250 lines
14 KiB
Markdown
250 lines
14 KiB
Markdown
# Server workspace-registry installation
|
|
|
|
This is the production workspace-registry companion to [the autonomous Linux server guide](server.md).
|
|
The application image is read-only, secrets are mounted read-only, and sessions use immutable
|
|
Git-validated snapshots. Expose the application only behind an authenticated same-origin reverse
|
|
proxy; never publish the core port directly.
|
|
|
|
## Service account, storage, and firewall
|
|
|
|
Create a dedicated host service account and an operator root such as `/srv/thothii`. The core
|
|
container is non-root UID/GID `10001` (`thoth`), so give that identity read/write ownership before
|
|
first startup. Keep storage separated:
|
|
|
|
```text
|
|
/srv/thothii/data/ # settings, session data, Pi state as applicable
|
|
/srv/thothii/workspace-registry/ # repo/, snapshots/, state/, locks/
|
|
/srv/thothii/secrets/ # Git and connector secret files, setgid mode 2750
|
|
/srv/thothii/operator/ # untracked operator files, setgid mode 2770
|
|
```
|
|
|
|
After cloning the source and before the first render/start, initialize the empty Pi-state root with
|
|
the repository setup command:
|
|
|
|
```sh
|
|
sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh \
|
|
/srv/thothii/pi-state 10001 10001
|
|
```
|
|
|
|
The active server profile mounts that writable parent at `/home/thoth/.pi` and overlays three
|
|
read-only files beneath `agent/`. The setup command atomically creates the required hidden regular
|
|
targets with runtime ownership without copying secret or tracked file contents into writable
|
|
state. Rerun it after a restore and before Compose or `thothctl` startup; it is idempotent and does
|
|
not overwrite existing targets.
|
|
|
|
Permit outbound TCP only to approved Git/Gitea, DWH, vector, embedding, and bastion endpoints.
|
|
Allow inbound traffic only from the reverse proxy/Docker network. Do not give the runtime service
|
|
account Gitea administration, database-superuser rights, or a shell in the Git host.
|
|
|
|
## Gitea and remote Git setup
|
|
|
|
Create a private Gitea (or compatible Git) repository such as `platform/thoth-workspaces`. Protect
|
|
`main` according to the release policy and grant the ThothII publisher only the intended repository
|
|
scope. Commit canonical schema-v2 descriptors and generated `.md`/`.env.example` artifacts only;
|
|
do not commit installation bindings or secret material.
|
|
|
|
For SSH, create a least-privilege deploy key, record Gitea's host key in managed known-hosts, and
|
|
use `ssh://git@git.example.invalid/platform/thoth-workspaces.git`. For HTTPS, create a scoped
|
|
machine credential in the secret manager and mount the Gitea/private CA separately. Never use a
|
|
Gitea admin credential in the application.
|
|
|
|
Bootstrap an empty remote from a temporary review clone: migrate legacy descriptors, review their
|
|
schema-v2 identity and generated artifacts, commit, and push `main`. The running server is not an
|
|
authoring environment for migration.
|
|
|
|
## Git credentials, CA, SSH key, and known-hosts mounts
|
|
|
|
Use the secret manager or a protected host-only procedure to create independent regular files under
|
|
`/srv/thothii/secrets`. As established in the server guide, use owner UID 10001, group
|
|
`thothii-ops`, file mode `0640`, and setgid directory mode `2750`. This lets the UID 10001
|
|
container and the reviewed Docker operator running `thothctl` read the files without making them
|
|
public. These path-only variables are mounted read-only by Compose:
|
|
|
|
```dotenv
|
|
THT_WORKSPACE_GIT_CREDENTIALS_FILE=/srv/thothii/secrets/git-credentials
|
|
THT_WORKSPACE_GIT_CA_FILE=/srv/thothii/secrets/git-ca.pem
|
|
THT_WORKSPACE_GIT_SSH_KEY_FILE=/srv/thothii/secrets/git-ssh-key
|
|
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/srv/thothii/secrets/git-known-hosts
|
|
PI_AUTH_FILE=/srv/thothii/secrets/pi-auth.json
|
|
THT_SECRETS_FILE=/srv/thothii/secrets/thothii.secrets
|
|
```
|
|
|
|
Use the credential file for HTTPS, or key and known-hosts for SSH. The base server Compose file
|
|
mounts neither transport; add exactly one `deploy/compose.git-https.yaml`
|
|
or `deploy/compose.git-ssh.yaml` override. Strict host-key checking stays enabled
|
|
and Git stderr is not exposed by the API. Rotate by atomically replacing the secret file,
|
|
restarting `core`, and performing pull/status; never put the material in an environment variable or
|
|
rendered Compose output.
|
|
|
|
## Shared Git values, local bindings, and secret files
|
|
|
|
Git describes workspace schema, immutable ID, DWH/vector identity, semantic-index dimensions and
|
|
distance, embedding contract, and LLM policy. The installation supplies remote/branch/installation
|
|
ID and one absolute `THT_WORKSPACE_BINDINGS_ENV_FILE` containing only `THT_WS_*` transport,
|
|
endpoint, user, and `/run/secrets/...` path bindings. The base Compose loads that file only into
|
|
`core`. Secret contents are only in host files, never the values stored in Git or browser-local
|
|
drafts.
|
|
|
|
The runtime registry layout is persistent and must be backed up together:
|
|
|
|
```text
|
|
/data/workspace-registry/repo/
|
|
/data/workspace-registry/snapshots/
|
|
/data/workspace-registry/state/
|
|
/data/workspace-registry/locks/
|
|
```
|
|
|
|
Variable names derive from the immutable ID: `north-star-research` becomes `NORTH_STAR_RESEARCH`, producing
|
|
`THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE`. A declared vector writer uses the distinct
|
|
`THT_WS_NORTH_STAR_RESEARCH_VECTOR_WRITER_API_KEY_FILE`; a reader file is never a writer substitute.
|
|
Copy [the bindings env example](examples/workspace-bindings.env.example) to the protected operator
|
|
directory. Every path-valued `*_FILE` entry needs an absolute host-only `*_SOURCE` path. Generate
|
|
the untracked connector override from those files during bootstrap; do not copy or maintain a
|
|
workspace-specific Compose override. Operator-managed path-only files use owner UID 10001, group
|
|
`thothii-ops`, and mode `0660`; secret files remain `0640` and non-group-writable.
|
|
|
|
## Direct PostgreSQL, REST, and SSH tunnel bindings
|
|
|
|
Select only a transport allowed by canonical YAML; preserve database/schema/collection, model,
|
|
dimensions, and distance as Git-shared identity.
|
|
|
|
```dotenv
|
|
# Direct PostgreSQL with verified native TLS if a CA path is supplied.
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.internal.example
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
|
|
```
|
|
|
|
```dotenv
|
|
# REST needs API-key file paths only when the descriptor declares authenticated diagnostics.
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.internal.example
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
|
|
```
|
|
|
|
```dotenv
|
|
# SSH tunnel diagnostic only; runtime sessions are fail-closed in this release.
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.internal.example
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
|
|
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
|
|
```
|
|
|
|
REST diagnostics refuse private per-request CAs
|
|
rather than disable verification; use runtime-trusted HTTPS or verified direct/SSH native TLS. See
|
|
the [diagnostic protocol](../workspace-diagnostic-protocol.md) for its read-only checks and optional
|
|
reversible writer probe.
|
|
|
|
An SSH connector can be tested with strict host-key and target verification, but it intentionally
|
|
returns `workspace_not_activatable`; configure direct or REST transport before starting sessions.
|
|
The Git registry itself may still use SSH normally.
|
|
|
|
## Same-origin reverse proxy, bootstrap, and health
|
|
|
|
Use the repository's canonical `compose.yaml` plus `deploy/compose.server.yaml`; they always
|
|
start the mandatory `frontend` and `core` services. Do not copy or maintain a standalone
|
|
application Compose file. Copy `docs/install/examples/thothii-installation.server.yaml` to the
|
|
protected operator directory and preserve its required session-server overlay, exactly one Git
|
|
transport override, and generated connector-secret override.
|
|
|
|
Review `deploy/workspaces/server-sessions.yaml.example`, materialize it as a protected host file,
|
|
and set its absolute `THT_SERVER_WORKSPACE_CONFIG` path. Copy the bindings env example into the
|
|
operator directory, then set absolute `PI_AUTH_FILE`, `THT_SECRETS_FILE`,
|
|
`THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths. The same operator env must set
|
|
`THT_SESSION_DB_HOST`, `THT_SESSION_DB_NAME`, `THT_SESSION_RUNTIME_USER`,
|
|
`THT_SESSION_RUNTIME_PASSWORD_SOURCE`, and `THT_SESSION_CA_SOURCE`;
|
|
`deploy/compose.session-server.yaml.example` wires `postgres`, `verify-full`, and separate
|
|
runtime/CA secret targets under `/run/secrets`. This public server profile never falls back to
|
|
filesystem sessions. The path-only environment file is not shell code; do not source it.
|
|
|
|
Generate the connector override, then use the installation-aware operator CLI. Building
|
|
`thothctl` requires only Docker and no Go knowledge. From a trusted maintenance shell:
|
|
|
|
```sh
|
|
umask 0007
|
|
THT_SOURCE_ROOT=/srv/thothii/source/ThothII
|
|
THT_OPERATOR_ENV=/srv/thothii/operator/server.env
|
|
THT_WORKSPACE_BINDINGS_ENV_FILE=/srv/thothii/operator/workspace-bindings.env
|
|
THT_CONNECTOR_OVERRIDE=/srv/thothii/operator/connector-secrets.server.yaml
|
|
THTCTL=/srv/thothii/operator/thothctl
|
|
INSTALLATION=/srv/thothii/operator/thothii-installation.yaml
|
|
sudo "$THT_SOURCE_ROOT/scripts/prepare-server-pi-state.sh" /srv/thothii/pi-state 10001 10001
|
|
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
|
|
-f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \
|
|
-f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" config --quiet
|
|
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" \
|
|
--bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" \
|
|
--operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
|
|
"$THTCTL" --installation "$INSTALLATION" update --check-only
|
|
"$THTCTL" --installation "$INSTALLATION" start
|
|
"$THTCTL" --installation "$INSTALLATION" status
|
|
"$THTCTL" --installation "$INSTALLATION" doctor
|
|
"$THTCTL" --installation "$INSTALLATION" pi doctor
|
|
"$THTCTL" --installation "$INSTALLATION" pi test
|
|
```
|
|
|
|
Configure [Nginx](reverse-proxy-nginx.md) or [Caddy](reverse-proxy-caddy.md) so frontend and `/api`
|
|
share one TLS origin. The proxy authenticates first, clears client identity headers, and carries
|
|
only successful authentication claims over the private `X-Thoth-Trusted-*` hop. Forwarding claims
|
|
without authenticating the request is not an identity boundary.
|
|
|
|
`/health` is liveness. The authenticated Workspace Management page's registry status verifies
|
|
branch/head/degraded state and the active validated snapshot; its workspace listing verifies
|
|
application access. A server with no active snapshot is not ready for workspace sessions even if
|
|
liveness succeeds.
|
|
|
|
## Pull, publish, upgrade, backup, and recovery
|
|
|
|
Use the authenticated Workspace Management UI or `POST /workspace-registry/pull`. Drafts are
|
|
browser-local. Publish takes a canonical diff, validates before commit, and pushes under a registry
|
|
lock. On `workspace_conflict`, pull, resolve the reviewed field-level draft, validate/test, and
|
|
publish; never edit `repo/` inside a running volume.
|
|
|
|
For upgrades, record active status/head, finish active work, use the documented `thothctl pi update
|
|
--drain` transaction when Pi/core changes, and take a stopped, filesystem-consistent backup of
|
|
`/srv/thothii/workspace-registry` plus `/srv/thothii/data` and Pi state. Exclude
|
|
`/srv/thothii/secrets` from the ordinary archive. Validate the descriptor with `thothctl update
|
|
--check-only`, deploy the compatible image through `thothctl`, verify health/status, then resume
|
|
proxy traffic.
|
|
|
|
For legacy descriptor migration, use a temporary review clone and the legacy transformer with absolute paths.
|
|
Its schema-v1 output is `migration_required`; explicitly supply vector database/schema, collection
|
|
identity, diagnostics, and the reviewed v2 contract before commit. Never import `${ENV}` values or
|
|
copy secret files.
|
|
|
|
After valid bootstrap, Git outage retains the active snapshot with `degraded: true`. Repair
|
|
egress/DNS/CA/credentials, pull, and confirm healthy status. Roll back a bad descriptor through a
|
|
reviewed Git revert/release branch, advance the remote through normal policy, pull it, and confirm
|
|
the replacement snapshot. Restore a registry backup only while stopped and with a compatible image;
|
|
do not delete snapshots as a rollback shortcut.
|
|
|
|
## Troubleshooting and snapshot rollback
|
|
|
|
| Stable code | Meaning and safe response |
|
|
| --- | --- |
|
|
| `workspace_invalid` | Invalid descriptor/path/snapshot; restore a reviewed canonical Git revision. |
|
|
| `binding_missing` | Missing/invalid local value or readable `*_FILE`; correct mount and permissions. |
|
|
| `workspace_not_activatable` | Bindings/diagnostics cannot activate; use sanitized fields to fix selected transport. |
|
|
| `workspace_stale` | Checkout changed/locked; stop concurrent registry work, never force Git in the volume. |
|
|
| `workspace_conflict` | Draft base stale; pull, resolve, validate, republish. |
|
|
| `git_unavailable` | Storage/remote/DNS/firewall/lock failed; preserve degraded active state while repairing it. |
|
|
| `git_auth_failed` | SSH/HTTPS material rejected or unreadable; rotate/fix file without printing it. |
|
|
| `git_non_fast_forward` | Checkout diverged; reconcile through registry workflow and branch policy. |
|
|
| `git_push_rejected` | Gitea policy rejected publish; review hooks/branch protection. |
|
|
| `connector_unavailable` | DNS/TLS/auth/resource identity failed; check egress and local bindings. |
|
|
| `semantic_index_incompatible` | Collection/model/dimensions/distance differs; perform explicit index migration. |
|
|
|
|
If the current snapshot is valid but Git remains down, continue only work safe on that pinned
|
|
revision and monitor status. If snapshots are missing or corrupt, stop the service, restore the
|
|
newest verified registry backup, start it privately, verify status, and then reopen proxy traffic.
|
|
A first-bootstrap failure has no fallback: repair remote trust rather than creating an unreviewed
|
|
runtime checkout.
|