ThothII/docs/install/server-workspace-registry.md

Server workspace-registry installation

This is the production workspace-registry companion to the autonomous Linux server guide. The application image is read-only, secrets are mounted read-only, and sessions use immutable Git-validated snapshots. Expose the application only behind an authenticated same-origin reverse proxy; never publish the core port directly.

Architecture ownership contract

Component	Ownership	Operator contract
DWH	External	Approved installation/server endpoint; not part of the private semantic Compose stack.
LLM	External	Approved installation/server endpoint or provider policy outside the semantic stack.
Qdrant	Internal	Mandatory private Compose semantic service; persistent qdrant-data volume.
Ollama embedding	Internal	Mandatory private Compose semantic service for qwen3-embedding:0.6b.

Service account, storage, and firewall

Create a dedicated host service account and an operator root such as /srv/thothii. The core container is non-root UID/GID 10001 (thoth), so give that identity read/write ownership before first startup. Keep storage separated:

/srv/thothii/data/                # settings, session data, Pi state as applicable
/srv/thothii/workspace-registry/  # repo/, snapshots/, state/, locks/
/srv/thothii/secrets/             # Git and connector secret files, setgid mode 2750
/srv/thothii/operator/            # untracked operator files, setgid mode 2770

After cloning the source and before the first render/start, initialize the empty Pi-state root with the repository setup command:

sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh \
  /srv/thothii/pi-state 10001 10001

The active server profile mounts that writable parent at /home/thoth/.pi and overlays three read-only files beneath agent/. The setup command atomically creates the required hidden regular targets with runtime ownership without copying secret or tracked file contents into writable state. Rerun it after a restore and before Compose or thothctl startup; it is idempotent and does not overwrite existing targets.

Permit outbound TCP only to approved Git/Gitea, DWH, LLM, and optional bastion endpoints. Qdrant and Ollama run inside the Compose stack. Allow inbound traffic only from the reverse proxy/Docker network. Do not give the runtime service account Gitea administration, database-superuser rights, or a shell in the Git host.

Gitea and remote Git setup

Create a private Gitea (or compatible Git) repository such as platform/thoth-workspaces. Protect main according to the release policy and grant the ThothII publisher only the intended repository scope. Commit canonical schema-v3 descriptors under workspaces/<id>.yaml, curated Evidence under workspace-content/<id>/evidence/**, and generated public artifacts only at workspace-docs/<id>/README.md and workspace-docs/<id>/contract.env.example; do not commit installation bindings or secret material.

For SSH, create a least-privilege deploy key, record Gitea's host key in managed known-hosts, and use ssh://git@git.example.invalid/platform/thoth-workspaces.git. For HTTPS, create a scoped machine credential in the secret manager and mount the Gitea/private CA separately. Never use a Gitea admin credential in the application.

Bootstrap an empty remote from a temporary review clone only after its canonical v3 descriptors and generated public artifacts have been reviewed; commit and push main. The running server is not a descriptor authoring or conversion environment.

Curator flow for shared-registry Evidence

Follow this order; the canonical Evidence contract defines the source shapes and safety boundary.

1. Clone the one shared registry, or update the review clone with git pull --ff-only.
2. Add source bytes below workspace-content/<id>/evidence, then commit and push.
3. Validate and publish the descriptor against that base commit.
4. Inspect workspace-docs/<id>/contract.env.example and workspace-docs/<id>/README.md.
5. Provision only the selected Evidence *_FILE files outside Git and strictly below a root in THT_WORKSPACE_SECRET_ROOTS; add matching host-only *_SOURCE paths for the generated connector override.
6. Render or acquire the runtime config, then run tht config check -c <path>.
7. Stop: P2/P6 later performs preprocessing and materialization.

For example, separate signed-HTTP and static-S3 workspaces can use these host-only connector source paths:

THT_WS_SIGNED_HTTP_EVIDENCE_SIGNED_URLS_SOURCE=/srv/thothii/secrets/signed-http-evidence-urls.json
THT_WS_STATIC_S3_EVIDENCE_ACCESS_KEY_SOURCE=/srv/thothii/secrets/static-s3-evidence-access-key
THT_WS_STATIC_S3_EVIDENCE_SECRET_KEY_SOURCE=/srv/thothii/secrets/static-s3-evidence-secret-key
THT_WS_STATIC_S3_EVIDENCE_SESSION_TOKEN_SOURCE=/srv/thothii/secrets/static-s3-evidence-session-token

The descriptor and declared filesystem root are validated at the same registry commit. The browser shows a read-only Evidence summary, while exports omit Evidence bytes.

Git credentials, CA, SSH key, and known-hosts mounts

Use the secret manager or a protected host-only procedure to create independent regular files under /srv/thothii/secrets. As established in the server guide, use owner UID 10001, group thothii-ops, file mode 0640, and setgid directory mode 2750. This lets the UID 10001 container and the reviewed Docker operator running thothctl read the files without making them public. These path-only variables are mounted read-only by Compose:

THT_WORKSPACE_GIT_CREDENTIALS_FILE=/srv/thothii/secrets/git-credentials
THT_WORKSPACE_GIT_CA_FILE=/srv/thothii/secrets/git-ca.pem
THT_WORKSPACE_GIT_SSH_KEY_FILE=/srv/thothii/secrets/git-ssh-key
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/srv/thothii/secrets/git-known-hosts
PI_AUTH_FILE=/srv/thothii/secrets/pi-auth.json
THT_SECRETS_FILE=/srv/thothii/secrets/thothii.secrets

Use the credential file for HTTPS, or key and known-hosts for SSH. The base server Compose file mounts neither transport; add exactly one deploy/compose.git-https.yaml or deploy/compose.git-ssh.yaml override. Strict host-key checking stays enabled and Git stderr is not exposed by the API. Rotate by atomically replacing the secret file, restarting core, and performing pull/status; never put the material in an environment variable or rendered Compose output.

Shared Git values, local bindings, and secret files

Git describes workspace schema, immutable ID, DWH identity, semantic-index dimensions and distance, internal embedding contract, and LLM policy. The installation supplies remote/branch/installation ID and one absolute THT_WORKSPACE_BINDINGS_ENV_FILE containing only THT_WS_* transport, endpoint, user, and /run/secrets/... path bindings. The base Compose loads that file only into core. Secret contents are only in host files, never the values stored in Git or browser-local drafts.

The runtime registry layout is persistent and must be backed up together:

/data/workspace-registry/repo/
/data/workspace-registry/snapshots/
/data/workspace-registry/state/
/data/workspace-registry/locks/

Variable names derive from the immutable ID: north-star-research becomes NORTH_STAR_RESEARCH, producing THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE. Keep the bindings file limited to DWH transport, endpoint, user, and secret-path values; internal semantic services are supplied by Compose and do not require workspace-local vector or embedding bindings. Copy the bindings env example to the protected operator directory. Every path-valued *_FILE entry needs an absolute host-only *_SOURCE path. Generate the untracked connector override from those files during bootstrap; do not copy or maintain a workspace-specific Compose override. Operator-managed path-only files use owner UID 10001, group thothii-ops, and mode 0660; secret files remain 0640 and non-group-writable.

Direct PostgreSQL, REST, and SSH tunnel bindings

Select only a transport allowed by canonical YAML; preserve database/schema/collection, model, dimensions, and distance as Git-shared identity.

# Direct PostgreSQL with verified native TLS if a CA path is supplied.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password

# REST needs API-key file paths only when the descriptor declares authenticated diagnostics.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key

# SSH tunnel diagnostic only; runtime sessions are fail-closed in this release.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432

REST diagnostics refuse private per-request CAs rather than disable verification; use runtime-trusted HTTPS or verified direct/SSH native TLS. See the diagnostic protocol for its read-only checks and optional reversible writer probe.

An SSH connector can be tested with strict host-key and target verification, but it intentionally returns workspace_not_activatable; configure direct or REST transport before starting sessions. The Git registry itself may still use SSH normally.

Same-origin reverse proxy, bootstrap, and health

Use the repository's canonical compose.yaml plus deploy/compose.server.yaml; they always start frontend, core, qdrant, embedding, and the one-shot embedding-model-init. Startup is CPU-first; use THOTH_ENABLE_EMBEDDING_GPU=1 only when the server intentionally exposes a supported GPU device to Docker. Qdrant is a derived but persistent index, and the Ollama model cache persists the exact qwen3-embedding:0.6b model (1024 dimensions, cosine distance) for offline reuse. Do not copy or maintain a standalone application Compose file. Copy docs/install/examples/thothii-installation.server.yaml to the protected operator directory and preserve its required session-server overlay, exactly one Git transport override, and generated connector-secret override.

Review deploy/workspaces/server-sessions.yaml.example, materialize it as a protected host file, and set its absolute THT_SERVER_WORKSPACE_CONFIG path. Copy the bindings env example into the operator directory, then set absolute PI_AUTH_FILE, THT_SECRETS_FILE, THT_WORKSPACE_BINDINGS_ENV_FILE, and connector *_SOURCE paths. The same operator env must set THT_SESSION_DB_HOST, THT_SESSION_DB_NAME, THT_SESSION_RUNTIME_USER, THT_SESSION_RUNTIME_PASSWORD_SOURCE, and THT_SESSION_CA_SOURCE; deploy/compose.session-server.yaml.example wires postgres, verify-full, and separate runtime/CA secret targets under /run/secrets. This public server profile never falls back to filesystem sessions. The path-only environment file is not shell code; do not source it.

Generate the connector override, then use the installation-aware operator CLI. Building thothctl requires only Docker and no Go knowledge. From a trusted maintenance shell:

umask 0007
THT_SOURCE_ROOT=/srv/thothii/source/ThothII
THT_OPERATOR_ENV=/srv/thothii/operator/server.env
THT_WORKSPACE_BINDINGS_ENV_FILE=/srv/thothii/operator/workspace-bindings.env
THT_CONNECTOR_OVERRIDE=/srv/thothii/operator/connector-secrets.server.yaml
THTCTL=/srv/thothii/operator/thothctl
INSTALLATION=/srv/thothii/operator/thothii-installation.yaml
sudo "$THT_SOURCE_ROOT/scripts/prepare-server-pi-state.sh" /srv/thothii/pi-state 10001 10001
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
  -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \
  -f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" config --quiet
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" \
  --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" \
  --operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
"$THTCTL" --installation "$INSTALLATION" update --check-only
"$THTCTL" --installation "$INSTALLATION" start
"$THTCTL" --installation "$INSTALLATION" status
"$THTCTL" --installation "$INSTALLATION" doctor
"$THTCTL" --installation "$INSTALLATION" pi doctor
"$THTCTL" --installation "$INSTALLATION" pi test

Configure Nginx or Caddy so frontend and /api share one TLS origin. The proxy authenticates first, clears client identity headers, and carries only successful authentication claims over the private X-Thoth-Trusted-* hop. Forwarding claims without authenticating the request is not an identity boundary.

/health is liveness. The authenticated Workspace Management page's registry status verifies branch/head/degraded state and the active validated snapshot; its workspace listing verifies application access. A server with no active snapshot is not ready for workspace sessions even if liveness succeeds.

Pull, publish, upgrade, backup, and recovery

Use the authenticated Workspace Management UI or POST /workspace-registry/pull. Drafts are browser-local. Publish takes a canonical diff, validates before commit, and pushes under a registry lock. On workspace_conflict, pull, resolve the reviewed field-level draft, validate/test, and publish; never edit repo/ inside a running volume.

For upgrades, record active status/head, finish active work, use the documented thothctl pi update --drain transaction when Pi/core changes, and take a stopped, filesystem-consistent backup of /srv/thothii/workspace-registry plus /srv/thothii/data and Pi state. Exclude /srv/thothii/secrets from the ordinary archive. Validate the descriptor with thothctl update --check-only, deploy the compatible image through thothctl, verify health/status, then resume proxy traffic.

Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are rejected before activation. Candidate snapshot validation makes initial activation or a pull fail atomically and leaves the prior active snapshot unchanged. There is no in-product migrator or automatic conversion. The repository must already contain reviewed v3 descriptors. One workspace owns one Qdrant collection; schema, Evidence, and Memory records share it and stay separated by payload kind.

If source material needs conversion, perform it outside ThothII in a separate reviewed process. Commit only the resulting reviewed v3 descriptors. That external process must not import ${ENV} values, secret values, certificates, keys, or secret files into the repository.

Semantic index ownership contract

Scope	Ownership rule	Isolation rule
Workspace semantic index	Each workspace reserves a single Qdrant collection.	Schema, Evidence, and Memory share that one collection and stay separated by payload kind.

After valid bootstrap, Git outage retains the active snapshot with degraded: true. Repair egress/DNS/CA/credentials, pull, and confirm healthy status. Roll back a bad descriptor through a reviewed Git revert/release branch, advance the remote through normal policy, pull it, and confirm the replacement snapshot. Restore a registry backup only while stopped and with a compatible image; do not delete snapshots as a rollback shortcut.

Troubleshooting and snapshot rollback

Stable code	Meaning and safe response
workspace_invalid	Invalid descriptor/path/snapshot; restore a reviewed canonical Git revision.
binding_missing	Missing/invalid local value or readable *_FILE; correct mount and permissions.
workspace_not_activatable	Bindings/diagnostics cannot activate; use sanitized fields to fix selected transport.
workspace_stale	Checkout changed/locked; stop concurrent registry work, never force Git in the volume.
workspace_conflict	Draft base stale; pull, resolve, validate, republish.
git_unavailable	Storage/remote/DNS/firewall/lock failed; preserve degraded active state while repairing it.
git_auth_failed	SSH/HTTPS material rejected or unreadable; rotate/fix file without printing it.
git_non_fast_forward	Checkout diverged; reconcile through registry workflow and branch policy.
git_push_rejected	Gitea policy rejected publish; review hooks/branch protection.
connector_unavailable	DNS/TLS/auth/resource identity failed; check egress and local bindings.
semantic_index_incompatible	Collection/model/dimensions/distance differs; perform explicit index migration.

If the current snapshot is valid but Git remains down, continue only work safe on that pinned revision and monitor status. If snapshots are missing or corrupt, stop the service, restore the newest verified registry backup, start it privately, verify status, and then reopen proxy traffic. A first-bootstrap failure has no fallback: repair remote trust rather than creating an unreviewed runtime checkout.

Qdrant backup/restore and cache recovery

Use the repository helpers for Qdrant backup/restore:

./scripts/vector-backup.sh --project-name thothii --output /secure/backups/thoth-qdrant-2026-08-08.tar
./scripts/vector-restore.sh --project-name thothii --input /secure/backups/thoth-qdrant-2026-08-08.tar --confirm-project thothii

Qdrant backup/restore targets exactly one labeled qdrant-data volume for the named Compose project. Restore requires the exact repeated project confirmation, validates the archive before stopping qdrant, stages rollback content, and restores semantic storage in place only for that project-scoped volume. Before recovery, the registry must already contain a reviewed v3 descriptor revision compatible with the restored collection. The helper does not restore descriptors, rename collections, or resolve semantic-index incompatibilities.

The Ollama model cache is a recoverable local cache, not the canonical semantic source of truth. You may back up embedding-models for faster offline recovery, but a cache loss is recoverable by re-pulling qwen3-embedding:0.6b through embedding-model-init.

Only the Git remote, DWH, LLM, and optional bastion endpoints stay external.