# Server workspace-registry installation

This is the production workspace-registry companion to [the autonomous Linux server guide](server.md).
The application image is read-only, secrets are mounted read-only, and sessions use immutable
Git-validated snapshots. Expose the application only behind an authenticated same-origin reverse
proxy; never publish the core port directly.

## Service account, storage, and firewall

Create a dedicated host service account and an operator root such as `/srv/thothii`. The core
container is non-root UID/GID `10001` (`thoth`), so give that identity read/write ownership before
first startup. Keep storage separated:

```text
/srv/thothii/data/                # settings, session data, Pi state as applicable
/srv/thothii/workspace-registry/  # repo/, snapshots/, state/, locks/
/srv/thothii/secrets/             # Git and connector secret files, setgid mode 2750
/srv/thothii/operator/            # untracked operator files, setgid mode 2770
```

After cloning the source and before the first render/start, initialize the empty Pi-state root with
the repository setup command:

```sh
sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh \
  /srv/thothii/pi-state 10001 10001
```

The active server profile mounts that writable parent at `/home/thoth/.pi` and overlays three
read-only files beneath `agent/`. The setup command atomically creates the required hidden regular
targets with runtime ownership without copying secret or tracked file contents into writable
state. Rerun it after a restore and before Compose or `thothctl` startup; it is idempotent and does
not overwrite existing targets.

Permit outbound TCP only to approved Git/Gitea, DWH, Qdrant, embedding, and bastion endpoints.
Allow inbound traffic only from the reverse proxy/Docker network. Do not give the runtime service
account Gitea administration, database-superuser rights, or a shell in the Git host.

## Gitea and remote Git setup

Create a private Gitea (or compatible Git) repository such as `platform/thoth-workspaces`. Protect
`main` according to the release policy and grant the ThothII publisher only the intended repository
scope. Commit canonical schema-v3 descriptors and generated `.md`/`.env.example` artifacts only;
do not commit installation bindings or secret material.

For SSH, create a least-privilege deploy key, record Gitea's host key in managed known-hosts, and
use `ssh://git@git.example.invalid/platform/thoth-workspaces.git`. For HTTPS, create a scoped
machine credential in the secret manager and mount the Gitea/private CA separately. Never use a
Gitea admin credential in the application.

Bootstrap an empty remote from a temporary review clone: migrate legacy descriptors, review their
schema-v3 identity and generated artifacts, commit, and push `main`. The running server is not an
authoring environment for migration.

## Git credentials, CA, SSH key, and known-hosts mounts

Use the secret manager or a protected host-only procedure to create independent regular files under
`/srv/thothii/secrets`. As established in the server guide, use owner UID 10001, group
`thothii-ops`, file mode `0640`, and setgid directory mode `2750`. This lets the UID 10001
container and the reviewed Docker operator running `thothctl` read the files without making them
public. These path-only variables are mounted read-only by Compose:

```dotenv
THT_WORKSPACE_GIT_CREDENTIALS_FILE=/srv/thothii/secrets/git-credentials
THT_WORKSPACE_GIT_CA_FILE=/srv/thothii/secrets/git-ca.pem
THT_WORKSPACE_GIT_SSH_KEY_FILE=/srv/thothii/secrets/git-ssh-key
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/srv/thothii/secrets/git-known-hosts
PI_AUTH_FILE=/srv/thothii/secrets/pi-auth.json
THT_SECRETS_FILE=/srv/thothii/secrets/thothii.secrets
```

Use the credential file for HTTPS, or key and known-hosts for SSH. The base server Compose file
mounts neither transport; add exactly one `deploy/compose.git-https.yaml`
or `deploy/compose.git-ssh.yaml` override. Strict host-key checking stays enabled
and Git stderr is not exposed by the API. Rotate by atomically replacing the secret file,
restarting `core`, and performing pull/status; never put the material in an environment variable or
rendered Compose output.

## Shared Git values, local bindings, and secret files

Git describes workspace schema, immutable ID, DWH identity, semantic-index dimensions and
distance, internal embedding contract, and LLM policy. The installation supplies remote/branch/installation
ID and one absolute `THT_WORKSPACE_BINDINGS_ENV_FILE` containing only `THT_WS_*` transport,
endpoint, user, and `/run/secrets/...` path bindings. The base Compose loads that file only into
`core`. Secret contents are only in host files, never the values stored in Git or browser-local
drafts.

The runtime registry layout is persistent and must be backed up together:

```text
/data/workspace-registry/repo/
/data/workspace-registry/snapshots/
/data/workspace-registry/state/
/data/workspace-registry/locks/
```

Variable names derive from the immutable ID: `north-star-research` becomes `NORTH_STAR_RESEARCH`, producing
`THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE`. Keep the bindings file limited to DWH transport,
endpoint, user, and secret-path values; internal semantic services are supplied by Compose and do
not require workspace-local vector or embedding bindings.
Copy [the bindings env example](examples/workspace-bindings.env.example) to the protected operator
directory. Every path-valued `*_FILE` entry needs an absolute host-only `*_SOURCE` path. Generate
the untracked connector override from those files during bootstrap; do not copy or maintain a
workspace-specific Compose override. Operator-managed path-only files use owner UID 10001, group
`thothii-ops`, and mode `0660`; secret files remain `0640` and non-group-writable.

## Direct PostgreSQL, REST, and SSH tunnel bindings

Select only a transport allowed by canonical YAML; preserve database/schema/collection, model,
dimensions, and distance as Git-shared identity.

```dotenv
# Direct PostgreSQL with verified native TLS if a CA path is supplied.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
```

```dotenv
# REST needs API-key file paths only when the descriptor declares authenticated diagnostics.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
```

```dotenv
# SSH tunnel diagnostic only; runtime sessions are fail-closed in this release.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
```

REST diagnostics refuse private per-request CAs
rather than disable verification; use runtime-trusted HTTPS or verified direct/SSH native TLS. See
the [diagnostic protocol](../workspace-diagnostic-protocol.md) for its read-only checks and optional
reversible writer probe.

An SSH connector can be tested with strict host-key and target verification, but it intentionally
returns `workspace_not_activatable`; configure direct or REST transport before starting sessions.
The Git registry itself may still use SSH normally.

## Same-origin reverse proxy, bootstrap, and health

Use the repository's canonical `compose.yaml` plus `deploy/compose.server.yaml`; they always
start the mandatory `frontend` and `core` services. Do not copy or maintain a standalone
application Compose file. Copy `docs/install/examples/thothii-installation.server.yaml` to the
protected operator directory and preserve its required session-server overlay, exactly one Git
transport override, and generated connector-secret override.

Review `deploy/workspaces/server-sessions.yaml.example`, materialize it as a protected host file,
and set its absolute `THT_SERVER_WORKSPACE_CONFIG` path. Copy the bindings env example into the
operator directory, then set absolute `PI_AUTH_FILE`, `THT_SECRETS_FILE`,
`THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths. The same operator env must set
`THT_SESSION_DB_HOST`, `THT_SESSION_DB_NAME`, `THT_SESSION_RUNTIME_USER`,
`THT_SESSION_RUNTIME_PASSWORD_SOURCE`, and `THT_SESSION_CA_SOURCE`;
`deploy/compose.session-server.yaml.example` wires `postgres`, `verify-full`, and separate
runtime/CA secret targets under `/run/secrets`. This public server profile never falls back to
filesystem sessions. The path-only environment file is not shell code; do not source it.

Generate the connector override, then use the installation-aware operator CLI. Building
`thothctl` requires only Docker and no Go knowledge. From a trusted maintenance shell:

```sh
umask 0007
THT_SOURCE_ROOT=/srv/thothii/source/ThothII
THT_OPERATOR_ENV=/srv/thothii/operator/server.env
THT_WORKSPACE_BINDINGS_ENV_FILE=/srv/thothii/operator/workspace-bindings.env
THT_CONNECTOR_OVERRIDE=/srv/thothii/operator/connector-secrets.server.yaml
THTCTL=/srv/thothii/operator/thothctl
INSTALLATION=/srv/thothii/operator/thothii-installation.yaml
sudo "$THT_SOURCE_ROOT/scripts/prepare-server-pi-state.sh" /srv/thothii/pi-state 10001 10001
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
  -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \
  -f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" config --quiet
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" \
  --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" \
  --operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
"$THTCTL" --installation "$INSTALLATION" update --check-only
"$THTCTL" --installation "$INSTALLATION" start
"$THTCTL" --installation "$INSTALLATION" status
"$THTCTL" --installation "$INSTALLATION" doctor
"$THTCTL" --installation "$INSTALLATION" pi doctor
"$THTCTL" --installation "$INSTALLATION" pi test
```

Configure [Nginx](reverse-proxy-nginx.md) or [Caddy](reverse-proxy-caddy.md) so frontend and `/api`
share one TLS origin. The proxy authenticates first, clears client identity headers, and carries
only successful authentication claims over the private `X-Thoth-Trusted-*` hop. Forwarding claims
without authenticating the request is not an identity boundary.

`/health` is liveness. The authenticated Workspace Management page's registry status verifies
branch/head/degraded state and the active validated snapshot; its workspace listing verifies
application access. A server with no active snapshot is not ready for workspace sessions even if
liveness succeeds.

## Pull, publish, upgrade, backup, and recovery

Use the authenticated Workspace Management UI or `POST /workspace-registry/pull`. Drafts are
browser-local. Publish takes a canonical diff, validates before commit, and pushes under a registry
lock. On `workspace_conflict`, pull, resolve the reviewed field-level draft, validate/test, and
publish; never edit `repo/` inside a running volume.

For upgrades, record active status/head, finish active work, use the documented `thothctl pi update
--drain` transaction when Pi/core changes, and take a stopped, filesystem-consistent backup of
`/srv/thothii/workspace-registry` plus `/srv/thothii/data` and Pi state. Exclude
`/srv/thothii/secrets` from the ordinary archive. Validate the descriptor with `thothctl update
--check-only`, deploy the compatible image through `thothctl`, verify health/status, then resume
proxy traffic.

For legacy descriptor migration, use a temporary review clone and the legacy transformer with absolute paths.
Its schema-v1 output is `migration_required`; explicitly supply collection identity, diagnostics,
and the reviewed v3 contract before commit. Never import `${ENV}` values or
copy secret files.

After valid bootstrap, Git outage retains the active snapshot with `degraded: true`. Repair
egress/DNS/CA/credentials, pull, and confirm healthy status. Roll back a bad descriptor through a
reviewed Git revert/release branch, advance the remote through normal policy, pull it, and confirm
the replacement snapshot. Restore a registry backup only while stopped and with a compatible image;
do not delete snapshots as a rollback shortcut.

## Troubleshooting and snapshot rollback

| Stable code | Meaning and safe response |
| --- | --- |
| `workspace_invalid` | Invalid descriptor/path/snapshot; restore a reviewed canonical Git revision. |
| `binding_missing` | Missing/invalid local value or readable `*_FILE`; correct mount and permissions. |
| `workspace_not_activatable` | Bindings/diagnostics cannot activate; use sanitized fields to fix selected transport. |
| `workspace_stale` | Checkout changed/locked; stop concurrent registry work, never force Git in the volume. |
| `workspace_conflict` | Draft base stale; pull, resolve, validate, republish. |
| `git_unavailable` | Storage/remote/DNS/firewall/lock failed; preserve degraded active state while repairing it. |
| `git_auth_failed` | SSH/HTTPS material rejected or unreadable; rotate/fix file without printing it. |
| `git_non_fast_forward` | Checkout diverged; reconcile through registry workflow and branch policy. |
| `git_push_rejected` | Gitea policy rejected publish; review hooks/branch protection. |
| `connector_unavailable` | DNS/TLS/auth/resource identity failed; check egress and local bindings. |
| `semantic_index_incompatible` | Collection/model/dimensions/distance differs; perform explicit index migration. |

If the current snapshot is valid but Git remains down, continue only work safe on that pinned
revision and monitor status. If snapshots are missing or corrupt, stop the service, restore the
newest verified registry backup, start it privately, verify status, and then reopen proxy traffic.
A first-bootstrap failure has no fallback: repair remote trust rather than creating an unreviewed
runtime checkout.