196 lines
10 KiB
Markdown
196 lines
10 KiB
Markdown
# Server workspace-registry installation
|
|
|
|
This is the production operator guide. The application image is read-only, secrets are mounted
|
|
read-only, and sessions use immutable Git-validated snapshots. Expose the application only behind
|
|
an authenticated same-origin reverse proxy; never publish the core port directly.
|
|
|
|
## Service account, storage, and firewall
|
|
|
|
Create a dedicated host service account and an operator root such as `/srv/thothii`. The core
|
|
container is non-root UID/GID `10001` (`thoth`), so give that identity read/write ownership before
|
|
first startup. Keep storage separated:
|
|
|
|
```text
|
|
/srv/thothii/data/ # settings, session data, Pi state as applicable
|
|
/srv/thothii/workspace-registry/ # repo/, snapshots/, state/, locks/
|
|
/srv/thothii/secrets/ # Git and connector secret files, mode 0700
|
|
/srv/thothii/operator/ # untracked Compose/.env, mode 0700
|
|
```
|
|
|
|
Permit outbound TCP only to approved Git/Gitea, DWH, vector, embedding, and bastion endpoints.
|
|
Allow inbound traffic only from the reverse proxy/Docker network. Do not give the runtime service
|
|
account Gitea administration, database-superuser rights, or a shell in the Git host.
|
|
|
|
## Gitea and remote Git setup
|
|
|
|
Create a private Gitea (or compatible Git) repository such as `platform/thoth-workspaces`. Protect
|
|
`main` according to the release policy and grant the ThothII publisher only the intended repository
|
|
scope. Commit canonical schema-v2 descriptors and generated `.md`/`.env.example` artifacts only;
|
|
do not commit installation bindings or secret material.
|
|
|
|
For SSH, create a least-privilege deploy key, record Gitea's host key in managed known-hosts, and
|
|
use `ssh://git@git.example.invalid/platform/thoth-workspaces.git`. For HTTPS, create a scoped
|
|
machine credential in the secret manager and mount the Gitea/private CA separately. Never use a
|
|
Gitea admin credential in the application.
|
|
|
|
Bootstrap an empty remote from a temporary review clone: migrate legacy descriptors, review their
|
|
schema-v2 identity and generated artifacts, commit, and push `main`. The running server is not an
|
|
authoring environment for migration.
|
|
|
|
## Git credentials, CA, SSH key, and known-hosts mounts
|
|
|
|
Use the secret manager or a protected host-only procedure to create independent regular files under
|
|
`/srv/thothii/secrets`. Set individual mode `0600`, directory mode `0700`, and ownership readable
|
|
by the service account. These path-only variables are mounted read-only by Compose:
|
|
|
|
```dotenv
|
|
THT_WORKSPACE_GIT_CREDENTIALS_FILE=/srv/thothii/secrets/git-credentials
|
|
THT_WORKSPACE_GIT_CA_FILE=/srv/thothii/secrets/git-ca.pem
|
|
THT_WORKSPACE_GIT_SSH_KEY_FILE=/srv/thothii/secrets/git-ssh-key
|
|
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/srv/thothii/secrets/git-known-hosts
|
|
```
|
|
|
|
Use the credential file for HTTPS, or key and known-hosts for SSH. Strict host-key checking stays
|
|
enabled and Git stderr is not exposed by the API. Rotate by atomically replacing the secret file,
|
|
restarting `core`, and performing pull/status; never put the material in an environment variable or
|
|
`docker compose config` output.
|
|
|
|
## Shared Git values, local bindings, and secret files
|
|
|
|
Git describes workspace schema, immutable ID, DWH/vector identity, semantic-index dimensions and
|
|
distance, embedding contract, and LLM policy. The installation supplies remote/branch/installation
|
|
ID, transport, endpoints, users, ports, and `THT_WS_*` bindings. Secret contents are only in files,
|
|
never the values stored in Git or browser-local drafts.
|
|
|
|
The runtime registry layout is persistent and must be backed up together:
|
|
|
|
```text
|
|
/data/workspace-registry/repo/
|
|
/data/workspace-registry/snapshots/
|
|
/data/workspace-registry/state/
|
|
/data/workspace-registry/locks/
|
|
```
|
|
|
|
Variable names derive from the immutable ID: `psd-clinical` becomes `PSD_CLINICAL`, producing
|
|
`THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE`. A declared vector writer uses the distinct
|
|
`THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE`; a reader file is never a writer substitute.
|
|
|
|
## Direct PostgreSQL, REST, and SSH tunnel bindings
|
|
|
|
Select only a transport allowed by canonical YAML; preserve database/schema/collection, model,
|
|
dimensions, and distance as Git-shared identity.
|
|
|
|
```dotenv
|
|
# Direct PostgreSQL/pgvector with verified native TLS if a CA path is supplied.
|
|
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=postgres_direct
|
|
THT_WS_PSD_CLINICAL_DWH_HOST=dwh.internal.example
|
|
THT_WS_PSD_CLINICAL_DWH_PORT=5432
|
|
THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader
|
|
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader
|
|
THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=pgvector_direct
|
|
THT_WS_PSD_CLINICAL_VECTOR_HOST=vector.internal.example
|
|
THT_WS_PSD_CLINICAL_VECTOR_PORT=5432
|
|
THT_WS_PSD_CLINICAL_VECTOR_USER=thoth_vector_reader
|
|
THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE=/run/secrets/psd-vector-reader
|
|
```
|
|
|
|
```dotenv
|
|
# REST needs API-key file paths only when the descriptor declares authenticated diagnostics.
|
|
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=rest_api
|
|
THT_WS_PSD_CLINICAL_DWH_BASE_URL=https://dwh.internal.example
|
|
THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE=/run/secrets/psd-dwh-api-key
|
|
THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=rest_api
|
|
THT_WS_PSD_CLINICAL_VECTOR_BASE_URL=https://vectors.internal.example
|
|
THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE=/run/secrets/psd-vector-api-key
|
|
THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL=https://embeddings.internal.example
|
|
```
|
|
|
|
```dotenv
|
|
# SSH tunnel requires explicit host-key verification and TLS target identity.
|
|
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=ssh_tunnel
|
|
THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader
|
|
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader
|
|
THT_WS_PSD_CLINICAL_DWH_SSH_HOST=bastion.internal.example
|
|
THT_WS_PSD_CLINICAL_DWH_SSH_PORT=22
|
|
THT_WS_PSD_CLINICAL_DWH_SSH_USER=thoth_tunnel
|
|
THT_WS_PSD_CLINICAL_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/psd-dwh-tunnel-key
|
|
THT_WS_PSD_CLINICAL_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/psd-dwh-known-hosts
|
|
THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_HOST=dwh.internal.example
|
|
THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_PORT=5432
|
|
```
|
|
|
|
Repeat SSH variables for `VECTOR` when selected. REST diagnostics refuse private per-request CAs
|
|
rather than disable verification; use runtime-trusted HTTPS or verified direct/SSH native TLS. See
|
|
the [diagnostic protocol](../workspace-diagnostic-protocol.md) for its read-only checks and optional
|
|
reversible writer probe.
|
|
|
|
## Same-origin reverse proxy, bootstrap, and health
|
|
|
|
Copy [the server Compose example](examples/server-compose.workspace-registry.yaml) to the protected
|
|
operator directory, set host paths/remote/branch/installation ID/portal network in local `.env`,
|
|
then render it before deployment.
|
|
|
|
<!-- verify:command -->
|
|
```sh
|
|
docker compose -f docs/install/examples/server-compose.workspace-registry.yaml config --quiet
|
|
```
|
|
|
|
Configure the portal proxy so the frontend and `/api` share one origin. It authenticates first and
|
|
forwards the trusted identity expected by `AUTH_MODE=upstream`; it is the only public listener.
|
|
From a trusted maintenance shell:
|
|
|
|
```sh
|
|
docker compose -f compose.workspace-registry.yaml up --build -d
|
|
docker compose -f compose.workspace-registry.yaml exec -T core curl --fail --silent http://127.0.0.1:8787/health
|
|
docker compose -f compose.workspace-registry.yaml exec -T core curl --fail --silent http://127.0.0.1:8787/workspace-registry/status
|
|
```
|
|
|
|
`/health` is liveness. Registry status verifies branch/head/degraded state and the active validated
|
|
snapshot; authenticated `/workspaces` verifies application access. A server with no active snapshot
|
|
is not ready for workspace sessions even if liveness succeeds.
|
|
|
|
## Pull, publish, upgrade, backup, and recovery
|
|
|
|
Use the authenticated Workspace Management UI or `POST /workspace-registry/pull`. Drafts are
|
|
browser-local. Publish takes a canonical diff, validates before commit, and pushes under a registry
|
|
lock. On `workspace_conflict`, pull, resolve the reviewed field-level draft, validate/test, and
|
|
publish; never edit `repo/` inside a running volume.
|
|
|
|
For upgrades, record active status/head, drain active Pi work, stop `core`, and take a
|
|
filesystem-consistent backup of `/srv/thothii/workspace-registry` plus `/srv/thothii/data`. Exclude
|
|
`/srv/thothii/secrets`. Render Compose, deploy the compatible image, verify health/status, then
|
|
resume proxy traffic.
|
|
|
|
For PSD migration, use a temporary review clone and the legacy transformer with absolute paths.
|
|
Its schema-v1 output is `migration_required`; explicitly supply vector database/schema, collection
|
|
identity, diagnostics, and the reviewed v2 contract before commit. Never import `${ENV}` values or
|
|
copy secret files.
|
|
|
|
After valid bootstrap, Git outage retains the active snapshot with `degraded: true`. Repair
|
|
egress/DNS/CA/credentials, pull, and confirm healthy status. Roll back a bad descriptor through a
|
|
reviewed Git revert/release branch, advance the remote through normal policy, pull it, and confirm
|
|
the replacement snapshot. Restore a registry backup only while stopped and with a compatible image;
|
|
do not delete snapshots as a rollback shortcut.
|
|
|
|
## Troubleshooting and snapshot rollback
|
|
|
|
| Stable code | Meaning and safe response |
|
|
| --- | --- |
|
|
| `workspace_invalid` | Invalid descriptor/path/snapshot; restore a reviewed canonical Git revision. |
|
|
| `binding_missing` | Missing/invalid local value or readable `*_FILE`; correct mount and permissions. |
|
|
| `workspace_not_activatable` | Bindings/diagnostics cannot activate; use sanitized fields to fix selected transport. |
|
|
| `workspace_stale` | Checkout changed/locked; stop concurrent registry work, never force Git in the volume. |
|
|
| `workspace_conflict` | Draft base stale; pull, resolve, validate, republish. |
|
|
| `git_unavailable` | Storage/remote/DNS/firewall/lock failed; preserve degraded active state while repairing it. |
|
|
| `git_auth_failed` | SSH/HTTPS material rejected or unreadable; rotate/fix file without printing it. |
|
|
| `git_non_fast_forward` | Checkout diverged; reconcile through registry workflow and branch policy. |
|
|
| `git_push_rejected` | Gitea policy rejected publish; review hooks/branch protection. |
|
|
| `connector_unavailable` | DNS/TLS/auth/resource identity failed; check egress and local bindings. |
|
|
| `semantic_index_incompatible` | Collection/model/dimensions/distance differs; perform explicit index migration. |
|
|
|
|
If the current snapshot is valid but Git remains down, continue only work safe on that pinned
|
|
revision and monitor status. If snapshots are missing or corrupt, stop the service, restore the
|
|
newest verified registry backup, start it privately, verify status, and then reopen proxy traffic.
|
|
A first-bootstrap failure has no fallback: repair remote trust rather than creating an unreviewed
|
|
runtime checkout.
|