mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
09eeea17e5d6355c37a9588cd868a2ee22bbe486
ThothII/docs/install/local-workspace-registry.md
T

12 KiB
Raw Blame History

Local workspace-registry installation (Mac and PC)

This guide runs a single-user ThothII registry on Docker Desktop (macOS or Windows) or a local Linux Docker Engine. It is intentionally loopback-only. Git is shared; the checkout, connector bindings, credentials, and session data are local. Never put credentials in workspace YAML, Git, browser drafts, diagnostics, or .env.example.

Prerequisites

  • macOS: Docker Desktop, Git, and sufficient volume disk space. Git Credential Manager is useful for HTTPS sign-in.
  • Windows: Docker Desktop with WSL2, Git for Windows, and the clone enabled in Docker file sharing. Use absolute paths/WSL paths; PowerShell uses ; rather than : in COMPOSE_FILE.
  • Linux PC: Docker Engine, Compose plugin, Git, and a user permitted to run Docker.
  • Outbound access to the Git remote. A local installation needs no inbound firewall rule.

Keep the operator .env and installation-secrets/ outside the workspace-registry Git checkout. On macOS/Linux use mode 0600 for individual secret files. On Windows apply an ACL that grants read access only to the Docker Desktop user. Do not use an empty file to silently bypass a selected authentication method.

Git remote: SSH and HTTPS

Create one private repository such as thoth-workspaces.git. It contains canonical workspace definitions and generated artifacts only:

thoth-workspaces.yaml
workspaces/<workspace-id>.yaml
workspaces/<workspace-id>.env.example
workspaces/<workspace-id>.md

For SSH, use a scoped deploy key, a verified known_hosts file, and strict host-key checking. For HTTPS, use Git Credential Manager or a secret-manager-created credentials file. Mount a private HTTPS CA as its own file. Do not disable host or certificate verification. The base Compose file does not mount a Git credential: add exactly one optional git-ssh.workspace-registry.yaml or git-https.workspace-registry.yaml override, so unused credential paths are never bind-mounted.

THT_WORKSPACE_GIT_REMOTE=ssh://git@git.example.invalid/platform/thoth-workspaces.git
THT_WORKSPACE_GIT_BRANCH=main
THT_WORKSPACE_INSTALLATION_ID=local-laptop
THT_SOURCE_ROOT=/absolute/path/to/ThothII
THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/installation-secrets/git-ssh-key
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/installation-secrets/git-known-hosts
THT_WORKSPACE_GIT_CA_FILE=/absolute/path/installation-secrets/git-ca.pem

For HTTPS set THT_WORKSPACE_GIT_CREDENTIALS_FILE instead of the SSH key/known-hosts pair. Remote and branch are non-secret; every *_FILE is a local path whose content never enters Git or logs.

Shared Git values, local bindings, and secret files

Location Contains Never contains
Git workspace repository schema v2 YAML, generated binding names, LLM policy, model/index identity installation hostnames, keys, passwords, certificates, SSH keys
local .env remote, branch, installation ID, THT_WORKSPACE_BINDINGS_ENV_FILE, and secret source paths secret contents or THT_WS_* values
workspace bindings env file only THT_WS_* transport, endpoint, user, and /run/secrets/... path bindings secret contents or unrelated application settings
local secret directory Git credentials/key, known hosts, CA, connector secret files a copied registry checkout
Docker volumes registry checkout/snapshots/state/locks and local data host-only secret source files

The persistent volume is /data/workspace-registry:

repo/       # persistent Git checkout
snapshots/  # immutable validated revisions used by sessions
state/      # active revision and registry state
locks/      # short-lived publish locks

Installation variables are deterministic: psd-clinical becomes PSD_CLINICAL, and every name is THT_WS_<NAMESPACE>_<ROLE>_<SUFFIX>. Copy the bindings env example to an untracked operator file and set its absolute path as THT_WORKSPACE_BINDINGS_ENV_FILE. It is loaded only into core. Credentials and certificates use *_FILE path variables that must point inside /run/secrets. If declared, THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE is distinct from the vector reader file; a reader credential is never repurposed for writing.

Direct PostgreSQL, REST, and SSH tunnel bindings

Set only fields for the selected transport in the dedicated bindings env file. Canonical YAML keeps database/schema/collection, distance, embedding model, and dimensions shared in Git. Every *_FILE=/run/secrets/<target> binding needs one matching host-only *_SOURCE path in operator .env. Generate the untracked connector override from those two files during bootstrap; do not copy or maintain a workspace-specific Compose override.

# Direct PostgreSQL and pgvector
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=postgres_direct
THT_WS_PSD_CLINICAL_DWH_HOST=dwh.example.invalid
THT_WS_PSD_CLINICAL_DWH_PORT=5432
THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader
THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=pgvector_direct
THT_WS_PSD_CLINICAL_VECTOR_HOST=vector.example.invalid
THT_WS_PSD_CLINICAL_VECTOR_PORT=5432
THT_WS_PSD_CLINICAL_VECTOR_USER=thoth_vector_reader
THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE=/run/secrets/psd-vector-reader
THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL=https://embeddings.example.invalid

# REST; an API-key file is needed only for a declared bearer/x-api-key diagnostic.
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=rest_api
THT_WS_PSD_CLINICAL_DWH_BASE_URL=https://dwh.example.invalid
THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE=/run/secrets/psd-dwh-api-key
THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=rest_api
THT_WS_PSD_CLINICAL_VECTOR_BASE_URL=https://vectors.example.invalid
THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE=/run/secrets/psd-vector-api-key

# SSH tunnel diagnostic only; runtime sessions are fail-closed in this release.
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=ssh_tunnel
THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader
THT_WS_PSD_CLINICAL_DWH_SSH_HOST=bastion.example.invalid
THT_WS_PSD_CLINICAL_DWH_SSH_PORT=22
THT_WS_PSD_CLINICAL_DWH_SSH_USER=thoth_tunnel
THT_WS_PSD_CLINICAL_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/psd-dwh-tunnel-key
THT_WS_PSD_CLINICAL_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/psd-dwh-known-hosts
THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_HOST=dwh.internal.example
THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_PORT=5432

Repeat the SSH names for VECTOR where needed. REST diagnostics reject a private per-request CA rather than weakening TLS; use runtime-trusted HTTPS or verified direct/SSH native TLS. See the diagnostic protocol.

An SSH connector can prove installation reachability, host-key verification, authentication, and target identity, but it intentionally returns workspace_not_activatable; select direct or REST before creating sessions. Git pull/push over SSH remains fully supported and is independent.

Bootstrap, first pull, and diagnostics

Copy the local Compose example, exactly one selected SSH Git override or HTTPS Git override, and the bindings env example into an untracked operator directory. Keep THT_SOURCE_ROOT and the absolute THT_WORKSPACE_BINDINGS_ENV_FILE in its .env for Compose interpolation; this keeps the copied Compose file buildable and confines THT_WS_* values to core. A Compose .env file is not a shell environment, so do not import it into the maintenance shell. Instead, explicitly export the two non-secret paths before running the commands. Create the host secret files named by the selected Git transport and every declared connector *_SOURCE, then generate the connector override and render through the preflight wrapper. The wrapper is required: it rejects unsafe source paths and a combined SSH+HTTPS Git selection before Compose runs.

export THT_SOURCE_ROOT=/absolute/path/to/ThothII
export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env"
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env .env --output connector-secrets.local.yaml
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \
  -f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml config --quiet

From the operator directory:

"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \
  -f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml up --build -d
curl --fail --silent http://127.0.0.1:8787/health
curl --fail --silent http://127.0.0.1:8787/workspace-registry/status
curl --fail --silent http://127.0.0.1:8787/workspaces

The first status request clones, validates all descriptors, and atomically activates a snapshot. Use POST /workspace-registry/pull to fetch later revisions. Run workspace diagnostics only after required bindings are mounted. The optional writer probe uses a distinct writer file and removes its uniquely named temporary record; ordinary diagnostics are read-only.

To migrate an existing PSD descriptor, create/clone an empty private remote, set the absolute THT_SOURCE_ROOT, transform with absolute paths, review the schema-v1 result, explicitly add vector database/schema and the complete schema-v2 contract, then commit/push. The transformer never imports ${ENV} values or secrets.

THT_SOURCE_ROOT=/absolute/path/to/ThothII
npm --prefix "$THT_SOURCE_ROOT/backend" run build
node "$THT_SOURCE_ROOT/backend/dist/workspaces/migrate-legacy.js" --input /absolute/path/psd.yaml --output /absolute/path/thoth-workspaces

Publish, update, backup, outage recovery, and rollback

Drafts live only in browser storage. Review the canonical field diff, validate/test locally, then publish. If conflicted, pull first and create a new reviewed field-level draft; never hand-edit the running repo/ volume. Before upgrading, record registry status, stop Compose, and take a timestamped ownership-preserving backup of both registry and local data volumes while excluding installation-secrets/. Render Compose, rebuild, start, and check status before resuming work.

After a valid bootstrap, remote outage retains the last valid snapshot and reports degraded: true. Pinned sessions continue. Repair network/authentication, pull, and confirm non-degraded status. To undo a bad remote change, create a reviewed Git revert/release branch, advance the remote through normal policy, pull, and confirm its new snapshot. Do not delete snapshots/ as rollback.

Troubleshooting

Stable code Meaning and safe action
workspace_invalid Invalid descriptor/path/snapshot; restore a reviewed canonical revision.
binding_missing A selected value or readable *_FILE is absent; fix the local binding/mount.
workspace_not_activatable Diagnostics cannot activate the workspace; correct the selected transport.
workspace_stale Checkout changed or is busy; stop competing pull/publish work.
workspace_conflict Draft base differs from Git; pull, resolve the diff, validate, republish.
git_unavailable Remote, path, network, or lock unavailable; preserve the degraded valid snapshot.
git_auth_failed Mounted SSH/HTTPS material rejected/unreadable; rotate or fix permissions without logging it.
git_non_fast_forward Checkout diverged; reconcile through the registry workflow.
git_push_rejected Remote policy rejected the change; review branch protection/hooks.
connector_unavailable DNS/TLS/auth/resource identity diagnostic failed; inspect local bindings and egress.
semantic_index_incompatible Collection/model/dimensions/distance differs from Git; perform an explicit index migration.

On macOS, restart Docker Desktop if a named volume disappears. On Windows, check WSL2 and Docker file sharing. A failed first bootstrap has no snapshot fallback: repair remote trust and retry; never create an unreviewed local registry repository.

Reference in New Issue View Git Blame Copy Permalink