mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
4e3fecbe8edfdf24c79a8bcb84f482aafef77c16
ThothII/docs/install/local-workspace-registry.md
T

12 KiB
Raw Blame History

Local workspace-registry installation (Mac and PC)

Complete the local PC/Mac/Linux installation first. This guide continues with the Git-backed workspace source of truth, installation-local connector bindings, and diagnostics. Use the Pi management manual for provider configuration and image recovery.

This guide runs a single-user ThothII registry on Docker Desktop (macOS or Windows) or a local Linux Docker Engine. It is intentionally loopback-only. Git is shared; the checkout, connector bindings, credentials, and session data are local. Never put credentials in workspace YAML, Git, browser drafts, diagnostics, or .env.example.

Prerequisites

  • macOS: Docker Desktop, Git, and sufficient volume disk space. Git Credential Manager is useful for HTTPS sign-in.
  • Windows: Docker Desktop with WSL2, Git for Windows, and the clone enabled in Docker file sharing. Use absolute paths/WSL paths; PowerShell uses ; rather than : in COMPOSE_FILE.
  • Linux PC: Docker Engine, Compose plugin, Git, and a user permitted to run Docker.
  • Outbound access to the Git remote. A local installation needs no inbound firewall rule.

Keep the operator .env and installation-secrets/ outside the workspace-registry Git checkout. On macOS/Linux use mode 0600 for individual secret files. On Windows apply an ACL that grants read access only to the Docker Desktop user. Do not use an empty file to silently bypass a selected authentication method.

Git remote: SSH and HTTPS

Create one private repository such as thoth-workspaces.git. It contains canonical workspace definitions and generated artifacts only:

thoth-workspaces.yaml
workspaces/<workspace-id>.yaml
workspaces/<workspace-id>.env.example
workspaces/<workspace-id>.md

For SSH, use a scoped deploy key, a verified known_hosts file, and strict host-key checking. For HTTPS, use Git Credential Manager or a secret-manager-created credentials file. Mount a private HTTPS CA as its own file. Do not disable host or certificate verification. The base Compose file does not mount a Git credential: add exactly one optional deploy/compose.git-ssh.yaml or deploy/compose.git-https.yaml override, so unused credential paths are never bind-mounted.

THT_WORKSPACE_GIT_REMOTE=ssh://git@git.example.invalid/platform/thoth-workspaces.git
THT_WORKSPACE_GIT_BRANCH=main
THT_WORKSPACE_INSTALLATION_ID=local-laptop
THT_SOURCE_ROOT=/absolute/path/to/ThothII
PI_AUTH_FILE=/absolute/path/installation-secrets/pi-auth.json
THT_SECRETS_FILE=/absolute/path/installation-secrets/thothii.secrets
THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/installation-secrets/git-ssh-key
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/installation-secrets/git-known-hosts
THT_WORKSPACE_GIT_CA_FILE=/absolute/path/installation-secrets/git-ca.pem

For HTTPS set THT_WORKSPACE_GIT_CREDENTIALS_FILE instead of the SSH key/known-hosts pair. Remote and branch are non-secret; every *_FILE is a local path whose content never enters Git or logs.

Shared Git values, local bindings, and secret files

Location Contains Never contains
Git workspace repository schema v2 YAML, generated binding names, LLM policy, model/index identity installation hostnames, keys, passwords, certificates, SSH keys
local .env remote, branch, installation ID, THT_WORKSPACE_BINDINGS_ENV_FILE, and secret source paths secret contents or THT_WS_* values
workspace bindings env file only THT_WS_* transport, endpoint, user, and /run/secrets/... path bindings secret contents or unrelated application settings
local secret directory Git credentials/key, known hosts, CA, connector secret files a copied registry checkout
Docker volumes registry checkout/snapshots/state/locks and local data host-only secret source files

The persistent volume is /data/workspace-registry:

repo/       # persistent Git checkout
snapshots/  # immutable validated revisions used by sessions
state/      # active revision and registry state
locks/      # short-lived publish locks

Installation variables are deterministic: north-star-research becomes NORTH_STAR_RESEARCH, and every name is THT_WS_<NAMESPACE>_<ROLE>_<SUFFIX>. Copy the bindings env example to an untracked operator file and set its absolute path as THT_WORKSPACE_BINDINGS_ENV_FILE. It is loaded only into core. Credentials and certificates use *_FILE path variables that must point inside /run/secrets.

Direct PostgreSQL, REST, and SSH tunnel bindings

Set only fields for the selected transport in the dedicated bindings env file. Canonical YAML keeps database/schema shared in Git. Every *_FILE=/run/secrets/<target> binding needs one matching host-only *_SOURCE path in operator .env. Generate the untracked connector override from those two files during bootstrap; do not copy or maintain a workspace-specific Compose override.

# Direct PostgreSQL
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password

# REST; an API-key file is needed only for a declared bearer/x-api-key diagnostic.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key

# SSH tunnel diagnostic only; runtime sessions are fail-closed in this release.
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432

REST diagnostics reject a private per-request CA rather than weakening TLS; use runtime-trusted HTTPS or verified direct/SSH native TLS. See the diagnostic protocol.

An SSH connector can prove installation reachability, host-key verification, authentication, and target identity, but it intentionally returns workspace_not_activatable; select direct or REST before creating sessions. Git pull/push over SSH remains fully supported and is independent.

Bootstrap, first pull, and diagnostics

Use the repository's canonical compose.yaml plus deploy/compose.local.yaml; they always start the mandatory frontend and core services. Do not copy or maintain a standalone application Compose file. Copy the bindings env example into an untracked operator directory and create a protected operator env file from deploy/env/local.env.example. It must contain absolute PI_AUTH_FILE, THT_SECRETS_FILE, THT_WORKSPACE_BINDINGS_ENV_FILE, and connector *_SOURCE paths. The Pi auth JSON, runtime secret bundle, and each connector credential remain separate protected host files and are mounted read-only; their contents never enter the operator env or rendered Compose.

Select exactly one repository Git transport override, deploy/compose.git-ssh.yaml or deploy/compose.git-https.yaml. A Compose env file is not a shell environment, so export only the non-secret paths required by the maintenance commands. Generate the connector override and render through the preflight wrapper, which rejects unsafe paths and combined SSH+HTTPS selection. Record the selected Git and generated connector overrides in the operator thothii-installation.yaml example, using absolute paths, so thothctl remains the ordinary lifecycle interface.

export THT_SOURCE_ROOT=/absolute/path/to/ThothII
export THT_OPERATOR_ENV=/absolute/path/to/operator/local.env
export THT_WORKSPACE_BINDINGS_ENV_FILE=/absolute/path/to/operator/workspace-bindings.env
export THT_CONNECTOR_OVERRIDE=/absolute/path/to/operator/connector-secrets.local.yaml
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
  -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \
  -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" config --quiet

"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
  -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \
  -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" up --build -d
curl --fail --silent http://127.0.0.1:8787/health
curl --fail --silent http://127.0.0.1:8787/workspace-registry/status
curl --fail --silent http://127.0.0.1:8787/workspaces

The first status request clones, validates all descriptors, and atomically activates a snapshot. Use POST /workspace-registry/pull to fetch later revisions. Run workspace diagnostics only after required bindings are mounted. The optional writer probe uses a distinct writer file and removes its uniquely named temporary record; ordinary diagnostics are read-only.

To migrate an existing legacy descriptor, create/clone an empty private remote, set the absolute THT_SOURCE_ROOT, transform with absolute paths, review the schema-v1 result, explicitly add vector database/schema and the complete schema-v2 contract, then commit/push. The transformer never imports ${ENV} values or secrets.

THT_SOURCE_ROOT=/absolute/path/to/ThothII
npm --prefix "$THT_SOURCE_ROOT/backend" run build
node "$THT_SOURCE_ROOT/backend/dist/workspaces/migrate-legacy.js" --input /absolute/path/legacy.yaml --output /absolute/path/thoth-workspaces

Publish, update, backup, outage recovery, and rollback

Drafts live only in browser storage. Review the canonical field diff, validate/test locally, then publish. If conflicted, pull first and create a new reviewed field-level draft; never hand-edit the running repo/ volume. Before upgrading, record registry status, stop Compose, and take a timestamped ownership-preserving backup of both registry and local data volumes while excluding installation-secrets/. Render Compose, rebuild, start, and check status before resuming work.

After a valid bootstrap, remote outage retains the last valid snapshot and reports degraded: true. Pinned sessions continue. Repair network/authentication, pull, and confirm non-degraded status. To undo a bad remote change, create a reviewed Git revert/release branch, advance the remote through normal policy, pull, and confirm its new snapshot. Do not delete snapshots/ as rollback.

Troubleshooting

Stable code Meaning and safe action
workspace_invalid Invalid descriptor/path/snapshot; restore a reviewed canonical revision.
binding_missing A selected value or readable *_FILE is absent; fix the local binding/mount.
workspace_not_activatable Diagnostics cannot activate the workspace; correct the selected transport.
workspace_stale Checkout changed or is busy; stop competing pull/publish work.
workspace_conflict Draft base differs from Git; pull, resolve the diff, validate, republish.
git_unavailable Remote, path, network, or lock unavailable; preserve the degraded valid snapshot.
git_auth_failed Mounted SSH/HTTPS material rejected/unreadable; rotate or fix permissions without logging it.
git_non_fast_forward Checkout diverged; reconcile through the registry workflow.
git_push_rejected Remote policy rejected the change; review branch protection/hooks.
connector_unavailable DNS/TLS/auth/resource identity diagnostic failed; inspect local bindings and egress.
semantic_index_incompatible Collection/model/dimensions/distance differs from Git; perform an explicit index migration.

On macOS, restart Docker Desktop if a named volume disappears. On Windows, check WSL2 and Docker file sharing. A failed first bootstrap has no snapshot fallback: repair remote trust and retry; never create an unreviewed local registry repository.

Reference in New Issue View Git Blame Copy Permalink