ThothII/README.md

ThothII

ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/tht core. The portable deployment runs exactly two application services; data services remain external in this profile, except for the mandatory internal semantic services bundled in Compose.

Docker Compose: local startup

Requirements: Docker Engine with Compose v2. The mandatory stack is frontend, core, qdrant, embedding, and the one-shot embedding-model-init. DWH and LLM remain external, configurable endpoints—even when they are co-located with ThothII.

From a fresh clone, run these commands from the repository root:

cp deploy/env/local.env.example deploy/env/local.env
# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
docker compose --env-file deploy/env/local.env \
  -f compose.yaml -f deploy/compose.local.yaml up --build -d

./scripts/run-stack.sh runs this same base+local command in the foreground. The core image contains its Pi runtime; no host pi executable is used. For a server installation:

cp deploy/env/server.env.example deploy/env/server.env
# Edit all absolute storage, Pi/secret/session files, and endpoint paths.
sudo scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001
docker compose --env-file deploy/env/server.env \
  -f compose.yaml -f deploy/compose.server.yaml \
  -f deploy/compose.session-server.yaml.example up --build -d

The initializer is required for an empty or restored server Pi-state bind. It atomically creates the three regular targets hidden below the writable parent bind; protected Pi auth and tracked model/settings sources remain separate read-only mounts. See the server manual before substituting a root other than /srv/thothii/pi-state.

Workspace descriptors come from the Git remote configured by THT_WORKSPACE_GIT_REMOTE; their runtime endpoint and secret bindings remain installation-local. Open http://127.0.0.1:8080 (set THOTH_HTTP_PORT in deploy/env/local.env to choose another loopback port).

Credentials and certificates are local protected files. Do not put them in environment examples, workspace YAML, URLs, or Compose interpolation values.

Application state is split across the named settings, pi-state, workspace-registry, and sessions volumes. docker compose down keeps them. Only an explicit destructive command such as docker compose down --volumes removes them.

The frontend depends on the core health check and proxies /health and /api/* to it. The application health endpoint intentionally checks process readiness only; external dependency diagnostics are exposed by tht doctor and do not prevent the UI from starting.

Git-backed workspace registry

Workspace descriptors are shared through a validated Git repository while endpoint bindings and secret files remain installation-local. Use the local Mac/PC installation manual for Docker Desktop or a local engine, and the server installation manual for the Gitea, reverse-proxy, backup, migration, and recovery workflow. The isolated deployment exercise is ./scripts/workspace-registry-smoke.sh; both manuals are checked with ./scripts/verify-workspace-install-docs.sh --profile local or --profile server.

The operator workflow is: update and review canonical YAML in the shared Git remote, Pull latest registry from each ThothII installation, run Validate workspace and Test on this installation, then select the workspace locally before creating sessions. Each new session pins the Git revision it used; a later pull or publish cannot change a Resume. Snapshot cleanup retains every revision referenced by an open, closed, or failed unarchived session. It reconciles from the single local installation list or from a server administrator's complete session list, never from a remote user's partial list.

Connector ssh_tunnel bindings are diagnostic-only in this release: their bounded probe always cleans up the loopback forward and returns workspace_not_activatable; session creation is rejected before persistence. Git registry access over SSH is unaffected. Use direct or REST connector transport for runtime sessions.

docker-compose.dev.yml is deliberately local: both published ports bind to 127.0.0.1, THT_SESSION_STORAGE=local, and THT_HOME=/data/local-home. Do not set THOTH_PUBLIC_EXPOSURE=true for that profile; the backend rejects that public/local combination at startup.

Run the end-to-end packaging check with:

./scripts/docker-smoke.sh

The smoke script validates Compose, builds and waits for both services, checks health through the frontend, verifies SSE response headers, restarts the core, and confirms /data survives. Each run uses a unique Compose project and removes that project's containers, network, and test volume afterward. It never targets the fixed thothii operator project or its volume. Set SMOKE_PROJECT to a different explicit project name for reproducible debugging, and set KEEP_SMOKE_RESOURCES=1 to retain that smoke project's resources for inspection; remove them later with docker compose --project-name "$SMOKE_PROJECT" down --volumes.

Unified deployment release gates

Task 13 adds no-secret release gates around the canonical local and server Compose profiles. Its deterministic safety check does not contact the Docker daemon:

bash scripts/unified-deployment-smoke.sh --self-test

The three Linux Docker smokes are separate release commands. Each creates a unique Compose project, temporary Git workspace remote, fixture provider, image names, and run label. Its exit trap removes only resources carrying that exact run identity and never performs a global Docker prune. Cleanup enumerates running and stopped project containers immediately before compose down and refuses the teardown if any container, volume, or network has a foreign run label.

bash scripts/unified-deployment-smoke.sh
bash scripts/thothctl-update-smoke.sh
bash scripts/server-deployment-smoke.sh

The unified smoke builds and starts frontend and core, verifies the embedded Pi and internal registry, recreates with the Git remote offline, activates a valid Git update, rejects invalid Git content while retaining the valid snapshot, and checks the four persistence volumes. The unified and update-only smokes inject a digest-pinned non-core candidate under a deliberately mismatched Pi version and require thothctl pi update to roll back while preserving settings, sessions, Pi state, registry revision, and mount identity. The rollback candidate is the digest-pinned hello-world executable: a preflight proves that it exits successfully, so the failed replacement core satisfies thothctl's stopped-core compensation precondition. The server smoke uses the same smoke-built core/frontend images with the server and required session overlays, disposable bind roots and secret files, upstream-auth checks, and a fail-closed 503 assertion for its deliberately unavailable disposable session endpoint. No real provider, database credential, or repository secret is required.

For a clean server bind, scripts/prepare-server-pi-state.sh creates the hidden regular agent/auth.json, agent/models.json, and agent/settings.json mount targets atomically before Compose. The server smoke starts from an empty Pi-state root and applies this same preflight; the real protected/tracked sources remain separate read-only mounts. Deterministic fixture tests render both profiles, verify that bindings stay on core, check mount readability, and run the production workspace resolver. Wrong-service, wrong-value, and broken-secret-mount mutations must fail.

Each public smoke has its own 30-minute process-group supervisor with TERM/KILL cleanup; CI retains an independent 32-minute outer timeout and does not retry a failed command.

Current release status (2026-08-05): clean-root render/setup and the production runtime-binding resolver contracts are green. The server fixture supplies all four private trusted claims, including exact non-admin value 0, and a focused test proves nginx normalization produces the accepted non-admin backend principal. Canonical schema-v2 registry descriptors now pass through one backend-owned, secret-safe runtime handoff for inventory and session execution; canonical identity and durable session/artifact/index roots are retained. The fresh update-only smoke passed bad-candidate mutation, automatic rolled_back compensation, exact prior-image restoration, unchanged registry/mount identity, all four sentinels, post-rollback doctor/workspace checks, and exact cleanup. The one authorized server-smoke invocation was denied access to the Docker socket by its execution sandbox before startup, so the complete authenticated workspace and fail-closed session assertions still require a fresh authorized release run. Native Windows Docker Desktop/WSL2 remains a separate manual/self-hosted gate.

The deterministic native Windows contract is:

.\scripts\test-windows-clone-contract.ps1

It checks Git's CRLF/LF attributes and bytes, copies tracked source into a temporary path containing spaces, builds and invokes native Windows thothctl there, and renders exactly core plus frontend without starting containers. On a supported self-hosted Windows Docker Desktop/WSL2 runner, dispatch the deployment workflow with windows_docker_startup=true; that job executes:

.\scripts\test-windows-clone-contract.ps1 -DockerStartup

Startup mode adds bounded image build/two-service health startup, installation-aware thothctl status, stopped-container-aware ownership checks, and exact cleanup. The ordinary hosted Windows job remains deterministic and does not claim Docker startup.

Preprocessing jobs and S3 Evidence

The included preprocessing services reuse the internal Qdrant/Ollama stack. Mount Evidence at /data/source/evidence, then run the explicit preprocessing preset:

docker compose --env-file deploy/env/local.env \
  -f compose.yaml -f deploy/compose.local.yaml \
  -f deploy/compose.preprocess.yaml --profile preprocess run --rm preprocess-evidence

Replace the final service with preprocess-dwh when required. The overlay makes each job wait for the vector service health checks and embedding model initialization; no separate semantic-service startup is required.

S3 Evidence uses the optional tht[s3] dependency and canonical s3://bucket/key provenance. AWS endpoints are used when no custom URL is supplied. Every custom endpoint is an explicit egress trust-boundary opt-in and uses path-style addressing; private and HTTP endpoints require additional independent opt-ins. Literal non-global IPv4/IPv6 addresses are classified locally; hostnames are not DNS-pinned, so trusted custom-endpoint deployments must enforce their destination with network egress policy. Store access key, secret key, and session token as secret references in deployment configuration—never in Compose environment values or source URIs. Discovery and reads are bounded by configured page, object, and byte limits.

Create a versioned PostgreSQL custom-format backup (the filename is operator-controlled, so use an immutable timestamp or release identifier):

./scripts/vector-backup.sh \
  --host 127.0.0.1 --port 5432 --database thoth --user thoth_backup \
  --password-file /secure/thoth/vector-backup-password \
  --output /secure/backups/thoth-vectors-2026-07-12.dump

The dump contains the three allowlisted vectors tables, their data and ACLs, plus the public.tht_vector_migrations ledger. Login roles and passwords are deliberately not copied: provision/reconcile the approved role names on the target first, and install the vector extension in its vectors schema. The target must otherwise contain no vector tables or ledger.

Restore always names both the currently active source and a target on a physically distinct PostgreSQL cluster. The script compares PostgreSQL system identity, so host aliases or a different database in the active cluster cannot bypass the guard. It refuses a non-empty target unless --force-nonempty is explicit, and the clean restore is one transaction:

./scripts/vector-restore.sh \
  --active-host vector-db --active-database thoth --active-user thoth_backup \
  --active-password-file /secure/thoth/vector-active-password \
  --target-host vector-db-restore --target-database thoth --target-user thoth_restore \
  --target-password-file /secure/thoth/vector-restore-password \
  --input /secure/backups/thoth-vectors-2026-07-12.dump

After restore, run tht vector migrate --status --json, adapter health, and a known retrieval query against the target before changing any migration/export endpoint.

Production trust boundary and secrets

ThothII does not implement OIDC. Do not expose its application port directly to a network. The production pattern is an authenticated host reverse proxy that:

• terminates TLS and authenticates every request;
• removes any client-supplied identity header;
• injects one trusted X-Authenticated-User value;
• proxies to the loopback-only ThothII frontend.

deploy/nginx-authenticated-proxy.conf.example shows the contract using nginx auth_request; replace the placeholder authentication gateway with the organization's reviewed identity proxy. AUTH_MODE=upstream trusts this boundary and rejects requests without the identity header. Setting THOTH_PUBLIC_EXPOSURE=true with any other auth mode fails during core startup.

Production credentials use the existing Compose secret-bundle contract, never environment values. Copy deploy/secrets/thothii.secrets.example to a protected host file, include only the required keys, and set its absolute path as THT_SECRETS_FILE in the operator env. Keep Pi's native provider auth in the separate protected file named by PI_AUTH_FILE.

The bundle is mounted read-only as /run/secrets/thothii.secrets and must be mode 0600 or 0400 on the host. Docker's runtime 0444 mode is accepted only beneath /run/secrets; see deploy/secrets/README.md. A PEM CA chain is deliberately not a bundle value: PEM contains whitespace and is rejected by the strict parser. Keep the CA chain in the host/secret-manager materialization and add a reviewed Compose override that mounts it at /run/secrets/ca-chain.pem and sets THT_SSL_CA when a private CA is required. The base bundle does not create that mount. The frontend remains on loopback; the authenticated host proxy is the only public listener.

Set the selected model provider in application settings (or PI_PROVIDER). For each Pi spawn the backend validates and reads THT_MODEL_API_KEY from the bundle, then exposes its value only as the provider's recognized child variable (for example ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, or ZAI_API_KEY). Neither the generic file path nor deprecated PI_PROVIDER_API_KEY is inherited by Pi. Local providers such as Ollama require no model key.

THT_MODEL_API_KEY supports Pi providers whose authentication is exactly one key: ant-ling, anthropic, cerebras, deepseek, fireworks, github-copilot, google (including the gemini alias), google-vertex when using its API-key mode, groq, huggingface, kimi-coding, minimax, minimax-cn, mistral, moonshotai, moonshotai-cn, nvidia, openai, opencode, opencode-go, openrouter, together, vercel-ai-gateway, xai, the four xiaomi* providers, zai, and zai-coding-cn. Compound providers are deliberately unsupported: amazon-bedrock, azure-openai-responses, cloudflare-workers-ai, and cloudflare-ai-gateway require multiple credential/configuration values. Selecting one fails before Pi starts; ambient AWS, Azure, and Cloudflare credentials are still scrubbed. Supporting them requires a future dedicated provider-specific configuration.

User-owned session server cutover

The server profile stores sessions and per-user preferences directly in PostgreSQL schema thoth_sessions; it does not use PostgREST, browser storage, a shared session directory, or a dual write. Use deploy/compose.session-server.yaml.example with the canonical base+server files and set THT_SERVER_WORKSPACE_CONFIG to an absolute, protected copy of deploy/workspaces/server-sessions.yaml.example.

The runtime login needs membership in the no-login database role thoth_sessions_runtime only. The distinct, one-shot migrator login needs migration authority and uses thoth_sessions_migrator; it must never be mounted into core. Set the non-secret endpoint and role fields in the protected deployment environment:

AUTH_MODE=upstream
THOTH_PUBLIC_EXPOSURE=true
THT_SESSION_STORAGE=postgres
THT_SESSION_DB_HOST=sessions-db.internal
THT_SESSION_DB_PORT=5432
THT_SESSION_DB_NAME=thoth
THT_SESSION_RUNTIME_USER=thoth_sessions_app
THT_SESSION_MIGRATOR_USER=thoth_sessions_migrate
THT_SESSION_DB_SSLMODE=verify-full
THT_SESSION_RUNTIME_PASSWORD_SOURCE=/secure/thoth/session-runtime-password
THT_SESSION_MIGRATOR_PASSWORD_SOURCE=/secure/thoth/session-migrator-password
THT_SESSION_CA_SOURCE=/secure/thoth/session-ca.pem

The overlay mounts the runtime password at /run/secrets/session_runtime_password, the CA at /run/secrets/session_ca.pem, and passes those paths—not their contents—to the server workspace. It mounts session_migrator_password only to session-migrate. The backend refuses a server session store without upstream authentication, direct DB host/name/runtime user/password-file, verify-ca or verify-full, and an absolute CA path. The migrator independently rejects every other TLS mode before reading its password secret or constructing a database URL.

Perform the cutover in one maintenance window, with the upstream identity-proxy headers and backend principal parser deployed together. Neither change is safe to deploy independently: the proxy clears the legacy identity header and the backend rejects it. Drain/stop active Pi work, enable a maintenance response at the proxy, then run the migrator once and inspect its pristine JSON:

docker compose --env-file deploy/env/server.env \
  -f compose.yaml -f deploy/compose.server.yaml -f deploy/compose.session-server.yaml.example \
  --profile session-migrate run --rm session-migrate

It must report no pending or drifted migrations before starting the replacement core. /health is a liveness probe and remains 200; any request that needs unavailable repository storage returns a fixed 503 before a Pi process starts. Verify this with an authenticated request after the replacement core is healthy, then remove maintenance mode.

Do not import the three legacy server filesystem sessions: they have no trusted owner binding. During the same maintenance window, archive the exact three reviewed IDs, verify the generated archive and .sha256, then rerun the command with --delete to remove only those three source directories:

./docker/cutover-legacy-sessions.sh \
  /secure/thoth/legacy-sessions /secure/backups/thoth-legacy-sessions-2026-07-16.tar \
  SESSION_ID_1 SESSION_ID_2 SESSION_ID_3
# After independent archive review, use a new backup filename:
./docker/cutover-legacy-sessions.sh --delete \
  /secure/thoth/legacy-sessions /secure/backups/thoth-legacy-sessions-2026-07-16-delete.tar \
  SESSION_ID_1 SESSION_ID_2 SESSION_ID_3

The helper refuses to overwrite an existing backup and refuses any count other than three distinct IDs. Never run it against a live path without the maintenance gate. Roll back application code only by keeping PostgreSQL as the single source of truth and deploying a compatible fixed release. Do not restore filesystem persistence, do not re-import the archive, and never dual-write sessions to database and files.

Reproducible image verification

Base images use exact tags and immutable multi-platform manifest digests. Dependency update and residual OS-repository limitations are documented in docker/LOCKS.md. Run the shared architecture gate with PLATFORM=linux/amd64 or PLATFORM=linux/arm64:

PLATFORM=linux/arm64 ./scripts/verify-container-images.sh

It builds both images, runs common version/runtime/security smokes, and emits an image/package inventory beneath .artifacts/container-images/. CI runs the same script for both architectures.