# ThothII ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/`tht` core. The portable deployment runs two application services plus the installation-local metadata catalog; DWH and LLM services remain external. Semantic services are bundled in Compose. The same frontend supports **full** (its own header) and **embedded** (inside a portal). This choice is independent of authentication: the Mac uses full/local, Omics uses embedded/upstream with its existing login, and a standalone server can use full/OIDC. See [rendering architecture](docs/architecture/application-shell.md) and [configuration, Omics delivery and deploy](docs/operations/shell-and-localization.md). For the current server upgrade with Omics Portal, follow the ordered [Codex server handoff](docs/operations/server-codex-handoff.md), including source integration, embedded/upstream configuration, coordinated rollout and rollback. Local/OIDC authentication is configured through the host CLI `tht`; portal authentication is established by the trusted server proxy. See the [local guide](docs/install/authentication-local.md), [OIDC guide](docs/install/authentication-oidc.md), [upstream integration](docs/install/authentication-upstream.md), and [manual acceptance matrix](docs/testing/authentication-manual-acceptance.md). For the clone-based manual standalone installation test on macOS, Windows, and Linux, use the [Italian procedure](docs/install/standalone-manual-it.md) or the [English procedure](docs/install/standalone-manual-en.md). The [public manual](https://git.tylconsulting.it/thothii-docs/) covers the product, installation, use and administration. Developer architecture, contracts, ADRs, tests, plans and release records remain in this repository but are excluded from MkDocs pages and search. This is an editorial boundary, not an access restriction on the public repository. See the [documentation cleanup review](docs/maintenance/2026-09-15-documentation-cleanup.md) for the executed consolidation and the inventory of historical sources retained in Git. ## Docker Compose and installation For a fresh installation, follow the complete manual procedure in [Italian](docs/install/standalone-manual-it.md) or [English](docs/install/standalone-manual-en.md). Configure protected files first; then run the documented build, explicit migrations and startup commands with the same installation descriptor and Compose project. There is no installer or launcher. The mandatory stack includes frontend, core, PostgreSQL catalog, Qdrant, Ollama and the embedding initializer. DWH and LLM endpoints remain external dependencies. Pi is included in the core image. Credentials and certificates belong in protected installation-local files, never in the workspace repository. For developer topology, overlays and lifecycle details, see the internal [Compose reference](docs/operations/compose-reference.md). Ordinary stop/down keeps persistent data; removing volumes is destructive and is not an upgrade step. Process health is distinct from external dependency checks performed by doctor. ## Git-backed workspace repository Workspace descriptors are shared through a validated Git repository while endpoint bindings and secret files remain installation-local. The supported operating sequence is documented in [Workspace operations](docs/operations/workspaces.md); it covers curator publication, installation activation, runtime bindings, and preprocessing. The host setup and lifecycle path is in [Install and first start](docs/install/first-start.md). The curator-owned repository layout is: ```text thoth-workspaces.yaml /workspace.yaml /evidence/** ``` `thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered `workspaces` list of `{id, name, description?}` entries. It is authoritative for workspace ID, name, description, and display order. Every catalog entry must have a matching descriptor in the same commit; otherwise the complete candidate is rejected. Descriptors remain curator-owned and change only through a Git commit and push from a separate authoring clone, followed by an installation pull. ThothII never writes any workspace repository content. The operator workflow is: curate catalog/descriptor/Evidence changes in Git, commit and push, **Update workspace repository** from each ThothII installation, select the workspace, complete its write-only runtime-secret fields, run **Validate workspace source** and **Test workspace connections**, then select the workspace locally before creating sessions. Each new session pins the Git revision it used; a later pull cannot change a Resume. Snapshot cleanup retains every revision referenced by an open, closed, or failed unarchived session. It reconciles from the single local installation list or from a server administrator's complete session list, never from a remote user's partial list. The isolated deployment exercise is `./scripts/workspace-registry-smoke.sh`; both manuals are checked with `./scripts/verify-workspace-install-docs.sh --profile local` or `--profile server`. Schema v4 is the only accepted workspace descriptor. It contains workspace identity and optional Evidence configuration only; PostgreSQL Metadata Catalog owns every database fact and binding. Schema v1, v2, and v3 descriptors are rejected before activation. Candidate snapshot validation therefore makes activation or a pull fail atomically while the prior valid snapshot remains active. Each workspace owns separate Qdrant `reference` and `memory` collections: Schema, relationships, and Evidence are replaceable reference data; Memory and solved questions have a persistent lifecycle. Create a clean v4 descriptor containing only `workspace` and optional `evidence`. Do not copy the legacy database, diagnostics, `llm_policy`, or `semantic_index` blocks; configure the database in Database Management. For NL→SQL runtime sessions, connector `ssh_tunnel` bindings remain diagnostic-only: their bounded probe cleans up the loopback forward and returns `workspace_not_activatable`; session creation is rejected before persistence. Database management is a separate boundary and supports a strict OpenSSH tunnel for **Test connection** and **Sync tables**, using a private key, optional passphrase, mandatory `known_hosts`, and optional PostgreSQL TLS CA/server name. Git registry access over SSH is unaffected. Use direct or REST connector transport for runtime sessions. `docker-compose.dev.yml` is deliberately local: both published ports bind to `127.0.0.1`, `THT_SESSION_STORAGE=local`, and `THT_HOME=/data/local-home`. Do not set `THOTH_PUBLIC_EXPOSURE=true` for that profile; the backend rejects that public/local combination at startup. Run the end-to-end packaging check with: ```sh ./scripts/docker-smoke.sh ``` The smoke script validates Compose, builds and waits for both services, checks health through the frontend, verifies SSE response headers, restarts the core, and confirms `/data` survives. Each run uses a unique Compose project and removes that project's containers, network, and test volume afterward. It never targets the fixed `thothii` operator project or its volume. Set `SMOKE_PROJECT` to a different explicit project name for reproducible debugging, and set `KEEP_SMOKE_RESOURCES=1` to retain that smoke project's resources for inspection; remove them later with `docker compose --project-name "$SMOKE_PROJECT" down --volumes`. ## Unified deployment release gates Task 13 adds no-secret release gates around the canonical local and server Compose profiles. Its deterministic safety check does not contact the Docker daemon: ```sh bash scripts/unified-deployment-smoke.sh --self-test ``` The three Linux Docker smokes are separate release commands. Each creates a unique Compose project, temporary Git workspace remote, fixture provider, image names, and run label. Its exit trap removes only resources carrying that exact run identity and never performs a global Docker prune. Cleanup enumerates running and stopped project containers immediately before `compose down` and refuses the teardown if any container, volume, or network has a foreign run label. ```sh bash scripts/unified-deployment-smoke.sh bash scripts/tht-update-smoke.sh bash scripts/server-deployment-smoke.sh ``` The unified smoke builds and starts `frontend` and `core`, verifies the embedded Pi and internal registry, recreates with the Git remote offline, activates a valid Git update, rejects invalid Git content while retaining the valid snapshot, and checks the four persistence volumes. The unified and update-only smokes inject a digest-pinned non-core candidate under a deliberately mismatched Pi version and require `tht pi update` to roll back while preserving settings, sessions, Pi state, registry revision, and mount identity. The rollback candidate is the digest-pinned `hello-world` executable: a preflight proves that it exits successfully, so the failed replacement core satisfies `tht`'s stopped-core compensation precondition. The server smoke uses the same smoke-built core/frontend images with the server and required session overlays, disposable bind roots and secret files, upstream-auth checks, and a fail-closed `503` assertion for its deliberately unavailable disposable session endpoint. No real provider, database credential, or repository secret is required. For a clean server bind, `scripts/prepare-server-pi-state.sh` creates the hidden regular Pi agent mount targets atomically before Compose. The auth target receives the protected credential bind; the model and settings targets receive generated read-only projections. The server smoke starts from an empty Pi-state root and applies this same preflight. Deterministic fixture tests render both profiles, verify that bindings stay on `core`, check mount readability, and run the production workspace resolver. Wrong-service, wrong-value, and broken-secret-mount mutations must fail. Each public smoke has its own 30-minute process-group supervisor with TERM/KILL cleanup; CI retains an independent 32-minute outer timeout and does not retry a failed command. Current release status (2026-08-05): clean-root render/setup and the production runtime-binding resolver contracts are green. The server fixture supplies all four private trusted claims, including exact non-admin value `0`, and a focused test proves nginx normalization produces the accepted non-admin backend principal. Canonical schema-v4 registry descriptors now pass through one backend-owned, secret-safe runtime handoff for inventory and session execution; canonical identity and durable session/artifact/index roots are retained. The fresh update-only smoke passed bad-candidate mutation, automatic `rolled_back` compensation, exact prior-image restoration, unchanged registry/mount identity, all four sentinels, post-rollback doctor/workspace checks, and exact cleanup. The one authorized server-smoke invocation was denied access to the Docker socket by its execution sandbox before startup, so the complete authenticated workspace and fail-closed session assertions still require a fresh authorized release run. Native Windows Docker Desktop/WSL2 remains a separate manual/self-hosted gate. The deterministic native Windows contract is: ```powershell .\scripts\test-windows-clone-contract.ps1 ``` It checks Git's CRLF/LF attributes and bytes, copies tracked source into a temporary path containing spaces, builds and invokes native Windows `tht` there, and renders exactly `core` plus `frontend` without starting containers. On a supported self-hosted Windows Docker Desktop/WSL2 runner, dispatch the deployment workflow with `windows_docker_startup=true`; that job executes: ```powershell .\scripts\test-windows-clone-contract.ps1 -DockerStartup ``` Startup mode adds bounded image build/two-service health startup, installation-aware `tht` status, stopped-container-aware ownership checks, and exact cleanup. The ordinary hosted Windows job remains deterministic and does not claim Docker startup. ## Workspace preprocessing and S3 Evidence For an interactive run, select the workspace, expand **Administration** in the right sidebar, and use its **Preprocessing** control. The control explains any unmet prerequisite and exposes only the latest safe failure diagnostic. For unattended operation, use the native host CLI and installation descriptor: ```sh tht --installation /absolute/path/thothii-installation.yaml \ workspace preprocess run --workspace tht --installation /absolute/path/thothii-installation.yaml \ workspace preprocess clear --workspace ``` The one-shot command starts the profile-gated `workspace-maintenance` service, reads database metadata from PostgreSQL, and rebuilds LSH plus schema/Evidence vectors. The clear command removes those derived artifacts while preserving the separate Memory collection. The core remains unavailable until preprocessing completes. See [Evidence](docs/evidence.md) and the [workspace preprocessing CLI contract](docs/contracts/workspace-preprocessing-cli.md). S3 Evidence uses the optional `tht[s3]` dependency and canonical `s3://bucket/key` provenance. AWS endpoints are used when no custom URL is supplied. Every custom endpoint is an explicit egress trust-boundary opt-in and uses path-style addressing; private and HTTP endpoints require additional independent opt-ins. Literal non-global IPv4/IPv6 addresses are classified locally; hostnames are not DNS-pinned, so trusted custom-endpoint deployments must enforce their destination with network egress policy. Store access key, secret key, and session token as secret references in deployment configuration—never in Compose environment values or source URIs. Discovery and reads are bounded by configured page, object, and byte limits. Create a versioned Qdrant volume backup for one exact Compose project (the filename is operator-controlled, so use an immutable timestamp or release identifier): ```sh ./scripts/vector-backup.sh \ --project-name thothii \ --output /secure/backups/thoth-qdrant-2026-08-08.tar ``` The script resolves exactly one Docker volume with the labels `com.docker.compose.project=` and `com.docker.compose.volume=qdrant-data`, stops the `qdrant` service if it is running, archives that volume's persistent contents, then restores the prior service state. It never performs global Docker cleanup and refuses to overwrite an existing archive path. Restore targets that same exact project-scoped `qdrant-data` volume. Because restore replaces the persistent Qdrant data in place, it requires an explicit confirmation that exactly repeats the Compose project name by passing `--confirm-project`: ```sh ./scripts/vector-restore.sh \ --project-name thothii \ --input /secure/backups/thoth-qdrant-2026-08-08.tar \ --confirm-project thothii ``` The restore script stops `qdrant`, validates the exact labeled target, stages the current volume contents for rollback, extracts the requested archive into the volume, and then returns the service to its prior running state. It restores semantic storage only. Before reopening write traffic, the workspace registry must already be at a reviewed v4 descriptor revision compatible with the restored collection; then run backend health checks and a known retrieval query. The helper does not restore descriptors, rename collections, or reconcile an incompatible collection contract. ## Production trust boundary and secrets ThothII does not implement OIDC. Do not expose its application port directly to a network. The production pattern is an authenticated host reverse proxy that: - terminates TLS and authenticates every request; - removes any client-supplied identity header; - injects one trusted `X-Authenticated-User` value; - proxies to the loopback-only ThothII frontend. [`deploy/nginx-authenticated-proxy.conf.example`](deploy/nginx-authenticated-proxy.conf.example) shows the contract using nginx `auth_request`; replace the placeholder authentication gateway with the organization's reviewed identity proxy. `AUTH_MODE=upstream` trusts this boundary and rejects requests without the identity header. Setting `THOTH_PUBLIC_EXPOSURE=true` with any other auth mode fails during core startup. Production credentials use the existing Compose secret-bundle contract, never environment values. Copy `deploy/secrets/thothii.secrets.example` to a protected host file, include only the required keys, and set its absolute path as `THT_SECRETS_FILE` in the operator env. Keep Pi's native provider auth in the separate protected file named by `PI_AUTH_FILE`. Interactive sessions, Description Generation, and embedding share the protected installation descriptor's `modelCatalog`. Set `THT_INSTALLATION_CONFIG_SOURCE` to that exact host file; `tht` validates it and generates the runtime catalog, Pi adapters, and Compose override before startup. Each authenticated provider stores only an audited `apiKeyEnv` reference; the referenced value stays in the secret bundle. A provider may use `authentication.mode: none` only with an explicit keyless endpoint. The browser receives only eligible model IDs, labels, and the catalog default. Before enabling Description Generation, approve the selected model provider for bounded source-data disclosure. Every catalog column has a **Sensitive** flag that defaults to `false`. Administrators can request an AI proposal based only on structural metadata, then must review and save the resulting checkboxes themselves. The proposal never reads column contents and is not persisted automatically. For unprotected columns, a request may send up to five real source rows and five representative distinct, non-null example values. Protected columns are omitted from source reads and replaced in the prompt by deterministic plausible values derived only from column metadata. Samples are transient and are not stored in generation runs, run logs, application logs, API responses, or catalog metadata; prompt and sample snapshots are not retained. A flag change applies to later generations and does not regenerate existing descriptions. Description Generation is an interactive Database Management operation, not a user-facing CLI. The installation runs at most one sequential generation at a time. The run drawer exposes safe ordered events through SSE with polling fallback, Stop terminates the current helper while keeping already stored results, and Run history retains terminal runs for inspection. A backend restart marks queued or running work interrupted instead of resuming it; use Generate Missing to continue. Unlock is reserved for a stale recorded run and is rejected while a local start, worker, or helper is still live. The bundle is mounted read-only as `/run/secrets/thothii.secrets` and must be mode `0600` or `0400` on the host. Docker's runtime `0444` mode is accepted only beneath `/run/secrets`; see [`deploy/secrets/README.md`](deploy/secrets/README.md). A PEM CA chain is deliberately not a bundle value: PEM contains whitespace and is rejected by the strict parser. Keep the CA chain in the host/secret-manager materialization and add a reviewed Compose override that mounts it at `/run/secrets/ca-chain.pem` and sets `THT_SSL_CA` when a private CA is required. The base bundle does not create that mount. The frontend remains on loopback; the authenticated host proxy is the only public listener. For each Pi spawn, the backend resolves the selected canonical provider/model in the runtime catalog, reads exactly that provider's declared `apiKeyEnv` value from the bundle, and exposes only that key to the child. Ambient provider credentials and secret-bundle paths are scrubbed. Providers needing a compound credential bundle remain unsupported until the catalog gains an explicit generic contract for them. ## User-owned session server cutover The server profile stores sessions and per-user preferences directly in PostgreSQL schema `thoth_sessions`; it does not use PostgREST, browser storage, a shared session directory, or a dual write. Use [`deploy/compose.session-server.yaml.example`](deploy/compose.session-server.yaml.example) with the canonical base+server files and set `THT_SERVER_WORKSPACE_CONFIG` to an absolute, protected copy of [`deploy/workspaces/server-sessions.yaml.example`](deploy/workspaces/server-sessions.yaml.example). That file is an installation runtime template, not an authored workspace descriptor; database bindings are injected from the PostgreSQL Metadata Catalog for each runtime lease. The runtime login needs membership in the no-login database role `thoth_sessions_runtime` only. The distinct, one-shot migrator login needs migration authority and uses `thoth_sessions_migrator`; it must never be mounted into `core`. Set the non-secret endpoint and role fields in the protected deployment environment: ```dotenv AUTH_MODE=upstream THOTH_PUBLIC_EXPOSURE=true THT_SESSION_STORAGE=postgres THT_SESSION_DB_HOST=sessions-db.internal THT_SESSION_DB_PORT=5432 THT_SESSION_DB_NAME=thoth THT_SESSION_RUNTIME_USER=thoth_sessions_app THT_SESSION_MIGRATOR_USER=thoth_sessions_migrate THT_SESSION_DB_SSLMODE=verify-full THT_SESSION_RUNTIME_PASSWORD_SOURCE=/secure/thoth/session-runtime-password THT_SESSION_MIGRATOR_PASSWORD_SOURCE=/secure/thoth/session-migrator-password THT_SESSION_CA_SOURCE=/secure/thoth/session-ca.pem ``` The overlay mounts the runtime password at `/run/secrets/session_runtime_password`, the CA at `/run/secrets/session_ca.pem`, and passes those paths—not their contents—to the server workspace. It mounts `session_migrator_password` only to `session-migrate`. The backend refuses a server session store without upstream authentication, direct DB host/name/runtime user/password-file, `verify-ca` or `verify-full`, and an absolute CA path. The migrator independently rejects every other TLS mode before reading its password secret or constructing a database URL. Perform the cutover in one maintenance window, with the upstream identity-proxy headers and backend principal parser deployed together. Neither change is safe to deploy independently: the proxy clears the legacy identity header and the backend rejects it. Drain/stop active Pi work, enable a maintenance response at the proxy, then run the migrator once and inspect its pristine JSON: ```sh docker compose --env-file deploy/env/server.env \ -f compose.yaml -f deploy/compose.server.yaml -f deploy/compose.session-server.yaml.example \ --profile session-migrate run --rm session-migrate ``` It must report no pending or drifted migrations before starting the replacement core. `/health` is a liveness probe and remains `200`; any request that needs unavailable repository storage returns a fixed `503` before a Pi process starts. Verify this with an authenticated request after the replacement core is healthy, then remove maintenance mode. Do not import the three legacy server filesystem sessions: they have no trusted owner binding. During the same maintenance window, archive the exact three reviewed IDs, verify the generated archive and `.sha256`, then rerun the command with `--delete` to remove only those three source directories: ```sh ./docker/cutover-legacy-sessions.sh \ /secure/thoth/legacy-sessions /secure/backups/thoth-legacy-sessions-2026-07-16.tar \ SESSION_ID_1 SESSION_ID_2 SESSION_ID_3 # After independent archive review, use a new backup filename: ./docker/cutover-legacy-sessions.sh --delete \ /secure/thoth/legacy-sessions /secure/backups/thoth-legacy-sessions-2026-07-16-delete.tar \ SESSION_ID_1 SESSION_ID_2 SESSION_ID_3 ``` The helper refuses to overwrite an existing backup and refuses any count other than three distinct IDs. Never run it against a live path without the maintenance gate. Roll back application code only by keeping PostgreSQL as the single source of truth and deploying a compatible fixed release. Do not restore filesystem persistence, do not re-import the archive, and never dual-write sessions to database and files. ## Reproducible image verification Base images use exact tags and immutable multi-platform manifest digests. Dependency update and residual OS-repository limitations are documented in [`docker/LOCKS.md`](docker/LOCKS.md). Run the shared architecture gate with `PLATFORM=linux/amd64` or `PLATFORM=linux/arm64`: ```sh PLATFORM=linux/arm64 ./scripts/verify-container-images.sh ``` It builds both images, runs common version/runtime/security smokes, and emits an image/package inventory beneath `.artifacts/container-images/`. CI runs the same script for both architectures.