ThothII/README.md

# ThothII

ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/`tht`
core. The portable deployment runs exactly two application services; data services remain
external in this profile, except for the mandatory internal semantic services bundled in Compose.

Authentication is configured through the single host CLI tht: see the [local authentication guide](docs/install/authentication-local.md),
[generic OIDC guide](docs/install/authentication-oidc.md), and [manual acceptance matrix](docs/testing/authentication-manual-acceptance.md).

## Docker Compose: local startup

Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external,
configurable endpoints—even when they are co-located with ThothII.

From a fresh clone, run these commands from the repository root:

```sh
cp deploy/env/local.env.example deploy/env/local.env
# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
docker compose --env-file deploy/env/local.env \
  -f compose.yaml -f deploy/compose.local.yaml up --build -d
```

`./scripts/run-stack.sh` runs this same base+local command in the foreground. The core image
contains its Pi runtime; no host `pi` executable is used. For a server installation:

```sh
cp deploy/env/server.env.example deploy/env/server.env
# Edit all absolute storage, Pi/secret/session files, and endpoint paths.
sudo scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001
docker compose --env-file deploy/env/server.env \
  -f compose.yaml -f deploy/compose.server.yaml \
  -f deploy/compose.session-server.yaml.example up --build -d
```

The initializer is required for an empty or restored server Pi-state bind. It atomically creates
the three regular targets hidden below the writable parent bind; protected Pi auth and tracked
model/settings sources remain separate read-only mounts. See the server manual before substituting
a root other than `/srv/thothii/pi-state`.

Workspace descriptors come from the Git remote configured by `THT_WORKSPACE_GIT_REMOTE`; their
runtime endpoint and secret bindings remain installation-local. Open
<http://127.0.0.1:8080> (set `THOTH_HTTP_PORT` in `deploy/env/local.env` to choose another
loopback port).

Credentials and certificates are local protected files. Do not put them in environment examples,
workspace YAML, URLs, or Compose interpolation values.

Application state is split across the named `settings`, `pi-state`, `workspace-registry`,
`sessions`, `qdrant-data`, and `embedding-models` volumes. `docker compose down` keeps them.
`qdrant-data` is a derived but persistent index store; `embedding-models` is an Ollama model
cache for `qwen3-embedding:0.6b` with fixed `1024`-dimension embeddings. Only an explicit destructive command such as `docker compose
down --volumes` removes them.

The frontend depends on the core health check and proxies `/health` and `/api/*` to it. The
application health endpoint intentionally checks process readiness only; external dependency
diagnostics are exposed by `tht doctor` and do not prevent the UI from starting.

## Git-backed workspace repository

Workspace descriptors are shared through a validated Git repository while endpoint bindings and
secret files remain installation-local. Use the [local Mac/PC installation manual](docs/install/local-workspace-registry.md)
for Docker Desktop or a local engine, the [server installation manual](docs/install/server-workspace-registry.md)
for the Gitea, reverse-proxy, backup, upgrade, and recovery workflow, and the
[P1→P1.1 migration guide](docs/migrations/p1-to-p1-1-registry-layout.md) before upgrading an
older flat-layout registry.

The curator-owned repository layout is:

```text
thoth-workspaces.yaml
<id>/workspace.yaml
<id>/evidence/**
```

`thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered `workspaces` list of
`{id, name, description?}` entries. It is authoritative for workspace ID, name, description, and
display order. Every catalog entry must have a matching descriptor in the same commit; otherwise
the complete candidate is rejected. Descriptors remain curator-owned and change only through a
Git commit and push from a separate authoring clone, followed by an installation pull. ThothII
never writes any workspace repository content.

The operator workflow is: curate catalog/descriptor/Evidence changes in Git, commit and push,
**Update workspace repository** from each ThothII installation, select the workspace, complete its
write-only runtime-secret fields, run **Validate workspace source** and **Test workspace
connections**, then select the workspace locally before creating sessions. Each new session
pins the Git revision it used; a later pull cannot change a Resume. Snapshot cleanup retains every
revision referenced by an open, closed, or failed unarchived session. It reconciles from the
single local installation list or from a server administrator's complete session list, never from
a remote user's partial list. The isolated deployment exercise is
`./scripts/workspace-registry-smoke.sh`; both manuals are checked with
`./scripts/verify-workspace-install-docs.sh --profile local` or `--profile server`.

<!-- workspace-descriptor-contract:start -->
Schema v3 is the only accepted workspace descriptor. Schema v1 and v2 workspace descriptors are
rejected before activation. Candidate snapshot validation therefore makes activation or a pull fail
atomically while the prior valid snapshot remains active. There is no in-product migrator or
automatic conversion. A repository must already contain reviewed v3 descriptors. One workspace
owns one Qdrant collection;
schema, Evidence, and Memory records share that collection and stay separated by indexed payload
`kind`.
<!-- workspace-descriptor-contract:end -->

Connector `ssh_tunnel` bindings are diagnostic-only in this release: their bounded probe always
cleans up the loopback forward and returns `workspace_not_activatable`; session creation is rejected
before persistence. Git registry access over SSH is unaffected. Use direct or REST connector
transport for runtime sessions.

`docker-compose.dev.yml` is deliberately local: both published ports bind to `127.0.0.1`,
`THT_SESSION_STORAGE=local`, and `THT_HOME=/data/local-home`. Do not set
`THOTH_PUBLIC_EXPOSURE=true` for that profile; the backend rejects that public/local combination
at startup.

Run the end-to-end packaging check with:

```sh
./scripts/docker-smoke.sh
```

The smoke script validates Compose, builds and waits for both services, checks health through
the frontend, verifies SSE response headers, restarts the core, and confirms `/data` survives.
Each run uses a unique Compose project and removes that project's containers, network, and test
volume afterward. It never targets the fixed `thothii` operator project or its volume. Set
`SMOKE_PROJECT` to a different explicit project name for reproducible debugging, and set
`KEEP_SMOKE_RESOURCES=1` to retain that smoke project's resources for inspection; remove them
later with `docker compose --project-name "$SMOKE_PROJECT" down --volumes`.

## Unified deployment release gates

Task 13 adds no-secret release gates around the canonical local and server Compose profiles. Its
deterministic safety check does not contact the Docker daemon:

```sh
bash scripts/unified-deployment-smoke.sh --self-test
```

The three Linux Docker smokes are separate release commands. Each creates a unique Compose project, temporary
Git workspace remote, fixture provider, image names, and run label. Its exit trap removes only
resources carrying that exact run identity and never performs a global Docker prune. Cleanup
enumerates running and stopped project containers immediately before `compose down` and refuses
the teardown if any container, volume, or network has a foreign run label.

```sh
bash scripts/unified-deployment-smoke.sh
bash scripts/thothctl-update-smoke.sh
bash scripts/server-deployment-smoke.sh
```

The unified smoke builds and starts `frontend` and `core`, verifies the embedded Pi and internal
registry, recreates with the Git remote offline, activates a valid Git update, rejects invalid Git
content while retaining the valid snapshot, and checks the four persistence volumes. The unified
and update-only smokes
inject a digest-pinned non-core candidate under a deliberately mismatched Pi version and require
`thothctl pi update` to roll back while preserving settings, sessions, Pi state, registry revision,
and mount identity. The rollback candidate is the digest-pinned `hello-world` executable: a
preflight proves that it exits successfully, so the failed replacement core satisfies
`thothctl`'s stopped-core compensation precondition. The server smoke uses the same smoke-built
core/frontend images with the server and required session overlays, disposable bind roots and
secret files, upstream-auth checks, and a fail-closed `503` assertion for its deliberately
unavailable disposable session endpoint. No real provider, database credential, or repository
secret is required.

For a clean server bind, `scripts/prepare-server-pi-state.sh` creates the hidden regular
`agent/auth.json`, `agent/models.json`, and `agent/settings.json` mount targets atomically before
Compose. The server smoke starts from an empty Pi-state root and applies this same preflight; the
real protected/tracked sources remain separate read-only mounts. Deterministic fixture tests render
both profiles, verify that bindings stay on `core`, check mount readability, and run the production
workspace resolver. Wrong-service, wrong-value, and broken-secret-mount mutations must fail.

Each public smoke has its own 30-minute process-group supervisor with TERM/KILL cleanup; CI retains
an independent 32-minute outer timeout and does not retry a failed command.

Current release status (2026-08-05): clean-root render/setup and the production runtime-binding
resolver contracts are green. The server fixture supplies all four private trusted claims,
including exact non-admin value `0`, and a focused test proves nginx normalization produces the
accepted non-admin backend principal. Canonical schema-v3 registry descriptors now pass through
one backend-owned, secret-safe runtime handoff for inventory and session execution; canonical
identity and durable session/artifact/index roots are retained. The fresh update-only smoke passed
bad-candidate mutation, automatic `rolled_back` compensation, exact prior-image restoration,
unchanged registry/mount identity, all four sentinels, post-rollback doctor/workspace checks, and
exact cleanup. The one authorized server-smoke invocation was denied access to the Docker socket
by its execution sandbox before startup, so the complete authenticated workspace and fail-closed
session assertions still require a fresh authorized release run. Native Windows Docker
Desktop/WSL2 remains a separate manual/self-hosted gate.

The deterministic native Windows contract is:

```powershell
.\scripts\test-windows-clone-contract.ps1
```

It checks Git's CRLF/LF attributes and bytes, copies tracked source into a temporary path containing
spaces, builds and invokes native Windows `thothctl` there, and renders exactly `core` plus
`frontend` without starting containers. On a supported self-hosted Windows Docker Desktop/WSL2
runner, dispatch the deployment workflow with `windows_docker_startup=true`; that job executes:

```powershell
.\scripts\test-windows-clone-contract.ps1 -DockerStartup
```

Startup mode adds bounded image build/two-service health startup, installation-aware `thothctl`
status, stopped-container-aware ownership checks, and exact cleanup. The ordinary hosted Windows
job remains deterministic and does not claim Docker startup.

## Preprocessing jobs and S3 Evidence

The included preprocessing services reuse the internal Qdrant/Ollama stack. Mount Evidence at
`/data/source/evidence`, then run the explicit preprocessing preset:

```sh
docker compose --env-file deploy/env/local.env \
  -f compose.yaml -f deploy/compose.local.yaml \
  -f deploy/compose.preprocess.yaml --profile preprocess run --rm preprocess-evidence
```

Replace the final service with `preprocess-dwh` when required. The overlay makes each job wait for the internal Qdrant
service health checks and embedding model initialization; no separate semantic-service startup is
required.

S3 Evidence uses the optional `tht[s3]` dependency and canonical `s3://bucket/key` provenance.
AWS endpoints are used when no custom URL is supplied. Every custom endpoint is an explicit egress
trust-boundary opt-in and uses path-style addressing; private and HTTP endpoints require additional
independent opt-ins. Literal non-global IPv4/IPv6 addresses are classified locally; hostnames are
not DNS-pinned, so trusted custom-endpoint deployments must enforce their destination with network
egress policy. Store access key, secret key, and session token as secret references in
deployment configuration—never in Compose environment values or source URIs. Discovery and reads
are bounded by configured page, object, and byte limits.

Create a versioned Qdrant volume backup for one exact Compose project (the filename is
operator-controlled, so use an immutable timestamp or release identifier):

```sh
./scripts/vector-backup.sh \
  --project-name thothii \
  --output /secure/backups/thoth-qdrant-2026-08-08.tar
```

The script resolves exactly one Docker volume with the labels
`com.docker.compose.project=<project>` and `com.docker.compose.volume=qdrant-data`, stops the
`qdrant` service if it is running, archives that volume's persistent contents, then restores the
prior service state. It never performs global Docker cleanup and refuses to overwrite an existing
archive path.

Restore targets that same exact project-scoped `qdrant-data` volume. Because restore replaces the
persistent Qdrant data in place, it requires an explicit confirmation that exactly repeats the
Compose project name by passing `--confirm-project`:

```sh
./scripts/vector-restore.sh \
  --project-name thothii \
  --input /secure/backups/thoth-qdrant-2026-08-08.tar \
  --confirm-project thothii
```

The restore script stops `qdrant`, validates the exact labeled target, stages the current volume
contents for rollback, extracts the requested archive into the volume, and then returns the
service to its prior running state. It restores semantic storage only. Before reopening write
traffic, the workspace registry must already be at a reviewed v3 descriptor revision compatible
with the restored collection; then run backend health checks and a known retrieval query. The
helper does not restore descriptors, rename collections, or reconcile an incompatible collection
contract.

## Production trust boundary and secrets

ThothII does not implement OIDC. Do not expose its application port directly to a network.
The production pattern is an authenticated host reverse proxy that:

- terminates TLS and authenticates every request;
- removes any client-supplied identity header;
- injects one trusted `X-Authenticated-User` value;
- proxies to the loopback-only ThothII frontend.

[`deploy/nginx-authenticated-proxy.conf.example`](deploy/nginx-authenticated-proxy.conf.example)
shows the contract using nginx `auth_request`; replace the placeholder authentication gateway
with the organization's reviewed identity proxy. `AUTH_MODE=upstream` trusts this boundary and
rejects requests without the identity header. Setting `THOTH_PUBLIC_EXPOSURE=true` with any other
auth mode fails during core startup.

Production credentials use the existing Compose secret-bundle contract, never environment values.
Copy `deploy/secrets/thothii.secrets.example` to a protected host file, include only the required
keys, and set its absolute path as `THT_SECRETS_FILE` in the operator env. Keep Pi's native
provider auth in the separate protected file named by `PI_AUTH_FILE`.

The bundle is mounted read-only as `/run/secrets/thothii.secrets` and must be mode `0600` or
`0400` on the host. Docker's runtime `0444` mode is accepted only beneath `/run/secrets`; see
[`deploy/secrets/README.md`](deploy/secrets/README.md). A PEM CA chain is deliberately not a
bundle value: PEM contains whitespace and is rejected by the strict parser. Keep the CA chain in
the host/secret-manager materialization and add a reviewed Compose override that mounts it at
`/run/secrets/ca-chain.pem` and sets `THT_SSL_CA` when a private CA is required. The base bundle
does not create that mount. The frontend remains on loopback; the authenticated host proxy is the
only public listener.

Set the selected model provider in application settings (or `PI_PROVIDER`). For each Pi spawn the
backend validates and reads `THT_MODEL_API_KEY` from the bundle, then exposes its value only as the provider's
recognized child variable (for example `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, or
`ZAI_API_KEY`). Neither the generic file path nor deprecated `PI_PROVIDER_API_KEY` is inherited by
Pi. Local providers such as Ollama require no model key.

`THT_MODEL_API_KEY` supports Pi providers whose authentication is exactly one key:
`ant-ling`, `anthropic`, `cerebras`, `deepseek`, `fireworks`, `github-copilot`, `google`
(including the `gemini` alias), `google-vertex` when using its API-key mode, `groq`,
`huggingface`, `kimi-coding`, `minimax`, `minimax-cn`, `mistral`, `moonshotai`,
`moonshotai-cn`, `nvidia`, `openai`, `opencode`, `opencode-go`, `openrouter`, `together`,
`vercel-ai-gateway`, `xai`, the four `xiaomi*` providers, `zai`, and `zai-coding-cn`.
Compound providers are deliberately unsupported: `amazon-bedrock`, `azure-openai-responses`,
`cloudflare-workers-ai`, and `cloudflare-ai-gateway` require multiple credential/configuration
values. Selecting one fails before Pi starts; ambient AWS, Azure, and Cloudflare credentials are
still scrubbed. Supporting them requires a future dedicated provider-specific configuration.

## User-owned session server cutover

The server profile stores sessions and per-user preferences directly in PostgreSQL schema
`thoth_sessions`; it does not use PostgREST, browser storage, a shared session directory, or a
dual write. Use [`deploy/compose.session-server.yaml.example`](deploy/compose.session-server.yaml.example)
with the canonical base+server files and set `THT_SERVER_WORKSPACE_CONFIG` to an absolute,
protected copy of [`deploy/workspaces/server-sessions.yaml.example`](deploy/workspaces/server-sessions.yaml.example).

The runtime login needs membership in the no-login database role `thoth_sessions_runtime` only.
The distinct, one-shot migrator login needs migration authority and uses
`thoth_sessions_migrator`; it must never be mounted into `core`. Set the non-secret endpoint and
role fields in the protected deployment environment:

```dotenv
AUTH_MODE=upstream
THOTH_PUBLIC_EXPOSURE=true
THT_SESSION_STORAGE=postgres
THT_SESSION_DB_HOST=sessions-db.internal
THT_SESSION_DB_PORT=5432
THT_SESSION_DB_NAME=thoth
THT_SESSION_RUNTIME_USER=thoth_sessions_app
THT_SESSION_MIGRATOR_USER=thoth_sessions_migrate
THT_SESSION_DB_SSLMODE=verify-full
THT_SESSION_RUNTIME_PASSWORD_SOURCE=/secure/thoth/session-runtime-password
THT_SESSION_MIGRATOR_PASSWORD_SOURCE=/secure/thoth/session-migrator-password
THT_SESSION_CA_SOURCE=/secure/thoth/session-ca.pem
```

The overlay mounts the runtime password at `/run/secrets/session_runtime_password`, the CA at
`/run/secrets/session_ca.pem`, and passes those paths—not their contents—to the server workspace.
It mounts `session_migrator_password` only to `session-migrate`. The backend refuses a server
session store without upstream authentication, direct DB host/name/runtime user/password-file,
`verify-ca` or `verify-full`, and an absolute CA path.
The migrator independently rejects every other TLS mode before reading its password secret or
constructing a database URL.

Perform the cutover in one maintenance window, with the upstream identity-proxy headers and
backend principal parser deployed together. Neither change is safe to deploy independently: the
proxy clears the legacy identity header and the backend rejects it. Drain/stop active Pi work,
enable a maintenance response at the proxy, then run the migrator once and inspect its pristine JSON:

```sh
docker compose --env-file deploy/env/server.env \
  -f compose.yaml -f deploy/compose.server.yaml -f deploy/compose.session-server.yaml.example \
  --profile session-migrate run --rm session-migrate
```

It must report no pending or drifted migrations before starting the replacement core. `/health`
is a liveness probe and remains `200`; any request that needs unavailable repository storage
returns a fixed `503` before a Pi process starts. Verify this with an authenticated request after
the replacement core is healthy, then remove maintenance mode.

Do not import the three legacy server filesystem sessions: they have no trusted owner binding.
During the same maintenance window, archive the exact three reviewed IDs, verify the generated
archive and `.sha256`, then rerun the command with `--delete` to remove only those three source
directories:

```sh
./docker/cutover-legacy-sessions.sh \
  /secure/thoth/legacy-sessions /secure/backups/thoth-legacy-sessions-2026-07-16.tar \
  SESSION_ID_1 SESSION_ID_2 SESSION_ID_3
# After independent archive review, use a new backup filename:
./docker/cutover-legacy-sessions.sh --delete \
  /secure/thoth/legacy-sessions /secure/backups/thoth-legacy-sessions-2026-07-16-delete.tar \
  SESSION_ID_1 SESSION_ID_2 SESSION_ID_3
```

The helper refuses to overwrite an existing backup and refuses any count other than three
distinct IDs. Never run it against a live path without the maintenance gate. Roll back application
code only by keeping PostgreSQL as the single source of truth and deploying a compatible fixed
release. Do not restore filesystem persistence, do not re-import the archive, and never dual-write
sessions to database and files.

## Reproducible image verification

Base images use exact tags and immutable multi-platform manifest digests. Dependency update and
residual OS-repository limitations are documented in [`docker/LOCKS.md`](docker/LOCKS.md).
Run the shared architecture gate with `PLATFORM=linux/amd64` or `PLATFORM=linux/arm64`:

```sh
PLATFORM=linux/arm64 ./scripts/verify-container-images.sh
```

It builds both images, runs common version/runtime/security smokes, and emits an image/package
inventory beneath `.artifacts/container-images/`. CI runs the same script for both architectures.