ThothII/README.md

# ThothII

ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/`tht`
core. The portable deployment runs exactly two application services; data services remain
external in this profile.

## Docker Compose: one-command startup

Requirements: Docker Engine with Compose v2. The default project starts only the two
application images; DWH, vector and embedding services can be remote or supplied by an
optional overlay.

From a fresh clone, run these commands from the repository root:

```sh
cp .env.example .env
cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets
chmod 600 deploy/secrets/thothii.secrets
# Edit .env (non-secret endpoints) and deploy/secrets/thothii.secrets (KEY=VALUE lines).
docker compose up --build -d
```

The root `.env` is loaded automatically by Compose. It defaults to `compose.yaml`, an empty
profile, and `THT_SECRETS_FILE=deploy/secrets/thothii.secrets`; no `--env-file`, `-f`, or
`--profile` flag is required for the normal installation. Add or edit YAML workspace descriptors
under `deploy/workspaces/`; they are mounted read-only and relative `roots` resolve beneath
`/data/workspaces/<workspace-name>`. Open <http://127.0.0.1:8080> (set `THOTH_HTTP_PORT` in
`.env` to choose another loopback port).

The bundle contains only values, one per line (`THT_MODEL_API_KEY=...`, DWH/vector keys, and
the optional local-vector passwords). It is ignored by Git and never copied into either image.
Do not put credentials in `.env`, workspace YAML, URLs, or Compose interpolation values.

### Optional overlays

Overlays are selected in `.env`, so the operational command remains the same. On Unix-like
systems use `:` between files; on Windows use `;`:

```dotenv
# Remote DWH/vector/embedding services with authenticated reverse proxy:
COMPOSE_FILE=compose.yaml:deploy/compose.production.yaml
COMPOSE_PROFILES=

# Local pgvector (Mac/Windows or a standalone application server):
COMPOSE_FILE=compose.yaml:deploy/compose.local-vector.yaml
COMPOSE_PROFILES=local-vector
```

After changing `.env`, apply the selected configuration with `docker compose up --build -d`.
Preprocessing is an explicit opt-in preset: append
`deploy/compose.preprocess.yaml:deploy/compose.preprocess-local-vector.yaml` and set
`COMPOSE_PROFILES=local-vector,preprocess`; then run the job with
`docker compose run --rm preprocess-evidence` or `preprocess-dwh`.

Application state, including settings, sessions, artifacts, and indexes, lives in the named
`thoth_data` volume mounted at `/data`. `docker compose down` keeps that volume. Only an
explicit destructive command such as `docker compose down --volumes` removes it.

The frontend depends on the core health check and proxies `/health` and `/api/*` to it. The
application health endpoint intentionally checks process readiness only; external dependency
diagnostics are exposed by `tht doctor` and do not prevent the UI from starting.

`docker-compose.dev.yml` is deliberately local: both published ports bind to `127.0.0.1`,
`THT_SESSION_STORAGE=local`, and `THT_HOME=/data/local-home`. Do not set
`THOTH_PUBLIC_EXPOSURE=true` for that profile; the backend rejects that public/local combination
at startup.

Run the end-to-end packaging check with:

```sh
./scripts/docker-smoke.sh
```

The smoke script validates Compose, builds and waits for both services, checks health through
the frontend, verifies SSE response headers, restarts the core, and confirms `/data` survives.
Each run uses a unique Compose project and removes that project's containers, network, and test
volume afterward. It never targets the fixed `thothii` operator project or its volume. Set
`SMOKE_PROJECT` to a different explicit project name for reproducible debugging, and set
`KEEP_SMOKE_RESOURCES=1` to retain that smoke project's resources for inspection; remove them
later with `docker compose --project-name "$SMOKE_PROJECT" down --volumes`.

## Optional local pgvector and recovery

The local-vector overlay reads `THT_VECTOR_BOOTSTRAP_PASSWORD`,
`THT_VECTOR_MIGRATOR_PASSWORD`, `THT_VECTOR_READER_PASSWORD`, and
`THT_VECTOR_WRITER_PASSWORD` from the same bundle. Its `vector_data` volume is independent of
application state; passwords are selected at runtime and are never passed as URL arguments.

## Preprocessing jobs and S3 Evidence

The included job workspaces target the local-vector profile. Put the four local-vector password
keys in the bundle, set `THT_OLLAMA_URL`, mount Evidence at `/data/source/evidence`, then select
the preprocessing preset in `.env`:

```dotenv
COMPOSE_FILE=compose.yaml:deploy/compose.local-vector.yaml:deploy/compose.preprocess.yaml:deploy/compose.preprocess-local-vector.yaml
COMPOSE_PROFILES=local-vector,preprocess
```

Run `docker compose run --rm preprocess-evidence` or
`docker compose run --rm preprocess-dwh`. The overlay makes each job wait for the vector
database health check, role reconciliation, and a successful migration; no separate database
startup or migration command is required.

S3 Evidence uses the optional `tht[s3]` dependency and canonical `s3://bucket/key` provenance.
AWS endpoints are used when no custom URL is supplied. Every custom endpoint is an explicit egress
trust-boundary opt-in and uses path-style addressing; private and HTTP endpoints require additional
independent opt-ins. Literal non-global IPv4/IPv6 addresses are classified locally; hostnames are
not DNS-pinned, so trusted custom-endpoint deployments must enforce their destination with network
egress policy. Store access key, secret key, and session token as secret references in
deployment configuration—never in Compose environment values or source URIs. Discovery and reads
are bounded by configured page, object, and byte limits.

Create a versioned PostgreSQL custom-format backup (the filename is operator-controlled, so use
an immutable timestamp or release identifier):

```sh
./scripts/vector-backup.sh \
  --host 127.0.0.1 --port 5432 --database thoth --user thoth_backup \
  --password-file /secure/thoth/vector-backup-password \
  --output /secure/backups/thoth-vectors-2026-07-12.dump
```

The dump contains the three allowlisted `vectors` tables, their data and ACLs, plus the
`public.tht_vector_migrations` ledger. Login roles and passwords are deliberately not copied:
provision/reconcile the approved role names on the target first, and install the `vector`
extension in its `vectors` schema. The target must otherwise contain no vector tables or ledger.

Restore always names both the currently active source and a target on a physically distinct
PostgreSQL cluster. The script compares PostgreSQL system identity, so host aliases or a different
database in the active cluster cannot bypass the guard. It refuses a non-empty target unless
`--force-nonempty` is explicit, and the clean restore is one transaction:

```sh
./scripts/vector-restore.sh \
  --active-host vector-db --active-database thoth --active-user thoth_backup \
  --active-password-file /secure/thoth/vector-active-password \
  --target-host vector-db-restore --target-database thoth --target-user thoth_restore \
  --target-password-file /secure/thoth/vector-restore-password \
  --input /secure/backups/thoth-vectors-2026-07-12.dump
```

After restore, run `tht vector migrate --status --json`, adapter health, and a known retrieval
query against the target before changing any deployment endpoint. Never test recovery against the
active `vector_data` volume. `./scripts/local-vector-smoke.sh --backup-restore` performs this drill
with disposable source and target volumes.

## Production trust boundary and secrets

ThothII does not implement OIDC. Do not expose its application port directly to a network.
The production pattern is an authenticated host reverse proxy that:

- terminates TLS and authenticates every request;
- removes any client-supplied identity header;
- injects one trusted `X-Authenticated-User` value;
- proxies to the loopback-only ThothII frontend.

[`deploy/nginx-authenticated-proxy.conf.example`](deploy/nginx-authenticated-proxy.conf.example)
shows the contract using nginx `auth_request`; replace the placeholder authentication gateway
with the organization's reviewed identity proxy. `AUTH_MODE=upstream` trusts this boundary and
rejects requests without the identity header. Setting `THOTH_PUBLIC_EXPOSURE=true` with any other
auth mode fails during core startup.

Production credentials use the one Compose secret bundle, not `.env`. Put the required keys in
`deploy/secrets/thothii.secrets` and select the production overlay in `.env`:

```dotenv
THT_MODEL_API_KEY=replace-me
THT_DWH_API_KEY=replace-me
THT_VEC_API_KEY=replace-me
THT_VEC_WRITE_API_KEY=replace-me
```

The bundle is mounted read-only as `/run/secrets/thothii.secrets` and must be mode `0600` or
`0400` on the host. Docker's runtime `0444` mode is accepted only beneath `/run/secrets`; see
[`deploy/secrets/README.md`](deploy/secrets/README.md). A PEM CA chain is deliberately not a
bundle value: PEM contains whitespace and is rejected by the strict parser. Keep the CA chain in
the host/secret-manager materialization and add a reviewed Compose override that mounts it at
`/run/secrets/ca-chain.pem` and sets `THT_SSL_CA` when a private CA is required. The base bundle
does not create that mount. The frontend remains on loopback; the authenticated host proxy is the
only public listener.

Set the selected model provider in application settings (or `PI_PROVIDER`). For each Pi spawn the
backend validates and reads `THT_MODEL_API_KEY` from the bundle, then exposes its value only as the provider's
recognized child variable (for example `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, or
`ZAI_API_KEY`). Neither the generic file path nor deprecated `PI_PROVIDER_API_KEY` is inherited by
Pi. Local providers such as Ollama require no model key.

`THT_MODEL_API_KEY` supports Pi providers whose authentication is exactly one key:
`ant-ling`, `anthropic`, `cerebras`, `deepseek`, `fireworks`, `github-copilot`, `google`
(including the `gemini` alias), `google-vertex` when using its API-key mode, `groq`,
`huggingface`, `kimi-coding`, `minimax`, `minimax-cn`, `mistral`, `moonshotai`,
`moonshotai-cn`, `nvidia`, `openai`, `opencode`, `opencode-go`, `openrouter`, `together`,
`vercel-ai-gateway`, `xai`, the four `xiaomi*` providers, `zai`, and `zai-coding-cn`.
Compound providers are deliberately unsupported: `amazon-bedrock`, `azure-openai-responses`,
`cloudflare-workers-ai`, and `cloudflare-ai-gateway` require multiple credential/configuration
values. Selecting one fails before Pi starts; ambient AWS, Azure, and Cloudflare credentials are
still scrubbed. Supporting them requires a future dedicated provider-specific configuration.

## User-owned session server cutover

The server profile stores sessions and per-user preferences directly in PostgreSQL schema
`thoth_sessions`; it does not use PostgREST, browser storage, a shared session directory, or a
dual write. Start from [`deploy/compose.session-server.yaml.example`](deploy/compose.session-server.yaml.example)
and copy [`deploy/workspaces/server-sessions.yaml.example`](deploy/workspaces/server-sessions.yaml.example)
to the untracked `deploy/workspaces/server-sessions.yaml` mounted into the core container.

The runtime login needs membership in the no-login database role `thoth_sessions_runtime` only.
The distinct, one-shot migrator login needs migration authority and uses
`thoth_sessions_migrator`; it must never be mounted into `core`. Set the non-secret endpoint and
role fields in the protected deployment environment:

```dotenv
AUTH_MODE=upstream
THOTH_PUBLIC_EXPOSURE=true
THT_SESSION_STORAGE=postgres
THT_SESSION_DB_HOST=sessions-db.internal
THT_SESSION_DB_PORT=5432
THT_SESSION_DB_NAME=thoth
THT_SESSION_RUNTIME_USER=thoth_sessions_app
THT_SESSION_MIGRATOR_USER=thoth_sessions_migrate
THT_SESSION_DB_SSLMODE=verify-full
THT_SESSION_RUNTIME_PASSWORD_SOURCE=/secure/thoth/session-runtime-password
THT_SESSION_MIGRATOR_PASSWORD_SOURCE=/secure/thoth/session-migrator-password
THT_SESSION_CA_SOURCE=/secure/thoth/session-ca.pem
```

The overlay mounts the runtime password at `/run/secrets/session_runtime_password`, the CA at
`/run/secrets/session_ca.pem`, and passes those paths—not their contents—to the server workspace.
It mounts `session_migrator_password` only to `session-migrate`. The backend refuses a server
session store without upstream authentication, direct DB host/name/runtime user/password-file,
`verify-ca` or `verify-full`, and an absolute CA path.
The migrator independently rejects every other TLS mode before reading its password secret or
constructing a database URL.

Perform the cutover in one maintenance window, with the Task 4 portal proxy headers and Task 5
backend principal parser deployed together. Neither change is safe to deploy independently: Task
4 clears the legacy identity header and Task 5 rejects it. Drain/stop active Pi work, enable a
maintenance response at the portal, then run the migrator once and inspect its pristine JSON:

```sh
docker compose -f compose.yaml -f deploy/compose.session-server.yaml \
  --profile session-migrate run --rm session-migrate
```

It must report no pending or drifted migrations before starting the replacement core. `/health`
is a liveness probe and remains `200`; any request that needs unavailable repository storage
returns a fixed `503` before a Pi process starts. Verify this with an authenticated request after
the replacement core is healthy, then remove maintenance mode.

Do not import the three legacy server filesystem sessions: they have no trusted owner binding.
During the same maintenance window, archive the exact three reviewed IDs, verify the generated
archive and `.sha256`, then rerun the command with `--delete` to remove only those three source
directories:

```sh
./docker/cutover-legacy-sessions.sh \
  /secure/thoth/legacy-sessions /secure/backups/thoth-legacy-sessions-2026-07-16.tar \
  SESSION_ID_1 SESSION_ID_2 SESSION_ID_3
# After independent archive review, use a new backup filename:
./docker/cutover-legacy-sessions.sh --delete \
  /secure/thoth/legacy-sessions /secure/backups/thoth-legacy-sessions-2026-07-16-delete.tar \
  SESSION_ID_1 SESSION_ID_2 SESSION_ID_3
```

The helper refuses to overwrite an existing backup and refuses any count other than three
distinct IDs. Never run it against a live path without the maintenance gate. Roll back application
code only by keeping PostgreSQL as the single source of truth and deploying a compatible fixed
release. Do not restore filesystem persistence, do not re-import the archive, and never dual-write
sessions to database and files.

## Reproducible image verification

Base images use exact tags and immutable multi-platform manifest digests. Dependency update and
residual OS-repository limitations are documented in [`docker/LOCKS.md`](docker/LOCKS.md).
Run the shared architecture gate with `PLATFORM=linux/amd64` or `PLATFORM=linux/arm64`:

```sh
PLATFORM=linux/arm64 ./scripts/verify-container-images.sh
```

It builds both images, runs common version/runtime/security smokes, and emits an image/package
inventory beneath `.artifacts/container-images/`. CI runs the same script for both architectures.