291 lines
16 KiB
Markdown
291 lines
16 KiB
Markdown
# ThothII
|
|
|
|
ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/`tht`
|
|
core. The portable deployment runs exactly two application services; data services remain
|
|
external in this profile.
|
|
|
|
## Docker Compose: local startup
|
|
|
|
Requirements: Docker Engine with Compose v2. The mandatory stack is exactly the `core` and
|
|
`frontend` application images. DWH, vector DB, embedding, and LLM services are external,
|
|
configurable endpoints—even when they are co-located with ThothII.
|
|
|
|
From a fresh clone, run these commands from the repository root:
|
|
|
|
```sh
|
|
cp deploy/env/local.env.example deploy/env/local.env
|
|
# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
|
|
docker compose --env-file deploy/env/local.env \
|
|
-f compose.yaml -f deploy/compose.local.yaml up --build -d
|
|
```
|
|
|
|
`./scripts/run-stack.sh` runs this same base+local command in the foreground. The core image
|
|
contains its Pi runtime; no host `pi` executable is used. For a server installation:
|
|
|
|
```sh
|
|
cp deploy/env/server.env.example deploy/env/server.env
|
|
# Edit all absolute storage, Pi/secret/session files, and endpoint paths.
|
|
docker compose --env-file deploy/env/server.env \
|
|
-f compose.yaml -f deploy/compose.server.yaml \
|
|
-f deploy/compose.session-server.yaml.example up --build -d
|
|
```
|
|
|
|
Workspace descriptors come from the Git remote configured by `THT_WORKSPACE_GIT_REMOTE`; their
|
|
runtime endpoint and secret bindings remain installation-local. Open
|
|
<http://127.0.0.1:8080> (set `THOTH_HTTP_PORT` in `deploy/env/local.env` to choose another
|
|
loopback port).
|
|
|
|
Credentials and certificates are local protected files. Do not put them in environment examples,
|
|
workspace YAML, URLs, or Compose interpolation values. The optional `local-vector` and
|
|
preprocessing overlays are development presets; they do not change the two-service mandatory
|
|
stack or the external-endpoint contract.
|
|
|
|
Application state is split across the named `settings`, `pi-state`, `workspace-registry`, and
|
|
`sessions` volumes. `docker compose down` keeps them. Only an explicit destructive command such
|
|
as `docker compose down --volumes` removes them.
|
|
|
|
The frontend depends on the core health check and proxies `/health` and `/api/*` to it. The
|
|
application health endpoint intentionally checks process readiness only; external dependency
|
|
diagnostics are exposed by `tht doctor` and do not prevent the UI from starting.
|
|
|
|
## Git-backed workspace registry
|
|
|
|
Workspace descriptors are shared through a validated Git repository while endpoint bindings and
|
|
secret files remain installation-local. Use the [local Mac/PC installation manual](docs/install/local-workspace-registry.md)
|
|
for Docker Desktop or a local engine, and the [server installation manual](docs/install/server-workspace-registry.md)
|
|
for the Gitea, reverse-proxy, backup, migration, and recovery workflow. The isolated deployment
|
|
exercise is `./scripts/workspace-registry-smoke.sh`; both manuals are checked with
|
|
`./scripts/verify-workspace-install-docs.sh --profile local` or `--profile server`.
|
|
|
|
The operator workflow is: update and review canonical YAML in the shared Git remote, **Pull latest
|
|
registry** from each ThothII installation, run **Validate workspace** and **Test on this
|
|
installation**, then select the workspace locally before creating sessions. Each new session pins
|
|
the Git revision it used; a later pull or publish cannot change a Resume. Snapshot cleanup retains
|
|
every revision referenced by an open, closed, or failed unarchived session. It reconciles from the
|
|
single local installation list or from a server administrator's complete session list, never from
|
|
a remote user's partial list.
|
|
|
|
Connector `ssh_tunnel` bindings are diagnostic-only in this release: their bounded probe always
|
|
cleans up the loopback forward and returns `workspace_not_activatable`; session creation is rejected
|
|
before persistence. Git registry access over SSH is unaffected. Use direct or REST connector
|
|
transport for runtime sessions.
|
|
|
|
`docker-compose.dev.yml` is deliberately local: both published ports bind to `127.0.0.1`,
|
|
`THT_SESSION_STORAGE=local`, and `THT_HOME=/data/local-home`. Do not set
|
|
`THOTH_PUBLIC_EXPOSURE=true` for that profile; the backend rejects that public/local combination
|
|
at startup.
|
|
|
|
Run the end-to-end packaging check with:
|
|
|
|
```sh
|
|
./scripts/docker-smoke.sh
|
|
```
|
|
|
|
The smoke script validates Compose, builds and waits for both services, checks health through
|
|
the frontend, verifies SSE response headers, restarts the core, and confirms `/data` survives.
|
|
Each run uses a unique Compose project and removes that project's containers, network, and test
|
|
volume afterward. It never targets the fixed `thothii` operator project or its volume. Set
|
|
`SMOKE_PROJECT` to a different explicit project name for reproducible debugging, and set
|
|
`KEEP_SMOKE_RESOURCES=1` to retain that smoke project's resources for inspection; remove them
|
|
later with `docker compose --project-name "$SMOKE_PROJECT" down --volumes`.
|
|
|
|
## Optional local pgvector and recovery
|
|
|
|
The local-vector overlay reads `THT_VECTOR_BOOTSTRAP_PASSWORD`,
|
|
`THT_VECTOR_MIGRATOR_PASSWORD`, `THT_VECTOR_READER_PASSWORD`, and
|
|
`THT_VECTOR_WRITER_PASSWORD` from the same bundle. Its `vector_data` volume is independent of
|
|
application state; passwords are selected at runtime and are never passed as URL arguments.
|
|
|
|
## Preprocessing jobs and S3 Evidence
|
|
|
|
The included job workspaces target the optional local-vector profile. Put the four local-vector
|
|
password keys in the bundle, set `THT_OLLAMA_URL`, mount Evidence at `/data/source/evidence`, then
|
|
run the explicit preprocessing preset:
|
|
|
|
```sh
|
|
docker compose --env-file deploy/env/local.env \
|
|
-f compose.yaml -f deploy/compose.local.yaml -f deploy/compose.local-vector.yaml \
|
|
-f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml \
|
|
--profile local-vector --profile preprocess run --rm preprocess-evidence
|
|
```
|
|
|
|
Replace the final service with `preprocess-dwh` when required. The overlay makes each job wait for the vector
|
|
database health check, role reconciliation, and a successful migration; no separate database
|
|
startup or migration command is required.
|
|
|
|
S3 Evidence uses the optional `tht[s3]` dependency and canonical `s3://bucket/key` provenance.
|
|
AWS endpoints are used when no custom URL is supplied. Every custom endpoint is an explicit egress
|
|
trust-boundary opt-in and uses path-style addressing; private and HTTP endpoints require additional
|
|
independent opt-ins. Literal non-global IPv4/IPv6 addresses are classified locally; hostnames are
|
|
not DNS-pinned, so trusted custom-endpoint deployments must enforce their destination with network
|
|
egress policy. Store access key, secret key, and session token as secret references in
|
|
deployment configuration—never in Compose environment values or source URIs. Discovery and reads
|
|
are bounded by configured page, object, and byte limits.
|
|
|
|
Create a versioned PostgreSQL custom-format backup (the filename is operator-controlled, so use
|
|
an immutable timestamp or release identifier):
|
|
|
|
```sh
|
|
./scripts/vector-backup.sh \
|
|
--host 127.0.0.1 --port 5432 --database thoth --user thoth_backup \
|
|
--password-file /secure/thoth/vector-backup-password \
|
|
--output /secure/backups/thoth-vectors-2026-07-12.dump
|
|
```
|
|
|
|
The dump contains the three allowlisted `vectors` tables, their data and ACLs, plus the
|
|
`public.tht_vector_migrations` ledger. Login roles and passwords are deliberately not copied:
|
|
provision/reconcile the approved role names on the target first, and install the `vector`
|
|
extension in its `vectors` schema. The target must otherwise contain no vector tables or ledger.
|
|
|
|
Restore always names both the currently active source and a target on a physically distinct
|
|
PostgreSQL cluster. The script compares PostgreSQL system identity, so host aliases or a different
|
|
database in the active cluster cannot bypass the guard. It refuses a non-empty target unless
|
|
`--force-nonempty` is explicit, and the clean restore is one transaction:
|
|
|
|
```sh
|
|
./scripts/vector-restore.sh \
|
|
--active-host vector-db --active-database thoth --active-user thoth_backup \
|
|
--active-password-file /secure/thoth/vector-active-password \
|
|
--target-host vector-db-restore --target-database thoth --target-user thoth_restore \
|
|
--target-password-file /secure/thoth/vector-restore-password \
|
|
--input /secure/backups/thoth-vectors-2026-07-12.dump
|
|
```
|
|
|
|
After restore, run `tht vector migrate --status --json`, adapter health, and a known retrieval
|
|
query against the target before changing any deployment endpoint. Never test recovery against the
|
|
active `vector_data` volume. `./scripts/local-vector-smoke.sh --backup-restore` performs this drill
|
|
with disposable source and target volumes.
|
|
|
|
## Production trust boundary and secrets
|
|
|
|
ThothII does not implement OIDC. Do not expose its application port directly to a network.
|
|
The production pattern is an authenticated host reverse proxy that:
|
|
|
|
- terminates TLS and authenticates every request;
|
|
- removes any client-supplied identity header;
|
|
- injects one trusted `X-Authenticated-User` value;
|
|
- proxies to the loopback-only ThothII frontend.
|
|
|
|
[`deploy/nginx-authenticated-proxy.conf.example`](deploy/nginx-authenticated-proxy.conf.example)
|
|
shows the contract using nginx `auth_request`; replace the placeholder authentication gateway
|
|
with the organization's reviewed identity proxy. `AUTH_MODE=upstream` trusts this boundary and
|
|
rejects requests without the identity header. Setting `THOTH_PUBLIC_EXPOSURE=true` with any other
|
|
auth mode fails during core startup.
|
|
|
|
Production credentials use the existing Compose secret-bundle contract, never environment values.
|
|
Copy `deploy/secrets/thothii.secrets.example` to a protected host file, include only the required
|
|
keys, and set its absolute path as `THT_SECRETS_FILE` in the operator env. Keep Pi's native
|
|
provider auth in the separate protected file named by `PI_AUTH_FILE`.
|
|
|
|
The bundle is mounted read-only as `/run/secrets/thothii.secrets` and must be mode `0600` or
|
|
`0400` on the host. Docker's runtime `0444` mode is accepted only beneath `/run/secrets`; see
|
|
[`deploy/secrets/README.md`](deploy/secrets/README.md). A PEM CA chain is deliberately not a
|
|
bundle value: PEM contains whitespace and is rejected by the strict parser. Keep the CA chain in
|
|
the host/secret-manager materialization and add a reviewed Compose override that mounts it at
|
|
`/run/secrets/ca-chain.pem` and sets `THT_SSL_CA` when a private CA is required. The base bundle
|
|
does not create that mount. The frontend remains on loopback; the authenticated host proxy is the
|
|
only public listener.
|
|
|
|
Set the selected model provider in application settings (or `PI_PROVIDER`). For each Pi spawn the
|
|
backend validates and reads `THT_MODEL_API_KEY` from the bundle, then exposes its value only as the provider's
|
|
recognized child variable (for example `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`, or
|
|
`ZAI_API_KEY`). Neither the generic file path nor deprecated `PI_PROVIDER_API_KEY` is inherited by
|
|
Pi. Local providers such as Ollama require no model key.
|
|
|
|
`THT_MODEL_API_KEY` supports Pi providers whose authentication is exactly one key:
|
|
`ant-ling`, `anthropic`, `cerebras`, `deepseek`, `fireworks`, `github-copilot`, `google`
|
|
(including the `gemini` alias), `google-vertex` when using its API-key mode, `groq`,
|
|
`huggingface`, `kimi-coding`, `minimax`, `minimax-cn`, `mistral`, `moonshotai`,
|
|
`moonshotai-cn`, `nvidia`, `openai`, `opencode`, `opencode-go`, `openrouter`, `together`,
|
|
`vercel-ai-gateway`, `xai`, the four `xiaomi*` providers, `zai`, and `zai-coding-cn`.
|
|
Compound providers are deliberately unsupported: `amazon-bedrock`, `azure-openai-responses`,
|
|
`cloudflare-workers-ai`, and `cloudflare-ai-gateway` require multiple credential/configuration
|
|
values. Selecting one fails before Pi starts; ambient AWS, Azure, and Cloudflare credentials are
|
|
still scrubbed. Supporting them requires a future dedicated provider-specific configuration.
|
|
|
|
## User-owned session server cutover
|
|
|
|
The server profile stores sessions and per-user preferences directly in PostgreSQL schema
|
|
`thoth_sessions`; it does not use PostgREST, browser storage, a shared session directory, or a
|
|
dual write. Use [`deploy/compose.session-server.yaml.example`](deploy/compose.session-server.yaml.example)
|
|
with the canonical base+server files and set `THT_SERVER_WORKSPACE_CONFIG` to an absolute,
|
|
protected copy of [`deploy/workspaces/server-sessions.yaml.example`](deploy/workspaces/server-sessions.yaml.example).
|
|
|
|
The runtime login needs membership in the no-login database role `thoth_sessions_runtime` only.
|
|
The distinct, one-shot migrator login needs migration authority and uses
|
|
`thoth_sessions_migrator`; it must never be mounted into `core`. Set the non-secret endpoint and
|
|
role fields in the protected deployment environment:
|
|
|
|
```dotenv
|
|
AUTH_MODE=upstream
|
|
THOTH_PUBLIC_EXPOSURE=true
|
|
THT_SESSION_STORAGE=postgres
|
|
THT_SESSION_DB_HOST=sessions-db.internal
|
|
THT_SESSION_DB_PORT=5432
|
|
THT_SESSION_DB_NAME=thoth
|
|
THT_SESSION_RUNTIME_USER=thoth_sessions_app
|
|
THT_SESSION_MIGRATOR_USER=thoth_sessions_migrate
|
|
THT_SESSION_DB_SSLMODE=verify-full
|
|
THT_SESSION_RUNTIME_PASSWORD_SOURCE=/secure/thoth/session-runtime-password
|
|
THT_SESSION_MIGRATOR_PASSWORD_SOURCE=/secure/thoth/session-migrator-password
|
|
THT_SESSION_CA_SOURCE=/secure/thoth/session-ca.pem
|
|
```
|
|
|
|
The overlay mounts the runtime password at `/run/secrets/session_runtime_password`, the CA at
|
|
`/run/secrets/session_ca.pem`, and passes those paths—not their contents—to the server workspace.
|
|
It mounts `session_migrator_password` only to `session-migrate`. The backend refuses a server
|
|
session store without upstream authentication, direct DB host/name/runtime user/password-file,
|
|
`verify-ca` or `verify-full`, and an absolute CA path.
|
|
The migrator independently rejects every other TLS mode before reading its password secret or
|
|
constructing a database URL.
|
|
|
|
Perform the cutover in one maintenance window, with the upstream identity-proxy headers and
|
|
backend principal parser deployed together. Neither change is safe to deploy independently: the
|
|
proxy clears the legacy identity header and the backend rejects it. Drain/stop active Pi work,
|
|
enable a maintenance response at the proxy, then run the migrator once and inspect its pristine JSON:
|
|
|
|
```sh
|
|
docker compose --env-file deploy/env/server.env \
|
|
-f compose.yaml -f deploy/compose.server.yaml -f deploy/compose.session-server.yaml.example \
|
|
--profile session-migrate run --rm session-migrate
|
|
```
|
|
|
|
It must report no pending or drifted migrations before starting the replacement core. `/health`
|
|
is a liveness probe and remains `200`; any request that needs unavailable repository storage
|
|
returns a fixed `503` before a Pi process starts. Verify this with an authenticated request after
|
|
the replacement core is healthy, then remove maintenance mode.
|
|
|
|
Do not import the three legacy server filesystem sessions: they have no trusted owner binding.
|
|
During the same maintenance window, archive the exact three reviewed IDs, verify the generated
|
|
archive and `.sha256`, then rerun the command with `--delete` to remove only those three source
|
|
directories:
|
|
|
|
```sh
|
|
./docker/cutover-legacy-sessions.sh \
|
|
/secure/thoth/legacy-sessions /secure/backups/thoth-legacy-sessions-2026-07-16.tar \
|
|
SESSION_ID_1 SESSION_ID_2 SESSION_ID_3
|
|
# After independent archive review, use a new backup filename:
|
|
./docker/cutover-legacy-sessions.sh --delete \
|
|
/secure/thoth/legacy-sessions /secure/backups/thoth-legacy-sessions-2026-07-16-delete.tar \
|
|
SESSION_ID_1 SESSION_ID_2 SESSION_ID_3
|
|
```
|
|
|
|
The helper refuses to overwrite an existing backup and refuses any count other than three
|
|
distinct IDs. Never run it against a live path without the maintenance gate. Roll back application
|
|
code only by keeping PostgreSQL as the single source of truth and deploying a compatible fixed
|
|
release. Do not restore filesystem persistence, do not re-import the archive, and never dual-write
|
|
sessions to database and files.
|
|
|
|
## Reproducible image verification
|
|
|
|
Base images use exact tags and immutable multi-platform manifest digests. Dependency update and
|
|
residual OS-repository limitations are documented in [`docker/LOCKS.md`](docker/LOCKS.md).
|
|
Run the shared architecture gate with `PLATFORM=linux/amd64` or `PLATFORM=linux/arm64`:
|
|
|
|
```sh
|
|
PLATFORM=linux/arm64 ./scripts/verify-container-images.sh
|
|
```
|
|
|
|
It builds both images, runs common version/runtime/security smokes, and emits an image/package
|
|
inventory beneath `.artifacts/container-images/`. CI runs the same script for both architectures.
|