165 lines
8.3 KiB
Markdown
165 lines
8.3 KiB
Markdown
# ThothII
|
|
|
|
ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/`tht`
|
|
core. The portable deployment runs exactly two application services; data services remain
|
|
external in this profile.
|
|
|
|
## Docker Compose: external services
|
|
|
|
Requirements: Docker Engine with Compose v2 and reachable DWH, vector, and embeddings
|
|
services.
|
|
|
|
1. For local development only, copy `deploy/env.example` to `deploy/.env` and fill in runtime
|
|
credentials. The file is gitignored and is never copied into either image.
|
|
2. Add or edit YAML workspace descriptors under `deploy/workspaces/`. These files are mounted
|
|
read-only. Use relative `roots`; they resolve beneath `/data/workspaces/<workspace-name>`.
|
|
3. Start the external-service profile:
|
|
|
|
```sh
|
|
docker compose -f compose.yaml -f deploy/compose.local.yaml \
|
|
--profile external up --build --wait
|
|
```
|
|
|
|
4. Open <http://127.0.0.1:8080>. The published port is loopback-only. Set `THOTH_HTTP_PORT`
|
|
before starting to use another loopback port.
|
|
|
|
Application state, including settings, sessions, artifacts, and indexes, lives in the named
|
|
`thoth_data` volume mounted at `/data`. `docker compose down` keeps that volume. Only an
|
|
explicit destructive command such as `docker compose down --volumes` removes it.
|
|
|
|
The frontend depends on the core health check and proxies `/health` and `/api/*` to it. The
|
|
application health endpoint intentionally checks process readiness only; external dependency
|
|
diagnostics are exposed by `tht doctor` and do not prevent the UI from starting.
|
|
|
|
Run the end-to-end packaging check with:
|
|
|
|
```sh
|
|
./scripts/docker-smoke.sh
|
|
```
|
|
|
|
The smoke script validates Compose, builds and waits for both services, checks health through
|
|
the frontend, verifies SSE response headers, restarts the core, and confirms `/data` survives.
|
|
Each run uses a unique Compose project and removes that project's containers, network, and test
|
|
volume afterward. It never targets the fixed `thothii` operator project or its volume. Set
|
|
`SMOKE_PROJECT` to a different explicit project name for reproducible debugging, and set
|
|
`KEEP_SMOKE_RESOURCES=1` to retain that smoke project's resources for inspection; remove them
|
|
later with `docker compose --project-name "$SMOKE_PROJECT" --profile external down --volumes`.
|
|
|
|
## Optional local pgvector and recovery
|
|
|
|
Start the persistent local vector profile with `docker compose -f compose.yaml -f
|
|
deploy/compose.local-vector.yaml --profile local-vector up --build --wait`. Its `vector_data`
|
|
volume is independent of application state. Reader, writer,
|
|
migrator, and bootstrap credentials remain separate; password files must be mode `0600` and
|
|
must not be passed as URL arguments.
|
|
|
|
## Preprocessing jobs and S3 Evidence
|
|
|
|
The included job workspaces target the local-vector profile. Point the four
|
|
`THT_VECTOR_*_PASSWORD_SECRET_FILE` variables at owner-only files, set `THT_OLLAMA_URL`, mount
|
|
Evidence at `/data/source/evidence`, then run the explicit overlays (which are inert for normal
|
|
runtime):
|
|
|
|
```sh
|
|
docker compose -f compose.yaml -f deploy/compose.local-vector.yaml \
|
|
-f deploy/compose.preprocess.yaml --profile local-vector --profile preprocess \
|
|
run --rm preprocess-evidence
|
|
docker compose -f compose.yaml -f deploy/compose.local-vector.yaml \
|
|
-f deploy/compose.preprocess.yaml --profile local-vector --profile preprocess \
|
|
run --rm preprocess-dwh
|
|
```
|
|
|
|
S3 Evidence uses the optional `tht[s3]` dependency and canonical `s3://bucket/key` provenance.
|
|
AWS endpoints are used when no custom URL is supplied. Every custom endpoint is an explicit egress
|
|
trust-boundary opt-in and uses path-style addressing; private and HTTP endpoints require additional
|
|
independent opt-ins. Literal non-global IPv4/IPv6 addresses are classified locally; hostnames are
|
|
not DNS-pinned, so trusted custom-endpoint deployments must enforce their destination with network
|
|
egress policy. Store access key, secret key, and session token as secret references in
|
|
deployment configuration—never in Compose environment values or source URIs. Discovery and reads
|
|
are bounded by configured page, object, and byte limits.
|
|
|
|
Create a versioned PostgreSQL custom-format backup (the filename is operator-controlled, so use
|
|
an immutable timestamp or release identifier):
|
|
|
|
```sh
|
|
./scripts/vector-backup.sh \
|
|
--host 127.0.0.1 --port 5432 --database thoth --user thoth_backup \
|
|
--password-file /secure/thoth/vector-backup-password \
|
|
--output /secure/backups/thoth-vectors-2026-07-12.dump
|
|
```
|
|
|
|
The dump contains the three allowlisted `vectors` tables, their data and ACLs, plus the
|
|
`public.tht_vector_migrations` ledger. Login roles and passwords are deliberately not copied:
|
|
provision/reconcile the approved role names on the target first, and install the `vector`
|
|
extension in its `vectors` schema. The target must otherwise contain no vector tables or ledger.
|
|
|
|
Restore always names both the currently active source and a target on a physically distinct
|
|
PostgreSQL cluster. The script compares PostgreSQL system identity, so host aliases or a different
|
|
database in the active cluster cannot bypass the guard. It refuses a non-empty target unless
|
|
`--force-nonempty` is explicit, and the clean restore is one transaction:
|
|
|
|
```sh
|
|
./scripts/vector-restore.sh \
|
|
--active-host vector-db --active-database thoth --active-user thoth_backup \
|
|
--active-password-file /secure/thoth/vector-active-password \
|
|
--target-host vector-db-restore --target-database thoth --target-user thoth_restore \
|
|
--target-password-file /secure/thoth/vector-restore-password \
|
|
--input /secure/backups/thoth-vectors-2026-07-12.dump
|
|
```
|
|
|
|
After restore, run `tht vector migrate --status --json`, adapter health, and a known retrieval
|
|
query against the target before changing any deployment endpoint. Never test recovery against the
|
|
active `vector_data` volume. `./scripts/local-vector-smoke.sh --backup-restore` performs this drill
|
|
with disposable source and target volumes.
|
|
|
|
## Production trust boundary and secrets
|
|
|
|
ThothII does not implement OIDC. Do not expose its application port directly to a network.
|
|
The production pattern is an authenticated host reverse proxy that:
|
|
|
|
- terminates TLS and authenticates every request;
|
|
- removes any client-supplied identity header;
|
|
- injects one trusted `X-Authenticated-User` value;
|
|
- proxies to the loopback-only ThothII frontend.
|
|
|
|
[`deploy/nginx-authenticated-proxy.conf.example`](deploy/nginx-authenticated-proxy.conf.example)
|
|
shows the contract using nginx `auth_request`; replace the placeholder authentication gateway
|
|
with the organization's reviewed identity proxy. `AUTH_MODE=upstream` trusts this boundary and
|
|
rejects requests without the identity header. Setting `THOTH_PUBLIC_EXPOSURE=true` with any other
|
|
auth mode fails during core startup.
|
|
|
|
Production credentials use Compose secrets, not `deploy/.env`. Create four files outside the
|
|
repository, restrict their host permissions, and point these variables to them:
|
|
|
|
```sh
|
|
export THT_DWH_API_KEY_SECRET_FILE=/secure/thoth/dwh-api-key
|
|
export THT_VEC_API_KEY_SECRET_FILE=/secure/thoth/vector-reader-api-key
|
|
export THT_VEC_WRITE_API_KEY_SECRET_FILE=/secure/thoth/vector-writer-api-key
|
|
export THT_CA_SECRET_FILE=/secure/thoth/ca-chain.pem
|
|
export THT_DB_NAME=warehouse
|
|
export THT_DWH_REST_URL=https://dwh.example.test
|
|
export THT_VEC_REST_URL=https://vectors.example.test
|
|
export THT_OLLAMA_URL=https://embeddings.example.test
|
|
docker compose -f compose.yaml -f deploy/compose.production.yaml \
|
|
--profile external up --build --wait
|
|
```
|
|
|
|
The secrets and public CA chain are mounted read-only under `/run/secrets` and must be readable by
|
|
the core's UID 10001. Host secret files must be `0600` or `0400`; Docker's runtime `0444` mount is
|
|
accepted only beneath `/run/secrets`. See [`deploy/secrets/README.md`](deploy/secrets/README.md) for
|
|
the verification command. The frontend remains on loopback; the authenticated host proxy is the
|
|
only public listener.
|
|
|
|
## Reproducible image verification
|
|
|
|
Base images use exact tags and immutable multi-platform manifest digests. Dependency update and
|
|
residual OS-repository limitations are documented in [`docker/LOCKS.md`](docker/LOCKS.md).
|
|
Run the shared architecture gate with `PLATFORM=linux/amd64` or `PLATFORM=linux/arm64`:
|
|
|
|
```sh
|
|
PLATFORM=linux/arm64 ./scripts/verify-container-images.sh
|
|
```
|
|
|
|
It builds both images, runs common version/runtime/security smokes, and emits an image/package
|
|
inventory beneath `.artifacts/container-images/`. CI runs the same script for both architectures.
|