mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

refactor: remove portal deployment coupling

Browse Source
This commit is contained in:
marcopan
2026-08-05 06:58:19 +02:00
parent fd1fd2f802
commit 5d037e97c4
25 changed files with 265 additions and 452 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+37 -56
View File
@@ -4,57 +4,37 @@ ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fasti
core. The portable deployment runs exactly two application services; data services remain
external in this profile.
## Docker Compose: one-command startup
## Docker Compose: local startup
Requirements: Docker Engine with Compose v2. The default project starts only the two
application images; DWH, vector and embedding services can be remote or supplied by an
optional overlay.
Requirements: Docker Engine with Compose v2. The mandatory stack is exactly the `core` and
`frontend` application images. DWH, vector DB, embedding, and LLM services are external,
configurable endpoints—even when they are co-located with ThothII.
From a fresh clone, run these commands from the repository root:
```sh
cp .env.example .env
cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets
chmod 600 deploy/secrets/thothii.secrets
# Edit .env (non-secret endpoints) and deploy/secrets/thothii.secrets (KEY=VALUE lines).
docker compose up --build -d
cp deploy/env/local.env.example deploy/env/local.env
# Edit deploy/env/local.env, including PI_AUTH_FILE and the external endpoint URLs.
docker compose --env-file deploy/env/local.env \
-f compose.yaml -f deploy/compose.local.yaml up --build -d
```
The root `.env` is loaded automatically by Compose. It defaults to `compose.yaml`, an empty
profile, and `THT_SECRETS_FILE=deploy/secrets/thothii.secrets`; no `--env-file`, `-f`, or
`--profile` flag is required for the normal installation. Add or edit YAML workspace descriptors
under `deploy/workspaces/`; they are mounted read-only and relative `roots` resolve beneath
`/data/workspaces/<workspace-name>`. Open <http://127.0.0.1:8080> (set `THOTH_HTTP_PORT` in
`.env` to choose another loopback port).
`./scripts/run-stack.sh` runs this same base+local command in the foreground. The core image
contains its Pi runtime; no host `pi` executable is used. For a server installation, copy and
fill `deploy/env/server.env.example`, then use `-f compose.yaml -f deploy/compose.server.yaml`.
Workspace descriptors come from the Git remote configured by `THT_WORKSPACE_GIT_REMOTE`; their
runtime endpoint and secret bindings remain installation-local. Open
<http://127.0.0.1:8080> (set `THOTH_HTTP_PORT` in `deploy/env/local.env` to choose another
loopback port).
The bundle contains only values, one per line (`THT_MODEL_API_KEY=...`, DWH/vector keys, and
the optional local-vector passwords). It is ignored by Git and never copied into either image.
Do not put credentials in `.env`, workspace YAML, URLs, or Compose interpolation values.
Credentials and certificates are local protected files. Do not put them in environment examples,
workspace YAML, URLs, or Compose interpolation values. The optional `local-vector` and
preprocessing overlays are development presets; they do not change the two-service mandatory
stack or the external-endpoint contract.
### Optional overlays
Overlays are selected in `.env`, so the operational command remains the same. On Unix-like
systems use `:` between files; on Windows use `;`:
```dotenv
# Remote DWH/vector/embedding services with authenticated reverse proxy:
COMPOSE_FILE=compose.yaml:deploy/compose.production.yaml
COMPOSE_PROFILES=
# Local pgvector (Mac/Windows or a standalone application server):
COMPOSE_FILE=compose.yaml:deploy/compose.local-vector.yaml
COMPOSE_PROFILES=local-vector
```
After changing `.env`, apply the selected configuration with `docker compose up --build -d`.
Preprocessing is an explicit opt-in preset: append
`deploy/compose.preprocess.yaml:deploy/compose.preprocess-local-vector.yaml` and set
`COMPOSE_PROFILES=local-vector,preprocess`; then run the job with
`docker compose run --rm preprocess-evidence` or `preprocess-dwh`.
Application state, including settings, sessions, artifacts, and indexes, lives in the named
`thoth_data` volume mounted at `/data`. `docker compose down` keeps that volume. Only an
explicit destructive command such as `docker compose down --volumes` removes it.
Application state is split across the named `settings`, `pi-state`, `workspace-registry`, and
`sessions` volumes. `docker compose down` keeps them. Only an explicit destructive command such
as `docker compose down --volumes` removes them.
The frontend depends on the core health check and proxies `/health` and `/api/*` to it. The
application health endpoint intentionally checks process readiness only; external dependency
@@ -110,17 +90,18 @@ application state; passwords are selected at runtime and are never passed as URL
## Preprocessing jobs and S3 Evidence
The included job workspaces target the local-vector profile. Put the four local-vector password
keys in the bundle, set `THT_OLLAMA_URL`, mount Evidence at `/data/source/evidence`, then select
the preprocessing preset in `.env`:
The included job workspaces target the optional local-vector profile. Put the four local-vector
password keys in the bundle, set `THT_OLLAMA_URL`, mount Evidence at `/data/source/evidence`, then
run the explicit preprocessing preset:
```dotenv
COMPOSE_FILE=compose.yaml:deploy/compose.local-vector.yaml:deploy/compose.preprocess.yaml:deploy/compose.preprocess-local-vector.yaml
COMPOSE_PROFILES=local-vector,preprocess
```sh
docker compose --env-file deploy/env/local.env \
-f compose.yaml -f deploy/compose.local.yaml -f deploy/compose.local-vector.yaml \
-f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml \
--profile local-vector --profile preprocess run --rm preprocess-evidence
```
Run `docker compose run --rm preprocess-evidence` or
`docker compose run --rm preprocess-dwh`. The overlay makes each job wait for the vector
Replace the final service with `preprocess-dwh` when required. The overlay makes each job wait for the vector
database health check, role reconciliation, and a successful migration; no separate database
startup or migration command is required.
@@ -183,8 +164,8 @@ with the organization's reviewed identity proxy. `AUTH_MODE=upstream` trusts thi
rejects requests without the identity header. Setting `THOTH_PUBLIC_EXPOSURE=true` with any other
auth mode fails during core startup.
Production credentials use the one Compose secret bundle, not `.env`. Put the required keys in
`deploy/secrets/thothii.secrets` and select the production overlay in `.env`:
Production credentials use the one Compose secret bundle, not an environment example. Put the
required keys in `deploy/secrets/thothii.secrets` for the selected base+server installation:
```dotenv
THT_MODEL_API_KEY=replace-me
@@ -255,10 +236,10 @@ session store without upstream authentication, direct DB host/name/runtime user/
The migrator independently rejects every other TLS mode before reading its password secret or
constructing a database URL.
Perform the cutover in one maintenance window, with the Task 4 portal proxy headers and Task 5
backend principal parser deployed together. Neither change is safe to deploy independently: Task
4 clears the legacy identity header and Task 5 rejects it. Drain/stop active Pi work, enable a
maintenance response at the portal, then run the migrator once and inspect its pristine JSON:
Perform the cutover in one maintenance window, with the upstream identity-proxy headers and
backend principal parser deployed together. Neither change is safe to deploy independently: the
proxy clears the legacy identity header and the backend rejects it. Drain/stop active Pi work,
enable a maintenance response at the proxy, then run the migrator once and inspect its pristine JSON:
```sh
docker compose -f compose.yaml -f deploy/compose.session-server.yaml \