ThothII/docs/architecture/overview.md

Architecture overview

For details about modules and flows, see Components, modules, and flows.

ThothII is a human-in-the-loop datamart builder. It turns a natural-language question into validated SQL, and optionally a dbt datamart, through a deterministic eight-phase NL-to-SQL workflow in which the model proposes and a human reviewer decides at gates.

Production authentication uses local authentication or generic OIDC. tht is the only operator CLI. For sessions, roles, groups, diagnostics, and recovery, see the authentication documentation.

flowchart LR
    USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
    FE --> BE["Backend\nFastify"]
    BE --> CATALOG["Metadata catalog\nPostgreSQL"]
    BE --> MODEL["Configured AI model\nvia short-lived LiteLLM helper"]
    BE -->|bounded read-only samples| DWH
    BE --> PI["Pi\nRPC per sessione"]
    PI --> THT["tht and harness\nworkflow and persistence"]
    THT --> DWH["DWH\nread only"]
    THT --> EVIDENCE["Evidence\ncurated corpus"]
    EVIDENCE --> THT
    THT --> FE

The three independently built layers

frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)

Layer	Stack	Ruolo
harness/	Python (tht CLI) + Pi gate extension (JS)	Owns the workflow and all session persistence
backend/	Fastify + TypeScript + Kysely	Session bridge plus the isolated administrative metadata catalog
frontend/	React 18 + Vite	UI that renders gate widgets and rebuilds the live transcript from the SSE stream

The harness owns the workflow

tht (Python) is a deterministic CLI. harness/.pi/extensions/tht-gate.js is a Pi extension that guides the eight-phase workflow. harness/workflow.yaml is the single source of workflow truth, and harness/.pi/skills/tht-sessione/SKILL.md contains the orchestration rules the model must follow. The "current phase" is not stored. It is computed by folding the decision ledger (harness/tht/phase.py), which must be read before reasoning about phase logic.

Persistence means phase documents, not chat

A session is a directory under sessions/ (the workspace defines the path): session_manifest.yaml, phase artifacts (question.md, schema_linking.json, sql_final.sql, and others), and review_decisions.jsonl. The contract says: "persisted state is the truth; what is not recorded did not happen". There is no verbatim transcript store. A resumed Pi process rebuilds context from tht session show <id> and the artifacts on disk.

The backend bridges sessions and owns the metadata catalog

• ThtRunner runs tht subcommands in a shell.
• PiProcessManager runs one Pi child process per session and bridges its RPC stream.
• SessionBridge maps Pi RPC events to client events (ui_request / text_delta / info).
• SseHub distributes these events to the browser over SSE.
• CatalogService joins authoritative YAML workspace identities with installation-local database configurations stored in PostgreSQL through Kysely.
• CatalogTableService reconciles persisted Catalog Tables with a successful external schema scan; ConcreteCatalogTableIntrospector isolates direct PostgreSQL, typed REST, and SSH-tunnel access.
• DescriptionGenerationWorker admits one installation-wide run and processes its table or column targets sequentially.
• PostgresDescriptionSourceSampler reads bounded real rows and five representative examples from the configured DWH connection; ProcessModelCompleter invokes the short-lived Python LiteLLM helper with the installation-selected model.

Application settings remain in backend/data/settings.json; session state remains in harness phase documents. PostgreSQL stores only the administrative database catalog, bindings, observed tables, curated and generated descriptions, description-generation runs, and sanitized run events. Connector secrets remain write-only in the encrypted workspace secret store; model credentials remain in the protected installation secret bundle. Catalog SSH support is limited to connection tests, table synchronization, and bounded description-generation sampling; it does not change the session runtime binding contract.

Human-in-the-loop gate contract

The model proposes; a human reviewer decides at gates through widgets:

• reviewer_select: single choice. An option with a decision payload confirms and persists directly; an option without a payload only asks.
• reviewer_decide: multiselect. Each choice is a decision.
• reviewer_confirm: artifact or phase gate.

The frontend renders these widget descriptors (registry in src/widgets/). It rebuilds the live transcript in memory from the SSE stream (src/store/sessionStore.ts); it is not persisted.

Curated and immutable Evidence

The workspace repository is the publication boundary. The curator prepares evidence/source/, reviews units in evidence/curated/, validates them, and merges them. With evidence.schema_version: 2, the runtime materializes the full evidence/ tree from the exact Git commit, but the renderer passes only curated/**/*.md from the immutable revision root to preprocessing. Sources, manifests, and evaluation data remain available for traceability. The runtime never modifies, stages, commits, or publishes the authoring repository.

Before indexing, the curated corpus from the pinned revision is validated. The shared Qdrant collection keeps the unnamed dense vector used by Schema and Memory. Evidence preprocessing may add only the sparse bm25 vector with idf, without deleting, renaming, or recreating the collection. workspace preprocess evidence and the Evidence part of workspace preprocess run are the only public operations that perform this upgrade.

Recurring points of attention

• tht -c/--config is a per-command option. It must follow the subcommand, never precede it (ThtRunner.buildArgv enforces this).
• --json output must be plain JSON on stdout. It is a machine-readable contract.
• UI strings are in English. Document content stays in the workspace language because it is the actual data; only chrome and labels are in English.
• Each workspace defines its DWH target and working directories. Secrets remain in protected installation files, not in the workspace repository.
• Settings are global (backend/data/settings.json: workspace/provider/model/thinking); the new-session form asks only for the question.
• Resume: a resumable session returns to its last incomplete phase. The backend rejects resume with 409 when finalized or archived; PiProcessManager.spawnFor must send /riprendi-sessione <id> for resume and /nuova-domanda for a new session. The wrong prompt silently turns a resume into a new question.

Runtime composition

The local stack is started by ./scripts/run-stack.sh after the installation descriptor and deploy/env/local.env exist. core includes Pi. catalog-db, Qdrant, and the Ollama embedding service are internal Compose services; only the DWH and model-provider endpoint remain external. The launcher runs the explicit catalog-migrate one-shot service before application startup.

For the operator sequence, see Install and first start. For the two user-facing paths, see User guide and Database management.