ThothII/docs/architecture/overview.md

Architecture overview

For details about modules and flows, see Components, modules, and flows.

ThothII is a human-in-the-loop datamart builder. It turns a natural-language question into validated SQL, and optionally a dbt datamart, through a deterministic eight-phase NL-to-SQL workflow in which the model proposes and a human reviewer decides at gates.

Production authentication uses local authentication, generic OIDC, or the trusted-proxy upstream path used by Omics. tht is the operator CLI for the installation and local/OIDC configuration. For roles and recovery, see authentication. One React build supports full and embedded; rendering architecture separates presentation from identity.

flowchart LR
    USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
    FE --> BE["Backend\nFastify"]
    BE --> CATALOG["Metadata catalog\nPostgreSQL"]
    BE --> MODEL["Configured AI model\nvia short-lived LiteLLM helper"]
    BE -->|bounded read-only samples| DWH
    BE --> PI["Pi\nRPC per sessione"]
    PI --> THT["tht and harness\nworkflow and persistence"]
    THT --> DWH["DWH\nread only"]
    THT --> EVIDENCE["Evidence\ncurated corpus"]
    EVIDENCE --> THT
    THT --> FE

The three independently built layers

frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)

Layer	Stack	Ruolo
harness/	Python (tht CLI) + Pi gate extension (JS)	Owns the workflow and all session persistence
backend/	Fastify + TypeScript + Kysely	Session bridge plus the isolated administrative metadata catalog
frontend/	React 18 + Vite	UI that renders gate widgets and rebuilds the live transcript from the SSE stream

The harness owns the workflow

tht (Python) is a deterministic CLI. harness/.pi/extensions/tht-gate.js is a Pi extension that guides the eight-phase workflow. harness/workflow.yaml is the single source of workflow truth, and harness/.pi/skills/tht-sessione/SKILL.md contains the orchestration rules the model must follow. The "current phase" is not stored. It is computed by folding the decision ledger (harness/tht/phase.py), which must be read before reasoning about phase logic.

Persistence means phase documents, not chat

A session is a directory under sessions/ (the workspace defines the path): session_manifest.yaml, phase artifacts (question.md, schema_linking.json, sql_final.sql, and others), and review_decisions.jsonl. The contract says: "persisted state is the truth; what is not recorded did not happen". There is no verbatim transcript store. A resumed Pi process rebuilds context from tht session show <id> and the artifacts on disk.

The backend bridges sessions and owns the metadata catalog

• ThtRunner runs tht subcommands in a shell.
• PiProcessManager runs one Pi child process per session and bridges its RPC stream.
• SessionBridge maps Pi RPC events to client events (ui_request / text_delta / info).
• SseHub distributes these events to the browser over SSE.
• CatalogService joins authoritative YAML workspace identities and Evidence settings with the sole database authority: installation-local PostgreSQL Metadata Catalog records.
• CatalogTableService reconciles persisted Catalog Tables with a successful external schema scan; ConcreteCatalogTableIntrospector isolates direct PostgreSQL, typed REST, and SSH-tunnel access.
• DescriptionGenerationWorker admits one installation-wide run and processes its table or column targets sequentially.
• PostgresDescriptionSourceSampler reads bounded real rows and five representative examples from the configured DWH connection; ProcessModelCompleter invokes the short-lived Python LiteLLM helper with the installation-selected model.

Application settings keep only workspace and thinking preferences in backend/data/settings.json; provider/model defaults come from the generated Installation Model Catalog. Session state remains in harness phase documents. PostgreSQL stores the administrative database catalog, bindings, observed tables, curated and generated descriptions, description-generation runs, sanitized run events, and the authoritative preprocessing state/revisions. Connector secrets remain write-only in the encrypted workspace secret store; model credentials remain in the protected installation secret bundle. Catalog SSH support is limited to connection tests, table synchronization, and bounded description-generation sampling; it does not change the session runtime binding contract.

Human-in-the-loop gate contract

The model proposes; a human reviewer decides at gates through widgets:

• reviewer_select: single choice. An option with a decision payload confirms and persists directly; an option without a payload only asks.
• reviewer_decide: multiselect. Each choice is a decision.
• reviewer_confirm: artifact or phase gate.

The frontend renders these widget descriptors (registry in src/widgets/). It rebuilds the live transcript in memory from the SSE stream (src/store/sessionStore.ts); it is not persisted.

Evidence sources, local authority and search projections

The workspace repository publishes revision-pinned workspace identities and Evidence sources. An initialized editable Evidence archive is maintained locally through external editors and explicit consolidation; normal preprocessing or source refresh must not overwrite its manual corrections. File save, active search generation and a later human Git commit/push are separate outcomes. See the current Evidence contract and source/publication boundaries.

Each workspace has separate Qdrant collections. reference contains Schema, relationships and Evidence and may be replaced or cleared by preprocessing. Memory uses its own dense/BM25 projection, rebuilt from authoritative PostgreSQL cards rather than old vector payloads or session artifacts. Both collections can contain sparse vectors; their ownership and cleanup lifecycles remain separate. See Memory.

The Administration control can clear the replaceable reference collection, LSH, corpus, and derived checkpoints. The operation preserves the memory collection and makes preprocessing required before the core can admit a new session.

Recurring points of attention

• tht -c/--config is a per-command option. It must follow the subcommand, never precede it (ThtRunner.buildArgv enforces this).
• --json output must be plain JSON on stdout. It is a machine-readable contract.
• UI strings support English and Italian, with English fallback. Session interaction language is pinned at creation; document content remains in the workspace language. See shell and localization.
• Each workspace defines identity and optional Evidence only. The PostgreSQL Metadata Catalog defines its DWH target and binding; secrets remain in the protected workspace secret store.
• Settings are global (backend/data/settings.json: workspace/thinking); provider/model choices are canonical catalog references selected per session.
• Resume: a resumable session returns to its last incomplete phase. The backend rejects resume with 409 when finalized or archived; PiProcessManager.spawnFor must send /riprendi-sessione <id> for resume and /nuova-domanda for a new session. The wrong prompt silently turns a resume into a new question.

Runtime composition

PostgreSQL is the current database-metadata authority for all core consumers, not a deferred catalog-to-core integration. The historical research on a separate catalog service and annotations.yaml publication is superseded by ADRs 0004 and 0016. Preserve read-only DWH access, installation-local bindings/secrets, revision-pinned workspace identity, and fail-closed readiness when changing those boundaries. The exact snapshot interface is the Catalog Schema Snapshot contract.

The local stack is started by ./scripts/run-stack.sh after the installation descriptor and deploy/env/local.env exist. core includes Pi. catalog-db, Qdrant, and the Ollama embedding service are internal Compose services; only the DWH and model-provider endpoint remain external. The launcher runs the explicit catalog-migrate one-shot service before application startup.

For the operator sequence, see Install and first start. For the two user-facing paths, see User guide and Database management.