# Architecture overview

> For details about modules and flows, see [Components, modules, and flows](components.md).

ThothII is a **human-in-the-loop datamart builder**. It turns a natural-language question into validated SQL, and optionally a dbt datamart, through a **deterministic eight-phase NL-to-SQL workflow** in which the model *proposes* and a human reviewer *decides* at gates.

Production authentication uses local authentication or generic OIDC. `tht` is the only operator CLI.
For sessions, roles, groups, diagnostics, and recovery, see the [authentication documentation](authentication.md).

```mermaid
flowchart LR
    USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
    FE --> BE["Backend\nFastify"]
    BE --> CATALOG["Metadata catalog\nPostgreSQL"]
    BE --> MODEL["Configured AI model\nvia short-lived LiteLLM helper"]
    BE -->|bounded read-only samples| DWH
    BE --> PI["Pi\nRPC per sessione"]
    PI --> THT["tht and harness\nworkflow and persistence"]
    THT --> DWH["DWH\nread only"]
    THT --> EVIDENCE["Evidence\ncurated corpus"]
    EVIDENCE --> THT
    THT --> FE
```

## The three independently built layers

```
frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)
```

| Layer | Stack | Ruolo |
|---|---|---|
| **harness/** | Python (`tht` CLI) + Pi gate extension (JS) | Owns the workflow and all session persistence |
| **backend/** | Fastify + TypeScript + Kysely | Session bridge plus the isolated administrative metadata catalog |
| **frontend/** | React 18 + Vite | UI that renders gate widgets and rebuilds the live transcript from the SSE stream |

## The harness owns the workflow

`tht` (Python) is a deterministic CLI. `harness/.pi/extensions/tht-gate.js` is a Pi extension that guides the eight-phase workflow. `harness/workflow.yaml` is the single source of workflow truth, and `harness/.pi/skills/tht-sessione/SKILL.md` contains the orchestration rules the model must follow. The "current phase" is **not stored**. It is computed by folding the decision ledger (`harness/tht/phase.py`), which must be read before reasoning about phase logic.

## Persistence means phase documents, not chat

A session is a directory under `sessions/` (the workspace defines the path): `session_manifest.yaml`, phase artifacts (`question.md`, `schema_linking.json`, `sql_final.sql`, and others), and `review_decisions.jsonl`. The contract says: *"persisted state is the truth; what is not recorded did not happen"*. There is no verbatim transcript store. A resumed Pi process rebuilds context from `tht session show <id>` and the artifacts on disk.

## The backend bridges sessions and owns the metadata catalog

- `ThtRunner` runs `tht` subcommands in a shell.
- `PiProcessManager` runs one Pi child process per session and bridges its RPC stream.
- `SessionBridge` maps Pi RPC events to client events (`ui_request` / `text_delta` / `info`).
- `SseHub` distributes these events to the browser over SSE.
- `CatalogService` joins authoritative YAML workspace identities with installation-local database
  configurations stored in PostgreSQL through Kysely.
- `CatalogTableService` reconciles persisted Catalog Tables with a successful external schema scan;
  `ConcreteCatalogTableIntrospector` isolates direct PostgreSQL, typed REST, and SSH-tunnel access.
- `DescriptionGenerationWorker` admits one installation-wide run and processes its table or column
  targets sequentially.
- `PostgresDescriptionSourceSampler` reads bounded real rows and five representative examples from
  the configured DWH connection; `ProcessModelCompleter` invokes the short-lived Python LiteLLM
  helper with the installation-selected model.

Application settings keep only workspace and thinking preferences in `backend/data/settings.json`;
provider/model defaults come from the generated Installation Model Catalog. Session state remains in
harness phase documents. PostgreSQL stores only the administrative database catalog, bindings, observed tables,
curated and generated descriptions, description-generation runs, and sanitized run events.
Connector secrets remain write-only in the encrypted workspace secret store; model credentials
remain in the protected installation secret bundle. Catalog SSH support is limited to connection
tests, table synchronization, and bounded description-generation sampling; it does not change the
session runtime binding contract.

## Human-in-the-loop gate contract

The model proposes; a human reviewer decides at gates through widgets:

- **`reviewer_select`**: single choice. An option with a `decision` payload confirms and persists directly; an option without a payload only asks.
- **`reviewer_decide`**: multiselect. Each choice is a decision.
- **`reviewer_confirm`**: artifact or phase gate.

The frontend renders these widget descriptors (registry in `src/widgets/`). It rebuilds the live transcript in memory from the SSE stream (`src/store/sessionStore.ts`); it is **not persisted**.

## Curated and immutable Evidence

The workspace repository is the publication boundary. The curator prepares `evidence/source/`,
reviews units in `evidence/curated/`, validates them, and merges them. With `evidence.schema_version: 2`,
the runtime materializes the full `evidence/` tree from the exact Git commit, but the renderer passes
only `curated/**/*.md` from the immutable revision root to preprocessing. Sources, manifests, and
evaluation data remain available for traceability. The runtime never modifies, stages, commits, or
publishes the authoring repository.

Before indexing, the curated corpus from the pinned revision is validated. The shared Qdrant
collection keeps the unnamed dense vector used by Schema and Memory. Evidence preprocessing may
add only the sparse `bm25` vector with `idf`, without deleting, renaming, or recreating the collection.
`workspace preprocess evidence` and the Evidence part of `workspace preprocess run` are the only public
operations that perform this upgrade.

## Recurring points of attention

- `tht -c`/`--config` is a **per-command** option. It must follow the subcommand, never precede it (`ThtRunner.buildArgv` enforces this).
- `--json` output must be plain JSON on stdout. It is a machine-readable contract.
- UI strings are in English. Document *content* stays in the workspace language because it is the actual data; only chrome and labels are in English.
- Each workspace defines its DWH target and working directories. Secrets remain in protected installation files, not in the workspace repository.
- Settings are global (`backend/data/settings.json`: workspace/thinking); provider/model choices are
  canonical catalog references selected per session.
- **Resume**: a resumable session returns to its last incomplete phase. The backend rejects resume with 409 when `finalized` or `archived`; `PiProcessManager.spawnFor` must send `/riprendi-sessione <id>` for resume and `/nuova-domanda` for a new session. The wrong prompt silently turns a resume into a new question.

## Runtime composition

The local stack is started by `./scripts/run-stack.sh` after the installation descriptor and
`deploy/env/local.env` exist. `core` includes Pi. `catalog-db`, Qdrant, and the Ollama embedding
service are internal Compose services; only the DWH and model-provider endpoint remain external.
The launcher runs the explicit `catalog-migrate` one-shot service before application startup.

For the operator sequence, see [Install and first start](../install/first-start.md). For the
two user-facing paths, see [User guide](../guida-utente.md) and
[Database management](../operations/database-management.md).