ThothII/docs/architecture/overview.md

# Architecture overview

> For details about modules and flows, see [Components, modules, and flows](components.md).

ThothII is a **human-in-the-loop datamart builder**. It turns a natural-language question into validated SQL, and optionally a dbt datamart, through a **deterministic eight-phase NL-to-SQL workflow** in which the model *proposes* and a human reviewer *decides* at gates.

Production authentication uses local authentication, generic OIDC, or the
trusted-proxy upstream path used by Omics. `tht` is the operator CLI for the
installation and local/OIDC configuration. For roles and recovery, see
[authentication](authentication.md). One React build supports full and embedded;
[rendering architecture](application-shell.md) separates presentation from identity.

```mermaid
flowchart LR
    USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
    FE --> BE["Backend\nFastify"]
    BE --> CATALOG["Metadata catalog\nPostgreSQL"]
    BE --> MODEL["Configured AI model\nvia short-lived LiteLLM helper"]
    BE -->|bounded read-only samples| DWH
    BE --> PI["Pi\nRPC per sessione"]
    PI --> THT["tht and harness\nworkflow and persistence"]
    THT --> DWH["DWH\nread only"]
    THT --> EVIDENCE["Evidence\ncurated corpus"]
    EVIDENCE --> THT
    THT --> FE
```

## The three independently built layers

```
frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)
```

| Layer | Stack | Ruolo |
|---|---|---|
| **harness/** | Python (`tht` CLI) + Pi gate extension (JS) | Owns the workflow and all session persistence |
| **backend/** | Fastify + TypeScript + Kysely | Session bridge plus the isolated administrative metadata catalog |
| **frontend/** | React 18 + Vite | UI that renders gate widgets and rebuilds the live transcript from the SSE stream |

## The harness owns the workflow

`tht` (Python) is a deterministic CLI. `harness/.pi/extensions/tht-gate.js` is a Pi extension that guides the eight-phase workflow. `harness/workflow.yaml` is the single source of workflow truth, and `harness/.pi/skills/tht-sessione/SKILL.md` contains the orchestration rules the model must follow. The "current phase" is **not stored**. It is computed by folding the decision ledger (`harness/tht/phase.py`), which must be read before reasoning about phase logic.

## Persistence means phase documents, not chat

A session is a directory under `sessions/` (the workspace defines the path): `session_manifest.yaml`, phase artifacts (`question.md`, `schema_linking.json`, `sql_final.sql`, and others), and `review_decisions.jsonl`. The contract says: *"persisted state is the truth; what is not recorded did not happen"*. There is no verbatim transcript store. A resumed Pi process rebuilds context from `tht session show <id>` and the artifacts on disk.

## The backend bridges sessions and owns the metadata catalog

- `ThtRunner` runs `tht` subcommands in a shell.
- `PiProcessManager` runs one Pi child process per session and bridges its RPC stream.
- `SessionBridge` maps Pi RPC events to client events (`ui_request` / `text_delta` / `info`).
- `SseHub` distributes these events to the browser over SSE.
- `CatalogService` joins authoritative YAML workspace identities and Evidence settings with the
  sole database authority: installation-local PostgreSQL Metadata Catalog records.
- `CatalogTableService` reconciles persisted Catalog Tables with a successful external schema scan;
  `ConcreteCatalogTableIntrospector` isolates direct PostgreSQL, typed REST, and SSH-tunnel access.
- `DescriptionGenerationWorker` admits one installation-wide run and processes its table or column
  targets sequentially.
- `PostgresDescriptionSourceSampler` reads bounded real rows and five representative examples from
  the configured DWH connection; `ProcessModelCompleter` invokes the short-lived Python LiteLLM
  helper with the installation-selected model.

Application settings keep only workspace and thinking preferences in `backend/data/settings.json`;
provider/model defaults come from the generated Installation Model Catalog. Session state remains in
harness phase documents. PostgreSQL stores the administrative database catalog, bindings, observed tables,
curated and generated descriptions, description-generation runs, sanitized run events, and the
authoritative preprocessing state/revisions.
Connector secrets remain write-only in the encrypted workspace secret store; model credentials
remain in the protected installation secret bundle. Catalog SSH support is limited to connection
tests, table synchronization, and bounded description-generation sampling; it does not change the
session runtime binding contract.

## Human-in-the-loop gate contract

The model proposes; a human reviewer decides at gates through widgets:

- **`reviewer_select`**: single choice. An option with a `decision` payload confirms and persists directly; an option without a payload only asks.
- **`reviewer_decide`**: multiselect. Each choice is a decision.
- **`reviewer_confirm`**: artifact or phase gate.

The frontend renders these widget descriptors (registry in `src/widgets/`). It rebuilds the live transcript in memory from the SSE stream (`src/store/sessionStore.ts`); it is **not persisted**.

## Evidence sources, local authority and search projections

The workspace repository publishes revision-pinned workspace identities and Evidence sources.
An initialized editable Evidence archive is maintained locally through external editors and
explicit consolidation; normal preprocessing or source refresh must not overwrite its manual
corrections. File save, active search generation and a later human Git commit/push are separate
outcomes. See [the current Evidence contract](../contracts/curated-evidence-v4.md) and
[source/publication boundaries](../contracts/workspace-evidence-v3.md).

Each workspace has separate Qdrant collections. `reference` contains Schema, relationships and
Evidence and may be replaced or cleared by preprocessing. Memory uses its own dense/BM25
projection, rebuilt from authoritative PostgreSQL cards rather than old vector payloads or
session artifacts. Both collections can contain sparse vectors; their ownership and cleanup
lifecycles remain separate. See [Memory](../gestione-memory.md).

The Administration control can clear the replaceable reference collection, LSH, corpus, and derived
checkpoints. The operation preserves the memory collection and makes preprocessing required before
the core can admit a new session.

## Recurring points of attention

- `tht -c`/`--config` is a **per-command** option. It must follow the subcommand, never precede it (`ThtRunner.buildArgv` enforces this).
- `--json` output must be plain JSON on stdout. It is a machine-readable contract.
- UI strings support English and Italian, with English fallback. Session interaction language
  is pinned at creation; document content remains in the workspace language. See
  [shell and localization](../operations/shell-and-localization.md).
- Each workspace defines identity and optional Evidence only. The PostgreSQL Metadata Catalog defines
  its DWH target and binding; secrets remain in the protected workspace secret store.
- Settings are global (`backend/data/settings.json`: workspace/thinking); provider/model choices are
  canonical catalog references selected per session.
- **Resume**: a resumable session returns to its last incomplete phase. The backend rejects resume with 409 when `finalized` or `archived`; `PiProcessManager.spawnFor` must send `/riprendi-sessione <id>` for resume and `/nuova-domanda` for a new session. The wrong prompt silently turns a resume into a new question.

## Runtime composition

PostgreSQL is the current database-metadata authority for all core consumers, not a deferred
catalog-to-core integration. The historical research on a separate catalog service and
`annotations.yaml` publication is superseded by ADRs 0004 and 0016. Preserve read-only DWH access,
installation-local bindings/secrets, revision-pinned workspace identity, and fail-closed readiness
when changing those boundaries. The exact snapshot interface is the
[Catalog Schema Snapshot contract](../contracts/catalog-schema-snapshot.md).

The local stack is started by `./scripts/run-stack.sh` after the installation descriptor and
`deploy/env/local.env` exist. `core` includes Pi. `catalog-db`, Qdrant, and the Ollama embedding
service are internal Compose services; only the DWH and model-provider endpoint remain external.
The launcher runs the explicit `catalog-migrate` one-shot service before application startup.

For the operator sequence, see [Install and first start](../install/first-start.md). For the
two user-facing paths, see [User guide](../guida-utente.md) and
[Database management](../operations/database-management.md).