Architecture overview

For details about modules and flows, see Components, modules, and flows.

ThothII is a human-in-the-loop datamart builder. It turns a natural-language question into validated SQL, and optionally a dbt datamart, through a deterministic eight-phase NL-to-SQL workflow in which the model proposes and a human reviewer decides at gates.

Production authentication uses local authentication or generic OIDC. tht is the only operator CLI. For sessions, roles, groups, diagnostics, and recovery, see the authentication documentation.

flowchart LR
    USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
    FE --> BE["Backend\nFastify"]
    BE --> PI["Pi\nRPC per sessione"]
    PI --> THT["tht and harness\nworkflow and persistence"]
    THT --> DWH["DWH\nread only"]
    THT --> EVIDENCE["Evidence\ncurated corpus"]
    EVIDENCE --> THT
    THT --> FE

The three independent projects

frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)

Layer	Stack	Ruolo
harness/	Python (`tht` CLI) + Pi gate extension (JS)	Owns the workflow and all persistence
backend/	Fastify + TypeScript	Thin bridge with no database of its own
frontend/	React 18 + Vite	UI that renders gate widgets and rebuilds the live transcript from the SSE stream

The harness owns the workflow

tht (Python) is a deterministic CLI. harness/.pi/extensions/tht-gate.js is a Pi extension that guides the eight-phase workflow. harness/workflow.yaml is the single source of workflow truth, and harness/.pi/skills/tht-sessione/SKILL.md contains the orchestration rules the model must follow. The "current phase" is not stored. It is computed by folding the decision ledger (harness/tht/phase.py), which must be read before reasoning about phase logic.

Persistence means phase documents, not chat

A session is a directory under sessions/ (the workspace defines the path): session_manifest.yaml, phase artifacts (question.md, schema_linking.json, sql_final.sql, and others), and review_decisions.jsonl. The contract says: "persisted state is the truth; what is not recorded did not happen". There is no verbatim transcript store. A resumed Pi process rebuilds context from tht session show <id> and the artifacts on disk.

The backend is a thin bridge with no database

ThtRunner runs tht subcommands in a shell.
PiProcessManager runs one Pi child process per session and bridges its RPC stream.
SessionBridge maps Pi RPC events to client events (ui_request / text_delta / info).
SseHub distributes these events to the browser over SSE.

Application settings live in a JSON file (backend/data/settings.json), not in a database.

Human-in-the-loop gate contract

The model proposes; a human reviewer decides at gates through widgets:

reviewer_select: single choice. An option with a decision payload confirms and persists directly; an option without a payload only asks.
reviewer_decide: multiselect. Each choice is a decision.
reviewer_confirm: artifact or phase gate.

The frontend renders these widget descriptors (registry in src/widgets/). It rebuilds the live transcript in memory from the SSE stream (src/store/sessionStore.ts); it is not persisted.

Curated and immutable Evidence

The workspace repository is the publication boundary. The curator prepares evidence/source/, reviews units in evidence/curated/, validates them, and merges them. With evidence.schema_version: 2, the runtime materializes the full evidence/ tree from the exact Git commit, but the renderer passes only curated/**/*.md from the immutable revision root to preprocessing. Sources, manifests, and evaluation data remain available for traceability. The runtime never modifies, stages, commits, or publishes the authoring repository.

Before indexing, the curated corpus from the pinned revision is validated. The shared Qdrant collection keeps the unnamed dense vector used by Schema and Memory. Evidence preprocessing may add only the sparse bm25 vector with idf, without deleting, renaming, or recreating the collection. workspace preprocess evidence and the Evidence part of workspace preprocess run are the only public operations that perform this upgrade.

Recurring points of attention

tht -c/--config is a per-command option. It must follow the subcommand, never precede it (ThtRunner.buildArgv enforces this).
--json output must be plain JSON on stdout. It is a machine-readable contract.
UI strings are in English. Document content stays in the workspace language because it is the actual data; only chrome and labels are in English.
Each workspace defines its DWH target and working directories. Secrets remain in protected installation files, not in the workspace repository.
Settings are global (backend/data/settings.json: workspace/provider/model/thinking); the new-session form asks only for the question.
Resume: a resumable session returns to its last incomplete phase. The backend rejects resume with 409 when finalized or archived; PiProcessManager.spawnFor must send /riprendi-sessione <id> for resume and /nuova-domanda for a new session. The wrong prompt silently turns a resume into a new question.

Starting the stack

Start the local stack with ./scripts/run-stack.sh after creating deploy/env/local.env from deploy/env/local.env.example. The Compose core includes Pi; DWH, the vector database, embeddings, and the LLM are external endpoints configured in the local file.

5.6 KiB Raw Blame History