ThothII/docs/architecture/components.md

Components, modules, and flows

This page complements the architecture overview with the module structure and flows through ThothII. The diagrams describe the current code, not a future architecture.

Modules and dependencies

The frontend communicates with the backend through REST and SSE. The backend does not own session persistence: it starts Pi, invokes the tht CLI, and forwards events. It does own the separate installation-local database catalog. The harness contains the workflow, the Python CLI, and adapters for the DWH and vector store.

flowchart LR
    FE["frontend/\nReact + Vite"] -->|REST + SSE| BE["backend/\nFastify + TypeScript"]
    BE -->|RPC stdin/stdout| PI["Pi\n--mode rpc"]
    BE -->|subprocess\nJSON stdout| THT["harness/tht\nCLI Python"]
    PI --> EXT["harness/.pi/extensions/\ntht-gate.js"]
    EXT --> SKILL["harness/.pi/skills/\ntht-sessione"]
    EXT --> THT
    THT --> FS["Sessions and artifacts\nworkspace repository"]
    THT --> DWH["DWH\nread-only"]
    THT --> VDB["Qdrant / vector store"]
    BE --> CFG["settings.json\nworkspace registry"]
    BE --> CAT["catalog-db\nPostgreSQL + Kysely"]
    BE -->|catalog Test + Sync + bounded AI sampling| DWH
    BE -->|one request per subprocess| LLMHELPER["LiteLLM helper\nPython, short-lived"]
    LLMHELPER -->|configured model| PROVIDER["AI provider"]
    FE -.->|renders widgets| EXT

Dipendenze principali:

Module	Depends on	Responsibility
frontend/	Backend REST and SSE APIs	UI, gate widgets, and in-memory transcript
backend/src/	Pi, tht, configuration, workspace registry, catalog PostgreSQL, read-only DWH connectors, and the internal LiteLLM helper	Transport, session lifecycle, catalog CRUD, connection tests, table introspection, sequential AI description generation, and APIs
harness/.pi/	Pi and tht phase	Workflow orchestration and human-in-the-loop gates
harness/tht/	Filesystem, DWH, and vector store	Persistence, CLI, Evidence, schema, and preprocessing
workspace repository	source/, curated/, manifest, and artifacts	Versioned Evidence source and session output

Session sequence

The main path starts with a user question and ends with an SSE event. Reviewer decisions use the same channel and are persisted by the harness.

sequenceDiagram
    actor U as User or reviewer
    participant FE as Frontend
    participant BE as Backend
    participant PI as Pi RPC
    participant THT as CLI tht
    participant WS as Workspace
    participant DWH as DWH

    U->>FE: Send question or gate decision
    FE->>BE: POST session / risposta widget
    BE->>PI: RPC input o prompt di resume
    PI->>THT: phase/session/evidence commands
    THT->>WS: Read and write phase artifacts
    THT->>DWH: Introspection or read-only query
    DWH-->>THT: Schema, results, or diagnostics
    THT-->>PI: JSON and phase state
    PI-->>BE: RPC events and widget descriptor
    BE-->>FE: SSE text_delta, info, ui_request
    FE-->>U: Text, artifact, or review request

The backend uses ThtRunner for CLI subprocesses, PiProcessManager for one Pi process per session, SessionBridge to adapt RPC events, and SseHub to distribute them to clients.

Catalog description generation

Catalog description generation is a backend-owned administrative operation, separate from the Pi session workflow and from the public tht CLI. The frontend starts one run for selected catalog tables or columns. A single installation-wide worker processes targets sequentially, reads at most the configured bounded sample from the source DWH through its read-only connection, and invokes a short-lived Python LiteLLM helper once per target. The selected model comes from the installation descriptor; its API key remains in the protected installation secret bundle.

Each result is written immediately to Generated Description. Run state and sanitized activity events are stored in catalog-db and exposed to the drawer through REST and SSE. An administrator may later copy selected generated descriptions into Description. There is no parallel run queue, automatic retry policy, or second orchestration subsystem.

Main backend classes

The diagram shows the classes that form the bridge between the browser, Pi, and tht. Fastify routes receive requests and delegate to these services.

classDiagram
    class ThtRunner {
      +buildArgv(command, args) string[]
      +run(args) Promise~ThtResult~
      +sessionShow(id) Promise~unknown~
    }
    class PiProcessManager {
      -runtimes Map
      +spawnFor(sessionId, mode) SessionRuntime
      +resume(sessionId, tht) Promise~SessionRuntime~
      +stop(sessionId) Promise~void~
    }
    class SessionBridge {
      +handleRpcEvent(event) ClientEvent
      +handleUiResponse(response) Promise~void~
    }
    class SseHub {
      +subscribe(sessionId) AsyncIterable
      +publish(sessionId, event) void
      +close(sessionId) void
    }
    class SessionRoutes {
      +createSession(request) Response
      +resumeSession(id) Response
      +postInput(id, input) Response
    }
    class WorkspaceRegistry {
      +list() Workspace[]
      +resolve(id) Workspace
    }
    class SettingsStore {
      +get() Settings
      +update(patch) Settings
    }
    class App {
      +buildApp() FastifyInstance
    }

    App --> SessionRoutes
    App --> WorkspaceRegistry
    App --> SettingsStore
    SessionRoutes --> PiProcessManager
    SessionRoutes --> ThtRunner
    SessionRoutes --> SseHub
    PiProcessManager --> SessionBridge
    PiProcessManager --> ThtRunner
    SessionBridge --> SseHub

Python modules in the tht CLI

The CLI consists of Typer commands and domain modules. cli/ turns arguments into operations; evidence/, session/, db/, adapters/, and the other packages contain the application logic.

flowchart TB
    MAIN["tht/cli/__init__.py"] --> CMD["tht/cli/*_cmd.py"]
    CMD --> CONFIG["config.py\nworkspace.py\npaths.py"]
    CMD --> SESSION["session_cmd.py\nsession/"]
    CMD --> EVIDENCE["evidence_cmd.py\nevidence/"]
    CMD --> PRE["preprocess_cmd.py\nevidence/corpus/"]
    CMD --> PHASE["phase_cmd.py\nphase.py\nworkflow.py"]
    CMD --> SQL["sql_cmd.py\ndb/\nrest/"]
    EVIDENCE --> ACQ["evidence/acquisition.py\nadapters/ filesystem/http/s3"]
    EVIDENCE --> CANON["evidence/canonical.py\ncontracts.py\nmodel.py"]
    EVIDENCE --> AUTHOR["evidence/authoring.py"]
    PRE --> PIPE["evidence/corpus/pipeline.py\nchunk.py normalize.py store.py"]
    PRE --> VECTOR["adapters/vector/qdrant.py"]
    PRE --> DWH["jobs/dwh_pipeline.py\nadapters/dwh/"]
    SESSION --> REPO["session/filesystem_repository.py\npostgres_repository.py"]
    PHASE --> LEDGER["decisions.py\nreview_decisions"]

The operator command tht in tools/tht/ is separate from the harness Python CLI. The former handles installation, lifecycle, authentication, and workspaces; the latter runs the workflow and data operations.

Eight-phase workflow and gates

The source of truth is harness/workflow.yaml. The current phase is computed from the decision ledger, not from a manually updated field.

flowchart LR
    F1["F1\nChiarimento"] --> F2["F2\nMemoria"]
    F2 --> F3["F3\nRiscrittura"]
    F3 --> F4["F4\nSchema linking\nreviewer_decide"]
    F4 --> F5["F5\nSintesi"]
    F5 --> F6["F6\nCTE\nauto o skip"]
    F6 --> F7["F7\nSQL finale\nreviewer_confirm"]
    F7 --> F8["F8\nDatamart\nreviewer_decide"]
    F1 -.->|reviewer_confirm| F1
    F3 -.->|reviewer_confirm| F3
    F4 -.->|decisioni su tabelle, colonne, evidence| F4
    F6 -.->|cte_approved o cte_rejected| F6
    F7 -.->|sql_approved o sql_rejected| F7
    F8 -.->|datamart_requested o declined| F8

Fase	Nome	Avanzamento	Artefatti principali
F1	chiarimento	kind:phase	decisioni di chiarimento
F2	memoria	automatico se vuota	decisioni memoria
F3	riscrittura	kind:phase	question.md
F4	schema linking	reviewer_decide	schema_linking.json
F5	sintesi	kind:phase	verifica dello schema linking
F6	CTE	automatico, oppure skip	cte_plan.json, ctes/, cte_tests.json
F7	SQL finale	kind:phase dopo sql_approved	sql_final.sql
F8	datamart	reviewer_decide	decisione su richiesta o rifiuto

Un reviewer_select con decisione incorporata può confermare direttamente. Un reviewer_decide registra le scelte multiple. Un reviewer_confirm conferma un artefatto o la chiusura della fase. Il modello propone; il revisore decide e il ledger registrato è la fonte dello stato.