feat: add AI catalog description generation

This commit is contained in:
Codex
2026-08-29 16:42:56 +02:00
parent b0afba81ca
commit 376dd5a09d
76 changed files with 14860 additions and 102 deletions
+1 -1
View File
@@ -33,7 +33,7 @@ The production role expansion from `backend/src/auth/config.ts` is exact:
| Role | Permissions |
|---|---|
| `user` | `session.use` |
| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, `workspace.manage`, `workspace.secrets.manage`, `pi.manage`, `auth.diagnostics.read` |
| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, `workspace.manage`, `workspace.secrets.manage`, `database.manage`, `pi.manage`, `auth.diagnostics.read` |
`admin` therefore includes the ordinary `session.use` permission. No other role or permission
label is part of the production catalog.
+18 -2
View File
@@ -22,7 +22,9 @@ flowchart LR
THT --> VDB["Qdrant / vector store"]
BE --> CFG["settings.json\nworkspace registry"]
BE --> CAT["catalog-db\nPostgreSQL + Kysely"]
BE -->|catalog Test + Table Sync| DWH
BE -->|catalog Test + Sync + bounded AI sampling| DWH
BE -->|one request per subprocess| LLMHELPER["LiteLLM helper\nPython, short-lived"]
LLMHELPER -->|configured model| PROVIDER["AI provider"]
FE -.->|renders widgets| EXT
```
@@ -31,7 +33,7 @@ Dipendenze principali:
| Module | Depends on | Responsibility |
| --- | --- | --- |
| `frontend/` | Backend REST and SSE APIs | UI, gate widgets, and in-memory transcript |
| `backend/src/` | Pi, `tht`, configuration, workspace registry, catalog PostgreSQL, and read-only DWH connectors | Transport, session lifecycle, catalog CRUD, connection tests, table introspection, and APIs |
| `backend/src/` | Pi, `tht`, configuration, workspace registry, catalog PostgreSQL, read-only DWH connectors, and the internal LiteLLM helper | Transport, session lifecycle, catalog CRUD, connection tests, table introspection, sequential AI description generation, and APIs |
| `harness/.pi/` | Pi and `tht phase` | Workflow orchestration and human-in-the-loop gates |
| `harness/tht/` | Filesystem, DWH, and vector store | Persistence, CLI, Evidence, schema, and preprocessing |
| workspace repository | `source/`, `curated/`, manifest, and artifacts | Versioned Evidence source and session output |
@@ -65,6 +67,20 @@ sequenceDiagram
The backend uses `ThtRunner` for CLI subprocesses, `PiProcessManager` for one Pi process per session, `SessionBridge` to adapt RPC events, and `SseHub` to distribute them to clients.
## Catalog description generation
Catalog description generation is a backend-owned administrative operation, separate from the
Pi session workflow and from the public `tht` CLI. The frontend starts one run for selected catalog
tables or columns. A single installation-wide worker processes targets sequentially, reads at most
the configured bounded sample from the source DWH through its read-only connection, and invokes a
short-lived Python LiteLLM helper once per target. The selected model comes from the installation
descriptor; its API key remains in the protected installation secret bundle.
Each result is written immediately to `Generated Description`. Run state and sanitized activity
events are stored in `catalog-db` and exposed to the drawer through REST and SSE. An administrator
may later copy selected generated descriptions into `Description`. There is no parallel run queue,
automatic retry policy, or second orchestration subsystem.
## Main backend classes
The diagram shows the classes that form the bridge between the browser, Pi, and `tht`. Fastify routes receive requests and delegate to these services.
+12 -3
View File
@@ -12,6 +12,8 @@ flowchart LR
USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
FE --> BE["Backend\nFastify"]
BE --> CATALOG["Metadata catalog\nPostgreSQL"]
BE --> MODEL["Configured AI model\nvia short-lived LiteLLM helper"]
BE -->|bounded read-only samples| DWH
BE --> PI["Pi\nRPC per sessione"]
PI --> THT["tht and harness\nworkflow and persistence"]
THT --> DWH["DWH\nread only"]
@@ -50,12 +52,19 @@ A session is a directory under `sessions/` (the workspace defines the path): `se
configurations stored in PostgreSQL through Kysely.
- `CatalogTableService` reconciles persisted Catalog Tables with a successful external schema scan;
`ConcreteCatalogTableIntrospector` isolates direct PostgreSQL, typed REST, and SSH-tunnel access.
- `DescriptionGenerationWorker` admits one installation-wide run and processes its table or column
targets sequentially.
- `PostgresDescriptionSourceSampler` reads bounded real rows and five representative examples from
the configured DWH connection; `ProcessModelCompleter` invokes the short-lived Python LiteLLM
helper with the installation-selected model.
Application settings remain in `backend/data/settings.json`; session state remains in harness phase
documents. PostgreSQL stores only the administrative database catalog, bindings, observed tables,
and curated descriptions. Connector secrets remain write-only in the encrypted workspace secret
store. Catalog SSH support is limited to connection tests and table synchronization; it does not
change the session runtime binding contract.
curated and generated descriptions, description-generation runs, and sanitized run events.
Connector secrets remain write-only in the encrypted workspace secret store; model credentials
remain in the protected installation secret bundle. Catalog SSH support is limited to connection
tests, table synchronization, and bounded description-generation sampling; it does not change the
session runtime binding contract.
## Human-in-the-loop gate contract