feat: implement metadata catalog database management
This commit is contained in:
@@ -4,7 +4,10 @@ This page complements the [architecture overview](overview.md) with the module s
|
||||
|
||||
## Modules and dependencies
|
||||
|
||||
The frontend communicates with the backend through REST and SSE. The backend does not own session persistence: it starts Pi, invokes the `tht` CLI, and forwards events. The harness contains the workflow, the Python CLI, and adapters for the DWH and vector store.
|
||||
The frontend communicates with the backend through REST and SSE. The backend does not own session
|
||||
persistence: it starts Pi, invokes the `tht` CLI, and forwards events. It does own the separate
|
||||
installation-local database catalog. The harness contains the workflow, the Python CLI, and
|
||||
adapters for the DWH and vector store.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
@@ -18,6 +21,8 @@ flowchart LR
|
||||
THT --> DWH["DWH\nread-only"]
|
||||
THT --> VDB["Qdrant / vector store"]
|
||||
BE --> CFG["settings.json\nworkspace registry"]
|
||||
BE --> CAT["catalog-db\nPostgreSQL + Kysely"]
|
||||
BE -->|catalog Test + Table Sync| DWH
|
||||
FE -.->|renders widgets| EXT
|
||||
```
|
||||
|
||||
@@ -26,7 +31,7 @@ Dipendenze principali:
|
||||
| Module | Depends on | Responsibility |
|
||||
| --- | --- | --- |
|
||||
| `frontend/` | Backend REST and SSE APIs | UI, gate widgets, and in-memory transcript |
|
||||
| `backend/src/` | Pi, `tht`, configuration, and workspace registry | Transport, session lifecycle, and APIs |
|
||||
| `backend/src/` | Pi, `tht`, configuration, workspace registry, catalog PostgreSQL, and read-only DWH connectors | Transport, session lifecycle, catalog CRUD, connection tests, table introspection, and APIs |
|
||||
| `harness/.pi/` | Pi and `tht phase` | Workflow orchestration and human-in-the-loop gates |
|
||||
| `harness/tht/` | Filesystem, DWH, and vector store | Persistence, CLI, Evidence, schema, and preprocessing |
|
||||
| workspace repository | `source/`, `curated/`, manifest, and artifacts | Versioned Evidence source and session output |
|
||||
|
||||
@@ -11,6 +11,7 @@ For sessions, roles, groups, diagnostics, and recovery, see the [authentication
|
||||
flowchart LR
|
||||
USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
|
||||
FE --> BE["Backend\nFastify"]
|
||||
BE --> CATALOG["Metadata catalog\nPostgreSQL"]
|
||||
BE --> PI["Pi\nRPC per sessione"]
|
||||
PI --> THT["tht and harness\nworkflow and persistence"]
|
||||
THT --> DWH["DWH\nread only"]
|
||||
@@ -27,8 +28,8 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
|
||||
|
||||
| Layer | Stack | Ruolo |
|
||||
|---|---|---|
|
||||
| **harness/** | Python (`tht` CLI) + Pi gate extension (JS) | Owns the workflow and **all** persistence |
|
||||
| **backend/** | Fastify + TypeScript | Thin bridge with no database of its own |
|
||||
| **harness/** | Python (`tht` CLI) + Pi gate extension (JS) | Owns the workflow and all session persistence |
|
||||
| **backend/** | Fastify + TypeScript + Kysely | Session bridge plus the isolated administrative metadata catalog |
|
||||
| **frontend/** | React 18 + Vite | UI that renders gate widgets and rebuilds the live transcript from the SSE stream |
|
||||
|
||||
## The harness owns the workflow
|
||||
@@ -39,14 +40,22 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
|
||||
|
||||
A session is a directory under `sessions/` (the workspace defines the path): `session_manifest.yaml`, phase artifacts (`question.md`, `schema_linking.json`, `sql_final.sql`, and others), and `review_decisions.jsonl`. The contract says: *"persisted state is the truth; what is not recorded did not happen"*. There is no verbatim transcript store. A resumed Pi process rebuilds context from `tht session show <id>` and the artifacts on disk.
|
||||
|
||||
## The backend is a thin bridge with no database
|
||||
## The backend bridges sessions and owns the metadata catalog
|
||||
|
||||
- `ThtRunner` runs `tht` subcommands in a shell.
|
||||
- `PiProcessManager` runs one Pi child process per session and bridges its RPC stream.
|
||||
- `SessionBridge` maps Pi RPC events to client events (`ui_request` / `text_delta` / `info`).
|
||||
- `SseHub` distributes these events to the browser over SSE.
|
||||
- `CatalogService` joins authoritative YAML workspace identities with installation-local database
|
||||
configurations stored in PostgreSQL through Kysely.
|
||||
- `CatalogTableService` reconciles persisted Catalog Tables with a successful external schema scan;
|
||||
`ConcreteCatalogTableIntrospector` isolates direct PostgreSQL, typed REST, and SSH-tunnel access.
|
||||
|
||||
Application settings live in a JSON file (`backend/data/settings.json`), not in a database.
|
||||
Application settings remain in `backend/data/settings.json`; session state remains in harness phase
|
||||
documents. PostgreSQL stores only the administrative database catalog, bindings, observed tables,
|
||||
and curated descriptions. Connector secrets remain write-only in the encrypted workspace secret
|
||||
store. Catalog SSH support is limited to connection tests and table synchronization; it does not
|
||||
change the session runtime binding contract.
|
||||
|
||||
## Human-in-the-loop gate contract
|
||||
|
||||
|
||||
Reference in New Issue
Block a user