mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: reorganize operational documentation

Browse Source
This commit is contained in:
Codex
2026-08-29 20:08:48 +02:00
parent d504b1def1
commit 0ce05869cf
10 changed files with 371 additions and 337 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+4 -5
View File
@@ -74,11 +74,10 @@ diagnostics are exposed by `tht doctor` and do not prevent the UI from starting.
## Git-backed workspace repository ## Git-backed workspace repository
Workspace descriptors are shared through a validated Git repository while endpoint bindings and Workspace descriptors are shared through a validated Git repository while endpoint bindings and
secret files remain installation-local. Use the [local Mac/PC installation manual](docs/install/local-workspace-registry.md) secret files remain installation-local. The supported operating sequence is documented in
for Docker Desktop or a local engine, the [server installation manual](docs/install/server-workspace-registry.md) [Workspace operations](docs/operations/workspaces.md); it covers curator publication, installation
for the Gitea, reverse-proxy, backup, upgrade, and recovery workflow, and the activation, runtime bindings, and preprocessing. The host setup and lifecycle path is in
[P1→P1.1 migration guide](docs/migrations/p1-to-p1-1-registry-layout.md) before upgrading an [Install and first start](docs/install/first-start.md).
older flat-layout registry.
The curator-owned repository layout is: The curator-owned repository layout is:
docs/architecture/components.md
+4 -2
View File
@@ -78,8 +78,10 @@ descriptor; its API key remains in the protected installation secret bundle.
Each result is written immediately to `Generated Description`. Run state and sanitized activity Each result is written immediately to `Generated Description`. Run state and sanitized activity
events are stored in `catalog-db` and exposed to the drawer through REST and SSE. An administrator events are stored in `catalog-db` and exposed to the drawer through REST and SSE. An administrator
may later copy selected generated descriptions into `Description`. There is no parallel run queue, may later copy selected generated descriptions into `Description`. There is no parallel run queue
automatic retry policy, or second orchestration subsystem. or second orchestration subsystem. A target receives at most one provider retry; three consecutive
exhausted technical batches fail the run. Stale work is marked interrupted at startup and must be
explicitly unlocked; it never resumes automatically.
## Main backend classes ## Main backend classes
docs/architecture/overview.md
+10 -5
View File
@@ -22,7 +22,7 @@ flowchart LR
THT --> FE THT --> FE
``` ```
## The three independent projects ## The three independently built layers
``` ```
frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only) frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)
@@ -100,8 +100,13 @@ operations that perform this upgrade.
- Settings are global (`backend/data/settings.json`: workspace/provider/model/thinking); the new-session form asks only for the question. - Settings are global (`backend/data/settings.json`: workspace/provider/model/thinking); the new-session form asks only for the question.
- **Resume**: a resumable session returns to its last incomplete phase. The backend rejects resume with 409 when `finalized` or `archived`; `PiProcessManager.spawnFor` must send `/riprendi-sessione <id>` for resume and `/nuova-domanda` for a new session. The wrong prompt silently turns a resume into a new question. - **Resume**: a resumable session returns to its last incomplete phase. The backend rejects resume with 409 when `finalized` or `archived`; `PiProcessManager.spawnFor` must send `/riprendi-sessione <id>` for resume and `/nuova-domanda` for a new session. The wrong prompt silently turns a resume into a new question.
## Starting the stack ## Runtime composition
Start the local stack with `./scripts/run-stack.sh` after creating The local stack is started by `./scripts/run-stack.sh` after the installation descriptor and
`deploy/env/local.env` from `deploy/env/local.env.example`. The Compose core includes Pi; DWH, `deploy/env/local.env` exist. `core` includes Pi. `catalog-db`, Qdrant, and the Ollama embedding
the vector database, embeddings, and the LLM are external endpoints configured in the local file. service are internal Compose services; only the DWH and model-provider endpoint remain external.
The launcher runs the explicit `catalog-migrate` one-shot service before application startup.
For the operator sequence, see [Install and first start](../install/first-start.md). For the
two user-facing paths, see [User guide](../guida-utente.md) and
[Database management](../operations/database-management.md).
docs/guida-utente.md
+57 -282
View File
@@ -1,282 +1,57 @@
# ThothII user guide # User guide: from question to validated SQL
For login, **Remember me**, roles, session invalidation, OIDC groups, and recovery, see the This guide is for a reviewer using a configured ThothII installation. Installation, workspace
[local authentication guide](install/authentication-local.md) and the [generic OIDC guide](install/authentication-oidc.md). publication, preprocessing, and database administration are separate paths; links to them are at
the end of this page.
This guide walks you through **preparing** the workspace repository, **using ThothII's tools** for
that repository, and **using the application** to ask natural-language questions and obtain ## Before creating a session
validated SQL. It uses plain language and examples. The contracts listed at the end contain the
technical details. An administrator must have selected a workspace and configured the installation-wide provider,
model, and thinking settings. The New session form deliberately asks only for the question.
> **What is ThothII?** It is a *datamart builder* with human review. You write a natural-language
> question, the model proposes each step in turn (clarifications, schema, CTEs, and SQL), and a The workspace is a pinned Git revision. A subsequent workspace update cannot alter a session
> **human reviewer decides** at every important step. The final result is validated SQL ready to already created from an earlier revision. If a workspace cannot reach its configured runtime DWH,
> run on the data warehouse. new sessions are refused before any session state is written.
--- ## Create and review a session
## Part 1: prepare the workspace repository on Git 1. Sign in and select **New session**.
2. Enter a precise business question, including the relevant time period and desired output. For
### 1.1 Structure example: “List patients discharged in the last 30 days, with ward and discharge date.”
3. Review each gate and make the decision requested by the widget. A choice with a decision payload
The workspace repository is a **Git repository** that describes *which data* is available and can persist immediately; a multi-choice widget records each selected decision; a confirmation
*how to reach it*. It contains neither the data nor **secrets** such as passwords, tokens, or certificates. widget approves an artifact or phase.
4. Inspect the generated artifacts, especially schema linking, CTEs, and final SQL. The final SQL is
A valid repository contains: available only after the F7 review gate.
5. At F8, decide whether a datamart is requested. This is distinct from approving the SQL.
```text
thoth-workspaces.yaml # catalog: list of workspaces The workflow phases are fixed:
<workspace-id>/workspace.yaml # workspace descriptor (schema v3)
<workspace-id>/evidence/ # optional context documents, such as *.md | Phase | What is reviewed |
<workspace-id>/schema/annotations.yaml # optional manually curated logical joins (P5) | --- | --- |
``` | F1–F3 | clarification, reusable Memory, and the rewritten question |
| F4–F5 | proposed tables, columns, Evidence, then their summary |
- The **catalog** `thoth-workspaces.yaml` is a simple list: | F6 | a CTE plan or an explicit skip |
| F7 | `sql_final.sql` |
```yaml | F8 | the datamart request or refusal |
schema_version: 1
workspaces: ## Resume, archive, and the meaning of saved state
- id: acme-ebikes
name: ACME Limited The sidebar lists sessions and their current lifecycle. Resuming returns to the last incomplete
description: DWH for electric bicycle production phase. A finalized or archived session cannot be resumed.
```
The source of truth is the workspace session directory: `session_manifest.yaml`, phase artifacts,
- The **ID** must be lowercase, contain no spaces, and follow `acme-ebikes` (`[a-z][a-z0-9-]{2,62}`). and `review_decisions.jsonl`. The visible activity stream is rebuilt from live SSE events and is
- The **descriptor** `<id>/workspace.yaml` uses schema v3. It is the only valid description. not a transcript store. Therefore a decision or artifact that has not been persisted did not
happen from the workflow’s point of view.
### 1.2 Descriptor example (ACME Limited)
## Choose the correct neighbouring path
```yaml
workspace: - To prepare, update, validate, or preprocess a workspace, use
schema_version: 3 [Workspace operations](operations/workspaces.md).
id: acme-ebikes - To create a database configuration, refresh its physical schema, or generate catalog
name: ACME Limited descriptions, use [Database management](operations/database-management.md).
description: Industrial DWH for electric bicycle production - To author material the workflow can retrieve, use [Evidence](evidence.md). A proposal from a
language: en # descriptions and Evidence are in English session does not become Evidence automatically: a curator must review and publish it in Git.
- For login and access recovery, use [local authentication](install/authentication-local.md) or
dwh: [OIDC authentication](install/authentication-oidc.md).
engine: postgres
database: postgres
schema: datawarehouse
supported_transports: [rest_api] # access through the REST API (PostgREST)
semantic_index:
vector_store:
engine: qdrant
collection: acme-ebikes
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
diagnostics:
dwh_rest:
method: POST
path: /rpc/ping
auth: x-api-key
response: { database: database, schema: schema }
evidence:
source:
type: filesystem
uri: acme-ebikes/evidence # path inside the repository
policy:
max_chunk_chars: 4000
retain_published_generations: 3
```
Changes from older workspaces:
- NL→SQL sessions reach the database only through **REST** or **direct Postgres** (`rest_api` /
`postgres_direct`); `ssh_tunnel` remains disabled for session runtime. The separate Database
management surface supports SSH for **Test connection** and **Sync tables**, with a private key,
mandatory `known_hosts`, and an optional key passphrase;
- the semantic index is **internal** (Qdrant plus `qwen3-embedding:0.6b`, 1024 dimensions, cosine);
- **filesystem** Evidence lives in the repository (`<id>/evidence`) and is materialized from the
pinned Git commit (P6). HTTP Evidence is also supported.
### 1.3 Rules
1. **Git is the source of truth.** Change the descriptor, catalog, and Evidence only through a
*commit* and *push*, followed by an installation *pull*.
2. **No secrets in the repository.** Add passwords, tokens, private keys, and signed URLs at
runtime through Workspace management; the backend stores them encrypted.
3. **Schema v3 only.** Reject v1 and v2 descriptors before activation.
4. **The application does not push curated content.** The repository curator works in a separate
authoring clone.
---
## Part 2: use ThothII's repository tools
There are **two** tools: the **web application** (workspace management) and the **`tht` CLI**
(preprocessing and operations). The complete installation is described in
`docs/install/local-workspace-registry.md` (macOS/Windows/Linux) e
`docs/install/server-workspace-registry.md`.
### 2.1 `tht`: main commands
Always invoke `tht` with `--installation <path>/thothii-installation.yaml`. The usual commands are:
```bash
# 1) inspect workspace state (revision and identity)
tht --installation <install> workspace inspect --workspace <id> --json
# 2) inspect the DWH (generates physical.yaml and LSH)
tht --installation <install> workspace preprocess dwh --workspace <id> --json
# 3) suggest joins (FKs) from approved SQL
tht --installation <install> workspace schema suggest-fks --workspace <id> --from-sql <query>.sql --output <candidati>.yaml --json
# 4) after review, publish curated FKs in Git and accept them
tht --installation <install> workspace schema accept --workspace <id> --run <run-id> --yes --json
# 5) index the schema (Qdrant)
tht --installation <install> workspace index-schema --workspace <id> --json
# 6) preprocess Evidence
tht --installation <install> workspace preprocess evidence --workspace <id> --json
# 7) complete chain (DWH → FK → schema → Evidence)
tht --installation <install> workspace preprocess run --workspace <id> --json
# 8) inspect or rebuild the Qdrant collection (maintenance only)
tht --installation <install> workspace vector inspect --workspace <id> --json
tht --installation <install> workspace vector rebuild --workspace <id> --collection <nome> --confirm <nome> --destroy
```
Important notes:
- **`--json` writes JSON only to stdout** (machine contract); use it in scripts.
- **`preprocess run` stops for human review** when it finds new proposed joins: it exits with
`manual_review_required`. After review, continue with `schema accept ... --yes` and
`preprocess run --resume <run-id>`.
- **A filesystem Evidence file is materialized from the pinned Git commit** (there is no moving
checkout); symlinks, unsafe paths, and trees that are too large are rejected.
- **The CLI never writes to the repository** (it never pushes curated content).
### 2.2 Web application: workspace management
Workspace management has two distinct levels.
**Level 1: repository.** The first section shows that the workspace source lives in a separate
directory, is published by the curator to a repository hosted on a Git server such as GitHub,
GitLab, or Gitea, and is read by ThothII in read-only mode. It shows the host, repository, branch,
active revision, and status of the last update.
* **Update workspace repository** does not require a workspace to be selected. The backend fetches
or pulls the configured branch into ThothII's managed checkout, validates the entire candidate
revision, and activates it atomically. If validation fails, it keeps the previous revision. It
does not modify the remote source or save content from the GUI.
* To create a local workspace, prepare a source directory with the catalog, `workspace.yaml`, and
the expected subdirectories. Validate it, then commit and push from the authoring clone.
ThothII provides no commands to create, edit, or publish the source.
**Level 2: selected workspace.** These commands are separate because they require a workspace to be selected first.
These commands are separate because they require a workspace to be selected first.
- **Validate workspace source** checks the catalog, descriptor, Evidence, and invariants of the
selected active revision again. It does not contact the DWH or modify files.
- **Save entered secrets** replaces the entered values without displaying them. The fields depend
on the declared DWH transport and Evidence authentication. The backend returns only configured
or missing status.
* **Forget stored value** removes the selected secret from the encrypted vault. Future sessions or
operations that need it remain blocked until it is entered again.
The remote Git repository and its credentials are installation settings. Runtime DWH and Evidence
secrets persist in the backend's encrypted vault, not in the GUI's local storage. The GUI is only
the interface: after submission it clears the field values and cannot read them back.
---
## Part 3: use the ThothII application
### 3.1 New session
Open the application and use **New session**. Enter only the **natural-language question**;
workspace, model, and provider are already configured as global settings.
Example question:
> "List the electric bicycles completed in the last year, with model, frame number, and completion
> date."
### 3.2 The eight-phase workflow and gates
The question passes through **eight phases**. You see the intermediate documents and decide at the important points:
1. **F1 clarification**: the model removes ambiguity when needed;
2. **F2 Memory**: retrieves reusable Memory;
3. **F3 rewriting**: rewrites and approves the question;
4. **F4 schema linking**: proposes related tables and columns;
5. **F5 summary**: summarizes the selected schema;
6. **F6 CTE**: builds the CTEs;
7. **F7 final SQL**: produces `sql_final.sql`;
8. **F8 datamart**: execution or export (dbt, CSV, Excel).
**Review gates** appear as widgets: choose one option, select several items, or confirm an
artifact or phase. The model *proposes* and the reviewer *decides*. The right side shows artifacts
(schema linking, CTEs, and SQL); the Model activity panel shows the question and reasoning.
### 3.3 Sessions
Sessions appear in the sidebar with their ID, question, date, and author. A session
**resumes** from its last incomplete phase by rebuilding state from documents saved on disk
(`session_manifest.yaml`, phase artifacts, and `review_decisions.jsonl`). Saved state **is** the
truth: what is not recorded did not happen.
---
## Complete example: ACME Limited
### Step 0: repository
Create the workspace Git repository, for example `tht-workspace-acme`:
```text
thoth-workspaces.yaml # catalog containing acme-ebikes
acme-ebikes/workspace.yaml # v3 descriptor (see §1.2)
acme-ebikes/evidence/ # curated context .md documents
acme-ebikes/schema/annotations.yaml # when curated joins exist
```
Publish a new Git revision. In the installation, the application fetches and **activates** the
workspace, validates schema v3, materializes Evidence from the pinned revision, and prepares the
Qdrant collection (1024/cosine plus indexes).
### Step 1: preprocessing
```bash
tht --installation ~/thothii-installation.yaml workspace preprocess dwh --workspace acme-ebikes --json
tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --json
```
If the run stops for joins (`manual_review_required`):
```bash
# the curator reviews the candidates and publishes acme-ebikes/schema/annotations.yaml, then:
tht --installation ~/thothii-installation.yaml workspace schema accept --workspace acme-ebikes --run RUN_ID --yes --json
tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --resume RUN_ID --json
```
### Step 2: the question
In the application, select the `acme-ebikes` workspace and create a session with the question.
Follow the phases and confirm the gates. The model will propose schema linking (tables and columns
from the `datawarehouse` DWH), CTEs, and finally the SQL, which you can view, copy, and run.
---
## Where to find technical details
For DWH REST access, the key is installation-specific and applies only to `rest_api`; `postgres_direct`
and `ssh_tunnel` do not use it. See the [DWH server guide](install/dwh-auth-server.md), [client
enrollment guide](install/dwh-auth-client-enrollment.md), and [TLS guide](install/dwh-auth-tls.md).
- CLI contract: `docs/contracts/workspace-preprocessing-cli.md`
- `.tht-dwh` contract: `docs/contracts/tht-dwh.md`
- Evidence v3: `docs/contracts/workspace-evidence-v3.md`
docs/index.md
+22 -13
View File
@@ -1,20 +1,29 @@
# ThothII documentation # ThothII documentation
ThothII is a human-in-the-loop datamart builder. It turns natural-language questions into validated SQL through an eight-phase workflow orchestrated by Pi. ThothII is a human-reviewed datamart builder. It turns a question into validated SQL through an
eight-phase workflow: the model proposes; a reviewer makes the decisions that are persisted.
The documentation is divided into two areas: Start with the path that matches the work you need to do:
## ThothII technical documentation | I need to… | Start here |
| --- | --- |
| Install or operate one instance | [Install and first start](install/first-start.md) |
| Add, update, or prepare a workspace | [Workspace operations](operations/workspaces.md) |
| Ask a question and review the SQL workflow | [User guide](guida-utente.md) |
| Configure and refresh an authoritative database catalog | [Database management](operations/database-management.md) |
| Author and publish domain Evidence | [Evidence](evidence.md) |
| Understand boundaries and persistence | [Architecture overview](architecture/overview.md) |
This section explains the system architecture, workflow, operating contracts, Evidence, and Memory. Start with the [architecture overview](architecture/overview.md). ## How the documentation is organised
For local authentication, generic OIDC, and Authentik, see the [authentication documentation](architecture/authentication.md). - **Install and operate** documents host-side setup, authentication, lifecycle, workspaces, and
Pi administration.
- **Use ThothII** documents the two application paths: reviewed NL→SQL sessions and administrative
database management.
- **Architecture and contracts** explain why the system behaves as it does and define the
machine-facing boundaries. Consult them when integrating or changing an implementation; they
are not a substitute for an operator runbook.
To install the application in Docker across the four operating contexts, using the env file, The host-side `tht` command is the operator interface. The Python `tht` inside `harness/` is a
`compose.yaml`, the local/server overlay, and the mounted secret bundle, see [Docker installation in the four operating contexts](installazione-docker-4-contesti.md). different, internal workflow CLI invoked by the core. In examples, use an absolute installation
descriptor path whenever discovery is not unambiguous.
For DWH REST with an installation-specific revocable key, see the [server guide](install/dwh-auth-server.md), [client enrollment guide](install/dwh-auth-client-enrollment.md), and [TLS guide](install/dwh-auth-tls.md). This component remains separate from the ThothII Compose stack.
## General topics
Operating and configuration notes that are not specific to the ThothII domain, such as how Pi resolves built-in, user-level, and project-level models. Start with [Pi model configuration](general/pi-configuration.md).
docs/install/first-start.md
+82
View File
@@ -0,0 +1,82 @@
# Install and first start
This is the supported local installation path. It creates an installation-local configuration and
starts the Compose stack; it does not create a workspace repository or a database catalog entry.
## Prerequisites and boundaries
Install Docker Engine with Compose v2, plus the host `tht` command. On macOS or Linux, install the
host command from the repository with `./scripts/install-tht.sh`; Windows uses
`./scripts/install-tht.ps1`. The installer builds or verifies the native command and checks that
`tht` is resolvable on `PATH`.
The stack contains `frontend`, `core`, `catalog-db`, `qdrant`, `embedding`, and the one-shot
`embedding-model-init` and `catalog-migrate` services. DWH and model-provider endpoints are
external installation settings. Pi runs inside `core`; do not install a host Pi executable for
the application runtime.
Secrets, certificates, Pi authentication, and workspace endpoint bindings are protected
installation-local files. Never put them in a workspace descriptor, an env file intended for
version control, a URL, or a command line.
## Create the local installation
From the repository root, start the interactive setup and select the local profile:
```sh
tht setup --profile local
```
It writes the selected non-secret descriptor below `deploy/<installation-id>/`, the associated
operator env file, and can create protected secret templates. Keep the descriptor path: pass it
to commands as `--installation /absolute/path/thothii-installation.yaml` when more than one
installation can be discovered.
If the descriptor is prepared manually instead, begin with
[`thothii-installation.local.yaml`](examples/thothii-installation.local.yaml), set mode `0600` or
`0400`, and ensure `THT_INSTALLATION_CONFIG_SOURCE` in the selected env file points to that exact
file. Create the secret bundle from `deploy/secrets/thothii.secrets.example`, protect it, and set
the file locations and external endpoints in the env file. The required settings include:
- `PI_AUTH_FILE`, `THT_SECRETS_FILE`, and the catalog password source files;
- `THT_INSTALLATION_CONFIG_SOURCE` and the workspace Git remote/branch;
- the DWH and model-provider endpoints; and
- `THT_AUTH_CONFIG_ROOT` for local authentication or the OIDC configuration selected during setup.
For the supported secret names and the metadata-generation model credential boundary, see the
`deploy/secrets/README.md` file in the installation checkout. It is intentionally not published
as a documentation page because it describes a protected local-file contract.
## Start and verify
For the normal local path, use the launcher:
```sh
./scripts/run-stack.sh
```
It builds `core`, starts `catalog-db`, runs `catalog-migrate`, then keeps the base plus local
Compose profile in the foreground. Database migrations are deliberately not a hidden backend
startup action. Open `http://127.0.0.1:8080` unless `THOTH_HTTP_PORT` was changed.
In another terminal, verify the installation without changing it:
```sh
tht --installation /absolute/path/thothii-installation.yaml doctor --json
tht --installation /absolute/path/thothii-installation.yaml status
```
`/health` verifies application-process readiness; `tht doctor` is the diagnostic surface for
Compose, configuration, workspace, workflow, and Pi prerequisites.
## Routine lifecycle and next steps
Use `tht start [--build]`, `tht stop`, `tht logs`, and `tht doctor` rather than composing ad-hoc
container commands. Named volumes retain settings, Pi state, workspace registry, sessions,
Qdrant data, and embedding models across `docker compose down`; removing them requires the
explicit destructive `--volumes` form.
After the stack is healthy, configure authentication if setup did not do so, then continue with
[Workspace operations](../operations/workspaces.md). For server profile, reverse proxy, backups,
and recovery, use the deployment program and its manual gates; the server profile is not a
drop-in replacement for the local command above.
docs/installazione-docker-4-contesti.md
+19 -10
View File
@@ -4,9 +4,11 @@ ThothII uses one Compose topology:
- `frontend` - `frontend`
- `core` - `core`
- `catalog-db`
- `qdrant` - `qdrant`
- `embedding` - `embedding`
- `embedding-model-init` - `embedding-model-init`
- `catalog-migrate` (one-shot)
Qdrant and Ollama embedding are required internal Compose services. Only DWH and the LLM remain Qdrant and Ollama embedding are required internal Compose services. Only DWH and the LLM remain
external. The fixed model is `qwen3-embedding:0.6b` with 1024 dimensions and cosine distance; external. The fixed model is `qwen3-embedding:0.6b` with 1024 dimensions and cosine distance;
@@ -23,7 +25,7 @@ flowchart TB
SERVER --> BUNDLE SERVER --> BUNDLE
SESSION --> BUNDLE SESSION --> BUNDLE
AUTH --> BUNDLE AUTH --> BUNDLE
BUNDLE --> SERVICES["Frontend, core, vector, embedding"] BUNDLE --> SERVICES["Frontend, core, catalog DB, vector, embedding"]
``` ```
## Short ownership contract ## Short ownership contract
@@ -35,7 +37,14 @@ flowchart TB
| Qdrant | Internal | Required internal Compose service with persistent `qdrant-data` volume. | | Qdrant | Internal | Required internal Compose service with persistent `qdrant-data` volume. |
| Ollama embedding | Internal | Required internal Compose service for `qwen3-embedding:0.6b`. | | Ollama embedding | Internal | Required internal Compose service for `qwen3-embedding:0.6b`. |
## Comando standard locale ## Local path: setup, migration, start
The normal local path is [Install and first start](install/first-start.md). It uses `tht setup`
to create the selected installation-local descriptor and runs the catalog migration explicitly
before application startup.
If an operator intentionally prepares the descriptor and protected files by hand, the equivalent
foreground launch is:
```sh ```sh
cp deploy/env/local.env.example deploy/env/local.env cp deploy/env/local.env.example deploy/env/local.env
@@ -46,8 +55,7 @@ chmod 600 deploy/secrets/thothii.secrets
# replace every placeholder, chmod it 600, then set that exact path as # replace every placeholder, chmod it 600, then set that exact path as
# THT_INSTALLATION_CONFIG_SOURCE in deploy/env/local.env. # THT_INSTALLATION_CONFIG_SOURCE in deploy/env/local.env.
docker compose --env-file deploy/env/local.env \ ./scripts/run-stack.sh
-f compose.yaml -f deploy/compose.local.yaml up --build -d
``` ```
Set these values in `deploy/env/local.env`: Set these values in `deploy/env/local.env`:
@@ -90,8 +98,10 @@ A private PEM CA remains outside the bundle and must be mounted through a review
Preprocessing runs through the native host CLI and the installation descriptor: Preprocessing runs through the native host CLI and the installation descriptor:
```sh ```sh
tht --installation /percorso/assoluto/thothii-installation.yaml workspace preprocess evidence tht --installation /percorso/assoluto/thothii-installation.yaml \
tht --installation /percorso/assoluto/thothii-installation.yaml workspace preprocess dwh workspace preprocess evidence --workspace <workspace-id>
tht --installation /percorso/assoluto/thothii-installation.yaml \
workspace preprocess dwh --workspace <workspace-id>
``` ```
The CLI runs the profile-gated `workspace-maintenance` service. See the The CLI runs the profile-gated `workspace-maintenance` service. See the
@@ -108,7 +118,6 @@ docker compose --env-file deploy/env/server.env \
-f deploy/compose.session-server.yaml.example up --build -d -f deploy/compose.session-server.yaml.example up --build -d
``` ```
Also see: Also see [Workspace operations](operations/workspaces.md). Server deployment, reverse-proxy,
backup, and recovery remain manual-gated operations; do not treat the local profile as a server
- `docs/install/local-workspace-registry.md` replacement.
- `docs/install/server-workspace-registry.md`
docs/operations/database-management.md
+67
View File
@@ -0,0 +1,67 @@
# Database management
Database Management is an administrative catalog for an external PostgreSQL schema. It is separate
from workspace preprocessing and, today, does not change the DWH binding used by the NL→SQL
session workflow.
## What the catalog owns
For each YAML workspace, an administrator may create at most one database configuration. It holds
the database name, schema, connection binding, write-only encrypted secrets, observed physical
schema, optional curated descriptions, generated descriptions, and durable operation history.
It does **not** become the external source of truth. Tables, columns, types, defaults,
nullability, primary-key positions, and ordered foreign-key pairs are observations of the source
schema and cannot be manually created, renamed, or structurally edited. Descriptions are the
editable metadata.
## Configure and test a database
1. Open **Database Management** and choose a workspace.
2. Create its PostgreSQL configuration. Choose `postgres_direct`, `rest_api`, or `ssh_tunnel` and
complete the binding fields that the chosen transport requires.
3. Enter secrets only when replacing them. They remain write-only and are never returned by the
application.
4. Run **Test connection** before any synchronization.
SSH uses a private key, optional key passphrase, mandatory `known_hosts`, and optional PostgreSQL
TLS CA/server name. REST prefers `POST /rpc/schema_snapshot`; when it is absent, the catalog may
use the same strict v1 snapshot through one read-only `POST /rpc/run_query`. An unavailable
capability, malformed snapshot, or connector error applies no catalog changes. See the
[schema snapshot contract](../contracts/catalog-schema-snapshot.md).
## Synchronize authoritative schema metadata
Start a synchronization from a database or a selected table set. The available scopes are tables,
columns, relationships, and all. One database can have only one active catalog operation at a
time; cleanup shares this exclusion.
The run scans first and publishes a durable operation. If it detects a destructive difference, it
requires confirmation and re-scans before applying. You can cancel before apply; completed and
failed runs remain in history. The live log is delivered over SSE with a polling fallback.
Explicit cleanup is different from source synchronization: administrators can clear selected
table/relationship or column/relationship catalog metadata without changing the external source,
the connection binding, or secrets. Deleting a table cascades to its columns and relationships.
## Generate and consolidate descriptions
Generated descriptions can be requested for selected tables, selected columns, every eligible
target, or targets with a missing generated description. The backend accepts one installation-wide
run and processes targets sequentially. It reads at most five source rows and five representative
non-null values per relevant source through a read-only connector, then sends that bounded sample
transiently to the configured model provider.
Each successful result is persisted immediately. Stop terminates the active helper but retains
earlier results. A helper has at most one provider retry; three consecutively exhausted technical
batches fail the run. Stale queued/running work is marked interrupted at startup and can be
unlocked only when no local worker/helper is live. There is no automatic resume and no public
description-generation CLI.
Review generated text before copying it into the curated **Description** field. The sampling rule
is a deliberate data-disclosure boundary: do not use this facility for fields whose values must
not be sent to the configured provider until a Sensitive Data Policy is in place.
The decisions behind this surface are [ADRs 0001–0010](../adr/0001-postgres-metadata-catalog.md)
and the detailed acceptance record is
[AI catalog description generation acceptance](../testing/2026-08-29-ai-catalog-description-generation-acceptance.md).
docs/operations/workspaces.md
+65
View File
@@ -0,0 +1,65 @@
# Workspace operations
A workspace is curator-owned Git content plus installation-local runtime bindings. It is the
boundary between what can be published and what can be used by an installation.
## Roles and ownership
| Role | Owns | Does not own |
| --- | --- | --- |
| Curator | `thoth-workspaces.yaml`, `<id>/workspace.yaml`, Evidence, and curated schema annotations | installation secrets or active runtime bindings |
| Installation operator | Git source, selected workspace, write-only runtime secrets, validation, connectivity, and preprocessing | commits or pushes to the workspace repository |
| Reviewer | NL→SQL decisions in a pinned session | workspace publication or preprocessing |
The root catalog has `schema_version: 1` and an ordered list of workspace identities. Each entry
must have a matching schema-v3 descriptor at `<id>/workspace.yaml` in the same Git commit. The
application validates a complete candidate revision and activates it atomically; invalid content
leaves the preceding active revision in place.
## Operator sequence
1. Curate and push a complete repository revision. Do not put DWH passwords, API keys, private
keys, or signed URLs in this repository.
2. In the application, update the workspace repository. This fetches and validates the candidate;
it never edits the remote repository.
3. Select the workspace. Supply or replace its write-only runtime secrets, then run **Validate
workspace source** and **Test workspace connections**.
4. Select it as the installation workspace before creating sessions.
5. Use the host CLI for preprocessing. It dispatches a profile-gated maintenance service and
returns a single structured result; `--json` keeps stdout machine-readable.
```sh
INSTALLATION=/absolute/path/thothii-installation.yaml
WORKSPACE=example-workspace
tht --installation "$INSTALLATION" workspace inspect --workspace "$WORKSPACE" --json
tht --installation "$INSTALLATION" workspace preprocess dwh --workspace "$WORKSPACE" --json
tht --installation "$INSTALLATION" workspace preprocess evidence --workspace "$WORKSPACE" --json
```
For the full DWH → review → schema-index → Evidence chain, run
`workspace preprocess run`. It may stop with `manual_review_required` when FK candidates need a
curator decision. Publish the reviewed annotations, update the repository, then accept that exact
run and resume it:
```sh
tht --installation "$INSTALLATION" workspace schema accept \
--workspace "$WORKSPACE" --run <run-id> --yes --json
tht --installation "$INSTALLATION" workspace preprocess run \
--workspace "$WORKSPACE" --resume <run-id> --json
```
The contract gives exact validation, exit code, and JSON rules in
[Workspace preprocessing CLI](../contracts/workspace-preprocessing-cli.md). For Evidence source
forms and the schema-v3 descriptor contract, see
[Workspace Evidence v3](../contracts/workspace-evidence-v3.md).
## Transport and revision rules
Runtime sessions support direct PostgreSQL and REST bindings. SSH tunnel bindings are diagnostic
only for this path, so they cannot admit an NL→SQL session. Database Management has its own
strict known-host SSH path for connection tests and schema synchronization.
Every new session pins the active Git revision. Snapshot cleanup retains revisions still
referenced by unarchived sessions. A later pull can prepare a future session but cannot alter a
resume.
mkdocs.yml
+41 -20
View File
@@ -44,27 +44,48 @@ markdown_extensions:
alternate_style: true alternate_style: true
nav: nav:
- Home: index.md - Home: index.md
- User guide: guida-utente.md - Install and operate:
- DWH REST per installation: - Install and first start: install/first-start.md
- dwh-auth server: install/dwh-auth-server.md - Local authentication: install/authentication-local.md
- DWH client enrollment: install/dwh-auth-client-enrollment.md - Generic OIDC: install/authentication-oidc.md
- DWH REST TLS: install/dwh-auth-tls.md - Authentik: install/authentik.md
- Contracts and CLI: - Workspace operations: operations/workspaces.md
- Pi model configuration: general/pi-configuration.md
- Docker installation contexts: installazione-docker-4-contesti.md
- Use ThothII:
- User guide: guida-utente.md
- Database management: operations/database-management.md
- Evidence authoring and publication: evidence.md
- Workflow references:
- Initial disambiguation: disambiguazione-iniziale.md
- Memory management: gestione-memory.md
- Operating skills: skills.md
- Architecture:
- Overview: architecture/overview.md
- Components, modules, and flows: architecture/components.md
- Authentication and authorization: architecture/authentication.md
- Contracts and integration:
- Catalog schema snapshot RPC: contracts/catalog-schema-snapshot.md
- Workspace preprocessing CLI: contracts/workspace-preprocessing-cli.md - Workspace preprocessing CLI: contracts/workspace-preprocessing-cli.md
- tht–DWH contract: contracts/tht-dwh.md - tht–DWH contract: contracts/tht-dwh.md
- Workspace Evidence v3 contract: contracts/workspace-evidence-v3.md - Workspace Evidence v3 contract: contracts/workspace-evidence-v3.md
- ThothII technical documentation: - DWH REST server: install/dwh-auth-server.md
- Architecture overview: architecture/overview.md - DWH client enrollment: install/dwh-auth-client-enrollment.md
- Components, modules, and flows: architecture/components.md - DWH REST TLS: install/dwh-auth-tls.md
- Evidence: evidence.md - Decisions and acceptance:
- Architecture decisions:
- 0001 Metadata catalog: adr/0001-postgres-metadata-catalog.md
- 0002 Workspace database secrets: adr/0002-workspace-database-secret-references.md
- 0003 Installation-local bindings: adr/0003-installation-local-database-bindings.md
- 0004 Fastify/Kysely catalog: adr/0004-fastify-kysely-metadata-catalog.md
- 0005 Table deletion during synchronization: adr/0005-hard-delete-catalog-tables-during-synchronization.md
- 0006 Physical and logical relationships: adr/0006-separate-physical-and-logical-relationships.md
- 0007 Authoritative schema synchronization: adr/0007-durable-authoritative-schema-synchronization.md
- 0008 Manual catalog cleanup: adr/0008-allow-manual-catalog-metadata-cleanup.md
- 0009 Sequential description generation: adr/0009-use-one-sequential-description-generation-run.md
- 0010 Bounded source samples: adr/0010-allow-bounded-real-source-samples-for-description-generation.md
- AI catalog description acceptance: testing/2026-08-29-ai-catalog-description-generation-acceptance.md - AI catalog description acceptance: testing/2026-08-29-ai-catalog-description-generation-acceptance.md
- Authentication: architecture/authentication.md - Design records:
- Local authentication installation: install/authentication-local.md - Metadata catalog design: plans/2026-08-26-metadata-catalog-from-thothai.md
- Generic OIDC: install/authentication-oidc.md - Description generation design: plans/2026-08-28-ai-catalog-description-generation.md
- Authentik: install/authentik.md - Description generation specification: plans/2026-08-28-ai-catalog-description-generation-spec.md
- Docker installation (4 contexts): installazione-docker-4-contesti.md
- Memory management: gestione-memory.md
- Operating skills: skills.md
- Initial disambiguation: disambiguazione-iniziale.md
- General topics:
- Pi model configuration: general/pi-configuration.md