From 0ce05869cff3a03a3a5c948fc4beb3548ff1a670 Mon Sep 17 00:00:00 2001 From: Codex Date: Sat, 29 Aug 2026 20:08:48 +0200 Subject: [PATCH] docs: reorganize operational documentation --- README.md | 9 +- docs/architecture/components.md | 6 +- docs/architecture/overview.md | 15 +- docs/guida-utente.md | 339 ++++-------------------- docs/index.md | 35 ++- docs/install/first-start.md | 82 ++++++ docs/installazione-docker-4-contesti.md | 29 +- docs/operations/database-management.md | 67 +++++ docs/operations/workspaces.md | 65 +++++ mkdocs.yml | 61 +++-- 10 files changed, 371 insertions(+), 337 deletions(-) create mode 100644 docs/install/first-start.md create mode 100644 docs/operations/database-management.md create mode 100644 docs/operations/workspaces.md diff --git a/README.md b/README.md index 3eb33aa6..5676c628 100644 --- a/README.md +++ b/README.md @@ -74,11 +74,10 @@ diagnostics are exposed by `tht doctor` and do not prevent the UI from starting. ## Git-backed workspace repository Workspace descriptors are shared through a validated Git repository while endpoint bindings and -secret files remain installation-local. Use the [local Mac/PC installation manual](docs/install/local-workspace-registry.md) -for Docker Desktop or a local engine, the [server installation manual](docs/install/server-workspace-registry.md) -for the Gitea, reverse-proxy, backup, upgrade, and recovery workflow, and the -[P1→P1.1 migration guide](docs/migrations/p1-to-p1-1-registry-layout.md) before upgrading an -older flat-layout registry. +secret files remain installation-local. The supported operating sequence is documented in +[Workspace operations](docs/operations/workspaces.md); it covers curator publication, installation +activation, runtime bindings, and preprocessing. The host setup and lifecycle path is in +[Install and first start](docs/install/first-start.md). The curator-owned repository layout is: diff --git a/docs/architecture/components.md b/docs/architecture/components.md index 61d7e9bf..084a9946 100644 --- a/docs/architecture/components.md +++ b/docs/architecture/components.md @@ -78,8 +78,10 @@ descriptor; its API key remains in the protected installation secret bundle. Each result is written immediately to `Generated Description`. Run state and sanitized activity events are stored in `catalog-db` and exposed to the drawer through REST and SSE. An administrator -may later copy selected generated descriptions into `Description`. There is no parallel run queue, -automatic retry policy, or second orchestration subsystem. +may later copy selected generated descriptions into `Description`. There is no parallel run queue +or second orchestration subsystem. A target receives at most one provider retry; three consecutive +exhausted technical batches fail the run. Stale work is marked interrupted at startup and must be +explicitly unlocked; it never resumes automatically. ## Main backend classes diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 01b0cd7f..e30c76f6 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -22,7 +22,7 @@ flowchart LR THT --> FE ``` -## The three independent projects +## The three independently built layers ``` frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only) @@ -100,8 +100,13 @@ operations that perform this upgrade. - Settings are global (`backend/data/settings.json`: workspace/provider/model/thinking); the new-session form asks only for the question. - **Resume**: a resumable session returns to its last incomplete phase. The backend rejects resume with 409 when `finalized` or `archived`; `PiProcessManager.spawnFor` must send `/riprendi-sessione ` for resume and `/nuova-domanda` for a new session. The wrong prompt silently turns a resume into a new question. -## Starting the stack +## Runtime composition -Start the local stack with `./scripts/run-stack.sh` after creating -`deploy/env/local.env` from `deploy/env/local.env.example`. The Compose core includes Pi; DWH, -the vector database, embeddings, and the LLM are external endpoints configured in the local file. +The local stack is started by `./scripts/run-stack.sh` after the installation descriptor and +`deploy/env/local.env` exist. `core` includes Pi. `catalog-db`, Qdrant, and the Ollama embedding +service are internal Compose services; only the DWH and model-provider endpoint remain external. +The launcher runs the explicit `catalog-migrate` one-shot service before application startup. + +For the operator sequence, see [Install and first start](../install/first-start.md). For the +two user-facing paths, see [User guide](../guida-utente.md) and +[Database management](../operations/database-management.md). diff --git a/docs/guida-utente.md b/docs/guida-utente.md index 7e2a54fa..5cfe8d0c 100644 --- a/docs/guida-utente.md +++ b/docs/guida-utente.md @@ -1,282 +1,57 @@ -# ThothII user guide - -For login, **Remember me**, roles, session invalidation, OIDC groups, and recovery, see the -[local authentication guide](install/authentication-local.md) and the [generic OIDC guide](install/authentication-oidc.md). - -This guide walks you through **preparing** the workspace repository, **using ThothII's tools** for -that repository, and **using the application** to ask natural-language questions and obtain -validated SQL. It uses plain language and examples. The contracts listed at the end contain the -technical details. - -> **What is ThothII?** It is a *datamart builder* with human review. You write a natural-language -> question, the model proposes each step in turn (clarifications, schema, CTEs, and SQL), and a -> **human reviewer decides** at every important step. The final result is validated SQL ready to -> run on the data warehouse. - ---- - -## Part 1: prepare the workspace repository on Git - -### 1.1 Structure - -The workspace repository is a **Git repository** that describes *which data* is available and -*how to reach it*. It contains neither the data nor **secrets** such as passwords, tokens, or certificates. - -A valid repository contains: - -```text -thoth-workspaces.yaml # catalog: list of workspaces -/workspace.yaml # workspace descriptor (schema v3) -/evidence/ # optional context documents, such as *.md -/schema/annotations.yaml # optional manually curated logical joins (P5) -``` - -- The **catalog** `thoth-workspaces.yaml` is a simple list: - -```yaml -schema_version: 1 -workspaces: - - id: acme-ebikes - name: ACME Limited - description: DWH for electric bicycle production -``` - -- The **ID** must be lowercase, contain no spaces, and follow `acme-ebikes` (`[a-z][a-z0-9-]{2,62}`). -- The **descriptor** `/workspace.yaml` uses schema v3. It is the only valid description. - -### 1.2 Descriptor example (ACME Limited) - -```yaml -workspace: - schema_version: 3 - id: acme-ebikes - name: ACME Limited - description: Industrial DWH for electric bicycle production - language: en # descriptions and Evidence are in English - -dwh: - engine: postgres - database: postgres - schema: datawarehouse - supported_transports: [rest_api] # access through the REST API (PostgREST) - -semantic_index: - vector_store: - engine: qdrant - collection: acme-ebikes - dimensions: 1024 - distance: cosine - embedding: - provider: ollama_internal - model: qwen3-embedding:0.6b - dimensions: 1024 - -llm_policy: - allowed: [zai/glm-5.2] - -diagnostics: - dwh_rest: - method: POST - path: /rpc/ping - auth: x-api-key - response: { database: database, schema: schema } - -evidence: - source: - type: filesystem - uri: acme-ebikes/evidence # path inside the repository - policy: - max_chunk_chars: 4000 - retain_published_generations: 3 -``` - -Changes from older workspaces: - -- NL→SQL sessions reach the database only through **REST** or **direct Postgres** (`rest_api` / - `postgres_direct`); `ssh_tunnel` remains disabled for session runtime. The separate Database - management surface supports SSH for **Test connection** and **Sync tables**, with a private key, - mandatory `known_hosts`, and an optional key passphrase; -- the semantic index is **internal** (Qdrant plus `qwen3-embedding:0.6b`, 1024 dimensions, cosine); -- **filesystem** Evidence lives in the repository (`/evidence`) and is materialized from the - pinned Git commit (P6). HTTP Evidence is also supported. - -### 1.3 Rules - -1. **Git is the source of truth.** Change the descriptor, catalog, and Evidence only through a - *commit* and *push*, followed by an installation *pull*. -2. **No secrets in the repository.** Add passwords, tokens, private keys, and signed URLs at - runtime through Workspace management; the backend stores them encrypted. -3. **Schema v3 only.** Reject v1 and v2 descriptors before activation. -4. **The application does not push curated content.** The repository curator works in a separate - authoring clone. - ---- - -## Part 2: use ThothII's repository tools - -There are **two** tools: the **web application** (workspace management) and the **`tht` CLI** -(preprocessing and operations). The complete installation is described in -`docs/install/local-workspace-registry.md` (macOS/Windows/Linux) e -`docs/install/server-workspace-registry.md`. - -### 2.1 `tht`: main commands - -Always invoke `tht` with `--installation /thothii-installation.yaml`. The usual commands are: - -```bash -# 1) inspect workspace state (revision and identity) -tht --installation workspace inspect --workspace --json - -# 2) inspect the DWH (generates physical.yaml and LSH) -tht --installation workspace preprocess dwh --workspace --json - -# 3) suggest joins (FKs) from approved SQL -tht --installation workspace schema suggest-fks --workspace --from-sql .sql --output .yaml --json - -# 4) after review, publish curated FKs in Git and accept them -tht --installation workspace schema accept --workspace --run --yes --json - -# 5) index the schema (Qdrant) -tht --installation workspace index-schema --workspace --json - -# 6) preprocess Evidence -tht --installation workspace preprocess evidence --workspace --json - -# 7) complete chain (DWH → FK → schema → Evidence) -tht --installation workspace preprocess run --workspace --json - -# 8) inspect or rebuild the Qdrant collection (maintenance only) -tht --installation workspace vector inspect --workspace --json -tht --installation workspace vector rebuild --workspace --collection --confirm --destroy -``` - -Important notes: - -- **`--json` writes JSON only to stdout** (machine contract); use it in scripts. -- **`preprocess run` stops for human review** when it finds new proposed joins: it exits with - `manual_review_required`. After review, continue with `schema accept ... --yes` and - `preprocess run --resume `. -- **A filesystem Evidence file is materialized from the pinned Git commit** (there is no moving - checkout); symlinks, unsafe paths, and trees that are too large are rejected. -- **The CLI never writes to the repository** (it never pushes curated content). - -### 2.2 Web application: workspace management - -Workspace management has two distinct levels. - -**Level 1: repository.** The first section shows that the workspace source lives in a separate -directory, is published by the curator to a repository hosted on a Git server such as GitHub, -GitLab, or Gitea, and is read by ThothII in read-only mode. It shows the host, repository, branch, -active revision, and status of the last update. - -* **Update workspace repository** does not require a workspace to be selected. The backend fetches - or pulls the configured branch into ThothII's managed checkout, validates the entire candidate - revision, and activates it atomically. If validation fails, it keeps the previous revision. It - does not modify the remote source or save content from the GUI. -* To create a local workspace, prepare a source directory with the catalog, `workspace.yaml`, and - the expected subdirectories. Validate it, then commit and push from the authoring clone. - ThothII provides no commands to create, edit, or publish the source. - -**Level 2: selected workspace.** These commands are separate because they require a workspace to be selected first. -These commands are separate because they require a workspace to be selected first. - -- **Validate workspace source** checks the catalog, descriptor, Evidence, and invariants of the - selected active revision again. It does not contact the DWH or modify files. -- **Save entered secrets** replaces the entered values without displaying them. The fields depend - on the declared DWH transport and Evidence authentication. The backend returns only configured - or missing status. -* **Forget stored value** removes the selected secret from the encrypted vault. Future sessions or - operations that need it remain blocked until it is entered again. -The remote Git repository and its credentials are installation settings. Runtime DWH and Evidence -secrets persist in the backend's encrypted vault, not in the GUI's local storage. The GUI is only -the interface: after submission it clears the field values and cannot read them back. - ---- - -## Part 3: use the ThothII application - -### 3.1 New session - -Open the application and use **New session**. Enter only the **natural-language question**; -workspace, model, and provider are already configured as global settings. - -Example question: - -> "List the electric bicycles completed in the last year, with model, frame number, and completion -> date." - -### 3.2 The eight-phase workflow and gates - -The question passes through **eight phases**. You see the intermediate documents and decide at the important points: - -1. **F1 clarification**: the model removes ambiguity when needed; -2. **F2 Memory**: retrieves reusable Memory; -3. **F3 rewriting**: rewrites and approves the question; -4. **F4 schema linking**: proposes related tables and columns; -5. **F5 summary**: summarizes the selected schema; -6. **F6 CTE**: builds the CTEs; -7. **F7 final SQL**: produces `sql_final.sql`; -8. **F8 datamart**: execution or export (dbt, CSV, Excel). - -**Review gates** appear as widgets: choose one option, select several items, or confirm an -artifact or phase. The model *proposes* and the reviewer *decides*. The right side shows artifacts -(schema linking, CTEs, and SQL); the Model activity panel shows the question and reasoning. - -### 3.3 Sessions - -Sessions appear in the sidebar with their ID, question, date, and author. A session -**resumes** from its last incomplete phase by rebuilding state from documents saved on disk -(`session_manifest.yaml`, phase artifacts, and `review_decisions.jsonl`). Saved state **is** the -truth: what is not recorded did not happen. - ---- - -## Complete example: ACME Limited - -### Step 0: repository - -Create the workspace Git repository, for example `tht-workspace-acme`: - -```text -thoth-workspaces.yaml # catalog containing acme-ebikes -acme-ebikes/workspace.yaml # v3 descriptor (see §1.2) -acme-ebikes/evidence/ # curated context .md documents -acme-ebikes/schema/annotations.yaml # when curated joins exist -``` - -Publish a new Git revision. In the installation, the application fetches and **activates** the -workspace, validates schema v3, materializes Evidence from the pinned revision, and prepares the -Qdrant collection (1024/cosine plus indexes). - -### Step 1: preprocessing - -```bash -tht --installation ~/thothii-installation.yaml workspace preprocess dwh --workspace acme-ebikes --json -tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --json -``` - -If the run stops for joins (`manual_review_required`): - -```bash -# the curator reviews the candidates and publishes acme-ebikes/schema/annotations.yaml, then: -tht --installation ~/thothii-installation.yaml workspace schema accept --workspace acme-ebikes --run RUN_ID --yes --json -tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --resume RUN_ID --json -``` - -### Step 2: the question - -In the application, select the `acme-ebikes` workspace and create a session with the question. -Follow the phases and confirm the gates. The model will propose schema linking (tables and columns -from the `datawarehouse` DWH), CTEs, and finally the SQL, which you can view, copy, and run. - ---- - -## Where to find technical details - -For DWH REST access, the key is installation-specific and applies only to `rest_api`; `postgres_direct` -and `ssh_tunnel` do not use it. See the [DWH server guide](install/dwh-auth-server.md), [client -enrollment guide](install/dwh-auth-client-enrollment.md), and [TLS guide](install/dwh-auth-tls.md). - -- CLI contract: `docs/contracts/workspace-preprocessing-cli.md` -- `.tht-dwh` contract: `docs/contracts/tht-dwh.md` -- Evidence v3: `docs/contracts/workspace-evidence-v3.md` +# User guide: from question to validated SQL + +This guide is for a reviewer using a configured ThothII installation. Installation, workspace +publication, preprocessing, and database administration are separate paths; links to them are at +the end of this page. + +## Before creating a session + +An administrator must have selected a workspace and configured the installation-wide provider, +model, and thinking settings. The New session form deliberately asks only for the question. + +The workspace is a pinned Git revision. A subsequent workspace update cannot alter a session +already created from an earlier revision. If a workspace cannot reach its configured runtime DWH, +new sessions are refused before any session state is written. + +## Create and review a session + +1. Sign in and select **New session**. +2. Enter a precise business question, including the relevant time period and desired output. For + example: “List patients discharged in the last 30 days, with ward and discharge date.” +3. Review each gate and make the decision requested by the widget. A choice with a decision payload + can persist immediately; a multi-choice widget records each selected decision; a confirmation + widget approves an artifact or phase. +4. Inspect the generated artifacts, especially schema linking, CTEs, and final SQL. The final SQL is + available only after the F7 review gate. +5. At F8, decide whether a datamart is requested. This is distinct from approving the SQL. + +The workflow phases are fixed: + +| Phase | What is reviewed | +| --- | --- | +| F1–F3 | clarification, reusable Memory, and the rewritten question | +| F4–F5 | proposed tables, columns, Evidence, then their summary | +| F6 | a CTE plan or an explicit skip | +| F7 | `sql_final.sql` | +| F8 | the datamart request or refusal | + +## Resume, archive, and the meaning of saved state + +The sidebar lists sessions and their current lifecycle. Resuming returns to the last incomplete +phase. A finalized or archived session cannot be resumed. + +The source of truth is the workspace session directory: `session_manifest.yaml`, phase artifacts, +and `review_decisions.jsonl`. The visible activity stream is rebuilt from live SSE events and is +not a transcript store. Therefore a decision or artifact that has not been persisted did not +happen from the workflow’s point of view. + +## Choose the correct neighbouring path + +- To prepare, update, validate, or preprocess a workspace, use + [Workspace operations](operations/workspaces.md). +- To create a database configuration, refresh its physical schema, or generate catalog + descriptions, use [Database management](operations/database-management.md). +- To author material the workflow can retrieve, use [Evidence](evidence.md). A proposal from a + session does not become Evidence automatically: a curator must review and publish it in Git. +- For login and access recovery, use [local authentication](install/authentication-local.md) or + [OIDC authentication](install/authentication-oidc.md). diff --git a/docs/index.md b/docs/index.md index 80168b88..ffc5e74d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,20 +1,29 @@ # ThothII documentation -ThothII is a human-in-the-loop datamart builder. It turns natural-language questions into validated SQL through an eight-phase workflow orchestrated by Pi. +ThothII is a human-reviewed datamart builder. It turns a question into validated SQL through an +eight-phase workflow: the model proposes; a reviewer makes the decisions that are persisted. -The documentation is divided into two areas: +Start with the path that matches the work you need to do: -## ThothII technical documentation +| I need to… | Start here | +| --- | --- | +| Install or operate one instance | [Install and first start](install/first-start.md) | +| Add, update, or prepare a workspace | [Workspace operations](operations/workspaces.md) | +| Ask a question and review the SQL workflow | [User guide](guida-utente.md) | +| Configure and refresh an authoritative database catalog | [Database management](operations/database-management.md) | +| Author and publish domain Evidence | [Evidence](evidence.md) | +| Understand boundaries and persistence | [Architecture overview](architecture/overview.md) | -This section explains the system architecture, workflow, operating contracts, Evidence, and Memory. Start with the [architecture overview](architecture/overview.md). +## How the documentation is organised -For local authentication, generic OIDC, and Authentik, see the [authentication documentation](architecture/authentication.md). +- **Install and operate** documents host-side setup, authentication, lifecycle, workspaces, and + Pi administration. +- **Use ThothII** documents the two application paths: reviewed NL→SQL sessions and administrative + database management. +- **Architecture and contracts** explain why the system behaves as it does and define the + machine-facing boundaries. Consult them when integrating or changing an implementation; they + are not a substitute for an operator runbook. -To install the application in Docker across the four operating contexts, using the env file, -`compose.yaml`, the local/server overlay, and the mounted secret bundle, see [Docker installation in the four operating contexts](installazione-docker-4-contesti.md). - -For DWH REST with an installation-specific revocable key, see the [server guide](install/dwh-auth-server.md), [client enrollment guide](install/dwh-auth-client-enrollment.md), and [TLS guide](install/dwh-auth-tls.md). This component remains separate from the ThothII Compose stack. - -## General topics - -Operating and configuration notes that are not specific to the ThothII domain, such as how Pi resolves built-in, user-level, and project-level models. Start with [Pi model configuration](general/pi-configuration.md). +The host-side `tht` command is the operator interface. The Python `tht` inside `harness/` is a +different, internal workflow CLI invoked by the core. In examples, use an absolute installation +descriptor path whenever discovery is not unambiguous. diff --git a/docs/install/first-start.md b/docs/install/first-start.md new file mode 100644 index 00000000..52466f64 --- /dev/null +++ b/docs/install/first-start.md @@ -0,0 +1,82 @@ +# Install and first start + +This is the supported local installation path. It creates an installation-local configuration and +starts the Compose stack; it does not create a workspace repository or a database catalog entry. + +## Prerequisites and boundaries + +Install Docker Engine with Compose v2, plus the host `tht` command. On macOS or Linux, install the +host command from the repository with `./scripts/install-tht.sh`; Windows uses +`./scripts/install-tht.ps1`. The installer builds or verifies the native command and checks that +`tht` is resolvable on `PATH`. + +The stack contains `frontend`, `core`, `catalog-db`, `qdrant`, `embedding`, and the one-shot +`embedding-model-init` and `catalog-migrate` services. DWH and model-provider endpoints are +external installation settings. Pi runs inside `core`; do not install a host Pi executable for +the application runtime. + +Secrets, certificates, Pi authentication, and workspace endpoint bindings are protected +installation-local files. Never put them in a workspace descriptor, an env file intended for +version control, a URL, or a command line. + +## Create the local installation + +From the repository root, start the interactive setup and select the local profile: + +```sh +tht setup --profile local +``` + +It writes the selected non-secret descriptor below `deploy//`, the associated +operator env file, and can create protected secret templates. Keep the descriptor path: pass it +to commands as `--installation /absolute/path/thothii-installation.yaml` when more than one +installation can be discovered. + +If the descriptor is prepared manually instead, begin with +[`thothii-installation.local.yaml`](examples/thothii-installation.local.yaml), set mode `0600` or +`0400`, and ensure `THT_INSTALLATION_CONFIG_SOURCE` in the selected env file points to that exact +file. Create the secret bundle from `deploy/secrets/thothii.secrets.example`, protect it, and set +the file locations and external endpoints in the env file. The required settings include: + +- `PI_AUTH_FILE`, `THT_SECRETS_FILE`, and the catalog password source files; +- `THT_INSTALLATION_CONFIG_SOURCE` and the workspace Git remote/branch; +- the DWH and model-provider endpoints; and +- `THT_AUTH_CONFIG_ROOT` for local authentication or the OIDC configuration selected during setup. + +For the supported secret names and the metadata-generation model credential boundary, see the +`deploy/secrets/README.md` file in the installation checkout. It is intentionally not published +as a documentation page because it describes a protected local-file contract. + +## Start and verify + +For the normal local path, use the launcher: + +```sh +./scripts/run-stack.sh +``` + +It builds `core`, starts `catalog-db`, runs `catalog-migrate`, then keeps the base plus local +Compose profile in the foreground. Database migrations are deliberately not a hidden backend +startup action. Open `http://127.0.0.1:8080` unless `THOTH_HTTP_PORT` was changed. + +In another terminal, verify the installation without changing it: + +```sh +tht --installation /absolute/path/thothii-installation.yaml doctor --json +tht --installation /absolute/path/thothii-installation.yaml status +``` + +`/health` verifies application-process readiness; `tht doctor` is the diagnostic surface for +Compose, configuration, workspace, workflow, and Pi prerequisites. + +## Routine lifecycle and next steps + +Use `tht start [--build]`, `tht stop`, `tht logs`, and `tht doctor` rather than composing ad-hoc +container commands. Named volumes retain settings, Pi state, workspace registry, sessions, +Qdrant data, and embedding models across `docker compose down`; removing them requires the +explicit destructive `--volumes` form. + +After the stack is healthy, configure authentication if setup did not do so, then continue with +[Workspace operations](../operations/workspaces.md). For server profile, reverse proxy, backups, +and recovery, use the deployment program and its manual gates; the server profile is not a +drop-in replacement for the local command above. diff --git a/docs/installazione-docker-4-contesti.md b/docs/installazione-docker-4-contesti.md index 29816526..b80a4c69 100644 --- a/docs/installazione-docker-4-contesti.md +++ b/docs/installazione-docker-4-contesti.md @@ -4,9 +4,11 @@ ThothII uses one Compose topology: - `frontend` - `core` +- `catalog-db` - `qdrant` - `embedding` - `embedding-model-init` +- `catalog-migrate` (one-shot) Qdrant and Ollama embedding are required internal Compose services. Only DWH and the LLM remain external. The fixed model is `qwen3-embedding:0.6b` with 1024 dimensions and cosine distance; @@ -23,7 +25,7 @@ flowchart TB SERVER --> BUNDLE SESSION --> BUNDLE AUTH --> BUNDLE - BUNDLE --> SERVICES["Frontend, core, vector, embedding"] + BUNDLE --> SERVICES["Frontend, core, catalog DB, vector, embedding"] ``` ## Short ownership contract @@ -35,7 +37,14 @@ flowchart TB | Qdrant | Internal | Required internal Compose service with persistent `qdrant-data` volume. | | Ollama embedding | Internal | Required internal Compose service for `qwen3-embedding:0.6b`. | -## Comando standard locale +## Local path: setup, migration, start + +The normal local path is [Install and first start](install/first-start.md). It uses `tht setup` +to create the selected installation-local descriptor and runs the catalog migration explicitly +before application startup. + +If an operator intentionally prepares the descriptor and protected files by hand, the equivalent +foreground launch is: ```sh cp deploy/env/local.env.example deploy/env/local.env @@ -46,8 +55,7 @@ chmod 600 deploy/secrets/thothii.secrets # replace every placeholder, chmod it 600, then set that exact path as # THT_INSTALLATION_CONFIG_SOURCE in deploy/env/local.env. -docker compose --env-file deploy/env/local.env \ - -f compose.yaml -f deploy/compose.local.yaml up --build -d +./scripts/run-stack.sh ``` Set these values in `deploy/env/local.env`: @@ -90,8 +98,10 @@ A private PEM CA remains outside the bundle and must be mounted through a review Preprocessing runs through the native host CLI and the installation descriptor: ```sh -tht --installation /percorso/assoluto/thothii-installation.yaml workspace preprocess evidence -tht --installation /percorso/assoluto/thothii-installation.yaml workspace preprocess dwh +tht --installation /percorso/assoluto/thothii-installation.yaml \ + workspace preprocess evidence --workspace +tht --installation /percorso/assoluto/thothii-installation.yaml \ + workspace preprocess dwh --workspace ``` The CLI runs the profile-gated `workspace-maintenance` service. See the @@ -108,7 +118,6 @@ docker compose --env-file deploy/env/server.env \ -f deploy/compose.session-server.yaml.example up --build -d ``` -Also see: - -- `docs/install/local-workspace-registry.md` -- `docs/install/server-workspace-registry.md` +Also see [Workspace operations](operations/workspaces.md). Server deployment, reverse-proxy, +backup, and recovery remain manual-gated operations; do not treat the local profile as a server +replacement. diff --git a/docs/operations/database-management.md b/docs/operations/database-management.md new file mode 100644 index 00000000..4b2e2614 --- /dev/null +++ b/docs/operations/database-management.md @@ -0,0 +1,67 @@ +# Database management + +Database Management is an administrative catalog for an external PostgreSQL schema. It is separate +from workspace preprocessing and, today, does not change the DWH binding used by the NL→SQL +session workflow. + +## What the catalog owns + +For each YAML workspace, an administrator may create at most one database configuration. It holds +the database name, schema, connection binding, write-only encrypted secrets, observed physical +schema, optional curated descriptions, generated descriptions, and durable operation history. + +It does **not** become the external source of truth. Tables, columns, types, defaults, +nullability, primary-key positions, and ordered foreign-key pairs are observations of the source +schema and cannot be manually created, renamed, or structurally edited. Descriptions are the +editable metadata. + +## Configure and test a database + +1. Open **Database Management** and choose a workspace. +2. Create its PostgreSQL configuration. Choose `postgres_direct`, `rest_api`, or `ssh_tunnel` and + complete the binding fields that the chosen transport requires. +3. Enter secrets only when replacing them. They remain write-only and are never returned by the + application. +4. Run **Test connection** before any synchronization. + +SSH uses a private key, optional key passphrase, mandatory `known_hosts`, and optional PostgreSQL +TLS CA/server name. REST prefers `POST /rpc/schema_snapshot`; when it is absent, the catalog may +use the same strict v1 snapshot through one read-only `POST /rpc/run_query`. An unavailable +capability, malformed snapshot, or connector error applies no catalog changes. See the +[schema snapshot contract](../contracts/catalog-schema-snapshot.md). + +## Synchronize authoritative schema metadata + +Start a synchronization from a database or a selected table set. The available scopes are tables, +columns, relationships, and all. One database can have only one active catalog operation at a +time; cleanup shares this exclusion. + +The run scans first and publishes a durable operation. If it detects a destructive difference, it +requires confirmation and re-scans before applying. You can cancel before apply; completed and +failed runs remain in history. The live log is delivered over SSE with a polling fallback. + +Explicit cleanup is different from source synchronization: administrators can clear selected +table/relationship or column/relationship catalog metadata without changing the external source, +the connection binding, or secrets. Deleting a table cascades to its columns and relationships. + +## Generate and consolidate descriptions + +Generated descriptions can be requested for selected tables, selected columns, every eligible +target, or targets with a missing generated description. The backend accepts one installation-wide +run and processes targets sequentially. It reads at most five source rows and five representative +non-null values per relevant source through a read-only connector, then sends that bounded sample +transiently to the configured model provider. + +Each successful result is persisted immediately. Stop terminates the active helper but retains +earlier results. A helper has at most one provider retry; three consecutively exhausted technical +batches fail the run. Stale queued/running work is marked interrupted at startup and can be +unlocked only when no local worker/helper is live. There is no automatic resume and no public +description-generation CLI. + +Review generated text before copying it into the curated **Description** field. The sampling rule +is a deliberate data-disclosure boundary: do not use this facility for fields whose values must +not be sent to the configured provider until a Sensitive Data Policy is in place. + +The decisions behind this surface are [ADRs 0001–0010](../adr/0001-postgres-metadata-catalog.md) +and the detailed acceptance record is +[AI catalog description generation acceptance](../testing/2026-08-29-ai-catalog-description-generation-acceptance.md). diff --git a/docs/operations/workspaces.md b/docs/operations/workspaces.md new file mode 100644 index 00000000..a50815f5 --- /dev/null +++ b/docs/operations/workspaces.md @@ -0,0 +1,65 @@ +# Workspace operations + +A workspace is curator-owned Git content plus installation-local runtime bindings. It is the +boundary between what can be published and what can be used by an installation. + +## Roles and ownership + +| Role | Owns | Does not own | +| --- | --- | --- | +| Curator | `thoth-workspaces.yaml`, `/workspace.yaml`, Evidence, and curated schema annotations | installation secrets or active runtime bindings | +| Installation operator | Git source, selected workspace, write-only runtime secrets, validation, connectivity, and preprocessing | commits or pushes to the workspace repository | +| Reviewer | NL→SQL decisions in a pinned session | workspace publication or preprocessing | + +The root catalog has `schema_version: 1` and an ordered list of workspace identities. Each entry +must have a matching schema-v3 descriptor at `/workspace.yaml` in the same Git commit. The +application validates a complete candidate revision and activates it atomically; invalid content +leaves the preceding active revision in place. + +## Operator sequence + +1. Curate and push a complete repository revision. Do not put DWH passwords, API keys, private + keys, or signed URLs in this repository. +2. In the application, update the workspace repository. This fetches and validates the candidate; + it never edits the remote repository. +3. Select the workspace. Supply or replace its write-only runtime secrets, then run **Validate + workspace source** and **Test workspace connections**. +4. Select it as the installation workspace before creating sessions. +5. Use the host CLI for preprocessing. It dispatches a profile-gated maintenance service and + returns a single structured result; `--json` keeps stdout machine-readable. + +```sh +INSTALLATION=/absolute/path/thothii-installation.yaml +WORKSPACE=example-workspace + +tht --installation "$INSTALLATION" workspace inspect --workspace "$WORKSPACE" --json +tht --installation "$INSTALLATION" workspace preprocess dwh --workspace "$WORKSPACE" --json +tht --installation "$INSTALLATION" workspace preprocess evidence --workspace "$WORKSPACE" --json +``` + +For the full DWH → review → schema-index → Evidence chain, run +`workspace preprocess run`. It may stop with `manual_review_required` when FK candidates need a +curator decision. Publish the reviewed annotations, update the repository, then accept that exact +run and resume it: + +```sh +tht --installation "$INSTALLATION" workspace schema accept \ + --workspace "$WORKSPACE" --run --yes --json +tht --installation "$INSTALLATION" workspace preprocess run \ + --workspace "$WORKSPACE" --resume --json +``` + +The contract gives exact validation, exit code, and JSON rules in +[Workspace preprocessing CLI](../contracts/workspace-preprocessing-cli.md). For Evidence source +forms and the schema-v3 descriptor contract, see +[Workspace Evidence v3](../contracts/workspace-evidence-v3.md). + +## Transport and revision rules + +Runtime sessions support direct PostgreSQL and REST bindings. SSH tunnel bindings are diagnostic +only for this path, so they cannot admit an NL→SQL session. Database Management has its own +strict known-host SSH path for connection tests and schema synchronization. + +Every new session pins the active Git revision. Snapshot cleanup retains revisions still +referenced by unarchived sessions. A later pull can prepare a future session but cannot alter a +resume. diff --git a/mkdocs.yml b/mkdocs.yml index 42096720..eb3108c8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -44,27 +44,48 @@ markdown_extensions: alternate_style: true nav: - Home: index.md -- User guide: guida-utente.md -- DWH REST per installation: - - dwh-auth server: install/dwh-auth-server.md - - DWH client enrollment: install/dwh-auth-client-enrollment.md - - DWH REST TLS: install/dwh-auth-tls.md -- Contracts and CLI: +- Install and operate: + - Install and first start: install/first-start.md + - Local authentication: install/authentication-local.md + - Generic OIDC: install/authentication-oidc.md + - Authentik: install/authentik.md + - Workspace operations: operations/workspaces.md + - Pi model configuration: general/pi-configuration.md + - Docker installation contexts: installazione-docker-4-contesti.md +- Use ThothII: + - User guide: guida-utente.md + - Database management: operations/database-management.md + - Evidence authoring and publication: evidence.md + - Workflow references: + - Initial disambiguation: disambiguazione-iniziale.md + - Memory management: gestione-memory.md + - Operating skills: skills.md +- Architecture: + - Overview: architecture/overview.md + - Components, modules, and flows: architecture/components.md + - Authentication and authorization: architecture/authentication.md +- Contracts and integration: + - Catalog schema snapshot RPC: contracts/catalog-schema-snapshot.md - Workspace preprocessing CLI: contracts/workspace-preprocessing-cli.md - tht–DWH contract: contracts/tht-dwh.md - Workspace Evidence v3 contract: contracts/workspace-evidence-v3.md -- ThothII technical documentation: - - Architecture overview: architecture/overview.md - - Components, modules, and flows: architecture/components.md - - Evidence: evidence.md + - DWH REST server: install/dwh-auth-server.md + - DWH client enrollment: install/dwh-auth-client-enrollment.md + - DWH REST TLS: install/dwh-auth-tls.md +- Decisions and acceptance: + - Architecture decisions: + - 0001 Metadata catalog: adr/0001-postgres-metadata-catalog.md + - 0002 Workspace database secrets: adr/0002-workspace-database-secret-references.md + - 0003 Installation-local bindings: adr/0003-installation-local-database-bindings.md + - 0004 Fastify/Kysely catalog: adr/0004-fastify-kysely-metadata-catalog.md + - 0005 Table deletion during synchronization: adr/0005-hard-delete-catalog-tables-during-synchronization.md + - 0006 Physical and logical relationships: adr/0006-separate-physical-and-logical-relationships.md + - 0007 Authoritative schema synchronization: adr/0007-durable-authoritative-schema-synchronization.md + - 0008 Manual catalog cleanup: adr/0008-allow-manual-catalog-metadata-cleanup.md + - 0009 Sequential description generation: adr/0009-use-one-sequential-description-generation-run.md + - 0010 Bounded source samples: adr/0010-allow-bounded-real-source-samples-for-description-generation.md - AI catalog description acceptance: testing/2026-08-29-ai-catalog-description-generation-acceptance.md - - Authentication: architecture/authentication.md - - Local authentication installation: install/authentication-local.md - - Generic OIDC: install/authentication-oidc.md - - Authentik: install/authentik.md - - Docker installation (4 contexts): installazione-docker-4-contesti.md - - Memory management: gestione-memory.md - - Operating skills: skills.md - - Initial disambiguation: disambiguazione-iniziale.md -- General topics: - - Pi model configuration: general/pi-configuration.md + - Design records: + - Metadata catalog design: plans/2026-08-26-metadata-catalog-from-thothai.md + - Description generation design: plans/2026-08-28-ai-catalog-description-generation.md + - Description generation specification: plans/2026-08-28-ai-catalog-description-generation-spec.md