docs: reorganize operational documentation
This commit is contained in:
Codex
@@ -74,11 +74,10 @@ diagnostics are exposed by `tht doctor` and do not prevent the UI from starting.
|
||||
## Git-backed workspace repository
|
||||
|
||||
Workspace descriptors are shared through a validated Git repository while endpoint bindings and
|
||||
secret files remain installation-local. Use the [local Mac/PC installation manual](docs/install/local-workspace-registry.md)
|
||||
for Docker Desktop or a local engine, the [server installation manual](docs/install/server-workspace-registry.md)
|
||||
for the Gitea, reverse-proxy, backup, upgrade, and recovery workflow, and the
|
||||
[P1→P1.1 migration guide](docs/migrations/p1-to-p1-1-registry-layout.md) before upgrading an
|
||||
older flat-layout registry.
|
||||
secret files remain installation-local. The supported operating sequence is documented in
|
||||
[Workspace operations](docs/operations/workspaces.md); it covers curator publication, installation
|
||||
activation, runtime bindings, and preprocessing. The host setup and lifecycle path is in
|
||||
[Install and first start](docs/install/first-start.md).
|
||||
|
||||
The curator-owned repository layout is:
|
||||
|
||||
|
||||
@@ -78,8 +78,10 @@ descriptor; its API key remains in the protected installation secret bundle.
|
||||
|
||||
Each result is written immediately to `Generated Description`. Run state and sanitized activity
|
||||
events are stored in `catalog-db` and exposed to the drawer through REST and SSE. An administrator
|
||||
may later copy selected generated descriptions into `Description`. There is no parallel run queue,
|
||||
automatic retry policy, or second orchestration subsystem.
|
||||
may later copy selected generated descriptions into `Description`. There is no parallel run queue
|
||||
or second orchestration subsystem. A target receives at most one provider retry; three consecutive
|
||||
exhausted technical batches fail the run. Stale work is marked interrupted at startup and must be
|
||||
explicitly unlocked; it never resumes automatically.
|
||||
|
||||
## Main backend classes
|
||||
|
||||
|
||||
@@ -22,7 +22,7 @@ flowchart LR
|
||||
THT --> FE
|
||||
```
|
||||
|
||||
## The three independent projects
|
||||
## The three independently built layers
|
||||
|
||||
```
|
||||
frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only)
|
||||
@@ -100,8 +100,13 @@ operations that perform this upgrade.
|
||||
- Settings are global (`backend/data/settings.json`: workspace/provider/model/thinking); the new-session form asks only for the question.
|
||||
- **Resume**: a resumable session returns to its last incomplete phase. The backend rejects resume with 409 when `finalized` or `archived`; `PiProcessManager.spawnFor` must send `/riprendi-sessione <id>` for resume and `/nuova-domanda` for a new session. The wrong prompt silently turns a resume into a new question.
|
||||
|
||||
## Starting the stack
|
||||
## Runtime composition
|
||||
|
||||
Start the local stack with `./scripts/run-stack.sh` after creating
|
||||
`deploy/env/local.env` from `deploy/env/local.env.example`. The Compose core includes Pi; DWH,
|
||||
the vector database, embeddings, and the LLM are external endpoints configured in the local file.
|
||||
The local stack is started by `./scripts/run-stack.sh` after the installation descriptor and
|
||||
`deploy/env/local.env` exist. `core` includes Pi. `catalog-db`, Qdrant, and the Ollama embedding
|
||||
service are internal Compose services; only the DWH and model-provider endpoint remain external.
|
||||
The launcher runs the explicit `catalog-migrate` one-shot service before application startup.
|
||||
|
||||
For the operator sequence, see [Install and first start](../install/first-start.md). For the
|
||||
two user-facing paths, see [User guide](../guida-utente.md) and
|
||||
[Database management](../operations/database-management.md).
|
||||
|
||||
+57
-282
@@ -1,282 +1,57 @@
|
||||
# ThothII user guide
|
||||
|
||||
For login, **Remember me**, roles, session invalidation, OIDC groups, and recovery, see the
|
||||
[local authentication guide](install/authentication-local.md) and the [generic OIDC guide](install/authentication-oidc.md).
|
||||
|
||||
This guide walks you through **preparing** the workspace repository, **using ThothII's tools** for
|
||||
that repository, and **using the application** to ask natural-language questions and obtain
|
||||
validated SQL. It uses plain language and examples. The contracts listed at the end contain the
|
||||
technical details.
|
||||
|
||||
> **What is ThothII?** It is a *datamart builder* with human review. You write a natural-language
|
||||
> question, the model proposes each step in turn (clarifications, schema, CTEs, and SQL), and a
|
||||
> **human reviewer decides** at every important step. The final result is validated SQL ready to
|
||||
> run on the data warehouse.
|
||||
|
||||
---
|
||||
|
||||
## Part 1: prepare the workspace repository on Git
|
||||
|
||||
### 1.1 Structure
|
||||
|
||||
The workspace repository is a **Git repository** that describes *which data* is available and
|
||||
*how to reach it*. It contains neither the data nor **secrets** such as passwords, tokens, or certificates.
|
||||
|
||||
A valid repository contains:
|
||||
|
||||
```text
|
||||
thoth-workspaces.yaml # catalog: list of workspaces
|
||||
<workspace-id>/workspace.yaml # workspace descriptor (schema v3)
|
||||
<workspace-id>/evidence/ # optional context documents, such as *.md
|
||||
<workspace-id>/schema/annotations.yaml # optional manually curated logical joins (P5)
|
||||
```
|
||||
|
||||
- The **catalog** `thoth-workspaces.yaml` is a simple list:
|
||||
|
||||
```yaml
|
||||
schema_version: 1
|
||||
workspaces:
|
||||
- id: acme-ebikes
|
||||
name: ACME Limited
|
||||
description: DWH for electric bicycle production
|
||||
```
|
||||
|
||||
- The **ID** must be lowercase, contain no spaces, and follow `acme-ebikes` (`[a-z][a-z0-9-]{2,62}`).
|
||||
- The **descriptor** `<id>/workspace.yaml` uses schema v3. It is the only valid description.
|
||||
|
||||
### 1.2 Descriptor example (ACME Limited)
|
||||
|
||||
```yaml
|
||||
workspace:
|
||||
schema_version: 3
|
||||
id: acme-ebikes
|
||||
name: ACME Limited
|
||||
description: Industrial DWH for electric bicycle production
|
||||
language: en # descriptions and Evidence are in English
|
||||
|
||||
dwh:
|
||||
engine: postgres
|
||||
database: postgres
|
||||
schema: datawarehouse
|
||||
supported_transports: [rest_api] # access through the REST API (PostgREST)
|
||||
|
||||
semantic_index:
|
||||
vector_store:
|
||||
engine: qdrant
|
||||
collection: acme-ebikes
|
||||
dimensions: 1024
|
||||
distance: cosine
|
||||
embedding:
|
||||
provider: ollama_internal
|
||||
model: qwen3-embedding:0.6b
|
||||
dimensions: 1024
|
||||
|
||||
llm_policy:
|
||||
allowed: [zai/glm-5.2]
|
||||
|
||||
diagnostics:
|
||||
dwh_rest:
|
||||
method: POST
|
||||
path: /rpc/ping
|
||||
auth: x-api-key
|
||||
response: { database: database, schema: schema }
|
||||
|
||||
evidence:
|
||||
source:
|
||||
type: filesystem
|
||||
uri: acme-ebikes/evidence # path inside the repository
|
||||
policy:
|
||||
max_chunk_chars: 4000
|
||||
retain_published_generations: 3
|
||||
```
|
||||
|
||||
Changes from older workspaces:
|
||||
|
||||
- NL→SQL sessions reach the database only through **REST** or **direct Postgres** (`rest_api` /
|
||||
`postgres_direct`); `ssh_tunnel` remains disabled for session runtime. The separate Database
|
||||
management surface supports SSH for **Test connection** and **Sync tables**, with a private key,
|
||||
mandatory `known_hosts`, and an optional key passphrase;
|
||||
- the semantic index is **internal** (Qdrant plus `qwen3-embedding:0.6b`, 1024 dimensions, cosine);
|
||||
- **filesystem** Evidence lives in the repository (`<id>/evidence`) and is materialized from the
|
||||
pinned Git commit (P6). HTTP Evidence is also supported.
|
||||
|
||||
### 1.3 Rules
|
||||
|
||||
1. **Git is the source of truth.** Change the descriptor, catalog, and Evidence only through a
|
||||
*commit* and *push*, followed by an installation *pull*.
|
||||
2. **No secrets in the repository.** Add passwords, tokens, private keys, and signed URLs at
|
||||
runtime through Workspace management; the backend stores them encrypted.
|
||||
3. **Schema v3 only.** Reject v1 and v2 descriptors before activation.
|
||||
4. **The application does not push curated content.** The repository curator works in a separate
|
||||
authoring clone.
|
||||
|
||||
---
|
||||
|
||||
## Part 2: use ThothII's repository tools
|
||||
|
||||
There are **two** tools: the **web application** (workspace management) and the **`tht` CLI**
|
||||
(preprocessing and operations). The complete installation is described in
|
||||
`docs/install/local-workspace-registry.md` (macOS/Windows/Linux) e
|
||||
`docs/install/server-workspace-registry.md`.
|
||||
|
||||
### 2.1 `tht`: main commands
|
||||
|
||||
Always invoke `tht` with `--installation <path>/thothii-installation.yaml`. The usual commands are:
|
||||
|
||||
```bash
|
||||
# 1) inspect workspace state (revision and identity)
|
||||
tht --installation <install> workspace inspect --workspace <id> --json
|
||||
|
||||
# 2) inspect the DWH (generates physical.yaml and LSH)
|
||||
tht --installation <install> workspace preprocess dwh --workspace <id> --json
|
||||
|
||||
# 3) suggest joins (FKs) from approved SQL
|
||||
tht --installation <install> workspace schema suggest-fks --workspace <id> --from-sql <query>.sql --output <candidati>.yaml --json
|
||||
|
||||
# 4) after review, publish curated FKs in Git and accept them
|
||||
tht --installation <install> workspace schema accept --workspace <id> --run <run-id> --yes --json
|
||||
|
||||
# 5) index the schema (Qdrant)
|
||||
tht --installation <install> workspace index-schema --workspace <id> --json
|
||||
|
||||
# 6) preprocess Evidence
|
||||
tht --installation <install> workspace preprocess evidence --workspace <id> --json
|
||||
|
||||
# 7) complete chain (DWH → FK → schema → Evidence)
|
||||
tht --installation <install> workspace preprocess run --workspace <id> --json
|
||||
|
||||
# 8) inspect or rebuild the Qdrant collection (maintenance only)
|
||||
tht --installation <install> workspace vector inspect --workspace <id> --json
|
||||
tht --installation <install> workspace vector rebuild --workspace <id> --collection <nome> --confirm <nome> --destroy
|
||||
```
|
||||
|
||||
Important notes:
|
||||
|
||||
- **`--json` writes JSON only to stdout** (machine contract); use it in scripts.
|
||||
- **`preprocess run` stops for human review** when it finds new proposed joins: it exits with
|
||||
`manual_review_required`. After review, continue with `schema accept ... --yes` and
|
||||
`preprocess run --resume <run-id>`.
|
||||
- **A filesystem Evidence file is materialized from the pinned Git commit** (there is no moving
|
||||
checkout); symlinks, unsafe paths, and trees that are too large are rejected.
|
||||
- **The CLI never writes to the repository** (it never pushes curated content).
|
||||
|
||||
### 2.2 Web application: workspace management
|
||||
|
||||
Workspace management has two distinct levels.
|
||||
|
||||
**Level 1: repository.** The first section shows that the workspace source lives in a separate
|
||||
directory, is published by the curator to a repository hosted on a Git server such as GitHub,
|
||||
GitLab, or Gitea, and is read by ThothII in read-only mode. It shows the host, repository, branch,
|
||||
active revision, and status of the last update.
|
||||
|
||||
* **Update workspace repository** does not require a workspace to be selected. The backend fetches
|
||||
or pulls the configured branch into ThothII's managed checkout, validates the entire candidate
|
||||
revision, and activates it atomically. If validation fails, it keeps the previous revision. It
|
||||
does not modify the remote source or save content from the GUI.
|
||||
* To create a local workspace, prepare a source directory with the catalog, `workspace.yaml`, and
|
||||
the expected subdirectories. Validate it, then commit and push from the authoring clone.
|
||||
ThothII provides no commands to create, edit, or publish the source.
|
||||
|
||||
**Level 2: selected workspace.** These commands are separate because they require a workspace to be selected first.
|
||||
These commands are separate because they require a workspace to be selected first.
|
||||
|
||||
- **Validate workspace source** checks the catalog, descriptor, Evidence, and invariants of the
|
||||
selected active revision again. It does not contact the DWH or modify files.
|
||||
- **Save entered secrets** replaces the entered values without displaying them. The fields depend
|
||||
on the declared DWH transport and Evidence authentication. The backend returns only configured
|
||||
or missing status.
|
||||
* **Forget stored value** removes the selected secret from the encrypted vault. Future sessions or
|
||||
operations that need it remain blocked until it is entered again.
|
||||
The remote Git repository and its credentials are installation settings. Runtime DWH and Evidence
|
||||
secrets persist in the backend's encrypted vault, not in the GUI's local storage. The GUI is only
|
||||
the interface: after submission it clears the field values and cannot read them back.
|
||||
|
||||
---
|
||||
|
||||
## Part 3: use the ThothII application
|
||||
|
||||
### 3.1 New session
|
||||
|
||||
Open the application and use **New session**. Enter only the **natural-language question**;
|
||||
workspace, model, and provider are already configured as global settings.
|
||||
|
||||
Example question:
|
||||
|
||||
> "List the electric bicycles completed in the last year, with model, frame number, and completion
|
||||
> date."
|
||||
|
||||
### 3.2 The eight-phase workflow and gates
|
||||
|
||||
The question passes through **eight phases**. You see the intermediate documents and decide at the important points:
|
||||
|
||||
1. **F1 clarification**: the model removes ambiguity when needed;
|
||||
2. **F2 Memory**: retrieves reusable Memory;
|
||||
3. **F3 rewriting**: rewrites and approves the question;
|
||||
4. **F4 schema linking**: proposes related tables and columns;
|
||||
5. **F5 summary**: summarizes the selected schema;
|
||||
6. **F6 CTE**: builds the CTEs;
|
||||
7. **F7 final SQL**: produces `sql_final.sql`;
|
||||
8. **F8 datamart**: execution or export (dbt, CSV, Excel).
|
||||
|
||||
**Review gates** appear as widgets: choose one option, select several items, or confirm an
|
||||
artifact or phase. The model *proposes* and the reviewer *decides*. The right side shows artifacts
|
||||
(schema linking, CTEs, and SQL); the Model activity panel shows the question and reasoning.
|
||||
|
||||
### 3.3 Sessions
|
||||
|
||||
Sessions appear in the sidebar with their ID, question, date, and author. A session
|
||||
**resumes** from its last incomplete phase by rebuilding state from documents saved on disk
|
||||
(`session_manifest.yaml`, phase artifacts, and `review_decisions.jsonl`). Saved state **is** the
|
||||
truth: what is not recorded did not happen.
|
||||
|
||||
---
|
||||
|
||||
## Complete example: ACME Limited
|
||||
|
||||
### Step 0: repository
|
||||
|
||||
Create the workspace Git repository, for example `tht-workspace-acme`:
|
||||
|
||||
```text
|
||||
thoth-workspaces.yaml # catalog containing acme-ebikes
|
||||
acme-ebikes/workspace.yaml # v3 descriptor (see §1.2)
|
||||
acme-ebikes/evidence/ # curated context .md documents
|
||||
acme-ebikes/schema/annotations.yaml # when curated joins exist
|
||||
```
|
||||
|
||||
Publish a new Git revision. In the installation, the application fetches and **activates** the
|
||||
workspace, validates schema v3, materializes Evidence from the pinned revision, and prepares the
|
||||
Qdrant collection (1024/cosine plus indexes).
|
||||
|
||||
### Step 1: preprocessing
|
||||
|
||||
```bash
|
||||
tht --installation ~/thothii-installation.yaml workspace preprocess dwh --workspace acme-ebikes --json
|
||||
tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --json
|
||||
```
|
||||
|
||||
If the run stops for joins (`manual_review_required`):
|
||||
|
||||
```bash
|
||||
# the curator reviews the candidates and publishes acme-ebikes/schema/annotations.yaml, then:
|
||||
tht --installation ~/thothii-installation.yaml workspace schema accept --workspace acme-ebikes --run RUN_ID --yes --json
|
||||
tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --resume RUN_ID --json
|
||||
```
|
||||
|
||||
### Step 2: the question
|
||||
|
||||
In the application, select the `acme-ebikes` workspace and create a session with the question.
|
||||
Follow the phases and confirm the gates. The model will propose schema linking (tables and columns
|
||||
from the `datawarehouse` DWH), CTEs, and finally the SQL, which you can view, copy, and run.
|
||||
|
||||
---
|
||||
|
||||
## Where to find technical details
|
||||
|
||||
For DWH REST access, the key is installation-specific and applies only to `rest_api`; `postgres_direct`
|
||||
and `ssh_tunnel` do not use it. See the [DWH server guide](install/dwh-auth-server.md), [client
|
||||
enrollment guide](install/dwh-auth-client-enrollment.md), and [TLS guide](install/dwh-auth-tls.md).
|
||||
|
||||
- CLI contract: `docs/contracts/workspace-preprocessing-cli.md`
|
||||
- `.tht-dwh` contract: `docs/contracts/tht-dwh.md`
|
||||
- Evidence v3: `docs/contracts/workspace-evidence-v3.md`
|
||||
# User guide: from question to validated SQL
|
||||
|
||||
This guide is for a reviewer using a configured ThothII installation. Installation, workspace
|
||||
publication, preprocessing, and database administration are separate paths; links to them are at
|
||||
the end of this page.
|
||||
|
||||
## Before creating a session
|
||||
|
||||
An administrator must have selected a workspace and configured the installation-wide provider,
|
||||
model, and thinking settings. The New session form deliberately asks only for the question.
|
||||
|
||||
The workspace is a pinned Git revision. A subsequent workspace update cannot alter a session
|
||||
already created from an earlier revision. If a workspace cannot reach its configured runtime DWH,
|
||||
new sessions are refused before any session state is written.
|
||||
|
||||
## Create and review a session
|
||||
|
||||
1. Sign in and select **New session**.
|
||||
2. Enter a precise business question, including the relevant time period and desired output. For
|
||||
example: “List patients discharged in the last 30 days, with ward and discharge date.”
|
||||
3. Review each gate and make the decision requested by the widget. A choice with a decision payload
|
||||
can persist immediately; a multi-choice widget records each selected decision; a confirmation
|
||||
widget approves an artifact or phase.
|
||||
4. Inspect the generated artifacts, especially schema linking, CTEs, and final SQL. The final SQL is
|
||||
available only after the F7 review gate.
|
||||
5. At F8, decide whether a datamart is requested. This is distinct from approving the SQL.
|
||||
|
||||
The workflow phases are fixed:
|
||||
|
||||
| Phase | What is reviewed |
|
||||
| --- | --- |
|
||||
| F1–F3 | clarification, reusable Memory, and the rewritten question |
|
||||
| F4–F5 | proposed tables, columns, Evidence, then their summary |
|
||||
| F6 | a CTE plan or an explicit skip |
|
||||
| F7 | `sql_final.sql` |
|
||||
| F8 | the datamart request or refusal |
|
||||
|
||||
## Resume, archive, and the meaning of saved state
|
||||
|
||||
The sidebar lists sessions and their current lifecycle. Resuming returns to the last incomplete
|
||||
phase. A finalized or archived session cannot be resumed.
|
||||
|
||||
The source of truth is the workspace session directory: `session_manifest.yaml`, phase artifacts,
|
||||
and `review_decisions.jsonl`. The visible activity stream is rebuilt from live SSE events and is
|
||||
not a transcript store. Therefore a decision or artifact that has not been persisted did not
|
||||
happen from the workflow’s point of view.
|
||||
|
||||
## Choose the correct neighbouring path
|
||||
|
||||
- To prepare, update, validate, or preprocess a workspace, use
|
||||
[Workspace operations](operations/workspaces.md).
|
||||
- To create a database configuration, refresh its physical schema, or generate catalog
|
||||
descriptions, use [Database management](operations/database-management.md).
|
||||
- To author material the workflow can retrieve, use [Evidence](evidence.md). A proposal from a
|
||||
session does not become Evidence automatically: a curator must review and publish it in Git.
|
||||
- For login and access recovery, use [local authentication](install/authentication-local.md) or
|
||||
[OIDC authentication](install/authentication-oidc.md).
|
||||
|
||||
+22
-13
@@ -1,20 +1,29 @@
|
||||
# ThothII documentation
|
||||
|
||||
ThothII is a human-in-the-loop datamart builder. It turns natural-language questions into validated SQL through an eight-phase workflow orchestrated by Pi.
|
||||
ThothII is a human-reviewed datamart builder. It turns a question into validated SQL through an
|
||||
eight-phase workflow: the model proposes; a reviewer makes the decisions that are persisted.
|
||||
|
||||
The documentation is divided into two areas:
|
||||
Start with the path that matches the work you need to do:
|
||||
|
||||
## ThothII technical documentation
|
||||
| I need to… | Start here |
|
||||
| --- | --- |
|
||||
| Install or operate one instance | [Install and first start](install/first-start.md) |
|
||||
| Add, update, or prepare a workspace | [Workspace operations](operations/workspaces.md) |
|
||||
| Ask a question and review the SQL workflow | [User guide](guida-utente.md) |
|
||||
| Configure and refresh an authoritative database catalog | [Database management](operations/database-management.md) |
|
||||
| Author and publish domain Evidence | [Evidence](evidence.md) |
|
||||
| Understand boundaries and persistence | [Architecture overview](architecture/overview.md) |
|
||||
|
||||
This section explains the system architecture, workflow, operating contracts, Evidence, and Memory. Start with the [architecture overview](architecture/overview.md).
|
||||
## How the documentation is organised
|
||||
|
||||
For local authentication, generic OIDC, and Authentik, see the [authentication documentation](architecture/authentication.md).
|
||||
- **Install and operate** documents host-side setup, authentication, lifecycle, workspaces, and
|
||||
Pi administration.
|
||||
- **Use ThothII** documents the two application paths: reviewed NL→SQL sessions and administrative
|
||||
database management.
|
||||
- **Architecture and contracts** explain why the system behaves as it does and define the
|
||||
machine-facing boundaries. Consult them when integrating or changing an implementation; they
|
||||
are not a substitute for an operator runbook.
|
||||
|
||||
To install the application in Docker across the four operating contexts, using the env file,
|
||||
`compose.yaml`, the local/server overlay, and the mounted secret bundle, see [Docker installation in the four operating contexts](installazione-docker-4-contesti.md).
|
||||
|
||||
For DWH REST with an installation-specific revocable key, see the [server guide](install/dwh-auth-server.md), [client enrollment guide](install/dwh-auth-client-enrollment.md), and [TLS guide](install/dwh-auth-tls.md). This component remains separate from the ThothII Compose stack.
|
||||
|
||||
## General topics
|
||||
|
||||
Operating and configuration notes that are not specific to the ThothII domain, such as how Pi resolves built-in, user-level, and project-level models. Start with [Pi model configuration](general/pi-configuration.md).
|
||||
The host-side `tht` command is the operator interface. The Python `tht` inside `harness/` is a
|
||||
different, internal workflow CLI invoked by the core. In examples, use an absolute installation
|
||||
descriptor path whenever discovery is not unambiguous.
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
# Install and first start
|
||||
|
||||
This is the supported local installation path. It creates an installation-local configuration and
|
||||
starts the Compose stack; it does not create a workspace repository or a database catalog entry.
|
||||
|
||||
## Prerequisites and boundaries
|
||||
|
||||
Install Docker Engine with Compose v2, plus the host `tht` command. On macOS or Linux, install the
|
||||
host command from the repository with `./scripts/install-tht.sh`; Windows uses
|
||||
`./scripts/install-tht.ps1`. The installer builds or verifies the native command and checks that
|
||||
`tht` is resolvable on `PATH`.
|
||||
|
||||
The stack contains `frontend`, `core`, `catalog-db`, `qdrant`, `embedding`, and the one-shot
|
||||
`embedding-model-init` and `catalog-migrate` services. DWH and model-provider endpoints are
|
||||
external installation settings. Pi runs inside `core`; do not install a host Pi executable for
|
||||
the application runtime.
|
||||
|
||||
Secrets, certificates, Pi authentication, and workspace endpoint bindings are protected
|
||||
installation-local files. Never put them in a workspace descriptor, an env file intended for
|
||||
version control, a URL, or a command line.
|
||||
|
||||
## Create the local installation
|
||||
|
||||
From the repository root, start the interactive setup and select the local profile:
|
||||
|
||||
```sh
|
||||
tht setup --profile local
|
||||
```
|
||||
|
||||
It writes the selected non-secret descriptor below `deploy/<installation-id>/`, the associated
|
||||
operator env file, and can create protected secret templates. Keep the descriptor path: pass it
|
||||
to commands as `--installation /absolute/path/thothii-installation.yaml` when more than one
|
||||
installation can be discovered.
|
||||
|
||||
If the descriptor is prepared manually instead, begin with
|
||||
[`thothii-installation.local.yaml`](examples/thothii-installation.local.yaml), set mode `0600` or
|
||||
`0400`, and ensure `THT_INSTALLATION_CONFIG_SOURCE` in the selected env file points to that exact
|
||||
file. Create the secret bundle from `deploy/secrets/thothii.secrets.example`, protect it, and set
|
||||
the file locations and external endpoints in the env file. The required settings include:
|
||||
|
||||
- `PI_AUTH_FILE`, `THT_SECRETS_FILE`, and the catalog password source files;
|
||||
- `THT_INSTALLATION_CONFIG_SOURCE` and the workspace Git remote/branch;
|
||||
- the DWH and model-provider endpoints; and
|
||||
- `THT_AUTH_CONFIG_ROOT` for local authentication or the OIDC configuration selected during setup.
|
||||
|
||||
For the supported secret names and the metadata-generation model credential boundary, see the
|
||||
`deploy/secrets/README.md` file in the installation checkout. It is intentionally not published
|
||||
as a documentation page because it describes a protected local-file contract.
|
||||
|
||||
## Start and verify
|
||||
|
||||
For the normal local path, use the launcher:
|
||||
|
||||
```sh
|
||||
./scripts/run-stack.sh
|
||||
```
|
||||
|
||||
It builds `core`, starts `catalog-db`, runs `catalog-migrate`, then keeps the base plus local
|
||||
Compose profile in the foreground. Database migrations are deliberately not a hidden backend
|
||||
startup action. Open `http://127.0.0.1:8080` unless `THOTH_HTTP_PORT` was changed.
|
||||
|
||||
In another terminal, verify the installation without changing it:
|
||||
|
||||
```sh
|
||||
tht --installation /absolute/path/thothii-installation.yaml doctor --json
|
||||
tht --installation /absolute/path/thothii-installation.yaml status
|
||||
```
|
||||
|
||||
`/health` verifies application-process readiness; `tht doctor` is the diagnostic surface for
|
||||
Compose, configuration, workspace, workflow, and Pi prerequisites.
|
||||
|
||||
## Routine lifecycle and next steps
|
||||
|
||||
Use `tht start [--build]`, `tht stop`, `tht logs`, and `tht doctor` rather than composing ad-hoc
|
||||
container commands. Named volumes retain settings, Pi state, workspace registry, sessions,
|
||||
Qdrant data, and embedding models across `docker compose down`; removing them requires the
|
||||
explicit destructive `--volumes` form.
|
||||
|
||||
After the stack is healthy, configure authentication if setup did not do so, then continue with
|
||||
[Workspace operations](../operations/workspaces.md). For server profile, reverse proxy, backups,
|
||||
and recovery, use the deployment program and its manual gates; the server profile is not a
|
||||
drop-in replacement for the local command above.
|
||||
@@ -4,9 +4,11 @@ ThothII uses one Compose topology:
|
||||
|
||||
- `frontend`
|
||||
- `core`
|
||||
- `catalog-db`
|
||||
- `qdrant`
|
||||
- `embedding`
|
||||
- `embedding-model-init`
|
||||
- `catalog-migrate` (one-shot)
|
||||
|
||||
Qdrant and Ollama embedding are required internal Compose services. Only DWH and the LLM remain
|
||||
external. The fixed model is `qwen3-embedding:0.6b` with 1024 dimensions and cosine distance;
|
||||
@@ -23,7 +25,7 @@ flowchart TB
|
||||
SERVER --> BUNDLE
|
||||
SESSION --> BUNDLE
|
||||
AUTH --> BUNDLE
|
||||
BUNDLE --> SERVICES["Frontend, core, vector, embedding"]
|
||||
BUNDLE --> SERVICES["Frontend, core, catalog DB, vector, embedding"]
|
||||
```
|
||||
|
||||
## Short ownership contract
|
||||
@@ -35,7 +37,14 @@ flowchart TB
|
||||
| Qdrant | Internal | Required internal Compose service with persistent `qdrant-data` volume. |
|
||||
| Ollama embedding | Internal | Required internal Compose service for `qwen3-embedding:0.6b`. |
|
||||
|
||||
## Comando standard locale
|
||||
## Local path: setup, migration, start
|
||||
|
||||
The normal local path is [Install and first start](install/first-start.md). It uses `tht setup`
|
||||
to create the selected installation-local descriptor and runs the catalog migration explicitly
|
||||
before application startup.
|
||||
|
||||
If an operator intentionally prepares the descriptor and protected files by hand, the equivalent
|
||||
foreground launch is:
|
||||
|
||||
```sh
|
||||
cp deploy/env/local.env.example deploy/env/local.env
|
||||
@@ -46,8 +55,7 @@ chmod 600 deploy/secrets/thothii.secrets
|
||||
# replace every placeholder, chmod it 600, then set that exact path as
|
||||
# THT_INSTALLATION_CONFIG_SOURCE in deploy/env/local.env.
|
||||
|
||||
docker compose --env-file deploy/env/local.env \
|
||||
-f compose.yaml -f deploy/compose.local.yaml up --build -d
|
||||
./scripts/run-stack.sh
|
||||
```
|
||||
|
||||
Set these values in `deploy/env/local.env`:
|
||||
@@ -90,8 +98,10 @@ A private PEM CA remains outside the bundle and must be mounted through a review
|
||||
Preprocessing runs through the native host CLI and the installation descriptor:
|
||||
|
||||
```sh
|
||||
tht --installation /percorso/assoluto/thothii-installation.yaml workspace preprocess evidence
|
||||
tht --installation /percorso/assoluto/thothii-installation.yaml workspace preprocess dwh
|
||||
tht --installation /percorso/assoluto/thothii-installation.yaml \
|
||||
workspace preprocess evidence --workspace <workspace-id>
|
||||
tht --installation /percorso/assoluto/thothii-installation.yaml \
|
||||
workspace preprocess dwh --workspace <workspace-id>
|
||||
```
|
||||
|
||||
The CLI runs the profile-gated `workspace-maintenance` service. See the
|
||||
@@ -108,7 +118,6 @@ docker compose --env-file deploy/env/server.env \
|
||||
-f deploy/compose.session-server.yaml.example up --build -d
|
||||
```
|
||||
|
||||
Also see:
|
||||
|
||||
- `docs/install/local-workspace-registry.md`
|
||||
- `docs/install/server-workspace-registry.md`
|
||||
Also see [Workspace operations](operations/workspaces.md). Server deployment, reverse-proxy,
|
||||
backup, and recovery remain manual-gated operations; do not treat the local profile as a server
|
||||
replacement.
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
# Database management
|
||||
|
||||
Database Management is an administrative catalog for an external PostgreSQL schema. It is separate
|
||||
from workspace preprocessing and, today, does not change the DWH binding used by the NL→SQL
|
||||
session workflow.
|
||||
|
||||
## What the catalog owns
|
||||
|
||||
For each YAML workspace, an administrator may create at most one database configuration. It holds
|
||||
the database name, schema, connection binding, write-only encrypted secrets, observed physical
|
||||
schema, optional curated descriptions, generated descriptions, and durable operation history.
|
||||
|
||||
It does **not** become the external source of truth. Tables, columns, types, defaults,
|
||||
nullability, primary-key positions, and ordered foreign-key pairs are observations of the source
|
||||
schema and cannot be manually created, renamed, or structurally edited. Descriptions are the
|
||||
editable metadata.
|
||||
|
||||
## Configure and test a database
|
||||
|
||||
1. Open **Database Management** and choose a workspace.
|
||||
2. Create its PostgreSQL configuration. Choose `postgres_direct`, `rest_api`, or `ssh_tunnel` and
|
||||
complete the binding fields that the chosen transport requires.
|
||||
3. Enter secrets only when replacing them. They remain write-only and are never returned by the
|
||||
application.
|
||||
4. Run **Test connection** before any synchronization.
|
||||
|
||||
SSH uses a private key, optional key passphrase, mandatory `known_hosts`, and optional PostgreSQL
|
||||
TLS CA/server name. REST prefers `POST /rpc/schema_snapshot`; when it is absent, the catalog may
|
||||
use the same strict v1 snapshot through one read-only `POST /rpc/run_query`. An unavailable
|
||||
capability, malformed snapshot, or connector error applies no catalog changes. See the
|
||||
[schema snapshot contract](../contracts/catalog-schema-snapshot.md).
|
||||
|
||||
## Synchronize authoritative schema metadata
|
||||
|
||||
Start a synchronization from a database or a selected table set. The available scopes are tables,
|
||||
columns, relationships, and all. One database can have only one active catalog operation at a
|
||||
time; cleanup shares this exclusion.
|
||||
|
||||
The run scans first and publishes a durable operation. If it detects a destructive difference, it
|
||||
requires confirmation and re-scans before applying. You can cancel before apply; completed and
|
||||
failed runs remain in history. The live log is delivered over SSE with a polling fallback.
|
||||
|
||||
Explicit cleanup is different from source synchronization: administrators can clear selected
|
||||
table/relationship or column/relationship catalog metadata without changing the external source,
|
||||
the connection binding, or secrets. Deleting a table cascades to its columns and relationships.
|
||||
|
||||
## Generate and consolidate descriptions
|
||||
|
||||
Generated descriptions can be requested for selected tables, selected columns, every eligible
|
||||
target, or targets with a missing generated description. The backend accepts one installation-wide
|
||||
run and processes targets sequentially. It reads at most five source rows and five representative
|
||||
non-null values per relevant source through a read-only connector, then sends that bounded sample
|
||||
transiently to the configured model provider.
|
||||
|
||||
Each successful result is persisted immediately. Stop terminates the active helper but retains
|
||||
earlier results. A helper has at most one provider retry; three consecutively exhausted technical
|
||||
batches fail the run. Stale queued/running work is marked interrupted at startup and can be
|
||||
unlocked only when no local worker/helper is live. There is no automatic resume and no public
|
||||
description-generation CLI.
|
||||
|
||||
Review generated text before copying it into the curated **Description** field. The sampling rule
|
||||
is a deliberate data-disclosure boundary: do not use this facility for fields whose values must
|
||||
not be sent to the configured provider until a Sensitive Data Policy is in place.
|
||||
|
||||
The decisions behind this surface are [ADRs 0001–0010](../adr/0001-postgres-metadata-catalog.md)
|
||||
and the detailed acceptance record is
|
||||
[AI catalog description generation acceptance](../testing/2026-08-29-ai-catalog-description-generation-acceptance.md).
|
||||
@@ -0,0 +1,65 @@
|
||||
# Workspace operations
|
||||
|
||||
A workspace is curator-owned Git content plus installation-local runtime bindings. It is the
|
||||
boundary between what can be published and what can be used by an installation.
|
||||
|
||||
## Roles and ownership
|
||||
|
||||
| Role | Owns | Does not own |
|
||||
| --- | --- | --- |
|
||||
| Curator | `thoth-workspaces.yaml`, `<id>/workspace.yaml`, Evidence, and curated schema annotations | installation secrets or active runtime bindings |
|
||||
| Installation operator | Git source, selected workspace, write-only runtime secrets, validation, connectivity, and preprocessing | commits or pushes to the workspace repository |
|
||||
| Reviewer | NL→SQL decisions in a pinned session | workspace publication or preprocessing |
|
||||
|
||||
The root catalog has `schema_version: 1` and an ordered list of workspace identities. Each entry
|
||||
must have a matching schema-v3 descriptor at `<id>/workspace.yaml` in the same Git commit. The
|
||||
application validates a complete candidate revision and activates it atomically; invalid content
|
||||
leaves the preceding active revision in place.
|
||||
|
||||
## Operator sequence
|
||||
|
||||
1. Curate and push a complete repository revision. Do not put DWH passwords, API keys, private
|
||||
keys, or signed URLs in this repository.
|
||||
2. In the application, update the workspace repository. This fetches and validates the candidate;
|
||||
it never edits the remote repository.
|
||||
3. Select the workspace. Supply or replace its write-only runtime secrets, then run **Validate
|
||||
workspace source** and **Test workspace connections**.
|
||||
4. Select it as the installation workspace before creating sessions.
|
||||
5. Use the host CLI for preprocessing. It dispatches a profile-gated maintenance service and
|
||||
returns a single structured result; `--json` keeps stdout machine-readable.
|
||||
|
||||
```sh
|
||||
INSTALLATION=/absolute/path/thothii-installation.yaml
|
||||
WORKSPACE=example-workspace
|
||||
|
||||
tht --installation "$INSTALLATION" workspace inspect --workspace "$WORKSPACE" --json
|
||||
tht --installation "$INSTALLATION" workspace preprocess dwh --workspace "$WORKSPACE" --json
|
||||
tht --installation "$INSTALLATION" workspace preprocess evidence --workspace "$WORKSPACE" --json
|
||||
```
|
||||
|
||||
For the full DWH → review → schema-index → Evidence chain, run
|
||||
`workspace preprocess run`. It may stop with `manual_review_required` when FK candidates need a
|
||||
curator decision. Publish the reviewed annotations, update the repository, then accept that exact
|
||||
run and resume it:
|
||||
|
||||
```sh
|
||||
tht --installation "$INSTALLATION" workspace schema accept \
|
||||
--workspace "$WORKSPACE" --run <run-id> --yes --json
|
||||
tht --installation "$INSTALLATION" workspace preprocess run \
|
||||
--workspace "$WORKSPACE" --resume <run-id> --json
|
||||
```
|
||||
|
||||
The contract gives exact validation, exit code, and JSON rules in
|
||||
[Workspace preprocessing CLI](../contracts/workspace-preprocessing-cli.md). For Evidence source
|
||||
forms and the schema-v3 descriptor contract, see
|
||||
[Workspace Evidence v3](../contracts/workspace-evidence-v3.md).
|
||||
|
||||
## Transport and revision rules
|
||||
|
||||
Runtime sessions support direct PostgreSQL and REST bindings. SSH tunnel bindings are diagnostic
|
||||
only for this path, so they cannot admit an NL→SQL session. Database Management has its own
|
||||
strict known-host SSH path for connection tests and schema synchronization.
|
||||
|
||||
Every new session pins the active Git revision. Snapshot cleanup retains revisions still
|
||||
referenced by unarchived sessions. A later pull can prepare a future session but cannot alter a
|
||||
resume.
|
||||
+40
-19
@@ -44,27 +44,48 @@ markdown_extensions:
|
||||
alternate_style: true
|
||||
nav:
|
||||
- Home: index.md
|
||||
- Install and operate:
|
||||
- Install and first start: install/first-start.md
|
||||
- Local authentication: install/authentication-local.md
|
||||
- Generic OIDC: install/authentication-oidc.md
|
||||
- Authentik: install/authentik.md
|
||||
- Workspace operations: operations/workspaces.md
|
||||
- Pi model configuration: general/pi-configuration.md
|
||||
- Docker installation contexts: installazione-docker-4-contesti.md
|
||||
- Use ThothII:
|
||||
- User guide: guida-utente.md
|
||||
- DWH REST per installation:
|
||||
- dwh-auth server: install/dwh-auth-server.md
|
||||
- DWH client enrollment: install/dwh-auth-client-enrollment.md
|
||||
- DWH REST TLS: install/dwh-auth-tls.md
|
||||
- Contracts and CLI:
|
||||
- Database management: operations/database-management.md
|
||||
- Evidence authoring and publication: evidence.md
|
||||
- Workflow references:
|
||||
- Initial disambiguation: disambiguazione-iniziale.md
|
||||
- Memory management: gestione-memory.md
|
||||
- Operating skills: skills.md
|
||||
- Architecture:
|
||||
- Overview: architecture/overview.md
|
||||
- Components, modules, and flows: architecture/components.md
|
||||
- Authentication and authorization: architecture/authentication.md
|
||||
- Contracts and integration:
|
||||
- Catalog schema snapshot RPC: contracts/catalog-schema-snapshot.md
|
||||
- Workspace preprocessing CLI: contracts/workspace-preprocessing-cli.md
|
||||
- tht–DWH contract: contracts/tht-dwh.md
|
||||
- Workspace Evidence v3 contract: contracts/workspace-evidence-v3.md
|
||||
- ThothII technical documentation:
|
||||
- Architecture overview: architecture/overview.md
|
||||
- Components, modules, and flows: architecture/components.md
|
||||
- Evidence: evidence.md
|
||||
- DWH REST server: install/dwh-auth-server.md
|
||||
- DWH client enrollment: install/dwh-auth-client-enrollment.md
|
||||
- DWH REST TLS: install/dwh-auth-tls.md
|
||||
- Decisions and acceptance:
|
||||
- Architecture decisions:
|
||||
- 0001 Metadata catalog: adr/0001-postgres-metadata-catalog.md
|
||||
- 0002 Workspace database secrets: adr/0002-workspace-database-secret-references.md
|
||||
- 0003 Installation-local bindings: adr/0003-installation-local-database-bindings.md
|
||||
- 0004 Fastify/Kysely catalog: adr/0004-fastify-kysely-metadata-catalog.md
|
||||
- 0005 Table deletion during synchronization: adr/0005-hard-delete-catalog-tables-during-synchronization.md
|
||||
- 0006 Physical and logical relationships: adr/0006-separate-physical-and-logical-relationships.md
|
||||
- 0007 Authoritative schema synchronization: adr/0007-durable-authoritative-schema-synchronization.md
|
||||
- 0008 Manual catalog cleanup: adr/0008-allow-manual-catalog-metadata-cleanup.md
|
||||
- 0009 Sequential description generation: adr/0009-use-one-sequential-description-generation-run.md
|
||||
- 0010 Bounded source samples: adr/0010-allow-bounded-real-source-samples-for-description-generation.md
|
||||
- AI catalog description acceptance: testing/2026-08-29-ai-catalog-description-generation-acceptance.md
|
||||
- Authentication: architecture/authentication.md
|
||||
- Local authentication installation: install/authentication-local.md
|
||||
- Generic OIDC: install/authentication-oidc.md
|
||||
- Authentik: install/authentik.md
|
||||
- Docker installation (4 contexts): installazione-docker-4-contesti.md
|
||||
- Memory management: gestione-memory.md
|
||||
- Operating skills: skills.md
|
||||
- Initial disambiguation: disambiguazione-iniziale.md
|
||||
- General topics:
|
||||
- Pi model configuration: general/pi-configuration.md
|
||||
- Design records:
|
||||
- Metadata catalog design: plans/2026-08-26-metadata-catalog-from-thothai.md
|
||||
- Description generation design: plans/2026-08-28-ai-catalog-description-generation.md
|
||||
- Description generation specification: plans/2026-08-28-ai-catalog-description-generation-spec.md
|
||||
|
||||
Reference in New Issue
Block a user