mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: reorganize operational documentation

Browse Source
This commit is contained in:
Codex
2026-08-29 20:08:48 +02:00
parent d504b1def1
commit 0ce05869cf
10 changed files with 371 additions and 337 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/guida-utente.md
+57 -282
View File
@@ -1,282 +1,57 @@
# ThothII user guide
For login, **Remember me**, roles, session invalidation, OIDC groups, and recovery, see the
[local authentication guide](install/authentication-local.md) and the [generic OIDC guide](install/authentication-oidc.md).
This guide walks you through **preparing** the workspace repository, **using ThothII's tools** for
that repository, and **using the application** to ask natural-language questions and obtain
validated SQL. It uses plain language and examples. The contracts listed at the end contain the
technical details.
> **What is ThothII?** It is a *datamart builder* with human review. You write a natural-language
> question, the model proposes each step in turn (clarifications, schema, CTEs, and SQL), and a
> **human reviewer decides** at every important step. The final result is validated SQL ready to
> run on the data warehouse.
---
## Part 1: prepare the workspace repository on Git
### 1.1 Structure
The workspace repository is a **Git repository** that describes *which data* is available and
*how to reach it*. It contains neither the data nor **secrets** such as passwords, tokens, or certificates.
A valid repository contains:
```text
thoth-workspaces.yaml # catalog: list of workspaces
<workspace-id>/workspace.yaml # workspace descriptor (schema v3)
<workspace-id>/evidence/ # optional context documents, such as *.md
<workspace-id>/schema/annotations.yaml # optional manually curated logical joins (P5)
```
- The **catalog** `thoth-workspaces.yaml` is a simple list:
```yaml
schema_version: 1
workspaces:
- id: acme-ebikes
name: ACME Limited
description: DWH for electric bicycle production
```
- The **ID** must be lowercase, contain no spaces, and follow `acme-ebikes` (`[a-z][a-z0-9-]{2,62}`).
- The **descriptor** `<id>/workspace.yaml` uses schema v3. It is the only valid description.
### 1.2 Descriptor example (ACME Limited)
```yaml
workspace:
schema_version: 3
id: acme-ebikes
name: ACME Limited
description: Industrial DWH for electric bicycle production
language: en # descriptions and Evidence are in English
dwh:
engine: postgres
database: postgres
schema: datawarehouse
supported_transports: [rest_api] # access through the REST API (PostgREST)
semantic_index:
vector_store:
engine: qdrant
collection: acme-ebikes
dimensions: 1024
distance: cosine
embedding:
provider: ollama_internal
model: qwen3-embedding:0.6b
dimensions: 1024
llm_policy:
allowed: [zai/glm-5.2]
diagnostics:
dwh_rest:
method: POST
path: /rpc/ping
auth: x-api-key
response: { database: database, schema: schema }
evidence:
source:
type: filesystem
uri: acme-ebikes/evidence # path inside the repository
policy:
max_chunk_chars: 4000
retain_published_generations: 3
```
Changes from older workspaces:
- NL→SQL sessions reach the database only through **REST** or **direct Postgres** (`rest_api` /
`postgres_direct`); `ssh_tunnel` remains disabled for session runtime. The separate Database
management surface supports SSH for **Test connection** and **Sync tables**, with a private key,
mandatory `known_hosts`, and an optional key passphrase;
- the semantic index is **internal** (Qdrant plus `qwen3-embedding:0.6b`, 1024 dimensions, cosine);
- **filesystem** Evidence lives in the repository (`<id>/evidence`) and is materialized from the
pinned Git commit (P6). HTTP Evidence is also supported.
### 1.3 Rules
1. **Git is the source of truth.** Change the descriptor, catalog, and Evidence only through a
*commit* and *push*, followed by an installation *pull*.
2. **No secrets in the repository.** Add passwords, tokens, private keys, and signed URLs at
runtime through Workspace management; the backend stores them encrypted.
3. **Schema v3 only.** Reject v1 and v2 descriptors before activation.
4. **The application does not push curated content.** The repository curator works in a separate
authoring clone.
---
## Part 2: use ThothII's repository tools
There are **two** tools: the **web application** (workspace management) and the **`tht` CLI**
(preprocessing and operations). The complete installation is described in
`docs/install/local-workspace-registry.md` (macOS/Windows/Linux) e
`docs/install/server-workspace-registry.md`.
### 2.1 `tht`: main commands
Always invoke `tht` with `--installation <path>/thothii-installation.yaml`. The usual commands are:
```bash
# 1) inspect workspace state (revision and identity)
tht --installation <install> workspace inspect --workspace <id> --json
# 2) inspect the DWH (generates physical.yaml and LSH)
tht --installation <install> workspace preprocess dwh --workspace <id> --json
# 3) suggest joins (FKs) from approved SQL
tht --installation <install> workspace schema suggest-fks --workspace <id> --from-sql <query>.sql --output <candidati>.yaml --json
# 4) after review, publish curated FKs in Git and accept them
tht --installation <install> workspace schema accept --workspace <id> --run <run-id> --yes --json
# 5) index the schema (Qdrant)
tht --installation <install> workspace index-schema --workspace <id> --json
# 6) preprocess Evidence
tht --installation <install> workspace preprocess evidence --workspace <id> --json
# 7) complete chain (DWH → FK → schema → Evidence)
tht --installation <install> workspace preprocess run --workspace <id> --json
# 8) inspect or rebuild the Qdrant collection (maintenance only)
tht --installation <install> workspace vector inspect --workspace <id> --json
tht --installation <install> workspace vector rebuild --workspace <id> --collection <nome> --confirm <nome> --destroy
```
Important notes:
- **`--json` writes JSON only to stdout** (machine contract); use it in scripts.
- **`preprocess run` stops for human review** when it finds new proposed joins: it exits with
`manual_review_required`. After review, continue with `schema accept ... --yes` and
`preprocess run --resume <run-id>`.
- **A filesystem Evidence file is materialized from the pinned Git commit** (there is no moving
checkout); symlinks, unsafe paths, and trees that are too large are rejected.
- **The CLI never writes to the repository** (it never pushes curated content).
### 2.2 Web application: workspace management
Workspace management has two distinct levels.
**Level 1: repository.** The first section shows that the workspace source lives in a separate
directory, is published by the curator to a repository hosted on a Git server such as GitHub,
GitLab, or Gitea, and is read by ThothII in read-only mode. It shows the host, repository, branch,
active revision, and status of the last update.
* **Update workspace repository** does not require a workspace to be selected. The backend fetches
or pulls the configured branch into ThothII's managed checkout, validates the entire candidate
revision, and activates it atomically. If validation fails, it keeps the previous revision. It
does not modify the remote source or save content from the GUI.
* To create a local workspace, prepare a source directory with the catalog, `workspace.yaml`, and
the expected subdirectories. Validate it, then commit and push from the authoring clone.
ThothII provides no commands to create, edit, or publish the source.
**Level 2: selected workspace.** These commands are separate because they require a workspace to be selected first.
These commands are separate because they require a workspace to be selected first.
- **Validate workspace source** checks the catalog, descriptor, Evidence, and invariants of the
selected active revision again. It does not contact the DWH or modify files.
- **Save entered secrets** replaces the entered values without displaying them. The fields depend
on the declared DWH transport and Evidence authentication. The backend returns only configured
or missing status.
* **Forget stored value** removes the selected secret from the encrypted vault. Future sessions or
operations that need it remain blocked until it is entered again.
The remote Git repository and its credentials are installation settings. Runtime DWH and Evidence
secrets persist in the backend's encrypted vault, not in the GUI's local storage. The GUI is only
the interface: after submission it clears the field values and cannot read them back.
---
## Part 3: use the ThothII application
### 3.1 New session
Open the application and use **New session**. Enter only the **natural-language question**;
workspace, model, and provider are already configured as global settings.
Example question:
> "List the electric bicycles completed in the last year, with model, frame number, and completion
> date."
### 3.2 The eight-phase workflow and gates
The question passes through **eight phases**. You see the intermediate documents and decide at the important points:
1. **F1 clarification**: the model removes ambiguity when needed;
2. **F2 Memory**: retrieves reusable Memory;
3. **F3 rewriting**: rewrites and approves the question;
4. **F4 schema linking**: proposes related tables and columns;
5. **F5 summary**: summarizes the selected schema;
6. **F6 CTE**: builds the CTEs;
7. **F7 final SQL**: produces `sql_final.sql`;
8. **F8 datamart**: execution or export (dbt, CSV, Excel).
**Review gates** appear as widgets: choose one option, select several items, or confirm an
artifact or phase. The model *proposes* and the reviewer *decides*. The right side shows artifacts
(schema linking, CTEs, and SQL); the Model activity panel shows the question and reasoning.
### 3.3 Sessions
Sessions appear in the sidebar with their ID, question, date, and author. A session
**resumes** from its last incomplete phase by rebuilding state from documents saved on disk
(`session_manifest.yaml`, phase artifacts, and `review_decisions.jsonl`). Saved state **is** the
truth: what is not recorded did not happen.
---
## Complete example: ACME Limited
### Step 0: repository
Create the workspace Git repository, for example `tht-workspace-acme`:
```text
thoth-workspaces.yaml # catalog containing acme-ebikes
acme-ebikes/workspace.yaml # v3 descriptor (see §1.2)
acme-ebikes/evidence/ # curated context .md documents
acme-ebikes/schema/annotations.yaml # when curated joins exist
```
Publish a new Git revision. In the installation, the application fetches and **activates** the
workspace, validates schema v3, materializes Evidence from the pinned revision, and prepares the
Qdrant collection (1024/cosine plus indexes).
### Step 1: preprocessing
```bash
tht --installation ~/thothii-installation.yaml workspace preprocess dwh --workspace acme-ebikes --json
tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --json
```
If the run stops for joins (`manual_review_required`):
```bash
# the curator reviews the candidates and publishes acme-ebikes/schema/annotations.yaml, then:
tht --installation ~/thothii-installation.yaml workspace schema accept --workspace acme-ebikes --run RUN_ID --yes --json
tht --installation ~/thothii-installation.yaml workspace preprocess run --workspace acme-ebikes --resume RUN_ID --json
```
### Step 2: the question
In the application, select the `acme-ebikes` workspace and create a session with the question.
Follow the phases and confirm the gates. The model will propose schema linking (tables and columns
from the `datawarehouse` DWH), CTEs, and finally the SQL, which you can view, copy, and run.
---
## Where to find technical details
For DWH REST access, the key is installation-specific and applies only to `rest_api`; `postgres_direct`
and `ssh_tunnel` do not use it. See the [DWH server guide](install/dwh-auth-server.md), [client
enrollment guide](install/dwh-auth-client-enrollment.md), and [TLS guide](install/dwh-auth-tls.md).
- CLI contract: `docs/contracts/workspace-preprocessing-cli.md`
- `.tht-dwh` contract: `docs/contracts/tht-dwh.md`
- Evidence v3: `docs/contracts/workspace-evidence-v3.md`
# User guide: from question to validated SQL
This guide is for a reviewer using a configured ThothII installation. Installation, workspace
publication, preprocessing, and database administration are separate paths; links to them are at
the end of this page.
## Before creating a session
An administrator must have selected a workspace and configured the installation-wide provider,
model, and thinking settings. The New session form deliberately asks only for the question.
The workspace is a pinned Git revision. A subsequent workspace update cannot alter a session
already created from an earlier revision. If a workspace cannot reach its configured runtime DWH,
new sessions are refused before any session state is written.
## Create and review a session
1. Sign in and select **New session**.
2. Enter a precise business question, including the relevant time period and desired output. For
example: “List patients discharged in the last 30 days, with ward and discharge date.”
3. Review each gate and make the decision requested by the widget. A choice with a decision payload
can persist immediately; a multi-choice widget records each selected decision; a confirmation
widget approves an artifact or phase.
4. Inspect the generated artifacts, especially schema linking, CTEs, and final SQL. The final SQL is
available only after the F7 review gate.
5. At F8, decide whether a datamart is requested. This is distinct from approving the SQL.
The workflow phases are fixed:
| Phase | What is reviewed |
| --- | --- |
| F1–F3 | clarification, reusable Memory, and the rewritten question |
| F4–F5 | proposed tables, columns, Evidence, then their summary |
| F6 | a CTE plan or an explicit skip |
| F7 | `sql_final.sql` |
| F8 | the datamart request or refusal |
## Resume, archive, and the meaning of saved state
The sidebar lists sessions and their current lifecycle. Resuming returns to the last incomplete
phase. A finalized or archived session cannot be resumed.
The source of truth is the workspace session directory: `session_manifest.yaml`, phase artifacts,
and `review_decisions.jsonl`. The visible activity stream is rebuilt from live SSE events and is
not a transcript store. Therefore a decision or artifact that has not been persisted did not
happen from the workflow’s point of view.
## Choose the correct neighbouring path
- To prepare, update, validate, or preprocess a workspace, use
[Workspace operations](operations/workspaces.md).
- To create a database configuration, refresh its physical schema, or generate catalog
descriptions, use [Database management](operations/database-management.md).
- To author material the workflow can retrieve, use [Evidence](evidence.md). A proposal from a
session does not become Evidence automatically: a curator must review and publish it in Git.
- For login and access recovery, use [local authentication](install/authentication-local.md) or
[OIDC authentication](install/authentication-oidc.md).