ThothII/docs/guida-utente.md

# ThothII user guide

For login, **Remember me**, roles, session invalidation, OIDC groups, and recovery, see the
[local authentication guide](install/authentication-local.md) and the [generic OIDC guide](install/authentication-oidc.md).

This guide walks you through **preparing** the workspace repository, **using ThothII's tools** for
that repository, and **using the application** to ask natural-language questions and obtain
validated SQL. It uses plain language and examples. The contracts listed at the end contain the
technical details.

> **What is ThothII?** It is a *datamart builder* with human review. You write a natural-language
> question, the model proposes each step in turn (clarifications, schema, CTEs, and SQL), and a
> **human reviewer decides** at every important step. The final result is validated SQL ready to
> run on the data warehouse.

---

## Part 1: prepare the workspace repository on Git

### 1.1 Structure

The workspace repository is a **Git repository** that describes *which data* is available and
*how to reach it*. It contains neither the data nor **secrets** such as passwords, tokens, or certificates.

A valid repository contains:

```text
thoth-workspaces.yaml              # catalog: list of workspaces
<workspace-id>/workspace.yaml      # workspace descriptor (schema v3)
<workspace-id>/evidence/            # optional context documents, such as *.md
<workspace-id>/schema/annotations.yaml   # optional manually curated logical joins (P5)
```

- The **catalog** `thoth-workspaces.yaml` is a simple list:

```yaml
schema_version: 1
workspaces:
  - id: acme-ebikes
    name: ACME Limited
    description: DWH for electric bicycle production
```

- The **ID** must be lowercase, contain no spaces, and follow `acme-ebikes` (`[a-z][a-z0-9-]{2,62}`).
- The **descriptor** `<id>/workspace.yaml` uses schema v3. It is the only valid description.

### 1.2 Descriptor example (ACME Limited)

```yaml
workspace:
  schema_version: 3
  id: acme-ebikes
  name: ACME Limited
  description: Industrial DWH for electric bicycle production
  language: en                      # descriptions and Evidence are in English

dwh:
  engine: postgres
  database: postgres
  schema: datawarehouse
    supported_transports: [rest_api]  # access through the REST API (PostgREST)

semantic_index:
  vector_store:
    engine: qdrant
    collection: acme-ebikes
    dimensions: 1024
    distance: cosine
  embedding:
    provider: ollama_internal
    model: qwen3-embedding:0.6b
    dimensions: 1024

llm_policy:
  allowed: [zai/glm-5.2]

diagnostics:
  dwh_rest:
    method: POST
    path: /rpc/ping
    auth: x-api-key
    response: { database: database, schema: schema }

evidence:
  source:
    type: filesystem
    uri: acme-ebikes/evidence      # path inside the repository
  policy:
    max_chunk_chars: 4000
    retain_published_generations: 3
```

Changes from older workspaces:

- the database is reached only through **REST** or **direct Postgres** (`rest_api` /
  `postgres_direct`); the SSH tunnel remains disabled;
- the semantic index is **internal** (Qdrant plus `qwen3-embedding:0.6b`, 1024 dimensions, cosine);
- **filesystem** Evidence lives in the repository (`<id>/evidence`) and is materialized from the
  pinned Git commit (P6). HTTP Evidence is also supported.

### 1.3 Rules

1. **Git is the source of truth.** Change the descriptor, catalog, and Evidence only through a
   *commit* and *push*, followed by an installation *pull*.
2. **No secrets in the repository.** Add passwords, tokens, private keys, and signed URLs at
   runtime through Workspace management; the backend stores them encrypted.
3. **Schema v3 only.** Reject v1 and v2 descriptors before activation.
4. **The application does not push curated content.** The repository curator works in a separate
   authoring clone.

---

## Part 2: use ThothII's repository tools

There are **two** tools: the **web application** (workspace management) and the **`tht` CLI**
(preprocessing and operations). The complete installation is described in
`docs/install/local-workspace-registry.md` (macOS/Windows/Linux) e
`docs/install/server-workspace-registry.md`.

### 2.1 `tht`: main commands

Always invoke `tht` with `--installation <path>/thothii-installation.yaml`. The usual commands are:

```bash
# 1) inspect workspace state (revision and identity)
tht --installation <install> workspace inspect --workspace <id> --json

# 2) inspect the DWH (generates physical.yaml and LSH)
tht --installation <install> workspace preprocess dwh --workspace <id> --json

# 3) suggest joins (FKs) from approved SQL
tht --installation <install> workspace schema suggest-fks --workspace <id>   --from-sql <query>.sql --output <candidati>.yaml --json

# 4) after review, publish curated FKs in Git and accept them
tht --installation <install> workspace schema accept --workspace <id>   --run <run-id> --yes --json

# 5) index the schema (Qdrant)
tht --installation <install> workspace index-schema --workspace <id> --json

# 6) preprocess Evidence
tht --installation <install> workspace preprocess evidence --workspace <id> --json

# 7) complete chain (DWH → FK → schema → Evidence)
tht --installation <install> workspace preprocess run --workspace <id> --json

# 8) inspect or rebuild the Qdrant collection (maintenance only)
tht --installation <install> workspace vector inspect --workspace <id> --json
tht --installation <install> workspace vector rebuild --workspace <id>   --collection <nome> --confirm <nome> --destroy
```

Important notes:

- **`--json` writes JSON only to stdout** (machine contract); use it in scripts.
- **`preprocess run` stops for human review** when it finds new proposed joins: it exits with
  `manual_review_required`. After review, continue with `schema accept ... --yes` and
  `preprocess run --resume <run-id>`.
- **A filesystem Evidence file is materialized from the pinned Git commit** (there is no moving
  checkout); symlinks, unsafe paths, and trees that are too large are rejected.
- **The CLI never writes to the repository** (it never pushes curated content).

### 2.2 Web application: workspace management

Workspace management has two distinct levels.

**Level 1: repository.** The first section shows that the workspace source lives in a separate
directory, is published by the curator to a repository hosted on a Git server such as GitHub,
GitLab, or Gitea, and is read by ThothII in read-only mode. It shows the host, repository, branch,
active revision, and status of the last update.

* **Update workspace repository** does not require a workspace to be selected. The backend fetches
  or pulls the configured branch into ThothII's managed checkout, validates the entire candidate
  revision, and activates it atomically. If validation fails, it keeps the previous revision. It
  does not modify the remote source or save content from the GUI.
* To create a local workspace, prepare a source directory with the catalog, `workspace.yaml`, and
  the expected subdirectories. Validate it, then commit and push from the authoring clone.
  ThothII provides no commands to create, edit, or publish the source.

**Level 2: selected workspace.** These commands are separate because they require a workspace to be selected first.
These commands are separate because they require a workspace to be selected first.

- **Validate workspace source** checks the catalog, descriptor, Evidence, and invariants of the
  selected active revision again. It does not contact the DWH or modify files.
- **Save entered secrets** replaces the entered values without displaying them. The fields depend
  on the declared DWH transport and Evidence authentication. The backend returns only configured
  or missing status.
* **Forget stored value** removes the selected secret from the encrypted vault. Future sessions or
  operations that need it remain blocked until it is entered again.
The remote Git repository and its credentials are installation settings. Runtime DWH and Evidence
secrets persist in the backend's encrypted vault, not in the GUI's local storage. The GUI is only
the interface: after submission it clears the field values and cannot read them back.

---

## Part 3: use the ThothII application

### 3.1 New session

Open the application and use **New session**. Enter only the **natural-language question**;
workspace, model, and provider are already configured as global settings.

Example question:

> "List the electric bicycles completed in the last year, with model, frame number, and completion
> date."

### 3.2 The eight-phase workflow and gates

The question passes through **eight phases**. You see the intermediate documents and decide at the important points:

1. **F1 clarification**: the model removes ambiguity when needed;
2. **F2 Memory**: retrieves reusable Memory;
3. **F3 rewriting**: rewrites and approves the question;
4. **F4 schema linking**: proposes related tables and columns;
5. **F5 summary**: summarizes the selected schema;
6. **F6 CTE**: builds the CTEs;
7. **F7 final SQL**: produces `sql_final.sql`;
8. **F8 datamart**: execution or export (dbt, CSV, Excel).

**Review gates** appear as widgets: choose one option, select several items, or confirm an
artifact or phase. The model *proposes* and the reviewer *decides*. The right side shows artifacts
(schema linking, CTEs, and SQL); the Model activity panel shows the question and reasoning.

### 3.3 Sessions

Sessions appear in the sidebar with their ID, question, date, and author. A session
**resumes** from its last incomplete phase by rebuilding state from documents saved on disk
(`session_manifest.yaml`, phase artifacts, and `review_decisions.jsonl`). Saved state **is** the
truth: what is not recorded did not happen.

---

## Complete example: ACME Limited

### Step 0: repository

Create the workspace Git repository, for example `tht-workspace-acme`:

```text
thoth-workspaces.yaml          # catalog containing acme-ebikes
acme-ebikes/workspace.yaml     # v3 descriptor (see §1.2)
acme-ebikes/evidence/           # curated context .md documents
acme-ebikes/schema/annotations.yaml   # when curated joins exist
```

Publish a new Git revision. In the installation, the application fetches and **activates** the
workspace, validates schema v3, materializes Evidence from the pinned revision, and prepares the
Qdrant collection (1024/cosine plus indexes).

### Step 1: preprocessing

```bash
tht --installation ~/thothii-installation.yaml workspace preprocess dwh   --workspace acme-ebikes --json
tht --installation ~/thothii-installation.yaml workspace preprocess run   --workspace acme-ebikes --json
```

If the run stops for joins (`manual_review_required`):

```bash
# the curator reviews the candidates and publishes acme-ebikes/schema/annotations.yaml, then:
tht --installation ~/thothii-installation.yaml workspace schema accept   --workspace acme-ebikes --run RUN_ID --yes --json
tht --installation ~/thothii-installation.yaml workspace preprocess run   --workspace acme-ebikes --resume RUN_ID --json
```

### Step 2: the question

In the application, select the `acme-ebikes` workspace and create a session with the question.
Follow the phases and confirm the gates. The model will propose schema linking (tables and columns
from the `datawarehouse` DWH), CTEs, and finally the SQL, which you can view, copy, and run.

---

## Where to find technical details

For DWH REST access, the key is installation-specific and applies only to `rest_api`; `postgres_direct`
and `ssh_tunnel` do not use it. See the [DWH server guide](install/dwh-auth-server.md), [client
enrollment guide](install/dwh-auth-client-enrollment.md), and [TLS guide](install/dwh-auth-tls.md).

- CLI contract: `docs/contracts/workspace-preprocessing-cli.md`
- `.tht-dwh` contract: `docs/contracts/tht-dwh.md`
- Evidence v3: `docs/contracts/workspace-evidence-v3.md`