mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
79c4c925b5743fb0276c762b27c3542eda1995da
ThothII/docs/guida-utente.md
T

12 KiB
Raw Blame History

ThothII user guide

For login, Remember me, roles, session invalidation, OIDC groups, and recovery, see the local authentication guide and the generic OIDC guide.

This guide walks you through preparing the workspace repository, using ThothII's tools for that repository, and using the application to ask natural-language questions and obtain validated SQL. It uses plain language and examples. The contracts listed at the end contain the technical details.

What is ThothII? It is a datamart builder with human review. You write a natural-language question, the model proposes each step in turn (clarifications, schema, CTEs, and SQL), and a human reviewer decides at every important step. The final result is validated SQL ready to run on the data warehouse.

Part 1: prepare the workspace repository on Git

1.1 Structure

The workspace repository is a Git repository that describes which data is available and how to reach it. It contains neither the data nor secrets such as passwords, tokens, or certificates.

A valid repository contains:

thoth-workspaces.yaml              # catalog: list of workspaces
<workspace-id>/workspace.yaml      # workspace descriptor (schema v3)
<workspace-id>/evidence/            # optional context documents, such as *.md
<workspace-id>/schema/annotations.yaml   # optional manually curated logical joins (P5)
  • The catalog thoth-workspaces.yaml is a simple list:
schema_version: 1
workspaces:
  - id: acme-ebikes
    name: ACME Limited
    description: DWH for electric bicycle production
  • The ID must be lowercase, contain no spaces, and follow acme-ebikes ([a-z][a-z0-9-]{2,62}).
  • The descriptor <id>/workspace.yaml uses schema v3. It is the only valid description.

1.2 Descriptor example (ACME Limited)

workspace:
  schema_version: 3
  id: acme-ebikes
  name: ACME Limited
  description: Industrial DWH for electric bicycle production
  language: en                      # descriptions and Evidence are in English

dwh:
  engine: postgres
  database: postgres
  schema: datawarehouse
    supported_transports: [rest_api]  # access through the REST API (PostgREST)

semantic_index:
  vector_store:
    engine: qdrant
    collection: acme-ebikes
    dimensions: 1024
    distance: cosine
  embedding:
    provider: ollama_internal
    model: qwen3-embedding:0.6b
    dimensions: 1024

llm_policy:
  allowed: [zai/glm-5.2]

diagnostics:
  dwh_rest:
    method: POST
    path: /rpc/ping
    auth: x-api-key
    response: { database: database, schema: schema }

evidence:
  source:
    type: filesystem
    uri: acme-ebikes/evidence      # path inside the repository
  policy:
    max_chunk_chars: 4000
    retain_published_generations: 3

Changes from older workspaces:

  • NL→SQL sessions reach the database only through REST or direct Postgres (rest_api / postgres_direct); ssh_tunnel remains disabled for session runtime. The separate Database management surface supports SSH for Test connection and Sync tables, with a private key, mandatory known_hosts, and an optional key passphrase;
  • the semantic index is internal (Qdrant plus qwen3-embedding:0.6b, 1024 dimensions, cosine);
  • filesystem Evidence lives in the repository (<id>/evidence) and is materialized from the pinned Git commit (P6). HTTP Evidence is also supported.

1.3 Rules

  1. Git is the source of truth. Change the descriptor, catalog, and Evidence only through a commit and push, followed by an installation pull.
  2. No secrets in the repository. Add passwords, tokens, private keys, and signed URLs at runtime through Workspace management; the backend stores them encrypted.
  3. Schema v3 only. Reject v1 and v2 descriptors before activation.
  4. The application does not push curated content. The repository curator works in a separate authoring clone.

Part 2: use ThothII's repository tools

There are two tools: the web application (workspace management) and the tht CLI (preprocessing and operations). The complete installation is described in docs/install/local-workspace-registry.md (macOS/Windows/Linux) e docs/install/server-workspace-registry.md.

2.1 tht: main commands

Always invoke tht with --installation <path>/thothii-installation.yaml. The usual commands are:

# 1) inspect workspace state (revision and identity)
tht --installation <install> workspace inspect --workspace <id> --json

# 2) inspect the DWH (generates physical.yaml and LSH)
tht --installation <install> workspace preprocess dwh --workspace <id> --json

# 3) suggest joins (FKs) from approved SQL
tht --installation <install> workspace schema suggest-fks --workspace <id>   --from-sql <query>.sql --output <candidati>.yaml --json

# 4) after review, publish curated FKs in Git and accept them
tht --installation <install> workspace schema accept --workspace <id>   --run <run-id> --yes --json

# 5) index the schema (Qdrant)
tht --installation <install> workspace index-schema --workspace <id> --json

# 6) preprocess Evidence
tht --installation <install> workspace preprocess evidence --workspace <id> --json

# 7) complete chain (DWH → FK → schema → Evidence)
tht --installation <install> workspace preprocess run --workspace <id> --json

# 8) inspect or rebuild the Qdrant collection (maintenance only)
tht --installation <install> workspace vector inspect --workspace <id> --json
tht --installation <install> workspace vector rebuild --workspace <id>   --collection <nome> --confirm <nome> --destroy

Important notes:

  • --json writes JSON only to stdout (machine contract); use it in scripts.
  • preprocess run stops for human review when it finds new proposed joins: it exits with manual_review_required. After review, continue with schema accept ... --yes and preprocess run --resume <run-id>.
  • A filesystem Evidence file is materialized from the pinned Git commit (there is no moving checkout); symlinks, unsafe paths, and trees that are too large are rejected.
  • The CLI never writes to the repository (it never pushes curated content).

2.2 Web application: workspace management

Workspace management has two distinct levels.

Level 1: repository. The first section shows that the workspace source lives in a separate directory, is published by the curator to a repository hosted on a Git server such as GitHub, GitLab, or Gitea, and is read by ThothII in read-only mode. It shows the host, repository, branch, active revision, and status of the last update.

  • Update workspace repository does not require a workspace to be selected. The backend fetches or pulls the configured branch into ThothII's managed checkout, validates the entire candidate revision, and activates it atomically. If validation fails, it keeps the previous revision. It does not modify the remote source or save content from the GUI.
  • To create a local workspace, prepare a source directory with the catalog, workspace.yaml, and the expected subdirectories. Validate it, then commit and push from the authoring clone. ThothII provides no commands to create, edit, or publish the source.

Level 2: selected workspace. These commands are separate because they require a workspace to be selected first. These commands are separate because they require a workspace to be selected first.

  • Validate workspace source checks the catalog, descriptor, Evidence, and invariants of the selected active revision again. It does not contact the DWH or modify files.
  • Save entered secrets replaces the entered values without displaying them. The fields depend on the declared DWH transport and Evidence authentication. The backend returns only configured or missing status.
  • Forget stored value removes the selected secret from the encrypted vault. Future sessions or operations that need it remain blocked until it is entered again. The remote Git repository and its credentials are installation settings. Runtime DWH and Evidence secrets persist in the backend's encrypted vault, not in the GUI's local storage. The GUI is only the interface: after submission it clears the field values and cannot read them back.

Part 3: use the ThothII application

3.1 New session

Open the application and use New session. Enter only the natural-language question; workspace, model, and provider are already configured as global settings.

Example question:

"List the electric bicycles completed in the last year, with model, frame number, and completion date."

3.2 The eight-phase workflow and gates

The question passes through eight phases. You see the intermediate documents and decide at the important points:

  1. F1 clarification: the model removes ambiguity when needed;
  2. F2 Memory: retrieves reusable Memory;
  3. F3 rewriting: rewrites and approves the question;
  4. F4 schema linking: proposes related tables and columns;
  5. F5 summary: summarizes the selected schema;
  6. F6 CTE: builds the CTEs;
  7. F7 final SQL: produces sql_final.sql;
  8. F8 datamart: execution or export (dbt, CSV, Excel).

Review gates appear as widgets: choose one option, select several items, or confirm an artifact or phase. The model proposes and the reviewer decides. The right side shows artifacts (schema linking, CTEs, and SQL); the Model activity panel shows the question and reasoning.

3.3 Sessions

Sessions appear in the sidebar with their ID, question, date, and author. A session resumes from its last incomplete phase by rebuilding state from documents saved on disk (session_manifest.yaml, phase artifacts, and review_decisions.jsonl). Saved state is the truth: what is not recorded did not happen.

Complete example: ACME Limited

Step 0: repository

Create the workspace Git repository, for example tht-workspace-acme:

thoth-workspaces.yaml          # catalog containing acme-ebikes
acme-ebikes/workspace.yaml     # v3 descriptor (see §1.2)
acme-ebikes/evidence/           # curated context .md documents
acme-ebikes/schema/annotations.yaml   # when curated joins exist

Publish a new Git revision. In the installation, the application fetches and activates the workspace, validates schema v3, materializes Evidence from the pinned revision, and prepares the Qdrant collection (1024/cosine plus indexes).

Step 1: preprocessing

tht --installation ~/thothii-installation.yaml workspace preprocess dwh   --workspace acme-ebikes --json
tht --installation ~/thothii-installation.yaml workspace preprocess run   --workspace acme-ebikes --json

If the run stops for joins (manual_review_required):

# the curator reviews the candidates and publishes acme-ebikes/schema/annotations.yaml, then:
tht --installation ~/thothii-installation.yaml workspace schema accept   --workspace acme-ebikes --run RUN_ID --yes --json
tht --installation ~/thothii-installation.yaml workspace preprocess run   --workspace acme-ebikes --resume RUN_ID --json

Step 2: the question

In the application, select the acme-ebikes workspace and create a session with the question. Follow the phases and confirm the gates. The model will propose schema linking (tables and columns from the datawarehouse DWH), CTEs, and finally the SQL, which you can view, copy, and run.

Where to find technical details

For DWH REST access, the key is installation-specific and applies only to rest_api; postgres_direct and ssh_tunnel do not use it. See the DWH server guide, client enrollment guide, and TLS guide.

  • CLI contract: docs/contracts/workspace-preprocessing-cli.md
  • .tht-dwh contract: docs/contracts/tht-dwh.md
  • Evidence v3: docs/contracts/workspace-evidence-v3.md
Reference in New Issue View Git Blame Copy Permalink