ThothII/harness/README.md

ThothII harness — tht

The self-contained Pi layer of ThothII: a deterministic Python CLI (tht) + a Pi gate extension (.pi/extensions/tht-gate.js) that runs the 8-phase NL→SQL workflow, emitting and consuming widget-descriptor JSON. Derived from the reference implementation as a validated starting point, adapted (not assumed reliable) to the ThothII contract.

Install

cd harness
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

The tht command is now on PATH. Node ≥ 20 is needed for the gate JS tests (npm test).

Configure

.env (gitignored — secrets live ONLY here)

cp .env.example .env
# fill in: THT_PROFILE, THT_DB_*, THT_DWH_API_KEY, THT_SSL_CA, ...

Keys are never logged; URLs are fine. Rotate any key that appeared in chat.

workspaces/<name>.yaml

A schema-v4 workspace wires the external relational DWH to one internal Qdrant collection. The embedding identity and dimensions come from the Installation Model Catalog, not this workspace descriptor. Evidence paths remain workspace-local, while Qdrant stores the derived semantic projection for schema, Evidence, Memory, and solved-question records. The legacy files under workspaces/ are retained as migration fixtures; new operator-facing descriptors live in the workspace Git registry. ${THT_*} tokens expand from .env.

DB support (MVP): the direct transport supports PostgreSQL only (psycopg2 driver, pg_* catalog introspection, postgres-dialect sqlcheck/EXPLAIN). The central production DWH is reached via the rest transport. Multi-dialect direct support (sqlserver/mariadb/informix, cf. Thoth/thoth_sqldb2) is separate future work.

Per-customer workspace repo (deployment shape)

ThothII is generic; evidence content and LSH indexes are per-customer and live in a separate repo, not copied into harness/. The deployment is two checkouts + one .env:

ThothII/                      generic harness (this repo)
└── harness/                  tht CLI + .env (THT_* secrets, points at the customer repo)

tht-workspace-<customer>/     per-customer workspace repo (separate git repo)
├── <customer>.yaml           workspace YAML (paths ABSOLUTE, anchored here)
├── evidence/                 curated evidence markdown (frontmatter-structured, by domain)
├── artifacts/                derived artifacts (gitignored; regenerated by tht)
├── indexes/                  LSH index (gitignored; regenerated by `tht preprocess dwh --steps lsh`)
└── sessions/                 per-session artifacts (gitignored)

THT_DOCS_ROOT in .env points at the root of the customer repo (evidence.source_root + evidence.evidence_dir resolve the curated folder). All tht commands take the customer workspace via -c:

tht schema introspect -c /path/to/tht-workspace-<customer>/<customer>.yaml
tht preprocess dwh --steps lsh -c /path/to/tht-workspace-<customer>/<customer>.yaml
tht session new      -c /path/to/tht-workspace-<customer>/<customer>.yaml

Efficiency levers setup (one-time per workspace)

To enable the three deployed optimization levers on a new workspace:

# Lever 1: Foreign key suggestions from approved SQL + heuristics.
# Run once after schema introspection to populate annotations.yaml.
tht schema suggest-fks \
  --from-sql /path/to/customer/sessions \
  --assume cod_paz=dim_patient \
  -c /path/to/customer/workspace.yaml \
  --write

# (psd workspace: 228 FK suggestions already generated via this command 2026-07-08)

# Levers 2 + 3: automatic at next session (context-pack + recap from ledger).
# No setup needed — SKILL.md prescribes them automatically.

See PROJECT_STATE.md for what each lever does.

The workflow

workflow.yaml is the single source of workflow truth (spec F2). Eight phases (F1 chiarimento → F8 datamart). Edit it to change the workflow; phase.py, the gate, and the skill all read from it — no mirrored constants to drift.

tht phase meta --json        # the gate reads workflow facts from here
tht phase meta               # human-readable

Run

The harness runs inside Pi (pi --mode rpc, cwd = harness/):

pi --mode rpc
# in Pi: /nuova-domanda "<your question in natural language>"

The gate (tht-gate.js) presents reviewer widgets (widget-descriptor), the reviewer decides, and tht persists decisions to the append-only ledger (review_decisions.jsonl). See docs/testing.md for what each interaction level validates.

Test

pytest            # L0 (testcontainers, real Postgres) + L1 (pure logic + gate builders)
npm test          # gate widget-builder golden + fuzzy tests (JS)
pytest -m l2      # L2: real GLM 5.2 + remote DWH (pre-release; needs .env + VPN + CA bundle)

L0 needs Docker (present on the dev machine). L2 is manual, non-deterministic, and skips cleanly without .env. See docs/testing.md for the honest split of what each level covers and — crucially — does not cover.

Layout

tht/                 Python package (CLI + workflow + phase + decisions + db/rest/mschema/
                     vectorstore/evidence/search/session + memory)
.pi/                 Pi project (settings, prompts, themes, extensions/tht-gate.js + gate/)
workflow.yaml        single source of workflow truth (F2)
workspaces/          workspace YAML definitions (D3)
scripts/             retained legacy SQL fixtures and workspace utilities
tests/               L0 (testcontainers), L1 (logic + builders), L2 (real model + DB)
docs/                testing guide + workflow editing

Reference

• Repository architecture: ../docs/architecture/overview.md
• Components and flows: ../docs/architecture/components.md
• Runtime contracts: ../docs/contracts/