# ThothII — Project State

> Starting-point snapshot for new sessions. Last updated: 2026-06-29.
> Point a fresh session here ("read PROJECT_STATE.md") before substantial work.

## What ThothII is

A **human-in-the-loop datamart builder**: it turns a natural-language question into
validated SQL (and optionally a dbt datamart) through a deterministic **8-phase
NL→SQL workflow**, where the model *proposes* and a human *reviewer decides* at gates.
The UI is meant to embed inside the Omics Portal (GSD design system) and is **English**.

## Architecture — three layers

```
frontend (React, :5173) → backend (Fastify, :8787) → pi --mode rpc → tht / harness → DWH (read-only)
```

- **harness/** — the Pi layer. A deterministic Python CLI **`tht`** + a Pi gate extension
  (`.pi/extensions/tht-gate.js`) that runs the 8-phase workflow and emits/consumes
  widget-descriptor JSON. **Owns all persistence.** Workflow truth is `harness/workflow.yaml`;
  orchestration rules are `harness/.pi/skills/tht-sessione/SKILL.md`.
- **backend/** — Fastify + TypeScript. A **thin bridge**: proxies REST routes to the `tht`
  CLI (`ThtRunner`), manages Pi processes (`PiProcessManager`, one child per session),
  bridges Pi RPC events to SSE (`SessionBridge` + `SseHub`). No application database.
- **frontend/** — React 18 + base-ui + Tailwind + TanStack Query + Zustand. Chat-style
  shell (`src/shell/AppShell.tsx`); the live transcript is rebuilt in-memory from the SSE
  stream (`src/store/sessionStore.ts`), **not persisted**.

### Persistence model (the load-bearing premise)
There is **no verbatim chat store**. Each workflow phase persists its own document into the
session directory, and that **IS** the persistence. A session = a directory under the
workspace's `sessions/` path containing `session_manifest.yaml` + phase artifacts
(`question.md`, `schema_linking.json`, `cte_plan.json`, `sql_final.sql`,
`validation_report.md`, `review_decisions.jsonl`, …). A fresh Pi process resumes by reading
`tht session show <id>` + the on-disk artifacts — never by replaying chat.

## The 8 phases (harness/workflow.yaml)
F1 chiarimento · F2 memoria · F3 riscrittura (`question.md`) · F4 schema_linking
(`schema_linking.json`) · F5 sintesi · F6 cte (`cte_plan.json`, `cte_tests.json`) ·
F7 sql_finale (`sql_final.sql`) · F8 datamart. Current phase is a fold over the decision
ledger (`harness/tht/phase.py`); statuses: `open` / `closed` / `finalized`.

## How to run

**Full stack (real Pi + DWH):** `./scripts/run-stack.sh`
Prereqs: VPN on; `pi` on PATH (configured model); `harness/.env` populated;
`harness/config/tht.yaml` → a workspace; deps installed in all three projects.
Opens frontend at http://localhost:5173 → backend :8787.

**Individual dev:**
- backend: `cd backend && npm run dev` (tsx watch; env: `PORT`, `THT_HARNESS_DIR`, `THT_BIN`, `PI_BIN`, `AUTH_MODE`)
- frontend: `cd frontend && npm run dev` (Vite; `VITE_BACKEND_URL` → backend)
- harness install: `cd harness && python -m venv .venv && pip install -e ".[dev]"` → `tht` on PATH

## How to test (all green as of 2026-06-29: harness 248 / backend 59 / frontend 73)
- harness: `cd harness && .venv/bin/pytest -q` (5 L2/real-DB tests are deselected by default)
- backend: `cd backend && npx vitest run` · typecheck `npx tsc --noEmit -p .`
- frontend: `cd frontend && npx vitest run` · typecheck `npx tsc -b` · e2e `npm run e2e` (Playwright)

## Config & workspaces
- Workspaces: `harness/workspaces/*.yaml` (`psd`, `tht-test`, `tht.example`). A workspace sets
  the DB target and the **absolute** `paths.sessions/artifacts/indexes` (psd → a *separate*
  repo `tht-workspace-psd/`, NOT committed here).
- Secrets live ONLY in `harness/.env` (gitignored; `THT_*` — DB, DWH REST, vector, SSL CA…).
  See `harness/.env.example` for the variable list.
- App settings (global): `backend/data/settings.json` (gitignored) — `{ workspace, provider,
  model, thinking }`. The "New session" form is question-only; these settings supply the rest.

## Conventions & contracts (don't relearn the hard way)
- **`-c`/`--config` is a PER-COMMAND option in `tht`** — append it AFTER the subcommand,
  never globally (`ThtRunner.buildArgv` handles this).
- **`--json` output must be pristine** (only valid JSON on stdout).
- **UI strings are English.** Document *content* stays in the workspace language (Italian
  for psd) because it's the real data; only chrome/labels are English.
- **Settings are global**, not per-question.
- TDD throughout; tests assert real behavior, not mocks. Frequent, scoped commits.
- Global user rules (`~/.claude/CLAUDE.md`): think before coding, simplicity first, surgical
  changes, goal-driven verification.

## Most recent feature — Session management (MERGED to main @ 2c21e46)
Full session management modeled on Claude's UI, all three layers:
- **Read-only "split view" panel** (left drawer, `SessionDocumentsPanel`) showing a session's
  phase documents read-only (reuses `SqlViewer`/`SchemaLinkingViewer`/`MarkdownView`).
- **Rename / Move to group / Archive / Delete** via a kebab menu (`SessionMenu`) → REST →
  `tht session set-name/set-group/archive/unarchive/delete`. Archive = a manifest `archived`
  flag (not a dir move); groups = a manifest `group` field; delete = hard `rmtree` + confirm.
- Rail: collapsible group headers + "No group" + a separate **Archive** view.
- **Resume correctness:** read-only **guard** (HTTP 409 when `finalized` or `archived`);
  `PiProcessManager.spawnFor` now has a `new`/`resume` mode (resume sends
  `/riprendi-sessione <id>`); a "Phase 0 — Resume" cold-start section in `SKILL.md`.
- Design docs: `docs/superpowers/specs/2026-06-29-session-management-design.md` +
  `docs/superpowers/plans/2026-06-29-session-management.md`.

### ⚠️ Open items / pending gates
1. **Resume cold-start e2e (MANUAL, not yet run).** The resume chain is structurally
   complete but unproven live: drive a session to F4, stop it, resume in a fresh process,
   confirm it re-enters at F4 (not a new-question kickoff). **Do not advertise the "Resume"
   button as working until this passes.**
2. **Full Playwright live-stack verification (MANUAL, not yet run).**
3. **Minor backlog (non-blocking):** explicit id-traversal guard in `delete_session`
   (today gated by `load_session`); `close_session` could reuse `_save_touched` (DRY);
   delete-via-kebab integration test skipped (base-ui Menu portal not drivable in jsdom —
   the dialog itself is unit-tested); a couple of test-file lint nits.

## Where design history lives
- Specs: `docs/superpowers/specs/` · Plans: `docs/superpowers/plans/`
- SDD execution ledger (gitignored scratch): `.superpowers/sdd/progress.md`
- Auto-memory index: `~/.claude/projects/-Users-mp-projects-ThothII/memory/MEMORY.md`
  (notes on the Pi RPC event vocabulary and the Omics Portal/GSD design system).

## Git
`main` @ `2c21e46`, pushed to `origin` (github.com/mptyl/ThothII), in sync. Other local
branches (`feat/settings-menu`, `feat/thothII-debugging`) are pre-existing, untouched.