ThothII/PROJECT_STATE.md

# ThothII — Project State

> Starting-point snapshot for new sessions. Last updated: 2026-07-01.
> Point a fresh session here ("read PROJECT_STATE.md") before substantial work.

## What ThothII is

A **human-in-the-loop datamart builder**: it turns a natural-language question into
validated SQL (and optionally a dbt datamart) through a deterministic **8-phase
NL→SQL workflow**, where the model *proposes* and a human *reviewer decides* at gates.
The UI is meant to embed inside the Omics Portal (GSD design system) and is **English**.

## Architecture — three layers

```
frontend (React, :5173) → backend (Fastify, :8787) → pi --mode rpc → tht / harness → DWH (read-only)
```

- **harness/** — the Pi layer. A deterministic Python CLI **`tht`** + a Pi gate extension
  (`.pi/extensions/tht-gate.js`) that runs the 8-phase workflow and emits/consumes
  widget-descriptor JSON. **Owns all persistence.** Workflow truth is `harness/workflow.yaml`;
  orchestration rules are `harness/.pi/skills/tht-sessione/SKILL.md`.
- **backend/** — Fastify + TypeScript. A **thin bridge**: proxies REST routes to the `tht`
  CLI (`ThtRunner`), manages Pi processes (`PiProcessManager`, one child per session),
  bridges Pi RPC events to SSE (`SessionBridge` + `SseHub`). No application database.
- **frontend/** — React 18 + base-ui + Tailwind + TanStack Query + Zustand. Chat-style
  shell (`src/shell/AppShell.tsx`); the live transcript is rebuilt in-memory from the SSE
  stream (`src/store/sessionStore.ts`), **not persisted**.

### Persistence model (the load-bearing premise)
There is **no verbatim chat store**. Each workflow phase persists its own document into the
session directory, and that **IS** the persistence. A session = a directory under the
workspace's `sessions/` path containing `session_manifest.yaml` + phase artifacts
(`question.md`, `schema_linking.json`, `cte_plan.json`, `sql_final.sql`,
`validation_report.md`, `review_decisions.jsonl`, …). A fresh Pi process resumes by reading
`tht session show <id>` + the on-disk artifacts — never by replaying chat.

## The 8 phases (harness/workflow.yaml)
F1 chiarimento · F2 memoria · F3 riscrittura (`question.md`) · F4 schema_linking
(`schema_linking.json`) · F5 sintesi · F6 cte (`cte_plan.json`, `cte_tests.json`) ·
F7 sql_finale (`sql_final.sql`) · F8 datamart. Current phase is a fold over the decision
ledger (`harness/tht/phase.py`); statuses: `open` / `closed` / `finalized`.

## How to run

**Full stack (real Pi + DWH):** `./scripts/run-stack.sh`
Prereqs: VPN on; `pi` on PATH (configured model); `harness/.env` populated;
`harness/config/tht.yaml` → a workspace; deps installed in all three projects.
Opens frontend at http://localhost:5173 → backend :8787.

**Individual dev:**
- backend: `cd backend && npm run dev` (tsx watch; env: `PORT`, `THT_HARNESS_DIR`, `THT_BIN`, `PI_BIN`, `AUTH_MODE`)
- frontend: `cd frontend && npm run dev` (Vite; `VITE_BACKEND_URL` → backend)
- harness install: `cd harness && python -m venv .venv && pip install -e ".[dev]"` → `tht` on PATH

## How to test (all green as of 2026-07-01: harness 281 / backend 67 / frontend 96)
- harness: `cd harness && .venv/bin/pytest -q` (5 L2/real-DB tests are deselected by default)
- backend: `cd backend && npx vitest run` · typecheck `npx tsc --noEmit -p .`
- frontend: `cd frontend && npx vitest run` · typecheck `npx tsc -b` · e2e `npm run e2e` (Playwright)

## Config & workspaces
- Workspaces: `harness/workspaces/*.yaml` (`psd`, `tht-test`, `tht.example`). A workspace sets
  the DB target and the **absolute** `paths.sessions/artifacts/indexes` (psd → a *separate*
  repo `tht-workspace-psd/`, NOT committed here).
- Secrets live ONLY in `harness/.env` (gitignored; `THT_*` — DB, DWH REST, vector, SSL CA…).
  See `harness/.env.example` for the variable list.
- App settings (global): `backend/data/settings.json` (gitignored) — `{ workspace, provider,
  model, thinking }`. The "New session" form is question-only; these settings supply the rest.

## Conventions & contracts (don't relearn the hard way)
- **`-c`/`--config` is a PER-COMMAND option in `tht`** — append it AFTER the subcommand,
  never globally (`ThtRunner.buildArgv` handles this).
- **`--json` output must be pristine** (only valid JSON on stdout).
- **UI strings are English.** Document *content* stays in the workspace language (Italian
  for psd) because it's the real data; only chrome/labels are English.
- **Settings are global**, not per-question.
- TDD throughout; tests assert real behavior, not mocks. Frequent, scoped commits.
- Global user rules (`~/.claude/CLAUDE.md`): think before coding, simplicity first, surgical
  changes, goal-driven verification.

## Workflow contract hardening — COMPLETE (2026-07-01)

Spec: `docs/superpowers/specs/2026-07-01-workflow-contract-hardening-design.md` · Plan:
`docs/superpowers/plans/2026-07-01-workflow-contract-hardening.md`. Merged to `main` @ `3dadc6f`
(pushed). Driven by analysis of Pi session `2026-06-30-165708` (GLM 5.2), where the model spent
~80% of its tool calls reverse-engineering the harness because `SKILL.md` mis-stated the
phase-advance contract — and Phase 6 was a hard dead-end. Three coordinated harness fixes (TDD):

- **F6 CTE-approval dead-end FIXED.** The gate's `reviewer_confirm kind:"cte_result"` used to
  register `cte_approved --subject phase:6`, which `decision_cmd` rejects (exit 5 — it needs a
  real CTE name from the plan) → F6 could never close. New `tht cte next --session <id>` returns
  the first unapproved plan CTE; the gate now approves **by name**. (`tht/cli/cte_cmd.py`,
  `.pi/extensions/tht-gate.js`.)
- **`schema_linking.json` writer/validator.** `store.set_schema_linking` (validates against the
  `SchemaLinking` model, THEN writes — no partial file) → CLI `tht session set-schema-linking
  <id> --file <path|->` (exit 5 on bad JSON / ValidationError) → gate tool `write_schema_linking`
  (stdin). Replaces the model hand-writing the F4 artifact + ad-hoc python validation.
- **`SKILL.md` corrected to match the code.** Only F2-empty / F6-skipped auto-advance
  (`_AUTO_ADVANCE_PHASES={2,6}`); every substantive phase closes with `reviewer_confirm
  kind:"phase"` (F7 is **two-step**: `kind:"sql"` records `sql_approved`, then `kind:"phase"`
  advances). Fixed Discipline 2 + Phase 1/3/4, added a per-phase **cheat-sheet**, documented the
  `SchemaLinking` shape. The old false "the reviewer_decide already advances" (F3) claim — the
  exact cause of the observed thrash — is gone.

Verified: harness pytest **281 passed** / 5 deselected, gate JS **34/34**, changed-files ruff
clean (the 36 `ruff check .` errors are pre-existing on `main`). Final whole-branch review
(opus): READY TO MERGE, no Critical/Important. Executed via subagent-driven-development
(implementer + task-review per task, final opus review). **DEFERRED (needs VPN): live F4/F6
end-to-end** — resuming session `2026-06-30-165708` (stuck at F6) is the ideal live probe.

## UI/UX redesign + Resume — COMPLETE (2026-06-30)

Plan: **`~/.claude/plans/foamy-forging-dahl.md`**. Memory: `thothii-ui-redesign-inprogress.md`.
**All workstreams done and pushed to origin/main:** D + E @ `0eeb3f7`, B + C @ `b056ff3`,
F @ `cef9ae4`, A @ `0a13f71`, G @ `e8cdd00`(scope) + `cbb8e18`(results). Nothing pending from this
plan. Per-workstream detail below for reference.

- **D — DONE** (`c12bdcd`): session display `name` = 3-5 Italian keywords via **YAKE** (no LLM),
  derived in `tht session new` (CLI layer); `create_session` core unchanged (`name=None` default).
  `yake` added to `harness/pyproject.toml`. TDD `tests/test_session_name.py`; harness 269 passed.
- **E — DONE** (`0eeb3f7`): rotating activity icon replaces the red dot in `CentralStatus`
  (inline, clickable → opens the panel); `ModelActivityPanel` is a **5-line expandable
  model-stream tail**; `WorkingSpinner` extracted to its own module; the separate spinner button
  + orphaned `Transcript.tsx` removed. Frontend 87/87, tsc clean. **Live visual check DONE
  (2026-06-30):** inline spinner opens the panel; 5-line collapsed tail; expand → full transcript.
- **B — DONE** (`b056ff3`): `WorkflowBar` is now colored **dots** F1..F8, no phase-name text
  (amber-translucent=running, green=done, red=error, gray=pending; green connectors lead the active
  dot). Each dot carries `data-state`. Error is lightweight: store `phaseError` set when an `info`
  `level=error` arrives during the phase, cleared on the next `ui_request` (`sessionStore.ts`).
  **All four states live-verified** via Playwright.
- **C — DONE** (`b056ff3`): right sidebar — single-line denser rows (inline status dot + name,
  `py-1`), a 3-level type hierarchy via **`/impeccable`** (L1 `SESSIONS` red/bold/wide-tracking ·
  L2 section + group headers muted uppercase · L3 names normal-case), and the **"No group" label
  removed** (ungrouped sessions render after the last group; guarded so the empty-state still
  teaches when there are no groups). **Live-verified.** (Resume in `SessionMenu` stays with A1.)
- **Tests:** frontend **93/93** (was 87; +3 store `phaseError`, +2 `WorkflowBar` dot-state, +1
  AppShell no-"No group"), `tsc -b` clean.
- **F — DONE** (uncommitted; live check deferred to G): single-select answers **auto-confirm**.
  `reviewer_select` options may carry a `decision` payload (`{type, subject, detail?, rationale?}`)
  and an optional `advance`; picking such an option persists the decision directly via
  `tht decision add` (shared `decisionAddArgs` helper, also used by `reviewer_decide`) — no redundant
  `reviewer_decide`/`reviewer_confirm` gate. Options without a payload stay ask-only; back/exit/Other
  never persist. Pure logic extracted to `resolveSelectOutcome`/`decisionAddArgs` (exported, unit-
  tested). Contract docs updated: `reviewer_select` tool desc + `SKILL.md` (widget summary,
  disciplines 2-3, Phase-1 single-pick) + the `CLAUDE.md` gate note. Gate JS **33/33**, harness 269.
  **Live verification (model actually uses `reviewer_select`+decision, no follow-up gate, decision in
  `review_decisions.jsonl`) deferred to G** — it is model-behavior-dependent.
- **A — DONE** (uncommitted): **A1** — `SessionMenu` gains a **Resume** item (gated to
  `status!=="finalized" && !archived`), wired in `AppShell` to the existing `doResume` → `POST
  /sessions/:id/resume`. 3 tests (`SessionMenu.test.tsx`); frontend **96/96**, tsc clean. **A2** —
  diagnosis-first clean-room repro shows the **resume cold-start stall NO LONGER reproduces on pi
  0.79.4** (8/8 chained into the tool calls, fresh + partway; GLM 5.2 now narrates AND emits
  `tht session show`+`read SKILL.md` in-turn). The earlier narrate-and-stop predates the pi upgrade.
  Defense-in-depth applied: `RIPRENDI_KICKOFF` hardened to force the in-turn tool call (gate test +
  live regression 2/2). The cross-model angle (weaker/older models) lives in **G**.
- **G — DONE** (`e8cdd00`+`cbb8e18`): cross-model behavior matrix via a committed clean-room harness
  (`harness/scripts/model-matrix.mjs`). **Tier 1** — kickoff + resume first-turn: all *available*
  models chain in-turn (`zai/glm-5.2`, `deepseek/deepseek-v4-{pro,flash}`, `aritmolab/qwen3.6-35b-a3b`,
  `zai/glm-4.5-air`); the resume stall recurs on none (closes A's cross-model robustness).
  `aritmolab/gemma4-26b-a4b` = **404 unavailable** at the endpoint (listed but not served) — infra
  gap, not a workflow issue. **Tier 2** — F single-select auto-confirm verified live on `glm-5.2`:
  answering the first `reviewer_select` persisted a `concept_clarified` decision **0→1** with **no
  follow-up gate** (closes F's deferred live check). Full results: the G plan doc + memory
  `thothii-cross-model-matrix`. No prompt hardening needed.

**Status:** **All UI-redesign + resume workstreams done and pushed — D, E, B, C, F, A, G.**
Nothing pending from the plan. Optional nice-to-haves (not required): Tier-2 F/multiselect live for
the non-baseline models (cheap re-run with `harness/scripts/model-matrix.mjs` + the Tier-2 method),
and a one-off manual Playwright kebab→resume pass in the live UI.

## Live verification + reviewer_select fix (2026-06-30, afternoon)

Drove the real stack (Playwright → backend → real Pi → GLM 5.2 → DWH) end-to-end.

- **F1 hang fix (`418187a`) VERIFIED LIVE.** Answered an F1 reviewer widget; Pi resumed (model
  socket reopened) and the gate produced new output — vs the old silent hang. The transition
  "silent hang → gate re-presents/advances" proves `ctx.ui.input` now resolves.
- **New bug found + fixed: reviewer_select `choices` vs `choice`.** The gate's `reviewer_select`
  (and `reviewer_confirm` reject) read `resp.choice` (singular) but the frontend uniformly sends
  `choices: [id]` (array) — so every single-select gate answered "Nessuna scelta ricevuta" and
  re-proposed forever (multiselect was fine; it already read `choices`). Fix: a shared
  `selectedChoice(resp)` helper (`harness/.pi/extensions/tht-gate.js`) reading the array; both
  handlers use it. TDD: `gate/__tests__/gate_choice.test.js` RED→GREEN, full gate suite **28/28**.
  VERIFIED LIVE: a single-select answer is now accepted and the workflow advances (2/4 → 3/4).
- **Resume cold-start STALL confirmed (open item #1).** On `/riprendi-sessione`, GLM 5.2 narrates
  the bootstrap step then ends the turn without the tool call → Pi idle, unrecoverable from the UI.
  Memory: `thothii-resume-cold-start-stall.md`.
- **GLM 5.2 F1 is slow (~3-4 min, ~50+ reads) but works** — looks stuck but isn't; don't hit
  "Stop and save" (it `POST /close`s → kills Pi). Memory: `thothii-glm52-f1-slow-not-stuck.md`.

## Earlier work — F1 reviewer-widget hang fix + multiselect guidance (committed 2026-06-30; authored 2026-06-29)

Two fixes, **committed to `main`** (7 files):

1. **Bug: every reviewer widget hung "stuck with no output" after the human answered** — F1
   disambiguation (and any gate) dead-ended. Root cause, confirmed from Pi's own source
   (`@mariozechner/pi-coding-agent` `dist/modes/rpc/rpc-mode.js`, `createDialogPromise`):
   `ctx.ui.input` assigns its OWN RPC id (`crypto.randomUUID`) and correlates
   `extension_ui_response` on THAT id, silently dropping unknown ids. The gate puts a
   different id (`u${Date.now()}`) inside the descriptor carried in `title`. `SessionBridge`
   was replying with the **descriptor** id, so real Pi never resolved `ctx.ui.input` → the
   model never continued. **Fix:** `SessionBridge` now stores Pi's top-level `m.id`
   (`pendingPiId`) on the incoming request and replies `extension_ui_response{ id: pendingPiId,
   value: <uiResponse JSON> }` (value still carries the descriptor id, so the gate's internal
   `resp.id === descriptor.id` check holds). File: `backend/src/bridge/session-bridge.ts`.
   Full write-up: memory `pi-ui-input-id-correlation.md`.
   - **The test double was masking it:** `harness/tests/fake_pi/fake_pi_rpc.mjs` had forced
     `m.id == descriptor.id`. Corrected to mirror real Pi (distinct `randomUUID` top-level id,
     correlate on it, drop unknown ids); `test_fake_pi_contract.mjs` gained a negative
     regression test ("respond with descriptor id → no follow-up").
   - TDD: `backend/test/session-bridge.test.ts` (unit) + `backend/test/e2e-f1.test.ts`
     (integration — now asserts the model's follow-up arrives after the answer) went
     RED→GREEN.

2. **UX: multi-answer disambiguation** — `harness/.pi/skills/tht-sessione/SKILL.md` Phase 1
   now tells the model to use `reviewer_decide` (the existing multiselect/checkbox widget)
   when an ambiguity admits several simultaneously-true answers, instead of single-pick
   `reviewer_select`. Guidance-only — no new widget (`frontend MultiselectWidget` already
   exists).

Verified at commit time: backend `npx vitest run` **67/67 green**; `tsc --noEmit -p .` **OK**;
fake-pi contract `node --test test_fake_pi_contract.mjs` **2/2 green**. **Verified LIVE
2026-06-30** (see the top "Live verification" section).

## Most recent feature — Session management (MERGED to main @ 2c21e46)
Full session management modeled on Claude's UI, all three layers:
- **Read-only "split view" panel** (left drawer, `SessionDocumentsPanel`) showing a session's
  phase documents read-only (reuses `SqlViewer`/`SchemaLinkingViewer`/`MarkdownView`).
- **Rename / Move to group / Archive / Delete** via a kebab menu (`SessionMenu`) → REST →
  `tht session set-name/set-group/archive/unarchive/delete`. Archive = a manifest `archived`
  flag (not a dir move); groups = a manifest `group` field; delete = hard `rmtree` + confirm.
- Rail: collapsible group headers + "No group" + a separate **Archive** view.
- **Resume correctness:** read-only **guard** (HTTP 409 when `finalized` or `archived`);
  `PiProcessManager.spawnFor` now has a `new`/`resume` mode (resume sends
  `/riprendi-sessione <id>`); a "Phase 0 — Resume" cold-start section in `SKILL.md`.
- Design docs: `docs/superpowers/specs/2026-06-29-session-management-design.md` +
  `docs/superpowers/plans/2026-06-29-session-management.md`.

### ⚠️ Open items / pending gates
1. **Resume cold-start stall — RESOLVED on pi 0.79.4 (workstream A, 2026-06-30).** The earlier
   narrate-and-stop (GLM 5.2 narrating the bootstrap step then ending the turn without the tool
   call) **no longer reproduces**: a clean-room repro of the backend's exact resume handshake
   chained into `tht session show`+`read SKILL.md` in-turn **8/8** (fresh + partway sessions). The
   pi upgrade is the likely fix. Defense-in-depth: `RIPRENDI_KICKOFF` hardened to force the in-turn
   tool call (gate test + live 2/2). Memory: `thothii-resume-cold-start-stall.md`. **Remaining:**
   the cross-model angle (older/weaker models) is folded into **G**; a full Playwright kebab→resume
   pass through the live UI is still worth one manual run (item 2).
2. **Full Playwright live-stack verification (MANUAL, not yet run).**
3. **Minor backlog (non-blocking):** explicit id-traversal guard in `delete_session`
   (today gated by `load_session`); `close_session` could reuse `_save_touched` (DRY);
   delete-via-kebab integration test skipped (base-ui Menu portal not drivable in jsdom —
   the dialog itself is unit-tested); a couple of test-file lint nits.
4. **DONE — F1 hang fix live-verified 2026-06-30** (see top section). The live verification
   also surfaced + fixed the reviewer_select `choices` mismatch.
5. **Resolved: settings use `zai/glm-5.2`/medium** (not deepseek-flash); GLM 5.2 drives F1
   fine, just slowly (~3-4 min, ~50+ reads). To tell a truly stalled Pi from a merely-slow one,
   check its sockets/children. Memory: `thothii-glm52-f1-slow-not-stuck.md`.
6. **DONE — Pi migrated to `@earendil-works/pi-coding-agent@0.80.3` (2026-07-04).** The old
   scope `@mariozechner/pi-coding-agent` is frozen at 0.73.1; every release ≥0.74 lives under
   the new scope `@earendil-works` (latest 0.80.3). The live `pi` (`~/.local/bin/pi`) was
   repointed to 0.80.3. Validated by an API/RPC-surface diff (0.73.1→0.80.3: `ExtensionUIContext`
   byte-identical, `rpc-types` additive-only, `createDialogPromise` + provider-registration API
   unchanged) **plus** a live `model-matrix` smoke (GLM 5.2, `new`+`resume` both `CHAINED`,
   full RPC event vocabulary incl. `extension_ui_request` intact). Notable: 0.80.3 adds
   `ctx.mode: "tui"|"rpc"|"json"|"print"` (a proper mode discriminator; the earlier `0.79.4`
   references above are superseded). Rollback: the old package is still on disk — repoint the
   symlink to `@mariozechner/.../dist/cli.js`. Memory: `thothii-pi-earendil-migration.md`.

## Where design history lives
- Specs: `docs/superpowers/specs/` · Plans: `docs/superpowers/plans/`
- SDD execution ledger (gitignored scratch): `.superpowers/sdd/progress.md`
- Auto-memory index: `~/.claude/projects/-Users-mp-projects-ThothII/memory/MEMORY.md`
  (Pi RPC event vocabulary, ui.input id correlation, Omics Portal/GSD design system,
  resume cold-start stall, GLM 5.2 F1 slow≠stuck).

## Git
`main` @ `3dadc6f`, **pushed to `origin`** (github.com/mptyl/ThothII); working tree clean, no stashes.
Workflow-hardening landed on top of the redesign: `d5af7b5` (spec) · `0193341` (plan) · `47a5217`
(cte next) · `160d7f0` (F6 by-name) · `92caaac`+`0077792` (schema_linking store+CLI) · `20bc3de`
(write_schema_linking gate) · `01de6f3`+`3dadc6f` (SKILL corrections + F7). Feature branch
`feat/workflow-contract-hardening` deleted after the ff-merge.
**`main` is the only branch** — the three stale branches (`feat/settings-menu` `bcaf670`,
`feat/thothII-debugging` `d67409f`, `feat/ui-redesign-resume` `0eeb3f7`) were fully merged ancestors
(no unique work) and were deleted on 2026-06-30 (SHAs recoverable via reflog).
Redesign+resume commits on top of `0eeb3f7`: `b056ff3` (B/C dots+sidebar) · `cef9ae4` (F
auto-confirm) · `0a13f71` (A resume menu + kickoff hardening) · `e8cdd00` (G scope) · `cbb8e18`
(G results). Earlier today: `418187a` (F1 hang fix) · `d27afd0` (reviewer_select choices fix) ·
`c12bdcd` (D) · `0eeb3f7` (E).