ThothII/PROJECT_STATE.md

# ThothII — Project State

> Starting-point snapshot for new sessions. Last updated: 2026-06-30.
> Point a fresh session here ("read PROJECT_STATE.md") before substantial work.

## What ThothII is

A **human-in-the-loop datamart builder**: it turns a natural-language question into
validated SQL (and optionally a dbt datamart) through a deterministic **8-phase
NL→SQL workflow**, where the model *proposes* and a human *reviewer decides* at gates.
The UI is meant to embed inside the Omics Portal (GSD design system) and is **English**.

## Architecture — three layers

```
frontend (React, :5173) → backend (Fastify, :8787) → pi --mode rpc → tht / harness → DWH (read-only)
```

- **harness/** — the Pi layer. A deterministic Python CLI **`tht`** + a Pi gate extension
  (`.pi/extensions/tht-gate.js`) that runs the 8-phase workflow and emits/consumes
  widget-descriptor JSON. **Owns all persistence.** Workflow truth is `harness/workflow.yaml`;
  orchestration rules are `harness/.pi/skills/tht-sessione/SKILL.md`.
- **backend/** — Fastify + TypeScript. A **thin bridge**: proxies REST routes to the `tht`
  CLI (`ThtRunner`), manages Pi processes (`PiProcessManager`, one child per session),
  bridges Pi RPC events to SSE (`SessionBridge` + `SseHub`). No application database.
- **frontend/** — React 18 + base-ui + Tailwind + TanStack Query + Zustand. Chat-style
  shell (`src/shell/AppShell.tsx`); the live transcript is rebuilt in-memory from the SSE
  stream (`src/store/sessionStore.ts`), **not persisted**.

### Persistence model (the load-bearing premise)
There is **no verbatim chat store**. Each workflow phase persists its own document into the
session directory, and that **IS** the persistence. A session = a directory under the
workspace's `sessions/` path containing `session_manifest.yaml` + phase artifacts
(`question.md`, `schema_linking.json`, `cte_plan.json`, `sql_final.sql`,
`validation_report.md`, `review_decisions.jsonl`, …). A fresh Pi process resumes by reading
`tht session show <id>` + the on-disk artifacts — never by replaying chat.

## The 8 phases (harness/workflow.yaml)
F1 chiarimento · F2 memoria · F3 riscrittura (`question.md`) · F4 schema_linking
(`schema_linking.json`) · F5 sintesi · F6 cte (`cte_plan.json`, `cte_tests.json`) ·
F7 sql_finale (`sql_final.sql`) · F8 datamart. Current phase is a fold over the decision
ledger (`harness/tht/phase.py`); statuses: `open` / `closed` / `finalized`.

## How to run

**Full stack (real Pi + DWH):** `./scripts/run-stack.sh`
Prereqs: VPN on; `pi` on PATH (configured model); `harness/.env` populated;
`harness/config/tht.yaml` → a workspace; deps installed in all three projects.
Opens frontend at http://localhost:5173 → backend :8787.

**Individual dev:**
- backend: `cd backend && npm run dev` (tsx watch; env: `PORT`, `THT_HARNESS_DIR`, `THT_BIN`, `PI_BIN`, `AUTH_MODE`)
- frontend: `cd frontend && npm run dev` (Vite; `VITE_BACKEND_URL` → backend)
- harness install: `cd harness && python -m venv .venv && pip install -e ".[dev]"` → `tht` on PATH

## How to test (all green as of 2026-06-30: harness 269 / backend 67 / frontend 87)
- harness: `cd harness && .venv/bin/pytest -q` (5 L2/real-DB tests are deselected by default)
- backend: `cd backend && npx vitest run` · typecheck `npx tsc --noEmit -p .`
- frontend: `cd frontend && npx vitest run` · typecheck `npx tsc -b` · e2e `npm run e2e` (Playwright)

## Config & workspaces
- Workspaces: `harness/workspaces/*.yaml` (`psd`, `tht-test`, `tht.example`). A workspace sets
  the DB target and the **absolute** `paths.sessions/artifacts/indexes` (psd → a *separate*
  repo `tht-workspace-psd/`, NOT committed here).
- Secrets live ONLY in `harness/.env` (gitignored; `THT_*` — DB, DWH REST, vector, SSL CA…).
  See `harness/.env.example` for the variable list.
- App settings (global): `backend/data/settings.json` (gitignored) — `{ workspace, provider,
  model, thinking }`. The "New session" form is question-only; these settings supply the rest.

## Conventions & contracts (don't relearn the hard way)
- **`-c`/`--config` is a PER-COMMAND option in `tht`** — append it AFTER the subcommand,
  never globally (`ThtRunner.buildArgv` handles this).
- **`--json` output must be pristine** (only valid JSON on stdout).
- **UI strings are English.** Document *content* stays in the workspace language (Italian
  for psd) because it's the real data; only chrome/labels are English.
- **Settings are global**, not per-question.
- TDD throughout; tests assert real behavior, not mocks. Frequent, scoped commits.
- Global user rules (`~/.claude/CLAUDE.md`): think before coding, simplicity first, surgical
  changes, goal-driven verification.

## UI/UX redesign + Resume — IN PROGRESS (2026-06-30, evening)

Approved multi-workstream plan: **`~/.claude/plans/foamy-forging-dahl.md`** (read it to resume).
Memory: `thothii-ui-redesign-inprogress.md`. **D + E are merged to `main` @ `0eeb3f7` and pushed.**

- **D — DONE** (`c12bdcd`): session display `name` = 3-5 Italian keywords via **YAKE** (no LLM),
  derived in `tht session new` (CLI layer); `create_session` core unchanged (`name=None` default).
  `yake` added to `harness/pyproject.toml`. TDD `tests/test_session_name.py`; harness 269 passed.
- **E — DONE** (`0eeb3f7`): rotating activity icon replaces the red dot in `CentralStatus`
  (inline, clickable → opens the panel); `ModelActivityPanel` is a **5-line expandable
  model-stream tail**; `WorkingSpinner` extracted to its own module; the separate spinner button
  + orphaned `Transcript.tsx` removed. Frontend 87/87, tsc clean. **Live visual check PENDING**
  (needs a *working* session — do it in the B/C pass).
- **B (pending):** `WorkflowBar` → colored **dots** F1..F8, no text (amber-translucent=running,
  green=done, red=error, gray=pending); error is lightweight (store `phaseError` set on an `info`
  `level=error` during the phase, cleared on the next `ui_request`).
- **C (pending):** right sidebar — tighter rows, 3-level type hierarchy via **`/impeccable`**,
  remove the "No group" label (ungrouped after the last group accordion); add **Resume** to `SessionMenu`.
- **F (pending):** single-select answers **auto-confirm** — `reviewer_select` persists the
  decision directly on a concrete choice (no redundant `reviewer_decide` gate); contract change
  (update its tool desc + `SKILL.md` + the `CLAUDE.md` note). back/exit/Other stay non-persisting.
- **A (pending, riskiest):** Resume command + **FIX the resume cold-start stall** (open item #1).
- **G (later):** cross-model behavior matrix (Qwen3.6 / GLM 5.2 / Deepseek V4 / others).

**Next chunk:** the **B/C visual pass** (`/impeccable` + a live verification, incl. E's deferred check).

## Live verification + reviewer_select fix (2026-06-30, afternoon)

Drove the real stack (Playwright → backend → real Pi → GLM 5.2 → DWH) end-to-end.

- **F1 hang fix (`418187a`) VERIFIED LIVE.** Answered an F1 reviewer widget; Pi resumed (model
  socket reopened) and the gate produced new output — vs the old silent hang. The transition
  "silent hang → gate re-presents/advances" proves `ctx.ui.input` now resolves.
- **New bug found + fixed: reviewer_select `choices` vs `choice`.** The gate's `reviewer_select`
  (and `reviewer_confirm` reject) read `resp.choice` (singular) but the frontend uniformly sends
  `choices: [id]` (array) — so every single-select gate answered "Nessuna scelta ricevuta" and
  re-proposed forever (multiselect was fine; it already read `choices`). Fix: a shared
  `selectedChoice(resp)` helper (`harness/.pi/extensions/tht-gate.js`) reading the array; both
  handlers use it. TDD: `gate/__tests__/gate_choice.test.js` RED→GREEN, full gate suite **28/28**.
  VERIFIED LIVE: a single-select answer is now accepted and the workflow advances (2/4 → 3/4).
- **Resume cold-start STALL confirmed (open item #1).** On `/riprendi-sessione`, GLM 5.2 narrates
  the bootstrap step then ends the turn without the tool call → Pi idle, unrecoverable from the UI.
  Memory: `thothii-resume-cold-start-stall.md`.
- **GLM 5.2 F1 is slow (~3-4 min, ~50+ reads) but works** — looks stuck but isn't; don't hit
  "Stop and save" (it `POST /close`s → kills Pi). Memory: `thothii-glm52-f1-slow-not-stuck.md`.

## Earlier work — F1 reviewer-widget hang fix + multiselect guidance (committed 2026-06-30; authored 2026-06-29)

Two fixes, **committed to `main`** (7 files):

1. **Bug: every reviewer widget hung "stuck with no output" after the human answered** — F1
   disambiguation (and any gate) dead-ended. Root cause, confirmed from Pi's own source
   (`@mariozechner/pi-coding-agent` `dist/modes/rpc/rpc-mode.js`, `createDialogPromise`):
   `ctx.ui.input` assigns its OWN RPC id (`crypto.randomUUID`) and correlates
   `extension_ui_response` on THAT id, silently dropping unknown ids. The gate puts a
   different id (`u${Date.now()}`) inside the descriptor carried in `title`. `SessionBridge`
   was replying with the **descriptor** id, so real Pi never resolved `ctx.ui.input` → the
   model never continued. **Fix:** `SessionBridge` now stores Pi's top-level `m.id`
   (`pendingPiId`) on the incoming request and replies `extension_ui_response{ id: pendingPiId,
   value: <uiResponse JSON> }` (value still carries the descriptor id, so the gate's internal
   `resp.id === descriptor.id` check holds). File: `backend/src/bridge/session-bridge.ts`.
   Full write-up: memory `pi-ui-input-id-correlation.md`.
   - **The test double was masking it:** `harness/tests/fake_pi/fake_pi_rpc.mjs` had forced
     `m.id == descriptor.id`. Corrected to mirror real Pi (distinct `randomUUID` top-level id,
     correlate on it, drop unknown ids); `test_fake_pi_contract.mjs` gained a negative
     regression test ("respond with descriptor id → no follow-up").
   - TDD: `backend/test/session-bridge.test.ts` (unit) + `backend/test/e2e-f1.test.ts`
     (integration — now asserts the model's follow-up arrives after the answer) went
     RED→GREEN.

2. **UX: multi-answer disambiguation** — `harness/.pi/skills/tht-sessione/SKILL.md` Phase 1
   now tells the model to use `reviewer_decide` (the existing multiselect/checkbox widget)
   when an ambiguity admits several simultaneously-true answers, instead of single-pick
   `reviewer_select`. Guidance-only — no new widget (`frontend MultiselectWidget` already
   exists).

Verified at commit time: backend `npx vitest run` **67/67 green**; `tsc --noEmit -p .` **OK**;
fake-pi contract `node --test test_fake_pi_contract.mjs` **2/2 green**. **Verified LIVE
2026-06-30** (see the top "Live verification" section).

## Most recent feature — Session management (MERGED to main @ 2c21e46)
Full session management modeled on Claude's UI, all three layers:
- **Read-only "split view" panel** (left drawer, `SessionDocumentsPanel`) showing a session's
  phase documents read-only (reuses `SqlViewer`/`SchemaLinkingViewer`/`MarkdownView`).
- **Rename / Move to group / Archive / Delete** via a kebab menu (`SessionMenu`) → REST →
  `tht session set-name/set-group/archive/unarchive/delete`. Archive = a manifest `archived`
  flag (not a dir move); groups = a manifest `group` field; delete = hard `rmtree` + confirm.
- Rail: collapsible group headers + "No group" + a separate **Archive** view.
- **Resume correctness:** read-only **guard** (HTTP 409 when `finalized` or `archived`);
  `PiProcessManager.spawnFor` now has a `new`/`resume` mode (resume sends
  `/riprendi-sessione <id>`); a "Phase 0 — Resume" cold-start section in `SKILL.md`.
- Design docs: `docs/superpowers/specs/2026-06-29-session-management-design.md` +
  `docs/superpowers/plans/2026-06-29-session-management.md`.

### ⚠️ Open items / pending gates
1. **Resume cold-start STALLS (root cause confirmed 2026-06-30) — NOT fixed.** On
   `/riprendi-sessione` GLM 5.2 narrates the bootstrap step ("esamino la sessione…") then ends
   the turn without the tool call → Pi idle, unrecoverable from the UI (steer doesn't revive an
   ended turn). The new-question path works, so it's resume-specific. Fix direction: harden the
   resume kickoff/SKILL so the model chains into the tool call. Memory:
   `thothii-resume-cold-start-stall.md`. **Do not advertise "Resume" as working until fixed.**
   *(Scheduled as workstream A of the redesign plan above.)*
2. **Full Playwright live-stack verification (MANUAL, not yet run).**
3. **Minor backlog (non-blocking):** explicit id-traversal guard in `delete_session`
   (today gated by `load_session`); `close_session` could reuse `_save_touched` (DRY);
   delete-via-kebab integration test skipped (base-ui Menu portal not drivable in jsdom —
   the dialog itself is unit-tested); a couple of test-file lint nits.
4. **DONE — F1 hang fix live-verified 2026-06-30** (see top section). The live verification
   also surfaced + fixed the reviewer_select `choices` mismatch.
5. **Resolved: settings use `zai/glm-5.2`/medium** (not deepseek-flash); GLM 5.2 drives F1
   fine, just slowly (~3-4 min, ~50+ reads). To tell a truly stalled Pi from a merely-slow one,
   check its sockets/children. Memory: `thothii-glm52-f1-slow-not-stuck.md`.

## Where design history lives
- Specs: `docs/superpowers/specs/` · Plans: `docs/superpowers/plans/`
- SDD execution ledger (gitignored scratch): `.superpowers/sdd/progress.md`
- Auto-memory index: `~/.claude/projects/-Users-mp-projects-ThothII/memory/MEMORY.md`
  (Pi RPC event vocabulary, ui.input id correlation, Omics Portal/GSD design system,
  resume cold-start stall, GLM 5.2 F1 slow≠stuck).

## Git
`main` @ `0eeb3f7`, **pushed to `origin`** (github.com/mptyl/ThothII). Today's commits on top of
`4f60b38`: `418187a` (F1 hang fix) · `d27afd0` (reviewer_select `choices` fix) · `c12bdcd`
(D — YAKE naming) · `0eeb3f7` (E — activity icon/panel). The `feat/ui-redesign-resume` branch ==
`main` (redundant, safe to delete). Other local branches (`feat/settings-menu`,
`feat/thothII-debugging`) are pre-existing, untouched.