ThothII/PROJECT_STATE.md

ThothII — Project State

Starting-point snapshot for new sessions. Last updated: 2026-06-30. Point a fresh session here ("read PROJECT_STATE.md") before substantial work.

What ThothII is

A human-in-the-loop datamart builder: it turns a natural-language question into validated SQL (and optionally a dbt datamart) through a deterministic 8-phase NL→SQL workflow, where the model proposes and a human reviewer decides at gates. The UI is meant to embed inside the Omics Portal (GSD design system) and is English.

Architecture — three layers

frontend (React, :5173) → backend (Fastify, :8787) → pi --mode rpc → tht / harness → DWH (read-only)

• harness/ — the Pi layer. A deterministic Python CLI tht + a Pi gate extension (.pi/extensions/tht-gate.js) that runs the 8-phase workflow and emits/consumes widget-descriptor JSON. Owns all persistence. Workflow truth is harness/workflow.yaml; orchestration rules are harness/.pi/skills/tht-sessione/SKILL.md.
• backend/ — Fastify + TypeScript. A thin bridge: proxies REST routes to the tht CLI (ThtRunner), manages Pi processes (PiProcessManager, one child per session), bridges Pi RPC events to SSE (SessionBridge + SseHub). No application database.
• frontend/ — React 18 + base-ui + Tailwind + TanStack Query + Zustand. Chat-style shell (src/shell/AppShell.tsx); the live transcript is rebuilt in-memory from the SSE stream (src/store/sessionStore.ts), not persisted.

Persistence model (the load-bearing premise)

There is no verbatim chat store. Each workflow phase persists its own document into the session directory, and that IS the persistence. A session = a directory under the workspace's sessions/ path containing session_manifest.yaml + phase artifacts (question.md, schema_linking.json, cte_plan.json, sql_final.sql, validation_report.md, review_decisions.jsonl, …). A fresh Pi process resumes by reading tht session show <id> + the on-disk artifacts — never by replaying chat.

The 8 phases (harness/workflow.yaml)

F1 chiarimento · F2 memoria · F3 riscrittura (question.md) · F4 schema_linking (schema_linking.json) · F5 sintesi · F6 cte (cte_plan.json, cte_tests.json) · F7 sql_finale (sql_final.sql) · F8 datamart. Current phase is a fold over the decision ledger (harness/tht/phase.py); statuses: open / closed / finalized.

How to run

Full stack (real Pi + DWH): ./scripts/run-stack.sh Prereqs: VPN on; pi on PATH (configured model); harness/.env populated; harness/config/tht.yaml → a workspace; deps installed in all three projects. Opens frontend at http://localhost:5173 → backend :8787.

Individual dev:

• backend: cd backend && npm run dev (tsx watch; env: PORT, THT_HARNESS_DIR, THT_BIN, PI_BIN, AUTH_MODE)
• frontend: cd frontend && npm run dev (Vite; VITE_BACKEND_URL → backend)
• harness install: cd harness && python -m venv .venv && pip install -e ".[dev]" → tht on PATH

How to test (all green as of 2026-06-30: harness 269 / backend 67 / frontend 87)

• harness: cd harness && .venv/bin/pytest -q (5 L2/real-DB tests are deselected by default)
• backend: cd backend && npx vitest run · typecheck npx tsc --noEmit -p .
• frontend: cd frontend && npx vitest run · typecheck npx tsc -b · e2e npm run e2e (Playwright)

Config & workspaces

• Workspaces: harness/workspaces/*.yaml (psd, tht-test, tht.example). A workspace sets the DB target and the absolute paths.sessions/artifacts/indexes (psd → a separate repo tht-workspace-psd/, NOT committed here).
• Secrets live ONLY in harness/.env (gitignored; THT_* — DB, DWH REST, vector, SSL CA…). See harness/.env.example for the variable list.
• App settings (global): backend/data/settings.json (gitignored) — { workspace, provider, model, thinking }. The "New session" form is question-only; these settings supply the rest.

Conventions & contracts (don't relearn the hard way)

• -c/--config is a PER-COMMAND option in tht — append it AFTER the subcommand, never globally (ThtRunner.buildArgv handles this).
• --json output must be pristine (only valid JSON on stdout).
• UI strings are English. Document content stays in the workspace language (Italian for psd) because it's the real data; only chrome/labels are English.
• Settings are global, not per-question.
• TDD throughout; tests assert real behavior, not mocks. Frequent, scoped commits.
• Global user rules (~/.claude/CLAUDE.md): think before coding, simplicity first, surgical changes, goal-driven verification.

UI/UX redesign + Resume — IN PROGRESS (2026-06-30, evening)

Approved multi-workstream plan: ~/.claude/plans/foamy-forging-dahl.md (read it to resume). Memory: thothii-ui-redesign-inprogress.md. D + E merged @ 0eeb3f7 (pushed); B + C @ b056ff3 (live-verified); F implemented + committed (live check deferred to G). All on main, not yet pushed.

• D — DONE (c12bdcd): session display name = 3-5 Italian keywords via YAKE (no LLM), derived in tht session new (CLI layer); create_session core unchanged (name=None default). yake added to harness/pyproject.toml. TDD tests/test_session_name.py; harness 269 passed.
• E — DONE (0eeb3f7): rotating activity icon replaces the red dot in CentralStatus (inline, clickable → opens the panel); ModelActivityPanel is a 5-line expandable model-stream tail; WorkingSpinner extracted to its own module; the separate spinner button
  • orphaned Transcript.tsx removed. Frontend 87/87, tsc clean. Live visual check DONE (2026-06-30): inline spinner opens the panel; 5-line collapsed tail; expand → full transcript.
• B — DONE (b056ff3): WorkflowBar is now colored dots F1..F8, no phase-name text (amber-translucent=running, green=done, red=error, gray=pending; green connectors lead the active dot). Each dot carries data-state. Error is lightweight: store phaseError set when an info level=error arrives during the phase, cleared on the next ui_request (sessionStore.ts). All four states live-verified via Playwright.
• C — DONE (b056ff3): right sidebar — single-line denser rows (inline status dot + name, py-1), a 3-level type hierarchy via /impeccable (L1 SESSIONS red/bold/wide-tracking · L2 section + group headers muted uppercase · L3 names normal-case), and the "No group" label removed (ungrouped sessions render after the last group; guarded so the empty-state still teaches when there are no groups). Live-verified. (Resume in SessionMenu stays with A1.)
• Tests: frontend 93/93 (was 87; +3 store phaseError, +2 WorkflowBar dot-state, +1 AppShell no-"No group"), tsc -b clean.
• F — DONE (uncommitted; live check deferred to G): single-select answers auto-confirm. reviewer_select options may carry a decision payload ({type, subject, detail?, rationale?}) and an optional advance; picking such an option persists the decision directly via tht decision add (shared decisionAddArgs helper, also used by reviewer_decide) — no redundant reviewer_decide/reviewer_confirm gate. Options without a payload stay ask-only; back/exit/Other never persist. Pure logic extracted to resolveSelectOutcome/decisionAddArgs (exported, unit- tested). Contract docs updated: reviewer_select tool desc + SKILL.md (widget summary, disciplines 2-3, Phase-1 single-pick) + the CLAUDE.md gate note. Gate JS 33/33, harness 269. Live verification (model actually uses reviewer_select+decision, no follow-up gate, decision in review_decisions.jsonl) deferred to G — it is model-behavior-dependent.
• A (pending, riskiest): Resume command + FIX the resume cold-start stall (open item #1).
• G (later): cross-model behavior matrix (Qwen3.6 / GLM 5.2 / Deepseek V4 / others) — also the home for F's live verification.

Next chunk: A (Resume in the kebab + the resume cold-start stall fix, diagnosis-first). G (cross-model, incl. F's live check) later.

Live verification + reviewer_select fix (2026-06-30, afternoon)

Drove the real stack (Playwright → backend → real Pi → GLM 5.2 → DWH) end-to-end.

• F1 hang fix (418187a) VERIFIED LIVE. Answered an F1 reviewer widget; Pi resumed (model socket reopened) and the gate produced new output — vs the old silent hang. The transition "silent hang → gate re-presents/advances" proves ctx.ui.input now resolves.
• New bug found + fixed: reviewer_select choices vs choice. The gate's reviewer_select (and reviewer_confirm reject) read resp.choice (singular) but the frontend uniformly sends choices: [id] (array) — so every single-select gate answered "Nessuna scelta ricevuta" and re-proposed forever (multiselect was fine; it already read choices). Fix: a shared selectedChoice(resp) helper (harness/.pi/extensions/tht-gate.js) reading the array; both handlers use it. TDD: gate/__tests__/gate_choice.test.js RED→GREEN, full gate suite 28/28. VERIFIED LIVE: a single-select answer is now accepted and the workflow advances (2/4 → 3/4).
• Resume cold-start STALL confirmed (open item #1). On /riprendi-sessione, GLM 5.2 narrates the bootstrap step then ends the turn without the tool call → Pi idle, unrecoverable from the UI. Memory: thothii-resume-cold-start-stall.md.
• GLM 5.2 F1 is slow (~3-4 min, ~50+ reads) but works — looks stuck but isn't; don't hit "Stop and save" (it POST /closes → kills Pi). Memory: thothii-glm52-f1-slow-not-stuck.md.

Earlier work — F1 reviewer-widget hang fix + multiselect guidance (committed 2026-06-30; authored 2026-06-29)

Two fixes, committed to main (7 files):

1. Bug: every reviewer widget hung "stuck with no output" after the human answered — F1 disambiguation (and any gate) dead-ended. Root cause, confirmed from Pi's own source (@mariozechner/pi-coding-agent dist/modes/rpc/rpc-mode.js, createDialogPromise): ctx.ui.input assigns its OWN RPC id (crypto.randomUUID) and correlates extension_ui_response on THAT id, silently dropping unknown ids. The gate puts a different id (u${Date.now()}) inside the descriptor carried in title. SessionBridge was replying with the descriptor id, so real Pi never resolved ctx.ui.input → the model never continued. Fix: SessionBridge now stores Pi's top-level m.id (pendingPiId) on the incoming request and replies extension_ui_response{ id: pendingPiId, value: <uiResponse JSON> } (value still carries the descriptor id, so the gate's internal resp.id === descriptor.id check holds). File: backend/src/bridge/session-bridge.ts. Full write-up: memory pi-ui-input-id-correlation.md.

   • The test double was masking it: harness/tests/fake_pi/fake_pi_rpc.mjs had forced m.id == descriptor.id. Corrected to mirror real Pi (distinct randomUUID top-level id, correlate on it, drop unknown ids); test_fake_pi_contract.mjs gained a negative regression test ("respond with descriptor id → no follow-up").
   • TDD: backend/test/session-bridge.test.ts (unit) + backend/test/e2e-f1.test.ts (integration — now asserts the model's follow-up arrives after the answer) went RED→GREEN.

2. UX: multi-answer disambiguation — harness/.pi/skills/tht-sessione/SKILL.md Phase 1 now tells the model to use reviewer_decide (the existing multiselect/checkbox widget) when an ambiguity admits several simultaneously-true answers, instead of single-pick reviewer_select. Guidance-only — no new widget (frontend MultiselectWidget already exists).

Verified at commit time: backend npx vitest run 67/67 green; tsc --noEmit -p . OK; fake-pi contract node --test test_fake_pi_contract.mjs 2/2 green. Verified LIVE 2026-06-30 (see the top "Live verification" section).

Most recent feature — Session management (MERGED to main @ 2c21e46)

Full session management modeled on Claude's UI, all three layers:

• Read-only "split view" panel (left drawer, SessionDocumentsPanel) showing a session's phase documents read-only (reuses SqlViewer/SchemaLinkingViewer/MarkdownView).
• Rename / Move to group / Archive / Delete via a kebab menu (SessionMenu) → REST → tht session set-name/set-group/archive/unarchive/delete. Archive = a manifest archived flag (not a dir move); groups = a manifest group field; delete = hard rmtree + confirm.
• Rail: collapsible group headers + "No group" + a separate Archive view.
• Resume correctness: read-only guard (HTTP 409 when finalized or archived); PiProcessManager.spawnFor now has a new/resume mode (resume sends /riprendi-sessione <id>); a "Phase 0 — Resume" cold-start section in SKILL.md.
• Design docs: docs/superpowers/specs/2026-06-29-session-management-design.md + docs/superpowers/plans/2026-06-29-session-management.md.

⚠️ Open items / pending gates

1. Resume cold-start STALLS (root cause confirmed 2026-06-30) — NOT fixed. On /riprendi-sessione GLM 5.2 narrates the bootstrap step ("esamino la sessione…") then ends the turn without the tool call → Pi idle, unrecoverable from the UI (steer doesn't revive an ended turn). The new-question path works, so it's resume-specific. Fix direction: harden the resume kickoff/SKILL so the model chains into the tool call. Memory: thothii-resume-cold-start-stall.md. Do not advertise "Resume" as working until fixed. (Scheduled as workstream A of the redesign plan above.)
2. Full Playwright live-stack verification (MANUAL, not yet run).
3. Minor backlog (non-blocking): explicit id-traversal guard in delete_session (today gated by load_session); close_session could reuse _save_touched (DRY); delete-via-kebab integration test skipped (base-ui Menu portal not drivable in jsdom — the dialog itself is unit-tested); a couple of test-file lint nits.
4. DONE — F1 hang fix live-verified 2026-06-30 (see top section). The live verification also surfaced + fixed the reviewer_select choices mismatch.
5. Resolved: settings use zai/glm-5.2/medium (not deepseek-flash); GLM 5.2 drives F1 fine, just slowly (~3-4 min, ~50+ reads). To tell a truly stalled Pi from a merely-slow one, check its sockets/children. Memory: thothii-glm52-f1-slow-not-stuck.md.

Where design history lives

• Specs: docs/superpowers/specs/ · Plans: docs/superpowers/plans/
• SDD execution ledger (gitignored scratch): .superpowers/sdd/progress.md
• Auto-memory index: ~/.claude/projects/-Users-mp-projects-ThothII/memory/MEMORY.md (Pi RPC event vocabulary, ui.input id correlation, Omics Portal/GSD design system, resume cold-start stall, GLM 5.2 F1 slow≠stuck).

Git

main @ 0eeb3f7, pushed to origin (github.com/mptyl/ThothII). Today's commits on top of 4f60b38: 418187a (F1 hang fix) · d27afd0 (reviewer_select choices fix) · c12bdcd (D — YAKE naming) · 0eeb3f7 (E — activity icon/panel). The feat/ui-redesign-resume branch == main (redundant, safe to delete). Other local branches (feat/settings-menu, feat/thothII-debugging) are pre-existing, untouched.