diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md new file mode 100644 index 00000000..89da49e2 --- /dev/null +++ b/PROJECT_STATE.md @@ -0,0 +1,114 @@ +# ThothII — Project State + +> Starting-point snapshot for new sessions. Last updated: 2026-06-29. +> Point a fresh session here ("read PROJECT_STATE.md") before substantial work. + +## What ThothII is + +A **human-in-the-loop datamart builder**: it turns a natural-language question into +validated SQL (and optionally a dbt datamart) through a deterministic **8-phase +NL→SQL workflow**, where the model *proposes* and a human *reviewer decides* at gates. +The UI is meant to embed inside the Omics Portal (GSD design system) and is **English**. + +## Architecture — three layers + +``` +frontend (React, :5173) → backend (Fastify, :8787) → pi --mode rpc → tht / harness → DWH (read-only) +``` + +- **harness/** — the Pi layer. A deterministic Python CLI **`tht`** + a Pi gate extension + (`.pi/extensions/tht-gate.js`) that runs the 8-phase workflow and emits/consumes + widget-descriptor JSON. **Owns all persistence.** Workflow truth is `harness/workflow.yaml`; + orchestration rules are `harness/.pi/skills/tht-sessione/SKILL.md`. +- **backend/** — Fastify + TypeScript. A **thin bridge**: proxies REST routes to the `tht` + CLI (`ThtRunner`), manages Pi processes (`PiProcessManager`, one child per session), + bridges Pi RPC events to SSE (`SessionBridge` + `SseHub`). No application database. +- **frontend/** — React 18 + base-ui + Tailwind + TanStack Query + Zustand. Chat-style + shell (`src/shell/AppShell.tsx`); the live transcript is rebuilt in-memory from the SSE + stream (`src/store/sessionStore.ts`), **not persisted**. + +### Persistence model (the load-bearing premise) +There is **no verbatim chat store**. Each workflow phase persists its own document into the +session directory, and that **IS** the persistence. A session = a directory under the +workspace's `sessions/` path containing `session_manifest.yaml` + phase artifacts +(`question.md`, `schema_linking.json`, `cte_plan.json`, `sql_final.sql`, +`validation_report.md`, `review_decisions.jsonl`, …). A fresh Pi process resumes by reading +`tht session show ` + the on-disk artifacts — never by replaying chat. + +## The 8 phases (harness/workflow.yaml) +F1 chiarimento · F2 memoria · F3 riscrittura (`question.md`) · F4 schema_linking +(`schema_linking.json`) · F5 sintesi · F6 cte (`cte_plan.json`, `cte_tests.json`) · +F7 sql_finale (`sql_final.sql`) · F8 datamart. Current phase is a fold over the decision +ledger (`harness/tht/phase.py`); statuses: `open` / `closed` / `finalized`. + +## How to run + +**Full stack (real Pi + DWH):** `./scripts/run-stack.sh` +Prereqs: VPN on; `pi` on PATH (configured model); `harness/.env` populated; +`harness/config/tht.yaml` → a workspace; deps installed in all three projects. +Opens frontend at http://localhost:5173 → backend :8787. + +**Individual dev:** +- backend: `cd backend && npm run dev` (tsx watch; env: `PORT`, `THT_HARNESS_DIR`, `THT_BIN`, `PI_BIN`, `AUTH_MODE`) +- frontend: `cd frontend && npm run dev` (Vite; `VITE_BACKEND_URL` → backend) +- harness install: `cd harness && python -m venv .venv && pip install -e ".[dev]"` → `tht` on PATH + +## How to test (all green as of 2026-06-29: harness 248 / backend 59 / frontend 73) +- harness: `cd harness && .venv/bin/pytest -q` (5 L2/real-DB tests are deselected by default) +- backend: `cd backend && npx vitest run` · typecheck `npx tsc --noEmit -p .` +- frontend: `cd frontend && npx vitest run` · typecheck `npx tsc -b` · e2e `npm run e2e` (Playwright) + +## Config & workspaces +- Workspaces: `harness/workspaces/*.yaml` (`psd`, `tht-test`, `tht.example`). A workspace sets + the DB target and the **absolute** `paths.sessions/artifacts/indexes` (psd → a *separate* + repo `tht-workspace-psd/`, NOT committed here). +- Secrets live ONLY in `harness/.env` (gitignored; `THT_*` — DB, DWH REST, vector, SSL CA…). + See `harness/.env.example` for the variable list. +- App settings (global): `backend/data/settings.json` (gitignored) — `{ workspace, provider, + model, thinking }`. The "New session" form is question-only; these settings supply the rest. + +## Conventions & contracts (don't relearn the hard way) +- **`-c`/`--config` is a PER-COMMAND option in `tht`** — append it AFTER the subcommand, + never globally (`ThtRunner.buildArgv` handles this). +- **`--json` output must be pristine** (only valid JSON on stdout). +- **UI strings are English.** Document *content* stays in the workspace language (Italian + for psd) because it's the real data; only chrome/labels are English. +- **Settings are global**, not per-question. +- TDD throughout; tests assert real behavior, not mocks. Frequent, scoped commits. +- Global user rules (`~/.claude/CLAUDE.md`): think before coding, simplicity first, surgical + changes, goal-driven verification. + +## Most recent feature — Session management (MERGED to main @ 2c21e46) +Full session management modeled on Claude's UI, all three layers: +- **Read-only "split view" panel** (left drawer, `SessionDocumentsPanel`) showing a session's + phase documents read-only (reuses `SqlViewer`/`SchemaLinkingViewer`/`MarkdownView`). +- **Rename / Move to group / Archive / Delete** via a kebab menu (`SessionMenu`) → REST → + `tht session set-name/set-group/archive/unarchive/delete`. Archive = a manifest `archived` + flag (not a dir move); groups = a manifest `group` field; delete = hard `rmtree` + confirm. +- Rail: collapsible group headers + "No group" + a separate **Archive** view. +- **Resume correctness:** read-only **guard** (HTTP 409 when `finalized` or `archived`); + `PiProcessManager.spawnFor` now has a `new`/`resume` mode (resume sends + `/riprendi-sessione `); a "Phase 0 — Resume" cold-start section in `SKILL.md`. +- Design docs: `docs/superpowers/specs/2026-06-29-session-management-design.md` + + `docs/superpowers/plans/2026-06-29-session-management.md`. + +### ⚠️ Open items / pending gates +1. **Resume cold-start e2e (MANUAL, not yet run).** The resume chain is structurally + complete but unproven live: drive a session to F4, stop it, resume in a fresh process, + confirm it re-enters at F4 (not a new-question kickoff). **Do not advertise the "Resume" + button as working until this passes.** +2. **Full Playwright live-stack verification (MANUAL, not yet run).** +3. **Minor backlog (non-blocking):** explicit id-traversal guard in `delete_session` + (today gated by `load_session`); `close_session` could reuse `_save_touched` (DRY); + delete-via-kebab integration test skipped (base-ui Menu portal not drivable in jsdom — + the dialog itself is unit-tested); a couple of test-file lint nits. + +## Where design history lives +- Specs: `docs/superpowers/specs/` · Plans: `docs/superpowers/plans/` +- SDD execution ledger (gitignored scratch): `.superpowers/sdd/progress.md` +- Auto-memory index: `~/.claude/projects/-Users-mp-projects-ThothII/memory/MEMORY.md` + (notes on the Pi RPC event vocabulary and the Omics Portal/GSD design system). + +## Git +`main` @ `2c21e46`, pushed to `origin` (github.com/mptyl/ThothII), in sync. Other local +branches (`feat/settings-menu`, `feat/thothII-debugging`) are pre-existing, untouched.