# ThothII — Project State

> Starting-point snapshot for new sessions. Last updated: 2026-08-08 (final verification).
> Point a fresh session here ("read PROJECT_STATE.md") before substantial work.

## P1 configuration-process automated integration — PASS 2026-08-09

- Retained evidence: `.artifacts/p1-integration/p1-f9327d6f41dafbd8a8cd8f6c9efc0276/report.md`
- automated integration: PASS
- manual acceptance: PENDING
- The contextual negatives use a separate `invalid-context` branch, remote, checkout, data,
  runtime, registry, and second production Fastify listener, all beneath the owned run root.
  Persisted before/after semantic-state proofs show primary `main` remains valid, a missing-tree
  publish is state-neutral, and an invalid pull advances only its disposable checkout while the
  remote and last-valid active/snapshot/data/runtime state remain unchanged.
- The retained audit hashes every request/response and descriptor/negative fixture, production
  Git Trace2 plus runner command events, and all declared artifacts with unique paths. The final
  whole-run, reachable-Git, and virtual-report scan found no fixture canaries outside the excluded
  secret fixture directory.

## Internal Qdrant + Ollama semantic infrastructure — LIVE 2026-08-08

- **Compose topology.** The mandatory application stack is `frontend`, `core`, `qdrant`,
  `embedding`, and the one-shot `embedding-model-init`. Startup is CPU-first by default; Linux
  hosts may opt into GPU exposure with `THOTH_ENABLE_EMBEDDING_GPU=1`. Qdrant is private on the
  Compose network and persists `/qdrant/storage` in `qdrant-data`. Ollama persists its local model
  cache in `embedding-models`, and `embedding-model-init` blocks `core` until
  `qwen3-embedding:0.6b` is present.
- **Semantic contract.** Internal semantic indexing is fixed to `qwen3-embedding:0.6b`,
  `1024` dimensions, and cosine distance. Schema-v3 descriptors are operational; schema-v1/v2 descriptors remain `migration_required` until an explicit reviewed migration writes schema version 3. One workspace owns one Qdrant collection, and schema, Evidence, and Memory records
  coexist inside that collection with payload `kind` separation.
- **Final review runtime barriers.** Operational routes, retained session pins, and runtime
  rendering now require schema version 3 before resolving bindings, readiness, diagnostics, or
  Pi. Session admission verifies the exact internal Qdrant collection (dimensions, cosine
  distance, and required keyword payload indexes) before Ollama and before manifest persistence.
  The Qdrant adapter binds every search/list/delete filter to its constructed workspace identity
  and rejects conflicting caller namespaces.
- **Boundary and persistence.** Only DWH and LLM remain external runtime application endpoints.
  There are no active external vector or embedding endpoint instructions, bindings, or secrets in
  the supported operator manuals. Qdrant remains a derived but persistent semantic index: the
  canonical sources of truth stay the workspace Git descriptors, phase artifacts, and memory
  registry/ledger. The Ollama model cache is recoverable for offline startup but is not the
  canonical source of semantic content.
- **Backup and recovery.** `./scripts/vector-backup.sh --project-name <name> --output <file>`
  archives exactly one labeled `<project>_qdrant-data` volume and preserves the prior `qdrant`
  running state. `./scripts/vector-restore.sh --project-name <name> --input <file>
  --confirm-project <name>` requires the exact repeated project confirmation, validates manifest
  and archive safety before stopping `qdrant`, stages rollback content, restores in place, and
  restarts `qdrant` only if it was previously running. Restore does not migrate legacy workspace
  descriptors, rename collections, or repair a semantic-index incompatibility. Backup and restore
  share one atomic Docker-daemon lock per Compose project/Qdrant volume; contenders fail before
  volume resolution, and cleanup removes the lock only when its ownership labels still match.
- **Verification recorded for Task 13 final audit.** On Apple M4 Pro
  (`Darwin 25.5.0`, Docker Server `29.6.2 linux/arm64`), harness pytest passed
  **827 passed / 4 deselected**; backend Vitest passed **477/477** plus TypeScript and build;
  frontend Vitest passed **374/374** plus TypeScript and build; `git diff --check` passed.
  Deployment contracts passed:
  `test-default-compose.sh`, `test-unified-compose.sh`, `test-internal-semantic-compose.sh`,
  `test-no-deployment-coupling.sh`, `test-compose-secret-policy.sh`, and
  `verify-workspace-install-docs.sh --fixtures-only`.
- **Task 13 Docker smoke evidence.** CPU semantic smoke passed in **217.34s** and proved
  offline Qdrant/Ollama persistence plus exact cleanup. Workspace registry smoke passed in
  **42.06s** in fix round 1 with a per-run image tag derived from the unique Compose project,
  and proves exact cleanup of compose containers, volumes, networks, and only that smoke image.
  Unified deployment smoke passed in **125.57s**; update-only rollback smoke
  passed in **85.40s**; Linux server deployment smoke passed in **55.99s**. The previously
  observed `thothctl` rollback failure did not recur.
- **Task 13 image and manual-gate notes.** Verified pinned runtime images:
  `qdrant/qdrant:v1.18.2@sha256:75eab8c4ba42096724fdcfde8b4de0b5713d529dde32f285a1f86fdcb2c9e50c`
  and
  `ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a`.
  The workspace-registry smoke fix-round image used tag
  `thothii-workspace-registry-smoke:thoth-workspace-registry-smoke-thoth-workspace-registry-smoke-10vi3a-19157`,
  built manifest list `sha256:715b943057929418cad4aa71806d9edbaf823555d19bda6b875297617463fd4a`
  with config `sha256:a566521981e08958aae9a12bfc7803bb5f3f835536b4bb8c39df8fcf26063161`,
  and removed that exact reference during cleanup. Local GPU exposure (`THOTH_ENABLE_EMBEDDING_GPU=1`) and
  Windows Docker Desktop startup were not manually executed in this run.
- **Task 13 known limitations.** Broad harness Ruff remains existing unrelated debt
  (**220 errors**); touched harness files were verified Ruff-clean. The final active-reference
  audit remains non-empty only in categorized legacy parser/migration compatibility, legacy
  descriptor/config fixtures, deterministic negative guards, retained off-repository migration
  SQL, L2 legacy fixtures, gitignored task notes, and historical reference notes. No active
  schema-v3 operator manual or supported runtime deployment path retains external vector or
  embedding endpoint coupling.
- **Final review fix verification.** Backend Vitest passed **477/477** plus TypeScript and build;
  harness pytest passed **827 passed / 4 deselected** with the existing 74 warnings; touched Python
  files are Ruff-clean. The complete `thothctl` Go suite, deterministic backup/restore safety test,
  internal semantic Compose contract, no-deployment-coupling gate, CPU/offline semantic smoke, and
  unified deployment smoke all pass after the final fix. The intermittent `thothctl` rollback failure was
  traced to Docker Desktop alternating equivalent bind sources between `/private/...` and
  `/host_mnt/private/...`. Exact state-v4 source hashes remain unchanged; only fresh bind
  observations made by a Darwin `thothctl` carry non-serialized aliases for the rollback
  comparison, so pre-fix recovery state remains readable and Linux `/host_mnt` paths remain
  distinct. The rollback-only smoke passed twice consecutively after each fix revision, and the
  subsequent full unified smoke passed with exact cleanup.

## Historical snapshots and archived reference notes

### Historical snapshot — Unified deployment release gate, Task 13 (2026-08-05)

- **Release coverage.** `scripts/unified-deployment-smoke.sh` gates the two-service render/build,
  frontend-to-core routing, embedded pinned Pi, Git registry bootstrap, offline recreation, valid
  update, invalid-update retention, and the four persistent stores. `scripts/thothctl-update-smoke.sh`
  independently exercises the bad-Pi update and automatic rollback path.
  `scripts/server-deployment-smoke.sh` starts the server plus required session overlays with the
  same smoke-built core/frontend images, disposable bind roots/secrets/session configuration,
  upstream-auth checks, and fail-closed unavailable-session behavior.
- **Isolation and disclosure boundary.** Every run generates a unique temporary root, Compose
  project, container/image names, transaction image tags, and run label. The rollback fixture uses
  an immutable `hello-world` digest whose preflight exits successfully, guaranteeing the stopped
  core state required by `thothctl` compensation. Cleanup includes stopped project containers in
  its final ownership check immediately before teardown and removes only exact containers,
  Compose resources, image references, control state, and temporary files. There is no global
  prune. Failure diagnostics are bounded and sanitized, and all credentials/endpoints used by the
  smokes are disposable fixtures rather than operator or repository secrets. Every public smoke
  also has an internal 30-minute process-group supervisor with TERM/KILL of the complete group.
- **Cross-platform CI contract.** `.github/workflows/deployment.yml` uses immutable action commits,
  pinned supported Node and Go versions, runs LF/Compose/secret/coupling/docs/TypeScript gates on
  Linux, runs each Linux Docker smoke once under its own outer timeout, and copies the Windows
  source into a path containing spaces before building/invoking native `thothctl` and rendering
  Compose. The optional `windows_docker_startup` dispatch targets a labelled self-hosted Windows
  Docker Desktop/WSL2 runner and performs bounded two-service startup and exact cleanup. No local
  Windows or Windows Docker execution is claimed until that manual job is recorded.
- **Validation status.** Deterministic Phase A gates, backend **434/434** plus TypeScript,
  frontend **386/386** plus TypeScript, and harness **862 passed / 5 L2 deselected** are green.
  Review round 1 ran each Docker smoke exactly once without retry. Unified (`103.86s`) and
  update-only (`46.45s`) passed build/start, core/Pi/registry/persistence setup and the stopped
  candidate preflight, but `thothctl` stopped before mutation at its active-session inventory gate.
  Round 2 replaces presence-only fixture checks with generated Compose renders plus the production
  workspace resolver; this found and fixed missing explicit direct transport selections. The
  clean-server preflight now atomically initializes the three hidden Pi-agent targets under the
  writable parent bind while protected/tracked sources remain separate read-only mounts. Clean
  empty-root render/setup and wrong-service/value/mount mutations are green. The corrected server
  one-shot built and started both healthy services from an empty Pi-state root, then stopped at an
  incorrectly addressed authenticated frontend hop. Fix round 3 adds the exact fourth private
  non-admin claim and proves its nginx/backend transformation in a focused auth test. It also
  centralizes schema-v2 registry descriptor resolution and secret-safe runtime rendering in
  `ThtRunner`, preserving canonical revision identity and durable session roots for inventory,
  create/resume/show, SQL, and Pi calls. The fresh update-only one-shot now passes mutation,
  automatic `rolled_back` compensation, exact prior-image restoration, unchanged registry head
  and mount identities, all four persistence sentinels, post-rollback doctor/workspace checks,
  and exact labeled-resource cleanup. The one authorized server invocation was blocked at its
  first Docker readiness call by the execution sandbox's socket permission before any Compose
  resource could be created, so authenticated workspace/fail-closed session behavior remains an
  explicit release gate. Native Windows PowerShell/Docker execution also remains pending.

### Historical snapshot — Portable deployment decoupling (superseded 2026-08-08)

- **Mandatory stack.** The supported Compose stack is exactly `frontend` plus `core`; use the
  base file with `deploy/compose.local.yaml`, or with `deploy/compose.server.yaml` plus the
  required public-server session overlay. `run-stack.sh`
  invokes the base+local Compose command and the core image provides Pi, so no host Pi binary is
  part of the launch contract.
- **External boundaries.** DWH, vector DB, embedding, LLM, and reverse-proxy services are
  external configurable endpoints even when deployed on the same infrastructure. The two
  superseded PSD/portal deployment overlays were removed. Workspace descriptors and migration
  utilities remain separate from deployment runtime configuration.
- **Legacy PSD deployment ruling.** The PSD bootstrap was deleted because it generated the
  retired overlay and was therefore deployment machinery, not a data migration utility. Its
  remaining live contract checks were renamed for the generic local Compose profile. The coupling
  gate rejects stale active deployment filenames and content while deliberately excluding
  historical plans/specs, canonical workspace descriptors, and non-runtime migration helpers.
- **Fresh provider and secret contract.** Local, server, and standalone development mount the
  protected Pi auth JSON plus tracked declarative model/settings files read-only under
  `/home/thoth/.pi/agent`. The existing strict application bundle is a core-only Docker secret at
  `/run/secrets/thothii.secrets`; operator env files contain only its absolute source path.
  Provider readiness is exercised from a fresh Compose volume through model listing, configuration,
  and sanitized credential status.
- **Install and scan closure.** Superseded copied one-service installation examples and the
  provider-owned-network test are retired. Active manuals use the canonical base plus local/server
  and optional overrides, while the category-based coupling scan covers runtime, Docker smoke,
  install, operator, and positive deployment-test contracts and propagates scanner errors.

### Historical snapshot — Portable Git workspace registry, pre-schema-v3 (superseded 2026-08-08)

- **Source of truth and scope.** The canonical workspace repository is a generic Git remote,
  configured only by `THT_WORKSPACE_GIT_REMOTE` and `THT_WORKSPACE_GIT_BRANCH` (there is no
  committed PSD/Chirone remote or branch default). Both a local Docker installation and a server
  persist its checkout, validated snapshots, state, and locks at `/data/workspace-registry`.
  Connector endpoints, transport choices, and secret-file paths remain local bindings; secret
  contents are never stored in Git, API responses, browser storage, diagnostics, or bundles.
- **Migration and session safety.** Schema-v2 descriptors are operational; legacy descriptors are
  visible as `migration_required` until migrated by the documented operator workflow. New sessions
  acquire a persistent revision lease before readiness and persist workspace ID plus immutable Git
  revision. Retention hands that lease off only after an authoritative scan observes the manifest,
  so a stale concurrent scan cannot prune the pinned snapshot. Resume resolves that historical
  snapshot, while retention preserves every revision referenced by an open, closed, or failed
  unarchived manifest.
  Reconciliation runs only with a complete local installation list or an administrator's complete
  server list, never from a remote user's partial view.
- **SSH connector boundary.** The current OpenSSH forward is owned by one bounded diagnostic and is
  always cleaned up afterward. DWH/vector `ssh_tunnel` bindings therefore return
  `workspace_not_activatable`, and new-session creation rejects them before persistence. Direct and
  REST runtime connectors remain supported; Git remote access over SSH is unaffected.
- **Operator manuals.** Follow [the local manual](docs/install/local-workspace-registry.md) for
  macOS/Windows/Linux Docker Desktop deployment and [the server manual](docs/install/server-workspace-registry.md)
  for Gitea-compatible remotes, reverse proxy, migration, backup, and recovery. The release
  workflow is Git review/push → installation pull → validate → local diagnostic test → browser-local
  workspace/model/reasoning selection → revision-pinned session.
- **Verification recorded for this source branch.** `git diff --check` passed; backend Vitest
  **371/371** and TypeScript passed; frontend Vitest **398/398** and TypeScript passed; the
  harness document regression passed **10/10**. `./scripts/workspace-registry-smoke.sh` and the
  executable installation-manual fixture verifier passed with Docker. A final unrestricted full
  harness run remains a release command for the deployment environment; the earlier local
  long-running harness run was intentionally cancelled before it produced a final result.

### Historical snapshot — Session summary redesign (2026-07-23)

- Session documents are projected at read time in outcome-first order: original question,
  final SQL, persisted data preview, revised question, assumptions, one memory list, then
  remaining technical documents. This applies to existing filesystem and repository-backed
  sessions without rewriting their artifacts.
- Final SQL has an always-visible clipboard action. All prose, including original/revised
  questions, assumptions, memory content, and remaining decisions, renders as Markdown.
- Memories are shown once, approved before declined. The generic decision list suppresses
  memory ledger records plus `phase_approved`, `phase_auto_approved`, `table_approved`,
  `table_promoted`, and `column_promoted`.
- The session summary has its own accessible pointer/keyboard resize separator, persists its
  width independently from Model activity, reaches 50% when space permits, and preserves a
  512 px right-side minimum on narrower desktop layouts.
- Verification: harness **861 passed / 5 L2 deselected**, Pi gate **163/163**, full frontend
  suite, TypeScript check, production build, Ruff, and `git diff --check` all passed. A live
  pre-existing session returned the new canonical order and none of the suppressed decision
  labels. Compose rebuilt and force-recreated both services; core image
  `sha256:3566d1258b956f8ca96d3b5ff8f625503247a7fd3b0dffe77020ae03956403d8`
  is healthy and frontend image
  `sha256:8311ca1308b459ece7236bf143da7b1a226ff4082fed924e1b5a207c24b6ca29`
  is running. Frontend and `/api/health` both returned HTTP 200.

### Historical snapshot — Local Pi user auth + startup failure handling (2026-07-21)

- The PSD Docker profile now bind-mounts the configurable host `PI_AUTH_FILE` read-only at
  `/home/thoth/.pi/agent/auth.json`; on this Mac it resolves to the real user profile
  `/Users/mp/.pi/agent/auth.json`. The container keeps its correct Linux identity
  `HOME=/home/thoth` while Pi sees the user's independent `deepseek` and `zai` credentials.
- `deploy/pi/settings.json` is the non-secret model policy and exposes, in order,
  `zai/glm-5.2`, `deepseek/deepseek-v4-flash`, `deepseek/deepseek-v4-pro`, and
  `aritmolab/qwen3.6-35b-a3b`. The core image is aligned to Pi 0.80.3.
- New-session creation now validates the saved provider/model against Pi before persistence;
  unavailable selections return sanitized `503 model_unavailable` without creating a manifest.
  A synchronous runtime-construction failure after persistence marks that session `failed` and
  returns the fixed startup-recovery message instead of leaving an ambiguous `open` session.
- Verification: backend 235/235, TypeScript clean, dedicated Compose auth/model contract green
  with a demonstrated RED→GREEN cycle. Rebuilt core image
  `sha256:a8b4dd9f016c2335e4da897073bc6d5bdf1e8ce9b60dcfcca171b6677f563228`
  is healthy; live `/models` returned all four models; a real `deepseek-v4-pro` smoke reached its
  first reviewer gate, deleted only its own session, and restored the exact prior settings.
- Deleted the three explicitly approved incomplete DeepSeek attempts:
  `a390c8b8-0a91-4a37-967b-ce7ff9be9797`, `a2f974b2-4c48-4967-b4b6-afdbc2b2d541`, and
  `f66e1959-3c71-4b10-8aa1-606992046b7e` (API delete 204, subsequent lookup 404 for each).

### Historical snapshot — User-owned sessions cutover (2026-07-16)

- **Target contract:** the public server runs `AUTH_MODE=upstream` with Task 4 portal identity
  forwarding and Task 5 principal enforcement deployed together. Its session source of truth is
  direct TLS-verified PostgreSQL `thoth_sessions`; local development remains loopback-only with
  filesystem sessions under `THT_HOME`. The core never receives the migrator credential.
- **Deployment material:** copy `deploy/compose.session-server.yaml.example` and
  `deploy/workspaces/server-sessions.yaml.example` into reviewed, untracked operator files. The
  runtime password, migrator password, and CA are three separate Docker secret mounts; server
  startup rejects public/local storage and incomplete server DB/TLS configuration.
- **Readiness behavior:** `/health` remains the unauthenticated process liveness endpoint. Any
  route requiring unavailable session/preferences storage returns fixed HTTP 503 before starting
  Pi; this is intentional and must not be hidden by changing liveness to a database check.
- **Manual cutover only:** schedule maintenance, drain Pi work, run the one-shot migrator and
  require `pending=[]` and `drifted=[]`, then replace core and perform an authenticated storage
  smoke. Archive/checksum the three reviewed legacy filesystem session directories before deleting
  exactly those three with `docker/cutover-legacy-sessions.sh --delete`; no deletion has been run
  from this repository task. Do not import their untrusted ownership.
- **Rollback:** PostgreSQL remains the single source of truth. Revert only to a compatible fixed
  release; never re-enable filesystem persistence, restore the archive into production, or
  dual-write during rollback.

### Historical snapshot — Docker locale deployment, Profile A (superseded 2026-08-05)

ThothII gira in Docker sul server co-locato, **embedded nel portale omics_portal** a `https://aritmolab.policlinicosandonato.it/datamart-builder` (backend invisibile, tutto same-origin via nginx del portale).

- **2 container** su `compose.yaml`: `thothii-core` (Fastify + harness tht + Pi) + `thothii-frontend` (Vite + nginx-unprivileged). Rete `omics_portal_omics_network` (external) con alias `thothii-core`/`thothii-frontend`.
- **DB**: Postgres diretto `:5438` (stessa istanza: schema `datawarehouse` 163 tabelle + `vectors` pgvector). Ruoli dedicati `thoth_dwh_reader` (read-only) + `thoth_vector_rw` (read+write). Embeddings: Ollama `:11434`.
- **Secrets**: `deploy/thothii.env` (env_file, gitignored) + `THT_MODEL_API_KEY_FILE` (key modello, file 0600 — meccanismo provider-credentials di Codex) + bind-mount `~/.pi` (pi-config).
- **Backend**: merge di `codex/portable-deployment` (secret-bundle, provider-credentials, auth `upstream`, security hardening, CI multiarch). Setup Docker MIO tenuto (il modello Docker-secrets di Codex è in `deploy/` come alternativa inerte).
- **Portale** (repo `omics_portal`, branch `agent/patient-capabilities-datamart-ui`): `nginx.conf` rotte `/datamart-builder/api`+`/assets` + `auth_request`, template `datamart_builder.html` (mount `<div id="root">` + tag `{% vite_assets %}`), vista `datamart_builder_api_auth`. Auth: `authentik Admins` bypass; utenti normali necessitano gruppo `omics-datamart-builder`.
- **Fix load-bearing**: `configPath` da `THT_CONFIG` (route senza workspace), `vite_assets` `mark_safe` (SPA bianca), `COPY harness/`+`cp workflow.yaml`+`pip install .` (pip 26 / tht module-relative), entrypoint `server` case.
- **Standalone/dev**: `docker-compose.dev.yml` (rete propria, porte host 8787/8090) + `scripts/docker-smoke.sh`.
- Piano dettagliato: `docs/superpowers/plans/2026-07-12-local-docker-deploy-implementation.md`.

### Archived snapshot — Runtime incident fixes (2026-07-13)

- The bind-mounted Pi profile came from host paths and did not trust `/app/harness`.
  Pi 0.80 consequently loaded **zero** project extensions, prompts and skills, silently
  sending `/nuova-domanda`/`/riprendi-sessione` to the model as plain text. The core
  entrypoint now idempotently adds only `/app/harness` to the persistent
  `/home/thoth/.pi/agent/trust.json`, preserving all existing decisions.
- The gate embeds the canonical `tht-sessione/SKILL.md` in the one-shot kickoff system
  prompt and explicitly prohibits repository discovery. A live RPC `get_commands` must
  show `torna`, `nuova-domanda`, `riprendi-sessione`, and `skill:tht-sessione` after deploy.
- Workspace identity is derived from the resolved config path, so
  `config/tht.yaml -> workspaces/local.yaml` matches DWH artifact ownership (`local`).
- Direct pgvector now discovers the actual namespaces of the `vector` type and cosine
  operator from PostgreSQL catalogs. This supports server layout `vectors.*` tables with
  the extension installed in `public`.
- Live verification: session `2026-07-13-074712-dammi-la-lista-dei-pazienti-che-haoo-fat`
  resumed directly at F1, ran `tht session show`, and completed `tht search pack`
  (12 tables, 0 evidence, 2 solved) without repository exploration or adapter errors.

### Archived snapshot — Workflow/UI regression fixes (2026-07-14)

- **F1 Model Activity restored.** Session create/resume now preserves configured/persisted
  thinking instead of forcing `off`. Pi's nested `thinking_delta` is bridged to a dedicated
  named SSE `activity_delta`; EventSource subscribes to that name and the panel keeps it separate
  from final assistant text. Reasoning remains in-memory and is not persisted to session artifacts.
- **F3 rewrite confirmation remains bypassed.** `rewrite_question` records approval and advances
  automatically without a reviewer widget. The repeated prompt came from old running containers:
  images had been rebuilt but services had not been recreated.
- **Join review is read-only and complete-set safe.** Join-only proposals render informational
  cards with only `Continue` and `Other — specify`. Continue requires the exact complete id set;
  all joins are persisted together by `decision add-join-set`, using an atomic ledger replacement
  under a per-session cross-process writer lock. Other persists none of the rejected proposal.
- **CTE presentation fixed.** F6 CTE cards now structure purpose, rationale, tables, filters, keys,
  and output columns with responsive wrapping/alignment. The Horizontal/Vertical switch is hidden
  for a single SQL block (the per-CTE view), because it only affects multi-block layouts.
- **Latest render failure diagnosed and hardened.** Session
  `2026-07-14-115847-estrai-i-pazienti-che-hanno-fatto-un-abl` sent an object in
  `open_questions`, which React cannot render as a child. The v2 gate now enforces
  `open_questions?: string[]`; the frontend also safely normalizes legacy malformed payloads.
- **Verification/deploy:** Python harness 798 passed / 5 L2 deselected; gate JS 126; backend 143;
  frontend 250; TypeScript/build gates green. Compose rebuilt and force-recreated both services.
  Running image ids: core `sha256:55acef2f12151ea97144c2f5e9164d63f2ca734bc2746fef553df94849e3fb3f`;
  frontend `sha256:1043f79392420149655cc63d70461e2ca2005b2290a1e3e21dcf845ec3bd1c81`.

### Archived snapshot — Pi-enabled model selector (2026-07-14)

- **Pi is the allowlist authority.** `/models` reads the mounted Pi `enabledModels`, intersects
  it with models currently available from Pi, and preserves the configured order. Enumeration
  does not require `PI_PROVIDER`, does not inject generic/provider credentials, and fails closed
  for missing or malformed scope.
- **Live scope:** exactly `deepseek/deepseek-v4-flash`, `zai/glm-5.2`, and
  `local-qwen/qwen3.6-35b-a3b`. The live endpoint returned those three composite IDs once each and
  in that order; `zai/glm-5v-turbo` and all other authenticated Pi models are hidden.
- **Validation/process smoke:** live settings updates returned 200 for DeepSeek Flash and local
  Qwen, while hidden GLM-5V returned 400; a post-restore equality check confirmed the original app
  settings were restored. The real `PiProcessManager` configure path succeeded for DeepSeek and
  local Qwen without sending a prompt or starting a DWH operation; local Qwen required no hosted
  provider key. Unknown and compound providers remain fail-closed in the verified backend suite.
- **Verification/deploy (`2026-07-14T19:24:22+02:00`):** backend **154/154** and frontend
  **251/251** passed; both TypeScript gates and `git diff --check` were green. Compose built and
  force-recreated only `core`; container start was `2026-07-14T17:24:01.992971976Z`, health was
  `healthy`, and sanitized post-recreate logs contained only the backend listen line. Rebuilt image
  ID and running container image ID both equal
  `sha256:577f99754fd0731251c8ddd8608b1b8baee09d02fad66c759b23f8221083e676`.

### Archived snapshot — Qwen connectivity + state-aware Resume recovery (2026-07-14)

- **Pi turns have an explicit lifecycle.** The bridge tracks `idle`, `running`, `waiting`,
  and `failed`; a reviewer gate is `waiting`, responses/steering return to `running`, and an
  assistant provider error or unexpected Pi child exit becomes `failed`. Provider error details
  are never forwarded to the client; the UI receives a fixed sanitized recovery message.
- **Resume preserves only active work.** `running`/`waiting` runtimes return as already active.
  Every validated cold path—including recovery after a child has already exited—clears stale SSE
  state before reopening and restarts from persisted provider/model/thinking with
  `/riprendi-sessione`; `idle`/`failed` runtimes are torn down at that point. Failed validation does
  not detach the existing stream. A successful Resume of the currently selected session also
  closes and recreates its EventSource, so the replacement runtime cannot be left behind an old
  same-ID stream.
- **Private Qwen routing is live.** Core is attached to both `omics_portal_omics_network` and
  external `localllm_default`; frontend remains only on the portal network. The mounted Pi profile
  resolves `local-qwen/qwen3.6-35b-a3b` at the sanitized base URL
  `http://localllm-vllm:8000/v1`. A direct probe from core verified the model catalog and received
  a non-empty real chat completion.
- **Verification/deploy (`2026-07-14T21:41:52+02:00`):** backend **168/168** and frontend
  **256/256** passed; both TypeScript gates, both production builds, the Qwen Compose network
  contract, and `git diff --check` exited 0. The initial deployment built and force-recreated
  `core` and `frontend`; after the final crash-recovery review, only the affected `core` image was
  rebuilt and force-recreated with no active Pi session. Core is healthy and its post-recreate
  Qwen catalog/completion probe succeeded. Built and running image IDs match: core
  `sha256:9867c2fa002b6f117da9a1d02c73b47bfe0372b8c1c137a5174c3e7ecb1e1db1`, frontend
  `sha256:5a47f81bc887423e05cef8fd3feb075aa600cb33217247186124f60ca5a3005b`.
  The frontend entry hash changed, so only `omics_portal-web-1` was restarted to invalidate its
  indefinite Vite-manifest cache. The application-level Qwen smoke reached its first
  `ui_request`, persisted the expected provider/model, deleted only its uniquely named smoke
  session, restored the exact saved settings object, and left no smoke session or Pi runtime. The
  supplied probe's success path left its keep-alive SSE reader open, so only that probe process was
  terminated (exit 143), without touching backend, Pi, or unrelated runtimes. The same smoke then
  exited 0 with `controller.abort()` in cleanup, preserving the gate, session cleanup, and exact
  settings-restoration evidence.

### Archived snapshot — Complete activity timeline + CTE spacing (2026-07-15)

- **Model activity is complete from F1.** The left panel now records the submitted prompt before
  session creation completes, then projects thinking, assistant output, sanitized tool lifecycle,
  reviewer gates, status, and turn lifecycle in chronological order. It remains in-memory only;
  closing and reopening the panel does not discard it, while Resume intentionally starts a fresh
  live timeline.
- **Tool activity is a narrow public contract.** Only call id, tool name, and
  `running`/`completed`/`failed` status cross Pi → backend → SSE. Updates, arguments, results,
  commands, output, credentials, and raw errors stay server-side. Resume and SSE replay were also
  hardened for process restarts, concurrent lifecycle requests, stale callbacks, cursor reset, and
  multi-client reconnects.
- **CTE plan layout uses real Tailwind 3 spacing.** Shared cards use concrete 16 px default / 12 px
  compact padding utilities; F6 CTE headers and content use responsive 16/20 px edge padding.
  Semantic ordered steps, single-boundary divided filter/table lists, long-value wrapping, and a
  plain top-divider rationale preserve the artifact content while improving scanability.
- **Final verification/deploy (`2026-07-15T02:18:54+02:00`, HEAD `56d73d2`):** backend
  **204/204** and frontend **285/285** passed; both TypeScript gates and production builds exited
  0, and `git diff --check` was clean. The final lifecycle fixes make restarted-hub cursor replay
  generation-aware and mutate frontend delete/resume state only for sessions actually deleted.
  Compose rebuilt and force-recreated `core` and `frontend`; built and running image ids match:
  core `sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147`
  (`healthy`), frontend
  `sha256:dd7ef721b661b57b8c5422c47088e1a9a2e9821af021541cab372c9892fbd638`
  (`running`). The entry changed from `index-DcZviApa.js` to `index-CuIt1NQg.js`, so only
  `omics_portal-web-1` was restarted to refresh its indefinite manifest cache.
- **Final live local-Qwen smoke:** session
  `2026-07-15-001922-final-no-thinking-activity-smoke-2026-07` observed **0**
  `activity_delta` events while receiving **5** strictly allowlisted tool lifecycle events and the
  first reviewer gate (`bash` running/completed and `reviewer_select` running). No forbidden tool
  field crossed SSE. Cleanup closed with 200, deleted only that session with 204, restored the
  exact settings object, and left no Pi runtime or smoke session.

### Archived snapshot — Filtered Model activity projection (2026-07-15)

- **Resolved contract.** `activityLog` still folds the complete in-memory prompt, thinking,
  assistant, sanitized tool, reviewer-gate, status, and turn-lifecycle history. The left panel now
  applies a default-deny rendering boundary and shows only prompt, thinking, status, and gate;
  assistant text remains available to the central transcript, while tool, lifecycle, and unknown
  future activity kinds do not render or move the panel scroll.
- **Frontend-only verification/deploy (`2026-07-15T03:44:02+02:00`, source HEAD `7208599`):**
  frontend **287/287** passed; `npx tsc -b`, `npm run build`, and `git diff --check` exited 0.
  Compose rebuilt and force-recreated only `frontend`; its image changed from
  `sha256:dd7ef721b661b57b8c5422c47088e1a9a2e9821af021541cab372c9892fbd638` to
  `sha256:75fc0b7786c75ded1b488e9fad3232aa061c82d89140c67ad62b5285954d1d36`, with container start
  `2026-07-15T01:42:45.582760612Z`. The active Vite entry changed from
  `index-CuIt1NQg.js` to `index-BdZpFO7j.js`, so only `omics_portal-web-1` was restarted at
  `2026-07-15T01:42:53.715330158Z`; core retained image
  `sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and start
  `2026-07-15T00:18:37.040145636Z`.
- **Real local-Qwen no-COT smoke:** session
  `2026-07-15-014328-activity-filter-smoke-2026-07-15t01-43-2` reached its first reviewer gate
  with **0** `activity_delta` events and **3** tool lifecycle events, each containing exactly the
  four public fields. The probe accepted the close response, deleted only that session with 204,
  restored the exact saved settings object, confirmed the session absent, and left no Pi runtime.

### Archived snapshot — Central activity log + compact CTE density (2026-07-15)

- **Resolved UI contract.** The central working body now renders every chronological non-blank
  assistant transcript line in one bounded accessible log, without user-entry echoes,
  timer/spinner labels, or step messages. The left Model activity panel is a default-deny
  projection of only thinking and status, while the complete raw activity fold and the existing
  reviewer widgets, artifacts, and workflow state remain unchanged. F6 CTE headers, content,
  table rows, and filter rows use 8 px vertical padding with 12 px lateral padding below `sm` and
  16 px from `sm` upward; divider top padding is 8 px. The final review amendment keeps historical
  log rows at the full muted-foreground token so their normal-size text retains AA contrast.
- **Source verification (source HEAD
  `09f9bdffe582ff3c66845ca219f60c11472a550a`).** Frontend tests passed **292/292** across
  **43/43** files. `npx tsc -b`, `npm run build`, the Impeccable layout detector, and
  `git diff --check` all exited 0; the detector returned `[]`.
- **Frontend-only deployment.** The pre-deploy frontend was image
  `sha256:8c9aaee6453b26c69e4057d3f9e53aa791d449b88ba4a0669a25cd442667d582`, started
  `2026-07-15T10:17:16.919166255Z`, serving `index-CveSyban.js`. Compose built and
  force-recreated only `frontend` with `--no-deps`; the final running frontend is image
  `sha256:39dc47d81abcb13466218500476fbb78d1be66a424f4293470437700c848c26d`, started
  `2026-07-15T10:30:30.869532521Z`, serving `index-CihtpQJV.js`. Because the entry changed,
  exactly `omics_portal-web-1` was restarted: it retained image
  `sha256:4580cf2bc85f3656ee515996c24ed02d9096d24ed0d8afe384a2f8bd8cf4bac9` and moved from
  start `2026-07-15T10:17:36.151611451Z` to `2026-07-15T10:30:46.369205604Z`.
- **Isolation and final state.** Core remained `running`/`healthy` with exactly its original image
  `sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and start
  `2026-07-15T00:18:37.040145636Z`; frontend and portal were `running` with no container
  healthcheck. Pre/post `docker top` showed only core's supervisor and backend server, so no
  unrelated Pi runtime existed to disturb. The count-only frontend sensitive/error pattern scan
  was **0**. No live model smoke was run, and settings and sessions were intentionally untouched.

### Archived snapshot — Resizable activity split + compact CTE rows (2026-07-15)

- **Resolved UI contract.** `activityLog` remains the complete in-memory chronological fold. The
  left Model activity panel default-denies every kind except prompt, thinking, and assistant,
  labels those entries Question, Reasoning, and Response in source order, and hides status, tool,
  gate, lifecycle, and unknown kinds. The desktop panel is pointer/keyboard resizable from 288–576
  px while preserving 512 px centrally, persists its global width in localStorage, and becomes an
  overlay drawer below `lg` or whenever the measured app shell is narrower than 800 px. F6 CTE
  cards retain their semantic structure and responsive grids;
  lateral padding is 12/16 px, header/content edge padding is 8 px, internal section gaps are 12
  px, heading/divider spacing is 4 px, and table/filter rows use 4 px vertical padding with compact
  line heights.
- **Source verification (`2026-07-15T15:22:17+02:00`, source HEAD
  `f1af1f909b387ae12a10f1b534bf6a529ea42505`).** Frontend tests passed **295/295** across
  **44/44** files. `npx tsc -b`, `npm run build`, and `git diff --check` exited 0; the Impeccable
  layout detector returned `[]`. The local source build emitted Vite entry
  `assets/index-PlvQqhNG.js`.
- **Frontend-only deployment.** The pre-deploy frontend was image
  `sha256:39dc47d81abcb13466218500476fbb78d1be66a424f4293470437700c848c26d`, started
  `2026-07-15T10:30:30.869532521Z`, serving `index-CihtpQJV.js`. Compose built and force-recreated
  only `frontend` with `--no-deps`; the final running frontend is image
  `sha256:0947211373784a860c7507d03c0fcf1f901ac3d1547cd0aa62b914d158f15bd5`, started
  `2026-07-15T13:21:08.29613952Z`, serving `index-Dn7T524a.js`. Because the entry changed, exactly
  `omics_portal-web-1` was restarted: it retained image
  `sha256:4580cf2bc85f3656ee515996c24ed02d9096d24ed0d8afe384a2f8bd8cf4bac9` and moved from
  start `2026-07-15T10:30:46.369205604Z` to `2026-07-15T13:21:21.968808414Z`.
- **Isolation and final state.** Core remained `running`/`healthy` with exactly its original image
  `sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and start
  `2026-07-15T00:18:37.040145636Z`; frontend and portal were `running` with no container
  healthcheck. Pre/post `docker top` showed only core's supervisor and backend server, so no
  unrelated Pi process existed and no Pi process was stopped or steered. The count-only frontend
  sensitive/error pattern scan was **0**. No live model smoke was run; settings and sessions were
  intentionally untouched.

### Archived snapshot — Final activity-split fix (2026-07-15)

- **Source and verification (`2026-07-15T15:56:57+02:00`).** Deployed source commit
  `1f540fcb78ac9e552e56a21e47edf66e9872b323` (`1f540fc`). Frontend Vitest passed **298/298**
  tests across **44/44** files; `npx tsc -b` and `npm run build` exited 0. The Impeccable detector
  scoped to AppShell, ModelActivityPanel, index.css, and CtePlanViewer returned `[]`; `git diff
  --check` exited 0.
- **Frontend-only deployment.** Compose built and force-recreated only `frontend` with `--no-deps`.
  The frontend image changed from
  `sha256:0947211373784a860c7507d03c0fcf1f901ac3d1547cd0aa62b914d158f15bd5` to
  `sha256:6e14f55092b7e3aca9a396220394ae484147674d81b051771e394e59b73b1c88`; its active Vite entry
  changed from `index-Dn7T524a.js` to `index-BIznZeLH.js`. Therefore exactly
  `omics_portal-web-1` was restarted to refresh its manifest cache; it retained image
  `sha256:4580cf2bc85f3656ee515996c24ed02d9096d24ed0d8afe384a2f8bd8cf4bac9` and started at
  `2026-07-15T13:56:32.786693915Z`.
- **Isolation and final state.** Core retained image
  `sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and exact original
  start `2026-07-15T00:18:37.040145636Z`, remaining `running`/`healthy`. Final frontend and portal
  states are `running` (no healthcheck). Pre/post core process tables contained only the supervisor
  and backend server, so Pi was preserved and no Pi process was stopped or steered. The count-only
  frontend sensitive/error-pattern scan was **0**. No model smoke was run; settings and sessions
  were intentionally untouched.

## What ThothII is

A **human-in-the-loop datamart builder**: it turns a natural-language question into
validated SQL (and optionally a dbt datamart) through a deterministic **8-phase
NL→SQL workflow**, where the model *proposes* and a human *reviewer decides* at gates.
The UI is meant to embed inside the Omics Portal (GSD design system) and is **English**.

## Architecture — three layers

```
frontend (React, :5173) → backend (Fastify, :8787) → pi --mode rpc → tht / harness → DWH (read-only)
```

- **harness/** — the Pi layer. A deterministic Python CLI **`tht`** + a Pi gate extension
  (`.pi/extensions/tht-gate.js`) that runs the 8-phase workflow and emits/consumes
  widget-descriptor JSON. **Owns all persistence.** Workflow truth is `harness/workflow.yaml`;
  orchestration rules are `harness/.pi/skills/tht-sessione/SKILL.md`.
- **backend/** — Fastify + TypeScript. A **thin bridge**: proxies REST routes to the `tht`
  CLI (`ThtRunner`), manages Pi processes (`PiProcessManager`, one child per session),
  bridges Pi RPC events to SSE (`SessionBridge` + `SseHub`). No application database.
- **frontend/** — React 18 + base-ui + Tailwind + TanStack Query + Zustand. Chat-style
  shell (`src/shell/AppShell.tsx`); the live transcript is rebuilt in-memory from the SSE
  stream (`src/store/sessionStore.ts`), **not persisted**.

### Persistence model (the load-bearing premise)
There is **no verbatim chat store**. Each workflow phase persists its own document into the
session directory, and that **IS** the persistence. A session = a directory under the
workspace's `sessions/` path containing `session_manifest.yaml` + phase artifacts
(`question.md`, `schema_linking.json`, `cte_plan.json`, `sql_final.sql`,
`validation_report.md`, `review_decisions.jsonl`, …). A fresh Pi process resumes by reading
`tht session show <id>` + the on-disk artifacts — never by replaying chat.

## The 8 phases (harness/workflow.yaml)
F1 chiarimento · F2 memoria · F3 riscrittura (`question.md`) · F4 schema_linking
(`schema_linking.json`) · F5 sintesi · F6 cte (`cte_plan.json`, `cte_tests.json`) ·
F7 sql_finale (`sql_final.sql`) · F8 datamart. Current phase is a fold over the decision
ledger (`harness/tht/phase.py`); statuses: `open` / `closed` / `finalized`.

## How to run

**Full stack (real Pi + DWH):** from project root,
```bash
./scripts/run-stack.sh
# Opens frontend: http://localhost:5173 (proxies backend :8787)
```
**Prereqs:** VPN on; `pi` on PATH (with a configured model, e.g., `pi model set claude-fable-5`); 
`harness/.env` populated (see `.env.example`); `harness/config/tht.yaml` → workspace (psd recommended for testing).
All three layers' deps installed (`npm install` in each, `python -m venv + pip install -e ".[dev]"` in harness).

**Individual dev:**
- backend: `cd backend && npm run dev` (tsx watch; env: `PORT`, `THT_HARNESS_DIR`, `THT_BIN`, `PI_BIN`, `AUTH_MODE`)
- frontend: `cd frontend && npm run dev` (Vite; `VITE_BACKEND_URL` → backend)
- harness install: `cd harness && python -m venv .venv && pip install -e ".[dev]"` → `tht` on PATH

## How to test (latest TS gates green 2026-07-15: backend 204 / frontend 292 (43 files); harness 798 / gate JS 126 last recorded 2026-07-14)
- harness: `cd harness && .venv/bin/pytest -q` (5 L2/real-DB tests are deselected by default)
- backend: `cd backend && npx vitest run` · typecheck `npx tsc --noEmit -p .`
- frontend: `cd frontend && npx vitest run` · typecheck `npx tsc -b` · e2e `npm run e2e` (Playwright)

## Config & workspaces
- Workspaces: `harness/workspaces/*.yaml` (`psd`, `tht-test`, `tht.example`). A workspace sets
  the DB target and the **absolute** `paths.sessions/artifacts/indexes` (psd → a *separate*
  repo `tht-workspace-psd/`, NOT committed here).
- Secrets live ONLY in `harness/.env` (gitignored; `THT_*` — DB, DWH REST, vector, SSL CA…).
  See `harness/.env.example` for the variable list.
- App settings (global): `{ workspace, provider, model, thinking }`, persisted via harness
  preferences (`tht session preferences get/set` → the configured session repository —
  filesystem or Postgres in server mode). `backend/data/settings.json` (gitignored) remains
  only the file fallback for injected runners/tests. The "New session" form is question-only;
  these settings supply the rest.

## Efficiency levers (NL→SQL workflow optimization, 2026-07-08)

Three deployed optimizations target model thinking time (the dominant cost, ~220s in F1 alone):

1. **Join-graph via FK annotations + `tht schema suggest-fks`**
   - DWH has no FK constraints declared. Annotations file (`tht-workspace-*/artifacts/mschema/annotations.yaml`) now stores curated logical FKs.
   - Three ranking rules: mine from approved SQL (highest confidence), heuristics (`*_time_key → dim_time.day_key`), same-name PK discovery with `--assume` flag for disambiguity.
   - **Psd workspace:** 228 FK suggestions already generated (139 tables); `tht schema suggest-fks --from-sql <session-dir> --assume cod_paz=dim_patient` for updates.
   - **Activation:** automatic. The mschema renderer populates the `【Foreign keys】` section. F4 in SKILL.md now reads FK joins from there instead of the model re-deriving them.

2. **Context-pack consolidation at F1 kickoff (`tht search pack`)**
   - Single embedding of the question, reused for schema + evidence + solved-question searches.
   - Command: `tht search pack "<question>" --session <id>` → `sessions/<id>/retrieval_pack.md` (tabelle candidate, relevant evidence, solved exemplars).
   - Graceful degradation: if Ollama or vector store unreachable (no VPN), sections are empty but exit 0 — session continues with live searches.
   - **Activation:** automatic at next session. SKILL.md F1 now prescribes as first call; reduces exploratory turns.

3. **Phase-summary recap auto-construction from session ledger**
   - `tht session show --json` includes the full decisions ledger; `tht phase meta --json` exports decision types per phase.
   - Gate appends deterministic `【Decisioni registrate in questa fase】` section to v2 phase-summary artifacts.
   - Model authors only `summary` + `checks`; the gate fills the recap table from persisted state → exact by construction.
   - **Activation:** automatic at next session and Pi restart. SKILL.md Disciplina 6 updated: model keeps output brief, gate enriches from catalog + ledger.

**Tests:** 358 Python + 111 JS gate, all pass. L2 (live DWH) verification on psd workspace recommended when time permits.

## Conventions & contracts (don't relearn the hard way)
- **`-c`/`--config` is a PER-COMMAND option in `tht`** — append it AFTER the subcommand,
  never globally (`ThtRunner.buildArgv` handles this).
- **`--json` output must be pristine** (only valid JSON on stdout).
- **UI strings are English.** Document *content* stays in the workspace language (Italian
  for psd) because it's the real data; only chrome/labels are English.
- **Settings are global**, not per-question.
- TDD throughout; tests assert real behavior, not mocks. Frequent, scoped commits.
- Global user rules (`~/.claude/CLAUDE.md`): think before coding, simplicity first, surgical
  changes, goal-driven verification.

## Active memory — F8 promotion gate + solved-question recall — SHIPPED, L2 pending (2026-07-07)

Two additions to close the loop on reusable memory, on top of the existing `tht memory
search` (Phase 2) reuse:

- **F8 promotion gate.** `reviewer_memory_promote` (Phase 8, called with only the session
  id): the gate computes candidates deterministically via `tht memory promote --preview
  --json` (the 3 reusable decision types, already excluding previously promoted/declined
  ones) and shows a pre-selected checklist. Selected → `tht memory save-one` persists to
  the vectordb + records `memory_promoted`; deselected → `memory_promotion_declined`
  (ledger detail `seq:<n>`) so it is never re-proposed. `tht memory promote`/`save-one` were
  added to the gate's anti-bypass FORBIDDEN list (model must go through the gate tool).
- **Solved-question exemplars.** New vector kind `solved_question` reusing the existing
  `memory` pgvector table (no server-side DDL); `harness/tht/solved.py` does a one-row
  upsert keyed by a hash of question+SQL. CLI: `tht memory solved-index` / `solved-search`.
  `tht session finalize` auto-indexes the pair (best-effort: green line on upsert, cyan
  "già aggiornata" on dedup no-op, yellow warning + the recovery command
  `tht memory solved-index <id>` on failure). `SKILL.md` now prescribes calling
  `solved-search` as reference-only context in F4 (schema linking), F6 (CTE plan) and F7
  (final SQL), and documents the finalize auto-index in "Session end".

**Pending L2 gate (not yet run — needs VPN + writer key):** one live end-to-end session on
workspace `psd` via `./scripts/run-stack.sh` to verify (a) the promotion checklist renders
pre-selected and persists selected/declined correctly, (b) finalize indexes the pair,
(c) `tht memory solved-search` returns it with sql + tables.

**Fast-follow:**
- RestSearcher top-k dilution — **client-side DONE** (2026-07-07): `search_similar` manda
  `kinds` alla RPC (filtro server-side esatto) con fallback automatico su server legacy
  (404 → retry senza filtro, post-filter client). **Resta la migrazione server** della
  funzione SQL `search_similar` (+`kinds text[] DEFAULT NULL`): istruzioni pronte in
  `harness/docs/vector-rest-kinds-migration.md`; l'ordine di deploy è libero, ma fino
  alla migrazione il filtro resta client-side e la diluizione persiste.
- ~~`tht memory solved-search` muore con traceback grezzo se il vectordb è irraggiungibile~~
  **DONE** (2026-07-07): degrada a warning di una riga su stderr, stdout puro (`[]` in
  --json), exit 0 — copre VectorRestError/EmbeddingsError/OperationalError.

## Review gates v2 — payload strutturati + viewer dedicati — COMPLETE (2026-07-07)

Plan: `~/.claude/plans/prima-di-passare-ai-inherited-marshmallow.md`. Merged to `main` @ `2410f01`
(ff, pushed). Executed via subagent-driven-development (4 workstreams, task reviews, final
whole-branch review + fix wave).

- **Contracts:** `artifact.data.schema_version: 2` for `cte_plan` / `cte_result` / `phase`,
  built **deterministically by the gate** (catalog descriptions via `tht schema columns`; SQL
  from `ctes/<name>.sql`; preview rows persisted by `tht cte test`); the model contributes only
  purpose/rationale/note. Non-v2 payloads fall through to the legacy renderers — old sessions
  and `tools/replay/replay.json` keep working.
- **Harness (Python):** `CteTestRecord.preview_rows` (+ `_jsonable` coercer, ≤10 rows, cells
  ≤200 chars); new read-only `tht cte info <name> --session <id> --json` (index/total from
  `cte_plan.json`, same source as `next_cte`); `tht cte plan --doc -` writes
  `cte_plan_doc.json` (chain documentation; `cte_plan.json` stays a load-bearing `list[str]`).
- **Gate (JS):** `gate/artifact-contracts.js` (soft validators → self-corrective `textResult`,
  TypeBox untouched) + `gate/enrich.js` (pure, catalog lookups injected);
  `prepareReviewerArguments` now coerces `artifact.data` too (GLM stringified-param
  mitigation); `SKILL.md` Phase 5/6 + disciplines rewritten (plan via `reviewer_confirm
  kind:"cte_plan"` with payload A; `cte_result` gates send THIN data only — never SQL/preview
  as text).
- **Frontend:** `artifactV2.ts` types; `CtePlanViewer` (per-CTE cards + chain strip),
  `CteResultViewer` (shiki SQL + AG Grid preview), `PhaseSummaryViewer` (checks + criteria
  with the VALUES driving choices), `PreviewGrid` extracted from `ResultsPanel`,
  `statusBadge.ts` shared success/warn/error tokens.
- **Replay:** v2 fixtures + `tools/replay/augment-review-gates.mjs`; `replay.json` regenerated;
  offline visual pass ok (screenshots in the SDD scratch dir).
- **LIVE E2E (session `2026-07-07-011858`, GLM 5.2):** all 8 phases completed with the v2
  gates; session **finalized** (DWH validation battery green, needs VPN).
- **Bug found live + FIXED (`2410f01`):** infinite spinner at workflow end — the bridge dropped
  Pi's `agent_end` (the ONLY end-of-turn signal) and `working` was released only by the next
  gate, which the final turn doesn't have. Now: bridge maps `agent_end` → SSE
  `system_event`; FE tracks `agentActive`; an unexpected Pi child exit notifies the client
  (info error + synthetic `agent_end`). Memory: `pi-rpc-event-vocabulary`.
- **Open (non-blocking):** `tht.sqlcheck` maps table aliases by first occurrence (found and
  worked around by the model in F6 — spawned as a separate task); one more live confirmation
  that the spinner stops at F8 (the chain is unit-tested end to end).

## F4 schema-linking column curation + look&feel v2 — COMPLETE (2026-07-06)

Branches `feat/f4-schema-linking-column-curation` (PR #1) + `feat/frontend-lookfeel-v2`, landed
on `main` (`d942635` … `7491e8c`). The F4 gate (`reviewer_schema_linking`) presents
catalog-enriched tables/columns (descriptions from `tht schema columns`, hardened enrichment),
per-table columns modal (suggested pre-checked, suggested-first ordering + filter box);
decisions `column_promoted`/`column_excluded` + deterministic `tht session
sync-schema-linking` projection into `schema_linking.json`. Live-verified including the
clobber test (the model's joins write preserves curated columns). Look&feel v2: shadows/radii/
mono labels, 70% gate modal, structured cards (colors untouched). Memory:
`thothii-visual-language-v2`.

## Workflow contract hardening — COMPLETE (2026-07-01)

Spec: `docs/superpowers/specs/2026-07-01-workflow-contract-hardening-design.md` · Plan:
`docs/superpowers/plans/2026-07-01-workflow-contract-hardening.md`. Merged to `main` @ `3dadc6f`
(pushed). Driven by analysis of Pi session `2026-06-30-165708` (GLM 5.2), where the model spent
~80% of its tool calls reverse-engineering the harness because `SKILL.md` mis-stated the
phase-advance contract — and Phase 6 was a hard dead-end. Three coordinated harness fixes (TDD):

- **F6 CTE-approval dead-end FIXED.** The gate's `reviewer_confirm kind:"cte_result"` used to
  register `cte_approved --subject phase:6`, which `decision_cmd` rejects (exit 5 — it needs a
  real CTE name from the plan) → F6 could never close. New `tht cte next --session <id>` returns
  the first unapproved plan CTE; the gate now approves **by name**. (`tht/cli/cte_cmd.py`,
  `.pi/extensions/tht-gate.js`.)
- **`schema_linking.json` writer/validator.** `store.set_schema_linking` (validates against the
  `SchemaLinking` model, THEN writes — no partial file) → CLI `tht session set-schema-linking
  <id> --file <path|->` (exit 5 on bad JSON / ValidationError) → gate tool `write_schema_linking`
  (stdin). Replaces the model hand-writing the F4 artifact + ad-hoc python validation.
- **`SKILL.md` corrected to match the code.** Only F2-empty / F6-skipped auto-advance
  (`_AUTO_ADVANCE_PHASES={2,6}`); every substantive phase closes with `reviewer_confirm
  kind:"phase"` (F7 is **two-step**: `kind:"sql"` records `sql_approved`, then `kind:"phase"`
  advances). Fixed Discipline 2 + Phase 1/3/4, added a per-phase **cheat-sheet**, documented the
  `SchemaLinking` shape. The old false "the reviewer_decide already advances" (F3) claim — the
  exact cause of the observed thrash — is gone.

Verified: harness pytest **281 passed** / 5 deselected, gate JS **34/34**, changed-files ruff
clean (the 36 `ruff check .` errors are pre-existing on `main`). Final whole-branch review
(opus): READY TO MERGE, no Critical/Important. Executed via subagent-driven-development
(implementer + task-review per task, final opus review). **DEFERRED (needs VPN): live F4/F6
end-to-end** — resuming session `2026-06-30-165708` (stuck at F6) is the ideal live probe.

## UI/UX redesign + Resume — COMPLETE (2026-06-30)

Plan: **`~/.claude/plans/foamy-forging-dahl.md`**. Memory: `thothii-ui-redesign-inprogress.md`.
**All workstreams done and pushed to origin/main:** D + E @ `0eeb3f7`, B + C @ `b056ff3`,
F @ `cef9ae4`, A @ `0a13f71`, G @ `e8cdd00`(scope) + `cbb8e18`(results). Nothing pending from this
plan. Per-workstream detail below for reference.

- **D — DONE** (`c12bdcd`): session display `name` = 3-5 Italian keywords via **YAKE** (no LLM),
  derived in `tht session new` (CLI layer); `create_session` core unchanged (`name=None` default).
  `yake` added to `harness/pyproject.toml`. TDD `tests/test_session_name.py`; harness 269 passed.
- **E — DONE** (`0eeb3f7`): rotating activity icon replaces the red dot in `CentralStatus`
  (inline, clickable → opens the panel); `ModelActivityPanel` is a **5-line expandable
  model-stream tail**; `WorkingSpinner` extracted to its own module; the separate spinner button
  + orphaned `Transcript.tsx` removed. Frontend 87/87, tsc clean. **Live visual check DONE
  (2026-06-30):** inline spinner opens the panel; 5-line collapsed tail; expand → full transcript.
- **B — DONE** (`b056ff3`): `WorkflowBar` is now colored **dots** F1..F8, no phase-name text
  (amber-translucent=running, green=done, red=error, gray=pending; green connectors lead the active
  dot). Each dot carries `data-state`. Error is lightweight: store `phaseError` set when an `info`
  `level=error` arrives during the phase, cleared on the next `ui_request` (`sessionStore.ts`).
  **All four states live-verified** via Playwright.
- **C — DONE** (`b056ff3`): right sidebar — single-line denser rows (inline status dot + name,
  `py-1`), a 3-level type hierarchy via **`/impeccable`** (L1 `SESSIONS` red/bold/wide-tracking ·
  L2 section + group headers muted uppercase · L3 names normal-case), and the **"No group" label
  removed** (ungrouped sessions render after the last group; guarded so the empty-state still
  teaches when there are no groups). **Live-verified.** (Resume in `SessionMenu` stays with A1.)
- **Tests:** frontend **93/93** (was 87; +3 store `phaseError`, +2 `WorkflowBar` dot-state, +1
  AppShell no-"No group"), `tsc -b` clean.
- **F — DONE** (uncommitted; live check deferred to G): single-select answers **auto-confirm**.
  `reviewer_select` options may carry a `decision` payload (`{type, subject, detail?, rationale?}`)
  and an optional `advance`; picking such an option persists the decision directly via
  `tht decision add` (shared `decisionAddArgs` helper, also used by `reviewer_decide`) — no redundant
  `reviewer_decide`/`reviewer_confirm` gate. Options without a payload stay ask-only; back/exit/Other
  never persist. Pure logic extracted to `resolveSelectOutcome`/`decisionAddArgs` (exported, unit-
  tested). Contract docs updated: `reviewer_select` tool desc + `SKILL.md` (widget summary,
  disciplines 2-3, Phase-1 single-pick) + the `CLAUDE.md` gate note. Gate JS **33/33**, harness 269.
  **Live verification (model actually uses `reviewer_select`+decision, no follow-up gate, decision in
  `review_decisions.jsonl`) deferred to G** — it is model-behavior-dependent.
- **A — DONE** (uncommitted): **A1** — `SessionMenu` gains a **Resume** item (gated to
  `status!=="finalized" && !archived`), wired in `AppShell` to the existing `doResume` → `POST
  /sessions/:id/resume`. 3 tests (`SessionMenu.test.tsx`); frontend **96/96**, tsc clean. **A2** —
  diagnosis-first clean-room repro shows the **resume cold-start stall NO LONGER reproduces on pi
  0.79.4** (8/8 chained into the tool calls, fresh + partway; GLM 5.2 now narrates AND emits
  `tht session show`+`read SKILL.md` in-turn). The earlier narrate-and-stop predates the pi upgrade.
  Defense-in-depth applied: `RIPRENDI_KICKOFF` hardened to force the in-turn tool call (gate test +
  live regression 2/2). The cross-model angle (weaker/older models) lives in **G**.
- **G — DONE** (`e8cdd00`+`cbb8e18`): cross-model behavior matrix via a committed clean-room harness
  (`harness/scripts/model-matrix.mjs`). **Tier 1** — kickoff + resume first-turn: all *available*
  models chain in-turn (`zai/glm-5.2`, `deepseek/deepseek-v4-{pro,flash}`, `aritmolab/qwen3.6-35b-a3b`,
  `zai/glm-4.5-air`); the resume stall recurs on none (closes A's cross-model robustness).
  `aritmolab/gemma4-26b-a4b` = **404 unavailable** at the endpoint (listed but not served) — infra
  gap, not a workflow issue. **Tier 2** — F single-select auto-confirm verified live on `glm-5.2`:
  answering the first `reviewer_select` persisted a `concept_clarified` decision **0→1** with **no
  follow-up gate** (closes F's deferred live check). Full results: the G plan doc + memory
  `thothii-cross-model-matrix`. No prompt hardening needed.

**Status:** **All UI-redesign + resume workstreams done and pushed — D, E, B, C, F, A, G.**
Nothing pending from the plan. Optional nice-to-haves (not required): Tier-2 F/multiselect live for
the non-baseline models (cheap re-run with `harness/scripts/model-matrix.mjs` + the Tier-2 method),
and a one-off manual Playwright kebab→resume pass in the live UI.

## Live verification + reviewer_select fix (2026-06-30, afternoon)

Drove the real stack (Playwright → backend → real Pi → GLM 5.2 → DWH) end-to-end.

- **F1 hang fix (`418187a`) VERIFIED LIVE.** Answered an F1 reviewer widget; Pi resumed (model
  socket reopened) and the gate produced new output — vs the old silent hang. The transition
  "silent hang → gate re-presents/advances" proves `ctx.ui.input` now resolves.
- **New bug found + fixed: reviewer_select `choices` vs `choice`.** The gate's `reviewer_select`
  (and `reviewer_confirm` reject) read `resp.choice` (singular) but the frontend uniformly sends
  `choices: [id]` (array) — so every single-select gate answered "Nessuna scelta ricevuta" and
  re-proposed forever (multiselect was fine; it already read `choices`). Fix: a shared
  `selectedChoice(resp)` helper (`harness/.pi/extensions/tht-gate.js`) reading the array; both
  handlers use it. TDD: `gate/__tests__/gate_choice.test.js` RED→GREEN, full gate suite **28/28**.
  VERIFIED LIVE: a single-select answer is now accepted and the workflow advances (2/4 → 3/4).
- **Resume cold-start STALL confirmed (open item #1).** On `/riprendi-sessione`, GLM 5.2 narrates
  the bootstrap step then ends the turn without the tool call → Pi idle, unrecoverable from the UI.
  Memory: `thothii-resume-cold-start-stall.md`.
- **GLM 5.2 F1 is slow (~3-4 min, ~50+ reads) but works** — looks stuck but isn't; don't hit
  "Stop and save" (it `POST /close`s → kills Pi). Memory: `thothii-glm52-f1-slow-not-stuck.md`.

## Earlier work — F1 reviewer-widget hang fix + multiselect guidance (committed 2026-06-30; authored 2026-06-29)

Two fixes, **committed to `main`** (7 files):

1. **Bug: every reviewer widget hung "stuck with no output" after the human answered** — F1
   disambiguation (and any gate) dead-ended. Root cause, confirmed from Pi's own source
   (`@mariozechner/pi-coding-agent` `dist/modes/rpc/rpc-mode.js`, `createDialogPromise`):
   `ctx.ui.input` assigns its OWN RPC id (`crypto.randomUUID`) and correlates
   `extension_ui_response` on THAT id, silently dropping unknown ids. The gate puts a
   different id (`u${Date.now()}`) inside the descriptor carried in `title`. `SessionBridge`
   was replying with the **descriptor** id, so real Pi never resolved `ctx.ui.input` → the
   model never continued. **Fix:** `SessionBridge` now stores Pi's top-level `m.id`
   (`pendingPiId`) on the incoming request and replies `extension_ui_response{ id: pendingPiId,
   value: <uiResponse JSON> }` (value still carries the descriptor id, so the gate's internal
   `resp.id === descriptor.id` check holds). File: `backend/src/bridge/session-bridge.ts`.
   Full write-up: memory `pi-ui-input-id-correlation.md`.
   - **The test double was masking it:** `harness/tests/fake_pi/fake_pi_rpc.mjs` had forced
     `m.id == descriptor.id`. Corrected to mirror real Pi (distinct `randomUUID` top-level id,
     correlate on it, drop unknown ids); `test_fake_pi_contract.mjs` gained a negative
     regression test ("respond with descriptor id → no follow-up").
   - TDD: `backend/test/session-bridge.test.ts` (unit) + `backend/test/e2e-f1.test.ts`
     (integration — now asserts the model's follow-up arrives after the answer) went
     RED→GREEN.

2. **UX: multi-answer disambiguation** — `harness/.pi/skills/tht-sessione/SKILL.md` Phase 1
   now tells the model to use `reviewer_decide` (the existing multiselect/checkbox widget)
   when an ambiguity admits several simultaneously-true answers, instead of single-pick
   `reviewer_select`. Guidance-only — no new widget (`frontend MultiselectWidget` already
   exists).

Verified at commit time: backend `npx vitest run` **67/67 green**; `tsc --noEmit -p .` **OK**;
fake-pi contract `node --test test_fake_pi_contract.mjs` **2/2 green**. **Verified LIVE
2026-06-30** (see the top "Live verification" section).

## Most recent feature — Session management (MERGED to main @ 2c21e46)
Full session management modeled on Claude's UI, all three layers:
- **Read-only "split view" panel** (left drawer, `SessionDocumentsPanel`) showing a session's
  phase documents read-only (reuses `SqlViewer`/`SchemaLinkingViewer`/`MarkdownView`).
- **Rename / Move to group / Archive / Delete** via a kebab menu (`SessionMenu`) → REST →
  `tht session set-name/set-group/archive/unarchive/delete`. Archive = a manifest `archived`
  flag (not a dir move); groups = a manifest `group` field; delete = hard `rmtree` + confirm.
- Rail: collapsible group headers + "No group" + a separate **Archive** view.
- **Resume correctness:** read-only **guard** (HTTP 409 when `finalized` or `archived`);
  `PiProcessManager.spawnFor` now has a `new`/`resume` mode (resume sends
  `/riprendi-sessione <id>`); a "Phase 0 — Resume" cold-start section in `SKILL.md`.
- Design docs: `docs/superpowers/specs/2026-06-29-session-management-design.md` +
  `docs/superpowers/plans/2026-06-29-session-management.md`.

### ⚠️ Open items / pending gates
1. **Resume cold-start stall — RESOLVED on pi 0.79.4 (workstream A, 2026-06-30).** The earlier
   narrate-and-stop (GLM 5.2 narrating the bootstrap step then ending the turn without the tool
   call) **no longer reproduces**: a clean-room repro of the backend's exact resume handshake
   chained into `tht session show`+`read SKILL.md` in-turn **8/8** (fresh + partway sessions). The
   pi upgrade is the likely fix. Defense-in-depth: `RIPRENDI_KICKOFF` hardened to force the in-turn
   tool call (gate test + live 2/2). Memory: `thothii-resume-cold-start-stall.md`. **Remaining:**
   the cross-model angle (older/weaker models) is folded into **G**; a full Playwright kebab→resume
   pass through the live UI is still worth one manual run (item 2).
2. **Full Playwright live-stack verification (MANUAL, not yet run).**
3. **Minor backlog (non-blocking):** explicit id-traversal guard in `delete_session`
   (today gated by `load_session`); `close_session` could reuse `_save_touched` (DRY);
   delete-via-kebab integration test skipped (base-ui Menu portal not drivable in jsdom —
   the dialog itself is unit-tested); a couple of test-file lint nits.
4. **DONE — F1 hang fix live-verified 2026-06-30** (see top section). The live verification
   also surfaced + fixed the reviewer_select `choices` mismatch.
5. **Resolved: settings use `zai/glm-5.2`/medium** (not deepseek-flash); GLM 5.2 drives F1
   fine, just slowly (~3-4 min, ~50+ reads). To tell a truly stalled Pi from a merely-slow one,
   check its sockets/children. Memory: `thothii-glm52-f1-slow-not-stuck.md`.
6. **DONE — Pi migrated to `@earendil-works/pi-coding-agent@0.80.3` (2026-07-04).** The old
   scope `@mariozechner/pi-coding-agent` is frozen at 0.73.1; every release ≥0.74 lives under
   the new scope `@earendil-works` (latest 0.80.3). The live `pi` (`~/.local/bin/pi`) was
   repointed to 0.80.3. Validated by an API/RPC-surface diff (0.73.1→0.80.3: `ExtensionUIContext`
   byte-identical, `rpc-types` additive-only, `createDialogPromise` + provider-registration API
   unchanged) **plus** a live `model-matrix` smoke (GLM 5.2, `new`+`resume` both `CHAINED`,
   full RPC event vocabulary incl. `extension_ui_request` intact). Notable: 0.80.3 adds
   `ctx.mode: "tui"|"rpc"|"json"|"print"` (a proper mode discriminator; the earlier `0.79.4`
   references above are superseded). Rollback: the old package is still on disk — repoint the
   symlink to `@mariozechner/.../dist/cli.js`. Memory: `thothii-pi-earendil-migration.md`.

## Where design history lives
- Specs: `docs/superpowers/specs/` · Plans: `docs/superpowers/plans/`
- SDD execution ledger (gitignored scratch): `.superpowers/sdd/progress.md`
- Auto-memory index: `~/.claude/projects/-Users-mp-projects-ThothII/memory/MEMORY.md`
  (Pi RPC event vocabulary, ui.input id correlation, Omics Portal/GSD design system,
  resume cold-start stall, GLM 5.2 F1 slow≠stuck).

## Git
`main` @ `2410f01`, **pushed to `origin`** (github.com/mptyl/ThothII); working tree clean, no stashes.
Latest arc (2026-07-07): review-gates-v2 ff-merged — `9b4f6b9` (WS1 harness) · `3ad93cd` (WS2
gate+SKILL) · `1c97289` (WS3 viewers) · `4042d0b`+`b089482` (WS4 replay) · `b59b57c` (review fix
wave) · `2410f01` (agent_end spinner fix). Branch `feat/review-gates-v2` still exists (local +
origin), fully merged. Before that (2026-07-05/06): F4 column curation (PR #1) + look&feel v2 —
`0d9e035` · `0a63ba9` · `d942635` · `9fe1c93` · `e9b2934` · `9403147` · `7491e8c`.
Older history (workflow hardening 2026-07-01, UI redesign 2026-06-30): see the sections above;
stale branches were pruned on 2026-06-30 (SHAs recoverable via reflog).