1283 lines
98 KiB
Markdown
1283 lines
98 KiB
Markdown
# ThothII — Project State
|
||
|
||
> Starting-point snapshot for new sessions.
|
||
> **Requisito finale del progetto (owner, 2026-08-11):** al termine dell'ultima fase tecnica deve
|
||
> essere prodotto un documento unico che guidi l'utente passo-passo su (1) come preparare il
|
||
> repository dei workspace su Git secondo le regole del progetto, (2) come usare gli strumenti di
|
||
> ThothII per il repository (app + CLI `tht`), (3) come usare l'applicazione ThothII di base
|
||
> (sessioni, domande, gate). Il documento userà parole semplici ed esempi; i dettagli tecnici
|
||
> resteranno nei contratti esistenti. Esempio pratico completo: Policlinico San Donato.
|
||
> Last updated: 2026-08-21 (DWH per-installation authentication is active in dual-key mode;
|
||
> the owner deferred Mac acceptance and legacy revocation to the mandatory pre-Project-B gate,
|
||
> authorized the read-only survey and Project A private preparation, and did not authorize either
|
||
> stopping the legacy stack or starting the new stack).
|
||
> Point a fresh session here ("read PROJECT_STATE.md") before substantial work.
|
||
|
||
### PSD server deployment program — design approved, execution PENDING (2026-08-20)
|
||
|
||
- **Approved design:** `docs/plans/2026-08-20-psd-server-deployment-program-design.md`; design
|
||
commit `3fd177b`. The owner approved a common read-only survey followed by two independently
|
||
accepted projects: A installs/proves a private local-auth stack through F1-F8; B starts only
|
||
after A PASS and integrates Supabase schema storage, Authentik OIDC, Nginx, the load balancer,
|
||
and the existing Aritmolab sidebar journey.
|
||
- **Executable entrypoint:** `docs/plans/2026-08-20-psd-server-deployment-program.md`, with separate
|
||
plans for the survey, Project A, and Project B. The server-local Sol agent must execute them with
|
||
`superpowers:executing-plans`, checkpointing every verified step and stopping on the documented
|
||
owner/secret/rollback boundaries.
|
||
- **Human gates:** `docs/testing/psd-server-project-a-manual.md` and
|
||
`docs/testing/psd-server-project-b-manual.md`; survey and Project A/B report templates are under
|
||
`docs/testing/evidence/`. Automated evidence never substitutes for the two explicit human PASS
|
||
decisions.
|
||
- **Workspace decision:** keep one `psd-clinical` descriptor in `tht-workspace-psd`; publish
|
||
`supported_transports: [rest_api, postgres_direct]`. Mac selects REST, server selects direct;
|
||
all installation bindings/secrets remain outside Git.
|
||
- **Data/runtime decision:** migrate configuration only. Legacy work sessions, Qdrant indexes, and
|
||
Ollama cache are not imported. Project A rebuilds internal Qdrant/Ollama and uses filesystem work
|
||
sessions. Project B uses the existing Supabase PostgreSQL database with isolated schema
|
||
`thoth_sessions`, dedicated migrator/runtime roles, forced RLS, and no PostgREST exposure.
|
||
- **Network/auth decision:** no SSH tunnel. Project A is loopback-only unless the surveyed load
|
||
balancer can prove an operator-only temporary endpoint. Project B preserves the real user flow
|
||
`Aritmolab homepage -> sidebar -> load balancer -> Nginx -> ThothII`, with direct ThothII-managed
|
||
OIDC and no second Nginx `auth_request`.
|
||
- **State:** survey `SURVEY_NO_GO` for Project A private; Project A
|
||
`BLOCKED_BY_SURVEY_AND_MUTATION_GATE`; Project B `BLOCKED_BY_PROJECT_A_AND_PRE_B_GATE`.
|
||
Remaining private-scope blockers are legacy rollback/backup, approved installation paths and UID
|
||
strategy, dedicated read-only workspace access, a dedicated direct-DWH role/route, and Pi/LLM
|
||
metadata. The catalog-only survey proved the currently available `postgres` identity owns
|
||
`datawarehouse` and has full write/DDL privileges, so it must not be reused by the new core.
|
||
Pi metadata resolves to 0.80.3, `deepseek/deepseek-v4-pro`, thinking `high`, but the bounded
|
||
no-session/no-tool reachability probe is FAIL and must be diagnosed without exposing auth data.
|
||
- **Sequencing amendment (owner, 2026-08-21):** use two survey decisions. Project A private may
|
||
proceed only after `SURVEY_GO_PROJECT_A_PRIVATE` and a separate stop/start authorization. Mac
|
||
`rest_api` acceptance, the 48-hour/two-ETL observation, and revocation of `legacy-shared` are
|
||
mandatory before `SURVEY_GO_PROJECT_B`. Current authorization covers read-only survey and
|
||
preparation only; old-stack stop and new-stack start remain forbidden.
|
||
|
||
### Authentication final-review fix round 2 — remediation PASS, release gates remain (2026-08-18)
|
||
|
||
- Frozen source is `2a9359071257f9b8a71d36ec2bbb25b161003f81` on `feat/thoth-auth`.
|
||
Source and evidence are separate commits; generated local runtime-state directories remain
|
||
untracked and must not be staged.
|
||
- Local PASS on the frozen source: exact `safeio`/`backup`/`authstorage` tests, full Go race suite,
|
||
`go vet`, macOS host build, Windows amd64 package cross-compiles, and Windows CLI build.
|
||
- The lifecycle tests now use context-aware gate publication/release, bounded waits for stages,
|
||
outcomes and admission, and cancel plus bounded worker join before lock-release assertions. A
|
||
deterministic withheld-gate case proves timeout, cancellation, join, and eventual lock release.
|
||
The temporary Windows relative-open diagnostic matrix was removed without reducing DACL, NT
|
||
normalization, or retained no-delete assertions.
|
||
- Authorized exact-source workflow run `32147345625` completed on the exact frozen SHA and
|
||
executed the unfiltered native command
|
||
`go test ./internal/safeio ./internal/backup ./internal/authstorage -count=1`. The required step
|
||
passed: safeio `22.058s`, backup `7.161s`, authstorage `16.088s`. Native Windows StageArchive and
|
||
concurrent claim-consume evidence are therefore PASS, not inferred from cross-compilation.
|
||
- The same Windows job later failed the unrelated clone-contract script at
|
||
`scripts/test-windows-clone-contract.ps1:208` because `$remoteYaml:` is not a valid PowerShell
|
||
variable reference. LF/Compose and Linux Docker baseline failures also repeated. The optional
|
||
Windows Docker startup job was skipped without executing and is `NOT_RUN` / `BLOCKED`; the
|
||
overall completed run conclusion is `failure` because the baseline jobs remain red.
|
||
- Historical Node/auth/browser/docs PASS and harness/Ruff/Compose FAIL evidence remains bound to
|
||
its recorded source where not rerun. L2, PSD/manual, and provider prerequisites remain
|
||
`PENDING`; no new Docker image manifest was generated.
|
||
- Durable evidence: `.artifacts/task-15/automated-gates.json`,
|
||
`.superpowers/sdd/2026-08-18-thothii-authentication-remediation/task-4-report.md`, and
|
||
`.superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-2-report.md`.
|
||
- Current automated-gates SHA-256 is
|
||
`6c516db5c2064c4a4a2e5f25961b993cd4a8fe020bbbb822fbac7faa0c119599`; the historical Docker
|
||
manifest remains bound to its recorded older source and was not reused for this candidate.
|
||
- **State:** the three original remediation Important findings remain `RESOLVED`; the fix-round-2
|
||
lifecycle Important is `ADDRESSED`; the Windows diagnostics Minor is `ADDRESSED`; authentication
|
||
remediation is `PASS`. Separately, release readiness remains `FAIL`, with L2, PSD/manual, and
|
||
provider gates `PENDING`, until unrelated deployment, runner, baseline, and external gates close.
|
||
|
||
### P3 effective configuration and `.tht-dwh` — implementation complete, automated PASS, manual PASS (2026-08-13)
|
||
|
||
- **Scope:** P3 (PRD D3): a versioned shared canonicalizer produces the non-secret effective
|
||
DWH/preprocessing configuration and a stable logical identity
|
||
(`workspace://<id>@v1:<sha256>`), used identically by the application sessions and the operator
|
||
CLI. `OWNER.json` writes are versioned; legacy roots remain readable; content-only/Evidence-only
|
||
changes keep the identity (no forced reconfiguration), while DWH-affecting changes fail closed
|
||
(never silently reusing the old generation).
|
||
- **Memory:** explicit workspace-global `paths.memory` root with a guarded migration command
|
||
(`tht memory migrate`) that copies and verifies exactly one legacy JSONL under the workspace
|
||
lock and fails closed on conflicts.
|
||
- **Revision-scoped records:** schema and Evidence Qdrant point IDs, payloads and queries include
|
||
`workspace_revision`; memory/solved stay workspace-wide.
|
||
- **Operator contract:** `tht` now carries `effectiveConfigIdentity`/`configFingerprint`/
|
||
`inputFingerprint` in results; the operator config lease path is deterministic for the same
|
||
revision+identity.
|
||
- **Retained evidence:** `.artifacts/p3-integration/p3-da9428d84f152fe059d41a89436496b7/`
|
||
(15/15 checks PASS), bound to clean source commit
|
||
`3b0726472e15c157…`.
|
||
- **Manual gate:** P3 walkthrough in `docs/testing/p2-p6-manual-verification.md`; decision
|
||
**PASS** (owner approval 2026-08-13).
|
||
### P4 Qdrant collection lifecycle — implementation complete, automated PASS, manual PASS (2026-08-13)
|
||
|
||
- **Scope:** P4 (PRD D4): one shared TypeScript collection manager owns the Qdrant collection
|
||
and payload-index contract; session admission self-heals a missing collection (1024/cosine +
|
||
the 8 required keyword payload indexes) and adds missing indexes, but never mutates an
|
||
incompatible collection (`semantic_index_incompatible`); the operator path keeps
|
||
`require_existing` semantics.
|
||
- **Host CLI:** `tht workspace vector inspect` (read-only contract report) and
|
||
`tht workspace vector rebuild --workspace <id> --collection <name> --confirm <name> --destroy`
|
||
(guarded delete/recreate of only the descriptor-owned collection, with durable state before
|
||
deletion and verification after recreation; mismatched confirmation or missing `--destroy`
|
||
→ exit 2).
|
||
- **Key files:** `backend/src/workspaces/qdrant-collection.ts` (+test), `backend/src/tht/tht-runner.ts`
|
||
(`qdrantEnsure` self-heal for admission; default `require_existing` elsewhere),
|
||
`backend/src/workspaces/runtime-config-lease.ts` (lease exposes `semanticQdrantUrl`),
|
||
`backend/src/workspace-maintenance.ts` + `preprocessing-service.ts` (`vector-inspect`/`vector-rebuild`
|
||
operator commands), `tools/tht/internal/workspaceops/operations.go` (+tests).
|
||
- **Automated acceptance:** PASS 11/11 (run `p4-466bbfdea9ef3111f36baa99fc2d64aa`,
|
||
report `.artifacts/p4-integration/p4-466bbfdea9ef3111f36baa99fc2d64aa/` retained via `--keep`,
|
||
bound to clean source commit `e056c19e6214254a9e3b2390e24c389920b84e95`): preflight, clean_state,
|
||
ownership, qdrant_up, self_heal_create_missing, self_heal_repairs_missing_index,
|
||
incompatible_refused, require_existing_refused, rebuild_recreates_contract, secret_scan,
|
||
cleanup_confinement.
|
||
- **Gates:** backend 666/666 + tsc clean; Go build+test 9/9; p4 runner unit tests 3/3; harness
|
||
841 passed (only the two pre-existing debt failures unchanged).
|
||
- **Manual acceptance:** PASS (owner approval 2026-08-13) — walkthrough section P4 in
|
||
`docs/testing/p2-p6-manual-verification.md`.
|
||
|
||
|
||
### P5 curated FK annotations in Git — implementation complete, automated PASS, manual PASS (2026-08-13)
|
||
|
||
- **Scope:** P5 (PRD D5): the canonical curated FK file is `<workspace-id>/schema/annotations.yaml`,
|
||
a regular Git blob at the same commit as the descriptor. Absence is compatible (empty canonical set
|
||
+ warning); symlinks, trees/gitlinks, cross-namespace paths, oversized (>16 MiB), non-UTF-8, and
|
||
malformed objects are refused at activation. Activation synchronizes the blob to the immutable
|
||
revision root `/data/sessions/<id>/revisions/<commit>/artifacts/mschema/annotations.yaml` with a
|
||
restrictive mode and an adjacent ownership manifest (`workspace`, `commit`, `blobId`,
|
||
`contentDigest`, `destination`); re-sync is idempotent and re-verifies, and tampered destinations
|
||
fail closed.
|
||
- **Runtime root:** the backend renders `paths.annotations_root` for the pinned revision while
|
||
`paths.artifacts`/`indexes`/`memory`/`sessions` stay workspace-global (the binding-keyed DWH cache
|
||
at `artifacts.parent` is untouched); the harness resolves annotations from `annotations_root` with a
|
||
legacy fallback.
|
||
- **Review primitive:** `tht ... workspace schema accept --run <id> --yes` is the only human FK
|
||
review path. It validates the current synced Git blob with the harness parser and records
|
||
`{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision, blobId }`. Missing `--yes`, an
|
||
unknown run, an empty/malformed blob, or a non-matching candidate fails closed (`annotation_invalid`)
|
||
without recording a review. The P2 host-file `schema check --annotations --reviewed-candidates`
|
||
review write is superseded (read-only validation only).
|
||
- **Continuation gate:** `preprocess run` continues only when the accepted review's blob digest equals
|
||
the current revision's synced annotations digest and the DWH binding is compatible; otherwise it
|
||
records a new `manual_review_required` checkpoint.
|
||
- **Key files:** `backend/src/workspaces/annotations-sync.ts` (+test), `backend/src/workspaces/
|
||
annotations.ts`, `backend/src/workspaces/git-repository.ts` (`annotationsObject`),
|
||
`backend/src/workspaces/registry.ts` (activation validation + sync), `backend/src/workspaces/
|
||
preprocessing-service.ts` (`acceptSchema` + continuation gate), `backend/src/workspace-maintenance.ts`
|
||
(`schema-accept`), `tools/tht/internal/workspaceops/operations.go` (+tests), `harness/tht/
|
||
config.py` + `cli/schema_cmd.py` (`paths.annotations_root`), `docs/contracts/
|
||
workspace-preprocessing-cli.md`.
|
||
- **Gates:** backend **689/689** + tsc clean; Go build+test 9/9; harness focused schema/annotations
|
||
52 passed. Full-suite re-run and the clean-state process goal are recorded at the acceptance gate.
|
||
- **Automated acceptance:** PASS 10/10 (run `p5-66b1f1e74f147a23c0a4bff04e6d2a4c`, report
|
||
`.artifacts/p5-integration/p5-66b1f1e74f147a23c0a4bff04e6d2a4c/` retained via `--keep`, bound to
|
||
clean source commit `9db0299063d5068198c05dc467d7f86dc34de85b`): preflight, clean_state, ownership,
|
||
activation_sync, accept_happy_path, revision_isolation, accept_negatives, continuation_gate,
|
||
secret_scan, cleanup_confinement. Runner: `scripts/p5-acceptance.sh` /
|
||
`backend/scripts/p5-acceptance.mjs` (+unit test `scripts/test-p5-acceptance.sh`).
|
||
- **Manual acceptance:** PASS (owner approval 2026-08-13) — walkthrough section P5 in
|
||
`docs/testing/p2-p6-manual-verification.md`.
|
||
|
||
### P6 commit-addressed Evidence materialization — implementation complete, automated PASS, manual PASS (2026-08-13)
|
||
|
||
- **Scope:** P6 (PRD D6): filesystem Evidence `<id>/evidence` is materialized from the exact pinned
|
||
Git commit into the immutable revision content root `<registry>/snapshots/<commit>/<id>/evidence`
|
||
at activation, with a sibling bounded manifest `<id>/evidence.manifest.json` whose digest is chained
|
||
into `snapshot.json`.
|
||
- **Safety:** fixed Git plumbing (`ls-tree -r -z` + `cat-file blob`), no shell, no mobile checkout;
|
||
symlinks/gitlinks at any depth, traversal/absolute/duplicate/cross-namespace paths, and non-regular
|
||
modes are refused. Installation-local bounds (defaults): 4096 entries, 64 MiB total, 8 MiB per
|
||
file, 4096 path bytes, 1 MiB manifest; a size-sum preflight runs before writing and no partial root
|
||
is published. Re-activation reuses a valid root and fails closed on a tampered manifest.
|
||
- **Engine:** `evidencePolicy` no longer stops filesystem sources (`evidence_materialization_required`
|
||
retired); `preprocess evidence`/`preprocess run` operate on the materialized root. Evidence Qdrant
|
||
records remain revision-scoped; corpus ACTIVE is revision-qualified. HTTP/S3 Evidence is unchanged.
|
||
- **Retention:** materialized roots live inside the commit-addressed snapshot directory, so they are
|
||
retained while pinned and removed by the existing snapshot retention scan when unreferenced.
|
||
- **Key files:** `backend/src/workspaces/evidence-materialization.ts` (+test),
|
||
`backend/src/workspaces/git-repository.ts` (`evidenceTreeObjects`/`evidenceTreeId`/
|
||
`evidenceBlobBytes`/`gitObjectSize`), `backend/src/workspaces/registry.ts` (activation staging +
|
||
integrity chain), `backend/src/workspaces/preprocessing-service.ts` (stop removal),
|
||
`backend/src/workspaces/types.ts` + `config.ts` (limits), `docs/contracts/
|
||
workspace-preprocessing-cli.md`.
|
||
- **Gates:** backend **698/698** + tsc clean; Go build+test 9/9 (unchanged); harness focused suites
|
||
pass. Full-suite re-run and the clean-state process goal recorded at the acceptance gate.
|
||
- **Automated acceptance:** PASS 10/10 (run `p6-7a4c4d0ebb63399cfa9f674738b9e8fc`, report
|
||
`.artifacts/p6-integration/p6-7a4c4d0ebb63399cfa9f674738b9e8fc/` retained via `--keep`, bound to
|
||
clean source commit `124891bbfe8dc8270e8b58c4206150eb8bebeaa7`): preflight, clean_state, ownership,
|
||
activation_materialization, evidence_preprocess, revision_isolation, unsafe_tree_refused,
|
||
bound_refused, secret_scan, cleanup_confinement. Runner: `scripts/p6-acceptance.sh` /
|
||
`backend/scripts/p6-acceptance.mjs` (+unit test `scripts/test-p6-acceptance.sh`).
|
||
- **Manual acceptance:** PASS (owner approval 2026-08-13) — walkthrough section P6 in
|
||
`docs/testing/p2-p6-manual-verification.md`.
|
||
|
||
### P7 PSD migration — plan + repository restructured + local validation PASS; owner-gated (2026-08-13)
|
||
|
||
- **Plan:** `docs/superpowers/plans/2026-08-13-p7-psd-migration.md`.
|
||
- **Done (autonomous):** `/Users/mp/projects/tht-workspace-psd` restructured to the P1.1 layout and
|
||
committed (`thoth-workspaces.yaml` + `psd-clinical/workspace.yaml` schema v3 + `psd-clinical/
|
||
evidence/` 36 `.md` + `psd-clinical/schema/annotations.yaml` 42 KB); legacy runtime dirs gitignored
|
||
and the old flat `psd.yaml` retired. A local `WorkspaceRegistry.bootstrap()` against a scratch bare
|
||
clone **activated `psd-clinical`** (descriptor valid, 36 Evidence materialized + manifest, 42 KB
|
||
annotations synced, `workspace-docs` generated) with no DWH/secret access.
|
||
- **Templates:** `deploy/psd/{workspace-bindings,operator,thothii-installation}.env.example` +
|
||
gitignored `secrets/`; operator checklist in `docs/install/psd-workspace-setup.md` (registered in
|
||
MkDocs nav).
|
||
- **Published (2026-08-13):** private repo `https://github.com/mptyl/tht-workspace-psd` (main =
|
||
`d4f9185`), consumed via SSH deploy key `thothii-psd` (read-write, passphrase-less, generated in
|
||
`deploy/psd/secrets/`). Real operator config is wired (gitignored): `deploy/psd/operator.env`,
|
||
`workspace-bindings.env`, `thothii-installation.yaml`, `connector-secrets.yaml` + `secrets/`
|
||
(DWH X-API-Key reused from the legacy `.env`; no CA — the DWH REST is public HTTPS).
|
||
- **Stack live:** started via `tht start` (project `thothii-70417a3e30ea`), all services
|
||
healthy, `qwen3-embedding:0.6b` present; the registry cloned + activated `psd-clinical`
|
||
(`ready`); `tht workspace inspect` returns `ok` with descriptor/catalog/runtime identities.
|
||
Gotcha recorded: `tht` uses a per-descriptor Compose project name, so the stack must be
|
||
started with `tht start` (not a raw `compose-with-preflight.sh up`).
|
||
- **Preprocessing live (2026-08-13):** with VPN active, `tht workspace preprocess run
|
||
--workspace psd-clinical` **succeeded** against the real PSD DWH — DWH introspection + LSH
|
||
(163 tables / 2275 columns), FK review (no new candidates: the 42 KB curated annotations are
|
||
authoritative), schema index (2438 records) and filesystem Evidence index (36 docs / 43 chunks).
|
||
Qdrant `psd-clinical` now holds **2482 revision-scoped points** (`schema_table` 164,
|
||
`schema_column` 2275, `evidence` 43; all carry `workspace_revision`). Rerun is idempotent
|
||
(Evidence `unchanged: 36`).
|
||
- **Fixes shipped during the live run** (real-DWH scale revealed them): (1) pruned ~95 GB of orphaned
|
||
acceptance-run Docker volumes; (2) raised `workspace-maintenance` tmpfs `/tmp` 64 MiB → 1 GiB
|
||
(PSD LSH snapshot is ~105 MB); (3) `vector rebuild` now recreates the 8 keyword payload indexes
|
||
(it only created dimensions/distance); (4) Qdrant upserts are chunked (256 points/batch) — a 2438-
|
||
record schema batch exceeded Qdrant's 32 MiB JSON limit; (5) frozen Evidence metadata lists now
|
||
stay lists (`FrozenList`) instead of tuples, preserving JSON shape; (6) embedding timeout 30 s →
|
||
300 s and batch 32 → 16 for large CPU corpora.
|
||
- **Remaining:** live session smoke on `psd-clinical` (P8 L2) — create a session with a real
|
||
natural-language question and reach the first reviewer gate.
|
||
|
||
### Final aggregate P2–P6 verification — automated PASS, manual PENDING (2026-08-13)
|
||
|
||
- **Aggregate process goal:** one clean-state run exercises the complete DWH → FK → schema →
|
||
filesystem Evidence chain through `tht`/the operator surface, proves idempotency and
|
||
revision isolation, proves a second installation consumes the same Git workspace with its own
|
||
state, exercises unsafe-tree and bound negatives, and cleans only owned resources.
|
||
- **Automated acceptance:** PASS 12/12 (run `p2p6-ee542112c526ef0d4c25ddf6c8bc164b`, report
|
||
`.artifacts/p2p6-integration/p2p6-ee542112c526ef0d4c25ddf6c8bc164b/` retained via `--keep`, bound
|
||
to clean source commit `1dcf4051b0d9db8ae163e4d7c53871564ca3c564`): preflight, clean_state,
|
||
ownership, activation_materialization, dwh_chain, fk_schema_evidence_chain, revision_isolation,
|
||
second_installation, unsafe_tree_refused, bound_refused, secret_scan, cleanup_confinement. Runner:
|
||
`scripts/p2p6-acceptance.sh` / `backend/scripts/p2p6-acceptance.mjs` (+unit test).
|
||
- **Full suites + builds (design §10):** harness **873 passed / 4 deselected** (with color disabled;
|
||
the forced-color environment splits `--help` flags and trips the gate-CLI consistency test only);
|
||
backend **698/698** + tsc + build; frontend **364/364** + `tsc -b` + build; `tht` Go
|
||
build+test **9/9**; `git diff --check` clean.
|
||
- **Manual acceptance:** PENDING — "Final aggregate P2–P6 verification" in
|
||
`docs/testing/p2-p6-manual-verification.md`.
|
||
|
||
### User-guide deliverable (owner requirement) — written, review PENDING (2026-08-13)
|
||
|
||
- **`docs/guida-utente.md`** (Italian, simple words + examples) covers: (1) preparing the workspace
|
||
Git repository (catalog + schema-v3 descriptor + Evidence + curated annotations), (2) using the
|
||
ThothII tools for the repository (`tht` commands + read-only workspace management), and (3)
|
||
using the base ThothII application (sessions, questions, gates). It ends with a complete
|
||
Policlinico San Donato walkthrough and links to the technical contracts.
|
||
- Registered in the MkDocs nav (`mkdocs.yml`). Owner review PENDING.
|
||
|
||
### P2 host preprocessing CLI — implementation complete, automated PASS, manual PENDING (2026-08-11)
|
||
|
||
- **Scope:** P2 (PRD D2, based on the P1.1 registry contract): the installed native `tht`
|
||
binary is the only host interface for workspace preprocessing. Commands: `workspace inspect`,
|
||
`preprocess dwh`, `schema suggest-fks`, `schema check`, `index-schema`, `preprocess evidence`,
|
||
`preprocess run`, with the exact grammar, file-ingress bounds, result contract and exit codes in
|
||
`docs/contracts/workspace-preprocessing-cli.md`.
|
||
- **Operator:** `workspace-maintenance` is a profile-gated Compose service sharing the core image,
|
||
with no Pi auth/state, no backend/Pi/frontend listener, no Git credentials, and a compiled Node
|
||
entrypoint (`backend/src/workspace-maintenance.ts`) driving the existing harness engine through
|
||
pristine JSON machine interfaces (`schema_cmd.py`, `vector_cmd.py`, `preprocess_cmd.py`).
|
||
- **Boundaries honored:** FK review is digest-bound (candidate digest == persisted artifact; a
|
||
review accepted for the same candidate content counts); Qdrant collections are never created by
|
||
the product path (`require_existing` + pre-provisioned fixture, P4 owns lifecycle); filesystem
|
||
Evidence stops with `evidence_materialization_required` (P6); HTTP Evidence enforces an
|
||
installation private-host allowlist; `ssh_tunnel` stays fail-closed (P10); cross-revision DWH
|
||
reuse is explicitly P3.
|
||
- **Retained evidence:** `.artifacts/p2-integration/p2-b109757b26388a5ed6b1d173dee86584/`
|
||
(11/11 checks PASS), bound to clean source commit
|
||
`de5de36f9a4edfd4fbebf277822090871ccdd61f`.
|
||
- **Manual gate:** P2 walkthrough in `docs/testing/p2-p6-manual-verification.md`; the owner
|
||
approved P2 on 2026-08-11 (manual acceptance PASS). P3 and later start only after an explicit
|
||
new authorization.
|
||
|
||
### P1.1 workspace-directory registry — automated integration PASS, manual PENDING (2026-08-11)
|
||
|
||
- **Scope:** P1 correction (not preprocessing). Root curator-owned catalog `thoth-workspaces.yaml`;
|
||
one self-contained directory per workspace (`<id>/workspace.yaml`, optional `<id>/evidence/**`);
|
||
generated docs stay API-owned under `workspace-docs/<id>`; internal immutable snapshots remain
|
||
flat (`<snapshots>/<commit>/<id>.yaml`) to preserve session pins and runtime trust.
|
||
- **Ownership:** the API may create a descriptor once when its catalog slot exists and the
|
||
descriptor Git object is absent at the exact base commit. Existing descriptors and curated
|
||
content are curator-owned and change only through Git commit/push then installation pull.
|
||
Update/delete publish payloads are refused as HTTP 409 `workspace_curator_owned`. Catalog and
|
||
Evidence are never written/staged/cleaned by the API. Explicit pull may produce one deterministic
|
||
docs-only follow-up commit that never touches curator bytes.
|
||
- **Schema/UI:** schema v3 remains the only descriptor schema; filesystem Evidence URI is exactly
|
||
`<id>/evidence`. Browser workspace management is read-only for ready workspaces (Pull/Sync,
|
||
Validate, installation Test, Export, Evidence summary, curator Git guidance) and offers an
|
||
editable bootstrap form only for `configuration_required` catalog slots.
|
||
- **Retained evidence:** `.artifacts/p11-integration/p11-ac0b047024fb09eeca218512526a6b23/`
|
||
(`report.json` sha256 `44250145fede36de5de941262beb833c920e8c73366d987cdd738856aac6f6d6`),
|
||
19/19 checks PASS, bound to clean source commit
|
||
`eac472011e465c24572d9a6bae14de0fb3e246c0` / tree `4fd15ec28d3b7967b7b8757158013307fb20f3b9`.
|
||
- **Verification:** backend Vitest **634 passed / 41 files** + tsc + build; frontend Vitest
|
||
**364 passed / 54 files** + tsc + build; harness focused Evidence/config pytest **39 passed**;
|
||
install-docs and schema-v3-only gates PASS; `workspace-registry-smoke.sh` and
|
||
`unified-deployment-smoke.sh` full Docker runs PASS with exact cleanup.
|
||
- **Known limitations:** P2–P6 plans/designs are unchanged and their old source paths are
|
||
inventoried for a later owner-approved adaptation plan. Windows Docker startup and native
|
||
PowerShell contract were not executed on a Windows host. P1's accepted historical evidence and
|
||
process artifacts remain untouched; the old P1 process commands are not rerunnable against the
|
||
superseding P1.1 repository contract.
|
||
- **Manual gate:** `.artifacts/manual-acceptance/p11/` prepared for the reviewer;
|
||
follow `docs/testing/p11-manual-acceptance.md`. The owner reviewed the walkthrough and
|
||
approved the implementation on 2026-08-11.
|
||
|
||
```text
|
||
P1.1 automated integration: PASS
|
||
P1.1 manual acceptance: PASS (owner approval 2026-08-11)
|
||
```
|
||
|
||
# P1 configuration process — ACCEPTED 2026-08-10
|
||
|
||
- Retained evidence: `.artifacts/p1-integration/p1-038bf31360180dc831220b33fbadcfe6/report.md`
|
||
- Final report hashes: `report.json` `f07d49097966de6f0307490089fdb2ae61379c04b7fc7177d3c44cf001e1b46a`; `report.md` `09b6a9e9ad9eed2b049e286af452e12fa1f3174ea8d633253542604470890c6c`.
|
||
- automated integration: PASS
|
||
- manual acceptance: PASS — explicitly approved by the project reviewer on 2026-08-10.
|
||
- The retained run is bound to clean source commit
|
||
`c7338969d7c7c1c396d9099b7ab2d309b70ab6cf` and tree
|
||
`5f7013904806054b9f587230be89f34cbc80f5fc`. Its hash-bound provenance contains exact
|
||
43-file backend source and 39-file compiled `dist` manifests (manifest SHA-256
|
||
`eb6d6c77c78d14c798b50d0be430ad124b8fd8965afbc4bb07d358089d23f49` and `9f9e8899f8aca882ff08d49ec6cd00caeea75691c8280095a9b88bc74939a30a`).
|
||
- The retained audit has exactly 15 PASS checks and 134 unique declared artifacts whose final
|
||
bytes match every SHA-256 declaration. It records 749 PASS command events and 1,664 production
|
||
child/network events, with listener shutdown and refusal checks recorded in the final ownership
|
||
artifact. Raw Git rejects configured executable diff drivers and other helper-bearing state.
|
||
- Manual production acceptance binds every regular compiled distribution file through an immutable
|
||
manifest and cached verified module bytes; imported dependency replacement is refused before
|
||
RUNNING. Snapshot rendering validates the bounded `snapshot.json`, expected digest, and Git blob
|
||
identity, refusing regular source replacement without publishing output.
|
||
- Final Task 8/9 focused suites pass (48/48 Task 8; 59/59 manual acceptance and renderer checks),
|
||
backend TypeScript/build pass, frontend tests/build pass. Historical harness pytest/Ruff debt
|
||
remains unrelated to this P1 work.
|
||
|
||
## Internal Qdrant + Ollama semantic infrastructure — LIVE 2026-08-08
|
||
|
||
- **Compose topology.** The mandatory application stack is `frontend`, `core`, `qdrant`,
|
||
`embedding`, and the one-shot `embedding-model-init`. Startup is CPU-first by default; Linux
|
||
hosts may opt into GPU exposure with `THOTH_ENABLE_EMBEDDING_GPU=1`. Qdrant is private on the
|
||
Compose network and persists `/qdrant/storage` in `qdrant-data`. Ollama persists its local model
|
||
cache in `embedding-models`, and `embedding-model-init` blocks `core` until
|
||
`qwen3-embedding:0.6b` is present.
|
||
<!-- workspace-descriptor-contract:start -->
|
||
- **Semantic contract.** Internal semantic indexing is fixed to `qwen3-embedding:0.6b`,
|
||
`1024` dimensions, and cosine distance. Schema v3 is the only accepted workspace descriptor.
|
||
Schema v1 and v2 workspace descriptors are rejected before activation. Candidate snapshot
|
||
validation makes activation or a pull fail atomically and leaves the prior valid snapshot active;
|
||
there is no in-product migrator or automatic conversion. One workspace owns one Qdrant
|
||
collection, and
|
||
schema, Evidence, and Memory records coexist inside that collection with payload `kind`
|
||
separation.
|
||
<!-- workspace-descriptor-contract:end -->
|
||
- **Final review runtime barriers.** Operational routes, retained session pins, and runtime
|
||
rendering now require schema version 3 before resolving bindings, readiness, diagnostics, or
|
||
Pi. Session admission verifies the exact internal Qdrant collection (dimensions, cosine
|
||
distance, and required keyword payload indexes) before Ollama and before manifest persistence.
|
||
The Qdrant adapter binds every search/list/delete filter to its constructed workspace identity
|
||
and rejects conflicting caller namespaces.
|
||
- **Boundary and persistence.** Only DWH and LLM remain external runtime application endpoints.
|
||
There are no active external vector or embedding endpoint instructions, bindings, or secrets in
|
||
the supported operator manuals. Qdrant remains a derived but persistent semantic index: the
|
||
canonical sources of truth stay the workspace Git descriptors, phase artifacts, and memory
|
||
registry/ledger. The Ollama model cache is recoverable for offline startup but is not the
|
||
canonical source of semantic content.
|
||
- **Backup and recovery.** `./scripts/vector-backup.sh --project-name <name> --output <file>`
|
||
archives exactly one labeled `<project>_qdrant-data` volume and preserves the prior `qdrant`
|
||
running state. `./scripts/vector-restore.sh --project-name <name> --input <file>
|
||
--confirm-project <name>` requires the exact repeated project confirmation, validates manifest
|
||
and archive safety before stopping `qdrant`, stages rollback content, restores semantic storage
|
||
in place, and restarts `qdrant` only if it was previously running. Recovery requires the registry
|
||
to already hold a reviewed v3 descriptor revision compatible with the restored collection; the
|
||
helper does not restore descriptors, rename collections, or repair a semantic-index
|
||
incompatibility. Backup and restore share one atomic Docker-daemon lock per Compose
|
||
project/Qdrant volume; contenders fail before volume resolution, and cleanup removes the lock
|
||
only when its ownership labels still match.
|
||
- **Verification recorded for Task 13 final audit.** On Apple M4 Pro
|
||
(`Darwin 25.5.0`, Docker Server `29.6.2 linux/arm64`), harness pytest passed
|
||
**827 passed / 4 deselected**; backend Vitest passed **477/477** plus TypeScript and build;
|
||
frontend Vitest passed **374/374** plus TypeScript and build; `git diff --check` passed.
|
||
Deployment contracts passed:
|
||
`test-default-compose.sh`, `test-unified-compose.sh`, `test-internal-semantic-compose.sh`,
|
||
`test-no-deployment-coupling.sh`, `test-compose-secret-policy.sh`, and
|
||
`verify-workspace-install-docs.sh --fixtures-only`.
|
||
- **Task 13 Docker smoke evidence.** CPU semantic smoke passed in **217.34s** and proved
|
||
offline Qdrant/Ollama persistence plus exact cleanup. Workspace registry smoke passed in
|
||
**42.06s** in fix round 1 with a per-run image tag derived from the unique Compose project,
|
||
and proves exact cleanup of compose containers, volumes, networks, and only that smoke image.
|
||
Unified deployment smoke passed in **125.57s**; update-only rollback smoke
|
||
passed in **85.40s**; Linux server deployment smoke passed in **55.99s**. The previously
|
||
observed `tht` rollback failure did not recur.
|
||
- **Task 13 image and manual-gate notes.** Verified pinned runtime images:
|
||
`qdrant/qdrant:v1.18.2@sha256:75eab8c4ba42096724fdcfde8b4de0b5713d529dde32f285a1f86fdcb2c9e50c`
|
||
and
|
||
`ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a`.
|
||
The workspace-registry smoke fix-round image used tag
|
||
`thothii-workspace-registry-smoke:thoth-workspace-registry-smoke-thoth-workspace-registry-smoke-10vi3a-19157`,
|
||
built manifest list `sha256:715b943057929418cad4aa71806d9edbaf823555d19bda6b875297617463fd4a`
|
||
with config `sha256:a566521981e08958aae9a12bfc7803bb5f3f835536b4bb8c39df8fcf26063161`,
|
||
and removed that exact reference during cleanup. Local GPU exposure (`THOTH_ENABLE_EMBEDDING_GPU=1`) and
|
||
Windows Docker Desktop startup were not manually executed in this run.
|
||
- **Task 13 known limitations.** Broad harness Ruff remains existing unrelated debt
|
||
(**220 errors**); touched harness files were verified Ruff-clean. The final active-reference
|
||
audit remains non-empty only in deterministic negative guards, retained off-repository migration
|
||
SQL, L2 compatibility fixtures, gitignored task notes, and historical reference notes. No active
|
||
schema-v3 operator manual or supported runtime deployment path retains external vector or
|
||
embedding endpoint coupling.
|
||
- **Final review fix verification.** Backend Vitest passed **477/477** plus TypeScript and build;
|
||
harness pytest passed **827 passed / 4 deselected** with the existing 74 warnings; touched Python
|
||
files are Ruff-clean. The complete `tht` Go suite, deterministic backup/restore safety test,
|
||
internal semantic Compose contract, no-deployment-coupling gate, CPU/offline semantic smoke, and
|
||
unified deployment smoke all pass after the final fix. The intermittent `tht` rollback failure was
|
||
traced to Docker Desktop alternating equivalent bind sources between `/private/...` and
|
||
`/host_mnt/private/...`. Exact state-v4 source hashes remain unchanged; only fresh bind
|
||
observations made by a Darwin `tht` carry non-serialized aliases for the rollback
|
||
comparison, so pre-fix recovery state remains readable and Linux `/host_mnt` paths remain
|
||
distinct. The rollback-only smoke passed twice consecutively after each fix revision, and the
|
||
subsequent full unified smoke passed with exact cleanup.
|
||
|
||
# Historical archive
|
||
## Historical snapshots and archived reference notes
|
||
|
||
### Historical snapshot — Unified deployment release gate, Task 13 (2026-08-05)
|
||
|
||
- **Release coverage.** `scripts/unified-deployment-smoke.sh` gates the two-service render/build,
|
||
frontend-to-core routing, embedded pinned Pi, Git registry bootstrap, offline recreation, valid
|
||
update, invalid-update retention, and the four persistent stores. `scripts/tht-update-smoke.sh`
|
||
independently exercises the bad-Pi update and automatic rollback path.
|
||
`scripts/server-deployment-smoke.sh` starts the server plus required session overlays with the
|
||
same smoke-built core/frontend images, disposable bind roots/secrets/session configuration,
|
||
upstream-auth checks, and fail-closed unavailable-session behavior.
|
||
- **Isolation and disclosure boundary.** Every run generates a unique temporary root, Compose
|
||
project, container/image names, transaction image tags, and run label. The rollback fixture uses
|
||
an immutable `hello-world` digest whose preflight exits successfully, guaranteeing the stopped
|
||
core state required by `tht` compensation. Cleanup includes stopped project containers in
|
||
its final ownership check immediately before teardown and removes only exact containers,
|
||
Compose resources, image references, control state, and temporary files. There is no global
|
||
prune. Failure diagnostics are bounded and sanitized, and all credentials/endpoints used by the
|
||
smokes are disposable fixtures rather than operator or repository secrets. Every public smoke
|
||
also has an internal 30-minute process-group supervisor with TERM/KILL of the complete group.
|
||
- **Cross-platform CI contract.** `.github/workflows/deployment.yml` uses immutable action commits,
|
||
pinned supported Node and Go versions, runs LF/Compose/secret/coupling/docs/TypeScript gates on
|
||
Linux, runs each Linux Docker smoke once under its own outer timeout, and copies the Windows
|
||
source into a path containing spaces before building/invoking native `tht` and rendering
|
||
Compose. The optional `windows_docker_startup` dispatch targets a labelled self-hosted Windows
|
||
Docker Desktop/WSL2 runner and performs bounded two-service startup and exact cleanup. No local
|
||
Windows or Windows Docker execution is claimed until that manual job is recorded.
|
||
- **Validation status.** Deterministic Phase A gates, backend **434/434** plus TypeScript,
|
||
frontend **386/386** plus TypeScript, and harness **862 passed / 5 L2 deselected** are green.
|
||
Review round 1 ran each Docker smoke exactly once without retry. Unified (`103.86s`) and
|
||
update-only (`46.45s`) passed build/start, core/Pi/registry/persistence setup and the stopped
|
||
candidate preflight, but `tht` stopped before mutation at its active-session inventory gate.
|
||
Round 2 replaces presence-only fixture checks with generated Compose renders plus the production
|
||
workspace resolver; this found and fixed missing explicit direct transport selections. The
|
||
clean-server preflight now atomically initializes the three hidden Pi-agent targets under the
|
||
writable parent bind while protected/tracked sources remain separate read-only mounts. Clean
|
||
empty-root render/setup and wrong-service/value/mount mutations are green. The corrected server
|
||
one-shot built and started both healthy services from an empty Pi-state root, then stopped at an
|
||
incorrectly addressed authenticated frontend hop. Fix round 3 adds the exact fourth private
|
||
non-admin claim and proves its nginx/backend transformation in a focused auth test. It also
|
||
centralizes schema-v2 registry descriptor resolution and secret-safe runtime rendering in
|
||
`ThtRunner`, preserving canonical revision identity and durable session roots for inventory,
|
||
create/resume/show, SQL, and Pi calls. The fresh update-only one-shot now passes mutation,
|
||
automatic `rolled_back` compensation, exact prior-image restoration, unchanged registry head
|
||
and mount identities, all four persistence sentinels, post-rollback doctor/workspace checks,
|
||
and exact labeled-resource cleanup. The one authorized server invocation was blocked at its
|
||
first Docker readiness call by the execution sandbox's socket permission before any Compose
|
||
resource could be created, so authenticated workspace/fail-closed session behavior remains an
|
||
explicit release gate. Native Windows PowerShell/Docker execution also remains pending.
|
||
|
||
### Historical snapshot — Portable deployment decoupling (superseded 2026-08-08)
|
||
|
||
- **Mandatory stack.** The supported Compose stack is exactly `frontend` plus `core`; use the
|
||
base file with `deploy/compose.local.yaml`, or with `deploy/compose.server.yaml` plus the
|
||
required public-server session overlay. `run-stack.sh`
|
||
invokes the base+local Compose command and the core image provides Pi, so no host Pi binary is
|
||
part of the launch contract.
|
||
- **External boundaries.** DWH, vector DB, embedding, LLM, and reverse-proxy services are
|
||
external configurable endpoints even when deployed on the same infrastructure. The two
|
||
superseded PSD/portal deployment overlays were removed. Workspace descriptors and migration
|
||
utilities remain separate from deployment runtime configuration.
|
||
- **Legacy PSD deployment ruling.** The PSD bootstrap was deleted because it generated the
|
||
retired overlay and was therefore deployment machinery, not a data migration utility. Its
|
||
remaining live contract checks were renamed for the generic local Compose profile. The coupling
|
||
gate rejects stale active deployment filenames and content while deliberately excluding
|
||
historical plans/specs, canonical workspace descriptors, and non-runtime migration helpers.
|
||
- **Fresh provider and secret contract.** Local, server, and standalone development mount the
|
||
protected Pi auth JSON plus tracked declarative model/settings files read-only under
|
||
`/home/thoth/.pi/agent`. The existing strict application bundle is a core-only Docker secret at
|
||
`/run/secrets/thothii.secrets`; operator env files contain only its absolute source path.
|
||
Provider readiness is exercised from a fresh Compose volume through model listing, configuration,
|
||
and sanitized credential status.
|
||
- **Install and scan closure.** Superseded copied one-service installation examples and the
|
||
provider-owned-network test are retired. Active manuals use the canonical base plus local/server
|
||
and optional overrides, while the category-based coupling scan covers runtime, Docker smoke,
|
||
install, operator, and positive deployment-test contracts and propagates scanner errors.
|
||
|
||
### Historical snapshot — Portable Git workspace registry, pre-schema-v3 (superseded 2026-08-08)
|
||
|
||
- **Source of truth and scope.** The canonical workspace repository is a generic Git remote,
|
||
configured only by `THT_WORKSPACE_GIT_REMOTE` and `THT_WORKSPACE_GIT_BRANCH` (there is no
|
||
committed PSD/Chirone remote or branch default). Both a local Docker installation and a server
|
||
persist its checkout, validated snapshots, state, and locks at `/data/workspace-registry`.
|
||
Connector endpoints, transport choices, and secret-file paths remain local bindings; secret
|
||
contents are never stored in Git, API responses, browser storage, diagnostics, or bundles.
|
||
- **Migration and session safety.** Schema-v2 descriptors are operational; legacy descriptors are
|
||
visible as `migration_required` until migrated by the documented operator workflow. New sessions
|
||
acquire a persistent revision lease before readiness and persist workspace ID plus immutable Git
|
||
revision. Retention hands that lease off only after an authoritative scan observes the manifest,
|
||
so a stale concurrent scan cannot prune the pinned snapshot. Resume resolves that historical
|
||
snapshot, while retention preserves every revision referenced by an open, closed, or failed
|
||
unarchived manifest.
|
||
Reconciliation runs only with a complete local installation list or an administrator's complete
|
||
server list, never from a remote user's partial view.
|
||
- **SSH connector boundary.** The current OpenSSH forward is owned by one bounded diagnostic and is
|
||
always cleaned up afterward. DWH/vector `ssh_tunnel` bindings therefore return
|
||
`workspace_not_activatable`, and new-session creation rejects them before persistence. Direct and
|
||
REST runtime connectors remain supported; Git remote access over SSH is unaffected.
|
||
- **Operator manuals.** Follow [the local manual](docs/install/local-workspace-registry.md) for
|
||
macOS/Windows/Linux Docker Desktop deployment and [the server manual](docs/install/server-workspace-registry.md)
|
||
for Gitea-compatible remotes, reverse proxy, migration, backup, and recovery. The release
|
||
workflow is Git review/push → installation pull → validate → local diagnostic test → browser-local
|
||
workspace/model/reasoning selection → revision-pinned session.
|
||
- **Verification recorded for this source branch.** `git diff --check` passed; backend Vitest
|
||
**371/371** and TypeScript passed; frontend Vitest **398/398** and TypeScript passed; the
|
||
harness document regression passed **10/10**. `./scripts/workspace-registry-smoke.sh` and the
|
||
executable installation-manual fixture verifier passed with Docker. A final unrestricted full
|
||
harness run remains a release command for the deployment environment; the earlier local
|
||
long-running harness run was intentionally cancelled before it produced a final result.
|
||
|
||
### Historical snapshot — Session summary redesign (2026-07-23)
|
||
|
||
- Session documents are projected at read time in outcome-first order: original question,
|
||
final SQL, persisted data preview, revised question, assumptions, one memory list, then
|
||
remaining technical documents. This applies to existing filesystem and repository-backed
|
||
sessions without rewriting their artifacts.
|
||
- Final SQL has an always-visible clipboard action. All prose, including original/revised
|
||
questions, assumptions, memory content, and remaining decisions, renders as Markdown.
|
||
- Memories are shown once, approved before declined. The generic decision list suppresses
|
||
memory ledger records plus `phase_approved`, `phase_auto_approved`, `table_approved`,
|
||
`table_promoted`, and `column_promoted`.
|
||
- The session summary has its own accessible pointer/keyboard resize separator, persists its
|
||
width independently from Model activity, reaches 50% when space permits, and preserves a
|
||
512 px right-side minimum on narrower desktop layouts.
|
||
- Verification: harness **861 passed / 5 L2 deselected**, Pi gate **163/163**, full frontend
|
||
suite, TypeScript check, production build, Ruff, and `git diff --check` all passed. A live
|
||
pre-existing session returned the new canonical order and none of the suppressed decision
|
||
labels. Compose rebuilt and force-recreated both services; core image
|
||
`sha256:3566d1258b956f8ca96d3b5ff8f625503247a7fd3b0dffe77020ae03956403d8`
|
||
is healthy and frontend image
|
||
`sha256:8311ca1308b459ece7236bf143da7b1a226ff4082fed924e1b5a207c24b6ca29`
|
||
is running. Frontend and `/api/health` both returned HTTP 200.
|
||
|
||
### Historical snapshot — Local Pi user auth + startup failure handling (2026-07-21)
|
||
|
||
- The PSD Docker profile now bind-mounts the configurable host `PI_AUTH_FILE` read-only at
|
||
`/home/thoth/.pi/agent/auth.json`; on this Mac it resolves to the real user profile
|
||
`/Users/mp/.pi/agent/auth.json`. The container keeps its correct Linux identity
|
||
`HOME=/home/thoth` while Pi sees the user's independent `deepseek` and `zai` credentials.
|
||
- `deploy/pi/settings.json` is the non-secret model policy and exposes, in order,
|
||
`zai/glm-5.2`, `deepseek/deepseek-v4-flash`, `deepseek/deepseek-v4-pro`, and
|
||
`aritmolab/qwen3.6-35b-a3b`. The core image is aligned to Pi 0.80.3.
|
||
- New-session creation now validates the saved provider/model against Pi before persistence;
|
||
unavailable selections return sanitized `503 model_unavailable` without creating a manifest.
|
||
A synchronous runtime-construction failure after persistence marks that session `failed` and
|
||
returns the fixed startup-recovery message instead of leaving an ambiguous `open` session.
|
||
- Verification: backend 235/235, TypeScript clean, dedicated Compose auth/model contract green
|
||
with a demonstrated RED→GREEN cycle. Rebuilt core image
|
||
`sha256:a8b4dd9f016c2335e4da897073bc6d5bdf1e8ce9b60dcfcca171b6677f563228`
|
||
is healthy; live `/models` returned all four models; a real `deepseek-v4-pro` smoke reached its
|
||
first reviewer gate, deleted only its own session, and restored the exact prior settings.
|
||
- Deleted the three explicitly approved incomplete DeepSeek attempts:
|
||
`a390c8b8-0a91-4a37-967b-ce7ff9be9797`, `a2f974b2-4c48-4967-b4b6-afdbc2b2d541`, and
|
||
`f66e1959-3c71-4b10-8aa1-606992046b7e` (API delete 204, subsequent lookup 404 for each).
|
||
|
||
### Historical snapshot — User-owned sessions cutover (2026-07-16)
|
||
|
||
- **Target contract:** the public server runs `AUTH_MODE=upstream` with Task 4 portal identity
|
||
forwarding and Task 5 principal enforcement deployed together. Its session source of truth is
|
||
direct TLS-verified PostgreSQL `thoth_sessions`; local development remains loopback-only with
|
||
filesystem sessions under `THT_HOME`. The core never receives the migrator credential.
|
||
- **Deployment material:** copy `deploy/compose.session-server.yaml.example` and
|
||
`deploy/workspaces/server-sessions.yaml.example` into reviewed, untracked operator files. The
|
||
runtime password, migrator password, and CA are three separate Docker secret mounts; server
|
||
startup rejects public/local storage and incomplete server DB/TLS configuration.
|
||
- **Readiness behavior:** `/health` remains the unauthenticated process liveness endpoint. Any
|
||
route requiring unavailable session/preferences storage returns fixed HTTP 503 before starting
|
||
Pi; this is intentional and must not be hidden by changing liveness to a database check.
|
||
- **Manual cutover only:** schedule maintenance, drain Pi work, run the one-shot migrator and
|
||
require `pending=[]` and `drifted=[]`, then replace core and perform an authenticated storage
|
||
smoke. Archive/checksum the three reviewed legacy filesystem session directories before deleting
|
||
exactly those three with `docker/cutover-legacy-sessions.sh --delete`; no deletion has been run
|
||
from this repository task. Do not import their untrusted ownership.
|
||
- **Rollback:** PostgreSQL remains the single source of truth. Revert only to a compatible fixed
|
||
release; never re-enable filesystem persistence, restore the archive into production, or
|
||
dual-write during rollback.
|
||
|
||
### Historical snapshot — Docker locale deployment, Profile A (superseded 2026-08-05)
|
||
|
||
ThothII gira in Docker sul server co-locato, **embedded nel portale omics_portal** a `https://aritmolab.policlinicosandonato.it/datamart-builder` (backend invisibile, tutto same-origin via nginx del portale).
|
||
|
||
- **2 container** su `compose.yaml`: `thothii-core` (Fastify + harness tht + Pi) + `thothii-frontend` (Vite + nginx-unprivileged). Rete `omics_portal_omics_network` (external) con alias `thothii-core`/`thothii-frontend`.
|
||
- **DB**: Postgres diretto `:5438` (stessa istanza: schema `datawarehouse` 163 tabelle + `vectors` pgvector). Ruoli dedicati `thoth_dwh_reader` (read-only) + `thoth_vector_rw` (read+write). Embeddings: Ollama `:11434`.
|
||
- **Secrets**: `deploy/thothii.env` (env_file, gitignored) + `THT_MODEL_API_KEY_FILE` (key modello, file 0600 — meccanismo provider-credentials di Codex) + bind-mount `~/.pi` (pi-config).
|
||
- **Backend**: merge di `codex/portable-deployment` (secret-bundle, provider-credentials, auth `upstream`, security hardening, CI multiarch). Setup Docker MIO tenuto (il modello Docker-secrets di Codex è in `deploy/` come alternativa inerte).
|
||
- **Portale** (repo `omics_portal`, branch `agent/patient-capabilities-datamart-ui`): `nginx.conf` rotte `/datamart-builder/api`+`/assets` + `auth_request`, template `datamart_builder.html` (mount `<div id="root">` + tag `{% vite_assets %}`), vista `datamart_builder_api_auth`. Auth: `authentik Admins` bypass; utenti normali necessitano gruppo `omics-datamart-builder`.
|
||
- **Fix load-bearing**: `configPath` da `THT_CONFIG` (route senza workspace), `vite_assets` `mark_safe` (SPA bianca), `COPY harness/`+`cp workflow.yaml`+`pip install .` (pip 26 / tht module-relative), entrypoint `server` case.
|
||
- **Standalone/dev**: `docker-compose.dev.yml` (rete propria, porte host 8787/8090) + `scripts/docker-smoke.sh`.
|
||
- Piano dettagliato: `docs/superpowers/plans/2026-07-12-local-docker-deploy-implementation.md`.
|
||
|
||
### Archived snapshot — Runtime incident fixes (2026-07-13)
|
||
|
||
- The bind-mounted Pi profile came from host paths and did not trust `/app/harness`.
|
||
Pi 0.80 consequently loaded **zero** project extensions, prompts and skills, silently
|
||
sending `/nuova-domanda`/`/riprendi-sessione` to the model as plain text. The core
|
||
entrypoint now idempotently adds only `/app/harness` to the persistent
|
||
`/home/thoth/.pi/agent/trust.json`, preserving all existing decisions.
|
||
- The gate embeds the canonical `tht-sessione/SKILL.md` in the one-shot kickoff system
|
||
prompt and explicitly prohibits repository discovery. A live RPC `get_commands` must
|
||
show `torna`, `nuova-domanda`, `riprendi-sessione`, and `skill:tht-sessione` after deploy.
|
||
- Workspace identity is derived from the resolved config path, so
|
||
`config/tht.yaml -> workspaces/local.yaml` matches DWH artifact ownership (`local`).
|
||
- Direct pgvector now discovers the actual namespaces of the `vector` type and cosine
|
||
operator from PostgreSQL catalogs. This supports server layout `vectors.*` tables with
|
||
the extension installed in `public`.
|
||
- Live verification: session `2026-07-13-074712-dammi-la-lista-dei-pazienti-che-haoo-fat`
|
||
resumed directly at F1, ran `tht session show`, and completed `tht search pack`
|
||
(12 tables, 0 evidence, 2 solved) without repository exploration or adapter errors.
|
||
|
||
### Archived snapshot — Workflow/UI regression fixes (2026-07-14)
|
||
|
||
- **F1 Model Activity restored.** Session create/resume now preserves configured/persisted
|
||
thinking instead of forcing `off`. Pi's nested `thinking_delta` is bridged to a dedicated
|
||
named SSE `activity_delta`; EventSource subscribes to that name and the panel keeps it separate
|
||
from final assistant text. Reasoning remains in-memory and is not persisted to session artifacts.
|
||
- **F3 rewrite confirmation remains bypassed.** `rewrite_question` records approval and advances
|
||
automatically without a reviewer widget. The repeated prompt came from old running containers:
|
||
images had been rebuilt but services had not been recreated.
|
||
- **Join review is read-only and complete-set safe.** Join-only proposals render informational
|
||
cards with only `Continue` and `Other — specify`. Continue requires the exact complete id set;
|
||
all joins are persisted together by `decision add-join-set`, using an atomic ledger replacement
|
||
under a per-session cross-process writer lock. Other persists none of the rejected proposal.
|
||
- **CTE presentation fixed.** F6 CTE cards now structure purpose, rationale, tables, filters, keys,
|
||
and output columns with responsive wrapping/alignment. The Horizontal/Vertical switch is hidden
|
||
for a single SQL block (the per-CTE view), because it only affects multi-block layouts.
|
||
- **Latest render failure diagnosed and hardened.** Session
|
||
`2026-07-14-115847-estrai-i-pazienti-che-hanno-fatto-un-abl` sent an object in
|
||
`open_questions`, which React cannot render as a child. The v2 gate now enforces
|
||
`open_questions?: string[]`; the frontend also safely normalizes legacy malformed payloads.
|
||
- **Verification/deploy:** Python harness 798 passed / 5 L2 deselected; gate JS 126; backend 143;
|
||
frontend 250; TypeScript/build gates green. Compose rebuilt and force-recreated both services.
|
||
Running image ids: core `sha256:55acef2f12151ea97144c2f5e9164d63f2ca734bc2746fef553df94849e3fb3f`;
|
||
frontend `sha256:1043f79392420149655cc63d70461e2ca2005b2290a1e3e21dcf845ec3bd1c81`.
|
||
|
||
### Archived snapshot — Pi-enabled model selector (2026-07-14)
|
||
|
||
- **Pi is the allowlist authority.** `/models` reads the mounted Pi `enabledModels`, intersects
|
||
it with models currently available from Pi, and preserves the configured order. Enumeration
|
||
does not require `PI_PROVIDER`, does not inject generic/provider credentials, and fails closed
|
||
for missing or malformed scope.
|
||
- **Live scope:** exactly `deepseek/deepseek-v4-flash`, `zai/glm-5.2`, and
|
||
`local-qwen/qwen3.6-35b-a3b`. The live endpoint returned those three composite IDs once each and
|
||
in that order; `zai/glm-5v-turbo` and all other authenticated Pi models are hidden.
|
||
- **Validation/process smoke:** live settings updates returned 200 for DeepSeek Flash and local
|
||
Qwen, while hidden GLM-5V returned 400; a post-restore equality check confirmed the original app
|
||
settings were restored. The real `PiProcessManager` configure path succeeded for DeepSeek and
|
||
local Qwen without sending a prompt or starting a DWH operation; local Qwen required no hosted
|
||
provider key. Unknown and compound providers remain fail-closed in the verified backend suite.
|
||
- **Verification/deploy (`2026-07-14T19:24:22+02:00`):** backend **154/154** and frontend
|
||
**251/251** passed; both TypeScript gates and `git diff --check` were green. Compose built and
|
||
force-recreated only `core`; container start was `2026-07-14T17:24:01.992971976Z`, health was
|
||
`healthy`, and sanitized post-recreate logs contained only the backend listen line. Rebuilt image
|
||
ID and running container image ID both equal
|
||
`sha256:577f99754fd0731251c8ddd8608b1b8baee09d02fad66c759b23f8221083e676`.
|
||
|
||
### Archived snapshot — Qwen connectivity + state-aware Resume recovery (2026-07-14)
|
||
|
||
- **Pi turns have an explicit lifecycle.** The bridge tracks `idle`, `running`, `waiting`,
|
||
and `failed`; a reviewer gate is `waiting`, responses/steering return to `running`, and an
|
||
assistant provider error or unexpected Pi child exit becomes `failed`. Provider error details
|
||
are never forwarded to the client; the UI receives a fixed sanitized recovery message.
|
||
- **Resume preserves only active work.** `running`/`waiting` runtimes return as already active.
|
||
Every validated cold path—including recovery after a child has already exited—clears stale SSE
|
||
state before reopening and restarts from persisted provider/model/thinking with
|
||
`/riprendi-sessione`; `idle`/`failed` runtimes are torn down at that point. Failed validation does
|
||
not detach the existing stream. A successful Resume of the currently selected session also
|
||
closes and recreates its EventSource, so the replacement runtime cannot be left behind an old
|
||
same-ID stream.
|
||
- **Private Qwen routing is live.** Core is attached to both `omics_portal_omics_network` and
|
||
external `localllm_default`; frontend remains only on the portal network. The mounted Pi profile
|
||
resolves `local-qwen/qwen3.6-35b-a3b` at the sanitized base URL
|
||
`http://localllm-vllm:8000/v1`. A direct probe from core verified the model catalog and received
|
||
a non-empty real chat completion.
|
||
- **Verification/deploy (`2026-07-14T21:41:52+02:00`):** backend **168/168** and frontend
|
||
**256/256** passed; both TypeScript gates, both production builds, the Qwen Compose network
|
||
contract, and `git diff --check` exited 0. The initial deployment built and force-recreated
|
||
`core` and `frontend`; after the final crash-recovery review, only the affected `core` image was
|
||
rebuilt and force-recreated with no active Pi session. Core is healthy and its post-recreate
|
||
Qwen catalog/completion probe succeeded. Built and running image IDs match: core
|
||
`sha256:9867c2fa002b6f117da9a1d02c73b47bfe0372b8c1c137a5174c3e7ecb1e1db1`, frontend
|
||
`sha256:5a47f81bc887423e05cef8fd3feb075aa600cb33217247186124f60ca5a3005b`.
|
||
The frontend entry hash changed, so only `omics_portal-web-1` was restarted to invalidate its
|
||
indefinite Vite-manifest cache. The application-level Qwen smoke reached its first
|
||
`ui_request`, persisted the expected provider/model, deleted only its uniquely named smoke
|
||
session, restored the exact saved settings object, and left no smoke session or Pi runtime. The
|
||
supplied probe's success path left its keep-alive SSE reader open, so only that probe process was
|
||
terminated (exit 143), without touching backend, Pi, or unrelated runtimes. The same smoke then
|
||
exited 0 with `controller.abort()` in cleanup, preserving the gate, session cleanup, and exact
|
||
settings-restoration evidence.
|
||
|
||
### Archived snapshot — Complete activity timeline + CTE spacing (2026-07-15)
|
||
|
||
- **Model activity is complete from F1.** The left panel now records the submitted prompt before
|
||
session creation completes, then projects thinking, assistant output, sanitized tool lifecycle,
|
||
reviewer gates, status, and turn lifecycle in chronological order. It remains in-memory only;
|
||
closing and reopening the panel does not discard it, while Resume intentionally starts a fresh
|
||
live timeline.
|
||
- **Tool activity is a narrow public contract.** Only call id, tool name, and
|
||
`running`/`completed`/`failed` status cross Pi → backend → SSE. Updates, arguments, results,
|
||
commands, output, credentials, and raw errors stay server-side. Resume and SSE replay were also
|
||
hardened for process restarts, concurrent lifecycle requests, stale callbacks, cursor reset, and
|
||
multi-client reconnects.
|
||
- **CTE plan layout uses real Tailwind 3 spacing.** Shared cards use concrete 16 px default / 12 px
|
||
compact padding utilities; F6 CTE headers and content use responsive 16/20 px edge padding.
|
||
Semantic ordered steps, single-boundary divided filter/table lists, long-value wrapping, and a
|
||
plain top-divider rationale preserve the artifact content while improving scanability.
|
||
- **Final verification/deploy (`2026-07-15T02:18:54+02:00`, HEAD `56d73d2`):** backend
|
||
**204/204** and frontend **285/285** passed; both TypeScript gates and production builds exited
|
||
0, and `git diff --check` was clean. The final lifecycle fixes make restarted-hub cursor replay
|
||
generation-aware and mutate frontend delete/resume state only for sessions actually deleted.
|
||
Compose rebuilt and force-recreated `core` and `frontend`; built and running image ids match:
|
||
core `sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147`
|
||
(`healthy`), frontend
|
||
`sha256:dd7ef721b661b57b8c5422c47088e1a9a2e9821af021541cab372c9892fbd638`
|
||
(`running`). The entry changed from `index-DcZviApa.js` to `index-CuIt1NQg.js`, so only
|
||
`omics_portal-web-1` was restarted to refresh its indefinite manifest cache.
|
||
- **Final live local-Qwen smoke:** session
|
||
`2026-07-15-001922-final-no-thinking-activity-smoke-2026-07` observed **0**
|
||
`activity_delta` events while receiving **5** strictly allowlisted tool lifecycle events and the
|
||
first reviewer gate (`bash` running/completed and `reviewer_select` running). No forbidden tool
|
||
field crossed SSE. Cleanup closed with 200, deleted only that session with 204, restored the
|
||
exact settings object, and left no Pi runtime or smoke session.
|
||
|
||
### Archived snapshot — Filtered Model activity projection (2026-07-15)
|
||
|
||
- **Resolved contract.** `activityLog` still folds the complete in-memory prompt, thinking,
|
||
assistant, sanitized tool, reviewer-gate, status, and turn-lifecycle history. The left panel now
|
||
applies a default-deny rendering boundary and shows only prompt, thinking, status, and gate;
|
||
assistant text remains available to the central transcript, while tool, lifecycle, and unknown
|
||
future activity kinds do not render or move the panel scroll.
|
||
- **Frontend-only verification/deploy (`2026-07-15T03:44:02+02:00`, source HEAD `7208599`):**
|
||
frontend **287/287** passed; `npx tsc -b`, `npm run build`, and `git diff --check` exited 0.
|
||
Compose rebuilt and force-recreated only `frontend`; its image changed from
|
||
`sha256:dd7ef721b661b57b8c5422c47088e1a9a2e9821af021541cab372c9892fbd638` to
|
||
`sha256:75fc0b7786c75ded1b488e9fad3232aa061c82d89140c67ad62b5285954d1d36`, with container start
|
||
`2026-07-15T01:42:45.582760612Z`. The active Vite entry changed from
|
||
`index-CuIt1NQg.js` to `index-BdZpFO7j.js`, so only `omics_portal-web-1` was restarted at
|
||
`2026-07-15T01:42:53.715330158Z`; core retained image
|
||
`sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and start
|
||
`2026-07-15T00:18:37.040145636Z`.
|
||
- **Real local-Qwen no-COT smoke:** session
|
||
`2026-07-15-014328-activity-filter-smoke-2026-07-15t01-43-2` reached its first reviewer gate
|
||
with **0** `activity_delta` events and **3** tool lifecycle events, each containing exactly the
|
||
four public fields. The probe accepted the close response, deleted only that session with 204,
|
||
restored the exact saved settings object, confirmed the session absent, and left no Pi runtime.
|
||
|
||
### Archived snapshot — Central activity log + compact CTE density (2026-07-15)
|
||
|
||
- **Resolved UI contract.** The central working body now renders every chronological non-blank
|
||
assistant transcript line in one bounded accessible log, without user-entry echoes,
|
||
timer/spinner labels, or step messages. The left Model activity panel is a default-deny
|
||
projection of only thinking and status, while the complete raw activity fold and the existing
|
||
reviewer widgets, artifacts, and workflow state remain unchanged. F6 CTE headers, content,
|
||
table rows, and filter rows use 8 px vertical padding with 12 px lateral padding below `sm` and
|
||
16 px from `sm` upward; divider top padding is 8 px. The final review amendment keeps historical
|
||
log rows at the full muted-foreground token so their normal-size text retains AA contrast.
|
||
- **Source verification (source HEAD
|
||
`09f9bdffe582ff3c66845ca219f60c11472a550a`).** Frontend tests passed **292/292** across
|
||
**43/43** files. `npx tsc -b`, `npm run build`, the Impeccable layout detector, and
|
||
`git diff --check` all exited 0; the detector returned `[]`.
|
||
- **Frontend-only deployment.** The pre-deploy frontend was image
|
||
`sha256:8c9aaee6453b26c69e4057d3f9e53aa791d449b88ba4a0669a25cd442667d582`, started
|
||
`2026-07-15T10:17:16.919166255Z`, serving `index-CveSyban.js`. Compose built and
|
||
force-recreated only `frontend` with `--no-deps`; the final running frontend is image
|
||
`sha256:39dc47d81abcb13466218500476fbb78d1be66a424f4293470437700c848c26d`, started
|
||
`2026-07-15T10:30:30.869532521Z`, serving `index-CihtpQJV.js`. Because the entry changed,
|
||
exactly `omics_portal-web-1` was restarted: it retained image
|
||
`sha256:4580cf2bc85f3656ee515996c24ed02d9096d24ed0d8afe384a2f8bd8cf4bac9` and moved from
|
||
start `2026-07-15T10:17:36.151611451Z` to `2026-07-15T10:30:46.369205604Z`.
|
||
- **Isolation and final state.** Core remained `running`/`healthy` with exactly its original image
|
||
`sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and start
|
||
`2026-07-15T00:18:37.040145636Z`; frontend and portal were `running` with no container
|
||
healthcheck. Pre/post `docker top` showed only core's supervisor and backend server, so no
|
||
unrelated Pi runtime existed to disturb. The count-only frontend sensitive/error pattern scan
|
||
was **0**. No live model smoke was run, and settings and sessions were intentionally untouched.
|
||
|
||
### Archived snapshot — Resizable activity split + compact CTE rows (2026-07-15)
|
||
|
||
- **Resolved UI contract.** `activityLog` remains the complete in-memory chronological fold. The
|
||
left Model activity panel default-denies every kind except prompt, thinking, and assistant,
|
||
labels those entries Question, Reasoning, and Response in source order, and hides status, tool,
|
||
gate, lifecycle, and unknown kinds. The desktop panel is pointer/keyboard resizable from 288–576
|
||
px while preserving 512 px centrally, persists its global width in localStorage, and becomes an
|
||
overlay drawer below `lg` or whenever the measured app shell is narrower than 800 px. F6 CTE
|
||
cards retain their semantic structure and responsive grids;
|
||
lateral padding is 12/16 px, header/content edge padding is 8 px, internal section gaps are 12
|
||
px, heading/divider spacing is 4 px, and table/filter rows use 4 px vertical padding with compact
|
||
line heights.
|
||
- **Source verification (`2026-07-15T15:22:17+02:00`, source HEAD
|
||
`f1af1f909b387ae12a10f1b534bf6a529ea42505`).** Frontend tests passed **295/295** across
|
||
**44/44** files. `npx tsc -b`, `npm run build`, and `git diff --check` exited 0; the Impeccable
|
||
layout detector returned `[]`. The local source build emitted Vite entry
|
||
`assets/index-PlvQqhNG.js`.
|
||
- **Frontend-only deployment.** The pre-deploy frontend was image
|
||
`sha256:39dc47d81abcb13466218500476fbb78d1be66a424f4293470437700c848c26d`, started
|
||
`2026-07-15T10:30:30.869532521Z`, serving `index-CihtpQJV.js`. Compose built and force-recreated
|
||
only `frontend` with `--no-deps`; the final running frontend is image
|
||
`sha256:0947211373784a860c7507d03c0fcf1f901ac3d1547cd0aa62b914d158f15bd5`, started
|
||
`2026-07-15T13:21:08.29613952Z`, serving `index-Dn7T524a.js`. Because the entry changed, exactly
|
||
`omics_portal-web-1` was restarted: it retained image
|
||
`sha256:4580cf2bc85f3656ee515996c24ed02d9096d24ed0d8afe384a2f8bd8cf4bac9` and moved from
|
||
start `2026-07-15T10:30:46.369205604Z` to `2026-07-15T13:21:21.968808414Z`.
|
||
- **Isolation and final state.** Core remained `running`/`healthy` with exactly its original image
|
||
`sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and start
|
||
`2026-07-15T00:18:37.040145636Z`; frontend and portal were `running` with no container
|
||
healthcheck. Pre/post `docker top` showed only core's supervisor and backend server, so no
|
||
unrelated Pi process existed and no Pi process was stopped or steered. The count-only frontend
|
||
sensitive/error pattern scan was **0**. No live model smoke was run; settings and sessions were
|
||
intentionally untouched.
|
||
|
||
### Archived snapshot — Final activity-split fix (2026-07-15)
|
||
|
||
- **Source and verification (`2026-07-15T15:56:57+02:00`).** Deployed source commit
|
||
`1f540fcb78ac9e552e56a21e47edf66e9872b323` (`1f540fc`). Frontend Vitest passed **298/298**
|
||
tests across **44/44** files; `npx tsc -b` and `npm run build` exited 0. The Impeccable detector
|
||
scoped to AppShell, ModelActivityPanel, index.css, and CtePlanViewer returned `[]`; `git diff
|
||
--check` exited 0.
|
||
- **Frontend-only deployment.** Compose built and force-recreated only `frontend` with `--no-deps`.
|
||
The frontend image changed from
|
||
`sha256:0947211373784a860c7507d03c0fcf1f901ac3d1547cd0aa62b914d158f15bd5` to
|
||
`sha256:6e14f55092b7e3aca9a396220394ae484147674d81b051771e394e59b73b1c88`; its active Vite entry
|
||
changed from `index-Dn7T524a.js` to `index-BIznZeLH.js`. Therefore exactly
|
||
`omics_portal-web-1` was restarted to refresh its manifest cache; it retained image
|
||
`sha256:4580cf2bc85f3656ee515996c24ed02d9096d24ed0d8afe384a2f8bd8cf4bac9` and started at
|
||
`2026-07-15T13:56:32.786693915Z`.
|
||
- **Isolation and final state.** Core retained image
|
||
`sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and exact original
|
||
start `2026-07-15T00:18:37.040145636Z`, remaining `running`/`healthy`. Final frontend and portal
|
||
states are `running` (no healthcheck). Pre/post core process tables contained only the supervisor
|
||
and backend server, so Pi was preserved and no Pi process was stopped or steered. The count-only
|
||
frontend sensitive/error-pattern scan was **0**. No model smoke was run; settings and sessions
|
||
were intentionally untouched.
|
||
|
||
## What ThothII is
|
||
|
||
A **human-in-the-loop datamart builder**: it turns a natural-language question into
|
||
validated SQL (and optionally a dbt datamart) through a deterministic **8-phase
|
||
NL→SQL workflow**, where the model *proposes* and a human *reviewer decides* at gates.
|
||
The UI is meant to embed inside the Omics Portal (GSD design system) and is **English**.
|
||
|
||
## Architecture — three layers
|
||
|
||
```
|
||
frontend (React, :5173) → backend (Fastify, :8787) → pi --mode rpc → tht / harness → DWH (read-only)
|
||
```
|
||
|
||
- **harness/** — the Pi layer. A deterministic Python CLI **`tht`** + a Pi gate extension
|
||
(`.pi/extensions/tht-gate.js`) that runs the 8-phase workflow and emits/consumes
|
||
widget-descriptor JSON. **Owns all persistence.** Workflow truth is `harness/workflow.yaml`;
|
||
orchestration rules are `harness/.pi/skills/tht-sessione/SKILL.md`.
|
||
- **backend/** — Fastify + TypeScript. A **thin bridge**: proxies REST routes to the `tht`
|
||
CLI (`ThtRunner`), manages Pi processes (`PiProcessManager`, one child per session),
|
||
bridges Pi RPC events to SSE (`SessionBridge` + `SseHub`). No application database.
|
||
- **frontend/** — React 18 + base-ui + Tailwind + TanStack Query + Zustand. Chat-style
|
||
shell (`src/shell/AppShell.tsx`); the live transcript is rebuilt in-memory from the SSE
|
||
stream (`src/store/sessionStore.ts`), **not persisted**.
|
||
|
||
### Persistence model (the load-bearing premise)
|
||
There is **no verbatim chat store**. Each workflow phase persists its own document into the
|
||
session directory, and that **IS** the persistence. A session = a directory under the
|
||
workspace's `sessions/` path containing `session_manifest.yaml` + phase artifacts
|
||
(`question.md`, `schema_linking.json`, `cte_plan.json`, `sql_final.sql`,
|
||
`validation_report.md`, `review_decisions.jsonl`, …). A fresh Pi process resumes by reading
|
||
`tht session show <id>` + the on-disk artifacts — never by replaying chat.
|
||
|
||
## The 8 phases (harness/workflow.yaml)
|
||
F1 chiarimento · F2 memoria · F3 riscrittura (`question.md`) · F4 schema_linking
|
||
(`schema_linking.json`) · F5 sintesi · F6 cte (`cte_plan.json`, `cte_tests.json`) ·
|
||
F7 sql_finale (`sql_final.sql`) · F8 datamart. Current phase is a fold over the decision
|
||
ledger (`harness/tht/phase.py`); statuses: `open` / `closed` / `finalized`.
|
||
|
||
## How to run
|
||
|
||
**Full stack (real Pi + DWH):** from project root,
|
||
```bash
|
||
./scripts/run-stack.sh
|
||
# Opens frontend: http://localhost:5173 (proxies backend :8787)
|
||
```
|
||
**Prereqs:** VPN on; `pi` on PATH (with a configured model, e.g., `pi model set claude-fable-5`);
|
||
`harness/.env` populated (see `.env.example`); `harness/config/tht.yaml` → workspace (psd recommended for testing).
|
||
All three layers' deps installed (`npm install` in each, `python -m venv + pip install -e ".[dev]"` in harness).
|
||
|
||
**Individual dev:**
|
||
- backend: `cd backend && npm run dev` (tsx watch; env: `PORT`, `THT_HARNESS_DIR`, `THT_BIN`, `PI_BIN`, `AUTH_MODE`)
|
||
- frontend: `cd frontend && npm run dev` (Vite; `VITE_BACKEND_URL` → backend)
|
||
- harness install: `cd harness && python -m venv .venv && pip install -e ".[dev]"` → `tht` on PATH
|
||
|
||
## How to test (latest TS gates green 2026-07-15: backend 204 / frontend 292 (43 files); harness 798 / gate JS 126 last recorded 2026-07-14)
|
||
- harness: `cd harness && .venv/bin/pytest -q` (5 L2/real-DB tests are deselected by default)
|
||
- backend: `cd backend && npx vitest run` · typecheck `npx tsc --noEmit -p .`
|
||
- frontend: `cd frontend && npx vitest run` · typecheck `npx tsc -b` · e2e `npm run e2e` (Playwright)
|
||
|
||
## Config & workspaces
|
||
- Workspaces: `harness/workspaces/*.yaml` (`psd`, `tht-test`, `tht.example`). A workspace sets
|
||
the DB target and the **absolute** `paths.sessions/artifacts/indexes` (psd → a *separate*
|
||
repo `tht-workspace-psd/`, NOT committed here).
|
||
- Secrets live ONLY in `harness/.env` (gitignored; `THT_*` — DB, DWH REST, vector, SSL CA…).
|
||
See `harness/.env.example` for the variable list.
|
||
- App settings (global): `{ workspace, provider, model, thinking }`, persisted via harness
|
||
preferences (`tht session preferences get/set` → the configured session repository —
|
||
filesystem or Postgres in server mode). `backend/data/settings.json` (gitignored) remains
|
||
only the file fallback for injected runners/tests. The "New session" form is question-only;
|
||
these settings supply the rest.
|
||
|
||
## Efficiency levers (NL→SQL workflow optimization, 2026-07-08)
|
||
|
||
Three deployed optimizations target model thinking time (the dominant cost, ~220s in F1 alone):
|
||
|
||
1. **Join-graph via FK annotations + `tht schema suggest-fks`**
|
||
- DWH has no FK constraints declared. Annotations file (`tht-workspace-*/artifacts/mschema/annotations.yaml`) now stores curated logical FKs.
|
||
- Three ranking rules: mine from approved SQL (highest confidence), heuristics (`*_time_key → dim_time.day_key`), same-name PK discovery with `--assume` flag for disambiguity.
|
||
- **Psd workspace:** 228 FK suggestions already generated (139 tables); `tht schema suggest-fks --from-sql <session-dir> --assume cod_paz=dim_patient` for updates.
|
||
- **Activation:** automatic. The mschema renderer populates the `【Foreign keys】` section. F4 in SKILL.md now reads FK joins from there instead of the model re-deriving them.
|
||
|
||
2. **Context-pack consolidation at F1 kickoff (`tht search pack`)**
|
||
- Single embedding of the question, reused for schema + evidence + solved-question searches.
|
||
- Command: `tht search pack "<question>" --session <id>` → `sessions/<id>/retrieval_pack.md` (tabelle candidate, relevant evidence, solved exemplars).
|
||
- Graceful degradation: if Ollama or vector store unreachable (no VPN), sections are empty but exit 0 — session continues with live searches.
|
||
- **Activation:** automatic at next session. SKILL.md F1 now prescribes as first call; reduces exploratory turns.
|
||
|
||
3. **Phase-summary recap auto-construction from session ledger**
|
||
- `tht session show --json` includes the full decisions ledger; `tht phase meta --json` exports decision types per phase.
|
||
- Gate appends deterministic `【Decisioni registrate in questa fase】` section to v2 phase-summary artifacts.
|
||
- Model authors only `summary` + `checks`; the gate fills the recap table from persisted state → exact by construction.
|
||
- **Activation:** automatic at next session and Pi restart. SKILL.md Disciplina 6 updated: model keeps output brief, gate enriches from catalog + ledger.
|
||
|
||
**Tests:** 358 Python + 111 JS gate, all pass. L2 (live DWH) verification on psd workspace recommended when time permits.
|
||
|
||
## Conventions & contracts (don't relearn the hard way)
|
||
- **`-c`/`--config` is a PER-COMMAND option in `tht`** — append it AFTER the subcommand,
|
||
never globally (`ThtRunner.buildArgv` handles this).
|
||
- **`--json` output must be pristine** (only valid JSON on stdout).
|
||
- **UI strings are English.** Document *content* stays in the workspace language (Italian
|
||
for psd) because it's the real data; only chrome/labels are English.
|
||
- **Settings are global**, not per-question.
|
||
- TDD throughout; tests assert real behavior, not mocks. Frequent, scoped commits.
|
||
- Global user rules (`~/.claude/CLAUDE.md`): think before coding, simplicity first, surgical
|
||
changes, goal-driven verification.
|
||
|
||
## Active memory — F8 promotion gate + solved-question recall — SHIPPED, L2 pending (2026-07-07)
|
||
|
||
Two additions to close the loop on reusable memory, on top of the existing `tht memory
|
||
search` (Phase 2) reuse:
|
||
|
||
- **F8 promotion gate.** `reviewer_memory_promote` (Phase 8, called with only the session
|
||
id): the gate computes candidates deterministically via `tht memory promote --preview
|
||
--json` (the 3 reusable decision types, already excluding previously promoted/declined
|
||
ones) and shows a pre-selected checklist. Selected → `tht memory save-one` persists to
|
||
the vectordb + records `memory_promoted`; deselected → `memory_promotion_declined`
|
||
(ledger detail `seq:<n>`) so it is never re-proposed. `tht memory promote`/`save-one` were
|
||
added to the gate's anti-bypass FORBIDDEN list (model must go through the gate tool).
|
||
- **Solved-question exemplars.** New vector kind `solved_question` reusing the existing
|
||
`memory` pgvector table (no server-side DDL); `harness/tht/solved.py` does a one-row
|
||
upsert keyed by a hash of question+SQL. CLI: `tht memory solved-index` / `solved-search`.
|
||
`tht session finalize` auto-indexes the pair (best-effort: green line on upsert, cyan
|
||
"già aggiornata" on dedup no-op, yellow warning + the recovery command
|
||
`tht memory solved-index <id>` on failure). `SKILL.md` now prescribes calling
|
||
`solved-search` as reference-only context in F4 (schema linking), F6 (CTE plan) and F7
|
||
(final SQL), and documents the finalize auto-index in "Session end".
|
||
|
||
**Pending L2 gate (not yet run — needs VPN + writer key):** one live end-to-end session on
|
||
workspace `psd` via `./scripts/run-stack.sh` to verify (a) the promotion checklist renders
|
||
pre-selected and persists selected/declined correctly, (b) finalize indexes the pair,
|
||
(c) `tht memory solved-search` returns it with sql + tables.
|
||
|
||
**Fast-follow:**
|
||
- RestSearcher top-k dilution — **client-side DONE** (2026-07-07): `search_similar` manda
|
||
`kinds` alla RPC (filtro server-side esatto) con fallback automatico su server legacy
|
||
(404 → retry senza filtro, post-filter client). **Resta la migrazione server** della
|
||
funzione SQL `search_similar` (+`kinds text[] DEFAULT NULL`): istruzioni pronte in
|
||
`harness/docs/vector-rest-kinds-migration.md`; l'ordine di deploy è libero, ma fino
|
||
alla migrazione il filtro resta client-side e la diluizione persiste.
|
||
- ~~`tht memory solved-search` muore con traceback grezzo se il vectordb è irraggiungibile~~
|
||
**DONE** (2026-07-07): degrada a warning di una riga su stderr, stdout puro (`[]` in
|
||
--json), exit 0 — copre VectorRestError/EmbeddingsError/OperationalError.
|
||
|
||
## Review gates v2 — payload strutturati + viewer dedicati — COMPLETE (2026-07-07)
|
||
|
||
Plan: `~/.claude/plans/prima-di-passare-ai-inherited-marshmallow.md`. Merged to `main` @ `2410f01`
|
||
(ff, pushed). Executed via subagent-driven-development (4 workstreams, task reviews, final
|
||
whole-branch review + fix wave).
|
||
|
||
- **Contracts:** `artifact.data.schema_version: 2` for `cte_plan` / `cte_result` / `phase`,
|
||
built **deterministically by the gate** (catalog descriptions via `tht schema columns`; SQL
|
||
from `ctes/<name>.sql`; preview rows persisted by `tht cte test`); the model contributes only
|
||
purpose/rationale/note. Non-v2 payloads fall through to the legacy renderers — old sessions
|
||
and `tools/replay/replay.json` keep working.
|
||
- **Harness (Python):** `CteTestRecord.preview_rows` (+ `_jsonable` coercer, ≤10 rows, cells
|
||
≤200 chars); new read-only `tht cte info <name> --session <id> --json` (index/total from
|
||
`cte_plan.json`, same source as `next_cte`); `tht cte plan --doc -` writes
|
||
`cte_plan_doc.json` (chain documentation; `cte_plan.json` stays a load-bearing `list[str]`).
|
||
- **Gate (JS):** `gate/artifact-contracts.js` (soft validators → self-corrective `textResult`,
|
||
TypeBox untouched) + `gate/enrich.js` (pure, catalog lookups injected);
|
||
`prepareReviewerArguments` now coerces `artifact.data` too (GLM stringified-param
|
||
mitigation); `SKILL.md` Phase 5/6 + disciplines rewritten (plan via `reviewer_confirm
|
||
kind:"cte_plan"` with payload A; `cte_result` gates send THIN data only — never SQL/preview
|
||
as text).
|
||
- **Frontend:** `artifactV2.ts` types; `CtePlanViewer` (per-CTE cards + chain strip),
|
||
`CteResultViewer` (shiki SQL + AG Grid preview), `PhaseSummaryViewer` (checks + criteria
|
||
with the VALUES driving choices), `PreviewGrid` extracted from `ResultsPanel`,
|
||
`statusBadge.ts` shared success/warn/error tokens.
|
||
- **Replay:** v2 fixtures + `tools/replay/augment-review-gates.mjs`; `replay.json` regenerated;
|
||
offline visual pass ok (screenshots in the SDD scratch dir).
|
||
- **LIVE E2E (session `2026-07-07-011858`, GLM 5.2):** all 8 phases completed with the v2
|
||
gates; session **finalized** (DWH validation battery green, needs VPN).
|
||
- **Bug found live + FIXED (`2410f01`):** infinite spinner at workflow end — the bridge dropped
|
||
Pi's `agent_end` (the ONLY end-of-turn signal) and `working` was released only by the next
|
||
gate, which the final turn doesn't have. Now: bridge maps `agent_end` → SSE
|
||
`system_event`; FE tracks `agentActive`; an unexpected Pi child exit notifies the client
|
||
(info error + synthetic `agent_end`). Memory: `pi-rpc-event-vocabulary`.
|
||
- **Open (non-blocking):** `tht.sqlcheck` maps table aliases by first occurrence (found and
|
||
worked around by the model in F6 — spawned as a separate task); one more live confirmation
|
||
that the spinner stops at F8 (the chain is unit-tested end to end).
|
||
|
||
## F4 schema-linking column curation + look&feel v2 — COMPLETE (2026-07-06)
|
||
|
||
Branches `feat/f4-schema-linking-column-curation` (PR #1) + `feat/frontend-lookfeel-v2`, landed
|
||
on `main` (`d942635` … `7491e8c`). The F4 gate (`reviewer_schema_linking`) presents
|
||
catalog-enriched tables/columns (descriptions from `tht schema columns`, hardened enrichment),
|
||
per-table columns modal (suggested pre-checked, suggested-first ordering + filter box);
|
||
decisions `column_promoted`/`column_excluded` + deterministic `tht session
|
||
sync-schema-linking` projection into `schema_linking.json`. Live-verified including the
|
||
clobber test (the model's joins write preserves curated columns). Look&feel v2: shadows/radii/
|
||
mono labels, 70% gate modal, structured cards (colors untouched). Memory:
|
||
`thothii-visual-language-v2`.
|
||
|
||
## Workflow contract hardening — COMPLETE (2026-07-01)
|
||
|
||
Spec: `docs/superpowers/specs/2026-07-01-workflow-contract-hardening-design.md` · Plan:
|
||
`docs/superpowers/plans/2026-07-01-workflow-contract-hardening.md`. Merged to `main` @ `3dadc6f`
|
||
(pushed). Driven by analysis of Pi session `2026-06-30-165708` (GLM 5.2), where the model spent
|
||
~80% of its tool calls reverse-engineering the harness because `SKILL.md` mis-stated the
|
||
phase-advance contract — and Phase 6 was a hard dead-end. Three coordinated harness fixes (TDD):
|
||
|
||
- **F6 CTE-approval dead-end FIXED.** The gate's `reviewer_confirm kind:"cte_result"` used to
|
||
register `cte_approved --subject phase:6`, which `decision_cmd` rejects (exit 5 — it needs a
|
||
real CTE name from the plan) → F6 could never close. New `tht cte next --session <id>` returns
|
||
the first unapproved plan CTE; the gate now approves **by name**. (`tht/cli/cte_cmd.py`,
|
||
`.pi/extensions/tht-gate.js`.)
|
||
- **`schema_linking.json` writer/validator.** `store.set_schema_linking` (validates against the
|
||
`SchemaLinking` model, THEN writes — no partial file) → CLI `tht session set-schema-linking
|
||
<id> --file <path|->` (exit 5 on bad JSON / ValidationError) → gate tool `write_schema_linking`
|
||
(stdin). Replaces the model hand-writing the F4 artifact + ad-hoc python validation.
|
||
- **`SKILL.md` corrected to match the code.** Only F2-empty / F6-skipped auto-advance
|
||
(`_AUTO_ADVANCE_PHASES={2,6}`); every substantive phase closes with `reviewer_confirm
|
||
kind:"phase"` (F7 is **two-step**: `kind:"sql"` records `sql_approved`, then `kind:"phase"`
|
||
advances). Fixed Discipline 2 + Phase 1/3/4, added a per-phase **cheat-sheet**, documented the
|
||
`SchemaLinking` shape. The old false "the reviewer_decide already advances" (F3) claim — the
|
||
exact cause of the observed thrash — is gone.
|
||
|
||
Verified: harness pytest **281 passed** / 5 deselected, gate JS **34/34**, changed-files ruff
|
||
clean (the 36 `ruff check .` errors are pre-existing on `main`). Final whole-branch review
|
||
(opus): READY TO MERGE, no Critical/Important. Executed via subagent-driven-development
|
||
(implementer + task-review per task, final opus review). **DEFERRED (needs VPN): live F4/F6
|
||
end-to-end** — resuming session `2026-06-30-165708` (stuck at F6) is the ideal live probe.
|
||
|
||
## UI/UX redesign + Resume — COMPLETE (2026-06-30)
|
||
|
||
Plan: **`~/.claude/plans/foamy-forging-dahl.md`**. Memory: `thothii-ui-redesign-inprogress.md`.
|
||
**All workstreams done and pushed to origin/main:** D + E @ `0eeb3f7`, B + C @ `b056ff3`,
|
||
F @ `cef9ae4`, A @ `0a13f71`, G @ `e8cdd00`(scope) + `cbb8e18`(results). Nothing pending from this
|
||
plan. Per-workstream detail below for reference.
|
||
|
||
- **D — DONE** (`c12bdcd`): session display `name` = 3-5 Italian keywords via **YAKE** (no LLM),
|
||
derived in `tht session new` (CLI layer); `create_session` core unchanged (`name=None` default).
|
||
`yake` added to `harness/pyproject.toml`. TDD `tests/test_session_name.py`; harness 269 passed.
|
||
- **E — DONE** (`0eeb3f7`): rotating activity icon replaces the red dot in `CentralStatus`
|
||
(inline, clickable → opens the panel); `ModelActivityPanel` is a **5-line expandable
|
||
model-stream tail**; `WorkingSpinner` extracted to its own module; the separate spinner button
|
||
+ orphaned `Transcript.tsx` removed. Frontend 87/87, tsc clean. **Live visual check DONE
|
||
(2026-06-30):** inline spinner opens the panel; 5-line collapsed tail; expand → full transcript.
|
||
- **B — DONE** (`b056ff3`): `WorkflowBar` is now colored **dots** F1..F8, no phase-name text
|
||
(amber-translucent=running, green=done, red=error, gray=pending; green connectors lead the active
|
||
dot). Each dot carries `data-state`. Error is lightweight: store `phaseError` set when an `info`
|
||
`level=error` arrives during the phase, cleared on the next `ui_request` (`sessionStore.ts`).
|
||
**All four states live-verified** via Playwright.
|
||
- **C — DONE** (`b056ff3`): right sidebar — single-line denser rows (inline status dot + name,
|
||
`py-1`), a 3-level type hierarchy via **`/impeccable`** (L1 `SESSIONS` red/bold/wide-tracking ·
|
||
L2 section + group headers muted uppercase · L3 names normal-case), and the **"No group" label
|
||
removed** (ungrouped sessions render after the last group; guarded so the empty-state still
|
||
teaches when there are no groups). **Live-verified.** (Resume in `SessionMenu` stays with A1.)
|
||
- **Tests:** frontend **93/93** (was 87; +3 store `phaseError`, +2 `WorkflowBar` dot-state, +1
|
||
AppShell no-"No group"), `tsc -b` clean.
|
||
- **F — DONE** (uncommitted; live check deferred to G): single-select answers **auto-confirm**.
|
||
`reviewer_select` options may carry a `decision` payload (`{type, subject, detail?, rationale?}`)
|
||
and an optional `advance`; picking such an option persists the decision directly via
|
||
`tht decision add` (shared `decisionAddArgs` helper, also used by `reviewer_decide`) — no redundant
|
||
`reviewer_decide`/`reviewer_confirm` gate. Options without a payload stay ask-only; back/exit/Other
|
||
never persist. Pure logic extracted to `resolveSelectOutcome`/`decisionAddArgs` (exported, unit-
|
||
tested). Contract docs updated: `reviewer_select` tool desc + `SKILL.md` (widget summary,
|
||
disciplines 2-3, Phase-1 single-pick) + the `CLAUDE.md` gate note. Gate JS **33/33**, harness 269.
|
||
**Live verification (model actually uses `reviewer_select`+decision, no follow-up gate, decision in
|
||
`review_decisions.jsonl`) deferred to G** — it is model-behavior-dependent.
|
||
- **A — DONE** (uncommitted): **A1** — `SessionMenu` gains a **Resume** item (gated to
|
||
`status!=="finalized" && !archived`), wired in `AppShell` to the existing `doResume` → `POST
|
||
/sessions/:id/resume`. 3 tests (`SessionMenu.test.tsx`); frontend **96/96**, tsc clean. **A2** —
|
||
diagnosis-first clean-room repro shows the **resume cold-start stall NO LONGER reproduces on pi
|
||
0.79.4** (8/8 chained into the tool calls, fresh + partway; GLM 5.2 now narrates AND emits
|
||
`tht session show`+`read SKILL.md` in-turn). The earlier narrate-and-stop predates the pi upgrade.
|
||
Defense-in-depth applied: `RIPRENDI_KICKOFF` hardened to force the in-turn tool call (gate test +
|
||
live regression 2/2). The cross-model angle (weaker/older models) lives in **G**.
|
||
- **G — DONE** (`e8cdd00`+`cbb8e18`): cross-model behavior matrix via a committed clean-room harness
|
||
(`harness/scripts/model-matrix.mjs`). **Tier 1** — kickoff + resume first-turn: all *available*
|
||
models chain in-turn (`zai/glm-5.2`, `deepseek/deepseek-v4-{pro,flash}`, `aritmolab/qwen3.6-35b-a3b`,
|
||
`zai/glm-4.5-air`); the resume stall recurs on none (closes A's cross-model robustness).
|
||
`aritmolab/gemma4-26b-a4b` = **404 unavailable** at the endpoint (listed but not served) — infra
|
||
gap, not a workflow issue. **Tier 2** — F single-select auto-confirm verified live on `glm-5.2`:
|
||
answering the first `reviewer_select` persisted a `concept_clarified` decision **0→1** with **no
|
||
follow-up gate** (closes F's deferred live check). Full results: the G plan doc + memory
|
||
`thothii-cross-model-matrix`. No prompt hardening needed.
|
||
|
||
**Status:** **All UI-redesign + resume workstreams done and pushed — D, E, B, C, F, A, G.**
|
||
Nothing pending from the plan. Optional nice-to-haves (not required): Tier-2 F/multiselect live for
|
||
the non-baseline models (cheap re-run with `harness/scripts/model-matrix.mjs` + the Tier-2 method),
|
||
and a one-off manual Playwright kebab→resume pass in the live UI.
|
||
|
||
## Live verification + reviewer_select fix (2026-06-30, afternoon)
|
||
|
||
Drove the real stack (Playwright → backend → real Pi → GLM 5.2 → DWH) end-to-end.
|
||
|
||
- **F1 hang fix (`418187a`) VERIFIED LIVE.** Answered an F1 reviewer widget; Pi resumed (model
|
||
socket reopened) and the gate produced new output — vs the old silent hang. The transition
|
||
"silent hang → gate re-presents/advances" proves `ctx.ui.input` now resolves.
|
||
- **New bug found + fixed: reviewer_select `choices` vs `choice`.** The gate's `reviewer_select`
|
||
(and `reviewer_confirm` reject) read `resp.choice` (singular) but the frontend uniformly sends
|
||
`choices: [id]` (array) — so every single-select gate answered "Nessuna scelta ricevuta" and
|
||
re-proposed forever (multiselect was fine; it already read `choices`). Fix: a shared
|
||
`selectedChoice(resp)` helper (`harness/.pi/extensions/tht-gate.js`) reading the array; both
|
||
handlers use it. TDD: `gate/__tests__/gate_choice.test.js` RED→GREEN, full gate suite **28/28**.
|
||
VERIFIED LIVE: a single-select answer is now accepted and the workflow advances (2/4 → 3/4).
|
||
- **Resume cold-start STALL confirmed (open item #1).** On `/riprendi-sessione`, GLM 5.2 narrates
|
||
the bootstrap step then ends the turn without the tool call → Pi idle, unrecoverable from the UI.
|
||
Memory: `thothii-resume-cold-start-stall.md`.
|
||
- **GLM 5.2 F1 is slow (~3-4 min, ~50+ reads) but works** — looks stuck but isn't; don't hit
|
||
"Stop and save" (it `POST /close`s → kills Pi). Memory: `thothii-glm52-f1-slow-not-stuck.md`.
|
||
|
||
## Earlier work — F1 reviewer-widget hang fix + multiselect guidance (committed 2026-06-30; authored 2026-06-29)
|
||
|
||
Two fixes, **committed to `main`** (7 files):
|
||
|
||
1. **Bug: every reviewer widget hung "stuck with no output" after the human answered** — F1
|
||
disambiguation (and any gate) dead-ended. Root cause, confirmed from Pi's own source
|
||
(`@mariozechner/pi-coding-agent` `dist/modes/rpc/rpc-mode.js`, `createDialogPromise`):
|
||
`ctx.ui.input` assigns its OWN RPC id (`crypto.randomUUID`) and correlates
|
||
`extension_ui_response` on THAT id, silently dropping unknown ids. The gate puts a
|
||
different id (`u${Date.now()}`) inside the descriptor carried in `title`. `SessionBridge`
|
||
was replying with the **descriptor** id, so real Pi never resolved `ctx.ui.input` → the
|
||
model never continued. **Fix:** `SessionBridge` now stores Pi's top-level `m.id`
|
||
(`pendingPiId`) on the incoming request and replies `extension_ui_response{ id: pendingPiId,
|
||
value: <uiResponse JSON> }` (value still carries the descriptor id, so the gate's internal
|
||
`resp.id === descriptor.id` check holds). File: `backend/src/bridge/session-bridge.ts`.
|
||
Full write-up: memory `pi-ui-input-id-correlation.md`.
|
||
- **The test double was masking it:** `harness/tests/fake_pi/fake_pi_rpc.mjs` had forced
|
||
`m.id == descriptor.id`. Corrected to mirror real Pi (distinct `randomUUID` top-level id,
|
||
correlate on it, drop unknown ids); `test_fake_pi_contract.mjs` gained a negative
|
||
regression test ("respond with descriptor id → no follow-up").
|
||
- TDD: `backend/test/session-bridge.test.ts` (unit) + `backend/test/e2e-f1.test.ts`
|
||
(integration — now asserts the model's follow-up arrives after the answer) went
|
||
RED→GREEN.
|
||
|
||
2. **UX: multi-answer disambiguation** — `harness/.pi/skills/tht-sessione/SKILL.md` Phase 1
|
||
now tells the model to use `reviewer_decide` (the existing multiselect/checkbox widget)
|
||
when an ambiguity admits several simultaneously-true answers, instead of single-pick
|
||
`reviewer_select`. Guidance-only — no new widget (`frontend MultiselectWidget` already
|
||
exists).
|
||
|
||
Verified at commit time: backend `npx vitest run` **67/67 green**; `tsc --noEmit -p .` **OK**;
|
||
fake-pi contract `node --test test_fake_pi_contract.mjs` **2/2 green**. **Verified LIVE
|
||
2026-06-30** (see the top "Live verification" section).
|
||
|
||
## Most recent feature — Session management (MERGED to main @ 2c21e46)
|
||
Full session management modeled on Claude's UI, all three layers:
|
||
- **Read-only "split view" panel** (left drawer, `SessionDocumentsPanel`) showing a session's
|
||
phase documents read-only (reuses `SqlViewer`/`SchemaLinkingViewer`/`MarkdownView`).
|
||
- **Rename / Move to group / Archive / Delete** via a kebab menu (`SessionMenu`) → REST →
|
||
`tht session set-name/set-group/archive/unarchive/delete`. Archive = a manifest `archived`
|
||
flag (not a dir move); groups = a manifest `group` field; delete = hard `rmtree` + confirm.
|
||
- Rail: collapsible group headers + "No group" + a separate **Archive** view.
|
||
- **Resume correctness:** read-only **guard** (HTTP 409 when `finalized` or `archived`);
|
||
`PiProcessManager.spawnFor` now has a `new`/`resume` mode (resume sends
|
||
`/riprendi-sessione <id>`); a "Phase 0 — Resume" cold-start section in `SKILL.md`.
|
||
- Design docs: `docs/superpowers/specs/2026-06-29-session-management-design.md` +
|
||
`docs/superpowers/plans/2026-06-29-session-management.md`.
|
||
|
||
### ⚠️ Open items / pending gates
|
||
1. **Resume cold-start stall — RESOLVED on pi 0.79.4 (workstream A, 2026-06-30).** The earlier
|
||
narrate-and-stop (GLM 5.2 narrating the bootstrap step then ending the turn without the tool
|
||
call) **no longer reproduces**: a clean-room repro of the backend's exact resume handshake
|
||
chained into `tht session show`+`read SKILL.md` in-turn **8/8** (fresh + partway sessions). The
|
||
pi upgrade is the likely fix. Defense-in-depth: `RIPRENDI_KICKOFF` hardened to force the in-turn
|
||
tool call (gate test + live 2/2). Memory: `thothii-resume-cold-start-stall.md`. **Remaining:**
|
||
the cross-model angle (older/weaker models) is folded into **G**; a full Playwright kebab→resume
|
||
pass through the live UI is still worth one manual run (item 2).
|
||
2. **Full Playwright live-stack verification (MANUAL, not yet run).**
|
||
3. **Minor backlog (non-blocking):** explicit id-traversal guard in `delete_session`
|
||
(today gated by `load_session`); `close_session` could reuse `_save_touched` (DRY);
|
||
delete-via-kebab integration test skipped (base-ui Menu portal not drivable in jsdom —
|
||
the dialog itself is unit-tested); a couple of test-file lint nits.
|
||
4. **DONE — F1 hang fix live-verified 2026-06-30** (see top section). The live verification
|
||
also surfaced + fixed the reviewer_select `choices` mismatch.
|
||
5. **Resolved: settings use `zai/glm-5.2`/medium** (not deepseek-flash); GLM 5.2 drives F1
|
||
fine, just slowly (~3-4 min, ~50+ reads). To tell a truly stalled Pi from a merely-slow one,
|
||
check its sockets/children. Memory: `thothii-glm52-f1-slow-not-stuck.md`.
|
||
6. **DONE — Pi migrated to `@earendil-works/pi-coding-agent@0.80.3` (2026-07-04).** The old
|
||
scope `@mariozechner/pi-coding-agent` is frozen at 0.73.1; every release ≥0.74 lives under
|
||
the new scope `@earendil-works` (latest 0.80.3). The live `pi` (`~/.local/bin/pi`) was
|
||
repointed to 0.80.3. Validated by an API/RPC-surface diff (0.73.1→0.80.3: `ExtensionUIContext`
|
||
byte-identical, `rpc-types` additive-only, `createDialogPromise` + provider-registration API
|
||
unchanged) **plus** a live `model-matrix` smoke (GLM 5.2, `new`+`resume` both `CHAINED`,
|
||
full RPC event vocabulary incl. `extension_ui_request` intact). Notable: 0.80.3 adds
|
||
`ctx.mode: "tui"|"rpc"|"json"|"print"` (a proper mode discriminator; the earlier `0.79.4`
|
||
references above are superseded). Rollback: the old package is still on disk — repoint the
|
||
symlink to `@mariozechner/.../dist/cli.js`. Memory: `thothii-pi-earendil-migration.md`.
|
||
|
||
## Where design history lives
|
||
- Specs: `docs/superpowers/specs/` · Plans: `docs/superpowers/plans/`
|
||
- SDD execution ledger (gitignored scratch): `.superpowers/sdd/progress.md`
|
||
- Auto-memory index: `~/.claude/projects/-Users-mp-projects-ThothII/memory/MEMORY.md`
|
||
(Pi RPC event vocabulary, ui.input id correlation, Omics Portal/GSD design system,
|
||
resume cold-start stall, GLM 5.2 F1 slow≠stuck).
|
||
|
||
## Git
|
||
`main` @ `2410f01`, **pushed to `origin`** (github.com/mptyl/ThothII); working tree clean, no stashes.
|
||
Latest arc (2026-07-07): review-gates-v2 ff-merged — `9b4f6b9` (WS1 harness) · `3ad93cd` (WS2
|
||
gate+SKILL) · `1c97289` (WS3 viewers) · `4042d0b`+`b089482` (WS4 replay) · `b59b57c` (review fix
|
||
wave) · `2410f01` (agent_end spinner fix). Branch `feat/review-gates-v2` still exists (local +
|
||
origin), fully merged. Before that (2026-07-05/06): F4 column curation (PR #1) + look&feel v2 —
|
||
`0d9e035` · `0a63ba9` · `d942635` · `9fe1c93` · `e9b2934` · `9403147` · `7491e8c`.
|
||
Older history (workflow hardening 2026-07-01, UI redesign 2026-06-30): see the sections above;
|
||
stale branches were pruned on 2026-06-30 (SHAs recoverable via reflog).
|