diff --git a/docs/superpowers/plans/2026-08-13-p5-curated-fk-annotations-in-git.md b/docs/superpowers/plans/2026-08-13-p5-curated-fk-annotations-in-git.md new file mode 100644 index 00000000..a4186310 --- /dev/null +++ b/docs/superpowers/plans/2026-08-13-p5-curated-fk-annotations-in-git.md @@ -0,0 +1,197 @@ +# P5 — Curated FK annotations in Git — Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to apply this plan task-by-task. + +**Goal:** Make the curated FK annotation file the canonical, revision-pinned human review input. The +registry validates `/schema/annotations.yaml` as a regular Git blob at the same commit +as the descriptor, synchronizes it to an immutable revision-qualified runtime root on activation, and +replaces the P2 host-file FK review with an explicit operator command +`workspace schema accept --run --yes`. A pinned historical runtime keeps reading its own +revision's annotations; a newer active revision writes a different directory. + +**Source of truth:** PRD D5 (`docs/prd/2026-08-09-workspace-preprocessing-prd.md`) and design §7 +(`docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md`). + +**Architecture:** The backend registry and the compiled operator entrypoint share the sync logic. Git +reads use fixed plumbing (`rev-parse`, `cat-file -t`, `show`) at an exact 40-hex commit — never a +mobile checkout and never author files. The harness keeps owning the annotation *parser* (Pydantic +`Annotations`) and the physical-schema orphan check. + +**Tech Stack:** TypeScript (backend registry/preprocessing/runtime rendering), Go (`thothctl`), +Python (`tht schema`), YAML. TDD throughout. + +--- + +## Current-state findings recorded by this plan + +- P3 implemented revision-scoped Qdrant schema/Evidence records and a binding-keyed DWH cache + (`.tht-dwh` at `paths.artifacts.parent`), but it did **not** repurpose `paths.artifacts`/`indexes` + into a revision root (they remain workspace-global under `/data/sessions//`). +- The harness resolves curated annotations at `paths.artifacts/mschema/annotations.yaml` and already + parses them with `Annotations.from_yaml`; `tht schema check` performs the physical orphan check. +- P2 already records FK candidates and an `FkReviewRecord` + (`{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision }`) and writes host-file reviews + from `schema check --annotations --reviewed-candidates`. + +## Explicit decisions frozen by this plan + +1. **Canonical path** is fixed `/schema/annotations.yaml` (not descriptor-configurable). + Absence is compatible and yields an empty canonical annotation set plus a warning. Symlinks, + submodules/trees at the file path, cross-namespace paths, oversized, non-UTF-8, and malformed + annotations are rejected before activation. +2. **Revision-qualified annotations root** is rendered as a new explicit path + `paths.annotations_root = /data/sessions//revisions//artifacts`; the immutable synced + file is `/mschema/annotations.yaml`. `paths.artifacts`/`indexes`/`memory`/ + `sessions` remain exactly as accepted by P3 because the binding-keyed DWH cache lives at + `artifacts.parent` and must stay shared across content-only revisions. This is a deliberate, + surgical refinement of design §7's literal "artifacts and indexes select the revision root": the + revision-pinned *annotations* requirement is satisfied without destabilizing the accepted P3 cache + contract. The harness resolves annotations from `paths.annotations_root` when present and falls + back to the legacy `artifacts/mschema/annotations.yaml` for unmigrated workspaces. +3. **Bounds:** the annotation blob is ≤ 16 MiB, UTF-8, and structurally parsed (Pydantic `Annotations`) + before synchronization; the full physical orphan check still runs at review time. +4. **Sync trigger:** on registry activation (pull/validate) and before session admission or + preprocessing, each active revision's annotation blob is read with fixed Git argv, validated, and + atomically written no-follow to its revision root with restrictive mode, alongside an ownership + manifest `{ workspace, commit, blobId, contentDigest, destination }`. Re-sync is idempotent and + re-verifies the manifest. +5. **Human review primitive is `workspace schema accept --run --yes`.** After commit/push/pull, + the operator reviews the current Git blob against the recorded candidate, then runs the accept + command. It parses the current blob, validates it against the physical schema via the harness + parser, and records `{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision }` plus the + current Git blob id and the new revision. `--yes` is required. An empty file or `schema check` + alone is **not** evidence of human review. +6. **Continuation gate:** `preprocess run` FK review now requires an accepted review whose + `annotationsDigest` equals the *current* revision's synced blob digest and a compatible reusable + DWH binding; otherwise the run starts a new review. The P2 host-file review path + (`schema check --annotations --reviewed-candidates` writing an `FkReviewRecord`) is superseded: + `schema check` remains available as read-only validation but no longer records a review. +7. **Error/output:** reuse `annotation_invalid`, `manual_review_required`, and + `preprocessing_resume_mismatch`; JSON results gain the accepted `blobId`/`annotationsDigest` + artifact identities. No new public error code is introduced unless a gap is proven by a test. +8. **No push/curation:** the preprocessing CLI never stages, commits, or pushes curated content. + Curators work in an ordinary author clone. + +## Completion contract + +The phase is complete when: + +1. A workspace whose Git tree contains a valid `/schema/annotations.yaml` blob activates and + syncs it to exactly `/data/sessions//revisions//artifacts/mschema/annotations.yaml` + with a verified ownership manifest; a workspace without the file activates with a warning and an + empty canonical set. +2. Symlink/tree-at-path, cross-namespace, oversized (>16 MiB), non-UTF-8, and malformed annotation + objects are refused without mutating the snapshot or runtime roots. +3. The harness resolves annotations from `paths.annotations_root` (legacy fallback preserved); a + session pinned to an older revision reads that revision's synced annotations, and a newer active + revision writes/reads a different directory. +4. `thothctl ... workspace schema accept --run --yes` records the accepted candidate/current-blob + digests and the new revision; `--yes` missing, an unknown run, an empty file, a malformed blob, or + a blob not matching the recorded candidate fails closed without recording a review. +5. `preprocess run` continuation succeeds only with the exact accepted blob digest and compatible DWH + binding; the superseded host-file `schema check` path no longer records a review. +6. `docs/contracts/workspace-preprocessing-cli.md` documents `schema accept` and the annotations + lifecycle; the P5 manual walkthrough section is runnable; PROJECT_STATE.md records the result. +7. The clean-state automated process goal passes 1/1 (no retry), and backend/Go/harness focused + suites plus the existing P2–P4 gates do not regress. + +--- + +### Task 1: Registry reads and validates the annotation blob at the exact commit + +**Files:** modify `backend/src/workspaces/git-repository.ts`, `backend/src/workspaces/registry.ts`; add +tests `backend/test/registry-annotations.test.ts`. + +1. Failing tests: `gitObjectType`-style read of `/schema/annotations.yaml` at an exact commit + returns `blob` or absent; a `tree`/`submodule`/other type is refused; the blob id (`rev-parse`) and + bytes (`show`) match; UTF-8 and 16 MiB bounds are enforced; path grammar rejects + `workspace-docs/...` and cross-namespace paths. +2. Implement `GitWorkspaceRepository.annotationsObject(revision, id)` returning + `{ blobId, type, contents } | undefined` with fixed Git argv and bounded sanitized errors. +3. In `WorkspaceRegistry.activate`, validate every active revision's annotation object; a present-but- + invalid object fails activation closed (`workspace_invalid`), absence is a safe warning. +4. Commit: `feat: read and validate curated FK annotations at the pinned commit (P5)`. + +### Task 2: Atomic revision-qualified annotations sync + ownership manifest + +**Files:** add `backend/src/workspaces/annotations-sync.ts`; wire into activation and +`renderActiveWorkspaceRuntime`; tests `backend/test/annotations-sync.test.ts`. + +1. Failing tests: sync writes `/sessions//revisions//artifacts/mschema/ + annotations.yaml` (mode restrictive, no-follow, exclusive staging + atomic rename + fsync) and an + adjacent ownership manifest `{ workspace, commit, blobId, contentDigest, destination }`; re-sync is + idempotent and re-verifies the manifest; a tampered destination or wrong manifest fails closed; + a different revision writes a different directory. +2. Implement the sync (shared by registry activation and the operator/session runtime render). +3. Commit: `feat: atomic revision-qualified annotations sync with ownership manifest (P5)`. + +### Task 3: Render `paths.annotations_root` and make the harness resolve it + +**Files:** modify `backend/src/workspaces/runtime-config-lease.ts`, `harness/tht/config.py`, +`harness/tht/cli/schema_cmd.py`; tests both layers. + +1. Failing tests: rendered config includes `paths.annotations_root = + /data/sessions//revisions//artifacts` while `paths.artifacts`/`indexes`/`memory`/ + `sessions` stay unchanged; `tht schema` `annotations_path` prefers `paths.annotations_root` and + falls back to the legacy `artifacts/mschema/annotations.yaml` when absent; a missing annotations + file yields an empty canonical set (not a crash). +2. Implement the render field and harness resolution with the legacy fallback. +3. Commit: `feat: revision-qualified annotations root for pinned runtimes (P5)`. + +### Task 4: Operator `workspace schema accept --run --yes` + +**Files:** modify `backend/src/workspaces/preprocessing-service.ts`, +`backend/src/workspace-maintenance.ts`, `tools/thothctl/internal/workspaceops/operations.go`, +`tools/thothctl/cmd/thothctl/main.go`; tests `workspace-preprocessing-service.test.ts` and +`operations_test.go`. + +1. Failing tests: the accept command reads the current synced Git blob, stages it, validates it with + the harness parser (structural + orphan check against the recorded candidate), and records + `{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision }` plus `blobId`; missing + `--yes`, unknown run, empty/malformed blob, and non-matching candidate fail closed with no review; + the recorded review is keyed by the run id. +2. Implement `WorkspacePreprocessingService.acceptSchema`, `workspace-maintenance` dispatch + (`schema-accept`), and the `thothctl` grammar/validation/execute path. +3. Commit: `feat: operator schema accept command for curated FK review (P5)`. + +### Task 5: Continuation gate on the accepted blob; supersede host-file review + +**Files:** modify `backend/src/workspaces/preprocessing-service.ts` (+ tests). + +1. Failing tests: `preprocess run` FK review requires an accepted review whose `annotationsDigest` + equals the current revision's synced blob digest and a compatible DWH binding; a digest mismatch + starts a new review (`manual_review_required`); the host-file `schema check --annotations + --reviewed-candidates` path validates but does not record a review. +2. Implement the gate and the supersession. +3. Commit: `feat: gate FK review on the accepted revision blob (P5)`. + +### Task 6: Contract, manual walkthrough, and clean-state acceptance + +**Files:** modify `docs/contracts/workspace-preprocessing-cli.md`, +`docs/testing/p2-p6-manual-verification.md` (P5 section), `PROJECT_STATE.md`; add +`scripts/p5-acceptance.sh`, `scripts/test-p5-acceptance.sh`, `backend/scripts/p5-acceptance.mjs`, +`backend/scripts/p5-acceptance.test.mjs` (pattern: P4 acceptance, owned root +`.artifacts/p5-integration/p5-/`). + +1. Update the CLI contract (new command, annotations lifecycle, exit codes, JSON fields). +2. Implement the clean-state scenario: fixture P1.1 registry + curated annotations + REST DWH + + pre-provisioned Qdrant; run `thothctl` product commands; prove activation sync + ownership + manifest, revision isolation, accept happy path, `--yes`/empty/malformed/mismatch negatives, the + continuation gate, no push of curated content, secret scan, exact cleanup. +3. Finalize the P5 manual walkthrough section and record the phase in PROJECT_STATE.md. +4. Commit: `feat: P5 curated FK annotations in Git (acceptance + docs)`. + +--- + +## Owner checkpoint + +After Task 6 the implementation stops for recap. The owner records the P5 manual acceptance +(automated PASS is never recorded as manual PASS), then authorizes P6. + +## Non-goals + +- No GUI/backend preprocessing endpoint, no push/stage/commit of curated content. +- No P6 filesystem Evidence materialization (the `evidence_materialization_required` stop remains). +- No real PSD migration, no SSH runtime transport, no policy-driven GC. +- No change to the accepted P1/P1.1/P2/P3/P4 contracts or retained evidence beyond the documented + P5 supersession of the host-file FK review.