ThothII/docs/superpowers/plans/2026-08-13-p5-curated-fk-annotations-in-git.md

P5 — Curated FK annotations in Git — Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:executing-plans to apply this plan task-by-task.

Goal: Make the curated FK annotation file the canonical, revision-pinned human review input. The registry validates <workspace-id>/schema/annotations.yaml as a regular Git blob at the same commit as the descriptor, synchronizes it to an immutable revision-qualified runtime root on activation, and replaces the P2 host-file FK review with an explicit operator command workspace schema accept --run <id> --yes. A pinned historical runtime keeps reading its own revision's annotations; a newer active revision writes a different directory.

Source of truth: PRD D5 (docs/prd/2026-08-09-workspace-preprocessing-prd.md) and design §7 (docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md).

Architecture: The backend registry and the compiled operator entrypoint share the sync logic. Git reads use fixed plumbing (rev-parse, cat-file -t, show) at an exact 40-hex commit — never a mobile checkout and never author files. The harness keeps owning the annotation parser (Pydantic Annotations) and the physical-schema orphan check.

Tech Stack: TypeScript (backend registry/preprocessing/runtime rendering), Go (thothctl), Python (tht schema), YAML. TDD throughout.

---

Current-state findings recorded by this plan

• P3 implemented revision-scoped Qdrant schema/Evidence records and a binding-keyed DWH cache (.tht-dwh at paths.artifacts.parent), but it did not repurpose paths.artifacts/indexes into a revision root (they remain workspace-global under /data/sessions/<id>/).
• The harness resolves curated annotations at paths.artifacts/mschema/annotations.yaml and already parses them with Annotations.from_yaml; tht schema check performs the physical orphan check.
• P2 already records FK candidates and an FkReviewRecord ({ reviewedCandidatesDigest, annotationsDigest, workspaceRevision }) and writes host-file reviews from schema check --annotations --reviewed-candidates.

Explicit decisions frozen by this plan

1. Canonical path is fixed <workspace-id>/schema/annotations.yaml (not descriptor-configurable). Absence is compatible and yields an empty canonical annotation set plus a warning. Symlinks, submodules/trees at the file path, cross-namespace paths, oversized, non-UTF-8, and malformed annotations are rejected before activation.
2. Revision-qualified annotations root is rendered as a new explicit path paths.annotations_root = /data/sessions/<id>/revisions/<commit>/artifacts; the immutable synced file is <annotations_root>/mschema/annotations.yaml. paths.artifacts/indexes/memory/ sessions remain exactly as accepted by P3 because the binding-keyed DWH cache lives at artifacts.parent and must stay shared across content-only revisions. This is a deliberate, surgical refinement of design §7's literal "artifacts and indexes select the revision root": the revision-pinned annotations requirement is satisfied without destabilizing the accepted P3 cache contract. The harness resolves annotations from paths.annotations_root when present and falls back to the legacy artifacts/mschema/annotations.yaml for unmigrated workspaces.
3. Bounds: the annotation blob is ≤ 16 MiB, UTF-8, and structurally parsed (Pydantic Annotations) before synchronization; the full physical orphan check still runs at review time.
4. Sync trigger: on registry activation (pull/validate) and before session admission or preprocessing, each active revision's annotation blob is read with fixed Git argv, validated, and atomically written no-follow to its revision root with restrictive mode, alongside an ownership manifest { workspace, commit, blobId, contentDigest, destination }. Re-sync is idempotent and re-verifies the manifest.
5. Human review primitive is workspace schema accept --run <id> --yes. After commit/push/pull, the operator reviews the current Git blob against the recorded candidate, then runs the accept command. It parses the current blob, validates it against the physical schema via the harness parser, and records { reviewedCandidatesDigest, annotationsDigest, workspaceRevision } plus the current Git blob id and the new revision. --yes is required. An empty file or schema check alone is not evidence of human review.
6. Continuation gate: preprocess run FK review now requires an accepted review whose annotationsDigest equals the current revision's synced blob digest and a compatible reusable DWH binding; otherwise the run starts a new review. The P2 host-file review path (schema check --annotations --reviewed-candidates writing an FkReviewRecord) is superseded: schema check remains available as read-only validation but no longer records a review.
7. Error/output: reuse annotation_invalid, manual_review_required, and preprocessing_resume_mismatch; JSON results gain the accepted blobId/annotationsDigest artifact identities. No new public error code is introduced unless a gap is proven by a test.
8. No push/curation: the preprocessing CLI never stages, commits, or pushes curated content. Curators work in an ordinary author clone.

Completion contract

The phase is complete when:

1. A workspace whose Git tree contains a valid <id>/schema/annotations.yaml blob activates and syncs it to exactly /data/sessions/<id>/revisions/<commit>/artifacts/mschema/annotations.yaml with a verified ownership manifest; a workspace without the file activates with a warning and an empty canonical set.
2. Symlink/tree-at-path, cross-namespace, oversized (>16 MiB), non-UTF-8, and malformed annotation objects are refused without mutating the snapshot or runtime roots.
3. The harness resolves annotations from paths.annotations_root (legacy fallback preserved); a session pinned to an older revision reads that revision's synced annotations, and a newer active revision writes/reads a different directory.
4. thothctl ... workspace schema accept --run <id> --yes records the accepted candidate/current-blob digests and the new revision; --yes missing, an unknown run, an empty file, a malformed blob, or a blob not matching the recorded candidate fails closed without recording a review.
5. preprocess run continuation succeeds only with the exact accepted blob digest and compatible DWH binding; the superseded host-file schema check path no longer records a review.
6. docs/contracts/workspace-preprocessing-cli.md documents schema accept and the annotations lifecycle; the P5 manual walkthrough section is runnable; PROJECT_STATE.md records the result.
7. The clean-state automated process goal passes 1/1 (no retry), and backend/Go/harness focused suites plus the existing P2–P4 gates do not regress.

---

Task 1: Registry reads and validates the annotation blob at the exact commit

Files: modify backend/src/workspaces/git-repository.ts, backend/src/workspaces/registry.ts; add tests backend/test/registry-annotations.test.ts.

1. Failing tests: gitObjectType-style read of <id>/schema/annotations.yaml at an exact commit returns blob or absent; a tree/submodule/other type is refused; the blob id (rev-parse) and bytes (show) match; UTF-8 and 16 MiB bounds are enforced; path grammar rejects workspace-docs/... and cross-namespace paths.
2. Implement GitWorkspaceRepository.annotationsObject(revision, id) returning { blobId, type, contents } | undefined with fixed Git argv and bounded sanitized errors.
3. In WorkspaceRegistry.activate, validate every active revision's annotation object; a present-but- invalid object fails activation closed (workspace_invalid), absence is a safe warning.
4. Commit: feat: read and validate curated FK annotations at the pinned commit (P5).

Task 2: Atomic revision-qualified annotations sync + ownership manifest

Files: add backend/src/workspaces/annotations-sync.ts; wire into activation and renderActiveWorkspaceRuntime; tests backend/test/annotations-sync.test.ts.

1. Failing tests: sync writes <dataRoot>/sessions/<id>/revisions/<commit>/artifacts/mschema/ annotations.yaml (mode restrictive, no-follow, exclusive staging + atomic rename + fsync) and an adjacent ownership manifest { workspace, commit, blobId, contentDigest, destination }; re-sync is idempotent and re-verifies the manifest; a tampered destination or wrong manifest fails closed; a different revision writes a different directory.
2. Implement the sync (shared by registry activation and the operator/session runtime render).
3. Commit: feat: atomic revision-qualified annotations sync with ownership manifest (P5).

Task 3: Render paths.annotations_root and make the harness resolve it

Files: modify backend/src/workspaces/runtime-config-lease.ts, harness/tht/config.py, harness/tht/cli/schema_cmd.py; tests both layers.

1. Failing tests: rendered config includes paths.annotations_root = /data/sessions/<id>/revisions/<commit>/artifacts while paths.artifacts/indexes/memory/ sessions stay unchanged; tht schema annotations_path prefers paths.annotations_root and falls back to the legacy artifacts/mschema/annotations.yaml when absent; a missing annotations file yields an empty canonical set (not a crash).
2. Implement the render field and harness resolution with the legacy fallback.
3. Commit: feat: revision-qualified annotations root for pinned runtimes (P5).

Task 4: Operator workspace schema accept --run <id> --yes

Files: modify backend/src/workspaces/preprocessing-service.ts, backend/src/workspace-maintenance.ts, tools/thothctl/internal/workspaceops/operations.go, tools/thothctl/cmd/thothctl/main.go; tests workspace-preprocessing-service.test.ts and operations_test.go.

1. Failing tests: the accept command reads the current synced Git blob, stages it, validates it with the harness parser (structural + orphan check against the recorded candidate), and records { reviewedCandidatesDigest, annotationsDigest, workspaceRevision } plus blobId; missing --yes, unknown run, empty/malformed blob, and non-matching candidate fail closed with no review; the recorded review is keyed by the run id.
2. Implement WorkspacePreprocessingService.acceptSchema, workspace-maintenance dispatch (schema-accept), and the thothctl grammar/validation/execute path.
3. Commit: feat: operator schema accept command for curated FK review (P5).

Task 5: Continuation gate on the accepted blob; supersede host-file review

Files: modify backend/src/workspaces/preprocessing-service.ts (+ tests).

1. Failing tests: preprocess run FK review requires an accepted review whose annotationsDigest equals the current revision's synced blob digest and a compatible DWH binding; a digest mismatch starts a new review (manual_review_required); the host-file schema check --annotations --reviewed-candidates path validates but does not record a review.
2. Implement the gate and the supersession.
3. Commit: feat: gate FK review on the accepted revision blob (P5).

Task 6: Contract, manual walkthrough, and clean-state acceptance

Files: modify docs/contracts/workspace-preprocessing-cli.md, docs/testing/p2-p6-manual-verification.md (P5 section), PROJECT_STATE.md; add scripts/p5-acceptance.sh, scripts/test-p5-acceptance.sh, backend/scripts/p5-acceptance.mjs, backend/scripts/p5-acceptance.test.mjs (pattern: P4 acceptance, owned root .artifacts/p5-integration/p5-<run-id>/).

1. Update the CLI contract (new command, annotations lifecycle, exit codes, JSON fields).
2. Implement the clean-state scenario: fixture P1.1 registry + curated annotations + REST DWH + pre-provisioned Qdrant; run thothctl product commands; prove activation sync + ownership manifest, revision isolation, accept happy path, --yes/empty/malformed/mismatch negatives, the continuation gate, no push of curated content, secret scan, exact cleanup.
3. Finalize the P5 manual walkthrough section and record the phase in PROJECT_STATE.md.
4. Commit: feat: P5 curated FK annotations in Git (acceptance + docs).

---

Owner checkpoint

After Task 6 the implementation stops for recap. The owner records the P5 manual acceptance (automated PASS is never recorded as manual PASS), then authorizes P6.

Non-goals

• No GUI/backend preprocessing endpoint, no push/stage/commit of curated content.
• No P6 filesystem Evidence materialization (the evidence_materialization_required stop remains).
• No real PSD migration, no SSH runtime transport, no policy-driven GC.
• No change to the accepted P1/P1.1/P2/P3/P4 contracts or retained evidence beyond the documented P5 supersession of the host-file FK review.