ThothII/docs/superpowers/plans/2026-08-13-p5-curated-fk-annotations-in-git.md

# P5 — Curated FK annotations in Git — Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to apply this plan task-by-task.

**Goal:** Make the curated FK annotation file the canonical, revision-pinned human review input. The
registry validates `<workspace-id>/schema/annotations.yaml` as a regular Git blob at the same commit
as the descriptor, synchronizes it to an immutable revision-qualified runtime root on activation, and
replaces the P2 host-file FK review with an explicit operator command
`workspace schema accept --run <id> --yes`. A pinned historical runtime keeps reading its own
revision's annotations; a newer active revision writes a different directory.

**Source of truth:** PRD D5 (`docs/prd/2026-08-09-workspace-preprocessing-prd.md`) and design §7
(`docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md`).

**Architecture:** The backend registry and the compiled operator entrypoint share the sync logic. Git
reads use fixed plumbing (`rev-parse`, `cat-file -t`, `show`) at an exact 40-hex commit — never a
mobile checkout and never author files. The harness keeps owning the annotation *parser* (Pydantic
`Annotations`) and the physical-schema orphan check.

**Tech Stack:** TypeScript (backend registry/preprocessing/runtime rendering), Go (`thothctl`),
Python (`tht schema`), YAML. TDD throughout.

---

## Current-state findings recorded by this plan

- P3 implemented revision-scoped Qdrant schema/Evidence records and a binding-keyed DWH cache
  (`.tht-dwh` at `paths.artifacts.parent`), but it did **not** repurpose `paths.artifacts`/`indexes`
  into a revision root (they remain workspace-global under `/data/sessions/<id>/`).
- The harness resolves curated annotations at `paths.artifacts/mschema/annotations.yaml` and already
  parses them with `Annotations.from_yaml`; `tht schema check` performs the physical orphan check.
- P2 already records FK candidates and an `FkReviewRecord`
  (`{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision }`) and writes host-file reviews
  from `schema check --annotations --reviewed-candidates`.

## Explicit decisions frozen by this plan

1. **Canonical path** is fixed `<workspace-id>/schema/annotations.yaml` (not descriptor-configurable).
   Absence is compatible and yields an empty canonical annotation set plus a warning. Symlinks,
   submodules/trees at the file path, cross-namespace paths, oversized, non-UTF-8, and malformed
   annotations are rejected before activation.
2. **Revision-qualified annotations root** is rendered as a new explicit path
   `paths.annotations_root = /data/sessions/<id>/revisions/<commit>/artifacts`; the immutable synced
   file is `<annotations_root>/mschema/annotations.yaml`. `paths.artifacts`/`indexes`/`memory`/
   `sessions` remain exactly as accepted by P3 because the binding-keyed DWH cache lives at
   `artifacts.parent` and must stay shared across content-only revisions. This is a deliberate,
   surgical refinement of design §7's literal "artifacts and indexes select the revision root": the
   revision-pinned *annotations* requirement is satisfied without destabilizing the accepted P3 cache
   contract. The harness resolves annotations from `paths.annotations_root` when present and falls
   back to the legacy `artifacts/mschema/annotations.yaml` for unmigrated workspaces.
3. **Bounds:** the annotation blob is ≤ 16 MiB, UTF-8, and structurally parsed (Pydantic `Annotations`)
   before synchronization; the full physical orphan check still runs at review time.
4. **Sync trigger:** on registry activation (pull/validate) and before session admission or
   preprocessing, each active revision's annotation blob is read with fixed Git argv, validated, and
   atomically written no-follow to its revision root with restrictive mode, alongside an ownership
   manifest `{ workspace, commit, blobId, contentDigest, destination }`. Re-sync is idempotent and
   re-verifies the manifest.
5. **Human review primitive is `workspace schema accept --run <id> --yes`.** After commit/push/pull,
   the operator reviews the current Git blob against the recorded candidate, then runs the accept
   command. It parses the current blob, validates it against the physical schema via the harness
   parser, and records `{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision }` plus the
   current Git blob id and the new revision. `--yes` is required. An empty file or `schema check`
   alone is **not** evidence of human review.
6. **Continuation gate:** `preprocess run` FK review now requires an accepted review whose
   `annotationsDigest` equals the *current* revision's synced blob digest and a compatible reusable
   DWH binding; otherwise the run starts a new review. The P2 host-file review path
   (`schema check --annotations --reviewed-candidates` writing an `FkReviewRecord`) is superseded:
   `schema check` remains available as read-only validation but no longer records a review.
7. **Error/output:** reuse `annotation_invalid`, `manual_review_required`, and
   `preprocessing_resume_mismatch`; JSON results gain the accepted `blobId`/`annotationsDigest`
   artifact identities. No new public error code is introduced unless a gap is proven by a test.
8. **No push/curation:** the preprocessing CLI never stages, commits, or pushes curated content.
   Curators work in an ordinary author clone.

## Completion contract

The phase is complete when:

1. A workspace whose Git tree contains a valid `<id>/schema/annotations.yaml` blob activates and
   syncs it to exactly `/data/sessions/<id>/revisions/<commit>/artifacts/mschema/annotations.yaml`
   with a verified ownership manifest; a workspace without the file activates with a warning and an
   empty canonical set.
2. Symlink/tree-at-path, cross-namespace, oversized (>16 MiB), non-UTF-8, and malformed annotation
   objects are refused without mutating the snapshot or runtime roots.
3. The harness resolves annotations from `paths.annotations_root` (legacy fallback preserved); a
   session pinned to an older revision reads that revision's synced annotations, and a newer active
   revision writes/reads a different directory.
4. `thothctl ... workspace schema accept --run <id> --yes` records the accepted candidate/current-blob
   digests and the new revision; `--yes` missing, an unknown run, an empty file, a malformed blob, or
   a blob not matching the recorded candidate fails closed without recording a review.
5. `preprocess run` continuation succeeds only with the exact accepted blob digest and compatible DWH
   binding; the superseded host-file `schema check` path no longer records a review.
6. `docs/contracts/workspace-preprocessing-cli.md` documents `schema accept` and the annotations
   lifecycle; the P5 manual walkthrough section is runnable; PROJECT_STATE.md records the result.
7. The clean-state automated process goal passes 1/1 (no retry), and backend/Go/harness focused
   suites plus the existing P2–P4 gates do not regress.

---

### Task 1: Registry reads and validates the annotation blob at the exact commit

**Files:** modify `backend/src/workspaces/git-repository.ts`, `backend/src/workspaces/registry.ts`; add
tests `backend/test/registry-annotations.test.ts`.

1. Failing tests: `gitObjectType`-style read of `<id>/schema/annotations.yaml` at an exact commit
   returns `blob` or absent; a `tree`/`submodule`/other type is refused; the blob id (`rev-parse`) and
   bytes (`show`) match; UTF-8 and 16 MiB bounds are enforced; path grammar rejects
   `workspace-docs/...` and cross-namespace paths.
2. Implement `GitWorkspaceRepository.annotationsObject(revision, id)` returning
   `{ blobId, type, contents } | undefined` with fixed Git argv and bounded sanitized errors.
3. In `WorkspaceRegistry.activate`, validate every active revision's annotation object; a present-but-
   invalid object fails activation closed (`workspace_invalid`), absence is a safe warning.
4. Commit: `feat: read and validate curated FK annotations at the pinned commit (P5)`.

### Task 2: Atomic revision-qualified annotations sync + ownership manifest

**Files:** add `backend/src/workspaces/annotations-sync.ts`; wire into activation and
`renderActiveWorkspaceRuntime`; tests `backend/test/annotations-sync.test.ts`.

1. Failing tests: sync writes `<dataRoot>/sessions/<id>/revisions/<commit>/artifacts/mschema/
   annotations.yaml` (mode restrictive, no-follow, exclusive staging + atomic rename + fsync) and an
   adjacent ownership manifest `{ workspace, commit, blobId, contentDigest, destination }`; re-sync is
   idempotent and re-verifies the manifest; a tampered destination or wrong manifest fails closed;
   a different revision writes a different directory.
2. Implement the sync (shared by registry activation and the operator/session runtime render).
3. Commit: `feat: atomic revision-qualified annotations sync with ownership manifest (P5)`.

### Task 3: Render `paths.annotations_root` and make the harness resolve it

**Files:** modify `backend/src/workspaces/runtime-config-lease.ts`, `harness/tht/config.py`,
`harness/tht/cli/schema_cmd.py`; tests both layers.

1. Failing tests: rendered config includes `paths.annotations_root =
   /data/sessions/<id>/revisions/<commit>/artifacts` while `paths.artifacts`/`indexes`/`memory`/
   `sessions` stay unchanged; `tht schema` `annotations_path` prefers `paths.annotations_root` and
   falls back to the legacy `artifacts/mschema/annotations.yaml` when absent; a missing annotations
   file yields an empty canonical set (not a crash).
2. Implement the render field and harness resolution with the legacy fallback.
3. Commit: `feat: revision-qualified annotations root for pinned runtimes (P5)`.

### Task 4: Operator `workspace schema accept --run <id> --yes`

**Files:** modify `backend/src/workspaces/preprocessing-service.ts`,
`backend/src/workspace-maintenance.ts`, `tools/thothctl/internal/workspaceops/operations.go`,
`tools/thothctl/cmd/thothctl/main.go`; tests `workspace-preprocessing-service.test.ts` and
`operations_test.go`.

1. Failing tests: the accept command reads the current synced Git blob, stages it, validates it with
   the harness parser (structural + orphan check against the recorded candidate), and records
   `{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision }` plus `blobId`; missing
   `--yes`, unknown run, empty/malformed blob, and non-matching candidate fail closed with no review;
   the recorded review is keyed by the run id.
2. Implement `WorkspacePreprocessingService.acceptSchema`, `workspace-maintenance` dispatch
   (`schema-accept`), and the `thothctl` grammar/validation/execute path.
3. Commit: `feat: operator schema accept command for curated FK review (P5)`.

### Task 5: Continuation gate on the accepted blob; supersede host-file review

**Files:** modify `backend/src/workspaces/preprocessing-service.ts` (+ tests).

1. Failing tests: `preprocess run` FK review requires an accepted review whose `annotationsDigest`
   equals the current revision's synced blob digest and a compatible DWH binding; a digest mismatch
   starts a new review (`manual_review_required`); the host-file `schema check --annotations
   --reviewed-candidates` path validates but does not record a review.
2. Implement the gate and the supersession.
3. Commit: `feat: gate FK review on the accepted revision blob (P5)`.

### Task 6: Contract, manual walkthrough, and clean-state acceptance

**Files:** modify `docs/contracts/workspace-preprocessing-cli.md`,
`docs/testing/p2-p6-manual-verification.md` (P5 section), `PROJECT_STATE.md`; add
`scripts/p5-acceptance.sh`, `scripts/test-p5-acceptance.sh`, `backend/scripts/p5-acceptance.mjs`,
`backend/scripts/p5-acceptance.test.mjs` (pattern: P4 acceptance, owned root
`.artifacts/p5-integration/p5-<run-id>/`).

1. Update the CLI contract (new command, annotations lifecycle, exit codes, JSON fields).
2. Implement the clean-state scenario: fixture P1.1 registry + curated annotations + REST DWH +
   pre-provisioned Qdrant; run `thothctl` product commands; prove activation sync + ownership
   manifest, revision isolation, accept happy path, `--yes`/empty/malformed/mismatch negatives, the
   continuation gate, no push of curated content, secret scan, exact cleanup.
3. Finalize the P5 manual walkthrough section and record the phase in PROJECT_STATE.md.
4. Commit: `feat: P5 curated FK annotations in Git (acceptance + docs)`.

---

## Owner checkpoint

After Task 6 the implementation stops for recap. The owner records the P5 manual acceptance
(automated PASS is never recorded as manual PASS), then authorizes P6.

## Non-goals

- No GUI/backend preprocessing endpoint, no push/stage/commit of curated content.
- No P6 filesystem Evidence materialization (the `evidence_materialization_required` stop remains).
- No real PSD migration, no SSH runtime transport, no policy-driven GC.
- No change to the accepted P1/P1.1/P2/P3/P4 contracts or retained evidence beyond the documented
  P5 supersession of the host-file FK review.