docs: plan P5 curated FK annotations in Git
This commit is contained in:
@@ -0,0 +1,197 @@
|
||||
# P5 — Curated FK annotations in Git — Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to apply this plan task-by-task.
|
||||
|
||||
**Goal:** Make the curated FK annotation file the canonical, revision-pinned human review input. The
|
||||
registry validates `<workspace-id>/schema/annotations.yaml` as a regular Git blob at the same commit
|
||||
as the descriptor, synchronizes it to an immutable revision-qualified runtime root on activation, and
|
||||
replaces the P2 host-file FK review with an explicit operator command
|
||||
`workspace schema accept --run <id> --yes`. A pinned historical runtime keeps reading its own
|
||||
revision's annotations; a newer active revision writes a different directory.
|
||||
|
||||
**Source of truth:** PRD D5 (`docs/prd/2026-08-09-workspace-preprocessing-prd.md`) and design §7
|
||||
(`docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md`).
|
||||
|
||||
**Architecture:** The backend registry and the compiled operator entrypoint share the sync logic. Git
|
||||
reads use fixed plumbing (`rev-parse`, `cat-file -t`, `show`) at an exact 40-hex commit — never a
|
||||
mobile checkout and never author files. The harness keeps owning the annotation *parser* (Pydantic
|
||||
`Annotations`) and the physical-schema orphan check.
|
||||
|
||||
**Tech Stack:** TypeScript (backend registry/preprocessing/runtime rendering), Go (`thothctl`),
|
||||
Python (`tht schema`), YAML. TDD throughout.
|
||||
|
||||
---
|
||||
|
||||
## Current-state findings recorded by this plan
|
||||
|
||||
- P3 implemented revision-scoped Qdrant schema/Evidence records and a binding-keyed DWH cache
|
||||
(`.tht-dwh` at `paths.artifacts.parent`), but it did **not** repurpose `paths.artifacts`/`indexes`
|
||||
into a revision root (they remain workspace-global under `/data/sessions/<id>/`).
|
||||
- The harness resolves curated annotations at `paths.artifacts/mschema/annotations.yaml` and already
|
||||
parses them with `Annotations.from_yaml`; `tht schema check` performs the physical orphan check.
|
||||
- P2 already records FK candidates and an `FkReviewRecord`
|
||||
(`{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision }`) and writes host-file reviews
|
||||
from `schema check --annotations --reviewed-candidates`.
|
||||
|
||||
## Explicit decisions frozen by this plan
|
||||
|
||||
1. **Canonical path** is fixed `<workspace-id>/schema/annotations.yaml` (not descriptor-configurable).
|
||||
Absence is compatible and yields an empty canonical annotation set plus a warning. Symlinks,
|
||||
submodules/trees at the file path, cross-namespace paths, oversized, non-UTF-8, and malformed
|
||||
annotations are rejected before activation.
|
||||
2. **Revision-qualified annotations root** is rendered as a new explicit path
|
||||
`paths.annotations_root = /data/sessions/<id>/revisions/<commit>/artifacts`; the immutable synced
|
||||
file is `<annotations_root>/mschema/annotations.yaml`. `paths.artifacts`/`indexes`/`memory`/
|
||||
`sessions` remain exactly as accepted by P3 because the binding-keyed DWH cache lives at
|
||||
`artifacts.parent` and must stay shared across content-only revisions. This is a deliberate,
|
||||
surgical refinement of design §7's literal "artifacts and indexes select the revision root": the
|
||||
revision-pinned *annotations* requirement is satisfied without destabilizing the accepted P3 cache
|
||||
contract. The harness resolves annotations from `paths.annotations_root` when present and falls
|
||||
back to the legacy `artifacts/mschema/annotations.yaml` for unmigrated workspaces.
|
||||
3. **Bounds:** the annotation blob is ≤ 16 MiB, UTF-8, and structurally parsed (Pydantic `Annotations`)
|
||||
before synchronization; the full physical orphan check still runs at review time.
|
||||
4. **Sync trigger:** on registry activation (pull/validate) and before session admission or
|
||||
preprocessing, each active revision's annotation blob is read with fixed Git argv, validated, and
|
||||
atomically written no-follow to its revision root with restrictive mode, alongside an ownership
|
||||
manifest `{ workspace, commit, blobId, contentDigest, destination }`. Re-sync is idempotent and
|
||||
re-verifies the manifest.
|
||||
5. **Human review primitive is `workspace schema accept --run <id> --yes`.** After commit/push/pull,
|
||||
the operator reviews the current Git blob against the recorded candidate, then runs the accept
|
||||
command. It parses the current blob, validates it against the physical schema via the harness
|
||||
parser, and records `{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision }` plus the
|
||||
current Git blob id and the new revision. `--yes` is required. An empty file or `schema check`
|
||||
alone is **not** evidence of human review.
|
||||
6. **Continuation gate:** `preprocess run` FK review now requires an accepted review whose
|
||||
`annotationsDigest` equals the *current* revision's synced blob digest and a compatible reusable
|
||||
DWH binding; otherwise the run starts a new review. The P2 host-file review path
|
||||
(`schema check --annotations --reviewed-candidates` writing an `FkReviewRecord`) is superseded:
|
||||
`schema check` remains available as read-only validation but no longer records a review.
|
||||
7. **Error/output:** reuse `annotation_invalid`, `manual_review_required`, and
|
||||
`preprocessing_resume_mismatch`; JSON results gain the accepted `blobId`/`annotationsDigest`
|
||||
artifact identities. No new public error code is introduced unless a gap is proven by a test.
|
||||
8. **No push/curation:** the preprocessing CLI never stages, commits, or pushes curated content.
|
||||
Curators work in an ordinary author clone.
|
||||
|
||||
## Completion contract
|
||||
|
||||
The phase is complete when:
|
||||
|
||||
1. A workspace whose Git tree contains a valid `<id>/schema/annotations.yaml` blob activates and
|
||||
syncs it to exactly `/data/sessions/<id>/revisions/<commit>/artifacts/mschema/annotations.yaml`
|
||||
with a verified ownership manifest; a workspace without the file activates with a warning and an
|
||||
empty canonical set.
|
||||
2. Symlink/tree-at-path, cross-namespace, oversized (>16 MiB), non-UTF-8, and malformed annotation
|
||||
objects are refused without mutating the snapshot or runtime roots.
|
||||
3. The harness resolves annotations from `paths.annotations_root` (legacy fallback preserved); a
|
||||
session pinned to an older revision reads that revision's synced annotations, and a newer active
|
||||
revision writes/reads a different directory.
|
||||
4. `thothctl ... workspace schema accept --run <id> --yes` records the accepted candidate/current-blob
|
||||
digests and the new revision; `--yes` missing, an unknown run, an empty file, a malformed blob, or
|
||||
a blob not matching the recorded candidate fails closed without recording a review.
|
||||
5. `preprocess run` continuation succeeds only with the exact accepted blob digest and compatible DWH
|
||||
binding; the superseded host-file `schema check` path no longer records a review.
|
||||
6. `docs/contracts/workspace-preprocessing-cli.md` documents `schema accept` and the annotations
|
||||
lifecycle; the P5 manual walkthrough section is runnable; PROJECT_STATE.md records the result.
|
||||
7. The clean-state automated process goal passes 1/1 (no retry), and backend/Go/harness focused
|
||||
suites plus the existing P2–P4 gates do not regress.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Registry reads and validates the annotation blob at the exact commit
|
||||
|
||||
**Files:** modify `backend/src/workspaces/git-repository.ts`, `backend/src/workspaces/registry.ts`; add
|
||||
tests `backend/test/registry-annotations.test.ts`.
|
||||
|
||||
1. Failing tests: `gitObjectType`-style read of `<id>/schema/annotations.yaml` at an exact commit
|
||||
returns `blob` or absent; a `tree`/`submodule`/other type is refused; the blob id (`rev-parse`) and
|
||||
bytes (`show`) match; UTF-8 and 16 MiB bounds are enforced; path grammar rejects
|
||||
`workspace-docs/...` and cross-namespace paths.
|
||||
2. Implement `GitWorkspaceRepository.annotationsObject(revision, id)` returning
|
||||
`{ blobId, type, contents } | undefined` with fixed Git argv and bounded sanitized errors.
|
||||
3. In `WorkspaceRegistry.activate`, validate every active revision's annotation object; a present-but-
|
||||
invalid object fails activation closed (`workspace_invalid`), absence is a safe warning.
|
||||
4. Commit: `feat: read and validate curated FK annotations at the pinned commit (P5)`.
|
||||
|
||||
### Task 2: Atomic revision-qualified annotations sync + ownership manifest
|
||||
|
||||
**Files:** add `backend/src/workspaces/annotations-sync.ts`; wire into activation and
|
||||
`renderActiveWorkspaceRuntime`; tests `backend/test/annotations-sync.test.ts`.
|
||||
|
||||
1. Failing tests: sync writes `<dataRoot>/sessions/<id>/revisions/<commit>/artifacts/mschema/
|
||||
annotations.yaml` (mode restrictive, no-follow, exclusive staging + atomic rename + fsync) and an
|
||||
adjacent ownership manifest `{ workspace, commit, blobId, contentDigest, destination }`; re-sync is
|
||||
idempotent and re-verifies the manifest; a tampered destination or wrong manifest fails closed;
|
||||
a different revision writes a different directory.
|
||||
2. Implement the sync (shared by registry activation and the operator/session runtime render).
|
||||
3. Commit: `feat: atomic revision-qualified annotations sync with ownership manifest (P5)`.
|
||||
|
||||
### Task 3: Render `paths.annotations_root` and make the harness resolve it
|
||||
|
||||
**Files:** modify `backend/src/workspaces/runtime-config-lease.ts`, `harness/tht/config.py`,
|
||||
`harness/tht/cli/schema_cmd.py`; tests both layers.
|
||||
|
||||
1. Failing tests: rendered config includes `paths.annotations_root =
|
||||
/data/sessions/<id>/revisions/<commit>/artifacts` while `paths.artifacts`/`indexes`/`memory`/
|
||||
`sessions` stay unchanged; `tht schema` `annotations_path` prefers `paths.annotations_root` and
|
||||
falls back to the legacy `artifacts/mschema/annotations.yaml` when absent; a missing annotations
|
||||
file yields an empty canonical set (not a crash).
|
||||
2. Implement the render field and harness resolution with the legacy fallback.
|
||||
3. Commit: `feat: revision-qualified annotations root for pinned runtimes (P5)`.
|
||||
|
||||
### Task 4: Operator `workspace schema accept --run <id> --yes`
|
||||
|
||||
**Files:** modify `backend/src/workspaces/preprocessing-service.ts`,
|
||||
`backend/src/workspace-maintenance.ts`, `tools/thothctl/internal/workspaceops/operations.go`,
|
||||
`tools/thothctl/cmd/thothctl/main.go`; tests `workspace-preprocessing-service.test.ts` and
|
||||
`operations_test.go`.
|
||||
|
||||
1. Failing tests: the accept command reads the current synced Git blob, stages it, validates it with
|
||||
the harness parser (structural + orphan check against the recorded candidate), and records
|
||||
`{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision }` plus `blobId`; missing
|
||||
`--yes`, unknown run, empty/malformed blob, and non-matching candidate fail closed with no review;
|
||||
the recorded review is keyed by the run id.
|
||||
2. Implement `WorkspacePreprocessingService.acceptSchema`, `workspace-maintenance` dispatch
|
||||
(`schema-accept`), and the `thothctl` grammar/validation/execute path.
|
||||
3. Commit: `feat: operator schema accept command for curated FK review (P5)`.
|
||||
|
||||
### Task 5: Continuation gate on the accepted blob; supersede host-file review
|
||||
|
||||
**Files:** modify `backend/src/workspaces/preprocessing-service.ts` (+ tests).
|
||||
|
||||
1. Failing tests: `preprocess run` FK review requires an accepted review whose `annotationsDigest`
|
||||
equals the current revision's synced blob digest and a compatible DWH binding; a digest mismatch
|
||||
starts a new review (`manual_review_required`); the host-file `schema check --annotations
|
||||
--reviewed-candidates` path validates but does not record a review.
|
||||
2. Implement the gate and the supersession.
|
||||
3. Commit: `feat: gate FK review on the accepted revision blob (P5)`.
|
||||
|
||||
### Task 6: Contract, manual walkthrough, and clean-state acceptance
|
||||
|
||||
**Files:** modify `docs/contracts/workspace-preprocessing-cli.md`,
|
||||
`docs/testing/p2-p6-manual-verification.md` (P5 section), `PROJECT_STATE.md`; add
|
||||
`scripts/p5-acceptance.sh`, `scripts/test-p5-acceptance.sh`, `backend/scripts/p5-acceptance.mjs`,
|
||||
`backend/scripts/p5-acceptance.test.mjs` (pattern: P4 acceptance, owned root
|
||||
`.artifacts/p5-integration/p5-<run-id>/`).
|
||||
|
||||
1. Update the CLI contract (new command, annotations lifecycle, exit codes, JSON fields).
|
||||
2. Implement the clean-state scenario: fixture P1.1 registry + curated annotations + REST DWH +
|
||||
pre-provisioned Qdrant; run `thothctl` product commands; prove activation sync + ownership
|
||||
manifest, revision isolation, accept happy path, `--yes`/empty/malformed/mismatch negatives, the
|
||||
continuation gate, no push of curated content, secret scan, exact cleanup.
|
||||
3. Finalize the P5 manual walkthrough section and record the phase in PROJECT_STATE.md.
|
||||
4. Commit: `feat: P5 curated FK annotations in Git (acceptance + docs)`.
|
||||
|
||||
---
|
||||
|
||||
## Owner checkpoint
|
||||
|
||||
After Task 6 the implementation stops for recap. The owner records the P5 manual acceptance
|
||||
(automated PASS is never recorded as manual PASS), then authorizes P6.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No GUI/backend preprocessing endpoint, no push/stage/commit of curated content.
|
||||
- No P6 filesystem Evidence materialization (the `evidence_materialization_required` stop remains).
|
||||
- No real PSD migration, no SSH runtime transport, no policy-driven GC.
|
||||
- No change to the accepted P1/P1.1/P2/P3/P4 contracts or retained evidence beyond the documented
|
||||
P5 supersession of the host-file FK review.
|
||||
Reference in New Issue
Block a user