diff --git a/docs/superpowers/plans/2026-08-13-p6-commit-addressed-evidence-materialization.md b/docs/superpowers/plans/2026-08-13-p6-commit-addressed-evidence-materialization.md new file mode 100644 index 00000000..5a15963a --- /dev/null +++ b/docs/superpowers/plans/2026-08-13-p6-commit-addressed-evidence-materialization.md @@ -0,0 +1,179 @@ +# P6 — Commit-addressed Evidence materialization — Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to apply this plan task-by-task. + +**Goal:** Materialize the filesystem Evidence tree `/evidence` from the exact pinned Git +commit into an immutable revision content root, verify real containment (no symlink/gitlink/ +traversal/special-file escape), and make P2's filesystem Evidence path operational end-to-end by +removing the temporary `evidence_materialization_required` stop. + +**Source of truth:** PRD D6 (`docs/prd/2026-08-09-workspace-preprocessing-prd.md`) and design §8 +(`docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md`). + +**Architecture:** A new shared TypeScript materializer enumerates the tree with fixed Git plumbing +(`ls-tree -r -z` + `cat-file blob`) and writes regular files no-follow/exclusive beneath a fresh +owned staging directory, hashing every file into a bounded manifest. The registry runs it during +snapshot staging so the materialized root lands atomically inside the already-retained, commit- +addressed snapshot directory; a tampered or mismatched manifest fails closed. The runtime renderer +and the harness filesystem Evidence adapter are already rooted at that directory and need no change. + +**Tech Stack:** TypeScript (registry + materializer), Python (existing filesystem Evidence adapter), +YAML. TDD throughout. + +--- + +## Current-state findings recorded by this plan + +- The registry already validates the filesystem Evidence *root object* is a Git tree at the pinned + commit (`assertTreeAtRevision`) and freezes it as `revisionContentRoot = dirname(snapshotPath)`. +- `renderEvidence` already resolves filesystem Evidence to `join(revisionContentRoot, source.uri)` + (`///evidence`), and the harness `FilesystemEvidenceSource` reads exactly + that directory with no-follow opens and `**/*.md` discovery. +- `WorkspacePreprocessingService.evidencePolicy` currently returns + `evidence_materialization_required` for filesystem sources (the P2 temporary stop). +- `reconcileSnapshotRetention` removes whole commit-addressed `//` directories, + so materialized evidence beneath that directory is automatically retained while pinned and removed + only when the revision becomes unreferenced. +- The registry `activate()` staging already writes immutable `.yaml`/`.env.example`/`.md` + + `snapshot.json` and renames atomically; the comment at `expectedSnapshotFiles` marks P6 as the + owner of workspace-content materialization. + +## Explicit decisions frozen by this plan + +1. **Target layout.** Materialized filesystem Evidence lives at + `/snapshots///evidence/` with a sibling bounded manifest + `/snapshots///evidence.manifest.json`. The manifest records + `{ workspace, commit, tree, entryCount, totalBytes, files: { "": { mode, oid, digest, bytes } } }`. + The sibling manifest is outside the discovery root so the Evidence adapter never ingests it. +2. **Eager, fail-closed materialization at activation.** During `activate()` snapshot staging, every + filesystem-Evidence workspace is materialized before the staging directory is atomically renamed. + A missing Evidence root, an unsafe object, a bound violation, or a write failure aborts activation + (`workspace_invalid`); no partial root is published. An empty Evidence tree is valid (empty root + + zero-entry manifest). +3. **Fixed Git plumbing, no shell, no mobile checkout.** Enumeration is + `git ls-tree -r -z -- /evidence`; blob bytes come from `git cat-file blob ` + (buffer, per-object bound). No archive is extracted and no author files are consulted. +4. **Object safety.** Reject at any depth: symlink (`120000`), gitlink/submodule (`160000`), non- + regular modes other than `100644`/`100755`, non-`blob` type, absolute/`.`/`..`/NUL/newline/ + non-normalized paths, duplicate normalized paths, cross-workspace namespaces, and any object whose + id or bytes change between enumeration and read. +5. **Bounds.** Installation-local non-secret limits with conservative defaults: + `maxEvidenceEntries` (4096 files), `maxEvidenceBytes` (64 MiB total), `maxEvidenceFileBytes` + (8 MiB per file), `maxEvidencePathBytes` (4096 total, 255 per segment), `maxEvidenceManifestBytes` + (1 MiB). The materializer sums `cat-file -s` sizes before writing as a disk-space preflight and + streams blobs so per-file-valid adversarial trees cannot exhaust memory or inodes. +6. **Integrity chain.** The snapshot `snapshot.json` manifest gains an entry + `.evidence.manifest.json` (its sha256) for every filesystem-Evidence workspace; the existing + `assertManifestFiles` chain therefore verifies the evidence manifest before reuse. On re-activation + of a commit, an already-materialized root is reused only when its manifest digest matches the + snapshot manifest; a missing or mismatched manifest fails closed (never silently reuses). +7. **Stop removal.** `evidencePolicy` no longer blocks filesystem sources; `preprocess evidence` and + `preprocess run` proceed against the materialized root. HTTP/S3 evidence behavior is unchanged. +8. **No GC change.** Retention of materialized roots is inherited from the commit-addressed snapshot + directory; no separate cleanup owns Evidence files. + +## Completion contract + +The phase is complete when: + +1. A workspace whose pinned commit contains a valid `/evidence` tree activates and materializes + every regular blob to `///evidence/` with a verified sibling manifest whose + digest appears in `snapshot.json`; a filesystem-Evidence workspace preprocesses, indexes, and + re-runs idempotently through the existing engine (no `evidence_materialization_required`). +2. Symlink/gitlink at any depth, traversal/absolute/duplicate/cross-namespace paths, oversized + files, and total/entry/path/manifest bound violations are refused without publishing a partial + root; the previous valid snapshot remains active. +3. Re-activation of the same commit reuses a valid materialized root and fails closed on a tampered + evidence manifest or file digest mismatch. +4. A pinned historical revision retains its materialized root; an unreferenced revision's root is + removed together with its snapshot directory by the existing retention scan. +5. `docs/contracts/workspace-preprocessing-cli.md` (or a dedicated P6 contract section) and the P6 + manual walkthrough are runnable; PROJECT_STATE.md records the result. +6. The clean-state automated process goal passes 1/1 (no retry), and backend/Go/harness focused + suites plus the existing P1.1–P5 gates do not regress. + +--- + +### Task 1: Safe Git tree enumeration + bounded blob streaming + +**Files:** modify `backend/src/workspaces/git-repository.ts`; tests +`backend/test/workspaces-git-evidence.test.ts`. + +1. Failing tests: `evidenceTreeObjects(revision, id)` returns ordered regular-blob entries + (`mode`, `oid`, `posixPath`) for a valid `/evidence` tree, and refuses symlink/gitlink/ + non-regular modes, non-blob types, traversal/absolute/NUL/newline/duplicate/cross-namespace paths, + and malformed revisions; `gitBlobBuffer` returns bounded bytes and refuses oversized objects. +2. Implement enumeration (`ls-tree -r -z`, path grammar, mode/type checks, duplicate detection) and + bounded blob reads (`cat-file blob`, `maxBuffer` + size guard). +3. Commit: `feat: safe Evidence tree enumeration and bounded blob streaming (P6)`. + +### Task 2: Evidence materializer with manifest and atomic publication + +**Files:** add `backend/src/workspaces/evidence-materialization.ts`; tests +`backend/test/evidence-materialization.test.ts`. + +1. Failing tests: materialize a fixture tree into a fresh owned staging root with exclusive/no-follow + writes, per-file hashes, an ordered manifest, fsync + atomic rename; refuse symlink/gitlink/ + special-file/traversal entries; enforce entry/total/per-file/path/manifest bounds (including a + size-sum preflight); a tampered destination or manifest fails closed on reuse. +2. Implement `materializeEvidenceTree({ repository, revision, id, stagingParent, limits })` returning + `{ root, manifestPath, manifest, manifestDigest }`. +3. Commit: `feat: bounded Evidence materializer with manifest and atomic publication (P6)`. + +### Task 3: Registry activation integration + integrity chain + +**Files:** modify `backend/src/workspaces/registry.ts`, `backend/src/workspaces/types.ts`, +`backend/src/config.ts`; tests `backend/test/registry-evidence.test.ts`. + +1. Failing tests: activation with a filesystem-Evidence workspace materializes the tree inside the + staged snapshot directory, writes the sibling manifest, records its digest in `snapshot.json`, + and atomically renames; re-activation reuses a valid root and fails closed on a tampered manifest; + an unsafe tree leaves the previous valid snapshot active; the evidence limits are configurable + through `WorkspaceRegistryConfig`. +2. Implement the staging integration, manifest-digest recording, integrity verification, and the new + config limits with env defaults. +3. Commit: `feat: activate commit-addressed Evidence materialization with an integrity chain (P6)`. + +### Task 4: Remove the filesystem Evidence stop + +**Files:** modify `backend/src/workspaces/preprocessing-service.ts`; tests +`workspace-preprocessing-service.test.ts`. + +1. Failing tests: `preprocess evidence` and `preprocess run` on a filesystem-Evidence workspace no + longer return `evidence_materialization_required` and instead invoke the evidence stage; HTTP/S3 + policy guards still apply unchanged. +2. Implement the `evidencePolicy` change. +3. Commit: `feat: make filesystem Evidence operational after materialization (P6)`. + +### Task 5: Contract, manual walkthrough, and clean-state acceptance + +**Files:** modify `docs/contracts/workspace-preprocessing-cli.md`, +`docs/testing/p2-p6-manual-verification.md` (P6 section), `PROJECT_STATE.md`; add +`scripts/p6-acceptance.sh`, `scripts/test-p6-acceptance.sh`, `backend/scripts/p6-acceptance.mjs`, +`backend/scripts/p6-acceptance.test.mjs` (pattern: P5 acceptance, owned root +`.artifacts/p6-integration/p6-/`). + +1. Document the Evidence lifecycle, limits, and exit codes. +2. Implement the clean-state scenario: fixture P1.1 registry with a filesystem Evidence tree + REST + DWH + pre-provisioned Qdrant; run `thothctl` product commands; prove materialization + manifest, + preprocessing/idempotency, revision-filtered Qdrant retrieval and corpus ACTIVE, unsafe-tree and + bound negatives without partial publication, retention while pinned and cleanup after release, + secret scan, exact cleanup. +3. Finalize the P6 manual walkthrough section and record the phase in PROJECT_STATE.md. +4. Commit: `feat: P6 commit-addressed Evidence materialization (acceptance + docs)`. + +--- + +## Owner checkpoint + +After Task 5 the implementation stops for recap. The owner records the P6 manual acceptance +(automated PASS is never recorded as manual PASS), then authorizes the final aggregate P2–P6 +verification and the user-guide deliverable. + +## Non-goals + +- No HTTP/S3 Evidence changes (they remain supported as before). +- No real PSD migration, SSH runtime transport, or policy-driven GC beyond the existing snapshot + retention. +- No change to the accepted P1/P1.1/P2/P3/P4/P5 contracts or retained evidence beyond the documented + P6 removal of the temporary filesystem stop.