P6 — Commit-addressed Evidence materialization — Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:executing-plans to apply this plan task-by-task.

Goal: Materialize the filesystem Evidence tree <workspace-id>/evidence from the exact pinned Git commit into an immutable revision content root, verify real containment (no symlink/gitlink/ traversal/special-file escape), and make P2's filesystem Evidence path operational end-to-end by removing the temporary evidence_materialization_required stop.

Source of truth: PRD D6 (docs/prd/2026-08-09-workspace-preprocessing-prd.md) and design §8 (docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md).

Architecture: A new shared TypeScript materializer enumerates the tree with fixed Git plumbing (ls-tree -r -z + cat-file blob) and writes regular files no-follow/exclusive beneath a fresh owned staging directory, hashing every file into a bounded manifest. The registry runs it during snapshot staging so the materialized root lands atomically inside the already-retained, commit- addressed snapshot directory; a tampered or mismatched manifest fails closed. The runtime renderer and the harness filesystem Evidence adapter are already rooted at that directory and need no change.

Tech Stack: TypeScript (registry + materializer), Python (existing filesystem Evidence adapter), YAML. TDD throughout.

Current-state findings recorded by this plan

The registry already validates the filesystem Evidence root object is a Git tree at the pinned commit (assertTreeAtRevision) and freezes it as revisionContentRoot = dirname(snapshotPath).
renderEvidence already resolves filesystem Evidence to join(revisionContentRoot, source.uri) (<snapshots>/<commit>/<id>/evidence), and the harness FilesystemEvidenceSource reads exactly that directory with no-follow opens and **/*.md discovery.
WorkspacePreprocessingService.evidencePolicy currently returns evidence_materialization_required for filesystem sources (the P2 temporary stop).
reconcileSnapshotRetention removes whole commit-addressed <snapshots>/<commit>/ directories, so materialized evidence beneath that directory is automatically retained while pinned and removed only when the revision becomes unreferenced.
The registry activate() staging already writes immutable <id>.yaml/.env.example/.md + snapshot.json and renames atomically; the comment at expectedSnapshotFiles marks P6 as the owner of workspace-content materialization.

Explicit decisions frozen by this plan

Target layout. Materialized filesystem Evidence lives at <registryRoot>/snapshots/<commit>/<id>/evidence/ with a sibling bounded manifest <registryRoot>/snapshots/<commit>/<id>/evidence.manifest.json. The manifest records { workspace, commit, tree, entryCount, totalBytes, files: { "<posix-path>": { mode, oid, digest, bytes } } }. The sibling manifest is outside the discovery root so the Evidence adapter never ingests it.
Eager, fail-closed materialization at activation. During activate() snapshot staging, every filesystem-Evidence workspace is materialized before the staging directory is atomically renamed. A missing Evidence root, an unsafe object, a bound violation, or a write failure aborts activation (workspace_invalid); no partial root is published. An empty Evidence tree is valid (empty root + zero-entry manifest).
Fixed Git plumbing, no shell, no mobile checkout. Enumeration is git ls-tree -r -z <commit> -- <id>/evidence; blob bytes come from git cat-file blob <oid> (buffer, per-object bound). No archive is extracted and no author files are consulted.
Object safety. Reject at any depth: symlink (120000), gitlink/submodule (160000), non- regular modes other than 100644/100755, non-blob type, absolute/./../NUL/newline/ non-normalized paths, duplicate normalized paths, cross-workspace namespaces, and any object whose id or bytes change between enumeration and read.
Bounds. Installation-local non-secret limits with conservative defaults: maxEvidenceEntries (4096 files), maxEvidenceBytes (64 MiB total), maxEvidenceFileBytes (8 MiB per file), maxEvidencePathBytes (4096 total, 255 per segment), maxEvidenceManifestBytes (1 MiB). The materializer sums cat-file -s sizes before writing as a disk-space preflight and streams blobs so per-file-valid adversarial trees cannot exhaust memory or inodes.
Integrity chain. The snapshot snapshot.json manifest gains an entry <id>.evidence.manifest.json (its sha256) for every filesystem-Evidence workspace; the existing assertManifestFiles chain therefore verifies the evidence manifest before reuse. On re-activation of a commit, an already-materialized root is reused only when its manifest digest matches the snapshot manifest; a missing or mismatched manifest fails closed (never silently reuses).
Stop removal. evidencePolicy no longer blocks filesystem sources; preprocess evidence and preprocess run proceed against the materialized root. HTTP/S3 evidence behavior is unchanged.
No GC change. Retention of materialized roots is inherited from the commit-addressed snapshot directory; no separate cleanup owns Evidence files.

Completion contract

The phase is complete when:

A workspace whose pinned commit contains a valid <id>/evidence tree activates and materializes every regular blob to <snapshots>/<commit>/<id>/evidence/ with a verified sibling manifest whose digest appears in snapshot.json; a filesystem-Evidence workspace preprocesses, indexes, and re-runs idempotently through the existing engine (no evidence_materialization_required).
Symlink/gitlink at any depth, traversal/absolute/duplicate/cross-namespace paths, oversized files, and total/entry/path/manifest bound violations are refused without publishing a partial root; the previous valid snapshot remains active.
Re-activation of the same commit reuses a valid materialized root and fails closed on a tampered evidence manifest or file digest mismatch.
A pinned historical revision retains its materialized root; an unreferenced revision's root is removed together with its snapshot directory by the existing retention scan.
docs/contracts/workspace-preprocessing-cli.md (or a dedicated P6 contract section) and the P6 manual walkthrough are runnable; PROJECT_STATE.md records the result.
The clean-state automated process goal passes 1/1 (no retry), and backend/Go/harness focused suites plus the existing P1.1–P5 gates do not regress.

Task 1: Safe Git tree enumeration + bounded blob streaming

Files: modify backend/src/workspaces/git-repository.ts; tests backend/test/workspaces-git-evidence.test.ts.

Failing tests: evidenceTreeObjects(revision, id) returns ordered regular-blob entries (mode, oid, posixPath) for a valid <id>/evidence tree, and refuses symlink/gitlink/ non-regular modes, non-blob types, traversal/absolute/NUL/newline/duplicate/cross-namespace paths, and malformed revisions; gitBlobBuffer returns bounded bytes and refuses oversized objects.
Implement enumeration (ls-tree -r -z, path grammar, mode/type checks, duplicate detection) and bounded blob reads (cat-file blob, maxBuffer + size guard).
Commit: feat: safe Evidence tree enumeration and bounded blob streaming (P6).

Task 2: Evidence materializer with manifest and atomic publication

Files: add backend/src/workspaces/evidence-materialization.ts; tests backend/test/evidence-materialization.test.ts.

Failing tests: materialize a fixture tree into a fresh owned staging root with exclusive/no-follow writes, per-file hashes, an ordered manifest, fsync + atomic rename; refuse symlink/gitlink/ special-file/traversal entries; enforce entry/total/per-file/path/manifest bounds (including a size-sum preflight); a tampered destination or manifest fails closed on reuse.
Implement materializeEvidenceTree({ repository, revision, id, stagingParent, limits }) returning { root, manifestPath, manifest, manifestDigest }.
Commit: feat: bounded Evidence materializer with manifest and atomic publication (P6).

Task 3: Registry activation integration + integrity chain

Files: modify backend/src/workspaces/registry.ts, backend/src/workspaces/types.ts, backend/src/config.ts; tests backend/test/registry-evidence.test.ts.

Failing tests: activation with a filesystem-Evidence workspace materializes the tree inside the staged snapshot directory, writes the sibling manifest, records its digest in snapshot.json, and atomically renames; re-activation reuses a valid root and fails closed on a tampered manifest; an unsafe tree leaves the previous valid snapshot active; the evidence limits are configurable through WorkspaceRegistryConfig.
Implement the staging integration, manifest-digest recording, integrity verification, and the new config limits with env defaults.
Commit: feat: activate commit-addressed Evidence materialization with an integrity chain (P6).

Task 4: Remove the filesystem Evidence stop

Files: modify backend/src/workspaces/preprocessing-service.ts; tests workspace-preprocessing-service.test.ts.

Failing tests: preprocess evidence and preprocess run on a filesystem-Evidence workspace no longer return evidence_materialization_required and instead invoke the evidence stage; HTTP/S3 policy guards still apply unchanged.
Implement the evidencePolicy change.
Commit: feat: make filesystem Evidence operational after materialization (P6).

Task 5: Contract, manual walkthrough, and clean-state acceptance

Files: modify docs/contracts/workspace-preprocessing-cli.md, docs/testing/p2-p6-manual-verification.md (P6 section), PROJECT_STATE.md; add scripts/p6-acceptance.sh, scripts/test-p6-acceptance.sh, backend/scripts/p6-acceptance.mjs, backend/scripts/p6-acceptance.test.mjs (pattern: P5 acceptance, owned root .artifacts/p6-integration/p6-<run-id>/).

Document the Evidence lifecycle, limits, and exit codes.
Implement the clean-state scenario: fixture P1.1 registry with a filesystem Evidence tree + REST DWH + pre-provisioned Qdrant; run thothctl product commands; prove materialization + manifest, preprocessing/idempotency, revision-filtered Qdrant retrieval and corpus ACTIVE, unsafe-tree and bound negatives without partial publication, retention while pinned and cleanup after release, secret scan, exact cleanup.
Finalize the P6 manual walkthrough section and record the phase in PROJECT_STATE.md.
Commit: feat: P6 commit-addressed Evidence materialization (acceptance + docs).

Owner checkpoint

After Task 5 the implementation stops for recap. The owner records the P6 manual acceptance (automated PASS is never recorded as manual PASS), then authorizes the final aggregate P2–P6 verification and the user-guide deliverable.

Non-goals

No HTTP/S3 Evidence changes (they remain supported as before).
No real PSD migration, SSH runtime transport, or policy-driven GC beyond the existing snapshot retention.
No change to the accepted P1/P1.1/P2/P3/P4/P5 contracts or retained evidence beyond the documented P6 removal of the temporary filesystem stop.

11 KiB Raw Blame History Unescape Escape