180 lines
11 KiB
Markdown
180 lines
11 KiB
Markdown
# P6 — Commit-addressed Evidence materialization — Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to apply this plan task-by-task.
|
||
|
||
**Goal:** Materialize the filesystem Evidence tree `<workspace-id>/evidence` from the exact pinned Git
|
||
commit into an immutable revision content root, verify real containment (no symlink/gitlink/
|
||
traversal/special-file escape), and make P2's filesystem Evidence path operational end-to-end by
|
||
removing the temporary `evidence_materialization_required` stop.
|
||
|
||
**Source of truth:** PRD D6 (`docs/prd/2026-08-09-workspace-preprocessing-prd.md`) and design §8
|
||
(`docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md`).
|
||
|
||
**Architecture:** A new shared TypeScript materializer enumerates the tree with fixed Git plumbing
|
||
(`ls-tree -r -z` + `cat-file blob`) and writes regular files no-follow/exclusive beneath a fresh
|
||
owned staging directory, hashing every file into a bounded manifest. The registry runs it during
|
||
snapshot staging so the materialized root lands atomically inside the already-retained, commit-
|
||
addressed snapshot directory; a tampered or mismatched manifest fails closed. The runtime renderer
|
||
and the harness filesystem Evidence adapter are already rooted at that directory and need no change.
|
||
|
||
**Tech Stack:** TypeScript (registry + materializer), Python (existing filesystem Evidence adapter),
|
||
YAML. TDD throughout.
|
||
|
||
---
|
||
|
||
## Current-state findings recorded by this plan
|
||
|
||
- The registry already validates the filesystem Evidence *root object* is a Git tree at the pinned
|
||
commit (`assertTreeAtRevision`) and freezes it as `revisionContentRoot = dirname(snapshotPath)`.
|
||
- `renderEvidence` already resolves filesystem Evidence to `join(revisionContentRoot, source.uri)`
|
||
(`<snapshots>/<commit>/<id>/evidence`), and the harness `FilesystemEvidenceSource` reads exactly
|
||
that directory with no-follow opens and `**/*.md` discovery.
|
||
- `WorkspacePreprocessingService.evidencePolicy` currently returns
|
||
`evidence_materialization_required` for filesystem sources (the P2 temporary stop).
|
||
- `reconcileSnapshotRetention` removes whole commit-addressed `<snapshots>/<commit>/` directories,
|
||
so materialized evidence beneath that directory is automatically retained while pinned and removed
|
||
only when the revision becomes unreferenced.
|
||
- The registry `activate()` staging already writes immutable `<id>.yaml`/`.env.example`/`.md` +
|
||
`snapshot.json` and renames atomically; the comment at `expectedSnapshotFiles` marks P6 as the
|
||
owner of workspace-content materialization.
|
||
|
||
## Explicit decisions frozen by this plan
|
||
|
||
1. **Target layout.** Materialized filesystem Evidence lives at
|
||
`<registryRoot>/snapshots/<commit>/<id>/evidence/` with a sibling bounded manifest
|
||
`<registryRoot>/snapshots/<commit>/<id>/evidence.manifest.json`. The manifest records
|
||
`{ workspace, commit, tree, entryCount, totalBytes, files: { "<posix-path>": { mode, oid, digest, bytes } } }`.
|
||
The sibling manifest is outside the discovery root so the Evidence adapter never ingests it.
|
||
2. **Eager, fail-closed materialization at activation.** During `activate()` snapshot staging, every
|
||
filesystem-Evidence workspace is materialized before the staging directory is atomically renamed.
|
||
A missing Evidence root, an unsafe object, a bound violation, or a write failure aborts activation
|
||
(`workspace_invalid`); no partial root is published. An empty Evidence tree is valid (empty root +
|
||
zero-entry manifest).
|
||
3. **Fixed Git plumbing, no shell, no mobile checkout.** Enumeration is
|
||
`git ls-tree -r -z <commit> -- <id>/evidence`; blob bytes come from `git cat-file blob <oid>`
|
||
(buffer, per-object bound). No archive is extracted and no author files are consulted.
|
||
4. **Object safety.** Reject at any depth: symlink (`120000`), gitlink/submodule (`160000`), non-
|
||
regular modes other than `100644`/`100755`, non-`blob` type, absolute/`.`/`..`/NUL/newline/
|
||
non-normalized paths, duplicate normalized paths, cross-workspace namespaces, and any object whose
|
||
id or bytes change between enumeration and read.
|
||
5. **Bounds.** Installation-local non-secret limits with conservative defaults:
|
||
`maxEvidenceEntries` (4096 files), `maxEvidenceBytes` (64 MiB total), `maxEvidenceFileBytes`
|
||
(8 MiB per file), `maxEvidencePathBytes` (4096 total, 255 per segment), `maxEvidenceManifestBytes`
|
||
(1 MiB). The materializer sums `cat-file -s` sizes before writing as a disk-space preflight and
|
||
streams blobs so per-file-valid adversarial trees cannot exhaust memory or inodes.
|
||
6. **Integrity chain.** The snapshot `snapshot.json` manifest gains an entry
|
||
`<id>.evidence.manifest.json` (its sha256) for every filesystem-Evidence workspace; the existing
|
||
`assertManifestFiles` chain therefore verifies the evidence manifest before reuse. On re-activation
|
||
of a commit, an already-materialized root is reused only when its manifest digest matches the
|
||
snapshot manifest; a missing or mismatched manifest fails closed (never silently reuses).
|
||
7. **Stop removal.** `evidencePolicy` no longer blocks filesystem sources; `preprocess evidence` and
|
||
`preprocess run` proceed against the materialized root. HTTP/S3 evidence behavior is unchanged.
|
||
8. **No GC change.** Retention of materialized roots is inherited from the commit-addressed snapshot
|
||
directory; no separate cleanup owns Evidence files.
|
||
|
||
## Completion contract
|
||
|
||
The phase is complete when:
|
||
|
||
1. A workspace whose pinned commit contains a valid `<id>/evidence` tree activates and materializes
|
||
every regular blob to `<snapshots>/<commit>/<id>/evidence/` with a verified sibling manifest whose
|
||
digest appears in `snapshot.json`; a filesystem-Evidence workspace preprocesses, indexes, and
|
||
re-runs idempotently through the existing engine (no `evidence_materialization_required`).
|
||
2. Symlink/gitlink at any depth, traversal/absolute/duplicate/cross-namespace paths, oversized
|
||
files, and total/entry/path/manifest bound violations are refused without publishing a partial
|
||
root; the previous valid snapshot remains active.
|
||
3. Re-activation of the same commit reuses a valid materialized root and fails closed on a tampered
|
||
evidence manifest or file digest mismatch.
|
||
4. A pinned historical revision retains its materialized root; an unreferenced revision's root is
|
||
removed together with its snapshot directory by the existing retention scan.
|
||
5. `docs/contracts/workspace-preprocessing-cli.md` (or a dedicated P6 contract section) and the P6
|
||
manual walkthrough are runnable; PROJECT_STATE.md records the result.
|
||
6. The clean-state automated process goal passes 1/1 (no retry), and backend/Go/harness focused
|
||
suites plus the existing P1.1–P5 gates do not regress.
|
||
|
||
---
|
||
|
||
### Task 1: Safe Git tree enumeration + bounded blob streaming
|
||
|
||
**Files:** modify `backend/src/workspaces/git-repository.ts`; tests
|
||
`backend/test/workspaces-git-evidence.test.ts`.
|
||
|
||
1. Failing tests: `evidenceTreeObjects(revision, id)` returns ordered regular-blob entries
|
||
(`mode`, `oid`, `posixPath`) for a valid `<id>/evidence` tree, and refuses symlink/gitlink/
|
||
non-regular modes, non-blob types, traversal/absolute/NUL/newline/duplicate/cross-namespace paths,
|
||
and malformed revisions; `gitBlobBuffer` returns bounded bytes and refuses oversized objects.
|
||
2. Implement enumeration (`ls-tree -r -z`, path grammar, mode/type checks, duplicate detection) and
|
||
bounded blob reads (`cat-file blob`, `maxBuffer` + size guard).
|
||
3. Commit: `feat: safe Evidence tree enumeration and bounded blob streaming (P6)`.
|
||
|
||
### Task 2: Evidence materializer with manifest and atomic publication
|
||
|
||
**Files:** add `backend/src/workspaces/evidence-materialization.ts`; tests
|
||
`backend/test/evidence-materialization.test.ts`.
|
||
|
||
1. Failing tests: materialize a fixture tree into a fresh owned staging root with exclusive/no-follow
|
||
writes, per-file hashes, an ordered manifest, fsync + atomic rename; refuse symlink/gitlink/
|
||
special-file/traversal entries; enforce entry/total/per-file/path/manifest bounds (including a
|
||
size-sum preflight); a tampered destination or manifest fails closed on reuse.
|
||
2. Implement `materializeEvidenceTree({ repository, revision, id, stagingParent, limits })` returning
|
||
`{ root, manifestPath, manifest, manifestDigest }`.
|
||
3. Commit: `feat: bounded Evidence materializer with manifest and atomic publication (P6)`.
|
||
|
||
### Task 3: Registry activation integration + integrity chain
|
||
|
||
**Files:** modify `backend/src/workspaces/registry.ts`, `backend/src/workspaces/types.ts`,
|
||
`backend/src/config.ts`; tests `backend/test/registry-evidence.test.ts`.
|
||
|
||
1. Failing tests: activation with a filesystem-Evidence workspace materializes the tree inside the
|
||
staged snapshot directory, writes the sibling manifest, records its digest in `snapshot.json`,
|
||
and atomically renames; re-activation reuses a valid root and fails closed on a tampered manifest;
|
||
an unsafe tree leaves the previous valid snapshot active; the evidence limits are configurable
|
||
through `WorkspaceRegistryConfig`.
|
||
2. Implement the staging integration, manifest-digest recording, integrity verification, and the new
|
||
config limits with env defaults.
|
||
3. Commit: `feat: activate commit-addressed Evidence materialization with an integrity chain (P6)`.
|
||
|
||
### Task 4: Remove the filesystem Evidence stop
|
||
|
||
**Files:** modify `backend/src/workspaces/preprocessing-service.ts`; tests
|
||
`workspace-preprocessing-service.test.ts`.
|
||
|
||
1. Failing tests: `preprocess evidence` and `preprocess run` on a filesystem-Evidence workspace no
|
||
longer return `evidence_materialization_required` and instead invoke the evidence stage; HTTP/S3
|
||
policy guards still apply unchanged.
|
||
2. Implement the `evidencePolicy` change.
|
||
3. Commit: `feat: make filesystem Evidence operational after materialization (P6)`.
|
||
|
||
### Task 5: Contract, manual walkthrough, and clean-state acceptance
|
||
|
||
**Files:** modify `docs/contracts/workspace-preprocessing-cli.md`,
|
||
`docs/testing/p2-p6-manual-verification.md` (P6 section), `PROJECT_STATE.md`; add
|
||
`scripts/p6-acceptance.sh`, `scripts/test-p6-acceptance.sh`, `backend/scripts/p6-acceptance.mjs`,
|
||
`backend/scripts/p6-acceptance.test.mjs` (pattern: P5 acceptance, owned root
|
||
`.artifacts/p6-integration/p6-<run-id>/`).
|
||
|
||
1. Document the Evidence lifecycle, limits, and exit codes.
|
||
2. Implement the clean-state scenario: fixture P1.1 registry with a filesystem Evidence tree + REST
|
||
DWH + pre-provisioned Qdrant; run `thothctl` product commands; prove materialization + manifest,
|
||
preprocessing/idempotency, revision-filtered Qdrant retrieval and corpus ACTIVE, unsafe-tree and
|
||
bound negatives without partial publication, retention while pinned and cleanup after release,
|
||
secret scan, exact cleanup.
|
||
3. Finalize the P6 manual walkthrough section and record the phase in PROJECT_STATE.md.
|
||
4. Commit: `feat: P6 commit-addressed Evidence materialization (acceptance + docs)`.
|
||
|
||
---
|
||
|
||
## Owner checkpoint
|
||
|
||
After Task 5 the implementation stops for recap. The owner records the P6 manual acceptance
|
||
(automated PASS is never recorded as manual PASS), then authorizes the final aggregate P2–P6
|
||
verification and the user-guide deliverable.
|
||
|
||
## Non-goals
|
||
|
||
- No HTTP/S3 Evidence changes (they remain supported as before).
|
||
- No real PSD migration, SSH runtime transport, or policy-driven GC beyond the existing snapshot
|
||
retention.
|
||
- No change to the accepted P1/P1.1/P2/P3/P4/P5 contracts or retained evidence beyond the documented
|
||
P6 removal of the temporary filesystem stop.
|