diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index 658f112d..78fcb328 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -7,7 +7,7 @@ > ThothII per il repository (app + CLI `thothctl`), (3) come usare l'applicazione ThothII di base > (sessioni, domande, gate). Il documento userà parole semplici ed esempi; i dettagli tecnici > resteranno nei contratti esistenti. Esempio pratico completo: Policlinico San Donato. - Last updated: 2026-08-13 (P3+P4+P5 manual acceptance). + Last updated: 2026-08-13 (P3+P4+P5 manual acceptance; P6 automated PASS). > Point a fresh session here ("read PROJECT_STATE.md") before substantial work. ### P3 effective configuration and `.tht-dwh` — implementation complete, automated PASS, manual PASS (2026-08-13) @@ -101,6 +101,38 @@ - **Manual acceptance:** PASS (owner approval 2026-08-13) — walkthrough section P5 in `docs/testing/p2-p6-manual-verification.md`. +### P6 commit-addressed Evidence materialization — implementation complete, automated PASS, manual PENDING (2026-08-13) + +- **Scope:** P6 (PRD D6): filesystem Evidence `/evidence` is materialized from the exact pinned + Git commit into the immutable revision content root `/snapshots///evidence` + at activation, with a sibling bounded manifest `/evidence.manifest.json` whose digest is chained + into `snapshot.json`. +- **Safety:** fixed Git plumbing (`ls-tree -r -z` + `cat-file blob`), no shell, no mobile checkout; + symlinks/gitlinks at any depth, traversal/absolute/duplicate/cross-namespace paths, and non-regular + modes are refused. Installation-local bounds (defaults): 4096 entries, 64 MiB total, 8 MiB per + file, 4096 path bytes, 1 MiB manifest; a size-sum preflight runs before writing and no partial root + is published. Re-activation reuses a valid root and fails closed on a tampered manifest. +- **Engine:** `evidencePolicy` no longer stops filesystem sources (`evidence_materialization_required` + retired); `preprocess evidence`/`preprocess run` operate on the materialized root. Evidence Qdrant + records remain revision-scoped; corpus ACTIVE is revision-qualified. HTTP/S3 Evidence is unchanged. +- **Retention:** materialized roots live inside the commit-addressed snapshot directory, so they are + retained while pinned and removed by the existing snapshot retention scan when unreferenced. +- **Key files:** `backend/src/workspaces/evidence-materialization.ts` (+test), + `backend/src/workspaces/git-repository.ts` (`evidenceTreeObjects`/`evidenceTreeId`/ + `evidenceBlobBytes`/`gitObjectSize`), `backend/src/workspaces/registry.ts` (activation staging + + integrity chain), `backend/src/workspaces/preprocessing-service.ts` (stop removal), + `backend/src/workspaces/types.ts` + `config.ts` (limits), `docs/contracts/ + workspace-preprocessing-cli.md`. +- **Gates:** backend **698/698** + tsc clean; Go build+test 9/9 (unchanged); harness focused suites + pass. Full-suite re-run and the clean-state process goal recorded at the acceptance gate. +- **Automated acceptance:** PASS 10/10 (run `p6-7a4c4d0ebb63399cfa9f674738b9e8fc`, report + `.artifacts/p6-integration/p6-7a4c4d0ebb63399cfa9f674738b9e8fc/` retained via `--keep`, bound to + clean source commit `124891bbfe8dc8270e8b58c4206150eb8bebeaa7`): preflight, clean_state, ownership, + activation_materialization, evidence_preprocess, revision_isolation, unsafe_tree_refused, + bound_refused, secret_scan, cleanup_confinement. Runner: `scripts/p6-acceptance.sh` / + `backend/scripts/p6-acceptance.mjs` (+unit test `scripts/test-p6-acceptance.sh`). +- **Manual acceptance:** PENDING — walkthrough section P6 in `docs/testing/p2-p6-manual-verification.md`. + ### P2 host preprocessing CLI — implementation complete, automated PASS, manual PENDING (2026-08-11) - **Scope:** P2 (PRD D2, based on the P1.1 registry contract): the installed native `thothctl` diff --git a/docs/contracts/workspace-preprocessing-cli.md b/docs/contracts/workspace-preprocessing-cli.md index 204300a9..353ef405 100644 --- a/docs/contracts/workspace-preprocessing-cli.md +++ b/docs/contracts/workspace-preprocessing-cli.md @@ -79,6 +79,23 @@ thothctl --installation /thothii-installation.yaml workspace vector re - `preprocess run` continues only with the exact accepted blob digest and a compatible reusable DWH binding; otherwise it records a new review checkpoint. +## Commit-addressed Evidence materialization (P6) + +- Filesystem Evidence `/evidence` is materialized from the exact pinned Git commit + into the immutable revision content root `/snapshots///evidence` at + activation, with a sibling bounded manifest `/evidence.manifest.json` whose digest is chained + into `snapshot.json`. +- Materialization uses fixed Git plumbing (`ls-tree -r -z` + `cat-file blob`) and refuses symlinks + and gitlinks at any depth, traversal/absolute/duplicate/cross-namespace paths, and non-regular + modes. Installation-local limits bound entry count (default 4096), total bytes (64 MiB), + per-file bytes (8 MiB), path bytes (4096), and manifest bytes (1 MiB); a size-sum preflight runs + before any bytes are written and no partial root is published. +- `preprocess evidence` and `preprocess run` operate directly on the materialized root; the + temporary `evidence_materialization_required` stop is retired (the code remains only for + pre-P6 compatibility). HTTP/S3 Evidence is unchanged. +- Materialized roots are retained with their commit-addressed snapshot directory and removed only + when the revision becomes unreferenced. + ## Validation - `--installation` is mandatory and absolute. @@ -133,6 +150,9 @@ The request is streamed as one schema-versioned JSON document over stdin. Public `thothctl --json` parses the operator stdout strictly and re-encodes only the public fields above. +`evidence_materialization_required` is retained for pre-P6 compatibility; since P6, filesystem +Evidence is materialized at activation and preprocesses directly. + ## Exit codes - `0`: `succeeded`, `unchanged`, or `dry_run` diff --git a/docs/testing/p2-p6-manual-verification.md b/docs/testing/p2-p6-manual-verification.md index 585f00d0..f564ce0d 100644 --- a/docs/testing/p2-p6-manual-verification.md +++ b/docs/testing/p2-p6-manual-verification.md @@ -164,25 +164,43 @@ Checks: Decision: **PASS** (owner approval 2026-08-13). ## P6 — Commit-addressed Evidence materialization -**Status:** instructions to be finalized by P6 implementation; not yet runnable. +**Status:** P6 implementation complete; automated integration PASS; manual acceptance PENDING. Manual goal: materialize filesystem Evidence from the pinned Git commit, inspect its bounded manifest, preprocess/index it, retrieve only the pinned revision, and exercise unsafe-tree and aggregate-limit failures without partial publication. -Checks to fill during P6: +Commands (contract: `docs/contracts/workspace-preprocessing-cli.md`): -1. exact commit/tree/object identities; -2. successful atomic materialization; -3. manifest and file digest verification; -4. filesystem Evidence dry-run/run/idempotency; -5. revision-filtered Qdrant retrieval and corpus ACTIVE; -6. nested symlink/gitlink/traversal/special-file refusal; -7. file-count/total-byte/path/manifest limit refusal; -8. retention while pinned and owned cleanup after release. +```bash +thothctl --installation /thothii-installation.yaml workspace inspect --workspace --json +thothctl --installation /thothii-installation.yaml workspace preprocess evidence --workspace --dry-run --json +thothctl --installation /thothii-installation.yaml workspace preprocess evidence --workspace --json +thothctl --installation /thothii-installation.yaml workspace preprocess evidence --workspace --json # idempotent rerun +``` + +Checks: + +1. exact commit/tree/object identities: after activation the materialized root is + `/snapshots///evidence` and its sibling manifest + `/evidence.manifest.json` records `workspace`, `commit`, `tree`, per-file `oid`/`digest`, + `entryCount`, `totalBytes`; `snapshot.json` chains the manifest digest; +2. successful atomic materialization: every regular blob is present byte-for-byte; the manifest + digests match; +3. manifest and file digest verification: re-activation reuses a valid root and fails closed on a + tampered manifest; +4. filesystem Evidence dry-run/run/idempotency: `--dry-run` returns `dry_run`, the real run + publishes, rerun is `unchanged`; +5. revision-filtered Qdrant retrieval and corpus ACTIVE: Evidence records carry the pinned + `workspace_revision`; +6. nested symlink/gitlink/traversal/special-file refusal: a commit introducing one of these fails + activation (`workspace_invalid`) and the previous valid revision stays active; +7. file-count/total-byte/path/manifest limit refusal: an oversized or over-count tree fails closed + without a partial publication; +8. retention while pinned and owned cleanup after release: the materialized root persists for a + pinned revision and is removed with its snapshot directory once unreferenced. Decision: **PENDING**. - ## Final aggregate P2–P6 verification **Status:** runnable only after P6.