ThothII/.superpowers/sdd/evidence-task-7-report.md

Evidence preprocessing Task 7 report

Implemented the S3-compatible Evidence adapter, explicit preprocessing Compose overlay, and operational gates.

• S3 discovery uses bounded paginator pages, page size, and total objects; acquisition enforces a byte ceiling and always closes streaming bodies.
• Provenance is canonical s3://bucket/key. Versioned objects use s3-version:<version>; unversioned objects use a hashed exact ETag, and acquisition refuses validator drift.
• The adapter uses boto3/botocore rather than custom signing. TLS verification is enabled by default. Custom HTTP and private endpoints require independent explicit opt-ins; endpoint userinfo is rejected and public custom endpoints are DNS-policy checked.
• Access, secret, and session credentials support file-secret resolution into masked SecretStr config fields. They are never emitted in provenance, reports, errors, or Compose environment.
• deploy/compose.preprocess.yaml provides separate one-shot Evidence and DWH jobs and is inert unless explicitly included with the preprocess profile.
• scripts/preprocess-smoke.sh verifies both services render without secret material and pins an unchanged rerun plus a modified generation through deterministic pipeline tests.

Verification: focused S3/HTTP/filesystem/config tests 34 passed; operational smoke 2 passed; core image with locked boto3 extra built; full harness 702 passed, 5 deselected; scoped Ruff and diff checks passed.

Operational risk: custom S3-compatible endpoints remain part of the deployment trust boundary. Private endpoint access must be explicitly enabled and should be restricted by container egress policy in production. S3 list consistency semantics are provider-defined; version IDs are preferred over ETags wherever bucket versioning is available.

Review correction

The Compose overlay now uses committed, purpose-built Evidence and DWH workspace files with job-specific dependencies. Its services create their lock roots and mount only the vector secrets they consume. The operational smoke is a real isolated Compose project: real pgvector migrations, a deterministic in-project embeddings endpoint, actual Evidence CLI JSON across initial/unchanged/ mutated runs, exact ACTIVE verification, an actual DWH introspection job, and owned cleanup.

S3 custom endpoints now fail closed unless declared trusted; HTTP and private loopback endpoints need additional independent opt-ins. Boto uses forced path-style addressing. Custom endpoints reject userinfo, query, fragment, and non-root paths. Buckets use strict DNS syntax; listed keys must remain under prefix and within the S3 byte bound; validators must be nonempty/bounded. Because ListObjectsV2 does not provide version IDs, discovery honestly fingerprints the exact ETag and acquisition rejects ETag drift.

Final correction verification: S3/config focused 20 passed; full harness 721 passed, 5 deselected; real Compose smoke and image build passed; scoped Ruff, shell syntax, and diff checks passed.

Final security review correction

Literal non-global IPv4/IPv6 endpoints now require the private-endpoint opt-in without claiming DNS pinning for hostnames. Pagination uses explicit continuation requests and never fetches page max_pages + 1. IP-shaped buckets, leading-slash prefixes, empty/overlong/control-character keys, and absent validators fail closed. Acquisition accepts only the exact stored SourceObject and compares the response ETag with the stored discovery validator. The real smoke snapshots generation directory counts after every run and has an injected-failure cleanup mode; cleanup fails if Compose down fails or any owned container, volume, or network remains.

The canonical smoke correction counts only root-level corpus/gen-<32 hex> directories. It exposed that the durable job path still published an empty unchanged generation; the pipeline now returns the existing ACTIVE generation without staging a directory when compatibility and all source fingerprints are unchanged. The smoke therefore proves directory deltas +1, +0, +1. Failure injection runs a real exit-97 command after resources exist and reaches the EXIT trap. Cleanup aggregates Compose-down, residual container/volume/network, and temp-directory failures while preserving the original failure status. S3 prefixes are validated before any client request for leading slash, UTF-8 byte length, controls, and DEL.

Canonical unchanged-run correction

The durable job now persists a deterministic source snapshot keyed by source identity. Each entry binds canonical URI, exact source fingerprint, UTC modification time, canonical immutable metadata, and explicit media type and size contract fields. The manifest also binds document-to-source provenance, supplied config/input fingerprints, compatibility, embedding settings, and pipeline and chunk-policy versions.

An unchanged run reuses ACTIVE only when ownership, bindings, the complete snapshot, document provenance, materialized document hashes, and every required vector ID/content hash match exactly. Snapshot changes rebuild only the affected sources; job input/config changes publish a new manifest while retaining valid stable vector-generation dependencies. Missing or corrupt legacy contract metadata, documents, or vectors fails closed and rebuilds. The Compose smoke now explicitly expects the unchanged no-op to report published=false while proving generation deltas +1, +0, +1.

Corrupt ACTIVE reconstruction correction

ACTIVE reuse now reconstructs each source contract from the persisted discovery snapshot and checks the deterministic document identity, canonical URI, source fingerprint, UTC modification time, source metadata, applicable media type, content hash, and pipeline identity against the owned materialized document. The persisted document-source map carries the same exact binding.

Chunks are recomputed under the current chunk policy and must match the manifest exactly in count, order, IDs, ordinals, content, hashes, linkage, provenance, and policy metadata. Vector health must report the configured dimension, and every recomputed chunk must have its generation-scoped vector ID with the exact content hash. Missing, altered, or extra chunks and corrupt document or vector contracts therefore disable the no-op and rebuild, while a valid unchanged run still performs no source acquisition.