ThothII/.superpowers/sdd/evidence-task-7-report.md

Evidence preprocessing Task 7 report

Implemented the S3-compatible Evidence adapter, explicit preprocessing Compose overlay, and operational gates.

• S3 discovery uses bounded paginator pages, page size, and total objects; acquisition enforces a byte ceiling and always closes streaming bodies.
• Provenance is canonical s3://bucket/key. Versioned objects use s3-version:<version>; unversioned objects use a hashed exact ETag, and acquisition refuses validator drift.
• The adapter uses boto3/botocore rather than custom signing. TLS verification is enabled by default. Custom HTTP and private endpoints require independent explicit opt-ins; endpoint userinfo is rejected and public custom endpoints are DNS-policy checked.
• Access, secret, and session credentials support file-secret resolution into masked SecretStr config fields. They are never emitted in provenance, reports, errors, or Compose environment.
• deploy/compose.preprocess.yaml provides separate one-shot Evidence and DWH jobs and is inert unless explicitly included with the preprocess profile.
• scripts/preprocess-smoke.sh verifies both services render without secret material and pins an unchanged rerun plus a modified generation through deterministic pipeline tests.

Verification: focused S3/HTTP/filesystem/config tests 34 passed; operational smoke 2 passed; core image with locked boto3 extra built; full harness 702 passed, 5 deselected; scoped Ruff and diff checks passed.

Operational risk: custom S3-compatible endpoints remain part of the deployment trust boundary. Private endpoint access must be explicitly enabled and should be restricted by container egress policy in production. S3 list consistency semantics are provider-defined; version IDs are preferred over ETags wherever bucket versioning is available.

Review correction

The Compose overlay now uses committed, purpose-built Evidence and DWH workspace files with job-specific dependencies. Its services create their lock roots and mount only the vector secrets they consume. The operational smoke is a real isolated Compose project: real pgvector migrations, a deterministic in-project embeddings endpoint, actual Evidence CLI JSON across initial/unchanged/ mutated runs, exact ACTIVE verification, an actual DWH introspection job, and owned cleanup.

S3 custom endpoints now fail closed unless declared trusted; HTTP and private loopback endpoints need additional independent opt-ins. Boto uses forced path-style addressing. Custom endpoints reject userinfo, query, fragment, and non-root paths. Buckets use strict DNS syntax; listed keys must remain under prefix and within the S3 byte bound; validators must be nonempty/bounded. Because ListObjectsV2 does not provide version IDs, discovery honestly fingerprints the exact ETag and acquisition rejects ETag drift.

Final correction verification: S3/config focused 20 passed; full harness 721 passed, 5 deselected; real Compose smoke and image build passed; scoped Ruff, shell syntax, and diff checks passed.

Final security review correction

Literal non-global IPv4/IPv6 endpoints now require the private-endpoint opt-in without claiming DNS pinning for hostnames. Pagination uses explicit continuation requests and never fetches page max_pages + 1. IP-shaped buckets, leading-slash prefixes, empty/overlong/control-character keys, and absent validators fail closed. Acquisition accepts only the exact stored SourceObject and compares the response ETag with the stored discovery validator. The real smoke snapshots generation directory counts after every run and has an injected-failure cleanup mode; cleanup fails if Compose down fails or any owned container, volume, or network remains.