feat(preprocess): add deployment jobs and S3 source

2026-07-12 05:42:07 +02:00
parent 6656a69630
commit 4028ef7821
12 changed files with 380 additions and 10 deletions
@@ -0,0 +1,27 @@
+# Evidence preprocessing Task 7 report
+
+Implemented the S3-compatible Evidence adapter, explicit preprocessing Compose overlay, and
+operational gates.
+
+- S3 discovery uses bounded paginator pages, page size, and total objects; acquisition enforces a
+  byte ceiling and always closes streaming bodies.
+- Provenance is canonical `s3://bucket/key`. Versioned objects use `s3-version:<version>`;
+  unversioned objects use a hashed exact ETag, and acquisition refuses validator drift.
+- The adapter uses boto3/botocore rather than custom signing. TLS verification is enabled by
+  default. Custom HTTP and private endpoints require independent explicit opt-ins; endpoint
+  userinfo is rejected and public custom endpoints are DNS-policy checked.
+- Access, secret, and session credentials support file-secret resolution into masked `SecretStr`
+  config fields. They are never emitted in provenance, reports, errors, or Compose environment.
+- `deploy/compose.preprocess.yaml` provides separate one-shot Evidence and DWH jobs and is inert
+  unless explicitly included with the `preprocess` profile.
+- `scripts/preprocess-smoke.sh` verifies both services render without secret material and pins an
+  unchanged rerun plus a modified generation through deterministic pipeline tests.
+
+Verification: focused S3/HTTP/filesystem/config tests 34 passed; operational smoke 2 passed; core
+image with locked boto3 extra built; full harness 702 passed, 5 deselected; scoped Ruff and diff
+checks passed.
+
+Operational risk: custom S3-compatible endpoints remain part of the deployment trust boundary.
+Private endpoint access must be explicitly enabled and should be restricted by container egress
+policy in production. S3 list consistency semantics are provider-defined; version IDs are preferred
+over ETags wherever bucket versioning is available.