46 lines
3.0 KiB
Markdown
46 lines
3.0 KiB
Markdown
# Evidence preprocessing Task 7 report
|
|
|
|
Implemented the S3-compatible Evidence adapter, explicit preprocessing Compose overlay, and
|
|
operational gates.
|
|
|
|
- S3 discovery uses bounded paginator pages, page size, and total objects; acquisition enforces a
|
|
byte ceiling and always closes streaming bodies.
|
|
- Provenance is canonical `s3://bucket/key`. Versioned objects use `s3-version:<version>`;
|
|
unversioned objects use a hashed exact ETag, and acquisition refuses validator drift.
|
|
- The adapter uses boto3/botocore rather than custom signing. TLS verification is enabled by
|
|
default. Custom HTTP and private endpoints require independent explicit opt-ins; endpoint
|
|
userinfo is rejected and public custom endpoints are DNS-policy checked.
|
|
- Access, secret, and session credentials support file-secret resolution into masked `SecretStr`
|
|
config fields. They are never emitted in provenance, reports, errors, or Compose environment.
|
|
- `deploy/compose.preprocess.yaml` provides separate one-shot Evidence and DWH jobs and is inert
|
|
unless explicitly included with the `preprocess` profile.
|
|
- `scripts/preprocess-smoke.sh` verifies both services render without secret material and pins an
|
|
unchanged rerun plus a modified generation through deterministic pipeline tests.
|
|
|
|
Verification: focused S3/HTTP/filesystem/config tests 34 passed; operational smoke 2 passed; core
|
|
image with locked boto3 extra built; full harness 702 passed, 5 deselected; scoped Ruff and diff
|
|
checks passed.
|
|
|
|
Operational risk: custom S3-compatible endpoints remain part of the deployment trust boundary.
|
|
Private endpoint access must be explicitly enabled and should be restricted by container egress
|
|
policy in production. S3 list consistency semantics are provider-defined; version IDs are preferred
|
|
over ETags wherever bucket versioning is available.
|
|
|
|
## Review correction
|
|
|
|
The Compose overlay now uses committed, purpose-built Evidence and DWH workspace files with
|
|
job-specific dependencies. Its services create their lock roots and mount only the vector secrets
|
|
they consume. The operational smoke is a real isolated Compose project: real pgvector migrations,
|
|
a deterministic in-project embeddings endpoint, actual Evidence CLI JSON across initial/unchanged/
|
|
mutated runs, exact ACTIVE verification, an actual DWH introspection job, and owned cleanup.
|
|
|
|
S3 custom endpoints now fail closed unless declared trusted; HTTP and private loopback endpoints
|
|
need additional independent opt-ins. Boto uses forced path-style addressing. Custom endpoints reject
|
|
userinfo, query, fragment, and non-root paths. Buckets use strict DNS syntax; listed keys must remain
|
|
under prefix and within the S3 byte bound; validators must be nonempty/bounded. Because
|
|
ListObjectsV2 does not provide version IDs, discovery honestly fingerprints the exact ETag and
|
|
acquisition rejects ETag drift.
|
|
|
|
Final correction verification: S3/config focused 20 passed; full harness 721 passed, 5 deselected;
|
|
real Compose smoke and image build passed; scoped Ruff, shell syntax, and diff checks passed.
|