65 lines
4.4 KiB
Markdown
65 lines
4.4 KiB
Markdown
# Evidence preprocessing Task 7 report
|
|
|
|
Implemented the S3-compatible Evidence adapter, explicit preprocessing Compose overlay, and
|
|
operational gates.
|
|
|
|
- S3 discovery uses bounded paginator pages, page size, and total objects; acquisition enforces a
|
|
byte ceiling and always closes streaming bodies.
|
|
- Provenance is canonical `s3://bucket/key`. Versioned objects use `s3-version:<version>`;
|
|
unversioned objects use a hashed exact ETag, and acquisition refuses validator drift.
|
|
- The adapter uses boto3/botocore rather than custom signing. TLS verification is enabled by
|
|
default. Custom HTTP and private endpoints require independent explicit opt-ins; endpoint
|
|
userinfo is rejected and public custom endpoints are DNS-policy checked.
|
|
- Access, secret, and session credentials support file-secret resolution into masked `SecretStr`
|
|
config fields. They are never emitted in provenance, reports, errors, or Compose environment.
|
|
- `deploy/compose.preprocess.yaml` provides separate one-shot Evidence and DWH jobs and is inert
|
|
unless explicitly included with the `preprocess` profile.
|
|
- `scripts/preprocess-smoke.sh` verifies both services render without secret material and pins an
|
|
unchanged rerun plus a modified generation through deterministic pipeline tests.
|
|
|
|
Verification: focused S3/HTTP/filesystem/config tests 34 passed; operational smoke 2 passed; core
|
|
image with locked boto3 extra built; full harness 702 passed, 5 deselected; scoped Ruff and diff
|
|
checks passed.
|
|
|
|
Operational risk: custom S3-compatible endpoints remain part of the deployment trust boundary.
|
|
Private endpoint access must be explicitly enabled and should be restricted by container egress
|
|
policy in production. S3 list consistency semantics are provider-defined; version IDs are preferred
|
|
over ETags wherever bucket versioning is available.
|
|
|
|
## Review correction
|
|
|
|
The Compose overlay now uses committed, purpose-built Evidence and DWH workspace files with
|
|
job-specific dependencies. Its services create their lock roots and mount only the vector secrets
|
|
they consume. The operational smoke is a real isolated Compose project: real pgvector migrations,
|
|
a deterministic in-project embeddings endpoint, actual Evidence CLI JSON across initial/unchanged/
|
|
mutated runs, exact ACTIVE verification, an actual DWH introspection job, and owned cleanup.
|
|
|
|
S3 custom endpoints now fail closed unless declared trusted; HTTP and private loopback endpoints
|
|
need additional independent opt-ins. Boto uses forced path-style addressing. Custom endpoints reject
|
|
userinfo, query, fragment, and non-root paths. Buckets use strict DNS syntax; listed keys must remain
|
|
under prefix and within the S3 byte bound; validators must be nonempty/bounded. Because
|
|
ListObjectsV2 does not provide version IDs, discovery honestly fingerprints the exact ETag and
|
|
acquisition rejects ETag drift.
|
|
|
|
Final correction verification: S3/config focused 20 passed; full harness 721 passed, 5 deselected;
|
|
real Compose smoke and image build passed; scoped Ruff, shell syntax, and diff checks passed.
|
|
|
|
## Final security review correction
|
|
|
|
Literal non-global IPv4/IPv6 endpoints now require the private-endpoint opt-in without claiming DNS
|
|
pinning for hostnames. Pagination uses explicit continuation requests and never fetches page
|
|
`max_pages + 1`. IP-shaped buckets, leading-slash prefixes, empty/overlong/control-character keys,
|
|
and absent validators fail closed. Acquisition accepts only the exact stored `SourceObject` and
|
|
compares the response ETag with the stored discovery validator. The real smoke snapshots generation
|
|
directory counts after every run and has an injected-failure cleanup mode; cleanup fails if Compose
|
|
down fails or any owned container, volume, or network remains.
|
|
|
|
The canonical smoke correction counts only root-level `corpus/gen-<32 hex>` directories. It exposed
|
|
that the durable job path still published an empty unchanged generation; the pipeline now returns
|
|
the existing ACTIVE generation without staging a directory when compatibility and all source
|
|
fingerprints are unchanged. The smoke therefore proves directory deltas `+1`, `+0`, `+1`.
|
|
Failure injection runs a real exit-97 command after resources exist and reaches the EXIT trap.
|
|
Cleanup aggregates Compose-down, residual container/volume/network, and temp-directory failures
|
|
while preserving the original failure status. S3 prefixes are validated before any client request
|
|
for leading slash, UTF-8 byte length, controls, and DEL.
|