94 lines
6.3 KiB
Markdown
94 lines
6.3 KiB
Markdown
# Evidence preprocessing Task 7 report
|
|
|
|
Implemented the S3-compatible Evidence adapter, explicit preprocessing Compose overlay, and
|
|
operational gates.
|
|
|
|
- S3 discovery uses bounded paginator pages, page size, and total objects; acquisition enforces a
|
|
byte ceiling and always closes streaming bodies.
|
|
- Provenance is canonical `s3://bucket/key`. Versioned objects use `s3-version:<version>`;
|
|
unversioned objects use a hashed exact ETag, and acquisition refuses validator drift.
|
|
- The adapter uses boto3/botocore rather than custom signing. TLS verification is enabled by
|
|
default. Custom HTTP and private endpoints require independent explicit opt-ins; endpoint
|
|
userinfo is rejected and public custom endpoints are DNS-policy checked.
|
|
- Access, secret, and session credentials support file-secret resolution into masked `SecretStr`
|
|
config fields. They are never emitted in provenance, reports, errors, or Compose environment.
|
|
- `deploy/compose.preprocess.yaml` provides separate one-shot Evidence and DWH jobs and is inert
|
|
unless explicitly included with the `preprocess` profile.
|
|
- `scripts/preprocess-smoke.sh` verifies both services render without secret material and pins an
|
|
unchanged rerun plus a modified generation through deterministic pipeline tests.
|
|
|
|
Verification: focused S3/HTTP/filesystem/config tests 34 passed; operational smoke 2 passed; core
|
|
image with locked boto3 extra built; full harness 702 passed, 5 deselected; scoped Ruff and diff
|
|
checks passed.
|
|
|
|
Operational risk: custom S3-compatible endpoints remain part of the deployment trust boundary.
|
|
Private endpoint access must be explicitly enabled and should be restricted by container egress
|
|
policy in production. S3 list consistency semantics are provider-defined; version IDs are preferred
|
|
over ETags wherever bucket versioning is available.
|
|
|
|
## Review correction
|
|
|
|
The Compose overlay now uses committed, purpose-built Evidence and DWH workspace files with
|
|
job-specific dependencies. Its services create their lock roots and mount only the vector secrets
|
|
they consume. The operational smoke is a real isolated Compose project: real pgvector migrations,
|
|
a deterministic in-project embeddings endpoint, actual Evidence CLI JSON across initial/unchanged/
|
|
mutated runs, exact ACTIVE verification, an actual DWH introspection job, and owned cleanup.
|
|
|
|
S3 custom endpoints now fail closed unless declared trusted; HTTP and private loopback endpoints
|
|
need additional independent opt-ins. Boto uses forced path-style addressing. Custom endpoints reject
|
|
userinfo, query, fragment, and non-root paths. Buckets use strict DNS syntax; listed keys must remain
|
|
under prefix and within the S3 byte bound; validators must be nonempty/bounded. Because
|
|
ListObjectsV2 does not provide version IDs, discovery honestly fingerprints the exact ETag and
|
|
acquisition rejects ETag drift.
|
|
|
|
Final correction verification: S3/config focused 20 passed; full harness 721 passed, 5 deselected;
|
|
real Compose smoke and image build passed; scoped Ruff, shell syntax, and diff checks passed.
|
|
|
|
## Final security review correction
|
|
|
|
Literal non-global IPv4/IPv6 endpoints now require the private-endpoint opt-in without claiming DNS
|
|
pinning for hostnames. Pagination uses explicit continuation requests and never fetches page
|
|
`max_pages + 1`. IP-shaped buckets, leading-slash prefixes, empty/overlong/control-character keys,
|
|
and absent validators fail closed. Acquisition accepts only the exact stored `SourceObject` and
|
|
compares the response ETag with the stored discovery validator. The real smoke snapshots generation
|
|
directory counts after every run and has an injected-failure cleanup mode; cleanup fails if Compose
|
|
down fails or any owned container, volume, or network remains.
|
|
|
|
The canonical smoke correction counts only root-level `corpus/gen-<32 hex>` directories. It exposed
|
|
that the durable job path still published an empty unchanged generation; the pipeline now returns
|
|
the existing ACTIVE generation without staging a directory when compatibility and all source
|
|
fingerprints are unchanged. The smoke therefore proves directory deltas `+1`, `+0`, `+1`.
|
|
Failure injection runs a real exit-97 command after resources exist and reaches the EXIT trap.
|
|
Cleanup aggregates Compose-down, residual container/volume/network, and temp-directory failures
|
|
while preserving the original failure status. S3 prefixes are validated before any client request
|
|
for leading slash, UTF-8 byte length, controls, and DEL.
|
|
|
|
## Canonical unchanged-run correction
|
|
|
|
The durable job now persists a deterministic source snapshot keyed by source identity. Each entry
|
|
binds canonical URI, exact source fingerprint, UTC modification time, canonical immutable metadata,
|
|
and explicit media type and size contract fields. The manifest also binds document-to-source
|
|
provenance, supplied config/input fingerprints, compatibility, embedding settings, and pipeline and
|
|
chunk-policy versions.
|
|
|
|
An unchanged run reuses ACTIVE only when ownership, bindings, the complete snapshot, document
|
|
provenance, materialized document hashes, and every required vector ID/content hash match exactly.
|
|
Snapshot changes rebuild only the affected sources; job input/config changes publish a new manifest
|
|
while retaining valid stable vector-generation dependencies. Missing or corrupt legacy contract
|
|
metadata, documents, or vectors fails closed and rebuilds. The Compose smoke now explicitly expects
|
|
the unchanged no-op to report `published=false` while proving generation deltas `+1`, `+0`, `+1`.
|
|
|
|
## Corrupt ACTIVE reconstruction correction
|
|
|
|
ACTIVE reuse now reconstructs each source contract from the persisted discovery snapshot and checks
|
|
the deterministic document identity, canonical URI, source fingerprint, UTC modification time,
|
|
source metadata, applicable media type, content hash, and pipeline identity against the owned
|
|
materialized document. The persisted document-source map carries the same exact binding.
|
|
|
|
Chunks are recomputed under the current chunk policy and must match the manifest exactly in count,
|
|
order, IDs, ordinals, content, hashes, linkage, provenance, and policy metadata. Vector health must
|
|
report the configured dimension, and every recomputed chunk must have its generation-scoped vector
|
|
ID with the exact content hash. Missing, altered, or extra chunks and corrupt document or vector
|
|
contracts therefore disable the no-op and rebuild, while a valid unchanged run still performs no
|
|
source acquisition.
|