# Evidence / Preprocessing Task 1 Report

## Outcome

Implemented the additive Evidence source port and canonical corpus records. Existing evidence,
search, vector, and session runtime code is unchanged.

## Contract

- `EvidenceSource` is a runtime-checkable protocol with `discover` and `acquire` operations.
- `SourceObject` and `AcquiredDocument` are frozen, reject extra fields, use independent metadata
  defaults, and restrict metadata to Pydantic `JsonValue` values.
- `CanonicalDocument`, `CanonicalChunk`, and `CorpusManifest` are frozen and reject extra fields.
- Provenance includes stable source IDs, canonical URIs, fingerprints, modification time, and
  content hashes.
- Pipeline versions are recorded on documents, chunks, and manifests. Manifests also carry schema
  version, optional publish ID/vector generation, and paired embedding model/dimension fields.
- Credential-like metadata keys are rejected recursively. Credentials are not model fields and
  therefore cannot enter serialized canonical artifacts through extras.

## TDD evidence

The initial focused run failed during collection because `tht.ports.evidence` and `tht.corpus`
did not exist. After implementation, the focused suite passed.

## Verification

- Focused models/protocol tests: 13 passed.
- Harness excluding Docker-backed L0 and the network-dependent wheel packaging test: 444 passed,
  5 deselected.
- Focused Ruff: passed.
- Full-repository Ruff remains blocked by 34 pre-existing findings outside the task files.
- An unrestricted `pytest -q` attempt reached 453 passed and 5 deselected, but reported 47 Docker
  setup errors plus 4 Docker parity failures because the sandbox cannot access the Docker socket;
  the wheel packaging test also failed because its isolated `uv build` needs unavailable network.

## Concerns / follow-up

- Pydantic's `frozen=True` prevents model field reassignment but does not recursively freeze list
  and dict contents. `default_factory` prevents shared mutable defaults. Later pipeline stages should
  treat these value objects as immutable and construct replacements rather than mutate collections.
- The adapter and normalization tasks should preserve the credential-free boundary by passing only
  these records beyond acquisition.

## Review hardening follow-up

All six binding review areas were addressed in a separate TDD pass:

- JSON metadata is recursively converted to immutable `FrozenDict`/tuple values while retaining
  stable object/array JSON serialization. Manifest document and chunk collections are tuples.
- Secret-key matching now normalizes camelCase and punctuation. It rejects credential-specific
  names (passwords, API keys, access/refresh tokens, client/private keys, session cookies and
  authorization) recursively, while deliberate benign labels such as generic `token` and `secret`
  remain valid.
- Canonical URIs require a scheme and reject userinfo or credential-bearing query parameters.
- Namespaced IDs, SHA-256 content hashes, timezone-aware UTC timestamps, embedding/vector
  compatibility, unique IDs, chunk referential/provenance integrity, contiguous per-document
  ordinals and pipeline-version consistency are validated. Nested Pydantic instances are always
  revalidated so `model_copy(update=...)` cannot bypass a manifest boundary.
- Acquired arbitrary bytes have explicit base64 JSON encoding and validation, covered by a JSON
  round-trip test.
- `EvidenceSourceError` classifies transient/retryable versus permanent failures and exposes only
  recursively immutable, credential-screened JSON details.

Follow-up verification:

- Focused contract suite: 39 passed.
- Focused Ruff: passed.
- Harness excluding Docker-backed L0 and the network-dependent wheel packaging test: 470 passed,
  5 deselected.
- Fresh unrestricted harness attempt: 479 passed, 5 deselected; the same environmental boundary
  remains (47 Docker socket setup errors, four Docker parity failures, one isolated `uv build`
  network failure).

## Final blocker follow-up

The remaining four contract blockers were closed in a third TDD cycle:

- `EvidenceSourceError` now always exposes the fixed public message/`args` value `evidence source
  operation failed`; caller diagnostics are not retained. Category, details and args cannot be
  reassigned, details remain recursively frozen and credential-screened, and an original exception
  is available only when callers use standard exception chaining.
- Canonical document/chunk provenance stores only URI scheme, authority and path. Userinfo is
  rejected; query strings and fragments are removed unconditionally, including AWS `X-Amz-*`, SAS
  `sig`, and fragment token material.
- Binding model bases override Pydantic's unchecked `model_copy(update=...)`: merged values always
  pass full field/model validation, so invalid copied records and top-level manifests fail.
- A canonical document/chunk `content_hash` must equal SHA-256 of the exact stored text encoded as
  UTF-8. This establishes the normalization boundary explicitly: line-ending/frontmatter/text
  normalization happens before model construction; the canonical models never rewrite content.

Final follow-up verification:

- Focused contract suite: 45 passed.
- Focused Ruff: passed.
- Harness excluding Docker-backed L0 and network-dependent packaging: 476 passed, 5 deselected.
- Fresh unrestricted harness attempt: 486 passed, 5 deselected, with the unchanged environmental
  failures (47 Docker setup errors, four Docker parity failures, one isolated `uv build` failure).