95 lines
5.4 KiB
Markdown
95 lines
5.4 KiB
Markdown
# Evidence Task 5B implementation report
|
|
|
|
## Status
|
|
|
|
Integrated Evidence preprocessing with the Task 4 `JobRunner`. The CLI now accepts only a
|
|
32-character JobRunner run ID for `--resume`; generation IDs remain outputs. Runs persist the
|
|
exact ordered stages `discover`, `acquire_normalize_chunk`, `embed`, `vector_upsert`,
|
|
`stage_validate`, `publish`, and `retention_cleanup`.
|
|
|
|
Successful-stage artifacts are copied into the new resume run before execution, allowing later
|
|
stages to continue without rediscovery, acquisition, normalization, chunking, or embedding.
|
|
Job compatibility includes workspace, configuration, discovered-input, pipeline, embedding, and
|
|
chunk-policy fingerprints. Generation-specific filesystem/vector compensation is retained, and a
|
|
compensated generation is rotated before retry. `ACTIVE` is mutated only by `publish`.
|
|
|
|
Dry-run executes discovery/planning and makes every side-effecting stage a no-op. JSON output is
|
|
pristine and includes the JobRunner `run_id`, `resumed_from`, generation, plan, and publish status.
|
|
|
|
## TDD evidence
|
|
|
|
- RED: run-ID rejection and resume-artifact tests failed because generation IDs reached
|
|
configuration and resume runs had empty artifact directories.
|
|
- GREEN: the two regression tests passed after strict CLI validation and durable artifact carryover.
|
|
- Added pipeline job-plan and dry-run counting-fake coverage; both passed.
|
|
|
|
## Fresh verification
|
|
|
|
- Focused integration/search suite: `62 passed, 4 warnings`.
|
|
- Available harness suite excluding sandbox-blocked Docker, loopback HTTP-server, and networked
|
|
wheel-build tests: `559 passed, 5 deselected, 18 warnings`.
|
|
- Scoped Ruff: `All checks passed!`.
|
|
- `git diff --check`: clean.
|
|
|
|
## Environment limitations and concerns
|
|
|
|
The literal full harness invocation cannot complete in the managed sandbox: Docker socket access,
|
|
loopback HTTP test servers, and the `uv build` dependency resolution path are denied. It reached
|
|
`575 passed, 5 deselected` before those environment errors. The available-suite rerun above is
|
|
green.
|
|
|
|
One pre-existing Pydantic serialization warning is exposed by the new end-to-end job test when
|
|
canonical metadata contains frozen tuple values; it does not contaminate CLI stdout. Retention is
|
|
an explicit stable no-op until a retention policy is configured.
|
|
|
|
## Review fix wave — crash consistency and artifact integrity
|
|
|
|
Addressed all five follow-up findings:
|
|
|
|
- `JobRunner` now supports a test-only post-call/pre-checkpoint fault hook. Each stage seals a
|
|
canonical artifact manifest containing required flat filenames, SHA-256, byte size, producer
|
|
stage, and the full spec compatibility fingerprint. Resume validates the checkpoint and every
|
|
sealed artifact before allocating/copying a new run, rejecting missing, tampered, extra, nested,
|
|
or symlinked state. A sealed `running` stage is promoted after a simulated process crash; a
|
|
sealed `failed` stage is deliberately retried.
|
|
- Vector intent (exact record IDs and content hashes) is sealed before upsert. Execution reconciles
|
|
`existing_hashes` and writes only missing/mismatched rows. Crash-after-effect tests prove no
|
|
duplicate acquire, embed, or vector upsert.
|
|
- Raw upsert, stage, recovery-upsert, recovery-stage, and publish exceptions compensate the exact
|
|
generation. Compensation markers survive failed checkpoints; resume rotates the generation,
|
|
refreshes generation-bound artifacts, reconciles vectors, and stages idempotently.
|
|
- `CorpusStore.publish` is idempotent and failure-atomic. If replace succeeds but directory fsync
|
|
fails, it restores the previous `ACTIVE` value (or removes a newly created pointer), fsyncs the
|
|
rollback, and re-raises. Pipeline cleanup refuses to discard a generation referenced by ACTIVE.
|
|
- Added crash/resume coverage after all seven ordered stages; corrupt/missing plan, manifest, and
|
|
embeddings; unsafe extra paths; nonexistent run IDs; raw vector/stage failures; and post-replace
|
|
ACTIVE rollback.
|
|
|
|
Fresh fix-wave verification:
|
|
|
|
- Focused jobs/corpus/CLI/search suite: `82 passed, 17 warnings`.
|
|
- Available harness suite (same sandbox exclusions described above):
|
|
`579 passed, 5 deselected, 31 warnings`.
|
|
- Scoped Ruff and `git diff --check`: clean.
|
|
|
|
## Final P1 fix — effect state and checkpoint-bound manifest roots
|
|
|
|
- Stage checkpoints now distinguish `intent` from `completed`. Vector intent is atomically sealed
|
|
and checkpointed before upsert. A process-level `BaseException` after a partial multi-record
|
|
write leaves the stage `running/intent`; resume never promotes it and instead reconciles
|
|
`existing_hashes`, writing only the missing records. The completed state is persisted only after
|
|
reconciliation returns successfully.
|
|
- Every stage now persists its completed artifact state while still `running`, before the
|
|
post-call fault hook. The checkpoint binds the SHA-256 of canonical `artifact-manifest.json`,
|
|
effect state, exact producer stage, and exact required-file mapping. Resume validates this root
|
|
and all bindings before promotion or copying.
|
|
- Added process-interruption coverage proving the already-written vector record is not submitted
|
|
twice, remaining records are written, and publish completes only after reconciliation. Added
|
|
coordinated artifact/manifest, spec-binding, and producer-binding tamper rejection tests.
|
|
|
|
Fresh verification:
|
|
|
|
- Focused jobs/corpus/CLI/search suite: `86 passed, 18 warnings`.
|
|
- Available broad harness suite: `583 passed, 5 deselected, 32 warnings`.
|
|
- Scoped Ruff and `git diff --check`: clean.
|