feat(preprocess): resume evidence jobs by run id

2026-07-12 04:39:32 +02:00
parent 981b149249
commit 1d5f8c76a7
7 changed files with 413 additions and 5 deletions
@@ -0,0 +1,43 @@
+# Evidence Task 5B implementation report
+
+## Status
+
+Integrated Evidence preprocessing with the Task 4 `JobRunner`. The CLI now accepts only a
+32-character JobRunner run ID for `--resume`; generation IDs remain outputs. Runs persist the
+exact ordered stages `discover`, `acquire_normalize_chunk`, `embed`, `vector_upsert`,
+`stage_validate`, `publish`, and `retention_cleanup`.
+
+Successful-stage artifacts are copied into the new resume run before execution, allowing later
+stages to continue without rediscovery, acquisition, normalization, chunking, or embedding.
+Job compatibility includes workspace, configuration, discovered-input, pipeline, embedding, and
+chunk-policy fingerprints. Generation-specific filesystem/vector compensation is retained, and a
+compensated generation is rotated before retry. `ACTIVE` is mutated only by `publish`.
+
+Dry-run executes discovery/planning and makes every side-effecting stage a no-op. JSON output is
+pristine and includes the JobRunner `run_id`, `resumed_from`, generation, plan, and publish status.
+
+## TDD evidence
+
+- RED: run-ID rejection and resume-artifact tests failed because generation IDs reached
+  configuration and resume runs had empty artifact directories.
+- GREEN: the two regression tests passed after strict CLI validation and durable artifact carryover.
+- Added pipeline job-plan and dry-run counting-fake coverage; both passed.
+
+## Fresh verification
+
+- Focused integration/search suite: `62 passed, 4 warnings`.
+- Available harness suite excluding sandbox-blocked Docker, loopback HTTP-server, and networked
+  wheel-build tests: `559 passed, 5 deselected, 18 warnings`.
+- Scoped Ruff: `All checks passed!`.
+- `git diff --check`: clean.
+
+## Environment limitations and concerns
+
+The literal full harness invocation cannot complete in the managed sandbox: Docker socket access,
+loopback HTTP test servers, and the `uv build` dependency resolution path are denied. It reached
+`575 passed, 5 deselected` before those environment errors. The available-suite rerun above is
+green.
+
+One pre-existing Pydantic serialization warning is exposed by the new end-to-end job test when
+canonical metadata contains frozen tuple values; it does not contaminate CLI stdout. Retention is
+an explicit stable no-op until a retention policy is configured.