fix(preprocess): harden crash recovery integrity

2026-07-12 04:49:15 +02:00
parent 1d5f8c76a7
commit c964920f16
7 changed files with 440 additions and 37 deletions
@@ -41,3 +41,33 @@ green.
 One pre-existing Pydantic serialization warning is exposed by the new end-to-end job test when
 canonical metadata contains frozen tuple values; it does not contaminate CLI stdout. Retention is
 an explicit stable no-op until a retention policy is configured.
+
+## Review fix wave — crash consistency and artifact integrity
+
+Addressed all five follow-up findings:
+
+- `JobRunner` now supports a test-only post-call/pre-checkpoint fault hook. Each stage seals a
+  canonical artifact manifest containing required flat filenames, SHA-256, byte size, producer
+  stage, and the full spec compatibility fingerprint. Resume validates the checkpoint and every
+  sealed artifact before allocating/copying a new run, rejecting missing, tampered, extra, nested,
+  or symlinked state. A sealed `running` stage is promoted after a simulated process crash; a
+  sealed `failed` stage is deliberately retried.
+- Vector intent (exact record IDs and content hashes) is sealed before upsert. Execution reconciles
+  `existing_hashes` and writes only missing/mismatched rows. Crash-after-effect tests prove no
+  duplicate acquire, embed, or vector upsert.
+- Raw upsert, stage, recovery-upsert, recovery-stage, and publish exceptions compensate the exact
+  generation. Compensation markers survive failed checkpoints; resume rotates the generation,
+  refreshes generation-bound artifacts, reconciles vectors, and stages idempotently.
+- `CorpusStore.publish` is idempotent and failure-atomic. If replace succeeds but directory fsync
+  fails, it restores the previous `ACTIVE` value (or removes a newly created pointer), fsyncs the
+  rollback, and re-raises. Pipeline cleanup refuses to discard a generation referenced by ACTIVE.
+- Added crash/resume coverage after all seven ordered stages; corrupt/missing plan, manifest, and
+  embeddings; unsafe extra paths; nonexistent run IDs; raw vector/stage failures; and post-replace
+  ACTIVE rollback.
+
+Fresh fix-wave verification:
+
+- Focused jobs/corpus/CLI/search suite: `82 passed, 17 warnings`.
+- Available harness suite (same sandbox exclusions described above):
+  `579 passed, 5 deselected, 31 warnings`.
+- Scoped Ruff and `git diff --check`: clean.