fix(evidence): resolve final review findings
This commit is contained in:
@@ -0,0 +1,175 @@
|
||||
# Final-review fix report — Evidence #43–#46
|
||||
|
||||
Date: 2026-08-25
|
||||
|
||||
Base ThothII revision: `d4818c8cc33b8b11204377ff3cc65c6c9ee1425e`
|
||||
|
||||
Binding inputs:
|
||||
|
||||
- requirements: `docs/plans/2026-08-24-evidence-restructuring.md`;
|
||||
- approved design: `docs/plans/2026-08-24-evidence-restructuring-design.md`;
|
||||
- final review: `.superpowers/sdd/2026-08-24-evidence-restructuring/final-review.md`.
|
||||
|
||||
No PSD path was read or mutated during this fix wave. No issue was closed.
|
||||
|
||||
## Verdict
|
||||
|
||||
All three Important findings are fixed. Candidate evaluation remains mandatory for schema-v2
|
||||
filesystem curated corpora and is absent for legacy v1, HTTP-only, and S3-only acquisition.
|
||||
Reviewed formula migration now fails closed unless the caller supplies the exact original source,
|
||||
the source parses back to the same formula, and every supporting excerpt is present after the
|
||||
canonical mechanical normalization. Current owner-gate summaries explicitly distinguish the
|
||||
immutable 225-test owner-gate observation from the expanded 230-test final-review verification and
|
||||
place stale 222/224-era statements under an explicitly superseded historical section.
|
||||
|
||||
## RED/GREEN record
|
||||
|
||||
### Finding 1 — evaluator compatibility boundary
|
||||
|
||||
RED:
|
||||
|
||||
```text
|
||||
cd harness
|
||||
.venv/bin/pytest -q tests/test_preprocess_cli.py tests/test_evidence_formula_migration.py \
|
||||
tests/test_evidence_restructuring_fixture.py \
|
||||
-k 'candidate_evaluation_is_not_required or runtime_identity'
|
||||
6 failed, 18 deselected, 1 warning
|
||||
```
|
||||
|
||||
The v1 filesystem run still received a callable evaluator; v1 HTTP/S3 and v2 HTTP/S3 attempted to
|
||||
resolve a filesystem evaluation root before the pipeline could run.
|
||||
|
||||
GREEN:
|
||||
|
||||
```text
|
||||
6 passed, 9 deselected, 1 warning
|
||||
```
|
||||
|
||||
`_requires_candidate_evaluation()` is now the single boundary used by both curated-corpus
|
||||
validation and evaluator construction. It returns true only for `schema_version: 2` with a
|
||||
filesystem source (including an explicit filesystem `source_root`). The existing integrated v2
|
||||
candidate test remains the positive proof: it validates the corpus, performs all nine
|
||||
candidate-generation searches while inactive, publishes only after PASS, and compensates a failed
|
||||
candidate.
|
||||
|
||||
### Finding 2 — formula provenance
|
||||
|
||||
RED:
|
||||
|
||||
```text
|
||||
cd harness
|
||||
.venv/bin/pytest -q tests/test_evidence_formula_migration.py \
|
||||
tests/test_evidence_restructuring_fixture.py
|
||||
5 failed, 4 passed, 1 warning
|
||||
```
|
||||
|
||||
The converter rejected the new `source_content` argument, returned clean Curated Evidence when no
|
||||
original source was supplied, and the documentation consistency assertion still failed.
|
||||
|
||||
GREEN:
|
||||
|
||||
```text
|
||||
cd harness
|
||||
.venv/bin/pytest -q tests/test_evidence_formula_migration.py tests/test_formula.py \
|
||||
tests/test_formula_wiring.py tests/test_preprocess_cli.py \
|
||||
tests/test_evidence_candidate_publication.py
|
||||
39 passed, 1 warning
|
||||
```
|
||||
|
||||
The migration now:
|
||||
|
||||
1. returns `None` unchanged for `auto` and `draft` formulas;
|
||||
2. requires original source content for a reviewed formula;
|
||||
3. normalizes it with the authoring validator's `normalize_source_text()`;
|
||||
4. parses it and requires equality with the supplied `ConceptFormula`;
|
||||
5. requires one or more legacy provenance notes and verifies each note in normalized source text;
|
||||
6. hashes the verified normalized original source, never `formula.dump()`;
|
||||
7. returns `LegacyFormulaMigrationFailure(code="legacy_formula_requires_manual_review")` with
|
||||
bounded problem codes for missing, invalid, mismatched, or excerpt-incomplete provenance.
|
||||
|
||||
The regression matrix covers exact original formatting, deterministic digest, absent original
|
||||
source, absent excerpts, a YAML-folded excerpt that cannot validate literally, mismatched original
|
||||
formula content, invalid full-query Formula Evidence, stable path-derived IDs, and unreviewed
|
||||
session-only behavior.
|
||||
|
||||
### Finding 3 — current owner-gate record
|
||||
|
||||
RED:
|
||||
|
||||
```text
|
||||
cd harness
|
||||
.venv/bin/pytest -q tests/test_evidence_restructuring_fixture.py \
|
||||
-k current_summaries
|
||||
1 failed, 3 deselected, 1 warning
|
||||
```
|
||||
|
||||
The current report did not contain the fresh expanded-suite result. Earlier RED also showed that
|
||||
the leading report still lacked an authoritative read-only PSD summary and `PROJECT_STATE.md`
|
||||
still said 222.
|
||||
|
||||
GREEN:
|
||||
|
||||
```text
|
||||
4 passed, 1 warning
|
||||
```
|
||||
|
||||
The task report now begins with one authoritative current record and moves all earlier scope/count
|
||||
statements below `Historical record — explicitly superseded`. The manual and `PROJECT_STATE.md`
|
||||
record both immutable observations without conflating them: 225 passed at the reviewed `d4818c8`
|
||||
owner gate; 230 passed after five final-review regressions entered the selected acceptance files.
|
||||
Both summaries describe the authorized PSD package as read-only, with 35 moveable sources plus the
|
||||
retained README, narrow no-payload/no-vector observations, no secrets, and no mutation. Issue #47
|
||||
still owns migration and manual acceptance.
|
||||
|
||||
## Requirement mapping
|
||||
|
||||
| Requirement | Implementation evidence | Verification |
|
||||
| --- | --- | --- |
|
||||
| v1 filesystem remains viable without `evaluation.yaml` or a new curated tree | `_requires_candidate_evaluation()` returns false for schema v1; `run_from_config()` passes `None` | v1 runtime identity test and parameterized filesystem case |
|
||||
| HTTP/S3 preserve existing behavior | evaluator construction returns `None` for HTTP-only/S3-only configs in schema v1 and v2 | four parameterized HTTP/S3 cases |
|
||||
| v2 filesystem evaluation remains mandatory | shared predicate drives validation and evaluator; no permissive missing-fixture path was added | integrated inactive-candidate publication/failure-compensation test |
|
||||
| formula digest represents verified normalized original | caller supplies `source_content`; authoring normalization computes digest | literal SHA-256 assertion from independently normalized original fixture |
|
||||
| formula and source cannot silently mismatch | original source is parsed and compared with the supplied model | `original_source_mismatch` regression |
|
||||
| supporting excerpts are real and nonempty | notes are required and each normalized note must occur in normalized source | missing and folded/unverifiable excerpt regressions |
|
||||
| unverifiable migration requires human resolution | every provenance failure returns the existing bounded manual-review result | exact code/problem assertions preserve original path and formula |
|
||||
| current acceptance record is unambiguous | authoritative current sections plus explicit historical supersession | owner-gate document consistency test |
|
||||
| PSD scope remains read-only | no PSD tool/path used in fix; current docs retain issue #47 authorization gate | diff review and acceptance isolation probe |
|
||||
|
||||
## Verification evidence
|
||||
|
||||
```text
|
||||
focused Evidence compatibility/formula/candidate: 39 passed, 1 warning
|
||||
owner-gate document contract: 4 passed, 1 warning
|
||||
harness full: 1093 passed, 4 deselected, 53 warnings
|
||||
harness Ruff: All checks passed!
|
||||
acceptance runner: 230 passed, 1 warning; PASS
|
||||
backend TypeScript: PASS
|
||||
native tht build: PASS
|
||||
git diff --check: PASS
|
||||
```
|
||||
|
||||
The full backend Vitest gate remains outside this patch's modified files and reported 13 failures:
|
||||
the ten previously recorded auth-runtime projection failures, the recorded Node 25 versus Node 24
|
||||
contract failure, the recorded Argon2 401/429 timing assertion, plus one Windows timeout-helper
|
||||
marker failure that is timing-sensitive and was not part of the prior stable 12-failure baseline.
|
||||
The native Go suite with `-timeout 20s` reproduced the recorded authconfig timeout and backup/setup
|
||||
failures. These unrelated failures were not repaired or hidden; backend TypeScript and native build
|
||||
both pass.
|
||||
|
||||
## SHA-256 artifact hashes
|
||||
|
||||
```text
|
||||
193f9e83634c17160dc5f589ee392f6cefbca2e24efa0770181500e7dce008a7 PROJECT_STATE.md
|
||||
4f9cfb213fdfbab481fcad9eeb0002ab0bd69d83d7ef0c459c5fccd5f5b44048 docs/testing/evidence-restructuring-manual.md
|
||||
e47862d4bf9b846f059ef133b77a6311fbeb297671843bfeda0d0445eb2e4933 harness/tht/cli/preprocess_cmd.py
|
||||
4017f66dc6d3aeaa4f0294bbca581b8d447e3b60bd8daff4efff08048dc43d4b harness/tht/evidence/formula_store.py
|
||||
35bd73f6d0759cf0c8cb15a9125cc97b5dda64eb02417eda9dd1a9c82ed8a558 harness/tests/test_preprocess_cli.py
|
||||
a0affbf9213c41c95d5fea8e596ef37ae21836e0df9bdfa8f6adda95d9238fb6 harness/tests/test_evidence_formula_migration.py
|
||||
6ed7d7679c860c6edc54cbcfc2a04db89c7d32ecab80c7daa72be96cdc14f164 harness/tests/test_formula.py
|
||||
26d886074244be7cf6ca084f859dcbc0c0ade74c302077a506198a2e96bfc07e harness/tests/test_evidence_restructuring_fixture.py
|
||||
62743e885230823f8942d14feadae511b6d09b5d391807a04d10a164f71dd919 .superpowers/sdd/2026-08-24-evidence-restructuring/task-12-report.md
|
||||
e753674bb55d4880448a51e7fbdb3ccf506800ecc5fd3b73f59ed655cf9b63c7 tracked working-tree diff before this report
|
||||
```
|
||||
|
||||
The final commit hash is reported with the completed handoff because a commit cannot include its
|
||||
own hash without changing itself.
|
||||
Reference in New Issue
Block a user