176 lines
8.6 KiB
Markdown
176 lines
8.6 KiB
Markdown
# Final-review fix report — Evidence #43–#46
|
||
|
||
Date: 2026-08-25
|
||
|
||
Base ThothII revision: `d4818c8cc33b8b11204377ff3cc65c6c9ee1425e`
|
||
|
||
Binding inputs:
|
||
|
||
- requirements: `docs/plans/2026-08-24-evidence-restructuring.md`;
|
||
- approved design: `docs/plans/2026-08-24-evidence-restructuring-design.md`;
|
||
- final review: `.superpowers/sdd/2026-08-24-evidence-restructuring/final-review.md`.
|
||
|
||
No PSD path was read or mutated during this fix wave. No issue was closed.
|
||
|
||
## Verdict
|
||
|
||
All three Important findings are fixed. Candidate evaluation remains mandatory for schema-v2
|
||
filesystem curated corpora and is absent for legacy v1, HTTP-only, and S3-only acquisition.
|
||
Reviewed formula migration now fails closed unless the caller supplies the exact original source,
|
||
the source parses back to the same formula, and every supporting excerpt is present after the
|
||
canonical mechanical normalization. Current owner-gate summaries explicitly distinguish the
|
||
immutable 225-test owner-gate observation from the expanded 230-test final-review verification and
|
||
place stale 222/224-era statements under an explicitly superseded historical section.
|
||
|
||
## RED/GREEN record
|
||
|
||
### Finding 1 — evaluator compatibility boundary
|
||
|
||
RED:
|
||
|
||
```text
|
||
cd harness
|
||
.venv/bin/pytest -q tests/test_preprocess_cli.py tests/test_evidence_formula_migration.py \
|
||
tests/test_evidence_restructuring_fixture.py \
|
||
-k 'candidate_evaluation_is_not_required or runtime_identity'
|
||
6 failed, 18 deselected, 1 warning
|
||
```
|
||
|
||
The v1 filesystem run still received a callable evaluator; v1 HTTP/S3 and v2 HTTP/S3 attempted to
|
||
resolve a filesystem evaluation root before the pipeline could run.
|
||
|
||
GREEN:
|
||
|
||
```text
|
||
6 passed, 9 deselected, 1 warning
|
||
```
|
||
|
||
`_requires_candidate_evaluation()` is now the single boundary used by both curated-corpus
|
||
validation and evaluator construction. It returns true only for `schema_version: 2` with a
|
||
filesystem source (including an explicit filesystem `source_root`). The existing integrated v2
|
||
candidate test remains the positive proof: it validates the corpus, performs all nine
|
||
candidate-generation searches while inactive, publishes only after PASS, and compensates a failed
|
||
candidate.
|
||
|
||
### Finding 2 — formula provenance
|
||
|
||
RED:
|
||
|
||
```text
|
||
cd harness
|
||
.venv/bin/pytest -q tests/test_evidence_formula_migration.py \
|
||
tests/test_evidence_restructuring_fixture.py
|
||
5 failed, 4 passed, 1 warning
|
||
```
|
||
|
||
The converter rejected the new `source_content` argument, returned clean Curated Evidence when no
|
||
original source was supplied, and the documentation consistency assertion still failed.
|
||
|
||
GREEN:
|
||
|
||
```text
|
||
cd harness
|
||
.venv/bin/pytest -q tests/test_evidence_formula_migration.py tests/test_formula.py \
|
||
tests/test_formula_wiring.py tests/test_preprocess_cli.py \
|
||
tests/test_evidence_candidate_publication.py
|
||
39 passed, 1 warning
|
||
```
|
||
|
||
The migration now:
|
||
|
||
1. returns `None` unchanged for `auto` and `draft` formulas;
|
||
2. requires original source content for a reviewed formula;
|
||
3. normalizes it with the authoring validator's `normalize_source_text()`;
|
||
4. parses it and requires equality with the supplied `ConceptFormula`;
|
||
5. requires one or more legacy provenance notes and verifies each note in normalized source text;
|
||
6. hashes the verified normalized original source, never `formula.dump()`;
|
||
7. returns `LegacyFormulaMigrationFailure(code="legacy_formula_requires_manual_review")` with
|
||
bounded problem codes for missing, invalid, mismatched, or excerpt-incomplete provenance.
|
||
|
||
The regression matrix covers exact original formatting, deterministic digest, absent original
|
||
source, absent excerpts, a YAML-folded excerpt that cannot validate literally, mismatched original
|
||
formula content, invalid full-query Formula Evidence, stable path-derived IDs, and unreviewed
|
||
session-only behavior.
|
||
|
||
### Finding 3 — current owner-gate record
|
||
|
||
RED:
|
||
|
||
```text
|
||
cd harness
|
||
.venv/bin/pytest -q tests/test_evidence_restructuring_fixture.py \
|
||
-k current_summaries
|
||
1 failed, 3 deselected, 1 warning
|
||
```
|
||
|
||
The current report did not contain the fresh expanded-suite result. Earlier RED also showed that
|
||
the leading report still lacked an authoritative read-only PSD summary and `PROJECT_STATE.md`
|
||
still said 222.
|
||
|
||
GREEN:
|
||
|
||
```text
|
||
4 passed, 1 warning
|
||
```
|
||
|
||
The task report now begins with one authoritative current record and moves all earlier scope/count
|
||
statements below `Historical record — explicitly superseded`. The manual and `PROJECT_STATE.md`
|
||
record both immutable observations without conflating them: 225 passed at the reviewed `d4818c8`
|
||
owner gate; 230 passed after five final-review regressions entered the selected acceptance files.
|
||
Both summaries describe the authorized PSD package as read-only, with 35 moveable sources plus the
|
||
retained README, narrow no-payload/no-vector observations, no secrets, and no mutation. Issue #47
|
||
still owns migration and manual acceptance.
|
||
|
||
## Requirement mapping
|
||
|
||
| Requirement | Implementation evidence | Verification |
|
||
| --- | --- | --- |
|
||
| v1 filesystem remains viable without `evaluation.yaml` or a new curated tree | `_requires_candidate_evaluation()` returns false for schema v1; `run_from_config()` passes `None` | v1 runtime identity test and parameterized filesystem case |
|
||
| HTTP/S3 preserve existing behavior | evaluator construction returns `None` for HTTP-only/S3-only configs in schema v1 and v2 | four parameterized HTTP/S3 cases |
|
||
| v2 filesystem evaluation remains mandatory | shared predicate drives validation and evaluator; no permissive missing-fixture path was added | integrated inactive-candidate publication/failure-compensation test |
|
||
| formula digest represents verified normalized original | caller supplies `source_content`; authoring normalization computes digest | literal SHA-256 assertion from independently normalized original fixture |
|
||
| formula and source cannot silently mismatch | original source is parsed and compared with the supplied model | `original_source_mismatch` regression |
|
||
| supporting excerpts are real and nonempty | notes are required and each normalized note must occur in normalized source | missing and folded/unverifiable excerpt regressions |
|
||
| unverifiable migration requires human resolution | every provenance failure returns the existing bounded manual-review result | exact code/problem assertions preserve original path and formula |
|
||
| current acceptance record is unambiguous | authoritative current sections plus explicit historical supersession | owner-gate document consistency test |
|
||
| PSD scope remains read-only | no PSD tool/path used in fix; current docs retain issue #47 authorization gate | diff review and acceptance isolation probe |
|
||
|
||
## Verification evidence
|
||
|
||
```text
|
||
focused Evidence compatibility/formula/candidate: 39 passed, 1 warning
|
||
owner-gate document contract: 4 passed, 1 warning
|
||
harness full: 1093 passed, 4 deselected, 53 warnings
|
||
harness Ruff: All checks passed!
|
||
acceptance runner: 230 passed, 1 warning; PASS
|
||
backend TypeScript: PASS
|
||
native tht build: PASS
|
||
git diff --check: PASS
|
||
```
|
||
|
||
The full backend Vitest gate remains outside this patch's modified files and reported 13 failures:
|
||
the ten previously recorded auth-runtime projection failures, the recorded Node 25 versus Node 24
|
||
contract failure, the recorded Argon2 401/429 timing assertion, plus one Windows timeout-helper
|
||
marker failure that is timing-sensitive and was not part of the prior stable 12-failure baseline.
|
||
The native Go suite with `-timeout 20s` reproduced the recorded authconfig timeout and backup/setup
|
||
failures. These unrelated failures were not repaired or hidden; backend TypeScript and native build
|
||
both pass.
|
||
|
||
## SHA-256 artifact hashes
|
||
|
||
```text
|
||
193f9e83634c17160dc5f589ee392f6cefbca2e24efa0770181500e7dce008a7 PROJECT_STATE.md
|
||
4f9cfb213fdfbab481fcad9eeb0002ab0bd69d83d7ef0c459c5fccd5f5b44048 docs/testing/evidence-restructuring-manual.md
|
||
e47862d4bf9b846f059ef133b77a6311fbeb297671843bfeda0d0445eb2e4933 harness/tht/cli/preprocess_cmd.py
|
||
4017f66dc6d3aeaa4f0294bbca581b8d447e3b60bd8daff4efff08048dc43d4b harness/tht/evidence/formula_store.py
|
||
35bd73f6d0759cf0c8cb15a9125cc97b5dda64eb02417eda9dd1a9c82ed8a558 harness/tests/test_preprocess_cli.py
|
||
a0affbf9213c41c95d5fea8e596ef37ae21836e0df9bdfa8f6adda95d9238fb6 harness/tests/test_evidence_formula_migration.py
|
||
6ed7d7679c860c6edc54cbcfc2a04db89c7d32ecab80c7daa72be96cdc14f164 harness/tests/test_formula.py
|
||
26d886074244be7cf6ca084f859dcbc0c0ade74c302077a506198a2e96bfc07e harness/tests/test_evidence_restructuring_fixture.py
|
||
62743e885230823f8942d14feadae511b6d09b5d391807a04d10a164f71dd919 .superpowers/sdd/2026-08-24-evidence-restructuring/task-12-report.md
|
||
e753674bb55d4880448a51e7fbdb3ccf506800ecc5fd3b73f59ed655cf9b63c7 tracked working-tree diff before this report
|
||
```
|
||
|
||
The final commit hash is reported with the completed handoff because a commit cannot include its
|
||
own hash without changing itself.
|