1327 lines
48 KiB
Markdown
1327 lines
48 KiB
Markdown
# Evidence Restructuring Implementation Plan
|
||
|
||
> **Execution:** GitHub issue #35 is the approved parent specification. Issues #36–#47
|
||
> are the executable tracer-bullet tickets; implement one unblocked ticket at a time
|
||
> with `/implement`. This document remains the detailed technical reference and must
|
||
> not be executed as a second, parallel work queue.
|
||
|
||
**Goal:** Build a Git-reviewed, typed Evidence authoring pipeline and publish its approved output to the existing revision-scoped Qdrant lifecycle with dense+BM25 hybrid retrieval.
|
||
|
||
**Architecture:** Keep authoring outside NL→SQL sessions: deterministic preparation wraps one read-only Pi restructuring call, writes reviewable Markdown into the workspace repository, and blocks publication on unresolved review items. Reuse the current Evidence module, corpus generations, rollback, active-revision checks, and workspace-owned semantic collection; extend them instead of introducing a parallel store.
|
||
|
||
**Tech Stack:** Python 3.12, Pydantic 2, Typer, PyYAML, sqlglot, pytest, Pi CLI, TypeScript, Fastify workspace maintenance, Vitest, Qdrant 1.18.2 Query API, server-side `qdrant/bm25`, Git.
|
||
|
||
---
|
||
|
||
## Preconditions
|
||
|
||
- Work in this dedicated worktree.
|
||
- Read `PROJECT_STATE.md`, `CONTEXT.md`,
|
||
`docs/plans/2026-08-24-evidence-restructuring-design.md`,
|
||
`docs/contracts/workspace-evidence-v3.md`, and
|
||
`docs/contracts/workspace-preprocessing-cli.md`.
|
||
- Preserve the public facade in `harness/tht/evidence/__init__.py`.
|
||
- Do not edit `harness/.pi/skills/tht-sessione/SKILL.md` directly; regenerate it with
|
||
`python -m tht.pi_skill_projection --write`.
|
||
- Keep runtime workspace access read-only. Only the authoring CLI may write
|
||
`evidence/curated/` and `evidence/manifest.yaml` in a curator clone.
|
||
- Do not migrate the external PSD repository until all code and contract gates pass.
|
||
|
||
### Task 1: Add the typed Curated Evidence model
|
||
|
||
**Files:**
|
||
|
||
- Create: `harness/tht/evidence/canonical.py`
|
||
- Modify: `harness/tht/evidence/__init__.py`
|
||
- Create: `harness/tests/test_evidence_canonical.py`
|
||
|
||
**Step 1: Write failing tests for the common envelope**
|
||
|
||
Cover:
|
||
|
||
- all eight `kind` values;
|
||
- all four `purpose` values;
|
||
- immutable provenance;
|
||
- strict unknown-field rejection;
|
||
- kind-independent stable IDs;
|
||
- SHA-256 syntax;
|
||
- directory/`kind` agreement;
|
||
- parsing and dumping Markdown with YAML frontmatter.
|
||
- review items containing a stable `code`, human `message` and optional `field`, with no
|
||
status or history fields;
|
||
- IDs matching `evidence:<slug>` and remaining independent from `kind`.
|
||
|
||
Start with:
|
||
|
||
```python
|
||
def test_formula_requires_formula_payload():
|
||
with pytest.raises(ValidationError):
|
||
CuratedEvidence.model_validate({
|
||
**COMMON,
|
||
"kind": "formula",
|
||
"payload": {"concept": "fascia pediatrica"},
|
||
})
|
||
|
||
|
||
def test_reference_rejects_formula_payload():
|
||
with pytest.raises(ValidationError):
|
||
CuratedEvidence.model_validate({
|
||
**COMMON,
|
||
"kind": "reference",
|
||
"payload": {
|
||
"concept": "x",
|
||
"columns": ["clinical.patient.birth_date"],
|
||
"sql": "CASE WHEN true THEN 1 END",
|
||
},
|
||
})
|
||
```
|
||
|
||
**Step 2: Run the focused tests and confirm RED**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
cd harness
|
||
.venv/bin/pytest tests/test_evidence_canonical.py -q
|
||
```
|
||
|
||
Expected: import failure for `tht.evidence.canonical`.
|
||
|
||
**Step 3: Implement the discriminated model**
|
||
|
||
Use a strict Pydantic model with these public types:
|
||
|
||
```python
|
||
EvidenceKind = Literal[
|
||
"glossary", "domain", "enum", "example", "mapping",
|
||
"normalization", "formula", "reference",
|
||
]
|
||
EvidencePurpose = Literal[
|
||
"disambiguation", "rewriting", "schema_linking", "sql_generation",
|
||
]
|
||
|
||
class EvidenceScope(StrictModel):
|
||
concepts: tuple[str, ...] = ()
|
||
tables: tuple[str, ...] = ()
|
||
columns: tuple[str, ...] = ()
|
||
|
||
class EvidenceProvenance(StrictModel):
|
||
source_file: str
|
||
source_sha256: str
|
||
supporting_excerpts: tuple[str, ...]
|
||
|
||
class FormulaPayload(StrictModel):
|
||
concept: str
|
||
columns: tuple[str, ...]
|
||
sql: str
|
||
|
||
class ReviewItem(StrictModel):
|
||
code: str
|
||
message: str
|
||
field: str | None = None
|
||
|
||
class ReferencePayload(StrictModel):
|
||
url: AnyHttpUrl
|
||
label: str
|
||
description: str
|
||
```
|
||
|
||
Define equally strict payloads for the other six kinds and expose a single
|
||
`CuratedEvidence` API. The implementation may use an internal Pydantic discriminated
|
||
union, but callers must not switch between eight unrelated loaders.
|
||
|
||
Add:
|
||
|
||
```python
|
||
def parse_curated_markdown(text: str, *, path: Path | None = None) -> CuratedEvidence: ...
|
||
def dump_curated_markdown(value: CuratedEvidence) -> str: ...
|
||
def load_curated_tree(root: Path) -> list[CuratedEvidence]: ...
|
||
```
|
||
|
||
Validate formula SQL with `sqlglot` as one PostgreSQL expression, rejecting complete
|
||
`SELECT`/`WITH` statements, DDL, DML and multiple statements. Validate `schema.table` /
|
||
`schema.table.column` identifiers without querying the DWH.
|
||
Require one to five supporting excerpts, each nonempty and at most 1,000 characters.
|
||
|
||
**Step 4: Export only the stable API**
|
||
|
||
Add the model and loader functions to `harness/tht/evidence/__init__.py`. Do not export
|
||
internal union member helpers unless another module needs them.
|
||
|
||
**Step 5: Run tests and lint**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
cd harness
|
||
.venv/bin/pytest tests/test_evidence_canonical.py tests/test_evidence_facade_contract.py -q
|
||
.venv/bin/ruff check tht/evidence/canonical.py tests/test_evidence_canonical.py
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
**Step 6: Commit**
|
||
|
||
```bash
|
||
git add harness/tht/evidence/canonical.py harness/tht/evidence/__init__.py \
|
||
harness/tests/test_evidence_canonical.py
|
||
git commit -m "feat(evidence): add typed curated evidence model"
|
||
```
|
||
|
||
### Task 2: Add deterministic validation and the Evidence manifest
|
||
|
||
**Files:**
|
||
|
||
- Create: `harness/tht/evidence/authoring.py`
|
||
- Create: `harness/tests/test_evidence_authoring.py`
|
||
- Modify: `harness/tht/evidence/__init__.py`
|
||
|
||
**Step 1: Write failing tests for publication validation**
|
||
|
||
Test that:
|
||
|
||
- `review_items != []` is valid as a draft but blocks publication;
|
||
- `orphans != []` is preserved but blocks publication;
|
||
- a missing or mismatched source hash blocks publication;
|
||
- every supporting excerpt is found after applying the same mechanical normalization
|
||
to the excerpt and its source;
|
||
- two units cannot share an ID;
|
||
- one unit cannot claim two source files;
|
||
- `curated/formula/x.md` must contain `kind: formula`;
|
||
- credentials in URLs, YAML, or body are rejected;
|
||
- only Markdown, `.txt`, and `.sql.md` sources are accepted;
|
||
- UTF-8 and per-file limits are enforced.
|
||
|
||
Expose errors as bounded structured values:
|
||
|
||
```python
|
||
@dataclass(frozen=True)
|
||
class ValidationFinding:
|
||
severity: Literal["error", "warning"]
|
||
code: str
|
||
path: str
|
||
message: str
|
||
```
|
||
|
||
**Step 2: Write failing tests for the versioned manifest**
|
||
|
||
Use this minimum shape:
|
||
|
||
```yaml
|
||
schema_version: 1
|
||
pipeline_version: evidence-authoring-v1
|
||
sources:
|
||
source/domain/patient.md:
|
||
sha256: sha256:...
|
||
units:
|
||
- domain:patient
|
||
orphans: []
|
||
```
|
||
|
||
Test deterministic key ordering, stable round-trip, unknown fields, duplicate unit IDs,
|
||
orphan preservation and incompatible `pipeline_version` refusal.
|
||
|
||
**Step 3: Run and confirm RED**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
cd harness
|
||
.venv/bin/pytest tests/test_evidence_authoring.py -q
|
||
```
|
||
|
||
Expected: missing authoring API.
|
||
|
||
**Step 4: Implement `EvidenceManifest` and `validate_workspace_evidence`**
|
||
|
||
Provide:
|
||
|
||
```python
|
||
def load_manifest(path: Path) -> EvidenceManifest: ...
|
||
def dump_manifest(manifest: EvidenceManifest) -> str: ...
|
||
def validate_workspace_evidence(workspace_root: Path) -> ValidationReport: ...
|
||
```
|
||
|
||
`ValidationReport.publishable` is true only when there are no errors and no unresolved
|
||
review items or orphaned units. Warnings alone do not block publication.
|
||
|
||
Do not use status fields such as `draft/reviewed` as an approval mechanism. The approved
|
||
Git revision is the publication boundary.
|
||
|
||
**Step 5: Run tests and lint**
|
||
|
||
```bash
|
||
cd harness
|
||
.venv/bin/pytest tests/test_evidence_authoring.py tests/test_evidence_canonical.py -q
|
||
.venv/bin/ruff check tht/evidence/authoring.py tests/test_evidence_authoring.py
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
**Step 6: Commit**
|
||
|
||
```bash
|
||
git add harness/tht/evidence/authoring.py harness/tht/evidence/__init__.py \
|
||
harness/tests/test_evidence_authoring.py
|
||
git commit -m "feat(evidence): validate curated corpus and manifest"
|
||
```
|
||
|
||
### Task 3: Implement incremental preparation with one Pi restructuring call
|
||
|
||
**Files:**
|
||
|
||
- Modify: `harness/tht/evidence/authoring.py`
|
||
- Create: `harness/.pi/skills/tht-evidence-authoring/SKILL.md`
|
||
- Modify: `harness/tests/test_evidence_authoring.py`
|
||
- Create: `harness/tests/test_evidence_pi_restructurer.py`
|
||
|
||
**Step 1: Define the restructuring port and request/response models**
|
||
|
||
Add:
|
||
|
||
```python
|
||
class EvidenceRestructurer(Protocol):
|
||
def restructure(self, request: RestructureRequest) -> tuple[RestructureCandidate, ...]: ...
|
||
|
||
class RestructureRequest(StrictModel):
|
||
source_file: str
|
||
source_sha256: str
|
||
normalized_text: str
|
||
previous_units: tuple[CuratedEvidence, ...] = ()
|
||
|
||
class RestructureCandidate(StrictModel):
|
||
existing_id: str | None = None
|
||
# The same title, kind, purposes, scope, provenance excerpts and typed payload
|
||
# needed to construct CuratedEvidence, but no model-assigned canonical ID.
|
||
```
|
||
|
||
`existing_id`, when present, must belong to `previous_units`. The preparer rejects any
|
||
unknown ID and deterministically allocates `evidence:<slug>` plus a collision suffix for
|
||
every candidate without one. The response is converted to and validated through the
|
||
models from Task 1 before any write.
|
||
|
||
**Step 2: Write RED tests for the incremental rules**
|
||
|
||
Test:
|
||
|
||
- unchanged source: no model call and no file write;
|
||
- changed source: exactly one model call;
|
||
- new source: new stable IDs;
|
||
- removed source: old units become orphans and remain on disk;
|
||
- uniquely renamed source with the same hash: provenance changes and unit IDs remain;
|
||
- existing source no longer supporting a prior unit: retain it with
|
||
`source_no_longer_supports_unit` and block publication;
|
||
- kind-only reclassification: preserve the unit ID;
|
||
- semantic split: allocate new IDs for the new independent units;
|
||
- a semantic split retains the previous unit as a blocking retirement candidate until
|
||
the curator explicitly retires it;
|
||
- one source may produce several units;
|
||
- every returned unit contains one to five exact supporting excerpts found in its
|
||
normalized source;
|
||
- no returned unit may cite another source;
|
||
- prior curated units are included in the request;
|
||
- a dirty `evidence/curated/` or `evidence/manifest.yaml` fails before the model call;
|
||
- committed human edits remain recoverable in Git and every proposed change is exposed
|
||
in the working-tree diff;
|
||
- all output writes are staged and atomically replaced only after complete validation.
|
||
- one invalid source leaves the complete batch and manifest unchanged;
|
||
|
||
Inject the Git status runner and filesystem writer in tests; do not require a real Git
|
||
repository for every unit test.
|
||
|
||
**Step 3: Implement deterministic source normalization**
|
||
|
||
Normalize UTF-8 text with NFC, LF newlines and terminal newline. Preserve meaningful
|
||
Markdown, tables, fenced SQL, URLs and list structure. Do not rewrite vocabulary or
|
||
infer domain facts in this step.
|
||
|
||
**Step 4: Implement `PiEvidenceRestructurer`**
|
||
|
||
Invoke Pi as an ephemeral, no-tools process using an argument list, never a shell:
|
||
|
||
```python
|
||
argv = [
|
||
pi_executable,
|
||
"--mode", "text",
|
||
"--print",
|
||
"--no-session",
|
||
"--no-tools",
|
||
"--no-extensions",
|
||
"--no-context-files",
|
||
"--skill", str(skill_path),
|
||
f"@{request_path}",
|
||
"Return only the JSON object required by the Evidence authoring skill.",
|
||
]
|
||
```
|
||
|
||
Use a private `TemporaryDirectory`, a bounded timeout, bounded stdout/stderr, and strict
|
||
JSON parsing. Do not forward the model's raw output to public JSON errors. The skill must
|
||
state:
|
||
|
||
- use only facts present in `normalized_text`;
|
||
- preserve prior reviewed wording where it remains supported;
|
||
- retain unsupported prior units with a `source_no_longer_supports_unit` review item;
|
||
- never merge sources;
|
||
- emit `review_items` for uncertainty;
|
||
- copy short exact supporting excerpts from `normalized_text`;
|
||
- use only a supplied `existing_id` or omit it for deterministic allocation;
|
||
- emit exactly the schema-versioned JSON object and no Markdown fence.
|
||
|
||
Do not retry automatically after a timeout, nonzero exit, malformed JSON or invalid
|
||
response. Return a stable error code and the affected source so the curator can rerun
|
||
the command explicitly.
|
||
|
||
**Step 5: Implement `prepare_workspace_evidence`**
|
||
|
||
Return a bounded report with `changed`, `unchanged`, `created`, `orphaned`, `findings`,
|
||
and `model_calls`. Keep the model implementation behind `EvidenceRestructurer`. Build
|
||
the complete batch in a private staging directory and replace curated files plus the
|
||
manifest only after every candidate validates; never expose partial success.
|
||
|
||
**Step 6: Run tests**
|
||
|
||
```bash
|
||
cd harness
|
||
.venv/bin/pytest tests/test_evidence_authoring.py tests/test_evidence_pi_restructurer.py -q
|
||
.venv/bin/ruff check tht/evidence/authoring.py tests/test_evidence_pi_restructurer.py
|
||
```
|
||
|
||
Expected: PASS with no live model call.
|
||
|
||
**Step 7: Commit**
|
||
|
||
```bash
|
||
git add harness/tht/evidence/authoring.py \
|
||
harness/.pi/skills/tht-evidence-authoring/SKILL.md \
|
||
harness/tests/test_evidence_authoring.py harness/tests/test_evidence_pi_restructurer.py
|
||
git commit -m "feat(evidence): prepare curated evidence incrementally"
|
||
```
|
||
|
||
### Task 4: Add the authoring CLI
|
||
|
||
**Files:**
|
||
|
||
- Create: `harness/tht/cli/evidence_cmd.py`
|
||
- Modify: `harness/tht/cli/__init__.py`
|
||
- Create: `harness/tests/test_evidence_cli.py`
|
||
|
||
**Step 1: Write CLI grammar tests**
|
||
|
||
Cover:
|
||
|
||
```text
|
||
tht evidence prepare <workspace-root> [--upgrade] [--json]
|
||
tht evidence validate <workspace-root> [--json]
|
||
tht evidence resolve <workspace-root> <evidence-id> --retire [--json]
|
||
tht evidence resolve <workspace-root> <evidence-id> --source <source-path> [--json]
|
||
```
|
||
|
||
Require an existing canonical Git worktree root. Reject unknown flags, symlinks, a
|
||
workspace path outside the Git root, duplicate options and dirty curated state. A
|
||
pipeline-version mismatch must fail without writes unless `--upgrade` is explicit;
|
||
`--upgrade` reprocesses every source. Ensure `--json` writes pristine JSON to stdout.
|
||
For `resolve`, require exactly one of `--retire` and `--source`, an existing Evidence ID,
|
||
an existing in-root source path for relinking and a clean worktree. Test atomic updates
|
||
to the curated file and manifest, with no commit or publication side effect.
|
||
|
||
**Step 2: Run and confirm RED**
|
||
|
||
```bash
|
||
cd harness
|
||
.venv/bin/pytest tests/test_evidence_cli.py -q
|
||
```
|
||
|
||
Expected: `evidence` command group is unknown.
|
||
|
||
**Step 3: Implement the Typer group**
|
||
|
||
Use one top-level authoring group:
|
||
|
||
```python
|
||
evidence_app = typer.Typer(help="Prepare and validate workspace Evidence")
|
||
|
||
@evidence_app.command("prepare")
|
||
def prepare_cmd(workspace_root: Path, json_output: bool = False) -> None: ...
|
||
|
||
@evidence_app.command("validate")
|
||
def validate_cmd(workspace_root: Path, json_output: bool = False) -> None: ...
|
||
|
||
@evidence_app.command("resolve")
|
||
def resolve_cmd(
|
||
workspace_root: Path,
|
||
evidence_id: str,
|
||
retire: bool = False,
|
||
source: Path | None = None,
|
||
json_output: bool = False,
|
||
) -> None: ...
|
||
```
|
||
|
||
Register it in `harness/tht/cli/__init__.py`. Keep this separate from the existing
|
||
runtime command `tht preprocess evidence`.
|
||
|
||
Exit codes:
|
||
|
||
- `0`: prepared/unchanged or valid;
|
||
- `2`: unsafe path or CLI misuse;
|
||
- `3`: valid drafts but review required;
|
||
- `1`: operational/model/structural failure.
|
||
|
||
**Step 4: Run tests and CLI help**
|
||
|
||
```bash
|
||
cd harness
|
||
.venv/bin/pytest tests/test_evidence_cli.py tests/test_preprocess_cli.py -q
|
||
.venv/bin/tht evidence --help
|
||
.venv/bin/tht preprocess evidence --help
|
||
```
|
||
|
||
Expected: both command families are present and unambiguous.
|
||
|
||
**Step 5: Commit**
|
||
|
||
```bash
|
||
git add harness/tht/cli/evidence_cmd.py harness/tht/cli/__init__.py \
|
||
harness/tests/test_evidence_cli.py
|
||
git commit -m "feat(evidence): expose prepare and validate commands"
|
||
```
|
||
|
||
### Task 5: Project typed units into semantic Evidence Fragments
|
||
|
||
**Files:**
|
||
|
||
- Modify: `harness/tht/evidence/corpus/models.py`
|
||
- Modify: `harness/tht/evidence/corpus/chunk.py`
|
||
- Modify: `harness/tht/evidence/corpus/normalize.py`
|
||
- Modify: `harness/tht/evidence/corpus/pipeline.py`
|
||
- Modify: `harness/tht/evidence/preprocessing.py`
|
||
- Modify: `harness/tests/test_corpus_models.py`
|
||
- Modify: `harness/tests/test_corpus_chunk.py`
|
||
- Modify: `harness/tests/test_corpus_pipeline.py`
|
||
|
||
**Step 1: Write failing projection tests**
|
||
|
||
Test that:
|
||
|
||
- a short formula remains one fragment;
|
||
- a domain document splits only at semantic section boundaries;
|
||
- enum entries are not split in the middle of a value/meaning pair;
|
||
- formulas, value/meaning pairs, mappings, rules and URLs are never split;
|
||
- the complete rendered fragment, including labels and textual metadata, uses the
|
||
existing `max_chunk_chars` setting whose default is 4,000 characters;
|
||
- an atomic element over `max_chunk_chars` produces the blocking
|
||
`atomic_content_too_large` review item instead of fixed-size chunks, with no second
|
||
size setting;
|
||
- all fragments carry `evidence_id`, `evidence_kind`, `purposes`, scope, language and
|
||
provenance;
|
||
- fragment IDs and ordinals are deterministic;
|
||
- only `curated/**/*.md` is accepted as canonical filesystem content.
|
||
|
||
**Step 2: Extend the immutable corpus models**
|
||
|
||
Keep `CanonicalDocument` and `CanonicalChunk` as transport-neutral storage models. Put
|
||
typed Evidence metadata in their already-safe `metadata` field, with exact allowlisted
|
||
keys. Do not make corpus storage depend on Pydantic subtype classes at read time.
|
||
|
||
**Step 3: Implement kind-aware fragment rendering**
|
||
|
||
Construct embedding text from title, purpose, scope and type-specific data. Example for
|
||
a formula:
|
||
|
||
```text
|
||
Formula: Fascia pediatrica
|
||
Concept: fascia pediatrica
|
||
Columns: clinical.patient.birth_date
|
||
SQL: CASE WHEN ... END
|
||
Limitations: ...
|
||
```
|
||
|
||
The rendered text is derived; provenance and canonical content remain in the corpus
|
||
manifest.
|
||
|
||
**Step 4: Run focused pipeline tests**
|
||
|
||
```bash
|
||
cd harness
|
||
.venv/bin/pytest tests/test_corpus_models.py tests/test_corpus_chunk.py \
|
||
tests/test_corpus_normalize.py tests/test_corpus_pipeline.py \
|
||
tests/test_corpus_publish.py -q
|
||
```
|
||
|
||
Expected: PASS, including existing rollback, compensation, retention and resume tests.
|
||
|
||
**Step 5: Commit**
|
||
|
||
```bash
|
||
git add harness/tht/evidence/corpus harness/tht/evidence/preprocessing.py \
|
||
harness/tests/test_corpus_models.py harness/tests/test_corpus_chunk.py \
|
||
harness/tests/test_corpus_normalize.py harness/tests/test_corpus_pipeline.py
|
||
git commit -m "feat(evidence): build semantic fragments from typed units"
|
||
```
|
||
|
||
### Task 6: Add BM25 to the existing Qdrant collection without rebuilding it
|
||
|
||
**Files:**
|
||
|
||
- Modify: `backend/src/workspaces/qdrant-collection.ts`
|
||
- Modify: `backend/src/workspaces/evidence/preprocessing.ts`
|
||
- Modify: `backend/test/qdrant-collection.test.ts`
|
||
- Modify: `backend/test/workspaces/evidence/preprocessing.test.ts`
|
||
- Modify: `harness/tht/adapters/vector/qdrant.py`
|
||
- Modify: `harness/tests/test_qdrant_vector_store.py`
|
||
- Modify: `docs/contracts/workspace-preprocessing-cli.md`
|
||
|
||
**Step 1: Write RED TypeScript collection-contract tests**
|
||
|
||
The required vector contract is:
|
||
|
||
```json
|
||
{
|
||
"vectors": {"size": 1024, "distance": "Cosine"},
|
||
"sparse_vectors": {
|
||
"bm25": {"modifier": "idf"}
|
||
}
|
||
}
|
||
```
|
||
|
||
Keep the current unnamed dense vector. Test three distinct states:
|
||
|
||
- compatible: unnamed dense is correct and `bm25` has `modifier: idf`;
|
||
- Evidence-upgradeable: unnamed dense is correct and `bm25` is absent;
|
||
- incompatible: dense dimension/distance is wrong, dense is unexpectedly named, or
|
||
`bm25` exists with a different configuration.
|
||
|
||
Only the Evidence maintenance path may turn the upgradeable state into compatible by
|
||
adding `bm25`. Session admission and searches remain read-only. Missing payload indexes
|
||
may still use the existing additive reconciliation; no path may delete or rename a
|
||
vector.
|
||
|
||
Add keyword indexes only for fields used by filters:
|
||
|
||
```text
|
||
content_hash, document_id, kind, record_key, record_kind,
|
||
vector_generation, workspace_id, workspace_revision,
|
||
evidence_id, evidence_kind, purposes, concepts, tables, columns, language
|
||
```
|
||
|
||
**Step 2: Run the TypeScript test and confirm RED**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run test/qdrant-collection.test.ts
|
||
```
|
||
|
||
Expected: the new additive BM25 expectations fail.
|
||
|
||
**Step 3: Implement additive BM25 reconciliation**
|
||
|
||
Keep base `vectorCompatibility` concerned with the unnamed dense contract used by
|
||
Schema and Memory. Add an Evidence-specific compatibility result that also classifies
|
||
`bm25` as compatible, upgradeable or incompatible. Update `createCollection` and the
|
||
Evidence preprocessing preflight: a new collection is created with unnamed dense plus
|
||
`bm25`; on an existing upgradeable collection, `workspace preprocess evidence` calls
|
||
Qdrant's additive vector-schema endpoint to create only `bm25` with IDF, then verifies
|
||
the result before upload. Retain `require_existing` behavior for ordinary runtime
|
||
validation: it never mutates. An incompatible state fails without changes. Missing
|
||
BM25 does not make Schema or Memory unavailable; only Evidence reports `unavailable`.
|
||
|
||
**Step 4: Update the Python adapter's collection validation**
|
||
|
||
`QdrantVectorStore.health()` and `_ensure_collection()` must recognize exactly the same
|
||
contract as TypeScript. Add a shared test fixture shape even though the two languages do
|
||
not share implementation code.
|
||
|
||
**Step 5: Run backend and harness tests**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run test/qdrant-collection.test.ts test/workspaces/evidence/preprocessing.test.ts
|
||
npx tsc --noEmit -p .
|
||
cd ../harness
|
||
.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -q
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
**Step 6: Document the additive maintenance behavior**
|
||
|
||
Update the CLI contract to state that `workspace preprocess evidence` may add the
|
||
missing `bm25` definition but may not delete or rename vectors. The existing destructive
|
||
`workspace vector rebuild` command remains available for unrelated operator recovery
|
||
and is not used by this migration.
|
||
|
||
**Step 7: Commit**
|
||
|
||
```bash
|
||
git add backend/src/workspaces/qdrant-collection.ts \
|
||
backend/src/workspaces/evidence/preprocessing.ts \
|
||
backend/test/qdrant-collection.test.ts backend/test/workspaces/evidence/preprocessing.test.ts \
|
||
harness/tht/adapters/vector/qdrant.py \
|
||
harness/tests/test_qdrant_vector_store.py docs/contracts/workspace-preprocessing-cli.md
|
||
git commit -m "feat(evidence): add qdrant bm25 vector in place"
|
||
```
|
||
|
||
### Task 7: Add server-side BM25 ingestion and hybrid Query API retrieval
|
||
|
||
**Files:**
|
||
|
||
- Modify: `harness/tht/ports/vector.py`
|
||
- Modify: `harness/tht/adapters/vector/qdrant.py`
|
||
- Modify: `harness/tht/vectorstore/records.py`
|
||
- Modify: `harness/tht/evidence/corpus/pipeline.py`
|
||
- Modify: `harness/tests/test_vector_port_contract.py`
|
||
- Modify: `harness/tests/test_qdrant_vector_store.py`
|
||
- Modify: `harness/tests/test_corpus_pipeline.py`
|
||
- Create: `harness/tests/l0/test_qdrant_bm25_inference.py`
|
||
|
||
**Step 1: Write RED port tests**
|
||
|
||
Extend, do not replace, the current record:
|
||
|
||
```python
|
||
@dataclass(frozen=True)
|
||
class VectorWriteRecord:
|
||
record: VectorRecord
|
||
embedding: list[float]
|
||
content_hash: str
|
||
sparse_text: str | None = None
|
||
sparse_language: str | None = None
|
||
```
|
||
|
||
Extend `VectorStore.search` with keyword-only `query_text` and `query_language`. Existing
|
||
callers that omit them remain dense-only.
|
||
|
||
**Step 2: Write exact Qdrant request tests**
|
||
|
||
For Evidence upsert, assert:
|
||
|
||
```json
|
||
"vector": {
|
||
"": [0.1, 0.2],
|
||
"bm25": {
|
||
"text": "...",
|
||
"model": "qdrant/bm25",
|
||
"options": {"language": "italian"}
|
||
}
|
||
}
|
||
```
|
||
|
||
For hybrid search, assert two filtered prefetches and default RRF:
|
||
|
||
```json
|
||
{
|
||
"prefetch": [
|
||
{"query": [0.1, 0.2], "limit": 20, "filter": {}},
|
||
{
|
||
"query": {
|
||
"text": "fascia pediatrica",
|
||
"model": "qdrant/bm25",
|
||
"options": {"language": "italian"}
|
||
},
|
||
"using": "bm25",
|
||
"limit": 20,
|
||
"filter": {}
|
||
}
|
||
],
|
||
"query": {"rrf": {}},
|
||
"limit": 10,
|
||
"with_payload": true
|
||
}
|
||
```
|
||
|
||
Both prefetch filters must include workspace, revision, active generation and record
|
||
kind. Do not add hand-tuned weights.
|
||
|
||
**Step 3: Prove BM25 on the actual local Qdrant image**
|
||
|
||
Add an `l0` test that reads the Qdrant image reference from the root `compose.yaml`,
|
||
starts that exact image with testcontainers, creates a uniquely named temporary
|
||
collection, adds `bm25` with IDF, indexes two Italian texts through server-side
|
||
`qdrant/bm25`, retrieves the expected text and deletes the collection. The test must
|
||
not use FastEmbed or accept a dense fallback. It fails if the Compose reference and the
|
||
tested image diverge.
|
||
|
||
**Step 4: Preserve dense writes for all existing semantic records**
|
||
|
||
Schema, Memory and solved-question records keep their current unnamed dense writes.
|
||
Evidence records use the empty default-vector name plus `bm25` in the mixed-vector
|
||
upsert shape. Only records with `sparse_text` receive `bm25`; no full reindex of Schema
|
||
or Memory is performed.
|
||
|
||
**Step 5: Implement BM25 Evidence ingestion**
|
||
|
||
Populate `sparse_text` and map workspace language `it` to Qdrant's `italian`. Reject an
|
||
unsupported language before uploading the generation. Use the same options at ingest
|
||
and query time.
|
||
|
||
**Step 6: Implement hybrid search with dense fallback only for non-Evidence callers**
|
||
|
||
An Evidence hybrid request must fail as unavailable if the configured Qdrant version or
|
||
collection contract does not support BM25. It must not silently claim to have run hybrid
|
||
search. Existing non-Evidence dense requests continue to work.
|
||
|
||
**Step 7: Run focused tests**
|
||
|
||
```bash
|
||
cd harness
|
||
.venv/bin/pytest tests/test_vector_port_contract.py tests/test_qdrant_vector_store.py \
|
||
tests/test_corpus_pipeline.py tests/test_search_pack.py -q
|
||
.venv/bin/pytest tests/l0/test_qdrant_bm25_inference.py -m l0 -q
|
||
.venv/bin/ruff check tht/ports/vector.py tht/adapters/vector/qdrant.py \
|
||
tht/evidence/corpus/pipeline.py
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
**Step 8: Commit**
|
||
|
||
```bash
|
||
git add harness/tht/ports/vector.py harness/tht/adapters/vector/qdrant.py \
|
||
harness/tht/vectorstore/records.py harness/tht/evidence/corpus/pipeline.py \
|
||
harness/tests/test_vector_port_contract.py harness/tests/test_qdrant_vector_store.py \
|
||
harness/tests/test_corpus_pipeline.py harness/tests/l0/test_qdrant_bm25_inference.py
|
||
git commit -m "feat(evidence): add qdrant bm25 hybrid retrieval"
|
||
```
|
||
|
||
### Task 8: Add the typed Evidence search contract and workflow-owned purposes
|
||
|
||
**Files:**
|
||
|
||
- Modify: `harness/tht/evidence/search.py`
|
||
- Modify: `harness/tht/evidence/__init__.py`
|
||
- Modify: `harness/tht/evidence/session.py`
|
||
- Modify: `harness/tht/cli/search_cmd.py`
|
||
- Modify: `harness/tht/session/filesystem_repository.py`
|
||
- Modify: `harness/tht/session/postgres_repository.py`
|
||
- Modify: `harness/tests/test_evidence_facade_contract.py`
|
||
- Create: `harness/tests/test_evidence_session_receipts.py`
|
||
- Modify: `harness/tests/test_search_pack.py`
|
||
- Create: `harness/.pi/skills/tht-sessione/modules/evidence/runtime-search.md`
|
||
- Modify: `harness/.pi/skills/tht-sessione/projection.md.tmpl`
|
||
- Modify: `harness/tht/pi_skill_projection.py`
|
||
- Modify: `harness/tests/test_pi_skill_projection.py`
|
||
- Regenerate: `harness/.pi/skills/tht-sessione/SKILL.md`
|
||
|
||
**Step 1: Write RED search-facade tests**
|
||
|
||
Introduce:
|
||
|
||
```python
|
||
class EvidenceSearchContext(BaseModel):
|
||
concepts: tuple[str, ...] = ()
|
||
tables: tuple[str, ...] = ()
|
||
columns: tuple[str, ...] = ()
|
||
required_kinds: tuple[EvidenceKind, ...] = ()
|
||
required_concepts: tuple[str, ...] = ()
|
||
required_tables: tuple[str, ...] = ()
|
||
required_columns: tuple[str, ...] = ()
|
||
|
||
def search_evidence(
|
||
query: str,
|
||
purpose: EvidencePurpose,
|
||
context: EvidenceSearchContext,
|
||
*,
|
||
searcher: ActiveEvidenceSearcher,
|
||
embedder: EvidenceQueryEmbedder,
|
||
top_n: int = 10,
|
||
) -> EvidenceSearchOutcome: ...
|
||
```
|
||
|
||
Test hard filters for workspace, revision, generation, purpose and every explicit
|
||
`required_*` constraint. Concepts, tables and columns without the `required_` prefix
|
||
enrich the dense and BM25 query text but do not create payload filters. Do not add
|
||
implicit kind or scope bonuses in v1.
|
||
|
||
Render that enrichment once, identically for both retrieval branches, in this fixed
|
||
order:
|
||
|
||
```text
|
||
Domanda: <original query>
|
||
Concetti: <normalized, deduplicated, sorted values>
|
||
Tabelle: <normalized, deduplicated, sorted values>
|
||
Colonne: <normalized, deduplicated, sorted values>
|
||
```
|
||
|
||
Omit empty lines and do not rewrite the original question. Test that permutations and
|
||
duplicates in context produce the same rendered text and that the dense embedder and
|
||
BM25 request receive exactly that same value.
|
||
|
||
The renderer normalizes the question to Unicode NFC, converts CRLF and CR to `\n`,
|
||
strips only leading and trailing whitespace and rejects an empty result. It preserves
|
||
case, punctuation and internal whitespace. Apply NFC and `strip` to context values,
|
||
remove empty strings and exact duplicates, then sort by Unicode value. Do not call
|
||
`lower()` or `casefold()` because quoted PostgreSQL identifiers can be case-sensitive.
|
||
|
||
`EvidenceSearchOutcome` has an `available` state with a generation and zero or more
|
||
results, and an `unavailable` state with a stable error code, a bounded message and no
|
||
results. `memory` is not an `EvidencePurpose`: the `memory` stage remains owned by the
|
||
Memory Module.
|
||
|
||
**Step 2: Test fragment grouping**
|
||
|
||
Two returned fragments with the same `evidence_id` must become one `EvidenceResult`,
|
||
with the best score, ordered matching excerpts, canonical citation, a reference for
|
||
resolving the full document and no duplicate unit. Do not insert the full document into
|
||
the search pack automatically.
|
||
|
||
**Step 3: Distinguish an empty search from technical unavailability**
|
||
|
||
Test a successful search with zero matches separately from absent ACTIVE corpus,
|
||
revision mismatch, unavailable Qdrant and malformed payload. The first returns
|
||
`available` with an empty result list and may continue. Every technical case returns
|
||
`unavailable`, blocks the calling stage until retry, and never uses stale rows or a
|
||
fallback purpose.
|
||
|
||
**Step 4: Implement the facade and CLI mapping**
|
||
|
||
Keep Qdrant syntax inside `tht.evidence`. `search_cmd.py` translates command inputs to
|
||
the facade and renders results; it must not duplicate ranking logic.
|
||
|
||
**Step 5: Integrate the contributor with semantic stages**
|
||
|
||
Move the shared Evidence rules from the projection template into
|
||
`modules/evidence/runtime-search.md`. The fragment must state:
|
||
|
||
- candidates are not truth;
|
||
- pass the phase-appropriate purpose;
|
||
- show provenance;
|
||
- formulas are `kind=formula`, not a separate search store;
|
||
- an available empty result is visible and does not block the stage;
|
||
- an unavailable outcome blocks the stage and is retriable;
|
||
- Evidence never writes decisions, canonical artifacts or workflow state.
|
||
|
||
Map semantic stages, independently of their display codes:
|
||
|
||
```text
|
||
clarification -> disambiguation
|
||
rewriting -> rewriting
|
||
schema_linking -> schema_linking
|
||
cte -> sql_generation
|
||
final_sql -> sql_generation
|
||
```
|
||
|
||
Do not invoke Evidence from `memory` or `synthesis`. Run an independent search in every
|
||
mapped stage; the `final_sql` query includes the approved CTE plan.
|
||
|
||
**Step 6: Persist the minimal Evidence receipt**
|
||
|
||
Use the existing session artifact repositories to maintain one
|
||
`evidence_receipts.json` artifact. For every available search, append or replace the
|
||
receipt identified by semantic stage with exactly `stage`, `purpose`,
|
||
`vector_generation` and ordered `evidence_ids`. Do not copy excerpts or complete
|
||
Evidence text. The workflow integration owns the write; `tht.evidence` only constructs
|
||
and returns the typed receipt. Test filesystem and PostgreSQL repositories, resume and
|
||
retry replacement.
|
||
|
||
Register the fragment in the static `FRAGMENT_ORDER` and regenerate.
|
||
|
||
**Step 7: Run tests**
|
||
|
||
```bash
|
||
cd harness
|
||
python -m tht.pi_skill_projection --write
|
||
python -m tht.pi_skill_projection --check
|
||
.venv/bin/pytest tests/test_evidence_facade_contract.py tests/test_search_pack.py \
|
||
tests/test_evidence_session_receipts.py \
|
||
tests/test_pi_skill_projection.py -q
|
||
```
|
||
|
||
Expected: PASS and no direct edit drift in generated `SKILL.md`.
|
||
|
||
**Step 7: Commit**
|
||
|
||
```bash
|
||
git add harness/tht/evidence/search.py harness/tht/evidence/__init__.py \
|
||
harness/tht/cli/search_cmd.py harness/tests/test_evidence_facade_contract.py \
|
||
harness/tests/test_search_pack.py \
|
||
harness/.pi/skills/tht-sessione/modules/evidence/runtime-search.md \
|
||
harness/.pi/skills/tht-sessione/projection.md.tmpl \
|
||
harness/.pi/skills/tht-sessione/SKILL.md harness/tht/pi_skill_projection.py \
|
||
harness/tests/test_pi_skill_projection.py
|
||
git commit -m "refactor(evidence): own typed runtime retrieval"
|
||
```
|
||
|
||
### Task 9: Migrate formulas into Curated Evidence
|
||
|
||
**Files:**
|
||
|
||
- Modify: `harness/tht/evidence/formula_store.py`
|
||
- Modify: `harness/tht/evidence/session.py`
|
||
- Modify: `harness/tht/cli/search_cmd.py`
|
||
- Create: `harness/.pi/skills/tht-sessione/modules/evidence/formula-proposals.md`
|
||
- Modify: `harness/.pi/skills/tht-sessione/projection.md.tmpl`
|
||
- Modify: `harness/tht/pi_skill_projection.py`
|
||
- Modify: `harness/tests/test_formula.py`
|
||
- Modify: `harness/tests/test_formula_wiring.py`
|
||
- Create: `harness/tests/test_evidence_formula_migration.py`
|
||
- Modify: `harness/tests/test_pi_skill_projection.py`
|
||
- Regenerate: `harness/.pi/skills/tht-sessione/SKILL.md`
|
||
|
||
**Step 1: Write RED migration tests**
|
||
|
||
Map an approved legacy `ConceptFormula` to a Curated Evidence formula while preserving:
|
||
|
||
- concept;
|
||
- SQL;
|
||
- columns;
|
||
- sources as provenance notes;
|
||
- stable deterministic ID;
|
||
- reviewed content wording.
|
||
|
||
Reject `auto` and unresolved `draft` formulas from direct publication; they become
|
||
Formula proposals.
|
||
|
||
**Step 2: Define the session proposal contract**
|
||
|
||
Add a schema-versioned formula-proposal projection in the session artifact. Keep
|
||
`concept_formula_approved` / `concept_formula_rejected` decisions unchanged because they
|
||
record a session-local choice, not repository publication.
|
||
|
||
**Step 3: Remove the separate runtime formula lookup**
|
||
|
||
Change `tht search find --kind formula` to call typed Evidence search with
|
||
`required_kinds=("formula",)`. Keep the legacy store readable only for the migration
|
||
command/window, with a deprecation warning in human output and no warning leakage into
|
||
pristine JSON.
|
||
|
||
**Step 4: Add and project formula instructions**
|
||
|
||
The Evidence fragment must say that a newly synthesized formula is a session proposal
|
||
and cannot be treated as Published Evidence.
|
||
|
||
**Step 5: Run tests**
|
||
|
||
```bash
|
||
cd harness
|
||
python -m tht.pi_skill_projection --write
|
||
.venv/bin/pytest tests/test_formula.py tests/test_formula_wiring.py \
|
||
tests/test_evidence_formula_migration.py tests/test_pi_skill_projection.py \
|
||
tests/test_decision_min_phase.py tests/test_workflow_observable_contract.py -q
|
||
```
|
||
|
||
Expected: PASS; decision phase ownership remains F4.
|
||
|
||
**Step 6: Commit**
|
||
|
||
```bash
|
||
git add harness/tht/evidence/formula_store.py harness/tht/evidence/session.py \
|
||
harness/tht/cli/search_cmd.py \
|
||
harness/.pi/skills/tht-sessione/modules/evidence/formula-proposals.md \
|
||
harness/.pi/skills/tht-sessione/projection.md.tmpl \
|
||
harness/.pi/skills/tht-sessione/SKILL.md harness/tht/pi_skill_projection.py \
|
||
harness/tests/test_formula.py harness/tests/test_formula_wiring.py \
|
||
harness/tests/test_evidence_formula_migration.py harness/tests/test_pi_skill_projection.py
|
||
git commit -m "refactor(evidence): unify formulas with typed evidence"
|
||
```
|
||
|
||
### Task 10: Add the small retrieval evaluation command
|
||
|
||
**Files:**
|
||
|
||
- Create: `harness/tht/evidence/evaluation.py`
|
||
- Modify: `harness/tht/cli/evidence_cmd.py`
|
||
- Create: `harness/tests/test_evidence_evaluation.py`
|
||
- Modify: `harness/tests/test_evidence_cli.py`
|
||
|
||
**Step 1: Write RED schema tests**
|
||
|
||
Use a deliberately small format:
|
||
|
||
```yaml
|
||
schema_version: 1
|
||
queries:
|
||
- id: pediatric-formula
|
||
query: Come distinguo i pazienti pediatrici?
|
||
profile: semantic
|
||
purpose: sql_generation
|
||
expected:
|
||
- evidence:fascia-pediatrica
|
||
```
|
||
|
||
Require unique query IDs, nonempty expected IDs, one of `lexical`, `semantic` or
|
||
`mixed` for every profile, only public purpose values and at least one query of every
|
||
profile in the complete file.
|
||
|
||
**Step 2: Write RED metric tests**
|
||
|
||
Compute `hit_at_5`, `hit_at_10`, missing expected IDs, empty-result queries and counts by
|
||
expected `kind`. For each expected ID, also report its nullable dense-only, BM25-only
|
||
and fused rank. The evaluator runs both branches separately for diagnosis and the same
|
||
hybrid request used by runtime. The report passes only when every query finds at least
|
||
one expected ID in its first ten fused results; branch ranks and `hit_at_5` are
|
||
informative. Do not add nDCG, relevance grading or an evaluation database in v1.
|
||
|
||
**Step 3: Implement the evaluator and CLI**
|
||
|
||
Expose:
|
||
|
||
```text
|
||
tht evidence evaluate <workspace-root> -c <runtime-config>
|
||
[--generation <id>] [--json]
|
||
```
|
||
|
||
The report includes workspace revision, evaluated vector generation and the fixed
|
||
default RRF configuration. It is read-only. With no `--generation`, it evaluates the
|
||
active generation for monitoring. Before publication, the preprocessing pipeline calls
|
||
the same evaluator against its candidate generation and atomically activates it only
|
||
when the report passes. A failed candidate remains inactive.
|
||
|
||
**Step 4: Run tests**
|
||
|
||
```bash
|
||
cd harness
|
||
.venv/bin/pytest tests/test_evidence_evaluation.py tests/test_evidence_cli.py -q
|
||
.venv/bin/ruff check tht/evidence/evaluation.py tests/test_evidence_evaluation.py
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
**Step 5: Commit**
|
||
|
||
```bash
|
||
git add harness/tht/evidence/evaluation.py harness/tht/cli/evidence_cmd.py \
|
||
harness/tests/test_evidence_evaluation.py harness/tests/test_evidence_cli.py
|
||
git commit -m "feat(evidence): evaluate retrieval with a small fixture"
|
||
```
|
||
|
||
### Task 11: Enforce curated-only runtime ingestion and update contracts
|
||
|
||
**Files:**
|
||
|
||
- Modify: `backend/src/workspaces/schema.ts`
|
||
- Modify: `backend/src/workspaces/runtime-renderer.ts`
|
||
- Modify: `backend/test/workspaces/schema.test.ts`
|
||
- Modify: `backend/test/workspaces/runtime-renderer.test.ts`
|
||
- Modify: `docs/contracts/workspace-evidence-v3.md`
|
||
- Modify: `docs/contracts/workspace-preprocessing-cli.md`
|
||
- Modify: `docs/architecture/overview.md`
|
||
- Modify: `PROJECT_STATE.md`
|
||
|
||
**Step 1: Write RED descriptor/rendering tests**
|
||
|
||
For filesystem Evidence, make `patterns: ["curated/**/*.md"]` the documented and
|
||
generated default for the new authoring layout. Continue accepting an explicitly
|
||
configured safe pattern for non-Git HTTP/S3 compatibility, but reject a filesystem
|
||
descriptor that includes both `source/**` and `curated/**` once it declares the new
|
||
layout version.
|
||
|
||
If a schema-version field is required to preserve compatibility, add it to the Evidence
|
||
subcontract, not to the whole workspace descriptor.
|
||
|
||
**Step 2: Implement the narrowest compatible descriptor change**
|
||
|
||
The materializer continues to copy the entire `evidence/` tree at the pinned commit.
|
||
Only the rendered runtime acquisition patterns restrict preprocessing to `curated/`.
|
||
Do not duplicate or move P6 materialization logic.
|
||
|
||
**Step 3: Update documentation contracts**
|
||
|
||
Document:
|
||
|
||
- source/curated layout;
|
||
- Git publication boundary;
|
||
- no runtime writes;
|
||
- validation before indexing;
|
||
- unnamed-dense plus BM25 contract and additive Evidence upgrade;
|
||
- exact public operation names and JSON status additions, if any.
|
||
|
||
**Step 4: Run backend and harness contract gates**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run test/workspaces/schema.test.ts test/workspaces/runtime-renderer.test.ts \
|
||
test/workspaces/evidence/materialization.test.ts \
|
||
test/workspaces/evidence/preprocessing.test.ts
|
||
npx tsc --noEmit -p .
|
||
cd ../harness
|
||
.venv/bin/pytest tests/test_registry_evidence_config.py \
|
||
tests/test_filesystem_evidence_source.py tests/test_preprocess_cli.py -q
|
||
```
|
||
|
||
Expected: PASS; P6 materialization safety remains unchanged.
|
||
|
||
**Step 5: Commit**
|
||
|
||
```bash
|
||
git add backend/src/workspaces/schema.ts backend/src/workspaces/runtime-renderer.ts \
|
||
backend/test/workspaces/schema.test.ts \
|
||
backend/test/workspaces/runtime-renderer.test.ts \
|
||
docs/contracts/workspace-evidence-v3.md \
|
||
docs/contracts/workspace-preprocessing-cli.md docs/architecture/overview.md PROJECT_STATE.md
|
||
git commit -m "docs(evidence): publish curated-only workspace contract"
|
||
```
|
||
|
||
### Task 12: Migrate PSD and perform acceptance
|
||
|
||
**Files in ThothII:**
|
||
|
||
- Create: `docs/testing/evidence-restructuring-manual.md`
|
||
- Create: `scripts/evidence-restructuring-acceptance.sh`
|
||
- Create: `harness/tests/fixtures/evidence_authoring/poorly_structured.md`
|
||
- Modify: `PROJECT_STATE.md`
|
||
|
||
**Files in the external authoring repository:**
|
||
|
||
- Move: `/Users/mp/projects/tht-workspace-psd/psd-clinical/evidence/<current-folders>`
|
||
to `/Users/mp/projects/tht-workspace-psd/psd-clinical/evidence/source/`
|
||
- Create: `/Users/mp/projects/tht-workspace-psd/psd-clinical/evidence/curated/<kind>/`
|
||
- Create: `/Users/mp/projects/tht-workspace-psd/psd-clinical/evidence/manifest.yaml`
|
||
- Create: `/Users/mp/projects/tht-workspace-psd/psd-clinical/evidence/evaluation.yaml`
|
||
- Modify: `/Users/mp/projects/tht-workspace-psd/psd-clinical/evidence/README.md`
|
||
- Modify: `/Users/mp/projects/tht-workspace-psd/psd-clinical/workspace.yaml`
|
||
|
||
Do not modify the external repository until the owner confirms the migration window and
|
||
the exact target branch. Treat that as the only manual authorization gate in this task.
|
||
|
||
**Step 1: Add a hermetic badly-structured fixture**
|
||
|
||
The fixture must contain prose, a rough list, an enum, an URL, an ambiguous statement
|
||
and a SQL formula candidate. The acceptance runner must prove:
|
||
|
||
- split into multiple typed units;
|
||
- ambiguity becomes `review_items`;
|
||
- no cross-source merge;
|
||
- unchanged rerun is a no-op;
|
||
- committed human content remains recoverable and proposed changes are visible in Git;
|
||
- dirty-tree refusal;
|
||
- validation blocks unresolved review and orphaned units;
|
||
- pipeline-version mismatch refusal and explicit full-corpus `--upgrade`;
|
||
- validated corpus builds an inactive candidate and searches it hybrid;
|
||
- every evaluation query retrieves at least one expected ID in the first ten results
|
||
before the candidate becomes active;
|
||
- the evaluation set contains lexical, semantic and mixed cases and reports dense,
|
||
BM25 and fused ranks separately;
|
||
- dense and BM25 receive the same deterministic query text;
|
||
- no atomic formula, enum pair, mapping, rule or URL is split into fixed-size chunks;
|
||
- the complete rendered fragment stays within the existing 4,000-character
|
||
`max_chunk_chars` default and no parallel size option exists;
|
||
- the L0 probe passes against the Qdrant image referenced by `compose.yaml` without
|
||
FastEmbed or fallback;
|
||
- query normalization is limited to NFC, newline canonicalization and outer trimming,
|
||
preserving internal whitespace, punctuation and case-sensitive identifiers;
|
||
- Formula Evidence accepts a PostgreSQL expression and rejects a complete query;
|
||
- Qdrant failure remains fail-closed.
|
||
|
||
Use a fake restructurer for hermetic CI. The real Pi call is a separate manual check.
|
||
|
||
**Step 2: Run the complete automated gates before external writes**
|
||
|
||
```bash
|
||
cd harness
|
||
.venv/bin/pytest -q
|
||
.venv/bin/ruff check .
|
||
cd ../backend
|
||
npx vitest run
|
||
npx tsc --noEmit -p .
|
||
cd ../tools/tht
|
||
go test ./...
|
||
go build ./cmd/tht
|
||
cd ../..
|
||
bash scripts/evidence-restructuring-acceptance.sh
|
||
```
|
||
|
||
Expected: all suites and acceptance checks PASS. If pre-existing unrelated failures
|
||
remain, record exact names and prove they reproduce at the baseline commit before
|
||
continuing.
|
||
|
||
**Step 3: Stop for the owner migration gate**
|
||
|
||
Provide:
|
||
|
||
- clean ThothII commit;
|
||
- test and acceptance summary;
|
||
- proposed PSD branch name;
|
||
- exact list of 36 source files to move;
|
||
- rollback command based on the pre-migration PSD commit;
|
||
- before/after Schema and Memory counts plus sample IDs used to prove non-regression.
|
||
|
||
Do not infer approval from prior design acceptance.
|
||
|
||
**Step 4: Migrate the PSD repository after approval**
|
||
|
||
Use `tht evidence prepare`, review the Git diff, resolve all review items manually, run
|
||
`tht evidence validate`, and create the approximately twenty evaluation queries. Do not
|
||
auto-merge or auto-push unless separately requested.
|
||
|
||
**Step 5: Activate and perform the additive BM25 upgrade**
|
||
|
||
After the PSD merge/pull and activation, inspect first:
|
||
|
||
```text
|
||
tht --installation <absolute>/thothii-installation.yaml workspace vector inspect
|
||
--workspace psd-clinical --json
|
||
```
|
||
|
||
Before preprocessing, record counts and representative IDs for `schema_table`,
|
||
`schema_column`, `memory` and `solved_question`. Run `workspace preprocess evidence`:
|
||
it adds `bm25` when absent, builds the candidate, evaluates that exact generation and
|
||
publishes it only if the report passes. Repeat the counts, verify the representative IDs
|
||
and run dense smoke searches for Schema and Memory before completing the manual
|
||
walkthrough of `clarification`, `rewriting`, `schema_linking`, `cte` and `final_sql`.
|
||
|
||
**Step 6: Record acceptance and commit ThothII documentation**
|
||
|
||
`docs/testing/evidence-restructuring-manual.md` must record separate outcomes for:
|
||
|
||
- authoring and Git review;
|
||
- additive BM25 schema upgrade;
|
||
- Schema and Memory before/after non-regression evidence;
|
||
- preprocessing generation publication;
|
||
- hybrid retrieval evaluation;
|
||
- formula retrieval;
|
||
- distinzione fra risultato vuoto e indisponibilità bloccante;
|
||
- complete session behavior.
|
||
|
||
Update `PROJECT_STATE.md` only with observed results and immutable commit/run IDs.
|
||
|
||
```bash
|
||
git add docs/testing/evidence-restructuring-manual.md \
|
||
scripts/evidence-restructuring-acceptance.sh \
|
||
harness/tests/fixtures/evidence_authoring/poorly_structured.md PROJECT_STATE.md
|
||
git commit -m "test(evidence): record restructuring acceptance"
|
||
```
|
||
|
||
## Final verification checklist
|
||
|
||
Before claiming completion, verify:
|
||
|
||
- [ ] `CONTEXT.md` and both Evidence plan documents use the same terminology.
|
||
- [ ] All eight Evidence kinds have type-specific positive and negative tests.
|
||
- [ ] Pi is called once per changed source, with no tools and no saved session.
|
||
- [ ] `prepare` refuses dirty curated state, exposes every proposal in Git and never
|
||
deletes orphans.
|
||
- [ ] `validate` blocks unresolved review items and orphaned units.
|
||
- [ ] Source renames and kind-only reclassifications preserve unit IDs; semantic splits
|
||
receive new IDs.
|
||
- [ ] IDs use `evidence:<slug>`, remain independent from kind and are not recomputed
|
||
after their initial assignment.
|
||
- [ ] Formula Evidence accepts one PostgreSQL expression and rejects full queries.
|
||
- [ ] A pipeline-version mismatch performs no writes without explicit `--upgrade`.
|
||
- [ ] Runtime reads only `curated/**/*.md` from the pinned Git revision.
|
||
- [ ] The TypeScript and Python Qdrant compatibility checks agree.
|
||
- [ ] Qdrant retains the unnamed dense vector and adds only `bm25` with IDF.
|
||
- [ ] Evidence preprocessing adds a missing `bm25` definition but never deletes,
|
||
renames or destructively rebuilds collection vectors.
|
||
- [ ] Schema and Memory counts, sample IDs and dense searches remain unchanged across
|
||
the additive upgrade.
|
||
- [ ] Italian BM25 options are identical during ingest and query.
|
||
- [ ] Hybrid search uses two prefetches and default RRF.
|
||
- [ ] Hard filters always include workspace, revision, active generation and purpose;
|
||
only explicit `required_*` context values add further filters.
|
||
- [ ] Formula runtime lookup uses typed Evidence; session proposals remain non-published.
|
||
- [ ] Fragment hits are grouped into one Evidence Result with excerpts and a reference;
|
||
full documents are loaded only on demand.
|
||
- [ ] Evidence is queried independently from the five mapped semantic stages and is not
|
||
invoked from `memory` or `synthesis`.
|
||
- [ ] An available empty result can continue; technical unavailability blocks the
|
||
current stage without stale-generation or purpose fallback.
|
||
- [ ] Each available stage search persists only its minimal Evidence receipt.
|
||
- [ ] Evaluation reports hit@5 and hit@10 against a versioned fixture and passes only
|
||
when every query has at least one expected result in the first ten.
|
||
- [ ] Evaluation runs against the candidate generation before atomic activation; a
|
||
failed candidate remains invisible to sessions.
|
||
- [ ] Existing corpus rollback, compensation, retention and resume tests still pass.
|
||
- [ ] Backend Vitest and TypeScript gates pass.
|
||
- [ ] Harness pytest and Ruff gates pass.
|
||
- [ ] Native `tht` Go tests/build pass.
|
||
- [ ] External PSD writes occurred only after explicit migration authorization.
|