ThothII/docs/contracts/curated-evidence-v4.md

# Editable Curated Evidence v4

Curated unit v4 makes the visible Markdown body authoritative. It uses the existing
typed Evidence payloads and stable identifiers. Workspace descriptor v4 and Evidence
descriptor v1/v2 are separate version numbers.

E1 implements the format, explicit conversion, local archive and consolidation API.
E2 connects that API to the installed consolidation command, Evidence management
page and runtime source selection. The local PSD preview now uses 35 converted units.

## Write or edit a file

Place units in `<workspace-root>/evidence/curated/<kind>/<name>.md`. Keep the existing
`id` when editing. The directory must match `kind`. A new manual unit needs no external
document, hash or encoded metadata:

```markdown
---
schema_version: 4
id: evidence:order-key
kind: domain
language: en
purposes: [sql_generation]
applies_to:
  tables: [sales.orders]
---

# Order key

## Rule

Join orders using the order number, financial year and company.
```

Required metadata is `schema_version`, `id`, `kind`, `language`, and a nonempty
`purposes` list. Purposes are `disambiguation`, `rewriting`, `schema_linking`, and
`sql_generation`. Optional `applies_to` contains `concepts`, `tables`, and `columns`.
Tables use `schema.table`; columns use `schema.table.column`.

The first H1 is the title. H2 headings identify the payload fields below. Heading
spelling follows `language`: Italian for `it` and its regional variants, English
otherwise. Keep structural headings when editing the text below them.

| Kind | English field headings | Italian field headings |
| --- | --- | --- |
| `domain` | Rule | Regola |
| `glossary` | Definition, Synonyms, Variants | Definizione, Sinonimi, Varianti |
| `enum` | Column, Values | Colonna, Valori |
| `example` | Question, Interpretation | Domanda, Interpretazione |
| `mapping` | Concept, Tables, Columns | Concetto, Tabelle, Colonne |
| `normalization` | Input, Output, Rule | Input, Output, Regola |
| `formula` | Concept, Columns, SQL | Concetto, Colonne, SQL |
| `reference` | URL, Label, Description | URL, Etichetta, Descrizione |

List fields use one `- value` per line. Empty optional lists may be omitted. Quoted
JSON strings within bullets preserve unusual or multiline values during conversion.
Enum values use `### "stored value"`, followed by their meaning; `### ""` represents
an empty stored value. Formula SQL uses a fenced `sql` block containing one PostgreSQL
expression. Whole queries and mutation statements remain invalid.

Nested prose headings and fenced examples are supported inside text fields. An H2
matching a field heading is structural outside a code fence. Duplicate fields, missing
required fields, duplicate metadata keys and malformed payloads are rejected. There
is no second title or payload in frontmatter and no hidden authoritative rule text.
Conversion fails explicitly if a legacy payload cannot be represented losslessly.

## Current provenance and original source

The host supplies document provenance when refining source material: relative source
path, normalized source hash and exact supporting excerpts. A manual unit can omit
`provenance`. Consolidation records `kind: manual` and the supplied curator identity.

A visible change to a previously recorded unit becomes a manual declaration. If that
unit originated from a document, its former document provenance is retained under
`original`. The original excerpts establish lineage; they do not assert that the
source contains the new wording. Retrieval carries this distinction through typed
metadata. Curators edit content; the service updates managed provenance.

Unchanged document declarations still require matching source bytes and excerpts.
Replacing a source requires an explicit refresh through the E3 source-review path. Ordinary preparation
is blocked on an initialized local archive, preventing regenerated source material
from overwriting corrections or restoring deletions. Legacy `resolve` is likewise
blocked there; local file corrections and the archive API own those changes.

## Persistent archive and activation

```text
<workspace-root>/
  .evidence-archive.lock
  evidence/
    source/                      # acquired original documents
    curated/<kind>/*.md           # editable primary content
    local-manifest.yaml          # derived declarations and deletion records
    .local/
      state.yaml                 # baseline, pending and active revisions
      snapshots/<revision>/      # immutable units, source bytes and manifest
```

Keep the complete Evidence tree and its managed metadata in backups and the operator's
Git review. Historical source bytes and deletion records are needed to preserve manual
care across subsequent imports. The separate corpus cache and Qdrant index are derived.
The lock file only coordinates local service operations.

`LocalEvidenceArchive.initialize()` records the pre-edit baseline without activation.
`consolidate(actor=..., activate=...)` validates files, derives provenance, records
deletions and source suppression, and creates an immutable candidate. The activation
callback receives that snapshot and must raise if indexing is blocked or fails. Only
successful activation advances `active_snapshot()`. Without a callback the result is
explicitly `pending_activation`; saving a file alone never changes this pointer.

The same candidate can be retried after an index failure. Interrupted managed writes
are replayed only when the operator's file bytes have not changed. A missing curated
directory is an availability error, not proof that all units were deleted. Removing
unit files from an accessible directory records deletion without deleting their sources.
The API also provides `get`, revision-checked `save`, and revision-checked `remove` for
future deliberate workflow corrections. Concurrent external edits produce conflicts.

These boundaries are exercised with the existing corpus pipeline and real Qdrant.
An initialized installation reads only its active local snapshot, including during
ordinary preprocessing. Unconsolidated edits are visible in Administration but do
not enter retrieval. Before initialization, the pinned repository source still works.

## Administration and installed command

Open **Administration → Evidence management**. This independent page requires the
`evidence.manage` permission and no active session. It shows complete units, source
lineage, review items, file errors, filters, and changes relative to the active snapshot.
Edit the displayed Markdown path using an external editor. Refresh files to inspect
the result, then run the command shown by the page:

```sh
tht --installation /absolute/path/thothii-installation.yaml \
  workspace evidence consolidate --workspace psd-clinical
```

The installed command accepts `--json`. It validates and activates Evidence using
the existing corpus pipeline and embedding service. It does not scan the DWH, run
full workspace preprocessing, change Catalog/Schema readiness, or execute Git.
If indexing fails after saving, the archive retains the candidate for retry and the
previous active revision remains selected. Validation errors identify corrections
to make in the files. Structural and review checks still apply; initialized archives
do not require the legacy repository's fixed retrieval-evaluation fixture, whose
expected IDs would otherwise prevent deliberate local deletions.

The canonical workspace root is `<workspace-registry-root>/repo/<workspace-id>`.
Snapshots and the derived corpus are separate. For a registry in a Docker volume,
copy its existing `repo` to a persistent host directory before enabling
`deploy/compose.evidence-host.yaml`; set `THT_EVIDENCE_HOST_REGISTRY_ROOT` to that
directory and include the override in the installation descriptor. Core and
workspace-maintenance must mount the same checkout. Keep registry state/snapshots
on their existing volume. The page only presents a host path when configured; it
does not label an internal container path as a usable editor path.

After successful consolidation, inspect and commit the complete workspace Evidence
tree, including managed manifests, snapshots, and deletion records, then push manually.
The page provides quoted POSIX-shell examples for status, diff, add, commit, and push.
Do not commit only the edited Markdown. Ordinary Git operations remain the operator's
responsibility. E3 source decisions use this same activation boundary.

**Clear** removes derived Reference/corpus data while retaining editable files,
snapshots and Memory. It still requires full workspace preprocessing to recreate
Reference/Schema readiness; Evidence consolidation does not satisfy that gate.

## Import drafts and refresh sources

Place externally authored Markdown drafts in `<workspace-root>/evidence/incoming/`.
The specialist needs no installation account or database access to write a draft.
Copying the draft to the installation and choosing **Import or refresh sources** in
Evidence management explicitly starts acquisition and refinement. The equivalent
installed command is:

```sh
tht --installation /absolute/path/thothii-installation.yaml \
  workspace evidence refresh --workspace psd-clinical
```

This reads local `incoming/**/*.md` and original `source/**/*.md` files, excluding
managed `source/acquired/` versions. It also reads configured HTTP and S3 sources
through the existing read-only adapters and network policies. After local archive
initialization, filesystem descriptors use this local authoring tree; ordinary
runtime/preprocessing never fetches remote source changes. No source-server write
credential is needed. The existing Pi authoring refiner runs once for each changed
document, without session state or tools. Unchanged source hashes skip refinement.

All acquisitions and proposals must succeed before the new comparison set is
recorded. An access or refinement failure preserves previous comparisons and active
Evidence. A source absent from a successful discovery is marked missing and never
treated as permission to delete units. Restore an accidentally missing local original
file before consolidating its document-derived units, or explicitly retire those units.

Each changed source has a durable comparison showing current local units, complete
proposed content, supporting excerpts, and IDs that replacement would retire:

- **Keep local Evidence** retains the current wording as a manual declaration, with
  its original documentary lineage preserved. The changed source is acknowledged;
  the next unchanged refresh does not reopen that decision.
- **Use proposed Evidence** adopts the displayed proposal and explicitly retires the
  displayed omitted IDs. The source version and provenance change together.

Both choices save and activate through the same local consolidation/index pipeline.
Review items block adoption; correct the original draft and refresh, or keep local
content. There is no automatic merge based on a model's semantic conflict assessment.
Any change to an affected curated file invalidates the comparison and requires a new
refresh. If indexing fails after the decision is saved, use **Retry saved decision**;
this reuses acquired content without fetching sources again. An intervening external
edit is never silently overwritten by recovery.

Headless operators can make the same decision using the source ID and comparison
revision from the local source registry or administration response:

```sh
tht --installation /absolute/path/thothii-installation.yaml \
  workspace evidence decide --workspace psd-clinical \
  --source-id <64-hex-source-id> --revision <64-hex-comparison-revision> \
  --decision keep
```

Use `--decision replace` to adopt the proposal. All installed commands accept `--json`.
The Python `evidence sources` worker is internal to this installed command/API surface.

`evidence/.local/sources.json` stores source identities, comparisons and retry journals.
`evidence/.local/acquisitions/` preserves original acquired bytes and credential-free
remote provenance. Adopted normalized documents live under
`evidence/source/acquired/<source-id>/<content-hash>.md`. Versioned paths let a new
document and an older manual declaration's original source coexist. Include all of
these files in the existing manual Git/backup sequence. Do not edit managed acquired
versions: edit the original local draft or refresh its remote origin.

Deleted IDs remain reserved. Once a source has had a curated deletion, fresh model
IDs from that source are conservatively suppressed as well: changing an ID must not
restore retired knowledge. Existing surviving IDs can still receive reviewed updates;
deliberate new knowledge can be written as a manual Evidence file. Refresh is bounded
to 200 documents and 100 MiB per operation, in addition to each adapter's limits.

## Convert an existing workspace

The existing workflow CLI command `tht evidence migrate <workspace-root>` converts
unit versions 1–3 to 4 deterministically and initializes the archive baseline. It
preserves IDs, typed content, provenance and review items, with no model call or Git
commit. It does not activate a local index. Review items still block consolidation.
The command's existing Git-worktree path check remains in effect.

The first installed consolidation performs this conversion automatically when the
legacy manifest is present, then validates and activates the result. Preserve the
existing checkout in backups before upgrading. The E1 validation used an isolated
copy; E2 also converted and indexed all 35 units on the running local preview.
See the [E1 validation report](../reports/knowledge-archives-release.md) and
[E2 validation report](../reports/knowledge-archives-release.md).