ThothII/docs/contracts/curated-evidence-v4.md

Editable Curated Evidence v4

Curated unit v4 makes the visible Markdown body authoritative. It uses the existing typed Evidence payloads and stable identifiers. Workspace descriptor v4 and Evidence descriptor v1/v2 are separate version numbers.

E1 implements the format, explicit conversion, local archive and consolidation API. E2 connects that API to the installed consolidation command, Evidence management page and runtime source selection. The local PSD preview now uses 35 converted units.

Write or edit a file

Place units in <workspace-root>/evidence/curated/<kind>/<name>.md. Keep the existing id when editing. The directory must match kind. A new manual unit needs no external document, hash or encoded metadata:

---
schema_version: 4
id: evidence:order-key
kind: domain
language: en
purposes: [sql_generation]
applies_to:
  tables: [sales.orders]
---

# Order key

## Rule

Join orders using the order number, financial year and company.

Required metadata is schema_version, id, kind, language, and a nonempty purposes list. Purposes are disambiguation, rewriting, schema_linking, and sql_generation. Optional applies_to contains concepts, tables, and columns. Tables use schema.table; columns use schema.table.column.

The first H1 is the title. H2 headings identify the payload fields below. Heading spelling follows language: Italian for it and its regional variants, English otherwise. Keep structural headings when editing the text below them.

Kind	English field headings	Italian field headings
domain	Rule	Regola
glossary	Definition, Synonyms, Variants	Definizione, Sinonimi, Varianti
enum	Column, Values	Colonna, Valori
example	Question, Interpretation	Domanda, Interpretazione
mapping	Concept, Tables, Columns	Concetto, Tabelle, Colonne
normalization	Input, Output, Rule	Input, Output, Regola
formula	Concept, Columns, SQL	Concetto, Colonne, SQL
reference	URL, Label, Description	URL, Etichetta, Descrizione

List fields use one - value per line. Empty optional lists may be omitted. Quoted JSON strings within bullets preserve unusual or multiline values during conversion. Enum values use ### "stored value", followed by their meaning; ### "" represents an empty stored value. Formula SQL uses a fenced sql block containing one PostgreSQL expression. Whole queries and mutation statements remain invalid.

Nested prose headings and fenced examples are supported inside text fields. An H2 matching a field heading is structural outside a code fence. Duplicate fields, missing required fields, duplicate metadata keys and malformed payloads are rejected. There is no second title or payload in frontmatter and no hidden authoritative rule text. Conversion fails explicitly if a legacy payload cannot be represented losslessly.

Current provenance and original source

The host supplies document provenance when refining source material: relative source path, normalized source hash and exact supporting excerpts. A manual unit can omit provenance. Consolidation records kind: manual and the supplied curator identity.

A visible change to a previously recorded unit becomes a manual declaration. If that unit originated from a document, its former document provenance is retained under original. The original excerpts establish lineage; they do not assert that the source contains the new wording. Retrieval carries this distinction through typed metadata. Curators edit content; the service updates managed provenance.

Unchanged document declarations still require matching source bytes and excerpts. Replacing a source requires an explicit refresh through the E3 source-review path. Ordinary preparation is blocked on an initialized local archive, preventing regenerated source material from overwriting corrections or restoring deletions. Legacy resolve is likewise blocked there; local file corrections and the archive API own those changes.

Persistent archive and activation

<workspace-root>/
  .evidence-archive.lock
  evidence/
    source/                      # acquired original documents
    curated/<kind>/*.md           # editable primary content
    local-manifest.yaml          # derived declarations and deletion records
    .local/
      state.yaml                 # baseline, pending and active revisions
      snapshots/<revision>/      # immutable units, source bytes and manifest

Keep the complete Evidence tree and its managed metadata in backups and the operator's Git review. Historical source bytes and deletion records are needed to preserve manual care across subsequent imports. The separate corpus cache and Qdrant index are derived. The lock file only coordinates local service operations.

LocalEvidenceArchive.initialize() records the pre-edit baseline without activation. consolidate(actor=..., activate=...) validates files, derives provenance, records deletions and source suppression, and creates an immutable candidate. The activation callback receives that snapshot and must raise if indexing is blocked or fails. Only successful activation advances active_snapshot(). Without a callback the result is explicitly pending_activation; saving a file alone never changes this pointer.

The same candidate can be retried after an index failure. Interrupted managed writes are replayed only when the operator's file bytes have not changed. A missing curated directory is an availability error, not proof that all units were deleted. Removing unit files from an accessible directory records deletion without deleting their sources. The API also provides get, revision-checked save, and revision-checked remove for future deliberate workflow corrections. Concurrent external edits produce conflicts.

These boundaries are exercised with the existing corpus pipeline and real Qdrant. An initialized installation reads only its active local snapshot, including during ordinary preprocessing. Unconsolidated edits are visible in Administration but do not enter retrieval. Before initialization, the pinned repository source still works.

Administration and installed command

Open Administration → Evidence management. This independent page requires the evidence.manage permission and no active session. It shows complete units, source lineage, review items, file errors, filters, and changes relative to the active snapshot. Edit the displayed Markdown path using an external editor. Refresh files to inspect the result, then run the command shown by the page:

tht --installation /absolute/path/thothii-installation.yaml \
  workspace evidence consolidate --workspace psd-clinical

The installed command accepts --json. It validates and activates Evidence using the existing corpus pipeline and embedding service. It does not scan the DWH, run full workspace preprocessing, change Catalog/Schema readiness, or execute Git. If indexing fails after saving, the archive retains the candidate for retry and the previous active revision remains selected. Validation errors identify corrections to make in the files. Structural and review checks still apply; initialized archives do not require the legacy repository's fixed retrieval-evaluation fixture, whose expected IDs would otherwise prevent deliberate local deletions.

The canonical workspace root is <workspace-registry-root>/repo/<workspace-id>. Snapshots and the derived corpus are separate. For a registry in a Docker volume, copy its existing repo to a persistent host directory before enabling deploy/compose.evidence-host.yaml; set THT_EVIDENCE_HOST_REGISTRY_ROOT to that directory and include the override in the installation descriptor. Core and workspace-maintenance must mount the same checkout. Keep registry state/snapshots on their existing volume. The page only presents a host path when configured; it does not label an internal container path as a usable editor path.

After successful consolidation, inspect and commit the complete workspace Evidence tree, including managed manifests, snapshots, and deletion records, then push manually. The page provides quoted POSIX-shell examples for status, diff, add, commit, and push. Do not commit only the edited Markdown. Ordinary Git operations remain the operator's responsibility. E3 source decisions use this same activation boundary.

Clear removes derived Reference/corpus data while retaining editable files, snapshots and Memory. It still requires full workspace preprocessing to recreate Reference/Schema readiness; Evidence consolidation does not satisfy that gate.

Import drafts and refresh sources

Place externally authored Markdown drafts in <workspace-root>/evidence/incoming/. The specialist needs no installation account or database access to write a draft. Copying the draft to the installation and choosing Import or refresh sources in Evidence management explicitly starts acquisition and refinement. The equivalent installed command is:

tht --installation /absolute/path/thothii-installation.yaml \
  workspace evidence refresh --workspace psd-clinical

This reads local incoming/**/*.md and original source/**/*.md files, excluding managed source/acquired/ versions. It also reads configured HTTP and S3 sources through the existing read-only adapters and network policies. After local archive initialization, filesystem descriptors use this local authoring tree; ordinary runtime/preprocessing never fetches remote source changes. No source-server write credential is needed. The existing Pi authoring refiner runs once for each changed document, without session state or tools. Unchanged source hashes skip refinement.

All acquisitions and proposals must succeed before the new comparison set is recorded. An access or refinement failure preserves previous comparisons and active Evidence. A source absent from a successful discovery is marked missing and never treated as permission to delete units. Restore an accidentally missing local original file before consolidating its document-derived units, or explicitly retire those units.

Each changed source has a durable comparison showing current local units, complete proposed content, supporting excerpts, and IDs that replacement would retire:

• Keep local Evidence retains the current wording as a manual declaration, with its original documentary lineage preserved. The changed source is acknowledged; the next unchanged refresh does not reopen that decision.
• Use proposed Evidence adopts the displayed proposal and explicitly retires the displayed omitted IDs. The source version and provenance change together.

Both choices save and activate through the same local consolidation/index pipeline. Review items block adoption; correct the original draft and refresh, or keep local content. There is no automatic merge based on a model's semantic conflict assessment. Any change to an affected curated file invalidates the comparison and requires a new refresh. If indexing fails after the decision is saved, use Retry saved decision; this reuses acquired content without fetching sources again. An intervening external edit is never silently overwritten by recovery.

Headless operators can make the same decision using the source ID and comparison revision from the local source registry or administration response:

tht --installation /absolute/path/thothii-installation.yaml \
  workspace evidence decide --workspace psd-clinical \
  --source-id <64-hex-source-id> --revision <64-hex-comparison-revision> \
  --decision keep

Use --decision replace to adopt the proposal. All installed commands accept --json. The Python evidence sources worker is internal to this installed command/API surface.

evidence/.local/sources.json stores source identities, comparisons and retry journals. evidence/.local/acquisitions/ preserves original acquired bytes and credential-free remote provenance. Adopted normalized documents live under evidence/source/acquired/<source-id>/<content-hash>.md. Versioned paths let a new document and an older manual declaration's original source coexist. Include all of these files in the existing manual Git/backup sequence. Do not edit managed acquired versions: edit the original local draft or refresh its remote origin.

Deleted IDs remain reserved. Once a source has had a curated deletion, fresh model IDs from that source are conservatively suppressed as well: changing an ID must not restore retired knowledge. Existing surviving IDs can still receive reviewed updates; deliberate new knowledge can be written as a manual Evidence file. Refresh is bounded to 200 documents and 100 MiB per operation, in addition to each adapter's limits.

Convert an existing workspace

The existing workflow CLI command tht evidence migrate <workspace-root> converts unit versions 1–3 to 4 deterministically and initializes the archive baseline. It preserves IDs, typed content, provenance and review items, with no model call or Git commit. It does not activate a local index. Review items still block consolidation. The command's existing Git-worktree path check remains in effect.

The first installed consolidation performs this conversion automatically when the legacy manifest is present, then validates and activates the result. Preserve the existing checkout in backups before upgrading. The E1 validation used an isolated copy; E2 also converted and indexed all 35 units on the running local preview. See the E1 validation report and E2 validation report.