ThothII/docs/evidence.md

# Evidence: sources, preparation, and review

This page describes the complete workspace Evidence lifecycle: where original material lives, how curated units are produced, when they become available at runtime, and what the author and reviewer are responsible for.

## Editable local Evidence

E1 adds [Curated Evidence v4 and a persistent local archive](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md).
The visible title and payload fields are authoritative Markdown. New manual units need no
external source; consolidation records their curator and distinguishes later corrections
from original documentary provenance. The core archive API creates immutable candidates
and advances its active pointer only after successful indexing.

E2 adds **Administration → Evidence management**, actual host file paths, complete
browsing and filtering, and the installed `tht workspace evidence consolidate
--workspace <id>` command. Edit files externally, consolidate to activate them, then
review and run Git manually. Runtime consumes only the active local snapshot.
The [v4 contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md#administration-and-installed-command)
describes host mounting, first conversion, failure recovery and Clear behavior.
The local PSD preview has 35 converted and indexed units. E3 adds explicit import/refresh
from local drafts and configured HTTP/S3 sources, durable current/proposed comparisons,
and keep/replace decisions with activation and retry. See
[Import drafts and refresh sources](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md#import-drafts-and-refresh-sources).
Workflow gate corrections remain the subsequent shared increment, X1.

## Existing repository publication path

The remainder describes the legacy, uninitialized repository source path. Initialized
v4 local archives use the lifecycle above; manual declarations need no source document.

The workspace repository is the versioned source. ThothII reads it, validates it, and publishes an atomic generation. It does not modify, commit, or push the author's repository.

Evidence becomes available to the workflow only when:

1. the original material is in `source/`;
2. the derived unit is in `curated/`;
3. the manifest links the unit, source, and hash;
4. validation finds no errors or unresolved review items;
5. preprocessing builds and activates an indexed generation.

A proposal generated during a session is not automatically published Evidence. The model may propose a formula or explanation, but a curator must import, review, and publish it in the repository before another session can retrieve it.

## Where the original source belongs

For filesystem Evidence v2, the authoritative original source must be in the `source/` directory of the workspace repository. `curated/` contains the reviewed and indexed result, not the original material.

```text
<workspace-repository>/
├── source/                 # materiale originale, preservato
│   └── <domain>/<file>.md
├── curated/                # reviewed Evidence Units
│   └── <domain>/<unit>.md
├── manifest.yaml           # preparation links, hashes, and metadata
└── example/                # examples and supporting material
```

The workspace descriptor must declare `evidence.schema_version: 2` and use exactly this configuration for a filesystem source:

```yaml
evidence:
  schema_version: 2
  source:
    type: filesystem
    uri: "<workspace.id>/evidence"
    patterns:
      - "curated/**/*.md"
```

The legacy configuration may expose `source_root`, such as `${THT_DOCS_ROOT}` or `/data`. For the v2 structure, the runtime pattern must select only `curated/**/*.md`. Do not index `source/` directly, mix `source/` and `curated/`, use broader globs, or include non-Markdown files.

HTTP and S3 are separate adapters. They do not use the filesystem structure `source/` and `curated/`, but they must still provide stable provenance, without credentials in URIs, under the adapter-specific contract.

## What an editable curated unit contains

New preparation produces unit schema v4. The first H1 contains its title; documented H2
sections contain the typed payload. A minimal manual unit is:

```markdown
---
schema_version: 4
id: evidence:order-key
kind: domain
language: en
purposes: [sql_generation]
---

# Order key

## Rule

Join orders using the order number, financial year and company.
```

Use `tht evidence migrate <workspace-root>` for deterministic legacy conversion. The
conversion preserves typed content and initializes an archive baseline; it does not
activate the local corpus. See the [v4 contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/curated-evidence-v4.md) for
all eight kinds, provenance, file layout and the E1/E2 boundary.

## Legacy v3 representation

Canonical Curated Evidence v3 hides canonical machine metadata in an HTML comment and renders the
whole review surface as real Markdown. GitHub therefore shows no frontmatter table. The body layout
is deterministic for each Evidence kind: prose uses sections and paragraphs, scopes and enum values
use wrapping lists, formulas use fenced SQL, supporting excerpts use blockquotes, and unresolved
review items use dedicated blocks. Long domain rules are split into readable paragraphs, labelled
subsections, and lists at existing semicolon boundaries. Their exact original text remains canonical
in an invisible marker, so the formatting cannot change their meaning or bytes.

Preprocessing parses the unit first and builds semantic chunks from the typed payload. The vector
store therefore receives the original rule text and not headings, list markers, or invisible
presentation metadata.

```markdown
<!-- tht:metadata:<canonical metadata> -->
# Fascia pediatrica

> **Dominio** · Italiano
>
> **Scopi:** Disambiguazione

## Ambito di applicazione

### Concetti

- fascia pediatrica

## Regola

La fascia pediatrica comprende i pazienti con età inferiore a 18 anni.

## Estratti di supporto

> I pazienti sotto i 18 anni sono pediatrici.

<details>
<summary>Dettagli tecnici e provenienza</summary>

- **ID:** `evidence:fascia-pediatrica`
- **File sorgente:** `source/domain/paziente.md`

</details>
```

The actual files contain invisible `tht:` comments for canonical metadata and typed-field
boundaries. Removing, duplicating, or desynchronizing them makes validation fail closed instead of
silently ignoring content. Unit schemas v1 and v2 remain readable for compatibility, but newly
prepared units use v4. Unit v3 remains readable as a conversion input.

Curated units must be atomic, readable by a second reviewer, and supported by the source.
Provenance references must lead back to the original file and the passage that supports the claim.
Do not put secrets, tokens, passwords, or credentials in metadata or URIs.

The modern canonical form also stores the Evidence kind, provenance, supporting excerpts, `source_file`, and `source_sha256`. The canonical contract rejects unknown fields, mutable metadata, and URIs containing credentials. Identifiers must remain stable even when a unit's kind changes.

## Preparation: from source to active generation

```mermaid
flowchart TD
    SRC["source/DOMAIN/*.md\noriginal material"] --> PREP["tht evidence prepare\ncandidate preparation"]
    PREP --> CAND["curated/DOMAIN/*.md\nproposed or updated units"]
    CAND --> VAL["tht evidence validate\nstructure and link checks"]
    VAL -->|errors or review items| FIX["Author corrections\nand review"]
    FIX --> PREP
    VAL -->|publishable| COMMIT["Commit del repository\nauthoring clone"]
    COMMIT --> ING["tht workspace preprocess run\nnormalization and chunking"]
    ING --> BM25["BM25 index"]
    ING --> VEC["Embeddings and vector store"]
    BM25 --> GEN["Candidate generation"]
    VEC --> GEN
    GEN --> ACT["Replace active Evidence slice"]
    ACT --> RUNTIME["Evidence retrieval in the workflow"]
```

Preparation can restructure changed sources, but it does not publish by itself. `prepare` produces a proposal and can identify the document involved in an error. `validate` does not write or publish. The curator publishes the revision. The complete workspace preprocessing command reads that exact revision and replaces the active Evidence slice. There is no application-level rollback; rerun complete preprocessing after correcting a failure.

Runtime retrieval is hybrid. The dense branch uses embeddings, the BM25 branch uses lexical search,
and deterministic fusion orders the results. Evidence fragments live in the workspace `reference`
collection together with Schema and relationships; runtime Memory and solved questions live in a
separate `memory` collection. The published unit keeps its provenance, which the model must cite when
it uses the Evidence.

## Author responsibilities

The author prepares the material and makes every unit verifiable. The author must:

- put the original material in `source/` without changing its meaning during curation;
- split the content into atomic units, with one rule or definition per unit when possible;
- assign a stable identifier and a clear title;
- provide provenance, tables, and related concepts when known;
- keep the text in the workspace language;
- separate facts, rules, examples, formulas, and limits;
- include excerpts that support the unit without extending the conclusion beyond the source;
- run `tht evidence prepare` and `tht evidence validate`;
- resolve every error and review item before proposing a commit;
- give the reviewer the necessary context, including source changes and the reason for any rename or retirement.

The author must not:

- write directly to the active production corpus;
- treat a model proposal as a verified fact;
- delete an unsupported unit without recording its retirement or relink;
- put credentials in metadata, files, or provenance URLs;
- manually change manifests, hashes, or generations to make validation pass.

## Reviewer responsibilities

The reviewer does not approve text merely because it is clear. The reviewer checks the relationship between source, unit, and intended use. For each unit, the reviewer must check:

1. the cited source exists in the reviewed revision;
2. the excerpt actually supports the claim;
3. the unit does not combine incompatible rules or independent concepts;
4. tables, columns, and concepts are identified correctly;
5. the identifier is stable and does not duplicate another unit;
6. the text distinguishes the definition, condition, exception, and example;
7. it contains no sensitive information or details absent from the source;
8. retrieval evaluation covers relevant queries and does not hide empty results.

The reviewer can approve, request changes, reject, retire, or relink a unit to a new source. Retirement must be explicit. A relink must name the new file and leave a verifiable record of the decision. Approval does not publish immediately: the repository must pass validation and the generation must pass evaluation before activation.

## Available commands

Authoring commands operate on the workspace repository and do not publish directly.

```bash
# Prepare changed sources. Does not commit or publish.
tht evidence prepare <workspace-root>

# Reprocess all sources with the installed pipeline.
tht evidence prepare <workspace-root> --upgrade

# Rewrite legacy v1/v2 units as table-free v3 Markdown without model calls.
tht evidence migrate <workspace-root>

# Validate structure, manifest, links, and review items.
tht evidence validate <workspace-root>

# Return JSON for CI or automated tools.
tht evidence validate <workspace-root> --json

# Evaluate retrieval on a generation or the active generation.
tht evidence evaluate <workspace-root> --config <workspace-config>
tht evidence evaluate <workspace-root> --config <workspace-config> --generation <id> --json

# Resolve a unit without publishing: retire it or link it to a new source.
tht evidence resolve <workspace-root> evidence:<id> --retire
tht evidence resolve <workspace-root> evidence:<id> --source source/domain/nuovo.md

# Materialize Catalog-derived schema/LSH and the revision-pinned Evidence in one run.
tht --installation <absolute>/thothii-installation.yaml \
  workspace preprocess run --workspace <workspace-id>
```

`evidence prepare`, `evidence migrate`, `evidence validate`, and `evidence resolve` are authoring
operations and require the repository path. Runtime publication is available only through the
complete host-side `workspace preprocess run`; there is no public Evidence-only preprocessing
command.

Exit codes are part of the operating contract: `evidence validate` returns `0` when the corpus is publishable, `1` for validation errors, and `3` when only review items or orphaned units remain. With `--json`, stdout must contain valid JSON only.

## Formulas and session proposals

Formulas use a format distinct from document Evidence. A formula proposed during a session may be cited in the current proposal, but it does not enter the runtime corpus, receive a usable `evidence:` ID, or write to the repository. To become published, it must follow the same import, review, and preprocessing path as other units.

## Contract references

- [Workspace Evidence v3 contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-evidence-v3.md)
- [Preprocessing CLI contract](https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-preprocessing-cli.md)