# Evidence: sources, preparation, and review This page describes the complete workspace Evidence lifecycle: where original material lives, how curated units are produced, when they become available at runtime, and what the author and reviewer are responsible for. ## Publication rule The workspace repository is the versioned source. ThothII reads it, validates it, and publishes an atomic generation. It does not modify, commit, or push the author's repository. Evidence becomes available to the workflow only when: 1. the original material is in `source/`; 2. the derived unit is in `curated/`; 3. the manifest links the unit, source, and hash; 4. validation finds no errors or unresolved review items; 5. preprocessing builds and activates an indexed generation. A proposal generated during a session is not automatically published Evidence. The model may propose a formula or explanation, but a curator must import, review, and publish it in the repository before another session can retrieve it. ## Where the original source belongs For filesystem Evidence v2, the authoritative original source must be in the `source/` directory of the workspace repository. `curated/` contains the reviewed and indexed result, not the original material. ```text / ├── source/ # materiale originale, preservato │ └── /.md ├── curated/ # reviewed Evidence Units │ └── /.md ├── manifest.yaml # preparation links, hashes, and metadata └── example/ # examples and supporting material ``` The workspace descriptor must declare `evidence.schema_version: 2` and use exactly this configuration for a filesystem source: ```yaml evidence: schema_version: 2 source: type: filesystem uri: "/evidence" patterns: - "curated/**/*.md" ``` The legacy configuration may expose `source_root`, such as `${THT_DOCS_ROOT}` or `/data`. For the v2 structure, the runtime pattern must select only `curated/**/*.md`. Do not index `source/` directly, mix `source/` and `curated/`, use broader globs, or include non-Markdown files. HTTP and S3 are separate adapters. They do not use the filesystem structure `source/` and `curated/`, but they must still provide stable provenance, without credentials in URIs, under the adapter-specific contract. ## What a curated unit must contain Markdown units read by the legacy CLI loader use YAML frontmatter. The minimum fields are `id` and `title`; `tier`, `status`, `sources`, `tables`, and `concepts` describe the unit's context. ```markdown --- id: evidence:autonomia-batteria title: Nominal battery range tier: structural status: reviewed sources: - source/domain/bicycle.md tables: - bicycle_model concepts: - concept:battery-range --- Verified definition of nominal range for an electric bicycle model. The rule must be atomic enough to cite without reconstructing an entire chapter. The text must distinguish the definition, conditions, and limits. ``` Curated units must be atomic, readable by a second reviewer, and supported by the source. Provenance references must lead back to the original file and the passage that supports the claim. Do not put secrets, tokens, passwords, or credentials in metadata or URIs. The modern canonical form also stores the Evidence kind, provenance, supporting excerpts, `source_file`, and `source_sha256`. The canonical contract rejects unknown fields, mutable metadata, and URIs containing credentials. Identifiers must remain stable even when a unit's kind changes. ## Preparation: from source to active generation ```mermaid flowchart TD SRC["source/DOMAIN/*.md\noriginal material"] --> PREP["tht evidence prepare\ncandidate preparation"] PREP --> CAND["curated/DOMAIN/*.md\nproposed or updated units"] CAND --> VAL["tht evidence validate\nstructure and link checks"] VAL -->|errors or review items| FIX["Author corrections\nand review"] FIX --> PREP VAL -->|publishable| COMMIT["Commit del repository\nauthoring clone"] COMMIT --> ING["tht preprocess evidence\nnormalization and chunking"] ING --> BM25["BM25 index"] ING --> VEC["Embeddings and vector store"] BM25 --> GEN["Candidate generation"] VEC --> GEN GEN --> ACT["Active generation"] ACT --> RUNTIME["Evidence retrieval in the workflow"] ``` Preparation can restructure changed sources, but it does not publish by itself. `prepare` produces a proposal and can identify the document involved in an error. `validate` does not write or publish. The curator publishes the revision. The runtime reads a complete, validated revision, then the pipeline creates a versioned generation. Activation is atomic, and a previous generation remains available under the retention policy. Runtime retrieval is hybrid. The dense branch uses embeddings, the BM25 branch uses lexical search, and deterministic fusion orders the results. The published unit keeps its provenance, which the model must cite when it uses the Evidence. ## Author responsibilities The author prepares the material and makes every unit verifiable. The author must: - put the original material in `source/` without changing its meaning during curation; - split the content into atomic units, with one rule or definition per unit when possible; - assign a stable identifier and a clear title; - provide provenance, tables, and related concepts when known; - keep the text in the workspace language; - separate facts, rules, examples, formulas, and limits; - include excerpts that support the unit without extending the conclusion beyond the source; - run `tht evidence prepare` and `tht evidence validate`; - resolve every error and review item before proposing a commit; - give the reviewer the necessary context, including source changes and the reason for any rename or retirement. The author must not: - write directly to the active production corpus; - treat a model proposal as a verified fact; - delete an unsupported unit without recording its retirement or relink; - put credentials in metadata, files, or provenance URLs; - manually change manifests, hashes, or generations to make validation pass. ## Reviewer responsibilities The reviewer does not approve text merely because it is clear. The reviewer checks the relationship between source, unit, and intended use. For each unit, the reviewer must check: 1. the cited source exists in the reviewed revision; 2. the excerpt actually supports the claim; 3. the unit does not combine incompatible rules or independent concepts; 4. tables, columns, and concepts are identified correctly; 5. the identifier is stable and does not duplicate another unit; 6. the text distinguishes the definition, condition, exception, and example; 7. it contains no sensitive information or details absent from the source; 8. retrieval evaluation covers relevant queries and does not hide empty results. The reviewer can approve, request changes, reject, retire, or relink a unit to a new source. Retirement must be explicit. A relink must name the new file and leave a verifiable record of the decision. Approval does not publish immediately: the repository must pass validation and the generation must pass evaluation before activation. ## Available commands Authoring commands operate on the workspace repository and do not publish directly. ```bash # Prepare changed sources. Does not commit or publish. tht evidence prepare # Reprocess all sources with the installed pipeline. tht evidence prepare --upgrade # Validate structure, manifest, links, and review items. tht evidence validate # Return JSON for CI or automated tools. tht evidence validate --json # Evaluate retrieval on a generation or the active generation. tht evidence evaluate --config tht evidence evaluate --config --generation --json # Resolve a unit without publishing: retire it or link it to a new source. tht evidence resolve evidence: --retire tht evidence resolve evidence: --source source/domain/nuovo.md # Materialize and index a versioned generation. tht preprocess evidence --config # Dry run and resume a job when supported by the configuration. tht preprocess evidence --config --dry-run tht preprocess evidence --config --resume ``` `evidence prepare`, `evidence validate`, and `evidence resolve` require the repository path. `preprocess evidence` uses the workspace configuration because it needs the embedding, vector store, retention policy, and artifact directory. Exit codes are part of the operating contract: `evidence validate` returns `0` when the corpus is publishable, `1` for validation errors, and `3` when only review items or orphaned units remain. With `--json`, stdout must contain valid JSON only. ## Formulas and session proposals Formulas use a format distinct from document Evidence. A formula proposed during a session may be cited in the current proposal, but it does not enter the runtime corpus, receive a usable `evidence:` ID, or write to the repository. To become published, it must follow the same import, review, and preprocessing path as other units. ## Contract references - [Workspace Evidence v3 contract](contracts/workspace-evidence-v3.md) - [Preprocessing CLI contract](contracts/workspace-preprocessing-cli.md)