ThothII/docs/contracts/workspace-evidence-v3.md

Workspace Evidence contract

This is the canonical public contract for the optional evidence object in a schema-v4 workspace descriptor. Evidence is optional: a valid v4 descriptor without it remains operational. When present, evidence is strict: it contains source and a defaulted strict policy; every source variant and the policy reject unknown keys.

The version numbers are intentionally separate: the Evidence descriptor supports v1/v2, while the latest Curated Evidence Unit format is v4. There is no Evidence descriptor v3/v4. The editable local archive is implemented in E1; installation integration is the next increment, E2.

flowchart LR
    REGISTRY["Workspace registry"] --> DESCRIPTOR["Evidence descriptor"]
    DESCRIPTOR --> FILESYSTEM["Filesystem adapter"]
    DESCRIPTOR --> HTTP["HTTP adapter"]
    DESCRIPTOR --> S3["S3 adapter"]
    FILESYSTEM --> CURATED["Curated markdown"]
    HTTP --> CURATED
    S3 --> CURATED
    CURATED --> VALIDATE["Validate schema\nand provenance"]
    VALIDATE --> PREPROCESS["Preprocess pinned\nrevision"]
    PREPROCESS --> GENERATION["Versioned generation"]
    GENERATION --> ACTIVE["Active corpus"]

Filesystem source

A filesystem source uses the exact URI <workspace.id>/evidence. patterns is a nonempty list of unique, normalized relative POSIX globs. The Evidence-local schema_version defaults to 1 for compatibility, where an omitted filesystem pattern defaults to patterns: ["**/*.md"] and max_bytes: 10485760.

evidence.schema_version: 2 declares the source/curated authoring layout. Its omitted filesystem pattern defaults to patterns: ["curated/**/*.md"]; if declared, the only accepted v2 filesystem pattern list is exactly patterns: ["curated/**/*.md"]. A v2 descriptor that selects source/, spans both source/ and curated/, uses a broader curated glob, or selects a non-Markdown file is rejected. Explicit safe legacy filesystem patterns remain supported under Evidence version 1. HTTP and S3 sources do not use filesystem layout patterns and retain their existing contracts.

The v2 authoring tree is:

evidence/
├── source/       # preserved original material
├── curated/      # reviewed Evidence Units indexed at runtime
├── manifest.yaml
└── evaluation.yaml

source/, the manifest, the evaluation set, and other support files are materialized for traceability but never acquired by v2 runtime preprocessing.

Curated unit representation

The schema_version inside each curated/**/*.md file is distinct from the workspace descriptor version above. Unit schema v1 stores the complete typed unit in YAML frontmatter. Unit schema v2 keeps short metadata in frontmatter and stores the typed payload in the body. Both remain readable for compatibility.

Unit schema v3 stores canonical machine metadata in an invisible tht:metadata comment and renders the complete review surface as deterministic Markdown. It uses headings, paragraphs, wrapping lists, fenced SQL, blockquotes, and a collapsed technical-details block. It never emits YAML frontmatter or Markdown tables. Invisible tht: comments delimit typed fields. Parsers must reject missing, duplicate, unknown, desynchronized, or unstructured body content; they must never silently ignore it. Domain rules also retain their exact canonical text in an invisible tht:raw-rule comment while presenting long prose as paragraphs, labelled subsections, and semicolon-derived lists. Runtime chunking reads the parsed canonical rule, not this review-only presentation.

The representations above are legacy conversion inputs. Newly prepared units use editable v4: short YAML metadata, a visible H1 title and typed H2 payload fields, with no hidden content copy. tht evidence migrate <workspace-root> converts v1–v3 to v4 and initializes a local archive baseline without a model call, commit, activation, or semantic change. See the v4 editing and consolidation contract.

Example: filesystem

evidence:
  schema_version: 2
  source:
    type: filesystem
    uri: example/evidence
    patterns:
      - "curated/**/*.md"
    max_bytes: 10485760
  policy:
    max_chunk_chars: 4000
    retain_published_generations: 3

Safe: example/evidence. Unsafe filesystem identities include /srv/evidence, another/evidence and example/evidence/../another because cross-namespace and traversal paths are not canonical. Legacy split-layout paths are rejected as noncanonical.

HTTP source

uris is a nonempty, unique list of canonical HTTP or HTTPS provenance identities. Each URI must have no whitespace, control character, backslash, userinfo, query, or fragment. Public mode uses authentication: none; signed mode uses authentication: signed_urls_file. Defaults are authentication: none, connect_timeout_ms: 5000, read_timeout_ms: 30000, max_bytes: 10485760, max_redirects: 5, allow_private_hosts: false, and max_cache_bytes: 67108864. Public HTTP (authentication: none) uses the declared query-free URIs directly and requires no Evidence credential file. Signed HTTP uses the installation file contract below and preserves a mandatory one-to-one provenance mapping in declared-URI order.

Example: http

evidence:
  source:
    type: http
    uris:
      - https://evidence.example.invalid/report.md
    authentication: signed_urls_file
    connect_timeout_ms: 5000
    read_timeout_ms: 30000
    max_bytes: 10485760
    max_redirects: 5
    allow_private_hosts: false
    max_cache_bytes: 67108864
  policy:
    max_chunk_chars: 4000
    retain_published_generations: 3

The shown provenance URI is safe and query-free. An HTTP fragment identity such as https://evidence.example.invalid/report.md#section is unsafe. Query-bearing and userinfo-bearing HTTP identities are rejected; public documentation must not spell or publish a signed transport URL.

S3 source

uri is a canonical s3:// identity with a valid bucket and no port, userinfo, query, or fragment. endpoint_url, when present, is an origin-only HTTP(S) URL; region, when present, is nonblank. Defaults are credentials: ambient, trusted_endpoint: false, allow_private_endpoint: false, allow_insecure_endpoint: false, max_bytes: 10485760, max_objects: 10000, max_pages: 100, and page_size: 1000 (and page size cannot exceed 1000). A custom endpoint requires trusted_endpoint: true; an HTTP endpoint additionally requires allow_insecure_endpoint: true. Endpoint-policy flags cannot be enabled without endpoint_url. Ambient S3 (credentials: ambient) uses the runtime provider chain and requires no Evidence credential file. Static mode uses credentials: static_files, requires access-key and secret-key files together, and permits an optional session-token file.

Example: s3

evidence:
  source:
    type: s3
    uri: s3://example-evidence/curated/
    credentials: static_files
    trusted_endpoint: false
    allow_private_endpoint: false
    allow_insecure_endpoint: false
    max_bytes: 10485760
    max_objects: 10000
    max_pages: 100
    page_size: 1000
  policy:
    max_chunk_chars: 4000
    retain_published_generations: 3

Safe: s3://example-evidence/curated/. Unsafe identities include s3://Invalid_Bucket/evidence and s3://example-evidence/evidence#section. S3 userinfo and query identities are rejected in prose and implementation; no credential-bearing example is published.

Policy and numeric domains

The strict policy defaults to max_chunk_chars: 4000 and retain_published_generations: 3. Filesystem max_bytes; HTTP connect_timeout_ms, read_timeout_ms, max_bytes, and max_cache_bytes; S3 max_bytes, max_objects, max_pages, and page_size; and both policy values must be positive safe integers from 1 through 9007199254740991. HTTP max_redirects must be a nonnegative safe integer from 0 through 9007199254740991. S3 page_size has the stricter maximum of 1000.

Installation files

The namespace is the workspace ID uppercased with every - changed to _. All Evidence variables hold file paths, never credential or signed-URL values.

Mode	Variable	File contract
Signed HTTP	THT_WS_<NAMESPACE>_EVIDENCE_SIGNED_URLS_FILE	Required for signed_urls_file; at most 1048576 bytes; nonempty UTF-8 JSON string array in declared-URI order; query-stripped identities must match uris one-to-one.
Static S3 pair	THT_WS_<NAMESPACE>_EVIDENCE_ACCESS_KEY_FILE and THT_WS_<NAMESPACE>_EVIDENCE_SECRET_KEY_FILE	Required together for static_files; each file is at most 65536 bytes.
Static S3 session	THT_WS_<NAMESPACE>_EVIDENCE_SESSION_TOKEN_FILE	Optional, valid only with the required access/secret pair, and at most 65536 bytes.

At the connector boundary every variable is an absolute path to a readable regular file whose resolved target is strictly below one of the roots configured by THT_WORKSPACE_SECRET_ROOTS. Users enter the corresponding values through Workspace management; the backend stores them as authenticated ciphertext and materializes these files only for a runtime lease. Scalar S3 files are nonempty UTF-8 tokens without whitespace or NUL. Public docs, APIs, and rendered YAML never expose file contents. changeme, replace-me, YOUR_SECRET, <secret>, access-key-looking strings, and any credential-bearing or query-bearing URI are forbidden as public placeholder values.

One shared workspace repository

All workspace namespaces live in one Git repository:

workspace-repository.git/
├── thoth-workspaces.yaml
├── example/
│   ├── workspace.yaml
│   └── evidence/...
└── another/
    └── workspace.yaml

The curator-owned root catalog thoth-workspaces.yaml uses the schema_version value 1 and the ordered workspaces list of {id, name, description?} entries. It is authoritative for workspace ID, name, description, and display order. The descriptor at <id>/workspace.yaml must match the catalog metadata exactly. Every catalog entry must have its descriptor at that same commit; catalog-only entries are invalid and reject the complete candidate revision.

Workspace source changes only through curator Git commit/push in a separate authoring clone, followed by an installation pull. Curator validation occurs before merge; activation and preprocessing consume only the merged, pinned commit. The API and runtime never write thoth-workspaces.yaml, <id>/workspace.yaml, <id>/schema/**, or <id>/evidence/** in the authoring repository.

Registry revision and phase ownership

Relationship	Contract
Revision identity	The catalog blob, descriptor blob, and filesystem Evidence root tree are checked at the same 40-hex Git commit.
Content-only revision	An Evidence-only commit changes authoritative revision.commit even when the catalog and descriptor blobs are unchanged.
Repository consumer	ThothII fetches and validates a complete candidate, atomically activates it only on success, and never edits, commits, or pushes repository content.
Runtime secrets	Workspace management returns configured/missing status only; decrypted values exist only for the lifetime of a diagnostic or runtime lease.
P1.1	Validates the lexical URI <id>/evidence and proves the declared filesystem root object is a Git tree at that same commit; it does not recursively inspect nested symlinks. Evidence materialization stays out of scope for P1.1.
P6	Owns commit-addressed materialization of the complete Evidence tree, realpath and recursive containment, nested-symlink checks, and race checks.

P1.1 performs no acquisition, extraction, preprocessing/indexing, embeddings, Qdrant writes, active-snapshot retention, or GC.

Runtime publication

After the curator has published a valid revision, the installation operator publishes it only as part of complete workspace preprocessing:

tht --installation <absolute>/thothii-installation.yaml \
  workspace preprocess run --workspace <workspace-id>

This command also consumes the PostgreSQL Catalog snapshot and rebuilds schema/LSH output. There is no public Evidence-only preprocessing command.