ThothII/docs/contracts/workspace-evidence-v3.md

# Workspace Evidence contract

This is the canonical public contract for the optional `evidence` object in a schema-v4
workspace descriptor. Evidence is optional: a valid v4 descriptor without it remains operational.
When present, `evidence` is strict: it contains `source` and a defaulted strict `policy`; every
source variant and the policy reject unknown keys.

The version numbers are intentionally separate: the Evidence descriptor supports v1/v2, while the
latest Curated Evidence Unit format is [v4](curated-evidence-v4.md). There is no Evidence
descriptor v3/v4. The editable local archive is implemented in E1; installation integration
is the next increment, E2.

```mermaid
flowchart LR
    REGISTRY["Workspace registry"] --> DESCRIPTOR["Evidence descriptor"]
    DESCRIPTOR --> FILESYSTEM["Filesystem adapter"]
    DESCRIPTOR --> HTTP["HTTP adapter"]
    DESCRIPTOR --> S3["S3 adapter"]
    FILESYSTEM --> CURATED["Curated markdown"]
    HTTP --> CURATED
    S3 --> CURATED
    CURATED --> VALIDATE["Validate schema\nand provenance"]
    VALIDATE --> PREPROCESS["Preprocess pinned\nrevision"]
    PREPROCESS --> GENERATION["Versioned generation"]
    GENERATION --> ACTIVE["Active corpus"]
```

## Filesystem source

A filesystem source uses the exact URI `<workspace.id>/evidence`. `patterns` is a nonempty list of
unique, normalized relative POSIX globs. The Evidence-local `schema_version` defaults to `1` for
compatibility, where an omitted filesystem pattern defaults to `patterns: ["**/*.md"]` and
`max_bytes: 10485760`.

`evidence.schema_version: 2` declares the source/curated authoring layout. Its omitted filesystem
pattern defaults to `patterns: ["curated/**/*.md"]`; if declared, the only accepted v2 filesystem
pattern list is exactly `patterns: ["curated/**/*.md"]`. A v2 descriptor that selects `source/`,
spans both `source/` and `curated/`, uses a broader curated glob, or selects a non-Markdown file is
rejected. Explicit safe legacy filesystem patterns remain supported under Evidence version 1. HTTP
and S3 sources do not use filesystem layout patterns and retain their existing contracts.

The v2 authoring tree is:

```text
evidence/
├── source/       # preserved original material
├── curated/      # reviewed Evidence Units indexed at runtime
├── manifest.yaml
└── evaluation.yaml
```

`source/`, the manifest, the evaluation set, and other support files are materialized for
traceability but never acquired by v2 runtime preprocessing.

### Curated unit representation

The `schema_version` inside each `curated/**/*.md` file is distinct from the workspace descriptor
version above. Unit schema v1 stores the complete typed unit in YAML frontmatter. Unit schema v2
keeps short metadata in frontmatter and stores the typed payload in the body. Both remain readable
for compatibility.

Unit schema v3 stores canonical machine metadata in an invisible `tht:metadata` comment and renders
the complete review surface as deterministic Markdown. It uses headings, paragraphs, wrapping
lists, fenced SQL, blockquotes, and a collapsed technical-details block. It never emits YAML
frontmatter or Markdown tables. Invisible `tht:` comments delimit typed fields. Parsers must reject
missing, duplicate, unknown, desynchronized, or unstructured body content; they must never silently
ignore it. Domain rules also retain their exact canonical text in an invisible `tht:raw-rule`
comment while presenting long prose as paragraphs, labelled subsections, and semicolon-derived
lists. Runtime chunking reads the parsed canonical rule, not this review-only presentation.

The representations above are legacy conversion inputs. Newly prepared units use editable v4:
short YAML metadata, a visible H1 title and typed H2 payload fields, with no hidden content copy.
`tht evidence migrate <workspace-root>` converts v1–v3 to v4 and initializes a local archive
baseline without a model call, commit, activation, or semantic change. See the
[v4 editing and consolidation contract](curated-evidence-v4.md).

### Example: filesystem

```yaml
evidence:
  schema_version: 2
  source:
    type: filesystem
    uri: example/evidence
    patterns:
      - "curated/**/*.md"
    max_bytes: 10485760
  policy:
    max_chunk_chars: 4000
    retain_published_generations: 3
```

Safe: `example/evidence`. Unsafe filesystem identities include `/srv/evidence`,
`another/evidence` and `example/evidence/../another` because cross-namespace and traversal
paths are not canonical. Legacy split-layout paths are rejected as noncanonical.

## HTTP source

`uris` is a nonempty, unique list of canonical HTTP or HTTPS provenance identities. Each URI must
have no whitespace, control character, backslash, userinfo, query, or fragment. Public mode uses
`authentication: none`; signed mode uses `authentication: signed_urls_file`. Defaults are
`authentication: none`, `connect_timeout_ms: 5000`, `read_timeout_ms: 30000`,
`max_bytes: 10485760`, `max_redirects: 5`, `allow_private_hosts: false`, and
`max_cache_bytes: 67108864`. Public HTTP (`authentication: none`) uses the declared query-free
URIs directly and requires no Evidence credential file. Signed HTTP uses the installation file
contract below and preserves a mandatory one-to-one provenance mapping in declared-URI order.

### Example: http

```yaml
evidence:
  source:
    type: http
    uris:
      - https://evidence.example.invalid/report.md
    authentication: signed_urls_file
    connect_timeout_ms: 5000
    read_timeout_ms: 30000
    max_bytes: 10485760
    max_redirects: 5
    allow_private_hosts: false
    max_cache_bytes: 67108864
  policy:
    max_chunk_chars: 4000
    retain_published_generations: 3
```

The shown provenance URI is safe and query-free. An HTTP fragment identity such as
`https://evidence.example.invalid/report.md#section` is unsafe. Query-bearing and userinfo-bearing
HTTP identities are rejected; public documentation must not spell or publish a signed transport
URL.

## S3 source

`uri` is a canonical `s3://` identity with a valid bucket and no port, userinfo, query, or fragment.
`endpoint_url`, when present, is an origin-only HTTP(S) URL; `region`, when present, is nonblank.
Defaults are `credentials: ambient`, `trusted_endpoint: false`, `allow_private_endpoint: false`,
`allow_insecure_endpoint: false`, `max_bytes: 10485760`, `max_objects: 10000`, `max_pages: 100`,
and `page_size: 1000` (and page size cannot exceed 1000). A custom endpoint requires
`trusted_endpoint: true`; an HTTP endpoint additionally requires `allow_insecure_endpoint: true`.
Endpoint-policy flags cannot be enabled without `endpoint_url`. Ambient S3
(`credentials: ambient`) uses the runtime provider chain and requires no Evidence credential file.
Static mode uses `credentials: static_files`, requires access-key and secret-key files together,
and permits an optional session-token file.

### Example: s3

```yaml
evidence:
  source:
    type: s3
    uri: s3://example-evidence/curated/
    credentials: static_files
    trusted_endpoint: false
    allow_private_endpoint: false
    allow_insecure_endpoint: false
    max_bytes: 10485760
    max_objects: 10000
    max_pages: 100
    page_size: 1000
  policy:
    max_chunk_chars: 4000
    retain_published_generations: 3
```

Safe: `s3://example-evidence/curated/`. Unsafe identities include
`s3://Invalid_Bucket/evidence` and `s3://example-evidence/evidence#section`. S3 userinfo and query
identities are rejected in prose and implementation; no credential-bearing example is published.

## Policy and numeric domains

The strict policy defaults to `max_chunk_chars: 4000` and
`retain_published_generations: 3`. Filesystem `max_bytes`; HTTP `connect_timeout_ms`,
`read_timeout_ms`, `max_bytes`, and `max_cache_bytes`; S3 `max_bytes`, `max_objects`, `max_pages`,
and `page_size`; and both policy values must be positive safe integers from 1 through
9007199254740991. HTTP `max_redirects` must be a nonnegative safe integer from 0 through
9007199254740991. S3 `page_size` has the stricter maximum of 1000.

## Installation files

The namespace is the workspace ID uppercased with every `-` changed to `_`. All Evidence variables
hold file paths, never credential or signed-URL values.

| Mode | Variable | File contract |
| --- | --- | --- |
| Signed HTTP | `THT_WS_<NAMESPACE>_EVIDENCE_SIGNED_URLS_FILE` | Required for `signed_urls_file`; at most 1048576 bytes; nonempty UTF-8 JSON string array in declared-URI order; query-stripped identities must match `uris` one-to-one. |
| Static S3 pair | `THT_WS_<NAMESPACE>_EVIDENCE_ACCESS_KEY_FILE` and `THT_WS_<NAMESPACE>_EVIDENCE_SECRET_KEY_FILE` | Required together for `static_files`; each file is at most 65536 bytes. |
| Static S3 session | `THT_WS_<NAMESPACE>_EVIDENCE_SESSION_TOKEN_FILE` | Optional, valid only with the required access/secret pair, and at most 65536 bytes. |

At the connector boundary every variable is an absolute path to a readable regular file whose
resolved target is strictly below one of the roots configured by `THT_WORKSPACE_SECRET_ROOTS`.
Users enter the corresponding values through Workspace management; the backend stores them as
authenticated ciphertext and materializes these files only for a runtime lease. Scalar S3 files
are nonempty UTF-8 tokens without whitespace or NUL. Public docs, APIs, and rendered YAML never
expose file contents. `changeme`, `replace-me`, `YOUR_SECRET`, `<secret>`, access-key-looking strings, and any
credential-bearing or query-bearing URI are forbidden as public placeholder values.

## One shared workspace repository

All workspace namespaces live in one Git repository:

```text
workspace-repository.git/
├── thoth-workspaces.yaml
├── example/
│   ├── workspace.yaml
│   └── evidence/...
└── another/
    └── workspace.yaml
```

The curator-owned root catalog `thoth-workspaces.yaml` uses the `schema_version` value `1` and the ordered
`workspaces` list of `{id, name, description?}` entries. It is authoritative for workspace ID,
name, description, and display order. The descriptor at `<id>/workspace.yaml` must match the
catalog metadata exactly. Every catalog entry must have its descriptor at that same commit;
catalog-only entries are invalid and reject the complete candidate revision.

Workspace source changes only through curator Git commit/push in a separate authoring clone,
followed by an installation pull. Curator validation occurs before merge; activation and
preprocessing consume only the merged, pinned commit. The API and runtime never write
`thoth-workspaces.yaml`, `<id>/workspace.yaml`, `<id>/schema/**`, or `<id>/evidence/**` in the
authoring repository.

## Registry revision and phase ownership

| Relationship | Contract |
| --- | --- |
| Revision identity | The catalog blob, descriptor blob, and filesystem Evidence root tree are checked at the same 40-hex Git commit. |
| Content-only revision | An Evidence-only commit changes authoritative `revision.commit` even when the catalog and descriptor blobs are unchanged. |
| Repository consumer | ThothII fetches and validates a complete candidate, atomically activates it only on success, and never edits, commits, or pushes repository content. |
| Runtime secrets | Workspace management returns configured/missing status only; decrypted values exist only for the lifetime of a diagnostic or runtime lease. |
| P1.1 | Validates the lexical URI `<id>/evidence` and proves the declared filesystem root object is a Git tree at that same commit; it does not recursively inspect nested symlinks. Evidence materialization stays out of scope for P1.1. |
| P6 | Owns commit-addressed materialization of the complete Evidence tree, realpath and recursive containment, nested-symlink checks, and race checks. |

P1.1 performs no acquisition, extraction, preprocessing/indexing, embeddings, Qdrant writes,
active-snapshot retention, or GC.

## Runtime publication

After the curator has published a valid revision, the installation operator publishes it only as
part of complete workspace preprocessing:

```sh
tht --installation <absolute>/thothii-installation.yaml \
  workspace preprocess run --workspace <workspace-id>
```

This command also consumes the PostgreSQL Catalog snapshot and rebuilds schema/LSH output. There is
no public Evidence-only preprocessing command.