feat: P3 effective configuration, memory root, and revision-scoped records

2026-08-12 16:01:25 +02:00
parent beaba548c1
commit e23e526966
21 changed files with 1108 additions and 44 deletions
@@ -0,0 +1,128 @@
+# `.tht-dwh` — DWH generations, `OWNER.json`, ACTIVE, and fingerprints
+
+> Operator contract. P3 makes the effective DWH/preprocessing configuration reproducible and
+> versioned across the operator CLI and the application sessions, and documents what `.tht-dwh`
+> is so operators can reason about why a rerun is instant or why it takes minutes.
+
+## What `.tht-dwh` is
+
+`.tht-dwh` is the workspace-local directory that stores the **prepared snapshots of the data
+warehouse structure** (the catalog `physical.yaml` plus the LSH hashes used for fuzzy search).
+ThothII does not re-read the whole database for every question: it prepares it once, stores the
+result here, and reuses it. The directory lives under the workspace runtime root, for example:
+
+```text
+/data/sessions/<workspace-id>/.tht-dwh/
+```
+
+## Immutable generations
+
+Each preparation run produces a **generation**: an immutable directory containing the catalog and
+the LSH artifacts for one exact "effective configuration" (see fingerprints below). Generations
+are never modified in place; a new run writes a new generation, and an `ACTIVE` pointer selects
+which generation the workspace currently uses. Keeping the old generations makes rollback and
+diagnosis safe.
+
+## `OWNER.json`
+
+Every generation root contains an `OWNER.json` that records who owns it:
+
+```json
+{
+  "workspace_id": "<workspace-id>",
+  "config_fingerprint": "sha256:<64 hex>",
+  "input_fingerprint": "sha256:<64 hex>"
+}
+```
+
+- `config_fingerprint` is the digest of the **canonical effective configuration** (see below).
+- `input_fingerprint` is the digest of the **logical configuration identity**.
+
+Before reusing a generation, the harness compares the current canonical identity with the one in
+`OWNER.json`. If they differ, the generation is **refused** (never silently reused) and a new one
+is produced. This is what protects ThothII from using artifacts prepared for a different database,
+endpoint, user, schema, or index contract.
+
+The reader is compatible with the historical schema-v1 `OWNER.json` (same three keys, `sha256:`
+values) so existing installations keep working; new writes use the versioned computation. There is
+no automatic in-place reinterpretation: operators regenerate explicitly when a root is old.
+
+## The canonical effective configuration and the logical identity
+
+The **canonical effective configuration** is the non-secret subset of the rendered runtime
+configuration that determines whether a prepared DWH generation is still valid:
+
+```json
+{
+  "schemaVersion": 1,
+  "dwh": {
+    "engine": "postgres",
+    "database": "<database>",
+    "schema": "<schema>",
+    "transport": "postgres_direct | rest_api | ...",
+    "host": "<host>",
+    "port": 5432,
+    "baseUrl": "<base-url>",
+    "user": "<user>"
+  },
+  "vector": { "collection": "<collection>", "dimensions": 1024, "distance": "cosine" },
+  "embedding": { "model": "<model>", "dimensions": 1024 },
+  "roots": { "artifacts": "<abs-path>", "indexes": "<abs-path>" }
+}
+```
+
+Deliberately **excluded** (their change must not invalidate a DWH generation):
+
+- `session_storage` and `runtime_identity` (a content-only Git commit or an Evidence-only change
+  must not force a full database re-introspection);
+- Evidence source/policy (P6 materialization and Evidence preprocessing are separate);
+- memory, search, and execution settings;
+- **all credentials** (passwords, API keys, signed URLs, and secret-file paths).
+
+The **logical configuration identity** is:
+
+```text
+workspace://<workspace-id>@v1:<sha256 of the canonical effective configuration>
+```
+
+It is the same for the operator CLI and for application sessions, because both derive it from the
+same rendered configuration. That is the guarantee that the work prepared by `thothctl` is exactly
+what the sessions will consume.
+
+## Why a rerun can be instant or take minutes
+
+- Same canonical identity (e.g., only Evidence files changed) → the generation is reused → the
+  DWH step is `unchanged` and fast.
+- Changed canonical identity (different database, address, user, schema, collection, model, or
+  artifact/index roots) → the old generation is refused → ThothII re-introspects and writes a new
+  generation → the step takes as long as the first preparation.
+
+## Safe migration, regeneration, and recovery
+
+- **Migration**: existing schema-v1 `OWNER.json` roots are readable; to switch them to the
+  versioned identity, run a normal regeneration (explicit `--refresh`/new run). No automatic
+  in-place rewrite.
+- **Regeneration**: a new run produces a new immutable generation and moves `ACTIVE`; the previous
+  generations remain for rollback.
+- **Recovery**: if the active generation is corrupt or owned by another configuration, ThothII
+  fails closed (never mixes artifacts) and tells the operator to regenerate; the old generations
+  are still available for inspection.
+
+## Memory root
+
+P3 also gives each workspace an explicit **workspace-global memory root**:
+
+```text
+/data/sessions/<workspace-id>/memory/
+```
+
+All memory commands, locks, the canonical JSONL registry, and the Qdrant projection use this root
+when present. A guarded migration copies and verifies exactly one legacy canonical JSONL from the
+old `artifacts/memory` location under the workspace lock and rebuilds the projection; conflicting
+legacy registries fail closed. There is no in-place reinterpretation.
+
+## Revision-scoped search records
+
+Schema and Evidence records in the Qdrant collection include the pinned `workspace_revision`, so
+searches never mix descriptions or documents from different versions of the workspace. Memory and
+solved-question records remain workspace-wide on purpose.