129 lines
5.5 KiB
Markdown
129 lines
5.5 KiB
Markdown
# `.tht-dwh` — DWH generations, `OWNER.json`, ACTIVE, and fingerprints
|
|
|
|
> Operator contract. P3 makes the effective DWH/preprocessing configuration reproducible and
|
|
> versioned across the operator CLI and the application sessions, and documents what `.tht-dwh`
|
|
> is so operators can reason about why a rerun is instant or why it takes minutes.
|
|
|
|
## What `.tht-dwh` is
|
|
|
|
`.tht-dwh` is the workspace-local directory that stores the **prepared snapshots of the data
|
|
warehouse structure** (the catalog `physical.yaml` plus the LSH hashes used for fuzzy search).
|
|
ThothII does not re-read the whole database for every question: it prepares it once, stores the
|
|
result here, and reuses it. The directory lives under the workspace runtime root, for example:
|
|
|
|
```text
|
|
/data/sessions/<workspace-id>/.tht-dwh/
|
|
```
|
|
|
|
## Immutable generations
|
|
|
|
Each preparation run produces a **generation**: an immutable directory containing the catalog and
|
|
the LSH artifacts for one exact "effective configuration" (see fingerprints below). Generations
|
|
are never modified in place; a new run writes a new generation, and an `ACTIVE` pointer selects
|
|
which generation the workspace currently uses. Keeping the old generations makes rollback and
|
|
diagnosis safe.
|
|
|
|
## `OWNER.json`
|
|
|
|
Every generation root contains an `OWNER.json` that records who owns it:
|
|
|
|
```json
|
|
{
|
|
"workspace_id": "<workspace-id>",
|
|
"config_fingerprint": "sha256:<64 hex>",
|
|
"input_fingerprint": "sha256:<64 hex>"
|
|
}
|
|
```
|
|
|
|
- `config_fingerprint` is the digest of the **canonical effective configuration** (see below).
|
|
- `input_fingerprint` is the digest of the **logical configuration identity**.
|
|
|
|
Before reusing a generation, the harness compares the current canonical identity with the one in
|
|
`OWNER.json`. If they differ, the generation is **refused** (never silently reused) and a new one
|
|
is produced. This is what protects ThothII from using artifacts prepared for a different database,
|
|
endpoint, user, schema, or index contract.
|
|
|
|
The reader is compatible with the historical schema-v1 `OWNER.json` (same three keys, `sha256:`
|
|
values) so existing installations keep working; new writes use the versioned computation. There is
|
|
no automatic in-place reinterpretation: operators regenerate explicitly when a root is old.
|
|
|
|
## The canonical effective configuration and the logical identity
|
|
|
|
The **canonical effective configuration** is the non-secret subset of the rendered runtime
|
|
configuration that determines whether a prepared DWH generation is still valid:
|
|
|
|
```json
|
|
{
|
|
"schemaVersion": 1,
|
|
"dwh": {
|
|
"engine": "postgres",
|
|
"database": "<database>",
|
|
"schema": "<schema>",
|
|
"transport": "postgres_direct | rest_api | ...",
|
|
"host": "<host>",
|
|
"port": 5432,
|
|
"baseUrl": "<base-url>",
|
|
"user": "<user>"
|
|
},
|
|
"vector": { "collection": "<collection>", "dimensions": 1024, "distance": "cosine" },
|
|
"embedding": { "model": "<model>", "dimensions": 1024 },
|
|
"roots": { "artifacts": "<abs-path>", "indexes": "<abs-path>" }
|
|
}
|
|
```
|
|
|
|
Deliberately **excluded** (their change must not invalidate a DWH generation):
|
|
|
|
- `session_storage` and `runtime_identity` (a content-only Git commit or an Evidence-only change
|
|
must not force a full database re-introspection);
|
|
- Evidence source/policy (P6 materialization and Evidence preprocessing are separate);
|
|
- memory, search, and execution settings;
|
|
- **all credentials** (passwords, API keys, signed URLs, and secret-file paths).
|
|
|
|
The **logical configuration identity** is:
|
|
|
|
```text
|
|
workspace://<workspace-id>@v1:<sha256 of the canonical effective configuration>
|
|
```
|
|
|
|
It is the same for the operator CLI and for application sessions, because both derive it from the
|
|
same rendered configuration. That is the guarantee that the work prepared by `thothctl` is exactly
|
|
what the sessions will consume.
|
|
|
|
## Why a rerun can be instant or take minutes
|
|
|
|
- Same canonical identity (e.g., only Evidence files changed) → the generation is reused → the
|
|
DWH step is `unchanged` and fast.
|
|
- Changed canonical identity (different database, address, user, schema, collection, model, or
|
|
artifact/index roots) → the old generation is refused → ThothII re-introspects and writes a new
|
|
generation → the step takes as long as the first preparation.
|
|
|
|
## Safe migration, regeneration, and recovery
|
|
|
|
- **Migration**: existing schema-v1 `OWNER.json` roots are readable; to switch them to the
|
|
versioned identity, run a normal regeneration (explicit `--refresh`/new run). No automatic
|
|
in-place rewrite.
|
|
- **Regeneration**: a new run produces a new immutable generation and moves `ACTIVE`; the previous
|
|
generations remain for rollback.
|
|
- **Recovery**: if the active generation is corrupt or owned by another configuration, ThothII
|
|
fails closed (never mixes artifacts) and tells the operator to regenerate; the old generations
|
|
are still available for inspection.
|
|
|
|
## Memory root
|
|
|
|
P3 also gives each workspace an explicit **workspace-global memory root**:
|
|
|
|
```text
|
|
/data/sessions/<workspace-id>/memory/
|
|
```
|
|
|
|
All memory commands, locks, the canonical JSONL registry, and the Qdrant projection use this root
|
|
when present. A guarded migration copies and verifies exactly one legacy canonical JSONL from the
|
|
old `artifacts/memory` location under the workspace lock and rebuilds the projection; conflicting
|
|
legacy registries fail closed. There is no in-place reinterpretation.
|
|
|
|
## Revision-scoped search records
|
|
|
|
Schema and Evidence records in the Qdrant collection include the pinned `workspace_revision`, so
|
|
searches never mix descriptions or documents from different versions of the workspace. Memory and
|
|
solved-question records remain workspace-wide on purpose.
|