feat: complete catalog-driven preprocessing
Publish documentation / publish (push) Successful in 2m12s
Publish documentation / publish (push) Successful in 2m12s
This commit is contained in:
Codex
+28
-121
@@ -1,128 +1,35 @@
|
||||
# `.tht-dwh` — DWH generations, `OWNER.json`, ACTIVE, and fingerprints
|
||||
# Internal Catalog-bound DWH artifacts
|
||||
|
||||
> Operator contract. P3 makes the effective DWH/preprocessing configuration reproducible and
|
||||
> versioned across the operator CLI and the application sessions, and documents what `.tht-dwh`
|
||||
> is so operators can reason about why a rerun is instant or why it takes minutes.
|
||||
`.tht-dwh`, `OWNER.json`, and the `ACTIVE` generation pointer are private preprocessing artifacts,
|
||||
not an operator-facing generation or rollback contract. Complete preprocessing materializes the
|
||||
physical schema from the immutable PostgreSQL Catalog Metadata Snapshot and samples only eligible
|
||||
DWH values to build LSH. It never obtains schema metadata from workspace YAML or by introspecting the
|
||||
DWH in the harness.
|
||||
|
||||
## What `.tht-dwh` is
|
||||
`OWNER.json` binds the artifact root to all of:
|
||||
|
||||
`.tht-dwh` is the workspace-local directory that stores the **prepared snapshots of the data
|
||||
warehouse structure** (the catalog `physical.yaml` plus the LSH hashes used for fuzzy search).
|
||||
ThothII does not re-read the whole database for every question: it prepares it once, stores the
|
||||
result here, and reuses it. The directory lives under the workspace runtime root, for example:
|
||||
- workspace ID;
|
||||
- Catalog database ID;
|
||||
- Metadata Content Revision;
|
||||
- effective configuration fingerprint;
|
||||
- input fingerprint.
|
||||
|
||||
```text
|
||||
/data/sessions/<workspace-id>/.tht-dwh/
|
||||
The complete binding must match before artifacts can be reused. This prevents an LSH built from one
|
||||
database, workspace, Catalog revision, or effective configuration from being associated with
|
||||
another. PostgreSQL separately owns readiness through `running | succeeded | failed`, the processed
|
||||
revision, and the preprocessing input fingerprint.
|
||||
|
||||
The internal generation is overwritten by normal preprocessing and is not retained for rollback.
|
||||
Recovery is always:
|
||||
|
||||
```sh
|
||||
tht --installation <absolute>/thothii-installation.yaml \
|
||||
workspace preprocess run --workspace <workspace-id>
|
||||
```
|
||||
|
||||
## Immutable generations
|
||||
The clear operation removes `.tht-dwh` together with the LSH, private Catalog snapshot, Evidence
|
||||
corpus, and derived checkpoints. The next run recreates them from PostgreSQL and the pinned workspace
|
||||
revision.
|
||||
|
||||
Each preparation run produces a **generation**: an immutable directory containing the catalog and
|
||||
the LSH artifacts for one exact "effective configuration" (see fingerprints below). Generations
|
||||
are never modified in place; a new run writes a new generation, and an `ACTIVE` pointer selects
|
||||
which generation the workspace currently uses. Keeping the old generations makes rollback and
|
||||
diagnosis safe.
|
||||
|
||||
## `OWNER.json`
|
||||
|
||||
Every generation root contains an `OWNER.json` that records who owns it:
|
||||
|
||||
```json
|
||||
{
|
||||
"workspace_id": "<workspace-id>",
|
||||
"config_fingerprint": "sha256:<64 hex>",
|
||||
"input_fingerprint": "sha256:<64 hex>"
|
||||
}
|
||||
```
|
||||
|
||||
- `config_fingerprint` is the digest of the **canonical effective configuration** (see below).
|
||||
- `input_fingerprint` is the digest of the **logical configuration identity**.
|
||||
|
||||
Before reusing a generation, the harness compares the current canonical identity with the one in
|
||||
`OWNER.json`. If they differ, the generation is **refused** (never silently reused) and a new one
|
||||
is produced. This is what protects ThothII from using artifacts prepared for a different database,
|
||||
endpoint, user, schema, or index contract.
|
||||
|
||||
The reader is compatible with the historical schema-v1 `OWNER.json` (same three keys, `sha256:`
|
||||
values) so existing installations keep working; new writes use the versioned computation. There is
|
||||
no automatic in-place reinterpretation: operators regenerate explicitly when a root is old.
|
||||
|
||||
## The canonical effective configuration and the logical identity
|
||||
|
||||
The **canonical effective configuration** is the non-secret subset of the rendered runtime
|
||||
configuration that determines whether a prepared DWH generation is still valid:
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"dwh": {
|
||||
"engine": "postgres",
|
||||
"database": "<database>",
|
||||
"schema": "<schema>",
|
||||
"transport": "postgres_direct | rest_api | ...",
|
||||
"host": "<host>",
|
||||
"port": 5432,
|
||||
"baseUrl": "<base-url>",
|
||||
"user": "<user>"
|
||||
},
|
||||
"vector": { "collection": "<collection>", "dimensions": 1024, "distance": "cosine" },
|
||||
"embedding": { "model": "<model>", "dimensions": 1024 },
|
||||
"roots": { "artifacts": "<abs-path>", "indexes": "<abs-path>" }
|
||||
}
|
||||
```
|
||||
|
||||
Deliberately **excluded** (their change must not invalidate a DWH generation):
|
||||
|
||||
- `session_storage` and `runtime_identity` (a content-only Git commit or an Evidence-only change
|
||||
must not force a full database re-introspection);
|
||||
- Evidence source/policy (P6 materialization and Evidence preprocessing are separate);
|
||||
- memory, search, and execution settings;
|
||||
- **all credentials** (passwords, API keys, signed URLs, and secret-file paths).
|
||||
|
||||
The **logical configuration identity** is:
|
||||
|
||||
```text
|
||||
workspace://<workspace-id>@v1:<sha256 of the canonical effective configuration>
|
||||
```
|
||||
|
||||
It is the same for the operator CLI and for application sessions, because both derive it from the
|
||||
same rendered configuration. That is the guarantee that the work prepared by `tht` is exactly
|
||||
what the sessions will consume.
|
||||
|
||||
## Why a rerun can be instant or take minutes
|
||||
|
||||
- Same canonical identity (e.g., only Evidence files changed) → the generation is reused → the
|
||||
DWH step is `unchanged` and fast.
|
||||
- Changed canonical identity (different database, address, user, schema, collection, model, or
|
||||
artifact/index roots) → the old generation is refused → ThothII re-introspects and writes a new
|
||||
generation → the step takes as long as the first preparation.
|
||||
|
||||
## Safe migration, regeneration, and recovery
|
||||
|
||||
- **Migration**: existing schema-v1 `OWNER.json` roots are readable; to switch them to the
|
||||
versioned identity, run a normal regeneration (explicit `--refresh`/new run). No automatic
|
||||
in-place rewrite.
|
||||
- **Regeneration**: a new run produces a new immutable generation and moves `ACTIVE`; the previous
|
||||
generations remain for rollback.
|
||||
- **Recovery**: if the active generation is corrupt or owned by another configuration, ThothII
|
||||
fails closed (never mixes artifacts) and tells the operator to regenerate; the old generations
|
||||
are still available for inspection.
|
||||
|
||||
## Memory root
|
||||
|
||||
P3 also gives each workspace an explicit **workspace-global memory root**:
|
||||
|
||||
```text
|
||||
/data/sessions/<workspace-id>/memory/
|
||||
```
|
||||
|
||||
All memory commands, locks, the canonical JSONL registry, and the Qdrant projection use this root
|
||||
when present. A guarded migration copies and verifies exactly one legacy canonical JSONL from the
|
||||
old `artifacts/memory` location under the workspace lock and rebuilds the projection; conflicting
|
||||
legacy registries fail closed. There is no in-place reinterpretation.
|
||||
|
||||
## Revision-scoped search records
|
||||
|
||||
Schema and Evidence records in the Qdrant collection include the pinned `workspace_revision`, so
|
||||
searches never mix descriptions or documents from different versions of the workspace. Memory and
|
||||
solved-question records remain workspace-wide on purpose.
|
||||
Workspace-global `memory` and Qdrant `solved_question` data are not preprocessing output and remain
|
||||
preserved both when reference data is overwritten and when preprocessing is cleared.
|
||||
|
||||
Reference in New Issue
Block a user