`.tht-dwh` — DWH generations, `OWNER.json`, ACTIVE, and fingerprints

Operator contract. P3 makes the effective DWH/preprocessing configuration reproducible and versioned across the operator CLI and the application sessions, and documents what .tht-dwh is so operators can reason about why a rerun is instant or why it takes minutes.

What `.tht-dwh` is

.tht-dwh is the workspace-local directory that stores the prepared snapshots of the data warehouse structure (the catalog physical.yaml plus the LSH hashes used for fuzzy search). ThothII does not re-read the whole database for every question: it prepares it once, stores the result here, and reuses it. The directory lives under the workspace runtime root, for example:

/data/sessions/<workspace-id>/.tht-dwh/

Immutable generations

Each preparation run produces a generation: an immutable directory containing the catalog and the LSH artifacts for one exact "effective configuration" (see fingerprints below). Generations are never modified in place; a new run writes a new generation, and an ACTIVE pointer selects which generation the workspace currently uses. Keeping the old generations makes rollback and diagnosis safe.

`OWNER.json`

Every generation root contains an OWNER.json that records who owns it:

{
  "workspace_id": "<workspace-id>",
  "config_fingerprint": "sha256:<64 hex>",
  "input_fingerprint": "sha256:<64 hex>"
}

config_fingerprint is the digest of the canonical effective configuration (see below).
input_fingerprint is the digest of the logical configuration identity.

Before reusing a generation, the harness compares the current canonical identity with the one in OWNER.json. If they differ, the generation is refused (never silently reused) and a new one is produced. This is what protects ThothII from using artifacts prepared for a different database, endpoint, user, schema, or index contract.

The reader is compatible with the historical schema-v1 OWNER.json (same three keys, sha256: values) so existing installations keep working; new writes use the versioned computation. There is no automatic in-place reinterpretation: operators regenerate explicitly when a root is old.

The canonical effective configuration and the logical identity

The canonical effective configuration is the non-secret subset of the rendered runtime configuration that determines whether a prepared DWH generation is still valid:

{
  "schemaVersion": 1,
  "dwh": {
    "engine": "postgres",
    "database": "<database>",
    "schema": "<schema>",
    "transport": "postgres_direct | rest_api | ...",
    "host": "<host>",
    "port": 5432,
    "baseUrl": "<base-url>",
    "user": "<user>"
  },
  "vector": { "collection": "<collection>", "dimensions": 1024, "distance": "cosine" },
  "embedding": { "model": "<model>", "dimensions": 1024 },
  "roots": { "artifacts": "<abs-path>", "indexes": "<abs-path>" }
}

Deliberately excluded (their change must not invalidate a DWH generation):

session_storage and runtime_identity (a content-only Git commit or an Evidence-only change must not force a full database re-introspection);
Evidence source/policy (P6 materialization and Evidence preprocessing are separate);
memory, search, and execution settings;
all credentials (passwords, API keys, signed URLs, and secret-file paths).

The logical configuration identity is:

workspace://<workspace-id>@v1:<sha256 of the canonical effective configuration>

It is the same for the operator CLI and for application sessions, because both derive it from the same rendered configuration. That is the guarantee that the work prepared by thothctl is exactly what the sessions will consume.

Why a rerun can be instant or take minutes

Same canonical identity (e.g., only Evidence files changed) → the generation is reused → the DWH step is unchanged and fast.
Changed canonical identity (different database, address, user, schema, collection, model, or artifact/index roots) → the old generation is refused → ThothII re-introspects and writes a new generation → the step takes as long as the first preparation.

Safe migration, regeneration, and recovery

Migration: existing schema-v1 OWNER.json roots are readable; to switch them to the versioned identity, run a normal regeneration (explicit --refresh/new run). No automatic in-place rewrite.
Regeneration: a new run produces a new immutable generation and moves ACTIVE; the previous generations remain for rollback.
Recovery: if the active generation is corrupt or owned by another configuration, ThothII fails closed (never mixes artifacts) and tells the operator to regenerate; the old generations are still available for inspection.

Memory root

P3 also gives each workspace an explicit workspace-global memory root:

/data/sessions/<workspace-id>/memory/

All memory commands, locks, the canonical JSONL registry, and the Qdrant projection use this root when present. A guarded migration copies and verifies exactly one legacy canonical JSONL from the old artifacts/memory location under the workspace lock and rebuilds the projection; conflicting legacy registries fail closed. There is no in-place reinterpretation.

Revision-scoped search records

Schema and Evidence records in the Qdrant collection include the pinned workspace_revision, so searches never mix descriptions or documents from different versions of the workspace. Memory and solved-question records remain workspace-wide on purpose.

5.5 KiB Raw Blame History

.tht-dwh — DWH generations, OWNER.json, ACTIVE, and fingerprints

What .tht-dwh is