feat: P3 effective configuration, memory root, and revision-scoped records
This commit is contained in:
@@ -0,0 +1,128 @@
|
||||
# `.tht-dwh` — DWH generations, `OWNER.json`, ACTIVE, and fingerprints
|
||||
|
||||
> Operator contract. P3 makes the effective DWH/preprocessing configuration reproducible and
|
||||
> versioned across the operator CLI and the application sessions, and documents what `.tht-dwh`
|
||||
> is so operators can reason about why a rerun is instant or why it takes minutes.
|
||||
|
||||
## What `.tht-dwh` is
|
||||
|
||||
`.tht-dwh` is the workspace-local directory that stores the **prepared snapshots of the data
|
||||
warehouse structure** (the catalog `physical.yaml` plus the LSH hashes used for fuzzy search).
|
||||
ThothII does not re-read the whole database for every question: it prepares it once, stores the
|
||||
result here, and reuses it. The directory lives under the workspace runtime root, for example:
|
||||
|
||||
```text
|
||||
/data/sessions/<workspace-id>/.tht-dwh/
|
||||
```
|
||||
|
||||
## Immutable generations
|
||||
|
||||
Each preparation run produces a **generation**: an immutable directory containing the catalog and
|
||||
the LSH artifacts for one exact "effective configuration" (see fingerprints below). Generations
|
||||
are never modified in place; a new run writes a new generation, and an `ACTIVE` pointer selects
|
||||
which generation the workspace currently uses. Keeping the old generations makes rollback and
|
||||
diagnosis safe.
|
||||
|
||||
## `OWNER.json`
|
||||
|
||||
Every generation root contains an `OWNER.json` that records who owns it:
|
||||
|
||||
```json
|
||||
{
|
||||
"workspace_id": "<workspace-id>",
|
||||
"config_fingerprint": "sha256:<64 hex>",
|
||||
"input_fingerprint": "sha256:<64 hex>"
|
||||
}
|
||||
```
|
||||
|
||||
- `config_fingerprint` is the digest of the **canonical effective configuration** (see below).
|
||||
- `input_fingerprint` is the digest of the **logical configuration identity**.
|
||||
|
||||
Before reusing a generation, the harness compares the current canonical identity with the one in
|
||||
`OWNER.json`. If they differ, the generation is **refused** (never silently reused) and a new one
|
||||
is produced. This is what protects ThothII from using artifacts prepared for a different database,
|
||||
endpoint, user, schema, or index contract.
|
||||
|
||||
The reader is compatible with the historical schema-v1 `OWNER.json` (same three keys, `sha256:`
|
||||
values) so existing installations keep working; new writes use the versioned computation. There is
|
||||
no automatic in-place reinterpretation: operators regenerate explicitly when a root is old.
|
||||
|
||||
## The canonical effective configuration and the logical identity
|
||||
|
||||
The **canonical effective configuration** is the non-secret subset of the rendered runtime
|
||||
configuration that determines whether a prepared DWH generation is still valid:
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"dwh": {
|
||||
"engine": "postgres",
|
||||
"database": "<database>",
|
||||
"schema": "<schema>",
|
||||
"transport": "postgres_direct | rest_api | ...",
|
||||
"host": "<host>",
|
||||
"port": 5432,
|
||||
"baseUrl": "<base-url>",
|
||||
"user": "<user>"
|
||||
},
|
||||
"vector": { "collection": "<collection>", "dimensions": 1024, "distance": "cosine" },
|
||||
"embedding": { "model": "<model>", "dimensions": 1024 },
|
||||
"roots": { "artifacts": "<abs-path>", "indexes": "<abs-path>" }
|
||||
}
|
||||
```
|
||||
|
||||
Deliberately **excluded** (their change must not invalidate a DWH generation):
|
||||
|
||||
- `session_storage` and `runtime_identity` (a content-only Git commit or an Evidence-only change
|
||||
must not force a full database re-introspection);
|
||||
- Evidence source/policy (P6 materialization and Evidence preprocessing are separate);
|
||||
- memory, search, and execution settings;
|
||||
- **all credentials** (passwords, API keys, signed URLs, and secret-file paths).
|
||||
|
||||
The **logical configuration identity** is:
|
||||
|
||||
```text
|
||||
workspace://<workspace-id>@v1:<sha256 of the canonical effective configuration>
|
||||
```
|
||||
|
||||
It is the same for the operator CLI and for application sessions, because both derive it from the
|
||||
same rendered configuration. That is the guarantee that the work prepared by `thothctl` is exactly
|
||||
what the sessions will consume.
|
||||
|
||||
## Why a rerun can be instant or take minutes
|
||||
|
||||
- Same canonical identity (e.g., only Evidence files changed) → the generation is reused → the
|
||||
DWH step is `unchanged` and fast.
|
||||
- Changed canonical identity (different database, address, user, schema, collection, model, or
|
||||
artifact/index roots) → the old generation is refused → ThothII re-introspects and writes a new
|
||||
generation → the step takes as long as the first preparation.
|
||||
|
||||
## Safe migration, regeneration, and recovery
|
||||
|
||||
- **Migration**: existing schema-v1 `OWNER.json` roots are readable; to switch them to the
|
||||
versioned identity, run a normal regeneration (explicit `--refresh`/new run). No automatic
|
||||
in-place rewrite.
|
||||
- **Regeneration**: a new run produces a new immutable generation and moves `ACTIVE`; the previous
|
||||
generations remain for rollback.
|
||||
- **Recovery**: if the active generation is corrupt or owned by another configuration, ThothII
|
||||
fails closed (never mixes artifacts) and tells the operator to regenerate; the old generations
|
||||
are still available for inspection.
|
||||
|
||||
## Memory root
|
||||
|
||||
P3 also gives each workspace an explicit **workspace-global memory root**:
|
||||
|
||||
```text
|
||||
/data/sessions/<workspace-id>/memory/
|
||||
```
|
||||
|
||||
All memory commands, locks, the canonical JSONL registry, and the Qdrant projection use this root
|
||||
when present. A guarded migration copies and verifies exactly one legacy canonical JSONL from the
|
||||
old `artifacts/memory` location under the workspace lock and rebuilds the projection; conflicting
|
||||
legacy registries fail closed. There is no in-place reinterpretation.
|
||||
|
||||
## Revision-scoped search records
|
||||
|
||||
Schema and Evidence records in the Qdrant collection include the pinned `workspace_revision`, so
|
||||
searches never mix descriptions or documents from different versions of the workspace. Memory and
|
||||
solved-question records remain workspace-wide on purpose.
|
||||
@@ -30,6 +30,15 @@ per `docs/contracts/workspace-preprocessing-cli.md` and the P2 walkthrough in
|
||||
`workspace-maintenance` Compose service; it never starts a backend/Pi/frontend listener and never
|
||||
attaches Git credentials.
|
||||
|
||||
|
||||
## Effective configuration and `.tht-dwh` (P3)
|
||||
|
||||
Prepared DWH generations are reusable and safe: `thothctl` and the application derive the same
|
||||
canonical effective configuration and logical identity, so prepared work is reused when nothing
|
||||
relevant changed and refused when the database/endpoint/identity changed. See
|
||||
`docs/contracts/tht-dwh.md` for generations, `OWNER.json`, `ACTIVE`, fingerprints, migration and
|
||||
recovery. A content-only or Evidence-only change never forces a full re-introspection.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- macOS: Docker Desktop, Git, and sufficient volume disk space. Git Credential Manager is useful
|
||||
|
||||
@@ -25,6 +25,15 @@ per `docs/contracts/workspace-preprocessing-cli.md` and the P2 walkthrough in
|
||||
`workspace-maintenance` Compose service; it never starts a backend/Pi/frontend listener and never
|
||||
attaches Git credentials.
|
||||
|
||||
|
||||
## Effective configuration and `.tht-dwh` (P3)
|
||||
|
||||
Prepared DWH generations are reusable and safe: `thothctl` and the application derive the same
|
||||
canonical effective configuration and logical identity, so prepared work is reused when nothing
|
||||
relevant changed and refused when the database/endpoint/identity changed. See
|
||||
`docs/contracts/tht-dwh.md` for generations, `OWNER.json`, `ACTIVE`, fingerprints, migration and
|
||||
recovery. A content-only or Evidence-only change never forces a full re-introspection.
|
||||
|
||||
## Service account, storage, and firewall
|
||||
|
||||
Create a dedicated host service account and an operator root such as `/srv/thothii`. The core
|
||||
|
||||
@@ -56,25 +56,32 @@ Checks:
|
||||
|
||||
Decision: **PENDING** (independent manual gate; automation never records PASS).
|
||||
|
||||
## P3 — Effective config and `.tht-dwh`
|
||||
## P3 — Effective configuration and `.tht-dwh`
|
||||
|
||||
**Status:** instructions to be finalized by P3 implementation; not yet runnable.
|
||||
**Status:** P3 implementation complete; automated integration PASS; manual acceptance PENDING.
|
||||
|
||||
Manual goal: compare operator and session effective DWH identities, inspect `OWNER.json` and
|
||||
`ACTIVE` without exposing secrets, prove safe reuse after a content-only revision, and prove
|
||||
fail-closed behavior after a DWH-affecting change.
|
||||
Manual goal: prove that the operator CLI and application sessions derive the same effective
|
||||
configuration, that a content-only revision reuses the prepared DWH generation (fast, `unchanged`),
|
||||
that a DWH-affecting change fails closed and regenerates, that the workspace memory migration is
|
||||
safe, and that search records are revision-scoped. See `docs/contracts/tht-dwh.md`.
|
||||
|
||||
Checks to fill during P3:
|
||||
Checks:
|
||||
|
||||
1. canonical fingerprint comparison;
|
||||
2. stable logical config-source identity;
|
||||
3. schema-v1 ownership compatibility/migration;
|
||||
4. DWH cache reuse across equivalent revisions;
|
||||
5. revision-scoped schema/Evidence state;
|
||||
6. mismatch rejection and recovery.
|
||||
1. run `thothctl ... workspace preprocess dwh` twice with only an Evidence/content change between
|
||||
them: the second run reports `unchanged` and does not re-introspect;
|
||||
2. change a DWH-affecting field (host/port/database/schema/user/collection) in the descriptor,
|
||||
push, pull: the next run refuses the old generation and regenerates, with a clear
|
||||
`effective_config_mismatch`-style outcome and no mixed artifacts;
|
||||
3. inspect `.tht-dwh` generations: immutable directories, `OWNER.json` with the canonical
|
||||
fingerprints, `ACTIVE` pointer; old generations still present;
|
||||
4. memory: after the guarded migration the workspace uses
|
||||
`<dataRoot>/sessions/<workspace-id>/memory/`; the JSONL registry and Qdrant projection are
|
||||
rebuilt and consistent; a conflicting legacy registry fails closed;
|
||||
5. search records: schema/Evidence points carry the pinned `workspace_revision`; memory/solved
|
||||
records remain workspace-wide;
|
||||
6. documentation: `docs/contracts/tht-dwh.md` matches the observed behavior.
|
||||
|
||||
Decision: **PENDING**.
|
||||
|
||||
## P4 — Qdrant bootstrap and guarded rebuild
|
||||
|
||||
**Status:** instructions to be finalized by P4 implementation; not yet runnable.
|
||||
|
||||
Reference in New Issue
Block a user