feat: complete catalog-driven preprocessing
Publish documentation / publish (push) Successful in 2m12s
Publish documentation / publish (push) Successful in 2m12s
This commit is contained in:
Codex
+28
-121
@@ -1,128 +1,35 @@
|
||||
# `.tht-dwh` — DWH generations, `OWNER.json`, ACTIVE, and fingerprints
|
||||
# Internal Catalog-bound DWH artifacts
|
||||
|
||||
> Operator contract. P3 makes the effective DWH/preprocessing configuration reproducible and
|
||||
> versioned across the operator CLI and the application sessions, and documents what `.tht-dwh`
|
||||
> is so operators can reason about why a rerun is instant or why it takes minutes.
|
||||
`.tht-dwh`, `OWNER.json`, and the `ACTIVE` generation pointer are private preprocessing artifacts,
|
||||
not an operator-facing generation or rollback contract. Complete preprocessing materializes the
|
||||
physical schema from the immutable PostgreSQL Catalog Metadata Snapshot and samples only eligible
|
||||
DWH values to build LSH. It never obtains schema metadata from workspace YAML or by introspecting the
|
||||
DWH in the harness.
|
||||
|
||||
## What `.tht-dwh` is
|
||||
`OWNER.json` binds the artifact root to all of:
|
||||
|
||||
`.tht-dwh` is the workspace-local directory that stores the **prepared snapshots of the data
|
||||
warehouse structure** (the catalog `physical.yaml` plus the LSH hashes used for fuzzy search).
|
||||
ThothII does not re-read the whole database for every question: it prepares it once, stores the
|
||||
result here, and reuses it. The directory lives under the workspace runtime root, for example:
|
||||
- workspace ID;
|
||||
- Catalog database ID;
|
||||
- Metadata Content Revision;
|
||||
- effective configuration fingerprint;
|
||||
- input fingerprint.
|
||||
|
||||
```text
|
||||
/data/sessions/<workspace-id>/.tht-dwh/
|
||||
The complete binding must match before artifacts can be reused. This prevents an LSH built from one
|
||||
database, workspace, Catalog revision, or effective configuration from being associated with
|
||||
another. PostgreSQL separately owns readiness through `running | succeeded | failed`, the processed
|
||||
revision, and the preprocessing input fingerprint.
|
||||
|
||||
The internal generation is overwritten by normal preprocessing and is not retained for rollback.
|
||||
Recovery is always:
|
||||
|
||||
```sh
|
||||
tht --installation <absolute>/thothii-installation.yaml \
|
||||
workspace preprocess run --workspace <workspace-id>
|
||||
```
|
||||
|
||||
## Immutable generations
|
||||
The clear operation removes `.tht-dwh` together with the LSH, private Catalog snapshot, Evidence
|
||||
corpus, and derived checkpoints. The next run recreates them from PostgreSQL and the pinned workspace
|
||||
revision.
|
||||
|
||||
Each preparation run produces a **generation**: an immutable directory containing the catalog and
|
||||
the LSH artifacts for one exact "effective configuration" (see fingerprints below). Generations
|
||||
are never modified in place; a new run writes a new generation, and an `ACTIVE` pointer selects
|
||||
which generation the workspace currently uses. Keeping the old generations makes rollback and
|
||||
diagnosis safe.
|
||||
|
||||
## `OWNER.json`
|
||||
|
||||
Every generation root contains an `OWNER.json` that records who owns it:
|
||||
|
||||
```json
|
||||
{
|
||||
"workspace_id": "<workspace-id>",
|
||||
"config_fingerprint": "sha256:<64 hex>",
|
||||
"input_fingerprint": "sha256:<64 hex>"
|
||||
}
|
||||
```
|
||||
|
||||
- `config_fingerprint` is the digest of the **canonical effective configuration** (see below).
|
||||
- `input_fingerprint` is the digest of the **logical configuration identity**.
|
||||
|
||||
Before reusing a generation, the harness compares the current canonical identity with the one in
|
||||
`OWNER.json`. If they differ, the generation is **refused** (never silently reused) and a new one
|
||||
is produced. This is what protects ThothII from using artifacts prepared for a different database,
|
||||
endpoint, user, schema, or index contract.
|
||||
|
||||
The reader is compatible with the historical schema-v1 `OWNER.json` (same three keys, `sha256:`
|
||||
values) so existing installations keep working; new writes use the versioned computation. There is
|
||||
no automatic in-place reinterpretation: operators regenerate explicitly when a root is old.
|
||||
|
||||
## The canonical effective configuration and the logical identity
|
||||
|
||||
The **canonical effective configuration** is the non-secret subset of the rendered runtime
|
||||
configuration that determines whether a prepared DWH generation is still valid:
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"dwh": {
|
||||
"engine": "postgres",
|
||||
"database": "<database>",
|
||||
"schema": "<schema>",
|
||||
"transport": "postgres_direct | rest_api | ...",
|
||||
"host": "<host>",
|
||||
"port": 5432,
|
||||
"baseUrl": "<base-url>",
|
||||
"user": "<user>"
|
||||
},
|
||||
"vector": { "collection": "<collection>", "dimensions": 1024, "distance": "cosine" },
|
||||
"embedding": { "model": "<model>", "dimensions": 1024 },
|
||||
"roots": { "artifacts": "<abs-path>", "indexes": "<abs-path>" }
|
||||
}
|
||||
```
|
||||
|
||||
Deliberately **excluded** (their change must not invalidate a DWH generation):
|
||||
|
||||
- `session_storage` and `runtime_identity` (a content-only Git commit or an Evidence-only change
|
||||
must not force a full database re-introspection);
|
||||
- Evidence source/policy (P6 materialization and Evidence preprocessing are separate);
|
||||
- memory, search, and execution settings;
|
||||
- **all credentials** (passwords, API keys, signed URLs, and secret-file paths).
|
||||
|
||||
The **logical configuration identity** is:
|
||||
|
||||
```text
|
||||
workspace://<workspace-id>@v1:<sha256 of the canonical effective configuration>
|
||||
```
|
||||
|
||||
It is the same for the operator CLI and for application sessions, because both derive it from the
|
||||
same rendered configuration. That is the guarantee that the work prepared by `tht` is exactly
|
||||
what the sessions will consume.
|
||||
|
||||
## Why a rerun can be instant or take minutes
|
||||
|
||||
- Same canonical identity (e.g., only Evidence files changed) → the generation is reused → the
|
||||
DWH step is `unchanged` and fast.
|
||||
- Changed canonical identity (different database, address, user, schema, collection, model, or
|
||||
artifact/index roots) → the old generation is refused → ThothII re-introspects and writes a new
|
||||
generation → the step takes as long as the first preparation.
|
||||
|
||||
## Safe migration, regeneration, and recovery
|
||||
|
||||
- **Migration**: existing schema-v1 `OWNER.json` roots are readable; to switch them to the
|
||||
versioned identity, run a normal regeneration (explicit `--refresh`/new run). No automatic
|
||||
in-place rewrite.
|
||||
- **Regeneration**: a new run produces a new immutable generation and moves `ACTIVE`; the previous
|
||||
generations remain for rollback.
|
||||
- **Recovery**: if the active generation is corrupt or owned by another configuration, ThothII
|
||||
fails closed (never mixes artifacts) and tells the operator to regenerate; the old generations
|
||||
are still available for inspection.
|
||||
|
||||
## Memory root
|
||||
|
||||
P3 also gives each workspace an explicit **workspace-global memory root**:
|
||||
|
||||
```text
|
||||
/data/sessions/<workspace-id>/memory/
|
||||
```
|
||||
|
||||
All memory commands, locks, the canonical JSONL registry, and the Qdrant projection use this root
|
||||
when present. A guarded migration copies and verifies exactly one legacy canonical JSONL from the
|
||||
old `artifacts/memory` location under the workspace lock and rebuilds the projection; conflicting
|
||||
legacy registries fail closed. There is no in-place reinterpretation.
|
||||
|
||||
## Revision-scoped search records
|
||||
|
||||
Schema and Evidence records in the Qdrant collection include the pinned `workspace_revision`, so
|
||||
searches never mix descriptions or documents from different versions of the workspace. Memory and
|
||||
solved-question records remain workspace-wide on purpose.
|
||||
Workspace-global `memory` and Qdrant `solved_question` data are not preprocessing output and remain
|
||||
preserved both when reference data is overwritten and when preprocessing is cleared.
|
||||
|
||||
@@ -5,6 +5,10 @@ workspace descriptor. Evidence is optional: a valid v4 descriptor without it rem
|
||||
When present, `evidence` is strict: it contains `source` and a defaulted strict `policy`; every
|
||||
source variant and the policy reject unknown keys.
|
||||
|
||||
The version numbers are intentionally separate: the Evidence descriptor supports v1/v2, while the
|
||||
latest Curated Evidence Unit format is v3. There is no Evidence descriptor v3/v4 and no Curated
|
||||
Evidence Unit v4.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
REGISTRY["Workspace registry"] --> DESCRIPTOR["Evidence descriptor"]
|
||||
@@ -228,13 +232,15 @@ authoring repository.
|
||||
P1.1 performs no acquisition, extraction, preprocessing/indexing, embeddings, Qdrant writes,
|
||||
active-snapshot retention, or GC.
|
||||
|
||||
## Operator validation
|
||||
## Runtime publication
|
||||
|
||||
After the runtime configuration is rendered or acquired, validate it with the exact per-command
|
||||
option ordering:
|
||||
After the curator has published a valid revision, the installation operator publishes it only as
|
||||
part of complete workspace preprocessing:
|
||||
|
||||
```sh
|
||||
tht config check -c <path>
|
||||
tht --installation <absolute>/thothii-installation.yaml \
|
||||
workspace preprocess run --workspace <workspace-id>
|
||||
```
|
||||
|
||||
Stop after validation. P2/P6 later owns preprocessing and materialization.
|
||||
This command also consumes the PostgreSQL Catalog snapshot and rebuilds schema/LSH output. There is
|
||||
no public Evidence-only preprocessing command.
|
||||
|
||||
@@ -1,193 +1,182 @@
|
||||
# Workspace preprocessing CLI contract
|
||||
|
||||
`tht` is the only supported host entrypoint for workspace preprocessing.
|
||||
`tht` exposes one complete, one-shot mutating preprocessing operation. It must succeed before the
|
||||
workspace can be used by the core.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
OP["Operator"] --> INVOKE["tht workspace preprocess"]
|
||||
INVOKE --> VALIDATE["Validate descriptor\nand paths"]
|
||||
VALIDATE --> SOURCE["Read source and\ncurated workspace"]
|
||||
SOURCE --> NORMALIZE["Normalize and chunk"]
|
||||
NORMALIZE --> INDEX["Update vector and\nBM25 indexes"]
|
||||
INDEX --> VERIFY["Verify collection\nand generation"]
|
||||
VERIFY --> READY["Generation ready"]
|
||||
VALIDATE -->|"invalid"| STOP["Exit with diagnostic"]
|
||||
```
|
||||
|
||||
## Invocation
|
||||
## Public commands
|
||||
|
||||
```text
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace inspect
|
||||
--workspace <id> [--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace preprocess dwh
|
||||
--workspace <id> [--resume <32hex>] [--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace schema suggest-fks
|
||||
--workspace <id>
|
||||
[--from-sql <regular-file>]... [--assume <column=table>]...
|
||||
[--output <new-file>] [--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace schema check
|
||||
--workspace <id>
|
||||
[--annotations <regular-file> --reviewed-candidates <sha256:hex>]
|
||||
[--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace schema accept
|
||||
--workspace <id> --run <32hex> --yes [--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace index-schema
|
||||
--workspace <id> [--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace preprocess evidence
|
||||
--workspace <id> [--dry-run] [--resume <32hex>] [--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace preprocess run
|
||||
--workspace <id> [--resume <32hex>] [--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace vector inspect
|
||||
--workspace <id> [--json]
|
||||
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace vector rebuild
|
||||
--workspace <id> --collection <name> --confirm <name> --destroy [--json]
|
||||
tht --installation <absolute>/thothii-installation.yaml workspace preprocess clear
|
||||
--workspace <id> [--json]
|
||||
```
|
||||
|
||||
## Qdrant collection lifecycle (P4)
|
||||
There are no public partial commands for DWH introspection, LSH, FK suggestions, schema indexing,
|
||||
Evidence indexing, or Qdrant rebuild. `preprocess run` does not accept `--resume`, `--dry-run`, a
|
||||
generation identifier, or a rollback option. Re-running it replaces the preceding derived output.
|
||||
`preprocess clear` removes all replaceable preprocessing output and preserves runtime memory.
|
||||
|
||||
- `workspace vector inspect` reports the descriptor-owned Qdrant collection contract
|
||||
(name, dimensions, distance, keyword indexes) **without mutation**.
|
||||
- `workspace vector rebuild` deletes and recreates the descriptor-owned collection
|
||||
with the exact contract (1024 dimensions, cosine distance, the 8 required keyword
|
||||
payload indexes) under guards:
|
||||
- `--collection <name>` must equal the descriptor's `semantic_index.vector_store.collection`;
|
||||
- `--confirm <name>` must equal `--collection` (exact repetition);
|
||||
- `--destroy` is required to confirm the destructive operation;
|
||||
- the operator refuses any other combination with exit code 2 (usage).
|
||||
- Self-heal at session admission: a missing collection is created and missing
|
||||
keyword indexes are added by the shared collection manager; incompatible
|
||||
dimensions/distance/index types are never mutated (`semantic_index_incompatible`).
|
||||
- The operator path (`workspace-maintenance.js vector-inspect|vector-rebuild`)
|
||||
performs the guarded rebuild; rebuild state is written before deletion and the
|
||||
collection is verified after recreation. No prefix matching or global Qdrant
|
||||
mutation is performed.
|
||||
## Sources of truth
|
||||
|
||||
## Additive BM25 for Evidence
|
||||
- The Workspace Descriptor v4 contains workspace identity and optional Evidence configuration.
|
||||
It contains no database identity, connection, table, column, description, sensitivity, or
|
||||
relationship data.
|
||||
- PostgreSQL Metadata Catalog is the sole database authority. It owns the workspace/database
|
||||
association, installation-local binding, tables, columns, descriptions, sensitivity flags,
|
||||
physical foreign keys, and active logical relationships.
|
||||
- The workspace Git revision remains authoritative for Evidence. Evidence Descriptor v1/v2 and
|
||||
Curated Evidence Unit v3 are unchanged; there is no Evidence v4.
|
||||
|
||||
- Only `workspace preprocess evidence` (and the Evidence portion of `workspace preprocess run`)
|
||||
may add the named sparse vector `bm25` with Qdrant modifier `idf`.
|
||||
- The upgrade uses Qdrant's additive named-vector operation. It preserves the existing unnamed
|
||||
dense vector and never deletes, renames, or rebuilds the shared collection.
|
||||
- Session readiness remains read-only with respect to BM25. Schema, Memory, and solved-question
|
||||
records therefore continue to use their existing dense-only points during and after an Evidence
|
||||
upgrade.
|
||||
- A missing `bm25` is added and reread before Evidence preprocessing starts. An existing definition
|
||||
other than `modifier: idf` fails as `semantic_index_incompatible` without any collection mutation.
|
||||
If a later Evidence candidate fails, the compatible additive schema remains in place; it does not
|
||||
make the dense-only records unavailable.
|
||||
No metadata is imported from legacy workspace YAML or `physical.yaml`/`annotations.yaml`.
|
||||
|
||||
## Preconditions and lock
|
||||
|
||||
The maintenance process obtains the PostgreSQL preprocessing lease for the workspace. Acquisition
|
||||
fails unless:
|
||||
|
||||
- the workspace has a Catalog database;
|
||||
- its latest schema synchronization matches the current database configuration version;
|
||||
- no catalog sync, description-generation, or sensitivity-analysis run is active;
|
||||
- no other preprocessing run is active.
|
||||
|
||||
While the state is `running`, PostgreSQL rejects database binding changes and every write to the
|
||||
catalog tables, columns, physical relationships, relationship columns, and logical relationships.
|
||||
It also rejects the start of the three conflicting background operations. The accepted operating
|
||||
model assumes no core session is running and nobody attempts core admission during preprocessing;
|
||||
there is therefore no drain protocol or session pinning.
|
||||
|
||||
## Complete pipeline
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
CLI["workspace preprocess run"] --> LOCK["Acquire PostgreSQL lease"]
|
||||
LOCK --> SNAP["Build Catalog Metadata Snapshot"]
|
||||
SNAP --> LSH["Sample DWH and rebuild LSH"]
|
||||
SNAP --> SCHEMA["Replace schema vectors"]
|
||||
SCHEMA --> REL["Index relationships as first-class records"]
|
||||
LSH --> EVIDENCE["Validate and replace Evidence vectors"]
|
||||
REL --> EVIDENCE
|
||||
EVIDENCE --> VERIFY["Verify complete result"]
|
||||
VERIFY --> READY["Commit succeeded state"]
|
||||
```
|
||||
|
||||
## Curated FK annotations (P5)
|
||||
The backend reads PostgreSQL and writes one private `Catalog Metadata Snapshot` JSON file. The
|
||||
Python child receives the snapshot path and the Catalog-derived DWH runtime binding, but never the
|
||||
Metadata Catalog credentials. It does not query the Catalog.
|
||||
|
||||
- The canonical curated annotations file is `<workspace-id>/schema/annotations.yaml`, a regular
|
||||
Git blob at the same commit as the descriptor. Absence is compatible (empty canonical set +
|
||||
warning); symlinks, trees/gitlinks, oversized (>16 MiB), non-UTF-8, and malformed objects are
|
||||
refused at activation.
|
||||
- Activation synchronizes the blob to the immutable revision-qualified root
|
||||
`/data/sessions/<id>/revisions/<commit>/artifacts/mschema/annotations.yaml` with a restrictive
|
||||
mode and an adjacent ownership manifest `{ workspace, commit, blobId, contentDigest,
|
||||
destination }`. Pinned runtimes resolve annotations from `paths.annotations_root`.
|
||||
- `workspace schema accept --run <id> --yes` is the only human FK review primitive: after
|
||||
commit/push/pull, it reads the current synced blob, validates it with the harness parser against
|
||||
the physical schema and the recorded candidate digest, and records
|
||||
`{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision, blobId }`. `--yes` is
|
||||
required; an empty file, an unknown run, a malformed blob, or a non-matching candidate fails
|
||||
closed without recording a review. `schema check` alone is not evidence of human review.
|
||||
- `preprocess run` continues only with the exact accepted blob digest and a compatible reusable
|
||||
DWH binding; otherwise it records a new review checkpoint.
|
||||
The snapshot contains the complete table and column structure, sensitivity, and effective active
|
||||
relationships. Effective descriptions use this precedence:
|
||||
|
||||
## Commit-addressed Evidence materialization (P6)
|
||||
1. curated `description`;
|
||||
2. `generatedDescription`;
|
||||
3. PostgreSQL `sourceComment`.
|
||||
|
||||
- Filesystem Evidence `<workspace-id>/evidence` is materialized from the exact pinned Git commit
|
||||
into the immutable revision content root `<registry>/snapshots/<commit>/<id>/evidence` at
|
||||
activation, with a sibling bounded manifest `<id>/evidence.manifest.json` whose digest is chained
|
||||
into `snapshot.json`.
|
||||
- Materialization uses fixed Git plumbing (`ls-tree -r -z` + `cat-file blob`) and refuses symlinks
|
||||
and gitlinks at any depth, traversal/absolute/duplicate/cross-namespace paths, and non-regular
|
||||
modes. Installation-local limits bound entry count (default 4096), total bytes (64 MiB),
|
||||
per-file bytes (8 MiB), path bytes (4096), and manifest bytes (1 MiB); a size-sum preflight runs
|
||||
before any bytes are written and no partial root is published.
|
||||
- `preprocess evidence` and `preprocess run` operate directly on the materialized root; the
|
||||
temporary `evidence_materialization_required` stop is retired (the code remains only for
|
||||
pre-P6 compatibility). For `evidence.schema_version: 2`, runtime acquisition receives exactly
|
||||
`curated/**/*.md`; `source/` and support files remain in the materialized tree for traceability.
|
||||
HTTP/S3 Evidence is unchanged.
|
||||
- The curator validates Evidence before merge. Preprocessing validates the pinned curated corpus
|
||||
again before it constructs a candidate generation, so an invalid revision is never indexed.
|
||||
- The runtime writes only its immutable materialized snapshot and derived index state. It never
|
||||
writes, stages, commits, or pushes the workspace authoring repository.
|
||||
- Materialized roots are retained with their commit-addressed snapshot directory and removed only
|
||||
when the revision becomes unreferenced.
|
||||
The DWH is queried only for derived value samples used by LSH. Sensitive columns are never sampled.
|
||||
Eligibility is computed deterministically; legacy annotation concepts, synonyms, notes, Evidence
|
||||
links, and manual eligibility overrides are not part of the snapshot.
|
||||
|
||||
## Validation
|
||||
Qdrant receives first-class `schema_table`, `schema_column`, and `schema_relationship` records. A
|
||||
relationship search hit promotes both endpoint tables. Each workspace has two physical collections:
|
||||
|
||||
- `--installation` is mandatory and absolute.
|
||||
- `--workspace` is mandatory exactly once and must match `[a-z][a-z0-9-]{2,62}`.
|
||||
- `--resume` values must be 32 lowercase hex characters.
|
||||
- `--json` may be supplied once.
|
||||
- `schema suggest-fks`
|
||||
- allows at most 32 `--from-sql` files;
|
||||
- each SQL file must be a canonical regular file, UTF-8, non-symlink, max 1 MiB;
|
||||
- total SQL ingress must not exceed 16 MiB;
|
||||
- allows at most 256 `--assume` values, each `column=table`, max 256 bytes;
|
||||
- `--output` must name a new canonical path; existing targets are refused.
|
||||
- `schema check`
|
||||
- `--annotations` and `--reviewed-candidates` are all-or-nothing;
|
||||
- annotations must be UTF-8, canonical, non-symlink, max 16 MiB;
|
||||
- `--reviewed-candidates` must match `sha256:<64 lowercase hex>`.
|
||||
- `schema accept`
|
||||
- `--run` is mandatory and must be 32 lowercase hex characters;
|
||||
- `--yes` is mandatory and may be supplied once;
|
||||
- `--annotations`/`--reviewed-candidates`/`--from-sql`/`--assume` are not accepted.
|
||||
- Unknown flags, passthrough separators, and shell fragments are rejected before Docker runs.
|
||||
- `<workspace>-reference` contains `schema_table`, `schema_column`, `schema_relationship`, and
|
||||
`evidence` records;
|
||||
- `<workspace>-memory` contains `memory` and `solved_question` records.
|
||||
|
||||
A rerun replaces the relevant records in `reference` and leaves `memory` untouched. A workspace
|
||||
without Evidence is valid and completes with an explicit warning.
|
||||
|
||||
The workspace LSH generation is bound to the workspace ID, Catalog database ID, Metadata Content
|
||||
Revision, effective configuration fingerprint, and input fingerprint. A mismatch cannot reuse a
|
||||
generation produced for another database or revision.
|
||||
|
||||
## Clear lifecycle
|
||||
|
||||
`workspace preprocess clear` is intentionally narrower than deleting all semantic data. It:
|
||||
|
||||
1. copies any `memory` and `solved_question` records still present in the pre-split workspace
|
||||
collection to `<workspace>-memory`, then retires that legacy collection (a normal preprocessing
|
||||
write performs the same one-time cutover if clear was not invoked first);
|
||||
2. deletes `<workspace>-reference`;
|
||||
3. removes the active LSH generation, Evidence corpus, private Catalog snapshot, and derived job
|
||||
checkpoints;
|
||||
4. records `derived_data_cleared` in PostgreSQL so readiness projects to `required`.
|
||||
|
||||
It never deletes `<workspace>-memory`, session artifacts, the workspace repository, Catalog metadata,
|
||||
or source database data. There is no clear history or rollback. The next complete preprocessing run
|
||||
recreates the reference collection and all local derived artifacts from their authoritative sources.
|
||||
|
||||
## PostgreSQL preprocessing state
|
||||
|
||||
PostgreSQL is authoritative for:
|
||||
|
||||
- `preprocessing_status`: `running | succeeded | failed`;
|
||||
- current `metadata_content_revision`;
|
||||
- last successful `preprocessed_metadata_revision`;
|
||||
- `preprocessing_input_fingerprint`;
|
||||
- start/finish timestamps and a bounded failure code.
|
||||
|
||||
Every relevant Catalog mutation increments `metadata_content_revision` and marks the prior result
|
||||
stale in the same transaction. The input fingerprint covers both the immutable workspace Git
|
||||
revision (including revision-pinned Evidence) and the effective Catalog-derived DWH/semantic
|
||||
configuration.
|
||||
|
||||
Core admission requires all of the following:
|
||||
|
||||
- status is `succeeded`;
|
||||
- processed and current Metadata Content Revision are equal;
|
||||
- stored and current preprocessing input fingerprints are equal.
|
||||
|
||||
Failure leaves the workspace unavailable to the core. There is no rollback: fix the cause and run
|
||||
the complete command again.
|
||||
|
||||
## Administration sidebar
|
||||
|
||||
The Administration sidebar exposes the same one-shot operation for the workspace selected in the
|
||||
composer. It reads `GET /workspaces/:workspaceId/preprocessing`, invokes
|
||||
`POST /workspaces/:workspaceId/preprocessing` for a run, and invokes
|
||||
`DELETE /workspaces/:workspaceId/preprocessing` for clear. Both mutations delegate to the complete
|
||||
workspace preprocessing service and do not expose partial stages.
|
||||
|
||||
The compact control has five states: `ready`, `required`, `running`, `blocked`, and `failed`.
|
||||
**Run again** is available for `ready`, **Run** for `required`, and **Retry** for `failed`. **Clear**
|
||||
appears to the left of that action whenever replaceable data can be cleared. It opens an inline
|
||||
confirmation that names both the data removed and the memory retained. All run actions invoke the
|
||||
same complete, idempotent preprocessing operation. A blocked state
|
||||
names the current unmet prerequisite and links it to an operator action, but has no run diagnostic
|
||||
because no preprocessing run started. A failed state shows only the latest bounded diagnostic:
|
||||
failed stage, safe error code, and finish time. PostgreSQL overwrites that diagnostic when the next
|
||||
run starts or finishes; there is no preprocessing history or rollback UI. The full service log is
|
||||
available to operators with `docker compose logs core`.
|
||||
|
||||
Once a non-ready state is known, **New session** is disabled. Core admission remains the
|
||||
authoritative enforcement boundary if the browser has not loaded the state yet.
|
||||
|
||||
## Container boundary
|
||||
|
||||
`tht` resolves the selected `core` image from the rendered installation, converts it to an immutable local image ID, writes a one-shot final override that pins both `core` and `workspace-maintenance` to that ID with `pull_policy: never`, and runs only:
|
||||
The host CLI resolves the selected core image to an immutable local image ID and runs only:
|
||||
|
||||
```text
|
||||
docker compose run --rm --no-deps --no-TTY --name <owned-name> workspace-maintenance <fixed-command>
|
||||
docker compose run --rm --no-deps --no-TTY --name <owned-name> \
|
||||
workspace-maintenance preprocess-run
|
||||
|
||||
docker compose run --rm --no-deps --no-TTY --name <owned-name> \
|
||||
workspace-maintenance preprocess-clear
|
||||
```
|
||||
|
||||
The request is streamed as one schema-versioned JSON document over stdin. Public stdout is always one schema-versioned JSON result; human mode is rendered from an allowlisted subset of that same result.
|
||||
The request is one bounded schema-versioned JSON document on stdin. The maintenance process strips
|
||||
all `THT_CATALOG_*` variables before spawning Python. JSON mode keeps stdout pristine.
|
||||
|
||||
## Public JSON result
|
||||
## Result and exit codes
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"status": "succeeded|unchanged|dry_run|blocked|failed",
|
||||
"code": "ok|workspace_not_found|workspace_not_activatable|binding_missing|preprocessing_conflict|preprocessing_resume_mismatch|manual_review_required|evidence_materialization_required|effective_config_mismatch|semantic_index_incompatible|annotation_invalid|egress_policy_refused",
|
||||
"workspaceId": "abc",
|
||||
"workspaceRevision": "1234567890abcdef1234567890abcdef12345678",
|
||||
"descriptorBlob": "sha256:<64 lowercase hex>",
|
||||
"operation": "inspect|preprocess-dwh|schema-suggest-fks|schema-check|index-schema|preprocess-evidence|preprocess-run",
|
||||
"runId": "<optional 32hex>",
|
||||
"childRuns": {"stage": "<optional 32hex>"},
|
||||
"completedStages": ["stage"],
|
||||
"counts": {"name": 1},
|
||||
"artifactIdentities": [{"kind": "fk_candidates", "digest": "sha256:<64 lowercase hex>"}],
|
||||
"warnings": ["safe warning"]
|
||||
}
|
||||
```
|
||||
The public result includes `schemaVersion`, `status`, `code`, workspace/revision identity,
|
||||
`operation`, completed stages, safe counts, artifact digests, and the input fingerprint. It never
|
||||
contains credentials, connection strings, sampled values, SQL, or raw child errors.
|
||||
|
||||
`tht --json` parses the operator stdout strictly and re-encodes only the public fields above.
|
||||
|
||||
`evidence_materialization_required` is retained for pre-P6 compatibility; since P6, filesystem
|
||||
Evidence is materialized at activation and preprocesses directly.
|
||||
|
||||
## Exit codes
|
||||
|
||||
- `0`: `succeeded`, `unchanged`, or `dry_run`
|
||||
- `3`: `blocked`
|
||||
- `2`: host-side grammar or local file safety failure
|
||||
- `1`: operational failure or operator-reported `failed`
|
||||
- `0`: preprocessing run or clear succeeded;
|
||||
- `1`: operational or preprocessing failure;
|
||||
- `2`: invalid host-side invocation.
|
||||
|
||||
Reference in New Issue
Block a user