183 lines
8.7 KiB
Markdown
183 lines
8.7 KiB
Markdown
# Workspace preprocessing CLI contract
|
|
|
|
`tht` exposes one complete, one-shot mutating preprocessing operation. It must succeed before the
|
|
workspace can be used by the core.
|
|
|
|
## Public commands
|
|
|
|
```text
|
|
tht --installation <absolute>/thothii-installation.yaml workspace inspect
|
|
--workspace <id> [--json]
|
|
|
|
tht --installation <absolute>/thothii-installation.yaml workspace preprocess run
|
|
--workspace <id> [--json]
|
|
|
|
tht --installation <absolute>/thothii-installation.yaml workspace preprocess clear
|
|
--workspace <id> [--json]
|
|
```
|
|
|
|
There are no public partial commands for DWH introspection, LSH, FK suggestions, schema indexing,
|
|
Evidence indexing, or Qdrant rebuild. `preprocess run` does not accept `--resume`, `--dry-run`, a
|
|
generation identifier, or a rollback option. Re-running it replaces the preceding derived output.
|
|
`preprocess clear` removes all replaceable preprocessing output and preserves runtime memory.
|
|
|
|
## Sources of truth
|
|
|
|
- The Workspace Descriptor v4 contains workspace identity and optional Evidence configuration.
|
|
It contains no database identity, connection, table, column, description, sensitivity, or
|
|
relationship data.
|
|
- PostgreSQL Metadata Catalog is the sole database authority. It owns the workspace/database
|
|
association, installation-local binding, tables, columns, descriptions, sensitivity flags,
|
|
physical foreign keys, and active logical relationships.
|
|
- The workspace Git revision remains authoritative for Evidence. Evidence Descriptor v1/v2 and
|
|
Curated Evidence Unit v3 are unchanged; there is no Evidence v4.
|
|
|
|
No metadata is imported from legacy workspace YAML or `physical.yaml`/`annotations.yaml`.
|
|
|
|
## Preconditions and lock
|
|
|
|
The maintenance process obtains the PostgreSQL preprocessing lease for the workspace. Acquisition
|
|
fails unless:
|
|
|
|
- the workspace has a Catalog database;
|
|
- its latest schema synchronization matches the current database configuration version;
|
|
- no catalog sync, description-generation, or sensitivity-analysis run is active;
|
|
- no other preprocessing run is active.
|
|
|
|
While the state is `running`, PostgreSQL rejects database binding changes and every write to the
|
|
catalog tables, columns, physical relationships, relationship columns, and logical relationships.
|
|
It also rejects the start of the three conflicting background operations. The accepted operating
|
|
model assumes no core session is running and nobody attempts core admission during preprocessing;
|
|
there is therefore no drain protocol or session pinning.
|
|
|
|
## Complete pipeline
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
CLI["workspace preprocess run"] --> LOCK["Acquire PostgreSQL lease"]
|
|
LOCK --> SNAP["Build Catalog Metadata Snapshot"]
|
|
SNAP --> LSH["Sample DWH and rebuild LSH"]
|
|
SNAP --> SCHEMA["Replace schema vectors"]
|
|
SCHEMA --> REL["Index relationships as first-class records"]
|
|
LSH --> EVIDENCE["Validate and replace Evidence vectors"]
|
|
REL --> EVIDENCE
|
|
EVIDENCE --> VERIFY["Verify complete result"]
|
|
VERIFY --> READY["Commit succeeded state"]
|
|
```
|
|
|
|
The backend reads PostgreSQL and writes one private `Catalog Metadata Snapshot` JSON file. The
|
|
Python child receives the snapshot path and the Catalog-derived DWH runtime binding, but never the
|
|
Metadata Catalog credentials. It does not query the Catalog.
|
|
|
|
The snapshot contains the complete table and column structure, sensitivity, and effective active
|
|
relationships. Effective descriptions use this precedence:
|
|
|
|
1. curated `description`;
|
|
2. `generatedDescription`;
|
|
3. PostgreSQL `sourceComment`.
|
|
|
|
The DWH is queried only for derived value samples used by LSH. Sensitive columns are never sampled.
|
|
Eligibility is computed deterministically; legacy annotation concepts, synonyms, notes, Evidence
|
|
links, and manual eligibility overrides are not part of the snapshot.
|
|
|
|
Qdrant receives first-class `schema_table`, `schema_column`, and `schema_relationship` records. A
|
|
relationship search hit promotes both endpoint tables. Each workspace has two physical collections:
|
|
|
|
- `<workspace>-reference` contains `schema_table`, `schema_column`, `schema_relationship`, and
|
|
`evidence` records;
|
|
- `<workspace>-memory` contains `memory` and `solved_question` records.
|
|
|
|
A rerun replaces the relevant records in `reference` and leaves `memory` untouched. A workspace
|
|
without Evidence is valid and completes with an explicit warning.
|
|
|
|
The workspace LSH generation is bound to the workspace ID, Catalog database ID, Metadata Content
|
|
Revision, effective configuration fingerprint, and input fingerprint. A mismatch cannot reuse a
|
|
generation produced for another database or revision.
|
|
|
|
## Clear lifecycle
|
|
|
|
`workspace preprocess clear` is intentionally narrower than deleting all semantic data. It:
|
|
|
|
1. copies any `memory` and `solved_question` records still present in the pre-split workspace
|
|
collection to `<workspace>-memory`, then retires that legacy collection (a normal preprocessing
|
|
write performs the same one-time cutover if clear was not invoked first);
|
|
2. deletes `<workspace>-reference`;
|
|
3. removes the active LSH generation, Evidence corpus, private Catalog snapshot, and derived job
|
|
checkpoints;
|
|
4. records `derived_data_cleared` in PostgreSQL so readiness projects to `required`.
|
|
|
|
It never deletes `<workspace>-memory`, session artifacts, the workspace repository, Catalog metadata,
|
|
or source database data. There is no clear history or rollback. The next complete preprocessing run
|
|
recreates the reference collection and all local derived artifacts from their authoritative sources.
|
|
|
|
## PostgreSQL preprocessing state
|
|
|
|
PostgreSQL is authoritative for:
|
|
|
|
- `preprocessing_status`: `running | succeeded | failed`;
|
|
- current `metadata_content_revision`;
|
|
- last successful `preprocessed_metadata_revision`;
|
|
- `preprocessing_input_fingerprint`;
|
|
- start/finish timestamps and a bounded failure code.
|
|
|
|
Every relevant Catalog mutation increments `metadata_content_revision` and marks the prior result
|
|
stale in the same transaction. The input fingerprint covers both the immutable workspace Git
|
|
revision (including revision-pinned Evidence) and the effective Catalog-derived DWH/semantic
|
|
configuration.
|
|
|
|
Core admission requires all of the following:
|
|
|
|
- status is `succeeded`;
|
|
- processed and current Metadata Content Revision are equal;
|
|
- stored and current preprocessing input fingerprints are equal.
|
|
|
|
Failure leaves the workspace unavailable to the core. There is no rollback: fix the cause and run
|
|
the complete command again.
|
|
|
|
## Administration sidebar
|
|
|
|
The Administration sidebar exposes the same one-shot operation for the workspace selected in the
|
|
composer. It reads `GET /workspaces/:workspaceId/preprocessing`, invokes
|
|
`POST /workspaces/:workspaceId/preprocessing` for a run, and invokes
|
|
`DELETE /workspaces/:workspaceId/preprocessing` for clear. Both mutations delegate to the complete
|
|
workspace preprocessing service and do not expose partial stages.
|
|
|
|
The compact control has five states: `ready`, `required`, `running`, `blocked`, and `failed`.
|
|
**Run again** is available for `ready`, **Run** for `required`, and **Retry** for `failed`. **Clear**
|
|
appears to the left of that action whenever replaceable data can be cleared. It opens an inline
|
|
confirmation that names both the data removed and the memory retained. All run actions invoke the
|
|
same complete, idempotent preprocessing operation. A blocked state
|
|
names the current unmet prerequisite and links it to an operator action, but has no run diagnostic
|
|
because no preprocessing run started. A failed state shows only the latest bounded diagnostic:
|
|
failed stage, safe error code, and finish time. PostgreSQL overwrites that diagnostic when the next
|
|
run starts or finishes; there is no preprocessing history or rollback UI. The full service log is
|
|
available to operators with `docker compose logs core`.
|
|
|
|
Once a non-ready state is known, **New session** is disabled. Core admission remains the
|
|
authoritative enforcement boundary if the browser has not loaded the state yet.
|
|
|
|
## Container boundary
|
|
|
|
The host CLI resolves the selected core image to an immutable local image ID and runs only:
|
|
|
|
```text
|
|
docker compose run --rm --no-deps --no-TTY --name <owned-name> \
|
|
workspace-maintenance preprocess-run
|
|
|
|
docker compose run --rm --no-deps --no-TTY --name <owned-name> \
|
|
workspace-maintenance preprocess-clear
|
|
```
|
|
|
|
The request is one bounded schema-versioned JSON document on stdin. The maintenance process strips
|
|
all `THT_CATALOG_*` variables before spawning Python. JSON mode keeps stdout pristine.
|
|
|
|
## Result and exit codes
|
|
|
|
The public result includes `schemaVersion`, `status`, `code`, workspace/revision identity,
|
|
`operation`, completed stages, safe counts, artifact digests, and the input fingerprint. It never
|
|
contains credentials, connection strings, sampled values, SQL, or raw child errors.
|
|
|
|
- `0`: preprocessing run or clear succeeded;
|
|
- `1`: operational or preprocessing failure;
|
|
- `2`: invalid host-side invocation.
|