# Workspace preprocessing CLI contract `tht` exposes one complete, one-shot mutating preprocessing operation. It must succeed before the workspace can be used by the core. ## Public commands ```text tht --installation /thothii-installation.yaml workspace inspect --workspace [--json] tht --installation /thothii-installation.yaml workspace preprocess run --workspace [--json] tht --installation /thothii-installation.yaml workspace preprocess clear --workspace [--json] tht --installation /thothii-installation.yaml workspace evidence consolidate --workspace [--json] tht --installation /thothii-installation.yaml workspace evidence refresh --workspace [--json] tht --installation /thothii-installation.yaml workspace evidence decide --workspace --source-id <64-hex> --revision <64-hex> --decision keep|replace [--json] ``` There are no public partial commands for DWH introspection, LSH, FK suggestions, schema indexing, or Qdrant rebuild. The separate Evidence curation command validates editable local files and activates only their index; it does not satisfy workspace preprocessing readiness or execute Git. See [Curated Evidence v4](curated-evidence-v4.md#administration-and-installed-command). Source refresh acquires and refines only on explicit request, saving comparisons without changing active Evidence. Source decisions activate through the same Evidence-only pipeline and preserve Catalog readiness. Their envelopes reject arbitrary URLs, paths and extra flags; source locations and credentials come from installation configuration. `preprocess run` does not accept `--resume`, `--dry-run`, a generation identifier, or a rollback option. Re-running it replaces the preceding derived output. `preprocess clear` removes all replaceable preprocessing output and preserves runtime memory and the canonical local Evidence archive. Full preprocessing remains required afterward. ## Sources of truth - The Workspace Descriptor v4 contains workspace identity and optional Evidence configuration. It contains no database identity, connection, table, column, description, sensitivity, or relationship data. - PostgreSQL Metadata Catalog is the sole database authority. It owns the workspace/database association, installation-local binding, tables, columns, descriptions, sensitivity flags, physical foreign keys, and active logical relationships. - Evidence Descriptor v1/v2 still configures the initial source. Initialized archives use editable Curated Evidence Unit v4 and the last successfully activated local snapshot. Ordinary preprocessing never imports unconsolidated working-tree edits. Before initialization, the pinned workspace Git source remains supported. No metadata is imported from legacy workspace YAML or `physical.yaml`/`annotations.yaml`. ## Preconditions and lock The maintenance process obtains the PostgreSQL preprocessing lease for the workspace. Acquisition fails unless: - the workspace has a Catalog database; - its latest schema synchronization matches the current database configuration version; - no catalog sync, description-generation, or sensitivity-analysis run is active; - no other preprocessing run is active. While the state is `running`, PostgreSQL rejects database binding changes and every write to the catalog tables, columns, physical relationships, relationship columns, and logical relationships. It also rejects the start of the three conflicting background operations. The accepted operating model assumes no core session is running and nobody attempts core admission during preprocessing; there is therefore no drain protocol or session pinning. ## Complete pipeline ```mermaid flowchart LR CLI["workspace preprocess run"] --> LOCK["Acquire PostgreSQL lease"] LOCK --> SNAP["Build Catalog Metadata Snapshot"] SNAP --> LSH["Sample DWH and rebuild LSH"] SNAP --> SCHEMA["Replace schema vectors"] SCHEMA --> REL["Index relationships as first-class records"] LSH --> EVIDENCE["Validate and replace Evidence vectors"] REL --> EVIDENCE EVIDENCE --> VERIFY["Verify complete result"] VERIFY --> READY["Commit succeeded state"] ``` The backend reads PostgreSQL and writes one private `Catalog Metadata Snapshot` JSON file. The Python child receives the snapshot path and the Catalog-derived DWH runtime binding, but never the Metadata Catalog credentials. It does not query the Catalog. The snapshot contains the complete table and column structure, sensitivity, and effective active relationships. Effective descriptions use this precedence: 1. curated `description`; 2. `generatedDescription`; 3. PostgreSQL `sourceComment`. The DWH is queried only for derived value samples used by LSH. Sensitive columns are never sampled. Eligibility is computed deterministically; legacy annotation concepts, synonyms, notes, Evidence links, and manual eligibility overrides are not part of the snapshot. Qdrant receives first-class `schema_table`, `schema_column`, and `schema_relationship` records. A relationship search hit promotes both endpoint tables. Each workspace has two physical collections: - `-reference` contains `schema_table`, `schema_column`, `schema_relationship`, and `evidence` records; - `-memory` contains `memory` and `solved_question` records. A rerun replaces the relevant records in `reference` and leaves `memory` untouched. A workspace without Evidence is valid and completes with an explicit warning. The workspace LSH generation is bound to the workspace ID, Catalog database ID, Metadata Content Revision, effective configuration fingerprint, and input fingerprint. A mismatch cannot reuse a generation produced for another database or revision. ## Clear lifecycle `workspace preprocess clear` is intentionally narrower than deleting all semantic data. It: 1. leaves `-memory` unchanged; Memory projections are reconstructed only from the authoritative PostgreSQL archive, never imported from legacy vector payloads; 2. deletes `-reference`; 3. removes the active LSH generation, Evidence corpus, private Catalog snapshot, and derived job checkpoints; 4. records `derived_data_cleared` in PostgreSQL so readiness projects to `required`. It never deletes `-memory`, session artifacts, the workspace repository, Catalog metadata, or source database data. There is no clear history or rollback. The next complete preprocessing run recreates the reference collection and all local derived artifacts from their authoritative sources. ## PostgreSQL preprocessing state PostgreSQL is authoritative for: - `preprocessing_status`: `running | succeeded | failed`; - current `metadata_content_revision`; - last successful `preprocessed_metadata_revision`; - `preprocessing_input_fingerprint`; - start/finish timestamps and a bounded failure code. Every relevant Catalog mutation increments `metadata_content_revision` and marks the prior result stale in the same transaction. The input fingerprint covers both the immutable workspace Git revision (including revision-pinned Evidence) and the effective Catalog-derived DWH/semantic configuration. Core admission requires all of the following: - status is `succeeded`; - processed and current Metadata Content Revision are equal; - stored and current preprocessing input fingerprints are equal. Failure leaves the workspace unavailable to the core. There is no rollback: fix the cause and run the complete command again. ## Administration sidebar The Administration sidebar exposes the same one-shot operation for the workspace selected in the composer. It reads `GET /workspaces/:workspaceId/preprocessing`, invokes `POST /workspaces/:workspaceId/preprocessing` for a run, and invokes `DELETE /workspaces/:workspaceId/preprocessing` for clear. Both mutations delegate to the complete workspace preprocessing service and do not expose partial stages. The compact control has five states: `ready`, `required`, `running`, `blocked`, and `failed`. **Run again** is available for `ready`, **Run** for `required`, and **Retry** for `failed`. **Clear** appears to the left of that action whenever replaceable data can be cleared. It opens an inline confirmation that names both the data removed and the memory retained. All run actions invoke the same complete, idempotent preprocessing operation. A blocked state names the current unmet prerequisite and links it to an operator action, but has no run diagnostic because no preprocessing run started. A failed state shows only the latest bounded diagnostic: failed stage, safe error code, and finish time. PostgreSQL overwrites that diagnostic when the next run starts or finishes; there is no preprocessing history or rollback UI. The full service log is available to operators with `docker compose logs core`. Once a non-ready state is known, **New session** is disabled. Core admission remains the authoritative enforcement boundary if the browser has not loaded the state yet. ## Container boundary The host CLI resolves the selected core image to an immutable local image ID and runs only: ```text docker compose run --rm --no-deps --no-TTY --name \ workspace-maintenance preprocess-run docker compose run --rm --no-deps --no-TTY --name \ workspace-maintenance preprocess-clear ``` The request is one bounded schema-versioned JSON document on stdin. The maintenance process strips all `THT_CATALOG_*` variables before spawning Python. JSON mode keeps stdout pristine. ## Result and exit codes The public result includes `schemaVersion`, `status`, `code`, workspace/revision identity, `operation`, completed stages, safe counts, artifact digests, and the input fingerprint. It never contains credentials, connection strings, sampled values, SQL, or raw child errors. - `0`: preprocessing run or clear succeeded; - `1`: operational or preprocessing failure; - `2`: invalid host-side invocation.