ThothII/docs/contracts/workspace-preprocessing-cli.md

Workspace preprocessing CLI contract

tht exposes one complete, one-shot mutating preprocessing operation. It must succeed before the workspace can be used by the core.

Public commands

tht --installation <absolute>/thothii-installation.yaml workspace inspect
  --workspace <id> [--json]

tht --installation <absolute>/thothii-installation.yaml workspace preprocess run
  --workspace <id> [--json]

tht --installation <absolute>/thothii-installation.yaml workspace preprocess clear
  --workspace <id> [--json]

tht --installation <absolute>/thothii-installation.yaml workspace evidence consolidate
  --workspace <id> [--json]

tht --installation <absolute>/thothii-installation.yaml workspace evidence refresh
  --workspace <id> [--json]

tht --installation <absolute>/thothii-installation.yaml workspace evidence decide
  --workspace <id> --source-id <64-hex> --revision <64-hex>
  --decision keep|replace [--json]

There are no public partial commands for DWH introspection, LSH, FK suggestions, schema indexing, or Qdrant rebuild. The separate Evidence curation command validates editable local files and activates only their index; it does not satisfy workspace preprocessing readiness or execute Git. See Curated Evidence v4. Source refresh acquires and refines only on explicit request, saving comparisons without changing active Evidence. Source decisions activate through the same Evidence-only pipeline and preserve Catalog readiness. Their envelopes reject arbitrary URLs, paths and extra flags; source locations and credentials come from installation configuration. preprocess run does not accept --resume, --dry-run, a generation identifier, or a rollback option. Re-running it replaces the preceding derived output. preprocess clear removes all replaceable preprocessing output and preserves runtime memory and the canonical local Evidence archive. Full preprocessing remains required afterward.

Sources of truth

• The Workspace Descriptor v4 contains workspace identity and optional Evidence configuration. It contains no database identity, connection, table, column, description, sensitivity, or relationship data.
• PostgreSQL Metadata Catalog is the sole database authority. It owns the workspace/database association, installation-local binding, tables, columns, descriptions, sensitivity flags, physical foreign keys, and active logical relationships.
• Evidence Descriptor v1/v2 still configures the initial source. Initialized archives use editable Curated Evidence Unit v4 and the last successfully activated local snapshot. Ordinary preprocessing never imports unconsolidated working-tree edits. Before initialization, the pinned workspace Git source remains supported.

No metadata is imported from legacy workspace YAML or physical.yaml/annotations.yaml.

Preconditions and lock

The maintenance process obtains the PostgreSQL preprocessing lease for the workspace. Acquisition fails unless:

• the workspace has a Catalog database;
• its latest schema synchronization matches the current database configuration version;
• no catalog sync, description-generation, or sensitivity-analysis run is active;
• no other preprocessing run is active.

While the state is running, PostgreSQL rejects database binding changes and every write to the catalog tables, columns, physical relationships, relationship columns, and logical relationships. It also rejects the start of the three conflicting background operations. The accepted operating model assumes no core session is running and nobody attempts core admission during preprocessing; there is therefore no drain protocol or session pinning.

Complete pipeline

flowchart LR
    CLI["workspace preprocess run"] --> LOCK["Acquire PostgreSQL lease"]
    LOCK --> SNAP["Build Catalog Metadata Snapshot"]
    SNAP --> LSH["Sample DWH and rebuild LSH"]
    SNAP --> SCHEMA["Replace schema vectors"]
    SCHEMA --> REL["Index relationships as first-class records"]
    LSH --> EVIDENCE["Validate and replace Evidence vectors"]
    REL --> EVIDENCE
    EVIDENCE --> VERIFY["Verify complete result"]
    VERIFY --> READY["Commit succeeded state"]

The backend reads PostgreSQL and writes one private Catalog Metadata Snapshot JSON file. The Python child receives the snapshot path and the Catalog-derived DWH runtime binding, but never the Metadata Catalog credentials. It does not query the Catalog.

The snapshot contains the complete table and column structure, sensitivity, and effective active relationships. Effective descriptions use this precedence:

1. curated description;
2. generatedDescription;
3. PostgreSQL sourceComment.

The DWH is queried only for derived value samples used by LSH. Sensitive columns are never sampled. Eligibility is computed deterministically; legacy annotation concepts, synonyms, notes, Evidence links, and manual eligibility overrides are not part of the snapshot.

Qdrant receives first-class schema_table, schema_column, and schema_relationship records. A relationship search hit promotes both endpoint tables. Each workspace has two physical collections:

• <workspace>-reference contains schema_table, schema_column, schema_relationship, and evidence records;
• <workspace>-memory contains memory and solved_question records.

A rerun replaces the relevant records in reference and leaves memory untouched. A workspace without Evidence is valid and completes with an explicit warning.

The workspace LSH generation is bound to the workspace ID, Catalog database ID, Metadata Content Revision, effective configuration fingerprint, and input fingerprint. A mismatch cannot reuse a generation produced for another database or revision.

Clear lifecycle

workspace preprocess clear is intentionally narrower than deleting all semantic data. It:

1. leaves <workspace>-memory unchanged; Memory projections are reconstructed only from the authoritative PostgreSQL archive, never imported from legacy vector payloads;
2. deletes <workspace>-reference;
3. removes the active LSH generation, Evidence corpus, private Catalog snapshot, and derived job checkpoints;
4. records derived_data_cleared in PostgreSQL so readiness projects to required.

It never deletes <workspace>-memory, session artifacts, the workspace repository, Catalog metadata, or source database data. There is no clear history or rollback. The next complete preprocessing run recreates the reference collection and all local derived artifacts from their authoritative sources.

PostgreSQL preprocessing state

PostgreSQL is authoritative for:

• preprocessing_status: running | succeeded | failed;
• current metadata_content_revision;
• last successful preprocessed_metadata_revision;
• preprocessing_input_fingerprint;
• start/finish timestamps and a bounded failure code.

Every relevant Catalog mutation increments metadata_content_revision and marks the prior result stale in the same transaction. The input fingerprint covers both the immutable workspace Git revision (including revision-pinned Evidence) and the effective Catalog-derived DWH/semantic configuration.

Core admission requires all of the following:

• status is succeeded;
• processed and current Metadata Content Revision are equal;
• stored and current preprocessing input fingerprints are equal.

Failure leaves the workspace unavailable to the core. There is no rollback: fix the cause and run the complete command again.

Administration sidebar

The Administration sidebar exposes the same one-shot operation for the workspace selected in the composer. It reads GET /workspaces/:workspaceId/preprocessing, invokes POST /workspaces/:workspaceId/preprocessing for a run, and invokes DELETE /workspaces/:workspaceId/preprocessing for clear. Both mutations delegate to the complete workspace preprocessing service and do not expose partial stages.

The compact control has five states: ready, required, running, blocked, and failed. Run again is available for ready, Run for required, and Retry for failed. Clear appears to the left of that action whenever replaceable data can be cleared. It opens an inline confirmation that names both the data removed and the memory retained. All run actions invoke the same complete, idempotent preprocessing operation. A blocked state names the current unmet prerequisite and links it to an operator action, but has no run diagnostic because no preprocessing run started. A failed state shows only the latest bounded diagnostic: failed stage, safe error code, and finish time. PostgreSQL overwrites that diagnostic when the next run starts or finishes; there is no preprocessing history or rollback UI. The full service log is available to operators with docker compose logs core.

Once a non-ready state is known, New session is disabled. Core admission remains the authoritative enforcement boundary if the browser has not loaded the state yet.

Container boundary

The host CLI resolves the selected core image to an immutable local image ID and runs only:

docker compose run --rm --no-deps --no-TTY --name <owned-name> \
  workspace-maintenance preprocess-run

docker compose run --rm --no-deps --no-TTY --name <owned-name> \
  workspace-maintenance preprocess-clear

The request is one bounded schema-versioned JSON document on stdin. The maintenance process strips all THT_CATALOG_* variables before spawning Python. JSON mode keeps stdout pristine.

Result and exit codes

The public result includes schemaVersion, status, code, workspace/revision identity, operation, completed stages, safe counts, artifact digests, and the input fingerprint. It never contains credentials, connection strings, sampled values, SQL, or raw child errors.

• 0: preprocessing run or clear succeeded;
• 1: operational or preprocessing failure;
• 2: invalid host-side invocation.