ThothII/docs/contracts/workspace-preprocessing-cli.md

# Workspace preprocessing CLI contract

`tht` exposes one complete, one-shot mutating preprocessing operation. It must succeed before the
workspace can be used by the core.

## Public commands

```text
tht --installation <absolute>/thothii-installation.yaml workspace inspect
  --workspace <id> [--json]

tht --installation <absolute>/thothii-installation.yaml workspace preprocess run
  --workspace <id> [--json]

tht --installation <absolute>/thothii-installation.yaml workspace preprocess clear
  --workspace <id> [--json]

tht --installation <absolute>/thothii-installation.yaml workspace evidence consolidate
  --workspace <id> [--json]

tht --installation <absolute>/thothii-installation.yaml workspace evidence refresh
  --workspace <id> [--json]

tht --installation <absolute>/thothii-installation.yaml workspace evidence decide
  --workspace <id> --source-id <64-hex> --revision <64-hex>
  --decision keep|replace [--json]
```

There are no public partial commands for DWH introspection, LSH, FK suggestions, schema indexing,
or Qdrant rebuild. The separate Evidence curation command validates editable local files and
activates only their index; it does not satisfy workspace preprocessing readiness or execute Git.
See [Curated Evidence v4](curated-evidence-v4.md#administration-and-installed-command).
Source refresh acquires and refines only on explicit request, saving comparisons without
changing active Evidence. Source decisions activate through the same Evidence-only
pipeline and preserve Catalog readiness. Their envelopes reject arbitrary URLs, paths
and extra flags; source locations and credentials come from installation configuration.
`preprocess run` does not accept `--resume`, `--dry-run`, a
generation identifier, or a rollback option. Re-running it replaces the preceding derived output.
`preprocess clear` removes all replaceable preprocessing output and preserves runtime memory
and the canonical local Evidence archive. Full preprocessing remains required afterward.

## Sources of truth

- The Workspace Descriptor v4 contains workspace identity and optional Evidence configuration.
  It contains no database identity, connection, table, column, description, sensitivity, or
  relationship data.
- PostgreSQL Metadata Catalog is the sole database authority. It owns the workspace/database
  association, installation-local binding, tables, columns, descriptions, sensitivity flags,
  physical foreign keys, and active logical relationships.
- Evidence Descriptor v1/v2 still configures the initial source. Initialized archives use
  editable Curated Evidence Unit v4 and the last successfully activated local snapshot.
  Ordinary preprocessing never imports unconsolidated working-tree edits. Before initialization,
  the pinned workspace Git source remains supported.

No metadata is imported from legacy workspace YAML or `physical.yaml`/`annotations.yaml`.

## Preconditions and lock

The maintenance process obtains the PostgreSQL preprocessing lease for the workspace. Acquisition
fails unless:

- the workspace has a Catalog database;
- its latest schema synchronization matches the current database configuration version;
- no catalog sync, description-generation, or sensitivity-analysis run is active;
- no other preprocessing run is active.

While the state is `running`, PostgreSQL rejects database binding changes and every write to the
catalog tables, columns, physical relationships, relationship columns, and logical relationships.
It also rejects the start of the three conflicting background operations. The accepted operating
model assumes no core session is running and nobody attempts core admission during preprocessing;
there is therefore no drain protocol or session pinning.

## Complete pipeline

```mermaid
flowchart LR
    CLI["workspace preprocess run"] --> LOCK["Acquire PostgreSQL lease"]
    LOCK --> SNAP["Build Catalog Metadata Snapshot"]
    SNAP --> LSH["Sample DWH and rebuild LSH"]
    SNAP --> SCHEMA["Replace schema vectors"]
    SCHEMA --> REL["Index relationships as first-class records"]
    LSH --> EVIDENCE["Validate and replace Evidence vectors"]
    REL --> EVIDENCE
    EVIDENCE --> VERIFY["Verify complete result"]
    VERIFY --> READY["Commit succeeded state"]
```

The backend reads PostgreSQL and writes one private `Catalog Metadata Snapshot` JSON file. The
Python child receives the snapshot path and the Catalog-derived DWH runtime binding, but never the
Metadata Catalog credentials. It does not query the Catalog.

The snapshot contains the complete table and column structure, sensitivity, and effective active
relationships. Effective descriptions use this precedence:

1. curated `description`;
2. `generatedDescription`;
3. PostgreSQL `sourceComment`.

The DWH is queried only for derived value samples used by LSH. Sensitive columns are never sampled.
Eligibility is computed deterministically; legacy annotation concepts, synonyms, notes, Evidence
links, and manual eligibility overrides are not part of the snapshot.

Qdrant receives first-class `schema_table`, `schema_column`, and `schema_relationship` records. A
relationship search hit promotes both endpoint tables. Each workspace has two physical collections:

- `<workspace>-reference` contains `schema_table`, `schema_column`, `schema_relationship`, and
  `evidence` records;
- `<workspace>-memory` contains `memory` and `solved_question` records.

A rerun replaces the relevant records in `reference` and leaves `memory` untouched. A workspace
without Evidence is valid and completes with an explicit warning.

The workspace LSH generation is bound to the workspace ID, Catalog database ID, Metadata Content
Revision, effective configuration fingerprint, and input fingerprint. A mismatch cannot reuse a
generation produced for another database or revision.

## Clear lifecycle

`workspace preprocess clear` is intentionally narrower than deleting all semantic data. It:

1. leaves `<workspace>-memory` unchanged; Memory projections are reconstructed only from
   the authoritative PostgreSQL archive, never imported from legacy vector payloads;
2. deletes `<workspace>-reference`;
3. removes the active LSH generation, Evidence corpus, private Catalog snapshot, and derived job
   checkpoints;
4. records `derived_data_cleared` in PostgreSQL so readiness projects to `required`.

It never deletes `<workspace>-memory`, session artifacts, the workspace repository, Catalog metadata,
or source database data. There is no clear history or rollback. The next complete preprocessing run
recreates the reference collection and all local derived artifacts from their authoritative sources.

## PostgreSQL preprocessing state

PostgreSQL is authoritative for:

- `preprocessing_status`: `running | succeeded | failed`;
- current `metadata_content_revision`;
- last successful `preprocessed_metadata_revision`;
- `preprocessing_input_fingerprint`;
- start/finish timestamps and a bounded failure code.

Every relevant Catalog mutation increments `metadata_content_revision` and marks the prior result
stale in the same transaction. The input fingerprint covers both the immutable workspace Git
revision (including revision-pinned Evidence) and the effective Catalog-derived DWH/semantic
configuration.

Core admission requires all of the following:

- status is `succeeded`;
- processed and current Metadata Content Revision are equal;
- stored and current preprocessing input fingerprints are equal.

Failure leaves the workspace unavailable to the core. There is no rollback: fix the cause and run
the complete command again.

## Administration sidebar

The Administration sidebar exposes the same one-shot operation for the workspace selected in the
composer. It reads `GET /workspaces/:workspaceId/preprocessing`, invokes
`POST /workspaces/:workspaceId/preprocessing` for a run, and invokes
`DELETE /workspaces/:workspaceId/preprocessing` for clear. Both mutations delegate to the complete
workspace preprocessing service and do not expose partial stages.

The compact control has five states: `ready`, `required`, `running`, `blocked`, and `failed`.
**Run again** is available for `ready`, **Run** for `required`, and **Retry** for `failed`. **Clear**
appears to the left of that action whenever replaceable data can be cleared. It opens an inline
confirmation that names both the data removed and the memory retained. All run actions invoke the
same complete, idempotent preprocessing operation. A blocked state
names the current unmet prerequisite and links it to an operator action, but has no run diagnostic
because no preprocessing run started. A failed state shows only the latest bounded diagnostic:
failed stage, safe error code, and finish time. PostgreSQL overwrites that diagnostic when the next
run starts or finishes; there is no preprocessing history or rollback UI. The full service log is
available to operators with `docker compose logs core`.

Once a non-ready state is known, **New session** is disabled. Core admission remains the
authoritative enforcement boundary if the browser has not loaded the state yet.

## Container boundary

The host CLI resolves the selected core image to an immutable local image ID and runs only:

```text
docker compose run --rm --no-deps --no-TTY --name <owned-name> \
  workspace-maintenance preprocess-run

docker compose run --rm --no-deps --no-TTY --name <owned-name> \
  workspace-maintenance preprocess-clear
```

The request is one bounded schema-versioned JSON document on stdin. The maintenance process strips
all `THT_CATALOG_*` variables before spawning Python. JSON mode keeps stdout pristine.

## Result and exit codes

The public result includes `schemaVersion`, `status`, `code`, workspace/revision identity,
`operation`, completed stages, safe counts, artifact digests, and the input fingerprint. It never
contains credentials, connection strings, sampled values, SQL, or raw child errors.

- `0`: preprocessing run or clear succeeded;
- `1`: operational or preprocessing failure;
- `2`: invalid host-side invocation.