ThothII/docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md

P2–P6 Registry-Aware Workspace Preprocessing Design

Status: Reviewed execution design; P1 automated and manual acceptance PASS Date: 2026-08-10 Source: docs/prd/2026-08-09-workspace-preprocessing-prd.md D2–D6

1. Goal and delivery protocol

Connect the existing preprocessing engine to a revision-pinned schema-v3 Git workspace without requiring Python, Node, or Pi on the host. Delivery remains on codex/git-workspace-registry, with separately reviewable commits and a hard user checkpoint after each of P2, P3, P4, P5, and P6.

Each plan has its own clean-state automated process goal. Focused tests run while implementing each plan; the aggregate P2–P6 Docker/process smoke and full repository verification run only after P6. docs/testing/p2-p6-manual-verification.md is the single living manual walkthrough and records one independent section and decision for each plan.

2. Chosen architecture

The existing native thothctl binary becomes the only host interface. New workspace commands use the installation descriptor to reconstruct the exact Compose project and launch a one-shot maintenance process from the selected core image. The process uses the installation's registry, sessions, Qdrant, embedding, Git credentials, connector bindings, and secret mounts. It does not call a running backend HTTP server and does not require a host language runtime.

thothctl --installation ... workspace <command>
  -> docker compose ... run --rm workspace-maintenance <fixed argv>
  -> compiled Node operator entrypoint in the core image
  -> WorkspaceRegistry + runtime renderer + installation bindings
  -> restrictive temporary harness config
  -> existing tht commands
  -> DWH / artifacts / Qdrant / internal Ollama

The operator entrypoint shares production classes with the backend, but is a separate process and interface. It writes pristine JSON to stdout, bounded sanitized diagnostics to stderr, and never returns a secret value or rendered secret-bearing transport URL.

3. Common identity and persistence

Every operation binds these values before doing work:

• workspace ID;
• exact 40-hex active Git commit;
• catalog entry (thoth-workspaces.yaml) and exact canonical descriptor snapshot (the catalog blob and descriptor blob at that same commit; a docs-only or content-only commit is still a distinct revision even when the descriptor blob is unchanged, because the commit is authoritative);
• installation-local bindings resolved under configured secret roots;
• runtime roots beneath /data/sessions/<workspace-id>;
• internal Qdrant/Ollama contract;
• the existing harness DWH ownership binding in P2 and its compatible, versioned P3 refinement.

Mutable preprocessing state and outputs remain beneath the workspace runtime boundary. Operator job state is an atomic, versioned JSON document under /data/sessions/<workspace-id>/preprocessing/. It records the operation, revision, existing harness ownership binding, child run IDs, completed stages, manual checkpoint, and terminal status. P2 resume is same-revision only. P5 deliberately upgrades FK review to a controlled revision transition and does not pretend an old same-revision resume token remains valid after a Git push.

P3 separates reusable DWH-derived state (keyed by the revision-independent effective DWH binding) from revision-scoped curated/semantic state. Schema/Evidence points and reads bind workspace_revision; their point identities include that revision. Memory/solved records remain workspace-wide. The harness config gains an explicit workspace-global paths.memory root rendered as /data/sessions/<workspace-id>/memory; legacy configs without it continue to resolve memory beneath artifacts/memory until explicitly migrated. All memory commands, locking, registry JSONL, and projection rebuild use the explicit root when present. Revision-scoped artifact/corpus roots therefore cannot split the canonical memory registry. DWH generations may be safely reused when their existing effective DWH binding is unchanged.

4. P2 — host preprocessing CLI

Command contract

The initial public family is intentionally small:

thothctl ... workspace inspect --workspace <id> [--json]
thothctl ... workspace preprocess dwh --workspace <id> [--resume <id>] [--json]
thothctl ... workspace schema suggest-fks --workspace <id> [tht-safe options] [--json]
thothctl ... workspace schema check --workspace <id> [--json]
thothctl ... workspace index-schema --workspace <id> [--json]
thothctl ... workspace preprocess evidence --workspace <id> [--dry-run] [--resume <id>] [--json]
thothctl ... workspace preprocess run --workspace <id> [--resume <id>] [--json]

preprocess run orders DWH introspection/LSH, FK review, schema indexing, and Evidence. When newly suggested FK changes need human review it records manual_review_required and exits without indexing schema or Evidence. A later explicit resume continues only after schema check succeeds. A fixture with already-curated FK can complete without a human pause.

P2 proves the complete chain with controlled REST DWH and HTTP Evidence fixtures. A filesystem Evidence source is parsed and rendered but the operational command stops with the stable code evidence_materialization_required; commit-addressed filesystem consumption belongs to P6.

The command never edits or pushes the registry repository. Curators use an ordinary review clone. P2 retains the existing engine's generation publication, idempotency, dry-run, and child resume semantics rather than adding a second preprocessing engine.

P2 includes the missing machine interfaces in the harness: JSON forms for FK suggest/check and schema indexing, Evidence job identity from runtime_identity.workspace_id rather than a temporary config filename, and bounded safe ingress/egress for --from-sql and annotation export. Host input is a readable regular non-symlink file, is size-bounded by thothctl, and is streamed over the one-shot process stdin rather than mounted as an arbitrary host directory.

The one-shot job is a dedicated Compose profile/service, not compose run core. It has no Pi auth mount, no writable Pi state, and an entrypoint that does not run Pi trust initialization. Git and connector override generators attach only the credentials required by the selected workspace operation to this service.

5. P3 — effective configuration and .tht-dwh ownership

P2 initially consumes the existing harness schema-v1 ownership contract unchanged. P3 makes that contract reproducible across the operator and session paths without silently invalidating existing generations. The current exclusion of session_storage and runtime_identity is preserved: a Git content-only commit must not force DWH introspection when the effective DWH configuration is unchanged.

P3 introduces a versioned shared canonicalizer for the non-secret effective DWH/preprocessing configuration and a stable logical config-source identity. It replaces dependence on random temporary config filenames while retaining a compatibility reader and explicit migration for existing OWNER.json schema-v1 roots. It also adds the explicit workspace-global memory root and a migration that copies and verifies one legacy canonical JSONL under the workspace lock before rebuilding its Qdrant projection; conflicting legacy registries fail closed. No in-place reinterpretation is allowed.

Reusable DWH cache roots are keyed by the versioned effective DWH binding. Revision-scoped runtime roots receive verified physical/LSH snapshots from that cache, while annotations, corpus ACTIVE, and schema/Evidence Qdrant records remain revision-specific. Qdrant schema/Evidence point IDs and queries include workspace_revision; memory/solved identities and queries remain workspace-wide.

P3 proves that the operator and a session render the same effective DWH binding, that semantically identical revisions reuse it, and that a changed endpoint/transport/database/schema/root-affecting policy fails closed. Documentation explains .tht-dwh, immutable generations, OWNER.json, ACTIVE, input versus config fingerprints, safe migration, regeneration, and recovery.

6. P4 — Qdrant collection lifecycle

A shared TypeScript collection manager owns Qdrant collection and payload-index reconciliation. Session admission and the operator path both call it.

Self-heal may:

• create a missing collection with exactly 1024 dimensions and cosine distance;
• create any missing required keyword index;
• tolerate an already-compatible concurrent creator and re-read final state.

Self-heal never mutates incompatible dimensions, distance, or index types. Those return semantic_index_incompatible.

The host CLI adds guarded collection inspection and rebuild. Rebuild requires the exact workspace ID, exact collection name repeated as confirmation, and an explicit destructive flag. It uses a cross-process quiescence protocol rather than trusting the one-shot job: thothctl acquires the installation lifecycle lock, asks the running backend to durably activate maintenance, verifies the complete session inventory is closed/finalized/archived, and polls a new loopback-only internal quiescence endpoint until both admission leases and PiProcessManager.count() are zero. It then stops core, rechecks that the container is stopped, and starts the dedicated maintenance service. The maintenance marker prevents a racing restart from admitting work. This backend-mediated drain is an explicit exception to the ordinary preprocessing path's no-backend-HTTP rule.

The job also verifies no preprocessing lock is held. It deletes only the descriptor-owned collection, recreates the complete contract, verifies it, and emits JSON. No prefix matching or global Qdrant mutation is allowed. A durable rebuild state is written before deletion; success restarts core and clears maintenance only after health verification. Failure after deletion leaves maintenance active and provides an explicit recovery/recreate command rather than claiming rollback of lost vector data.

7. P5 — curated FK annotations in Git

The canonical path is fixed, not descriptor-configurable:

<workspace-id>/schema/annotations.yaml

The registry validates that the object is a regular Git blob at the same commit as the descriptor. Absence remains compatible and produces an empty canonical annotation set plus a warning until a curator publishes one. Symlinks, submodules, trees at the file path, cross-namespace paths, and malformed annotations are rejected.

On activation and before preprocessing/session use, the exact blob is read with fixed Git argv, validated by the harness annotation parser, and atomically synchronized to the immutable revision-qualified runtime root:

/data/sessions/<workspace-id>/revisions/<commit>/artifacts/mschema/annotations.yaml

P3 changes operator and session rendering so artifacts and indexes select that exact revision root; the shared session-manifest root remains /data/sessions/<workspace-id>/sessions. Existing workspace-global artifact roots are treated as legacy input and require the explicit P3 migration; there is no mutable compatibility symlink or pointer used by pinned runtimes.

The synchronized file has restrictive mode and an adjacent ownership manifest containing workspace, commit, blob ID, content digest, and destination. A newer active revision writes a different directory, so a pinned historical runtime continues to receive its own revision. physical.yaml remains generated locally and is never published.

The annotation blob is bounded (16 MiB), UTF-8, and parsed before synchronization. The preprocessing CLI never pushes curated content. P2 same-revision local review is superseded in P5 by an explicit controlled transition: after commit/push/pull, the operator reviews the current Git blob against the recorded candidate, runs workspace schema accept --run <id> --yes, and records the accepted candidate/current-blob digests and new revision. Continuation requires that exact accepted blob and compatible reusable DWH binding; otherwise it starts a new run. An empty file or schema check alone is not evidence of human review.

8. P6 — commit-addressed Evidence materialization

For filesystem Evidence, the registry materializes exactly <id>/evidence from the pinned commit into an immutable revision content root. It does not consume the mobile registry checkout and does not resolve against author files.

Materialization uses fixed Git plumbing to enumerate object type, mode, path, object ID, and bytes. It rejects every symlink at any depth, gitlink/submodule, device/FIFO/socket, unsupported mode, absolute/traversing/non-normalized path, cross-workspace namespace, duplicate normalized path, oversized individual source object, or object identity change. No archive is extracted by a shell.

Files are written with no-follow/exclusive semantics beneath a fresh owned staging directory. Every file is hashed and recorded in a bounded manifest. Installation-local non-secret limits bound entry count, cumulative bytes, path/segment bytes, and manifest bytes; conservative defaults are documented and may be raised deliberately for large repositories. Materialization streams blobs and performs a disk-space preflight, so per-file-valid adversarial trees cannot exhaust memory or inodes silently. The complete tree and manifest are fsynced and atomically renamed only after all checks pass. A subsequent consumer revalidates destination ownership and the manifest before reuse. Partial staging is removed without following links.

The runtime renderer receives the verified immutable content root, after which the existing filesystem Evidence adapter may discover only beneath that root. Corpus ACTIVE and Evidence vector records are revision-scoped, and retrieval requires the pinned revision. Retention keeps materialized and derived roots for every retained/pinned workspace revision and removes only unreferenced, manifest-owned roots. P6 also makes P2's filesystem Evidence path operational and removes the temporary stable stop.

9. Error and output contract

Stable public codes include:

• workspace_not_found / workspace_not_activatable;
• binding_missing;
• preprocessing_conflict;
• preprocessing_resume_mismatch;
• manual_review_required;
• evidence_materialization_required (P2–P5 only);
• effective_config_mismatch;
• semantic_index_incompatible;
• annotation_invalid;
• evidence_materialization_unsafe.

JSON output contains status, stable code, workspace ID, revision, operation/run ID, completed stage names, counts, and safe artifact identities. It excludes endpoint credentials, secret contents, query-bearing signed URLs, raw child stderr, and arbitrary exception strings.

One writer lock per workspace serializes preprocessing, annotation synchronization, collection rebuild, and materialization publication where they could conflict. Read-only inspection remains concurrent.

10. Verification and manual acceptance

Each Px provides one clean-state process goal with unique ownership under .artifacts/p<id>-..., no retry, fixture-only secrets, machine/readable reports, secret scan, exact cleanup, and retained --keep mode. Focused unit/integration/type/lint tests are evidence for that Px. After each Px the user receives a report and an independently runnable manual section, and work stops for explicit authorization.

After P6, one aggregate test starts from a new Git registry and installation, executes P2 through P6 with real local Git, REST fixtures, Qdrant, and Ollama, proves DWH/FK/schema/Evidence outputs, repeats for idempotency, proves a second installation can consume the same Git workspace with its own local state, exercises negative security cases, and cleans only owned resources. Only then are the full harness/backend/frontend suites and builds run.

A GUI/backend preprocessing endpoint, real PSD migration, SSH runtime transport, and policy-driven long-term GC beyond the existing engine remain outside P2–P6.