# P2 Host Workspace Preprocessing CLI Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Implement P2/D2 as a native `thothctl workspace` interface that runs the existing preprocessing engine in a hardened one-shot container, derives its configuration from one active schema-v3 Git workspace revision plus installation-local bindings, and requires no Python, Node, Pi, or running backend HTTP service on the host. **Architecture:** `thothctl` validates a closed command grammar, reconstructs the exact installation Compose project, resolves the selected core image to an immutable Docker image ID, and starts only the profile-gated `workspace-maintenance` service with `--no-deps`. A compiled Node entrypoint reads an already-active immutable registry snapshot, uses the same binding resolver and runtime renderer as sessions, writes a deterministic revision-owned protected harness config, and invokes fixed existing `tht` commands. A versioned coordinator state and one kernel-released writer lock serialize mutation, preserve outer/child resume identity, and stop at a digest-bound FK review checkpoint. **Tech Stack:** Go 1.24 (`thothctl`), Docker Compose v2, Node.js 22, TypeScript 5, Python 3.12, Typer, Pydantic 2, Qdrant 1.18.2, the internal Ollama-compatible embedding interface, Vitest, pytest, Bash/Node acceptance tooling. **Source PRD and design:** `docs/prd/2026-08-09-workspace-preprocessing-prd.md` D2/P2, RF1.2–RF1.4, RF2, RF3.1, RF4.1, RF5.2, RF8.5–RF8.6, RNF1–RNF9; `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` §§1–4, 9–10; `docs/testing/p2-p6-manual-verification.md` P2. **Planning status:** DESIGN/PLAN ONLY. Do not change production code, start P2 implementation, or create a persistent implementation goal until the reviewer asks for plan validation and then gives explicit implementation approval. --- ## P2 completion contract P2 is complete only when all of the following are true: 1. The only public host interface is the installed native `thothctl` binary. Docker/Compose is required, but host Python, Node, Pi, `tht`, and a running Fastify backend are not. 2. Every command consumes an already-active, validated P1.1 registry snapshot and binds the exact workspace ID, 40-hex commit, catalog blob (`thoth-workspaces.yaml`), descriptor blob/digest, installation bindings, runtime roots, and selected internal semantic contract before mutation. A docs-only or content-only commit is still a distinct revision even when the descriptor blob is unchanged, because the commit is authoritative. 3. Operator and session configuration use the same `resolveRuntimeBindings` and `renderRuntimeConfig` implementation. P2 uses one deterministic same-revision config-source path so current schema-v1 DWH/Evidence resume works; P3 later introduces cross-revision canonical effective identity and explicit migrations. 4. DWH introspection+LSH, FK suggestion/check, schema indexing, and HTTP Evidence preprocessing invoke the existing harness engine through fixed argv and pristine JSON machine interfaces. No second preprocessing engine is added. 5. A full run with new FK candidates stops before schema/Evidence writes. Continuation requires a reviewer-supplied annotations file and an explicit acknowledgement of the exact candidate digest; `schema check` alone is not treated as human approval. 6. Mutating P2 operations are safe while roots and semantic point IDs are still workspace-global: under the workspace writer lock they refuse if any resumable session is pinned to a different workspace revision. P3 removes this temporary restriction by introducing revision-scoped curated/semantic state. 7. P2 never creates, repairs, deletes, or rebuilds a Qdrant collection. Schema/Evidence writes require an already-existing, exactly compatible collection and a harness `require_existing` mode that cannot race into auto-create. P4 owns lifecycle reconciliation. 8. HTTP Evidence is operational only under installation-local egress policy. Private hosts require an exact installation allowlist; redirects are rechecked; metadata/link-local targets are always refused. S3 custom/private/insecure endpoints and ambient credentials remain fail-closed in P2 unless a later separately reviewed plan expands policy. 9. Filesystem Evidence is rendered but execution stops before discovery with `evidence_materialization_required` and no partial corpus/vector publication. P6 owns materialization and symlink/containment checks. 10. `postgres_direct` and `rest_api` routing remain supported and are regression-tested; the clean P2 process goal uses controlled REST. `ssh_tunnel` returns a stable fail-closed result until P10. 11. One clean-state product-path integration command passes without retry, produces retained machine/human reports and a secret scan, and proves exact cleanup. Manual P2 acceptance remains independent and PENDING. 12. Work stops after the P2 handoff. No P3 work begins without a new explicit user authorization. ## Truthful command status at the P2 checkpoint | Command | P2 status | Deliberate boundary | |---|---|---| | `workspace inspect` | Operational | Reads active snapshot only; does not pull/activate Git | | `workspace preprocess dwh` | Operational for REST/direct | Same-revision config identity; cross-revision reuse is P3 | | `workspace schema suggest-fks` | Operational, machine-safe | Candidate export only; no automatic human acceptance | | `workspace schema check` | Operational | Imports reviewed annotations and records digest-bound local P2 acknowledgement | | `workspace index-schema` | Operational with compatible pre-existing collection | Collection create/repair/rebuild is P4 | | `workspace preprocess evidence` | Operational for policy-allowed HTTP; filesystem deferred | Filesystem materialization is P6; S3 expansion needs separate policy review | | `workspace preprocess run` | Operational with FK checkpoint | Git-canonical annotations are P5; revision-global writes use the P2 session-inventory guard | P2 is therefore the host CLI/orchestration checkpoint, not final acceptance of the PSD filesystem path or the complete PRD chain. ## Frozen host command grammar ```text thothctl --installation /thothii-installation.yaml workspace inspect --workspace [--json] thothctl ... workspace preprocess dwh --workspace [--resume ] [--json] thothctl ... workspace schema suggest-fks --workspace [--from-sql ]... [--assume ]... [--output ] [--json] thothctl ... workspace schema check --workspace [--annotations --reviewed-candidates ] [--json] thothctl ... workspace index-schema --workspace [--json] thothctl ... workspace preprocess evidence --workspace [--dry-run] [--resume ] [--json] thothctl ... workspace preprocess run --workspace [--resume ] [--json] ``` Rules: - `--workspace` occurs exactly once and matches `[a-z][a-z0-9-]{2,62}`. - All run IDs are 32 lowercase hex characters and identify outer P2 state, never a path or raw child checkpoint. - At most 32 `--from-sql` files, 1 MiB each and 16 MiB total. `thothctl` opens each as a canonical regular non-symlink/reparse-point file, rechecks identity after reading, and streams a schema-versioned request over stdin. No host directory is mounted. - `--assume` occurs at most 256 times; each value is at most 256 bytes and is validated before Compose. - `--annotations` is a single UTF-8 YAML file, at most 16 MiB. `--reviewed-candidates` is mandatory with it and must equal the persisted candidate artifact digest. The pair is invalid without both flags. - `--output` is created exclusively with restrictive permissions after the returned workspace/run/digest identity has been verified. Existing files, symlinks, hardlinks, and Windows reparse targets are refused. - Existing harness `suggest-fks --write` is intentionally not exposed: an automatic merge is not a human review decision. - No unknown flag, passthrough separator, environment-selected command, shell fragment, or arbitrary container entrypoint is accepted. ## Public result and exit contract The one-shot entrypoint always emits exactly one bounded schema-versioned JSON object. `thothctl --json` parses it strictly and re-encodes it, so Compose progress cannot contaminate stdout. Human mode renders only allowlisted fields. ```ts interface WorkspaceOperationResult { schemaVersion: 1; status: "succeeded" | "unchanged" | "dry_run" | "blocked" | "failed"; code: | "ok" | "workspace_not_found" | "workspace_not_activatable" | "binding_missing" | "preprocessing_conflict" | "preprocessing_resume_mismatch" | "manual_review_required" | "evidence_materialization_required" | "effective_config_mismatch" | "semantic_index_incompatible" | "annotation_invalid" | "egress_policy_refused"; workspaceId: string; workspaceRevision: string; descriptorBlob: string; operation: string; runId?: string; childRuns?: Record; completedStages: string[]; counts?: Record; artifactIdentities?: Array<{ kind: string; digest: string }>; warnings?: string[]; } ``` - Exit `0`: `succeeded`, `unchanged`, or `dry_run`. - Exit `3`: expected operator checkpoint/block (`manual_review_required`, `evidence_materialization_required`, lock/revision conflict). - Exit `2`: host grammar or unsafe local file error. - Exit `1`: operational failure. - Stdout JSON maximum: 1 MiB. Sanitized stderr maximum: 64 KiB. Child stdout/stderr and every stage have explicit limits/timeouts. - Never return descriptor endpoints with credentials/query strings, secret contents or paths, signed URLs, raw SQL, rendered configuration, raw child stderr, arbitrary exception text, Qdrant payload contents, or host/container environment dumps. ## P2 state and identity layout ```text /data/sessions//preprocessing/ ├── writer.lock ├── runtime-config/ │ └── <40-hex-revision>.yaml ├── runtime-config-manifests/ │ └── <40-hex-revision>.json ├── jobs/ │ └── <32-hex-outer-run-id>.json ├── fk-candidates/ │ └── <32-hex-outer-run-id>.yaml └── fk-reviews/ └── <32-hex-outer-run-id>.json ``` - `writer.lock` is a regular `0600` file held by a Linux kernel advisory lock for the entire outer operation. The file may persist; the kernel lock is released on crash/container death. `inspect` never takes it. - Lock order is always P2 workspace writer lock → existing harness stage lock. Harness code never acquires the P2 lock, preventing inversion/deadlock. - Every directory component is opened/validated without following symlinks. State files are `0600`, written to an exclusive sibling, fsynced, renamed, and parent-fsynced. Hardlink count must be one. - The deterministic config path fixes P2 same-revision `config_source` identity. Its manifest binds workspace, revision, descriptor blob, config SHA-256, file identity, and the current existing harness ownership binding. Same path + different bytes returns `effective_config_mismatch`; P3 introduces semantic cross-revision equivalence. - Job state binds operation, revision, catalog blob, descriptor blob, config digest, non-secret binding identity, completed stage records, child run IDs, candidate/review digests, and terminal status. Resume revalidates all fields and reconciles a child publication that completed immediately before an outer-state crash. - Before any schema/Evidence mutation, enumerate resumable session manifests for the workspace. A different pinned revision returns `preprocessing_conflict`; no write begins. This is the explicit P2 bridge until P3 revision isolation. ## One-shot service security contract `workspace-maintenance` is a dedicated profile service, not `compose run core`: - same exact selected core image, resolved to its immutable local image ID; generated final override uses that ID and `pull_policy: never`; - `docker compose run --rm --no-deps --no-TTY --name workspace-maintenance ...`; - no `build`, frontend, published port, Pi auth, Pi state, Pi trust initialization, Docker socket, home credential directory, or arbitrary command; - non-root `10001`, `read_only: true`, `cap_drop: [ALL]`, `no-new-privileges:true`, restrictive tmpfs, registry active snapshots read-only, sessions root writable; - only operation-required connector/Evidence secret files are mounted; AWS ambient environment is cleared; - semantic commands require already-running healthy Qdrant/embedding services and do not start/stop them; DWH/inspect commands do not start dependencies; - exact operation labels and container identity are recorded. Cancellation terminates the process group, verifies the owned container labels/image, removes only that container, and preserves all pre-existing services, volumes, and networks; - fully rendered Compose is validated before launch, and post-run container/image identity is checked before accepting output. --- ## Target file map **Native host CLI** - `tools/thothctl/cmd/thothctl/main.go`, `main_test.go`: public grammar/help, dispatch, exit codes. - Create `tools/thothctl/internal/workspaceops/operations.go`, `operations_test.go`: immutable image resolution, generated override, Compose run, cancellation cleanup, JSON validation. - `tools/thothctl/internal/config/installation.go`, `installation_test.go`: maintenance service override and operation-specific binding discovery. - `tools/thothctl/internal/safeio/files.go`, platform files/tests: bounded no-follow input and exclusive output. - `tools/thothctl/internal/output/sanitize.go`, tests: bounded redaction. - `tools/thothctl/internal/pi/update.go`, tests: selected image override must pin both `core` and `workspace-maintenance`. **Compose/image boundary** - `compose.yaml`: dedicated profile-gated service with shared image identity and least privilege. - `deploy/compose.local.yaml`, `deploy/compose.server.yaml`: correct registry/session storage semantics. - `deploy/compose.git-https.yaml`, `deploy/compose.git-ssh.yaml`: do not attach Git credentials to P2 active-snapshot operations. - `scripts/generate-connector-secrets-override.sh`: operation-specific maintenance secrets. - Create `docker/workspace-maintenance-entrypoint.sh`; modify `docker/core.Dockerfile`. - Retire/redirect fixture-only `deploy/compose.preprocess.yaml` as a non-public compatibility test path; do not leave two operator commands. **Shared Node operator** - Create `backend/src/workspaces/runtime-config-lease.ts`: shared snapshot read/render and deterministic protected config lease. - Modify `backend/src/tht/tht-runner.ts` to delegate session/operator rendering to the shared component without changing route behavior. - Create `backend/src/workspaces/preprocessing-state.ts`: state schema, durable writes, locks, resume reconciliation. - Create `backend/src/workspaces/preprocessing-service.ts`: closed stage coordinator and security preflights. - Create `backend/src/workspace-maintenance.ts`: compiled stdin/argv entrypoint and pristine result encoder. - Add tests: `backend/test/workspace-runtime-config-lease.test.ts`, `workspace-preprocessing-state.test.ts`, `workspace-preprocessing-service.test.ts`, `workspace-maintenance.test.ts`. **Harness machine contracts** - `harness/tht/cli/preprocess_cmd.py`: authoritative runtime workspace identity; require-existing collection mode. - `harness/tht/cli/schema_cmd.py`: extracted deterministic helpers, JSON suggest/check, safe SQL staging and annotation validation. - `harness/tht/cli/vector_cmd.py`: JSON schema-index result. - `harness/tht/adapters/vector/qdrant.py`: explicit non-creating strict mode for P2. - Tests: `harness/tests/test_preprocess_cli.py`, `test_schema_fk_annotations.py`, `test_qdrant_cli_commands.py`, `test_registry_evidence_config.py`, `test_http_evidence_source.py`, plus new focused security cases. **Docs and gates** - Create `docs/contracts/workspace-preprocessing-cli.md`. - Update local/server installation manuals and `docs/testing/p2-p6-manual-verification.md` P2 only. - Create `scripts/p2-acceptance.sh`, `backend/scripts/p2-acceptance.mjs`, `backend/scripts/p2-acceptance.test.mjs`. - Create `scripts/p2-manual-acceptance.sh`, `backend/scripts/p2-manual-acceptance.mjs` only if needed to generate the isolated walkthrough lab; automation must never create PASS. - Update `PROJECT_STATE.md` only after implementation evidence exists. --- ### Task 1: Freeze the native CLI, file-ingress, and result contracts **Files:** - Modify: `tools/thothctl/cmd/thothctl/main.go` - Modify: `tools/thothctl/cmd/thothctl/main_test.go` - Modify: `tools/thothctl/internal/safeio/files.go` - Modify platform-specific safe-I/O tests - Create: `tools/thothctl/internal/workspaceops/operations.go` - Create: `tools/thothctl/internal/workspaceops/operations_test.go` - Create: `docs/contracts/workspace-preprocessing-cli.md` - [ ] **Step 1: Write RED parser-table tests** for every valid command above and for duplicate/missing/unknown flags, invalid IDs, incompatible annotation flags, option-count/size limits, and passthrough/shell attempts. - [ ] **Step 2: Run** `cd tools/thothctl && go test ./cmd/thothctl ./internal/workspaceops -run 'Workspace|workspace' -v` and verify the new tests fail because `workspace` is unknown. - [ ] **Step 3: Add closed request types** (`InspectRequest`, `DwhRequest`, `SuggestFksRequest`, `CheckSchemaRequest`, `IndexSchemaRequest`, `EvidenceRequest`, `RunRequest`) and a parser that cannot represent arbitrary argv. - [ ] **Step 4: Write RED safe-I/O tests** for symlinks, hardlinks, directory input, replacement during read, Windows reparse points, existing output, >1 MiB SQL, >16 MiB total, and non-UTF-8 annotation input. - [ ] **Step 5: Implement bounded reads and exclusive restrictive output** using existing platform seams; return only generic file errors. - [ ] **Step 6: Add schema-v1 stdin request/result validation** with exact field allowlists and output bounds. - [ ] **Step 7: Run focused Go tests and `gofmt -w`**, then `go test ./...`. - [ ] **Step 8: Commit:** `feat: define P2 host workspace command contract`. ### Task 2: Add pristine harness JSON interfaces without changing the engine **Files:** - Modify: `harness/tht/cli/schema_cmd.py` - Modify: `harness/tht/cli/vector_cmd.py` - Modify: `harness/tht/cli/preprocess_cmd.py` - Modify: `harness/tests/test_schema_fk_annotations.py` - Modify: `harness/tests/test_qdrant_cli_commands.py` - Modify: `harness/tests/test_preprocess_cli.py` - [ ] **Step 1: Write RED tests** requiring `schema suggest-fks --json`, `schema check --json`, and `vector index-schema --json` to emit exactly one JSON object on stdout for success and failure, with no color/prose contamination. - [ ] **Step 2: Write RED deterministic FK tests** for bounded staged SQL files, stable candidate ordering, candidate SHA-256, annotation import, orphan counts, and no implicit review/write. - [ ] **Step 3: Write RED Evidence identity test** showing a config named `/dev/fd/3` still uses `runtime_identity.workspace_id`, never the config basename. - [ ] **Step 4: Run:** ```bash cd harness .venv/bin/pytest -q \ tests/test_schema_fk_annotations.py \ tests/test_qdrant_cli_commands.py \ tests/test_preprocess_cli.py ``` Expected: FAIL only on the new machine-contract assertions. - [ ] **Step 5: Extract pure helpers** returning typed dictionaries/models; keep existing human commands as renderers over the same helpers. - [ ] **Step 6: Implement the JSON flags and authoritative workspace identity**. Catch expected exceptions and emit stable safe codes; never serialize arbitrary exception text. - [ ] **Step 7: Run the three focused files and touched Ruff**: ```bash .venv/bin/ruff check \ tht/cli/schema_cmd.py tht/cli/vector_cmd.py tht/cli/preprocess_cmd.py \ tests/test_schema_fk_annotations.py tests/test_qdrant_cli_commands.py tests/test_preprocess_cli.py ``` - [ ] **Step 8: Commit:** `feat: add P2 harness machine contracts`. ### Task 3: Add a non-creating semantic writer mode **Files:** - Modify: `harness/tht/config.py` - Modify: `harness/tht/adapters/factory.py` - Modify: `harness/tht/adapters/vector/qdrant.py` - Modify: `harness/tests/test_qdrant_vector_store.py` - Modify: `harness/tests/test_qdrant_cli_commands.py` - Modify: `harness/tests/test_registry_evidence_config.py` - [ ] **Step 1: Write RED tests** proving operator mode refuses a missing collection without issuing create/index mutations, refuses wrong dimensions/distance/index type, and still writes to an existing compatible collection. - [ ] **Step 2: Run the focused tests** and confirm current `_ensure_collection(strict=True)` incorrectly creates the collection. - [ ] **Step 3: Add an internal rendered field** such as `vectors.collection_lifecycle: require_existing`; it is not a descriptor option and defaults to legacy behavior for non-operator configs. - [ ] **Step 4: Thread the mode through the factory/store** and perform a read-only exact collection/index preflight before any upsert. - [ ] **Step 5: Add a race regression**: delete the collection after preflight and prove the write fails rather than recreates it. - [ ] **Step 6: Run focused pytest and touched Ruff.** - [ ] **Step 7: Commit:** `fix: prevent P2 from owning Qdrant lifecycle`. ### Task 4: Extract the shared runtime configuration lease **Files:** - Create: `backend/src/workspaces/runtime-config-lease.ts` - Create: `backend/test/workspace-runtime-config-lease.test.ts` - Modify: `backend/src/tht/tht-runner.ts` - Modify: `backend/test/tht-runner.test.ts` - Modify: `backend/test/workspace-runtime-handoff.test.ts` - [ ] **Step 1: Write RED equivalence tests** feeding the same immutable snapshot, env, roots, installation overlay, and semantic contract to the session and operator callers and requiring byte-identical YAML. - [ ] **Step 2: Write RED identity/safety tests** for snapshot replacement, wrong commit/path, symlink/hardlink, wrong workspace ID, unstable config destination, same-revision changed bytes, mode, fsync/rename failure, and cleanup. - [ ] **Step 3: Run:** ```bash cd backend npx vitest run \ test/workspace-runtime-config-lease.test.ts \ test/tht-runner.test.ts \ test/workspace-runtime-handoff.test.ts ``` - [ ] **Step 4: Move snapshot validation, runtime roots, installation-overlay parsing, binding resolution, and rendering** out of `ThtRunner` into one explicit-input component. - [ ] **Step 5: Preserve session behavior**: `ThtRunner.acquireWorkspaceRuntime` delegates to the component and retains its current opaque FD-backed temporary lease. - [ ] **Step 6: Add operator mode**: deterministically publish `/data/sessions//preprocessing/runtime-config/.yaml` plus a manifest, mode `0400/0600`, and set `collection_lifecycle: require_existing`. - [ ] **Step 7: Prove same-revision rerun path identity** and changed config/binding refusal. Do not implement P3 semantic cross-revision canonicalization. - [ ] **Step 8: Run focused tests, `npx tsc --noEmit -p .`, and `npm run build`.** - [ ] **Step 9: Commit:** `refactor: share registry runtime configuration leases`. ### Task 5: Build durable outer state, locking, and revision guard **Files:** - Create: `backend/src/workspaces/preprocessing-state.ts` - Create: `backend/test/workspace-preprocessing-state.test.ts` - Modify: `docker/core.Dockerfile` (install/pin the kernel lock utility only when the implementation proves it is absent) - [ ] **Step 1: Write RED state-schema tests** for valid state, same-operation resume, cross-workspace/revision/operation/config mismatch, tampering, run-ID traversal, restrictive modes, atomic failure, and bounded fields. - [ ] **Step 2: Write RED cross-process lock tests** with two processes/containers: one wins, one receives `preprocessing_conflict`, and SIGKILL releases the kernel lock without deleting unrelated state. - [ ] **Step 3: Write RED session-inventory tests**: no sessions/current-only sessions permit mutation; a resumable different-revision manifest blocks; finalized/archived sessions follow existing resume policy. - [ ] **Step 4: Implement the exact state layout and durable write protocol** described above. - [ ] **Step 5: Implement lock acquisition ordering and safe conflict mapping.** Do not invent stale-PID deletion; the kernel owns lock lifetime. - [ ] **Step 6: Implement active-snapshot revalidation immediately before each mutating child stage** and the different-revision resumable-session guard. - [ ] **Step 7: Add crash reconciliation tests** where a child publishes DWH/corpus state but outer state has not yet advanced. - [ ] **Step 8: Run focused Vitest, typecheck, and build.** - [ ] **Step 9: Commit:** `feat: add P2 preprocessing operation state`. ### Task 6: Build the compiled inspect/operator boundary **Files:** - Create: `backend/src/workspace-maintenance.ts` - Create: `backend/src/workspaces/preprocessing-service.ts` - Create: `backend/test/workspace-maintenance.test.ts` - Create: `backend/test/workspace-preprocessing-service.test.ts` - Modify: `backend/src/workspaces/types.ts` only if a separate operator-code union cannot stay private - [ ] **Step 1: Write RED entrypoint process tests** for exact JSON, malformed/extra stdin, unknown command/field, stdout/stderr bounds, timeout, signal, raw exception/stderr redaction, and no Fastify listener. - [ ] **Step 2: Write RED `inspect` tests** for absent/corrupt/stale active state, migration-required descriptor, missing bindings, exact commit/blob/config identities, safe capability warnings, and no URL/secret output. - [ ] **Step 3: Run focused Vitest** and verify no operator exists. - [ ] **Step 4: Implement a closed `WorkspacePreprocessingService` dependency interface**: active registry reader, shared config lease, fixed child runner, state store, session inventory, semantic preflight, egress policy. - [ ] **Step 5: Implement active-snapshot-only acquisition.** P2 does not pull or activate Git; clean installations receive `workspace_not_activatable` with safe instructions. - [ ] **Step 6: Implement bounded child execution** with fixed executable/argv, `-c` after the subcommand, FD-backed config/input, process-group cancellation, per-stage timeout, and strict one-document child JSON parsing. - [ ] **Step 7: Implement `inspect` and result encoding.** - [ ] **Step 8: Run tests, typecheck, build, and verify `dist/workspace-maintenance.js` exists.** - [ ] **Step 9: Commit:** `feat: add P2 workspace maintenance operator`. ### Task 7: Implement DWH preprocessing and outer resume **Files:** - Modify: `backend/src/workspaces/preprocessing-service.ts` - Modify: `backend/test/workspace-preprocessing-service.test.ts` - Modify: `harness/tests/test_dwh_preprocess_job.py` - Modify: `harness/tests/test_lsh_job_resume.py` - [ ] **Step 1: Write RED service tests** requiring fixed `preprocess dwh --steps introspect,lsh --json -c /dev/fd/N`, child result validation, outer/child run IDs, completed stages, safe artifact digests, and failure mapping. - [ ] **Step 2: Add real harness regressions** for deterministic same-revision config-source path, second clean-process rerun, resume after introspection, changed binding/config refusal, and ACTIVE preservation on failure. - [ ] **Step 3: Run focused backend and harness tests.** - [ ] **Step 4: Implement `preprocess dwh`** under the outer writer lock and persist state before/after every child transition. - [ ] **Step 5: Reconcile a published child run after an injected outer crash** without rerunning or corrupting ACTIVE. - [ ] **Step 6: Verify REST and direct rendered routing.** SSH returns `workspace_not_activatable` with a P10 warning. - [ ] **Step 7: Run focused gates and commit:** `feat: run workspace DWH preprocessing from thothctl operator`. ### Task 8: Implement FK candidate export and digest-bound review **Files:** - Modify: `backend/src/workspaces/preprocessing-service.ts` - Modify: `backend/src/workspaces/preprocessing-state.ts` - Modify: relevant backend tests - Modify: `tools/thothctl/internal/workspaceops/operations.go` - Modify: Go tests - [ ] **Step 1: Write RED end-to-end unit/process tests**: new candidates create a bounded artifact and return `manual_review_required`; schema/Evidence child calls are absent. - [ ] **Step 2: Add safe host ingress tests** proving SQL and annotations travel only over stdin, are absent from Compose argv/state/logs, and staging files are removed. - [ ] **Step 3: Add safe host egress tests** for candidate export identity/digest, existing destination refusal, and sanitized JSON mode. - [ ] **Step 4: Implement `schema suggest-fks`** with candidate count/digest and optional exclusive output. - [ ] **Step 5: Implement `schema check`** in two modes: read-only orphan validation; or reviewed annotation import requiring the exact candidate digest. Persist review digest + annotation digest + workspace/revision. - [ ] **Step 6: Require the review record on full-run resume.** A mere zero-orphan result without reviewer digest is insufficient. - [ ] **Step 7: Preserve the boundary:** P2 updates runtime-local annotations only; it never writes Git. Output warns that P5 will supersede this local acknowledgement. - [ ] **Step 8: Run Go/backend/harness focused gates and commit:** `feat: add P2 FK review checkpoint`. ### Task 9: Implement schema indexing and Evidence policy boundaries **Files:** - Modify: `backend/src/workspaces/preprocessing-service.ts` - Modify: backend service tests - Modify: `harness/tests/test_http_evidence_source.py` - Modify: `harness/tests/test_registry_evidence_config.py` - Modify: `harness/tests/test_semantic_kind_isolation.py` - [ ] **Step 1: Write RED schema-index tests** for compatible pre-existing collection, deterministic JSON counts, idempotent repeat, missing/incompatible refusal, and no collection-create request. - [ ] **Step 2: Write RED Evidence tests** for no-Evidence warning/skip, HTTP dry-run/run/resume/unchanged/mutation, authoritative workspace identity, ACTIVE preservation, and filesystem early stop before adapter/Qdrant calls. - [ ] **Step 3: Write RED egress tests** for exact private-host allowlist, DNS re-resolution, redirect to private/link-local/metadata, signed URL query redaction, and refusal of S3 ambient/custom/private/insecure modes. - [ ] **Step 4: Implement installation-local egress-policy parsing** with exact bounded hostnames and no wildcard. Descriptor flags alone never grant network access. - [ ] **Step 5: Implement `index-schema` and `preprocess evidence`** with semantic preflight and stable result mapping. - [ ] **Step 6: Revalidate active revision and session inventory immediately before each write.** - [ ] **Step 7: Run focused tests/touched Ruff/backend typecheck/build and commit:** `feat: add guarded P2 semantic preprocessing`. ### Task 10: Implement the ordered full-run coordinator **Files:** - Modify: `backend/src/workspaces/preprocessing-service.ts` - Modify: `backend/test/workspace-preprocessing-service.test.ts` - Modify: `backend/test/workspace-maintenance.test.ts` - [ ] **Step 1: Write a RED stage-table test** for exact order `dwh → fk_suggest → fk_review/check → schema_index → evidence` and for no hidden/skipped mutation. - [ ] **Step 2: Add scenarios**: pre-curated/no-new-candidate completion; new-candidate block; digest-reviewed resume; no-Evidence warning; filesystem deferred block; each child failure; resume mismatch; outer crash reconciliation. - [ ] **Step 3: Implement the coordinator as an explicit state machine**, not recursive command dispatch. - [ ] **Step 4: Persist completion after each verified child artifact** and never mark a stage based only on exit code. - [ ] **Step 5: Prove unchanged rerun creates no duplicate schema/Evidence points or generation.** - [ ] **Step 6: Run focused Vitest, typecheck, build, and commit:** `feat: orchestrate the P2 preprocessing chain`. ### Task 11: Add the hardened maintenance service and selected-image handoff **Files:** - Modify: `compose.yaml` - Modify: `deploy/compose.local.yaml` - Modify: `deploy/compose.server.yaml` - Modify: connector/Git override files and generators as required - Create: `docker/workspace-maintenance-entrypoint.sh` - Modify: `docker/core.Dockerfile` - Modify: `tools/thothctl/internal/workspaceops/operations.go` - Modify: `tools/thothctl/internal/pi/update.go` - Modify relevant Go/Bash/Compose tests - [ ] **Step 1: Write RED Compose contract tests** for the exact security contract, profile, mounts, no Pi/no port/no build, local/server storage, and absence of Git credentials on active-snapshot operations. - [ ] **Step 2: Write RED image-precedence tests** across base/profile/operator overrides/current-image override; `core` and maintenance must resolve to the same immutable ID. - [ ] **Step 3: Write RED lifecycle tests** for `--no-deps`, pre-existing service preservation, interruption cleanup, hostile container name/label collision, tag replacement, and output rejection on post-run image mismatch. - [ ] **Step 4: Add the dedicated service and entrypoint**; the entrypoint executes only the compiled operator and never calls Pi trust setup. - [ ] **Step 5: Generate a per-operation final override** that pins immutable image ID, exact secrets, egress policy, and owned labels. Validate `docker compose config` structurally before run. - [ ] **Step 6: Extend Pi update/rollback override generation** so future selected images cannot split core and maintenance. - [ ] **Step 7: Run:** ```bash bash scripts/test-preprocess-compose-config.sh bash scripts/test-compose-secret-policy.sh bash scripts/test-default-compose.sh bash scripts/test-unified-compose.sh bash scripts/test-no-deployment-coupling.sh cd tools/thothctl && go test ./... ``` - [ ] **Step 8: Build the core image and invoke operator `--help` through the exact service** without starting Pi/backend/dependencies. - [ ] **Step 9: Commit:** `feat: package the P2 maintenance service`. ### Task 12: Complete host dispatch and supported-platform build contract **Files:** - Modify: `tools/thothctl/cmd/thothctl/main.go`, tests - Modify: `tools/thothctl/internal/workspaceops/operations.go`, tests - Modify: `scripts/build-thothctl.sh` - Modify: `scripts/test-thothctl-build-contract.sh` - Update operator docs - [ ] **Step 1: Add RED command-to-request-to-Compose tests** for all seven public commands, JSON/human output, exit mapping, secret redaction, and exact stdin. - [ ] **Step 2: Implement dispatcher integration** using only typed requests. - [ ] **Step 3: Cross-build the existing release matrix** and verify Windows input/output safety compiles. Do not claim Windows Docker behavior without a Windows Docker run. - [ ] **Step 4: Run `go test ./...`, build contract, `go vet ./...`, and `gofmt` check.** - [ ] **Step 5: Commit:** `feat: expose P2 workspace commands in thothctl`. ### Task 13: Build the clean-state automated P2 process goal **Files:** - Create: `scripts/p2-acceptance.sh` - Create: `backend/scripts/p2-acceptance.mjs` - Create: `backend/scripts/p2-acceptance.test.mjs` - Update: `.gitignore` only if the existing `.artifacts/` rule is insufficient The public command is: ```bash ./scripts/p2-acceptance.sh integration --keep ``` - [ ] **Step 1: Write RED acceptance-runner tests** for ownership-first state, unique run/project/container/image names, exact cleanup, `--keep`, injected failure, signal cleanup, report bounds, and no automatic retry. - [ ] **Step 2: Build a clean owned topology** under `.artifacts/p2-integration/p2-/`: local bare Git + author clone, active P1.1 snapshot (root catalog + `/workspace.yaml` + `/evidence`), installation descriptor/env, fixture-only secrets, controlled REST DWH, controlled HTTP Evidence, real compatible Qdrant, deterministic Ollama-compatible embedding fixture, selected core image, and no backend/Pi/frontend. - [ ] **Step 3: Pre-provision the exact compatible Qdrant collection** outside the product operation and record that setup as a P4-deferred fixture step. - [ ] **Step 4: Exercise only built `thothctl` product commands** and assert: 1. exact inspect revision/config identity; 2. DWH introspection+LSH, clean-process rerun/resume, physical/LSH artifacts; 3. FK pristine JSON, pause-before-index, candidate export, explicit digest review, resume; 4. pre-curated full-run completion; 5. schema index counts and unchanged rerun; 6. HTTP Evidence dry-run, publish, unchanged rerun, input mutation/new generation/ACTIVE; 7. no-Evidence warning/skip; 8. filesystem `evidence_materialization_required` with no partial output; 9. direct renderer regression and SSH fail-closed result; 10. missing workspace/binding, resume mismatch, different-revision resumable session, concurrent writer, annotation invalid, egress refusal, and semantic incompatibility; 11. no collection creation, no backend listener, no Pi init, no arbitrary mount; 12. exact cleanup preserving all foreign/pre-existing resources. - [ ] **Step 5: Produce bounded `report.json` and `report.md`**, declare hashes for every retained owned artifact, and scan raw Git objects, names, state, reports, logs, configs, candidates, and Qdrant payloads for fixture canaries/signed queries/raw SQL. - [ ] **Step 6: Run runner unit tests, then one clean integration run without retry.** On failure, diagnose/fix/regress and start one new clean run; never loop blindly. - [ ] **Step 7: Commit tooling:** `test: add P2 host preprocessing acceptance`. ### Task 14: Finalize P2 documentation and independent manual walkthrough **Files:** - Update: `docs/testing/p2-p6-manual-verification.md` P2 section only - Update: local/server installation manuals - Update: `docs/contracts/workspace-preprocessing-cli.md` - Optionally create manual lab helper files if concrete setup cannot remain concise - [ ] **Step 1: Document prerequisites and boundaries**: Docker/Compose, active registry snapshot, existing compatible collection, running semantic services for semantic commands, no host language runtimes, no backend/Pi. - [ ] **Step 2: Fill exact P2 commands** for inspect, DWH/resume, FK export/review digest, check/import, index, HTTP dry/run, full run, unchanged rerun, filesystem deferred result, secret scan, and cleanup. - [ ] **Step 3: Explain each observed component/artifact** without exposing config or secret contents. - [ ] **Step 4: Require a new manual root and `VERDICT.md`** with reviewer, UTC time, explicit result for every P2 check, observations, and exactly `P2 manual acceptance: PASS|FAIL`. Automation never writes it. - [ ] **Step 5: Add mechanical docs tests** for all released commands and stable codes. - [ ] **Step 6: Commit:** `docs: add P2 preprocessing operator walkthrough`. ### Task 15: Run affected-layer verification and hand off the hard checkpoint **Files:** - Modify after evidence exists: `PROJECT_STATE.md` - [ ] **Step 1: Run complete affected Go gates:** `cd tools/thothctl && go test ./... && go vet ./...`, plus the release build contract. - [ ] **Step 2: Run complete backend gates:** `cd backend && npx vitest run && npx tsc --noEmit -p . && npm run build`. - [ ] **Step 3: Run focused harness tests listed in the target map, then full `.venv/bin/pytest -q` if feasible.** Any baseline failure must be reported exactly; touched Python files must be Ruff-clean. - [ ] **Step 4: Run all affected Compose/security contracts** from Task 11 and `git diff --check`. - [ ] **Step 5: Run exactly one final clean P2 integration at the final source commit:** ```bash ./scripts/p2-acceptance.sh integration --keep ``` Expected final lines: ```text P2 automated integration: PASS P2 manual acceptance: PENDING ``` - [ ] **Step 6: Verify report hashes, declared artifacts, secret scan, closed listeners, no maintenance container, and exact cleanup/retention.** - [ ] **Step 7: Update and commit only tracked project state** with retained report path and truthful scope: ```bash git add PROJECT_STATE.md git commit -m "docs: record P2 automated acceptance" ``` - [ ] **Step 8: Report separate statuses and STOP:** ```text P2 automated integration: PASS — P2 manual acceptance: PENDING — docs/testing/p2-p6-manual-verification.md#p2 P3 authorization: PENDING — awaiting explicit user decision ``` Do not begin P3, mark manual PASS, or infer implementation approval from plan approval or automated evidence. --- ## Requirement traceability | Requirement/decision | P2 implementation/proof | Deferred truth | |---|---|---| | D2 / P2 / RNF7 | Native `thothctl`, dedicated one-shot service, existing engine | GUI/backend endpoint excluded | | RF1.2 / RNF1 | File-only bindings, operation-specific mounts, redaction/scan | No secret in Git/rendered output | | RF1.3 / RNF5 | Same binding+renderer code and byte-equivalence test | P3 canonical cross-revision identity/migration | | RF1.4 / RF2.2 | REST process goal; direct regression | SSH operational support P10 | | RF2.1 | DWH command, JSON, outer+child resume | — | | RF2.3 | Physical/LSH production then schema-index consumption | — | | RF2.4 / RNF2 | Immutable engine generations, unchanged rerun, ACTIVE preservation | Cross-revision DWH reuse P3 | | RF3.1 | JSON candidates/check, bounded SQL ingress, explicit digest review | Git-canonical review/sync P5 | | RF3.2 / D5 | Runtime-local P2 annotations only | Repository annotations and pinned sync P5 | | RF3.3 | No model-derived FK path added | Existing workflow invariant preserved | | RF4.1 | Existing schema hash/upsert + JSON counts | Collection lifecycle P4 | | RF5.2 | Policy-allowed HTTP dry/run/resume/publish | Filesystem P6; broader S3 policy separately reviewed | | RF5.3 / D9 | Existing per-run behavior only | Long-term GC/retention P9 | | RNF3 | Safe errors, prior ACTIVE preserved, no-Evidence warning | — | | RNF4 | Workspace binding + P2 different-revision session guard | Revision-scoped points/roots P3 | | RF8.5–8.6 / RNF8–9 | Clean process goal + separate walkthrough | Aggregate P2–P6 verification after P6 | | PRD AC2 | Native release binary on local/server installation profiles | Windows Docker is a separate manual claim | | PRD AC8 | HTTP unchanged/mutation cases | Filesystem change/GC P6/P9 | ## Explicit exclusions - No frontend/GUI or backend HTTP preprocessing endpoint. - No host Python, Node, Pi, `tht`, arbitrary shell, arbitrary entrypoint, or arbitrary host mount. - No Git pull/publish/push, active-revision transition, or authoring API in P2. - No P3 canonical effective fingerprint, ownership migration, revision-scoped roots/points, or `.tht-dwh` operator chapter. - No P4 collection creation/index repair/rebuild/maintenance drain. - No P5 Git-canonical FK annotation sync/acceptance. - No P6 filesystem Evidence materialization, realpath/symlink containment, or pinned-tree retention. - No PSD migration/re-embedding (P7), final search/session/L2 gate (P8), policy-driven long-term GC (P9), or SSH runtime (P10). - No changed embedding model/dimensions/distance, external vector service, pgvector compatibility path, or NL→SQL workflow change. ## Execution notes - Every implementation task is RED → minimal GREEN → focused verification → commit. - Never weaken an existing P1 security invariant to simplify P2. - P2's session-inventory guard and deterministic same-revision config path are deliberate temporary safety mechanisms, not substitutes for P3. - Keep automated FK mechanics distinct from human review and from the later P5 Git decision. - A passed plan review authorizes only plan acceptance. Implementation starts only after the user's separate explicit approval.