ThothII/docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md

P2 Host Workspace Preprocessing CLI Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Implement P2/D2 as a native thothctl workspace interface that runs the existing preprocessing engine in a hardened one-shot container, derives its configuration from one active schema-v3 Git workspace revision plus installation-local bindings, and requires no Python, Node, Pi, or running backend HTTP service on the host.

Architecture: thothctl validates a closed command grammar, reconstructs the exact installation Compose project, resolves the selected core image to an immutable Docker image ID, and starts only the profile-gated workspace-maintenance service with --no-deps. A compiled Node entrypoint reads an already-active immutable registry snapshot, uses the same binding resolver and runtime renderer as sessions, writes a deterministic revision-owned protected harness config, and invokes fixed existing tht commands. A versioned coordinator state and one kernel-released writer lock serialize mutation, preserve outer/child resume identity, and stop at a digest-bound FK review checkpoint.

Tech Stack: Go 1.24 (thothctl), Docker Compose v2, Node.js 22, TypeScript 5, Python 3.12, Typer, Pydantic 2, Qdrant 1.18.2, the internal Ollama-compatible embedding interface, Vitest, pytest, Bash/Node acceptance tooling.

Source PRD and design: docs/prd/2026-08-09-workspace-preprocessing-prd.md D2/P2, RF1.2–RF1.4, RF2, RF3.1, RF4.1, RF5.2, RF8.5–RF8.6, RNF1–RNF9; docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md §§1–4, 9–10; docs/testing/p2-p6-manual-verification.md P2.

Planning status: DESIGN/PLAN ONLY. Do not change production code, start P2 implementation, or create a persistent implementation goal until the reviewer asks for plan validation and then gives explicit implementation approval.

---

P2 completion contract

P2 is complete only when all of the following are true:

1. The only public host interface is the installed native thothctl binary. Docker/Compose is required, but host Python, Node, Pi, tht, and a running Fastify backend are not.
2. Every command consumes an already-active, validated P1.1 registry snapshot and binds the exact workspace ID, 40-hex commit, catalog blob (thoth-workspaces.yaml), descriptor blob/digest, installation bindings, runtime roots, and selected internal semantic contract before mutation. A docs-only or content-only commit is still a distinct revision even when the descriptor blob is unchanged, because the commit is authoritative.
3. Operator and session configuration use the same resolveRuntimeBindings and renderRuntimeConfig implementation. P2 uses one deterministic same-revision config-source path so current schema-v1 DWH/Evidence resume works; P3 later introduces cross-revision canonical effective identity and explicit migrations.
4. DWH introspection+LSH, FK suggestion/check, schema indexing, and HTTP Evidence preprocessing invoke the existing harness engine through fixed argv and pristine JSON machine interfaces. No second preprocessing engine is added.
5. A full run with new FK candidates stops before schema/Evidence writes. Continuation requires a reviewer-supplied annotations file and an explicit acknowledgement of the exact candidate digest; schema check alone is not treated as human approval.
6. Mutating P2 operations are safe while roots and semantic point IDs are still workspace-global: under the workspace writer lock they refuse if any resumable session is pinned to a different workspace revision. P3 removes this temporary restriction by introducing revision-scoped curated/semantic state.
7. P2 never creates, repairs, deletes, or rebuilds a Qdrant collection. Schema/Evidence writes require an already-existing, exactly compatible collection and a harness require_existing mode that cannot race into auto-create. P4 owns lifecycle reconciliation.
8. HTTP Evidence is operational only under installation-local egress policy. Private hosts require an exact installation allowlist; redirects are rechecked; metadata/link-local targets are always refused. S3 custom/private/insecure endpoints and ambient credentials remain fail-closed in P2 unless a later separately reviewed plan expands policy.
9. Filesystem Evidence is rendered but execution stops before discovery with evidence_materialization_required and no partial corpus/vector publication. P6 owns materialization and symlink/containment checks.
10. postgres_direct and rest_api routing remain supported and are regression-tested; the clean P2 process goal uses controlled REST. ssh_tunnel returns a stable fail-closed result until P10.
11. One clean-state product-path integration command passes without retry, produces retained machine/human reports and a secret scan, and proves exact cleanup. Manual P2 acceptance remains independent and PENDING.
12. Work stops after the P2 handoff. No P3 work begins without a new explicit user authorization.

Truthful command status at the P2 checkpoint

Command	P2 status	Deliberate boundary
workspace inspect	Operational	Reads active snapshot only; does not pull/activate Git
workspace preprocess dwh	Operational for REST/direct	Same-revision config identity; cross-revision reuse is P3
workspace schema suggest-fks	Operational, machine-safe	Candidate export only; no automatic human acceptance
workspace schema check	Operational	Imports reviewed annotations and records digest-bound local P2 acknowledgement
workspace index-schema	Operational with compatible pre-existing collection	Collection create/repair/rebuild is P4
workspace preprocess evidence	Operational for policy-allowed HTTP; filesystem deferred	Filesystem materialization is P6; S3 expansion needs separate policy review
workspace preprocess run	Operational with FK checkpoint	Git-canonical annotations are P5; revision-global writes use the P2 session-inventory guard

P2 is therefore the host CLI/orchestration checkpoint, not final acceptance of the PSD filesystem path or the complete PRD chain.

Frozen host command grammar

thothctl --installation <absolute>/thothii-installation.yaml workspace inspect
  --workspace <id> [--json]

thothctl ... workspace preprocess dwh
  --workspace <id> [--resume <outer-run-id>] [--json]

thothctl ... workspace schema suggest-fks
  --workspace <id>
  [--from-sql <regular-file>]... [--assume <column=table>]...
  [--output <new-file>] [--json]

thothctl ... workspace schema check
  --workspace <id>
  [--annotations <regular-file> --reviewed-candidates <sha256:hex>]
  [--json]

thothctl ... workspace index-schema
  --workspace <id> [--json]

thothctl ... workspace preprocess evidence
  --workspace <id> [--dry-run] [--resume <outer-run-id>] [--json]

thothctl ... workspace preprocess run
  --workspace <id> [--resume <outer-run-id>] [--json]

Rules:

• --workspace occurs exactly once and matches [a-z][a-z0-9-]{2,62}.
• All run IDs are 32 lowercase hex characters and identify outer P2 state, never a path or raw child checkpoint.
• At most 32 --from-sql files, 1 MiB each and 16 MiB total. thothctl opens each as a canonical regular non-symlink/reparse-point file, rechecks identity after reading, and streams a schema-versioned request over stdin. No host directory is mounted.
• --assume occurs at most 256 times; each value is at most 256 bytes and is validated before Compose.
• --annotations is a single UTF-8 YAML file, at most 16 MiB. --reviewed-candidates is mandatory with it and must equal the persisted candidate artifact digest. The pair is invalid without both flags.
• --output is created exclusively with restrictive permissions after the returned workspace/run/digest identity has been verified. Existing files, symlinks, hardlinks, and Windows reparse targets are refused.
• Existing harness suggest-fks --write is intentionally not exposed: an automatic merge is not a human review decision.
• No unknown flag, passthrough separator, environment-selected command, shell fragment, or arbitrary container entrypoint is accepted.

Public result and exit contract

The one-shot entrypoint always emits exactly one bounded schema-versioned JSON object. thothctl --json parses it strictly and re-encodes it, so Compose progress cannot contaminate stdout. Human mode renders only allowlisted fields.

interface WorkspaceOperationResult {
  schemaVersion: 1;
  status: "succeeded" | "unchanged" | "dry_run" | "blocked" | "failed";
  code:
    | "ok" | "workspace_not_found" | "workspace_not_activatable"
    | "binding_missing" | "preprocessing_conflict"
    | "preprocessing_resume_mismatch" | "manual_review_required"
    | "evidence_materialization_required" | "effective_config_mismatch"
    | "semantic_index_incompatible" | "annotation_invalid"
    | "egress_policy_refused";
  workspaceId: string;
  workspaceRevision: string;
  descriptorBlob: string;
  operation: string;
  runId?: string;
  childRuns?: Record<string, string>;
  completedStages: string[];
  counts?: Record<string, number>;
  artifactIdentities?: Array<{ kind: string; digest: string }>;
  warnings?: string[];
}

• Exit 0: succeeded, unchanged, or dry_run.
• Exit 3: expected operator checkpoint/block (manual_review_required, evidence_materialization_required, lock/revision conflict).
• Exit 2: host grammar or unsafe local file error.
• Exit 1: operational failure.
• Stdout JSON maximum: 1 MiB. Sanitized stderr maximum: 64 KiB. Child stdout/stderr and every stage have explicit limits/timeouts.
• Never return descriptor endpoints with credentials/query strings, secret contents or paths, signed URLs, raw SQL, rendered configuration, raw child stderr, arbitrary exception text, Qdrant payload contents, or host/container environment dumps.

P2 state and identity layout

/data/sessions/<workspace-id>/preprocessing/
├── writer.lock
├── runtime-config/
│   └── <40-hex-revision>.yaml
├── runtime-config-manifests/
│   └── <40-hex-revision>.json
├── jobs/
│   └── <32-hex-outer-run-id>.json
├── fk-candidates/
│   └── <32-hex-outer-run-id>.yaml
└── fk-reviews/
    └── <32-hex-outer-run-id>.json

• writer.lock is a regular 0600 file held by a Linux kernel advisory lock for the entire outer operation. The file may persist; the kernel lock is released on crash/container death. inspect never takes it.
• Lock order is always P2 workspace writer lock → existing harness stage lock. Harness code never acquires the P2 lock, preventing inversion/deadlock.
• Every directory component is opened/validated without following symlinks. State files are 0600, written to an exclusive sibling, fsynced, renamed, and parent-fsynced. Hardlink count must be one.
• The deterministic config path fixes P2 same-revision config_source identity. Its manifest binds workspace, revision, descriptor blob, config SHA-256, file identity, and the current existing harness ownership binding. Same path + different bytes returns effective_config_mismatch; P3 introduces semantic cross-revision equivalence.
• Job state binds operation, revision, catalog blob, descriptor blob, config digest, non-secret binding identity, completed stage records, child run IDs, candidate/review digests, and terminal status. Resume revalidates all fields and reconciles a child publication that completed immediately before an outer-state crash.
• Before any schema/Evidence mutation, enumerate resumable session manifests for the workspace. A different pinned revision returns preprocessing_conflict; no write begins. This is the explicit P2 bridge until P3 revision isolation.

One-shot service security contract

workspace-maintenance is a dedicated profile service, not compose run core:

• same exact selected core image, resolved to its immutable local image ID; generated final override uses that ID and pull_policy: never;
• docker compose run --rm --no-deps --no-TTY --name <owned-name> workspace-maintenance ...;
• no build, frontend, published port, Pi auth, Pi state, Pi trust initialization, Docker socket, home credential directory, or arbitrary command;
• non-root 10001, read_only: true, cap_drop: [ALL], no-new-privileges:true, restrictive tmpfs, registry active snapshots read-only, sessions root writable;
• only operation-required connector/Evidence secret files are mounted; AWS ambient environment is cleared;
• semantic commands require already-running healthy Qdrant/embedding services and do not start/stop them; DWH/inspect commands do not start dependencies;
• exact operation labels and container identity are recorded. Cancellation terminates the process group, verifies the owned container labels/image, removes only that container, and preserves all pre-existing services, volumes, and networks;
• fully rendered Compose is validated before launch, and post-run container/image identity is checked before accepting output.

---

Target file map

Native host CLI

• tools/thothctl/cmd/thothctl/main.go, main_test.go: public grammar/help, dispatch, exit codes.
• Create tools/thothctl/internal/workspaceops/operations.go, operations_test.go: immutable image resolution, generated override, Compose run, cancellation cleanup, JSON validation.
• tools/thothctl/internal/config/installation.go, installation_test.go: maintenance service override and operation-specific binding discovery.
• tools/thothctl/internal/safeio/files.go, platform files/tests: bounded no-follow input and exclusive output.
• tools/thothctl/internal/output/sanitize.go, tests: bounded redaction.
• tools/thothctl/internal/pi/update.go, tests: selected image override must pin both core and workspace-maintenance.

Compose/image boundary

• compose.yaml: dedicated profile-gated service with shared image identity and least privilege.
• deploy/compose.local.yaml, deploy/compose.server.yaml: correct registry/session storage semantics.
• deploy/compose.git-https.yaml, deploy/compose.git-ssh.yaml: do not attach Git credentials to P2 active-snapshot operations.
• scripts/generate-connector-secrets-override.sh: operation-specific maintenance secrets.
• Create docker/workspace-maintenance-entrypoint.sh; modify docker/core.Dockerfile.
• Retire/redirect fixture-only deploy/compose.preprocess.yaml as a non-public compatibility test path; do not leave two operator commands.

Shared Node operator

• Create backend/src/workspaces/runtime-config-lease.ts: shared snapshot read/render and deterministic protected config lease.
• Modify backend/src/tht/tht-runner.ts to delegate session/operator rendering to the shared component without changing route behavior.
• Create backend/src/workspaces/preprocessing-state.ts: state schema, durable writes, locks, resume reconciliation.
• Create backend/src/workspaces/preprocessing-service.ts: closed stage coordinator and security preflights.
• Create backend/src/workspace-maintenance.ts: compiled stdin/argv entrypoint and pristine result encoder.
• Add tests: backend/test/workspace-runtime-config-lease.test.ts, workspace-preprocessing-state.test.ts, workspace-preprocessing-service.test.ts, workspace-maintenance.test.ts.

Harness machine contracts

• harness/tht/cli/preprocess_cmd.py: authoritative runtime workspace identity; require-existing collection mode.
• harness/tht/cli/schema_cmd.py: extracted deterministic helpers, JSON suggest/check, safe SQL staging and annotation validation.
• harness/tht/cli/vector_cmd.py: JSON schema-index result.
• harness/tht/adapters/vector/qdrant.py: explicit non-creating strict mode for P2.
• Tests: harness/tests/test_preprocess_cli.py, test_schema_fk_annotations.py, test_qdrant_cli_commands.py, test_registry_evidence_config.py, test_http_evidence_source.py, plus new focused security cases.

Docs and gates

• Create docs/contracts/workspace-preprocessing-cli.md.
• Update local/server installation manuals and docs/testing/p2-p6-manual-verification.md P2 only.
• Create scripts/p2-acceptance.sh, backend/scripts/p2-acceptance.mjs, backend/scripts/p2-acceptance.test.mjs.
• Create scripts/p2-manual-acceptance.sh, backend/scripts/p2-manual-acceptance.mjs only if needed to generate the isolated walkthrough lab; automation must never create PASS.
• Update PROJECT_STATE.md only after implementation evidence exists.

---

Task 1: Freeze the native CLI, file-ingress, and result contracts

Files:

• Modify: tools/thothctl/cmd/thothctl/main.go

• Modify: tools/thothctl/cmd/thothctl/main_test.go

• Modify: tools/thothctl/internal/safeio/files.go

• Modify platform-specific safe-I/O tests

• Create: tools/thothctl/internal/workspaceops/operations.go

• Create: tools/thothctl/internal/workspaceops/operations_test.go

• Create: docs/contracts/workspace-preprocessing-cli.md

• Step 1: Write RED parser-table tests for every valid command above and for duplicate/missing/unknown flags, invalid IDs, incompatible annotation flags, option-count/size limits, and passthrough/shell attempts.

• Step 2: Run cd tools/thothctl && go test ./cmd/thothctl ./internal/workspaceops -run 'Workspace|workspace' -v and verify the new tests fail because workspace is unknown.

• Step 3: Add closed request types (InspectRequest, DwhRequest, SuggestFksRequest, CheckSchemaRequest, IndexSchemaRequest, EvidenceRequest, RunRequest) and a parser that cannot represent arbitrary argv.

• Step 4: Write RED safe-I/O tests for symlinks, hardlinks, directory input, replacement during read, Windows reparse points, existing output, >1 MiB SQL, >16 MiB total, and non-UTF-8 annotation input.

• Step 5: Implement bounded reads and exclusive restrictive output using existing platform seams; return only generic file errors.

• Step 6: Add schema-v1 stdin request/result validation with exact field allowlists and output bounds.

• Step 7: Run focused Go tests and gofmt -w, then go test ./....

• Step 8: Commit: feat: define P2 host workspace command contract.

Task 2: Add pristine harness JSON interfaces without changing the engine

Files:

• Modify: harness/tht/cli/schema_cmd.py

• Modify: harness/tht/cli/vector_cmd.py

• Modify: harness/tht/cli/preprocess_cmd.py

• Modify: harness/tests/test_schema_fk_annotations.py

• Modify: harness/tests/test_qdrant_cli_commands.py

• Modify: harness/tests/test_preprocess_cli.py

• Step 1: Write RED tests requiring schema suggest-fks --json, schema check --json, and vector index-schema --json to emit exactly one JSON object on stdout for success and failure, with no color/prose contamination.

• Step 2: Write RED deterministic FK tests for bounded staged SQL files, stable candidate ordering, candidate SHA-256, annotation import, orphan counts, and no implicit review/write.

• Step 3: Write RED Evidence identity test showing a config named /dev/fd/3 still uses runtime_identity.workspace_id, never the config basename.

• Step 4: Run:

cd harness
.venv/bin/pytest -q \
  tests/test_schema_fk_annotations.py \
  tests/test_qdrant_cli_commands.py \
  tests/test_preprocess_cli.py

Expected: FAIL only on the new machine-contract assertions.

• Step 5: Extract pure helpers returning typed dictionaries/models; keep existing human commands as renderers over the same helpers.
• Step 6: Implement the JSON flags and authoritative workspace identity. Catch expected exceptions and emit stable safe codes; never serialize arbitrary exception text.
• Step 7: Run the three focused files and touched Ruff:

.venv/bin/ruff check \
  tht/cli/schema_cmd.py tht/cli/vector_cmd.py tht/cli/preprocess_cmd.py \
  tests/test_schema_fk_annotations.py tests/test_qdrant_cli_commands.py tests/test_preprocess_cli.py

• Step 8: Commit: feat: add P2 harness machine contracts.

Task 3: Add a non-creating semantic writer mode

Files:

• Modify: harness/tht/config.py

• Modify: harness/tht/adapters/factory.py

• Modify: harness/tht/adapters/vector/qdrant.py

• Modify: harness/tests/test_qdrant_vector_store.py

• Modify: harness/tests/test_qdrant_cli_commands.py

• Modify: harness/tests/test_registry_evidence_config.py

• Step 1: Write RED tests proving operator mode refuses a missing collection without issuing create/index mutations, refuses wrong dimensions/distance/index type, and still writes to an existing compatible collection.

• Step 2: Run the focused tests and confirm current _ensure_collection(strict=True) incorrectly creates the collection.

• Step 3: Add an internal rendered field such as vectors.collection_lifecycle: require_existing; it is not a descriptor option and defaults to legacy behavior for non-operator configs.

• Step 4: Thread the mode through the factory/store and perform a read-only exact collection/index preflight before any upsert.

• Step 5: Add a race regression: delete the collection after preflight and prove the write fails rather than recreates it.

• Step 6: Run focused pytest and touched Ruff.

• Step 7: Commit: fix: prevent P2 from owning Qdrant lifecycle.

Task 4: Extract the shared runtime configuration lease

Files:

• Create: backend/src/workspaces/runtime-config-lease.ts

• Create: backend/test/workspace-runtime-config-lease.test.ts

• Modify: backend/src/tht/tht-runner.ts

• Modify: backend/test/tht-runner.test.ts

• Modify: backend/test/workspace-runtime-handoff.test.ts

• Step 1: Write RED equivalence tests feeding the same immutable snapshot, env, roots, installation overlay, and semantic contract to the session and operator callers and requiring byte-identical YAML.

• Step 2: Write RED identity/safety tests for snapshot replacement, wrong commit/path, symlink/hardlink, wrong workspace ID, unstable config destination, same-revision changed bytes, mode, fsync/rename failure, and cleanup.

• Step 3: Run:

cd backend
npx vitest run \
  test/workspace-runtime-config-lease.test.ts \
  test/tht-runner.test.ts \
  test/workspace-runtime-handoff.test.ts

• Step 4: Move snapshot validation, runtime roots, installation-overlay parsing, binding resolution, and rendering out of ThtRunner into one explicit-input component.
• Step 5: Preserve session behavior: ThtRunner.acquireWorkspaceRuntime delegates to the component and retains its current opaque FD-backed temporary lease.
• Step 6: Add operator mode: deterministically publish /data/sessions/<id>/preprocessing/runtime-config/<revision>.yaml plus a manifest, mode 0400/0600, and set collection_lifecycle: require_existing.
• Step 7: Prove same-revision rerun path identity and changed config/binding refusal. Do not implement P3 semantic cross-revision canonicalization.
• Step 8: Run focused tests, npx tsc --noEmit -p ., and npm run build.
• Step 9: Commit: refactor: share registry runtime configuration leases.

Task 5: Build durable outer state, locking, and revision guard

Files:

• Create: backend/src/workspaces/preprocessing-state.ts

• Create: backend/test/workspace-preprocessing-state.test.ts

• Modify: docker/core.Dockerfile (install/pin the kernel lock utility only when the implementation proves it is absent)

• Step 1: Write RED state-schema tests for valid state, same-operation resume, cross-workspace/revision/operation/config mismatch, tampering, run-ID traversal, restrictive modes, atomic failure, and bounded fields.

• Step 2: Write RED cross-process lock tests with two processes/containers: one wins, one receives preprocessing_conflict, and SIGKILL releases the kernel lock without deleting unrelated state.

• Step 3: Write RED session-inventory tests: no sessions/current-only sessions permit mutation; a resumable different-revision manifest blocks; finalized/archived sessions follow existing resume policy.

• Step 4: Implement the exact state layout and durable write protocol described above.

• Step 5: Implement lock acquisition ordering and safe conflict mapping. Do not invent stale-PID deletion; the kernel owns lock lifetime.

• Step 6: Implement active-snapshot revalidation immediately before each mutating child stage and the different-revision resumable-session guard.

• Step 7: Add crash reconciliation tests where a child publishes DWH/corpus state but outer state has not yet advanced.

• Step 8: Run focused Vitest, typecheck, and build.

• Step 9: Commit: feat: add P2 preprocessing operation state.

Task 6: Build the compiled inspect/operator boundary

Files:

• Create: backend/src/workspace-maintenance.ts

• Create: backend/src/workspaces/preprocessing-service.ts

• Create: backend/test/workspace-maintenance.test.ts

• Create: backend/test/workspace-preprocessing-service.test.ts

• Modify: backend/src/workspaces/types.ts only if a separate operator-code union cannot stay private

• Step 1: Write RED entrypoint process tests for exact JSON, malformed/extra stdin, unknown command/field, stdout/stderr bounds, timeout, signal, raw exception/stderr redaction, and no Fastify listener.

• Step 2: Write RED inspect tests for absent/corrupt/stale active state, migration-required descriptor, missing bindings, exact commit/blob/config identities, safe capability warnings, and no URL/secret output.

• Step 3: Run focused Vitest and verify no operator exists.

• Step 4: Implement a closed WorkspacePreprocessingService dependency interface: active registry reader, shared config lease, fixed child runner, state store, session inventory, semantic preflight, egress policy.

• Step 5: Implement active-snapshot-only acquisition. P2 does not pull or activate Git; clean installations receive workspace_not_activatable with safe instructions.

• Step 6: Implement bounded child execution with fixed executable/argv, -c after the subcommand, FD-backed config/input, process-group cancellation, per-stage timeout, and strict one-document child JSON parsing.

• Step 7: Implement inspect and result encoding.

• Step 8: Run tests, typecheck, build, and verify dist/workspace-maintenance.js exists.

• Step 9: Commit: feat: add P2 workspace maintenance operator.

Task 7: Implement DWH preprocessing and outer resume

Files:

• Modify: backend/src/workspaces/preprocessing-service.ts

• Modify: backend/test/workspace-preprocessing-service.test.ts

• Modify: harness/tests/test_dwh_preprocess_job.py

• Modify: harness/tests/test_lsh_job_resume.py

• Step 1: Write RED service tests requiring fixed preprocess dwh --steps introspect,lsh --json -c /dev/fd/N, child result validation, outer/child run IDs, completed stages, safe artifact digests, and failure mapping.

• Step 2: Add real harness regressions for deterministic same-revision config-source path, second clean-process rerun, resume after introspection, changed binding/config refusal, and ACTIVE preservation on failure.

• Step 3: Run focused backend and harness tests.

• Step 4: Implement preprocess dwh under the outer writer lock and persist state before/after every child transition.

• Step 5: Reconcile a published child run after an injected outer crash without rerunning or corrupting ACTIVE.

• Step 6: Verify REST and direct rendered routing. SSH returns workspace_not_activatable with a P10 warning.

• Step 7: Run focused gates and commit: feat: run workspace DWH preprocessing from thothctl operator.

Task 8: Implement FK candidate export and digest-bound review

Files:

• Modify: backend/src/workspaces/preprocessing-service.ts

• Modify: backend/src/workspaces/preprocessing-state.ts

• Modify: relevant backend tests

• Modify: tools/thothctl/internal/workspaceops/operations.go

• Modify: Go tests

• Step 1: Write RED end-to-end unit/process tests: new candidates create a bounded artifact and return manual_review_required; schema/Evidence child calls are absent.

• Step 2: Add safe host ingress tests proving SQL and annotations travel only over stdin, are absent from Compose argv/state/logs, and staging files are removed.

• Step 3: Add safe host egress tests for candidate export identity/digest, existing destination refusal, and sanitized JSON mode.

• Step 4: Implement schema suggest-fks with candidate count/digest and optional exclusive output.

• Step 5: Implement schema check in two modes: read-only orphan validation; or reviewed annotation import requiring the exact candidate digest. Persist review digest + annotation digest + workspace/revision.

• Step 6: Require the review record on full-run resume. A mere zero-orphan result without reviewer digest is insufficient.

• Step 7: Preserve the boundary: P2 updates runtime-local annotations only; it never writes Git. Output warns that P5 will supersede this local acknowledgement.

• Step 8: Run Go/backend/harness focused gates and commit: feat: add P2 FK review checkpoint.

Task 9: Implement schema indexing and Evidence policy boundaries

Files:

• Modify: backend/src/workspaces/preprocessing-service.ts

• Modify: backend service tests

• Modify: harness/tests/test_http_evidence_source.py

• Modify: harness/tests/test_registry_evidence_config.py

• Modify: harness/tests/test_semantic_kind_isolation.py

• Step 1: Write RED schema-index tests for compatible pre-existing collection, deterministic JSON counts, idempotent repeat, missing/incompatible refusal, and no collection-create request.

• Step 2: Write RED Evidence tests for no-Evidence warning/skip, HTTP dry-run/run/resume/unchanged/mutation, authoritative workspace identity, ACTIVE preservation, and filesystem early stop before adapter/Qdrant calls.

• Step 3: Write RED egress tests for exact private-host allowlist, DNS re-resolution, redirect to private/link-local/metadata, signed URL query redaction, and refusal of S3 ambient/custom/private/insecure modes.

• Step 4: Implement installation-local egress-policy parsing with exact bounded hostnames and no wildcard. Descriptor flags alone never grant network access.

• Step 5: Implement index-schema and preprocess evidence with semantic preflight and stable result mapping.

• Step 6: Revalidate active revision and session inventory immediately before each write.

• Step 7: Run focused tests/touched Ruff/backend typecheck/build and commit: feat: add guarded P2 semantic preprocessing.

Task 10: Implement the ordered full-run coordinator

Files:

• Modify: backend/src/workspaces/preprocessing-service.ts

• Modify: backend/test/workspace-preprocessing-service.test.ts

• Modify: backend/test/workspace-maintenance.test.ts

• Step 1: Write a RED stage-table test for exact order dwh → fk_suggest → fk_review/check → schema_index → evidence and for no hidden/skipped mutation.

• Step 2: Add scenarios: pre-curated/no-new-candidate completion; new-candidate block; digest-reviewed resume; no-Evidence warning; filesystem deferred block; each child failure; resume mismatch; outer crash reconciliation.

• Step 3: Implement the coordinator as an explicit state machine, not recursive command dispatch.

• Step 4: Persist completion after each verified child artifact and never mark a stage based only on exit code.

• Step 5: Prove unchanged rerun creates no duplicate schema/Evidence points or generation.

• Step 6: Run focused Vitest, typecheck, build, and commit: feat: orchestrate the P2 preprocessing chain.

Task 11: Add the hardened maintenance service and selected-image handoff

Files:

• Modify: compose.yaml

• Modify: deploy/compose.local.yaml

• Modify: deploy/compose.server.yaml

• Modify: connector/Git override files and generators as required

• Create: docker/workspace-maintenance-entrypoint.sh

• Modify: docker/core.Dockerfile

• Modify: tools/thothctl/internal/workspaceops/operations.go

• Modify: tools/thothctl/internal/pi/update.go

• Modify relevant Go/Bash/Compose tests

• Step 1: Write RED Compose contract tests for the exact security contract, profile, mounts, no Pi/no port/no build, local/server storage, and absence of Git credentials on active-snapshot operations.

• Step 2: Write RED image-precedence tests across base/profile/operator overrides/current-image override; core and maintenance must resolve to the same immutable ID.

• Step 3: Write RED lifecycle tests for --no-deps, pre-existing service preservation, interruption cleanup, hostile container name/label collision, tag replacement, and output rejection on post-run image mismatch.

• Step 4: Add the dedicated service and entrypoint; the entrypoint executes only the compiled operator and never calls Pi trust setup.

• Step 5: Generate a per-operation final override that pins immutable image ID, exact secrets, egress policy, and owned labels. Validate docker compose config structurally before run.

• Step 6: Extend Pi update/rollback override generation so future selected images cannot split core and maintenance.

• Step 7: Run:

bash scripts/test-preprocess-compose-config.sh
bash scripts/test-compose-secret-policy.sh
bash scripts/test-default-compose.sh
bash scripts/test-unified-compose.sh
bash scripts/test-no-deployment-coupling.sh
cd tools/thothctl && go test ./...

• Step 8: Build the core image and invoke operator --help through the exact service without starting Pi/backend/dependencies.
• Step 9: Commit: feat: package the P2 maintenance service.

Task 12: Complete host dispatch and supported-platform build contract

Files:

• Modify: tools/thothctl/cmd/thothctl/main.go, tests

• Modify: tools/thothctl/internal/workspaceops/operations.go, tests

• Modify: scripts/build-thothctl.sh

• Modify: scripts/test-thothctl-build-contract.sh

• Update operator docs

• Step 1: Add RED command-to-request-to-Compose tests for all seven public commands, JSON/human output, exit mapping, secret redaction, and exact stdin.

• Step 2: Implement dispatcher integration using only typed requests.

• Step 3: Cross-build the existing release matrix and verify Windows input/output safety compiles. Do not claim Windows Docker behavior without a Windows Docker run.

• Step 4: Run go test ./..., build contract, go vet ./..., and gofmt check.

• Step 5: Commit: feat: expose P2 workspace commands in thothctl.

Task 13: Build the clean-state automated P2 process goal

Files:

• Create: scripts/p2-acceptance.sh
• Create: backend/scripts/p2-acceptance.mjs
• Create: backend/scripts/p2-acceptance.test.mjs
• Update: .gitignore only if the existing .artifacts/ rule is insufficient

The public command is:

./scripts/p2-acceptance.sh integration --keep

• Step 1: Write RED acceptance-runner tests for ownership-first state, unique run/project/container/image names, exact cleanup, --keep, injected failure, signal cleanup, report bounds, and no automatic retry.
• Step 2: Build a clean owned topology under .artifacts/p2-integration/p2-<run-id>/: local bare Git + author clone, active P1.1 snapshot (root catalog + <id>/workspace.yaml + <id>/evidence), installation descriptor/env, fixture-only secrets, controlled REST DWH, controlled HTTP Evidence, real compatible Qdrant, deterministic Ollama-compatible embedding fixture, selected core image, and no backend/Pi/frontend.
• Step 3: Pre-provision the exact compatible Qdrant collection outside the product operation and record that setup as a P4-deferred fixture step.
• Step 4: Exercise only built thothctl product commands and assert:
  1. exact inspect revision/config identity;
  2. DWH introspection+LSH, clean-process rerun/resume, physical/LSH artifacts;
  3. FK pristine JSON, pause-before-index, candidate export, explicit digest review, resume;
  4. pre-curated full-run completion;
  5. schema index counts and unchanged rerun;
  6. HTTP Evidence dry-run, publish, unchanged rerun, input mutation/new generation/ACTIVE;
  7. no-Evidence warning/skip;
  8. filesystem evidence_materialization_required with no partial output;
  9. direct renderer regression and SSH fail-closed result;
  10. missing workspace/binding, resume mismatch, different-revision resumable session, concurrent writer, annotation invalid, egress refusal, and semantic incompatibility;
  11. no collection creation, no backend listener, no Pi init, no arbitrary mount;
  12. exact cleanup preserving all foreign/pre-existing resources.
• Step 5: Produce bounded report.json and report.md, declare hashes for every retained owned artifact, and scan raw Git objects, names, state, reports, logs, configs, candidates, and Qdrant payloads for fixture canaries/signed queries/raw SQL.
• Step 6: Run runner unit tests, then one clean integration run without retry. On failure, diagnose/fix/regress and start one new clean run; never loop blindly.
• Step 7: Commit tooling: test: add P2 host preprocessing acceptance.

Task 14: Finalize P2 documentation and independent manual walkthrough

Files:

• Update: docs/testing/p2-p6-manual-verification.md P2 section only

• Update: local/server installation manuals

• Update: docs/contracts/workspace-preprocessing-cli.md

• Optionally create manual lab helper files if concrete setup cannot remain concise

• Step 1: Document prerequisites and boundaries: Docker/Compose, active registry snapshot, existing compatible collection, running semantic services for semantic commands, no host language runtimes, no backend/Pi.

• Step 2: Fill exact P2 commands for inspect, DWH/resume, FK export/review digest, check/import, index, HTTP dry/run, full run, unchanged rerun, filesystem deferred result, secret scan, and cleanup.

• Step 3: Explain each observed component/artifact without exposing config or secret contents.

• Step 4: Require a new manual root and VERDICT.md with reviewer, UTC time, explicit result for every P2 check, observations, and exactly P2 manual acceptance: PASS|FAIL. Automation never writes it.

• Step 5: Add mechanical docs tests for all released commands and stable codes.

• Step 6: Commit: docs: add P2 preprocessing operator walkthrough.

Task 15: Run affected-layer verification and hand off the hard checkpoint

Files:

• Modify after evidence exists: PROJECT_STATE.md

• Step 1: Run complete affected Go gates: cd tools/thothctl && go test ./... && go vet ./..., plus the release build contract.

• Step 2: Run complete backend gates: cd backend && npx vitest run && npx tsc --noEmit -p . && npm run build.

• Step 3: Run focused harness tests listed in the target map, then full .venv/bin/pytest -q if feasible. Any baseline failure must be reported exactly; touched Python files must be Ruff-clean.

• Step 4: Run all affected Compose/security contracts from Task 11 and git diff --check.

• Step 5: Run exactly one final clean P2 integration at the final source commit:

./scripts/p2-acceptance.sh integration --keep

Expected final lines:

P2 automated integration: PASS
P2 manual acceptance: PENDING

• Step 6: Verify report hashes, declared artifacts, secret scan, closed listeners, no maintenance container, and exact cleanup/retention.
• Step 7: Update and commit only tracked project state with retained report path and truthful scope:

git add PROJECT_STATE.md
git commit -m "docs: record P2 automated acceptance"

• Step 8: Report separate statuses and STOP:

P2 automated integration: PASS — <retained report>
P2 manual acceptance: PENDING — docs/testing/p2-p6-manual-verification.md#p2
P3 authorization: PENDING — awaiting explicit user decision

Do not begin P3, mark manual PASS, or infer implementation approval from plan approval or automated evidence.

---

Requirement traceability

Requirement/decision	P2 implementation/proof	Deferred truth
D2 / P2 / RNF7	Native thothctl, dedicated one-shot service, existing engine	GUI/backend endpoint excluded
RF1.2 / RNF1	File-only bindings, operation-specific mounts, redaction/scan	No secret in Git/rendered output
RF1.3 / RNF5	Same binding+renderer code and byte-equivalence test	P3 canonical cross-revision identity/migration
RF1.4 / RF2.2	REST process goal; direct regression	SSH operational support P10
RF2.1	DWH command, JSON, outer+child resume	—
RF2.3	Physical/LSH production then schema-index consumption	—
RF2.4 / RNF2	Immutable engine generations, unchanged rerun, ACTIVE preservation	Cross-revision DWH reuse P3
RF3.1	JSON candidates/check, bounded SQL ingress, explicit digest review	Git-canonical review/sync P5
RF3.2 / D5	Runtime-local P2 annotations only	Repository annotations and pinned sync P5
RF3.3	No model-derived FK path added	Existing workflow invariant preserved
RF4.1	Existing schema hash/upsert + JSON counts	Collection lifecycle P4
RF5.2	Policy-allowed HTTP dry/run/resume/publish	Filesystem P6; broader S3 policy separately reviewed
RF5.3 / D9	Existing per-run behavior only	Long-term GC/retention P9
RNF3	Safe errors, prior ACTIVE preserved, no-Evidence warning	—
RNF4	Workspace binding + P2 different-revision session guard	Revision-scoped points/roots P3
RF8.5–8.6 / RNF8–9	Clean process goal + separate walkthrough	Aggregate P2–P6 verification after P6
PRD AC2	Native release binary on local/server installation profiles	Windows Docker is a separate manual claim
PRD AC8	HTTP unchanged/mutation cases	Filesystem change/GC P6/P9

Explicit exclusions

• No frontend/GUI or backend HTTP preprocessing endpoint.
• No host Python, Node, Pi, tht, arbitrary shell, arbitrary entrypoint, or arbitrary host mount.
• No Git pull/publish/push, active-revision transition, or authoring API in P2.
• No P3 canonical effective fingerprint, ownership migration, revision-scoped roots/points, or .tht-dwh operator chapter.
• No P4 collection creation/index repair/rebuild/maintenance drain.
• No P5 Git-canonical FK annotation sync/acceptance.
• No P6 filesystem Evidence materialization, realpath/symlink containment, or pinned-tree retention.
• No PSD migration/re-embedding (P7), final search/session/L2 gate (P8), policy-driven long-term GC (P9), or SSH runtime (P10).
• No changed embedding model/dimensions/distance, external vector service, pgvector compatibility path, or NL→SQL workflow change.

Execution notes

• Every implementation task is RED → minimal GREEN → focused verification → commit.
• Never weaken an existing P1 security invariant to simplify P2.
• P2's session-inventory guard and deterministic same-revision config path are deliberate temporary safety mechanisms, not substitutes for P3.
• Keep automated FK mechanics distinct from human review and from the later P5 Git decision.
• A passed plan review authorizes only plan acceptance. Implementation starts only after the user's separate explicit approval.