mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: add P2 host preprocessing plan

Browse Source
This commit is contained in:
marcopan
2026-08-10 19:04:51 +02:00
parent 16ee92c8f9
commit ce90b4410d
1 changed files with 593 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md
+593
View File
@@ -0,0 +1,593 @@
# P2 Host Workspace Preprocessing CLI Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Implement P2/D2 as a native `thothctl workspace` interface that runs the existing preprocessing engine in a hardened one-shot container, derives its configuration from one active schema-v3 Git workspace revision plus installation-local bindings, and requires no Python, Node, Pi, or running backend HTTP service on the host.
**Architecture:** `thothctl` validates a closed command grammar, reconstructs the exact installation Compose project, resolves the selected core image to an immutable Docker image ID, and starts only the profile-gated `workspace-maintenance` service with `--no-deps`. A compiled Node entrypoint reads an already-active immutable registry snapshot, uses the same binding resolver and runtime renderer as sessions, writes a deterministic revision-owned protected harness config, and invokes fixed existing `tht` commands. A versioned coordinator state and one kernel-released writer lock serialize mutation, preserve outer/child resume identity, and stop at a digest-bound FK review checkpoint.
**Tech Stack:** Go 1.24 (`thothctl`), Docker Compose v2, Node.js 22, TypeScript 5, Python 3.12, Typer, Pydantic 2, Qdrant 1.18.2, the internal Ollama-compatible embedding interface, Vitest, pytest, Bash/Node acceptance tooling.
**Source PRD and design:** `docs/prd/2026-08-09-workspace-preprocessing-prd.md` D2/P2, RF1.2–RF1.4, RF2, RF3.1, RF4.1, RF5.2, RF8.5–RF8.6, RNF1–RNF9; `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` §§1–4, 9–10; `docs/testing/p2-p6-manual-verification.md` P2.
**Planning status:** DESIGN/PLAN ONLY. Do not change production code, start P2 implementation, or create a persistent implementation goal until the reviewer asks for plan validation and then gives explicit implementation approval.
---
## P2 completion contract
P2 is complete only when all of the following are true:
1. The only public host interface is the installed native `thothctl` binary. Docker/Compose is required, but host Python, Node, Pi, `tht`, and a running Fastify backend are not.
2. Every command consumes an already-active, validated registry snapshot and binds the exact workspace ID, 40-hex commit, descriptor blob/digest, installation bindings, runtime roots, and selected internal semantic contract before mutation.
3. Operator and session configuration use the same `resolveRuntimeBindings` and `renderRuntimeConfig` implementation. P2 uses one deterministic same-revision config-source path so current schema-v1 DWH/Evidence resume works; P3 later introduces cross-revision canonical effective identity and explicit migrations.
4. DWH introspection+LSH, FK suggestion/check, schema indexing, and HTTP Evidence preprocessing invoke the existing harness engine through fixed argv and pristine JSON machine interfaces. No second preprocessing engine is added.
5. A full run with new FK candidates stops before schema/Evidence writes. Continuation requires a reviewer-supplied annotations file and an explicit acknowledgement of the exact candidate digest; `schema check` alone is not treated as human approval.
6. Mutating P2 operations are safe while roots and semantic point IDs are still workspace-global: under the workspace writer lock they refuse if any resumable session is pinned to a different workspace revision. P3 removes this temporary restriction by introducing revision-scoped curated/semantic state.
7. P2 never creates, repairs, deletes, or rebuilds a Qdrant collection. Schema/Evidence writes require an already-existing, exactly compatible collection and a harness `require_existing` mode that cannot race into auto-create. P4 owns lifecycle reconciliation.
8. HTTP Evidence is operational only under installation-local egress policy. Private hosts require an exact installation allowlist; redirects are rechecked; metadata/link-local targets are always refused. S3 custom/private/insecure endpoints and ambient credentials remain fail-closed in P2 unless a later separately reviewed plan expands policy.
9. Filesystem Evidence is rendered but execution stops before discovery with `evidence_materialization_required` and no partial corpus/vector publication. P6 owns materialization and symlink/containment checks.
10. `postgres_direct` and `rest_api` routing remain supported and are regression-tested; the clean P2 process goal uses controlled REST. `ssh_tunnel` returns a stable fail-closed result until P10.
11. One clean-state product-path integration command passes without retry, produces retained machine/human reports and a secret scan, and proves exact cleanup. Manual P2 acceptance remains independent and PENDING.
12. Work stops after the P2 handoff. No P3 work begins without a new explicit user authorization.
## Truthful command status at the P2 checkpoint
| Command | P2 status | Deliberate boundary |
|---|---|---|
| `workspace inspect` | Operational | Reads active snapshot only; does not pull/activate Git |
| `workspace preprocess dwh` | Operational for REST/direct | Same-revision config identity; cross-revision reuse is P3 |
| `workspace schema suggest-fks` | Operational, machine-safe | Candidate export only; no automatic human acceptance |
| `workspace schema check` | Operational | Imports reviewed annotations and records digest-bound local P2 acknowledgement |
| `workspace index-schema` | Operational with compatible pre-existing collection | Collection create/repair/rebuild is P4 |
| `workspace preprocess evidence` | Operational for policy-allowed HTTP; filesystem deferred | Filesystem materialization is P6; S3 expansion needs separate policy review |
| `workspace preprocess run` | Operational with FK checkpoint | Git-canonical annotations are P5; revision-global writes use the P2 session-inventory guard |
P2 is therefore the host CLI/orchestration checkpoint, not final acceptance of the PSD filesystem path or the complete PRD chain.
## Frozen host command grammar
```text
thothctl --installation <absolute>/thothii-installation.yaml workspace inspect
--workspace <id> [--json]
thothctl ... workspace preprocess dwh
--workspace <id> [--resume <outer-run-id>] [--json]
thothctl ... workspace schema suggest-fks
--workspace <id>
[--from-sql <regular-file>]... [--assume <column=table>]...
[--output <new-file>] [--json]
thothctl ... workspace schema check
--workspace <id>
[--annotations <regular-file> --reviewed-candidates <sha256:hex>]
[--json]
thothctl ... workspace index-schema
--workspace <id> [--json]
thothctl ... workspace preprocess evidence
--workspace <id> [--dry-run] [--resume <outer-run-id>] [--json]
thothctl ... workspace preprocess run
--workspace <id> [--resume <outer-run-id>] [--json]
```
Rules:
- `--workspace` occurs exactly once and matches `[a-z][a-z0-9-]{2,62}`.
- All run IDs are 32 lowercase hex characters and identify outer P2 state, never a path or raw child checkpoint.
- At most 32 `--from-sql` files, 1 MiB each and 16 MiB total. `thothctl` opens each as a canonical regular non-symlink/reparse-point file, rechecks identity after reading, and streams a schema-versioned request over stdin. No host directory is mounted.
- `--assume` occurs at most 256 times; each value is at most 256 bytes and is validated before Compose.
- `--annotations` is a single UTF-8 YAML file, at most 16 MiB. `--reviewed-candidates` is mandatory with it and must equal the persisted candidate artifact digest. The pair is invalid without both flags.
- `--output` is created exclusively with restrictive permissions after the returned workspace/run/digest identity has been verified. Existing files, symlinks, hardlinks, and Windows reparse targets are refused.
- Existing harness `suggest-fks --write` is intentionally not exposed: an automatic merge is not a human review decision.
- No unknown flag, passthrough separator, environment-selected command, shell fragment, or arbitrary container entrypoint is accepted.
## Public result and exit contract
The one-shot entrypoint always emits exactly one bounded schema-versioned JSON object. `thothctl --json` parses it strictly and re-encodes it, so Compose progress cannot contaminate stdout. Human mode renders only allowlisted fields.
```ts
interface WorkspaceOperationResult {
schemaVersion: 1;
status: "succeeded" | "unchanged" | "dry_run" | "blocked" | "failed";
code:
| "ok" | "workspace_not_found" | "workspace_not_activatable"
| "binding_missing" | "preprocessing_conflict"
| "preprocessing_resume_mismatch" | "manual_review_required"
| "evidence_materialization_required" | "effective_config_mismatch"
| "semantic_index_incompatible" | "annotation_invalid"
| "egress_policy_refused";
workspaceId: string;
workspaceRevision: string;
descriptorBlob: string;
operation: string;
runId?: string;
childRuns?: Record<string, string>;
completedStages: string[];
counts?: Record<string, number>;
artifactIdentities?: Array<{ kind: string; digest: string }>;
warnings?: string[];
}
```
- Exit `0`: `succeeded`, `unchanged`, or `dry_run`.
- Exit `3`: expected operator checkpoint/block (`manual_review_required`, `evidence_materialization_required`, lock/revision conflict).
- Exit `2`: host grammar or unsafe local file error.
- Exit `1`: operational failure.
- Stdout JSON maximum: 1 MiB. Sanitized stderr maximum: 64 KiB. Child stdout/stderr and every stage have explicit limits/timeouts.
- Never return descriptor endpoints with credentials/query strings, secret contents or paths, signed URLs, raw SQL, rendered configuration, raw child stderr, arbitrary exception text, Qdrant payload contents, or host/container environment dumps.
## P2 state and identity layout
```text
/data/sessions/<workspace-id>/preprocessing/
├── writer.lock
├── runtime-config/
│ └── <40-hex-revision>.yaml
├── runtime-config-manifests/
│ └── <40-hex-revision>.json
├── jobs/
│ └── <32-hex-outer-run-id>.json
├── fk-candidates/
│ └── <32-hex-outer-run-id>.yaml
└── fk-reviews/
└── <32-hex-outer-run-id>.json
```
- `writer.lock` is a regular `0600` file held by a Linux kernel advisory lock for the entire outer operation. The file may persist; the kernel lock is released on crash/container death. `inspect` never takes it.
- Lock order is always P2 workspace writer lock → existing harness stage lock. Harness code never acquires the P2 lock, preventing inversion/deadlock.
- Every directory component is opened/validated without following symlinks. State files are `0600`, written to an exclusive sibling, fsynced, renamed, and parent-fsynced. Hardlink count must be one.
- The deterministic config path fixes P2 same-revision `config_source` identity. Its manifest binds workspace, revision, descriptor blob, config SHA-256, file identity, and the current existing harness ownership binding. Same path + different bytes returns `effective_config_mismatch`; P3 introduces semantic cross-revision equivalence.
- Job state binds operation, revision, descriptor blob, config digest, non-secret binding identity, completed stage records, child run IDs, candidate/review digests, and terminal status. Resume revalidates all fields and reconciles a child publication that completed immediately before an outer-state crash.
- Before any schema/Evidence mutation, enumerate resumable session manifests for the workspace. A different pinned revision returns `preprocessing_conflict`; no write begins. This is the explicit P2 bridge until P3 revision isolation.
## One-shot service security contract
`workspace-maintenance` is a dedicated profile service, not `compose run core`:
- same exact selected core image, resolved to its immutable local image ID; generated final override uses that ID and `pull_policy: never`;
- `docker compose run --rm --no-deps --no-TTY --name <owned-name> workspace-maintenance ...`;
- no `build`, frontend, published port, Pi auth, Pi state, Pi trust initialization, Docker socket, home credential directory, or arbitrary command;
- non-root `10001`, `read_only: true`, `cap_drop: [ALL]`, `no-new-privileges:true`, restrictive tmpfs, registry active snapshots read-only, sessions root writable;
- only operation-required connector/Evidence secret files are mounted; AWS ambient environment is cleared;
- semantic commands require already-running healthy Qdrant/embedding services and do not start/stop them; DWH/inspect commands do not start dependencies;
- exact operation labels and container identity are recorded. Cancellation terminates the process group, verifies the owned container labels/image, removes only that container, and preserves all pre-existing services, volumes, and networks;
- fully rendered Compose is validated before launch, and post-run container/image identity is checked before accepting output.
---
## Target file map
**Native host CLI**
- `tools/thothctl/cmd/thothctl/main.go`, `main_test.go`: public grammar/help, dispatch, exit codes.
- Create `tools/thothctl/internal/workspaceops/operations.go`, `operations_test.go`: immutable image resolution, generated override, Compose run, cancellation cleanup, JSON validation.
- `tools/thothctl/internal/config/installation.go`, `installation_test.go`: maintenance service override and operation-specific binding discovery.
- `tools/thothctl/internal/safeio/files.go`, platform files/tests: bounded no-follow input and exclusive output.
- `tools/thothctl/internal/output/sanitize.go`, tests: bounded redaction.
- `tools/thothctl/internal/pi/update.go`, tests: selected image override must pin both `core` and `workspace-maintenance`.
**Compose/image boundary**
- `compose.yaml`: dedicated profile-gated service with shared image identity and least privilege.
- `deploy/compose.local.yaml`, `deploy/compose.server.yaml`: correct registry/session storage semantics.
- `deploy/compose.git-https.yaml`, `deploy/compose.git-ssh.yaml`: do not attach Git credentials to P2 active-snapshot operations.
- `scripts/generate-connector-secrets-override.sh`: operation-specific maintenance secrets.
- Create `docker/workspace-maintenance-entrypoint.sh`; modify `docker/core.Dockerfile`.
- Retire/redirect fixture-only `deploy/compose.preprocess.yaml` as a non-public compatibility test path; do not leave two operator commands.
**Shared Node operator**
- Create `backend/src/workspaces/runtime-config-lease.ts`: shared snapshot read/render and deterministic protected config lease.
- Modify `backend/src/tht/tht-runner.ts` to delegate session/operator rendering to the shared component without changing route behavior.
- Create `backend/src/workspaces/preprocessing-state.ts`: state schema, durable writes, locks, resume reconciliation.
- Create `backend/src/workspaces/preprocessing-service.ts`: closed stage coordinator and security preflights.
- Create `backend/src/workspace-maintenance.ts`: compiled stdin/argv entrypoint and pristine result encoder.
- Add tests: `backend/test/workspace-runtime-config-lease.test.ts`, `workspace-preprocessing-state.test.ts`, `workspace-preprocessing-service.test.ts`, `workspace-maintenance.test.ts`.
**Harness machine contracts**
- `harness/tht/cli/preprocess_cmd.py`: authoritative runtime workspace identity; require-existing collection mode.
- `harness/tht/cli/schema_cmd.py`: extracted deterministic helpers, JSON suggest/check, safe SQL staging and annotation validation.
- `harness/tht/cli/vector_cmd.py`: JSON schema-index result.
- `harness/tht/adapters/vector/qdrant.py`: explicit non-creating strict mode for P2.
- Tests: `harness/tests/test_preprocess_cli.py`, `test_schema_fk_annotations.py`, `test_qdrant_cli_commands.py`, `test_registry_evidence_config.py`, `test_http_evidence_source.py`, plus new focused security cases.
**Docs and gates**
- Create `docs/contracts/workspace-preprocessing-cli.md`.
- Update local/server installation manuals and `docs/testing/p2-p6-manual-verification.md` P2 only.
- Create `scripts/p2-acceptance.sh`, `backend/scripts/p2-acceptance.mjs`, `backend/scripts/p2-acceptance.test.mjs`.
- Create `scripts/p2-manual-acceptance.sh`, `backend/scripts/p2-manual-acceptance.mjs` only if needed to generate the isolated walkthrough lab; automation must never create PASS.
- Update `PROJECT_STATE.md` only after implementation evidence exists.
---
### Task 1: Freeze the native CLI, file-ingress, and result contracts
**Files:**
- Modify: `tools/thothctl/cmd/thothctl/main.go`
- Modify: `tools/thothctl/cmd/thothctl/main_test.go`
- Modify: `tools/thothctl/internal/safeio/files.go`
- Modify platform-specific safe-I/O tests
- Create: `tools/thothctl/internal/workspaceops/operations.go`
- Create: `tools/thothctl/internal/workspaceops/operations_test.go`
- Create: `docs/contracts/workspace-preprocessing-cli.md`
- [ ] **Step 1: Write RED parser-table tests** for every valid command above and for duplicate/missing/unknown flags, invalid IDs, incompatible annotation flags, option-count/size limits, and passthrough/shell attempts.
- [ ] **Step 2: Run** `cd tools/thothctl && go test ./cmd/thothctl ./internal/workspaceops -run 'Workspace|workspace' -v` and verify the new tests fail because `workspace` is unknown.
- [ ] **Step 3: Add closed request types** (`InspectRequest`, `DwhRequest`, `SuggestFksRequest`, `CheckSchemaRequest`, `IndexSchemaRequest`, `EvidenceRequest`, `RunRequest`) and a parser that cannot represent arbitrary argv.
- [ ] **Step 4: Write RED safe-I/O tests** for symlinks, hardlinks, directory input, replacement during read, Windows reparse points, existing output, >1 MiB SQL, >16 MiB total, and non-UTF-8 annotation input.
- [ ] **Step 5: Implement bounded reads and exclusive restrictive output** using existing platform seams; return only generic file errors.
- [ ] **Step 6: Add schema-v1 stdin request/result validation** with exact field allowlists and output bounds.
- [ ] **Step 7: Run focused Go tests and `gofmt -w`**, then `go test ./...`.
- [ ] **Step 8: Commit:** `feat: define P2 host workspace command contract`.
### Task 2: Add pristine harness JSON interfaces without changing the engine
**Files:**
- Modify: `harness/tht/cli/schema_cmd.py`
- Modify: `harness/tht/cli/vector_cmd.py`
- Modify: `harness/tht/cli/preprocess_cmd.py`
- Modify: `harness/tests/test_schema_fk_annotations.py`
- Modify: `harness/tests/test_qdrant_cli_commands.py`
- Modify: `harness/tests/test_preprocess_cli.py`
- [ ] **Step 1: Write RED tests** requiring `schema suggest-fks --json`, `schema check --json`, and `vector index-schema --json` to emit exactly one JSON object on stdout for success and failure, with no color/prose contamination.
- [ ] **Step 2: Write RED deterministic FK tests** for bounded staged SQL files, stable candidate ordering, candidate SHA-256, annotation import, orphan counts, and no implicit review/write.
- [ ] **Step 3: Write RED Evidence identity test** showing a config named `/dev/fd/3` still uses `runtime_identity.workspace_id`, never the config basename.
- [ ] **Step 4: Run:**
```bash
cd harness
.venv/bin/pytest -q \
tests/test_schema_fk_annotations.py \
tests/test_qdrant_cli_commands.py \
tests/test_preprocess_cli.py
```
Expected: FAIL only on the new machine-contract assertions.
- [ ] **Step 5: Extract pure helpers** returning typed dictionaries/models; keep existing human commands as renderers over the same helpers.
- [ ] **Step 6: Implement the JSON flags and authoritative workspace identity**. Catch expected exceptions and emit stable safe codes; never serialize arbitrary exception text.
- [ ] **Step 7: Run the three focused files and touched Ruff**:
```bash
.venv/bin/ruff check \
tht/cli/schema_cmd.py tht/cli/vector_cmd.py tht/cli/preprocess_cmd.py \
tests/test_schema_fk_annotations.py tests/test_qdrant_cli_commands.py tests/test_preprocess_cli.py
```
- [ ] **Step 8: Commit:** `feat: add P2 harness machine contracts`.
### Task 3: Add a non-creating semantic writer mode
**Files:**
- Modify: `harness/tht/config.py`
- Modify: `harness/tht/adapters/factory.py`
- Modify: `harness/tht/adapters/vector/qdrant.py`
- Modify: `harness/tests/test_qdrant_vector_store.py`
- Modify: `harness/tests/test_qdrant_cli_commands.py`
- Modify: `harness/tests/test_registry_evidence_config.py`
- [ ] **Step 1: Write RED tests** proving operator mode refuses a missing collection without issuing create/index mutations, refuses wrong dimensions/distance/index type, and still writes to an existing compatible collection.
- [ ] **Step 2: Run the focused tests** and confirm current `_ensure_collection(strict=True)` incorrectly creates the collection.
- [ ] **Step 3: Add an internal rendered field** such as `vectors.collection_lifecycle: require_existing`; it is not a descriptor option and defaults to legacy behavior for non-operator configs.
- [ ] **Step 4: Thread the mode through the factory/store** and perform a read-only exact collection/index preflight before any upsert.
- [ ] **Step 5: Add a race regression**: delete the collection after preflight and prove the write fails rather than recreates it.
- [ ] **Step 6: Run focused pytest and touched Ruff.**
- [ ] **Step 7: Commit:** `fix: prevent P2 from owning Qdrant lifecycle`.
### Task 4: Extract the shared runtime configuration lease
**Files:**
- Create: `backend/src/workspaces/runtime-config-lease.ts`
- Create: `backend/test/workspace-runtime-config-lease.test.ts`
- Modify: `backend/src/tht/tht-runner.ts`
- Modify: `backend/test/tht-runner.test.ts`
- Modify: `backend/test/workspace-runtime-handoff.test.ts`
- [ ] **Step 1: Write RED equivalence tests** feeding the same immutable snapshot, env, roots, installation overlay, and semantic contract to the session and operator callers and requiring byte-identical YAML.
- [ ] **Step 2: Write RED identity/safety tests** for snapshot replacement, wrong commit/path, symlink/hardlink, wrong workspace ID, unstable config destination, same-revision changed bytes, mode, fsync/rename failure, and cleanup.
- [ ] **Step 3: Run:**
```bash
cd backend
npx vitest run \
test/workspace-runtime-config-lease.test.ts \
test/tht-runner.test.ts \
test/workspace-runtime-handoff.test.ts
```
- [ ] **Step 4: Move snapshot validation, runtime roots, installation-overlay parsing, binding resolution, and rendering** out of `ThtRunner` into one explicit-input component.
- [ ] **Step 5: Preserve session behavior**: `ThtRunner.acquireWorkspaceRuntime` delegates to the component and retains its current opaque FD-backed temporary lease.
- [ ] **Step 6: Add operator mode**: deterministically publish `/data/sessions/<id>/preprocessing/runtime-config/<revision>.yaml` plus a manifest, mode `0400/0600`, and set `collection_lifecycle: require_existing`.
- [ ] **Step 7: Prove same-revision rerun path identity** and changed config/binding refusal. Do not implement P3 semantic cross-revision canonicalization.
- [ ] **Step 8: Run focused tests, `npx tsc --noEmit -p .`, and `npm run build`.**
- [ ] **Step 9: Commit:** `refactor: share registry runtime configuration leases`.
### Task 5: Build durable outer state, locking, and revision guard
**Files:**
- Create: `backend/src/workspaces/preprocessing-state.ts`
- Create: `backend/test/workspace-preprocessing-state.test.ts`
- Modify: `docker/core.Dockerfile` (install/pin the kernel lock utility only when the implementation proves it is absent)
- [ ] **Step 1: Write RED state-schema tests** for valid state, same-operation resume, cross-workspace/revision/operation/config mismatch, tampering, run-ID traversal, restrictive modes, atomic failure, and bounded fields.
- [ ] **Step 2: Write RED cross-process lock tests** with two processes/containers: one wins, one receives `preprocessing_conflict`, and SIGKILL releases the kernel lock without deleting unrelated state.
- [ ] **Step 3: Write RED session-inventory tests**: no sessions/current-only sessions permit mutation; a resumable different-revision manifest blocks; finalized/archived sessions follow existing resume policy.
- [ ] **Step 4: Implement the exact state layout and durable write protocol** described above.
- [ ] **Step 5: Implement lock acquisition ordering and safe conflict mapping.** Do not invent stale-PID deletion; the kernel owns lock lifetime.
- [ ] **Step 6: Implement active-snapshot revalidation immediately before each mutating child stage** and the different-revision resumable-session guard.
- [ ] **Step 7: Add crash reconciliation tests** where a child publishes DWH/corpus state but outer state has not yet advanced.
- [ ] **Step 8: Run focused Vitest, typecheck, and build.**
- [ ] **Step 9: Commit:** `feat: add P2 preprocessing operation state`.
### Task 6: Build the compiled inspect/operator boundary
**Files:**
- Create: `backend/src/workspace-maintenance.ts`
- Create: `backend/src/workspaces/preprocessing-service.ts`
- Create: `backend/test/workspace-maintenance.test.ts`
- Create: `backend/test/workspace-preprocessing-service.test.ts`
- Modify: `backend/src/workspaces/types.ts` only if a separate operator-code union cannot stay private
- [ ] **Step 1: Write RED entrypoint process tests** for exact JSON, malformed/extra stdin, unknown command/field, stdout/stderr bounds, timeout, signal, raw exception/stderr redaction, and no Fastify listener.
- [ ] **Step 2: Write RED `inspect` tests** for absent/corrupt/stale active state, migration-required descriptor, missing bindings, exact commit/blob/config identities, safe capability warnings, and no URL/secret output.
- [ ] **Step 3: Run focused Vitest** and verify no operator exists.
- [ ] **Step 4: Implement a closed `WorkspacePreprocessingService` dependency interface**: active registry reader, shared config lease, fixed child runner, state store, session inventory, semantic preflight, egress policy.
- [ ] **Step 5: Implement active-snapshot-only acquisition.** P2 does not pull or activate Git; clean installations receive `workspace_not_activatable` with safe instructions.
- [ ] **Step 6: Implement bounded child execution** with fixed executable/argv, `-c` after the subcommand, FD-backed config/input, process-group cancellation, per-stage timeout, and strict one-document child JSON parsing.
- [ ] **Step 7: Implement `inspect` and result encoding.**
- [ ] **Step 8: Run tests, typecheck, build, and verify `dist/workspace-maintenance.js` exists.**
- [ ] **Step 9: Commit:** `feat: add P2 workspace maintenance operator`.
### Task 7: Implement DWH preprocessing and outer resume
**Files:**
- Modify: `backend/src/workspaces/preprocessing-service.ts`
- Modify: `backend/test/workspace-preprocessing-service.test.ts`
- Modify: `harness/tests/test_dwh_preprocess_job.py`
- Modify: `harness/tests/test_lsh_job_resume.py`
- [ ] **Step 1: Write RED service tests** requiring fixed `preprocess dwh --steps introspect,lsh --json -c /dev/fd/N`, child result validation, outer/child run IDs, completed stages, safe artifact digests, and failure mapping.
- [ ] **Step 2: Add real harness regressions** for deterministic same-revision config-source path, second clean-process rerun, resume after introspection, changed binding/config refusal, and ACTIVE preservation on failure.
- [ ] **Step 3: Run focused backend and harness tests.**
- [ ] **Step 4: Implement `preprocess dwh`** under the outer writer lock and persist state before/after every child transition.
- [ ] **Step 5: Reconcile a published child run after an injected outer crash** without rerunning or corrupting ACTIVE.
- [ ] **Step 6: Verify REST and direct rendered routing.** SSH returns `workspace_not_activatable` with a P10 warning.
- [ ] **Step 7: Run focused gates and commit:** `feat: run workspace DWH preprocessing from thothctl operator`.
### Task 8: Implement FK candidate export and digest-bound review
**Files:**
- Modify: `backend/src/workspaces/preprocessing-service.ts`
- Modify: `backend/src/workspaces/preprocessing-state.ts`
- Modify: relevant backend tests
- Modify: `tools/thothctl/internal/workspaceops/operations.go`
- Modify: Go tests
- [ ] **Step 1: Write RED end-to-end unit/process tests**: new candidates create a bounded artifact and return `manual_review_required`; schema/Evidence child calls are absent.
- [ ] **Step 2: Add safe host ingress tests** proving SQL and annotations travel only over stdin, are absent from Compose argv/state/logs, and staging files are removed.
- [ ] **Step 3: Add safe host egress tests** for candidate export identity/digest, existing destination refusal, and sanitized JSON mode.
- [ ] **Step 4: Implement `schema suggest-fks`** with candidate count/digest and optional exclusive output.
- [ ] **Step 5: Implement `schema check`** in two modes: read-only orphan validation; or reviewed annotation import requiring the exact candidate digest. Persist review digest + annotation digest + workspace/revision.
- [ ] **Step 6: Require the review record on full-run resume.** A mere zero-orphan result without reviewer digest is insufficient.
- [ ] **Step 7: Preserve the boundary:** P2 updates runtime-local annotations only; it never writes Git. Output warns that P5 will supersede this local acknowledgement.
- [ ] **Step 8: Run Go/backend/harness focused gates and commit:** `feat: add P2 FK review checkpoint`.
### Task 9: Implement schema indexing and Evidence policy boundaries
**Files:**
- Modify: `backend/src/workspaces/preprocessing-service.ts`
- Modify: backend service tests
- Modify: `harness/tests/test_http_evidence_source.py`
- Modify: `harness/tests/test_registry_evidence_config.py`
- Modify: `harness/tests/test_semantic_kind_isolation.py`
- [ ] **Step 1: Write RED schema-index tests** for compatible pre-existing collection, deterministic JSON counts, idempotent repeat, missing/incompatible refusal, and no collection-create request.
- [ ] **Step 2: Write RED Evidence tests** for no-Evidence warning/skip, HTTP dry-run/run/resume/unchanged/mutation, authoritative workspace identity, ACTIVE preservation, and filesystem early stop before adapter/Qdrant calls.
- [ ] **Step 3: Write RED egress tests** for exact private-host allowlist, DNS re-resolution, redirect to private/link-local/metadata, signed URL query redaction, and refusal of S3 ambient/custom/private/insecure modes.
- [ ] **Step 4: Implement installation-local egress-policy parsing** with exact bounded hostnames and no wildcard. Descriptor flags alone never grant network access.
- [ ] **Step 5: Implement `index-schema` and `preprocess evidence`** with semantic preflight and stable result mapping.
- [ ] **Step 6: Revalidate active revision and session inventory immediately before each write.**
- [ ] **Step 7: Run focused tests/touched Ruff/backend typecheck/build and commit:** `feat: add guarded P2 semantic preprocessing`.
### Task 10: Implement the ordered full-run coordinator
**Files:**
- Modify: `backend/src/workspaces/preprocessing-service.ts`
- Modify: `backend/test/workspace-preprocessing-service.test.ts`
- Modify: `backend/test/workspace-maintenance.test.ts`
- [ ] **Step 1: Write a RED stage-table test** for exact order `dwh → fk_suggest → fk_review/check → schema_index → evidence` and for no hidden/skipped mutation.
- [ ] **Step 2: Add scenarios**: pre-curated/no-new-candidate completion; new-candidate block; digest-reviewed resume; no-Evidence warning; filesystem deferred block; each child failure; resume mismatch; outer crash reconciliation.
- [ ] **Step 3: Implement the coordinator as an explicit state machine**, not recursive command dispatch.
- [ ] **Step 4: Persist completion after each verified child artifact** and never mark a stage based only on exit code.
- [ ] **Step 5: Prove unchanged rerun creates no duplicate schema/Evidence points or generation.**
- [ ] **Step 6: Run focused Vitest, typecheck, build, and commit:** `feat: orchestrate the P2 preprocessing chain`.
### Task 11: Add the hardened maintenance service and selected-image handoff
**Files:**
- Modify: `compose.yaml`
- Modify: `deploy/compose.local.yaml`
- Modify: `deploy/compose.server.yaml`
- Modify: connector/Git override files and generators as required
- Create: `docker/workspace-maintenance-entrypoint.sh`
- Modify: `docker/core.Dockerfile`
- Modify: `tools/thothctl/internal/workspaceops/operations.go`
- Modify: `tools/thothctl/internal/pi/update.go`
- Modify relevant Go/Bash/Compose tests
- [ ] **Step 1: Write RED Compose contract tests** for the exact security contract, profile, mounts, no Pi/no port/no build, local/server storage, and absence of Git credentials on active-snapshot operations.
- [ ] **Step 2: Write RED image-precedence tests** across base/profile/operator overrides/current-image override; `core` and maintenance must resolve to the same immutable ID.
- [ ] **Step 3: Write RED lifecycle tests** for `--no-deps`, pre-existing service preservation, interruption cleanup, hostile container name/label collision, tag replacement, and output rejection on post-run image mismatch.
- [ ] **Step 4: Add the dedicated service and entrypoint**; the entrypoint executes only the compiled operator and never calls Pi trust setup.
- [ ] **Step 5: Generate a per-operation final override** that pins immutable image ID, exact secrets, egress policy, and owned labels. Validate `docker compose config` structurally before run.
- [ ] **Step 6: Extend Pi update/rollback override generation** so future selected images cannot split core and maintenance.
- [ ] **Step 7: Run:**
```bash
bash scripts/test-preprocess-compose-config.sh
bash scripts/test-compose-secret-policy.sh
bash scripts/test-default-compose.sh
bash scripts/test-unified-compose.sh
bash scripts/test-no-deployment-coupling.sh
cd tools/thothctl && go test ./...
```
- [ ] **Step 8: Build the core image and invoke operator `--help` through the exact service** without starting Pi/backend/dependencies.
- [ ] **Step 9: Commit:** `feat: package the P2 maintenance service`.
### Task 12: Complete host dispatch and supported-platform build contract
**Files:**
- Modify: `tools/thothctl/cmd/thothctl/main.go`, tests
- Modify: `tools/thothctl/internal/workspaceops/operations.go`, tests
- Modify: `scripts/build-thothctl.sh`
- Modify: `scripts/test-thothctl-build-contract.sh`
- Update operator docs
- [ ] **Step 1: Add RED command-to-request-to-Compose tests** for all seven public commands, JSON/human output, exit mapping, secret redaction, and exact stdin.
- [ ] **Step 2: Implement dispatcher integration** using only typed requests.
- [ ] **Step 3: Cross-build the existing release matrix** and verify Windows input/output safety compiles. Do not claim Windows Docker behavior without a Windows Docker run.
- [ ] **Step 4: Run `go test ./...`, build contract, `go vet ./...`, and `gofmt` check.**
- [ ] **Step 5: Commit:** `feat: expose P2 workspace commands in thothctl`.
### Task 13: Build the clean-state automated P2 process goal
**Files:**
- Create: `scripts/p2-acceptance.sh`
- Create: `backend/scripts/p2-acceptance.mjs`
- Create: `backend/scripts/p2-acceptance.test.mjs`
- Update: `.gitignore` only if the existing `.artifacts/` rule is insufficient
The public command is:
```bash
./scripts/p2-acceptance.sh integration --keep
```
- [ ] **Step 1: Write RED acceptance-runner tests** for ownership-first state, unique run/project/container/image names, exact cleanup, `--keep`, injected failure, signal cleanup, report bounds, and no automatic retry.
- [ ] **Step 2: Build a clean owned topology** under `.artifacts/p2-integration/p2-<run-id>/`: local bare Git + author clone, active P1 snapshot, installation descriptor/env, fixture-only secrets, controlled REST DWH, controlled HTTP Evidence, real compatible Qdrant, deterministic Ollama-compatible embedding fixture, selected core image, and no backend/Pi/frontend.
- [ ] **Step 3: Pre-provision the exact compatible Qdrant collection** outside the product operation and record that setup as a P4-deferred fixture step.
- [ ] **Step 4: Exercise only built `thothctl` product commands** and assert:
1. exact inspect revision/config identity;
2. DWH introspection+LSH, clean-process rerun/resume, physical/LSH artifacts;
3. FK pristine JSON, pause-before-index, candidate export, explicit digest review, resume;
4. pre-curated full-run completion;
5. schema index counts and unchanged rerun;
6. HTTP Evidence dry-run, publish, unchanged rerun, input mutation/new generation/ACTIVE;
7. no-Evidence warning/skip;
8. filesystem `evidence_materialization_required` with no partial output;
9. direct renderer regression and SSH fail-closed result;
10. missing workspace/binding, resume mismatch, different-revision resumable session, concurrent writer, annotation invalid, egress refusal, and semantic incompatibility;
11. no collection creation, no backend listener, no Pi init, no arbitrary mount;
12. exact cleanup preserving all foreign/pre-existing resources.
- [ ] **Step 5: Produce bounded `report.json` and `report.md`**, declare hashes for every retained owned artifact, and scan raw Git objects, names, state, reports, logs, configs, candidates, and Qdrant payloads for fixture canaries/signed queries/raw SQL.
- [ ] **Step 6: Run runner unit tests, then one clean integration run without retry.** On failure, diagnose/fix/regress and start one new clean run; never loop blindly.
- [ ] **Step 7: Commit tooling:** `test: add P2 host preprocessing acceptance`.
### Task 14: Finalize P2 documentation and independent manual walkthrough
**Files:**
- Update: `docs/testing/p2-p6-manual-verification.md` P2 section only
- Update: local/server installation manuals
- Update: `docs/contracts/workspace-preprocessing-cli.md`
- Optionally create manual lab helper files if concrete setup cannot remain concise
- [ ] **Step 1: Document prerequisites and boundaries**: Docker/Compose, active registry snapshot, existing compatible collection, running semantic services for semantic commands, no host language runtimes, no backend/Pi.
- [ ] **Step 2: Fill exact P2 commands** for inspect, DWH/resume, FK export/review digest, check/import, index, HTTP dry/run, full run, unchanged rerun, filesystem deferred result, secret scan, and cleanup.
- [ ] **Step 3: Explain each observed component/artifact** without exposing config or secret contents.
- [ ] **Step 4: Require a new manual root and `VERDICT.md`** with reviewer, UTC time, explicit result for every P2 check, observations, and exactly `P2 manual acceptance: PASS|FAIL`. Automation never writes it.
- [ ] **Step 5: Add mechanical docs tests** for all released commands and stable codes.
- [ ] **Step 6: Commit:** `docs: add P2 preprocessing operator walkthrough`.
### Task 15: Run affected-layer verification and hand off the hard checkpoint
**Files:**
- Modify after evidence exists: `PROJECT_STATE.md`
- [ ] **Step 1: Run complete affected Go gates:** `cd tools/thothctl && go test ./... && go vet ./...`, plus the release build contract.
- [ ] **Step 2: Run complete backend gates:** `cd backend && npx vitest run && npx tsc --noEmit -p . && npm run build`.
- [ ] **Step 3: Run focused harness tests listed in the target map, then full `.venv/bin/pytest -q` if feasible.** Any baseline failure must be reported exactly; touched Python files must be Ruff-clean.
- [ ] **Step 4: Run all affected Compose/security contracts** from Task 11 and `git diff --check`.
- [ ] **Step 5: Run exactly one final clean P2 integration at the final source commit:**
```bash
./scripts/p2-acceptance.sh integration --keep
```
Expected final lines:
```text
P2 automated integration: PASS
P2 manual acceptance: PENDING
```
- [ ] **Step 6: Verify report hashes, declared artifacts, secret scan, closed listeners, no maintenance container, and exact cleanup/retention.**
- [ ] **Step 7: Update and commit only tracked project state** with retained report path and truthful scope:
```bash
git add PROJECT_STATE.md
git commit -m "docs: record P2 automated acceptance"
```
- [ ] **Step 8: Report separate statuses and STOP:**
```text
P2 automated integration: PASS — <retained report>
P2 manual acceptance: PENDING — docs/testing/p2-p6-manual-verification.md#p2
P3 authorization: PENDING — awaiting explicit user decision
```
Do not begin P3, mark manual PASS, or infer implementation approval from plan approval or automated evidence.
---
## Requirement traceability
| Requirement/decision | P2 implementation/proof | Deferred truth |
|---|---|---|
| D2 / P2 / RNF7 | Native `thothctl`, dedicated one-shot service, existing engine | GUI/backend endpoint excluded |
| RF1.2 / RNF1 | File-only bindings, operation-specific mounts, redaction/scan | No secret in Git/rendered output |
| RF1.3 / RNF5 | Same binding+renderer code and byte-equivalence test | P3 canonical cross-revision identity/migration |
| RF1.4 / RF2.2 | REST process goal; direct regression | SSH operational support P10 |
| RF2.1 | DWH command, JSON, outer+child resume | — |
| RF2.3 | Physical/LSH production then schema-index consumption | — |
| RF2.4 / RNF2 | Immutable engine generations, unchanged rerun, ACTIVE preservation | Cross-revision DWH reuse P3 |
| RF3.1 | JSON candidates/check, bounded SQL ingress, explicit digest review | Git-canonical review/sync P5 |
| RF3.2 / D5 | Runtime-local P2 annotations only | Repository annotations and pinned sync P5 |
| RF3.3 | No model-derived FK path added | Existing workflow invariant preserved |
| RF4.1 | Existing schema hash/upsert + JSON counts | Collection lifecycle P4 |
| RF5.2 | Policy-allowed HTTP dry/run/resume/publish | Filesystem P6; broader S3 policy separately reviewed |
| RF5.3 / D9 | Existing per-run behavior only | Long-term GC/retention P9 |
| RNF3 | Safe errors, prior ACTIVE preserved, no-Evidence warning | — |
| RNF4 | Workspace binding + P2 different-revision session guard | Revision-scoped points/roots P3 |
| RF8.5–8.6 / RNF8–9 | Clean process goal + separate walkthrough | Aggregate P2–P6 verification after P6 |
| PRD AC2 | Native release binary on local/server installation profiles | Windows Docker is a separate manual claim |
| PRD AC8 | HTTP unchanged/mutation cases | Filesystem change/GC P6/P9 |
## Explicit exclusions
- No frontend/GUI or backend HTTP preprocessing endpoint.
- No host Python, Node, Pi, `tht`, arbitrary shell, arbitrary entrypoint, or arbitrary host mount.
- No Git pull/publish/push, active-revision transition, or authoring API in P2.
- No P3 canonical effective fingerprint, ownership migration, revision-scoped roots/points, or `.tht-dwh` operator chapter.
- No P4 collection creation/index repair/rebuild/maintenance drain.
- No P5 Git-canonical FK annotation sync/acceptance.
- No P6 filesystem Evidence materialization, realpath/symlink containment, or pinned-tree retention.
- No PSD migration/re-embedding (P7), final search/session/L2 gate (P8), policy-driven long-term GC (P9), or SSH runtime (P10).
- No changed embedding model/dimensions/distance, external vector service, pgvector compatibility path, or NL→SQL workflow change.
## Execution notes
- Every implementation task is RED → minimal GREEN → focused verification → commit.
- Never weaken an existing P1 security invariant to simplify P2.
- P2's session-inventory guard and deterministic same-revision config path are deliberate temporary safety mechanisms, not substitutes for P3.
- Keep automated FK mechanics distinct from human review and from the later P5 Git decision.
- A passed plan review authorizes only plan acceptance. Implementation starts only after the user's separate explicit approval.