mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
7356d6794bb9d0480394bd645a165fbc57e6136e
ThothII/docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md
T

43 KiB
Raw Blame History

P2 Host Workspace Preprocessing CLI Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Implement P2/D2 as a native thothctl workspace interface that runs the existing preprocessing engine in a hardened one-shot container, derives its configuration from one active schema-v3 Git workspace revision plus installation-local bindings, and requires no Python, Node, Pi, or running backend HTTP service on the host.

Architecture: thothctl validates a closed command grammar, reconstructs the exact installation Compose project, resolves the selected core image to an immutable Docker image ID, and starts only the profile-gated workspace-maintenance service with --no-deps. A compiled Node entrypoint reads an already-active immutable registry snapshot, uses the same binding resolver and runtime renderer as sessions, writes a deterministic revision-owned protected harness config, and invokes fixed existing tht commands. A versioned coordinator state and one kernel-released writer lock serialize mutation, preserve outer/child resume identity, and stop at a digest-bound FK review checkpoint.

Tech Stack: Go 1.24 (thothctl), Docker Compose v2, Node.js 22, TypeScript 5, Python 3.12, Typer, Pydantic 2, Qdrant 1.18.2, the internal Ollama-compatible embedding interface, Vitest, pytest, Bash/Node acceptance tooling.

Source PRD and design: docs/prd/2026-08-09-workspace-preprocessing-prd.md D2/P2, RF1.2–RF1.4, RF2, RF3.1, RF4.1, RF5.2, RF8.5–RF8.6, RNF1–RNF9; docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md §§1–4, 9–10; docs/testing/p2-p6-manual-verification.md P2.

Planning status: DESIGN/PLAN ONLY. Do not change production code, start P2 implementation, or create a persistent implementation goal until the reviewer asks for plan validation and then gives explicit implementation approval.

P2 completion contract

P2 is complete only when all of the following are true:

  1. The only public host interface is the installed native thothctl binary. Docker/Compose is required, but host Python, Node, Pi, tht, and a running Fastify backend are not.
  2. Every command consumes an already-active, validated registry snapshot and binds the exact workspace ID, 40-hex commit, descriptor blob/digest, installation bindings, runtime roots, and selected internal semantic contract before mutation.
  3. Operator and session configuration use the same resolveRuntimeBindings and renderRuntimeConfig implementation. P2 uses one deterministic same-revision config-source path so current schema-v1 DWH/Evidence resume works; P3 later introduces cross-revision canonical effective identity and explicit migrations.
  4. DWH introspection+LSH, FK suggestion/check, schema indexing, and HTTP Evidence preprocessing invoke the existing harness engine through fixed argv and pristine JSON machine interfaces. No second preprocessing engine is added.
  5. A full run with new FK candidates stops before schema/Evidence writes. Continuation requires a reviewer-supplied annotations file and an explicit acknowledgement of the exact candidate digest; schema check alone is not treated as human approval.
  6. Mutating P2 operations are safe while roots and semantic point IDs are still workspace-global: under the workspace writer lock they refuse if any resumable session is pinned to a different workspace revision. P3 removes this temporary restriction by introducing revision-scoped curated/semantic state.
  7. P2 never creates, repairs, deletes, or rebuilds a Qdrant collection. Schema/Evidence writes require an already-existing, exactly compatible collection and a harness require_existing mode that cannot race into auto-create. P4 owns lifecycle reconciliation.
  8. HTTP Evidence is operational only under installation-local egress policy. Private hosts require an exact installation allowlist; redirects are rechecked; metadata/link-local targets are always refused. S3 custom/private/insecure endpoints and ambient credentials remain fail-closed in P2 unless a later separately reviewed plan expands policy.
  9. Filesystem Evidence is rendered but execution stops before discovery with evidence_materialization_required and no partial corpus/vector publication. P6 owns materialization and symlink/containment checks.
  10. postgres_direct and rest_api routing remain supported and are regression-tested; the clean P2 process goal uses controlled REST. ssh_tunnel returns a stable fail-closed result until P10.
  11. One clean-state product-path integration command passes without retry, produces retained machine/human reports and a secret scan, and proves exact cleanup. Manual P2 acceptance remains independent and PENDING.
  12. Work stops after the P2 handoff. No P3 work begins without a new explicit user authorization.

Truthful command status at the P2 checkpoint

Command P2 status Deliberate boundary
workspace inspect Operational Reads active snapshot only; does not pull/activate Git
workspace preprocess dwh Operational for REST/direct Same-revision config identity; cross-revision reuse is P3
workspace schema suggest-fks Operational, machine-safe Candidate export only; no automatic human acceptance
workspace schema check Operational Imports reviewed annotations and records digest-bound local P2 acknowledgement
workspace index-schema Operational with compatible pre-existing collection Collection create/repair/rebuild is P4
workspace preprocess evidence Operational for policy-allowed HTTP; filesystem deferred Filesystem materialization is P6; S3 expansion needs separate policy review
workspace preprocess run Operational with FK checkpoint Git-canonical annotations are P5; revision-global writes use the P2 session-inventory guard

P2 is therefore the host CLI/orchestration checkpoint, not final acceptance of the PSD filesystem path or the complete PRD chain.

Frozen host command grammar

thothctl --installation <absolute>/thothii-installation.yaml workspace inspect
  --workspace <id> [--json]

thothctl ... workspace preprocess dwh
  --workspace <id> [--resume <outer-run-id>] [--json]

thothctl ... workspace schema suggest-fks
  --workspace <id>
  [--from-sql <regular-file>]... [--assume <column=table>]...
  [--output <new-file>] [--json]

thothctl ... workspace schema check
  --workspace <id>
  [--annotations <regular-file> --reviewed-candidates <sha256:hex>]
  [--json]

thothctl ... workspace index-schema
  --workspace <id> [--json]

thothctl ... workspace preprocess evidence
  --workspace <id> [--dry-run] [--resume <outer-run-id>] [--json]

thothctl ... workspace preprocess run
  --workspace <id> [--resume <outer-run-id>] [--json]

Rules:

  • --workspace occurs exactly once and matches [a-z][a-z0-9-]{2,62}.
  • All run IDs are 32 lowercase hex characters and identify outer P2 state, never a path or raw child checkpoint.
  • At most 32 --from-sql files, 1 MiB each and 16 MiB total. thothctl opens each as a canonical regular non-symlink/reparse-point file, rechecks identity after reading, and streams a schema-versioned request over stdin. No host directory is mounted.
  • --assume occurs at most 256 times; each value is at most 256 bytes and is validated before Compose.
  • --annotations is a single UTF-8 YAML file, at most 16 MiB. --reviewed-candidates is mandatory with it and must equal the persisted candidate artifact digest. The pair is invalid without both flags.
  • --output is created exclusively with restrictive permissions after the returned workspace/run/digest identity has been verified. Existing files, symlinks, hardlinks, and Windows reparse targets are refused.
  • Existing harness suggest-fks --write is intentionally not exposed: an automatic merge is not a human review decision.
  • No unknown flag, passthrough separator, environment-selected command, shell fragment, or arbitrary container entrypoint is accepted.

Public result and exit contract

The one-shot entrypoint always emits exactly one bounded schema-versioned JSON object. thothctl --json parses it strictly and re-encodes it, so Compose progress cannot contaminate stdout. Human mode renders only allowlisted fields.

interface WorkspaceOperationResult {
  schemaVersion: 1;
  status: "succeeded" | "unchanged" | "dry_run" | "blocked" | "failed";
  code:
    | "ok" | "workspace_not_found" | "workspace_not_activatable"
    | "binding_missing" | "preprocessing_conflict"
    | "preprocessing_resume_mismatch" | "manual_review_required"
    | "evidence_materialization_required" | "effective_config_mismatch"
    | "semantic_index_incompatible" | "annotation_invalid"
    | "egress_policy_refused";
  workspaceId: string;
  workspaceRevision: string;
  descriptorBlob: string;
  operation: string;
  runId?: string;
  childRuns?: Record<string, string>;
  completedStages: string[];
  counts?: Record<string, number>;
  artifactIdentities?: Array<{ kind: string; digest: string }>;
  warnings?: string[];
}
  • Exit 0: succeeded, unchanged, or dry_run.
  • Exit 3: expected operator checkpoint/block (manual_review_required, evidence_materialization_required, lock/revision conflict).
  • Exit 2: host grammar or unsafe local file error.
  • Exit 1: operational failure.
  • Stdout JSON maximum: 1 MiB. Sanitized stderr maximum: 64 KiB. Child stdout/stderr and every stage have explicit limits/timeouts.
  • Never return descriptor endpoints with credentials/query strings, secret contents or paths, signed URLs, raw SQL, rendered configuration, raw child stderr, arbitrary exception text, Qdrant payload contents, or host/container environment dumps.

P2 state and identity layout

/data/sessions/<workspace-id>/preprocessing/
├── writer.lock
├── runtime-config/
│   └── <40-hex-revision>.yaml
├── runtime-config-manifests/
│   └── <40-hex-revision>.json
├── jobs/
│   └── <32-hex-outer-run-id>.json
├── fk-candidates/
│   └── <32-hex-outer-run-id>.yaml
└── fk-reviews/
    └── <32-hex-outer-run-id>.json
  • writer.lock is a regular 0600 file held by a Linux kernel advisory lock for the entire outer operation. The file may persist; the kernel lock is released on crash/container death. inspect never takes it.
  • Lock order is always P2 workspace writer lock → existing harness stage lock. Harness code never acquires the P2 lock, preventing inversion/deadlock.
  • Every directory component is opened/validated without following symlinks. State files are 0600, written to an exclusive sibling, fsynced, renamed, and parent-fsynced. Hardlink count must be one.
  • The deterministic config path fixes P2 same-revision config_source identity. Its manifest binds workspace, revision, descriptor blob, config SHA-256, file identity, and the current existing harness ownership binding. Same path + different bytes returns effective_config_mismatch; P3 introduces semantic cross-revision equivalence.
  • Job state binds operation, revision, descriptor blob, config digest, non-secret binding identity, completed stage records, child run IDs, candidate/review digests, and terminal status. Resume revalidates all fields and reconciles a child publication that completed immediately before an outer-state crash.
  • Before any schema/Evidence mutation, enumerate resumable session manifests for the workspace. A different pinned revision returns preprocessing_conflict; no write begins. This is the explicit P2 bridge until P3 revision isolation.

One-shot service security contract

workspace-maintenance is a dedicated profile service, not compose run core:

  • same exact selected core image, resolved to its immutable local image ID; generated final override uses that ID and pull_policy: never;
  • docker compose run --rm --no-deps --no-TTY --name <owned-name> workspace-maintenance ...;
  • no build, frontend, published port, Pi auth, Pi state, Pi trust initialization, Docker socket, home credential directory, or arbitrary command;
  • non-root 10001, read_only: true, cap_drop: [ALL], no-new-privileges:true, restrictive tmpfs, registry active snapshots read-only, sessions root writable;
  • only operation-required connector/Evidence secret files are mounted; AWS ambient environment is cleared;
  • semantic commands require already-running healthy Qdrant/embedding services and do not start/stop them; DWH/inspect commands do not start dependencies;
  • exact operation labels and container identity are recorded. Cancellation terminates the process group, verifies the owned container labels/image, removes only that container, and preserves all pre-existing services, volumes, and networks;
  • fully rendered Compose is validated before launch, and post-run container/image identity is checked before accepting output.

Target file map

Native host CLI

  • tools/thothctl/cmd/thothctl/main.go, main_test.go: public grammar/help, dispatch, exit codes.
  • Create tools/thothctl/internal/workspaceops/operations.go, operations_test.go: immutable image resolution, generated override, Compose run, cancellation cleanup, JSON validation.
  • tools/thothctl/internal/config/installation.go, installation_test.go: maintenance service override and operation-specific binding discovery.
  • tools/thothctl/internal/safeio/files.go, platform files/tests: bounded no-follow input and exclusive output.
  • tools/thothctl/internal/output/sanitize.go, tests: bounded redaction.
  • tools/thothctl/internal/pi/update.go, tests: selected image override must pin both core and workspace-maintenance.

Compose/image boundary

  • compose.yaml: dedicated profile-gated service with shared image identity and least privilege.
  • deploy/compose.local.yaml, deploy/compose.server.yaml: correct registry/session storage semantics.
  • deploy/compose.git-https.yaml, deploy/compose.git-ssh.yaml: do not attach Git credentials to P2 active-snapshot operations.
  • scripts/generate-connector-secrets-override.sh: operation-specific maintenance secrets.
  • Create docker/workspace-maintenance-entrypoint.sh; modify docker/core.Dockerfile.
  • Retire/redirect fixture-only deploy/compose.preprocess.yaml as a non-public compatibility test path; do not leave two operator commands.

Shared Node operator

  • Create backend/src/workspaces/runtime-config-lease.ts: shared snapshot read/render and deterministic protected config lease.
  • Modify backend/src/tht/tht-runner.ts to delegate session/operator rendering to the shared component without changing route behavior.
  • Create backend/src/workspaces/preprocessing-state.ts: state schema, durable writes, locks, resume reconciliation.
  • Create backend/src/workspaces/preprocessing-service.ts: closed stage coordinator and security preflights.
  • Create backend/src/workspace-maintenance.ts: compiled stdin/argv entrypoint and pristine result encoder.
  • Add tests: backend/test/workspace-runtime-config-lease.test.ts, workspace-preprocessing-state.test.ts, workspace-preprocessing-service.test.ts, workspace-maintenance.test.ts.

Harness machine contracts

  • harness/tht/cli/preprocess_cmd.py: authoritative runtime workspace identity; require-existing collection mode.
  • harness/tht/cli/schema_cmd.py: extracted deterministic helpers, JSON suggest/check, safe SQL staging and annotation validation.
  • harness/tht/cli/vector_cmd.py: JSON schema-index result.
  • harness/tht/adapters/vector/qdrant.py: explicit non-creating strict mode for P2.
  • Tests: harness/tests/test_preprocess_cli.py, test_schema_fk_annotations.py, test_qdrant_cli_commands.py, test_registry_evidence_config.py, test_http_evidence_source.py, plus new focused security cases.

Docs and gates

  • Create docs/contracts/workspace-preprocessing-cli.md.
  • Update local/server installation manuals and docs/testing/p2-p6-manual-verification.md P2 only.
  • Create scripts/p2-acceptance.sh, backend/scripts/p2-acceptance.mjs, backend/scripts/p2-acceptance.test.mjs.
  • Create scripts/p2-manual-acceptance.sh, backend/scripts/p2-manual-acceptance.mjs only if needed to generate the isolated walkthrough lab; automation must never create PASS.
  • Update PROJECT_STATE.md only after implementation evidence exists.

Task 1: Freeze the native CLI, file-ingress, and result contracts

Files:

  • Modify: tools/thothctl/cmd/thothctl/main.go

  • Modify: tools/thothctl/cmd/thothctl/main_test.go

  • Modify: tools/thothctl/internal/safeio/files.go

  • Modify platform-specific safe-I/O tests

  • Create: tools/thothctl/internal/workspaceops/operations.go

  • Create: tools/thothctl/internal/workspaceops/operations_test.go

  • Create: docs/contracts/workspace-preprocessing-cli.md

  • Step 1: Write RED parser-table tests for every valid command above and for duplicate/missing/unknown flags, invalid IDs, incompatible annotation flags, option-count/size limits, and passthrough/shell attempts.

  • Step 2: Run cd tools/thothctl && go test ./cmd/thothctl ./internal/workspaceops -run 'Workspace|workspace' -v and verify the new tests fail because workspace is unknown.

  • Step 3: Add closed request types (InspectRequest, DwhRequest, SuggestFksRequest, CheckSchemaRequest, IndexSchemaRequest, EvidenceRequest, RunRequest) and a parser that cannot represent arbitrary argv.

  • Step 4: Write RED safe-I/O tests for symlinks, hardlinks, directory input, replacement during read, Windows reparse points, existing output, >1 MiB SQL, >16 MiB total, and non-UTF-8 annotation input.

  • Step 5: Implement bounded reads and exclusive restrictive output using existing platform seams; return only generic file errors.

  • Step 6: Add schema-v1 stdin request/result validation with exact field allowlists and output bounds.

  • Step 7: Run focused Go tests and gofmt -w, then go test ./....

  • Step 8: Commit: feat: define P2 host workspace command contract.

Task 2: Add pristine harness JSON interfaces without changing the engine

Files:

  • Modify: harness/tht/cli/schema_cmd.py

  • Modify: harness/tht/cli/vector_cmd.py

  • Modify: harness/tht/cli/preprocess_cmd.py

  • Modify: harness/tests/test_schema_fk_annotations.py

  • Modify: harness/tests/test_qdrant_cli_commands.py

  • Modify: harness/tests/test_preprocess_cli.py

  • Step 1: Write RED tests requiring schema suggest-fks --json, schema check --json, and vector index-schema --json to emit exactly one JSON object on stdout for success and failure, with no color/prose contamination.

  • Step 2: Write RED deterministic FK tests for bounded staged SQL files, stable candidate ordering, candidate SHA-256, annotation import, orphan counts, and no implicit review/write.

  • Step 3: Write RED Evidence identity test showing a config named /dev/fd/3 still uses runtime_identity.workspace_id, never the config basename.

  • Step 4: Run:

cd harness
.venv/bin/pytest -q \
  tests/test_schema_fk_annotations.py \
  tests/test_qdrant_cli_commands.py \
  tests/test_preprocess_cli.py

Expected: FAIL only on the new machine-contract assertions.

  • Step 5: Extract pure helpers returning typed dictionaries/models; keep existing human commands as renderers over the same helpers.
  • Step 6: Implement the JSON flags and authoritative workspace identity. Catch expected exceptions and emit stable safe codes; never serialize arbitrary exception text.
  • Step 7: Run the three focused files and touched Ruff:
.venv/bin/ruff check \
  tht/cli/schema_cmd.py tht/cli/vector_cmd.py tht/cli/preprocess_cmd.py \
  tests/test_schema_fk_annotations.py tests/test_qdrant_cli_commands.py tests/test_preprocess_cli.py
  • Step 8: Commit: feat: add P2 harness machine contracts.

Task 3: Add a non-creating semantic writer mode

Files:

  • Modify: harness/tht/config.py

  • Modify: harness/tht/adapters/factory.py

  • Modify: harness/tht/adapters/vector/qdrant.py

  • Modify: harness/tests/test_qdrant_vector_store.py

  • Modify: harness/tests/test_qdrant_cli_commands.py

  • Modify: harness/tests/test_registry_evidence_config.py

  • Step 1: Write RED tests proving operator mode refuses a missing collection without issuing create/index mutations, refuses wrong dimensions/distance/index type, and still writes to an existing compatible collection.

  • Step 2: Run the focused tests and confirm current _ensure_collection(strict=True) incorrectly creates the collection.

  • Step 3: Add an internal rendered field such as vectors.collection_lifecycle: require_existing; it is not a descriptor option and defaults to legacy behavior for non-operator configs.

  • Step 4: Thread the mode through the factory/store and perform a read-only exact collection/index preflight before any upsert.

  • Step 5: Add a race regression: delete the collection after preflight and prove the write fails rather than recreates it.

  • Step 6: Run focused pytest and touched Ruff.

  • Step 7: Commit: fix: prevent P2 from owning Qdrant lifecycle.

Task 4: Extract the shared runtime configuration lease

Files:

  • Create: backend/src/workspaces/runtime-config-lease.ts

  • Create: backend/test/workspace-runtime-config-lease.test.ts

  • Modify: backend/src/tht/tht-runner.ts

  • Modify: backend/test/tht-runner.test.ts

  • Modify: backend/test/workspace-runtime-handoff.test.ts

  • Step 1: Write RED equivalence tests feeding the same immutable snapshot, env, roots, installation overlay, and semantic contract to the session and operator callers and requiring byte-identical YAML.

  • Step 2: Write RED identity/safety tests for snapshot replacement, wrong commit/path, symlink/hardlink, wrong workspace ID, unstable config destination, same-revision changed bytes, mode, fsync/rename failure, and cleanup.

  • Step 3: Run:

cd backend
npx vitest run \
  test/workspace-runtime-config-lease.test.ts \
  test/tht-runner.test.ts \
  test/workspace-runtime-handoff.test.ts
  • Step 4: Move snapshot validation, runtime roots, installation-overlay parsing, binding resolution, and rendering out of ThtRunner into one explicit-input component.
  • Step 5: Preserve session behavior: ThtRunner.acquireWorkspaceRuntime delegates to the component and retains its current opaque FD-backed temporary lease.
  • Step 6: Add operator mode: deterministically publish /data/sessions/<id>/preprocessing/runtime-config/<revision>.yaml plus a manifest, mode 0400/0600, and set collection_lifecycle: require_existing.
  • Step 7: Prove same-revision rerun path identity and changed config/binding refusal. Do not implement P3 semantic cross-revision canonicalization.
  • Step 8: Run focused tests, npx tsc --noEmit -p ., and npm run build.
  • Step 9: Commit: refactor: share registry runtime configuration leases.

Task 5: Build durable outer state, locking, and revision guard

Files:

  • Create: backend/src/workspaces/preprocessing-state.ts

  • Create: backend/test/workspace-preprocessing-state.test.ts

  • Modify: docker/core.Dockerfile (install/pin the kernel lock utility only when the implementation proves it is absent)

  • Step 1: Write RED state-schema tests for valid state, same-operation resume, cross-workspace/revision/operation/config mismatch, tampering, run-ID traversal, restrictive modes, atomic failure, and bounded fields.

  • Step 2: Write RED cross-process lock tests with two processes/containers: one wins, one receives preprocessing_conflict, and SIGKILL releases the kernel lock without deleting unrelated state.

  • Step 3: Write RED session-inventory tests: no sessions/current-only sessions permit mutation; a resumable different-revision manifest blocks; finalized/archived sessions follow existing resume policy.

  • Step 4: Implement the exact state layout and durable write protocol described above.

  • Step 5: Implement lock acquisition ordering and safe conflict mapping. Do not invent stale-PID deletion; the kernel owns lock lifetime.

  • Step 6: Implement active-snapshot revalidation immediately before each mutating child stage and the different-revision resumable-session guard.

  • Step 7: Add crash reconciliation tests where a child publishes DWH/corpus state but outer state has not yet advanced.

  • Step 8: Run focused Vitest, typecheck, and build.

  • Step 9: Commit: feat: add P2 preprocessing operation state.

Task 6: Build the compiled inspect/operator boundary

Files:

  • Create: backend/src/workspace-maintenance.ts

  • Create: backend/src/workspaces/preprocessing-service.ts

  • Create: backend/test/workspace-maintenance.test.ts

  • Create: backend/test/workspace-preprocessing-service.test.ts

  • Modify: backend/src/workspaces/types.ts only if a separate operator-code union cannot stay private

  • Step 1: Write RED entrypoint process tests for exact JSON, malformed/extra stdin, unknown command/field, stdout/stderr bounds, timeout, signal, raw exception/stderr redaction, and no Fastify listener.

  • Step 2: Write RED inspect tests for absent/corrupt/stale active state, migration-required descriptor, missing bindings, exact commit/blob/config identities, safe capability warnings, and no URL/secret output.

  • Step 3: Run focused Vitest and verify no operator exists.

  • Step 4: Implement a closed WorkspacePreprocessingService dependency interface: active registry reader, shared config lease, fixed child runner, state store, session inventory, semantic preflight, egress policy.

  • Step 5: Implement active-snapshot-only acquisition. P2 does not pull or activate Git; clean installations receive workspace_not_activatable with safe instructions.

  • Step 6: Implement bounded child execution with fixed executable/argv, -c after the subcommand, FD-backed config/input, process-group cancellation, per-stage timeout, and strict one-document child JSON parsing.

  • Step 7: Implement inspect and result encoding.

  • Step 8: Run tests, typecheck, build, and verify dist/workspace-maintenance.js exists.

  • Step 9: Commit: feat: add P2 workspace maintenance operator.

Task 7: Implement DWH preprocessing and outer resume

Files:

  • Modify: backend/src/workspaces/preprocessing-service.ts

  • Modify: backend/test/workspace-preprocessing-service.test.ts

  • Modify: harness/tests/test_dwh_preprocess_job.py

  • Modify: harness/tests/test_lsh_job_resume.py

  • Step 1: Write RED service tests requiring fixed preprocess dwh --steps introspect,lsh --json -c /dev/fd/N, child result validation, outer/child run IDs, completed stages, safe artifact digests, and failure mapping.

  • Step 2: Add real harness regressions for deterministic same-revision config-source path, second clean-process rerun, resume after introspection, changed binding/config refusal, and ACTIVE preservation on failure.

  • Step 3: Run focused backend and harness tests.

  • Step 4: Implement preprocess dwh under the outer writer lock and persist state before/after every child transition.

  • Step 5: Reconcile a published child run after an injected outer crash without rerunning or corrupting ACTIVE.

  • Step 6: Verify REST and direct rendered routing. SSH returns workspace_not_activatable with a P10 warning.

  • Step 7: Run focused gates and commit: feat: run workspace DWH preprocessing from thothctl operator.

Task 8: Implement FK candidate export and digest-bound review

Files:

  • Modify: backend/src/workspaces/preprocessing-service.ts

  • Modify: backend/src/workspaces/preprocessing-state.ts

  • Modify: relevant backend tests

  • Modify: tools/thothctl/internal/workspaceops/operations.go

  • Modify: Go tests

  • Step 1: Write RED end-to-end unit/process tests: new candidates create a bounded artifact and return manual_review_required; schema/Evidence child calls are absent.

  • Step 2: Add safe host ingress tests proving SQL and annotations travel only over stdin, are absent from Compose argv/state/logs, and staging files are removed.

  • Step 3: Add safe host egress tests for candidate export identity/digest, existing destination refusal, and sanitized JSON mode.

  • Step 4: Implement schema suggest-fks with candidate count/digest and optional exclusive output.

  • Step 5: Implement schema check in two modes: read-only orphan validation; or reviewed annotation import requiring the exact candidate digest. Persist review digest + annotation digest + workspace/revision.

  • Step 6: Require the review record on full-run resume. A mere zero-orphan result without reviewer digest is insufficient.

  • Step 7: Preserve the boundary: P2 updates runtime-local annotations only; it never writes Git. Output warns that P5 will supersede this local acknowledgement.

  • Step 8: Run Go/backend/harness focused gates and commit: feat: add P2 FK review checkpoint.

Task 9: Implement schema indexing and Evidence policy boundaries

Files:

  • Modify: backend/src/workspaces/preprocessing-service.ts

  • Modify: backend service tests

  • Modify: harness/tests/test_http_evidence_source.py

  • Modify: harness/tests/test_registry_evidence_config.py

  • Modify: harness/tests/test_semantic_kind_isolation.py

  • Step 1: Write RED schema-index tests for compatible pre-existing collection, deterministic JSON counts, idempotent repeat, missing/incompatible refusal, and no collection-create request.

  • Step 2: Write RED Evidence tests for no-Evidence warning/skip, HTTP dry-run/run/resume/unchanged/mutation, authoritative workspace identity, ACTIVE preservation, and filesystem early stop before adapter/Qdrant calls.

  • Step 3: Write RED egress tests for exact private-host allowlist, DNS re-resolution, redirect to private/link-local/metadata, signed URL query redaction, and refusal of S3 ambient/custom/private/insecure modes.

  • Step 4: Implement installation-local egress-policy parsing with exact bounded hostnames and no wildcard. Descriptor flags alone never grant network access.

  • Step 5: Implement index-schema and preprocess evidence with semantic preflight and stable result mapping.

  • Step 6: Revalidate active revision and session inventory immediately before each write.

  • Step 7: Run focused tests/touched Ruff/backend typecheck/build and commit: feat: add guarded P2 semantic preprocessing.

Task 10: Implement the ordered full-run coordinator

Files:

  • Modify: backend/src/workspaces/preprocessing-service.ts

  • Modify: backend/test/workspace-preprocessing-service.test.ts

  • Modify: backend/test/workspace-maintenance.test.ts

  • Step 1: Write a RED stage-table test for exact order dwh → fk_suggest → fk_review/check → schema_index → evidence and for no hidden/skipped mutation.

  • Step 2: Add scenarios: pre-curated/no-new-candidate completion; new-candidate block; digest-reviewed resume; no-Evidence warning; filesystem deferred block; each child failure; resume mismatch; outer crash reconciliation.

  • Step 3: Implement the coordinator as an explicit state machine, not recursive command dispatch.

  • Step 4: Persist completion after each verified child artifact and never mark a stage based only on exit code.

  • Step 5: Prove unchanged rerun creates no duplicate schema/Evidence points or generation.

  • Step 6: Run focused Vitest, typecheck, build, and commit: feat: orchestrate the P2 preprocessing chain.

Task 11: Add the hardened maintenance service and selected-image handoff

Files:

  • Modify: compose.yaml

  • Modify: deploy/compose.local.yaml

  • Modify: deploy/compose.server.yaml

  • Modify: connector/Git override files and generators as required

  • Create: docker/workspace-maintenance-entrypoint.sh

  • Modify: docker/core.Dockerfile

  • Modify: tools/thothctl/internal/workspaceops/operations.go

  • Modify: tools/thothctl/internal/pi/update.go

  • Modify relevant Go/Bash/Compose tests

  • Step 1: Write RED Compose contract tests for the exact security contract, profile, mounts, no Pi/no port/no build, local/server storage, and absence of Git credentials on active-snapshot operations.

  • Step 2: Write RED image-precedence tests across base/profile/operator overrides/current-image override; core and maintenance must resolve to the same immutable ID.

  • Step 3: Write RED lifecycle tests for --no-deps, pre-existing service preservation, interruption cleanup, hostile container name/label collision, tag replacement, and output rejection on post-run image mismatch.

  • Step 4: Add the dedicated service and entrypoint; the entrypoint executes only the compiled operator and never calls Pi trust setup.

  • Step 5: Generate a per-operation final override that pins immutable image ID, exact secrets, egress policy, and owned labels. Validate docker compose config structurally before run.

  • Step 6: Extend Pi update/rollback override generation so future selected images cannot split core and maintenance.

  • Step 7: Run:

bash scripts/test-preprocess-compose-config.sh
bash scripts/test-compose-secret-policy.sh
bash scripts/test-default-compose.sh
bash scripts/test-unified-compose.sh
bash scripts/test-no-deployment-coupling.sh
cd tools/thothctl && go test ./...
  • Step 8: Build the core image and invoke operator --help through the exact service without starting Pi/backend/dependencies.
  • Step 9: Commit: feat: package the P2 maintenance service.

Task 12: Complete host dispatch and supported-platform build contract

Files:

  • Modify: tools/thothctl/cmd/thothctl/main.go, tests

  • Modify: tools/thothctl/internal/workspaceops/operations.go, tests

  • Modify: scripts/build-thothctl.sh

  • Modify: scripts/test-thothctl-build-contract.sh

  • Update operator docs

  • Step 1: Add RED command-to-request-to-Compose tests for all seven public commands, JSON/human output, exit mapping, secret redaction, and exact stdin.

  • Step 2: Implement dispatcher integration using only typed requests.

  • Step 3: Cross-build the existing release matrix and verify Windows input/output safety compiles. Do not claim Windows Docker behavior without a Windows Docker run.

  • Step 4: Run go test ./..., build contract, go vet ./..., and gofmt check.

  • Step 5: Commit: feat: expose P2 workspace commands in thothctl.

Task 13: Build the clean-state automated P2 process goal

Files:

  • Create: scripts/p2-acceptance.sh
  • Create: backend/scripts/p2-acceptance.mjs
  • Create: backend/scripts/p2-acceptance.test.mjs
  • Update: .gitignore only if the existing .artifacts/ rule is insufficient

The public command is:

./scripts/p2-acceptance.sh integration --keep
  • Step 1: Write RED acceptance-runner tests for ownership-first state, unique run/project/container/image names, exact cleanup, --keep, injected failure, signal cleanup, report bounds, and no automatic retry.
  • Step 2: Build a clean owned topology under .artifacts/p2-integration/p2-<run-id>/: local bare Git + author clone, active P1 snapshot, installation descriptor/env, fixture-only secrets, controlled REST DWH, controlled HTTP Evidence, real compatible Qdrant, deterministic Ollama-compatible embedding fixture, selected core image, and no backend/Pi/frontend.
  • Step 3: Pre-provision the exact compatible Qdrant collection outside the product operation and record that setup as a P4-deferred fixture step.
  • Step 4: Exercise only built thothctl product commands and assert:
    1. exact inspect revision/config identity;
    2. DWH introspection+LSH, clean-process rerun/resume, physical/LSH artifacts;
    3. FK pristine JSON, pause-before-index, candidate export, explicit digest review, resume;
    4. pre-curated full-run completion;
    5. schema index counts and unchanged rerun;
    6. HTTP Evidence dry-run, publish, unchanged rerun, input mutation/new generation/ACTIVE;
    7. no-Evidence warning/skip;
    8. filesystem evidence_materialization_required with no partial output;
    9. direct renderer regression and SSH fail-closed result;
    10. missing workspace/binding, resume mismatch, different-revision resumable session, concurrent writer, annotation invalid, egress refusal, and semantic incompatibility;
    11. no collection creation, no backend listener, no Pi init, no arbitrary mount;
    12. exact cleanup preserving all foreign/pre-existing resources.
  • Step 5: Produce bounded report.json and report.md, declare hashes for every retained owned artifact, and scan raw Git objects, names, state, reports, logs, configs, candidates, and Qdrant payloads for fixture canaries/signed queries/raw SQL.
  • Step 6: Run runner unit tests, then one clean integration run without retry. On failure, diagnose/fix/regress and start one new clean run; never loop blindly.
  • Step 7: Commit tooling: test: add P2 host preprocessing acceptance.

Task 14: Finalize P2 documentation and independent manual walkthrough

Files:

  • Update: docs/testing/p2-p6-manual-verification.md P2 section only

  • Update: local/server installation manuals

  • Update: docs/contracts/workspace-preprocessing-cli.md

  • Optionally create manual lab helper files if concrete setup cannot remain concise

  • Step 1: Document prerequisites and boundaries: Docker/Compose, active registry snapshot, existing compatible collection, running semantic services for semantic commands, no host language runtimes, no backend/Pi.

  • Step 2: Fill exact P2 commands for inspect, DWH/resume, FK export/review digest, check/import, index, HTTP dry/run, full run, unchanged rerun, filesystem deferred result, secret scan, and cleanup.

  • Step 3: Explain each observed component/artifact without exposing config or secret contents.

  • Step 4: Require a new manual root and VERDICT.md with reviewer, UTC time, explicit result for every P2 check, observations, and exactly P2 manual acceptance: PASS|FAIL. Automation never writes it.

  • Step 5: Add mechanical docs tests for all released commands and stable codes.

  • Step 6: Commit: docs: add P2 preprocessing operator walkthrough.

Task 15: Run affected-layer verification and hand off the hard checkpoint

Files:

  • Modify after evidence exists: PROJECT_STATE.md

  • Step 1: Run complete affected Go gates: cd tools/thothctl && go test ./... && go vet ./..., plus the release build contract.

  • Step 2: Run complete backend gates: cd backend && npx vitest run && npx tsc --noEmit -p . && npm run build.

  • Step 3: Run focused harness tests listed in the target map, then full .venv/bin/pytest -q if feasible. Any baseline failure must be reported exactly; touched Python files must be Ruff-clean.

  • Step 4: Run all affected Compose/security contracts from Task 11 and git diff --check.

  • Step 5: Run exactly one final clean P2 integration at the final source commit:

./scripts/p2-acceptance.sh integration --keep

Expected final lines:

P2 automated integration: PASS
P2 manual acceptance: PENDING
  • Step 6: Verify report hashes, declared artifacts, secret scan, closed listeners, no maintenance container, and exact cleanup/retention.
  • Step 7: Update and commit only tracked project state with retained report path and truthful scope:
git add PROJECT_STATE.md
git commit -m "docs: record P2 automated acceptance"
  • Step 8: Report separate statuses and STOP:
P2 automated integration: PASS — <retained report>
P2 manual acceptance: PENDING — docs/testing/p2-p6-manual-verification.md#p2
P3 authorization: PENDING — awaiting explicit user decision

Do not begin P3, mark manual PASS, or infer implementation approval from plan approval or automated evidence.

Requirement traceability

Requirement/decision P2 implementation/proof Deferred truth
D2 / P2 / RNF7 Native thothctl, dedicated one-shot service, existing engine GUI/backend endpoint excluded
RF1.2 / RNF1 File-only bindings, operation-specific mounts, redaction/scan No secret in Git/rendered output
RF1.3 / RNF5 Same binding+renderer code and byte-equivalence test P3 canonical cross-revision identity/migration
RF1.4 / RF2.2 REST process goal; direct regression SSH operational support P10
RF2.1 DWH command, JSON, outer+child resume —
RF2.3 Physical/LSH production then schema-index consumption —
RF2.4 / RNF2 Immutable engine generations, unchanged rerun, ACTIVE preservation Cross-revision DWH reuse P3
RF3.1 JSON candidates/check, bounded SQL ingress, explicit digest review Git-canonical review/sync P5
RF3.2 / D5 Runtime-local P2 annotations only Repository annotations and pinned sync P5
RF3.3 No model-derived FK path added Existing workflow invariant preserved
RF4.1 Existing schema hash/upsert + JSON counts Collection lifecycle P4
RF5.2 Policy-allowed HTTP dry/run/resume/publish Filesystem P6; broader S3 policy separately reviewed
RF5.3 / D9 Existing per-run behavior only Long-term GC/retention P9
RNF3 Safe errors, prior ACTIVE preserved, no-Evidence warning —
RNF4 Workspace binding + P2 different-revision session guard Revision-scoped points/roots P3
RF8.5–8.6 / RNF8–9 Clean process goal + separate walkthrough Aggregate P2–P6 verification after P6
PRD AC2 Native release binary on local/server installation profiles Windows Docker is a separate manual claim
PRD AC8 HTTP unchanged/mutation cases Filesystem change/GC P6/P9

Explicit exclusions

  • No frontend/GUI or backend HTTP preprocessing endpoint.
  • No host Python, Node, Pi, tht, arbitrary shell, arbitrary entrypoint, or arbitrary host mount.
  • No Git pull/publish/push, active-revision transition, or authoring API in P2.
  • No P3 canonical effective fingerprint, ownership migration, revision-scoped roots/points, or .tht-dwh operator chapter.
  • No P4 collection creation/index repair/rebuild/maintenance drain.
  • No P5 Git-canonical FK annotation sync/acceptance.
  • No P6 filesystem Evidence materialization, realpath/symlink containment, or pinned-tree retention.
  • No PSD migration/re-embedding (P7), final search/session/L2 gate (P8), policy-driven long-term GC (P9), or SSH runtime (P10).
  • No changed embedding model/dimensions/distance, external vector service, pgvector compatibility path, or NL→SQL workflow change.

Execution notes

  • Every implementation task is RED → minimal GREEN → focused verification → commit.
  • Never weaken an existing P1 security invariant to simplify P2.
  • P2's session-inventory guard and deterministic same-revision config path are deliberate temporary safety mechanisms, not substitutes for P3.
  • Keep automated FK mechanics distinct from human review and from the later P5 Git decision.
  • A passed plan review authorizes only plan acceptance. Implementation starts only after the user's separate explicit approval.
Reference in New Issue View Git Blame Copy Permalink