From 38c8398bc1f27d21399bb8ea52e9060a30a7f2dd Mon Sep 17 00:00:00 2001 From: mptyl Date: Tue, 11 Aug 2026 00:09:43 +0200 Subject: [PATCH] docs: add executable P2-P6 implementation plans --- ...-10-p2-host-workspace-preprocessing-cli.md | 1570 ++++++++++- ...6-08-10-p3-effective-config-fingerprint.md | 2342 +++++++++++++++++ .../2026-08-10-p4-qdrant-bootstrap-rebuild.md | 1010 +++++++ .../2026-08-10-p5-curated-fk-annotations.md | 1670 ++++++++++++ .../2026-08-10-p6-evidence-materialization.md | 1624 ++++++++++++ 5 files changed, 8122 insertions(+), 94 deletions(-) create mode 100644 docs/superpowers/plans/2026-08-10-p3-effective-config-fingerprint.md create mode 100644 docs/superpowers/plans/2026-08-10-p4-qdrant-bootstrap-rebuild.md create mode 100644 docs/superpowers/plans/2026-08-10-p5-curated-fk-annotations.md create mode 100644 docs/superpowers/plans/2026-08-10-p6-evidence-materialization.md diff --git a/docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md b/docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md index 89dab8d9..199ec1f7 100644 --- a/docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md +++ b/docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md @@ -1,12 +1,12 @@ # P2 Host Workspace Preprocessing CLI Implementation Plan -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Implement P2/D2 as a native `thothctl workspace` interface that runs the existing preprocessing engine in a hardened one-shot container, derives its configuration from one active schema-v3 Git workspace revision plus installation-local bindings, and requires no Python, Node, Pi, or running backend HTTP service on the host. **Architecture:** `thothctl` validates a closed command grammar, reconstructs the exact installation Compose project, resolves the selected core image to an immutable Docker image ID, and starts only the profile-gated `workspace-maintenance` service with `--no-deps`. A compiled Node entrypoint reads an already-active immutable registry snapshot, uses the same binding resolver and runtime renderer as sessions, writes a deterministic revision-owned protected harness config, and invokes fixed existing `tht` commands. A versioned coordinator state and one kernel-released writer lock serialize mutation, preserve outer/child resume identity, and stop at a digest-bound FK review checkpoint. -**Tech Stack:** Go 1.24 (`thothctl`), Docker Compose v2, Node.js 22, TypeScript 5, Python 3.12, Typer, Pydantic 2, Qdrant 1.18.2, the internal Ollama-compatible embedding interface, Vitest, pytest, Bash/Node acceptance tooling. +**Tech Stack:** Go 1.26.5 (`thothctl`; matches `go.mod` `toolchain go1.26.5`), Docker Compose v2, Node.js 22, TypeScript 5, Python 3.12, Typer, Pydantic 2, Qdrant 1.18.2, the internal Ollama-compatible embedding interface, Vitest, pytest, Bash/Node acceptance tooling. **Source PRD and design:** `docs/prd/2026-08-09-workspace-preprocessing-prd.md` D2/P2, RF1.2–RF1.4, RF2, RF3.1, RF4.1, RF5.2, RF8.5–RF8.6, RNF1–RNF9; `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` §§1–4, 9–10; `docs/testing/p2-p6-manual-verification.md` P2. @@ -19,11 +19,18 @@ P2 is complete only when all of the following are true: 1. The only public host interface is the installed native `thothctl` binary. Docker/Compose is required, but host Python, Node, Pi, `tht`, and a running Fastify backend are not. -2. Every command consumes an already-active, validated registry snapshot and binds the exact workspace ID, 40-hex commit, descriptor blob/digest, installation bindings, runtime roots, and selected internal semantic contract before mutation. -3. Operator and session configuration use the same `resolveRuntimeBindings` and `renderRuntimeConfig` implementation. P2 uses one deterministic same-revision config-source path so current schema-v1 DWH/Evidence resume works; P3 later introduces cross-revision canonical effective identity and explicit migrations. +2. `workspace inspect` consumes the active validated snapshot or, only when the repository-locked + recheck proves no active state exists, submits the `registry_bootstrap` variant to the same addressed + publication owner used by pull, lazy list, status, and author-publication activation. A queued caller + that observed earlier absence returns `already_active` with the locked recheck's exact snapshot; + corrupt active state fails closed. Bootstrap has no base and its complete changed + set is every target workspace ID. Mutating preprocessing commands require that pinned active snapshot + and never pull. Every command binds the exact workspace ID, 40-hex commit, descriptor blob/digest, + installation bindings, runtime roots, and selected internal semantic contract before mutation. +3. Operator and session configuration use the same `resolveRuntimeBindings` and `renderRuntimeConfig` implementation. P2 renders byte-identical YAML for the same snapshot/bindings in both modes and uses one deterministic same-revision config-source path so current schema-v1 DWH/Evidence resume works. Every registry-rendered config, session and maintenance alike, sets `vectors.collection_lifecycle: require_existing`; only legacy non-registry configs retain the old create-capable default. P3 later introduces cross-revision canonical effective identity and explicit migrations. 4. DWH introspection+LSH, FK suggestion/check, schema indexing, and HTTP Evidence preprocessing invoke the existing harness engine through fixed argv and pristine JSON machine interfaces. No second preprocessing engine is added. 5. A full run with new FK candidates stops before schema/Evidence writes. Continuation requires a reviewer-supplied annotations file and an explicit acknowledgement of the exact candidate digest; `schema check` alone is not treated as human approval. -6. Mutating P2 operations are safe while roots and semantic point IDs are still workspace-global: under the workspace writer lock they refuse if any resumable session is pinned to a different workspace revision. P3 removes this temporary restriction by introducing revision-scoped curated/semantic state. +6. Mutating P2 operations are safe while roots and semantic point IDs are still workspace-global: under the workspace writer lock they refuse if any resumable session is pinned to a different workspace revision. Registry bootstrap/pull/author-publication activation uses one repository-first addressed owner and `runUnderOrderedWorkspaceWriterLocks`: it holds the verified root and writer FDs for every changed workspace in lexical order through quiescence/readers, participants, publication, and terminal durability, then closes in reverse. Therefore the active revision cannot change during a mutating publication window. P3 removes the temporary session restriction by introducing revision-scoped curated/semantic state. 7. P2 never creates, repairs, deletes, or rebuilds a Qdrant collection. Schema/Evidence writes require an already-existing, exactly compatible collection and a harness `require_existing` mode that cannot race into auto-create. P4 owns lifecycle reconciliation. 8. HTTP Evidence is operational only under installation-local egress policy. Private hosts require an exact installation allowlist; redirects are rechecked; metadata/link-local targets are always refused. S3 custom/private/insecure endpoints and ambient credentials remain fail-closed in P2 unless a later separately reviewed plan expands policy. 9. Filesystem Evidence is rendered but execution stops before discovery with `evidence_materialization_required` and no partial corpus/vector publication. P6 owns materialization and symlink/containment checks. @@ -35,10 +42,10 @@ P2 is complete only when all of the following are true: | Command | P2 status | Deliberate boundary | |---|---|---| -| `workspace inspect` | Operational | Reads active snapshot only; does not pull/activate Git | +| `workspace inspect` | Operational | Reads active snapshot; under `repository.lock`, it first revalidates active state and returns `already_active`, otherwise an empty registry automatically resumes its sole exact-identity nonterminal bootstrap or creates one; it never pulls an existing active state | | `workspace preprocess dwh` | Operational for REST/direct | Same-revision config identity; cross-revision reuse is P3 | | `workspace schema suggest-fks` | Operational, machine-safe | Candidate export only; no automatic human acceptance | -| `workspace schema check` | Operational | Imports reviewed annotations and records digest-bound local P2 acknowledgement | +| `workspace schema check` | Operational | Requires the exact outer `--resume` run, imports reviewed annotations, and records a digest-bound local P2 acknowledgement | | `workspace index-schema` | Operational with compatible pre-existing collection | Collection create/repair/rebuild is P4 | | `workspace preprocess evidence` | Operational for policy-allowed HTTP; filesystem deferred | Filesystem materialization is P6; S3 expansion needs separate policy review | | `workspace preprocess run` | Operational with FK checkpoint | Git-canonical annotations are P5; revision-global writes use the P2 session-inventory guard | @@ -60,7 +67,7 @@ thothctl ... workspace schema suggest-fks [--output ] [--json] thothctl ... workspace schema check - --workspace + --workspace --resume <32-hex-outer-run-id> [--annotations --reviewed-candidates ] [--json] @@ -77,12 +84,13 @@ thothctl ... workspace preprocess run Rules: - `--workspace` occurs exactly once and matches `[a-z][a-z0-9-]{2,62}`. -- All run IDs are 32 lowercase hex characters and identify outer P2 state, never a path or raw child checkpoint. +- All run IDs are 32 lowercase hex characters and identify outer P2 state, never a path or raw child checkpoint. `schema check` always requires the exact `--resume <32-hex-outer-run-id>` returned by `schema suggest-fks` or the blocked full run; the candidate digest is not a run selector and no automatic digest lookup is permitted. - At most 32 `--from-sql` files, 1 MiB each and 16 MiB total. `thothctl` opens each as a canonical regular non-symlink/reparse-point file, rechecks identity after reading, and streams a schema-versioned request over stdin. No host directory is mounted. - `--assume` occurs at most 256 times; each value is at most 256 bytes and is validated before Compose. - `--annotations` is a single UTF-8 YAML file, at most 16 MiB. `--reviewed-candidates` is mandatory with it and must equal the persisted candidate artifact digest. The pair is invalid without both flags. - `--output` is created exclusively with restrictive permissions after the returned workspace/run/digest identity has been verified. Existing files, symlinks, hardlinks, and Windows reparse targets are refused. - Existing harness `suggest-fks --write` is intentionally not exposed: an automatic merge is not a human review decision. +- Bootstrap recovery is automatic and installation-scoped; `workspace inspect` deliberately has no public bootstrap `--resume` selector. Inspect, lazy list, and status all invoke the same bounded `ensureBootstrapAddressed` selector under `repository.lock`; it rechecks valid/corrupt/absent active state before any job scan or bootstrap network operation, so queued stale-absence callers return `already_active`. - No unknown flag, passthrough separator, environment-selected command, shell fragment, or arbitrary container entrypoint is accepted. ## Public result and exit contract @@ -99,7 +107,7 @@ interface WorkspaceOperationResult { | "preprocessing_resume_mismatch" | "manual_review_required" | "evidence_materialization_required" | "effective_config_mismatch" | "semantic_index_incompatible" | "annotation_invalid" - | "egress_policy_refused"; + | "egress_policy_refused" | "registry_bootstrap_recovery_conflict"; workspaceId: string; workspaceRevision: string; descriptorBlob: string; @@ -114,9 +122,12 @@ interface WorkspaceOperationResult { ``` - Exit `0`: `succeeded`, `unchanged`, or `dry_run`. -- Exit `3`: expected operator checkpoint/block (`manual_review_required`, `evidence_materialization_required`, lock/revision conflict). +- Exit `3`: expected operator checkpoint/block (`manual_review_required`, `evidence_materialization_required`, lock/revision conflict, or `registry_bootstrap_recovery_conflict`). The Go JSON encoder preserves that exact code; human mode prints only `Bootstrap recovery is ambiguous or corrupt; inspect the installation registry jobs.` and never selects or prints a candidate/newest run ID. - Exit `2`: host grammar or unsafe local file error. - Exit `1`: operational failure. +- The existing lazy-list/status HTTP shapes stay unchanged; automatic recovery conflict is HTTP `409` + with `{ "code": "registry_bootstrap_recovery_conflict" }` and no run ID, filename, raw parser error, + repository path, or remote detail. - Stdout JSON maximum: 1 MiB. Sanitized stderr maximum: 64 KiB. Child stdout/stderr and every stage have explicit limits/timeouts. - Never return descriptor endpoints with credentials/query strings, secret contents or paths, signed URLs, raw SQL, rendered configuration, raw child stderr, arbitrary exception text, Qdrant payload contents, or host/container environment dumps. @@ -137,12 +148,18 @@ interface WorkspaceOperationResult { └── <32-hex-outer-run-id>.json ``` -- `writer.lock` is a regular `0600` file held by a Linux kernel advisory lock for the entire outer operation. The file may persist; the kernel lock is released on crash/container death. `inspect` never takes it. -- Lock order is always P2 workspace writer lock → existing harness stage lock. Harness code never acquires the P2 lock, preventing inversion/deadlock. +- `workspace-lock-root-lease.ts` is the sole producer of an exported, non-forgeable `VerifiedWorkspaceLockRootLease`. Its factory is bound at construction to the installation-derived sessions root, accepts only a factory-produced `CanonicalWorkspaceLockRootInput` for a canonical workspace ID (never a request path), opens every component with no-follow directory semantics, and retains the workspace-root directory FD plus device/inode identity until the reader or writer lifetime ends. A structurally forged/cast input, an input from another factory, a changed root inode, a closed/consumed lease, or a root replacement between derivation, open, and lock-file open fails `preprocessing_conflict`. Writer, session admission, route, and maintenance code consume the exact operations on this lease/capability contract rather than branding/casting strings or recovering the retained root. +- For a changed-set ID whose canonical leaf does not exist, only `VerifiedWorkspaceLockRootLeaseFactory.acquireOrProvision` may create it, and only while the caller already holds `repository.lock`. The factory retains and verifies the installation sessions-parent FD, takes its private parent provisioning lock, uses the repo-owned `workspace-fs-at` Node-API seam for literal `mkdirat(parentFd, workspaceId, 0700)` plus `openat(..., O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC)`, requires the exact service UID/mode, fsyncs the new leaf and verified parent, and handles `EEXIST` only by opening and revalidating the winning directory. Symlink, non-directory, wrong-owner/mode, parent replacement, or a different inode at any recheck fails. An unused canonical leaf created for a pull that later fails is deliberately retained; it contains no published workspace data and a later add/remove retry reuses and reverifies it. Ordinary existing-root callers continue to use `acquire`; registry changed-set acquisition uses `acquireOrProvision` for every ID, including absent added and never-used removed roots. +- The FD-relative seam is concrete and frozen: raw C++ source `backend/native/workspace-fs-at/workspace_fs_at.cc`, `binding.gyp` with `NAPI_VERSION=8`, type declaration `backend/src/native/workspace-fs-at-binding.d.ts`, sole TypeScript wrapper `backend/src/workspaces/workspace-fs-at.ts`, and build driver `backend/scripts/build-workspace-fs-at.mjs`. It exposes only typed `openat`, `mkdirat`, no-follow `fstatat`, directory `fsync`, explicit `close`, the closed lock-file name union, and typed owned-handle flock/child-stdio operations. The raw numeric FD borrow is module-private to `workspace-fs-at.ts` and is authorized only inside `flockOwnedLock` and `duplicateForChildStdio`; neither operation returns a number or passes one to its caller. It never accepts an arbitrary flags integer or multi-component relative name. `backend/package.json` and `backend/package-lock.json` pin `node-gyp@11.2.0`, require Node 22, and build the addon from committed source; no downloaded/prebuilt addon is accepted. +- `writer.lock` and the P3 handoff file `session-readers.lock` are the exact `LockFileName` values. Each is a regular `0600` file opened relative to the retained root FD by `workspace-fs-at` with `O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC` and verified through its `fstatat`/owned-FD stat result as a single-link regular file owned by the service UID. `flockOwnedLock` is the only TypeScript locking boundary: it synchronously borrows the owned regular-file FD inside the wrapper module and calls exact `fs-ext@2.1.1.flockSync` with the typed shared/exclusive plus blocking/nonblocking combination (`sh`, `ex`, `shnb`, or `exnb`). Writer acquisition uses exclusive/nonblocking and holds it through the entire outer operation. Before and after lock open/flock, the opener verifies that the canonical pathname still names the retained root device/inode. `fs-ext` is never credited with `openat`, `mkdirat`, `fstatat`, directory fsync, or close. `runUnderWorkspaceWriterLock` consumes one root lease. The sole multi-lock producer is `runUnderOrderedWorkspaceWriterLocks(leases, action)`: it rejects duplicate lease identities, sorts a private copy, acquires each writer lock in lexical workspace-ID order, retains every root and writer FD until `action(set)` settles, invalidates the opaque set and every capability before returning, and closes writer/root ownership in reverse order on success, throw, cancellation, or partial acquisition. Neither API permits nested public acquisition. +- `VerifiedWorkspaceLockRootLease.acquireSessionReadersShared()` is the only session-side reader-gate acquisition. It takes no name, path, handle, flags, FD, or mode parameter. It opens exactly `session-readers.lock` at `0600` through its retained root and the private `WorkspaceFsAtV1` instance, calls `flockOwnedLock(owned, "shared", "nonblocking")`, and repeats the root/path identity checks. After an open/flock/root-recheck failure, any opened lock is closed exactly once; if that cleanup succeeds, the root owner remains live so its caller can close or deliberately retry it. If cleanup close is uncertain, the method instead invalidates the root owner, attempts its close exactly once, and rejects fail-closed. Only after all checks and flock succeed does the method atomically consume the root owner and return a `WorkspaceSessionReadersLockLease` owning both the shared lock open description and retained root. That lease is the transferable/closable owner passed into the Pi session owner and remains held until every Pi/session child and stream has settled. `transfer()` requires a live owner, invalidates its source without closing either open description, and returns the sole new owner; `close()` on that transferred source is an idempotent no-op, while another transfer rejects. Owning `close()` invalidates first, is idempotent, attempts lock close and then root close exactly once even if the first close throws, and never retries an uncertain close. Acquisition/root-race/lock-contention/transfer-after-invalid failures and any close uncertainty reject as stable `preprocessing_conflict`; a failed acquisition never returns a lease, and a close failure leaves the lease invalid rather than reusable. +- `WorkspaceWriterLockCapability.runUnderSessionReadersExclusive(action)` is the only maintenance/registry-side reader-gate acquisition. It is callable only on the live callback-scoped writer capability, takes no lock argument, and through that capability's retained root and private `WorkspaceFsAtV1` opens exactly `session-readers.lock` at `0600`, then calls `flockOwnedLock(owned, "exclusive", "nonblocking")`. It rejects acquisition while a child spawn is already in flight and rejects nested/concurrent reader-gate acquisition; while `action` is live, the same capability's closed-union `spawnChild` remains usable and the reader lock is held through that child settlement. Contention or open/flock failure closes any opened lock once and never calls `action`. On success it passes only a private-constructor `BorrowedWorkspaceSessionReadersExclusiveLockLease` containing diagnostic workspace/root identities and `assertLive()`—never the directory handle/path/FD/lock name—holds the exclusive lock until the returned/raised action settles, invalidates the borrowed lease first, and closes the lock exactly once. The retained root stays owned by the outer writer capability. A callback rejection is rethrown after successful cleanup; close uncertainty instead rejects as stable `preprocessing_conflict`, retains the callback error only as a non-serialized cause, permanently poisons the writer capability, and forces its outer ordered owner to fail after completing writer/root cleanup. Captured leases fail `assertLive()` after settlement. Registry quiescence owners must call this operation rather than construct `BorrowedWorkspaceSessionReadersExclusiveLockLease` or open/flock independently. +- Every harness child that can migrate or publish is spawned only by `WorkspaceWriterLockCapability.spawnChild`. Internally, and without exposing either number or an accessor, it installs the *actual locked writer open file description* as child FD 3 and the same retained verified workspace-root directory FD as child FD 4 for every mutating child. The capability accepts only the closed, exhaustively switched `WorkspaceLockedChildRequest` union already validated by the coordinator; P3, P5, and P6 extend that same union with exact variants and no raw path/argv member. Python `require_workspace_writer_lock(cfg)` first validates FD 4 as the expected retained root directory, opens `preprocessing/writer.lock` relative to FD 4 with no-follow semantics, proves it is the same device/inode as FD 3, and calls `fcntl.flock(3, LOCK_EX|LOCK_NB)` before mutation. Missing/closed/substituted/cross-root FD 3 or FD 4, pathname replacement after acquisition, a forged marker, ordinary direct CLI invocation, post-settlement capability use, or a request for another workspace returns `preprocessing_conflict` before writes. There is no raw-root, ambient-root, direct-spawn, or general-process overload. +- Lock order is session admission: verified root lease → shared session-reader lease → Pi/session children; ordinary mutation: verified root lease → P2 workspace writer lock → quiescence/drain → `runUnderSessionReadersExclusive` → existing harness stage lock. Harness code verifies inherited writer/root FDs but never opens/acquires a different P2 lock, preventing inversion/deadlock. The addressed registry exception is repository lock → durable request claim → durable advertised target pin → exact-OID fetch into a run-specific immutable ref → immutable base/target plan → `runUnderOrderedWorkspaceWriterLocks` for the complete changed set → each changed workspace's quiescence and `runUnderSessionReadersExclusive` callback → participants → publication → terminal durability, all inside the ordered-owner callback. Nested exclusive callbacks acquire in the already-frozen lexical workspace order and then invalidate/close reader borrows in reverse order before quiescence/writers/roots and repository release. - Every directory component is opened/validated without following symlinks. State files are `0600`, written to an exclusive sibling, fsynced, renamed, and parent-fsynced. Hardlink count must be one. -- The deterministic config path fixes P2 same-revision `config_source` identity. Its manifest binds workspace, revision, descriptor blob, config SHA-256, file identity, and the current existing harness ownership binding. Same path + different bytes returns `effective_config_mismatch`; P3 introduces semantic cross-revision equivalence. -- Job state binds operation, revision, descriptor blob, config digest, non-secret binding identity, completed stage records, child run IDs, candidate/review digests, and terminal status. Resume revalidates all fields and reconciles a child publication that completed immediately before an outer-state crash. -- Before any schema/Evidence mutation, enumerate resumable session manifests for the workspace. A different pinned revision returns `preprocessing_conflict`; no write begins. This is the explicit P2 bridge until P3 revision isolation. +- The deterministic config path fixes P2 same-revision `config_source` identity. Its manifest binds workspace, revision, descriptor blob, config SHA-256, file identity, and the current existing harness ownership binding. Session and maintenance leases for equal inputs must have byte-identical YAML and byte-identical `config_dwh_binding()` output. Same path + different bytes returns `effective_config_mismatch`; P3 introduces semantic cross-revision equivalence. +- Job state binds operation, revision, descriptor blob, config digest, non-secret binding identity, completed stage records, child run IDs, candidate/review digests, and terminal status. Registry addressed state additionally owns the complete request, advertised OID, immutable-ref identity, full base/target plan, participant/synchronizer digests, publication bytes, and terminal result in P2; P3 consumes rather than reinvents that record. +- Before any schema/Evidence mutation, enumerate resumable session manifests for the workspace. A different pinned revision returns `preprocessing_conflict`; no write begins. Every product bootstrap caller—inspect, lazy list, and status—delegates to `WorkspaceRegistry.ensureBootstrapAddressed`; while continuously holding `repository.lock`, that one method first revalidates active state, returns `already_active` or fails on corruption before any job scan, and only for proven absence performs the bounded exact job scan and calls the same private addressed executor used by `publishAddressed`. Pull, author-publication activation, and internal active-pointer change delegate to `WorkspaceRegistry.publishAddressed`; both paths use the same callback-scoped `CapabilityAwareRegistryPublicationLifecycleOwner`, and no old `bootstrap`, `pull`, `activate`, or direct pointer writer remains callable. Bootstrap uses no base and all target IDs; pull/author publication use the exact base/target symmetric changed set. All registry state, root provisioning, quiescence/readers, participant/synchronizer work, pointer publication/reconciliation, and `terminal_durable` persistence occur before the ordered callback settles. This is the explicit P2 bridge and frozen P3–P6 handoff. ## One-shot service security contract @@ -151,8 +168,18 @@ interface WorkspaceOperationResult { - same exact selected core image, resolved to its immutable local image ID; generated final override uses that ID and `pull_policy: never`; - `docker compose run --rm --no-deps --no-TTY --name workspace-maintenance ...`; - no `build`, frontend, published port, Pi auth, Pi state, Pi trust initialization, Docker socket, home credential directory, or arbitrary command; -- non-root `10001`, `read_only: true`, `cap_drop: [ALL]`, `no-new-privileges:true`, restrictive tmpfs, registry active snapshots read-only, sessions root writable; -- only operation-required connector/Evidence secret files are mounted; AWS ambient environment is cleared; +- non-root `10001`, `read_only: true`, `cap_drop: [ALL]`, `no-new-privileges:true`, restrictive + tmpfs; the registry volume is writable only for inspect's empty-registry bootstrap and read-only + for every mutating operation; the sessions root is writable; +- only operation-required credentials are mounted: Git transport files only for the inspect + capability that may bootstrap an empty registry, DWH files only for DWH/full-run, and Evidence + files only for the selected Evidence/full-run source; AWS ambient environment is cleared; +- the exact installation-local setting is `THT_HTTP_PRIVATE_HOST_ALLOWLIST`, a comma-separated list + of at most 32 lower-case DNS hostnames, each at most 253 ASCII bytes. Empty means no private host. + Whitespace, empty entries, IP literals, wildcard/glob syntax, trailing dots, duplicates, IDNA + ambiguity, and non-canonical hostnames are refused before Compose. The same validated list is + passed to both core session rendering and maintenance rendering as + `egress.http_private_host_allowlist`, so their YAML and schema-v1 binding stay equal; - semantic commands require already-running healthy Qdrant/embedding services and do not start/stop them; DWH/inspect commands do not start dependencies; - exact operation labels and container identity are recorded. Cancellation terminates the process group, verifies the owned container labels/image, removes only that container, and preserves all pre-existing services, volumes, and networks; - fully rendered Compose is validated before launch, and post-run container/image identity is checked before accepting output. @@ -164,7 +191,8 @@ interface WorkspaceOperationResult { **Native host CLI** - `tools/thothctl/cmd/thothctl/main.go`, `main_test.go`: public grammar/help, dispatch, exit codes. -- Create `tools/thothctl/internal/workspaceops/operations.go`, `operations_test.go`: immutable image resolution, generated override, Compose run, cancellation cleanup, JSON validation. +- Create `tools/thothctl/internal/workspaceops/operations.go`, `operations_test.go`: immutable image resolution, generated override, bounded Compose run, cancellation cleanup, JSON validation. +- Modify `tools/thothctl/internal/compose/runner.go`; create/update platform process-group files and tests: streaming bounded capture and immediate process-group termination on overflow. - `tools/thothctl/internal/config/installation.go`, `installation_test.go`: maintenance service override and operation-specific binding discovery. - `tools/thothctl/internal/safeio/files.go`, platform files/tests: bounded no-follow input and exclusive output. - `tools/thothctl/internal/output/sanitize.go`, tests: bounded redaction. @@ -174,26 +202,35 @@ interface WorkspaceOperationResult { - `compose.yaml`: dedicated profile-gated service with shared image identity and least privilege. - `deploy/compose.local.yaml`, `deploy/compose.server.yaml`: correct registry/session storage semantics. -- `deploy/compose.git-https.yaml`, `deploy/compose.git-ssh.yaml`: do not attach Git credentials to P2 active-snapshot operations. +- `deploy/compose.git-https.yaml`, `deploy/compose.git-ssh.yaml`: expose validated Git transport metadata so the generated inspect-only maintenance override can attach exact credentials without copying them to mutating operations. - `scripts/generate-connector-secrets-override.sh`: operation-specific maintenance secrets. -- Create `docker/workspace-maintenance-entrypoint.sh`; modify `docker/core.Dockerfile`. +- Create `docker/workspace-maintenance-entrypoint.sh`; modify `docker/core.Dockerfile` with an explicit Linux native-compiler build stage, addon load smoke check, exact addon copy, and compiler-free runtime stage. - Retire/redirect fixture-only `deploy/compose.preprocess.yaml` as a non-public compatibility test path; do not leave two operator commands. **Shared Node operator** - Create `backend/src/workspaces/runtime-config-lease.ts`: shared snapshot read/render and deterministic protected config lease. - Modify `backend/src/tht/tht-runner.ts` to delegate session/operator rendering to the shared component without changing route behavior. -- Create `backend/src/workspaces/preprocessing-state.ts`: state schema, durable writes, locks, resume reconciliation. +- Create `backend/native/workspace-fs-at/workspace_fs_at.cc`, `backend/native/workspace-fs-at/binding.gyp`, `backend/src/native/workspace-fs-at-binding.d.ts`, `backend/src/workspaces/workspace-fs-at.ts`, and `backend/scripts/build-workspace-fs-at.mjs`: the repo-owned Node-API v8 FD-relative syscall seam and pinned build. +- Create `backend/src/workspaces/workspace-lock-root-lease.ts`: the sole canonical input/factory and retained-FD root-lease implementation, plus the P2-owned opaque transferable shared `session-readers.lock` owner, consuming only the typed `workspace-fs-at` wrapper. +- Create `backend/src/workspaces/preprocessing-state.ts`: state schema, durable writes, writer capability, the P2-owned callback-scoped exclusive `session-readers.lock` operation, inherited-FD verification contract, and resume reconciliation. Exact `fs-ext@2.1.1` remains imported only by `workspace-fs-at.ts` for typed kernel `flock`. +- Modify `backend/src/workspaces/registry.ts`: durable addressed bootstrap/pull/author-publication planning and all-or-nothing active-state publication; remove direct activation paths. +- Create `backend/src/workspaces/registry-publication.ts`: the single addressed request/job/export owner plus callback-scoped synchronizer/participant protocol; no lock reacquisition. +- Modify `backend/src/routes/workspaces.ts` and `backend/src/app.ts`: status, pull, lazy list/bootstrap, and dependency wiring all use the same addressed owner. +- Add tests: `backend/test/workspace-fs-at-native.test.ts`, `backend/test/workspace-lock-root-lease.test.ts`, `backend/test/workspace-session-readers-lock.test.ts`, `backend/test/workspace-registry-addressed-publication.test.ts`, `backend/test/workspace-registry-addressed-process.test.ts`, `backend/test/routes-workspaces.test.ts`, `backend/test/app.test.ts`, `backend/test/registry-pull-job-exports.test.ts`, `backend/test/fixtures/workspace-fs-at-race-worker.mjs`, `backend/test/fixtures/workspace-lock-root-worker.mjs`, `backend/test/fixtures/workspace-session-readers-worker.mjs`, and `backend/test/fixtures/workspace-registry-addressed-worker.mjs`. +- Modify `backend/package.json` and `backend/package-lock.json`: pin `node-gyp@11.2.0`, exact `fs-ext@2.1.1` for `flock(2)` only, the Node 22 engine/build scripts, and the committed addon build inputs. - Create `backend/src/workspaces/preprocessing-service.ts`: closed stage coordinator and security preflights. - Create `backend/src/workspace-maintenance.ts`: compiled stdin/argv entrypoint and pristine result encoder. - Add tests: `backend/test/workspace-runtime-config-lease.test.ts`, `workspace-preprocessing-state.test.ts`, `workspace-preprocessing-service.test.ts`, `workspace-maintenance.test.ts`. **Harness machine contracts** -- `harness/tht/cli/preprocess_cmd.py`: authoritative runtime workspace identity; require-existing collection mode. +- `harness/tht/cli/preprocess_cmd.py`: authoritative runtime workspace identity; require-existing collection mode; locked-FD verification before publishing operations. - `harness/tht/cli/schema_cmd.py`: extracted deterministic helpers, JSON suggest/check, safe SQL staging and annotation validation. - `harness/tht/cli/vector_cmd.py`: JSON schema-index result. -- `harness/tht/adapters/vector/qdrant.py`: explicit non-creating strict mode for P2. +- `harness/tht/config.py`, `harness/tht/adapters/factory.py`, `harness/tht/adapters/evidence/http.py`: exact rendered HTTP allowlist, adapter wiring, per-hop DNS/peer enforcement. +- `backend/src/workspaces/runtime-renderer.ts`: render the same collection lifecycle and egress policy for session and maintenance configs. +- `harness/tht/adapters/vector/qdrant.py`: explicit non-creating strict mode for every registry-rendered config. - Tests: `harness/tests/test_preprocess_cli.py`, `test_schema_fk_annotations.py`, `test_qdrant_cli_commands.py`, `test_registry_evidence_config.py`, `test_http_evidence_source.py`, plus new focused security cases. **Docs and gates** @@ -206,6 +243,763 @@ interface WorkspaceOperationResult { --- +## Frozen implementation interfaces + +These names are part of P2's handoff to P3–P6; implementation must not introduce a parallel +operator, renderer, state store, or lock under different names. + +```go +// tools/thothctl/internal/workspaceops/operations.go +type Command interface { workspaceCommand() } +type InspectCommand struct { WorkspaceID string; JSON bool } // no bootstrap run-ID member +const CodeRegistryBootstrapRecoveryConflict = "registry_bootstrap_recovery_conflict" +func ParseWorkspaceCommand(args []string) (Command, error) +func Run(ctx context.Context, installation config.Installation, runner compose.Runner, + command Command, stdin io.Reader) (Result, error) + +// tools/thothctl/internal/compose/runner.go +type CaptureLimits struct { StdoutBytes int64; StderrBytes int64 } +func (r Runner) RunBounded(ctx context.Context, args []string, stdin io.Reader, + limits CaptureLimits) (Result, error) + +// tools/thothctl/internal/safeio/files.go plus platform implementations +func ReadCanonicalUTF8(path string, maximum int64) ([]byte, error) +func WriteCanonicalExclusive(path string, contents []byte, mode fs.FileMode) error +``` + +`ParseWorkspaceCommand` is the only translator from host argv to the closed command union. `Run` +may build only the fixed Compose invocation for the corresponding union member. It must not accept +raw child argv. `workspaceops.Run` must call `RunBounded` with exactly 1,048,576 stdout bytes and +65,536 stderr bytes. `RunBounded` streams into fixed-capacity collectors and, on the first byte over +either limit, cancels and terminates the complete owned process group before returning +`compose.ErrOutputLimit`; it never buffers the overflow tail. Existing `Runner.Run` remains for +unrelated callers but is forbidden at the workspace public boundary. `ReadCanonicalUTF8` reuses the +existing component-by-component no-follow reader and adds UTF-8/identity rechecks; +`WriteCanonicalExclusive` refuses an existing leaf, hardlink/reparse ambiguity, and parent +replacement. + +```ts +// backend/src/native/workspace-fs-at-binding.d.ts -- the complete raw addon surface +declare const nativeWorkspaceFsAtHandleBrand: unique symbol; +declare const nativeWorkspaceFsAtComponentBrand: unique symbol; +export interface NativeWorkspaceFsAtHandleV1 { + readonly [nativeWorkspaceFsAtHandleBrand]: true; +} +export type NativeWorkspaceFsAtComponentV1 = string & { + readonly [nativeWorkspaceFsAtComponentBrand]: true; +}; +export interface NativeWorkspaceFsAtStatV1 { + readonly device: bigint; readonly inode: bigint; readonly mode: number; + readonly uid: number; readonly gid: number; readonly nlink: bigint; +} +export interface NativeWorkspaceFsAtErrorV1 extends Error { + readonly code: string; readonly errno: number; readonly syscall: + "openat" | "mkdirat" | "fstat" | "fstatat" | "fsync" | "fcntl" | "close"; +} +export interface NativeWorkspaceFsAtOpenResultV1 { + readonly handle: NativeWorkspaceFsAtHandleV1; + readonly openedStat: NativeWorkspaceFsAtStatV1; +} +export interface WorkspaceFsAtBindingV1 { + openat(input: { + readonly parent: NativeWorkspaceFsAtHandleV1 | null; + readonly name: "/" | NativeWorkspaceFsAtComponentV1; + readonly kind: "directory" | "regular_lock"; + readonly createMode: 0 | 0o600; + }): NativeWorkspaceFsAtOpenResultV1; + mkdirat(parent: NativeWorkspaceFsAtHandleV1, name: NativeWorkspaceFsAtComponentV1, mode: 0o700): void; + fstatat(parent: NativeWorkspaceFsAtHandleV1, name: NativeWorkspaceFsAtComponentV1): NativeWorkspaceFsAtStatV1; + fsyncDirectory(handle: NativeWorkspaceFsAtHandleV1): void; + close(handle: NativeWorkspaceFsAtHandleV1): void; + fdNumberForSynchronousBorrow(handle: NativeWorkspaceFsAtHandleV1): number; +} + +// backend/src/workspaces/workspace-fs-at.ts -- the only importer of the .node binding +import type { NativeWorkspaceFsAtStatV1 } from "../native/workspace-fs-at-binding.js"; +export interface WorkspaceFsAtStatV1 extends NativeWorkspaceFsAtStatV1 {} +export class OwnedWorkspaceFsAtDirectory { + private constructor(); + stat(): WorkspaceFsAtStatV1; + close(): void; +} +export class OwnedWorkspaceFsAtRegularFile { + private constructor(); + stat(): WorkspaceFsAtStatV1; + close(): void; +} +export type LockFileName = "writer.lock" | "session-readers.lock"; +export type WorkspaceFlockKindV1 = "shared" | "exclusive"; +export type WorkspaceFlockWaitV1 = "blocking" | "nonblocking"; +export class WorkspaceFsAtV1 { + openRoot(): OwnedWorkspaceFsAtDirectory; + openDirectoryAt(parent: OwnedWorkspaceFsAtDirectory, component: string): OwnedWorkspaceFsAtDirectory; + openOrCreateLockAt(parent: OwnedWorkspaceFsAtDirectory, component: LockFileName, + mode: 0o600): OwnedWorkspaceFsAtRegularFile; + mkdirAt(parent: OwnedWorkspaceFsAtDirectory, component: string, mode: 0o700): void; + statAtNoFollow(parent: OwnedWorkspaceFsAtDirectory, component: string): WorkspaceFsAtStatV1; + fsyncDirectory(directory: OwnedWorkspaceFsAtDirectory): void; + flockOwnedLock(owned: OwnedWorkspaceFsAtRegularFile, kind: WorkspaceFlockKindV1, + wait: WorkspaceFlockWaitV1): void; +} + +// backend/src/workspaces/runtime-config-lease.ts +export class WorkspaceRuntimeConfigLeaseFactory { + acquireSession(snapshotPath: string): RuntimeConfigLease; + acquireMaintenance(input: MaintenanceRuntimeInput): RuntimeConfigLease; +} + +// backend/src/workspaces/workspace-lock-root-lease.ts +declare const canonicalWorkspaceIdBrand: unique symbol; +declare const revision40Brand: unique symbol; +declare const sha256HexBrand: unique symbol; +export type CanonicalWorkspaceId = string & { readonly [canonicalWorkspaceIdBrand]: true }; +export type Revision40 = string & { readonly [revision40Brand]: true }; +export type Sha256Hex = string & { readonly [sha256HexBrand]: true }; +export interface WorkspaceLockRootIdentityV1 { + readonly schemaVersion: 1; + readonly workspaceId: CanonicalWorkspaceId; + readonly device: bigint; + readonly inode: bigint; +} +export class CanonicalWorkspaceLockRootInput { + private constructor(); + readonly workspaceId: CanonicalWorkspaceId; +} +export class BorrowedVerifiedWorkspaceLockRootLease { + private constructor(); + readonly identity: WorkspaceLockRootIdentityV1; +} +export class WorkspaceSessionReadersLockLease { + private constructor(); + readonly rootIdentity: WorkspaceLockRootIdentityV1; + transfer(): WorkspaceSessionReadersLockLease; + close(): Promise; +} +export class VerifiedWorkspaceLockRootLease { + private constructor(); + readonly identity: WorkspaceLockRootIdentityV1; + borrow(action: (borrowed: BorrowedVerifiedWorkspaceLockRootLease) => Promise): Promise; + acquireSessionReadersShared(): Promise; + transfer(): VerifiedWorkspaceLockRootLease; + close(): Promise; +} +export class VerifiedWorkspaceLockRootLeaseFactory { + constructor(input: { + readonly workspaceFsAt: WorkspaceFsAtV1; + readonly installationId: string; + readonly sessionsRootFromValidatedInstallationConfig: string; + readonly serviceUid: number; + readonly provisionedWorkspaceMode: 0o700; + }); + canonicalInput(workspaceId: string): CanonicalWorkspaceLockRootInput; + acquire(input: CanonicalWorkspaceLockRootInput): Promise; + acquireOrProvision( + input: CanonicalWorkspaceLockRootInput, + ): Promise; +} + +// backend/src/workspaces/preprocessing-state.ts +import type { RuntimeConfigLease } from "./runtime-config-lease.js"; +import type { + BorrowedVerifiedWorkspaceLockRootLease, + CanonicalWorkspaceId, + Revision40, + VerifiedWorkspaceLockRootLease, + WorkspaceLockRootIdentityV1, +} from "./workspace-lock-root-lease.js"; +export class PreprocessingStateStore { + create(input: CreateRunInput): Promise; + loadForResume(input: ResumeRunInput): Promise; + transition(runId: string, transition: RunTransition): Promise; + writeFkCandidate(runId: string, yaml: Uint8Array): Promise; + recordFkReview(runId: string, review: FkReviewInput): Promise; +} +interface WorkspaceLockedChildRequestBase { + readonly workspaceId: CanonicalWorkspaceId; + readonly revision: Revision40; + readonly rootIdentity: WorkspaceLockRootIdentityV1; + readonly runtimeConfig: RuntimeConfigLease; + readonly childRunId: string; +} +export interface DwhLockedChildRequest extends WorkspaceLockedChildRequestBase { + readonly kind: "dwh_preprocess"; + readonly stage: "introspect" | "lsh"; +} +export interface SchemaLockedChildRequest extends WorkspaceLockedChildRequestBase { + readonly kind: "schema_preprocess"; + readonly stage: "fk_suggest" | "fk_check" | "schema_index"; + readonly reviewedArtifact: ArtifactIdentity | null; +} +export interface EvidenceLockedChildRequest extends WorkspaceLockedChildRequestBase { + readonly kind: "evidence_preprocess"; + readonly stage: "http_publish"; +} +export type WorkspaceLockedChildRequest = + | DwhLockedChildRequest | SchemaLockedChildRequest | EvidenceLockedChildRequest; +export interface WorkspaceLockedChildResult { exitCode: number; stdout: Uint8Array; stderr: Uint8Array; } +export class BorrowedWorkspaceSessionReadersExclusiveLockLease { + private constructor(); + readonly workspaceId: CanonicalWorkspaceId; + readonly rootIdentity: WorkspaceLockRootIdentityV1; + assertLive(): void; +} +export class WorkspaceWriterLockCapability { + private constructor(); + readonly workspaceId: CanonicalWorkspaceId; + readonly rootIdentity: WorkspaceLockRootIdentityV1; + runUnderSessionReadersExclusive( + action: (lease: BorrowedWorkspaceSessionReadersExclusiveLockLease) => Promise, + ): Promise; + spawnChild(request: WorkspaceLockedChildRequest): Promise; +} +export interface BorrowedOrderedWorkspaceWriterLeaseV1 { + readonly workspaceId: CanonicalWorkspaceId; + readonly rootLease: BorrowedVerifiedWorkspaceLockRootLease; + readonly writerCapability: WorkspaceWriterLockCapability; +} +export class OrderedWorkspaceWriterCapabilitySet { + private constructor(); + readonly workspaceIds: readonly CanonicalWorkspaceId[]; + forWorkspace( + workspaceId: CanonicalWorkspaceId, + action: (lease: BorrowedOrderedWorkspaceWriterLeaseV1) => Promise, + ): Promise; + forEachWorkspace( + action: (lease: BorrowedOrderedWorkspaceWriterLeaseV1) => Promise, + ): Promise; +} +export function runUnderOrderedWorkspaceWriterLocks( + rootLeases: readonly VerifiedWorkspaceLockRootLease[], + action: (capabilities: OrderedWorkspaceWriterCapabilitySet) => Promise, +): Promise; +export function runUnderWorkspaceWriterLock( + rootLease: VerifiedWorkspaceLockRootLease, + action: (capability: WorkspaceWriterLockCapability) => Promise, +): Promise; +export function probeWorkspaceWriterLock( + rootLease: VerifiedWorkspaceLockRootLease, +): Promise<"available" | "held">; + +// backend/src/workspaces/registry-publication.ts +import type { + CanonicalWorkspaceId, + Revision40, + Sha256Hex, + VerifiedWorkspaceLockRootLeaseFactory, +} from "./workspace-lock-root-lease.js"; +import type { + BorrowedOrderedWorkspaceWriterLeaseV1, + BorrowedWorkspaceSessionReadersExclusiveLockLease, + OrderedWorkspaceWriterCapabilitySet, +} from "./preprocessing-state.js"; +declare const registryRunId32Brand: unique symbol; +export type RegistryRunId32 = string & { readonly [registryRunId32Brand]: true }; +export type RegistryAddressedOperationV1 = "registry_bootstrap" | "registry_pull"; +export type RegistryAddressedPublicationPhaseV1 = + | "request_claimed" | "target_advertised" | "target_fetched" | "planned" + | "participants_prepared" | "publication_intent_durable" + | "target_published" | "terminal_durable"; +export type RegistryAddressedJobArtifactPathV1 = + `addressed-publication-jobs/${RegistryRunId32}.json`; +export interface RegistryWorkspaceManifestIdentityV1 { + readonly workspaceId: CanonicalWorkspaceId; + readonly revision: Revision40; + readonly descriptorBlob: Revision40; + readonly manifestSha256: Sha256Hex; +} +export interface RegistryActiveSnapshotV1 { + readonly schemaVersion: 1; + readonly commit: Revision40; + readonly manifestSha256: Sha256Hex; + readonly workspaces: readonly RegistryWorkspaceManifestIdentityV1[]; +} +interface RegistryAddressedPlanFieldsV1 { + readonly schemaVersion: 1; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + readonly jobArtifactPath: RegistryAddressedJobArtifactPathV1; + readonly advertisedTargetCommit: Revision40; + readonly immutableTargetRef: `refs/thoth/addressed-runs/${RegistryRunId32}/target`; + readonly fetchedTargetCommit: Revision40; + readonly targetCommit: Revision40; + readonly targetManifestSha256: Sha256Hex; + readonly targetWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[]; + readonly changedWorkspaceIds: readonly CanonicalWorkspaceId[]; + readonly changedSetSha256: Sha256Hex; +} +export interface RegistryBootstrapAddressedPlanV1 extends RegistryAddressedPlanFieldsV1 { + readonly operation: "registry_bootstrap"; + readonly changedSetRule: "all_target_workspace_ids"; + readonly baseCommit: null; + readonly baseManifestSha256: null; + readonly baseWorkspaces: readonly []; +} +export interface RegistryPullAddressedPlanV1 extends RegistryAddressedPlanFieldsV1 { + readonly operation: "registry_pull"; + readonly changedSetRule: "symmetric_base_target_workspace_difference"; + readonly baseCommit: Revision40; + readonly baseManifestSha256: Sha256Hex; + readonly baseWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[]; +} +export type RegistryAddressedPlanV1 = + | RegistryBootstrapAddressedPlanV1 | RegistryPullAddressedPlanV1; +interface RegistryAddressedPublicationStateFieldsV1 { + readonly schemaVersion: 1; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly jobArtifactPath: RegistryAddressedJobArtifactPathV1; + readonly phase: RegistryAddressedPublicationPhaseV1; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + readonly advertisedTargetCommit: Revision40 | null; + readonly immutableTargetRef: `refs/thoth/addressed-runs/${RegistryRunId32}/target` | null; + readonly fetchedTargetCommit: Revision40 | null; + readonly targetCommit: Revision40 | null; + readonly targetManifestSha256: Sha256Hex | null; + readonly targetWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[] | null; + readonly changedWorkspaceIds: readonly CanonicalWorkspaceId[] | null; + readonly planSha256: Sha256Hex | null; + readonly changedSetSha256: Sha256Hex | null; + readonly participantsSha256: Sha256Hex | null; + readonly synchronizersSha256: Sha256Hex | null; + readonly publicationIntentSha256: Sha256Hex | null; + readonly publishedActiveStateSha256: Sha256Hex | null; + readonly terminalResultSha256: Sha256Hex | null; + readonly priorStateSha256: Sha256Hex | null; +} +export interface RegistryBootstrapAddressedPublicationStateV1 + extends RegistryAddressedPublicationStateFieldsV1 { + readonly operation: "registry_bootstrap"; + readonly baseCommit: null; + readonly baseManifestSha256: null; + readonly baseWorkspaces: readonly []; + readonly changedSetRule: "all_target_workspace_ids" | null; +} +export interface RegistryPullAddressedPublicationStateV1 + extends RegistryAddressedPublicationStateFieldsV1 { + readonly operation: "registry_pull"; + readonly baseCommit: Revision40; + readonly baseManifestSha256: Sha256Hex; + readonly baseWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[]; + readonly changedSetRule: "symmetric_base_target_workspace_difference" | null; +} +export type RegistryAddressedPublicationStateV1 = + | RegistryBootstrapAddressedPublicationStateV1 + | RegistryPullAddressedPublicationStateV1; +export class BorrowedWorkspaceMaintenanceQuiescenceLease { + private constructor(); + readonly workspaceId: CanonicalWorkspaceId; +} +export interface AddressedWorkspacePublicationLeaseV1 + extends BorrowedOrderedWorkspaceWriterLeaseV1 { + readonly quiescence: BorrowedWorkspaceMaintenanceQuiescenceLease; + readonly readers: BorrowedWorkspaceSessionReadersExclusiveLockLease; +} +export interface CapabilityAwareRegistryPublicationParticipant { + readonly participantId: string; + prepare( + plan: RegistryAddressedPlanV1, + workspace: AddressedWorkspacePublicationLeaseV1, + ): Promise; + reconcile( + plan: RegistryAddressedPlanV1, + workspace: AddressedWorkspacePublicationLeaseV1, + prepared: T, + phase: RegistryAddressedPublicationPhaseV1, + ): Promise; +} +export interface RegistrySynchronizerPreparedV1 { + readonly synchronizerId: string; + readonly preparedSha256: Sha256Hex; +} +export interface CapabilityAwareRegistryPublicationSynchronizer { + readonly synchronizerId: string; + ensureForPublication( + plan: RegistryAddressedPlanV1, + capabilities: OrderedWorkspaceWriterCapabilitySet, + phase: "planned" | "participants_prepared" | "publication_intent_durable" | "target_published", + ): Promise; +} +export class CapabilityAwareRegistryPublicationLifecycleOwner { + constructor(); + run(input: { + readonly plan: RegistryAddressedPlanV1; + readonly capabilities: OrderedWorkspaceWriterCapabilitySet; + readonly participants: readonly CapabilityAwareRegistryPublicationParticipant[]; + readonly synchronizers: readonly CapabilityAwareRegistryPublicationSynchronizer[]; + readonly action: () => Promise; + }): Promise; +} +export type RegistryAddressedRequestV1 = + | { + readonly mode: "create"; + readonly operation: "registry_bootstrap"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly expectedBaseCommit: null; + readonly remoteRefIdentitySha256: Sha256Hex; + } + | { + readonly mode: "resume"; + readonly operation: "registry_bootstrap"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + } + | { + readonly mode: "create"; + readonly operation: "registry_pull"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly expectedBaseCommit: Revision40; + readonly remoteRefIdentitySha256: Sha256Hex; + } + | { + readonly mode: "resume"; + readonly operation: "registry_pull"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + }; +export interface RegistryBootstrapAddressedResultV1 { + readonly operation: "registry_bootstrap"; + readonly runId: RegistryRunId32; + readonly jobArtifactPath: RegistryAddressedJobArtifactPathV1; + readonly plan: RegistryBootstrapAddressedPlanV1; + readonly planSha256: Sha256Hex; + readonly phase: "terminal_durable"; + readonly publication: "target" | "reconciled_target" | "unchanged"; +} +export interface RegistryPullAddressedResultV1 { + readonly operation: "registry_pull"; + readonly runId: RegistryRunId32; + readonly jobArtifactPath: RegistryAddressedJobArtifactPathV1; + readonly plan: RegistryPullAddressedPlanV1; + readonly planSha256: Sha256Hex; + readonly phase: "terminal_durable"; + readonly publication: "target" | "reconciled_target" | "unchanged"; +} +export type RegistryAddressedResultV1 = + | RegistryBootstrapAddressedResultV1 | RegistryPullAddressedResultV1; + +export interface RegistryBootstrapRecoveryIdentityV1 { + readonly operation: "registry_bootstrap"; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; +} +export type RegistryEnsureBootstrapAddressedResultV1 = + | { + readonly kind: "already_active"; + readonly snapshot: RegistryActiveSnapshotV1; + } + | { + readonly kind: "bootstrap_terminal"; + readonly snapshot: RegistryActiveSnapshotV1; + readonly result: RegistryBootstrapAddressedResultV1; + }; +export interface RegistryBootstrapRecoveryScanLimitsV1 { + readonly maximumDirectoryEntries: 4096; + readonly maximumArtifactBytes: 1048576; + readonly maximumTotalArtifactBytes: 67108864; +} + +// backend/src/workspaces/registry.ts +import type { VerifiedWorkspaceLockRootLeaseFactory } from "./workspace-lock-root-lease.js"; +import type { + CapabilityAwareRegistryPublicationLifecycleOwner, + CapabilityAwareRegistryPublicationParticipant, + CapabilityAwareRegistryPublicationSynchronizer, + RegistryAddressedRequestV1, + RegistryAddressedResultV1, + RegistryEnsureBootstrapAddressedResultV1, + RegistryBootstrapRecoveryIdentityV1, +} from "./registry-publication.js"; +export class WorkspaceRegistry { + constructor(input: { + readonly rootLeaseFactory: VerifiedWorkspaceLockRootLeaseFactory; + readonly lifecycleOwner: CapabilityAwareRegistryPublicationLifecycleOwner; + readonly participants: readonly CapabilityAwareRegistryPublicationParticipant[]; + readonly synchronizers: readonly CapabilityAwareRegistryPublicationSynchronizer[]; + }); + ensureBootstrapAddressed( + identity: RegistryBootstrapRecoveryIdentityV1, + ): Promise; + publishAddressed(request: RegistryAddressedRequestV1): Promise; +} + +// backend/src/workspaces/preprocessing-service.ts +export class WorkspacePreprocessingService { + execute(command: MaintenanceCommand, ingress: MaintenanceIngress): Promise; +} + +// backend/src/workspace-maintenance.ts +export async function main( + argv: readonly string[], stdin: NodeJS.ReadableStream, + stdout: NodeJS.WritableStream, stderr: NodeJS.WritableStream, +): Promise; +``` + +`workspace_fs_at.cc` uses only the stable C Node-API at pinned `NAPI_VERSION=8`; `binding.gyp` +compiles it as C++17 on Node 22 and has an explicit `OS=="linux" or OS=="mac"` condition, with every +other target rejected. `build-workspace-fs-at.mjs` rejects a non-22 Node, requires +`process.versions.napi >= 8`, locates already-installed Node 22 headers (or an explicit +`npm_config_nodedir`), invokes exact lockfile `node-gyp@11.2.0`, and verifies that the sole output is +`native/workspace-fs-at/build/Release/workspace_fs_at.node`; it never downloads headers or a binary. +`package.json` defines `build:native` as this driver, `build:ts` as `tsc -p tsconfig.json`, `build` as +`npm run build:native && npm run build:ts`, and `test:native` as the driver followed by the native +Vitest file. The wrapper is the only `.node` importer and checks the frozen export names at load. + +The binding owns every returned descriptor in a native handle with `FD_CLOEXEC`; each successful +`openat` immediately `fstat`s the opened description and atomically returns its immutable handle plus +that stat identity, closing on any conversion failure. `openat` admits a null parent if and only if +opening root `/` as a directory; otherwise it requires an owned directory parent and one validated +single canonical component. It admits only kind `directory` (fixed +`O_RDONLY|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC`) or `regular_lock` (fixed +`O_RDWR|O_CREAT|O_NOFOLLOW|O_CLOEXEC`, mode `0600`); the wrapper further restricts regular-lock names to +exact `LockFileName = "writer.lock" | "session-readers.lock"` and requires literal mode `0o600`. +`mkdirat` admits only a single canonical component and mode `0700`; `fstatat` always uses +`AT_SYMLINK_NOFOLLOW`; directory fsync checks the handle type; and `close` consumes it exactly once. The +wrapper rejects empty, `.`, `..`, slash-containing, NUL, non-UTF-8, over-255-byte, wrong-kind, +cross-addon, closed, and concurrent-close inputs before a syscall, and does not export the raw binding, +an FD accessor, or the numeric-borrow callback. A synchronous non-reentrant numeric borrow exists only +inside the wrapper module and only `flockOwnedLock` and `duplicateForChildStdio` may invoke it; close +cannot run during either borrow. `flockOwnedLock` accepts only an owned regular-lock handle plus typed +`"shared" | "exclusive"` and `"blocking" | "nonblocking"`, maps those four combinations exactly to +`fs-ext.flockSync(fd, "sh" | "ex" | "shnb" | "exnb")`, and never exposes `fd`. The child duplication +helper likewise exposes no number and only installs the owned writer/root descriptions into the fixed +child FD 3/4 slots. Native finalization is leak containment only, never a successful durability path. + +All syscalls except `close` retry `EINTR`. Native errors are ordinary `Error` objects with stable +`code`, numeric `errno`, and allowlisted `syscall`, with no supplied path text; the wrapper maps expected +filesystem races to the frozen conflict code without hiding the errno class. `EEXIST` is returned to +the factory for the explicit winning-creator branch. Close marks ownership consumed before calling the +kernel and is never retried; an `EINTR`/uncertain close becomes +`ERR_WORKSPACE_FS_AT_CLOSE_UNCERTAIN` and fails the outer operation so a recycled descriptor cannot be +closed. Double close/borrow-after-close is `ERR_WORKSPACE_FS_AT_HANDLE_CLOSED`; wrong handle/type is +`ERR_WORKSPACE_FS_AT_ARGUMENT`. + +Linux uses literal `openat(2)`, `mkdirat(2)`, `fstatat(2)`, and `fsync(2)`. Darwin uses the same POSIX +calls and flags; directory durability first calls `fsync`, then only for `EINVAL`/`ENOTSUP` calls +`fcntl(F_FULLFSYNC)`. If both are unsupported it returns stable +`ERR_WORKSPACE_FS_AT_DIRECTORY_FSYNC_UNSUPPORTED` and the root factory fails closed—there is no fake +success. Linux and Darwin native tests run against the host temporary filesystem and assert either +successful durable directory sync or that exact Darwin fail-closed result; the Linux container gate +requires success. Windows neither builds nor loads this backend addon; the supported Windows Go CLI +continues to target the Linux core image. + +`docker/core.Dockerfile` names the compiler stage exactly `backend-native-build`: from the pinned Node +22 Bookworm digest it installs only `python3`, `make`, and `g++`, runs `npm ci`, `npm run build:native`, +and the Linux native test. The following `backend-build` stage starts again from the same clean pinned +Node image, copies the locked `node_modules` and exact `.node` output from `backend-native-build`, invokes no +native compiler and runs `npm run build:ts`, then `npm prune --omit=dev --ignore-scripts` so node-gyp +and build-only packages are absent while the already-built exact `fs-ext` binding remains. The final +Python-slim `runtime` copies `dist`, the pruned runtime `node_modules`, and only +`native/workspace-fs-at/build/Release/workspace_fs_at.node` at that same relative path; it does not +copy node-gyp, headers, source, make, g++, or compiler-stage caches. Image build checks Node major 22, +N-API >=8, loads both `workspace_fs_at.node` and `fs-ext`, and invokes one temp-directory Linux fsync +smoke test as UID 10001 before accepting the image. + +`WorkspaceRuntimeConfigLeaseFactory` is the sole reader/binder/renderer handoff for both +`ThtRunner.acquireWorkspaceRuntime` and maintenance. `VerifiedWorkspaceLockRootLeaseFactory` is the +sole root producer and receives the sole `WorkspaceFsAtV1`. Its constructor no-follow opens and retains the installation sessions-parent FD; +`canonicalInput` accepts a workspace ID, not a path, and returns a private-constructor value carrying +factory provenance. `acquire` opens an existing canonical leaf. `acquireOrProvision` is the only +missing-leaf path and is callable by registry publication only while `repository.lock` is held: under +one factory-private parent provisioning lock it performs anchored +`mkdirat(parentFd, workspaceId, 0700)`, opens with `O_DIRECTORY|O_NOFOLLOW`, requires UID exactly equal +to `serviceUid` and mode exactly `0700`, fsyncs the created leaf and retained parent, and rechecks +parent plus leaf device/inode before returning. `EEXIST` means a concurrent creator won and is accepted +only after the same open/owner/mode/identity checks. Symlink/non-directory/wrong UID or mode, parent +replacement, or leaf substitution fails closed. A provisioned unused leaf is retained on every later +failure, including failure before publication; retry reverifies and reuses it. Both absent added IDs +and absent never-used removed IDs follow this rule. + +The public root identity is diagnostic data only and cannot reopen the FD. `borrow` is callback-scoped +and cannot outlive the owning handle. `transfer` requires zero active borrows, atomically invalidates +the source, and returns the only new owner. `close` is idempotent, waits for current borrows, then closes +the FD; borrow/transfer after close or transfer fails `preprocessing_conflict`. Path-to-retained-inode +checks run before and after every relative open. `acquireSessionReadersShared()` is the one additional +consuming operation: with no caller-selected operand it uses the retained root to open the literal +`session-readers.lock` at `0600`, acquire typed shared/nonblocking flock, and on success move root plus +lock ownership into `WorkspaceSessionReadersLockLease`. That lease alone may transfer into the Pi +session owner; it remains live through child/stream teardown and closes lock then root exactly once. + +`runUnderOrderedWorkspaceWriterLocks(rootLeases, action)` is the sole producer of an ordered +capability set and the sole constructor path for `WorkspaceWriterLockCapability`. +It consumes all root leases, rejects duplicate workspace identities, sorts a private copy by canonical +workspace ID, opens each relative `writer.lock` with `openOrCreateLockAt(..., 0o600)` and invokes +`flockOwnedLock(owned, "exclusive", "nonblocking")` in lexical order without calling or nesting the +single-lock public API, and enters `action(set)` only after all roots and writer locks are held. +The set, every borrowed workspace lease, and every capability are valid only during that callback. +On success, throw, cancellation, or partial acquisition it invalidates them first and closes writer FDs +and owned root leases in exact reverse lexical order. `runUnderWorkspaceWriterLock` is only the +one-element adapter over this function; it adds no producer or acquisition path. `probeWorkspaceWriterLock` +likewise consumes and closes its lease. + +Each `WorkspaceWriterLockCapability` owns (rather than merely borrows) its transferred retained root +lease for its whole lifetime. Its `runUnderSessionReadersExclusive(action)` method is the only consumer +of that retained root for the reader gate: it opens the literal `session-readers.lock` at `0600`, takes +typed exclusive/nonblocking flock, supplies only a callback-scoped +`BorrowedWorkspaceSessionReadersExclusiveLockLease`, and invalidates then closes that lock when the +action settles. `spawnChild` accepts only the frozen discriminated union, checks request +workspace ID, revision-bound runtime-config lease, and root device/inode against the capability, and +rejects post-settlement/concurrent/cross-workspace use. It passes the actual locked writer open file +description as FD 3 and the same retained root directory open file description as FD 4 to **every** +mutating harness child. Python first requires both descriptors, verifies FD 4 is the expected owned +root directory, opens `preprocessing/writer.lock` relative to FD 4 with no-follow semantics, proves that +file's device/inode equals FD 3, verifies FD 3 is a single-link `0600` service-owned regular file, and +requires `flock(3, LOCK_EX|LOCK_NB)` to report already held before any mutation. Missing, closed, +renumbered, substituted, cross-root, or independently locked FD 3/4 fails `preprocessing_conflict`. +No environment marker is authorization. P3/P5/P6 may add exact variants only to this same +`WorkspaceLockedChildRequest` alias; they may not add another capability, raw argv/path/stdio member, +ambient lookup, direct process spawn, or general escape hatch. + +The registry-root-relative durable artifact is exactly +`addressed-publication-jobs/<32-hex-run-id>.json`; it is created with `0600` sibling-write/file-fsync/ +rename/parent-fsync semantics. `WorkspaceRegistry.publishAddressed` accepts the exact +`registry_bootstrap`/`registry_pull` request union above. Bootstrap requires no active base, persists +`baseCommit`/`baseManifestSha256` as `null` and `baseWorkspaces` as `[]`, and defines +`changedWorkspaceIds` as every target workspace ID in lexical order. Pull requires the request's exact +active base and uses the lexical symmetric difference of canonical base/target workspace identities. +Pull and author publication call `publishAddressed`; inspect, lazy list bootstrap, and status call only +`ensureBootstrapAddressed`. Both enter one private addressed executor under the same repository-lock +ownership; the old public/internal `bootstrap`, `pull`, `activate`, and direct active-pointer writer are +removed, not retained as alternate publication paths. + +`ensureBootstrapAddressed(identity)` acquires `repository.lock` before its authoritative inspection and +retains it without a gap through selection, claim/resume, all network work, publication, and terminal +durability. Its first action under that lock is to reread and strictly validate the active pointer and +referenced immutable snapshot through the same active-state reader used by ordinary registry reads. +A valid active state returns `{ kind: "already_active", snapshot }` immediately, before opening or +scanning the jobs directory and without network; any caller-side absence check is only an optimization. +A present but malformed, incompatible, missing-target, identity-mismatched, or otherwise corrupt active +state fails closed with `registry_bootstrap_recovery_conflict` and neither scans jobs nor bootstraps. +Only a strictly proven absent pointer proceeds. It then opens exactly the registry-root-relative +`addressed-publication-jobs/` directory no-follow and reads one lexically sorted initial snapshot capped +at 4,096 directory entries. Durable writers use only final +`[0-9a-f]{32}\.json` and deterministic sibling `.[0-9a-f]{32}.json.tmp` names. Before candidate +selection, a sibling is unlinked and the directory fsynced only after proving it is a single-link +service-owned `0600` regular file and that its encoded run ID has at most that one final; this is the +exact pre-claim/interrupted-transition cleanup, not a candidate. A malformed/cross-owned/hardlinked or +duplicate sibling fails closed. Final files must be single-link service-owned `0600` regular files; +each is capped at 1,048,576 bytes and total final bytes read at 67,108,864, with identity rechecks after +every read. After deterministic sibling cleanup and final-file reads, it performs one second bounded +sorted enumeration and requires byte-identical final names and identities to the expected finals. +Overflow, entry churn between enumeration/recheck, any other entry name/type, invalid +JSON/schema/hash/phase, filename/run-ID disagreement, or duplicate run ID returns stable +`registry_bootstrap_recovery_conflict`; it never truncates, skips, quarantines, or guesses. + +The selector validates every record first, then ignores valid `terminal_durable` records for automatic +nonterminal selection. A direct exact same-ID addressed resume may replay such a terminal record only +when its stored operation, request hash, installation identity, repository identity, and remote-ref +identity all equal the caller's exact identity; replay is stored-result-only and performs no network. +For automatic empty-registry recovery: zero nonterminal addressed records creates one fresh 32-hex ID +and its exact create request; exactly one nonterminal record resumes that exact run ID before any +network only if it is `registry_bootstrap` and all five operation/request/installation/repository/remote +identity fields match; multiple nonterminal records, a nonterminal pull, or any identity mismatch fails +with `registry_bootstrap_recovery_conflict`. Terminal records are never ranked against nonterminals. +Successful create/resume strictly rereads the published active pointer/snapshot while the lock remains +held and returns `{ kind: "bootstrap_terminal", result, snapshot }`; that snapshot has the same commit, +manifest digest, and ordered workspace identities as the durable result plan. Thus two or more callers +that all observed absence before queueing converge: the first publishes once, while each later lock +holder takes `already_active` before a job scan, terminal ranking, run creation, or network, and every +caller consumes the exact same validated active snapshot. There is no mtime, lexical-last, newest, +advertised-OID, or remote-head heuristic. + +The bootstrap `requestSha256` is the canonical digest of schema version, operation, installation +identity, repository identity, and remote-ref identity—never the random run ID or mode—so create and +resume compare the same request. The installation identity is derived once from the validated +installation descriptor and registry volume identity; repository and remote-ref identities use the +same canonical functions as addressed publication. Every create state and plan persists all three +identity digests, and resume requires byte-equality before reconciliation. + +With `repository.lock` held, create first writes and parent-fsyncs phase `request_claimed`, including +the operation, request hash, installation/repository/remote-ref identities, artifact path, and the +complete base fields, **before any `ls-remote`, fetch, or other network call**. Advertisement then reads the remote +ref once and durably records its 40-hex OID as `advertisedTargetCommit` at phase +`target_advertised`. Only afterward may exact-OID fetch write +`refs/thoth/addressed-runs//target`; that ref is create-only and immutable for the run. The +implementation verifies the ref resolves byte-for-byte to the advertised OID and durably records the +same OID as `fetchedTargetCommit` at phase `target_fetched` before reading the target manifest and +persisting the full base/target/changed plan at `planned`. `planSha256` covers the operation, installation/repository/remote-ref identities, artifact path, +base fields, advertised OID, immutable-ref identity, fetched/target fields, and changed set. + +Pre-pin recovery is explicit. Before `request_claimed` is durable no network was allowed and a repeated +same request may recreate the claim. From durable `request_claimed` until `target_advertised` is +durable, no target has been promised: same-ID resume repeats advertisement and may observe a newer OID. +Once `target_advertised` is durable, that OID is permanent. Death before/during its exact-OID fetch +retries only that OID if the immutable ref is absent; if the ref already exists at that OID, resume +performs no network and advances to `target_fetched`; any different ref OID is terminal corruption. +Death after `target_fetched` never fetches or re-advertises. A terminal job always replays its stored +result even if a later independent pull moved remote-tracking refs; drift checks apply only while +reconciling a nonterminal job. Kill tests cover before claim rename, after claim fsync, during/after +advertisement but before its durable pin, after advertisement fsync, during/after exact fetch, after +immutable-ref creation but before `target_fetched`, after its fsync, and every later publication phase. + +After `planned`, the registry acquires/provisions one root lease per complete changed ID, passes all of +them once to `runUnderOrderedWorkspaceWriterLocks`, and performs quiescence, exclusive-reader gates, +participant preparation/reconciliation, synchronizers, pointer publication, and terminal persistence +inside that one callback and the same production +`CapabilityAwareRegistryPublicationLifecycleOwner`. For each already-ordered capability the owner calls +only `runUnderSessionReadersExclusive`; it never opens/flocks a reader file itself, and +`AddressedWorkspacePublicationLeaseV1.readers` is that method's exact +`BorrowedWorkspaceSessionReadersExclusiveLockLease`. Participants receive only callback-borrowed +addressed leases, synchronizers receive the same callback-scoped opaque set, and neither can reacquire +locks. Any failure invalidates borrows and closes readers/quiescence and then writers/roots in reverse +lexical order, leaving active state at exact base (or absent for bootstrap) before repository-lock +release. A compile/runtime test uses the production lifecycle owner—not a fake—to hold A and B +simultaneously and proves neither capability survives callback settlement. + +All participant `prepare` calls and durable digests complete before `publication_intent_durable`. +Publication stages immutable target workspace records, then performs one sibling write + file fsync + +atomic rename of the installation-wide active-snapshot pointer + parent fsync. Readers resolve only the +pointer, so staged or crash-left projection records are never a partial active publication. A +synchronous no-follow reread of pointer and target snapshot must be byte-equal before phase +`target_published`. Before pointer rename, active remains exact base/absent; at or after the boundary, +reconcile all-base/all-target/mixed exact projections deterministically to target and fail closed on a +third identity. `terminal_durable` is written before the ordered callback settles. +`WorkspacePreprocessingService.execute` remains the ordinary single-workspace coordinator. `main` +parses and encodes one result only; it never starts Fastify. +The harness keeps Typer wrappers thin over these testable helpers: + +```python +# harness/tht/cli/schema_cmd.py +def suggest_fks(config: Path, *, sql_roots: tuple[Path, ...], assumptions: tuple[str, ...]) -> dict: ... +def check_annotations(config: Path, *, annotations: Path | None = None) -> dict: ... + +# harness/tht/cli/vector_cmd.py +def index_schema(config: Path) -> dict: ... + +# harness/tht/cli/preprocess_cmd.py +# run_from_config and run_dwh_from_config remain the engine entrypoints, but both derive +# workspace identity through workspace_id_for_config(cfg, config). +``` + +Host ingress is one bounded schema-v1 JSON document on Compose stdin. SQL/annotation bytes are +base64 fields with a declared SHA-256 and logical basename; Node verifies both before exclusive +staging under the owned run root. Candidate export is an internal bounded +`hostExport: {mediaType, sha256, contentBase64}` field emitted only by `schema suggest-fks`. +Candidate YAML is capped at exactly 700 KiB (716,800 decoded bytes); its base64 plus the complete +schema-v1 envelope must remain at or below the 1 MiB stdout limit, otherwise the operation fails +before host output publication. +`workspaceops.Run` verifies workspace/revision/run/digest, writes `--output` exclusively, removes +`hostExport`, and only then re-encodes the public result. Raw SQL, annotation bytes, and base64 are +never public JSON, human output, state JSON, argv, Compose config, or retained logs. + +--- + ### Task 1: Freeze the native CLI, file-ingress, and result contracts **Files:** @@ -215,16 +1009,31 @@ interface WorkspaceOperationResult { - Modify platform-specific safe-I/O tests - Create: `tools/thothctl/internal/workspaceops/operations.go` - Create: `tools/thothctl/internal/workspaceops/operations_test.go` +- Modify: `tools/thothctl/internal/compose/runner.go` +- Create/modify: `tools/thothctl/internal/compose/runner_test.go` and platform process-group helpers/tests - Create: `docs/contracts/workspace-preprocessing-cli.md` -- [ ] **Step 1: Write RED parser-table tests** for every valid command above and for duplicate/missing/unknown flags, invalid IDs, incompatible annotation flags, option-count/size limits, and passthrough/shell attempts. +- [ ] **Step 1: Write RED parser-table tests** for every valid command above and for duplicate/missing/unknown flags, invalid IDs, incompatible annotation flags, option-count/size limits, and passthrough/shell attempts. Assert inspect rejects `--resume`, `--bootstrap-run-id`, and any equivalent spelling; automatic bootstrap recovery is never host-selected. - [ ] **Step 2: Run** `cd tools/thothctl && go test ./cmd/thothctl ./internal/workspaceops -run 'Workspace|workspace' -v` and verify the new tests fail because `workspace` is unknown. -- [ ] **Step 3: Add closed request types** (`InspectRequest`, `DwhRequest`, `SuggestFksRequest`, `CheckSchemaRequest`, `IndexSchemaRequest`, `EvidenceRequest`, `RunRequest`) and a parser that cannot represent arbitrary argv. +- [ ] **Step 3: Add closed request types** (`InspectCommand` exactly as frozen above, `DwhRequest`, `SuggestFksRequest`, `CheckSchemaRequest`, `IndexSchemaRequest`, `EvidenceRequest`, `RunRequest`) and a parser that cannot represent arbitrary argv or a bootstrap run ID. - [ ] **Step 4: Write RED safe-I/O tests** for symlinks, hardlinks, directory input, replacement during read, Windows reparse points, existing output, >1 MiB SQL, >16 MiB total, and non-UTF-8 annotation input. - [ ] **Step 5: Implement bounded reads and exclusive restrictive output** using existing platform seams; return only generic file errors. -- [ ] **Step 6: Add schema-v1 stdin request/result validation** with exact field allowlists and output bounds. -- [ ] **Step 7: Run focused Go tests and `gofmt -w`**, then `go test ./...`. -- [ ] **Step 8: Commit:** `feat: define P2 host workspace command contract`. +- [ ] **Step 6: Write RED bounded-runner tests** for exactly-at-limit output, one-byte-over stdout/stderr, infinite stdout, infinite stderr, context cancellation, and a grandchild that keeps writing. Assert fixed-capacity capture, immediate owned process-group termination, and `compose.ErrOutputLimit`; no test may retain more than the configured limit. +- [ ] **Step 7: Implement `Runner.RunBounded`** as the frozen streaming API above and require `workspaceops.Run` to use it. Add schema-v1 stdin/result validation with exact field allowlists, the exact 700 KiB candidate export cap, and complete-envelope bound. +- [ ] **Step 8: Run GREEN native tests:** + +```bash +cd tools/thothctl +gofmt -w cmd/thothctl/main.go cmd/thothctl/main_test.go \ + internal/workspaceops/operations.go internal/workspaceops/operations_test.go \ + internal/compose/*.go internal/safeio/*.go +go test ./cmd/thothctl ./internal/workspaceops ./internal/safeio ./internal/compose -run 'Workspace|workspace|Canonical|Exclusive|Bounded|OutputLimit' -v +go test ./... +``` + +Expected: focused and complete Go suites PASS; unsafe-file cases return usage exit 2 and never invoke +the fake Compose runner. +- [ ] **Step 9: Commit:** `feat: define P2 host workspace command contract`. ### Task 2: Add pristine harness JSON interfaces without changing the engine @@ -253,14 +1062,22 @@ Expected: FAIL only on the new machine-contract assertions. - [ ] **Step 5: Extract pure helpers** returning typed dictionaries/models; keep existing human commands as renderers over the same helpers. - [ ] **Step 6: Implement the JSON flags and authoritative workspace identity**. Catch expected exceptions and emit stable safe codes; never serialize arbitrary exception text. -- [ ] **Step 7: Run the three focused files and touched Ruff**: +- [ ] **Step 7: Run GREEN harness gates:** ```bash +cd harness +.venv/bin/pytest -q \ + tests/test_schema_fk_annotations.py \ + tests/test_qdrant_cli_commands.py \ + tests/test_preprocess_cli.py .venv/bin/ruff check \ tht/cli/schema_cmd.py tht/cli/vector_cmd.py tht/cli/preprocess_cmd.py \ tests/test_schema_fk_annotations.py tests/test_qdrant_cli_commands.py tests/test_preprocess_cli.py ``` +Expected: all three pytest files PASS and Ruff exits 0; captured JSON stdout parses as exactly one +document in every new case. + - [ ] **Step 8: Commit:** `feat: add P2 harness machine contracts`. ### Task 3: Add a non-creating semantic writer mode @@ -273,12 +1090,38 @@ Expected: FAIL only on the new machine-contract assertions. - Modify: `harness/tests/test_qdrant_cli_commands.py` - Modify: `harness/tests/test_registry_evidence_config.py` -- [ ] **Step 1: Write RED tests** proving operator mode refuses a missing collection without issuing create/index mutations, refuses wrong dimensions/distance/index type, and still writes to an existing compatible collection. -- [ ] **Step 2: Run the focused tests** and confirm current `_ensure_collection(strict=True)` incorrectly creates the collection. -- [ ] **Step 3: Add an internal rendered field** such as `vectors.collection_lifecycle: require_existing`; it is not a descriptor option and defaults to legacy behavior for non-operator configs. -- [ ] **Step 4: Thread the mode through the factory/store** and perform a read-only exact collection/index preflight before any upsert. +- [ ] **Step 1: Write RED tests** proving every registry-rendered config (session and maintenance) refuses a missing collection without issuing create/index mutations, refuses wrong dimensions/distance/index type, and still writes to an existing compatible collection. Legacy non-registry fixture configs retain their old default. +- [ ] **Step 2: Run the focused RED tests:** + +```bash +cd harness +.venv/bin/pytest -q \ + tests/test_qdrant_vector_store.py \ + tests/test_qdrant_cli_commands.py \ + tests/test_registry_evidence_config.py \ + -k 'require_existing or missing_collection or incompatible_collection' +``` + +Expected: FAIL because current `_ensure_collection(strict=True)` issues collection creation instead +of returning `semantic_index_incompatible`. No unrelated test may fail. +- [ ] **Step 3: Add the exact internal field** `vectors.collection_lifecycle: require_existing`; it is not a descriptor option. `renderRuntimeConfig` emits it for *all* schema-v3 registry configs, whether acquired by a session or maintenance. Only configs without registry `runtime_identity` retain the legacy create-capable default. +- [ ] **Step 4: Thread the mode through the factory/store** and perform a read-only exact collection/index preflight before any upsert. Add a schema-v1 binding regression proving equal session/maintenance rendered configs produce equal `config_dwh_binding()` values. - [ ] **Step 5: Add a race regression**: delete the collection after preflight and prove the write fails rather than recreates it. -- [ ] **Step 6: Run focused pytest and touched Ruff.** +- [ ] **Step 6: Run focused GREEN gates:** + +```bash +cd harness +.venv/bin/pytest -q \ + tests/test_qdrant_vector_store.py \ + tests/test_qdrant_cli_commands.py \ + tests/test_registry_evidence_config.py +.venv/bin/ruff check \ + tht/config.py tht/adapters/factory.py tht/adapters/vector/qdrant.py \ + tests/test_qdrant_vector_store.py tests/test_qdrant_cli_commands.py \ + tests/test_registry_evidence_config.py +``` + +Expected: all selected pytest tests PASS and Ruff exits 0. - [ ] **Step 7: Commit:** `fix: prevent P2 from owning Qdrant lifecycle`. ### Task 4: Extract the shared runtime configuration lease @@ -290,7 +1133,7 @@ Expected: FAIL only on the new machine-contract assertions. - Modify: `backend/test/tht-runner.test.ts` - Modify: `backend/test/workspace-runtime-handoff.test.ts` -- [ ] **Step 1: Write RED equivalence tests** feeding the same immutable snapshot, env, roots, installation overlay, and semantic contract to the session and operator callers and requiring byte-identical YAML. +- [ ] **Step 1: Write RED equivalence tests** feeding the same immutable snapshot, env, roots, installation overlay (including the normalized private-host allowlist), and semantic contract to the session and operator callers and requiring byte-identical YAML *and* identical harness `config_dwh_binding()` output. - [ ] **Step 2: Write RED identity/safety tests** for snapshot replacement, wrong commit/path, symlink/hardlink, wrong workspace ID, unstable config destination, same-revision changed bytes, mode, fsync/rename failure, and cleanup. - [ ] **Step 3: Run:** @@ -302,29 +1145,343 @@ npx vitest run \ test/workspace-runtime-handoff.test.ts ``` +Expected: RED only because `runtime-config-lease.ts` and the new shared-lease behavior do not yet +exist; existing `ThtRunner` regressions remain green. + - [ ] **Step 4: Move snapshot validation, runtime roots, installation-overlay parsing, binding resolution, and rendering** out of `ThtRunner` into one explicit-input component. -- [ ] **Step 5: Preserve session behavior**: `ThtRunner.acquireWorkspaceRuntime` delegates to the component and retains its current opaque FD-backed temporary lease. -- [ ] **Step 6: Add operator mode**: deterministically publish `/data/sessions//preprocessing/runtime-config/.yaml` plus a manifest, mode `0400/0600`, and set `collection_lifecycle: require_existing`. +- [ ] **Step 5: Preserve session semantics but replace the random source path**: `ThtRunner.acquireWorkspaceRuntime` delegates to the component and opens the same verified deterministic `/data/sessions//preprocessing/runtime-config/.yaml` publication used by maintenance. It may retain an opaque FD for TOCTOU resistance, but the harness-visible `-c` source is that canonical path in both modes; no `/dev/fd/*` or random temporary filename may enter schema-v1 binding. +- [ ] **Step 6: Add operator mode**: deterministically publish `/data/sessions//preprocessing/runtime-config/.yaml` plus a manifest, mode `0400/0600`. Do not add any operator-only rendered field: `collection_lifecycle: require_existing` and `egress.http_private_host_allowlist` are inputs to the one shared render and therefore appear identically in the session lease. - [ ] **Step 7: Prove same-revision rerun path identity** and changed config/binding refusal. Do not implement P3 semantic cross-revision canonicalization. -- [ ] **Step 8: Run focused tests, `npx tsc --noEmit -p .`, and `npm run build`.** +- [ ] **Step 8: Run focused GREEN gates:** + +```bash +cd backend +npx vitest run \ + test/workspace-runtime-config-lease.test.ts \ + test/tht-runner.test.ts \ + test/workspace-runtime-handoff.test.ts +npx tsc --noEmit -p . +npm run build +``` + +Expected: all selected Vitest tests PASS, TypeScript exits 0, and the build emits `dist/` without +changing tracked files. - [ ] **Step 9: Commit:** `refactor: share registry runtime configuration leases`. -### Task 5: Build durable outer state, locking, and revision guard +### Task 5: Build the retained root lease, durable state, and addressed registry publication **Files:** +- Create: `backend/native/workspace-fs-at/workspace_fs_at.cc` +- Create: `backend/native/workspace-fs-at/binding.gyp` +- Create: `backend/src/native/workspace-fs-at-binding.d.ts` +- Create: `backend/src/workspaces/workspace-fs-at.ts` +- Create: `backend/scripts/build-workspace-fs-at.mjs` +- Create: `backend/test/workspace-fs-at-native.test.ts` +- Create: `backend/test/fixtures/workspace-fs-at-race-worker.mjs` +- Create: `backend/src/workspaces/workspace-lock-root-lease.ts` +- Create: `backend/test/workspace-lock-root-lease.test.ts` +- Create: `backend/test/workspace-session-readers-lock.test.ts` +- Create: `backend/test/fixtures/workspace-session-readers-worker.mjs` - Create: `backend/src/workspaces/preprocessing-state.ts` - Create: `backend/test/workspace-preprocessing-state.test.ts` -- Modify: `docker/core.Dockerfile` (install/pin the kernel lock utility only when the implementation proves it is absent) +- Create: `backend/src/workspaces/registry-publication.ts` +- Create: `backend/test/workspace-registry-addressed-publication.test.ts` +- Create: `backend/test/workspace-registry-addressed-process.test.ts` +- Create: `backend/test/registry-pull-job-exports.test.ts` +- Create: `backend/test/fixtures/workspace-lock-root-worker.mjs` +- Create: `backend/test/fixtures/workspace-registry-addressed-worker.mjs` +- Modify: `backend/src/workspaces/registry.ts` +- Modify: `backend/test/workspace-registry.test.ts` +- Modify: `backend/src/routes/workspaces.ts` +- Modify: `backend/test/routes-workspaces.test.ts` +- Modify: `backend/src/app.ts` +- Create: `backend/test/app.test.ts` +- Modify: `backend/package.json` and `backend/package-lock.json` (Node 22 engine, exact `node-gyp@11.2.0`, exact `fs-ext@2.1.1` for flock only, native build scripts) +- Modify: `harness/tht/cli/preprocess_cmd.py` +- Modify: `harness/tht/cli/schema_cmd.py` +- Modify: `harness/tht/cli/vector_cmd.py` +- Create: `harness/tht/workspace_writer_lock.py` +- Create: `harness/tests/test_workspace_writer_lock.py` +- Modify: `harness/tests/test_preprocess_cli.py` +- Modify: `harness/tests/test_schema_fk_annotations.py` +- Modify: `harness/tests/test_qdrant_cli_commands.py` +- Modify: `docker/core.Dockerfile` to compile the repo-owned addon in an explicit compiler stage, copy only its `.node` output plus runtime dependencies, and smoke-load it in the compiler-free Node 22 Linux runtime -- [ ] **Step 1: Write RED state-schema tests** for valid state, same-operation resume, cross-workspace/revision/operation/config mismatch, tampering, run-ID traversal, restrictive modes, atomic failure, and bounded fields. -- [ ] **Step 2: Write RED cross-process lock tests** with two processes/containers: one wins, one receives `preprocessing_conflict`, and SIGKILL releases the kernel lock without deleting unrelated state. -- [ ] **Step 3: Write RED session-inventory tests**: no sessions/current-only sessions permit mutation; a resumable different-revision manifest blocks; finalized/archived sessions follow existing resume policy. -- [ ] **Step 4: Implement the exact state layout and durable write protocol** described above. -- [ ] **Step 5: Implement lock acquisition ordering and safe conflict mapping.** Do not invent stale-PID deletion; the kernel owns lock lifetime. -- [ ] **Step 6: Implement active-snapshot revalidation immediately before each mutating child stage** and the different-revision resumable-session guard. -- [ ] **Step 7: Add crash reconciliation tests** where a child publishes DWH/corpus state but outer state has not yet advanced. -- [ ] **Step 8: Run focused Vitest, typecheck, and build.** -- [ ] **Step 9: Commit:** `feat: add P2 preprocessing operation state`. +- [ ] **Step 1: Write RED native-seam and canonical-root factory tests.** In + `workspace-fs-at-native.test.ts`, load only the typed wrapper and prove the exact Node-API v8 exports, + Node 22 gate, descriptor ownership, close-on-conversion-failure, explicit/idempotent wrapper close, + no raw flags/path/FD export, exact `LockFileName = "writer.lock" | "session-readers.lock"`, and + stable errno mapping. Prove both names open only as `0600` regular locks and every other name/mode is + rejected. With real `fs-ext@2.1.1`, spy and contend on `flockOwnedLock` to prove exact shared/exclusive + and blocking/nonblocking mapping (`sh`/`ex`/`shnb`/`exnb`), actual writer exclusion, compatible shared + reader acquisition, nonblocking contention, blocking release/wakeup, no numeric escape, and close + exclusion during both locking and child-stdio duplication. Negative cases cover empty/`.`/`..`/slash/NUL/ + 256-byte components, symlink/non-directory/wrong handle kind, forged/cross-addon/closed handles, + `EEXIST`, `ENOENT`, `ELOOP` or platform-equivalent no-follow refusal, and `EBADF`; a source-level + assertion rejects any close retry loop and checks the frozen close-uncertain branch. The race worker repeatedly substitutes parent and leaf + directories/files/symlinks between `fstatat`, `mkdirat`, `openat`, and fsync; every iteration must + return the retained inode, the verified winning creator, or a stable fail-closed error—never the + attacker inode. Run the same suite on Linux and Darwin with the frozen directory-fsync behavior, + and require real fsync success in the Linux container. Then instantiate two factories against distinct + canonical installation session roots. Prove `canonicalInput` accepts only canonical IDs, never a + path; factory A rejects factory B's input; acquisition retains the original root FD/device/inode + across pathname replacement; relative opens detect replacement; and forged objects, symlinks, + hardlinks, wrong ownership/mode, and non-directory components fail. Prove one owner, callback-only + borrow, atomic transfer invalidating the source, idempotent close, close waiting for a borrow, and + use-after-transfer/close rejection. For `acquireOrProvision`, cover absent added and absent + never-used removed workspace leaves, exact UID/`0700`, anchored no-follow `mkdirat`/open, leaf and + parent fsync, concurrent creators/`EEXIST`, parent or leaf substitution, symlink races, failure + before publication retaining the unused leaf, and successful subsequent retry/revalidation. Add a + subprocess root-replacement race at derivation/open, provision/open, open/lock-open, and callback + settlement. Compile the exact public reader surface: `acquireSessionReadersShared()` has no argument and + returns only `Promise`; that lease exposes only diagnostic + `rootIdentity`, `transfer()`, and `close()`. Source/type fences reject a directory handle, path, numeric + FD, lock name, flags/mode, arbitrary open/flock callback, public constructor, or cast escape on either + type. +- [ ] **Step 2: Run the native-build/root-lease RED tests:** + +```bash +cd backend +npm run build:native +npx vitest run test/workspace-fs-at-native.test.ts \ + test/workspace-lock-root-lease.test.ts \ + test/workspace-session-readers-lock.test.ts +``` + +Expected: FAIL because the committed addon/build driver, typed wrapper, exported factory, +`acquireOrProvision`, opaque root lease, and P2-owned shared reader-lock lease do not exist. + +- [ ] **Step 3: Implement the pinned native seam and `workspace-lock-root-lease.ts`.** Write the raw + Node-API v8 C++ addon, exact declaration, binding.gyp, Node-22/offline node-gyp build driver, and sole + wrapper with the frozen flags, result stat, ownership, error, Linux, and Darwin contracts. Implement + exact `LockFileName`, `openOrCreateLockAt(..., 0o600)`, and typed `flockOwnedLock`; its module-private + synchronous numeric borrow calls exact `fs-ext@2.1.1.flockSync` for the four typed modes. The only + other authorized numeric-borrow consumer is module-private `duplicateForChildStdio`, which exposes no + FD. Pin exact lockfile inputs; do not use `fs-ext` for any filesystem-at operation. Make the + root factory depend only on `WorkspaceFsAtV1` and use its retained installation-parent FD, + component-relative no-follow directory opens, factory provenance, + exact service UID/`0700`, private provisioning serialization, fsync and identity rechecks, retained + FD plus device/inode, single ownership, scoped borrowing, atomic transfer, and idempotent close. + Implement `acquireSessionReadersShared()` and `WorkspaceSessionReadersLockLease` exactly as frozen: + wrapper-only literal open plus shared/nonblocking flock, successful root consumption, sole-owner + transfer, lock-then-root close, one close attempt per owned object, and stable fail-closed cleanup. + Retain unused provisioned leaves after all failures. No string overload, branded path, persistent FD + accessor, public numeric-borrow callback, public lease constructor, caller path, or unanchored + filesystem call survives. +- [ ] **Step 4: Write RED state/kernel-capability tests.** Cover valid state, exact replay, mismatches, + tampering, traversal, modes, atomic failure, and bounds. Two processes/containers contend; one wins; + SIGKILL releases the kernel lock. Prove writer acquisition invokes `flockOwnedLock` with + `"exclusive", "nonblocking"` on the owned `writer.lock` handle (and therefore the real + `fs-ext.flockSync`, not a mock ownership marker). Prove `runUnderOrderedWorkspaceWriterLocks` + itself—not nested `runUnderWorkspaceWriterLock` calls—sorts A/B lexically, holds both root/writer open descriptions + simultaneously for one callback, gives the production lifecycle owner the same callback-scoped set, + and invalidates captured set/capabilities before reverse B/A close on success, throw, cancellation, + and partial acquisition. Prove the one-lock API delegates to this sole producer. Every mutating + child receives the locked writer OFD as FD 3 and the retained root directory OFD as FD 4; Python + validates root identity, relative no-follow `writer.lock` identity, type/link/mode/UID, and already-held + `flock`. Missing/closed/substituted/renumbered/cross-root FD 3 or FD 4, independent lock descriptions, + forged markers, request/runtime-config/root mismatch, concurrent spawn misuse, and post-settlement + use all fail before writes and close exactly once. In the dedicated + `workspace-session-readers-lock.test.ts`, call the production methods rather than a shadow factory. + For session ownership, acquire a root, call `acquireSessionReadersShared()`, prove the source root is + consumed only after success, transfer the lease into a fixture session owner, and keep a real child + process plus stdout/stderr drain open; an exclusive production callback must contend until child exit, + stream settlement, and the owner's exactly-once teardown close, after which it succeeds. Cover spawn, + configure, child `exit`, child `close`, replacement, explicit teardown, and shutdown-shaped failure + paths with exactly one final lease close and no early unlock. Prove two production shared acquisitions + coexist; shared blocks exclusive; exclusive blocks a new shared acquisition; release permits the + contender; and root/path substitution always fails closed. + + Compile and call the exact maintenance signature + `runUnderSessionReadersExclusive(action: (lease: + BorrowedWorkspaceSessionReadersExclusiveLockLease) => Promise): Promise`. Prove it invokes only + `openOrCreateLockAt(retainedRoot, "session-readers.lock", 0o600)` and + `flockOwnedLock(owned, "exclusive", "nonblocking")`; holds the real lock across awaited work; rejects + contention without entering the callback; rejects nested/concurrent calls and acquisition while an + earlier child is in flight; allows the callback's awaited closed-union `spawnChild` and holds the reader + lock through its teardown; invalidates a captured borrow before settlement so `assertLive()` fails + afterward; and closes the owned lock once + on resolve, throw, cancellation, and acquisition failure. Inject production-wrapper close failure for + shared acquisition cleanup, shared lease teardown, and exclusive callback cleanup: require no retry, + all remaining owned objects still receive their one close attempt, the owner/capability stays invalid/poisoned, the outer + writer owner cannot report success, and the public failure is stable `preprocessing_conflict` with no + FD/path/name leak. Type/source fences allow only P2 `workspace-fs-at.ts` to open/flock and reject raw + addon/`fs-ext` imports, directory-handle/FD/path/name/flags fields or parameters, casts, alternate + factories, and a path fallback. +- [ ] **Step 5: Write RED addressed-registry, recovery-selector, and route tests.** Compile-check the + exact create/resume variants, installation/repository/remote identities, results, recovery identity, + scan limits, and the discriminated `RegistryEnsureBootstrapAddressedResultV1` signature, including + exact `{ kind: "already_active", snapshot }` and `{ kind: "bootstrap_terminal", result, snapshot }` + branches. Empty bootstrap has `baseCommit`/base digest `null`, base workspace list `[]`, and every + target ID as its lexical changed set; existing-active pull has full base/target identities and the + exact symmetric changed set. Under a held `repository.lock`, prove the active pointer/snapshot is + rechecked before any job-directory open: valid active returns `already_active` with no scan/network; + corrupt/incompatible active fails closed with no scan/network; absence alone reaches the exact bounded + sorted directory scan. There, zero nonterminal jobs creates one ID; exactly one matching nonterminal resumes that exact ID before a + network spy fires; a valid deterministic pre-claim/transition sibling is cleaned+directory-fsynced + and zero creates, while malformed/cross-owned/hardlinked/duplicate siblings fail closed; valid + terminal jobs do not enter selection and exact same-ID terminal resume replays stored result without + network; multiple nonterminals, sole pull, request/install/repo/remote mismatch, invalid name/type/ + mode/link/JSON/hash/phase/run ID, entry churn, and each entry/per-file/aggregate bound fail with + `registry_bootstrap_recovery_conflict`. Prove no mtime/newest/lexical-last/ + remote-head heuristic is observable. Exercise inspect, status, and lazy list/bootstrap through + `ensureBootstrapAddressed`; each adapter must consume both union branches and use the returned exact + snapshot, including exact HTTP 409/code-only conflict bodies, and pull/author + publication through `publishAddressed`, including app dependency wiring. Prove no callable old `bootstrap`, `pull`, `activate`, or direct pointer + publication path remains and every caller reaches the same production lifecycle owner and complete + ordered set. +- [ ] **Step 6: Write RED durable phase and process-kill tests.** Assert repository lock precedes a + `0600` `addressed-publication-jobs/.json` claim and that `request_claimed` rename+parent-fsync + completes before any network. Cover kills before claim rename, after claim fsync, during/after + advertisement but before durable pin, after `target_advertised` fsync, during/after exact-OID fetch, + after immutable-ref creation but before `target_fetched`, after its fsync, before/after `planned`, + every participant phase, and before/after active-pointer rename/fsync. Before the advertised pin, + same-ID resume may re-advertise; afterward it must retain that exact OID, reuse an already-correct + run ref without network, retry only the pinned exact OID when the ref is absent, reject a mismatched + ref, never fetch after `target_fetched`, and replay terminal success despite later tracking-ref drift. + Repeat every preterminal kill through the actual inspect, lazy-list, and status product callers: + after restart each must enter the same selector, resume the same ID before network when the one + matching job exists, and fail closed on multiple/corrupt/mismatched jobs. Add real two- and + three-process barriers where inspect, lazy list, and status all observe initial absence before + queueing on `repository.lock`: prove exactly one run claim, advertisement/fetch, publication, and + terminal job; queued callers return `already_active` before scanning jobs or touching network; and + all three adapters expose the same commit/manifest/workspace snapshot. Repeat with corrupt active + state after lock handoff and require fail-closed/no-job/no-network behavior. Include absent added/removed + roots, concurrent provisioning, complete-set contention, participant failure, concurrent session + admission, all-or-nothing base/target visibility, third-identity refusal, and reverse + readers/quiescence/writer/root release. +- [ ] **Step 7: Run all RED files:** + +```bash +cd backend +npm run build:native +npx vitest run test/workspace-fs-at-native.test.ts \ + test/workspace-lock-root-lease.test.ts \ + test/workspace-session-readers-lock.test.ts \ + test/workspace-preprocessing-state.test.ts \ + test/workspace-registry-addressed-publication.test.ts \ + test/workspace-registry-addressed-process.test.ts \ + test/registry-pull-job-exports.test.ts \ + test/workspace-registry.test.ts \ + test/routes-workspaces.test.ts \ + test/app.test.ts +``` + +Expected: FAIL on the missing native addon/wrapper, retained/provisioned root, opaque shared/exclusive +reader operations, ordered callback producer, FD-3/FD-4 capability, bootstrap/pull variants, bounded +automatic recovery selector, pre-plan +durable phases, route delegation, and addressed publication interfaces; no contender may exceed the +five-second test timeout. + +- [ ] **Step 8: Implement state and writer capability ownership.** Implement + `runUnderOrderedWorkspaceWriterLocks(rootLeases, action)` as the sole capability/set producer with + lexical acquisition through `openOrCreateLockAt(root, "writer.lock", 0o600)` followed by + `flockOwnedLock(lock, "exclusive", "nonblocking")`, callback-only borrows, and reverse + invalidation/close. Make the one-workspace + API its one-element adapter. A capability owns its root lease until settlement. Implement its exact + no-operand `runUnderSessionReadersExclusive(action)` with wrapper-only literal open, + exclusive/nonblocking flock, callback-borrow invalidation, one close attempt, poisoning on close + uncertainty, and no handle/path/FD/name surface. Make the production lifecycle owner nest these + callbacks in lexical order and pass their exact borrowed leases to participants. `spawnChild` + accepts only the frozen request union, remains usable from inside the exclusive reader callback, + installs writer OFD 3 plus root-directory OFD 4 for every mutating child, and performs exact + workspace/revision/config/root checks. Implement the shared Python + FD-3/FD-4 verifier and call it before every mutating harness path; preserve the same-union extension + seam for P3/P5/P6 and add no raw argv/path/direct-spawn overload. +- [ ] **Step 9: Implement addressed claim/fetch/plan/publication and automatic recovery.** Implement + the full P2 `RegistryAddressedPublicationStateV1`, all three installation/repository/remote identity + digests, canonical bootstrap request digest, artifact path, `request_claimed`, `target_advertised`, + `target_fetched`, and `planned` transitions plus both exact plan variants. Implement the one bounded + no-follow `ensureBootstrapAddressed` active-recheck/scan/zero-create/sole-exact-resume/fail-closed + selector while continuously holding `repository.lock`. Before the jobs scan, return + `{ kind: "already_active", snapshot }` for a strictly valid active snapshot and reject corrupt active + state; after publication return `{ kind: "bootstrap_terminal", result, snapshot }` from a strict + locked reread. Share a private already-locked executor with `publishAddressed` rather than + nesting/releasing the lock. Persist the claim before network, durably + pin the advertisement OID before exact-OID fetch to the create-only run ref, and implement the frozen + pre-pin/post-pin resume and exact-identity terminal replay rules. For every changed ID call + `acquireOrProvision` while retaining `repository.lock`, then pass all leases once to + `runUnderOrderedWorkspaceWriterLocks`; keep + quiescence, each exact `writerCapability.runUnderSessionReadersExclusive(...)` callback, participants, + synchronizers, atomic active publication/reconciliation, and `terminal_durable` inside the production + lifecycle-owner callback, followed by reverse reader-borrow invalidation/close and + repository release. +- [ ] **Step 10: Route every registry caller through the sole addressed APIs.** Implement + `WorkspaceRegistry.ensureBootstrapAddressed` and `publishAddressed`; delete/private the old + bootstrap/pull/activate/pointer methods. Make inspect, lazy list/bootstrap, and status delegate only + to the automatic selector and exhaustively adapt both `already_active` and `bootstrap_terminal` to the + returned exact snapshot; make pull and author activation delegate to `publishAddressed`; update + routes and `app.ts` construction. Preserve accepted HTTP request shapes (there is no bootstrap run-ID + field) while returning the stable fail-closed code. Synchronizers/adapters may not call either + writer-lock function or acquire capabilities. +- [ ] **Step 11: Run GREEN gates:** + +```bash +cd backend +npm run build:native +npx vitest run test/workspace-fs-at-native.test.ts \ + test/workspace-lock-root-lease.test.ts \ + test/workspace-session-readers-lock.test.ts \ + test/workspace-preprocessing-state.test.ts \ + test/workspace-registry-addressed-publication.test.ts \ + test/workspace-registry-addressed-process.test.ts \ + test/registry-pull-job-exports.test.ts \ + test/workspace-registry.test.ts \ + test/routes-workspaces.test.ts \ + test/app.test.ts +npx tsc --noEmit -p . +npm run build +node -e 'if (process.versions.node.split(".")[0] !== "22" || Number(process.versions.napi) < 8) process.exit(1); require("./native/workspace-fs-at/build/Release/workspace_fs_at.node")' +cd .. +docker build --target backend-native-build -f docker/core.Dockerfile . +docker build --target backend-build -f docker/core.Dockerfile . +docker build -f docker/core.Dockerfile -t thothii-core:p2-native . +docker run --rm --entrypoint node thothii-core:p2-native -e 'const a=require("/app/backend/native/workspace-fs-at/build/Release/workspace_fs_at.node"); const r=a.openat({parent:null,name:"/",kind:"directory",createMode:0}); const t=a.openat({parent:r.handle,name:"tmp",kind:"directory",createMode:0}); a.fsyncDirectory(t.handle); a.close(t.handle); a.close(r.handle); require("fs-ext")' +cd harness +.venv/bin/pytest -q tests/test_workspace_writer_lock.py \ + tests/test_preprocess_cli.py \ + tests/test_schema_fk_annotations.py \ + tests/test_qdrant_cli_commands.py +``` + +Expected: all selected Linux/Darwin-appropriate tests PASS, including real shared/exclusive contention, +session transfer/child teardown, maintenance callback invalidation, and injected close-failure cases; +TypeScript exits 0, the Node 22/N-API v8 load check succeeds, both Linux image stages build, the +compiler-free runtime loads the copied addon, and harness tests pass. + +- [ ] **Step 12: Commit:** + +```bash +git add backend/native/workspace-fs-at/workspace_fs_at.cc \ + backend/native/workspace-fs-at/binding.gyp \ + backend/src/native/workspace-fs-at-binding.d.ts \ + backend/src/workspaces/workspace-fs-at.ts \ + backend/scripts/build-workspace-fs-at.mjs \ + backend/test/workspace-fs-at-native.test.ts \ + backend/test/fixtures/workspace-fs-at-race-worker.mjs \ + backend/src/workspaces/workspace-lock-root-lease.ts \ + backend/test/workspace-lock-root-lease.test.ts \ + backend/test/workspace-session-readers-lock.test.ts \ + backend/test/fixtures/workspace-session-readers-worker.mjs \ + backend/src/workspaces/preprocessing-state.ts \ + backend/test/workspace-preprocessing-state.test.ts \ + backend/src/workspaces/registry-publication.ts \ + backend/test/workspace-registry-addressed-publication.test.ts \ + backend/test/workspace-registry-addressed-process.test.ts \ + backend/test/registry-pull-job-exports.test.ts \ + backend/test/fixtures/workspace-lock-root-worker.mjs \ + backend/test/fixtures/workspace-registry-addressed-worker.mjs \ + backend/src/workspaces/registry.ts backend/test/workspace-registry.test.ts \ + backend/src/routes/workspaces.ts backend/test/routes-workspaces.test.ts \ + backend/src/app.ts backend/test/app.test.ts \ + backend/package.json backend/package-lock.json \ + harness/tht/cli/preprocess_cmd.py harness/tht/cli/schema_cmd.py \ + harness/tht/cli/vector_cmd.py harness/tht/workspace_writer_lock.py \ + harness/tests/test_workspace_writer_lock.py harness/tests/test_preprocess_cli.py \ + harness/tests/test_schema_fk_annotations.py harness/tests/test_qdrant_cli_commands.py \ + docker/core.Dockerfile +git commit -m "feat: add durable addressed registry publication" +``` ### Task 6: Build the compiled inspect/operator boundary @@ -336,13 +1493,52 @@ npx vitest run \ - Modify: `backend/src/workspaces/types.ts` only if a separate operator-code union cannot stay private - [ ] **Step 1: Write RED entrypoint process tests** for exact JSON, malformed/extra stdin, unknown command/field, stdout/stderr bounds, timeout, signal, raw exception/stderr redaction, and no Fastify listener. -- [ ] **Step 2: Write RED `inspect` tests** for absent/corrupt/stale active state, migration-required descriptor, missing bindings, exact commit/blob/config identities, safe capability warnings, and no URL/secret output. -- [ ] **Step 3: Run focused Vitest** and verify no operator exists. +- [ ] **Step 2: Write RED `inspect` tests** for successful automatic locked bootstrap when active + state is absent, zero-job create, sole-exact-ID nonterminal resume before network, multiple/corrupt/ + identity-mismatched fail-closed output, and exact terminal replay behavior. Prove inspect has no + bootstrap-resume input field and delegates to the same `ensureBootstrapAddressed` spy as lazy list + and status. Exercise both result branches: a queued stale-absence caller must consume + `already_active` and the exact returned snapshot with no scan/fetch/pull, while bootstrap consumes + `bootstrap_terminal`; corrupt/stale active state after lock acquisition must fail closed. + Also cover active-state read without a fetch/pull, + migration-required descriptor, missing bindings, exact commit/blob/config identities, safe + capability warnings, and no URL/secret/job-ID output. +- [ ] **Step 3: Run the RED files:** + +```bash +cd backend +npx vitest run \ + test/workspace-maintenance.test.ts \ + test/workspace-preprocessing-service.test.ts +``` + +Expected: FAIL because the compiled entrypoint/service do not exist; no test may open a listener. - [ ] **Step 4: Implement a closed `WorkspacePreprocessingService` dependency interface**: active registry reader, shared config lease, fixed child runner, state store, session inventory, semantic preflight, egress policy. -- [ ] **Step 5: Implement active-snapshot-only acquisition.** P2 does not pull or activate Git; clean installations receive `workspace_not_activatable` with safe instructions. -- [ ] **Step 6: Implement bounded child execution** with fixed executable/argv, `-c` after the subcommand, FD-backed config/input, process-group cancellation, per-stage timeout, and strict one-document child JSON parsing. +- [ ] **Step 5: Implement pinned snapshot acquisition.** Only when its initial unlocked read observes + active state absent, `inspect` derives `RegistryBootstrapRecoveryIdentityV1` from the validated + installation/repository/remote and + calls `WorkspaceRegistry.ensureBootstrapAddressed`; it never constructs or selects a run ID. The + selector rechecks active state under its retained repository lock, returns the `already_active` + snapshot to stale queued callers, or creates/resumes and returns the `bootstrap_terminal` snapshot; + the service exhaustively consumes either branch rather than rereading an unlocked pointer. Corrupt + active state fails closed. With an active state, no maintenance command fetches or pulls. Mutating commands never bootstrap or activate Git and return + `workspace_not_activatable` with safe instructions if the inspect/bootstrap prerequisite is absent. +- [ ] **Step 6: Implement bounded child execution** with fixed executable/argv, `-c` after the subcommand and pointing at the verified deterministic +revision config, FD-backed ingress only, process-group cancellation, per-stage timeout, and strict one-document child JSON parsing. - [ ] **Step 7: Implement `inspect` and result encoding.** -- [ ] **Step 8: Run tests, typecheck, build, and verify `dist/workspace-maintenance.js` exists.** +- [ ] **Step 8: Run GREEN gates:** + +```bash +cd backend +npx vitest run \ + test/workspace-maintenance.test.ts \ + test/workspace-preprocessing-service.test.ts +npx tsc --noEmit -p . +npm run build +test -f dist/workspace-maintenance.js +``` + +Expected: both Vitest files PASS, typecheck/build exit 0, and the compiled entrypoint exists. - [ ] **Step 9: Commit:** `feat: add P2 workspace maintenance operator`. ### Task 7: Implement DWH preprocessing and outer resume @@ -353,13 +1549,36 @@ npx vitest run \ - Modify: `harness/tests/test_dwh_preprocess_job.py` - Modify: `harness/tests/test_lsh_job_resume.py` -- [ ] **Step 1: Write RED service tests** requiring fixed `preprocess dwh --steps introspect,lsh --json -c /dev/fd/N`, child result validation, outer/child run IDs, completed stages, safe artifact digests, and failure mapping. +- [ ] **Step 1: Write RED service tests** requiring fixed `preprocess dwh --steps introspect,lsh --json -c `, child result validation, outer/child run IDs, completed stages, safe artifact digests, and failure mapping. - [ ] **Step 2: Add real harness regressions** for deterministic same-revision config-source path, second clean-process rerun, resume after introspection, changed binding/config refusal, and ACTIVE preservation on failure. -- [ ] **Step 3: Run focused backend and harness tests.** +- [ ] **Step 3: Run the RED focused tests:** + +```bash +cd backend +npx vitest run test/workspace-preprocessing-service.test.ts -t 'DWH|resume' +cd ../harness +.venv/bin/pytest -q tests/test_dwh_preprocess_job.py tests/test_lsh_job_resume.py +``` + +Expected: new backend DWH coordinator tests FAIL before implementation; existing harness tests stay +green except newly added same-revision identity assertions. - [ ] **Step 4: Implement `preprocess dwh`** under the outer writer lock and persist state before/after every child transition. - [ ] **Step 5: Reconcile a published child run after an injected outer crash** without rerunning or corrupting ACTIVE. - [ ] **Step 6: Verify REST and direct rendered routing.** SSH returns `workspace_not_activatable` with a P10 warning. -- [ ] **Step 7: Run focused gates and commit:** `feat: run workspace DWH preprocessing from thothctl operator`. +- [ ] **Step 7: Run GREEN focused gates:** + +```bash +cd backend +npx vitest run test/workspace-preprocessing-service.test.ts -t 'DWH|resume' +npx tsc --noEmit -p . +cd ../harness +.venv/bin/pytest -q tests/test_dwh_preprocess_job.py tests/test_lsh_job_resume.py +.venv/bin/ruff check tht/cli/preprocess_cmd.py tests/test_dwh_preprocess_job.py tests/test_lsh_job_resume.py +``` + +Expected: all selected tests PASS, TypeScript exits 0, and touched Python is Ruff-clean. + +- [ ] **Step 8: Commit:** `feat: run workspace DWH preprocessing from thothctl operator`. ### Task 8: Implement FK candidate export and digest-bound review @@ -370,31 +1589,95 @@ npx vitest run \ - Modify: `tools/thothctl/internal/workspaceops/operations.go` - Modify: Go tests -- [ ] **Step 1: Write RED end-to-end unit/process tests**: new candidates create a bounded artifact and return `manual_review_required`; schema/Evidence child calls are absent. +- [ ] **Step 1: Write RED end-to-end unit/process tests**: new candidates create a bounded artifact and return `manual_review_required` with its 32-hex outer run ID; `schema check` without `--resume`, with another run's ID, or with a same-digest ambiguous run is refused; the exact resume ID loads only that run; schema/Evidence child calls are absent. Run: + +```bash +cd backend +npx vitest run test/workspace-preprocessing-service.test.ts -t 'FK|manual review' +cd ../tools/thothctl +go test ./internal/workspaceops ./cmd/thothctl -run 'Fk|Annotation|Export' -v +``` + +Expected: FAIL because FK dispatch, digest review, and exclusive export are not implemented. - [ ] **Step 2: Add safe host ingress tests** proving SQL and annotations travel only over stdin, are absent from Compose argv/state/logs, and staging files are removed. - [ ] **Step 3: Add safe host egress tests** for candidate export identity/digest, existing destination refusal, and sanitized JSON mode. - [ ] **Step 4: Implement `schema suggest-fks`** with candidate count/digest and optional exclusive output. -- [ ] **Step 5: Implement `schema check`** in two modes: read-only orphan validation; or reviewed annotation import requiring the exact candidate digest. Persist review digest + annotation digest + workspace/revision. +- [ ] **Step 5: Implement `schema check`** in two modes: read-only orphan validation; or reviewed annotation import requiring the exact candidate digest. Both modes require `--resume <32-hex-outer-run-id>`, call `PreprocessingStateStore.loadForResume` for that exact workspace/revision/operation, and never select by candidate digest. Persist outer run ID + review digest + annotation digest + workspace/revision. - [ ] **Step 6: Require the review record on full-run resume.** A mere zero-orphan result without reviewer digest is insufficient. - [ ] **Step 7: Preserve the boundary:** P2 updates runtime-local annotations only; it never writes Git. Output warns that P5 will supersede this local acknowledgement. -- [ ] **Step 8: Run Go/backend/harness focused gates and commit:** `feat: add P2 FK review checkpoint`. +- [ ] **Step 8: Run GREEN focused gates:** + +```bash +cd backend +npx vitest run test/workspace-preprocessing-service.test.ts -t 'FK|manual review' +npx tsc --noEmit -p . +cd ../tools/thothctl +go test ./internal/workspaceops ./cmd/thothctl -run 'Fk|Annotation|Export' -v +cd ../../harness +.venv/bin/pytest -q tests/test_schema_fk_annotations.py +.venv/bin/ruff check tht/cli/schema_cmd.py tests/test_schema_fk_annotations.py +``` + +Expected: all selected tests PASS and no candidate/annotation bytes appear in captured argv, logs, +or public JSON. + +- [ ] **Step 9: Commit:** `feat: add P2 FK review checkpoint`. ### Task 9: Implement schema indexing and Evidence policy boundaries **Files:** - Modify: `backend/src/workspaces/preprocessing-service.ts` -- Modify: backend service tests +- Modify: `backend/src/workspaces/runtime-renderer.ts` +- Modify: backend service/renderer/config-lease tests +- Modify: `tools/thothctl/internal/config/installation.go` and tests +- Modify: `harness/tht/config.py` +- Modify: `harness/tht/adapters/factory.py` +- Modify: `harness/tht/adapters/evidence/http.py` - Modify: `harness/tests/test_http_evidence_source.py` - Modify: `harness/tests/test_registry_evidence_config.py` - Modify: `harness/tests/test_semantic_kind_isolation.py` -- [ ] **Step 1: Write RED schema-index tests** for compatible pre-existing collection, deterministic JSON counts, idempotent repeat, missing/incompatible refusal, and no collection-create request. +- [ ] **Step 1: Write RED schema-index tests** for compatible pre-existing collection, deterministic JSON counts, idempotent repeat, missing/incompatible refusal, and no collection-create request. Run: + +```bash +cd backend +npx vitest run test/workspace-preprocessing-service.test.ts -t 'schema index|Evidence|egress' +cd ../harness +.venv/bin/pytest -q \ + tests/test_http_evidence_source.py \ + tests/test_registry_evidence_config.py \ + tests/test_semantic_kind_isolation.py +``` + +Expected: the new operator boundary assertions FAIL; existing adapter tests remain green. - [ ] **Step 2: Write RED Evidence tests** for no-Evidence warning/skip, HTTP dry-run/run/resume/unchanged/mutation, authoritative workspace identity, ACTIVE preservation, and filesystem early stop before adapter/Qdrant calls. -- [ ] **Step 3: Write RED egress tests** for exact private-host allowlist, DNS re-resolution, redirect to private/link-local/metadata, signed URL query redaction, and refusal of S3 ambient/custom/private/insecure modes. -- [ ] **Step 4: Implement installation-local egress-policy parsing** with exact bounded hostnames and no wildcard. Descriptor flags alone never grant network access. -- [ ] **Step 5: Implement `index-schema` and `preprocess evidence`** with semantic preflight and stable result mapping. -- [ ] **Step 6: Revalidate active revision and session inventory immediately before each write.** -- [ ] **Step 7: Run focused tests/touched Ruff/backend typecheck/build and commit:** `feat: add guarded P2 semantic preprocessing`. +- [ ] **Step 3: Write RED tests at the real Python request boundary.** Cover the exact `THT_HTTP_PRIVATE_HOST_ALLOWLIST` grammar, session/operator rendered byte and `config_dwh_binding()` equality, public DNS, an allowlisted RFC1918/IPv6-ULA hostname, non-allowlisted private DNS, direct private IP literals, redirect chains, redirect to private/link-local/metadata, DNS rebinding where the connected peer differs from the approved resolution, mixed safe/unsafe DNS answers, signed-query redaction, and refusal of S3 ambient/custom/private/insecure modes. +- [ ] **Step 4: Implement the exact installation setting and shared rendering.** Parse at most 32 canonical lower-case ASCII DNS names (253 bytes each), no IPs/wildcards/trailing dot/duplicates, and render `egress.http_private_host_allowlist` identically into session and maintenance YAML. The descriptor's `allow_private_hosts` is necessary but never sufficient for private access; the exact hostname must also be installation-allowlisted. +- [ ] **Step 5: Enforce policy inside `HttpManifestEvidenceSource`, not in a Node-only preflight.** Disable automatic redirects. Before each hop, freshly resolve every A/AAAA result; refuse the hop if any address is loopback, link-local, metadata (`169.254.169.254` and IPv6 equivalents), multicast, unspecified, reserved/documentation/benchmark, or otherwise non-routable. RFC1918 and IPv6 ULA are allowed only for a descriptor-enabled exact allowlisted DNS hostname. Connect, then verify the actual peer IP belongs to that hop's approved resolution and class before accepting response bytes. Resolve and peer-check again for every redirect; a changed/rebound peer fails closed. Descriptor `allow_private_hosts` alone never bypasses any DNS or peer check. +- [ ] **Step 6: Implement `index-schema` and `preprocess evidence`** with semantic preflight and stable result mapping. Hold the shared workspace writer lock across every mutating publication; active revision cannot change because registry activation honors the same lock. Immediate revision/session rechecks remain defense in depth. +- [ ] **Step 7: Run GREEN focused gates:** + +```bash +cd backend +npx vitest run test/workspace-preprocessing-service.test.ts -t 'schema index|Evidence|egress' +npx tsc --noEmit -p . +npm run build +cd ../harness +.venv/bin/pytest -q \ + tests/test_http_evidence_source.py \ + tests/test_registry_evidence_config.py \ + tests/test_semantic_kind_isolation.py +.venv/bin/ruff check \ + tht/cli/preprocess_cmd.py tht/cli/vector_cmd.py tht/config.py \ + tht/adapters/factory.py tht/adapters/evidence/http.py \ + tests/test_http_evidence_source.py tests/test_registry_evidence_config.py \ + tests/test_semantic_kind_isolation.py +``` + +Expected: all selected tests PASS; filesystem returns `evidence_materialization_required` before +adapter discovery; collection-create call count is zero. + +- [ ] **Step 8: Commit:** `feat: add guarded P2 semantic preprocessing`. ### Task 10: Implement the ordered full-run coordinator @@ -403,12 +1686,30 @@ npx vitest run \ - Modify: `backend/test/workspace-preprocessing-service.test.ts` - Modify: `backend/test/workspace-maintenance.test.ts` -- [ ] **Step 1: Write a RED stage-table test** for exact order `dwh → fk_suggest → fk_review/check → schema_index → evidence` and for no hidden/skipped mutation. +- [ ] **Step 1: Write a RED stage-table test** for exact order `dwh → fk_suggest → fk_review/check → schema_index → evidence` and for no hidden/skipped mutation. Run: + +```bash +cd backend +npx vitest run test/workspace-preprocessing-service.test.ts -t 'full run|stage order' +``` + +Expected: FAIL because the full-run state machine is not implemented. - [ ] **Step 2: Add scenarios**: pre-curated/no-new-candidate completion; new-candidate block; digest-reviewed resume; no-Evidence warning; filesystem deferred block; each child failure; resume mismatch; outer crash reconciliation. - [ ] **Step 3: Implement the coordinator as an explicit state machine**, not recursive command dispatch. - [ ] **Step 4: Persist completion after each verified child artifact** and never mark a stage based only on exit code. - [ ] **Step 5: Prove unchanged rerun creates no duplicate schema/Evidence points or generation.** -- [ ] **Step 6: Run focused Vitest, typecheck, build, and commit:** `feat: orchestrate the P2 preprocessing chain`. +- [ ] **Step 6: Run GREEN coordinator gates:** + +```bash +cd backend +npx vitest run test/workspace-preprocessing-service.test.ts -t 'full run|stage order' +npx tsc --noEmit -p . +npm run build +``` + +Expected: all coordinator scenarios PASS; blocked paths prove later child call count is zero. + +- [ ] **Step 7: Commit:** `feat: orchestrate the P2 preprocessing chain`. ### Task 11: Add the hardened maintenance service and selected-image handoff @@ -423,13 +1724,28 @@ npx vitest run \ - Modify: `tools/thothctl/internal/pi/update.go` - Modify relevant Go/Bash/Compose tests -- [ ] **Step 1: Write RED Compose contract tests** for the exact security contract, profile, mounts, no Pi/no port/no build, local/server storage, and absence of Git credentials on active-snapshot operations. +- [ ] **Step 1: Write RED Compose contract tests** for the exact security contract, profile, mounts, no Pi/no port/no build, local/server storage, and Git credentials attached only to inspect's empty-registry bootstrap capability. - [ ] **Step 2: Write RED image-precedence tests** across base/profile/operator overrides/current-image override; `core` and maintenance must resolve to the same immutable ID. -- [ ] **Step 3: Write RED lifecycle tests** for `--no-deps`, pre-existing service preservation, interruption cleanup, hostile container name/label collision, tag replacement, and output rejection on post-run image mismatch. -- [ ] **Step 4: Add the dedicated service and entrypoint**; the entrypoint executes only the compiled operator and never calls Pi trust setup. -- [ ] **Step 5: Generate a per-operation final override** that pins immutable image ID, exact secrets, egress policy, and owned labels. Validate `docker compose config` structurally before run. -- [ ] **Step 6: Extend Pi update/rollback override generation** so future selected images cannot split core and maintenance. -- [ ] **Step 7: Run:** +- [ ] **Step 3: Write RED lifecycle tests** for `--no-deps`, pre-existing service preservation, interruption cleanup, hostile container name/label collision, tag replacement, output rejection on post-run image mismatch, and infinite/oversized maintenance stdout or stderr causing immediate owned process-group/container termination through `RunBounded`. +- [ ] **Step 4: Run RED packaging contracts:** + +```bash +bash scripts/test-preprocess-compose-config.sh +bash scripts/test-compose-secret-policy.sh +cd tools/thothctl +go test ./internal/workspaceops ./internal/pi -run 'Maintenance|Workspace|Image' -v +``` + +Expected: FAIL because `workspace-maintenance`, its entrypoint, selected-image handoff, and scoped +secret override do not exist; no Compose resource is started by these contract tests. + +- [ ] **Step 5: Add the dedicated service and entrypoint**; the entrypoint executes only the compiled + operator and never calls Pi trust setup. Extend the HTTPS/SSH transport contracts and connector + generator metadata so `workspaceops.Run` can select exact inspect/DWH/Evidence files without + mounting an unrelated workspace or role credential. +- [ ] **Step 6: Generate a per-operation final override** that pins immutable image ID, exact secrets, egress policy, and owned labels. Validate `docker compose config` structurally before run. +- [ ] **Step 7: Extend Pi update/rollback override generation** so future selected images cannot split core and maintenance. +- [ ] **Step 8: Run:** ```bash bash scripts/test-preprocess-compose-config.sh @@ -440,7 +1756,18 @@ bash scripts/test-no-deployment-coupling.sh cd tools/thothctl && go test ./... ``` -- [ ] **Step 8: Build the core image and invoke operator `--help` through the exact service** without starting Pi/backend/dependencies. +Expected: every script exits 0 and every Go package PASS. Then use the contract script's owned +fixture mode (add this mode in Step 1) to build and invoke the exact service: + +```bash +bash scripts/test-preprocess-compose-config.sh --image-smoke +``` + +Expected: the script prints `workspace-maintenance image smoke: PASS`; operator help exits 0; its +final owned-resource assertion finds no maintenance container; core/frontend/Pi were never started. +The script creates and destroys its own fixture env/overrides and never reads an operator +installation. + - [ ] **Step 9: Commit:** `feat: package the P2 maintenance service`. ### Task 12: Complete host dispatch and supported-platform build contract @@ -452,10 +1779,29 @@ cd tools/thothctl && go test ./... - Modify: `scripts/test-thothctl-build-contract.sh` - Update operator docs -- [ ] **Step 1: Add RED command-to-request-to-Compose tests** for all seven public commands, JSON/human output, exit mapping, secret redaction, and exact stdin. -- [ ] **Step 2: Implement dispatcher integration** using only typed requests. +- [ ] **Step 1: Add RED command-to-request-to-Compose tests** for all seven public commands, JSON/human output, exit mapping, secret redaction, and exact stdin. For inspect, assert Go accepts no `--resume`/bootstrap-ID field, sends the same typed inspect request on every invocation, preserves exact JSON code `registry_bootstrap_recovery_conflict`, maps it to exit 3, and emits only the frozen safe human sentence. Run: + +```bash +cd tools/thothctl +go test ./cmd/thothctl ./internal/workspaceops -run 'Workspace' -v +``` + +Expected: FAIL on dispatch/output cases not yet connected; parser-only tests from Task 1 remain green. +- [ ] **Step 2: Implement dispatcher integration** using only typed requests. Add the exact Go result-code constant and exhaustive exit/human renderer case for `registry_bootstrap_recovery_conflict`; do not add a Go or HTTP bootstrap run-ID selector. - [ ] **Step 3: Cross-build the existing release matrix** and verify Windows input/output safety compiles. Do not claim Windows Docker behavior without a Windows Docker run. -- [ ] **Step 4: Run `go test ./...`, build contract, `go vet ./...`, and `gofmt` check.** +- [ ] **Step 4: Run GREEN native gates:** + +```bash +cd tools/thothctl +test -z "$(gofmt -l .)" +go test ./... +go vet ./... +cd ../.. +bash scripts/test-thothctl-build-contract.sh +``` + +Expected: no `gofmt` output; all Go packages PASS; vet and the Linux/macOS/Windows release build +contract exit 0. - [ ] **Step 5: Commit:** `feat: expose P2 workspace commands in thothctl`. ### Task 13: Build the clean-state automated P2 process goal @@ -472,8 +1818,14 @@ The public command is: ./scripts/p2-acceptance.sh integration --keep ``` -- [ ] **Step 1: Write RED acceptance-runner tests** for ownership-first state, unique run/project/container/image names, exact cleanup, `--keep`, injected failure, signal cleanup, report bounds, and no automatic retry. -- [ ] **Step 2: Build a clean owned topology** under `.artifacts/p2-integration/p2-/`: local bare Git + author clone, active P1 snapshot, installation descriptor/env, fixture-only secrets, controlled REST DWH, controlled HTTP Evidence, real compatible Qdrant, deterministic Ollama-compatible embedding fixture, selected core image, and no backend/Pi/frontend. +- [ ] **Step 1: Write RED acceptance-runner tests** for ownership-first state, unique run/project/container/image names, exact cleanup, `--keep`, injected failure, signal cleanup, report bounds, and no automatic retry. Run: + +```bash +node --test backend/scripts/p2-acceptance.test.mjs +``` + +Expected: FAIL because the runner and ownership/report contract do not exist. +- [ ] **Step 2: Build a clean owned topology** under `.artifacts/p2-integration/p2-/`: local bare Git + author clone, an empty installation registry that `workspace inspect` bootstraps, installation descriptor/env, fixture-only secrets, controlled REST DWH, controlled HTTP Evidence, real compatible Qdrant, deterministic Ollama-compatible embedding fixture, selected core image, and no backend/Pi/frontend. - [ ] **Step 3: Pre-provision the exact compatible Qdrant collection** outside the product operation and record that setup as a P4-deferred fixture step. - [ ] **Step 4: Exercise only built `thothctl` product commands** and assert: 1. exact inspect revision/config identity; @@ -485,11 +1837,29 @@ The public command is: 7. no-Evidence warning/skip; 8. filesystem `evidence_materialization_required` with no partial output; 9. direct renderer regression and SSH fail-closed result; - 10. missing workspace/binding, resume mismatch, different-revision resumable session, concurrent writer, annotation invalid, egress refusal, and semantic incompatibility; - 11. no collection creation, no backend listener, no Pi init, no arbitrary mount; - 12. exact cleanup preserving all foreign/pre-existing resources. + 10. missing workspace/binding, resume mismatch, different-revision resumable session, concurrent writer, annotation invalid, egress refusal (including redirect/rebinding), and semantic incompatibility; + 11. session/operator rendered bytes and `config_dwh_binding()` are equal; concurrent pull/activation is refused during each mutating child and subsequent session admission remains on the unchanged active revision; + 12. 700 KiB candidate export succeeds inside the 1 MiB envelope; one-byte-over/infinite stdout and stderr terminate the owned process group/container without excess capture; + 13. no collection creation, no backend listener, no Pi init, no arbitrary mount; + 14. exact cleanup preserving all foreign/pre-existing resources; + 15. for every preterminal bootstrap kill point, restart separately through built inspect, backend lazy + list, and status caller harnesses and prove same-ID resume precedes network; inject zero, multiple, + corrupt, and request/install/repository/remote mismatch scans and prove the exact stable code with + no newest heuristic or leaked candidate ID; then barrier-start two and all three callers after each + observed absence and prove one bootstrap/network/publication, queued `already_active` results, and + byte-identical returned active snapshot identity. Corrupt active state at the lock handoff must + produce no job scan, run, or network. - [ ] **Step 5: Produce bounded `report.json` and `report.md`**, declare hashes for every retained owned artifact, and scan raw Git objects, names, state, reports, logs, configs, candidates, and Qdrant payloads for fixture canaries/signed queries/raw SQL. -- [ ] **Step 6: Run runner unit tests, then one clean integration run without retry.** On failure, diagnose/fix/regress and start one new clean run; never loop blindly. +- [ ] **Step 6: Run GREEN runner tests, then one clean integration without retry:** + +```bash +node --test backend/scripts/p2-acceptance.test.mjs +./scripts/p2-acceptance.sh integration --keep +``` + +Expected: unit tests PASS; the integration exits 0 once and prints `P2 automated integration: PASS` +plus one retained report path. On failure, diagnose/fix/regress and start one new clean run; never +loop blindly. - [ ] **Step 7: Commit tooling:** `test: add P2 host preprocessing acceptance`. ### Task 14: Finalize P2 documentation and independent manual walkthrough @@ -500,11 +1870,19 @@ The public command is: - Update: `docs/contracts/workspace-preprocessing-cli.md` - Optionally create manual lab helper files if concrete setup cannot remain concise -- [ ] **Step 1: Document prerequisites and boundaries**: Docker/Compose, active registry snapshot, existing compatible collection, running semantic services for semantic commands, no host language runtimes, no backend/Pi. -- [ ] **Step 2: Fill exact P2 commands** for inspect, DWH/resume, FK export/review digest, check/import, index, HTTP dry/run, full run, unchanged rerun, filesystem deferred result, secret scan, and cleanup. +- [ ] **Step 1: Document prerequisites and boundaries**: Docker/Compose, inspect/bootstrap then active registry snapshot, existing compatible collection, running semantic services for semantic commands, no host language runtimes, no backend/Pi. `docs/contracts/workspace-preprocessing-cli.md` must state that inspect exposes no bootstrap resume flag; inspect/lazy list/status all use the same automatic repository-locked bounded selector; its first locked action revalidates active state, valid active returns `already_active` before job scan/network, corrupt active fails closed, and only proven absence reaches job selection. Document that zero creates, one exact nonterminal resumes its exact ID before network, multiple/corrupt/mismatched fails closed, terminals are excluded from selection and replay only by exact identity, and no newest heuristic exists. Include the queued-caller guarantee: concurrent inspect/lazy-list/status calls publish once and converge on the same returned snapshot. Update the local/server backup inventories to state that the sessions store contains P2 `.tht-dwh`, corpus, runtime-local annotations, and preprocessing state; reuse existing volume backup mechanics and do not design P3 ownership recovery here. +- [ ] **Step 2: Fill exact P2 commands** for inspect, DWH/resume, FK export/review digest, check/import, index, HTTP dry/run, full run, unchanged rerun, filesystem deferred result, secret scan, and cleanup. Document exact JSON code `registry_bootstrap_recovery_conflict`, exit 3, the sole human sentence, and lazy-list/status HTTP 409 code-only body; examples must not suggest a bootstrap run-ID selector or manual newest-job choice. - [ ] **Step 3: Explain each observed component/artifact** without exposing config or secret contents. - [ ] **Step 4: Require a new manual root and `VERDICT.md`** with reviewer, UTC time, explicit result for every P2 check, observations, and exactly `P2 manual acceptance: PASS|FAIL`. Automation never writes it. -- [ ] **Step 5: Add mechanical docs tests** for all released commands and stable codes. +- [ ] **Step 5: Add mechanical docs tests** for all released commands and stable codes, including the automatic bootstrap recovery matrix, locked `already_active`/corrupt-active behavior, two-/three-caller convergence, absent inspect `--resume`, exact recovery-conflict JSON/exit/human output, and no newest-job wording, then run: + +```bash +bash scripts/test-verify-workspace-install-docs.sh +bash scripts/verify-workspace-install-docs.sh --fixtures-only +``` + +Expected: both exit 0; the living manual contains every frozen P2 command and keeps P3–P6 sections +PENDING. - [ ] **Step 6: Commit:** `docs: add P2 preprocessing operator walkthrough`. ### Task 15: Run affected-layer verification and hand off the hard checkpoint @@ -512,7 +1890,7 @@ The public command is: **Files:** - Modify after evidence exists: `PROJECT_STATE.md` -- [ ] **Step 1: Run complete affected Go gates:** `cd tools/thothctl && go test ./... && go vet ./...`, plus the release build contract. +- [ ] **Step 1: Assert `go version` is exactly Go 1.26.5, then run complete affected Go gates:** `(cd tools/thothctl && go test ./... && go vet ./...)`, plus the release build contract. Record Go 1.26.5 in the acceptance report; offline verification must not download a toolchain. - [ ] **Step 2: Run complete backend gates:** `cd backend && npx vitest run && npx tsc --noEmit -p . && npm run build`. - [ ] **Step 3: Run focused harness tests listed in the target map, then full `.venv/bin/pytest -q` if feasible.** Any baseline failure must be reported exactly; touched Python files must be Ruff-clean. - [ ] **Step 4: Run all affected Compose/security contracts** from Task 11 and `git diff --check`. @@ -554,6 +1932,7 @@ Do not begin P3, mark manual PASS, or infer implementation approval from plan ap | Requirement/decision | P2 implementation/proof | Deferred truth | |---|---|---| | D2 / P2 / RNF7 | Native `thothctl`, dedicated one-shot service, existing engine | GUI/backend endpoint excluded | +| RF1.1 / RF1.5 | Consume P1's complete source contract and bind descriptor/source to the same bootstrapped or active commit | Filesystem object materialization P6 | | RF1.2 / RNF1 | File-only bindings, operation-specific mounts, redaction/scan | No secret in Git/rendered output | | RF1.3 / RNF5 | Same binding+renderer code and byte-equivalence test | P3 canonical cross-revision identity/migration | | RF1.4 / RF2.2 | REST process goal; direct regression | SSH operational support P10 | @@ -564,11 +1943,14 @@ Do not begin P3, mark manual PASS, or infer implementation approval from plan ap | RF3.2 / D5 | Runtime-local P2 annotations only | Repository annotations and pinned sync P5 | | RF3.3 | No model-derived FK path added | Existing workflow invariant preserved | | RF4.1 | Existing schema hash/upsert + JSON counts | Collection lifecycle P4 | -| RF5.2 | Policy-allowed HTTP dry/run/resume/publish | Filesystem P6; broader S3 policy separately reviewed | -| RF5.3 / D9 | Existing per-run behavior only | Long-term GC/retention P9 | +| RF5.1–RF5.2 | Consume P1 HTTP declaration for dry/run/resume/publish; parse/render filesystem and stop stably | Filesystem materialization P6; broader S3 policy separately reviewed | +| RF5.3 / D9 | Prove the existing engine retention behavior during HTTP mutation without changing policy/defaults | New configurable or long-term GC behavior P9 | +| RF5.4 | Assert corpus ACTIVE and every Evidence write/search identity belong to the selected workspace | Revision-scoped Evidence payload/read filtering P3 | | RNF3 | Safe errors, prior ACTIVE preserved, no-Evidence warning | — | | RNF4 | Workspace binding + P2 different-revision session guard | Revision-scoped points/roots P3 | -| RF8.5–8.6 / RNF8–9 | Clean process goal + separate walkthrough | Aggregate P2–P6 verification after P6 | +| P2→P3 root-lock handoff | Sole factory-produced retained root-FD lease backed by the repo-owned Node-API v8 `workspace-fs-at` seam; exact `LockFileName` admits `writer.lock` and `session-readers.lock` at `0600`; typed `flockOwnedLock` owns all `fs-ext.flockSync` SH/EX blocking/nonblocking calls; numeric borrowing remains module-private to flock/child duplication; exact `*at`/directory-fsync ownership and Linux/Darwin negative/race tests; consumed writer API, transfer/borrow/close and replacement races | Reader-gate acquisition is exercised in P3 through this frozen seam, with no cast/raw binding/path reopen | +| P2→P3 registry handoff | Repository-first immutable base/target plan, complete lexical capability set, participant no-reentry, atomic/reconciled active publication, and one repository-locked bounded `ensureBootstrapAddressed` recovery path for inspect/lazy list/status; locked active recheck returns `already_active`, corrupt active fails closed, and queued callers converge without a second scan/network/publication | Public `registry pull` command released in P3 | +| RF8.1 / RF8.3–8.6 / RNF8–9 | Local/server manuals, P2 product-path smoke, retained report/scan/cleanup, separate walkthrough, and sessions-volume backup inventory | Aggregate P2–P6 smoke after P6; `.tht-dwh` ownership chapter P3 | | PRD AC2 | Native release binary on local/server installation profiles | Windows Docker is a separate manual claim | | PRD AC8 | HTTP unchanged/mutation cases | Filesystem change/GC P6/P9 | @@ -576,7 +1958,7 @@ Do not begin P3, mark manual PASS, or infer implementation approval from plan ap - No frontend/GUI or backend HTTP preprocessing endpoint. - No host Python, Node, Pi, `tht`, arbitrary shell, arbitrary entrypoint, or arbitrary host mount. -- No Git pull/publish/push, active-revision transition, or authoring API in P2. +- No released host Git pull/push or authoring command in P2. P2 does implement and process-test the internal repository-first `publishAddressed` publication boundary required by P3; only P3 exposes it. - No P3 canonical effective fingerprint, ownership migration, revision-scoped roots/points, or `.tht-dwh` operator chapter. - No P4 collection creation/index repair/rebuild/maintenance drain. - No P5 Git-canonical FK annotation sync/acceptance. diff --git a/docs/superpowers/plans/2026-08-10-p3-effective-config-fingerprint.md b/docs/superpowers/plans/2026-08-10-p3-effective-config-fingerprint.md new file mode 100644 index 00000000..3930e459 --- /dev/null +++ b/docs/superpowers/plans/2026-08-10-p3-effective-config-fingerprint.md @@ -0,0 +1,2342 @@ +# P3 Effective Configuration Fingerprint and Revision Isolation Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Make DWH-derived preprocessing safely reusable across semantically equivalent Git revisions while keeping curated artifacts, corpus state, and schema/Evidence vectors revision-pinned and Memory workspace-global, with explicit compatibility migrations from the P2 layout. + +**Architecture:** The Python harness owns one versioned canonical effective-DWH binding computed from an allowlist of non-secret, output-affecting values. Both the P2 dedicated one-shot maintenance process and session runtime consume the same rendered config and ask the harness for that binding; neither hashes temporary YAML paths independently. `thothctl` launches only the profile-gated `workspace-maintenance` Compose job: that process owns client-addressed durable run creation/replay, owner-qualified quiescence, the exclusive reader gate, and the inherited P2 writer-lock capability without requiring Fastify or stopping `core`. A binding-keyed workspace-global DWH cache feeds verified immutable snapshots into revision-qualified runtime roots, while explicit migrations copy and verify schema-v1 owners and legacy Memory state without reinterpreting or merging it in place. Qdrant partitions identity by `(workspace_id, workspace_revision)` for schema/Evidence and by `workspace_id` alone for Memory/solved records. + +**Tech Stack:** Python 3.12, Pydantic v2, Typer, pytest, Node.js 22, TypeScript, Vitest, repo-owned Node-API v8 `workspace-fs-at` with a closed typed lock API and wrapper-internal exact `fs-ext@2.1.1` for `flock(2)` only, Go 1.26.5 `thothctl` (matching `go.mod` `toolchain go1.26.5`), Docker Compose, Qdrant, Ollama, Bash, Git, YAML/JSON, SHA-256, UUIDv5, POSIX `*at`/`flock`/`fsync`/atomic rename. + +--- + +## Scope, dependency gate, and invariants + +This is **P3 / D3 only**. P2 must already be implemented, automatically green, manually accepted, and present in this worktree. P3 extends—without renaming, wrapping in a parallel tree, or duplicating—P2's exact frozen handoff: + +- `backend/src/workspaces/runtime-config-lease.ts` / `WorkspaceRuntimeConfigLeaseFactory.acquireSession` and `.acquireMaintenance`; +- `backend/native/workspace-fs-at/{workspace_fs_at.cc,binding.gyp}`, `backend/src/native/workspace-fs-at-binding.d.ts`, `backend/src/workspaces/workspace-fs-at.ts`, and `backend/scripts/build-workspace-fs-at.mjs` / the repo-owned Node-API v8 `WorkspaceFsAtV1` seam with typed `openat`/`mkdirat`/no-follow `fstatat`/directory-fsync/close ownership, exact `LockFileName = "writer.lock" | "session-readers.lock"`, `openOrCreateLockAt(...)`, and `flockOwnedLock(handle, ownership, wait)`; exact `fs-ext@2.1.1` remains wrapper-internal and `flock(2)`-only; +- `backend/src/workspaces/workspace-lock-root-lease.ts` / `CanonicalWorkspaceLockRootInput`, `BorrowedVerifiedWorkspaceLockRootLease`, `VerifiedWorkspaceLockRootLease`, `WorkspaceSessionReadersLockLease`, `VerifiedWorkspaceLockRootLease.acquireSessionReadersShared()`, `VerifiedWorkspaceLockRootLeaseFactory.acquire`/`.acquireOrProvision`, and the retained root identity, all consuming only that typed FD-relative seam; +- `backend/src/workspaces/preprocessing-state.ts` / `PreprocessingStateStore`, `BorrowedWorkspaceSessionReadersExclusiveLockLease`, `WorkspaceWriterLockCapability.runUnderSessionReadersExclusive(...)`, `OrderedWorkspaceWriterCapabilitySet`, `runUnderWorkspaceWriterLock`, `runUnderOrderedWorkspaceWriterLocks`, and the exact inherited writer FD 3/root FD 4 contract; +- `backend/src/workspaces/registry-publication.ts` / the exact `RegistryAddressedRequestV1` bootstrap/pull union, `RegistryAddressedPlanV1` bootstrap/pull union, `RegistryAddressedPublicationStateV1` bootstrap/pull union, `RegistryBootstrapRecoveryIdentityV1`, `RegistryBootstrapRecoveryScanLimitsV1`, `RegistryActiveSnapshotV1`, and exact `RegistryEnsureBootstrapAddressedResultV1` (`already_active` snapshot or `bootstrap_terminal` result), phases `request_claimed` through `terminal_durable`, `AddressedWorkspacePublicationLeaseV1`, `CapabilityAwareRegistryPublicationParticipant`, `CapabilityAwareRegistryPublicationSynchronizer`, and `CapabilityAwareRegistryPublicationLifecycleOwner.run`; +- `backend/src/workspaces/registry.ts` / `WorkspaceRegistry.ensureBootstrapAddressed` as the sole bounded automatic bootstrap selector and `WorkspaceRegistry.publishAddressed` for explicit addressed mutation; +- `backend/src/routes/workspaces.ts` and `backend/src/app.ts` / inspect, lazy bootstrap, and status routed only through `ensureBootstrapAddressed`, pull and author publication routed only through `publishAddressed`, with no old `bootstrap`/`pull`/`activate`/direct-pointer escape; +- `backend/src/workspaces/preprocessing-service.ts` / `WorkspacePreprocessingService.execute`; +- the single `backend/src/workspace-maintenance.ts` / exported `main` entrypoint; +- `tools/thothctl/internal/workspaceops/operations.go` / `ParseWorkspaceCommand` and `Run`; +- P2 tests `workspace-fs-at-native.test.ts`, `workspace-lock-root-lease.test.ts`, `workspace-session-readers-lock.test.ts`, `workspace-runtime-config-lease.test.ts`, `workspace-preprocessing-state.test.ts`, `workspace-registry-addressed-publication.test.ts`, `workspace-registry-addressed-process.test.ts`, `routes-workspaces.test.ts`, `app.test.ts`, `workspace-preprocessing-service.test.ts`, and `workspace-maintenance.test.ts`; +- `scripts/p2-acceptance.sh`. + +No `backend/src/workspace-maintenance/` directory, `WorkspaceMaintenanceOperator`, `tools/thothctl/internal/workspace` package, `workspace.Parse`, or `workspace.Run` may be introduced: those names are nonexistent and conflict with the P2 handoff. + +Do not start P3 against the current pre-P2 tree. Do not implement P4 collection creation/rebuild, P5 Git annotation synchronization, P6 filesystem Evidence materialization, PSD migration, SSH runtime transport, GUI/API preprocessing, or aggregate P2–P6 verification. + +Preserve these invariants throughout: + +1. P2 reads/writes the existing `OWNER.json` schema-v1 contract unchanged until the explicit P3 migration code and tests exist. Capture a valid P2 schema-v1 fixture before changing the writer. +2. The reusable digest excludes secrets, credential values/files, Git commit, random config filename, `runtime_identity`, `session_storage`, vector/embedding/search/execution settings, and revision-qualified output paths. +3. The digest includes every non-secret value that can change physical/LSH output: DWH transport and canonical endpoint identity, database/schema and non-secret login identity, examples/introspection, eligibility and LSH policies, plus an explicit artifact-layout policy version. +4. A content-only Git commit reuses a cache. A changed endpoint, transport, database, schema, or included policy gets another cache and never falls back to the old cache. +5. Runtime layout is exactly: + + ```text + /data/sessions// + sessions/ # workspace-global session manifests + memory/ # workspace-global canonical registry + preprocessing/dwh-cache// # reusable immutable generations + revisions/<40-hex>/readiness//READY.json + revisions/<40-hex>/dwh-snapshots//{artifacts,indexes}/ + revisions/<40-hex>/artifacts/ # annotations and revision-owned artifacts + revisions/<40-hex>/indexes/ # revision-owned indexes + revisions/<40-hex>/corpus/ # corpus generations + ACTIVE + ``` + +6. `paths.memory` is authoritative when present. Only a config that lacks it may use the compatibility fallback `paths.artifacts/memory`. +7. Schema/Evidence Qdrant reads, hashes, writes, lists, and deletes require a 40-hex revision in point identity and filter. Memory/solved records deliberately omit revision from identity and filters. +8. P3 is additive until workspace layout enablement. `preprocessing/layout-version.json` is one workspace-global version marker and never names a revision or binding. Readiness is immutable and keyed by the exact pair `(40-hex Git commit, 64-hex effective-DWH cache key)` at `revisions//readiness//READY.json`, but **at most one effective binding may ever become READY for a Git revision**. Before preparation or activation, securely enumerate that revision's readiness directory: an existing valid READY for the selected key may only byte-match; any valid READY for another key makes activation fail `effective_config_mismatch` before cache/snapshot/semantic writes. A changed effective binding therefore requires a new content commit/revision. Once the global marker exists, a session probes its harness-owned effective binding, selects only that exact readiness generation and binding-qualified DWH snapshot, and fails `migration_required` before Pi/session child spawn when it is absent. Prepare maintenance is available for an unready revision only when no different binding is already READY there, through the trusted future resolver and never through an active fallback. +9. Every mutating migration/activation child runs under P2's exact opaque `WorkspaceWriterLockCapability`; its sole `spawnChild` method passes the actual locked writer open file description as FD 3 and the same retained verified root directory open file description as FD 4. P3 extends P2's one closed request union in place. Before content access, the child validates FD 4 as the expected service-owned root, opens `preprocessing/writer.lock` relative to FD 4, proves that inode is FD 3, and proves FD 3 is the already-held exclusive lock. P3 introduces no root brand, verified-root string, second capability, direct spawn, or ambient/path authorization seam. Source/destination bytes are reverified, publication is atomic/idempotent, and legacy filesystem sources remain for rollback. Conflicting Memory registries or cache destinations fail closed. +10. The public non-activating `semantic-revision` command is inventory/rebuild-readiness-only: it persists exact legacy IDs/digests and verifies that current schema/Evidence sources are rebuildable, but performs **zero Qdrant replacement upserts and zero deletes**. Semantic publish-before-delete occurs only inside one quiesced activation transaction: after durable admission blocking and exclusive acquisition of the no-follow reader gate, reverify the persisted inventory and sources, publish and read back every replacement, publish/verify the global reader-layout marker on initial enablement, delete only unchanged exact legacy IDs, and publish that commit+binding READY last. `core` may remain running because the durable marker prevents new admission and the exclusive gate proves zero active session readers. No successful non-activation command can create a mixed legacy/replacement reader interval. +11. The four ordinary operations (`migrate_dwh_cache`, `migrate_memory`, `migrate_semantic_revision`, and `activate_revision_layout`) keep the selected-workspace **writer-first** lifecycle: the sole P2 factory `acquire`s the existing root, transfers it into `runUnderWorkspaceWriterLock`, then the still-live exact capability owns ordinary job/quiescence, exclusive reader-gate acquisition, participants/locked children, publication, reverse release, and root close. The client pre-generates the run ID; process death retains state and owner quiescence for exact same-ID dead-`core` resume. Only activation may publish Qdrant replacements, switch reader mode/publish the sole binding READY, or delete legacy IDs. +12. `registry_pull` is the exact repository-first P2 callback exception and never enters the ordinary writer-first function. It invokes only `WorkspaceRegistry.publishAddressed` with a P2 `registry_pull` create/resume literal whose validated installation boundary supplies `installationIdentitySha256`, `repositoryIdentitySha256`, and `remoteRefIdentitySha256`, and whose create branch also supplies `expectedBaseCommit`. Inspect, lazy list/bootstrap, and status instead invoke only `WorkspaceRegistry.ensureBootstrapAddressed`. Its continuously repository-locked selector first revalidates active state: a valid snapshot returns P2's exact `already_active` result without job scanning, state writes, or network; only absence enters bounded exact-identity zero-create/one-resume selection. Corrupt or incompatible active state, corrupt/mismatched/pull/multiple nonterminal jobs, and every identity/base mismatch fail closed before network or state mutation. Under `repository.lock`, P2 durably advances `request_claimed` → `target_advertised` → `target_fetched` → `planned`, then `acquireOrProvision`s every complete changed-ID root through the sole typed `WorkspaceFsAtV1` seam and passes all roots once to `runUnderOrderedWorkspaceWriterLocks`. The exact same callback-scoped set flows through the production lifecycle owner, per-workspace addressed participant leases, every synchronizer, pointer publication, and `terminal_durable`; reverse invalidation/release occurs before repository release. The same owner and callbacks remain valid for P2's no-active-base `registry_bootstrap` plan/result variant. Same-ID recovery is phase-exact: before a durable advertisement it may repeat advertisement; at/after `target_advertised` the recorded OID is permanent, exact-OID fetch/ref reconciliation is the only allowed recovery, and after `target_fetched` no network or target reselection occurs. `migrate_dwh_cache` remains pre-READY preparation and writes no global marker, semantic point, or READY. +13. Maintenance control is profile-independent and host-unpublished: `workspaceops.Run` invokes only the selected installation's existing profile-gated `docker compose run --rm --no-deps --no-TTY workspace-maintenance` job with bounded canonical stdin/result. It never calls Fastify, an HTTP/internal route, `compose exec core`, a frontend proxy, host Python/Node/Pi/`tht`, or a core stop/start command. Request/result bounds, lock order, no-follow files, deadlines, state transitions, and failure mapping are the exact contract frozen in Task 8. +14. Every admitted session owns P2's exact `WorkspaceSessionReadersLockLease` from the final quiescence recheck until all Pi/session children and their stdout/stderr streams have settled. `WorkspaceReaderLeaseFactory.acquireForSession` delegates only to `VerifiedWorkspaceLockRootLease.acquireSessionReadersShared()`, and the resulting owner is transferred into `PiProcessManager` for exactly-once teardown. Maintenance publishes/drains quiescence first, then `WorkspaceReaderLeaseFactory.acquireForMaintenance` delegates only to `WorkspaceWriterLockCapability.runUnderSessionReadersExclusive(...)` around the full quiesced action through terminal durability and matching owner clear; its borrowed lease never escapes the callback. P3 never opens/flocks directly, imports `WorkspaceFsAtV1`, `fs-ext`, or the raw addon for reader acquisition, exposes a directory handle/path/FD/lock name/flags, casts a handle/name, or adds another lock wrapper. `--json` remains pristine and all public errors use bounded stable codes, especially `effective_config_mismatch`, `migration_required`, `preprocessing_conflict`, `workspace_quiesced`, `registry_bootstrap_recovery_conflict`, and `semantic_index_incompatible`. + +## Automated process goal + +At implementation start, register this persistent goal if the agent runtime supports goals: + +> From a clean P3 source tree and clean Docker/fixture namespace, use only the released `thothctl` host interface to launch dedicated maintenance jobs that atomically create/resume client-addressed runs, durably block admission, drain shared reader leases, and hold the exclusive reader gate plus the exact P2 capability's writer FD 3 and retained-root FD 4 without Fastify or stopping `core`. Migrate a valid P2 schema-v1 DWH owner and legacy Memory registry; prove `dwh-cache` materializes the selected binding-qualified revision snapshot without READY activation; prove every non-activating command preserves one unmixed reader mode; then use one activation to publish/verify revision-scoped schema/Evidence replacements, switch the global layout, delete exact verified legacy IDs, publish immutable commit+binding READY, and owner-clear quiescence. Pull content-only revision B, block it until its exact READY, and reuse the cache while isolating revision state. Change binding X to Y at READY B, prove B admission and same-revision activation are refused without mutation, publish/pull C, and prove new v2 Y cache → C/Y snapshot → immutable C/Y READY. Restore installation binding X and admit historical B/X, then restore Y and admit C/Y, proving immutable B/X bytes rather than pinned-revision-only operability; retain global Memory/solved, reject conflicts, survive dead-core resume and SIGKILL, scan fixture secrets, and remove exactly owned resources. + +Keep the goal open until `./scripts/p3-acceptance.sh integration --keep` succeeds once from clean state with no automatic retry and its report has been verified. Focused tests are progress evidence, not completion of the process goal. + +### Controlled topology and report contract + +The P3 process test extends P2's private fixture topology: a bare local Git remote with revisions A/B/C, installation descriptor and fixture secret files, the real dedicated `workspace-maintenance` image/job, an optionally running or deliberately unavailable `core`, real Qdrant, real internal Ollama, and the P2 controlled REST-DWH fixture. No production credential, registry, volume, or network is permitted. The process owns a unique Compose project and `.artifacts/p3-effective-config//ownership.json`. + +`report.json` and `report.md` must record the clean source commit/tree, built image digest, fixture Git commits, workspace ID/revisions, safe canonical binding hashes, owner schema/digests, DWH ACTIVE generation and file digests, commit+binding readiness paths and binding-qualified snapshot identities, Qdrant counts by safe kind/revision, Memory registry/projection counts, command event results, secret-scan result, cleanup inventory/result, and overall PASS/FAIL. They must never contain endpoint credentials, fixture secret values, signed URLs, rendered config, raw child stderr, or unbounded logs. `--keep` retains the owned run for review; `cleanup --run ` later removes only resources in its ownership manifest while retaining the sanitized reports and a final cleanup report. + +There are no unavoidable human steps inside automation. Manual acceptance is a separate clean environment after automated PASS. + +--- + +### Task 1: Freeze P2 schema-v1 ownership and baseline the dependency + +**Files:** +- Create: `harness/tests/fixtures/dwh-owner-v1/README.md` +- Create: `harness/tests/fixtures/dwh-owner-v1/OWNER.json` +- Create: `harness/tests/fixtures/dwh-owner-v1/ACTIVE` +- Create: `harness/tests/fixtures/dwh-owner-v1/generations/11111111111111111111111111111111/{physical.yaml,analytics_lsh.pkl,analytics_minhashes.pkl,analytics_meta.json,generation-manifest.json}` +- Modify: `harness/tests/test_dwh_preprocess_job.py` + +**Step 1: Verify the P2 checkpoint before editing.** + +Run: + +```bash +git status --short +./scripts/p2-acceptance.sh integration --keep +(cd backend && npm run build:native && npx vitest run \ + test/workspace-fs-at-native.test.ts test/workspace-lock-root-lease.test.ts \ + test/workspace-session-readers-lock.test.ts \ + test/workspace-registry-addressed-publication.test.ts \ + test/workspace-registry-addressed-process.test.ts test/routes-workspaces.test.ts \ + test/app.test.ts && npx tsc --noEmit -p .) +cd harness && .venv/bin/pytest -q tests/test_dwh_preprocess_job.py +``` + +Expected: clean tracked tree before the retained P2 report is created; P2 report says `automated integration: PASS`; the repo-owned `workspace-fs-at` addon/root lease and automatic bootstrap recovery route/app/process suites pass with TypeScript green; and the existing DWH suite passes. If P2 has not been manually accepted, stop for the P2 checkpoint. + +**Step 2: Create the fixture through the unmodified P2 operational writer.** + +Use the P2 controlled config to run `tht preprocess dwh`, copy only the bounded owner/ACTIVE/generation files above, replace source-specific values with deterministic fixture values, recompute all declared SHA-256 values, and document the exact generation command in `README.md`. Do not hand-wave a structurally plausible owner. + +**Step 3: Write the RED compatibility test.** + +Add `test_schema_v1_fixture_is_a_valid_p2_owner_and_remains_readable` and `test_p2_writer_still_emits_schema_v1_before_explicit_migration` to `test_dwh_preprocess_job.py`. The first loads the fixture with the current reader and verifies all files/ACTIVE; the second asserts the P2 writer still emits `schema_version == 1` at this checkpoint. + +**Step 4: Run the focused tests.** + +```bash +cd harness && .venv/bin/pytest -q \ + tests/test_dwh_preprocess_job.py::test_schema_v1_fixture_is_a_valid_p2_owner_and_remains_readable \ + tests/test_dwh_preprocess_job.py::test_p2_writer_still_emits_schema_v1_before_explicit_migration +``` + +Expected: PASS. This is a characterization task, not production behavior change. + +**Step 5: Commit the fixture checkpoint.** + +```bash +git add harness/tests/fixtures/dwh-owner-v1 harness/tests/test_dwh_preprocess_job.py +git commit -m "test: preserve P2 DWH owner compatibility fixture" +``` + +--- + +### Task 2: Add the harness-owned versioned effective-DWH canonicalizer + +**Files:** +- Create: `harness/tht/effective_dwh.py` +- Create: `harness/tests/test_effective_dwh_binding.py` +- Modify: `harness/tht/config.py` +- Modify: `harness/tht/cli/config_cmd.py` +- Modify: `harness/tht/jobs/dwh_pipeline.py` + +**Step 1: Write RED tests for the exact canonical contract.** + +Create tests named: + +- `test_binding_is_canonical_versioned_secret_free_and_source_stable` +- `test_content_revision_temp_path_session_and_semantic_changes_do_not_change_binding` +- `test_each_dwh_output_affecting_field_changes_binding` +- `test_missing_registry_identity_cannot_claim_a_v2_cache` +- `test_effective_dwh_json_is_pristine_and_safe` + +Use parameterized mutations for transport, direct host/port/user, REST canonical base URL, database, schema, examples, eligibility, every LSH field, language if it affects generated descriptions, and `artifact_layout_version`. Include obvious fixture passwords/API keys and assert neither secret nor secret-file path appears in canonical JSON, CLI output, exceptions, or `repr`. + +Run: + +```bash +cd harness && .venv/bin/pytest -q tests/test_effective_dwh_binding.py +``` + +Expected: RED because `tht.effective_dwh` and `tht config effective-dwh` do not exist. + +**Step 2: Implement only the canonical contract.** + +In `effective_dwh.py`, define immutable models/constants and functions with these public names: + +```python +EFFECTIVE_DWH_SCHEMA_VERSION = 2 +CANONICALIZER_VERSION = "effective-dwh-v1" +ARTIFACT_LAYOUT_VERSION = "dwh-cache-v1" + +class EffectiveDwhBinding(BaseModel): + schema_version: Literal[2] + workspace_id: str + logical_source_identity: str + canonicalizer_version: Literal["effective-dwh-v1"] + effective_config_sha256: str + input_fingerprint: str + + +def canonical_effective_dwh_config(cfg: Config) -> dict[str, object]: ... +def effective_dwh_binding(cfg: Config) -> EffectiveDwhBinding: ... +def effective_dwh_binding_json(cfg: Config) -> str: ... # sort_keys, compact, trailing newline +def effective_dwh_cache_key(binding: EffectiveDwhBinding) -> str: ... # 64 lowercase hex +def effective_dwh_cache_root(cfg: Config) -> Path: ... +def legacy_schema_v1_binding(cfg: Config) -> dict[str, str]: ... +``` + +`canonical_effective_dwh_config` must build an explicit allowlist, normalize URLs/host case/default ports without DNS/network access, and reject query/userinfo/fragments. Never start with `cfg.model_dump()` and subtract fields. `effective_dwh_cache_root` appends the binding key to the configured cache **base**; it must reject missing registry identity and any symlink/path escape. + +Add `Config._logical_source_identity` if needed, but keep `_config_source` as the compatibility source used by `legacy_schema_v1_binding`. Registry configs set both to `workspace://`; legacy file configs retain their resolved filename only for schema-v1 compatibility. + +Add `tht config effective-dwh --json -c ` in `config_cmd.py`. It emits only the safe binding, cache key, and layout version. + +Replace `dwh_pipeline.config_dwh_binding` internals with a compatibility wrapper that delegates to `effective_dwh_binding` only after the new owner path is enabled in Task 4; until then it must keep schema-v1 behavior so the characterization test remains green. + +**Step 3: Run RED/GREEN tests.** + +```bash +cd harness && .venv/bin/pytest -q tests/test_effective_dwh_binding.py tests/test_config_resources.py +``` + +Expected: PASS; exact stdout from the JSON test parses as one object and contains no secret fixture. + +**Step 4: Lint the touched Python.** + +```bash +cd harness && .venv/bin/ruff check tht/effective_dwh.py tht/config.py tht/cli/config_cmd.py tht/jobs/dwh_pipeline.py tests/test_effective_dwh_binding.py +``` + +Expected: PASS. + +**Step 5: Commit.** + +```bash +git add harness/tht/effective_dwh.py harness/tht/config.py harness/tht/cli/config_cmd.py \ + harness/tht/jobs/dwh_pipeline.py harness/tests/test_effective_dwh_binding.py +git commit -m "feat: define canonical effective DWH binding" +``` + +--- + +### Task 3: Add global layout and immutable commit+binding readiness models without activation + +**Files:** +- Modify: `backend/src/workspaces/runtime-renderer.ts` +- Create: `backend/src/workspaces/revision-layout.ts` +- Modify: `backend/test/workspace-runtime-renderer.test.ts` +- Create: `backend/test/workspace-revision-layout.test.ts` +- Modify: `harness/tht/config.py` +- Modify: `harness/tht/paths.py` +- Create: `harness/tht/dwh_snapshot.py` +- Create: `harness/tests/test_dwh_snapshot.py` +- Modify: `harness/tests/test_config_resources.py` +- Modify: `harness/tests/test_portable_paths.py` + +This task is deliberately additive. Session and ordinary maintenance continue to render the exact P2 +workspace-global roots. No marker is written and no runtime consumer changes behavior here. + +**Step 1: Write RED model, secure-reader, and resolver tests.** + +Add exact future layout support for: + +```text +/data/sessions// + preprocessing/layout-version.json # workspace-global version only + preprocessing/dwh-cache// + revisions/<40-hex>/readiness//READY.json + revisions/<40-hex>/dwh-snapshots//artifacts/ACTIVE + revisions/<40-hex>/dwh-snapshots//artifacts/generations//physical.yaml + revisions/<40-hex>/dwh-snapshots//artifacts/generations//snapshot-manifest.json + revisions/<40-hex>/dwh-snapshots//indexes/generations//analytics_lsh.pkl + revisions/<40-hex>/dwh-snapshots//indexes/generations//analytics_minhashes.pkl + revisions/<40-hex>/dwh-snapshots//indexes/generations//analytics_meta.json + revisions/<40-hex>/corpus/ + memory/ + sessions/ +``` + +Freeze `LayoutVersionMarkerV1` as an exact-key schema containing only `schemaVersion: 1`, +`layoutVersion: "revision-layout-v1"`, workspace ID, and enabled UTC time. It never contains an +active revision or binding. Freeze `RevisionReadyManifestV1` as an exact-key schema containing +workspace ID, 40-hex revision, descriptor blob, complete effective DWH binding plus its binding SHA +and 64-hex cache key, the binding-qualified DWH snapshot generation+manifest digest, Memory registry +digest/status, semantic inventory/replacement/deletion digests and status, preparing outer run ID, +and ready UTC time. Its canonical bytes are published exclusively at +`revisions//readiness//READY.json`; an existing file must byte-match after strict +reverification and is never replaced. Thus readiness is an immutable generation for one exact +`(commit, effective binding)` pair, and the sole session admission proof is the manifest selected by +the session's harness-reported cache key. The secure readiness-directory reader additionally enforces +at most one READY key per revision: a second valid key, an unsafe entry, or ambiguous directory state +fails closed. A same-commit installation binding change cannot create a sibling READY or snapshot; +activation returns `effective_config_mismatch` before mutation and the changed binding must be paired +with a new content commit/revision. + +Freeze these shared synchronous TypeScript exports in +`backend/src/workspaces/revision-layout.ts`: + +```ts +import type { + BorrowedVerifiedWorkspaceLockRootLease, + CanonicalWorkspaceId, + Revision40, + Sha256Hex, +} from "./workspace-lock-root-lease.js"; +export type WorkspaceLayout = "p2-global" | "revision-layout-v1"; +export type RevisionLayoutState = + | { layout: "p2-global"; ready: null } + | { layout: "revision-layout-v1"; ready: RevisionReadyManifestV1 | null }; +export function futureWorkspaceLayoutPaths( + rootLease: BorrowedVerifiedWorkspaceLockRootLease, + workspaceId: CanonicalWorkspaceId, + workspaceRevision: Revision40, +): RuntimePaths; +export function bindingQualifiedWorkspaceLayoutPaths( + futurePaths: RuntimePaths, effectiveDwhCacheKey: Sha256Hex, +): RuntimePaths; +export function workspaceRuntimePaths( + rootLease: BorrowedVerifiedWorkspaceLockRootLease, + workspaceId: CanonicalWorkspaceId, + workspaceRevision: Revision40, + state: RevisionLayoutState, +): RuntimePaths; +export function readRevisionLayoutState( + rootLease: BorrowedVerifiedWorkspaceLockRootLease, + expected: { + workspaceId: CanonicalWorkspaceId; workspaceRevision: Revision40; descriptorBlob: Revision40; + effectiveDwhCacheKey: Sha256Hex; + }, +): RevisionLayoutState; +``` + +`readRevisionLayoutState` is deliberately synchronous inside an already-active P2 root-lease borrow, +so `WorkspaceRuntimeConfigLeaseFactory.acquireSession` and `.acquireMaintenance` retain their released +signatures and no raw root string crosses the handoff. It is called only after the fixed read-only +harness binding probe has returned the exact cache key. The module uses P2's installation-bound +root-relative safe reader; it never derives/reopens an ambient path. It opens every component and leaf +with `O_NOFOLLOW`, `fstat`, owner/mode/nlink/regular-file and post-read identity checks, bounds both JSON +files, and closes every relative child FD while leaving the borrowed root owned by its caller. An absent global marker returns `p2-global`. A present invalid +marker fails closed. With a valid global marker, absence of the exact +`readiness//READY.json` returns `{layout: "revision-layout-v1", ready: null}` when +the exact key is absent; it never selects newest, the sole other-key entry, or a revision-only READY, +so session admission maps the absence to `migration_required`. The activation/prepare guard separately +reports `effective_config_mismatch` when that sole other-key READY exists. Multiple READY keys, an +unsafe entry, or a present invalid, descriptor-, revision-, or binding-mismatched selected READY fail +closed as corrupted state. No asynchronous wrapper, `readFile`, or symlink-following convenience API +is permitted. + +`futureWorkspaceLayoutPaths` is the trusted migration-destination resolver. It validates the +already verified workspace root, workspace ID, and 40-hex revision, then derives binding-independent +future roots without reading the global marker, READY, current rendered config, or any P2 +compatibility fallback. `bindingQualifiedWorkspaceLayoutPaths` validates the harness-provided 64-hex +cache key and derives only the exact cache, DWH snapshot, and READY generation beneath those trusted +roots. Neither accepts a destination path from argv/stdin. `workspaceRuntimePaths` may return P2 paths +only when the global marker is absent; when layout v1 is enabled it uses the selected binding-qualified +resolver and never falls back because READY is absent. + +In `harness/tht/dwh_snapshot.py`, freeze: + +```python +@dataclass(frozen=True) +class DwhArtifactSnapshot: + generation_id: str + physical: Path + analytics_lsh: Path + analytics_minhashes: Path + analytics_meta: Path + manifest_sha256: str + + +def resolve_revision_dwh_snapshot(cfg: Config) -> DwhArtifactSnapshot: ... +def resolve_effective_dwh_inputs(cfg: Config) -> DwhArtifactSnapshot | None: ... +``` + +`resolve_revision_dwh_snapshot` securely validates the exact `ACTIVE`, generation directory, +manifest, modes/link counts, revision identity, and every declared digest. `resolve_effective_dwh_inputs` +is the only compatibility branch: when `paths.dwh_snapshot` is absent it returns `None`, telling +existing consumers to use unchanged P2 paths. No caller constructs a snapshot path itself. + +Run: + +```bash +(cd backend && npx vitest run \ + test/workspace-runtime-renderer.test.ts \ + test/workspace-revision-layout.test.ts) +(cd harness && .venv/bin/pytest -q \ + tests/test_config_resources.py tests/test_portable_paths.py tests/test_dwh_snapshot.py) +``` + +Expected: RED on missing models/resolver, while every existing P2 rendering assertion stays green. + +**Step 2: Implement additive models only.** + +Extend `RuntimePaths`/`PathsConfig` with optional `memory`, `corpus`, `dwh_cache`, and +`dwh_snapshot`. Keep P2 fallback semantics exactly for ordinary active rendering. Implement the +pure trusted future resolver and synchronous secure state reader, but do not call either from +`ThtRunner` or `WorkspaceRuntimeConfigLeaseFactory` yet. + +**Step 3: Run GREEN tests and the P2 regression boundary.** + +```bash +(cd backend && npx vitest run \ + test/workspace-runtime-renderer.test.ts \ + test/workspace-revision-layout.test.ts \ + test/workspace-runtime-config-lease.test.ts \ + test/workspace-preprocessing-service.test.ts \ + test/workspace-maintenance.test.ts && npx tsc --noEmit -p .) +(cd harness && .venv/bin/pytest -q \ + tests/test_config_resources.py tests/test_portable_paths.py tests/test_dwh_snapshot.py \ + tests/test_dwh_preprocess_job.py tests/test_lsh_job_resume.py) +``` + +Expected: PASS; session and maintenance still render byte-identical P2 YAML and equal schema-v1 +`config_dwh_binding()`. + +**Step 4: Commit.** + +```bash +git add backend/src/workspaces/runtime-renderer.ts backend/src/workspaces/revision-layout.ts \ + backend/test/workspace-runtime-renderer.test.ts backend/test/workspace-revision-layout.test.ts \ + harness/tht/config.py harness/tht/paths.py harness/tht/dwh_snapshot.py \ + harness/tests/test_config_resources.py harness/tests/test_portable_paths.py \ + harness/tests/test_dwh_snapshot.py +git commit -m "feat: add inactive revision layout models" +``` + +--- + +### Task 4: Add schema-v2 OWNER reading/writing without weakening schema-v1 validation + +**Files:** +- Create: `harness/tht/dwh_owner.py` +- Create: `harness/tests/test_dwh_owner_v2.py` +- Modify: `harness/tht/jobs/dwh_pipeline.py` +- Modify: `harness/tests/test_dwh_preprocess_job.py` + +**Step 1: Write RED owner tests.** + +Cover new empty binding-key cache schema 2; complete `EffectiveDwhBinding`; canonical +`binding_sha256`; strict exact keys/modes/UID/nlink; malformed ACTIVE/manifest; v1 compatibility +read-only; content-only revision reuse; and mismatch refusal without sibling-key fallback. + +```bash +(cd harness && .venv/bin/pytest -q tests/test_dwh_owner_v2.py tests/test_dwh_preprocess_job.py) +``` + +Expected: RED because v2 owner code does not exist. + +**Step 2: Implement strict dual readers and a v2-only writer.** + +```python +class OwnerSchemaV1(BaseModel): ... +class OwnerSchemaV2(BaseModel): ... + + +def read_owner_at(root_fd: int) -> OwnerSchemaV1 | OwnerSchemaV2: ... +def validate_owner_at(root_fd: int, expected: EffectiveDwhBinding) -> OwnerSchemaV2: ... +def write_owner_v2_at(root_fd: int, binding: EffectiveDwhBinding) -> None: ... +``` + +Select only by literal `schema_version`; never reinterpret v1 as v2. New v2 cache generations bind +the complete v2 owner digest. Existing P2 runtime roots remain selected until Task 9 activation. + +**Step 3: Run focused/regression suites and lint.** + +```bash +(cd harness && .venv/bin/pytest -q \ + tests/test_dwh_owner_v2.py tests/test_dwh_preprocess_job.py tests/test_search_pack.py && \ + .venv/bin/ruff check tht/dwh_owner.py tht/jobs/dwh_pipeline.py \ + tests/test_dwh_owner_v2.py tests/test_dwh_preprocess_job.py) +``` + +Expected: PASS, including the untouched valid schema-v1 fixture. + +**Step 4: Commit.** + +```bash +git add harness/tht/dwh_owner.py harness/tht/jobs/dwh_pipeline.py \ + harness/tests/test_dwh_owner_v2.py harness/tests/test_dwh_preprocess_job.py +git commit -m "feat: version DWH cache ownership" +``` + +--- + +### Task 5: Implement verified schema-v1 migration, prepare-mode DWH builds, and immutable binding snapshots + +**Files:** +- Create: `harness/tht/dwh_migration.py` +- Create: `harness/tests/test_dwh_owner_migration.py` +- Modify: `harness/tht/dwh_snapshot.py` +- Modify: `harness/tht/effective_dwh.py` +- Modify: `harness/tht/jobs/dwh_pipeline.py` +- Modify: `harness/tht/cli/config_cmd.py` +- Modify: `harness/tht/cli/preprocess_cmd.py` + +**Step 1: Write RED migration/lock tests.** + +Cover verified v1 copy with source preservation; prepare-mode introspection/LSH build when the exact +binding-key cache is absent or the v1 binding does not match; v2-only owner/manifests; idempotent full +reverification; mismatch/corruption/collision/partial destination refusal; copy/build/fsync/rename +fault injection; and exact binding-qualified snapshot layout/digests. Assert the migration source is +exactly the legacy +`/.tht-dwh` and destinations are exactly the cache/snapshot roots returned by +`futureWorkspaceLayoutPaths`, even while active P2 rendering still points at compatibility roots. +Reject a config-derived fallback destination and every caller-supplied destination. Require P2's +actual inherited writer FD 3 and retained-root FD 4: missing/closed descriptors, another writer/root inode, a cross-root pair, or a forged environment marker +without both descriptors, and ordinary direct internal CLI all fail `preprocessing_conflict`; SIGKILL releases +the parent Node lock and leaves no readable partial. Add the complete changed-binding boundary test: READY for revision B/binding X exists; installation +binding changes to Y; session admission refuses B/X and activation fails `effective_config_mismatch` +before DWH access, cache/snapshot creation, semantic calls, or READY publication. After a content-only +commit C is published and pulled, prepare mode builds and verifies Y's new v2 owner/cache and C/Y +binding-qualified snapshot; immutable B/X READY and snapshot remain byte-identical; C/Y READY is later +published. While installation binding Y is selected, B admission remains intentionally refused although B/X bytes are immutable. The test then explicitly restores installation binding X and admits B/X, restores Y, and admits C/Y; revision pinning alone is never claimed to restore a historical installation binding. + +```bash +(cd harness && .venv/bin/pytest -q tests/test_dwh_owner_migration.py) +``` + +Expected: RED. + +**Step 2: Implement migration with the fixed kernel-lock capability.** + +```python +@dataclass(frozen=True) +class DwhMigrationReport: ... + + +def migrate_schema_v1_cache(cfg: Config) -> DwhMigrationReport: ... +def prepare_effective_dwh_cache(cfg: Config) -> DwhMigrationReport: ... +def materialize_revision_dwh_snapshot(cfg: Config) -> DwhArtifactSnapshot: ... +``` + +Both functions begin with P2's `require_workspace_writer_lock(cfg)`, which validates the retained root directory, opens the canonical lock relative to FD 4, proves that inode is FD 3, and calls `fcntl.flock(3, LOCK_EX|LOCK_NB)`. There is no callback, boolean, path, independently reopened root, +or environment-marker seam. + +Derive the sole P2 migration source as `/.tht-dwh`; never accept a caller +path. The backend first performs the fixed read-only effective-binding probe, then acquires +maintenance with `layoutIntent: "prepare-revision-layout-v1"` and binds the returned key through +`bindingQualifiedWorkspaceLayoutPaths`. Python re-derives and asserts the configured cache and +snapshot roots equal `/preprocessing/dwh-cache/` and +`/revisions//dwh-snapshots//...`, never `effective_dwh_cache_root` under an +active P2 fallback. If the exact legacy schema-v1 binding matches, verify every generation/digest and +ACTIVE, copy to a temporary sibling of that trusted binding-key cache, write schema-v2 +owner/manifests, re-read through strict v2 code, rename, and fsync. Preserve source bytes. + +If no exact v2 cache exists and the legacy binding does not match, activation must call +`prepare_effective_dwh_cache`: run the existing DWH introspection and LSH pipeline in explicit +`prepare-revision-layout-v1` mode, writing only a temporary sibling of that binding-key cache; publish +`OWNER.json` schema v2, generation manifest and ACTIVE in the existing safe order; strictly re-read +the complete owner/binding/digests; then rename/fsync the cache. Prepare mode is allowed for an unready +commit+binding pair, requires the exact FD 3/FD 4 pair, cannot read or fall back to any old cache, and on failure removes +or quarantines only its unpublished temporary sibling while leaving legacy and other binding caches +unchanged. Existing exact destinations must fully reverify and return `already_prepared`; different +bytes at the same binding key fail closed. + +`materialize_revision_dwh_snapshot` writes the exact Task 3 binding-qualified generation layout. It +copies only the validated cache ACTIVE generation, writes and revalidates +`snapshot-manifest.json`, fsyncs, renames, then publishes that binding snapshot's `ACTIVE` last. It +never symlinks the revision to the cache and never overwrites another binding snapshot. + +Add fixed internal `tht config migrate-dwh-cache --json -c `, `tht preprocess dwh +--prepare-revision-layout-v1 --json -c `, and `tht config materialize-dwh-snapshot --json -c +`. Direct invocation of any mutating operation without the actual inherited writer FD 3 and retained-root FD 4 fails +before source inspection or DWH access. + +**Step 3: Run focused tests and the P2 regression boundary.** + +```bash +(cd harness && .venv/bin/pytest -q \ + tests/test_dwh_owner_migration.py tests/test_dwh_owner_v2.py \ + tests/test_dwh_preprocess_job.py tests/test_search_pack.py tests/test_dwh_snapshot.py && \ + .venv/bin/ruff check tht/dwh_migration.py tht/dwh_snapshot.py tht/effective_dwh.py \ + tht/jobs/dwh_pipeline.py tht/cli/config_cmd.py tht/cli/preprocess_cmd.py \ + tests/test_dwh_owner_migration.py) +``` + +Expected: PASS; v1 source and P2 active paths remain byte-identical. + +**Step 4: Commit.** + +```bash +git add harness/tht/dwh_migration.py harness/tht/dwh_snapshot.py harness/tht/effective_dwh.py \ + harness/tht/jobs/dwh_pipeline.py harness/tht/cli/config_cmd.py \ + harness/tht/cli/preprocess_cmd.py harness/tests/test_dwh_owner_migration.py +git commit -m "feat: migrate legacy DWH caches explicitly" +``` + +--- + +### Task 6: Add revision semantic identities, inventory/readiness preparation, and activation-only publish/delete + +**Files:** +- Modify: `harness/tht/vectorstore/records.py` +- Modify: `harness/tht/adapters/vector/qdrant.py` +- Modify: `harness/tht/ports/vector.py` +- Modify: `harness/tht/adapters/factory.py` +- Modify: `harness/tht/search/evidence.py` +- Create: `harness/tht/semantic_migration.py` +- Modify: `harness/tht/cli/vector_cmd.py` +- Create: `harness/tests/test_semantic_revision_migration.py` +- Modify: `harness/tests/test_qdrant_vector_store.py` +- Modify: `harness/tests/test_semantic_kind_isolation.py` +- Modify: `harness/tests/test_qdrant_cli_commands.py` +- Modify: `harness/tests/test_corpus_pipeline.py` +- Modify: `harness/tests/test_memory_save_one.py` +- Modify: `harness/tests/test_solved_question.py` + +**Step 1: Write RED scoped identity/filter tests.** + +Test every operation: schema/Evidence IDs and filters include exact workspace+40-hex revision; +Memory/solved remain workspace-only; mixed searches partition and merge deterministically; caller +namespace conflicts fail before network; and revisionless points report `migration_required`. +Runtime selection stays in P2 legacy mode until Task 9 publishes the global layout version and the +exact A+binding READY. The public non-activating semantic command may only inventory legacy points +and prove canonical schema/Evidence inputs are rebuildable; it must not call Qdrant upsert or delete. +Only Task 9 activation may write/verify revision-scoped targets. After global enablement, an unready +revision/binding never falls back to P2. + +**Step 2: Write RED publish-before-delete/resume tests.** + +Freeze semantic state phases in the P2 `PreprocessingStateStore` record: +`legacy_inventory_persisted`, `semantic_sources_ready`, `replacement_published`, +`replacement_verified`, `legacy_delete_complete`. The non-activating command can reach only the first +two; the last three are activation-only. The inventory contains each exact legacy schema/Evidence point ID and the +SHA-256 of canonical bounded payload bytes, plus the inventory digest and source artifact/corpus +digests. + +Tests must prove: + +- after every successful non-activation command, a newly admitted legacy-mode test session observes + no revision-scoped replacement points and Qdrant's upsert/delete call counts remain exactly zero; +- only activation can invoke schema/Evidence replacement upserts; readback verification completes + before any delete call; +- embed/upsert/verification failure deletes zero legacy IDs; +- unavailable Evidence canonical source returns `migration_required` and deletes zero IDs; +- resume after replacement verification does not re-embed verified points; +- final deletion addresses only inventory IDs and first re-reads each legacy payload digest; +- changed/missing legacy payload stops with conflict and does not broaden deletion; +- crash during deletion resumes the exact remaining set; +- Memory/solved and other workspaces/revisions are never listed or deleted. + +```bash +(cd harness && .venv/bin/pytest -q \ + tests/test_qdrant_vector_store.py tests/test_semantic_kind_isolation.py \ + tests/test_qdrant_cli_commands.py tests/test_corpus_pipeline.py \ + tests/test_memory_save_one.py tests/test_solved_question.py \ + tests/test_semantic_revision_migration.py) +``` + +Expected: RED. + +**Step 3: Implement the exact contracts.** + +```python +def point_id( + workspace_id: str, + kind: str, + record_key: str, + workspace_revision: str | None = None, +) -> str: ... + +@dataclass(frozen=True) +class LegacySemanticPoint: + point_id: str + payload_sha256: str + kind: Literal["schema", "evidence"] + + +def inventory_legacy_semantic_points(cfg: Config) -> SemanticLegacyInventory: ... +def verify_semantic_rebuild_readiness(cfg: Config, inventory: SemanticLegacyInventory) -> SemanticReadinessReport: ... +def publish_semantic_replacements(cfg: Config, inventory: SemanticLegacyInventory) -> SemanticMigrationReport: ... +def verify_semantic_replacements(cfg: Config, inventory: SemanticLegacyInventory) -> SemanticMigrationReport: ... +def delete_confirmed_legacy_semantic_points(cfg: Config, inventory: SemanticLegacyInventory) -> SemanticMigrationReport: ... +``` + +Use UUIDv5 input `thothii::::` for schema/Evidence +and the existing workspace-global form for Memory. Apply identity filters to search, hashes, scroll, +list, and delete. Preparation begins by verifying the inherited writer FD 3 and retained-root FD 4 pair. The fixed Python `inventory` +phase returns a bounded exact legacy inventory; `WorkspacePreprocessingService.execute` persists +those exact bytes and digest through `PreprocessingStateStore`. The fixed `readiness` phase opens and +hashes the verified DWH snapshot and canonical Evidence corpus/source, proves they are complete and +rebuildable, and returns only bounded counts/digests. It has no vector-store writer dependency and +must be proven incapable of Qdrant upsert/delete. A successful released `semantic-revision` command +stops after persisting `semantic_sources_ready`, durably completes and owner-clears quiescence while +still holding the exclusive reader gate, then admits a test legacy reader that sees exactly the +pre-command legacy set. + +Only activation may invoke fixed `publish` and `verify` phases. With durable quiescence installed +and the exclusive reader gate held, they reverify the persisted inventory/readiness bytes, rebuild current-revision schema from the binding-qualified DWH +snapshot and Evidence from the canonical corpus/source, then upsert/read back every replacement +identity/hash. Only after the backend durably records `replacement_verified` and publishes/verifies +the initial global reader marker may fixed `delete-confirmed` receive the exact persisted inventory +over bounded child stdin, re-read each legacy digest, and delete those IDs. Python never imports or +impersonates the TypeScript store, and no phase accepts an arbitrary state path. No phase creates or +deletes the collection. + +**Step 4: Run tests and lint.** + +```bash +(cd harness && .venv/bin/pytest -q \ + tests/test_qdrant_vector_store.py tests/test_semantic_kind_isolation.py \ + tests/test_qdrant_cli_commands.py tests/test_corpus_pipeline.py \ + tests/test_memory_save_one.py tests/test_solved_question.py \ + tests/test_semantic_revision_migration.py && \ + .venv/bin/ruff check tht/vectorstore/records.py tht/adapters/vector/qdrant.py \ + tht/ports/vector.py tht/adapters/factory.py tht/search/evidence.py \ + tht/semantic_migration.py tht/cli/vector_cmd.py \ + tests/test_semantic_revision_migration.py) +``` + +Expected: PASS; failure-before-delete assertions observe zero deletion calls. + +**Step 5: Commit.** + +```bash +git add harness/tht/vectorstore/records.py harness/tht/adapters/vector/qdrant.py \ + harness/tht/ports/vector.py harness/tht/adapters/factory.py harness/tht/search/evidence.py \ + harness/tht/semantic_migration.py harness/tht/cli/vector_cmd.py \ + harness/tests/test_semantic_revision_migration.py \ + harness/tests/test_qdrant_vector_store.py harness/tests/test_semantic_kind_isolation.py \ + harness/tests/test_qdrant_cli_commands.py harness/tests/test_corpus_pipeline.py \ + harness/tests/test_memory_save_one.py harness/tests/test_solved_question.py +git commit -m "feat: migrate semantic state publish before delete" +``` + +--- + +### Task 7: Add the explicit Memory root and migrate one canonical registry without activating it + +**Files:** +- Create: `harness/tht/memory_migration.py` +- Create: `harness/tests/test_memory_migration.py` +- Modify: `harness/tht/cli/memory_cmd.py` +- Modify: `harness/tht/memory.py` +- Modify: `harness/tht/solved.py` +- Modify: `harness/tests/test_memory_promotion.py` +- Modify: `harness/tests/test_memory_save_one.py` +- Modify: `harness/tests/test_solved_search_cli.py` +- Modify: `harness/tests/test_repository_memory_sql_paths.py` + +**Step 1: Write RED path/migration tests.** + +Cover all Memory commands; solved global identity; compatibility fallback only when `paths.memory` +is absent; exact known P2/revision candidates; zero/one/equivalent/conflicting registries; unsafe +file cases; fault injection; idempotence; source preservation; and rebuild of only Memory/solved +projections. Assert every known legacy source separately from the exact future destination +`/memory` returned by `futureWorkspaceLayoutPaths`; a P2 +`paths.artifacts/memory` fallback must never become the destination. The P2-rendered active config +still lacks explicit `paths.memory`, so ordinary runtime behavior remains on its legacy root here. + +```bash +(cd harness && .venv/bin/pytest -q \ + tests/test_memory_migration.py tests/test_memory_promotion.py \ + tests/test_memory_save_one.py tests/test_solved_search_cli.py) +``` + +Expected: RED. + +**Step 2: Centralize future and fallback paths.** + +```python +def memory_root(cfg: Config) -> Path: ... +def registry_path(cfg: Config) -> Path: ... +class MemoryRegistryLock: ... +``` + +Every command uses these helpers. Explicit `paths.memory` selects the future global root; absence +selects exactly the P2 fallback. Do not change Memory IDs or merge semantics. + +**Step 3: Implement migration with the actual writer FD 3 and retained-root FD 4.** + +```python +@dataclass(frozen=True) +class MemoryMigrationReport: ... + + +def migrate_memory_root(cfg: Config) -> MemoryMigrationReport: ... +def rebuild_global_memory_projection(cfg: Config) -> dict[str, int]: ... +``` + +Derive source candidates from the verified workspace root, never caller paths. The backend passes a +`layoutIntent: "prepare-revision-layout-v1"` maintenance config derived only from +`futureWorkspaceLayoutPaths`; Python re-derives and asserts that `paths.memory` is exactly the future +workspace-global `/memory` destination before writing or rebuilding projections. Begin with +`require_workspace_writer_lock(cfg)`. Validate/canonicalize every `MemoryRecord`; conflicting +semantic content fails without merge. Publish the future canonical JSONL atomically and reverify, +then rebuild only global Memory/solved projections from that exact registry. Add fixed internal +`tht memory migrate-root --json -c `; missing/closed/substituted FD fails. + +**Step 4: Run tests and lint.** + +```bash +(cd harness && .venv/bin/pytest -q \ + tests/test_memory_migration.py tests/test_memory_promotion.py tests/test_memory_save_one.py \ + tests/test_solved_search_cli.py tests/test_solved_question.py \ + tests/test_repository_memory_sql_paths.py && \ + .venv/bin/ruff check tht/memory_migration.py tht/memory.py tht/solved.py \ + tht/cli/memory_cmd.py tests/test_memory_migration.py) +``` + +Expected: PASS; P2 fallback paths remain green. + +**Step 5: Commit.** + +```bash +git add harness/tht/memory_migration.py harness/tht/memory.py harness/tht/solved.py \ + harness/tht/cli/memory_cmd.py harness/tests/test_memory_migration.py \ + harness/tests/test_memory_promotion.py harness/tests/test_memory_save_one.py \ + harness/tests/test_solved_search_cli.py harness/tests/test_repository_memory_sql_paths.py +git commit -m "feat: migrate workspace-global memory state" +``` + +--- + +### Task 8: Expose registry pull and durable quiesced migrations through the exact P2 one-shot API + +**Files:** +- Reuse unchanged from accepted P2: `backend/src/workspaces/workspace-fs-at.ts` +- Reuse unchanged from accepted P2: `backend/src/native/workspace-fs-at-binding.d.ts` +- Re-run unchanged P2 ownership test: `backend/test/workspace-fs-at-native.test.ts` +- Reuse unchanged from accepted P2: `backend/src/workspaces/workspace-lock-root-lease.ts` +- Re-run unchanged P2 ownership tests: `backend/test/workspace-lock-root-lease.test.ts`, `backend/test/workspace-session-readers-lock.test.ts` +- Modify only for the P3 child-request union (not reader locking): `backend/src/workspaces/preprocessing-state.ts` +- Modify: `backend/src/workspaces/preprocessing-service.ts` +- Modify: `backend/src/workspace-maintenance.ts` +- Modify: `backend/src/workspaces/registry.ts` +- Modify: `backend/src/workspaces/registry-publication.ts` +- Create: `backend/src/workspaces/registry-pull-job.ts` +- Create: `backend/test/registry-pull-job-imports.compile.ts` +- Modify: `backend/test/workspace-registry.test.ts` +- Modify: `backend/test/workspace-registry-addressed-publication.test.ts` +- Modify: `backend/test/workspace-registry-addressed-process.test.ts` +- Modify: `backend/test/fixtures/workspace-lock-root-worker.mjs` +- Modify: `backend/test/fixtures/workspace-registry-addressed-worker.mjs` +- Create: `backend/src/workspaces/workspace-reader-lease.ts` +- Modify: `backend/src/routes/workspaces.ts` +- Modify: `backend/src/routes/sessions.ts` +- Modify: `backend/src/app.ts` +- Modify: `backend/test/app.test.ts` +- Modify: `backend/src/pi/pi-process-manager.ts` +- Modify: `backend/test/workspace-preprocessing-state.test.ts` +- Modify: `backend/test/workspace-preprocessing-service.test.ts` +- Modify: `backend/test/workspace-maintenance.test.ts` +- Create: `backend/test/workspace-reader-lease.test.ts` +- Modify: `backend/test/routes-workspaces.test.ts` +- Modify: `backend/test/routes-sessions.test.ts` +- Modify: `backend/test/pi-process-manager.test.ts` +- Create: `harness/tht/locked_child_stdin.py` +- Create: `harness/tht/layout_markers.py` +- Modify: `harness/tht/cli/config_cmd.py` +- Modify: `harness/tht/cli/vector_cmd.py` +- Modify: `harness/tht/semantic_migration.py` +- Create: `harness/tests/test_locked_child_stdin.py` +- Create: `harness/tests/test_layout_marker_commands.py` +- Create: `harness/tests/test_p3_internal_cli.py` +- Modify: `harness/tests/test_semantic_revision_migration.py` +- Modify: `harness/tests/test_qdrant_cli_commands.py` +- Modify: `tools/thothctl/internal/workspaceops/operations.go` +- Modify: `tools/thothctl/internal/workspaceops/operations_test.go` +- Modify: `tools/thothctl/cmd/thothctl/main.go` +- Modify: `tools/thothctl/cmd/thothctl/main_test.go` +- Modify: `deploy/compose.git-https.yaml` +- Modify: `deploy/compose.git-ssh.yaml` +- Modify: `scripts/generate-connector-secrets-override.sh` +- Modify: `scripts/test-preprocess-compose-config.sh` + +**Step 1: Write RED public-command, exact-P2 registry, route, provisioning, and compile-import tests.** + +Add these released public commands, with no aliases: + +```text +thothctl --installation workspace registry pull + --workspace [--resume <32-hex-outer-run-id>] [--json] +thothctl --installation workspace migrate dwh-cache + --workspace [--resume <32-hex-outer-run-id>] [--json] +thothctl --installation workspace migrate memory + --workspace [--resume <32-hex-outer-run-id>] [--json] +thothctl --installation workspace migrate semantic-revision + --workspace --yes [--resume <32-hex-outer-run-id>] [--json] +``` + +Before a fresh call launches Compose, `thothctl` obtains exactly 16 bytes from `crypto/rand`, formats +one 32-lowercase-hex outer run ID, builds the canonical request/digest below, and retains that ID in +every result or error. For these four released pull/migration commands, `--resume` uses exactly the +caller-supplied ID; no newest-run, digest, marker, or automatic selection exists. P2's separate +empty-registry inspect/lazy-list/status bootstrap path remains the sole automatic-recovery exception via +`ensureBootstrapAddressed`. The four ordinary operations reject wrong workspace, operation, active +revision/descriptor, selected binding, or request digest before mutation. `registry_pull` is different: +the outer request addresses the P2 job, and all base/target authority comes from the exact durable P2 +`registry_pull` state. An ambiguous Compose exit/timeout/truncated result returns the known ID and exact +`--resume` instruction; it never allocates a replacement ID. + +Freeze `RegistryPullCommand` and operation literal `registry_pull`. It alone receives writable registry +storage plus the selected HTTPS/SSH Git transport capability and calls only +`WorkspaceRegistry.publishAddressed`. It never acquires the selected workspace writer first. Every +ordinary migration gets a read-only registry and no Git credential. Rendered Compose tests inspect +mounts, environment, profile, image ID, and command; `inspect` cannot pull an existing registry. + +The RED registry matrix must consume P2's exact request/plan/state/callback surface rather than shadow +it. Test both `registry_bootstrap` (no base; changed set is all target IDs) and `registry_pull` (exact +base; changed set is the lexical symmetric base/target identity difference). Test every phase in order: +`request_claimed`, `target_advertised`, `target_fetched`, `planned`, `participants_prepared`, +`publication_intent_durable`, `target_published`, `terminal_durable`. Test the exact +`addressed-publication-jobs/.json` path and reject the removed pull-only job-directory spelling. +Require repository lock before the claim/network/pin sequence; `acquireOrProvision` for every complete +changed ID while the repository lock is held; one call to `runUnderOrderedWorkspaceWriterLocks`; the +same callback-scoped capability set in the production lifecycle owner and every synchronizer; the same +addressed lease objects in participants; no reacquisition; and all-or-nothing pointer publication. + +Root tests include newly added and never-used removed IDs with absent leaves, concurrent creators, +symlink substitution, wrong owner/mode, parent replacement, failure before publication, retained unused +leaf, and successful same-ID retry. Re-run P2's unchanged native seam and +`workspace-session-readers-lock.test.ts` production ownership tests: exact +`LockFileName = "writer.lock" | "session-readers.lock"`, wrapper-only typed open/flock ownership, +`VerifiedWorkspaceLockRootLease.acquireSessionReadersShared()`, exact +`WorkspaceSessionReadersLockLease` transfer/close, and +`WorkspaceWriterLockCapability.runUnderSessionReadersExclusive(...)`. P3 RED tests compile the adapter +against those exact types, spy that each adapter method calls only its matching P2 method, and exercise +real contention through the production P2 methods rather than a shadow opener. Compile/source fences +reject any P3 `WorkspaceFsAtV1` dependency for reader acquisition, raw-addon/`fs-ext` import, direct +open/flock, directory handle, path, numeric FD, lock name/flags, cast, wrapper lease, public constructor, +or second lock factory. + +Route/app tests prove inspect, lazy list bootstrap, and status call only `ensureBootstrapAddressed`; +explicit pull and author-publication activation call only `publishAddressed`. Across every bootstrap +preterminal kill point, each actual product caller automatically resumes the sole exact-identity +nonterminal run under continuously held `repository.lock`; zero nonterminal jobs creates one fresh run. +Add a two-/three-process barrier test in which inspect, lazy list, and status all observe initial absence: +the first locked caller publishes exactly one bootstrap, while queued callers revalidate and receive the +same exact P2 `already_active` snapshot result without scanning zero into another create, state writes, +or network. Ordinary calls made after active state exists take the same branch. Corrupt/incompatible +active state and corrupt/mismatched/pull/multiple nonterminals return the exact fail-closed code before +network/state and never choose newest/mtime/lexical/OID. Each route/service adapter consumes P2's exact +`RegistryEnsureBootstrapAddressedResultV1` rather than assuming a terminal bootstrap result, and compile +imports pin its `RegistryActiveSnapshotV1` snapshot. The removed public/internal `bootstrap`, `pull`, +`activate`, and direct active-pointer writer are absent. Session routes use `canonicalInput(id)` plus +existing-leaf `acquire`, never provisioning. Application construction injects the sole P2 +`WorkspaceFsAtV1` into the installation-bound root factory and production lifecycle owner once; +`WorkspaceReaderLeaseFactory` receives neither that wrapper nor any filesystem operand. + +For the actual production pull call, add compile assertions on both literal branches and runtime spies +through the released create and resume commands. Create must pass the validated boundary's +`installationIdentitySha256`, `repositoryIdentitySha256`, `remoteRefIdentitySha256`, and +`expectedBaseCommit`; resume must pass all three identity digests and omit only +`expectedBaseCommit`. Omission, substitution, cross-installation/repository/ref reuse, and create-base +mismatch must fail before the first advertisement/network call and before addressed state creation or +transition. Assert zero calls to the network and state-mutation spies, and prove the host request cannot +supply or override any of these four values. + +Create `backend/test/registry-pull-job-imports.compile.ts` with one `import type` declaration from the +single exact pull-job module below and references to all six released names. In that same compile gate, +import P2's exact `LockFileName` from `workspace-fs-at.js` and exact +`RegistryActiveSnapshotV1`/`RegistryEnsureBootstrapAddressedResultV1` from +`registry-publication.js`; no alias or alternate module path is permitted: + +```ts +import type { + RegistryPullPublicJobRequestV1, + RegistryPullPhaseV1, + RegistryPullAddressedJobRequestV1, + RegistryPullParticipantStateV1, + RegistryPullSynchronizerStateV1, + RegistryPullJobStateV1, +} from "../src/workspaces/registry-pull-job.js"; +import type { LockFileName } from "../src/workspaces/workspace-fs-at.js"; +import type { + Revision40, + Sha256Hex, +} from "../src/workspaces/workspace-lock-root-lease.js"; +import type { + RegistryActiveSnapshotV1, + RegistryAddressedPublicationPhaseV1, + RegistryAddressedRequestV1, + RegistryEnsureBootstrapAddressedResultV1, + RegistryPullAddressedPublicationStateV1, +} from "../src/workspaces/registry-publication.js"; + +type Equal = + (() => T extends A ? 1 : 2) extends (() => T extends B ? 1 : 2) + ? (() => T extends B ? 1 : 2) extends (() => T extends A ? 1 : 2) + ? true + : false + : false; +type Assert = T; +type AllRegistryPullExports = readonly [ + RegistryPullPublicJobRequestV1, + RegistryPullPhaseV1, + RegistryPullAddressedJobRequestV1, + RegistryPullParticipantStateV1, + RegistryPullSynchronizerStateV1, + RegistryPullJobStateV1, +]; +type RegistryPullCreateRequestV1 = Extract< + RegistryPullAddressedJobRequestV1, + { readonly mode: "create" } +>; +type RegistryPullResumeRequestV1 = Extract< + RegistryPullAddressedJobRequestV1, + { readonly mode: "resume" } +>; +type ExactP2HandoffSymbols = readonly [ + Assert>, + RegistryActiveSnapshotV1, + RegistryEnsureBootstrapAddressedResultV1, + Assert["snapshot"], + RegistryActiveSnapshotV1 + >>, +]; +type ExactP2Parity = readonly [ + Assert>, + Assert + >>, + Assert, + { + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + readonly expectedBaseCommit: Revision40; + } + >>, + Assert, + { + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + } + >>, + Assert, never>>, + Assert>, +]; +export type { AllRegistryPullExports, ExactP2HandoffSymbols, ExactP2Parity }; +``` + +Run: + +```bash +(cd tools/thothctl && test "$(go env GOVERSION)" = "go1.26.5" && \ + go test ./internal/workspaceops ./cmd/thothctl \ + -run 'Workspace|RegistryPull|Migrate|Resume|Capability|DedicatedJob' -v) +(cd backend && npx vitest run \ + test/workspace-lock-root-lease.test.ts test/workspace-session-readers-lock.test.ts \ + test/workspace-preprocessing-state.test.ts \ + test/workspace-preprocessing-service.test.ts test/workspace-maintenance.test.ts \ + test/workspace-registry.test.ts test/workspace-registry-addressed-publication.test.ts \ + test/workspace-registry-addressed-process.test.ts test/workspace-reader-lease.test.ts \ + test/routes-workspaces.test.ts test/routes-sessions.test.ts test/pi-process-manager.test.ts) +(cd backend && npx tsc --noEmit --target ES2022 --module ES2022 \ + --moduleResolution Bundler --strict --skipLibCheck \ + test/registry-pull-job-imports.compile.ts) +``` + +Expected: RED on new command members, reader lifecycle, P3 callback participants, root-provisioning +coverage, route regression, and the not-yet-created exact export module; existing P2 names themselves +compile unchanged. + +**Step 2: Export one exact RegistryPull type surface without forking P2 state.** + +Create exactly `backend/src/workspaces/registry-pull-job.ts`. It is the sole module that exports the six +`RegistryPull*` job names consumed by P4-P6. It imports P2's brands and addressed types; it declares no +second workspace/revision/SHA/run-ID brand and no second durable job shape: + +```ts +// backend/src/workspaces/registry-pull-job.ts +import type { + CanonicalWorkspaceId, + Sha256Hex, +} from "./workspace-lock-root-lease.js"; +import type { + RegistryAddressedPublicationPhaseV1, + RegistryAddressedRequestV1, + RegistryPullAddressedPublicationStateV1, + RegistryRunId32, +} from "./registry-publication.js"; + +export interface RegistryPullPublicJobRequestV1 { + readonly mode: "create" | "resume"; + readonly operation: "registry_pull"; + readonly requestSha256: Sha256Hex; + readonly runId: RegistryRunId32; + readonly schemaVersion: 1; + readonly workspaceId: CanonicalWorkspaceId; // result selector, never the changed set +} +export type RegistryPullPhaseV1 = RegistryAddressedPublicationPhaseV1; +export type RegistryPullAddressedJobRequestV1 = Extract< + RegistryAddressedRequestV1, + { readonly operation: "registry_pull" } +>; +export interface RegistryPullParticipantStateV1 { + readonly workspaceId: CanonicalWorkspaceId; + readonly baseWorkspaceIdentitySha256: Sha256Hex | null; + readonly targetWorkspaceIdentitySha256: Sha256Hex | null; + readonly preparedSha256: Sha256Hex | null; +} +export interface RegistryPullSynchronizerStateV1 { + readonly synchronizerId: string; + readonly preparedSha256: Sha256Hex | null; +} +export type RegistryPullJobStateV1 = RegistryPullAddressedPublicationStateV1; +``` + +`RegistryPullPhaseV1`, `RegistryPullAddressedJobRequestV1`, and `RegistryPullJobStateV1` are deliberate +exact aliases to P2's released types, not parallel brands or persisted wrappers. Thus the pull state is +field-for-field P2: `jobArtifactPath`; operation/base fields; `installationIdentitySha256`, +`repositoryIdentitySha256`, and `remoteRefIdentitySha256`; nullable `advertisedTargetCommit`, `immutableTargetRef`, +`fetchedTargetCommit`, target/changed-plan fields and digests before their phases; and all publication, +terminal, and prior-state digests. `RegistryPullParticipantStateV1` and +`RegistryPullSynchronizerStateV1` are bounded callback receipts used to compute P2's aggregate digests; +they never create another job artifact or add fields to `RegistryPullJobStateV1`. + +The compile-import test is a release gate and its bidirectional `Equal` assertions are the exact P2 symbol/field parity fence. Add an AST/source-boundary test that all six names have one +exporting module, that consumers import that exact `.js` path, and that none is redeclared in +`preprocessing-state.ts`, `registry-publication.ts`, or P4-P6. No barrel, compatibility export, or P5/P6 +alias is permitted. + +The public process request remains canonical compact JSON plus LF. `requestSha256` hashes exactly +`{"operation":"...","runId":"...","schemaVersion":1,"workspaceId":"..."}\n`; the stdin object has +exactly `mode,operation,requestSha256,runId,schemaVersion,workspaceId` in lexicographic order and is at +most 4,096 bytes. Go and TypeScript share golden vectors. Reject duplicate/reordered/unknown keys, BOM, +noncanonical whitespace/escaping, trailing bytes, invalid UTF-8, bad identities, and oversize input +before state access. + +The state locations are exactly: + +```text +/data/sessions//preprocessing/ + writer.lock + session-readers.lock + maintenance-quiescence.json + jobs/<32-hex-run-id>.json # four ordinary operations only +/ + repository.lock + addressed-publication-jobs/<32-hex-run-id>.json # exact P2 addressed state +/data/sessions//preprocessing/ + maintenance-quiescence.json # addressed callback owner + session-readers.lock +``` + +Do not create a pull-only job directory, a P3 registry state directory, or another active-pointer writer. + +**Step 3: Implement writer-first ordinary ownership and consume P2's repository-first callback.** + +`thothctl` invokes no backend transport. Fresh and resume commands launch only P2's selected, +immutable-image-pinned, profile-gated dedicated job: + +```text +docker compose ... run --rm --no-deps --no-TTY --name thoth-workspace-maintenance- workspace-maintenance --request-json-stdin +``` + +The job is runnable with `core` absent. No path uses `compose exec core`, curl/HTTP/internal routes, +Fastify, frontend, host Python/Node/Pi/`tht`, or a core stop/start command. + +The four ordinary operations retain this exact order: + +```text +VerifiedWorkspaceLockRootLeaseFactory.acquire(existing canonical selected ID) +-> transfer into runUnderWorkspaceWriterLock +-> exact WorkspaceWriterLockCapability callback (writer FD 3 + retained root FD 4) +-> ordinary job create/replay, owner quiescence publication, and reader drain +-> WorkspaceReaderLeaseFactory.acquireForMaintenance(writerCapability, full quiesced callback) +-> P2 runUnderSessionReadersExclusive holds LOCK_EX across final recheck, children, and publication +-> terminal durability and owner clear inside that same callback +-> callback settlement invalidates/closes reader borrow, then writer/root close +``` + +They must neither call `acquireOrProvision` nor enter `publishAddressed`. Implement +`WorkspaceReaderLeaseFactory` as a zero-filesystem adapter over P2's exact owner methods—without a P3 +verified-root alias or wrapper lease: + +```ts +import type { + VerifiedWorkspaceLockRootLease, + WorkspaceSessionReadersLockLease, +} from "./workspace-lock-root-lease.js"; +import type { + BorrowedWorkspaceSessionReadersExclusiveLockLease, + WorkspaceWriterLockCapability, +} from "./preprocessing-state.js"; + +export class WorkspaceReaderLeaseFactory { + acquireForSession( + rootLease: VerifiedWorkspaceLockRootLease, + ): Promise { + return rootLease.acquireSessionReadersShared(); + } + + acquireForMaintenance( + writerCapability: WorkspaceWriterLockCapability, + action: (lease: BorrowedWorkspaceSessionReadersExclusiveLockLease) => Promise, + ): Promise { + return writerCapability.runUnderSessionReadersExclusive(action); + } +} +``` + +There is no P3 `WorkspaceReaderLease`, `release()` wrapper, constructor, lock opener, or copied lifetime +logic. `acquireForSession` calls only P2's consuming `acquireSessionReadersShared()` and returns its exact +`WorkspaceSessionReadersLockLease`. `acquireForMaintenance` returns the exact result of +`writerCapability.runUnderSessionReadersExclusive(action)`; it never returns an owned maintenance lease, +and the exact `BorrowedWorkspaceSessionReadersExclusiveLockLease` may exist only inside `action`. +`WorkspaceReaderLeaseFactory` has no constructor dependency and never imports or receives +`WorkspaceFsAtV1`, a directory/regular-file handle, path, FD, lock name, flags, or mode. P2 alone opens, +flocks, invalidates, and closes the lock. + +The session route calls only `canonicalInput(id)` then existing-leaf `acquire(input)`, checks +quiescence, calls `acquireForSession(rootLease)`, and rechecks quiescence while holding that exact P2 +shared owner. `RuntimeOptions.sessionReadersLease` is typed as `WorkspaceSessionReadersLockLease`. +`PiProcessManager` synchronously calls `transfer()` at the ownership handoff and from then on closes its +owned lease exactly once only after every Pi/session child **and every stdout/stderr/read stream** has +settled. The route closes an untransferred owner on every pre-handoff failure; after transfer its source +close is the P2 idempotent no-op and the manager owns cleanup across configure/start failure, `exit`, +`close`, replacement, explicit teardown, shutdown, cancellation, and stream error. + +For ordinary maintenance, publish the owner-qualified quiescence marker and drain existing readers +first, then call `acquireForMaintenance(writerCapability, async (readerBorrow) => { ... })` exactly once. +That callback contains the complete quiesced action: final drain/admission-state recheck, every locked +child/participant, publication, terminal durability, and matching owner clear. It calls +`readerBorrow.assertLive()` at its protected boundaries and settles only after child stdout/stderr +collection settles. The P2 exclusive owner remains held for the callback's resolve/reject lifetime; the +borrow invalidates before return, and no existing reader is killed. + +Adapter tests use compile-time exact P2 imports and runtime spies to prove one call to each matching P2 +method, exact returned/result identity, no other call, and no maintenance-borrow escape. Production tests +reuse `workspace-session-readers-lock.test.ts`: acquire a real shared owner, transfer it into a fixture +`PiProcessManager`, keep a child plus stdout/stderr drains open, and prove the production exclusive +callback contends until all child/stream teardown and exactly-once close. Conversely, hold the production +exclusive callback around the full quiesced action and prove new shared admission contends, captured +borrows fail after settlement, callback failure closes once, and release permits admission. Source/type +fences reject a direct wrapper/raw-addon/`fs-ext` import, directory or file handle, path, numeric FD, lock +name/flags/mode, cast, direct open/flock, path fallback, wrapper lease, or alternate factory. + +`registry_pull` has no selected-workspace wrapper. Its production path first enters the validated +installation/registry boundary and obtains one immutable identity tuple. The boundary derives and +revalidates all four values against the retained installation descriptor, owned registry volume, +canonical repository, configured remote/ref, and current active snapshot; the host request cannot +supply or override any of them: + +```ts +const { + installationIdentitySha256, + repositoryIdentitySha256, + remoteRefIdentitySha256, + expectedBaseCommit, +} = await validatedInstallationRegistryBoundary.deriveRegistryPullIdentity(); + +const result = await registry.publishAddressed( + request.mode === "create" + ? ({ + mode: "create", + operation: "registry_pull", + runId: request.runId, + requestSha256: request.requestSha256, + installationIdentitySha256, + repositoryIdentitySha256, + remoteRefIdentitySha256, + expectedBaseCommit, + } satisfies Extract< + RegistryPullAddressedJobRequestV1, + { readonly mode: "create" } + >) + : ({ + mode: "resume", + operation: "registry_pull", + runId: request.runId, + requestSha256: request.requestSha256, + installationIdentitySha256, + repositoryIdentitySha256, + remoteRefIdentitySha256, + } satisfies Extract< + RegistryPullAddressedJobRequestV1, + { readonly mode: "resume" } + >), +); +``` + +These are the actual production call literals, not test-only examples: their two `satisfies` clauses are +compile gates against P2's exact request union, while runtime command/service tests spy on this call for +both modes. A create base mismatch, or any create/resume installation/repository/remote identity mismatch +with validated active state or a durable addressed record, fails before creating/transitioning addressed +state and before advertisement or other network. Do not redeclare P2's callback. Consume it exactly: +`CapabilityAwareRegistryPublicationLifecycleOwner.run({ plan, capabilities, participants, +synchronizers, action })`. Its `plan` is the P2 `RegistryAddressedPlanV1` union, so P3 participants and +synchronizers handle both `RegistryBootstrapAddressedPlanV1` and `RegistryPullAddressedPlanV1` by +discriminating `plan.operation`. Each participant receives exactly +`AddressedWorkspacePublicationLeaseV1`, including P2's borrowed retained root, writer capability, +quiescence, and reader lease. Each synchronizer receives the exact same +`OrderedWorkspaceWriterCapabilitySet`. No callback calls root acquisition, writer-lock acquisition, +reader acquisition, registry publication, or a nested lifecycle owner. + +For both addressed variants, P2 holds `repository.lock`, computes the exact complete changed IDs, calls +`acquireOrProvision(canonicalInput(id))` for each ID, and passes the full root array once to +`runUnderOrderedWorkspaceWriterLocks`. Missing added and never-used removed leaves are securely +provisioned under the retained parent FD with P2's fixed UID/mode/fsync/identity rules and retained on +failure. The factory consumes only P2's `WorkspaceFsAtV1` wrapper over +`backend/native/workspace-fs-at/workspace_fs_at.cc`; literal `openat`/`mkdirat`/no-follow `fstatat`, +directory fsync, owned close, the closed `"writer.lock" | "session-readers.lock"` open surface, typed +flock modes, flags, errors, Node 22 build, and Darwin/Linux behavior remain the exact P2 contract. The +wrapper's module-private synchronous numeric borrow is the only bridge to exact `fs-ext@2.1.1` and is +used internally for typed flock; P3 cannot import either underlying module, accept/return/borrow/cast an +FD, cast an owned handle/name, accept flags, implement another mkdir/open/flock factory, or introduce a +path fallback. Bootstrap requires no +active base, null base fields, `[]` base workspaces, and all target IDs. Pull requires the exact active +base and lexical symmetric base/target workspace-identity difference. Empty changed sets still use the +same lifecycle and publish/terminal protocol without inventing a selected lock. + +Product bootstrap recovery is automatic rather than a new public selector. Inspect, lazy list, and +status call only `ensureBootstrapAddressed(identity)` and consume P2's exact discriminated result union. +Under one continuously held `repository.lock`, the selector first reads and validates active state. A +valid compatible snapshot returns the exact `already_active` branch with that snapshot and performs no +job scan, state create/transition, or network; corrupt/incompatible active state fails closed. Only +validated absence no-follow scans one lexically sorted `addressed-publication-jobs/` snapshot with P2's +exact 4,096-entry, 1,048,576-byte-per-artifact, and 67,108,864-byte-total bounds and validates every +record before selection. Zero nonterminals creates one fresh ID; exactly one matching +`registry_bootstrap` nonterminal resumes that exact ID before any network. Unknown/corrupt/churning +entries, identity mismatch, nonterminal pull, or multiple nonterminals fail +`registry_bootstrap_recovery_conflict`; terminal jobs are ignored for automatic selection, and no +mtime/newest/lexical-last/OID/remote-head heuristic exists. A queued inspect/list/status caller never +acts on its pre-lock observation: after acquiring the lock it returns the new `already_active` snapshot +published by the winning caller. The bootstrap request digest excludes run ID/mode and binds schema, +operation, installation, repository, and remote-ref identity exactly. + +Fresh addressed create durably writes `request_claimed` before `ls-remote`, fetch, or any network call. +Then it advertises once and records the OID at `target_advertised`; exact-OID fetch creates only +`refs/thoth/addressed-runs//target`, verifies it equals the advertisement, and records +`target_fetched`; only then may manifest reading and the full plan produce `planned`. All writes use the +exact P2 sibling-write/file-fsync/rename/parent-fsync state transition and `priorStateSha256` chain. + +Same-ID recovery is split at the durable pin and tested literally: + +1. Before `request_claimed` rename is durable, no network was allowed; the same request may recreate + that claim. +2. From durable `request_claimed` until `target_advertised` is durable, no target was promised; resume + repeats advertisement and may observe a newer OID. +3. At/after durable `target_advertised`, that OID is permanent. If the immutable run ref is absent, + resume retries fetch of only that exact OID; if it already resolves to that OID, resume performs no + network and advances; a different OID is corruption. +4. At/after `target_fetched`, resume never advertises, fetches, consults remote target selection, or + accepts another OID. A terminal job replays its stored result even if a later independent pull moved + remote-tracking refs. Drift checks apply only while reconciling a nonterminal run. + +After `planned`, participant preparation and aggregate digests precede +`publication_intent_durable`. Publication stages immutable target records, atomically renames and +parent-fsyncs the installation-wide active pointer, rereads exact bytes, advances +`target_published`, and persists `terminal_durable` before callback settlement. Before pointer rename, +active is exact base (or absent for bootstrap). At/after it, same-ID resume accepts only exact recorded +base/target projections, converges all-base or mixed to target, recognizes all-target lost +acknowledgement, and refuses every third identity. It reuses the same full callback capability set. +Owner markers clear only after terminal durability while readers remain exclusive; then callback +borrows are invalidated and readers/quiescence/writers/roots release in reverse lexical order before +repository release. + +Kill/process tests cover before claim rename, after claim fsync, during/after advertisement before its +durable record, after advertisement fsync, during/after exact fetch, after immutable-ref creation before +`target_fetched`, after its fsync, after `planned`, before/after pointer file fsync/rename/parent fsync, +after target reread before phase persistence, and after terminal before clear. Assert exact same-ID +behavior at every boundary, no post-pin target reselection, no post-`target_fetched` network, terminal +replay despite a later tracking-ref move, and deterministic base/target/mixed recovery. Repeat every +bootstrap preterminal boundary through actual inspect, lazy-list, and status callers and prove automatic +same-ID selection occurs before network; exercise zero/one/multiple/pull/mismatched/corrupt/churning job +sets and all three P2 scan bounds. Repository/ref/installation identity change, a missing/changed pinned +object, third active identity, wrong owner/digest, unsafe state, or nonterminal clear fails closed without +publication, and automatic conflicts preserve exact `registry_bootstrap_recovery_conflict`. + +**Step 4: Extend P2's closed locked-child union in place and implement the one-shot lifecycle.** + +In `backend/src/workspaces/preprocessing-state.ts`, preserve P2's three variants and extend the same +exported alias—never create a parallel spawner—with these exact P3 contracts: + +```ts +type P3LockedOperation = + | "p3_migrate_dwh_cache" | "p3_prepare_dwh_cache" | "p3_materialize_dwh_snapshot" + | "p3_migrate_memory_root" | "p3_rebuild_memory_projection" + | "p3_inventory_semantic_legacy" | "p3_check_semantic_readiness" + | "p3_publish_semantic_replacements" | "p3_verify_semantic_replacements" + | "p3_delete_confirmed_semantic_legacy" | "p3_prepare_layout_markers" + | "p3_publish_layout_version" | "p3_publish_revision_ready" + | "p3_verify_revision_readiness"; +import type { + CanonicalWorkspaceId, + Revision40, + Sha256Hex, + WorkspaceLockRootIdentityV1, +} from "./workspace-lock-root-lease.js"; +interface P3LockedChildContextV1 { + schemaVersion: 1; + workspaceId: CanonicalWorkspaceId; + rootIdentity: WorkspaceLockRootIdentityV1; // from the consumed P2 lease; never argv/stdin + workspaceRevision: Revision40; + descriptorBlob: Revision40; + effectiveDwhBindingSha256: Sha256Hex; + effectiveDwhCacheKey: Sha256Hex; + outerRunId: string; // exactly 32 lowercase hex + configLease: MaintenanceRuntimeConfigLease; // opaque owned lease, not a caller path +} +interface P3LockedChildStdinV1 { + artifactKind: "semantic-legacy-inventory-v1" | "revision-ready-input-v1"; + contentBase64: string; + contentByteLength: number; + contentSha256: Sha256Hex; + descriptorBlob: Revision40; + effectiveDwhBindingSha256: Sha256Hex; + effectiveDwhCacheKey: Sha256Hex; + outerRunId: string; // exactly 32 lowercase hex + schemaVersion: 1; + stateArtifact: "semantic-legacy-inventory.json" | "revision-ready-input.json"; + workspaceId: CanonicalWorkspaceId; + workspaceRevision: Revision40; +} +type P3LockedChildRequest = + | { kind: "p3_migrate_dwh_cache"; context: P3LockedChildContextV1 } + | { kind: "p3_prepare_dwh_cache"; context: P3LockedChildContextV1 } + | { kind: "p3_materialize_dwh_snapshot"; context: P3LockedChildContextV1 } + | { kind: "p3_migrate_memory_root"; context: P3LockedChildContextV1 } + | { kind: "p3_rebuild_memory_projection"; context: P3LockedChildContextV1 } + | { kind: "p3_inventory_semantic_legacy"; context: P3LockedChildContextV1 } + | { kind: "p3_check_semantic_readiness"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 } + | { kind: "p3_publish_semantic_replacements"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 } + | { kind: "p3_verify_semantic_replacements"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 } + | { kind: "p3_delete_confirmed_semantic_legacy"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 } + | { kind: "p3_prepare_layout_markers"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 } + | { kind: "p3_publish_layout_version"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 } + | { kind: "p3_publish_revision_ready"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 } + | { kind: "p3_verify_revision_readiness"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 }; +export type WorkspaceLockedChildRequest = + | DwhLockedChildRequest | SchemaLockedChildRequest | EvidenceLockedChildRequest + | P3LockedChildRequest; +interface P3ResultBaseV1 { + schemaVersion: 1; workspaceId: CanonicalWorkspaceId; workspaceRevision: Revision40; + effectiveDwhCacheKey: Sha256Hex; status: "succeeded" | "unchanged" | "not_ready"; +} +type P3LockedChildParsedResult = + | (P3ResultBaseV1 & { operation: "p3_migrate_dwh_cache" | "p3_prepare_dwh_cache"; + ownerSha256: Sha256Hex; generationId: string; generationManifestSha256: Sha256Hex }) + | (P3ResultBaseV1 & { operation: "p3_materialize_dwh_snapshot"; + snapshotGenerationId: string; snapshotManifestSha256: Sha256Hex }) + | (P3ResultBaseV1 & { operation: "p3_migrate_memory_root"; + registrySha256: Sha256Hex; recordCount: number }) + | (P3ResultBaseV1 & { operation: "p3_rebuild_memory_projection"; + registrySha256: Sha256Hex; memoryPointCount: number; solvedPointCount: number }) + | (P3ResultBaseV1 & { operation: "p3_inventory_semantic_legacy"; + inventorySha256: Sha256Hex; inventoryByteLength: number; pointCount: number; + artifactKind: "semantic-legacy-inventory-v1"; contentBase64: string }) + | (P3ResultBaseV1 & { operation: "p3_check_semantic_readiness"; + inventorySha256: Sha256Hex; schemaSourceSha256: Sha256Hex; + evidenceSourceSha256: Sha256Hex; schemaCount: number; evidenceCount: number }) + | (P3ResultBaseV1 & { operation: "p3_publish_semantic_replacements" | + "p3_verify_semantic_replacements" | "p3_delete_confirmed_semantic_legacy"; + inventorySha256: Sha256Hex; replacementSetSha256: Sha256Hex; + schemaCount: number; evidenceCount: number; deletedLegacyCount: number }) + | (P3ResultBaseV1 & { operation: "p3_prepare_layout_markers"; + layoutMarkerInputSha256: Sha256Hex; readyInputSha256: Sha256Hex }) + | (P3ResultBaseV1 & { operation: "p3_publish_layout_version"; + layoutMarkerInputSha256: Sha256Hex; layoutMarkerSha256: Sha256Hex }) + | (P3ResultBaseV1 & { operation: "p3_publish_revision_ready"; + readyInputSha256: Sha256Hex; readyManifestSha256: Sha256Hex }) + | (P3ResultBaseV1 & { operation: "p3_verify_revision_readiness"; + readyInputSha256: Sha256Hex; readyManifestSha256: Sha256Hex }); +``` + +Only semantic operations accept `semantic-legacy-inventory-v1`; all four marker/readiness variants +accept only `revision-ready-input-v1`. Marker preparation returns canonical bounded layout/READY input +digests but writes neither marker. The coordinator then calls the two exact capability variants in +transaction order: `p3_publish_layout_version` before confirmed legacy delete and +`p3_publish_revision_ready` after deletion. Both publishers mutate only through inherited FD 4 while +FD 3 remains verified/held; readiness verification rereads the exact published bytes. Node never +publishes either marker by pathname. + +The child stdin wire is frozen byte-for-byte. It is one UTF-8 compact JSON object with **exactly** the +12 keys in lexicographic order `artifactKind,contentBase64,contentByteLength,contentSha256,descriptorBlob,effectiveDwhBindingSha256,effectiveDwhCacheKey,outerRunId,schemaVersion,stateArtifact,workspaceId,workspaceRevision`, no BOM, duplicate/reordered/unknown key, optional whitespace, alternate escaping, or trailing byte, followed by exactly one LF. `contentBase64` is strict padded RFC 4648 base64 of the canonical persisted artifact bytes; `contentByteLength` and `contentSha256` describe the decoded bytes, never the JSON/base64 text. The remaining identity fields must equal the locked request context, rendered config, and exact identity fields inside the decoded artifact. + +Inventory content is at most 716,800 decoded bytes and readiness content at most 65,536. With the +inherited canonical ASCII workspace-ID bound of 128 bytes and all fixed-width identities above, the +exact largest inventory stdin is `4*ceil(716800/3) + 745 = 956,481` bytes; the largest readiness stdin +is 88,118 bytes. Both are below the fixed 1,048,576-byte stdin cap. Node computes the envelope once +from state-store bytes, validates it before spawn, and writes those exact bytes without +`JSON.stringify(Uint8Array)` or a second serialization. No variant can represent raw argv, executable, +env, cwd, stdio, FD, config path, workspace destination, or arbitrary state path. + +The capability's exhaustive switch is the sole argv builder and emits exactly these internal argv +after the `tht` executable (where `CONFIG` comes only from the opaque lease): + +```text +p3_migrate_dwh_cache config migrate-dwh-cache --json -c CONFIG +p3_prepare_dwh_cache preprocess dwh --prepare-revision-layout-v1 --json -c CONFIG +p3_materialize_dwh_snapshot config materialize-dwh-snapshot --json -c CONFIG +p3_migrate_memory_root memory migrate-root --json -c CONFIG +p3_rebuild_memory_projection memory rebuild-projection --json -c CONFIG +p3_inventory_semantic_legacy vector semantic-inventory --json -c CONFIG +p3_check_semantic_readiness vector semantic-readiness --json -c CONFIG +p3_publish_semantic_replacements vector semantic-publish --json -c CONFIG +p3_verify_semantic_replacements vector semantic-verify --json -c CONFIG +p3_delete_confirmed_semantic_legacy vector semantic-delete-confirmed --json -c CONFIG +p3_prepare_layout_markers config prepare-layout-markers --json -c CONFIG +p3_publish_layout_version config publish-layout-version --json -c CONFIG +p3_publish_revision_ready config publish-revision-ready --json -c CONFIG +p3_verify_revision_readiness config verify-revision-readiness --json -c CONFIG +``` + +Implement `harness/tht/locked_child_stdin.py` now, not in Task 9, with one bounded binary-reader API +whose caller supplies only a compile-time literal expected artifact kind/state-artifact and decoded +limit. It requires actual writer FD 3 and retained-root FD 4 first; streams at most 1,048,576 stdin bytes; enforces the canonical +wire above; strict-base64 decodes; checks decoded length/digest; derives workspace, revision, +descriptor, and both binding identities from `Config`; and verifies outer-run/artifact identities +inside the decoded exact-key artifact. All four stdin-consuming semantic wrappers and all four marker/readiness +wrappers must call it; inventory has no stdin. Python negatives cover zero/two LF, BOM, invalid UTF-8, empty/truncated/oversize JSON, +duplicate/missing/unknown/reordered keys, whitespace/noncanonical escaping, bool-for-integer, invalid +base64/padding, decoded oversize, length/digest mismatch, wrong artifact/state/run/workspace/revision/ +descriptor/binding, content-identity mismatch, and missing/substituted/cross-root FD 3 or FD 4 before content use. + +Implement `harness/tht/layout_markers.py` and register all four exact Typer targets in +`harness/tht/cli/config_cmd.py` in this task. `config prepare-layout-markers` accepts only the frozen +`revision-ready-input-v1` envelope, validates the single-READY-per-revision rule, and returns the +canonical bounded layout/READY input digests without publication. `config +verify-revision-readiness` accepts the same contract and securely reopens the coordinator-published +marker and selected READY, checks exact bytes/digests/identities and rejects another READY key. Unit +tests cover help/registration, happy paths, strict stdin negatives, symlink/hardlink/replacement, +other-key READY, and no-write preparation. A table-driven Python test proves all fourteen argv targets +above resolve to the intended Typer command. A built-image test runs each target's fixed prefix with +`--help` in the selected `workspace-maintenance` image and compares the fourteen-target set exactly, so coordinator tests +cannot go GREEN against an unimplemented image command. + +All mutating commands inherit only the actual writer FD 3 and retained-root FD 4 from the exact P2 capability. Inventory content remains capped +at 716,800 decoded bytes, its RFC 4648 base64 is exactly at most 955,736 bytes, `pointCount` is bounded +`0..716800`, and canonical workspace ID is ASCII at most 128 bytes. With the exact inventory-result +keys/types above, compact lexicographically keyed JSON, maximum-width values, and exactly one LF, the +proved worst-case child stdout is `955736 + 581 = 956,317` bytes. Therefore inventory child stdout is +bounded to **1,048,576 bytes** (matching P2's 1 MiB boundary), not 921,600; every other variant remains +262,144 and stderr 65,536. A real bounded-collector round-trip test emits a 716,800-byte inventory, +observes exactly 956,317 serialized bytes at maximum-width fields, strictly parses/decodes it, and +compares all original bytes/digests; 1,048,576 total bytes is accepted and byte 1,048,577 terminates +the process group without retaining overflow. Timeouts are 60 seconds for inventory/readiness +verification, 300 seconds for migrations/snapshot/semantic verify/delete, 900 seconds for Memory +projection, and 1,800 seconds for DWH prepare and semantic publish. Overflow/timeout terminates the +child process group and returns a stable failed result. + +Parse exactly `P3LockedChildParsedResult` above; validate base identity, literal operation, SHA-256, +generation ID, integer counts, and status, and reject unknown keys, impossible per-operation fields, +identity mismatch, output other than the one canonical JSON object plus one LF, invalid +base64/decoded length, and public unsafe fields. Inventory `contentBase64` is decoded, hash/length +checked, persisted by `PreprocessingStateStore`, then removed before any public result encoding. + +Add TypeScript compile-time `satisfies Record` and runtime `assertNever` tests, +plus AST/source-boundary tests that fail on `spawn`, `exec`, `fork`, or raw argv outside the +capability. Exercise every variant's exact argv/stdin/result/timeout; missing/substituted/cross-root FD 3 or FD 4, +cross-workspace/revision/binding context, wrong artifact, post-settlement use, and arbitrary +path/argv constructions fail before spawn. Coordinator tests monkeypatch all general process spawn +APIs to throw and prove every migration still succeeds only through `capability.spawnChild`. + + +The exhaustive `WorkspaceWriterLockCapability.spawnChild` switch remains P2's sole process producer. +Every P3 variant passes the locked writer open description as child FD 3 and the same capability-owned +retained root directory open description as child FD 4. Before reading config/stdin or touching a +source/destination, the Python shared child guard verifies FD 4's expected root identity and ownership, opens +`preprocessing/writer.lock` relative to FD 4 with no-follow semantics, proves it is FD 3's inode, and +proves FD 3 is already exclusively held. Missing, closed, renumbered, independently locked, substituted, +cross-root, cross-workspace, or post-callback FD use fails `preprocessing_conflict`. No P3 raw root path, verified-root alias, capability wrapper/brand, environment marker, or direct helper spawn exists. + +On an ordinary failure/signal, retain state and owner quiescence, return the exact run ID, and require +same-ID resume; never wait for or restart `core`. Resume skips only phases whose exact persisted inputs +and outputs reverify. After every successful non-activation operation, owner-clear while the exclusive +reader lease remains held, then test a fresh admission. Before initial activation no revision-scoped +semantic replacement exists. After enablement a session selects only its currently probed exact +commit+binding READY; a second run cannot steal or clear the workspace. + +**Step 5: Run type/symbol/process gates and commit the complete release surface.** + +Run: + +```bash +(cd tools/thothctl && test "$(go env GOVERSION)" = "go1.26.5" && go test ./...) +(cd backend && npm run build:native && npx vitest run \ + test/workspace-fs-at-native.test.ts test/workspace-lock-root-lease.test.ts \ + test/workspace-session-readers-lock.test.ts \ + test/workspace-preprocessing-state.test.ts test/workspace-preprocessing-service.test.ts \ + test/workspace-maintenance.test.ts test/workspace-registry.test.ts \ + test/workspace-registry-addressed-publication.test.ts \ + test/workspace-registry-addressed-process.test.ts test/workspace-reader-lease.test.ts \ + test/routes-workspaces.test.ts test/routes-sessions.test.ts test/app.test.ts \ + test/pi-process-manager.test.ts && npx tsc --noEmit -p . && npm run build) +(cd backend && npx tsc --noEmit --target ES2022 --module ES2022 \ + --moduleResolution Bundler --strict --skipLibCheck \ + test/registry-pull-job-imports.compile.ts) +(cd harness && .venv/bin/pytest -q \ + tests/test_locked_child_stdin.py tests/test_layout_marker_commands.py \ + tests/test_p3_internal_cli.py tests/test_semantic_revision_migration.py \ + tests/test_qdrant_cli_commands.py) +bash scripts/test-preprocess-compose-config.sh +bash scripts/test-compose-secret-policy.sh +``` + +Expected: PASS with no Go toolchain download. The four ordinary operations retain writer-first order, +request bytes, phase transitions, fixed child argv/results, admission semantics, and existing-root +`acquire`. The P2 native addon and typed wrapper pass their Linux/Darwin contract and remain the sole +FD-relative provisioning/open/flock seam. P3's zero-filesystem adapter calls only +`acquireSessionReadersShared()` and `runUnderSessionReadersExclusive(...)`; the exact shared owner is +transferred into `PiProcessManager` through child/stream teardown, and the exclusive callback encloses +the full quiesced action. No P3 wrapper/handle/path/FD/name/flags/direct-open factory exists. The pull path +calls only `publishAddressed`; its actual create literal passes all three validated identity digests plus +`expectedBaseCommit`, its resume literal passes all three digests, and compile/runtime mismatch tests +fail before network/state. It proves all eight exact P2 phases, pre-advertisement and +post-advertisement same-ID recovery, exact-OID immutable-ref recovery, no network at/after +`target_fetched`, bootstrap/pull callback parity, complete lexical ownership, missing-root provisioning, +same participant/capability objects, no reentry, exact state path, all-or-nothing publication, +base/target/mixed reconciliation, terminal-before-clear recovery, and reverse release. Inspect, lazy +list, and status call only `ensureBootstrapAddressed` and consume its full result union: bounded +exact-identity automatic same-ID bootstrap recovery works at every preterminal kill point, and queued or +ordinary active callers converge through the exact `already_active` snapshot with one publication and +no second network/state work; ambiguous/corrupt/incompatible cases fail before network/state. Routes +retain the sole bootstrap/pull/activation mutation lifecycle. Every mutating child validates both writer +FD 3 and retained-root FD 4. The standalone compile command imports all six released job types from +`../src/workspaces/registry-pull-job.js`, exact `LockFileName` from `workspace-fs-at.js`, and exact +`RegistryActiveSnapshotV1`/`RegistryEnsureBootstrapAddressedResultV1` from +`registry-publication.js`. + +Run a final symbol fence before commit: + +```bash +python3 - <<'PY' +from pathlib import Path +roots = [Path("backend/src"), Path("backend/test")] +text = "\n".join(p.read_text() for root in roots for p in root.rglob("*.ts")) +for stale in ( + "RegistryPullBa" + "seTargetPlanV1", + "RegistryAddressedW" + "orkspaceLeaseOwner", + "acquireCo" + "mpleteSet", + "releaseCo" + "mpleteSet", + "pullAndPubl" + "ishAddressed", + "registry-" + "pull-jobs", + "VerifiedWo" + "rkspaceRoot", + "WorkspaceLockFi" + "leNameV1", + "RegistryValidatedActive" + "SnapshotV1", + "RegistryBootstrapEnsure" + "AddressedResultV1", + "interface Workspace" + "ReaderLease", +): + assert stale not in text, stale +module = Path("backend/src/workspaces/registry-pull-job.ts").read_text() +for name in ( + "RegistryPullPublicJobRequestV1", + "RegistryPullPhaseV1", + "RegistryPullAddressedJobRequestV1", + "RegistryPullParticipantStateV1", + "RegistryPullSynchronizerStateV1", + "RegistryPullJobStateV1", +): + assert f"export " in module and name in module, name +print("P3 registry symbol fence: PASS") +PY +``` + +Expected: `P3 registry symbol fence: PASS`. + +Commit all owning files, including the exact export and compile-import test: + +```bash +git add backend/src/workspaces/preprocessing-state.ts \ + backend/src/workspaces/preprocessing-service.ts backend/src/workspace-maintenance.ts \ + backend/src/workspaces/registry.ts backend/src/workspaces/registry-publication.ts \ + backend/src/workspaces/registry-pull-job.ts \ + backend/test/registry-pull-job-imports.compile.ts \ + backend/test/workspace-registry.test.ts \ + backend/test/workspace-registry-addressed-publication.test.ts \ + backend/test/workspace-registry-addressed-process.test.ts \ + backend/test/fixtures/workspace-lock-root-worker.mjs \ + backend/test/fixtures/workspace-registry-addressed-worker.mjs \ + backend/src/workspaces/workspace-reader-lease.ts \ + backend/src/routes/workspaces.ts backend/src/routes/sessions.ts backend/src/app.ts \ + backend/test/app.test.ts backend/src/pi/pi-process-manager.ts \ + backend/test/workspace-preprocessing-state.test.ts \ + backend/test/workspace-preprocessing-service.test.ts backend/test/workspace-maintenance.test.ts \ + backend/test/workspace-reader-lease.test.ts backend/test/routes-workspaces.test.ts \ + backend/test/routes-sessions.test.ts backend/test/pi-process-manager.test.ts \ + harness/tht/locked_child_stdin.py harness/tht/layout_markers.py \ + harness/tht/cli/config_cmd.py harness/tht/cli/vector_cmd.py harness/tht/semantic_migration.py \ + harness/tests/test_locked_child_stdin.py harness/tests/test_layout_marker_commands.py \ + harness/tests/test_p3_internal_cli.py harness/tests/test_semantic_revision_migration.py \ + harness/tests/test_qdrant_cli_commands.py \ + tools/thothctl/internal/workspaceops/operations.go \ + tools/thothctl/internal/workspaceops/operations_test.go \ + tools/thothctl/cmd/thothctl/main.go tools/thothctl/cmd/thothctl/main_test.go \ + deploy/compose.git-https.yaml deploy/compose.git-ssh.yaml \ + scripts/generate-connector-secrets-override.sh scripts/test-preprocess-compose-config.sh +git commit -m "feat: add exact addressed registry pull and resumable maintenance" +``` +--- + +### Task 9: Prepare one exact binding generation per commit and activate the revision layout + +**Files:** +- Modify: `backend/src/workspaces/revision-layout.ts` +- Modify: `backend/src/workspaces/runtime-config-lease.ts` +- Modify: `backend/src/workspaces/runtime-renderer.ts` +- Modify: `backend/src/tht/tht-runner.ts` +- Modify: `backend/src/workspaces/preprocessing-state.ts` +- Modify: `backend/src/workspaces/preprocessing-service.ts` +- Modify: `backend/src/workspace-maintenance.ts` +- Reuse unchanged from Task 8: `backend/src/workspaces/workspace-reader-lease.ts` +- Modify: `backend/src/pi/pi-process-manager.ts` +- Modify: `backend/test/workspace-revision-layout.test.ts` +- Modify: `backend/test/workspace-runtime-renderer.test.ts` +- Modify: `backend/test/workspace-runtime-config-lease.test.ts` +- Modify: `backend/test/workspace-runtime-handoff.test.ts` +- Create: `backend/test/workspace-effective-config-equivalence.test.ts` +- Modify: `backend/test/workspace-preprocessing-service.test.ts` +- Modify: `backend/test/workspace-maintenance.test.ts` +- Modify: `backend/test/workspace-reader-lease.test.ts` +- Modify: `backend/test/pi-process-manager.test.ts` +- Modify: `harness/tht/layout_markers.py` +- Modify: `harness/tht/cli/config_cmd.py` +- Modify: `harness/tests/test_layout_marker_commands.py` +- Modify: `harness/tests/test_p3_internal_cli.py` +- Modify: `harness/tht/cli/schema_cmd.py` +- Modify: `harness/tht/cli/lsh_cmd.py` +- Modify: `harness/tht/cli/preprocess_cmd.py` +- Modify: `harness/tht/jobs/dwh_pipeline.py` +- Modify: `harness/tht/taskdoc.py` +- Modify: `harness/tht/search/evidence.py` +- Modify: `harness/tht/cli/memory_cmd.py` +- Modify: `harness/tht/memory.py` +- Modify: `harness/tht/solved.py` +- Modify: `harness/tests/test_dwh_snapshot.py` +- Modify: `harness/tests/test_dwh_owner_migration.py` +- Modify: `harness/tests/test_dwh_preprocess_job.py` +- Modify: `harness/tests/test_lsh_job_resume.py` +- Modify: `harness/tests/test_search_pack.py` +- Modify: `harness/tests/test_schema_fk_annotations.py` +- Modify: `harness/tests/test_qdrant_cli_commands.py` +- Modify: `harness/tests/test_corpus_pipeline.py` +- Modify: `harness/tests/test_memory_migration.py` +- Modify: `harness/tests/test_memory_promotion.py` +- Modify: `harness/tests/test_memory_save_one.py` +- Modify: `harness/tests/test_solved_search_cli.py` +- Modify: `tools/thothctl/internal/workspaceops/operations.go`, `operations_test.go` +- Modify: `tools/thothctl/cmd/thothctl/main.go`, `main_test.go` + +**Step 1: Write the RED complete-consumer, admission, and transaction tests.** + +Add the exact released host command: + +```text +thothctl --installation workspace migrate activate-revision-layout + --workspace --yes [--resume <32-hex-outer-run-id>] [--json] +``` + +Fresh pre-generates one outer run ID and launches Task 8's single dedicated job with the canonical +create request. The one-shot atomically creates or exactly replays that ID/digest and can produce no +second run. Every retry after a confirmed nonterminal result uses that exact `--resume`; an ambiguous +host result exposes the generated ID with `run_durability_unconfirmed`. Freeze durable phases: + +```text +run_addressed -> quiescence_published -> reader_gate_exclusive -> effective_binding_probed -> +future_paths_bound -> dwh_cache_prepared -> dwh_snapshot_ready -> memory_ready -> +semantic_inventory_persisted -> semantic_sources_ready -> +semantic_replacements_published -> semantic_replacements_verified -> +layout_version_published -> legacy_delete_complete -> ready_published -> +terminal_durable -> quiescence_owner_cleared +``` + +Transitions are closed, monotonic, and artifact-digest bound. Resume re-verifies a phase before +skipping it. It must resume after a crash during reader drain, partial legacy deletion, either atomic +marker rename, terminal-state fsync, and before owner-qualified quiescence clear, including while +`core` is unavailable. + +Initial revision A plus effective binding X requires all of these before its READY publication: + +1. matching durable quiescence/run ownership, safe bounded session inventory, and an exclusive + `session-readers.lock` lease proving zero active readers while new admission is blocked; +2. P2 writer lock held continuously inside maintenance and active revision/descriptor/config + revalidated; +3. fixed read-only harness binding probe returns X, and every later child/result repeats X's binding + SHA/cache key; +4. matching schema-v1 DWH is migrated, or otherwise prepare-mode DWH builds, then strictly reverifies + the trusted X cache and materializes the exact A/X binding-qualified snapshot; +5. canonical Memory is migrated/reverified at the future global root; +6. exact legacy semantic inventory and source-readiness bytes are persisted, then all A replacements + are published and read back while the exclusive reader lease remains held; +7. global `layout-version.json` is published atomically (or strictly reverified if present); +8. only then, exact digest-confirmed legacy semantic deletion completes; +9. exact `revisions/A/readiness/X/READY.json` is exclusively published last and strictly re-read. + +The layout marker never names A or X. For later revision B, released `workspace registry pull` +publishes B through P2's repository-first addressed callback and its complete changed-set capability set. With layout v1 already global, B/X +has no READY: the fixed binding probe selects X and session admission fails `migration_required`, while +prepare maintenance returns only trusted B/X paths. Activation reuses the verified X cache, creates a +B/X snapshot and readiness generation without rewriting the marker; A/X and B/X coexist. + +Also test a changed installation binding Y while commit B is already READY for X. The session probe +selects Y, refuses B/X cache/snapshot/READY, and fails `migration_required`; activation securely finds +the existing B/X READY and returns `effective_config_mismatch` **before** DWH prepare/migration, +snapshot writes, semantic inventory/publish/delete, or marker/READY writes. B/X READY, snapshot, +semantic points, and owner remain byte-identical. The fixture then makes and publishes content-only +commit C and invokes released registry pull. C/Y is unready and isolated; activation now chooses the +Task 5 prepare-mode DWH build (not legacy migration or X fallback), publishes/verifies the new +schema-v2 Y cache and C/Y snapshot, publishes/read-backs C semantic replacements while quiesced, and +exclusively creates `revisions/C/readiness/Y/READY.json`. With Y still selected, C/Y is admitted and +B admission remains intentionally refused; B/X READY, snapshot, semantic points, and owner bytes remain +unchanged but revision pinning alone does not restore binding X. The test then explicitly restores the +installation binding to X, probes X and admits historical B/X; restores the installation binding to Y, +probes Y and admits C/Y. Repeat the refusal -> newly published successor commit -> changed-binding +activation for endpoint, transport, database, schema, and one included policy mutation, using a +distinct fresh revision for every newly selected binding; no test may activate B/Y or any second READY +binding at one revision. + +Inventory every physical/LSH consumer and fail the test if it references `.tht-dwh`, +`artifacts/mschema/physical.yaml`, or `indexes/lsh` outside `dwh_snapshot.py`. Explicitly exercise +`schema_cmd.physical_path`, schema introspection/check/index, `lsh_cmd` build/read, preprocess DWH, +`dwh_pipeline`, `taskdoc`/search-pack generation, and mschema readers through +`resolve_revision_dwh_snapshot`. Exercise all Memory commands through `memory_root/registry_path` and +Evidence preprocessing/search through explicit `paths.corpus`. + +Add fault tests at every phase and prove throughout the publish/delete/READY interval that the durable +marker blocks the new-admission race, the exclusive reader gate proves all shared readers drained, +persistent inventory was accepted, and no session child remains. Replacement publish/readback must +precede the first legacy delete. READY must follow the final delete. On failure/SIGKILL the OS locks +release but durable owner state and quiescence remain, sessions stay refused, and only exact same-ID +resume continues even with dead `core`. No failure clears quiescence or rolls back to P2 once the +global marker exists; wrong owner/digest can neither resume nor clear. + +Run: + +```bash +(cd backend && npx vitest run \ + test/workspace-revision-layout.test.ts \ + test/workspace-runtime-renderer.test.ts \ + test/workspace-runtime-config-lease.test.ts \ + test/workspace-runtime-handoff.test.ts \ + test/workspace-effective-config-equivalence.test.ts \ + test/workspace-preprocessing-service.test.ts \ + test/workspace-maintenance.test.ts test/workspace-reader-lease.test.ts \ + test/routes-sessions.test.ts test/pi-process-manager.test.ts) +(cd harness && .venv/bin/pytest -q \ + tests/test_dwh_snapshot.py tests/test_dwh_owner_migration.py \ + tests/test_dwh_preprocess_job.py tests/test_lsh_job_resume.py tests/test_search_pack.py \ + tests/test_schema_fk_annotations.py tests/test_qdrant_cli_commands.py \ + tests/test_corpus_pipeline.py tests/test_memory_migration.py \ + tests/test_memory_promotion.py tests/test_memory_save_one.py tests/test_solved_search_cli.py) +(cd tools/thothctl && test "$(go env GOVERSION)" = "go1.26.5" && \ + go test ./internal/workspaceops ./cmd/thothctl -run 'Activate|RegistryPull|Resume|Quiesc|Reader|DedicatedJob' -v) +``` + +Expected: RED on activation, readiness gating, and remaining direct paths. + +**Step 2: Implement synchronous marker-gated session rendering and trusted prepare rendering.** + +Keep P2's exact synchronous signatures: + +```ts +export class WorkspaceRuntimeConfigLeaseFactory { + acquireSession(snapshotPath: string): RuntimeConfigLease; + acquireMaintenance(input: MaintenanceRuntimeInput): RuntimeConfigLease; +} +``` + +Extend `MaintenanceRuntimeInput` with exact `layoutIntent: "active" | +"prepare-revision-layout-v1"`; keep both factory signatures synchronous and do not add a TypeScript +effective-DWH canonicalizer. In layout v1 the returned `RuntimeConfigLease` begins as a bounded +binding-probe lease: its YAML has the exact descriptor/DWH inputs and trusted future base roots, may be +used only for fixed `tht config effective-dwh --json -c CONFIG`, and cannot spawn Pi or any mutating +child. The existing async `ThtRunner` startup/coordinator path runs that read-only probe with 65,536 +stdout bytes, 16,384 stderr bytes and a 30-second timeout, validates the exact `EffectiveDwhBinding` +and key, then calls a new synchronous lease method +`selectEffectiveDwh(binding): BoundRuntimeConfigLease`. That method invokes +`readRevisionLayoutState(..., effectiveDwhCacheKey)` and +`bindingQualifiedWorkspaceLayoutPaths`; it never selects newest/only READY. This adds no alternate +factory entrypoint and preserves P2's exact `acquireSession(snapshotPath)` and +`acquireMaintenance(input)` signatures. + +Session behavior after selection is exact: + +- no global marker: byte-identical P2 roots and normal P2 handoff; +- valid marker + exact commit+binding READY: render its binding-qualified cache/snapshot and revision + roots, reverify READY identity, then and only then spawn Pi/session work; +- valid marker + absent exact READY: `migration_required` before Pi/session or semantic child spawn, + even if another READY exists for that commit; +- invalid marker/selected READY or binding mismatch: fail closed. + +Active maintenance follows the same rule. Prepare maintenance is the only exception: before READY and +only after the secure directory guard proves that revision has no READY for another binding, it uses +the probed binding key with the trusted resolvers and renders only: + +```text +sessions = /data/sessions//sessions +memory = /data/sessions//memory +dwh_cache = /data/sessions//preprocessing/dwh-cache/ +dwh_snapshot = /data/sessions//revisions//dwh-snapshots/ +ready = /data/sessions//revisions//readiness//READY.json +artifacts = /data/sessions//revisions//artifacts +indexes = /data/sessions//revisions//indexes +corpus = /data/sessions//revisions//corpus +``` + +It never consults P2 `memory`, `.tht-dwh`, artifacts, indexes, corpus, another binding cache/snapshot, +or a sibling READY as a destination. Tests prove the probe lease cannot escape to a session/mutating +spawn and that session and maintenance selection use byte-identical binding-qualified YAML. + +**Step 3: Implement the quiesced activation transaction in the flat coordinator.** + +`workspaceops.Run` launches only Task 8's dedicated job. Inside that one-shot, +`WorkspacePreprocessingService.execute` obtains `CanonicalWorkspaceLockRootInput` from the sole P2 +factory, uses existing-root `acquire`, transfers the exact `VerifiedWorkspaceLockRootLease`, and enters +one `runUnderWorkspaceWriterLock(rootLease, async (writerCapability) => ...)` action **before** ordinary +job or quiescence. Inside that still-live writer callback, publish the durable owner marker, drain +readers, then call Task 8's exact +`readerLeaseFactory.acquireForMaintenance(writerCapability, async (readerBorrow) => ...)`. The complete +quiesced activation—including final recheck, all children/publication, terminal durability, and matching +owner clear—settles inside that callback; `readerBorrow` never escapes. The writer action consumes and +closes the root only after the adapter has returned and P2 has invalidated/closed the exclusive reader. +Each fixed internal migration request runs only through `writerCapability.spawnChild`; thus the actual locked open file +description is child FD 3 and the same retained root description is FD 4; neither is reopened. Revalidate target snapshot and inventory at +every child boundary. + +Activation is the sole semantic cutover and one continuous quiesced transaction. It binds the exact +commit+effective key, then **before DWH/Memory/semantic mutation** securely enumerates the revision's +readiness directory. An empty directory permits preparation; the selected READY may only byte-match +for idempotent resume; one READY at another key returns `effective_config_mismatch`; multiple/unsafe +entries fail closed. Only after this guard does activation migrate or prepare-build/reverify the cache +and binding snapshot; reuse/recheck Memory and persisted semantic inventory/readiness; publish all +replacement points and read back their exact identities/digests; atomically publish/reverify the +global marker for initial enablement; delete only unchanged exact legacy inventory IDs; and +exclusively publish the selected commit+binding READY last. For later **revisions** it strictly verifies +the unchanged global marker and does not broaden deletion. All exclusive files use sibling write + +file fsync + rename + parent fsync and synchronous no-follow reread. Different bytes at the same +`(commit,key)` fail closed; a changed binding never writes a sibling generation for that commit and +must use a newly published revision. + +Only after READY and terminal run state are durably re-read does the same one-shot owner remove its +byte-matching quiescence marker while still holding the exclusive reader gate, then release the gate. +No HTTP/backend acknowledgment and no core lifecycle action exists. Failure before the global marker +leaves P2 rendering but stays durably quiesced; failure after the marker leaves the revision unready +and stays quiesced. Process death releases OS locks but exact same-ID resume is the only recovery path. + +**Step 4: Switch every consumer only through the shared resolvers.** + +Remove every direct path found by Step 1. P3 configs require a valid revision snapshot; legacy configs +without `dwh_snapshot` retain their compatibility branch only when the global marker is absent. +Schema/LSH reads never probe cache/old roots after layout enablement. Memory and corpus use explicit +config paths. Schema/Evidence vector operations select strict revision scope only from a ready P3 +config. + +**Step 5: Prove A to B transition and effective-binding equivalence.** + +For A/B with identical DWH inputs, run `tht config effective-dwh --json` against session and +maintenance leases and assert byte-identical binding/cache key, distinct revision roots, and stable +`workspace://`. Use the released command, not raw Git or the backend route: + +```bash +"$THOTHCTL" --installation "$INSTALLATION" workspace registry pull \ + --workspace "$WORKSPACE" --json +"$THOTHCTL" --installation "$INSTALLATION" workspace migrate dwh-cache \ + --workspace "$WORKSPACE" --json +``` + +Assert B/X session admission is `migration_required` before B/X READY. The released `dwh-cache` +command resolves only B/X, materializes/reverifies the binding-qualified B/X snapshot for pre-READY +preparation/P5 validation, and proves no layout marker, READY, or semantic point changed; then activate +and admit B/X. Then, without changing commit B, change effective binding to Y: +admission fails and activation returns `effective_config_mismatch` before +`p3_prepare_dwh_cache`, cache/snapshot/semantic/marker writes, or READY publication. Publish and pull +content-only revision C, prove C/Y is unready, then run the full released activation path: +`p3_prepare_dwh_cache` creates the new v2 Y owner/cache, C/Y snapshot, semantic verification, and +immutable C/Y READY. With Y selected, admit C/Y and continue refusing B; prove B/X bytes unchanged. +Explicitly restore installation binding X and admit B/X, then restore Y and admit C/Y. Repeat the +complete refusal -> fresh successor revision -> new-binding activation path for endpoint, transport, +database, schema, and one included policy, never reusing an already-READY revision for the next +binding. Hold the registry addressed callback-scoped ordered set open to prove every changed-workspace writer +capability contends without reacquisition; hold each ordinary child open for its selected writer lock; +hold multiple session reader leases to prove drain, and rely on durable admission blocking plus the exclusive reader gate for +semantic cutover while `core` may keep running. + +**Step 6: Run GREEN activation and full P2 regression gates.** + +```bash +(cd backend && npx vitest run \ + test/workspace-revision-layout.test.ts \ + test/workspace-runtime-renderer.test.ts \ + test/workspace-runtime-config-lease.test.ts \ + test/workspace-runtime-handoff.test.ts \ + test/workspace-effective-config-equivalence.test.ts \ + test/workspace-preprocessing-state.test.ts \ + test/workspace-preprocessing-service.test.ts \ + test/workspace-maintenance.test.ts test/workspace-reader-lease.test.ts \ + test/routes-sessions.test.ts test/pi-process-manager.test.ts test/tht-runner.test.ts && npx tsc --noEmit -p . && npm run build) +(cd harness && .venv/bin/pytest -q \ + tests/test_effective_dwh_binding.py tests/test_dwh_snapshot.py tests/test_dwh_owner_v2.py \ + tests/test_dwh_owner_migration.py tests/test_dwh_preprocess_job.py \ + tests/test_locked_child_stdin.py tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py \ + tests/test_lsh_job_resume.py tests/test_search_pack.py tests/test_schema_fk_annotations.py \ + tests/test_qdrant_vector_store.py tests/test_semantic_kind_isolation.py \ + tests/test_qdrant_cli_commands.py tests/test_corpus_pipeline.py \ + tests/test_memory_migration.py tests/test_memory_promotion.py \ + tests/test_memory_save_one.py tests/test_solved_question.py tests/test_solved_search_cli.py) +(cd tools/thothctl && test "$(go env GOVERSION)" = "go1.26.5" && go test ./...) +./scripts/p2-acceptance.sh integration --keep +``` + +Expected: every focused gate passes; fresh P2 acceptance still passes before global enablement; +A/X, B/X, same-revision Y refusal, C/Y READY, explicit X restore/B admission, and Y restore/C +admission tests pass. New admission cannot cross quiescence publication; semantic delete and READY +publication occur only under the exclusive reader gate, regardless of whether `core` is running. + +**Step 7: Commit.** + +```bash +git add backend/src/workspaces/revision-layout.ts \ + backend/src/workspaces/runtime-config-lease.ts backend/src/workspaces/runtime-renderer.ts \ + backend/src/tht/tht-runner.ts backend/src/workspaces/preprocessing-state.ts \ + backend/src/workspaces/preprocessing-service.ts backend/src/workspace-maintenance.ts \ + backend/src/pi/pi-process-manager.ts \ + backend/test/workspace-revision-layout.test.ts backend/test/workspace-runtime-renderer.test.ts \ + backend/test/workspace-runtime-config-lease.test.ts backend/test/workspace-runtime-handoff.test.ts \ + backend/test/workspace-effective-config-equivalence.test.ts \ + backend/test/workspace-preprocessing-service.test.ts backend/test/workspace-maintenance.test.ts \ + backend/test/workspace-reader-lease.test.ts backend/test/pi-process-manager.test.ts \ + harness/tht/cli/schema_cmd.py harness/tht/cli/lsh_cmd.py harness/tht/cli/preprocess_cmd.py \ + harness/tht/jobs/dwh_pipeline.py harness/tht/taskdoc.py harness/tht/search/evidence.py \ + harness/tht/cli/memory_cmd.py harness/tht/memory.py harness/tht/solved.py \ + harness/tht/layout_markers.py harness/tht/cli/config_cmd.py \ + harness/tests/test_layout_marker_commands.py harness/tests/test_p3_internal_cli.py \ + harness/tests/test_dwh_snapshot.py harness/tests/test_dwh_owner_migration.py \ + harness/tests/test_dwh_preprocess_job.py harness/tests/test_lsh_job_resume.py \ + harness/tests/test_search_pack.py harness/tests/test_schema_fk_annotations.py \ + harness/tests/test_qdrant_cli_commands.py harness/tests/test_corpus_pipeline.py \ + harness/tests/test_memory_migration.py harness/tests/test_memory_promotion.py \ + harness/tests/test_memory_save_one.py harness/tests/test_solved_search_cli.py \ + tools/thothctl/internal/workspaceops/operations.go \ + tools/thothctl/internal/workspaceops/operations_test.go \ + tools/thothctl/cmd/thothctl/main.go tools/thothctl/cmd/thothctl/main_test.go +git commit -m "feat: activate revision-isolated workspace layout" +``` + +--- + +### Task 10: Document `.tht-dwh`, migration, recovery, and the P3 manual walkthrough + +**Files:** +- Modify: `docs/install/local-workspace-registry.md` +- Modify: `docs/install/server-workspace-registry.md` +- Modify: `docs/testing/p2-p6-manual-verification.md` +- Create: `docs/architecture/effective-dwh-cache.md` +- Modify: `docs/index.md` + +**Step 1: Write documentation contract tests first.** + +Create or extend `scripts/test-verify-workspace-install-docs.sh` assertions requiring both manuals and the architecture page to name: + +- effective config vs input fingerprint; +- logical source identity and excluded revision/temp filename/secrets; +- `.tht-dwh`, immutable `generations/`, `OWNER.json` v1/v2, `ACTIVE`; +- cache key, revision snapshot roots, and why mismatch fails closed; +- exact inspect/registry-pull/migrate/`--resume`/preprocess/reindex/recovery commands, including that explicit `--resume` is for pull/migration while empty-registry bootstrap recovery is automatic through inspect, lazy list, and status; +- the repo-owned Node-API v8 `workspace-fs-at` seam for anchored missing-root provisioning, its closed typed writer/reader lock-name and flock surface, its Darwin/Linux durability behavior, and that exact `fs-ext@2.1.1` is wrapper-internal and `flock(2)` only with no P3 raw FD/import/cast/path fallback; +- client-generated addressed run IDs/digests, the pull create/resume identity/base fields derived inside the validated boundary, bounded exact-identity `ensureBootstrapAddressed` active-state/zero/one/multiple/corrupt rules (including the `already_active` no-network/no-state branch) and `registry_bootstrap_recovery_conflict`, no-follow durable owner quiescence, shared session reader leases, exclusive drain, dead-core same-ID resume, and owner-only clear; +- global layout-version marker vs immutable commit+effective-binding READY, with at most one READY binding per revision and a required new content commit for changed binding; +- Memory canonical JSONL vs Qdrant projection; +- backup scope for cache, revisions, corpus, Memory, Qdrant, and registry; +- warning not to edit OWNER/ACTIVE or copy an unverified generation manually. + +Run: + +```bash +./scripts/test-verify-workspace-install-docs.sh +``` + +Expected: RED on missing P3 content. + +**Step 2: Write operator-facing architecture and recovery content.** + +Explain safe recovery choices: inspect; inspect/lazy list/status always enter the continuously repository-locked selector, return its validated `already_active` snapshot without job scan/network/state when active state exists (including queued callers after another caller publishes), and only for validated absence automatically resume the sole exact-identity nonterminal bootstrap or create one, while incompatible active state or multiple, pull, mismatched, corrupt, churning, or over-bound job sets stop with `registry_bootstrap_recovery_conflict` and no operator-selected run ID; preserve legacy filesystem sources; run non-activating DWH/Memory preparation; use `semantic-revision` only to inventory legacy points and prove rebuild readiness, explicitly documenting that it writes no replacement Qdrant point and deletes nothing. `workspace migrate dwh-cache` must be documented as pre-READY materialization of the selected cache plus binding-qualified pulled-revision snapshot, usable before P5 acceptance but incapable of publishing layout/READY or admitting that revision. Perform replacement publish/readback, global reader-mode switch, exact legacy deletion, and immutable commit+binding READY only in one addressed one-shot run after durable admission blocking and exclusive reader-gate acquisition. The CLI never calls Fastify/HTTP or stops `core`; blocked admission plus zero shared readers makes publication safe while `core` may run. Use the client-generated run ID with exact `--resume` after interruption or ambiguous response, including dead-core recovery; SIGKILL releases OS locks but retains owner marker/run state. Prepare a changed binding only after publishing/pulling a new content commit; never activate a second binding READY for one revision. While Y is selected, B/X bytes remain immutable but B admission is refused; historical B/X is demonstrated only by restoring installation X, then Y is restored for C/Y. Rebuild Memory projection from canonical JSONL; never hand-clear quiescence or weaken digest checks. Evidence that cannot be rebuilt stays `migration_required` and its legacy points are not deleted. A global marker without the current exact commit+binding READY intentionally blocks sessions while prepare mode remains available only if that revision has no other READY binding. State that schema-v1 remains readable only to verify/migrate, not writable by P3. Explain that absent changed-set roots are provisioned only by P2's owned `WorkspaceFsAtV1` Node-API seam using literal FD-relative syscalls, with Linux directory-fsync success and Darwin success-or-exact fail-closed behavior. For readers, P3 calls only `VerifiedWorkspaceLockRootLease.acquireSessionReadersShared()` and `WorkspaceWriterLockCapability.runUnderSessionReadersExclusive(...)` through the zero-filesystem adapter; it never opens/flocks directly or receives a wrapper, directory handle, path, numeric FD, lock name, flags, or mode, and exact `fs-ext@2.1.1` remains P2-wrapper-internal and `flock(2)` only. + +**Step 3: Replace the P3 placeholder in the living manual with an independently runnable clean walkthrough.** + +Use variables `INSTALLATION`, `THOTHCTL`, `WORKSPACE`, and a new fixture/private Git remote. Include these exact operator calls: + +```bash +"$THOTHCTL" --installation "$INSTALLATION" workspace inspect \ + --workspace "$WORKSPACE" --json +"$THOTHCTL" --installation "$INSTALLATION" workspace migrate dwh-cache \ + --workspace "$WORKSPACE" --json +"$THOTHCTL" --installation "$INSTALLATION" workspace migrate memory \ + --workspace "$WORKSPACE" --json +"$THOTHCTL" --installation "$INSTALLATION" workspace migrate semantic-revision \ + --workspace "$WORKSPACE" --yes --json +"$THOTHCTL" --installation "$INSTALLATION" workspace migrate activate-revision-layout \ + --workspace "$WORKSPACE" --yes --json +"$THOTHCTL" --installation "$INSTALLATION" workspace preprocess dwh \ + --workspace "$WORKSPACE" --json + +# After committing/pushing content-only B, use the released product pull, never raw registry mutation: +"$THOTHCTL" --installation "$INSTALLATION" workspace registry pull \ + --workspace "$WORKSPACE" --json +# Pre-READY: materialize the pulled revision's selected cache/snapshot; do not activate READY. +"$THOTHCTL" --installation "$INSTALLATION" workspace migrate dwh-cache \ + --workspace "$WORKSPACE" --json +"$THOTHCTL" --installation "$INSTALLATION" workspace migrate activate-revision-layout \ + --workspace "$WORKSPACE" --yes --json + +# Exact crash recovery pattern: export RUN_ID from the returned 32-hex runId first. +RUN_ID="${RUN_ID:?export RUN_ID as the returned 32-hex outer run ID}" +[[ "$RUN_ID" =~ ^[0-9a-f]{32}$ ]] +"$THOTHCTL" --installation "$INSTALLATION" workspace migrate activate-revision-layout \ + --workspace "$WORKSPACE" --yes --resume "$RUN_ID" --json +``` + +Then walk the reviewer through: + +1. during the initial empty-registry bootstrap, inject one preterminal death for each of inspect, lazy list, and status, then prove the next actual caller automatically resumes the same exact-identity run before network; prove multiple/mismatched/corrupt jobs stop with `registry_bootstrap_recovery_conflict`; after terminal recovery, record revision A and compare the safe operator/session effective binding; +2. inspect migrated v2 OWNER, ACTIVE, binding-qualified immutable snapshot manifests/digests, the workspace-global `layout-version.json`, and exact A/binding `readiness//READY.json` without printing config/secrets; +3. after each non-activating command, prove owner-clear permits a session that sees no mixed legacy/replacement state; specifically prove `dwh-cache` materializes the selected binding-qualified snapshot but no READY/layout/semantic bytes, and `semantic-revision` makes zero Qdrant upsert/delete calls; +4. hold two live sessions with shared reader leases, start activation, prove the durable marker refuses a racing new admission, let both readers drain without killing them, and prove publication starts only after the exclusive reader lease establishes zero active readers; keep `core` running and show no Fastify/HTTP or stop/start command occurs; +5. inject SIGKILL during partial deletion and after READY rename, make `core` unavailable, then use the returned exact `--resume` ID to reach success without re-embedding or broad deletion; prove the OS lock released, the durable owner marker remained, and wrong owner/digest could not clear it; +6. commit/push content-only revision B, invoke released `workspace registry pull`, run `workspace migrate dwh-cache`, prove the B/X cache and binding-qualified snapshot exist pre-READY and are usable for later P5 validation while B admission still fails, then activate and prove the same cache generation but distinct revision roots and schema/Evidence points; +7. prove A/X READY remains, and Memory/solved results remain visible at B with global identity; +8. hold activation/mutation open and prove concurrent registry publication is refused; no existing reader is killed to reach exclusive ownership; +9. while commit B is READY for X, change the fixture binding to Y; prove B admission and same-revision activation are refused with zero mutation while B/X bytes remain immutable. Publish/pull C, run pre-READY `dwh-cache`, activate C/Y, and admit C/Y. Then explicitly restore installation X and admit B/X; restore Y and admit C/Y. Repeat on distinct successor revisions for endpoint, transport, database, schema, and one included policy; +10. create a conflicting legacy Memory registry, prove durable owner quiescence, remove only the conflict, and resume the same ID; +11. scan captured output for fixture secret canaries and perform exact cleanup. + +For every step explain component, state read, artifact produced, invariant, evidence to retain, and PASS/FAIL criterion. Set `automated integration: PENDING` until Task 12; keep `manual acceptance: PENDING` and `Decision: PENDING` until reviewer action. The manual environment must not reuse automated state. + +**Step 4: Run docs checks.** + +```bash +./scripts/test-verify-workspace-install-docs.sh +``` + +Expected: PASS. + +**Step 5: Commit.** + +```bash +git add docs/install/local-workspace-registry.md docs/install/server-workspace-registry.md \ + docs/testing/p2-p6-manual-verification.md docs/architecture/effective-dwh-cache.md \ + docs/index.md scripts/test-verify-workspace-install-docs.sh +git commit -m "docs: explain effective DWH cache ownership" +``` + +--- + +### Task 11: Build the clean-state P3 acceptance process and self-tests + +**Files:** +- Create: `scripts/p3-acceptance.sh` +- Create: `scripts/test-p3-acceptance.sh` +- Create: `backend/scripts/p3-acceptance.mjs` +- Modify: `.gitignore` only if `.artifacts/p3-effective-config/` is not already covered + +**Step 1: Write RED acceptance-harness tests.** + +`test-p3-acceptance.sh` must test, without requiring the full expensive success run: + +- exact CLI grammar: `integration [--keep]` and `cleanup --run `; +- dirty tracked source refusal before build/start; +- unique run/project IDs and exclusive ownership manifest creation; +- no retry loop or recursive self-invocation; +- trap cleanup on pre-service and post-service injected failures; +- refusal to clean an unowned/mismatched project, volume, network, path, or symlink; +- `--keep` retains owned resources and reports; cleanup removes owned resources only and is idempotent; +- report schema and finalization on PASS and injected FAIL; +- secret canaries absent from stdout/stderr/report/public retained files; +- Qdrant/Ollama listener shutdown and Compose resource absence after normal cleanup. + +Run: + +```bash +./scripts/test-p3-acceptance.sh +``` + +Expected: RED because the runner does not exist. + +**Step 2: Implement the process runner.** + +Follow the hardened P1/P2 runner conventions: trusted fixed toolchain discovery, sanitized environment, no shell-evaluated fixture data, bounded subprocess capture, exact ownership labels, raw-Git safety, single attempt, atomic report writes, SHA-256 artifact manifest, and signal-safe cleanup. Reuse P2 fixture setup functions rather than copying them if they are already factored into sourceable non-executable helpers. + +The full process assertions are: + +1. clean source commit/tree, exact `go1.26.5`, and no pre-existing owned resources; +2. revision A P2 schema-v1 fixture and legacy Memory/schema/Evidence state detected as `migration_required`, with byte-identical P2 session/maintenance roots; +3. the synchronous no-follow layout reader rejects symlinks/replacement/malformed state, and the trusted future resolver selects exact migration destinations without active fallback; +4. every released operation pre-generates a 32-hex ID and launches only the immutable-image-pinned, profile-gated `workspace-maintenance` Compose job with canonical 4,096-byte-bounded request stdin and bounded exact JSON result; no Fastify/HTTP/internal route, `compose exec core`, frontend, host Python/Node/Pi/`tht`, or core stop/start is invoked. The four ordinary operations each use the selected-workspace P2 existing-root/writer-first lifecycle: shared session ownership is acquired only by `acquireSessionReadersShared()` and transferred into `PiProcessManager` through all child/stream teardown, while the full quiesced maintenance action is enclosed only by `runUnderSessionReadersExclusive(...)` via the zero-filesystem adapter. `registry_pull` uses only the repository-first P2 addressed callback; its actual create request passes validated installation/repository/remote identities plus expected base and its resume request passes all three identities, with compile/runtime mismatch-before-network/state proof. It owns the complete lexical changed set, provisions absent roots and opens/flocks writer/reader gates only through P2's typed repo-owned `WorkspaceFsAtV1` seam (with native Linux/Darwin tests green, `fs-ext` wrapper-internal, and no P3 raw FD/import/cast/path factory), uses the exact addressed state artifact, and resumes the same ID by its durable pin phase while `core` is dead; +5. every P3 request variant is exhaustively built by the opaque capability with fixed argv, the canonical exact-key base64 stdin wire, strict Python parser, bounded result and the actual writer FD 3/retained-root FD 4 pair; every argv target, including all four marker/readiness commands, exists in the built image; maximum 716,800-byte inventory round-trips through the 1,048,576-byte collector; DWH/Memory retain sources and reverify destinations; arbitrary spawn/argv/path, malformed stdin, missing/substituted/cross-root/post-callback/nested/concurrent capability misuse, and SIGKILL fail safely; +6. after each successful non-activation command, terminal durability plus owner-only clear permits admission into one unmixed mode or expected `migration_required`; `dwh-cache` materializes the selected pre-READY cache and binding-qualified revision snapshot with zero READY/layout/semantic writes, and `semantic-revision` persists inventory/readiness with exactly zero replacement upserts/deletes; +7. durable marker publication blocks a racing admission; multiple existing shared reader leases drain without being killed; the exclusive no-follow OS lease proves zero active readers and is held while activation publishes/verifies A/X replacements, publishes the global reader marker, deletes only unchanged exact legacy IDs, and publishes immutable A/X READY last while `core` may remain running; +8. success durably records terminal state before the matching owner clears quiescence and releases exclusive ownership. SIGKILL releases OS locks but retains marker/run state; dead-core same-ID resume succeeds; wrong owner/digest and a second run cannot resume, mutate, or clear; +9. every schema/LSH/search-pack/mschema reader resolves the selected binding snapshot, every Memory command uses the global root, and no direct old-root/sibling-binding probe remains after enablement; +10. released registry pull alone receives writable registry/Git capability; repository lock precedes durable `request_claimed`, `target_advertised`, exact-OID `target_fetched`, and immutable exact A→B `planned`; complete changed IDs use P2 `acquireOrProvision` backed only by `WorkspaceFsAtV1` and one callback-scoped root/writer/quiescence/readers set lexically; bootstrap and pull callbacks remain field-for-field P2; inspect/lazy-list/status use only bounded `ensureBootstrapAddressed`, whose locked active recheck returns the exact `already_active` snapshot without scan/network/state and whose absent branch performs automatic zero/create or one/resume selection. A three-caller initial-absence race publishes exactly once and all callers converge; incompatible active state or pull/multiple/mismatched/corrupt/churning/over-bound job sets fail closed before network/state; participants cannot reenter; active publication is all-or-nothing; and B/X sessions fail before B/X READY while ordinary prepare resolves only B/X; +11. resumed B/X activation reuses X cache, publishes distinct revision state and immutable B/X READY, preserves A/X READY, and retains global Memory/solved retrieval; +12. ordinary marker-before-exclusive ordering, new-admission race refusal, multi-reader drain, writer-before-reader order, typed shared/exclusive reader locking, publish-before-delete, partial-delete/after-READY SIGKILL resume, dead-core recovery, and owner-only clear are proved. Separately, dead-core registry SIGKILL spans before/after claim, advertisement, immutable-ref fetch, `target_fetched`, active rename/fsync, and terminal-before-clear. Same-ID pre-advertisement recovery may re-advertise; post-advertisement recovery keeps the recorded OID; at/after `target_fetched` it never uses the network. For bootstrap, every preterminal death is resumed through each actual inspect/lazy-list/status caller by the sole matching exact-identity run before any fresh advertisement; queued callers recheck and return the resulting exact `already_active` snapshot; and every incompatible-active/ambiguity/corruption/bound failure is stable and network/state-free. Exact mixed base/target reconciles, a third installation/repository/remote identity, create base mismatch, or missing/changed pinned target is rejected before network/state, and terminal replay succeeds even if a later independent pull moved tracking refs; +13. after pull, `migrate dwh-cache` proves pre-READY cache plus binding-qualified snapshot materialization usable before P5 acceptance with no READY activation. At READY B/X, binding Y first refuses B admission and same-revision activation with zero mutation; then C is published/pulled, C/Y is prepared and activated, and C/Y is admitted while B/X bytes remain unchanged. Historical B/X is proved only by explicitly restoring installation X and admitting B/X, followed by restoring Y and admitting C/Y; transport, database, schema, and one root-affecting policy repeat on distinct successors with no second READY at one revision; +14. corrupted owner, conflicting Memory registry, caller namespace conflict, unsafe marker/READY, partial migration, wrong `--resume`, and interrupted activation fail closed with stable codes and no secret leakage; +15. safe reports pass secret scan and declared hashes; cleanup removes exactly owned Docker/filesystem state. + +**Step 3: Run the runner self-tests.** + +```bash +./scripts/test-p3-acceptance.sh +``` + +Expected: PASS. + +**Step 4: Commit.** + +```bash +git add scripts/p3-acceptance.sh scripts/test-p3-acceptance.sh backend/scripts/p3-acceptance.mjs .gitignore +git commit -m "test: add P3 clean-state process goal" +``` + +--- + +### Task 12: Focused verification, full clean process run, retained report, and checkpoint + +**Files:** +- Create: `docs/reports/2026-08-10-p3-effective-config-checkpoint.md` +- Modify: `docs/testing/p2-p6-manual-verification.md` +- Modify: `PROJECT_STATE.md` +- Create: `scripts/lint-plan-shell-fences.py` +- Create: `scripts/test-lint-plan-shell-fences.py` + +**Step 1: Verify the source is clean before the evidence run.** + +```bash +git status --short +git rev-parse HEAD +git rev-parse 'HEAD^{tree}' +``` + +Expected: empty status and recorded commit/tree. If documentation/report changes are still uncommitted, commit them before running; the acceptance runner must not accept a dirty tracked tree. + +**Step 2: Run all P3-focused gates.** + +```bash +(cd harness && .venv/bin/pytest -q \ + tests/test_effective_dwh_binding.py tests/test_dwh_snapshot.py tests/test_dwh_owner_v2.py \ + tests/test_dwh_owner_migration.py tests/test_dwh_preprocess_job.py \ + tests/test_config_resources.py tests/test_portable_paths.py \ + tests/test_locked_child_stdin.py tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py \ + tests/test_lsh_job_resume.py \ + tests/test_search_pack.py tests/test_schema_fk_annotations.py \ + tests/test_qdrant_vector_store.py tests/test_semantic_kind_isolation.py \ + tests/test_semantic_revision_migration.py tests/test_qdrant_cli_commands.py \ + tests/test_corpus_pipeline.py tests/test_memory_migration.py \ + tests/test_memory_promotion.py tests/test_memory_save_one.py \ + tests/test_solved_question.py tests/test_solved_search_cli.py \ + tests/test_repository_memory_sql_paths.py) +(cd harness && .venv/bin/ruff check \ + tht/effective_dwh.py tht/config.py tht/cli/config_cmd.py tht/jobs/dwh_pipeline.py \ + tht/paths.py tht/dwh_snapshot.py tht/dwh_owner.py tht/dwh_migration.py \ + tht/cli/preprocess_cmd.py tht/vectorstore/records.py tht/adapters/vector/qdrant.py \ + tht/ports/vector.py tht/adapters/factory.py tht/search/evidence.py \ + tht/semantic_migration.py tht/cli/vector_cmd.py tht/memory_migration.py \ + tht/cli/memory_cmd.py tht/memory.py tht/solved.py tht/cli/schema_cmd.py \ + tht/cli/lsh_cmd.py tht/taskdoc.py tht/locked_child_stdin.py tht/layout_markers.py \ + tests/test_dwh_preprocess_job.py tests/test_effective_dwh_binding.py \ + tests/test_config_resources.py tests/test_portable_paths.py tests/test_dwh_snapshot.py \ + tests/test_dwh_owner_v2.py tests/test_dwh_owner_migration.py tests/test_search_pack.py \ + tests/test_qdrant_vector_store.py tests/test_semantic_kind_isolation.py \ + tests/test_qdrant_cli_commands.py tests/test_corpus_pipeline.py \ + tests/test_memory_save_one.py tests/test_solved_question.py \ + tests/test_semantic_revision_migration.py tests/test_memory_migration.py \ + tests/test_memory_promotion.py tests/test_solved_search_cli.py \ + tests/test_repository_memory_sql_paths.py tests/test_locked_child_stdin.py \ + tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py tests/test_lsh_job_resume.py \ + tests/test_schema_fk_annotations.py) +(cd backend && npx vitest run \ + test/workspace-revision-layout.test.ts test/workspace-runtime-renderer.test.ts \ + test/workspace-runtime-config-lease.test.ts test/workspace-runtime-handoff.test.ts \ + test/workspace-effective-config-equivalence.test.ts test/workspace-lock-root-lease.test.ts \ + test/workspace-preprocessing-state.test.ts test/workspace-preprocessing-service.test.ts \ + test/workspace-maintenance.test.ts test/workspace-registry.test.ts \ + test/workspace-registry-addressed-publication.test.ts \ + test/workspace-registry-addressed-process.test.ts test/workspace-reader-lease.test.ts \ + test/routes-workspaces.test.ts test/routes-sessions.test.ts \ + test/pi-process-manager.test.ts test/tht-runner.test.ts && \ + npx tsc --noEmit -p . && npm run build) +(cd backend && npx tsc --noEmit --target ES2022 --module ES2022 \ + --moduleResolution Bundler --strict --skipLibCheck \ + test/registry-pull-job-imports.compile.ts) +(cd tools/thothctl && test "$(go env GOVERSION)" = "go1.26.5" && go test ./...) +./scripts/test-verify-workspace-install-docs.sh +./scripts/test-p3-acceptance.sh +python3 scripts/test-lint-plan-shell-fences.py +python3 scripts/lint-plan-shell-fences.py \ + docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md \ + docs/superpowers/plans/2026-08-10-p3-effective-config-fingerprint.md +``` + +The plan-shell linter extracts every fenced `bash`/`sh` block byte-for-byte, rejects shell blocks +containing angle-bracket placeholders, and runs `bash -n` on each block independently. Deliberate +metavariables may remain only in `text` fences; runnable shell must use quoted environment checks such +as `${RUN_ID:?message}` and an explicit regex validation. Its self-test includes unterminated quotes, +redirection-shaped ``, heredocs, and both P2/P3 documents. + +Expected: every command exits 0. Record exact counts and versions, including Go 1.26.5, in the checkpoint report; no offline gate may download another Go toolchain. Do not run the full unrelated repository suites here; aggregate/full-stack verification belongs after P6 unless a touched-layer regression requires it. + +**Step 3: Run the complete process once, without retry.** + +```bash +./scripts/p3-acceptance.sh integration --keep +``` + +Expected: exit 0, one new `.artifacts/p3-effective-config//report.json`, matching `report.md`, ownership manifest, `overall: PASS`, `automated integration: PASS`, secret scan PASS, cleanup test PASS, and all fifteen assertions above PASS. Do not rerun a failure blindly: diagnose, add a regression test, fix, commit to regain clean state, then perform a new complete run with a new run ID. + +**Step 4: Independently validate retained evidence.** + +Use a small bounded script to parse `report.json`, verify every declared SHA-256, compare source commit/tree to current HEAD, confirm no undeclared public files, scan report/stdout/stderr/public manifests for every fixture canary, and query Docker by exact ownership labels. Then exercise retained cleanup: + +```bash +RUN_ID="${RUN_ID:?export RUN_ID as the retained 32-hex acceptance run ID}" +[[ "$RUN_ID" =~ ^[0-9a-f]{32}$ ]] +./scripts/p3-acceptance.sh cleanup --run "$RUN_ID" +./scripts/p3-acceptance.sh cleanup --run "$RUN_ID" +``` + +Expected: first cleanup removes exactly declared containers/networks/volumes/temp roots and writes cleanup PASS; second reports idempotent `already_clean`; sanitized reports remain. Confirm no owned listener accepts connections. + +**Step 5: Write and commit the automated checkpoint report.** + +`docs/reports/2026-08-10-p3-effective-config-checkpoint.md` must record commit/tree, tool versions, focused command counts, retained report path and SHA-256, automated result, cleanup result, known unrelated debt (if any), and `manual acceptance: PENDING`. Update the P3 manual header and `PROJECT_STATE.md` with the same retained evidence and explicitly state that P4 is blocked pending reviewer approval. + +```bash +git add docs/reports/2026-08-10-p3-effective-config-checkpoint.md \ + docs/testing/p2-p6-manual-verification.md PROJECT_STATE.md \ + scripts/lint-plan-shell-fences.py scripts/test-lint-plan-shell-fences.py +git commit -m "docs: record P3 automated verification" +``` + +Because that documentation commit is after the retained source commit, do not claim the report is bound to the doc commit; record both precisely. If policy requires a report bound to final docs too, run a fresh clean acceptance once and replace the retained reference rather than editing provenance. + +**Step 6: Stop for the human checkpoint.** + +Send the reviewer: + +- retained report path and hashes; +- concise architecture summary; +- focused verification results; +- exact P3 manual walkthrough section; +- explicit choices `PASS`, `FAIL with notes`, or `DEFER`. + +Do not begin P4. After the reviewer runs the walkthrough in a new environment and explicitly approves, update `Decision: PASS`, `manual acceptance: PASS`, `PROJECT_STATE.md`, and the checkpoint report in one scoped documentation commit: + +```bash +git add docs/testing/p2-p6-manual-verification.md \ + docs/reports/2026-08-10-p3-effective-config-checkpoint.md PROJECT_STATE.md +git commit -m "docs: record P3 manual acceptance" +``` + +Only that explicit decision completes P3 and unblocks P4. diff --git a/docs/superpowers/plans/2026-08-10-p4-qdrant-bootstrap-rebuild.md b/docs/superpowers/plans/2026-08-10-p4-qdrant-bootstrap-rebuild.md new file mode 100644 index 00000000..a0baeded --- /dev/null +++ b/docs/superpowers/plans/2026-08-10-p4-qdrant-bootstrap-rebuild.md @@ -0,0 +1,1010 @@ +# P4 Qdrant Bootstrap and Guarded Rebuild Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Implement PRD D4 so session admission safely self-heals a missing workspace Qdrant collection or missing keyword indexes, while `thothctl` can inspect, destructively rebuild, and recover only the descriptor-owned collection under durable maintenance and complete quiescence. + +**Architecture:** Move Qdrant collection inspection/reconciliation out of `ThtRunner` into one shared TypeScript manager used by session admission and P2's `WorkspacePreprocessingService.execute`. The manager has safe readiness reconciliation plus narrow destructive primitives—never a public monolithic `rebuild()`—so the guarded service can durably record every mutation boundary. Readiness may create only the fixed 1024/cosine collection contract and missing keyword indexes; incompatible vector or index types remain fail-closed. Destructive rebuild is a host-orchestrated transaction: `thothctl` holds one installation lifecycle lock, activates the backend's durable admission barrier, proves the complete session inventory and live process/admission counts are quiescent, stops `core`, and runs the dedicated Pi-free `backend/src/workspace-maintenance.ts::main` entrypoint under P2's writer lock. Success persists `deleting` before DELETE, `deleted` after a confirmed 404, `recreated` after exact collection/index creation, and `verified` only after a separate final inspection; a post-delete failure leaves maintenance active for explicitly confirmed recovery. + +**Tech Stack:** TypeScript 5, Node.js 22 `fetch`, Fastify 5, Vitest, Qdrant REST API v1.18, Go 1.26, `gofrs/flock`, Docker Compose, Bash, real Git fixtures, and JSON/YAML. + +--- + +## Source, prerequisites, and hard boundary + +- Source requirement: `docs/prd/2026-08-09-workspace-preprocessing-prd.md` D4, RF4.2–RF4.4, RNF1–RNF9, and section 8. +- Reviewed design: `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` section 6 and the common error/verification contracts. +- P1 is already accepted. Execute and checkpoint P2, then P3, before this plan because this implementation deliberately **extends**, rather than duplicates, P2's exact maintenance surface: + - `backend/src/workspaces/{runtime-config-lease.ts,preprocessing-state.ts,preprocessing-service.ts}` and `backend/src/workspace-maintenance.ts`; + - `tools/thothctl/internal/workspaceops/operations.go`; + - profile-only `workspace-maintenance` in `compose.yaml`; + - `/data/sessions//preprocessing/writer.lock` and `runUnderWorkspaceWriterLock(...)`. +- This plan is rebased on P2/P3's frozen handoff: `backend/src/workspace-maintenance.ts::main`, `WorkspacePreprocessingService.execute`, `PreprocessingStateStore`, `WorkspaceRuntimeConfigLeaseFactory`, `backend/src/workspaces/revision-layout.ts::{workspaceRuntimePaths,readRevisionLayoutState}`, `runUnderWorkspaceWriterLock` / `probeWorkspaceWriterLock`, and `tools/thothctl/internal/workspaceops::{ParseWorkspaceCommand,Run}`. Run the recorded P2/P3 focused gates before Task 1. Do not create a second operator, renderer, state root, lock, or `internal/workspace` package. +- P4 does not implement P5 Git annotations, P6 Evidence materialization, P7 migration, P8 aggregate smoke, P9 retention policy, P10 SSH runtime support, a GUI maintenance endpoint, or automatic semantic reindexing after rebuild. +- Qdrant remains derived data. Rebuild intentionally discards the selected collection's schema/Evidence/Memory projection. Canonical Git descriptors, phase artifacts, corpus state, and the memory registry remain untouched; operators must rerun the already-available indexing commands afterward. +- No prefix matching, collection enumeration followed by bulk deletion, Qdrant-wide cleanup, forced Pi termination, automatic session closure, or automatic retry is permitted. + +## Exact public command and result contract + +P2's read-only command gains a `semantic_index` and `collection_recovery` section: + +```text +thothctl --installation /abs/thothii-installation.yaml \ + workspace inspect --workspace research --json +``` + +P4 adds only these destructive commands: + +```text +thothctl --installation /abs/thothii-installation.yaml \ + workspace collection rebuild \ + --workspace research \ + --confirm-workspace research \ + --confirm-collection research-semantic \ + --destructive --json + +thothctl --installation /abs/thothii-installation.yaml \ + workspace collection recover \ + --workspace research \ + --confirm-workspace research \ + --confirm-collection research-semantic \ + --destructive --json +``` + +Rules: + +1. `--workspace`, `--confirm-workspace`, `--confirm-collection`, and the literal `--destructive` are all required once and only once for rebuild/recover. Values are compared byte-for-byte after the normal identifier syntax validation; no case folding, trimming, defaults, interactive prompts, or `--yes` alias. +2. The collection is always re-derived from the active, operational schema-v3 descriptor. A caller-supplied collection is never used as a mutation target until it exactly equals that value. +3. Confirmation mismatch exits 2 before maintenance activation, lifecycle state creation, collection mutation, or stopping `core`. +4. `inspect` is read-only and may run concurrently. It reports `missing`, `compatible`, `repairable` (only keyword indexes are absent), or `incompatible`, plus safe expected/observed dimensions, distance, required index names/types, active revision, maintenance state, and recovery phase. It never self-heals. +5. Machine output is one pristine JSON object. Stable P4 codes are `semantic_index_incompatible`, `preprocessing_conflict`, `collection_confirmation_mismatch`, `collection_recovery_required`, `collection_recovery_not_required`, `session_inventory_active`, `maintenance_not_quiescent`, and `workspace_not_activatable`. No response includes a Qdrant response body, arbitrary exception, raw child stderr, endpoint, credential, signed URL, or secret-file content. + +The fixed collection contract remains: + +```ts +export const REQUIRED_QDRANT_KEYWORD_INDEXES = [ + "content_hash", + "document_id", + "kind", + "record_key", + "record_kind", + "vector_generation", + "workspace_id", + "workspace_revision", +] as const; +``` + +Vectors are exactly size `1024`, distance `Cosine`; every listed payload index is exactly `keyword`. + +## Durable transaction and fail-safe matrix + +Store the collection transaction at: + +```text +/data/sessions//preprocessing/qdrant-rebuild-state.json +``` + +Use strict schema version 1: + +```ts +interface CollectionRebuildStateV1 { + version: 1; + transaction_id: string; + operation: "collection-rebuild"; + workspace_id: string; + workspace_revision: string; // exact 40-hex active commit + collection: string; // descriptor-derived exact name + phase: "prepared" | "deleting" | "deleted" | "recreated" | "verified" | "failed_pre_delete"; + mutation_started: boolean; + created_at: string; + updated_at: string; + error_code?: "semantic_index_incompatible" | "workspace_not_activatable"; +} +``` + +Write `prepared` durably before DELETE and write `deleting` with `mutation_started: true` durably **before** issuing DELETE. Therefore a crash with `mutation_started: true` is always treated as potentially destructive even when Qdrant still contains the collection. Atomic replace must fsync the file and owning directory, use regular no-follow 0600 files under the P2-owned preprocessing root, reject unknown keys/versions/identity drift, and never interpolate an exception into `error_code`. + +- Failure before `mutation_started` may mark `failed_pre_delete`, restart `core`, verify health, and clear maintenance. +- Failure or ambiguous subprocess output after `mutation_started` leaves `core` stopped and the durable maintenance marker active. The CLI prints the exact recover command using only safe workspace/collection identifiers. +- Recovery accepts only a nonterminal state matching the current descriptor workspace, revision, and collection. If the collection is missing it recreates it; if compatible it verifies it (covering a crash after recreate); if incompatible it may delete/recreate the exact same collection only because recovery repeats all destructive confirmations and the transaction proves mutation already began. It never targets a different or prefix-matched collection. +- `verified` is retained for inspection/audit. A later rebuild may atomically supersede it only after proving it is terminal. + +## Automated P4 process goal + +Start the execution goal before Task 1 and complete it only after this single clean-state command succeeds: + +```bash +./scripts/p4-acceptance.sh integration --keep +``` + +The command must refuse a dirty tracked source tree, bind the run to the exact Git commit/tree, use one unique Compose project and ownership manifest below `.artifacts/p4-integration//`, execute once with no automatic retry, and retain `report.json`, `report.md`, bounded command events, safe Qdrant observations, recovery state copies, artifact hashes, and cleanup proof. External LLM/DWH/Ollama behavior is outside D4 and uses fixture-only values; Qdrant, the compiled core image, backend routes, `thothctl`, Git registry snapshot, dedicated maintenance service, persistent volumes, and Compose lifecycle are real. + +The integration must prove: missing collection admission self-heal; missing-index self-heal; concurrent compatible creation/index reconciliation; incompatible dimension/distance/index refusal without PUT/DELETE; exact confirmation refusal before maintenance; complete open-session refusal; live admission/Pi count quiescence; held P2 writer-lock refusal before deletion; exact target-only deletion with a neighbor collection unchanged; verified rebuild/core restart/maintenance clear; a deterministic interruption after deletion with core stopped and marker retained; explicit recovery; secret scan; no retry; and exact owned-resource cleanup. `--keep` retains filesystem evidence, never live containers/networks/volumes or secrets. + +No unavoidable human action exists inside this automated goal. Manual acceptance happens afterward in a new independent environment. + +--- + +### Task 0: Establish the execution baseline and persistent goal + +**Files:** +- Read: `docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md` +- Read: `docs/superpowers/plans/2026-08-10-p3-effective-config-fingerprint.md` +- Read: `backend/src/workspaces/{runtime-config-lease.ts,revision-layout.ts,preprocessing-state.ts,preprocessing-service.ts}` and `backend/src/workspace-maintenance.ts` +- Read: `tools/thothctl/internal/workspaceops/operations.go` +- Read: `compose.yaml`, `deploy/compose.local.yaml`, `deploy/compose.server.yaml` + +- [ ] **Step 1: Confirm the worktree and prerequisites** + +```bash +git status --short +git log -1 --format='%H %T' +test -f backend/src/workspaces/preprocessing-state.ts +test -f tools/thothctl/internal/workspaceops/operations.go +``` + +Expected: clean output from `git status`; one commit/tree line; both files exist. Confirm P2 and P3 checkpoint reports say automated integration PASS. If not, stop—do not fold their scope into P4. + +- [ ] **Step 2: Start the process goal** + +If the execution environment supports persistent goals, create: “P4 clean-state Qdrant bootstrap/rebuild/recovery process passes once without retry, retains a secret-clean report, and cleans only owned resources.” Keep it open through Task 11. + +- [ ] **Step 3: Run prerequisite focused gates** + +Run the exact P2/P3 focused verification commands recorded in their checkpoint reports. + +Expected: PASS. A failure is prerequisite drift; fix it in its owning plan before P4. + +No commit. + +--- + +### Task 1: Extract the shared Qdrant collection manager + +**Files:** +- Create: `backend/src/semantic/qdrant-collection-manager.ts` +- Create: `backend/test/qdrant-collection-manager.test.ts` +- Modify: `backend/src/tht/tht-runner.ts` (`QdrantEnsureResult`, `REQUIRED_QDRANT_PAYLOAD_INDEXES`, `qdrantEnsure`) +- Modify: `backend/test/tht-qdrant-readiness.test.ts` + +**Required interface:** + +```ts +export type CollectionState = "missing" | "compatible" | "repairable" | "incompatible"; +export interface QdrantCollectionSpec { + collection: string; + dimensions: 1024; + distance: "cosine"; + keywordIndexes: readonly string[]; +} +export interface CollectionInspection { + state: CollectionState; + expected: { dimensions: 1024; distance: "cosine"; keyword_indexes: readonly string[] }; + observed?: { dimensions?: number; distance?: string; keyword_indexes: Record }; + missing_keyword_indexes: string[]; + incompatible_fields: string[]; +} +export class QdrantCollectionManager { + constructor(options: { baseUrl: string; request?: typeof fetch }); + inspect(spec: QdrantCollectionSpec, timeoutMs: number): Promise; + ensure(spec: QdrantCollectionSpec, timeoutMs: number): Promise<{ ok: boolean; code?: SemanticReadinessCode }>; + deleteExactAndConfirmMissing(spec: QdrantCollectionSpec, timeoutMs: number): Promise; + ensureCollection(spec: QdrantCollectionSpec, timeoutMs: number): Promise; + ensureIndexes(spec: QdrantCollectionSpec, timeoutMs: number): Promise; +} +export function qdrantCollectionSpec(workspace: WorkspaceDescriptor): QdrantCollectionSpec; +``` + +There is deliberately no exported `rebuild()` and no manager method that can cross more than one durable destructive phase. `deleteExactAndConfirmMissing` issues one exact DELETE and succeeds only after a separate GET confirms 404. `ensureCollection` creates or converges only the exact 1024/Cosine collection and finishes with a read proving the vector contract (keyword indexes may still be absent). `ensureIndexes` creates only missing required keyword indexes in canonical order and finishes with a read proving the exact complete contract. The guarded service in Task 3 is the sole production composer of these primitives; readiness calls only `ensure`. + +- [ ] **Step 1: Write RED inspection tests** + +Use a scripted `vi.fn` fetch boundary. Cover exact URL encoding, GET 404 → `missing`, exact 1024/Cosine/eight keyword indexes → `compatible`, missing index → `repairable`, and wrong size/distance/index type → `incompatible`. Reject malformed success JSON as `workspace_not_activatable`; never propagate a body canary. + +- [ ] **Step 2: Verify RED** + +```bash +cd backend +npx vitest run test/qdrant-collection-manager.test.ts +``` + +Expected: FAIL because the module is missing. + +- [ ] **Step 3: Implement read-only inspection** + +Use only `new URL('/collections/' + encodeURIComponent(name), baseUrl)`, an operation-wide `AbortController`, allowlisted parsed fields, and lowercase comparison for Qdrant's distance/type response. Distinguish missing from incompatible; an unavailable/unparseable service is not semantic incompatibility. + +- [ ] **Step 4: Write RED self-heal/concurrency tests** + +Cover these exact request sequences: + +1. GET 404 → PUT collection with `{"vectors":{"size":1024,"distance":"Cosine"}}` → final GET compatible. +2. GET 404 → PUT 409/already exists → final GET compatible (concurrent compatible creator). +3. GET repairable → PUT `/collections//index?wait=true` with `{"field_name":"kind","field_schema":"keyword"}` → final GET compatible. +4. Index PUT conflict → final GET compatible (concurrent compatible index creator). +5. A barrier-controlled unit race launches two independent `manager.ensure` calls (not a shared promise): both initial GETs observe 404/repairable, one PUT succeeds, the other receives the allowed conflict, both perform their own final GET, and both converge compatible. Repeat for collection creation and one missing index; assert no retry loop. +6. Concurrent creator/index ends incompatible → `semantic_index_incompatible`. +7. Existing incompatible size/distance/index performs no PUT or DELETE. +8. A missing-index run never recreates the collection and creates only missing indexes in canonical order. +9. Timeout/unreachable response returns `workspace_not_activatable` with no endpoint/body leak. +10. `deleteExactAndConfirmMissing` issues DELETE for the one URL-encoded exact collection and then independently reads 404; DELETE failure or a non-404 post-delete state fails. +11. `ensureCollection` and `ensureIndexes` are separately observable: collection creation ends with the exact vector contract present, index creation ends with the exact complete contract, and neither ever deletes. A later independent `inspect` supplies final verification. No other collection name, list endpoint, prefix, or global mutation is ever requested. +12. No exported `rebuild`, callback that hides multiple phases, or test-only destructive shortcut exists. + +- [ ] **Step 5: Verify RED, then implement minimal reconciliation** + +```bash +cd backend +npx vitest run test/qdrant-collection-manager.test.ts +``` + +Expected before implementation: FAIL on PUT sequences. Implement create/index calls, tolerate only conflict/already-exists as a reason to re-read, and always decide success from one final GET. Do not accept the mutating response as proof. + +- [ ] **Step 6: Delegate `ThtRunner.qdrantEnsure`** + +Keep descriptor operational validation in `qdrantEnsure`, derive `qdrantCollectionSpec`, and call the shared manager. Remove the duplicate constant/parsing logic. Preserve `ThtConfig.qdrantRequest` as the injectable fetch boundary. + +- [ ] **Step 7: Run focused tests and typecheck** + +```bash +cd backend +npx vitest run test/qdrant-collection-manager.test.ts test/tht-qdrant-readiness.test.ts test/readiness-manager.test.ts +npx tsc --noEmit -p . +``` + +Expected: PASS. Update the old missing-collection assertion from incompatibility to success after the exact create/final-read sequence; existing incompatible cases remain fail-closed. + +- [ ] **Step 8: Commit** + +```bash +git add backend/src/semantic/qdrant-collection-manager.ts \ + backend/src/tht/tht-runner.ts \ + backend/test/qdrant-collection-manager.test.ts \ + backend/test/tht-qdrant-readiness.test.ts \ + backend/test/readiness-manager.test.ts +git commit -m "feat: self-heal workspace qdrant collections" +``` + +--- + +### Task 2: Prove session admission uses self-heal before Ollama and persistence + +**Files:** +- Modify: `backend/test/routes-sessions.test.ts` (readiness cases around current Qdrant tests) +- Modify if dependency injection requires it: `backend/src/app.ts` (`ThtRunner` construction only) + +- [ ] **Step 1: Write RED route tests** + +At the real `buildApp` boundary, inject a `ThtRunner`/fetch script and assert: + +- missing collection creates the exact contract, then Ollama runs, then session persistence is allowed; +- missing index creates only that index before Ollama; +- incompatible collection returns 503 with `semantic_index_incompatible`, does not call Ollama, does not call `sessionNew`, acquire a revision lease permanently, or spawn Pi; +- two simultaneous **local-mode, same-principal, same-workspace** admissions hit `ReadinessManager`'s in-flight key and share one promise: exactly one Qdrant create/reconciliation occurs, both callers receive the same compatible result, and no false incompatibility is emitted; +- one separate `ReadinessManager` unit test proves the same-principal dedup key and cleanup after resolution/rejection; +- one **upstream-auth, two-distinct-principal** route test uses distinct readiness keys, coordinates both initial GET 404 reads, lets one PUT create and the other PUT receive the compatible-creator conflict, and proves both converge after their final reads. Do not describe this as a local same-principal race; +- raw Qdrant body/endpoint canaries do not appear in HTTP JSON or logs captured by the test. + +Keep the compatible-creator and compatible-index conflict sequences in `qdrant-collection-manager.test.ts` as the direct manager race proof. Route tests prove the real dedup/cross-principal semantics rather than attempting to bypass `ReadinessManager`. + +- [ ] **Step 2: Run and verify RED** + +```bash +cd backend +npx vitest run test/routes-sessions.test.ts -t "Qdrant|semantic index|concurrent collection" +``` + +Expected: the new call-order, local deduplication, and distinct-principal convergence assertions FAIL until route fixtures use the real manager behavior and upstream identities produce distinct readiness keys. + +- [ ] **Step 3: Make the minimal wiring change** + +Do not add a new route. Session admission continues through `ReadinessManager.ensure(..., descriptor)` and `ThtRunner.qdrantEnsure`; change only construction/injection necessary to share the manager. + +- [ ] **Step 4: Verify** + +```bash +cd backend +npx vitest run \ + test/routes-sessions.test.ts \ + test/readiness-manager.test.ts \ + test/tht-qdrant-readiness.test.ts +npx tsc --noEmit -p . +``` + +Expected: PASS, including Qdrant-before-Ollama ordering, exactly one create for local same-principal deduplication, two independently convergent manager calls for distinct upstream principals, and no persistence on incompatibility. + +- [ ] **Step 5: Commit** + +```bash +git add backend/src/app.ts backend/test/routes-sessions.test.ts backend/test/readiness-manager.test.ts +git commit -m "test: prove qdrant admission self-heal" +``` + +Omit unchanged paths from `git add`. + +--- + +### Task 3: Add safe inspection and durable rebuild/recovery to the P2 operator + +**Files:** +- Create: `backend/src/workspaces/qdrant-rebuild-state.ts` +- Create: `backend/test/qdrant-rebuild-state.test.ts` +- Modify: `backend/src/workspaces/preprocessing-service.ts` (`MaintenanceCommand`/result union) +- Modify: `backend/src/workspaces/preprocessing-service.ts` (`WorkspacePreprocessingService.execute`) +- Modify: `backend/src/workspace-maintenance.ts` (`main` parser/encoder) +- Modify: `backend/src/workspaces/preprocessing-state.ts` +- Modify: `backend/test/workspace-preprocessing-service.test.ts` +- Modify: `backend/test/workspace-maintenance.test.ts` +- Modify: `backend/test/workspace-preprocessing-state.test.ts` + +**Operator additions:** + +```ts +type MaintenanceCommand = + | /* P2/P3 requests */ + | { operation: "collection-rebuild"; workspace_id: string; confirm_workspace: string; + confirm_collection: string; destructive: true } + | { operation: "collection-recover"; workspace_id: string; confirm_workspace: string; + confirm_collection: string; destructive: true }; +``` + +`inspect` gains safe `semantic_index` and `collection_recovery`; it never acquires a writer lock or mutates Qdrant. Rebuild/recover must call `runUnderWorkspaceWriterLock` for the complete state-read → mutation → verification transaction. A conflict maps to `preprocessing_conflict` before state or Qdrant mutation. + +- [ ] **Step 1: Write RED state-store tests** + +Test restrictive creation, atomic durable transitions, same-inode/root safety, unknown schema/keys, workspace/revision/collection mismatch, symlink/path swap, injected file-fsync and directory-fsync ambiguity, and terminal supersession. Table-test the only legal forward edges `prepared → deleting → deleted → recreated → verified` plus `prepared → failed_pre_delete`; reject skips, rewinds, cross-transaction updates, and `failed_pre_delete` after mutation. Add crash-visible assertions at every boundary: `deleting/mutation_started` is durable before mocked DELETE, `deleted` only after the manager confirms 404, `recreated` only after both exact collection and required indexes are confirmed, and `verified` only after one additional independent `inspect` returns compatible. + +- [ ] **Step 2: Verify RED** + +```bash +cd backend +npx vitest run test/qdrant-rebuild-state.test.ts +``` + +Expected: FAIL because `CollectionRebuildStateStore` does not exist. + +- [ ] **Step 3: Implement the strict store** + +Reuse P2's trusted preprocessing-root and atomic state primitives rather than another root policy. Expose only `read`, `begin`, and `transition(expectedTransaction, next)`; enforce legal monotonic phase transitions. Report durability uncertainty as recovery-required whenever the intended bytes may have reached the canonical path. + +- [ ] **Step 4: Write RED operator tests** + +Cover: + +- `inspect` returns active workspace/revision, exact descriptor collection, manager inspection, and safe recovery state without mutation/lock; +- confirmation mismatch and missing `destructive: true` return `collection_confirmation_mismatch` before marker/lock/Qdrant; +- rebuild refuses unless `/data/settings/maintenance.json` is a trusted regular file whose strict JSON says active; +- held `/preprocessing/writer.lock` returns `preprocessing_conflict` before DELETE; +- rebuild writes/fsyncs `prepared`, then writes/fsyncs `deleting` with `mutation_started:true` before `deleteExactAndConfirmMissing`; after that primitive's confirmed 404 it writes/fsyncs `deleted`; after `ensureCollection` and `ensureIndexes` have separately confirmed the exact contract it writes/fsyncs `recreated`; only a subsequent standalone `inspect` returning compatible permits `verified`; +- injected crashes after each durable phase leave exactly that phase visible, never infer a later phase from a mutating response, and resume through the same narrow manager primitives; +- neighbor collections are never requested; +- failure before `deleting` becomes `failed_pre_delete` with `mutation_started: false`; +- failure/timeout at or after `deleting` retains the last durable nonterminal state with `mutation_started: true` and returns a safe recovery-required envelope; +- recover handles every nonterminal phase: `prepared` may start deletion only after repeated confirmations; `deleting` first inspects and conservatively establishes missing/compatible/incompatible exact state; `deleted` recreates; `recreated` independently verifies; missing, already-compatible-after-crash, and explicitly confirmed incompatible exact targets all converge without skipping a durable boundary; +- recover rejects absent/verified state, changed active revision, changed descriptor collection, and unsafe/unknown state; +- error output excludes Qdrant body, marker contents beyond allowlisted booleans, paths, endpoints, and canaries. + +- [ ] **Step 5: Run and verify RED** + +```bash +cd backend +npx vitest run \ + test/workspace-preprocessing-service.test.ts \ + test/workspace-preprocessing-state.test.ts \ + test/workspace-maintenance.test.ts +``` + +Expected: new operation cases FAIL because dispatch/state enforcement is absent. + +- [ ] **Step 6: Implement guarded dispatch** + +Derive the descriptor and revision through `WorkspacePreprocessingService.execute` and its existing P2 resolver, validate confirmations, verify the marker through `lstat/open(O_NOFOLLOW)/fstat` and strict JSON, then enter `runUnderWorkspaceWriterLock`. Compose only `inspect`, `deleteExactAndConfirmMissing`, `ensureCollection`, and `ensureIndexes`, persisting/fsyncing the state between calls exactly as specified above. Neither `backend/src/workspace-maintenance.ts::main` nor any other caller receives a monolithic destructive API. The CLI accepts fixed argv generated by `workspaceops.Run` and preserves one pristine JSON envelope; direct manual invocation without marker/confirmations remains harmless. + +- [ ] **Step 7: Verify focused backend tests** + +```bash +cd backend +npx vitest run \ + test/qdrant-collection-manager.test.ts \ + test/qdrant-rebuild-state.test.ts \ + test/workspace-preprocessing-state.test.ts \ + test/workspace-preprocessing-service.test.ts \ + test/workspace-maintenance.test.ts +npx tsc --noEmit -p . +npm run build +``` + +Expected: PASS; build emits `/app/backend/dist/workspace-maintenance.js` and the new shared manager/state module. + +- [ ] **Step 8: Commit** + +```bash +git add backend/src/workspaces/qdrant-rebuild-state.ts \ + backend/src/workspaces/preprocessing-state.ts \ + backend/src/workspaces/preprocessing-service.ts \ + backend/src/workspace-maintenance.ts \ + backend/test/qdrant-rebuild-state.test.ts \ + backend/test/workspace-preprocessing-state.test.ts \ + backend/test/workspace-preprocessing-service.test.ts \ + backend/test/workspace-maintenance.test.ts +git commit -m "feat: add guarded qdrant maintenance operations" +``` + +--- + +### Task 4: Expose loopback-only maintenance quiescence + +**Files:** +- Modify: `backend/src/app.ts` (`maintenance status handlers`, `isMaintenanceControl`) +- Modify: `backend/src/runtime/maintenance-gate.ts` only if a named status type is needed +- Modify: `backend/test/routes-sessions.test.ts` (current maintenance endpoint tests) +- Modify: `backend/test/maintenance-gate.test.ts` only for status typing/durability regressions +- Modify: `backend/test/pi-process-manager.test.ts` + +**Response:** + +```json +{ + "active": true, + "admissions": 0, + "piProcesses": 0, + "quiescent": true +} +``` + +Include `recoveryRequired: true` only when already produced by `MaintenanceBarrier`. `quiescent` is true only when the durable marker is active, `admissions === 0`, `mgr.count() === 0`, and no durability recovery is pending. + +- [ ] **Step 1: Write RED endpoint tests** + +Test `POST /internal/maintenance/activate`, `GET /internal/maintenance/status`, and new `GET /internal/maintenance/quiescence` with injected manager counts 0/1. Prove activation waits for an in-flight admission lease, later admissions receive 503, Pi count is observational (never killed), spoofed non-loopback callers receive 403 in `none` and `upstream` auth, and durability ambiguity makes `quiescent: false`. + +- [ ] **Step 2: Verify RED** + +```bash +cd backend +npx vitest run test/routes-sessions.test.ts -t "maintenance|quiescence" +``` + +Expected: FAIL because responses omit `piProcesses`/`quiescent` and the route is not allowlisted. + +- [ ] **Step 3: Implement one response composer** + +In `buildApp`, define `maintenanceStatus()` from `maintenanceBarrier.status()` plus `mgr.count()`. Use it for activate/deactivate/status/quiescence responses so the fields cannot drift. Do not expose runtime IDs, principals, session questions, or child process details. + +- [ ] **Step 4: Verify** + +```bash +cd backend +npx vitest run \ + test/maintenance-gate.test.ts \ + test/routes-sessions.test.ts \ + test/pi-process-manager.test.ts +npx tsc --noEmit -p . +``` + +Expected: PASS. Existing admission behavior remains unchanged except for additive internal response fields. + +- [ ] **Step 5: Commit** + +```bash +git add backend/src/app.ts backend/src/runtime/maintenance-gate.ts \ + backend/test/routes-sessions.test.ts backend/test/maintenance-gate.test.ts \ + backend/test/pi-process-manager.test.ts +git commit -m "feat: report complete maintenance quiescence" +``` + +--- + +### Task 5: Share one installation lifecycle lock with Pi operations + +**Files:** +- Create: `tools/thothctl/internal/lifecycle/lock.go` +- Create: `tools/thothctl/internal/lifecycle/lock_test.go` +- Modify: `tools/thothctl/internal/config/installation.go` (`LifecycleLockPath`, `CollectionRebuildStatePath` only if host metadata is retained) +- Modify: `tools/thothctl/internal/config/installation_test.go` +- Modify: `tools/thothctl/internal/pi/state.go` (`acquireLock`, `updateLock`, `ErrLockHeld` removal/delegation) +- Modify: `tools/thothctl/internal/pi/update.go` (`Update`, `Rollback`, `RecoverMaintenance` lock acquisition) +- Modify: `tools/thothctl/internal/pi/update_test.go` +- Modify: `tools/thothctl/internal/pi/state_test.go` + +**Path:** `.thothctl//lifecycle.lock`, with owner metadata `.thothctl//lifecycle.lock.owner.json`. + +- [ ] **Step 1: Write RED lifecycle tests** + +Test stable 0600 lock inode, 0600 atomic owner metadata, nonblocking cross-process contention, metadata removal only by the owner on release, crash-safe kernel release, symlink/unsafe control-directory rejection, and distinct installation descriptors not contending. Add a test that a Pi update lock blocks a simulated collection rebuild and vice versa. + +- [ ] **Step 2: Verify RED** + +```bash +cd tools/thothctl +go test ./internal/lifecycle ./internal/config +``` + +Expected: FAIL because the lifecycle package/path does not exist. + +- [ ] **Step 3: Extract the current lock without changing Pi transaction semantics** + +Move the `gofrs/flock` implementation from `internal/pi/state.go` into `lifecycle.Acquire(path, operation)`. Owner JSON contains PID, host, start time, transaction, and an allowlisted operation (`pi-update`, `pi-rollback`, `pi-maintenance-recover`, `qdrant-rebuild`, `qdrant-recover`), never argv/environment. Add `LifecycleLockPath string` to `pi.Request`; change `Rollback` and `RecoverMaintenance` to accept `(statePath, lifecycleLockPath, confirm)`; and make `main.go` pass `installation.LifecycleLockPath()` to all three paths. Reject an empty/noncanonical lock path. The update state remains `UpdateStatePath()`; only mutual exclusion moves from `update-state.json.lock` to the installation lock. + +- [ ] **Step 4: Run Pi regressions** + +```bash +cd tools/thothctl +go test ./internal/lifecycle ./internal/config ./internal/pi +``` + +Expected: PASS; interrupted update/rollback behavior and recovery metadata remain identical, but P4 and Pi now contend on one installation lock. + +- [ ] **Step 5: Commit** + +```bash +git add tools/thothctl/internal/lifecycle \ + tools/thothctl/internal/config/installation.go \ + tools/thothctl/internal/config/installation_test.go \ + tools/thothctl/internal/pi/state.go tools/thothctl/internal/pi/state_test.go \ + tools/thothctl/internal/pi/update.go tools/thothctl/internal/pi/update_test.go +git commit -m "refactor: share installation lifecycle lock" +``` + +--- + +### Task 6: Parse exact collection commands and confirmations in `thothctl` + +**Files:** +- Modify: `tools/thothctl/internal/workspaceops/operations.go` +- Modify: `tools/thothctl/internal/workspaceops/operations_test.go` +- Create: `tools/thothctl/internal/workspaceops/collection.go` +- Create: `tools/thothctl/internal/workspaceops/collection_test.go` +- Modify: `tools/thothctl/cmd/thothctl/main.go` +- Modify: `tools/thothctl/cmd/thothctl/main_test.go` + +- [ ] **Step 1: Write RED parse tests** + +Table-test the exact commands above. Reject missing/duplicate flags, swapped confirmations, whitespace/case differences, unknown flags/subcommands, positional targets, `--yes`, absent `--destructive`, unsafe workspace/collection identifiers, and rebuild/recover input or output file flags inherited from P2. Confirm usage errors exit 2 and never call Docker. + +- [ ] **Step 2: Verify RED** + +```bash +cd tools/thothctl +go test ./internal/workspaceops ./cmd/thothctl -run 'Collection|collection|Confirmation' +``` + +Expected: FAIL because P2 parsing knows no `workspace collection` family. + +- [ ] **Step 3: Extend `workspaceops.ParseWorkspaceCommand` and main dispatch** + +Use typed operations `CollectionRebuild` and `CollectionRecover`; do not add another top-level parser. Perform syntactic equality checks in Go before Docker, then pass all values to TypeScript for authoritative descriptor equality checks. + +- [ ] **Step 4: Extend inspect parsing/output** + +P2 `workspace inspect` keeps the same command syntax. Validate the returned semantic/recovery JSON structure and print pristine JSON under `--json`; human output lists exact workspace, revision, collection, state, missing/incompatible fields, maintenance, and recovery phase without endpoint details. + +- [ ] **Step 5: Verify** + +```bash +cd tools/thothctl +go test ./internal/workspaceops ./cmd/thothctl +``` + +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add tools/thothctl/internal/workspaceops/operations.go \ + tools/thothctl/internal/workspaceops/operations_test.go \ + tools/thothctl/internal/workspaceops/collection.go \ + tools/thothctl/internal/workspaceops/collection_test.go \ + tools/thothctl/cmd/thothctl/main.go tools/thothctl/cmd/thothctl/main_test.go +git commit -m "feat: add qdrant collection operator commands" +``` + +--- + +### Task 7: Orchestrate maintenance, inventory, stop, rebuild, restart, and recovery + +**Files:** +- Modify: `tools/thothctl/internal/workspaceops/collection.go` +- Modify: `tools/thothctl/internal/workspaceops/collection_test.go` +- Modify: `tools/thothctl/internal/workspaceops/operations.go` +- Modify: `tools/thothctl/internal/workspaceops/operations_test.go` +- Modify: `tools/thothctl/cmd/thothctl/main.go` +- Modify: `tools/thothctl/cmd/thothctl/main_test.go` +- Modify: `tools/thothctl/internal/pi/update.go` only to share strict maintenance parsing/helpers where appropriate +- Modify: `tools/thothctl/internal/pi/update_test.go` + +**Required sequence for rebuild:** + +```text +Acquire installation lifecycle lock +→ workspace inspect and authoritative confirmation recheck +→ POST core /internal/maintenance/activate +→ GET complete /sessions?scope=mine(local)|all(server) +→ reject unless every item is archived OR status closed/finalized +→ poll /internal/maintenance/quiescence until active + admissions=0 + piProcesses=0 +→ compose stop core +→ compose ps --status running -q core must be empty +→ compose --profile workspace-maintenance run --rm --no-deps -T + workspace-maintenance collection-rebuild +→ parse exact safe JSON and require verified state +→ compose up --detach --no-deps core +→ poll Compose health and loopback /health +→ require maintenance still active and quiescent +→ POST /internal/maintenance/deactivate +→ require active=false +→ release lifecycle lock +``` + +Use a bounded monotonic timeout (document 60 seconds for quiescence and 120 seconds for core health), one-second polling, and **no retry of a failed operation**. Polling observation is not retrying a mutation. + +- [ ] **Step 1: Write RED orchestration tests with a scripted runner** + +Assert exact argv/order and failure behavior for: + +- confirmation mismatch before lock/HTTP; +- lifecycle contention with Pi; +- activation durability failure; +- malformed/incomplete/duplicate session inventory; +- open/failed unarchived session refusal; archived, closed, and finalized acceptance; +- `admissions > 0` and `piProcesses > 0` poll, then success; +- timeout without forced teardown; +- core stop failure or still-running proof; +- P2 writer-lock conflict returned by the maintenance service before deletion, followed by safe core restart/health/maintenance clear; +- successful operator `verified` response, exact core restart, health, and marker clear; +- malformed/ambiguous operator result treated as mutation-started/recovery-required unless a trusted response proves `mutation_started:false`; +- post-delete error leaves core stopped, marker active, state retained, and prints exact recover command; +- neighbor services/collections never appear in stop/delete argv; +- all child stderr is sanitized and bounded. + +- [ ] **Step 2: Verify RED** + +```bash +cd tools/thothctl +go test ./internal/workspaceops -run 'Rebuild|Quiescence|Inventory|Recovery' +``` + +Expected: FAIL because lifecycle orchestration is not implemented. + +- [ ] **Step 3: Implement strict inventory and quiescence clients** + +Reuse the existing core-side curl identity headers. Decode exactly one bounded JSON document. A complete local install uses `scope=mine`; server uses admin `scope=all`. Never infer quiescence only from manifests: both `admissions` and `piProcesses` must reach zero after the durable marker is active. + +- [ ] **Step 4: Implement core stop/start proof and fail-safe cleanup** + +Use fixed Compose argv and explicit `core`. Do not call `down`, stop Qdrant, stop frontend, use `--remove-orphans`, or prune. Clear maintenance only after a trusted pre-delete failure or complete verified success and healthy core. + +- [ ] **Step 5: Write RED recovery tests** + +Test core-already-stopped recovery, core-running-but-maintained recovery, missing marker refusal, no/nonterminal/verified state, descriptor revision drift, recovery service failure, compatible-after-crash verification, missing recreate, explicitly confirmed incompatible recreate, successful restart/health/deactivate, and repeated recover returning `collection_recovery_not_required` without mutation. + +- [ ] **Step 6: Implement recovery** + +When core is stopped, the durable marker plus matching nonterminal state is the admission barrier; the maintenance service re-verifies both before mutation. When core is running, repeat activation/inventory/quiescence before stopping it. Never clear the marker merely because recovery cannot read state. + +- [ ] **Step 7: Run Go suites** + +```bash +cd tools/thothctl +go test ./internal/workspaceops ./internal/lifecycle ./internal/pi ./cmd/thothctl +go test ./... +``` + +Expected: PASS. + +- [ ] **Step 8: Commit** + +```bash +git add tools/thothctl/internal/workspaceops/collection.go \ + tools/thothctl/internal/workspaceops/collection_test.go \ + tools/thothctl/internal/workspaceops/operations.go \ + tools/thothctl/internal/workspaceops/operations_test.go \ + tools/thothctl/internal/pi/update.go tools/thothctl/internal/pi/update_test.go \ + tools/thothctl/cmd/thothctl/main.go tools/thothctl/cmd/thothctl/main_test.go +git commit -m "feat: orchestrate recoverable qdrant rebuilds" +``` + +--- + +### Task 8: Harden the dedicated maintenance service and Compose contract + +**Files:** +- Modify: `compose.yaml` (`workspace-maintenance` only) +- Modify: `deploy/compose.server.yaml` (`workspace-maintenance` bind roots) +- Verify/modify as P2 created them: `deploy/compose.git-https.yaml`, `deploy/compose.git-ssh.yaml` +- Modify: `docker/core.Dockerfile` (ensure `/usr/bin/flock` from `util-linux` is present) +- Modify: `scripts/test-internal-semantic-compose.sh` +- Modify: `scripts/test-unified-compose.sh` +- Modify: `scripts/test-compose-secret-policy.sh` +- Modify: `scripts/test-no-deployment-coupling.sh` +- Modify: `scripts/test-deployment-command-contract.sh` + +The service gains the shared settings mount needed to read the durable maintenance marker. It retains P2's sessions/registry mounts, internal Qdrant network, same selected core image, and direct Node entrypoint. It must still have no `/home/thoth/.pi`, PI auth/models/settings, writable Pi state, frontend dependency, host Docker socket, published port, or normal core entrypoint/trust initialization. + +- [ ] **Step 1: Extend RED Compose contract tests** + +Assert local named volumes and server bind roots make the **same** `/data/settings`, `/data/sessions`, and `/data/workspace-registry` visible to `core` and `workspace-maintenance`; Qdrant is reachable only on the private network; only the exact required Git/connector secret files attach; service is profile-only; `flock` exists in the built core image; and service cannot start Pi or expose HTTP. + +- [ ] **Step 2: Verify RED** + +```bash +./scripts/test-internal-semantic-compose.sh +./scripts/test-unified-compose.sh +./scripts/test-compose-secret-policy.sh +./scripts/test-deployment-command-contract.sh +``` + +Expected: at least the settings/marker and `flock` assertions FAIL. + +- [ ] **Step 3: Make minimal Compose/image changes** + +Add `util-linux` explicitly rather than relying on a transitive base package. Extend the P2 service only; do not introduce a second maintenance service or mount Pi state. + +- [ ] **Step 4: Run deployment gates** + +```bash +./scripts/test-default-compose.sh +./scripts/test-unified-compose.sh +./scripts/test-internal-semantic-compose.sh +./scripts/test-compose-secret-policy.sh +./scripts/test-no-deployment-coupling.sh +./scripts/test-deployment-command-contract.sh +``` + +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add compose.yaml deploy/compose.server.yaml \ + deploy/compose.git-https.yaml deploy/compose.git-ssh.yaml \ + docker/core.Dockerfile scripts/test-internal-semantic-compose.sh \ + scripts/test-unified-compose.sh scripts/test-compose-secret-policy.sh \ + scripts/test-no-deployment-coupling.sh scripts/test-deployment-command-contract.sh +git commit -m "build: isolate qdrant maintenance service" +``` + +--- + +### Task 9: Build the clean-state P4 process goal and retained report + +**Files:** +- Create: `scripts/p4-acceptance.sh` +- Create: `scripts/test-p4-acceptance.sh` +- Create: `backend/scripts/p4-acceptance.mjs` +- Create: `backend/scripts/p4-acceptance.test.mjs` +- Create: `backend/scripts/fixtures/p4-qdrant-proxy.mjs` +- Modify: `.gitignore` only if the existing `.artifacts/` rule is insufficient + +**Owned layout:** + +```text +.artifacts/p4-integration// +├── ownership.json +├── source.json +├── installation/thothii-installation.yaml +├── registry/{remote.git,author}/ +├── fixture-secrets/ +├── compose/acceptance.override.yaml +├── requests/ +├── responses/ +├── observations/ +├── recovery/qdrant-rebuild-state.json +├── logs/ +├── artifact-manifest.json +├── cleanup.json +├── report.json +└── report.md +``` + +- [ ] **Step 1: Write RED wrapper/orchestrator tests** + +Test argument parsing (`integration [--keep]` only), dirty tracked-tree refusal, unique run/project naming, fixture-only credentials, fixed toolchain resolution, no retry loop, bounded output, ownership checks, report schema, artifact hashes, secret-canary scan, cleanup behavior on success/failure/signal, and `--keep` retaining only the owned filesystem root. + +- [ ] **Step 2: Verify RED** + +```bash +./scripts/test-p4-acceptance.sh +``` + +Expected: FAIL because the wrapper/orchestrator are missing. + +- [ ] **Step 3: Implement the isolated topology** + +Use a real local bare Git remote and schema-v3 descriptor, compiled core image, real Qdrant storage, and production `thothctl`. An acceptance-only private proxy named `qdrant` fronts a real uniquely named Qdrant service; it forwards normal REST calls and provides a deterministic, test-owned barrier after DELETE so the one-shot maintenance container can be killed after durable `mutation_started` state and before recreate. It must not alter production code paths or accept secrets. Ollama/DWH/LLM use safe fixtures because they are outside D4. + +- [ ] **Step 4: Implement positive and negative checks** + +Record named PASS checks for every assertion in the automated goal. For the session self-heal check, traverse the real backend admission route and intentionally stop at a later fixture readiness boundary only after Qdrant creation has been final-read compatible; assert no session manifest/Pi process survives. For concurrency, prove both layers separately: coordinate two direct real manager calls through the proxy so one compatible create/index conflicts, and send two simultaneous local same-principal route admissions to prove `ReadinessManager` performs exactly one create. Also run the upstream-auth route with two distinct principals if the cross-principal route proof is retained. Do not loop/retry until green and do not claim two local same-principal route calls both reached Qdrant. + +Preseed one target point and one neighbor collection/point. Successful rebuild must remove/recreate only the target, leave the neighbor byte/count identity unchanged, retain canonical source files, and report that reindexing is required. + +- [ ] **Step 5: Implement interruption and recovery check** + +Block recreate after the exact target DELETE, kill only the uniquely labelled one-shot container, and assert: command failure; core stopped; maintenance marker active; state `mutation_started:true`; admissions refused if core is deliberately restarted under the marker. Release the proxy barrier and run the exact confirmed `collection recover` once; assert verified collection, healthy core, inactive marker, terminal state, and neighbor preservation. + +- [ ] **Step 6: Implement secret scan, artifact binding, and exact cleanup** + +Scan every retained regular file and bounded report/log field for all fixture secret canaries and credential-shaped URLs. Hash all declared artifacts, then remove only containers/volumes/network/proxy image/reference carrying this run's exact Compose project/run label. Prove no owned resource remains and no pre-existing resource was changed. There is no global `docker system prune`, volume prefix wildcard, or broad process kill. + +- [ ] **Step 7: Verify harness syntax/unit contract** + +```bash +bash -n scripts/p4-acceptance.sh scripts/test-p4-acceptance.sh +node --check backend/scripts/p4-acceptance.mjs +node --check backend/scripts/p4-acceptance.test.mjs +node --check backend/scripts/fixtures/p4-qdrant-proxy.mjs +./scripts/test-p4-acceptance.sh +``` + +Expected: PASS without starting the full process from the unit-contract test. + +- [ ] **Step 8: Commit the process goal** + +```bash +git add scripts/p4-acceptance.sh scripts/test-p4-acceptance.sh \ + backend/scripts/p4-acceptance.mjs backend/scripts/p4-acceptance.test.mjs \ + backend/scripts/fixtures/p4-qdrant-proxy.mjs .gitignore +git commit -m "test: add p4 qdrant lifecycle acceptance" +``` + +- [ ] **Step 9: Confirm the goal is runnable, but defer the one authoritative full run** + +```bash +git status --short +./scripts/test-p4-acceptance.sh +``` + +Expected: clean status and PASS. Do not launch the authoritative Docker/process run yet: Task 10 still adds the manual helper/docs included in the final source boundary. The one retained release run occurs in Task 11 from the final clean implementation/manual-documentation commit. + +--- + +### Task 10: Publish the independent manual walkthrough + +**Files:** +- Modify: `docs/testing/p2-p6-manual-verification.md` (replace only the P4 placeholder section) +- Create: `scripts/p4-manual-verification.sh` +- Create: `scripts/test-p4-manual-verification.sh` +- Modify: `docs/install/local-workspace-registry.md` +- Modify: `docs/install/server-workspace-registry.md` +- Modify: `tools/thothctl/cmd/thothctl/main.go` usage text if not already complete + +- [ ] **Step 1: Write RED manual-helper contract tests** + +The helper accepts only `prepare`, `interrupt-after-delete`, `release-recovery`, `status`, and `cleanup`; owns `.artifacts/manual-acceptance/p4`; refuses reuse without cleanup; emits `GUIDE.md`; uses a new Git remote/Compose project/volumes distinct from automated state; never performs the reviewer commands or records PASS on the reviewer's behalf. + +```bash +./scripts/test-p4-manual-verification.sh +``` + +Expected: FAIL until the helper exists. + +- [ ] **Step 2: Implement prepare/status/fault-control/cleanup only** + +`prepare` builds the fixture and prints exact non-secret variables. `interrupt-after-delete` arms the deterministic fixture barrier but does not invoke rebuild. `release-recovery` releases only that barrier. `cleanup` removes only the manual run's labelled resources/root after explicit reviewer confirmation. No helper calls `workspace collection rebuild/recover` for the reviewer. + +- [ ] **Step 3: Replace the P4 walkthrough placeholder with exact commands** + +Document, in order: + +1. prepare new manual state; +2. `workspace inspect --json` on missing collection; +3. trigger one admission and inspect exact self-healed contract; +4. delete one safe fixture keyword index, trigger admission, and inspect its repair; +5. seed dimension, distance, and index-type incompatibilities and record nonmutation refusal; +6. run workspace/collection confirmation mismatches and prove marker/core/collection unchanged; +7. create a fixture open session and prove `session_inventory_active`; +8. hold the P2 writer lock and prove `preprocessing_conflict` before delete with safe restart/clear; +9. run the exact confirmed rebuild, inspect terminal state, neighbor preservation, healthy core, and inactive maintenance; +10. arm interruption, run rebuild, observe core stopped/maintenance retained/recovery state, release barrier, run exact confirmed recovery, and verify final state; +11. inspect report/state without secrets, decide PASS/FAIL, then cleanup. + +For every step explain the component crossed, expected JSON fields/code, state/artifact produced, and invariant. Include `Decision: **PENDING**` and blank reviewer/date/evidence fields; only the human changes it to PASS/FAIL. + +- [ ] **Step 4: Update operator manuals** + +Explain self-heal vs destructive rebuild, exact confirmations, closed/finalized/archived inventory, lifecycle lock contention with Pi update, maintenance marker semantics, writer-lock refusal, post-delete recovery, intentional vector loss/reindex sequence, and backup recommendation. Do not claim P5/P6 materialization or automated reindexing. + +- [ ] **Step 5: Verify docs/helper** + +```bash +bash -n scripts/p4-manual-verification.sh scripts/test-p4-manual-verification.sh +./scripts/test-p4-manual-verification.sh +./scripts/verify-workspace-install-docs.sh --fixtures-only +``` + +Expected: PASS; the living P4 section contains no placeholder text and remains PENDING. + +- [ ] **Step 6: Commit** + +```bash +git add docs/testing/p2-p6-manual-verification.md \ + docs/install/local-workspace-registry.md docs/install/server-workspace-registry.md \ + scripts/p4-manual-verification.sh scripts/test-p4-manual-verification.sh \ + tools/thothctl/cmd/thothctl/main.go +git commit -m "docs: add p4 qdrant lifecycle walkthrough" +``` + +--- + +### Task 11: Final verification, checkpoint report, and hard stop + +**Files:** +- Create after green evidence exists: `docs/testing/p4-qdrant-bootstrap-rebuild-checkpoint.md` +- Modify: `PROJECT_STATE.md` + +- [ ] **Step 1: Run the scoped P4 verification from a clean commit** + +```bash +git status --short +cd backend && npx vitest run \ + test/qdrant-collection-manager.test.ts \ + test/tht-qdrant-readiness.test.ts \ + test/readiness-manager.test.ts \ + test/routes-sessions.test.ts \ + test/maintenance-gate.test.ts \ + test/pi-process-manager.test.ts \ + test/qdrant-rebuild-state.test.ts \ + test/workspace-preprocessing-state.test.ts \ + test/workspace-preprocessing-service.test.ts \ + test/workspace-maintenance.test.ts +cd backend && npx tsc --noEmit -p . && npm run build +cd tools/thothctl && go test ./... +./scripts/test-default-compose.sh +./scripts/test-unified-compose.sh +./scripts/test-internal-semantic-compose.sh +./scripts/test-compose-secret-policy.sh +./scripts/test-no-deployment-coupling.sh +./scripts/test-deployment-command-contract.sh +./scripts/test-p4-acceptance.sh +./scripts/test-p4-manual-verification.sh +./scripts/verify-workspace-install-docs.sh --fixtures-only +git diff --check +git status --short +``` + +Expected: every command exits 0; TypeScript/build and all Go tests PASS; final tracked status is clean. Do not run or claim the aggregate P2–P6 smoke, full harness/frontend suites, real PSD, or P5/P6 flows—those remain the post-P6 gate. + +- [ ] **Step 2: Run the one authoritative clean-state process and retain it** + +```bash +./scripts/p4-acceptance.sh integration --keep +``` + +Expected: exit 0; `report.json` and `report.md` say every named check PASS, `automated integration: PASS`, `manual acceptance: PENDING`, `secret_scan: PASS`, `cleanup: PASS`, and `retry_count: 0`. If it fails, retain the failed run, diagnose the root cause, add a focused regression, fix and commit, rerun Step 1, then launch a **new run ID from the beginning**—never resume a partial run or retry a mutation inside a run. + +- [ ] **Step 3: Verify the retained process report independently** + +Check the retained P4 run is bound to the clean Step 1 source commit/tree, contains one scenario execution and no automatic retry, has all named checks PASS, secret scan PASS, exact cleanup PASS, a valid artifact SHA-256 manifest, no undeclared regular files, and no owned live Docker resource. Record `report.json` and `report.md` hashes. + +- [ ] **Step 4: Write the checkpoint report** + +`docs/testing/p4-qdrant-bootstrap-rebuild-checkpoint.md` must state: + +- source commit/tree and plan/design/PRD references; +- exact commands and outcomes (do not invent counts); +- retained report path and hashes; +- self-heal, manager compatible-creator race, local route deduplication, distinct-principal route convergence, and incompatibility evidence; +- lifecycle lock, inventory, admissions/Pi quiescence, writer lock, stop/restart evidence; +- successful rebuild and interrupted recovery evidence; +- secret-scan and exact-cleanup evidence; +- `automated integration: PASS` and `manual acceptance: PENDING`; +- explicit P5/P6/post-P6 exclusions and any real-platform checks not run. + +Update only the evolving P4 section of `PROJECT_STATE.md` with the same truthful status. + +- [ ] **Step 5: Commit checkpoint documentation** + +```bash +git add docs/testing/p4-qdrant-bootstrap-rebuild-checkpoint.md PROJECT_STATE.md +git commit -m "docs: record p4 automated checkpoint" +``` + +The retained run is intentionally bound to this checkpoint commit's parent: the final clean implementation and manual-documentation source boundary. The docs-only checkpoint records immutable report hashes and is not falsely claimed as input to its own report. + +- [ ] **Step 6: Complete the persistent process goal and stop** + +Require a clean worktree, then complete the goal only after the authoritative run and cleanup proof. Send the user the checkpoint report and exact P4 manual section. Stop for explicit authorization/manual decision; do not begin P5, P6, aggregate verification, or unrelated cleanup. + +## Manual acceptance completion (later, reviewer-owned) + +After the reviewer executes the independent P4 walkthrough, record only their actual decision and evidence. If PASS, update the P4 checkpoint and `PROJECT_STATE.md` from `manual acceptance: PENDING` to PASS in one docs-only commit. If FAIL, retain the failure evidence, reopen the P4 goal, add a focused regression, and repeat the entire automated process from a new clean run before asking for another manual decision. diff --git a/docs/superpowers/plans/2026-08-10-p5-curated-fk-annotations.md b/docs/superpowers/plans/2026-08-10-p5-curated-fk-annotations.md new file mode 100644 index 00000000..a5a55ee1 --- /dev/null +++ b/docs/superpowers/plans/2026-08-10-p5-curated-fk-annotations.md @@ -0,0 +1,1670 @@ +# P5 Curated FK Annotations in Git Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** Implement PRD D5 so the only curated FK source is the bounded, validated Git blob at `workspace-content//schema/annotations.yaml`, materialized for one exact workspace commit and accepted through an explicit, auditable P2-run revision transition. + +**Architecture:** Extend the P1 registry plumbing with a fixed-argv Git blob reader built on one shared bounded-child lifecycle and one shared `AnnotationSynchronizer`; the synchronizer asks the harness's strict parser to validate bytes, then publishes exact bytes and a hash-bound manifest through one dirfd/openat-style protected-filesystem primitive under P3's immutable revision root. The synchronizer has an explicit prepare lifecycle for repository pull and pre-READY acceptance, resolved only with P3's `futureWorkspaceLayoutPaths`, and a runtime lifecycle for active/pinned consumption that requires the exact committed READY state before calling the four-argument `workspaceRuntimePaths`. Extend P2's `PreprocessingStateStore` and `WorkspacePreprocessingService.execute` rather than creating another workflow: one legal V1→V2 checkpoint migration preserves audit facts but requires explicit acceptance, a bounded ambiguity-safe `resolveRun` owns workspace discovery, and `workspace schema export-fks --run` exports the paused run's exact candidate. After the curator pushes, P5 registers an exact P2 `CapabilityAwareRegistryPublicationParticipant` with P2's `RegistryAddressedRequestV1` and invokes only `WorkspaceRegistry.publishAddressed`—never the writer-first common migration lifecycle. That released `registry_pull` exception alone has RW/Git capability: `repository.lock` pins `RegistryPullAddressedPlanV1`; `WorkspaceRegistry` enters the sole `runUnderOrderedWorkspaceWriterLocks` callback for the complete `changedWorkspaceIds` in strict lexical order and exposes one `OrderedWorkspaceWriterCapabilitySet`; the participant receives each exact `AddressedWorkspacePublicationLeaseV1`. The set remains held through the single active-snapshot-pointer write, file fsync, atomic rename, parent fsync, target-byte verification, and `terminal_durable`; ordered callback settlement invalidates and closes the set in reverse lexical order, and only then is `repository.lock` released. Exact same-ID resume uses `RegistryPullJobStateV1`: at `publication_intent_durable` an all-base state resumes without a fetch, while an all-target state after rename+fsync reconciles lost acknowledgement without refetch or a second publication; third identity or target drift refuses. The curator must then run P3's exact non-activating `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json` operation: its fixed `p3_migrate_dwh_cache`/`p3_prepare_dwh_cache` plus `p3_materialize_dwh_snapshot` stages select the pulled commit and effective binding, publish and strictly reverify that revision's binding-qualified physical/LSH snapshot without reading or publishing READY, and never fall back to the prior revision's snapshot. Only after that preparation does `schema accept` consume the already-active immutable annotation and DWH snapshots under the writer lock; it never pulls, takes the repository lock, or receives Git capability. Admission and preprocessing resume remain `migration_required` after pull, after DWH snapshot preparation, and after acceptance. P3's exact `workspace migrate activate-revision-layout` operation must then publish and verify the accepted commit+binding READY before either can succeed. + +**Tech Stack:** TypeScript 5, Node.js 22, Git plumbing (`git ls-tree`, `git cat-file` with fixed argv), Python 3.12, Pydantic 2, PyYAML, Typer, Go, Docker Compose, Vitest, pytest, Ruff, Go test, Bash. + +--- + +## Scope, prerequisites, and non-negotiable contracts + +**Source:** `docs/prd/2026-08-09-workspace-preprocessing-prd.md` D5/RF3 and `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` §7. The reviewed planning baseline was commit `16ee92c8f980b363a82cb0e2777c354ed25519fa` (tree `ced13dcedff78dfba40313608f97946f716d2e11`); execution must record its own clean source commit and tree in the retained report. + +Do not begin Task 1 until the P1, P2, P3, and P4 checkpoints are accepted and the worktree is clean. P5 technically builds on P1 plus the P2/P3 operator/state/path work; P4 is an independent delivery checkpoint but precedes P5 in the P2–P6 release sequence. First run: + +```bash +git status --short +git rev-parse HEAD +git rev-parse HEAD^{tree} +test -f docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md +test -f docs/superpowers/plans/2026-08-10-p3-effective-config-fingerprint.md +test -f docs/superpowers/plans/2026-08-10-p4-qdrant-bootstrap-rebuild.md +test -f backend/src/workspaces/registry-pull-job.ts +test -f backend/test/registry-pull-job-imports.compile.ts +``` + +Expected: empty `git status --short`; three plan files plus P3's exact job export/compile-import files exist; `PROJECT_STATE.md` records accepted P1–P4 checkpoints. This plan is already rebased on the frozen P2/P3 handoff: `backend/src/workspace-maintenance.ts::main`, `WorkspacePreprocessingService.execute`, `PreprocessingStateStore`, `WorkspaceRuntimeConfigLeaseFactory`, `backend/src/workspaces/revision-layout.ts::{WorkspaceLayout,futureWorkspaceLayoutPaths,workspaceRuntimePaths,readRevisionLayoutState}`, `runUnderWorkspaceWriterLock` / `probeWorkspaceWriterLock`, P2 `RegistryPullAddressedPlanV1`, `AddressedWorkspacePublicationLeaseV1`, `OrderedWorkspaceWriterCapabilitySet`, `CapabilityAwareRegistryPublicationLifecycleOwner`, `CapabilityAwareRegistryPublicationParticipant`, `RegistryAddressedRequestV1`, `RegistryPullAddressedResultV1`, and `WorkspaceRegistry.publishAddressed`, plus `tools/thothctl/internal/workspaceops::{ParseWorkspaceCommand,Run}`. It also consumes P3's released `thothctl --installation workspace registry pull --workspace [--resume <32-hex-outer-run-id>] [--json]`, closed Go `RegistryPullCommand`, literal operation/result/Compose capability label `registry_pull`, and exact `RegistryPullPublicJobRequestV1`, `RegistryPullPhaseV1`, `RegistryPullAddressedJobRequestV1`, `RegistryPullParticipantStateV1`, `RegistryPullSynchronizerStateV1`, and `RegistryPullJobStateV1`; the existing read-only active immutable-snapshot API `WorkspaceRegistry.read(id)`; and released `thothctl --installation workspace migrate dwh-cache --workspace [--resume <32-hex-outer-run-id>] [--json]` / `migrate_dwh_cache` operation. `registry_pull` is the explicit exception to P3's writer-first common migration lifecycle. Its exact repository-first order and fields are quoted below: the complete lexical capability set stays owned through active-pointer rename+parent fsync, byte verification, and terminal durability; reverse set release precedes repository release. Resume reads only the recorded `RegistryPullJobStateV1`, never refetches or reselects a target, and accepts only exact all-base/all-target/mixed recorded identities while refusing third identity or target drift. The DWH operation's exact `p3_migrate_dwh_cache` or `p3_prepare_dwh_cache` cache stage is followed by `p3_materialize_dwh_snapshot` for the selected unready commit+binding and must publish and strictly reverify its revision-qualified physical/LSH snapshot without READY. Only `registry_pull` receives writable registry storage plus the selected validated HTTPS/SSH Git transport files; DWH migration and ordinary maintenance mutations remain registry RO with no Git credentials. Do not create duplicate operator, state-store, lock, renderer, `internal/workspace` package, path abstraction, or second pull implementation. + +### Exact released P2/P3 addressed registry-pull exception + +P5 must import and consume the released names verbatim. The six P3 job names have one owning module; +production consumers import them only as follows (tests use the corresponding +`../src/workspaces/registry-pull-job.js` path): + +```ts +import type { + RegistryPullPublicJobRequestV1, + RegistryPullPhaseV1, + RegistryPullAddressedJobRequestV1, + RegistryPullParticipantStateV1, + RegistryPullSynchronizerStateV1, + RegistryPullJobStateV1, +} from "./registry-pull-job.js"; +``` + +`backend/src/workspaces/registry-pull-job.ts` exports all six names. It defines +`RegistryPullPhaseV1 = RegistryAddressedPublicationPhaseV1`, +`RegistryPullAddressedJobRequestV1 = Extract`, and +`RegistryPullJobStateV1 = RegistryPullAddressedPublicationStateV1`; these are exact aliases owned by +P3, not names P5 may redeclare. `RegistryPullPublicJobRequestV1`, +`RegistryPullParticipantStateV1`, and `RegistryPullSynchronizerStateV1` are the other three exact P3 +exports. Add a compile-only import in P5 and preserve P3's bidirectional parity assertions. No barrel, +compatibility export, local alias, or second job shape is permitted. Before P5 begins, P3's production +`registry.publishAddressed(...)` call must compile against this exact surface: the validated boundary +passes all three identities on create/resume and the expected base on create. If that owning P3 call or +its mismatch-before-mutation tests are absent, stop and reopen P3; P5 must not patch it or add an overload. + +P5 consumes these exact P2 exports from `backend/src/workspaces/registry-publication.ts` and +`backend/src/workspaces/preprocessing-state.ts`. `AddressedWorkspacePublicationLeaseV1` is owned by +`registry-publication.ts`, and its exact `readers` field type +`BorrowedWorkspaceSessionReadersExclusiveLockLease` is owned by `preprocessing-state.ts`: + +```ts +export type RegistryAddressedPublicationPhaseV1 = + | "request_claimed" | "target_advertised" | "target_fetched" | "planned" + | "participants_prepared" | "publication_intent_durable" + | "target_published" | "terminal_durable"; +interface RegistryAddressedPlanFieldsV1 { + readonly schemaVersion: 1; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + readonly jobArtifactPath: RegistryAddressedJobArtifactPathV1; + readonly advertisedTargetCommit: Revision40; + readonly immutableTargetRef: `refs/thoth/addressed-runs/${RegistryRunId32}/target`; + readonly fetchedTargetCommit: Revision40; + readonly targetCommit: Revision40; + readonly targetManifestSha256: Sha256Hex; + readonly targetWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[]; + readonly changedWorkspaceIds: readonly CanonicalWorkspaceId[]; + readonly changedSetSha256: Sha256Hex; +} +export interface RegistryPullAddressedPlanV1 extends RegistryAddressedPlanFieldsV1 { + readonly operation: "registry_pull"; + readonly changedSetRule: "symmetric_base_target_workspace_difference"; + readonly baseCommit: Revision40; + readonly baseManifestSha256: Sha256Hex; + readonly baseWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[]; +} +export type RegistryAddressedPlanV1 = + | RegistryBootstrapAddressedPlanV1 | RegistryPullAddressedPlanV1; +interface RegistryAddressedPublicationStateFieldsV1 { + readonly schemaVersion: 1; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly jobArtifactPath: RegistryAddressedJobArtifactPathV1; + readonly phase: RegistryAddressedPublicationPhaseV1; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + readonly advertisedTargetCommit: Revision40 | null; + readonly immutableTargetRef: `refs/thoth/addressed-runs/${RegistryRunId32}/target` | null; + readonly fetchedTargetCommit: Revision40 | null; + readonly targetCommit: Revision40 | null; + readonly targetManifestSha256: Sha256Hex | null; + readonly targetWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[] | null; + readonly changedWorkspaceIds: readonly CanonicalWorkspaceId[] | null; + readonly planSha256: Sha256Hex | null; + readonly changedSetSha256: Sha256Hex | null; + readonly participantsSha256: Sha256Hex | null; + readonly synchronizersSha256: Sha256Hex | null; + readonly publicationIntentSha256: Sha256Hex | null; + readonly publishedActiveStateSha256: Sha256Hex | null; + readonly terminalResultSha256: Sha256Hex | null; + readonly priorStateSha256: Sha256Hex | null; +} +export interface RegistryPullAddressedPublicationStateV1 + extends RegistryAddressedPublicationStateFieldsV1 { + readonly operation: "registry_pull"; + readonly baseCommit: Revision40; + readonly baseManifestSha256: Sha256Hex; + readonly baseWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[]; + readonly changedSetRule: "symmetric_base_target_workspace_difference" | null; +} +export interface AddressedWorkspacePublicationLeaseV1 + extends BorrowedOrderedWorkspaceWriterLeaseV1 { + readonly quiescence: BorrowedWorkspaceMaintenanceQuiescenceLease; + readonly readers: BorrowedWorkspaceSessionReadersExclusiveLockLease; +} +export interface CapabilityAwareRegistryPublicationParticipant { + readonly participantId: string; + prepare(plan: RegistryAddressedPlanV1, + workspace: AddressedWorkspacePublicationLeaseV1): Promise; + reconcile(plan: RegistryAddressedPlanV1, + workspace: AddressedWorkspacePublicationLeaseV1, prepared: T, + phase: RegistryAddressedPublicationPhaseV1): Promise; +} +export class CapabilityAwareRegistryPublicationLifecycleOwner { + run(input: { + readonly plan: RegistryAddressedPlanV1; + readonly capabilities: OrderedWorkspaceWriterCapabilitySet; + readonly participants: readonly CapabilityAwareRegistryPublicationParticipant[]; + readonly synchronizers: readonly CapabilityAwareRegistryPublicationSynchronizer[]; + readonly action: () => Promise; + }): Promise; +} +export type RegistryAddressedRequestV1 = + | { + readonly mode: "create"; + readonly operation: "registry_bootstrap"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly expectedBaseCommit: null; + readonly remoteRefIdentitySha256: Sha256Hex; + } + | { + readonly mode: "resume"; + readonly operation: "registry_bootstrap"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + } + | { + readonly mode: "create"; + readonly operation: "registry_pull"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly expectedBaseCommit: Revision40; + readonly remoteRefIdentitySha256: Sha256Hex; + } + | { + readonly mode: "resume"; + readonly operation: "registry_pull"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + }; +export class WorkspaceRegistry { + publishAddressed(request: RegistryAddressedRequestV1): Promise; +} +``` + +This is a field-for-field dependency quotation, not a compatibility sketch. The validated +installation/registry boundary derives the production `installationIdentitySha256`, +`repositoryIdentitySha256`, and `remoteRefIdentitySha256` and supplies all three on every create and +resume member; pull create additionally supplies the exact `expectedBaseCommit`. The same three +identities are required, non-optional fields in the immutable plan and durable state, and same-ID +resume revalidates them before network or state mutation. `P5` adds no optional identity, partial +request, constructor decoration, compatibility overload, or second `publishAddressed` signature. + +The production registry constructor, not the request, owns the single +`VerifiedWorkspaceLockRootLeaseFactory`, `CapabilityAwareRegistryPublicationLifecycleOwner`, +participant list, and synchronizer list. The sole pull order is repository lock → durable +`request_claimed` → durable target advertisement → exact-OID fetch to the create-only immutable ref → +`target_fetched` → immutable pull plan → `planned` → acquire/provision every complete changed root → +one `runUnderOrderedWorkspaceWriterLocks` callback → lexical quiescence/readers → participant prepare +→ `participants_prepared` → durable publication intent → active-state publication/reconciliation → +`target_published` → `terminal_durable` → callback settlement invalidates and reverse-closes +readers/quiescence/writers/roots → repository lock release. The artifact is exactly +`addressed-publication-jobs/.json`. + +P5 calls only `WorkspaceRegistry.publishAddressed` with the `registry_pull` member. Its participant is +called by the existing lifecycle owner with the same callback-scoped +`AddressedWorkspacePublicationLeaseV1`; it consumes that lease's opaque `rootLease` and +`writerCapability`, never reacquires either, and never calls a repository method. The complete ordered +capability set stays live through active-pointer sibling write, file fsync, rename, parent fsync, +byte verification, and terminal durability. Same-ID recovery uses only +`RegistryPullJobStateV1`: it never refetches/reselects after the durable target pin, accepts only the +recorded all-base/all-target/mixed identities, reconciles a lost publication acknowledgement without a +second rename, and refuses a third identity or target drift. A new process may reacquire the recorded +complete set once; no live attempt permits nested acquisition or participant reentry. + +### Canonical content and runtime contract + +```text +Git path (fixed): +workspace-content//schema/annotations.yaml + +Runtime bytes (exact Git blob or canonical absent value): +/data/sessions//revisions//artifacts/mschema/annotations.yaml + +Adjacent publication record: +/data/sessions//revisions//artifacts/mschema/annotations.manifest.json +``` + +Use these constants and shapes: + +```ts +export const MAX_FK_CANDIDATE_BYTES = 716_800; // exact P2 decoded candidate/export maximum +export const MAX_CURATED_ANNOTATION_BYTES = 16 * 1024 * 1024; // Git blob only +export const EMPTY_ANNOTATIONS = Buffer.from("tables: {}\n", "utf8"); + +export interface AnnotationGitObject { + source: "git" | "absent"; + path: string; + mode: "100644" | "100755" | null; + blobId: string | null; // exact 40-hex object ID when present + bytes: Buffer; // exact blob bytes; absent uses EMPTY_ANNOTATIONS + sha256: string; + warning?: "annotation_missing"; +} + +export interface AnnotationOwnershipManifest { + schema_version: 1; + kind: "workspace_fk_annotations"; + workspace_id: string; + workspace_revision: string; // exact 40-hex commit + source_path: string; + source: "git" | "absent"; + blob_id: string | null; + content_sha256: string; + byte_length: number; + destination: string; // exact absolute annotations.yaml destination +} +``` + +The manifest JSON is canonical single-line JSON plus `\n`, mode `0400`; the annotation is mode `0400`; created directories are `0700`. A present object may use Git regular-file mode `100644` or `100755`; the synchronized result is never executable. Reject mode `120000` (symlink), `160000` (gitlink), `040000` (tree), other types/modes, a curated Git blob greater than exactly 16 MiB, malformed UTF-8, NUL, duplicate YAML keys, multiple documents, aliases/merge keys, unknown fields, and invalid annotation models. Missing path is compatible: validate and publish `tables: {}\n`, set `source: "absent"`, `blob_id: null`, and return a safe warning. A present zero-byte/empty YAML document is `annotation_invalid`; absence and an empty file are not equivalent. + +The two byte limits are intentionally different and must never share a constant or fallback. Every P2/P5 state-owned suggested candidate and the decoded `schema-export-fks` payload is at most exactly 716,800 bytes. Its base64 `hostExport` plus the complete machine envelope remains at most exactly 1 MiB. Only the separately curated Git annotation blob may be as large as exactly 16 MiB; it is read from Git after the author edits/commits it and never traverses `hostExport`. Tests must pass at 716,800/16 MiB and fail before buffering/publication at 716,801/16 MiB + 1 respectively. + +`physical.yaml` remains local DWH-derived content. Never add it to Git, an author export, a registry bundle, or an annotation manifest. + +### Shared fixed-Git child contract + +P5 introduces the single reusable lifecycle boundary `backend/src/workspaces/fixed-git-child.ts::runFixedGitChild`; P6 must extend/reuse it rather than adding another subprocess wrapper. Repository methods supply only reviewed, fixed Git argv vectors and one caller-owned monotonic absolute deadline shared across every stage of the logical operation. The helper starts Git shell-free in a new owned process group, writes only the exact declared stdin (empty for P5 plumbing) and closes it, streams rather than `execFile`-buffers, enforces both a per-stage stdout byte bound and parser record bound plus a small fixed stderr bound, and returns only parsed allowlisted values. + +On deadline, caller cancellation, parser rejection, early/abnormal exit, stdout/record overflow, or stderr overflow, stop parsing and close stdin, send TERM only to the owned process group, wait the fixed grace, send KILL only to that group if necessary, and always await child exit plus stream settlement before returning. No lock-sensitive caller may continue while a child or descendant is alive. Raw stdout, stderr, argv-derived paths, repository endpoints, and child exceptions never enter public errors or logs. Real-process tests cover a hang, stdout and record floods, stderr flood, early exit, parser abort, cancellation, a descendant holding a pipe open, TERM→KILL escalation, and proof that exit/stdio cleanup was awaited. P5's `ls-tree`, `cat-file -s`, and `cat-file blob` stages share one operation deadline; their stdout/record bounds are respectively the exact bounded control record, exact bounded size record, and advertised blob length no greater than `MAX_CURATED_ANNOTATION_BYTES`. + +### Controlled acceptance contract + +P2's same-revision manual-review state is explicitly migrated, not silently reinterpreted. P2 stores state in `/data/sessions//preprocessing/jobs/.json`, candidates under `fk-candidates/.yaml`, reviews under `fk-reviews/.json`, all guarded by the persistent regular `0600` advisory lock `/data/sessions//preprocessing/writer.lock`. P5 keeps those exact `PreprocessingStateStore` paths and freezes one V2 checkpoint shape: + +```ts +interface FkReviewCheckpointV2Base { + kind: "fk_review"; + candidateDigest: `sha256:${string}`; + candidateCount: number; + baseAnnotationsDigest: `sha256:${string}`; + reviewedDigest?: `sha256:${string}`; // copied P2 audit fact only; never canonical/accepted + baseWorkspaceRevision: string; + baseGitBlob: string | null; +} +interface PendingFkReviewCheckpointV2 extends FkReviewCheckpointV2Base { + status: "pending"; +} +interface AcceptedFkReviewCheckpointV2 extends FkReviewCheckpointV2Base { + status: "accepted"; + acceptedWorkspaceRevision: string; + acceptedGitBlob: string; // exact present 40-hex Git blob + acceptedCandidateDigest: `sha256:${string}`; + acceptedBlobDigest: `sha256:${string}`; + acceptedEffectiveDwhBinding: string; + acceptedAt: string; +} +type FkReviewCheckpointV2 = PendingFkReviewCheckpointV2 | AcceptedFkReviewCheckpointV2; +``` + +There is no V2 `required` or `reviewed` status. `manual_review_required` remains the public P2 operation result/error code, not an on-disk checkpoint status. The only normal state edge is `pending → accepted`; an exact replay of the identical complete acceptance object is idempotent, while any other `accepted → *`, rewind, skip, or partial acceptance is invalid. + +The V1 source in this table means the exact canonical JSON bytes emitted by the completed P2 `PreprocessingStateStore` and `FkReviewRecordV1`, not a structurally plausible handwritten object. At Task 5 start, run the unmodified P2 writer to commit two byte fixtures—paused without a review and paused with a valid same-revision review—under `backend/test/fixtures/preprocessing-v1-fk-review/`, record their SHA-256 and candidate bytes in `README.md`, and prove the P2 reader accepts them before adding a V2 reader. Any P2 V1 field/version drift is a prerequisite failure to repair at P2, not a permissive P5 parser change. + +Freeze this migration/transition table before code changes: + +| Exact source bytes/state | Required validation | V2 result / behavior | +|---|---|---| +| P2 V1 nonterminal job at `manual_review_required`, state-owned candidate present, no `FkReviewRecordV1` | strict V1 keys/version/identity; exact regular no-follow candidate; rehash equals recorded digest/count; exact base revision is still available; synchronize/verify that revision's Git annotation identity | atomically write V2 `pending`; copy candidate fields and verified base revision/blob/content digest; no `reviewedDigest` | +| Same paused P2 V1 job plus a valid same-revision `FkReviewRecordV1`, with no later stage recorded | all checks above; review candidate/workspace/revision/digest exactly match the V1 job | atomically write V2 `pending`; copy the V1 reviewed annotation digest to optional `reviewedDigest` for audit only; still require explicit Git acceptance | +| V2 `pending` plus a complete verified acceptance object | candidate rehash matches; repository→writer protocol below; new active revision, present blob, checked bytes, unchanged effective-DWH binding | atomically write V2 `accepted` with all accepted fields | +| V2 `accepted` plus byte-identical complete acceptance object | revalidate the stored object, candidate, active identity, synchronized blob, and binding | return the existing `accepted` value without rewriting | +| P2 V1 run that already recorded any stage after FK review or is terminal | strict V1 validation only | preserve immutable legacy audit state; it is not exportable/acceptable/resumable under P5; return `preprocessing_resume_mismatch` and require a new run | +| Unknown V1/V2 keys/version/status, missing or mismatched candidate/review, partial accepted fields, identity conflict, or any unlisted source/edge | none may be repaired heuristically | fail closed before publication or later-stage work | + +The migration service supplies the verified base Git identity; the state store never invents a blob ID from P2 bytes. Migration and transition use `PreprocessingStateStore`'s closed transition union and atomic writer, never a partial merge. + +`workspace schema accept --run --yes` is the only human-decision transition. Because the public grammar intentionally omits a workspace, `WorkspacePreprocessingService.execute` first calls the bounded `resolveRun(runId)` contract: validate the exact 32-hex run ID; enumerate only IDs from the strict active workspace set (bounded by the existing registry limit, never recursively walk `/data/sessions`); for each ID open only `//preprocessing/jobs/.json` as an exact regular no-follow file through P2 `PreprocessingStateStore`'s installation-root-relative no-follow reader; bound every file and the whole operation; verify embedded run/workspace identity; and require exactly one match. Zero matches returns `preprocessing_run_not_found`; two or more returns `preprocessing_run_ambiguous`; malformed IDs, symlink/hardlink/embedded-identity mismatch, unsafe state, or traversal input returns `preprocessing_state_invalid`; a run whose former workspace is not in the active set is deliberately indistinguishable from not found. These three codes are stable P5 public codes with no searched paths or workspace list. The same resolver is used by exact candidate export. After the later writer lock is acquired, the service reopens and revalidates the exact resolved state so a stale locator result is never trusted. + +The curator must run P3's released `thothctl --installation /thothii-installation.yaml workspace registry pull --workspace --json` after the curated commit is pushed and before acceptance (`--resume <32-hex-outer-run-id>` is only that exact interrupted pull's continuation). That distinct `RegistryPullCommand` / `registry_pull` operation alone receives writable registry storage plus the selected validated HTTPS/SSH Git transport files and invokes P2 `WorkspaceRegistry.publishAddressed(request: RegistryAddressedRequestV1)` rather than the common writer-first migration action. Inside the validated installation/registry boundary, create and resume both supply the exact production `installationIdentitySha256`, `repositoryIdentitySha256`, and `remoteRefIdentitySha256`; create also supplies the exact active `expectedBaseCommit`. While holding `repository.lock`, create fetches once, pins all three identities in the exact `RegistryPullAddressedPlanV1`, and persists them in `RegistryPullJobStateV1` at `planned`. `WorkspaceRegistry` enters the sole `runUnderOrderedWorkspaceWriterLocks` callback for the complete strict-lexical `changedWorkspaceIds`, and P5's exact `CapabilityAwareRegistryPublicationParticipant` receives each matching `AddressedWorkspacePublicationLeaseV1`. It prepares target manifests (or a closed base-only removal result) before `publication_intent_durable`; it never calls `runUnderWorkspaceWriterLock`, acquires roots/readers, invokes `publishAddressed`, or calls repository. The `OrderedWorkspaceWriterCapabilitySet` stays live through the single active-snapshot-pointer write+file fsync+atomic rename+parent fsync and synchronous target-byte verification, then through `target_published` and `terminal_durable`; only then does ordered callback settlement invalidate and reverse-close readers/quiescence/capabilities/root leases, followed by `repository.lock` release. Same-ID resume loads only the persisted addressed state: all-base before publication continues without fetch, all-target after rename+fsync advances lost acknowledgement without refetch or a second publication, and mixed exact base/target converges to target while the reacquired full recorded set is held. Any third active identity, changed pinned target object/inventory/digest, installation/repository/remote-ref identity drift, or moved local remote-tracking OID returns `preprocessing_resume_mismatch` before network or state mutation and preserves nonterminal ownership. This is the sole supported pull path. + +Immediately after pull and still before acceptance, the curator must run the exact released command `thothctl --installation /thothii-installation.yaml workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json`, where `WORKSPACE_ID` is the resolved workspace ID (`--resume <32-hex-outer-run-id>` is only that interrupted `migrate_dwh_cache` run's continuation). This is a non-activating P3 maintenance transition: the fixed `p3_migrate_dwh_cache`/`p3_prepare_dwh_cache` cache stage plus `p3_materialize_dwh_snapshot` must bind to the pulled revision and probed effective binding, reuse or prepare the immutable cache, publish the exact `revisions//dwh-snapshots//{artifacts,indexes}` physical/LSH snapshot, and strictly reverify its identities/digests without requiring or publishing `READY.json`. It must never read the prior commit's revision snapshot as a fallback. After the dedicated job records terminal durability, performs owner-only clear, and releases its exclusive reader gate, session admission and preprocessing resume still return `migration_required`; that remains true through acceptance and changes only after activation publishes READY. Every migration and ordinary mutation—including `schema-accept`—keeps P2's registry RO mount and receives no Git transport. + +`schema accept` itself never pulls, spawns Git, calls a repository method, or takes the repository lock. After bounded `resolveRun`, it reads the already-active immutable snapshot and prepared annotation identity, acquires the existing root through P2's installation-bound factory, and enters a one-element `runUnderOrderedWorkspaceWriterLocks` callback. Inside `forWorkspace`, it receives the callback-scoped `rootLease` plus `writerCapability`, rereads `active.json`, reopens and revalidates the resolved paused state/candidate, and calls only `AnnotationSynchronizer.verifyPrepared(..., { kind: "prepare", rootLease, writerCapability })`. The verifier never acquires a lock or spawns directly: its exact `annotation_verify` request goes through `writerCapability.spawnChild`, and it reads only the already-materialized revision selected by `futureWorkspaceLayoutPaths(rootLease, workspaceId, workspaceRevision)`. Because P3 deliberately leaves a newly pulled layout-v1 revision without `READY.json`, accept is a maintenance-only check using P3's trusted `layoutIntent: "prepare-revision-layout-v1"` future resolver; it never calls `readRevisionLayoutState`/`workspaceRuntimePaths`, admits a session, or publishes READY. It runs `schema check` against the already-published and strictly reverified P3 physical/LSH snapshot for that exact future revision+binding (never a prior-revision fallback), compares and reports candidate/current digests (they may differ after editing), requires a new revision and present blob, requires the paused `canonical_effective_dwh_binding(cfg)`, and atomically publishes V2 `accepted`. A racing pull may hold repository but must block on writer before changing `active.json`; acceptance never requests repository and therefore sees one stable identity without deadlock. The exact `CapabilityAwareRegistryPublicationParticipant` retains the released repository-first order without acquiring locks itself; P2/P3 holds the complete `OrderedWorkspaceWriterCapabilitySet` through active-pointer rename+parent fsync before reverse release, and tests structurally reject every writer→repository path. + +After acceptance, admission and the explicit preprocessing run still fail `migration_required`. Run P3's released `thothctl --installation workspace migrate activate-revision-layout --workspace --yes [--resume <32-hex-outer-run-id>] [--json]` to reverify the prepared snapshot/materializations and publish the accepted revision's exact `READY.json`; only then may the run resume on its human-accepted A→B transition. The P5 acceptance record is the only authority for that preprocessing-run rebind. Ordinary sessions remain pinned to their recorded revision and are never silently retargeted. + +An explicit `--resume ` fails with `preprocessing_resume_mismatch` if the accepted commit/blob/digest or effective DWH binding no longer matches. An unqualified new run may start with the newer revision. `schema check`, an empty file, digest equality, Git publication, or the legacy P2 review record alone never records human acceptance; no silent rebinding is allowed. + +### Hard boundaries + +- Do not publish or push curated bytes from the application checkout or maintenance container. Curators use an ordinary Git author clone. +- Do not broaden P1 HTTP workspace ZIP import/export to carry `workspace-content`; P5's “export/import” means candidate export to an author clone, ordinary Git commit/push, registry pull, pre-READY DWH snapshot preparation, then acceptance/import from the exact Git blob. +- Do not create a mutable `current` symlink, copy annotations into a workspace-global artifact root, or rewrite an old revision directory. +- Do not implement Evidence enumeration/materialization, nested Evidence limits, Evidence retention, Qdrant Evidence writes, or P6 cleanup. +- Do not change DWH fingerprint semantics, memory migration, or collection lifecycle; consume P3/P4 APIs. +- Do not add a GUI or backend preprocessing HTTP endpoint. +- Do not broaden maintenance capability to make acceptance work: only P3 `registry_pull` is registry RW/Git; released `migrate_dwh_cache`, `schema-accept`, export, resume, and every other ordinary mutation remain registry RO/no-Git. The DWH migration's existing bounded DWH capability does not grant Git or registry mutation. +- Do not route `registry_pull` through P3's common writer-first migration lifecycle, enter repository while any writer is already held, or let `AnnotationSynchronizer.ensure` acquire/reacquire a writer. Invoke only `WorkspaceRegistry.publishAddressed(RegistryAddressedRequestV1)` with the exact `CapabilityAwareRegistryPublicationLifecycleOwner`, `OrderedWorkspaceWriterCapabilitySet`, and `CapabilityAwareRegistryPublicationParticipant`; keep the set held through active-pointer rename+parent fsync and terminal durability, reverse-release it, then release repository. + +## Target file map + +**Harness parser** +- Modify: `harness/tht/mschema/models.py` +- Modify: `harness/tht/cli/schema_cmd.py` +- Modify: `harness/tests/test_schema_fk_annotations.py` +- Create: `harness/tests/test_annotation_validation_cli.py` +- Create: `harness/tht/protected_fs_helper.py` (internal fixed-protocol Linux `openat`/`renameat` helper; not a public `tht` command) +- Create: `harness/tests/test_protected_fs_helper.py` +- Modify: `harness/tht/locked_child_stdin.py` +- Modify: `harness/tests/test_locked_child_stdin.py` + +**Git and materialization** +- Create: `backend/src/workspaces/fixed-git-child.ts` +- Create: `backend/test/fixed-git-child.test.ts` +- Modify: `backend/src/workspaces/git-repository.ts` +- Modify: `backend/src/workspaces/types.ts` +- Create: `backend/src/workspaces/annotations.ts` +- Create: `backend/src/workspaces/protected-workspace-fs.ts` (capability-only adapter over P2 `rootLease` + `writerCapability.spawnChild`; no path/direct-spawn fallback) +- Create: `backend/src/workspaces/registry-factory.ts` +- Modify: `backend/src/workspaces/registry.ts` +- Modify: `backend/src/app.ts` +- Modify: `backend/test/workspaces-git-repository.test.ts` +- Create: `backend/test/workspace-annotations.test.ts` +- Create: `backend/test/protected-workspace-fs.test.ts` +- Create: `backend/test/workspace-registry-factory.test.ts` +- Modify: `backend/test/workspace-registry.test.ts` +- Modify: `backend/test/workspace-runtime-handoff.test.ts` + +**P2/P3 operator and runtime integration (these files must already exist)** +- Modify: `backend/src/workspaces/preprocessing-state.ts` (sole locked-child union/dispatcher extension, strict V1→V2 migration, bounded `resolveRun` support; no second filesystem/spawn path) +- Modify: `backend/src/workspaces/preprocessing-service.ts` +- Modify: `backend/src/workspace-maintenance.ts` +- Modify: `backend/src/workspaces/runtime-config-lease.ts` +- Read/consume unchanged: `backend/src/workspaces/revision-layout.ts` +- Modify: `backend/test/workspace-preprocessing-state.test.ts` +- Modify: `backend/test/workspace-preprocessing-service.test.ts` +- Modify: `backend/test/workspace-maintenance.test.ts` +- Modify: `backend/test/workspace-runtime-config-lease.test.ts` +- Modify: `backend/test/workspace-revision-layout.test.ts` +- Modify: `backend/test/workspace-runtime-renderer.test.ts` +- Modify: `backend/test/routes-sessions.test.ts` + +**Host CLI** +- Modify: `tools/thothctl/internal/workspaceops/operations.go` +- Modify: `tools/thothctl/internal/workspaceops/operations_test.go` +- Read/verify unchanged from repaired P3: `tools/thothctl/internal/config/installation.go` and tests (operation-specific registry/Git bindings) +- Read/verify unchanged from repaired P3: `compose.yaml`, `deploy/compose.git-{https,ssh}.yaml`, `scripts/generate-connector-secrets-override.sh`, and their Compose/secret-policy tests (`registry_pull` RW/Git; ordinary mutations RO/no-Git) +- Modify: `tools/thothctl/internal/safeio/files.go` +- Modify: `tools/thothctl/internal/safeio/files_unix_test.go` +- Modify: `tools/thothctl/internal/safeio/files_windows_test.go` +- Modify: `tools/thothctl/cmd/thothctl/main.go` +- Modify: `tools/thothctl/cmd/thothctl/main_test.go` + +**Process goal, docs, and checkpoint** +- Create: `backend/scripts/p5-acceptance.mjs` +- Create: `backend/scripts/p5-acceptance.test.mjs` +- Create: `scripts/p5-acceptance.sh` +- Create: `scripts/test-p5-acceptance.sh` +- Create: `backend/scripts/p5-manual-verification.mjs` +- Create: `backend/scripts/p5-manual-verification.test.mjs` +- Create: `scripts/p5-manual-verification.sh` +- Modify: `docs/testing/p2-p6-manual-verification.md` +- Create: `docs/contracts/workspace-annotations.md` +- Modify: `docs/install/local-workspace-registry.md` +- Modify: `docs/install/server-workspace-registry.md` +- Modify: `PROJECT_STATE.md` + +--- + +### Task 1: Make the harness annotation parser a strict reusable validation boundary + +**Files:** +- Modify: `harness/tht/mschema/models.py` +- Modify: `harness/tht/cli/schema_cmd.py` +- Modify: `harness/tests/test_schema_fk_annotations.py` +- Create: `harness/tests/test_annotation_validation_cli.py` + +- [ ] **Step 1 (RED): write parser tests before production code.** + +Add `parse_annotations_yaml(source: bytes) -> Annotations` tests for valid Italian UTF-8, `tables: {}`, all existing FK fields, and deterministic counts. Add one parameterized rejection test for invalid UTF-8, NUL, zero bytes, whitespace-only content, duplicate keys, multiple YAML documents, anchors/aliases, merge keys, unknown root/table/column/FK keys, non-mapping root, mismatched/empty FK column lists, and a 16 MiB + 1 input. Preserve `Annotations.from_yaml(path)` missing-path compatibility, but make an existing malformed file fail. + +Add CLI tests using `typer.testing.CliRunner`: + +```python +def test_validate_annotations_reads_stdin_and_emits_pristine_json(): + result = runner.invoke(app, ["schema", "validate-annotations", "--stdin", "--json"], + input="tables: {}\n") + assert result.exit_code == 0 + assert json.loads(result.stdout) == { + "valid": True, "tables": 0, "foreign_keys": 0, "byte_length": 11 + } + assert result.stderr == "" +``` + +A bad document must exit nonzero with no stdout and a bounded generic stderr message that does not echo input. + +- [ ] **Step 2 (verify RED):** + +```bash +cd harness +.venv/bin/pytest tests/test_schema_fk_annotations.py tests/test_annotation_validation_cli.py -q +``` + +Expected: FAIL because `parse_annotations_yaml` and `validate-annotations` do not exist and current Pydantic/PyYAML parsing accepts extras/duplicates. + +- [ ] **Step 3 (GREEN): implement the minimal strict parser.** + +In `models.py`, add `MAX_ANNOTATION_BYTES = 16 * 1024 * 1024`, strict annotation-only Pydantic configs (`extra="forbid"`), annotation-FK validators requiring equal nonempty `columns`/`ref_columns`, and a `yaml.SafeLoader` subclass whose mapping constructor rejects duplicate keys and whose alias/merge handling fails closed. Decode bytes with `utf-8` strict, reject NUL/empty, call `yaml.load_all`, require exactly one mapping document, then `Annotations.model_validate` it. Keep physical-schema compatibility unchanged: do not globally make `_YamlModel` or the shared physical `ForeignKey` stricter. Reject annotation FK extras with an annotation-only model or a pre-validation allowlist, then map to the shared FK value object. + +Add `schema validate-annotations` with exact `--stdin --json` flags. Use `sys.stdin.buffer.read(MAX_ANNOTATION_BYTES + 1)`, never accept a caller path, and emit only the count envelope on stdout. + +- [ ] **Step 4 (verify GREEN and lint):** + +```bash +cd harness +.venv/bin/pytest tests/test_schema_fk_annotations.py tests/test_annotation_validation_cli.py \ + tests/test_protected_fs_helper.py -q +.venv/bin/ruff check tht/mschema/models.py tht/cli/schema_cmd.py tht/protected_fs_helper.py \ + tests/test_schema_fk_annotations.py tests/test_annotation_validation_cli.py \ + tests/test_protected_fs_helper.py +``` + +Expected: all focused tests PASS and Ruff reports no errors. + +- [ ] **Step 5: commit.** + +```bash +git add harness/tht/mschema/models.py harness/tht/cli/schema_cmd.py \ + harness/tests/test_schema_fk_annotations.py harness/tests/test_annotation_validation_cli.py +git commit -m "feat: validate curated annotation documents strictly" +``` + +--- + +### Task 2: Read one exact bounded annotation Git blob without checkout semantics + +**Files:** +- Create: `backend/src/workspaces/fixed-git-child.ts` +- Create: `backend/test/fixed-git-child.test.ts` +- Modify: `backend/src/workspaces/git-repository.ts` +- Modify: `backend/src/workspaces/types.ts` +- Modify: `backend/test/workspaces-git-repository.test.ts` + +- [ ] **Step 1 (RED): add real-Git tests for `readAnnotationBlobAtRevision`.** + +Extend the local bare-remote fixture with present, missing, executable regular blob, symlink, tree, and gitlink annotation paths. Tests must prove: + +1. path is derived only as `workspace-content//schema/annotations.yaml`; +2. lookup uses the supplied historical 40-hex commit, not `HEAD`; +3. a present regular blob returns exact mode/blob/bytes/SHA-256; +4. absence returns the canonical absent object and warning identity; +5. symlink/tree/gitlink are `annotation_invalid`; +6. exactly 16 MiB passes and 16 MiB + 1 fails before buffering the body; +7. malformed revision/ID, caller-controlled namespace/path attempts, and shell-like strings never create a marker file or escape the derived requested path; +8. a commit containing valid `workspace-content/other-workspace/schema/annotations.yaml` coexists: a request for the target reads only the target blob, never rejects or mutates the unrelated namespace, and a target absence remains the canonical absent value even when the unrelated blob exists; +9. corrupt Git storage is `git_unavailable`, not falsely “absent”; Git stderr/path/remote is redacted. + +First test the shared `runFixedGitChild` with real fixture processes: one monotonic absolute deadline spans all stages and is never reset; stdin is written exactly then closed; stdout bytes, parsed record length, and stderr are independently bounded. Cover a hang, infinite stdout, an individual record crossing its bound while aggregate output is still below its limit, infinite stderr, early/nonzero exit, parser throw, `AbortSignal` cancellation, and a descendant that keeps a pipe open or ignores TERM. Each failure must immediately stop parsing, TERM then KILL only the owned group, await the exit/close and every stdio settlement, retain at most the configured bounds, and return only a safe classification. Assert the parent/test process and an unrelated process group survive. + +Instrument the repository seam and assert only fixed vectors are used through that shared helper: `ls-tree -z -- `, `cat-file -s `, `cat-file blob `. All three share one logical-operation deadline; empty stdin is closed for each stage. Bound `ls-tree` to one complete derived-path NUL record, `cat-file -s` to one complete decimal size record, `cat-file blob` to the already-validated advertised length at most `MAX_CURATED_ANNOTATION_BYTES`, and stderr to the shared small fixed cap. Do not use `git show`, a shell, checkout, archive extraction, a mobile author file, `execFile`/`maxBuffer`, or a private second child runner. + +- [ ] **Step 2 (verify RED):** + +```bash +cd backend +npx vitest run test/fixed-git-child.test.ts \ + test/workspaces-git-repository.test.ts -t "fixed Git child|annotation" +``` + +Expected: FAIL because `runFixedGitChild`, the repository method, and `annotation_invalid` do not exist. + +- [ ] **Step 3 (GREEN): implement the fixed-argv reader.** + +Add `annotation_invalid` to `WorkspaceErrorCode`. Parse the single NUL-terminated `ls-tree` record as ` <40hex>\t\0`; zero records means absent, more than one or any mismatch is invalid. Derive the path only from the validated requested ID; another valid workspace namespace in the same commit is unrelated content and must not affect this lookup. Accept only `type=blob` and mode `100644|100755`. Query size before bytes; require safe integer `0..MAX_CURATED_ANNOTATION_BYTES`. + +Implement `runFixedGitChild` in the shared module exactly as frozen above and make the repository reader use it for all three stages with one caller-created monotonic deadline. The helper must stream bounded binary data, enforce record and stderr bounds during reads, close exact stdin, abort on parser/caller/limit/deadline failures, own and terminate only its process group with TERM→KILL, and await exit/stream teardown on every path. There is no private `spawnGit`, `execFile` buffer, or returned raw child output. Require exit 0 and exact advertised blob size, then compute SHA-256 after the exact bounded read. + +- [ ] **Step 4 (GREEN): add regression coverage to the complete repository test.** + +```bash +cd backend +npx vitest run test/fixed-git-child.test.ts test/workspaces-git-repository.test.ts +npx tsc --noEmit -p . +``` + +Expected: PASS; no existing Evidence-tree behavior regresses. + +- [ ] **Step 5: commit.** + +```bash +git add backend/src/workspaces/fixed-git-child.ts backend/test/fixed-git-child.test.ts \ + backend/src/workspaces/git-repository.ts backend/src/workspaces/types.ts \ + backend/test/workspaces-git-repository.test.ts +git commit -m "feat: read revision-pinned annotation blobs" +``` + +--- + +### Task 3: Atomically synchronize exact bytes with a verified ownership manifest + +**Files:** +- Create: `backend/src/workspaces/protected-workspace-fs.ts` +- Create: `backend/test/protected-workspace-fs.test.ts` +- Create: `harness/tht/protected_fs_helper.py` +- Create: `harness/tests/test_protected_fs_helper.py` +- Create: `backend/src/workspaces/annotations.ts` +- Create: `backend/test/workspace-annotations.test.ts` +- Modify: `backend/src/workspaces/preprocessing-state.ts` (extend the sole `WorkspaceLockedChildRequest` union and capability dispatcher; keep P2/P3 members and P2 state I/O unchanged) +- Create: `backend/test/workspace-locked-child-cumulative.compile.ts` (exact compile-time base-three + P3-fourteen + P5-two union fence) +- Modify: `backend/test/workspace-preprocessing-state.test.ts` (P3 locked-child focused cumulative runtime dispatcher regression) +- Modify: `backend/test/fixtures/workspace-lock-root-worker.mjs` +- Modify: `harness/tht/locked_child_stdin.py` +- Modify: `harness/tests/test_locked_child_stdin.py` + +- [ ] **Step 1 (RED): specify `AnnotationSynchronizer.ensure`.** + +Use real temporary directories plus an injected repository and validator. Add tests for: + +- present and absent publication at the exact revision-qualified destination; +- validator is called before any destination becomes visible; +- byte-for-byte equality and manifest equality to the contract above; +- `0700` directories and `0400` files on POSIX; +- idempotent reuse only after no-follow type/mode/size/digest/manifest validation; +- conflict if an existing manifest differs, a destination/ancestor is a symlink, or either file is modified; +- validator failure, injected short write, rename failure, and process interruption leave no manifest publication and remove only owned staging files; +- two concurrent repository-owned transactions serialize through P2's real workspace writer lock and end with one valid immutable publication; `ensure` receives the already-held matching capability and never acquires or re-enters that lock itself; +- old and new commits publish separate roots and never mutate each other; +- a deterministic ancestor-replacement race at each boundary (after validating inherited FD 4, after opening `revisions//artifacts/mschema` relative to it, after staging fsync, and immediately before each publication rename) either completes only in the originally retained directory or fails closed; it never writes into the replacement tree; +- unrelated valid workspace roots and Git namespaces remain byte-identical and are never inspected as candidate destinations; +- a compile-only cumulative-union fence defines the fourteen exact P3 kind literals and the two exact + P5 annotation kind literals as separate readonly tuples, rejects duplicates, and uses bidirectional + `Equal`/`Assert` checks to prove `WorkspaceLockedChildRequest["kind"]` is exactly the original three + P2 kinds plus those fourteen plus those two—neither a subset nor a widened string; +- a table-driven runtime regression creates one real `WorkspaceWriterLockCapability`, submits the + original three requests, all fourteen P3 requests, and both annotation requests through that same + capability, and asserts the sole exhaustive dispatcher selects each existing fixed argv/result path + exactly once. Its switch has an `assertNever` default, and the fixture records one capability/root + identity throughout; no parallel P3 or annotation spawner may make the test pass. + +Freeze the compile-only fence around the discriminants (with local `Equal`, `Assert`, and recursive +`Unique` type helpers) so the member count cannot be satisfied by duplicate or widened entries: + +```ts +const p3LockedKinds = [ + "p3_migrate_dwh_cache", "p3_prepare_dwh_cache", "p3_materialize_dwh_snapshot", + "p3_migrate_memory_root", "p3_rebuild_memory_projection", + "p3_inventory_semantic_legacy", "p3_check_semantic_readiness", + "p3_publish_semantic_replacements", "p3_verify_semantic_replacements", + "p3_delete_confirmed_semantic_legacy", "p3_prepare_layout_markers", + "p3_publish_layout_version", "p3_publish_revision_ready", + "p3_verify_revision_readiness", +] as const satisfies readonly WorkspaceLockedChildRequest["kind"][]; +const p2LockedKinds = ["dwh_preprocess", "schema_preprocess", "evidence_preprocess"] as const + satisfies readonly WorkspaceLockedChildRequest["kind"][]; +const p5LockedKinds = ["annotation_publish", "annotation_verify"] as const + satisfies readonly WorkspaceLockedChildRequest["kind"][]; +const p5CumulativeLockedKinds = [ + ...p2LockedKinds, ...p3LockedKinds, ...p5LockedKinds, +] as const; +type P5CumulativeLockedKind = typeof p5CumulativeLockedKinds[number]; +type _P5CumulativeKindsAreUnique = Assert>; +type _P5CumulativeUnionIsExact = Assert< + Equal +>; +``` + +The shared filesystem boundary is capability-only and cumulatively extends P3's single closed child +union in `backend/src/workspaces/preprocessing-state.ts`; it never accepts or derives an absolute workspace +path. `P3LockedChildRequest` remains the exact fourteen-member alias frozen by P3 and is not copied, +renamed, narrowed, or re-declared: + +```ts +export const ANNOTATION_BYTES_MAX = 16_777_216; +export const ANNOTATION_MANIFEST_BYTES_MAX = 65_536; +export const ANNOTATION_CHILD_STDIN_MAX = 16_908_288; +export const PROTECTED_FS_CHILD_STDOUT_MAX = 16_384; +export const PROTECTED_FS_CHILD_STDERR_MAX = 65_536; +export const PROTECTED_FS_CHILD_RESULT_MAX = 81_920; +export interface AnnotationPublishLockedChildRequest { + readonly kind: "annotation_publish"; + readonly workspaceId: CanonicalWorkspaceId; + readonly revision: Revision40; + readonly rootIdentity: WorkspaceLockRootIdentityV1; + readonly childRunId: string; + readonly annotationBytes: Uint8Array; + readonly annotationSha256: Sha256Hex; + readonly manifestBytes: Uint8Array; + readonly manifestSha256: Sha256Hex; +} +export interface AnnotationVerifyLockedChildRequest { + readonly kind: "annotation_verify"; + readonly workspaceId: CanonicalWorkspaceId; + readonly revision: Revision40; + readonly rootIdentity: WorkspaceLockRootIdentityV1; + readonly childRunId: string; + readonly annotationSha256: Sha256Hex; + readonly manifestSha256: Sha256Hex; +} +export type WorkspaceLockedChildRequest = + | DwhLockedChildRequest | SchemaLockedChildRequest | EvidenceLockedChildRequest + | P3LockedChildRequest + | AnnotationPublishLockedChildRequest | AnnotationVerifyLockedChildRequest; +export interface ProtectedWorkspaceFs { + publishImmutablePair(input: ImmutablePairPublication): Promise; + verifyImmutablePair(input: ImmutablePairVerification): Promise; +} +export function openProtectedWorkspaceFs(input: { + readonly rootLease: BorrowedVerifiedWorkspaceLockRootLease; + readonly writerCapability: WorkspaceWriterLockCapability; +}): ProtectedWorkspaceFs; +``` + +`openProtectedWorkspaceFs` checks the opaque lease/capability workspace and root identities match and +retains neither beyond the caller's borrow. Each method calls only +`input.writerCapability.spawnChild()` with the matching exact request variant. There is no overload for +`workspaceRoot`, `dataRoot`, a root FD/number, raw argv, raw stdio, a spawn function, or ambient lookup. +The capability implementation owns the only spawn and fixes the executable plus argv to the selected +image's Python and exactly `-I -m tht.protected_fs_helper`. It canonical-frames the semantic request, +rejects an annotation or manifest over its individual cap and a complete stdin frame over 16,908,288 +bytes before spawn, streams into fixed-capacity collectors, caps stdout at 16,384 bytes and stderr at +65,536 bytes (81,920 combined), enforces the fixed protected-helper deadline, terminates its owned +process group on the first overflow/cancellation/deadline, and awaits exit and both stdio pipelines. +The result parser accepts one allowlisted identity envelope only. The request union has no path, argv, +stdio, environment, executable, or callback field. + +The child receives the actual locked writer open file description on FD 3 and the same retained root +directory open file description on FD 4. `protected_fs_helper.py` calls P2's shared +`require_workspace_writer_lock` before any workspace read or write: it `fstat`s FD 4 as the expected +root identity, opens `preprocessing/writer.lock` relative to FD 4 with `O_NOFOLLOW`, proves its +(device,inode) equals FD 3, validates FD 3 owner/mode/nlink/type and already-held flock, and rejects +missing, closed, substituted, independently locked, or cross-root descriptors. It derives the fixed +revision annotation leaves solely from the validated workspace/revision request and FD 4. It retains an +FD for every accepted component and uses only `openat`/`mkdirat`/`unlinkat` plus +`renameat2(RENAME_NOREPLACE)`, all relative to retained dirfds with no-follow semantics. It rechecks +root and ancestor identities before/after every read, write, fsync, and rename; there is no path reopen, +`lstat`-then-path fallback, or overwrite-capable rename fallback. + +Publication writes/fsyncs annotation and manifest staging leaves, publishes annotation then the manifest +marker, fsyncs/revalidates the retained parent and ancestors, and removes only an operation-owned partial +whose retained inode/digest still matches. Verification performs the same FD-3/FD-4 authorization and +anchored reads and returns identities only. The helper is not a public `tht` command and direct Python +invocation without both inherited descriptors fails before workspace access. + +Freeze real-process tests, not mock locks: pause the helper on an explicit barrier after the parent has +acquired the writer and installed child FDs but before the child validates/uses FD 4; replace the +canonical workspace pathname with a different directory/symlink, then release the barrier. Publication +may complete only in the retained original inode or fail, and the replacement tree must remain empty. +Also test missing FD 3, missing FD 4, substituted writer FD, substituted root FD, FD 3 from workspace A +with FD 4 from B, another independently locked writer, a forged marker, direct spawn, wrong requested +workspace/root identity, capability use after the ordered callback settles, and helper timeout/output +flood. Every case returns `preprocessing_conflict` or the bounded safe helper error with zero publication. + +Desired single-synchronizer API (do not add a prepare/runtime synchronizer pair): + +```ts +export type AnnotationMaterializationLifecycle = + | { + readonly kind: "prepare"; + readonly rootLease: BorrowedVerifiedWorkspaceLockRootLease; + readonly writerCapability: WorkspaceWriterLockCapability; + } + | { + readonly kind: "runtime"; + readonly rootLease: BorrowedVerifiedWorkspaceLockRootLease; + readonly descriptorBlob: Revision40; + readonly effectiveDwhCacheKey: Sha256Hex; + }; + +export class AnnotationSynchronizer { + ensure( + workspaceId: CanonicalWorkspaceId, + commit: Revision40, + lifecycle: Extract, + ): Promise; + verifyPrepared( + workspaceId: CanonicalWorkspaceId, + commit: Revision40, + lifecycle: Extract, + ): Promise; + readRuntime( + workspaceId: CanonicalWorkspaceId, + commit: Revision40, + lifecycle: Extract, + ): Promise; +} +``` + +The exact P2 participant first narrows `RegistryAddressedPlanV1` to `operation === "registry_pull"`. +For a target it calls `ensure(target.workspaceId, target.revision, +{ kind: "prepare", rootLease: workspace.rootLease, writerCapability: +workspace.writerCapability })`; for a base-only removal it returns the closed removal variant without +publication. `reconcile` uses `verifyPrepared` with those same borrowed capabilities. Thus the +participant consumes both objects supplied by `AddressedWorkspacePublicationLeaseV1` and cannot invent +a root path or acquire another lock. The prepare resolver calls P3 +`futureWorkspaceLayoutPaths(rootLease, workspaceId, revision)` only to verify the fixed semantic +layout; the helper itself receives no path and derives/opens beneath inherited FD 4. + +Pre-READY acceptance must be executed inside the existing P2 writer-capability lifetime and uses the +same `annotation_verify` child variant; it never directly spawns the helper. Runtime/session reads do +not call the mutating helper: while their existing verified root borrow and reader lease are alive they +use P3's root-relative safe reader after +`readRevisionLayoutState(rootLease, expected)` and +`workspaceRuntimePaths(rootLease, workspaceId, revision, state)`. This preserves runtime read-only +capability while ensuring every `protected_fs_helper` invocation, including verification, is authorized +by FD 3 and FD 4. + +- [ ] **Step 2 (verify RED):** + +```bash +cd harness +.venv/bin/pytest -q tests/test_protected_fs_helper.py tests/test_locked_child_stdin.py \ + tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py +cd ../backend +npx vitest run test/protected-workspace-fs.test.ts test/workspace-annotations.test.ts \ + test/workspace-preprocessing-state.test.ts +npx tsc --noEmit --target ES2022 --module ES2022 --moduleResolution Bundler \ + --strict --skipLibCheck test/workspace-locked-child-cumulative.compile.ts +``` + +Expected: the existing P3 locked-child focused cases remain green, while the cumulative dispatcher, +compile-only union fence, helper, shared wrapper, and `annotations.ts` cases fail for only the missing +P5 surface. + +- [ ] **Step 3 (GREEN): implement safe publication.** + +Validate the branded workspace ID, revision, opaque borrowed-root identity, and matching writer capability. Construct `openProtectedWorkspaceFs({ rootLease, writerCapability })` only inside that borrow, then call `publishImmutablePair` or `verifyImmutablePair`; both dispatch the exact union variant through `writerCapability.spawnChild`. The child derives the fixed `revisions//artifacts/mschema` components from semantic fields and inherited FD 4, creates unique same-directory staging leaves, writes/fsyncs/rechecks both, publishes annotation first and manifest last, then fsyncs and revalidates the retained parent and every ancestor. The manifest is the publication marker. Do not overwrite an existing immutable publication—verify it or fail closed. Map parser/model failures to `annotation_invalid`; descriptor/capability mismatch, ancestor replacement, helper/syscall uncertainty, or filesystem corruption remains a safe workspace error. Tests must use the real pre-FD-validation process barrier, replace the canonical root after writer acquisition, and prove the replacement tree receives no file. + +Production validation must spawn the installed `tht` directly with fixed argv: + +```text +tht schema validate-annotations --stdin --json +``` + +Use no shell, pass exact bytes on stdin, bound stdout/stderr, enforce a short timeout, parse the exact success envelope, and discard raw errors. Put this adapter in `annotations.ts` as `createHarnessAnnotationValidator(...)` so backend and maintenance use one implementation. + +- [ ] **Step 4 (verify GREEN):** + +```bash +cd harness +.venv/bin/pytest -q tests/test_protected_fs_helper.py tests/test_locked_child_stdin.py \ + tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py +.venv/bin/ruff check tht/protected_fs_helper.py tht/locked_child_stdin.py \ + tests/test_protected_fs_helper.py tests/test_locked_child_stdin.py \ + tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py +cd ../backend +npx vitest run test/protected-workspace-fs.test.ts test/workspace-annotations.test.ts \ + test/workspace-preprocessing-state.test.ts +npx tsc --noEmit --target ES2022 --module ES2022 --moduleResolution Bundler \ + --strict --skipLibCheck test/workspace-locked-child-cumulative.compile.ts +npx tsc --noEmit -p . +``` + +Expected: PASS. The compile fence proves the exact base-three + fourteen-P3 + two-annotation kind +union, and the runtime table proves all nineteen requests still traverse one exhaustive +`WorkspaceWriterLockCapability.spawnChild` dispatcher. + +- [ ] **Step 5: commit.** + +```bash +git add harness/tht/protected_fs_helper.py harness/tests/test_protected_fs_helper.py \ + backend/src/workspaces/protected-workspace-fs.ts \ + backend/test/protected-workspace-fs.test.ts \ + backend/src/workspaces/annotations.ts backend/test/workspace-annotations.test.ts \ + backend/src/workspaces/preprocessing-state.ts \ + backend/test/workspace-locked-child-cumulative.compile.ts \ + backend/test/workspace-preprocessing-state.test.ts \ + backend/test/fixtures/workspace-lock-root-worker.mjs \ + harness/tht/locked_child_stdin.py harness/tests/test_locked_child_stdin.py +git commit -m "feat: materialize immutable workspace annotations" +``` + +--- + +### Task 4: Gate activation, active sessions, pinned sessions, and runtime rendering on synchronization + +**Files:** +- Modify: `backend/src/workspaces/registry.ts` +- Create: `backend/src/workspaces/registry-factory.ts` +- Modify: `backend/src/app.ts` +- Modify: `backend/src/workspace-maintenance.ts` +- Modify: `backend/src/workspaces/runtime-config-lease.ts` +- Read/consume unchanged: `backend/src/workspaces/revision-layout.ts` +- Modify: `backend/test/workspace-registry.test.ts` +- Create: `backend/test/workspace-registry-factory.test.ts` +- Create: `backend/test/p5-registry-pull-imports.compile.ts` (imports all six names only from `../src/workspaces/registry-pull-job.js`) +- Modify: `backend/test/workspace-runtime-handoff.test.ts` +- Modify: `backend/test/workspace-runtime-config-lease.test.ts` +- Modify: `backend/test/workspace-revision-layout.test.ts` +- Modify: `backend/test/workspace-runtime-renderer.test.ts` +- Modify: `backend/test/routes-sessions.test.ts` + +Freeze `backend/test/p5-registry-pull-imports.compile.ts` as a real field-parity fence, not merely an +import smoke test. It imports the six released P3 names only from +`../src/workspaces/registry-pull-job.js`, imports their owning P2 types directly—including +`AddressedWorkspacePublicationLeaseV1` from `registry-publication.js` and +`BorrowedWorkspaceSessionReadersExclusiveLockLease` from `preprocessing-state.js`—and contains these +exact bidirectional assertions (plus an `AllRegistryPullExports` tuple referencing the other three +released names): + +```ts +import type { + RegistryPullPublicJobRequestV1, + RegistryPullPhaseV1, + RegistryPullAddressedJobRequestV1, + RegistryPullParticipantStateV1, + RegistryPullSynchronizerStateV1, + RegistryPullJobStateV1, +} from "../src/workspaces/registry-pull-job.js"; +import type { + AddressedWorkspacePublicationLeaseV1, + RegistryAddressedPublicationPhaseV1, + RegistryAddressedRequestV1, + RegistryPullAddressedPlanV1, + RegistryPullAddressedPublicationStateV1, + RegistryRunId32, +} from "../src/workspaces/registry-publication.js"; +import type { + BorrowedWorkspaceSessionReadersExclusiveLockLease, +} from "../src/workspaces/preprocessing-state.js"; +import type { WorkspaceRegistry } from "../src/workspaces/registry.js"; +import type { + Revision40, + Sha256Hex, +} from "../src/workspaces/workspace-lock-root-lease.js"; + +type Equal = + (() => T extends A ? 1 : 2) extends (() => T extends B ? 1 : 2) + ? (() => T extends B ? 1 : 2) extends (() => T extends A ? 1 : 2) + ? true : false + : false; +type Assert = T; +type RequiredKey = {} extends Pick ? false : true; +type ExpectedPullAddressedRequestV1 = + | { + readonly mode: "create"; + readonly operation: "registry_pull"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly expectedBaseCommit: Revision40; + readonly remoteRefIdentitySha256: Sha256Hex; + } + | { + readonly mode: "resume"; + readonly operation: "registry_pull"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + }; +type PullCreate = Extract; +type PullResume = Extract; +type AllRegistryPullExports = readonly [ + RegistryPullPublicJobRequestV1, + RegistryPullPhaseV1, + RegistryPullAddressedJobRequestV1, + RegistryPullParticipantStateV1, + RegistryPullSynchronizerStateV1, + RegistryPullJobStateV1, +]; +type ExactP2Parity = readonly [ + Assert>, + Assert>, + Assert + >>, + Assert>, + Assert>, + Assert[0], RegistryAddressedRequestV1>>, + Assert>, + Assert>, + Assert>, + Assert>, + Assert>, + Assert>, + Assert>, + Assert>, + Assert>, + Assert>, + Assert>, + Assert>, + Assert>, +]; +export type { AllRegistryPullExports, ExactP2Parity }; +``` + +Also add source-boundary assertions that `WorkspaceRegistry` has exactly one `publishAddressed` +declaration and it takes exactly the required `RegistryAddressedRequestV1`; an omitted identity on +create or resume is covered by `@ts-expect-error`, and adding an optional/partial compatibility overload +must make the boundary test fail. + +- [ ] **Step 1 (RED): add registry/session/runtime tests.** + +Register exactly one object returned by `createAnnotationRegistryPublicationParticipant(...)` in the `WorkspaceRegistry` constructor-owned participant list; its public type is P2 `CapabilityAwareRegistryPublicationParticipant`, not a P5 hook alias. Assert `WorkspaceRegistry.publishAddressed`, not the participant or `AnnotationSynchronizer`, owns the released lifecycle: inside the validated installation/registry boundary it derives the exact production installation, repository, and remote-ref identities, passes all three required fields on both pull create and pull resume, passes the exact expected active base on pull create, takes real `repository.lock`, fetches once, pins every exact `RegistryPullAddressedPlanV1` field/digest including `installationIdentitySha256`, persists all three identities in `RegistryPullJobStateV1`, computes complete strict-lexical `changedWorkspaceIds` including target additions and base removals, and calls the participant with each exact `AddressedWorkspacePublicationLeaseV1`. For target records, `prepare` calls `ensure(id, exact target revision, { kind: "prepare", rootLease: workspace.rootLease, writerCapability: workspace.writerCapability })`; for base-only removals it records the closed removal result without file publication. Every target annotation prepares before `publication_intent_durable`; failure preserves exact base. This path succeeds for a deliberately unready target and derives its immutable annotation destination with exact `futureWorkspaceLayoutPaths(rootLease, workspaceId, workspaceRevision)` without READY. Assert `acquireSessionRevision()` and `readPinned()` instead call the same synchronizer with the `runtime` lifecycle and exact descriptor blob/effective-DWH cache key before returning. A content-only annotation commit has the same descriptor blob but a new workspace commit and distinct runtime annotation path. A resumed historical session still renders the old commit's bytes after a newer pull. Replaced manifest/file refuses both new and resumed sessions. + +Use real independent `flock`-contending processes, the real `addressed-publication-jobs/.json`, and exact P2/P3 types—never a mock mutex, anonymous job shape, or timeout-only trace. Present at least two changed workspace identities in reverse input order, including an addition/removal, and assert the persisted `RegistryPullJobStateV1.changedWorkspaceIds` is the unique complete strict-lexical symmetric difference and `changedSetSha256`/`planSha256` match canonical bytes. Barriers must prove `repository.lock acquire → request_claimed → target_advertised → exact-OID immutable-ref fetch → target_fetched → RegistryPullJobStateV1 phase planned → one runUnderOrderedWorkspaceWriterLocks callback → lexical root/writer for the complete set → lexical quiescence/readers for the complete set → CapabilityAwareRegistryPublicationParticipant.prepare with each matching AddressedWorkspacePublicationLeaseV1 → participants_prepared → publication_intent_durable → active-pointer sibling write+file fsync+atomic rename+parent fsync+target-byte verification → target_published → terminal_durable → ordered callback settlement and reverse close → repository.lock release`. The complete `OrderedWorkspaceWriterCapabilitySet` must still contend at every changed ID until after publication rename+parent fsync; a competing repository operation serializes, and a process holding the later lexical writer makes pull wait while repository remains held. AST/capability fakes fail if participant `prepare`/`reconcile` can call repository, root/reader acquisition, `runUnderWorkspaceWriterLock`, or `publishAddressed`; counters prove each live create/resume attempt acquires each ID once, uses the same opaque capability in participant calls, and has no nested/duplicate reentry. + +Run the exact same-ID SIGKILL dependency matrix with `core` absent and a fetch/remote-resolution/publication counter. Case 1 kills the real one-shot at persisted phase `publication_intent_durable` immediately before the active-pointer rename: the active pointer and projections remain exact base; resuming the same 32-hex `runId` and identical `requestSha256` loads the recorded `RegistryPullJobStateV1`, performs no fetch/ls-remote/remote resolution/target selection, reacquires the recorded complete lexical set once in the new process, and converges to target. Case 2 kills immediately after the active-pointer rename and parent fsync but before the `target_published` state rewrite: exact target is active while durable job phase is still `publication_intent_durable`; same-ID resume accepts all-target as lost acknowledgement, performs no fetch and no second active publication rename, advances through `target_published`/`terminal_durable`, owner-clears, reverse-releases the reacquired set, then releases repository. Assert one fetch total and one publication rename total in each case. Separately inject a third active revision/descriptor/manifest identity, changed/deleted pinned target object, target manifest/inventory/digest drift, installation identity drift, repository identity drift, remote-ref identity drift, moved local remote-tracking OID, different ID, same ID with different digest, and cross-operation resume; each returns `preprocessing_resume_mismatch` or the released closed conflict code before network or state mutation and without refetch, participant entry, or owner clear. A crash releases OS locks but never silently clears nonterminal ownership. + +Freeze the lifecycle transition test explicitly: pull revision B and prove its prepared annotation manifest verifies at the future revision path while B has no READY; session admission and preprocessing resume both refuse B with `migration_required`. Run the exact released `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json` transition and prove its `p3_migrate_dwh_cache`/`p3_prepare_dwh_cache` plus `p3_materialize_dwh_snapshot` stages publish and strictly reverify B's binding-qualified physical/LSH snapshot under the future B/binding root without reading or publishing READY or falling back to A; admission and resume must still refuse `migration_required`. Only then may `schema accept` with the prepare lifecycle succeed; assert admission/resume remain `migration_required` after accept. Run the exact P3 `workspace migrate activate-revision-layout --workspace --yes --json` operation and strictly re-read the READY for B plus the probed binding; only then do session admission, pinned/runtime annotation consumption, and accepted preprocessing resume succeed. A READY or snapshot for another revision or binding must not satisfy this test. + +Update renderer tests to consume P3's exact roots: + +```ts +const state = readRevisionLayoutState(rootLease, { + workspaceId, + workspaceRevision, + descriptorBlob, + effectiveDwhCacheKey, +}); // exact synchronous P3 no-follow state read +const runtimePaths = workspaceRuntimePaths( + rootLease, + workspaceId, + workspaceRevision, + state, +); +// artifacts/indexes/corpus under .../revisions// +// memory workspace-global; dwh_cache is /preprocessing/dwh-cache +// acquireSession and ordinary committed pinned/runtime acquisition use this helper +// pre-READY acceptance maintenance instead uses futureWorkspaceLayoutPaths and never this state +// harness/tht/effective_dwh.py::effective_dwh_cache_root(cfg) appends the v2 binding digest +``` + +Assert `paths.artifacts + /mschema/annotations.yaml` is the verified synchronized destination and that there is no workspace-global annotations path or compatibility symlink. + +- [ ] **Step 2 (verify RED):** + +```bash +cd backend +npx vitest run \ + test/workspace-registry.test.ts \ + test/workspace-registry-factory.test.ts \ + test/workspace-runtime-handoff.test.ts \ + test/workspace-runtime-config-lease.test.ts \ + test/workspace-revision-layout.test.ts \ + test/workspace-runtime-renderer.test.ts \ + test/routes-sessions.test.ts -t "annotation|pinned|revision" +``` + +Expected: FAIL because registry/runtime paths do not yet require annotation synchronization. + +- [ ] **Step 3 (GREEN): wire one shared synchronizer.** + +Create `createWorkspaceRegistry(...)` in `registry-factory.ts`; it constructs one production `AnnotationSynchronizer` from the registry repository and the fixed selected-image annotation validator, then creates exactly one `CapabilityAwareRegistryPublicationParticipant` with `createAnnotationRegistryPublicationParticipant`. Both `buildApp()` and `backend/src/workspace-maintenance.ts::main` call this factory and pass that exact participant to the `WorkspaceRegistry` constructor-owned participant list; do not create a second synchronizer, participant interface, lease-set alias, lock file, or publication path. The pull path delegates once to `WorkspaceRegistry.publishAddressed`. P2 `WorkspaceRegistry.publishAddressed` alone acquires the complete set and passes it to the `CapabilityAwareRegistryPublicationLifecycleOwner`, which owns it through active-pointer write/file-fsync/rename/parent-fsync/verification and `terminal_durable`, then lets the ordered callback settle and reverse-close before repository release. `AnnotationSynchronizer.verifyPrepared` takes no lock itself and uses only the supplied writer capability; participant `prepare`/`reconcile` use only the supplied `AddressedWorkspacePublicationLeaseV1` and never re-enter repository or a writer acquisition. + +Implement the synchronizer's lifecycle split exactly: + +- `prepare`: validate the borrowed root identity, workspace ID, and 40-hex revision, then call only P3's exact `futureWorkspaceLayoutPaths(rootLease, workspaceId, workspaceRevision)`. Repository-owned pull/activation synchronization uses `ensure(..., { kind: "prepare", rootLease })`; pre-READY acceptance uses capability-only `verifyPrepared(..., { kind: "prepare", rootLease, writerCapability })`. Neither path calls `readRevisionLayoutState`, requires READY, accepts a caller path, or falls back to currently active/P2 roots. It publishes or validates the manifest at `futurePaths.artifacts/mschema` before `active.json` changes. +- `runtime`: session and pinned runtime consumption synchronously call the exact P3 state reader `readRevisionLayoutState(rootLease, { workspaceId, workspaceRevision, descriptorBlob, effectiveDwhCacheKey })`, require the returned strict committed `RevisionLayoutState` to be `revision-layout-v1` with the exact non-null READY for that commit+binding, and only then call the exact four-argument P3 export `workspaceRuntimePaths(rootLease, workspaceId, workspaceRevision, state)`. It must not omit the borrowed root lease, pass a layout string in place of `state`, duplicate joins, use `futureWorkspaceLayoutPaths`, or fall back to `p2-global`/another READY. + +Make `WorkspaceRuntimeConfigLeaseFactory.acquireSession(snapshotPath)` and normal post-READY pinned/resume runtime paths use the runtime lifecycle before `renderRuntimeConfig`; the prepare-mode acceptance maintenance path uses only the prepare lifecycle and cannot spawn a session. Render only P3's DWH-cache base. `harness/tht/effective_dwh.py::effective_dwh_cache_root(cfg)` appends the v2 binding digest returned by `canonical_effective_dwh_binding(cfg)`; do not compute or reimplement that digest in TypeScript. + +A missing/relative data root is `workspace_not_activatable`; there is no annotation-unaware production constructor path. Keep `revisionContentRoot` reserved for P6 Evidence; annotation consumption is through revision-qualified `RuntimePaths.artifacts`. + +- [ ] **Step 4 (GREEN): run focused and complete backend gates.** + +```bash +cd backend +npx vitest run \ + test/workspace-registry.test.ts \ + test/workspace-registry-factory.test.ts \ + test/workspace-runtime-handoff.test.ts \ + test/workspace-runtime-config-lease.test.ts \ + test/workspace-revision-layout.test.ts \ + test/workspace-runtime-renderer.test.ts \ + test/routes-sessions.test.ts +npx tsc --noEmit -p . +npx tsc --noEmit --target ES2022 --module ES2022 --moduleResolution Bundler \ + --strict --skipLibCheck test/registry-pull-job-imports.compile.ts \ + test/p5-registry-pull-imports.compile.ts +npm run build +``` + +Expected: PASS; `dist/workspaces/annotations.js` exists. + +- [ ] **Step 5: commit.** + +```bash +git add backend/src/workspaces/registry.ts backend/src/workspaces/registry-factory.ts \ + backend/src/app.ts backend/src/workspace-maintenance.ts \ + backend/src/workspaces/runtime-config-lease.ts \ + backend/test/workspace-registry.test.ts backend/test/workspace-registry-factory.test.ts \ + backend/test/p5-registry-pull-imports.compile.ts \ + backend/test/workspace-runtime-handoff.test.ts \ + backend/test/workspace-runtime-config-lease.test.ts \ + backend/test/workspace-revision-layout.test.ts \ + backend/test/workspace-runtime-renderer.test.ts backend/test/routes-sessions.test.ts +git commit -m "feat: bind session annotations to workspace revisions" +``` + +--- + +### Task 5: Upgrade P2 run state to a digest-bound explicit revision transition + +**Files:** +- Create from the unmodified P2 writer: `backend/test/fixtures/preprocessing-v1-fk-review/README.md` +- Create from the unmodified P2 writer: `backend/test/fixtures/preprocessing-v1-fk-review/{paused-unreviewed,paused-reviewed}/{job.json,candidate.yaml}` +- Create from the unmodified P2 writer: `backend/test/fixtures/preprocessing-v1-fk-review/paused-reviewed/review.json` +- Modify: `backend/src/workspaces/preprocessing-state.ts` +- Modify: `backend/test/workspace-preprocessing-state.test.ts` + +- [ ] **Step 1 (characterization, before RED): capture and verify exact P2 V1 bytes.** + +Use the final P2 test writer/API to produce both fixture directories; do not hand-author JSON. Record the generation command and SHA-256 of every file. Run the existing P2 reader tests plus new characterization assertions that the exact fixtures load as their documented paused states. Expected: PASS before any production edit. If not, stop and amend the P2 checkpoint. + +- [ ] **Step 2 (RED): add state-store migration/transition tests.** + +Create byte fixtures for every row of the frozen V1→V2 table above and a table-driven transition test. Tests must cover: + +- a strict P2 V1 `manual_review_required` job records the existing `fk-candidates/.yaml`, count/`candidateDigest`, and base revision under the state-store-owned `preprocessing/fk-candidates/` directory; +- candidate and optional V1 review are restrictive regular non-symlinks, match embedded workspace/run/revision identity, are rehashed on every migration/export/accept, accept exactly 716,800 decoded candidate bytes, and reject 716,801 before state transition or export; +- V1 paused-without-review maps only to V2 `pending`; V1 paused-with-valid-local-review also maps to `pending` with `reviewedDigest` audit-only; neither is accepted; +- progressed/terminal V1 remains immutable legacy audit state and cannot be exported, accepted, or resumed; malformed/conflicting/unknown V1 fails closed; +- `acceptAnnotationRevision(runId, acceptance)` is allowed once only from V2 `pending`, under the workspace writer lock; +- acceptance requires candidate digest equality and every canonical accepted field shown above; partial accepted fields are invalid; +- exact `accepted → accepted` replay is idempotent only after revalidation; a different commit/blob/digest/binding fails; every skip, rewind, `required`, or `reviewed` status is rejected; +- the migration cannot manufacture `baseGitBlob`: it requires a caller-supplied, already synchronized base identity and rejects a revision/content mismatch; +- a resume validator accepts exactly the recorded accepted revision/blob/content/binding and rejects each single-field mutation with `preprocessing_resume_mismatch`. + +- [ ] **Step 3 (verify RED):** + +```bash +cd backend +npx vitest run test/workspace-preprocessing-state.test.ts -t "annotation|candidate|accept" +``` + +Expected: FAIL because P2 state has no acceptance record or controlled cross-revision transition. + +- [ ] **Step 4 (GREEN): implement atomic state/candidate ownership.** + +Use the existing `PreprocessingStateStore` closed transitions, the shared `ProtectedWorkspaceFs`, state directory, and writer lock. Extend the existing candidate/review path and add only `migrateFkReviewCheckpoint`, `acceptAnnotationRevision`, and `assertAcceptedAnnotation`; do not invent caller-supplied paths. Implement exactly the frozen table and two-state V2 union—do not retain `required`/`reviewed` aliases or infer acceptance from P2 local review. Never accept a candidate path from CLI JSON. Reuse P2's exact `MAX_FK_CANDIDATE_BYTES = 716_800` decoded-byte limit for state-owned suggestions, V1 fixtures, migration, reread, and export; never substitute the 16 MiB curated-blob limit. Write candidates exclusively/no-follow, store digest before publishing state, and delete an unreferenced staging candidate on failure. Keep timestamps audit-only; no timestamp participates in identity. + +- [ ] **Step 5 (verify GREEN):** + +```bash +cd backend +npx vitest run test/workspace-preprocessing-state.test.ts +npx tsc --noEmit -p . +``` + +Expected: PASS. + +- [ ] **Step 6: commit.** + +```bash +git add backend/test/fixtures/preprocessing-v1-fk-review \ + backend/src/workspaces/preprocessing-state.ts \ + backend/test/workspace-preprocessing-state.test.ts \ + backend/test/fixtures/workspace-lock-root-worker.mjs \ + harness/tht/locked_child_stdin.py harness/tests/test_locked_child_stdin.py +git commit -m "feat: record annotation review acceptance" +``` + +--- + +### Task 6: Add `schema accept` and enforce the accepted blob on continuation + +**Files:** +- Modify: `backend/src/workspaces/preprocessing-state.ts` +- Modify: `backend/src/workspaces/preprocessing-service.ts` +- Modify: `backend/src/workspace-maintenance.ts` +- Modify: `backend/test/workspace-preprocessing-state.test.ts` +- Modify: `backend/test/workspace-preprocessing-service.test.ts` +- Modify: `backend/test/workspace-maintenance.test.ts` +- Modify: `backend/test/workspace-registry.test.ts` (strict active-ID enumeration and lock-order barriers) + +- [ ] **Step 1 (RED): write run-resolution, operator, and machine-envelope tests.** + +Add two closed request discriminators: + +```ts +{ operation: "schema-export-fks"; runId: string } +{ operation: "schema-accept"; runId: string; yes: true } +``` + +`schema-export-fks` uses `resolveRun`, requires the run to be the paused P2 `manual_review_required` / V2 `pending` checkpoint, opens the exact state-owned `fk-candidates/.yaml` through `PreprocessingStateStore`, rejects more than exactly 716,800 decoded bytes, rehashes it against `candidateDigest`, and returns only P2's bounded internal `hostExport` plus run/workspace/digest/count. Base64 plus the complete child envelope must remain within P2's exact 1 MiB machine-output maximum. It never invokes FK suggestion, DWH, Git, or the annotation synchronizer, and it never exports a progressed, terminal, ambiguous, removed-workspace, or accepted run. + +First freeze and test the bounded locator: + +```ts +export interface ResolvedPreprocessingRun { + workspaceId: string; + rootIdentity: WorkspaceLockRootIdentityV1; + runId: string; +} +export function resolveRun(input: { + dataRoot: string; + activeWorkspaceIds: readonly string[]; + runId: string; +}): Promise; +``` + +Tests use the real protected filesystem and cover zero and one match, duplicate IDs in two active workspaces, traversal/shell-like/non-32-hex input, removed/inactive workspace, a job symlink, a job hardlink, swapped ancestor, oversized/unknown JSON, and embedded workspace/run mismatch. Assert there is no recursive directory walk, every candidate path is the one exact job leaf under a validated active ID, total work is capped by the existing registry workspace limit and P2 state-size limit, and all public errors are the three frozen safe codes. After resolution, replace the file before writer acquisition and prove the mandatory locked reread refuses it. + +Then prove this exact `schema-accept` transaction, assuming the curator already completed the separate P3 pull followed by exact `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json`, and the latter strictly reverified the selected unready revision's binding-qualified physical/LSH snapshot without READY: + +1. validate `runId`/`yes` and obtain the strict bounded active workspace ID set without taking repository or writer lock; +2. call `resolveRun`, open/rehash the paused V1/V2 state and exact state-owned candidate (at most 716,800 bytes) only to establish the candidate workspace, then close all handles; +3. call only P3's existing read-only `WorkspaceRegistry.read(workspaceId)` active immutable-snapshot API, then read the prepared annotation manifest identity and the strict P3-selected physical/LSH snapshot identity at that exact future revision+binding path; refuse a missing/mismatched snapshot, prior-revision fallback, or READY-dependent resolver; do not call pull, migration, a repository-owned method/lock, Git, or `AnnotationSynchronizer.ensure()`; +4. acquire that workspace's P2 writer lock; +5. reread `active.json` and require the exact same active revision/snapshot, then reopen/revalidate the resolved paused state and candidate; call only repository-free `AnnotationSynchronizer.verifyPrepared(..., { kind: "prepare", rootLease, writerCapability })` for that exact already-materialized active identity, using `futureWorkspaceLayoutPaths` and no READY/runtime-state requirement; +6. reject changed active identity, unchanged base revision, absent blob, invalid materialization, wrong workspace, removed workspace, or already advanced run; +7. compare the P3 harness-owned effective-DWH binding to the paused binding; +8. run harness `schema check --json` against the current revision paths and exact binding-qualified physical/LSH snapshot that the preceding P3 DWH migration published and strictly reverified without READY; +9. atomically record candidate/current blob identities only after all checks pass; +10. return a safe envelope with run/workspace/base/current revisions, candidate/current `sha256:<64hex>` digests, blob ID, `matches_candidate`, and status `accepted`. + +Add deterministic lock-seam tests for both directions. The P3 `registry_pull` path must record `repository.lock acquire → request_claimed → target_advertised → exact-OID fetch → target_fetched → planned → one runUnderOrderedWorkspaceWriterLocks callback in strict lexical order → participant prepare/reconcile with the matching AddressedWorkspacePublicationLeaseV1 objects → active-pointer sibling write + file fsync + atomic rename + parent fsync + target-byte verification while every capability remains held → target_published → terminal_durable → ordered callback settlement and reverse close → repository.lock release`; fail if any writer/root/quiescence/reader lease is released before the active rename+parent fsync or if repository releases before the reverse set release. A normal `schema-accept` records `active snapshot read → one-element ordered writer callback → rootLease/writerCapability verification → active/state/materialization revalidation → callback settlement` with zero repository/Git events. In the race, acceptance holds writer after its snapshot read, a pull acquires repository and blocks while entering the sole `runUnderOrderedWorkspaceWriterLocks` callback before active publication, acceptance revalidates and completes without requesting repository, releases writer, and the pull then prepares and publishes while its full set remains held. A pull that wins before acceptance acquires writer changes the active pointer, so acceptance's locked identity reread fails closed and the curator retries against the new active snapshot. Fail structurally on any writer→repository attempt, participant lock reentry, or repository/Git dependency reachable from the accept callback. This proves the global order and absence of the P5/P6 deadlock rather than relying on a timeout-only assertion. + +Add the pre-READY/READY resume tests as one ordered transition: after pull, the prepare manifest verifies and both session admission and `preprocess run --resume ` return `migration_required`. Execute exact released `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json`; assert the selected unready commit+binding's physical/LSH snapshot is published and strictly reverified by the fixed cache/materialization stages without READY, and admission/resume still return `migration_required`. Only then does `schema accept` succeed; immediately retest both refusals and prove no later schema/Evidence stage ran. Execute P3's exact `workspace migrate activate-revision-layout --workspace --yes --json` command, strictly re-read the accepted revision's exact commit+effective-binding READY, and only then require the same accepted resume to continue at the stage after FK review. A READY for another commit/binding, a subsequent Git update, byte replacement, manifest corruption, or DWH binding change returns `preprocessing_resume_mismatch` or `migration_required` as owned by the violated boundary and does not index schema/Evidence. A new invocation without `--resume` creates a new run. Assert raw parser/Git/harness errors and endpoints never enter JSON. + +- [ ] **Step 2 (verify RED):** + +```bash +cd backend +npx vitest run \ + test/workspace-preprocessing-state.test.ts \ + test/workspace-preprocessing-service.test.ts \ + test/workspace-registry.test.ts \ + test/workspace-maintenance.test.ts -t "resolve run|lock order|schema export|schema accept|accepted annotation|resume" +``` + +Expected: FAIL because `schema-export-fks`, secure run resolution, and `schema-accept` are unknown. + +- [ ] **Step 3 (GREEN): implement minimal dispatch and resume checks.** + +Extend `WorkspacePreprocessingService.execute()`; reuse `AnnotationSynchronizer`, `PreprocessingStateStore`, `resolveRun`, `WorkspaceRuntimeConfigLeaseFactory`, the P3 harness-owned effective-binding result, P3 `WorkspaceRegistry.read(id)` active immutable-snapshot API, and P2's child invocation. Require `yes === true`; do not prompt inside the container. The service, not `backend/src/workspace-maintenance.ts::main`, owns the transaction. Inject only the RO active-snapshot reader, P3 strict pre-READY physical/LSH snapshot resolver/verifier, and capability-only prepared-annotation verifier into the accept path: no repository method, repository lock, pull or migration function, `ensure()`, or Git child is present in its capability/type. The DWH verifier must accept only the selected revision+binding publication produced by `migrate_dwh_cache`, require its manifest/digests to strictly reverify without READY, and reject a prior-revision or sibling-binding snapshot. `verifyPrepared(..., { kind: "prepare", rootLease, writerCapability })` accepts an already prepared immutable identity, resolves only through `futureWorkspaceLayoutPaths`, and performs no lock or READY acquisition. `schema check` success is necessary but not sufficient—the subsequent state-store acceptance call is the human decision. Update `preprocess-run` resume handling to call `assertAcceptedAnnotation` before any later stage. + +- [ ] **Step 4 (verify GREEN):** + +```bash +cd backend +npx vitest run \ + test/workspace-preprocessing-state.test.ts \ + test/workspace-preprocessing-service.test.ts \ + test/workspace-registry.test.ts \ + test/workspace-maintenance.test.ts +npx tsc --noEmit -p . +npm run build +``` + +Expected: PASS and stdout remains one pristine JSON document. + +- [ ] **Step 5: commit.** + +```bash +git add backend/src/workspaces/preprocessing-state.ts \ + backend/src/workspaces/preprocessing-service.ts backend/src/workspace-maintenance.ts \ + backend/test/workspace-preprocessing-state.test.ts \ + backend/test/workspace-preprocessing-service.test.ts \ + backend/test/workspace-maintenance.test.ts backend/test/workspace-registry.test.ts +git commit -m "feat: accept reviewed annotation revisions" +``` + +--- + +### Task 7: Expose safe author-clone export and explicit acceptance through `thothctl` + +**Files:** +- Modify: `tools/thothctl/internal/workspaceops/operations.go` +- Modify: `tools/thothctl/internal/workspaceops/operations_test.go` +- Modify: `tools/thothctl/internal/safeio/files.go` +- Modify: `tools/thothctl/internal/safeio/files_unix_test.go` +- Modify: `tools/thothctl/internal/safeio/files_windows_test.go` +- Modify: `tools/thothctl/cmd/thothctl/main.go` +- Modify: `tools/thothctl/cmd/thothctl/main_test.go` + +- [ ] **Step 1 (RED): add parse/run/file tests.** + +The released P5 host syntax is below; the pull, DWH snapshot preparation, and activation commands are required, already-released P3 operations and are not reimplemented by P5: + +```text +thothctl --installation /thothii-installation.yaml \ + workspace schema export-fks --run <32-hex-id> --output --json +# after edit + ordinary git commit/push: +thothctl --installation /thothii-installation.yaml \ + workspace registry pull --workspace --json +WORKSPACE_ID= +thothctl --installation /thothii-installation.yaml \ + workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json +# only after strict pre-READY physical/LSH snapshot verification: +thothctl --installation /thothii-installation.yaml \ + workspace schema accept --run <32-hex-id> --yes --json +# the accepted revision is still unready; this exact P3 operation must complete next: +thothctl --installation /thothii-installation.yaml \ + workspace migrate activate-revision-layout --workspace "$WORKSPACE_ID" --yes --json +# only after strict commit+binding READY verification: +thothctl --installation /thothii-installation.yaml \ + workspace preprocess run --resume <32-hex-id> --json +``` + +P2's `workspace schema suggest-fks --workspace ...` remains the operation that creates a new candidate, but it is **not** the curator export after a full run pauses and is never substituted for `export-fks`. `export-fks` must require exactly one run and output, use the same secure `resolveRun` and state-owned candidate read as acceptance, and report a `sha256:<64hex>` equal to the paused state's recorded `candidateDigest`. It must not accept `--workspace`, SQL, assumptions, input, resume, or suggestion flags. + +Keep P2's `--output` safety contract and name; do not add a second synonym. Tests require an absolute canonical output below an existing author-owned directory, refuse a symlink/reparse point in every existing component, refuse an existing destination, accept exactly 716,800 decoded candidate bytes and reject 716,801, require base64 plus the complete child envelope to remain within exactly 1 MiB, write a same-directory exclusive temp file, fsync, rename, and leave no partial on child failure. Candidate bytes come only from P2's dedicated framed/bounded `hostExport`, never logs/public JSON and never the separate 16 MiB curated-blob channel. Verify the child envelope run/workspace/digest/count, recompute the host bytes, require both equal the paused state digest, then remove `hostExport` before pristine JSON. On Windows use the existing safeio retained-handle/reparse policy; document its trusted-local-ACL limitation rather than weakening it. + +`schema accept` must require exactly one `--run`, literal `--yes`, optional `--json`, and no workspace/path/input flags. Both P5 commands reject traversal, zero/multiple run matches, removed workspaces, unsafe job/candidate files, and mismatched embedded identity using the frozen safe codes. Freeze the host sequence test: dispatch pull; observe admission/resume `migration_required`; dispatch the exact already-released `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json`; prove its returned operation is `migrate_dwh_cache`, its fixed cache plus `p3_materialize_dwh_snapshot` stages publish and strictly reverify the active unready revision's binding-qualified physical/LSH snapshot without READY, and admission/resume remain `migration_required`; dispatch accept and prove both remain refused; then dispatch exact `workspace migrate activate-revision-layout --workspace "$WORKSPACE_ID" --yes` (after interruption, only the corresponding command's returned 32-hex outer run ID may be supplied to that same command with `--resume`). Verify commit+binding READY and only then dispatch preprocessing resume successfully. Parser/dispatcher tests reject missing/duplicate `--workspace`, arbitrary paths, `--yes` on DWH preparation, and cross-operation/cross-workspace resume IDs. Assert the Go layer sends only fixed maintenance requests and redacts Compose stderr/secrets. Add usage text. + +Add capability-regression tests against P3's generated final override. `workspace registry pull` alone must select the registry RW mount and only the exact validated HTTPS/SSH Git transport needed by that installation. `workspace migrate dwh-cache`, `schema accept`, export, resume, and representative ordinary mutations must select the registry RO mount, omit every Git transport file/environment variable, and fail a fixture attempt to create the repository lock, alter the checkout, or contact the remote. Conversely, pull must not inherit unrelated DWH/Evidence/Pi secrets. Validate the rendered Compose structure before invocation and assert no P5 code broadens or duplicates P3's binding-discovery/generator surface. + +- [ ] **Step 2 (verify RED):** + +```bash +cd tools/thothctl +go test ./internal/workspaceops ./cmd/thothctl -run 'Annotation|SchemaAccept|Export|RegistryPull|MigrateDwhCache' -count=1 +``` + +Expected: FAIL because P5 acceptance parsing is absent and P2 export does not yet enforce the P5 author-clone contract. + +- [ ] **Step 3 (GREEN): implement host orchestration.** + +Extend the frozen `workspaceops.ParseWorkspaceCommand` / `workspaceops.Run`; continue using the exact P2 maintenance invocation for the P5 export/accept processes: + +```text +docker compose ... --profile workspace-maintenance run --rm --no-deps -T \ + workspace-maintenance +``` + +Reuse P3's existing `registry_pull`, `migrate_dwh_cache`, and `activate_revision_layout` orchestration and generated capability overrides unchanged. Do not mount the author clone. `export-fks` exports only the already-persisted candidate owned by the paused run through the bounded child channel and safe host writer; it never calls `suggest-fks`. Git add/commit/push remains a manual ordinary Git action outside `thothctl`. The curator invokes P3 pull explicitly after push, then explicitly invokes P3 `migrate dwh-cache` and preserves that operation's own durable returned run ID for same-command `--resume` recovery. The DWH preparation must finish and strictly reverify the selected unready revision's physical/LSH snapshot before the later accept process, which has only RO/no-Git capability and validates the registry's already-active exact blob/snapshot; it never pulls or migrates implicitly. Acceptance does not make the revision runnable: dispatch the exact P3 activation after accept, preserve its distinct durable returned run ID for exact same-command `--resume` recovery, verify the accepted commit+binding READY, and dispatch preprocessing resume only afterward. + +- [ ] **Step 4 (verify GREEN):** + +```bash +cd tools/thothctl +gofmt -w internal/workspaceops/operations.go internal/workspaceops/operations_test.go \ + internal/safeio/files.go \ + cmd/thothctl/main.go cmd/thothctl/main_test.go +go test ./... -count=1 +cd ../.. +bash scripts/test-preprocess-compose-config.sh +bash scripts/test-compose-secret-policy.sh +``` + +Expected: all Go tests and P3/P5 grammar/capability regressions PASS; the exact DWH migration is dispatched between pull and accept, generated `registry_pull` is RW/Git, generated `migrate_dwh_cache`, `schema-accept`, and ordinary mutation overrides are registry RO/no-Git, and no production credential is read. + +- [ ] **Step 5: commit.** + +```bash +git add tools/thothctl/internal/workspaceops tools/thothctl/internal/safeio \ + tools/thothctl/cmd/thothctl/main.go \ + tools/thothctl/cmd/thothctl/main_test.go +git commit -m "feat: add curated annotation acceptance commands" +``` + +--- + +### Task 8: Document the Git author workflow and create an independent manual walkthrough + +**Files:** +- Create: `docs/contracts/workspace-annotations.md` +- Modify: `docs/install/local-workspace-registry.md` +- Modify: `docs/install/server-workspace-registry.md` +- Modify: `docs/testing/p2-p6-manual-verification.md` +- Create: `backend/scripts/p5-manual-verification.mjs` +- Create: `backend/scripts/p5-manual-verification.test.mjs` +- Create: `scripts/p5-manual-verification.sh` + +- [ ] **Step 1 (RED): write manual-helper contract tests.** + +Follow the P1 helper's ownership discipline but use independent `.artifacts/manual-acceptance/p5`. Test only `prepare`, `status`, and `cleanup`; the helper must not run the reviewer commands or decide PASS. `prepare` creates a new local bare remote, ordinary `author/` clone, isolated installation/env/fixture-secret files, generated immutable command scripts, `ownership.json`, and `GUIDE.md`. Contract tests parse the generated guide/commands and fail unless pull → validated `WORKSPACE_ID` → exact `migrate dwh-cache --workspace "$WORKSPACE_ID" --json` → strict pre-READY physical/LSH verification → accept → activation/READY → resume appears in that order, with `migration_required` checks after pull, migration, and accept and distinct same-command recovery IDs for migration and activation. `cleanup` refuses live/unowned/replaced roots and deletes only the exact owned Compose project/resources and manual root; no prune or broad process matching. + +- [ ] **Step 2 (verify RED):** + +```bash +node --test backend/scripts/p5-manual-verification.test.mjs +bash -n scripts/p5-manual-verification.sh +``` + +Expected: FAIL because helpers do not exist. + +- [ ] **Step 3 (GREEN): write the exact P5 section and generated guide.** + +Replace only the P5 placeholder in `docs/testing/p2-p6-manual-verification.md`. Keep P2/P3/P4/P6 decisions untouched. The generated guide must have the reviewer personally: + +1. inspect clean author and installation roots plus exact ThothII commit/tree; +2. run `workspace preprocess run --workspace p5-fk --json` and record `manual_review_required`, the 32-hex run ID, and candidate digest; +3. run `workspace schema export-fks --run --output /workspace-content/p5-fk/schema/annotations.yaml --json`, hash the exported file, and require both the command digest and file digest to equal the paused run's recorded candidate digest **before** editing; +4. inspect/edit the exported YAML in the author clone and run the provided harness validation command without printing secrets; +5. prove `git status` contains annotations but no `physical.yaml`, then `git add`, `git commit`, `git push`; +6. record `git rev-parse HEAD`, `git rev-parse HEAD:workspace-content/p5-fk/schema/annotations.yaml`, mode/type/size, and SHA-256; +7. explicitly run the P3 `workspace registry pull --workspace --json` command, record the activated-but-unready revision, verify its prepared annotation manifest exists at the exact future revision path, and verify its generated maintenance override alone used registry RW plus exact Git transport; +8. immediately prove production session admission and `workspace preprocess run --resume --json` both refuse with `migration_required`, with no Pi/later preprocessing stage; +9. export `WORKSPACE_ID` as that exact resolved ID and run released `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json`; if interrupted, validate and use only its returned 32-hex outer run ID with the same DWH command plus `--resume`; inspect the fixed `p3_migrate_dwh_cache`/`p3_prepare_dwh_cache` and `p3_materialize_dwh_snapshot` evidence, require the selected revision+binding physical/LSH snapshot and manifest/digests under `revisions//dwh-snapshots//{artifacts,indexes}`, prove no prior-revision fallback and no READY publication, and recheck that admission/resume still refuse `migration_required`; +10. while still pre-READY, run `workspace schema accept --run --yes --json` as a distinct invocation, verify its override was registry RO/no-Git, prove it performed no pull/migration/repository-lock/Git child, inspect acceptance candidate/current digests plus `matches_candidate` (no requirement that it be true after curation), and recheck admission/resume remain `migration_required`; +11. inspect synchronized exact bytes, restrictive modes, and `annotations.manifest.json` fields; +12. run the exact P3 command `workspace migrate activate-revision-layout --workspace "$WORKSPACE_ID" --yes --json`; if interrupted, export only its distinct returned 32-hex outer run ID and recover with the same command plus `--resume `; then strictly inspect `READY.json` and require the accepted 40-hex commit plus the probed effective-DWH binding/cache key; +13. only after that READY verification, resume the accepted preprocessing run and confirm it crosses FK review with the accepted blob; admit a session and prove runtime consumption now succeeds; +14. pin/open the historical revision, publish a second annotation commit, explicitly pull again, repeat refusal → DWH cache/snapshot preparation → refusal → accept → refusal → exact activation → READY verification before use, and prove old/new runtime files remain different and unchanged; +15. prove an unrelated valid `workspace-content/other-workspace/schema/annotations.yaml` coexists and is never read/modified for `p5-fk`; separately attempt a caller-controlled mismatched ID/path and confirm it cannot escape the derived namespace; +16. run prepared negatives separately for candidate/export 716,801 decoded bytes and curated Git blob 16 MiB + 1, plus malformed, symlink, tree/gitlink, run traversal/duplicate/removed-workspace/symlink cases, accept without pull, accept without pre-READY DWH preparation, resume before READY at every boundary, wrong-commit/wrong-binding physical snapshot or READY, interrupted DWH/activation recovery with a wrong or cross-operation run ID, post-accept blob change, and DWH-binding mismatch; confirm no partial publication or later-stage work; +17. run the generated secret scan and exact cleanup; +18. record `Decision: **PASS**` or `**FAIL**` manually with report paths and notes. + +The install docs must state: only the canonical path is supported; absence yields warning/empty set; `physical.yaml` is never committed; the API does not push curated bytes; after a pause, `schema export-fks --run` (not a new suggestion) must reproduce the recorded candidate digest before editing; the candidate/export maximum is exactly 716,800 decoded bytes while the separate curated Git blob maximum is exactly 16 MiB. Freeze the supported order as author clone → Git commit/push → explicit P3 `workspace registry pull --workspace --json` → pre-READY admission/resume refusal with `migration_required` → exact released `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json` → strict selected-revision+binding physical/LSH snapshot verification without READY or prior-revision fallback → repeated `migration_required` refusal → distinct explicit `workspace schema accept --run --yes --json` → repeated `migration_required` refusal → exact P3 `workspace migrate activate-revision-layout --workspace "$WORKSPACE_ID" --yes --json` → strict accepted commit+effective-binding READY verification → preprocessing resume. Pull alone is registry RW/Git; DWH migration, accept, and ordinary mutations remain registry RO/no-Git. Both local and server install/recovery sections must show that interrupted DWH preparation and activation are resumed only with the distinct returned 32-hex outer run ID on the same exact originating command; operators must validate each ID, never cross-use it, never hand-create/edit READY or snapshot files, and never reuse the preprocessing run ID as a maintenance run ID. Unrelated workspace namespaces coexist; old pinned revisions keep old annotations. + +- [ ] **Step 4 (verify GREEN):** + +```bash +node --test backend/scripts/p5-manual-verification.test.mjs +./scripts/p5-manual-verification.sh prepare +./scripts/p5-manual-verification.sh status +./scripts/p5-manual-verification.sh cleanup +rg -n "schema export-fks|migrate dwh-cache|schema accept|annotations.yaml|physical.yaml|Decision: \*\*PENDING\*\*" \ + docs/testing/p2-p6-manual-verification.md \ + docs/install/local-workspace-registry.md docs/install/server-workspace-registry.md +``` + +Expected: tests PASS; prepare/status/cleanup PASS; P5 remains `PENDING` until a human decision; no other Px decision changed. + +- [ ] **Step 5: commit.** + +```bash +git add docs/contracts/workspace-annotations.md docs/install/local-workspace-registry.md \ + docs/install/server-workspace-registry.md docs/testing/p2-p6-manual-verification.md \ + backend/scripts/p5-manual-verification.mjs \ + backend/scripts/p5-manual-verification.test.mjs scripts/p5-manual-verification.sh +git commit -m "docs: add curated annotation review walkthrough" +``` + +--- + +### Task 9: Build the clean-state P5 automated process goal + +**Files:** +- Create: `backend/scripts/p5-acceptance.mjs` +- Create: `backend/scripts/p5-acceptance.test.mjs` +- Create: `scripts/p5-acceptance.sh` +- Create: `scripts/test-p5-acceptance.sh` + +The one-command interface is: + +```text +./scripts/p5-acceptance.sh integration [--keep] +``` + +Each run owns only: + +```text +.artifacts/p5-integration// +├── ownership.json +├── remote.git/ +├── author/ +├── installation/ +├── runtime-data/ +├── fixture-secrets/ +├── candidates/ +├── responses/ +├── manifests/ +├── logs/ +├── report.json +└── report.md +``` + +and a unique Compose project/container/network/volume/image reference labelled with the run ID. Use a child process group and one hard deadline; TERM then KILL only that group. There are no retries. All remote/DWH credentials are generated fixture canaries. Every command event records bounded sanitized argv category, exit status, start/end, and artifact hashes—not raw secret-bearing environment or unbounded stderr. + +- [ ] **Step 1 (RED): write harness unit/contract tests.** + +Test canonical run-root validation, exclusive ownership creation, symlink/root replacement refusal, unique run/Compose identities, fixed check IDs, no-retry execution, report schema/hash validation, secret scanning, external-network guard, cleanup refusal for foreign resources, process-group shutdown, and `--keep` semantics. Freeze the production event order as pull → first refusal → exact DWH cache/snapshot preparation → second refusal → accept → third refusal → activation/READY → resume, and fail if `pre_ready_dwh_snapshot_publication` or the all-three-boundaries `pre_ready_admission_resume_refusal` event is missing, duplicated, or reordered. Include mutation tests showing removal of any required negative case, secret scan, final ownership check, or report hash makes the harness test fail. + +Required report check IDs: + +```text +clean_source_identity +isolated_real_git_author_flow +paused_candidate_export_and_digest +secure_run_resolution +exact_commit_blob_validation +fixed_git_child_lifecycle +harness_parser_validation +dirfd_atomic_revision_materialization +operation_specific_registry_capability +registry_pull_same_id_recovery +registry_pull_target_drift_refusal +pre_ready_dwh_snapshot_publication +explicit_acceptance_transition +pre_ready_admission_resume_refusal +accepted_revision_layout_activation +repository_writer_lock_order +unrelated_namespace_coexistence +accepted_resume_and_mismatch_refusal +pinned_revision_isolation +negative_object_and_size_cases +distinct_candidate_blob_limits +physical_yaml_not_published +secret_scan +owned_resource_shutdown +exact_cleanup +``` + +- [ ] **Step 2 (verify RED):** + +```bash +node --test backend/scripts/p5-acceptance.test.mjs +bash scripts/test-p5-acceptance.sh +``` + +Expected: FAIL because the P5 harness does not exist. + +- [ ] **Step 3 (GREEN): implement the acceptance scenario through production interfaces.** + +From a clean source commit, the harness must build/use the real `thothctl`, real `workspace-maintenance` service/core image, real local bare Git remote and author clone, real registry pull, real P3 `migrate_dwh_cache` maintenance path, production parser/synchronizer/state store, and controlled REST DWH fixture sufficient to generate/check FK. It must: + +1. create a descriptor with no annotations and prove the safe absent publication/warning; +2. start a P2 run and capture run ID/candidate digest at `manual_review_required`; +3. invoke production `workspace schema export-fks --run --output ...`, require its returned digest and a fresh SHA-256 of the exact output to equal the paused run digest before editing, assert no suggestion/DWH child ran, and prove 716,800 decoded bytes fits the complete 1 MiB child envelope while 716,801 is refused before host publication; +4. edit that file in the ordinary author clone to a valid curated document, commit/push, and record exact commit/blob/mode/size/SHA-256; +5. invoke P3 production `workspace registry pull --workspace --json` explicitly through the exact `RegistryAddressedRequestV1`/`RegistryPullJobStateV1` path; prove create/resume both carry required production installation/repository/remote-ref identities, create also carries the exact expected base, plan/state retain all three identities, and only this operation receives registry RW/exact Git transport. In isolated real-process subcases with at least two reverse-presented changed workspace identities, SIGKILL at `publication_intent_durable` immediately before active-pointer rename and resume the exact returned 32-hex ID, then SIGKILL a fresh fixture immediately after active-pointer rename+parent fsync but before `target_published` persistence and resume that exact ID. Require complete strict-lexical `changedWorkspaceIds`, matching `AddressedWorkspacePublicationLeaseV1` participant calls, no participant/root/writer/reader reentry within each attempt, no fetch/ls-remote/target reselection on either resume, no second publication rename in the post-publication case, full-set ownership through rename+fsync, reverse release, then repository release. Inject third active identity and changed/deleted target object/inventory/digest and require refusal without mutation/owner clear. Only after those dependency subcases pass, run the scenario's successful C1 pull and prove the annotation participant prepares/verifies through exact `futureWorkspaceLayoutPaths` without READY; record `operation_specific_registry_capability`, `registry_pull_same_id_recovery`, `registry_pull_target_drift_refusal`, and `repository_writer_lock_order`. +6. before READY, invoke production session admission and `workspace preprocess run --resume --json`; require both to return `migration_required` and prove no Pi or later preprocessing stage starts; +7. set and validate the exact resolved `WORKSPACE_ID`, then invoke production `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json`; prove the `migrate_dwh_cache` run uses registry RO/no-Git and its fixed `p3_migrate_dwh_cache`/`p3_prepare_dwh_cache` plus `p3_materialize_dwh_snapshot` stages publish and strictly reverify the curated commit+binding physical/LSH snapshot under its own future revision root without reading or publishing READY or falling back to the base revision; exercise interrupted recovery only with this command's returned 32-hex outer run ID and record `pre_ready_dwh_snapshot_publication`; +8. retest session admission and preprocessing resume after DWH preparation; require `migration_required` and prove no Pi/later stage; +9. still before READY, invoke production `workspace schema accept --run --yes --json` separately; prove its generated container has registry RO/no-Git, its service calls no repository lock/pull/migration/Git child, its prepare verifiers succeed against the exact annotation and physical/LSH snapshots without READY, and acceptance records both digests and P3 effective binding; immediately prove admission/resume remain `migration_required`, then record `pre_ready_admission_resume_refusal` only after all three post-pull, post-DWH-preparation, and post-accept refusal pairs pass; +10. invoke the exact P3 production `workspace migrate activate-revision-layout --workspace "$WORKSPACE_ID" --yes --json`; exercise one interrupted activation and recover only with its distinct returned 32-hex outer run ID plus `--resume`; strictly verify the resulting READY names the accepted commit and exact probed effective-DWH binding/cache key before recording `accepted_revision_layout_activation`; +11. only after that check, resume the accepted preprocessing run and admit a session; require runtime annotation consumption to use strict committed state plus exact four-argument `workspaceRuntimePaths`, then verify exact runtime bytes and manifest, including restrictive modes and destination; +12. retain an old pin across a second commit and explicit second pull, repeat pre-READY refusal → exact DWH cache/snapshot preparation → refusal → accept → refusal → exact activation → READY verification, then prove byte/path isolation; +13. exercise `resolveRun` through production commands for traversal, duplicate run ID across two active workspaces, removed workspace, job/candidate symlink, hardlink, and embedded identity mismatch; require stable safe codes and no Git/DWH/later stage; +14. deterministically replace an already-opened revision/artifacts/mschema ancestor at the pull fixture's dirfd barrier and prove publication remains in the retained inode or fails with no file in the replacement tree; +15. add a valid annotation blob under another workspace namespace, prove the target lookup/accept ignores and preserves it, then separately attempt a mismatched caller-controlled lookup and prove confinement; +16. use separate fresh remotes/commits for malformed UTF-8/YAML, curated blob exactly 16 MiB and 16 MiB + 1, candidate/export exactly 716,800 and 716,801, symlink, tree, gitlink, missing-compatible case, accept without prior pull, accept without pre-READY DWH preparation, resume before READY at every boundary, wrong-commit/wrong-binding physical snapshot or READY, wrong/cross-operation DWH or activation resume ID, blob changed after acceptance, and DWH-binding mismatch; +17. exercise the shared fixed-Git child with real hang, stdout/record/stderr flood, early exit, parser abort, cancellation, and TERM-ignoring descendant fixtures; prove bounded capture, owned-group TERM→KILL, awaited exit, and no output exposure; +18. run the controlled pull/DWH-preparation/accept barriers and record repository→writer for pull, independent P3 maintenance for snapshot publication, snapshot→writer-only for accept, stable accepted identity, completion without deadlock, and the later pull advancing only after writer release; +19. prove every failed case preserves the last active state, publishes no incomplete annotation or DWH snapshot, performs no later preprocessing stage, and emits only stable safe codes; +20. inspect the Git tree to prove no `physical.yaml` exists; +21. shut down every owned child/listener/container, refuse connections, remove exact owned Compose resources (never global prune), remove fixture secrets, run the final secret scan, and record cleanup evidence. + +No frontend, browser API, Qdrant rebuild, Evidence materialization, Evidence adapter, or P6 retention is part of this process goal. + +`--keep` retains the bounded disk evidence/report after secret-file deletion and live-resource cleanup. Without `--keep`, copy the final report to a caller-selected safe location only if such behavior already exists in P2; otherwise remove the owned root. The release evidence run uses `--keep`. + +- [ ] **Step 4 (verify GREEN):** + +```bash +node --test backend/scripts/p5-acceptance.test.mjs +bash scripts/test-p5-acceptance.sh +``` + +Expected: all harness/mutation tests PASS without running the expensive integration scenario. + +- [ ] **Step 5: commit.** + +```bash +git add backend/scripts/p5-acceptance.mjs backend/scripts/p5-acceptance.test.mjs \ + scripts/p5-acceptance.sh scripts/test-p5-acceptance.sh +git commit -m "test: add curated annotation process goal" +``` + +--- + +### Task 10: Verify P5, run one clean retained goal, and publish the checkpoint report + +**Files:** +- Modify: `PROJECT_STATE.md` +- Modify only if evidence requires it: `docs/testing/p2-p6-manual-verification.md` + +- [ ] **Step 1: prove a clean committed source before the process goal.** + +```bash +git status --short +git diff --check +git rev-parse HEAD +git rev-parse HEAD^{tree} +``` + +Expected: clean status, no whitespace errors, exact commit/tree captured. Do not run the goal from uncommitted production bytes. + +- [ ] **Step 2: run focused layer verification.** + +```bash +cd harness +.venv/bin/pytest tests/test_schema_fk_annotations.py tests/test_annotation_validation_cli.py \ + tests/test_protected_fs_helper.py tests/test_locked_child_stdin.py \ + tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py -q +.venv/bin/ruff check tht/mschema/models.py tht/cli/schema_cmd.py tht/protected_fs_helper.py \ + tht/locked_child_stdin.py tests/test_schema_fk_annotations.py \ + tests/test_annotation_validation_cli.py tests/test_protected_fs_helper.py \ + tests/test_locked_child_stdin.py tests/test_layout_marker_commands.py \ + tests/test_p3_internal_cli.py + +cd ../backend +npx vitest run \ + test/workspaces-git-repository.test.ts \ + test/protected-workspace-fs.test.ts \ + test/workspace-annotations.test.ts \ + test/workspace-registry.test.ts \ + test/workspace-registry-factory.test.ts \ + test/workspace-runtime-handoff.test.ts \ + test/workspace-runtime-config-lease.test.ts \ + test/workspace-revision-layout.test.ts \ + test/workspace-runtime-renderer.test.ts \ + test/workspace-preprocessing-state.test.ts \ + test/workspace-preprocessing-service.test.ts \ + test/workspace-maintenance.test.ts \ + test/routes-sessions.test.ts +npx tsc --noEmit --target ES2022 --module ES2022 --moduleResolution Bundler \ + --strict --skipLibCheck test/workspace-locked-child-cumulative.compile.ts +npx tsc --noEmit -p . +npm run build +node --test scripts/p5-acceptance.test.mjs scripts/p5-manual-verification.test.mjs + +cd ../tools/thothctl +go test ./... -count=1 + +cd ../.. +bash scripts/test-p5-acceptance.sh +bash -n scripts/p5-acceptance.sh scripts/p5-manual-verification.sh +git diff --check +``` + +Expected: every command exits 0. Record exact test counts; do not summarize a failure as PASS. P6 owns the aggregate P2–P6 Docker smoke and full repository suites, so do not expand P5 into P6 scope. + +- [ ] **Step 3: run the process goal once, with no retry.** + +```bash +RETAINED_REPORT="$(./scripts/p5-acceptance.sh integration --keep)" +[[ "$RETAINED_REPORT" =~ ^\.artifacts/p5-integration/([0-9a-f]{32})/report\.md$ ]] || { + printf 'unexpected retained report path: %q\n' "$RETAINED_REPORT" >&2 + exit 2 +} +RUN_ID="${BASH_REMATCH[1]}" +printf 'export RUN_ID=%q\n' "$RUN_ID" +``` + +Expected: exit 0, exactly one captured retained path `.artifacts/p5-integration/<32-hex-run-id>/report.md`, a validated 32-lowercase-hex `RUN_ID`, and one safe export line for the next independently runnable step. If it fails, use @superpowers:systematic-debugging, add the smallest failing regression test, fix it, commit the scoped fix, and rerun the **entire scenario from a new run ID/root**. Never relabel or overwrite a failed report; the checkpoint names only the final green run. + +- [ ] **Step 4: independently inspect the retained report and cleanup evidence.** + +```bash +RUN_ID="${RUN_ID:?export RUN_ID as the 32-hex ID captured and printed by Step 3}" +[[ "$RUN_ID" =~ ^[0-9a-f]{32}$ ]] || { + printf 'invalid RUN_ID\n' >&2 + exit 2 +} +RUN=".artifacts/p5-integration/$RUN_ID" +[[ -d "$RUN" && ! -L "$RUN" ]] || { + printf 'missing or unsafe retained run root: %q\n' "$RUN" >&2 + exit 2 +} +python3 - "$RUN" <<'PY' +import hashlib, json, pathlib, sys +root = pathlib.Path(sys.argv[1]) +report = json.loads((root / "report.json").read_text()) +assert report["overall"] == "PASS" +assert all(check["status"] == "PASS" for check in report["checks"]) +assert report["cleanup"]["owned_resources_remaining"] == [] +for name in ("report.json", "report.md"): + print(name, hashlib.sha256((root / name).read_bytes()).hexdigest()) +PY +``` + +Then run the harness's documented retained-artifact secret scan verification. Expected: all required check IDs present exactly once, all PASS, no fixture secret files/canary values, no live owned resources, report hashes printed. Do not manually grep secret contents into terminal history. + +- [ ] **Step 5: update the checkpoint without claiming manual acceptance.** + +Add a P5 section to `PROJECT_STATE.md` containing: + +- implementation commit and tree; +- retained run/report path; +- `report.json` and `report.md` SHA-256; +- focused command/test counts; +- `automated integration: PASS`; +- `manual acceptance: PENDING`; +- known platform/manual limitations; +- explicit statement that P6 Evidence materialization and aggregate P2–P6 verification have not run. + +Keep the P5 manual section decision `PENDING` until the reviewer performs it. Run: + +```bash +git diff --check +git status --short +git add PROJECT_STATE.md +git commit -m "docs: record P5 automated verification checkpoint" +git status --short +``` + +Expected: final status clean. + +- [ ] **Step 6: stop for the required user checkpoint.** + +Provide a checkpoint report with scoped commits, exact commands/counts, retained path/hashes, secret-scan and cleanup result, and the exact manual entry command: + +```bash +./scripts/p5-manual-verification.sh prepare +``` + +State plainly: `automated integration: PASS / manual acceptance: PENDING`. Do not begin P6 or the aggregate test until the user records the P5 decision and explicitly authorizes continuation. + +## Final verification checklist + +- [ ] Canonical Git path is fixed and cross-workspace paths are impossible. +- [ ] Object is resolved at the descriptor's exact commit and only regular blob modes pass. +- [ ] Suggested candidate/export is exactly bounded at 716,800 decoded bytes and complete `hostExport` at 1 MiB; the separate curated Git blob alone is exactly bounded at 16 MiB; both boundaries and +1 failures are tested. +- [ ] Absence is compatible and distinguishable from an invalid empty file. +- [ ] Runtime publication is immutable, revision-qualified, restrictive, atomic, and manifest-owned through retained dirfds/openat-style operations; real ancestor-replacement races cannot redirect it. +- [ ] Active and pinned sessions/operators revalidate the exact manifest before use. +- [ ] Unrelated valid workspace namespaces coexist and are neither rejected nor read/modified; caller-controlled mismatch attempts cannot escape the requested namespace. +- [ ] P3 revision roots and effective DWH binding are reused; no global annotations pointer exists. +- [ ] `schema export-fks --run` resolves exactly one active workspace and exports/re-hashes the paused run's state-owned candidate; it never reruns suggestion. +- [ ] Run resolution is bounded/no-follow/ambiguity-safe and rejects traversal, duplicates, removed workspaces, links, and embedded identity drift with stable safe codes. +- [ ] Candidate export goes to an ordinary author clone; application code never pushes curated bytes. +- [ ] The only V2 checkpoint states are `pending` and `accepted`; every exact V1 row and legal transition is tested. +- [ ] Explicit accept records candidate digest, current blob ID/digest, new revision, and DWH binding. +- [ ] Curator explicitly runs P3 `workspace registry pull --workspace --json` after push; only `RegistryPullCommand` / `registry_pull` has writable registry storage and selected exact Git transport, while DWH migration, accept, and ordinary mutations remain registry RO/no-Git. +- [ ] After every pull and before every accept, exact released `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json` completes; its `p3_migrate_dwh_cache`/`p3_prepare_dwh_cache` plus `p3_materialize_dwh_snapshot` stages publish and strictly reverify the selected unready revision's binding-qualified physical/LSH snapshot without READY or prior-revision fallback, and `pre_ready_dwh_snapshot_publication` is present exactly once. +- [ ] The one `AnnotationSynchronizer` uses exact `futureWorkspaceLayoutPaths(rootLease, workspaceId, workspaceRevision)` for repository pull and pre-READY accept. Admission/resume are explicitly tested as `migration_required` after pull, after DWH snapshot preparation, and after accept; session/pinned runtime alone requires the strict committed READY state and exact four-argument `workspaceRuntimePaths(rootLease, workspaceId, workspaceRevision, state)`. +- [ ] After every successful accept and before every resume, the exact P3 `workspace migrate activate-revision-layout --workspace --yes [--resume ] --json` operation completes and the accepted commit+effective-binding READY is strictly verified; success occurs only afterward, and both refusal/activation report check IDs are present. +- [ ] `registry_pull` uses only `publishAddressed`: complete strict-lexical `OrderedWorkspaceWriterCapabilitySet` acquisition, exact participant leases, active-pointer write/file-fsync/rename/parent-fsync while the full set remains held, `terminal_durable`, reverse set release, then repository release. Accept uses active-snapshot-read→writer/revalidation only; controlled races prove no inversion, reentry, deadlock, or repository/Git call from accept. +- [ ] Real-process dependency tests use multiple reverse-presented changed IDs and exact `RegistryPullJobStateV1` fields/phases; same-ID SIGKILL before active rename and after rename+parent fsync resumes without refetch/reentry (and without a second post-publication rename), while target drift/third identity refuses without mutation or owner clear. +- [ ] The final sole `WorkspaceLockedChildRequest` alias is cumulative: the original three P2 members, unchanged fourteen-member `P3LockedChildRequest`, and both P5 annotation members. A bidirectional compile-only exact-kind fence and a table-driven runtime `assertNever` regression send all nineteen variants through the same `WorkspaceWriterLockCapability.spawnChild`; the P3 locked-child focused gates remain in the P5 release set. +- [ ] Every annotation publish/verify uses that sole cumulative `WorkspaceLockedChildRequest` union and `writerCapability.spawnChild`, validates actual FD 3 plus retained-root FD 4 before reads/writes, enforces fixed stdin/result caps, and has real barrier replacement, missing/substituted/cross-root, direct-spawn, and post-settlement tests. +- [ ] Every P5 Git stage uses shared `runFixedGitChild` with one monotonic operation deadline, bounded stdout/record/stderr, parser/cancel abort, owned-group TERM→KILL, and awaited exit/stdio cleanup. +- [ ] Resume refuses any later blob/revision/binding mismatch before schema/Evidence stages. +- [ ] `physical.yaml` remains local and absent from Git/export. +- [ ] Clean-state goal passes once without retry; its exact retained report output yields a validated 32-hex `RUN_ID`, every retained path use is quoted, and the report is hash-bound. +- [ ] Secret scan and exact owned-resource cleanup pass. +- [ ] Manual P5 guide is runnable and still PENDING until human decision. +- [ ] No P6 Evidence materialization, retention, or aggregate scope was implemented. diff --git a/docs/superpowers/plans/2026-08-10-p6-evidence-materialization.md b/docs/superpowers/plans/2026-08-10-p6-evidence-materialization.md new file mode 100644 index 00000000..9e7bb95d --- /dev/null +++ b/docs/superpowers/plans/2026-08-10-p6-evidence-materialization.md @@ -0,0 +1,1624 @@ +# P6 Commit-Addressed Evidence Materialization Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Materialize one workspace's filesystem Evidence tree safely from its pinned Git commit, run the existing P2 preprocessing workflow against that immutable tree, reclaim only content that no authoritative pin retains, and close the P2–P6 two-installation acceptance gates. + +**Architecture:** P6 extends P5's fixed Git-child protocol and selected-image `ProtectedWorkspaceFs`; it does not add a checkout, archive extractor, second operator, second lock, or second path model. A repository-owned boundary holds the repository lock, resolves and inventories the exact Git tree, then takes the P2 writer lock in repository → writer order to publish or verify a dirfd-anchored immutable tree. It releases both locks before `WorkspacePreprocessingService.execute` later enters a writer-only one-element ordered callback, rereads the active revision and manifest/tree identity, acquires the unchanged P3 maintenance lease, and invokes the existing harness pipeline. Every reuse is anchored again to the Git tree OID. Registry snapshot and Evidence retention consume one authoritative set built from active commits, validated revision leases, a complete session-manifest scan, and nonterminal preprocessing runs. + +**Tech Stack:** Node.js 22, TypeScript 5, P5 `runFixedGitChild`, Git fixed plumbing, P5 `ProtectedWorkspaceFs`, Python 3.12 internal protected-filesystem helper, `renameat2(RENAME_NOREPLACE)`, Go 1.26.5 `thothctl`, Docker Compose v2, Qdrant 1.18.2, internal Ollama `qwen3-embedding:0.6b`, Vitest, pytest, Ruff, Bash/Node acceptance tooling. + +**Source:** `docs/prd/2026-08-09-workspace-preprocessing-prd.md` D6/RF5 and `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` §§8–10. + +**Planning status:** DESIGN/PLAN ONLY until P1–P5 are implemented and accepted and the reviewer explicitly authorizes P6 execution. + +--- + +## Scope, dependency gate, and frozen handoff + +P6 begins only from a clean worktree whose `PROJECT_STATE.md` records automated and manual PASS for P1–P5. It consumes these exact flat P2/P3/P5 surfaces: + +- `backend/src/workspace-maintenance.ts::main`; +- `backend/src/workspaces/preprocessing-service.ts::WorkspacePreprocessingService.execute`; +- `backend/src/workspaces/preprocessing-state.ts::{PreprocessingStateStore,runUnderOrderedWorkspaceWriterLocks,runUnderWorkspaceWriterLock,probeWorkspaceWriterLock,WorkspaceLockedChildRequest,BorrowedWorkspaceSessionReadersExclusiveLockLease}`; +- `backend/src/workspaces/runtime-config-lease.ts::WorkspaceRuntimeConfigLeaseFactory`; +- `backend/src/workspaces/revision-layout.ts::{WorkspaceLayout,workspaceRuntimePaths,readRevisionLayoutState}`; +- `tools/thothctl/internal/workspaceops::{ParseWorkspaceCommand,Run}`; +- P5 `backend/src/workspaces/fixed-git-child.ts::runFixedGitChild`; +- P5 `backend/src/workspaces/protected-workspace-fs.ts::{ProtectedWorkspaceFs,openProtectedWorkspaceFs}` and `harness/tht/protected_fs_helper.py`; +- P5 `AnnotationSynchronizer.verifyPrepared` / `readRuntime` and its adapter typed as exact P2 `CapabilityAwareRegistryPublicationParticipant`; P2 `RegistryPullAddressedPlanV1`, `AddressedWorkspacePublicationLeaseV1`, `OrderedWorkspaceWriterCapabilitySet`, `CapabilityAwareRegistryPublicationLifecycleOwner`, `RegistryAddressedRequestV1`, `RegistryPullAddressedResultV1`, and `WorkspaceRegistry.publishAddressed`; P3 `RegistryPullPublicJobRequestV1`, `RegistryPullPhaseV1`, `RegistryPullAddressedJobRequestV1`, `RegistryPullParticipantStateV1`, `RegistryPullSynchronizerStateV1`, and `RegistryPullJobStateV1`; P3's explicit `workspace registry pull --workspace --json` author workflow and exact `registry_pull` capability; and P3's released non-activating `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json` / `migrate_dwh_cache` transition whose fixed `p3_migrate_dwh_cache`/`p3_prepare_dwh_cache` plus `p3_materialize_dwh_snapshot` stages publish and strictly reverify the selected unready revision's binding-qualified physical/LSH snapshot without READY. + +P6 must not rename or wrap those APIs. It must not modify any P3 production file. In particular, `revision-layout.ts`, `runtime-config-lease.ts`, `runtime-renderer.ts`, `tht-runner.ts`, and all P3 harness path/corpus/Qdrant/Memory production modules are read-only dependencies in P6. P6 may add cross-layer regression tests; a failing P3 contract stops P6, reopens P3, reruns P3's focused and clean-state acceptance, checkpoints P3, and then rebases P6. It is forbidden to commit a P3 repair as P6 work. + +### Exact released P2/P3 addressed registry-pull exception + +P6 dependency gates must import and consume the released names verbatim. The six P3 job names have one owning module; +production consumers import them only as follows (tests use the corresponding +`../src/workspaces/registry-pull-job.js` path): + +```ts +import type { + RegistryPullPublicJobRequestV1, + RegistryPullPhaseV1, + RegistryPullAddressedJobRequestV1, + RegistryPullParticipantStateV1, + RegistryPullSynchronizerStateV1, + RegistryPullJobStateV1, +} from "./registry-pull-job.js"; +``` + +`backend/src/workspaces/registry-pull-job.ts` exports all six names. It defines +`RegistryPullPhaseV1 = RegistryAddressedPublicationPhaseV1`, +`RegistryPullAddressedJobRequestV1 = Extract`, and +`RegistryPullJobStateV1 = RegistryPullAddressedPublicationStateV1`; these are exact aliases owned by +P3, not names P6 may redeclare. `RegistryPullPublicJobRequestV1`, +`RegistryPullParticipantStateV1`, and `RegistryPullSynchronizerStateV1` are the other three exact P3 +exports. Use both P3's existing `backend/test/registry-pull-job-imports.compile.ts` and P5's +`backend/test/p5-registry-pull-imports.compile.ts` as compile-only gates. Preserve the bidirectional +alias parity assertions and P5's exact field fence: it imports +`AddressedWorkspacePublicationLeaseV1` from `registry-publication.js` and +`BorrowedWorkspaceSessionReadersExclusiveLockLease` from `preprocessing-state.js`, proves their exact +bidirectional parity through `AddressedWorkspacePublicationLeaseV1["readers"]`, proves all three +identities are required on pull create/resume, `expectedBaseCommit` is required on pull create, +installation/repository/remote identity is required in plan/state, and `publishAddressed` takes the +exact addressed union. No barrel, compatibility export, optional-field overload, local alias, or second +job shape is permitted. P3's +production `registry.publishAddressed(...)` call and mismatch-before-mutation tests must already satisfy +that fence; otherwise stop and reopen P3 rather than repairing or decorating the call in P6. + +P6 consumes these exact P2 exports from `backend/src/workspaces/registry-publication.ts` and +`backend/src/workspaces/preprocessing-state.ts`. `AddressedWorkspacePublicationLeaseV1` is owned by +`registry-publication.ts`, and its exact `readers` field type +`BorrowedWorkspaceSessionReadersExclusiveLockLease` is owned by `preprocessing-state.ts`: + +```ts +export type RegistryAddressedPublicationPhaseV1 = + | "request_claimed" | "target_advertised" | "target_fetched" | "planned" + | "participants_prepared" | "publication_intent_durable" + | "target_published" | "terminal_durable"; +interface RegistryAddressedPlanFieldsV1 { + readonly schemaVersion: 1; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + readonly jobArtifactPath: RegistryAddressedJobArtifactPathV1; + readonly advertisedTargetCommit: Revision40; + readonly immutableTargetRef: `refs/thoth/addressed-runs/${RegistryRunId32}/target`; + readonly fetchedTargetCommit: Revision40; + readonly targetCommit: Revision40; + readonly targetManifestSha256: Sha256Hex; + readonly targetWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[]; + readonly changedWorkspaceIds: readonly CanonicalWorkspaceId[]; + readonly changedSetSha256: Sha256Hex; +} +export interface RegistryPullAddressedPlanV1 extends RegistryAddressedPlanFieldsV1 { + readonly operation: "registry_pull"; + readonly changedSetRule: "symmetric_base_target_workspace_difference"; + readonly baseCommit: Revision40; + readonly baseManifestSha256: Sha256Hex; + readonly baseWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[]; +} +export type RegistryAddressedPlanV1 = + | RegistryBootstrapAddressedPlanV1 | RegistryPullAddressedPlanV1; +interface RegistryAddressedPublicationStateFieldsV1 { + readonly schemaVersion: 1; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly jobArtifactPath: RegistryAddressedJobArtifactPathV1; + readonly phase: RegistryAddressedPublicationPhaseV1; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + readonly advertisedTargetCommit: Revision40 | null; + readonly immutableTargetRef: `refs/thoth/addressed-runs/${RegistryRunId32}/target` | null; + readonly fetchedTargetCommit: Revision40 | null; + readonly targetCommit: Revision40 | null; + readonly targetManifestSha256: Sha256Hex | null; + readonly targetWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[] | null; + readonly changedWorkspaceIds: readonly CanonicalWorkspaceId[] | null; + readonly planSha256: Sha256Hex | null; + readonly changedSetSha256: Sha256Hex | null; + readonly participantsSha256: Sha256Hex | null; + readonly synchronizersSha256: Sha256Hex | null; + readonly publicationIntentSha256: Sha256Hex | null; + readonly publishedActiveStateSha256: Sha256Hex | null; + readonly terminalResultSha256: Sha256Hex | null; + readonly priorStateSha256: Sha256Hex | null; +} +export interface RegistryPullAddressedPublicationStateV1 + extends RegistryAddressedPublicationStateFieldsV1 { + readonly operation: "registry_pull"; + readonly baseCommit: Revision40; + readonly baseManifestSha256: Sha256Hex; + readonly baseWorkspaces: readonly RegistryWorkspaceManifestIdentityV1[]; + readonly changedSetRule: "symmetric_base_target_workspace_difference" | null; +} +export interface AddressedWorkspacePublicationLeaseV1 + extends BorrowedOrderedWorkspaceWriterLeaseV1 { + readonly quiescence: BorrowedWorkspaceMaintenanceQuiescenceLease; + readonly readers: BorrowedWorkspaceSessionReadersExclusiveLockLease; +} +export interface CapabilityAwareRegistryPublicationParticipant { + readonly participantId: string; + prepare(plan: RegistryAddressedPlanV1, + workspace: AddressedWorkspacePublicationLeaseV1): Promise; + reconcile(plan: RegistryAddressedPlanV1, + workspace: AddressedWorkspacePublicationLeaseV1, prepared: T, + phase: RegistryAddressedPublicationPhaseV1): Promise; +} +export class CapabilityAwareRegistryPublicationLifecycleOwner { + run(input: { + readonly plan: RegistryAddressedPlanV1; + readonly capabilities: OrderedWorkspaceWriterCapabilitySet; + readonly participants: readonly CapabilityAwareRegistryPublicationParticipant[]; + readonly synchronizers: readonly CapabilityAwareRegistryPublicationSynchronizer[]; + readonly action: () => Promise; + }): Promise; +} +export type RegistryAddressedRequestV1 = + | { + readonly mode: "create"; + readonly operation: "registry_bootstrap"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly expectedBaseCommit: null; + readonly remoteRefIdentitySha256: Sha256Hex; + } + | { + readonly mode: "resume"; + readonly operation: "registry_bootstrap"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + } + | { + readonly mode: "create"; + readonly operation: "registry_pull"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly expectedBaseCommit: Revision40; + readonly remoteRefIdentitySha256: Sha256Hex; + } + | { + readonly mode: "resume"; + readonly operation: "registry_pull"; + readonly runId: RegistryRunId32; + readonly requestSha256: Sha256Hex; + readonly installationIdentitySha256: Sha256Hex; + readonly repositoryIdentitySha256: Sha256Hex; + readonly remoteRefIdentitySha256: Sha256Hex; + }; +export class WorkspaceRegistry { + publishAddressed(request: RegistryAddressedRequestV1): Promise; +} +``` + +This is a field-for-field dependency quotation, not a compatibility sketch. The validated +installation/registry boundary derives the production `installationIdentitySha256`, +`repositoryIdentitySha256`, and `remoteRefIdentitySha256` and supplies all three on every create and +resume member; pull create additionally supplies the exact `expectedBaseCommit`. The same three +identities are required, non-optional fields in the immutable plan and durable state, and same-ID +resume revalidates them before network or state mutation. `P6` adds no optional identity, partial +request, constructor decoration, compatibility overload, or second `publishAddressed` signature. + +The production registry constructor, not the request, owns the single +`VerifiedWorkspaceLockRootLeaseFactory`, `CapabilityAwareRegistryPublicationLifecycleOwner`, +participant list, and synchronizer list. The sole pull order is repository lock → durable +`request_claimed` → durable target advertisement → exact-OID fetch to the create-only immutable ref → +`target_fetched` → immutable pull plan → `planned` → acquire/provision every complete changed root → +one `runUnderOrderedWorkspaceWriterLocks` callback → lexical quiescence/readers → participant prepare +→ `participants_prepared` → durable publication intent → active-state publication/reconciliation → +`target_published` → `terminal_durable` → callback settlement invalidates and reverse-closes +readers/quiescence/writers/roots → repository lock release. The artifact is exactly +`addressed-publication-jobs/.json`. + +P6 preserves P5's annotation participant and calls only `WorkspaceRegistry.publishAddressed` with the `registry_pull` member. The annotation participant is +called by the existing lifecycle owner with the same callback-scoped +`AddressedWorkspacePublicationLeaseV1`; it consumes that lease's opaque `rootLease` and +`writerCapability`, never reacquires either, and never calls a repository method. The complete ordered +capability set stays live through active-pointer sibling write, file fsync, rename, parent fsync, +byte verification, and terminal durability. Same-ID recovery uses only +`RegistryPullJobStateV1`: it never refetches/reselects after the durable target pin, accepts only the +recorded all-base/all-target/mixed identities, reconciles a lost publication acknowledgement without a +second rename, and refuses a third identity or target drift. A new process may reacquire the recorded +complete set once; no live attempt permits nested acquisition or participant reentry. + +Freeze these declarations exactly once in the plan and implement them in `backend/src/workspaces/types.ts` during Task 2: + +```ts +export interface RepositoryMaterializationInput { + workspaceId: string; + expectedRevision: string; + signal: AbortSignal; + deadline: number; // caller-owned monotonic absolute deadline +} + +export interface PreparedEvidenceMaterialization { + workspaceId: string; // active identity revalidated under writer + workspaceRevision: string; // exact expected/active 40-hex revision + descriptorBlob: string; // exact active descriptor blob identity + descriptorSnapshotPath: string; // exact protected snapshot reopened under writer + contentRoot: string; // verified immutable revisions//content root + sourceRoot: string; // verified filesystem Evidence root rendered to the harness + sourceTreeOid: string; // Git anchor re-resolved during materialize/reuse + manifestSha256: string; // exact validated materialization.json bytes + entryCount: number; + totalBytes: number; +} +``` + +`PreparedEvidenceMaterialization` is a closed identity handoff, not a bag of optional metadata: Task 7 must re-read `active.json` plus `descriptorSnapshotPath`, require workspace/revision/descriptor equality, and fully revalidate `contentRoot`, `manifestSha256`, `sourceTreeOid`, and `sourceRoot` before rendering or child spawn. Counts must also equal the revalidated manifest. Do not redefine either type in `evidence-git-tree.ts`, `registry.ts`, `evidence-materializer.ts`, or the service; import and use these names throughout. + +Global lock order is exactly: + +```text +repository lock → workspace writer lock → existing harness stage lock +``` + +A repository-owned materialization callback may enter the writer lock while it holds repository. Once the repository lock has been released, the preprocessing service may enter the writer lock alone and call only lock-free verification and the fixed harness child. No repository method, Git child, registry pull, or materializing `ensure` is available inside that later writer callback. Tests must reject the opposite edge structurally and with both deterministic race directions. + +P6 does not add a GUI/API maintenance endpoint, SSH runtime transport, PSD content, LLM/Pi preprocessing, archive extraction, checkout/worktree reads, arbitrary Git argv, user-selected destinations, or long-term policy GC. P4's released session-admission route is used only by the aggregate test to prove self-heal, with exact failed-admission cleanup described below. + +### Fixed published layout + +For workspace `alpha` and exact commit `<40hex>`: + +```text +/data/sessions/alpha/revisions/<40hex>/ +├── artifacts/ # P3/P5; never P6-deleted +├── indexes/ # P3; never P6-deleted +├── corpus/ # P3; never P6-deleted +├── .content-staging-<32hex>.owner-reserved.json +├── .content-staging-<32hex>.owner-bound.json # adjacent until publication recovery completes +└── content/ # one P6 immutable publication unit + ├── materialization.json + └── workspace-content/alpha/evidence/** +``` + +The two adjacent owner files exist only during a live/recovering publication. The reserved record is durable before staging-directory creation and binds the retained revision-directory identity, staging basename, target basename, workspace, revision, and a 128-bit nonce. Before any source file is created, the bound record is durably published and additionally binds the staging directory's device and inode. After no-replace rename, the same device/inode identifies `content/`; the records remain adjacent until parent fsync, full validation, and final owner-record unlink+parent fsync complete. A crash never depends on an owner file inside a directory that was just renamed. + +`materialization.json` is canonical UTF-8 JSON plus one newline, mode `0400`, with exact keys: + +```ts +export interface EvidenceMaterializationManifestV1 { + schema_version: 1; + workspace_id: string; + workspace_revision: string; + source_path: string; // workspace-content//evidence + source_tree_oid: string; // exact Git tree OID resolved at this commit + entry_count: number; + directory_count: number; // every created source directory, including prefix directories + total_bytes: number; + limits: EvidenceMaterializationLimits; + files: Array<{ + path: string; // repository-relative NFC path, bytewise sorted + mode: "100644" | "100755"; + git_oid: string; + size: number; + sha256: string; + }>; +} +``` + +No timestamp, host path, endpoint, credential path/value, Git stderr, exception text, or raw content enters the manifest or public result. Public failures use `evidence_materialization_unsafe`; safe success fields are workspace, revision, run, source tree OID, manifest SHA-256, counts, and completed stages. + +### Installation-local limits and exact inode accounting + +Defaults are: + +```ts +{ + maxEntries: 10_000, + maxTotalBytes: 1024 * 1024 * 1024, + maxPathBytes: 1024, + maxSegmentBytes: 255, + maxManifestBytes: 16 * 1024 * 1024, + minFreeBytesAfterPublish: 64 * 1024 * 1024, + minFreeInodesAfterPublish: 1024, +} +``` + +`directoryCount` includes `workspace-content`, the workspace-ID directory, `evidence`, and every nested directory not already present in the new staging tree. Peak new inodes are exactly: + +```text +entryCount ++ directoryCount ++ 1 staging root ++ 1 materialization.json ++ 1 reserved adjacent owner record ++ 1 bound adjacent owner record +``` + +Thus preflight requires `ffree >= entryCount + directoryCount + 4 + minFreeInodesAfterPublish`. Publication is a rename of the staging inode, not another content inode. Byte preflight includes total blob bytes, the bounded actual manifest estimate, both bounded canonical owner records, and the configured reserve. Missing, negative, or overflowing `statfs` values fail closed. Tests independently count actual created peak inodes and fail if this formula undercounts. + +### P6 operation capability and Git-child boundary + +P3 owns and releases the exact general capability literal `registry_pull`: it alone has registry RW plus exact validated Git transport, while P5 accept/export and every ordinary mutation are registry RO/no-Git. P6 adds one internal `filesystem-materialization` capability selected only for a validated filesystem descriptor during `preprocess evidence` or `preprocess run`: + +- it may mount the registry volume RW solely because the existing repository lock opens/creates `locks/repository.lock`; +- it mounts sessions RW, repository/object storage locally, and no Git HTTPS/SSH credential, agent socket, home credentials, DWH secret beyond the later operation's existing needs, Pi state, Docker socket, or arbitrary host path; +- its TypeScript dependency is a narrowed `EvidenceMaterializationRepository`, not the mutation-capable repository surface; +- it exposes only the repository lock plus fixed `ls-tree`, `rev-parse`/`cat-file -t`, and `cat-file --batch` reads at an already-active 40-hex commit; +- bootstrap, pull, fetch, checkout, reset, clean, update-ref, commit, push, config writes, registry publication, arbitrary Git argv, and shell execution are absent from the command union and fail before spawn; +- HTTP/S3 Evidence and every unrelated mutation keep the P2 registry RO/no-Git capability. + +Image/Compose/Go contract tests hash the repository state/ref/object/FETCH_HEAD files before and after materialization (excluding the owned lock file) and prove byte identity. Mutation-negative tests attempt every forbidden operation through public grammar, direct machine ingress, capability selection, and an injected Git argv seam and prove zero child/mutation. A RW mount is not permission to expose a generic write API. + +Every P6 Git child reuses P5 `runFixedGitChild`: shell-free fixed argv; one caller-supplied monotonic absolute operation deadline shared by tree resolution, inventory, and batch streaming; bounded stdout, record, stderr, and total bytes; exact stdin writes followed by close; abort on parser error, sink error, `AbortSignal`, deadline, or any bound; TERM then KILL only the owned process group; always await exit and all stdio settlement. No child output reaches public errors. P6 adds stage-specific tests for hang, stdout/record/stderr flood, early exit, parser abort, cancellation, sink abort, descendant survival attempts, and awaited teardown while locks are held. + +--- + +### Task 1: Enforce the accepted flat prerequisites and stop on dependency drift + +**Files:** +- Read: `PROJECT_STATE.md` +- Read: P2–P5 checkpoint reports and plans +- Read/verify unchanged: the exact production/test files named in the frozen handoff +- Read/verify unchanged: `backend/src/workspaces/registry-publication.ts` +- Read/verify unchanged: `backend/src/workspaces/registry-pull-job.ts` +- Read/verify unchanged: `backend/test/registry-pull-job-imports.compile.ts` +- Read/verify unchanged: `backend/test/p5-registry-pull-imports.compile.ts` +- Read/verify unchanged: `backend/src/workspaces/registry.ts` +- Read/verify unchanged: `backend/test/workspace-registry-addressed-publication.test.ts` +- Read/verify unchanged: `backend/test/workspace-registry-addressed-process.test.ts` +- Read/verify unchanged: `backend/test/workspace-maintenance.test.ts` +- Read/verify unchanged: P5 `backend/test/workspace-registry.test.ts` +- No production, test, or documentation modification; missing coverage reopens P2/P3/P5 + +- [ ] **Step 1: verify clean accepted prerequisites.** + +```bash +git status --short +git log -1 --format='%H %T' +rg -n 'P[1-5].*automated.*PASS|P[1-5].*manual.*PASS' PROJECT_STATE.md +test -f backend/src/workspace-maintenance.ts +test -f backend/src/workspaces/preprocessing-service.ts +test -f backend/src/workspaces/preprocessing-state.ts +test -f backend/src/workspaces/runtime-config-lease.ts +test -f backend/src/workspaces/revision-layout.ts +test -f backend/src/workspaces/registry-publication.ts +test -f backend/src/workspaces/registry-pull-job.ts +test -f backend/test/registry-pull-job-imports.compile.ts +test -f backend/test/p5-registry-pull-imports.compile.ts +test -f backend/src/workspaces/registry.ts +test -f backend/test/workspace-registry-addressed-publication.test.ts +test -f backend/test/workspace-registry-addressed-process.test.ts +test -f backend/test/workspace-maintenance.test.ts +test -f backend/test/workspace-registry.test.ts +test -f backend/src/workspaces/fixed-git-child.ts +test -f backend/src/workspaces/protected-workspace-fs.ts +test -f harness/tht/protected_fs_helper.py +test -f tools/thothctl/internal/workspaceops/operations.go +``` + +Expected: clean status; one commit/tree; accepted P1–P5; every exact file exists. A pending manual gate or missing file stops P6. + +- [ ] **Step 2: fail on superseded aliases rather than renaming around them.** + +Run this exact negative inventory. The split literals deliberately construct disallowed legacy names without teaching later plan steps to use them. + +```bash +bad=( + 'backend/src/workspace-maintenance''/' + 'WorkspaceMaintenance''Operator' + 'tools/thothctl/internal/workspace''/' + 'workspace-effective-config''.test.ts' + 'runtime''Paths(' + 'RegistryPullBa''seTargetPlanV1' + 'RegistryAddressedWo''rkspaceLeaseOwner' + 'acquireCo''mpleteSet' + 'releaseCo''mpleteSet' + 'pullAndPubl''ishAddressed' + 'registry-''pull-jobs' + 'VerifiedWo''rkspaceRoot' + 'BorrowedWorkspaceExclusive''ReaderLease' +) +for needle in "${bad[@]}"; do + if rg -nF "$needle" backend tools/thothctl harness; then + echo "superseded P2/P3 surface present" >&2 + exit 1 + fi +done +``` + +Expected: no matches. Any match reopens the owning P2/P3 checkpoint; P6 does not add a conditional rename or compatibility wrapper. + +- [ ] **Step 3: run the exact registry-pull exception dependency matrix before P6 work.** + +The accepted P2/P3/P5 test files above must already execute the real one-shot and import the exact +released names quoted in this plan. In particular, P5's compile-only fence must directly import +`AddressedWorkspacePublicationLeaseV1` from `registry-publication.js` and +`BorrowedWorkspaceSessionReadersExclusiveLockLease` from `preprocessing-state.js`, and its +bidirectional `Equal` assertion must pin `AddressedWorkspacePublicationLeaseV1["readers"]` exactly to +that owner type. Coverage is a gate, not prose: if any assertion below or this corrected type fence is +absent, stop P6 and reopen the owning P2/P3/P5 checkpoint rather than adding a P6 alias or production +repair. + +1. Build base/target manifests whose complete symmetric difference contains at least two workspace + IDs presented in reverse order, including an addition/removal. Capture the real addressed create and + resume objects and prove each carries required `installationIdentitySha256`, + `repositoryIdentitySha256`, and `remoteRefIdentitySha256` equal to the validated production registry + identities; create also carries the exact active `expectedBaseCommit`. Assert the immutable + `RegistryPullAddressedPlanV1` and persisted `RegistryPullJobStateV1` retain all three identities, + `RegistryPullJobStateV1.changedWorkspaceIds` is unique/complete/strict lexical, and + `changedSetSha256`, `planSha256`, every ordered `RegistryPullParticipantStateV1`, and every + `AddressedWorkspacePublicationLeaseV1.workspaceId` match exact canonical bytes/order. +2. Assert the live trace is exactly `repository.lock` → durable `request_claimed` → durable + `target_advertised` → exact-OID immutable-ref fetch → `target_fetched` → durable `planned` → one + `runUnderOrderedWorkspaceWriterLocks` callback → lexical root/writer set → lexical + quiescence/readers set → exact `CapabilityAwareRegistryPublicationParticipant.prepare` calls → + `participants_prepared` → `publication_intent_durable` → active-pointer sibling write/file fsync/ + atomic rename/parent fsync/target-byte verification while the entire + `OrderedWorkspaceWriterCapabilitySet` remains held → `target_published` → `terminal_durable` → + ordered callback settlement and reverse close → repository release. Hold every changed-ID writer + from an independent process and prove contention persists through rename+parent fsync. Reject any + participant repository call, root/reader acquisition, `runUnderWorkspaceWriterLock`, + `publishAddressed`, nested/duplicate acquisition, post-settlement use, or cross-workspace + capability use. +3. With `core` absent, SIGKILL a real create at persisted `publication_intent_durable` immediately + before active-pointer rename. Exact same `runId` + `requestSha256` resume must load the recorded + `RegistryPullJobStateV1`, observe exact base, do no fetch/ls-remote/remote resolution/target + selection, reacquire the recorded complete lexical set once in the new process, and converge to + target. +4. In a fresh fixture, SIGKILL immediately after active-pointer rename+parent fsync but before the + `target_published` rewrite, leaving durable phase `publication_intent_durable` and exact target + active. Same-ID resume must treat all-target as lost acknowledgement, do no fetch and no second + active publication rename, advance `target_published` → `terminal_durable`, owner-clear, reverse + release, then release repository. Assert one fetch and one active publication rename total. +5. Inject a third active revision/descriptor/manifest identity, changed/deleted pinned target object, + target inventory/manifest/digest drift, installation identity drift, repository identity drift, + remote-ref identity drift, moved local remote-tracking OID, different ID, same ID/different digest, + and cross-operation resume. Each must refuse before network or state mutation and without refetch, + participant reentry, or nonterminal owner clear. + +```bash +(cd backend && npx vitest run test/workspace-registry-addressed-publication.test.ts test/workspace-registry-addressed-process.test.ts test/workspace-registry.test.ts test/workspace-runtime-config-lease.test.ts test/workspace-preprocessing-state.test.ts test/workspace-preprocessing-service.test.ts test/workspace-maintenance.test.ts test/workspace-effective-config-equivalence.test.ts test/workspace-revision-layout.test.ts test/fixed-git-child.test.ts test/protected-workspace-fs.test.ts test/workspace-annotations.test.ts && npx tsc --noEmit -p .) +(cd backend && npx tsc --noEmit --target ES2022 --module ES2022 \ + --moduleResolution Bundler --strict --skipLibCheck \ + test/registry-pull-job-imports.compile.ts \ + test/p5-registry-pull-imports.compile.ts) +(cd harness && .venv/bin/pytest -q tests/test_effective_dwh_binding.py tests/test_dwh_snapshot.py tests/test_corpus_pipeline.py tests/test_qdrant_vector_store.py tests/test_semantic_kind_isolation.py tests/test_protected_fs_helper.py) +(cd tools/thothctl && test "$(go env GOVERSION)" = 'go1.26.5' && go test ./internal/workspaceops ./cmd/thothctl -run 'RegistryPull|Resume|Capability|EvidenceMaterialization|RegistryMutation' -count=1) +``` + +Expected: PASS with both same-ID SIGKILL cases, exact strict-lexical multi-workspace set/order, +pre/post-publication recovery, drift/third-identity refusal, zero resume refetch/republication, and zero +participant/capability reentry. Record counts and the P2/P3/P5 checkpoint hashes. No commit. + +--- + +### Task 2: Add bounded materialization configuration and the least-capability container path + +**Files:** +- Modify: `backend/src/workspaces/types.ts` +- Modify: `backend/src/config.ts` +- Modify: `backend/test/config.test.ts` +- Modify: `backend/src/workspace-maintenance.ts` +- Modify: `backend/test/workspace-maintenance.test.ts` +- Modify: `tools/thothctl/internal/workspaceops/operations.go` +- Modify: `tools/thothctl/internal/workspaceops/operations_test.go` +- Modify: `tools/thothctl/internal/config/installation.go` +- Modify: `tools/thothctl/internal/config/installation_test.go` +- Modify: `compose.yaml` +- Modify: `deploy/compose.local.yaml` +- Modify: `deploy/compose.server.yaml` +- Modify: `scripts/generate-connector-secrets-override.sh` +- Modify: `scripts/test-compose-secret-policy.sh` +- Modify: `scripts/test-deployment-command-contract.sh` +- Modify: `scripts/test-unified-compose.sh` +- Modify: `deploy/env/local.env.example` +- Modify: `deploy/env/server.env.example` + +- [ ] **Step 1 (RED): add typed limit and cross-field tests.** + +Add `EvidenceMaterializationLimits` to `workspaces/types.ts` and tests for the exact defaults above, every environment override, safe-integer parsing, zero/negative/fraction/overflow rejection, `maxSegmentBytes <= maxPathBytes`, and a manifest limit large enough for the fixed envelope. In that same named file, implement the single frozen `RepositoryMaterializationInput` and `PreparedEvidenceMaterialization` declarations from the handoff above before this task's TypeScript gate; their field schema is not repeated here. Add compile-time tests that require every field and reject optional/extra/anonymous substitute shapes. Add `evidence_materialization_unsafe` to the existing closed result/error union in the flat P2 service/entrypoint; preserve decoding of historical `evidence_materialization_required` but do not emit it after Task 7. + +```bash +(cd backend && npx vitest run test/config.test.ts test/workspace-maintenance.test.ts \ + -t 'Evidence materialization|materialization unsafe') +``` + +Expected: RED because limits/code are absent. + +- [ ] **Step 2 (RED): freeze operation capability selection and mutation-negative cases.** + +In Go/Compose tests, require: + +1. validated filesystem `preprocess evidence` and `preprocess run` select the closed internal `filesystem-materialization` capability and pass expected workspace/revision/source identities; +2. HTTP/S3 and all unrelated commands retain registry RO/no-Git; +3. P3 `registry_pull` remains the only capability with Git transport and general repository mutation; +4. materialization receives registry RW, sessions RW, no Git transport secret, and the exact selected immutable image; +5. unknown capability, raw command, extra argv, environment-selected argv, bootstrap/pull/fetch/checkout/reset/clean/update-ref/commit/push/config request, and descriptor type drift are refused before a child starts; +6. direct calls to `main` cannot manufacture a capability token or repository mutator; +7. registry refs, object inventory, checkout, active state, snapshots, and `FETCH_HEAD` hash identically before/after positive and negative materialization; only the exact lock file may be created/touched; +8. container cancellation uses the existing bounded process-group cleanup and removes only its labelled one-shot container. + +```bash +(cd tools/thothctl && go test ./internal/workspaceops ./internal/config ./cmd/thothctl \ + -run 'EvidenceMaterialization|Capability|RegistryMutation' -count=1) +./scripts/test-compose-secret-policy.sh +./scripts/test-deployment-command-contract.sh +``` + +Expected: RED on the new capability. + +- [ ] **Step 3 (GREEN): implement the minimum capability.** + +Parse the seven non-secret limits into one immutable config object. Extend `workspaceops.Run`, not a parallel dispatcher. It may select materialization only after strict read-only inspection returns a filesystem source and exact active identity; the selected maintenance ingress repeats those expected fields and the service revalidates them. Generated Compose uses the selected image ID with `pull_policy: never`, fixed entrypoint, and the operation-specific mount/secret set above. Do not mount an author clone or give the narrowed repository interface a mutation method. + +- [ ] **Step 4: verify.** + +```bash +(cd backend && npx vitest run test/config.test.ts test/workspace-maintenance.test.ts && \ + npx tsc --noEmit -p .) +(cd tools/thothctl && gofmt -w internal/workspaceops/operations.go \ + internal/workspaceops/operations_test.go internal/config/installation.go \ + internal/config/installation_test.go && go test ./... -count=1) +docker compose --env-file deploy/env/local.env.example \ + -f compose.yaml -f deploy/compose.local.yaml config --quiet +./scripts/test-compose-secret-policy.sh +./scripts/test-deployment-command-contract.sh +./scripts/test-unified-compose.sh +``` + +Expected: PASS; rendered output exposes no secret value. + +- [ ] **Step 5: commit.** + +```bash +git add backend/src/workspaces/types.ts backend/src/config.ts backend/test/config.test.ts \ + backend/src/workspace-maintenance.ts backend/test/workspace-maintenance.test.ts \ + tools/thothctl/internal/workspaceops/operations.go \ + tools/thothctl/internal/workspaceops/operations_test.go \ + tools/thothctl/internal/config/installation.go \ + tools/thothctl/internal/config/installation_test.go \ + compose.yaml deploy/compose.local.yaml deploy/compose.server.yaml \ + scripts/generate-connector-secrets-override.sh scripts/test-compose-secret-policy.sh \ + scripts/test-deployment-command-contract.sh scripts/test-unified-compose.sh \ + deploy/env/local.env.example deploy/env/server.env.example +git commit -m 'feat: isolate filesystem materialization capability' +``` + +--- + +### Task 3: Inventory the exact pinned Evidence tree with the shared bounded Git protocol + +**Files:** +- Create: `backend/src/workspaces/evidence-git-tree.ts` +- Create: `backend/test/workspace-evidence-git-tree.test.ts` +- Modify: `backend/src/workspaces/git-repository.ts` +- Modify: `backend/test/workspaces-git-repository.test.ts` +- Reuse unchanged: `backend/src/workspaces/fixed-git-child.ts` +- Modify only to add P6 lifecycle regressions, not semantics: `backend/test/fixed-git-child.test.ts` + +**Frozen interfaces:** + +```ts +export interface EvidenceGitEntry { + mode: "100644" | "100755"; + type: "blob"; + oid: string; + size: number; + path: string; +} +export interface EvidenceGitInventory { + revision: string; + sourcePath: string; + treeOid: string; + entries: readonly EvidenceGitEntry[]; + entryCount: number; + directoryCount: number; + totalBytes: number; + estimatedManifestBytes: number; +} +export interface EvidenceMaterializationRepository { + materializeEvidenceForActiveRevision(input: RepositoryMaterializationInput): + Promise; +} +``` + +Import `RepositoryMaterializationInput` and `PreparedEvidenceMaterialization` from `backend/src/workspaces/types.ts`, where Task 2 defined them before the first implementation `tsc`; do not restate their fields here or create local aliases. The narrowed interface exposes no generic repository or Git call. + +- [ ] **Step 1 (RED): write byte parser and real-Git tests.** + +Test chunk splits at every byte, multiple NUL records, empty tree, invalid UTF-8, NUL/control characters, absolute/`.`/`..`/double/trailing separators, backslashes, non-NFC, cross-workspace prefixes, duplicate normalized paths, path/segment/count/total/manifest overflow, unsafe/nondecimal sizes, and duplicate OID size disagreement. Real Git tests cover regular/executable blobs; symlink at every depth; gitlink; injected special/unknown modes/types; per-object limit before body read; shell/pathspec strings; exact historical commit; and unrelated valid workspace namespaces that coexist unchanged. + +Tree identity is resolved on **every** new publication and reuse from exact `:workspace-content//evidence` with fixed read plumbing, required type `tree`, and compared to `source_tree_oid`. Inventory uses only: + +```text +git -c core.hooksPath= ls-tree -r -z --full-tree --long \ + -- :(literal)workspace-content//evidence +``` + +The root lookup uses fixed `rev-parse --verify` plus fixed `cat-file -t`; no revision/path is accepted from arbitrary caller text. + +```bash +(cd backend && npx vitest run \ + test/workspace-evidence-git-tree.test.ts \ + test/workspaces-git-repository.test.ts \ + test/fixed-git-child.test.ts -t 'Evidence|reuse anchor|deadline|abort') +``` + +Expected: RED because the reader is absent. + +- [ ] **Step 2 (GREEN): implement bounded inventory.** + +Use `runFixedGitChild` for every stage with one absolute deadline. Parse `MODE SP TYPE SP OID SP SIZE TAB PATH NUL` as bytes; bound the unterminated record by `maxPathBytes + 256`; decode fatal UTF-8; require NFC, exact POSIX normalization and source prefix; accept only blob modes `100644|100755`, 40-hex OIDs, and safe sizes. Sort by UTF-8 bytes, reject duplicates, calculate every prefix directory, and conservatively calculate manifest bytes before any blob body starts. Empty Evidence tree is valid. All failures map to one safe error without path/OID/stderr/remote. + +- [ ] **Step 3 (RED then GREEN): stream batch objects with complete teardown.** + +Use one fixed `cat-file --batch`. Send one expected OID only when the prior blob has been consumed; close stdin after the final OID. Require exact header, size, delimiter, SHA-1 Git object identity, backpressure, and sink close. Never buffer a full blob. Add hang, stderr flood, stdout/record overflow, early EOF/exit, parser throw, sink abort, cancellation, deadline, grandchild, and held-lock teardown tests. Each asserts stdin closure, TERM→KILL of only the owned group, awaited exit/stdio, no live descendant, and lock release only after settlement. + +- [ ] **Step 4: verify and commit.** + +```bash +(cd backend && npx vitest run \ + test/workspace-evidence-git-tree.test.ts \ + test/workspaces-git-repository.test.ts test/fixed-git-child.test.ts && \ + npx tsc --noEmit -p .) +git add backend/src/workspaces/evidence-git-tree.ts \ + backend/src/workspaces/git-repository.ts \ + backend/test/workspace-evidence-git-tree.test.ts \ + backend/test/workspaces-git-repository.test.ts backend/test/fixed-git-child.test.ts +git commit -m 'feat: inventory pinned Evidence Git trees' +``` + +--- + +### Task 4: Extend P5's protected filesystem with crash-safe immutable tree publication + +**Files:** +- Modify: `backend/src/workspaces/protected-workspace-fs.ts` +- Modify: `backend/test/protected-workspace-fs.test.ts` +- Modify: `backend/src/workspaces/preprocessing-state.ts` (cumulatively extend the sole locked-child union/dispatcher without removing P3 or P5 members) +- Modify: `backend/test/workspace-locked-child-cumulative.compile.ts` (exact compile-time base-three + P3-fourteen + P5-two + P6-two union fence) +- Modify: `backend/test/workspace-preprocessing-state.test.ts` (P3 locked-child focused cumulative runtime dispatcher regression) +- Modify: `backend/test/fixtures/workspace-lock-root-worker.mjs` +- Modify: `harness/tht/locked_child_stdin.py` +- Modify: `harness/tests/test_locked_child_stdin.py` +- Modify: `harness/tht/protected_fs_helper.py` +- Modify: `harness/tests/test_protected_fs_helper.py` +- Create: `backend/src/workspaces/evidence-materializer.ts` +- Create: `backend/test/workspace-evidence-materializer.test.ts` + +P6 extends P5's same capability-bound interface and sole cumulative +`WorkspaceLockedChildRequest` union. It retains the original three P2 members, the exact fourteen-member +`P3LockedChildRequest`, and P5's exact `AnnotationPublishLockedChildRequest` kind +`"annotation_publish"` plus `AnnotationVerifyLockedChildRequest` kind `"annotation_verify"`, then adds +only the two Evidence-tree members. It does not copy/redeclare a prior alias, create another filesystem +abstraction, or add any path/direct-spawn fallback: + +```ts +export const EVIDENCE_TREE_ENTRIES_MAX = 10_000; +export const EVIDENCE_TREE_TOTAL_BYTES_MAX = 1_073_741_824; +export const EVIDENCE_TREE_MANIFEST_BYTES_MAX = 16_777_216; +export const EVIDENCE_TREE_CHILD_STDIN_MAX = 1_107_296_256; +export interface EvidenceTreePublishLockedChildRequest { + readonly kind: "evidence_tree_publish"; + readonly workspaceId: CanonicalWorkspaceId; + readonly revision: Revision40; + readonly rootIdentity: WorkspaceLockRootIdentityV1; + readonly childRunId: string; + readonly sourceTreeOid: Revision40; + readonly inventory: VerifiedEvidenceTreeInventoryV1; + readonly blobSource: BoundedEvidenceBlobSource; + readonly limits: EvidenceMaterializationLimitsV1; +} +export interface EvidenceTreeVerifyLockedChildRequest { + readonly kind: "evidence_tree_verify"; + readonly workspaceId: CanonicalWorkspaceId; + readonly revision: Revision40; + readonly rootIdentity: WorkspaceLockRootIdentityV1; + readonly childRunId: string; + readonly sourceTreeOid: Revision40; + readonly inventorySha256: Sha256Hex; + readonly manifestSha256: Sha256Hex; + readonly limits: EvidenceMaterializationLimitsV1; +} +export type WorkspaceLockedChildRequest = + | DwhLockedChildRequest | SchemaLockedChildRequest | EvidenceLockedChildRequest + | P3LockedChildRequest + | AnnotationPublishLockedChildRequest | AnnotationVerifyLockedChildRequest + | EvidenceTreePublishLockedChildRequest | EvidenceTreeVerifyLockedChildRequest; +export interface ProtectedWorkspaceFs { + publishImmutablePair(input: ImmutablePairPublication): Promise; + verifyImmutablePair(input: ImmutablePairVerification): Promise; + publishImmutableTree(input: ImmutableTreePublication): Promise; + verifyImmutableTree(input: ImmutableTreeVerification): Promise; +} +export function openProtectedWorkspaceFs(input: { + readonly rootLease: BorrowedVerifiedWorkspaceLockRootLease; + readonly writerCapability: WorkspaceWriterLockCapability; +}): ProtectedWorkspaceFs; +``` + +`VerifiedEvidenceTreeInventoryV1` and `BoundedEvidenceBlobSource` have private constructors and are +created only after the fixed Git inventory/deadline/limit checks; they are semantic bounded inputs, not +raw argv, stdio, callbacks, workspace roots, or destination paths. The capability dispatcher fixes the +selected-image executable and exact argv `-I -m tht.protected_fs_helper`. It rejects more than 10,000 +entries, more than 1,073,741,824 blob bytes, more than 16,777,216 manifest bytes, any inventory/path +limit violation, or more than 1,107,296,256 total framed stdin bytes before/while streaming. It reuses +P5's exact 16,384-byte stdout, 65,536-byte stderr, 81,920-byte combined result caps, fixed deadline, +owned-group termination, and awaited exit/stdio settlement. No P6 code calls `spawn`, `exec`, or +`child_process` for the helper; every publish and verify call is +`writerCapability.spawnChild(exactVariant)`. + +The helper validates inherited writer FD 3 and retained-root FD 4 with P2's shared verifier before any +workspace read or write, then derives the immutable revision content destination only from the branded +workspace/revision request beneath FD 4. All opens use `O_NOFOLLOW`; directory creation, owner evidence, cleanup, publication, and +verification are anchored `openat`/`mkdirat`/`unlinkat` plus +`renameat2(RENAME_NOREPLACE)` with no-follow and retained identity rechecks. There is no ambient root, +absolute-root frame, pathname reopen, Node rename, overwrite fallback, or direct helper spawn. + +The P5 compile-only fence is extended rather than replaced: its separate exact fourteen-P3 and +two-annotation readonly tuples remain unchanged, a separate two-Evidence-tree tuple is added, +duplicates are rejected, and bidirectional `Equal`/`Assert` checks prove +`WorkspaceLockedChildRequest["kind"]` is exactly the original three P2 kinds plus the cumulative +14 + 2 + 2 extension kinds. The runtime table likewise retains the original three, all fourteen P3, +and both P5 annotation requests, adds only the two tree requests, and submits all twenty-one through +one real `WorkspaceWriterLockCapability`. The fixture must observe one capability/root identity and +one exhaustive production dispatcher; a test-side switch uses `assertNever`, and a second capability, +P3 spawner, annotation spawner, or tree spawner is forbidden. Preserve P5's compile-file declarations +and extend its expected type exactly as follows: + +```ts +const p6LockedKinds = ["evidence_tree_publish", "evidence_tree_verify"] as const + satisfies readonly WorkspaceLockedChildRequest["kind"][]; +const p6CumulativeLockedKinds = [...p5CumulativeLockedKinds, ...p6LockedKinds] as const; +type P6CumulativeLockedKind = typeof p6CumulativeLockedKinds[number]; +type _P6CumulativeKindsAreUnique = Assert>; +type _P6CumulativeUnionIsExact = Assert< + Equal +>; +``` + +- [ ] **Step 1 (RED): specify selected-image framed tree publication.** + +Use real temporary filesystems and invoke the installed helper positively only through `WorkspaceWriterLockCapability.spawnChild`. Test exact layout; modes (`0700` directories, `0400` files); UID/nlink/type; one-byte chunks; bounded memory; actual SHA-256; deterministic manifest; and exact inode accounting. Test real ancestor replacement after inherited-FD-4 validation, revision open, staging creation, every directory/file open, every file fsync, every directory fsync, manifest fsync, owner reservation/binding, no-replace rename, revision-parent fsync, validation, owner unlink, and final fsync. The helper retains accepted dirfds and must either finish in the retained inode or fail without writing into a replacement tree. Add a real parent/child barrier after the parent acquires the writer and installs FD 3/FD 4 but before the helper validates/uses FD 4; replace the canonical workspace leaf, then release the child. Require zero bytes in the replacement tree. Separately run the real child with missing FD 3, missing FD 4, substituted/cross-root FD pairs, wrong request root identity, direct spawn, and use of the saved capability after the ordered callback settles; all must fail before read, owner-record creation, staging, or publication. + +Publication uses only retained-dirfd `openat`/`mkdirat`/`unlinkat` and the existing `renameat2(RENAME_NOREPLACE)` binding. There is no Node rename, overwrite-capable fallback, recursive path API, or lstat-then-normal-path operation. A competing `content` created immediately before rename must win with `EEXIST`; P6 verifies it as immutable reuse or fails closed and never overwrites it. + +```bash +(cd harness && .venv/bin/pytest -q tests/test_protected_fs_helper.py \ + tests/test_locked_child_stdin.py tests/test_layout_marker_commands.py \ + tests/test_p3_internal_cli.py) +(cd backend && npx vitest run test/protected-workspace-fs.test.ts \ + test/workspace-preprocessing-state.test.ts test/workspace-evidence-materializer.test.ts \ + -t 'tree|ancestor|noreplace|inode|cumulative locked child') +(cd backend && npx tsc --noEmit --target ES2022 --module ES2022 \ + --moduleResolution Bundler --strict --skipLibCheck \ + test/workspace-locked-child-cumulative.compile.ts) +``` + +Expected: the retained P3/P5 locked-child cases remain green, while the cumulative runtime table, +14 + 2 + 2 compile-only fence, `evidence_tree_publish`, `evidence_tree_verify`, and materializer cases +fail for only the missing P6 surface. + +- [ ] **Step 2 (RED): freeze adjacent owner evidence and kill recovery.** + +Test the exact state machine: + +1. fsync reserved adjacent record before staging mkdir; +2. create/fstat an empty staging directory; +3. fsync bound adjacent record with staging dev/inode before creating source entries; +4. stream/write/fsync all files; create and fsync all directories bottom-up; +5. write/fsync manifest inside staging; fsync staging; +6. recheck retained parent and staging identities; +7. `renameat2(..., RENAME_NOREPLACE)` staging → `content`; +8. fsync retained revision parent; +9. fully validate `content`, including same dev/inode as bound owner; +10. unlink both exact owner records and fsync parent. + +Kill a real helper at every boundary. Recovery rules are closed: reserved/no staging removes only the matching reservation; reserved plus an empty exact-name/UID/mode staging may remove that empty directory; once bound exists, cleanup/recovery acts only when nonce/name/parent/dev/inode all match; post-rename recovery validates the same inode at `content`, finishes fsync/validation, and removes evidence; neither/multiple/mismatched objects fail closed without deletion. Add owner-record replacement, hardlink, symlink, foreign UID, nonce, dev/inode, and nonempty-unbound staging negatives. + +- [ ] **Step 3 (GREEN): implement streaming and validation.** + +`ProtectedWorkspaceFs.publishImmutableTree` constructs the exact `evidence_tree_publish` semantic request and calls only `writerCapability.spawnChild`; `verifyImmutableTree` similarly uses `evidence_tree_verify`. The capability dispatcher alone starts the fixed selected-image `python` with argv `-I -m tht.protected_fs_helper`, owns canonical inventory/blob framing, applies the exact stdin/result/deadline caps above, and tears down/awaits the owned process group. The Python helper first validates FD 3 and FD 4, then owns all anchored tree syscalls/FDs. It computes file SHA-256, verifies exact byte counts, serializes the canonical manifest, rejects actual manifest overflow, and never echoes bytes, paths, or exceptions. + +`EvidenceMaterializer.validatePublished` calls `ProtectedWorkspaceFs.verifyImmutableTree` inside the same borrowed-root/writer-capability lifetime, strictly parses the manifest, compares inventory/tree OID and limits, streams/hashes every declared local file, verifies modes/UID/nlink/size, validates every directory and absence of undeclared entries, and returns: + +```ts +export interface VerifiedRevisionContent { + workspaceId: string; + workspaceRevision: string; + root: string; + sourceRoot: string; + sourceTreeOid: string; + manifestSha256: string; + entryCount: number; + totalBytes: number; +} +``` + +A suspect final root is never overwritten, repaired, or deleted. + +- [ ] **Step 4: run kill, race, and focused gates.** + +```bash +(cd harness && .venv/bin/pytest -q tests/test_protected_fs_helper.py \ + tests/test_locked_child_stdin.py tests/test_layout_marker_commands.py \ + tests/test_p3_internal_cli.py && \ + .venv/bin/ruff check tht/protected_fs_helper.py tht/locked_child_stdin.py \ + tests/test_protected_fs_helper.py tests/test_locked_child_stdin.py \ + tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py) +(cd backend && npx vitest run test/protected-workspace-fs.test.ts \ + test/workspace-preprocessing-state.test.ts test/workspace-evidence-materializer.test.ts && \ + npx tsc --noEmit --target ES2022 --module ES2022 --moduleResolution Bundler \ + --strict --skipLibCheck test/workspace-locked-child-cumulative.compile.ts && \ + npx tsc --noEmit -p .) +``` + +Expected: every kill point either recovers the owned inode or reports safe residue; no replacement-tree +write, overwrite, or foreign deletion occurs. The P3 locked-child focused gates pass, the exact +base-three + 14 + 2 + 2 kind fence compiles, and all twenty-one requests traverse the one exhaustive +capability dispatcher. + +- [ ] **Step 5: commit.** + +```bash +git add backend/src/workspaces/protected-workspace-fs.ts \ + backend/test/protected-workspace-fs.test.ts \ + backend/src/workspaces/preprocessing-state.ts \ + backend/test/workspace-locked-child-cumulative.compile.ts \ + backend/test/workspace-preprocessing-state.test.ts \ + backend/test/fixtures/workspace-lock-root-worker.mjs \ + backend/src/workspaces/evidence-materializer.ts \ + backend/test/workspace-evidence-materializer.test.ts \ + harness/tht/locked_child_stdin.py harness/tests/test_locked_child_stdin.py \ + harness/tht/protected_fs_helper.py harness/tests/test_protected_fs_helper.py +git commit -m 'feat: publish immutable Evidence trees safely' +``` + +--- + +### Task 5: Add repository-owned materialize/reuse with Git anchoring and deadlock barriers + +**Files:** +- Modify: `backend/src/workspaces/git-repository.ts` +- Modify: `backend/src/workspaces/registry.ts` +- Modify: `backend/src/workspaces/registry-factory.ts` +- Modify: `backend/src/workspaces/evidence-materializer.ts` +- Modify: `backend/test/workspaces-git-repository.test.ts` +- Modify: `backend/test/workspace-registry.test.ts` +- Modify: `backend/test/workspace-registry-factory.test.ts` +- Modify: `backend/test/workspace-evidence-materializer.test.ts` + +- [ ] **Step 1 (RED): define the single lock-owning boundary.** + +Add one method to the narrowed materialization view, implemented by `WorkspaceRegistry`, using the Task 2 named types without an anonymous restatement: + +```ts +materializeEvidenceForActiveRevision( + input: RepositoryMaterializationInput, +): Promise; +``` + +It performs exactly: + +```text +repository acquire +→ reread operational active descriptor/snapshot and expected identity +→ fixed Git root-tree OID resolution + exact inventory +→ writer acquire while repository remains held +→ publish new tree OR validate existing tree against Git OID+inventory and stream/hash local files +→ writer release +→ repository release +``` + +The caller cannot supply a repo path, destination, source path, Git argv, writer callback, or repository mutator. The source is derived from the validated descriptor only. + +Tests assert the normal trace and two races. Race A: materialization holds repository before writer while another normal writer waits; it completes. Race B: preprocessing holds a writer-only one-element ordered callback after its materialization; a pull holds repository and waits for writer; preprocessing performs no repository call, completes, then pull advances. Instrumented types fail immediately on any writer → repository edge, not merely on timeout. + +- [ ] **Step 2 (RED): require a Git anchor on every reuse.** + +A second invocation must spawn fixed tree lookup/inventory Git reads even if local content and manifest appear valid. It compares exact root OID and every inventory entry to the manifest, then streams/hashes the local tree. Blob bodies need not be reread from Git. Git/object unavailability, tree OID mismatch, inventory mismatch, local tamper, unsigned reconstructed manifest, extra/missing file, link/type/mode/UID drift, or changed limit contract fails closed. Assert no mtime/write/blob-body read on valid reuse, but assert Git tree lookup occurred exactly once under the shared deadline. + +Include an attack that replaces local files and rewrites the manifest with matching new SHA-256 values while retaining a claimed old `source_tree_oid`; the fixed Git inventory/OID comparison must reject it. + +- [ ] **Step 3 (GREEN): implement and keep mutation APIs structurally absent.** + +`registry-factory.ts` constructs the narrowed interface only for the P6 capability. The implementation imports and uses `RepositoryMaterializationInput` and returns the one named `PreparedEvidenceMaterialization` from `workspaces/types.ts`; no anonymous input/result shape or duplicate declaration is permitted. Populate every prepared field only after the repository-held active descriptor/snapshot identity, Git tree anchor, manifest, immutable content root, source root, counts, and local bytes have all validated. The implementation uses P5 `runFixedGitChild`, P5 `ProtectedWorkspaceFs`, the installation-bound P2 root factory, and a one-element `runUnderOrderedWorkspaceWriterLocks` callback. Inside `forWorkspace` it passes the exact borrowed `rootLease` plus matching `writerCapability` to `openProtectedWorkspaceFs`; it never passes a root path or spawns directly. No exported callback allows a service to call arbitrary code under repository lock. No Git or repository object escapes in `PreparedEvidenceMaterialization`. + +- [ ] **Step 4: run focused gates and capability mutation negatives.** + +```bash +(cd backend && npx vitest run \ + test/fixed-git-child.test.ts test/workspaces-git-repository.test.ts \ + test/workspace-evidence-git-tree.test.ts test/protected-workspace-fs.test.ts \ + test/workspace-evidence-materializer.test.ts test/workspace-registry.test.ts \ + test/workspace-registry-factory.test.ts -t 'material|reuse|lock order|mutation' && \ + npx tsc --noEmit -p .) +(cd tools/thothctl && go test ./internal/workspaceops -run \ + 'EvidenceMaterialization|RegistryMutation' -count=1) +``` + +Expected: PASS; valid reuse has Git metadata reads and zero blob-body/local writes; every mutation attempt has zero spawned mutator. + +- [ ] **Step 5: commit.** + +```bash +git add backend/src/workspaces/git-repository.ts backend/src/workspaces/registry.ts \ + backend/src/workspaces/registry-factory.ts backend/src/workspaces/evidence-materializer.ts \ + backend/test/workspaces-git-repository.test.ts backend/test/workspace-registry.test.ts \ + backend/test/workspace-registry-factory.test.ts \ + backend/test/workspace-evidence-materializer.test.ts +git commit -m 'feat: materialize Evidence under repository writer order' +``` + +--- + +### Task 6: Reconcile snapshots and Evidence from one authoritative retention API + +**Files:** +- Modify: `backend/src/workspaces/registry.ts` +- Modify: `backend/src/workspaces/preprocessing-state.ts` +- Create: `backend/src/workspaces/session-manifest-pins.ts` +- Create: `backend/test/workspace-session-manifest-pins.test.ts` +- Create: `backend/src/workspaces/evidence-retention.ts` +- Create: `backend/test/workspace-evidence-retention.test.ts` +- Modify: `backend/test/workspace-registry.test.ts` +- Modify: `backend/test/workspace-preprocessing-state.test.ts` +- Modify: `backend/src/routes/sessions.ts` +- Modify: `backend/test/routes-sessions.test.ts` + +The only public retention source is: + +```ts +export interface AuthoritativeRevisionRetention { + workspaceId: string; + revisions: readonly string[]; + reasons: Readonly>; +} +export class WorkspaceRegistry { + authoritativeRevisionRetention(workspaceId: string): Promise; + reconcileAuthoritativeRetention(workspaceId: string): Promise; +} +``` + +- [ ] **Step 1 (RED): prove complete pin enumeration and fail-closed behavior.** + +Tests require the union of: + +1. the active operational commit for the workspace; +2. every strict, persisted registry revision-lease record (`creating` or `persisted`) whose workspace/commit/token/state/file identity validates; +3. every nonterminal/resumable session manifest pin from a complete direct scan of the workspace's canonical `sessions/` root, independent of principal, HTTP pagination, RLS, or a caller-supplied list; +4. every strict nonterminal P2/P5/P6 job revision from `PreprocessingStateStore`'s canonical `jobs/` directory. + +Manifest and job scans are bounded, direct-child only where the persistence schema requires, no-follow, regular single-link, size-bounded, strict schema/identity/status, and total-count bounded. An unsafe/unknown pin file fails retention without deletion. Finalized/archived nonresumable sessions and terminal runs do not pin; tests use the exact final schemas rather than heuristic status strings. + +Explicitly prove that physical snapshot/content directories, a partial `/sessions?scope=mine` result, a caller list, and a removed user's invisible list are never pin sources. Delete the last logical pin while leaving the physical descriptor snapshot and prove the revision becomes reclaimable. + +```bash +(cd backend && npx vitest run \ + test/workspace-session-manifest-pins.test.ts \ + test/workspace-evidence-retention.test.ts \ + test/workspace-registry.test.ts test/workspace-preprocessing-state.test.ts \ + test/routes-sessions.test.ts -t 'authoritative retention|manifest pin|partial list') +``` + +Expected: RED because the authoritative API does not exist. + +- [ ] **Step 2 (GREEN): implement one repository-owned reconciliation.** + +Under repository → writer order, compute and freeze the authoritative set once; reconcile descriptor snapshots from exactly that set first; then inspect only direct `revisions/<40hex>/content` targets and remove an unpinned target only if its complete manifest/owner/type/UID/nlink/tree validates as P6-owned. Never remove a revision directory, `artifacts`, `indexes`, `corpus`, Memory, sessions, run state, a suspect content root, another workspace, or an owned root whose pin scan was incomplete. Interrupted deletion is durable/idempotent and cannot broaden its target. + +Remove the session route's retention side effect based on a list response. Routes may request authoritative reconciliation after a mutation, but cannot provide revisions/principal views. Tests prove route behavior and RLS listings remain unchanged. + +- [ ] **Step 3: run retention and lock-order gates.** + +```bash +(cd backend && npx vitest run \ + test/workspace-session-manifest-pins.test.ts \ + test/workspace-evidence-retention.test.ts test/workspace-registry.test.ts \ + test/workspace-preprocessing-state.test.ts test/routes-sessions.test.ts && \ + npx tsc --noEmit -p .) +``` + +Expected: PASS, including active/lease/complete-manifest/nonterminal-run pins and physical-directory non-pinning. + +- [ ] **Step 4: commit.** + +```bash +git add backend/src/workspaces/registry.ts backend/src/workspaces/preprocessing-state.ts \ + backend/src/workspaces/session-manifest-pins.ts \ + backend/src/workspaces/evidence-retention.ts backend/src/routes/sessions.ts \ + backend/test/workspace-session-manifest-pins.test.ts \ + backend/test/workspace-evidence-retention.test.ts backend/test/workspace-registry.test.ts \ + backend/test/workspace-preprocessing-state.test.ts backend/test/routes-sessions.test.ts +git commit -m 'feat: reconcile authoritative revision retention' +``` + +--- + +### Task 7: Run the flat P2 service against verified filesystem Evidence without changing P3 + +**Files:** +- Modify: `backend/src/workspaces/preprocessing-service.ts` +- Modify: `backend/src/workspaces/preprocessing-state.ts` +- Modify: `backend/src/workspace-maintenance.ts` +- Modify: `backend/src/workspaces/registry-factory.ts` +- Modify: `backend/test/workspace-preprocessing-service.test.ts` +- Modify: `backend/test/workspace-preprocessing-state.test.ts` +- Modify: `backend/test/workspace-maintenance.test.ts` +- Modify: `backend/test/workspace-runtime-config-lease.test.ts` (test only) +- Modify: `tools/thothctl/internal/workspaceops/operations_test.go` +- Modify: `tools/thothctl/cmd/thothctl/main_test.go` +- Read unchanged: `backend/src/workspaces/runtime-config-lease.ts` +- Read unchanged: `backend/src/workspaces/revision-layout.ts` +- Read unchanged: `backend/src/workspaces/runtime-renderer.ts` +- Read unchanged: `backend/src/tht/tht-runner.ts` + +- [ ] **Step 1 (RED): freeze the two-lock-phase orchestration.** + +For `preprocess-evidence` and the Evidence stage of `preprocess-run`, require: + +```text +repository-owned materializeEvidenceForActiveRevision( + input: RepositoryMaterializationInput +) → PreparedEvidenceMaterialization + (repository → writer, Git tree anchor, publish/verify, both released) +→ writer-only one-element ordered callback (`forWorkspace` supplies rootLease + writerCapability) +→ reread exact active revision and protected descriptor snapshot without repository/Git call +→ require workspaceId/workspaceRevision/descriptorBlob equal the named prepared result +→ fully revalidate contentRoot + manifestSha256 + sourceTreeOid + sourceRoot + counts/local bytes + against that same PreparedEvidenceMaterialization +→ reread exact nonterminal run state and P5 accepted annotation identity +→ WorkspaceRuntimeConfigLeaseFactory.acquireMaintenance (unchanged P3 API) +→ require rendered filesystem root equals prepared.sourceRoot +→ fixed existing tht Evidence/full-run child +→ persist evidence_materialized before child and evidence_preprocessed only after success +→ writer release +→ repository-owned authoritative retention as a later separate operation +``` + +A pull that advances before writer reacquisition returns `preprocessing_conflict`; it is not retried or silently rebound. A pull waiting behind the later writer completes only after the child, because the child never asks for repository. Add deterministic traces for both directions and a compile-time fake whose repository methods throw if captured by the writer callback. + +Tests cover dry-run (materialize/verify but no corpus/Qdrant publication), full publish, resume, cancel, child failure, unsafe materialization before child, prior corpus ACTIVE preservation, pristine JSON, and HTTP/S3 byte-for-byte regression. The public Go grammar is unchanged. + +```bash +(cd backend && npx vitest run \ + test/workspace-preprocessing-service.test.ts \ + test/workspace-preprocessing-state.test.ts test/workspace-maintenance.test.ts \ + test/workspace-runtime-config-lease.test.ts -t 'filesystem|material|lock order') +``` + +Expected: RED because filesystem still stops at the P2 deferral. + +- [ ] **Step 2 (GREEN): extend only the P2 service/state/entrypoint.** + +`WorkspacePreprocessingService.execute` owns orchestration; `main` only parses/encodes. Its repository dependency and local variable use the Task 2 named `RepositoryMaterializationInput` and `PreparedEvidenceMaterialization` imports directly; do not introduce another interface, `Pick`, anonymous signature, or inferred partial result. Reuse the P3 lease and its already-derived `workspaceRuntimePaths`; do not add a path helper or edit P3 production. Under the later writer lock, reopen `prepared.descriptorSnapshotPath`, revalidate every prepared identity field named above, and require `prepared.sourceRoot` to equal the unchanged lease's rendered filesystem root before invoking the existing harness argv. Preserve P2's run ID, completed-stage, dry-run, resume, stdout/stderr, and child FD 3 contracts. Preserve P5 explicit acceptance; P6 cannot skip or infer it. + +- [ ] **Step 3: verify backend/Go and old-code decoding.** + +```bash +(cd backend && npx vitest run \ + test/workspace-preprocessing-service.test.ts test/workspace-preprocessing-state.test.ts \ + test/workspace-maintenance.test.ts test/workspace-runtime-config-lease.test.ts \ + test/workspace-revision-layout.test.ts test/workspace-runtime-renderer.test.ts \ + test/workspace-runtime-handoff.test.ts && npx tsc --noEmit -p . && npm run build) +(cd tools/thothctl && go test ./internal/workspaceops ./cmd/thothctl -count=1) +``` + +Expected: filesystem succeeds; historical result decoding still recognizes the old deferral code; no production path emits it. + +- [ ] **Step 4: commit.** + +```bash +git add backend/src/workspaces/preprocessing-service.ts \ + backend/src/workspaces/preprocessing-state.ts backend/src/workspace-maintenance.ts \ + backend/src/workspaces/registry-factory.ts \ + backend/test/workspace-preprocessing-service.test.ts \ + backend/test/workspace-preprocessing-state.test.ts backend/test/workspace-maintenance.test.ts \ + backend/test/workspace-runtime-config-lease.test.ts \ + tools/thothctl/internal/workspaceops/operations_test.go \ + tools/thothctl/cmd/thothctl/main_test.go +git commit -m 'feat: preprocess verified filesystem Evidence' +``` + +--- + +### Task 8: Add a cross-layer P3 regression test and stop rather than repairing P3 + +**Files:** +- Create: `harness/tests/test_registry_evidence_revision.py` +- Modify test only: `backend/test/workspace-preprocessing-service.test.ts` +- Do not modify any P3 production file + +- [ ] **Step 1: add the regression.** + +Build two configs for the same workspace/collection at revisions A/B with separate P3 `paths.corpus`. With fake embeddings and the Qdrant request recorder, prove: + +- A and B corpus ACTIVE/manifests stay in their exact roots; +- schema/Evidence point IDs and every list/search/existing-hash/upsert/delete filter include exact revision; +- identical document IDs/generations retrieve only their own revision; +- filesystem resolution never crosses corpus roots; +- Memory/solved registry, IDs, and filters remain workspace-global and do not fork; +- service materialized source commit, runtime identity, corpus manifest, and Qdrant payload/filter revision are equal. + +```bash +(cd harness && .venv/bin/pytest -q tests/test_registry_evidence_revision.py) +(cd backend && npx vitest run test/workspace-preprocessing-service.test.ts \ + -t 'materialized source corpus and Qdrant share one revision') +``` + +Expected after Tasks 1–7: PASS against finalized P3. If either exposes P3 production drift, stop P6. Amend P3 in a separate owning checkpoint, run P3 focused suites and `./scripts/p3-acceptance.sh integration --keep`, record the checkpoint, rebase, and rerun Task 1. Do not add P3 production files to a P6 commit. + +- [ ] **Step 2: run the P3 contract regression suite.** + +```bash +(cd harness && .venv/bin/pytest -q \ + tests/test_registry_evidence_revision.py tests/test_preprocess_cli.py \ + tests/test_corpus_pipeline.py tests/test_qdrant_vector_store.py \ + tests/test_semantic_kind_isolation.py tests/test_filesystem_evidence_source.py && \ + .venv/bin/ruff check tests/test_registry_evidence_revision.py) +``` + +Expected: PASS with no network. + +- [ ] **Step 3: commit tests only.** + +```bash +git add harness/tests/test_registry_evidence_revision.py \ + backend/test/workspace-preprocessing-service.test.ts +git commit -m 'test: bind filesystem Evidence to its revision' +``` + +--- + +### Task 9: Document the released Evidence path and bounded recovery model + +**Files:** +- Modify: `docs/contracts/workspace-evidence-v3.md` +- Modify: `docs/install/local-workspace-registry.md` +- Modify: `docs/install/server-workspace-registry.md` +- Modify: `docs/testing/p2-p6-manual-verification.md` +- Modify: `deploy/workspace-registry.env.example` +- Modify: `scripts/verify-workspace-install-docs.sh` +- Modify: `scripts/test-verify-workspace-install-docs.sh` + +- [ ] **Step 1 (RED): add documentation contract checks.** + +Require the fixed Git source and published layout; exact `thothctl ... preprocess evidence --dry-run --json`, real run, and full-run commands; the inherited P5 operator order `registry pull` → exact `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json` → strict selected-revision+binding physical/LSH snapshot verification without READY → `schema accept` → activation/READY; `migration_required` admission/resume at every pre-activation boundary; distinct exact-run-ID recovery for DWH preparation and activation; per-object vs aggregate/disk/inode limits; every symlink depth/gitlink/special/path rejection; adjacent owner recovery; every-reuse Git OID anchor; repository → writer order; operation-specific RW-lock/no-transport capability; authoritative active/lease/complete-manifest/nonterminal-run retention; revision-scoped corpus/schema/Evidence; workspace-global Memory/solved; safe inspection; and no checkout/archive/author-clone mount/generic Git mutation. + +```bash +bash scripts/test-verify-workspace-install-docs.sh +``` + +Expected: RED on deferred P2 wording. + +- [ ] **Step 2 (GREEN): update supported manuals.** + +Retain historical explanation of old P2 reports, but supported P6 execution no longer expects a deferral. Preserve the P5 install/recovery sequence in both local and server manuals: after every content pull and before accept, validate `WORKSPACE_ID`, run exact released DWH preparation, inspect the selected commit+binding physical/LSH snapshot while READY is absent, and prove admission remains `migration_required` until activation. Interrupted DWH preparation and activation each resume only with their own returned 32-hex outer run ID on the same command; never cross-use IDs or hand-edit snapshots/READY. Document Evidence recovery without instructing operators to edit owner/manifest files. Do not document P7/PSD/GUI/SSH. + +- [ ] **Step 3: verify and commit.** + +```bash +bash scripts/test-verify-workspace-install-docs.sh +bash scripts/verify-workspace-install-docs.sh --fixtures-only +git add docs/contracts/workspace-evidence-v3.md docs/install/local-workspace-registry.md \ + docs/install/server-workspace-registry.md docs/testing/p2-p6-manual-verification.md \ + deploy/workspace-registry.env.example scripts/verify-workspace-install-docs.sh \ + scripts/test-verify-workspace-install-docs.sh +git commit -m 'docs: explain pinned Evidence materialization' +``` + +--- + +### Task 10: Build the clean-state P6 process goal with full-root secret proof + +**Files:** +- Create: `scripts/p6-acceptance.sh` +- Create: `scripts/test-p6-acceptance.sh` +- Create: `backend/scripts/p6-acceptance.mjs` +- Create: `backend/scripts/p6-acceptance.test.mjs` + +**Public command:** + +```bash +./scripts/p6-acceptance.sh integration --keep +``` + +- [ ] **Step 1 (RED): test ownership, lifecycle, mutation coverage, and reports.** + +The root is `.artifacts/p6-integration/p6-<32hex>/`. Test exclusive/no-follow ownership, clean source commit/tree, unique Compose labels, no retry, one hard monotonic deadline, bounded children, TERM→KILL group cleanup, report schema/hashes, kill points, capability mutation negatives, authoritative retention, `--keep`, signal/failure cleanup, and exact foreign-resource refusal. The dependency portion must also execute the exact P2/P3 multi-workspace registry-pull matrix: strict-lexical complete `changedWorkspaceIds`; same-ID SIGKILL immediately before active-pointer rename and immediately after rename+parent fsync; exact `RegistryPullJobStateV1` phase/digest assertions; no resume refetch/republication or participant/capability reentry; full-set ownership through publication; reverse release before repository; and target-drift/third-identity refusal without mutation/owner clear. Mutation tests must fail if any required unsafe fixture, Git anchor, registry exception case, lock barrier, owner-evidence kill point, secret deletion, full-root scan, or report check is removed. + +```bash +bash scripts/test-p6-acceptance.sh +``` + +Expected: RED because the runner is absent. + +- [ ] **Step 2: implement the isolated production topology.** + +Use a real local bare remote/author clone, real selected core/maintenance image, real `thothctl`, production registry/materializer/service, controlled REST DWH, real Qdrant, real internal Ollama/model initialization, and fixture-only secrets. Host Node is only the bounded orchestrator/report writer; the product path requires no host Python/Node/Pi. Build once from clean HEAD. Record safe source/image/Git identities and exact ownership. + +- [ ] **Step 3: execute positive P6 once.** + +Through production commands, inspect, dry-run, publish, retrieve, run full preprocessing, and explicitly rerun. Prove exact commit/tree/blob identity; every-reuse Git OID lookup; no blob reread/local rewrite on valid reuse; manifest/file hashes; no staging/owner residue; corpus ACTIVE; Qdrant revision payload/filter/retrieval; idempotent counts and no duplicate points; historical pinned revision retention; release of the last pin followed by authoritative content-only reclamation; and preservation of artifacts/indexes/corpus/Memory. + +- [ ] **Step 4: execute each negative once, without retry.** + +Use independent commits/workspaces for the two registry-pull SIGKILL/resume cases, target drift/third identity, nested/root symlink, gitlink, parser-injected special mode, absolute/traversal/non-NFC/cross-workspace/duplicate path, per-file/aggregate/count/path/segment/manifest/disk/inode bounds, object header/OID drift, local manifest+file rewrite, Git unavailable on reuse, ancestor swap, target race, every owner/fsync/rename kill point, unsafe pin file, partial-user-list non-authority, foreign retention root, forbidden registry mutation, and bounded-child hang/flood/parser/cancel. The pull fixtures contain multiple reverse-presented changed IDs and use the exact P2/P3 request/job/lease-set/participant types; the pre-publication case resumes all-base without fetch, the post-rename+parent-fsync case resumes all-target without fetch or a second publication, and drift/third identity preserves nonterminal ownership. Each negative must yield a stable safe code, no harness child after unsafe materialization, no new corpus/Qdrant state, no overwrite, no foreign cleanup, and complete owned-child teardown. + +Required check IDs include: + +```text +clean_source ownership capability_confinement registry_mutation_negative +registry_pull_multiworkspace_lexical_set registry_pull_same_id_prepublication +registry_pull_same_id_postpublication registry_pull_target_drift_third_identity_refusal +registry_pull_no_refetch_reentry registry_pull_publication_release_order +pinned_inventory bounded_git_children streaming_preflight inode_accounting +adjacent_owner_recovery dirfd_ancestor_races noreplace_atomic_publish +manifest_revalidation git_anchored_reuse filesystem_dry_run filesystem_publish +full_run idempotent_rerun revision_corpus revision_qdrant revision_retrieval +authoritative_retention physical_snapshot_not_pin no_partial_publish +fixture_secrets_deleted full_retained_root_secret_scan exact_cleanup +``` + +- [ ] **Step 5: delete fixture secrets before report finalization and scan all retained bytes.** + +On success, failure, or signal: stop listeners/children/containers; export only safe hashes/identities needed by reports; delete every fixture secret file and secret directory; assert those paths are absent; then scan **every regular file under the retained run root with no directory exclusion**, every reachable fixture Git blob, and every declared report artifact for all secret canaries, credential-shaped URLs, signed query values, and private endpoints. Only after that scan passes may final `report.json`/`report.md` and artifact hashes be finalized. Run the complete-root scan again including the final reports. `--keep` retains sanitized evidence only, never secret files or live resources. + +- [ ] **Step 6: verify runner contracts and commit tooling.** + +```bash +bash -n scripts/p6-acceptance.sh scripts/test-p6-acceptance.sh +node --check backend/scripts/p6-acceptance.mjs +node --test backend/scripts/p6-acceptance.test.mjs +bash scripts/test-p6-acceptance.sh +git add scripts/p6-acceptance.sh scripts/test-p6-acceptance.sh \ + backend/scripts/p6-acceptance.mjs backend/scripts/p6-acceptance.test.mjs +git commit -m 'test: add P6 materialization acceptance' +``` + +Do not run the authoritative retained process until manual tooling is committed and the source is clean. + +--- + +### Task 11: Build the exact two-installation aggregate P2–P6 process + +**Files:** +- Create: `scripts/p2-p6-acceptance.sh` +- Create: `scripts/test-p2-p6-acceptance.sh` +- Create: `backend/scripts/p2-p6-acceptance.mjs` +- Create: `backend/scripts/p2-p6-acceptance.test.mjs` +- Modify: `scripts/preprocess-smoke.sh` (delegation/deprecation note only) + +**Public command:** + +```bash +./scripts/p2-p6-acceptance.sh integration --keep +``` + +- [ ] **Step 1 (RED): freeze two-installation ownership and exact choreography.** + +Use `.artifacts/p2-p6-integration/p2-p6-<32hex>/`, one shared bare remote/author clone, and two independent installations A/B with distinct descriptors, Compose projects, registry/sessions/Qdrant/Ollama volumes, fixture-secret roots, collections, run IDs, and reports. They may share the one immutable built image and Git remote only. Tests assert the following numbered production events occur exactly in order; no step is optional, preseeded, or replaced by a test seam. + +- [ ] **Step 2: bootstrap A and B at the same base commit C0.** + +Start both private stacks. With the released `workspace inspect --workspace --json`, bootstrap each empty registry against C0. Record equal C0 descriptor/tree/blob/content identities and distinct registry roots/installations. No curated annotation commit exists yet. + +- [ ] **Step 3: prove P2 DWH and P3 layout/Memory/effective state in both.** + +For A, then B, run the released P2 DWH command and exact P3 migrations: + +```text +workspace preprocess dwh --workspace --json +workspace migrate dwh-cache --workspace --json +workspace migrate memory --workspace --json +workspace migrate semantic-revision --workspace --yes --json +workspace migrate activate-revision-layout --workspace --yes --json +``` + +Prove maintenance/session effective-binding equivalence through the released inspect/runtime-snapshot evidence without starting Pi; same logical DWH binding/cache generation for equal installation inputs; revision-qualified artifacts/indexes/corpus; one workspace-global Memory registry/projection; and no mixed old/new layout. + +- [ ] **Step 4: self-heal and strict-index A and B while C0 is still active and READY.** + +Delete/omit each exact fixture collection while the accepted C0 revision layout remains READY. For A, then B, call the released core `POST /sessions` admission route with valid fixture auth/question. This is the production `ReadinessManager → ThtRunner.qdrantEnsure → QdrantCollectionManager.ensure` path, not direct manager invocation. Configure the controlled Ollama fixture to fail at its later readiness boundary only after the collection and keyword indexes have been independently read compatible. Require the admission response to fail at that later boundary and prove cleanup: no session manifest, persisted revision lease, live Pi process, admission lease, or resumable run survives. Restore normal fixture Ollama, inspect exact collection compatibility, and run the released strict schema index in A and B at C0. No hidden session/Pi is permitted. + +- [ ] **Step 5: exercise P4 guarded rebuild and deterministic interrupted recovery at READY C0, then reindex.** + +On A, run one fully confirmed production `workspace collection rebuild`; verify maintenance activation, complete session inventory, admissions/Pi quiescence, core stop, exact target deletion/recreation, neighbor preservation, core health, marker clear, and terminal state. Then arm the acceptance-owned P4 delete barrier, run another confirmed rebuild, kill only the labelled one-shot child after durable deletion, require core stopped/maintenance active/nonterminal state, release the barrier, and run the exact confirmed `workspace collection recover` once. Prove verified empty target and cleanup. Because rebuild/recovery intentionally discarded A's vectors, run the released strict schema index again in A before continuing. B remains independently indexed from Step 4 and is never rebuilt through A's project. Do not delete B's collection or repeat its missing-collection admission self-heal. + +- [ ] **Step 6: pause one real P5 run in both before any curated commit.** + +Invoke `workspace preprocess run --workspace --json` once in A and once in B. Both must stop at the real `manual_review_required` checkpoint with separate 32-hex run IDs and state-owned candidates. Use production `workspace schema export-fks --run --output --json` in both; rehash exact outputs; require equal candidate digest/count and prove no second FK suggestion occurred during export. + +- [ ] **Step 7: publish one curated content-only C1, then pull, prepare its DWH snapshot, accept, and activate C1 independently in A and B.** + +Edit one exported candidate in the ordinary author clone, validate it, create exactly one curated content-only Git commit C1, and push once. Freeze this exact production command order, using each installation's own `--installation`, workspace ID, and paused run ID: + +```text +# pull C1 independently into both installations through P3-owned registry_pull +workspace registry pull --workspace --json +workspace registry pull --workspace --json +# bind each installation's exact ID, then prepare C1's physical/LSH snapshot without READY +WORKSPACE_ID= +workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json +WORKSPACE_ID= +workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json +# P5 accept remains registry RO/no-Git and writer-only one-element ordered callbacks in each installation +workspace schema accept --run --yes --json +workspace schema accept --run --yes --json +# C1 remains unready after pull/preparation/accept; publish READY independently in both +workspace migrate activate-revision-layout --workspace --yes --json +workspace migrate activate-revision-layout --workspace --yes --json +``` + +Each pull uses P3's exact `RegistryAddressedRequestV1` / `RegistryPullJobStateV1` path and `registry_pull` capability. Before the response-loss matrix, prove both create and resume carry the validated installation's required installation/repository/remote-ref identities, create carries its exact expected base, and the plan/state preserve all three identities. Exercise the main C1 pulls as response-loss cases without creating another curated commit: for A, arm the owned barrier at durable `publication_intent_durable`, SIGKILL immediately before active-pointer rename, then resume only A's returned exact 32-hex ID; for B, SIGKILL immediately after active-pointer rename+parent fsync but before `target_published` persistence, then resume only B's returned exact ID. A must remain all-base until resume; B must be all-target despite its still-`publication_intent_durable` job. In both installations prove no resume fetch/ls-remote/target reselection, no participant lock/repository reentry, complete strict-lexical `changedWorkspaceIds`, matching `AddressedWorkspacePublicationLeaseV1` calls, full `OrderedWorkspaceWriterCapabilitySet` ownership through rename+parent fsync and terminal durability, reverse set release before repository release, and one active publication rename total. Record `registry_pull_same_id_before_publication_a` and `registry_pull_same_id_after_publication_b` only after exact field/digest/phase checks. Run target-drift and third-active-identity refusal in isolated copied fixtures so the one shared C1 and main A/B state remain unchanged; record `registry_pull_drift_third_identity_refusal_ab` only when each refusal performs no mutation/refetch/owner clear. + +After its recovered pull and before its accept, each installation must execute exact released `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json` with its own validated ID and outer run. For A and B separately, prove the `migrate_dwh_cache` operation remains registry RO/no-Git; its `p3_migrate_dwh_cache`/`p3_prepare_dwh_cache` cache stage plus `p3_materialize_dwh_snapshot` reuse or prepare C1's binding cache, publish C1's own `revisions/C1/dwh-snapshots//{artifacts,indexes}` physical/LSH snapshot, and strictly reverify all identities/digests without reading or publishing READY or falling back to C0's revision snapshot. Record `dwh_snapshot_c1_a` and `dwh_snapshot_c1_b` only after those strict checks. Each accept then uses registry RO/no-Git, a writer-only one-element ordered callback, and `AnnotationSynchronizer.verifyPrepared` through `writerCapability.spawnChild`. Assert C1 session admission and preprocessing resume fail `migration_required` after pull, after DWH preparation, and after accept in each installation; record `pre_ready_migration_required_c1_ab` only after all six A/B boundary refusal pairs pass. Record separate `c1_layout_activated_a` and `c1_layout_activated_b` only after the corresponding activation proves C1's exact `READY.json`; only then may admission/resume succeed. After activation, assert each installation resolves the same C1 annotation blob/digest while retaining distinct state/run IDs. Rerun DWH/effective inspection and prove C1 reuses C0's DWH cache generation/binding but never its revision snapshot, while C1 artifacts/corpus/schema/Evidence roots fork from C0 and Memory/solved remain global. + +- [ ] **Step 8: strict-index, resume, materialize, retrieve, and prove idempotency in A at READY C1.** + +Run A's released strict schema index only after `c1_layout_activated_a`. Resume the exact P5-accepted outer run ID through `workspace preprocess run --resume`; require that production resume to pass the P6 `materializeEvidenceForActiveRevision` boundary, materialize/index filesystem Evidence, and reach terminal success. Run revision-filtered retrieval, then one explicit idempotent materialize/full-run rerun. Prove Git tree OID/materialization/corpus/Qdrant revision equality, no duplicate points/generations, P5 accepted identity, exact DWH reuse, and authoritative retention. The rerun is an assertion, not a retry. + +- [ ] **Step 9: execute the same READY-C1 chain in B with fully separate semantic state.** + +Only after `c1_layout_activated_b`, run B's strict schema index, resume its exact accepted outer run, require filesystem Evidence materialization/indexing, run revision-filtered retrieval, and run one explicit idempotent rerun through its own maintenance service, sessions root, corpus, Qdrant collection/storage, and state. Stop all A services before one B retrieval. Require equal shared Git tree/blob/content hashes and logical DWH binding where inputs match, but distinct installation IDs, registry paths, manifests, run IDs, corpus ACTIVE files, Qdrant storage/points, Memory registries, secrets, and ownership. Neither report may contain the other's canary or path. + +- [ ] **Step 10: run aggregate negative/security cases and exact cleanup.** + +Cover incompatible Qdrant, unaccepted new annotation, revision resume mismatch, nested symlink/gitlink, aggregate/disk/inode limits, tampered local tree+manifest, Git-unavailable reuse, forbidden registry mutation, unsafe pin, stale foreign root, and cross-installation path/collection attempts. Preserve last valid state. Stop bounded children/listeners; remove only both labelled projects/resources; delete both fixture-secret trees; then perform a no-exclusion scan of the complete retained aggregate root before and after final report generation. Retain only safe hashes/identities/reports. + +Required aggregate check IDs separately cover A, B, and shared choreography, including: + +```text +bootstrap_ab_base p2_dwh_ab p3_layout_memory_effective_ab +production_admission_self_heal_c0_ab failed_admission_cleanup_c0_ab strict_index_c0_ab +p4_guarded_rebuild p4_interrupted_recovery reindex_a_after_recovery +p5_paused_ab one_curated_commit registry_pull_c1_a registry_pull_c1_b +registry_pull_same_id_before_publication_a registry_pull_same_id_after_publication_b +registry_pull_drift_third_identity_refusal_ab +dwh_snapshot_c1_a dwh_snapshot_c1_b pre_ready_migration_required_c1_ab +p5_accept_c1_a p5_accept_c1_b c1_layout_activated_a c1_layout_activated_b dwh_reuse_content_fork +strict_index_c1_a resume_materialize_retrieve_idempotent_a +strict_index_c1_b resume_materialize_retrieve_idempotent_b +cross_installation_isolation fixture_secrets_deleted full_retained_root_secret_scan exact_cleanup +``` + +- [ ] **Step 11: test and commit aggregate tooling.** + +```bash +bash -n scripts/p2-p6-acceptance.sh scripts/test-p2-p6-acceptance.sh +node --check backend/scripts/p2-p6-acceptance.mjs +node --test backend/scripts/p2-p6-acceptance.test.mjs +bash scripts/test-p2-p6-acceptance.sh +git add scripts/p2-p6-acceptance.sh scripts/test-p2-p6-acceptance.sh \ + backend/scripts/p2-p6-acceptance.mjs backend/scripts/p2-p6-acceptance.test.mjs \ + scripts/preprocess-smoke.sh +git commit -m 'test: add aggregate P2-P6 acceptance' +``` + +--- + +### Task 12: Make focused P6 and aggregate manual verification independently runnable + +**Files:** +- Create: `scripts/p6-manual-verification.sh` +- Create: `scripts/test-p6-manual-verification.sh` +- Create: `backend/scripts/p6-manual-verification.mjs` +- Create: `backend/scripts/p6-manual-verification.test.mjs` +- Modify: `docs/testing/p2-p6-manual-verification.md` +- Modify only after a reviewer verdict: `PROJECT_STATE.md` + +**Lifecycle:** + +```bash +./scripts/p6-manual-verification.sh prepare +./scripts/p6-manual-verification.sh start +./scripts/p6-manual-verification.sh stop +./scripts/p6-manual-verification.sh cleanup +``` + +- [ ] **Step 1 (RED): test helper ownership and reviewer separation.** + +Require fresh `.artifacts/manual-acceptance/p6/`, exact two-project ownership, symlink/foreign refusal, bounded start/readiness, no scenario retry, stop with TERM→KILL owned groups, cleanup refusal while live, no broad Docker/Git command, fixture-secret deletion before complete-root scan, and no helper-created verdict/PASS. Generated commands use real `thothctl` and bounded readers; they never print Compose environments, rendered credentials, or secret files. + +```bash +bash scripts/test-p6-manual-verification.sh +``` + +Expected: RED until tooling exists. + +- [ ] **Step 2: implement prepare/start/stop/cleanup only.** + +`prepare` creates a new shared remote/author tree, C0, unsafe independent commits, controlled DWH/Ollama, two installations, fixture secrets, command scripts, `GUIDE.md`, and ownership. It does not execute reviewer checks. `start` starts only owned fixture services/projects and bounded readiness. `stop` validates and stops exact resources, exports safe identities, deletes fixture secrets, and scans the entire retained root without exclusions before declaring stopped. `cleanup` requires stopped state and exact nonce/labels. + +- [ ] **Step 3: write the exact reviewer walkthrough.** + +The reviewer personally performs, in order: + +1. bootstrap A/B at C0; +2. complete P2 DWH and every P3 layout/Memory/effective-equivalence check in both so C0 is READY; +3. use production session admission to self-heal each missing C0 collection, fail only at later Ollama readiness, inspect complete failed-admission cleanup, and strict-index C0 in A/B; +4. run A's guarded rebuild, interruption, exact recovery, and post-recovery reindex without repeating B's missing-collection self-heal; +5. pause real P5 runs in both and export/re-hash both state-owned candidates; +6. create one curated C1; run exact P3 `workspace registry pull --workspace ` independently in A/B through the quoted P2/P3 exception: SIGKILL A at `publication_intent_durable` immediately before active-pointer rename and resume the same ID; SIGKILL B immediately after rename+parent fsync but before `target_published` persistence and resume the same ID. Inspect exact `RegistryPullJobStateV1` fields/phases, strict-lexical complete multi-workspace capability acquisition, no resume fetch/republication or participant reentry, full-set ownership through rename+fsync, reverse release before repository, and isolated target-drift/third-identity refusal. After each recovered pull set/validate that installation's `WORKSPACE_ID` and run exact released `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json`; inspect and record `dwh_snapshot_c1_a` / `dwh_snapshot_c1_b` only after C1's binding-qualified physical/LSH snapshot is published and strictly reverified without READY or C0 snapshot fallback; accept each exact paused run; then run `workspace migrate activate-revision-layout --workspace "$WORKSPACE_ID" --yes` independently and record `c1_layout_activated_a` / `c1_layout_activated_b`; +7. prove C1 DWH cache-generation reuse but distinct revision snapshot, revision artifact/corpus/schema/Evidence fork, global Memory/solved, and admission/resume `migration_required` after pull, after DWH preparation, and after accept until each C1 READY event; +8. after A's C1 activation only, strict-index, resume the accepted run through P6 materialization, retrieve, and prove idempotency with Git OID/manifest/corpus/Qdrant equality; +9. after B's C1 activation only, execute the same strict-index/resume/materialize/retrieve/idempotency chain and retrieve with A stopped; +10. exercise nested symlink, gitlink, limits, tamper+manifest rewrite, Git-unavailable reuse, retention, and capability-mutation negatives; +11. release an old pin and prove only its valid `content/` is reclaimed; +12. run generated bounded report/secret checks and `stop`; +13. write `VERDICT.md` with reviewer, UTC time, observed hashes, each item PASS/FAIL, and separate `P6 manual acceptance` and `aggregate P2-P6 manual acceptance` decisions. + +No helper writes or edits the verdict. + +- [ ] **Step 4: verify and commit manual tooling/docs.** + +```bash +bash -n scripts/p6-manual-verification.sh scripts/test-p6-manual-verification.sh +node --test backend/scripts/p6-manual-verification.test.mjs +bash scripts/test-p6-manual-verification.sh +bash scripts/test-verify-workspace-install-docs.sh +git add scripts/p6-manual-verification.sh scripts/test-p6-manual-verification.sh \ + backend/scripts/p6-manual-verification.mjs \ + backend/scripts/p6-manual-verification.test.mjs \ + docs/testing/p2-p6-manual-verification.md +git commit -m 'test: add P6 and aggregate manual walkthrough' +``` + +--- + +### Task 13: Run focused/full suites, retain final evidence, and stop at the manual gate + +**Files:** +- Modify after final green reports: `PROJECT_STATE.md` +- Read: retained P6 and aggregate reports +- Do not modify production in this task without a new focused RED test and owning-task repair + +- [ ] **Step 1: verify a clean committed source and no superseded surface.** + +```bash +git diff --check +git status --short --branch +git fsck --no-reflogs --full +bad=( + 'backend/src/workspace-maintenance''/' + 'WorkspaceMaintenance''Operator' + 'tools/thothctl/internal/workspace''/' + 'workspace-effective-config''.test.ts' + 'runtime''Paths(' +) +for needle in "${bad[@]}"; do ! rg -nF "$needle" backend tools/thothctl harness; done +``` + +Expected: clean source, Git fsck PASS, no superseded names. + +- [ ] **Step 2: run focused P5/P6 backend and harness suites.** + +```bash +(cd backend && npx vitest run \ + test/fixed-git-child.test.ts test/protected-workspace-fs.test.ts \ + test/workspace-evidence-git-tree.test.ts test/workspace-evidence-materializer.test.ts \ + test/workspace-evidence-retention.test.ts test/workspace-session-manifest-pins.test.ts \ + test/workspaces-git-repository.test.ts test/workspace-registry.test.ts \ + test/workspace-registry-factory.test.ts test/workspace-preprocessing-state.test.ts \ + test/workspace-preprocessing-service.test.ts test/workspace-maintenance.test.ts \ + test/workspace-runtime-config-lease.test.ts test/workspace-revision-layout.test.ts \ + test/workspace-runtime-renderer.test.ts test/workspace-runtime-handoff.test.ts \ + test/workspace-annotations.test.ts test/routes-sessions.test.ts && \ + npx tsc --noEmit --target ES2022 --module ES2022 --moduleResolution Bundler \ + --strict --skipLibCheck test/workspace-locked-child-cumulative.compile.ts && \ + npx tsc --noEmit -p . && npm run build) +(cd harness && .venv/bin/pytest -q \ + tests/test_protected_fs_helper.py tests/test_locked_child_stdin.py \ + tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py \ + tests/test_registry_evidence_revision.py tests/test_preprocess_cli.py \ + tests/test_corpus_pipeline.py tests/test_qdrant_vector_store.py \ + tests/test_semantic_kind_isolation.py tests/test_filesystem_evidence_source.py && \ + .venv/bin/ruff check tht/protected_fs_helper.py tht/locked_child_stdin.py \ + tests/test_protected_fs_helper.py tests/test_locked_child_stdin.py \ + tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py \ + tests/test_registry_evidence_revision.py) +(cd tools/thothctl && test "$(go env GOVERSION)" = 'go1.26.5' && go test ./... -count=1) +``` + +Expected: PASS with exact counts recorded. + +- [ ] **Step 3: run every repository full suite/gate.** + +```bash +(cd backend && npx vitest run && npx tsc --noEmit -p . && npm run build) +(cd harness && .venv/bin/pytest -q && .venv/bin/ruff check .) +(cd frontend && npx vitest run && npx tsc -b && npm run build) +(cd tools/thothctl && go test ./... -count=1) +./scripts/test-default-compose.sh +./scripts/test-unified-compose.sh +./scripts/test-internal-semantic-compose.sh +./scripts/test-compose-secret-policy.sh +./scripts/test-no-deployment-coupling.sh +./scripts/test-deployment-command-contract.sh +./scripts/test-thothctl-build-contract.sh +./scripts/test-p6-acceptance.sh +./scripts/test-p2-p6-acceptance.sh +./scripts/test-p6-manual-verification.sh +./scripts/test-verify-workspace-install-docs.sh +./scripts/verify-workspace-install-docs.sh --fixtures-only +git diff --check +git status --short +``` + +Expected: every command exits 0. Existing unrelated lint debt, if still accepted by project policy, must be reported with an exact baseline and touched files must remain clean; never summarize a failing required gate as PASS. + +- [ ] **Step 4: run final clean P6 and aggregate integrations once each.** + +```bash +./scripts/p6-acceptance.sh integration --keep +./scripts/p2-p6-acceptance.sh integration --keep +``` + +Expected: new roots, no retry, same exact clean source commit/tree/image, every check PASS, fixture secrets absent, complete retained-root scans PASS, and no owned live child/listener/container/network/volume. + +- [ ] **Step 5: independently verify reports and retained roots.** + +Use bounded report parsers to verify schemas, required check IDs exactly once, declared SHA-256 against every final artifact, source/image/Git identities, exact command order, A/B independence, and zero owned resources. Assert no fixture-secret path exists. Scan every regular file below each retained root with **no excluded directory**, then scan the final reports again. Do not grep secret contents into terminal history. + +- [ ] **Step 6: record the automated checkpoint and commit only status docs.** + +`PROJECT_STATE.md` records exact source commit/tree, report paths and SHA-256, test/check counts, limitations, and: + +```text +P6 automated integration: PASS +P6 manual acceptance: PENDING +aggregate P2-P6 automated integration: PASS +aggregate P2-P6 manual acceptance: PENDING +``` + +```bash +git add PROJECT_STATE.md +git commit -m 'docs: record P6 automated verification checkpoint' +git status --short +``` + +- [ ] **Step 7: stop for the human gate.** + +Provide exact commands: + +```bash +./scripts/p6-manual-verification.sh prepare +./scripts/p6-manual-verification.sh start +# reviewer follows .artifacts/manual-acceptance/p6/GUIDE.md and writes VERDICT.md +./scripts/p6-manual-verification.sh stop +``` + +Only an explicit reviewer PASS for both decisions may update the living manual and `PROJECT_STATE.md`; FAIL/PENDING reopens a focused owning plan. Cleanup is offered only after the verdict hashes are recorded: + +```bash +./scripts/p6-manual-verification.sh cleanup +``` + +Stop. No P7/PSD/GUI/SSH work begins without new authorization. + +--- + +## Final acceptance checklist + +- [ ] Only the finalized flat P2/P3/P5 APIs are used; registry pull names/fields/orders match the quoted `RegistryPullAddressedPlanV1`, `AddressedWorkspacePublicationLeaseV1`, `OrderedWorkspaceWriterCapabilitySet`, `RegistryAddressedRequestV1`, exact participant interfaces, `RegistryPullPhaseV1`, and `RegistryPullJobStateV1`, with required installation/repository/remote-ref identity parity, pull-create expected-base parity, and no alias, optional overload, or replacement. +- [ ] Registry-pull dependency/integration tests cover a complete reverse-presented multi-workspace lexical set, exact same-ID SIGKILL immediately before active-pointer rename and immediately after rename+parent fsync, no resume refetch/republication or participant/capability reentry, target drift/third identity refusal, full-set ownership through publication/terminal durability, reverse release, then repository release. +- [ ] Repository → writer is the only nested lock order; later writer-only one-element ordered execution has no repository/Git capability. +- [ ] The filesystem operation's registry RW mount exposes only the exact lock and fixed read Git plumbing; mutation-negative tests pass. +- [ ] Every Git child is deadline/output/record/stderr/cancel bounded, closes stdin, TERM→KILLs only its owned group, and awaits exit/stdio. +- [ ] The final sole `WorkspaceLockedChildRequest` alias remains the original three P2 members plus the unchanged fourteen-member `P3LockedChildRequest`, both P5 annotation members, and both P6 Evidence-tree members. A bidirectional compile-only exact-kind fence proves the cumulative 14 + 2 + 2 extension surface, and a table-driven runtime `assertNever` regression routes all twenty-one requests through the same `WorkspaceWriterLockCapability.spawnChild`; P3 locked-child focused tests remain in the P6 file, commit, and release gates. +- [ ] P5 `ProtectedWorkspaceFs` and selected-image helper provide retained-dirfd tree publication with `RENAME_NOREPLACE`; no path-only/overwrite fallback exists. Every annotation/tree publish or verify uses that sole cumulative `WorkspaceWriterLockCapability.spawnChild` union, actual FD 3 + FD 4, fixed framing/result caps, and no ambient/direct spawn. +- [ ] Adjacent reserved/bound owner evidence makes every pre/post-rename kill point explicit and deletion inode-confined. +- [ ] Preflight counts every file, source directory, staging root, manifest, and both owner records. +- [ ] Every reuse resolves the exact Git tree OID/inventory before streaming/hashing local bytes; Git unavailability fails closed. +- [ ] Retention is active + validated leases + complete manifest pins + nonterminal runs; physical snapshot/content directories never self-pin. +- [ ] Snapshot reconciliation precedes Evidence reconciliation from the same frozen authoritative set. +- [ ] P6 adds only P3 regression tests; any P3 production defect is repaired/checkpointed under P3 before continuing. +- [ ] Fixture secrets are deleted before finalization, and every retained-root scan has no secret-directory exclusion. +- [ ] Aggregate order is exactly bootstrap A/B; P2/P3 C0 READY proof; one C0 admission self-heal plus strict index in A/B; A guarded rebuild/interrupted recovery plus reindex; both P5 pauses; one curated C1 commit; independent P3 `workspace registry pull --workspace ` in A/B with A same-ID pre-publication SIGKILL/resume, B same-ID post-rename+parent-fsync SIGKILL/resume, and isolated target-drift/third-identity refusal; exact `workspace migrate dwh-cache --workspace "$WORKSPACE_ID" --json` plus strict pre-READY C1/binding physical/LSH snapshot publication verification in A and B; both accepts; independent C1 `activate-revision-layout` in A/B; then C1 strict-index/resume/materialize/retrieve/idempotency in A and B. Admission/resume remain `migration_required` through pull, DWH preparation, and accept and succeed only after the matching activation. +- [ ] Production self-heal traverses the released admission route and proves no failed-session manifest, lease, admission, or Pi residue. +- [ ] Focused tests, full backend/harness/frontend/Go suites, Compose/docs gates, P6 integration, aggregate integration, and independent manual tooling are all explicit. +- [ ] Automated statuses are PASS only from final clean retained reports; both manual decisions remain PENDING until the reviewer acts.