# P4 Qdrant Bootstrap and Guarded Rebuild Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. **Goal:** Implement PRD D4 so session admission safely self-heals a missing workspace Qdrant collection or missing keyword indexes, while `thothctl` can inspect, destructively rebuild, and recover only the descriptor-owned collection under durable maintenance and complete quiescence. **Architecture:** Move Qdrant collection inspection/reconciliation out of `ThtRunner` into one shared TypeScript manager used by session admission and P2's `WorkspacePreprocessingService.execute`. The manager has safe readiness reconciliation plus narrow destructive primitives—never a public monolithic `rebuild()`—so the guarded service can durably record every mutation boundary. Readiness may create only the fixed 1024/cosine collection contract and missing keyword indexes; incompatible vector or index types remain fail-closed. Destructive rebuild is a host-orchestrated transaction: `thothctl` holds one installation lifecycle lock, activates the backend's durable admission barrier, proves the complete session inventory and live process/admission counts are quiescent, stops `core`, and runs the dedicated Pi-free `backend/src/workspace-maintenance.ts::main` entrypoint under P2's writer lock. Success persists `deleting` before DELETE, `deleted` after a confirmed 404, `recreated` after exact collection/index creation, and `verified` only after a separate final inspection; a post-delete failure leaves maintenance active for explicitly confirmed recovery. **Tech Stack:** TypeScript 5, Node.js 22 `fetch`, Fastify 5, Vitest, Qdrant REST API v1.18, Go 1.26, `gofrs/flock`, Docker Compose, Bash, real Git fixtures, and JSON/YAML. --- ## Source, prerequisites, and hard boundary - Source requirement: `docs/prd/2026-08-09-workspace-preprocessing-prd.md` D4, RF4.2–RF4.4, RNF1–RNF9, and section 8. - Reviewed design: `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` section 6 and the common error/verification contracts. - P1 is already accepted. Execute and checkpoint P2, then P3, before this plan because this implementation deliberately **extends**, rather than duplicates, P2's exact maintenance surface: - `backend/src/workspaces/{runtime-config-lease.ts,preprocessing-state.ts,preprocessing-service.ts}` and `backend/src/workspace-maintenance.ts`; - `tools/thothctl/internal/workspaceops/operations.go`; - profile-only `workspace-maintenance` in `compose.yaml`; - `/data/sessions//preprocessing/writer.lock` and `runUnderWorkspaceWriterLock(...)`. - This plan is rebased on P2/P3's frozen handoff: `backend/src/workspace-maintenance.ts::main`, `WorkspacePreprocessingService.execute`, `PreprocessingStateStore`, `WorkspaceRuntimeConfigLeaseFactory`, `backend/src/workspaces/revision-layout.ts::{workspaceRuntimePaths,readRevisionLayoutState}`, `runUnderWorkspaceWriterLock` / `probeWorkspaceWriterLock`, and `tools/thothctl/internal/workspaceops::{ParseWorkspaceCommand,Run}`. Run the recorded P2/P3 focused gates before Task 1. Do not create a second operator, renderer, state root, lock, or `internal/workspace` package. - P4 does not implement P5 Git annotations, P6 Evidence materialization, P7 migration, P8 aggregate smoke, P9 retention policy, P10 SSH runtime support, a GUI maintenance endpoint, or automatic semantic reindexing after rebuild. - Qdrant remains derived data. Rebuild intentionally discards the selected collection's schema/Evidence/Memory projection. Canonical Git descriptors, phase artifacts, corpus state, and the memory registry remain untouched; operators must rerun the already-available indexing commands afterward. - No prefix matching, collection enumeration followed by bulk deletion, Qdrant-wide cleanup, forced Pi termination, automatic session closure, or automatic retry is permitted. ## Exact public command and result contract P2's read-only command gains a `semantic_index` and `collection_recovery` section: ```text thothctl --installation /abs/thothii-installation.yaml \ workspace inspect --workspace research --json ``` P4 adds only these destructive commands: ```text thothctl --installation /abs/thothii-installation.yaml \ workspace collection rebuild \ --workspace research \ --confirm-workspace research \ --confirm-collection research-semantic \ --destructive --json thothctl --installation /abs/thothii-installation.yaml \ workspace collection recover \ --workspace research \ --confirm-workspace research \ --confirm-collection research-semantic \ --destructive --json ``` Rules: 1. `--workspace`, `--confirm-workspace`, `--confirm-collection`, and the literal `--destructive` are all required once and only once for rebuild/recover. Values are compared byte-for-byte after the normal identifier syntax validation; no case folding, trimming, defaults, interactive prompts, or `--yes` alias. 2. The collection is always re-derived from the active, operational schema-v3 descriptor. A caller-supplied collection is never used as a mutation target until it exactly equals that value. 3. Confirmation mismatch exits 2 before maintenance activation, lifecycle state creation, collection mutation, or stopping `core`. 4. `inspect` is read-only and may run concurrently. It reports `missing`, `compatible`, `repairable` (only keyword indexes are absent), or `incompatible`, plus safe expected/observed dimensions, distance, required index names/types, active revision, maintenance state, and recovery phase. It never self-heals. 5. Machine output is one pristine JSON object. Stable P4 codes are `semantic_index_incompatible`, `preprocessing_conflict`, `collection_confirmation_mismatch`, `collection_recovery_required`, `collection_recovery_not_required`, `session_inventory_active`, `maintenance_not_quiescent`, and `workspace_not_activatable`. No response includes a Qdrant response body, arbitrary exception, raw child stderr, endpoint, credential, signed URL, or secret-file content. The fixed collection contract remains: ```ts export const REQUIRED_QDRANT_KEYWORD_INDEXES = [ "content_hash", "document_id", "kind", "record_key", "record_kind", "vector_generation", "workspace_id", "workspace_revision", ] as const; ``` Vectors are exactly size `1024`, distance `Cosine`; every listed payload index is exactly `keyword`. ## Durable transaction and fail-safe matrix Store the collection transaction at: ```text /data/sessions//preprocessing/qdrant-rebuild-state.json ``` Use strict schema version 1: ```ts interface CollectionRebuildStateV1 { version: 1; transaction_id: string; operation: "collection-rebuild"; workspace_id: string; workspace_revision: string; // exact 40-hex active commit collection: string; // descriptor-derived exact name phase: "prepared" | "deleting" | "deleted" | "recreated" | "verified" | "failed_pre_delete"; mutation_started: boolean; created_at: string; updated_at: string; error_code?: "semantic_index_incompatible" | "workspace_not_activatable"; } ``` Write `prepared` durably before DELETE and write `deleting` with `mutation_started: true` durably **before** issuing DELETE. Therefore a crash with `mutation_started: true` is always treated as potentially destructive even when Qdrant still contains the collection. Atomic replace must fsync the file and owning directory, use regular no-follow 0600 files under the P2-owned preprocessing root, reject unknown keys/versions/identity drift, and never interpolate an exception into `error_code`. - Failure before `mutation_started` may mark `failed_pre_delete`, restart `core`, verify health, and clear maintenance. - Failure or ambiguous subprocess output after `mutation_started` leaves `core` stopped and the durable maintenance marker active. The CLI prints the exact recover command using only safe workspace/collection identifiers. - Recovery accepts only a nonterminal state matching the current descriptor workspace, revision, and collection. If the collection is missing it recreates it; if compatible it verifies it (covering a crash after recreate); if incompatible it may delete/recreate the exact same collection only because recovery repeats all destructive confirmations and the transaction proves mutation already began. It never targets a different or prefix-matched collection. - `verified` is retained for inspection/audit. A later rebuild may atomically supersede it only after proving it is terminal. ## Automated P4 process goal Start the execution goal before Task 1 and complete it only after this single clean-state command succeeds: ```bash ./scripts/p4-acceptance.sh integration --keep ``` The command must refuse a dirty tracked source tree, bind the run to the exact Git commit/tree, use one unique Compose project and ownership manifest below `.artifacts/p4-integration//`, execute once with no automatic retry, and retain `report.json`, `report.md`, bounded command events, safe Qdrant observations, recovery state copies, artifact hashes, and cleanup proof. External LLM/DWH/Ollama behavior is outside D4 and uses fixture-only values; Qdrant, the compiled core image, backend routes, `thothctl`, Git registry snapshot, dedicated maintenance service, persistent volumes, and Compose lifecycle are real. The integration must prove: missing collection admission self-heal; missing-index self-heal; concurrent compatible creation/index reconciliation; incompatible dimension/distance/index refusal without PUT/DELETE; exact confirmation refusal before maintenance; complete open-session refusal; live admission/Pi count quiescence; held P2 writer-lock refusal before deletion; exact target-only deletion with a neighbor collection unchanged; verified rebuild/core restart/maintenance clear; a deterministic interruption after deletion with core stopped and marker retained; explicit recovery; secret scan; no retry; and exact owned-resource cleanup. `--keep` retains filesystem evidence, never live containers/networks/volumes or secrets. No unavoidable human action exists inside this automated goal. Manual acceptance happens afterward in a new independent environment. --- ### Task 0: Establish the execution baseline and persistent goal **Files:** - Read: `docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md` - Read: `docs/superpowers/plans/2026-08-10-p3-effective-config-fingerprint.md` - Read: `backend/src/workspaces/{runtime-config-lease.ts,revision-layout.ts,preprocessing-state.ts,preprocessing-service.ts}` and `backend/src/workspace-maintenance.ts` - Read: `tools/thothctl/internal/workspaceops/operations.go` - Read: `compose.yaml`, `deploy/compose.local.yaml`, `deploy/compose.server.yaml` - [ ] **Step 1: Confirm the worktree and prerequisites** ```bash git status --short git log -1 --format='%H %T' test -f backend/src/workspaces/preprocessing-state.ts test -f tools/thothctl/internal/workspaceops/operations.go ``` Expected: clean output from `git status`; one commit/tree line; both files exist. Confirm P2 and P3 checkpoint reports say automated integration PASS. If not, stop—do not fold their scope into P4. - [ ] **Step 2: Start the process goal** If the execution environment supports persistent goals, create: “P4 clean-state Qdrant bootstrap/rebuild/recovery process passes once without retry, retains a secret-clean report, and cleans only owned resources.” Keep it open through Task 11. - [ ] **Step 3: Run prerequisite focused gates** Run the exact P2/P3 focused verification commands recorded in their checkpoint reports. Expected: PASS. A failure is prerequisite drift; fix it in its owning plan before P4. No commit. --- ### Task 1: Extract the shared Qdrant collection manager **Files:** - Create: `backend/src/semantic/qdrant-collection-manager.ts` - Create: `backend/test/qdrant-collection-manager.test.ts` - Modify: `backend/src/tht/tht-runner.ts` (`QdrantEnsureResult`, `REQUIRED_QDRANT_PAYLOAD_INDEXES`, `qdrantEnsure`) - Modify: `backend/test/tht-qdrant-readiness.test.ts` **Required interface:** ```ts export type CollectionState = "missing" | "compatible" | "repairable" | "incompatible"; export interface QdrantCollectionSpec { collection: string; dimensions: 1024; distance: "cosine"; keywordIndexes: readonly string[]; } export interface CollectionInspection { state: CollectionState; expected: { dimensions: 1024; distance: "cosine"; keyword_indexes: readonly string[] }; observed?: { dimensions?: number; distance?: string; keyword_indexes: Record }; missing_keyword_indexes: string[]; incompatible_fields: string[]; } export class QdrantCollectionManager { constructor(options: { baseUrl: string; request?: typeof fetch }); inspect(spec: QdrantCollectionSpec, timeoutMs: number): Promise; ensure(spec: QdrantCollectionSpec, timeoutMs: number): Promise<{ ok: boolean; code?: SemanticReadinessCode }>; deleteExactAndConfirmMissing(spec: QdrantCollectionSpec, timeoutMs: number): Promise; ensureCollection(spec: QdrantCollectionSpec, timeoutMs: number): Promise; ensureIndexes(spec: QdrantCollectionSpec, timeoutMs: number): Promise; } export function qdrantCollectionSpec(workspace: WorkspaceDescriptor): QdrantCollectionSpec; ``` There is deliberately no exported `rebuild()` and no manager method that can cross more than one durable destructive phase. `deleteExactAndConfirmMissing` issues one exact DELETE and succeeds only after a separate GET confirms 404. `ensureCollection` creates or converges only the exact 1024/Cosine collection and finishes with a read proving the vector contract (keyword indexes may still be absent). `ensureIndexes` creates only missing required keyword indexes in canonical order and finishes with a read proving the exact complete contract. The guarded service in Task 3 is the sole production composer of these primitives; readiness calls only `ensure`. - [ ] **Step 1: Write RED inspection tests** Use a scripted `vi.fn` fetch boundary. Cover exact URL encoding, GET 404 → `missing`, exact 1024/Cosine/eight keyword indexes → `compatible`, missing index → `repairable`, and wrong size/distance/index type → `incompatible`. Reject malformed success JSON as `workspace_not_activatable`; never propagate a body canary. - [ ] **Step 2: Verify RED** ```bash cd backend npx vitest run test/qdrant-collection-manager.test.ts ``` Expected: FAIL because the module is missing. - [ ] **Step 3: Implement read-only inspection** Use only `new URL('/collections/' + encodeURIComponent(name), baseUrl)`, an operation-wide `AbortController`, allowlisted parsed fields, and lowercase comparison for Qdrant's distance/type response. Distinguish missing from incompatible; an unavailable/unparseable service is not semantic incompatibility. - [ ] **Step 4: Write RED self-heal/concurrency tests** Cover these exact request sequences: 1. GET 404 → PUT collection with `{"vectors":{"size":1024,"distance":"Cosine"}}` → final GET compatible. 2. GET 404 → PUT 409/already exists → final GET compatible (concurrent compatible creator). 3. GET repairable → PUT `/collections//index?wait=true` with `{"field_name":"kind","field_schema":"keyword"}` → final GET compatible. 4. Index PUT conflict → final GET compatible (concurrent compatible index creator). 5. A barrier-controlled unit race launches two independent `manager.ensure` calls (not a shared promise): both initial GETs observe 404/repairable, one PUT succeeds, the other receives the allowed conflict, both perform their own final GET, and both converge compatible. Repeat for collection creation and one missing index; assert no retry loop. 6. Concurrent creator/index ends incompatible → `semantic_index_incompatible`. 7. Existing incompatible size/distance/index performs no PUT or DELETE. 8. A missing-index run never recreates the collection and creates only missing indexes in canonical order. 9. Timeout/unreachable response returns `workspace_not_activatable` with no endpoint/body leak. 10. `deleteExactAndConfirmMissing` issues DELETE for the one URL-encoded exact collection and then independently reads 404; DELETE failure or a non-404 post-delete state fails. 11. `ensureCollection` and `ensureIndexes` are separately observable: collection creation ends with the exact vector contract present, index creation ends with the exact complete contract, and neither ever deletes. A later independent `inspect` supplies final verification. No other collection name, list endpoint, prefix, or global mutation is ever requested. 12. No exported `rebuild`, callback that hides multiple phases, or test-only destructive shortcut exists. - [ ] **Step 5: Verify RED, then implement minimal reconciliation** ```bash cd backend npx vitest run test/qdrant-collection-manager.test.ts ``` Expected before implementation: FAIL on PUT sequences. Implement create/index calls, tolerate only conflict/already-exists as a reason to re-read, and always decide success from one final GET. Do not accept the mutating response as proof. - [ ] **Step 6: Delegate `ThtRunner.qdrantEnsure`** Keep descriptor operational validation in `qdrantEnsure`, derive `qdrantCollectionSpec`, and call the shared manager. Remove the duplicate constant/parsing logic. Preserve `ThtConfig.qdrantRequest` as the injectable fetch boundary. - [ ] **Step 7: Run focused tests and typecheck** ```bash cd backend npx vitest run test/qdrant-collection-manager.test.ts test/tht-qdrant-readiness.test.ts test/readiness-manager.test.ts npx tsc --noEmit -p . ``` Expected: PASS. Update the old missing-collection assertion from incompatibility to success after the exact create/final-read sequence; existing incompatible cases remain fail-closed. - [ ] **Step 8: Commit** ```bash git add backend/src/semantic/qdrant-collection-manager.ts \ backend/src/tht/tht-runner.ts \ backend/test/qdrant-collection-manager.test.ts \ backend/test/tht-qdrant-readiness.test.ts \ backend/test/readiness-manager.test.ts git commit -m "feat: self-heal workspace qdrant collections" ``` --- ### Task 2: Prove session admission uses self-heal before Ollama and persistence **Files:** - Modify: `backend/test/routes-sessions.test.ts` (readiness cases around current Qdrant tests) - Modify if dependency injection requires it: `backend/src/app.ts` (`ThtRunner` construction only) - [ ] **Step 1: Write RED route tests** At the real `buildApp` boundary, inject a `ThtRunner`/fetch script and assert: - missing collection creates the exact contract, then Ollama runs, then session persistence is allowed; - missing index creates only that index before Ollama; - incompatible collection returns 503 with `semantic_index_incompatible`, does not call Ollama, does not call `sessionNew`, acquire a revision lease permanently, or spawn Pi; - two simultaneous **local-mode, same-principal, same-workspace** admissions hit `ReadinessManager`'s in-flight key and share one promise: exactly one Qdrant create/reconciliation occurs, both callers receive the same compatible result, and no false incompatibility is emitted; - one separate `ReadinessManager` unit test proves the same-principal dedup key and cleanup after resolution/rejection; - one **upstream-auth, two-distinct-principal** route test uses distinct readiness keys, coordinates both initial GET 404 reads, lets one PUT create and the other PUT receive the compatible-creator conflict, and proves both converge after their final reads. Do not describe this as a local same-principal race; - raw Qdrant body/endpoint canaries do not appear in HTTP JSON or logs captured by the test. Keep the compatible-creator and compatible-index conflict sequences in `qdrant-collection-manager.test.ts` as the direct manager race proof. Route tests prove the real dedup/cross-principal semantics rather than attempting to bypass `ReadinessManager`. - [ ] **Step 2: Run and verify RED** ```bash cd backend npx vitest run test/routes-sessions.test.ts -t "Qdrant|semantic index|concurrent collection" ``` Expected: the new call-order, local deduplication, and distinct-principal convergence assertions FAIL until route fixtures use the real manager behavior and upstream identities produce distinct readiness keys. - [ ] **Step 3: Make the minimal wiring change** Do not add a new route. Session admission continues through `ReadinessManager.ensure(..., descriptor)` and `ThtRunner.qdrantEnsure`; change only construction/injection necessary to share the manager. - [ ] **Step 4: Verify** ```bash cd backend npx vitest run \ test/routes-sessions.test.ts \ test/readiness-manager.test.ts \ test/tht-qdrant-readiness.test.ts npx tsc --noEmit -p . ``` Expected: PASS, including Qdrant-before-Ollama ordering, exactly one create for local same-principal deduplication, two independently convergent manager calls for distinct upstream principals, and no persistence on incompatibility. - [ ] **Step 5: Commit** ```bash git add backend/src/app.ts backend/test/routes-sessions.test.ts backend/test/readiness-manager.test.ts git commit -m "test: prove qdrant admission self-heal" ``` Omit unchanged paths from `git add`. --- ### Task 3: Add safe inspection and durable rebuild/recovery to the P2 operator **Files:** - Create: `backend/src/workspaces/qdrant-rebuild-state.ts` - Create: `backend/test/qdrant-rebuild-state.test.ts` - Modify: `backend/src/workspaces/preprocessing-service.ts` (`MaintenanceCommand`/result union) - Modify: `backend/src/workspaces/preprocessing-service.ts` (`WorkspacePreprocessingService.execute`) - Modify: `backend/src/workspace-maintenance.ts` (`main` parser/encoder) - Modify: `backend/src/workspaces/preprocessing-state.ts` - Modify: `backend/test/workspace-preprocessing-service.test.ts` - Modify: `backend/test/workspace-maintenance.test.ts` - Modify: `backend/test/workspace-preprocessing-state.test.ts` **Operator additions:** ```ts type MaintenanceCommand = | /* P2/P3 requests */ | { operation: "collection-rebuild"; workspace_id: string; confirm_workspace: string; confirm_collection: string; destructive: true } | { operation: "collection-recover"; workspace_id: string; confirm_workspace: string; confirm_collection: string; destructive: true }; ``` `inspect` gains safe `semantic_index` and `collection_recovery`; it never acquires a writer lock or mutates Qdrant. Rebuild/recover must call `runUnderWorkspaceWriterLock` for the complete state-read → mutation → verification transaction. A conflict maps to `preprocessing_conflict` before state or Qdrant mutation. - [ ] **Step 1: Write RED state-store tests** Test restrictive creation, atomic durable transitions, same-inode/root safety, unknown schema/keys, workspace/revision/collection mismatch, symlink/path swap, injected file-fsync and directory-fsync ambiguity, and terminal supersession. Table-test the only legal forward edges `prepared → deleting → deleted → recreated → verified` plus `prepared → failed_pre_delete`; reject skips, rewinds, cross-transaction updates, and `failed_pre_delete` after mutation. Add crash-visible assertions at every boundary: `deleting/mutation_started` is durable before mocked DELETE, `deleted` only after the manager confirms 404, `recreated` only after both exact collection and required indexes are confirmed, and `verified` only after one additional independent `inspect` returns compatible. - [ ] **Step 2: Verify RED** ```bash cd backend npx vitest run test/qdrant-rebuild-state.test.ts ``` Expected: FAIL because `CollectionRebuildStateStore` does not exist. - [ ] **Step 3: Implement the strict store** Reuse P2's trusted preprocessing-root and atomic state primitives rather than another root policy. Expose only `read`, `begin`, and `transition(expectedTransaction, next)`; enforce legal monotonic phase transitions. Report durability uncertainty as recovery-required whenever the intended bytes may have reached the canonical path. - [ ] **Step 4: Write RED operator tests** Cover: - `inspect` returns active workspace/revision, exact descriptor collection, manager inspection, and safe recovery state without mutation/lock; - confirmation mismatch and missing `destructive: true` return `collection_confirmation_mismatch` before marker/lock/Qdrant; - rebuild refuses unless `/data/settings/maintenance.json` is a trusted regular file whose strict JSON says active; - held `/preprocessing/writer.lock` returns `preprocessing_conflict` before DELETE; - rebuild writes/fsyncs `prepared`, then writes/fsyncs `deleting` with `mutation_started:true` before `deleteExactAndConfirmMissing`; after that primitive's confirmed 404 it writes/fsyncs `deleted`; after `ensureCollection` and `ensureIndexes` have separately confirmed the exact contract it writes/fsyncs `recreated`; only a subsequent standalone `inspect` returning compatible permits `verified`; - injected crashes after each durable phase leave exactly that phase visible, never infer a later phase from a mutating response, and resume through the same narrow manager primitives; - neighbor collections are never requested; - failure before `deleting` becomes `failed_pre_delete` with `mutation_started: false`; - failure/timeout at or after `deleting` retains the last durable nonterminal state with `mutation_started: true` and returns a safe recovery-required envelope; - recover handles every nonterminal phase: `prepared` may start deletion only after repeated confirmations; `deleting` first inspects and conservatively establishes missing/compatible/incompatible exact state; `deleted` recreates; `recreated` independently verifies; missing, already-compatible-after-crash, and explicitly confirmed incompatible exact targets all converge without skipping a durable boundary; - recover rejects absent/verified state, changed active revision, changed descriptor collection, and unsafe/unknown state; - error output excludes Qdrant body, marker contents beyond allowlisted booleans, paths, endpoints, and canaries. - [ ] **Step 5: Run and verify RED** ```bash cd backend npx vitest run \ test/workspace-preprocessing-service.test.ts \ test/workspace-preprocessing-state.test.ts \ test/workspace-maintenance.test.ts ``` Expected: new operation cases FAIL because dispatch/state enforcement is absent. - [ ] **Step 6: Implement guarded dispatch** Derive the descriptor and revision through `WorkspacePreprocessingService.execute` and its existing P2 resolver, validate confirmations, verify the marker through `lstat/open(O_NOFOLLOW)/fstat` and strict JSON, then enter `runUnderWorkspaceWriterLock`. Compose only `inspect`, `deleteExactAndConfirmMissing`, `ensureCollection`, and `ensureIndexes`, persisting/fsyncing the state between calls exactly as specified above. Neither `backend/src/workspace-maintenance.ts::main` nor any other caller receives a monolithic destructive API. The CLI accepts fixed argv generated by `workspaceops.Run` and preserves one pristine JSON envelope; direct manual invocation without marker/confirmations remains harmless. - [ ] **Step 7: Verify focused backend tests** ```bash cd backend npx vitest run \ test/qdrant-collection-manager.test.ts \ test/qdrant-rebuild-state.test.ts \ test/workspace-preprocessing-state.test.ts \ test/workspace-preprocessing-service.test.ts \ test/workspace-maintenance.test.ts npx tsc --noEmit -p . npm run build ``` Expected: PASS; build emits `/app/backend/dist/workspace-maintenance.js` and the new shared manager/state module. - [ ] **Step 8: Commit** ```bash git add backend/src/workspaces/qdrant-rebuild-state.ts \ backend/src/workspaces/preprocessing-state.ts \ backend/src/workspaces/preprocessing-service.ts \ backend/src/workspace-maintenance.ts \ backend/test/qdrant-rebuild-state.test.ts \ backend/test/workspace-preprocessing-state.test.ts \ backend/test/workspace-preprocessing-service.test.ts \ backend/test/workspace-maintenance.test.ts git commit -m "feat: add guarded qdrant maintenance operations" ``` --- ### Task 4: Expose loopback-only maintenance quiescence **Files:** - Modify: `backend/src/app.ts` (`maintenance status handlers`, `isMaintenanceControl`) - Modify: `backend/src/runtime/maintenance-gate.ts` only if a named status type is needed - Modify: `backend/test/routes-sessions.test.ts` (current maintenance endpoint tests) - Modify: `backend/test/maintenance-gate.test.ts` only for status typing/durability regressions - Modify: `backend/test/pi-process-manager.test.ts` **Response:** ```json { "active": true, "admissions": 0, "piProcesses": 0, "quiescent": true } ``` Include `recoveryRequired: true` only when already produced by `MaintenanceBarrier`. `quiescent` is true only when the durable marker is active, `admissions === 0`, `mgr.count() === 0`, and no durability recovery is pending. - [ ] **Step 1: Write RED endpoint tests** Test `POST /internal/maintenance/activate`, `GET /internal/maintenance/status`, and new `GET /internal/maintenance/quiescence` with injected manager counts 0/1. Prove activation waits for an in-flight admission lease, later admissions receive 503, Pi count is observational (never killed), spoofed non-loopback callers receive 403 in `none` and `upstream` auth, and durability ambiguity makes `quiescent: false`. - [ ] **Step 2: Verify RED** ```bash cd backend npx vitest run test/routes-sessions.test.ts -t "maintenance|quiescence" ``` Expected: FAIL because responses omit `piProcesses`/`quiescent` and the route is not allowlisted. - [ ] **Step 3: Implement one response composer** In `buildApp`, define `maintenanceStatus()` from `maintenanceBarrier.status()` plus `mgr.count()`. Use it for activate/deactivate/status/quiescence responses so the fields cannot drift. Do not expose runtime IDs, principals, session questions, or child process details. - [ ] **Step 4: Verify** ```bash cd backend npx vitest run \ test/maintenance-gate.test.ts \ test/routes-sessions.test.ts \ test/pi-process-manager.test.ts npx tsc --noEmit -p . ``` Expected: PASS. Existing admission behavior remains unchanged except for additive internal response fields. - [ ] **Step 5: Commit** ```bash git add backend/src/app.ts backend/src/runtime/maintenance-gate.ts \ backend/test/routes-sessions.test.ts backend/test/maintenance-gate.test.ts \ backend/test/pi-process-manager.test.ts git commit -m "feat: report complete maintenance quiescence" ``` --- ### Task 5: Share one installation lifecycle lock with Pi operations **Files:** - Create: `tools/thothctl/internal/lifecycle/lock.go` - Create: `tools/thothctl/internal/lifecycle/lock_test.go` - Modify: `tools/thothctl/internal/config/installation.go` (`LifecycleLockPath`, `CollectionRebuildStatePath` only if host metadata is retained) - Modify: `tools/thothctl/internal/config/installation_test.go` - Modify: `tools/thothctl/internal/pi/state.go` (`acquireLock`, `updateLock`, `ErrLockHeld` removal/delegation) - Modify: `tools/thothctl/internal/pi/update.go` (`Update`, `Rollback`, `RecoverMaintenance` lock acquisition) - Modify: `tools/thothctl/internal/pi/update_test.go` - Modify: `tools/thothctl/internal/pi/state_test.go` **Path:** `.thothctl//lifecycle.lock`, with owner metadata `.thothctl//lifecycle.lock.owner.json`. - [ ] **Step 1: Write RED lifecycle tests** Test stable 0600 lock inode, 0600 atomic owner metadata, nonblocking cross-process contention, metadata removal only by the owner on release, crash-safe kernel release, symlink/unsafe control-directory rejection, and distinct installation descriptors not contending. Add a test that a Pi update lock blocks a simulated collection rebuild and vice versa. - [ ] **Step 2: Verify RED** ```bash cd tools/thothctl go test ./internal/lifecycle ./internal/config ``` Expected: FAIL because the lifecycle package/path does not exist. - [ ] **Step 3: Extract the current lock without changing Pi transaction semantics** Move the `gofrs/flock` implementation from `internal/pi/state.go` into `lifecycle.Acquire(path, operation)`. Owner JSON contains PID, host, start time, transaction, and an allowlisted operation (`pi-update`, `pi-rollback`, `pi-maintenance-recover`, `qdrant-rebuild`, `qdrant-recover`), never argv/environment. Add `LifecycleLockPath string` to `pi.Request`; change `Rollback` and `RecoverMaintenance` to accept `(statePath, lifecycleLockPath, confirm)`; and make `main.go` pass `installation.LifecycleLockPath()` to all three paths. Reject an empty/noncanonical lock path. The update state remains `UpdateStatePath()`; only mutual exclusion moves from `update-state.json.lock` to the installation lock. - [ ] **Step 4: Run Pi regressions** ```bash cd tools/thothctl go test ./internal/lifecycle ./internal/config ./internal/pi ``` Expected: PASS; interrupted update/rollback behavior and recovery metadata remain identical, but P4 and Pi now contend on one installation lock. - [ ] **Step 5: Commit** ```bash git add tools/thothctl/internal/lifecycle \ tools/thothctl/internal/config/installation.go \ tools/thothctl/internal/config/installation_test.go \ tools/thothctl/internal/pi/state.go tools/thothctl/internal/pi/state_test.go \ tools/thothctl/internal/pi/update.go tools/thothctl/internal/pi/update_test.go git commit -m "refactor: share installation lifecycle lock" ``` --- ### Task 6: Parse exact collection commands and confirmations in `thothctl` **Files:** - Modify: `tools/thothctl/internal/workspaceops/operations.go` - Modify: `tools/thothctl/internal/workspaceops/operations_test.go` - Create: `tools/thothctl/internal/workspaceops/collection.go` - Create: `tools/thothctl/internal/workspaceops/collection_test.go` - Modify: `tools/thothctl/cmd/thothctl/main.go` - Modify: `tools/thothctl/cmd/thothctl/main_test.go` - [ ] **Step 1: Write RED parse tests** Table-test the exact commands above. Reject missing/duplicate flags, swapped confirmations, whitespace/case differences, unknown flags/subcommands, positional targets, `--yes`, absent `--destructive`, unsafe workspace/collection identifiers, and rebuild/recover input or output file flags inherited from P2. Confirm usage errors exit 2 and never call Docker. - [ ] **Step 2: Verify RED** ```bash cd tools/thothctl go test ./internal/workspaceops ./cmd/thothctl -run 'Collection|collection|Confirmation' ``` Expected: FAIL because P2 parsing knows no `workspace collection` family. - [ ] **Step 3: Extend `workspaceops.ParseWorkspaceCommand` and main dispatch** Use typed operations `CollectionRebuild` and `CollectionRecover`; do not add another top-level parser. Perform syntactic equality checks in Go before Docker, then pass all values to TypeScript for authoritative descriptor equality checks. - [ ] **Step 4: Extend inspect parsing/output** P2 `workspace inspect` keeps the same command syntax. Validate the returned semantic/recovery JSON structure and print pristine JSON under `--json`; human output lists exact workspace, revision, collection, state, missing/incompatible fields, maintenance, and recovery phase without endpoint details. - [ ] **Step 5: Verify** ```bash cd tools/thothctl go test ./internal/workspaceops ./cmd/thothctl ``` Expected: PASS. - [ ] **Step 6: Commit** ```bash git add tools/thothctl/internal/workspaceops/operations.go \ tools/thothctl/internal/workspaceops/operations_test.go \ tools/thothctl/internal/workspaceops/collection.go \ tools/thothctl/internal/workspaceops/collection_test.go \ tools/thothctl/cmd/thothctl/main.go tools/thothctl/cmd/thothctl/main_test.go git commit -m "feat: add qdrant collection operator commands" ``` --- ### Task 7: Orchestrate maintenance, inventory, stop, rebuild, restart, and recovery **Files:** - Modify: `tools/thothctl/internal/workspaceops/collection.go` - Modify: `tools/thothctl/internal/workspaceops/collection_test.go` - Modify: `tools/thothctl/internal/workspaceops/operations.go` - Modify: `tools/thothctl/internal/workspaceops/operations_test.go` - Modify: `tools/thothctl/cmd/thothctl/main.go` - Modify: `tools/thothctl/cmd/thothctl/main_test.go` - Modify: `tools/thothctl/internal/pi/update.go` only to share strict maintenance parsing/helpers where appropriate - Modify: `tools/thothctl/internal/pi/update_test.go` **Required sequence for rebuild:** ```text Acquire installation lifecycle lock → workspace inspect and authoritative confirmation recheck → POST core /internal/maintenance/activate → GET complete /sessions?scope=mine(local)|all(server) → reject unless every item is archived OR status closed/finalized → poll /internal/maintenance/quiescence until active + admissions=0 + piProcesses=0 → compose stop core → compose ps --status running -q core must be empty → compose --profile workspace-maintenance run --rm --no-deps -T workspace-maintenance collection-rebuild → parse exact safe JSON and require verified state → compose up --detach --no-deps core → poll Compose health and loopback /health → require maintenance still active and quiescent → POST /internal/maintenance/deactivate → require active=false → release lifecycle lock ``` Use a bounded monotonic timeout (document 60 seconds for quiescence and 120 seconds for core health), one-second polling, and **no retry of a failed operation**. Polling observation is not retrying a mutation. - [ ] **Step 1: Write RED orchestration tests with a scripted runner** Assert exact argv/order and failure behavior for: - confirmation mismatch before lock/HTTP; - lifecycle contention with Pi; - activation durability failure; - malformed/incomplete/duplicate session inventory; - open/failed unarchived session refusal; archived, closed, and finalized acceptance; - `admissions > 0` and `piProcesses > 0` poll, then success; - timeout without forced teardown; - core stop failure or still-running proof; - P2 writer-lock conflict returned by the maintenance service before deletion, followed by safe core restart/health/maintenance clear; - successful operator `verified` response, exact core restart, health, and marker clear; - malformed/ambiguous operator result treated as mutation-started/recovery-required unless a trusted response proves `mutation_started:false`; - post-delete error leaves core stopped, marker active, state retained, and prints exact recover command; - neighbor services/collections never appear in stop/delete argv; - all child stderr is sanitized and bounded. - [ ] **Step 2: Verify RED** ```bash cd tools/thothctl go test ./internal/workspaceops -run 'Rebuild|Quiescence|Inventory|Recovery' ``` Expected: FAIL because lifecycle orchestration is not implemented. - [ ] **Step 3: Implement strict inventory and quiescence clients** Reuse the existing core-side curl identity headers. Decode exactly one bounded JSON document. A complete local install uses `scope=mine`; server uses admin `scope=all`. Never infer quiescence only from manifests: both `admissions` and `piProcesses` must reach zero after the durable marker is active. - [ ] **Step 4: Implement core stop/start proof and fail-safe cleanup** Use fixed Compose argv and explicit `core`. Do not call `down`, stop Qdrant, stop frontend, use `--remove-orphans`, or prune. Clear maintenance only after a trusted pre-delete failure or complete verified success and healthy core. - [ ] **Step 5: Write RED recovery tests** Test core-already-stopped recovery, core-running-but-maintained recovery, missing marker refusal, no/nonterminal/verified state, descriptor revision drift, recovery service failure, compatible-after-crash verification, missing recreate, explicitly confirmed incompatible recreate, successful restart/health/deactivate, and repeated recover returning `collection_recovery_not_required` without mutation. - [ ] **Step 6: Implement recovery** When core is stopped, the durable marker plus matching nonterminal state is the admission barrier; the maintenance service re-verifies both before mutation. When core is running, repeat activation/inventory/quiescence before stopping it. Never clear the marker merely because recovery cannot read state. - [ ] **Step 7: Run Go suites** ```bash cd tools/thothctl go test ./internal/workspaceops ./internal/lifecycle ./internal/pi ./cmd/thothctl go test ./... ``` Expected: PASS. - [ ] **Step 8: Commit** ```bash git add tools/thothctl/internal/workspaceops/collection.go \ tools/thothctl/internal/workspaceops/collection_test.go \ tools/thothctl/internal/workspaceops/operations.go \ tools/thothctl/internal/workspaceops/operations_test.go \ tools/thothctl/internal/pi/update.go tools/thothctl/internal/pi/update_test.go \ tools/thothctl/cmd/thothctl/main.go tools/thothctl/cmd/thothctl/main_test.go git commit -m "feat: orchestrate recoverable qdrant rebuilds" ``` --- ### Task 8: Harden the dedicated maintenance service and Compose contract **Files:** - Modify: `compose.yaml` (`workspace-maintenance` only) - Modify: `deploy/compose.server.yaml` (`workspace-maintenance` bind roots) - Verify/modify as P2 created them: `deploy/compose.git-https.yaml`, `deploy/compose.git-ssh.yaml` - Modify: `docker/core.Dockerfile` (ensure `/usr/bin/flock` from `util-linux` is present) - Modify: `scripts/test-internal-semantic-compose.sh` - Modify: `scripts/test-unified-compose.sh` - Modify: `scripts/test-compose-secret-policy.sh` - Modify: `scripts/test-no-deployment-coupling.sh` - Modify: `scripts/test-deployment-command-contract.sh` The service gains the shared settings mount needed to read the durable maintenance marker. It retains P2's sessions/registry mounts, internal Qdrant network, same selected core image, and direct Node entrypoint. It must still have no `/home/thoth/.pi`, PI auth/models/settings, writable Pi state, frontend dependency, host Docker socket, published port, or normal core entrypoint/trust initialization. - [ ] **Step 1: Extend RED Compose contract tests** Assert local named volumes and server bind roots make the **same** `/data/settings`, `/data/sessions`, and `/data/workspace-registry` visible to `core` and `workspace-maintenance`; Qdrant is reachable only on the private network; only the exact required Git/connector secret files attach; service is profile-only; `flock` exists in the built core image; and service cannot start Pi or expose HTTP. - [ ] **Step 2: Verify RED** ```bash ./scripts/test-internal-semantic-compose.sh ./scripts/test-unified-compose.sh ./scripts/test-compose-secret-policy.sh ./scripts/test-deployment-command-contract.sh ``` Expected: at least the settings/marker and `flock` assertions FAIL. - [ ] **Step 3: Make minimal Compose/image changes** Add `util-linux` explicitly rather than relying on a transitive base package. Extend the P2 service only; do not introduce a second maintenance service or mount Pi state. - [ ] **Step 4: Run deployment gates** ```bash ./scripts/test-default-compose.sh ./scripts/test-unified-compose.sh ./scripts/test-internal-semantic-compose.sh ./scripts/test-compose-secret-policy.sh ./scripts/test-no-deployment-coupling.sh ./scripts/test-deployment-command-contract.sh ``` Expected: PASS. - [ ] **Step 5: Commit** ```bash git add compose.yaml deploy/compose.server.yaml \ deploy/compose.git-https.yaml deploy/compose.git-ssh.yaml \ docker/core.Dockerfile scripts/test-internal-semantic-compose.sh \ scripts/test-unified-compose.sh scripts/test-compose-secret-policy.sh \ scripts/test-no-deployment-coupling.sh scripts/test-deployment-command-contract.sh git commit -m "build: isolate qdrant maintenance service" ``` --- ### Task 9: Build the clean-state P4 process goal and retained report **Files:** - Create: `scripts/p4-acceptance.sh` - Create: `scripts/test-p4-acceptance.sh` - Create: `backend/scripts/p4-acceptance.mjs` - Create: `backend/scripts/p4-acceptance.test.mjs` - Create: `backend/scripts/fixtures/p4-qdrant-proxy.mjs` - Modify: `.gitignore` only if the existing `.artifacts/` rule is insufficient **Owned layout:** ```text .artifacts/p4-integration// ├── ownership.json ├── source.json ├── installation/thothii-installation.yaml ├── registry/{remote.git,author}/ ├── fixture-secrets/ ├── compose/acceptance.override.yaml ├── requests/ ├── responses/ ├── observations/ ├── recovery/qdrant-rebuild-state.json ├── logs/ ├── artifact-manifest.json ├── cleanup.json ├── report.json └── report.md ``` - [ ] **Step 1: Write RED wrapper/orchestrator tests** Test argument parsing (`integration [--keep]` only), dirty tracked-tree refusal, unique run/project naming, fixture-only credentials, fixed toolchain resolution, no retry loop, bounded output, ownership checks, report schema, artifact hashes, secret-canary scan, cleanup behavior on success/failure/signal, and `--keep` retaining only the owned filesystem root. - [ ] **Step 2: Verify RED** ```bash ./scripts/test-p4-acceptance.sh ``` Expected: FAIL because the wrapper/orchestrator are missing. - [ ] **Step 3: Implement the isolated topology** Use a real local bare Git remote and schema-v3 descriptor, compiled core image, real Qdrant storage, and production `thothctl`. An acceptance-only private proxy named `qdrant` fronts a real uniquely named Qdrant service; it forwards normal REST calls and provides a deterministic, test-owned barrier after DELETE so the one-shot maintenance container can be killed after durable `mutation_started` state and before recreate. It must not alter production code paths or accept secrets. Ollama/DWH/LLM use safe fixtures because they are outside D4. - [ ] **Step 4: Implement positive and negative checks** Record named PASS checks for every assertion in the automated goal. For the session self-heal check, traverse the real backend admission route and intentionally stop at a later fixture readiness boundary only after Qdrant creation has been final-read compatible; assert no session manifest/Pi process survives. For concurrency, prove both layers separately: coordinate two direct real manager calls through the proxy so one compatible create/index conflicts, and send two simultaneous local same-principal route admissions to prove `ReadinessManager` performs exactly one create. Also run the upstream-auth route with two distinct principals if the cross-principal route proof is retained. Do not loop/retry until green and do not claim two local same-principal route calls both reached Qdrant. Preseed one target point and one neighbor collection/point. Successful rebuild must remove/recreate only the target, leave the neighbor byte/count identity unchanged, retain canonical source files, and report that reindexing is required. - [ ] **Step 5: Implement interruption and recovery check** Block recreate after the exact target DELETE, kill only the uniquely labelled one-shot container, and assert: command failure; core stopped; maintenance marker active; state `mutation_started:true`; admissions refused if core is deliberately restarted under the marker. Release the proxy barrier and run the exact confirmed `collection recover` once; assert verified collection, healthy core, inactive marker, terminal state, and neighbor preservation. - [ ] **Step 6: Implement secret scan, artifact binding, and exact cleanup** Scan every retained regular file and bounded report/log field for all fixture secret canaries and credential-shaped URLs. Hash all declared artifacts, then remove only containers/volumes/network/proxy image/reference carrying this run's exact Compose project/run label. Prove no owned resource remains and no pre-existing resource was changed. There is no global `docker system prune`, volume prefix wildcard, or broad process kill. - [ ] **Step 7: Verify harness syntax/unit contract** ```bash bash -n scripts/p4-acceptance.sh scripts/test-p4-acceptance.sh node --check backend/scripts/p4-acceptance.mjs node --check backend/scripts/p4-acceptance.test.mjs node --check backend/scripts/fixtures/p4-qdrant-proxy.mjs ./scripts/test-p4-acceptance.sh ``` Expected: PASS without starting the full process from the unit-contract test. - [ ] **Step 8: Commit the process goal** ```bash git add scripts/p4-acceptance.sh scripts/test-p4-acceptance.sh \ backend/scripts/p4-acceptance.mjs backend/scripts/p4-acceptance.test.mjs \ backend/scripts/fixtures/p4-qdrant-proxy.mjs .gitignore git commit -m "test: add p4 qdrant lifecycle acceptance" ``` - [ ] **Step 9: Confirm the goal is runnable, but defer the one authoritative full run** ```bash git status --short ./scripts/test-p4-acceptance.sh ``` Expected: clean status and PASS. Do not launch the authoritative Docker/process run yet: Task 10 still adds the manual helper/docs included in the final source boundary. The one retained release run occurs in Task 11 from the final clean implementation/manual-documentation commit. --- ### Task 10: Publish the independent manual walkthrough **Files:** - Modify: `docs/testing/p2-p6-manual-verification.md` (replace only the P4 placeholder section) - Create: `scripts/p4-manual-verification.sh` - Create: `scripts/test-p4-manual-verification.sh` - Modify: `docs/install/local-workspace-registry.md` - Modify: `docs/install/server-workspace-registry.md` - Modify: `tools/thothctl/cmd/thothctl/main.go` usage text if not already complete - [ ] **Step 1: Write RED manual-helper contract tests** The helper accepts only `prepare`, `interrupt-after-delete`, `release-recovery`, `status`, and `cleanup`; owns `.artifacts/manual-acceptance/p4`; refuses reuse without cleanup; emits `GUIDE.md`; uses a new Git remote/Compose project/volumes distinct from automated state; never performs the reviewer commands or records PASS on the reviewer's behalf. ```bash ./scripts/test-p4-manual-verification.sh ``` Expected: FAIL until the helper exists. - [ ] **Step 2: Implement prepare/status/fault-control/cleanup only** `prepare` builds the fixture and prints exact non-secret variables. `interrupt-after-delete` arms the deterministic fixture barrier but does not invoke rebuild. `release-recovery` releases only that barrier. `cleanup` removes only the manual run's labelled resources/root after explicit reviewer confirmation. No helper calls `workspace collection rebuild/recover` for the reviewer. - [ ] **Step 3: Replace the P4 walkthrough placeholder with exact commands** Document, in order: 1. prepare new manual state; 2. `workspace inspect --json` on missing collection; 3. trigger one admission and inspect exact self-healed contract; 4. delete one safe fixture keyword index, trigger admission, and inspect its repair; 5. seed dimension, distance, and index-type incompatibilities and record nonmutation refusal; 6. run workspace/collection confirmation mismatches and prove marker/core/collection unchanged; 7. create a fixture open session and prove `session_inventory_active`; 8. hold the P2 writer lock and prove `preprocessing_conflict` before delete with safe restart/clear; 9. run the exact confirmed rebuild, inspect terminal state, neighbor preservation, healthy core, and inactive maintenance; 10. arm interruption, run rebuild, observe core stopped/maintenance retained/recovery state, release barrier, run exact confirmed recovery, and verify final state; 11. inspect report/state without secrets, decide PASS/FAIL, then cleanup. For every step explain the component crossed, expected JSON fields/code, state/artifact produced, and invariant. Include `Decision: **PENDING**` and blank reviewer/date/evidence fields; only the human changes it to PASS/FAIL. - [ ] **Step 4: Update operator manuals** Explain self-heal vs destructive rebuild, exact confirmations, closed/finalized/archived inventory, lifecycle lock contention with Pi update, maintenance marker semantics, writer-lock refusal, post-delete recovery, intentional vector loss/reindex sequence, and backup recommendation. Do not claim P5/P6 materialization or automated reindexing. - [ ] **Step 5: Verify docs/helper** ```bash bash -n scripts/p4-manual-verification.sh scripts/test-p4-manual-verification.sh ./scripts/test-p4-manual-verification.sh ./scripts/verify-workspace-install-docs.sh --fixtures-only ``` Expected: PASS; the living P4 section contains no placeholder text and remains PENDING. - [ ] **Step 6: Commit** ```bash git add docs/testing/p2-p6-manual-verification.md \ docs/install/local-workspace-registry.md docs/install/server-workspace-registry.md \ scripts/p4-manual-verification.sh scripts/test-p4-manual-verification.sh \ tools/thothctl/cmd/thothctl/main.go git commit -m "docs: add p4 qdrant lifecycle walkthrough" ``` --- ### Task 11: Final verification, checkpoint report, and hard stop **Files:** - Create after green evidence exists: `docs/testing/p4-qdrant-bootstrap-rebuild-checkpoint.md` - Modify: `PROJECT_STATE.md` - [ ] **Step 1: Run the scoped P4 verification from a clean commit** ```bash git status --short cd backend && npx vitest run \ test/qdrant-collection-manager.test.ts \ test/tht-qdrant-readiness.test.ts \ test/readiness-manager.test.ts \ test/routes-sessions.test.ts \ test/maintenance-gate.test.ts \ test/pi-process-manager.test.ts \ test/qdrant-rebuild-state.test.ts \ test/workspace-preprocessing-state.test.ts \ test/workspace-preprocessing-service.test.ts \ test/workspace-maintenance.test.ts cd backend && npx tsc --noEmit -p . && npm run build cd tools/thothctl && go test ./... ./scripts/test-default-compose.sh ./scripts/test-unified-compose.sh ./scripts/test-internal-semantic-compose.sh ./scripts/test-compose-secret-policy.sh ./scripts/test-no-deployment-coupling.sh ./scripts/test-deployment-command-contract.sh ./scripts/test-p4-acceptance.sh ./scripts/test-p4-manual-verification.sh ./scripts/verify-workspace-install-docs.sh --fixtures-only git diff --check git status --short ``` Expected: every command exits 0; TypeScript/build and all Go tests PASS; final tracked status is clean. Do not run or claim the aggregate P2–P6 smoke, full harness/frontend suites, real PSD, or P5/P6 flows—those remain the post-P6 gate. - [ ] **Step 2: Run the one authoritative clean-state process and retain it** ```bash ./scripts/p4-acceptance.sh integration --keep ``` Expected: exit 0; `report.json` and `report.md` say every named check PASS, `automated integration: PASS`, `manual acceptance: PENDING`, `secret_scan: PASS`, `cleanup: PASS`, and `retry_count: 0`. If it fails, retain the failed run, diagnose the root cause, add a focused regression, fix and commit, rerun Step 1, then launch a **new run ID from the beginning**—never resume a partial run or retry a mutation inside a run. - [ ] **Step 3: Verify the retained process report independently** Check the retained P4 run is bound to the clean Step 1 source commit/tree, contains one scenario execution and no automatic retry, has all named checks PASS, secret scan PASS, exact cleanup PASS, a valid artifact SHA-256 manifest, no undeclared regular files, and no owned live Docker resource. Record `report.json` and `report.md` hashes. - [ ] **Step 4: Write the checkpoint report** `docs/testing/p4-qdrant-bootstrap-rebuild-checkpoint.md` must state: - source commit/tree and plan/design/PRD references; - exact commands and outcomes (do not invent counts); - retained report path and hashes; - self-heal, manager compatible-creator race, local route deduplication, distinct-principal route convergence, and incompatibility evidence; - lifecycle lock, inventory, admissions/Pi quiescence, writer lock, stop/restart evidence; - successful rebuild and interrupted recovery evidence; - secret-scan and exact-cleanup evidence; - `automated integration: PASS` and `manual acceptance: PENDING`; - explicit P5/P6/post-P6 exclusions and any real-platform checks not run. Update only the evolving P4 section of `PROJECT_STATE.md` with the same truthful status. - [ ] **Step 5: Commit checkpoint documentation** ```bash git add docs/testing/p4-qdrant-bootstrap-rebuild-checkpoint.md PROJECT_STATE.md git commit -m "docs: record p4 automated checkpoint" ``` The retained run is intentionally bound to this checkpoint commit's parent: the final clean implementation and manual-documentation source boundary. The docs-only checkpoint records immutable report hashes and is not falsely claimed as input to its own report. - [ ] **Step 6: Complete the persistent process goal and stop** Require a clean worktree, then complete the goal only after the authoritative run and cleanup proof. Send the user the checkpoint report and exact P4 manual section. Stop for explicit authorization/manual decision; do not begin P5, P6, aggregate verification, or unrelated cleanup. ## Manual acceptance completion (later, reviewer-owned) After the reviewer executes the independent P4 walkthrough, record only their actual decision and evidence. If PASS, update the P4 checkpoint and `PROJECT_STATE.md` from `manual acceptance: PENDING` to PASS in one docs-only commit. If FAIL, retain the failure evidence, reopen the P4 goal, add a focused regression, and repeat the entire automated process from a new clean run before asking for another manual decision.