diff --git a/docs/superpowers/plans/2026-08-11-p4-qdrant-collection-lifecycle.md b/docs/superpowers/plans/2026-08-11-p4-qdrant-collection-lifecycle.md new file mode 100644 index 00000000..e8d0f537 --- /dev/null +++ b/docs/superpowers/plans/2026-08-11-p4-qdrant-collection-lifecycle.md @@ -0,0 +1,61 @@ +# P4 Qdrant Collection Lifecycle — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development (recommended) or superpowers:executing-plans. Use TDD and verification-before-completion. + +**Goal:** Own the Qdrant collection lifecycle with one shared manager: self-heal a missing/incomplete collection at session admission and in the operator path, refuse incompatible collections, and provide a guarded host CLI for inspection and destructive rebuild under a durable maintenance/quiescence protocol. + +**Architecture:** A shared TypeScript collection manager (`qdrant-collection.ts`) reconciles collection + payload keyword indexes. Session admission (`qdrantEnsure`) uses it to self-heal (create missing, add missing indexes) but never mutates incompatible collections (`semantic_index_incompatible`). The operator path uses the same manager with `require_existing` (no auto-create outside admission). `thothctl workspace vector inspect|rebuild` drive the maintenance service; rebuild requires exact workspace id + collection name confirmation + explicit destructive flag, and runs under a backend-mediated maintenance marker with a loopback quiescence endpoint (admission leases and PiProcessManager count zero), stopping core before the destructive job. + +**Tech Stack:** TypeScript 5, Node 22, Qdrant HTTP API, Go (thothctl), Fastify, Vitest, Go tests, Bash. + +**Source contract:** `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` §6 (P4), PRD D4, and P3 effective-config/revision contracts. + +## P4 completion contract + +1. One shared TS manager owns collection create/index reconciliation and validation; both session admission and the operator use it (operator path stays `require_existing`). +2. Admission self-heal: a missing collection is created with exactly 1024 dimensions, cosine distance, and the 8 required keyword payload indexes; missing indexes are added; an already-compatible concurrent creator is tolerated (re-read final state). +3. Incompatible dimensions/distance/index types are never mutated: admission and operator both return `semantic_index_incompatible`. +4. `thothctl workspace vector inspect --workspace [--json]` reports the collection contract without mutation. +5. `thothctl workspace vector rebuild --workspace --collection --confirm --destroy` deletes only the descriptor-owned collection and recreates the complete contract, under: installation lifecycle lock; backend maintenance marker activated durably; session inventory all closed/finalized/archived; loopback quiescence endpoint with zero admission leases and zero PiProcessManager count; core stopped and rechecked; no preprocessing lock held; durable rebuild state written before deletion. No prefix matching or global Qdrant mutation. +6. Failure after deletion leaves maintenance active and provides an explicit recovery/recreate command (never claims rollback of lost data); success restarts core and clears maintenance only after health verification. +7. Memory/solved and schema/Evidence payload contracts (P3 revision scoping) are preserved by the recreated collection. +8. A clean-state P4 automated process goal passes; P4 manual walkthrough stays PENDING. No P5/P6 work. + +## Target file map + +**TS collection manager + admission** +- Create `backend/src/workspaces/qdrant-collection.ts`, `qdrant-collection.test.ts`. +- Modify `backend/src/tht/tht-runner.ts` (`qdrantEnsure`) to use the manager (self-heal). +- Modify `backend/src/workspaces/preprocessing-service.ts` (operator path uses manager with `require_existing`). +- Modify `backend/src/runtime/maintenance-gate.ts`/maintenance state and add a loopback internal quiescence endpoint + admission-lease drain, tests. + +**Go host CLI** +- Modify `tools/thothctl/internal/workspaceops/operations.go` (+tests): `workspace vector inspect`, `workspace vector rebuild` with exact grammar, confirmation, destructive flag, lifecycle lock, maintenance activation (calls backend loopback), quiescence polling, stop/start core, durable rebuild state, recovery command. + +**Docs** +- Modify `docs/contracts/workspace-preprocessing-cli.md`, install manuals, `docs/testing/p2-p6-manual-verification.md` (P4 section). +- Modify after evidence: `PROJECT_STATE.md`. + +**Acceptance** +- Create `scripts/p4-acceptance.sh`, `scripts/test-p4-acceptance.sh`, `backend/scripts/p4-acceptance.mjs`, `backend/scripts/p4-acceptance.test.mjs` (pattern: P3 acceptance runner). + +### Task 1 — Shared TS collection manager (create/reconcile/validate) +Failing tests: creates missing collection with 1024/cosine; adds missing keyword indexes; tolerates concurrent compatible creator (re-read); refuses incompatible dimensions/distance/index with `semantic_index_incompatible`; never mutates incompatible. Implement manager over the Qdrant HTTP API; wire into `qdrantEnsure` (self-heal) and the operator (`require_existing`). Commit. + +### Task 2 — Durable maintenance marker + loopback quiescence endpoint +Failing tests: backend can durably activate maintenance (marker survives restart); a new loopback-only internal endpoint reports admission leases and PiProcessManager count; drain waits until both are zero; a maintenance marker refuses new admission; health/restart behavior defined. Implement; keep the endpoint loopback-only and unauthenticated-but-internal. Commit. + +### Task 3 — thothctl vector inspect and guarded rebuild +Failing tests (Go): `workspace vector inspect` reads the collection contract and emits pristine JSON; `workspace vector rebuild` requires exact `--workspace`, `--collection`, `--confirm `, `--destroy`; refuses mismatched confirmation or missing `--destroy`; acquires the installation lifecycle lock; asks the backend to activate maintenance; verifies session inventory closed; polls quiescence; stops core, rechecks; verifies no preprocessing lock; deletes only the descriptor collection; recreates + verifies; writes durable rebuild state before deletion; restarts core and clears maintenance only after health; on failure after deletion keeps maintenance and prints the explicit recovery command. No prefix/global mutation. Commit. + +### Task 4 — Docs + P4 walkthrough +Update `workspace-preprocessing-cli.md`, install manuals, and the P4 section of `docs/testing/p2-p6-manual-verification.md` (decision PENDING). Commit. + +### Task 5 — Clean-state P4 automated process goal +P4 acceptance runner (pattern P3): bootstrap fixtures (P1.1 registry + REST DWH + HTTP evidence + embedding stub); pre-provision Qdrant; run thothctl product commands; prove: admission self-heal creates the collection on a fresh volume, incompatible collection refused, `vector inspect` contract, `vector rebuild` guarded flow with confirmation/destroy and exact cleanup, collection recreated with the 8 indexes + revision-scoped payload contract, no P5/P6 scope, secret scan, cleanup. Run runner tests, then one clean integration run without retry; report ends `P4 automated integration: PASS` / `P4 manual acceptance: PENDING`. Commit. + +### Task 6 — Final verification + owner handoff +Full backend/frontend/harness/Go gates; one final clean P4 acceptance run; update `PROJECT_STATE.md`; stop. No P5 work without a new authorization. + +## Exclusions +No Evidence materialization (P6), no Git FK annotations (P5), no changes to accepted P1.1/P2/P3 evidence, no migration of real PSD content.