docs: plan P4 qdrant collection lifecycle

2026-08-12 18:55:33 +02:00
parent 4912b49f28
commit 230a876314
1 changed files with 61 additions and 0 deletions
@@ -0,0 +1,61 @@
+# P4 Qdrant Collection Lifecycle — Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:subagent-driven-development (recommended) or superpowers:executing-plans. Use TDD and verification-before-completion.
+
+**Goal:** Own the Qdrant collection lifecycle with one shared manager: self-heal a missing/incomplete collection at session admission and in the operator path, refuse incompatible collections, and provide a guarded host CLI for inspection and destructive rebuild under a durable maintenance/quiescence protocol.
+
+**Architecture:** A shared TypeScript collection manager (`qdrant-collection.ts`) reconciles collection + payload keyword indexes. Session admission (`qdrantEnsure`) uses it to self-heal (create missing, add missing indexes) but never mutates incompatible collections (`semantic_index_incompatible`). The operator path uses the same manager with `require_existing` (no auto-create outside admission). `thothctl workspace vector inspect|rebuild` drive the maintenance service; rebuild requires exact workspace id + collection name confirmation + explicit destructive flag, and runs under a backend-mediated maintenance marker with a loopback quiescence endpoint (admission leases and PiProcessManager count zero), stopping core before the destructive job.
+
+**Tech Stack:** TypeScript 5, Node 22, Qdrant HTTP API, Go (thothctl), Fastify, Vitest, Go tests, Bash.
+
+**Source contract:** `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` §6 (P4), PRD D4, and P3 effective-config/revision contracts.
+
+## P4 completion contract
+
+1. One shared TS manager owns collection create/index reconciliation and validation; both session admission and the operator use it (operator path stays `require_existing`).
+2. Admission self-heal: a missing collection is created with exactly 1024 dimensions, cosine distance, and the 8 required keyword payload indexes; missing indexes are added; an already-compatible concurrent creator is tolerated (re-read final state).
+3. Incompatible dimensions/distance/index types are never mutated: admission and operator both return `semantic_index_incompatible`.
+4. `thothctl workspace vector inspect --workspace <id> [--json]` reports the collection contract without mutation.
+5. `thothctl workspace vector rebuild --workspace <id> --collection <name> --confirm <name> --destroy` deletes only the descriptor-owned collection and recreates the complete contract, under: installation lifecycle lock; backend maintenance marker activated durably; session inventory all closed/finalized/archived; loopback quiescence endpoint with zero admission leases and zero PiProcessManager count; core stopped and rechecked; no preprocessing lock held; durable rebuild state written before deletion. No prefix matching or global Qdrant mutation.
+6. Failure after deletion leaves maintenance active and provides an explicit recovery/recreate command (never claims rollback of lost data); success restarts core and clears maintenance only after health verification.
+7. Memory/solved and schema/Evidence payload contracts (P3 revision scoping) are preserved by the recreated collection.
+8. A clean-state P4 automated process goal passes; P4 manual walkthrough stays PENDING. No P5/P6 work.
+
+## Target file map
+
+**TS collection manager + admission**
+- Create `backend/src/workspaces/qdrant-collection.ts`, `qdrant-collection.test.ts`.
+- Modify `backend/src/tht/tht-runner.ts` (`qdrantEnsure`) to use the manager (self-heal).
+- Modify `backend/src/workspaces/preprocessing-service.ts` (operator path uses manager with `require_existing`).
+- Modify `backend/src/runtime/maintenance-gate.ts`/maintenance state and add a loopback internal quiescence endpoint + admission-lease drain, tests.
+
+**Go host CLI**
+- Modify `tools/thothctl/internal/workspaceops/operations.go` (+tests): `workspace vector inspect`, `workspace vector rebuild` with exact grammar, confirmation, destructive flag, lifecycle lock, maintenance activation (calls backend loopback), quiescence polling, stop/start core, durable rebuild state, recovery command.
+
+**Docs**
+- Modify `docs/contracts/workspace-preprocessing-cli.md`, install manuals, `docs/testing/p2-p6-manual-verification.md` (P4 section).
+- Modify after evidence: `PROJECT_STATE.md`.
+
+**Acceptance**
+- Create `scripts/p4-acceptance.sh`, `scripts/test-p4-acceptance.sh`, `backend/scripts/p4-acceptance.mjs`, `backend/scripts/p4-acceptance.test.mjs` (pattern: P3 acceptance runner).
+
+### Task 1 — Shared TS collection manager (create/reconcile/validate)
+Failing tests: creates missing collection with 1024/cosine; adds missing keyword indexes; tolerates concurrent compatible creator (re-read); refuses incompatible dimensions/distance/index with `semantic_index_incompatible`; never mutates incompatible. Implement manager over the Qdrant HTTP API; wire into `qdrantEnsure` (self-heal) and the operator (`require_existing`). Commit.
+
+### Task 2 — Durable maintenance marker + loopback quiescence endpoint
+Failing tests: backend can durably activate maintenance (marker survives restart); a new loopback-only internal endpoint reports admission leases and PiProcessManager count; drain waits until both are zero; a maintenance marker refuses new admission; health/restart behavior defined. Implement; keep the endpoint loopback-only and unauthenticated-but-internal. Commit.
+
+### Task 3 — thothctl vector inspect and guarded rebuild
+Failing tests (Go): `workspace vector inspect` reads the collection contract and emits pristine JSON; `workspace vector rebuild` requires exact `--workspace`, `--collection`, `--confirm <same>`, `--destroy`; refuses mismatched confirmation or missing `--destroy`; acquires the installation lifecycle lock; asks the backend to activate maintenance; verifies session inventory closed; polls quiescence; stops core, rechecks; verifies no preprocessing lock; deletes only the descriptor collection; recreates + verifies; writes durable rebuild state before deletion; restarts core and clears maintenance only after health; on failure after deletion keeps maintenance and prints the explicit recovery command. No prefix/global mutation. Commit.
+
+### Task 4 — Docs + P4 walkthrough
+Update `workspace-preprocessing-cli.md`, install manuals, and the P4 section of `docs/testing/p2-p6-manual-verification.md` (decision PENDING). Commit.
+
+### Task 5 — Clean-state P4 automated process goal
+P4 acceptance runner (pattern P3): bootstrap fixtures (P1.1 registry + REST DWH + HTTP evidence + embedding stub); pre-provision Qdrant; run thothctl product commands; prove: admission self-heal creates the collection on a fresh volume, incompatible collection refused, `vector inspect` contract, `vector rebuild` guarded flow with confirmation/destroy and exact cleanup, collection recreated with the 8 indexes + revision-scoped payload contract, no P5/P6 scope, secret scan, cleanup. Run runner tests, then one clean integration run without retry; report ends `P4 automated integration: PASS` / `P4 manual acceptance: PENDING`. Commit.
+
+### Task 6 — Final verification + owner handoff
+Full backend/frontend/harness/Go gates; one final clean P4 acceptance run; update `PROJECT_STATE.md`; stop. No P5 work without a new authorization.
+
+## Exclusions
+No Evidence materialization (P6), no Git FK annotations (P5), no changes to accepted P1.1/P2/P3 evidence, no migration of real PSD content.