ThothII/docs/superpowers/plans/2026-08-11-p4-qdrant-collection-lifecycle.md

P4 Qdrant Collection Lifecycle — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: superpowers:subagent-driven-development (recommended) or superpowers:executing-plans. Use TDD and verification-before-completion.

Goal: Own the Qdrant collection lifecycle with one shared manager: self-heal a missing/incomplete collection at session admission and in the operator path, refuse incompatible collections, and provide a guarded host CLI for inspection and destructive rebuild under a durable maintenance/quiescence protocol.

Architecture: A shared TypeScript collection manager (qdrant-collection.ts) reconciles collection + payload keyword indexes. Session admission (qdrantEnsure) uses it to self-heal (create missing, add missing indexes) but never mutates incompatible collections (semantic_index_incompatible). The operator path uses the same manager with require_existing (no auto-create outside admission). thothctl workspace vector inspect|rebuild drive the maintenance service; rebuild requires exact workspace id + collection name confirmation + explicit destructive flag, and runs under a backend-mediated maintenance marker with a loopback quiescence endpoint (admission leases and PiProcessManager count zero), stopping core before the destructive job.

Tech Stack: TypeScript 5, Node 22, Qdrant HTTP API, Go (thothctl), Fastify, Vitest, Go tests, Bash.

Source contract: docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md §6 (P4), PRD D4, and P3 effective-config/revision contracts.

P4 completion contract

1. One shared TS manager owns collection create/index reconciliation and validation; both session admission and the operator use it (operator path stays require_existing).
2. Admission self-heal: a missing collection is created with exactly 1024 dimensions, cosine distance, and the 8 required keyword payload indexes; missing indexes are added; an already-compatible concurrent creator is tolerated (re-read final state).
3. Incompatible dimensions/distance/index types are never mutated: admission and operator both return semantic_index_incompatible.
4. thothctl workspace vector inspect --workspace <id> [--json] reports the collection contract without mutation.
5. thothctl workspace vector rebuild --workspace <id> --collection <name> --confirm <name> --destroy deletes only the descriptor-owned collection and recreates the complete contract, under: installation lifecycle lock; backend maintenance marker activated durably; session inventory all closed/finalized/archived; loopback quiescence endpoint with zero admission leases and zero PiProcessManager count; core stopped and rechecked; no preprocessing lock held; durable rebuild state written before deletion. No prefix matching or global Qdrant mutation.
6. Failure after deletion leaves maintenance active and provides an explicit recovery/recreate command (never claims rollback of lost data); success restarts core and clears maintenance only after health verification.
7. Memory/solved and schema/Evidence payload contracts (P3 revision scoping) are preserved by the recreated collection.
8. A clean-state P4 automated process goal passes; P4 manual walkthrough stays PENDING. No P5/P6 work.

Target file map

TS collection manager + admission

• Create backend/src/workspaces/qdrant-collection.ts, qdrant-collection.test.ts.
• Modify backend/src/tht/tht-runner.ts (qdrantEnsure) to use the manager (self-heal).
• Modify backend/src/workspaces/preprocessing-service.ts (operator path uses manager with require_existing).
• Modify backend/src/runtime/maintenance-gate.ts/maintenance state and add a loopback internal quiescence endpoint + admission-lease drain, tests.

Go host CLI

• Modify tools/thothctl/internal/workspaceops/operations.go (+tests): workspace vector inspect, workspace vector rebuild with exact grammar, confirmation, destructive flag, lifecycle lock, maintenance activation (calls backend loopback), quiescence polling, stop/start core, durable rebuild state, recovery command.

Docs

• Modify docs/contracts/workspace-preprocessing-cli.md, install manuals, docs/testing/p2-p6-manual-verification.md (P4 section).
• Modify after evidence: PROJECT_STATE.md.

Acceptance

• Create scripts/p4-acceptance.sh, scripts/test-p4-acceptance.sh, backend/scripts/p4-acceptance.mjs, backend/scripts/p4-acceptance.test.mjs (pattern: P3 acceptance runner).

Task 1 — Shared TS collection manager (create/reconcile/validate)

Failing tests: creates missing collection with 1024/cosine; adds missing keyword indexes; tolerates concurrent compatible creator (re-read); refuses incompatible dimensions/distance/index with semantic_index_incompatible; never mutates incompatible. Implement manager over the Qdrant HTTP API; wire into qdrantEnsure (self-heal) and the operator (require_existing). Commit.

Task 2 — Durable maintenance marker + loopback quiescence endpoint

Failing tests: backend can durably activate maintenance (marker survives restart); a new loopback-only internal endpoint reports admission leases and PiProcessManager count; drain waits until both are zero; a maintenance marker refuses new admission; health/restart behavior defined. Implement; keep the endpoint loopback-only and unauthenticated-but-internal. Commit.

Task 3 — thothctl vector inspect and guarded rebuild

Failing tests (Go): workspace vector inspect reads the collection contract and emits pristine JSON; workspace vector rebuild requires exact --workspace, --collection, --confirm <same>, --destroy; refuses mismatched confirmation or missing --destroy; acquires the installation lifecycle lock; asks the backend to activate maintenance; verifies session inventory closed; polls quiescence; stops core, rechecks; verifies no preprocessing lock; deletes only the descriptor collection; recreates + verifies; writes durable rebuild state before deletion; restarts core and clears maintenance only after health; on failure after deletion keeps maintenance and prints the explicit recovery command. No prefix/global mutation. Commit.

Task 4 — Docs + P4 walkthrough

Update workspace-preprocessing-cli.md, install manuals, and the P4 section of docs/testing/p2-p6-manual-verification.md (decision PENDING). Commit.

Task 5 — Clean-state P4 automated process goal

P4 acceptance runner (pattern P3): bootstrap fixtures (P1.1 registry + REST DWH + HTTP evidence + embedding stub); pre-provision Qdrant; run thothctl product commands; prove: admission self-heal creates the collection on a fresh volume, incompatible collection refused, vector inspect contract, vector rebuild guarded flow with confirmation/destroy and exact cleanup, collection recreated with the 8 indexes + revision-scoped payload contract, no P5/P6 scope, secret scan, cleanup. Run runner tests, then one clean integration run without retry; report ends P4 automated integration: PASS / P4 manual acceptance: PENDING. Commit.

Task 6 — Final verification + owner handoff

Full backend/frontend/harness/Go gates; one final clean P4 acceptance run; update PROJECT_STATE.md; stop. No P5 work without a new authorization.

Exclusions

No Evidence materialization (P6), no Git FK annotations (P5), no changes to accepted P1.1/P2/P3 evidence, no migration of real PSD content.