1011 lines
56 KiB
Markdown
1011 lines
56 KiB
Markdown
# P4 Qdrant Bootstrap and Guarded Rebuild Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** Implement PRD D4 so session admission safely self-heals a missing workspace Qdrant collection or missing keyword indexes, while `thothctl` can inspect, destructively rebuild, and recover only the descriptor-owned collection under durable maintenance and complete quiescence.
|
||
|
||
**Architecture:** Move Qdrant collection inspection/reconciliation out of `ThtRunner` into one shared TypeScript manager used by session admission and P2's `WorkspacePreprocessingService.execute`. The manager has safe readiness reconciliation plus narrow destructive primitives—never a public monolithic `rebuild()`—so the guarded service can durably record every mutation boundary. Readiness may create only the fixed 1024/cosine collection contract and missing keyword indexes; incompatible vector or index types remain fail-closed. Destructive rebuild is a host-orchestrated transaction: `thothctl` holds one installation lifecycle lock, activates the backend's durable admission barrier, proves the complete session inventory and live process/admission counts are quiescent, stops `core`, and runs the dedicated Pi-free `backend/src/workspace-maintenance.ts::main` entrypoint under P2's writer lock. Success persists `deleting` before DELETE, `deleted` after a confirmed 404, `recreated` after exact collection/index creation, and `verified` only after a separate final inspection; a post-delete failure leaves maintenance active for explicitly confirmed recovery.
|
||
|
||
**Tech Stack:** TypeScript 5, Node.js 22 `fetch`, Fastify 5, Vitest, Qdrant REST API v1.18, Go 1.26, `gofrs/flock`, Docker Compose, Bash, real Git fixtures, and JSON/YAML.
|
||
|
||
---
|
||
|
||
## Source, prerequisites, and hard boundary
|
||
|
||
- Source requirement: `docs/prd/2026-08-09-workspace-preprocessing-prd.md` D4, RF4.2–RF4.4, RNF1–RNF9, and section 8.
|
||
- Reviewed design: `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` section 6 and the common error/verification contracts.
|
||
- P1 is already accepted. Execute and checkpoint P2, then P3, before this plan because this implementation deliberately **extends**, rather than duplicates, P2's exact maintenance surface:
|
||
- `backend/src/workspaces/{runtime-config-lease.ts,preprocessing-state.ts,preprocessing-service.ts}` and `backend/src/workspace-maintenance.ts`;
|
||
- `tools/thothctl/internal/workspaceops/operations.go`;
|
||
- profile-only `workspace-maintenance` in `compose.yaml`;
|
||
- `/data/sessions/<workspace-id>/preprocessing/writer.lock` and `runUnderWorkspaceWriterLock(...)`.
|
||
- This plan is rebased on P2/P3's frozen handoff: `backend/src/workspace-maintenance.ts::main`, `WorkspacePreprocessingService.execute`, `PreprocessingStateStore`, `WorkspaceRuntimeConfigLeaseFactory`, `backend/src/workspaces/revision-layout.ts::{workspaceRuntimePaths,readRevisionLayoutState}`, `runUnderWorkspaceWriterLock` / `probeWorkspaceWriterLock`, and `tools/thothctl/internal/workspaceops::{ParseWorkspaceCommand,Run}`. Run the recorded P2/P3 focused gates before Task 1. Do not create a second operator, renderer, state root, lock, or `internal/workspace` package.
|
||
- P4 does not implement P5 Git annotations, P6 Evidence materialization, P7 migration, P8 aggregate smoke, P9 retention policy, P10 SSH runtime support, a GUI maintenance endpoint, or automatic semantic reindexing after rebuild.
|
||
- Qdrant remains derived data. Rebuild intentionally discards the selected collection's schema/Evidence/Memory projection. Canonical Git descriptors, phase artifacts, corpus state, and the memory registry remain untouched; operators must rerun the already-available indexing commands afterward.
|
||
- No prefix matching, collection enumeration followed by bulk deletion, Qdrant-wide cleanup, forced Pi termination, automatic session closure, or automatic retry is permitted.
|
||
|
||
## Exact public command and result contract
|
||
|
||
P2's read-only command gains a `semantic_index` and `collection_recovery` section:
|
||
|
||
```text
|
||
thothctl --installation /abs/thothii-installation.yaml \
|
||
workspace inspect --workspace research --json
|
||
```
|
||
|
||
P4 adds only these destructive commands:
|
||
|
||
```text
|
||
thothctl --installation /abs/thothii-installation.yaml \
|
||
workspace collection rebuild \
|
||
--workspace research \
|
||
--confirm-workspace research \
|
||
--confirm-collection research-semantic \
|
||
--destructive --json
|
||
|
||
thothctl --installation /abs/thothii-installation.yaml \
|
||
workspace collection recover \
|
||
--workspace research \
|
||
--confirm-workspace research \
|
||
--confirm-collection research-semantic \
|
||
--destructive --json
|
||
```
|
||
|
||
Rules:
|
||
|
||
1. `--workspace`, `--confirm-workspace`, `--confirm-collection`, and the literal `--destructive` are all required once and only once for rebuild/recover. Values are compared byte-for-byte after the normal identifier syntax validation; no case folding, trimming, defaults, interactive prompts, or `--yes` alias.
|
||
2. The collection is always re-derived from the active, operational schema-v3 descriptor. A caller-supplied collection is never used as a mutation target until it exactly equals that value.
|
||
3. Confirmation mismatch exits 2 before maintenance activation, lifecycle state creation, collection mutation, or stopping `core`.
|
||
4. `inspect` is read-only and may run concurrently. It reports `missing`, `compatible`, `repairable` (only keyword indexes are absent), or `incompatible`, plus safe expected/observed dimensions, distance, required index names/types, active revision, maintenance state, and recovery phase. It never self-heals.
|
||
5. Machine output is one pristine JSON object. Stable P4 codes are `semantic_index_incompatible`, `preprocessing_conflict`, `collection_confirmation_mismatch`, `collection_recovery_required`, `collection_recovery_not_required`, `session_inventory_active`, `maintenance_not_quiescent`, and `workspace_not_activatable`. No response includes a Qdrant response body, arbitrary exception, raw child stderr, endpoint, credential, signed URL, or secret-file content.
|
||
|
||
The fixed collection contract remains:
|
||
|
||
```ts
|
||
export const REQUIRED_QDRANT_KEYWORD_INDEXES = [
|
||
"content_hash",
|
||
"document_id",
|
||
"kind",
|
||
"record_key",
|
||
"record_kind",
|
||
"vector_generation",
|
||
"workspace_id",
|
||
"workspace_revision",
|
||
] as const;
|
||
```
|
||
|
||
Vectors are exactly size `1024`, distance `Cosine`; every listed payload index is exactly `keyword`.
|
||
|
||
## Durable transaction and fail-safe matrix
|
||
|
||
Store the collection transaction at:
|
||
|
||
```text
|
||
/data/sessions/<workspace-id>/preprocessing/qdrant-rebuild-state.json
|
||
```
|
||
|
||
Use strict schema version 1:
|
||
|
||
```ts
|
||
interface CollectionRebuildStateV1 {
|
||
version: 1;
|
||
transaction_id: string;
|
||
operation: "collection-rebuild";
|
||
workspace_id: string;
|
||
workspace_revision: string; // exact 40-hex active commit
|
||
collection: string; // descriptor-derived exact name
|
||
phase: "prepared" | "deleting" | "deleted" | "recreated" | "verified" | "failed_pre_delete";
|
||
mutation_started: boolean;
|
||
created_at: string;
|
||
updated_at: string;
|
||
error_code?: "semantic_index_incompatible" | "workspace_not_activatable";
|
||
}
|
||
```
|
||
|
||
Write `prepared` durably before DELETE and write `deleting` with `mutation_started: true` durably **before** issuing DELETE. Therefore a crash with `mutation_started: true` is always treated as potentially destructive even when Qdrant still contains the collection. Atomic replace must fsync the file and owning directory, use regular no-follow 0600 files under the P2-owned preprocessing root, reject unknown keys/versions/identity drift, and never interpolate an exception into `error_code`.
|
||
|
||
- Failure before `mutation_started` may mark `failed_pre_delete`, restart `core`, verify health, and clear maintenance.
|
||
- Failure or ambiguous subprocess output after `mutation_started` leaves `core` stopped and the durable maintenance marker active. The CLI prints the exact recover command using only safe workspace/collection identifiers.
|
||
- Recovery accepts only a nonterminal state matching the current descriptor workspace, revision, and collection. If the collection is missing it recreates it; if compatible it verifies it (covering a crash after recreate); if incompatible it may delete/recreate the exact same collection only because recovery repeats all destructive confirmations and the transaction proves mutation already began. It never targets a different or prefix-matched collection.
|
||
- `verified` is retained for inspection/audit. A later rebuild may atomically supersede it only after proving it is terminal.
|
||
|
||
## Automated P4 process goal
|
||
|
||
Start the execution goal before Task 1 and complete it only after this single clean-state command succeeds:
|
||
|
||
```bash
|
||
./scripts/p4-acceptance.sh integration --keep
|
||
```
|
||
|
||
The command must refuse a dirty tracked source tree, bind the run to the exact Git commit/tree, use one unique Compose project and ownership manifest below `.artifacts/p4-integration/<run-id>/`, execute once with no automatic retry, and retain `report.json`, `report.md`, bounded command events, safe Qdrant observations, recovery state copies, artifact hashes, and cleanup proof. External LLM/DWH/Ollama behavior is outside D4 and uses fixture-only values; Qdrant, the compiled core image, backend routes, `thothctl`, Git registry snapshot, dedicated maintenance service, persistent volumes, and Compose lifecycle are real.
|
||
|
||
The integration must prove: missing collection admission self-heal; missing-index self-heal; concurrent compatible creation/index reconciliation; incompatible dimension/distance/index refusal without PUT/DELETE; exact confirmation refusal before maintenance; complete open-session refusal; live admission/Pi count quiescence; held P2 writer-lock refusal before deletion; exact target-only deletion with a neighbor collection unchanged; verified rebuild/core restart/maintenance clear; a deterministic interruption after deletion with core stopped and marker retained; explicit recovery; secret scan; no retry; and exact owned-resource cleanup. `--keep` retains filesystem evidence, never live containers/networks/volumes or secrets.
|
||
|
||
No unavoidable human action exists inside this automated goal. Manual acceptance happens afterward in a new independent environment.
|
||
|
||
---
|
||
|
||
### Task 0: Establish the execution baseline and persistent goal
|
||
|
||
**Files:**
|
||
- Read: `docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md`
|
||
- Read: `docs/superpowers/plans/2026-08-10-p3-effective-config-fingerprint.md`
|
||
- Read: `backend/src/workspaces/{runtime-config-lease.ts,revision-layout.ts,preprocessing-state.ts,preprocessing-service.ts}` and `backend/src/workspace-maintenance.ts`
|
||
- Read: `tools/thothctl/internal/workspaceops/operations.go`
|
||
- Read: `compose.yaml`, `deploy/compose.local.yaml`, `deploy/compose.server.yaml`
|
||
|
||
- [ ] **Step 1: Confirm the worktree and prerequisites**
|
||
|
||
```bash
|
||
git status --short
|
||
git log -1 --format='%H %T'
|
||
test -f backend/src/workspaces/preprocessing-state.ts
|
||
test -f tools/thothctl/internal/workspaceops/operations.go
|
||
```
|
||
|
||
Expected: clean output from `git status`; one commit/tree line; both files exist. Confirm P2 and P3 checkpoint reports say automated integration PASS. If not, stop—do not fold their scope into P4.
|
||
|
||
- [ ] **Step 2: Start the process goal**
|
||
|
||
If the execution environment supports persistent goals, create: “P4 clean-state Qdrant bootstrap/rebuild/recovery process passes once without retry, retains a secret-clean report, and cleans only owned resources.” Keep it open through Task 11.
|
||
|
||
- [ ] **Step 3: Run prerequisite focused gates**
|
||
|
||
Run the exact P2/P3 focused verification commands recorded in their checkpoint reports.
|
||
|
||
Expected: PASS. A failure is prerequisite drift; fix it in its owning plan before P4.
|
||
|
||
No commit.
|
||
|
||
---
|
||
|
||
### Task 1: Extract the shared Qdrant collection manager
|
||
|
||
**Files:**
|
||
- Create: `backend/src/semantic/qdrant-collection-manager.ts`
|
||
- Create: `backend/test/qdrant-collection-manager.test.ts`
|
||
- Modify: `backend/src/tht/tht-runner.ts` (`QdrantEnsureResult`, `REQUIRED_QDRANT_PAYLOAD_INDEXES`, `qdrantEnsure`)
|
||
- Modify: `backend/test/tht-qdrant-readiness.test.ts`
|
||
|
||
**Required interface:**
|
||
|
||
```ts
|
||
export type CollectionState = "missing" | "compatible" | "repairable" | "incompatible";
|
||
export interface QdrantCollectionSpec {
|
||
collection: string;
|
||
dimensions: 1024;
|
||
distance: "cosine";
|
||
keywordIndexes: readonly string[];
|
||
}
|
||
export interface CollectionInspection {
|
||
state: CollectionState;
|
||
expected: { dimensions: 1024; distance: "cosine"; keyword_indexes: readonly string[] };
|
||
observed?: { dimensions?: number; distance?: string; keyword_indexes: Record<string, string> };
|
||
missing_keyword_indexes: string[];
|
||
incompatible_fields: string[];
|
||
}
|
||
export class QdrantCollectionManager {
|
||
constructor(options: { baseUrl: string; request?: typeof fetch });
|
||
inspect(spec: QdrantCollectionSpec, timeoutMs: number): Promise<CollectionInspection>;
|
||
ensure(spec: QdrantCollectionSpec, timeoutMs: number): Promise<{ ok: boolean; code?: SemanticReadinessCode }>;
|
||
deleteExactAndConfirmMissing(spec: QdrantCollectionSpec, timeoutMs: number): Promise<void>;
|
||
ensureCollection(spec: QdrantCollectionSpec, timeoutMs: number): Promise<void>;
|
||
ensureIndexes(spec: QdrantCollectionSpec, timeoutMs: number): Promise<void>;
|
||
}
|
||
export function qdrantCollectionSpec(workspace: WorkspaceDescriptor): QdrantCollectionSpec;
|
||
```
|
||
|
||
There is deliberately no exported `rebuild()` and no manager method that can cross more than one durable destructive phase. `deleteExactAndConfirmMissing` issues one exact DELETE and succeeds only after a separate GET confirms 404. `ensureCollection` creates or converges only the exact 1024/Cosine collection and finishes with a read proving the vector contract (keyword indexes may still be absent). `ensureIndexes` creates only missing required keyword indexes in canonical order and finishes with a read proving the exact complete contract. The guarded service in Task 3 is the sole production composer of these primitives; readiness calls only `ensure`.
|
||
|
||
- [ ] **Step 1: Write RED inspection tests**
|
||
|
||
Use a scripted `vi.fn` fetch boundary. Cover exact URL encoding, GET 404 → `missing`, exact 1024/Cosine/eight keyword indexes → `compatible`, missing index → `repairable`, and wrong size/distance/index type → `incompatible`. Reject malformed success JSON as `workspace_not_activatable`; never propagate a body canary.
|
||
|
||
- [ ] **Step 2: Verify RED**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run test/qdrant-collection-manager.test.ts
|
||
```
|
||
|
||
Expected: FAIL because the module is missing.
|
||
|
||
- [ ] **Step 3: Implement read-only inspection**
|
||
|
||
Use only `new URL('/collections/' + encodeURIComponent(name), baseUrl)`, an operation-wide `AbortController`, allowlisted parsed fields, and lowercase comparison for Qdrant's distance/type response. Distinguish missing from incompatible; an unavailable/unparseable service is not semantic incompatibility.
|
||
|
||
- [ ] **Step 4: Write RED self-heal/concurrency tests**
|
||
|
||
Cover these exact request sequences:
|
||
|
||
1. GET 404 → PUT collection with `{"vectors":{"size":1024,"distance":"Cosine"}}` → final GET compatible.
|
||
2. GET 404 → PUT 409/already exists → final GET compatible (concurrent compatible creator).
|
||
3. GET repairable → PUT `/collections/<name>/index?wait=true` with `{"field_name":"kind","field_schema":"keyword"}` → final GET compatible.
|
||
4. Index PUT conflict → final GET compatible (concurrent compatible index creator).
|
||
5. A barrier-controlled unit race launches two independent `manager.ensure` calls (not a shared promise): both initial GETs observe 404/repairable, one PUT succeeds, the other receives the allowed conflict, both perform their own final GET, and both converge compatible. Repeat for collection creation and one missing index; assert no retry loop.
|
||
6. Concurrent creator/index ends incompatible → `semantic_index_incompatible`.
|
||
7. Existing incompatible size/distance/index performs no PUT or DELETE.
|
||
8. A missing-index run never recreates the collection and creates only missing indexes in canonical order.
|
||
9. Timeout/unreachable response returns `workspace_not_activatable` with no endpoint/body leak.
|
||
10. `deleteExactAndConfirmMissing` issues DELETE for the one URL-encoded exact collection and then independently reads 404; DELETE failure or a non-404 post-delete state fails.
|
||
11. `ensureCollection` and `ensureIndexes` are separately observable: collection creation ends with the exact vector contract present, index creation ends with the exact complete contract, and neither ever deletes. A later independent `inspect` supplies final verification. No other collection name, list endpoint, prefix, or global mutation is ever requested.
|
||
12. No exported `rebuild`, callback that hides multiple phases, or test-only destructive shortcut exists.
|
||
|
||
- [ ] **Step 5: Verify RED, then implement minimal reconciliation**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run test/qdrant-collection-manager.test.ts
|
||
```
|
||
|
||
Expected before implementation: FAIL on PUT sequences. Implement create/index calls, tolerate only conflict/already-exists as a reason to re-read, and always decide success from one final GET. Do not accept the mutating response as proof.
|
||
|
||
- [ ] **Step 6: Delegate `ThtRunner.qdrantEnsure`**
|
||
|
||
Keep descriptor operational validation in `qdrantEnsure`, derive `qdrantCollectionSpec`, and call the shared manager. Remove the duplicate constant/parsing logic. Preserve `ThtConfig.qdrantRequest` as the injectable fetch boundary.
|
||
|
||
- [ ] **Step 7: Run focused tests and typecheck**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run test/qdrant-collection-manager.test.ts test/tht-qdrant-readiness.test.ts test/readiness-manager.test.ts
|
||
npx tsc --noEmit -p .
|
||
```
|
||
|
||
Expected: PASS. Update the old missing-collection assertion from incompatibility to success after the exact create/final-read sequence; existing incompatible cases remain fail-closed.
|
||
|
||
- [ ] **Step 8: Commit**
|
||
|
||
```bash
|
||
git add backend/src/semantic/qdrant-collection-manager.ts \
|
||
backend/src/tht/tht-runner.ts \
|
||
backend/test/qdrant-collection-manager.test.ts \
|
||
backend/test/tht-qdrant-readiness.test.ts \
|
||
backend/test/readiness-manager.test.ts
|
||
git commit -m "feat: self-heal workspace qdrant collections"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: Prove session admission uses self-heal before Ollama and persistence
|
||
|
||
**Files:**
|
||
- Modify: `backend/test/routes-sessions.test.ts` (readiness cases around current Qdrant tests)
|
||
- Modify if dependency injection requires it: `backend/src/app.ts` (`ThtRunner` construction only)
|
||
|
||
- [ ] **Step 1: Write RED route tests**
|
||
|
||
At the real `buildApp` boundary, inject a `ThtRunner`/fetch script and assert:
|
||
|
||
- missing collection creates the exact contract, then Ollama runs, then session persistence is allowed;
|
||
- missing index creates only that index before Ollama;
|
||
- incompatible collection returns 503 with `semantic_index_incompatible`, does not call Ollama, does not call `sessionNew`, acquire a revision lease permanently, or spawn Pi;
|
||
- two simultaneous **local-mode, same-principal, same-workspace** admissions hit `ReadinessManager`'s in-flight key and share one promise: exactly one Qdrant create/reconciliation occurs, both callers receive the same compatible result, and no false incompatibility is emitted;
|
||
- one separate `ReadinessManager` unit test proves the same-principal dedup key and cleanup after resolution/rejection;
|
||
- one **upstream-auth, two-distinct-principal** route test uses distinct readiness keys, coordinates both initial GET 404 reads, lets one PUT create and the other PUT receive the compatible-creator conflict, and proves both converge after their final reads. Do not describe this as a local same-principal race;
|
||
- raw Qdrant body/endpoint canaries do not appear in HTTP JSON or logs captured by the test.
|
||
|
||
Keep the compatible-creator and compatible-index conflict sequences in `qdrant-collection-manager.test.ts` as the direct manager race proof. Route tests prove the real dedup/cross-principal semantics rather than attempting to bypass `ReadinessManager`.
|
||
|
||
- [ ] **Step 2: Run and verify RED**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run test/routes-sessions.test.ts -t "Qdrant|semantic index|concurrent collection"
|
||
```
|
||
|
||
Expected: the new call-order, local deduplication, and distinct-principal convergence assertions FAIL until route fixtures use the real manager behavior and upstream identities produce distinct readiness keys.
|
||
|
||
- [ ] **Step 3: Make the minimal wiring change**
|
||
|
||
Do not add a new route. Session admission continues through `ReadinessManager.ensure(..., descriptor)` and `ThtRunner.qdrantEnsure`; change only construction/injection necessary to share the manager.
|
||
|
||
- [ ] **Step 4: Verify**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run \
|
||
test/routes-sessions.test.ts \
|
||
test/readiness-manager.test.ts \
|
||
test/tht-qdrant-readiness.test.ts
|
||
npx tsc --noEmit -p .
|
||
```
|
||
|
||
Expected: PASS, including Qdrant-before-Ollama ordering, exactly one create for local same-principal deduplication, two independently convergent manager calls for distinct upstream principals, and no persistence on incompatibility.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add backend/src/app.ts backend/test/routes-sessions.test.ts backend/test/readiness-manager.test.ts
|
||
git commit -m "test: prove qdrant admission self-heal"
|
||
```
|
||
|
||
Omit unchanged paths from `git add`.
|
||
|
||
---
|
||
|
||
### Task 3: Add safe inspection and durable rebuild/recovery to the P2 operator
|
||
|
||
**Files:**
|
||
- Create: `backend/src/workspaces/qdrant-rebuild-state.ts`
|
||
- Create: `backend/test/qdrant-rebuild-state.test.ts`
|
||
- Modify: `backend/src/workspaces/preprocessing-service.ts` (`MaintenanceCommand`/result union)
|
||
- Modify: `backend/src/workspaces/preprocessing-service.ts` (`WorkspacePreprocessingService.execute`)
|
||
- Modify: `backend/src/workspace-maintenance.ts` (`main` parser/encoder)
|
||
- Modify: `backend/src/workspaces/preprocessing-state.ts`
|
||
- Modify: `backend/test/workspace-preprocessing-service.test.ts`
|
||
- Modify: `backend/test/workspace-maintenance.test.ts`
|
||
- Modify: `backend/test/workspace-preprocessing-state.test.ts`
|
||
|
||
**Operator additions:**
|
||
|
||
```ts
|
||
type MaintenanceCommand =
|
||
| /* P2/P3 requests */
|
||
| { operation: "collection-rebuild"; workspace_id: string; confirm_workspace: string;
|
||
confirm_collection: string; destructive: true }
|
||
| { operation: "collection-recover"; workspace_id: string; confirm_workspace: string;
|
||
confirm_collection: string; destructive: true };
|
||
```
|
||
|
||
`inspect` gains safe `semantic_index` and `collection_recovery`; it never acquires a writer lock or mutates Qdrant. Rebuild/recover must call `runUnderWorkspaceWriterLock` for the complete state-read → mutation → verification transaction. A conflict maps to `preprocessing_conflict` before state or Qdrant mutation.
|
||
|
||
- [ ] **Step 1: Write RED state-store tests**
|
||
|
||
Test restrictive creation, atomic durable transitions, same-inode/root safety, unknown schema/keys, workspace/revision/collection mismatch, symlink/path swap, injected file-fsync and directory-fsync ambiguity, and terminal supersession. Table-test the only legal forward edges `prepared → deleting → deleted → recreated → verified` plus `prepared → failed_pre_delete`; reject skips, rewinds, cross-transaction updates, and `failed_pre_delete` after mutation. Add crash-visible assertions at every boundary: `deleting/mutation_started` is durable before mocked DELETE, `deleted` only after the manager confirms 404, `recreated` only after both exact collection and required indexes are confirmed, and `verified` only after one additional independent `inspect` returns compatible.
|
||
|
||
- [ ] **Step 2: Verify RED**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run test/qdrant-rebuild-state.test.ts
|
||
```
|
||
|
||
Expected: FAIL because `CollectionRebuildStateStore` does not exist.
|
||
|
||
- [ ] **Step 3: Implement the strict store**
|
||
|
||
Reuse P2's trusted preprocessing-root and atomic state primitives rather than another root policy. Expose only `read`, `begin`, and `transition(expectedTransaction, next)`; enforce legal monotonic phase transitions. Report durability uncertainty as recovery-required whenever the intended bytes may have reached the canonical path.
|
||
|
||
- [ ] **Step 4: Write RED operator tests**
|
||
|
||
Cover:
|
||
|
||
- `inspect` returns active workspace/revision, exact descriptor collection, manager inspection, and safe recovery state without mutation/lock;
|
||
- confirmation mismatch and missing `destructive: true` return `collection_confirmation_mismatch` before marker/lock/Qdrant;
|
||
- rebuild refuses unless `/data/settings/maintenance.json` is a trusted regular file whose strict JSON says active;
|
||
- held `/preprocessing/writer.lock` returns `preprocessing_conflict` before DELETE;
|
||
- rebuild writes/fsyncs `prepared`, then writes/fsyncs `deleting` with `mutation_started:true` before `deleteExactAndConfirmMissing`; after that primitive's confirmed 404 it writes/fsyncs `deleted`; after `ensureCollection` and `ensureIndexes` have separately confirmed the exact contract it writes/fsyncs `recreated`; only a subsequent standalone `inspect` returning compatible permits `verified`;
|
||
- injected crashes after each durable phase leave exactly that phase visible, never infer a later phase from a mutating response, and resume through the same narrow manager primitives;
|
||
- neighbor collections are never requested;
|
||
- failure before `deleting` becomes `failed_pre_delete` with `mutation_started: false`;
|
||
- failure/timeout at or after `deleting` retains the last durable nonterminal state with `mutation_started: true` and returns a safe recovery-required envelope;
|
||
- recover handles every nonterminal phase: `prepared` may start deletion only after repeated confirmations; `deleting` first inspects and conservatively establishes missing/compatible/incompatible exact state; `deleted` recreates; `recreated` independently verifies; missing, already-compatible-after-crash, and explicitly confirmed incompatible exact targets all converge without skipping a durable boundary;
|
||
- recover rejects absent/verified state, changed active revision, changed descriptor collection, and unsafe/unknown state;
|
||
- error output excludes Qdrant body, marker contents beyond allowlisted booleans, paths, endpoints, and canaries.
|
||
|
||
- [ ] **Step 5: Run and verify RED**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run \
|
||
test/workspace-preprocessing-service.test.ts \
|
||
test/workspace-preprocessing-state.test.ts \
|
||
test/workspace-maintenance.test.ts
|
||
```
|
||
|
||
Expected: new operation cases FAIL because dispatch/state enforcement is absent.
|
||
|
||
- [ ] **Step 6: Implement guarded dispatch**
|
||
|
||
Derive the descriptor and revision through `WorkspacePreprocessingService.execute` and its existing P2 resolver, validate confirmations, verify the marker through `lstat/open(O_NOFOLLOW)/fstat` and strict JSON, then enter `runUnderWorkspaceWriterLock`. Compose only `inspect`, `deleteExactAndConfirmMissing`, `ensureCollection`, and `ensureIndexes`, persisting/fsyncing the state between calls exactly as specified above. Neither `backend/src/workspace-maintenance.ts::main` nor any other caller receives a monolithic destructive API. The CLI accepts fixed argv generated by `workspaceops.Run` and preserves one pristine JSON envelope; direct manual invocation without marker/confirmations remains harmless.
|
||
|
||
- [ ] **Step 7: Verify focused backend tests**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run \
|
||
test/qdrant-collection-manager.test.ts \
|
||
test/qdrant-rebuild-state.test.ts \
|
||
test/workspace-preprocessing-state.test.ts \
|
||
test/workspace-preprocessing-service.test.ts \
|
||
test/workspace-maintenance.test.ts
|
||
npx tsc --noEmit -p .
|
||
npm run build
|
||
```
|
||
|
||
Expected: PASS; build emits `/app/backend/dist/workspace-maintenance.js` and the new shared manager/state module.
|
||
|
||
- [ ] **Step 8: Commit**
|
||
|
||
```bash
|
||
git add backend/src/workspaces/qdrant-rebuild-state.ts \
|
||
backend/src/workspaces/preprocessing-state.ts \
|
||
backend/src/workspaces/preprocessing-service.ts \
|
||
backend/src/workspace-maintenance.ts \
|
||
backend/test/qdrant-rebuild-state.test.ts \
|
||
backend/test/workspace-preprocessing-state.test.ts \
|
||
backend/test/workspace-preprocessing-service.test.ts \
|
||
backend/test/workspace-maintenance.test.ts
|
||
git commit -m "feat: add guarded qdrant maintenance operations"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: Expose loopback-only maintenance quiescence
|
||
|
||
**Files:**
|
||
- Modify: `backend/src/app.ts` (`maintenance status handlers`, `isMaintenanceControl`)
|
||
- Modify: `backend/src/runtime/maintenance-gate.ts` only if a named status type is needed
|
||
- Modify: `backend/test/routes-sessions.test.ts` (current maintenance endpoint tests)
|
||
- Modify: `backend/test/maintenance-gate.test.ts` only for status typing/durability regressions
|
||
- Modify: `backend/test/pi-process-manager.test.ts`
|
||
|
||
**Response:**
|
||
|
||
```json
|
||
{
|
||
"active": true,
|
||
"admissions": 0,
|
||
"piProcesses": 0,
|
||
"quiescent": true
|
||
}
|
||
```
|
||
|
||
Include `recoveryRequired: true` only when already produced by `MaintenanceBarrier`. `quiescent` is true only when the durable marker is active, `admissions === 0`, `mgr.count() === 0`, and no durability recovery is pending.
|
||
|
||
- [ ] **Step 1: Write RED endpoint tests**
|
||
|
||
Test `POST /internal/maintenance/activate`, `GET /internal/maintenance/status`, and new `GET /internal/maintenance/quiescence` with injected manager counts 0/1. Prove activation waits for an in-flight admission lease, later admissions receive 503, Pi count is observational (never killed), spoofed non-loopback callers receive 403 in `none` and `upstream` auth, and durability ambiguity makes `quiescent: false`.
|
||
|
||
- [ ] **Step 2: Verify RED**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run test/routes-sessions.test.ts -t "maintenance|quiescence"
|
||
```
|
||
|
||
Expected: FAIL because responses omit `piProcesses`/`quiescent` and the route is not allowlisted.
|
||
|
||
- [ ] **Step 3: Implement one response composer**
|
||
|
||
In `buildApp`, define `maintenanceStatus()` from `maintenanceBarrier.status()` plus `mgr.count()`. Use it for activate/deactivate/status/quiescence responses so the fields cannot drift. Do not expose runtime IDs, principals, session questions, or child process details.
|
||
|
||
- [ ] **Step 4: Verify**
|
||
|
||
```bash
|
||
cd backend
|
||
npx vitest run \
|
||
test/maintenance-gate.test.ts \
|
||
test/routes-sessions.test.ts \
|
||
test/pi-process-manager.test.ts
|
||
npx tsc --noEmit -p .
|
||
```
|
||
|
||
Expected: PASS. Existing admission behavior remains unchanged except for additive internal response fields.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add backend/src/app.ts backend/src/runtime/maintenance-gate.ts \
|
||
backend/test/routes-sessions.test.ts backend/test/maintenance-gate.test.ts \
|
||
backend/test/pi-process-manager.test.ts
|
||
git commit -m "feat: report complete maintenance quiescence"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: Share one installation lifecycle lock with Pi operations
|
||
|
||
**Files:**
|
||
- Create: `tools/thothctl/internal/lifecycle/lock.go`
|
||
- Create: `tools/thothctl/internal/lifecycle/lock_test.go`
|
||
- Modify: `tools/thothctl/internal/config/installation.go` (`LifecycleLockPath`, `CollectionRebuildStatePath` only if host metadata is retained)
|
||
- Modify: `tools/thothctl/internal/config/installation_test.go`
|
||
- Modify: `tools/thothctl/internal/pi/state.go` (`acquireLock`, `updateLock`, `ErrLockHeld` removal/delegation)
|
||
- Modify: `tools/thothctl/internal/pi/update.go` (`Update`, `Rollback`, `RecoverMaintenance` lock acquisition)
|
||
- Modify: `tools/thothctl/internal/pi/update_test.go`
|
||
- Modify: `tools/thothctl/internal/pi/state_test.go`
|
||
|
||
**Path:** `.thothctl/<project-name>/lifecycle.lock`, with owner metadata `.thothctl/<project-name>/lifecycle.lock.owner.json`.
|
||
|
||
- [ ] **Step 1: Write RED lifecycle tests**
|
||
|
||
Test stable 0600 lock inode, 0600 atomic owner metadata, nonblocking cross-process contention, metadata removal only by the owner on release, crash-safe kernel release, symlink/unsafe control-directory rejection, and distinct installation descriptors not contending. Add a test that a Pi update lock blocks a simulated collection rebuild and vice versa.
|
||
|
||
- [ ] **Step 2: Verify RED**
|
||
|
||
```bash
|
||
cd tools/thothctl
|
||
go test ./internal/lifecycle ./internal/config
|
||
```
|
||
|
||
Expected: FAIL because the lifecycle package/path does not exist.
|
||
|
||
- [ ] **Step 3: Extract the current lock without changing Pi transaction semantics**
|
||
|
||
Move the `gofrs/flock` implementation from `internal/pi/state.go` into `lifecycle.Acquire(path, operation)`. Owner JSON contains PID, host, start time, transaction, and an allowlisted operation (`pi-update`, `pi-rollback`, `pi-maintenance-recover`, `qdrant-rebuild`, `qdrant-recover`), never argv/environment. Add `LifecycleLockPath string` to `pi.Request`; change `Rollback` and `RecoverMaintenance` to accept `(statePath, lifecycleLockPath, confirm)`; and make `main.go` pass `installation.LifecycleLockPath()` to all three paths. Reject an empty/noncanonical lock path. The update state remains `UpdateStatePath()`; only mutual exclusion moves from `update-state.json.lock` to the installation lock.
|
||
|
||
- [ ] **Step 4: Run Pi regressions**
|
||
|
||
```bash
|
||
cd tools/thothctl
|
||
go test ./internal/lifecycle ./internal/config ./internal/pi
|
||
```
|
||
|
||
Expected: PASS; interrupted update/rollback behavior and recovery metadata remain identical, but P4 and Pi now contend on one installation lock.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add tools/thothctl/internal/lifecycle \
|
||
tools/thothctl/internal/config/installation.go \
|
||
tools/thothctl/internal/config/installation_test.go \
|
||
tools/thothctl/internal/pi/state.go tools/thothctl/internal/pi/state_test.go \
|
||
tools/thothctl/internal/pi/update.go tools/thothctl/internal/pi/update_test.go
|
||
git commit -m "refactor: share installation lifecycle lock"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: Parse exact collection commands and confirmations in `thothctl`
|
||
|
||
**Files:**
|
||
- Modify: `tools/thothctl/internal/workspaceops/operations.go`
|
||
- Modify: `tools/thothctl/internal/workspaceops/operations_test.go`
|
||
- Create: `tools/thothctl/internal/workspaceops/collection.go`
|
||
- Create: `tools/thothctl/internal/workspaceops/collection_test.go`
|
||
- Modify: `tools/thothctl/cmd/thothctl/main.go`
|
||
- Modify: `tools/thothctl/cmd/thothctl/main_test.go`
|
||
|
||
- [ ] **Step 1: Write RED parse tests**
|
||
|
||
Table-test the exact commands above. Reject missing/duplicate flags, swapped confirmations, whitespace/case differences, unknown flags/subcommands, positional targets, `--yes`, absent `--destructive`, unsafe workspace/collection identifiers, and rebuild/recover input or output file flags inherited from P2. Confirm usage errors exit 2 and never call Docker.
|
||
|
||
- [ ] **Step 2: Verify RED**
|
||
|
||
```bash
|
||
cd tools/thothctl
|
||
go test ./internal/workspaceops ./cmd/thothctl -run 'Collection|collection|Confirmation'
|
||
```
|
||
|
||
Expected: FAIL because P2 parsing knows no `workspace collection` family.
|
||
|
||
- [ ] **Step 3: Extend `workspaceops.ParseWorkspaceCommand` and main dispatch**
|
||
|
||
Use typed operations `CollectionRebuild` and `CollectionRecover`; do not add another top-level parser. Perform syntactic equality checks in Go before Docker, then pass all values to TypeScript for authoritative descriptor equality checks.
|
||
|
||
- [ ] **Step 4: Extend inspect parsing/output**
|
||
|
||
P2 `workspace inspect` keeps the same command syntax. Validate the returned semantic/recovery JSON structure and print pristine JSON under `--json`; human output lists exact workspace, revision, collection, state, missing/incompatible fields, maintenance, and recovery phase without endpoint details.
|
||
|
||
- [ ] **Step 5: Verify**
|
||
|
||
```bash
|
||
cd tools/thothctl
|
||
go test ./internal/workspaceops ./cmd/thothctl
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add tools/thothctl/internal/workspaceops/operations.go \
|
||
tools/thothctl/internal/workspaceops/operations_test.go \
|
||
tools/thothctl/internal/workspaceops/collection.go \
|
||
tools/thothctl/internal/workspaceops/collection_test.go \
|
||
tools/thothctl/cmd/thothctl/main.go tools/thothctl/cmd/thothctl/main_test.go
|
||
git commit -m "feat: add qdrant collection operator commands"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 7: Orchestrate maintenance, inventory, stop, rebuild, restart, and recovery
|
||
|
||
**Files:**
|
||
- Modify: `tools/thothctl/internal/workspaceops/collection.go`
|
||
- Modify: `tools/thothctl/internal/workspaceops/collection_test.go`
|
||
- Modify: `tools/thothctl/internal/workspaceops/operations.go`
|
||
- Modify: `tools/thothctl/internal/workspaceops/operations_test.go`
|
||
- Modify: `tools/thothctl/cmd/thothctl/main.go`
|
||
- Modify: `tools/thothctl/cmd/thothctl/main_test.go`
|
||
- Modify: `tools/thothctl/internal/pi/update.go` only to share strict maintenance parsing/helpers where appropriate
|
||
- Modify: `tools/thothctl/internal/pi/update_test.go`
|
||
|
||
**Required sequence for rebuild:**
|
||
|
||
```text
|
||
Acquire installation lifecycle lock
|
||
→ workspace inspect and authoritative confirmation recheck
|
||
→ POST core /internal/maintenance/activate
|
||
→ GET complete /sessions?scope=mine(local)|all(server)
|
||
→ reject unless every item is archived OR status closed/finalized
|
||
→ poll /internal/maintenance/quiescence until active + admissions=0 + piProcesses=0
|
||
→ compose stop core
|
||
→ compose ps --status running -q core must be empty
|
||
→ compose --profile workspace-maintenance run --rm --no-deps -T
|
||
workspace-maintenance collection-rebuild <fixed argv>
|
||
→ parse exact safe JSON and require verified state
|
||
→ compose up --detach --no-deps core
|
||
→ poll Compose health and loopback /health
|
||
→ require maintenance still active and quiescent
|
||
→ POST /internal/maintenance/deactivate
|
||
→ require active=false
|
||
→ release lifecycle lock
|
||
```
|
||
|
||
Use a bounded monotonic timeout (document 60 seconds for quiescence and 120 seconds for core health), one-second polling, and **no retry of a failed operation**. Polling observation is not retrying a mutation.
|
||
|
||
- [ ] **Step 1: Write RED orchestration tests with a scripted runner**
|
||
|
||
Assert exact argv/order and failure behavior for:
|
||
|
||
- confirmation mismatch before lock/HTTP;
|
||
- lifecycle contention with Pi;
|
||
- activation durability failure;
|
||
- malformed/incomplete/duplicate session inventory;
|
||
- open/failed unarchived session refusal; archived, closed, and finalized acceptance;
|
||
- `admissions > 0` and `piProcesses > 0` poll, then success;
|
||
- timeout without forced teardown;
|
||
- core stop failure or still-running proof;
|
||
- P2 writer-lock conflict returned by the maintenance service before deletion, followed by safe core restart/health/maintenance clear;
|
||
- successful operator `verified` response, exact core restart, health, and marker clear;
|
||
- malformed/ambiguous operator result treated as mutation-started/recovery-required unless a trusted response proves `mutation_started:false`;
|
||
- post-delete error leaves core stopped, marker active, state retained, and prints exact recover command;
|
||
- neighbor services/collections never appear in stop/delete argv;
|
||
- all child stderr is sanitized and bounded.
|
||
|
||
- [ ] **Step 2: Verify RED**
|
||
|
||
```bash
|
||
cd tools/thothctl
|
||
go test ./internal/workspaceops -run 'Rebuild|Quiescence|Inventory|Recovery'
|
||
```
|
||
|
||
Expected: FAIL because lifecycle orchestration is not implemented.
|
||
|
||
- [ ] **Step 3: Implement strict inventory and quiescence clients**
|
||
|
||
Reuse the existing core-side curl identity headers. Decode exactly one bounded JSON document. A complete local install uses `scope=mine`; server uses admin `scope=all`. Never infer quiescence only from manifests: both `admissions` and `piProcesses` must reach zero after the durable marker is active.
|
||
|
||
- [ ] **Step 4: Implement core stop/start proof and fail-safe cleanup**
|
||
|
||
Use fixed Compose argv and explicit `core`. Do not call `down`, stop Qdrant, stop frontend, use `--remove-orphans`, or prune. Clear maintenance only after a trusted pre-delete failure or complete verified success and healthy core.
|
||
|
||
- [ ] **Step 5: Write RED recovery tests**
|
||
|
||
Test core-already-stopped recovery, core-running-but-maintained recovery, missing marker refusal, no/nonterminal/verified state, descriptor revision drift, recovery service failure, compatible-after-crash verification, missing recreate, explicitly confirmed incompatible recreate, successful restart/health/deactivate, and repeated recover returning `collection_recovery_not_required` without mutation.
|
||
|
||
- [ ] **Step 6: Implement recovery**
|
||
|
||
When core is stopped, the durable marker plus matching nonterminal state is the admission barrier; the maintenance service re-verifies both before mutation. When core is running, repeat activation/inventory/quiescence before stopping it. Never clear the marker merely because recovery cannot read state.
|
||
|
||
- [ ] **Step 7: Run Go suites**
|
||
|
||
```bash
|
||
cd tools/thothctl
|
||
go test ./internal/workspaceops ./internal/lifecycle ./internal/pi ./cmd/thothctl
|
||
go test ./...
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 8: Commit**
|
||
|
||
```bash
|
||
git add tools/thothctl/internal/workspaceops/collection.go \
|
||
tools/thothctl/internal/workspaceops/collection_test.go \
|
||
tools/thothctl/internal/workspaceops/operations.go \
|
||
tools/thothctl/internal/workspaceops/operations_test.go \
|
||
tools/thothctl/internal/pi/update.go tools/thothctl/internal/pi/update_test.go \
|
||
tools/thothctl/cmd/thothctl/main.go tools/thothctl/cmd/thothctl/main_test.go
|
||
git commit -m "feat: orchestrate recoverable qdrant rebuilds"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 8: Harden the dedicated maintenance service and Compose contract
|
||
|
||
**Files:**
|
||
- Modify: `compose.yaml` (`workspace-maintenance` only)
|
||
- Modify: `deploy/compose.server.yaml` (`workspace-maintenance` bind roots)
|
||
- Verify/modify as P2 created them: `deploy/compose.git-https.yaml`, `deploy/compose.git-ssh.yaml`
|
||
- Modify: `docker/core.Dockerfile` (ensure `/usr/bin/flock` from `util-linux` is present)
|
||
- Modify: `scripts/test-internal-semantic-compose.sh`
|
||
- Modify: `scripts/test-unified-compose.sh`
|
||
- Modify: `scripts/test-compose-secret-policy.sh`
|
||
- Modify: `scripts/test-no-deployment-coupling.sh`
|
||
- Modify: `scripts/test-deployment-command-contract.sh`
|
||
|
||
The service gains the shared settings mount needed to read the durable maintenance marker. It retains P2's sessions/registry mounts, internal Qdrant network, same selected core image, and direct Node entrypoint. It must still have no `/home/thoth/.pi`, PI auth/models/settings, writable Pi state, frontend dependency, host Docker socket, published port, or normal core entrypoint/trust initialization.
|
||
|
||
- [ ] **Step 1: Extend RED Compose contract tests**
|
||
|
||
Assert local named volumes and server bind roots make the **same** `/data/settings`, `/data/sessions`, and `/data/workspace-registry` visible to `core` and `workspace-maintenance`; Qdrant is reachable only on the private network; only the exact required Git/connector secret files attach; service is profile-only; `flock` exists in the built core image; and service cannot start Pi or expose HTTP.
|
||
|
||
- [ ] **Step 2: Verify RED**
|
||
|
||
```bash
|
||
./scripts/test-internal-semantic-compose.sh
|
||
./scripts/test-unified-compose.sh
|
||
./scripts/test-compose-secret-policy.sh
|
||
./scripts/test-deployment-command-contract.sh
|
||
```
|
||
|
||
Expected: at least the settings/marker and `flock` assertions FAIL.
|
||
|
||
- [ ] **Step 3: Make minimal Compose/image changes**
|
||
|
||
Add `util-linux` explicitly rather than relying on a transitive base package. Extend the P2 service only; do not introduce a second maintenance service or mount Pi state.
|
||
|
||
- [ ] **Step 4: Run deployment gates**
|
||
|
||
```bash
|
||
./scripts/test-default-compose.sh
|
||
./scripts/test-unified-compose.sh
|
||
./scripts/test-internal-semantic-compose.sh
|
||
./scripts/test-compose-secret-policy.sh
|
||
./scripts/test-no-deployment-coupling.sh
|
||
./scripts/test-deployment-command-contract.sh
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```bash
|
||
git add compose.yaml deploy/compose.server.yaml \
|
||
deploy/compose.git-https.yaml deploy/compose.git-ssh.yaml \
|
||
docker/core.Dockerfile scripts/test-internal-semantic-compose.sh \
|
||
scripts/test-unified-compose.sh scripts/test-compose-secret-policy.sh \
|
||
scripts/test-no-deployment-coupling.sh scripts/test-deployment-command-contract.sh
|
||
git commit -m "build: isolate qdrant maintenance service"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 9: Build the clean-state P4 process goal and retained report
|
||
|
||
**Files:**
|
||
- Create: `scripts/p4-acceptance.sh`
|
||
- Create: `scripts/test-p4-acceptance.sh`
|
||
- Create: `backend/scripts/p4-acceptance.mjs`
|
||
- Create: `backend/scripts/p4-acceptance.test.mjs`
|
||
- Create: `backend/scripts/fixtures/p4-qdrant-proxy.mjs`
|
||
- Modify: `.gitignore` only if the existing `.artifacts/` rule is insufficient
|
||
|
||
**Owned layout:**
|
||
|
||
```text
|
||
.artifacts/p4-integration/<run-id>/
|
||
├── ownership.json
|
||
├── source.json
|
||
├── installation/thothii-installation.yaml
|
||
├── registry/{remote.git,author}/
|
||
├── fixture-secrets/
|
||
├── compose/acceptance.override.yaml
|
||
├── requests/
|
||
├── responses/
|
||
├── observations/
|
||
├── recovery/qdrant-rebuild-state.json
|
||
├── logs/
|
||
├── artifact-manifest.json
|
||
├── cleanup.json
|
||
├── report.json
|
||
└── report.md
|
||
```
|
||
|
||
- [ ] **Step 1: Write RED wrapper/orchestrator tests**
|
||
|
||
Test argument parsing (`integration [--keep]` only), dirty tracked-tree refusal, unique run/project naming, fixture-only credentials, fixed toolchain resolution, no retry loop, bounded output, ownership checks, report schema, artifact hashes, secret-canary scan, cleanup behavior on success/failure/signal, and `--keep` retaining only the owned filesystem root.
|
||
|
||
- [ ] **Step 2: Verify RED**
|
||
|
||
```bash
|
||
./scripts/test-p4-acceptance.sh
|
||
```
|
||
|
||
Expected: FAIL because the wrapper/orchestrator are missing.
|
||
|
||
- [ ] **Step 3: Implement the isolated topology**
|
||
|
||
Use a real local bare Git remote and schema-v3 descriptor, compiled core image, real Qdrant storage, and production `thothctl`. An acceptance-only private proxy named `qdrant` fronts a real uniquely named Qdrant service; it forwards normal REST calls and provides a deterministic, test-owned barrier after DELETE so the one-shot maintenance container can be killed after durable `mutation_started` state and before recreate. It must not alter production code paths or accept secrets. Ollama/DWH/LLM use safe fixtures because they are outside D4.
|
||
|
||
- [ ] **Step 4: Implement positive and negative checks**
|
||
|
||
Record named PASS checks for every assertion in the automated goal. For the session self-heal check, traverse the real backend admission route and intentionally stop at a later fixture readiness boundary only after Qdrant creation has been final-read compatible; assert no session manifest/Pi process survives. For concurrency, prove both layers separately: coordinate two direct real manager calls through the proxy so one compatible create/index conflicts, and send two simultaneous local same-principal route admissions to prove `ReadinessManager` performs exactly one create. Also run the upstream-auth route with two distinct principals if the cross-principal route proof is retained. Do not loop/retry until green and do not claim two local same-principal route calls both reached Qdrant.
|
||
|
||
Preseed one target point and one neighbor collection/point. Successful rebuild must remove/recreate only the target, leave the neighbor byte/count identity unchanged, retain canonical source files, and report that reindexing is required.
|
||
|
||
- [ ] **Step 5: Implement interruption and recovery check**
|
||
|
||
Block recreate after the exact target DELETE, kill only the uniquely labelled one-shot container, and assert: command failure; core stopped; maintenance marker active; state `mutation_started:true`; admissions refused if core is deliberately restarted under the marker. Release the proxy barrier and run the exact confirmed `collection recover` once; assert verified collection, healthy core, inactive marker, terminal state, and neighbor preservation.
|
||
|
||
- [ ] **Step 6: Implement secret scan, artifact binding, and exact cleanup**
|
||
|
||
Scan every retained regular file and bounded report/log field for all fixture secret canaries and credential-shaped URLs. Hash all declared artifacts, then remove only containers/volumes/network/proxy image/reference carrying this run's exact Compose project/run label. Prove no owned resource remains and no pre-existing resource was changed. There is no global `docker system prune`, volume prefix wildcard, or broad process kill.
|
||
|
||
- [ ] **Step 7: Verify harness syntax/unit contract**
|
||
|
||
```bash
|
||
bash -n scripts/p4-acceptance.sh scripts/test-p4-acceptance.sh
|
||
node --check backend/scripts/p4-acceptance.mjs
|
||
node --check backend/scripts/p4-acceptance.test.mjs
|
||
node --check backend/scripts/fixtures/p4-qdrant-proxy.mjs
|
||
./scripts/test-p4-acceptance.sh
|
||
```
|
||
|
||
Expected: PASS without starting the full process from the unit-contract test.
|
||
|
||
- [ ] **Step 8: Commit the process goal**
|
||
|
||
```bash
|
||
git add scripts/p4-acceptance.sh scripts/test-p4-acceptance.sh \
|
||
backend/scripts/p4-acceptance.mjs backend/scripts/p4-acceptance.test.mjs \
|
||
backend/scripts/fixtures/p4-qdrant-proxy.mjs .gitignore
|
||
git commit -m "test: add p4 qdrant lifecycle acceptance"
|
||
```
|
||
|
||
- [ ] **Step 9: Confirm the goal is runnable, but defer the one authoritative full run**
|
||
|
||
```bash
|
||
git status --short
|
||
./scripts/test-p4-acceptance.sh
|
||
```
|
||
|
||
Expected: clean status and PASS. Do not launch the authoritative Docker/process run yet: Task 10 still adds the manual helper/docs included in the final source boundary. The one retained release run occurs in Task 11 from the final clean implementation/manual-documentation commit.
|
||
|
||
---
|
||
|
||
### Task 10: Publish the independent manual walkthrough
|
||
|
||
**Files:**
|
||
- Modify: `docs/testing/p2-p6-manual-verification.md` (replace only the P4 placeholder section)
|
||
- Create: `scripts/p4-manual-verification.sh`
|
||
- Create: `scripts/test-p4-manual-verification.sh`
|
||
- Modify: `docs/install/local-workspace-registry.md`
|
||
- Modify: `docs/install/server-workspace-registry.md`
|
||
- Modify: `tools/thothctl/cmd/thothctl/main.go` usage text if not already complete
|
||
|
||
- [ ] **Step 1: Write RED manual-helper contract tests**
|
||
|
||
The helper accepts only `prepare`, `interrupt-after-delete`, `release-recovery`, `status`, and `cleanup`; owns `.artifacts/manual-acceptance/p4`; refuses reuse without cleanup; emits `GUIDE.md`; uses a new Git remote/Compose project/volumes distinct from automated state; never performs the reviewer commands or records PASS on the reviewer's behalf.
|
||
|
||
```bash
|
||
./scripts/test-p4-manual-verification.sh
|
||
```
|
||
|
||
Expected: FAIL until the helper exists.
|
||
|
||
- [ ] **Step 2: Implement prepare/status/fault-control/cleanup only**
|
||
|
||
`prepare` builds the fixture and prints exact non-secret variables. `interrupt-after-delete` arms the deterministic fixture barrier but does not invoke rebuild. `release-recovery` releases only that barrier. `cleanup` removes only the manual run's labelled resources/root after explicit reviewer confirmation. No helper calls `workspace collection rebuild/recover` for the reviewer.
|
||
|
||
- [ ] **Step 3: Replace the P4 walkthrough placeholder with exact commands**
|
||
|
||
Document, in order:
|
||
|
||
1. prepare new manual state;
|
||
2. `workspace inspect --json` on missing collection;
|
||
3. trigger one admission and inspect exact self-healed contract;
|
||
4. delete one safe fixture keyword index, trigger admission, and inspect its repair;
|
||
5. seed dimension, distance, and index-type incompatibilities and record nonmutation refusal;
|
||
6. run workspace/collection confirmation mismatches and prove marker/core/collection unchanged;
|
||
7. create a fixture open session and prove `session_inventory_active`;
|
||
8. hold the P2 writer lock and prove `preprocessing_conflict` before delete with safe restart/clear;
|
||
9. run the exact confirmed rebuild, inspect terminal state, neighbor preservation, healthy core, and inactive maintenance;
|
||
10. arm interruption, run rebuild, observe core stopped/maintenance retained/recovery state, release barrier, run exact confirmed recovery, and verify final state;
|
||
11. inspect report/state without secrets, decide PASS/FAIL, then cleanup.
|
||
|
||
For every step explain the component crossed, expected JSON fields/code, state/artifact produced, and invariant. Include `Decision: **PENDING**` and blank reviewer/date/evidence fields; only the human changes it to PASS/FAIL.
|
||
|
||
- [ ] **Step 4: Update operator manuals**
|
||
|
||
Explain self-heal vs destructive rebuild, exact confirmations, closed/finalized/archived inventory, lifecycle lock contention with Pi update, maintenance marker semantics, writer-lock refusal, post-delete recovery, intentional vector loss/reindex sequence, and backup recommendation. Do not claim P5/P6 materialization or automated reindexing.
|
||
|
||
- [ ] **Step 5: Verify docs/helper**
|
||
|
||
```bash
|
||
bash -n scripts/p4-manual-verification.sh scripts/test-p4-manual-verification.sh
|
||
./scripts/test-p4-manual-verification.sh
|
||
./scripts/verify-workspace-install-docs.sh --fixtures-only
|
||
```
|
||
|
||
Expected: PASS; the living P4 section contains no placeholder text and remains PENDING.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```bash
|
||
git add docs/testing/p2-p6-manual-verification.md \
|
||
docs/install/local-workspace-registry.md docs/install/server-workspace-registry.md \
|
||
scripts/p4-manual-verification.sh scripts/test-p4-manual-verification.sh \
|
||
tools/thothctl/cmd/thothctl/main.go
|
||
git commit -m "docs: add p4 qdrant lifecycle walkthrough"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 11: Final verification, checkpoint report, and hard stop
|
||
|
||
**Files:**
|
||
- Create after green evidence exists: `docs/testing/p4-qdrant-bootstrap-rebuild-checkpoint.md`
|
||
- Modify: `PROJECT_STATE.md`
|
||
|
||
- [ ] **Step 1: Run the scoped P4 verification from a clean commit**
|
||
|
||
```bash
|
||
git status --short
|
||
cd backend && npx vitest run \
|
||
test/qdrant-collection-manager.test.ts \
|
||
test/tht-qdrant-readiness.test.ts \
|
||
test/readiness-manager.test.ts \
|
||
test/routes-sessions.test.ts \
|
||
test/maintenance-gate.test.ts \
|
||
test/pi-process-manager.test.ts \
|
||
test/qdrant-rebuild-state.test.ts \
|
||
test/workspace-preprocessing-state.test.ts \
|
||
test/workspace-preprocessing-service.test.ts \
|
||
test/workspace-maintenance.test.ts
|
||
cd backend && npx tsc --noEmit -p . && npm run build
|
||
cd tools/thothctl && go test ./...
|
||
./scripts/test-default-compose.sh
|
||
./scripts/test-unified-compose.sh
|
||
./scripts/test-internal-semantic-compose.sh
|
||
./scripts/test-compose-secret-policy.sh
|
||
./scripts/test-no-deployment-coupling.sh
|
||
./scripts/test-deployment-command-contract.sh
|
||
./scripts/test-p4-acceptance.sh
|
||
./scripts/test-p4-manual-verification.sh
|
||
./scripts/verify-workspace-install-docs.sh --fixtures-only
|
||
git diff --check
|
||
git status --short
|
||
```
|
||
|
||
Expected: every command exits 0; TypeScript/build and all Go tests PASS; final tracked status is clean. Do not run or claim the aggregate P2–P6 smoke, full harness/frontend suites, real PSD, or P5/P6 flows—those remain the post-P6 gate.
|
||
|
||
- [ ] **Step 2: Run the one authoritative clean-state process and retain it**
|
||
|
||
```bash
|
||
./scripts/p4-acceptance.sh integration --keep
|
||
```
|
||
|
||
Expected: exit 0; `report.json` and `report.md` say every named check PASS, `automated integration: PASS`, `manual acceptance: PENDING`, `secret_scan: PASS`, `cleanup: PASS`, and `retry_count: 0`. If it fails, retain the failed run, diagnose the root cause, add a focused regression, fix and commit, rerun Step 1, then launch a **new run ID from the beginning**—never resume a partial run or retry a mutation inside a run.
|
||
|
||
- [ ] **Step 3: Verify the retained process report independently**
|
||
|
||
Check the retained P4 run is bound to the clean Step 1 source commit/tree, contains one scenario execution and no automatic retry, has all named checks PASS, secret scan PASS, exact cleanup PASS, a valid artifact SHA-256 manifest, no undeclared regular files, and no owned live Docker resource. Record `report.json` and `report.md` hashes.
|
||
|
||
- [ ] **Step 4: Write the checkpoint report**
|
||
|
||
`docs/testing/p4-qdrant-bootstrap-rebuild-checkpoint.md` must state:
|
||
|
||
- source commit/tree and plan/design/PRD references;
|
||
- exact commands and outcomes (do not invent counts);
|
||
- retained report path and hashes;
|
||
- self-heal, manager compatible-creator race, local route deduplication, distinct-principal route convergence, and incompatibility evidence;
|
||
- lifecycle lock, inventory, admissions/Pi quiescence, writer lock, stop/restart evidence;
|
||
- successful rebuild and interrupted recovery evidence;
|
||
- secret-scan and exact-cleanup evidence;
|
||
- `automated integration: PASS` and `manual acceptance: PENDING`;
|
||
- explicit P5/P6/post-P6 exclusions and any real-platform checks not run.
|
||
|
||
Update only the evolving P4 section of `PROJECT_STATE.md` with the same truthful status.
|
||
|
||
- [ ] **Step 5: Commit checkpoint documentation**
|
||
|
||
```bash
|
||
git add docs/testing/p4-qdrant-bootstrap-rebuild-checkpoint.md PROJECT_STATE.md
|
||
git commit -m "docs: record p4 automated checkpoint"
|
||
```
|
||
|
||
The retained run is intentionally bound to this checkpoint commit's parent: the final clean implementation and manual-documentation source boundary. The docs-only checkpoint records immutable report hashes and is not falsely claimed as input to its own report.
|
||
|
||
- [ ] **Step 6: Complete the persistent process goal and stop**
|
||
|
||
Require a clean worktree, then complete the goal only after the authoritative run and cleanup proof. Send the user the checkpoint report and exact P4 manual section. Stop for explicit authorization/manual decision; do not begin P5, P6, aggregate verification, or unrelated cleanup.
|
||
|
||
## Manual acceptance completion (later, reviewer-owned)
|
||
|
||
After the reviewer executes the independent P4 walkthrough, record only their actual decision and evidence. If PASS, update the P4 checkpoint and `PROJECT_STATE.md` from `manual acceptance: PENDING` to PASS in one docs-only commit. If FAIL, retain the failure evidence, reopen the P4 goal, add a focused regression, and repeat the entire automated process from a new clean run before asking for another manual decision.
|