770 lines
26 KiB
Markdown
770 lines
26 KiB
Markdown
# Internal Qdrant and Ollama Implementation Plan
|
|
|
|
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
|
|
|
**Goal:** Make Qdrant and Ollama mandatory internal ThothII services while keeping the analytical
|
|
DWH external and associating each workspace with one Qdrant collection for schema, Evidence, and
|
|
Memory embeddings.
|
|
|
|
**Architecture:** Introduce workspace schema v3, preserve v1/v2 only as migration inputs, and keep
|
|
the existing harness `VectorStore` port behind a new Qdrant REST adapter. Base Compose owns Qdrant,
|
|
Ollama, their persistent volumes, and model initialization; workspace descriptors contain semantic
|
|
identity but no vector/embedding endpoints or credentials.
|
|
|
|
**Tech Stack:** TypeScript/Fastify/Zod, Python 3.12/Pydantic/requests, React 18, Docker Compose,
|
|
Qdrant REST API, Ollama `/api/embed`, Vitest, pytest.
|
|
|
|
---
|
|
|
|
## Guardrails
|
|
|
|
- Apply `@superpowers:test-driven-development` to every behavior change: add one focused failing
|
|
test, observe the expected failure, implement the minimum, and rerun the focused test.
|
|
- Do not run broad suites until the corresponding code/config changes exist; this preserves the
|
|
requested ordering while still using TDD.
|
|
- Preserve the external DWH connector contract and session persistence model.
|
|
- Do not retain an operational fallback to pgvector or an external embedding endpoint.
|
|
- Do not delete or rewrite user workspace repositories or Qdrant data. Migration is descriptor-only;
|
|
semantic data is rebuilt explicitly.
|
|
- Commit after each task only when focused tests are green.
|
|
|
|
### Task 1: Define workspace schema v3
|
|
|
|
**Files:**
|
|
|
|
- Modify: `backend/src/workspaces/schema.ts`
|
|
- Modify: `backend/src/workspaces/types.ts`
|
|
- Modify: `backend/test/workspaces-schema.test.ts`
|
|
- Modify: `backend/test/workspaces-migrate-legacy.test.ts`
|
|
- Create: `backend/src/workspaces/migrate-v2-qdrant.ts`
|
|
- Create: `backend/test/workspaces-migrate-v2-qdrant.test.ts`
|
|
|
|
**Step 1: Write the failing schema tests**
|
|
|
|
Add tests proving that schema v3 accepts only this semantic shape:
|
|
|
|
```ts
|
|
const semantic_index = {
|
|
vector_store: {
|
|
engine: "qdrant",
|
|
collection: "psd-clinical",
|
|
dimensions: 1024,
|
|
distance: "cosine",
|
|
},
|
|
embedding: {
|
|
provider: "ollama_internal",
|
|
model: "qwen3-embedding:0.6b",
|
|
dimensions: 1024,
|
|
},
|
|
};
|
|
```
|
|
|
|
Add separate rejection cases for `pgvector`, `supported_transports`, external embedding providers,
|
|
non-1024 dimensions, non-cosine distance, and unknown fields. Assert v1/v2 remain parseable as
|
|
legacy descriptors but `isOperationalWorkspace()` returns false.
|
|
|
|
**Step 2: Run the tests and verify RED**
|
|
|
|
Run:
|
|
|
|
```bash
|
|
cd backend
|
|
npx vitest run test/workspaces-schema.test.ts test/workspaces-migrate-v2-qdrant.test.ts
|
|
```
|
|
|
|
Expected: failure because schema version 3 and `migrateWorkspaceV2ToV3` do not exist.
|
|
|
|
**Step 3: Implement the minimum schema and migration**
|
|
|
|
Add `QdrantVectorStore`, `InternalEmbedding`, and `WorkspaceV3` types. Replace the operational type
|
|
guard with schema-v3-only semantics. Implement:
|
|
|
|
```ts
|
|
export function migrateWorkspaceV2ToV3(
|
|
legacy: WorkspaceV2,
|
|
collection: string,
|
|
): WorkspaceV3 {
|
|
return validateOperationalWorkspace({
|
|
workspace: { ...legacy.workspace, schema_version: 3 },
|
|
dwh: legacy.dwh,
|
|
semantic_index: {
|
|
vector_store: {
|
|
engine: "qdrant",
|
|
collection,
|
|
dimensions: 1024,
|
|
distance: "cosine",
|
|
},
|
|
embedding: {
|
|
provider: "ollama_internal",
|
|
model: "qwen3-embedding:0.6b",
|
|
dimensions: 1024,
|
|
},
|
|
},
|
|
llm_policy: legacy.llm_policy,
|
|
...(legacy.diagnostics?.dwh_rest
|
|
? { diagnostics: { dwh_rest: legacy.diagnostics.dwh_rest } }
|
|
: {}),
|
|
});
|
|
}
|
|
```
|
|
|
|
Do not copy vector/embedding diagnostics or transports.
|
|
|
|
**Step 4: Verify GREEN**
|
|
|
|
Run the command from Step 2. Expected: all selected tests pass.
|
|
|
|
**Step 5: Commit**
|
|
|
|
```bash
|
|
git add backend/src/workspaces/schema.ts backend/src/workspaces/types.ts \
|
|
backend/src/workspaces/migrate-v2-qdrant.ts backend/test/workspaces-schema.test.ts \
|
|
backend/test/workspaces-migrate-legacy.test.ts backend/test/workspaces-migrate-v2-qdrant.test.ts
|
|
git commit -m "feat: define internal semantic workspace schema"
|
|
```
|
|
|
|
### Task 2: Make collection ownership unique in the Git registry
|
|
|
|
**Files:**
|
|
|
|
- Modify: `backend/src/workspaces/registry.ts`
|
|
- Modify: `backend/src/workspaces/migrate-legacy.ts`
|
|
- Modify: `backend/test/workspace-registry.test.ts`
|
|
- Modify: `backend/test/workspaces-migrate-legacy.test.ts`
|
|
|
|
**Step 1: Write failing registry tests**
|
|
|
|
Add fixtures with two schema-v3 workspaces claiming `collection: shared`. Assert snapshot activation
|
|
fails with `workspace_invalid` and retains the previous active snapshot. Assert v1/v2 entries are
|
|
listed as `migration_required` and cannot be acquired with `acquireSessionRevision()`.
|
|
|
|
**Step 2: Verify RED**
|
|
|
|
```bash
|
|
cd backend
|
|
npx vitest run test/workspace-registry.test.ts test/workspaces-migrate-legacy.test.ts \
|
|
-t "collection|migration_required"
|
|
```
|
|
|
|
Expected: duplicate collections are currently accepted and v2 is currently operational.
|
|
|
|
**Step 3: Implement uniqueness and migration state**
|
|
|
|
During snapshot validation, build `Map<collection, workspaceId>` for operational descriptors and
|
|
raise a sanitized `workspace_invalid` error on a duplicate. Update migration output and CLI wording
|
|
to require an explicit target collection and schema v3.
|
|
|
|
**Step 4: Verify GREEN and commit**
|
|
|
|
```bash
|
|
cd backend
|
|
npx vitest run test/workspace-registry.test.ts test/workspaces-migrate-legacy.test.ts \
|
|
-t "collection|migration_required"
|
|
cd ..
|
|
git add backend/src/workspaces/registry.ts backend/src/workspaces/migrate-legacy.ts \
|
|
backend/test/workspace-registry.test.ts backend/test/workspaces-migrate-legacy.test.ts
|
|
git commit -m "feat: reserve one qdrant collection per workspace"
|
|
```
|
|
|
|
### Task 3: Remove external semantic bindings and render internal endpoints
|
|
|
|
**Files:**
|
|
|
|
- Modify: `backend/src/workspaces/contracts.ts`
|
|
- Modify: `backend/src/workspaces/bindings.ts`
|
|
- Modify: `backend/src/workspaces/runtime-renderer.ts`
|
|
- Modify: `backend/src/config.ts`
|
|
- Modify: `backend/test/workspaces-contracts.test.ts`
|
|
- Modify: `backend/test/workspaces-bindings.test.ts`
|
|
- Modify: `backend/test/workspace-runtime-renderer.test.ts`
|
|
- Modify: `backend/test/config.test.ts`
|
|
|
|
**Step 1: Write failing contract tests**
|
|
|
|
Assert schema-v3 installation contracts contain DWH variables only. Assert environment variables
|
|
matching `*_VECTOR_*`, `*_EMBEDDING_BASE_URL`, or semantic API-key suffixes are ignored/rejected.
|
|
Assert the rendered harness config always contains:
|
|
|
|
```yaml
|
|
resources:
|
|
vector:
|
|
engine: qdrant
|
|
base_url: http://qdrant:6333
|
|
collection: psd-clinical
|
|
embeddings:
|
|
provider: ollama_internal
|
|
base_url: http://embedding:11434
|
|
model: qwen3-embedding:0.6b
|
|
dimensions: 1024
|
|
```
|
|
|
|
**Step 2: Verify RED**
|
|
|
|
```bash
|
|
cd backend
|
|
npx vitest run test/workspaces-contracts.test.ts test/workspaces-bindings.test.ts \
|
|
test/workspace-runtime-renderer.test.ts test/config.test.ts
|
|
```
|
|
|
|
Expected: current contracts require external vector and embedding bindings.
|
|
|
|
**Step 3: Implement internal runtime configuration**
|
|
|
|
Add typed backend config fields with Compose defaults:
|
|
|
|
```ts
|
|
internalQdrantUrl: "http://qdrant:6333"
|
|
internalEmbeddingUrl: "http://embedding:11434"
|
|
internalEmbeddingModel: "qwen3-embedding:0.6b"
|
|
internalEmbeddingDimensions: 1024
|
|
```
|
|
|
|
Accept only `qdrant`, `embedding`, `localhost`, or loopback hosts. Keep these values out of Git
|
|
workspace descriptors, API payloads, and generated installation docs. Render them into the
|
|
ephemeral backend-owned harness config after descriptor validation.
|
|
|
|
**Step 4: Verify GREEN and commit**
|
|
|
|
Run Step 2, then:
|
|
|
|
```bash
|
|
git add backend/src/config.ts backend/src/workspaces/contracts.ts backend/src/workspaces/bindings.ts \
|
|
backend/src/workspaces/runtime-renderer.ts backend/test/config.test.ts \
|
|
backend/test/workspaces-contracts.test.ts backend/test/workspaces-bindings.test.ts \
|
|
backend/test/workspace-runtime-renderer.test.ts
|
|
git commit -m "feat: render private semantic service endpoints"
|
|
```
|
|
|
|
### Task 4: Narrow harness embedding configuration to internal Ollama
|
|
|
|
**Files:**
|
|
|
|
- Modify: `harness/tht/config.py`
|
|
- Modify: `harness/tht/config_compat.py`
|
|
- Modify: `harness/tht/vectorstore/embeddings.py`
|
|
- Modify: `harness/tht/cli/ollama_cmd.py`
|
|
- Modify: `harness/tests/test_config_resources.py`
|
|
- Create: `harness/tests/test_internal_embeddings.py`
|
|
|
|
**Step 1: Write failing embedding tests**
|
|
|
|
Use a fake `requests.Session` to prove `OllamaInternalEmbeddings.embed()` calls `/api/embed` with
|
|
model and batch input, returns 1024-dimensional finite vectors, and rejects count/dimension/NaN
|
|
mismatches. Add config tests rejecting external providers, API keys, and non-private base URLs.
|
|
|
|
**Step 2: Verify RED**
|
|
|
|
```bash
|
|
cd harness
|
|
.venv/bin/pytest tests/test_internal_embeddings.py tests/test_config_resources.py -q
|
|
```
|
|
|
|
Expected: `OllamaInternalEmbeddings` and internal-only config do not exist.
|
|
|
|
**Step 3: Implement the client**
|
|
|
|
Implement one bounded `/api/embed` request per batch:
|
|
|
|
```python
|
|
response = self._session.post(
|
|
f"{self.base_url}/api/embed",
|
|
json={"model": self.model, "input": texts},
|
|
timeout=self.timeout,
|
|
)
|
|
```
|
|
|
|
Validate response shape before returning any vector. Keep retry behavior bounded and sanitize URLs
|
|
and response bodies from raised errors.
|
|
|
|
**Step 4: Verify GREEN and commit**
|
|
|
|
```bash
|
|
cd harness
|
|
.venv/bin/pytest tests/test_internal_embeddings.py tests/test_config_resources.py -q
|
|
cd ..
|
|
git add harness/tht/config.py harness/tht/config_compat.py harness/tht/vectorstore/embeddings.py \
|
|
harness/tht/cli/ollama_cmd.py harness/tests/test_config_resources.py \
|
|
harness/tests/test_internal_embeddings.py
|
|
git commit -m "feat: use internal ollama embeddings"
|
|
```
|
|
|
|
### Task 5: Implement the Qdrant VectorStore adapter
|
|
|
|
**Files:**
|
|
|
|
- Create: `harness/tht/adapters/vector/qdrant.py`
|
|
- Modify: `harness/tht/adapters/vector/__init__.py`
|
|
- Modify: `harness/tht/ports/vector.py`
|
|
- Modify: `harness/tht/vectorstore/records.py`
|
|
- Modify: `harness/tht/vectorstore/store.py`
|
|
- Create: `harness/tests/test_qdrant_vector_store.py`
|
|
- Modify: `harness/tests/test_vector_port_contract.py`
|
|
|
|
**Step 1: Write failing adapter tests**
|
|
|
|
Test a real adapter against a deterministic fake HTTP server. Cover:
|
|
|
|
- idempotent collection create with 1024/Cosine;
|
|
- mismatch fails without delete/recreate;
|
|
- keyword payload-index creation;
|
|
- deterministic UUIDv5 point IDs;
|
|
- upsert payload for `schema`, `evidence`, and `memory`;
|
|
- query filtered by workspace and allowed kinds;
|
|
- `existing_hashes`, exact Evidence generation list/delete, and health;
|
|
- sanitized timeouts and malformed responses.
|
|
|
|
The point ID helper must satisfy:
|
|
|
|
```python
|
|
def point_id(workspace_id: str, kind: str, record_key: str) -> str:
|
|
return str(uuid5(NAMESPACE_URL, f"thothii:{workspace_id}:{kind}:{record_key}"))
|
|
```
|
|
|
|
**Step 2: Verify RED**
|
|
|
|
```bash
|
|
cd harness
|
|
.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -q
|
|
```
|
|
|
|
Expected: import failure for the Qdrant adapter.
|
|
|
|
**Step 3: Implement minimal REST mappings**
|
|
|
|
Use existing `requests` dependency and these endpoints:
|
|
|
|
```text
|
|
GET /collections/{collection}
|
|
PUT /collections/{collection}
|
|
PUT /collections/{collection}/index
|
|
PUT /collections/{collection}/points?wait=true
|
|
POST /collections/{collection}/points/query
|
|
POST /collections/{collection}/points/scroll
|
|
POST /collections/{collection}/points/delete?wait=true
|
|
```
|
|
|
|
Every operation must include the workspace filter even though the collection is workspace-owned.
|
|
Map Qdrant scores and payloads back into existing `VectorHit` objects.
|
|
|
|
**Step 4: Verify GREEN and commit**
|
|
|
|
```bash
|
|
cd harness
|
|
.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -q
|
|
cd ..
|
|
git add harness/tht/adapters/vector/qdrant.py harness/tht/adapters/vector/__init__.py \
|
|
harness/tht/ports/vector.py harness/tht/vectorstore/records.py \
|
|
harness/tht/vectorstore/store.py harness/tests/test_qdrant_vector_store.py \
|
|
harness/tests/test_vector_port_contract.py
|
|
git commit -m "feat: add qdrant vector adapter"
|
|
```
|
|
|
|
### Task 6: Wire schema, Evidence, and Memory through Qdrant
|
|
|
|
**Files:**
|
|
|
|
- Modify: `harness/tht/vectorstore/reader.py`
|
|
- Modify: `harness/tht/cli/vector_cmd.py`
|
|
- Modify: `harness/tht/cli/memory_cmd.py`
|
|
- Modify: `harness/tht/corpus/pipeline.py`
|
|
- Modify: `harness/tht/search/evidence.py`
|
|
- Modify: `harness/tht/cli/schema_cmd.py`
|
|
- Modify: `harness/tests/test_memory_save_one.py`
|
|
- Modify: `harness/tests/test_search_pack.py`
|
|
- Create: `harness/tests/test_semantic_kind_isolation.py`
|
|
|
|
**Step 1: Write failing integration tests**
|
|
|
|
Use an in-memory fake implementing the `VectorStore` port. Assert:
|
|
|
|
- schema records use `kind=schema`;
|
|
- corpus records use `kind=evidence` and exact generation;
|
|
- approved memories use `kind=memory`;
|
|
- search pack requests only its allowed kind set;
|
|
- all three paths share `workspace_id`, `workspace_revision`, hashing, and point-key construction;
|
|
- retries do not duplicate points.
|
|
|
|
**Step 2: Verify RED**
|
|
|
|
```bash
|
|
cd harness
|
|
.venv/bin/pytest tests/test_semantic_kind_isolation.py tests/test_memory_save_one.py \
|
|
tests/test_search_pack.py -q
|
|
```
|
|
|
|
Expected: current factories select pgvector/HTTP adapters and payloads lack the v3 identity fields.
|
|
|
|
**Step 3: Wire the adapter**
|
|
|
|
Make schema-v3 `qdrant` the only operational vector factory branch. Reuse the current canonical
|
|
record builders; add only missing identity fields. Keep the JSONL Memory registry and filesystem
|
|
Evidence corpus as sources of truth.
|
|
|
|
**Step 4: Verify GREEN and commit**
|
|
|
|
Run Step 2, then commit the listed files with:
|
|
|
|
```bash
|
|
git commit -m "feat: index semantic records in qdrant"
|
|
```
|
|
|
|
### Task 7: Add mandatory Qdrant and Ollama Compose services
|
|
|
|
**Files:**
|
|
|
|
- Modify: `compose.yaml`
|
|
- Create: `deploy/compose.embedding-gpu.yaml`
|
|
- Create: `docker/embedding-model-init.sh`
|
|
- Modify: `docker/core.Dockerfile`
|
|
- Modify: `deploy/env/local.env.example`
|
|
- Modify: `deploy/env/server.env.example`
|
|
- Modify: `scripts/run-stack.sh`
|
|
- Modify: `scripts/test-default-compose.sh`
|
|
- Modify: `scripts/test-unified-compose.sh`
|
|
- Create: `scripts/test-internal-semantic-compose.sh`
|
|
|
|
**Step 1: Write failing Compose contract tests**
|
|
|
|
Assert the rendered base profile has `core`, `frontend`, `qdrant`, `embedding`, and
|
|
`embedding-model-init`; private services have no published ports; persistent volumes exist; core
|
|
depends on Qdrant health and successful model init; no external vector/embedding binding is required.
|
|
|
|
Also assert all service images use version plus immutable digest. Resolve and record supported
|
|
multi-architecture digests for Qdrant v1.18.x and Ollama v0.32.x during implementation:
|
|
|
|
```bash
|
|
docker buildx imagetools inspect qdrant/qdrant:v1.18.2
|
|
docker buildx imagetools inspect ollama/ollama:0.32.0
|
|
```
|
|
|
|
**Step 2: Verify RED**
|
|
|
|
```bash
|
|
./scripts/test-default-compose.sh
|
|
./scripts/test-unified-compose.sh
|
|
./scripts/test-internal-semantic-compose.sh
|
|
```
|
|
|
|
Expected: required services and volumes are absent.
|
|
|
|
**Step 3: Implement the services**
|
|
|
|
`embedding-model-init.sh` must wait with a bounded deadline, call `ollama pull` for the exact model,
|
|
and verify it appears in `/api/tags`. The Qdrant healthcheck uses its HTTP health endpoint. The CPU
|
|
base has no device reservation; the GPU override adds only the supported device stanza.
|
|
|
|
**Step 4: Verify GREEN and commit**
|
|
|
|
Run Step 2, then:
|
|
|
|
```bash
|
|
git add compose.yaml deploy/compose.embedding-gpu.yaml docker/embedding-model-init.sh \
|
|
docker/core.Dockerfile deploy/env/local.env.example deploy/env/server.env.example \
|
|
scripts/run-stack.sh scripts/test-default-compose.sh scripts/test-unified-compose.sh \
|
|
scripts/test-internal-semantic-compose.sh
|
|
git commit -m "feat: run qdrant and ollama inside thothii"
|
|
```
|
|
|
|
### Task 8: Retire pgvector deployment and external semantic connectors
|
|
|
|
**Files:**
|
|
|
|
- Delete: `deploy/compose.local-vector.yaml`
|
|
- Delete: `deploy/compose.preprocess-local-vector.yaml`
|
|
- Delete: `deploy/sql/20-vector-roles.sql`
|
|
- Delete: `deploy/vector/reconcile-roles.sh`
|
|
- Delete: `deploy/vector/rotate-bootstrap-password.py`
|
|
- Delete: `deploy/vector/secret-policy.sh`
|
|
- Delete: `deploy/vector/vector-db-entrypoint.sh`
|
|
- Delete: `scripts/local-vector-smoke.sh`
|
|
- Delete: `scripts/test-local-vector-smoke-safety.sh`
|
|
- Delete: `scripts/test-local-vector-smoke-live-collision.sh`
|
|
- Delete: `scripts/test-vector-bootstrap-rotation.sh`
|
|
- Delete: `scripts/test-vector-migration-image.sh`
|
|
- Delete: `scripts/test-vector-secret-policy.sh`
|
|
- Modify: `scripts/test-no-deployment-coupling.sh`
|
|
- Modify: `scripts/test-no-deployment-coupling-scope.sh`
|
|
- Modify: `scripts/test-compose-secret-policy.sh`
|
|
- Modify: `.github/workflows/deployment.yml`
|
|
|
|
**Step 1: Write the failing coupling test**
|
|
|
|
Teach the coupling gate to reject active `pgvector`, `local-vector`, `THT_VECTOR_*`, workspace
|
|
embedding URLs/API keys, and external vector transports while allowing historical specs and the
|
|
explicit descriptor migration module.
|
|
|
|
**Step 2: Verify RED**
|
|
|
|
```bash
|
|
./scripts/test-no-deployment-coupling-scope.sh
|
|
./scripts/test-no-deployment-coupling.sh
|
|
./scripts/test-compose-secret-policy.sh
|
|
```
|
|
|
|
Expected: active pgvector deployment paths are reported.
|
|
|
|
**Step 3: Remove the retired paths and update CI**
|
|
|
|
Remove only repository deployment machinery. Retain harness pgvector code temporarily only if it
|
|
is needed to read/export legacy data during migration; it must not be reachable from schema v3 or
|
|
Compose. Remove it in a follow-up task once migration fixtures no longer import it.
|
|
|
|
**Step 4: Verify GREEN and commit**
|
|
|
|
Run Step 2 and the workflow fixture tests, then commit all deletions and modifications:
|
|
|
|
```bash
|
|
git add -A deploy scripts .github/workflows/deployment.yml
|
|
git commit -m "refactor: retire external vector deployment"
|
|
```
|
|
|
|
### Task 9: Update frontend workspace editing and examples
|
|
|
|
**Files:**
|
|
|
|
- Modify: `frontend/src/api/workspaces.ts`
|
|
- Modify: `frontend/src/shell/WorkspaceEditor.tsx`
|
|
- Modify: `frontend/src/shell/WorkspaceEditor.test.tsx`
|
|
- Modify: `frontend/src/shell/WorkspaceManager.test.tsx`
|
|
- Modify: `frontend/src/api/workspaces.test.ts`
|
|
- Modify: `frontend/src/workspaces/drafts.test.ts`
|
|
- Modify: `deploy/workspaces/example.yaml`
|
|
- Modify: `deploy/workspaces/psd.yaml.example`
|
|
|
|
**Step 1: Write failing UI tests**
|
|
|
|
Assert editor/preview show Qdrant collection and fixed internal embedding model, expose no vector
|
|
endpoint/credential fields, and publish schema v3. Assert legacy descriptors display a migration
|
|
banner and cannot be selected for a new session.
|
|
|
|
**Step 2: Verify RED**
|
|
|
|
```bash
|
|
cd frontend
|
|
npx vitest run src/shell/WorkspaceEditor.test.tsx src/shell/WorkspaceManager.test.tsx \
|
|
src/api/workspaces.test.ts src/workspaces/drafts.test.ts
|
|
```
|
|
|
|
Expected: fixtures and controls still use pgvector/external embedding.
|
|
|
|
**Step 3: Implement fixed semantic controls**
|
|
|
|
Collection remains editable and validated. Engine, provider, model, dimensions, and distance render
|
|
as fixed architecture values. Remove external semantic diagnostics from drafts and publish payloads.
|
|
|
|
**Step 4: Verify GREEN and commit**
|
|
|
|
Run Step 2, then commit the listed files with:
|
|
|
|
```bash
|
|
git commit -m "feat: edit qdrant workspace collections"
|
|
```
|
|
|
|
### Task 10: Add a real internal semantic smoke
|
|
|
|
**Files:**
|
|
|
|
- Create: `scripts/internal-semantic-smoke.sh`
|
|
- Modify: `scripts/unified-deployment-smoke.sh`
|
|
- Modify: `scripts/server-deployment-smoke.sh`
|
|
- Modify: `scripts/task13-runtime-fixture-check.ts`
|
|
- Modify: `scripts/test-task13-runtime-fixtures.sh`
|
|
|
|
**Step 1: Write failing smoke fixture assertions**
|
|
|
|
The fixture must require private Qdrant/Ollama services, model volume, Qdrant volume, fixed internal
|
|
URLs, and no host ports. It must reject wrong service names, external URLs, collection reuse, and
|
|
dimension changes.
|
|
|
|
**Step 2: Verify RED**
|
|
|
|
```bash
|
|
./scripts/test-task13-runtime-fixtures.sh local
|
|
./scripts/test-task13-runtime-fixtures.sh server
|
|
```
|
|
|
|
Expected: current fixture expects the two-service topology.
|
|
|
|
**Step 3: Implement the live smoke**
|
|
|
|
Using disposable volumes and a fixture workspace, start the stack on CPU, wait for the model, ensure
|
|
the collection, embed one record of each kind, query each kind with filters, restart offline, and
|
|
prove all points and the model remain available. Cleanup must remain exact and must not prune global
|
|
Docker resources.
|
|
|
|
**Step 4: Verify GREEN and commit**
|
|
|
|
```bash
|
|
./scripts/test-task13-runtime-fixtures.sh local
|
|
./scripts/test-task13-runtime-fixtures.sh server
|
|
./scripts/internal-semantic-smoke.sh
|
|
git add scripts/internal-semantic-smoke.sh scripts/unified-deployment-smoke.sh \
|
|
scripts/server-deployment-smoke.sh scripts/task13-runtime-fixture-check.ts \
|
|
scripts/test-task13-runtime-fixtures.sh
|
|
git commit -m "test: cover internal semantic services"
|
|
```
|
|
|
|
### Task 11: Update operator documentation and state
|
|
|
|
**Files:**
|
|
|
|
- Modify: `README.md`
|
|
- Modify: `AGENTS.md`
|
|
- Modify: `PROJECT_STATE.md`
|
|
- Modify: `docs/install/local-workspace-registry.md`
|
|
- Modify: `docs/install/server-workspace-registry.md`
|
|
- Modify: `docs/installazione-docker-4-contesti.md`
|
|
- Modify: `docs/workspace-diagnostic-protocol.md`
|
|
- Modify: `docs/gestione-memory.md`
|
|
- Modify: `deploy/secrets/README.md`
|
|
- Modify: `scripts/verify-workspace-install-docs.sh`
|
|
- Modify: `scripts/test-verify-workspace-install-docs.sh`
|
|
|
|
**Step 1: Write failing documentation contract assertions**
|
|
|
|
Require the four-service topology, CPU/GPU behavior, volume backup/restore, schema-v3 migration,
|
|
Qdrant collection ownership, and removal of external vector/embedding variables from active manuals.
|
|
|
|
**Step 2: Verify RED**
|
|
|
|
```bash
|
|
./scripts/test-verify-workspace-install-docs.sh
|
|
./scripts/verify-workspace-install-docs.sh --fixtures-only
|
|
```
|
|
|
|
Expected: manuals still describe external pgvector/embedding and a two-service mandatory stack.
|
|
|
|
**Step 3: Update documentation**
|
|
|
|
Document Qdrant as a derived but persistent index, Ollama model cache behavior, CPU-first startup,
|
|
optional GPU override, snapshot/restore, explicit legacy migration, and the fact that only the DWH
|
|
and LLM remain external application endpoints.
|
|
|
|
**Step 4: Verify GREEN and commit**
|
|
|
|
Run Step 2, then:
|
|
|
|
```bash
|
|
git add README.md AGENTS.md PROJECT_STATE.md docs deploy/secrets/README.md \
|
|
scripts/verify-workspace-install-docs.sh scripts/test-verify-workspace-install-docs.sh
|
|
git commit -m "docs: document internal semantic infrastructure"
|
|
```
|
|
|
|
### Task 12: Remove unreachable pgvector runtime code
|
|
|
|
**Files:**
|
|
|
|
- Delete: `harness/tht/adapters/vector/pgvector.py`
|
|
- Delete: `harness/tht/adapters/vector/legacy_direct.py`
|
|
- Delete: `harness/tht/adapters/vector/thoth_http.py`
|
|
- Delete: `harness/tht/vectorstore/rest_client.py`
|
|
- Delete: `harness/tht/vectorstore/rest_writer.py`
|
|
- Delete: `harness/tht/migrations/vector/001_extensions.sql`
|
|
- Delete: `harness/tht/migrations/vector/002_schema_tables.sql`
|
|
- Delete: `harness/tht/migrations/vector/003_roles.sql`
|
|
- Delete: `harness/tht/migrations/vector/004_evidence_generation_gc.sql`
|
|
- Modify: `harness/pyproject.toml`
|
|
- Modify/Delete: affected pgvector and migration tests under `harness/tests/l0/`
|
|
|
|
**Step 1: Prove the code is unreachable**
|
|
|
|
```bash
|
|
rg -n "PgVectorStore|ThothHttpVectorStore|LegacyDirectVectorStore|migrations/vector" \
|
|
harness backend frontend compose.yaml deploy scripts docker docs \
|
|
--glob '!docs/plans/**' --glob '!docs/superpowers/**'
|
|
```
|
|
|
|
Expected before cleanup: matches only in the files scheduled for deletion and legacy tests. If an
|
|
operational call site remains, stop and migrate it before deleting anything.
|
|
|
|
**Step 2: Delete obsolete runtime and tests**
|
|
|
|
Retain descriptor migration tests, but remove PostgreSQL vector runtime/migration packaging tests.
|
|
Remove `psycopg2-binary` only if the DWH/session PostgreSQL paths do not need it; otherwise keep it.
|
|
|
|
**Step 3: Verify focused imports and packaging**
|
|
|
|
```bash
|
|
cd harness
|
|
.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py \
|
|
tests/test_semantic_kind_isolation.py tests/test_vector_migration_packaging.py -q
|
|
python -m build
|
|
```
|
|
|
|
Expected: Qdrant tests pass and the wheel contains no pgvector migrations. Adjust the packaging test
|
|
to assert Qdrant has no SQL migration payload.
|
|
|
|
**Step 4: Commit**
|
|
|
|
```bash
|
|
git add -A harness
|
|
git commit -m "refactor: remove pgvector runtime"
|
|
```
|
|
|
|
### Task 13: Run complete verification
|
|
|
|
**Files:**
|
|
|
|
- Modify only if a genuine regression is discovered.
|
|
|
|
**Step 1: Deterministic layer gates**
|
|
|
|
```bash
|
|
cd harness && .venv/bin/pytest -q && .venv/bin/ruff check .
|
|
cd ../backend && npx vitest run && npx tsc --noEmit -p . && npm run build
|
|
cd ../frontend && npx vitest run && npx tsc -b && npm run build
|
|
cd .. && git diff --check
|
|
```
|
|
|
|
Expected: all gates pass. Existing unrelated Ruff debt must be reported separately if it remains;
|
|
new/modified files must be Ruff-clean.
|
|
|
|
**Step 2: Deployment contracts**
|
|
|
|
```bash
|
|
./scripts/test-default-compose.sh
|
|
./scripts/test-unified-compose.sh
|
|
./scripts/test-internal-semantic-compose.sh
|
|
./scripts/test-no-deployment-coupling.sh
|
|
./scripts/test-compose-secret-policy.sh
|
|
./scripts/verify-workspace-install-docs.sh --fixtures-only
|
|
```
|
|
|
|
Expected: all pass without external vector/embedding settings.
|
|
|
|
**Step 3: Docker smokes**
|
|
|
|
```bash
|
|
./scripts/internal-semantic-smoke.sh
|
|
./scripts/workspace-registry-smoke.sh
|
|
./scripts/unified-deployment-smoke.sh
|
|
./scripts/thothctl-update-smoke.sh
|
|
./scripts/server-deployment-smoke.sh
|
|
```
|
|
|
|
Expected: CPU semantic smoke passes, persistence survives offline restart, and every script proves
|
|
exact cleanup. Investigate the previously observed `thothctl` rollback failure independently if it
|
|
recurs; do not weaken the new semantic gate to hide it.
|
|
|
|
**Step 4: Final audit**
|
|
|
|
```bash
|
|
rg -n "pgvector|local-vector|THT_VECTOR_|EMBEDDING_BASE_URL|openai_compatible|ollama_compatible" \
|
|
. --glob '!docs/plans/**' --glob '!docs/superpowers/**' --glob '!**/node_modules/**' \
|
|
--glob '!**/.venv/**' --glob '!**/.git/**'
|
|
git status --short
|
|
```
|
|
|
|
Expected: no active operational references; only explicit legacy descriptor migration fixtures may
|
|
remain. Worktree contains only intentional changes.
|
|
|
|
**Step 5: Commit verification metadata**
|
|
|
|
Update `PROJECT_STATE.md` with exact counts, image digests, smoke durations, CPU hardware, and any
|
|
manual GPU/Windows gates. Commit only verified claims:
|
|
|
|
```bash
|
|
git add PROJECT_STATE.md
|
|
git commit -m "docs: record qdrant ollama verification"
|
|
```
|