212 lines
7.3 KiB
Markdown
212 lines
7.3 KiB
Markdown
# Optional Local pgvector Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Let server and desktop deployments run an optional persistent pgvector service with behavior equivalent to the HTTP vector adapter.
|
|
|
|
**Architecture:** Implement the final direct `VectorStore`, version vector schema migrations, add a standard pgvector image to Compose, and provide operational backup/restore commands.
|
|
|
|
**Tech Stack:** PostgreSQL 16, pgvector, psycopg2/SQLAlchemy, Alembic or ordered SQL migrations, Docker Compose, pytest/testcontainers.
|
|
|
|
## Global Constraints
|
|
|
|
- Plans 1 and 2 are complete.
|
|
- pgvector is an infrastructure image, not a ThothII-owned application image.
|
|
- Reader and writer roles are distinct even for local deployments.
|
|
- Existing remote HTTP vector behavior remains supported.
|
|
- Persistent data must survive application image replacement.
|
|
|
|
---
|
|
|
|
### Task 1: Implement direct pgvector store behind `VectorStore`
|
|
|
|
**Files:**
|
|
- Create: `harness/tht/adapters/vector/pgvector.py`
|
|
- Test: `harness/tests/l0/test_pgvector_store.py`
|
|
- Modify: `harness/tht/adapters/factory.py`
|
|
- Modify: `harness/tht/config.py`
|
|
|
|
**Interfaces:**
|
|
- Produces: `PgVectorStore(read_config, write_config=None)` implementing the Plan 1 port.
|
|
|
|
- [ ] **Step 1: Add L0 contract tests for search, kind filters, hashes, and upsert**
|
|
|
|
```python
|
|
def test_pgvector_round_trip(store):
|
|
assert store.upsert("memory", [record("a", [1.0, 0.0])]) == 1
|
|
hits = store.search(["memory"], [1.0, 0.0], limit=5, kinds=["memory"])
|
|
assert hits[0].metadata["content_hash"] == "a"
|
|
```
|
|
|
|
- [ ] **Step 2: Verify failure**
|
|
|
|
Run: `cd harness && .venv/bin/pytest tests/l0/test_pgvector_store.py -q`
|
|
Expected: FAIL because `PgVectorStore` is absent.
|
|
|
|
- [ ] **Step 3: Implement parameterized SQL with allowlisted collection names**
|
|
|
|
```python
|
|
def _collection(name: str) -> sql.Identifier:
|
|
if name not in ALLOWED_COLLECTIONS:
|
|
raise VectorStoreError(f"Collection not allowed: {name}")
|
|
return sql.Identifier("vectors", name)
|
|
```
|
|
|
|
Do not interpolate untrusted identifiers; reuse current metadata shapes and cosine distance ordering.
|
|
|
|
- [ ] **Step 4: Run direct and HTTP parity tests**
|
|
|
|
Run: `cd harness && .venv/bin/pytest tests/l0/test_pgvector_store.py tests/test_vector_port_contract.py tests/test_search_similar_kinds.py -q`
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add harness/tht/adapters/vector/pgvector.py harness/tht/adapters/factory.py harness/tht/config.py harness/tests/l0/test_pgvector_store.py
|
|
git commit -m "feat(vector): add direct pgvector adapter"
|
|
```
|
|
|
|
### Task 2: Version schema and roles
|
|
|
|
**Files:**
|
|
- Create: `harness/migrations/vector/001_extensions.sql`
|
|
- Create: `harness/migrations/vector/002_schema_tables.sql`
|
|
- Create: `harness/migrations/vector/003_roles.sql`
|
|
- Create: `harness/tht/cli/vector_migrate_cmd.py`
|
|
- Test: `harness/tests/l0/test_vector_migrations.py`
|
|
|
|
**Interfaces:**
|
|
- Produces: `tht vector migrate`, `tht vector migrate --status --json`.
|
|
|
|
- [ ] **Step 1: Test clean install and idempotent rerun**
|
|
|
|
```python
|
|
def test_migrations_are_idempotent(database_url):
|
|
migrate(database_url)
|
|
migrate(database_url)
|
|
assert migration_status(database_url).pending == []
|
|
```
|
|
|
|
- [ ] **Step 2: Verify failure**
|
|
|
|
Run: `cd harness && .venv/bin/pytest tests/l0/test_vector_migrations.py -q`
|
|
Expected: FAIL.
|
|
|
|
- [ ] **Step 3: Add ordered migrations and least-privilege roles**
|
|
|
|
```sql
|
|
CREATE SCHEMA IF NOT EXISTS vectors;
|
|
CREATE TABLE IF NOT EXISTS vectors.schema (..., embedding vector(768) NOT NULL);
|
|
CREATE TABLE IF NOT EXISTS vectors.memory (..., embedding vector(768) NOT NULL);
|
|
REVOKE ALL ON SCHEMA vectors FROM PUBLIC;
|
|
```
|
|
|
|
Create reader grants for SELECT/search and writer grants for controlled insert/update; credentials are injected at deployment, not stored in SQL files.
|
|
|
|
- [ ] **Step 4: Run migration and existing vector tests**
|
|
|
|
Run: `cd harness && .venv/bin/pytest tests/l0/test_vector_migrations.py tests/l0/test_pgvector_store.py -q`
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add harness/migrations/vector harness/tht/cli/vector_migrate_cmd.py harness/tests/l0/test_vector_migrations.py
|
|
git commit -m "feat(vector): version pgvector schema"
|
|
```
|
|
|
|
### Task 3: Add `local-vector` Compose profile
|
|
|
|
**Files:**
|
|
- Modify: `compose.yaml`
|
|
- Create: `deploy/vector/init/00-bootstrap.sh`
|
|
- Modify: `deploy/env.example`
|
|
- Create: `scripts/local-vector-smoke.sh`
|
|
|
|
**Interfaces:**
|
|
- Produces service `vector-db`, volume `vector_data`, health-gated core dependency in the profile.
|
|
|
|
- [ ] **Step 1: Add failing profile smoke command**
|
|
|
|
Run: `docker compose --profile local-vector config --services`
|
|
Expected: output does not yet contain `vector-db`.
|
|
|
|
- [ ] **Step 2: Define the standard infrastructure service**
|
|
|
|
```yaml
|
|
vector-db:
|
|
image: pgvector/pgvector:pg16
|
|
profiles: ["local-vector"]
|
|
volumes: ["vector_data:/var/lib/postgresql/data"]
|
|
healthcheck:
|
|
test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
|
|
interval: 5s
|
|
timeout: 3s
|
|
retries: 20
|
|
```
|
|
|
|
- [ ] **Step 3: Wire the local resource config and migration job**
|
|
|
|
Add a one-shot `vector-migrate` service using `thothii-core`; it must complete successfully before preprocessing writes.
|
|
|
|
- [ ] **Step 4: Run local vector smoke**
|
|
|
|
Run: `./scripts/local-vector-smoke.sh`
|
|
Expected: migration succeeds, one record is indexed, and search returns it after restarting `core`.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add compose.yaml deploy/vector deploy/env.example scripts/local-vector-smoke.sh
|
|
git commit -m "feat(deploy): add optional local pgvector profile"
|
|
```
|
|
|
|
### Task 4: Add backup, restore, and parity gates
|
|
|
|
**Files:**
|
|
- Create: `scripts/vector-backup.sh`
|
|
- Create: `scripts/vector-restore.sh`
|
|
- Create: `harness/tests/l0/test_vector_adapter_parity.py`
|
|
- Modify: `README.md`
|
|
|
|
**Interfaces:**
|
|
- Produces versioned custom-format dumps and explicit restore into an empty target.
|
|
|
|
- [ ] **Step 1: Add adapter parity scenarios**
|
|
|
|
```python
|
|
@pytest.mark.parametrize("store_fixture", ["direct_store", "http_store"])
|
|
def test_kind_filtered_search_parity(request, store_fixture):
|
|
store = request.getfixturevalue(store_fixture)
|
|
assert normalize(store.search(...)) == EXPECTED_HITS
|
|
```
|
|
|
|
- [ ] **Step 2: Verify parity test exposes any semantic differences**
|
|
|
|
Run: `cd harness && .venv/bin/pytest tests/l0/test_vector_adapter_parity.py -q`
|
|
Expected: FAIL until result ordering/error mapping is aligned.
|
|
|
|
- [ ] **Step 3: Implement backup/restore safety checks**
|
|
|
|
```sh
|
|
pg_dump --format=custom --schema=vectors --file="$OUTPUT" "$DATABASE_URL"
|
|
pg_restore --exit-on-error --clean --if-exists --dbname="$TARGET_DATABASE_URL" "$INPUT"
|
|
```
|
|
|
|
Require explicit target and refuse restore when it equals the active source URL.
|
|
|
|
- [ ] **Step 4: Run parity, backup/restore, and full harness gates**
|
|
|
|
Run: `cd harness && .venv/bin/pytest tests/l0/test_vector_adapter_parity.py tests/l0/test_pgvector_store.py -q`
|
|
Run: `./scripts/local-vector-smoke.sh --backup-restore`
|
|
Run: `cd harness && .venv/bin/pytest -q`
|
|
Expected: PASS.
|
|
|
|
- [ ] **Step 5: Commit**
|
|
|
|
```bash
|
|
git add scripts/vector-backup.sh scripts/vector-restore.sh harness/tests/l0/test_vector_adapter_parity.py README.md
|
|
git commit -m "docs(vector): add local backup restore and parity gate"
|
|
```
|
|
|