ThothII/docs/superpowers/plans/2026-07-11-local-pgvector-profile.md

Optional Local pgvector Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Let server and desktop deployments run an optional persistent pgvector service with behavior equivalent to the HTTP vector adapter.

Architecture: Implement the final direct VectorStore, version vector schema migrations, add a standard pgvector image to Compose, and provide operational backup/restore commands.

Tech Stack: PostgreSQL 16, pgvector, psycopg2/SQLAlchemy, Alembic or ordered SQL migrations, Docker Compose, pytest/testcontainers.

Global Constraints

• Plans 1 and 2 are complete.
• pgvector is an infrastructure image, not a ThothII-owned application image.
• Reader and writer roles are distinct even for local deployments.
• Existing remote HTTP vector behavior remains supported.
• Persistent data must survive application image replacement.

---

Task 1: Implement direct pgvector store behind VectorStore

Files:

• Create: harness/tht/adapters/vector/pgvector.py
• Test: harness/tests/l0/test_pgvector_store.py
• Modify: harness/tht/adapters/factory.py
• Modify: harness/tht/config.py

Interfaces:

• Produces: PgVectorStore(read_config, write_config=None) implementing the Plan 1 port.

• Step 1: Add L0 contract tests for search, kind filters, hashes, and upsert

def test_pgvector_round_trip(store):
    assert store.upsert("memory", [record("a", [1.0, 0.0])]) == 1
    hits = store.search(["memory"], [1.0, 0.0], limit=5, kinds=["memory"])
    assert hits[0].metadata["content_hash"] == "a"

• Step 2: Verify failure

Run: cd harness && .venv/bin/pytest tests/l0/test_pgvector_store.py -q
Expected: FAIL because PgVectorStore is absent.

• Step 3: Implement parameterized SQL with allowlisted collection names

def _collection(name: str) -> sql.Identifier:
    if name not in ALLOWED_COLLECTIONS:
        raise VectorStoreError(f"Collection not allowed: {name}")
    return sql.Identifier("vectors", name)

Do not interpolate untrusted identifiers; reuse current metadata shapes and cosine distance ordering.

• Step 4: Run direct and HTTP parity tests

Run: cd harness && .venv/bin/pytest tests/l0/test_pgvector_store.py tests/test_vector_port_contract.py tests/test_search_similar_kinds.py -q
Expected: PASS.

• Step 5: Commit

git add harness/tht/adapters/vector/pgvector.py harness/tht/adapters/factory.py harness/tht/config.py harness/tests/l0/test_pgvector_store.py
git commit -m "feat(vector): add direct pgvector adapter"

Task 2: Version schema and roles

Files:

• Create: harness/migrations/vector/001_extensions.sql
• Create: harness/migrations/vector/002_schema_tables.sql
• Create: harness/migrations/vector/003_roles.sql
• Create: harness/tht/cli/vector_migrate_cmd.py
• Test: harness/tests/l0/test_vector_migrations.py

Interfaces:

• Produces: tht vector migrate, tht vector migrate --status --json.

• Step 1: Test clean install and idempotent rerun

def test_migrations_are_idempotent(database_url):
    migrate(database_url)
    migrate(database_url)
    assert migration_status(database_url).pending == []

• Step 2: Verify failure

Run: cd harness && .venv/bin/pytest tests/l0/test_vector_migrations.py -q
Expected: FAIL.

• Step 3: Add ordered migrations and least-privilege roles

CREATE SCHEMA IF NOT EXISTS vectors;
CREATE TABLE IF NOT EXISTS vectors.schema (..., embedding vector(768) NOT NULL);
CREATE TABLE IF NOT EXISTS vectors.memory (..., embedding vector(768) NOT NULL);
REVOKE ALL ON SCHEMA vectors FROM PUBLIC;

Create reader grants for SELECT/search and writer grants for controlled insert/update; credentials are injected at deployment, not stored in SQL files.

• Step 4: Run migration and existing vector tests

Run: cd harness && .venv/bin/pytest tests/l0/test_vector_migrations.py tests/l0/test_pgvector_store.py -q
Expected: PASS.

• Step 5: Commit

git add harness/migrations/vector harness/tht/cli/vector_migrate_cmd.py harness/tests/l0/test_vector_migrations.py
git commit -m "feat(vector): version pgvector schema"

Task 3: Add local-vector Compose profile

Files:

• Modify: compose.yaml
• Create: deploy/vector/init/00-bootstrap.sh
• Modify: deploy/env.example
• Create: scripts/local-vector-smoke.sh

Interfaces:

• Produces service vector-db, volume vector_data, health-gated core dependency in the profile.

• Step 1: Add failing profile smoke command

Run: docker compose --profile local-vector config --services
Expected: output does not yet contain vector-db.

• Step 2: Define the standard infrastructure service

vector-db:
  image: pgvector/pgvector:pg16
  profiles: ["local-vector"]
  volumes: ["vector_data:/var/lib/postgresql/data"]
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
    interval: 5s
    timeout: 3s
    retries: 20

• Step 3: Wire the local resource config and migration job

Add a one-shot vector-migrate service using thothii-core; it must complete successfully before preprocessing writes.

• Step 4: Run local vector smoke

Run: ./scripts/local-vector-smoke.sh
Expected: migration succeeds, one record is indexed, and search returns it after restarting core.

• Step 5: Commit

git add compose.yaml deploy/vector deploy/env.example scripts/local-vector-smoke.sh
git commit -m "feat(deploy): add optional local pgvector profile"

Task 4: Add backup, restore, and parity gates

Files:

• Create: scripts/vector-backup.sh
• Create: scripts/vector-restore.sh
• Create: harness/tests/l0/test_vector_adapter_parity.py
• Modify: README.md

Interfaces:

• Produces versioned custom-format dumps and explicit restore into an empty target.

• Step 1: Add adapter parity scenarios

@pytest.mark.parametrize("store_fixture", ["direct_store", "http_store"])
def test_kind_filtered_search_parity(request, store_fixture):
    store = request.getfixturevalue(store_fixture)
    assert normalize(store.search(...)) == EXPECTED_HITS

• Step 2: Verify parity test exposes any semantic differences

Run: cd harness && .venv/bin/pytest tests/l0/test_vector_adapter_parity.py -q
Expected: FAIL until result ordering/error mapping is aligned.

• Step 3: Implement backup/restore safety checks

pg_dump --format=custom --schema=vectors --file="$OUTPUT" "$DATABASE_URL"
pg_restore --exit-on-error --clean --if-exists --dbname="$TARGET_DATABASE_URL" "$INPUT"

Require explicit target and refuse restore when it equals the active source URL.

• Step 4: Run parity, backup/restore, and full harness gates

Run: cd harness && .venv/bin/pytest tests/l0/test_vector_adapter_parity.py tests/l0/test_pgvector_store.py -q
Run: ./scripts/local-vector-smoke.sh --backup-restore
Run: cd harness && .venv/bin/pytest -q
Expected: PASS.

• Step 5: Commit

git add scripts/vector-backup.sh scripts/vector-restore.sh harness/tests/l0/test_vector_adapter_parity.py README.md
git commit -m "docs(vector): add local backup restore and parity gate"