docs(vector): add local backup restore and parity gate
This commit is contained in:
@@ -0,0 +1,64 @@
|
|||||||
|
# Local pgvector Task 4 report
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Implemented adapter parity gates and an operator-safe custom-format backup/restore workflow.
|
||||||
|
|
||||||
|
- Direct and HTTP stores now share validation, configured-dimension rejection, and deterministic
|
||||||
|
similarity ordering with record ID as the tie-break.
|
||||||
|
- The parity fixture exercises identical records through real pgvector and the HTTP RPC contract:
|
||||||
|
kind filtering, ordering, hashes, replacement upserts, invalid collection/kind errors, and query
|
||||||
|
plus write dimensions.
|
||||||
|
- Backup explicitly allowlists the three vector tables and migration ledger, refuses overwrite,
|
||||||
|
writes through a partial file, and uses a custom compressed archive.
|
||||||
|
- Restore requires explicit active-source and target coordinates. It compares PostgreSQL system
|
||||||
|
identifier plus database OID (robust across DNS aliases), refuses the active database, checks for
|
||||||
|
an empty target unless force is explicit, and restores with exit-on-error.
|
||||||
|
- Passwords are accepted only through validated secret files, converted to private temporary
|
||||||
|
`PGPASSFILE`s, and never placed in command arguments or success/error logs.
|
||||||
|
- Role passwords/login identities are deliberately not dumped. The target must have the approved
|
||||||
|
passwordless group roles and pgvector extension reconciled before restore; archived ACLs restore
|
||||||
|
the reader/writer grants.
|
||||||
|
|
||||||
|
## TDD and semantic alignment
|
||||||
|
|
||||||
|
The first parity run exposed the intended HTTP differences: it accepted unknown collections and
|
||||||
|
wrong dimensions. Direct pgvector also had no stable order for equal cosine distance. The adapters
|
||||||
|
were aligned, and the final focused real-pgvector gate passed: **25 passed**.
|
||||||
|
|
||||||
|
The first recovery run caught an incorrect probe username before restore. The second caught an
|
||||||
|
intersection between `pg_dump --schema` and the explicit public ledger table. The third confirmed
|
||||||
|
the archive contents but caught missing target group roles. Each defect was corrected and the
|
||||||
|
complete drill was rerun from a fresh generated project.
|
||||||
|
|
||||||
|
## Live recovery smoke
|
||||||
|
|
||||||
|
`./scripts/local-vector-smoke.sh --backup-restore`: **PASS**.
|
||||||
|
|
||||||
|
- generated/owned source Compose project and source `vector_data`
|
||||||
|
- distinct restore container and distinct named restore volume
|
||||||
|
- migration and role health, secret rotation, restart persistence
|
||||||
|
- real custom backup, then deliberate mutation of the active source record
|
||||||
|
- same-database identity guard evaluated before restore
|
||||||
|
- restore into the separate target only
|
||||||
|
- restored hash equals the pre-mutation backup, proving retrieval parity
|
||||||
|
- migration ledger has all three applied versions
|
||||||
|
- all three restored embedding columns report `vectors.vector(768)`
|
||||||
|
- ownership-checked cleanup; the active operator project/volume is never addressed
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- parity + direct adapter: 25 passed
|
||||||
|
- full harness: 485 passed, 5 deselected
|
||||||
|
- changed Python files: Ruff clean
|
||||||
|
- shell syntax: clean
|
||||||
|
- `git diff --check`: clean
|
||||||
|
- full Ruff: unchanged repository baseline of 34 unrelated pre-existing test-file violations
|
||||||
|
|
||||||
|
## Self-review and operational constraints
|
||||||
|
|
||||||
|
The restore account must be able to read `pg_control_system()` for the robust cluster-identity
|
||||||
|
comparison and create/restore the selected objects. This is intentionally an administrative
|
||||||
|
recovery operation, not a runtime reader/writer action. `--force-nonempty` is explicit but still
|
||||||
|
uses `pg_restore --clean --if-exists`; operators should prefer a new database/volume and validate
|
||||||
|
migration status, health, and known retrieval before endpoint cutover.
|
||||||
@@ -45,6 +45,46 @@ volume afterward. It never targets the fixed `thothii` operator project or its v
|
|||||||
`KEEP_SMOKE_RESOURCES=1` to retain that smoke project's resources for inspection; remove them
|
`KEEP_SMOKE_RESOURCES=1` to retain that smoke project's resources for inspection; remove them
|
||||||
later with `docker compose --project-name "$SMOKE_PROJECT" --profile external down --volumes`.
|
later with `docker compose --project-name "$SMOKE_PROJECT" --profile external down --volumes`.
|
||||||
|
|
||||||
|
## Optional local pgvector and recovery
|
||||||
|
|
||||||
|
Start the persistent local vector profile with `docker compose --profile local-vector up
|
||||||
|
--build --wait`. Its `vector_data` volume is independent of application state. Reader, writer,
|
||||||
|
migrator, and bootstrap credentials remain separate; password files must be mode `0600` and
|
||||||
|
must not be passed as URL arguments.
|
||||||
|
|
||||||
|
Create a versioned PostgreSQL custom-format backup (the filename is operator-controlled, so use
|
||||||
|
an immutable timestamp or release identifier):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
./scripts/vector-backup.sh \
|
||||||
|
--host 127.0.0.1 --port 5432 --database thoth --user thoth_backup \
|
||||||
|
--password-file /secure/thoth/vector-backup-password \
|
||||||
|
--output /secure/backups/thoth-vectors-2026-07-12.dump
|
||||||
|
```
|
||||||
|
|
||||||
|
The dump contains the three allowlisted `vectors` tables, their data and ACLs, plus the
|
||||||
|
`public.tht_vector_migrations` ledger. Login roles and passwords are deliberately not copied:
|
||||||
|
provision/reconcile the approved role names on the target first, and install the `vector`
|
||||||
|
extension in its `vectors` schema. The target must otherwise contain no vector tables or ledger.
|
||||||
|
|
||||||
|
Restore always names both the currently active source and a distinct target. The script compares
|
||||||
|
PostgreSQL cluster identity plus database OID, so host aliases cannot bypass the active-database
|
||||||
|
guard. It refuses a non-empty target unless `--force-nonempty` is explicit:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
./scripts/vector-restore.sh \
|
||||||
|
--active-host vector-db --active-database thoth --active-user thoth_backup \
|
||||||
|
--active-password-file /secure/thoth/vector-active-password \
|
||||||
|
--target-host vector-db-restore --target-database thoth --target-user thoth_restore \
|
||||||
|
--target-password-file /secure/thoth/vector-restore-password \
|
||||||
|
--input /secure/backups/thoth-vectors-2026-07-12.dump
|
||||||
|
```
|
||||||
|
|
||||||
|
After restore, run `tht vector migrate --status --json`, adapter health, and a known retrieval
|
||||||
|
query against the target before changing any deployment endpoint. Never test recovery against the
|
||||||
|
active `vector_data` volume. `./scripts/local-vector-smoke.sh --backup-restore` performs this drill
|
||||||
|
with disposable source and target volumes.
|
||||||
|
|
||||||
## Production trust boundary and secrets
|
## Production trust boundary and secrets
|
||||||
|
|
||||||
ThothII does not implement OIDC. Do not expose its application port directly to a network.
|
ThothII does not implement OIDC. Do not expose its application port directly to a network.
|
||||||
|
|||||||
@@ -0,0 +1,148 @@
|
|||||||
|
import math
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy import create_engine
|
||||||
|
from testcontainers.postgres import PostgresContainer
|
||||||
|
|
||||||
|
from tht.adapters.vector.pgvector import PgVectorStore
|
||||||
|
from tht.adapters.vector.thoth_http import ThothHttpVectorStore
|
||||||
|
from tht.config import DatabaseConfig
|
||||||
|
from tht.ports.vector import VectorRecord, VectorStoreError, VectorWriteRecord
|
||||||
|
|
||||||
|
|
||||||
|
def _write(record_id, kind, embedding, content_hash):
|
||||||
|
return VectorWriteRecord(
|
||||||
|
VectorRecord(
|
||||||
|
id=record_id,
|
||||||
|
kind=kind,
|
||||||
|
ref="fixture",
|
||||||
|
title=record_id,
|
||||||
|
content=f"content {record_id}",
|
||||||
|
metadata={"fixture": True},
|
||||||
|
),
|
||||||
|
embedding,
|
||||||
|
content_hash,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
FIXTURE = [
|
||||||
|
_write("memory:a", "memory", [1.0, 0.0], "hash-a"),
|
||||||
|
_write("memory:b", "memory", [1.0, 0.0], "hash-b"),
|
||||||
|
_write("solved:a", "solved_question", [0.8, 0.2], "hash-solved"),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class FixtureHttpClient:
|
||||||
|
def __init__(self):
|
||||||
|
self.rows = {}
|
||||||
|
|
||||||
|
def list_tables(self):
|
||||||
|
return [{"table_name": "memory", "vector_dimensions": 2}]
|
||||||
|
|
||||||
|
def upsert_records(self, table_name, rows):
|
||||||
|
for row in rows:
|
||||||
|
self.rows[(table_name, row["record_key"])] = row
|
||||||
|
return len(rows)
|
||||||
|
|
||||||
|
def existing_hashes(self, table_name, kinds):
|
||||||
|
return {
|
||||||
|
row["record_key"]: row["content_hash"]
|
||||||
|
for (table, _), row in self.rows.items()
|
||||||
|
if table == table_name and row["kind"] in kinds
|
||||||
|
}
|
||||||
|
|
||||||
|
def search_similar(self, table_name, embedding, limit, kinds=None):
|
||||||
|
def similarity(row):
|
||||||
|
left, right = row["embedding"], embedding
|
||||||
|
return sum(a * b for a, b in zip(left, right)) / (
|
||||||
|
math.sqrt(sum(a * a for a in left))
|
||||||
|
* math.sqrt(sum(b * b for b in right))
|
||||||
|
)
|
||||||
|
|
||||||
|
rows = [
|
||||||
|
{"metadata": row["metadata"], "similarity": similarity(row)}
|
||||||
|
for (table, _), row in self.rows.items()
|
||||||
|
if table == table_name and (not kinds or row["kind"] in kinds)
|
||||||
|
]
|
||||||
|
return sorted(
|
||||||
|
rows,
|
||||||
|
key=lambda row: (-row["similarity"], row["metadata"]["record_key"]),
|
||||||
|
)[:limit]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def direct_store():
|
||||||
|
with PostgresContainer("pgvector/pgvector:pg16") as postgres:
|
||||||
|
config = DatabaseConfig(
|
||||||
|
host=postgres.get_container_host_ip(),
|
||||||
|
port=int(postgres.get_exposed_port(5432)),
|
||||||
|
database=postgres.dbname,
|
||||||
|
schema="vectors",
|
||||||
|
user=postgres.username,
|
||||||
|
password=postgres.password,
|
||||||
|
)
|
||||||
|
engine = create_engine(postgres.get_connection_url())
|
||||||
|
with engine.begin() as connection:
|
||||||
|
connection.exec_driver_sql("CREATE SCHEMA vectors")
|
||||||
|
connection.exec_driver_sql("CREATE EXTENSION vector WITH SCHEMA vectors")
|
||||||
|
connection.exec_driver_sql(
|
||||||
|
"CREATE TABLE vectors.memory ("
|
||||||
|
"id bigserial PRIMARY KEY, record_key text UNIQUE NOT NULL, "
|
||||||
|
"kind text NOT NULL, content_hash text NOT NULL, metadata jsonb NOT NULL, "
|
||||||
|
"embedding vectors.vector(2) NOT NULL, indexed_at timestamptz NOT NULL "
|
||||||
|
"DEFAULT now())"
|
||||||
|
)
|
||||||
|
engine.dispose()
|
||||||
|
reader, writer = config, config
|
||||||
|
store = PgVectorStore(reader, writer, expected_dimension=2)
|
||||||
|
store.upsert("memory", FIXTURE)
|
||||||
|
yield store
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def http_store():
|
||||||
|
client = FixtureHttpClient()
|
||||||
|
store = ThothHttpVectorStore(client, client, expected_dimension=2)
|
||||||
|
store.upsert("memory", FIXTURE)
|
||||||
|
return store
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("store_fixture", ["direct_store", "http_store"])
|
||||||
|
def test_kind_filtered_search_has_identical_order(request, store_fixture):
|
||||||
|
store = request.getfixturevalue(store_fixture)
|
||||||
|
hits = store.search(["memory"], [1.0, 0.0], limit=3, kinds=["memory"])
|
||||||
|
assert [(hit.id, hit.kind, round(hit.similarity, 6)) for hit in hits] == [
|
||||||
|
("memory:a", "memory", 1.0),
|
||||||
|
("memory:b", "memory", 1.0),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("store_fixture", ["direct_store", "http_store"])
|
||||||
|
def test_hash_and_upsert_parity(request, store_fixture):
|
||||||
|
store = request.getfixturevalue(store_fixture)
|
||||||
|
assert store.existing_hashes("memory", ["memory"]) == {
|
||||||
|
"memory:a": "hash-a",
|
||||||
|
"memory:b": "hash-b",
|
||||||
|
}
|
||||||
|
replacement = _write("memory:a", "memory", [0.0, 1.0], "hash-a-2")
|
||||||
|
assert store.upsert("memory", [replacement]) == 1
|
||||||
|
assert store.existing_hashes("memory", ["memory"])["memory:a"] == "hash-a-2"
|
||||||
|
assert store.search(["memory"], [0.0, 1.0], limit=1, kinds=["memory"])[0].id == "memory:a"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("store_fixture", ["direct_store", "http_store"])
|
||||||
|
def test_validation_error_parity(request, store_fixture):
|
||||||
|
store = request.getfixturevalue(store_fixture)
|
||||||
|
with pytest.raises(VectorStoreError, match="Collection not allowed"):
|
||||||
|
store.search(["not_allowed"], [1.0, 0.0], limit=1)
|
||||||
|
with pytest.raises(VectorStoreError, match="Kind not allowed"):
|
||||||
|
store.search(["memory"], [1.0, 0.0], limit=1, kinds=["not_allowed"])
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("store_fixture", ["direct_store", "http_store"])
|
||||||
|
def test_dimension_error_parity(request, store_fixture):
|
||||||
|
store = request.getfixturevalue(store_fixture)
|
||||||
|
with pytest.raises(VectorStoreError, match="Query embedding dimension"):
|
||||||
|
store.search(["memory"], [1.0], limit=1)
|
||||||
|
with pytest.raises(VectorStoreError, match="Embedding dimension"):
|
||||||
|
store.upsert("memory", [_write("bad", "memory", [1.0], "bad")])
|
||||||
@@ -257,7 +257,7 @@ class PgVectorStore:
|
|||||||
where = sql.SQL(" WHERE kind = ANY(%s)") if collection_kinds else sql.SQL("")
|
where = sql.SQL(" WHERE kind = ANY(%s)") if collection_kinds else sql.SQL("")
|
||||||
query = sql.SQL(
|
query = sql.SQL(
|
||||||
"SELECT metadata, 1 - (embedding {} %s::{}) AS similarity "
|
"SELECT metadata, 1 - (embedding {} %s::{}) AS similarity "
|
||||||
"FROM {}{} ORDER BY embedding {} %s::{} LIMIT %s"
|
"FROM {}{} ORDER BY embedding {} %s::{}, record_key LIMIT %s"
|
||||||
).format(
|
).format(
|
||||||
_cosine_operator(self._schema),
|
_cosine_operator(self._schema),
|
||||||
_vector_type(self._schema),
|
_vector_type(self._schema),
|
||||||
@@ -274,7 +274,7 @@ class PgVectorStore:
|
|||||||
hits.extend(hit_from_metadata(row[1], row[0]) for row in cursor.fetchall())
|
hits.extend(hit_from_metadata(row[1], row[0]) for row in cursor.fetchall())
|
||||||
finally:
|
finally:
|
||||||
raw.close()
|
raw.close()
|
||||||
return sorted(hits, key=lambda hit: hit.similarity, reverse=True)[:limit]
|
return sorted(hits, key=lambda hit: (-hit.similarity, hit.id))[:limit]
|
||||||
|
|
||||||
def _require_writer(self) -> Engine:
|
def _require_writer(self) -> Engine:
|
||||||
if self._writer is None:
|
if self._writer is None:
|
||||||
|
|||||||
@@ -5,16 +5,22 @@ from tht.ports.vector import (
|
|||||||
VectorHealth,
|
VectorHealth,
|
||||||
VectorHit,
|
VectorHit,
|
||||||
VectorReadUnavailable,
|
VectorReadUnavailable,
|
||||||
|
VectorStoreError,
|
||||||
VectorWriteRecord,
|
VectorWriteRecord,
|
||||||
VectorWriteUnavailable,
|
VectorWriteUnavailable,
|
||||||
require_positive_limit,
|
require_positive_limit,
|
||||||
)
|
)
|
||||||
from tht.vectorstore.rest_client import VectorRestClient
|
from tht.vectorstore.rest_client import VectorRestClient
|
||||||
from tht.vectorstore.store import hit_from_metadata
|
from tht.vectorstore.store import hit_from_metadata
|
||||||
|
from tht.adapters.vector.pgvector import (
|
||||||
|
_collection,
|
||||||
|
_validate_collection_kinds,
|
||||||
|
_validate_known_kinds,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _merge(hits: list[VectorHit], limit: int) -> list[VectorHit]:
|
def _merge(hits: list[VectorHit], limit: int) -> list[VectorHit]:
|
||||||
return sorted(hits, key=lambda hit: hit.similarity, reverse=True)[:limit]
|
return sorted(hits, key=lambda hit: (-hit.similarity, hit.id))[:limit]
|
||||||
|
|
||||||
|
|
||||||
class ThothHttpVectorStore:
|
class ThothHttpVectorStore:
|
||||||
@@ -89,8 +95,13 @@ class ThothHttpVectorStore:
|
|||||||
require_positive_limit(limit)
|
require_positive_limit(limit)
|
||||||
if self._reader is None:
|
if self._reader is None:
|
||||||
raise VectorReadUnavailable("Vector reader credential is not configured")
|
raise VectorReadUnavailable("Vector reader credential is not configured")
|
||||||
|
if self._expected_dimension is not None and len(embedding) != self._expected_dimension:
|
||||||
|
raise VectorStoreError("Query embedding dimension does not match configured dimension")
|
||||||
|
if kinds:
|
||||||
|
_validate_known_kinds(kinds)
|
||||||
hits: list[VectorHit] = []
|
hits: list[VectorHit] = []
|
||||||
for collection in collections:
|
for collection in collections:
|
||||||
|
_collection("vectors", collection)
|
||||||
rows = self._reader.search_similar(collection, embedding, limit, kinds=kinds)
|
rows = self._reader.search_similar(collection, embedding, limit, kinds=kinds)
|
||||||
hits.extend(
|
hits.extend(
|
||||||
hit_from_metadata(row.get("similarity", 0.0), row.get("metadata"))
|
hit_from_metadata(row.get("similarity", 0.0), row.get("metadata"))
|
||||||
@@ -107,10 +118,20 @@ class ThothHttpVectorStore:
|
|||||||
return self._writer
|
return self._writer
|
||||||
|
|
||||||
def existing_hashes(self, collection: str, kinds: list[str]) -> dict[str, str]:
|
def existing_hashes(self, collection: str, kinds: list[str]) -> dict[str, str]:
|
||||||
|
_collection("vectors", collection)
|
||||||
|
_validate_collection_kinds(collection, kinds)
|
||||||
return self._require_writer().existing_hashes(collection, kinds)
|
return self._require_writer().existing_hashes(collection, kinds)
|
||||||
|
|
||||||
def upsert(self, collection: str, records: list[VectorWriteRecord]) -> int:
|
def upsert(self, collection: str, records: list[VectorWriteRecord]) -> int:
|
||||||
writer = self._require_writer()
|
writer = self._require_writer()
|
||||||
|
_collection("vectors", collection)
|
||||||
|
for record in records:
|
||||||
|
_validate_collection_kinds(collection, [record.record.kind])
|
||||||
|
if (
|
||||||
|
self._expected_dimension is not None
|
||||||
|
and len(record.embedding) != self._expected_dimension
|
||||||
|
):
|
||||||
|
raise VectorStoreError("Embedding dimension does not match configured dimension")
|
||||||
rows = [self._row(record) for record in records]
|
rows = [self._row(record) for record in records]
|
||||||
return writer.upsert_records(collection, rows)
|
return writer.upsert_records(collection, rows)
|
||||||
|
|
||||||
|
|||||||
@@ -5,8 +5,8 @@ cd "$(dirname "$0")/.."
|
|||||||
|
|
||||||
mode=${1:-run}
|
mode=${1:-run}
|
||||||
case "$mode" in
|
case "$mode" in
|
||||||
run|--live-collision-test) ;;
|
run|--live-collision-test|--backup-restore) ;;
|
||||||
*) echo "usage: $0 [--live-collision-test]" >&2; exit 2 ;;
|
*) echo "usage: $0 [--live-collision-test|--backup-restore]" >&2; exit 2 ;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
keep_resources=${KEEP_SMOKE_RESOURCES:-0}
|
keep_resources=${KEEP_SMOKE_RESOURCES:-0}
|
||||||
@@ -19,6 +19,8 @@ suffix=$(basename "$secret_dir" | tr -cd 'a-z0-9')
|
|||||||
smoke_project="thothii-vector-smoke-$(date +%s)-$$-$suffix"
|
smoke_project="thothii-vector-smoke-$(date +%s)-$$-$suffix"
|
||||||
smoke_owner="$smoke_project-owner"
|
smoke_owner="$smoke_project-owner"
|
||||||
marker="local-vector-$smoke_project"
|
marker="local-vector-$smoke_project"
|
||||||
|
restore_container="${smoke_project}-restore"
|
||||||
|
restore_volume="${smoke_project}-restore-data"
|
||||||
|
|
||||||
for secret in bootstrap migrator reader writer; do
|
for secret in bootstrap migrator reader writer; do
|
||||||
password="smoke-${secret}-${smoke_project}"
|
password="smoke-${secret}-${smoke_project}"
|
||||||
@@ -82,6 +84,8 @@ cleanup() {
|
|||||||
echo "Keeping smoke resources for project $smoke_project (KEEP_SMOKE_RESOURCES=1)." >&2
|
echo "Keeping smoke resources for project $smoke_project (KEEP_SMOKE_RESOURCES=1)." >&2
|
||||||
else
|
else
|
||||||
if verify_owned_resources; then
|
if verify_owned_resources; then
|
||||||
|
docker rm -f "$restore_container" >/dev/null 2>&1 || true
|
||||||
|
docker volume rm "$restore_volume" >/dev/null 2>&1 || true
|
||||||
compose down --volumes >/dev/null 2>&1 || true
|
compose down --volumes >/dev/null 2>&1 || true
|
||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
@@ -249,4 +253,70 @@ compose restart vector-db core
|
|||||||
compose up --wait vector-db core
|
compose up --wait vector-db core
|
||||||
probe_vector read
|
probe_vector read
|
||||||
|
|
||||||
|
if [ "$mode" = "--backup-restore" ]; then
|
||||||
|
image=$(compose images -q vector-db)
|
||||||
|
network="${smoke_project}_default"
|
||||||
|
docker volume create \
|
||||||
|
--label "com.docker.compose.project=$smoke_project" \
|
||||||
|
--label "io.thothii.smoke-owner=$smoke_owner" "$restore_volume" >/dev/null
|
||||||
|
docker run -d --name "$restore_container" \
|
||||||
|
--label "com.docker.compose.project=$smoke_project" \
|
||||||
|
--label "io.thothii.smoke-owner=$smoke_owner" \
|
||||||
|
--network "$network" --network-alias vector-db-restore \
|
||||||
|
--mount "type=volume,source=$restore_volume,target=/var/lib/postgresql/data" \
|
||||||
|
--mount "type=bind,source=$secret_dir/bootstrap,target=/run/secrets/bootstrap,readonly" \
|
||||||
|
-e POSTGRES_DB=thoth -e POSTGRES_USER="$THT_VECTOR_BOOTSTRAP_USER" \
|
||||||
|
-e POSTGRES_PASSWORD_FILE=/run/secrets/bootstrap "$image" >/dev/null
|
||||||
|
attempts=0
|
||||||
|
until docker exec "$restore_container" pg_isready \
|
||||||
|
-U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth >/dev/null 2>&1; do
|
||||||
|
attempts=$((attempts + 1))
|
||||||
|
[ "$attempts" -lt 30 ] || { echo "restore database did not become ready" >&2; exit 1; }
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql -X \
|
||||||
|
-U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth -v ON_ERROR_STOP=1 --command \
|
||||||
|
"CREATE SCHEMA vectors; CREATE EXTENSION vector WITH SCHEMA vectors;
|
||||||
|
CREATE ROLE vector_reader NOLOGIN; CREATE ROLE vector_writer NOLOGIN;" >/dev/null
|
||||||
|
|
||||||
|
docker run --rm --network "$network" \
|
||||||
|
--mount "type=bind,source=$(pwd),target=/repo,readonly" \
|
||||||
|
--mount "type=bind,source=$secret_dir,target=/scratch" "$image" \
|
||||||
|
/repo/scripts/vector-backup.sh --host vector-db --database thoth \
|
||||||
|
--user "$THT_VECTOR_BOOTSTRAP_USER" --password-file /scratch/bootstrap \
|
||||||
|
--output /scratch/vector.dump
|
||||||
|
|
||||||
|
compose exec -T vector-db sh -ec '
|
||||||
|
export PGPASSWORD=$(cat /run/secrets/vector_bootstrap_password)
|
||||||
|
psql -X -U "$POSTGRES_USER" -d thoth -v ON_ERROR_STOP=1 --command \
|
||||||
|
"UPDATE vectors.memory SET content_hash = '\''mutated-after-backup'\'' WHERE record_key = '\''$1'\''"' \
|
||||||
|
sh "$marker" >/dev/null
|
||||||
|
|
||||||
|
docker run --rm --network "$network" \
|
||||||
|
--mount "type=bind,source=$(pwd),target=/repo,readonly" \
|
||||||
|
--mount "type=bind,source=$secret_dir,target=/scratch" "$image" \
|
||||||
|
/repo/scripts/vector-restore.sh \
|
||||||
|
--active-host vector-db --active-database thoth --active-user "$THT_VECTOR_BOOTSTRAP_USER" \
|
||||||
|
--active-password-file /scratch/bootstrap \
|
||||||
|
--target-host vector-db-restore --target-database thoth \
|
||||||
|
--target-user "$THT_VECTOR_BOOTSTRAP_USER" --target-password-file /scratch/bootstrap \
|
||||||
|
--input /scratch/vector.dump
|
||||||
|
|
||||||
|
restored=$(docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql \
|
||||||
|
-XAt -U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth --command \
|
||||||
|
"SELECT content_hash <> 'mutated-after-backup' FROM vectors.memory WHERE record_key = '$marker'")
|
||||||
|
test "$restored" = t
|
||||||
|
pending=$(docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql \
|
||||||
|
-XAt -U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth --command \
|
||||||
|
"SELECT count(*) = 3 FROM public.tht_vector_migrations")
|
||||||
|
test "$pending" = t
|
||||||
|
dimensions=$(docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql \
|
||||||
|
-XAt -U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth --command \
|
||||||
|
"SELECT count(*) = 3 FROM pg_attribute a JOIN pg_class c ON c.oid=a.attrelid
|
||||||
|
JOIN pg_namespace n ON n.oid=c.relnamespace
|
||||||
|
WHERE n.nspname='vectors' AND a.attname='embedding' AND format_type(a.atttypid,a.atttypmod)='vectors.vector(768)'")
|
||||||
|
test "$dimensions" = t
|
||||||
|
echo "Disposable-volume backup, mutation, restore, ledger, health, and retrieval parity passed."
|
||||||
|
fi
|
||||||
|
|
||||||
echo "Local pgvector runtime/bootstrap rotation, least-privilege roles, and persistence passed."
|
echo "Local pgvector runtime/bootstrap rotation, least-privilege roles, and persistence passed."
|
||||||
|
|||||||
Executable
+45
@@ -0,0 +1,45 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
root=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
|
||||||
|
. "$root/deploy/vector/secret-policy.sh"
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
echo "usage: $0 --host HOST --database DB --user USER --password-file FILE --output FILE [--port PORT]" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
host= database= user= password_file= output= port=5432
|
||||||
|
while [ "$#" -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--host) host=${2-}; shift 2 ;;
|
||||||
|
--port) port=${2-}; shift 2 ;;
|
||||||
|
--database) database=${2-}; shift 2 ;;
|
||||||
|
--user) user=${2-}; shift 2 ;;
|
||||||
|
--password-file) password_file=${2-}; shift 2 ;;
|
||||||
|
--output) output=${2-}; shift 2 ;;
|
||||||
|
*) usage ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
[ -n "$host" ] && [ -n "$database" ] && [ -n "$user" ] || usage
|
||||||
|
[ -n "$password_file" ] && [ -n "$output" ] || usage
|
||||||
|
validate_secret_file "$password_file" "backup password file"
|
||||||
|
[ ! -e "$output" ] || { echo "refusing to overwrite existing backup: $output" >&2; exit 2; }
|
||||||
|
|
||||||
|
password=$(read_secret_file "$password_file" "backup password file")
|
||||||
|
|
||||||
|
umask 077
|
||||||
|
passfile=$(mktemp "${TMPDIR:-/tmp}/thoth-vector-pgpass.XXXXXX")
|
||||||
|
cleanup() { rm -f "$passfile" "$output.partial"; }
|
||||||
|
trap cleanup EXIT HUP INT TERM
|
||||||
|
escaped=$(printf '%s' "$password" | sed 's/\\/\\\\/g; s/:/\\:/g')
|
||||||
|
printf '%s:%s:%s:%s:%s\n' "$host" "$port" "$database" "$user" "$escaped" >"$passfile"
|
||||||
|
chmod 0600 "$passfile"
|
||||||
|
|
||||||
|
PGPASSFILE=$passfile pg_dump \
|
||||||
|
--host="$host" --port="$port" --username="$user" --dbname="$database" \
|
||||||
|
--format=custom --compress=9 \
|
||||||
|
--table=vectors.schema_records --table=vectors.evidence --table=vectors.memory \
|
||||||
|
--table=public.tht_vector_migrations --file="$output.partial"
|
||||||
|
mv "$output.partial" "$output"
|
||||||
|
echo "Vector backup written: $output"
|
||||||
Executable
+78
@@ -0,0 +1,78 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
root=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
|
||||||
|
. "$root/deploy/vector/secret-policy.sh"
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
echo "usage: $0 --active-host HOST --active-database DB --active-user USER --active-password-file FILE --target-host HOST --target-database DB --target-user USER --target-password-file FILE --input FILE [--active-port PORT] [--target-port PORT] [--force-nonempty]" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
active_host= active_database= active_user= active_password_file= active_port=5432
|
||||||
|
target_host= target_database= target_user= target_password_file= target_port=5432
|
||||||
|
input= force=0
|
||||||
|
while [ "$#" -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
--active-host) active_host=${2-}; shift 2 ;;
|
||||||
|
--active-port) active_port=${2-}; shift 2 ;;
|
||||||
|
--active-database) active_database=${2-}; shift 2 ;;
|
||||||
|
--active-user) active_user=${2-}; shift 2 ;;
|
||||||
|
--active-password-file) active_password_file=${2-}; shift 2 ;;
|
||||||
|
--target-host) target_host=${2-}; shift 2 ;;
|
||||||
|
--target-port) target_port=${2-}; shift 2 ;;
|
||||||
|
--target-database) target_database=${2-}; shift 2 ;;
|
||||||
|
--target-user) target_user=${2-}; shift 2 ;;
|
||||||
|
--target-password-file) target_password_file=${2-}; shift 2 ;;
|
||||||
|
--input) input=${2-}; shift 2 ;;
|
||||||
|
--force-nonempty) force=1; shift ;;
|
||||||
|
*) usage ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
for value in "$active_host" "$active_database" "$active_user" "$active_password_file" \
|
||||||
|
"$target_host" "$target_database" "$target_user" "$target_password_file" "$input"; do
|
||||||
|
[ -n "$value" ] || usage
|
||||||
|
done
|
||||||
|
[ -r "$input" ] || { echo "backup input is not readable" >&2; exit 2; }
|
||||||
|
validate_secret_file "$active_password_file" "active source password file"
|
||||||
|
validate_secret_file "$target_password_file" "target password file"
|
||||||
|
|
||||||
|
umask 077
|
||||||
|
active_pass=$(mktemp "${TMPDIR:-/tmp}/thoth-vector-active-pgpass.XXXXXX")
|
||||||
|
target_pass=$(mktemp "${TMPDIR:-/tmp}/thoth-vector-target-pgpass.XXXXXX")
|
||||||
|
cleanup() { rm -f "$active_pass" "$target_pass"; }
|
||||||
|
trap cleanup EXIT HUP INT TERM
|
||||||
|
make_passfile() {
|
||||||
|
secret=$(read_secret_file "$5" "database password file")
|
||||||
|
escaped=$(printf '%s' "$secret" | sed 's/\\/\\\\/g; s/:/\\:/g')
|
||||||
|
printf '%s:%s:%s:%s:%s\n' "$1" "$2" "$3" "$4" "$escaped" >"$6"
|
||||||
|
chmod 0600 "$6"
|
||||||
|
}
|
||||||
|
make_passfile "$active_host" "$active_port" "$active_database" "$active_user" \
|
||||||
|
"$active_password_file" "$active_pass"
|
||||||
|
make_passfile "$target_host" "$target_port" "$target_database" "$target_user" \
|
||||||
|
"$target_password_file" "$target_pass"
|
||||||
|
|
||||||
|
identity_sql="SELECT system_identifier::text || ':' || d.oid::text FROM pg_control_system(), pg_database d WHERE d.datname = current_database()"
|
||||||
|
active_identity=$(PGPASSFILE=$active_pass psql -XAt --host="$active_host" --port="$active_port" \
|
||||||
|
--username="$active_user" --dbname="$active_database" --command="$identity_sql")
|
||||||
|
target_identity=$(PGPASSFILE=$target_pass psql -XAt --host="$target_host" --port="$target_port" \
|
||||||
|
--username="$target_user" --dbname="$target_database" --command="$identity_sql")
|
||||||
|
[ "$active_identity" != "$target_identity" ] || {
|
||||||
|
echo "refusing restore: active source and target are the same database" >&2; exit 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
object_count=$(PGPASSFILE=$target_pass psql -XAt --host="$target_host" --port="$target_port" \
|
||||||
|
--username="$target_user" --dbname="$target_database" --command="
|
||||||
|
SELECT count(*) FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace
|
||||||
|
WHERE (n.nspname='vectors' OR (n.nspname='public' AND c.relname='tht_vector_migrations'))
|
||||||
|
AND c.relkind IN ('r','p','S','v','m');")
|
||||||
|
if [ "$object_count" != 0 ] && [ "$force" != 1 ]; then
|
||||||
|
echo "refusing restore into non-empty target; use --force-nonempty explicitly" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
PGPASSFILE=$target_pass pg_restore --exit-on-error --clean --if-exists --no-owner \
|
||||||
|
--host="$target_host" --port="$target_port" --username="$target_user" \
|
||||||
|
--dbname="$target_database" "$input"
|
||||||
|
echo "Vector restore completed into explicit target $target_host:$target_port/$target_database"
|
||||||
Reference in New Issue
Block a user