docs(vector): add local backup restore and parity gate

This commit is contained in:
2026-07-12 02:18:29 +02:00
parent 1145ae20bc
commit e4db2ea5e1
8 changed files with 471 additions and 5 deletions
@@ -0,0 +1,64 @@
# Local pgvector Task 4 report
## Outcome
Implemented adapter parity gates and an operator-safe custom-format backup/restore workflow.
- Direct and HTTP stores now share validation, configured-dimension rejection, and deterministic
similarity ordering with record ID as the tie-break.
- The parity fixture exercises identical records through real pgvector and the HTTP RPC contract:
kind filtering, ordering, hashes, replacement upserts, invalid collection/kind errors, and query
plus write dimensions.
- Backup explicitly allowlists the three vector tables and migration ledger, refuses overwrite,
writes through a partial file, and uses a custom compressed archive.
- Restore requires explicit active-source and target coordinates. It compares PostgreSQL system
identifier plus database OID (robust across DNS aliases), refuses the active database, checks for
an empty target unless force is explicit, and restores with exit-on-error.
- Passwords are accepted only through validated secret files, converted to private temporary
`PGPASSFILE`s, and never placed in command arguments or success/error logs.
- Role passwords/login identities are deliberately not dumped. The target must have the approved
passwordless group roles and pgvector extension reconciled before restore; archived ACLs restore
the reader/writer grants.
## TDD and semantic alignment
The first parity run exposed the intended HTTP differences: it accepted unknown collections and
wrong dimensions. Direct pgvector also had no stable order for equal cosine distance. The adapters
were aligned, and the final focused real-pgvector gate passed: **25 passed**.
The first recovery run caught an incorrect probe username before restore. The second caught an
intersection between `pg_dump --schema` and the explicit public ledger table. The third confirmed
the archive contents but caught missing target group roles. Each defect was corrected and the
complete drill was rerun from a fresh generated project.
## Live recovery smoke
`./scripts/local-vector-smoke.sh --backup-restore`: **PASS**.
- generated/owned source Compose project and source `vector_data`
- distinct restore container and distinct named restore volume
- migration and role health, secret rotation, restart persistence
- real custom backup, then deliberate mutation of the active source record
- same-database identity guard evaluated before restore
- restore into the separate target only
- restored hash equals the pre-mutation backup, proving retrieval parity
- migration ledger has all three applied versions
- all three restored embedding columns report `vectors.vector(768)`
- ownership-checked cleanup; the active operator project/volume is never addressed
## Verification
- parity + direct adapter: 25 passed
- full harness: 485 passed, 5 deselected
- changed Python files: Ruff clean
- shell syntax: clean
- `git diff --check`: clean
- full Ruff: unchanged repository baseline of 34 unrelated pre-existing test-file violations
## Self-review and operational constraints
The restore account must be able to read `pg_control_system()` for the robust cluster-identity
comparison and create/restore the selected objects. This is intentionally an administrative
recovery operation, not a runtime reader/writer action. `--force-nonempty` is explicit but still
uses `pg_restore --clean --if-exists`; operators should prefer a new database/volume and validate
migration status, health, and known retrieval before endpoint cutover.
+40
View File
@@ -45,6 +45,46 @@ volume afterward. It never targets the fixed `thothii` operator project or its v
`KEEP_SMOKE_RESOURCES=1` to retain that smoke project's resources for inspection; remove them `KEEP_SMOKE_RESOURCES=1` to retain that smoke project's resources for inspection; remove them
later with `docker compose --project-name "$SMOKE_PROJECT" --profile external down --volumes`. later with `docker compose --project-name "$SMOKE_PROJECT" --profile external down --volumes`.
## Optional local pgvector and recovery
Start the persistent local vector profile with `docker compose --profile local-vector up
--build --wait`. Its `vector_data` volume is independent of application state. Reader, writer,
migrator, and bootstrap credentials remain separate; password files must be mode `0600` and
must not be passed as URL arguments.
Create a versioned PostgreSQL custom-format backup (the filename is operator-controlled, so use
an immutable timestamp or release identifier):
```sh
./scripts/vector-backup.sh \
--host 127.0.0.1 --port 5432 --database thoth --user thoth_backup \
--password-file /secure/thoth/vector-backup-password \
--output /secure/backups/thoth-vectors-2026-07-12.dump
```
The dump contains the three allowlisted `vectors` tables, their data and ACLs, plus the
`public.tht_vector_migrations` ledger. Login roles and passwords are deliberately not copied:
provision/reconcile the approved role names on the target first, and install the `vector`
extension in its `vectors` schema. The target must otherwise contain no vector tables or ledger.
Restore always names both the currently active source and a distinct target. The script compares
PostgreSQL cluster identity plus database OID, so host aliases cannot bypass the active-database
guard. It refuses a non-empty target unless `--force-nonempty` is explicit:
```sh
./scripts/vector-restore.sh \
--active-host vector-db --active-database thoth --active-user thoth_backup \
--active-password-file /secure/thoth/vector-active-password \
--target-host vector-db-restore --target-database thoth --target-user thoth_restore \
--target-password-file /secure/thoth/vector-restore-password \
--input /secure/backups/thoth-vectors-2026-07-12.dump
```
After restore, run `tht vector migrate --status --json`, adapter health, and a known retrieval
query against the target before changing any deployment endpoint. Never test recovery against the
active `vector_data` volume. `./scripts/local-vector-smoke.sh --backup-restore` performs this drill
with disposable source and target volumes.
## Production trust boundary and secrets ## Production trust boundary and secrets
ThothII does not implement OIDC. Do not expose its application port directly to a network. ThothII does not implement OIDC. Do not expose its application port directly to a network.
@@ -0,0 +1,148 @@
import math
import pytest
from sqlalchemy import create_engine
from testcontainers.postgres import PostgresContainer
from tht.adapters.vector.pgvector import PgVectorStore
from tht.adapters.vector.thoth_http import ThothHttpVectorStore
from tht.config import DatabaseConfig
from tht.ports.vector import VectorRecord, VectorStoreError, VectorWriteRecord
def _write(record_id, kind, embedding, content_hash):
return VectorWriteRecord(
VectorRecord(
id=record_id,
kind=kind,
ref="fixture",
title=record_id,
content=f"content {record_id}",
metadata={"fixture": True},
),
embedding,
content_hash,
)
FIXTURE = [
_write("memory:a", "memory", [1.0, 0.0], "hash-a"),
_write("memory:b", "memory", [1.0, 0.0], "hash-b"),
_write("solved:a", "solved_question", [0.8, 0.2], "hash-solved"),
]
class FixtureHttpClient:
def __init__(self):
self.rows = {}
def list_tables(self):
return [{"table_name": "memory", "vector_dimensions": 2}]
def upsert_records(self, table_name, rows):
for row in rows:
self.rows[(table_name, row["record_key"])] = row
return len(rows)
def existing_hashes(self, table_name, kinds):
return {
row["record_key"]: row["content_hash"]
for (table, _), row in self.rows.items()
if table == table_name and row["kind"] in kinds
}
def search_similar(self, table_name, embedding, limit, kinds=None):
def similarity(row):
left, right = row["embedding"], embedding
return sum(a * b for a, b in zip(left, right)) / (
math.sqrt(sum(a * a for a in left))
* math.sqrt(sum(b * b for b in right))
)
rows = [
{"metadata": row["metadata"], "similarity": similarity(row)}
for (table, _), row in self.rows.items()
if table == table_name and (not kinds or row["kind"] in kinds)
]
return sorted(
rows,
key=lambda row: (-row["similarity"], row["metadata"]["record_key"]),
)[:limit]
@pytest.fixture
def direct_store():
with PostgresContainer("pgvector/pgvector:pg16") as postgres:
config = DatabaseConfig(
host=postgres.get_container_host_ip(),
port=int(postgres.get_exposed_port(5432)),
database=postgres.dbname,
schema="vectors",
user=postgres.username,
password=postgres.password,
)
engine = create_engine(postgres.get_connection_url())
with engine.begin() as connection:
connection.exec_driver_sql("CREATE SCHEMA vectors")
connection.exec_driver_sql("CREATE EXTENSION vector WITH SCHEMA vectors")
connection.exec_driver_sql(
"CREATE TABLE vectors.memory ("
"id bigserial PRIMARY KEY, record_key text UNIQUE NOT NULL, "
"kind text NOT NULL, content_hash text NOT NULL, metadata jsonb NOT NULL, "
"embedding vectors.vector(2) NOT NULL, indexed_at timestamptz NOT NULL "
"DEFAULT now())"
)
engine.dispose()
reader, writer = config, config
store = PgVectorStore(reader, writer, expected_dimension=2)
store.upsert("memory", FIXTURE)
yield store
@pytest.fixture
def http_store():
client = FixtureHttpClient()
store = ThothHttpVectorStore(client, client, expected_dimension=2)
store.upsert("memory", FIXTURE)
return store
@pytest.mark.parametrize("store_fixture", ["direct_store", "http_store"])
def test_kind_filtered_search_has_identical_order(request, store_fixture):
store = request.getfixturevalue(store_fixture)
hits = store.search(["memory"], [1.0, 0.0], limit=3, kinds=["memory"])
assert [(hit.id, hit.kind, round(hit.similarity, 6)) for hit in hits] == [
("memory:a", "memory", 1.0),
("memory:b", "memory", 1.0),
]
@pytest.mark.parametrize("store_fixture", ["direct_store", "http_store"])
def test_hash_and_upsert_parity(request, store_fixture):
store = request.getfixturevalue(store_fixture)
assert store.existing_hashes("memory", ["memory"]) == {
"memory:a": "hash-a",
"memory:b": "hash-b",
}
replacement = _write("memory:a", "memory", [0.0, 1.0], "hash-a-2")
assert store.upsert("memory", [replacement]) == 1
assert store.existing_hashes("memory", ["memory"])["memory:a"] == "hash-a-2"
assert store.search(["memory"], [0.0, 1.0], limit=1, kinds=["memory"])[0].id == "memory:a"
@pytest.mark.parametrize("store_fixture", ["direct_store", "http_store"])
def test_validation_error_parity(request, store_fixture):
store = request.getfixturevalue(store_fixture)
with pytest.raises(VectorStoreError, match="Collection not allowed"):
store.search(["not_allowed"], [1.0, 0.0], limit=1)
with pytest.raises(VectorStoreError, match="Kind not allowed"):
store.search(["memory"], [1.0, 0.0], limit=1, kinds=["not_allowed"])
@pytest.mark.parametrize("store_fixture", ["direct_store", "http_store"])
def test_dimension_error_parity(request, store_fixture):
store = request.getfixturevalue(store_fixture)
with pytest.raises(VectorStoreError, match="Query embedding dimension"):
store.search(["memory"], [1.0], limit=1)
with pytest.raises(VectorStoreError, match="Embedding dimension"):
store.upsert("memory", [_write("bad", "memory", [1.0], "bad")])
+2 -2
View File
@@ -257,7 +257,7 @@ class PgVectorStore:
where = sql.SQL(" WHERE kind = ANY(%s)") if collection_kinds else sql.SQL("") where = sql.SQL(" WHERE kind = ANY(%s)") if collection_kinds else sql.SQL("")
query = sql.SQL( query = sql.SQL(
"SELECT metadata, 1 - (embedding {} %s::{}) AS similarity " "SELECT metadata, 1 - (embedding {} %s::{}) AS similarity "
"FROM {}{} ORDER BY embedding {} %s::{} LIMIT %s" "FROM {}{} ORDER BY embedding {} %s::{}, record_key LIMIT %s"
).format( ).format(
_cosine_operator(self._schema), _cosine_operator(self._schema),
_vector_type(self._schema), _vector_type(self._schema),
@@ -274,7 +274,7 @@ class PgVectorStore:
hits.extend(hit_from_metadata(row[1], row[0]) for row in cursor.fetchall()) hits.extend(hit_from_metadata(row[1], row[0]) for row in cursor.fetchall())
finally: finally:
raw.close() raw.close()
return sorted(hits, key=lambda hit: hit.similarity, reverse=True)[:limit] return sorted(hits, key=lambda hit: (-hit.similarity, hit.id))[:limit]
def _require_writer(self) -> Engine: def _require_writer(self) -> Engine:
if self._writer is None: if self._writer is None:
+22 -1
View File
@@ -5,16 +5,22 @@ from tht.ports.vector import (
VectorHealth, VectorHealth,
VectorHit, VectorHit,
VectorReadUnavailable, VectorReadUnavailable,
VectorStoreError,
VectorWriteRecord, VectorWriteRecord,
VectorWriteUnavailable, VectorWriteUnavailable,
require_positive_limit, require_positive_limit,
) )
from tht.vectorstore.rest_client import VectorRestClient from tht.vectorstore.rest_client import VectorRestClient
from tht.vectorstore.store import hit_from_metadata from tht.vectorstore.store import hit_from_metadata
from tht.adapters.vector.pgvector import (
_collection,
_validate_collection_kinds,
_validate_known_kinds,
)
def _merge(hits: list[VectorHit], limit: int) -> list[VectorHit]: def _merge(hits: list[VectorHit], limit: int) -> list[VectorHit]:
return sorted(hits, key=lambda hit: hit.similarity, reverse=True)[:limit] return sorted(hits, key=lambda hit: (-hit.similarity, hit.id))[:limit]
class ThothHttpVectorStore: class ThothHttpVectorStore:
@@ -89,8 +95,13 @@ class ThothHttpVectorStore:
require_positive_limit(limit) require_positive_limit(limit)
if self._reader is None: if self._reader is None:
raise VectorReadUnavailable("Vector reader credential is not configured") raise VectorReadUnavailable("Vector reader credential is not configured")
if self._expected_dimension is not None and len(embedding) != self._expected_dimension:
raise VectorStoreError("Query embedding dimension does not match configured dimension")
if kinds:
_validate_known_kinds(kinds)
hits: list[VectorHit] = [] hits: list[VectorHit] = []
for collection in collections: for collection in collections:
_collection("vectors", collection)
rows = self._reader.search_similar(collection, embedding, limit, kinds=kinds) rows = self._reader.search_similar(collection, embedding, limit, kinds=kinds)
hits.extend( hits.extend(
hit_from_metadata(row.get("similarity", 0.0), row.get("metadata")) hit_from_metadata(row.get("similarity", 0.0), row.get("metadata"))
@@ -107,10 +118,20 @@ class ThothHttpVectorStore:
return self._writer return self._writer
def existing_hashes(self, collection: str, kinds: list[str]) -> dict[str, str]: def existing_hashes(self, collection: str, kinds: list[str]) -> dict[str, str]:
_collection("vectors", collection)
_validate_collection_kinds(collection, kinds)
return self._require_writer().existing_hashes(collection, kinds) return self._require_writer().existing_hashes(collection, kinds)
def upsert(self, collection: str, records: list[VectorWriteRecord]) -> int: def upsert(self, collection: str, records: list[VectorWriteRecord]) -> int:
writer = self._require_writer() writer = self._require_writer()
_collection("vectors", collection)
for record in records:
_validate_collection_kinds(collection, [record.record.kind])
if (
self._expected_dimension is not None
and len(record.embedding) != self._expected_dimension
):
raise VectorStoreError("Embedding dimension does not match configured dimension")
rows = [self._row(record) for record in records] rows = [self._row(record) for record in records]
return writer.upsert_records(collection, rows) return writer.upsert_records(collection, rows)
+72 -2
View File
@@ -5,8 +5,8 @@ cd "$(dirname "$0")/.."
mode=${1:-run} mode=${1:-run}
case "$mode" in case "$mode" in
run|--live-collision-test) ;; run|--live-collision-test|--backup-restore) ;;
*) echo "usage: $0 [--live-collision-test]" >&2; exit 2 ;; *) echo "usage: $0 [--live-collision-test|--backup-restore]" >&2; exit 2 ;;
esac esac
keep_resources=${KEEP_SMOKE_RESOURCES:-0} keep_resources=${KEEP_SMOKE_RESOURCES:-0}
@@ -19,6 +19,8 @@ suffix=$(basename "$secret_dir" | tr -cd 'a-z0-9')
smoke_project="thothii-vector-smoke-$(date +%s)-$$-$suffix" smoke_project="thothii-vector-smoke-$(date +%s)-$$-$suffix"
smoke_owner="$smoke_project-owner" smoke_owner="$smoke_project-owner"
marker="local-vector-$smoke_project" marker="local-vector-$smoke_project"
restore_container="${smoke_project}-restore"
restore_volume="${smoke_project}-restore-data"
for secret in bootstrap migrator reader writer; do for secret in bootstrap migrator reader writer; do
password="smoke-${secret}-${smoke_project}" password="smoke-${secret}-${smoke_project}"
@@ -82,6 +84,8 @@ cleanup() {
echo "Keeping smoke resources for project $smoke_project (KEEP_SMOKE_RESOURCES=1)." >&2 echo "Keeping smoke resources for project $smoke_project (KEEP_SMOKE_RESOURCES=1)." >&2
else else
if verify_owned_resources; then if verify_owned_resources; then
docker rm -f "$restore_container" >/dev/null 2>&1 || true
docker volume rm "$restore_volume" >/dev/null 2>&1 || true
compose down --volumes >/dev/null 2>&1 || true compose down --volumes >/dev/null 2>&1 || true
fi fi
fi fi
@@ -249,4 +253,70 @@ compose restart vector-db core
compose up --wait vector-db core compose up --wait vector-db core
probe_vector read probe_vector read
if [ "$mode" = "--backup-restore" ]; then
image=$(compose images -q vector-db)
network="${smoke_project}_default"
docker volume create \
--label "com.docker.compose.project=$smoke_project" \
--label "io.thothii.smoke-owner=$smoke_owner" "$restore_volume" >/dev/null
docker run -d --name "$restore_container" \
--label "com.docker.compose.project=$smoke_project" \
--label "io.thothii.smoke-owner=$smoke_owner" \
--network "$network" --network-alias vector-db-restore \
--mount "type=volume,source=$restore_volume,target=/var/lib/postgresql/data" \
--mount "type=bind,source=$secret_dir/bootstrap,target=/run/secrets/bootstrap,readonly" \
-e POSTGRES_DB=thoth -e POSTGRES_USER="$THT_VECTOR_BOOTSTRAP_USER" \
-e POSTGRES_PASSWORD_FILE=/run/secrets/bootstrap "$image" >/dev/null
attempts=0
until docker exec "$restore_container" pg_isready \
-U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth >/dev/null 2>&1; do
attempts=$((attempts + 1))
[ "$attempts" -lt 30 ] || { echo "restore database did not become ready" >&2; exit 1; }
sleep 1
done
docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql -X \
-U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth -v ON_ERROR_STOP=1 --command \
"CREATE SCHEMA vectors; CREATE EXTENSION vector WITH SCHEMA vectors;
CREATE ROLE vector_reader NOLOGIN; CREATE ROLE vector_writer NOLOGIN;" >/dev/null
docker run --rm --network "$network" \
--mount "type=bind,source=$(pwd),target=/repo,readonly" \
--mount "type=bind,source=$secret_dir,target=/scratch" "$image" \
/repo/scripts/vector-backup.sh --host vector-db --database thoth \
--user "$THT_VECTOR_BOOTSTRAP_USER" --password-file /scratch/bootstrap \
--output /scratch/vector.dump
compose exec -T vector-db sh -ec '
export PGPASSWORD=$(cat /run/secrets/vector_bootstrap_password)
psql -X -U "$POSTGRES_USER" -d thoth -v ON_ERROR_STOP=1 --command \
"UPDATE vectors.memory SET content_hash = '\''mutated-after-backup'\'' WHERE record_key = '\''$1'\''"' \
sh "$marker" >/dev/null
docker run --rm --network "$network" \
--mount "type=bind,source=$(pwd),target=/repo,readonly" \
--mount "type=bind,source=$secret_dir,target=/scratch" "$image" \
/repo/scripts/vector-restore.sh \
--active-host vector-db --active-database thoth --active-user "$THT_VECTOR_BOOTSTRAP_USER" \
--active-password-file /scratch/bootstrap \
--target-host vector-db-restore --target-database thoth \
--target-user "$THT_VECTOR_BOOTSTRAP_USER" --target-password-file /scratch/bootstrap \
--input /scratch/vector.dump
restored=$(docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql \
-XAt -U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth --command \
"SELECT content_hash <> 'mutated-after-backup' FROM vectors.memory WHERE record_key = '$marker'")
test "$restored" = t
pending=$(docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql \
-XAt -U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth --command \
"SELECT count(*) = 3 FROM public.tht_vector_migrations")
test "$pending" = t
dimensions=$(docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql \
-XAt -U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth --command \
"SELECT count(*) = 3 FROM pg_attribute a JOIN pg_class c ON c.oid=a.attrelid
JOIN pg_namespace n ON n.oid=c.relnamespace
WHERE n.nspname='vectors' AND a.attname='embedding' AND format_type(a.atttypid,a.atttypmod)='vectors.vector(768)'")
test "$dimensions" = t
echo "Disposable-volume backup, mutation, restore, ledger, health, and retrieval parity passed."
fi
echo "Local pgvector runtime/bootstrap rotation, least-privilege roles, and persistence passed." echo "Local pgvector runtime/bootstrap rotation, least-privilege roles, and persistence passed."
+45
View File
@@ -0,0 +1,45 @@
#!/bin/sh
set -eu
root=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
. "$root/deploy/vector/secret-policy.sh"
usage() {
echo "usage: $0 --host HOST --database DB --user USER --password-file FILE --output FILE [--port PORT]" >&2
exit 2
}
host= database= user= password_file= output= port=5432
while [ "$#" -gt 0 ]; do
case "$1" in
--host) host=${2-}; shift 2 ;;
--port) port=${2-}; shift 2 ;;
--database) database=${2-}; shift 2 ;;
--user) user=${2-}; shift 2 ;;
--password-file) password_file=${2-}; shift 2 ;;
--output) output=${2-}; shift 2 ;;
*) usage ;;
esac
done
[ -n "$host" ] && [ -n "$database" ] && [ -n "$user" ] || usage
[ -n "$password_file" ] && [ -n "$output" ] || usage
validate_secret_file "$password_file" "backup password file"
[ ! -e "$output" ] || { echo "refusing to overwrite existing backup: $output" >&2; exit 2; }
password=$(read_secret_file "$password_file" "backup password file")
umask 077
passfile=$(mktemp "${TMPDIR:-/tmp}/thoth-vector-pgpass.XXXXXX")
cleanup() { rm -f "$passfile" "$output.partial"; }
trap cleanup EXIT HUP INT TERM
escaped=$(printf '%s' "$password" | sed 's/\\/\\\\/g; s/:/\\:/g')
printf '%s:%s:%s:%s:%s\n' "$host" "$port" "$database" "$user" "$escaped" >"$passfile"
chmod 0600 "$passfile"
PGPASSFILE=$passfile pg_dump \
--host="$host" --port="$port" --username="$user" --dbname="$database" \
--format=custom --compress=9 \
--table=vectors.schema_records --table=vectors.evidence --table=vectors.memory \
--table=public.tht_vector_migrations --file="$output.partial"
mv "$output.partial" "$output"
echo "Vector backup written: $output"
+78
View File
@@ -0,0 +1,78 @@
#!/bin/sh
set -eu
root=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
. "$root/deploy/vector/secret-policy.sh"
usage() {
echo "usage: $0 --active-host HOST --active-database DB --active-user USER --active-password-file FILE --target-host HOST --target-database DB --target-user USER --target-password-file FILE --input FILE [--active-port PORT] [--target-port PORT] [--force-nonempty]" >&2
exit 2
}
active_host= active_database= active_user= active_password_file= active_port=5432
target_host= target_database= target_user= target_password_file= target_port=5432
input= force=0
while [ "$#" -gt 0 ]; do
case "$1" in
--active-host) active_host=${2-}; shift 2 ;;
--active-port) active_port=${2-}; shift 2 ;;
--active-database) active_database=${2-}; shift 2 ;;
--active-user) active_user=${2-}; shift 2 ;;
--active-password-file) active_password_file=${2-}; shift 2 ;;
--target-host) target_host=${2-}; shift 2 ;;
--target-port) target_port=${2-}; shift 2 ;;
--target-database) target_database=${2-}; shift 2 ;;
--target-user) target_user=${2-}; shift 2 ;;
--target-password-file) target_password_file=${2-}; shift 2 ;;
--input) input=${2-}; shift 2 ;;
--force-nonempty) force=1; shift ;;
*) usage ;;
esac
done
for value in "$active_host" "$active_database" "$active_user" "$active_password_file" \
"$target_host" "$target_database" "$target_user" "$target_password_file" "$input"; do
[ -n "$value" ] || usage
done
[ -r "$input" ] || { echo "backup input is not readable" >&2; exit 2; }
validate_secret_file "$active_password_file" "active source password file"
validate_secret_file "$target_password_file" "target password file"
umask 077
active_pass=$(mktemp "${TMPDIR:-/tmp}/thoth-vector-active-pgpass.XXXXXX")
target_pass=$(mktemp "${TMPDIR:-/tmp}/thoth-vector-target-pgpass.XXXXXX")
cleanup() { rm -f "$active_pass" "$target_pass"; }
trap cleanup EXIT HUP INT TERM
make_passfile() {
secret=$(read_secret_file "$5" "database password file")
escaped=$(printf '%s' "$secret" | sed 's/\\/\\\\/g; s/:/\\:/g')
printf '%s:%s:%s:%s:%s\n' "$1" "$2" "$3" "$4" "$escaped" >"$6"
chmod 0600 "$6"
}
make_passfile "$active_host" "$active_port" "$active_database" "$active_user" \
"$active_password_file" "$active_pass"
make_passfile "$target_host" "$target_port" "$target_database" "$target_user" \
"$target_password_file" "$target_pass"
identity_sql="SELECT system_identifier::text || ':' || d.oid::text FROM pg_control_system(), pg_database d WHERE d.datname = current_database()"
active_identity=$(PGPASSFILE=$active_pass psql -XAt --host="$active_host" --port="$active_port" \
--username="$active_user" --dbname="$active_database" --command="$identity_sql")
target_identity=$(PGPASSFILE=$target_pass psql -XAt --host="$target_host" --port="$target_port" \
--username="$target_user" --dbname="$target_database" --command="$identity_sql")
[ "$active_identity" != "$target_identity" ] || {
echo "refusing restore: active source and target are the same database" >&2; exit 2;
}
object_count=$(PGPASSFILE=$target_pass psql -XAt --host="$target_host" --port="$target_port" \
--username="$target_user" --dbname="$target_database" --command="
SELECT count(*) FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace
WHERE (n.nspname='vectors' OR (n.nspname='public' AND c.relname='tht_vector_migrations'))
AND c.relkind IN ('r','p','S','v','m');")
if [ "$object_count" != 0 ] && [ "$force" != 1 ]; then
echo "refusing restore into non-empty target; use --force-nonempty explicitly" >&2
exit 2
fi
PGPASSFILE=$target_pass pg_restore --exit-on-error --clean --if-exists --no-owner \
--host="$target_host" --port="$target_port" --username="$target_user" \
--dbname="$target_database" "$input"
echo "Vector restore completed into explicit target $target_host:$target_port/$target_database"