diff --git a/.dockerignore b/.dockerignore index 6d6c2571..e1c8604d 100644 --- a/.dockerignore +++ b/.dockerignore @@ -29,3 +29,5 @@ harness/artifacts harness/indexes harness/corpus deploy +!deploy/vector/ +!deploy/vector/secret-policy.sh diff --git a/.superpowers/sdd/task-3-report.md b/.superpowers/sdd/task-3-report.md new file mode 100644 index 00000000..c1e9b183 --- /dev/null +++ b/.superpowers/sdd/task-3-report.md @@ -0,0 +1,139 @@ +# Task 3 report — vector port and wrappers + +## Status + +Complete. Added the transport-neutral vector port, HTTP and legacy-direct wrappers, and +routed the existing `RestSearcher` and `DirectSearcher` through them while retaining the +canonical `VectorRecord` and `VectorHit` models. + +## TDD evidence + +- RED: `cd harness && .venv/bin/pytest tests/test_vector_port_contract.py -q` + - Result: collection failed with `ModuleNotFoundError: No module named + 'tht.adapters.vector'` (expected missing-port failure). +- GREEN: `cd harness && .venv/bin/pytest tests/test_vector_port_contract.py -q` + - Result: `4 passed in 0.11s`. + +## Verification + +- Required regression command: + `cd harness && .venv/bin/pytest tests/test_vector_port_contract.py + tests/test_vector_dual_key.py tests/test_search_similar_kinds.py + tests/test_memory_save_one.py tests/test_solved_question.py -q` + - Result: `26 passed in 0.12s` (fresh final run; earlier run: 26 passed in 0.16s). +- Broader vector-focused regression: + `cd harness && .venv/bin/pytest tests -q -k 'vector or search_similar or + memory_save_one or solved_question'` + - Result: `29 passed, 370 deselected in 0.49s`. +- Scoped lint: + `cd harness && .venv/bin/ruff check tht/ports/vector.py tht/adapters/vector + tht/vectorstore/reader.py tests/test_vector_port_contract.py` + - Result: `All checks passed!`. +- Whitespace check: `git diff --check` + - Result: exit 0, no output. +- Staged whitespace check: `git diff --cached --check` + - Result: exit 0, no output. + +## Commit + +`ff4d662 refactor(vector): define store contract` + +Only the six Task 3 implementation/test files were included in the commit. + +## Self-review / concerns + +- Reader and writer clients remain separate: search and health use only the reader; + hashes and upsert use only the writer. +- A missing writer reports `upsert=False` and raises `VectorWriteUnavailable`, including + for an empty upsert, so deployments cannot silently bypass the write credential gate. +- Existing REST kind forwarding, legacy-404 fallback, post-filtering, global similarity + merge, and direct table-per-kind behavior remain delegated to their established code. +- The port re-exports existing vector models instead of duplicating result types. +- Non-blocking design constraint: embedded values for the new generic `upsert` contract are + carried in `VectorRecord.metadata['embedding']`; this preserves the existing canonical + record model and REST payload without changing out-of-scope `records.py`. A later direct + pgvector implementation may choose to formalize that field across adapters. + +## Authorized contract correction + +Status: complete. Commit: `fe8d70d fix(vector): separate write transport fields`. + +The earlier metadata-envelope concern above is superseded. The contract now exports a +dedicated frozen `VectorWriteRecord` containing the canonical `VectorRecord`, precomputed +embedding, and content hash. `VectorStore.upsert` accepts only that envelope. HTTP row +serialization takes transport fields from the envelope and preserves `record.metadata` +unchanged, including legitimate metadata keys named `embedding` and `content_hash`. + +### Corrective TDD evidence + +- RED: `cd harness && .venv/bin/pytest tests/test_vector_port_contract.py -q` + - Result: collection failed with `ImportError: cannot import name 'VectorWriteRecord' from + 'tht.ports.vector'` (expected missing-envelope failure). +- GREEN: `cd harness && .venv/bin/pytest tests/test_vector_port_contract.py -q` + - Result: `7 passed in 0.11s`. +- Required regression suite: + `cd harness && .venv/bin/pytest tests/test_vector_port_contract.py + tests/test_vector_dual_key.py tests/test_search_similar_kinds.py + tests/test_memory_save_one.py tests/test_solved_question.py -q` + - Result: `29 passed in 0.15s` (fresh final run; earlier run: 29 passed in 0.14s). +- Scoped lint: + `cd harness && .venv/bin/ruff check tht/ports/__init__.py tht/ports/vector.py + tht/adapters/vector tht/vectorstore/reader.py tests/test_vector_port_contract.py` + - Result: `All checks passed!`. +- Whitespace check: `git diff --check` + - Result: exit 0, no output. + +### Corrective self-review + +- A real `evidence_records(...)` canonical builder record is serialized in the contract tests. +- Collision coverage proves semantic `embedding` and `content_hash` metadata survive while + distinct envelope values occupy the RPC row's top-level transport fields. +- Reader health/search still use only the reader client; hashes/upsert still use only writer. +- Existing kind forwarding, legacy 404 fallback, post-filtering, similarity merge, and direct + search behavior remain unchanged and covered by the required regression suite. +- The tracked plan, regenerated Task 3 scratch brief, port exports, adapter signature, and this + report now consistently describe `VectorWriteRecord`. +- Concerns: none known. + +--- + +# Task simple-config-3 report — one bundle for local-vector and preprocess + +## Status + +Complete. Local pgvector bootstrap, reconciliation, migration, and preprocess services now +mount only `/run/secrets/thothii.secrets`. `deploy/vector/secret-policy.sh` validates the +whole bundle (allowlist, duplicate/empty/unknown keys, comments/blank lines, mode and symlink +policy) and returns only the requested value. The core entrypoint exposes DWH/vector/CA values +to the harness and materializes short-lived 0600 password files for workspace resolution. + +## TDD evidence + +- RED: `./scripts/test-preprocess-compose-config.sh` failed on the pre-existing + `vector_reader_password` Compose secret declaration. +- GREEN: the same command passes after the bundle conversion and verifies local-vector + workspace interpolation and shared secret mounts. +- Added bundle parser regressions to `./scripts/test-vector-secret-policy.sh`: comments/blank + lines are accepted and an unrelated duplicate key is rejected. + +## Verification + +- `./scripts/test-vector-secret-policy.sh` — passed. +- `./scripts/test-preprocess-compose-config.sh` — passed. +- `./scripts/test-vector-backup-restore-safety.sh` — passed. +- `./scripts/test-default-compose.sh` — passed. +- `./scripts/test-container-deployment.sh` — passed. +- `./scripts/local-vector-smoke.sh` — passed (real Docker; bootstrap rotation, role + reconciliation, migration, persistence and restart). +- `./scripts/preprocess-smoke.sh` — passed (real Docker; unchanged rerun, mutation, DWH job, + ACTIVE publication and cleanup). +- `git diff --check` and `sh -n` gates — passed. + +## Commit + +Pending: `feat(compose): use one secret bundle for local services`. + +## Concerns + +The rotation helper still accepts old/new scratch files because that is its explicit CLI +contract; the smoke script keeps those files outside Compose and mounts only the bundle. diff --git a/deploy/compose.local-vector.yaml b/deploy/compose.local-vector.yaml index 53b01364..0bf41337 100644 --- a/deploy/compose.local-vector.yaml +++ b/deploy/compose.local-vector.yaml @@ -6,9 +6,8 @@ services: THT_VECTOR_BOOTSTRAP_USER: "${THT_VECTOR_BOOTSTRAP_USER:-postgres}" THT_VECTOR_READER_USER: "${THT_VECTOR_READER_USER:-thoth_vector_reader}" THT_VECTOR_WRITER_USER: "${THT_VECTOR_WRITER_USER:-thoth_vector_writer}" - THT_VECTOR_READER_PASSWORD_FILE: /run/secrets/vector_reader_password - THT_VECTOR_WRITER_PASSWORD_FILE: /run/secrets/vector_writer_password - secrets: [vector_reader_password, vector_writer_password] + THT_SECRETS_FILE: /run/secrets/thothii.secrets + secrets: [{source: thothii_secrets, target: thothii.secrets}] depends_on: vector-migrate: condition: service_completed_successfully @@ -23,12 +22,16 @@ services: environment: POSTGRES_DB: "${THT_VECTOR_DATABASE:-thoth}" POSTGRES_USER: "${THT_VECTOR_BOOTSTRAP_USER:-postgres}" - POSTGRES_PASSWORD_FILE: /run/secrets/vector_bootstrap_password THT_VECTOR_MIGRATOR_USER: "${THT_VECTOR_MIGRATOR_USER:-thoth_vector_migrator}" THT_VECTOR_READER_USER: "${THT_VECTOR_READER_USER:-thoth_vector_reader}" THT_VECTOR_WRITER_USER: "${THT_VECTOR_WRITER_USER:-thoth_vector_writer}" - secrets: [vector_bootstrap_password, vector_migrator_password, vector_reader_password, vector_writer_password] - volumes: [vector_data:/var/lib/postgresql/data] + THT_SECRETS_FILE: /run/secrets/thothii.secrets + secrets: [{source: thothii_secrets, target: thothii.secrets}] + entrypoint: [/opt/thoth/vector-db-entrypoint.sh] + volumes: + - vector_data:/var/lib/postgresql/data + - ./deploy/vector/vector-db-entrypoint.sh:/opt/thoth/vector-db-entrypoint.sh:ro + - ./deploy/vector/secret-policy.sh:/opt/thoth/secret-policy.sh:ro healthcheck: test: [CMD-SHELL, "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"] interval: 5s @@ -50,8 +53,9 @@ services: THT_VECTOR_MIGRATOR_USER: "${THT_VECTOR_MIGRATOR_USER:-thoth_vector_migrator}" THT_VECTOR_READER_USER: "${THT_VECTOR_READER_USER:-thoth_vector_reader}" THT_VECTOR_WRITER_USER: "${THT_VECTOR_WRITER_USER:-thoth_vector_writer}" + THT_SECRETS_FILE: /run/secrets/thothii.secrets entrypoint: [/opt/thoth/reconcile-roles.sh] - secrets: [vector_bootstrap_password, vector_migrator_password, vector_reader_password, vector_writer_password] + secrets: [{source: thothii_secrets, target: thothii.secrets}] volumes: - ./deploy/vector/reconcile-roles.sh:/opt/thoth/reconcile-roles.sh:ro - ./deploy/vector/secret-policy.sh:/opt/thoth/secret-policy.sh:ro @@ -69,10 +73,14 @@ services: entrypoint: [sh, -ec] command: - | - password=$$(cat /run/secrets/vector_migrator_password) - encoded=$$(python -c 'import sys, urllib.parse; print(urllib.parse.quote(sys.argv[1], safe=""))' "$$password") - exec /opt/venv/bin/tht vector migrate --database-url "postgresql+psycopg2://${THT_VECTOR_MIGRATOR_USER:-thoth_vector_migrator}:$$encoded@vector-db:5432/${THT_VECTOR_DATABASE:-thoth}" --json - secrets: [vector_migrator_password] + . /opt/thoth/secret-policy.sh + export PGPASSWORD=$$(read_bundle_secret /run/secrets/thothii.secrets THT_VECTOR_MIGRATOR_PASSWORD) + exec /opt/venv/bin/tht vector migrate --database-url "postgresql+psycopg2://${THT_VECTOR_MIGRATOR_USER:-thoth_vector_migrator}@vector-db:5432/${THT_VECTOR_DATABASE:-thoth}" --json + secrets: [{source: thothii_secrets, target: thothii.secrets}] + environment: + THT_SECRETS_FILE: /run/secrets/thothii.secrets + volumes: + - ./deploy/vector/secret-policy.sh:/opt/thoth/secret-policy.sh:ro depends_on: vector-reconcile: {condition: service_completed_successfully} restart: "no" @@ -80,13 +88,3 @@ services: volumes: vector_data: labels: {io.thothii.smoke-owner: "${THOTH_SMOKE_OWNER:-operator}"} - -secrets: - vector_bootstrap_password: - file: ${THT_VECTOR_BOOTSTRAP_PASSWORD_SECRET_FILE:-deploy/secrets/vector_bootstrap_password} - vector_migrator_password: - file: ${THT_VECTOR_MIGRATOR_PASSWORD_SECRET_FILE:-deploy/secrets/vector_migrator_password} - vector_reader_password: - file: ${THT_VECTOR_READER_PASSWORD_SECRET_FILE:-deploy/secrets/vector_reader_password} - vector_writer_password: - file: ${THT_VECTOR_WRITER_PASSWORD_SECRET_FILE:-deploy/secrets/vector_writer_password} diff --git a/deploy/compose.preprocess-local-vector.yaml b/deploy/compose.preprocess-local-vector.yaml index 3b8b8376..8ce48d61 100644 --- a/deploy/compose.preprocess-local-vector.yaml +++ b/deploy/compose.preprocess-local-vector.yaml @@ -1,21 +1,14 @@ services: preprocess-evidence: environment: - THT_VECTOR_READER_PASSWORD_FILE: /run/secrets/vector_reader_password - THT_VECTOR_WRITER_PASSWORD_FILE: /run/secrets/vector_writer_password - secrets: [vector_reader_password, vector_writer_password] + THT_SECRETS_FILE: /run/secrets/thothii.secrets + secrets: [{source: thothii_secrets, target: thothii.secrets}] depends_on: vector-migrate: {condition: service_completed_successfully} preprocess-dwh: environment: - THT_VECTOR_READER_PASSWORD_FILE: /run/secrets/vector_reader_password - secrets: [vector_reader_password] + THT_SECRETS_FILE: /run/secrets/thothii.secrets + secrets: [{source: thothii_secrets, target: thothii.secrets}] depends_on: vector-migrate: {condition: service_completed_successfully} - -secrets: - vector_reader_password: - file: ${THT_VECTOR_READER_PASSWORD_SECRET_FILE:?set THT_VECTOR_READER_PASSWORD_SECRET_FILE} - vector_writer_password: - file: ${THT_VECTOR_WRITER_PASSWORD_SECRET_FILE:?set THT_VECTOR_WRITER_PASSWORD_SECRET_FILE} diff --git a/deploy/compose.preprocess.yaml b/deploy/compose.preprocess.yaml index 794a9e58..0b18e140 100644 --- a/deploy/compose.preprocess.yaml +++ b/deploy/compose.preprocess.yaml @@ -10,6 +10,8 @@ services: environment: THT_DATA_ROOT: /data THT_OLLAMA_URL: "${THT_OLLAMA_URL:-http://host.docker.internal:11434}" + THT_SECRETS_FILE: /run/secrets/thothii.secrets + secrets: [{source: thothii_secrets, target: thothii.secrets}] volumes: - thoth_data:/data - ./deploy/workspaces:/app/harness/workspaces:ro @@ -25,6 +27,8 @@ services: command: ["mkdir -p /data/workspaces/preprocess-dwh && exec /app/docker/core-entrypoint.sh preprocess dwh --steps introspect --json -c /app/harness/workspaces/preprocess-dwh.yaml"] environment: THT_DATA_ROOT: /data + THT_SECRETS_FILE: /run/secrets/thothii.secrets + secrets: [{source: thothii_secrets, target: thothii.secrets}] volumes: - thoth_data:/data - ./deploy/workspaces:/app/harness/workspaces:ro diff --git a/deploy/vector/reconcile-roles.sh b/deploy/vector/reconcile-roles.sh index b2f09719..74f14afe 100755 --- a/deploy/vector/reconcile-roles.sh +++ b/deploy/vector/reconcile-roles.sh @@ -3,10 +3,11 @@ set -eu . /opt/thoth/secret-policy.sh -export PGPASSWORD=$(read_secret_file /run/secrets/vector_bootstrap_password vector_bootstrap_password) -migrator_password=$(read_secret_file /run/secrets/vector_migrator_password vector_migrator_password) -reader_password=$(read_secret_file /run/secrets/vector_reader_password vector_reader_password) -writer_password=$(read_secret_file /run/secrets/vector_writer_password vector_writer_password) +bundle=${THT_SECRETS_FILE:-/run/secrets/thothii.secrets} +export PGPASSWORD=$(read_bundle_secret "$bundle" THT_VECTOR_BOOTSTRAP_PASSWORD) +migrator_password=$(read_bundle_secret "$bundle" THT_VECTOR_MIGRATOR_PASSWORD) +reader_password=$(read_bundle_secret "$bundle" THT_VECTOR_READER_PASSWORD) +writer_password=$(read_bundle_secret "$bundle" THT_VECTOR_WRITER_PASSWORD) psql --set=ON_ERROR_STOP=1 \ --set=migrator_user="$THT_VECTOR_MIGRATOR_USER" \ diff --git a/deploy/vector/secret-policy.sh b/deploy/vector/secret-policy.sh index 6f039011..09f34823 100755 --- a/deploy/vector/secret-policy.sh +++ b/deploy/vector/secret-policy.sh @@ -3,7 +3,7 @@ validate_secret_file() { secret_path=$1 secret_name=$2 - if [ ! -f "$secret_path" ] || [ ! -r "$secret_path" ] || [ ! -s "$secret_path" ]; then + if [ -L "$secret_path" ] || [ ! -f "$secret_path" ] || [ ! -r "$secret_path" ] || [ ! -s "$secret_path" ]; then echo "$secret_name must be a readable, non-empty regular file" >&2 return 2 fi @@ -22,3 +22,49 @@ read_secret_file() { validate_secret_file "$1" "$2" || return cat "$1" } + +# Read one value from the deployment bundle without putting the bundle itself in +# a service environment. The parser is deliberately strict: one KEY=VALUE per +# line, no duplicate keys, no unknown syntax, and no whitespace in credentials. +read_bundle_secret() { + bundle_path=$1 + bundle_key=$2 + if [ -L "$bundle_path" ] || [ ! -f "$bundle_path" ] || [ ! -r "$bundle_path" ] || [ ! -s "$bundle_path" ]; then + echo "secret bundle must be a readable, non-empty regular file" >&2 + return 2 + fi + mode=$(stat -c '%a' "$bundle_path" 2>/dev/null || stat -f '%Lp' "$bundle_path" 2>/dev/null) || return 2 + case "$bundle_path:$mode" in + /run/secrets/*:444|/run/secrets/*:400|/run/secrets/*:600|*:600|*:400) ;; + *) echo "secret bundle must have mode 0600 or stricter (Docker secrets may be 0444)" >&2; return 2 ;; + esac + case "$bundle_key" in + THT_[A-Z0-9_]*|PI_PROVIDER_API_KEY) ;; + *) echo "invalid secret bundle key" >&2; return 2 ;; + esac + value=$(awk -v wanted="$bundle_key" ' + /^[[:space:]]*$/ || /^[[:space:]]*#/ { next } + /^[A-Z][A-Z0-9_]*=/ { + key=$0; sub(/=.*/, "", key) + val=$0; sub(/^[^=]*=/, "", val) + if (key !~ /^(THT_MODEL_API_KEY|THT_DWH_API_KEY|THT_VEC_API_KEY|THT_VEC_WRITE_API_KEY|THT_CA|THT_SSL_CA|THT_VECTOR_BOOTSTRAP_PASSWORD|THT_VECTOR_MIGRATOR_PASSWORD|THT_VECTOR_READER_PASSWORD|THT_VECTOR_WRITER_PASSWORD|PI_PROVIDER_API_KEY)$/) exit 6 + if (val == "") exit 7 + if (++seen[key] > 1) exit 8 + if (key == wanted) { + if (found) exit 3 + found=1; print val + } + next + } + { exit 4 } + END { if (!found) exit 5 } + ' "$bundle_path") || { + echo "$bundle_key is unavailable in secret bundle" >&2 + return 2 + } + if [ -z "$value" ] || printf '%s' "$value" | LC_ALL=C grep -q '[[:space:]]'; then + echo "$bundle_key must contain no whitespace" >&2 + return 2 + fi + printf '%s' "$value" +} diff --git a/deploy/vector/vector-db-entrypoint.sh b/deploy/vector/vector-db-entrypoint.sh new file mode 100755 index 00000000..75cd0e40 --- /dev/null +++ b/deploy/vector/vector-db-entrypoint.sh @@ -0,0 +1,9 @@ +#!/bin/sh +set -eu + +. /opt/thoth/secret-policy.sh + +bundle=${THT_SECRETS_FILE:-/run/secrets/thothii.secrets} +export POSTGRES_PASSWORD=$(read_bundle_secret "$bundle" THT_VECTOR_BOOTSTRAP_PASSWORD) +unset THT_SECRETS_FILE +exec /usr/local/bin/docker-entrypoint.sh postgres diff --git a/docker/core-entrypoint.sh b/docker/core-entrypoint.sh index c0da216b..900b4f40 100755 --- a/docker/core-entrypoint.sh +++ b/docker/core-entrypoint.sh @@ -1,6 +1,12 @@ #!/bin/sh set -eu +if [ -r /app/docker/secret-policy.sh ]; then + . /app/docker/secret-policy.sh +else + . "$(CDPATH= cd -- "$(dirname -- "$0")/../deploy/vector" && pwd)/secret-policy.sh" +fi + if [ -n "${PI_PROVIDER_API_KEY:-}" ]; then echo "PI_PROVIDER_API_KEY is unsupported; configure THT_MODEL_API_KEY_FILE" >&2 exit 2 @@ -24,6 +30,49 @@ load_secret THT_DWH_API_KEY THT_DWH_API_KEY_FILE load_secret THT_VEC_API_KEY THT_VEC_API_KEY_FILE load_secret THT_VEC_WRITE_API_KEY THT_VEC_WRITE_API_KEY_FILE +secret_tmp_dir= +cleanup_secret_tmp() { + if [ -n "${secret_tmp_dir:-}" ]; then rm -rf "$secret_tmp_dir"; fi +} +trap cleanup_secret_tmp EXIT HUP INT TERM + +bundle=${THT_SECRETS_FILE:-} +if [ -n "$bundle" ] && [ -r "$bundle" ]; then + # REST adapters consume these values while the harness is running. They are + # loaded here (before `tht` starts), not only in the backend/Pi process. + load_bundle_env() { + env_name=$1 + key=$2 + value=$(read_bundle_secret "$bundle" "$key" 2>/dev/null || true) + if [ -n "$value" ]; then export "$env_name=$value"; fi + } + load_bundle_env THT_DWH_API_KEY THT_DWH_API_KEY + load_bundle_env THT_VEC_API_KEY THT_VEC_API_KEY + load_bundle_env THT_VEC_WRITE_API_KEY THT_VEC_WRITE_API_KEY + load_bundle_env THT_SSL_CA THT_SSL_CA + if [ -z "${THT_SSL_CA:-}" ]; then load_bundle_env THT_SSL_CA THT_CA; fi + + # DatabaseConfig intentionally accepts password_file paths. Materialize + # short-lived 0600 files so workspace interpolation never puts a password in + # YAML, a URL, or the container's immutable Config.Env. + umask 077 + secret_tmp_dir=$(mktemp -d "${TMPDIR:-/tmp}/thothii-secrets.XXXXXX") + materialize_password() { + env_name=$1 + key=$2 + value=$(read_bundle_secret "$bundle" "$key" 2>/dev/null || true) + if [ -n "$value" ]; then + file="$secret_tmp_dir/$key" + printf '%s' "$value" >"$file" + export "$env_name=$file" + fi + } + materialize_password THT_VECTOR_BOOTSTRAP_PASSWORD_FILE THT_VECTOR_BOOTSTRAP_PASSWORD + materialize_password THT_VECTOR_MIGRATOR_PASSWORD_FILE THT_VECTOR_MIGRATOR_PASSWORD + materialize_password THT_VECTOR_READER_PASSWORD_FILE THT_VECTOR_READER_PASSWORD + materialize_password THT_VECTOR_WRITER_PASSWORD_FILE THT_VECTOR_WRITER_PASSWORD +fi + case "${1:-server}" in server) shift || true diff --git a/docker/core.Dockerfile b/docker/core.Dockerfile index 8af924f4..a199e489 100644 --- a/docker/core.Dockerfile +++ b/docker/core.Dockerfile @@ -46,8 +46,9 @@ RUN python -m venv /opt/venv \ COPY --from=backend-build /src/backend/dist /app/backend/dist COPY --from=backend-build /src/backend/node_modules /app/backend/node_modules COPY docker/core-entrypoint.sh /app/docker/core-entrypoint.sh +COPY deploy/vector/secret-policy.sh /app/docker/secret-policy.sh COPY docker/smoke/core-smoke.sh /app/docker/smoke/core-smoke.sh -RUN chmod 0555 /app/docker/core-entrypoint.sh /app/docker/smoke/core-smoke.sh \ +RUN chmod 0555 /app/docker/core-entrypoint.sh /app/docker/secret-policy.sh /app/docker/smoke/core-smoke.sh \ && test ! -e /app/harness/.env \ && test ! -d /app/harness/workspaces diff --git a/scripts/local-vector-smoke.sh b/scripts/local-vector-smoke.sh index 3ec07fcc..071757ac 100755 --- a/scripts/local-vector-smoke.sh +++ b/scripts/local-vector-smoke.sh @@ -22,16 +22,27 @@ marker="local-vector-$smoke_project" restore_container="${smoke_project}-restore" restore_volume="${smoke_project}-restore-data" -for secret in bootstrap migrator reader writer; do - password="smoke-${secret}-${smoke_project}" - printf '%s' "$password" >"$secret_dir/$secret" - chmod 0600 "$secret_dir/$secret" -done - -export THT_VECTOR_BOOTSTRAP_PASSWORD_SECRET_FILE="$secret_dir/bootstrap" -export THT_VECTOR_MIGRATOR_PASSWORD_SECRET_FILE="$secret_dir/migrator" -export THT_VECTOR_READER_PASSWORD_SECRET_FILE="$secret_dir/reader" -export THT_VECTOR_WRITER_PASSWORD_SECRET_FILE="$secret_dir/writer" +bootstrap_password="smoke-bootstrap-$smoke_project" +migrator_password="smoke-migrator-$smoke_project" +reader_password="smoke-reader-$smoke_project" +writer_password="smoke-writer-$smoke_project" +bundle="$secret_dir/thothii.secrets" +write_bundle() { + umask 077 + { + printf 'THT_VECTOR_BOOTSTRAP_PASSWORD=%s\n' "$bootstrap_password" + printf 'THT_VECTOR_MIGRATOR_PASSWORD=%s\n' "$migrator_password" + printf 'THT_VECTOR_READER_PASSWORD=%s\n' "$reader_password" + printf 'THT_VECTOR_WRITER_PASSWORD=%s\n' "$writer_password" + } >"$bundle" + chmod 0600 "$bundle" +} +write_bundle +export THT_SECRETS_FILE="$bundle" +# The rotation helper has an old/new file interface; these are test-only +# scratch files and are never mounted into a Compose service. +printf '%s' "$bootstrap_password" >"$secret_dir/bootstrap" +chmod 0600 "$secret_dir/bootstrap" export THT_VECTOR_BOOTSTRAP_USER=thoth_bootstrap_smoke export THOTH_SMOKE_OWNER="$smoke_owner" @@ -109,7 +120,16 @@ if [ "$mode" = "--live-collision-test" ]; then fi probe_vector() { - compose exec -T core /opt/venv/bin/python - "$marker" "$1" <<'PY' + compose exec -T core sh -ec ' + . /app/docker/secret-policy.sh + tmp=$(mktemp -d); trap "rm -rf \"$tmp\"" EXIT + for role in READER WRITER; do + file="$tmp/$role" + read_bundle_secret /run/secrets/thothii.secrets "THT_VECTOR_${role}_PASSWORD" >"$file" + export "THT_VECTOR_${role}_PASSWORD_FILE=$file" + done + exec /opt/venv/bin/python - "$1" "$2" + ' sh "$marker" "$1" <<'PY' import hashlib import os import sys @@ -173,23 +193,23 @@ if printf '%s' "$inspect_env" | grep -q "smoke-\(reader\|writer\)-${smoke_projec echo "docker inspect exposed a direct vector password" >&2 exit 1 fi -printf '%s' "$inspect_env" | grep -q 'THT_VECTOR_READER_PASSWORD_FILE=/run/secrets/vector_reader_password' -printf '%s' "$inspect_env" | grep -q 'THT_VECTOR_WRITER_PASSWORD_FILE=/run/secrets/vector_writer_password' +printf '%s' "$inspect_env" | grep -q 'THT_SECRETS_FILE=/run/secrets/thothii.secrets' migration_status=$(compose run --rm --no-deps vector-migrate) printf '%s\n' "$migration_status" | grep -q '"pending": \[\]' migrator_flags=$(compose run --rm --no-deps --entrypoint sh vector-reconcile -ec ' - export PGPASSWORD=$(cat /run/secrets/vector_bootstrap_password) + . /opt/thoth/secret-policy.sh + export PGPASSWORD=$(read_bundle_secret /run/secrets/thothii.secrets THT_VECTOR_BOOTSTRAP_PASSWORD) psql -At --host vector-db --username "$THT_VECTOR_BOOTSTRAP_USER" --dbname thoth \ --command "SELECT (NOT rolcreaterole) AND (NOT rolcreatedb) AND (NOT rolsuper) FROM pg_roles WHERE rolname = '\''thoth_vector_migrator'\''" ') test "$migrator_flags" = t probe_vector write -old_reader_password=$(cat "$secret_dir/reader") -for secret in migrator reader writer; do - password="rotated-${secret}-${smoke_project}" - printf '%s' "$password" >"$secret_dir/$secret" -done +old_reader_password="$reader_password" +migrator_password="rotated-migrator-$smoke_project" +reader_password="rotated-reader-$smoke_project" +writer_password="rotated-writer-$smoke_project" +write_bundle compose run --rm vector-reconcile rotation_status=$(compose run --rm --no-deps vector-migrate) @@ -205,7 +225,7 @@ fi compose up --force-recreate --no-deps --wait core probe_vector read -old_bootstrap_password=$(cat "$secret_dir/bootstrap") +old_bootstrap_password="$bootstrap_password" printf '%s' "wrong-bootstrap-${smoke_project}" >"$secret_dir/bootstrap-wrong" printf '%s' "next-bootstrap-'quoted-${smoke_project}" >"$secret_dir/bootstrap-next" cp "$secret_dir/bootstrap" "$secret_dir/bootstrap-before-negative" @@ -238,6 +258,8 @@ COMPOSE_PROJECT_NAME="$smoke_project" \ ./scripts/vector-rotate-bootstrap-password.sh \ "$secret_dir/bootstrap" "$secret_dir/bootstrap-next" new_bootstrap_password=$(cat "$secret_dir/bootstrap") +bootstrap_password="$new_bootstrap_password" +write_bundle test "$new_bootstrap_password" != "$old_bootstrap_password" if compose run --rm --no-deps --entrypoint psql \ -e PGPASSWORD="$old_bootstrap_password" vector-reconcile \ @@ -271,9 +293,12 @@ if [ "$mode" = "--backup-restore" ]; then --label "io.thothii.smoke-owner=$smoke_owner" \ --network "$network" --network-alias vector-db-restore \ --mount "type=volume,source=$restore_volume,target=/var/lib/postgresql/data" \ - --mount "type=bind,source=$secret_dir/bootstrap,target=/run/secrets/bootstrap,readonly" \ + --mount "type=bind,source=$bundle,target=/run/secrets/thothii.secrets,readonly" \ + --mount "type=bind,source=$(pwd)/deploy/vector/vector-db-entrypoint.sh,target=/opt/thoth/vector-db-entrypoint.sh,readonly" \ + --mount "type=bind,source=$(pwd)/deploy/vector/secret-policy.sh,target=/opt/thoth/secret-policy.sh,readonly" \ -e POSTGRES_DB=thoth -e POSTGRES_USER="$THT_VECTOR_BOOTSTRAP_USER" \ - -e POSTGRES_PASSWORD_FILE=/run/secrets/bootstrap "$image" >/dev/null + -e THT_SECRETS_FILE=/run/secrets/thothii.secrets \ + --entrypoint /opt/thoth/vector-db-entrypoint.sh "$image" >/dev/null attempts=0 until docker exec "$restore_container" pg_isready \ -U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth >/dev/null 2>&1; do @@ -300,7 +325,8 @@ if [ "$mode" = "--backup-restore" ]; then --output /scratch/vector.dump compose exec -T vector-db sh -ec ' - export PGPASSWORD=$(cat /run/secrets/vector_bootstrap_password) + . /opt/thoth/secret-policy.sh + export PGPASSWORD=$(read_bundle_secret /run/secrets/thothii.secrets THT_VECTOR_BOOTSTRAP_PASSWORD) psql -X -U "$POSTGRES_USER" -d thoth -v ON_ERROR_STOP=1 --command \ "UPDATE vectors.memory SET content_hash = '\''mutated-after-backup'\'' WHERE record_key = '\''$1'\''"' \ sh "$marker" >/dev/null @@ -339,23 +365,25 @@ if [ "$mode" = "--backup-restore" ]; then docker run --rm --network "$network" \ --mount "type=bind,source=$(pwd)/deploy/vector/reconcile-roles.sh,target=/opt/thoth/reconcile-roles.sh,readonly" \ --mount "type=bind,source=$(pwd)/deploy/vector/secret-policy.sh,target=/opt/thoth/secret-policy.sh,readonly" \ - --mount "type=bind,source=$secret_dir/bootstrap,target=/run/secrets/vector_bootstrap_password,readonly" \ - --mount "type=bind,source=$secret_dir/migrator,target=/run/secrets/vector_migrator_password,readonly" \ - --mount "type=bind,source=$secret_dir/reader,target=/run/secrets/vector_reader_password,readonly" \ - --mount "type=bind,source=$secret_dir/writer,target=/run/secrets/vector_writer_password,readonly" \ + --mount "type=bind,source=$bundle,target=/run/secrets/thothii.secrets,readonly" \ -e PGHOST=vector-db-restore -e PGDATABASE=thoth \ -e PGUSER="$THT_VECTOR_BOOTSTRAP_USER" \ + -e THT_SECRETS_FILE=/run/secrets/thothii.secrets \ -e THT_VECTOR_MIGRATOR_USER=thoth_vector_migrator \ -e THT_VECTOR_READER_USER=thoth_vector_reader \ -e THT_VECTOR_WRITER_USER=thoth_vector_writer \ --entrypoint /opt/thoth/reconcile-roles.sh "$image" >/dev/null - docker run --rm -i --network "$network" \ - --mount "type=bind,source=$secret_dir/reader,target=/run/secrets/vector_reader_password,readonly" \ - --mount "type=bind,source=$secret_dir/writer,target=/run/secrets/vector_writer_password,readonly" \ - -e THT_VECTOR_READER_PASSWORD_FILE=/run/secrets/vector_reader_password \ - -e THT_VECTOR_WRITER_PASSWORD_FILE=/run/secrets/vector_writer_password \ - --entrypoint /opt/venv/bin/python thothii-core:local - "$marker" <<'PY' + compose exec -T core sh -ec ' + . /app/docker/secret-policy.sh + tmp=$(mktemp -d); trap "rm -rf \"$tmp\"" EXIT + for role in READER WRITER; do + file="$tmp/$role" + read_bundle_secret /run/secrets/thothii.secrets "THT_VECTOR_${role}_PASSWORD" >"$file" + export "THT_VECTOR_${role}_PASSWORD_FILE=$file" + done + exec /opt/venv/bin/python - "$1" + ' sh "$marker" <<'PY' import hashlib import os import sys diff --git a/scripts/preprocess-smoke.sh b/scripts/preprocess-smoke.sh index ae3d4820..c9a6486d 100755 --- a/scripts/preprocess-smoke.sh +++ b/scripts/preprocess-smoke.sh @@ -48,14 +48,15 @@ trap 'exit 130' INT trap 'exit 143' TERM mkdir -p "$tmp/source/evidence" printf '%s\n' '# Evidence' 'generation one' >"$tmp/source/evidence/a.md" -for name in bootstrap migrator reader writer; do - printf '%s' "smoke-$name-$project" >"$tmp/$name" - chmod 0600 "$tmp/$name" -done -export THT_VECTOR_BOOTSTRAP_PASSWORD_SECRET_FILE="$tmp/bootstrap" -export THT_VECTOR_MIGRATOR_PASSWORD_SECRET_FILE="$tmp/migrator" -export THT_VECTOR_READER_PASSWORD_SECRET_FILE="$tmp/reader" -export THT_VECTOR_WRITER_PASSWORD_SECRET_FILE="$tmp/writer" +bundle="$tmp/thothii.secrets" +{ + printf 'THT_VECTOR_BOOTSTRAP_PASSWORD=smoke-bootstrap-%s\n' "$project" + printf 'THT_VECTOR_MIGRATOR_PASSWORD=smoke-migrator-%s\n' "$project" + printf 'THT_VECTOR_READER_PASSWORD=smoke-reader-%s\n' "$project" + printf 'THT_VECTOR_WRITER_PASSWORD=smoke-writer-%s\n' "$project" +} >"$bundle" +chmod 0600 "$bundle" +export THT_SECRETS_FILE="$bundle" export THT_OLLAMA_URL=http://mock-embeddings:8081 cat >"$tmp/smoke.yaml" <"$tmp_bundle" <<'EOF' +THT_VECTOR_BOOTSTRAP_PASSWORD=test-bootstrap +THT_VECTOR_MIGRATOR_PASSWORD=test-migrator +THT_VECTOR_READER_PASSWORD=test-reader +THT_VECTOR_WRITER_PASSWORD=test-writer +EOF +chmod 0600 "$tmp_bundle" +export THT_SECRETS_FILE="$tmp_bundle" local_files="-f compose.yaml -f deploy/compose.local-vector.yaml -f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml" local_json=$(docker compose $local_files --profile local-vector --profile preprocess config --format json) @@ -16,6 +22,16 @@ import json, sys config = json.load(sys.stdin) services = config["services"] +assert "thothii_secrets" in config.get("secrets", {}), config.get("secrets") +assert "vector_bootstrap_password" not in config.get("secrets", {}) +assert "vector_migrator_password" not in config.get("secrets", {}) +assert "vector_reader_password" not in config.get("secrets", {}) +assert "vector_writer_password" not in config.get("secrets", {}) +for name, service in services.items(): + if name.startswith("vector-") or name.startswith("preprocess-") or name == "core": + assert any(item.get("target") == "thothii.secrets" for item in service.get("secrets", []) if isinstance(item, dict)), (name, service.get("secrets")) + assert "vector_reader_password" not in str(service) + assert "vector_writer_password" not in str(service) for name in ("preprocess-evidence", "preprocess-dwh"): dependency = services[name].get("depends_on", {}).get("vector-migrate") assert dependency is not None, f"{name} does not depend on vector-migrate" @@ -37,7 +53,32 @@ assert "vector-reconcile" not in services for name in ("preprocess-evidence", "preprocess-dwh"): service = services[name] assert "depends_on" not in service - assert not service.get("secrets"), service.get("secrets") + assert all(item.get("target") == "thothii.secrets" for item in service.get("secrets", []) if isinstance(item, dict)), service.get("secrets") + assert "vector_reader_password" not in str(service) + assert "vector_writer_password" not in str(service) ' +python3 - <<'PY' +import os +from pathlib import Path + +os.environ.update({ + "THT_DB_NAME": "thoth", + "THT_DWH_REST_URL": "http://dwh.invalid", + "THT_DWH_API_KEY": "dwh", + "THT_VECTOR_DATABASE": "thoth", + "THT_VECTOR_READER_USER": "reader", + "THT_VECTOR_WRITER_USER": "writer", + "THT_VECTOR_READER_PASSWORD_FILE": "/tmp/generated-reader", + "THT_VECTOR_WRITER_PASSWORD_FILE": "/tmp/generated-writer", + "THT_DOCS_ROOT": "/data/source", + "THT_OLLAMA_URL": "http://ollama.invalid", +}) +text = Path("deploy/workspaces/local-vector.yaml").read_text() +assert "password_file: ${THT_VECTOR_READER_PASSWORD_FILE}" in text +assert "password_file: ${THT_VECTOR_WRITER_PASSWORD_FILE}" in text +assert "${THT_SECRETS_FILE}" not in text +print("local-vector workspace resolution contract: ok") +PY + echo "preprocess compose config: ok" diff --git a/scripts/test-vector-backup-restore-safety.sh b/scripts/test-vector-backup-restore-safety.sh index 68188fd6..76845eea 100755 --- a/scripts/test-vector-backup-restore-safety.sh +++ b/scripts/test-vector-backup-restore-safety.sh @@ -88,6 +88,13 @@ grep -q -- '--exit-on-error' "$tmp/restore.log" # The live restore smoke must follow the packaged migration set instead of a stale # hard-coded count when a new migration is added. +if grep -Eq 'vector_(bootstrap|migrator|reader|writer)_password' \ + deploy/compose.local-vector.yaml deploy/compose.preprocess-local-vector.yaml; then + echo "local-vector Compose still declares legacy per-password secrets" >&2 + exit 1 +fi +grep -Fq 'thothii_secrets' deploy/compose.local-vector.yaml +grep -Fq 'thothii_secrets' deploy/compose.preprocess-local-vector.yaml if grep -Fq 'SELECT count(*) = 3 FROM public.tht_vector_migrations' \ scripts/local-vector-smoke.sh; then echo "local vector smoke hard-codes the pre-004 migration count" >&2 diff --git a/scripts/test-vector-secret-policy.sh b/scripts/test-vector-secret-policy.sh index d271ae01..8369ed68 100755 --- a/scripts/test-vector-secret-policy.sh +++ b/scripts/test-vector-secret-policy.sh @@ -14,10 +14,13 @@ printf 'safe-quoted-\047-dollar-$' >"$tmp/valid" printf 'docker-secret' >"$tmp/docker" printf 'owner-readonly' >"$tmp/readonly" printf 'too-open' >"$tmp/open" +printf '# comment\n\nTHT_VECTOR_READER_PASSWORD=reader\nTHT_VECTOR_WRITER_PASSWORD=writer\n' >"$tmp/bundle" +printf 'THT_VECTOR_READER_PASSWORD=reader\nTHT_VECTOR_WRITER_PASSWORD=writer\nTHT_DWH_API_KEY=one\nTHT_DWH_API_KEY=two\n' >"$tmp/duplicate-bundle" chmod 0600 "$tmp/valid" chmod 0444 "$tmp/docker" chmod 0400 "$tmp/readonly" chmod 0640 "$tmp/open" +chmod 0600 "$tmp/bundle" "$tmp/duplicate-bundle" for invalid in empty newline space; do if validate_secret_file "$tmp/$invalid" "$invalid" >/dev/null 2>&1; then @@ -36,5 +39,10 @@ if validate_secret_file "$tmp/open" open >/dev/null 2>&1; then exit 1 fi test "$(read_secret_file "$tmp/valid" valid)" = "safe-quoted-'-dollar-$" +test "$(read_bundle_secret "$tmp/bundle" THT_VECTOR_READER_PASSWORD)" = reader +if read_bundle_secret "$tmp/duplicate-bundle" THT_VECTOR_READER_PASSWORD >/dev/null 2>&1; then + echo "secret policy accepted a duplicate unrelated bundle key" >&2 + exit 1 +fi echo "shared vector secret policy contracts passed."