134 lines
7.5 KiB
Markdown
134 lines
7.5 KiB
Markdown
# Task 3 report — optional local pgvector profile
|
|
|
|
## Status
|
|
|
|
Implemented and verified the `local-vector` Compose profile.
|
|
|
|
- `vector-db` uses pgvector 0.8.5 on PostgreSQL 16, pinned to the official multi-arch
|
|
manifest digest.
|
|
- `vector_data` is a project-scoped named volume and is not shared with application data.
|
|
- database readiness gates the packaged one-shot `vector-migrate` job; core declares the
|
|
migration completion dependency while remaining usable in the pre-existing external profile.
|
|
- bootstrap, migrator, reader, and writer identities are distinct. Bootstrap and migration
|
|
credentials are supplied as Compose secrets; the application receives only reader/writer
|
|
credentials.
|
|
- `deploy/workspaces/local-vector.yaml` selects `pgvector_direct` with separate reader and
|
|
writer connections.
|
|
- the base loopback port binding, `AUTH_MODE=none`, and `THOTH_PUBLIC_EXPOSURE=false` defaults
|
|
are unchanged.
|
|
|
|
## Red/green evidence
|
|
|
|
The initial Compose contract did not list `vector-db`, as required by the brief. The first real
|
|
smoke then failed migration 002 because bootstrap installed the vector extension in `public`.
|
|
The bootstrap was corrected to create the `vectors` schema under the migration owner and install
|
|
the extension there. A clean-volume rerun passed.
|
|
|
|
## Verification
|
|
|
|
- `./scripts/local-vector-smoke.sh`: PASS
|
|
- isolated generated Compose project and credentials
|
|
- clean migration plus idempotent status rerun
|
|
- reader/writer privilege health
|
|
- one-record upsert and similarity search
|
|
- restart of both `core` and `vector-db`
|
|
- persisted search result after restart
|
|
- project-only volume cleanup
|
|
- `./scripts/test-container-deployment.sh`: PASS
|
|
- `./scripts/test-backend-url-policy.sh`: PASS
|
|
- `docker compose --profile local-vector config --quiet`: PASS
|
|
- harness: 477 passed, 5 deselected
|
|
- backend: 84 passed; TypeScript typecheck PASS
|
|
- frontend: 226 passed; TypeScript typecheck PASS
|
|
- `git diff --check`: PASS
|
|
|
|
## Self-review / concerns
|
|
|
|
- Compose cannot make a dependency required only under one profile. The core dependency uses
|
|
`required: false` so the established `external` profile does not activate local infrastructure;
|
|
under `local-vector`, `compose up --wait` still fails if `vector-migrate` exits nonzero, and the
|
|
smoke verifies that successful migration precedes the healthy stack.
|
|
- Reader/writer passwords are injected into core environment variables because Compose service
|
|
attributes cannot be conditional by profile. Bootstrap and migrator credentials remain
|
|
file-backed secrets and are never exposed to core.
|
|
- The smoke intentionally refuses the operator project name `thothii` and removes only its unique
|
|
project namespace and volumes.
|
|
|
|
## Follow-up hardening — credential reconciliation and cleanup ownership
|
|
|
|
Review findings were resolved in a separate follow-up:
|
|
|
|
- Replaced fresh-volume-only initialization with `vector-reconcile`, an idempotent one-shot that
|
|
runs after database health and before `vector-migrate`. It authenticates with only the bootstrap
|
|
admin secret, safely creates missing identities, reconciles role attributes and passwords on
|
|
existing volumes, restores memberships/ownership, and leaves vector data untouched.
|
|
- The migrator is explicitly `NOSUPERUSER NOCREATEDB NOCREATEROLE`. Schema/database ownership is
|
|
sufficient for all packaged migrations because reconciliation creates the two group roles first.
|
|
- The live smoke rotates migrator, reader, and writer secrets on the same populated volume, rejects
|
|
the old reader credential, reruns migrations, recreates core with the new runtime credentials,
|
|
and retrieves the record written before rotation and again after database/core restart.
|
|
- Smoke project names are no longer caller-controlled. Each run creates a unique namespace and
|
|
ownership token. Containers, networks, and volumes carry the ownership label; preflight refuses
|
|
any collision and cleanup verifies every discovered resource before `down --volumes`.
|
|
- Added a dynamic fake-Docker contract suite for caller override, collision, and mismatched cleanup
|
|
labels, plus a real-Docker collision probe using a unique labeled volume.
|
|
|
|
Follow-up verification:
|
|
|
|
- `./scripts/local-vector-smoke.sh`: PASS, including live secret rotation and persisted retrieval
|
|
- `./scripts/test-local-vector-smoke-safety.sh`: PASS
|
|
- `./scripts/test-local-vector-smoke-live-collision.sh`: PASS
|
|
- harness: 477 passed, 5 deselected
|
|
- backend: 84 passed; TypeScript typecheck PASS
|
|
- frontend: 226 passed; TypeScript typecheck PASS
|
|
- Compose security, backend URL, config, shell syntax, and diff checks: PASS
|
|
|
|
Remaining operational constraint: the bootstrap admin secret must continue to match the PostgreSQL
|
|
bootstrap account stored in the volume. Runtime migrator/reader/writer rotation is supported without
|
|
data deletion; bootstrap-account password rotation is a distinct database-administration operation.
|
|
|
|
## Final hardening — bootstrap account rotation
|
|
|
|
The remaining operational constraint is now covered by
|
|
`scripts/vector-rotate-bootstrap-password.sh OLD_SECRET_FILE NEW_SECRET_FILE`:
|
|
|
|
- It does not rely on `POSTGRES_PASSWORD_FILE` after initialization.
|
|
- It pre-stages the deployment-file replacement in the same directory, authenticates to the live
|
|
database with the explicit old file, and changes only the authenticated bootstrap role.
|
|
- Passwords are passed as connection parameters and rendered with psycopg2 SQL composition, so
|
|
shell and SQL metacharacters are not interpolated.
|
|
- A second connection must authenticate with the new password before the command succeeds. If that
|
|
verification fails, the still-open old connection restores the old database password.
|
|
- Only after verified database login does an atomic rename replace the current deployment secret.
|
|
Wrong-old authentication and verification failures leave deployment configuration unchanged.
|
|
|
|
Final live smoke evidence on one existing `vector_data` volume:
|
|
|
|
- wrong-old bootstrap rotation rejected; current deployment secret unchanged
|
|
- bootstrap password with quote characters rotated successfully
|
|
- old bootstrap login rejected and new login accepted
|
|
- `vector-reconcile`, packaged migrations, and core health passed afterward
|
|
- the vector record written before rotation remained searchable after rotation and after a further
|
|
database/core restart
|
|
|
|
Final tests:
|
|
|
|
- `./scripts/test-vector-bootstrap-rotation.sh`: PASS
|
|
- `./scripts/local-vector-smoke.sh`: PASS with negative and positive live bootstrap rotation
|
|
- existing local-vector collision/safety and Compose deployment contracts: PASS
|
|
|
|
## Final identity and secret-policy alignment
|
|
|
|
- `THT_VECTOR_BOOTSTRAP_USER` is now passed through core as well as vector-db and reconciliation,
|
|
so the rotation helper uses the authoritative configured role instead of defaulting to `postgres`.
|
|
- Rotation and reconciliation source the same raw-file `secret-policy.sh`: non-empty and no
|
|
whitespace, including trailing newlines. Rotation validates both files before Docker,
|
|
PostgreSQL, or atomic replacement staging; `test-vector-secret-policy.sh` pins empty, newline,
|
|
internal-space, and valid metacharacter cases.
|
|
- Fake-Docker tests prove a non-default identity reaches the helper path and whitespace rejection
|
|
performs no Docker call and creates no staged replacement.
|
|
- The real smoke runs the entire stack as `thoth_bootstrap_smoke`. Its whitespace-negative case
|
|
leaves the deployment file unchanged and proves the existing database login still succeeds;
|
|
non-default-account bootstrap rotation, reconciliation, migration, core health, restart, and
|
|
persisted retrieval all pass.
|