fix(vector): close local pgvector final review
This commit is contained in:
@@ -0,0 +1,49 @@
|
||||
# Local pgvector whole-plan final fix report
|
||||
|
||||
## Outcome
|
||||
|
||||
All four binding final-review findings are closed.
|
||||
|
||||
1. `PgVectorStore.health()` checks namespace `USAGE` independently for reader and writer
|
||||
before inspecting vector types. Real PostgreSQL tests revoke only schema `USAGE`, prove both
|
||||
health sides false and operations unavailable, then grant it back and prove recovery.
|
||||
2. Direct reader/writer passwords use workspace `password_file` references. Compose mounts the
|
||||
two files read-only into core and exposes only `_FILE` paths. Rendered Compose and live
|
||||
`docker inspect` checks prove secret contents are absent.
|
||||
3. Direct search failures map to `VectorReadUnavailable`; hash/upsert failures map to
|
||||
`VectorWriteUnavailable`. Messages are fixed and sanitized, original exceptions remain chained,
|
||||
and upsert rollback is preserved.
|
||||
4. The shared secret policy uses Linux `stat -c` with macOS `stat -f` fallback. Host files permit
|
||||
only `0600`/`0400`; Docker's read-only `0444` is accepted only beneath `/run/secrets`. Tests and
|
||||
operator docs pin this exact policy.
|
||||
|
||||
## TDD evidence
|
||||
|
||||
The new config, mode, schema-usage, unavailable-connection, and permission regressions failed
|
||||
before their implementations. The first live secret-policy run also caught GNU `stat -f` accepting
|
||||
an incompatible format invocation; detection now tries the native Linux form first. The next live
|
||||
run caught smoke-generated rotation fixtures at `0644`; fixtures now model the documented host
|
||||
policy.
|
||||
|
||||
## Verification
|
||||
|
||||
- Real direct pgvector + HTTP parity: `31 passed`.
|
||||
- Full harness from `harness/`: `493 passed, 5 deselected`.
|
||||
- Live `local-vector` rotation, restart persistence, inspect boundary, and backup/restore: pass.
|
||||
- Core image vector migration discovery/status smoke: pass.
|
||||
- External and local Compose deployment security contracts: pass.
|
||||
- Config/port focused suite: `26 passed`.
|
||||
- Secret policy, bootstrap rotation, and backup/restore safety scripts: pass.
|
||||
- Changed Python Ruff, shell syntax, and `git diff --check`: pass.
|
||||
|
||||
One attempted full-harness invocation from the repository root produced a path-dependent failure
|
||||
in an existing test that opens `workflow.yaml` relative to CWD. It was immediately rerun using the
|
||||
documented `cd harness && .venv/bin/pytest -q` command and passed completely.
|
||||
|
||||
## Operational notes
|
||||
|
||||
Workspace files contain file paths, never direct passwords. Secret contents necessarily exist in
|
||||
the in-process validated `DatabaseConfig` used to establish PostgreSQL connections, but are not
|
||||
serialized by doctor/Compose/inspect paths. Docker Desktop file-backed secrets may appear as bind
|
||||
mounts; the safe runtime exception is therefore based on the read-only service mount location
|
||||
`/run/secrets`, while source files remain owner-only on the host.
|
||||
@@ -119,8 +119,10 @@ docker compose -f compose.yaml -f deploy/compose.production.yaml \
|
||||
```
|
||||
|
||||
The secrets and public CA chain are mounted read-only under `/run/secrets` and must be readable by
|
||||
the core's UID 10001. See [`deploy/secrets/README.md`](deploy/secrets/README.md) for the verification
|
||||
command. The frontend remains on loopback; the authenticated host proxy is the only public listener.
|
||||
the core's UID 10001. Host secret files must be `0600` or `0400`; Docker's runtime `0444` mount is
|
||||
accepted only beneath `/run/secrets`. See [`deploy/secrets/README.md`](deploy/secrets/README.md) for
|
||||
the verification command. The frontend remains on loopback; the authenticated host proxy is the
|
||||
only public listener.
|
||||
|
||||
## Reproducible image verification
|
||||
|
||||
|
||||
+5
-2
@@ -20,8 +20,11 @@ services:
|
||||
THT_VECTOR_BOOTSTRAP_USER: "${THT_VECTOR_BOOTSTRAP_USER:-postgres}"
|
||||
THT_VECTOR_READER_USER: "${THT_VECTOR_READER_USER:-thoth_vector_reader}"
|
||||
THT_VECTOR_WRITER_USER: "${THT_VECTOR_WRITER_USER:-thoth_vector_writer}"
|
||||
THT_VECTOR_READER_PASSWORD: "${THT_VECTOR_READER_PASSWORD:-}"
|
||||
THT_VECTOR_WRITER_PASSWORD: "${THT_VECTOR_WRITER_PASSWORD:-}"
|
||||
THT_VECTOR_READER_PASSWORD_FILE: /run/secrets/vector_reader_password
|
||||
THT_VECTOR_WRITER_PASSWORD_FILE: /run/secrets/vector_writer_password
|
||||
secrets:
|
||||
- vector_reader_password
|
||||
- vector_writer_password
|
||||
volumes:
|
||||
- thoth_data:/data
|
||||
- ./deploy/workspaces:/app/harness/workspaces:ro
|
||||
|
||||
@@ -24,8 +24,6 @@ THT_VECTOR_BOOTSTRAP_USER=postgres
|
||||
THT_VECTOR_MIGRATOR_USER=thoth_vector_migrator
|
||||
THT_VECTOR_READER_USER=thoth_vector_reader
|
||||
THT_VECTOR_WRITER_USER=thoth_vector_writer
|
||||
THT_VECTOR_READER_PASSWORD=
|
||||
THT_VECTOR_WRITER_PASSWORD=
|
||||
THT_VECTOR_BOOTSTRAP_PASSWORD_SECRET_FILE=/absolute/path/to/vector_bootstrap_password
|
||||
# Changing the file alone does not rotate an initialized DB; use
|
||||
# scripts/vector-rotate-bootstrap-password.sh OLD_SECRET_FILE NEW_SECRET_FILE.
|
||||
|
||||
@@ -5,7 +5,9 @@ repository and point the `*_SECRET_FILE` variables documented in the root README
|
||||
|
||||
Compose mounts each file read-only beneath `/run/secrets`. The core process runs as UID 10001;
|
||||
the mounted files must be readable by that UID. Docker Compose file-backed secrets are normally
|
||||
mounted read-only with mode `0444`; verify with:
|
||||
mounted read-only with mode `0444`. This mode is accepted only for runtime paths beneath
|
||||
`/run/secrets`, where the container mount is read-only and scoped to services that declare the
|
||||
secret. Source files on the host must have no group/other bits (`0600` or `0400`). Verify with:
|
||||
|
||||
```sh
|
||||
docker compose -f compose.yaml -f deploy/compose.production.yaml \
|
||||
|
||||
@@ -11,6 +11,11 @@ validate_secret_file() {
|
||||
echo "$secret_name must contain no whitespace" >&2
|
||||
return 2
|
||||
fi
|
||||
mode=$(stat -c '%a' "$secret_path" 2>/dev/null || stat -f '%Lp' "$secret_path" 2>/dev/null) || return 2
|
||||
case "$secret_path:$mode" in
|
||||
/run/secrets/*:444|/run/secrets/*:400|/run/secrets/*:600|*:600|*:400) ;;
|
||||
*) echo "$secret_name must have mode 0600 or stricter (Docker secrets may be 0444)" >&2; return 2 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
read_secret_file() {
|
||||
|
||||
@@ -17,14 +17,14 @@ vectors:
|
||||
database: ${THT_VECTOR_DATABASE}
|
||||
schema: vectors
|
||||
user: ${THT_VECTOR_READER_USER}
|
||||
password: ${THT_VECTOR_READER_PASSWORD}
|
||||
password_file: ${THT_VECTOR_READER_PASSWORD_FILE}
|
||||
writer:
|
||||
host: vector-db
|
||||
port: 5432
|
||||
database: ${THT_VECTOR_DATABASE}
|
||||
schema: vectors
|
||||
user: ${THT_VECTOR_WRITER_USER}
|
||||
password: ${THT_VECTOR_WRITER_PASSWORD}
|
||||
password_file: ${THT_VECTOR_WRITER_PASSWORD_FILE}
|
||||
|
||||
roots:
|
||||
artifacts: artifacts
|
||||
|
||||
@@ -245,8 +245,9 @@ def test_pgvector_writer_health_requires_sequence_usage(vector_configs):
|
||||
assert health.write_detail == (
|
||||
"vector schema incomplete: missing sequence privileges evidence, memory, schema_records"
|
||||
)
|
||||
with pytest.raises(InsufficientPrivilege):
|
||||
with pytest.raises(VectorWriteUnavailable) as error:
|
||||
store.upsert("memory", [_record("needs-sequence", [1.0, 0.0])])
|
||||
assert isinstance(error.value.__cause__, InsufficientPrivilege)
|
||||
|
||||
admin_engine = create_engine(
|
||||
f"postgresql+psycopg2://{admin_config.user}:{admin_config.password}"
|
||||
@@ -262,6 +263,53 @@ def test_pgvector_writer_health_requires_sequence_usage(vector_configs):
|
||||
assert store.upsert("memory", [_record("has-sequence", [1.0, 0.0])]) == 1
|
||||
|
||||
|
||||
def test_pgvector_health_requires_schema_usage_for_reader_and_writer(vector_configs):
|
||||
from tht.adapters.vector.pgvector import PgVectorStore
|
||||
|
||||
admin_config, reader_config, writer_config, _ = vector_configs
|
||||
admin_engine = create_engine(
|
||||
f"postgresql+psycopg2://{admin_config.user}:{admin_config.password}"
|
||||
f"@{admin_config.host}:{admin_config.port}/{admin_config.database}"
|
||||
)
|
||||
store = PgVectorStore(reader_config, writer_config, expected_dimension=2)
|
||||
with admin_engine.begin() as connection:
|
||||
connection.exec_driver_sql(
|
||||
f"REVOKE USAGE ON SCHEMA vectors FROM {reader_config.user}, {writer_config.user}"
|
||||
)
|
||||
health = store.health()
|
||||
assert health.read_reachable is False and health.write_reachable is False
|
||||
assert "missing schema usage" in health.read_detail
|
||||
assert "missing schema usage" in health.write_detail
|
||||
with pytest.raises(VectorReadUnavailable, match="Vector read operation unavailable"):
|
||||
store.search(["memory"], [1.0, 0.0], limit=1)
|
||||
with pytest.raises(VectorWriteUnavailable, match="Vector write operation unavailable"):
|
||||
store.upsert("memory", [_record("blocked", [1.0, 0.0])])
|
||||
with admin_engine.begin() as connection:
|
||||
connection.exec_driver_sql(
|
||||
f"GRANT USAGE ON SCHEMA vectors TO {reader_config.user}, {writer_config.user}"
|
||||
)
|
||||
admin_engine.dispose()
|
||||
assert store.health().ok is True
|
||||
|
||||
|
||||
def test_pgvector_maps_unavailable_connections_without_leaking_password(vector_configs):
|
||||
from tht.adapters.vector.pgvector import PgVectorStore
|
||||
|
||||
_, reader_config, writer_config, _ = vector_configs
|
||||
password = "never-leak-this"
|
||||
reader = reader_config.model_copy(update={"port": 1, "password": password})
|
||||
writer = writer_config.model_copy(update={"port": 1, "password": password})
|
||||
with pytest.raises(VectorReadUnavailable) as read_error:
|
||||
PgVectorStore(reader, None).search(["memory"], [1.0, 0.0], limit=1)
|
||||
with pytest.raises(VectorWriteUnavailable) as hash_error:
|
||||
PgVectorStore(None, writer).existing_hashes("memory", ["memory"])
|
||||
with pytest.raises(VectorWriteUnavailable) as write_error:
|
||||
PgVectorStore(None, writer).upsert("memory", [_record("x", [1.0, 0.0])])
|
||||
assert password not in str(read_error.value)
|
||||
assert password not in str(hash_error.value)
|
||||
assert password not in str(write_error.value)
|
||||
|
||||
|
||||
def test_pgvector_health_reports_dimension_and_each_connection(vector_configs):
|
||||
from tht.adapters.vector.pgvector import PgVectorStore
|
||||
|
||||
|
||||
@@ -1,4 +1,7 @@
|
||||
import pytest
|
||||
|
||||
from tht.config import (
|
||||
ConfigError,
|
||||
PgvectorDirectConfig,
|
||||
PostgresDwhConfig,
|
||||
ThothRestDwhConfig,
|
||||
@@ -7,6 +10,44 @@ from tht.config import (
|
||||
)
|
||||
|
||||
|
||||
def test_direct_vector_passwords_load_from_file_references(monkeypatch, tmp_path):
|
||||
reader = tmp_path / "reader"
|
||||
writer = tmp_path / "writer"
|
||||
reader.write_text("reader-secret")
|
||||
writer.write_text("writer-secret")
|
||||
monkeypatch.setenv("READER_FILE", str(reader))
|
||||
monkeypatch.setenv("WRITER_FILE", str(writer))
|
||||
workspace = tmp_path / "workspace.yaml"
|
||||
workspace.write_text("""
|
||||
dwh:
|
||||
type: postgres_direct
|
||||
connection: {database: d, schema: public, user: u, password: p}
|
||||
vectors:
|
||||
type: pgvector_direct
|
||||
reader: {database: d, schema: vectors, user: r, password_file: '${READER_FILE}'}
|
||||
writer: {database: d, schema: vectors, user: w, password_file: '${WRITER_FILE}'}
|
||||
""")
|
||||
config = load_config(workspace)
|
||||
assert config.vectors.reader.password == "reader-secret"
|
||||
assert config.vectors.writer.password == "writer-secret"
|
||||
|
||||
|
||||
def test_direct_vector_secret_file_rejects_whitespace(tmp_path):
|
||||
secret = tmp_path / "reader"
|
||||
secret.write_text("bad secret")
|
||||
workspace = tmp_path / "workspace.yaml"
|
||||
workspace.write_text(f"""
|
||||
dwh:
|
||||
type: postgres_direct
|
||||
connection: {{database: d, schema: public, user: u, password: p}}
|
||||
vectors:
|
||||
type: pgvector_direct
|
||||
reader: {{database: d, schema: vectors, user: r, password_file: {secret}}}
|
||||
""")
|
||||
with pytest.raises(ConfigError, match="secret file"):
|
||||
load_config(workspace)
|
||||
|
||||
|
||||
def test_loads_discriminated_dwh_and_vector_resources(tmp_path):
|
||||
workspace = tmp_path / "workspace.yaml"
|
||||
workspace.write_text(
|
||||
|
||||
@@ -97,6 +97,13 @@ class PgVectorStore:
|
||||
try:
|
||||
with raw.cursor() as cursor:
|
||||
cursor.execute("SELECT 1")
|
||||
cursor.execute(
|
||||
"SELECT has_schema_privilege(current_user, %s, 'USAGE')",
|
||||
(self._schema,),
|
||||
)
|
||||
schema_usage = bool(cursor.fetchone()[0])
|
||||
if not schema_usage:
|
||||
return False, "vector schema incomplete: missing schema usage", set()
|
||||
cursor.execute(
|
||||
"""SELECT c.relname, format_type(a.atttypid, a.atttypmod),
|
||||
has_table_privilege(current_user, c.oid, 'SELECT'),
|
||||
@@ -244,8 +251,9 @@ class PgVectorStore:
|
||||
if kinds:
|
||||
_validate_known_kinds(kinds)
|
||||
hits: list[VectorHit] = []
|
||||
raw = self._reader.raw_connection()
|
||||
raw = None
|
||||
try:
|
||||
raw = self._reader.raw_connection()
|
||||
with raw.cursor() as cursor:
|
||||
for collection in collections:
|
||||
table = _collection(self._schema, collection)
|
||||
@@ -272,8 +280,13 @@ class PgVectorStore:
|
||||
params.extend((_vector_literal(embedding), limit))
|
||||
cursor.execute(query, params)
|
||||
hits.extend(hit_from_metadata(row[1], row[0]) for row in cursor.fetchall())
|
||||
except VectorStoreError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise VectorReadUnavailable("Vector read operation unavailable") from exc
|
||||
finally:
|
||||
raw.close()
|
||||
if raw is not None:
|
||||
raw.close()
|
||||
return sorted(hits, key=lambda hit: (-hit.similarity, hit.id))[:limit]
|
||||
|
||||
def _require_writer(self) -> Engine:
|
||||
@@ -285,8 +298,9 @@ class PgVectorStore:
|
||||
engine = self._require_writer()
|
||||
table = _collection(self._schema, collection)
|
||||
_validate_collection_kinds(collection, kinds)
|
||||
raw = engine.raw_connection()
|
||||
raw = None
|
||||
try:
|
||||
raw = engine.raw_connection()
|
||||
with raw.cursor() as cursor:
|
||||
cursor.execute(
|
||||
sql.SQL("SELECT record_key, content_hash FROM {} WHERE kind = ANY(%s)").format(
|
||||
@@ -295,8 +309,13 @@ class PgVectorStore:
|
||||
(kinds,),
|
||||
)
|
||||
return dict(cursor.fetchall())
|
||||
except VectorStoreError:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise VectorWriteUnavailable("Vector write operation unavailable") from exc
|
||||
finally:
|
||||
raw.close()
|
||||
if raw is not None:
|
||||
raw.close()
|
||||
|
||||
def upsert(self, collection: str, records: list[VectorWriteRecord]) -> int:
|
||||
engine = self._require_writer()
|
||||
@@ -317,8 +336,9 @@ class PgVectorStore:
|
||||
"UPDATE {} SET kind = %s, content_hash = %s, metadata = %s::jsonb, "
|
||||
"embedding = %s::{}, indexed_at = pg_catalog.now() WHERE record_key = %s"
|
||||
).format(table, _vector_type(self._schema))
|
||||
raw = engine.raw_connection()
|
||||
raw = None
|
||||
try:
|
||||
raw = engine.raw_connection()
|
||||
with raw.cursor() as cursor:
|
||||
for write_record in records:
|
||||
record = write_record.record
|
||||
@@ -348,11 +368,17 @@ class PgVectorStore:
|
||||
),
|
||||
)
|
||||
raw.commit()
|
||||
except Exception:
|
||||
raw.rollback()
|
||||
except VectorStoreError:
|
||||
if raw is not None:
|
||||
raw.rollback()
|
||||
raise
|
||||
except Exception as exc:
|
||||
if raw is not None:
|
||||
raw.rollback()
|
||||
raise VectorWriteUnavailable("Vector write operation unavailable") from exc
|
||||
finally:
|
||||
raw.close()
|
||||
if raw is not None:
|
||||
raw.close()
|
||||
return len(records)
|
||||
|
||||
|
||||
|
||||
+21
-1
@@ -36,6 +36,26 @@ def _expand_env(value: Any) -> Any:
|
||||
return value
|
||||
|
||||
|
||||
def _resolve_secret_files(value: Any) -> Any:
|
||||
if isinstance(value, dict):
|
||||
resolved = {key: _resolve_secret_files(item) for key, item in value.items()}
|
||||
if "password_file" in resolved:
|
||||
if "password" in resolved:
|
||||
raise ConfigError("password and password_file are mutually exclusive")
|
||||
path = Path(resolved.pop("password_file"))
|
||||
try:
|
||||
secret = path.read_text()
|
||||
except (OSError, UnicodeError) as exc:
|
||||
raise ConfigError(f"Cannot read secret file: {path}") from exc
|
||||
if not secret or any(char.isspace() for char in secret) or "\x00" in secret:
|
||||
raise ConfigError(f"Invalid secret file: {path}")
|
||||
resolved["password"] = secret
|
||||
return resolved
|
||||
if isinstance(value, list):
|
||||
return [_resolve_secret_files(item) for item in value]
|
||||
return value
|
||||
|
||||
|
||||
class DatabaseConfig(BaseModel):
|
||||
host: str = "localhost"
|
||||
port: int = 5432
|
||||
@@ -254,7 +274,7 @@ def load_config(path: Path) -> Config:
|
||||
raw = yaml.safe_load(path.read_text())
|
||||
if not isinstance(raw, dict):
|
||||
raise ConfigError(f"Configurazione non valida (atteso un mapping YAML): {path}")
|
||||
expanded = _expand_env(raw)
|
||||
expanded = _resolve_secret_files(_expand_env(raw))
|
||||
translated, used_legacy = translate_legacy_config(expanded)
|
||||
_populate_legacy_views(translated)
|
||||
try:
|
||||
|
||||
@@ -32,8 +32,6 @@ export THT_VECTOR_BOOTSTRAP_PASSWORD_SECRET_FILE="$secret_dir/bootstrap"
|
||||
export THT_VECTOR_MIGRATOR_PASSWORD_SECRET_FILE="$secret_dir/migrator"
|
||||
export THT_VECTOR_READER_PASSWORD_SECRET_FILE="$secret_dir/reader"
|
||||
export THT_VECTOR_WRITER_PASSWORD_SECRET_FILE="$secret_dir/writer"
|
||||
export THT_VECTOR_READER_PASSWORD="smoke-reader-${smoke_project}"
|
||||
export THT_VECTOR_WRITER_PASSWORD="smoke-writer-${smoke_project}"
|
||||
export THT_VECTOR_BOOTSTRAP_USER=thoth_bootstrap_smoke
|
||||
export THOTH_SMOKE_OWNER="$smoke_owner"
|
||||
|
||||
@@ -132,7 +130,7 @@ def credential(role: str) -> DatabaseConfig:
|
||||
database=database,
|
||||
schema="vectors",
|
||||
user=f"thoth_vector_{role}",
|
||||
password=os.environ[f"THT_VECTOR_{role.upper()}_PASSWORD"],
|
||||
password=open(os.environ[f"THT_VECTOR_{role.upper()}_PASSWORD_FILE"]).read(),
|
||||
)
|
||||
|
||||
store = PgVectorStore(credential("reader"), credential("writer"), expected_dimension=768)
|
||||
@@ -168,6 +166,14 @@ printf '%s\n' "$services" | grep -qx vector-reconcile
|
||||
printf '%s\n' "$services" | grep -qx vector-migrate
|
||||
|
||||
compose up --build --wait vector-reconcile vector-migrate core
|
||||
core_id=$(compose ps -q core)
|
||||
inspect_env=$(docker inspect --format '{{json .Config.Env}}' "$core_id")
|
||||
if printf '%s' "$inspect_env" | grep -q "smoke-\(reader\|writer\)-${smoke_project}"; then
|
||||
echo "docker inspect exposed a direct vector password" >&2
|
||||
exit 1
|
||||
fi
|
||||
printf '%s' "$inspect_env" | grep -q 'THT_VECTOR_READER_PASSWORD_FILE=/run/secrets/vector_reader_password'
|
||||
printf '%s' "$inspect_env" | grep -q 'THT_VECTOR_WRITER_PASSWORD_FILE=/run/secrets/vector_writer_password'
|
||||
migration_status=$(compose run --rm --no-deps vector-migrate)
|
||||
printf '%s\n' "$migration_status" | grep -q '"pending": \[\]'
|
||||
migrator_flags=$(compose run --rm --no-deps --entrypoint sh vector-reconcile -ec '
|
||||
@@ -178,13 +184,11 @@ migrator_flags=$(compose run --rm --no-deps --entrypoint sh vector-reconcile -ec
|
||||
test "$migrator_flags" = t
|
||||
probe_vector write
|
||||
|
||||
old_reader_password=$THT_VECTOR_READER_PASSWORD
|
||||
old_reader_password=$(cat "$secret_dir/reader")
|
||||
for secret in migrator reader writer; do
|
||||
password="rotated-${secret}-${smoke_project}"
|
||||
printf '%s' "$password" >"$secret_dir/$secret"
|
||||
done
|
||||
export THT_VECTOR_READER_PASSWORD="rotated-reader-${smoke_project}"
|
||||
export THT_VECTOR_WRITER_PASSWORD="rotated-writer-${smoke_project}"
|
||||
|
||||
compose run --rm vector-reconcile
|
||||
rotation_status=$(compose run --rm --no-deps vector-migrate)
|
||||
@@ -205,6 +209,8 @@ printf '%s' "wrong-bootstrap-${smoke_project}" >"$secret_dir/bootstrap-wrong"
|
||||
printf '%s' "next-bootstrap-'quoted-${smoke_project}" >"$secret_dir/bootstrap-next"
|
||||
cp "$secret_dir/bootstrap" "$secret_dir/bootstrap-before-negative"
|
||||
printf 'invalid bootstrap password\n' >"$secret_dir/bootstrap-whitespace"
|
||||
chmod 0600 "$secret_dir/bootstrap-wrong" "$secret_dir/bootstrap-next" \
|
||||
"$secret_dir/bootstrap-before-negative" "$secret_dir/bootstrap-whitespace"
|
||||
if COMPOSE_PROJECT_NAME="$smoke_project" \
|
||||
./scripts/vector-rotate-bootstrap-password.sh \
|
||||
"$secret_dir/bootstrap" "$secret_dir/bootstrap-whitespace" \
|
||||
@@ -344,8 +350,10 @@ if [ "$mode" = "--backup-restore" ]; then
|
||||
--entrypoint /opt/thoth/reconcile-roles.sh "$image" >/dev/null
|
||||
|
||||
docker run --rm -i --network "$network" \
|
||||
-e THT_VECTOR_READER_PASSWORD="$THT_VECTOR_READER_PASSWORD" \
|
||||
-e THT_VECTOR_WRITER_PASSWORD="$THT_VECTOR_WRITER_PASSWORD" \
|
||||
--mount "type=bind,source=$secret_dir/reader,target=/run/secrets/vector_reader_password,readonly" \
|
||||
--mount "type=bind,source=$secret_dir/writer,target=/run/secrets/vector_writer_password,readonly" \
|
||||
-e THT_VECTOR_READER_PASSWORD_FILE=/run/secrets/vector_reader_password \
|
||||
-e THT_VECTOR_WRITER_PASSWORD_FILE=/run/secrets/vector_writer_password \
|
||||
--entrypoint /opt/venv/bin/python thothii-core:local - "$marker" <<'PY'
|
||||
import hashlib
|
||||
import os
|
||||
@@ -359,7 +367,8 @@ from tht.vectorstore.records import VectorRecord
|
||||
def config(role):
|
||||
return DatabaseConfig(
|
||||
host="vector-db-restore", port=5432, database="thoth", schema="vectors",
|
||||
user=f"thoth_vector_{role}", password=os.environ[f"THT_VECTOR_{role.upper()}_PASSWORD"],
|
||||
user=f"thoth_vector_{role}",
|
||||
password=open(os.environ[f"THT_VECTOR_{role.upper()}_PASSWORD_FILE"]).read(),
|
||||
)
|
||||
|
||||
store = PgVectorStore(config("reader"), config("writer"), expected_dimension=768)
|
||||
|
||||
@@ -15,6 +15,22 @@ if grep -q 'env_file:' "$tmp/base.yaml"; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
for secret in bootstrap migrator local_reader local_writer; do
|
||||
printf '%s' "contract-$secret" >"$tmp/$secret"
|
||||
chmod 0600 "$tmp/$secret"
|
||||
done
|
||||
THT_VECTOR_BOOTSTRAP_PASSWORD_SECRET_FILE="$tmp/bootstrap" \
|
||||
THT_VECTOR_MIGRATOR_PASSWORD_SECRET_FILE="$tmp/migrator" \
|
||||
THT_VECTOR_READER_PASSWORD_SECRET_FILE="$tmp/local_reader" \
|
||||
THT_VECTOR_WRITER_PASSWORD_SECRET_FILE="$tmp/local_writer" \
|
||||
docker compose --profile local-vector config >"$tmp/local-vector.yaml"
|
||||
grep -q 'THT_VECTOR_READER_PASSWORD_FILE: /run/secrets/vector_reader_password' "$tmp/local-vector.yaml"
|
||||
grep -q 'THT_VECTOR_WRITER_PASSWORD_FILE: /run/secrets/vector_writer_password' "$tmp/local-vector.yaml"
|
||||
if grep -q 'contract-local_' "$tmp/local-vector.yaml"; then
|
||||
echo "rendered local-vector config leaked a direct database secret" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
docker compose -f compose.yaml -f deploy/compose.local.yaml \
|
||||
--profile external config >"$tmp/local.yaml"
|
||||
grep -q 'env_file:' deploy/compose.local.yaml
|
||||
|
||||
@@ -7,6 +7,7 @@ trap 'rm -rf "$tmp"' EXIT HUP INT TERM
|
||||
fakebin="$tmp/bin"
|
||||
mkdir "$fakebin"
|
||||
printf '%s' secret >"$tmp/password"
|
||||
chmod 0600 "$tmp/password"
|
||||
|
||||
cat >"$fakebin/pg_dump" <<'SH'
|
||||
#!/bin/sh
|
||||
|
||||
@@ -20,6 +20,7 @@ printf '%s' "new-'quoted-\$-password" >"$tmp/new"
|
||||
cp "$tmp/old" "$tmp/original"
|
||||
|
||||
printf 'invalid password\n' >"$tmp/whitespace"
|
||||
chmod 0600 "$tmp/old" "$tmp/new" "$tmp/original" "$tmp/whitespace"
|
||||
: >"$log"
|
||||
if PATH="$tmp:$PATH" FAKE_DOCKER_LOG="$log" THT_VECTOR_BOOTSTRAP_USER=custom_admin \
|
||||
./scripts/vector-rotate-bootstrap-password.sh "$tmp/old" "$tmp/whitespace" \
|
||||
|
||||
@@ -11,6 +11,13 @@ trap 'rm -rf "$tmp"' EXIT HUP INT TERM
|
||||
printf 'has newline\n' >"$tmp/newline"
|
||||
printf 'has space' >"$tmp/space"
|
||||
printf 'safe-quoted-\047-dollar-$' >"$tmp/valid"
|
||||
printf 'docker-secret' >"$tmp/docker"
|
||||
printf 'owner-readonly' >"$tmp/readonly"
|
||||
printf 'too-open' >"$tmp/open"
|
||||
chmod 0600 "$tmp/valid"
|
||||
chmod 0444 "$tmp/docker"
|
||||
chmod 0400 "$tmp/readonly"
|
||||
chmod 0640 "$tmp/open"
|
||||
|
||||
for invalid in empty newline space; do
|
||||
if validate_secret_file "$tmp/$invalid" "$invalid" >/dev/null 2>&1; then
|
||||
@@ -19,6 +26,15 @@ for invalid in empty newline space; do
|
||||
fi
|
||||
done
|
||||
validate_secret_file "$tmp/valid" valid
|
||||
validate_secret_file "$tmp/readonly" readonly
|
||||
if validate_secret_file "$tmp/docker" docker >/dev/null 2>&1; then
|
||||
echo "secret policy accepted world-readable host secret" >&2
|
||||
exit 1
|
||||
fi
|
||||
if validate_secret_file "$tmp/open" open >/dev/null 2>&1; then
|
||||
echo "secret policy accepted group-readable host secret" >&2
|
||||
exit 1
|
||||
fi
|
||||
test "$(read_secret_file "$tmp/valid" valid)" = "safe-quoted-'-dollar-$"
|
||||
|
||||
echo "shared vector secret policy contracts passed."
|
||||
|
||||
Reference in New Issue
Block a user