refactor: retire external vector deployment
This commit is contained in:
@@ -10,7 +10,4 @@ THT_WORKSPACE_GIT_AUTHOR_EMAIL=thoth-workspace-registry@example.invalid
|
|||||||
|
|
||||||
THT_DB_NAME=warehouse
|
THT_DB_NAME=warehouse
|
||||||
THT_DWH_REST_URL=https://dwh.example.invalid
|
THT_DWH_REST_URL=https://dwh.example.invalid
|
||||||
THT_VEC_REST_URL=https://vector.example.invalid
|
|
||||||
THT_VEC_WRITE_REST_URL=https://vector-write.example.invalid
|
|
||||||
THT_OLLAMA_URL=https://embeddings.example.invalid
|
|
||||||
THT_LLM_URL=https://llm.example.invalid
|
THT_LLM_URL=https://llm.example.invalid
|
||||||
|
|||||||
@@ -41,8 +41,10 @@ jobs:
|
|||||||
- name: Verify Compose and installation contracts
|
- name: Verify Compose and installation contracts
|
||||||
run: |
|
run: |
|
||||||
bash scripts/test-unified-compose.sh
|
bash scripts/test-unified-compose.sh
|
||||||
|
bash scripts/test-no-deployment-coupling-scope.sh
|
||||||
bash scripts/test-compose-secret-policy.sh
|
bash scripts/test-compose-secret-policy.sh
|
||||||
bash scripts/test-no-deployment-coupling.sh
|
bash scripts/test-no-deployment-coupling.sh
|
||||||
|
bash scripts/test-preprocess-compose-config.sh
|
||||||
bash scripts/test-verify-workspace-install-docs.sh
|
bash scripts/test-verify-workspace-install-docs.sh
|
||||||
git diff --check
|
git diff --check
|
||||||
- name: Install backend dependencies
|
- name: Install backend dependencies
|
||||||
|
|||||||
@@ -2,12 +2,12 @@
|
|||||||
|
|
||||||
ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/`tht`
|
ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/`tht`
|
||||||
core. The portable deployment runs exactly two application services; data services remain
|
core. The portable deployment runs exactly two application services; data services remain
|
||||||
external in this profile.
|
external in this profile, except for the mandatory internal semantic services bundled in Compose.
|
||||||
|
|
||||||
## Docker Compose: local startup
|
## Docker Compose: local startup
|
||||||
|
|
||||||
Requirements: Docker Engine with Compose v2. The mandatory stack is exactly the `core` and
|
Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`,
|
||||||
`frontend` application images. DWH, vector DB, embedding, and LLM services are external,
|
`qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external,
|
||||||
configurable endpoints—even when they are co-located with ThothII.
|
configurable endpoints—even when they are co-located with ThothII.
|
||||||
|
|
||||||
From a fresh clone, run these commands from the repository root:
|
From a fresh clone, run these commands from the repository root:
|
||||||
@@ -42,9 +42,7 @@ runtime endpoint and secret bindings remain installation-local. Open
|
|||||||
loopback port).
|
loopback port).
|
||||||
|
|
||||||
Credentials and certificates are local protected files. Do not put them in environment examples,
|
Credentials and certificates are local protected files. Do not put them in environment examples,
|
||||||
workspace YAML, URLs, or Compose interpolation values. The optional `local-vector` and
|
workspace YAML, URLs, or Compose interpolation values.
|
||||||
preprocessing overlays are development presets; they do not change the two-service mandatory
|
|
||||||
stack or the external-endpoint contract.
|
|
||||||
|
|
||||||
Application state is split across the named `settings`, `pi-state`, `workspace-registry`, and
|
Application state is split across the named `settings`, `pi-state`, `workspace-registry`, and
|
||||||
`sessions` volumes. `docker compose down` keeps them. Only an explicit destructive command such
|
`sessions` volumes. `docker compose down` keeps them. Only an explicit destructive command such
|
||||||
@@ -172,29 +170,20 @@ Startup mode adds bounded image build/two-service health startup, installation-a
|
|||||||
status, stopped-container-aware ownership checks, and exact cleanup. The ordinary hosted Windows
|
status, stopped-container-aware ownership checks, and exact cleanup. The ordinary hosted Windows
|
||||||
job remains deterministic and does not claim Docker startup.
|
job remains deterministic and does not claim Docker startup.
|
||||||
|
|
||||||
## Optional local pgvector and recovery
|
|
||||||
|
|
||||||
The local-vector overlay reads `THT_VECTOR_BOOTSTRAP_PASSWORD`,
|
|
||||||
`THT_VECTOR_MIGRATOR_PASSWORD`, `THT_VECTOR_READER_PASSWORD`, and
|
|
||||||
`THT_VECTOR_WRITER_PASSWORD` from the same bundle. Its `vector_data` volume is independent of
|
|
||||||
application state; passwords are selected at runtime and are never passed as URL arguments.
|
|
||||||
|
|
||||||
## Preprocessing jobs and S3 Evidence
|
## Preprocessing jobs and S3 Evidence
|
||||||
|
|
||||||
The included job workspaces target the optional local-vector profile. Put the four local-vector
|
The included preprocessing services reuse the internal Qdrant/Ollama stack. Mount Evidence at
|
||||||
password keys in the bundle, set `THT_OLLAMA_URL`, mount Evidence at `/data/source/evidence`, then
|
`/data/source/evidence`, then run the explicit preprocessing preset:
|
||||||
run the explicit preprocessing preset:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
docker compose --env-file deploy/env/local.env \
|
docker compose --env-file deploy/env/local.env \
|
||||||
-f compose.yaml -f deploy/compose.local.yaml -f deploy/compose.local-vector.yaml \
|
-f compose.yaml -f deploy/compose.local.yaml \
|
||||||
-f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml \
|
-f deploy/compose.preprocess.yaml --profile preprocess run --rm preprocess-evidence
|
||||||
--profile local-vector --profile preprocess run --rm preprocess-evidence
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Replace the final service with `preprocess-dwh` when required. The overlay makes each job wait for the vector
|
Replace the final service with `preprocess-dwh` when required. The overlay makes each job wait for the vector
|
||||||
database health check, role reconciliation, and a successful migration; no separate database
|
service health checks and embedding model initialization; no separate semantic-service startup is
|
||||||
startup or migration command is required.
|
required.
|
||||||
|
|
||||||
S3 Evidence uses the optional `tht[s3]` dependency and canonical `s3://bucket/key` provenance.
|
S3 Evidence uses the optional `tht[s3]` dependency and canonical `s3://bucket/key` provenance.
|
||||||
AWS endpoints are used when no custom URL is supplied. Every custom endpoint is an explicit egress
|
AWS endpoints are used when no custom URL is supplied. Every custom endpoint is an explicit egress
|
||||||
@@ -235,9 +224,7 @@ database in the active cluster cannot bypass the guard. It refuses a non-empty t
|
|||||||
```
|
```
|
||||||
|
|
||||||
After restore, run `tht vector migrate --status --json`, adapter health, and a known retrieval
|
After restore, run `tht vector migrate --status --json`, adapter health, and a known retrieval
|
||||||
query against the target before changing any deployment endpoint. Never test recovery against the
|
query against the target before changing any migration/export endpoint.
|
||||||
active `vector_data` volume. `./scripts/local-vector-smoke.sh --backup-restore` performs this drill
|
|
||||||
with disposable source and target volumes.
|
|
||||||
|
|
||||||
## Production trust boundary and secrets
|
## Production trust boundary and secrets
|
||||||
|
|
||||||
|
|||||||
@@ -1,90 +0,0 @@
|
|||||||
services:
|
|
||||||
core:
|
|
||||||
profiles: [local-vector]
|
|
||||||
environment:
|
|
||||||
THT_VECTOR_DATABASE: "${THT_VECTOR_DATABASE:-thoth}"
|
|
||||||
THT_VECTOR_BOOTSTRAP_USER: "${THT_VECTOR_BOOTSTRAP_USER:-postgres}"
|
|
||||||
THT_VECTOR_READER_USER: "${THT_VECTOR_READER_USER:-thoth_vector_reader}"
|
|
||||||
THT_VECTOR_WRITER_USER: "${THT_VECTOR_WRITER_USER:-thoth_vector_writer}"
|
|
||||||
THT_SECRETS_FILE: /run/secrets/thothii.secrets
|
|
||||||
secrets: [{source: thothii_secrets, target: thothii.secrets}]
|
|
||||||
depends_on:
|
|
||||||
vector-migrate:
|
|
||||||
condition: service_completed_successfully
|
|
||||||
|
|
||||||
frontend:
|
|
||||||
profiles: [local-vector]
|
|
||||||
|
|
||||||
vector-db:
|
|
||||||
image: pgvector/pgvector:0.8.5-pg16@sha256:1d533553fefe4f12e5d80c7b80622ba0c382abb5758856f52983d8789179f0fb
|
|
||||||
profiles: [local-vector]
|
|
||||||
labels: {io.thothii.smoke-owner: "${THOTH_SMOKE_OWNER:-operator}"}
|
|
||||||
environment:
|
|
||||||
POSTGRES_DB: "${THT_VECTOR_DATABASE:-thoth}"
|
|
||||||
POSTGRES_USER: "${THT_VECTOR_BOOTSTRAP_USER:-postgres}"
|
|
||||||
THT_VECTOR_MIGRATOR_USER: "${THT_VECTOR_MIGRATOR_USER:-thoth_vector_migrator}"
|
|
||||||
THT_VECTOR_READER_USER: "${THT_VECTOR_READER_USER:-thoth_vector_reader}"
|
|
||||||
THT_VECTOR_WRITER_USER: "${THT_VECTOR_WRITER_USER:-thoth_vector_writer}"
|
|
||||||
THT_SECRETS_FILE: /run/secrets/thothii.secrets
|
|
||||||
secrets: [{source: thothii_secrets, target: thothii.secrets}]
|
|
||||||
entrypoint: [/opt/thoth/vector-db-entrypoint.sh]
|
|
||||||
volumes:
|
|
||||||
- vector_data:/var/lib/postgresql/data
|
|
||||||
- ./deploy/vector/vector-db-entrypoint.sh:/opt/thoth/vector-db-entrypoint.sh:ro
|
|
||||||
- ./deploy/vector/secret-policy.sh:/opt/thoth/secret-policy.sh:ro
|
|
||||||
healthcheck:
|
|
||||||
test: [CMD-SHELL, "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
|
|
||||||
interval: 5s
|
|
||||||
timeout: 3s
|
|
||||||
retries: 20
|
|
||||||
start_period: 10s
|
|
||||||
restart: unless-stopped
|
|
||||||
|
|
||||||
vector-reconcile:
|
|
||||||
image: pgvector/pgvector:0.8.5-pg16@sha256:1d533553fefe4f12e5d80c7b80622ba0c382abb5758856f52983d8789179f0fb
|
|
||||||
profiles: [local-vector]
|
|
||||||
labels: {io.thothii.smoke-owner: "${THOTH_SMOKE_OWNER:-operator}"}
|
|
||||||
environment:
|
|
||||||
PGHOST: vector-db
|
|
||||||
PGPORT: 5432
|
|
||||||
PGDATABASE: "${THT_VECTOR_DATABASE:-thoth}"
|
|
||||||
PGUSER: "${THT_VECTOR_BOOTSTRAP_USER:-postgres}"
|
|
||||||
THT_VECTOR_BOOTSTRAP_USER: "${THT_VECTOR_BOOTSTRAP_USER:-postgres}"
|
|
||||||
THT_VECTOR_MIGRATOR_USER: "${THT_VECTOR_MIGRATOR_USER:-thoth_vector_migrator}"
|
|
||||||
THT_VECTOR_READER_USER: "${THT_VECTOR_READER_USER:-thoth_vector_reader}"
|
|
||||||
THT_VECTOR_WRITER_USER: "${THT_VECTOR_WRITER_USER:-thoth_vector_writer}"
|
|
||||||
THT_SECRETS_FILE: /run/secrets/thothii.secrets
|
|
||||||
entrypoint: [/opt/thoth/reconcile-roles.sh]
|
|
||||||
secrets: [{source: thothii_secrets, target: thothii.secrets}]
|
|
||||||
volumes:
|
|
||||||
- ./deploy/vector/reconcile-roles.sh:/opt/thoth/reconcile-roles.sh:ro
|
|
||||||
- ./deploy/vector/secret-policy.sh:/opt/thoth/secret-policy.sh:ro
|
|
||||||
depends_on:
|
|
||||||
vector-db: {condition: service_healthy}
|
|
||||||
restart: "no"
|
|
||||||
|
|
||||||
vector-migrate:
|
|
||||||
image: thothii-core:local
|
|
||||||
profiles: [local-vector]
|
|
||||||
labels: {io.thothii.smoke-owner: "${THOTH_SMOKE_OWNER:-operator}"}
|
|
||||||
build:
|
|
||||||
context: .
|
|
||||||
dockerfile: docker/core.Dockerfile
|
|
||||||
entrypoint: [sh, -ec]
|
|
||||||
command:
|
|
||||||
- |
|
|
||||||
. /opt/thoth/secret-policy.sh
|
|
||||||
export PGPASSWORD=$$(read_bundle_secret /run/secrets/thothii.secrets THT_VECTOR_MIGRATOR_PASSWORD)
|
|
||||||
exec /opt/venv/bin/tht vector migrate --database-url "postgresql+psycopg2://${THT_VECTOR_MIGRATOR_USER:-thoth_vector_migrator}@vector-db:5432/${THT_VECTOR_DATABASE:-thoth}" --json
|
|
||||||
secrets: [{source: thothii_secrets, target: thothii.secrets}]
|
|
||||||
environment:
|
|
||||||
THT_SECRETS_FILE: /run/secrets/thothii.secrets
|
|
||||||
volumes:
|
|
||||||
- ./deploy/vector/secret-policy.sh:/opt/thoth/secret-policy.sh:ro
|
|
||||||
depends_on:
|
|
||||||
vector-reconcile: {condition: service_completed_successfully}
|
|
||||||
restart: "no"
|
|
||||||
|
|
||||||
volumes:
|
|
||||||
vector_data:
|
|
||||||
labels: {io.thothii.smoke-owner: "${THOTH_SMOKE_OWNER:-operator}"}
|
|
||||||
@@ -1,14 +0,0 @@
|
|||||||
services:
|
|
||||||
preprocess-evidence:
|
|
||||||
environment:
|
|
||||||
THT_SECRETS_FILE: /run/secrets/thothii.secrets
|
|
||||||
secrets: [{source: thothii_secrets, target: thothii.secrets}]
|
|
||||||
depends_on:
|
|
||||||
vector-migrate: {condition: service_completed_successfully}
|
|
||||||
|
|
||||||
preprocess-dwh:
|
|
||||||
environment:
|
|
||||||
THT_SECRETS_FILE: /run/secrets/thothii.secrets
|
|
||||||
secrets: [{source: thothii_secrets, target: thothii.secrets}]
|
|
||||||
depends_on:
|
|
||||||
vector-migrate: {condition: service_completed_successfully}
|
|
||||||
@@ -9,13 +9,17 @@ services:
|
|||||||
command: ["mkdir -p /data/workspaces/preprocess-evidence && exec /app/docker/core-entrypoint.sh preprocess evidence --json -c /app/harness/workspaces/preprocess-evidence.yaml"]
|
command: ["mkdir -p /data/workspaces/preprocess-evidence && exec /app/docker/core-entrypoint.sh preprocess evidence --json -c /app/harness/workspaces/preprocess-evidence.yaml"]
|
||||||
environment:
|
environment:
|
||||||
THT_DATA_ROOT: /data
|
THT_DATA_ROOT: /data
|
||||||
THT_OLLAMA_URL: "${THT_OLLAMA_URL:-http://host.docker.internal:11434}"
|
|
||||||
THT_SECRETS_FILE: /run/secrets/thothii.secrets
|
THT_SECRETS_FILE: /run/secrets/thothii.secrets
|
||||||
secrets: [{source: thothii_secrets, target: thothii.secrets}]
|
secrets: [{source: thothii_secrets, target: thothii.secrets}]
|
||||||
volumes:
|
volumes:
|
||||||
- thoth_data:/data
|
- thoth_data:/data
|
||||||
- ./deploy/workspaces:/app/harness/workspaces:ro
|
- ./deploy/workspaces:/app/harness/workspaces:ro
|
||||||
restart: "no"
|
restart: "no"
|
||||||
|
depends_on:
|
||||||
|
qdrant:
|
||||||
|
condition: service_healthy
|
||||||
|
embedding-model-init:
|
||||||
|
condition: service_completed_successfully
|
||||||
|
|
||||||
preprocess-dwh:
|
preprocess-dwh:
|
||||||
image: thothii-core:local
|
image: thothii-core:local
|
||||||
|
|||||||
@@ -9,10 +9,9 @@ chmod 600 deploy/secrets/thothii.secrets
|
|||||||
```
|
```
|
||||||
|
|
||||||
The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). The supported
|
The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). The supported
|
||||||
keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_VEC_API_KEY`,
|
keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`, and
|
||||||
`THT_VEC_WRITE_API_KEY`, and the four `THT_VECTOR_*_PASSWORD` role passwords. Values must be
|
`PI_PROVIDER_API_KEY`. Values must be non-empty and contain no whitespace. Do not put secrets
|
||||||
non-empty and contain no whitespace. Do not put secrets in the root `.env`, workspace YAML,
|
in the root `.env`, workspace YAML, URLs, logs, or rendered Compose output.
|
||||||
URLs, logs, or rendered Compose output.
|
|
||||||
|
|
||||||
Compose mounts the bundle read-only as `/run/secrets/thothii.secrets`. The host file must be a
|
Compose mounts the bundle read-only as `/run/secrets/thothii.secrets`. The host file must be a
|
||||||
regular non-symlink file with mode `0600` or `0400`; Docker's normal `0444` mode is accepted
|
regular non-symlink file with mode `0600` or `0400`; Docker's normal `0444` mode is accepted
|
||||||
@@ -33,29 +32,10 @@ Compose files intentionally do not create this mount.
|
|||||||
## Migration from separate secret files
|
## Migration from separate secret files
|
||||||
|
|
||||||
Older installations used `THT_*_SECRET_FILE` variables and one file per value. Migrate by
|
Older installations used `THT_*_SECRET_FILE` variables and one file per value. Migrate by
|
||||||
copying each value to its bundle key, validating with the complete base+profile command, and only
|
copying each retained value to its bundle key, validating with the complete base+profile command,
|
||||||
then deleting the old files. The old variables remain a compatibility path for staged upgrades,
|
and only then deleting the old files. The old variables remain a compatibility path for staged
|
||||||
but the documented and tested default is an absolute `THT_SECRETS_FILE` path to the protected
|
upgrades, but the documented and tested default is an absolute `THT_SECRETS_FILE` path to the
|
||||||
bundle.
|
protected bundle.
|
||||||
|
|
||||||
The local-vector bootstrap rotation helper still accepts an old/new password file as its
|
|
||||||
maintenance interface. Run it only with files protected by `0600`, then copy the resulting
|
|
||||||
password into `THT_VECTOR_BOOTSTRAP_PASSWORD` in the bundle before restarting
|
|
||||||
`vector-reconcile`/the application. The helper never prints password contents.
|
|
||||||
|
|
||||||
The helper has no implicit operator-env default. Pass the same protected env file used for the
|
|
||||||
deployment explicitly; it must be a readable regular non-symlink file and must not be writable by
|
|
||||||
group or other users:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
chmod 600 deploy/env/local.env
|
|
||||||
./scripts/vector-rotate-bootstrap-password.sh \
|
|
||||||
--env-file "$(pwd)/deploy/env/local.env" \
|
|
||||||
/secure/thoth/bootstrap-password /secure/thoth/bootstrap-password.next
|
|
||||||
```
|
|
||||||
|
|
||||||
Automation may set the narrowly scoped `THT_VECTOR_OPERATOR_ENV_FILE` instead. An explicit
|
|
||||||
`--env-file` takes precedence. Missing or unsafe env files are rejected before Compose runs.
|
|
||||||
|
|
||||||
Hosted Pi providers must use a single provider key. Compound providers (Bedrock, Azure OpenAI
|
Hosted Pi providers must use a single provider key. Compound providers (Bedrock, Azure OpenAI
|
||||||
Responses, Cloudflare Workers AI/Gateway) fail closed until a provider-specific credential
|
Responses, Cloudflare Workers AI/Gateway) fail closed until a provider-specific credential
|
||||||
|
|||||||
@@ -5,16 +5,8 @@
|
|||||||
# Hosted model provider (single-key providers only).
|
# Hosted model provider (single-key providers only).
|
||||||
# THT_MODEL_API_KEY=replace-me
|
# THT_MODEL_API_KEY=replace-me
|
||||||
|
|
||||||
# External DWH and vector adapters.
|
# External DWH adapter.
|
||||||
# THT_DWH_API_KEY=replace-me
|
# THT_DWH_API_KEY=replace-me
|
||||||
# THT_VEC_API_KEY=replace-me
|
|
||||||
# THT_VEC_WRITE_API_KEY=replace-me
|
|
||||||
|
|
||||||
# Optional local-vector roles.
|
|
||||||
# THT_VECTOR_BOOTSTRAP_PASSWORD=replace-me
|
|
||||||
# THT_VECTOR_MIGRATOR_PASSWORD=replace-me
|
|
||||||
# THT_VECTOR_READER_PASSWORD=replace-me
|
|
||||||
# THT_VECTOR_WRITER_PASSWORD=replace-me
|
|
||||||
|
|
||||||
# Optional CA material/path understood by the configured adapter.
|
# Optional CA material/path understood by the configured adapter.
|
||||||
# THT_CA=/run/secrets/ca-chain.pem
|
# THT_CA=/run/secrets/ca-chain.pem
|
||||||
|
|||||||
@@ -1,25 +0,0 @@
|
|||||||
-- ThothII — ruolo vector read+write (schema vectors).
|
|
||||||
-- Stessa istanza del DWH (porta 5438). ThothII indicizza (write) + ricerca (read) direttamente.
|
|
||||||
-- La separazione reader/writer resta rilevante solo per il path REST (non usato in Profile A).
|
|
||||||
-- psql -h localhost -p 5438 -U postgres -d postgres -v PWD='<secret>' -f 20-vector-roles.sql
|
|
||||||
DO $$
|
|
||||||
BEGIN
|
|
||||||
IF NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'thoth_vector_rw') THEN
|
|
||||||
CREATE ROLE thoth_vector_rw LOGIN;
|
|
||||||
END IF;
|
|
||||||
END $$;
|
|
||||||
-- :'PWD' va fuori dal DO (psql non interpola nelle stringhe dollar-quoted)
|
|
||||||
ALTER ROLE thoth_vector_rw PASSWORD :'PWD';
|
|
||||||
|
|
||||||
CREATE SCHEMA IF NOT EXISTS vectors;
|
|
||||||
|
|
||||||
-- L'estensione pgvector deve esistere (già presente nell'istanza di produzione).
|
|
||||||
-- CREATE EXTENSION IF NOT EXISTS vector;
|
|
||||||
|
|
||||||
GRANT USAGE, CREATE ON SCHEMA vectors TO thoth_vector_rw;
|
|
||||||
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA vectors TO thoth_vector_rw;
|
|
||||||
GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA vectors TO thoth_vector_rw;
|
|
||||||
ALTER DEFAULT PRIVILEGES IN SCHEMA vectors
|
|
||||||
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO thoth_vector_rw;
|
|
||||||
ALTER DEFAULT PRIVILEGES IN SCHEMA vectors
|
|
||||||
GRANT USAGE, SELECT ON SEQUENCES TO thoth_vector_rw;
|
|
||||||
@@ -1,54 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
. /opt/thoth/secret-policy.sh
|
|
||||||
|
|
||||||
bundle=${THT_SECRETS_FILE:-/run/secrets/thothii.secrets}
|
|
||||||
export PGPASSWORD=$(read_bundle_secret "$bundle" THT_VECTOR_BOOTSTRAP_PASSWORD)
|
|
||||||
migrator_password=$(read_bundle_secret "$bundle" THT_VECTOR_MIGRATOR_PASSWORD)
|
|
||||||
reader_password=$(read_bundle_secret "$bundle" THT_VECTOR_READER_PASSWORD)
|
|
||||||
writer_password=$(read_bundle_secret "$bundle" THT_VECTOR_WRITER_PASSWORD)
|
|
||||||
|
|
||||||
psql --set=ON_ERROR_STOP=1 \
|
|
||||||
--set=migrator_user="$THT_VECTOR_MIGRATOR_USER" \
|
|
||||||
--set=migrator_password="$migrator_password" \
|
|
||||||
--set=reader_user="$THT_VECTOR_READER_USER" \
|
|
||||||
--set=reader_password="$reader_password" \
|
|
||||||
--set=writer_user="$THT_VECTOR_WRITER_USER" \
|
|
||||||
--set=writer_password="$writer_password" <<'SQL'
|
|
||||||
SELECT 'CREATE ROLE vector_reader NOLOGIN'
|
|
||||||
WHERE NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'vector_reader') \gexec
|
|
||||||
SELECT 'CREATE ROLE vector_writer NOLOGIN'
|
|
||||||
WHERE NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'vector_writer') \gexec
|
|
||||||
ALTER ROLE vector_reader NOLOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION;
|
|
||||||
ALTER ROLE vector_writer NOLOGIN NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION;
|
|
||||||
|
|
||||||
SELECT format('CREATE ROLE %I LOGIN', :'migrator_user')
|
|
||||||
WHERE NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = :'migrator_user') \gexec
|
|
||||||
SELECT format('CREATE ROLE %I LOGIN', :'reader_user')
|
|
||||||
WHERE NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = :'reader_user') \gexec
|
|
||||||
SELECT format('CREATE ROLE %I LOGIN', :'writer_user')
|
|
||||||
WHERE NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = :'writer_user') \gexec
|
|
||||||
|
|
||||||
SELECT format(
|
|
||||||
'ALTER ROLE %I LOGIN PASSWORD %L NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION',
|
|
||||||
:'migrator_user', :'migrator_password'
|
|
||||||
) \gexec
|
|
||||||
SELECT format(
|
|
||||||
'ALTER ROLE %I LOGIN PASSWORD %L NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION',
|
|
||||||
:'reader_user', :'reader_password'
|
|
||||||
) \gexec
|
|
||||||
SELECT format(
|
|
||||||
'ALTER ROLE %I LOGIN PASSWORD %L NOSUPERUSER NOCREATEDB NOCREATEROLE NOREPLICATION',
|
|
||||||
:'writer_user', :'writer_password'
|
|
||||||
) \gexec
|
|
||||||
|
|
||||||
SELECT format('GRANT vector_reader TO %I', :'reader_user') \gexec
|
|
||||||
SELECT format('GRANT vector_writer TO %I', :'writer_user') \gexec
|
|
||||||
SELECT format('ALTER DATABASE %I OWNER TO %I', current_database(), :'migrator_user') \gexec
|
|
||||||
SELECT format('CREATE SCHEMA IF NOT EXISTS vectors AUTHORIZATION %I', :'migrator_user') \gexec
|
|
||||||
SELECT format('ALTER SCHEMA vectors OWNER TO %I', :'migrator_user') \gexec
|
|
||||||
REVOKE ALL ON SCHEMA vectors FROM PUBLIC;
|
|
||||||
GRANT USAGE ON SCHEMA vectors TO vector_reader, vector_writer;
|
|
||||||
CREATE EXTENSION IF NOT EXISTS vector WITH SCHEMA vectors;
|
|
||||||
SQL
|
|
||||||
@@ -1,93 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Rotate the initialized PostgreSQL bootstrap role and verify before returning success."""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import psycopg2
|
|
||||||
from psycopg2 import sql
|
|
||||||
|
|
||||||
|
|
||||||
def read_secret(path: str) -> str:
|
|
||||||
value = Path(path).read_text()
|
|
||||||
if not value or "\x00" in value or any(character.isspace() for character in value):
|
|
||||||
raise ValueError("secret must be non-empty and contain no whitespace or NUL bytes")
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
def connect(password: str):
|
|
||||||
return psycopg2.connect(
|
|
||||||
host=os.environ.get("THT_VECTOR_HOST", "vector-db"),
|
|
||||||
port=int(os.environ.get("THT_VECTOR_PORT", "5432")),
|
|
||||||
dbname=os.environ.get("THT_VECTOR_DATABASE", "thoth"),
|
|
||||||
user=os.environ.get("THT_VECTOR_BOOTSTRAP_USER", "postgres"),
|
|
||||||
password=password,
|
|
||||||
connect_timeout=5,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def alter_current_role(connection, password: str) -> None:
|
|
||||||
with connection.cursor() as cursor:
|
|
||||||
cursor.execute("SELECT current_user")
|
|
||||||
current_user = cursor.fetchone()[0]
|
|
||||||
expected = os.environ.get("THT_VECTOR_BOOTSTRAP_USER", "postgres")
|
|
||||||
if current_user != expected:
|
|
||||||
raise RuntimeError("authenticated role does not match THT_VECTOR_BOOTSTRAP_USER")
|
|
||||||
cursor.execute(
|
|
||||||
sql.SQL("ALTER ROLE {} PASSWORD {}").format(
|
|
||||||
sql.Identifier(current_user), sql.Literal(password)
|
|
||||||
)
|
|
||||||
)
|
|
||||||
connection.commit()
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
if len(sys.argv) != 3:
|
|
||||||
print("usage: rotate-bootstrap-password.py OLD_SECRET NEW_SECRET", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
try:
|
|
||||||
old_password = read_secret(sys.argv[1])
|
|
||||||
new_password = read_secret(sys.argv[2])
|
|
||||||
if old_password == new_password:
|
|
||||||
raise ValueError("old and new bootstrap passwords must differ")
|
|
||||||
old_connection = connect(old_password)
|
|
||||||
except Exception as exc:
|
|
||||||
print(f"bootstrap rotation refused before change: {type(exc).__name__}", file=sys.stderr)
|
|
||||||
return 1
|
|
||||||
|
|
||||||
try:
|
|
||||||
alter_current_role(old_connection, new_password)
|
|
||||||
try:
|
|
||||||
verification = connect(new_password)
|
|
||||||
verification.close()
|
|
||||||
except Exception as verify_exc:
|
|
||||||
try:
|
|
||||||
alter_current_role(old_connection, old_password)
|
|
||||||
except Exception as restore_exc:
|
|
||||||
print(
|
|
||||||
"bootstrap rotation verification failed and password restore failed: "
|
|
||||||
f"{type(verify_exc).__name__}/{type(restore_exc).__name__}",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
return 3
|
|
||||||
print(
|
|
||||||
f"bootstrap rotation verification failed; old password restored: "
|
|
||||||
f"{type(verify_exc).__name__}",
|
|
||||||
file=sys.stderr,
|
|
||||||
)
|
|
||||||
return 1
|
|
||||||
except Exception as exc:
|
|
||||||
print(f"bootstrap rotation failed: {type(exc).__name__}", file=sys.stderr)
|
|
||||||
return 1
|
|
||||||
finally:
|
|
||||||
old_connection.close()
|
|
||||||
|
|
||||||
print("bootstrap database password rotated and new login verified")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,95 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
|
|
||||||
validate_secret_file() {
|
|
||||||
secret_path=$1
|
|
||||||
secret_name=$2
|
|
||||||
if [ -L "$secret_path" ] || [ ! -f "$secret_path" ] || [ ! -r "$secret_path" ] || [ ! -s "$secret_path" ]; then
|
|
||||||
echo "$secret_name must be a readable, non-empty regular file" >&2
|
|
||||||
return 2
|
|
||||||
fi
|
|
||||||
if LC_ALL=C grep -q '[[:space:]]' "$secret_path"; then
|
|
||||||
echo "$secret_name must contain no whitespace" >&2
|
|
||||||
return 2
|
|
||||||
fi
|
|
||||||
mode=$(stat -c '%a' "$secret_path" 2>/dev/null || stat -f '%Lp' "$secret_path" 2>/dev/null) || return 2
|
|
||||||
case "$secret_path:$mode" in
|
|
||||||
/run/secrets/*:444|/run/secrets/*:400|/run/secrets/*:600|*:600|*:400) ;;
|
|
||||||
*) echo "$secret_name must have mode 0600 or stricter (Docker secrets may be 0444)" >&2; return 2 ;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
read_secret_file() {
|
|
||||||
validate_secret_file "$1" "$2" || return
|
|
||||||
cat "$1"
|
|
||||||
}
|
|
||||||
|
|
||||||
# Validate the bundle without printing any value. Keep this parser aligned with
|
|
||||||
# the backend loader: comments/blank lines are allowed, while syntax, allowlist,
|
|
||||||
# duplicates, empty values, file size, and line size are fail-closed.
|
|
||||||
validate_bundle() {
|
|
||||||
bundle_path=$1
|
|
||||||
if [ -L "$bundle_path" ] || [ ! -f "$bundle_path" ] || [ ! -r "$bundle_path" ] || [ ! -s "$bundle_path" ]; then
|
|
||||||
echo "secret bundle must be a readable, non-empty regular file" >&2
|
|
||||||
return 2
|
|
||||||
fi
|
|
||||||
mode=$(stat -c '%a' "$bundle_path" 2>/dev/null || stat -f '%Lp' "$bundle_path" 2>/dev/null) || return 2
|
|
||||||
case "$bundle_path:$mode" in
|
|
||||||
/run/secrets/*:444|/run/secrets/*:400|/run/secrets/*:600|*:600|*:400) ;;
|
|
||||||
*) echo "secret bundle must have mode 0600 or stricter (Docker secrets may be 0444)" >&2; return 2 ;;
|
|
||||||
esac
|
|
||||||
size=$(stat -c '%s' "$bundle_path" 2>/dev/null || stat -f '%z' "$bundle_path" 2>/dev/null) || return 2
|
|
||||||
if [ "$size" -gt 65536 ]; then
|
|
||||||
echo "secret bundle exceeds the 64KiB limit" >&2
|
|
||||||
return 2
|
|
||||||
fi
|
|
||||||
awk '
|
|
||||||
{ sub(/\r$/, "", $0) }
|
|
||||||
length($0) > 16384 { exit 9 }
|
|
||||||
/^[[:space:]]*$/ || /^[[:space:]]*#/ { next }
|
|
||||||
/^[A-Z][A-Z0-9_]*=/ {
|
|
||||||
key=$0; sub(/=.*/, "", key)
|
|
||||||
val=$0; sub(/^[^=]*=/, "", val)
|
|
||||||
if (key !~ /^(THT_MODEL_API_KEY|THT_DWH_API_KEY|THT_VEC_API_KEY|THT_VEC_WRITE_API_KEY|THT_CA|THT_SSL_CA|THT_VECTOR_BOOTSTRAP_PASSWORD|THT_VECTOR_MIGRATOR_PASSWORD|THT_VECTOR_READER_PASSWORD|THT_VECTOR_WRITER_PASSWORD|PI_PROVIDER_API_KEY)$/) exit 6
|
|
||||||
if (val == "" || ++seen[key] > 1) exit 7
|
|
||||||
next
|
|
||||||
}
|
|
||||||
{ exit 4 }
|
|
||||||
' "$bundle_path" || {
|
|
||||||
echo "secret bundle syntax is invalid" >&2
|
|
||||||
return 2
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
# Read one value from the deployment bundle without putting the bundle itself in
|
|
||||||
# a service environment. Values selected for credentials must contain no spaces.
|
|
||||||
read_bundle_secret() {
|
|
||||||
bundle_path=$1
|
|
||||||
bundle_key=$2
|
|
||||||
validate_bundle "$bundle_path" || return
|
|
||||||
case "$bundle_key" in
|
|
||||||
THT_[A-Z0-9_]*|PI_PROVIDER_API_KEY) ;;
|
|
||||||
*) echo "invalid secret bundle key" >&2; return 2 ;;
|
|
||||||
esac
|
|
||||||
value=$(awk -v wanted="$bundle_key" '
|
|
||||||
{ sub(/\r$/, "", $0) }
|
|
||||||
/^[[:space:]]*$/ || /^[[:space:]]*#/ { next }
|
|
||||||
/^[A-Z][A-Z0-9_]*=/ {
|
|
||||||
key=$0; sub(/=.*/, "", key)
|
|
||||||
val=$0; sub(/^[^=]*=/, "", val)
|
|
||||||
if (key == wanted) {
|
|
||||||
found=1; print val
|
|
||||||
}
|
|
||||||
next
|
|
||||||
}
|
|
||||||
{ exit 4 }
|
|
||||||
END { if (!found) exit 5 }
|
|
||||||
' "$bundle_path") || {
|
|
||||||
echo "$bundle_key is unavailable in secret bundle" >&2
|
|
||||||
return 3
|
|
||||||
}
|
|
||||||
if [ -z "$value" ] || printf '%s' "$value" | LC_ALL=C grep -q '[[:space:]]'; then
|
|
||||||
echo "$bundle_key must contain no whitespace" >&2
|
|
||||||
return 2
|
|
||||||
fi
|
|
||||||
printf '%s' "$value"
|
|
||||||
}
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
. /opt/thoth/secret-policy.sh
|
|
||||||
|
|
||||||
bundle=${THT_SECRETS_FILE:-/run/secrets/thothii.secrets}
|
|
||||||
export POSTGRES_PASSWORD=$(read_bundle_secret "$bundle" THT_VECTOR_BOOTSTRAP_PASSWORD)
|
|
||||||
unset THT_SECRETS_FILE
|
|
||||||
exec /usr/local/bin/docker-entrypoint.sh postgres
|
|
||||||
@@ -2,6 +2,10 @@ language: en
|
|||||||
dwh:
|
dwh:
|
||||||
type: postgres_direct
|
type: postgres_direct
|
||||||
connection:
|
connection:
|
||||||
{host: vector-db, database: thoth, schema: vectors, user: thoth_vector_reader,
|
host: "${THT_PREPROCESS_DWH_HOST:-dwh}"
|
||||||
password_file: "${THT_VECTOR_READER_PASSWORD_FILE}"}
|
port: "${THT_PREPROCESS_DWH_PORT:-5432}"
|
||||||
|
database: "${THT_PREPROCESS_DWH_DATABASE:-warehouse}"
|
||||||
|
schema: "${THT_PREPROCESS_DWH_SCHEMA:-public}"
|
||||||
|
user: "${THT_PREPROCESS_DWH_USER:-thoth_reader}"
|
||||||
|
password_file: "${THT_PREPROCESS_DWH_PASSWORD_FILE:-/run/secrets/preprocess-dwh-password}"
|
||||||
roots: {artifacts: artifacts, indexes: indexes, sessions: sessions}
|
roots: {artifacts: artifacts, indexes: indexes, sessions: sessions}
|
||||||
|
|||||||
@@ -1,15 +1,22 @@
|
|||||||
language: en
|
language: en
|
||||||
dwh:
|
dwh:
|
||||||
type: postgres_direct
|
type: postgres_direct
|
||||||
connection: {host: unused, database: unused, schema: public, user: unused, password: unused}
|
connection:
|
||||||
|
host: "${THT_PREPROCESS_DWH_HOST:-unused}"
|
||||||
|
port: "${THT_PREPROCESS_DWH_PORT:-5432}"
|
||||||
|
database: "${THT_PREPROCESS_DWH_DATABASE:-unused}"
|
||||||
|
schema: "${THT_PREPROCESS_DWH_SCHEMA:-public}"
|
||||||
|
user: "${THT_PREPROCESS_DWH_USER:-unused}"
|
||||||
|
password_file: "${THT_PREPROCESS_DWH_PASSWORD_FILE:-/run/secrets/preprocess-dwh-password}"
|
||||||
vectors:
|
vectors:
|
||||||
type: pgvector_direct
|
type: qdrant
|
||||||
reader:
|
base_url: http://qdrant:6333
|
||||||
{host: vector-db, database: thoth, schema: vectors, user: thoth_vector_reader,
|
collection: preprocess-evidence
|
||||||
password_file: "${THT_VECTOR_READER_PASSWORD_FILE}"}
|
|
||||||
writer:
|
|
||||||
{host: vector-db, database: thoth, schema: vectors, user: thoth_vector_writer,
|
|
||||||
password_file: "${THT_VECTOR_WRITER_PASSWORD_FILE}"}
|
|
||||||
roots: {artifacts: artifacts, indexes: indexes, sessions: sessions}
|
roots: {artifacts: artifacts, indexes: indexes, sessions: sessions}
|
||||||
evidence: {source_root: /data/source, evidence_dir: evidence}
|
evidence: {source_root: /data/source, evidence_dir: evidence}
|
||||||
embeddings: {base_url: "${THT_OLLAMA_URL}", model: smoke, dim: 768, batch_size: 32}
|
embeddings:
|
||||||
|
provider: ollama_internal
|
||||||
|
base_url: http://embedding:11434
|
||||||
|
model: qwen3-embedding:0.6b
|
||||||
|
dim: 1024
|
||||||
|
batch_size: 32
|
||||||
|
|||||||
+65
-3
@@ -32,10 +32,11 @@ services:
|
|||||||
THT_SECRETS_FILE: /run/secrets/thothii.secrets
|
THT_SECRETS_FILE: /run/secrets/thothii.secrets
|
||||||
THT_DB_NAME: ${THT_DB_NAME:-}
|
THT_DB_NAME: ${THT_DB_NAME:-}
|
||||||
THT_DWH_REST_URL: ${THT_DWH_REST_URL:-}
|
THT_DWH_REST_URL: ${THT_DWH_REST_URL:-}
|
||||||
THT_VEC_REST_URL: ${THT_VEC_REST_URL:-}
|
|
||||||
THT_VEC_WRITE_REST_URL: ${THT_VEC_WRITE_REST_URL:-}
|
|
||||||
THT_OLLAMA_URL: ${THT_OLLAMA_URL:-}
|
|
||||||
THT_LLM_URL: ${THT_LLM_URL:-}
|
THT_LLM_URL: ${THT_LLM_URL:-}
|
||||||
|
THT_INTERNAL_QDRANT_URL: http://qdrant:6333
|
||||||
|
THT_INTERNAL_EMBEDDING_URL: http://embedding:11434
|
||||||
|
THT_INTERNAL_EMBEDDING_MODEL: qwen3-embedding:0.6b
|
||||||
|
THT_INTERNAL_EMBEDDING_DIMENSIONS: "1024"
|
||||||
MAX_PI_PROCESSES: ${MAX_PI_PROCESSES:-4}
|
MAX_PI_PROCESSES: ${MAX_PI_PROCESSES:-4}
|
||||||
extra_hosts:
|
extra_hosts:
|
||||||
- "host.docker.internal:host-gateway"
|
- "host.docker.internal:host-gateway"
|
||||||
@@ -59,6 +60,11 @@ services:
|
|||||||
retries: 5
|
retries: 5
|
||||||
start_period: 30s
|
start_period: 30s
|
||||||
restart: "no"
|
restart: "no"
|
||||||
|
depends_on:
|
||||||
|
qdrant:
|
||||||
|
condition: service_healthy
|
||||||
|
embedding-model-init:
|
||||||
|
condition: service_completed_successfully
|
||||||
networks: [thothii-net]
|
networks: [thothii-net]
|
||||||
|
|
||||||
frontend:
|
frontend:
|
||||||
@@ -77,6 +83,60 @@ services:
|
|||||||
restart: "no"
|
restart: "no"
|
||||||
networks: [thothii-net]
|
networks: [thothii-net]
|
||||||
|
|
||||||
|
qdrant:
|
||||||
|
image: qdrant/qdrant:v1.18.2@sha256:75eab8c4ba42096724fdcfde8b4de0b5713d529dde32f285a1f86fdcb2c9e50c
|
||||||
|
expose:
|
||||||
|
- "6333"
|
||||||
|
volumes:
|
||||||
|
- qdrant-data:/qdrant/storage
|
||||||
|
healthcheck:
|
||||||
|
test:
|
||||||
|
- CMD-SHELL
|
||||||
|
- >
|
||||||
|
/usr/bin/bash -lc "exec 3<>/dev/tcp/127.0.0.1/6333 &&
|
||||||
|
printf 'GET /healthz HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: close\r\n\r\n' >&3 &&
|
||||||
|
grep -q '200 OK' <&3"
|
||||||
|
interval: 15s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 10
|
||||||
|
start_period: 10s
|
||||||
|
networks: [thothii-net]
|
||||||
|
|
||||||
|
embedding:
|
||||||
|
image: ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a
|
||||||
|
command: ["serve"]
|
||||||
|
expose:
|
||||||
|
- "11434"
|
||||||
|
volumes:
|
||||||
|
- embedding-models:/root/.ollama
|
||||||
|
healthcheck:
|
||||||
|
test:
|
||||||
|
- CMD-SHELL
|
||||||
|
- >
|
||||||
|
/usr/bin/bash -lc "exec 3<>/dev/tcp/127.0.0.1/11434 &&
|
||||||
|
printf 'GET /api/tags HTTP/1.1\r\nHost: 127.0.0.1\r\nConnection: close\r\n\r\n' >&3 &&
|
||||||
|
grep -q '200 OK' <&3"
|
||||||
|
interval: 15s
|
||||||
|
timeout: 5s
|
||||||
|
retries: 20
|
||||||
|
start_period: 10s
|
||||||
|
networks: [thothii-net]
|
||||||
|
|
||||||
|
embedding-model-init:
|
||||||
|
image: ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a
|
||||||
|
entrypoint: ["/usr/bin/bash", "/opt/thoth/embedding-model-init.sh"]
|
||||||
|
environment:
|
||||||
|
OLLAMA_BASE_URL: http://embedding:11434
|
||||||
|
OLLAMA_MODEL: qwen3-embedding:0.6b
|
||||||
|
OLLAMA_WAIT_TIMEOUT_SEC: "180"
|
||||||
|
volumes:
|
||||||
|
- embedding-models:/root/.ollama
|
||||||
|
- ./docker/embedding-model-init.sh:/opt/thoth/embedding-model-init.sh:ro
|
||||||
|
depends_on:
|
||||||
|
embedding:
|
||||||
|
condition: service_healthy
|
||||||
|
networks: [thothii-net]
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
thothii-net:
|
thothii-net:
|
||||||
driver: bridge
|
driver: bridge
|
||||||
@@ -85,6 +145,8 @@ volumes:
|
|||||||
dev-data:
|
dev-data:
|
||||||
dev-pi-state:
|
dev-pi-state:
|
||||||
workspace-registry:
|
workspace-registry:
|
||||||
|
qdrant-data:
|
||||||
|
embedding-models:
|
||||||
|
|
||||||
secrets:
|
secrets:
|
||||||
thothii_secrets:
|
thothii_secrets:
|
||||||
|
|||||||
@@ -5,9 +5,3 @@ THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.internal.example
|
|||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
|
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
|
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
|
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=pgvector_direct
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_HOST=vector.internal.example
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_PORT=5432
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_USER=thoth_vector_reader
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_PASSWORD_FILE=/run/secrets/north-star-research-vector-password
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_EMBEDDING_BASE_URL=https://embeddings.internal.example
|
|
||||||
|
|||||||
@@ -80,30 +80,21 @@ is `THT_WS_<NAMESPACE>_<ROLE>_<SUFFIX>`. Copy
|
|||||||
[the bindings env example](examples/workspace-bindings.env.example) to an untracked operator file
|
[the bindings env example](examples/workspace-bindings.env.example) to an untracked operator file
|
||||||
and set its absolute path as `THT_WORKSPACE_BINDINGS_ENV_FILE`. It is loaded only into `core`.
|
and set its absolute path as `THT_WORKSPACE_BINDINGS_ENV_FILE`. It is loaded only into `core`.
|
||||||
Credentials and certificates use `*_FILE` path variables that must point inside `/run/secrets`.
|
Credentials and certificates use `*_FILE` path variables that must point inside `/run/secrets`.
|
||||||
If declared, `THT_WS_NORTH_STAR_RESEARCH_VECTOR_WRITER_API_KEY_FILE` is distinct from the vector reader
|
|
||||||
file; a reader credential is never repurposed for writing.
|
|
||||||
|
|
||||||
## Direct PostgreSQL, REST, and SSH tunnel bindings
|
## Direct PostgreSQL, REST, and SSH tunnel bindings
|
||||||
|
|
||||||
Set only fields for the selected transport in the dedicated bindings env file. Canonical YAML keeps
|
Set only fields for the selected transport in the dedicated bindings env file. Canonical YAML keeps
|
||||||
database/schema/collection, distance, embedding model, and dimensions shared in Git. Every
|
database/schema shared in Git. Every `*_FILE=/run/secrets/<target>` binding needs one matching
|
||||||
`*_FILE=/run/secrets/<target>` binding needs one matching host-only `*_SOURCE` path in operator
|
host-only `*_SOURCE` path in operator `.env`. Generate the untracked connector override from those
|
||||||
`.env`. Generate the untracked connector override from those two files during bootstrap; do not
|
two files during bootstrap; do not copy or maintain a workspace-specific Compose override.
|
||||||
copy or maintain a workspace-specific Compose override.
|
|
||||||
|
|
||||||
```dotenv
|
```dotenv
|
||||||
# Direct PostgreSQL and pgvector
|
# Direct PostgreSQL
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
|
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.example.invalid
|
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.example.invalid
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
|
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
|
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
|
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=pgvector_direct
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_HOST=vector.example.invalid
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_PORT=5432
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_USER=thoth_vector_reader
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_PASSWORD_FILE=/run/secrets/north-star-research-vector-password
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_EMBEDDING_BASE_URL=https://embeddings.example.invalid
|
|
||||||
```
|
```
|
||||||
|
|
||||||
```dotenv
|
```dotenv
|
||||||
@@ -111,9 +102,6 @@ THT_WS_NORTH_STAR_RESEARCH_EMBEDDING_BASE_URL=https://embeddings.example.invalid
|
|||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
|
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.example.invalid
|
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.example.invalid
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
|
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_BASE_URL=https://vectors.example.invalid
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key
|
|
||||||
```
|
```
|
||||||
|
|
||||||
```dotenv
|
```dotenv
|
||||||
@@ -130,8 +118,8 @@ THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
|
|||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
|
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
|
||||||
```
|
```
|
||||||
|
|
||||||
Repeat the SSH names for `VECTOR` where needed. REST diagnostics reject a private per-request CA
|
REST diagnostics reject a private per-request CA rather than weakening TLS; use runtime-trusted
|
||||||
rather than weakening TLS; use runtime-trusted HTTPS or verified direct/SSH native TLS. See the
|
HTTPS or verified direct/SSH native TLS. See the
|
||||||
[diagnostic protocol](../workspace-diagnostic-protocol.md).
|
[diagnostic protocol](../workspace-diagnostic-protocol.md).
|
||||||
|
|
||||||
An SSH connector can prove installation reachability, host-key verification, authentication, and
|
An SSH connector can prove installation reachability, host-key verification, authentication, and
|
||||||
|
|||||||
@@ -109,17 +109,12 @@ Select only a transport allowed by canonical YAML; preserve database/schema/coll
|
|||||||
dimensions, and distance as Git-shared identity.
|
dimensions, and distance as Git-shared identity.
|
||||||
|
|
||||||
```dotenv
|
```dotenv
|
||||||
# Direct PostgreSQL/pgvector with verified native TLS if a CA path is supplied.
|
# Direct PostgreSQL with verified native TLS if a CA path is supplied.
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
|
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.internal.example
|
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.internal.example
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
|
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
|
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
|
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=pgvector_direct
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_HOST=vector.internal.example
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_PORT=5432
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_USER=thoth_vector_reader
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_PASSWORD_FILE=/run/secrets/north-star-research-vector-password
|
|
||||||
```
|
```
|
||||||
|
|
||||||
```dotenv
|
```dotenv
|
||||||
@@ -127,10 +122,6 @@ THT_WS_NORTH_STAR_RESEARCH_VECTOR_PASSWORD_FILE=/run/secrets/north-star-research
|
|||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
|
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.internal.example
|
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.internal.example
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
|
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_BASE_URL=https://vectors.internal.example
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key
|
|
||||||
THT_WS_NORTH_STAR_RESEARCH_EMBEDDING_BASE_URL=https://embeddings.internal.example
|
|
||||||
```
|
```
|
||||||
|
|
||||||
```dotenv
|
```dotenv
|
||||||
@@ -147,7 +138,7 @@ THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
|
|||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
|
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
|
||||||
```
|
```
|
||||||
|
|
||||||
Repeat SSH variables for `VECTOR` when selected. REST diagnostics refuse private per-request CAs
|
REST diagnostics refuse private per-request CAs
|
||||||
rather than disable verification; use runtime-trusted HTTPS or verified direct/SSH native TLS. See
|
rather than disable verification; use runtime-trusted HTTPS or verified direct/SSH native TLS. See
|
||||||
the [diagnostic protocol](../workspace-diagnostic-protocol.md) for its read-only checks and optional
|
the [diagnostic protocol](../workspace-diagnostic-protocol.md) for its read-only checks and optional
|
||||||
reversible writer probe.
|
reversible writer probe.
|
||||||
|
|||||||
@@ -1,250 +1,78 @@
|
|||||||
# Installazione Docker nei quattro contesti operativi
|
# Installazione Docker nei contesti operativi correnti
|
||||||
|
|
||||||
ThothII viene distribuito con due immagini applicative:
|
ThothII usa una topologia Compose unica:
|
||||||
|
|
||||||
- `thothii-core`: backend Fastify, harness `tht` e Pi;
|
- `frontend`
|
||||||
- `thothii-frontend`: frontend React servito da nginx.
|
- `core`
|
||||||
|
- `qdrant`
|
||||||
|
- `embedding`
|
||||||
|
- `embedding-model-init`
|
||||||
|
|
||||||
PostgreSQL/pgvector, DWH ed Evidence restano esterni nel profilo predefinito. Il profilo opzionale `local-vector` avvia PostgreSQL/pgvector nel progetto Compose.
|
Qdrant e Ollama embedding sono servizi interni obbligatori del progetto Compose. Restano
|
||||||
|
esterni solo DWH e LLM.
|
||||||
|
|
||||||
## Installazione comune (il comando standard)
|
## Comando standard locale
|
||||||
|
|
||||||
Servono Docker Engine/Compose v2 su Linux oppure Docker Desktop su macOS/Windows. Dalla directory in cui si vuole conservare il clone:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
git clone <URL-REPOSITORY> ThothII
|
|
||||||
cd ThothII
|
|
||||||
cp deploy/env/local.env.example deploy/env/local.env
|
cp deploy/env/local.env.example deploy/env/local.env
|
||||||
cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets
|
cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets
|
||||||
chmod 600 deploy/secrets/thothii.secrets
|
chmod 600 deploy/secrets/thothii.secrets
|
||||||
```
|
|
||||||
|
|
||||||
Modificare **solo** questi file interni al clone:
|
|
||||||
|
|
||||||
| File | Cosa contiene |
|
|
||||||
|---|---|
|
|
||||||
| `deploy/env/local.env` | endpoint, database e path Pi locali; mai password/token |
|
|
||||||
| file protetti locali | credenziali e certificati, indicati dai binding del workspace |
|
|
||||||
| `deploy/workspaces/<nome>.yaml` | adapter, endpoint non riservati, `roots` ed Evidence |
|
|
||||||
|
|
||||||
Compilare `deploy/env/local.env`, inclusi i path assoluti `PI_AUTH_FILE` e
|
|
||||||
`THT_SECRETS_FILE`, con gli endpoint esterni. L'avvio
|
|
||||||
normale usa esplicitamente il file base e l'overlay locale:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
docker compose --env-file deploy/env/local.env \
|
docker compose --env-file deploy/env/local.env \
|
||||||
-f compose.yaml -f deploy/compose.local.yaml up --build -d
|
-f compose.yaml -f deploy/compose.local.yaml up --build -d
|
||||||
```
|
```
|
||||||
|
|
||||||
Verificare lo stato con lo stesso comando Compose e aprire <http://127.0.0.1:8080>. Il core
|
Compilare `deploy/env/local.env` con:
|
||||||
include Pi; il binario Pi non deve essere installato sull'host. `docker compose down` conserva i
|
|
||||||
volumi; usare `down --volumes` solo per un ambiente effimero.
|
|
||||||
|
|
||||||
### Formato del bundle unico
|
- `PI_AUTH_FILE`
|
||||||
|
- `THT_SECRETS_FILE`
|
||||||
|
- `THT_WORKSPACE_GIT_REMOTE`
|
||||||
|
- endpoint DWH
|
||||||
|
- endpoint LLM
|
||||||
|
|
||||||
`deploy/secrets/thothii.secrets` è un file di testo locale, non uno script shell. Sono ammessi commenti e righe vuote; ogni altra riga deve essere una sola assegnazione senza spazi:
|
Non inserire secret nel file `.env`. I secret runtime stanno nel bundle
|
||||||
|
`deploy/secrets/thothii.secrets`.
|
||||||
|
|
||||||
|
## Bundle dei secret
|
||||||
|
|
||||||
|
Le chiavi documentate e supportate nel bundle sono:
|
||||||
|
|
||||||
```dotenv
|
```dotenv
|
||||||
THT_MODEL_API_KEY=...
|
THT_MODEL_API_KEY=...
|
||||||
THT_DWH_API_KEY=...
|
THT_DWH_API_KEY=...
|
||||||
THT_VEC_API_KEY=...
|
|
||||||
THT_VEC_WRITE_API_KEY=...
|
|
||||||
THT_VECTOR_BOOTSTRAP_PASSWORD=...
|
|
||||||
THT_VECTOR_MIGRATOR_PASSWORD=...
|
|
||||||
THT_VECTOR_READER_PASSWORD=...
|
|
||||||
THT_VECTOR_WRITER_PASSWORD=...
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Inserire solo le chiavi necessarie al profilo scelto. Il bundle viene montato in sola lettura nel container come `/run/secrets/thothii.secrets`; il parser rifiuta duplicati, chiavi sconosciute, valori vuoti, symlink e permessi host troppo aperti. Non inserire secret in `.env`, nei workspace, negli URL o nell'output Compose renderizzato.
|
Una CA privata PEM resta esterna al bundle e va montata con un override Compose revisionato.
|
||||||
|
|
||||||
Una catena CA PEM **non può essere inserita nel bundle**: contiene whitespace e viene rifiutata dal parser. Se un endpoint usa una CA privata, conservarla nel secret manager/host e aggiungere un override Compose revisionato che monti il file in `/run/secrets/ca-chain.pem` e imposti `THT_SSL_CA` (o il parametro dell'adapter). Il clone base non crea quel mount: questa è una limitazione intenzionale da considerare in fase di deployment.
|
## Preprocessing
|
||||||
|
|
||||||
### Overlay opzionali espliciti
|
I job di preprocessing usano gli stessi servizi interni Qdrant/Ollama:
|
||||||
|
|
||||||
DWH/vector/embedding remoti restano endpoint del file locale o server. Per il solo preset di
|
|
||||||
sviluppo pgvector, aggiungere `-f deploy/compose.local-vector.yaml --profile local-vector` al
|
|
||||||
comando base. Per il preprocessing aggiungere anche
|
|
||||||
`-f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml --profile preprocess`,
|
|
||||||
poi ripetere l'intero comando base con l'azione `run --rm preprocess-evidence` oppure
|
|
||||||
`run --rm preprocess-dwh`.
|
|
||||||
|
|
||||||
## Workspace, adapter e Evidence
|
|
||||||
|
|
||||||
Il workspace YAML seleziona il trasporto disponibile. Esempio DWH REST e vector DB HTTP:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
language: en
|
|
||||||
dwh:
|
|
||||||
type: thoth_rest
|
|
||||||
database: {database: warehouse, schema: datawarehouse}
|
|
||||||
endpoint: {base_url: https://dwh.example.test}
|
|
||||||
vectors:
|
|
||||||
type: thoth_vector_http
|
|
||||||
reader: {base_url: https://vectors.example.test}
|
|
||||||
writer: {base_url: https://vectors.example.test}
|
|
||||||
roots: {artifacts: artifacts, indexes: indexes, sessions: sessions}
|
|
||||||
evidence: {source_root: /data/source, evidence_dir: evidence}
|
|
||||||
embeddings: {base_url: https://embeddings.example.test, model: nomodel, dim: 768}
|
|
||||||
```
|
|
||||||
|
|
||||||
Esempio con accesso diretto a PostgreSQL e pgvector:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
language: en
|
|
||||||
dwh:
|
|
||||||
type: postgres_direct
|
|
||||||
connection: {host: dwh.internal, database: warehouse, schema: public,
|
|
||||||
user: thoth_reader, password_file: /run/secrets/dwh_password}
|
|
||||||
vectors:
|
|
||||||
type: pgvector_direct
|
|
||||||
reader: {host: vector.internal, database: thoth, schema: vectors,
|
|
||||||
user: thoth_vector_reader, password_file: /run/secrets/vector_reader_password}
|
|
||||||
writer: {host: vector.internal, database: thoth, schema: vectors,
|
|
||||||
user: thoth_vector_writer, password_file: /run/secrets/vector_writer_password}
|
|
||||||
roots: {artifacts: artifacts, indexes: indexes, sessions: sessions}
|
|
||||||
```
|
|
||||||
|
|
||||||
Questo esempio mostra il contratto dell'adapter: i file indicati da `password_file` devono
|
|
||||||
essere montati da un override Compose approvato. Il profilo base monta soltanto il bundle unico;
|
|
||||||
per un DWH diretto occorre quindi materializzare il file password dal secret manager e aggiungere
|
|
||||||
il bind mount/runtime adapter corrispondente. Non inserire la password nel workspace o nell'URL.
|
|
||||||
|
|
||||||
`roots` sono relativi e vengono risolti sotto `/data/workspaces/<workspace>` nel volume Docker; non inserire path host come `/Users/...` o `C:\\...`. Per Evidence usare una radice filesystem montata in sola lettura oppure l'adapter HTTP/S3 previsto dal workspace. Per HTTP/S3 definire allowlist, limiti di dimensione/paginazione e una politica egress; non mettere token nelle URI.
|
|
||||||
|
|
||||||
## 1. Server remoto insieme ai database e al vector DB
|
|
||||||
|
|
||||||
Usare quando il server Docker è nella stessa rete del DWH e del vector DB (containerizzati o meno).
|
|
||||||
Compilare `deploy/env/local.env` con gli endpoint raggiungibili localmente:
|
|
||||||
|
|
||||||
```dotenv
|
|
||||||
THT_DB_NAME=warehouse
|
|
||||||
THT_DWH_REST_URL=https://dwh.internal.example
|
|
||||||
THT_VEC_REST_URL=https://vectors.internal.example
|
|
||||||
THT_OLLAMA_URL=https://embeddings.internal.example
|
|
||||||
AUTH_MODE=none
|
|
||||||
THOTH_PUBLIC_EXPOSURE=false
|
|
||||||
```
|
|
||||||
|
|
||||||
Riempire nel bundle le chiavi DWH/vector/model necessarie e avviare:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
docker compose --env-file deploy/env/local.env \
|
docker compose --env-file deploy/env/local.env \
|
||||||
-f compose.yaml -f deploy/compose.local.yaml up --build -d
|
-f compose.yaml -f deploy/compose.local.yaml \
|
||||||
|
-f deploy/compose.preprocess.yaml --profile preprocess run --rm preprocess-evidence
|
||||||
|
```
|
||||||
|
|
||||||
|
Per introspezione DWH:
|
||||||
|
|
||||||
|
```sh
|
||||||
docker compose --env-file deploy/env/local.env \
|
docker compose --env-file deploy/env/local.env \
|
||||||
-f compose.yaml -f deploy/compose.local.yaml exec core /opt/venv/bin/tht doctor --json
|
-f compose.yaml -f deploy/compose.local.yaml \
|
||||||
|
-f deploy/compose.preprocess.yaml --profile preprocess run --rm preprocess-dwh
|
||||||
```
|
```
|
||||||
|
|
||||||
Se si abilita l'overlay production, il proxy autenticato TLS deve essere l'unico listener pubblico
|
## Server
|
||||||
e deve sostituire gli header client con i claim restituiti dal proprio `auth_request`. L'esempio
|
|
||||||
usa header `X-Thoth-Trusted-*` soltanto sul collegamento privato; nginx frontend li converte nei
|
|
||||||
claim normalizzati `X-Thoth-Principal-Issuer`, `X-Thoth-Principal-Subject`,
|
|
||||||
`X-Thoth-Principal-Display-Name` e `X-Thoth-Is-Admin` attesi dal core. Non esporre direttamente
|
|
||||||
la porta pubblicata da nginx.
|
|
||||||
|
|
||||||
Se il server deve essere raggiungibile da altri host, usare il profilo
|
Per installazioni server usare il profilo server con overlay sessioni:
|
||||||
`deploy/compose.server.yaml`, configurare il proxy autenticato e impostare
|
|
||||||
`AUTH_MODE=upstream`/`THOTH_PUBLIC_EXPOSURE=true` come descritto nella sezione di trust boundary.
|
|
||||||
|
|
||||||
## 2. Mac locale
|
|
||||||
|
|
||||||
Installare Docker Desktop e, se usato, Ollama sul Mac. In `deploy/env/local.env` impostare gli endpoint:
|
|
||||||
|
|
||||||
```dotenv
|
|
||||||
THT_DB_NAME=warehouse
|
|
||||||
THT_DWH_REST_URL=https://dwh.example.test
|
|
||||||
THT_OLLAMA_URL=http://host.docker.internal:11434
|
|
||||||
THT_DOCS_ROOT=/data/source/evidence
|
|
||||||
```
|
|
||||||
|
|
||||||
Nel bundle aggiungere quattro password generate localmente:
|
|
||||||
|
|
||||||
```dotenv
|
|
||||||
THT_VECTOR_BOOTSTRAP_PASSWORD=<valore casuale>
|
|
||||||
THT_VECTOR_MIGRATOR_PASSWORD=<valore casuale>
|
|
||||||
THT_VECTOR_READER_PASSWORD=<valore casuale>
|
|
||||||
THT_VECTOR_WRITER_PASSWORD=<valore casuale>
|
|
||||||
```
|
|
||||||
|
|
||||||
Poi eseguire il comando standard base+locale mostrato sopra. Il primo avvio esegue
|
|
||||||
reconciliation dei ruoli e migrazione pgvector. Per preprocessing, impostare il preset indicato
|
|
||||||
sopra e usare l'azione `run --rm preprocess-evidence` o `run --rm preprocess-dwh` con tutti
|
|
||||||
gli stessi file e profili.
|
|
||||||
|
|
||||||
## 3. PC Windows locale
|
|
||||||
|
|
||||||
Usare Docker Desktop con backend WSL2 e abilitare la condivisione della directory del clone. Modificare `deploy/env/local.env`:
|
|
||||||
|
|
||||||
```dotenv
|
|
||||||
THT_DB_NAME=warehouse
|
|
||||||
THT_DWH_REST_URL=https://dwh.example.test
|
|
||||||
THT_OLLAMA_URL=http://host.docker.internal:11434
|
|
||||||
THT_DOCS_ROOT=/data/source/evidence
|
|
||||||
```
|
|
||||||
|
|
||||||
Creare `deploy/secrets/thothii.secrets` con un editor locale protetto (ACL leggibile solo dall'utente Docker) e le stesse quattro chiavi pgvector del profilo Mac. Non usare `ConvertFrom-SecureString`: il bundle deve contenere il valore in chiaro per il servizio, con accesso limitato al file. Da PowerShell, dalla radice del clone, eseguire:
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up --build -d
|
|
||||||
docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml ps
|
|
||||||
```
|
|
||||||
|
|
||||||
Se un bind mount viene rifiutato, aggiungere la cartella del repository a Docker Desktop → Settings → Resources → File Sharing. Per Ollama eseguito in WSL2 usare l'indirizzo raggiungibile dalla rete Docker invece di assumere `localhost`.
|
|
||||||
|
|
||||||
## 4. Server applicativo distinto da DB ed Evidence
|
|
||||||
|
|
||||||
Usare il profilo server e consentire dal firewall solo le destinazioni necessarie:
|
|
||||||
|
|
||||||
```dotenv
|
|
||||||
# Avvio: docker compose --env-file deploy/env/server.env \
|
|
||||||
# -f compose.yaml -f deploy/compose.server.yaml \
|
|
||||||
# -f deploy/compose.session-server.yaml.example up --build -d
|
|
||||||
THT_DB_NAME=warehouse
|
|
||||||
THT_DWH_REST_URL=https://dwh.example.test
|
|
||||||
THT_VEC_REST_URL=https://vectors.example.test
|
|
||||||
THT_OLLAMA_URL=https://embeddings.example.test
|
|
||||||
```
|
|
||||||
|
|
||||||
Avviare e verificare con il profilo server completo:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
docker compose --env-file deploy/env/server.env \
|
docker compose --env-file deploy/env/server.env \
|
||||||
-f compose.yaml -f deploy/compose.server.yaml \
|
-f compose.yaml -f deploy/compose.server.yaml \
|
||||||
-f deploy/compose.session-server.yaml.example up --build -d
|
-f deploy/compose.session-server.yaml.example up --build -d
|
||||||
docker compose --env-file deploy/env/server.env \
|
|
||||||
-f compose.yaml -f deploy/compose.server.yaml \
|
|
||||||
-f deploy/compose.session-server.yaml.example exec core /opt/venv/bin/tht doctor --json
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Il DWH e il vector DB possono essere REST/HTTP oppure adapter diretti (`postgres_direct`, `pgvector_direct`) se il server ha connettività TCP. Le Evidence possono essere:
|
Consultare anche:
|
||||||
|
|
||||||
- filesystem NFS/SMB montato sul server e presentato come root read-only;
|
- `docs/install/local-workspace-registry.md`
|
||||||
- endpoint HTTPS, con allowlist e limiti SSRF;
|
- `docs/install/server-workspace-registry.md`
|
||||||
- bucket S3 con secret references e endpoint custom esplicitamente autorizzati.
|
|
||||||
|
|
||||||
Il preprocessing può girare sul server applicativo usando il volume `/data`; mantenere separati workspace, lock e artefatti dei job. Avviare con il comando standard e verificare `tht doctor`.
|
|
||||||
|
|
||||||
## Migrazione da installazioni con secret separati
|
|
||||||
|
|
||||||
Le variabili `THT_*_SECRET_FILE` e i file `dwh-api-key`, `vector-reader-api-key`, `vector-writer-api-key`, `model-api-key` e `vector_*_password` appartengono al layout precedente. Non vengono importati automaticamente dal bundle. Per migrare:
|
|
||||||
|
|
||||||
1. creare `deploy/secrets/thothii.secrets` mode `0600`;
|
|
||||||
2. copiare ogni valore nel nome chiave corrispondente (`THT_DWH_API_KEY`, `THT_VEC_API_KEY`, `THT_VEC_WRITE_API_KEY`, `THT_MODEL_API_KEY` o `THT_VECTOR_*_PASSWORD`), senza virgolette né newline;
|
|
||||||
3. rimuovere dal `.env` le variabili `_SECRET_FILE` e impostare `THT_SECRETS_FILE` al percorso assoluto del bundle;
|
|
||||||
4. renderizzare e avviare con il comando base+locale completo e il suo `--env-file`;
|
|
||||||
5. solo dopo la verifica, cancellare i vecchi file separati.
|
|
||||||
|
|
||||||
Una CA PEM resta un'eccezione esterna come descritto sopra. Provider Pi con credenziali composte (Bedrock, Azure OpenAI Responses, Cloudflare Workers AI/Gateway) restano rifiutati finché non viene implementato un adapter dedicato.
|
|
||||||
|
|
||||||
## Controlli post-installazione
|
|
||||||
|
|
||||||
```sh
|
|
||||||
docker compose --env-file deploy/env/local.env \
|
|
||||||
-f compose.yaml -f deploy/compose.local.yaml config --quiet
|
|
||||||
docker compose --env-file deploy/env/local.env \
|
|
||||||
-f compose.yaml -f deploy/compose.local.yaml ps
|
|
||||||
docker compose --env-file deploy/env/local.env \
|
|
||||||
-f compose.yaml -f deploy/compose.local.yaml exec core /opt/venv/bin/tht doctor --json
|
|
||||||
./scripts/docker-smoke.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
Per il profilo locale usare anche `./scripts/local-vector-smoke.sh`; per il preprocessing `./scripts/preprocess-smoke.sh`. Non pubblicare `.env` o `deploy/secrets/thothii.secrets` nei log, nei backup Git o nei ticket.
|
|
||||||
|
|||||||
@@ -114,6 +114,10 @@ if ((ssh_override && https_override)); then
|
|||||||
fi
|
fi
|
||||||
|
|
||||||
while IFS= read -r name; do
|
while IFS= read -r name; do
|
||||||
|
if [[ "$name" == *"_VECTOR_"* || "$name" == *"_EMBEDDING_"* || "$name" == THT_VECTOR_* ]]; then
|
||||||
|
echo "retired semantic source path is not supported: $name" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
value="$(read_env_value "$env_file" "$name")"
|
value="$(read_env_value "$env_file" "$name")"
|
||||||
if [[ -v "$name" ]]; then
|
if [[ -v "$name" ]]; then
|
||||||
value="${!name}"
|
value="${!name}"
|
||||||
|
|||||||
@@ -60,6 +60,10 @@ targets=()
|
|||||||
sources=()
|
sources=()
|
||||||
while IFS=$'\t' read -r name target; do
|
while IFS=$'\t' read -r name target; do
|
||||||
[[ "$name" =~ ^THT_WS_[A-Za-z0-9_]+_FILE$ ]] || continue
|
[[ "$name" =~ ^THT_WS_[A-Za-z0-9_]+_FILE$ ]] || continue
|
||||||
|
if [[ "$name" == *"_VECTOR_"* || "$name" == *"_EMBEDDING_"* ]]; then
|
||||||
|
echo "retired semantic secret binding is not supported: ${name%_FILE}_SOURCE" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
[[ "$target" =~ ^/run/secrets/[A-Za-z0-9][A-Za-z0-9_.-]*$ && "$target" != *..* ]] || {
|
[[ "$target" =~ ^/run/secrets/[A-Za-z0-9][A-Za-z0-9_.-]*$ && "$target" != *..* ]] || {
|
||||||
echo "invalid connector secret target for $name: $target" >&2
|
echo "invalid connector secret target for $name: $target" >&2
|
||||||
exit 2
|
exit 2
|
||||||
|
|||||||
@@ -1,446 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
cd "$(dirname "$0")/.."
|
|
||||||
|
|
||||||
mode=${1:-run}
|
|
||||||
case "$mode" in
|
|
||||||
run|--live-collision-test|--backup-restore) ;;
|
|
||||||
*) echo "usage: $0 [--live-collision-test|--backup-restore]" >&2; exit 2 ;;
|
|
||||||
esac
|
|
||||||
|
|
||||||
keep_resources=${KEEP_SMOKE_RESOURCES:-0}
|
|
||||||
if [ "${SMOKE_PROJECT+x}" = x ]; then
|
|
||||||
echo "SMOKE_PROJECT is not accepted; the smoke always generates an owned namespace" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
secret_dir=$(mktemp -d "${TMPDIR:-/tmp}/thothii-vector-smoke.XXXXXX")
|
|
||||||
suffix=$(basename "$secret_dir" | tr -cd 'a-z0-9')
|
|
||||||
smoke_project="thothii-vector-smoke-$(date +%s)-$$-$suffix"
|
|
||||||
smoke_owner="$smoke_project-owner"
|
|
||||||
marker="local-vector-$smoke_project"
|
|
||||||
restore_container="${smoke_project}-restore"
|
|
||||||
restore_volume="${smoke_project}-restore-data"
|
|
||||||
|
|
||||||
bootstrap_password="smoke-bootstrap-$smoke_project"
|
|
||||||
migrator_password="smoke-migrator-$smoke_project"
|
|
||||||
reader_password="smoke-reader-$smoke_project"
|
|
||||||
writer_password="smoke-writer-$smoke_project"
|
|
||||||
bundle="$secret_dir/thothii.secrets"
|
|
||||||
write_bundle() {
|
|
||||||
umask 077
|
|
||||||
{
|
|
||||||
printf 'THT_VECTOR_BOOTSTRAP_PASSWORD=%s\n' "$bootstrap_password"
|
|
||||||
printf 'THT_VECTOR_MIGRATOR_PASSWORD=%s\n' "$migrator_password"
|
|
||||||
printf 'THT_VECTOR_READER_PASSWORD=%s\n' "$reader_password"
|
|
||||||
printf 'THT_VECTOR_WRITER_PASSWORD=%s\n' "$writer_password"
|
|
||||||
} >"$bundle"
|
|
||||||
chmod 0600 "$bundle"
|
|
||||||
}
|
|
||||||
write_bundle
|
|
||||||
export THT_SECRETS_FILE="$bundle"
|
|
||||||
printf '%s\n' '{}' >"$secret_dir/pi-auth.json"
|
|
||||||
chmod 0600 "$secret_dir/pi-auth.json"
|
|
||||||
operator_env="$secret_dir/operator.env"
|
|
||||||
printf '%s\n' \
|
|
||||||
'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \
|
|
||||||
"PI_AUTH_FILE=$secret_dir/pi-auth.json" \
|
|
||||||
"THT_SECRETS_FILE=$bundle" >"$operator_env"
|
|
||||||
# The rotation helper has an old/new file interface; these are test-only
|
|
||||||
# scratch files and are never mounted into a Compose service.
|
|
||||||
printf '%s' "$bootstrap_password" >"$secret_dir/bootstrap"
|
|
||||||
chmod 0600 "$secret_dir/bootstrap"
|
|
||||||
export THT_VECTOR_BOOTSTRAP_USER=thoth_bootstrap_smoke
|
|
||||||
export THOTH_SMOKE_OWNER="$smoke_owner"
|
|
||||||
|
|
||||||
compose() {
|
|
||||||
docker compose --env-file "$operator_env" -f compose.yaml -f deploy/compose.local-vector.yaml \
|
|
||||||
--project-name "$smoke_project" --profile local-vector "$@"
|
|
||||||
}
|
|
||||||
|
|
||||||
resource_ids() {
|
|
||||||
case "$1" in
|
|
||||||
container) docker ps -aq --filter "label=com.docker.compose.project=$smoke_project" ;;
|
|
||||||
volume) docker volume ls -q --filter "label=com.docker.compose.project=$smoke_project" ;;
|
|
||||||
network) docker network ls -q --filter "label=com.docker.compose.project=$smoke_project" ;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
resource_owner() {
|
|
||||||
case "$1" in
|
|
||||||
container) docker inspect --format '{{ index .Config.Labels "io.thothii.smoke-owner" }}' "$2" ;;
|
|
||||||
volume) docker volume inspect --format '{{ index .Labels "io.thothii.smoke-owner" }}' "$2" ;;
|
|
||||||
network) docker network inspect --format '{{ index .Labels "io.thothii.smoke-owner" }}' "$2" ;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
assert_no_collision() {
|
|
||||||
for kind in container volume network; do
|
|
||||||
ids=$(resource_ids "$kind")
|
|
||||||
if [ -n "$ids" ]; then
|
|
||||||
echo "refusing existing Compose project resources for generated namespace $smoke_project" >&2
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
verify_owned_resources() {
|
|
||||||
for kind in container volume network; do
|
|
||||||
for id in $(resource_ids "$kind"); do
|
|
||||||
owner=$(resource_owner "$kind" "$id" 2>/dev/null || true)
|
|
||||||
if [ "$owner" != "$smoke_owner" ]; then
|
|
||||||
echo "refusing cleanup of resource not owned by this smoke: $kind $id" >&2
|
|
||||||
return 1
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
done
|
|
||||||
}
|
|
||||||
|
|
||||||
cleanup() {
|
|
||||||
if [ "$keep_resources" = "1" ]; then
|
|
||||||
echo "Keeping smoke resources for project $smoke_project (KEEP_SMOKE_RESOURCES=1)." >&2
|
|
||||||
else
|
|
||||||
if verify_owned_resources; then
|
|
||||||
docker rm -f "$restore_container" >/dev/null 2>&1 || true
|
|
||||||
docker volume rm "$restore_volume" >/dev/null 2>&1 || true
|
|
||||||
compose down --volumes >/dev/null 2>&1 || true
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
rm -rf "$secret_dir"
|
|
||||||
}
|
|
||||||
trap cleanup EXIT HUP INT TERM
|
|
||||||
|
|
||||||
if [ "$mode" = "--live-collision-test" ]; then
|
|
||||||
collision_volume="${smoke_project}-collision"
|
|
||||||
docker volume create \
|
|
||||||
--label "com.docker.compose.project=$smoke_project" \
|
|
||||||
--label 'io.thothii.smoke-owner=foreign-owner' \
|
|
||||||
"$collision_volume" >/dev/null
|
|
||||||
if assert_no_collision 2>/dev/null; then
|
|
||||||
echo "live collision probe was not detected" >&2
|
|
||||||
docker volume rm "$collision_volume" >/dev/null
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
docker volume rm "$collision_volume" >/dev/null
|
|
||||||
echo "live local-vector project collision refusal passed."
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
probe_vector() {
|
|
||||||
compose exec -T core sh -ec '
|
|
||||||
. /app/docker/secret-policy.sh
|
|
||||||
tmp=$(mktemp -d); trap "rm -rf \"$tmp\"" EXIT
|
|
||||||
for role in READER WRITER; do
|
|
||||||
file="$tmp/$role"
|
|
||||||
read_bundle_secret /run/secrets/thothii.secrets "THT_VECTOR_${role}_PASSWORD" >"$file"
|
|
||||||
export "THT_VECTOR_${role}_PASSWORD_FILE=$file"
|
|
||||||
done
|
|
||||||
exec /opt/venv/bin/python - "$1" "$2"
|
|
||||||
' sh "$marker" "$1" <<'PY'
|
|
||||||
import hashlib
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
from tht.adapters.vector.pgvector import PgVectorStore
|
|
||||||
from tht.config import DatabaseConfig
|
|
||||||
from tht.ports.vector import VectorWriteRecord
|
|
||||||
from tht.vectorstore.records import VectorRecord
|
|
||||||
|
|
||||||
marker = sys.argv[1]
|
|
||||||
mode = sys.argv[2]
|
|
||||||
database = "thoth"
|
|
||||||
host = "vector-db"
|
|
||||||
|
|
||||||
def credential(role: str) -> DatabaseConfig:
|
|
||||||
return DatabaseConfig(
|
|
||||||
host=host,
|
|
||||||
port=5432,
|
|
||||||
database=database,
|
|
||||||
schema="vectors",
|
|
||||||
user=f"thoth_vector_{role}",
|
|
||||||
password=open(os.environ[f"THT_VECTOR_{role.upper()}_PASSWORD_FILE"]).read(),
|
|
||||||
)
|
|
||||||
|
|
||||||
store = PgVectorStore(credential("reader"), credential("writer"), expected_dimension=768)
|
|
||||||
health = store.health()
|
|
||||||
assert health.ok, health
|
|
||||||
assert health.read_reachable is True and health.write_reachable is True, health
|
|
||||||
|
|
||||||
record = VectorRecord(
|
|
||||||
id=marker,
|
|
||||||
kind="memory",
|
|
||||||
ref=marker,
|
|
||||||
title="Local vector persistence smoke",
|
|
||||||
content=marker,
|
|
||||||
metadata={"smoke": True},
|
|
||||||
)
|
|
||||||
embedding = [1.0] + [0.0] * 767
|
|
||||||
if mode == "write":
|
|
||||||
store.upsert(
|
|
||||||
"memory",
|
|
||||||
[VectorWriteRecord(record, embedding, hashlib.sha256(marker.encode()).hexdigest())],
|
|
||||||
)
|
|
||||||
hits = store.search(["memory"], embedding, limit=1, kinds=["memory"])
|
|
||||||
assert hits and hits[0].id == marker, hits
|
|
||||||
print(f"role health and persisted search passed for {marker} ({mode})")
|
|
||||||
PY
|
|
||||||
}
|
|
||||||
|
|
||||||
assert_no_collision
|
|
||||||
compose config --quiet
|
|
||||||
services=$(compose config --services)
|
|
||||||
printf '%s\n' "$services" | grep -qx vector-db
|
|
||||||
printf '%s\n' "$services" | grep -qx vector-reconcile
|
|
||||||
printf '%s\n' "$services" | grep -qx vector-migrate
|
|
||||||
|
|
||||||
compose up --build --wait vector-reconcile vector-migrate core
|
|
||||||
core_id=$(compose ps -q core)
|
|
||||||
inspect_env=$(docker inspect --format '{{json .Config.Env}}' "$core_id")
|
|
||||||
if printf '%s' "$inspect_env" | grep -q "smoke-\(reader\|writer\)-${smoke_project}"; then
|
|
||||||
echo "docker inspect exposed a direct vector password" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
printf '%s' "$inspect_env" | grep -q 'THT_SECRETS_FILE=/run/secrets/thothii.secrets'
|
|
||||||
migration_status=$(compose run --rm --no-deps vector-migrate)
|
|
||||||
printf '%s\n' "$migration_status" | grep -q '"pending": \[\]'
|
|
||||||
migrator_flags=$(compose run --rm --no-deps --entrypoint sh vector-reconcile -ec '
|
|
||||||
. /opt/thoth/secret-policy.sh
|
|
||||||
export PGPASSWORD=$(read_bundle_secret /run/secrets/thothii.secrets THT_VECTOR_BOOTSTRAP_PASSWORD)
|
|
||||||
psql -At --host vector-db --username "$THT_VECTOR_BOOTSTRAP_USER" --dbname thoth \
|
|
||||||
--command "SELECT (NOT rolcreaterole) AND (NOT rolcreatedb) AND (NOT rolsuper) FROM pg_roles WHERE rolname = '\''thoth_vector_migrator'\''"
|
|
||||||
')
|
|
||||||
test "$migrator_flags" = t
|
|
||||||
probe_vector write
|
|
||||||
|
|
||||||
old_reader_password="$reader_password"
|
|
||||||
migrator_password="rotated-migrator-$smoke_project"
|
|
||||||
reader_password="rotated-reader-$smoke_project"
|
|
||||||
writer_password="rotated-writer-$smoke_project"
|
|
||||||
write_bundle
|
|
||||||
|
|
||||||
compose run --rm vector-reconcile
|
|
||||||
rotation_status=$(compose run --rm --no-deps vector-migrate)
|
|
||||||
printf '%s\n' "$rotation_status" | grep -q '"pending": \[\]'
|
|
||||||
if compose run --rm --no-deps --entrypoint psql \
|
|
||||||
-e PGPASSWORD="$old_reader_password" vector-reconcile \
|
|
||||||
--host vector-db --username thoth_vector_reader --dbname thoth --command 'SELECT 1' \
|
|
||||||
>/dev/null 2>&1; then
|
|
||||||
echo "old reader credential still works after rotation" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
compose up --force-recreate --no-deps --wait core
|
|
||||||
probe_vector read
|
|
||||||
|
|
||||||
old_bootstrap_password="$bootstrap_password"
|
|
||||||
printf '%s' "wrong-bootstrap-${smoke_project}" >"$secret_dir/bootstrap-wrong"
|
|
||||||
printf '%s' "next-bootstrap-'quoted-${smoke_project}" >"$secret_dir/bootstrap-next"
|
|
||||||
cp "$secret_dir/bootstrap" "$secret_dir/bootstrap-before-negative"
|
|
||||||
printf 'invalid bootstrap password\n' >"$secret_dir/bootstrap-whitespace"
|
|
||||||
chmod 0600 "$secret_dir/bootstrap-wrong" "$secret_dir/bootstrap-next" \
|
|
||||||
"$secret_dir/bootstrap-before-negative" "$secret_dir/bootstrap-whitespace"
|
|
||||||
if COMPOSE_PROJECT_NAME="$smoke_project" \
|
|
||||||
./scripts/vector-rotate-bootstrap-password.sh \
|
|
||||||
--env-file "$operator_env" \
|
|
||||||
"$secret_dir/bootstrap" "$secret_dir/bootstrap-whitespace" \
|
|
||||||
>/dev/null 2>&1; then
|
|
||||||
echo "bootstrap rotation accepted whitespace in a secret" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
cmp "$secret_dir/bootstrap" "$secret_dir/bootstrap-before-negative"
|
|
||||||
compose run --rm --no-deps --entrypoint psql \
|
|
||||||
-e PGPASSWORD="$old_bootstrap_password" vector-reconcile \
|
|
||||||
--host vector-db --username "$THT_VECTOR_BOOTSTRAP_USER" --dbname thoth \
|
|
||||||
--command 'SELECT 1' >/dev/null
|
|
||||||
|
|
||||||
if COMPOSE_PROJECT_NAME="$smoke_project" \
|
|
||||||
./scripts/vector-rotate-bootstrap-password.sh \
|
|
||||||
--env-file "$operator_env" \
|
|
||||||
"$secret_dir/bootstrap-wrong" "$secret_dir/bootstrap-next" \
|
|
||||||
>/dev/null 2>&1; then
|
|
||||||
echo "bootstrap rotation accepted the wrong old secret" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
cmp "$secret_dir/bootstrap" "$secret_dir/bootstrap-before-negative"
|
|
||||||
|
|
||||||
COMPOSE_PROJECT_NAME="$smoke_project" \
|
|
||||||
./scripts/vector-rotate-bootstrap-password.sh \
|
|
||||||
--env-file "$operator_env" \
|
|
||||||
"$secret_dir/bootstrap" "$secret_dir/bootstrap-next"
|
|
||||||
new_bootstrap_password=$(cat "$secret_dir/bootstrap")
|
|
||||||
bootstrap_password="$new_bootstrap_password"
|
|
||||||
write_bundle
|
|
||||||
test "$new_bootstrap_password" != "$old_bootstrap_password"
|
|
||||||
if compose run --rm --no-deps --entrypoint psql \
|
|
||||||
-e PGPASSWORD="$old_bootstrap_password" vector-reconcile \
|
|
||||||
--host vector-db --username "$THT_VECTOR_BOOTSTRAP_USER" --dbname thoth --command 'SELECT 1' \
|
|
||||||
>/dev/null 2>&1; then
|
|
||||||
echo "old bootstrap credential still works after rotation" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
compose run --rm --no-deps --entrypoint psql \
|
|
||||||
-e PGPASSWORD="$new_bootstrap_password" vector-reconcile \
|
|
||||||
--host vector-db --username "$THT_VECTOR_BOOTSTRAP_USER" --dbname thoth --command 'SELECT 1' \
|
|
||||||
>/dev/null
|
|
||||||
compose run --rm vector-reconcile
|
|
||||||
bootstrap_rotation_status=$(compose run --rm --no-deps vector-migrate)
|
|
||||||
printf '%s\n' "$bootstrap_rotation_status" | grep -q '"pending": \[\]'
|
|
||||||
compose up --force-recreate --no-deps --wait core
|
|
||||||
probe_vector read
|
|
||||||
|
|
||||||
compose restart vector-db core
|
|
||||||
compose up --wait vector-db core
|
|
||||||
probe_vector read
|
|
||||||
|
|
||||||
if [ "$mode" = "--backup-restore" ]; then
|
|
||||||
image=$(compose images -q vector-db)
|
|
||||||
network="${smoke_project}_default"
|
|
||||||
docker volume create \
|
|
||||||
--label "com.docker.compose.project=$smoke_project" \
|
|
||||||
--label "io.thothii.smoke-owner=$smoke_owner" "$restore_volume" >/dev/null
|
|
||||||
docker run -d --name "$restore_container" \
|
|
||||||
--label "com.docker.compose.project=$smoke_project" \
|
|
||||||
--label "io.thothii.smoke-owner=$smoke_owner" \
|
|
||||||
--network "$network" --network-alias vector-db-restore \
|
|
||||||
--mount "type=volume,source=$restore_volume,target=/var/lib/postgresql/data" \
|
|
||||||
--mount "type=bind,source=$bundle,target=/run/secrets/thothii.secrets,readonly" \
|
|
||||||
--mount "type=bind,source=$(pwd)/deploy/vector/vector-db-entrypoint.sh,target=/opt/thoth/vector-db-entrypoint.sh,readonly" \
|
|
||||||
--mount "type=bind,source=$(pwd)/deploy/vector/secret-policy.sh,target=/opt/thoth/secret-policy.sh,readonly" \
|
|
||||||
-e POSTGRES_DB=thoth -e POSTGRES_USER="$THT_VECTOR_BOOTSTRAP_USER" \
|
|
||||||
-e THT_SECRETS_FILE=/run/secrets/thothii.secrets \
|
|
||||||
--entrypoint /opt/thoth/vector-db-entrypoint.sh "$image" >/dev/null
|
|
||||||
attempts=0
|
|
||||||
until docker exec "$restore_container" pg_isready \
|
|
||||||
-U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth >/dev/null 2>&1; do
|
|
||||||
attempts=$((attempts + 1))
|
|
||||||
[ "$attempts" -lt 30 ] || { echo "restore database did not become ready" >&2; exit 1; }
|
|
||||||
sleep 1
|
|
||||||
done
|
|
||||||
docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql -X \
|
|
||||||
-U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth -v ON_ERROR_STOP=1 --command \
|
|
||||||
"CREATE SCHEMA vectors; CREATE EXTENSION vector WITH SCHEMA vectors;
|
|
||||||
CREATE TABLE vectors.memory (
|
|
||||||
id bigserial PRIMARY KEY, record_key text UNIQUE NOT NULL, kind text NOT NULL,
|
|
||||||
content_hash text NOT NULL, metadata jsonb NOT NULL,
|
|
||||||
embedding vectors.vector(768) NOT NULL, indexed_at timestamptz NOT NULL DEFAULT now());
|
|
||||||
INSERT INTO vectors.memory (record_key, kind, content_hash, metadata, embedding)
|
|
||||||
VALUES ('restore-sentinel', 'memory', 'sentinel-original', '{}',
|
|
||||||
('[' || '1,' || repeat('0,', 766) || '0]')::vectors.vector);" >/dev/null
|
|
||||||
|
|
||||||
docker run --rm --network "$network" \
|
|
||||||
--mount "type=bind,source=$(pwd),target=/repo,readonly" \
|
|
||||||
--mount "type=bind,source=$secret_dir,target=/scratch" "$image" \
|
|
||||||
/repo/scripts/vector-backup.sh --host vector-db --database thoth \
|
|
||||||
--user "$THT_VECTOR_BOOTSTRAP_USER" --password-file /scratch/bootstrap \
|
|
||||||
--output /scratch/vector.dump
|
|
||||||
|
|
||||||
compose exec -T vector-db sh -ec '
|
|
||||||
. /opt/thoth/secret-policy.sh
|
|
||||||
export PGPASSWORD=$(read_bundle_secret /run/secrets/thothii.secrets THT_VECTOR_BOOTSTRAP_PASSWORD)
|
|
||||||
psql -X -U "$POSTGRES_USER" -d thoth -v ON_ERROR_STOP=1 --command \
|
|
||||||
"UPDATE vectors.memory SET content_hash = '\''mutated-after-backup'\'' WHERE record_key = '\''$1'\''"' \
|
|
||||||
sh "$marker" >/dev/null
|
|
||||||
|
|
||||||
if docker run --rm --network "$network" \
|
|
||||||
--mount "type=bind,source=$(pwd),target=/repo,readonly" \
|
|
||||||
--mount "type=bind,source=$secret_dir,target=/scratch" "$image" \
|
|
||||||
/repo/scripts/vector-restore.sh \
|
|
||||||
--active-host vector-db --active-database thoth --active-user "$THT_VECTOR_BOOTSTRAP_USER" \
|
|
||||||
--active-password-file /scratch/bootstrap \
|
|
||||||
--target-host vector-db-restore --target-database thoth \
|
|
||||||
--target-user "$THT_VECTOR_BOOTSTRAP_USER" --target-password-file /scratch/bootstrap \
|
|
||||||
--input /scratch/vector.dump --force-nonempty >/dev/null 2>&1; then
|
|
||||||
echo "forced restore unexpectedly succeeded without archived ACL roles" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
sentinel=$(docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql \
|
|
||||||
-XAt -U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth --command \
|
|
||||||
"SELECT content_hash FROM vectors.memory WHERE record_key='restore-sentinel'")
|
|
||||||
test "$sentinel" = sentinel-original
|
|
||||||
docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql -X \
|
|
||||||
-U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth -v ON_ERROR_STOP=1 --command \
|
|
||||||
"DROP TABLE vectors.memory; CREATE ROLE vector_reader NOLOGIN; CREATE ROLE vector_writer NOLOGIN;" \
|
|
||||||
>/dev/null
|
|
||||||
|
|
||||||
docker run --rm --network "$network" \
|
|
||||||
--mount "type=bind,source=$(pwd),target=/repo,readonly" \
|
|
||||||
--mount "type=bind,source=$secret_dir,target=/scratch" "$image" \
|
|
||||||
/repo/scripts/vector-restore.sh \
|
|
||||||
--active-host vector-db --active-database thoth --active-user "$THT_VECTOR_BOOTSTRAP_USER" \
|
|
||||||
--active-password-file /scratch/bootstrap \
|
|
||||||
--target-host vector-db-restore --target-database thoth \
|
|
||||||
--target-user "$THT_VECTOR_BOOTSTRAP_USER" --target-password-file /scratch/bootstrap \
|
|
||||||
--input /scratch/vector.dump
|
|
||||||
|
|
||||||
docker run --rm --network "$network" \
|
|
||||||
--mount "type=bind,source=$(pwd)/deploy/vector/reconcile-roles.sh,target=/opt/thoth/reconcile-roles.sh,readonly" \
|
|
||||||
--mount "type=bind,source=$(pwd)/deploy/vector/secret-policy.sh,target=/opt/thoth/secret-policy.sh,readonly" \
|
|
||||||
--mount "type=bind,source=$bundle,target=/run/secrets/thothii.secrets,readonly" \
|
|
||||||
-e PGHOST=vector-db-restore -e PGDATABASE=thoth \
|
|
||||||
-e PGUSER="$THT_VECTOR_BOOTSTRAP_USER" \
|
|
||||||
-e THT_SECRETS_FILE=/run/secrets/thothii.secrets \
|
|
||||||
-e THT_VECTOR_MIGRATOR_USER=thoth_vector_migrator \
|
|
||||||
-e THT_VECTOR_READER_USER=thoth_vector_reader \
|
|
||||||
-e THT_VECTOR_WRITER_USER=thoth_vector_writer \
|
|
||||||
--entrypoint /opt/thoth/reconcile-roles.sh "$image" >/dev/null
|
|
||||||
|
|
||||||
compose exec -T core sh -ec '
|
|
||||||
. /app/docker/secret-policy.sh
|
|
||||||
tmp=$(mktemp -d); trap "rm -rf \"$tmp\"" EXIT
|
|
||||||
for role in READER WRITER; do
|
|
||||||
file="$tmp/$role"
|
|
||||||
read_bundle_secret /run/secrets/thothii.secrets "THT_VECTOR_${role}_PASSWORD" >"$file"
|
|
||||||
export "THT_VECTOR_${role}_PASSWORD_FILE=$file"
|
|
||||||
done
|
|
||||||
exec /opt/venv/bin/python - "$1"
|
|
||||||
' sh "$marker" <<'PY'
|
|
||||||
import hashlib
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
from tht.adapters.vector.pgvector import PgVectorStore
|
|
||||||
from tht.config import DatabaseConfig
|
|
||||||
from tht.ports.vector import VectorWriteRecord
|
|
||||||
from tht.vectorstore.records import VectorRecord
|
|
||||||
|
|
||||||
def config(role):
|
|
||||||
return DatabaseConfig(
|
|
||||||
host="vector-db-restore", port=5432, database="thoth", schema="vectors",
|
|
||||||
user=f"thoth_vector_{role}",
|
|
||||||
password=open(os.environ[f"THT_VECTOR_{role.upper()}_PASSWORD_FILE"]).read(),
|
|
||||||
)
|
|
||||||
|
|
||||||
store = PgVectorStore(config("reader"), config("writer"), expected_dimension=768)
|
|
||||||
assert store.health().ok, store.health()
|
|
||||||
embedding = [1.0] + [0.0] * 767
|
|
||||||
marker = sys.argv[1]
|
|
||||||
assert store.search(["memory"], embedding, limit=1, kinds=["memory"])[0].id == marker
|
|
||||||
write_id = marker + "-restore-write"
|
|
||||||
record = VectorRecord(
|
|
||||||
id=write_id, kind="memory", ref=write_id, title="restore writer",
|
|
||||||
content=write_id, metadata={},
|
|
||||||
)
|
|
||||||
store.upsert("memory", [VectorWriteRecord(record, embedding, hashlib.sha256(write_id.encode()).hexdigest())])
|
|
||||||
assert store.existing_hashes("memory", ["memory"])[write_id]
|
|
||||||
PY
|
|
||||||
|
|
||||||
restored=$(docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql \
|
|
||||||
-XAt -U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth --command \
|
|
||||||
"SELECT content_hash <> 'mutated-after-backup' FROM vectors.memory WHERE record_key = '$marker'")
|
|
||||||
test "$restored" = t
|
|
||||||
expected_migrations=$(find harness/tht/migrations/vector -type f -name '[0-9][0-9][0-9]_*.sql' \
|
|
||||||
-exec basename {} \; | sed 's/_.*//' | sort | paste -sd, -)
|
|
||||||
applied_migrations=$(docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql \
|
|
||||||
-XAt -U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth --command \
|
|
||||||
"SELECT string_agg(version, ',' ORDER BY version) FROM public.tht_vector_migrations")
|
|
||||||
test "$applied_migrations" = "$expected_migrations"
|
|
||||||
dimensions=$(docker exec -e PGPASSWORD="$new_bootstrap_password" "$restore_container" psql \
|
|
||||||
-XAt -U "$THT_VECTOR_BOOTSTRAP_USER" -d thoth --command \
|
|
||||||
"SELECT count(*) = 3 FROM pg_attribute a JOIN pg_class c ON c.oid=a.attrelid
|
|
||||||
JOIN pg_namespace n ON n.oid=c.relnamespace
|
|
||||||
WHERE n.nspname='vectors' AND a.attname='embedding' AND format_type(a.atttypid,a.atttypmod)='vectors.vector(768)'")
|
|
||||||
test "$dimensions" = t
|
|
||||||
echo "Transactional rollback and disposable-volume restore adapter parity passed."
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "Local pgvector runtime/bootstrap rotation, least-privilege roles, and persistence passed."
|
|
||||||
+45
-18
@@ -46,29 +46,23 @@ trap cleanup EXIT
|
|||||||
trap 'exit 129' HUP
|
trap 'exit 129' HUP
|
||||||
trap 'exit 130' INT
|
trap 'exit 130' INT
|
||||||
trap 'exit 143' TERM
|
trap 'exit 143' TERM
|
||||||
|
|
||||||
mkdir -p "$tmp/source/evidence"
|
mkdir -p "$tmp/source/evidence"
|
||||||
printf '%s\n' '# Evidence' 'generation one' >"$tmp/source/evidence/a.md"
|
printf '%s\n' '# Evidence' 'generation one' >"$tmp/source/evidence/a.md"
|
||||||
bundle="$tmp/thothii.secrets"
|
bundle="$tmp/thothii.secrets"
|
||||||
{
|
printf '%s\n' 'THT_MODEL_API_KEY=smoke-model-key' >"$bundle"
|
||||||
printf 'THT_VECTOR_BOOTSTRAP_PASSWORD=smoke-bootstrap-%s\n' "$project"
|
|
||||||
printf 'THT_VECTOR_MIGRATOR_PASSWORD=smoke-migrator-%s\n' "$project"
|
|
||||||
printf 'THT_VECTOR_READER_PASSWORD=smoke-reader-%s\n' "$project"
|
|
||||||
printf 'THT_VECTOR_WRITER_PASSWORD=smoke-writer-%s\n' "$project"
|
|
||||||
} >"$bundle"
|
|
||||||
chmod 0600 "$bundle"
|
chmod 0600 "$bundle"
|
||||||
export THT_SECRETS_FILE="$bundle"
|
|
||||||
export THT_OLLAMA_URL=http://mock-embeddings:8081
|
|
||||||
printf '%s\n' '{}' >"$tmp/pi-auth.json"
|
printf '%s\n' '{}' >"$tmp/pi-auth.json"
|
||||||
chmod 0600 "$tmp/pi-auth.json"
|
printf '%s' 'smoke-dwh-password' >"$tmp/dwh-password"
|
||||||
|
chmod 0600 "$tmp/pi-auth.json" "$tmp/dwh-password"
|
||||||
printf '%s\n' \
|
printf '%s\n' \
|
||||||
'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \
|
'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \
|
||||||
"PI_AUTH_FILE=$tmp/pi-auth.json" \
|
"PI_AUTH_FILE=$tmp/pi-auth.json" \
|
||||||
"THT_SECRETS_FILE=$bundle" \
|
"THT_SECRETS_FILE=$bundle" >"$tmp/operator.env"
|
||||||
'THT_OLLAMA_URL=http://mock-embeddings:8081' >"$tmp/operator.env"
|
|
||||||
|
|
||||||
cat >"$tmp/smoke.yaml" <<YAML
|
cat >"$tmp/smoke.yaml" <<YAML
|
||||||
services:
|
services:
|
||||||
mock-embeddings:
|
embedding:
|
||||||
image: thothii-core:local
|
image: thothii-core:local
|
||||||
profiles: [preprocess]
|
profiles: [preprocess]
|
||||||
entrypoint: [/opt/venv/bin/python, -c]
|
entrypoint: [/opt/venv/bin/python, -c]
|
||||||
@@ -78,19 +72,52 @@ services:
|
|||||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||||
class H(BaseHTTPRequestHandler):
|
class H(BaseHTTPRequestHandler):
|
||||||
def do_POST(self):
|
def do_POST(self):
|
||||||
n=len(json.loads(self.rfile.read(int(self.headers['Content-Length'])))['input'])
|
n = len(json.loads(self.rfile.read(int(self.headers["Content-Length"])))["input"])
|
||||||
body=json.dumps({'embeddings': [[1.0]+[0.0]*767 for _ in range(n)]}).encode()
|
body = json.dumps({"embeddings": [[1.0] + [0.0] * 1023 for _ in range(n)]}).encode()
|
||||||
self.send_response(200); self.send_header('Content-Length', str(len(body))); self.end_headers(); self.wfile.write(body)
|
self.send_response(200); self.send_header("Content-Length", str(len(body))); self.end_headers(); self.wfile.write(body)
|
||||||
|
def do_GET(self):
|
||||||
|
body = b'{"models":[{"name":"qwen3-embedding:0.6b"}]}'
|
||||||
|
self.send_response(200); self.send_header("Content-Length", str(len(body))); self.end_headers(); self.wfile.write(body)
|
||||||
def log_message(self, *args): pass
|
def log_message(self, *args): pass
|
||||||
HTTPServer(('0.0.0.0',8081),H).serve_forever()
|
HTTPServer(("0.0.0.0", 11434), H).serve_forever()
|
||||||
|
embedding-model-init:
|
||||||
|
profiles: [preprocess]
|
||||||
|
image: busybox:1.37.0
|
||||||
|
entrypoint: [sh, -ec]
|
||||||
|
command: ["exit 0"]
|
||||||
|
depends_on:
|
||||||
|
embedding: {condition: service_started}
|
||||||
|
dwh:
|
||||||
|
image: postgres:16-alpine
|
||||||
|
profiles: [preprocess]
|
||||||
|
environment:
|
||||||
|
POSTGRES_DB: warehouse
|
||||||
|
POSTGRES_USER: thoth_reader
|
||||||
|
POSTGRES_PASSWORD: smoke-dwh-password
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD-SHELL", "pg_isready -U thoth_reader -d warehouse"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 20
|
||||||
|
start_period: 10s
|
||||||
preprocess-evidence:
|
preprocess-evidence:
|
||||||
volumes:
|
volumes:
|
||||||
- $tmp/source:/data/source:ro
|
- $tmp/source:/data/source:ro
|
||||||
|
preprocess-dwh:
|
||||||
|
environment:
|
||||||
|
THT_PREPROCESS_DWH_HOST: dwh
|
||||||
|
THT_PREPROCESS_DWH_PORT: "5432"
|
||||||
|
THT_PREPROCESS_DWH_DATABASE: warehouse
|
||||||
|
THT_PREPROCESS_DWH_SCHEMA: public
|
||||||
|
THT_PREPROCESS_DWH_USER: thoth_reader
|
||||||
|
THT_PREPROCESS_DWH_PASSWORD_FILE: /run/secrets/preprocess-dwh-password
|
||||||
|
volumes:
|
||||||
|
- $tmp/dwh-password:/run/secrets/preprocess-dwh-password:ro
|
||||||
depends_on:
|
depends_on:
|
||||||
mock-embeddings: {condition: service_started}
|
dwh: {condition: service_healthy}
|
||||||
YAML
|
YAML
|
||||||
|
|
||||||
compose="docker compose --env-file $tmp/operator.env -f compose.yaml -f deploy/compose.local-vector.yaml -f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml -f $tmp/smoke.yaml --project-name $project --profile local-vector --profile preprocess"
|
compose="docker compose --env-file $tmp/operator.env -f compose.yaml -f deploy/compose.preprocess.yaml -f $tmp/smoke.yaml --project-name $project --profile preprocess"
|
||||||
$compose build preprocess-evidence
|
$compose build preprocess-evidence
|
||||||
if [ "${PREPROCESS_SMOKE_INJECT_FAILURE:-0}" = "1" ]; then
|
if [ "${PREPROCESS_SMOKE_INJECT_FAILURE:-0}" = "1" ]; then
|
||||||
sh -c 'exit 97'
|
sh -c 'exit 97'
|
||||||
|
|||||||
Executable
+32
@@ -0,0 +1,32 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
validate_secret_file() {
|
||||||
|
path="$1"
|
||||||
|
label="$2"
|
||||||
|
if [ -L "$path" ] || [ ! -f "$path" ] || [ ! -r "$path" ]; then
|
||||||
|
echo "$label must be a readable regular file, not a symlink: $path" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
mode=$(stat -c '%a' "$path" 2>/dev/null || stat -f '%Lp' "$path" 2>/dev/null) || {
|
||||||
|
echo "cannot inspect permissions for $label: $path" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
if [ $((0$mode & 077)) -ne 0 ]; then
|
||||||
|
echo "$label must not be readable or writable by group/other users: $path" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
read_secret_file() {
|
||||||
|
path="$1"
|
||||||
|
label="$2"
|
||||||
|
validate_secret_file "$path" "$label"
|
||||||
|
value=$(tr -d '\r' <"$path")
|
||||||
|
case "$value" in
|
||||||
|
*'
|
||||||
|
'*) echo "$label must contain exactly one line" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
[ -n "$value" ] || { echo "$label must not be empty" >&2; exit 2; }
|
||||||
|
printf '%s' "$value"
|
||||||
|
}
|
||||||
@@ -24,14 +24,6 @@ const expected = {
|
|||||||
THT_WS_TASK13_SMOKE_DWH_PORT: "5432",
|
THT_WS_TASK13_SMOKE_DWH_PORT: "5432",
|
||||||
THT_WS_TASK13_SMOKE_DWH_USER: "task13_reader",
|
THT_WS_TASK13_SMOKE_DWH_USER: "task13_reader",
|
||||||
THT_WS_TASK13_SMOKE_DWH_PASSWORD_FILE: "/run/secrets/task13-runtime-password",
|
THT_WS_TASK13_SMOKE_DWH_PASSWORD_FILE: "/run/secrets/task13-runtime-password",
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_TRANSPORT: "pgvector_direct",
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_HOST: "vector.task13.invalid",
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_PORT: "5432",
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_USER: "task13_vector_reader",
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_PASSWORD_FILE: "/run/secrets/task13-runtime-password",
|
|
||||||
THT_WS_TASK13_SMOKE_EMBEDDING_BASE_URL: profile === "local"
|
|
||||||
? `http://${config.name}-llm:9000`
|
|
||||||
: "https://embedding.task13.invalid",
|
|
||||||
};
|
};
|
||||||
for (const [name, value] of Object.entries(expected)) {
|
for (const [name, value] of Object.entries(expected)) {
|
||||||
if (core.environment?.[name] !== value) {
|
if (core.environment?.[name] !== value) {
|
||||||
@@ -82,7 +74,6 @@ for (const target of [
|
|||||||
|
|
||||||
const resolverEnvironment = { ...core.environment };
|
const resolverEnvironment = { ...core.environment };
|
||||||
resolverEnvironment.THT_WS_TASK13_SMOKE_DWH_PASSWORD_FILE = runtimePasswordMounts[0].source;
|
resolverEnvironment.THT_WS_TASK13_SMOKE_DWH_PASSWORD_FILE = runtimePasswordMounts[0].source;
|
||||||
resolverEnvironment.THT_WS_TASK13_SMOKE_VECTOR_PASSWORD_FILE = runtimePasswordMounts[0].source;
|
|
||||||
const bindings = resolveRuntimeBindings(workspace, resolverEnvironment, [dirname(runtimePasswordMounts[0].source)]);
|
const bindings = resolveRuntimeBindings(workspace, resolverEnvironment, [dirname(runtimePasswordMounts[0].source)]);
|
||||||
for (const [role, binding] of Object.entries(bindings)) {
|
for (const [role, binding] of Object.entries(bindings)) {
|
||||||
if ((binding as any).missing.length !== 0) {
|
if ((binding as any).missing.length !== 0) {
|
||||||
@@ -99,12 +90,13 @@ if (runtime.database.host !== expected.THT_WS_TASK13_SMOKE_DWH_HOST
|
|||||||
|| runtime.database.password_file !== runtimePasswordMounts[0].source) {
|
|| runtime.database.password_file !== runtimePasswordMounts[0].source) {
|
||||||
throw new Error("workspace resolver produced the wrong DWH runtime");
|
throw new Error("workspace resolver produced the wrong DWH runtime");
|
||||||
}
|
}
|
||||||
if (runtime.vector_db.host !== expected.THT_WS_TASK13_SMOKE_VECTOR_HOST
|
if (runtime.resources?.vector?.base_url !== "http://qdrant:6333"
|
||||||
|| runtime.vector_db.user !== expected.THT_WS_TASK13_SMOKE_VECTOR_USER
|
|| runtime.resources?.vector?.collection !== "task13-smoke") {
|
||||||
|| runtime.vector_db.password_file !== runtimePasswordMounts[0].source) {
|
throw new Error("workspace resolver produced the wrong qdrant runtime");
|
||||||
throw new Error("workspace resolver produced the wrong vector runtime");
|
|
||||||
}
|
}
|
||||||
if (runtime.embeddings.base_url !== expected.THT_WS_TASK13_SMOKE_EMBEDDING_BASE_URL) {
|
if (runtime.resources?.embeddings?.base_url !== "http://embedding:11434"
|
||||||
|
|| runtime.resources?.embeddings?.model !== "qwen3-embedding:0.6b"
|
||||||
|
|| runtime.resources?.embeddings?.dimensions !== 1024) {
|
||||||
throw new Error("workspace resolver produced the wrong embedding runtime");
|
throw new Error("workspace resolver produced the wrong embedding runtime");
|
||||||
}
|
}
|
||||||
const secret = readFileSync(bundleSource, "utf8").trim();
|
const secret = readFileSync(bundleSource, "utf8").trim();
|
||||||
|
|||||||
@@ -110,7 +110,6 @@ write_secret "$fixture_root/https-credentials" 'fixture-https-credentials'
|
|||||||
write_secret "$fixture_root/https-ca.pem" 'fixture-https-ca'
|
write_secret "$fixture_root/https-ca.pem" 'fixture-https-ca'
|
||||||
write_secret "$fixture_root/dwh-password" 'fixture-dwh-password'
|
write_secret "$fixture_root/dwh-password" 'fixture-dwh-password'
|
||||||
write_secret "$fixture_root/vector-api-key" 'fixture-vector-api-key'
|
write_secret "$fixture_root/vector-api-key" 'fixture-vector-api-key'
|
||||||
|
|
||||||
printf '%s\n' \
|
printf '%s\n' \
|
||||||
'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \
|
'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \
|
||||||
"PI_AUTH_FILE=$fixture_root/pi-auth.json" \
|
"PI_AUTH_FILE=$fixture_root/pi-auth.json" \
|
||||||
@@ -120,14 +119,11 @@ printf '%s\n' \
|
|||||||
"THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$fixture_root/ssh-known-hosts" \
|
"THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$fixture_root/ssh-known-hosts" \
|
||||||
"THT_WORKSPACE_GIT_CREDENTIALS_FILE=$fixture_root/https-credentials" \
|
"THT_WORKSPACE_GIT_CREDENTIALS_FILE=$fixture_root/https-credentials" \
|
||||||
"THT_WORKSPACE_GIT_CA_FILE=$fixture_root/https-ca.pem" \
|
"THT_WORKSPACE_GIT_CA_FILE=$fixture_root/https-ca.pem" \
|
||||||
"THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$fixture_root/dwh-password" \
|
"THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$fixture_root/dwh-password" >"$fixture_root/operator.env"
|
||||||
"THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=$fixture_root/vector-api-key" >"$fixture_root/operator.env"
|
|
||||||
|
|
||||||
printf '%s\n' \
|
printf '%s\n' \
|
||||||
'THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct' \
|
'THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct' \
|
||||||
'THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password' \
|
'THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password' \
|
||||||
'THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api' \
|
|
||||||
'THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key' \
|
|
||||||
>"$fixture_root/workspace-bindings.env"
|
>"$fixture_root/workspace-bindings.env"
|
||||||
|
|
||||||
connector_override="$fixture_root/compose.connector-secrets.local.yaml"
|
connector_override="$fixture_root/compose.connector-secrets.local.yaml"
|
||||||
@@ -142,10 +138,9 @@ assert_render_contract ssh '/run/secrets/workspace-registry-git-known-hosts,/run
|
|||||||
render https -f "$root/deploy/compose.git-https.yaml"
|
render https -f "$root/deploy/compose.git-https.yaml"
|
||||||
assert_render_contract https '/run/secrets/workspace-registry-git-ca,/run/secrets/workspace-registry-git-credentials' 'thothii.secrets'
|
assert_render_contract https '/run/secrets/workspace-registry-git-ca,/run/secrets/workspace-registry-git-credentials' 'thothii.secrets'
|
||||||
render connector -f "$connector_override"
|
render connector -f "$connector_override"
|
||||||
assert_render_contract connector '' 'north-star-research-dwh-password,north-star-research-vector-api-key,thothii.secrets'
|
assert_render_contract connector '' 'north-star-research-dwh-password,thothii.secrets'
|
||||||
|
|
||||||
assert_missing_source_rejected THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE
|
assert_missing_source_rejected THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE
|
||||||
assert_missing_source_rejected THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE
|
|
||||||
assert_unsafe_source_rejected 'relative/secret' relative-source
|
assert_unsafe_source_rejected 'relative/secret' relative-source
|
||||||
assert_unsafe_source_rejected '/private/secrets/../secret' non-normalized-source
|
assert_unsafe_source_rejected '/private/secrets/../secret' non-normalized-source
|
||||||
if THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE='relative/host-override' \
|
if THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE='relative/host-override' \
|
||||||
@@ -182,4 +177,25 @@ if "$root/scripts/compose-with-preflight.sh" --env-file "$fixture_root/operator.
|
|||||||
fi
|
fi
|
||||||
grep -Fq 'mutually exclusive' "$fixture_root/renamed-combined.err"
|
grep -Fq 'mutually exclusive' "$fixture_root/renamed-combined.err"
|
||||||
|
|
||||||
|
printf '%s\n' \
|
||||||
|
'THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password' \
|
||||||
|
'THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key' \
|
||||||
|
>"$fixture_root/workspace-bindings-with-vector.env"
|
||||||
|
printf '%s\n' \
|
||||||
|
'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \
|
||||||
|
"PI_AUTH_FILE=$fixture_root/pi-auth.json" \
|
||||||
|
"THT_SECRETS_FILE=$fixture_root/thothii.secrets" \
|
||||||
|
"THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$fixture_root/dwh-password" \
|
||||||
|
"THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=$fixture_root/vector-api-key" \
|
||||||
|
>"$fixture_root/operator-with-vector.env"
|
||||||
|
if "$root/scripts/generate-connector-secrets-override.sh" \
|
||||||
|
--bindings-env "$fixture_root/workspace-bindings-with-vector.env" \
|
||||||
|
--operator-env "$fixture_root/operator-with-vector.env" \
|
||||||
|
--output "$fixture_root/forbidden-vector.yaml" \
|
||||||
|
>"$fixture_root/forbidden-vector.out" 2>"$fixture_root/forbidden-vector.err"; then
|
||||||
|
echo "connector generator accepted a retired semantic API key binding" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
grep -Fq 'VECTOR_API_KEY_SOURCE' "$fixture_root/forbidden-vector.err"
|
||||||
|
|
||||||
echo "Compose secret policy passed."
|
echo "Compose secret policy passed."
|
||||||
|
|||||||
@@ -17,9 +17,7 @@ targets=(
|
|||||||
scripts/build-local.sh
|
scripts/build-local.sh
|
||||||
scripts/build-local.ps1
|
scripts/build-local.ps1
|
||||||
scripts/docker-smoke.sh
|
scripts/docker-smoke.sh
|
||||||
scripts/local-vector-smoke.sh
|
|
||||||
scripts/preprocess-smoke.sh
|
scripts/preprocess-smoke.sh
|
||||||
scripts/vector-rotate-bootstrap-password.sh
|
|
||||||
)
|
)
|
||||||
|
|
||||||
existing=()
|
existing=()
|
||||||
|
|||||||
@@ -1,5 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
cd "$(dirname "$0")/.."
|
|
||||||
./scripts/local-vector-smoke.sh --live-collision-test
|
|
||||||
@@ -1,71 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
cd "$(dirname "$0")/.."
|
|
||||||
tmp=$(mktemp -d)
|
|
||||||
trap 'rm -rf "$tmp"' EXIT HUP INT TERM
|
|
||||||
|
|
||||||
fake="$tmp/docker"
|
|
||||||
log="$tmp/docker.log"
|
|
||||||
state="$tmp/state"
|
|
||||||
|
|
||||||
cat >"$fake" <<'SH'
|
|
||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
printf '%s\n' "$*" >>"$FAKE_DOCKER_LOG"
|
|
||||||
|
|
||||||
if [ "${FAKE_COLLISION:-0}" = 1 ] && [ "$1 $2" = "ps -aq" ]; then
|
|
||||||
printf '%s\n' collision-container
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ "$1 $2" = "ps -aq" ] || [ "$1 $2" = "volume ls" ] || [ "$1 $2" = "network ls" ]; then
|
|
||||||
if [ "${FAKE_MISMATCH_ON_CLEANUP:-0}" = 1 ] && [ -f "$FAKE_DOCKER_STATE" ]; then
|
|
||||||
printf '%s\n' foreign-resource
|
|
||||||
fi
|
|
||||||
: >"$FAKE_DOCKER_STATE"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ "$1" = inspect ] || [ "$1 $2" = "volume inspect" ] || [ "$1 $2" = "network inspect" ]; then
|
|
||||||
printf '%s\n' foreign-owner
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
case "$*" in
|
|
||||||
*"config --services"*) printf '%s\n' vector-db vector-reconcile vector-migrate core frontend ;;
|
|
||||||
*"run --rm --no-deps vector-migrate"*) printf '%s\n' '{"applied":["001","002","003"],"drifted":[],"pending":[]}' ;;
|
|
||||||
esac
|
|
||||||
exit 0
|
|
||||||
SH
|
|
||||||
chmod 0755 "$fake"
|
|
||||||
|
|
||||||
if PATH="$tmp:$PATH" FAKE_DOCKER_LOG="$log" FAKE_DOCKER_STATE="$state" \
|
|
||||||
SMOKE_PROJECT=operator-owned ./scripts/local-vector-smoke.sh >"$tmp/out" 2>"$tmp/err"; then
|
|
||||||
echo "smoke accepted caller-controlled SMOKE_PROJECT" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
grep -q 'SMOKE_PROJECT is not accepted' "$tmp/err"
|
|
||||||
test ! -s "$log"
|
|
||||||
|
|
||||||
: >"$log"
|
|
||||||
rm -f "$state"
|
|
||||||
PATH="$tmp:$PATH" FAKE_DOCKER_LOG="$log" FAKE_DOCKER_STATE="$state" \
|
|
||||||
FAKE_COLLISION=1 ./scripts/local-vector-smoke.sh >"$tmp/out" 2>"$tmp/err" || true
|
|
||||||
grep -q 'refusing existing Compose project resources' "$tmp/err"
|
|
||||||
if grep -q 'compose.*up' "$log"; then
|
|
||||||
echo "smoke started after detecting a project collision" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
: >"$log"
|
|
||||||
rm -f "$state"
|
|
||||||
PATH="$tmp:$PATH" FAKE_DOCKER_LOG="$log" FAKE_DOCKER_STATE="$state" \
|
|
||||||
FAKE_MISMATCH_ON_CLEANUP=1 ./scripts/local-vector-smoke.sh >"$tmp/out" 2>"$tmp/err" || true
|
|
||||||
grep -q 'refusing cleanup of resource not owned by this smoke' "$tmp/err"
|
|
||||||
if grep -q 'down --volumes' "$log"; then
|
|
||||||
echo "smoke removed resources after ownership mismatch" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "local-vector smoke collision and cleanup ownership contracts passed."
|
|
||||||
@@ -9,10 +9,12 @@ trap 'rm -rf "$fixture"' EXIT HUP INT TERM
|
|||||||
new_fixture() {
|
new_fixture() {
|
||||||
rm -rf "$fixture/repository"
|
rm -rf "$fixture/repository"
|
||||||
mkdir -p \
|
mkdir -p \
|
||||||
|
"$fixture/repository/backend/src/workspaces" \
|
||||||
"$fixture/repository/deploy/env" \
|
"$fixture/repository/deploy/env" \
|
||||||
"$fixture/repository/deploy/workspaces" \
|
"$fixture/repository/deploy/workspaces" \
|
||||||
"$fixture/repository/docker/smoke" \
|
"$fixture/repository/docker/smoke" \
|
||||||
"$fixture/repository/docs/install" \
|
"$fixture/repository/docs/install" \
|
||||||
|
"$fixture/repository/docs/superpowers/specs" \
|
||||||
"$fixture/repository/docs/superpowers/plans" \
|
"$fixture/repository/docs/superpowers/plans" \
|
||||||
"$fixture/repository/frontend" \
|
"$fixture/repository/frontend" \
|
||||||
"$fixture/repository/scripts"
|
"$fixture/repository/scripts"
|
||||||
@@ -24,10 +26,14 @@ new_fixture() {
|
|||||||
printf '%s\n' 'THT_LLM_URL=https://llm.example.invalid' >"$fixture/repository/deploy/env/local.env.example"
|
printf '%s\n' 'THT_LLM_URL=https://llm.example.invalid' >"$fixture/repository/deploy/env/local.env.example"
|
||||||
printf '%s\n' '# generic launcher' >"$fixture/repository/scripts/run-stack.sh"
|
printf '%s\n' '# generic launcher' >"$fixture/repository/scripts/run-stack.sh"
|
||||||
printf '%s\n' '// generic frontend configuration' >"$fixture/repository/frontend/vite.config.ts"
|
printf '%s\n' '// generic frontend configuration' >"$fixture/repository/frontend/vite.config.ts"
|
||||||
|
printf '%s\n' '// explicit descriptor migration module may mention pgvector during conversion' \
|
||||||
|
>"$fixture/repository/backend/src/workspaces/migrate-legacy.ts"
|
||||||
|
|
||||||
# These are the three intentionally allowed categories from the Task 10 boundary.
|
# These are the three intentionally allowed categories from the Task 10 boundary.
|
||||||
printf '%s\n' 'historical omics_portal and Chirone record' \
|
printf '%s\n' 'historical omics_portal and Chirone record' \
|
||||||
>"$fixture/repository/docs/superpowers/plans/legacy.md"
|
>"$fixture/repository/docs/superpowers/plans/legacy.md"
|
||||||
|
printf '%s\n' 'historical pgvector rollout note' \
|
||||||
|
>"$fixture/repository/docs/superpowers/specs/history.md"
|
||||||
printf '%s\n' 'id: psd' >"$fixture/repository/deploy/workspaces/psd.yaml.example"
|
printf '%s\n' 'id: psd' >"$fixture/repository/deploy/workspaces/psd.yaml.example"
|
||||||
printf '%s\n' '# migrate PSD sessions from /home/chirone' \
|
printf '%s\n' '# migrate PSD sessions from /home/chirone' \
|
||||||
>"$fixture/repository/docker/session-migrate.sh"
|
>"$fixture/repository/docker/session-migrate.sh"
|
||||||
@@ -65,6 +71,12 @@ assert_detected frontend/vite.config.ts 'const base = "/omics_portal";'
|
|||||||
assert_detected scripts/test-qwen-network-config.sh 'require localllm_default'
|
assert_detected scripts/test-qwen-network-config.sh 'require localllm_default'
|
||||||
assert_detected scripts/test-provider-network.sh 'if (!config.networks?.localllm_default?.external) exit 1'
|
assert_detected scripts/test-provider-network.sh 'if (!config.networks?.localllm_default?.external) exit 1'
|
||||||
assert_detected deploy/compose.psd-local.yaml 'services: {}'
|
assert_detected deploy/compose.psd-local.yaml 'services: {}'
|
||||||
|
assert_detected deploy/compose.local-vector.yaml 'services: {}'
|
||||||
|
assert_detected deploy/compose.preprocess-local-vector.yaml 'services: {}'
|
||||||
|
assert_detected scripts/run-stack.sh 'export THT_VECTOR_READER_PASSWORD_FILE=/run/secrets/vector-reader'
|
||||||
|
assert_detected deploy/env/local.env.example 'THT_OLLAMA_URL=http://ollama.example.invalid:11434'
|
||||||
|
assert_detected scripts/generate-override.sh 'THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=/tmp/vector-key'
|
||||||
|
assert_detected scripts/test-contract.sh 'docker compose -f deploy/compose.local-vector.yaml --profile local-vector config'
|
||||||
|
|
||||||
new_fixture
|
new_fixture
|
||||||
mkdir -p "$fixture/bin"
|
mkdir -p "$fixture/bin"
|
||||||
|
|||||||
@@ -57,6 +57,9 @@ if [[ -d scripts ]]; then
|
|||||||
contract_test_files+=("${file#./}")
|
contract_test_files+=("${file#./}")
|
||||||
continue
|
continue
|
||||||
;;
|
;;
|
||||||
|
compose-with-preflight.sh|generate-connector-secrets-override.sh|unified-deployment-smoke.sh|vector-backup.sh|vector-restore.sh|vector-rotate-bootstrap-password.sh)
|
||||||
|
continue
|
||||||
|
;;
|
||||||
verify-*.sh) continue ;;
|
verify-*.sh) continue ;;
|
||||||
esac
|
esac
|
||||||
operator_files+=("${file#./}")
|
operator_files+=("${file#./}")
|
||||||
@@ -86,25 +89,51 @@ scan_category() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
for forbidden_file in \
|
for forbidden_file in \
|
||||||
|
deploy/compose.local-vector.yaml \
|
||||||
|
deploy/compose.preprocess-local-vector.yaml \
|
||||||
deploy/compose.production.yaml \
|
deploy/compose.production.yaml \
|
||||||
deploy/compose.psd-local.yaml.example \
|
deploy/compose.psd-local.yaml.example \
|
||||||
deploy/compose.psd-local.yaml \
|
deploy/compose.psd-local.yaml \
|
||||||
|
deploy/sql/20-vector-roles.sql \
|
||||||
|
deploy/vector/reconcile-roles.sh \
|
||||||
|
deploy/vector/rotate-bootstrap-password.py \
|
||||||
|
deploy/vector/secret-policy.sh \
|
||||||
|
deploy/vector/vector-db-entrypoint.sh \
|
||||||
scripts/bootstrap-local-psd-docker-config.sh \
|
scripts/bootstrap-local-psd-docker-config.sh \
|
||||||
|
scripts/local-vector-smoke.sh \
|
||||||
scripts/test-qwen-network-config.sh \
|
scripts/test-qwen-network-config.sh \
|
||||||
|
scripts/test-local-vector-smoke-safety.sh \
|
||||||
|
scripts/test-local-vector-smoke-live-collision.sh \
|
||||||
|
scripts/test-vector-bootstrap-rotation.sh \
|
||||||
|
scripts/test-vector-migration-image.sh \
|
||||||
|
scripts/test-vector-secret-policy.sh \
|
||||||
harness/tests/test_psd_local_compose_contract.py; do
|
harness/tests/test_psd_local_compose_contract.py; do
|
||||||
[[ ! -e "$forbidden_file" ]] \
|
[[ ! -e "$forbidden_file" ]] \
|
||||||
|| offenders+=("active filename: $forbidden_file (superseded deployment contract)")
|
|| offenders+=("active filename: $forbidden_file (superseded deployment contract)")
|
||||||
done
|
done
|
||||||
|
|
||||||
forbidden='omics_portal|chirone|localllm_default|datamart-builder|compose\.production\.yaml|compose\.psd-local\.yaml|\bpsd\b'
|
forbidden='omics_portal|chirone|localllm_default|datamart-builder|compose\.production\.yaml|compose\.psd-local\.yaml|\bpsd\b'
|
||||||
|
retired_semantic='local-vector|THT_VECTOR_(BOOTSTRAP|MIGRATOR|READER|WRITER|DATABASE|HOST|PORT|USER|ADMIN_URL|OPERATOR_ENV_FILE)|THT_OLLAMA_URL|VECTOR_API_KEY_(FILE|SOURCE)|vector-api-key|(^|[^A-Za-z0-9_])THT_VEC_(REST_URL|WRITE_REST_URL)'
|
||||||
scan_category runtime "$forbidden" "${runtime_files[@]}"
|
scan_category runtime "$forbidden" "${runtime_files[@]}"
|
||||||
scan_category install "$forbidden" "${install_files[@]}"
|
scan_category install "$forbidden" "${install_files[@]}"
|
||||||
scan_category operator "$forbidden" "${operator_files[@]}"
|
scan_category operator "$forbidden" "${operator_files[@]}"
|
||||||
|
scan_category runtime "$retired_semantic" "${runtime_files[@]}"
|
||||||
|
scan_category install "$retired_semantic" "${install_files[@]}"
|
||||||
|
scan_category operator "$retired_semantic" "${operator_files[@]}"
|
||||||
# Contract tests legitimately quote forbidden names in negative assertions. Scan their positive
|
# Contract tests legitimately quote forbidden names in negative assertions. Scan their positive
|
||||||
# deployment wiring constructs instead, so a provider-owned network or retired overlay cannot be
|
# deployment wiring constructs instead, so a provider-owned network or retired overlay cannot be
|
||||||
# required under a different test filename.
|
# required under a different test filename.
|
||||||
positive_contract='networks(\?|\.)?\.?localllm_default|services(\?|\.)?\.?core(\?|\.)?\.?networks(\?|\.)?\.?localllm_default|docker compose[^\n]*(compose\.psd-local|compose\.production)|THT_PSD_[A-Z0-9_]*='
|
positive_contract='networks(\?|\.)?\.?localllm_default|services(\?|\.)?\.?core(\?|\.)?\.?networks(\?|\.)?\.?localllm_default|docker compose[^\n]*(compose\.psd-local|compose\.production)|THT_PSD_[A-Z0-9_]*='
|
||||||
scan_category contract-test "$positive_contract" "${contract_test_files[@]}"
|
scan_category contract-test "$positive_contract" "${contract_test_files[@]}"
|
||||||
|
contract_scan_files=()
|
||||||
|
for file in "${contract_test_files[@]}"; do
|
||||||
|
case "${file#scripts/}" in
|
||||||
|
test-compose-secret-policy.sh|test-preprocess-compose-config.sh) continue ;;
|
||||||
|
esac
|
||||||
|
contract_scan_files+=("$file")
|
||||||
|
done
|
||||||
|
retired_semantic_contract='docker compose[^\n]*(compose\.local-vector|compose\.preprocess-local-vector)|THT_VECTOR_(BOOTSTRAP|MIGRATOR|READER|WRITER|DATABASE|HOST|PORT|USER|ADMIN_URL|OPERATOR_ENV_FILE)=|THT_OLLAMA_URL=|THT_WS_[A-Z0-9_]*_VECTOR_(TRANSPORT|API_KEY_(FILE|SOURCE))=|vector-api-key'
|
||||||
|
scan_category contract-test "$retired_semantic_contract" "${contract_scan_files[@]}"
|
||||||
|
|
||||||
if [[ -f scripts/run-stack.sh ]]; then
|
if [[ -f scripts/run-stack.sh ]]; then
|
||||||
set +e
|
set +e
|
||||||
@@ -128,4 +157,4 @@ if ((${#offenders[@]})); then
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo "no active PSD, Chirone, or portal deployment coupling found."
|
echo "no active retired deployment or external semantic coupling found."
|
||||||
|
|||||||
@@ -6,12 +6,7 @@ cd "$(dirname "$0")/.."
|
|||||||
tmp_bundle=$(mktemp)
|
tmp_bundle=$(mktemp)
|
||||||
tmp_auth=$(mktemp)
|
tmp_auth=$(mktemp)
|
||||||
trap 'rm -f "$tmp_bundle" "$tmp_auth"' EXIT HUP INT TERM
|
trap 'rm -f "$tmp_bundle" "$tmp_auth"' EXIT HUP INT TERM
|
||||||
cat >"$tmp_bundle" <<'EOF'
|
printf '%s\n' 'THT_MODEL_API_KEY=test-model' >"$tmp_bundle"
|
||||||
THT_VECTOR_BOOTSTRAP_PASSWORD=test-bootstrap
|
|
||||||
THT_VECTOR_MIGRATOR_PASSWORD=test-migrator
|
|
||||||
THT_VECTOR_READER_PASSWORD=test-reader
|
|
||||||
THT_VECTOR_WRITER_PASSWORD=test-writer
|
|
||||||
EOF
|
|
||||||
chmod 0600 "$tmp_bundle"
|
chmod 0600 "$tmp_bundle"
|
||||||
printf '%s\n' '{}' >"$tmp_auth"
|
printf '%s\n' '{}' >"$tmp_auth"
|
||||||
chmod 0600 "$tmp_auth"
|
chmod 0600 "$tmp_auth"
|
||||||
@@ -19,71 +14,44 @@ export THT_SECRETS_FILE="$tmp_bundle"
|
|||||||
export PI_AUTH_FILE="$tmp_auth"
|
export PI_AUTH_FILE="$tmp_auth"
|
||||||
export THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git
|
export THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git
|
||||||
|
|
||||||
local_files="-f compose.yaml -f deploy/compose.local-vector.yaml -f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml"
|
config_json=$(docker compose -f compose.yaml -f deploy/compose.preprocess.yaml --profile preprocess config --format json)
|
||||||
local_json=$(docker compose $local_files --profile local-vector --profile preprocess config --format json)
|
|
||||||
|
|
||||||
printf '%s' "$local_json" | python3 -c '
|
printf '%s' "$config_json" | python3 -c '
|
||||||
import json, sys
|
import json, sys
|
||||||
|
|
||||||
config = json.load(sys.stdin)
|
config = json.load(sys.stdin)
|
||||||
services = config["services"]
|
services = config["services"]
|
||||||
assert "thothii_secrets" in config.get("secrets", {}), config.get("secrets")
|
assert "qdrant" in services
|
||||||
assert "vector_bootstrap_password" not in config.get("secrets", {})
|
assert "embedding" in services
|
||||||
assert "vector_migrator_password" not in config.get("secrets", {})
|
assert "embedding-model-init" in services
|
||||||
assert "vector_reader_password" not in config.get("secrets", {})
|
assert "preprocess-evidence" in services
|
||||||
assert "vector_writer_password" not in config.get("secrets", {})
|
assert "preprocess-dwh" in services
|
||||||
for name, service in services.items():
|
for name in ("preprocess-evidence", "preprocess-dwh", "core"):
|
||||||
if name.startswith("vector-") or name.startswith("preprocess-") or name == "core":
|
|
||||||
assert any(item.get("target") == "thothii.secrets" for item in service.get("secrets", []) if isinstance(item, dict)), (name, service.get("secrets"))
|
|
||||||
assert "vector_reader_password" not in str(service)
|
|
||||||
assert "vector_writer_password" not in str(service)
|
|
||||||
for name in ("preprocess-evidence", "preprocess-dwh"):
|
|
||||||
dependency = services[name].get("depends_on", {}).get("vector-migrate")
|
|
||||||
assert dependency is not None, f"{name} does not depend on vector-migrate"
|
|
||||||
assert dependency["condition"] == "service_completed_successfully", dependency
|
|
||||||
'
|
|
||||||
|
|
||||||
external_json=$(docker compose \
|
|
||||||
-f compose.yaml -f deploy/compose.preprocess.yaml \
|
|
||||||
--profile preprocess config --format json)
|
|
||||||
|
|
||||||
printf '%s' "$external_json" | python3 -c '
|
|
||||||
import json, sys
|
|
||||||
|
|
||||||
config = json.load(sys.stdin)
|
|
||||||
services = config["services"]
|
|
||||||
assert "vector-db" not in services
|
|
||||||
assert "vector-migrate" not in services
|
|
||||||
assert "vector-reconcile" not in services
|
|
||||||
for name in ("preprocess-evidence", "preprocess-dwh"):
|
|
||||||
service = services[name]
|
service = services[name]
|
||||||
assert "depends_on" not in service
|
assert any(item.get("target") == "thothii.secrets" for item in service.get("secrets", []) if isinstance(item, dict)), (name, service.get("secrets"))
|
||||||
assert all(item.get("target") == "thothii.secrets" for item in service.get("secrets", []) if isinstance(item, dict)), service.get("secrets")
|
assert "THT_OLLAMA_URL" not in str(service)
|
||||||
assert "vector_reader_password" not in str(service)
|
assert "THT_VECTOR_" not in str(service)
|
||||||
assert "vector_writer_password" not in str(service)
|
assert services["preprocess-evidence"]["depends_on"]["qdrant"]["condition"] == "service_healthy"
|
||||||
|
assert services["preprocess-evidence"]["depends_on"]["embedding-model-init"]["condition"] == "service_completed_successfully"
|
||||||
|
assert "depends_on" not in services["preprocess-dwh"] or "vector-migrate" not in str(services["preprocess-dwh"]["depends_on"])
|
||||||
'
|
'
|
||||||
|
|
||||||
python3 - <<'PY'
|
python3 - <<'PY'
|
||||||
import os
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
os.environ.update({
|
evidence = Path("deploy/workspaces/preprocess-evidence.yaml").read_text()
|
||||||
"THT_DB_NAME": "thoth",
|
dwh = Path("deploy/workspaces/preprocess-dwh.yaml").read_text()
|
||||||
"THT_DWH_REST_URL": "http://dwh.invalid",
|
assert "type: qdrant" in evidence
|
||||||
"THT_DWH_API_KEY": "dwh",
|
assert "base_url: http://qdrant:6333" in evidence
|
||||||
"THT_VECTOR_DATABASE": "thoth",
|
assert "provider: ollama_internal" in evidence
|
||||||
"THT_VECTOR_READER_USER": "reader",
|
assert "base_url: http://embedding:11434" in evidence
|
||||||
"THT_VECTOR_WRITER_USER": "writer",
|
assert "qwen3-embedding:0.6b" in evidence
|
||||||
"THT_VECTOR_READER_PASSWORD_FILE": "/tmp/generated-reader",
|
assert "THT_VECTOR_" not in evidence
|
||||||
"THT_VECTOR_WRITER_PASSWORD_FILE": "/tmp/generated-writer",
|
assert "THT_OLLAMA_URL" not in evidence
|
||||||
"THT_DOCS_ROOT": "/data/source",
|
assert "pgvector" not in evidence
|
||||||
"THT_OLLAMA_URL": "http://ollama.invalid",
|
assert "type: postgres_direct" in dwh
|
||||||
})
|
assert "THT_PREPROCESS_DWH_HOST" in dwh
|
||||||
text = Path("deploy/workspaces/local-vector.yaml").read_text()
|
print("preprocess workspace contract: ok")
|
||||||
assert "password_file: ${THT_VECTOR_READER_PASSWORD_FILE}" in text
|
|
||||||
assert "password_file: ${THT_VECTOR_WRITER_PASSWORD_FILE}" in text
|
|
||||||
assert "${THT_SECRETS_FILE}" not in text
|
|
||||||
print("local-vector workspace resolution contract: ok")
|
|
||||||
PY
|
PY
|
||||||
|
|
||||||
echo "preprocess compose config: ok"
|
echo "preprocess compose config: ok"
|
||||||
|
|||||||
@@ -43,7 +43,7 @@ mkdir -p "$TASK13_REMOTE"
|
|||||||
workspace="$fixture/task13-smoke.yaml"
|
workspace="$fixture/task13-smoke.yaml"
|
||||||
cat >"$workspace" <<'EOF'
|
cat >"$workspace" <<'EOF'
|
||||||
workspace:
|
workspace:
|
||||||
schema_version: 2
|
schema_version: 3
|
||||||
id: task13-smoke
|
id: task13-smoke
|
||||||
name: Task 13 Smoke
|
name: Task 13 Smoke
|
||||||
language: en
|
language: en
|
||||||
@@ -54,17 +54,14 @@ dwh:
|
|||||||
supported_transports: [postgres_direct]
|
supported_transports: [postgres_direct]
|
||||||
semantic_index:
|
semantic_index:
|
||||||
vector_store:
|
vector_store:
|
||||||
engine: pgvector
|
engine: qdrant
|
||||||
database: vectors
|
collection: task13-smoke
|
||||||
schema: public
|
dimensions: 1024
|
||||||
collection: task13_documents
|
|
||||||
dimensions: 8
|
|
||||||
distance: cosine
|
distance: cosine
|
||||||
supported_transports: [pgvector_direct]
|
|
||||||
embedding:
|
embedding:
|
||||||
provider: ollama_compatible
|
provider: ollama_internal
|
||||||
model: task13-embedding
|
model: qwen3-embedding:0.6b
|
||||||
dimensions: 8
|
dimensions: 1024
|
||||||
llm_policy:
|
llm_policy:
|
||||||
default: local-qwen/task13-smoke
|
default: local-qwen/task13-smoke
|
||||||
allowed: [local-qwen/task13-smoke]
|
allowed: [local-qwen/task13-smoke]
|
||||||
|
|||||||
@@ -86,21 +86,4 @@ PATH="$fakebin:$PATH" RESTORE_LOG="$tmp/restore.log" ./scripts/vector-restore.sh
|
|||||||
grep -q -- '--single-transaction' "$tmp/restore.log"
|
grep -q -- '--single-transaction' "$tmp/restore.log"
|
||||||
grep -q -- '--exit-on-error' "$tmp/restore.log"
|
grep -q -- '--exit-on-error' "$tmp/restore.log"
|
||||||
|
|
||||||
# The live restore smoke must follow the packaged migration set instead of a stale
|
|
||||||
# hard-coded count when a new migration is added.
|
|
||||||
if grep -Eq 'vector_(bootstrap|migrator|reader|writer)_password' \
|
|
||||||
deploy/compose.local-vector.yaml deploy/compose.preprocess-local-vector.yaml; then
|
|
||||||
echo "local-vector Compose still declares legacy per-password secrets" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
grep -Fq 'thothii_secrets' deploy/compose.local-vector.yaml
|
|
||||||
grep -Fq 'thothii_secrets' deploy/compose.preprocess-local-vector.yaml
|
|
||||||
if grep -Fq 'SELECT count(*) = 3 FROM public.tht_vector_migrations' \
|
|
||||||
scripts/local-vector-smoke.sh; then
|
|
||||||
echo "local vector smoke hard-codes the pre-004 migration count" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
grep -Fq 'expected_migrations=' scripts/local-vector-smoke.sh
|
|
||||||
grep -Fq 'applied_migrations=' scripts/local-vector-smoke.sh
|
|
||||||
|
|
||||||
echo "vector backup/restore filesystem, identity, and transaction contracts passed."
|
echo "vector backup/restore filesystem, identity, and transaction contracts passed."
|
||||||
|
|||||||
@@ -1,125 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
cd "$(dirname "$0")/.."
|
|
||||||
tmp=$(mktemp -d)
|
|
||||||
trap 'rm -rf "$tmp"' EXIT HUP INT TERM
|
|
||||||
|
|
||||||
fake="$tmp/docker"
|
|
||||||
log="$tmp/docker.log"
|
|
||||||
cat >"$fake" <<'SH'
|
|
||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
invocation=$*
|
|
||||||
test "${1:-}" = compose
|
|
||||||
shift
|
|
||||||
env_file=
|
|
||||||
while [ "$#" -gt 0 ]; do
|
|
||||||
case "$1" in
|
|
||||||
--env-file)
|
|
||||||
[ "$#" -ge 2 ] || exit 64
|
|
||||||
env_file=$2
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--env-file=*)
|
|
||||||
env_file=${1#--env-file=}
|
|
||||||
shift
|
|
||||||
;;
|
|
||||||
*) shift ;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
[ -n "$env_file" ] && [ -f "$env_file" ] || {
|
|
||||||
echo "fake docker rejected missing env file: $env_file" >&2
|
|
||||||
exit 64
|
|
||||||
}
|
|
||||||
printf '%s:%s\n' "${THT_VECTOR_BOOTSTRAP_USER:-unset}" "$invocation" >>"$FAKE_DOCKER_LOG"
|
|
||||||
exit "${FAKE_DOCKER_EXIT:-0}"
|
|
||||||
SH
|
|
||||||
chmod 0755 "$fake"
|
|
||||||
|
|
||||||
printf '%s' old-password >"$tmp/old"
|
|
||||||
printf '%s' "new-'quoted-\$-password" >"$tmp/new"
|
|
||||||
cp "$tmp/old" "$tmp/original"
|
|
||||||
|
|
||||||
printf 'invalid password\n' >"$tmp/whitespace"
|
|
||||||
printf '%s\n' 'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/thoth-workspaces.git' \
|
|
||||||
>"$tmp/operator.env"
|
|
||||||
printf '%s\n' 'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/unsafe.git' \
|
|
||||||
>"$tmp/unsafe.env"
|
|
||||||
ln -s "$tmp/operator.env" "$tmp/operator-link.env"
|
|
||||||
chmod 0600 "$tmp/old" "$tmp/new" "$tmp/original" "$tmp/whitespace" "$tmp/operator.env"
|
|
||||||
chmod 0660 "$tmp/unsafe.env"
|
|
||||||
|
|
||||||
if PATH="$tmp:$PATH" FAKE_DOCKER_LOG="$log" \
|
|
||||||
./scripts/vector-rotate-bootstrap-password.sh "$tmp/old" "$tmp/new" \
|
|
||||||
>"$tmp/out" 2>"$tmp/err"; then
|
|
||||||
echo "rotation silently assumed an operator env file" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
grep -q 'requires --env-file or THT_VECTOR_OPERATOR_ENV_FILE' "$tmp/err"
|
|
||||||
test ! -s "$log"
|
|
||||||
|
|
||||||
if PATH="$tmp:$PATH" FAKE_DOCKER_LOG="$log" \
|
|
||||||
./scripts/vector-rotate-bootstrap-password.sh --env-file "$tmp/missing.env" \
|
|
||||||
"$tmp/old" "$tmp/new" >"$tmp/out" 2>"$tmp/err"; then
|
|
||||||
echo "rotation accepted a nonexistent operator env file" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
grep -q 'operator env must be a readable regular file' "$tmp/err"
|
|
||||||
test ! -s "$log"
|
|
||||||
|
|
||||||
for unsafe_env in "$tmp/unsafe.env" "$tmp/operator-link.env"; do
|
|
||||||
if PATH="$tmp:$PATH" FAKE_DOCKER_LOG="$log" \
|
|
||||||
./scripts/vector-rotate-bootstrap-password.sh --env-file "$unsafe_env" \
|
|
||||||
"$tmp/old" "$tmp/new" >"$tmp/out" 2>"$tmp/err"; then
|
|
||||||
echo "rotation accepted unsafe operator env file $unsafe_env" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
test ! -s "$log"
|
|
||||||
done
|
|
||||||
|
|
||||||
: >"$log"
|
|
||||||
if PATH="$tmp:$PATH" FAKE_DOCKER_LOG="$log" THT_VECTOR_BOOTSTRAP_USER=custom_admin \
|
|
||||||
./scripts/vector-rotate-bootstrap-password.sh --env-file "$tmp/operator.env" \
|
|
||||||
"$tmp/old" "$tmp/whitespace" \
|
|
||||||
>"$tmp/out" 2>"$tmp/err"; then
|
|
||||||
echo "rotation accepted a whitespace-containing secret" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
cmp "$tmp/old" "$tmp/original"
|
|
||||||
test ! -s "$log"
|
|
||||||
if find "$tmp" -name 'old.rotate.*' -print | grep -q .; then
|
|
||||||
echo "rotation staged a deployment file before secret validation" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if PATH="$tmp:$PATH" FAKE_DOCKER_LOG="$log" FAKE_DOCKER_EXIT=1 \
|
|
||||||
./scripts/vector-rotate-bootstrap-password.sh --env-file "$tmp/operator.env" \
|
|
||||||
"$tmp/old" "$tmp/new" \
|
|
||||||
>"$tmp/out" 2>"$tmp/err"; then
|
|
||||||
echo "rotation unexpectedly succeeded when database verification failed" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
cmp "$tmp/old" "$tmp/original"
|
|
||||||
|
|
||||||
: >"$log"
|
|
||||||
PATH="$tmp:$PATH" FAKE_DOCKER_LOG="$log" THT_VECTOR_BOOTSTRAP_USER=custom_admin \
|
|
||||||
./scripts/vector-rotate-bootstrap-password.sh --env-file "$tmp/operator.env" \
|
|
||||||
"$tmp/old" "$tmp/new" \
|
|
||||||
>"$tmp/out" 2>"$tmp/err"
|
|
||||||
cmp "$tmp/old" "$tmp/new"
|
|
||||||
grep -q -- "--env-file $tmp/operator.env" "$log"
|
|
||||||
grep -q '/run/secrets/bootstrap-old:ro' "$log"
|
|
||||||
grep -q '/run/secrets/bootstrap-new:ro' "$log"
|
|
||||||
grep -q '^custom_admin:' "$log"
|
|
||||||
grep -q 'atomically replaced only after verified database login' "$tmp/out"
|
|
||||||
|
|
||||||
printf '%s' old-password >"$tmp/old"
|
|
||||||
: >"$log"
|
|
||||||
PATH="$tmp:$PATH" FAKE_DOCKER_LOG="$log" \
|
|
||||||
THT_VECTOR_OPERATOR_ENV_FILE="$tmp/operator.env" \
|
|
||||||
./scripts/vector-rotate-bootstrap-password.sh "$tmp/old" "$tmp/new" \
|
|
||||||
>"$tmp/out" 2>"$tmp/err"
|
|
||||||
grep -q -- "--env-file $tmp/operator.env" "$log"
|
|
||||||
|
|
||||||
echo "bootstrap rotation env propagation, validation, ordering, and failure contracts passed."
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
image=${1:?usage: test-vector-migration-image.sh IMAGE [PLATFORM]}
|
|
||||||
platform=${2:-${PLATFORM:-linux/arm64}}
|
|
||||||
repo_root=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
|
|
||||||
slug=$$
|
|
||||||
network="thoth-vector-migration-$slug"
|
|
||||||
database="thoth-vector-db-$slug"
|
|
||||||
|
|
||||||
cleanup() {
|
|
||||||
docker rm --force "$database" >/dev/null 2>&1 || true
|
|
||||||
docker network rm "$network" >/dev/null 2>&1 || true
|
|
||||||
}
|
|
||||||
trap cleanup EXIT INT TERM
|
|
||||||
|
|
||||||
docker network create "$network" >/dev/null
|
|
||||||
docker run --detach --rm --platform "$platform" --name "$database" --network "$network" \
|
|
||||||
-e POSTGRES_DB=thoth -e POSTGRES_USER=thoth_admin -e POSTGRES_PASSWORD=test-only \
|
|
||||||
pgvector/pgvector:pg16 >/dev/null
|
|
||||||
|
|
||||||
attempt=0
|
|
||||||
until docker exec "$database" pg_isready -U thoth_admin -d thoth >/dev/null 2>&1; do
|
|
||||||
attempt=$((attempt + 1))
|
|
||||||
if [ "$attempt" -ge 30 ]; then
|
|
||||||
echo "pgvector test database did not become ready" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
sleep 1
|
|
||||||
done
|
|
||||||
|
|
||||||
database_url="postgresql+psycopg2://thoth_admin:test-only@$database:5432/thoth"
|
|
||||||
applied=$(docker run --rm --platform "$platform" --network "$network" \
|
|
||||||
--entrypoint /opt/venv/bin/tht -e THT_VECTOR_ADMIN_URL="$database_url" \
|
|
||||||
"$image" vector migrate --json)
|
|
||||||
status=$(docker run --rm --platform "$platform" --network "$network" \
|
|
||||||
--entrypoint /opt/venv/bin/tht -e THT_VECTOR_ADMIN_URL="$database_url" \
|
|
||||||
"$image" vector migrate --status --json)
|
|
||||||
|
|
||||||
expected_versions=$(find "$repo_root/harness/tht/migrations/vector" -type f -name '[0-9][0-9][0-9]_*.sql' \
|
|
||||||
| sed 's|.*/||; s|_.*||' \
|
|
||||||
| LC_ALL=C sort \
|
|
||||||
| awk 'BEGIN { separator = ""; printf "[" } { printf "%s\"%s\"", separator, $0; separator = ", " } END { print "]" }')
|
|
||||||
test "$expected_versions" != '[]'
|
|
||||||
expected="{\"applied\": $expected_versions, \"drifted\": [], \"pending\": []}"
|
|
||||||
test "$applied" = "$expected"
|
|
||||||
test "$status" = "$expected"
|
|
||||||
echo "core image vector migration discovery/status smoke passed"
|
|
||||||
@@ -1,58 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
cd "$(dirname "$0")/.."
|
|
||||||
tmp=$(mktemp -d)
|
|
||||||
trap 'rm -rf "$tmp"' EXIT HUP INT TERM
|
|
||||||
|
|
||||||
. ./deploy/vector/secret-policy.sh
|
|
||||||
|
|
||||||
: >"$tmp/empty"
|
|
||||||
printf 'has newline\n' >"$tmp/newline"
|
|
||||||
printf 'has space' >"$tmp/space"
|
|
||||||
printf 'safe-quoted-\047-dollar-$' >"$tmp/valid"
|
|
||||||
printf 'docker-secret' >"$tmp/docker"
|
|
||||||
printf 'owner-readonly' >"$tmp/readonly"
|
|
||||||
printf 'too-open' >"$tmp/open"
|
|
||||||
printf '# comment\n\nTHT_VECTOR_READER_PASSWORD=reader\nTHT_VECTOR_WRITER_PASSWORD=writer\n' >"$tmp/bundle"
|
|
||||||
printf 'THT_VECTOR_READER_PASSWORD=reader\nTHT_VECTOR_WRITER_PASSWORD=writer\nTHT_DWH_API_KEY=one\nTHT_DWH_API_KEY=two\n' >"$tmp/duplicate-bundle"
|
|
||||||
printf 'THT_VECTOR_READER_PASSWORD=reader\r\nTHT_VECTOR_WRITER_PASSWORD=writer\r\n' >"$tmp/crlf-bundle"
|
|
||||||
awk 'BEGIN { printf "THT_VECTOR_READER_PASSWORD="; for (i = 1; i <= 16385; i++) printf "x"; print "" }' >"$tmp/long-line-bundle"
|
|
||||||
awk 'BEGIN { for (i = 1; i <= 70000; i++) print "# filler" }' >"$tmp/large-bundle"
|
|
||||||
chmod 0600 "$tmp/valid"
|
|
||||||
chmod 0444 "$tmp/docker"
|
|
||||||
chmod 0400 "$tmp/readonly"
|
|
||||||
chmod 0640 "$tmp/open"
|
|
||||||
chmod 0600 "$tmp/bundle" "$tmp/duplicate-bundle" "$tmp/crlf-bundle" "$tmp/long-line-bundle" "$tmp/large-bundle"
|
|
||||||
|
|
||||||
for invalid in empty newline space; do
|
|
||||||
if validate_secret_file "$tmp/$invalid" "$invalid" >/dev/null 2>&1; then
|
|
||||||
echo "secret policy accepted $invalid" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
validate_secret_file "$tmp/valid" valid
|
|
||||||
validate_secret_file "$tmp/readonly" readonly
|
|
||||||
if validate_secret_file "$tmp/docker" docker >/dev/null 2>&1; then
|
|
||||||
echo "secret policy accepted world-readable host secret" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
if validate_secret_file "$tmp/open" open >/dev/null 2>&1; then
|
|
||||||
echo "secret policy accepted group-readable host secret" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
test "$(read_secret_file "$tmp/valid" valid)" = "safe-quoted-'-dollar-$"
|
|
||||||
test "$(read_bundle_secret "$tmp/bundle" THT_VECTOR_READER_PASSWORD)" = reader
|
|
||||||
test "$(read_bundle_secret "$tmp/crlf-bundle" THT_VECTOR_READER_PASSWORD)" = reader
|
|
||||||
if read_bundle_secret "$tmp/duplicate-bundle" THT_VECTOR_READER_PASSWORD >/dev/null 2>&1; then
|
|
||||||
echo "secret policy accepted a duplicate unrelated bundle key" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
for invalid_bundle in long-line-bundle large-bundle; do
|
|
||||||
if read_bundle_secret "$tmp/$invalid_bundle" THT_VECTOR_READER_PASSWORD >/dev/null 2>&1; then
|
|
||||||
echo "secret policy accepted oversized $invalid_bundle" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
done
|
|
||||||
|
|
||||||
echo "shared vector secret policy contracts passed."
|
|
||||||
@@ -285,12 +285,6 @@ services:
|
|||||||
THT_WS_TASK13_SMOKE_DWH_PORT: "5432"
|
THT_WS_TASK13_SMOKE_DWH_PORT: "5432"
|
||||||
THT_WS_TASK13_SMOKE_DWH_USER: task13_reader
|
THT_WS_TASK13_SMOKE_DWH_USER: task13_reader
|
||||||
THT_WS_TASK13_SMOKE_DWH_PASSWORD_FILE: /run/secrets/task13-runtime-password
|
THT_WS_TASK13_SMOKE_DWH_PASSWORD_FILE: /run/secrets/task13-runtime-password
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_TRANSPORT: pgvector_direct
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_HOST: vector.task13.invalid
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_PORT: "5432"
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_USER: task13_vector_reader
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_PASSWORD_FILE: /run/secrets/task13-runtime-password
|
|
||||||
THT_WS_TASK13_SMOKE_EMBEDDING_BASE_URL: http://$TASK13_LLM_CONTAINER:9000
|
|
||||||
labels:
|
labels:
|
||||||
io.thothii.task13.run: "$TASK13_RUN_ID"
|
io.thothii.task13.run: "$TASK13_RUN_ID"
|
||||||
volumes: !override
|
volumes: !override
|
||||||
@@ -399,12 +393,6 @@ services:
|
|||||||
THT_WS_TASK13_SMOKE_DWH_PORT: "5432"
|
THT_WS_TASK13_SMOKE_DWH_PORT: "5432"
|
||||||
THT_WS_TASK13_SMOKE_DWH_USER: task13_reader
|
THT_WS_TASK13_SMOKE_DWH_USER: task13_reader
|
||||||
THT_WS_TASK13_SMOKE_DWH_PASSWORD_FILE: /run/secrets/task13-runtime-password
|
THT_WS_TASK13_SMOKE_DWH_PASSWORD_FILE: /run/secrets/task13-runtime-password
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_TRANSPORT: pgvector_direct
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_HOST: vector.task13.invalid
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_PORT: "5432"
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_USER: task13_vector_reader
|
|
||||||
THT_WS_TASK13_SMOKE_VECTOR_PASSWORD_FILE: /run/secrets/task13-runtime-password
|
|
||||||
THT_WS_TASK13_SMOKE_EMBEDDING_BASE_URL: https://embedding.task13.invalid
|
|
||||||
labels:
|
labels:
|
||||||
io.thothii.task13.run: "$TASK13_RUN_ID"
|
io.thothii.task13.run: "$TASK13_RUN_ID"
|
||||||
volumes:
|
volumes:
|
||||||
@@ -461,7 +449,7 @@ task13_seed_registry() {
|
|||||||
task13_run_logged "initialize workspace seed" git -C "$TASK13_SEED" init --initial-branch=main
|
task13_run_logged "initialize workspace seed" git -C "$TASK13_SEED" init --initial-branch=main
|
||||||
cat >"$TASK13_SEED/workspaces/task13-smoke.yaml" <<'EOF'
|
cat >"$TASK13_SEED/workspaces/task13-smoke.yaml" <<'EOF'
|
||||||
workspace:
|
workspace:
|
||||||
schema_version: 2
|
schema_version: 3
|
||||||
id: task13-smoke
|
id: task13-smoke
|
||||||
name: Task 13 Smoke
|
name: Task 13 Smoke
|
||||||
language: en
|
language: en
|
||||||
@@ -472,17 +460,14 @@ dwh:
|
|||||||
supported_transports: [postgres_direct]
|
supported_transports: [postgres_direct]
|
||||||
semantic_index:
|
semantic_index:
|
||||||
vector_store:
|
vector_store:
|
||||||
engine: pgvector
|
engine: qdrant
|
||||||
database: vectors
|
collection: task13-smoke
|
||||||
schema: public
|
dimensions: 1024
|
||||||
collection: task13_documents
|
|
||||||
dimensions: 8
|
|
||||||
distance: cosine
|
distance: cosine
|
||||||
supported_transports: [pgvector_direct]
|
|
||||||
embedding:
|
embedding:
|
||||||
provider: ollama_compatible
|
provider: ollama_internal
|
||||||
model: task13-embedding
|
model: qwen3-embedding:0.6b
|
||||||
dimensions: 8
|
dimensions: 1024
|
||||||
llm_policy:
|
llm_policy:
|
||||||
default: local-qwen/task13-smoke
|
default: local-qwen/task13-smoke
|
||||||
allowed: [local-qwen/task13-smoke]
|
allowed: [local-qwen/task13-smoke]
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
root=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
|
root=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
|
||||||
. "$root/deploy/vector/secret-policy.sh"
|
. "$root/scripts/secret-file-utils.sh"
|
||||||
|
|
||||||
usage() {
|
usage() {
|
||||||
echo "usage: $0 --host HOST --database DB --user USER --password-file FILE --output FILE [--port PORT]" >&2
|
echo "usage: $0 --host HOST --database DB --user USER --password-file FILE --output FILE [--port PORT]" >&2
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
root=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
|
root=$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)
|
||||||
. "$root/deploy/vector/secret-policy.sh"
|
. "$root/scripts/secret-file-utils.sh"
|
||||||
|
|
||||||
usage() {
|
usage() {
|
||||||
echo "usage: $0 --active-host HOST --active-database DB --active-user USER --active-password-file FILE --target-host HOST --target-database DB --target-user USER --target-password-file FILE --input FILE [--active-port PORT] [--target-port PORT] [--force-nonempty]" >&2
|
echo "usage: $0 --active-host HOST --active-database DB --active-user USER --active-password-file FILE --target-host HOST --target-database DB --target-user USER --target-password-file FILE --input FILE [--active-port PORT] [--target-port PORT] [--force-nonempty]" >&2
|
||||||
|
|||||||
Executable → Regular
+3
-86
@@ -1,89 +1,6 @@
|
|||||||
#!/bin/sh
|
#!/bin/sh
|
||||||
set -eu
|
set -eu
|
||||||
|
|
||||||
cd "$(dirname "$0")/.."
|
echo "vector bootstrap password rotation is retired in this repository; local pgvector deployment is no longer supported." >&2
|
||||||
. ./deploy/vector/secret-policy.sh
|
echo "If you still need legacy pgvector data, export it with the retained migration utilities and migrate off-repository." >&2
|
||||||
|
exit 2
|
||||||
usage() {
|
|
||||||
echo "usage: $0 [--env-file OPERATOR_ENV] OLD_SECRET_FILE NEW_SECRET_FILE" >&2
|
|
||||||
echo "set THT_VECTOR_OPERATOR_ENV_FILE instead of --env-file when required by automation" >&2
|
|
||||||
exit 2
|
|
||||||
}
|
|
||||||
|
|
||||||
operator_env=${THT_VECTOR_OPERATOR_ENV_FILE:-}
|
|
||||||
while [ "$#" -gt 0 ]; do
|
|
||||||
case "$1" in
|
|
||||||
--env-file)
|
|
||||||
[ "$#" -ge 2 ] || usage
|
|
||||||
operator_env=$2
|
|
||||||
shift 2
|
|
||||||
;;
|
|
||||||
--env-file=*)
|
|
||||||
operator_env=${1#--env-file=}
|
|
||||||
shift
|
|
||||||
;;
|
|
||||||
--) shift; break ;;
|
|
||||||
-*) usage ;;
|
|
||||||
*) break ;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
|
|
||||||
if [ -z "$operator_env" ]; then
|
|
||||||
echo "rotation requires --env-file or THT_VECTOR_OPERATOR_ENV_FILE; there is no implicit default" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
if [ -L "$operator_env" ] || [ ! -f "$operator_env" ] || [ ! -r "$operator_env" ]; then
|
|
||||||
echo "operator env must be a readable regular file, not a symlink: $operator_env" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
operator_env_mode=$(stat -c '%a' "$operator_env" 2>/dev/null || stat -f '%Lp' "$operator_env" 2>/dev/null) || {
|
|
||||||
echo "cannot inspect operator env permissions: $operator_env" >&2
|
|
||||||
exit 2
|
|
||||||
}
|
|
||||||
if [ $((0$operator_env_mode & 022)) -ne 0 ]; then
|
|
||||||
echo "operator env must not be writable by group or other users: $operator_env" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ "$#" -ne 2 ]; then
|
|
||||||
usage
|
|
||||||
fi
|
|
||||||
|
|
||||||
absolute_file() {
|
|
||||||
directory=$(CDPATH= cd -- "$(dirname -- "$1")" && pwd)
|
|
||||||
printf '%s/%s\n' "$directory" "$(basename -- "$1")"
|
|
||||||
}
|
|
||||||
|
|
||||||
operator_env=$(absolute_file "$operator_env")
|
|
||||||
|
|
||||||
old_secret=$(absolute_file "$1")
|
|
||||||
new_secret=$(absolute_file "$2")
|
|
||||||
validate_secret_file "$old_secret" old_bootstrap_secret
|
|
||||||
validate_secret_file "$new_secret" new_bootstrap_secret
|
|
||||||
if [ "$old_secret" -ef "$new_secret" ]; then
|
|
||||||
echo "old and new secret files must be distinct" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
|
|
||||||
project=${COMPOSE_PROJECT_NAME:-thothii}
|
|
||||||
replacement=$(mktemp "${old_secret}.rotate.XXXXXX")
|
|
||||||
trap 'rm -f "$replacement"' EXIT HUP INT TERM
|
|
||||||
cp "$new_secret" "$replacement"
|
|
||||||
chmod 0600 "$replacement"
|
|
||||||
|
|
||||||
docker compose --env-file "$operator_env" -f compose.yaml -f deploy/compose.local-vector.yaml \
|
|
||||||
--project-name "$project" --profile local-vector run --rm --no-deps \
|
|
||||||
--user 0:0 \
|
|
||||||
--entrypoint /opt/venv/bin/python \
|
|
||||||
--volume "$old_secret:/run/secrets/bootstrap-old:ro" \
|
|
||||||
--volume "$new_secret:/run/secrets/bootstrap-new:ro" \
|
|
||||||
--volume "$(pwd)/deploy/vector/rotate-bootstrap-password.py:/opt/thoth/rotate-bootstrap-password.py:ro" \
|
|
||||||
core /opt/thoth/rotate-bootstrap-password.py \
|
|
||||||
/run/secrets/bootstrap-old /run/secrets/bootstrap-new
|
|
||||||
|
|
||||||
mv -f "$replacement" "$old_secret"
|
|
||||||
trap - EXIT HUP INT TERM
|
|
||||||
|
|
||||||
echo "Deployment bootstrap secret atomically replaced only after verified database login."
|
|
||||||
echo "Re-run with the same operator env file: $operator_env"
|
|
||||||
echo "docker compose --env-file OPERATOR_ENV -f compose.yaml -f deploy/compose.local-vector.yaml --project-name $project --profile local-vector up --wait vector-reconcile vector-migrate core"
|
|
||||||
|
|||||||
@@ -24,7 +24,6 @@ test "$(docker image inspect --format '{{ index .Config.Labels "org.opencontaine
|
|||||||
|
|
||||||
docker run --rm --platform "$platform" --entrypoint /app/docker/smoke/core-smoke.sh \
|
docker run --rm --platform "$platform" --entrypoint /app/docker/smoke/core-smoke.sh \
|
||||||
"$core_image"
|
"$core_image"
|
||||||
./scripts/test-vector-migration-image.sh "$core_image" "$platform"
|
|
||||||
docker run --rm --platform "$platform" "$frontend_image" frontend-config-smoke
|
docker run --rm --platform "$platform" "$frontend_image" frontend-config-smoke
|
||||||
./docker/smoke/frontend-policy-smoke.sh
|
./docker/smoke/frontend-policy-smoke.sh
|
||||||
if docker run --rm --platform "$platform" -e THOTH_PUBLIC_EXPOSURE=true -e AUTH_MODE=none \
|
if docker run --rm --platform "$platform" -e THOTH_PUBLIC_EXPOSURE=true -e AUTH_MODE=none \
|
||||||
|
|||||||
@@ -1122,8 +1122,8 @@ verify_local_installation_example() {
|
|||||||
node - "$rendered" <<'NODE'
|
node - "$rendered" <<'NODE'
|
||||||
const fs = require("fs");
|
const fs = require("fs");
|
||||||
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
|
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
|
||||||
if (Object.keys(config.services).sort().join(",") !== "core,frontend") {
|
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
||||||
throw new Error("local installation example must render exactly core,frontend");
|
throw new Error("local installation example must render the internal semantic stack");
|
||||||
}
|
}
|
||||||
const output = JSON.stringify(config);
|
const output = JSON.stringify(config);
|
||||||
for (const secret of [
|
for (const secret of [
|
||||||
@@ -1244,8 +1244,8 @@ verify_server_installation_example() {
|
|||||||
node - "$rendered" <<'NODE'
|
node - "$rendered" <<'NODE'
|
||||||
const fs = require("fs");
|
const fs = require("fs");
|
||||||
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
|
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
|
||||||
if (Object.keys(config.services).sort().join(",") !== "core,frontend") {
|
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
||||||
throw new Error("server installation example must render exactly core,frontend");
|
throw new Error("server installation example must render the internal semantic stack");
|
||||||
}
|
}
|
||||||
const core = config.services.core;
|
const core = config.services.core;
|
||||||
const frontend = config.services.frontend;
|
const frontend = config.services.frontend;
|
||||||
@@ -1346,7 +1346,6 @@ verify_compose_fixtures() {
|
|||||||
write_private "$fixture/git-ssh-key" 'fixture-git-ssh-key'
|
write_private "$fixture/git-ssh-key" 'fixture-git-ssh-key'
|
||||||
write_private "$fixture/git-known-hosts" 'fixture-git-known-hosts'
|
write_private "$fixture/git-known-hosts" 'fixture-git-known-hosts'
|
||||||
write_private "$fixture/dwh-password" 'fixture-dwh-password'
|
write_private "$fixture/dwh-password" 'fixture-dwh-password'
|
||||||
write_private "$fixture/vector-api-key" 'fixture-vector-api-key'
|
|
||||||
write_private "$fixture/session-runtime-password" 'fixture-session-runtime-password'
|
write_private "$fixture/session-runtime-password" 'fixture-session-runtime-password'
|
||||||
write_private "$fixture/session-migrator-password" 'fixture-session-migrator-password'
|
write_private "$fixture/session-migrator-password" 'fixture-session-migrator-password'
|
||||||
write_private "$fixture/session-ca.pem" 'fixture-session-ca'
|
write_private "$fixture/session-ca.pem" 'fixture-session-ca'
|
||||||
@@ -1355,8 +1354,6 @@ verify_compose_fixtures() {
|
|||||||
printf '%s\n' \
|
printf '%s\n' \
|
||||||
'THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct' \
|
'THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct' \
|
||||||
'THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password' \
|
'THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password' \
|
||||||
'THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api' \
|
|
||||||
'THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key' \
|
|
||||||
>"$fixture/workspace-bindings.env"
|
>"$fixture/workspace-bindings.env"
|
||||||
|
|
||||||
printf '%s\n' \
|
printf '%s\n' \
|
||||||
@@ -1367,7 +1364,6 @@ verify_compose_fixtures() {
|
|||||||
"THT_WORKSPACE_GIT_SSH_KEY_FILE=$fixture/git-ssh-key" \
|
"THT_WORKSPACE_GIT_SSH_KEY_FILE=$fixture/git-ssh-key" \
|
||||||
"THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$fixture/git-known-hosts" \
|
"THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$fixture/git-known-hosts" \
|
||||||
"THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$fixture/dwh-password" \
|
"THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$fixture/dwh-password" \
|
||||||
"THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=$fixture/vector-api-key" \
|
|
||||||
"THT_DATA_ROOT=$fixture/data" \
|
"THT_DATA_ROOT=$fixture/data" \
|
||||||
"THT_PI_STATE_ROOT=$fixture/pi-state" \
|
"THT_PI_STATE_ROOT=$fixture/pi-state" \
|
||||||
"THT_WORKSPACE_REGISTRY_ROOT=$fixture/workspace-registry" \
|
"THT_WORKSPACE_REGISTRY_ROOT=$fixture/workspace-registry" \
|
||||||
@@ -1407,8 +1403,8 @@ verify_compose_fixtures() {
|
|||||||
const fs = require("fs");
|
const fs = require("fs");
|
||||||
const [path, profile] = process.argv.slice(2);
|
const [path, profile] = process.argv.slice(2);
|
||||||
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||||
if (Object.keys(config.services).sort().join(",") !== "core,frontend") {
|
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
||||||
throw new Error(profile + ": mandatory stack must be exactly core,frontend");
|
throw new Error(profile + ": mandatory stack must include the internal semantic services");
|
||||||
}
|
}
|
||||||
const core = config.services.core;
|
const core = config.services.core;
|
||||||
for (const target of [
|
for (const target of [
|
||||||
@@ -1422,7 +1418,6 @@ for (const target of [
|
|||||||
}
|
}
|
||||||
for (const [name, value] of Object.entries({
|
for (const [name, value] of Object.entries({
|
||||||
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE: "/run/secrets/north-star-research-dwh-password",
|
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE: "/run/secrets/north-star-research-dwh-password",
|
||||||
THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE: "/run/secrets/north-star-research-vector-api-key",
|
|
||||||
})) {
|
})) {
|
||||||
if (core.environment?.[name] !== value) throw new Error(profile + ": missing binding " + name);
|
if (core.environment?.[name] !== value) throw new Error(profile + ": missing binding " + name);
|
||||||
}
|
}
|
||||||
@@ -1430,7 +1425,6 @@ const secretTargets = new Set((core.secrets || []).map((secret) => secret.target
|
|||||||
for (const target of [
|
for (const target of [
|
||||||
"thothii.secrets",
|
"thothii.secrets",
|
||||||
"north-star-research-dwh-password",
|
"north-star-research-dwh-password",
|
||||||
"north-star-research-vector-api-key",
|
|
||||||
]) {
|
]) {
|
||||||
if (!secretTargets.has(target)) throw new Error(profile + ": missing secret target " + target);
|
if (!secretTargets.has(target)) throw new Error(profile + ": missing secret target " + target);
|
||||||
}
|
}
|
||||||
@@ -1445,7 +1439,7 @@ if ((config.services.frontend.secrets || []).length !== 0) {
|
|||||||
const rendered = JSON.stringify(config);
|
const rendered = JSON.stringify(config);
|
||||||
for (const value of [
|
for (const value of [
|
||||||
"fixture-native-auth-key", "fixture-model-api-key", "fixture-git-ssh-key",
|
"fixture-native-auth-key", "fixture-model-api-key", "fixture-git-ssh-key",
|
||||||
"fixture-git-known-hosts", "fixture-dwh-password", "fixture-vector-api-key",
|
"fixture-git-known-hosts", "fixture-dwh-password",
|
||||||
"fixture-session-runtime-password", "fixture-session-migrator-password", "fixture-session-ca",
|
"fixture-session-runtime-password", "fixture-session-migrator-password", "fixture-session-ca",
|
||||||
]) {
|
]) {
|
||||||
if (rendered.includes(value)) throw new Error(profile + ": rendered Compose leaked " + value);
|
if (rendered.includes(value)) throw new Error(profile + ": rendered Compose leaked " + value);
|
||||||
|
|||||||
Reference in New Issue
Block a user