diff --git a/.env.example b/.env.example index 123c5210..7de6f78a 100644 --- a/.env.example +++ b/.env.example @@ -1,6 +1,6 @@ # Common non-secret Compose values. Select local.env or server.env with --env-file. # Run Compose with both files explicitly, for example: -# docker compose -f compose.yaml -f deploy/compose.local.yaml up -d --build +# docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up -d --build MAX_PI_PROCESSES=4 THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index be2df99b..f6e608ff 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -6,7 +6,8 @@ ## Portable deployment decoupling — LIVE 2026-08-05 - **Mandatory stack.** The supported Compose stack is exactly `frontend` plus `core`; use the - base file with `deploy/compose.local.yaml` or `deploy/compose.server.yaml`. `run-stack.sh` + base file with `deploy/compose.local.yaml`, or with `deploy/compose.server.yaml` plus the + required public-server session overlay. `run-stack.sh` invokes the base+local Compose command and the core image provides Pi, so no host Pi binary is part of the launch contract. - **External boundaries.** DWH, vector DB, embedding, LLM, and reverse-proxy services are @@ -18,6 +19,16 @@ remaining live contract checks were renamed for the generic local Compose profile. The coupling gate rejects stale active deployment filenames and content while deliberately excluding historical plans/specs, canonical workspace descriptors, and non-runtime migration helpers. +- **Fresh provider and secret contract.** Local, server, and standalone development mount the + protected Pi auth JSON plus tracked declarative model/settings files read-only under + `/home/thoth/.pi/agent`. The existing strict application bundle is a core-only Docker secret at + `/run/secrets/thothii.secrets`; operator env files contain only its absolute source path. + Provider readiness is exercised from a fresh Compose volume through model listing, configuration, + and sanitized credential status. +- **Install and scan closure.** Superseded copied one-service installation examples and the + provider-owned-network test are retired. Active manuals use the canonical base plus local/server + and optional overrides, while the category-based coupling scan covers runtime, Docker smoke, + install, operator, and positive deployment-test contracts and propagates scanner errors. ## Portable Git workspace registry — source integration (2026-08-04) diff --git a/README.md b/README.md index 6b50a234..c33557bb 100644 --- a/README.md +++ b/README.md @@ -14,14 +14,22 @@ From a fresh clone, run these commands from the repository root: ```sh cp deploy/env/local.env.example deploy/env/local.env -# Edit deploy/env/local.env, including PI_AUTH_FILE and the external endpoint URLs. +# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints. docker compose --env-file deploy/env/local.env \ -f compose.yaml -f deploy/compose.local.yaml up --build -d ``` `./scripts/run-stack.sh` runs this same base+local command in the foreground. The core image -contains its Pi runtime; no host `pi` executable is used. For a server installation, copy and -fill `deploy/env/server.env.example`, then use `-f compose.yaml -f deploy/compose.server.yaml`. +contains its Pi runtime; no host `pi` executable is used. For a server installation: + +```sh +cp deploy/env/server.env.example deploy/env/server.env +# Edit all absolute storage, Pi/secret/session files, and endpoint paths. +docker compose --env-file deploy/env/server.env \ + -f compose.yaml -f deploy/compose.server.yaml \ + -f deploy/compose.session-server.yaml.example up --build -d +``` + Workspace descriptors come from the Git remote configured by `THT_WORKSPACE_GIT_REMOTE`; their runtime endpoint and secret bindings remain installation-local. Open (set `THOTH_HTTP_PORT` in `deploy/env/local.env` to choose another @@ -164,15 +172,10 @@ with the organization's reviewed identity proxy. `AUTH_MODE=upstream` trusts thi rejects requests without the identity header. Setting `THOTH_PUBLIC_EXPOSURE=true` with any other auth mode fails during core startup. -Production credentials use the one Compose secret bundle, not an environment example. Put the -required keys in `deploy/secrets/thothii.secrets` for the selected base+server installation: - -```dotenv -THT_MODEL_API_KEY=replace-me -THT_DWH_API_KEY=replace-me -THT_VEC_API_KEY=replace-me -THT_VEC_WRITE_API_KEY=replace-me -``` +Production credentials use the existing Compose secret-bundle contract, never environment values. +Copy `deploy/secrets/thothii.secrets.example` to a protected host file, include only the required +keys, and set its absolute path as `THT_SECRETS_FILE` in the operator env. Keep Pi's native +provider auth in the separate protected file named by `PI_AUTH_FILE`. The bundle is mounted read-only as `/run/secrets/thothii.secrets` and must be mode `0600` or `0400` on the host. Docker's runtime `0444` mode is accepted only beneath `/run/secrets`; see @@ -204,9 +207,9 @@ still scrubbed. Supporting them requires a future dedicated provider-specific co The server profile stores sessions and per-user preferences directly in PostgreSQL schema `thoth_sessions`; it does not use PostgREST, browser storage, a shared session directory, or a -dual write. Start from [`deploy/compose.session-server.yaml.example`](deploy/compose.session-server.yaml.example) -and copy [`deploy/workspaces/server-sessions.yaml.example`](deploy/workspaces/server-sessions.yaml.example) -to the untracked `deploy/workspaces/server-sessions.yaml` mounted into the core container. +dual write. Use [`deploy/compose.session-server.yaml.example`](deploy/compose.session-server.yaml.example) +with the canonical base+server files and set `THT_SERVER_WORKSPACE_CONFIG` to an absolute, +protected copy of [`deploy/workspaces/server-sessions.yaml.example`](deploy/workspaces/server-sessions.yaml.example). The runtime login needs membership in the no-login database role `thoth_sessions_runtime` only. The distinct, one-shot migrator login needs migration authority and uses @@ -242,7 +245,8 @@ proxy clears the legacy identity header and the backend rejects it. Drain/stop a enable a maintenance response at the proxy, then run the migrator once and inspect its pristine JSON: ```sh -docker compose -f compose.yaml -f deploy/compose.session-server.yaml \ +docker compose --env-file deploy/env/server.env \ + -f compose.yaml -f deploy/compose.server.yaml -f deploy/compose.session-server.yaml.example \ --profile session-migrate run --rm session-migrate ``` diff --git a/compose.yaml b/compose.yaml index 3eec4da0..ff25b0cc 100644 --- a/compose.yaml +++ b/compose.yaml @@ -21,6 +21,7 @@ services: THT_WORKSPACE_GIT_AUTHOR_NAME: ${THT_WORKSPACE_GIT_AUTHOR_NAME:-Thoth Workspace Registry} THT_WORKSPACE_GIT_AUTHOR_EMAIL: ${THT_WORKSPACE_GIT_AUTHOR_EMAIL:-thoth-workspace-registry@localhost} THT_WORKSPACE_SECRET_ROOTS: /run/secrets + THT_SECRETS_FILE: /run/secrets/thothii.secrets THT_DB_NAME: ${THT_DB_NAME:-} THT_DWH_REST_URL: ${THT_DWH_REST_URL:-} THT_VEC_REST_URL: ${THT_VEC_REST_URL:-} @@ -32,8 +33,13 @@ services: - settings:/data/settings - pi-state:/home/thoth/.pi - ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro + - ./deploy/pi/models.json:/home/thoth/.pi/agent/models.json:ro + - ./deploy/pi/settings.json:/home/thoth/.pi/agent/settings.json:ro - workspace-registry:/data/workspace-registry - sessions:/data/sessions + secrets: + - source: thothii_secrets + target: thothii.secrets healthcheck: test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"] interval: 15s @@ -71,3 +77,7 @@ volumes: pi-state: workspace-registry: sessions: + +secrets: + thothii_secrets: + file: "${THT_SECRETS_FILE:?set THT_SECRETS_FILE}" diff --git a/deploy/compose.preprocess.yaml b/deploy/compose.preprocess.yaml index 0b18e140..46752d43 100644 --- a/deploy/compose.preprocess.yaml +++ b/deploy/compose.preprocess.yaml @@ -33,3 +33,6 @@ services: - thoth_data:/data - ./deploy/workspaces:/app/harness/workspaces:ro restart: "no" + +volumes: + thoth_data: diff --git a/deploy/compose.server.yaml b/deploy/compose.server.yaml index d684ae6b..789f061c 100644 --- a/deploy/compose.server.yaml +++ b/deploy/compose.server.yaml @@ -9,6 +9,8 @@ services: - ${THT_DATA_ROOT:?set THT_DATA_ROOT}:/data - ${THT_PI_STATE_ROOT:?set THT_PI_STATE_ROOT}:/home/thoth/.pi - ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro + - ./deploy/pi/models.json:/home/thoth/.pi/agent/models.json:ro + - ./deploy/pi/settings.json:/home/thoth/.pi/agent/settings.json:ro - ${THT_WORKSPACE_REGISTRY_ROOT:?set THT_WORKSPACE_REGISTRY_ROOT}:/data/workspace-registry restart: unless-stopped diff --git a/deploy/compose.session-server.yaml.example b/deploy/compose.session-server.yaml.example index fbb1bc4f..31643170 100644 --- a/deploy/compose.session-server.yaml.example +++ b/deploy/compose.session-server.yaml.example @@ -20,7 +20,7 @@ services: - source: session_ca target: session_ca.pem volumes: - - ./deploy/workspaces:/app/harness/workspaces:ro + - ${THT_SERVER_WORKSPACE_CONFIG:?set THT_SERVER_WORKSPACE_CONFIG}:/app/harness/workspaces/server-sessions.yaml:ro # Run manually during the maintenance window. It is not a dependency of core, # so the application never gains the schema-changing migrator credential. diff --git a/deploy/env.example b/deploy/env.example deleted file mode 100644 index 881dfca1..00000000 --- a/deploy/env.example +++ /dev/null @@ -1,40 +0,0 @@ -# Deprecated compatibility template; it is not loaded by Docker Compose automatically. -# New installations must copy ../.env.example to ../.env and run -# `docker compose up --build -d` from the repository root. Keep this file only for -# staged upgrades that still invoke `--env-file deploy/env.example` explicitly. -# Never put secret values in this file. - -COMPOSE_FILE=compose.yaml -COMPOSE_PROFILES= -THT_SECRETS_FILE=deploy/secrets/thothii.secrets - -PI_PROVIDER= -PI_MODEL= -PI_THINKING= -MAX_PI_PROCESSES=4 -AUTH_MODE=none - -# User-owned session storage. Keep local for the loopback-only development stack. -# The server-session overlay requires every THT_SESSION_* value below. -THT_SESSION_STORAGE=local -THT_SESSION_DB_HOST= -THT_SESSION_DB_PORT=5432 -THT_SESSION_DB_NAME= -THT_SESSION_RUNTIME_USER= -THT_SESSION_RUNTIME_PASSWORD_SOURCE= -THT_SESSION_MIGRATOR_USER= -THT_SESSION_MIGRATOR_PASSWORD_SOURCE= -THT_SESSION_DB_SSLMODE=verify-full -THT_SESSION_CA_SOURCE= - -THT_DB_NAME= -THT_DWH_REST_URL= -THT_VEC_REST_URL= -THT_OLLAMA_URL= -THT_DOCS_ROOT=/data/workspaces/example/evidence-source - -THT_VECTOR_DATABASE=thoth -THT_VECTOR_BOOTSTRAP_USER=postgres -THT_VECTOR_MIGRATOR_USER=thoth_vector_migrator -THT_VECTOR_READER_USER=thoth_vector_reader -THT_VECTOR_WRITER_USER=thoth_vector_writer diff --git a/deploy/env/local.env.example b/deploy/env/local.env.example index 40881f8f..ddd341a6 100644 --- a/deploy/env/local.env.example +++ b/deploy/env/local.env.example @@ -4,6 +4,7 @@ THOTH_HTTP_PORT=8080 THOTH_CORE_HTTP_PORT=8787 MAX_PI_PROCESSES=4 PI_AUTH_FILE=/absolute/path/to/pi-auth.json +THT_SECRETS_FILE=/absolute/path/to/thothii.secrets THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git THT_WORKSPACE_GIT_BRANCH=main diff --git a/deploy/env/server.env.example b/deploy/env/server.env.example index cc080031..e591630a 100644 --- a/deploy/env/server.env.example +++ b/deploy/env/server.env.example @@ -4,10 +4,12 @@ THOTH_SERVER_BIND=127.0.0.1 THOTH_HTTP_PORT=8080 MAX_PI_PROCESSES=4 PI_AUTH_FILE=/absolute/path/to/pi-auth.json +THT_SECRETS_FILE=/absolute/path/to/thothii.secrets THT_DATA_ROOT=/srv/thothii/data THT_PI_STATE_ROOT=/srv/thothii/pi-state THT_WORKSPACE_REGISTRY_ROOT=/srv/thothii/workspace-registry +THT_SERVER_WORKSPACE_CONFIG=/absolute/path/to/server-sessions.yaml THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git THT_WORKSPACE_GIT_BRANCH=main THT_WORKSPACE_GIT_AUTHOR_NAME="Thoth Workspace Registry" @@ -19,3 +21,14 @@ THT_VEC_REST_URL=https://vector.example.invalid THT_VEC_WRITE_REST_URL=https://vector-write.example.invalid THT_OLLAMA_URL=https://embeddings.example.invalid THT_LLM_URL=https://llm.example.invalid + +# Public server session storage. Values are endpoints, roles, or protected source-file paths. +THT_SESSION_DB_HOST=sessions-db.example.invalid +THT_SESSION_DB_PORT=5432 +THT_SESSION_DB_NAME=thoth_sessions +THT_SESSION_RUNTIME_USER=thoth_sessions_app +THT_SESSION_MIGRATOR_USER=thoth_sessions_migrate +THT_SESSION_DB_SSLMODE=verify-full +THT_SESSION_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/session-runtime-password +THT_SESSION_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/session-migrator-password +THT_SESSION_CA_SOURCE=/absolute/path/to/session-ca.pem diff --git a/deploy/secrets/README.md b/deploy/secrets/README.md index 8ea67f50..55fed4ad 100644 --- a/deploy/secrets/README.md +++ b/deploy/secrets/README.md @@ -12,7 +12,7 @@ The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). T keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_VEC_API_KEY`, `THT_VEC_WRITE_API_KEY`, and the four `THT_VECTOR_*_PASSWORD` role passwords. Values must be non-empty and contain no whitespace. Do not put secrets in the root `.env`, workspace YAML, -URLs, logs, or `docker compose config` output. +URLs, logs, or rendered Compose output. Compose mounts the bundle read-only as `/run/secrets/thothii.secrets`. The host file must be a regular non-symlink file with mode `0600` or `0400`; Docker's normal `0444` mode is accepted @@ -20,7 +20,9 @@ only for the runtime mount beneath `/run/secrets`. The core runs as UID 10001. V without printing its contents: ```sh -docker compose run --rm core sh -c 'id && test -r /run/secrets/thothii.secrets' +docker compose --env-file deploy/env/local.env \ + -f compose.yaml -f deploy/compose.local.yaml \ + run --rm core sh -c 'id && test -r /run/secrets/thothii.secrets' ``` A private CA PEM chain is not a bundle value: PEM whitespace is rejected by the strict parser. @@ -31,9 +33,10 @@ Compose files intentionally do not create this mount. ## Migration from separate secret files Older installations used `THT_*_SECRET_FILE` variables and one file per value. Migrate by -copying each value to its bundle key, validating with `docker compose config --quiet`, and only +copying each value to its bundle key, validating with the complete base+profile command, and only then deleting the old files. The old variables remain a compatibility path for staged upgrades, -but the documented and tested default is `THT_SECRETS_FILE=deploy/secrets/thothii.secrets`. +but the documented and tested default is an absolute `THT_SECRETS_FILE` path to the protected +bundle. The local-vector bootstrap rotation helper still accepts an old/new password file as its maintenance interface. Run it only with files protected by `0600`, then copy the resulting diff --git a/deploy/thothii.env.example b/deploy/thothii.env.example deleted file mode 100644 index 1fb73b94..00000000 --- a/deploy/thothii.env.example +++ /dev/null @@ -1,33 +0,0 @@ -# ThothII core — env di runtime (compose env_file). -# Copiare in deploy/thothii.env e completare. NON committare thothii.env. - -# --- DWH (direct, ruolo read-only su schema datawarehouse) --- -THT_DB_HOST=host.docker.internal -THT_DB_PORT=5438 -THT_DB_NAME=postgres -THT_DB_USER=thoth_dwh_reader -THT_DB_PASSWORD=__CHANGE_ME__ - -# --- Vector (direct, ruolo read+write su schema vectors; stessa istanza del DWH) --- -THT_VEC_HOST=host.docker.internal -THT_VEC_PORT=5438 -THT_VEC_USER=thoth_vector_rw -THT_VEC_PASSWORD=__CHANGE_ME__ - -# --- Embeddings (Ollama sull'host, modello nomic-embed-text-v2-moe) --- -THT_OLLAMA_URL=http://host.docker.internal:11434 - -# --- Backend --- -AUTH_MODE=none # none | mock | oidc (upstream auth is enforced at the proxy boundary) -MAX_PI_PROCESSES=4 -THT_DEV_EVIDENCE_HOST_PATH=/absolute/path/to/evidence - -# --- Git-backed workspace registry (no secret values belong in this file) --- -THT_WORKSPACE_REGISTRY_ROOT=/data/workspace-registry -THT_WORKSPACE_GIT_BRANCH=main -THT_WORKSPACE_INSTALLATION_ID=server -# THT_WORKSPACE_GIT_REMOTE=ssh://git@your-git-host/your-org/thoth-workspaces.git -# THT_WORKSPACE_GIT_CREDENTIALS_FILE=/absolute/path/to/git-credentials -# THT_WORKSPACE_GIT_CA_FILE=/absolute/path/to/git-ca.pem -# THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/to/git-ssh-key -# THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/to/git-known-hosts diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index 26443ba2..8803e539 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -1,7 +1,7 @@ -# ThothII — deploy STANDALONE locale (dev / smoke test). -# Rete propria + porte host per ispezione diretta. -# docker compose -f docker-compose.dev.yml up -d --build -# frontend: http://localhost:8090 backend: http://localhost:8787 +# ThothII standalone development/smoke stack. +# Run with the canonical local env file: +# docker compose --env-file deploy/env/local.env -f docker-compose.dev.yml up -d --build +# frontend: http://localhost:8090 backend: http://localhost:8787 name: thothii-dev services: @@ -10,19 +10,18 @@ services: context: . dockerfile: docker/core.Dockerfile image: thothii-core:local - env_file: - - path: deploy/thothii.env - required: false environment: HOST: 0.0.0.0 PORT: "8787" THT_HARNESS_DIR: /app/harness THT_BIN: /opt/venv/bin/tht PI_BIN: pi - AUTH_MODE: ${AUTH_MODE:-none} + AUTH_MODE: none THT_SESSION_STORAGE: local THT_HOME: /data/local-home + THT_DATA_ROOT: /data SETTINGS_FILE: /data/settings/settings.json + THT_MAINTENANCE_FILE: /data/settings/maintenance.json THT_WORKSPACE_REGISTRY_ROOT: /data/workspace-registry THT_WORKSPACE_GIT_REMOTE: ${THT_WORKSPACE_GIT_REMOTE:?set THT_WORKSPACE_GIT_REMOTE} THT_WORKSPACE_GIT_BRANCH: ${THT_WORKSPACE_GIT_BRANCH:-main} @@ -30,26 +29,35 @@ services: THT_WORKSPACE_GIT_AUTHOR_NAME: ${THT_WORKSPACE_GIT_AUTHOR_NAME:-Thoth Workspace Registry} THT_WORKSPACE_GIT_AUTHOR_EMAIL: ${THT_WORKSPACE_GIT_AUTHOR_EMAIL:-thoth-workspace-registry@localhost} THT_WORKSPACE_SECRET_ROOTS: /run/secrets - GIT_CONFIG_COUNT: "2" - GIT_CONFIG_KEY_0: credential.helper - GIT_CONFIG_VALUE_0: store --file=/run/secrets/workspace-registry-git-credentials - GIT_CONFIG_KEY_1: http.sslCAInfo - GIT_CONFIG_VALUE_1: /run/secrets/workspace-registry-git-ca - GIT_SSH_COMMAND: ssh -i /run/secrets/workspace-registry-git-ssh-key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/run/secrets/workspace-registry-git-known-hosts + THT_SECRETS_FILE: /run/secrets/thothii.secrets + THT_DB_NAME: ${THT_DB_NAME:-} + THT_DWH_REST_URL: ${THT_DWH_REST_URL:-} + THT_VEC_REST_URL: ${THT_VEC_REST_URL:-} + THT_VEC_WRITE_REST_URL: ${THT_VEC_WRITE_REST_URL:-} + THT_OLLAMA_URL: ${THT_OLLAMA_URL:-} + THT_LLM_URL: ${THT_LLM_URL:-} MAX_PI_PROCESSES: ${MAX_PI_PROCESSES:-4} extra_hosts: - "host.docker.internal:host-gateway" volumes: - dev-data:/data - - workspace-registry:/data/workspace-registry - dev-pi-state:/home/thoth/.pi + - ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro + - ./deploy/pi/models.json:/home/thoth/.pi/agent/models.json:ro + - ./deploy/pi/settings.json:/home/thoth/.pi/agent/settings.json:ro + - workspace-registry:/data/workspace-registry - ${THT_DEV_EVIDENCE_HOST_PATH:-./evidence}:/data/evidence:ro - - ${THT_WORKSPACE_GIT_CREDENTIALS_FILE:-/dev/null}:/run/secrets/workspace-registry-git-credentials:ro - - ${THT_WORKSPACE_GIT_CA_FILE:-/etc/ssl/certs/ca-certificates.crt}:/run/secrets/workspace-registry-git-ca:ro - - ${THT_WORKSPACE_GIT_SSH_KEY_FILE:-/dev/null}:/run/secrets/workspace-registry-git-ssh-key:ro - - ${THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE:-/dev/null}:/run/secrets/workspace-registry-git-known-hosts:ro + secrets: + - source: thothii_secrets + target: thothii.secrets ports: - "127.0.0.1:8787:8787" + healthcheck: + test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"] + interval: 15s + timeout: 3s + retries: 5 + start_period: 30s restart: "no" networks: [thothii-net] @@ -64,7 +72,8 @@ services: ports: - "127.0.0.1:8090:8080" depends_on: - - core + core: + condition: service_healthy restart: "no" networks: [thothii-net] @@ -76,3 +85,7 @@ volumes: dev-data: dev-pi-state: workspace-registry: + +secrets: + thothii_secrets: + file: "${THT_SECRETS_FILE:?set THT_SECRETS_FILE}" diff --git a/docs/index.md b/docs/index.md index fa2a74c7..e6ac4f2b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,8 +8,8 @@ La documentazione è divisa in due aree: Come funziona il sistema: architettura, specifiche di design delle singole funzionalità, piani di implementazione, report di test. Parte da qui: [Panoramica dell'architettura](architecture/overview.md). -Per installare l'applicazione in Docker nei quattro contesti operativi, partendo dal comando -predefinito `docker compose up --build -d` e dal bundle unico dei secret: +Per installare l'applicazione in Docker nei quattro contesti operativi, usando il file env, +`compose.yaml`, l'overlay locale/server e il bundle di secret montato: [Installazione Docker nei quattro contesti](installazione-docker-4-contesti.md). ## Considerazioni Generali diff --git a/docs/install/examples/git-https.workspace-registry.yaml b/docs/install/examples/git-https.workspace-registry.yaml deleted file mode 100644 index 562116df..00000000 --- a/docs/install/examples/git-https.workspace-registry.yaml +++ /dev/null @@ -1,13 +0,0 @@ -# Optional override for an HTTPS Git remote. Both source paths are required absolute paths to -# existing operator-managed files; neither file content belongs in the base Compose example. -services: - core: - environment: - GIT_CONFIG_COUNT: "2" - GIT_CONFIG_KEY_0: credential.helper - GIT_CONFIG_VALUE_0: store --file=/run/secrets/workspace-registry-git-credentials - GIT_CONFIG_KEY_1: http.sslCAInfo - GIT_CONFIG_VALUE_1: /run/secrets/workspace-registry-git-ca - volumes: - - ${THT_WORKSPACE_GIT_CREDENTIALS_FILE:?set THT_WORKSPACE_GIT_CREDENTIALS_FILE}:/run/secrets/workspace-registry-git-credentials:ro - - ${THT_WORKSPACE_GIT_CA_FILE:?set THT_WORKSPACE_GIT_CA_FILE}:/run/secrets/workspace-registry-git-ca:ro diff --git a/docs/install/examples/git-ssh.workspace-registry.yaml b/docs/install/examples/git-ssh.workspace-registry.yaml deleted file mode 100644 index 2c46be3f..00000000 --- a/docs/install/examples/git-ssh.workspace-registry.yaml +++ /dev/null @@ -1,9 +0,0 @@ -# Optional override for an SSH Git remote. Source paths are required absolute operator-managed -# files. Host-key checking remains strict; do not add a fallback known-hosts or key mount. -services: - core: - environment: - GIT_SSH_COMMAND: ssh -i /run/secrets/workspace-registry-git-ssh-key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/run/secrets/workspace-registry-git-known-hosts - volumes: - - ${THT_WORKSPACE_GIT_SSH_KEY_FILE:?set THT_WORKSPACE_GIT_SSH_KEY_FILE}:/run/secrets/workspace-registry-git-ssh-key:ro - - ${THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE:?set THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE}:/run/secrets/workspace-registry-git-known-hosts:ro diff --git a/docs/install/examples/local-compose.workspace-registry.yaml b/docs/install/examples/local-compose.workspace-registry.yaml deleted file mode 100644 index 460c91ed..00000000 --- a/docs/install/examples/local-compose.workspace-registry.yaml +++ /dev/null @@ -1,39 +0,0 @@ -# Standalone local registry example. Copy to an untracked operator directory and set the absolute -# THT_SOURCE_ROOT in .env. Add only the selected Git transport override from this directory. -name: thothii-workspace-registry-local - -services: - core: - image: thothii-core:local - build: - context: ${THT_SOURCE_ROOT:?set THT_SOURCE_ROOT to the absolute ThothII source checkout} - dockerfile: docker/core.Dockerfile - env_file: - - path: ${THT_WORKSPACE_BINDINGS_ENV_FILE:?set THT_WORKSPACE_BINDINGS_ENV_FILE to an absolute THT_WS bindings file} - required: true - environment: - HOST: 0.0.0.0 - PORT: "8787" - AUTH_MODE: none - THT_SESSION_STORAGE: local - THT_HOME: /data/local-home - SETTINGS_FILE: /data/settings/settings.json - THT_HARNESS_DIR: /app/harness - THT_BIN: /opt/venv/bin/tht - THT_WORKSPACE_REGISTRY_ROOT: /data/workspace-registry - THT_WORKSPACE_GIT_REMOTE: ${THT_WORKSPACE_GIT_REMOTE:-ssh://git@git.example.invalid/platform/thoth-workspaces.git} - THT_WORKSPACE_GIT_BRANCH: ${THT_WORKSPACE_GIT_BRANCH:-main} - THT_WORKSPACE_INSTALLATION_ID: ${THT_WORKSPACE_INSTALLATION_ID:-local-laptop} - THT_WORKSPACE_GIT_AUTHOR_NAME: ${THT_WORKSPACE_GIT_AUTHOR_NAME:-Thoth Workspace Registry} - THT_WORKSPACE_GIT_AUTHOR_EMAIL: ${THT_WORKSPACE_GIT_AUTHOR_EMAIL:-thoth-workspace-registry@localhost} - THT_WORKSPACE_SECRET_ROOTS: /run/secrets - ports: - - "127.0.0.1:8787:8787" - volumes: - - thoth-local-data:/data - - workspace-registry:/data/workspace-registry - restart: "no" - -volumes: - thoth-local-data: {} - workspace-registry: {} diff --git a/docs/install/examples/server-compose.workspace-registry.yaml b/docs/install/examples/server-compose.workspace-registry.yaml deleted file mode 100644 index fecc6dbb..00000000 --- a/docs/install/examples/server-compose.workspace-registry.yaml +++ /dev/null @@ -1,60 +0,0 @@ -# Server registry example. Copy to a reviewed, untracked operator directory and set absolute host -# paths and Git values in .env. Add a selected Git transport override from this directory. -name: thothii-workspace-registry-server - -services: - core: - image: thothii-core:local - build: - context: ${THT_SOURCE_ROOT:?set THT_SOURCE_ROOT to the absolute ThothII source checkout} - dockerfile: docker/core.Dockerfile - env_file: - - path: ${THT_WORKSPACE_BINDINGS_ENV_FILE:?set THT_WORKSPACE_BINDINGS_ENV_FILE to an absolute THT_WS bindings file} - required: true - environment: - HOST: 0.0.0.0 - PORT: "8787" - AUTH_MODE: upstream - THOTH_PUBLIC_EXPOSURE: "true" - THT_SESSION_STORAGE: postgres - THT_CONFIG: /app/harness/workspaces/server-sessions.yaml - THT_SESSION_DB_HOST: ${THT_SESSION_DB_HOST:?set THT_SESSION_DB_HOST} - THT_SESSION_DB_PORT: ${THT_SESSION_DB_PORT:-5432} - THT_SESSION_DB_NAME: ${THT_SESSION_DB_NAME:?set THT_SESSION_DB_NAME} - THT_SESSION_RUNTIME_USER: ${THT_SESSION_RUNTIME_USER:?set THT_SESSION_RUNTIME_USER} - THT_SESSION_RUNTIME_PASSWORD_FILE: /run/secrets/session_runtime_password - THT_SESSION_DB_SSLMODE: ${THT_SESSION_DB_SSLMODE:-verify-full} - THT_SESSION_DB_SSLROOTCERT: /run/secrets/session_ca.pem - THT_HARNESS_DIR: /app/harness - THT_BIN: /opt/venv/bin/tht - SETTINGS_FILE: /data/settings/settings.json - THT_WORKSPACE_REGISTRY_ROOT: /data/workspace-registry - THT_WORKSPACE_GIT_REMOTE: ${THT_WORKSPACE_GIT_REMOTE:-ssh://git@git.example.invalid/platform/thoth-workspaces.git} - THT_WORKSPACE_GIT_BRANCH: ${THT_WORKSPACE_GIT_BRANCH:-main} - THT_WORKSPACE_INSTALLATION_ID: ${THT_WORKSPACE_INSTALLATION_ID:-production-1} - THT_WORKSPACE_GIT_AUTHOR_NAME: ${THT_WORKSPACE_GIT_AUTHOR_NAME:-Thoth Workspace Registry} - THT_WORKSPACE_GIT_AUTHOR_EMAIL: ${THT_WORKSPACE_GIT_AUTHOR_EMAIL:-thoth-workspace-registry@localhost} - THT_WORKSPACE_SECRET_ROOTS: /run/secrets - volumes: - - ${THT_HOST_DATA_ROOT:-/srv/thothii/data}:/data - - ${THT_WORKSPACE_REGISTRY_HOST_PATH:-/srv/thothii/workspace-registry}:/data/workspace-registry - - ${THT_SERVER_WORKSPACE_CONFIG:?set THT_SERVER_WORKSPACE_CONFIG}:/app/harness/workspaces/server-sessions.yaml:ro - secrets: - - source: session_runtime_password - target: session_runtime_password - - source: session_ca - target: session_ca.pem - networks: - - upstream - restart: unless-stopped - -networks: - upstream: - external: true - name: ${THT_UPSTREAM_NETWORK:-thothii-upstream} - -secrets: - session_runtime_password: - file: ${THT_SESSION_RUNTIME_PASSWORD_SOURCE:?set THT_SESSION_RUNTIME_PASSWORD_SOURCE} - session_ca: - file: ${THT_SESSION_CA_SOURCE:?set THT_SESSION_CA_SOURCE} diff --git a/docs/install/examples/workspace-bindings.env.example b/docs/install/examples/workspace-bindings.env.example index fe511fc6..6b6a8263 100644 --- a/docs/install/examples/workspace-bindings.env.example +++ b/docs/install/examples/workspace-bindings.env.example @@ -1,13 +1,13 @@ # Copy to an untracked operator file. This file contains only non-secret THT_WS_* bindings. # Every *_FILE value is a container path supplied by the generated local connector override. -THT_WS_PSD_CLINICAL_DWH_TRANSPORT=postgres_direct -THT_WS_PSD_CLINICAL_DWH_HOST=dwh.internal.example -THT_WS_PSD_CLINICAL_DWH_PORT=5432 -THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader -THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-clinical-dwh-password -THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=pgvector_direct -THT_WS_PSD_CLINICAL_VECTOR_HOST=vector.internal.example -THT_WS_PSD_CLINICAL_VECTOR_PORT=5432 -THT_WS_PSD_CLINICAL_VECTOR_USER=thoth_vector_reader -THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE=/run/secrets/psd-clinical-vector-password -THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL=https://embeddings.internal.example +THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct +THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.internal.example +THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432 +THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader +THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password +THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=pgvector_direct +THT_WS_NORTH_STAR_RESEARCH_VECTOR_HOST=vector.internal.example +THT_WS_NORTH_STAR_RESEARCH_VECTOR_PORT=5432 +THT_WS_NORTH_STAR_RESEARCH_VECTOR_USER=thoth_vector_reader +THT_WS_NORTH_STAR_RESEARCH_VECTOR_PASSWORD_FILE=/run/secrets/north-star-research-vector-password +THT_WS_NORTH_STAR_RESEARCH_EMBEDDING_BASE_URL=https://embeddings.internal.example diff --git a/docs/install/local-workspace-registry.md b/docs/install/local-workspace-registry.md index 391f6877..1e06a900 100644 --- a/docs/install/local-workspace-registry.md +++ b/docs/install/local-workspace-registry.md @@ -34,14 +34,16 @@ workspaces/.md For SSH, use a scoped deploy key, a verified `known_hosts` file, and strict host-key checking. For HTTPS, use Git Credential Manager or a secret-manager-created credentials file. Mount a private HTTPS CA as its own file. Do not disable host or certificate verification. The base Compose file -does not mount a Git credential: add exactly one optional `git-ssh.workspace-registry.yaml` or -`git-https.workspace-registry.yaml` override, so unused credential paths are never bind-mounted. +does not mount a Git credential: add exactly one optional `deploy/compose.git-ssh.yaml` or +`deploy/compose.git-https.yaml` override, so unused credential paths are never bind-mounted. ```dotenv THT_WORKSPACE_GIT_REMOTE=ssh://git@git.example.invalid/platform/thoth-workspaces.git THT_WORKSPACE_GIT_BRANCH=main THT_WORKSPACE_INSTALLATION_ID=local-laptop THT_SOURCE_ROOT=/absolute/path/to/ThothII +PI_AUTH_FILE=/absolute/path/installation-secrets/pi-auth.json +THT_SECRETS_FILE=/absolute/path/installation-secrets/thothii.secrets THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/installation-secrets/git-ssh-key THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/installation-secrets/git-known-hosts THT_WORKSPACE_GIT_CA_FILE=/absolute/path/installation-secrets/git-ca.pem @@ -69,12 +71,12 @@ state/ # active revision and registry state locks/ # short-lived publish locks ``` -Installation variables are deterministic: `psd-clinical` becomes `PSD_CLINICAL`, and every name +Installation variables are deterministic: `north-star-research` becomes `NORTH_STAR_RESEARCH`, and every name is `THT_WS___`. Copy [the bindings env example](examples/workspace-bindings.env.example) to an untracked operator file and set its absolute path as `THT_WORKSPACE_BINDINGS_ENV_FILE`. It is loaded only into `core`. Credentials and certificates use `*_FILE` path variables that must point inside `/run/secrets`. -If declared, `THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE` is distinct from the vector reader +If declared, `THT_WS_NORTH_STAR_RESEARCH_VECTOR_WRITER_API_KEY_FILE` is distinct from the vector reader file; a reader credential is never repurposed for writing. ## Direct PostgreSQL, REST, and SSH tunnel bindings @@ -87,41 +89,41 @@ copy or maintain a workspace-specific Compose override. ```dotenv # Direct PostgreSQL and pgvector -THT_WS_PSD_CLINICAL_DWH_TRANSPORT=postgres_direct -THT_WS_PSD_CLINICAL_DWH_HOST=dwh.example.invalid -THT_WS_PSD_CLINICAL_DWH_PORT=5432 -THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader -THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader -THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=pgvector_direct -THT_WS_PSD_CLINICAL_VECTOR_HOST=vector.example.invalid -THT_WS_PSD_CLINICAL_VECTOR_PORT=5432 -THT_WS_PSD_CLINICAL_VECTOR_USER=thoth_vector_reader -THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE=/run/secrets/psd-vector-reader -THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL=https://embeddings.example.invalid +THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct +THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.example.invalid +THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432 +THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader +THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password +THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=pgvector_direct +THT_WS_NORTH_STAR_RESEARCH_VECTOR_HOST=vector.example.invalid +THT_WS_NORTH_STAR_RESEARCH_VECTOR_PORT=5432 +THT_WS_NORTH_STAR_RESEARCH_VECTOR_USER=thoth_vector_reader +THT_WS_NORTH_STAR_RESEARCH_VECTOR_PASSWORD_FILE=/run/secrets/north-star-research-vector-password +THT_WS_NORTH_STAR_RESEARCH_EMBEDDING_BASE_URL=https://embeddings.example.invalid ``` ```dotenv # REST; an API-key file is needed only for a declared bearer/x-api-key diagnostic. -THT_WS_PSD_CLINICAL_DWH_TRANSPORT=rest_api -THT_WS_PSD_CLINICAL_DWH_BASE_URL=https://dwh.example.invalid -THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE=/run/secrets/psd-dwh-api-key -THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=rest_api -THT_WS_PSD_CLINICAL_VECTOR_BASE_URL=https://vectors.example.invalid -THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE=/run/secrets/psd-vector-api-key +THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api +THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.example.invalid +THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key +THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api +THT_WS_NORTH_STAR_RESEARCH_VECTOR_BASE_URL=https://vectors.example.invalid +THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key ``` ```dotenv # SSH tunnel diagnostic only; runtime sessions are fail-closed in this release. -THT_WS_PSD_CLINICAL_DWH_TRANSPORT=ssh_tunnel -THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader -THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader -THT_WS_PSD_CLINICAL_DWH_SSH_HOST=bastion.example.invalid -THT_WS_PSD_CLINICAL_DWH_SSH_PORT=22 -THT_WS_PSD_CLINICAL_DWH_SSH_USER=thoth_tunnel -THT_WS_PSD_CLINICAL_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/psd-dwh-tunnel-key -THT_WS_PSD_CLINICAL_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/psd-dwh-known-hosts -THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_HOST=dwh.internal.example -THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_PORT=5432 +THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel +THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader +THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.example.invalid +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22 +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432 ``` Repeat the SSH names for `VECTOR` where needed. REST diagnostics reject a private per-request CA @@ -134,31 +136,36 @@ before creating sessions. Git pull/push over SSH remains fully supported and is ## Bootstrap, first pull, and diagnostics -Copy [the local Compose example](examples/local-compose.workspace-registry.yaml), exactly one -selected [SSH Git override](examples/git-ssh.workspace-registry.yaml) or [HTTPS Git override](examples/git-https.workspace-registry.yaml), -and [the bindings env example](examples/workspace-bindings.env.example) into an untracked operator -directory. Keep `THT_SOURCE_ROOT` and the absolute `THT_WORKSPACE_BINDINGS_ENV_FILE` in its `.env` -for Compose interpolation; this keeps the copied Compose file buildable and confines `THT_WS_*` -values to `core`. A Compose `.env` file is not a shell environment, so do not import it into the -maintenance shell. Instead, explicitly export the two non-secret paths before running the commands. -Create the host secret files named by the selected Git transport and every declared connector -`*_SOURCE`, then generate the connector override and render through the preflight wrapper. The -wrapper is required: it rejects unsafe source paths and a combined SSH+HTTPS Git selection before -Compose runs. +Use the repository's canonical `compose.yaml` plus `deploy/compose.local.yaml`; they always start +the mandatory `frontend` and `core` services. Do not copy or maintain a standalone application +Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an +untracked operator directory and create a protected operator env file from +`deploy/env/local.env.example`. It must contain absolute `PI_AUTH_FILE`, +`THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths. +The Pi auth JSON, runtime secret bundle, and each connector credential remain separate protected +host files and are mounted read-only; their contents never enter the operator env or rendered +Compose. + +Select exactly one repository Git transport override, `deploy/compose.git-ssh.yaml` or +`deploy/compose.git-https.yaml`. A Compose env file is not a shell environment, so export only the +non-secret paths required by the maintenance commands. Generate the connector override and render +through the preflight wrapper, which rejects unsafe paths and combined SSH+HTTPS selection. ```sh export THT_SOURCE_ROOT=/absolute/path/to/ThothII -export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env" -"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env .env --output connector-secrets.local.yaml -"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \ - -f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml config --quiet +export THT_OPERATOR_ENV=/absolute/path/to/operator/local.env +export THT_WORKSPACE_BINDINGS_ENV_FILE=/absolute/path/to/operator/workspace-bindings.env +export THT_CONNECTOR_OVERRIDE=/absolute/path/to/operator/connector-secrets.local.yaml +"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE" +"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \ + -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \ + -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" config --quiet ``` -From the operator directory: - ```sh -"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \ - -f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml up --build -d +"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \ + -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \ + -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" up --build -d curl --fail --silent http://127.0.0.1:8787/health curl --fail --silent http://127.0.0.1:8787/workspace-registry/status curl --fail --silent http://127.0.0.1:8787/workspaces @@ -169,7 +176,7 @@ Use `POST /workspace-registry/pull` to fetch later revisions. Run workspace diag required bindings are mounted. The optional writer probe uses a distinct writer file and removes its uniquely named temporary record; ordinary diagnostics are read-only. -To migrate an existing PSD descriptor, create/clone an empty private remote, set the absolute +To migrate an existing legacy descriptor, create/clone an empty private remote, set the absolute `THT_SOURCE_ROOT`, transform with absolute paths, review the schema-v1 result, explicitly add vector database/schema and the complete schema-v2 contract, then commit/push. The transformer never imports `${ENV}` values or secrets. @@ -177,7 +184,7 @@ never imports `${ENV}` values or secrets. ```sh THT_SOURCE_ROOT=/absolute/path/to/ThothII npm --prefix "$THT_SOURCE_ROOT/backend" run build -node "$THT_SOURCE_ROOT/backend/dist/workspaces/migrate-legacy.js" --input /absolute/path/psd.yaml --output /absolute/path/thoth-workspaces +node "$THT_SOURCE_ROOT/backend/dist/workspaces/migrate-legacy.js" --input /absolute/path/legacy.yaml --output /absolute/path/thoth-workspaces ``` ## Publish, update, backup, outage recovery, and rollback diff --git a/docs/install/server-workspace-registry.md b/docs/install/server-workspace-registry.md index 22281a59..bb3a7c5c 100644 --- a/docs/install/server-workspace-registry.md +++ b/docs/install/server-workspace-registry.md @@ -48,11 +48,13 @@ THT_WORKSPACE_GIT_CREDENTIALS_FILE=/srv/thothii/secrets/git-credentials THT_WORKSPACE_GIT_CA_FILE=/srv/thothii/secrets/git-ca.pem THT_WORKSPACE_GIT_SSH_KEY_FILE=/srv/thothii/secrets/git-ssh-key THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/srv/thothii/secrets/git-known-hosts +PI_AUTH_FILE=/srv/thothii/secrets/pi-auth.json +THT_SECRETS_FILE=/srv/thothii/secrets/thothii.secrets ``` Use the credential file for HTTPS, or key and known-hosts for SSH. The base server Compose file -mounts neither transport; add exactly one [HTTPS override](examples/git-https.workspace-registry.yaml) -or [SSH override](examples/git-ssh.workspace-registry.yaml). Strict host-key checking stays enabled +mounts neither transport; add exactly one `deploy/compose.git-https.yaml` +or `deploy/compose.git-ssh.yaml` override. Strict host-key checking stays enabled and Git stderr is not exposed by the API. Rotate by atomically replacing the secret file, restarting `core`, and performing pull/status; never put the material in an environment variable or rendered Compose output. @@ -75,9 +77,9 @@ The runtime registry layout is persistent and must be backed up together: /data/workspace-registry/locks/ ``` -Variable names derive from the immutable ID: `psd-clinical` becomes `PSD_CLINICAL`, producing -`THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE`. A declared vector writer uses the distinct -`THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE`; a reader file is never a writer substitute. +Variable names derive from the immutable ID: `north-star-research` becomes `NORTH_STAR_RESEARCH`, producing +`THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE`. A declared vector writer uses the distinct +`THT_WS_NORTH_STAR_RESEARCH_VECTOR_WRITER_API_KEY_FILE`; a reader file is never a writer substitute. Copy [the bindings env example](examples/workspace-bindings.env.example) to the protected operator directory. Every path-valued `*_FILE` entry needs an absolute host-only `*_SOURCE` path. Generate the untracked connector override from those files during bootstrap; do not copy or maintain a @@ -90,41 +92,41 @@ dimensions, and distance as Git-shared identity. ```dotenv # Direct PostgreSQL/pgvector with verified native TLS if a CA path is supplied. -THT_WS_PSD_CLINICAL_DWH_TRANSPORT=postgres_direct -THT_WS_PSD_CLINICAL_DWH_HOST=dwh.internal.example -THT_WS_PSD_CLINICAL_DWH_PORT=5432 -THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader -THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader -THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=pgvector_direct -THT_WS_PSD_CLINICAL_VECTOR_HOST=vector.internal.example -THT_WS_PSD_CLINICAL_VECTOR_PORT=5432 -THT_WS_PSD_CLINICAL_VECTOR_USER=thoth_vector_reader -THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE=/run/secrets/psd-vector-reader +THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct +THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.internal.example +THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432 +THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader +THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password +THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=pgvector_direct +THT_WS_NORTH_STAR_RESEARCH_VECTOR_HOST=vector.internal.example +THT_WS_NORTH_STAR_RESEARCH_VECTOR_PORT=5432 +THT_WS_NORTH_STAR_RESEARCH_VECTOR_USER=thoth_vector_reader +THT_WS_NORTH_STAR_RESEARCH_VECTOR_PASSWORD_FILE=/run/secrets/north-star-research-vector-password ``` ```dotenv # REST needs API-key file paths only when the descriptor declares authenticated diagnostics. -THT_WS_PSD_CLINICAL_DWH_TRANSPORT=rest_api -THT_WS_PSD_CLINICAL_DWH_BASE_URL=https://dwh.internal.example -THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE=/run/secrets/psd-dwh-api-key -THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=rest_api -THT_WS_PSD_CLINICAL_VECTOR_BASE_URL=https://vectors.internal.example -THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE=/run/secrets/psd-vector-api-key -THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL=https://embeddings.internal.example +THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api +THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.internal.example +THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key +THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api +THT_WS_NORTH_STAR_RESEARCH_VECTOR_BASE_URL=https://vectors.internal.example +THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key +THT_WS_NORTH_STAR_RESEARCH_EMBEDDING_BASE_URL=https://embeddings.internal.example ``` ```dotenv # SSH tunnel diagnostic only; runtime sessions are fail-closed in this release. -THT_WS_PSD_CLINICAL_DWH_TRANSPORT=ssh_tunnel -THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader -THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader -THT_WS_PSD_CLINICAL_DWH_SSH_HOST=bastion.internal.example -THT_WS_PSD_CLINICAL_DWH_SSH_PORT=22 -THT_WS_PSD_CLINICAL_DWH_SSH_USER=thoth_tunnel -THT_WS_PSD_CLINICAL_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/psd-dwh-tunnel-key -THT_WS_PSD_CLINICAL_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/psd-dwh-known-hosts -THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_HOST=dwh.internal.example -THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_PORT=5432 +THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel +THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader +THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.internal.example +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22 +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example +THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432 ``` Repeat SSH variables for `VECTOR` when selected. REST diagnostics refuse private per-request CAs @@ -138,15 +140,17 @@ The Git registry itself may still use SSH normally. ## Same-origin reverse proxy, bootstrap, and health -Copy [the server Compose example](examples/server-compose.workspace-registry.yaml) plus exactly one -selected Git override to the protected operator directory. Set `THT_SOURCE_ROOT` to the absolute -ThothII checkout; a copied file cannot use a relative build context. Copy -`deploy/workspaces/server-sessions.yaml.example` into that operator directory, review it, then set -the absolute `THT_SERVER_WORKSPACE_CONFIG` path. Copy the bindings env example, then set absolute -`THT_WORKSPACE_BINDINGS_ENV_FILE` and connector `*_SOURCE` paths. The same `.env` must set +Use the repository's canonical `compose.yaml` plus `deploy/compose.server.yaml`; they always +start the mandatory `frontend` and `core` services. Do not copy or maintain a standalone +application Compose file. Review `deploy/workspaces/server-sessions.yaml.example`, materialize it +as a protected host file, and set its absolute `THT_SERVER_WORKSPACE_CONFIG` path. Copy the +bindings env example into the operator directory, then set absolute `PI_AUTH_FILE`, +`THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths. +The same operator env must set `THT_SESSION_DB_HOST`, `THT_SESSION_DB_NAME`, `THT_SESSION_RUNTIME_USER`, -`THT_SESSION_RUNTIME_PASSWORD_SOURCE`, and `THT_SESSION_CA_SOURCE`; the base Compose file wires -`postgres`, `verify-full`, and the two Docker secret mount paths. This is the public server profile, +`THT_SESSION_RUNTIME_PASSWORD_SOURCE`, and `THT_SESSION_CA_SOURCE`; +`deploy/compose.session-server.yaml.example` wires `postgres`, `verify-full`, and separate +runtime/CA Docker secret mount paths. This is the public server profile, not a filesystem-session fallback. A Compose `.env` file is not a shell environment, so do not import it into the maintenance shell. Explicitly export the non-secret source and bindings paths before running the commands below. @@ -161,14 +165,24 @@ From a trusted maintenance shell: ```sh export THT_SOURCE_ROOT=/absolute/path/to/ThothII -export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env" -"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env .env --output connector-secrets.local.yaml -"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \ - -f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml up --build -d -"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \ - -f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml exec -T core curl --fail --silent http://127.0.0.1:8787/health -"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \ - -f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml exec -T core curl --fail --silent http://127.0.0.1:8787/workspace-registry/status +export THT_OPERATOR_ENV=/srv/thothii/operator/server.env +export THT_WORKSPACE_BINDINGS_ENV_FILE=/srv/thothii/operator/workspace-bindings.env +export THT_CONNECTOR_OVERRIDE=/srv/thothii/operator/connector-secrets.local.yaml +"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE" +"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \ + -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \ + -f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" \ + -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" up --build -d +"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \ + -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \ + -f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" \ + -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" \ + exec -T core curl --fail --silent http://127.0.0.1:8787/health +"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \ + -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \ + -f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" \ + -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" \ + exec -T core curl --fail --silent http://127.0.0.1:8787/workspace-registry/status ``` `/health` is liveness. Registry status verifies branch/head/degraded state and the active validated @@ -187,7 +201,7 @@ filesystem-consistent backup of `/srv/thothii/workspace-registry` plus `/srv/tho `/srv/thothii/secrets`. Render Compose, deploy the compatible image, verify health/status, then resume proxy traffic. -For PSD migration, use a temporary review clone and the legacy transformer with absolute paths. +For legacy descriptor migration, use a temporary review clone and the legacy transformer with absolute paths. Its schema-v1 output is `migration_required`; explicitly supply vector database/schema, collection identity, diagnostics, and the reviewed v2 contract before commit. Never import `${ENV}` values or copy secret files. diff --git a/docs/installazione-docker-4-contesti.md b/docs/installazione-docker-4-contesti.md index 42d9207c..0c82be2d 100644 --- a/docs/installazione-docker-4-contesti.md +++ b/docs/installazione-docker-4-contesti.md @@ -15,6 +15,8 @@ Servono Docker Engine/Compose v2 su Linux oppure Docker Desktop su macOS/Windows git clone ThothII cd ThothII cp deploy/env/local.env.example deploy/env/local.env +cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets +chmod 600 deploy/secrets/thothii.secrets ``` Modificare **solo** questi file interni al clone: @@ -25,7 +27,8 @@ Modificare **solo** questi file interni al clone: | file protetti locali | credenziali e certificati, indicati dai binding del workspace | | `deploy/workspaces/.yaml` | adapter, endpoint non riservati, `roots` ed Evidence | -Compilare `deploy/env/local.env`, incluso `PI_AUTH_FILE`, con gli endpoint esterni. L'avvio +Compilare `deploy/env/local.env`, inclusi i path assoluti `PI_AUTH_FILE` e +`THT_SECRETS_FILE`, con gli endpoint esterni. L'avvio normale usa esplicitamente il file base e l'overlay locale: ```sh @@ -52,7 +55,7 @@ THT_VECTOR_READER_PASSWORD=... THT_VECTOR_WRITER_PASSWORD=... ``` -Inserire solo le chiavi necessarie al profilo scelto. Il bundle viene montato in sola lettura nel container come `/run/secrets/thothii.secrets`; il parser rifiuta duplicati, chiavi sconosciute, valori vuoti, symlink e permessi host troppo aperti. Non inserire secret in `.env`, nei workspace, negli URL o nell'output di `docker compose config`. +Inserire solo le chiavi necessarie al profilo scelto. Il bundle viene montato in sola lettura nel container come `/run/secrets/thothii.secrets`; il parser rifiuta duplicati, chiavi sconosciute, valori vuoti, symlink e permessi host troppo aperti. Non inserire secret in `.env`, nei workspace, negli URL o nell'output Compose renderizzato. Una catena CA PEM **non può essere inserita nel bundle**: contiene whitespace e viene rifiutata dal parser. Se un endpoint usa una CA privata, conservarla nel secret manager/host e aggiungere un override Compose revisionato che monti il file in `/run/secrets/ca-chain.pem` e imposti `THT_SSL_CA` (o il parametro dell'adapter). Il clone base non crea quel mount: questa è una limitazione intenzionale da considerare in fase di deployment. @@ -62,7 +65,8 @@ DWH/vector/embedding remoti restano endpoint del file locale o server. Per il so sviluppo pgvector, aggiungere `-f deploy/compose.local-vector.yaml --profile local-vector` al comando base. Per il preprocessing aggiungere anche `-f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml --profile preprocess`, -poi usare `docker compose run --rm preprocess-evidence` oppure `preprocess-dwh` con gli stessi argomenti. +poi ripetere l'intero comando base con l'azione `run --rm preprocess-evidence` oppure +`run --rm preprocess-dwh`. ## Workspace, adapter e Evidence @@ -124,8 +128,10 @@ THOTH_PUBLIC_EXPOSURE=false Riempire nel bundle le chiavi DWH/vector/model necessarie e avviare: ```sh -docker compose up --build -d -docker compose exec core /opt/venv/bin/tht doctor --json +docker compose --env-file deploy/env/local.env \ + -f compose.yaml -f deploy/compose.local.yaml up --build -d +docker compose --env-file deploy/env/local.env \ + -f compose.yaml -f deploy/compose.local.yaml exec core /opt/venv/bin/tht doctor --json ``` Se si abilita l'overlay production, il proxy autenticato TLS deve essere l'unico listener pubblico @@ -159,7 +165,10 @@ THT_VECTOR_READER_PASSWORD= THT_VECTOR_WRITER_PASSWORD= ``` -Poi eseguire il comando standard `docker compose up --build -d`. Il primo avvio esegue reconciliation dei ruoli e migrazione pgvector. Per preprocessing, impostare il preset indicato sopra e usare `docker compose run --rm preprocess-evidence`/`preprocess-dwh`. +Poi eseguire il comando standard base+locale mostrato sopra. Il primo avvio esegue +reconciliation dei ruoli e migrazione pgvector. Per preprocessing, impostare il preset indicato +sopra e usare l'azione `run --rm preprocess-evidence` o `run --rm preprocess-dwh` con tutti +gli stessi file e profili. ## 3. PC Windows locale @@ -175,8 +184,8 @@ THT_DOCS_ROOT=/data/source/evidence Creare `deploy/secrets/thothii.secrets` con un editor locale protetto (ACL leggibile solo dall'utente Docker) e le stesse quattro chiavi pgvector del profilo Mac. Non usare `ConvertFrom-SecureString`: il bundle deve contenere il valore in chiaro per il servizio, con accesso limitato al file. Da PowerShell, dalla radice del clone, eseguire: ```powershell -docker compose up --build -d -docker compose ps +docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up --build -d +docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml ps ``` Se un bind mount viene rifiutato, aggiungere la cartella del repository a Docker Desktop → Settings → Resources → File Sharing. Per Ollama eseguito in WSL2 usare l'indirizzo raggiungibile dalla rete Docker invece di assumere `localhost`. @@ -187,13 +196,25 @@ Usare il profilo server e consentire dal firewall solo le destinazioni necessari ```dotenv # Avvio: docker compose --env-file deploy/env/server.env \ -# -f compose.yaml -f deploy/compose.server.yaml up --build -d +# -f compose.yaml -f deploy/compose.server.yaml \ +# -f deploy/compose.session-server.yaml.example up --build -d THT_DB_NAME=warehouse THT_DWH_REST_URL=https://dwh.example.test THT_VEC_REST_URL=https://vectors.example.test THT_OLLAMA_URL=https://embeddings.example.test ``` +Avviare e verificare con il profilo server completo: + +```sh +docker compose --env-file deploy/env/server.env \ + -f compose.yaml -f deploy/compose.server.yaml \ + -f deploy/compose.session-server.yaml.example up --build -d +docker compose --env-file deploy/env/server.env \ + -f compose.yaml -f deploy/compose.server.yaml \ + -f deploy/compose.session-server.yaml.example exec core /opt/venv/bin/tht doctor --json +``` + Il DWH e il vector DB possono essere REST/HTTP oppure adapter diretti (`postgres_direct`, `pgvector_direct`) se il server ha connettività TCP. Le Evidence possono essere: - filesystem NFS/SMB montato sul server e presentato come root read-only; @@ -208,8 +229,8 @@ Le variabili `THT_*_SECRET_FILE` e i file `dwh-api-key`, `vector-reader-api-key` 1. creare `deploy/secrets/thothii.secrets` mode `0600`; 2. copiare ogni valore nel nome chiave corrispondente (`THT_DWH_API_KEY`, `THT_VEC_API_KEY`, `THT_VEC_WRITE_API_KEY`, `THT_MODEL_API_KEY` o `THT_VECTOR_*_PASSWORD`), senza virgolette né newline; -3. rimuovere dal `.env` le variabili `_SECRET_FILE` e impostare `THT_SECRETS_FILE` al percorso del bundle (il default relativo è già corretto); -4. eseguire `docker compose config --quiet` e poi `docker compose up --build -d`; +3. rimuovere dal `.env` le variabili `_SECRET_FILE` e impostare `THT_SECRETS_FILE` al percorso assoluto del bundle; +4. renderizzare e avviare con il comando base+locale completo e il suo `--env-file`; 5. solo dopo la verifica, cancellare i vecchi file separati. Una CA PEM resta un'eccezione esterna come descritto sopra. Provider Pi con credenziali composte (Bedrock, Azure OpenAI Responses, Cloudflare Workers AI/Gateway) restano rifiutati finché non viene implementato un adapter dedicato. @@ -217,9 +238,12 @@ Una CA PEM resta un'eccezione esterna come descritto sopra. Provider Pi con cred ## Controlli post-installazione ```sh -docker compose config --quiet -docker compose ps -docker compose exec core /opt/venv/bin/tht doctor --json +docker compose --env-file deploy/env/local.env \ + -f compose.yaml -f deploy/compose.local.yaml config --quiet +docker compose --env-file deploy/env/local.env \ + -f compose.yaml -f deploy/compose.local.yaml ps +docker compose --env-file deploy/env/local.env \ + -f compose.yaml -f deploy/compose.local.yaml exec core /opt/venv/bin/tht doctor --json ./scripts/docker-smoke.sh ``` diff --git a/harness/workspaces/local.yaml b/harness/workspaces/local.yaml index f763db39..9a0753a6 100644 --- a/harness/workspaces/local.yaml +++ b/harness/workspaces/local.yaml @@ -1,5 +1,5 @@ # Workspace ThothII — Profilo A (server co-locato). DWH + vector BOTH direct, no REST. -# Segreti SOLO in env (compose env_file: deploy/thothii.env). Path assoluti interni al container (/data). +# Secret contents live only in protected mounted files; paths below are container-absolute. language: it database: diff --git a/scripts/build-local.ps1 b/scripts/build-local.ps1 index c1feb111..78d166aa 100644 --- a/scripts/build-local.ps1 +++ b/scripts/build-local.ps1 @@ -3,11 +3,11 @@ $ErrorActionPreference = "Continue" $repositoryRoot = Split-Path -Parent $PSScriptRoot Set-Location $repositoryRoot -& docker compose -f compose.yaml -f deploy/compose.local.yaml build --pull +& docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml build --pull $exitCode = $LASTEXITCODE if ($exitCode -eq 0) { - Write-Output "Next: docker compose -f compose.yaml -f deploy/compose.local.yaml up -d" + Write-Output "Next: docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up -d" } exit $exitCode diff --git a/scripts/build-local.sh b/scripts/build-local.sh index a026f2f7..fdd985e3 100755 --- a/scripts/build-local.sh +++ b/scripts/build-local.sh @@ -3,11 +3,12 @@ set -u cd "$(dirname "$0")/.." -docker compose -f compose.yaml -f deploy/compose.local.yaml build --pull +docker compose --env-file deploy/env/local.env \ + -f compose.yaml -f deploy/compose.local.yaml build --pull status=$? if [[ "$status" -eq 0 ]]; then - printf '%s\n' 'Next: docker compose -f compose.yaml -f deploy/compose.local.yaml up -d' + printf '%s\n' 'Next: docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up -d' fi exit "$status" diff --git a/scripts/docker-smoke.sh b/scripts/docker-smoke.sh index 53b3064c..e4596798 100755 --- a/scripts/docker-smoke.sh +++ b/scripts/docker-smoke.sh @@ -1,21 +1,21 @@ #!/usr/bin/env bash # Smoke test del deploy standalone ThothII (core + frontend). # Usa docker-compose.dev.yml (rete propria, porte host). -# Prereq: deploy/thothii.env popolato, endpoint esterni configurati e profilo Pi locale. +# Prereq: deploy/env/local.env popolato, endpoint esterni e file Pi/segreti configurati. set -euo pipefail cd "$(dirname "$0")/.." -DC="docker compose -f docker-compose.dev.yml" +DC=(docker compose --env-file deploy/env/local.env -f docker-compose.dev.yml) WS="/app/harness/workspaces/local.yaml" echo "== ThothII standalone smoke ==" -$DC config --quiet +"${DC[@]}" config --quiet echo "== Build ==" -$DC build +"${DC[@]}" build echo "== Up (wait health) ==" -$DC up -d --wait +"${DC[@]}" up -d --wait echo "== Core health ==" curl -fsS http://localhost:8787/health && echo @@ -24,12 +24,12 @@ echo "== Frontend serve ==" curl -fsSI http://localhost:8090/ | head -1 echo "== Wiring check (config + DWH ping; -c è per-command) ==" -$DC exec -T core tht config check -c "$WS" || \ +"${DC[@]}" exec -T core tht config check -c "$WS" || \ echo "(config check non verde: verificare .env/ruoli DB)" -$DC exec -T core tht db ping -c "$WS" || \ +"${DC[@]}" exec -T core tht db ping -c "$WS" || \ echo "(db ping non verde: verificare ruolo thoth_dwh_reader + rete)" echo "== Down ==" -$DC down +"${DC[@]}" down echo "OK: smoke standalone passato." diff --git a/scripts/generate-connector-secrets-override.sh b/scripts/generate-connector-secrets-override.sh index 5ccdf9cc..2600825b 100755 --- a/scripts/generate-connector-secrets-override.sh +++ b/scripts/generate-connector-secrets-override.sh @@ -101,7 +101,13 @@ done < <( { printf '%s\n' '# Generated by scripts/generate-connector-secrets-override.sh; keep this file untracked.' - printf '%s\n' 'services:' ' core:' ' secrets:' + printf '%s\n' \ + 'services:' \ + ' core:' \ + ' env_file:' \ + ' - path: ${THT_WORKSPACE_BINDINGS_ENV_FILE:?set THT_WORKSPACE_BINDINGS_ENV_FILE}' \ + ' required: true' \ + ' secrets:' for ((index = 0; index < ${#names[@]}; index += 1)); do printf ' - source: connector_secret_%d\n' "$((index + 1))" printf ' target: %s\n' "${targets[index]}" diff --git a/scripts/local-vector-smoke.sh b/scripts/local-vector-smoke.sh index 071757ac..eb45a916 100755 --- a/scripts/local-vector-smoke.sh +++ b/scripts/local-vector-smoke.sh @@ -39,6 +39,13 @@ write_bundle() { } write_bundle export THT_SECRETS_FILE="$bundle" +printf '%s\n' '{}' >"$secret_dir/pi-auth.json" +chmod 0600 "$secret_dir/pi-auth.json" +operator_env="$secret_dir/operator.env" +printf '%s\n' \ + 'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \ + "PI_AUTH_FILE=$secret_dir/pi-auth.json" \ + "THT_SECRETS_FILE=$bundle" >"$operator_env" # The rotation helper has an old/new file interface; these are test-only # scratch files and are never mounted into a Compose service. printf '%s' "$bootstrap_password" >"$secret_dir/bootstrap" @@ -47,7 +54,7 @@ export THT_VECTOR_BOOTSTRAP_USER=thoth_bootstrap_smoke export THOTH_SMOKE_OWNER="$smoke_owner" compose() { - docker compose -f compose.yaml -f deploy/compose.local-vector.yaml \ + docker compose --env-file "$operator_env" -f compose.yaml -f deploy/compose.local-vector.yaml \ --project-name "$smoke_project" --profile local-vector "$@" } diff --git a/scripts/preprocess-smoke.sh b/scripts/preprocess-smoke.sh index c9a6486d..3b112275 100755 --- a/scripts/preprocess-smoke.sh +++ b/scripts/preprocess-smoke.sh @@ -58,6 +58,13 @@ bundle="$tmp/thothii.secrets" chmod 0600 "$bundle" export THT_SECRETS_FILE="$bundle" export THT_OLLAMA_URL=http://mock-embeddings:8081 +printf '%s\n' '{}' >"$tmp/pi-auth.json" +chmod 0600 "$tmp/pi-auth.json" +printf '%s\n' \ + 'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \ + "PI_AUTH_FILE=$tmp/pi-auth.json" \ + "THT_SECRETS_FILE=$bundle" \ + 'THT_OLLAMA_URL=http://mock-embeddings:8081' >"$tmp/operator.env" cat >"$tmp/smoke.yaml" <&2 + exit 1 + fi +done + +printf '%s\n' '{}' >"$tmp/pi-auth.json" +printf '%s\n' 'THT_MODEL_API_KEY=fixture-model-api-key' >"$tmp/thothii.secrets" +chmod 0600 "$tmp/pi-auth.json" "$tmp/thothii.secrets" +mkdir -p "$tmp/data" "$tmp/pi-state" "$tmp/workspace-registry" +printf '%s\n' 'fixture-session-password' >"$tmp/session-runtime-password" +printf '%s\n' 'fixture-session-migrator-password' >"$tmp/session-migrator-password" +printf '%s\n' 'fixture-session-ca' >"$tmp/session-ca.pem" +cp "$root/deploy/workspaces/server-sessions.yaml.example" "$tmp/server-sessions.yaml" +chmod 0600 "$tmp/session-runtime-password" "$tmp/session-migrator-password" "$tmp/session-ca.pem" + +for profile in local server; do + env_file="$tmp/$profile.env" + { + printf '%s\n' \ + 'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \ + "PI_AUTH_FILE=$tmp/pi-auth.json" \ + "THT_SECRETS_FILE=$tmp/thothii.secrets" + if [[ "$profile" == server ]]; then + printf '%s\n' \ + "THT_DATA_ROOT=$tmp/data" \ + "THT_PI_STATE_ROOT=$tmp/pi-state" \ + "THT_WORKSPACE_REGISTRY_ROOT=$tmp/workspace-registry" \ + "THT_SERVER_WORKSPACE_CONFIG=$tmp/server-sessions.yaml" \ + 'THT_SESSION_DB_HOST=sessions.example.invalid' \ + 'THT_SESSION_DB_NAME=thoth_sessions' \ + 'THT_SESSION_RUNTIME_USER=thoth_sessions_app' \ + 'THT_SESSION_MIGRATOR_USER=thoth_sessions_migrate' \ + "THT_SESSION_RUNTIME_PASSWORD_SOURCE=$tmp/session-runtime-password" \ + "THT_SESSION_MIGRATOR_PASSWORD_SOURCE=$tmp/session-migrator-password" \ + "THT_SESSION_CA_SOURCE=$tmp/session-ca.pem" + fi + } >"$env_file" + + compose_files=(-f "$root/compose.yaml" -f "$root/deploy/compose.$profile.yaml") + if [[ "$profile" == server ]]; then + compose_files+=(-f "$root/deploy/compose.session-server.yaml.example") + fi + docker compose --env-file "$env_file" "${compose_files[@]}" \ + config --format json >"$tmp/$profile.json" + node - "$tmp/$profile.json" "$profile" <<'NODE' +const fs = require("fs"); +const [path, profile] = process.argv.slice(2); +const config = JSON.parse(fs.readFileSync(path, "utf8")); +if (Object.keys(config.services).sort().join(",") !== "core,frontend") { + throw new Error(profile + ": install stack must be exactly core,frontend"); +} +if (!config.services.core.secrets?.some((secret) => secret.target === "thothii.secrets")) { + throw new Error(profile + ": install stack lacks the runtime secret bundle"); +} +if (!config.services.core.volumes?.some( + (mount) => mount.target === "/home/thoth/.pi/agent/auth.json" && mount.read_only, +)) { + throw new Error(profile + ": install stack lacks the read-only Pi auth file"); +} +if ((config.services.frontend.secrets || []).length !== 0) { + throw new Error(profile + ": frontend received runtime secrets"); +} +if (profile === "server" && config.services.core.environment?.THT_SESSION_STORAGE !== "postgres") { + throw new Error("server: public startup must include the PostgreSQL session override"); +} +if (JSON.stringify(config).includes("fixture-model-api-key")) { + throw new Error(profile + ": rendered Compose leaked a secret value"); +} +NODE +done + +for profile in local server; do + manual="$root/docs/install/$profile-workspace-registry.md" + grep -Fq -- '--env-file "$THT_OPERATOR_ENV"' "$manual" \ + && grep -Fq -- "-f \"\$THT_SOURCE_ROOT/compose.yaml\" -f \"\$THT_SOURCE_ROOT/deploy/compose.$profile.yaml\"" "$manual" || { + echo "$profile manual lacks the canonical base+profile command" >&2 + exit 1 + } + if rg -q 'local-compose\.workspace-registry|server-compose\.workspace-registry' "$manual"; then + echo "$profile manual still references a superseded standalone Compose example" >&2 + exit 1 + fi +done + +echo "canonical install Compose contract passed." diff --git a/scripts/test-compose-provider-readiness.sh b/scripts/test-compose-provider-readiness.sh new file mode 100755 index 00000000..3b25268c --- /dev/null +++ b/scripts/test-compose-provider-readiness.sh @@ -0,0 +1,74 @@ +#!/usr/bin/env bash +# Fresh Compose flow: mounted Pi policy/auth must produce a selectable, credential-ready provider. +set -euo pipefail + +root="$(cd "$(dirname "$0")/.." && pwd -P)" +tmp="$(mktemp -d "${TMPDIR%/}/thoth-provider-readiness.XXXXXX")" +project="thothii-provider-readiness-$$" +compose=( + docker compose --project-name "$project" --env-file "$tmp/local.env" + -f "$root/compose.yaml" -f "$root/deploy/compose.local.yaml" +) +cleanup() { + "${compose[@]}" down --volumes --remove-orphans >/dev/null 2>&1 || true + rm -rf "$tmp" +} +trap cleanup EXIT HUP INT TERM + +printf '%s\n' '{"zai":{"type":"api_key","key":"fixture-native-auth-key"}}' >"$tmp/pi-auth.json" +printf '%s\n' 'THT_MODEL_API_KEY=fixture-model-api-key' >"$tmp/thothii.secrets" +chmod 0600 "$tmp/pi-auth.json" "$tmp/thothii.secrets" +printf '%s\n' \ + 'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \ + "PI_AUTH_FILE=$tmp/pi-auth.json" \ + "THT_SECRETS_FILE=$tmp/thothii.secrets" \ + 'THOTH_CORE_HTTP_PORT=0' \ + 'THOTH_HTTP_PORT=0' \ + >"$tmp/local.env" + +"${compose[@]}" up --detach --wait --wait-timeout 90 --build core +core_id="$("${compose[@]}" ps -q core)" +core_address="$("${compose[@]}" port core 8787 | head -n 1)" + +"${compose[@]}" exec -T core sh -ceu ' + test -r /home/thoth/.pi/agent/auth.json + test -r /home/thoth/.pi/agent/models.json + test -r /home/thoth/.pi/agent/settings.json + test -r /run/secrets/thothii.secrets +' + +curl --fail --silent --show-error "http://$core_address/models" >"$tmp/models.json" +node - "$tmp/models.json" <<'NODE' +const fs = require("fs"); +const body = JSON.parse(fs.readFileSync(process.argv[2], "utf8")); +if (!body.models?.some((model) => model.provider === "zai" && model.id === "glm-5.2")) { + throw new Error("fresh Compose did not expose the mounted Pi-enabled model"); +} +NODE + +curl --fail --silent --show-error -X PUT \ + -H 'content-type: application/json' \ + --data '{"provider":"zai","model":"glm-5.2","reasoning":"low"}' \ + "http://$core_address/pi-management/config" >"$tmp/configured.json" +curl --fail --silent --show-error \ + "http://$core_address/pi-management/status" >"$tmp/status.json" +node - "$tmp/status.json" <<'NODE' +const fs = require("fs"); +const body = JSON.parse(fs.readFileSync(process.argv[2], "utf8")); +if (!body.ready || body.credentials !== "present") { + throw new Error("mounted Pi provider is not credential-ready"); +} +if (body.config?.provider !== "zai" || body.config?.model !== "glm-5.2") { + throw new Error("Pi provider configuration was not persisted"); +} +NODE + +inspect="$(docker inspect "$core_id")" +for secret in fixture-native-auth-key fixture-model-api-key; do + if grep -Fq "$secret" <<<"$inspect"; then + echo "container inspection leaked $secret" >&2 + exit 1 + fi +done + +echo "Compose provider-readiness contract passed." diff --git a/scripts/test-compose-secret-policy.sh b/scripts/test-compose-secret-policy.sh index a30820c4..c4fbc2bf 100755 --- a/scripts/test-compose-secret-policy.sh +++ b/scripts/test-compose-secret-policy.sh @@ -48,7 +48,13 @@ for (const mount of (core.volumes || []).filter((item) => item.target?.startsWit const secretTargets = (core.secrets || []).map((secret) => secret.target).sort(); const expectedSecrets = expectedSecretTargets ? expectedSecretTargets.split(",").filter(Boolean).sort() : []; if (secretTargets.join(",") !== expectedSecrets.join(",")) { - throw new Error(`${name}: connector targets do not match generated THT_WS_*_FILE bindings`); + throw new Error(`${name}: Docker secret targets do not match the deployment contract`); +} +if (core.environment?.THT_SECRETS_FILE !== "/run/secrets/thothii.secrets") { + throw new Error(`${name}: core does not use the canonical /run/secrets bundle path`); +} +if ((config.services.frontend?.secrets || []).length !== 0) { + throw new Error(`${name}: frontend must not receive runtime secrets`); } if (name === "ssh") { @@ -62,7 +68,7 @@ if (name === "https" && core.environment?.GIT_CONFIG_VALUE_1 !== "/run/secrets/w } const rendered = JSON.stringify(config); -for (const secret of ["fixture-ssh-private-key", "fixture-ssh-known-hosts", "fixture-https-credentials", "fixture-https-ca", "fixture-dwh-password", "fixture-vector-api-key"]) { +for (const secret of ["fixture-model-api-key", "fixture-ssh-private-key", "fixture-ssh-known-hosts", "fixture-https-credentials", "fixture-https-ca", "fixture-dwh-password", "fixture-vector-api-key"]) { if (rendered.includes(secret)) throw new Error(`${name}: rendered Compose leaked fixture secret value`); } NODE @@ -97,6 +103,7 @@ assert_unsafe_source_rejected() { } write_secret "$fixture_root/pi-auth.json" 'fixture-pi-auth' +write_secret "$fixture_root/thothii.secrets" 'THT_MODEL_API_KEY=fixture-model-api-key' write_secret "$fixture_root/ssh-private-key" 'fixture-ssh-private-key' write_secret "$fixture_root/ssh-known-hosts" 'fixture-ssh-known-hosts' write_secret "$fixture_root/https-credentials" 'fixture-https-credentials' @@ -107,6 +114,8 @@ write_secret "$fixture_root/vector-api-key" 'fixture-vector-api-key' printf '%s\n' \ 'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \ "PI_AUTH_FILE=$fixture_root/pi-auth.json" \ + "THT_SECRETS_FILE=$fixture_root/thothii.secrets" \ + "THT_WORKSPACE_BINDINGS_ENV_FILE=$fixture_root/workspace-bindings.env" \ "THT_WORKSPACE_GIT_SSH_KEY_FILE=$fixture_root/ssh-private-key" \ "THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$fixture_root/ssh-known-hosts" \ "THT_WORKSPACE_GIT_CREDENTIALS_FILE=$fixture_root/https-credentials" \ @@ -127,13 +136,13 @@ connector_override="$fixture_root/compose.connector-secrets.local.yaml" --output "$connector_override" render base -assert_render_contract base '' '' +assert_render_contract base '' 'thothii.secrets' render ssh -f "$root/deploy/compose.git-ssh.yaml" -assert_render_contract ssh '/run/secrets/workspace-registry-git-known-hosts,/run/secrets/workspace-registry-git-ssh-key' '' +assert_render_contract ssh '/run/secrets/workspace-registry-git-known-hosts,/run/secrets/workspace-registry-git-ssh-key' 'thothii.secrets' render https -f "$root/deploy/compose.git-https.yaml" -assert_render_contract https '/run/secrets/workspace-registry-git-ca,/run/secrets/workspace-registry-git-credentials' '' +assert_render_contract https '/run/secrets/workspace-registry-git-ca,/run/secrets/workspace-registry-git-credentials' 'thothii.secrets' render connector -f "$connector_override" -assert_render_contract connector '' 'north-star-research-dwh-password,north-star-research-vector-api-key' +assert_render_contract connector '' 'north-star-research-dwh-password,north-star-research-vector-api-key,thothii.secrets' assert_missing_source_rejected THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE assert_missing_source_rejected THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE diff --git a/scripts/test-container-deployment.sh b/scripts/test-container-deployment.sh index 992f7e66..2bacefad 100755 --- a/scripts/test-container-deployment.sh +++ b/scripts/test-container-deployment.sh @@ -9,10 +9,13 @@ expected_pi_version=$(sed -n 's/^ARG PI_VERSION=//p' docker/core.Dockerfile) trap 'docker compose --project-name "$project" -f compose.yaml -f deploy/compose.local.yaml down --volumes --remove-orphans >/dev/null 2>&1 || true; rm -rf "$tmp"' EXIT HUP INT TERM test -n "$expected_pi_version" -printf '{}\n' >"$tmp/pi-auth.json" +printf '%s\n' '{"zai":{"type":"api_key","key":"fixture-native-auth-key"}}' >"$tmp/pi-auth.json" chmod 0600 "$tmp/pi-auth.json" +printf '%s\n' 'THT_MODEL_API_KEY=fixture-model-api-key' >"$tmp/thothii.secrets" +chmod 0600 "$tmp/thothii.secrets" export PI_AUTH_FILE="$tmp/pi-auth.json" +export THT_SECRETS_FILE="$tmp/thothii.secrets" export THT_WORKSPACE_GIT_REMOTE="https://git.example.invalid/thothii/workspaces.git" # Let Docker assign loopback ports so this isolated contract test never collides with an operator stack. export THOTH_CORE_HTTP_PORT=0 @@ -62,6 +65,10 @@ docker compose --project-name "$project" -f compose.yaml -f deploy/compose.local test "$(pi --version)" = "$PI_VERSION" command -v pi >/dev/null test ! -e /var/run/docker.sock + test -r /home/thoth/.pi/agent/auth.json + test -r /home/thoth/.pi/agent/models.json + test -r /home/thoth/.pi/agent/settings.json + test -r /run/secrets/thothii.secrets touch /data/.task5-writable rm /data/.task5-writable if find /app /home /data -xdev \( -iname "*chirone*" -o -iname "*omics*portal*" \) -print -quit | grep -q .; then diff --git a/scripts/test-deployment-command-contract.sh b/scripts/test-deployment-command-contract.sh new file mode 100755 index 00000000..f45305b4 --- /dev/null +++ b/scripts/test-deployment-command-contract.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# Prevent active operator-facing startup examples from bypassing required env/profile inputs. +set -euo pipefail + +root="$(cd "$(dirname "$0")/.." && pwd -P)" +cd "$root" + +targets=( + README.md + .env.example + docker-compose.dev.yml + deploy/env.example + deploy/secrets/README.md + docs/install + docs/index.md + docs/installazione-docker-4-contesti.md + scripts/build-local.sh + scripts/build-local.ps1 + scripts/docker-smoke.sh + scripts/local-vector-smoke.sh + scripts/preprocess-smoke.sh + scripts/vector-rotate-bootstrap-password.sh +) + +existing=() +for target in "${targets[@]}"; do + [[ ! -e "$target" ]] || existing+=("$target") +done + +set +e +matches="$(rg -n \ + 'docker compose (up|build|run|config|ps|exec|-f)|DC="docker compose -f|compose="docker compose -f' \ + "${existing[@]}" 2>&1)" +rg_status=$? +set -e +case "$rg_status" in + 0) + echo "active deployment command omits --env-file before its action/overrides:" >&2 + printf '%s\n' "$matches" >&2 + exit 1 + ;; + 1) ;; + *) + printf '%s\n' "$matches" >&2 + exit "$rg_status" + ;; +esac + +for document in README.md docs/installazione-docker-4-contesti.md; do + grep -Fq -- '-f deploy/compose.session-server.yaml.example' "$document" || { + echo "$document omits the required public-server session override" >&2 + exit 1 + } +done +for required in \ + THT_SERVER_WORKSPACE_CONFIG \ + THT_SESSION_RUNTIME_PASSWORD_SOURCE \ + THT_SESSION_MIGRATOR_PASSWORD_SOURCE \ + THT_SESSION_CA_SOURCE; do + grep -q "^$required=" deploy/env/server.env.example || { + echo "server env example omits $required" >&2 + exit 1 + } +done + +echo "deployment command contract passed." diff --git a/scripts/test-external-compose-lifecycle.sh b/scripts/test-external-compose-lifecycle.sh index 8b645ae7..1fd843dc 100755 --- a/scripts/test-external-compose-lifecycle.sh +++ b/scripts/test-external-compose-lifecycle.sh @@ -8,6 +8,7 @@ trap cleanup EXIT HUP INT TERM export THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git export PI_AUTH_FILE=/dev/null +export THT_SECRETS_FILE=/dev/null unset THT_VECTOR_BOOTSTRAP_PASSWORD_SECRET_FILE THT_VECTOR_MIGRATOR_PASSWORD_SECRET_FILE unset THT_VECTOR_READER_PASSWORD_SECRET_FILE THT_VECTOR_WRITER_PASSWORD_SECRET_FILE diff --git a/scripts/test-external-llm-network-config.sh b/scripts/test-external-llm-network-config.sh new file mode 100755 index 00000000..45c47b37 --- /dev/null +++ b/scripts/test-external-llm-network-config.sh @@ -0,0 +1,38 @@ +#!/usr/bin/env bash +# Model providers are external endpoints reached through the ordinary application network. +set -euo pipefail + +root="$(cd "$(dirname "$0")/.." && pwd -P)" +tmp="$(mktemp -d "${TMPDIR%/}/thoth-external-llm.XXXXXX")" +trap 'rm -rf "$tmp"' EXIT HUP INT TERM +printf '%s\n' '{}' >"$tmp/pi-auth.json" +printf '%s\n' 'THT_MODEL_API_KEY=fixture-model-api-key' >"$tmp/thothii.secrets" +chmod 0600 "$tmp/pi-auth.json" "$tmp/thothii.secrets" + +THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git \ +PI_AUTH_FILE="$tmp/pi-auth.json" \ +THT_SECRETS_FILE="$tmp/thothii.secrets" \ +THT_LLM_URL=https://llm.example.invalid/v1 \ + docker compose -f "$root/compose.yaml" config --format json >"$tmp/config.json" + +node - "$tmp/config.json" <<'NODE' +const fs = require("fs"); +const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8")); +if (Object.keys(config.services).sort().join(",") !== "core,frontend") { + throw new Error("external LLM deployment must retain the mandatory two-service stack"); +} +if (Object.keys(config.networks || {}).join(",") !== "thothii") { + throw new Error("external LLM endpoint must not require a provider-owned Docker network"); +} +if (config.services.core.environment?.THT_LLM_URL !== "https://llm.example.invalid/v1") { + throw new Error("core did not receive the generic external LLM endpoint"); +} +const joins = (service, network) => Array.isArray(service.networks) + ? service.networks.includes(network) + : Object.hasOwn(service.networks || {}, network); +if (!joins(config.services.core, "thothii") || !joins(config.services.frontend, "thothii")) { + throw new Error("frontend and core must share only the application network"); +} +NODE + +echo "external LLM network contract passed." diff --git a/scripts/test-no-deployment-coupling-scope.sh b/scripts/test-no-deployment-coupling-scope.sh new file mode 100755 index 00000000..34be5edd --- /dev/null +++ b/scripts/test-no-deployment-coupling-scope.sh @@ -0,0 +1,84 @@ +#!/usr/bin/env bash +# Regression coverage for coupling-scan categories, exact exclusions, and scanner failures. +set -euo pipefail + +root="$(cd "$(dirname "$0")/.." && pwd -P)" +fixture="$(mktemp -d "${TMPDIR%/}/thoth-coupling-scope.XXXXXX")" +trap 'rm -rf "$fixture"' EXIT HUP INT TERM + +new_fixture() { + rm -rf "$fixture/repository" + mkdir -p \ + "$fixture/repository/deploy/env" \ + "$fixture/repository/deploy/workspaces" \ + "$fixture/repository/docker/smoke" \ + "$fixture/repository/docs/install" \ + "$fixture/repository/docs/superpowers/plans" \ + "$fixture/repository/frontend" \ + "$fixture/repository/scripts" + + printf '%s\n' 'services: {}' >"$fixture/repository/compose.yaml" + printf '%s\n' '# generic runtime image' >"$fixture/repository/docker/core.Dockerfile" + printf '%s\n' '# generic smoke' >"$fixture/repository/docker/smoke/core-smoke.sh" + printf '%s\n' '# generic install' >"$fixture/repository/docs/install/local.md" + printf '%s\n' 'THT_LLM_URL=https://llm.example.invalid' >"$fixture/repository/deploy/env/local.env.example" + printf '%s\n' '# generic launcher' >"$fixture/repository/scripts/run-stack.sh" + printf '%s\n' '// generic frontend configuration' >"$fixture/repository/frontend/vite.config.ts" + + # These are the three intentionally allowed categories from the Task 10 boundary. + printf '%s\n' 'historical omics_portal and Chirone record' \ + >"$fixture/repository/docs/superpowers/plans/legacy.md" + printf '%s\n' 'id: psd' >"$fixture/repository/deploy/workspaces/psd.yaml.example" + printf '%s\n' '# migrate PSD sessions from /home/chirone' \ + >"$fixture/repository/docker/session-migrate.sh" +} + +assert_clean() { + "$root/scripts/test-no-deployment-coupling.sh" --root "$fixture/repository" >/dev/null +} + +assert_detected() { + local relative_path="$1" content="$2" output status + new_fixture + mkdir -p "$(dirname "$fixture/repository/$relative_path")" + printf '%s\n' "$content" >"$fixture/repository/$relative_path" + set +e + output="$("$root/scripts/test-no-deployment-coupling.sh" --root "$fixture/repository" 2>&1)" + status=$? + set -e + if [[ $status -ne 1 ]] || ! grep -Fq "$relative_path" <<<"$output"; then + echo "coupling scan missed $relative_path" >&2 + printf '%s\n' "$output" >&2 + exit 1 + fi +} + +new_fixture +assert_clean + +assert_detected compose.yaml 'services: # Chirone runtime coupling' +assert_detected docker/smoke/core-smoke.sh 'test -d /home/chirone' +assert_detected docs/install/local.md 'Install the PSD deployment profile.' +assert_detected deploy/env/local.env.example 'NETWORK=omics_portal' +assert_detected scripts/run-stack.sh 'exec datamart-builder' +assert_detected frontend/vite.config.ts 'const base = "/omics_portal";' +assert_detected scripts/test-qwen-network-config.sh 'require localllm_default' +assert_detected scripts/test-provider-network.sh 'if (!config.networks?.localllm_default?.external) exit 1' +assert_detected deploy/compose.psd-local.yaml 'services: {}' + +new_fixture +mkdir -p "$fixture/bin" +printf '%s\n' '#!/bin/sh' 'exit 2' >"$fixture/bin/rg" +chmod +x "$fixture/bin/rg" +set +e +PATH="$fixture/bin:$PATH" "$root/scripts/test-no-deployment-coupling.sh" \ + --root "$fixture/repository" >"$fixture/rg.out" 2>"$fixture/rg.err" +status=$? +set -e +if [[ $status -ne 2 ]]; then + echo "coupling scan masked an rg failure (status $status)" >&2 + cat "$fixture/rg.out" "$fixture/rg.err" >&2 + exit 1 +fi + +echo "no-coupling scope regression tests passed." diff --git a/scripts/test-no-deployment-coupling.sh b/scripts/test-no-deployment-coupling.sh index 510cf348..d56e2474 100755 --- a/scripts/test-no-deployment-coupling.sh +++ b/scripts/test-no-deployment-coupling.sh @@ -1,69 +1,125 @@ #!/usr/bin/env bash +# Category-based guard for active build, runtime, install, and launch coupling. set -euo pipefail -cd "$(dirname "$0")/.." +script_root="$(cd "$(dirname "$0")/.." && pwd -P)" +scan_root="$script_root" +if [[ "${1:-}" == --root ]]; then + [[ $# -eq 2 ]] || { echo "usage: $0 [--root PATH]" >&2; exit 2; } + scan_root="$2" +elif [[ $# -ne 0 ]]; then + echo "usage: $0 [--root PATH]" >&2 + exit 2 +fi +[[ -d "$scan_root" ]] || { echo "coupling scan root is not a directory: $scan_root" >&2; exit 2; } +cd "$scan_root" -content_targets=( - .dockerignore - compose.yaml - docker-compose.dev.yml - deploy - docker - frontend/vite.config.ts - README.md - docs/install - docs/installazione-docker-4-contesti.md - .env.example - scripts/run-stack.sh - scripts/docker-smoke.sh -) +runtime_files=() +install_files=() +operator_files=() +contract_test_files=() +add_file() { + local array_name="$1" file="$2" + [[ ! -f "$file" ]] || eval "$array_name+=(\"\$file\")" +} -matches=$( - rg -n -i \ - -g '!deploy/workspaces/**' \ - -g '!docker/session-migrate.sh' \ - -g '!docker/cutover-legacy-sessions.sh' \ - -g '!docker/smoke/**' \ - 'omics_portal|chirone|localllm_default|datamart-builder|compose\.production\.yaml|compose\.psd-local\.yaml' \ - "${content_targets[@]}" || true -) +for file in .dockerignore compose.yaml docker-compose.dev.yml frontend/vite.config.ts; do + add_file runtime_files "$file" +done +if [[ -d deploy ]]; then + while IFS= read -r -d '' file; do runtime_files+=("${file#./}"); done < <( + find deploy -type f ! -path 'deploy/workspaces/*' -print0 + ) +fi +if [[ -d docker ]]; then + while IFS= read -r -d '' file; do + case "$file" in + docker/session-migrate.sh|docker/cutover-legacy-sessions.sh) continue ;; + esac + runtime_files+=("${file#./}") + done < <(find docker -type f -print0) +fi -runtime_psd_matches=$( - rg -n -i \ - -g '!deploy/workspaces/**' \ - -g '!docker/session-migrate.sh' \ - -g '!docker/cutover-legacy-sessions.sh' \ - -g '!docker/smoke/**' \ - '\bpsd\b' \ - .dockerignore compose.yaml docker-compose.dev.yml deploy docker frontend/vite.config.ts \ - .env.example scripts/run-stack.sh scripts/docker-smoke.sh || true -) +for file in README.md .env.example docs/installazione-docker-4-contesti.md; do + add_file install_files "$file" +done +if [[ -d docs/install ]]; then + while IFS= read -r -d '' file; do install_files+=("${file#./}"); done < <( + find docs/install -type f -print0 + ) +fi + +if [[ -d scripts ]]; then + while IFS= read -r -d '' file; do + case "${file#scripts/}" in + test-no-deployment-coupling.sh|test-no-deployment-coupling-scope.sh) continue ;; + test-*.sh) + contract_test_files+=("${file#./}") + continue + ;; + verify-*.sh) continue ;; + esac + operator_files+=("${file#./}") + done < <(find scripts -maxdepth 1 -type f -print0) +fi offenders=() -for superseded_file in \ +scan_category() { + local label="$1" pattern="$2"; shift 2 + local output rg_status + (($#)) || return 0 + set +e + output="$(rg -n -i --with-filename -- "$pattern" "$@" 2>&1)" + rg_status=$? + set -e + case "$rg_status" in + 0) + while IFS= read -r match; do offenders+=("$label: $match"); done <<<"$output" + ;; + 1) ;; + *) + echo "coupling scan failed in $label (rg status $rg_status)" >&2 + printf '%s\n' "$output" >&2 + exit "$rg_status" + ;; + esac +} + +for forbidden_file in \ deploy/compose.production.yaml \ deploy/compose.psd-local.yaml.example \ deploy/compose.psd-local.yaml \ scripts/bootstrap-local-psd-docker-config.sh \ - harness/tests/test_psd_local_compose_contract.py -do - [[ ! -e "$superseded_file" ]] || offenders+=("$superseded_file (forbidden active deployment filename)") + scripts/test-qwen-network-config.sh \ + harness/tests/test_psd_local_compose_contract.py; do + [[ ! -e "$forbidden_file" ]] \ + || offenders+=("active filename: $forbidden_file (superseded deployment contract)") done -if [[ -n "$matches" ]]; then - while IFS= read -r match; do - offenders+=("$match") - done <<<"$matches" -fi +forbidden='omics_portal|chirone|localllm_default|datamart-builder|compose\.production\.yaml|compose\.psd-local\.yaml|\bpsd\b' +scan_category runtime "$forbidden" "${runtime_files[@]}" +scan_category install "$forbidden" "${install_files[@]}" +scan_category operator "$forbidden" "${operator_files[@]}" +# Contract tests legitimately quote forbidden names in negative assertions. Scan their positive +# deployment wiring constructs instead, so a provider-owned network or retired overlay cannot be +# required under a different test filename. +positive_contract='networks(\?|\.)?\.?localllm_default|services(\?|\.)?\.?core(\?|\.)?\.?networks(\?|\.)?\.?localllm_default|docker compose[^\n]*(compose\.psd-local|compose\.production)|THT_PSD_[A-Z0-9_]*=' +scan_category contract-test "$positive_contract" "${contract_test_files[@]}" -if [[ -n "$runtime_psd_matches" ]]; then - while IFS= read -r match; do - offenders+=("$match") - done <<<"$runtime_psd_matches" -fi - -if rg -n 'command -v pi|PI_BIN="pi"|PI_BIN=pi' scripts/run-stack.sh >/dev/null; then - offenders+=("scripts/run-stack.sh (requires a host Pi binary)") +if [[ -f scripts/run-stack.sh ]]; then + set +e + host_pi="$(rg -n 'command -v pi|PI_BIN="pi"|PI_BIN=pi' scripts/run-stack.sh 2>&1)" + host_pi_status=$? + set -e + case "$host_pi_status" in + 0) offenders+=("operator: $host_pi") ;; + 1) ;; + *) + echo "coupling scan failed in host-Pi contract (rg status $host_pi_status)" >&2 + printf '%s\n' "$host_pi" >&2 + exit "$host_pi_status" + ;; + esac fi if ((${#offenders[@]})); then diff --git a/scripts/test-pi-user-auth-compose.sh b/scripts/test-pi-user-auth-compose.sh index c50cb10c..10594825 100755 --- a/scripts/test-pi-user-auth-compose.sh +++ b/scripts/test-pi-user-auth-compose.sh @@ -8,13 +8,42 @@ trap 'rm -rf "$tmp"' EXIT HUP INT TERM auth_file="$tmp/auth.json" printf '%s\n' '{}' >"$auth_file" chmod 0600 "$auth_file" +secrets_file="$tmp/thothii.secrets" +printf '%s\n' 'THT_MODEL_API_KEY=fixture-model-api-key' >"$secrets_file" +chmod 0600 "$secrets_file" rendered=$(THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git \ - PI_AUTH_FILE="$auth_file" docker compose config) + PI_AUTH_FILE="$auth_file" THT_SECRETS_FILE="$secrets_file" docker compose config) printf '%s\n' "$rendered" | grep -q "source: $auth_file" printf '%s\n' "$rendered" | grep -q 'target: /home/thoth/.pi/agent/auth.json' printf '%s\n' "$rendered" | grep -A4 'target: /home/thoth/.pi/agent/auth.json' \ | grep -q 'read_only: true' +for target in \ + /home/thoth/.pi/agent/models.json \ + /home/thoth/.pi/agent/settings.json; do + printf '%s\n' "$rendered" | grep -q "target: $target" + printf '%s\n' "$rendered" | grep -A4 "target: $target" | grep -q 'read_only: true' +done +printf '%s\n' "$rendered" | grep -q "file: $secrets_file" +printf '%s\n' "$rendered" | grep -q 'target: thothii.secrets' +if grep -Fq 'fixture-model-api-key' <<<"$rendered"; then + echo "rendered base Compose leaked the model key" >&2 + exit 1 +fi + +dev_rendered=$(THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git \ + PI_AUTH_FILE="$auth_file" THT_SECRETS_FILE="$secrets_file" \ + docker compose --env-file deploy/env/local.env.example -f docker-compose.dev.yml config) +printf '%s\n' "$dev_rendered" | grep -q "source: $auth_file" +printf '%s\n' "$dev_rendered" | grep -q 'target: /home/thoth/.pi/agent/auth.json' +printf '%s\n' "$dev_rendered" | grep -q 'target: /home/thoth/.pi/agent/models.json' +printf '%s\n' "$dev_rendered" | grep -q 'target: /home/thoth/.pi/agent/settings.json' +printf '%s\n' "$dev_rendered" | grep -q "file: $secrets_file" +printf '%s\n' "$dev_rendered" | grep -q 'target: thothii.secrets' +if grep -Fq 'fixture-model-api-key' <<<"$dev_rendered"; then + echo "rendered development Compose leaked the model key" >&2 + exit 1 +fi python3 - <<'PY' import json @@ -32,5 +61,7 @@ PY grep -q '^ARG PI_VERSION=0.80.3$' docker/core.Dockerfile grep -q '^PI_AUTH_FILE=/absolute/path/to/pi-auth.json$' deploy/env/local.env.example grep -q '^PI_AUTH_FILE=/absolute/path/to/pi-auth.json$' deploy/env/server.env.example +grep -q '^THT_SECRETS_FILE=/absolute/path/to/thothii.secrets$' deploy/env/local.env.example +grep -q '^THT_SECRETS_FILE=/absolute/path/to/thothii.secrets$' deploy/env/server.env.example echo "Pi user-auth Compose contract passed." diff --git a/scripts/test-preprocess-compose-config.sh b/scripts/test-preprocess-compose-config.sh index 3249247a..9627e614 100755 --- a/scripts/test-preprocess-compose-config.sh +++ b/scripts/test-preprocess-compose-config.sh @@ -4,7 +4,8 @@ set -eu cd "$(dirname "$0")/.." tmp_bundle=$(mktemp) -trap 'rm -f "$tmp_bundle"' EXIT HUP INT TERM +tmp_auth=$(mktemp) +trap 'rm -f "$tmp_bundle" "$tmp_auth"' EXIT HUP INT TERM cat >"$tmp_bundle" <<'EOF' THT_VECTOR_BOOTSTRAP_PASSWORD=test-bootstrap THT_VECTOR_MIGRATOR_PASSWORD=test-migrator @@ -12,7 +13,11 @@ THT_VECTOR_READER_PASSWORD=test-reader THT_VECTOR_WRITER_PASSWORD=test-writer EOF chmod 0600 "$tmp_bundle" +printf '%s\n' '{}' >"$tmp_auth" +chmod 0600 "$tmp_auth" export THT_SECRETS_FILE="$tmp_bundle" +export PI_AUTH_FILE="$tmp_auth" +export THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git local_files="-f compose.yaml -f deploy/compose.local-vector.yaml -f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml" local_json=$(docker compose $local_files --profile local-vector --profile preprocess config --format json) diff --git a/scripts/test-qwen-network-config.sh b/scripts/test-qwen-network-config.sh deleted file mode 100755 index a40352b1..00000000 --- a/scripts/test-qwen-network-config.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/bin/sh -set -eu - -cd "$(dirname "$0")/.." -tmp=$(mktemp -d) -trap 'rm -rf "$tmp"' EXIT HUP INT TERM -mkdir -p "$tmp/deploy" -cp compose.yaml "$tmp/compose.yaml" -: >"$tmp/deploy/thothii.env" - -docker compose --project-directory "$tmp" -f "$tmp/compose.yaml" config --format json >"$tmp/config.json" -node - "$tmp/config.json" <<'NODE' -const fs = require("fs"); -const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8")); -if (!config.networks?.localllm_default?.external) { - throw new Error("localllm_default must be an external network"); -} -if (!config.services?.core?.networks?.localllm_default) { - throw new Error("core must join localllm_default"); -} -if (config.services?.frontend?.networks?.localllm_default) { - throw new Error("frontend must not join the model network"); -} -NODE diff --git a/scripts/test-unified-compose.sh b/scripts/test-unified-compose.sh index 91203646..4c78205f 100755 --- a/scripts/test-unified-compose.sh +++ b/scripts/test-unified-compose.sh @@ -11,8 +11,12 @@ render_profile() { local env_file=$2 local compose_file=$3 local rendered="$tmp/$profile.json" + local -a files=(-f compose.yaml -f "$compose_file") + if [[ "$profile" == server ]]; then + files+=(-f deploy/compose.session-server.yaml.example) + fi - docker compose --env-file "$env_file" -f compose.yaml -f "$compose_file" \ + docker compose --env-file "$env_file" "${files[@]}" \ config --format json >"$rendered" node - "$rendered" "$profile" <<'NODE' @@ -38,6 +42,28 @@ const piAuthMounts = (config.services.core.volumes || []).filter( if (piAuthMounts.length !== 1 || piAuthMounts[0].type !== "bind" || !piAuthMounts[0].read_only) { throw new Error("Pi auth must be one read-only file bind"); } +if (config.services.core.environment?.THT_SECRETS_FILE !== "/run/secrets/thothii.secrets") { + throw new Error("core must read the canonical runtime secret bundle from /run/secrets"); +} +const runtimeSecrets = config.services.core.secrets || []; +const bundleSecrets = runtimeSecrets.filter( + (secret) => secret.source === "thothii_secrets" && secret.target === "thothii.secrets", +); +if (bundleSecrets.length !== 1) { + throw new Error("core must receive exactly one canonical runtime secret bundle"); +} +if (profile === "local" && runtimeSecrets.length !== 1) { + throw new Error("local core must receive only the canonical runtime secret bundle"); +} +if (profile === "server") { + const targets = new Set(runtimeSecrets.map((secret) => secret.target)); + for (const target of ["session_runtime_password", "session_ca.pem"]) { + if (!targets.has(target)) throw new Error("server core lacks " + target); + } +} +if ((config.services.frontend.secrets || []).length !== 0) { + throw new Error("frontend must not receive runtime secrets"); +} const ports = Object.fromEntries( Object.entries(config.services).map(([name, service]) => [name, service.ports || []]), @@ -60,9 +86,12 @@ assert_remote_required() { local env_file=$1 local compose_file=$2 local without_remote="$tmp/without-remote.env" + local -a files=(-f compose.yaml -f "$compose_file") + [[ "$compose_file" != deploy/compose.server.yaml ]] \ + || files+=(-f deploy/compose.session-server.yaml.example) grep -v '^THT_WORKSPACE_GIT_REMOTE=' "$env_file" >"$without_remote" - if env -u THT_WORKSPACE_GIT_REMOTE docker compose --env-file "$without_remote" -f compose.yaml -f "$compose_file" \ + if env -u THT_WORKSPACE_GIT_REMOTE docker compose --env-file "$without_remote" "${files[@]}" \ config --format json >"$tmp/missing-remote.out" 2>"$tmp/missing-remote.err"; then echo "Compose must require THT_WORKSPACE_GIT_REMOTE" >&2 exit 1 @@ -72,6 +101,7 @@ assert_remote_required() { THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git \ PI_AUTH_FILE=/dev/null \ +THT_SECRETS_FILE=/dev/null \ docker compose -f compose.yaml config --format json >"$tmp/base.json" node - "$tmp/base.json" <<'NODE' const fs = require("fs"); @@ -98,6 +128,18 @@ const piAuthMounts = (config.services.core.volumes || []).filter( if (piAuthMounts.length !== 1 || piAuthMounts[0].type !== "bind" || !piAuthMounts[0].read_only) { throw new Error("Pi auth must be one read-only file bind"); } +if (config.services.core.environment?.THT_SECRETS_FILE !== "/run/secrets/thothii.secrets") { + throw new Error("core must read the canonical runtime secret bundle from /run/secrets"); +} +const runtimeSecrets = config.services.core.secrets || []; +if (runtimeSecrets.length !== 1 + || runtimeSecrets[0].source !== "thothii_secrets" + || runtimeSecrets[0].target !== "thothii.secrets") { + throw new Error("core must receive exactly the canonical runtime secret bundle"); +} +if ((config.services.frontend.secrets || []).length !== 0) { + throw new Error("frontend must not receive runtime secrets"); +} NODE render_profile local deploy/env/local.env.example deploy/compose.local.yaml diff --git a/scripts/test-verify-workspace-install-docs.sh b/scripts/test-verify-workspace-install-docs.sh index b293f8ee..6e7ead0c 100755 --- a/scripts/test-verify-workspace-install-docs.sh +++ b/scripts/test-verify-workspace-install-docs.sh @@ -9,20 +9,11 @@ trap 'rm -f "$output"' EXIT HUP INT TERM "$root/scripts/verify-workspace-install-docs.sh" --fixtures-only >"$output" for fixture in \ - "local manual requires generated connector override and Compose preflight" \ - "server manual requires generated connector override and Compose preflight" \ - "local documented shell environment fixture" \ - "server documented shell environment fixture" \ - "copied local base fixture" \ - "copied server PostgreSQL/TLS fixture" \ - "copied HTTPS Git override fixture" \ - "copied SSH Git override fixture" \ - "copied connector binding/secret fixture" \ - "core process sees connector bindings and secret files" \ - "non-path secret-file fixture rejected" \ - "literal secret-source fixture rejected" \ - "relative secret-source fixture rejected" \ - "non-normalized secret-source fixture rejected"; do + "local manual canonical base+override references" \ + "server manual canonical base+override references" \ + "canonical local base+override fixture" \ + "canonical server base+override fixture" \ + "relative secret-source fixture rejected"; do grep -Fqx "$fixture passed" "$output" >/dev/null || { echo "missing fixture verification: $fixture" >&2 cat "$output" >&2 @@ -37,7 +28,7 @@ for manual in \ echo "installation manual does not publish a self-contained THT_SOURCE_ROOT export: $manual" >&2 exit 1 } - grep -Fq 'export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env"' "$manual" || { + grep -Fq 'export THT_WORKSPACE_BINDINGS_ENV_FILE=' "$manual" || { echo "installation manual does not publish a self-contained bindings export: $manual" >&2 exit 1 } @@ -47,7 +38,7 @@ for manual in \ fi done -if rg -n 'connector-secrets\.workspace-registry|docker compose' \ +if rg -n 'local-compose\.workspace-registry|server-compose\.workspace-registry|connector-secrets\.workspace-registry|docker compose' \ "$root/docs/install/local-workspace-registry.md" \ "$root/docs/install/server-workspace-registry.md"; then echo "installation manuals still document a bypassed Compose or copied connector override path" >&2 diff --git a/scripts/vector-rotate-bootstrap-password.sh b/scripts/vector-rotate-bootstrap-password.sh index ae2836d5..af1e99e3 100755 --- a/scripts/vector-rotate-bootstrap-password.sh +++ b/scripts/vector-rotate-bootstrap-password.sh @@ -29,7 +29,7 @@ trap 'rm -f "$replacement"' EXIT HUP INT TERM cp "$new_secret" "$replacement" chmod 0600 "$replacement" -docker compose -f compose.yaml -f deploy/compose.local-vector.yaml \ +docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local-vector.yaml \ --project-name "$project" --profile local-vector run --rm --no-deps \ --user 0:0 \ --entrypoint /opt/venv/bin/python \ @@ -43,4 +43,4 @@ mv -f "$replacement" "$old_secret" trap - EXIT HUP INT TERM echo "Deployment bootstrap secret atomically replaced only after verified database login." -echo "Re-run: docker compose -f compose.yaml -f deploy/compose.local-vector.yaml --project-name $project --profile local-vector up --wait vector-reconcile vector-migrate core" +echo "Re-run: docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local-vector.yaml --project-name $project --profile local-vector up --wait vector-reconcile vector-migrate core" diff --git a/scripts/verify-workspace-install-docs.sh b/scripts/verify-workspace-install-docs.sh index 6df56dfb..d9875b96 100755 --- a/scripts/verify-workspace-install-docs.sh +++ b/scripts/verify-workspace-install-docs.sh @@ -1,9 +1,9 @@ #!/usr/bin/env bash -# Validate the installation manuals without reading an operator environment or production remote. +# Verify canonical local/server installation manuals and their base+override Compose paths. set -euo pipefail root="$(cd "$(dirname "$0")/.." && pwd -P)" -profile="${1:-}" +mode="${1:-}" trim() { local value="$1" @@ -41,266 +41,13 @@ verify_path_variable_values() { fi fi done <"$source" - return 0 } -verify_server_public_contract() { - local server_example="$root/docs/install/examples/server-compose.workspace-registry.yaml" - for expected in \ - 'THT_SESSION_STORAGE: postgres' \ - 'THT_SESSION_RUNTIME_PASSWORD_FILE: /run/secrets/session_runtime_password' \ - 'THT_SESSION_DB_SSLMODE: ${THT_SESSION_DB_SSLMODE:-verify-full}' \ - 'THT_SESSION_DB_SSLROOTCERT: /run/secrets/session_ca.pem' \ - 'session_runtime_password:' \ - 'session_ca:'; do - grep -Fq "$expected" "$server_example" || { - echo "server Compose example lacks required public PostgreSQL/TLS contract: $expected" >&2 - return 1 - } - done -} - -verify_manual_supported_path() { - local profile="$1" manual="$2" - local source_root_export='export THT_SOURCE_ROOT=/absolute/path/to/ThothII' - local bindings_export='export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env"' - local generator='"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env .env --output connector-secrets.local.yaml' - local wrapper='"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env' - - grep -Fq "$source_root_export" "$manual" || { - echo "$profile manual does not export THT_SOURCE_ROOT for its shell commands" >&2 - return 1 - } - grep -Fq "$bindings_export" "$manual" || { - echo "$profile manual does not export THT_WORKSPACE_BINDINGS_ENV_FILE for its shell commands" >&2 - return 1 - } - grep -Fq "$generator" "$manual" || { - echo "$profile manual does not document the connector override generator" >&2 - return 1 - } - grep -Fq "$wrapper" "$manual" || { - echo "$profile manual does not document the Compose preflight wrapper" >&2 - return 1 - } - if grep -Eq 'connector-secrets\.workspace-registry|docker compose' "$manual"; then - echo "$profile manual documents a bypassed Compose or copied connector override path" >&2 - return 1 - fi - echo "$profile manual requires generated connector override and Compose preflight passed" -} - -compose_fixture() { - local name="$1" directory="$2"; shift 2 - ( - cd "$directory" - "$root/scripts/compose-with-preflight.sh" --env-file .env "$@" config --quiet - ) - echo "$name passed" -} - -prepare_binding_fixture() { - local directory="$1" - printf '%s\n' \ - 'THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct' \ - 'THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password' \ - 'THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api' \ - 'THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key' \ - >"$directory/workspace-bindings.env" - printf 'THT_WORKSPACE_BINDINGS_ENV_FILE=%s\n' "$directory/workspace-bindings.env" >>"$directory/.env" -} - -verify_connector_fixture() { - local directory="$1" rendered project connector_override - project="thoth-install-connector-fixture-$$" - connector_override="$directory/connector-secrets.local.yaml" - "$root/scripts/generate-connector-secrets-override.sh" \ - --bindings-env "$directory/workspace-bindings.env" --operator-env "$directory/.env" \ - --output "$connector_override" >/dev/null - rendered="$( - cd "$directory" - "$root/scripts/compose-with-preflight.sh" --env-file .env \ - -f compose.workspace-registry.yaml -f connector-secrets.local.yaml config - )" - for expected in \ - 'THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT: postgres_direct' \ - 'THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE: /run/secrets/north-star-research-dwh-password' \ - 'THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE: /run/secrets/north-star-research-vector-api-key' \ - 'target: north-star-research-dwh-password' \ - 'target: north-star-research-vector-api-key'; do - grep -Fq "$expected" <<<"$rendered" || { - echo "connector fixture does not give core required binding or secret target: $expected" >&2 - return 1 - } - done - echo "copied connector binding/secret fixture passed" - if ! ( - cd "$directory" - "$root/scripts/compose-with-preflight.sh" --project-name "$project" --env-file .env \ - -f compose.workspace-registry.yaml -f connector-secrets.local.yaml run --rm --no-deps --build --entrypoint sh core -c ' - test "$THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT" = postgres_direct - test "$THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE" = /run/secrets/north-star-research-dwh-password - test "$THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE" = /run/secrets/north-star-research-vector-api-key - test -f "$THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE" - test -f "$THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE" - ' - ); then - ( - cd "$directory" - "$root/scripts/compose-with-preflight.sh" --project-name "$project" --env-file .env \ - -f compose.workspace-registry.yaml -f connector-secrets.local.yaml down --volumes --remove-orphans - ) || true - return 1 - fi - ( - cd "$directory" - "$root/scripts/compose-with-preflight.sh" --project-name "$project" --env-file .env \ - -f compose.workspace-registry.yaml -f connector-secrets.local.yaml down --volumes --remove-orphans - ) - echo "core process sees connector bindings and secret files passed" -} - -verify_documented_operator_path() { - local profile="$1" directory="$2" documented_source_root="$3" connector_override - connector_override="$directory/connector-secrets.local.yaml" - ( - cd "$directory" - unset THT_SOURCE_ROOT THT_WORKSPACE_BINDINGS_ENV_FILE - export THT_SOURCE_ROOT="$documented_source_root" - export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env" - "$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" \ - --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env .env \ - --output connector-secrets.local.yaml >/dev/null - "$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \ - -f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml \ - -f connector-secrets.local.yaml config --quiet - ) - echo "$profile documented shell environment fixture passed" -} - -verify_copied_operator_fixtures() { - local fixture_root local_dir server_dir https_dir ssh_dir connector_dir - fixture_root="$(mktemp -d "${TMPDIR%/}/thoth-install-fixtures.XXXXXX")" - trap 'rm -rf "$fixture_root"' RETURN - local_dir="$fixture_root/local"; server_dir="$fixture_root/server" - https_dir="$fixture_root/https"; ssh_dir="$fixture_root/ssh"; connector_dir="$fixture_root/connector" - mkdir -p "$local_dir" "$server_dir" "$https_dir" "$ssh_dir" "$connector_dir" - - cp "$root/docs/install/examples/local-compose.workspace-registry.yaml" "$local_dir/compose.workspace-registry.yaml" - printf 'THT_SOURCE_ROOT=%s\n' "$root" >"$local_dir/.env" - prepare_binding_fixture "$local_dir" - compose_fixture "copied local base fixture" "$local_dir" -f compose.workspace-registry.yaml - - cp "$root/docs/install/examples/server-compose.workspace-registry.yaml" "$server_dir/compose.workspace-registry.yaml" - cp "$root/deploy/workspaces/server-sessions.yaml.example" "$server_dir/server-sessions.yaml" - : >"$server_dir/session-runtime-password"; : >"$server_dir/session-ca.pem" - printf '%s\n' \ - "THT_SOURCE_ROOT=$root" \ - "THT_SERVER_WORKSPACE_CONFIG=$server_dir/server-sessions.yaml" \ - 'THT_SESSION_DB_HOST=sessions.example.invalid' \ - 'THT_SESSION_DB_NAME=thoth_sessions' \ - 'THT_SESSION_RUNTIME_USER=thoth_sessions_app' \ - "THT_SESSION_RUNTIME_PASSWORD_SOURCE=$server_dir/session-runtime-password" \ - "THT_SESSION_CA_SOURCE=$server_dir/session-ca.pem" >"$server_dir/.env" - prepare_binding_fixture "$server_dir" - compose_fixture "copied server PostgreSQL/TLS fixture" "$server_dir" -f compose.workspace-registry.yaml - - cp "$root/docs/install/examples/local-compose.workspace-registry.yaml" "$https_dir/compose.workspace-registry.yaml" - cp "$root/docs/install/examples/git-https.workspace-registry.yaml" "$https_dir/git-https.yaml" - : >"$https_dir/git-credentials"; : >"$https_dir/git-ca.pem" - printf '%s\n' \ - "THT_SOURCE_ROOT=$root" \ - "THT_WORKSPACE_GIT_CREDENTIALS_FILE=$https_dir/git-credentials" \ - "THT_WORKSPACE_GIT_CA_FILE=$https_dir/git-ca.pem" >"$https_dir/.env" - prepare_binding_fixture "$https_dir" - compose_fixture "copied HTTPS Git override fixture" "$https_dir" -f compose.workspace-registry.yaml -f git-https.yaml - - cp "$root/docs/install/examples/local-compose.workspace-registry.yaml" "$ssh_dir/compose.workspace-registry.yaml" - cp "$root/docs/install/examples/git-ssh.workspace-registry.yaml" "$ssh_dir/git-ssh.yaml" - : >"$ssh_dir/git-ssh-key"; : >"$ssh_dir/git-known-hosts" - printf '%s\n' \ - "THT_SOURCE_ROOT=$root" \ - "THT_WORKSPACE_GIT_SSH_KEY_FILE=$ssh_dir/git-ssh-key" \ - "THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$ssh_dir/git-known-hosts" >"$ssh_dir/.env" - prepare_binding_fixture "$ssh_dir" - compose_fixture "copied SSH Git override fixture" "$ssh_dir" -f compose.workspace-registry.yaml -f git-ssh.yaml - - : >"$server_dir/git-ssh-key"; : >"$server_dir/git-known-hosts" - : >"$server_dir/dwh-password"; : >"$server_dir/vector-api-key" - printf '%s\n' \ - "THT_WORKSPACE_GIT_SSH_KEY_FILE=$server_dir/git-ssh-key" \ - "THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$server_dir/git-known-hosts" \ - "THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$server_dir/dwh-password" \ - "THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=$server_dir/vector-api-key" >>"$server_dir/.env" - cp "$root/docs/install/examples/git-ssh.workspace-registry.yaml" "$server_dir/git-ssh.workspace-registry.yaml" - : >"$ssh_dir/dwh-password"; : >"$ssh_dir/vector-api-key" - printf '%s\n' \ - "THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$ssh_dir/dwh-password" \ - "THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=$ssh_dir/vector-api-key" >>"$ssh_dir/.env" - cp "$root/docs/install/examples/git-ssh.workspace-registry.yaml" "$ssh_dir/git-ssh.workspace-registry.yaml" - verify_documented_operator_path local "$ssh_dir" "$root" - verify_documented_operator_path server "$server_dir" "$root" - - cp "$root/docs/install/examples/local-compose.workspace-registry.yaml" "$connector_dir/compose.workspace-registry.yaml" - : >"$connector_dir/dwh-password"; : >"$connector_dir/vector-password" - printf '%s\n' \ - "THT_SOURCE_ROOT=$root" \ - "THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$connector_dir/dwh-password" \ - "THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=$connector_dir/vector-password" >"$connector_dir/.env" - prepare_binding_fixture "$connector_dir" - verify_connector_fixture "$connector_dir" - - printf 'THT_WS_EXAMPLE_DWH_PASSWORD_FILE=not-a-path\n' >"$fixture_root/non-path-secret.env" - if verify_path_variable_values "$fixture_root/non-path-secret.env" >/dev/null 2>&1; then - echo "non-path secret-file fixture was accepted" >&2 - return 1 - fi - echo "non-path secret-file fixture rejected passed" - - printf 'THT_WS_EXAMPLE_DWH_PASSWORD_SOURCE=literal-value\n' >"$fixture_root/literal-source.env" - if verify_path_variable_values "$fixture_root/literal-source.env" >/dev/null 2>&1; then - echo "literal secret-source fixture was accepted" >&2 - return 1 - fi - echo "literal secret-source fixture rejected passed" - - printf 'THT_WS_EXAMPLE_DWH_PASSWORD_SOURCE=installation-secrets/password\n' >"$fixture_root/relative-source.env" - if verify_path_variable_values "$fixture_root/relative-source.env" >/dev/null 2>&1; then - echo "relative secret-source fixture was accepted" >&2 - return 1 - fi - echo "relative secret-source fixture rejected passed" - - printf 'THT_WS_EXAMPLE_DWH_PASSWORD_SOURCE=/srv/thothii/secrets/../password\n' >"$fixture_root/non-normalized-source.env" - if verify_path_variable_values "$fixture_root/non-normalized-source.env" >/dev/null 2>&1; then - echo "non-normalized secret-source fixture was accepted" >&2 - return 1 - fi - echo "non-normalized secret-source fixture rejected passed" -} - -case "$profile" in - --fixtures-only) - [[ $# -eq 1 ]] || { echo "usage: $0 --fixtures-only" >&2; exit 2; } - verify_manual_supported_path local "$root/docs/install/local-workspace-registry.md" - verify_manual_supported_path server "$root/docs/install/server-workspace-registry.md" - verify_copied_operator_fixtures - exit 0 - ;; - --profile) - profile="${2:-}" - [[ $# -eq 2 ]] || { echo "usage: $0 --profile {local|server}" >&2; exit 2; } - ;; - *) - echo "usage: $0 --profile {local|server}" >&2 - exit 2 - ;; -esac - -case "$profile" in - local) - manual="$root/docs/install/local-workspace-registry.md" - example="$root/docs/install/examples/local-compose.workspace-registry.yaml" +verify_manual() { + local profile="$1" manual + manual="$root/docs/install/$profile-workspace-registry.md" + local -a headings + if [[ "$profile" == local ]]; then headings=( "Prerequisites" "Git remote: SSH and HTTPS" @@ -310,10 +57,7 @@ case "$profile" in "Publish, update, backup, outage recovery, and rollback" "Troubleshooting" ) - ;; - server) - manual="$root/docs/install/server-workspace-registry.md" - example="$root/docs/install/examples/server-compose.workspace-registry.yaml" + else headings=( "Service account, storage, and firewall" "Gitea and remote Git setup" @@ -324,50 +68,186 @@ case "$profile" in "Pull, publish, upgrade, backup, and recovery" "Troubleshooting and snapshot rollback" ) + fi + for heading in "${headings[@]}"; do + grep -Fqx "## $heading" "$manual" || { + echo "missing required heading in $profile manual: $heading" >&2 + return 1 + } + done + for expected in \ + 'export THT_SOURCE_ROOT=/absolute/path/to/ThothII' \ + '--env-file "$THT_OPERATOR_ENV"' \ + "-f \"\$THT_SOURCE_ROOT/compose.yaml\" -f \"\$THT_SOURCE_ROOT/deploy/compose.$profile.yaml\"" \ + '"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh"'; do + grep -Fq -- "$expected" "$manual" || { + echo "$profile manual lacks canonical operator step: $expected" >&2 + return 1 + } + done + if rg -n 'local-compose\.workspace-registry|server-compose\.workspace-registry|connector-secrets\.workspace-registry|docker compose' "$manual"; then + echo "$profile manual documents a superseded or bypassed Compose path" >&2 + return 1 + fi + verify_path_variable_values "$manual" + echo "$profile manual canonical base+override references passed" +} + +write_private() { + local path="$1" value="$2" + printf '%s\n' "$value" >"$path" + chmod 0600 "$path" +} + +verify_compose_fixtures() { + local fixture connector_override profile rendered + fixture="$(mktemp -d "${TMPDIR%/}/thoth-install-fixtures.XXXXXX")" + trap 'rm -rf "$fixture"' RETURN + mkdir -p "$fixture/data" "$fixture/pi-state" "$fixture/workspace-registry" + + write_private "$fixture/pi-auth.json" '{"zai":{"type":"api_key","key":"fixture-native-auth-key"}}' + write_private "$fixture/thothii.secrets" 'THT_MODEL_API_KEY=fixture-model-api-key' + write_private "$fixture/git-ssh-key" 'fixture-git-ssh-key' + write_private "$fixture/git-known-hosts" 'fixture-git-known-hosts' + write_private "$fixture/dwh-password" 'fixture-dwh-password' + write_private "$fixture/vector-api-key" 'fixture-vector-api-key' + write_private "$fixture/session-runtime-password" 'fixture-session-runtime-password' + write_private "$fixture/session-migrator-password" 'fixture-session-migrator-password' + write_private "$fixture/session-ca.pem" 'fixture-session-ca' + cp "$root/deploy/workspaces/server-sessions.yaml.example" "$fixture/server-sessions.yaml" + + printf '%s\n' \ + 'THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct' \ + 'THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password' \ + 'THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api' \ + 'THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key' \ + >"$fixture/workspace-bindings.env" + + printf '%s\n' \ + 'THT_WORKSPACE_GIT_REMOTE=ssh://git@git.example.invalid/platform/thoth-workspaces.git' \ + "PI_AUTH_FILE=$fixture/pi-auth.json" \ + "THT_SECRETS_FILE=$fixture/thothii.secrets" \ + "THT_WORKSPACE_BINDINGS_ENV_FILE=$fixture/workspace-bindings.env" \ + "THT_WORKSPACE_GIT_SSH_KEY_FILE=$fixture/git-ssh-key" \ + "THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$fixture/git-known-hosts" \ + "THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$fixture/dwh-password" \ + "THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=$fixture/vector-api-key" \ + "THT_DATA_ROOT=$fixture/data" \ + "THT_PI_STATE_ROOT=$fixture/pi-state" \ + "THT_WORKSPACE_REGISTRY_ROOT=$fixture/workspace-registry" \ + "THT_SERVER_WORKSPACE_CONFIG=$fixture/server-sessions.yaml" \ + 'THT_SESSION_DB_HOST=sessions.example.invalid' \ + 'THT_SESSION_DB_NAME=thoth_sessions' \ + 'THT_SESSION_RUNTIME_USER=thoth_sessions_app' \ + 'THT_SESSION_MIGRATOR_USER=thoth_sessions_migrate' \ + "THT_SESSION_RUNTIME_PASSWORD_SOURCE=$fixture/session-runtime-password" \ + "THT_SESSION_MIGRATOR_PASSWORD_SOURCE=$fixture/session-migrator-password" \ + "THT_SESSION_CA_SOURCE=$fixture/session-ca.pem" \ + >"$fixture/operator.env" + + connector_override="$fixture/connector-secrets.local.yaml" + "$root/scripts/generate-connector-secrets-override.sh" \ + --bindings-env "$fixture/workspace-bindings.env" \ + --operator-env "$fixture/operator.env" \ + --output "$connector_override" >/dev/null + + for profile in local server; do + rendered="$fixture/$profile.json" + files=( + -f "$root/compose.yaml" + -f "$root/deploy/compose.$profile.yaml" + ) + if [[ "$profile" == server ]]; then + files+=(-f "$root/deploy/compose.session-server.yaml.example") + fi + files+=( + -f "$root/deploy/compose.git-ssh.yaml" + -f "$connector_override" + ) + "$root/scripts/compose-with-preflight.sh" --env-file "$fixture/operator.env" \ + "${files[@]}" config --format json >"$rendered" + + node - "$rendered" "$profile" <<'NODE' +const fs = require("fs"); +const [path, profile] = process.argv.slice(2); +const config = JSON.parse(fs.readFileSync(path, "utf8")); +if (Object.keys(config.services).sort().join(",") !== "core,frontend") { + throw new Error(profile + ": mandatory stack must be exactly core,frontend"); +} +const core = config.services.core; +for (const target of [ + "/home/thoth/.pi/agent/auth.json", + "/home/thoth/.pi/agent/models.json", + "/home/thoth/.pi/agent/settings.json", +]) { + if (!(core.volumes || []).some((mount) => mount.target === target && mount.read_only)) { + throw new Error(profile + ": missing read-only Pi mount " + target); + } +} +for (const [name, value] of Object.entries({ + THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE: "/run/secrets/north-star-research-dwh-password", + THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE: "/run/secrets/north-star-research-vector-api-key", +})) { + if (core.environment?.[name] !== value) throw new Error(profile + ": missing binding " + name); +} +const secretTargets = new Set((core.secrets || []).map((secret) => secret.target)); +for (const target of [ + "thothii.secrets", + "north-star-research-dwh-password", + "north-star-research-vector-api-key", +]) { + if (!secretTargets.has(target)) throw new Error(profile + ": missing secret target " + target); +} +if (profile === "server") { + for (const target of ["session_runtime_password", "session_ca.pem"]) { + if (!secretTargets.has(target)) throw new Error("server: missing session secret " + target); + } +} +if ((config.services.frontend.secrets || []).length !== 0) { + throw new Error(profile + ": frontend received a runtime secret"); +} +const rendered = JSON.stringify(config); +for (const value of [ + "fixture-native-auth-key", "fixture-model-api-key", "fixture-git-ssh-key", + "fixture-git-known-hosts", "fixture-dwh-password", "fixture-vector-api-key", + "fixture-session-runtime-password", "fixture-session-migrator-password", "fixture-session-ca", +]) { + if (rendered.includes(value)) throw new Error(profile + ": rendered Compose leaked " + value); +} +NODE + echo "canonical $profile base+override fixture passed" + done + + printf 'THT_WS_EXAMPLE_DWH_PASSWORD_SOURCE=relative/secret\n' >"$fixture/unsafe.env" + if verify_path_variable_values "$fixture/unsafe.env" >/dev/null 2>&1; then + echo "relative secret-source fixture was accepted" >&2 + return 1 + fi + echo "relative secret-source fixture rejected passed" +} + +case "$mode" in + --fixtures-only) + [[ $# -eq 1 ]] || { echo "usage: $0 --fixtures-only" >&2; exit 2; } + verify_manual local + verify_manual server + verify_compose_fixtures + ;; + --profile) + profile="${2:-}" + [[ $# -eq 2 && "$profile" =~ ^(local|server)$ ]] \ + || { echo "usage: $0 --profile {local|server}" >&2; exit 2; } + verify_manual "$profile" + verify_compose_fixtures + echo "== Run isolated workspace-registry bootstrap and recovery smoke ==" + ( + cd "$root" + env -u WORKSPACE_GIT_REMOTE ./scripts/workspace-registry-smoke.sh + ) + echo "$profile installation documentation verification passed" ;; *) - echo "unknown documentation profile: $profile" >&2 + echo "usage: $0 --fixtures-only | --profile {local|server}" >&2 exit 2 ;; esac - -[[ -f "$manual" ]] || { echo "missing $profile installation manual: $manual" >&2; exit 1; } -[[ -f "$example" ]] || { echo "missing $profile Compose example: $example" >&2; exit 1; } - -for heading in "${headings[@]}"; do - grep -Fqx "## $heading" "$manual" >/dev/null || { - echo "missing required heading in $profile manual: $heading" >&2 - exit 1 - } -done - -grep -Fq "$(basename "$example")" "$manual" || { - echo "the $profile manual does not reference its Compose example" >&2 - exit 1 -} - -# Values for secret-bearing variables must be paths. These patterns catch common accidental -# credentials while allowing declarative *_FILE bindings and explicitly empty assignments. -if grep -Ein '(^|[[:space:]])(password|api[_-]?key|token|secret)[[:space:]]*[:=][[:space:]]*[^[:space:]#]' \ - "$manual" "$example" >/dev/null; then - echo "installation documentation contains a secret literal" >&2 - exit 1 -fi -verify_path_variable_values "$manual" -verify_path_variable_values "$example" -verify_path_variable_values "$root/docs/install/examples/git-https.workspace-registry.yaml" -verify_path_variable_values "$root/docs/install/examples/git-ssh.workspace-registry.yaml" -verify_path_variable_values "$root/docs/install/examples/workspace-bindings.env.example" -verify_server_public_contract -verify_manual_supported_path "$profile" "$manual" - -echo "== Validate copied operator fixtures and documented optional Git transports ==" -verify_copied_operator_fixtures - -echo "== Run isolated workspace-registry bootstrap and recovery smoke ==" -( - cd "$root" - env -u WORKSPACE_GIT_REMOTE ./scripts/workspace-registry-smoke.sh -) - -echo "$profile installation documentation verification passed"