fix: close deployment decoupling review

This commit is contained in:
2026-08-05 07:52:32 +02:00
parent 5d037e97c4
commit 09834d5cd4
45 changed files with 1082 additions and 791 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
# Common non-secret Compose values. Select local.env or server.env with --env-file. # Common non-secret Compose values. Select local.env or server.env with --env-file.
# Run Compose with both files explicitly, for example: # Run Compose with both files explicitly, for example:
# docker compose -f compose.yaml -f deploy/compose.local.yaml up -d --build # docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up -d --build
MAX_PI_PROCESSES=4 MAX_PI_PROCESSES=4
THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git
+12 -1
View File
@@ -6,7 +6,8 @@
## Portable deployment decoupling — LIVE 2026-08-05 ## Portable deployment decoupling — LIVE 2026-08-05
- **Mandatory stack.** The supported Compose stack is exactly `frontend` plus `core`; use the - **Mandatory stack.** The supported Compose stack is exactly `frontend` plus `core`; use the
base file with `deploy/compose.local.yaml` or `deploy/compose.server.yaml`. `run-stack.sh` base file with `deploy/compose.local.yaml`, or with `deploy/compose.server.yaml` plus the
required public-server session overlay. `run-stack.sh`
invokes the base+local Compose command and the core image provides Pi, so no host Pi binary is invokes the base+local Compose command and the core image provides Pi, so no host Pi binary is
part of the launch contract. part of the launch contract.
- **External boundaries.** DWH, vector DB, embedding, LLM, and reverse-proxy services are - **External boundaries.** DWH, vector DB, embedding, LLM, and reverse-proxy services are
@@ -18,6 +19,16 @@
remaining live contract checks were renamed for the generic local Compose profile. The coupling remaining live contract checks were renamed for the generic local Compose profile. The coupling
gate rejects stale active deployment filenames and content while deliberately excluding gate rejects stale active deployment filenames and content while deliberately excluding
historical plans/specs, canonical workspace descriptors, and non-runtime migration helpers. historical plans/specs, canonical workspace descriptors, and non-runtime migration helpers.
- **Fresh provider and secret contract.** Local, server, and standalone development mount the
protected Pi auth JSON plus tracked declarative model/settings files read-only under
`/home/thoth/.pi/agent`. The existing strict application bundle is a core-only Docker secret at
`/run/secrets/thothii.secrets`; operator env files contain only its absolute source path.
Provider readiness is exercised from a fresh Compose volume through model listing, configuration,
and sanitized credential status.
- **Install and scan closure.** Superseded copied one-service installation examples and the
provider-owned-network test are retired. Active manuals use the canonical base plus local/server
and optional overrides, while the category-based coupling scan covers runtime, Docker smoke,
install, operator, and positive deployment-test contracts and propagates scanner errors.
## Portable Git workspace registry — source integration (2026-08-04) ## Portable Git workspace registry — source integration (2026-08-04)
+20 -16
View File
@@ -14,14 +14,22 @@ From a fresh clone, run these commands from the repository root:
```sh ```sh
cp deploy/env/local.env.example deploy/env/local.env cp deploy/env/local.env.example deploy/env/local.env
# Edit deploy/env/local.env, including PI_AUTH_FILE and the external endpoint URLs. # Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
docker compose --env-file deploy/env/local.env \ docker compose --env-file deploy/env/local.env \
-f compose.yaml -f deploy/compose.local.yaml up --build -d -f compose.yaml -f deploy/compose.local.yaml up --build -d
``` ```
`./scripts/run-stack.sh` runs this same base+local command in the foreground. The core image `./scripts/run-stack.sh` runs this same base+local command in the foreground. The core image
contains its Pi runtime; no host `pi` executable is used. For a server installation, copy and contains its Pi runtime; no host `pi` executable is used. For a server installation:
fill `deploy/env/server.env.example`, then use `-f compose.yaml -f deploy/compose.server.yaml`.
```sh
cp deploy/env/server.env.example deploy/env/server.env
# Edit all absolute storage, Pi/secret/session files, and endpoint paths.
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example up --build -d
```
Workspace descriptors come from the Git remote configured by `THT_WORKSPACE_GIT_REMOTE`; their Workspace descriptors come from the Git remote configured by `THT_WORKSPACE_GIT_REMOTE`; their
runtime endpoint and secret bindings remain installation-local. Open runtime endpoint and secret bindings remain installation-local. Open
<http://127.0.0.1:8080> (set `THOTH_HTTP_PORT` in `deploy/env/local.env` to choose another <http://127.0.0.1:8080> (set `THOTH_HTTP_PORT` in `deploy/env/local.env` to choose another
@@ -164,15 +172,10 @@ with the organization's reviewed identity proxy. `AUTH_MODE=upstream` trusts thi
rejects requests without the identity header. Setting `THOTH_PUBLIC_EXPOSURE=true` with any other rejects requests without the identity header. Setting `THOTH_PUBLIC_EXPOSURE=true` with any other
auth mode fails during core startup. auth mode fails during core startup.
Production credentials use the one Compose secret bundle, not an environment example. Put the Production credentials use the existing Compose secret-bundle contract, never environment values.
required keys in `deploy/secrets/thothii.secrets` for the selected base+server installation: Copy `deploy/secrets/thothii.secrets.example` to a protected host file, include only the required
keys, and set its absolute path as `THT_SECRETS_FILE` in the operator env. Keep Pi's native
```dotenv provider auth in the separate protected file named by `PI_AUTH_FILE`.
THT_MODEL_API_KEY=replace-me
THT_DWH_API_KEY=replace-me
THT_VEC_API_KEY=replace-me
THT_VEC_WRITE_API_KEY=replace-me
```
The bundle is mounted read-only as `/run/secrets/thothii.secrets` and must be mode `0600` or The bundle is mounted read-only as `/run/secrets/thothii.secrets` and must be mode `0600` or
`0400` on the host. Docker's runtime `0444` mode is accepted only beneath `/run/secrets`; see `0400` on the host. Docker's runtime `0444` mode is accepted only beneath `/run/secrets`; see
@@ -204,9 +207,9 @@ still scrubbed. Supporting them requires a future dedicated provider-specific co
The server profile stores sessions and per-user preferences directly in PostgreSQL schema The server profile stores sessions and per-user preferences directly in PostgreSQL schema
`thoth_sessions`; it does not use PostgREST, browser storage, a shared session directory, or a `thoth_sessions`; it does not use PostgREST, browser storage, a shared session directory, or a
dual write. Start from [`deploy/compose.session-server.yaml.example`](deploy/compose.session-server.yaml.example) dual write. Use [`deploy/compose.session-server.yaml.example`](deploy/compose.session-server.yaml.example)
and copy [`deploy/workspaces/server-sessions.yaml.example`](deploy/workspaces/server-sessions.yaml.example) with the canonical base+server files and set `THT_SERVER_WORKSPACE_CONFIG` to an absolute,
to the untracked `deploy/workspaces/server-sessions.yaml` mounted into the core container. protected copy of [`deploy/workspaces/server-sessions.yaml.example`](deploy/workspaces/server-sessions.yaml.example).
The runtime login needs membership in the no-login database role `thoth_sessions_runtime` only. The runtime login needs membership in the no-login database role `thoth_sessions_runtime` only.
The distinct, one-shot migrator login needs migration authority and uses The distinct, one-shot migrator login needs migration authority and uses
@@ -242,7 +245,8 @@ proxy clears the legacy identity header and the backend rejects it. Drain/stop a
enable a maintenance response at the proxy, then run the migrator once and inspect its pristine JSON: enable a maintenance response at the proxy, then run the migrator once and inspect its pristine JSON:
```sh ```sh
docker compose -f compose.yaml -f deploy/compose.session-server.yaml \ docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml -f deploy/compose.session-server.yaml.example \
--profile session-migrate run --rm session-migrate --profile session-migrate run --rm session-migrate
``` ```
+10
View File
@@ -21,6 +21,7 @@ services:
THT_WORKSPACE_GIT_AUTHOR_NAME: ${THT_WORKSPACE_GIT_AUTHOR_NAME:-Thoth Workspace Registry} THT_WORKSPACE_GIT_AUTHOR_NAME: ${THT_WORKSPACE_GIT_AUTHOR_NAME:-Thoth Workspace Registry}
THT_WORKSPACE_GIT_AUTHOR_EMAIL: ${THT_WORKSPACE_GIT_AUTHOR_EMAIL:-thoth-workspace-registry@localhost} THT_WORKSPACE_GIT_AUTHOR_EMAIL: ${THT_WORKSPACE_GIT_AUTHOR_EMAIL:-thoth-workspace-registry@localhost}
THT_WORKSPACE_SECRET_ROOTS: /run/secrets THT_WORKSPACE_SECRET_ROOTS: /run/secrets
THT_SECRETS_FILE: /run/secrets/thothii.secrets
THT_DB_NAME: ${THT_DB_NAME:-} THT_DB_NAME: ${THT_DB_NAME:-}
THT_DWH_REST_URL: ${THT_DWH_REST_URL:-} THT_DWH_REST_URL: ${THT_DWH_REST_URL:-}
THT_VEC_REST_URL: ${THT_VEC_REST_URL:-} THT_VEC_REST_URL: ${THT_VEC_REST_URL:-}
@@ -32,8 +33,13 @@ services:
- settings:/data/settings - settings:/data/settings
- pi-state:/home/thoth/.pi - pi-state:/home/thoth/.pi
- ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro - ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro
- ./deploy/pi/models.json:/home/thoth/.pi/agent/models.json:ro
- ./deploy/pi/settings.json:/home/thoth/.pi/agent/settings.json:ro
- workspace-registry:/data/workspace-registry - workspace-registry:/data/workspace-registry
- sessions:/data/sessions - sessions:/data/sessions
secrets:
- source: thothii_secrets
target: thothii.secrets
healthcheck: healthcheck:
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"] test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"]
interval: 15s interval: 15s
@@ -71,3 +77,7 @@ volumes:
pi-state: pi-state:
workspace-registry: workspace-registry:
sessions: sessions:
secrets:
thothii_secrets:
file: "${THT_SECRETS_FILE:?set THT_SECRETS_FILE}"
+3
View File
@@ -33,3 +33,6 @@ services:
- thoth_data:/data - thoth_data:/data
- ./deploy/workspaces:/app/harness/workspaces:ro - ./deploy/workspaces:/app/harness/workspaces:ro
restart: "no" restart: "no"
volumes:
thoth_data:
+2
View File
@@ -9,6 +9,8 @@ services:
- ${THT_DATA_ROOT:?set THT_DATA_ROOT}:/data - ${THT_DATA_ROOT:?set THT_DATA_ROOT}:/data
- ${THT_PI_STATE_ROOT:?set THT_PI_STATE_ROOT}:/home/thoth/.pi - ${THT_PI_STATE_ROOT:?set THT_PI_STATE_ROOT}:/home/thoth/.pi
- ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro - ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro
- ./deploy/pi/models.json:/home/thoth/.pi/agent/models.json:ro
- ./deploy/pi/settings.json:/home/thoth/.pi/agent/settings.json:ro
- ${THT_WORKSPACE_REGISTRY_ROOT:?set THT_WORKSPACE_REGISTRY_ROOT}:/data/workspace-registry - ${THT_WORKSPACE_REGISTRY_ROOT:?set THT_WORKSPACE_REGISTRY_ROOT}:/data/workspace-registry
restart: unless-stopped restart: unless-stopped
+1 -1
View File
@@ -20,7 +20,7 @@ services:
- source: session_ca - source: session_ca
target: session_ca.pem target: session_ca.pem
volumes: volumes:
- ./deploy/workspaces:/app/harness/workspaces:ro - ${THT_SERVER_WORKSPACE_CONFIG:?set THT_SERVER_WORKSPACE_CONFIG}:/app/harness/workspaces/server-sessions.yaml:ro
# Run manually during the maintenance window. It is not a dependency of core, # Run manually during the maintenance window. It is not a dependency of core,
# so the application never gains the schema-changing migrator credential. # so the application never gains the schema-changing migrator credential.
-40
View File
@@ -1,40 +0,0 @@
# Deprecated compatibility template; it is not loaded by Docker Compose automatically.
# New installations must copy ../.env.example to ../.env and run
# `docker compose up --build -d` from the repository root. Keep this file only for
# staged upgrades that still invoke `--env-file deploy/env.example` explicitly.
# Never put secret values in this file.
COMPOSE_FILE=compose.yaml
COMPOSE_PROFILES=
THT_SECRETS_FILE=deploy/secrets/thothii.secrets
PI_PROVIDER=
PI_MODEL=
PI_THINKING=
MAX_PI_PROCESSES=4
AUTH_MODE=none
# User-owned session storage. Keep local for the loopback-only development stack.
# The server-session overlay requires every THT_SESSION_* value below.
THT_SESSION_STORAGE=local
THT_SESSION_DB_HOST=
THT_SESSION_DB_PORT=5432
THT_SESSION_DB_NAME=
THT_SESSION_RUNTIME_USER=
THT_SESSION_RUNTIME_PASSWORD_SOURCE=
THT_SESSION_MIGRATOR_USER=
THT_SESSION_MIGRATOR_PASSWORD_SOURCE=
THT_SESSION_DB_SSLMODE=verify-full
THT_SESSION_CA_SOURCE=
THT_DB_NAME=
THT_DWH_REST_URL=
THT_VEC_REST_URL=
THT_OLLAMA_URL=
THT_DOCS_ROOT=/data/workspaces/example/evidence-source
THT_VECTOR_DATABASE=thoth
THT_VECTOR_BOOTSTRAP_USER=postgres
THT_VECTOR_MIGRATOR_USER=thoth_vector_migrator
THT_VECTOR_READER_USER=thoth_vector_reader
THT_VECTOR_WRITER_USER=thoth_vector_writer
+1
View File
@@ -4,6 +4,7 @@ THOTH_HTTP_PORT=8080
THOTH_CORE_HTTP_PORT=8787 THOTH_CORE_HTTP_PORT=8787
MAX_PI_PROCESSES=4 MAX_PI_PROCESSES=4
PI_AUTH_FILE=/absolute/path/to/pi-auth.json PI_AUTH_FILE=/absolute/path/to/pi-auth.json
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git
THT_WORKSPACE_GIT_BRANCH=main THT_WORKSPACE_GIT_BRANCH=main
+13
View File
@@ -4,10 +4,12 @@ THOTH_SERVER_BIND=127.0.0.1
THOTH_HTTP_PORT=8080 THOTH_HTTP_PORT=8080
MAX_PI_PROCESSES=4 MAX_PI_PROCESSES=4
PI_AUTH_FILE=/absolute/path/to/pi-auth.json PI_AUTH_FILE=/absolute/path/to/pi-auth.json
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
THT_DATA_ROOT=/srv/thothii/data THT_DATA_ROOT=/srv/thothii/data
THT_PI_STATE_ROOT=/srv/thothii/pi-state THT_PI_STATE_ROOT=/srv/thothii/pi-state
THT_WORKSPACE_REGISTRY_ROOT=/srv/thothii/workspace-registry THT_WORKSPACE_REGISTRY_ROOT=/srv/thothii/workspace-registry
THT_SERVER_WORKSPACE_CONFIG=/absolute/path/to/server-sessions.yaml
THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git
THT_WORKSPACE_GIT_BRANCH=main THT_WORKSPACE_GIT_BRANCH=main
THT_WORKSPACE_GIT_AUTHOR_NAME="Thoth Workspace Registry" THT_WORKSPACE_GIT_AUTHOR_NAME="Thoth Workspace Registry"
@@ -19,3 +21,14 @@ THT_VEC_REST_URL=https://vector.example.invalid
THT_VEC_WRITE_REST_URL=https://vector-write.example.invalid THT_VEC_WRITE_REST_URL=https://vector-write.example.invalid
THT_OLLAMA_URL=https://embeddings.example.invalid THT_OLLAMA_URL=https://embeddings.example.invalid
THT_LLM_URL=https://llm.example.invalid THT_LLM_URL=https://llm.example.invalid
# Public server session storage. Values are endpoints, roles, or protected source-file paths.
THT_SESSION_DB_HOST=sessions-db.example.invalid
THT_SESSION_DB_PORT=5432
THT_SESSION_DB_NAME=thoth_sessions
THT_SESSION_RUNTIME_USER=thoth_sessions_app
THT_SESSION_MIGRATOR_USER=thoth_sessions_migrate
THT_SESSION_DB_SSLMODE=verify-full
THT_SESSION_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/session-runtime-password
THT_SESSION_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/session-migrator-password
THT_SESSION_CA_SOURCE=/absolute/path/to/session-ca.pem
+7 -4
View File
@@ -12,7 +12,7 @@ The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). T
keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_VEC_API_KEY`, keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_VEC_API_KEY`,
`THT_VEC_WRITE_API_KEY`, and the four `THT_VECTOR_*_PASSWORD` role passwords. Values must be `THT_VEC_WRITE_API_KEY`, and the four `THT_VECTOR_*_PASSWORD` role passwords. Values must be
non-empty and contain no whitespace. Do not put secrets in the root `.env`, workspace YAML, non-empty and contain no whitespace. Do not put secrets in the root `.env`, workspace YAML,
URLs, logs, or `docker compose config` output. URLs, logs, or rendered Compose output.
Compose mounts the bundle read-only as `/run/secrets/thothii.secrets`. The host file must be a Compose mounts the bundle read-only as `/run/secrets/thothii.secrets`. The host file must be a
regular non-symlink file with mode `0600` or `0400`; Docker's normal `0444` mode is accepted regular non-symlink file with mode `0600` or `0400`; Docker's normal `0444` mode is accepted
@@ -20,7 +20,9 @@ only for the runtime mount beneath `/run/secrets`. The core runs as UID 10001. V
without printing its contents: without printing its contents:
```sh ```sh
docker compose run --rm core sh -c 'id && test -r /run/secrets/thothii.secrets' docker compose --env-file deploy/env/local.env \
-f compose.yaml -f deploy/compose.local.yaml \
run --rm core sh -c 'id && test -r /run/secrets/thothii.secrets'
``` ```
A private CA PEM chain is not a bundle value: PEM whitespace is rejected by the strict parser. A private CA PEM chain is not a bundle value: PEM whitespace is rejected by the strict parser.
@@ -31,9 +33,10 @@ Compose files intentionally do not create this mount.
## Migration from separate secret files ## Migration from separate secret files
Older installations used `THT_*_SECRET_FILE` variables and one file per value. Migrate by Older installations used `THT_*_SECRET_FILE` variables and one file per value. Migrate by
copying each value to its bundle key, validating with `docker compose config --quiet`, and only copying each value to its bundle key, validating with the complete base+profile command, and only
then deleting the old files. The old variables remain a compatibility path for staged upgrades, then deleting the old files. The old variables remain a compatibility path for staged upgrades,
but the documented and tested default is `THT_SECRETS_FILE=deploy/secrets/thothii.secrets`. but the documented and tested default is an absolute `THT_SECRETS_FILE` path to the protected
bundle.
The local-vector bootstrap rotation helper still accepts an old/new password file as its The local-vector bootstrap rotation helper still accepts an old/new password file as its
maintenance interface. Run it only with files protected by `0600`, then copy the resulting maintenance interface. Run it only with files protected by `0600`, then copy the resulting
-33
View File
@@ -1,33 +0,0 @@
# ThothII core — env di runtime (compose env_file).
# Copiare in deploy/thothii.env e completare. NON committare thothii.env.
# --- DWH (direct, ruolo read-only su schema datawarehouse) ---
THT_DB_HOST=host.docker.internal
THT_DB_PORT=5438
THT_DB_NAME=postgres
THT_DB_USER=thoth_dwh_reader
THT_DB_PASSWORD=__CHANGE_ME__
# --- Vector (direct, ruolo read+write su schema vectors; stessa istanza del DWH) ---
THT_VEC_HOST=host.docker.internal
THT_VEC_PORT=5438
THT_VEC_USER=thoth_vector_rw
THT_VEC_PASSWORD=__CHANGE_ME__
# --- Embeddings (Ollama sull'host, modello nomic-embed-text-v2-moe) ---
THT_OLLAMA_URL=http://host.docker.internal:11434
# --- Backend ---
AUTH_MODE=none # none | mock | oidc (upstream auth is enforced at the proxy boundary)
MAX_PI_PROCESSES=4
THT_DEV_EVIDENCE_HOST_PATH=/absolute/path/to/evidence
# --- Git-backed workspace registry (no secret values belong in this file) ---
THT_WORKSPACE_REGISTRY_ROOT=/data/workspace-registry
THT_WORKSPACE_GIT_BRANCH=main
THT_WORKSPACE_INSTALLATION_ID=server
# THT_WORKSPACE_GIT_REMOTE=ssh://git@your-git-host/your-org/thoth-workspaces.git
# THT_WORKSPACE_GIT_CREDENTIALS_FILE=/absolute/path/to/git-credentials
# THT_WORKSPACE_GIT_CA_FILE=/absolute/path/to/git-ca.pem
# THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/to/git-ssh-key
# THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/to/git-known-hosts
+33 -20
View File
@@ -1,7 +1,7 @@
# ThothII — deploy STANDALONE locale (dev / smoke test). # ThothII standalone development/smoke stack.
# Rete propria + porte host per ispezione diretta. # Run with the canonical local env file:
# docker compose -f docker-compose.dev.yml up -d --build # docker compose --env-file deploy/env/local.env -f docker-compose.dev.yml up -d --build
# frontend: http://localhost:8090 backend: http://localhost:8787 # frontend: http://localhost:8090 backend: http://localhost:8787
name: thothii-dev name: thothii-dev
services: services:
@@ -10,19 +10,18 @@ services:
context: . context: .
dockerfile: docker/core.Dockerfile dockerfile: docker/core.Dockerfile
image: thothii-core:local image: thothii-core:local
env_file:
- path: deploy/thothii.env
required: false
environment: environment:
HOST: 0.0.0.0 HOST: 0.0.0.0
PORT: "8787" PORT: "8787"
THT_HARNESS_DIR: /app/harness THT_HARNESS_DIR: /app/harness
THT_BIN: /opt/venv/bin/tht THT_BIN: /opt/venv/bin/tht
PI_BIN: pi PI_BIN: pi
AUTH_MODE: ${AUTH_MODE:-none} AUTH_MODE: none
THT_SESSION_STORAGE: local THT_SESSION_STORAGE: local
THT_HOME: /data/local-home THT_HOME: /data/local-home
THT_DATA_ROOT: /data
SETTINGS_FILE: /data/settings/settings.json SETTINGS_FILE: /data/settings/settings.json
THT_MAINTENANCE_FILE: /data/settings/maintenance.json
THT_WORKSPACE_REGISTRY_ROOT: /data/workspace-registry THT_WORKSPACE_REGISTRY_ROOT: /data/workspace-registry
THT_WORKSPACE_GIT_REMOTE: ${THT_WORKSPACE_GIT_REMOTE:?set THT_WORKSPACE_GIT_REMOTE} THT_WORKSPACE_GIT_REMOTE: ${THT_WORKSPACE_GIT_REMOTE:?set THT_WORKSPACE_GIT_REMOTE}
THT_WORKSPACE_GIT_BRANCH: ${THT_WORKSPACE_GIT_BRANCH:-main} THT_WORKSPACE_GIT_BRANCH: ${THT_WORKSPACE_GIT_BRANCH:-main}
@@ -30,26 +29,35 @@ services:
THT_WORKSPACE_GIT_AUTHOR_NAME: ${THT_WORKSPACE_GIT_AUTHOR_NAME:-Thoth Workspace Registry} THT_WORKSPACE_GIT_AUTHOR_NAME: ${THT_WORKSPACE_GIT_AUTHOR_NAME:-Thoth Workspace Registry}
THT_WORKSPACE_GIT_AUTHOR_EMAIL: ${THT_WORKSPACE_GIT_AUTHOR_EMAIL:-thoth-workspace-registry@localhost} THT_WORKSPACE_GIT_AUTHOR_EMAIL: ${THT_WORKSPACE_GIT_AUTHOR_EMAIL:-thoth-workspace-registry@localhost}
THT_WORKSPACE_SECRET_ROOTS: /run/secrets THT_WORKSPACE_SECRET_ROOTS: /run/secrets
GIT_CONFIG_COUNT: "2" THT_SECRETS_FILE: /run/secrets/thothii.secrets
GIT_CONFIG_KEY_0: credential.helper THT_DB_NAME: ${THT_DB_NAME:-}
GIT_CONFIG_VALUE_0: store --file=/run/secrets/workspace-registry-git-credentials THT_DWH_REST_URL: ${THT_DWH_REST_URL:-}
GIT_CONFIG_KEY_1: http.sslCAInfo THT_VEC_REST_URL: ${THT_VEC_REST_URL:-}
GIT_CONFIG_VALUE_1: /run/secrets/workspace-registry-git-ca THT_VEC_WRITE_REST_URL: ${THT_VEC_WRITE_REST_URL:-}
GIT_SSH_COMMAND: ssh -i /run/secrets/workspace-registry-git-ssh-key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/run/secrets/workspace-registry-git-known-hosts THT_OLLAMA_URL: ${THT_OLLAMA_URL:-}
THT_LLM_URL: ${THT_LLM_URL:-}
MAX_PI_PROCESSES: ${MAX_PI_PROCESSES:-4} MAX_PI_PROCESSES: ${MAX_PI_PROCESSES:-4}
extra_hosts: extra_hosts:
- "host.docker.internal:host-gateway" - "host.docker.internal:host-gateway"
volumes: volumes:
- dev-data:/data - dev-data:/data
- workspace-registry:/data/workspace-registry
- dev-pi-state:/home/thoth/.pi - dev-pi-state:/home/thoth/.pi
- ${PI_AUTH_FILE:?set PI_AUTH_FILE}:/home/thoth/.pi/agent/auth.json:ro
- ./deploy/pi/models.json:/home/thoth/.pi/agent/models.json:ro
- ./deploy/pi/settings.json:/home/thoth/.pi/agent/settings.json:ro
- workspace-registry:/data/workspace-registry
- ${THT_DEV_EVIDENCE_HOST_PATH:-./evidence}:/data/evidence:ro - ${THT_DEV_EVIDENCE_HOST_PATH:-./evidence}:/data/evidence:ro
- ${THT_WORKSPACE_GIT_CREDENTIALS_FILE:-/dev/null}:/run/secrets/workspace-registry-git-credentials:ro secrets:
- ${THT_WORKSPACE_GIT_CA_FILE:-/etc/ssl/certs/ca-certificates.crt}:/run/secrets/workspace-registry-git-ca:ro - source: thothii_secrets
- ${THT_WORKSPACE_GIT_SSH_KEY_FILE:-/dev/null}:/run/secrets/workspace-registry-git-ssh-key:ro target: thothii.secrets
- ${THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE:-/dev/null}:/run/secrets/workspace-registry-git-known-hosts:ro
ports: ports:
- "127.0.0.1:8787:8787" - "127.0.0.1:8787:8787"
healthcheck:
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"]
interval: 15s
timeout: 3s
retries: 5
start_period: 30s
restart: "no" restart: "no"
networks: [thothii-net] networks: [thothii-net]
@@ -64,7 +72,8 @@ services:
ports: ports:
- "127.0.0.1:8090:8080" - "127.0.0.1:8090:8080"
depends_on: depends_on:
- core core:
condition: service_healthy
restart: "no" restart: "no"
networks: [thothii-net] networks: [thothii-net]
@@ -76,3 +85,7 @@ volumes:
dev-data: dev-data:
dev-pi-state: dev-pi-state:
workspace-registry: workspace-registry:
secrets:
thothii_secrets:
file: "${THT_SECRETS_FILE:?set THT_SECRETS_FILE}"
+2 -2
View File
@@ -8,8 +8,8 @@ La documentazione è divisa in due aree:
Come funziona il sistema: architettura, specifiche di design delle singole funzionalità, piani di implementazione, report di test. Parte da qui: [Panoramica dell'architettura](architecture/overview.md). Come funziona il sistema: architettura, specifiche di design delle singole funzionalità, piani di implementazione, report di test. Parte da qui: [Panoramica dell'architettura](architecture/overview.md).
Per installare l'applicazione in Docker nei quattro contesti operativi, partendo dal comando Per installare l'applicazione in Docker nei quattro contesti operativi, usando il file env,
predefinito `docker compose up --build -d` e dal bundle unico dei secret: `compose.yaml`, l'overlay locale/server e il bundle di secret montato:
[Installazione Docker nei quattro contesti](installazione-docker-4-contesti.md). [Installazione Docker nei quattro contesti](installazione-docker-4-contesti.md).
## Considerazioni Generali ## Considerazioni Generali
@@ -1,13 +0,0 @@
# Optional override for an HTTPS Git remote. Both source paths are required absolute paths to
# existing operator-managed files; neither file content belongs in the base Compose example.
services:
core:
environment:
GIT_CONFIG_COUNT: "2"
GIT_CONFIG_KEY_0: credential.helper
GIT_CONFIG_VALUE_0: store --file=/run/secrets/workspace-registry-git-credentials
GIT_CONFIG_KEY_1: http.sslCAInfo
GIT_CONFIG_VALUE_1: /run/secrets/workspace-registry-git-ca
volumes:
- ${THT_WORKSPACE_GIT_CREDENTIALS_FILE:?set THT_WORKSPACE_GIT_CREDENTIALS_FILE}:/run/secrets/workspace-registry-git-credentials:ro
- ${THT_WORKSPACE_GIT_CA_FILE:?set THT_WORKSPACE_GIT_CA_FILE}:/run/secrets/workspace-registry-git-ca:ro
@@ -1,9 +0,0 @@
# Optional override for an SSH Git remote. Source paths are required absolute operator-managed
# files. Host-key checking remains strict; do not add a fallback known-hosts or key mount.
services:
core:
environment:
GIT_SSH_COMMAND: ssh -i /run/secrets/workspace-registry-git-ssh-key -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=/run/secrets/workspace-registry-git-known-hosts
volumes:
- ${THT_WORKSPACE_GIT_SSH_KEY_FILE:?set THT_WORKSPACE_GIT_SSH_KEY_FILE}:/run/secrets/workspace-registry-git-ssh-key:ro
- ${THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE:?set THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE}:/run/secrets/workspace-registry-git-known-hosts:ro
@@ -1,39 +0,0 @@
# Standalone local registry example. Copy to an untracked operator directory and set the absolute
# THT_SOURCE_ROOT in .env. Add only the selected Git transport override from this directory.
name: thothii-workspace-registry-local
services:
core:
image: thothii-core:local
build:
context: ${THT_SOURCE_ROOT:?set THT_SOURCE_ROOT to the absolute ThothII source checkout}
dockerfile: docker/core.Dockerfile
env_file:
- path: ${THT_WORKSPACE_BINDINGS_ENV_FILE:?set THT_WORKSPACE_BINDINGS_ENV_FILE to an absolute THT_WS bindings file}
required: true
environment:
HOST: 0.0.0.0
PORT: "8787"
AUTH_MODE: none
THT_SESSION_STORAGE: local
THT_HOME: /data/local-home
SETTINGS_FILE: /data/settings/settings.json
THT_HARNESS_DIR: /app/harness
THT_BIN: /opt/venv/bin/tht
THT_WORKSPACE_REGISTRY_ROOT: /data/workspace-registry
THT_WORKSPACE_GIT_REMOTE: ${THT_WORKSPACE_GIT_REMOTE:-ssh://git@git.example.invalid/platform/thoth-workspaces.git}
THT_WORKSPACE_GIT_BRANCH: ${THT_WORKSPACE_GIT_BRANCH:-main}
THT_WORKSPACE_INSTALLATION_ID: ${THT_WORKSPACE_INSTALLATION_ID:-local-laptop}
THT_WORKSPACE_GIT_AUTHOR_NAME: ${THT_WORKSPACE_GIT_AUTHOR_NAME:-Thoth Workspace Registry}
THT_WORKSPACE_GIT_AUTHOR_EMAIL: ${THT_WORKSPACE_GIT_AUTHOR_EMAIL:-thoth-workspace-registry@localhost}
THT_WORKSPACE_SECRET_ROOTS: /run/secrets
ports:
- "127.0.0.1:8787:8787"
volumes:
- thoth-local-data:/data
- workspace-registry:/data/workspace-registry
restart: "no"
volumes:
thoth-local-data: {}
workspace-registry: {}
@@ -1,60 +0,0 @@
# Server registry example. Copy to a reviewed, untracked operator directory and set absolute host
# paths and Git values in .env. Add a selected Git transport override from this directory.
name: thothii-workspace-registry-server
services:
core:
image: thothii-core:local
build:
context: ${THT_SOURCE_ROOT:?set THT_SOURCE_ROOT to the absolute ThothII source checkout}
dockerfile: docker/core.Dockerfile
env_file:
- path: ${THT_WORKSPACE_BINDINGS_ENV_FILE:?set THT_WORKSPACE_BINDINGS_ENV_FILE to an absolute THT_WS bindings file}
required: true
environment:
HOST: 0.0.0.0
PORT: "8787"
AUTH_MODE: upstream
THOTH_PUBLIC_EXPOSURE: "true"
THT_SESSION_STORAGE: postgres
THT_CONFIG: /app/harness/workspaces/server-sessions.yaml
THT_SESSION_DB_HOST: ${THT_SESSION_DB_HOST:?set THT_SESSION_DB_HOST}
THT_SESSION_DB_PORT: ${THT_SESSION_DB_PORT:-5432}
THT_SESSION_DB_NAME: ${THT_SESSION_DB_NAME:?set THT_SESSION_DB_NAME}
THT_SESSION_RUNTIME_USER: ${THT_SESSION_RUNTIME_USER:?set THT_SESSION_RUNTIME_USER}
THT_SESSION_RUNTIME_PASSWORD_FILE: /run/secrets/session_runtime_password
THT_SESSION_DB_SSLMODE: ${THT_SESSION_DB_SSLMODE:-verify-full}
THT_SESSION_DB_SSLROOTCERT: /run/secrets/session_ca.pem
THT_HARNESS_DIR: /app/harness
THT_BIN: /opt/venv/bin/tht
SETTINGS_FILE: /data/settings/settings.json
THT_WORKSPACE_REGISTRY_ROOT: /data/workspace-registry
THT_WORKSPACE_GIT_REMOTE: ${THT_WORKSPACE_GIT_REMOTE:-ssh://git@git.example.invalid/platform/thoth-workspaces.git}
THT_WORKSPACE_GIT_BRANCH: ${THT_WORKSPACE_GIT_BRANCH:-main}
THT_WORKSPACE_INSTALLATION_ID: ${THT_WORKSPACE_INSTALLATION_ID:-production-1}
THT_WORKSPACE_GIT_AUTHOR_NAME: ${THT_WORKSPACE_GIT_AUTHOR_NAME:-Thoth Workspace Registry}
THT_WORKSPACE_GIT_AUTHOR_EMAIL: ${THT_WORKSPACE_GIT_AUTHOR_EMAIL:-thoth-workspace-registry@localhost}
THT_WORKSPACE_SECRET_ROOTS: /run/secrets
volumes:
- ${THT_HOST_DATA_ROOT:-/srv/thothii/data}:/data
- ${THT_WORKSPACE_REGISTRY_HOST_PATH:-/srv/thothii/workspace-registry}:/data/workspace-registry
- ${THT_SERVER_WORKSPACE_CONFIG:?set THT_SERVER_WORKSPACE_CONFIG}:/app/harness/workspaces/server-sessions.yaml:ro
secrets:
- source: session_runtime_password
target: session_runtime_password
- source: session_ca
target: session_ca.pem
networks:
- upstream
restart: unless-stopped
networks:
upstream:
external: true
name: ${THT_UPSTREAM_NETWORK:-thothii-upstream}
secrets:
session_runtime_password:
file: ${THT_SESSION_RUNTIME_PASSWORD_SOURCE:?set THT_SESSION_RUNTIME_PASSWORD_SOURCE}
session_ca:
file: ${THT_SESSION_CA_SOURCE:?set THT_SESSION_CA_SOURCE}
@@ -1,13 +1,13 @@
# Copy to an untracked operator file. This file contains only non-secret THT_WS_* bindings. # Copy to an untracked operator file. This file contains only non-secret THT_WS_* bindings.
# Every *_FILE value is a container path supplied by the generated local connector override. # Every *_FILE value is a container path supplied by the generated local connector override.
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=postgres_direct THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
THT_WS_PSD_CLINICAL_DWH_HOST=dwh.internal.example THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.internal.example
THT_WS_PSD_CLINICAL_DWH_PORT=5432 THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-clinical-dwh-password THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=pgvector_direct THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=pgvector_direct
THT_WS_PSD_CLINICAL_VECTOR_HOST=vector.internal.example THT_WS_NORTH_STAR_RESEARCH_VECTOR_HOST=vector.internal.example
THT_WS_PSD_CLINICAL_VECTOR_PORT=5432 THT_WS_NORTH_STAR_RESEARCH_VECTOR_PORT=5432
THT_WS_PSD_CLINICAL_VECTOR_USER=thoth_vector_reader THT_WS_NORTH_STAR_RESEARCH_VECTOR_USER=thoth_vector_reader
THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE=/run/secrets/psd-clinical-vector-password THT_WS_NORTH_STAR_RESEARCH_VECTOR_PASSWORD_FILE=/run/secrets/north-star-research-vector-password
THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL=https://embeddings.internal.example THT_WS_NORTH_STAR_RESEARCH_EMBEDDING_BASE_URL=https://embeddings.internal.example
+59 -52
View File
@@ -34,14 +34,16 @@ workspaces/<workspace-id>.md
For SSH, use a scoped deploy key, a verified `known_hosts` file, and strict host-key checking. For For SSH, use a scoped deploy key, a verified `known_hosts` file, and strict host-key checking. For
HTTPS, use Git Credential Manager or a secret-manager-created credentials file. Mount a private HTTPS, use Git Credential Manager or a secret-manager-created credentials file. Mount a private
HTTPS CA as its own file. Do not disable host or certificate verification. The base Compose file HTTPS CA as its own file. Do not disable host or certificate verification. The base Compose file
does not mount a Git credential: add exactly one optional `git-ssh.workspace-registry.yaml` or does not mount a Git credential: add exactly one optional `deploy/compose.git-ssh.yaml` or
`git-https.workspace-registry.yaml` override, so unused credential paths are never bind-mounted. `deploy/compose.git-https.yaml` override, so unused credential paths are never bind-mounted.
```dotenv ```dotenv
THT_WORKSPACE_GIT_REMOTE=ssh://git@git.example.invalid/platform/thoth-workspaces.git THT_WORKSPACE_GIT_REMOTE=ssh://git@git.example.invalid/platform/thoth-workspaces.git
THT_WORKSPACE_GIT_BRANCH=main THT_WORKSPACE_GIT_BRANCH=main
THT_WORKSPACE_INSTALLATION_ID=local-laptop THT_WORKSPACE_INSTALLATION_ID=local-laptop
THT_SOURCE_ROOT=/absolute/path/to/ThothII THT_SOURCE_ROOT=/absolute/path/to/ThothII
PI_AUTH_FILE=/absolute/path/installation-secrets/pi-auth.json
THT_SECRETS_FILE=/absolute/path/installation-secrets/thothii.secrets
THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/installation-secrets/git-ssh-key THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/installation-secrets/git-ssh-key
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/installation-secrets/git-known-hosts THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/installation-secrets/git-known-hosts
THT_WORKSPACE_GIT_CA_FILE=/absolute/path/installation-secrets/git-ca.pem THT_WORKSPACE_GIT_CA_FILE=/absolute/path/installation-secrets/git-ca.pem
@@ -69,12 +71,12 @@ state/ # active revision and registry state
locks/ # short-lived publish locks locks/ # short-lived publish locks
``` ```
Installation variables are deterministic: `psd-clinical` becomes `PSD_CLINICAL`, and every name Installation variables are deterministic: `north-star-research` becomes `NORTH_STAR_RESEARCH`, and every name
is `THT_WS_<NAMESPACE>_<ROLE>_<SUFFIX>`. Copy is `THT_WS_<NAMESPACE>_<ROLE>_<SUFFIX>`. Copy
[the bindings env example](examples/workspace-bindings.env.example) to an untracked operator file [the bindings env example](examples/workspace-bindings.env.example) to an untracked operator file
and set its absolute path as `THT_WORKSPACE_BINDINGS_ENV_FILE`. It is loaded only into `core`. and set its absolute path as `THT_WORKSPACE_BINDINGS_ENV_FILE`. It is loaded only into `core`.
Credentials and certificates use `*_FILE` path variables that must point inside `/run/secrets`. Credentials and certificates use `*_FILE` path variables that must point inside `/run/secrets`.
If declared, `THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE` is distinct from the vector reader If declared, `THT_WS_NORTH_STAR_RESEARCH_VECTOR_WRITER_API_KEY_FILE` is distinct from the vector reader
file; a reader credential is never repurposed for writing. file; a reader credential is never repurposed for writing.
## Direct PostgreSQL, REST, and SSH tunnel bindings ## Direct PostgreSQL, REST, and SSH tunnel bindings
@@ -87,41 +89,41 @@ copy or maintain a workspace-specific Compose override.
```dotenv ```dotenv
# Direct PostgreSQL and pgvector # Direct PostgreSQL and pgvector
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=postgres_direct THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
THT_WS_PSD_CLINICAL_DWH_HOST=dwh.example.invalid THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.example.invalid
THT_WS_PSD_CLINICAL_DWH_PORT=5432 THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=pgvector_direct THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=pgvector_direct
THT_WS_PSD_CLINICAL_VECTOR_HOST=vector.example.invalid THT_WS_NORTH_STAR_RESEARCH_VECTOR_HOST=vector.example.invalid
THT_WS_PSD_CLINICAL_VECTOR_PORT=5432 THT_WS_NORTH_STAR_RESEARCH_VECTOR_PORT=5432
THT_WS_PSD_CLINICAL_VECTOR_USER=thoth_vector_reader THT_WS_NORTH_STAR_RESEARCH_VECTOR_USER=thoth_vector_reader
THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE=/run/secrets/psd-vector-reader THT_WS_NORTH_STAR_RESEARCH_VECTOR_PASSWORD_FILE=/run/secrets/north-star-research-vector-password
THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL=https://embeddings.example.invalid THT_WS_NORTH_STAR_RESEARCH_EMBEDDING_BASE_URL=https://embeddings.example.invalid
``` ```
```dotenv ```dotenv
# REST; an API-key file is needed only for a declared bearer/x-api-key diagnostic. # REST; an API-key file is needed only for a declared bearer/x-api-key diagnostic.
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=rest_api THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
THT_WS_PSD_CLINICAL_DWH_BASE_URL=https://dwh.example.invalid THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.example.invalid
THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE=/run/secrets/psd-dwh-api-key THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=rest_api THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api
THT_WS_PSD_CLINICAL_VECTOR_BASE_URL=https://vectors.example.invalid THT_WS_NORTH_STAR_RESEARCH_VECTOR_BASE_URL=https://vectors.example.invalid
THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE=/run/secrets/psd-vector-api-key THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key
``` ```
```dotenv ```dotenv
# SSH tunnel diagnostic only; runtime sessions are fail-closed in this release. # SSH tunnel diagnostic only; runtime sessions are fail-closed in this release.
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=ssh_tunnel THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel
THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_PSD_CLINICAL_DWH_SSH_HOST=bastion.example.invalid THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.example.invalid
THT_WS_PSD_CLINICAL_DWH_SSH_PORT=22 THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22
THT_WS_PSD_CLINICAL_DWH_SSH_USER=thoth_tunnel THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel
THT_WS_PSD_CLINICAL_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/psd-dwh-tunnel-key THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key
THT_WS_PSD_CLINICAL_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/psd-dwh-known-hosts THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts
THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_HOST=dwh.internal.example THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_PORT=5432 THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
``` ```
Repeat the SSH names for `VECTOR` where needed. REST diagnostics reject a private per-request CA Repeat the SSH names for `VECTOR` where needed. REST diagnostics reject a private per-request CA
@@ -134,31 +136,36 @@ before creating sessions. Git pull/push over SSH remains fully supported and is
## Bootstrap, first pull, and diagnostics ## Bootstrap, first pull, and diagnostics
Copy [the local Compose example](examples/local-compose.workspace-registry.yaml), exactly one Use the repository's canonical `compose.yaml` plus `deploy/compose.local.yaml`; they always start
selected [SSH Git override](examples/git-ssh.workspace-registry.yaml) or [HTTPS Git override](examples/git-https.workspace-registry.yaml), the mandatory `frontend` and `core` services. Do not copy or maintain a standalone application
and [the bindings env example](examples/workspace-bindings.env.example) into an untracked operator Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an
directory. Keep `THT_SOURCE_ROOT` and the absolute `THT_WORKSPACE_BINDINGS_ENV_FILE` in its `.env` untracked operator directory and create a protected operator env file from
for Compose interpolation; this keeps the copied Compose file buildable and confines `THT_WS_*` `deploy/env/local.env.example`. It must contain absolute `PI_AUTH_FILE`,
values to `core`. A Compose `.env` file is not a shell environment, so do not import it into the `THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths.
maintenance shell. Instead, explicitly export the two non-secret paths before running the commands. The Pi auth JSON, runtime secret bundle, and each connector credential remain separate protected
Create the host secret files named by the selected Git transport and every declared connector host files and are mounted read-only; their contents never enter the operator env or rendered
`*_SOURCE`, then generate the connector override and render through the preflight wrapper. The Compose.
wrapper is required: it rejects unsafe source paths and a combined SSH+HTTPS Git selection before
Compose runs. Select exactly one repository Git transport override, `deploy/compose.git-ssh.yaml` or
`deploy/compose.git-https.yaml`. A Compose env file is not a shell environment, so export only the
non-secret paths required by the maintenance commands. Generate the connector override and render
through the preflight wrapper, which rejects unsafe paths and combined SSH+HTTPS selection.
```sh ```sh
export THT_SOURCE_ROOT=/absolute/path/to/ThothII export THT_SOURCE_ROOT=/absolute/path/to/ThothII
export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env" export THT_OPERATOR_ENV=/absolute/path/to/operator/local.env
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env .env --output connector-secrets.local.yaml export THT_WORKSPACE_BINDINGS_ENV_FILE=/absolute/path/to/operator/workspace-bindings.env
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \ export THT_CONNECTOR_OVERRIDE=/absolute/path/to/operator/connector-secrets.local.yaml
-f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml config --quiet "$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
-f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \
-f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" config --quiet
``` ```
From the operator directory:
```sh ```sh
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \ "$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
-f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml up --build -d -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \
-f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" up --build -d
curl --fail --silent http://127.0.0.1:8787/health curl --fail --silent http://127.0.0.1:8787/health
curl --fail --silent http://127.0.0.1:8787/workspace-registry/status curl --fail --silent http://127.0.0.1:8787/workspace-registry/status
curl --fail --silent http://127.0.0.1:8787/workspaces curl --fail --silent http://127.0.0.1:8787/workspaces
@@ -169,7 +176,7 @@ Use `POST /workspace-registry/pull` to fetch later revisions. Run workspace diag
required bindings are mounted. The optional writer probe uses a distinct writer file and removes required bindings are mounted. The optional writer probe uses a distinct writer file and removes
its uniquely named temporary record; ordinary diagnostics are read-only. its uniquely named temporary record; ordinary diagnostics are read-only.
To migrate an existing PSD descriptor, create/clone an empty private remote, set the absolute To migrate an existing legacy descriptor, create/clone an empty private remote, set the absolute
`THT_SOURCE_ROOT`, transform with absolute paths, review the schema-v1 result, explicitly add `THT_SOURCE_ROOT`, transform with absolute paths, review the schema-v1 result, explicitly add
vector database/schema and the complete schema-v2 contract, then commit/push. The transformer vector database/schema and the complete schema-v2 contract, then commit/push. The transformer
never imports `${ENV}` values or secrets. never imports `${ENV}` values or secrets.
@@ -177,7 +184,7 @@ never imports `${ENV}` values or secrets.
```sh ```sh
THT_SOURCE_ROOT=/absolute/path/to/ThothII THT_SOURCE_ROOT=/absolute/path/to/ThothII
npm --prefix "$THT_SOURCE_ROOT/backend" run build npm --prefix "$THT_SOURCE_ROOT/backend" run build
node "$THT_SOURCE_ROOT/backend/dist/workspaces/migrate-legacy.js" --input /absolute/path/psd.yaml --output /absolute/path/thoth-workspaces node "$THT_SOURCE_ROOT/backend/dist/workspaces/migrate-legacy.js" --input /absolute/path/legacy.yaml --output /absolute/path/thoth-workspaces
``` ```
## Publish, update, backup, outage recovery, and rollback ## Publish, update, backup, outage recovery, and rollback
+63 -49
View File
@@ -48,11 +48,13 @@ THT_WORKSPACE_GIT_CREDENTIALS_FILE=/srv/thothii/secrets/git-credentials
THT_WORKSPACE_GIT_CA_FILE=/srv/thothii/secrets/git-ca.pem THT_WORKSPACE_GIT_CA_FILE=/srv/thothii/secrets/git-ca.pem
THT_WORKSPACE_GIT_SSH_KEY_FILE=/srv/thothii/secrets/git-ssh-key THT_WORKSPACE_GIT_SSH_KEY_FILE=/srv/thothii/secrets/git-ssh-key
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/srv/thothii/secrets/git-known-hosts THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/srv/thothii/secrets/git-known-hosts
PI_AUTH_FILE=/srv/thothii/secrets/pi-auth.json
THT_SECRETS_FILE=/srv/thothii/secrets/thothii.secrets
``` ```
Use the credential file for HTTPS, or key and known-hosts for SSH. The base server Compose file Use the credential file for HTTPS, or key and known-hosts for SSH. The base server Compose file
mounts neither transport; add exactly one [HTTPS override](examples/git-https.workspace-registry.yaml) mounts neither transport; add exactly one `deploy/compose.git-https.yaml`
or [SSH override](examples/git-ssh.workspace-registry.yaml). Strict host-key checking stays enabled or `deploy/compose.git-ssh.yaml` override. Strict host-key checking stays enabled
and Git stderr is not exposed by the API. Rotate by atomically replacing the secret file, and Git stderr is not exposed by the API. Rotate by atomically replacing the secret file,
restarting `core`, and performing pull/status; never put the material in an environment variable or restarting `core`, and performing pull/status; never put the material in an environment variable or
rendered Compose output. rendered Compose output.
@@ -75,9 +77,9 @@ The runtime registry layout is persistent and must be backed up together:
/data/workspace-registry/locks/ /data/workspace-registry/locks/
``` ```
Variable names derive from the immutable ID: `psd-clinical` becomes `PSD_CLINICAL`, producing Variable names derive from the immutable ID: `north-star-research` becomes `NORTH_STAR_RESEARCH`, producing
`THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE`. A declared vector writer uses the distinct `THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE`. A declared vector writer uses the distinct
`THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE`; a reader file is never a writer substitute. `THT_WS_NORTH_STAR_RESEARCH_VECTOR_WRITER_API_KEY_FILE`; a reader file is never a writer substitute.
Copy [the bindings env example](examples/workspace-bindings.env.example) to the protected operator Copy [the bindings env example](examples/workspace-bindings.env.example) to the protected operator
directory. Every path-valued `*_FILE` entry needs an absolute host-only `*_SOURCE` path. Generate directory. Every path-valued `*_FILE` entry needs an absolute host-only `*_SOURCE` path. Generate
the untracked connector override from those files during bootstrap; do not copy or maintain a the untracked connector override from those files during bootstrap; do not copy or maintain a
@@ -90,41 +92,41 @@ dimensions, and distance as Git-shared identity.
```dotenv ```dotenv
# Direct PostgreSQL/pgvector with verified native TLS if a CA path is supplied. # Direct PostgreSQL/pgvector with verified native TLS if a CA path is supplied.
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=postgres_direct THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
THT_WS_PSD_CLINICAL_DWH_HOST=dwh.internal.example THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.internal.example
THT_WS_PSD_CLINICAL_DWH_PORT=5432 THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=pgvector_direct THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=pgvector_direct
THT_WS_PSD_CLINICAL_VECTOR_HOST=vector.internal.example THT_WS_NORTH_STAR_RESEARCH_VECTOR_HOST=vector.internal.example
THT_WS_PSD_CLINICAL_VECTOR_PORT=5432 THT_WS_NORTH_STAR_RESEARCH_VECTOR_PORT=5432
THT_WS_PSD_CLINICAL_VECTOR_USER=thoth_vector_reader THT_WS_NORTH_STAR_RESEARCH_VECTOR_USER=thoth_vector_reader
THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE=/run/secrets/psd-vector-reader THT_WS_NORTH_STAR_RESEARCH_VECTOR_PASSWORD_FILE=/run/secrets/north-star-research-vector-password
``` ```
```dotenv ```dotenv
# REST needs API-key file paths only when the descriptor declares authenticated diagnostics. # REST needs API-key file paths only when the descriptor declares authenticated diagnostics.
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=rest_api THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
THT_WS_PSD_CLINICAL_DWH_BASE_URL=https://dwh.internal.example THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.internal.example
THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE=/run/secrets/psd-dwh-api-key THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=rest_api THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api
THT_WS_PSD_CLINICAL_VECTOR_BASE_URL=https://vectors.internal.example THT_WS_NORTH_STAR_RESEARCH_VECTOR_BASE_URL=https://vectors.internal.example
THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE=/run/secrets/psd-vector-api-key THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key
THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL=https://embeddings.internal.example THT_WS_NORTH_STAR_RESEARCH_EMBEDDING_BASE_URL=https://embeddings.internal.example
``` ```
```dotenv ```dotenv
# SSH tunnel diagnostic only; runtime sessions are fail-closed in this release. # SSH tunnel diagnostic only; runtime sessions are fail-closed in this release.
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=ssh_tunnel THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel
THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_PSD_CLINICAL_DWH_SSH_HOST=bastion.internal.example THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.internal.example
THT_WS_PSD_CLINICAL_DWH_SSH_PORT=22 THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22
THT_WS_PSD_CLINICAL_DWH_SSH_USER=thoth_tunnel THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel
THT_WS_PSD_CLINICAL_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/psd-dwh-tunnel-key THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key
THT_WS_PSD_CLINICAL_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/psd-dwh-known-hosts THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts
THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_HOST=dwh.internal.example THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_PORT=5432 THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
``` ```
Repeat SSH variables for `VECTOR` when selected. REST diagnostics refuse private per-request CAs Repeat SSH variables for `VECTOR` when selected. REST diagnostics refuse private per-request CAs
@@ -138,15 +140,17 @@ The Git registry itself may still use SSH normally.
## Same-origin reverse proxy, bootstrap, and health ## Same-origin reverse proxy, bootstrap, and health
Copy [the server Compose example](examples/server-compose.workspace-registry.yaml) plus exactly one Use the repository's canonical `compose.yaml` plus `deploy/compose.server.yaml`; they always
selected Git override to the protected operator directory. Set `THT_SOURCE_ROOT` to the absolute start the mandatory `frontend` and `core` services. Do not copy or maintain a standalone
ThothII checkout; a copied file cannot use a relative build context. Copy application Compose file. Review `deploy/workspaces/server-sessions.yaml.example`, materialize it
`deploy/workspaces/server-sessions.yaml.example` into that operator directory, review it, then set as a protected host file, and set its absolute `THT_SERVER_WORKSPACE_CONFIG` path. Copy the
the absolute `THT_SERVER_WORKSPACE_CONFIG` path. Copy the bindings env example, then set absolute bindings env example into the operator directory, then set absolute `PI_AUTH_FILE`,
`THT_WORKSPACE_BINDINGS_ENV_FILE` and connector `*_SOURCE` paths. The same `.env` must set `THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths.
The same operator env must set
`THT_SESSION_DB_HOST`, `THT_SESSION_DB_NAME`, `THT_SESSION_RUNTIME_USER`, `THT_SESSION_DB_HOST`, `THT_SESSION_DB_NAME`, `THT_SESSION_RUNTIME_USER`,
`THT_SESSION_RUNTIME_PASSWORD_SOURCE`, and `THT_SESSION_CA_SOURCE`; the base Compose file wires `THT_SESSION_RUNTIME_PASSWORD_SOURCE`, and `THT_SESSION_CA_SOURCE`;
`postgres`, `verify-full`, and the two Docker secret mount paths. This is the public server profile, `deploy/compose.session-server.yaml.example` wires `postgres`, `verify-full`, and separate
runtime/CA Docker secret mount paths. This is the public server profile,
not a filesystem-session fallback. A Compose `.env` file is not a shell environment, so do not not a filesystem-session fallback. A Compose `.env` file is not a shell environment, so do not
import it into the maintenance shell. Explicitly export the non-secret source and bindings paths import it into the maintenance shell. Explicitly export the non-secret source and bindings paths
before running the commands below. before running the commands below.
@@ -161,14 +165,24 @@ From a trusted maintenance shell:
```sh ```sh
export THT_SOURCE_ROOT=/absolute/path/to/ThothII export THT_SOURCE_ROOT=/absolute/path/to/ThothII
export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env" export THT_OPERATOR_ENV=/srv/thothii/operator/server.env
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env .env --output connector-secrets.local.yaml export THT_WORKSPACE_BINDINGS_ENV_FILE=/srv/thothii/operator/workspace-bindings.env
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \ export THT_CONNECTOR_OVERRIDE=/srv/thothii/operator/connector-secrets.local.yaml
-f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml up --build -d "$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \ "$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
-f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml exec -T core curl --fail --silent http://127.0.0.1:8787/health -f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \ -f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" \
-f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml exec -T core curl --fail --silent http://127.0.0.1:8787/workspace-registry/status -f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" up --build -d
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
-f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \
-f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" \
-f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" \
exec -T core curl --fail --silent http://127.0.0.1:8787/health
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
-f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.server.yaml" \
-f "$THT_SOURCE_ROOT/deploy/compose.session-server.yaml.example" \
-f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" \
exec -T core curl --fail --silent http://127.0.0.1:8787/workspace-registry/status
``` ```
`/health` is liveness. Registry status verifies branch/head/degraded state and the active validated `/health` is liveness. Registry status verifies branch/head/degraded state and the active validated
@@ -187,7 +201,7 @@ filesystem-consistent backup of `/srv/thothii/workspace-registry` plus `/srv/tho
`/srv/thothii/secrets`. Render Compose, deploy the compatible image, verify health/status, then `/srv/thothii/secrets`. Render Compose, deploy the compatible image, verify health/status, then
resume proxy traffic. resume proxy traffic.
For PSD migration, use a temporary review clone and the legacy transformer with absolute paths. For legacy descriptor migration, use a temporary review clone and the legacy transformer with absolute paths.
Its schema-v1 output is `migration_required`; explicitly supply vector database/schema, collection Its schema-v1 output is `migration_required`; explicitly supply vector database/schema, collection
identity, diagnostics, and the reviewed v2 contract before commit. Never import `${ENV}` values or identity, diagnostics, and the reviewed v2 contract before commit. Never import `${ENV}` values or
copy secret files. copy secret files.
+38 -14
View File
@@ -15,6 +15,8 @@ Servono Docker Engine/Compose v2 su Linux oppure Docker Desktop su macOS/Windows
git clone <URL-REPOSITORY> ThothII git clone <URL-REPOSITORY> ThothII
cd ThothII cd ThothII
cp deploy/env/local.env.example deploy/env/local.env cp deploy/env/local.env.example deploy/env/local.env
cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets
chmod 600 deploy/secrets/thothii.secrets
``` ```
Modificare **solo** questi file interni al clone: Modificare **solo** questi file interni al clone:
@@ -25,7 +27,8 @@ Modificare **solo** questi file interni al clone:
| file protetti locali | credenziali e certificati, indicati dai binding del workspace | | file protetti locali | credenziali e certificati, indicati dai binding del workspace |
| `deploy/workspaces/<nome>.yaml` | adapter, endpoint non riservati, `roots` ed Evidence | | `deploy/workspaces/<nome>.yaml` | adapter, endpoint non riservati, `roots` ed Evidence |
Compilare `deploy/env/local.env`, incluso `PI_AUTH_FILE`, con gli endpoint esterni. L'avvio Compilare `deploy/env/local.env`, inclusi i path assoluti `PI_AUTH_FILE` e
`THT_SECRETS_FILE`, con gli endpoint esterni. L'avvio
normale usa esplicitamente il file base e l'overlay locale: normale usa esplicitamente il file base e l'overlay locale:
```sh ```sh
@@ -52,7 +55,7 @@ THT_VECTOR_READER_PASSWORD=...
THT_VECTOR_WRITER_PASSWORD=... THT_VECTOR_WRITER_PASSWORD=...
``` ```
Inserire solo le chiavi necessarie al profilo scelto. Il bundle viene montato in sola lettura nel container come `/run/secrets/thothii.secrets`; il parser rifiuta duplicati, chiavi sconosciute, valori vuoti, symlink e permessi host troppo aperti. Non inserire secret in `.env`, nei workspace, negli URL o nell'output di `docker compose config`. Inserire solo le chiavi necessarie al profilo scelto. Il bundle viene montato in sola lettura nel container come `/run/secrets/thothii.secrets`; il parser rifiuta duplicati, chiavi sconosciute, valori vuoti, symlink e permessi host troppo aperti. Non inserire secret in `.env`, nei workspace, negli URL o nell'output Compose renderizzato.
Una catena CA PEM **non può essere inserita nel bundle**: contiene whitespace e viene rifiutata dal parser. Se un endpoint usa una CA privata, conservarla nel secret manager/host e aggiungere un override Compose revisionato che monti il file in `/run/secrets/ca-chain.pem` e imposti `THT_SSL_CA` (o il parametro dell'adapter). Il clone base non crea quel mount: questa è una limitazione intenzionale da considerare in fase di deployment. Una catena CA PEM **non può essere inserita nel bundle**: contiene whitespace e viene rifiutata dal parser. Se un endpoint usa una CA privata, conservarla nel secret manager/host e aggiungere un override Compose revisionato che monti il file in `/run/secrets/ca-chain.pem` e imposti `THT_SSL_CA` (o il parametro dell'adapter). Il clone base non crea quel mount: questa è una limitazione intenzionale da considerare in fase di deployment.
@@ -62,7 +65,8 @@ DWH/vector/embedding remoti restano endpoint del file locale o server. Per il so
sviluppo pgvector, aggiungere `-f deploy/compose.local-vector.yaml --profile local-vector` al sviluppo pgvector, aggiungere `-f deploy/compose.local-vector.yaml --profile local-vector` al
comando base. Per il preprocessing aggiungere anche comando base. Per il preprocessing aggiungere anche
`-f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml --profile preprocess`, `-f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml --profile preprocess`,
poi usare `docker compose run --rm preprocess-evidence` oppure `preprocess-dwh` con gli stessi argomenti. poi ripetere l'intero comando base con l'azione `run --rm preprocess-evidence` oppure
`run --rm preprocess-dwh`.
## Workspace, adapter e Evidence ## Workspace, adapter e Evidence
@@ -124,8 +128,10 @@ THOTH_PUBLIC_EXPOSURE=false
Riempire nel bundle le chiavi DWH/vector/model necessarie e avviare: Riempire nel bundle le chiavi DWH/vector/model necessarie e avviare:
```sh ```sh
docker compose up --build -d docker compose --env-file deploy/env/local.env \
docker compose exec core /opt/venv/bin/tht doctor --json -f compose.yaml -f deploy/compose.local.yaml up --build -d
docker compose --env-file deploy/env/local.env \
-f compose.yaml -f deploy/compose.local.yaml exec core /opt/venv/bin/tht doctor --json
``` ```
Se si abilita l'overlay production, il proxy autenticato TLS deve essere l'unico listener pubblico Se si abilita l'overlay production, il proxy autenticato TLS deve essere l'unico listener pubblico
@@ -159,7 +165,10 @@ THT_VECTOR_READER_PASSWORD=<valore casuale>
THT_VECTOR_WRITER_PASSWORD=<valore casuale> THT_VECTOR_WRITER_PASSWORD=<valore casuale>
``` ```
Poi eseguire il comando standard `docker compose up --build -d`. Il primo avvio esegue reconciliation dei ruoli e migrazione pgvector. Per preprocessing, impostare il preset indicato sopra e usare `docker compose run --rm preprocess-evidence`/`preprocess-dwh`. Poi eseguire il comando standard base+locale mostrato sopra. Il primo avvio esegue
reconciliation dei ruoli e migrazione pgvector. Per preprocessing, impostare il preset indicato
sopra e usare l'azione `run --rm preprocess-evidence` o `run --rm preprocess-dwh` con tutti
gli stessi file e profili.
## 3. PC Windows locale ## 3. PC Windows locale
@@ -175,8 +184,8 @@ THT_DOCS_ROOT=/data/source/evidence
Creare `deploy/secrets/thothii.secrets` con un editor locale protetto (ACL leggibile solo dall'utente Docker) e le stesse quattro chiavi pgvector del profilo Mac. Non usare `ConvertFrom-SecureString`: il bundle deve contenere il valore in chiaro per il servizio, con accesso limitato al file. Da PowerShell, dalla radice del clone, eseguire: Creare `deploy/secrets/thothii.secrets` con un editor locale protetto (ACL leggibile solo dall'utente Docker) e le stesse quattro chiavi pgvector del profilo Mac. Non usare `ConvertFrom-SecureString`: il bundle deve contenere il valore in chiaro per il servizio, con accesso limitato al file. Da PowerShell, dalla radice del clone, eseguire:
```powershell ```powershell
docker compose up --build -d docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up --build -d
docker compose ps docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml ps
``` ```
Se un bind mount viene rifiutato, aggiungere la cartella del repository a Docker Desktop → Settings → Resources → File Sharing. Per Ollama eseguito in WSL2 usare l'indirizzo raggiungibile dalla rete Docker invece di assumere `localhost`. Se un bind mount viene rifiutato, aggiungere la cartella del repository a Docker Desktop → Settings → Resources → File Sharing. Per Ollama eseguito in WSL2 usare l'indirizzo raggiungibile dalla rete Docker invece di assumere `localhost`.
@@ -187,13 +196,25 @@ Usare il profilo server e consentire dal firewall solo le destinazioni necessari
```dotenv ```dotenv
# Avvio: docker compose --env-file deploy/env/server.env \ # Avvio: docker compose --env-file deploy/env/server.env \
# -f compose.yaml -f deploy/compose.server.yaml up --build -d # -f compose.yaml -f deploy/compose.server.yaml \
# -f deploy/compose.session-server.yaml.example up --build -d
THT_DB_NAME=warehouse THT_DB_NAME=warehouse
THT_DWH_REST_URL=https://dwh.example.test THT_DWH_REST_URL=https://dwh.example.test
THT_VEC_REST_URL=https://vectors.example.test THT_VEC_REST_URL=https://vectors.example.test
THT_OLLAMA_URL=https://embeddings.example.test THT_OLLAMA_URL=https://embeddings.example.test
``` ```
Avviare e verificare con il profilo server completo:
```sh
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example up --build -d
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example exec core /opt/venv/bin/tht doctor --json
```
Il DWH e il vector DB possono essere REST/HTTP oppure adapter diretti (`postgres_direct`, `pgvector_direct`) se il server ha connettività TCP. Le Evidence possono essere: Il DWH e il vector DB possono essere REST/HTTP oppure adapter diretti (`postgres_direct`, `pgvector_direct`) se il server ha connettività TCP. Le Evidence possono essere:
- filesystem NFS/SMB montato sul server e presentato come root read-only; - filesystem NFS/SMB montato sul server e presentato come root read-only;
@@ -208,8 +229,8 @@ Le variabili `THT_*_SECRET_FILE` e i file `dwh-api-key`, `vector-reader-api-key`
1. creare `deploy/secrets/thothii.secrets` mode `0600`; 1. creare `deploy/secrets/thothii.secrets` mode `0600`;
2. copiare ogni valore nel nome chiave corrispondente (`THT_DWH_API_KEY`, `THT_VEC_API_KEY`, `THT_VEC_WRITE_API_KEY`, `THT_MODEL_API_KEY` o `THT_VECTOR_*_PASSWORD`), senza virgolette né newline; 2. copiare ogni valore nel nome chiave corrispondente (`THT_DWH_API_KEY`, `THT_VEC_API_KEY`, `THT_VEC_WRITE_API_KEY`, `THT_MODEL_API_KEY` o `THT_VECTOR_*_PASSWORD`), senza virgolette né newline;
3. rimuovere dal `.env` le variabili `_SECRET_FILE` e impostare `THT_SECRETS_FILE` al percorso del bundle (il default relativo è già corretto); 3. rimuovere dal `.env` le variabili `_SECRET_FILE` e impostare `THT_SECRETS_FILE` al percorso assoluto del bundle;
4. eseguire `docker compose config --quiet` e poi `docker compose up --build -d`; 4. renderizzare e avviare con il comando base+locale completo e il suo `--env-file`;
5. solo dopo la verifica, cancellare i vecchi file separati. 5. solo dopo la verifica, cancellare i vecchi file separati.
Una CA PEM resta un'eccezione esterna come descritto sopra. Provider Pi con credenziali composte (Bedrock, Azure OpenAI Responses, Cloudflare Workers AI/Gateway) restano rifiutati finché non viene implementato un adapter dedicato. Una CA PEM resta un'eccezione esterna come descritto sopra. Provider Pi con credenziali composte (Bedrock, Azure OpenAI Responses, Cloudflare Workers AI/Gateway) restano rifiutati finché non viene implementato un adapter dedicato.
@@ -217,9 +238,12 @@ Una CA PEM resta un'eccezione esterna come descritto sopra. Provider Pi con cred
## Controlli post-installazione ## Controlli post-installazione
```sh ```sh
docker compose config --quiet docker compose --env-file deploy/env/local.env \
docker compose ps -f compose.yaml -f deploy/compose.local.yaml config --quiet
docker compose exec core /opt/venv/bin/tht doctor --json docker compose --env-file deploy/env/local.env \
-f compose.yaml -f deploy/compose.local.yaml ps
docker compose --env-file deploy/env/local.env \
-f compose.yaml -f deploy/compose.local.yaml exec core /opt/venv/bin/tht doctor --json
./scripts/docker-smoke.sh ./scripts/docker-smoke.sh
``` ```
+1 -1
View File
@@ -1,5 +1,5 @@
# Workspace ThothII — Profilo A (server co-locato). DWH + vector BOTH direct, no REST. # Workspace ThothII — Profilo A (server co-locato). DWH + vector BOTH direct, no REST.
# Segreti SOLO in env (compose env_file: deploy/thothii.env). Path assoluti interni al container (/data). # Secret contents live only in protected mounted files; paths below are container-absolute.
language: it language: it
database: database:
+2 -2
View File
@@ -3,11 +3,11 @@ $ErrorActionPreference = "Continue"
$repositoryRoot = Split-Path -Parent $PSScriptRoot $repositoryRoot = Split-Path -Parent $PSScriptRoot
Set-Location $repositoryRoot Set-Location $repositoryRoot
& docker compose -f compose.yaml -f deploy/compose.local.yaml build --pull & docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml build --pull
$exitCode = $LASTEXITCODE $exitCode = $LASTEXITCODE
if ($exitCode -eq 0) { if ($exitCode -eq 0) {
Write-Output "Next: docker compose -f compose.yaml -f deploy/compose.local.yaml up -d" Write-Output "Next: docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up -d"
} }
exit $exitCode exit $exitCode
+3 -2
View File
@@ -3,11 +3,12 @@ set -u
cd "$(dirname "$0")/.." cd "$(dirname "$0")/.."
docker compose -f compose.yaml -f deploy/compose.local.yaml build --pull docker compose --env-file deploy/env/local.env \
-f compose.yaml -f deploy/compose.local.yaml build --pull
status=$? status=$?
if [[ "$status" -eq 0 ]]; then if [[ "$status" -eq 0 ]]; then
printf '%s\n' 'Next: docker compose -f compose.yaml -f deploy/compose.local.yaml up -d' printf '%s\n' 'Next: docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up -d'
fi fi
exit "$status" exit "$status"
+8 -8
View File
@@ -1,21 +1,21 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Smoke test del deploy standalone ThothII (core + frontend). # Smoke test del deploy standalone ThothII (core + frontend).
# Usa docker-compose.dev.yml (rete propria, porte host). # Usa docker-compose.dev.yml (rete propria, porte host).
# Prereq: deploy/thothii.env popolato, endpoint esterni configurati e profilo Pi locale. # Prereq: deploy/env/local.env popolato, endpoint esterni e file Pi/segreti configurati.
set -euo pipefail set -euo pipefail
cd "$(dirname "$0")/.." cd "$(dirname "$0")/.."
DC="docker compose -f docker-compose.dev.yml" DC=(docker compose --env-file deploy/env/local.env -f docker-compose.dev.yml)
WS="/app/harness/workspaces/local.yaml" WS="/app/harness/workspaces/local.yaml"
echo "== ThothII standalone smoke ==" echo "== ThothII standalone smoke =="
$DC config --quiet "${DC[@]}" config --quiet
echo "== Build ==" echo "== Build =="
$DC build "${DC[@]}" build
echo "== Up (wait health) ==" echo "== Up (wait health) =="
$DC up -d --wait "${DC[@]}" up -d --wait
echo "== Core health ==" echo "== Core health =="
curl -fsS http://localhost:8787/health && echo curl -fsS http://localhost:8787/health && echo
@@ -24,12 +24,12 @@ echo "== Frontend serve =="
curl -fsSI http://localhost:8090/ | head -1 curl -fsSI http://localhost:8090/ | head -1
echo "== Wiring check (config + DWH ping; -c è per-command) ==" echo "== Wiring check (config + DWH ping; -c è per-command) =="
$DC exec -T core tht config check -c "$WS" || \ "${DC[@]}" exec -T core tht config check -c "$WS" || \
echo "(config check non verde: verificare .env/ruoli DB)" echo "(config check non verde: verificare .env/ruoli DB)"
$DC exec -T core tht db ping -c "$WS" || \ "${DC[@]}" exec -T core tht db ping -c "$WS" || \
echo "(db ping non verde: verificare ruolo thoth_dwh_reader + rete)" echo "(db ping non verde: verificare ruolo thoth_dwh_reader + rete)"
echo "== Down ==" echo "== Down =="
$DC down "${DC[@]}" down
echo "OK: smoke standalone passato." echo "OK: smoke standalone passato."
@@ -101,7 +101,13 @@ done < <(
{ {
printf '%s\n' '# Generated by scripts/generate-connector-secrets-override.sh; keep this file untracked.' printf '%s\n' '# Generated by scripts/generate-connector-secrets-override.sh; keep this file untracked.'
printf '%s\n' 'services:' ' core:' ' secrets:' printf '%s\n' \
'services:' \
' core:' \
' env_file:' \
' - path: ${THT_WORKSPACE_BINDINGS_ENV_FILE:?set THT_WORKSPACE_BINDINGS_ENV_FILE}' \
' required: true' \
' secrets:'
for ((index = 0; index < ${#names[@]}; index += 1)); do for ((index = 0; index < ${#names[@]}; index += 1)); do
printf ' - source: connector_secret_%d\n' "$((index + 1))" printf ' - source: connector_secret_%d\n' "$((index + 1))"
printf ' target: %s\n' "${targets[index]}" printf ' target: %s\n' "${targets[index]}"
+8 -1
View File
@@ -39,6 +39,13 @@ write_bundle() {
} }
write_bundle write_bundle
export THT_SECRETS_FILE="$bundle" export THT_SECRETS_FILE="$bundle"
printf '%s\n' '{}' >"$secret_dir/pi-auth.json"
chmod 0600 "$secret_dir/pi-auth.json"
operator_env="$secret_dir/operator.env"
printf '%s\n' \
'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \
"PI_AUTH_FILE=$secret_dir/pi-auth.json" \
"THT_SECRETS_FILE=$bundle" >"$operator_env"
# The rotation helper has an old/new file interface; these are test-only # The rotation helper has an old/new file interface; these are test-only
# scratch files and are never mounted into a Compose service. # scratch files and are never mounted into a Compose service.
printf '%s' "$bootstrap_password" >"$secret_dir/bootstrap" printf '%s' "$bootstrap_password" >"$secret_dir/bootstrap"
@@ -47,7 +54,7 @@ export THT_VECTOR_BOOTSTRAP_USER=thoth_bootstrap_smoke
export THOTH_SMOKE_OWNER="$smoke_owner" export THOTH_SMOKE_OWNER="$smoke_owner"
compose() { compose() {
docker compose -f compose.yaml -f deploy/compose.local-vector.yaml \ docker compose --env-file "$operator_env" -f compose.yaml -f deploy/compose.local-vector.yaml \
--project-name "$smoke_project" --profile local-vector "$@" --project-name "$smoke_project" --profile local-vector "$@"
} }
+8 -1
View File
@@ -58,6 +58,13 @@ bundle="$tmp/thothii.secrets"
chmod 0600 "$bundle" chmod 0600 "$bundle"
export THT_SECRETS_FILE="$bundle" export THT_SECRETS_FILE="$bundle"
export THT_OLLAMA_URL=http://mock-embeddings:8081 export THT_OLLAMA_URL=http://mock-embeddings:8081
printf '%s\n' '{}' >"$tmp/pi-auth.json"
chmod 0600 "$tmp/pi-auth.json"
printf '%s\n' \
'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \
"PI_AUTH_FILE=$tmp/pi-auth.json" \
"THT_SECRETS_FILE=$bundle" \
'THT_OLLAMA_URL=http://mock-embeddings:8081' >"$tmp/operator.env"
cat >"$tmp/smoke.yaml" <<YAML cat >"$tmp/smoke.yaml" <<YAML
services: services:
@@ -83,7 +90,7 @@ services:
mock-embeddings: {condition: service_started} mock-embeddings: {condition: service_started}
YAML YAML
compose="docker compose -f compose.yaml -f deploy/compose.local-vector.yaml -f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml -f $tmp/smoke.yaml --project-name $project --profile local-vector --profile preprocess" compose="docker compose --env-file $tmp/operator.env -f compose.yaml -f deploy/compose.local-vector.yaml -f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml -f $tmp/smoke.yaml --project-name $project --profile local-vector --profile preprocess"
$compose build preprocess-evidence $compose build preprocess-evidence
if [ "${PREPROCESS_SMOKE_INJECT_FAILURE:-0}" = "1" ]; then if [ "${PREPROCESS_SMOKE_INJECT_FAILURE:-0}" = "1" ]; then
sh -c 'exit 97' sh -c 'exit 97'
+99
View File
@@ -0,0 +1,99 @@
#!/usr/bin/env bash
# Active installation manuals must drive the canonical two-service base+profile stack.
set -euo pipefail
root="$(cd "$(dirname "$0")/.." && pwd -P)"
tmp="$(mktemp -d "${TMPDIR%/}/thoth-canonical-install.XXXXXX")"
trap 'rm -rf "$tmp"' EXIT HUP INT TERM
for retired_example in \
"$root/docs/install/examples/local-compose.workspace-registry.yaml" \
"$root/docs/install/examples/server-compose.workspace-registry.yaml" \
"$root/docs/install/examples/git-ssh.workspace-registry.yaml" \
"$root/docs/install/examples/git-https.workspace-registry.yaml"; do
if [[ -e "$retired_example" ]]; then
echo "superseded one-service install example remains active: ${retired_example#"$root/"}" >&2
exit 1
fi
done
printf '%s\n' '{}' >"$tmp/pi-auth.json"
printf '%s\n' 'THT_MODEL_API_KEY=fixture-model-api-key' >"$tmp/thothii.secrets"
chmod 0600 "$tmp/pi-auth.json" "$tmp/thothii.secrets"
mkdir -p "$tmp/data" "$tmp/pi-state" "$tmp/workspace-registry"
printf '%s\n' 'fixture-session-password' >"$tmp/session-runtime-password"
printf '%s\n' 'fixture-session-migrator-password' >"$tmp/session-migrator-password"
printf '%s\n' 'fixture-session-ca' >"$tmp/session-ca.pem"
cp "$root/deploy/workspaces/server-sessions.yaml.example" "$tmp/server-sessions.yaml"
chmod 0600 "$tmp/session-runtime-password" "$tmp/session-migrator-password" "$tmp/session-ca.pem"
for profile in local server; do
env_file="$tmp/$profile.env"
{
printf '%s\n' \
'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \
"PI_AUTH_FILE=$tmp/pi-auth.json" \
"THT_SECRETS_FILE=$tmp/thothii.secrets"
if [[ "$profile" == server ]]; then
printf '%s\n' \
"THT_DATA_ROOT=$tmp/data" \
"THT_PI_STATE_ROOT=$tmp/pi-state" \
"THT_WORKSPACE_REGISTRY_ROOT=$tmp/workspace-registry" \
"THT_SERVER_WORKSPACE_CONFIG=$tmp/server-sessions.yaml" \
'THT_SESSION_DB_HOST=sessions.example.invalid' \
'THT_SESSION_DB_NAME=thoth_sessions' \
'THT_SESSION_RUNTIME_USER=thoth_sessions_app' \
'THT_SESSION_MIGRATOR_USER=thoth_sessions_migrate' \
"THT_SESSION_RUNTIME_PASSWORD_SOURCE=$tmp/session-runtime-password" \
"THT_SESSION_MIGRATOR_PASSWORD_SOURCE=$tmp/session-migrator-password" \
"THT_SESSION_CA_SOURCE=$tmp/session-ca.pem"
fi
} >"$env_file"
compose_files=(-f "$root/compose.yaml" -f "$root/deploy/compose.$profile.yaml")
if [[ "$profile" == server ]]; then
compose_files+=(-f "$root/deploy/compose.session-server.yaml.example")
fi
docker compose --env-file "$env_file" "${compose_files[@]}" \
config --format json >"$tmp/$profile.json"
node - "$tmp/$profile.json" "$profile" <<'NODE'
const fs = require("fs");
const [path, profile] = process.argv.slice(2);
const config = JSON.parse(fs.readFileSync(path, "utf8"));
if (Object.keys(config.services).sort().join(",") !== "core,frontend") {
throw new Error(profile + ": install stack must be exactly core,frontend");
}
if (!config.services.core.secrets?.some((secret) => secret.target === "thothii.secrets")) {
throw new Error(profile + ": install stack lacks the runtime secret bundle");
}
if (!config.services.core.volumes?.some(
(mount) => mount.target === "/home/thoth/.pi/agent/auth.json" && mount.read_only,
)) {
throw new Error(profile + ": install stack lacks the read-only Pi auth file");
}
if ((config.services.frontend.secrets || []).length !== 0) {
throw new Error(profile + ": frontend received runtime secrets");
}
if (profile === "server" && config.services.core.environment?.THT_SESSION_STORAGE !== "postgres") {
throw new Error("server: public startup must include the PostgreSQL session override");
}
if (JSON.stringify(config).includes("fixture-model-api-key")) {
throw new Error(profile + ": rendered Compose leaked a secret value");
}
NODE
done
for profile in local server; do
manual="$root/docs/install/$profile-workspace-registry.md"
grep -Fq -- '--env-file "$THT_OPERATOR_ENV"' "$manual" \
&& grep -Fq -- "-f \"\$THT_SOURCE_ROOT/compose.yaml\" -f \"\$THT_SOURCE_ROOT/deploy/compose.$profile.yaml\"" "$manual" || {
echo "$profile manual lacks the canonical base+profile command" >&2
exit 1
}
if rg -q 'local-compose\.workspace-registry|server-compose\.workspace-registry' "$manual"; then
echo "$profile manual still references a superseded standalone Compose example" >&2
exit 1
fi
done
echo "canonical install Compose contract passed."
+74
View File
@@ -0,0 +1,74 @@
#!/usr/bin/env bash
# Fresh Compose flow: mounted Pi policy/auth must produce a selectable, credential-ready provider.
set -euo pipefail
root="$(cd "$(dirname "$0")/.." && pwd -P)"
tmp="$(mktemp -d "${TMPDIR%/}/thoth-provider-readiness.XXXXXX")"
project="thothii-provider-readiness-$$"
compose=(
docker compose --project-name "$project" --env-file "$tmp/local.env"
-f "$root/compose.yaml" -f "$root/deploy/compose.local.yaml"
)
cleanup() {
"${compose[@]}" down --volumes --remove-orphans >/dev/null 2>&1 || true
rm -rf "$tmp"
}
trap cleanup EXIT HUP INT TERM
printf '%s\n' '{"zai":{"type":"api_key","key":"fixture-native-auth-key"}}' >"$tmp/pi-auth.json"
printf '%s\n' 'THT_MODEL_API_KEY=fixture-model-api-key' >"$tmp/thothii.secrets"
chmod 0600 "$tmp/pi-auth.json" "$tmp/thothii.secrets"
printf '%s\n' \
'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \
"PI_AUTH_FILE=$tmp/pi-auth.json" \
"THT_SECRETS_FILE=$tmp/thothii.secrets" \
'THOTH_CORE_HTTP_PORT=0' \
'THOTH_HTTP_PORT=0' \
>"$tmp/local.env"
"${compose[@]}" up --detach --wait --wait-timeout 90 --build core
core_id="$("${compose[@]}" ps -q core)"
core_address="$("${compose[@]}" port core 8787 | head -n 1)"
"${compose[@]}" exec -T core sh -ceu '
test -r /home/thoth/.pi/agent/auth.json
test -r /home/thoth/.pi/agent/models.json
test -r /home/thoth/.pi/agent/settings.json
test -r /run/secrets/thothii.secrets
'
curl --fail --silent --show-error "http://$core_address/models" >"$tmp/models.json"
node - "$tmp/models.json" <<'NODE'
const fs = require("fs");
const body = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
if (!body.models?.some((model) => model.provider === "zai" && model.id === "glm-5.2")) {
throw new Error("fresh Compose did not expose the mounted Pi-enabled model");
}
NODE
curl --fail --silent --show-error -X PUT \
-H 'content-type: application/json' \
--data '{"provider":"zai","model":"glm-5.2","reasoning":"low"}' \
"http://$core_address/pi-management/config" >"$tmp/configured.json"
curl --fail --silent --show-error \
"http://$core_address/pi-management/status" >"$tmp/status.json"
node - "$tmp/status.json" <<'NODE'
const fs = require("fs");
const body = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
if (!body.ready || body.credentials !== "present") {
throw new Error("mounted Pi provider is not credential-ready");
}
if (body.config?.provider !== "zai" || body.config?.model !== "glm-5.2") {
throw new Error("Pi provider configuration was not persisted");
}
NODE
inspect="$(docker inspect "$core_id")"
for secret in fixture-native-auth-key fixture-model-api-key; do
if grep -Fq "$secret" <<<"$inspect"; then
echo "container inspection leaked $secret" >&2
exit 1
fi
done
echo "Compose provider-readiness contract passed."
+15 -6
View File
@@ -48,7 +48,13 @@ for (const mount of (core.volumes || []).filter((item) => item.target?.startsWit
const secretTargets = (core.secrets || []).map((secret) => secret.target).sort(); const secretTargets = (core.secrets || []).map((secret) => secret.target).sort();
const expectedSecrets = expectedSecretTargets ? expectedSecretTargets.split(",").filter(Boolean).sort() : []; const expectedSecrets = expectedSecretTargets ? expectedSecretTargets.split(",").filter(Boolean).sort() : [];
if (secretTargets.join(",") !== expectedSecrets.join(",")) { if (secretTargets.join(",") !== expectedSecrets.join(",")) {
throw new Error(`${name}: connector targets do not match generated THT_WS_*_FILE bindings`); throw new Error(`${name}: Docker secret targets do not match the deployment contract`);
}
if (core.environment?.THT_SECRETS_FILE !== "/run/secrets/thothii.secrets") {
throw new Error(`${name}: core does not use the canonical /run/secrets bundle path`);
}
if ((config.services.frontend?.secrets || []).length !== 0) {
throw new Error(`${name}: frontend must not receive runtime secrets`);
} }
if (name === "ssh") { if (name === "ssh") {
@@ -62,7 +68,7 @@ if (name === "https" && core.environment?.GIT_CONFIG_VALUE_1 !== "/run/secrets/w
} }
const rendered = JSON.stringify(config); const rendered = JSON.stringify(config);
for (const secret of ["fixture-ssh-private-key", "fixture-ssh-known-hosts", "fixture-https-credentials", "fixture-https-ca", "fixture-dwh-password", "fixture-vector-api-key"]) { for (const secret of ["fixture-model-api-key", "fixture-ssh-private-key", "fixture-ssh-known-hosts", "fixture-https-credentials", "fixture-https-ca", "fixture-dwh-password", "fixture-vector-api-key"]) {
if (rendered.includes(secret)) throw new Error(`${name}: rendered Compose leaked fixture secret value`); if (rendered.includes(secret)) throw new Error(`${name}: rendered Compose leaked fixture secret value`);
} }
NODE NODE
@@ -97,6 +103,7 @@ assert_unsafe_source_rejected() {
} }
write_secret "$fixture_root/pi-auth.json" 'fixture-pi-auth' write_secret "$fixture_root/pi-auth.json" 'fixture-pi-auth'
write_secret "$fixture_root/thothii.secrets" 'THT_MODEL_API_KEY=fixture-model-api-key'
write_secret "$fixture_root/ssh-private-key" 'fixture-ssh-private-key' write_secret "$fixture_root/ssh-private-key" 'fixture-ssh-private-key'
write_secret "$fixture_root/ssh-known-hosts" 'fixture-ssh-known-hosts' write_secret "$fixture_root/ssh-known-hosts" 'fixture-ssh-known-hosts'
write_secret "$fixture_root/https-credentials" 'fixture-https-credentials' write_secret "$fixture_root/https-credentials" 'fixture-https-credentials'
@@ -107,6 +114,8 @@ write_secret "$fixture_root/vector-api-key" 'fixture-vector-api-key'
printf '%s\n' \ printf '%s\n' \
'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \ 'THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git' \
"PI_AUTH_FILE=$fixture_root/pi-auth.json" \ "PI_AUTH_FILE=$fixture_root/pi-auth.json" \
"THT_SECRETS_FILE=$fixture_root/thothii.secrets" \
"THT_WORKSPACE_BINDINGS_ENV_FILE=$fixture_root/workspace-bindings.env" \
"THT_WORKSPACE_GIT_SSH_KEY_FILE=$fixture_root/ssh-private-key" \ "THT_WORKSPACE_GIT_SSH_KEY_FILE=$fixture_root/ssh-private-key" \
"THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$fixture_root/ssh-known-hosts" \ "THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$fixture_root/ssh-known-hosts" \
"THT_WORKSPACE_GIT_CREDENTIALS_FILE=$fixture_root/https-credentials" \ "THT_WORKSPACE_GIT_CREDENTIALS_FILE=$fixture_root/https-credentials" \
@@ -127,13 +136,13 @@ connector_override="$fixture_root/compose.connector-secrets.local.yaml"
--output "$connector_override" --output "$connector_override"
render base render base
assert_render_contract base '' '' assert_render_contract base '' 'thothii.secrets'
render ssh -f "$root/deploy/compose.git-ssh.yaml" render ssh -f "$root/deploy/compose.git-ssh.yaml"
assert_render_contract ssh '/run/secrets/workspace-registry-git-known-hosts,/run/secrets/workspace-registry-git-ssh-key' '' assert_render_contract ssh '/run/secrets/workspace-registry-git-known-hosts,/run/secrets/workspace-registry-git-ssh-key' 'thothii.secrets'
render https -f "$root/deploy/compose.git-https.yaml" render https -f "$root/deploy/compose.git-https.yaml"
assert_render_contract https '/run/secrets/workspace-registry-git-ca,/run/secrets/workspace-registry-git-credentials' '' assert_render_contract https '/run/secrets/workspace-registry-git-ca,/run/secrets/workspace-registry-git-credentials' 'thothii.secrets'
render connector -f "$connector_override" render connector -f "$connector_override"
assert_render_contract connector '' 'north-star-research-dwh-password,north-star-research-vector-api-key' assert_render_contract connector '' 'north-star-research-dwh-password,north-star-research-vector-api-key,thothii.secrets'
assert_missing_source_rejected THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE assert_missing_source_rejected THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE
assert_missing_source_rejected THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE assert_missing_source_rejected THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE
+8 -1
View File
@@ -9,10 +9,13 @@ expected_pi_version=$(sed -n 's/^ARG PI_VERSION=//p' docker/core.Dockerfile)
trap 'docker compose --project-name "$project" -f compose.yaml -f deploy/compose.local.yaml down --volumes --remove-orphans >/dev/null 2>&1 || true; rm -rf "$tmp"' EXIT HUP INT TERM trap 'docker compose --project-name "$project" -f compose.yaml -f deploy/compose.local.yaml down --volumes --remove-orphans >/dev/null 2>&1 || true; rm -rf "$tmp"' EXIT HUP INT TERM
test -n "$expected_pi_version" test -n "$expected_pi_version"
printf '{}\n' >"$tmp/pi-auth.json" printf '%s\n' '{"zai":{"type":"api_key","key":"fixture-native-auth-key"}}' >"$tmp/pi-auth.json"
chmod 0600 "$tmp/pi-auth.json" chmod 0600 "$tmp/pi-auth.json"
printf '%s\n' 'THT_MODEL_API_KEY=fixture-model-api-key' >"$tmp/thothii.secrets"
chmod 0600 "$tmp/thothii.secrets"
export PI_AUTH_FILE="$tmp/pi-auth.json" export PI_AUTH_FILE="$tmp/pi-auth.json"
export THT_SECRETS_FILE="$tmp/thothii.secrets"
export THT_WORKSPACE_GIT_REMOTE="https://git.example.invalid/thothii/workspaces.git" export THT_WORKSPACE_GIT_REMOTE="https://git.example.invalid/thothii/workspaces.git"
# Let Docker assign loopback ports so this isolated contract test never collides with an operator stack. # Let Docker assign loopback ports so this isolated contract test never collides with an operator stack.
export THOTH_CORE_HTTP_PORT=0 export THOTH_CORE_HTTP_PORT=0
@@ -62,6 +65,10 @@ docker compose --project-name "$project" -f compose.yaml -f deploy/compose.local
test "$(pi --version)" = "$PI_VERSION" test "$(pi --version)" = "$PI_VERSION"
command -v pi >/dev/null command -v pi >/dev/null
test ! -e /var/run/docker.sock test ! -e /var/run/docker.sock
test -r /home/thoth/.pi/agent/auth.json
test -r /home/thoth/.pi/agent/models.json
test -r /home/thoth/.pi/agent/settings.json
test -r /run/secrets/thothii.secrets
touch /data/.task5-writable touch /data/.task5-writable
rm /data/.task5-writable rm /data/.task5-writable
if find /app /home /data -xdev \( -iname "*chirone*" -o -iname "*omics*portal*" \) -print -quit | grep -q .; then if find /app /home /data -xdev \( -iname "*chirone*" -o -iname "*omics*portal*" \) -print -quit | grep -q .; then
+66
View File
@@ -0,0 +1,66 @@
#!/usr/bin/env bash
# Prevent active operator-facing startup examples from bypassing required env/profile inputs.
set -euo pipefail
root="$(cd "$(dirname "$0")/.." && pwd -P)"
cd "$root"
targets=(
README.md
.env.example
docker-compose.dev.yml
deploy/env.example
deploy/secrets/README.md
docs/install
docs/index.md
docs/installazione-docker-4-contesti.md
scripts/build-local.sh
scripts/build-local.ps1
scripts/docker-smoke.sh
scripts/local-vector-smoke.sh
scripts/preprocess-smoke.sh
scripts/vector-rotate-bootstrap-password.sh
)
existing=()
for target in "${targets[@]}"; do
[[ ! -e "$target" ]] || existing+=("$target")
done
set +e
matches="$(rg -n \
'docker compose (up|build|run|config|ps|exec|-f)|DC="docker compose -f|compose="docker compose -f' \
"${existing[@]}" 2>&1)"
rg_status=$?
set -e
case "$rg_status" in
0)
echo "active deployment command omits --env-file before its action/overrides:" >&2
printf '%s\n' "$matches" >&2
exit 1
;;
1) ;;
*)
printf '%s\n' "$matches" >&2
exit "$rg_status"
;;
esac
for document in README.md docs/installazione-docker-4-contesti.md; do
grep -Fq -- '-f deploy/compose.session-server.yaml.example' "$document" || {
echo "$document omits the required public-server session override" >&2
exit 1
}
done
for required in \
THT_SERVER_WORKSPACE_CONFIG \
THT_SESSION_RUNTIME_PASSWORD_SOURCE \
THT_SESSION_MIGRATOR_PASSWORD_SOURCE \
THT_SESSION_CA_SOURCE; do
grep -q "^$required=" deploy/env/server.env.example || {
echo "server env example omits $required" >&2
exit 1
}
done
echo "deployment command contract passed."
@@ -8,6 +8,7 @@ trap cleanup EXIT HUP INT TERM
export THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git export THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git
export PI_AUTH_FILE=/dev/null export PI_AUTH_FILE=/dev/null
export THT_SECRETS_FILE=/dev/null
unset THT_VECTOR_BOOTSTRAP_PASSWORD_SECRET_FILE THT_VECTOR_MIGRATOR_PASSWORD_SECRET_FILE unset THT_VECTOR_BOOTSTRAP_PASSWORD_SECRET_FILE THT_VECTOR_MIGRATOR_PASSWORD_SECRET_FILE
unset THT_VECTOR_READER_PASSWORD_SECRET_FILE THT_VECTOR_WRITER_PASSWORD_SECRET_FILE unset THT_VECTOR_READER_PASSWORD_SECRET_FILE THT_VECTOR_WRITER_PASSWORD_SECRET_FILE
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env bash
# Model providers are external endpoints reached through the ordinary application network.
set -euo pipefail
root="$(cd "$(dirname "$0")/.." && pwd -P)"
tmp="$(mktemp -d "${TMPDIR%/}/thoth-external-llm.XXXXXX")"
trap 'rm -rf "$tmp"' EXIT HUP INT TERM
printf '%s\n' '{}' >"$tmp/pi-auth.json"
printf '%s\n' 'THT_MODEL_API_KEY=fixture-model-api-key' >"$tmp/thothii.secrets"
chmod 0600 "$tmp/pi-auth.json" "$tmp/thothii.secrets"
THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git \
PI_AUTH_FILE="$tmp/pi-auth.json" \
THT_SECRETS_FILE="$tmp/thothii.secrets" \
THT_LLM_URL=https://llm.example.invalid/v1 \
docker compose -f "$root/compose.yaml" config --format json >"$tmp/config.json"
node - "$tmp/config.json" <<'NODE'
const fs = require("fs");
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
if (Object.keys(config.services).sort().join(",") !== "core,frontend") {
throw new Error("external LLM deployment must retain the mandatory two-service stack");
}
if (Object.keys(config.networks || {}).join(",") !== "thothii") {
throw new Error("external LLM endpoint must not require a provider-owned Docker network");
}
if (config.services.core.environment?.THT_LLM_URL !== "https://llm.example.invalid/v1") {
throw new Error("core did not receive the generic external LLM endpoint");
}
const joins = (service, network) => Array.isArray(service.networks)
? service.networks.includes(network)
: Object.hasOwn(service.networks || {}, network);
if (!joins(config.services.core, "thothii") || !joins(config.services.frontend, "thothii")) {
throw new Error("frontend and core must share only the application network");
}
NODE
echo "external LLM network contract passed."
+84
View File
@@ -0,0 +1,84 @@
#!/usr/bin/env bash
# Regression coverage for coupling-scan categories, exact exclusions, and scanner failures.
set -euo pipefail
root="$(cd "$(dirname "$0")/.." && pwd -P)"
fixture="$(mktemp -d "${TMPDIR%/}/thoth-coupling-scope.XXXXXX")"
trap 'rm -rf "$fixture"' EXIT HUP INT TERM
new_fixture() {
rm -rf "$fixture/repository"
mkdir -p \
"$fixture/repository/deploy/env" \
"$fixture/repository/deploy/workspaces" \
"$fixture/repository/docker/smoke" \
"$fixture/repository/docs/install" \
"$fixture/repository/docs/superpowers/plans" \
"$fixture/repository/frontend" \
"$fixture/repository/scripts"
printf '%s\n' 'services: {}' >"$fixture/repository/compose.yaml"
printf '%s\n' '# generic runtime image' >"$fixture/repository/docker/core.Dockerfile"
printf '%s\n' '# generic smoke' >"$fixture/repository/docker/smoke/core-smoke.sh"
printf '%s\n' '# generic install' >"$fixture/repository/docs/install/local.md"
printf '%s\n' 'THT_LLM_URL=https://llm.example.invalid' >"$fixture/repository/deploy/env/local.env.example"
printf '%s\n' '# generic launcher' >"$fixture/repository/scripts/run-stack.sh"
printf '%s\n' '// generic frontend configuration' >"$fixture/repository/frontend/vite.config.ts"
# These are the three intentionally allowed categories from the Task 10 boundary.
printf '%s\n' 'historical omics_portal and Chirone record' \
>"$fixture/repository/docs/superpowers/plans/legacy.md"
printf '%s\n' 'id: psd' >"$fixture/repository/deploy/workspaces/psd.yaml.example"
printf '%s\n' '# migrate PSD sessions from /home/chirone' \
>"$fixture/repository/docker/session-migrate.sh"
}
assert_clean() {
"$root/scripts/test-no-deployment-coupling.sh" --root "$fixture/repository" >/dev/null
}
assert_detected() {
local relative_path="$1" content="$2" output status
new_fixture
mkdir -p "$(dirname "$fixture/repository/$relative_path")"
printf '%s\n' "$content" >"$fixture/repository/$relative_path"
set +e
output="$("$root/scripts/test-no-deployment-coupling.sh" --root "$fixture/repository" 2>&1)"
status=$?
set -e
if [[ $status -ne 1 ]] || ! grep -Fq "$relative_path" <<<"$output"; then
echo "coupling scan missed $relative_path" >&2
printf '%s\n' "$output" >&2
exit 1
fi
}
new_fixture
assert_clean
assert_detected compose.yaml 'services: # Chirone runtime coupling'
assert_detected docker/smoke/core-smoke.sh 'test -d /home/chirone'
assert_detected docs/install/local.md 'Install the PSD deployment profile.'
assert_detected deploy/env/local.env.example 'NETWORK=omics_portal'
assert_detected scripts/run-stack.sh 'exec datamart-builder'
assert_detected frontend/vite.config.ts 'const base = "/omics_portal";'
assert_detected scripts/test-qwen-network-config.sh 'require localllm_default'
assert_detected scripts/test-provider-network.sh 'if (!config.networks?.localllm_default?.external) exit 1'
assert_detected deploy/compose.psd-local.yaml 'services: {}'
new_fixture
mkdir -p "$fixture/bin"
printf '%s\n' '#!/bin/sh' 'exit 2' >"$fixture/bin/rg"
chmod +x "$fixture/bin/rg"
set +e
PATH="$fixture/bin:$PATH" "$root/scripts/test-no-deployment-coupling.sh" \
--root "$fixture/repository" >"$fixture/rg.out" 2>"$fixture/rg.err"
status=$?
set -e
if [[ $status -ne 2 ]]; then
echo "coupling scan masked an rg failure (status $status)" >&2
cat "$fixture/rg.out" "$fixture/rg.err" >&2
exit 1
fi
echo "no-coupling scope regression tests passed."
+107 -51
View File
@@ -1,69 +1,125 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Category-based guard for active build, runtime, install, and launch coupling.
set -euo pipefail set -euo pipefail
cd "$(dirname "$0")/.." script_root="$(cd "$(dirname "$0")/.." && pwd -P)"
scan_root="$script_root"
if [[ "${1:-}" == --root ]]; then
[[ $# -eq 2 ]] || { echo "usage: $0 [--root PATH]" >&2; exit 2; }
scan_root="$2"
elif [[ $# -ne 0 ]]; then
echo "usage: $0 [--root PATH]" >&2
exit 2
fi
[[ -d "$scan_root" ]] || { echo "coupling scan root is not a directory: $scan_root" >&2; exit 2; }
cd "$scan_root"
content_targets=( runtime_files=()
.dockerignore install_files=()
compose.yaml operator_files=()
docker-compose.dev.yml contract_test_files=()
deploy add_file() {
docker local array_name="$1" file="$2"
frontend/vite.config.ts [[ ! -f "$file" ]] || eval "$array_name+=(\"\$file\")"
README.md }
docs/install
docs/installazione-docker-4-contesti.md
.env.example
scripts/run-stack.sh
scripts/docker-smoke.sh
)
matches=$( for file in .dockerignore compose.yaml docker-compose.dev.yml frontend/vite.config.ts; do
rg -n -i \ add_file runtime_files "$file"
-g '!deploy/workspaces/**' \ done
-g '!docker/session-migrate.sh' \ if [[ -d deploy ]]; then
-g '!docker/cutover-legacy-sessions.sh' \ while IFS= read -r -d '' file; do runtime_files+=("${file#./}"); done < <(
-g '!docker/smoke/**' \ find deploy -type f ! -path 'deploy/workspaces/*' -print0
'omics_portal|chirone|localllm_default|datamart-builder|compose\.production\.yaml|compose\.psd-local\.yaml' \ )
"${content_targets[@]}" || true fi
) if [[ -d docker ]]; then
while IFS= read -r -d '' file; do
case "$file" in
docker/session-migrate.sh|docker/cutover-legacy-sessions.sh) continue ;;
esac
runtime_files+=("${file#./}")
done < <(find docker -type f -print0)
fi
runtime_psd_matches=$( for file in README.md .env.example docs/installazione-docker-4-contesti.md; do
rg -n -i \ add_file install_files "$file"
-g '!deploy/workspaces/**' \ done
-g '!docker/session-migrate.sh' \ if [[ -d docs/install ]]; then
-g '!docker/cutover-legacy-sessions.sh' \ while IFS= read -r -d '' file; do install_files+=("${file#./}"); done < <(
-g '!docker/smoke/**' \ find docs/install -type f -print0
'\bpsd\b' \ )
.dockerignore compose.yaml docker-compose.dev.yml deploy docker frontend/vite.config.ts \ fi
.env.example scripts/run-stack.sh scripts/docker-smoke.sh || true
) if [[ -d scripts ]]; then
while IFS= read -r -d '' file; do
case "${file#scripts/}" in
test-no-deployment-coupling.sh|test-no-deployment-coupling-scope.sh) continue ;;
test-*.sh)
contract_test_files+=("${file#./}")
continue
;;
verify-*.sh) continue ;;
esac
operator_files+=("${file#./}")
done < <(find scripts -maxdepth 1 -type f -print0)
fi
offenders=() offenders=()
for superseded_file in \ scan_category() {
local label="$1" pattern="$2"; shift 2
local output rg_status
(($#)) || return 0
set +e
output="$(rg -n -i --with-filename -- "$pattern" "$@" 2>&1)"
rg_status=$?
set -e
case "$rg_status" in
0)
while IFS= read -r match; do offenders+=("$label: $match"); done <<<"$output"
;;
1) ;;
*)
echo "coupling scan failed in $label (rg status $rg_status)" >&2
printf '%s\n' "$output" >&2
exit "$rg_status"
;;
esac
}
for forbidden_file in \
deploy/compose.production.yaml \ deploy/compose.production.yaml \
deploy/compose.psd-local.yaml.example \ deploy/compose.psd-local.yaml.example \
deploy/compose.psd-local.yaml \ deploy/compose.psd-local.yaml \
scripts/bootstrap-local-psd-docker-config.sh \ scripts/bootstrap-local-psd-docker-config.sh \
harness/tests/test_psd_local_compose_contract.py scripts/test-qwen-network-config.sh \
do harness/tests/test_psd_local_compose_contract.py; do
[[ ! -e "$superseded_file" ]] || offenders+=("$superseded_file (forbidden active deployment filename)") [[ ! -e "$forbidden_file" ]] \
|| offenders+=("active filename: $forbidden_file (superseded deployment contract)")
done done
if [[ -n "$matches" ]]; then forbidden='omics_portal|chirone|localllm_default|datamart-builder|compose\.production\.yaml|compose\.psd-local\.yaml|\bpsd\b'
while IFS= read -r match; do scan_category runtime "$forbidden" "${runtime_files[@]}"
offenders+=("$match") scan_category install "$forbidden" "${install_files[@]}"
done <<<"$matches" scan_category operator "$forbidden" "${operator_files[@]}"
fi # Contract tests legitimately quote forbidden names in negative assertions. Scan their positive
# deployment wiring constructs instead, so a provider-owned network or retired overlay cannot be
# required under a different test filename.
positive_contract='networks(\?|\.)?\.?localllm_default|services(\?|\.)?\.?core(\?|\.)?\.?networks(\?|\.)?\.?localllm_default|docker compose[^\n]*(compose\.psd-local|compose\.production)|THT_PSD_[A-Z0-9_]*='
scan_category contract-test "$positive_contract" "${contract_test_files[@]}"
if [[ -n "$runtime_psd_matches" ]]; then if [[ -f scripts/run-stack.sh ]]; then
while IFS= read -r match; do set +e
offenders+=("$match") host_pi="$(rg -n 'command -v pi|PI_BIN="pi"|PI_BIN=pi' scripts/run-stack.sh 2>&1)"
done <<<"$runtime_psd_matches" host_pi_status=$?
fi set -e
case "$host_pi_status" in
if rg -n 'command -v pi|PI_BIN="pi"|PI_BIN=pi' scripts/run-stack.sh >/dev/null; then 0) offenders+=("operator: $host_pi") ;;
offenders+=("scripts/run-stack.sh (requires a host Pi binary)") 1) ;;
*)
echo "coupling scan failed in host-Pi contract (rg status $host_pi_status)" >&2
printf '%s\n' "$host_pi" >&2
exit "$host_pi_status"
;;
esac
fi fi
if ((${#offenders[@]})); then if ((${#offenders[@]})); then
+32 -1
View File
@@ -8,13 +8,42 @@ trap 'rm -rf "$tmp"' EXIT HUP INT TERM
auth_file="$tmp/auth.json" auth_file="$tmp/auth.json"
printf '%s\n' '{}' >"$auth_file" printf '%s\n' '{}' >"$auth_file"
chmod 0600 "$auth_file" chmod 0600 "$auth_file"
secrets_file="$tmp/thothii.secrets"
printf '%s\n' 'THT_MODEL_API_KEY=fixture-model-api-key' >"$secrets_file"
chmod 0600 "$secrets_file"
rendered=$(THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git \ rendered=$(THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git \
PI_AUTH_FILE="$auth_file" docker compose config) PI_AUTH_FILE="$auth_file" THT_SECRETS_FILE="$secrets_file" docker compose config)
printf '%s\n' "$rendered" | grep -q "source: $auth_file" printf '%s\n' "$rendered" | grep -q "source: $auth_file"
printf '%s\n' "$rendered" | grep -q 'target: /home/thoth/.pi/agent/auth.json' printf '%s\n' "$rendered" | grep -q 'target: /home/thoth/.pi/agent/auth.json'
printf '%s\n' "$rendered" | grep -A4 'target: /home/thoth/.pi/agent/auth.json' \ printf '%s\n' "$rendered" | grep -A4 'target: /home/thoth/.pi/agent/auth.json' \
| grep -q 'read_only: true' | grep -q 'read_only: true'
for target in \
/home/thoth/.pi/agent/models.json \
/home/thoth/.pi/agent/settings.json; do
printf '%s\n' "$rendered" | grep -q "target: $target"
printf '%s\n' "$rendered" | grep -A4 "target: $target" | grep -q 'read_only: true'
done
printf '%s\n' "$rendered" | grep -q "file: $secrets_file"
printf '%s\n' "$rendered" | grep -q 'target: thothii.secrets'
if grep -Fq 'fixture-model-api-key' <<<"$rendered"; then
echo "rendered base Compose leaked the model key" >&2
exit 1
fi
dev_rendered=$(THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git \
PI_AUTH_FILE="$auth_file" THT_SECRETS_FILE="$secrets_file" \
docker compose --env-file deploy/env/local.env.example -f docker-compose.dev.yml config)
printf '%s\n' "$dev_rendered" | grep -q "source: $auth_file"
printf '%s\n' "$dev_rendered" | grep -q 'target: /home/thoth/.pi/agent/auth.json'
printf '%s\n' "$dev_rendered" | grep -q 'target: /home/thoth/.pi/agent/models.json'
printf '%s\n' "$dev_rendered" | grep -q 'target: /home/thoth/.pi/agent/settings.json'
printf '%s\n' "$dev_rendered" | grep -q "file: $secrets_file"
printf '%s\n' "$dev_rendered" | grep -q 'target: thothii.secrets'
if grep -Fq 'fixture-model-api-key' <<<"$dev_rendered"; then
echo "rendered development Compose leaked the model key" >&2
exit 1
fi
python3 - <<'PY' python3 - <<'PY'
import json import json
@@ -32,5 +61,7 @@ PY
grep -q '^ARG PI_VERSION=0.80.3$' docker/core.Dockerfile grep -q '^ARG PI_VERSION=0.80.3$' docker/core.Dockerfile
grep -q '^PI_AUTH_FILE=/absolute/path/to/pi-auth.json$' deploy/env/local.env.example grep -q '^PI_AUTH_FILE=/absolute/path/to/pi-auth.json$' deploy/env/local.env.example
grep -q '^PI_AUTH_FILE=/absolute/path/to/pi-auth.json$' deploy/env/server.env.example grep -q '^PI_AUTH_FILE=/absolute/path/to/pi-auth.json$' deploy/env/server.env.example
grep -q '^THT_SECRETS_FILE=/absolute/path/to/thothii.secrets$' deploy/env/local.env.example
grep -q '^THT_SECRETS_FILE=/absolute/path/to/thothii.secrets$' deploy/env/server.env.example
echo "Pi user-auth Compose contract passed." echo "Pi user-auth Compose contract passed."
+6 -1
View File
@@ -4,7 +4,8 @@ set -eu
cd "$(dirname "$0")/.." cd "$(dirname "$0")/.."
tmp_bundle=$(mktemp) tmp_bundle=$(mktemp)
trap 'rm -f "$tmp_bundle"' EXIT HUP INT TERM tmp_auth=$(mktemp)
trap 'rm -f "$tmp_bundle" "$tmp_auth"' EXIT HUP INT TERM
cat >"$tmp_bundle" <<'EOF' cat >"$tmp_bundle" <<'EOF'
THT_VECTOR_BOOTSTRAP_PASSWORD=test-bootstrap THT_VECTOR_BOOTSTRAP_PASSWORD=test-bootstrap
THT_VECTOR_MIGRATOR_PASSWORD=test-migrator THT_VECTOR_MIGRATOR_PASSWORD=test-migrator
@@ -12,7 +13,11 @@ THT_VECTOR_READER_PASSWORD=test-reader
THT_VECTOR_WRITER_PASSWORD=test-writer THT_VECTOR_WRITER_PASSWORD=test-writer
EOF EOF
chmod 0600 "$tmp_bundle" chmod 0600 "$tmp_bundle"
printf '%s\n' '{}' >"$tmp_auth"
chmod 0600 "$tmp_auth"
export THT_SECRETS_FILE="$tmp_bundle" export THT_SECRETS_FILE="$tmp_bundle"
export PI_AUTH_FILE="$tmp_auth"
export THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git
local_files="-f compose.yaml -f deploy/compose.local-vector.yaml -f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml" local_files="-f compose.yaml -f deploy/compose.local-vector.yaml -f deploy/compose.preprocess.yaml -f deploy/compose.preprocess-local-vector.yaml"
local_json=$(docker compose $local_files --profile local-vector --profile preprocess config --format json) local_json=$(docker compose $local_files --profile local-vector --profile preprocess config --format json)
-24
View File
@@ -1,24 +0,0 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT HUP INT TERM
mkdir -p "$tmp/deploy"
cp compose.yaml "$tmp/compose.yaml"
: >"$tmp/deploy/thothii.env"
docker compose --project-directory "$tmp" -f "$tmp/compose.yaml" config --format json >"$tmp/config.json"
node - "$tmp/config.json" <<'NODE'
const fs = require("fs");
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
if (!config.networks?.localllm_default?.external) {
throw new Error("localllm_default must be an external network");
}
if (!config.services?.core?.networks?.localllm_default) {
throw new Error("core must join localllm_default");
}
if (config.services?.frontend?.networks?.localllm_default) {
throw new Error("frontend must not join the model network");
}
NODE
+44 -2
View File
@@ -11,8 +11,12 @@ render_profile() {
local env_file=$2 local env_file=$2
local compose_file=$3 local compose_file=$3
local rendered="$tmp/$profile.json" local rendered="$tmp/$profile.json"
local -a files=(-f compose.yaml -f "$compose_file")
if [[ "$profile" == server ]]; then
files+=(-f deploy/compose.session-server.yaml.example)
fi
docker compose --env-file "$env_file" -f compose.yaml -f "$compose_file" \ docker compose --env-file "$env_file" "${files[@]}" \
config --format json >"$rendered" config --format json >"$rendered"
node - "$rendered" "$profile" <<'NODE' node - "$rendered" "$profile" <<'NODE'
@@ -38,6 +42,28 @@ const piAuthMounts = (config.services.core.volumes || []).filter(
if (piAuthMounts.length !== 1 || piAuthMounts[0].type !== "bind" || !piAuthMounts[0].read_only) { if (piAuthMounts.length !== 1 || piAuthMounts[0].type !== "bind" || !piAuthMounts[0].read_only) {
throw new Error("Pi auth must be one read-only file bind"); throw new Error("Pi auth must be one read-only file bind");
} }
if (config.services.core.environment?.THT_SECRETS_FILE !== "/run/secrets/thothii.secrets") {
throw new Error("core must read the canonical runtime secret bundle from /run/secrets");
}
const runtimeSecrets = config.services.core.secrets || [];
const bundleSecrets = runtimeSecrets.filter(
(secret) => secret.source === "thothii_secrets" && secret.target === "thothii.secrets",
);
if (bundleSecrets.length !== 1) {
throw new Error("core must receive exactly one canonical runtime secret bundle");
}
if (profile === "local" && runtimeSecrets.length !== 1) {
throw new Error("local core must receive only the canonical runtime secret bundle");
}
if (profile === "server") {
const targets = new Set(runtimeSecrets.map((secret) => secret.target));
for (const target of ["session_runtime_password", "session_ca.pem"]) {
if (!targets.has(target)) throw new Error("server core lacks " + target);
}
}
if ((config.services.frontend.secrets || []).length !== 0) {
throw new Error("frontend must not receive runtime secrets");
}
const ports = Object.fromEntries( const ports = Object.fromEntries(
Object.entries(config.services).map(([name, service]) => [name, service.ports || []]), Object.entries(config.services).map(([name, service]) => [name, service.ports || []]),
@@ -60,9 +86,12 @@ assert_remote_required() {
local env_file=$1 local env_file=$1
local compose_file=$2 local compose_file=$2
local without_remote="$tmp/without-remote.env" local without_remote="$tmp/without-remote.env"
local -a files=(-f compose.yaml -f "$compose_file")
[[ "$compose_file" != deploy/compose.server.yaml ]] \
|| files+=(-f deploy/compose.session-server.yaml.example)
grep -v '^THT_WORKSPACE_GIT_REMOTE=' "$env_file" >"$without_remote" grep -v '^THT_WORKSPACE_GIT_REMOTE=' "$env_file" >"$without_remote"
if env -u THT_WORKSPACE_GIT_REMOTE docker compose --env-file "$without_remote" -f compose.yaml -f "$compose_file" \ if env -u THT_WORKSPACE_GIT_REMOTE docker compose --env-file "$without_remote" "${files[@]}" \
config --format json >"$tmp/missing-remote.out" 2>"$tmp/missing-remote.err"; then config --format json >"$tmp/missing-remote.out" 2>"$tmp/missing-remote.err"; then
echo "Compose must require THT_WORKSPACE_GIT_REMOTE" >&2 echo "Compose must require THT_WORKSPACE_GIT_REMOTE" >&2
exit 1 exit 1
@@ -72,6 +101,7 @@ assert_remote_required() {
THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git \ THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git \
PI_AUTH_FILE=/dev/null \ PI_AUTH_FILE=/dev/null \
THT_SECRETS_FILE=/dev/null \
docker compose -f compose.yaml config --format json >"$tmp/base.json" docker compose -f compose.yaml config --format json >"$tmp/base.json"
node - "$tmp/base.json" <<'NODE' node - "$tmp/base.json" <<'NODE'
const fs = require("fs"); const fs = require("fs");
@@ -98,6 +128,18 @@ const piAuthMounts = (config.services.core.volumes || []).filter(
if (piAuthMounts.length !== 1 || piAuthMounts[0].type !== "bind" || !piAuthMounts[0].read_only) { if (piAuthMounts.length !== 1 || piAuthMounts[0].type !== "bind" || !piAuthMounts[0].read_only) {
throw new Error("Pi auth must be one read-only file bind"); throw new Error("Pi auth must be one read-only file bind");
} }
if (config.services.core.environment?.THT_SECRETS_FILE !== "/run/secrets/thothii.secrets") {
throw new Error("core must read the canonical runtime secret bundle from /run/secrets");
}
const runtimeSecrets = config.services.core.secrets || [];
if (runtimeSecrets.length !== 1
|| runtimeSecrets[0].source !== "thothii_secrets"
|| runtimeSecrets[0].target !== "thothii.secrets") {
throw new Error("core must receive exactly the canonical runtime secret bundle");
}
if ((config.services.frontend.secrets || []).length !== 0) {
throw new Error("frontend must not receive runtime secrets");
}
NODE NODE
render_profile local deploy/env/local.env.example deploy/compose.local.yaml render_profile local deploy/env/local.env.example deploy/compose.local.yaml
+7 -16
View File
@@ -9,20 +9,11 @@ trap 'rm -f "$output"' EXIT HUP INT TERM
"$root/scripts/verify-workspace-install-docs.sh" --fixtures-only >"$output" "$root/scripts/verify-workspace-install-docs.sh" --fixtures-only >"$output"
for fixture in \ for fixture in \
"local manual requires generated connector override and Compose preflight" \ "local manual canonical base+override references" \
"server manual requires generated connector override and Compose preflight" \ "server manual canonical base+override references" \
"local documented shell environment fixture" \ "canonical local base+override fixture" \
"server documented shell environment fixture" \ "canonical server base+override fixture" \
"copied local base fixture" \ "relative secret-source fixture rejected"; do
"copied server PostgreSQL/TLS fixture" \
"copied HTTPS Git override fixture" \
"copied SSH Git override fixture" \
"copied connector binding/secret fixture" \
"core process sees connector bindings and secret files" \
"non-path secret-file fixture rejected" \
"literal secret-source fixture rejected" \
"relative secret-source fixture rejected" \
"non-normalized secret-source fixture rejected"; do
grep -Fqx "$fixture passed" "$output" >/dev/null || { grep -Fqx "$fixture passed" "$output" >/dev/null || {
echo "missing fixture verification: $fixture" >&2 echo "missing fixture verification: $fixture" >&2
cat "$output" >&2 cat "$output" >&2
@@ -37,7 +28,7 @@ for manual in \
echo "installation manual does not publish a self-contained THT_SOURCE_ROOT export: $manual" >&2 echo "installation manual does not publish a self-contained THT_SOURCE_ROOT export: $manual" >&2
exit 1 exit 1
} }
grep -Fq 'export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env"' "$manual" || { grep -Fq 'export THT_WORKSPACE_BINDINGS_ENV_FILE=' "$manual" || {
echo "installation manual does not publish a self-contained bindings export: $manual" >&2 echo "installation manual does not publish a self-contained bindings export: $manual" >&2
exit 1 exit 1
} }
@@ -47,7 +38,7 @@ for manual in \
fi fi
done done
if rg -n 'connector-secrets\.workspace-registry|docker compose' \ if rg -n 'local-compose\.workspace-registry|server-compose\.workspace-registry|connector-secrets\.workspace-registry|docker compose' \
"$root/docs/install/local-workspace-registry.md" \ "$root/docs/install/local-workspace-registry.md" \
"$root/docs/install/server-workspace-registry.md"; then "$root/docs/install/server-workspace-registry.md"; then
echo "installation manuals still document a bypassed Compose or copied connector override path" >&2 echo "installation manuals still document a bypassed Compose or copied connector override path" >&2
+2 -2
View File
@@ -29,7 +29,7 @@ trap 'rm -f "$replacement"' EXIT HUP INT TERM
cp "$new_secret" "$replacement" cp "$new_secret" "$replacement"
chmod 0600 "$replacement" chmod 0600 "$replacement"
docker compose -f compose.yaml -f deploy/compose.local-vector.yaml \ docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local-vector.yaml \
--project-name "$project" --profile local-vector run --rm --no-deps \ --project-name "$project" --profile local-vector run --rm --no-deps \
--user 0:0 \ --user 0:0 \
--entrypoint /opt/venv/bin/python \ --entrypoint /opt/venv/bin/python \
@@ -43,4 +43,4 @@ mv -f "$replacement" "$old_secret"
trap - EXIT HUP INT TERM trap - EXIT HUP INT TERM
echo "Deployment bootstrap secret atomically replaced only after verified database login." echo "Deployment bootstrap secret atomically replaced only after verified database login."
echo "Re-run: docker compose -f compose.yaml -f deploy/compose.local-vector.yaml --project-name $project --profile local-vector up --wait vector-reconcile vector-migrate core" echo "Re-run: docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local-vector.yaml --project-name $project --profile local-vector up --wait vector-reconcile vector-migrate core"
+186 -306
View File
@@ -1,9 +1,9 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Validate the installation manuals without reading an operator environment or production remote. # Verify canonical local/server installation manuals and their base+override Compose paths.
set -euo pipefail set -euo pipefail
root="$(cd "$(dirname "$0")/.." && pwd -P)" root="$(cd "$(dirname "$0")/.." && pwd -P)"
profile="${1:-}" mode="${1:-}"
trim() { trim() {
local value="$1" local value="$1"
@@ -41,266 +41,13 @@ verify_path_variable_values() {
fi fi
fi fi
done <"$source" done <"$source"
return 0
} }
verify_server_public_contract() { verify_manual() {
local server_example="$root/docs/install/examples/server-compose.workspace-registry.yaml" local profile="$1" manual
for expected in \ manual="$root/docs/install/$profile-workspace-registry.md"
'THT_SESSION_STORAGE: postgres' \ local -a headings
'THT_SESSION_RUNTIME_PASSWORD_FILE: /run/secrets/session_runtime_password' \ if [[ "$profile" == local ]]; then
'THT_SESSION_DB_SSLMODE: ${THT_SESSION_DB_SSLMODE:-verify-full}' \
'THT_SESSION_DB_SSLROOTCERT: /run/secrets/session_ca.pem' \
'session_runtime_password:' \
'session_ca:'; do
grep -Fq "$expected" "$server_example" || {
echo "server Compose example lacks required public PostgreSQL/TLS contract: $expected" >&2
return 1
}
done
}
verify_manual_supported_path() {
local profile="$1" manual="$2"
local source_root_export='export THT_SOURCE_ROOT=/absolute/path/to/ThothII'
local bindings_export='export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env"'
local generator='"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env .env --output connector-secrets.local.yaml'
local wrapper='"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env'
grep -Fq "$source_root_export" "$manual" || {
echo "$profile manual does not export THT_SOURCE_ROOT for its shell commands" >&2
return 1
}
grep -Fq "$bindings_export" "$manual" || {
echo "$profile manual does not export THT_WORKSPACE_BINDINGS_ENV_FILE for its shell commands" >&2
return 1
}
grep -Fq "$generator" "$manual" || {
echo "$profile manual does not document the connector override generator" >&2
return 1
}
grep -Fq "$wrapper" "$manual" || {
echo "$profile manual does not document the Compose preflight wrapper" >&2
return 1
}
if grep -Eq 'connector-secrets\.workspace-registry|docker compose' "$manual"; then
echo "$profile manual documents a bypassed Compose or copied connector override path" >&2
return 1
fi
echo "$profile manual requires generated connector override and Compose preflight passed"
}
compose_fixture() {
local name="$1" directory="$2"; shift 2
(
cd "$directory"
"$root/scripts/compose-with-preflight.sh" --env-file .env "$@" config --quiet
)
echo "$name passed"
}
prepare_binding_fixture() {
local directory="$1"
printf '%s\n' \
'THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct' \
'THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password' \
'THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api' \
'THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key' \
>"$directory/workspace-bindings.env"
printf 'THT_WORKSPACE_BINDINGS_ENV_FILE=%s\n' "$directory/workspace-bindings.env" >>"$directory/.env"
}
verify_connector_fixture() {
local directory="$1" rendered project connector_override
project="thoth-install-connector-fixture-$$"
connector_override="$directory/connector-secrets.local.yaml"
"$root/scripts/generate-connector-secrets-override.sh" \
--bindings-env "$directory/workspace-bindings.env" --operator-env "$directory/.env" \
--output "$connector_override" >/dev/null
rendered="$(
cd "$directory"
"$root/scripts/compose-with-preflight.sh" --env-file .env \
-f compose.workspace-registry.yaml -f connector-secrets.local.yaml config
)"
for expected in \
'THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT: postgres_direct' \
'THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE: /run/secrets/north-star-research-dwh-password' \
'THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE: /run/secrets/north-star-research-vector-api-key' \
'target: north-star-research-dwh-password' \
'target: north-star-research-vector-api-key'; do
grep -Fq "$expected" <<<"$rendered" || {
echo "connector fixture does not give core required binding or secret target: $expected" >&2
return 1
}
done
echo "copied connector binding/secret fixture passed"
if ! (
cd "$directory"
"$root/scripts/compose-with-preflight.sh" --project-name "$project" --env-file .env \
-f compose.workspace-registry.yaml -f connector-secrets.local.yaml run --rm --no-deps --build --entrypoint sh core -c '
test "$THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT" = postgres_direct
test "$THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE" = /run/secrets/north-star-research-dwh-password
test "$THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE" = /run/secrets/north-star-research-vector-api-key
test -f "$THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE"
test -f "$THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE"
'
); then
(
cd "$directory"
"$root/scripts/compose-with-preflight.sh" --project-name "$project" --env-file .env \
-f compose.workspace-registry.yaml -f connector-secrets.local.yaml down --volumes --remove-orphans
) || true
return 1
fi
(
cd "$directory"
"$root/scripts/compose-with-preflight.sh" --project-name "$project" --env-file .env \
-f compose.workspace-registry.yaml -f connector-secrets.local.yaml down --volumes --remove-orphans
)
echo "core process sees connector bindings and secret files passed"
}
verify_documented_operator_path() {
local profile="$1" directory="$2" documented_source_root="$3" connector_override
connector_override="$directory/connector-secrets.local.yaml"
(
cd "$directory"
unset THT_SOURCE_ROOT THT_WORKSPACE_BINDINGS_ENV_FILE
export THT_SOURCE_ROOT="$documented_source_root"
export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env"
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" \
--bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env .env \
--output connector-secrets.local.yaml >/dev/null
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \
-f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml \
-f connector-secrets.local.yaml config --quiet
)
echo "$profile documented shell environment fixture passed"
}
verify_copied_operator_fixtures() {
local fixture_root local_dir server_dir https_dir ssh_dir connector_dir
fixture_root="$(mktemp -d "${TMPDIR%/}/thoth-install-fixtures.XXXXXX")"
trap 'rm -rf "$fixture_root"' RETURN
local_dir="$fixture_root/local"; server_dir="$fixture_root/server"
https_dir="$fixture_root/https"; ssh_dir="$fixture_root/ssh"; connector_dir="$fixture_root/connector"
mkdir -p "$local_dir" "$server_dir" "$https_dir" "$ssh_dir" "$connector_dir"
cp "$root/docs/install/examples/local-compose.workspace-registry.yaml" "$local_dir/compose.workspace-registry.yaml"
printf 'THT_SOURCE_ROOT=%s\n' "$root" >"$local_dir/.env"
prepare_binding_fixture "$local_dir"
compose_fixture "copied local base fixture" "$local_dir" -f compose.workspace-registry.yaml
cp "$root/docs/install/examples/server-compose.workspace-registry.yaml" "$server_dir/compose.workspace-registry.yaml"
cp "$root/deploy/workspaces/server-sessions.yaml.example" "$server_dir/server-sessions.yaml"
: >"$server_dir/session-runtime-password"; : >"$server_dir/session-ca.pem"
printf '%s\n' \
"THT_SOURCE_ROOT=$root" \
"THT_SERVER_WORKSPACE_CONFIG=$server_dir/server-sessions.yaml" \
'THT_SESSION_DB_HOST=sessions.example.invalid' \
'THT_SESSION_DB_NAME=thoth_sessions' \
'THT_SESSION_RUNTIME_USER=thoth_sessions_app' \
"THT_SESSION_RUNTIME_PASSWORD_SOURCE=$server_dir/session-runtime-password" \
"THT_SESSION_CA_SOURCE=$server_dir/session-ca.pem" >"$server_dir/.env"
prepare_binding_fixture "$server_dir"
compose_fixture "copied server PostgreSQL/TLS fixture" "$server_dir" -f compose.workspace-registry.yaml
cp "$root/docs/install/examples/local-compose.workspace-registry.yaml" "$https_dir/compose.workspace-registry.yaml"
cp "$root/docs/install/examples/git-https.workspace-registry.yaml" "$https_dir/git-https.yaml"
: >"$https_dir/git-credentials"; : >"$https_dir/git-ca.pem"
printf '%s\n' \
"THT_SOURCE_ROOT=$root" \
"THT_WORKSPACE_GIT_CREDENTIALS_FILE=$https_dir/git-credentials" \
"THT_WORKSPACE_GIT_CA_FILE=$https_dir/git-ca.pem" >"$https_dir/.env"
prepare_binding_fixture "$https_dir"
compose_fixture "copied HTTPS Git override fixture" "$https_dir" -f compose.workspace-registry.yaml -f git-https.yaml
cp "$root/docs/install/examples/local-compose.workspace-registry.yaml" "$ssh_dir/compose.workspace-registry.yaml"
cp "$root/docs/install/examples/git-ssh.workspace-registry.yaml" "$ssh_dir/git-ssh.yaml"
: >"$ssh_dir/git-ssh-key"; : >"$ssh_dir/git-known-hosts"
printf '%s\n' \
"THT_SOURCE_ROOT=$root" \
"THT_WORKSPACE_GIT_SSH_KEY_FILE=$ssh_dir/git-ssh-key" \
"THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$ssh_dir/git-known-hosts" >"$ssh_dir/.env"
prepare_binding_fixture "$ssh_dir"
compose_fixture "copied SSH Git override fixture" "$ssh_dir" -f compose.workspace-registry.yaml -f git-ssh.yaml
: >"$server_dir/git-ssh-key"; : >"$server_dir/git-known-hosts"
: >"$server_dir/dwh-password"; : >"$server_dir/vector-api-key"
printf '%s\n' \
"THT_WORKSPACE_GIT_SSH_KEY_FILE=$server_dir/git-ssh-key" \
"THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$server_dir/git-known-hosts" \
"THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$server_dir/dwh-password" \
"THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=$server_dir/vector-api-key" >>"$server_dir/.env"
cp "$root/docs/install/examples/git-ssh.workspace-registry.yaml" "$server_dir/git-ssh.workspace-registry.yaml"
: >"$ssh_dir/dwh-password"; : >"$ssh_dir/vector-api-key"
printf '%s\n' \
"THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$ssh_dir/dwh-password" \
"THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=$ssh_dir/vector-api-key" >>"$ssh_dir/.env"
cp "$root/docs/install/examples/git-ssh.workspace-registry.yaml" "$ssh_dir/git-ssh.workspace-registry.yaml"
verify_documented_operator_path local "$ssh_dir" "$root"
verify_documented_operator_path server "$server_dir" "$root"
cp "$root/docs/install/examples/local-compose.workspace-registry.yaml" "$connector_dir/compose.workspace-registry.yaml"
: >"$connector_dir/dwh-password"; : >"$connector_dir/vector-password"
printf '%s\n' \
"THT_SOURCE_ROOT=$root" \
"THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$connector_dir/dwh-password" \
"THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=$connector_dir/vector-password" >"$connector_dir/.env"
prepare_binding_fixture "$connector_dir"
verify_connector_fixture "$connector_dir"
printf 'THT_WS_EXAMPLE_DWH_PASSWORD_FILE=not-a-path\n' >"$fixture_root/non-path-secret.env"
if verify_path_variable_values "$fixture_root/non-path-secret.env" >/dev/null 2>&1; then
echo "non-path secret-file fixture was accepted" >&2
return 1
fi
echo "non-path secret-file fixture rejected passed"
printf 'THT_WS_EXAMPLE_DWH_PASSWORD_SOURCE=literal-value\n' >"$fixture_root/literal-source.env"
if verify_path_variable_values "$fixture_root/literal-source.env" >/dev/null 2>&1; then
echo "literal secret-source fixture was accepted" >&2
return 1
fi
echo "literal secret-source fixture rejected passed"
printf 'THT_WS_EXAMPLE_DWH_PASSWORD_SOURCE=installation-secrets/password\n' >"$fixture_root/relative-source.env"
if verify_path_variable_values "$fixture_root/relative-source.env" >/dev/null 2>&1; then
echo "relative secret-source fixture was accepted" >&2
return 1
fi
echo "relative secret-source fixture rejected passed"
printf 'THT_WS_EXAMPLE_DWH_PASSWORD_SOURCE=/srv/thothii/secrets/../password\n' >"$fixture_root/non-normalized-source.env"
if verify_path_variable_values "$fixture_root/non-normalized-source.env" >/dev/null 2>&1; then
echo "non-normalized secret-source fixture was accepted" >&2
return 1
fi
echo "non-normalized secret-source fixture rejected passed"
}
case "$profile" in
--fixtures-only)
[[ $# -eq 1 ]] || { echo "usage: $0 --fixtures-only" >&2; exit 2; }
verify_manual_supported_path local "$root/docs/install/local-workspace-registry.md"
verify_manual_supported_path server "$root/docs/install/server-workspace-registry.md"
verify_copied_operator_fixtures
exit 0
;;
--profile)
profile="${2:-}"
[[ $# -eq 2 ]] || { echo "usage: $0 --profile {local|server}" >&2; exit 2; }
;;
*)
echo "usage: $0 --profile {local|server}" >&2
exit 2
;;
esac
case "$profile" in
local)
manual="$root/docs/install/local-workspace-registry.md"
example="$root/docs/install/examples/local-compose.workspace-registry.yaml"
headings=( headings=(
"Prerequisites" "Prerequisites"
"Git remote: SSH and HTTPS" "Git remote: SSH and HTTPS"
@@ -310,10 +57,7 @@ case "$profile" in
"Publish, update, backup, outage recovery, and rollback" "Publish, update, backup, outage recovery, and rollback"
"Troubleshooting" "Troubleshooting"
) )
;; else
server)
manual="$root/docs/install/server-workspace-registry.md"
example="$root/docs/install/examples/server-compose.workspace-registry.yaml"
headings=( headings=(
"Service account, storage, and firewall" "Service account, storage, and firewall"
"Gitea and remote Git setup" "Gitea and remote Git setup"
@@ -324,50 +68,186 @@ case "$profile" in
"Pull, publish, upgrade, backup, and recovery" "Pull, publish, upgrade, backup, and recovery"
"Troubleshooting and snapshot rollback" "Troubleshooting and snapshot rollback"
) )
fi
for heading in "${headings[@]}"; do
grep -Fqx "## $heading" "$manual" || {
echo "missing required heading in $profile manual: $heading" >&2
return 1
}
done
for expected in \
'export THT_SOURCE_ROOT=/absolute/path/to/ThothII' \
'--env-file "$THT_OPERATOR_ENV"' \
"-f \"\$THT_SOURCE_ROOT/compose.yaml\" -f \"\$THT_SOURCE_ROOT/deploy/compose.$profile.yaml\"" \
'"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh"'; do
grep -Fq -- "$expected" "$manual" || {
echo "$profile manual lacks canonical operator step: $expected" >&2
return 1
}
done
if rg -n 'local-compose\.workspace-registry|server-compose\.workspace-registry|connector-secrets\.workspace-registry|docker compose' "$manual"; then
echo "$profile manual documents a superseded or bypassed Compose path" >&2
return 1
fi
verify_path_variable_values "$manual"
echo "$profile manual canonical base+override references passed"
}
write_private() {
local path="$1" value="$2"
printf '%s\n' "$value" >"$path"
chmod 0600 "$path"
}
verify_compose_fixtures() {
local fixture connector_override profile rendered
fixture="$(mktemp -d "${TMPDIR%/}/thoth-install-fixtures.XXXXXX")"
trap 'rm -rf "$fixture"' RETURN
mkdir -p "$fixture/data" "$fixture/pi-state" "$fixture/workspace-registry"
write_private "$fixture/pi-auth.json" '{"zai":{"type":"api_key","key":"fixture-native-auth-key"}}'
write_private "$fixture/thothii.secrets" 'THT_MODEL_API_KEY=fixture-model-api-key'
write_private "$fixture/git-ssh-key" 'fixture-git-ssh-key'
write_private "$fixture/git-known-hosts" 'fixture-git-known-hosts'
write_private "$fixture/dwh-password" 'fixture-dwh-password'
write_private "$fixture/vector-api-key" 'fixture-vector-api-key'
write_private "$fixture/session-runtime-password" 'fixture-session-runtime-password'
write_private "$fixture/session-migrator-password" 'fixture-session-migrator-password'
write_private "$fixture/session-ca.pem" 'fixture-session-ca'
cp "$root/deploy/workspaces/server-sessions.yaml.example" "$fixture/server-sessions.yaml"
printf '%s\n' \
'THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct' \
'THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password' \
'THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api' \
'THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key' \
>"$fixture/workspace-bindings.env"
printf '%s\n' \
'THT_WORKSPACE_GIT_REMOTE=ssh://git@git.example.invalid/platform/thoth-workspaces.git' \
"PI_AUTH_FILE=$fixture/pi-auth.json" \
"THT_SECRETS_FILE=$fixture/thothii.secrets" \
"THT_WORKSPACE_BINDINGS_ENV_FILE=$fixture/workspace-bindings.env" \
"THT_WORKSPACE_GIT_SSH_KEY_FILE=$fixture/git-ssh-key" \
"THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=$fixture/git-known-hosts" \
"THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_SOURCE=$fixture/dwh-password" \
"THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_SOURCE=$fixture/vector-api-key" \
"THT_DATA_ROOT=$fixture/data" \
"THT_PI_STATE_ROOT=$fixture/pi-state" \
"THT_WORKSPACE_REGISTRY_ROOT=$fixture/workspace-registry" \
"THT_SERVER_WORKSPACE_CONFIG=$fixture/server-sessions.yaml" \
'THT_SESSION_DB_HOST=sessions.example.invalid' \
'THT_SESSION_DB_NAME=thoth_sessions' \
'THT_SESSION_RUNTIME_USER=thoth_sessions_app' \
'THT_SESSION_MIGRATOR_USER=thoth_sessions_migrate' \
"THT_SESSION_RUNTIME_PASSWORD_SOURCE=$fixture/session-runtime-password" \
"THT_SESSION_MIGRATOR_PASSWORD_SOURCE=$fixture/session-migrator-password" \
"THT_SESSION_CA_SOURCE=$fixture/session-ca.pem" \
>"$fixture/operator.env"
connector_override="$fixture/connector-secrets.local.yaml"
"$root/scripts/generate-connector-secrets-override.sh" \
--bindings-env "$fixture/workspace-bindings.env" \
--operator-env "$fixture/operator.env" \
--output "$connector_override" >/dev/null
for profile in local server; do
rendered="$fixture/$profile.json"
files=(
-f "$root/compose.yaml"
-f "$root/deploy/compose.$profile.yaml"
)
if [[ "$profile" == server ]]; then
files+=(-f "$root/deploy/compose.session-server.yaml.example")
fi
files+=(
-f "$root/deploy/compose.git-ssh.yaml"
-f "$connector_override"
)
"$root/scripts/compose-with-preflight.sh" --env-file "$fixture/operator.env" \
"${files[@]}" config --format json >"$rendered"
node - "$rendered" "$profile" <<'NODE'
const fs = require("fs");
const [path, profile] = process.argv.slice(2);
const config = JSON.parse(fs.readFileSync(path, "utf8"));
if (Object.keys(config.services).sort().join(",") !== "core,frontend") {
throw new Error(profile + ": mandatory stack must be exactly core,frontend");
}
const core = config.services.core;
for (const target of [
"/home/thoth/.pi/agent/auth.json",
"/home/thoth/.pi/agent/models.json",
"/home/thoth/.pi/agent/settings.json",
]) {
if (!(core.volumes || []).some((mount) => mount.target === target && mount.read_only)) {
throw new Error(profile + ": missing read-only Pi mount " + target);
}
}
for (const [name, value] of Object.entries({
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE: "/run/secrets/north-star-research-dwh-password",
THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE: "/run/secrets/north-star-research-vector-api-key",
})) {
if (core.environment?.[name] !== value) throw new Error(profile + ": missing binding " + name);
}
const secretTargets = new Set((core.secrets || []).map((secret) => secret.target));
for (const target of [
"thothii.secrets",
"north-star-research-dwh-password",
"north-star-research-vector-api-key",
]) {
if (!secretTargets.has(target)) throw new Error(profile + ": missing secret target " + target);
}
if (profile === "server") {
for (const target of ["session_runtime_password", "session_ca.pem"]) {
if (!secretTargets.has(target)) throw new Error("server: missing session secret " + target);
}
}
if ((config.services.frontend.secrets || []).length !== 0) {
throw new Error(profile + ": frontend received a runtime secret");
}
const rendered = JSON.stringify(config);
for (const value of [
"fixture-native-auth-key", "fixture-model-api-key", "fixture-git-ssh-key",
"fixture-git-known-hosts", "fixture-dwh-password", "fixture-vector-api-key",
"fixture-session-runtime-password", "fixture-session-migrator-password", "fixture-session-ca",
]) {
if (rendered.includes(value)) throw new Error(profile + ": rendered Compose leaked " + value);
}
NODE
echo "canonical $profile base+override fixture passed"
done
printf 'THT_WS_EXAMPLE_DWH_PASSWORD_SOURCE=relative/secret\n' >"$fixture/unsafe.env"
if verify_path_variable_values "$fixture/unsafe.env" >/dev/null 2>&1; then
echo "relative secret-source fixture was accepted" >&2
return 1
fi
echo "relative secret-source fixture rejected passed"
}
case "$mode" in
--fixtures-only)
[[ $# -eq 1 ]] || { echo "usage: $0 --fixtures-only" >&2; exit 2; }
verify_manual local
verify_manual server
verify_compose_fixtures
;;
--profile)
profile="${2:-}"
[[ $# -eq 2 && "$profile" =~ ^(local|server)$ ]] \
|| { echo "usage: $0 --profile {local|server}" >&2; exit 2; }
verify_manual "$profile"
verify_compose_fixtures
echo "== Run isolated workspace-registry bootstrap and recovery smoke =="
(
cd "$root"
env -u WORKSPACE_GIT_REMOTE ./scripts/workspace-registry-smoke.sh
)
echo "$profile installation documentation verification passed"
;; ;;
*) *)
echo "unknown documentation profile: $profile" >&2 echo "usage: $0 --fixtures-only | --profile {local|server}" >&2
exit 2 exit 2
;; ;;
esac esac
[[ -f "$manual" ]] || { echo "missing $profile installation manual: $manual" >&2; exit 1; }
[[ -f "$example" ]] || { echo "missing $profile Compose example: $example" >&2; exit 1; }
for heading in "${headings[@]}"; do
grep -Fqx "## $heading" "$manual" >/dev/null || {
echo "missing required heading in $profile manual: $heading" >&2
exit 1
}
done
grep -Fq "$(basename "$example")" "$manual" || {
echo "the $profile manual does not reference its Compose example" >&2
exit 1
}
# Values for secret-bearing variables must be paths. These patterns catch common accidental
# credentials while allowing declarative *_FILE bindings and explicitly empty assignments.
if grep -Ein '(^|[[:space:]])(password|api[_-]?key|token|secret)[[:space:]]*[:=][[:space:]]*[^[:space:]#]' \
"$manual" "$example" >/dev/null; then
echo "installation documentation contains a secret literal" >&2
exit 1
fi
verify_path_variable_values "$manual"
verify_path_variable_values "$example"
verify_path_variable_values "$root/docs/install/examples/git-https.workspace-registry.yaml"
verify_path_variable_values "$root/docs/install/examples/git-ssh.workspace-registry.yaml"
verify_path_variable_values "$root/docs/install/examples/workspace-bindings.env.example"
verify_server_public_contract
verify_manual_supported_path "$profile" "$manual"
echo "== Validate copied operator fixtures and documented optional Git transports =="
verify_copied_operator_fixtures
echo "== Run isolated workspace-registry bootstrap and recovery smoke =="
(
cd "$root"
env -u WORKSPACE_GIT_REMOTE ./scripts/workspace-registry-smoke.sh
)
echo "$profile installation documentation verification passed"