mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix: close deployment decoupling review

Browse Source
This commit is contained in:
marcopan
2026-08-05 07:52:32 +02:00
parent 5d037e97c4
commit 09834d5cd4
45 changed files with 1082 additions and 791 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/install/local-workspace-registry.md
+59 -52
View File
@@ -34,14 +34,16 @@ workspaces/<workspace-id>.md
For SSH, use a scoped deploy key, a verified `known_hosts` file, and strict host-key checking. For
HTTPS, use Git Credential Manager or a secret-manager-created credentials file. Mount a private
HTTPS CA as its own file. Do not disable host or certificate verification. The base Compose file
does not mount a Git credential: add exactly one optional `git-ssh.workspace-registry.yaml` or
`git-https.workspace-registry.yaml` override, so unused credential paths are never bind-mounted.
does not mount a Git credential: add exactly one optional `deploy/compose.git-ssh.yaml` or
`deploy/compose.git-https.yaml` override, so unused credential paths are never bind-mounted.
```dotenv
THT_WORKSPACE_GIT_REMOTE=ssh://git@git.example.invalid/platform/thoth-workspaces.git
THT_WORKSPACE_GIT_BRANCH=main
THT_WORKSPACE_INSTALLATION_ID=local-laptop
THT_SOURCE_ROOT=/absolute/path/to/ThothII
PI_AUTH_FILE=/absolute/path/installation-secrets/pi-auth.json
THT_SECRETS_FILE=/absolute/path/installation-secrets/thothii.secrets
THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/installation-secrets/git-ssh-key
THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/installation-secrets/git-known-hosts
THT_WORKSPACE_GIT_CA_FILE=/absolute/path/installation-secrets/git-ca.pem
@@ -69,12 +71,12 @@ state/ # active revision and registry state
locks/ # short-lived publish locks
```
Installation variables are deterministic: `psd-clinical` becomes `PSD_CLINICAL`, and every name
Installation variables are deterministic: `north-star-research` becomes `NORTH_STAR_RESEARCH`, and every name
is `THT_WS_<NAMESPACE>_<ROLE>_<SUFFIX>`. Copy
[the bindings env example](examples/workspace-bindings.env.example) to an untracked operator file
and set its absolute path as `THT_WORKSPACE_BINDINGS_ENV_FILE`. It is loaded only into `core`.
Credentials and certificates use `*_FILE` path variables that must point inside `/run/secrets`.
If declared, `THT_WS_PSD_CLINICAL_VECTOR_WRITER_API_KEY_FILE` is distinct from the vector reader
If declared, `THT_WS_NORTH_STAR_RESEARCH_VECTOR_WRITER_API_KEY_FILE` is distinct from the vector reader
file; a reader credential is never repurposed for writing.
## Direct PostgreSQL, REST, and SSH tunnel bindings
@@ -87,41 +89,41 @@ copy or maintain a workspace-specific Compose override.
```dotenv
# Direct PostgreSQL and pgvector
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=postgres_direct
THT_WS_PSD_CLINICAL_DWH_HOST=dwh.example.invalid
THT_WS_PSD_CLINICAL_DWH_PORT=5432
THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader
THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=pgvector_direct
THT_WS_PSD_CLINICAL_VECTOR_HOST=vector.example.invalid
THT_WS_PSD_CLINICAL_VECTOR_PORT=5432
THT_WS_PSD_CLINICAL_VECTOR_USER=thoth_vector_reader
THT_WS_PSD_CLINICAL_VECTOR_PASSWORD_FILE=/run/secrets/psd-vector-reader
THT_WS_PSD_CLINICAL_EMBEDDING_BASE_URL=https://embeddings.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=postgres_direct
THT_WS_NORTH_STAR_RESEARCH_DWH_HOST=dwh.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_PORT=5432
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=pgvector_direct
THT_WS_NORTH_STAR_RESEARCH_VECTOR_HOST=vector.example.invalid
THT_WS_NORTH_STAR_RESEARCH_VECTOR_PORT=5432
THT_WS_NORTH_STAR_RESEARCH_VECTOR_USER=thoth_vector_reader
THT_WS_NORTH_STAR_RESEARCH_VECTOR_PASSWORD_FILE=/run/secrets/north-star-research-vector-password
THT_WS_NORTH_STAR_RESEARCH_EMBEDDING_BASE_URL=https://embeddings.example.invalid
```
```dotenv
# REST; an API-key file is needed only for a declared bearer/x-api-key diagnostic.
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=rest_api
THT_WS_PSD_CLINICAL_DWH_BASE_URL=https://dwh.example.invalid
THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE=/run/secrets/psd-dwh-api-key
THT_WS_PSD_CLINICAL_VECTOR_TRANSPORT=rest_api
THT_WS_PSD_CLINICAL_VECTOR_BASE_URL=https://vectors.example.invalid
THT_WS_PSD_CLINICAL_VECTOR_API_KEY_FILE=/run/secrets/psd-vector-api-key
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=rest_api
THT_WS_NORTH_STAR_RESEARCH_DWH_BASE_URL=https://dwh.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_API_KEY_FILE=/run/secrets/north-star-research-dwh-api-key
THT_WS_NORTH_STAR_RESEARCH_VECTOR_TRANSPORT=rest_api
THT_WS_NORTH_STAR_RESEARCH_VECTOR_BASE_URL=https://vectors.example.invalid
THT_WS_NORTH_STAR_RESEARCH_VECTOR_API_KEY_FILE=/run/secrets/north-star-research-vector-api-key
```
```dotenv
# SSH tunnel diagnostic only; runtime sessions are fail-closed in this release.
THT_WS_PSD_CLINICAL_DWH_TRANSPORT=ssh_tunnel
THT_WS_PSD_CLINICAL_DWH_USER=thoth_reader
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE=/run/secrets/psd-dwh-reader
THT_WS_PSD_CLINICAL_DWH_SSH_HOST=bastion.example.invalid
THT_WS_PSD_CLINICAL_DWH_SSH_PORT=22
THT_WS_PSD_CLINICAL_DWH_SSH_USER=thoth_tunnel
THT_WS_PSD_CLINICAL_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/psd-dwh-tunnel-key
THT_WS_PSD_CLINICAL_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/psd-dwh-known-hosts
THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_HOST=dwh.internal.example
THT_WS_PSD_CLINICAL_DWH_SSH_TARGET_PORT=5432
THT_WS_NORTH_STAR_RESEARCH_DWH_TRANSPORT=ssh_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_USER=thoth_reader
THT_WS_NORTH_STAR_RESEARCH_DWH_PASSWORD_FILE=/run/secrets/north-star-research-dwh-password
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_HOST=bastion.example.invalid
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PORT=22
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_USER=thoth_tunnel
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_PRIVATE_KEY_FILE=/run/secrets/north-star-research-dwh-tunnel-key
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_KNOWN_HOSTS_FILE=/run/secrets/north-star-research-dwh-known-hosts
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_HOST=dwh.internal.example
THT_WS_NORTH_STAR_RESEARCH_DWH_SSH_TARGET_PORT=5432
```
Repeat the SSH names for `VECTOR` where needed. REST diagnostics reject a private per-request CA
@@ -134,31 +136,36 @@ before creating sessions. Git pull/push over SSH remains fully supported and is
## Bootstrap, first pull, and diagnostics
Copy [the local Compose example](examples/local-compose.workspace-registry.yaml), exactly one
selected [SSH Git override](examples/git-ssh.workspace-registry.yaml) or [HTTPS Git override](examples/git-https.workspace-registry.yaml),
and [the bindings env example](examples/workspace-bindings.env.example) into an untracked operator
directory. Keep `THT_SOURCE_ROOT` and the absolute `THT_WORKSPACE_BINDINGS_ENV_FILE` in its `.env`
for Compose interpolation; this keeps the copied Compose file buildable and confines `THT_WS_*`
values to `core`. A Compose `.env` file is not a shell environment, so do not import it into the
maintenance shell. Instead, explicitly export the two non-secret paths before running the commands.
Create the host secret files named by the selected Git transport and every declared connector
`*_SOURCE`, then generate the connector override and render through the preflight wrapper. The
wrapper is required: it rejects unsafe source paths and a combined SSH+HTTPS Git selection before
Compose runs.
Use the repository's canonical `compose.yaml` plus `deploy/compose.local.yaml`; they always start
the mandatory `frontend` and `core` services. Do not copy or maintain a standalone application
Compose file. Copy [the bindings env example](examples/workspace-bindings.env.example) into an
untracked operator directory and create a protected operator env file from
`deploy/env/local.env.example`. It must contain absolute `PI_AUTH_FILE`,
`THT_SECRETS_FILE`, `THT_WORKSPACE_BINDINGS_ENV_FILE`, and connector `*_SOURCE` paths.
The Pi auth JSON, runtime secret bundle, and each connector credential remain separate protected
host files and are mounted read-only; their contents never enter the operator env or rendered
Compose.
Select exactly one repository Git transport override, `deploy/compose.git-ssh.yaml` or
`deploy/compose.git-https.yaml`. A Compose env file is not a shell environment, so export only the
non-secret paths required by the maintenance commands. Generate the connector override and render
through the preflight wrapper, which rejects unsafe paths and combined SSH+HTTPS selection.
```sh
export THT_SOURCE_ROOT=/absolute/path/to/ThothII
export THT_WORKSPACE_BINDINGS_ENV_FILE="$(pwd -P)/workspace-bindings.env"
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env .env --output connector-secrets.local.yaml
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \
-f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml config --quiet
export THT_OPERATOR_ENV=/absolute/path/to/operator/local.env
export THT_WORKSPACE_BINDINGS_ENV_FILE=/absolute/path/to/operator/workspace-bindings.env
export THT_CONNECTOR_OVERRIDE=/absolute/path/to/operator/connector-secrets.local.yaml
"$THT_SOURCE_ROOT/scripts/generate-connector-secrets-override.sh" --bindings-env "$THT_WORKSPACE_BINDINGS_ENV_FILE" --operator-env "$THT_OPERATOR_ENV" --output "$THT_CONNECTOR_OVERRIDE"
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
-f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \
-f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" config --quiet
```
From the operator directory:
```sh
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file .env \
-f compose.workspace-registry.yaml -f git-ssh.workspace-registry.yaml -f connector-secrets.local.yaml up --build -d
"$THT_SOURCE_ROOT/scripts/compose-with-preflight.sh" --env-file "$THT_OPERATOR_ENV" \
-f "$THT_SOURCE_ROOT/compose.yaml" -f "$THT_SOURCE_ROOT/deploy/compose.local.yaml" \
-f "$THT_SOURCE_ROOT/deploy/compose.git-ssh.yaml" -f "$THT_CONNECTOR_OVERRIDE" up --build -d
curl --fail --silent http://127.0.0.1:8787/health
curl --fail --silent http://127.0.0.1:8787/workspace-registry/status
curl --fail --silent http://127.0.0.1:8787/workspaces
@@ -169,7 +176,7 @@ Use `POST /workspace-registry/pull` to fetch later revisions. Run workspace diag
required bindings are mounted. The optional writer probe uses a distinct writer file and removes
its uniquely named temporary record; ordinary diagnostics are read-only.
To migrate an existing PSD descriptor, create/clone an empty private remote, set the absolute
To migrate an existing legacy descriptor, create/clone an empty private remote, set the absolute
`THT_SOURCE_ROOT`, transform with absolute paths, review the schema-v1 result, explicitly add
vector database/schema and the complete schema-v2 contract, then commit/push. The transformer
never imports `${ENV}` values or secrets.
@@ -177,7 +184,7 @@ never imports `${ENV}` values or secrets.
```sh
THT_SOURCE_ROOT=/absolute/path/to/ThothII
npm --prefix "$THT_SOURCE_ROOT/backend" run build
node "$THT_SOURCE_ROOT/backend/dist/workspaces/migrate-legacy.js" --input /absolute/path/psd.yaml --output /absolute/path/thoth-workspaces
node "$THT_SOURCE_ROOT/backend/dist/workspaces/migrate-legacy.js" --input /absolute/path/legacy.yaml --output /absolute/path/thoth-workspaces
```
## Publish, update, backup, outage recovery, and rollback