From 08ae9e6d90ac3e949c275d9804bd79db81688918 Mon Sep 17 00:00:00 2001 From: mptyl Date: Tue, 4 Aug 2026 13:00:20 +0200 Subject: [PATCH] docs: design unified compose deployment --- ...08-04-unified-compose-deployment-design.md | 196 ++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-04-unified-compose-deployment-design.md diff --git a/docs/superpowers/specs/2026-08-04-unified-compose-deployment-design.md b/docs/superpowers/specs/2026-08-04-unified-compose-deployment-design.md new file mode 100644 index 00000000..f6379319 --- /dev/null +++ b/docs/superpowers/specs/2026-08-04-unified-compose-deployment-design.md @@ -0,0 +1,196 @@ +# Unified Docker Compose Deployment Design + +**Date:** 2026-08-04 + +**Status:** approved in conversation, pending written-spec review + +## Objective + +ThothII ships as one autonomous Docker Compose application that runs unchanged on a developer +PC/Mac or on a server. ThothII has no runtime, build, network, path, proxy, configuration, or +documentation dependency on PSD, Chirone, `omics_portal`, or any other application that happens +to provide databases or vector services. + +## Architecture + +The distribution contains the same `frontend` and `core` images in every environment. A portable +base Compose file defines services, health checks, internal networking, named volumes, registry +storage, and configuration contracts. Small local and server overrides select host exposure, +storage bindings, authentication policy, restart policy, and operational limits without copying +the complete service definitions. + +```text +Browser -> frontend container -> core container + |-> Git workspace registry + |-> configured DWH + |-> configured VectorDB + |-> configured embedding/LLM services + `-> configured session persistence +``` + +The frontend calls the core over the private Compose network through same-origin proxying. The +browser never needs the core port in the server profile. On a workstation, the frontend is bound +to loopback and the core may be bound to loopback for diagnostics. On a server, a generic external +reverse proxy forwards to the frontend host port; that proxy is an operator concern and is not a +ThothII dependency. + +## Compose layout + +- `compose.yaml`: portable base stack and the only complete service definition. +- `deploy/compose.local.yaml`: loopback ports, local named volumes, `AUTH_MODE=none`, workstation + defaults, and local installation identity. +- `deploy/compose.server.yaml`: frontend host binding suitable for a reverse proxy, no public core + binding, server storage policy, configurable authentication, restart policy, and resource limits. +- `deploy/compose.git-ssh.yaml` and `deploy/compose.git-https.yaml`: mutually exclusive Git secret + mounts and trust configuration. +- `deploy/compose.connector-secrets.yaml`: explicit connector secret mounts selected by an + installation. +- `.env.example`: non-secret common variables and documented absolute host paths. +- `deploy/env/local.env.example` and `deploy/env/server.env.example`: profile-specific examples + containing names and safe defaults, never credentials. + +The standard commands are intentionally symmetric: + +```sh +docker compose -f compose.yaml -f deploy/compose.local.yaml build +docker compose -f compose.yaml -f deploy/compose.local.yaml up -d +``` + +```sh +docker compose -f compose.yaml -f deploy/compose.server.yaml build +docker compose -f compose.yaml -f deploy/compose.server.yaml up -d +``` + +Published images remain optional. A server can build from a release checkout or consume pinned +images through an operator override without changing the application architecture. + +## Configuration and secrets + +Portable workspace descriptors remain in the Git workspace registry. Installation-specific +endpoints, ports, usernames, CA paths, and secret-file bindings remain local. Secret contents are +mounted as files and never enter Git, browser drafts, image layers, Compose output, or generated +workspace artifacts. + +The same deterministic naming contract continues to apply: +`THT_WS___` for bindings and `_FILE`/`_SOURCE` for secret paths. The +base Compose accepts generic DWH, vector, embedding, LLM, Git, and session-storage endpoints; none +has a default hostname, path, or network associated with PSD or `omics_portal`. + +## Networking and exposure + +The base stack owns a private Compose network. `frontend` reaches `core` by service name. External +DWH, vector, Git, embedding, and LLM services are reached through operator-configured DNS names or +URLs. `host.docker.internal` may be documented as a workstation option but is not hard-coded as a +product dependency. + +The local profile binds user-facing ports to `127.0.0.1`. The server profile exposes only the +frontend host port needed by a generic reverse proxy. Examples for Nginx and Caddy document TLS, +forwarded identity, websocket/SSE behavior, and timeouts, but neither proxy is embedded into or +required by the core architecture. + +## Persistence + +Named volumes are the portable default for workstation installations. Server documentation shows +explicit bind mounts under an operator-selected root such as `/srv/thothii`, with UID/GID and +backup requirements. The same container paths are used in both profiles: + +- `/data/workspace-registry` for Git checkout, immutable snapshots, leases, state, and locks; +- `/data/settings` for application settings; +- `/data/sessions` for filesystem sessions when selected; +- `/home/thoth/.pi` for Pi runtime state; +- `/run/secrets` for read-only secret files. + +Shared PostgreSQL session storage remains optional. It is required only when multiple ThothII +installations must see and resume the same sessions. + +## Cross-platform source and line endings + +The repository gains a root `.gitattributes` that makes line endings deterministic independently +of a developer's global Git configuration: + +```gitattributes +* text=auto +*.sh text eol=lf +Dockerfile* text eol=lf +*.Dockerfile text eol=lf +*.yml text eol=lf +*.yaml text eol=lf +*.json text eol=lf +*.ts text eol=lf +*.tsx text eol=lf +*.py text eol=lf +*.md text eol=lf +*.ps1 text eol=crlf +``` + +Executable shell scripts keep their executable bit and LF bytes. CI and a local verification +script scan Docker entrypoints, shell scripts, Compose/YAML, and Dockerfiles for carriage returns. +The Docker build also fails early with a clear message if an executable copied into an image has +CRLF. Documentation covers Git for Windows and WSL2, recommends cloning inside the WSL filesystem +for Linux-container work, and provides a safe one-time renormalization procedure for existing +clones. The documented process does not require changing global `core.autocrlf`. + +## Build and release behavior + +The local build uses the repository checkout as a BuildKit context and builds both images with +blocking TypeScript checks. The current non-blocking frontend typecheck is changed into a build +gate. A validation command renders each supported Compose combination before build. Image tags +include a local default and may be overridden with a release version or digest. + +Build inputs exclude `.git`, worktrees, local `.env` files, secrets, test output, caches, and +workspace runtime data through `.dockerignore`. Builds must work from macOS, Windows/WSL2, and +Linux without host-language runtimes beyond Docker, Compose, and Git. + +## Migration from the current deployment + +The current PSD/portal-oriented root Compose is replaced by the portable base. Reusable settings +from existing local and production overrides are folded into the new local/server overrides. +Portal network aliases, absolute Chirone paths, external `localllm_default`, PSD evidence mounts, +and `/datamart-builder` build arguments are removed from the product defaults. + +Existing operators migrate by copying only intentional values into the new environment and secret +files, rendering Compose, backing up volumes, building the new stack, and validating health and +workspace-registry status before switching the proxy. Legacy Compose examples remain in an +archive or are removed only after their replacement documentation and migration checks exist. + +## Error handling and operability + +Compose rendering fails when required non-secret values are absent. Entrypoints report stable, +sanitized errors for unreadable secret files, invalid registry configuration, incompatible line +endings, and storage permissions. Health checks distinguish process liveness from registry and +connector readiness. Remote outages preserve the last valid workspace snapshot as already +specified by the Git registry design. + +## Documentation + +Two complete guides are maintained against the same architecture: + +- local PC/Mac installation: Docker Desktop/Engine prerequisites, Windows/WSL2 line endings, + clone, environment creation, local build, first start, browser URL, update, backup, and recovery; +- server installation: service account, directories, firewall, generic reverse proxy, environment + and secrets, local image build or pinned image use, startup, health, upgrade, rollback, backup, + and recovery. + +Both guides use copy-pastable commands validated by scripts. They explain which configuration is +shared in Git, which is installation-local, and how a server that only provides DWH/VectorDB is +consumed without installing ThothII there. + +## Verification strategy + +Automated gates cover Compose rendering for local/server plus each Git transport, LF enforcement, +Docker builds, container health, frontend-to-core same-origin routing, registry bootstrap and +offline fallback, secret non-disclosure, and absence of PSD/Chirone/portal dependencies in active +deployment files. Existing backend, frontend, harness, registry, and installation-document tests +remain required. + +Manual acceptance covers a clean macOS build, a clean Windows Docker Desktop/WSL2 build from a +GitHub clone, a Linux server deployment behind a generic proxy, an update after `git pull`, volume +persistence, and restoration from backup. + +## Non-goals + +- Bundling a DWH, production VectorDB, embedding server, LLM server, or reverse proxy into the + mandatory ThothII stack. +- Making PSD, Chirone, or `omics_portal` supported product profiles. +- Synchronizing local filesystem sessions between installations without shared session storage. +- Implementing persistent runtime SSH tunnels for workspace connectors in this migration.