ThothII/docs/superpowers/specs/2026-08-04-unified-compose-deployment-design.md

Unified Docker Compose Deployment Design

Date: 2026-08-04

Status: approved in conversation, pending written-spec review

Objective

ThothII ships as one autonomous Docker Compose application that runs unchanged on a developer PC/Mac or on a server. ThothII has no runtime, build, network, path, proxy, configuration, or documentation dependency on PSD, Chirone, omics_portal, or any other application that happens to provide databases or vector services.

Architecture

The distribution contains the same frontend and core images in every environment. A portable base Compose file defines services, health checks, internal networking, named volumes, registry storage, and configuration contracts. Small local and server overrides select host exposure, storage bindings, authentication policy, restart policy, and operational limits without copying the complete service definitions.

Browser -> frontend container -> core container
                                  |-> Git workspace registry
                                  |-> configured external DWH endpoint
                                  |-> configured external VectorDB endpoint
                                  |-> configured external embedding/LLM endpoints
                                  `-> configured session persistence

The frontend calls the core over the private Compose network through same-origin proxying. The browser never needs the core port in the server profile. On a workstation, the frontend is bound to loopback and the core may be bound to loopback for diagnostics. On a server, a generic external reverse proxy forwards to the frontend host port; that proxy is an operator concern and is not a ThothII dependency.

Compose layout

• compose.yaml: portable base stack and the only complete service definition.
• deploy/compose.local.yaml: loopback ports, local named volumes, AUTH_MODE=none, workstation defaults, and local installation identity.
• deploy/compose.server.yaml: frontend host binding suitable for a reverse proxy, no public core binding, server storage policy, configurable authentication, restart policy, and resource limits.
• deploy/compose.git-ssh.yaml and deploy/compose.git-https.yaml: mutually exclusive Git secret mounts and trust configuration.
• deploy/compose.connector-secrets.yaml: explicit connector secret mounts selected by an installation.
• .env.example: non-secret common variables and documented absolute host paths.
• deploy/env/local.env.example and deploy/env/server.env.example: profile-specific examples containing names and safe defaults, never credentials.

The standard commands are intentionally symmetric:

docker compose -f compose.yaml -f deploy/compose.local.yaml build
docker compose -f compose.yaml -f deploy/compose.local.yaml up -d

docker compose -f compose.yaml -f deploy/compose.server.yaml build
docker compose -f compose.yaml -f deploy/compose.server.yaml up -d

Published images remain optional. A server can build from a release checkout or consume pinned images through an operator override without changing the application architecture.

Configuration and secrets

Portable workspace descriptors remain in the Git workspace registry. Installation-specific endpoints, ports, usernames, CA paths, and secret-file bindings remain local. Secret contents are mounted as files and never enter Git, browser drafts, image layers, Compose output, or generated workspace artifacts.

The same deterministic naming contract continues to apply: THT_WS_<WORKSPACE_ID>_<ROLE>_<FIELD> for bindings and _FILE/_SOURCE for secret paths. The base Compose accepts generic DWH, vector, embedding, LLM, Git, and session-storage endpoints; none has a default hostname, path, or network associated with PSD or omics_portal.

Networking and exposure

The base stack owns a private Compose network. frontend reaches core by service name. External DWH, vector, Git, embedding, and LLM services are reached through operator-configured DNS names or URLs. host.docker.internal may be documented as a workstation option but is not hard-coded as a product dependency.

The local profile binds user-facing ports to 127.0.0.1. The server profile exposes only the frontend host port needed by a generic reverse proxy. Examples for Nginx and Caddy document TLS, forwarded identity, websocket/SSE behavior, and timeouts, but neither proxy is embedded into or required by the core architecture.

Product boundary and external services

DWH, VectorDB, and embedding services are always external to the ThothII product boundary and Compose lifecycle. The mandatory ThothII stack neither defines nor starts them, never uses depends_on for them, and does not assume their implementation, installation path, container name, Docker network, or host. The same rule applies when all services happen to run on the same physical server: from ThothII's perspective they remain independently operated services reached through configurable addresses, ports, URLs, TLS settings, credentials, database/schema names, and vector collections.

An operator may address a co-resident service through a routable host address, a DNS name, a documented host-gateway alias, or an explicitly configured external Docker network. None of these becomes a product default. In particular, 127.0.0.1 inside core always means the core container, not the Docker host; the installation guides must show the correct Mac/Windows Docker Desktop and Linux server alternatives.

LLM endpoints follow the same configurable external-service model, while the Pi coding-agent runtime itself is part of ThothII as described below.

Embedded Pi runtime

Pi is an internal runtime dependency of ThothII and is installed at a pinned version while building the core image. The backend starts the image-bundled Pi executable; it never searches for or bind-mounts a Pi installation from the host. Therefore a clean Windows PC, Mac, Linux workstation, or server needs only Docker, Docker Compose, and Git to build and run ThothII.

Pi configuration and provider credentials are supplied to the container through the documented ThothII configuration and secret-file contracts. Pi writable state may use the ThothII-managed /home/thoth/.pi volume, but the binary and package installation remain immutable image content. Upgrading Pi requires changing the pinned build argument, rebuilding the image, and passing the normal ThothII regression and image-smoke gates. The container health/smoke test verifies that Pi exists in the image and can be invoked without any host executable.

Pi management for non-technical operators

Pi management uses a hybrid interface so an operator does not need Docker expertise while image updates remain reproducible and recoverable. Configuration and diagnostics are available in a ThothII “Pi Management” page; host lifecycle and upgrades are performed by a small ThothII control command named thothctl.

The Pi Management page shows the bundled Pi version, runtime state, configured provider, default model and reasoning level, writable Pi-state location, and sanitized diagnostics. It supports editing non-secret installation defaults, selecting only supported values, validating free-form fields, testing provider credentials without displaying them, running a Pi smoke request, and viewing sanitized logs. Per-user model/reasoning preferences remain browser-local until an authentication system provides durable user identities; installation defaults and Pi runtime configuration are stored in the mounted ThothII settings/Pi volumes.

The page may report that a newer supported Pi version exists, but it does not control Docker and does not mutate the package inside a running container. It presents the exact thothctl pi update command appropriate to the installation. On a loopback-only single-user installation, the normal configuration functions are available. On a server, privileged Pi management requires trusted upstream authentication/authorization; without it, privileged controls are disabled and host-side thothctl remains the only update path.

thothctl provides the stable operator commands:

thothctl status
thothctl doctor
thothctl logs
thothctl update
thothctl backup
thothctl restore
thothctl pi status
thothctl pi doctor
thothctl pi configure
thothctl pi test
thothctl pi update
thothctl pi logs

The tool wraps validated Docker Compose operations and uses the same behavior on Windows, macOS, and Linux. Distribution may use a small native executable or platform launchers, but command names, prompts, exit codes, backups, and rollback semantics are identical. Interactive configuration asks plain-language questions, offers closed choices where possible, writes only local non-secret configuration, and directs credentials into protected secret files.

thothctl pi update never runs npm install in the live container. It checks compatibility, records the current image/configuration, builds or pulls an image containing the selected pinned Pi version, recreates core, verifies health and pi --version, runs a smoke request, and rolls back to the recorded image if verification fails. The update preserves /data and /home/thoth/.pi volumes and prints a concise recovery result.

For advanced support, documentation may expose docker compose exec core pi ..., but no browser shell is enabled by default. The core container never mounts the Docker socket. A future updater service or web-triggered image update requires a separate authenticated design and is outside this scope.

Persistence

Named volumes are the portable default for workstation installations. Server documentation shows explicit bind mounts under an operator-selected root such as /srv/thothii, with UID/GID and backup requirements. The same container paths are used in both profiles:

• /data/workspace-registry for Git checkout, immutable snapshots, leases, state, and locks;
• /data/settings for application settings;
• /data/sessions for filesystem sessions when selected;
• /home/thoth/.pi for Pi runtime state;
• /run/secrets for read-only secret files.

Shared PostgreSQL session storage remains optional. It is required only when multiple ThothII installations must see and resume the same sessions.

Cross-platform source and line endings

The repository gains a root .gitattributes that makes line endings deterministic independently of a developer's global Git configuration:

* text=auto
*.sh text eol=lf
Dockerfile* text eol=lf
*.Dockerfile text eol=lf
*.yml text eol=lf
*.yaml text eol=lf
*.json text eol=lf
*.ts text eol=lf
*.tsx text eol=lf
*.py text eol=lf
*.md text eol=lf
*.ps1 text eol=crlf

Executable shell scripts keep their executable bit and LF bytes. CI and a local verification script scan Docker entrypoints, shell scripts, Compose/YAML, and Dockerfiles for carriage returns. The Docker build also fails early with a clear message if an executable copied into an image has CRLF. Documentation covers Git for Windows and WSL2, recommends cloning inside the WSL filesystem for Linux-container work, and provides a safe one-time renormalization procedure for existing clones. The documented process does not require changing global core.autocrlf.

Build and release behavior

The local build uses the repository checkout as a BuildKit context and builds both images with blocking TypeScript checks. The current non-blocking frontend typecheck is changed into a build gate. A validation command renders each supported Compose combination before build. Image tags include a local default and may be overridden with a release version or digest.

Build inputs exclude .git, worktrees, local .env files, secrets, test output, caches, and workspace runtime data through .dockerignore. Builds must work from macOS, Windows/WSL2, and Linux without host-language runtimes or a host Pi installation beyond Docker, Compose, and Git.

Migration from the current deployment

The current PSD/portal-oriented root Compose is replaced by the portable base. Reusable settings from existing local and production overrides are folded into the new local/server overrides. Portal network aliases, absolute Chirone paths, external localllm_default, PSD evidence mounts, and /datamart-builder build arguments are removed from the product defaults.

Existing operators migrate by copying only intentional values into the new environment and secret files, rendering Compose, backing up volumes, building the new stack, and validating health and workspace-registry status before switching the proxy. Legacy Compose examples remain in an archive or are removed only after their replacement documentation and migration checks exist.

Error handling and operability

Compose rendering fails when required non-secret values are absent. Entrypoints report stable, sanitized errors for unreadable secret files, invalid registry configuration, incompatible line endings, and storage permissions. Health checks distinguish process liveness from registry and connector readiness. Remote outages preserve the last valid workspace snapshot as already specified by the Git registry design.

Documentation

Two complete guides are maintained against the same architecture:

• local PC/Mac installation: Docker Desktop/Engine prerequisites, Windows/WSL2 line endings, clone, environment creation, local build, first start, browser URL, update, backup, and recovery;
• server installation: service account, directories, firewall, generic reverse proxy, environment and secrets, local image build or pinned image use, startup, health, upgrade, rollback, backup, and recovery.

Both guides include a non-technical operator section for thothctl, Pi configuration, Pi upgrade, failed-upgrade rollback, and obtaining sanitized diagnostic output for support. Windows examples use native PowerShell commands or a packaged executable rather than assuming a Unix shell.

Both guides use copy-pastable commands validated by scripts. They explain which configuration is shared in Git, which is installation-local, and how a server that only provides DWH/VectorDB is consumed without installing ThothII there.

Verification strategy

Automated gates cover Compose rendering for local/server plus each Git transport, LF enforcement, Docker builds, container health, frontend-to-core same-origin routing, registry bootstrap and offline fallback, secret non-disclosure, and absence of PSD/Chirone/portal dependencies in active deployment files. Image tests also prove that the pinned Pi executable is available inside core and that no host Pi path or Docker socket is mounted or required. Contract tests cover every thothctl pi command, non-interactive exit codes, update rollback, volume preservation, secret redaction, and parity of Windows/macOS/Linux launchers. Existing backend, frontend, harness, registry, and installation-document tests remain required.

Manual acceptance covers a clean macOS build, a clean Windows Docker Desktop/WSL2 build from a GitHub clone, a Linux server deployment behind a generic proxy, an update after git pull, volume persistence, and restoration from backup.

Non-goals

• Bundling a DWH, production VectorDB, embedding server, LLM server, or reverse proxy into the mandatory ThothII stack, even when those services are co-resident on the same physical host.
• Making PSD, Chirone, or omics_portal supported product profiles.
• Synchronizing local filesystem sessions between installations without shared session storage.
• Implementing persistent runtime SSH tunnels for workspace connectors in this migration.
• Providing a browser terminal or allowing the application container to control the Docker daemon.