ThothII/docs/superpowers/specs/2026-08-04-unified-compose-deployment-design.md

Unified Docker Compose Deployment Design

Date: 2026-08-04

Status: approved in conversation, pending written-spec review

Objective

ThothII ships as one autonomous Docker Compose application that runs unchanged on a developer PC/Mac or on a server. ThothII has no runtime, build, network, path, proxy, configuration, or documentation dependency on PSD, Chirone, omics_portal, or any other application that happens to provide databases or vector services.

Architecture

The distribution contains the same frontend and core images in every environment. A portable base Compose file defines services, health checks, internal networking, named volumes, registry storage, and configuration contracts. Small local and server overrides select host exposure, storage bindings, authentication policy, restart policy, and operational limits without copying the complete service definitions.

Browser -> frontend container -> core container
                                  |-> Git workspace registry
                                  |-> configured DWH
                                  |-> configured VectorDB
                                  |-> configured embedding/LLM services
                                  `-> configured session persistence

The frontend calls the core over the private Compose network through same-origin proxying. The browser never needs the core port in the server profile. On a workstation, the frontend is bound to loopback and the core may be bound to loopback for diagnostics. On a server, a generic external reverse proxy forwards to the frontend host port; that proxy is an operator concern and is not a ThothII dependency.

Compose layout

• compose.yaml: portable base stack and the only complete service definition.
• deploy/compose.local.yaml: loopback ports, local named volumes, AUTH_MODE=none, workstation defaults, and local installation identity.
• deploy/compose.server.yaml: frontend host binding suitable for a reverse proxy, no public core binding, server storage policy, configurable authentication, restart policy, and resource limits.
• deploy/compose.git-ssh.yaml and deploy/compose.git-https.yaml: mutually exclusive Git secret mounts and trust configuration.
• deploy/compose.connector-secrets.yaml: explicit connector secret mounts selected by an installation.
• .env.example: non-secret common variables and documented absolute host paths.
• deploy/env/local.env.example and deploy/env/server.env.example: profile-specific examples containing names and safe defaults, never credentials.

The standard commands are intentionally symmetric:

docker compose -f compose.yaml -f deploy/compose.local.yaml build
docker compose -f compose.yaml -f deploy/compose.local.yaml up -d

docker compose -f compose.yaml -f deploy/compose.server.yaml build
docker compose -f compose.yaml -f deploy/compose.server.yaml up -d

Published images remain optional. A server can build from a release checkout or consume pinned images through an operator override without changing the application architecture.

Configuration and secrets

Portable workspace descriptors remain in the Git workspace registry. Installation-specific endpoints, ports, usernames, CA paths, and secret-file bindings remain local. Secret contents are mounted as files and never enter Git, browser drafts, image layers, Compose output, or generated workspace artifacts.

The same deterministic naming contract continues to apply: THT_WS_<WORKSPACE_ID>_<ROLE>_<FIELD> for bindings and _FILE/_SOURCE for secret paths. The base Compose accepts generic DWH, vector, embedding, LLM, Git, and session-storage endpoints; none has a default hostname, path, or network associated with PSD or omics_portal.

Networking and exposure

The base stack owns a private Compose network. frontend reaches core by service name. External DWH, vector, Git, embedding, and LLM services are reached through operator-configured DNS names or URLs. host.docker.internal may be documented as a workstation option but is not hard-coded as a product dependency.

The local profile binds user-facing ports to 127.0.0.1. The server profile exposes only the frontend host port needed by a generic reverse proxy. Examples for Nginx and Caddy document TLS, forwarded identity, websocket/SSE behavior, and timeouts, but neither proxy is embedded into or required by the core architecture.

Persistence

Named volumes are the portable default for workstation installations. Server documentation shows explicit bind mounts under an operator-selected root such as /srv/thothii, with UID/GID and backup requirements. The same container paths are used in both profiles:

• /data/workspace-registry for Git checkout, immutable snapshots, leases, state, and locks;
• /data/settings for application settings;
• /data/sessions for filesystem sessions when selected;
• /home/thoth/.pi for Pi runtime state;
• /run/secrets for read-only secret files.

Shared PostgreSQL session storage remains optional. It is required only when multiple ThothII installations must see and resume the same sessions.

Cross-platform source and line endings

The repository gains a root .gitattributes that makes line endings deterministic independently of a developer's global Git configuration:

* text=auto
*.sh text eol=lf
Dockerfile* text eol=lf
*.Dockerfile text eol=lf
*.yml text eol=lf
*.yaml text eol=lf
*.json text eol=lf
*.ts text eol=lf
*.tsx text eol=lf
*.py text eol=lf
*.md text eol=lf
*.ps1 text eol=crlf

Executable shell scripts keep their executable bit and LF bytes. CI and a local verification script scan Docker entrypoints, shell scripts, Compose/YAML, and Dockerfiles for carriage returns. The Docker build also fails early with a clear message if an executable copied into an image has CRLF. Documentation covers Git for Windows and WSL2, recommends cloning inside the WSL filesystem for Linux-container work, and provides a safe one-time renormalization procedure for existing clones. The documented process does not require changing global core.autocrlf.

Build and release behavior

The local build uses the repository checkout as a BuildKit context and builds both images with blocking TypeScript checks. The current non-blocking frontend typecheck is changed into a build gate. A validation command renders each supported Compose combination before build. Image tags include a local default and may be overridden with a release version or digest.

Build inputs exclude .git, worktrees, local .env files, secrets, test output, caches, and workspace runtime data through .dockerignore. Builds must work from macOS, Windows/WSL2, and Linux without host-language runtimes beyond Docker, Compose, and Git.

Migration from the current deployment

The current PSD/portal-oriented root Compose is replaced by the portable base. Reusable settings from existing local and production overrides are folded into the new local/server overrides. Portal network aliases, absolute Chirone paths, external localllm_default, PSD evidence mounts, and /datamart-builder build arguments are removed from the product defaults.

Existing operators migrate by copying only intentional values into the new environment and secret files, rendering Compose, backing up volumes, building the new stack, and validating health and workspace-registry status before switching the proxy. Legacy Compose examples remain in an archive or are removed only after their replacement documentation and migration checks exist.

Error handling and operability

Compose rendering fails when required non-secret values are absent. Entrypoints report stable, sanitized errors for unreadable secret files, invalid registry configuration, incompatible line endings, and storage permissions. Health checks distinguish process liveness from registry and connector readiness. Remote outages preserve the last valid workspace snapshot as already specified by the Git registry design.

Documentation

Two complete guides are maintained against the same architecture:

• local PC/Mac installation: Docker Desktop/Engine prerequisites, Windows/WSL2 line endings, clone, environment creation, local build, first start, browser URL, update, backup, and recovery;
• server installation: service account, directories, firewall, generic reverse proxy, environment and secrets, local image build or pinned image use, startup, health, upgrade, rollback, backup, and recovery.

Both guides use copy-pastable commands validated by scripts. They explain which configuration is shared in Git, which is installation-local, and how a server that only provides DWH/VectorDB is consumed without installing ThothII there.

Verification strategy

Automated gates cover Compose rendering for local/server plus each Git transport, LF enforcement, Docker builds, container health, frontend-to-core same-origin routing, registry bootstrap and offline fallback, secret non-disclosure, and absence of PSD/Chirone/portal dependencies in active deployment files. Existing backend, frontend, harness, registry, and installation-document tests remain required.

Manual acceptance covers a clean macOS build, a clean Windows Docker Desktop/WSL2 build from a GitHub clone, a Linux server deployment behind a generic proxy, an update after git pull, volume persistence, and restoration from backup.

Non-goals

• Bundling a DWH, production VectorDB, embedding server, LLM server, or reverse proxy into the mandatory ThothII stack.
• Making PSD, Chirone, or omics_portal supported product profiles.
• Synchronizing local filesystem sessions between installations without shared session storage.
• Implementing persistent runtime SSH tunnels for workspace connectors in this migration.