# Unified Docker Compose Deployment Design **Date:** 2026-08-04 **Status:** approved in conversation, pending written-spec review ## Objective ThothII ships as one autonomous Docker Compose application that runs unchanged on a developer PC/Mac or on a server. ThothII has no runtime, build, network, path, proxy, configuration, or documentation dependency on PSD, Chirone, `omics_portal`, or any other application that happens to provide databases or vector services. ## Architecture The distribution contains the same `frontend` and `core` images in every environment. A portable base Compose file defines services, health checks, internal networking, named volumes, registry storage, and configuration contracts. Small local and server overrides select host exposure, storage bindings, authentication policy, restart policy, and operational limits without copying the complete service definitions. ```text Browser -> frontend container -> core container |-> Git workspace registry |-> configured external DWH endpoint |-> configured external VectorDB endpoint |-> configured external embedding/LLM endpoints `-> configured session persistence ``` The frontend calls the core over the private Compose network through same-origin proxying. The browser never needs the core port in the server profile. On a workstation, the frontend is bound to loopback and the core may be bound to loopback for diagnostics. On a server, a generic external reverse proxy forwards to the frontend host port; that proxy is an operator concern and is not a ThothII dependency. ## Compose layout - `compose.yaml`: portable base stack and the only complete service definition. - `deploy/compose.local.yaml`: loopback ports, local named volumes, `AUTH_MODE=none`, workstation defaults, and local installation identity. - `deploy/compose.server.yaml`: frontend host binding suitable for a reverse proxy, no public core binding, server storage policy, configurable authentication, restart policy, and resource limits. - `deploy/compose.git-ssh.yaml` and `deploy/compose.git-https.yaml`: mutually exclusive Git secret mounts and trust configuration. - `deploy/compose.connector-secrets.yaml`: explicit connector secret mounts selected by an installation. - `.env.example`: non-secret common variables and documented absolute host paths. - `deploy/env/local.env.example` and `deploy/env/server.env.example`: profile-specific examples containing names and safe defaults, never credentials. The standard commands are intentionally symmetric: ```sh docker compose -f compose.yaml -f deploy/compose.local.yaml build docker compose -f compose.yaml -f deploy/compose.local.yaml up -d ``` ```sh docker compose -f compose.yaml -f deploy/compose.server.yaml build docker compose -f compose.yaml -f deploy/compose.server.yaml up -d ``` Published images remain optional. A server can build from a release checkout or consume pinned images through an operator override without changing the application architecture. ## Configuration and secrets Portable workspace descriptors remain in the Git workspace registry. Installation-specific endpoints, ports, usernames, CA paths, and secret-file bindings remain local. Secret contents are mounted as files and never enter Git, browser drafts, image layers, Compose output, or generated workspace artifacts. The same deterministic naming contract continues to apply: `THT_WS___` for bindings and `_FILE`/`_SOURCE` for secret paths. The base Compose accepts generic DWH, vector, embedding, LLM, Git, and session-storage endpoints; none has a default hostname, path, or network associated with PSD or `omics_portal`. ## Networking and exposure The base stack owns a private Compose network. `frontend` reaches `core` by service name. External DWH, vector, Git, embedding, and LLM services are reached through operator-configured DNS names or URLs. `host.docker.internal` may be documented as a workstation option but is not hard-coded as a product dependency. The local profile binds user-facing ports to `127.0.0.1`. The server profile exposes only the frontend host port needed by a generic reverse proxy. Examples for Nginx and Caddy document TLS, forwarded identity, websocket/SSE behavior, and timeouts, but neither proxy is embedded into or required by the core architecture. ## Product boundary and external services DWH, VectorDB, and embedding services are always external to the ThothII product boundary and Compose lifecycle. The mandatory ThothII stack neither defines nor starts them, never uses `depends_on` for them, and does not assume their implementation, installation path, container name, Docker network, or host. The same rule applies when all services happen to run on the same physical server: from ThothII's perspective they remain independently operated services reached through configurable addresses, ports, URLs, TLS settings, credentials, database/schema names, and vector collections. An operator may address a co-resident service through a routable host address, a DNS name, a documented host-gateway alias, or an explicitly configured external Docker network. None of these becomes a product default. In particular, `127.0.0.1` inside `core` always means the core container, not the Docker host; the installation guides must show the correct Mac/Windows Docker Desktop and Linux server alternatives. LLM endpoints follow the same configurable external-service model, while the Pi coding-agent runtime itself is part of ThothII as described below. ## Embedded Pi runtime Pi is an internal runtime dependency of ThothII and is installed at a pinned version while building the `core` image. The backend starts the image-bundled Pi executable; it never searches for or bind-mounts a Pi installation from the host. Therefore a clean Windows PC, Mac, Linux workstation, or server needs only Docker, Docker Compose, and Git to build and run ThothII. Pi configuration and provider credentials are supplied to the container through the documented ThothII configuration and secret-file contracts. Pi writable state may use the ThothII-managed `/home/thoth/.pi` volume, but the binary and package installation remain immutable image content. Upgrading Pi requires changing the pinned build argument, rebuilding the image, and passing the normal ThothII regression and image-smoke gates. The container health/smoke test verifies that Pi exists in the image and can be invoked without any host executable. ## Pi management for non-technical operators Pi management uses a hybrid interface so an operator does not need Docker expertise while image updates remain reproducible and recoverable. Configuration and diagnostics are available in a ThothII “Pi Management” page; host lifecycle and upgrades are performed by a small ThothII control command named `thothctl`. The Pi Management page shows the bundled Pi version, runtime state, configured provider, default model and reasoning level, writable Pi-state location, and sanitized diagnostics. It supports editing non-secret installation defaults, selecting only supported values, validating free-form fields, testing provider credentials without displaying them, running a Pi smoke request, and viewing sanitized logs. Per-user model/reasoning preferences remain browser-local until an authentication system provides durable user identities; installation defaults and Pi runtime configuration are stored in the mounted ThothII settings/Pi volumes. The page may report that a newer supported Pi version exists, but it does not control Docker and does not mutate the package inside a running container. It presents the exact `thothctl pi update` command appropriate to the installation. On a loopback-only single-user installation, the normal configuration functions are available. On a server, privileged Pi management requires trusted upstream authentication/authorization; without it, privileged controls are disabled and host-side `thothctl` remains the only update path. `thothctl` provides the stable operator commands: ```text thothctl status thothctl doctor thothctl logs thothctl update thothctl backup thothctl restore thothctl pi status thothctl pi doctor thothctl pi configure thothctl pi test thothctl pi update thothctl pi logs ``` The tool wraps validated Docker Compose operations and uses the same behavior on Windows, macOS, and Linux. Distribution may use a small native executable or platform launchers, but command names, prompts, exit codes, backups, and rollback semantics are identical. Interactive configuration asks plain-language questions, offers closed choices where possible, writes only local non-secret configuration, and directs credentials into protected secret files. `thothctl pi update` never runs `npm install` in the live container. It checks compatibility, records the current image/configuration, builds or pulls an image containing the selected pinned Pi version, recreates `core`, verifies health and `pi --version`, runs a smoke request, and rolls back to the recorded image if verification fails. The update preserves `/data` and `/home/thoth/.pi` volumes and prints a concise recovery result. For advanced support, documentation may expose `docker compose exec core pi ...`, but no browser shell is enabled by default. The `core` container never mounts the Docker socket. A future updater service or web-triggered image update requires a separate authenticated design and is outside this scope. ## Persistence Named volumes are the portable default for workstation installations. Server documentation shows explicit bind mounts under an operator-selected root such as `/srv/thothii`, with UID/GID and backup requirements. The same container paths are used in both profiles: - `/data/workspace-registry` for Git checkout, immutable snapshots, leases, state, and locks; - `/data/settings` for application settings; - `/data/sessions` for filesystem sessions when selected; - `/home/thoth/.pi` for Pi runtime state; - `/run/secrets` for read-only secret files. Shared PostgreSQL session storage remains optional. It is required only when multiple ThothII installations must see and resume the same sessions. ## Cross-platform source and line endings The repository gains a root `.gitattributes` that makes line endings deterministic independently of a developer's global Git configuration: ```gitattributes * text=auto *.sh text eol=lf Dockerfile* text eol=lf *.Dockerfile text eol=lf *.yml text eol=lf *.yaml text eol=lf *.json text eol=lf *.ts text eol=lf *.tsx text eol=lf *.py text eol=lf *.md text eol=lf *.ps1 text eol=crlf ``` Executable shell scripts keep their executable bit and LF bytes. CI and a local verification script scan Docker entrypoints, shell scripts, Compose/YAML, and Dockerfiles for carriage returns. The Docker build also fails early with a clear message if an executable copied into an image has CRLF. Documentation covers Git for Windows and WSL2, recommends cloning inside the WSL filesystem for Linux-container work, and provides a safe one-time renormalization procedure for existing clones. The documented process does not require changing global `core.autocrlf`. ## Build and release behavior The local build uses the repository checkout as a BuildKit context and builds both images with blocking TypeScript checks. The current non-blocking frontend typecheck is changed into a build gate. A validation command renders each supported Compose combination before build. Image tags include a local default and may be overridden with a release version or digest. Build inputs exclude `.git`, worktrees, local `.env` files, secrets, test output, caches, and workspace runtime data through `.dockerignore`. Builds must work from macOS, Windows/WSL2, and Linux without host-language runtimes or a host Pi installation beyond Docker, Compose, and Git. ## Migration from the current deployment The current PSD/portal-oriented root Compose is replaced by the portable base. Reusable settings from existing local and production overrides are folded into the new local/server overrides. Portal network aliases, absolute Chirone paths, external `localllm_default`, PSD evidence mounts, and `/datamart-builder` build arguments are removed from the product defaults. Existing operators migrate by copying only intentional values into the new environment and secret files, rendering Compose, backing up volumes, building the new stack, and validating health and workspace-registry status before switching the proxy. Legacy Compose examples remain in an archive or are removed only after their replacement documentation and migration checks exist. ## Error handling and operability Compose rendering fails when required non-secret values are absent. Entrypoints report stable, sanitized errors for unreadable secret files, invalid registry configuration, incompatible line endings, and storage permissions. Health checks distinguish process liveness from registry and connector readiness. Remote outages preserve the last valid workspace snapshot as already specified by the Git registry design. ## Documentation Two complete guides are maintained against the same architecture: - local PC/Mac installation: Docker Desktop/Engine prerequisites, Windows/WSL2 line endings, clone, environment creation, local build, first start, browser URL, update, backup, and recovery; - server installation: service account, directories, firewall, generic reverse proxy, environment and secrets, local image build or pinned image use, startup, health, upgrade, rollback, backup, and recovery. Both guides include a non-technical operator section for `thothctl`, Pi configuration, Pi upgrade, failed-upgrade rollback, and obtaining sanitized diagnostic output for support. Windows examples use native PowerShell commands or a packaged executable rather than assuming a Unix shell. Both guides use copy-pastable commands validated by scripts. They explain which configuration is shared in Git, which is installation-local, and how a server that only provides DWH/VectorDB is consumed without installing ThothII there. ## Verification strategy Automated gates cover Compose rendering for local/server plus each Git transport, LF enforcement, Docker builds, container health, frontend-to-core same-origin routing, registry bootstrap and offline fallback, secret non-disclosure, and absence of PSD/Chirone/portal dependencies in active deployment files. Image tests also prove that the pinned Pi executable is available inside `core` and that no host Pi path or Docker socket is mounted or required. Contract tests cover every `thothctl pi` command, non-interactive exit codes, update rollback, volume preservation, secret redaction, and parity of Windows/macOS/Linux launchers. Existing backend, frontend, harness, registry, and installation-document tests remain required. Manual acceptance covers a clean macOS build, a clean Windows Docker Desktop/WSL2 build from a GitHub clone, a Linux server deployment behind a generic proxy, an update after `git pull`, volume persistence, and restoration from backup. ## Non-goals - Bundling a DWH, production VectorDB, embedding server, LLM server, or reverse proxy into the mandatory ThothII stack, even when those services are co-resident on the same physical host. - Making PSD, Chirone, or `omics_portal` supported product profiles. - Synchronizing local filesystem sessions between installations without shared session storage. - Implementing persistent runtime SSH tunnels for workspace connectors in this migration. - Providing a browser terminal or allowing the application container to control the Docker daemon.