294 lines
16 KiB
Markdown
294 lines
16 KiB
Markdown
# Unified Docker Compose Deployment Design
|
|
|
|
**Date:** 2026-08-04
|
|
|
|
**Status:** approved in conversation, pending written-spec review
|
|
|
|
## Objective
|
|
|
|
ThothII ships as one autonomous Docker Compose application that runs unchanged on a developer
|
|
PC/Mac or on a server. ThothII has no runtime, build, network, path, proxy, configuration, or
|
|
documentation dependency on PSD, Chirone, `omics_portal`, or any other application that happens
|
|
to provide databases or vector services.
|
|
|
|
## Architecture
|
|
|
|
The distribution contains the same `frontend` and `core` images in every environment. A portable
|
|
base Compose file defines services, health checks, internal networking, named volumes, registry
|
|
storage, and configuration contracts. Small local and server overrides select host exposure,
|
|
storage bindings, authentication policy, restart policy, and operational limits without copying
|
|
the complete service definitions.
|
|
|
|
```text
|
|
Browser -> frontend container -> core container
|
|
|-> Git workspace registry
|
|
|-> configured external DWH endpoint
|
|
|-> configured external VectorDB endpoint
|
|
|-> configured external embedding/LLM endpoints
|
|
`-> configured session persistence
|
|
```
|
|
|
|
The frontend calls the core over the private Compose network through same-origin proxying. The
|
|
browser never needs the core port in the server profile. On a workstation, the frontend is bound
|
|
to loopback and the core may be bound to loopback for diagnostics. On a server, a generic external
|
|
reverse proxy forwards to the frontend host port; that proxy is an operator concern and is not a
|
|
ThothII dependency.
|
|
|
|
## Compose layout
|
|
|
|
- `compose.yaml`: portable base stack and the only complete service definition.
|
|
- `deploy/compose.local.yaml`: loopback ports, local named volumes, `AUTH_MODE=none`, workstation
|
|
defaults, and local installation identity.
|
|
- `deploy/compose.server.yaml`: frontend host binding suitable for a reverse proxy, no public core
|
|
binding, server storage policy, configurable authentication, restart policy, and resource limits.
|
|
- `deploy/compose.git-ssh.yaml` and `deploy/compose.git-https.yaml`: mutually exclusive Git secret
|
|
mounts and trust configuration.
|
|
- `deploy/compose.connector-secrets.yaml`: explicit connector secret mounts selected by an
|
|
installation.
|
|
- `.env.example`: non-secret common variables and documented absolute host paths.
|
|
- `deploy/env/local.env.example` and `deploy/env/server.env.example`: profile-specific examples
|
|
containing names and safe defaults, never credentials.
|
|
|
|
The standard commands are intentionally symmetric:
|
|
|
|
```sh
|
|
docker compose -f compose.yaml -f deploy/compose.local.yaml build
|
|
docker compose -f compose.yaml -f deploy/compose.local.yaml up -d
|
|
```
|
|
|
|
```sh
|
|
docker compose -f compose.yaml -f deploy/compose.server.yaml build
|
|
docker compose -f compose.yaml -f deploy/compose.server.yaml up -d
|
|
```
|
|
|
|
Published images remain optional. A server can build from a release checkout or consume pinned
|
|
images through an operator override without changing the application architecture.
|
|
|
|
## Configuration and secrets
|
|
|
|
Portable workspace descriptors remain in the Git workspace registry. Installation-specific
|
|
endpoints, ports, usernames, CA paths, and secret-file bindings remain local. Secret contents are
|
|
mounted as files and never enter Git, browser drafts, image layers, Compose output, or generated
|
|
workspace artifacts.
|
|
|
|
The same deterministic naming contract continues to apply:
|
|
`THT_WS_<WORKSPACE_ID>_<ROLE>_<FIELD>` for bindings and `_FILE`/`_SOURCE` for secret paths. The
|
|
base Compose accepts generic DWH, vector, embedding, LLM, Git, and session-storage endpoints; none
|
|
has a default hostname, path, or network associated with PSD or `omics_portal`.
|
|
|
|
## Networking and exposure
|
|
|
|
The base stack owns a private Compose network. `frontend` reaches `core` by service name. External
|
|
DWH, vector, Git, embedding, and LLM services are reached through operator-configured DNS names or
|
|
URLs. `host.docker.internal` may be documented as a workstation option but is not hard-coded as a
|
|
product dependency.
|
|
|
|
The local profile binds user-facing ports to `127.0.0.1`. The server profile exposes only the
|
|
frontend host port needed by a generic reverse proxy. Examples for Nginx and Caddy document TLS,
|
|
forwarded identity, websocket/SSE behavior, and timeouts, but neither proxy is embedded into or
|
|
required by the core architecture.
|
|
|
|
## Product boundary and external services
|
|
|
|
DWH, VectorDB, and embedding services are always external to the ThothII product boundary and
|
|
Compose lifecycle. The mandatory ThothII stack neither defines nor starts them, never uses
|
|
`depends_on` for them, and does not assume their implementation, installation path, container
|
|
name, Docker network, or host. The same rule applies when all services happen to run on the same
|
|
physical server: from ThothII's perspective they remain independently operated services reached
|
|
through configurable addresses, ports, URLs, TLS settings, credentials, database/schema names,
|
|
and vector collections.
|
|
|
|
An operator may address a co-resident service through a routable host address, a DNS name, a
|
|
documented host-gateway alias, or an explicitly configured external Docker network. None of these
|
|
becomes a product default. In particular, `127.0.0.1` inside `core` always means the core container,
|
|
not the Docker host; the installation guides must show the correct Mac/Windows Docker Desktop and
|
|
Linux server alternatives.
|
|
|
|
LLM endpoints follow the same configurable external-service model, while the Pi coding-agent
|
|
runtime itself is part of ThothII as described below.
|
|
|
|
## Embedded Pi runtime
|
|
|
|
Pi is an internal runtime dependency of ThothII and is installed at a pinned version while building
|
|
the `core` image. The backend starts the image-bundled Pi executable; it never searches for or
|
|
bind-mounts a Pi installation from the host. Therefore a clean Windows PC, Mac, Linux workstation,
|
|
or server needs only Docker, Docker Compose, and Git to build and run ThothII.
|
|
|
|
Pi configuration and provider credentials are supplied to the container through the documented
|
|
ThothII configuration and secret-file contracts. Pi writable state may use the ThothII-managed
|
|
`/home/thoth/.pi` volume, but the binary and package installation remain immutable image content.
|
|
Upgrading Pi requires changing the pinned build argument, rebuilding the image, and passing the
|
|
normal ThothII regression and image-smoke gates. The container health/smoke test verifies that Pi
|
|
exists in the image and can be invoked without any host executable.
|
|
|
|
## Pi management for non-technical operators
|
|
|
|
Pi management uses a hybrid interface so an operator does not need Docker expertise while image
|
|
updates remain reproducible and recoverable. Configuration and diagnostics are available in a
|
|
ThothII “Pi Management” page; host lifecycle and upgrades are performed by a small ThothII control
|
|
command named `thothctl`.
|
|
|
|
The Pi Management page shows the bundled Pi version, runtime state, configured provider, default
|
|
model and reasoning level, writable Pi-state location, and sanitized diagnostics. It supports
|
|
editing non-secret installation defaults, selecting only supported values, validating free-form
|
|
fields, testing provider credentials without displaying them, running a Pi smoke request, and
|
|
viewing sanitized logs. Per-user model/reasoning preferences remain browser-local until an
|
|
authentication system provides durable user identities; installation defaults and Pi runtime
|
|
configuration are stored in the mounted ThothII settings/Pi volumes.
|
|
|
|
The page may report that a newer supported Pi version exists, but it does not control Docker and
|
|
does not mutate the package inside a running container. It presents the exact `thothctl pi update`
|
|
command appropriate to the installation. On a loopback-only single-user installation, the normal
|
|
configuration functions are available. On a server, privileged Pi management requires trusted
|
|
upstream authentication/authorization; without it, privileged controls are disabled and host-side
|
|
`thothctl` remains the only update path.
|
|
|
|
`thothctl` provides the stable operator commands:
|
|
|
|
```text
|
|
thothctl status
|
|
thothctl doctor
|
|
thothctl logs
|
|
thothctl update
|
|
thothctl backup
|
|
thothctl restore
|
|
thothctl pi status
|
|
thothctl pi doctor
|
|
thothctl pi configure
|
|
thothctl pi test
|
|
thothctl pi update
|
|
thothctl pi logs
|
|
```
|
|
|
|
The tool wraps validated Docker Compose operations and uses the same behavior on Windows, macOS,
|
|
and Linux. Distribution may use a small native executable or platform launchers, but command names,
|
|
prompts, exit codes, backups, and rollback semantics are identical. Interactive configuration asks
|
|
plain-language questions, offers closed choices where possible, writes only local non-secret
|
|
configuration, and directs credentials into protected secret files.
|
|
|
|
`thothctl pi update` never runs `npm install` in the live container. It checks compatibility,
|
|
records the current image/configuration, builds or pulls an image containing the selected pinned Pi
|
|
version, recreates `core`, verifies health and `pi --version`, runs a smoke request, and rolls back
|
|
to the recorded image if verification fails. The update preserves `/data` and `/home/thoth/.pi`
|
|
volumes and prints a concise recovery result.
|
|
|
|
For advanced support, documentation may expose `docker compose exec core pi ...`, but no browser
|
|
shell is enabled by default. The `core` container never mounts the Docker socket. A future updater
|
|
service or web-triggered image update requires a separate authenticated design and is outside this
|
|
scope.
|
|
|
|
## Persistence
|
|
|
|
Named volumes are the portable default for workstation installations. Server documentation shows
|
|
explicit bind mounts under an operator-selected root such as `/srv/thothii`, with UID/GID and
|
|
backup requirements. The same container paths are used in both profiles:
|
|
|
|
- `/data/workspace-registry` for Git checkout, immutable snapshots, leases, state, and locks;
|
|
- `/data/settings` for application settings;
|
|
- `/data/sessions` for filesystem sessions when selected;
|
|
- `/home/thoth/.pi` for Pi runtime state;
|
|
- `/run/secrets` for read-only secret files.
|
|
|
|
Shared PostgreSQL session storage remains optional. It is required only when multiple ThothII
|
|
installations must see and resume the same sessions.
|
|
|
|
## Cross-platform source and line endings
|
|
|
|
The repository gains a root `.gitattributes` that makes line endings deterministic independently
|
|
of a developer's global Git configuration:
|
|
|
|
```gitattributes
|
|
* text=auto
|
|
*.sh text eol=lf
|
|
Dockerfile* text eol=lf
|
|
*.Dockerfile text eol=lf
|
|
*.yml text eol=lf
|
|
*.yaml text eol=lf
|
|
*.json text eol=lf
|
|
*.ts text eol=lf
|
|
*.tsx text eol=lf
|
|
*.py text eol=lf
|
|
*.md text eol=lf
|
|
*.ps1 text eol=crlf
|
|
```
|
|
|
|
Executable shell scripts keep their executable bit and LF bytes. CI and a local verification
|
|
script scan Docker entrypoints, shell scripts, Compose/YAML, and Dockerfiles for carriage returns.
|
|
The Docker build also fails early with a clear message if an executable copied into an image has
|
|
CRLF. Documentation covers Git for Windows and WSL2, recommends cloning inside the WSL filesystem
|
|
for Linux-container work, and provides a safe one-time renormalization procedure for existing
|
|
clones. The documented process does not require changing global `core.autocrlf`.
|
|
|
|
## Build and release behavior
|
|
|
|
The local build uses the repository checkout as a BuildKit context and builds both images with
|
|
blocking TypeScript checks. The current non-blocking frontend typecheck is changed into a build
|
|
gate. A validation command renders each supported Compose combination before build. Image tags
|
|
include a local default and may be overridden with a release version or digest.
|
|
|
|
Build inputs exclude `.git`, worktrees, local `.env` files, secrets, test output, caches, and
|
|
workspace runtime data through `.dockerignore`. Builds must work from macOS, Windows/WSL2, and
|
|
Linux without host-language runtimes or a host Pi installation beyond Docker, Compose, and Git.
|
|
|
|
## Migration from the current deployment
|
|
|
|
The current PSD/portal-oriented root Compose is replaced by the portable base. Reusable settings
|
|
from existing local and production overrides are folded into the new local/server overrides.
|
|
Portal network aliases, absolute Chirone paths, external `localllm_default`, PSD evidence mounts,
|
|
and `/datamart-builder` build arguments are removed from the product defaults.
|
|
|
|
Existing operators migrate by copying only intentional values into the new environment and secret
|
|
files, rendering Compose, backing up volumes, building the new stack, and validating health and
|
|
workspace-registry status before switching the proxy. Legacy Compose examples remain in an
|
|
archive or are removed only after their replacement documentation and migration checks exist.
|
|
|
|
## Error handling and operability
|
|
|
|
Compose rendering fails when required non-secret values are absent. Entrypoints report stable,
|
|
sanitized errors for unreadable secret files, invalid registry configuration, incompatible line
|
|
endings, and storage permissions. Health checks distinguish process liveness from registry and
|
|
connector readiness. Remote outages preserve the last valid workspace snapshot as already
|
|
specified by the Git registry design.
|
|
|
|
## Documentation
|
|
|
|
Two complete guides are maintained against the same architecture:
|
|
|
|
- local PC/Mac installation: Docker Desktop/Engine prerequisites, Windows/WSL2 line endings,
|
|
clone, environment creation, local build, first start, browser URL, update, backup, and recovery;
|
|
- server installation: service account, directories, firewall, generic reverse proxy, environment
|
|
and secrets, local image build or pinned image use, startup, health, upgrade, rollback, backup,
|
|
and recovery.
|
|
|
|
Both guides include a non-technical operator section for `thothctl`, Pi configuration, Pi upgrade,
|
|
failed-upgrade rollback, and obtaining sanitized diagnostic output for support. Windows examples
|
|
use native PowerShell commands or a packaged executable rather than assuming a Unix shell.
|
|
|
|
Both guides use copy-pastable commands validated by scripts. They explain which configuration is
|
|
shared in Git, which is installation-local, and how a server that only provides DWH/VectorDB is
|
|
consumed without installing ThothII there.
|
|
|
|
## Verification strategy
|
|
|
|
Automated gates cover Compose rendering for local/server plus each Git transport, LF enforcement,
|
|
Docker builds, container health, frontend-to-core same-origin routing, registry bootstrap and
|
|
offline fallback, secret non-disclosure, and absence of PSD/Chirone/portal dependencies in active
|
|
deployment files. Image tests also prove that the pinned Pi executable is available inside `core`
|
|
and that no host Pi path or Docker socket is mounted or required. Contract tests cover every
|
|
`thothctl pi` command, non-interactive exit codes, update rollback, volume preservation, secret
|
|
redaction, and parity of Windows/macOS/Linux launchers. Existing backend, frontend, harness,
|
|
registry, and installation-document tests remain required.
|
|
|
|
Manual acceptance covers a clean macOS build, a clean Windows Docker Desktop/WSL2 build from a
|
|
GitHub clone, a Linux server deployment behind a generic proxy, an update after `git pull`, volume
|
|
persistence, and restoration from backup.
|
|
|
|
## Non-goals
|
|
|
|
- Bundling a DWH, production VectorDB, embedding server, LLM server, or reverse proxy into the
|
|
mandatory ThothII stack, even when those services are co-resident on the same physical host.
|
|
- Making PSD, Chirone, or `omics_portal` supported product profiles.
|
|
- Synchronizing local filesystem sessions between installations without shared session storage.
|
|
- Implementing persistent runtime SSH tunnels for workspace connectors in this migration.
|
|
- Providing a browser terminal or allowing the application container to control the Docker daemon.
|