docs: design unified compose deployment
This commit is contained in:
@@ -0,0 +1,196 @@
|
||||
# Unified Docker Compose Deployment Design
|
||||
|
||||
**Date:** 2026-08-04
|
||||
|
||||
**Status:** approved in conversation, pending written-spec review
|
||||
|
||||
## Objective
|
||||
|
||||
ThothII ships as one autonomous Docker Compose application that runs unchanged on a developer
|
||||
PC/Mac or on a server. ThothII has no runtime, build, network, path, proxy, configuration, or
|
||||
documentation dependency on PSD, Chirone, `omics_portal`, or any other application that happens
|
||||
to provide databases or vector services.
|
||||
|
||||
## Architecture
|
||||
|
||||
The distribution contains the same `frontend` and `core` images in every environment. A portable
|
||||
base Compose file defines services, health checks, internal networking, named volumes, registry
|
||||
storage, and configuration contracts. Small local and server overrides select host exposure,
|
||||
storage bindings, authentication policy, restart policy, and operational limits without copying
|
||||
the complete service definitions.
|
||||
|
||||
```text
|
||||
Browser -> frontend container -> core container
|
||||
|-> Git workspace registry
|
||||
|-> configured DWH
|
||||
|-> configured VectorDB
|
||||
|-> configured embedding/LLM services
|
||||
`-> configured session persistence
|
||||
```
|
||||
|
||||
The frontend calls the core over the private Compose network through same-origin proxying. The
|
||||
browser never needs the core port in the server profile. On a workstation, the frontend is bound
|
||||
to loopback and the core may be bound to loopback for diagnostics. On a server, a generic external
|
||||
reverse proxy forwards to the frontend host port; that proxy is an operator concern and is not a
|
||||
ThothII dependency.
|
||||
|
||||
## Compose layout
|
||||
|
||||
- `compose.yaml`: portable base stack and the only complete service definition.
|
||||
- `deploy/compose.local.yaml`: loopback ports, local named volumes, `AUTH_MODE=none`, workstation
|
||||
defaults, and local installation identity.
|
||||
- `deploy/compose.server.yaml`: frontend host binding suitable for a reverse proxy, no public core
|
||||
binding, server storage policy, configurable authentication, restart policy, and resource limits.
|
||||
- `deploy/compose.git-ssh.yaml` and `deploy/compose.git-https.yaml`: mutually exclusive Git secret
|
||||
mounts and trust configuration.
|
||||
- `deploy/compose.connector-secrets.yaml`: explicit connector secret mounts selected by an
|
||||
installation.
|
||||
- `.env.example`: non-secret common variables and documented absolute host paths.
|
||||
- `deploy/env/local.env.example` and `deploy/env/server.env.example`: profile-specific examples
|
||||
containing names and safe defaults, never credentials.
|
||||
|
||||
The standard commands are intentionally symmetric:
|
||||
|
||||
```sh
|
||||
docker compose -f compose.yaml -f deploy/compose.local.yaml build
|
||||
docker compose -f compose.yaml -f deploy/compose.local.yaml up -d
|
||||
```
|
||||
|
||||
```sh
|
||||
docker compose -f compose.yaml -f deploy/compose.server.yaml build
|
||||
docker compose -f compose.yaml -f deploy/compose.server.yaml up -d
|
||||
```
|
||||
|
||||
Published images remain optional. A server can build from a release checkout or consume pinned
|
||||
images through an operator override without changing the application architecture.
|
||||
|
||||
## Configuration and secrets
|
||||
|
||||
Portable workspace descriptors remain in the Git workspace registry. Installation-specific
|
||||
endpoints, ports, usernames, CA paths, and secret-file bindings remain local. Secret contents are
|
||||
mounted as files and never enter Git, browser drafts, image layers, Compose output, or generated
|
||||
workspace artifacts.
|
||||
|
||||
The same deterministic naming contract continues to apply:
|
||||
`THT_WS_<WORKSPACE_ID>_<ROLE>_<FIELD>` for bindings and `_FILE`/`_SOURCE` for secret paths. The
|
||||
base Compose accepts generic DWH, vector, embedding, LLM, Git, and session-storage endpoints; none
|
||||
has a default hostname, path, or network associated with PSD or `omics_portal`.
|
||||
|
||||
## Networking and exposure
|
||||
|
||||
The base stack owns a private Compose network. `frontend` reaches `core` by service name. External
|
||||
DWH, vector, Git, embedding, and LLM services are reached through operator-configured DNS names or
|
||||
URLs. `host.docker.internal` may be documented as a workstation option but is not hard-coded as a
|
||||
product dependency.
|
||||
|
||||
The local profile binds user-facing ports to `127.0.0.1`. The server profile exposes only the
|
||||
frontend host port needed by a generic reverse proxy. Examples for Nginx and Caddy document TLS,
|
||||
forwarded identity, websocket/SSE behavior, and timeouts, but neither proxy is embedded into or
|
||||
required by the core architecture.
|
||||
|
||||
## Persistence
|
||||
|
||||
Named volumes are the portable default for workstation installations. Server documentation shows
|
||||
explicit bind mounts under an operator-selected root such as `/srv/thothii`, with UID/GID and
|
||||
backup requirements. The same container paths are used in both profiles:
|
||||
|
||||
- `/data/workspace-registry` for Git checkout, immutable snapshots, leases, state, and locks;
|
||||
- `/data/settings` for application settings;
|
||||
- `/data/sessions` for filesystem sessions when selected;
|
||||
- `/home/thoth/.pi` for Pi runtime state;
|
||||
- `/run/secrets` for read-only secret files.
|
||||
|
||||
Shared PostgreSQL session storage remains optional. It is required only when multiple ThothII
|
||||
installations must see and resume the same sessions.
|
||||
|
||||
## Cross-platform source and line endings
|
||||
|
||||
The repository gains a root `.gitattributes` that makes line endings deterministic independently
|
||||
of a developer's global Git configuration:
|
||||
|
||||
```gitattributes
|
||||
* text=auto
|
||||
*.sh text eol=lf
|
||||
Dockerfile* text eol=lf
|
||||
*.Dockerfile text eol=lf
|
||||
*.yml text eol=lf
|
||||
*.yaml text eol=lf
|
||||
*.json text eol=lf
|
||||
*.ts text eol=lf
|
||||
*.tsx text eol=lf
|
||||
*.py text eol=lf
|
||||
*.md text eol=lf
|
||||
*.ps1 text eol=crlf
|
||||
```
|
||||
|
||||
Executable shell scripts keep their executable bit and LF bytes. CI and a local verification
|
||||
script scan Docker entrypoints, shell scripts, Compose/YAML, and Dockerfiles for carriage returns.
|
||||
The Docker build also fails early with a clear message if an executable copied into an image has
|
||||
CRLF. Documentation covers Git for Windows and WSL2, recommends cloning inside the WSL filesystem
|
||||
for Linux-container work, and provides a safe one-time renormalization procedure for existing
|
||||
clones. The documented process does not require changing global `core.autocrlf`.
|
||||
|
||||
## Build and release behavior
|
||||
|
||||
The local build uses the repository checkout as a BuildKit context and builds both images with
|
||||
blocking TypeScript checks. The current non-blocking frontend typecheck is changed into a build
|
||||
gate. A validation command renders each supported Compose combination before build. Image tags
|
||||
include a local default and may be overridden with a release version or digest.
|
||||
|
||||
Build inputs exclude `.git`, worktrees, local `.env` files, secrets, test output, caches, and
|
||||
workspace runtime data through `.dockerignore`. Builds must work from macOS, Windows/WSL2, and
|
||||
Linux without host-language runtimes beyond Docker, Compose, and Git.
|
||||
|
||||
## Migration from the current deployment
|
||||
|
||||
The current PSD/portal-oriented root Compose is replaced by the portable base. Reusable settings
|
||||
from existing local and production overrides are folded into the new local/server overrides.
|
||||
Portal network aliases, absolute Chirone paths, external `localllm_default`, PSD evidence mounts,
|
||||
and `/datamart-builder` build arguments are removed from the product defaults.
|
||||
|
||||
Existing operators migrate by copying only intentional values into the new environment and secret
|
||||
files, rendering Compose, backing up volumes, building the new stack, and validating health and
|
||||
workspace-registry status before switching the proxy. Legacy Compose examples remain in an
|
||||
archive or are removed only after their replacement documentation and migration checks exist.
|
||||
|
||||
## Error handling and operability
|
||||
|
||||
Compose rendering fails when required non-secret values are absent. Entrypoints report stable,
|
||||
sanitized errors for unreadable secret files, invalid registry configuration, incompatible line
|
||||
endings, and storage permissions. Health checks distinguish process liveness from registry and
|
||||
connector readiness. Remote outages preserve the last valid workspace snapshot as already
|
||||
specified by the Git registry design.
|
||||
|
||||
## Documentation
|
||||
|
||||
Two complete guides are maintained against the same architecture:
|
||||
|
||||
- local PC/Mac installation: Docker Desktop/Engine prerequisites, Windows/WSL2 line endings,
|
||||
clone, environment creation, local build, first start, browser URL, update, backup, and recovery;
|
||||
- server installation: service account, directories, firewall, generic reverse proxy, environment
|
||||
and secrets, local image build or pinned image use, startup, health, upgrade, rollback, backup,
|
||||
and recovery.
|
||||
|
||||
Both guides use copy-pastable commands validated by scripts. They explain which configuration is
|
||||
shared in Git, which is installation-local, and how a server that only provides DWH/VectorDB is
|
||||
consumed without installing ThothII there.
|
||||
|
||||
## Verification strategy
|
||||
|
||||
Automated gates cover Compose rendering for local/server plus each Git transport, LF enforcement,
|
||||
Docker builds, container health, frontend-to-core same-origin routing, registry bootstrap and
|
||||
offline fallback, secret non-disclosure, and absence of PSD/Chirone/portal dependencies in active
|
||||
deployment files. Existing backend, frontend, harness, registry, and installation-document tests
|
||||
remain required.
|
||||
|
||||
Manual acceptance covers a clean macOS build, a clean Windows Docker Desktop/WSL2 build from a
|
||||
GitHub clone, a Linux server deployment behind a generic proxy, an update after `git pull`, volume
|
||||
persistence, and restoration from backup.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Bundling a DWH, production VectorDB, embedding server, LLM server, or reverse proxy into the
|
||||
mandatory ThothII stack.
|
||||
- Making PSD, Chirone, or `omics_portal` supported product profiles.
|
||||
- Synchronizing local filesystem sessions between installations without shared session storage.
|
||||
- Implementing persistent runtime SSH tunnels for workspace connectors in this migration.
|
||||
Reference in New Issue
Block a user