688 lines
30 KiB
Markdown
688 lines
30 KiB
Markdown
# Unified Docker Compose Deployment Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||
|
||
**Goal:** Convert ThothII into one autonomous Docker Compose distribution for Windows, macOS, and Linux servers, with external configurable data/AI services, embedded Pi, guided Pi management, reproducible local builds, and deterministic line endings.
|
||
|
||
**Architecture:** `compose.yaml` is the sole complete stack definition; local and server files are small overrides. The mandatory stack contains only `frontend` and `core`; DWH, VectorDB, embedding, and LLM remain independently operated endpoints even when co-resident. Pi is pinned inside `core`; a Go `thothctl` executable wraps host-side Compose operations, while a web Pi Management page handles safe configuration and diagnostics.
|
||
|
||
**Tech Stack:** Docker BuildKit/Compose v2, Node.js 22, TypeScript/Fastify, React/Vite, nginx-unprivileged, Python 3.12 harness, Go 1.24 for `thothctl`, Vitest, Playwright, shell/PowerShell verification.
|
||
|
||
**Design:** `docs/superpowers/specs/2026-08-04-unified-compose-deployment-design.md`
|
||
|
||
## Global Constraints
|
||
|
||
- ThothII has no active build/runtime dependency on PSD, Chirone, or `omics_portal`.
|
||
- The mandatory stack contains only `frontend` and `core`; it never starts DWH, VectorDB, embedding, LLM, or a reverse proxy.
|
||
- External services use installation/workspace addresses even when they run on the Docker host.
|
||
- Pi is pinned inside `core`; no host Pi or host Node/Python/Go runtime is required.
|
||
- `core` never mounts a Docker daemon endpoint.
|
||
- Secrets remain file-mounted under `/run/secrets` and never enter Git, images, browser storage, rendered Compose, or logs.
|
||
- Local ports bind to `127.0.0.1`; server deployment publishes only the frontend port.
|
||
- Shell, YAML, Dockerfile, JSON, TypeScript, Python, and Markdown use LF; PowerShell uses CRLF.
|
||
- Every task follows red-green TDD and ends with a reviewable commit.
|
||
|
||
## Target file map
|
||
|
||
- `.gitattributes`, `.editorconfig`, `scripts/verify-line-endings.sh`: line-ending contract.
|
||
- `compose.yaml`, `deploy/compose.local.yaml`, `deploy/compose.server.yaml`: portable stack and profiles.
|
||
- `deploy/compose.git-*.yaml`, `deploy/compose.connector-secrets.yaml`: optional secret mounts.
|
||
- `.env.example`, `deploy/env/*.env.example`: non-secret operator contracts.
|
||
- `docker/core.Dockerfile`, `docker/frontend.Dockerfile`, `docker/nginx.conf.template`: reproducible images and same-origin routing.
|
||
- `tools/thothctl/`: cross-platform host control executable.
|
||
- `backend/src/pi/management.ts`, `backend/src/routes/pi-management.ts`: sanitized Pi APIs.
|
||
- `frontend/src/api/pi-management.ts`, `frontend/src/shell/PiManagement.tsx`: Pi operator UI.
|
||
- `docs/install/local.md`, `docs/install/server.md`, `docs/install/pi-management.md`: installation manuals.
|
||
|
||
---
|
||
|
||
### Task 1: Enforce deterministic line endings
|
||
|
||
**Files:**
|
||
- Create: `.gitattributes`
|
||
- Create: `.editorconfig`
|
||
- Create: `scripts/verify-line-endings.sh`
|
||
- Create: `scripts/test-verify-line-endings.sh`
|
||
- Modify: `docker/core.Dockerfile`
|
||
- Test: `scripts/test-verify-line-endings.sh`
|
||
|
||
**Interfaces:**
|
||
- Produces: `scripts/verify-line-endings.sh [root]`, exit `0` when compliant and `1` with offending paths for CRLF.
|
||
- Consumes: tracked files at repository root or an explicit fixture root; never scans volumes or secrets.
|
||
|
||
- [ ] **Step 1: Write the failing verifier test**
|
||
|
||
Create a temporary fixture with LF `ok.sh` and CRLF `bad.sh`, `compose.yaml`, and `Dockerfile`. Assert all bad paths are reported, `ok.sh` is absent, and LF-only input exits `0`.
|
||
|
||
```sh
|
||
fixture_root="$(mktemp -d)"
|
||
trap 'rm -rf "$fixture_root"' EXIT
|
||
printf '#!/bin/sh\r\nexit 0\r\n' > "$fixture_root/bad.sh"
|
||
if ./scripts/verify-line-endings.sh "$fixture_root"; then
|
||
echo "expected CRLF rejection" >&2
|
||
exit 1
|
||
fi
|
||
```
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `bash scripts/test-verify-line-endings.sh`
|
||
|
||
Expected: failure because the verifier does not exist.
|
||
|
||
- [ ] **Step 3: Add attributes, editor settings, and verifier**
|
||
|
||
Use this `.gitattributes` contract:
|
||
|
||
```gitattributes
|
||
* text=auto
|
||
*.sh text eol=lf
|
||
Dockerfile* text eol=lf
|
||
*.Dockerfile text eol=lf
|
||
*.yml text eol=lf
|
||
*.yaml text eol=lf
|
||
*.json text eol=lf
|
||
*.ts text eol=lf
|
||
*.tsx text eol=lf
|
||
*.py text eol=lf
|
||
*.md text eol=lf
|
||
*.ps1 text eol=crlf
|
||
```
|
||
|
||
Use `git ls-files` for the repository and `find` only for fixture mode. Detect carriage returns with `LC_ALL=C grep -Il $'\r'`. Add an image-build check before `chmod` of Docker scripts.
|
||
|
||
- [ ] **Step 4: Renormalize and verify**
|
||
|
||
Run:
|
||
|
||
```sh
|
||
git add --renormalize .
|
||
bash scripts/test-verify-line-endings.sh
|
||
bash scripts/verify-line-endings.sh
|
||
git diff --check
|
||
```
|
||
|
||
Expected: both scripts pass; review renormalized files to confirm line-ending-only changes.
|
||
|
||
- [ ] **Step 5: Commit**
|
||
|
||
```sh
|
||
git add .gitattributes .editorconfig scripts/verify-line-endings.sh scripts/test-verify-line-endings.sh docker/core.Dockerfile
|
||
git commit -m "build: enforce portable line endings"
|
||
```
|
||
|
||
### Task 2: Define the portable Compose contract
|
||
|
||
**Files:**
|
||
- Modify: `compose.yaml`
|
||
- Modify: `deploy/compose.local.yaml`
|
||
- Create: `deploy/compose.server.yaml`
|
||
- Create: `deploy/env/local.env.example`
|
||
- Create: `deploy/env/server.env.example`
|
||
- Create: `.env.example`
|
||
- Modify: `scripts/test-default-compose.sh`
|
||
- Create: `scripts/test-unified-compose.sh`
|
||
|
||
**Interfaces:**
|
||
- Produces: base services `core` and `frontend`, network `thothii`, and volumes `settings`, `pi-state`, `workspace-registry`, `sessions`.
|
||
- Produces: supported pairs base+local and base+server.
|
||
|
||
- [ ] **Step 1: Write failing structural assertions**
|
||
|
||
Render both profiles as JSON and assert:
|
||
|
||
```js
|
||
const services = Object.keys(config.services).sort();
|
||
if (services.join(",") !== "core,frontend") throw new Error("mandatory stack must be core,frontend");
|
||
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
|
||
throw new Error("forbidden application coupling");
|
||
}
|
||
```
|
||
|
||
Assert local publishes loopback frontend and optional loopback core; server publishes frontend only.
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `bash scripts/test-unified-compose.sh`
|
||
|
||
Expected: failure on current portal networks and host paths.
|
||
|
||
- [ ] **Step 3: Replace root Compose with the portable base**
|
||
|
||
Define only `core`, `frontend`, private network, health checks, and named volumes. `depends_on` may connect frontend to healthy core but never external services. Keep endpoint variables generic and secret-free.
|
||
|
||
- [ ] **Step 4: Add local/server overrides**
|
||
|
||
Local: `AUTH_MODE=none`, loopback ports, named volumes, installation ID `local`. Server: configurable frontend bind, `AUTH_MODE=upstream`, no core host port, `THT_DATA_ROOT` mounts, installation ID `server`.
|
||
|
||
- [ ] **Step 5: Add environment examples**
|
||
|
||
Use documentation domains such as `https://dwh.example.invalid`. Assert absent `THT_WORKSPACE_GIT_REMOTE` fails rendering with that exact variable name.
|
||
|
||
- [ ] **Step 6: Verify and commit**
|
||
|
||
```sh
|
||
bash scripts/test-default-compose.sh
|
||
bash scripts/test-unified-compose.sh
|
||
docker compose -f compose.yaml -f deploy/compose.local.yaml config --quiet
|
||
docker compose -f compose.yaml -f deploy/compose.server.yaml config --quiet
|
||
git add compose.yaml deploy/compose.local.yaml deploy/compose.server.yaml deploy/env .env.example scripts/test-default-compose.sh scripts/test-unified-compose.sh
|
||
git commit -m "deploy: unify local and server compose stack"
|
||
```
|
||
|
||
### Task 3: Isolate Git and connector secrets
|
||
|
||
**Files:**
|
||
- Create: `deploy/compose.git-ssh.yaml`
|
||
- Create: `deploy/compose.git-https.yaml`
|
||
- Create: `deploy/compose.connector-secrets.yaml`
|
||
- Modify: `deploy/workspace-registry.env.example`
|
||
- Create: `scripts/test-compose-secret-policy.sh`
|
||
- Modify: `scripts/verify-workspace-install-docs.sh`
|
||
|
||
**Interfaces:**
|
||
- Produces: mutually exclusive Git overrides and explicit connector targets under `/run/secrets`.
|
||
- Consumes: `THT_WS_*_FILE` and host-only `*_SOURCE` variables.
|
||
|
||
- [ ] **Step 1: Write failing rendered-secret tests**
|
||
|
||
Assert base has no `/dev/null` mounts; SSH mounts only key/known-hosts; HTTPS mounts only credentials/CA; connector mounts match declared `_FILE` targets; rendered output never contains fixture secret values.
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `bash scripts/test-compose-secret-policy.sh`
|
||
|
||
Expected: failure because secret mounts currently live in the portal-oriented base.
|
||
|
||
- [ ] **Step 3: Implement overrides**
|
||
|
||
Use read-only mounts and `${VAR:?message}` only in selected overrides. Keep strict SSH host checking and HTTPS CA verification. Reject relative/non-normalized `_SOURCE` paths.
|
||
|
||
- [ ] **Step 4: Verify and commit**
|
||
|
||
```sh
|
||
bash scripts/test-compose-secret-policy.sh
|
||
bash scripts/test-verify-workspace-install-docs.sh
|
||
bash scripts/workspace-registry-smoke.sh
|
||
git add deploy/compose.git-ssh.yaml deploy/compose.git-https.yaml deploy/compose.connector-secrets.yaml deploy/workspace-registry.env.example scripts/test-compose-secret-policy.sh scripts/verify-workspace-install-docs.sh
|
||
git commit -m "deploy: isolate git and connector secrets"
|
||
```
|
||
|
||
### Task 4: Make frontend-to-core routing same-origin
|
||
|
||
**Files:**
|
||
- Modify: `docker/nginx.conf.template`
|
||
- Modify: `docker/frontend-entrypoint.sh`
|
||
- Modify: `docker/frontend.Dockerfile`
|
||
- Modify: `frontend/src/api/runtime-config.ts`
|
||
- Test: `frontend/src/api/runtime-config.test.ts`
|
||
- Modify: `docker/smoke/frontend-policy-smoke.sh`
|
||
- Modify: `scripts/test-backend-url-policy.sh`
|
||
|
||
**Interfaces:**
|
||
- Produces: browser base `/api`; nginx proxies to `http://core:8787` privately.
|
||
- Consumes: optional internal `THT_FRONTEND_API_UPSTREAM` only.
|
||
|
||
- [ ] **Step 1: Write failing routing tests**
|
||
|
||
Assert `/api` default, rejection of browser-facing absolute production URLs, and nginx SSE settings:
|
||
|
||
```nginx
|
||
proxy_http_version 1.1;
|
||
proxy_buffering off;
|
||
proxy_read_timeout 3600s;
|
||
```
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
```sh
|
||
npm --prefix frontend test -- --run src/api/runtime-config.test.ts
|
||
bash scripts/test-backend-url-policy.sh
|
||
```
|
||
|
||
Expected: failure because deployment-specific build arguments remain.
|
||
|
||
- [ ] **Step 3: Implement runtime routing and blocking frontend build**
|
||
|
||
Build once with `/` assets and `/api`. Render private upstream at startup, strip `/api`, preserve SSE. Replace non-blocking typecheck with `RUN npm run build`. Remove `/datamart-builder` assumptions.
|
||
|
||
- [ ] **Step 4: Verify and commit**
|
||
|
||
```sh
|
||
npm --prefix frontend test -- --run src/api/runtime-config.test.ts
|
||
npm --prefix frontend run build
|
||
bash scripts/test-backend-url-policy.sh
|
||
bash docker/smoke/frontend-policy-smoke.sh
|
||
git add docker/nginx.conf.template docker/frontend-entrypoint.sh docker/frontend.Dockerfile docker/smoke/frontend-policy-smoke.sh frontend/src/api/runtime-config.ts frontend/src/api/runtime-config.test.ts scripts/test-backend-url-policy.sh
|
||
git commit -m "deploy: route frontend and core through one origin"
|
||
```
|
||
|
||
### Task 5: Harden local image builds and embedded Pi
|
||
|
||
**Files:**
|
||
- Modify: `docker/core.Dockerfile`
|
||
- Modify: `docker/pi-runtime/package.json`
|
||
- Modify: `docker/pi-runtime/package-lock.json`
|
||
- Create: `.dockerignore`
|
||
- Modify: `docker/smoke/core-smoke.sh`
|
||
- Modify: `scripts/verify-container-images.sh`
|
||
- Create: `scripts/build-local.sh`
|
||
- Create: `scripts/build-local.ps1`
|
||
- Test: `scripts/test-container-deployment.sh`
|
||
|
||
**Interfaces:**
|
||
- Produces: `core` containing the pinned `/usr/local/bin/pi` and a standalone `frontend` image.
|
||
- Produces: local build launchers requiring only Docker, Compose, and Git.
|
||
|
||
- [ ] **Step 1: Write failing image-contract assertions**
|
||
|
||
Inside `core`, assert `pi --version` matches `PI_VERSION`, UID is `10001`, Docker socket is absent, `/data` is writable, and no portal/Chirone path exists. Assert frontend health and `/api` routing.
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `bash scripts/test-container-deployment.sh`
|
||
|
||
Expected: failure on at least one old deployment contract.
|
||
|
||
- [ ] **Step 3: Pin Pi from its lock input**
|
||
|
||
Make `docker/pi-runtime/package-lock.json` the sole Pi dependency lock. Install with `npm ci --omit=dev` in a build stage, copy into `core`, and fail build when `pi --version` differs from `PI_VERSION`.
|
||
|
||
- [ ] **Step 4: Add exclusions and platform launchers**
|
||
|
||
Exclude `.git`, `.worktrees`, `.env`, secrets, dependencies, virtual environments, coverage, and runtime data. Both launchers run:
|
||
|
||
```text
|
||
docker compose -f compose.yaml -f deploy/compose.local.yaml build --pull
|
||
```
|
||
|
||
They print the same next command and preserve Docker's exit code.
|
||
|
||
- [ ] **Step 5: Verify and commit**
|
||
|
||
```sh
|
||
bash scripts/build-local.sh
|
||
bash scripts/test-container-deployment.sh
|
||
bash scripts/verify-container-images.sh
|
||
git add .dockerignore docker/core.Dockerfile docker/pi-runtime docker/smoke/core-smoke.sh scripts/build-local.sh scripts/build-local.ps1 scripts/test-container-deployment.sh scripts/verify-container-images.sh
|
||
git commit -m "build: make embedded pi images reproducible"
|
||
```
|
||
|
||
Windows gate: `powershell -ExecutionPolicy Bypass -File scripts/build-local.ps1`.
|
||
|
||
### Task 6: Build the cross-platform `thothctl` foundation
|
||
|
||
**Files:**
|
||
- Create: `tools/thothctl/go.mod`
|
||
- Create: `tools/thothctl/cmd/thothctl/main.go`
|
||
- Create: `tools/thothctl/internal/compose/runner.go`
|
||
- Create: `tools/thothctl/internal/config/installation.go`
|
||
- Create: `tools/thothctl/internal/output/sanitize.go`
|
||
- Test: `tools/thothctl/internal/compose/runner_test.go`
|
||
- Test: `tools/thothctl/internal/config/installation_test.go`
|
||
- Test: `tools/thothctl/internal/output/sanitize_test.go`
|
||
- Create: `docker/thothctl.Dockerfile`
|
||
- Create: `scripts/build-thothctl.sh`
|
||
|
||
**Interfaces:**
|
||
- Produces: `thothctl --installation <absolute-path> <command>` binaries for Windows amd64, macOS amd64/arm64, Linux amd64/arm64.
|
||
- Produces: `Runner.Run(ctx, args, stdin) (Result, error)` using argument arrays, never shell concatenation.
|
||
|
||
- [ ] **Step 1: Write failing tests**
|
||
|
||
Cover local/server Compose selection, paths containing spaces, missing Docker, propagated exit codes, and replacement of values matching password/token/key fields or secret-file contents with `[REDACTED]`.
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./...`
|
||
|
||
Expected: failure because packages do not exist.
|
||
|
||
- [ ] **Step 3: Implement installation discovery and safe runner**
|
||
|
||
Read `thothii-installation.yaml` fields `profile`, `projectDirectory`, `envFile`, and `overrides`. Resolve and validate absolute paths. Invoke `docker compose` with `exec.CommandContext` argument slices.
|
||
|
||
- [ ] **Step 4: Implement base commands**
|
||
|
||
Add `status`, `doctor`, `logs`, `start`, `stop`, and `update --check-only`. `doctor` validates Docker/Compose versions, rendered config, LF, volumes, and frontend/core health without printing environment values.
|
||
|
||
- [ ] **Step 5: Cross-compile with Docker and verify**
|
||
|
||
Produce `dist/thothctl/thothctl-windows-amd64.exe`, Darwin amd64/arm64, and Linux amd64/arm64 from `docker/thothctl.Dockerfile`.
|
||
|
||
```sh
|
||
docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./...
|
||
bash scripts/build-thothctl.sh
|
||
```
|
||
|
||
Run the host-matching binary with `--help`, then commit:
|
||
|
||
```sh
|
||
git add tools/thothctl docker/thothctl.Dockerfile scripts/build-thothctl.sh
|
||
git commit -m "feat: add cross-platform thothctl"
|
||
```
|
||
|
||
### Task 7: Implement safe Pi lifecycle commands in `thothctl`
|
||
|
||
**Files:**
|
||
- Create: `tools/thothctl/internal/pi/commands.go`
|
||
- Create: `tools/thothctl/internal/pi/update.go`
|
||
- Create: `tools/thothctl/internal/pi/state.go`
|
||
- Test: `tools/thothctl/internal/pi/commands_test.go`
|
||
- Test: `tools/thothctl/internal/pi/update_test.go`
|
||
- Modify: `tools/thothctl/cmd/thothctl/main.go`
|
||
- Create: `docs/contracts/tht-pi.md`
|
||
|
||
**Interfaces:**
|
||
- Produces: `pi status|doctor|configure|test|update|logs`.
|
||
- Produces: `.thothctl/update-state.json` with previous/new image references and phase, never credentials.
|
||
- Consumes: existing `/health`, `/models`, and `/settings` APIs plus `docker compose exec core pi --version`; Task 8 replaces the temporary composite checks with the dedicated Pi Management API.
|
||
|
||
- [ ] **Step 1: Write failing update-state tests**
|
||
|
||
With a fake Compose runner cover success and failure during build/pull, recreate, health, version, and smoke. Assert every post-recreate failure restores the prior image and leaves volume names unchanged.
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./internal/pi -v`
|
||
|
||
Expected: failure because Pi commands are absent.
|
||
|
||
- [ ] **Step 3: Implement read-only commands**
|
||
|
||
`status` runs `core pi --version`; `doctor` verifies image version, writable Pi volume, provider configuration presence, and `/health`; `test` combines `/models`, `/settings`, and a fixed `core pi --version` probe until Task 8 supplies the dedicated smoke endpoint; `logs` uses the shared sanitizer.
|
||
|
||
- [ ] **Step 4: Implement guided configuration**
|
||
|
||
Offer provider/model/reasoning choices returned by the backend. Atomically write non-secret defaults. For credentials print the expected secret filename and permissions; never accept secret text as an argument.
|
||
|
||
- [ ] **Step 5: Implement transactional update**
|
||
|
||
Record current image ID, pull/build requested pinned version, recreate only `core`, verify health/version/test, and roll back on failure. Refuse update while sessions are active unless `--drain` completes.
|
||
|
||
- [ ] **Step 6: Verify and commit**
|
||
|
||
```sh
|
||
docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./internal/pi -v
|
||
docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./...
|
||
git add tools/thothctl docs/contracts/tht-pi.md
|
||
git commit -m "feat: manage embedded pi with thothctl"
|
||
```
|
||
|
||
### Task 8: Add sanitized Pi Management backend APIs
|
||
|
||
**Files:**
|
||
- Create: `backend/src/pi/management.ts`
|
||
- Create: `backend/src/routes/pi-management.ts`
|
||
- Modify: `backend/src/app.ts`
|
||
- Modify: `backend/src/config.ts`
|
||
- Modify: `tools/thothctl/internal/pi/commands.go`
|
||
- Test: `backend/test/pi-management.test.ts`
|
||
- Test: `backend/test/routes-pi-management.test.ts`
|
||
- Test: `tools/thothctl/internal/pi/commands_test.go`
|
||
|
||
**Interfaces:**
|
||
- Produces: `GET /pi-management/status`, `GET /pi-management/options`, `PUT /pi-management/config`, `POST /pi-management/test`, `GET /pi-management/logs`.
|
||
- Produces: sanitized `PiStatus`, `PiOptions`, `PiInstallationConfig`, and stable errors.
|
||
|
||
- [ ] **Step 1: Write failing service/route tests**
|
||
|
||
Mock process execution and settings. Assert version parsing, closed choices, free-field validation, atomic writes, smoke timeout, 200-line log limit, redaction, and `403 pi_management_forbidden` in exposed server mode without trusted admin identity.
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `cd backend && npx vitest run test/pi-management.test.ts test/routes-pi-management.test.ts`
|
||
|
||
Expected: module-not-found failure.
|
||
|
||
- [ ] **Step 3: Implement service and authorization**
|
||
|
||
Use `execFile` with fixed argument arrays. Return only version, readiness, provider names, model IDs, reasoning choices, timestamps, and sanitized messages. Allow `AUTH_MODE=none` only when public exposure is false; upstream mode requires the documented admin claim. Expose no image-update API.
|
||
|
||
- [ ] **Step 4: Switch `thothctl` to the dedicated API**
|
||
|
||
Replace the Task 7 composite `/health`/`/models`/`/settings` test with `POST /pi-management/test`; load closed configuration choices from `GET /pi-management/options`. Preserve the direct in-container `pi --version` check as an independent image-integrity signal.
|
||
|
||
- [ ] **Step 5: Verify and commit**
|
||
|
||
```sh
|
||
cd backend
|
||
npx vitest run test/pi-management.test.ts test/routes-pi-management.test.ts
|
||
npx vitest run
|
||
npx tsc --noEmit -p .
|
||
cd ..
|
||
docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./internal/pi -v
|
||
git add backend/src/pi/management.ts backend/src/routes/pi-management.ts backend/src/app.ts backend/src/config.ts backend/test/pi-management.test.ts backend/test/routes-pi-management.test.ts tools/thothctl/internal/pi/commands.go tools/thothctl/internal/pi/commands_test.go
|
||
git commit -m "feat: expose safe pi management api"
|
||
```
|
||
|
||
### Task 9: Add the Pi Management interface
|
||
|
||
**Files:**
|
||
- Create: `frontend/src/api/pi-management.ts`
|
||
- Create: `frontend/src/api/pi-management.test.ts`
|
||
- Create: `frontend/src/shell/PiManagement.tsx`
|
||
- Create: `frontend/src/shell/PiManagement.test.tsx`
|
||
- Modify: `frontend/src/shell/AppShell.tsx`
|
||
- Modify: `frontend/src/shell/AppShell.session-mgmt.test.tsx`
|
||
|
||
**Interfaces:**
|
||
- Consumes: Task 8 Pi APIs.
|
||
- Produces: right-sidebar Pi Management panel without image-update execution or browser terminal.
|
||
|
||
- [ ] **Step 1: Write failing UI tests**
|
||
|
||
Cover loading/version, closed selects, validation, save, smoke test, sanitized logs, forbidden state, and copyable `thothctl pi update` instruction. Assert no secret input or browser terminal.
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `cd frontend && npx vitest run src/shell/PiManagement.test.tsx src/api/pi-management.test.ts`
|
||
|
||
Expected: module-not-found failure.
|
||
|
||
- [ ] **Step 3: Implement API and panel**
|
||
|
||
Use query keys `['pi-management','status']` and `['pi-management','options']`. Render provider/model/reasoning as closed choices, validate numeric fields before PUT, and show credentials only as present/missing.
|
||
|
||
- [ ] **Step 4: Attach to the right sidebar**
|
||
|
||
Add Pi next to Workspace Management, preserve current session/activity panels and accessibility, and remove AppShell comments/layout assumptions about Omics Portal chrome.
|
||
|
||
- [ ] **Step 5: Verify and commit**
|
||
|
||
```sh
|
||
cd frontend
|
||
npx vitest run src/shell/PiManagement.test.tsx src/api/pi-management.test.ts src/shell/AppShell.session-mgmt.test.tsx
|
||
npx vitest run
|
||
npx tsc -b
|
||
git add src/api/pi-management.ts src/api/pi-management.test.ts src/shell/PiManagement.tsx src/shell/PiManagement.test.tsx src/shell/AppShell.tsx src/shell/AppShell.session-mgmt.test.tsx
|
||
git commit -m "feat: add pi management interface"
|
||
```
|
||
|
||
### Task 10: Remove active PSD and portal deployment coupling
|
||
|
||
**Files:**
|
||
- Delete: `deploy/compose.production.yaml`
|
||
- Delete: `deploy/compose.psd-local.yaml.example`
|
||
- Modify: `README.md`
|
||
- Modify: `PROJECT_STATE.md`
|
||
- Modify: `scripts/run-stack.sh`
|
||
- Create: `scripts/test-no-deployment-coupling.sh`
|
||
- Modify: `scripts/test-external-compose-lifecycle.sh`
|
||
|
||
**Interfaces:**
|
||
- Produces: active deployment files free of PSD/Chirone/portal networks, paths, and prefixes.
|
||
- Preserves: generic external endpoint support and historical design documents.
|
||
|
||
- [ ] **Step 1: Write the failing coupling scan**
|
||
|
||
Scan active Compose, Docker, root README, install guides, env examples, and run scripts for `omics_portal`, `/home/chirone`, `localllm_default`, `datamart-builder`, and PSD deployment filenames. Exclude historical specs/plans and canonical workspace content.
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `bash scripts/test-no-deployment-coupling.sh`
|
||
|
||
Expected: failure listing current portal-oriented files.
|
||
|
||
- [ ] **Step 3: Remove superseded files and genericize launch behavior**
|
||
|
||
Delete only deployment files replaced by Tasks 2–5. Make `run-stack.sh` call base+local or direct users to `thothctl start`. Preserve data migration utilities that do not affect runtime coupling.
|
||
|
||
- [ ] **Step 4: Update root documentation and verify**
|
||
|
||
State that co-location never puts DWH/vector/embedding inside ThothII.
|
||
|
||
```sh
|
||
bash scripts/test-no-deployment-coupling.sh
|
||
bash scripts/test-unified-compose.sh
|
||
bash scripts/test-external-compose-lifecycle.sh
|
||
git add -A deploy/compose.production.yaml deploy/compose.psd-local.yaml.example README.md PROJECT_STATE.md scripts/run-stack.sh scripts/test-no-deployment-coupling.sh scripts/test-external-compose-lifecycle.sh
|
||
git commit -m "refactor: remove portal deployment coupling"
|
||
```
|
||
|
||
### Task 11: Write and verify the PC/Mac installation manual
|
||
|
||
**Files:**
|
||
- Create: `docs/install/local.md`
|
||
- Create: `docs/install/windows-line-endings.md`
|
||
- Create: `docs/install/pi-management.md`
|
||
- Create: `docs/install/examples/thothii-installation.local.yaml`
|
||
- Modify: `docs/install/local-workspace-registry.md`
|
||
- Modify: `scripts/verify-workspace-install-docs.sh`
|
||
- Modify: `scripts/test-verify-workspace-install-docs.sh`
|
||
|
||
**Interfaces:**
|
||
- Produces: complete clean-install, build, update, backup, restore, Pi, and CRLF instructions for Windows/WSL2, macOS, and Linux PC.
|
||
|
||
- [ ] **Step 1: Extend the verifier first**
|
||
|
||
Require prerequisites, Git clone, LF verification, `.env`, external-service addressing, build, start, health, browser URL, `thothctl`, Pi update/rollback, backup/restore, pull update, and data-preserving uninstall. Require `pi-management.md` to cover UI permissions, every `thothctl pi` command, secret handling, direct support access, and failed-update recovery. Copy examples to a temporary path containing spaces and render there.
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `bash scripts/test-verify-workspace-install-docs.sh`
|
||
|
||
Expected: failure naming missing local-guide sections.
|
||
|
||
- [ ] **Step 3: Write platform-specific instructions**
|
||
|
||
Document macOS, Windows PowerShell, Windows WSL2, and Linux separately. Recommend cloning inside the WSL filesystem. Include LF checks after clone/pull. For an existing CRLF clone, prefer recloning; describe repository-local `core.autocrlf=false` and renormalization. If mentioning `git reset --hard`, require explicit backup/commit and a destructive-action warning immediately before it.
|
||
|
||
- [ ] **Step 4: Document external services on the host**
|
||
|
||
Show `host.docker.internal` for Docker Desktop and `extra_hosts: host.docker.internal:host-gateway` for Linux. Explain that container `127.0.0.1` is not the host. All addresses remain configurable.
|
||
|
||
- [ ] **Step 5: Verify and commit**
|
||
|
||
```sh
|
||
bash scripts/test-verify-workspace-install-docs.sh
|
||
bash scripts/verify-workspace-install-docs.sh --profile local
|
||
bash scripts/docker-smoke.sh
|
||
git add docs/install/local.md docs/install/windows-line-endings.md docs/install/pi-management.md docs/install/examples/thothii-installation.local.yaml docs/install/local-workspace-registry.md scripts/verify-workspace-install-docs.sh scripts/test-verify-workspace-install-docs.sh
|
||
git commit -m "docs: add autonomous local installation guide"
|
||
```
|
||
|
||
### Task 12: Write and verify the server installation manual
|
||
|
||
**Files:**
|
||
- Create: `docs/install/server.md`
|
||
- Create: `docs/install/reverse-proxy-nginx.md`
|
||
- Create: `docs/install/reverse-proxy-caddy.md`
|
||
- Create: `docs/install/examples/thothii-installation.server.yaml`
|
||
- Modify: `docs/install/server-workspace-registry.md`
|
||
- Modify: `scripts/verify-workspace-install-docs.sh`
|
||
- Test: `scripts/test-verify-workspace-install-docs.sh`
|
||
|
||
**Interfaces:**
|
||
- Produces: generic Linux server deployment independent of another application's network.
|
||
- Consumes: server profile, secret overrides, `thothctl`, and same-origin frontend.
|
||
|
||
- [ ] **Step 1: Add failing server-guide assertions**
|
||
|
||
Require service account/UID, directories, firewall, co-resident external endpoints, DNS/host-gateway choices, generic TLS proxy, upstream auth, local build or pinned images, startup, readiness, Pi management, drain, rollback, backup, restore, and diagnostics.
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `bash scripts/test-verify-workspace-install-docs.sh`
|
||
|
||
Expected: failure naming missing server sections.
|
||
|
||
- [ ] **Step 3: Write the server and proxy guides**
|
||
|
||
Use `/srv/thothii` only as an example operator root. State that DWH/vector/embedding on the same physical machine remain independently addressed services. Nginx/Caddy proxy only to frontend, preserve SSE, terminate TLS, and forward identity only after authentication.
|
||
|
||
- [ ] **Step 4: Verify and commit**
|
||
|
||
```sh
|
||
bash scripts/test-verify-workspace-install-docs.sh
|
||
bash scripts/verify-workspace-install-docs.sh --profile server
|
||
bash scripts/test-unified-compose.sh
|
||
git add docs/install/server.md docs/install/reverse-proxy-nginx.md docs/install/reverse-proxy-caddy.md docs/install/examples/thothii-installation.server.yaml docs/install/server-workspace-registry.md scripts/verify-workspace-install-docs.sh scripts/test-verify-workspace-install-docs.sh
|
||
git commit -m "docs: add autonomous server installation guide"
|
||
```
|
||
|
||
### Task 13: Add end-to-end deployment and update gates
|
||
|
||
**Files:**
|
||
- Create: `scripts/unified-deployment-smoke.sh`
|
||
- Create: `scripts/thothctl-update-smoke.sh`
|
||
- Create: `scripts/test-windows-clone-contract.ps1`
|
||
- Create: `.github/workflows/deployment.yml`
|
||
- Modify: `PROJECT_STATE.md`
|
||
- Modify: `README.md`
|
||
|
||
**Interfaces:**
|
||
- Produces: release gate for rendering, image build, embedded Pi, registry persistence, offline recovery, and update rollback.
|
||
|
||
- [ ] **Step 1: Write failing orchestration smoke**
|
||
|
||
Create an isolated Compose project and bare Git registry, build, start local, verify frontend/core/Pi/registry, recreate offline, perform a valid Git update, and prove volumes survive. Inject a bad Pi image/version and prove rollback.
|
||
|
||
- [ ] **Step 2: Confirm red**
|
||
|
||
Run: `bash scripts/unified-deployment-smoke.sh`
|
||
|
||
Expected: failure because the script is absent.
|
||
|
||
- [ ] **Step 3: Implement exact-resource cleanup**
|
||
|
||
Generate a unique project name and temporary directory, label resources, and remove only those exact resources on exit. Never prune global Docker state. Sanitize captured logs.
|
||
|
||
- [ ] **Step 4: Add Windows and CI gates**
|
||
|
||
PowerShell scans tracked shell/YAML/Docker files for byte `0x0D`, renders Compose, and invokes Windows `thothctl`. CI runs LF and TypeScript gates everywhere, Docker smoke on Linux, and clone/Compose contract on Windows.
|
||
|
||
- [ ] **Step 5: Run final verification**
|
||
|
||
```sh
|
||
bash scripts/verify-line-endings.sh
|
||
bash scripts/test-unified-compose.sh
|
||
bash scripts/test-compose-secret-policy.sh
|
||
bash scripts/unified-deployment-smoke.sh
|
||
bash scripts/thothctl-update-smoke.sh
|
||
bash scripts/test-verify-workspace-install-docs.sh
|
||
cd backend && npx vitest run && npx tsc --noEmit -p .
|
||
cd ../frontend && npx vitest run && npx tsc -b
|
||
cd ../harness && .venv/bin/pytest -q
|
||
```
|
||
|
||
Expected: all non-L2 tests pass; Docker-dependent harness tests run on a Docker-capable host.
|
||
|
||
- [ ] **Step 6: Commit**
|
||
|
||
```sh
|
||
git add scripts/unified-deployment-smoke.sh scripts/thothctl-update-smoke.sh scripts/test-windows-clone-contract.ps1 .github/workflows/deployment.yml PROJECT_STATE.md README.md
|
||
git commit -m "test: gate unified compose deployment"
|
||
```
|
||
|
||
## Completion criteria
|
||
|
||
- Clean GitHub clones build/run on Windows Docker Desktop/WSL2 and macOS using only Docker, Compose, and Git.
|
||
- The same source/images deploy on Linux through the server override.
|
||
- Active deployment files contain no PSD, Chirone, or `omics_portal` dependency.
|
||
- DWH, VectorDB, embedding, and LLM are always configurable external endpoints.
|
||
- Pi exists in `core`, is configurable through Pi Management, and is safely updated by `thothctl`.
|
||
- Git attributes prevent CRLF and verification catches corruption before image startup.
|
||
- Local/server guides pass executable documentation checks.
|
||
- Failed updates restore the previous image while preserving registry, sessions, settings, and Pi state.
|