diff --git a/docs/superpowers/plans/2026-08-04-unified-compose-deployment.md b/docs/superpowers/plans/2026-08-04-unified-compose-deployment.md new file mode 100644 index 00000000..b9957dbf --- /dev/null +++ b/docs/superpowers/plans/2026-08-04-unified-compose-deployment.md @@ -0,0 +1,687 @@ +# Unified Docker Compose Deployment Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Convert ThothII into one autonomous Docker Compose distribution for Windows, macOS, and Linux servers, with external configurable data/AI services, embedded Pi, guided Pi management, reproducible local builds, and deterministic line endings. + +**Architecture:** `compose.yaml` is the sole complete stack definition; local and server files are small overrides. The mandatory stack contains only `frontend` and `core`; DWH, VectorDB, embedding, and LLM remain independently operated endpoints even when co-resident. Pi is pinned inside `core`; a Go `thothctl` executable wraps host-side Compose operations, while a web Pi Management page handles safe configuration and diagnostics. + +**Tech Stack:** Docker BuildKit/Compose v2, Node.js 22, TypeScript/Fastify, React/Vite, nginx-unprivileged, Python 3.12 harness, Go 1.24 for `thothctl`, Vitest, Playwright, shell/PowerShell verification. + +**Design:** `docs/superpowers/specs/2026-08-04-unified-compose-deployment-design.md` + +## Global Constraints + +- ThothII has no active build/runtime dependency on PSD, Chirone, or `omics_portal`. +- The mandatory stack contains only `frontend` and `core`; it never starts DWH, VectorDB, embedding, LLM, or a reverse proxy. +- External services use installation/workspace addresses even when they run on the Docker host. +- Pi is pinned inside `core`; no host Pi or host Node/Python/Go runtime is required. +- `core` never mounts a Docker daemon endpoint. +- Secrets remain file-mounted under `/run/secrets` and never enter Git, images, browser storage, rendered Compose, or logs. +- Local ports bind to `127.0.0.1`; server deployment publishes only the frontend port. +- Shell, YAML, Dockerfile, JSON, TypeScript, Python, and Markdown use LF; PowerShell uses CRLF. +- Every task follows red-green TDD and ends with a reviewable commit. + +## Target file map + +- `.gitattributes`, `.editorconfig`, `scripts/verify-line-endings.sh`: line-ending contract. +- `compose.yaml`, `deploy/compose.local.yaml`, `deploy/compose.server.yaml`: portable stack and profiles. +- `deploy/compose.git-*.yaml`, `deploy/compose.connector-secrets.yaml`: optional secret mounts. +- `.env.example`, `deploy/env/*.env.example`: non-secret operator contracts. +- `docker/core.Dockerfile`, `docker/frontend.Dockerfile`, `docker/nginx.conf.template`: reproducible images and same-origin routing. +- `tools/thothctl/`: cross-platform host control executable. +- `backend/src/pi/management.ts`, `backend/src/routes/pi-management.ts`: sanitized Pi APIs. +- `frontend/src/api/pi-management.ts`, `frontend/src/shell/PiManagement.tsx`: Pi operator UI. +- `docs/install/local.md`, `docs/install/server.md`, `docs/install/pi-management.md`: installation manuals. + +--- + +### Task 1: Enforce deterministic line endings + +**Files:** +- Create: `.gitattributes` +- Create: `.editorconfig` +- Create: `scripts/verify-line-endings.sh` +- Create: `scripts/test-verify-line-endings.sh` +- Modify: `docker/core.Dockerfile` +- Test: `scripts/test-verify-line-endings.sh` + +**Interfaces:** +- Produces: `scripts/verify-line-endings.sh [root]`, exit `0` when compliant and `1` with offending paths for CRLF. +- Consumes: tracked files at repository root or an explicit fixture root; never scans volumes or secrets. + +- [ ] **Step 1: Write the failing verifier test** + +Create a temporary fixture with LF `ok.sh` and CRLF `bad.sh`, `compose.yaml`, and `Dockerfile`. Assert all bad paths are reported, `ok.sh` is absent, and LF-only input exits `0`. + +```sh +fixture_root="$(mktemp -d)" +trap 'rm -rf "$fixture_root"' EXIT +printf '#!/bin/sh\r\nexit 0\r\n' > "$fixture_root/bad.sh" +if ./scripts/verify-line-endings.sh "$fixture_root"; then + echo "expected CRLF rejection" >&2 + exit 1 +fi +``` + +- [ ] **Step 2: Confirm red** + +Run: `bash scripts/test-verify-line-endings.sh` + +Expected: failure because the verifier does not exist. + +- [ ] **Step 3: Add attributes, editor settings, and verifier** + +Use this `.gitattributes` contract: + +```gitattributes +* text=auto +*.sh text eol=lf +Dockerfile* text eol=lf +*.Dockerfile text eol=lf +*.yml text eol=lf +*.yaml text eol=lf +*.json text eol=lf +*.ts text eol=lf +*.tsx text eol=lf +*.py text eol=lf +*.md text eol=lf +*.ps1 text eol=crlf +``` + +Use `git ls-files` for the repository and `find` only for fixture mode. Detect carriage returns with `LC_ALL=C grep -Il $'\r'`. Add an image-build check before `chmod` of Docker scripts. + +- [ ] **Step 4: Renormalize and verify** + +Run: + +```sh +git add --renormalize . +bash scripts/test-verify-line-endings.sh +bash scripts/verify-line-endings.sh +git diff --check +``` + +Expected: both scripts pass; review renormalized files to confirm line-ending-only changes. + +- [ ] **Step 5: Commit** + +```sh +git add .gitattributes .editorconfig scripts/verify-line-endings.sh scripts/test-verify-line-endings.sh docker/core.Dockerfile +git commit -m "build: enforce portable line endings" +``` + +### Task 2: Define the portable Compose contract + +**Files:** +- Modify: `compose.yaml` +- Modify: `deploy/compose.local.yaml` +- Create: `deploy/compose.server.yaml` +- Create: `deploy/env/local.env.example` +- Create: `deploy/env/server.env.example` +- Create: `.env.example` +- Modify: `scripts/test-default-compose.sh` +- Create: `scripts/test-unified-compose.sh` + +**Interfaces:** +- Produces: base services `core` and `frontend`, network `thothii`, and volumes `settings`, `pi-state`, `workspace-registry`, `sessions`. +- Produces: supported pairs base+local and base+server. + +- [ ] **Step 1: Write failing structural assertions** + +Render both profiles as JSON and assert: + +```js +const services = Object.keys(config.services).sort(); +if (services.join(",") !== "core,frontend") throw new Error("mandatory stack must be core,frontend"); +if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) { + throw new Error("forbidden application coupling"); +} +``` + +Assert local publishes loopback frontend and optional loopback core; server publishes frontend only. + +- [ ] **Step 2: Confirm red** + +Run: `bash scripts/test-unified-compose.sh` + +Expected: failure on current portal networks and host paths. + +- [ ] **Step 3: Replace root Compose with the portable base** + +Define only `core`, `frontend`, private network, health checks, and named volumes. `depends_on` may connect frontend to healthy core but never external services. Keep endpoint variables generic and secret-free. + +- [ ] **Step 4: Add local/server overrides** + +Local: `AUTH_MODE=none`, loopback ports, named volumes, installation ID `local`. Server: configurable frontend bind, `AUTH_MODE=upstream`, no core host port, `THT_DATA_ROOT` mounts, installation ID `server`. + +- [ ] **Step 5: Add environment examples** + +Use documentation domains such as `https://dwh.example.invalid`. Assert absent `THT_WORKSPACE_GIT_REMOTE` fails rendering with that exact variable name. + +- [ ] **Step 6: Verify and commit** + +```sh +bash scripts/test-default-compose.sh +bash scripts/test-unified-compose.sh +docker compose -f compose.yaml -f deploy/compose.local.yaml config --quiet +docker compose -f compose.yaml -f deploy/compose.server.yaml config --quiet +git add compose.yaml deploy/compose.local.yaml deploy/compose.server.yaml deploy/env .env.example scripts/test-default-compose.sh scripts/test-unified-compose.sh +git commit -m "deploy: unify local and server compose stack" +``` + +### Task 3: Isolate Git and connector secrets + +**Files:** +- Create: `deploy/compose.git-ssh.yaml` +- Create: `deploy/compose.git-https.yaml` +- Create: `deploy/compose.connector-secrets.yaml` +- Modify: `deploy/workspace-registry.env.example` +- Create: `scripts/test-compose-secret-policy.sh` +- Modify: `scripts/verify-workspace-install-docs.sh` + +**Interfaces:** +- Produces: mutually exclusive Git overrides and explicit connector targets under `/run/secrets`. +- Consumes: `THT_WS_*_FILE` and host-only `*_SOURCE` variables. + +- [ ] **Step 1: Write failing rendered-secret tests** + +Assert base has no `/dev/null` mounts; SSH mounts only key/known-hosts; HTTPS mounts only credentials/CA; connector mounts match declared `_FILE` targets; rendered output never contains fixture secret values. + +- [ ] **Step 2: Confirm red** + +Run: `bash scripts/test-compose-secret-policy.sh` + +Expected: failure because secret mounts currently live in the portal-oriented base. + +- [ ] **Step 3: Implement overrides** + +Use read-only mounts and `${VAR:?message}` only in selected overrides. Keep strict SSH host checking and HTTPS CA verification. Reject relative/non-normalized `_SOURCE` paths. + +- [ ] **Step 4: Verify and commit** + +```sh +bash scripts/test-compose-secret-policy.sh +bash scripts/test-verify-workspace-install-docs.sh +bash scripts/workspace-registry-smoke.sh +git add deploy/compose.git-ssh.yaml deploy/compose.git-https.yaml deploy/compose.connector-secrets.yaml deploy/workspace-registry.env.example scripts/test-compose-secret-policy.sh scripts/verify-workspace-install-docs.sh +git commit -m "deploy: isolate git and connector secrets" +``` + +### Task 4: Make frontend-to-core routing same-origin + +**Files:** +- Modify: `docker/nginx.conf.template` +- Modify: `docker/frontend-entrypoint.sh` +- Modify: `docker/frontend.Dockerfile` +- Modify: `frontend/src/api/runtime-config.ts` +- Test: `frontend/src/api/runtime-config.test.ts` +- Modify: `docker/smoke/frontend-policy-smoke.sh` +- Modify: `scripts/test-backend-url-policy.sh` + +**Interfaces:** +- Produces: browser base `/api`; nginx proxies to `http://core:8787` privately. +- Consumes: optional internal `THT_FRONTEND_API_UPSTREAM` only. + +- [ ] **Step 1: Write failing routing tests** + +Assert `/api` default, rejection of browser-facing absolute production URLs, and nginx SSE settings: + +```nginx +proxy_http_version 1.1; +proxy_buffering off; +proxy_read_timeout 3600s; +``` + +- [ ] **Step 2: Confirm red** + +```sh +npm --prefix frontend test -- --run src/api/runtime-config.test.ts +bash scripts/test-backend-url-policy.sh +``` + +Expected: failure because deployment-specific build arguments remain. + +- [ ] **Step 3: Implement runtime routing and blocking frontend build** + +Build once with `/` assets and `/api`. Render private upstream at startup, strip `/api`, preserve SSE. Replace non-blocking typecheck with `RUN npm run build`. Remove `/datamart-builder` assumptions. + +- [ ] **Step 4: Verify and commit** + +```sh +npm --prefix frontend test -- --run src/api/runtime-config.test.ts +npm --prefix frontend run build +bash scripts/test-backend-url-policy.sh +bash docker/smoke/frontend-policy-smoke.sh +git add docker/nginx.conf.template docker/frontend-entrypoint.sh docker/frontend.Dockerfile docker/smoke/frontend-policy-smoke.sh frontend/src/api/runtime-config.ts frontend/src/api/runtime-config.test.ts scripts/test-backend-url-policy.sh +git commit -m "deploy: route frontend and core through one origin" +``` + +### Task 5: Harden local image builds and embedded Pi + +**Files:** +- Modify: `docker/core.Dockerfile` +- Modify: `docker/pi-runtime/package.json` +- Modify: `docker/pi-runtime/package-lock.json` +- Create: `.dockerignore` +- Modify: `docker/smoke/core-smoke.sh` +- Modify: `scripts/verify-container-images.sh` +- Create: `scripts/build-local.sh` +- Create: `scripts/build-local.ps1` +- Test: `scripts/test-container-deployment.sh` + +**Interfaces:** +- Produces: `core` containing the pinned `/usr/local/bin/pi` and a standalone `frontend` image. +- Produces: local build launchers requiring only Docker, Compose, and Git. + +- [ ] **Step 1: Write failing image-contract assertions** + +Inside `core`, assert `pi --version` matches `PI_VERSION`, UID is `10001`, Docker socket is absent, `/data` is writable, and no portal/Chirone path exists. Assert frontend health and `/api` routing. + +- [ ] **Step 2: Confirm red** + +Run: `bash scripts/test-container-deployment.sh` + +Expected: failure on at least one old deployment contract. + +- [ ] **Step 3: Pin Pi from its lock input** + +Make `docker/pi-runtime/package-lock.json` the sole Pi dependency lock. Install with `npm ci --omit=dev` in a build stage, copy into `core`, and fail build when `pi --version` differs from `PI_VERSION`. + +- [ ] **Step 4: Add exclusions and platform launchers** + +Exclude `.git`, `.worktrees`, `.env`, secrets, dependencies, virtual environments, coverage, and runtime data. Both launchers run: + +```text +docker compose -f compose.yaml -f deploy/compose.local.yaml build --pull +``` + +They print the same next command and preserve Docker's exit code. + +- [ ] **Step 5: Verify and commit** + +```sh +bash scripts/build-local.sh +bash scripts/test-container-deployment.sh +bash scripts/verify-container-images.sh +git add .dockerignore docker/core.Dockerfile docker/pi-runtime docker/smoke/core-smoke.sh scripts/build-local.sh scripts/build-local.ps1 scripts/test-container-deployment.sh scripts/verify-container-images.sh +git commit -m "build: make embedded pi images reproducible" +``` + +Windows gate: `powershell -ExecutionPolicy Bypass -File scripts/build-local.ps1`. + +### Task 6: Build the cross-platform `thothctl` foundation + +**Files:** +- Create: `tools/thothctl/go.mod` +- Create: `tools/thothctl/cmd/thothctl/main.go` +- Create: `tools/thothctl/internal/compose/runner.go` +- Create: `tools/thothctl/internal/config/installation.go` +- Create: `tools/thothctl/internal/output/sanitize.go` +- Test: `tools/thothctl/internal/compose/runner_test.go` +- Test: `tools/thothctl/internal/config/installation_test.go` +- Test: `tools/thothctl/internal/output/sanitize_test.go` +- Create: `docker/thothctl.Dockerfile` +- Create: `scripts/build-thothctl.sh` + +**Interfaces:** +- Produces: `thothctl --installation ` binaries for Windows amd64, macOS amd64/arm64, Linux amd64/arm64. +- Produces: `Runner.Run(ctx, args, stdin) (Result, error)` using argument arrays, never shell concatenation. + +- [ ] **Step 1: Write failing tests** + +Cover local/server Compose selection, paths containing spaces, missing Docker, propagated exit codes, and replacement of values matching password/token/key fields or secret-file contents with `[REDACTED]`. + +- [ ] **Step 2: Confirm red** + +Run: `docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./...` + +Expected: failure because packages do not exist. + +- [ ] **Step 3: Implement installation discovery and safe runner** + +Read `thothii-installation.yaml` fields `profile`, `projectDirectory`, `envFile`, and `overrides`. Resolve and validate absolute paths. Invoke `docker compose` with `exec.CommandContext` argument slices. + +- [ ] **Step 4: Implement base commands** + +Add `status`, `doctor`, `logs`, `start`, `stop`, and `update --check-only`. `doctor` validates Docker/Compose versions, rendered config, LF, volumes, and frontend/core health without printing environment values. + +- [ ] **Step 5: Cross-compile with Docker and verify** + +Produce `dist/thothctl/thothctl-windows-amd64.exe`, Darwin amd64/arm64, and Linux amd64/arm64 from `docker/thothctl.Dockerfile`. + +```sh +docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./... +bash scripts/build-thothctl.sh +``` + +Run the host-matching binary with `--help`, then commit: + +```sh +git add tools/thothctl docker/thothctl.Dockerfile scripts/build-thothctl.sh +git commit -m "feat: add cross-platform thothctl" +``` + +### Task 7: Implement safe Pi lifecycle commands in `thothctl` + +**Files:** +- Create: `tools/thothctl/internal/pi/commands.go` +- Create: `tools/thothctl/internal/pi/update.go` +- Create: `tools/thothctl/internal/pi/state.go` +- Test: `tools/thothctl/internal/pi/commands_test.go` +- Test: `tools/thothctl/internal/pi/update_test.go` +- Modify: `tools/thothctl/cmd/thothctl/main.go` +- Create: `docs/contracts/thothctl-pi.md` + +**Interfaces:** +- Produces: `pi status|doctor|configure|test|update|logs`. +- Produces: `.thothctl/update-state.json` with previous/new image references and phase, never credentials. +- Consumes: existing `/health`, `/models`, and `/settings` APIs plus `docker compose exec core pi --version`; Task 8 replaces the temporary composite checks with the dedicated Pi Management API. + +- [ ] **Step 1: Write failing update-state tests** + +With a fake Compose runner cover success and failure during build/pull, recreate, health, version, and smoke. Assert every post-recreate failure restores the prior image and leaves volume names unchanged. + +- [ ] **Step 2: Confirm red** + +Run: `docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./internal/pi -v` + +Expected: failure because Pi commands are absent. + +- [ ] **Step 3: Implement read-only commands** + +`status` runs `core pi --version`; `doctor` verifies image version, writable Pi volume, provider configuration presence, and `/health`; `test` combines `/models`, `/settings`, and a fixed `core pi --version` probe until Task 8 supplies the dedicated smoke endpoint; `logs` uses the shared sanitizer. + +- [ ] **Step 4: Implement guided configuration** + +Offer provider/model/reasoning choices returned by the backend. Atomically write non-secret defaults. For credentials print the expected secret filename and permissions; never accept secret text as an argument. + +- [ ] **Step 5: Implement transactional update** + +Record current image ID, pull/build requested pinned version, recreate only `core`, verify health/version/test, and roll back on failure. Refuse update while sessions are active unless `--drain` completes. + +- [ ] **Step 6: Verify and commit** + +```sh +docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./internal/pi -v +docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./... +git add tools/thothctl docs/contracts/thothctl-pi.md +git commit -m "feat: manage embedded pi with thothctl" +``` + +### Task 8: Add sanitized Pi Management backend APIs + +**Files:** +- Create: `backend/src/pi/management.ts` +- Create: `backend/src/routes/pi-management.ts` +- Modify: `backend/src/app.ts` +- Modify: `backend/src/config.ts` +- Modify: `tools/thothctl/internal/pi/commands.go` +- Test: `backend/test/pi-management.test.ts` +- Test: `backend/test/routes-pi-management.test.ts` +- Test: `tools/thothctl/internal/pi/commands_test.go` + +**Interfaces:** +- Produces: `GET /pi-management/status`, `GET /pi-management/options`, `PUT /pi-management/config`, `POST /pi-management/test`, `GET /pi-management/logs`. +- Produces: sanitized `PiStatus`, `PiOptions`, `PiInstallationConfig`, and stable errors. + +- [ ] **Step 1: Write failing service/route tests** + +Mock process execution and settings. Assert version parsing, closed choices, free-field validation, atomic writes, smoke timeout, 200-line log limit, redaction, and `403 pi_management_forbidden` in exposed server mode without trusted admin identity. + +- [ ] **Step 2: Confirm red** + +Run: `cd backend && npx vitest run test/pi-management.test.ts test/routes-pi-management.test.ts` + +Expected: module-not-found failure. + +- [ ] **Step 3: Implement service and authorization** + +Use `execFile` with fixed argument arrays. Return only version, readiness, provider names, model IDs, reasoning choices, timestamps, and sanitized messages. Allow `AUTH_MODE=none` only when public exposure is false; upstream mode requires the documented admin claim. Expose no image-update API. + +- [ ] **Step 4: Switch `thothctl` to the dedicated API** + +Replace the Task 7 composite `/health`/`/models`/`/settings` test with `POST /pi-management/test`; load closed configuration choices from `GET /pi-management/options`. Preserve the direct in-container `pi --version` check as an independent image-integrity signal. + +- [ ] **Step 5: Verify and commit** + +```sh +cd backend +npx vitest run test/pi-management.test.ts test/routes-pi-management.test.ts +npx vitest run +npx tsc --noEmit -p . +cd .. +docker run --rm -v "$PWD:/src" -w /src/tools/thothctl golang:1.24 go test ./internal/pi -v +git add backend/src/pi/management.ts backend/src/routes/pi-management.ts backend/src/app.ts backend/src/config.ts backend/test/pi-management.test.ts backend/test/routes-pi-management.test.ts tools/thothctl/internal/pi/commands.go tools/thothctl/internal/pi/commands_test.go +git commit -m "feat: expose safe pi management api" +``` + +### Task 9: Add the Pi Management interface + +**Files:** +- Create: `frontend/src/api/pi-management.ts` +- Create: `frontend/src/api/pi-management.test.ts` +- Create: `frontend/src/shell/PiManagement.tsx` +- Create: `frontend/src/shell/PiManagement.test.tsx` +- Modify: `frontend/src/shell/AppShell.tsx` +- Modify: `frontend/src/shell/AppShell.session-mgmt.test.tsx` + +**Interfaces:** +- Consumes: Task 8 Pi APIs. +- Produces: right-sidebar Pi Management panel without image-update execution or browser terminal. + +- [ ] **Step 1: Write failing UI tests** + +Cover loading/version, closed selects, validation, save, smoke test, sanitized logs, forbidden state, and copyable `thothctl pi update` instruction. Assert no secret input or browser terminal. + +- [ ] **Step 2: Confirm red** + +Run: `cd frontend && npx vitest run src/shell/PiManagement.test.tsx src/api/pi-management.test.ts` + +Expected: module-not-found failure. + +- [ ] **Step 3: Implement API and panel** + +Use query keys `['pi-management','status']` and `['pi-management','options']`. Render provider/model/reasoning as closed choices, validate numeric fields before PUT, and show credentials only as present/missing. + +- [ ] **Step 4: Attach to the right sidebar** + +Add Pi next to Workspace Management, preserve current session/activity panels and accessibility, and remove AppShell comments/layout assumptions about Omics Portal chrome. + +- [ ] **Step 5: Verify and commit** + +```sh +cd frontend +npx vitest run src/shell/PiManagement.test.tsx src/api/pi-management.test.ts src/shell/AppShell.session-mgmt.test.tsx +npx vitest run +npx tsc -b +git add src/api/pi-management.ts src/api/pi-management.test.ts src/shell/PiManagement.tsx src/shell/PiManagement.test.tsx src/shell/AppShell.tsx src/shell/AppShell.session-mgmt.test.tsx +git commit -m "feat: add pi management interface" +``` + +### Task 10: Remove active PSD and portal deployment coupling + +**Files:** +- Delete: `deploy/compose.production.yaml` +- Delete: `deploy/compose.psd-local.yaml.example` +- Modify: `README.md` +- Modify: `PROJECT_STATE.md` +- Modify: `scripts/run-stack.sh` +- Create: `scripts/test-no-deployment-coupling.sh` +- Modify: `scripts/test-external-compose-lifecycle.sh` + +**Interfaces:** +- Produces: active deployment files free of PSD/Chirone/portal networks, paths, and prefixes. +- Preserves: generic external endpoint support and historical design documents. + +- [ ] **Step 1: Write the failing coupling scan** + +Scan active Compose, Docker, root README, install guides, env examples, and run scripts for `omics_portal`, `/home/chirone`, `localllm_default`, `datamart-builder`, and PSD deployment filenames. Exclude historical specs/plans and canonical workspace content. + +- [ ] **Step 2: Confirm red** + +Run: `bash scripts/test-no-deployment-coupling.sh` + +Expected: failure listing current portal-oriented files. + +- [ ] **Step 3: Remove superseded files and genericize launch behavior** + +Delete only deployment files replaced by Tasks 2–5. Make `run-stack.sh` call base+local or direct users to `thothctl start`. Preserve data migration utilities that do not affect runtime coupling. + +- [ ] **Step 4: Update root documentation and verify** + +State that co-location never puts DWH/vector/embedding inside ThothII. + +```sh +bash scripts/test-no-deployment-coupling.sh +bash scripts/test-unified-compose.sh +bash scripts/test-external-compose-lifecycle.sh +git add -A deploy/compose.production.yaml deploy/compose.psd-local.yaml.example README.md PROJECT_STATE.md scripts/run-stack.sh scripts/test-no-deployment-coupling.sh scripts/test-external-compose-lifecycle.sh +git commit -m "refactor: remove portal deployment coupling" +``` + +### Task 11: Write and verify the PC/Mac installation manual + +**Files:** +- Create: `docs/install/local.md` +- Create: `docs/install/windows-line-endings.md` +- Create: `docs/install/pi-management.md` +- Create: `docs/install/examples/thothii-installation.local.yaml` +- Modify: `docs/install/local-workspace-registry.md` +- Modify: `scripts/verify-workspace-install-docs.sh` +- Modify: `scripts/test-verify-workspace-install-docs.sh` + +**Interfaces:** +- Produces: complete clean-install, build, update, backup, restore, Pi, and CRLF instructions for Windows/WSL2, macOS, and Linux PC. + +- [ ] **Step 1: Extend the verifier first** + +Require prerequisites, Git clone, LF verification, `.env`, external-service addressing, build, start, health, browser URL, `thothctl`, Pi update/rollback, backup/restore, pull update, and data-preserving uninstall. Require `pi-management.md` to cover UI permissions, every `thothctl pi` command, secret handling, direct support access, and failed-update recovery. Copy examples to a temporary path containing spaces and render there. + +- [ ] **Step 2: Confirm red** + +Run: `bash scripts/test-verify-workspace-install-docs.sh` + +Expected: failure naming missing local-guide sections. + +- [ ] **Step 3: Write platform-specific instructions** + +Document macOS, Windows PowerShell, Windows WSL2, and Linux separately. Recommend cloning inside the WSL filesystem. Include LF checks after clone/pull. For an existing CRLF clone, prefer recloning; describe repository-local `core.autocrlf=false` and renormalization. If mentioning `git reset --hard`, require explicit backup/commit and a destructive-action warning immediately before it. + +- [ ] **Step 4: Document external services on the host** + +Show `host.docker.internal` for Docker Desktop and `extra_hosts: host.docker.internal:host-gateway` for Linux. Explain that container `127.0.0.1` is not the host. All addresses remain configurable. + +- [ ] **Step 5: Verify and commit** + +```sh +bash scripts/test-verify-workspace-install-docs.sh +bash scripts/verify-workspace-install-docs.sh --profile local +bash scripts/docker-smoke.sh +git add docs/install/local.md docs/install/windows-line-endings.md docs/install/pi-management.md docs/install/examples/thothii-installation.local.yaml docs/install/local-workspace-registry.md scripts/verify-workspace-install-docs.sh scripts/test-verify-workspace-install-docs.sh +git commit -m "docs: add autonomous local installation guide" +``` + +### Task 12: Write and verify the server installation manual + +**Files:** +- Create: `docs/install/server.md` +- Create: `docs/install/reverse-proxy-nginx.md` +- Create: `docs/install/reverse-proxy-caddy.md` +- Create: `docs/install/examples/thothii-installation.server.yaml` +- Modify: `docs/install/server-workspace-registry.md` +- Modify: `scripts/verify-workspace-install-docs.sh` +- Test: `scripts/test-verify-workspace-install-docs.sh` + +**Interfaces:** +- Produces: generic Linux server deployment independent of another application's network. +- Consumes: server profile, secret overrides, `thothctl`, and same-origin frontend. + +- [ ] **Step 1: Add failing server-guide assertions** + +Require service account/UID, directories, firewall, co-resident external endpoints, DNS/host-gateway choices, generic TLS proxy, upstream auth, local build or pinned images, startup, readiness, Pi management, drain, rollback, backup, restore, and diagnostics. + +- [ ] **Step 2: Confirm red** + +Run: `bash scripts/test-verify-workspace-install-docs.sh` + +Expected: failure naming missing server sections. + +- [ ] **Step 3: Write the server and proxy guides** + +Use `/srv/thothii` only as an example operator root. State that DWH/vector/embedding on the same physical machine remain independently addressed services. Nginx/Caddy proxy only to frontend, preserve SSE, terminate TLS, and forward identity only after authentication. + +- [ ] **Step 4: Verify and commit** + +```sh +bash scripts/test-verify-workspace-install-docs.sh +bash scripts/verify-workspace-install-docs.sh --profile server +bash scripts/test-unified-compose.sh +git add docs/install/server.md docs/install/reverse-proxy-nginx.md docs/install/reverse-proxy-caddy.md docs/install/examples/thothii-installation.server.yaml docs/install/server-workspace-registry.md scripts/verify-workspace-install-docs.sh scripts/test-verify-workspace-install-docs.sh +git commit -m "docs: add autonomous server installation guide" +``` + +### Task 13: Add end-to-end deployment and update gates + +**Files:** +- Create: `scripts/unified-deployment-smoke.sh` +- Create: `scripts/thothctl-update-smoke.sh` +- Create: `scripts/test-windows-clone-contract.ps1` +- Create: `.github/workflows/deployment.yml` +- Modify: `PROJECT_STATE.md` +- Modify: `README.md` + +**Interfaces:** +- Produces: release gate for rendering, image build, embedded Pi, registry persistence, offline recovery, and update rollback. + +- [ ] **Step 1: Write failing orchestration smoke** + +Create an isolated Compose project and bare Git registry, build, start local, verify frontend/core/Pi/registry, recreate offline, perform a valid Git update, and prove volumes survive. Inject a bad Pi image/version and prove rollback. + +- [ ] **Step 2: Confirm red** + +Run: `bash scripts/unified-deployment-smoke.sh` + +Expected: failure because the script is absent. + +- [ ] **Step 3: Implement exact-resource cleanup** + +Generate a unique project name and temporary directory, label resources, and remove only those exact resources on exit. Never prune global Docker state. Sanitize captured logs. + +- [ ] **Step 4: Add Windows and CI gates** + +PowerShell scans tracked shell/YAML/Docker files for byte `0x0D`, renders Compose, and invokes Windows `thothctl`. CI runs LF and TypeScript gates everywhere, Docker smoke on Linux, and clone/Compose contract on Windows. + +- [ ] **Step 5: Run final verification** + +```sh +bash scripts/verify-line-endings.sh +bash scripts/test-unified-compose.sh +bash scripts/test-compose-secret-policy.sh +bash scripts/unified-deployment-smoke.sh +bash scripts/thothctl-update-smoke.sh +bash scripts/test-verify-workspace-install-docs.sh +cd backend && npx vitest run && npx tsc --noEmit -p . +cd ../frontend && npx vitest run && npx tsc -b +cd ../harness && .venv/bin/pytest -q +``` + +Expected: all non-L2 tests pass; Docker-dependent harness tests run on a Docker-capable host. + +- [ ] **Step 6: Commit** + +```sh +git add scripts/unified-deployment-smoke.sh scripts/thothctl-update-smoke.sh scripts/test-windows-clone-contract.ps1 .github/workflows/deployment.yml PROJECT_STATE.md README.md +git commit -m "test: gate unified compose deployment" +``` + +## Completion criteria + +- Clean GitHub clones build/run on Windows Docker Desktop/WSL2 and macOS using only Docker, Compose, and Git. +- The same source/images deploy on Linux through the server override. +- Active deployment files contain no PSD, Chirone, or `omics_portal` dependency. +- DWH, VectorDB, embedding, and LLM are always configurable external endpoints. +- Pi exists in `core`, is configurable through Pi Management, and is safely updated by `thothctl`. +- Git attributes prevent CRLF and verification catches corruption before image startup. +- Local/server guides pass executable documentation checks. +- Failed updates restore the previous image while preserving registry, sessions, settings, and Pi state.