ThothII/docs/superpowers/plans/2026-07-11-container-packaging-portable-storage.md

Container Packaging and Portable Storage Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Run ThothII from exactly two application images with runtime configuration and host-independent persistent roots.

Architecture: Build a multi-runtime core image for Fastify, Pi, and tht, plus a static frontend image. Resolve workspace paths beneath mounted logical roots and provide a Compose base for external dependencies.

Tech Stack: Docker BuildKit, Docker Compose v2, Node 22, Python 3.11, Fastify, React/Vite, nginx or Caddy.

Global Constraints

• Plan 1 is complete and adapter factories are the only integration construction path.
• Do not bake secrets or customer workspace content into images.
• Containers run as non-root and write only beneath mounted data roots.
• pi, tht, CA certificates, and native dependencies must work on every advertised architecture.
• Frontend backend URL is runtime-configurable.

---

Task 1: Add logical workspace roots and migration diagnostics

Files:

• Create: harness/tht/paths.py
• Create: harness/tht/cli/doctor_cmd.py
• Modify: harness/tht/config.py
• Modify: harness/tht/cli/__init__.py
• Test: harness/tests/test_portable_paths.py
• Test: harness/tests/test_doctor_cli.py

Interfaces:

• Produces: resolve_workspace_paths(config_path, cfg, data_root) -> ResolvedPaths and tht doctor --json.

• Step 1: Test relative, absolute-legacy, and escape rejection

def test_relative_paths_resolve_under_workspace_root(tmp_path):
    resolved = resolve_workspace_paths(cfg_path, cfg, tmp_path / "data")
    assert resolved.sessions == tmp_path / "data/workspaces/demo/sessions"

def test_path_escape_is_rejected(tmp_path):
    with pytest.raises(ConfigError, match="outside workspace root"):
        resolve_workspace_paths(cfg_path, config_with_sessions("../../private"), tmp_path)

• Step 2: Verify failure

Run: cd harness && .venv/bin/pytest tests/test_portable_paths.py tests/test_doctor_cli.py -q
Expected: FAIL.

• Step 3: Implement resolution and JSON diagnostics

@dataclass(frozen=True)
class ResolvedPaths:
    workspace: Path
    sessions: Path
    artifacts: Path
    indexes: Path
    corpus: Path

doctor --json returns component statuses without printing warnings to stdout.

• Step 4: Run tests

Run: cd harness && .venv/bin/pytest tests/test_portable_paths.py tests/test_doctor_cli.py tests/test_workspace.py -q
Expected: PASS.

• Step 5: Commit

git add harness/tht/paths.py harness/tht/cli/doctor_cmd.py harness/tht/config.py harness/tht/cli/__init__.py harness/tests/test_portable_paths.py harness/tests/test_doctor_cli.py
git commit -m "feat(storage): resolve portable workspace roots"

Task 2: Make backend process paths and listening address container-safe

Files:

• Modify: backend/src/config.ts
• Modify: backend/src/server.ts
• Modify: backend/src/pi/pi-process-manager.ts
• Modify: backend/src/tht/tht-runner.ts
• Test: backend/test/config.test.ts
• Test: backend/test/pi-process-manager.test.ts

Interfaces:

• Produces env contract: HOST, PORT, THT_HARNESS_DIR, THT_BIN, PI_BIN, SETTINGS_FILE, THT_DATA_ROOT.

• Step 1: Add tests for explicit binaries and 0.0.0.0 listening

expect(loadConfig({ HOST: "0.0.0.0", THT_BIN: "/opt/venv/bin/tht" })).toMatchObject({
  host: "0.0.0.0", thtBin: "/opt/venv/bin/tht"
});

• Step 2: Verify failure

Run: cd backend && npx vitest run test/config.test.ts test/pi-process-manager.test.ts
Expected: FAIL for missing host/data-root behavior.

• Step 3: Remove the .venv/bin PATH assumption

const env = { ...process.env, THT_DATA_ROOT: cfg.dataRoot };
const child = spawn(cfg.piBin, ["--mode", "rpc"], { cwd: cfg.harnessDir, env });

• Step 4: Run backend tests and typecheck

Run: cd backend && npx vitest run && npx tsc --noEmit -p .
Expected: PASS.

• Step 5: Commit

git add backend/src backend/test
git commit -m "feat(backend): support container runtime paths"

Task 3: Build the core image

Files:

• Create: docker/core.Dockerfile
• Create: docker/core-entrypoint.sh
• Create: .dockerignore
• Create: docker/smoke/core-smoke.sh

Interfaces:

• Produces image entrypoints: server, doctor, preprocess, and arbitrary tht ....

• Step 1: Add a smoke script that asserts binaries and health

test "$(id -u)" != "0"
node --version
python --version
tht --help >/dev/null
pi --version >/dev/null
curl --fail http://127.0.0.1:8787/health

• Step 2: Build and observe the initial failure

Run: docker build -f docker/core.Dockerfile -t thothii-core:test .
Expected: FAIL because the Dockerfile is not present before implementation.

• Step 3: Implement a multi-stage core build

FROM node:22-bookworm AS backend-build
WORKDIR /src/backend
COPY backend/package*.json ./
RUN npm ci
COPY backend/ ./
RUN npm run build

FROM python:3.11-slim-bookworm AS runtime
RUN useradd --create-home --uid 10001 thoth
WORKDIR /app
COPY harness/ /app/harness/
RUN python -m venv /opt/venv && /opt/venv/bin/pip install --no-cache-dir /app/harness
COPY --from=backend-build /src/backend/dist /app/backend/dist
COPY --from=backend-build /src/backend/node_modules /app/backend/node_modules
ENV PATH="/opt/venv/bin:$PATH" HOST=0.0.0.0 PORT=8787
USER thoth
ENTRYPOINT ["/app/docker/core-entrypoint.sh"]
CMD ["server"]

Add Pi installation only from its pinned, redistributable source after the Phase 0 license/runtime gate; fail the build if pi --version is unavailable.

• Step 4: Build and run smoke test

Run: docker build -f docker/core.Dockerfile -t thothii-core:test .
Expected: success.
Run: docker run --rm thothii-core:test doctor
Expected: process starts and reports missing external configuration without traceback.

• Step 5: Commit

git add docker/core.Dockerfile docker/core-entrypoint.sh docker/smoke/core-smoke.sh .dockerignore
git commit -m "build(docker): add core application image"

Task 4: Add runtime frontend configuration and image

Files:

• Create: frontend/public/config.js
• Create: frontend/src/api/runtime-config.ts
• Modify: frontend/src/api/client.ts
• Modify: frontend/index.html
• Create: docker/frontend.Dockerfile
• Create: docker/frontend-entrypoint.sh
• Create: docker/nginx.conf.template
• Test: frontend/src/api/runtime-config.test.ts

Interfaces:

• Produces browser contract: window.__THOTHII_CONFIG__.backendBaseUrl.

• Step 1: Write fallback and injected-config tests

expect(resolveBackendUrl({ backendBaseUrl: "/api" })).toBe("/api");
expect(resolveBackendUrl(undefined)).toBe(import.meta.env.VITE_BACKEND_URL ?? "");

• Step 2: Verify failure

Run: cd frontend && npx vitest run src/api/runtime-config.test.ts
Expected: FAIL.

• Step 3: Implement runtime config generation and reverse proxy

sed "s|__BACKEND_BASE_URL__|${BACKEND_BASE_URL:-/api}|g" \
  /usr/share/nginx/html/config.template.js > /usr/share/nginx/html/config.js
exec nginx -g 'daemon off;'

• Step 4: Test and build

Run: cd frontend && npx vitest run && npx tsc -b && npm run build
Expected: PASS.
Run: docker build -f docker/frontend.Dockerfile -t thothii-frontend:test .
Expected: success.

• Step 5: Commit

git add frontend docker/frontend.Dockerfile docker/frontend-entrypoint.sh docker/nginx.conf.template
git commit -m "build(docker): add runtime-configured frontend image"

Task 5: Compose external profile and end-to-end smoke gate

Final-review security amendment (2026-07-12): the frontend port binds to 127.0.0.1 by default. Public deployment uses an authenticated upstream proxy with AUTH_MODE=upstream; THOTH_PUBLIC_EXPOSURE=true plus AUTH_MODE=none is invalid. Local env files are development only; production uses read-only Compose secrets. Image gates pin exact tags and multi-platform digests and verify both linux/amd64 and linux/arm64 using the shared container verification script.

Files:

• Create: compose.yaml
• Create: deploy/env.example
• Create: deploy/workspaces/example.yaml
• Create: scripts/docker-smoke.sh
• Modify: README.md
• Test: backend/test/health.test.ts

Interfaces:

• Produces services core and frontend; persistent volume/mount contract beneath /data.

• Step 1: Add a smoke script for Compose config and HTTP health

docker compose config --quiet
docker compose up --build --wait core frontend
curl --fail http://localhost:8080/health
docker compose down

• Step 2: Verify Compose is initially absent

Run: docker compose config --quiet
Expected: FAIL before compose.yaml is implemented.

• Step 3: Define the base deployment

services:
  core:
    build: { context: ., dockerfile: docker/core.Dockerfile }
    environment:
      THT_DATA_ROOT: /data
      SETTINGS_FILE: /data/settings/settings.json
    volumes: ["./deploy/workspaces:/data/workspaces:ro", "thoth_data:/data"]
  frontend:
    build: { context: ., dockerfile: docker/frontend.Dockerfile }
    environment: { BACKEND_BASE_URL: /api }
    ports: ["8080:8080"]
volumes: { thoth_data: {} }

• Step 4: Run Compose smoke and full layer gates

Run: ./scripts/docker-smoke.sh
Expected: both health checks PASS.
Run: cd harness && .venv/bin/pytest -q
Run: cd backend && npx vitest run && npx tsc --noEmit -p .
Run: cd frontend && npx vitest run && npx tsc -b
Expected: all PASS.

• Step 5: Commit

git add compose.yaml deploy scripts/docker-smoke.sh README.md
git commit -m "feat(deploy): add portable external-service stack"