ThothII/docs/superpowers/plans/2026-07-11-container-packaging-portable-storage.md

# Container Packaging and Portable Storage Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Run ThothII from exactly two application images with runtime configuration and host-independent persistent roots.

**Architecture:** Build a multi-runtime core image for Fastify, Pi, and `tht`, plus a static frontend image. Resolve workspace paths beneath mounted logical roots and provide a Compose base for external dependencies.

**Tech Stack:** Docker BuildKit, Docker Compose v2, Node 22, Python 3.11, Fastify, React/Vite, nginx or Caddy.

## Global Constraints

- Plan 1 is complete and adapter factories are the only integration construction path.
- Do not bake secrets or customer workspace content into images.
- Containers run as non-root and write only beneath mounted data roots.
- `pi`, `tht`, CA certificates, and native dependencies must work on every advertised architecture.
- Frontend backend URL is runtime-configurable.

---

### Task 1: Add logical workspace roots and migration diagnostics

**Files:**
- Create: `harness/tht/paths.py`
- Create: `harness/tht/cli/doctor_cmd.py`
- Modify: `harness/tht/config.py`
- Modify: `harness/tht/cli/__init__.py`
- Test: `harness/tests/test_portable_paths.py`
- Test: `harness/tests/test_doctor_cli.py`

**Interfaces:**
- Produces: `resolve_workspace_paths(config_path, cfg, data_root) -> ResolvedPaths` and `tht doctor --json`.

- [ ] **Step 1: Test relative, absolute-legacy, and escape rejection**

```python
def test_relative_paths_resolve_under_workspace_root(tmp_path):
    resolved = resolve_workspace_paths(cfg_path, cfg, tmp_path / "data")
    assert resolved.sessions == tmp_path / "data/workspaces/demo/sessions"

def test_path_escape_is_rejected(tmp_path):
    with pytest.raises(ConfigError, match="outside workspace root"):
        resolve_workspace_paths(cfg_path, config_with_sessions("../../private"), tmp_path)
```

- [ ] **Step 2: Verify failure**

Run: `cd harness && .venv/bin/pytest tests/test_portable_paths.py tests/test_doctor_cli.py -q`  
Expected: FAIL.

- [ ] **Step 3: Implement resolution and JSON diagnostics**

```python
@dataclass(frozen=True)
class ResolvedPaths:
    workspace: Path
    sessions: Path
    artifacts: Path
    indexes: Path
    corpus: Path
```

`doctor --json` returns component statuses without printing warnings to stdout.

- [ ] **Step 4: Run tests**

Run: `cd harness && .venv/bin/pytest tests/test_portable_paths.py tests/test_doctor_cli.py tests/test_workspace.py -q`  
Expected: PASS.

- [ ] **Step 5: Commit**

```bash
git add harness/tht/paths.py harness/tht/cli/doctor_cmd.py harness/tht/config.py harness/tht/cli/__init__.py harness/tests/test_portable_paths.py harness/tests/test_doctor_cli.py
git commit -m "feat(storage): resolve portable workspace roots"
```

### Task 2: Make backend process paths and listening address container-safe

**Files:**
- Modify: `backend/src/config.ts`
- Modify: `backend/src/server.ts`
- Modify: `backend/src/pi/pi-process-manager.ts`
- Modify: `backend/src/tht/tht-runner.ts`
- Test: `backend/test/config.test.ts`
- Test: `backend/test/pi-process-manager.test.ts`

**Interfaces:**
- Produces env contract: `HOST`, `PORT`, `THT_HARNESS_DIR`, `THT_BIN`, `PI_BIN`, `SETTINGS_FILE`, `THT_DATA_ROOT`.

- [ ] **Step 1: Add tests for explicit binaries and `0.0.0.0` listening**

```typescript
expect(loadConfig({ HOST: "0.0.0.0", THT_BIN: "/opt/venv/bin/tht" })).toMatchObject({
  host: "0.0.0.0", thtBin: "/opt/venv/bin/tht"
});
```

- [ ] **Step 2: Verify failure**

Run: `cd backend && npx vitest run test/config.test.ts test/pi-process-manager.test.ts`  
Expected: FAIL for missing host/data-root behavior.

- [ ] **Step 3: Remove the `.venv/bin` PATH assumption**

```typescript
const env = { ...process.env, THT_DATA_ROOT: cfg.dataRoot };
const child = spawn(cfg.piBin, ["--mode", "rpc"], { cwd: cfg.harnessDir, env });
```

- [ ] **Step 4: Run backend tests and typecheck**

Run: `cd backend && npx vitest run && npx tsc --noEmit -p .`  
Expected: PASS.

- [ ] **Step 5: Commit**

```bash
git add backend/src backend/test
git commit -m "feat(backend): support container runtime paths"
```

### Task 3: Build the core image

**Files:**
- Create: `docker/core.Dockerfile`
- Create: `docker/core-entrypoint.sh`
- Create: `.dockerignore`
- Create: `docker/smoke/core-smoke.sh`

**Interfaces:**
- Produces image entrypoints: `server`, `doctor`, `preprocess`, and arbitrary `tht ...`.

- [ ] **Step 1: Add a smoke script that asserts binaries and health**

```sh
test "$(id -u)" != "0"
node --version
python --version
tht --help >/dev/null
pi --version >/dev/null
curl --fail http://127.0.0.1:8787/health
```

- [ ] **Step 2: Build and observe the initial failure**

Run: `docker build -f docker/core.Dockerfile -t thothii-core:test .`  
Expected: FAIL because the Dockerfile is not present before implementation.

- [ ] **Step 3: Implement a multi-stage core build**

```dockerfile
FROM node:22-bookworm AS backend-build
WORKDIR /src/backend
COPY backend/package*.json ./
RUN npm ci
COPY backend/ ./
RUN npm run build

FROM python:3.11-slim-bookworm AS runtime
RUN useradd --create-home --uid 10001 thoth
WORKDIR /app
COPY harness/ /app/harness/
RUN python -m venv /opt/venv && /opt/venv/bin/pip install --no-cache-dir /app/harness
COPY --from=backend-build /src/backend/dist /app/backend/dist
COPY --from=backend-build /src/backend/node_modules /app/backend/node_modules
ENV PATH="/opt/venv/bin:$PATH" HOST=0.0.0.0 PORT=8787
USER thoth
ENTRYPOINT ["/app/docker/core-entrypoint.sh"]
CMD ["server"]
```

Add Pi installation only from its pinned, redistributable source after the Phase 0 license/runtime gate; fail the build if `pi --version` is unavailable.

- [ ] **Step 4: Build and run smoke test**

Run: `docker build -f docker/core.Dockerfile -t thothii-core:test .`  
Expected: success.  
Run: `docker run --rm thothii-core:test doctor`  
Expected: process starts and reports missing external configuration without traceback.

- [ ] **Step 5: Commit**

```bash
git add docker/core.Dockerfile docker/core-entrypoint.sh docker/smoke/core-smoke.sh .dockerignore
git commit -m "build(docker): add core application image"
```

### Task 4: Add runtime frontend configuration and image

**Files:**
- Create: `frontend/public/config.js`
- Create: `frontend/src/api/runtime-config.ts`
- Modify: `frontend/src/api/client.ts`
- Modify: `frontend/index.html`
- Create: `docker/frontend.Dockerfile`
- Create: `docker/frontend-entrypoint.sh`
- Create: `docker/nginx.conf.template`
- Test: `frontend/src/api/runtime-config.test.ts`

**Interfaces:**
- Produces browser contract: `window.__THOTHII_CONFIG__.backendBaseUrl`.

- [ ] **Step 1: Write fallback and injected-config tests**

```typescript
expect(resolveBackendUrl({ backendBaseUrl: "/api" })).toBe("/api");
expect(resolveBackendUrl(undefined)).toBe(import.meta.env.VITE_BACKEND_URL ?? "");
```

- [ ] **Step 2: Verify failure**

Run: `cd frontend && npx vitest run src/api/runtime-config.test.ts`  
Expected: FAIL.

- [ ] **Step 3: Implement runtime config generation and reverse proxy**

```sh
sed "s|__BACKEND_BASE_URL__|${BACKEND_BASE_URL:-/api}|g" \
  /usr/share/nginx/html/config.template.js > /usr/share/nginx/html/config.js
exec nginx -g 'daemon off;'
```

- [ ] **Step 4: Test and build**

Run: `cd frontend && npx vitest run && npx tsc -b && npm run build`  
Expected: PASS.  
Run: `docker build -f docker/frontend.Dockerfile -t thothii-frontend:test .`  
Expected: success.

- [ ] **Step 5: Commit**

```bash
git add frontend docker/frontend.Dockerfile docker/frontend-entrypoint.sh docker/nginx.conf.template
git commit -m "build(docker): add runtime-configured frontend image"
```

### Task 5: Compose external profile and end-to-end smoke gate

> **Final-review security amendment (2026-07-12):** the frontend port binds to `127.0.0.1` by
> default. Public deployment uses an authenticated upstream proxy with `AUTH_MODE=upstream`;
> `THOTH_PUBLIC_EXPOSURE=true` plus `AUTH_MODE=none` is invalid. Local env files are development
> only; production uses read-only Compose secrets. Image gates pin exact tags and multi-platform
> digests and verify both linux/amd64 and linux/arm64 using the shared container verification
> script.

**Files:**
- Create: `compose.yaml`
- Create: `deploy/env.example`
- Create: `deploy/workspaces/example.yaml`
- Create: `scripts/docker-smoke.sh`
- Modify: `README.md`
- Test: `backend/test/health.test.ts`

**Interfaces:**
- Produces services `core` and `frontend`; persistent volume/mount contract beneath `/data`.

- [ ] **Step 1: Add a smoke script for Compose config and HTTP health**

```sh
docker compose config --quiet
docker compose up --build --wait core frontend
curl --fail http://localhost:8080/health
docker compose down
```

- [ ] **Step 2: Verify Compose is initially absent**

Run: `docker compose config --quiet`  
Expected: FAIL before `compose.yaml` is implemented.

- [ ] **Step 3: Define the base deployment**

```yaml
services:
  core:
    build: { context: ., dockerfile: docker/core.Dockerfile }
    environment:
      THT_DATA_ROOT: /data
      SETTINGS_FILE: /data/settings/settings.json
    volumes: ["./deploy/workspaces:/data/workspaces:ro", "thoth_data:/data"]
  frontend:
    build: { context: ., dockerfile: docker/frontend.Dockerfile }
    environment: { BACKEND_BASE_URL: /api }
    ports: ["8080:8080"]
volumes: { thoth_data: {} }
```

- [ ] **Step 4: Run Compose smoke and full layer gates**

Run: `./scripts/docker-smoke.sh`  
Expected: both health checks PASS.  
Run: `cd harness && .venv/bin/pytest -q`  
Run: `cd backend && npx vitest run && npx tsc --noEmit -p .`  
Run: `cd frontend && npx vitest run && npx tsc -b`  
Expected: all PASS.

- [ ] **Step 5: Commit**

```bash
git add compose.yaml deploy scripts/docker-smoke.sh README.md
git commit -m "feat(deploy): add portable external-service stack"
```