# Datamart Builder accepted-release cutover implementation plan

> **Execution rule:** run this plan with `executing-plans`, one task at a time. Stop at both manual
> gates. Do not infer approval from silence.

**Goal:** Make the accepted ThothII release the only application behind the portal's canonical
`Datamart Builder` route, remove the legacy release and temporary test route after explicit visual
approval, then commit and push only reviewed non-secret changes.

**Architecture:** The first checkpoint is a routing-only switch: both canonical and test URLs point
to the accepted `thothii-test` containers while the legacy containers remain intact. After approval,
the accepted images are retagged and recreated as the canonical `thothii` Compose project. Its
Qdrant and Ollama services reuse the already validated named volumes as external storage. Django
and Nginx then lose every `-test` route, and exact legacy/test resources are removed by labels and
IDs. The `/srv/thothii/operator` Compose/runtime files are protected server-local state and are not
committed.

**Technology:** Docker Compose, Nginx, Django 5, React/Vite runtime config, Fastify/Node, Python
`unittest`, Vitest, Git.

---

## Safety invariants

- Never print, diff, copy into Git, or inspect the contents of environment/secret files.
- Never remove a Docker resource selected only by a substring or broad glob.
- Never pass `-v` to Compose cleanup. The accepted Qdrant/Ollama volumes are retained.
- Never remove `omics_portal_omics_network` or any DWH, Authentik, Supabase, Superset, Aritmolab,
  LocalLLM, ETL, or unrelated resource.
- Do not touch `/srv/thothii/workspace-registry`, auth roots, session data, Pi state, or vault data.
- Phase 1 changes no application containers except rebuilding/restarting the portal web service and
  reloading portal Nginx.
- Stop after Task 5. Resume only after the first explicit visual approval.
- Stop after Task 11. Commit/push only after the second explicit visual approval.

## Task 1: Record immutable baselines and rollback targets

**Files:** no modifications.

1. Record the two repository branches, remotes, worktrees, and the current commit without staging
   anything:

   ```bash
   git -C /home/chirone/ThothII-next status --short
   git -C /home/chirone/ThothII-next rev-parse HEAD
   git -C /home/chirone/omics_portal status --short
   git -C /home/chirone/omics_portal rev-parse HEAD
   ```

2. Inventory legacy and accepted containers separately by exact Compose project label. Save only
   IDs, names, image IDs, service labels, mounts, and network names to a mode-0600 temporary
   directory. Do not capture `Config.Env`:

   ```bash
   evidence_dir="$(mktemp -d /tmp/thothii-cutover.XXXXXX)"
   chmod 0700 "$evidence_dir"
   docker ps -a --filter label=com.docker.compose.project=thothii \
     --format '{{.ID}} {{.Names}} {{.Image}} {{.Label "com.docker.compose.service"}}' \
     > "$evidence_dir/legacy-containers.txt"
   docker ps -a --filter label=com.docker.compose.project=thothii-test \
     --format '{{.ID}} {{.Names}} {{.Image}} {{.Label "com.docker.compose.service"}}' \
     > "$evidence_dir/accepted-containers.txt"
   chmod 0600 "$evidence_dir"/*.txt
   ```

3. Assert the legacy set is exactly `core` and `frontend`; assert the accepted set is exactly
   `core`, `frontend`, `qdrant`, `embedding`, and the exited `embedding-model-init`. Abort if the
   inventory differs.

4. Resolve and retain the four core/frontend image IDs in shell variables or mode-0600 evidence.
   Confirm the accepted services are healthy and the public test route returns the existing login
   redirect or authenticated page.

## Task 2: Make the accepted frontend runtime config work on both temporary paths

**Files:**

- Create: `/home/chirone/ThothII-next/scripts/test-datamart-builder-path-config.mjs`
- Modify: `/home/chirone/ThothII-next/deploy/thothii-test-config.js`

1. Write a failing Node test that evaluates the runtime script in isolated VM contexts and asserts:

   - pathname `/datamart-builder/` produces `/datamart-builder/api`;
   - pathname `/datamart-builder-test/` produces `/datamart-builder-test/api`;
   - no absolute URL, hostname, credential key, or value is present;
   - every other pathname fails closed to the canonical same-origin API base.

2. Run RED:

   ```bash
   node scripts/test-datamart-builder-path-config.mjs
   ```

   Expected: the canonical-path assertion fails because the current file always selects the test
   prefix.

3. Change the runtime script to select the test prefix only when
   `window.location.pathname` starts with `/datamart-builder-test/`; otherwise select the canonical
   prefix. Keep the only public value as `window.__THOTHII_CONFIG__.backendBaseUrl`.

4. Run GREEN and the existing frontend URL policy tests:

   ```bash
   node scripts/test-datamart-builder-path-config.mjs
   cd /home/chirone/ThothII-next/frontend
   npx vitest run src/api/runtime-config.test.ts
   npx tsc -b
   ```

## Task 3: Specify the reversible portal routing switch test-first

**Files:**

- Modify: `/home/chirone/omics_portal/kokoro/test_page_titles.py`
- Modify: `/home/chirone/omics_portal/kokoro/test_superset_embed.py`
- Modify: `/home/chirone/omics_portal/test_thothii_nginx.py`

1. Add Django assertions that during the temporary dual-route phase:

   - `datamart_builder_public` loads `/datamart-builder/config.js` and canonical asset URLs from the
     accepted manifest variant;
   - `datamart_builder_test` still loads `/datamart-builder-test/config.js` and test asset URLs;
   - both menu entries remain visible to the capability-bearing user;
   - capability denial remains unchanged.

2. Extend the static Nginx contract to assert:

   - canonical `/datamart-builder/api/` proxies to `thothii_test_core`;
   - canonical assets and exact `config.js` proxy to `thothii_test_frontend`;
   - the canonical API receives the same HTTPS-to-internal-origin normalization as the test API;
   - Authentik `auth_request`, normalized principal headers, cookie/bearer stripping, SSE buffering,
     and timeouts remain present;
   - the test route still exists and still targets the accepted upstreams.

3. Build a disposable portal test image and run RED without replacing the live portal:

   ```bash
   cd /home/chirone/omics_portal
   docker compose build web
   docker compose run --rm --no-deps web \
     python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
   docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
   ```

   Expected: canonical accepted-manifest/config and Nginx-upstream assertions fail.

## Task 4: Implement and deploy the reversible portal switch

**Files:**

- Modify: `/home/chirone/omics_portal/kokoro/datamart_catalog_views.py`
- Modify: `/home/chirone/omics_portal/kokoro/templatetags/vite.py`
- Modify: `/home/chirone/omics_portal/templates/kokoro/datamart_builder.html`
- Modify: `/home/chirone/omics_portal/nginx/nginx.conf`
- Server-local update: `/srv/thothii/operator/thothii-test-config.js`

1. Add an accepted/canonical Vite manifest variant: fetch the manifest from
   `thothii-test-frontend`, but emit `/datamart-builder/assets/` URLs. Keep the test variant
   unchanged.

2. Make the canonical view choose the accepted/canonical variant and make the template load
   `/datamart-builder/config.js`; keep the test view/title/config intact.

3. In Nginx, point canonical API/assets/config to the test upstreams. Use one origin-normalization
   map for both accepted routes. Preserve the exact auth and SSE headers.

4. Run GREEN in disposable containers, then Django system checks:

   ```bash
   cd /home/chirone/omics_portal
   docker compose build web
   docker compose run --rm --no-deps web \
     python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
   docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
   docker compose run --rm --no-deps web python manage.py check
   ```

5. Install the reviewed non-secret path-aware runtime config atomically at the protected operator
   location, retaining ownership and mode. Do not read or rewrite `server.env`:

   ```bash
   sudo cp /home/chirone/ThothII-next/deploy/thothii-test-config.js \
     /srv/thothii/operator/thothii-test-config.js
   ```

6. Deploy only the portal web image, validate Nginx, then reload it:

   ```bash
   cd /home/chirone/omics_portal
   docker compose up -d --build web
   docker compose exec nginx nginx -t
   docker compose restart nginx
   docker compose ps
   ```

7. Verify accepted core/frontend health; probe both URLs without credentials; verify Nginx access
   logs show canonical requests reaching accepted assets/API and no new canonical traffic reaches
   legacy containers. Never dump backend environment or authentication payloads.

8. If any automated check fails, restore the four portal files and the previous operator runtime
   config from the captured diff/copy, rebuild web, validate Nginx, reload, and leave both ThothII
   projects untouched.

## Task 5: First manual visual gate — STOP

Ask the operator to open:

`https://aritmolab.policlinicosandonato.it/datamart-builder/`

Required manual checks:

1. Authentik/portal login succeeds.
2. Only the expected user identity/capability is used.
3. `psd-clinical` is visible and **Test Workspace Connection** succeeds.
4. The selected provider/model are correct.
5. A short disposable session starts and produces a response.
6. The parallel `/datamart-builder-test/` rollback URL still opens.

Do not stop/remove/tag any legacy resource before an explicit approval.

---

## Task 6: Prepare the canonical accepted Compose deployment after approval

**Files:**

- Create server-local: `/srv/thothii/operator/compose.datamart-builder-portal.yaml`
- Create server-local: `/srv/thothii/operator/thothii-config.js`
- No Git-tracked production file yet.

1. Re-run Task 1 inventory and compare the exact IDs with the saved baseline. Abort on unexpected
   drift.

2. Create a root/operator-protected canonical overlay derived from the tested overlay. It must:

   - use the accepted core/frontend image IDs through canonical tags;
   - set `AUTH_MODE=upstream` and preserve the already accepted non-secret workspace transport;
   - attach only core/frontend to `omics_portal_omics_network` with aliases `thothii-core` and
     `thothii-frontend`;
   - mount the canonical non-secret runtime config read-only into frontend;
   - declare `thothii-test_qdrant-data` and `thothii-test_embedding-models` as external named
     volumes so no accepted index/model state is copied or lost;
   - contain no secret value.

3. Write `/srv/thothii/operator/thothii-config.js` with the fixed same-origin base
   `/datamart-builder/api`. Set a read-only mode suitable for the frontend container.

4. Validate the complete canonical Compose rendering using the protected existing environment
   file without printing the rendered config:

   ```bash
   docker compose -p thothii \
     --env-file /srv/thothii/operator/server.env \
     -f /srv/thothii/source/ThothII/compose.yaml \
     -f /srv/thothii/source/ThothII/deploy/compose.server.yaml \
     -f /srv/thothii/source/ThothII/deploy/compose.git-ssh.yaml \
     -f /srv/thothii/operator/project-a-private.yaml \
     -f /srv/thothii/operator/compose.datamart-builder-portal.yaml \
     config --quiet
   ```

## Task 7: Promote accepted containers under the canonical Compose identity

**Files:** runtime only.

1. Tag the two legacy image IDs with temporary exact rollback tags. Tag the accepted image IDs with
   canonical `thothii-core:local` and `thothii-frontend:local` only after removing those canonical
   tags from the legacy images. Do not delete either legacy image ID yet.

2. Stop accepted services cleanly so Qdrant/Ollama volumes have a single writer. Stop the exact two
   legacy containers and remove only those two containers, resolving them from the verified label
   inventory.

3. Start the complete canonical project from the five-file Compose set in Task 6. Do not build or
   pull:

   ```bash
   docker compose -p thothii \
     --env-file /srv/thothii/operator/server.env \
     -f /srv/thothii/source/ThothII/compose.yaml \
     -f /srv/thothii/source/ThothII/deploy/compose.server.yaml \
     -f /srv/thothii/source/ThothII/deploy/compose.git-ssh.yaml \
     -f /srv/thothii/operator/project-a-private.yaml \
     -f /srv/thothii/operator/compose.datamart-builder-portal.yaml \
     up -d --no-build
   ```

4. Wait for canonical core/frontend/qdrant/embedding health. Verify their exact image IDs, mounts,
   external-volume names, portal-network aliases, and protected bind roots.

5. Run loopback workspace diagnostics and one create/delete disposable session against the
   canonical core using only synthetic normalized principal headers. Retain no content or secret
   output.

6. On failure, stop/remove the partial canonical project without `-v`, restore legacy canonical
   image tags, recreate the old two-service project from `/home/chirone/ThothII`, and restart the
   accepted test project. Stop the plan and report the failed invariant.

## Task 8: Specify final removal of the temporary portal route test-first

**Files:**

- Modify: `/home/chirone/omics_portal/kokoro/test_page_titles.py`
- Modify: `/home/chirone/omics_portal/kokoro/test_superset_embed.py`
- Modify: `/home/chirone/omics_portal/test_thothii_nginx.py`

1. Replace temporary-phase expectations with final-state assertions:

   - the canonical page loads `/datamart-builder/config.js` and canonical assets;
   - URL reversing `datamart_builder_test` fails and the sidebar has no test label/link;
   - no `thothii_test_*`, `/datamart-builder-test`, or test origin-map identifier remains in Nginx;
   - canonical API/assets/config target `thothii_core`/`thothii_frontend`;
   - Authentik principal and SSE contracts remain exact.

2. Build the disposable test image and run RED. Expected: all negative test-route assertions fail
   while the temporary route still exists.

## Task 9: Remove the temporary portal route and switch Nginx to canonical containers

**Files:**

- Modify: `/home/chirone/omics_portal/kokoro/datamart_catalog_views.py`
- Modify: `/home/chirone/omics_portal/kokoro/templatetags/vite.py`
- Modify: `/home/chirone/omics_portal/omics_portal/urls.py`
- Modify: `/home/chirone/omics_portal/templates/kokoro/datamart_builder.html`
- Modify: `/home/chirone/omics_portal/templates/partials/left-sidebar.html`
- Modify: `/home/chirone/omics_portal/nginx/nginx.conf`

1. Remove the test view, URL, Vite variant, template branch, menu link, Nginx upstreams, and all
   `/datamart-builder-test` locations. Restore the canonical manifest lookup to
   `thothii-frontend`; keep the canonical runtime config script and canonical asset prefix.

2. Point canonical API/assets/config to `thothii_core` and `thothii_frontend`. Retain the normalized
   origin mapping if required by the accepted backend's same-origin policy, but give it only a
   canonical name.

3. Run the final portal GREEN suite and checks in disposable containers:

   ```bash
   cd /home/chirone/omics_portal
   docker compose build web
   docker compose run --rm --no-deps web \
     python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
   docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
   docker compose run --rm --no-deps web python manage.py check
   ```

4. Deploy web, validate Nginx, reload, and probe the canonical URL. Confirm the test URL no longer
   resolves as a ThothII route and the menu item is absent.

5. If validation fails, restore canonical Nginx to the still-running canonical accepted project;
   do not remove test containers or image tags.

## Task 10: Remove exact legacy and temporary runtime resources

**Files:**

- Delete temporary untracked integration file:
  `/home/chirone/ThothII-next/deploy/compose.datamart-builder-test.yaml`
- Delete temporary untracked integration file:
  `/home/chirone/ThothII-next/deploy/thothii-test-config.js`
- Retain server-local canonical overlay/config under `/srv/thothii/operator`.

1. Confirm canonical portal, core, frontend, Qdrant, and embedding are healthy and that the portal no
   longer resolves either test upstream name.

2. Remove the stopped `thothii-test` project containers and its private network using the exact
   five-file test Compose set. Do not pass `-v`; the two named volumes are external storage for the
   canonical project.

3. Remove only the temporary test image tags. Confirm the same accepted image IDs remain reachable
   through the canonical tags.

4. Remove the two legacy rollback image tags and exact legacy image IDs only after proving no
   container references them.

5. Remove the obsolete legacy `/home/chirone/ThothII` tree only after a final read-only search proves
   no running container mount, installation descriptor, Compose file, systemd unit, or portal config
   references it. Use a recoverable trash/move operation when available; otherwise request a final
   explicit destructive confirmation with the resolved absolute target before recursive deletion.

6. Remove `/srv/thothii/operator/thothii-test-config.js` after confirming canonical frontend mounts
   only `thothii-config.js`. Remove no other operator file.

7. Report exact container/image/network/file targets removed and confirm the retained accepted
   volume names and canonical containers.

## Task 11: Full verification and second manual visual gate — STOP

1. Run targeted ThothII checks for every modified source file:

   ```bash
   cd /home/chirone/ThothII-next/frontend
   npx vitest run src/api/runtime-config.test.ts
   npx tsc -b
   cd /home/chirone/ThothII-next/backend
   npx vitest run test/app-auth-mode.test.ts test/routes-workspaces.test.ts test/routes-sessions.test.ts
   npx tsc --noEmit -p .
   ```

2. Run portal Django/Nginx checks from Task 9 and `docker compose ps`.

3. Verify final topology:

   - one canonical `thothii` project;
   - no legacy/test containers, network, or obsolete image ID;
   - canonical accepted images and healthy services;
   - no test route/menu/upstream/config;
   - shared service container IDs and network IDs unchanged from Task 1;
   - `/datamart-builder/` still redirects unauthenticated clients through the portal login path.

4. Ask the operator to repeat login, workspace connector, model/provider, short session, and a
   visual scan at the canonical URL. Stop. Do not stage, commit, or push.

## Task 12: Review, commit, and push after second approval

**Files:** both repositories; exact staged sets determined from reviewed diffs.

1. Inspect each worktree and classify every path as cutover work, earlier approved product fix, user
   scratch, protected runtime state, or unrelated. Never stage `.superpowers/sdd/progress.md`, Brain
   scratch, `/srv`, `/tmp`, environment files, API keys, sessions, logs, or generated evidence.

2. Run `git diff --check`, staged secret scans, and all tests affected by the exact staged files.

3. In `ThothII-next`, commit approved generic product fixes/tests/docs separately from server-only
   cleanup. Include this plan and the prior design; exclude deleted temporary files that were never
   tracked unless their removal is represented by the final intended source state.

4. In `omics_portal`, commit the canonical Datamart Builder integration and its tests. Confirm no
   unrelated clinical/data changes are staged.

5. Inspect all configured push URLs before pushing. Push the current intended branch of each
   repository only after local commits and tests succeed. Report both commit hashes and remote
   branches.

6. Recheck the live canonical route after push. Then report completion and celebrate.