ThothII/docs/superpowers/plans/2026-08-22-datamart-builder-cutover.md

Datamart Builder accepted-release cutover implementation plan

Execution rule: run this plan with executing-plans, one task at a time. Stop at both manual gates. Do not infer approval from silence.

Goal: Make the accepted ThothII release the only application behind the portal's canonical Datamart Builder route, remove the legacy release and temporary test route after explicit visual approval, then commit and push only reviewed non-secret changes.

Architecture: The first checkpoint is a routing-only switch: both canonical and test URLs point to the accepted thothii-test containers while the legacy containers remain intact. After approval, the accepted images are retagged and recreated as the canonical thothii Compose project. Its Qdrant and Ollama services reuse the already validated named volumes as external storage. Django and Nginx then lose every -test route, and exact legacy/test resources are removed by labels and IDs. The /srv/thothii/operator Compose/runtime files are protected server-local state and are not committed.

Technology: Docker Compose, Nginx, Django 5, React/Vite runtime config, Fastify/Node, Python unittest, Vitest, Git.

---

Safety invariants

• Never print, diff, copy into Git, or inspect the contents of environment/secret files.
• Never remove a Docker resource selected only by a substring or broad glob.
• Never pass -v to Compose cleanup. The accepted Qdrant/Ollama volumes are retained.
• Never remove omics_portal_omics_network or any DWH, Authentik, Supabase, Superset, Aritmolab, LocalLLM, ETL, or unrelated resource.
• Do not touch /srv/thothii/workspace-registry, auth roots, session data, Pi state, or vault data.
• Phase 1 changes no application containers except rebuilding/restarting the portal web service and reloading portal Nginx.
• Stop after Task 5. Resume only after the first explicit visual approval.
• Stop after Task 11. Commit/push only after the second explicit visual approval.

Task 1: Record immutable baselines and rollback targets

Files: no modifications.

1. Record the two repository branches, remotes, worktrees, and the current commit without staging anything:

   git -C /home/chirone/Thoth status --short
   git -C /home/chirone/Thoth rev-parse HEAD
   git -C /home/chirone/omics_portal status --short
   git -C /home/chirone/omics_portal rev-parse HEAD
   

2. Inventory legacy and accepted containers separately by exact Compose project label. Save only IDs, names, image IDs, service labels, mounts, and network names to a mode-0600 temporary directory. Do not capture Config.Env:

   evidence_dir="$(mktemp -d /tmp/thothii-cutover.XXXXXX)"
   chmod 0700 "$evidence_dir"
   docker ps -a --filter label=com.docker.compose.project=thothii \
     --format '{{.ID}} {{.Names}} {{.Image}} {{.Label "com.docker.compose.service"}}' \
     > "$evidence_dir/legacy-containers.txt"
   docker ps -a --filter label=com.docker.compose.project=thothii-test \
     --format '{{.ID}} {{.Names}} {{.Image}} {{.Label "com.docker.compose.service"}}' \
     > "$evidence_dir/accepted-containers.txt"
   chmod 0600 "$evidence_dir"/*.txt
   

3. Assert the legacy set is exactly core and frontend; assert the accepted set is exactly core, frontend, qdrant, embedding, and the exited embedding-model-init. Abort if the inventory differs.

4. Resolve and retain the four core/frontend image IDs in shell variables or mode-0600 evidence. Confirm the accepted services are healthy and the public test route returns the existing login redirect or authenticated page.

Task 2: Make the accepted frontend runtime config work on both temporary paths

Files:

• Create: /home/chirone/Thoth/scripts/test-datamart-builder-path-config.mjs
• Modify: /home/chirone/Thoth/deploy/thothii-test-config.js

1. Write a failing Node test that evaluates the runtime script in isolated VM contexts and asserts:

   • pathname /datamart-builder/ produces /datamart-builder/api;
   • pathname /datamart-builder-test/ produces /datamart-builder-test/api;
   • no absolute URL, hostname, credential key, or value is present;
   • every other pathname fails closed to the canonical same-origin API base.

2. Run RED:

   node scripts/test-datamart-builder-path-config.mjs
   

   Expected: the canonical-path assertion fails because the current file always selects the test prefix.

3. Change the runtime script to select the test prefix only when window.location.pathname starts with /datamart-builder-test/; otherwise select the canonical prefix. Keep the only public value as window.__THOTHII_CONFIG__.backendBaseUrl.

4. Run GREEN and the existing frontend URL policy tests:

   node scripts/test-datamart-builder-path-config.mjs
   cd /home/chirone/Thoth/frontend
   npx vitest run src/api/runtime-config.test.ts
   npx tsc -b
   

Task 3: Specify the reversible portal routing switch test-first

Files:

• Modify: /home/chirone/omics_portal/kokoro/test_page_titles.py
• Modify: /home/chirone/omics_portal/kokoro/test_superset_embed.py
• Modify: /home/chirone/omics_portal/test_thothii_nginx.py

1. Add Django assertions that during the temporary dual-route phase:

   • datamart_builder_public loads /datamart-builder/config.js and canonical asset URLs from the accepted manifest variant;
   • datamart_builder_test still loads /datamart-builder-test/config.js and test asset URLs;
   • both menu entries remain visible to the capability-bearing user;
   • capability denial remains unchanged.

2. Extend the static Nginx contract to assert:

   • canonical /datamart-builder/api/ proxies to thothii_test_core;
   • canonical assets and exact config.js proxy to thothii_test_frontend;
   • the canonical API receives the same HTTPS-to-internal-origin normalization as the test API;
   • Authentik auth_request, normalized principal headers, cookie/bearer stripping, SSE buffering, and timeouts remain present;
   • the test route still exists and still targets the accepted upstreams.

3. Build a disposable portal test image and run RED without replacing the live portal:

   cd /home/chirone/omics_portal
   docker compose build web
   docker compose run --rm --no-deps web \
     python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
   docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
   

   Expected: canonical accepted-manifest/config and Nginx-upstream assertions fail.

Task 4: Implement and deploy the reversible portal switch

Files:

• Modify: /home/chirone/omics_portal/kokoro/datamart_catalog_views.py
• Modify: /home/chirone/omics_portal/kokoro/templatetags/vite.py
• Modify: /home/chirone/omics_portal/templates/kokoro/datamart_builder.html
• Modify: /home/chirone/omics_portal/nginx/nginx.conf
• Server-local update: /srv/thothii/operator/thothii-test-config.js

1. Add an accepted/canonical Vite manifest variant: fetch the manifest from thothii-test-frontend, but emit /datamart-builder/assets/ URLs. Keep the test variant unchanged.

2. Make the canonical view choose the accepted/canonical variant and make the template load /datamart-builder/config.js; keep the test view/title/config intact.

3. In Nginx, point canonical API/assets/config to the test upstreams. Use one origin-normalization map for both accepted routes. Preserve the exact auth and SSE headers.

4. Run GREEN in disposable containers, then Django system checks:

   cd /home/chirone/omics_portal
   docker compose build web
   docker compose run --rm --no-deps web \
     python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
   docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
   docker compose run --rm --no-deps web python manage.py check
   

5. Install the reviewed non-secret path-aware runtime config atomically at the protected operator location, retaining ownership and mode. Do not read or rewrite server.env:

   sudo cp /home/chirone/Thoth/deploy/thothii-test-config.js \
     /srv/thothii/operator/thothii-test-config.js
   

6. Deploy only the portal web image, validate Nginx, then reload it:

   cd /home/chirone/omics_portal
   docker compose up -d --build web
   docker compose exec nginx nginx -t
   docker compose restart nginx
   docker compose ps
   

7. Verify accepted core/frontend health; probe both URLs without credentials; verify Nginx access logs show canonical requests reaching accepted assets/API and no new canonical traffic reaches legacy containers. Never dump backend environment or authentication payloads.

8. If any automated check fails, restore the four portal files and the previous operator runtime config from the captured diff/copy, rebuild web, validate Nginx, reload, and leave both ThothII projects untouched.

Task 5: First manual visual gate — STOP

Ask the operator to open:

https://aritmolab.policlinicosandonato.it/datamart-builder/

Required manual checks:

1. Authentik/portal login succeeds.
2. Only the expected user identity/capability is used.
3. psd-clinical is visible and Test Workspace Connection succeeds.
4. The selected provider/model are correct.
5. A short disposable session starts and produces a response.
6. The parallel /datamart-builder-test/ rollback URL still opens.

Do not stop/remove/tag any legacy resource before an explicit approval.

---

Task 6: Prepare the canonical accepted Compose deployment after approval

Files:

• Create server-local: /srv/thothii/operator/compose.datamart-builder-portal.yaml
• Create server-local: /srv/thothii/operator/thothii-config.js
• No Git-tracked production file yet.

1. Re-run Task 1 inventory and compare the exact IDs with the saved baseline. Abort on unexpected drift.

2. Create a root/operator-protected canonical overlay derived from the tested overlay. It must:

   • use the accepted core/frontend image IDs through canonical tags;
   • set AUTH_MODE=upstream and preserve the already accepted non-secret workspace transport;
   • attach only core/frontend to omics_portal_omics_network with aliases thothii-core and thothii-frontend;
   • mount the canonical non-secret runtime config read-only into frontend;
   • declare thothii-test_qdrant-data and thothii-test_embedding-models as external named volumes so no accepted index/model state is copied or lost;
   • contain no secret value.

3. Write /srv/thothii/operator/thothii-config.js with the fixed same-origin base /datamart-builder/api. Set a read-only mode suitable for the frontend container.

4. Validate the complete canonical Compose rendering using the protected existing environment file without printing the rendered config:

   docker compose -p thothii \
     --env-file /srv/thothii/operator/server.env \
     -f /srv/thothii/source/ThothII/compose.yaml \
     -f /srv/thothii/source/ThothII/deploy/compose.server.yaml \
     -f /srv/thothii/source/ThothII/deploy/compose.git-ssh.yaml \
     -f /srv/thothii/operator/project-a-private.yaml \
     -f /srv/thothii/operator/compose.datamart-builder-portal.yaml \
     config --quiet
   

Task 7: Promote accepted containers under the canonical Compose identity

Files: runtime only.

1. Tag the two legacy image IDs with temporary exact rollback tags. Tag the accepted image IDs with canonical thothii-core:local and thothii-frontend:local only after removing those canonical tags from the legacy images. Do not delete either legacy image ID yet.

2. Stop accepted services cleanly so Qdrant/Ollama volumes have a single writer. Stop the exact two legacy containers and remove only those two containers, resolving them from the verified label inventory.

3. Start the complete canonical project from the five-file Compose set in Task 6. Do not build or pull:

   docker compose -p thothii \
     --env-file /srv/thothii/operator/server.env \
     -f /srv/thothii/source/ThothII/compose.yaml \
     -f /srv/thothii/source/ThothII/deploy/compose.server.yaml \
     -f /srv/thothii/source/ThothII/deploy/compose.git-ssh.yaml \
     -f /srv/thothii/operator/project-a-private.yaml \
     -f /srv/thothii/operator/compose.datamart-builder-portal.yaml \
     up -d --no-build
   

4. Wait for canonical core/frontend/qdrant/embedding health. Verify their exact image IDs, mounts, external-volume names, portal-network aliases, and protected bind roots.

5. Run loopback workspace diagnostics and one create/delete disposable session against the canonical core using only synthetic normalized principal headers. Retain no content or secret output.

6. On failure, stop/remove the partial canonical project without -v, restore legacy canonical image tags, recreate the old two-service project from /home/chirone/ThothII, and restart the accepted test project. Stop the plan and report the failed invariant.

Task 8: Specify final removal of the temporary portal route test-first

Files:

• Modify: /home/chirone/omics_portal/kokoro/test_page_titles.py
• Modify: /home/chirone/omics_portal/kokoro/test_superset_embed.py
• Modify: /home/chirone/omics_portal/test_thothii_nginx.py

1. Replace temporary-phase expectations with final-state assertions:

   • the canonical page loads /datamart-builder/config.js and canonical assets;
   • URL reversing datamart_builder_test fails and the sidebar has no test label/link;
   • no thothii_test_*, /datamart-builder-test, or test origin-map identifier remains in Nginx;
   • canonical API/assets/config target thothii_core/thothii_frontend;
   • Authentik principal and SSE contracts remain exact.

2. Build the disposable test image and run RED. Expected: all negative test-route assertions fail while the temporary route still exists.

Task 9: Remove the temporary portal route and switch Nginx to canonical containers

Files:

• Modify: /home/chirone/omics_portal/kokoro/datamart_catalog_views.py
• Modify: /home/chirone/omics_portal/kokoro/templatetags/vite.py
• Modify: /home/chirone/omics_portal/omics_portal/urls.py
• Modify: /home/chirone/omics_portal/templates/kokoro/datamart_builder.html
• Modify: /home/chirone/omics_portal/templates/partials/left-sidebar.html
• Modify: /home/chirone/omics_portal/nginx/nginx.conf

1. Remove the test view, URL, Vite variant, template branch, menu link, Nginx upstreams, and all /datamart-builder-test locations. Restore the canonical manifest lookup to thothii-frontend; keep the canonical runtime config script and canonical asset prefix.

2. Point canonical API/assets/config to thothii_core and thothii_frontend. Retain the normalized origin mapping if required by the accepted backend's same-origin policy, but give it only a canonical name.

3. Run the final portal GREEN suite and checks in disposable containers:

   cd /home/chirone/omics_portal
   docker compose build web
   docker compose run --rm --no-deps web \
     python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
   docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
   docker compose run --rm --no-deps web python manage.py check
   

4. Deploy web, validate Nginx, reload, and probe the canonical URL. Confirm the test URL no longer resolves as a ThothII route and the menu item is absent.

5. If validation fails, restore canonical Nginx to the still-running canonical accepted project; do not remove test containers or image tags.

Task 10: Remove exact legacy and temporary runtime resources

Files:

• Delete temporary untracked integration file: /home/chirone/Thoth/deploy/compose.datamart-builder-test.yaml
• Delete temporary untracked integration file: /home/chirone/Thoth/deploy/thothii-test-config.js
• Retain server-local canonical overlay/config under /srv/thothii/operator.

1. Confirm canonical portal, core, frontend, Qdrant, and embedding are healthy and that the portal no longer resolves either test upstream name.

2. Remove the stopped thothii-test project containers and its private network using the exact five-file test Compose set. Do not pass -v; the two named volumes are external storage for the canonical project.

3. Remove only the temporary test image tags. Confirm the same accepted image IDs remain reachable through the canonical tags.

4. Remove the two legacy rollback image tags and exact legacy image IDs only after proving no container references them.

5. Remove the obsolete legacy /home/chirone/ThothII tree only after a final read-only search proves no running container mount, installation descriptor, Compose file, systemd unit, or portal config references it. Use a recoverable trash/move operation when available; otherwise request a final explicit destructive confirmation with the resolved absolute target before recursive deletion.

6. Remove /srv/thothii/operator/thothii-test-config.js after confirming canonical frontend mounts only thothii-config.js. Remove no other operator file.

7. Report exact container/image/network/file targets removed and confirm the retained accepted volume names and canonical containers.

Task 11: Full verification and second manual visual gate — STOP

1. Run targeted ThothII checks for every modified source file:

   cd /home/chirone/Thoth/frontend
   npx vitest run src/api/runtime-config.test.ts
   npx tsc -b
   cd /home/chirone/Thoth/backend
   npx vitest run test/app-auth-mode.test.ts test/routes-workspaces.test.ts test/routes-sessions.test.ts
   npx tsc --noEmit -p .
   

2. Run portal Django/Nginx checks from Task 9 and docker compose ps.

3. Verify final topology:

   • one canonical thothii project;
   • no legacy/test containers, network, or obsolete image ID;
   • canonical accepted images and healthy services;
   • no test route/menu/upstream/config;
   • shared service container IDs and network IDs unchanged from Task 1;
   • /datamart-builder/ still redirects unauthenticated clients through the portal login path.

4. Ask the operator to repeat login, workspace connector, model/provider, short session, and a visual scan at the canonical URL. Stop. Do not stage, commit, or push.

Task 12: Review, commit, and push after second approval

Files: both repositories; exact staged sets determined from reviewed diffs.

1. Inspect each worktree and classify every path as cutover work, earlier approved product fix, user scratch, protected runtime state, or unrelated. Never stage .superpowers/sdd/progress.md, Brain scratch, /srv, /tmp, environment files, API keys, sessions, logs, or generated evidence.

2. Run git diff --check, staged secret scans, and all tests affected by the exact staged files.

3. In Thoth, commit approved generic product fixes/tests/docs separately from server-only cleanup. Include this plan and the prior design; exclude deleted temporary files that were never tracked unless their removal is represented by the final intended source state.

4. In omics_portal, commit the canonical Datamart Builder integration and its tests. Confirm no unrelated clinical/data changes are staged.

5. Inspect all configured push URLs before pushing. Push the current intended branch of each repository only after local commits and tests succeed. Report both commit hashes and remote branches.

6. Recheck the live canonical route after push. Then report completion and celebrate.