437 lines
19 KiB
Markdown
437 lines
19 KiB
Markdown
# Datamart Builder accepted-release cutover implementation plan
|
|
|
|
> **Execution rule:** run this plan with `executing-plans`, one task at a time. Stop at both manual
|
|
> gates. Do not infer approval from silence.
|
|
|
|
**Goal:** Make the accepted ThothII release the only application behind the portal's canonical
|
|
`Datamart Builder` route, remove the legacy release and temporary test route after explicit visual
|
|
approval, then commit and push only reviewed non-secret changes.
|
|
|
|
**Architecture:** The first checkpoint is a routing-only switch: both canonical and test URLs point
|
|
to the accepted `thothii-test` containers while the legacy containers remain intact. After approval,
|
|
the accepted images are retagged and recreated as the canonical `thothii` Compose project. Its
|
|
Qdrant and Ollama services reuse the already validated named volumes as external storage. Django
|
|
and Nginx then lose every `-test` route, and exact legacy/test resources are removed by labels and
|
|
IDs. The `/srv/thothii/operator` Compose/runtime files are protected server-local state and are not
|
|
committed.
|
|
|
|
**Technology:** Docker Compose, Nginx, Django 5, React/Vite runtime config, Fastify/Node, Python
|
|
`unittest`, Vitest, Git.
|
|
|
|
---
|
|
|
|
## Safety invariants
|
|
|
|
- Never print, diff, copy into Git, or inspect the contents of environment/secret files.
|
|
- Never remove a Docker resource selected only by a substring or broad glob.
|
|
- Never pass `-v` to Compose cleanup. The accepted Qdrant/Ollama volumes are retained.
|
|
- Never remove `omics_portal_omics_network` or any DWH, Authentik, Supabase, Superset, Aritmolab,
|
|
LocalLLM, ETL, or unrelated resource.
|
|
- Do not touch `/srv/thothii/workspace-registry`, auth roots, session data, Pi state, or vault data.
|
|
- Phase 1 changes no application containers except rebuilding/restarting the portal web service and
|
|
reloading portal Nginx.
|
|
- Stop after Task 5. Resume only after the first explicit visual approval.
|
|
- Stop after Task 11. Commit/push only after the second explicit visual approval.
|
|
|
|
## Task 1: Record immutable baselines and rollback targets
|
|
|
|
**Files:** no modifications.
|
|
|
|
1. Record the two repository branches, remotes, worktrees, and the current commit without staging
|
|
anything:
|
|
|
|
```bash
|
|
git -C /home/chirone/ThothII-next status --short
|
|
git -C /home/chirone/ThothII-next rev-parse HEAD
|
|
git -C /home/chirone/omics_portal status --short
|
|
git -C /home/chirone/omics_portal rev-parse HEAD
|
|
```
|
|
|
|
2. Inventory legacy and accepted containers separately by exact Compose project label. Save only
|
|
IDs, names, image IDs, service labels, mounts, and network names to a mode-0600 temporary
|
|
directory. Do not capture `Config.Env`:
|
|
|
|
```bash
|
|
evidence_dir="$(mktemp -d /tmp/thothii-cutover.XXXXXX)"
|
|
chmod 0700 "$evidence_dir"
|
|
docker ps -a --filter label=com.docker.compose.project=thothii \
|
|
--format '{{.ID}} {{.Names}} {{.Image}} {{.Label "com.docker.compose.service"}}' \
|
|
> "$evidence_dir/legacy-containers.txt"
|
|
docker ps -a --filter label=com.docker.compose.project=thothii-test \
|
|
--format '{{.ID}} {{.Names}} {{.Image}} {{.Label "com.docker.compose.service"}}' \
|
|
> "$evidence_dir/accepted-containers.txt"
|
|
chmod 0600 "$evidence_dir"/*.txt
|
|
```
|
|
|
|
3. Assert the legacy set is exactly `core` and `frontend`; assert the accepted set is exactly
|
|
`core`, `frontend`, `qdrant`, `embedding`, and the exited `embedding-model-init`. Abort if the
|
|
inventory differs.
|
|
|
|
4. Resolve and retain the four core/frontend image IDs in shell variables or mode-0600 evidence.
|
|
Confirm the accepted services are healthy and the public test route returns the existing login
|
|
redirect or authenticated page.
|
|
|
|
## Task 2: Make the accepted frontend runtime config work on both temporary paths
|
|
|
|
**Files:**
|
|
|
|
- Create: `/home/chirone/ThothII-next/scripts/test-datamart-builder-path-config.mjs`
|
|
- Modify: `/home/chirone/ThothII-next/deploy/thothii-test-config.js`
|
|
|
|
1. Write a failing Node test that evaluates the runtime script in isolated VM contexts and asserts:
|
|
|
|
- pathname `/datamart-builder/` produces `/datamart-builder/api`;
|
|
- pathname `/datamart-builder-test/` produces `/datamart-builder-test/api`;
|
|
- no absolute URL, hostname, credential key, or value is present;
|
|
- every other pathname fails closed to the canonical same-origin API base.
|
|
|
|
2. Run RED:
|
|
|
|
```bash
|
|
node scripts/test-datamart-builder-path-config.mjs
|
|
```
|
|
|
|
Expected: the canonical-path assertion fails because the current file always selects the test
|
|
prefix.
|
|
|
|
3. Change the runtime script to select the test prefix only when
|
|
`window.location.pathname` starts with `/datamart-builder-test/`; otherwise select the canonical
|
|
prefix. Keep the only public value as `window.__THOTHII_CONFIG__.backendBaseUrl`.
|
|
|
|
4. Run GREEN and the existing frontend URL policy tests:
|
|
|
|
```bash
|
|
node scripts/test-datamart-builder-path-config.mjs
|
|
cd /home/chirone/ThothII-next/frontend
|
|
npx vitest run src/api/runtime-config.test.ts
|
|
npx tsc -b
|
|
```
|
|
|
|
## Task 3: Specify the reversible portal routing switch test-first
|
|
|
|
**Files:**
|
|
|
|
- Modify: `/home/chirone/omics_portal/kokoro/test_page_titles.py`
|
|
- Modify: `/home/chirone/omics_portal/kokoro/test_superset_embed.py`
|
|
- Modify: `/home/chirone/omics_portal/test_thothii_nginx.py`
|
|
|
|
1. Add Django assertions that during the temporary dual-route phase:
|
|
|
|
- `datamart_builder_public` loads `/datamart-builder/config.js` and canonical asset URLs from the
|
|
accepted manifest variant;
|
|
- `datamart_builder_test` still loads `/datamart-builder-test/config.js` and test asset URLs;
|
|
- both menu entries remain visible to the capability-bearing user;
|
|
- capability denial remains unchanged.
|
|
|
|
2. Extend the static Nginx contract to assert:
|
|
|
|
- canonical `/datamart-builder/api/` proxies to `thothii_test_core`;
|
|
- canonical assets and exact `config.js` proxy to `thothii_test_frontend`;
|
|
- the canonical API receives the same HTTPS-to-internal-origin normalization as the test API;
|
|
- Authentik `auth_request`, normalized principal headers, cookie/bearer stripping, SSE buffering,
|
|
and timeouts remain present;
|
|
- the test route still exists and still targets the accepted upstreams.
|
|
|
|
3. Build a disposable portal test image and run RED without replacing the live portal:
|
|
|
|
```bash
|
|
cd /home/chirone/omics_portal
|
|
docker compose build web
|
|
docker compose run --rm --no-deps web \
|
|
python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
|
|
docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
|
|
```
|
|
|
|
Expected: canonical accepted-manifest/config and Nginx-upstream assertions fail.
|
|
|
|
## Task 4: Implement and deploy the reversible portal switch
|
|
|
|
**Files:**
|
|
|
|
- Modify: `/home/chirone/omics_portal/kokoro/datamart_catalog_views.py`
|
|
- Modify: `/home/chirone/omics_portal/kokoro/templatetags/vite.py`
|
|
- Modify: `/home/chirone/omics_portal/templates/kokoro/datamart_builder.html`
|
|
- Modify: `/home/chirone/omics_portal/nginx/nginx.conf`
|
|
- Server-local update: `/srv/thothii/operator/thothii-test-config.js`
|
|
|
|
1. Add an accepted/canonical Vite manifest variant: fetch the manifest from
|
|
`thothii-test-frontend`, but emit `/datamart-builder/assets/` URLs. Keep the test variant
|
|
unchanged.
|
|
|
|
2. Make the canonical view choose the accepted/canonical variant and make the template load
|
|
`/datamart-builder/config.js`; keep the test view/title/config intact.
|
|
|
|
3. In Nginx, point canonical API/assets/config to the test upstreams. Use one origin-normalization
|
|
map for both accepted routes. Preserve the exact auth and SSE headers.
|
|
|
|
4. Run GREEN in disposable containers, then Django system checks:
|
|
|
|
```bash
|
|
cd /home/chirone/omics_portal
|
|
docker compose build web
|
|
docker compose run --rm --no-deps web \
|
|
python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
|
|
docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
|
|
docker compose run --rm --no-deps web python manage.py check
|
|
```
|
|
|
|
5. Install the reviewed non-secret path-aware runtime config atomically at the protected operator
|
|
location, retaining ownership and mode. Do not read or rewrite `server.env`:
|
|
|
|
```bash
|
|
sudo cp /home/chirone/ThothII-next/deploy/thothii-test-config.js \
|
|
/srv/thothii/operator/thothii-test-config.js
|
|
```
|
|
|
|
6. Deploy only the portal web image, validate Nginx, then reload it:
|
|
|
|
```bash
|
|
cd /home/chirone/omics_portal
|
|
docker compose up -d --build web
|
|
docker compose exec nginx nginx -t
|
|
docker compose restart nginx
|
|
docker compose ps
|
|
```
|
|
|
|
7. Verify accepted core/frontend health; probe both URLs without credentials; verify Nginx access
|
|
logs show canonical requests reaching accepted assets/API and no new canonical traffic reaches
|
|
legacy containers. Never dump backend environment or authentication payloads.
|
|
|
|
8. If any automated check fails, restore the four portal files and the previous operator runtime
|
|
config from the captured diff/copy, rebuild web, validate Nginx, reload, and leave both ThothII
|
|
projects untouched.
|
|
|
|
## Task 5: First manual visual gate — STOP
|
|
|
|
Ask the operator to open:
|
|
|
|
`https://aritmolab.policlinicosandonato.it/datamart-builder/`
|
|
|
|
Required manual checks:
|
|
|
|
1. Authentik/portal login succeeds.
|
|
2. Only the expected user identity/capability is used.
|
|
3. `psd-clinical` is visible and **Test Workspace Connection** succeeds.
|
|
4. The selected provider/model are correct.
|
|
5. A short disposable session starts and produces a response.
|
|
6. The parallel `/datamart-builder-test/` rollback URL still opens.
|
|
|
|
Do not stop/remove/tag any legacy resource before an explicit approval.
|
|
|
|
---
|
|
|
|
## Task 6: Prepare the canonical accepted Compose deployment after approval
|
|
|
|
**Files:**
|
|
|
|
- Create server-local: `/srv/thothii/operator/compose.datamart-builder-portal.yaml`
|
|
- Create server-local: `/srv/thothii/operator/thothii-config.js`
|
|
- No Git-tracked production file yet.
|
|
|
|
1. Re-run Task 1 inventory and compare the exact IDs with the saved baseline. Abort on unexpected
|
|
drift.
|
|
|
|
2. Create a root/operator-protected canonical overlay derived from the tested overlay. It must:
|
|
|
|
- use the accepted core/frontend image IDs through canonical tags;
|
|
- set `AUTH_MODE=upstream` and preserve the already accepted non-secret workspace transport;
|
|
- attach only core/frontend to `omics_portal_omics_network` with aliases `thothii-core` and
|
|
`thothii-frontend`;
|
|
- mount the canonical non-secret runtime config read-only into frontend;
|
|
- declare `thothii-test_qdrant-data` and `thothii-test_embedding-models` as external named
|
|
volumes so no accepted index/model state is copied or lost;
|
|
- contain no secret value.
|
|
|
|
3. Write `/srv/thothii/operator/thothii-config.js` with the fixed same-origin base
|
|
`/datamart-builder/api`. Set a read-only mode suitable for the frontend container.
|
|
|
|
4. Validate the complete canonical Compose rendering using the protected existing environment
|
|
file without printing the rendered config:
|
|
|
|
```bash
|
|
docker compose -p thothii \
|
|
--env-file /srv/thothii/operator/server.env \
|
|
-f /srv/thothii/source/ThothII/compose.yaml \
|
|
-f /srv/thothii/source/ThothII/deploy/compose.server.yaml \
|
|
-f /srv/thothii/source/ThothII/deploy/compose.git-ssh.yaml \
|
|
-f /srv/thothii/operator/project-a-private.yaml \
|
|
-f /srv/thothii/operator/compose.datamart-builder-portal.yaml \
|
|
config --quiet
|
|
```
|
|
|
|
## Task 7: Promote accepted containers under the canonical Compose identity
|
|
|
|
**Files:** runtime only.
|
|
|
|
1. Tag the two legacy image IDs with temporary exact rollback tags. Tag the accepted image IDs with
|
|
canonical `thothii-core:local` and `thothii-frontend:local` only after removing those canonical
|
|
tags from the legacy images. Do not delete either legacy image ID yet.
|
|
|
|
2. Stop accepted services cleanly so Qdrant/Ollama volumes have a single writer. Stop the exact two
|
|
legacy containers and remove only those two containers, resolving them from the verified label
|
|
inventory.
|
|
|
|
3. Start the complete canonical project from the five-file Compose set in Task 6. Do not build or
|
|
pull:
|
|
|
|
```bash
|
|
docker compose -p thothii \
|
|
--env-file /srv/thothii/operator/server.env \
|
|
-f /srv/thothii/source/ThothII/compose.yaml \
|
|
-f /srv/thothii/source/ThothII/deploy/compose.server.yaml \
|
|
-f /srv/thothii/source/ThothII/deploy/compose.git-ssh.yaml \
|
|
-f /srv/thothii/operator/project-a-private.yaml \
|
|
-f /srv/thothii/operator/compose.datamart-builder-portal.yaml \
|
|
up -d --no-build
|
|
```
|
|
|
|
4. Wait for canonical core/frontend/qdrant/embedding health. Verify their exact image IDs, mounts,
|
|
external-volume names, portal-network aliases, and protected bind roots.
|
|
|
|
5. Run loopback workspace diagnostics and one create/delete disposable session against the
|
|
canonical core using only synthetic normalized principal headers. Retain no content or secret
|
|
output.
|
|
|
|
6. On failure, stop/remove the partial canonical project without `-v`, restore legacy canonical
|
|
image tags, recreate the old two-service project from `/home/chirone/ThothII`, and restart the
|
|
accepted test project. Stop the plan and report the failed invariant.
|
|
|
|
## Task 8: Specify final removal of the temporary portal route test-first
|
|
|
|
**Files:**
|
|
|
|
- Modify: `/home/chirone/omics_portal/kokoro/test_page_titles.py`
|
|
- Modify: `/home/chirone/omics_portal/kokoro/test_superset_embed.py`
|
|
- Modify: `/home/chirone/omics_portal/test_thothii_nginx.py`
|
|
|
|
1. Replace temporary-phase expectations with final-state assertions:
|
|
|
|
- the canonical page loads `/datamart-builder/config.js` and canonical assets;
|
|
- URL reversing `datamart_builder_test` fails and the sidebar has no test label/link;
|
|
- no `thothii_test_*`, `/datamart-builder-test`, or test origin-map identifier remains in Nginx;
|
|
- canonical API/assets/config target `thothii_core`/`thothii_frontend`;
|
|
- Authentik principal and SSE contracts remain exact.
|
|
|
|
2. Build the disposable test image and run RED. Expected: all negative test-route assertions fail
|
|
while the temporary route still exists.
|
|
|
|
## Task 9: Remove the temporary portal route and switch Nginx to canonical containers
|
|
|
|
**Files:**
|
|
|
|
- Modify: `/home/chirone/omics_portal/kokoro/datamart_catalog_views.py`
|
|
- Modify: `/home/chirone/omics_portal/kokoro/templatetags/vite.py`
|
|
- Modify: `/home/chirone/omics_portal/omics_portal/urls.py`
|
|
- Modify: `/home/chirone/omics_portal/templates/kokoro/datamart_builder.html`
|
|
- Modify: `/home/chirone/omics_portal/templates/partials/left-sidebar.html`
|
|
- Modify: `/home/chirone/omics_portal/nginx/nginx.conf`
|
|
|
|
1. Remove the test view, URL, Vite variant, template branch, menu link, Nginx upstreams, and all
|
|
`/datamart-builder-test` locations. Restore the canonical manifest lookup to
|
|
`thothii-frontend`; keep the canonical runtime config script and canonical asset prefix.
|
|
|
|
2. Point canonical API/assets/config to `thothii_core` and `thothii_frontend`. Retain the normalized
|
|
origin mapping if required by the accepted backend's same-origin policy, but give it only a
|
|
canonical name.
|
|
|
|
3. Run the final portal GREEN suite and checks in disposable containers:
|
|
|
|
```bash
|
|
cd /home/chirone/omics_portal
|
|
docker compose build web
|
|
docker compose run --rm --no-deps web \
|
|
python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
|
|
docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
|
|
docker compose run --rm --no-deps web python manage.py check
|
|
```
|
|
|
|
4. Deploy web, validate Nginx, reload, and probe the canonical URL. Confirm the test URL no longer
|
|
resolves as a ThothII route and the menu item is absent.
|
|
|
|
5. If validation fails, restore canonical Nginx to the still-running canonical accepted project;
|
|
do not remove test containers or image tags.
|
|
|
|
## Task 10: Remove exact legacy and temporary runtime resources
|
|
|
|
**Files:**
|
|
|
|
- Delete temporary untracked integration file:
|
|
`/home/chirone/ThothII-next/deploy/compose.datamart-builder-test.yaml`
|
|
- Delete temporary untracked integration file:
|
|
`/home/chirone/ThothII-next/deploy/thothii-test-config.js`
|
|
- Retain server-local canonical overlay/config under `/srv/thothii/operator`.
|
|
|
|
1. Confirm canonical portal, core, frontend, Qdrant, and embedding are healthy and that the portal no
|
|
longer resolves either test upstream name.
|
|
|
|
2. Remove the stopped `thothii-test` project containers and its private network using the exact
|
|
five-file test Compose set. Do not pass `-v`; the two named volumes are external storage for the
|
|
canonical project.
|
|
|
|
3. Remove only the temporary test image tags. Confirm the same accepted image IDs remain reachable
|
|
through the canonical tags.
|
|
|
|
4. Remove the two legacy rollback image tags and exact legacy image IDs only after proving no
|
|
container references them.
|
|
|
|
5. Remove the obsolete legacy `/home/chirone/ThothII` tree only after a final read-only search proves
|
|
no running container mount, installation descriptor, Compose file, systemd unit, or portal config
|
|
references it. Use a recoverable trash/move operation when available; otherwise request a final
|
|
explicit destructive confirmation with the resolved absolute target before recursive deletion.
|
|
|
|
6. Remove `/srv/thothii/operator/thothii-test-config.js` after confirming canonical frontend mounts
|
|
only `thothii-config.js`. Remove no other operator file.
|
|
|
|
7. Report exact container/image/network/file targets removed and confirm the retained accepted
|
|
volume names and canonical containers.
|
|
|
|
## Task 11: Full verification and second manual visual gate — STOP
|
|
|
|
1. Run targeted ThothII checks for every modified source file:
|
|
|
|
```bash
|
|
cd /home/chirone/ThothII-next/frontend
|
|
npx vitest run src/api/runtime-config.test.ts
|
|
npx tsc -b
|
|
cd /home/chirone/ThothII-next/backend
|
|
npx vitest run test/app-auth-mode.test.ts test/routes-workspaces.test.ts test/routes-sessions.test.ts
|
|
npx tsc --noEmit -p .
|
|
```
|
|
|
|
2. Run portal Django/Nginx checks from Task 9 and `docker compose ps`.
|
|
|
|
3. Verify final topology:
|
|
|
|
- one canonical `thothii` project;
|
|
- no legacy/test containers, network, or obsolete image ID;
|
|
- canonical accepted images and healthy services;
|
|
- no test route/menu/upstream/config;
|
|
- shared service container IDs and network IDs unchanged from Task 1;
|
|
- `/datamart-builder/` still redirects unauthenticated clients through the portal login path.
|
|
|
|
4. Ask the operator to repeat login, workspace connector, model/provider, short session, and a
|
|
visual scan at the canonical URL. Stop. Do not stage, commit, or push.
|
|
|
|
## Task 12: Review, commit, and push after second approval
|
|
|
|
**Files:** both repositories; exact staged sets determined from reviewed diffs.
|
|
|
|
1. Inspect each worktree and classify every path as cutover work, earlier approved product fix, user
|
|
scratch, protected runtime state, or unrelated. Never stage `.superpowers/sdd/progress.md`, Brain
|
|
scratch, `/srv`, `/tmp`, environment files, API keys, sessions, logs, or generated evidence.
|
|
|
|
2. Run `git diff --check`, staged secret scans, and all tests affected by the exact staged files.
|
|
|
|
3. In `ThothII-next`, commit approved generic product fixes/tests/docs separately from server-only
|
|
cleanup. Include this plan and the prior design; exclude deleted temporary files that were never
|
|
tracked unless their removal is represented by the final intended source state.
|
|
|
|
4. In `omics_portal`, commit the canonical Datamart Builder integration and its tests. Confirm no
|
|
unrelated clinical/data changes are staged.
|
|
|
|
5. Inspect all configured push URLs before pushing. Push the current intended branch of each
|
|
repository only after local commits and tests succeed. Report both commit hashes and remote
|
|
branches.
|
|
|
|
6. Recheck the live canonical route after push. Then report completion and celebrate.
|