diff --git a/docs/superpowers/specs/2026-08-22-datamart-builder-cutover-design.md b/docs/superpowers/specs/2026-08-22-datamart-builder-cutover-design.md new file mode 100644 index 00000000..b62e72cf --- /dev/null +++ b/docs/superpowers/specs/2026-08-22-datamart-builder-cutover-design.md @@ -0,0 +1,148 @@ +# Datamart Builder — staged cutover to the accepted ThothII release + +Date: 2026-08-22 + +## Outcome + +Promote the ThothII release already accepted at `/datamart-builder-test/` to the canonical +`/datamart-builder/` entry point, retain a reversible checkpoint for the first visual test, then +remove the legacy release and the temporary test route only after the two explicit operator gates. + +The completed topology has one portal menu entry (`Datamart Builder`), one canonical route +(`/datamart-builder/`), and one ThothII Compose application running the accepted release. + +## Preconditions and authority + +- The new release has already passed browser login through the portal/Authentik path. +- Workspace discovery, `psd-clinical`, connector diagnostics, DeepSeek, Pi, and session creation + have been exercised successfully on the test route. +- The operator has explicitly authorized the staged cutover and, after the first visual approval, + deletion of the disconnected legacy containers and images because no legacy state must be + retained. +- The operator has chosen to keep `/datamart-builder-test/` active during the first visual + checkpoint as a rollback route. + +This authority does not extend to shared infrastructure: Omics/LocalLLM networks, DWH, +`dwh-auth`, Supabase, Authentik, Superset, Aritmolab, portal secrets, and unrelated containers or +images are never cleanup targets. + +## Current topology + +The portal currently exposes two independent paths: + +- `/datamart-builder/` reaches the legacy `thothii` Compose project (`thothii-core` and + `thothii-frontend`). +- `/datamart-builder-test/` reaches the accepted `thothii-test` project + (`thothii-test-core` and `thothii-test-frontend`). + +Both paths use the portal's existing Authentik admission flow. Nginx forwards the normalized +identity headers to the selected backend. The accepted release uses the server workspace and +secret layout under the protected `/srv/thothii` installation; those runtime values are not copied +into Git. + +## Phase 1 — reversible canonical switch + +Phase 1 changes routing only; it does not delete runtime resources. + +1. Update the canonical Django page and asset selection so `Datamart Builder` renders the accepted + frontend while retaining the canonical browser URL. +2. Point the canonical Nginx API, static asset, runtime-config, and streaming paths to the accepted + `thothii-test` frontend/core upstreams. +3. Preserve the existing Authentik subrequest and the exact normalized principal-header contract. +4. Keep `/datamart-builder-test/` and its `Datamart Builder Test` menu entry operational. +5. Leave the legacy `thothii` containers running but disconnected from canonical portal traffic. + +The runtime browser configuration must be path-aware during this temporary dual-route phase: the +canonical page uses `/datamart-builder/api`, while the test page uses +`/datamart-builder-test/api`. No frontend build may hard-code a server hostname or credential. + +### Phase 1 verification + +Before asking for the visual test: + +- run the portal Django tests covering the two pages, menu visibility, page titles, asset variants, + and Authentik behavior; +- run the Nginx routing tests and validate the live Nginx configuration before reload; +- validate the accepted Compose configuration and confirm frontend/core health; +- probe both public paths without credentials and confirm that admission still redirects through + Authentik rather than bypassing it; +- verify that the canonical API and asset requests now reach the accepted containers; +- verify that the test route remains a working rollback path; +- verify that the legacy containers receive no canonical application traffic after the switch. + +The operator then performs the first visual test at +`https://aritmolab.policlinicosandonato.it/datamart-builder/`, including login, workspace selection, +connector check, and creation of a short disposable session. + +### Phase 1 rollback + +Until the first visual approval, rollback consists only of restoring the prior Django/Nginx route +selection and reloading validated configuration. The legacy containers remain intact, so no image +rebuild or data recovery is required. + +## Phase 2 — canonicalize the accepted stack and remove legacy/test topology + +Phase 2 starts only after the operator explicitly approves the first visual test. + +1. Capture an exact inventory of the legacy `thothii` project resources, image IDs, mounts, + networks, and labels. Resolve cleanup targets by Compose project labels and exact IDs, never by + a broad name pattern or recursive directory deletion. +2. Stop and remove only the legacy `thothii` core/frontend containers and their obsolete images. + Remove no shared volume or network. Remove an old source/config directory only if the inventory + proves that the accepted installation does not mount or reference it and the target is entirely + legacy. +3. Recreate the accepted release under the canonical `thothii` Compose identity. It continues to + consume the approved `/srv/thothii` installation, workspace registry, authentication material, + and external service configuration; it must not inherit old `/home/chirone/ThothII` runtime + state. +4. Keep the accepted `thothii-test` project available until the canonical replacement is healthy + and the canonical Nginx route has passed automated probes. This bounds the promotion downtime + and preserves a known-good recovery target during recreation. +5. Point canonical Nginx upstreams at the canonical accepted containers, validate and reload. +6. Remove the `Datamart Builder Test` menu item, Django URL/view/asset variant, Nginx test routes + and upstreams, test-only runtime configuration, and test Compose overlay. +7. Stop and remove the now-redundant `thothii-test` containers and test-tagged images only after + confirming that canonical core/frontend are healthy and no longer depend on them. + +If canonical recreation or verification fails, leave the test project running and restore the +canonical Nginx route to that accepted upstream. Do not continue cleanup while the canonical route +is unhealthy. + +### Phase 2 verification + +Before asking for the second visual test: + +- confirm there is exactly one ThothII application project serving the portal; +- confirm `/datamart-builder/` authenticates through Authentik and reaches the accepted release; +- confirm `/datamart-builder-test/` and its menu entry are absent; +- confirm workspace discovery, connector diagnostics, SSE/session creation, and model/provider + configuration still work; +- confirm no legacy ThothII container or obsolete image remains; +- confirm all shared services and networks are unchanged and healthy; +- rerun the portal, Nginx, backend/frontend, installation, and focused acceptance gates appropriate + to the changed files. + +The operator then performs the second visual test on the canonical URL. No commit or push follows +until that approval. + +## Source control and evidence + +After the second visual approval: + +- inspect both repositories independently (`ThothII-next` and `omics_portal`); +- stage only the reviewed source/config/test/documentation files belonging to this work; +- exclude protected environment files, `/srv` runtime state, API keys, generated session data, + local logs, temporary evidence, `.superpowers/sdd/progress.md`, and unrelated user changes; +- run final tests and secret/diff checks on the exact staged sets; +- create clear repository-specific commits and push only their current intended branches; +- report commit hashes, pushed branches, verification evidence, and the exact runtime resources + removed. + +## Manual gates + +The execution intentionally pauses twice: + +1. after the reversible canonical route switch and before any legacy deletion; +2. after legacy/test cleanup and before commit/push. + +Silence or a partial test is not approval. Each continuation requires an explicit operator result.