docs: record Datamart Builder rollout
This commit is contained in:
User
@@ -0,0 +1,436 @@
|
||||
# Datamart Builder accepted-release cutover implementation plan
|
||||
|
||||
> **Execution rule:** run this plan with `executing-plans`, one task at a time. Stop at both manual
|
||||
> gates. Do not infer approval from silence.
|
||||
|
||||
**Goal:** Make the accepted ThothII release the only application behind the portal's canonical
|
||||
`Datamart Builder` route, remove the legacy release and temporary test route after explicit visual
|
||||
approval, then commit and push only reviewed non-secret changes.
|
||||
|
||||
**Architecture:** The first checkpoint is a routing-only switch: both canonical and test URLs point
|
||||
to the accepted `thothii-test` containers while the legacy containers remain intact. After approval,
|
||||
the accepted images are retagged and recreated as the canonical `thothii` Compose project. Its
|
||||
Qdrant and Ollama services reuse the already validated named volumes as external storage. Django
|
||||
and Nginx then lose every `-test` route, and exact legacy/test resources are removed by labels and
|
||||
IDs. The `/srv/thothii/operator` Compose/runtime files are protected server-local state and are not
|
||||
committed.
|
||||
|
||||
**Technology:** Docker Compose, Nginx, Django 5, React/Vite runtime config, Fastify/Node, Python
|
||||
`unittest`, Vitest, Git.
|
||||
|
||||
---
|
||||
|
||||
## Safety invariants
|
||||
|
||||
- Never print, diff, copy into Git, or inspect the contents of environment/secret files.
|
||||
- Never remove a Docker resource selected only by a substring or broad glob.
|
||||
- Never pass `-v` to Compose cleanup. The accepted Qdrant/Ollama volumes are retained.
|
||||
- Never remove `omics_portal_omics_network` or any DWH, Authentik, Supabase, Superset, Aritmolab,
|
||||
LocalLLM, ETL, or unrelated resource.
|
||||
- Do not touch `/srv/thothii/workspace-registry`, auth roots, session data, Pi state, or vault data.
|
||||
- Phase 1 changes no application containers except rebuilding/restarting the portal web service and
|
||||
reloading portal Nginx.
|
||||
- Stop after Task 5. Resume only after the first explicit visual approval.
|
||||
- Stop after Task 11. Commit/push only after the second explicit visual approval.
|
||||
|
||||
## Task 1: Record immutable baselines and rollback targets
|
||||
|
||||
**Files:** no modifications.
|
||||
|
||||
1. Record the two repository branches, remotes, worktrees, and the current commit without staging
|
||||
anything:
|
||||
|
||||
```bash
|
||||
git -C /home/chirone/ThothII-next status --short
|
||||
git -C /home/chirone/ThothII-next rev-parse HEAD
|
||||
git -C /home/chirone/omics_portal status --short
|
||||
git -C /home/chirone/omics_portal rev-parse HEAD
|
||||
```
|
||||
|
||||
2. Inventory legacy and accepted containers separately by exact Compose project label. Save only
|
||||
IDs, names, image IDs, service labels, mounts, and network names to a mode-0600 temporary
|
||||
directory. Do not capture `Config.Env`:
|
||||
|
||||
```bash
|
||||
evidence_dir="$(mktemp -d /tmp/thothii-cutover.XXXXXX)"
|
||||
chmod 0700 "$evidence_dir"
|
||||
docker ps -a --filter label=com.docker.compose.project=thothii \
|
||||
--format '{{.ID}} {{.Names}} {{.Image}} {{.Label "com.docker.compose.service"}}' \
|
||||
> "$evidence_dir/legacy-containers.txt"
|
||||
docker ps -a --filter label=com.docker.compose.project=thothii-test \
|
||||
--format '{{.ID}} {{.Names}} {{.Image}} {{.Label "com.docker.compose.service"}}' \
|
||||
> "$evidence_dir/accepted-containers.txt"
|
||||
chmod 0600 "$evidence_dir"/*.txt
|
||||
```
|
||||
|
||||
3. Assert the legacy set is exactly `core` and `frontend`; assert the accepted set is exactly
|
||||
`core`, `frontend`, `qdrant`, `embedding`, and the exited `embedding-model-init`. Abort if the
|
||||
inventory differs.
|
||||
|
||||
4. Resolve and retain the four core/frontend image IDs in shell variables or mode-0600 evidence.
|
||||
Confirm the accepted services are healthy and the public test route returns the existing login
|
||||
redirect or authenticated page.
|
||||
|
||||
## Task 2: Make the accepted frontend runtime config work on both temporary paths
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create: `/home/chirone/ThothII-next/scripts/test-datamart-builder-path-config.mjs`
|
||||
- Modify: `/home/chirone/ThothII-next/deploy/thothii-test-config.js`
|
||||
|
||||
1. Write a failing Node test that evaluates the runtime script in isolated VM contexts and asserts:
|
||||
|
||||
- pathname `/datamart-builder/` produces `/datamart-builder/api`;
|
||||
- pathname `/datamart-builder-test/` produces `/datamart-builder-test/api`;
|
||||
- no absolute URL, hostname, credential key, or value is present;
|
||||
- every other pathname fails closed to the canonical same-origin API base.
|
||||
|
||||
2. Run RED:
|
||||
|
||||
```bash
|
||||
node scripts/test-datamart-builder-path-config.mjs
|
||||
```
|
||||
|
||||
Expected: the canonical-path assertion fails because the current file always selects the test
|
||||
prefix.
|
||||
|
||||
3. Change the runtime script to select the test prefix only when
|
||||
`window.location.pathname` starts with `/datamart-builder-test/`; otherwise select the canonical
|
||||
prefix. Keep the only public value as `window.__THOTHII_CONFIG__.backendBaseUrl`.
|
||||
|
||||
4. Run GREEN and the existing frontend URL policy tests:
|
||||
|
||||
```bash
|
||||
node scripts/test-datamart-builder-path-config.mjs
|
||||
cd /home/chirone/ThothII-next/frontend
|
||||
npx vitest run src/api/runtime-config.test.ts
|
||||
npx tsc -b
|
||||
```
|
||||
|
||||
## Task 3: Specify the reversible portal routing switch test-first
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `/home/chirone/omics_portal/kokoro/test_page_titles.py`
|
||||
- Modify: `/home/chirone/omics_portal/kokoro/test_superset_embed.py`
|
||||
- Modify: `/home/chirone/omics_portal/test_thothii_nginx.py`
|
||||
|
||||
1. Add Django assertions that during the temporary dual-route phase:
|
||||
|
||||
- `datamart_builder_public` loads `/datamart-builder/config.js` and canonical asset URLs from the
|
||||
accepted manifest variant;
|
||||
- `datamart_builder_test` still loads `/datamart-builder-test/config.js` and test asset URLs;
|
||||
- both menu entries remain visible to the capability-bearing user;
|
||||
- capability denial remains unchanged.
|
||||
|
||||
2. Extend the static Nginx contract to assert:
|
||||
|
||||
- canonical `/datamart-builder/api/` proxies to `thothii_test_core`;
|
||||
- canonical assets and exact `config.js` proxy to `thothii_test_frontend`;
|
||||
- the canonical API receives the same HTTPS-to-internal-origin normalization as the test API;
|
||||
- Authentik `auth_request`, normalized principal headers, cookie/bearer stripping, SSE buffering,
|
||||
and timeouts remain present;
|
||||
- the test route still exists and still targets the accepted upstreams.
|
||||
|
||||
3. Build a disposable portal test image and run RED without replacing the live portal:
|
||||
|
||||
```bash
|
||||
cd /home/chirone/omics_portal
|
||||
docker compose build web
|
||||
docker compose run --rm --no-deps web \
|
||||
python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
|
||||
docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
|
||||
```
|
||||
|
||||
Expected: canonical accepted-manifest/config and Nginx-upstream assertions fail.
|
||||
|
||||
## Task 4: Implement and deploy the reversible portal switch
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `/home/chirone/omics_portal/kokoro/datamart_catalog_views.py`
|
||||
- Modify: `/home/chirone/omics_portal/kokoro/templatetags/vite.py`
|
||||
- Modify: `/home/chirone/omics_portal/templates/kokoro/datamart_builder.html`
|
||||
- Modify: `/home/chirone/omics_portal/nginx/nginx.conf`
|
||||
- Server-local update: `/srv/thothii/operator/thothii-test-config.js`
|
||||
|
||||
1. Add an accepted/canonical Vite manifest variant: fetch the manifest from
|
||||
`thothii-test-frontend`, but emit `/datamart-builder/assets/` URLs. Keep the test variant
|
||||
unchanged.
|
||||
|
||||
2. Make the canonical view choose the accepted/canonical variant and make the template load
|
||||
`/datamart-builder/config.js`; keep the test view/title/config intact.
|
||||
|
||||
3. In Nginx, point canonical API/assets/config to the test upstreams. Use one origin-normalization
|
||||
map for both accepted routes. Preserve the exact auth and SSE headers.
|
||||
|
||||
4. Run GREEN in disposable containers, then Django system checks:
|
||||
|
||||
```bash
|
||||
cd /home/chirone/omics_portal
|
||||
docker compose build web
|
||||
docker compose run --rm --no-deps web \
|
||||
python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
|
||||
docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
|
||||
docker compose run --rm --no-deps web python manage.py check
|
||||
```
|
||||
|
||||
5. Install the reviewed non-secret path-aware runtime config atomically at the protected operator
|
||||
location, retaining ownership and mode. Do not read or rewrite `server.env`:
|
||||
|
||||
```bash
|
||||
sudo cp /home/chirone/ThothII-next/deploy/thothii-test-config.js \
|
||||
/srv/thothii/operator/thothii-test-config.js
|
||||
```
|
||||
|
||||
6. Deploy only the portal web image, validate Nginx, then reload it:
|
||||
|
||||
```bash
|
||||
cd /home/chirone/omics_portal
|
||||
docker compose up -d --build web
|
||||
docker compose exec nginx nginx -t
|
||||
docker compose restart nginx
|
||||
docker compose ps
|
||||
```
|
||||
|
||||
7. Verify accepted core/frontend health; probe both URLs without credentials; verify Nginx access
|
||||
logs show canonical requests reaching accepted assets/API and no new canonical traffic reaches
|
||||
legacy containers. Never dump backend environment or authentication payloads.
|
||||
|
||||
8. If any automated check fails, restore the four portal files and the previous operator runtime
|
||||
config from the captured diff/copy, rebuild web, validate Nginx, reload, and leave both ThothII
|
||||
projects untouched.
|
||||
|
||||
## Task 5: First manual visual gate — STOP
|
||||
|
||||
Ask the operator to open:
|
||||
|
||||
`https://aritmolab.policlinicosandonato.it/datamart-builder/`
|
||||
|
||||
Required manual checks:
|
||||
|
||||
1. Authentik/portal login succeeds.
|
||||
2. Only the expected user identity/capability is used.
|
||||
3. `psd-clinical` is visible and **Test Workspace Connection** succeeds.
|
||||
4. The selected provider/model are correct.
|
||||
5. A short disposable session starts and produces a response.
|
||||
6. The parallel `/datamart-builder-test/` rollback URL still opens.
|
||||
|
||||
Do not stop/remove/tag any legacy resource before an explicit approval.
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Prepare the canonical accepted Compose deployment after approval
|
||||
|
||||
**Files:**
|
||||
|
||||
- Create server-local: `/srv/thothii/operator/compose.datamart-builder-portal.yaml`
|
||||
- Create server-local: `/srv/thothii/operator/thothii-config.js`
|
||||
- No Git-tracked production file yet.
|
||||
|
||||
1. Re-run Task 1 inventory and compare the exact IDs with the saved baseline. Abort on unexpected
|
||||
drift.
|
||||
|
||||
2. Create a root/operator-protected canonical overlay derived from the tested overlay. It must:
|
||||
|
||||
- use the accepted core/frontend image IDs through canonical tags;
|
||||
- set `AUTH_MODE=upstream` and preserve the already accepted non-secret workspace transport;
|
||||
- attach only core/frontend to `omics_portal_omics_network` with aliases `thothii-core` and
|
||||
`thothii-frontend`;
|
||||
- mount the canonical non-secret runtime config read-only into frontend;
|
||||
- declare `thothii-test_qdrant-data` and `thothii-test_embedding-models` as external named
|
||||
volumes so no accepted index/model state is copied or lost;
|
||||
- contain no secret value.
|
||||
|
||||
3. Write `/srv/thothii/operator/thothii-config.js` with the fixed same-origin base
|
||||
`/datamart-builder/api`. Set a read-only mode suitable for the frontend container.
|
||||
|
||||
4. Validate the complete canonical Compose rendering using the protected existing environment
|
||||
file without printing the rendered config:
|
||||
|
||||
```bash
|
||||
docker compose -p thothii \
|
||||
--env-file /srv/thothii/operator/server.env \
|
||||
-f /srv/thothii/source/ThothII/compose.yaml \
|
||||
-f /srv/thothii/source/ThothII/deploy/compose.server.yaml \
|
||||
-f /srv/thothii/source/ThothII/deploy/compose.git-ssh.yaml \
|
||||
-f /srv/thothii/operator/project-a-private.yaml \
|
||||
-f /srv/thothii/operator/compose.datamart-builder-portal.yaml \
|
||||
config --quiet
|
||||
```
|
||||
|
||||
## Task 7: Promote accepted containers under the canonical Compose identity
|
||||
|
||||
**Files:** runtime only.
|
||||
|
||||
1. Tag the two legacy image IDs with temporary exact rollback tags. Tag the accepted image IDs with
|
||||
canonical `thothii-core:local` and `thothii-frontend:local` only after removing those canonical
|
||||
tags from the legacy images. Do not delete either legacy image ID yet.
|
||||
|
||||
2. Stop accepted services cleanly so Qdrant/Ollama volumes have a single writer. Stop the exact two
|
||||
legacy containers and remove only those two containers, resolving them from the verified label
|
||||
inventory.
|
||||
|
||||
3. Start the complete canonical project from the five-file Compose set in Task 6. Do not build or
|
||||
pull:
|
||||
|
||||
```bash
|
||||
docker compose -p thothii \
|
||||
--env-file /srv/thothii/operator/server.env \
|
||||
-f /srv/thothii/source/ThothII/compose.yaml \
|
||||
-f /srv/thothii/source/ThothII/deploy/compose.server.yaml \
|
||||
-f /srv/thothii/source/ThothII/deploy/compose.git-ssh.yaml \
|
||||
-f /srv/thothii/operator/project-a-private.yaml \
|
||||
-f /srv/thothii/operator/compose.datamart-builder-portal.yaml \
|
||||
up -d --no-build
|
||||
```
|
||||
|
||||
4. Wait for canonical core/frontend/qdrant/embedding health. Verify their exact image IDs, mounts,
|
||||
external-volume names, portal-network aliases, and protected bind roots.
|
||||
|
||||
5. Run loopback workspace diagnostics and one create/delete disposable session against the
|
||||
canonical core using only synthetic normalized principal headers. Retain no content or secret
|
||||
output.
|
||||
|
||||
6. On failure, stop/remove the partial canonical project without `-v`, restore legacy canonical
|
||||
image tags, recreate the old two-service project from `/home/chirone/ThothII`, and restart the
|
||||
accepted test project. Stop the plan and report the failed invariant.
|
||||
|
||||
## Task 8: Specify final removal of the temporary portal route test-first
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `/home/chirone/omics_portal/kokoro/test_page_titles.py`
|
||||
- Modify: `/home/chirone/omics_portal/kokoro/test_superset_embed.py`
|
||||
- Modify: `/home/chirone/omics_portal/test_thothii_nginx.py`
|
||||
|
||||
1. Replace temporary-phase expectations with final-state assertions:
|
||||
|
||||
- the canonical page loads `/datamart-builder/config.js` and canonical assets;
|
||||
- URL reversing `datamart_builder_test` fails and the sidebar has no test label/link;
|
||||
- no `thothii_test_*`, `/datamart-builder-test`, or test origin-map identifier remains in Nginx;
|
||||
- canonical API/assets/config target `thothii_core`/`thothii_frontend`;
|
||||
- Authentik principal and SSE contracts remain exact.
|
||||
|
||||
2. Build the disposable test image and run RED. Expected: all negative test-route assertions fail
|
||||
while the temporary route still exists.
|
||||
|
||||
## Task 9: Remove the temporary portal route and switch Nginx to canonical containers
|
||||
|
||||
**Files:**
|
||||
|
||||
- Modify: `/home/chirone/omics_portal/kokoro/datamart_catalog_views.py`
|
||||
- Modify: `/home/chirone/omics_portal/kokoro/templatetags/vite.py`
|
||||
- Modify: `/home/chirone/omics_portal/omics_portal/urls.py`
|
||||
- Modify: `/home/chirone/omics_portal/templates/kokoro/datamart_builder.html`
|
||||
- Modify: `/home/chirone/omics_portal/templates/partials/left-sidebar.html`
|
||||
- Modify: `/home/chirone/omics_portal/nginx/nginx.conf`
|
||||
|
||||
1. Remove the test view, URL, Vite variant, template branch, menu link, Nginx upstreams, and all
|
||||
`/datamart-builder-test` locations. Restore the canonical manifest lookup to
|
||||
`thothii-frontend`; keep the canonical runtime config script and canonical asset prefix.
|
||||
|
||||
2. Point canonical API/assets/config to `thothii_core` and `thothii_frontend`. Retain the normalized
|
||||
origin mapping if required by the accepted backend's same-origin policy, but give it only a
|
||||
canonical name.
|
||||
|
||||
3. Run the final portal GREEN suite and checks in disposable containers:
|
||||
|
||||
```bash
|
||||
cd /home/chirone/omics_portal
|
||||
docker compose build web
|
||||
docker compose run --rm --no-deps web \
|
||||
python manage.py test kokoro.test_page_titles kokoro.test_superset_embed
|
||||
docker compose run --rm --no-deps web python -m unittest test_thothii_nginx.py
|
||||
docker compose run --rm --no-deps web python manage.py check
|
||||
```
|
||||
|
||||
4. Deploy web, validate Nginx, reload, and probe the canonical URL. Confirm the test URL no longer
|
||||
resolves as a ThothII route and the menu item is absent.
|
||||
|
||||
5. If validation fails, restore canonical Nginx to the still-running canonical accepted project;
|
||||
do not remove test containers or image tags.
|
||||
|
||||
## Task 10: Remove exact legacy and temporary runtime resources
|
||||
|
||||
**Files:**
|
||||
|
||||
- Delete temporary untracked integration file:
|
||||
`/home/chirone/ThothII-next/deploy/compose.datamart-builder-test.yaml`
|
||||
- Delete temporary untracked integration file:
|
||||
`/home/chirone/ThothII-next/deploy/thothii-test-config.js`
|
||||
- Retain server-local canonical overlay/config under `/srv/thothii/operator`.
|
||||
|
||||
1. Confirm canonical portal, core, frontend, Qdrant, and embedding are healthy and that the portal no
|
||||
longer resolves either test upstream name.
|
||||
|
||||
2. Remove the stopped `thothii-test` project containers and its private network using the exact
|
||||
five-file test Compose set. Do not pass `-v`; the two named volumes are external storage for the
|
||||
canonical project.
|
||||
|
||||
3. Remove only the temporary test image tags. Confirm the same accepted image IDs remain reachable
|
||||
through the canonical tags.
|
||||
|
||||
4. Remove the two legacy rollback image tags and exact legacy image IDs only after proving no
|
||||
container references them.
|
||||
|
||||
5. Remove the obsolete legacy `/home/chirone/ThothII` tree only after a final read-only search proves
|
||||
no running container mount, installation descriptor, Compose file, systemd unit, or portal config
|
||||
references it. Use a recoverable trash/move operation when available; otherwise request a final
|
||||
explicit destructive confirmation with the resolved absolute target before recursive deletion.
|
||||
|
||||
6. Remove `/srv/thothii/operator/thothii-test-config.js` after confirming canonical frontend mounts
|
||||
only `thothii-config.js`. Remove no other operator file.
|
||||
|
||||
7. Report exact container/image/network/file targets removed and confirm the retained accepted
|
||||
volume names and canonical containers.
|
||||
|
||||
## Task 11: Full verification and second manual visual gate — STOP
|
||||
|
||||
1. Run targeted ThothII checks for every modified source file:
|
||||
|
||||
```bash
|
||||
cd /home/chirone/ThothII-next/frontend
|
||||
npx vitest run src/api/runtime-config.test.ts
|
||||
npx tsc -b
|
||||
cd /home/chirone/ThothII-next/backend
|
||||
npx vitest run test/app-auth-mode.test.ts test/routes-workspaces.test.ts test/routes-sessions.test.ts
|
||||
npx tsc --noEmit -p .
|
||||
```
|
||||
|
||||
2. Run portal Django/Nginx checks from Task 9 and `docker compose ps`.
|
||||
|
||||
3. Verify final topology:
|
||||
|
||||
- one canonical `thothii` project;
|
||||
- no legacy/test containers, network, or obsolete image ID;
|
||||
- canonical accepted images and healthy services;
|
||||
- no test route/menu/upstream/config;
|
||||
- shared service container IDs and network IDs unchanged from Task 1;
|
||||
- `/datamart-builder/` still redirects unauthenticated clients through the portal login path.
|
||||
|
||||
4. Ask the operator to repeat login, workspace connector, model/provider, short session, and a
|
||||
visual scan at the canonical URL. Stop. Do not stage, commit, or push.
|
||||
|
||||
## Task 12: Review, commit, and push after second approval
|
||||
|
||||
**Files:** both repositories; exact staged sets determined from reviewed diffs.
|
||||
|
||||
1. Inspect each worktree and classify every path as cutover work, earlier approved product fix, user
|
||||
scratch, protected runtime state, or unrelated. Never stage `.superpowers/sdd/progress.md`, Brain
|
||||
scratch, `/srv`, `/tmp`, environment files, API keys, sessions, logs, or generated evidence.
|
||||
|
||||
2. Run `git diff --check`, staged secret scans, and all tests affected by the exact staged files.
|
||||
|
||||
3. In `ThothII-next`, commit approved generic product fixes/tests/docs separately from server-only
|
||||
cleanup. Include this plan and the prior design; exclude deleted temporary files that were never
|
||||
tracked unless their removal is represented by the final intended source state.
|
||||
|
||||
4. In `omics_portal`, commit the canonical Datamart Builder integration and its tests. Confirm no
|
||||
unrelated clinical/data changes are staged.
|
||||
|
||||
5. Inspect all configured push URLs before pushing. Push the current intended branch of each
|
||||
repository only after local commits and tests succeed. Report both commit hashes and remote
|
||||
branches.
|
||||
|
||||
6. Recheck the live canonical route after push. Then report completion and celebrate.
|
||||
@@ -0,0 +1,73 @@
|
||||
# Datamart Builder Test Route Implementation Plan
|
||||
|
||||
**Goal:** Expose the new ThothII release at `/datamart-builder-test/` through the existing portal domain while leaving `/datamart-builder/` unchanged until manual acceptance.
|
||||
|
||||
**Architecture:** Django remains the page and capability gate. The portal Nginx receives the test path and proxies its page/assets/API/SSE to an isolated ThothII test frontend/core pair on the existing Docker network. The test instance trusts the portal's normalized Authentik identity; no local ThothII login is introduced for this server.
|
||||
|
||||
**Tech Stack:** Django, Nginx, Docker Compose, React/Vite, Fastify/TypeScript, existing `omics_portal_omics_network`.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Do not change DNS or the external load balancer.
|
||||
- Preserve `/datamart-builder/` until the user explicitly accepts the test route.
|
||||
- Do not stop or replace the current ThothII instance during the test phase.
|
||||
- Do not expose the canonical authentication root or bypass the Django capability gate.
|
||||
- Use the existing portal Authentik session and normalized principal headers.
|
||||
- Keep secrets and generated runtime state outside Git.
|
||||
|
||||
### Task 1: Isolated test service and prefix contract
|
||||
|
||||
**Files:**
|
||||
- Modify: `compose.yaml` / an approved deployment override used on the server
|
||||
- Modify: `frontend/vite.config.ts` and frontend runtime routing only if required by tests
|
||||
- Test: frontend/backend route and same-origin API tests
|
||||
|
||||
- [ ] Build an isolated test frontend/core service pair with unique service/container names and no host port collision.
|
||||
- [ ] Configure the test frontend public base as `/datamart-builder-test/` and its API contract as `/datamart-builder-test/api/`, or implement an equivalent internal rewrite that preserves the browser same-origin contract.
|
||||
- [ ] Configure the core for trusted upstream identity headers and the public URL `https://aritmolab.policlinicosandonato.it/datamart-builder-test/`.
|
||||
- [ ] Attach only the existing portal Docker network needed for Nginx-to-test-service traffic.
|
||||
- [ ] Run focused frontend/backend tests and render the test Compose configuration without starting the stack.
|
||||
|
||||
### Task 2: Django test page and menu entry
|
||||
|
||||
**Files:**
|
||||
- Modify: `/home/chirone/omics_portal/omics_portal/urls.py`
|
||||
- Modify: `/home/chirone/omics_portal/kokoro/datamart_catalog_views.py` or a focused test view module
|
||||
- Modify: `/home/chirone/omics_portal/templates/partials/left-sidebar.html`
|
||||
- Modify: `/home/chirone/omics_portal/templates/kokoro/datamart_builder.html` or add a test-specific template
|
||||
- Test: Django URL, capability, menu, and template tests
|
||||
|
||||
- [ ] Add `/datamart-builder-test/` using the same capability check as the existing Datamart Builder page.
|
||||
- [ ] Add a visible `Datamart Builder Test` menu item without changing the existing item.
|
||||
- [ ] Ensure the test template emits asset URLs with the test prefix and does not expose credentials.
|
||||
- [ ] Run the focused Django tests and collect static/template validation output.
|
||||
|
||||
### Task 3: Portal Nginx test routing
|
||||
|
||||
**Files:**
|
||||
- Modify: `/home/chirone/omics_portal/nginx/nginx.conf`
|
||||
- Test: `nginx -t` in the portal container and deterministic config/route checks
|
||||
|
||||
- [ ] Add test-path locations for page, assets, API, and SSE.
|
||||
- [ ] Reuse the internal Django auth subrequest and normalized principal headers; clear client-controlled identity, cookie, and authorization headers before the core hop.
|
||||
- [ ] Proxy only to the isolated test frontend/core upstreams.
|
||||
- [ ] Verify `/datamart-builder/` remains byte-for-byte on its existing upstream rules.
|
||||
- [ ] Reload only the portal Nginx after config validation; do not reload the external balancer.
|
||||
|
||||
### Task 4: Manual test checkpoint
|
||||
|
||||
- [ ] Report the exact URL `https://aritmolab.policlinicosandonato.it/datamart-builder-test/` and test procedure.
|
||||
- [ ] Stop implementation and wait for the user's manual acceptance or failure report.
|
||||
|
||||
### Task 5: Cutover after explicit PASS
|
||||
|
||||
- [ ] Repoint the existing `Datamart Builder` Django menu/page and Nginx locations to the accepted new service.
|
||||
- [ ] Preserve the same Authentik capability gate and normalized identity contract.
|
||||
- [ ] Run automated checks and wait for the user's second manual acceptance.
|
||||
|
||||
### Task 6: Remove test-only surface after explicit PASS
|
||||
|
||||
- [ ] Remove the `Datamart Builder Test` menu entry and Django route.
|
||||
- [ ] Remove test-only Nginx locations/upstreams and test service definitions.
|
||||
- [ ] Leave only the accepted new implementation behind `/datamart-builder/`.
|
||||
- [ ] Validate Nginx, Django, Compose, and browser-facing health checks; report the final diff.
|
||||
@@ -0,0 +1,428 @@
|
||||
# Workspace PostgreSQL Diagnostic Alignment Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Make the `psd-clinical` workspace connection diagnostic match the PostgreSQL transport actually used by sessions, then prove that the isolated `/datamart-builder-test/` installation can validate the workspace and create a session without cutting over the production route.
|
||||
|
||||
**Architecture:** Keep the workspace descriptor, credential store, DWH endpoint, portal authentication, and deployment topology unchanged. Correct the concrete Node PostgreSQL probe so TLS is enabled only by explicit TLS bindings and the declared schema is checked with parameterized privilege SQL. Make the already-selected trusted upstream authentication mode diagnostically ready only when its protected session root is valid, so the aggregate workspace contract reflects the authentication path that already admitted the request.
|
||||
|
||||
**Tech Stack:** TypeScript, Node 24, `pg`, Fastify, Vitest, Docker, Docker Compose, Django/Nginx test route.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Work only in `/home/chirone/ThothII-next` for product code; preserve all unrelated dirty-worktree changes.
|
||||
- Do not change the `psd-clinical` workspace descriptor, DWH password, vault, Qdrant, embedding model, Pi provider/model, Django, Nginx, DNS, or external balancer.
|
||||
- Do not print credentials, decrypted secret files, database exception text, cookies, or session content.
|
||||
- Rebuild and recreate only the isolated `thothii-test` core service.
|
||||
- Do not touch the current `/datamart-builder/` route and do not perform the cutover.
|
||||
- Use test-first development: capture the intended focused RED before modifying production code.
|
||||
- Make a commit only from the exact Task 1–3 source/test paths; never stage pre-existing changes or the temporary deployment overlay.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Specify the PostgreSQL diagnostic transport contract
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/test/workspaces-diagnostics.test.ts`
|
||||
- Test: `backend/test/workspaces-diagnostics.test.ts`
|
||||
|
||||
- [ ] Add a small injected PostgreSQL wire-client constructor to the test setup through the planned `ConcreteDiagnosticAdapterDependencies.createPostgresClient` seam. The fake must expose `connect`, `query`, and `end`; it must retain no password after the assertion.
|
||||
|
||||
- [ ] Add `concrete DWH direct diagnostics disable TLS when no TLS binding is declared`:
|
||||
|
||||
```ts
|
||||
const createPostgresClient = vi.fn(() => ({ connect, query, end }));
|
||||
const adapter = createConcreteDiagnosticAdapters({ createPostgresClient });
|
||||
await adapter.probeConnector({
|
||||
role: "dwh",
|
||||
transport: "postgres_direct",
|
||||
host: "127.0.0.1",
|
||||
port: 5432,
|
||||
user: "reader",
|
||||
credentialFile: passwordFile,
|
||||
resource: { database: "warehouse", schema: "datawarehouse" },
|
||||
timeoutMs: 1_000,
|
||||
signal: new AbortController().signal,
|
||||
});
|
||||
expect(createPostgresClient).toHaveBeenCalledWith(expect.objectContaining({ ssl: false }));
|
||||
```
|
||||
|
||||
The fake query result is `{ database: "warehouse", schema: "datawarehouse" }`.
|
||||
|
||||
- [ ] Add `concrete DWH direct diagnostics use strict TLS only for explicit TLS bindings`. Create a temporary CA file, pass both `tlsCaFile` and `tlsServername`, and assert:
|
||||
|
||||
```ts
|
||||
expect(createPostgresClient).toHaveBeenCalledWith(expect.objectContaining({
|
||||
ssl: {
|
||||
ca: "test-ca",
|
||||
servername: "dwh.example.test",
|
||||
rejectUnauthorized: true,
|
||||
},
|
||||
}));
|
||||
```
|
||||
|
||||
- [ ] Strengthen `concrete DWH direct diagnostics authenticate, verify resource identity, and close` by retaining the fake `query` function and asserting both the SQL and values:
|
||||
|
||||
```ts
|
||||
expect(query).toHaveBeenCalledWith(
|
||||
expect.stringContaining("pg_catalog.has_schema_privilege"),
|
||||
["datawarehouse"],
|
||||
);
|
||||
```
|
||||
|
||||
Also assert the SQL contains `$1` and does not contain a quoted/interpolated `datawarehouse` identifier.
|
||||
|
||||
- [ ] Add `concrete DWH direct diagnostics fail closed when the declared schema is inaccessible`. Return `{ database: "warehouse", schema: null }`, expect the fixed internal error `direct probe failed`, and assert `end` was called once. Do not assert or expose a raw PostgreSQL error.
|
||||
|
||||
- [ ] Run the focused RED:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run test/workspaces-diagnostics.test.ts -t 'disable TLS|strict TLS|authenticate, verify resource identity|declared schema is inaccessible'
|
||||
```
|
||||
|
||||
Expected: compilation/test failure because `createPostgresClient` is not yet accepted, followed by behavioral failures if the seam is introduced without the production correction.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Align the concrete PostgreSQL diagnostic with runtime semantics
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/src/workspaces/diagnostics.ts`
|
||||
- Test: `backend/test/workspaces-diagnostics.test.ts`
|
||||
|
||||
- [ ] Import `ClientConfig` from `pg` as a type and define the narrow injected client contract:
|
||||
|
||||
```ts
|
||||
export interface PostgreSqlDiagnosticWireClient {
|
||||
connect(): Promise<void>;
|
||||
query(sql: string, values: readonly unknown[]): Promise<{ rows: Array<Record<string, unknown>> }>;
|
||||
end(): Promise<void>;
|
||||
}
|
||||
```
|
||||
|
||||
Add this optional dependency:
|
||||
|
||||
```ts
|
||||
createPostgresClient?: (config: ClientConfig) => PostgreSqlDiagnosticWireClient;
|
||||
```
|
||||
|
||||
- [ ] In `createConcreteDiagnosticAdapters`, select the injected constructor or the real `pg.Client`:
|
||||
|
||||
```ts
|
||||
const createPostgresClient = dependencies.createPostgresClient
|
||||
?? ((config: ClientConfig): PostgreSqlDiagnosticWireClient => new Client(config));
|
||||
```
|
||||
|
||||
- [ ] Build SSL options from explicit bindings only:
|
||||
|
||||
```ts
|
||||
const tlsConfigured = request.tlsCaFile !== undefined || request.tlsServername !== undefined;
|
||||
const ssl: ClientConfig["ssl"] = tlsConfigured
|
||||
? {
|
||||
...(request.tlsCaFile ? { ca: await readFile(request.tlsCaFile, "utf8") } : {}),
|
||||
...(request.tlsServername ? { servername: request.tlsServername } : {}),
|
||||
rejectUnauthorized: true,
|
||||
}
|
||||
: false;
|
||||
```
|
||||
|
||||
Pass `ssl` to `createPostgresClient`. Do not infer TLS from hostnames and do not add `sslmode` fallbacks.
|
||||
|
||||
- [ ] Replace the `current_schema()` comparison with a parameterized database/schema-access query:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
current_database() AS database,
|
||||
CASE
|
||||
WHEN pg_catalog.has_schema_privilege(
|
||||
current_user,
|
||||
(SELECT oid FROM pg_catalog.pg_namespace WHERE nspname = $1),
|
||||
'USAGE'
|
||||
)
|
||||
THEN $1
|
||||
ELSE NULL
|
||||
END AS schema
|
||||
```
|
||||
|
||||
Call it with `[schema]`. Continue requiring both returned `database === database` and `schema === schema`. This validates the declared namespace without changing `search_path` and without interpolating an identifier.
|
||||
|
||||
- [ ] Keep `tlsVerified: true` on a successful direct probe. In this result type the boolean means that the declared transport-security policy was satisfied: strict verification when TLS was configured, or an explicitly unconfigured/plain connection when it was not.
|
||||
|
||||
- [ ] Run the focused GREEN:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run test/workspaces-diagnostics.test.ts -t 'disable TLS|strict TLS|authenticate, verify resource identity|declared schema is inaccessible'
|
||||
```
|
||||
|
||||
Expected: all selected tests pass.
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Make trusted upstream authentication diagnostic-ready
|
||||
|
||||
**Files:**
|
||||
- Modify: `backend/test/auth-diagnostics.test.ts`
|
||||
- Modify: `backend/src/auth/diagnostics.ts`
|
||||
- Test: `backend/test/auth-diagnostics.test.ts`
|
||||
- Test: `backend/test/routes-workspaces.test.ts`
|
||||
|
||||
- [ ] Add `reports upstream authentication ready when its protected session root is valid`:
|
||||
|
||||
```ts
|
||||
const report = await createAuthDiagnoser({
|
||||
authMode: "upstream",
|
||||
authStateRoot: "/safe/auth-state",
|
||||
sessionRootValidator: async () => undefined,
|
||||
}).inspect({ live: true });
|
||||
expect(report).toEqual({
|
||||
ready: true,
|
||||
mode: "upstream",
|
||||
checks: [{ level: "info", code: "auth_ready", message: "Authentication is ready." }],
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] Add the complementary test `upstream authentication still fails when its protected session root is invalid`. Make the validator throw a synthetic sentinel and assert `ready: false`, exactly one `auth_session_store_invalid` check, and no sentinel in serialized output.
|
||||
|
||||
- [ ] Run the focused RED:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run test/auth-diagnostics.test.ts -t 'upstream authentication'
|
||||
```
|
||||
|
||||
Expected: the valid-root test fails because upstream mode is currently always labelled deprecated and uncertifiable.
|
||||
|
||||
- [ ] Replace the unconditional upstream failure in `createAuthDiagnoser` with the same ordered-check outcome used by `none` and `mock`:
|
||||
|
||||
```ts
|
||||
if (deps.authMode === "upstream") {
|
||||
const result = ordered(checks);
|
||||
return result.length === 0
|
||||
? {
|
||||
ready: true,
|
||||
mode: "upstream",
|
||||
checks: [{ level: "info", code: "auth_ready", message: "Authentication is ready." }],
|
||||
}
|
||||
: { ready: false, mode: "upstream", checks: result };
|
||||
}
|
||||
```
|
||||
|
||||
This does not accept browser-supplied identity headers or alter request authentication. It only makes the diagnostic match the explicitly selected `AUTH_MODE=upstream`; Nginx remains responsible for clearing client identity headers and adding the normalized portal principal.
|
||||
|
||||
- [ ] Run the focused GREEN and route regression:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run test/auth-diagnostics.test.ts -t 'upstream authentication'
|
||||
npx vitest run test/routes-workspaces.test.ts -t 'runs diagnostics for a schema v3 workspace'
|
||||
```
|
||||
|
||||
Expected: all selected tests pass and the aggregate workspace contract can be `activatable: true` only when connector and authentication diagnostics are both ready.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Run backend verification and commit only the correction
|
||||
|
||||
**Files:**
|
||||
- Verify: `backend/src/workspaces/diagnostics.ts`
|
||||
- Verify: `backend/src/auth/diagnostics.ts`
|
||||
- Verify: `backend/test/workspaces-diagnostics.test.ts`
|
||||
- Verify: `backend/test/auth-diagnostics.test.ts`
|
||||
|
||||
- [ ] Run the focused workspace/auth/session regression set:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run \
|
||||
test/workspaces-diagnostics.test.ts \
|
||||
test/routes-workspaces.test.ts \
|
||||
test/auth-diagnostics.test.ts \
|
||||
test/app-auth-mode.test.ts \
|
||||
test/routes-sessions.test.ts
|
||||
```
|
||||
|
||||
- [ ] Run the mandatory TypeScript gate:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx tsc --noEmit -p .
|
||||
```
|
||||
|
||||
- [ ] Run the full backend suite because both diagnostics are production admission dependencies:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
npx vitest run
|
||||
```
|
||||
|
||||
- [ ] Inspect only the scoped diff and whitespace:
|
||||
|
||||
```bash
|
||||
git diff --check
|
||||
git diff -- \
|
||||
backend/src/workspaces/diagnostics.ts \
|
||||
backend/src/auth/diagnostics.ts \
|
||||
backend/test/workspaces-diagnostics.test.ts \
|
||||
backend/test/auth-diagnostics.test.ts
|
||||
git status --short
|
||||
```
|
||||
|
||||
- [ ] Commit exactly those four files, leaving every unrelated modification unstaged:
|
||||
|
||||
```bash
|
||||
git add \
|
||||
backend/src/workspaces/diagnostics.ts \
|
||||
backend/src/auth/diagnostics.ts \
|
||||
backend/test/workspaces-diagnostics.test.ts \
|
||||
backend/test/auth-diagnostics.test.ts
|
||||
git commit -m "fix(workspaces): align postgres connection diagnostics"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Rebuild and recreate only the isolated test core
|
||||
|
||||
**Files:**
|
||||
- Read only: `docker/core.Dockerfile`
|
||||
- Read only: `deploy/compose.datamart-builder-test.yaml`
|
||||
- Read only: `/tmp/thothii-test-server.env`
|
||||
|
||||
- [ ] Confirm the active image/service names and that the current application remains separate:
|
||||
|
||||
```bash
|
||||
docker compose -p thothii-test \
|
||||
--env-file /tmp/thothii-test-server.env \
|
||||
-f /srv/thothii/source/ThothII/compose.yaml \
|
||||
-f /srv/thothii/source/ThothII/deploy/compose.server.yaml \
|
||||
-f /srv/thothii/source/ThothII/deploy/compose.git-ssh.yaml \
|
||||
-f /srv/thothii/operator/project-a-private.yaml \
|
||||
-f /home/chirone/ThothII-next/deploy/compose.datamart-builder-test.yaml \
|
||||
ps
|
||||
```
|
||||
|
||||
- [ ] Build only the test core image from the corrected source:
|
||||
|
||||
```bash
|
||||
docker build -f docker/core.Dockerfile -t thothii-project-a-core:test .
|
||||
```
|
||||
|
||||
- [ ] Recreate only `thothii-test-core-1`; do not recreate frontend, Qdrant, embedding, portal, or current ThothII:
|
||||
|
||||
```bash
|
||||
docker compose -p thothii-test \
|
||||
--env-file /tmp/thothii-test-server.env \
|
||||
-f /srv/thothii/source/ThothII/compose.yaml \
|
||||
-f /srv/thothii/source/ThothII/deploy/compose.server.yaml \
|
||||
-f /srv/thothii/source/ThothII/deploy/compose.git-ssh.yaml \
|
||||
-f /srv/thothii/operator/project-a-private.yaml \
|
||||
-f /home/chirone/ThothII-next/deploy/compose.datamart-builder-test.yaml \
|
||||
up -d --no-deps --force-recreate core
|
||||
```
|
||||
|
||||
- [ ] Wait for health without dumping environment or logs containing payloads:
|
||||
|
||||
```bash
|
||||
docker inspect --format '{{.State.Health.Status}}' thothii-test-core-1
|
||||
docker compose -p thothii-test \
|
||||
--env-file /tmp/thothii-test-server.env \
|
||||
-f /srv/thothii/source/ThothII/compose.yaml \
|
||||
-f /srv/thothii/source/ThothII/deploy/compose.server.yaml \
|
||||
-f /srv/thothii/source/ThothII/deploy/compose.git-ssh.yaml \
|
||||
-f /srv/thothii/operator/project-a-private.yaml \
|
||||
-f /home/chirone/ThothII-next/deploy/compose.datamart-builder-test.yaml \
|
||||
ps
|
||||
```
|
||||
|
||||
Expected: `core` becomes `healthy`; all other test services remain running with their prior container identities.
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Verify the live workspace and one disposable session
|
||||
|
||||
**Files:**
|
||||
- No source modifications.
|
||||
- Runtime scope: `thothii-test-core-1` and `https://aritmolab.policlinicosandonato.it/datamart-builder-test/` only.
|
||||
|
||||
- [ ] Probe the backend through its loopback interface with a synthetic normalized administrator principal. Keep the response in shell memory or a mode-0600 temporary file; never print credentials:
|
||||
|
||||
```bash
|
||||
docker exec thothii-test-core-1 node -e '
|
||||
const headers = {
|
||||
"x-thoth-principal-issuer": "portal",
|
||||
"x-thoth-principal-subject": "diagnostic-workspace-postgres",
|
||||
"x-thoth-principal-display-name": "Diagnostic",
|
||||
"x-thoth-is-admin": "true",
|
||||
"content-type": "application/json",
|
||||
};
|
||||
const response = await fetch("http://127.0.0.1:8787/workspaces/psd-clinical/test", { method: "POST", headers, body: "{}" });
|
||||
const body = await response.json();
|
||||
if (response.status !== 200 || body.activatable !== true || body.authentication?.ready !== true
|
||||
|| body.diagnostics?.some((item) => item.level === "error")) process.exit(1);
|
||||
'
|
||||
```
|
||||
|
||||
- [ ] Confirm the saved model remains available without printing API keys:
|
||||
|
||||
```bash
|
||||
docker exec thothii-test-core-1 node -e '
|
||||
const headers = {
|
||||
"x-thoth-principal-issuer": "portal",
|
||||
"x-thoth-principal-subject": "diagnostic-workspace-postgres",
|
||||
"x-thoth-principal-display-name": "Diagnostic",
|
||||
"x-thoth-is-admin": "true",
|
||||
};
|
||||
const response = await fetch("http://127.0.0.1:8787/models", { headers });
|
||||
const models = await response.json();
|
||||
if (response.status !== 200 || !models.some((item) => item.provider === "deepseek" && item.id === "deepseek-v4-flash")) process.exit(1);
|
||||
'
|
||||
```
|
||||
|
||||
- [ ] Create exactly one disposable session using the saved global workspace/model settings, retain only its returned ID, then delete it immediately:
|
||||
|
||||
```bash
|
||||
docker exec thothii-test-core-1 node -e '
|
||||
const headers = {
|
||||
"x-thoth-principal-issuer": "portal",
|
||||
"x-thoth-principal-subject": "diagnostic-workspace-postgres",
|
||||
"x-thoth-principal-display-name": "Diagnostic",
|
||||
"x-thoth-is-admin": "true",
|
||||
"content-type": "application/json",
|
||||
};
|
||||
const created = await fetch("http://127.0.0.1:8787/sessions", {
|
||||
method: "POST", headers, body: JSON.stringify({ question: "Diagnostic session creation check" }),
|
||||
});
|
||||
const payload = await created.json();
|
||||
if (created.status !== 200 || typeof payload.id !== "string") process.exit(1);
|
||||
const { ["content-type"]: _contentType, ...deleteHeaders } = headers;
|
||||
const removed = await fetch(`http://127.0.0.1:8787/sessions/${encodeURIComponent(payload.id)}`, {
|
||||
method: "DELETE", headers: deleteHeaders,
|
||||
});
|
||||
if (removed.status !== 204) process.exit(1);
|
||||
'
|
||||
```
|
||||
|
||||
- [ ] Verify the public test page still responds through the existing portal route without changing Nginx or Django:
|
||||
|
||||
```bash
|
||||
curl -k -sS -o /dev/null -w '%{http_code}\n' \
|
||||
https://aritmolab.policlinicosandonato.it/datamart-builder-test/
|
||||
```
|
||||
|
||||
Expected: an authenticated portal request reaches the page; an unauthenticated shell probe may return the existing login redirect and must not be treated as a route regression.
|
||||
|
||||
---
|
||||
|
||||
### Task 7: Manual browser checkpoint
|
||||
|
||||
- [ ] Report that only the isolated test core was rebuilt and provide this URL:
|
||||
|
||||
`https://aritmolab.policlinicosandonato.it/datamart-builder-test/`
|
||||
|
||||
- [ ] Ask the user to:
|
||||
1. refresh the page;
|
||||
2. open workspace management and run **Test Workspace Connection** for `psd-clinical`;
|
||||
3. create a new session and submit a short question.
|
||||
|
||||
- [ ] Stop and wait for the user's manual result. Do not repoint `Datamart Builder`, remove the `-test` route, or modify the current ThothII instance.
|
||||
Reference in New Issue
Block a user