diff --git a/docs/plans/2026-08-20-psd-server-deployment-program-design.md b/docs/plans/2026-08-20-psd-server-deployment-program-design.md new file mode 100644 index 00000000..190e8dfd --- /dev/null +++ b/docs/plans/2026-08-20-psd-server-deployment-program-design.md @@ -0,0 +1,348 @@ +# PSD Server Deployment Program — Design + +**Date:** 2026-08-20 + +**Status:** Approved by the owner + +**Design-time application baseline:** `main` at `5c0dc8c` (execution must freeze and record the +then-current `origin/main` SHA) + +**Design-time workspace baseline:** `tht-workspace-psd/main` at `bfbabf9` (execution must freeze and +record the then-current remote SHA) + +## Purpose + +Replace the unused legacy ThothII installation on the PSD server with the current application, +recovering only useful configuration and rebuilding runtime state from canonical sources. Complete +the work as two independently accepted projects: + +1. deploy and prove ThothII with local authentication, a direct read-only PSD DWH connection, + internal Qdrant, and internal Ollama; +2. only after Project A passes, integrate the accepted installation with the server's Authentik, + Nginx, load balancer, and the existing Aritmolab sidebar link. + +The program must be executable by the Sol LLM from a terminal local to the server. It must test +each step, retain redacted evidence, stop on unsafe uncertainty, and include a separate human +manual-test document for each project. + +## Binding decisions + +- The legacy application may be unavailable for days. Service continuity is not a goal. +- The new source clone is prepared beside the old source directory. The old and new stacks are not + kept running simultaneously: inventory and backup happen first, the old stack is stopped, and + only then is the new stack started. +- Preserve the old directory, configuration, and volumes as a recovery source until both projects + pass. Do not migrate legacy application sessions, Qdrant data, Ollama caches, or derived indexes. +- Recover configuration only: endpoints, non-secret policy, provider/model selection, relevant + paths, and references to protected credentials. Never copy an old setting without validating it + against the current contract. +- Use the canonical ThothII Compose distribution and reviewed installation-local overrides. Do not + modify an unrelated global Compose project or blindly adapt the legacy Compose file. +- Lifecycle operations use the installation-aware native `tht` CLI, not raw Compose commands. +- Never use `docker compose down --volumes`, global prune operations, broad recursive deletion, or + secret-bearing command arguments. + +## Repository and workspace model + +There are two Git sources with different responsibilities: + +- `ThothII` contains application code and non-secret deployment examples. +- `tht-workspace-psd` contains the shared, credential-free PSD workspace source. + +The workspace repository contains one logical workspace, `psd-clinical`. Its schema-v3 descriptor +will declare both supported DWH transports: + +```yaml +supported_transports: [rest_api, postgres_direct] +``` + +The Mac installation continues to select `rest_api`. The PSD server selects `postgres_direct`. +Endpoint values, user names, passwords, secret-file paths, machine paths, and Git credentials stay +in each installation's protected bindings and never enter the workspace repository. Evidence, +curated annotations, language, model policy, and semantic-index contract remain shared. + +Each installation owns its own Qdrant collection contents, Ollama model cache, preprocessing state, +and sessions even though both consume the same reviewed workspace commit. + +## Program structure + +The program consists of one non-mutating common survey followed by two independently gated +projects: + +```text +Common Survey PASS + -> Project A automated PASS + -> Project A human PASS + -> explicit authorization + -> Project B automated PASS + -> Project B human PASS + -> final cutover acceptance +``` + +Project B must not start from a partial or assumed Project A result. + +## Common Survey + +The survey is a prerequisite, not a third implementation project. It runs before any server +mutation and produces a redacted report, topology map, change-scope inventory, unknowns list, and +GO/NO-GO decision. + +Sol inventories from the server-local terminal: + +- operating system, architecture, Docker and Compose versions, available CPU/RAM/disk, and clock; +- legacy ThothII source, SHA, dirty state, images, containers, networks, ports, volumes, mounts, + health, installation state, and recovery state; +- effective Compose rendering and ownership of every relevant file; +- DWH database/schema, direct listener, TLS, read-only role, and reachability from containers; +- Supabase/PostgreSQL topology, schema conventions, exposed PostgREST schemas, migration policy, + backup mechanism, and suitable role boundaries; +- Pi provider/model policy, credential references, external LLM reachability, and current versions; +- Nginx effective configuration, ThothII virtual host/location, upstream, forwarded headers, SSE + settings, certificate metadata, certificate generation/renewal, and rollback files; +- load-balancer routes, health checks, allowlist capability, TLS boundary, and configuration owner; +- Aritmolab deployment, networks, homepage, sidebar source, current ThothII destination, and release + procedure; +- Authentik version, deployment, current Aritmolab integration, provider conventions, group + conventions, backup/export procedure, API access, and credential locations; +- public DNS/origin that the final sidebar link must preserve; +- protected files by path, ownership, mode, and readability only, without printing their contents. + +The survey may use hashes, metadata, redacted renders, and permission checks. It must not emit +passwords, bearer tokens, API keys, cookies, OIDC client secrets, private keys, password hashes, or +raw identity tokens. If credential discovery fails, the owner may help locate the existing +Authentik credentials. + +## Project A — Standalone Server Acceptance + +### Runtime architecture + +Project A installs a clean five-service stack: + +```text +operator terminal or local headless browser + -> frontend + -> core + Pi + workflow harness + -> direct read-only PSD PostgreSQL DWH + -> internal Qdrant + -> internal Ollama (qwen3-embedding:0.6b, 1024 dimensions) + -> local filesystem work-session storage +``` + +The stack uses local authentication. It must not be publicly reachable. Core, Qdrant, and Ollama +remain private; the frontend binds to loopback unless the optional restricted test route below is +proved safe. + +### Preparation and cutover + +1. Freeze exact application and workspace SHAs and require clean source trees. +2. Publish and validate the multi-transport `psd-clinical` descriptor through the curator workflow. +3. Prove the Mac installation still selects REST and remains valid. +4. Prepare the new source clone and protected operator/runtime directories beside the old source. +5. Extract only approved configuration facts from the legacy installation. +6. Create and verify backups and a restart recipe for the legacy stack. +7. Stop the legacy stack without deleting its source, configuration, images, or volumes. +8. Build/install the current native `tht` and application images from the frozen source. +9. Configure local authentication, direct DWH bindings, Pi/provider access, and the Git workspace. +10. Start through `tht`, then validate health, authentication, workspace, workflow, and Pi. +11. Rebuild schema/Evidence preprocessing, Qdrant contents, and the Ollama model cache from + canonical sources. Prove the complete preprocessing rerun is idempotent. + +### Optional restricted test route + +If and only if the survey proves that the load balancer can enforce a test-operator allowlist +before a request reaches ThothII, Project A may use a temporary private hostname to exercise the +same network path as production: + +```text +authorized operator -> allowlisted load-balancer route -> Nginx -> frontend -> core/local auth +``` + +The route has a distinct hostname, no Aritmolab sidebar link, a certificate created by the existing +managed mechanism, correct forwarded-origin and SSE behavior, and a negative test from an +unauthorized source. Its local-auth `publicUrl` matches the private test origin. It is not a public +production route and must be removed after Project B. + +If isolation cannot be demonstrated, Sol must not approximate it or misdeclare +`THOTH_PUBLIC_EXPOSURE=false`; tests run against loopback from the local terminal instead. + +### Acceptance + +Project A requires: + +- exact source identity and reproducible build evidence; +- healthy frontend, core, Qdrant, embedding, and completed model initializer; +- local authentication checks including admin/user separation, wrong password, logout, + disable/enable, invalidation, and restart persistence; +- direct DWH connectivity with a demonstrably read-only runtime identity; +- active `psd-clinical` at the expected Git revision; +- compatible Qdrant collection/index contract and correct Ollama model/dimensions; +- complete and idempotent DWH, schema, annotation, and Evidence preprocessing; +- one harmless real PSD work session completed through F1-F8, ending in read-only validated SQL; +- persisted manifest, artifacts, reviewer decisions, and final SQL inspection; +- optional private-route positive and negative isolation evidence when that route is used; +- completed human manual-test report with an explicit PASS. + +Project A does not modify the production Aritmolab sidebar, production public route, or Authentik. + +## Project B — Authentik and Aritmolab Integration + +Project B begins only from the frozen, accepted Project A source, images, workspace revision, and +PASS report. + +### Final request and data flow + +```text +user + -> aritmolab.policlinicosandonato.com + -> Aritmolab homepage/sidebar + -> load balancer + -> Nginx/TLS + -> ThothII frontend and same-origin /api + -> ThothII OIDC Authorization Code + PKCE with Authentik + -> PostgreSQL thoth_sessions schema for owned work sessions + -> direct read-only datawarehouse schema for clinical queries + -> internal Qdrant and Ollama +``` + +Nginx terminates/proxies according to the observed deployment but does not add a second +`auth_request` in front of ThothII. ThothII performs generic OIDC directly. Nginx preserves the +public host and HTTPS scheme, forwards the callback path unchanged, and supports SSE without +buffering or premature timeouts. + +### Authentik configuration + +Before mutation, export or back up the relevant Authentik configuration. Locate existing protected +administrative/API credentials without exposing them. Create or adapt: + +- one OAuth2/OIDC provider and one ThothII application; +- the exact callback `/api/auth/oidc/callback`; +- `openid`, `profile`, and `email` scopes; +- a direct, non-empty JSON string array claim named `groups`; +- exact user/admin group mappings selected after the survey; +- a separate group-view-only service account/API token for ThothII diagnostics. + +Dedicated `TOT Users` and `TOT Admin` groups are the default unless the survey finds existing groups +with exactly the intended semantics and the owner approves their reuse. Additional groups are +ignored. Missing, malformed, indirect, or ambiguous configured groups fail closed. + +### Supabase session storage + +Do not create a separate PostgreSQL database. Use the server's existing Supabase PostgreSQL +database and isolate ThothII work sessions in the dedicated `thoth_sessions` schema. This schema is +distinct from the clinical `datawarehouse` schema. + +- A one-shot migrator role owns only the required schema migration privileges. +- Core receives only the restricted runtime role, never the migrator credential. +- Forced RLS and application ownership checks isolate sessions by Authentik principal. +- The schema stores principals/preferences, manifests, phase artifacts, review decisions, and + audit records. +- Chat and live SSE output remain ephemeral; semantic vectors remain in Qdrant; clinical data + remains in `datawarehouse`; browser authentication sessions remain in protected auth state. +- `thoth_sessions` must not be added to Supabase/PostgREST exposed schemas. +- The direct runtime connection uses the TLS/CA contract required by the current application. + +### Safe activation order + +1. Back up the accepted Project A operator/auth configuration and every external configuration to + be changed. +2. Prepare Authentik objects without exposing the new route. +3. Run and verify additive session-schema migrations; require no pending or drifted migrations. +4. Prepare OIDC secrets and non-secret configuration in protected installation state. +5. Validate Authentik discovery, issuer/JWKS, catalog access, and mapped groups. +6. Validate Nginx, certificate, load-balancer route, callback, forwarded headers, and SSE while + production traffic remains closed. +7. Start ThothII in OIDC/public server mode with PostgreSQL session storage. +8. Open the final load-balancer route. +9. Preserve or update the Aritmolab sidebar link so the established user journey remains intact. +10. Complete automated and human acceptance, then remove the Project A temporary route. + +### Acceptance + +Project B requires: + +- successful redacted static, live, and interactive authentication diagnostics; +- trusted certificate chain, correct public origin, callback, and proxy headers; +- proven load-balancer/Nginx routing and SSE operation; +- successful migration status, RLS/role tests, and proof that `thoth_sessions` is not REST-exposed; +- ordinary, administrator, unmapped, malformed-claim, logout, and controlled provider-failure + cases; +- login to Aritmolab followed by the sidebar link to ThothII without a second credential prompt; +- no raw OIDC token in browser storage, logs, diagnostics, or evidence; +- session ownership and administrator-boundary tests; +- one harmless F1-F8 PSD session under an OIDC identity; +- tested rollback and a completed human manual-test report with an explicit PASS. + +## Error handling and stop rules + +Every executable step follows: + +```text +precondition -> action -> verification -> redacted evidence -> checkpoint +``` + +Sol stops and requests owner help rather than improvising when it encounters: + +- a dirty or unidentified source checkout; +- uncertain ownership of Compose, Nginx, load-balancer, Aritmolab, or Authentik configuration; +- missing or insufficient credentials; +- an unsafe secret path or risk of secret disclosure; +- an unverifiable backup or rollback path; +- a DWH identity that is not demonstrably read-only; +- a change that would affect unrelated Nginx virtual hosts or other applications; +- an unprovable temporary-route restriction; +- Supabase migration drift, excessive roles, or unintended REST exposure; +- a material difference between the surveyed server and this design. + +Unchanged external state is not failure. Sol records the observation and continues only when the +current gate is satisfied. + +## Rollback boundaries + +- **Project A:** stop the new stack and restart the preserved legacy installation. No new volume is + copied into the legacy installation. +- **Project B ingress:** close the public route first, then restore the prior Nginx, + load-balancer, certificate reference, and sidebar configuration. +- **Project B application:** return to the accepted Project A local-auth operator configuration + while the public route remains closed. +- **Authentik:** initially disable new objects instead of deleting them; retain the pre-change + export until final acceptance. +- **Supabase:** migrations are additive. Rollback does not automatically drop `thoth_sessions` or + destroy evidence; destructive cleanup requires a separate explicit decision. +- **Workspace Git:** publish the multi-transport change as an isolated commit and retain the + previous revision. A rejected candidate never replaces the installation's last valid snapshot. + +## Documents and evidence + +The implementation-planning phase creates: + +1. a general execution program with cross-project gates; +2. a survey checklist and report template; +3. an executable Project A plan for Sol; +4. a plain-language Project A manual-test guide; +5. a Project A evidence/PASS template; +6. an executable Project B plan for Sol; +7. a plain-language Project B manual-test guide; +8. a Project B evidence/PASS template. + +Sol maintains a protected server-local progress journal and resumes from the last verified +checkpoint. Detailed topology and command output remain in a protected evidence directory on the +server. Only intentionally redacted reports and reusable templates may enter Git. + +The Project A human guide is terminal-first and may use a local headless browser. When the optional +private endpoint exists, it also includes operator browser checks. The Project B guide covers the +real Aritmolab homepage/sidebar, Authentik SSO, roles, logout, final workflow, and negative cases. + +## Known documentation reconciliation + +Some older server documentation describes vector and embedding services as external and the +mandatory stack as only frontend/core. Current `compose.yaml`, repository instructions, and project +state define Qdrant and Ollama as mandatory internal services. The execution plans must treat the +current code/Compose contract as authoritative and include a documentation correction rather than +following the stale statements. + +## Success condition + +The program is complete only when both projects have exact-source evidence, all automated gates +pass, both human manuals are completed with explicit PASS decisions, the Aritmolab sidebar reaches +the final ThothII URL through the established load balancer and Nginx, Authentik provides SSO, and a +real read-only PSD session completes F1-F8 under an authorized OIDC identity.