docs: design PSD server deployment program
This commit is contained in:
@@ -0,0 +1,348 @@
|
||||
# PSD Server Deployment Program — Design
|
||||
|
||||
**Date:** 2026-08-20
|
||||
|
||||
**Status:** Approved by the owner
|
||||
|
||||
**Design-time application baseline:** `main` at `5c0dc8c` (execution must freeze and record the
|
||||
then-current `origin/main` SHA)
|
||||
|
||||
**Design-time workspace baseline:** `tht-workspace-psd/main` at `bfbabf9` (execution must freeze and
|
||||
record the then-current remote SHA)
|
||||
|
||||
## Purpose
|
||||
|
||||
Replace the unused legacy ThothII installation on the PSD server with the current application,
|
||||
recovering only useful configuration and rebuilding runtime state from canonical sources. Complete
|
||||
the work as two independently accepted projects:
|
||||
|
||||
1. deploy and prove ThothII with local authentication, a direct read-only PSD DWH connection,
|
||||
internal Qdrant, and internal Ollama;
|
||||
2. only after Project A passes, integrate the accepted installation with the server's Authentik,
|
||||
Nginx, load balancer, and the existing Aritmolab sidebar link.
|
||||
|
||||
The program must be executable by the Sol LLM from a terminal local to the server. It must test
|
||||
each step, retain redacted evidence, stop on unsafe uncertainty, and include a separate human
|
||||
manual-test document for each project.
|
||||
|
||||
## Binding decisions
|
||||
|
||||
- The legacy application may be unavailable for days. Service continuity is not a goal.
|
||||
- The new source clone is prepared beside the old source directory. The old and new stacks are not
|
||||
kept running simultaneously: inventory and backup happen first, the old stack is stopped, and
|
||||
only then is the new stack started.
|
||||
- Preserve the old directory, configuration, and volumes as a recovery source until both projects
|
||||
pass. Do not migrate legacy application sessions, Qdrant data, Ollama caches, or derived indexes.
|
||||
- Recover configuration only: endpoints, non-secret policy, provider/model selection, relevant
|
||||
paths, and references to protected credentials. Never copy an old setting without validating it
|
||||
against the current contract.
|
||||
- Use the canonical ThothII Compose distribution and reviewed installation-local overrides. Do not
|
||||
modify an unrelated global Compose project or blindly adapt the legacy Compose file.
|
||||
- Lifecycle operations use the installation-aware native `tht` CLI, not raw Compose commands.
|
||||
- Never use `docker compose down --volumes`, global prune operations, broad recursive deletion, or
|
||||
secret-bearing command arguments.
|
||||
|
||||
## Repository and workspace model
|
||||
|
||||
There are two Git sources with different responsibilities:
|
||||
|
||||
- `ThothII` contains application code and non-secret deployment examples.
|
||||
- `tht-workspace-psd` contains the shared, credential-free PSD workspace source.
|
||||
|
||||
The workspace repository contains one logical workspace, `psd-clinical`. Its schema-v3 descriptor
|
||||
will declare both supported DWH transports:
|
||||
|
||||
```yaml
|
||||
supported_transports: [rest_api, postgres_direct]
|
||||
```
|
||||
|
||||
The Mac installation continues to select `rest_api`. The PSD server selects `postgres_direct`.
|
||||
Endpoint values, user names, passwords, secret-file paths, machine paths, and Git credentials stay
|
||||
in each installation's protected bindings and never enter the workspace repository. Evidence,
|
||||
curated annotations, language, model policy, and semantic-index contract remain shared.
|
||||
|
||||
Each installation owns its own Qdrant collection contents, Ollama model cache, preprocessing state,
|
||||
and sessions even though both consume the same reviewed workspace commit.
|
||||
|
||||
## Program structure
|
||||
|
||||
The program consists of one non-mutating common survey followed by two independently gated
|
||||
projects:
|
||||
|
||||
```text
|
||||
Common Survey PASS
|
||||
-> Project A automated PASS
|
||||
-> Project A human PASS
|
||||
-> explicit authorization
|
||||
-> Project B automated PASS
|
||||
-> Project B human PASS
|
||||
-> final cutover acceptance
|
||||
```
|
||||
|
||||
Project B must not start from a partial or assumed Project A result.
|
||||
|
||||
## Common Survey
|
||||
|
||||
The survey is a prerequisite, not a third implementation project. It runs before any server
|
||||
mutation and produces a redacted report, topology map, change-scope inventory, unknowns list, and
|
||||
GO/NO-GO decision.
|
||||
|
||||
Sol inventories from the server-local terminal:
|
||||
|
||||
- operating system, architecture, Docker and Compose versions, available CPU/RAM/disk, and clock;
|
||||
- legacy ThothII source, SHA, dirty state, images, containers, networks, ports, volumes, mounts,
|
||||
health, installation state, and recovery state;
|
||||
- effective Compose rendering and ownership of every relevant file;
|
||||
- DWH database/schema, direct listener, TLS, read-only role, and reachability from containers;
|
||||
- Supabase/PostgreSQL topology, schema conventions, exposed PostgREST schemas, migration policy,
|
||||
backup mechanism, and suitable role boundaries;
|
||||
- Pi provider/model policy, credential references, external LLM reachability, and current versions;
|
||||
- Nginx effective configuration, ThothII virtual host/location, upstream, forwarded headers, SSE
|
||||
settings, certificate metadata, certificate generation/renewal, and rollback files;
|
||||
- load-balancer routes, health checks, allowlist capability, TLS boundary, and configuration owner;
|
||||
- Aritmolab deployment, networks, homepage, sidebar source, current ThothII destination, and release
|
||||
procedure;
|
||||
- Authentik version, deployment, current Aritmolab integration, provider conventions, group
|
||||
conventions, backup/export procedure, API access, and credential locations;
|
||||
- public DNS/origin that the final sidebar link must preserve;
|
||||
- protected files by path, ownership, mode, and readability only, without printing their contents.
|
||||
|
||||
The survey may use hashes, metadata, redacted renders, and permission checks. It must not emit
|
||||
passwords, bearer tokens, API keys, cookies, OIDC client secrets, private keys, password hashes, or
|
||||
raw identity tokens. If credential discovery fails, the owner may help locate the existing
|
||||
Authentik credentials.
|
||||
|
||||
## Project A — Standalone Server Acceptance
|
||||
|
||||
### Runtime architecture
|
||||
|
||||
Project A installs a clean five-service stack:
|
||||
|
||||
```text
|
||||
operator terminal or local headless browser
|
||||
-> frontend
|
||||
-> core + Pi + workflow harness
|
||||
-> direct read-only PSD PostgreSQL DWH
|
||||
-> internal Qdrant
|
||||
-> internal Ollama (qwen3-embedding:0.6b, 1024 dimensions)
|
||||
-> local filesystem work-session storage
|
||||
```
|
||||
|
||||
The stack uses local authentication. It must not be publicly reachable. Core, Qdrant, and Ollama
|
||||
remain private; the frontend binds to loopback unless the optional restricted test route below is
|
||||
proved safe.
|
||||
|
||||
### Preparation and cutover
|
||||
|
||||
1. Freeze exact application and workspace SHAs and require clean source trees.
|
||||
2. Publish and validate the multi-transport `psd-clinical` descriptor through the curator workflow.
|
||||
3. Prove the Mac installation still selects REST and remains valid.
|
||||
4. Prepare the new source clone and protected operator/runtime directories beside the old source.
|
||||
5. Extract only approved configuration facts from the legacy installation.
|
||||
6. Create and verify backups and a restart recipe for the legacy stack.
|
||||
7. Stop the legacy stack without deleting its source, configuration, images, or volumes.
|
||||
8. Build/install the current native `tht` and application images from the frozen source.
|
||||
9. Configure local authentication, direct DWH bindings, Pi/provider access, and the Git workspace.
|
||||
10. Start through `tht`, then validate health, authentication, workspace, workflow, and Pi.
|
||||
11. Rebuild schema/Evidence preprocessing, Qdrant contents, and the Ollama model cache from
|
||||
canonical sources. Prove the complete preprocessing rerun is idempotent.
|
||||
|
||||
### Optional restricted test route
|
||||
|
||||
If and only if the survey proves that the load balancer can enforce a test-operator allowlist
|
||||
before a request reaches ThothII, Project A may use a temporary private hostname to exercise the
|
||||
same network path as production:
|
||||
|
||||
```text
|
||||
authorized operator -> allowlisted load-balancer route -> Nginx -> frontend -> core/local auth
|
||||
```
|
||||
|
||||
The route has a distinct hostname, no Aritmolab sidebar link, a certificate created by the existing
|
||||
managed mechanism, correct forwarded-origin and SSE behavior, and a negative test from an
|
||||
unauthorized source. Its local-auth `publicUrl` matches the private test origin. It is not a public
|
||||
production route and must be removed after Project B.
|
||||
|
||||
If isolation cannot be demonstrated, Sol must not approximate it or misdeclare
|
||||
`THOTH_PUBLIC_EXPOSURE=false`; tests run against loopback from the local terminal instead.
|
||||
|
||||
### Acceptance
|
||||
|
||||
Project A requires:
|
||||
|
||||
- exact source identity and reproducible build evidence;
|
||||
- healthy frontend, core, Qdrant, embedding, and completed model initializer;
|
||||
- local authentication checks including admin/user separation, wrong password, logout,
|
||||
disable/enable, invalidation, and restart persistence;
|
||||
- direct DWH connectivity with a demonstrably read-only runtime identity;
|
||||
- active `psd-clinical` at the expected Git revision;
|
||||
- compatible Qdrant collection/index contract and correct Ollama model/dimensions;
|
||||
- complete and idempotent DWH, schema, annotation, and Evidence preprocessing;
|
||||
- one harmless real PSD work session completed through F1-F8, ending in read-only validated SQL;
|
||||
- persisted manifest, artifacts, reviewer decisions, and final SQL inspection;
|
||||
- optional private-route positive and negative isolation evidence when that route is used;
|
||||
- completed human manual-test report with an explicit PASS.
|
||||
|
||||
Project A does not modify the production Aritmolab sidebar, production public route, or Authentik.
|
||||
|
||||
## Project B — Authentik and Aritmolab Integration
|
||||
|
||||
Project B begins only from the frozen, accepted Project A source, images, workspace revision, and
|
||||
PASS report.
|
||||
|
||||
### Final request and data flow
|
||||
|
||||
```text
|
||||
user
|
||||
-> aritmolab.policlinicosandonato.com
|
||||
-> Aritmolab homepage/sidebar
|
||||
-> load balancer
|
||||
-> Nginx/TLS
|
||||
-> ThothII frontend and same-origin /api
|
||||
-> ThothII OIDC Authorization Code + PKCE with Authentik
|
||||
-> PostgreSQL thoth_sessions schema for owned work sessions
|
||||
-> direct read-only datawarehouse schema for clinical queries
|
||||
-> internal Qdrant and Ollama
|
||||
```
|
||||
|
||||
Nginx terminates/proxies according to the observed deployment but does not add a second
|
||||
`auth_request` in front of ThothII. ThothII performs generic OIDC directly. Nginx preserves the
|
||||
public host and HTTPS scheme, forwards the callback path unchanged, and supports SSE without
|
||||
buffering or premature timeouts.
|
||||
|
||||
### Authentik configuration
|
||||
|
||||
Before mutation, export or back up the relevant Authentik configuration. Locate existing protected
|
||||
administrative/API credentials without exposing them. Create or adapt:
|
||||
|
||||
- one OAuth2/OIDC provider and one ThothII application;
|
||||
- the exact callback `<public-origin>/api/auth/oidc/callback`;
|
||||
- `openid`, `profile`, and `email` scopes;
|
||||
- a direct, non-empty JSON string array claim named `groups`;
|
||||
- exact user/admin group mappings selected after the survey;
|
||||
- a separate group-view-only service account/API token for ThothII diagnostics.
|
||||
|
||||
Dedicated `TOT Users` and `TOT Admin` groups are the default unless the survey finds existing groups
|
||||
with exactly the intended semantics and the owner approves their reuse. Additional groups are
|
||||
ignored. Missing, malformed, indirect, or ambiguous configured groups fail closed.
|
||||
|
||||
### Supabase session storage
|
||||
|
||||
Do not create a separate PostgreSQL database. Use the server's existing Supabase PostgreSQL
|
||||
database and isolate ThothII work sessions in the dedicated `thoth_sessions` schema. This schema is
|
||||
distinct from the clinical `datawarehouse` schema.
|
||||
|
||||
- A one-shot migrator role owns only the required schema migration privileges.
|
||||
- Core receives only the restricted runtime role, never the migrator credential.
|
||||
- Forced RLS and application ownership checks isolate sessions by Authentik principal.
|
||||
- The schema stores principals/preferences, manifests, phase artifacts, review decisions, and
|
||||
audit records.
|
||||
- Chat and live SSE output remain ephemeral; semantic vectors remain in Qdrant; clinical data
|
||||
remains in `datawarehouse`; browser authentication sessions remain in protected auth state.
|
||||
- `thoth_sessions` must not be added to Supabase/PostgREST exposed schemas.
|
||||
- The direct runtime connection uses the TLS/CA contract required by the current application.
|
||||
|
||||
### Safe activation order
|
||||
|
||||
1. Back up the accepted Project A operator/auth configuration and every external configuration to
|
||||
be changed.
|
||||
2. Prepare Authentik objects without exposing the new route.
|
||||
3. Run and verify additive session-schema migrations; require no pending or drifted migrations.
|
||||
4. Prepare OIDC secrets and non-secret configuration in protected installation state.
|
||||
5. Validate Authentik discovery, issuer/JWKS, catalog access, and mapped groups.
|
||||
6. Validate Nginx, certificate, load-balancer route, callback, forwarded headers, and SSE while
|
||||
production traffic remains closed.
|
||||
7. Start ThothII in OIDC/public server mode with PostgreSQL session storage.
|
||||
8. Open the final load-balancer route.
|
||||
9. Preserve or update the Aritmolab sidebar link so the established user journey remains intact.
|
||||
10. Complete automated and human acceptance, then remove the Project A temporary route.
|
||||
|
||||
### Acceptance
|
||||
|
||||
Project B requires:
|
||||
|
||||
- successful redacted static, live, and interactive authentication diagnostics;
|
||||
- trusted certificate chain, correct public origin, callback, and proxy headers;
|
||||
- proven load-balancer/Nginx routing and SSE operation;
|
||||
- successful migration status, RLS/role tests, and proof that `thoth_sessions` is not REST-exposed;
|
||||
- ordinary, administrator, unmapped, malformed-claim, logout, and controlled provider-failure
|
||||
cases;
|
||||
- login to Aritmolab followed by the sidebar link to ThothII without a second credential prompt;
|
||||
- no raw OIDC token in browser storage, logs, diagnostics, or evidence;
|
||||
- session ownership and administrator-boundary tests;
|
||||
- one harmless F1-F8 PSD session under an OIDC identity;
|
||||
- tested rollback and a completed human manual-test report with an explicit PASS.
|
||||
|
||||
## Error handling and stop rules
|
||||
|
||||
Every executable step follows:
|
||||
|
||||
```text
|
||||
precondition -> action -> verification -> redacted evidence -> checkpoint
|
||||
```
|
||||
|
||||
Sol stops and requests owner help rather than improvising when it encounters:
|
||||
|
||||
- a dirty or unidentified source checkout;
|
||||
- uncertain ownership of Compose, Nginx, load-balancer, Aritmolab, or Authentik configuration;
|
||||
- missing or insufficient credentials;
|
||||
- an unsafe secret path or risk of secret disclosure;
|
||||
- an unverifiable backup or rollback path;
|
||||
- a DWH identity that is not demonstrably read-only;
|
||||
- a change that would affect unrelated Nginx virtual hosts or other applications;
|
||||
- an unprovable temporary-route restriction;
|
||||
- Supabase migration drift, excessive roles, or unintended REST exposure;
|
||||
- a material difference between the surveyed server and this design.
|
||||
|
||||
Unchanged external state is not failure. Sol records the observation and continues only when the
|
||||
current gate is satisfied.
|
||||
|
||||
## Rollback boundaries
|
||||
|
||||
- **Project A:** stop the new stack and restart the preserved legacy installation. No new volume is
|
||||
copied into the legacy installation.
|
||||
- **Project B ingress:** close the public route first, then restore the prior Nginx,
|
||||
load-balancer, certificate reference, and sidebar configuration.
|
||||
- **Project B application:** return to the accepted Project A local-auth operator configuration
|
||||
while the public route remains closed.
|
||||
- **Authentik:** initially disable new objects instead of deleting them; retain the pre-change
|
||||
export until final acceptance.
|
||||
- **Supabase:** migrations are additive. Rollback does not automatically drop `thoth_sessions` or
|
||||
destroy evidence; destructive cleanup requires a separate explicit decision.
|
||||
- **Workspace Git:** publish the multi-transport change as an isolated commit and retain the
|
||||
previous revision. A rejected candidate never replaces the installation's last valid snapshot.
|
||||
|
||||
## Documents and evidence
|
||||
|
||||
The implementation-planning phase creates:
|
||||
|
||||
1. a general execution program with cross-project gates;
|
||||
2. a survey checklist and report template;
|
||||
3. an executable Project A plan for Sol;
|
||||
4. a plain-language Project A manual-test guide;
|
||||
5. a Project A evidence/PASS template;
|
||||
6. an executable Project B plan for Sol;
|
||||
7. a plain-language Project B manual-test guide;
|
||||
8. a Project B evidence/PASS template.
|
||||
|
||||
Sol maintains a protected server-local progress journal and resumes from the last verified
|
||||
checkpoint. Detailed topology and command output remain in a protected evidence directory on the
|
||||
server. Only intentionally redacted reports and reusable templates may enter Git.
|
||||
|
||||
The Project A human guide is terminal-first and may use a local headless browser. When the optional
|
||||
private endpoint exists, it also includes operator browser checks. The Project B guide covers the
|
||||
real Aritmolab homepage/sidebar, Authentik SSO, roles, logout, final workflow, and negative cases.
|
||||
|
||||
## Known documentation reconciliation
|
||||
|
||||
Some older server documentation describes vector and embedding services as external and the
|
||||
mandatory stack as only frontend/core. Current `compose.yaml`, repository instructions, and project
|
||||
state define Qdrant and Ollama as mandatory internal services. The execution plans must treat the
|
||||
current code/Compose contract as authoritative and include a documentation correction rather than
|
||||
following the stale statements.
|
||||
|
||||
## Success condition
|
||||
|
||||
The program is complete only when both projects have exact-source evidence, all automated gates
|
||||
pass, both human manuals are completed with explicit PASS decisions, the Aritmolab sidebar reaches
|
||||
the final ThothII URL through the established load balancer and Nginx, Authentik provides SSO, and a
|
||||
real read-only PSD session completes F1-F8 under an authorized OIDC identity.
|
||||
Reference in New Issue
Block a user