ThothII/docs/plans/2026-08-20-psd-server-deployment-program-design.md

# PSD Server Deployment Program — Design

**Date:** 2026-08-20

**Status:** Approved by the owner

**Owner sequencing amendment (2026-08-21):** the Mac `rest_api` acceptance and revocation of
`legacy-shared` are deferred to one mandatory pre-Project-B gate. This permits the bounded survey
and static, non-mutating Project A private preparation to proceed without changing the Mac. It does not
authorize stopping the legacy stack, starting the new stack, opening ingress, or beginning Project B.

**Design-time application baseline:** `main` at `5c0dc8c` (execution must freeze and record the
then-current `origin/main` SHA)

**Design-time workspace baseline:** `tht-workspace-psd/main` at `bfbabf9` (execution must freeze and
record the then-current remote SHA)

## Purpose

Replace the unused legacy ThothII installation on the PSD server with the current application,
recovering only useful configuration and rebuilding runtime state from canonical sources. Complete
the work as two independently accepted projects:

1. deploy and prove ThothII with local authentication, a direct read-only PSD DWH connection,
   internal Qdrant, and internal Ollama;
2. only after Project A passes, integrate the accepted installation with the server's Authentik,
   Nginx, load balancer, and the existing Aritmolab sidebar link.

The program must be executable by the Sol LLM from a terminal local to the server. It must test
each step, retain redacted evidence, stop on unsafe uncertainty, and include a separate human
manual-test document for each project.

## Binding decisions

- The legacy application may be unavailable for days. Service continuity is not a goal.
- The new source clone is prepared beside the old source directory. The old and new stacks are not
  kept running simultaneously: inventory and backup happen first, the old stack is stopped, and
  only then is the new stack started.
- Keep the old directory, configuration, containers, images, and data only as a temporary recovery
  boundary until the real Aritmolab journey passes Project B. They are disposable after Project B
  automated, human, and owner PASS. Do not migrate legacy application sessions, Qdrant data,
  Ollama caches, or derived indexes.
- Do not create a host `thothii` user or group. Keep UID/GID `10001:10001` as the image's unmapped
  numeric runtime identity and confine numeric ownership to the new installation's writable bind
  trees. Stop if either number becomes mapped to a host account before installation.
- Recover configuration only: endpoints, non-secret policy, provider/model selection, relevant
  paths, and references to protected credentials. Never copy an old setting without validating it
  against the current contract.
- Use the canonical ThothII Compose distribution and reviewed installation-local overrides. Do not
  modify an unrelated global Compose project or blindly adapt the legacy Compose file.
- Lifecycle operations use the installation-aware native `tht` CLI, not raw Compose commands.
- Never use `docker compose down --volumes`, global prune operations, broad recursive deletion, or
  secret-bearing command arguments.

## Repository and workspace model

There are two Git sources with different responsibilities:

- `ThothII` contains application code and non-secret deployment examples.
- `tht-workspace-psd` contains the shared, credential-free PSD workspace source.

The workspace repository contains one logical workspace, `psd-clinical`. Its schema-v3 descriptor
will declare both supported DWH transports:

```yaml
supported_transports: [rest_api, postgres_direct]
```

The Mac installation continues to select `rest_api`. The PSD server selects `postgres_direct`.
Endpoint values, user names, passwords, secret-file paths, machine paths, and Git credentials stay
in each installation's protected bindings and never enter the workspace repository. Evidence,
curated annotations, language, model policy, and semantic-index contract remain shared.

Each installation owns its own Qdrant collection contents, Ollama model cache, preprocessing state,
and sessions even though both consume the same reviewed workspace commit.

## Program structure

The program consists of one non-mutating common survey followed by two independently gated
projects:

```text
Common Survey PASS for Project A private scope
  -> Project A automated PASS
  -> Project A human PASS
  -> Mac REST acceptance
  -> 48-hour dual-key observation covering two 03:00 ETL cycles
  -> legacy-shared revocation and negative proof
  -> full pre-Project-B survey PASS
  -> explicit Project B authorization
  -> Project B automated PASS
  -> Project B human PASS
  -> final cutover acceptance
```

Project B must not start from a partial or assumed Project A result.

## Common Survey

The survey is a prerequisite, not a third implementation project. It runs before any server
mutation and produces a redacted report, topology map, change-scope inventory, unknowns list, and
GO/NO-GO decision.

Sol inventories from the server-local terminal:

- operating system, architecture, Docker and Compose versions, available CPU/RAM/disk, and clock;
- legacy ThothII source, SHA, dirty state, images, containers, networks, ports, volumes, mounts,
  health, installation state, and recovery state;
- effective Compose rendering and ownership of every relevant file;
- DWH database/schema, direct listener, TLS, read-only role, and reachability from containers;
- Supabase/PostgreSQL topology, schema conventions, exposed PostgREST schemas, migration policy,
  backup mechanism, and suitable role boundaries;
- Pi provider/model policy, credential references, external LLM reachability, and current versions;
- Nginx effective configuration, ThothII virtual host/location, upstream, forwarded headers, SSE
  settings, certificate metadata, certificate generation/renewal, and rollback files;
- load-balancer routes, health checks, allowlist capability, TLS boundary, and configuration owner;
- Aritmolab deployment, networks, homepage, sidebar source, current ThothII destination, and release
  procedure;
- Authentik version, deployment, current Aritmolab integration, provider conventions, group
  conventions, backup/export procedure, API access, and credential locations;
- public DNS/origin that the final sidebar link must preserve;
- protected files by path, ownership, mode, and readability only, without printing their contents.

The survey may use hashes, metadata, redacted renders, and permission checks. It must not emit
passwords, bearer tokens, API keys, cookies, OIDC client secrets, private keys, password hashes, or
raw identity tokens. If credential discovery fails, the owner may help locate the existing
Authentik credentials.

## Project A — Standalone Server Acceptance

### Runtime architecture

Project A installs a clean five-service stack:

```text
operator terminal or local headless browser
  -> frontend
  -> core + Pi + workflow harness
       -> direct read-only PSD PostgreSQL DWH
       -> internal Qdrant
       -> internal Ollama (qwen3-embedding:0.6b, 1024 dimensions)
       -> local filesystem work-session storage
```

The stack uses local authentication. It must not be publicly reachable. Core, Qdrant, and Ollama
remain private; the frontend binds to loopback unless the optional restricted test route below is
proved safe.

### Preparation and cutover

1. Freeze exact application and workspace SHAs and require clean source trees.
2. Publish and validate the multi-transport `psd-clinical` descriptor through the curator workflow.
3. Preserve the Mac REST binding unchanged; its live acceptance is deferred to the mandatory
   pre-Project-B gate.
4. Prepare the new source clone and protected operator/runtime directories beside the old source.
5. Extract only approved configuration facts from the legacy installation.
6. Record and verify the exact restart recipe for the legacy stack. No data backup is required
   because the owner declared legacy sessions and configuration disposable; the still-present
   containers, images, source, and data are the temporary rollback boundary.
7. Stop the legacy stack without deleting its source, configuration, images, or data.
8. Build/install the current native `tht` and application images from the frozen source.
9. Configure local authentication, direct DWH bindings, Pi/provider access, and the Git workspace.
10. Start through `tht`, then validate health, authentication, workspace, workflow, and Pi.
11. Rebuild schema/Evidence preprocessing, Qdrant contents, and the Ollama model cache from
    canonical sources. Prove the complete preprocessing rerun is idempotent.

### Optional restricted test route

If and only if the survey proves that the load balancer can enforce a test-operator allowlist
before a request reaches ThothII, Project A may use a temporary private hostname to exercise the
same network path as production:

```text
authorized operator -> allowlisted load-balancer route -> Nginx -> frontend -> core/local auth
```

The route has a distinct hostname, no Aritmolab sidebar link, a certificate created by the existing
managed mechanism, correct forwarded-origin and SSE behavior, and a negative test from an
unauthorized source. Its local-auth `publicUrl` matches the private test origin. It is not a public
production route and must be removed after Project B.

If isolation cannot be demonstrated, Sol must not approximate it or misdeclare
`THOTH_PUBLIC_EXPOSURE=false`; tests run against loopback from the local terminal instead.

### Acceptance

Project A requires:

- exact source identity and reproducible build evidence;
- healthy frontend, core, Qdrant, embedding, and completed model initializer;
- local authentication checks including admin/user separation, wrong password, logout,
  disable/enable, invalidation, and restart persistence;
- direct DWH connectivity with a demonstrably read-only runtime identity;
- active `psd-clinical` at the expected Git revision;
- compatible Qdrant collection/index contract and correct Ollama model/dimensions;
- complete and idempotent DWH, schema, annotation, and Evidence preprocessing;
- one harmless real PSD work session completed through F1-F8, ending in read-only validated SQL;
- persisted manifest, artifacts, reviewer decisions, and final SQL inspection;
- optional private-route positive and negative isolation evidence when that route is used;
- completed human manual-test report with an explicit PASS.

The Mac row may be recorded only as `DEFERRED_PRE_PROJECT_B` under the dated owner amendment. It is
not part of the private server acceptance, but it must become PASS before Project B starts.

Project A does not modify the production Aritmolab sidebar, production public route, or Authentik.

## Project B — Authentik and Aritmolab Integration

Project B begins only from the frozen, accepted Project A source, images, workspace revision, and
PASS report. It also requires the deferred Mac REST acceptance, the full 48-hour observation
window (including two scheduled 03:00 ETL cycles), revocation of `legacy-shared`, proof that the
legacy credential receives `401`, and the full pre-Project-B survey gate.

### Final request and data flow

```text
user
  -> aritmolab.policlinicosandonato.com
  -> Aritmolab homepage/sidebar
  -> load balancer
  -> Nginx/TLS
  -> ThothII frontend and same-origin /api
  -> ThothII OIDC Authorization Code + PKCE with Authentik
  -> PostgreSQL thoth_sessions schema for owned work sessions
  -> direct read-only datawarehouse schema for clinical queries
  -> internal Qdrant and Ollama
```

Nginx terminates/proxies according to the observed deployment but does not add a second
`auth_request` in front of ThothII. ThothII performs generic OIDC directly. Nginx preserves the
public host and HTTPS scheme, forwards the callback path unchanged, and supports SSE without
buffering or premature timeouts.

### Authentik configuration

Before mutation, export or back up the relevant Authentik configuration. Locate existing protected
administrative/API credentials without exposing them. Create or adapt:

- one OAuth2/OIDC provider and one ThothII application;
- the exact callback `<public-origin>/api/auth/oidc/callback`;
- `openid`, `profile`, and `email` scopes;
- a direct, non-empty JSON string array claim named `groups`;
- exact user/admin group mappings selected after the survey;
- a separate group-view-only service account/API token for ThothII diagnostics.

Dedicated `TOT Users` and `TOT Admin` groups are the default unless the survey finds existing groups
with exactly the intended semantics and the owner approves their reuse. Additional groups are
ignored. Missing, malformed, indirect, or ambiguous configured groups fail closed.

### Supabase session storage

Do not create a separate PostgreSQL database. Use the server's existing Supabase PostgreSQL
database and isolate ThothII work sessions in the dedicated `thoth_sessions` schema. This schema is
distinct from the clinical `datawarehouse` schema.

- A one-shot migrator role owns only the required schema migration privileges.
- Core receives only the restricted runtime role, never the migrator credential.
- Forced RLS and application ownership checks isolate sessions by Authentik principal.
- The schema stores principals/preferences, manifests, phase artifacts, review decisions, and
  audit records.
- Chat and live SSE output remain ephemeral; semantic vectors remain in Qdrant; clinical data
  remains in `datawarehouse`; browser authentication sessions remain in protected auth state.
- `thoth_sessions` must not be added to Supabase/PostgREST exposed schemas.
- The direct runtime connection uses the TLS/CA contract required by the current application.

### Safe activation order

1. Back up the accepted Project A operator/auth configuration and every external configuration to
   be changed.
2. Prepare Authentik objects without exposing the new route.
3. Run and verify additive session-schema migrations; require no pending or drifted migrations.
4. Prepare OIDC secrets and non-secret configuration in protected installation state.
5. Validate Authentik discovery, issuer/JWKS, catalog access, and mapped groups.
6. Validate Nginx, certificate, load-balancer route, callback, forwarded headers, and SSE while
   production traffic remains closed.
7. Start ThothII in OIDC/public server mode with PostgreSQL session storage.
8. Open the final load-balancer route.
9. Preserve or update the Aritmolab sidebar link so the established user journey remains intact.
10. Complete automated and human acceptance, then remove the Project A temporary route.

### Acceptance

Project B requires:

- successful redacted static, live, and interactive authentication diagnostics;
- trusted certificate chain, correct public origin, callback, and proxy headers;
- proven load-balancer/Nginx routing and SSE operation;
- successful migration status, RLS/role tests, and proof that `thoth_sessions` is not REST-exposed;
- ordinary, administrator, unmapped, malformed-claim, logout, and controlled provider-failure
  cases;
- login to Aritmolab followed by the sidebar link to ThothII without a second credential prompt;
- no raw OIDC token in browser storage, logs, diagnostics, or evidence;
- session ownership and administrator-boundary tests;
- one harmless F1-F8 PSD session under an OIDC identity;
- tested rollback and a completed human manual-test report with an explicit PASS.

## Error handling and stop rules

Every executable step follows:

```text
precondition -> action -> verification -> redacted evidence -> checkpoint
```

Sol stops and requests owner help rather than improvising when it encounters:

- a dirty or unidentified source checkout;
- uncertain ownership of Compose, Nginx, load-balancer, Aritmolab, or Authentik configuration;
- missing or insufficient credentials;
- an unsafe secret path or risk of secret disclosure;
- an unverifiable backup or rollback path;
- a DWH identity that is not demonstrably read-only;
- a change that would affect unrelated Nginx virtual hosts or other applications;
- an unprovable temporary-route restriction;
- Supabase migration drift, excessive roles, or unintended REST exposure;
- a material difference between the surveyed server and this design.

Unchanged external state is not failure. Sol records the observation and continues only when the
current gate is satisfied.

## Rollback boundaries

- **Project A:** stop the new stack and restart the still-present legacy installation. No new
  volume is copied into it and no promise is made to retain it after Project B PASS.
- **Project B ingress:** close the public route first, then restore the prior Nginx,
  load-balancer, certificate reference, and sidebar configuration.
- **Project B application:** return to the accepted Project A local-auth operator configuration
  while the public route remains closed.
- **Authentik:** initially disable new objects instead of deleting them; retain the pre-change
  export until final acceptance.
- **Supabase:** migrations are additive. Rollback does not automatically drop `thoth_sessions` or
  destroy evidence; destructive cleanup requires a separate explicit decision.
- **Workspace Git:** publish the multi-transport change as an isolated commit and retain the
  previous revision. A rejected candidate never replaces the installation's last valid snapshot.

## Documents and evidence

The implementation-planning phase creates:

1. a general execution program with cross-project gates;
2. a survey checklist and report template;
3. an executable Project A plan for Sol;
4. a plain-language Project A manual-test guide;
5. a Project A evidence/PASS template;
6. an executable Project B plan for Sol;
7. a plain-language Project B manual-test guide;
8. a Project B evidence/PASS template.

Sol maintains a protected server-local progress journal and resumes from the last verified
checkpoint. Detailed topology and command output remain in a protected evidence directory on the
server. Only intentionally redacted reports and reusable templates may enter Git.

The Project A human guide is terminal-first and may use a local headless browser. When the optional
private endpoint exists, it also includes operator browser checks. The Project B guide covers the
real Aritmolab homepage/sidebar, Authentik SSO, roles, logout, final workflow, and negative cases.

## Known documentation reconciliation

Some older server documentation describes vector and embedding services as external and the
mandatory stack as only frontend/core. Current `compose.yaml`, repository instructions, and project
state define Qdrant and Ollama as mandatory internal services. The execution plans must treat the
current code/Compose contract as authoritative and include a documentation correction rather than
following the stale statements.

## Success condition

The program is complete only when both projects have exact-source evidence, all automated gates
pass, both human manuals are completed with explicit PASS decisions, the Aritmolab sidebar reaches
the final ThothII URL through the established load balancer and Nginx, Authentik provides SSO, and a
real read-only PSD session completes F1-F8 under an authorized OIDC identity.