ThothII/docs/plans/2026-08-20-psd-server-project-a-standalone.md

PSD Server Project A Standalone Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Install a clean PSD ThothII stack with local authentication, direct read-only DWH access, internal Qdrant/Ollama, rebuilt preprocessing, and one completed F1-F8 work session.

Architecture: Keep the stopped legacy installation intact only as a temporary rollback boundary and deploy the current canonical five-service Compose stack from an adjacent clean clone. Create no host service identity: the image's UID/GID 10001 remains numeric and unmapped, confined to the new writable bind roots. A reviewed server-local override disables public exposure and uses filesystem work sessions; an optional load-balancer route is permitted only when it is operator-only and its denial boundary is proved first.

Tech Stack: Git, Docker/Compose, native tht, local Argon2id authentication, PSD Supabase PostgreSQL direct transport, Qdrant, Ollama, Pi, Nginx/load-balancer test route where safe.

---

Preconditions

• Common survey result is SURVEY_GO_PROJECT_A_PRIVATE and its digest is recorded.
• Every path below is replaced by the exact survey result before execution.
• No production Nginx/load-balancer/sidebar/Authentik change is in scope.
• The old stack remains running until its exact inventory and restart recipe are verified; old and new stacks never run together.
• The server's workspace deploy credential remains read-only. A curator with write access publishes the workspace change.
• Owner amendment 2026-08-21 declares legacy sessions/configuration disposable. Retain old resources only until Project B proves the production Aritmolab journey, then delete them under a separate exact cleanup authorization. No legacy backup is required.
• Do not create a host user or group for 10001. Immediately before preparing /srv/thothii, both getent passwd 10001 and getent group 10001 must return no match.
• Owner amendment 2026-08-21 defers live Mac rest_api acceptance and legacy-shared revocation to the mandatory pre-Project-B gate. It does not authorize stop/start; those require a later explicit owner gate even after private preparation is complete.

Task 1: Freeze exact inputs

Files:

• Read: protected survey report
• Read: docs/plans/2026-08-20-psd-server-deployment-program-design.md
• Record: protected Project A journal

Step 1: Record application identity

Run in the new planning checkout:

git status --short --branch
git rev-parse HEAD
git rev-parse origin/main
git diff --check

Expected: clean and explicitly approved SHA.

Step 2: Record workspace remote identity

Use the surveyed read-only credential and run:

git ls-remote <workspace-remote> refs/heads/main

Expected: one SHA recorded as the pre-change workspace revision.

Step 3: Check old-stack recoverability

Expected: exact old start/stop procedure, source SHA, Compose identity, volumes/binds, proxy closure procedure, restart recipe, and shared-resource exclusions are present in the survey. Stop if any is missing.

Task 2: Publish the multi-transport workspace revision

Files:

• Modify in authorized curator clone: psd-clinical/workspace.yaml
• Verify: thoth-workspaces.yaml

Step 1: Create a clean curator branch

Run in a write-authorized clone, never in the application-managed registry checkout:

git status --short --branch
git fetch origin main
git switch --create codex/psd-direct-transport origin/main

Expected: clean branch at the recorded remote SHA.

Step 2: Make the minimal descriptor change

Change exactly:

supported_transports: [rest_api]

to:

supported_transports: [rest_api, postgres_direct]

Do not duplicate the workspace or change its ID, collection, Evidence, annotations, model policy, database, or schema.

Step 3: Review the descriptor-only diff

Run:

git diff --check
git diff -- psd-clinical/workspace.yaml thoth-workspaces.yaml

Expected: one semantic line changed; catalog metadata remains identical.

Step 4: Validate with the current ThothII contract

Use a disposable installation/registry or the repository's current registry validation harness to activate the candidate commit before publication. Expected: schema v3 accepts both transports, Evidence and annotations materialize, and no secret is required for source validation.

If no supported validator can be run in the curator environment, stop and request the owner to run the established Mac validation; do not publish based only on YAML parsing.

Step 5: Commit and publish through curator review

git add psd-clinical/workspace.yaml
git diff --cached --check
git commit -m "feat: support direct PSD DWH transport"
git push --set-upstream origin codex/psd-direct-transport

Merge through the repository's normal review path. Record the resulting main SHA.

Step 6: Record the deferred Mac REST proof

Do not change the Mac during Project A. Record DEFERRED_PRE_PROJECT_B, the unchanged expected transport rest_api, and the exact future diagnostics. The proof must become PASS before Project B, after protected delivery/configuration of the per-installation key.

Task 3: Back up and stop the legacy installation

Files:

• Create: surveyed protected legacy backup directory
• Record: Project A journal

Step 1: Capture final legacy state

Run the surveyed legacy status/doctor commands, record source SHA and image IDs, and confirm no active user work. Do not use the new tht against an incompatible old descriptor.

Step 2: Close or maintenance-gate the old ThothII route

Change only the surveyed ThothII-specific route using its established mechanism. Validate Nginx and load-balancer configuration before applying. Confirm external requests no longer reach the app.

Step 3: Record the disposable legacy boundary

Record exact container and image IDs plus filesystem device/inode/ownership/size for /home/chirone/ThothII and /home/chirone/thothii-data. Do not archive their contents: the owner declared them disposable. Prove that the external Evidence bind and both shared Docker networks are excluded from any later cleanup manifest.

Step 4: Stop the old stack

Use its own supported controller. Expected: old containers stopped, not removed; volumes and bind trees unchanged.

Step 5: Rehearse the restart command without executing it

Record the exact command, preconditions, port ownership, and route-restoration order. If it cannot be stated unambiguously, stop before creating the new stack.

Task 4: Prepare the adjacent clean installation

Files:

• Create: survey-selected new source root
• Create: survey-selected operator, secret, data, Pi-state, registry, and backup roots

Step 1: Create dedicated paths without creating identities

Follow docs/install/server.md numeric-ownership rules. Do not call useradd, groupadd, or usermod. The existing operator owns source/operator files; only the new data, Pi-state, and workspace-registry roots use unmapped numeric 10001:10001. Stop if UID or GID 10001 resolves to a host account, and do not change the image identity without a reviewed design amendment.

Step 2: Clone the frozen application source

git -c core.autocrlf=false clone <thothii-remote> <new-source-root>/ThothII
git -C <new-source-root>/ThothII config --local core.autocrlf false
git -C <new-source-root>/ThothII switch --detach <approved-application-sha>
git -C <new-source-root>/ThothII status --short --branch

Expected: detached exact SHA, clean tree.

Step 3: Verify source and platform

cd <new-source-root>/ThothII
bash scripts/verify-line-endings.sh
docker version
docker compose version

Expected: all pass.

Step 4: Prepare Pi state and build the operator

sudo scripts/prepare-server-pi-state.sh <new-pi-state-root> 10001 10001
THT_THT_OUTPUT_DIRECTORY=<protected-build-output> bash scripts/build-tht.sh

Install only the binary matching the surveyed server architecture. Run tht version --json and record its source identity.

Task 5: Create the protected Project A configuration

Files:

• Create outside Git: <project-a-operator-root>/server.env
• Create outside Git: <project-a-operator-root>/thothii-installation.yaml
• Create outside Git: <project-a-operator-root>/project-a-private.yaml
• Create outside Git: <project-a-auth-root>/auth.yaml through tht

Step 1: Start from current examples

Copy deploy/env/server.env.example and docs/install/examples/thothii-installation.server.yaml to the protected Project A operator root. Replace every placeholder with surveyed absolute paths. Never source server.env as shell code.

Step 2: Add the private/local-session override

Create this reviewed override:

services:
  core:
    environment:
      THOTH_PUBLIC_EXPOSURE: "false"
      THT_SESSION_STORAGE: local
  frontend:
    ports: !override
      - "127.0.0.1:<project-a-port>:8080"

Select an unused loopback port proved by ss -lntp. Do not publish core, Qdrant, or Ollama.

Step 3: Compose the installation descriptor

Use profile: server, the exact new source root/env/auth root, workspace remote/branch/read-only access, Project A override, and exactly one Git transport override. Do not include the public session-server overlay in Project A.

Step 4: Validate permissions and render

<new-tht> --installation <project-a-installation> update --check-only

Expected: Compose validates; only frontend has a loopback port; core declares public exposure false and local session storage; Qdrant/Ollama are internal.

Step 5: Configure the local administrator

Create a temporary mode-0600 password file using an echo-free prompt, then run:

<new-tht> --installation <project-a-installation> auth configure \
  --mode local --public-url <project-a-origin> \
  --admin-user <test-admin> --admin-display-name <display-name> \
  --password-file <protected-temporary-password-file>

Remove the temporary input file after success and record that removal. Do not delete generated auth.yaml or users.yaml.

Task 6: Build and start the clean stack

Files:

• Record: Project A evidence directory

Step 1: Build current images

cd <new-source-root>/ThothII
bash scripts/build-local.sh

Expected: current core/frontend images build; pinned Qdrant/Ollama references resolve.

Step 2: Run preflight

<new-tht> --installation <project-a-installation> update --check-only
<new-tht> --installation <project-a-installation> pi doctor

Expected: no mutation error and no secret in output.

Step 3: Start through tht

<new-tht> --installation <project-a-installation> start --build
<new-tht> --installation <project-a-installation> status
<new-tht> --installation <project-a-installation> doctor --json
<new-tht> --installation <project-a-installation> pi test

Expected: frontend, core, Qdrant, embedding healthy and model initializer completed. Doctor has the documented ordered checks and authentication PASS.

Step 4: Verify listener boundaries

Use ss -lntp and bounded Docker inspection. Expected: only the selected frontend loopback port is host-published; no external core, Qdrant, or Ollama listener.

Task 7: Activate the workspace and direct DWH binding

Files:

• Modify only through authenticated Workspace Management: encrypted workspace secret store

Step 1: Pull and inspect the reviewed workspace revision

<new-tht> --installation <project-a-installation> \
  workspace inspect --workspace psd-clinical --json

Expected: active workspace SHA equals the approved multi-transport revision.

Step 2: Configure runtime bindings

Through the authenticated Workspace Management API/UI, select postgres_direct and provide the surveyed host, port, runtime user, password, and optional TLS CA. Secret values go to the encrypted vault; they do not enter server.env, Git, shell arguments, or evidence.

The generated contract names are:

THT_WS_PSD_CLINICAL_DWH_TRANSPORT
THT_WS_PSD_CLINICAL_DWH_HOST
THT_WS_PSD_CLINICAL_DWH_PORT
THT_WS_PSD_CLINICAL_DWH_USER
THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE
THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE

Step 3: Validate and test connections

Run static validation, live connection test, workspace inspect, and doctor --json. Expected: DWH, workspace, internal embedding, and Qdrant checks pass with redacted output.

Step 4: Re-prove read-only grants

Use the survey's catalog query through the exact configured identity. Expected: no DML/DDL grant on datawarehouse. Stop if the runtime user is an owner, superuser, or write-capable role.

Task 8: Rebuild and verify semantic preprocessing

Files:

• Create: Project A preprocessing evidence

Step 1: Inspect the empty/new collection state

<new-tht> --installation <project-a-installation> \
  workspace vector inspect --workspace psd-clinical --json

Expected: either a compatible empty collection or the documented missing-collection state.

Step 2: Create the descriptor-owned collection when missing

Use the guarded vector rebuild only for collection psd-clinical, with exact repeated confirmation and --destroy. Do not run it against any other collection.

Step 3: Run complete preprocessing

<new-tht> --installation <project-a-installation> \
  workspace preprocess run --workspace psd-clinical --json

If it returns manual_review_required, inspect the exact run and curated annotations, obtain the required human decision, run workspace schema accept --workspace psd-clinical --run <run-id> --yes, then resume the same run. Never auto-approve unknown FK changes.

Step 4: Verify collection contract and counts

Run vector inspection and record dimensions, cosine distance, required keyword indexes, and bounded counts by payload kind/revision. Expected: all points carry the active workspace revision.

Step 5: Prove idempotency

Run the complete preprocessing command again. Expected: no new review, no duplicate logical points, unchanged Evidence reported as unchanged, and the same effective configuration identity.

Task 9: Configure and test local users

Files:

• Modify through tht auth user: protected local user registry

Step 1: Add an ordinary test user

Use an echo-free prompt or protected temporary password file:

<new-tht> --installation <project-a-installation> auth user add <test-user> \
  --role user --display-name <display-name> --password-file <protected-temporary-password-file>

Remove the temporary input file after success.

Step 2: Run authentication diagnostics

<new-tht> --installation <project-a-installation> auth status --json
<new-tht> --installation <project-a-installation> auth check --json

Expected: pristine redacted JSON and PASS.

Step 3: Execute automated local-auth cases

Use same-origin requests or a local headless browser to prove ordinary/admin authorization, generic wrong-password failure, disable/enable, password/role revision invalidation, logout-all, CSRF, and remembered-session survival after core restart. Do not retain cookie jars after the test.

Task 10: Optionally add the private network-path test

Current scope boundary (owner, 2026-08-21): omit this entire task and keep Project A loopback-only. Any future use requires a separate shared-infrastructure authorization after the public-origin and load-balancer activities pass; the Project A private survey decision alone is insufficient.

Files:

• Modify only surveyed test-specific load-balancer/Nginx files
• Create: test certificate through the existing managed mechanism

Step 1: Prove allowlist capability before proxying

Create a temporary hostname that returns a fixed maintenance response. From an approved operator source expect success; from an unapproved source expect denial. Do not point it at ThothII yet.

Step 2: Validate and activate the test proxy

Configure Nginx with the same Host/HTTPS forwarding and SSE settings intended for production. Run nginx -t, validate the load balancer, then reload through the established mechanism.

Step 3: Reconfigure local-auth public URL transactionally

If the exact private HTTPS origin differs from the loopback origin, use the supported authentication configuration workflow and invalidate prior test sessions. Re-run auth and doctor checks.

Step 4: Prove both sides

Expected: authorized operator reaches the local login; unauthorized source remains denied before ThothII. If this cannot be demonstrated, remove the test route and continue on loopback.

Task 11: Complete the F1-F8 acceptance session

Files:

• Complete: docs/testing/psd-server-project-a-manual.md
• Create: protected session evidence

Step 1: Select the approved harmless question

Use a known read-only PSD question agreed by the owner. Record the wording in the protected report; do not include patient-identifying values.

Step 2: Create the session as the ordinary local user

Use the private browser route when present; otherwise drive the same-origin frontend/API from the server-local terminal/headless browser. Record only session ID and sanitized milestones.

Step 3: Review every gate

Complete F1-F8 without auto-confirming human decisions. Confirm persisted phase/artifact state after each gate and resume once to prove recovery.

Step 4: Validate final SQL

Expected: finalized session, DWH validation PASS, SQL is read-only, and no clinical mutation occurs.

Step 5: Inspect persisted state

Confirm manifest, question, schema linking, Evidence, CTE plan/tests, final SQL, validation report, and decision ledger exist in local filesystem session storage. Chat/SSE need not persist.

Task 12: Close Project A and preserve rollback

Files:

• Complete: docs/testing/evidence/psd-server-project-a-report-template.md

Step 1: Run final diagnostics

Run status, doctor, auth check, workspace inspect, vector inspect, Pi test, and a bounded secret scan of the intended report.

Step 2: Create a transactional new-installation backup

Use tht backup --drain with a protected explicit output. Verify its checksum. Do not include secrets in the ordinary evidence archive.

Step 3: Complete human acceptance

Every private-server row in docs/testing/psd-server-project-a-manual.md must be PASS or explicitly blocking. Only the Mac REST row may be DEFERRED_PRE_PROJECT_B under the dated owner amendment.

Step 4: Record the gate

Record exact SHAs/images, workspace revision, preprocessing identity/counts, session ID, report digest, rollback status, and explicit PROJECT_A_PRIVATE_PASS or PROJECT_A_FAIL. A private PASS does not authorize Project B while the deferred gate remains open.

Step 5: Stop on FAIL

On FAIL, stop the new stack and use the surveyed old-stack recovery plan if service restoration is desired. Do not start Project B.