diff --git a/docs/superpowers/plans/2026-08-13-p7-psd-migration.md b/docs/superpowers/plans/2026-08-13-p7-psd-migration.md new file mode 100644 index 00000000..55d1fae3 --- /dev/null +++ b/docs/superpowers/plans/2026-08-13-p7-psd-migration.md @@ -0,0 +1,122 @@ +# P7 — PSD migration to the workspace registry — Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to apply this plan task-by-task. + +**Goal:** Move the existing Policlinico San Donato workspace from the legacy flat layout +(`tht-workspace-psd/psd.yaml` + root `evidence/` + `artifacts/mschema/annotations.yaml`) into the +P1.1 workspace registry repository, then re-embed/re-index it on the new internal semantic stack so +ThothII can run live against the real PSD DWH. + +**Source of truth:** PRD D7 (`docs/prd/2026-08-09-workspace-preprocessing-prd.md`), the P1.1 layout +(`docs/superpowers/plans/2026-08-11-p1-1-workspace-directory-registry.md`), and +`docs/workspace-diagnostic-protocol.md`. + +**Architecture:** The PSD content becomes an ordinary Git workspace repository consumed by ThothII +via `THT_WORKSPACE_GIT_REMOTE`. DWH access stays REST (PostgREST); Qdrant and Ollama embedding are +the internal Compose services. The legacy pgvector (768-dim, nomic) is superseded and must be +re-embedded with `qwen3-embedding:0.6b` (1024-dim). + +**Tech Stack:** Git + YAML (descriptor/catalog), the existing `thothctl`/registry for validation. +No new code is required unless a validator gap is proven by a test. + +--- + +## Current-state findings recorded by this plan + +- `/Users/mp/projects/tht-workspace-psd` is already a Git repo on `main` (38 tracked files: + `.gitignore`, legacy `psd.yaml`, and 36 curated `evidence/*.md`) with **no remote**. +- The legacy `psd.yaml` declares REST DWH (`database: postgres`, `schema: datawarehouse`), X-API-Key + auth, internal CA, and the old external pgvector + Ollama; it has **no** `llm_policy`. +- Curated FK annotations exist at `artifacts/mschema/annotations.yaml` (367 FK lines) in the new + canonical `Annotations` shape and can be copied directly to `/schema/annotations.yaml`. +- `physical.yaml` is absent, so DWH introspection must be re-run (requires VPN + DWH access). +- The legacy runtime dirs (`.tht-dwh/`, `.tht-jobs/`, `config/`, `corpus/`, `runtime-v2/`, + `.legacy-artifacts-backup-premerge/`) are untracked runtime state and must not enter the curated repo. + +## Explicit decisions frozen by this plan + +1. **Repository identity.** Reuse `/Users/mp/projects/tht-workspace-psd` as the author/curator clone; + publish it to a GitHub remote the owner creates. The ThothII installation clones that remote, not + any path inside the ThothII repo. +2. **Workspace id:** `psd-clinical` (catalog, descriptor, Qdrant collection, bindings namespace + `PSD_CLINICAL`). +3. **Descriptor (schema v3):** `dwh` = `postgres` / database `postgres` / schema `datawarehouse` / + `supported_transports: [rest_api]`; `semantic_index` = Qdrant `psd-clinical` 1024/cosine + + `qwen3-embedding:0.6b`; `language: it`; `llm_policy.allowed` starts from the historically active + PSD models (`zai/glm-5.2`, `deepseek/deepseek-v4-flash`, `deepseek/deepseek-v4-pro`, + `aritmolab/qwen3.6-35b-a3b`) and is owner-adjustable. +4. **DWH diagnostic:** `diagnostics.dwh_rest` = `POST /rpc/ping`, `auth: x-api-key`, response + `{ database: postgres, schema: datawarehouse }`. The exact ping RPC path/auth is verified by the + owner during the first live smoke and adjusted only in the Git descriptor. +5. **Curated content only:** the repo contains `thoth-workspaces.yaml`, `psd-clinical/workspace.yaml`, + `psd-clinical/evidence/`, `psd-clinical/schema/annotations.yaml`, plus API-generated + `workspace-docs/`. Legacy runtime dirs stay untracked (gitignore). +6. **Re-embedding:** the legacy pgvector is not reused. DWH introspection + schema/Evidence indexing + run afresh on the new stack (or, if the owner prefers, pgvector is exported and re-embedded); + this task is gated on VPN + DWH credentials + a running stack. +7. **Secrets stay out of Git:** DWH X-API-Key and CA are written as local secret files referenced by + `THT_WS_PSD_CLINICAL_DWH_API_KEY_FILE` / `_TLS_CA_FILE`. + +## Completion contract + +1. `tht-workspace-psd` is restructured to the P1.1 layout and commits cleanly (no legacy runtime dirs). +2. A local registry bootstrap against that repository activates `psd-clinical` (schema-v3 valid, + Evidence materialized, annotations parsed and synced, docs generated). +3. `thothctl … workspace inspect --workspace psd-clinical --json` succeeds once the installation + bindings + VPN are present. +4. `thothctl … workspace preprocess run --workspace psd-clinical --json` completes DWH → FK → schema → + Evidence against the real DWH and indexes into the internal Qdrant. +5. A live session on `psd-clinical` reaches the first reviewer gate (P8 L2 smoke). + +--- + +### Task 1: Restructure the repository (autonomous) + +**Repo:** `/Users/mp/projects/tht-workspace-psd` (separate checkout). + +1. Write `thoth-workspaces.yaml` (catalog with `psd-clinical`). +2. Write `psd-clinical/workspace.yaml` (schema v3, decisions 2–4). +3. `git mv` the 36 `evidence/*.md` files to `psd-clinical/evidence/`. +4. Copy the curated FK to `psd-clinical/schema/annotations.yaml`. +5. Add `.gitignore` for legacy runtime dirs; remove/leave legacy `psd.yaml` as a non-contract + historical note (do not commit the old flat paths as canonical). +6. Commit on `main` (no remote yet). + +### Task 2: Local registry validation (autonomous, no DWH/secret) + +1. From a scratch bare remote of the restructured repo, bootstrap a `WorkspaceRegistry` and assert + `psd-clinical` activates: descriptor valid, catalog matches, Evidence materialized with manifest, + annotations parsed/synced, `workspace-docs/psd-clinical` generated. +2. Verify `thothctl workspace inspect` fails only on the missing DWH bindings (not on the descriptor). + +### Task 3: Installation bindings + secrets (owner) + +1. Owner creates the GitHub remote and provides its URL + push credentials. +2. Owner provides (or confirms reuse of) the DWH X-API-Key and CA; write them as local secret files. +3. Fill `THT_WS_PSD_CLINICAL_DWH_*` bindings + `THT_WORKSPACE_GIT_REMOTE` + LLM provider in the + installation env (VPN active). + +### Task 4: Re-embedding/indexing (owner + stack) + +1. Start the stack; `embedding-model-init` pulls `qwen3-embedding:0.6b`. +2. `thothctl … workspace preprocess run --workspace psd-clinical --json` (DWH → FK → schema → Evidence) + against the real DWH; verify the Qdrant collection is populated and revision-scoped. + +### Task 5: Live smoke + manual acceptance (owner, P8 L2) + +1. New session on `psd-clinical` reaches the first reviewer gate; finalize one real query. +2. Record the P7/P8 manual acceptance in `docs/testing/p2-p6-manual-verification.md` and + PROJECT_STATE.md. + +--- + +## Owner checkpoint + +Tasks 1–2 are executed now by the agent. Tasks 3–5 are blocked on owner-provided secrets/access +(GitHub remote, VPN, DWH key/CA, LLM provider) and on the live stack. + +## Non-goals + +- No P8/P9/P10 work (end-to-end CI, retention policy changes, ssh_tunnel runtime). +- No change to the accepted P1.1–P6 contracts. +- No secret value, certificate, or response body is committed or printed.