diff --git a/.artifacts/reviews/task9-quality-audit-final5.md b/.artifacts/reviews/task9-quality-audit-final5.md deleted file mode 100644 index 665653b3..00000000 --- a/.artifacts/reviews/task9-quality-audit-final5.md +++ /dev/null @@ -1,66 +0,0 @@ -# Task 9 quality audit — final 5 - -**Scope:** the two blocking findings from `task9-quality-audit-final4.md` — unbound production -module graph at manual serve, and commit-addressed snapshots accepted without content identity at -render. Manual acceptance remains **PENDING**; no `VERDICT.md` was created. - -## Verdict: APPROVED for the two final integrity blockers - -### 1. Manual serve binds the complete `backend/dist` module graph, not only `server.js` - -`prepare` now builds a post-build manifest of every regular `backend/dist` file -(relative path, size, SHA-256, device, inode) and writes it as an exclusive `0600` record -(`installation/runtime/backend-dist.manifest.json`) inside the owned root; `ownership.json` -records that record's path/device/inode/size/SHA-256. `serve` revalidates the manifest record -identity and bytes, revalidates every distribution file against it (no-follow, single inode, -size and digest), and refuses before spawning. The manifest descriptor is passed to the child on -fd 4 together with the entrypoint on fd 3. The immutable preload parses the manifest, verifies -the entrypoint cross-digest, reads and hash-verifies **every** file at startup, caches the -verified bytes, and its load hook serves **only** those cached bytes for any import below -`backend/dist` (entry URL still served from the bound fd-3 bytes). A same-path regular -replacement of any imported dependency is therefore refused before `RUNNING` (serve-time -validation), refused at child startup (startup verification), or rendered harmless (cached -bytes), and the parent revalidates the full manifest at `RUNNING` publication and at `stop`. - -### 2. Renderer binds snapshot content to its commit identity - -The generated render command validates the bounded saved read/publish revisions, the -commit-addressed owned snapshot path, the installed Git HEAD, and the bounded -`snapshot.json` manifest of that commit: `head` equals the commit, `files[.yaml]` is the -SHA-256 of the snapshot bytes, the manifest revision binds commit/blob/snapshot path, the saved -revision blob equals the manifest blob, and `git rev-parse :workspaces/.yaml` plus -`git hash-object` of the snapshot bytes both equal that blob. It passes the expected digest as -`--snapshot-sha256`. The renderer re-reads the bounded `snapshot.json` (`head`, -`files[.yaml]` must equal the carried digest), opens the snapshot once with no-follow -semantics and bounded reads, renders only the digest-verified bytes, re-verifies around lease -publication, releases the lease in `finally`, and publishes no output on any refusal. - -## Deterministic regressions added - -- static regular replacement of an imported production dependency after `prepare` is refused, - no marker, no accepted PID record, no orphan; -- deterministic dependency check/load swap (`beforeSpawn` rename) is refused by the child's - startup verification, no marker, no PID record, no orphan; -- after `RUNNING`, a same-path regular dependency replacement is never executed: the loader - serves the verified cached bytes (health-visible source stays the original) and the marker is - absent; -- renderer refuses a same-path regular snapshot byte replacement against the carried digest and - manifest, with lease release and no output; -- renderer refuses manifest `head`, `files` digest, expected-digest, missing, and malformed - cases, with lease release and no output; -- wrapper refuses missing manifest, manifest head/digest/revision tampering, saved-revision blob - mismatch, Git blob mismatch, and snapshot-vs-Git-bytes mismatch, and passes the exact - `--snapshot-sha256` on the valid path (stub renderer records arguments). - -## Verification - -- `bash scripts/test-p1-manual-acceptance.sh` (backend build + both suites): **59 tests, 59 - pass, 0 fail**; no `8791/8792` listener and no `--p1-manual-nonce` process remain. -- `npx tsc --noEmit -p .` (backend): PASS. -- Real-repository `prepare` + `cleanup` cycle: 39 distribution files bound, entrypoint - cross-digest verified, owned root fully removed afterwards. -- Diff check: only the seven Task 9 paths are touched; no Task 8 file was modified. -- This report and the implementation contain no fixture secret or canary values. - -Manual acceptance remains **PENDING** by design; the walkthrough and human verdict are -unchanged. diff --git a/.artifacts/task-15/automated-gates.json b/.artifacts/task-15/automated-gates.json deleted file mode 100644 index 80664d7a..00000000 --- a/.artifacts/task-15/automated-gates.json +++ /dev/null @@ -1,282 +0,0 @@ -{ - "schema": "thothii-task4-certification-v1", - "generated_on": "2026-08-18", - "started_at_utc": "2026-08-18T14:16:40Z", - "ended_at_utc": "2026-08-18T14:20:10Z", - "source_commit": "2a9359071257f9b8a71d36ec2bbb25b161003f81", - "source_immutability": { - "status": "PASS", - "tracked_changes_after_freeze": false, - "allowed_untracked": [".playwright-cli/", ".thothctl/"] - }, - "source_commits": { - "task4_candidate": "b31b27e5845ffd3adf311429367319beaba263c7", - "task1": "d43738eeae6d14bb5e470093058b069a983f5372", - "task2": "5f9a3ae066a060b43a11a959b60a1efadd1c2425", - "task3": "0d8e707533fada938c99eb06f8457150e7ef2b40", - "task3_follow_up": "b31b27e5845ffd3adf311429367319beaba263c7", - "fix_round_1_source": "10cd66fe6a5b484a4dc569326a228c1c5484a5d4", - "fix_round_2_source": "2a9359071257f9b8a71d36ec2bbb25b161003f81", - "historical_task15_final": "74b062f1a737103524cbe706346cfd65f87cdfd1" - }, - "versions": { - "node_contract": "v24.16.0", - "node_host_default": "v25.6.1", - "go": "go1.26.5", - "pi": "0.80.3" - }, - "retained_report": ".superpowers/sdd/2026-08-16-thothii-authentication/task-15-report.md", - "task4_report": ".superpowers/sdd/2026-08-18-thothii-authentication-remediation/task-4-report.md", - "fix_round_2_report": ".superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-2-report.md", - "workflow": { - "run_id": "32147345625", - "url": "https://github.com/mptyl/ThothII/actions/runs/32147345625", - "event": "workflow_dispatch", - "head_sha": "2a9359071257f9b8a71d36ec2bbb25b161003f81", - "status": "completed", - "conclusion": "failure", - "windows_job": { - "name": "Windows clone and Compose contract", - "job_id": "95744249248", - "url": "https://github.com/mptyl/ThothII/actions/runs/32147345625/job/95744249248", - "conclusion": "failure", - "native_step": "Run native Windows retained-capability tests", - "native_step_conclusion": "success", - "command": "go test ./internal/safeio ./internal/backup ./internal/authstorage -count=1", - "requested_packages": ["internal/safeio", "internal/backup", "internal/authstorage"], - "executed_packages": ["internal/safeio", "internal/backup", "internal/authstorage"], - "not_executed_packages": [], - "package_results": { - "internal/safeio": "PASS (22.058s)", - "internal/backup": "PASS (7.161s)", - "internal/authstorage": "PASS (16.088s)" - }, - "failed_step": "Verify Windows clone contract", - "failure_category": "baseline_powershell_parser", - "failure_detail": "scripts/test-windows-clone-contract.ps1:208 parses $remoteYaml: as an invalid variable reference" - }, - "lf_compose_docs_typescript_job": { - "name": "LF, Compose, docs, and TypeScript", - "job_id": "95744249458", - "url": "https://github.com/mptyl/ThothII/actions/runs/32147345625/job/95744249458", - "conclusion": "failure", - "failed_step": "Verify Compose and installation contracts", - "category": "baseline_ci_contract", - "detail": "unified Compose contract passed; test-no-deployment-coupling-scope.sh stopped on TMPDIR: unbound variable", - "downstream_steps": "skipped" - }, - "linux_docker_job": { - "name": "Linux Docker deployment and rollback", - "job_id": "95744249354", - "url": "https://github.com/mptyl/ThothII/actions/runs/32147345625/job/95744249354", - "conclusion": "failure", - "failed_step": "Run unified deployment smoke", - "category": "infrastructure_prerequisite", - "detail": "Task 13 smoke failed before deployment because rg is required", - "cleanup": "PASS", - "image_manifest": "not_generated" - }, - "windows_docker_startup_job": { - "name": "Native Windows Docker Desktop/WSL2 startup", - "job_id": "95744250450", - "url": "https://github.com/mptyl/ThothII/actions/runs/32147345625/job/95744250450", - "status": "NOT_RUN", - "classification": "BLOCKED", - "workflow_conclusion": "skipped", - "reason": "workflow conditions skipped the job; no Windows Docker Desktop/WSL2 command executed" - } - }, - "docker_image_evidence": { - "authentication_smoke": { - "status": "PASS", - "docker_images": [], - "reason": "no_docker_images_exercised" - }, - "unified_docker_smoke": { - "status": "FAIL", - "source_commit": "2a9359071257f9b8a71d36ec2bbb25b161003f81", - "run_id": "32147345625", - "workflow_job_id": "95744249354", - "manifest": ".artifacts/task-15/unified-docker-images.json", - "reason": "workflow attempt stopped before deployment because rg is required", - "cleanup": "PASS", - "images": 0, - "historical": { - "status": "PASS", - "source_commit": "74b062f1a737103524cbe706346cfd65f87cdfd1", - "run_id": "20260818070637-66409-30058", - "manifest_sha256": "9c8dec4546909fd93799dbcf374bcb3a89bc46cfe0fd482472c0cbe757ddf5b6", - "images": 5, - "cleanup": "PASS" - } - } - }, - "gates": { - "posix_registry_ownership": { - "status": "PASS", - "source_commit": "b31b27e5845ffd3adf311429367319beaba263c7", - "evidence": "backend Node 24 full suite including local-registry ownership coverage" - }, - "stagearchive_unix_retained_capability": { - "status": "PASS", - "source_commit": "b31b27e5845ffd3adf311429367319beaba263c7", - "evidence": "focused safeio/backup tests, Go race suite, and Unix ancestor-swap coverage" - }, - "windows_stagearchive_retained_capability": { - "status": "PASS", - "source_commit": "2a9359071257f9b8a71d36ec2bbb25b161003f81", - "evidence": "native Windows backup package passed, including the two-file shared retained-root staging test" - }, - "windows_claim_retained_capability": { - "status": "PASS", - "source_commit": "2a9359071257f9b8a71d36ec2bbb25b161003f81", - "evidence": "native Windows safeio and authstorage packages passed concurrent claim/consume coverage" - }, - "workflow_lf_compose_docs_typescript": { - "status": "FAIL", - "classification": "baseline_ci_contract", - "reason": "TMPDIR was unset after the unified Compose contract passed" - }, - "workflow_linux_docker": { - "status": "FAIL", - "classification": "infrastructure_prerequisite", - "reason": "runner did not provide rg; cleanup proof passed and no image manifest was generated" - }, - "go_security_build": { - "status": "PASS", - "source_commit": "2a9359071257f9b8a71d36ec2bbb25b161003f81", - "focused_packages": 3, - "race_packages": 18, - "focused_test": "PASS", - "race": "PASS", - "vet": "PASS", - "host_build": "PASS" - }, - "windows_cross_compile": { - "status": "PASS", - "source_commit": "2a9359071257f9b8a71d36ec2bbb25b161003f81", - "focused_test_packages": 3, - "cli_build": "PASS", - "execution": "cross_compile_only_not_native_execution" - }, - "backend_node24": { - "status": "PASS", - "source_commit": "b31b27e5845ffd3adf311429367319beaba263c7", - "node": "v24.16.0", - "files": 76, - "tests": 1092, - "typecheck": "PASS", - "build": "PASS", - "note": "an initial full run had one workspace-registry timeout; focused rerun and complete rerun passed" - }, - "frontend_node24": { - "status": "PASS", - "source_commit": "b31b27e5845ffd3adf311429367319beaba263c7", - "node": "v24.16.0", - "files": 61, - "tests": 444, - "typecheck": "PASS", - "build": "PASS" - }, - "authentication_and_f1_smoke": { - "status": "PASS", - "source_commit": "b31b27e5845ffd3adf311429367319beaba263c7", - "node": "v24.16.0", - "filtered_e2e": "1 passed", - "sentinel_leak_scan": "PASS" - }, - "harness_pytest": { - "status": "FAIL", - "source_commit": "b31b27e5845ffd3adf311429367319beaba263c7", - "passed": 951, - "failed": 1, - "skipped": 4, - "subtests": 232, - "failure": "test_column_decisions::test_f4_emits_column_types: workflow.yaml not found from harness test cwd" - }, - "authentication_docs": { - "status": "PASS", - "source_commit": "b31b27e5845ffd3adf311429367319beaba263c7" - }, - "shell_syntax": { - "status": "PASS", - "source_commit": "b31b27e5845ffd3adf311429367319beaba263c7" - }, - "authentication_smoke_runtime": { - "status": "PASS", - "node": "v24.16.0", - "sentinel_leak_scan": "PASS" - }, - "compose_default": { - "status": "FAIL", - "reason": "required THT_WORKSPACE_GIT_REMOTE was not available" - }, - "compose_unified": { - "status": "FAIL", - "reason": "compose.unified.yaml is absent from the frozen source" - }, - "unified_docker_smoke": { - "status": "FAIL", - "source_commit": "2a9359071257f9b8a71d36ec2bbb25b161003f81", - "workflow_run_id": "32147345625", - "reason": "remote workflow attempted the smoke but stopped before deployment because rg is required", - "cleanup": "PASS", - "image_manifest": "not_generated" - }, - "ruff": { - "status": "FAIL", - "errors": 192, - "classification": "known_baseline" - }, - "mkdocs_strict": { - "status": "NOT_RUN", - "classification": "BLOCKED", - "historical_status": "FAIL", - "historical_warnings": 69 - }, - "canonical_install_docs": { - "status": "NOT_RUN", - "classification": "BLOCKED", - "historical_status": "FAIL" - }, - "workspace_install_docs": { - "status": "NOT_RUN", - "classification": "BLOCKED", - "historical_status": "FAIL" - }, - "pi_user_auth_compose": { - "status": "NOT_RUN", - "classification": "BLOCKED", - "historical_status": "FAIL" - }, - "deployment_coupling": { - "status": "NOT_RUN", - "classification": "BLOCKED", - "historical_status": "FAIL" - }, - "l2": { - "status": "PENDING", - "reason": "configured secret layout unavailable; gate not run after stop" - }, - "manual_psd": { - "status": "PENDING", - "reason": "approved real identity/access unavailable; gate not run after stop" - }, - "provider_readiness": { - "status": "PENDING", - "reason": "provider prerequisite unavailable; gate not run after stop" - } - }, - "review": { - "original_important_findings_resolved": 3, - "fix_round_2_important_lifecycle": "ADDRESSED", - "fix_round_2_minor_windows_diagnostics": "ADDRESSED", - "verdict": "PASS", - "reason": "the lifecycle controller is bounded and cancellation-aware with cancel, bounded join, and lock-release proof; the temporary Windows diagnostic matrix is removed; exact-source native safeio, backup, and authstorage all pass" - }, - "remediation_status": "PASS", - "release_complete": false, - "authentication_implementation_complete": true, - "release_readiness": "FAIL", - "release_readiness_pending_external_gates": true -} diff --git a/.artifacts/task-15/unified-docker-images.json b/.artifacts/task-15/unified-docker-images.json deleted file mode 100644 index 4cbfd1fa..00000000 --- a/.artifacts/task-15/unified-docker-images.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "gate": "unified-deployment-smoke", - "status": "pass", - "source_commit": "74b062f1a737103524cbe706346cfd65f87cdfd1", - "run_id": "20260818070637-66409-30058", - "images": [ - { - "id": "sha256:2d7b19491c7eb8c119c3cedb390aaeb2ff5593f6fc43ab66c317565560da6d7d", - "roles": [ - "compose-runtime", - "fixture-runtime" - ], - "repo_digests": [ - "sha256:2d7b19491c7eb8c119c3cedb390aaeb2ff5593f6fc43ab66c317565560da6d7d" - ] - }, - { - "id": "sha256:3b6c31a5d8f8fc58fa3233391b6175bd2fbc793eebb44d5e285ecc6e02e9e687", - "roles": [ - "compose-runtime" - ], - "repo_digests": [ - "sha256:3b6c31a5d8f8fc58fa3233391b6175bd2fbc793eebb44d5e285ecc6e02e9e687" - ] - }, - { - "id": "sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a", - "roles": [ - "compose-runtime" - ], - "repo_digests": [ - "sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a" - ] - }, - { - "id": "sha256:75eab8c4ba42096724fdcfde8b4de0b5713d529dde32f285a1f86fdcb2c9e50c", - "roles": [ - "compose-runtime" - ], - "repo_digests": [ - "sha256:75eab8c4ba42096724fdcfde8b4de0b5713d529dde32f285a1f86fdcb2c9e50c" - ] - }, - { - "id": "sha256:c3cbe1cc1aa588a64951ac6286e0df7b27fe2e6324b1001c619bb358770c0178", - "roles": [ - "rollback-candidate" - ], - "repo_digests": [ - "sha256:c3cbe1cc1aa588a64951ac6286e0df7b27fe2e6324b1001c619bb358770c0178" - ] - } - ] -} diff --git a/.claude/launch.json b/.claude/launch.json deleted file mode 100644 index 61be3335..00000000 --- a/.claude/launch.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "version": "0.0.1", - "configurations": [ - { - "name": "replay", - "runtimeExecutable": "node", - "runtimeArgs": ["tools/replay/server.mjs"], - "port": 5333 - } - ] -} diff --git a/.dockerignore b/.dockerignore index 492b9eef..7ac93658 100644 --- a/.dockerignore +++ b/.dockerignore @@ -27,5 +27,3 @@ coverage/ data/ sessions/ workspace-registry/ -# docs/site (mkdocs build) — non necessari nelle immagini -docs/superpowers/plans diff --git a/.github/workflows/deployment.yml b/.github/workflows/deployment.yml index dc994ac2..84b10285 100644 --- a/.github/workflows/deployment.yml +++ b/.github/workflows/deployment.yml @@ -36,6 +36,10 @@ jobs: with: node-version: "24.16.0" package-manager-cache: false + - name: Install release gate prerequisites + run: | + sudo apt-get update + sudo apt-get install --yes --no-install-recommends ripgrep - name: Verify shell syntax and LF policy run: | git ls-files -z '*.sh' | xargs -0 -n1 bash -n @@ -46,7 +50,6 @@ jobs: bash scripts/test-no-deployment-coupling-scope.sh bash scripts/test-compose-secret-policy.sh bash scripts/test-no-deployment-coupling.sh - bash scripts/test-preprocess-compose-config.sh bash scripts/test-verify-workspace-install-docs.sh git diff --check - name: Assert clean checkout before release trust bootstrap @@ -60,6 +63,14 @@ jobs: run: | bash scripts/test-server-pi-state-topology.sh bash scripts/unified-deployment-smoke.sh --self-test + - name: Install harness CLI for backend integration tests + working-directory: harness + run: | + python3 -m venv .venv + .venv/bin/python -m pip install -e . + - name: Install backend dependencies + working-directory: backend + run: npm ci - name: Test and type-check backend working-directory: backend run: | @@ -140,11 +151,23 @@ jobs: uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: persist-credentials: false + - name: Install release gate prerequisites + run: | + sudo apt-get update + sudo apt-get install --yes --no-install-recommends ripgrep + - name: Reclaim unused hosted-runner space + run: bash scripts/prepare-linux-docker-runner.sh - name: Run unified deployment smoke + env: + TASK13_IMAGE_EVIDENCE_OUTPUT: ${{ runner.temp }}/task13-images.json run: timeout --signal=TERM --kill-after=45s 32m bash scripts/unified-deployment-smoke.sh - name: Run tht update smoke + env: + TASK13_IMAGE_EVIDENCE_OUTPUT: ${{ runner.temp }}/task13-images.json run: timeout --signal=TERM --kill-after=45s 32m bash scripts/tht-update-smoke.sh - name: Run Linux server deployment smoke + env: + TASK13_IMAGE_EVIDENCE_OUTPUT: ${{ runner.temp }}/task13-images.json run: timeout --signal=TERM --kill-after=45s 32m bash scripts/server-deployment-smoke.sh windows-clone: diff --git a/.gitignore b/.gitignore index 604924df..8af7af0e 100644 --- a/.gitignore +++ b/.gitignore @@ -5,8 +5,6 @@ ChironeWp3/ Thoth/ -# === Visual companion brainstorming artifacts (local-only) === -.superpowers/ .worktrees/ .tht/ diff --git a/.impeccable/design.json b/.impeccable/design.json new file mode 100644 index 00000000..ff837348 --- /dev/null +++ b/.impeccable/design.json @@ -0,0 +1,203 @@ +{ + "schemaVersion": 2, + "generatedAt": "2026-08-26T10:10:15.926Z", + "title": "Design System: ThothII", + "extensions": { + "colorMeta": { + "instrument-red": { + "role": "primary", + "displayName": "Instrument Red", + "canonical": "oklch(55.87% 0.1881 23.2)", + "tonalRamp": ["oklch(15% 0.07 23.2)", "oklch(28% 0.12 23.2)", "oklch(42% 0.16 23.2)", "oklch(56% 0.1881 23.2)", "oklch(68% 0.17 23.2)", "oklch(78% 0.13 23.2)", "oklch(88% 0.07 23.2)", "oklch(95% 0.03 23.2)"] + }, + "instrument-red-hover": { + "role": "primary", + "displayName": "Instrument Red Pressed", + "canonical": "oklch(50.95% 0.1812 24.1)", + "tonalRamp": ["oklch(15% 0.07 24.1)", "oklch(28% 0.12 24.1)", "oklch(42% 0.16 24.1)", "oklch(51% 0.1812 24.1)", "oklch(68% 0.16 24.1)", "oklch(78% 0.12 24.1)", "oklch(88% 0.07 24.1)", "oklch(95% 0.03 24.1)"] + }, + "porcelain-background": { + "role": "neutral", + "displayName": "Porcelain Background", + "canonical": "oklch(99.18% 0.0011 17.2)", + "tonalRamp": ["oklch(15% 0.0011 17.2)", "oklch(28% 0.0011 17.2)", "oklch(42% 0.0011 17.2)", "oklch(56% 0.0011 17.2)", "oklch(68% 0.0011 17.2)", "oklch(78% 0.0011 17.2)", "oklch(88% 0.0011 17.2)", "oklch(95% 0.0011 17.2)"] + }, + "porcelain-card": { + "role": "neutral", + "displayName": "Porcelain Card", + "canonical": "oklch(99.85% 0.0006 17.2)", + "tonalRamp": ["oklch(15% 0.0006 17.2)", "oklch(28% 0.0006 17.2)", "oklch(42% 0.0006 17.2)", "oklch(56% 0.0006 17.2)", "oklch(68% 0.0006 17.2)", "oklch(78% 0.0006 17.2)", "oklch(88% 0.0006 17.2)", "oklch(95% 0.0006 17.2)"] + }, + "warm-surface": { + "role": "neutral", + "displayName": "Warm Surface", + "canonical": "oklch(97.09% 0.0011 17.2)", + "tonalRamp": ["oklch(15% 0.0011 17.2)", "oklch(28% 0.0011 17.2)", "oklch(42% 0.0011 17.2)", "oklch(56% 0.0011 17.2)", "oklch(68% 0.0011 17.2)", "oklch(78% 0.0011 17.2)", "oklch(88% 0.0011 17.2)", "oklch(95% 0.0011 17.2)"] + }, + "sunken-surface": { + "role": "neutral", + "displayName": "Sunken Surface", + "canonical": "oklch(94.08% 0.0011 17.2)", + "tonalRamp": ["oklch(15% 0.0011 17.2)", "oklch(28% 0.0011 17.2)", "oklch(42% 0.0011 17.2)", "oklch(56% 0.0011 17.2)", "oklch(68% 0.0011 17.2)", "oklch(78% 0.0011 17.2)", "oklch(88% 0.0011 17.2)", "oklch(95% 0.0011 17.2)"] + }, + "warm-graphite": { + "role": "neutral", + "displayName": "Warm Graphite", + "canonical": "oklch(26.78% 0.0097 355.6)", + "tonalRamp": ["oklch(15% 0.0097 355.6)", "oklch(28% 0.0097 355.6)", "oklch(42% 0.0097 355.6)", "oklch(56% 0.0097 355.6)", "oklch(68% 0.008 355.6)", "oklch(78% 0.006 355.6)", "oklch(88% 0.004 355.6)", "oklch(95% 0.002 355.6)"] + }, + "muted-graphite": { + "role": "neutral", + "displayName": "Muted Graphite", + "canonical": "oklch(51.33% 0.0088 345.6)", + "tonalRamp": ["oklch(15% 0.0088 345.6)", "oklch(28% 0.0088 345.6)", "oklch(42% 0.0088 345.6)", "oklch(56% 0.0088 345.6)", "oklch(68% 0.007 345.6)", "oklch(78% 0.005 345.6)", "oklch(88% 0.003 345.6)", "oklch(95% 0.002 345.6)"] + }, + "quiet-border": { + "role": "neutral", + "displayName": "Quiet Border", + "canonical": "oklch(90.93% 0.0035 354.7)", + "tonalRamp": ["oklch(15% 0.0035 354.7)", "oklch(28% 0.0035 354.7)", "oklch(42% 0.0035 354.7)", "oklch(56% 0.0035 354.7)", "oklch(68% 0.0035 354.7)", "oklch(78% 0.0035 354.7)", "oklch(88% 0.003 354.7)", "oklch(95% 0.002 354.7)"] + }, + "success-mint": { + "role": "secondary", + "displayName": "Success Mint", + "canonical": "oklch(75.77% 0.1581 165)", + "tonalRamp": ["oklch(15% 0.06 165)", "oklch(28% 0.1 165)", "oklch(42% 0.14 165)", "oklch(56% 0.1581 165)", "oklch(68% 0.15 165)", "oklch(78% 0.12 165)", "oklch(88% 0.07 165)", "oklch(95% 0.03 165)"] + }, + "warning-amber": { + "role": "tertiary", + "displayName": "Warning Amber", + "canonical": "oklch(85.23% 0.1386 78.9)", + "tonalRamp": ["oklch(15% 0.05 78.9)", "oklch(28% 0.09 78.9)", "oklch(42% 0.12 78.9)", "oklch(56% 0.1386 78.9)", "oklch(68% 0.13 78.9)", "oklch(78% 0.1 78.9)", "oklch(88% 0.06 78.9)", "oklch(95% 0.025 78.9)"] + }, + "information-blue": { + "role": "tertiary", + "displayName": "Information Blue", + "canonical": "oklch(70.35% 0.1128 221.3)", + "tonalRamp": ["oklch(15% 0.045 221.3)", "oklch(28% 0.075 221.3)", "oklch(42% 0.1 221.3)", "oklch(56% 0.1128 221.3)", "oklch(68% 0.105 221.3)", "oklch(78% 0.08 221.3)", "oklch(88% 0.045 221.3)", "oklch(95% 0.02 221.3)"] + } + }, + "typographyMeta": { + "display": {"displayName": "Display", "purpose": "Authentication and exceptional page-level statements only."}, + "headline": {"displayName": "Headline", "purpose": "Major page and persisted artifact titles."}, + "title": {"displayName": "Title", "purpose": "Panel and document section hierarchy."}, + "body": {"displayName": "Body", "purpose": "Operational prose and sustained reading."}, + "control": {"displayName": "Control", "purpose": "Buttons, inputs, tabs, and compact actions."}, + "label": {"displayName": "Machine Label", "purpose": "Uppercase metadata and machine-oriented micro-labels."} + }, + "shadows": [ + {"name": "contact", "value": "0 1px 2px oklch(var(--shadow-tint) / 0.05)", "purpose": "Contact shadow for controls and code blocks."}, + {"name": "panel", "value": "0 1px 2px oklch(var(--shadow-tint) / 0.05), 0 2px 6px -1px oklch(var(--shadow-tint) / 0.05)", "purpose": "Small structural lift for selected cards."}, + {"name": "overlay", "value": "0 2px 4px -2px oklch(var(--shadow-tint) / 0.06), 0 12px 32px -8px oklch(var(--shadow-tint) / 0.1)", "purpose": "Broad low-opacity lift for dialogs and floating layers."} + ], + "motion": [ + {"name": "control-feedback", "value": "140ms cubic-bezier(0.22, 1, 0.36, 1)", "purpose": "Button hover, focus, and press feedback."}, + {"name": "overlay-transition", "value": "100ms ease-out", "purpose": "Dialog fade and scale transitions."}, + {"name": "activity-pulse", "value": "1.5s ease-in-out infinite", "purpose": "Live model activity only; disabled for reduced motion."} + ], + "breakpoints": [ + {"name": "sm", "value": "640px"}, + {"name": "lg", "value": "1024px"} + ] + }, + "components": [ + { + "name": "Primary Button", + "kind": "button", + "refersTo": "button-primary", + "description": "The authoritative action for the current workflow step.", + "html": "", + "css": ".ds-button-primary { display:inline-flex; align-items:center; justify-content:center; height:32px; padding:0 14px; border:1px solid transparent; border-radius:8px; background:oklch(var(--primary)); color:oklch(var(--primary-foreground)); font:600 14px/1.25 var(--font-sans); letter-spacing:0.005em; box-shadow:var(--shadow-xs); transition:color 140ms cubic-bezier(0.22,1,0.36,1),background-color 140ms cubic-bezier(0.22,1,0.36,1),box-shadow 140ms cubic-bezier(0.22,1,0.36,1),transform 140ms cubic-bezier(0.22,1,0.36,1); } .ds-button-primary:hover { background:oklch(var(--primary-hover)); } .ds-button-primary:focus-visible { outline:3px solid oklch(var(--ring)/0.25); outline-offset:2px; } .ds-button-primary:active { transform:scale(0.97); box-shadow:none; }" + }, + { + "name": "Outline Button", + "kind": "button", + "refersTo": "button-secondary", + "description": "A compact secondary action that preserves the primary action hierarchy.", + "html": "", + "css": ".ds-button-outline { display:inline-flex; align-items:center; justify-content:center; height:32px; padding:0 14px; border:1px solid oklch(var(--border)); border-radius:8px; background:oklch(var(--card)); color:oklch(var(--foreground)); font:600 14px/1.25 var(--font-sans); box-shadow:var(--shadow-xs); transition:background-color 140ms cubic-bezier(0.22,1,0.36,1),transform 140ms cubic-bezier(0.22,1,0.36,1); } .ds-button-outline:hover { background:oklch(var(--muted)); } .ds-button-outline:focus-visible { outline:3px solid oklch(var(--ring)/0.25); outline-offset:2px; } .ds-button-outline:active { transform:scale(0.97); box-shadow:none; }" + }, + { + "name": "Status Badge", + "kind": "chip", + "refersTo": "badge-primary", + "description": "A compact state label that always carries readable text.", + "html": "Ready for review", + "css": ".ds-status-badge { display:inline-flex; align-items:center; height:20px; padding:2px 8px; border:1px solid transparent; border-radius:6px; background:oklch(var(--primary)); color:oklch(var(--primary-foreground)); font:600 12px/1.25 var(--font-sans); white-space:nowrap; } .ds-status-badge:focus-visible { outline:3px solid oklch(var(--ring)/0.5); outline-offset:2px; }" + }, + { + "name": "Text Field", + "kind": "input", + "refersTo": "input-default", + "description": "A readable operational field with an explicit focus state.", + "html": "", + "css": ".ds-text-field { width:280px; height:40px; padding:0 12px; border:1px solid oklch(var(--input)); border-radius:8px; background:oklch(var(--background)); color:oklch(var(--foreground)); font:400 14px/1.5 var(--font-sans); outline:none; } .ds-text-field:hover { border-color:oklch(var(--muted-foreground)/0.65); } .ds-text-field:focus-visible { border-color:oklch(var(--ring)); box-shadow:0 0 0 3px oklch(var(--ring)/0.25); } .ds-text-field:disabled { opacity:0.5; cursor:not-allowed; }" + }, + { + "name": "Work Card", + "kind": "card", + "refersTo": "card-default", + "description": "A single-level container for a coherent review surface.", + "html": "

Schema linking

Review the linked tables and columns before continuing.

", + "css": ".ds-work-card { width:320px; padding:16px; border:1px solid oklch(var(--border)/0.7); border-radius:12px; background:oklch(var(--card)); color:oklch(var(--card-foreground)); box-shadow:var(--shadow-sm); } .ds-work-card h3 { margin:0 0 8px; font:500 16px/1.35 var(--font-heading); letter-spacing:-0.01em; } .ds-work-card p { margin:0; color:oklch(var(--muted-foreground)); font:400 14px/1.6 var(--font-sans); } .ds-work-card:focus-within { outline:3px solid oklch(var(--ring)/0.25); outline-offset:2px; }" + }, + { + "name": "Session Navigation Item", + "kind": "nav", + "description": "A dense session row with restrained hover and active hierarchy.", + "html": "", + "css": ".ds-session-item { display:flex; width:260px; align-items:center; gap:8px; padding:4px 8px; border:0; border-radius:8px; background:transparent; color:oklch(var(--foreground)); text-align:left; font-family:var(--font-sans); transition:background-color 140ms cubic-bezier(0.22,1,0.36,1); } .ds-session-item:hover,.ds-session-item[aria-current=\"page\"] { background:oklch(var(--accent)); } .ds-session-item:focus-visible { outline:2px solid oklch(var(--ring)/0.4); outline-offset:1px; } .ds-session-dot { width:6px; height:6px; flex:none; border-radius:9999px; background:oklch(var(--success)); } .ds-session-item strong,.ds-session-item small { display:block; } .ds-session-item strong { font-size:13px; font-weight:600; } .ds-session-item small { margin-top:2px; color:oklch(var(--muted-foreground)); font-size:11px; }" + }, + { + "name": "Curated Evidence Document", + "kind": "custom", + "description": "The table-free reading hierarchy for persisted evidence.", + "html": "

Fascia pediatrica

Dominio · Italiano
Scopi: Disambiguazione · Generazione SQL

Ambito di applicazione

  • fascia pediatrica
  • paziente minore

Regola

La fascia pediatrica comprende i pazienti con età inferiore a 18 anni.

Dettagli tecnici e provenienzaevidence:fascia-pediatrica
", + "css": ".ds-evidence { max-width:70ch; color:oklch(var(--foreground)); font:400 15px/1.65 var(--font-sans); } .ds-evidence h2,.ds-evidence h3 { font-family:var(--font-heading); letter-spacing:-0.01em; } .ds-evidence h2 { margin:0 0 16px; font-size:24px; } .ds-evidence h3 { margin:24px 0 8px; font-size:18px; } .ds-evidence-summary { padding:12px 14px; border:1px solid oklch(var(--border)); border-radius:8px; background:oklch(var(--muted)); color:oklch(var(--muted-foreground)); } .ds-evidence-summary strong { color:oklch(var(--foreground)); } .ds-evidence ul { padding-left:20px; } .ds-evidence details { margin-top:24px; padding:10px 12px; border:1px solid oklch(var(--border)); border-radius:8px; background:oklch(var(--card)); } .ds-evidence summary { cursor:pointer; font-weight:600; } .ds-evidence code { font-family:var(--font-mono); }" + } + ], + "narrative": { + "northStar": "The Clinical Workbench", + "overview": "ThothII should feel like a well-kept clinical workbench: warm enough for sustained reading, exact enough for consequential review, and quiet enough that evidence, state, and decisions remain in the foreground. The visual system is calm, precise, and trustworthy. It uses familiar product patterns, restrained color, and deliberate density instead of decorative spectacle.\n\nThe primary physical scene is an analyst reviewing persisted evidence and SQL on a large monitor in a well-lit working environment. This makes the warm light theme the default. The supported dark theme serves lower-light work without becoming a separate neon aesthetic. Both themes preserve the same hierarchy and semantic roles.\n\nThe system rejects generic SaaS ornament, conspicuous ripples, bounce or elastic motion, long choreographed transitions, and effects that compete with the analytical task. Controls should feel disciplined and tactile, never playful, sluggish, or visually unstable.", + "keyCharacteristics": [ + "Warm, restrained surfaces with one scarce red accent.", + "Editorial headings paired with highly legible operational body text.", + "Dense information organized through hierarchy, rhythm, and progressive disclosure.", + "Persisted artifacts and reviewer decisions presented as the visual source of truth.", + "Fast state feedback with reduced-motion parity." + ], + "rules": [ + {"name": "The Workbench Rule", "body": "Every visual element must support inspection, action, state, or provenance. Decoration without an operational purpose is forbidden.", "section": "overview"}, + {"name": "The Persisted Truth Rule", "body": "Persisted artifacts and reviewer decisions receive stronger hierarchy than transient model narration.", "section": "overview"}, + {"name": "The Density with Rhythm Rule", "body": "Preserve information density, but vary spacing between groups so users can scan structure without adding nested containers.", "section": "overview"}, + {"name": "The One Voice Rule", "body": "Instrument Red should occupy no more than roughly ten percent of a screen. Its rarity is what makes it authoritative.", "section": "colors"}, + {"name": "The State Has a Name Rule", "body": "Success, warning, information, and destructive colors are reserved for their named states. Color is never the only state indicator.", "section": "colors"}, + {"name": "The Three Registers Rule", "body": "Serif means authority, sans means interaction and reading, mono means machine identity. Do not exchange these roles for novelty.", "section": "typography"}, + {"name": "The Read Once Rule", "body": "A heading, label, and body must be distinguishable on first glance through size and weight. Do not repeat headings in explanatory copy.", "section": "typography"}, + {"name": "The Flat by Default Rule", "body": "A resting surface has no shadow unless it is physically above another surface. If every panel floats, none of them has hierarchy.", "section": "elevation"}, + {"name": "The Borders Structure, Shadows Elevate Rule", "body": "Never use shadow as a substitute for grouping or a border as a decorative accent.", "section": "elevation"}, + {"name": "The Review Surface Rule", "body": "The visible Markdown must be readable without understanding the machine contract. Technical metadata belongs in progressive disclosure, not above the title.", "section": "components"} + ], + "dos": [ + "Do make every state change unmistakable without interrupting flow.", + "Do use Instrument Red only for primary action, current selection, focus identity, or explicit destructive meaning.", + "Do preserve information density with headings, rhythm, and progressive disclosure.", + "Do keep keyboard focus explicit and pair color with text, shape, icon, or position.", + "Do respect prefers-reduced-motion while preserving immediate non-kinetic feedback.", + "Do use English for interface chrome and the workspace language for persisted document content.", + "Do render curated metadata and scope as Markdown prose or lists, never as a frontmatter table." + ], + "donts": [ + "Don't add generic SaaS ornament, conspicuous ripples, bounce or elastic motion, long choreographed transitions, or effects that compete with the analytical task.", + "Don't make controls feel playful, sluggish, or visually unstable.", + "Don't use gradient text, decorative glassmorphism, or full-saturation accents on inactive states.", + "Don't use a colored side stripe greater than one pixel on cards, callouts, list items, or blockquotes. Use a full border, tonal background, icon, or heading instead.", + "Don't nest cards or wrap every section in a container.", + "Don't use a modal before exhausting inline or progressive alternatives.", + "Don't use tables for applies_to, metadata, enum values, or other one-dimensional content.", + "Don't use color as the sole carrier of success, warning, error, selection, or progress.", + "Don't use display typography for buttons, labels, or data.", + "Don't add em dashes to interface copy. Use commas, colons, semicolons, or parentheses." + ] + } +} diff --git a/.kilo/kilo.jsonc b/.kilo/kilo.jsonc deleted file mode 100644 index cc2fb594..00000000 --- a/.kilo/kilo.jsonc +++ /dev/null @@ -1,7 +0,0 @@ -{ - "$schema": "https://app.kilo.ai/config.json", - "indexing": { - "vectorStore": "qdrant", - "model": "sentence-transformers/all-minilm-l12-v2" - } -} \ No newline at end of file diff --git a/.superpowers/sdd/2026-08-03-diagnostic-contract-extension/task-3-report.md b/.superpowers/sdd/2026-08-03-diagnostic-contract-extension/task-3-report.md deleted file mode 100644 index 246794e4..00000000 --- a/.superpowers/sdd/2026-08-03-diagnostic-contract-extension/task-3-report.md +++ /dev/null @@ -1,105 +0,0 @@ -# Task 3 — Diagnostic contract remediation report - -Date: 2026-08-04 - -## Scope - -This remediation is limited to the four approved review findings for the workspace diagnostic -extension. It does not add registry routes, change workspace publication, alter session startup, -or expand transport support. - -## Changes - -1. `RuntimeBindings` now has an explicit `vectorWriter` binding. The new - `resolveRuntimeBindings()` resolves DWH, vector reader, vector writer, and embedding bindings - together. The diagnoser takes the writer credential only from `bindings.vectorWriter`, never - from vector-reader values. -2. Direct PostgreSQL and SSH-tunnelled direct probes accept an absent CA binding while retaining - certificate verification through the runtime system trust store. A supplied CA still uses - verified private-CA trust. REST private-CA refusal is unchanged. -3. A reversible vector probe now requires an authenticated POST declaration with a response map - containing `operation`. The adapter requires the successful JSON response to echo `create` or - `remove` respectively, so an arbitrary 2xx or an upsert-only response cannot activate the - write probe. -4. For DWH and vector REST diagnostics declared with `auth: none`, the resolver no longer - requires an API-key file and the adapter sends no credential. Credential-backed diagnostics - continue to require their local secret file. - -## TDD evidence - -The first focused RED run failed for the intended missing behavior: - -- `resolveRuntimeBindings is not a function` for unauthenticated resolver bindings; -- schema accepted a reversible probe without a response contract; and -- existing diagnostic fixtures rejected the new `response` declaration until schema support was - implemented. - -The focused GREEN run passed `43/43` tests across: - -- `test/workspaces-bindings.test.ts` -- `test/workspaces-schema.test.ts` -- `test/workspaces-diagnostics.test.ts` - -The regression coverage includes resolver-to-diagnoser writer propagation without manually -inserting the writer key into vector-reader bindings, no-CA direct/SSH system-trust requests, -operation-echo validation for create/remove, and `auth: none` bindings without secret files. - -## Documentation and design - -- `docs/workspace-diagnostic-protocol.md` now documents the verified system-trust fallback, - no-secret `auth: none` behavior, and required reversible response contract. -- `docs/superpowers/specs/2026-08-03-git-workspace-registry-design.md` now records the same - response, CA, SSH, and authentication rules. - -## Final verification - -The initial sandboxed full suite could not bind its local SSE listener (`listen EPERM: -operation not permitted 127.0.0.1`). It was rerun unchanged with local-listener permission. - -```text -backend: npx vitest run -31 test files passed; 329 tests passed - -backend: npx tsc --noEmit -p . -exit 0 - -repository: git diff --check -exit 0 -``` - -Expected test harness stderr from existing Pi/process failure-path tests remained present; no test -failed and no diagnostic secret was emitted. - -## Blockers - -None. - -## Round 2 remediation - -The final review found two remaining contract gaps. The binding resolver already treated -`auth: none` as credential-free, but the runtime renderer and diagnostic connector still required -the API-key file. Rendering and connector construction now make that requirement conditional on -the declared REST authentication mode, so a DWH/vector `auth: none` workspace passes resolver, -runtime rendering, and diagnostics with no API-key file. - -SSH forwarding previously changed the PostgreSQL connection host to `127.0.0.1` without retaining -the original target for TLS hostname validation. Forwarded probes now carry `SSH_TARGET_HOST` as -`tlsServername` into the PostgreSQL TLS options; private CA and verified system trust behavior are -unchanged. - -TDD RED: the new end-to-end no-key test failed at the unconditional runtime -`API_KEY_FILE` requirement, while the SSH test showed no `tlsServername` on the loopback probe or -database-client request. TDD GREEN: the focused backend workspace tests passed `40/40`. - -Round 2 final verification: - -```text -backend: npx vitest run -31 test files passed; 332 tests passed - -backend: npx tsc --noEmit -p . -exit 0 - -repository: git diff --check -exit 0 -``` diff --git a/.superpowers/sdd/2026-08-03-git-workspace-registry/task-7-report.md b/.superpowers/sdd/2026-08-03-git-workspace-registry/task-7-report.md deleted file mode 100644 index 53111d0d..00000000 --- a/.superpowers/sdd/2026-08-03-git-workspace-registry/task-7-report.md +++ /dev/null @@ -1,101 +0,0 @@ -# Task 7 report — revision-pinned sessions - -## Delivered - -- New-session requests may carry `workspaceId`, provider, model, and thinking. The backend - resolves the active operational registry revision, enforces its LLM policy, and persists the - workspace ID/revision with the selected LLM settings. -- The harness manifest and `tht session new` support the optional, backward-compatible - `workspace_id` and `workspace_revision` fields. -- Resume resolves the manifest's retained snapshot, including after later registry publication. - A missing retained revision returns a sanitized `workspace_revision_unavailable` response. - Legacy manifests retain the prior workspace behavior and are marked with a visible warning on - `GET /sessions/:id`. -- `/settings` is now a non-mutating compatibility endpoint: installation defaults remain - readable, while anonymous workspace/provider/model/thinking selections are no longer written - to backend settings or principal preferences. - -## TDD evidence - -- RED: `npx vitest run test/routes-sessions.test.ts test/routes-settings.test.ts` failed for the - new immutable-snapshot and no-settings-mutation assertions; the manifest test failed because - `new_session_manifest` did not accept workspace revision fields. -- GREEN: `npx vitest run test/tht-runner.test.ts test/routes-sessions.test.ts test/routes-settings.test.ts && npx tsc --noEmit -p .` - completed with 97 passing tests and a clean type check. -- GREEN: `THT_HOME=/private/tmp/thothii-task7-home .venv/bin/pytest tests/test_session_documents.py tests/test_session_mutations.py -q` - completed with 22 passing tests. -- `git diff --check` completed cleanly. - -## Review fixes — round 3 - -- The active registry snapshot that located a session now remains the authorization and mutation - config for response, steer, events, close/delete, archive/group/rename, documents, and detail. - A pruned historical revision cannot block an already-located session's active lifecycle. -- Only Resume resolves the retained pinned descriptor because Pi needs that immutable config to - restart safely. A pruned pin therefore returns the existing sanitized - `workspace_revision_unavailable` 409 solely for Resume. - -### Round 3 verification - -- RED: with a manifest found through an active registry snapshot and `readPinned` forced to fail, - `POST /sessions/:id/response` returned 409 instead of forwarding the active gate response. -- GREEN: `npx vitest run test/routes-sessions.test.ts test/tht-runner.test.ts test/routes-settings.test.ts && npx tsc --noEmit -p .` - — 102 tests passed with a clean type check. The regression confirms response, close, and delete - use the locating snapshot without calling `readPinned`, while Resume returns a sanitized 409. -- `git diff --check` completed cleanly. - -## Review fixes — round 2 - -- Lifecycle authorization no longer selects the installation-default workspace. The backend now - finds each session by querying every operational registry snapshot with the authenticated - principal, preserving RLS ownership concealment. -- After locating the manifest, durable pinned sessions resolve their retained descriptor before - any lifecycle mutation/reopen. Legacy sessions continue using the locating registry snapshot. -- Session listing aggregates the owner-visible rows from all operational registry snapshots; - detail, response, steer, resume, events, documents, and lifecycle mutations use the same - server-side locator. No route depends on browser-local workspace state. - -### Round 2 verification - -- RED: the new cross-workspace route integration test created a B session while installation - default A was selected, then demonstrated that `GET /sessions` returned an empty list. -- GREEN: `npx vitest run test/routes-sessions.test.ts test/tht-runner.test.ts test/routes-settings.test.ts && npx tsc --noEmit -p .` - — 101 tests passed with a clean type check. The integration test covers create B, list, detail, - response, and resume through B's pinned descriptor while default A remains configured. -- Full backend suite: 342 tests passed. The remaining 7 tests require binding `127.0.0.1` and - fail in this sandbox with `listen EPERM: operation not permitted`; no application assertion - failed. The focused typecheck above passed. -- `git diff --check` completed cleanly. - -## Verification note - -The unscoped backend suite was also run. The Task 7 code regressions in `test/tht-runner.test.ts` -were fixed; the remaining failures were existing sandbox restrictions on tests that listen on -`127.0.0.1` (`listen EPERM: operation not permitted` in SSE/e2e health tests), not application -assertions. - -## Review fixes — round 1 - -- Every new session now resolves `workspaceId` through the registry; an omitted value uses the - configured installation default and persists both the resolved ID and revision. Callers cannot - bypass revision pinning by supplying a workspace ID. -- Browser-local preferences now migrate once from the read-only legacy settings response and hold - workspace, provider, model, and thinking. Session creation includes those selections, including - direct entry points that run before the composer mounts. The frontend no longer `PUT`s shared - settings. -- The settings compatibility endpoint honors a stored installation workspace before falling back - to the first workspace configuration. -- Resume rejects finalized and archived sessions before looking up any pinned snapshot, preserving - the read-only response even when a historical snapshot is unavailable. - -### Review verification - -- RED: the added backend tests failed for omitted-default pinning, read-only resume ordering, and - stored-default precedence; the added frontend preference tests failed because preferences were - neither stored nor included in session requests. -- GREEN: `npx vitest run test/tht-runner.test.ts test/routes-sessions.test.ts test/routes-settings.test.ts && npx tsc --noEmit -p .` - — 100 tests passed with a clean type check. -- GREEN: `npx vitest run && npx tsc -b` — 332 frontend tests passed with a clean type check. -- GREEN: `THT_HOME=/private/tmp/thothii-task7-home .venv/bin/pytest tests/test_session_documents.py tests/test_session_mutations.py -q` - — 22 tests passed (one existing testcontainers deprecation warning). -- `git diff --check` completed cleanly. diff --git a/.superpowers/sdd/2026-08-03-git-workspace-registry/task-9-report.md b/.superpowers/sdd/2026-08-03-git-workspace-registry/task-9-report.md deleted file mode 100644 index 03b48d80..00000000 --- a/.superpowers/sdd/2026-08-03-git-workspace-registry/task-9-report.md +++ /dev/null @@ -1,73 +0,0 @@ -# Task 9 report — Workspace Management CRUD page - -## Delivered - -- Added the Workspace management dialog, launched from the persistent right sidebar and the - Model activity header without touching live-session/SSE state. -- Added a workspace list/detail editor for General, DWH, Semantic index, LLM policy, - Installation requirements, and Git status/history. -- Added browser-only New, Edit, Duplicate, Save draft, and Delete-draft workflows. A deletion - draft stores only ID and immutable revision references; publication remains a Task 10 action. -- Used closed native controls for languages, engines, transports, distance metrics, embedding - providers, and selectable default models. Free values have client-side, accessible errors. -- Made semantic-index dimensions atomic: one editor field always writes the same value to the - vector-store and embedding contracts. -- Added Validate and Test-on-this-installation actions. They display sanitized code/message - diagnostics only; neither action exposes or stores credentials, secrets, or raw response bodies. -- Explicitly excluded publish, pull, import, and export user flows from this task. - -## TDD evidence - -- RED: `npx vitest run src/shell/WorkspaceManager.test.tsx src/shell/WorkspaceEditor.test.tsx` - failed because the manager and editor modules did not exist. -- GREEN: focused manager/editor/AppShell coverage passed after the implementation. -- RED: a deletion-draft persistence regression failed with - `Cannot read properties of undefined (reading 'save')` before the sanitized draft store was added. -- GREEN: the draft-store and manager tests passed once deletion intent persisted locally. - -## Verification - -Executed from `frontend/`: - -```text -npx vitest run -50 test files passed, 358 tests passed -npx tsc -b -exit 0 -``` - -`git diff --check` passed before commit. No workspace secret value, secret-file path, raw -diagnostic body, publish call, import flow, or export flow was introduced. - -## Fix round 1 - -### Root causes and fixes - -- The original duplicate proposal appended `-copy` and then truncated at 63 characters. For an - already-maximal ID, truncation could remove the suffix and reproduce the immutable source ID. - The proposal now reserves suffix space and falls back to a distinct `-2` suffix when a maximal - source already ends in `-copy`. -- `dwh.timeout_ms` was rendered as a positive numeric field but was absent from the client - validation map. It now has the same immediate accessible error treatment as other numeric - fields, so a rejected save never reaches the manager’s saved-draft toast. -- Registry status, workspace list, and selected-detail React Query failures were rendered as - loading, empty, or unselected states. Each now has a named alert and a retry control, distinct - from its corresponding loading and empty state. - -### TDD evidence - -- RED: max-length duplication retained the original 63-character ID; the timeout field produced - no alert; and each of the three failed queries had no accessible retry control. -- GREEN: the focused manager/editor tests passed **12/12**, covering a valid changed duplicate - proposal, rejected zero timeout with no save toast, and status/list/detail retry recovery. - -### Verification - -Executed from `frontend/`: - -```text -npx vitest run -50 test files passed, 364 tests passed -npx tsc -b -exit 0 -``` diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-11-report.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-11-report.md deleted file mode 100644 index b9c381de..00000000 --- a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-11-report.md +++ /dev/null @@ -1,180 +0,0 @@ -# Task 11 report - -Status: completed on 2026-08-08. - -## Scope delivered - -- Updated operator-facing documentation for the internal Qdrant + Ollama architecture. -- Tightened documentation contract tests to require the current four-service-plus-init topology, - CPU-first/GPU-override guidance, fixed internal model/dimensions, schema-v3 migration wording, - one-collection-per-workspace ownership, and Qdrant backup/restore safety. -- Updated stable repo guidance in `AGENTS.md` and the current snapshot in `PROJECT_STATE.md`. -- Rewrote the workspace diagnostic protocol to the schema-v3/internal-semantic-service contract. -- Updated the memory guide to describe Qdrant as the derived persistent index. -- Updated the runtime secret-bundle guide to remove active vector/embedding secret guidance. - -## Files changed - -- `README.md` -- `AGENTS.md` -- `PROJECT_STATE.md` -- `docs/install/local-workspace-registry.md` -- `docs/install/server-workspace-registry.md` -- `docs/installazione-docker-4-contesti.md` -- `docs/workspace-diagnostic-protocol.md` -- `docs/gestione-memory.md` -- `deploy/secrets/README.md` -- `scripts/verify-workspace-install-docs.sh` -- `scripts/test-verify-workspace-install-docs.sh` - -## Verification - -Fresh successful runs: - -```sh -./scripts/test-verify-workspace-install-docs.sh -./scripts/verify-workspace-install-docs.sh --fixtures-only -git diff --check -``` - -Key outcomes: - -- internal semantic infrastructure documentation contract passed -- all existing install/manual fixture contracts still passed -- diff hygiene passed with no whitespace/errors - -## Self-review notes - -- The updated docs now match the code-backed Compose topology: `frontend`, `core`, `qdrant`, - `embedding`, and `embedding-model-init`. -- Active manuals no longer instruct operators to configure external vector or embedding runtime - endpoints/secrets. -- Qdrant backup/restore wording now matches the helper scripts' exact confirmation and rollback - behavior. -- Legacy descriptor handling is documented as explicit schema-v3 migration only; no silent - semantic-data migration is claimed. - -## Residual concerns - -- The broader repository still contains historical design/spec material that references older - pgvector/external-embedding architecture; this task intentionally updated operator/current-state - documentation and the corresponding contract tests, not historical planning documents. - -## Fix round 1/5 — 2026-08-08 - -Addressed reviewer findings: - -- Moved superseded rollout/state blocks in `PROJECT_STATE.md` behind an explicit - `## Historical snapshots and archived reference notes` boundary. -- Renamed superseded snapshot headings so historical notes no longer present as active `LIVE` - state. -- Added a current-state regression that rejects contradictory active blocks (for example: - schema-v2 operational, two-service active stack, or external vector/embedding runtime claims - before the historical boundary). -- Refactored new internal-semantic doc checks away from exact-sentence coupling: - - parse `compose.yaml` structurally with YAML; - - parse workspace examples structurally with YAML; - - inspect backup/restore stable usage interface; - - keep targeted forbidden-term checks for active docs while allowing historical sections; - - use regex/concept checks for prose. - -Evidence: - -```sh -./scripts/test-verify-workspace-install-docs.sh -./scripts/verify-workspace-install-docs.sh --fixtures-only -git diff --check -``` - -Observed RED before the fix: - -```text -PROJECT_STATE.md: missing Historical snapshots boundary -``` - -## Fix round 2/5 — 2026-08-08 - -Addressed reviewer findings: - -- Renamed every historical `PROJECT_STATE.md` heading after the historical boundary so no heading - level uses `LIVE` or current-state semantics there. -- Strengthened the historical-boundary regression to reject any Markdown heading level - (`#` through `######`) containing `LIVE` or current-state wording after the boundary. -- Added a fixture with a `### ... — LIVE ...` historical heading to prove RED then GREEN. -- Replaced remaining exact phrase checks with concept/semantic validation for: - - one-workspace/one-collection ownership; - - external boundary (DWH/LLM external; vector/embedding internal); - - the Italian compact install note. -- Added paraphrase fixtures that pass and omission/inversion fixtures that fail. - -Evidence: - -```sh -./scripts/test-verify-workspace-install-docs.sh -./scripts/verify-workspace-install-docs.sh --fixtures-only -git diff --check -``` - -## Fix round 4/5 — 2026-08-08 - -Addressed reviewer finding: - -- Eliminated semantic-index verifier/test contract drift by extracting the production - semantic-index ownership row matcher into `semantic_index_relationship_spec` and reusing it in - the fixture-level paraphrase, omission, and scattered-token checks. -- Kept the relationship constrained to one structured Markdown table row via - `verify_markdown_table_relationships`; the scattered-token fixture still removes the row and - appends the same words outside the table, where it must be rejected. -- Added a direct regression that copies the repository docs into an isolated root, applies the - accepted paraphrase “A workspace keeps exactly one Qdrant collection reserved for itself”, and - runs that root's actual `scripts/verify-workspace-install-docs.sh --fixtures-only` instead of a - separate temporary spec. - -Observed RED before the fix: - -```text -production verifier rejected the accepted semantic-index paraphrase -local workspace manual: missing relationship in 'Semantic index ownership contract': {'scope': 'workspace semantic index', 'ownership rule': '(each|one|single).*(workspace).*(single|one).*(Qdrant).*(collection)|(each workspace reserves a single qdrant collection)', 'isolation rule': 'schema.*evidence.*memory.*(one|that).*(collection).*(kind|payload)'} -``` - -Evidence: - -```sh -./scripts/test-verify-workspace-install-docs.sh -./scripts/verify-workspace-install-docs.sh --fixtures-only -git diff --check -``` - -Observed RED during this round: - -```text -PROJECT_STATE.md: historical section still contains active/live heading markers -compact manual paraphrase lacks required pattern: (esterni solo|solo esterni|restano esterni) -``` - -## Fix round 3/5 — 2026-08-08 - -Addressed reviewer findings: - -- Added table-driven historical-heading fixtures for every Markdown heading level `#` through - `######`; all are rejected after the historical boundary when they contain `LIVE`/current-state - semantics. -- Added small structured ownership tables to the active local/server manuals and to the compact - Italian operator note. -- Added small structured semantic-index ownership tables to the active local/server manuals. -- Replaced the remaining scattered-token relationship checks with explicit structured-section - parsing: - - architecture ownership rows map DWH → external, LLM → external, Qdrant → internal, - Ollama embedding → internal; - - semantic-index ownership rows localize the one-workspace/one-collection contract and the - schema/Evidence/Memory isolation rule. -- Added adversarial fixtures that fail when the same tokens are merely scattered in free text. -- Added structured paraphrase fixtures that pass and omission/inversion fixtures that fail. - -Evidence: - -```sh -./scripts/test-verify-workspace-install-docs.sh -./scripts/verify-workspace-install-docs.sh --fixtures-only -git diff --check -``` diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-12-report.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-12-report.md deleted file mode 100644 index 1ea763f5..00000000 --- a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-12-report.md +++ /dev/null @@ -1,43 +0,0 @@ -# Task 12 Report — Remove unreachable pgvector runtime code - -Status: completed - -Summary: -- Proved the retired pgvector runtime had no remaining operational adapter call sites after migration by re-running the required grep; only the packaging assertion still mentions `migrations/vector`. -- Removed the obsolete pgvector/HTTP/direct vector runtime modules, vector SQL migrations, and their affected runtime tests. -- Kept the operational semantic path on Qdrant and migrated the remaining runtime callers to that path. -- Kept `psycopg2-binary` because DWH direct PostgreSQL and session PostgreSQL code still depend on it. - -Implementation notes: -- Extracted shared collection/kind validation into `harness/tht/adapters/vector/_shared.py` so `QdrantVectorStore` no longer depends on the deleted pgvector module. -- Simplified `build_vector_store()` to return only `QdrantVectorStore`. -- Migrated vector/evidence/memory CLI paths away from legacy pgvector loaders and REST vector clients. -- Updated packaging coverage so the built wheel asserts session SQL migrations are present and vector SQL migrations are absent. - -Verification: -- `cd harness && .venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py tests/test_semantic_kind_isolation.py tests/test_vector_migration_packaging.py -q` -- `cd harness && .venv/bin/pytest tests/test_adapter_factory.py tests/test_solved_search_cli.py -q` -- `cd harness && .venv/bin/python -c "import tht.cli, tht.adapters.factory, tht.adapters.vector, tht.vectorstore.reader"` -- `cd harness && uv build` -- `harness/.venv/bin/ruff check harness/tests/test_adapter_factory.py harness/tests/test_solved_search_cli.py harness/tests/test_vector_migration_packaging.py harness/tests/test_vector_port_contract.py harness/tht/adapters/factory.py harness/tht/adapters/vector/__init__.py harness/tht/adapters/vector/_shared.py harness/tht/adapters/vector/qdrant.py harness/tht/cli/evidence_cmd.py harness/tht/cli/memory_cmd.py harness/tht/cli/search_cmd.py harness/tht/cli/vector_cmd.py harness/tht/solved.py harness/tht/vectorstore/reader.py` -- `git diff --check` - -Notes / concerns: -- Repository-wide `harness/.venv/bin/ruff check .` still reports many pre-existing findings outside this task’s touched files; it is not clean on this branch baseline. -- Some legacy config compatibility parsing still exists outside the deleted runtime path. This task removed the unreachable runtime/migration code without broad config-schema refactoring. - -## Fix round 1 evidence - -Changes: -- Removed dead `vector migrate` registration from `harness/tht/cli/__init__.py` and deleted `harness/tht/cli/vector_migrate_cmd.py`. -- Added CLI regressions proving `vector migrate` is absent while `vector init` and `vector index-schema` remain available. -- Restored the accidentally removed non-vector regressions by moving report coverage into `harness/tests/test_report.py` and restoring the taskdoc promoted-table slicing check in `harness/tests/test_taskdoc.py`. -- Reworded surviving active help/docstrings away from pgvector-specific wording in the touched Qdrant-backed command surface. - -Verification: -- `cd harness && .venv/bin/pytest tests/test_qdrant_cli_commands.py tests/test_report.py tests/test_taskdoc.py tests/test_vector_migration_packaging.py -q` -- `cd harness && .venv/bin/python -c "from typer.testing import CliRunner; from tht.cli import app; r=CliRunner().invoke(app, ['vector','--help']); assert r.exit_code == 0, r.output; assert 'migrate' not in r.output; r=CliRunner().invoke(app, ['vector','migrate','--help']); assert r.exit_code != 0, r.output; print('cli-help-ok')"` -- `cd harness && .venv/bin/python -c "import tht.cli, tht.cli.vector_cmd, tht.report, tht.taskdoc; print('imports-ok')"` -- `cd harness && uv build` -- `harness/.venv/bin/ruff check harness/tests/test_qdrant_cli_commands.py harness/tests/test_report.py harness/tests/test_taskdoc.py harness/tests/test_vector_migration_packaging.py harness/tht/cli/__init__.py harness/tht/cli/search_cmd.py harness/tht/cli/vector_cmd.py harness/tht/cli/memory_cmd.py harness/tht/solved.py` -- `git diff --check` diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-13-implementation.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-13-implementation.md deleted file mode 100644 index 9f798a3a..00000000 --- a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-13-implementation.md +++ /dev/null @@ -1,175 +0,0 @@ -# Task 13 Implementation Report - -## Status - -DONE_WITH_CONCERNS - -## Changes - -- Updated stale harness/backend/frontend tests and fixtures to the Task 13 internal Qdrant/Ollama contract. -- Made `deploy/workspaces/psd.yaml.example` generic while preserving schema-v3 Qdrant/Ollama shape. -- Fixed `scripts/workspace-registry-smoke.sh` to pass the required legacy migration `--collection` and prove exact Docker cleanup, including its smoke image. -- Updated `PROJECT_STATE.md` with only evidence observed in this run. - -Changed files: - -- `PROJECT_STATE.md` -- `backend/test/routes-workspaces.test.ts` -- `backend/test/workspace-runtime-handoff.test.ts` -- `backend/test/workspaces-contracts.test.ts` -- `backend/test/workspaces-git-repository.test.ts` -- `deploy/workspaces/psd.yaml.example` -- `frontend/src/shell/NewSessionDialog.test.tsx` -- `harness/tests/test_adapter_command_regressions.py` -- `harness/tests/test_workspace.py` -- `scripts/task13-runtime-fixture-check.ts` -- `scripts/test-verify-workspace-install-docs.sh` -- `scripts/workspace-registry-smoke.sh` - -## Verification - -Deterministic gates: - -- `cd harness && .venv/bin/pytest -q && .venv/bin/ruff check .` - - Initial red: 2 harness pytest failures. - - After fixture fixes: harness pytest passed `819 passed, 4 deselected, 74 warnings in 27.73s`. - - Ruff still failed with `Found 220 errors`; treated as existing unrelated debt. - - Touched harness files verified clean with `cd harness && .venv/bin/ruff check tests/test_adapter_command_regressions.py tests/test_workspace.py && .venv/bin/pytest -q tests/test_adapter_command_regressions.py::test_solved_index_writes_through_writer_only_factory_store tests/test_workspace.py::test_load_workspace_expands_env_vars`: `All checks passed!` and `2 passed, 2 warnings in 0.14s`. -- `cd backend && npx vitest run && npx tsc --noEmit -p . && npm run build` - - Initial red: 4 backend Vitest failures. - - After fixes: `Test Files 39 passed (39)`, `Tests 464 passed (464)`, TypeScript passed, build passed. -- `cd frontend && npx vitest run && npx tsc -b && npm run build` - - Initial red: 1 frontend Vitest failure. - - After fix: frontend Vitest passed `374/374`, TypeScript passed, build passed with Vite `built in 6.55s`. -- `git diff --check` - - Passed with no output. - -Focused reruns: - -- `cd backend && npx vitest run test/workspaces-migrate-legacy.test.ts test/workspaces-contracts.test.ts test/routes-workspaces.test.ts test/workspace-runtime-handoff.test.ts test/workspaces-git-repository.test.ts && cd .. && ./scripts/test-no-deployment-coupling.sh && ./scripts/verify-workspace-install-docs.sh --fixtures-only && git diff --check` - - `Test Files 5 passed (5)`, `Tests 35 passed (35)`. - - Coupling guard passed: `no active retired deployment or external semantic coupling found.` - - Install docs fixtures passed through `relative secret-source fixture rejected passed`. - -Deployment contracts: - -- `./scripts/test-default-compose.sh && ./scripts/test-unified-compose.sh && ./scripts/test-internal-semantic-compose.sh && ./scripts/test-no-deployment-coupling.sh && ./scripts/test-compose-secret-policy.sh && ./scripts/verify-workspace-install-docs.sh --fixtures-only` - - Passed. Output included: - - `default Compose contract passed.` - - `unified Compose contract passed.` - - `internal semantic Compose/script contracts passed.` - - `no active retired deployment or external semantic coupling found.` - - `Compose secret policy passed.` - - install-doc fixture checks through `relative secret-source fixture rejected passed`. - -Docker smokes: - -- `/usr/bin/time -p ./scripts/internal-semantic-smoke.sh` - - Passed: `Task 13 internal semantic smoke passed.` - - Cleanup proof: `no labeled containers, volumes, networks, or images remain for 20260808200245-83368-17823.` - - Duration: `real 217.34`. -- `/usr/bin/time -p ./scripts/workspace-registry-smoke.sh` - - Initial red: `usage: migrate-legacy --input --output --collection [--id ]`. - - After fix: `workspace registry smoke passed`. - - Cleanup proof: `no compose containers, volumes, networks, or image remain for thoth-workspace-registry-smoke-89671.` - - Duration: `real 9.93`. -- `/usr/bin/time -p ./scripts/unified-deployment-smoke.sh` - - Passed: `Task 13 full deployment smoke passed.` - - Cleanup proof: `no labeled containers, volumes, networks, or images remain for 20260808200706-85638-13391.` - - Duration: `real 125.57`. -- `/usr/bin/time -p ./scripts/thothctl-update-smoke.sh` - - Passed: `Task 13 update deployment smoke passed.` - - Cleanup proof: `no labeled containers, volumes, networks, or images remain for 20260808200918-87340-10404.` - - Duration: `real 85.40`. -- `/usr/bin/time -p ./scripts/server-deployment-smoke.sh` - - Passed: `Task 13 Linux server deployment smoke passed.` - - Cleanup proof: `no labeled containers, volumes, networks, or images remain for 20260808201047-88645-20675.` - - Duration: `real 55.99`. - -Final audit: - -- `rg -n "pgvector|local-vector|THT_VECTOR_|EMBEDDING_BASE_URL|openai_compatible|ollama_compatible" . --glob '!docs/plans/**' --glob '!docs/superpowers/**' --glob '!**/node_modules/**' --glob '!**/.venv/**' --glob '!**/.git/**'` - - Returned matches in legacy schema-v1/v2 support, migration tests, negative guards, historical notes, and older harness docs/code. - - This remains a concern: the audit is not clean under the brief's strict expected outcome. -- `git status --short` - - Before report/commit, contained only intentional Task 13 changes. - -## Image and Host Evidence - -- Host CPU: `Apple M4 Pro`. -- Host OS: `Darwin MacProM4-di-Marco.local 25.5.0 Darwin Kernel Version 25.5.0: Tue Jun 9 22:28:34 PDT 2026; root:xnu-12377.121.10~1/RELEASE_ARM64_T6041 arm64`. -- Docker server: `29.6.2 linux/arm64`. -- Verified pinned images: - - `qdrant/qdrant:v1.18.2@sha256:75eab8c4ba42096724fdcfde8b4de0b5713d529dde32f285a1f86fdcb2c9e50c`. - - `ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a`. -- Workspace registry smoke ephemeral image: - - Manifest list: `sha256:4d056bf2cb38d0e8ede91fbf121df1f9f18caee0d401581618ccef9ed8a55e73`. - - Config: `sha256:613f8fb28c0517adee4085f41bc447f2c3813b0fdbb7b26624bfb4cb192b6fd8`. - - Removed during cleanup. - -## Manual Gates - -- GPU exposure gate (`THOTH_ENABLE_EMBEDDING_GPU=1` on Linux): not executed in this run. -- Windows Docker Desktop startup/manual job: not executed in this run. - -## Commits - -- `4e810af` (`test: align qdrant ollama verification fixtures`) -- `7c09b98` (`docs: record qdrant ollama verification`) - -## Known Limitations - -- Broad harness Ruff remains existing unrelated debt: `Found 220 errors`. -- Final active-reference audit is not clean; it still finds legacy/negative-guard references outside explicit migration fixture files. -- Ephemeral Task 13 core/frontend image IDs from `internal-semantic-smoke.sh`, `unified-deployment-smoke.sh`, `thothctl-update-smoke.sh`, and `server-deployment-smoke.sh` were removed by exact cleanup and were not emitted in stdout; pinned Qdrant/Ollama digests and the workspace-registry smoke image digest were captured. - -## Fix Round 1 — reviewer findings - -Status: DONE - -Changes: - -- `scripts/workspace-registry-smoke.sh` now derives the smoke image reference from the already unique Compose project instead of using the global tag `thothii-workspace-registry-smoke:local`. -- The workspace-registry cleanup helpers remove and verify only the exact per-run image reference, plus Compose resources labeled with the exact project. -- Added deterministic self-test coverage in `backend/test/workspaces-migrate-legacy.test.ts` via `WORKSPACE_REGISTRY_SMOKE_SELF_TEST=image-cleanup-identity`; it stubs Docker and fails if cleanup touches same-repository foreign tags such as `:local` or another project tag. -- Updated active harness/testing/PRD docs and Python comments that still described the current semantic store as pgvector/vectordb. Preserved schema-v1/v2 and harness legacy compatibility fixtures. -- Updated `PROJECT_STATE.md` with fix-round smoke evidence and a precise, non-overclaiming audit limitation. - -Focused verification: - -- `cd backend && npx vitest run test/workspaces-migrate-legacy.test.ts` - - Passed: `7 passed`. -- `cd harness && .venv/bin/pytest -q tests/test_memory_save_one.py tests/test_adapter_command_regressions.py tests/test_solved_search_cli.py tests/test_search_pack.py` - - Passed: `22 passed, 14 warnings`. -- `cd harness && .venv/bin/ruff check tht/memory.py tht/search/__init__.py tht/workspace.py tht/vectorstore/store.py tests/test_memory_save_one.py tests/test_adapter_command_regressions.py tests/test_solved_search_cli.py` - - Passed: `All checks passed!` -- `bash -n scripts/workspace-registry-smoke.sh && WORKSPACE_REGISTRY_SMOKE_SELF_TEST=image-cleanup-identity bash scripts/workspace-registry-smoke.sh` - - Passed: `workspace registry smoke image cleanup identity self-test passed`. -- `./scripts/test-no-deployment-coupling.sh` - - Passed: `no active retired deployment or external semantic coupling found.` -- `./scripts/verify-workspace-install-docs.sh --fixtures-only` - - Passed through `relative secret-source fixture rejected passed`. -- `cd backend && npx tsc --noEmit -p .` - - Passed with no output. -- `/usr/bin/time -p ./scripts/workspace-registry-smoke.sh` - - Passed: `workspace registry smoke passed`. - - Built exact per-run tag: `thothii-workspace-registry-smoke:thoth-workspace-registry-smoke-thoth-workspace-registry-smoke-10vi3a-19157`. - - Manifest list: `sha256:715b943057929418cad4aa71806d9edbaf823555d19bda6b875297617463fd4a`. - - Config: `sha256:a566521981e08958aae9a12bfc7803bb5f3f835536b4bb8c39df8fcf26063161`. - - Cleanup proof: `no compose containers, volumes, networks, or image remain for thoth-workspace-registry-smoke-thoth-workspace-registry-smoke-10vi3a-19157.` - - Duration: `real 42.06`. - -Fix-round audit command: - -- `rg -n "pgvector|local-vector|THT_VECTOR_|EMBEDDING_BASE_URL|openai_compatible|ollama_compatible" . --glob '!docs/plans/**' --glob '!docs/superpowers/**' --glob '!**/node_modules/**' --glob '!**/.venv/**' --glob '!**/.git/**'` - -Categorized remaining hits: - -- Backend legacy parser/migration compatibility, kept deliberately non-operational for schema-v1/v2 descriptors: `backend/src/workspaces/schema.ts`, `types.ts`, `migrate-legacy.ts`, `runtime-renderer.ts`, `bindings.ts`, `contracts.ts`, `diagnostics.ts`. -- Backend negative guards and legacy fixture tests: `backend/test/workspaces-schema.test.ts`, `workspaces-migrate-v2-qdrant.test.ts`, `workspace-registry.test.ts`, `workspace-runtime-renderer.test.ts`, `workspaces-bindings.test.ts`, `workspaces-contracts.test.ts`, `workspaces-diagnostics.test.ts`, `workspaces-git-repository.test.ts`, `routes-workspaces.test.ts`, `routes-sessions.test.ts`, `provider-credentials.test.ts`. -- Secret/env scrub guards for retired variables: `backend/src/config.ts`, `backend/src/config/secret-bundle.ts`, `backend/src/pi/provider-credentials.ts`, `scripts/compose-with-preflight.sh`, `scripts/test-external-compose-lifecycle.sh`. -- Deployment negative guards and fixture-scope tests: `scripts/test-no-deployment-coupling.sh`, `scripts/test-no-deployment-coupling-scope.sh`, `scripts/test-preprocess-compose-config.sh`, `scripts/test-verify-workspace-install-docs.sh`, `scripts/verify-workspace-install-docs.sh`, `scripts/vector-rotate-bootstrap-password.sh`. -- Harness legacy config compatibility and fixtures: `harness/tht/config.py`, `harness/tht/config_compat.py`, `harness/tests/test_config_resources.py`, `harness/tests/l2/test_session_ablazione.py`, `harness/workspaces/tht.example.yaml`, `harness/workspaces/tht-test.yaml`. -- Retained off-repository migration SQL fixtures: `harness/scripts/create_vector_reader_rpc.sql`, `harness/scripts/create_vector_writer_rpc.sql`. -- Historical/reference notes, not active operator contracts: `brain/codebase/datamart-builder-deployment-gotchas.md`, `PROJECT_STATE.md`. -- Gitignored task report self-reference: `.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-13-implementation.md`. diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-2-report.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-2-report.md deleted file mode 100644 index 9f88efee..00000000 --- a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-2-report.md +++ /dev/null @@ -1,165 +0,0 @@ -Task 2 report — Make collection ownership unique in the Git registry - -Summary - -- Implemented unique Qdrant collection ownership enforcement during registry snapshot activation. -- Registry session revision leases now reject `migration_required` descriptors. -- Legacy migration now requires an explicit target collection and emits schema v3 descriptors. -- Preserved active snapshot rollback behavior on invalid pulled snapshots. - -RED evidence - -Focused RED command from the brief: - -```bash -cd backend -npx vitest run test/workspace-registry.test.ts test/workspaces-migrate-legacy.test.ts \ - -t "collection|migration_required" -``` - -Observed failures before implementation: - -- `rejects duplicate schema v3 collection ownership and keeps the previous active snapshot` - - `registry.pull()` resolved instead of rejecting. -- `does not acquire a session revision lease for a migration_required workspace` - - `acquireSessionRevision()` resolved instead of rejecting. -- `migrates a legacy descriptor only with an explicit target collection into schema v3` - - received schema version `1` instead of `3`. -- `requires an explicit target collection for legacy migration` - - migration did not throw without a collection. - -GREEN evidence - -Focused GREEN command from the brief: - -```bash -cd backend -npx vitest run test/workspace-registry.test.ts test/workspaces-migrate-legacy.test.ts \ - -t "collection|migration_required" -``` - -Fresh result after implementation: - -- 2 files passed -- 4 tests passed -- 0 failures - -Additional verification run after final cleanup: - -```bash -cd backend -npx vitest run test/routes-workspaces.test.ts -npx vitest run -npx tsc --noEmit -p . -git diff --check -``` - -Fresh results: - -- `test/routes-workspaces.test.ts`: 7 passed -- full backend Vitest: 39 files passed, 454 tests passed -- backend typecheck: passed -- `git diff --check`: passed - -Changed files - -- `backend/src/workspaces/registry.ts` -- `backend/src/workspaces/migrate-legacy.ts` -- `backend/test/workspace-registry.test.ts` -- `backend/test/workspaces-migrate-legacy.test.ts` -- `backend/test/routes-workspaces.test.ts` - -Why one extra file changed - -- `backend/test/routes-workspaces.test.ts` needed updating because Task 1 made schema v3 the only operational descriptor shape, and the route test still assumed the old pre-Task-3 runtime behavior. Updating that expectation was necessary to keep the required backend suite verification meaningful. - -Implementation notes - -- Duplicate collection detection is enforced only for operational schema v3 descriptors by tracking `collection -> workspaceId` during activation. -- Duplicate failures are sanitized back to `workspace_invalid` / `Workspace repository content is invalid`. -- `acquireSessionRevision()` now fails closed for `migration_required` revisions. -- Legacy migration CLI now requires `--collection `. -- Legacy migration output is schema v3 with the fixed internal semantic contract: - - `vector_store.engine = qdrant` - - explicit `collection` - - embedding provider `ollama_internal` - - embedding model `qwen3-embedding:0.6b` - -self-review - -- Confirmed invalid pulled snapshots do not replace the previous active snapshot. -- Confirmed duplicate collection enforcement does not affect legacy migration-required descriptors. -- Confirmed create/update publication tests still pass with unique per-workspace collections. -- Confirmed no JSON stdout contract regressions in the migration CLI. -- Kept runtime/data mutation scope descriptor-only; no user workspace repo or Qdrant data changes. - -Concerns - -- No code concerns remaining for Task 2. -- One deliberate scope exception: a route test was updated to align with the already-established Task 1 / Task 3 fail-closed contract. - -Fix round 1 - -Scope - -- Restored meaningful route-level diagnoser coverage without reopening schema-v3 semantic runtime paths. -- Added direct schema-v2 registry coverage for `migration_required` listing and lease rejection. - -Covering test files - -- `backend/test/routes-workspaces.test.ts` -- `backend/test/workspace-registry.test.ts` - -RED command and output - -Command: - -```bash -cd backend -npx vitest run test/routes-workspaces.test.ts test/workspace-registry.test.ts -``` - -Observed result on top of `76bc94d` after adding the restored/new assertions: - -- 2 files passed -- 37 tests passed -- 0 failures - -Why no RED appeared: - -- The review items exposed missing/weakened coverage, not a production behavior bug. -- `/workspaces/:id/test` already reaches the diagnoser for resolvable legacy v2 descriptors. -- Schema-v3 `/workspaces/:id/test` already fails closed before diagnoser entry. -- Schema-v2 descriptors were already listed as `migration_required` and already rejected by `acquireSessionRevision()`. - -GREEN command and output - -Command: - -```bash -cd backend -npx vitest run test/routes-workspaces.test.ts test/workspace-registry.test.ts -npx tsc --noEmit -p . -``` - -Fresh results: - -- covering tests: 2 files passed, 37 tests passed -- backend typecheck: passed - -Changed files - -- `backend/test/routes-workspaces.test.ts` -- `backend/test/workspace-registry.test.ts` -- `.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-2-report.md` - -What changed - -- Split route coverage so `POST /workspaces/validate` still checks canonical validation independently. -- Restored route-level diagnoser coverage through a migration-required schema-v2 descriptor with resolvable legacy bindings. -- Added an explicit schema-v3 fail-closed regression for `POST /workspaces/:id/test`. -- Added a direct schema-v2 registry regression proving `list()` returns `migration_required` and `acquireSessionRevision()` rejects it. - -Concerns - -- No production concerns. This round only tightened coverage and corrected the weakened test expectation. diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-3-report.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-3-report.md deleted file mode 100644 index 82bb1f00..00000000 --- a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-3-report.md +++ /dev/null @@ -1,132 +0,0 @@ -# Task 3 report — Remove external semantic bindings and render internal endpoints - -Date: 2026-08-08 - -## Scope - -Implemented backend-owned schema-v3 semantic runtime rendering so workspace descriptors and installation contracts remain free of external Qdrant/Ollama endpoints and credentials, while DWH bindings stay unchanged. - -## RED evidence - -Focused RED command: - -`cd backend && npx vitest run test/workspaces-contracts.test.ts test/workspaces-bindings.test.ts test/workspace-runtime-renderer.test.ts test/config.test.ts` - -Observed failures before implementation: - -- `config.test.ts` - - missing `internalQdrantUrl` - - missing `internalEmbeddingUrl` -- `workspaces-bindings.test.ts` - - schema v3 semantic binding resolution threw unsupported errors -- `workspace-runtime-renderer.test.ts` - - schema v3 runtime rendering threw `Schema version 3 runtime rendering is unsupported until the internal semantic runtime is implemented` - -## GREEN evidence - -Focused GREEN command: - -`cd backend && npx vitest run test/workspaces-contracts.test.ts test/workspaces-bindings.test.ts test/workspace-runtime-renderer.test.ts test/config.test.ts` - -Result: - -- 4 test files passed -- 36 tests passed - -Typecheck: - -`cd backend && npx tsc --noEmit -p .` - -Result: - -- passed - -Hygiene: - -- `git diff --check` passed - -## Files changed - -Listed-task files changed: - -- `backend/src/config.ts` -- `backend/src/workspaces/bindings.ts` -- `backend/src/workspaces/runtime-renderer.ts` -- `backend/test/config.test.ts` -- `backend/test/workspace-runtime-renderer.test.ts` -- `backend/test/workspaces-bindings.test.ts` -- `backend/test/workspaces-contracts.test.ts` - -Listed-task files inspected but not changed: - -- `backend/src/workspaces/contracts.ts` - -Unavoidable additional wiring changes: - -- `backend/src/app.ts` -- `backend/src/tht/tht-runner.ts` - -Reason: the new typed internal semantic runtime config had to flow from backend config into ephemeral harness config rendering at runtime. - -## Behavior delivered - -- schema-v3 installation contract exposes DWH bindings only -- schema-v3 binding resolution ignores external semantic env vars instead of sourcing runtime semantics from them -- runtime rendering for schema v3 emits backend-owned internal semantic endpoints: - - Qdrant: `http://qdrant:6333` - - Embedding: `http://embedding:11434` - - Model: `qwen3-embedding:0.6b` - - Dimensions: `1024` -- internal semantic URLs are validated to allow only `qdrant` / `embedding` / `localhost` / loopback hosts -- DWH transport/runtime behavior remains unchanged - -## Self-review - -- Confirmed schema-v3 contracts/docs no longer advertise VECTOR or EMBEDDING installation variables. -- Confirmed schema-v3 runtime output ignores injected external semantic endpoints from env bindings. -- Confirmed semantic endpoints are rendered only in the ephemeral backend-owned harness config path. -- Confirmed type wiring is explicit from `AppConfig` → `ThtRunner` → runtime renderer. - -## Concerns - -- Host validation currently permits both `http` and `https` on the allowed internal hosts. That keeps the configuration flexible, but if the installation contract intended `http` only, that restriction is not enforced here. - -## Fix round 1/5 - -Scope: - -- moved schema-v3 internal embeddings under `resources.embeddings` -- enforced `http`-only internal semantic URLs - -RED evidence: - -`cd backend && npx vitest run test/workspace-runtime-renderer.test.ts test/config.test.ts` - -Observed failures on `bc8afe0`: - -- `workspace-runtime-renderer.test.ts` - - schema-v3 output omitted `resources.embeddings` - - schema-v3 still exposed top-level `embeddings` -- `config.test.ts` - - `https://qdrant:6333` was accepted - -GREEN evidence: - -`cd backend && npx vitest run test/workspace-runtime-renderer.test.ts test/config.test.ts` - -Result: - -- 2 test files passed -- 16 tests passed - -Typecheck: - -`cd backend && npx tsc --noEmit -p .` - -Result: - -- passed - -Updated concerns: - -- none for this round beyond future tightening if exact-port rejection is later requested explicitly. diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-4-report.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-4-report.md deleted file mode 100644 index d3248faf..00000000 --- a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-4-report.md +++ /dev/null @@ -1,149 +0,0 @@ -# Task 4 Report — Narrow harness embedding configuration to internal Ollama - -## Status - -Implemented on 2026-08-08 in `/Users/mp/projects/ThothII/.worktrees/git-workspace-registry`. - -## RED evidence - -Command: - -```bash -cd harness -./.venv/bin/pytest tests/test_internal_embeddings.py tests/test_config_resources.py -q -``` - -Observed before implementation: - -- exit code `1` -- `10 failed, 10 passed` -- failures proved the missing `OllamaInternalEmbeddings` client and missing internal-only config validation - -Representative failures: - -- `ImportError: cannot import name 'OllamaInternalEmbeddings'` -- `AttributeError: 'EmbeddingsConfig' object has no attribute 'provider'` -- config tests `DID NOT RAISE ConfigError` for external provider, API key, and non-private base URL - -## GREEN evidence - -Focused behavior suite: - -```bash -cd harness -./.venv/bin/pytest tests/test_internal_embeddings.py tests/test_config_resources.py -q -``` - -- exit code `0` -- `20 passed` - -Relevant harness verification: - -```bash -cd harness -./.venv/bin/pytest tests/test_internal_embeddings.py tests/test_config_resources.py tests/test_ollama_ensure.py -q -``` - -- exit code `0` -- `36 passed, 2 warnings` - -Changed-file lint: - -```bash -cd harness -./.venv/bin/ruff check tht/config.py tht/config_compat.py tht/vectorstore/embeddings.py tht/cli/ollama_cmd.py tests/test_config_resources.py tests/test_internal_embeddings.py -``` - -- exit code `0` -- `All checks passed!` - -Patch hygiene: - -```bash -git diff --check -``` - -- exit code `0` - -## What changed - -- translated schema-v3 `resources.embeddings` into the harness-compatible embedding config view -- validated the internal embedding contract only for that runtime-owned `resources.embeddings` path: - - provider must be `ollama_internal` - - model must be `qwen3-embedding:0.6b` - - dimensions must be `1024` - - base URL must be `http://embedding:11434` or loopback HTTP on port `11434` - - extra fields like `api_key` are rejected -- replaced the active embed client with `OllamaInternalEmbeddings`, using one bounded `/api/embed` request per batch -- removed task/query prefix rewriting from the active embedding path -- validated response count, vector dimension, and finite numeric values before returning embeddings -- kept `tht ollama ensure --json` stdout pristine while warming through the internal client - -## Self-review - -- kept changes inside the brief-listed files -- preserved DWH and session-persistence behavior -- preserved the legacy `OllamaEmbeddings` import path as an alias to avoid unrelated call-site churn - -## Concerns - -- the focused harness verification still emits two pre-existing warnings: - - `DeprecationWarning` from `testcontainers.postgres` - - `FutureWarning` because `resources` currently flows through the legacy config translation path - -## Fix round 1 — 2026-08-08 - -### Findings addressed - -- HIGH: external top-level `embeddings` remained an operational fallback and could still load -- MEDIUM: non-object embed JSON payloads escaped as raw `AttributeError` - -### RED evidence - -Command: - -```bash -cd harness -./.venv/bin/pytest tests/test_internal_embeddings.py tests/test_config_resources.py tests/test_ollama_ensure.py -q -``` - -Observed before the fix: - -- exit code `1` -- `2 failed, 36 passed, 2 warnings` - -Representative failures: - -- `AttributeError: 'list' object has no attribute 'get'` from `response.json()` returning a JSON array -- `Failed: DID NOT RAISE ConfigError` for top-level external `embeddings.provider=openai_compatible` - -### GREEN evidence - -Command: - -```bash -cd harness -./.venv/bin/pytest tests/test_internal_embeddings.py tests/test_config_resources.py tests/test_ollama_ensure.py -q -``` - -Observed after the fix: - -- exit code `0` -- `38 passed, 2 warnings` - -Touched-file lint: - -```bash -cd harness -./.venv/bin/ruff check tht/config.py tht/vectorstore/embeddings.py tests/test_internal_embeddings.py tests/test_config_resources.py -``` - -- exit code `0` -- `All checks passed!` - -### Minimal fix - -- validated the final active `cfg.embeddings` contract after config loading, so legacy top-level - embedding inputs now fail explicitly unless they exactly match the internal Ollama contract -- converted non-mapping embed JSON payloads into controlled `EmbeddingsError` failures with - sanitized diagnostics instead of raw attribute errors diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-5-report.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-5-report.md deleted file mode 100644 index d4243cc5..00000000 --- a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-5-report.md +++ /dev/null @@ -1,170 +0,0 @@ -# Task 5 Report — Implement the Qdrant VectorStore adapter - -## Status - -Implemented on 2026-08-08 in `/Users/mp/projects/ThothII/.worktrees/git-workspace-registry`. - -## RED evidence - -Command: - -```bash -cd harness -./.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -q -``` - -Observed before implementation: - -- exit code `2` -- collection failed during import because the adapter did not exist yet - -Representative failures: - -- `ModuleNotFoundError: No module named 'tht.adapters.vector.qdrant'` - -## GREEN evidence - -Focused behavior suite: - -```bash -cd harness -./.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -q -``` - -- exit code `0` -- `31 passed, 1 warning` - -Touched-file lint: - -```bash -cd harness -./.venv/bin/ruff check tht/adapters/vector/qdrant.py tht/adapters/vector/__init__.py \ - tht/ports/vector.py tht/vectorstore/records.py tht/vectorstore/store.py \ - tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -``` - -- exit code `0` -- `All checks passed!` - -Patch hygiene: - -```bash -git diff --check -``` - -- exit code `0` - -## What changed - -- added `QdrantVectorStore` with direct `requests`-based REST calls for: - - `GET /collections/{collection}` - - `PUT /collections/{collection}` - - `PUT /collections/{collection}/index` - - `PUT /collections/{collection}/points?wait=true` - - `POST /collections/{collection}/points/query` - - `POST /collections/{collection}/points/scroll` - - `POST /collections/{collection}/points/delete?wait=true` -- implemented idempotent collection provisioning for `1024` dimensions and `Cosine` distance -- created deterministic UUIDv5 point IDs from workspace, semantic kind, and canonical record key -- preserved canonical record identity and only upserted/deleted points matching the exact workspace - and generation filters -- added Qdrant payload helpers so stored payloads carry: - - `workspace_id` - - grouped semantic `kind` (`schema`, `evidence`, `memory`) - - original `record_kind` - - canonical `record_key` - - `content_hash` - - existing Thoth metadata fields -- mapped Qdrant payloads back into existing `VectorHit` objects without losing the original - Thoth kind -- exported the new adapter from the public vector adapter package and added focused contract tests -- sanitized timeout and malformed-response failures so CLI-facing callers do not leak raw endpoint - details - -## Self-review - -- confirmed collection mismatch fails without any delete/recreate path -- confirmed every query/scroll/delete operation includes a workspace filter -- confirmed the adapter never deletes or rewrites unrelated Qdrant points -- added keyword payload indexes for all filter-critical fields used here, including `document_id` - for exact Evidence filtering - -## Concerns - -- the requested `adversarial-review` skill could not run its full external reviewer flow in this - environment because the skill’s referenced `brain/` files are missing at - `/Users/mp/.agents/skills/adversarial-review`; I performed a manual adversarial self-review - instead -- the focused suite still emits one pre-existing warning from `testcontainers.postgres` - -## Fix round 1 — 2026-08-08 - -### Findings addressed - -- IMPORTANT: metadata collisions could override canonical Qdrant payload identity fields and break - workspace isolation -- IMPORTANT: scroll-based operations only read the first page and did not follow - `next_page_offset`, making `existing_hashes`, `list_evidence_generations`, and delete counts - inexact beyond one page - -### RED evidence - -Command: - -```bash -cd harness -./.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -q -``` - -Observed before the fix: - -- exit code `1` -- `2 failed, 31 passed, 1 warning` - -Representative failures: - -- `assert payload["workspace_id"] == "demo"` failed because colliding `record.metadata` - overwrote canonical payload fields -- paginated scroll test missed later pages, so `existing_hashes` and generation cleanup counts - were incomplete - -### GREEN evidence - -Command: - -```bash -cd harness -./.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -q -``` - -Observed after the fix: - -- exit code `0` -- `33 passed, 1 warning` - -Touched-file lint: - -```bash -cd harness -./.venv/bin/ruff check tht/adapters/vector/qdrant.py tht/vectorstore/records.py \ - tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -``` - -- exit code `0` -- `All checks passed!` - -Patch hygiene: - -```bash -git diff --check -``` - -- exit code `0` - -### Minimal fix - -- made `qdrant_payload` apply canonical fields after `record.metadata` so workspace ID, semantic - kind, original record kind, canonical record key, and content hash cannot be overridden by - metadata collisions -- paginated `_scroll` until `next_page_offset` is absent, sent the returned `offset` back on the - next request, and reject repeated offsets as malformed to avoid infinite loops diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-6-report.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-6-report.md deleted file mode 100644 index 8b70773e..00000000 --- a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-6-report.md +++ /dev/null @@ -1,144 +0,0 @@ -# Task 6 Report - -Date: 2026-08-08 - -Status: implemented and verified - -Summary: - -- Added schema-v3 Qdrant runtime support to the harness config/resource layer and vector factory. -- Made Qdrant payloads carry `workspace_id` and `workspace_revision` on every point. -- Routed schema and memory bulk indexing through the transport-neutral vector port with canonical hash-based dedup. -- Kept Evidence canonical on filesystem and Memory canonical in JSONL; Qdrant remains derived/rebuildable. -- Added focused tests for semantic-kind isolation, shared identity fields, search-pack kind boundaries, and the schema-v3 factory/config path. - -Files changed: - -- `harness/tht/config.py` -- `harness/tht/config_compat.py` -- `harness/tht/adapters/factory.py` -- `harness/tht/adapters/vector/qdrant.py` -- `harness/tht/vectorstore/records.py` -- `harness/tht/cli/vector_cmd.py` -- `harness/tht/cli/memory_cmd.py` -- `harness/tests/test_semantic_kind_isolation.py` -- `harness/tests/test_memory_save_one.py` -- `harness/tests/test_search_pack.py` -- `harness/tests/test_qdrant_vector_store.py` -- `harness/tests/test_adapter_factory.py` -- `harness/tests/test_config_resources.py` - -Verification: - -- Focused RED/GREEN task suite: - - `cd harness && .venv/bin/pytest tests/test_semantic_kind_isolation.py tests/test_memory_save_one.py tests/test_search_pack.py -q` -- Relevant harness suite: - - `cd harness && .venv/bin/pytest tests/test_semantic_kind_isolation.py tests/test_memory_save_one.py tests/test_search_pack.py tests/test_qdrant_vector_store.py tests/test_adapter_factory.py tests/test_config_resources.py tests/test_vector_port_contract.py tests/test_corpus_pipeline.py -q` - - Result: `131 passed` -- Changed-file Ruff: - - `cd harness && .venv/bin/ruff check tht/vectorstore/records.py tht/adapters/vector/qdrant.py tht/config_compat.py tht/config.py tht/adapters/factory.py tht/cli/vector_cmd.py tht/cli/memory_cmd.py tests/test_memory_save_one.py tests/test_search_pack.py tests/test_semantic_kind_isolation.py tests/test_qdrant_vector_store.py tests/test_adapter_factory.py tests/test_config_resources.py` - - Result: clean - -Concerns / follow-up: - -- `memory clear` still retains its older direct-vector assumptions and was not expanded in this task because the brief focused on canonical builders and schema/evidence/memory routing through the active Qdrant path. -- The relevant suite still emits pre-existing warnings (legacy config deprecation in older fixtures, plus existing Pydantic serializer warnings in corpus tests), but they are not introduced by this task. - -## Fix round 1 (2026-08-08) - -Scope: - -- Fixed qdrant-only schema-v3 command gating for `vector index-schema`, `memory promote`, and `memory index`. -- Replaced `memory clear`'s direct-pgvector-only path with vector-port deletion by kind. -- Added focused qdrant-only CLI regression tests and refreshed older CLI fixtures to the enforced internal embedding contract. - -RED evidence: - -- `cd harness && .venv/bin/pytest tests/test_qdrant_cli_commands.py -q` -- Initial result against commit `5e39cfa`: `4 failed` -- Failure signatures: - - `ERRORE: sezioni mancanti nel workspace yaml: vector_db o vector_write_rest.` - - `ERRORE: sezioni mancanti nel workspace yaml: vector_db.` - -GREEN evidence: - -- Focused fix suite: - - `cd harness && .venv/bin/pytest tests/test_qdrant_cli_commands.py tests/test_qdrant_vector_store.py tests/test_adapter_factory.py tests/test_config_resources.py tests/test_memory_save_one.py tests/test_search_pack.py -q` - - Result: `51 passed` -- Relevant broader vector/memory/schema/search suite: - - `cd harness && .venv/bin/pytest tests/test_qdrant_cli_commands.py tests/test_qdrant_vector_store.py tests/test_adapter_factory.py tests/test_config_resources.py tests/test_memory_save_one.py tests/test_search_pack.py tests/test_vector_port_contract.py tests/test_adapter_command_regressions.py tests/test_solved_search_cli.py tests/test_schema_introspect_guard.py tests/test_semantic_kind_isolation.py tests/test_corpus_pipeline.py -q` - - Result: `154 passed` -- Ruff on the fix surface: - - `cd harness && .venv/bin/ruff check tht/ports/vector.py tht/adapters/vector/qdrant.py tht/adapters/vector/pgvector.py tht/adapters/vector/thoth_http.py tht/vectorstore/rest_client.py tht/cli/vector_cmd.py tht/cli/memory_cmd.py tests/test_qdrant_cli_commands.py tests/test_solved_search_cli.py` - - Result: clean - -Notes: - -- `memory clear` now deletes derived `kind=memory` points through the configured writable vector store, while leaving the JSONL registry as the source of truth until the registry file is removed by the command. -- The broader suite still carries the same pre-existing warnings noted above; this fix round did not add new warnings or failures. - -## Fix round 2 (2026-08-08) - -Scope: - -- Removed the accidental HTTP writer `delete_kinds` capability expansion from `ThothHttpVectorStore` and `VectorRestClient`. -- Reworked `memory clear` so schema-v3 Qdrant uses scoped `kind=memory` deletion, while legacy transports keep the pre-task direct-sync path instead of advertising a nonexistent RPC. -- Tightened the qdrant-only memory-clear regression to assert the exact `("memory", ["memory"])` delete scope. - -RED evidence: - -- Re-review found a transport contract mismatch in fix round 1: - - `ThothHttpVectorStore` exposed `delete_kinds(...)` - - `VectorRestClient` exposed `delete_kinds(...)` - - but the legacy HTTP writer migration only allowlists `delete_vector_generation`, not `delete_vector_kinds` -- The new regressions added in this round capture that mismatch and the missing qdrant delete-scope assertion: - - `tests/test_vector_port_contract.py::test_http_store_supports_writer_without_reader` - - `tests/l0/test_vector_adapter_parity.py::test_http_rest_client_does_not_advertise_nonexistent_delete_kinds_rpc` - - `tests/test_qdrant_cli_commands.py::test_memory_clear_accepts_qdrant_only_runtime_config` - -GREEN evidence: - -- Focused regression suite: - - `cd harness && .venv/bin/pytest tests/test_qdrant_cli_commands.py tests/test_vector_port_contract.py tests/l0/test_vector_adapter_parity.py tests/test_adapter_command_regressions.py -q` - - Result: `53 passed` -- Broader relevant vector/memory/search suite: - - `cd harness && .venv/bin/pytest tests/test_qdrant_cli_commands.py tests/test_adapter_command_regressions.py tests/test_vector_port_contract.py tests/l0/test_vector_adapter_parity.py tests/test_solved_search_cli.py tests/test_qdrant_vector_store.py tests/test_search_similar_kinds.py tests/test_corpus_pipeline.py -q` - - Result: `135 passed` -- Ruff on the changed fix surface: - - `cd harness && .venv/bin/ruff check tht/cli/memory_cmd.py tht/ports/vector.py tht/adapters/vector/thoth_http.py tht/vectorstore/rest_client.py tests/test_qdrant_cli_commands.py tests/test_vector_port_contract.py tests/l0/test_vector_adapter_parity.py` - - Result: clean - -Notes: - -- Legacy HTTP/vector-rest deployments do not gain a new destructive RPC surface from this fix; they keep their previous behavior and continue to fail closed for unsupported cleanup. -- The broader suite still emits the same pre-existing deprecation and serializer warnings already noted above; this round did not introduce new warnings. - -## Fix round 3 (2026-08-08) - -Scope: - -- Added an adapter-level Qdrant regression for mixed semantic kinds within one workspace plus a second workspace memory point. -- Verified that `delete_kinds("memory", ["memory"])` emits the real adapter filter with both `workspace_id=demo` and `record_kind=memory`. -- Verified that non-memory semantic kinds in the same workspace and memory from another workspace survive the delete. - -RED evidence: - -- Re-review identified a test gap rather than a confirmed runtime bug: - - existing coverage asserted only the CLI mock call shape for qdrant memory clear - - there was no adapter-level regression proving the real Qdrant delete filter and resulting fake-Qdrant state across mixed semantic kinds/workspaces -- Added regression: - - `tests/test_qdrant_vector_store.py::test_delete_kinds_is_workspace_scoped_and_preserves_other_semantic_kinds` - -GREEN evidence: - -- Requested focused suite: - - `cd harness && .venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_qdrant_cli_commands.py tests/test_semantic_kind_isolation.py -q` - - Result: `18 passed` -- Ruff on changed files: - - `cd harness && .venv/bin/ruff check tests/test_qdrant_vector_store.py` - - Result: clean - -Notes: - -- This round required no production change; the new adapter regression passed against the existing Qdrant implementation. -- The focused suite still emits the same pre-existing `testcontainers.postgres` deprecation warning from `tests/conftest.py`; no new warnings were introduced. diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-7-report.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-7-report.md deleted file mode 100644 index a0d72e66..00000000 --- a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-7-report.md +++ /dev/null @@ -1,98 +0,0 @@ -# Task 7 report — mandatory Qdrant and Ollama Compose services - -Date: 2026-08-08 - -Status: completed - -Summary: - -- Added mandatory private `qdrant`, `embedding`, and `embedding-model-init` services to the base Compose stack. -- Pinned Qdrant `v1.18.2` and Ollama `0.32.0` by immutable multi-arch digest. -- Persisted Qdrant storage in `qdrant-data` and Ollama model cache in `embedding-models`. -- Wired `core` to fixed internal semantic endpoints: - - `THT_INTERNAL_QDRANT_URL=http://qdrant:6333` - - `THT_INTERNAL_EMBEDDING_URL=http://embedding:11434` - - `THT_INTERNAL_EMBEDDING_MODEL=qwen3-embedding:0.6b` - - `THT_INTERNAL_EMBEDDING_DIMENSIONS=1024` -- Removed external vector / embedding endpoint requirements from the local and server env examples. -- Added an idempotent Ollama model bootstrap script that: - - waits up to a bounded deadline for `/api/tags` - - skips `ollama pull` when the model is already cached - - pulls `qwen3-embedding:0.6b` only when needed - - verifies the model appears in `/api/tags` after pull -- Added optional GPU override file `deploy/compose.embedding-gpu.yaml`; base Compose remains CPU-only. -- Updated `scripts/run-stack.sh` so the GPU override is included only when `THOTH_ENABLE_EMBEDDING_GPU=1`. - -Verification: - -- RED confirmed before implementation: - - `./scripts/test-default-compose.sh` failed on missing required services. - - `./scripts/test-unified-compose.sh` failed on missing required services. - - `./scripts/test-internal-semantic-compose.sh` failed because the GPU override file did not exist. -- GREEN after implementation: - - `./scripts/test-default-compose.sh` - - `./scripts/test-unified-compose.sh` - - `./scripts/test-internal-semantic-compose.sh` - - `git diff --check` -- Additional shell verification: - - `scripts/run-stack.sh --wait` includes only base + local Compose files by default. - - `THOTH_ENABLE_EMBEDDING_GPU=1 scripts/run-stack.sh --wait` adds `deploy/compose.embedding-gpu.yaml`. - -Resolved image digests: - -- `qdrant/qdrant:v1.18.2@sha256:75eab8c4ba42096724fdcfde8b4de0b5713d529dde32f285a1f86fdcb2c9e50c` -- `ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a` - -Self-review: - -- The first bootstrap-script draft depended on tools not guaranteed inside the Ollama image. This was corrected after image inspection; the final script uses only confirmed image tools (`bash`, `ollama`, `grep`) plus raw HTTP over `/dev/tcp`. -- The server overlay intentionally replaces most named core volumes with bind mounts, so the unified contract was tightened to require named semantic-cache volumes there while preserving the local/base named-volume checks. - -Concerns: - -- The model bootstrap waits for Ollama readiness and verifies cache state, but the first real cold-start will still take time to download `qwen3-embedding:0.6b`. -- The GPU override requests generic Docker GPU capability only; actual GPU availability remains host/runtime dependent and intentionally stays opt-in. - -## Fix round 1 / 5 — 2026-08-08 - -Rulings applied: - -- Kept the Task 1 boundary intact: schema-v3 remains the only operational workspace descriptor shape. -- Did not restore any external semantic fallback for schema-v2 live sessions. -- Treated `PROJECT_STATE.md` as stale documentation for this point, not runtime truth. - -Focused schema-v2 evidence: - -- Re-ran the existing targeted registry test: - - `cd backend && npx vitest run test/workspace-registry.test.ts -t "lists a schema v2 descriptor as migration_required and refuses to acquire it"` -- Result: pass. -- Evidence from that test: - - schema-v2 descriptors list as `migration_required` - - `acquireSessionRevision("psd-clinical")` rejects with `code: "workspace_invalid"` -- Conclusion: schema-v2 acquisition remains blocked; no external semantic fallback was reintroduced. - -Contract consistency fixes: - -- Updated `harness/tests/test_local_compose_contract.py` to assert the mandatory internal semantic stack, fixed internal core semantic env, private-service topology, persistent volumes, and Ollama health/dependency contract. -- Updated shell Compose contracts to require: - - Ollama healthcheck on `embedding` - - `embedding-model-init` dependency on `embedding: service_healthy` -- Updated `scripts/unified-deployment-smoke.sh` rendered-contract helper to expect the mandatory internal semantic topology and internal semantic env names, and to reject retired external semantic bindings. -- Updated `scripts/test-task13-runtime-fixtures.sh` to exercise `task13_assert_rendered_contract` for both local and server fixture renders. - -Fix round 1 verification: - -- RED before implementation: - - `cd harness && .venv/bin/pytest tests/test_local_compose_contract.py -q` failed because `embedding` had no healthcheck. - - `./scripts/test-default-compose.sh` failed because `embedding` had no healthcheck. - - `./scripts/test-unified-compose.sh` failed because `embedding` had no healthcheck. - - `./scripts/test-task13-runtime-fixtures.sh local` failed because `unified-deployment-smoke.sh` still expected `core,frontend`. -- GREEN after implementation: - - `./scripts/test-default-compose.sh` - - `./scripts/test-unified-compose.sh` - - `./scripts/test-internal-semantic-compose.sh` - - `cd harness && .venv/bin/pytest tests/test_local_compose_contract.py -q` - - `./scripts/test-task13-runtime-fixtures.sh local` - - `./scripts/test-task13-runtime-fixtures.sh server` - - `cd backend && npx vitest run test/workspace-registry.test.ts -t "lists a schema v2 descriptor as migration_required and refuses to acquire it"` - - `docker compose --env-file deploy/env/local.env.example -f compose.yaml -f deploy/compose.local.yaml config --format json` diff --git a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-9-report.md b/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-9-report.md deleted file mode 100644 index da2f169d..00000000 --- a/.superpowers/sdd/2026-08-08-internal-qdrant-ollama/task-9-report.md +++ /dev/null @@ -1,65 +0,0 @@ -Status: completed on August 8, 2026. - -Summary: -- Updated the frontend workspace contract from schema v2 editing to schema v3 publishing. -- Kept only `semantic_index.vector_store.collection` editable; rendered qdrant / internal Ollama semantic values as fixed read-only architecture values. -- Removed external vector transport / endpoint / credential / embedding diagnostics branches from frontend draft sanitization, conflict parsing, and editor UI. -- Added a migration-required banner in workspace management and blocked `migration_required` workspaces from new-session selection. -- Aligned the example workspace YAML comments with the fixed internal qdrant/Ollama architecture. - -Files changed: -- `frontend/src/api/workspaces.ts` -- `frontend/src/api/workspaces.test.ts` -- `frontend/src/workspaces/drafts.ts` -- `frontend/src/workspaces/drafts.test.ts` -- `frontend/src/shell/WorkspaceEditor.tsx` -- `frontend/src/shell/WorkspaceEditor.test.tsx` -- `frontend/src/shell/WorkspaceManager.tsx` -- `frontend/src/shell/WorkspaceManager.test.tsx` -- `frontend/src/shell/WorkspacePublishDialog.test.tsx` -- `frontend/src/api/sessions.ts` -- `frontend/src/shell/SteerInput.tsx` -- `frontend/src/shell/SteerInput.test.tsx` -- `deploy/workspaces/example.yaml` -- `deploy/workspaces/psd.yaml.example` - -Verification: -- `cd frontend && npx vitest run src/shell/SteerInput.test.tsx src/shell/WorkspaceEditor.test.tsx src/shell/WorkspaceManager.test.tsx src/shell/WorkspacePublishDialog.test.tsx src/workspaces/drafts.test.ts src/api/workspaces.test.ts` - - Result: 6 files passed, 59 tests passed. -- `cd frontend && npx tsc -b` - - Result: passed. -- `git diff --check` - - Result: passed. - -Self-review: -- The frontend now publishes the exact schema v3 semantic shape and no longer persists legacy semantic transport/credential branches. -- Migration-required workspaces are visible in management with an explicit banner and are excluded from the composer workspace selector. -- One dependent test file outside the original brief list (`WorkspacePublishDialog.test.tsx`) and the composer/session-selection path (`api/sessions.ts`, `SteerInput.tsx`, related test) were updated because they were directly coupled to the old v2 semantic/edit-selection behavior. - -Concerns: -- The composer still retains backward-compatible behavior for summaries that omit `revision` entirely; only explicit `revision.state === "migration_required"` is blocked. That matches the current mixed-test environment, but once summary responses are guaranteed to include `revision`, that fallback may be removable. - -Fix round 1/5 — August 8, 2026 - -Summary: -- Made missing or invalid workspace summaries fail safe in frontend session creation and composer selection instead of falling open as legacy. -- Added an actionable unavailable message in workspace management for incomplete summaries with no canonical revision. -- Replaced the old runtime-oriented example descriptor files with exact backend WorkspaceV3 descriptor YAML. - -Additional files changed: -- `frontend/src/api/sessions.test.ts` -- `backend/test/workspaces-schema.test.ts` - -Fix-round verification: -- `cd frontend && npx vitest run src/api/sessions.test.ts src/shell/SteerInput.test.tsx src/shell/WorkspaceManager.test.tsx src/shell/WorkspaceEditor.test.tsx src/shell/WorkspacePublishDialog.test.tsx src/workspaces/drafts.test.ts src/api/workspaces.test.ts` - - Result: 7 files passed, 73 tests passed. -- `cd frontend && npx tsc -b` - - Result: passed. -- `cd backend && npx vitest run test/workspaces-schema.test.ts` - - Result: 1 file passed, 17 tests passed. -- `git diff --check` - - Result: passed. - -Notes: -- Missing `revision` in a workspace summary now fails with the same session/composer safety posture as `migration_required`, using the existing safe workspace-policy error for session creation and an explicit unavailable message in workspace management. -- The committed example files now validate as actual schema-v3 descriptors instead of deployment/runtime templates with forbidden semantic endpoint fields. diff --git a/.superpowers/sdd/2026-08-15-unified-tht-cli-product-step/task-5-report.md b/.superpowers/sdd/2026-08-15-unified-tht-cli-product-step/task-5-report.md deleted file mode 100644 index ef8da22c..00000000 --- a/.superpowers/sdd/2026-08-15-unified-tht-cli-product-step/task-5-report.md +++ /dev/null @@ -1,83 +0,0 @@ -# Task 5 Report — `tht setup` lifecycle orchestration - -## Status - -Completed. `tht setup` now validates the checkout and host prerequisites, creates or validates -the non-secret installation files, validates Compose, and by default builds, starts, health-checks, -and verifies the installation. `tht setup --configure-only` stops immediately after successful -Compose rendering. - -## Implementation - -- Added `setup.Run`, with an ordered host preflight: project/worktree discovery, Docker Engine, - Docker Compose, supported architecture, and LF line-ending checks. -- Reused `config.Installation.ComposeArgs` for all Compose calls and added a narrow - `compose.InstallationRunner` adapter for Pi diagnostics; no shell command construction was added - to the top-level CLI parser. -- Default setup performs `compose build`, `compose up --detach --remove-orphans`, bounded polling - for `core`, `frontend`, `qdrant`, `embedding`, and `embedding-model-init`, then aggregate volume - diagnostics and `pi.Doctor`. -- Health timeout errors identify the last failing service and preserve containers for diagnosis, - with `tht logs ` and `tht status` guidance. -- Completion output includes the frontend URL, selected descriptor, and next action. - -## TDD evidence - -The initial focused test run failed because `setup.Run` did not exist. Tests were then written -against a fake Compose runner before the orchestration was implemented. They cover the complete -ordered flow, configure-only stop, preflight failure before writing configuration, health retry, -timeout guidance, and CLI default versus `--configure-only` dispatch. - -## Verification - -Executed from `tools/tht`: - -```bash -go test ./internal/setup ./internal/compose ./cmd/tht -run 'TestRun|TestSetupCommand|TestInstallationRunner' -count=1 -go test ./internal/setup ./internal/compose ./cmd/tht -count=1 -go test ./... -git diff --check -``` - -All commands passed. No actual Docker build, container start, live-stack restart, system -installation, Pi configuration edit, or documentation rewrite was performed. - -## Commit - -`feat(setup): build start and verify ThothII` (this report is included in that commit). - -## Concerns - -- The bounded health wait is verified with fakes only, as required for this task. Real Docker - lifecycle verification belongs to the later live acceptance task. -- The existing aggregate `tht doctor` command remains a separate implementation; Task 5 performs - its equivalent setup-time prerequisite checks plus `pi.Doctor` without invoking a nested CLI - process. - -## Fix round 1 - -The independent review identified three gaps. All were reproduced with RED tests before the -production change: - -- A rendered Compose document containing any one volume was accepted. `requireVolumes` now - requires `settings`, `pi-state`, `workspace-registry`, `workspace-secrets`, `sessions`, - `qdrant-data`, and `embedding-models`; tests reject each individual omission and an - unrelated-only volume set. -- Failures after `compose up` could return without recovery instructions. A single recovery - wrapper now preserves the underlying error while adding the retained-container, `tht logs - `, and `tht status` guidance for failed `up`, health, aggregate doctor, and Pi doctor - phases. Focused tests also prove build failure stops before attempting startup. -- LF inspection previously walked the full checkout. It now inspects only `compose.yaml`, - `deploy/`, and `docker/`; a test proves CRLF content under `node_modules/` is ignored. - -Verification added for this round: - -```bash -go test ./internal/setup -run 'TestRequireVolumes|TestRun(BuildFailure|UpFailure|AggregateDoctorFailure|PiDoctorFailure|IgnoresIrrelevant|TimesOut)' -count=1 -go test ./internal/setup -count=1 -``` - -Both passed before the final full-suite verification. No Docker or live operation was run. - -Implementation commit evidence: `ea70cc95b04532043744a9de6c5912e30a214595` — -`fix(setup): harden verification and recovery`. diff --git a/.superpowers/sdd/2026-08-15-unified-tht-cli-product-step/task-6-report.md b/.superpowers/sdd/2026-08-15-unified-tht-cli-product-step/task-6-report.md deleted file mode 100644 index 90a7d55b..00000000 --- a/.superpowers/sdd/2026-08-15-unified-tht-cli-product-step/task-6-report.md +++ /dev/null @@ -1,55 +0,0 @@ -# Task 6 — Version, aggregate doctor, and build-aware start - -Status: complete. - -Implemented the host-side `tht version`, aggregate `tht doctor [--json]`, and `tht start [--build]` contracts. - -- `version` is descriptor-free and reports semantic version, commit, build time, OS, and architecture. -- `doctor` emits typed, redacted checks for descriptor state, Docker/Compose, rendered volumes, file permissions, service health, workspace registry, the container-local workflow doctor, and Pi doctor. Its JSON mode writes exactly one JSON document to stdout. -- The Python workflow doctor is invoked only as `docker compose exec -T core tht doctor --json` after core is running. -- `start` uses the shared lifecycle service: default `up → health`; `--build` is `build → up → health`. -- `setup` now reuses the shared lifecycle and aggregate diagnostics rather than keeping parallel health/volume implementations. - -Verification performed without live Docker/container commands: - -```bash -cd tools/tht -go test ./internal/version ./internal/doctor ./internal/service ./cmd/tht \ - -run 'TestVersion|TestDoctor|TestStart|TestCurrent|TestRun' -count=1 -go test ./internal/setup -count=1 -run 'TestRun' -v -go test ./... -count=1 -git diff --check -``` - -All completed successfully. The intentionally fake runner coverage includes unavailable Docker, -stopped/running core, workflow failure redaction, pristine JSON output, and start ordering. - -Concerns: no live Docker validation or host installation was run, by explicit task constraint. - -## Fix round 1 - -Completed the independent-review follow-up without live Docker operations. - -- `workspace-registry` now executes a container-local, read-only Node validation of - `/data/workspace-registry/state/active.json` and every declared snapshot descriptor. It no - longer passes merely because Compose declares a volume. -- Host file permissions are checked before Docker/Compose availability and therefore remain - visible as failures when Docker is unavailable. -- Separate typed, bounded HTTP probes verify core (`curl --max-time 5`) and frontend - (`wget -T 5`) reachability, independently of Compose health. The probe is injectable in tests. -- The successful report tests assert the stable full checklist: - `descriptor`, `files`, `docker`, `compose`, `configuration`, `services`, `core-http`, - `frontend-http`, `workspace-registry`, `workflow`, `pi`. - -Additional verification: - -```bash -cd tools/tht -go test ./internal/doctor -run 'TestRun(ChecksUnsafeFilesEvenWhenDockerIsUnavailable|FailsAnInvalidContainerLocalRegistryState|ReportsEachHTTPReachabilityProbeFailure|UsesOnlyContainerLocalWorkflowAndPiDiagnosticsWhenCoreRuns)' -count=1 -v -go test ./internal/doctor ./internal/setup ./internal/service ./cmd/tht -count=1 -go test ./... -count=1 -git diff --check -``` - -All passed with fake runners/probes only. No live container, HTTP endpoint, or host installation -was touched. diff --git a/.superpowers/sdd/2026-08-16-thothii-authentication/task-15-report.md b/.superpowers/sdd/2026-08-16-thothii-authentication/task-15-report.md deleted file mode 100644 index 6b173538..00000000 --- a/.superpowers/sdd/2026-08-16-thothii-authentication/task-15-report.md +++ /dev/null @@ -1,163 +0,0 @@ -# Task 15 retained release-gate report — fix round 5 (sanitized) - -## Final-review fix-round-2 addendum — frozen source `2a9359071257f9b8a71d36ec2bbb25b161003f81` - -This addendum supersedes the fix-round-1 addendum for current authentication remediation status -while preserving the fix-round-5 material below as historical provenance. - -- Authentication remediation status: `PASS`. The three original remediation Important findings - remain `RESOLVED`; the fix-round-2 fully bounded lifecycle Important is `ADDRESSED`; and the - temporary Windows diagnostic-matrix Minor is `ADDRESSED`. -- Overall branch/release readiness is separately `FAIL`, with unavailable external/manual gates - `PENDING`. -- Completed exact-source workflow run `32147345625` concluded `failure` on baseline release jobs. - Its `Windows clone and Compose contract` job (`95744249248`) executed the unfiltered command - `go test ./internal/safeio ./internal/backup ./internal/authstorage -count=1`; the native step - passed all three packages: safeio `22.058s`, backup `7.161s`, authstorage `16.088s`. -- The Windows job failed only afterward in the baseline clone-contract script at - `scripts/test-windows-clone-contract.ps1:208`, where PowerShell rejects the undelimited - `$remoteYaml:` variable reference. -- `LF, Compose, docs, and TypeScript` job `95744249458` reproduced the baseline unset-`TMPDIR` - failure after unified Compose passed. Linux Docker job `95744249354` reproduced the missing-`rg` - prerequisite failure; cleanup passed and no image manifest was generated. -- The skipped Windows Docker Desktop/WSL2 job is recorded as `NOT_RUN` / `BLOCKED`, not FAIL. - Downstream commands skipped after executed baseline failures use the same classification. The - matrix contains an explicit native `windows_stagearchive_retained_capability` PASS row. -- Historical Node/auth/browser/docs PASS and harness/Ruff/Compose FAIL evidence remains bound to - its recorded source where not rerun. L2, real PSD/manual acceptance, and provider readiness - remain `PENDING`. -- Current machine-readable evidence and the requested Task 4 report are recorded in - `.artifacts/task-15/automated-gates.json` and - `.superpowers/sdd/2026-08-18-thothii-authentication-remediation/task-4-report.md`. -- The full fix-round-2 RED/GREEN and finding disposition is recorded in - `.superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-2-report.md`. -- Current automated-gates SHA-256: - `6c516db5c2064c4a4a2e5f25961b993cd4a8fe020bbbb822fbac7faa0c119599`. -- Historical unified Docker manifest SHA-256: `9c8dec4546909fd93799dbcf374bcb3a89bc46cfe0fd482472c0cbe757ddf5b6`. - -The complete sanitized Task 4 matrix and the separate remediation/release verdicts are in the -requested Task 4 report. - -- Final tested source commit: `74b062f1a737103524cbe706346cfd65f87cdfd1`. -- Historical retained source commits: fix-round-2 `fe190e7046acc173f510dddcb32f46ed142858c1`, - maintenance follow-up `4d230b87afdcd24f02264f8f937c8628b92db05a`, prior final Docker - source `e20bf33e2a00102192e5be66b178037aeca3a7b1`, and fix-round-4 streamed - archive privacy `54698e73400a54ce7c3e6c10099e14eb471ce8b9`. -- Versions: Node contract `v24.16.0`; host default Node `v25.6.1`; Go `go1.26.5`; - Pi `0.80.3`. -- Historical automated gate artifact: `.artifacts/task-15/automated-gates.json`; - SHA-256 `7d9ec93af15510605f1aa7179b26a7ee46d78122f647854300f7a9922057a63f`. -- Docker image manifest: `.artifacts/task-15/unified-docker-images.json`; - SHA-256 `9c8dec4546909fd93799dbcf374bcb3a89bc46cfe0fd482472c0cbe757ddf5b6`. - -## Fix-round-5 evidence - -- PASS, RED then GREEN: `TestCreateCanonicalNewPrivateFileUsesPinnedParentAfterAncestorSwap` - first failed because the creator had not retained its parent before creation. It now opens every - Unix ancestor once, creates the leaf with `openat(O_NOFOLLOW|O_CREAT|O_EXCL)`, applies and checks - `0600` by descriptor (`fchmod`/`fstat`), and uses `unlinkat` for creator failure cleanup. The - deterministic test moves the opened parent, replaces its lexical name with an outside symlink, - validates the archive under the moved original parent, and proves no outside archive was written. -- PASS: the Windows implementation uses NT `RootDirectory`-relative traversal for every component - after the volume root and for final file creation. The retained final parent receives only the - required child-create right (`FILE_WRITE_DATA` for a file, `FILE_APPEND_DATA` for a directory), - reparse points are rejected, and the owner-only protected DACL is installed in the same - `NtCreateFile` operation. The native-Windows test attempts the pre-create parent swap and calls - `safeio.ValidatePrivateRegular`; it is compiled but not executed on this host. -- PASS: `go test ./internal/safeio ./internal/backup -count=1`, `go test -race ./...` across - `18` packages, `go vet ./...`, and a native host `tht` CLI build. Existing StageArchive - capacity, lifecycle, rollback, streaming, and cleanup tests remain passing. -- PASS, compile-only: Windows amd64 static test/build compilation across `18` packages, including - the retained-handle Windows tests. No Windows executable was run; native execution remains - PENDING and is not inferred from compilation. -- PASS on Node `v24.16.0`: the hermetic OIDC/F1 authentication browser smoke passed all current - `8` checks in `frontend/e2e/auth.spec.ts` and `frontend/e2e/f1.spec.ts`; the runtime sentinel - leak scan passed. -- PASS: shell syntax, unified-smoke safety self-test, default Compose contract, unified Compose - contract, and Compose secret-policy contract. -- PASS: final unified Docker deployment smoke run `20260818070637-66409-30058`, bound exactly to - source `74b062f1a737103524cbe706346cfd65f87cdfd1`. It exercised maintenance-auth isolation, - restore, registry lifecycle, bad-candidate rollback, image revalidation, and task-scoped cleanup. - -## Sanitized final unified Docker output - -```text -== Build and start isolated local Compose distribution == -== Recreate offline and retain the validated registry snapshot == -== Pull a valid catalog+descriptor metadata update == -== Pull a content-only Git Evidence update == -== Reject catalog/descriptor metadata mismatch and retain the valid snapshot == -== Reject orphan descriptor directories not listed in the catalog == -== Reject the retired flat workspace layout and retain the valid snapshot == -== Inject a bad pinned Pi candidate and prove automatic rollback == -Task 13 full deployment smoke passed. -Task 13 cleanup proof: no labeled containers, volumes, networks, or images remain for 20260818070637-66409-30058. -``` - -## Sanitized Docker image identities - -- `sha256:2d7b19491c7eb8c119c3cedb390aaeb2ff5593f6fc43ab66c317565560da6d7d`; - roles `compose-runtime`, `fixture-runtime`. -- `sha256:3b6c31a5d8f8fc58fa3233391b6175bd2fbc793eebb44d5e285ecc6e02e9e687`; - role `compose-runtime`. -- `sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a`; - role `compose-runtime`. -- `sha256:75eab8c4ba42096724fdcfde8b4de0b5713d529dde32f285a1f86fdcb2c9e50c`; - role `compose-runtime`. -- `sha256:c3cbe1cc1aa588a64951ac6286e0df7b27fe2e6324b1001c619bb358770c0178`; - role `rollback-candidate`. - -For each image, the retained repository-digest component equals the listed image digest. Registry -names and credentials are deliberately omitted. - -## Complete observed matrix - -- PASS: Task 13 lifecycle carry-ins; retained-handle owner-private restore staging; provider fixture - round-one `6/6`; backend Node 24 round-one suite `75 files / 1081 tests`; frontend Node 24 - round-one suite `61 files / 444 tests`; current Node 24 authentication/F1 browser smoke `8/8`; - final-source Go race/build `18 packages`; Windows static cross-compile `18 packages`; harness - round-one suite `921 passed / 4 L2 deselected`; authentication docs round-one gate; shell/Compose - contracts; final unified Docker smoke; five-image traceability; and Docker cleanup. -- FAIL: Ruff `192` known-baseline errors; MkDocs strict `69` known-baseline warnings; existing - canonical/workspace install wording checks; existing Pi model-policy check; deployment-coupling - scan against preserved ignored private material. -- PENDING: native Windows execution because required host prerequisites are unavailable; L2 because - the configured secret layout is unavailable; real PSD/manual acceptance because no real - identity/access is available; isolated provider readiness because an unrelated host port is - occupied. - -## Final Task 15 review after fix round 5 - -The fresh Terra review verdict is **CHANGES REQUIRED**. The five-round breaker is exhausted; no -sixth implementation round was started. Two Important findings remain: - -- `StageArchive` does not retain the opaque parent/directory capability through the complete - stream and `Close` lifecycle. Staging-directory creation and final cleanup still use pathname - operations, so an ancestor swap after creation can strand the secret-bearing archive or redirect - cleanup. Deterministic StageArchive swap-and-cleanup coverage is still required on Unix and - native Windows. -- Windows claim removal closes its validated retained parent handles before calling pathname-based - `DeleteFile`. Removal must instead remain handle-relative (or delete through the opened handle), - with a native-Windows ancestor-swap test. - -The focused/full Go, cross-compile, Node 24, browser, Compose, Docker lifecycle, image-traceability, -and cleanup results above remain valid evidence for source `74b062f1a737103524cbe706346cfd65f87cdfd1`. -They do not override the final code-review verdict. Native Windows execution remains PENDING. - -The authentication feature is **not implementation-complete or release-complete** while these code -findings and the required FAIL/PENDING gates remain. No secret values, real identities, internal -endpoints, or registry names are retained. - -## Final whole-branch review - -The final read-only Terra review of `351361f..39b5453` also returned **CHANGES REQUIRED** and found -one additional Important issue: the POSIX local-user registry validates file type, link count, and -mode for `users.yaml` and its parent directory, but does not require ownership by the effective UID. -A foreign-owned `0600` registry inside a runtime-owned `0700` directory can remain writable by the -foreign owner and be used to alter credentials or grant the administrator role. The registry must -enforce effective-UID ownership on every POSIX `lstat`/`fstat` path and add foreign-owner rejection -coverage. - -No new Critical issue or load-bearing Minor issue was found. The branch is **not ready to merge**: -this ownership defect and the two retained-capability cleanup defects above require fixes and renewed -review, independently of the remaining FAIL/PENDING release gates. diff --git a/.superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-1-report.md b/.superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-1-report.md deleted file mode 100644 index 92d2a4cd..00000000 --- a/.superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-1-report.md +++ /dev/null @@ -1,162 +0,0 @@ -# Final-review fix round 1 report (sanitized) - -## Verdict - -- Base: `fa499a9bdd37011833691b0f447470d8b7e8a3a6`. -- Final frozen source: `10cd66fe6a5b484a4dc569326a228c1c5484a5d4` on - `feat/thoth-auth`. -- Authentication remediation: **PASS / ADDRESSED**. All four final-review Important findings are - resolved relative to the remediation brief. -- Terra Minor evidence corrections: **ADDRESSED**. -- Branch/release readiness: **FAIL**. The completed exact-source workflow still contains executed - baseline clone-contract, LF/Compose, and Linux Docker failures. Unavailable external/manual - gates remain **PENDING**. -- Source and evidence remain separate commits. No workflow was dispatched from the evidence-only - phase. - -## Finding disposition - -| Finding | Disposition | Evidence | -|---|---|---| -| Important 1 — exhaustive Windows cleanup | RESOLVED | Cleanup now attempts close/delete/validation operations in deterministic order and returns sanitized `ErrUnsafeFile` after aggregating failures. `TestWindowsPrivateRegularCleanupClosesAfterDeleteDispositionFailure` and `TestWindowsClaimCleanupAttemptsLaterOperationsAfterEarlierFailure` cover the non-short-circuit contract. Global no-delete sharing remains unchanged. | -| Important 2 — usable native Windows authority | RESOLVED | Owner-only descriptors use the current user SID, protected/non-defaulted DACL semantics, valid NT attributes/access masks, self-relative creation descriptors, and semantic full-control validation. Equal-or-stronger Windows fixture adaptations retain no-delete handles instead of weakening ACL/identity checks. The final native three-package gate passes. | -| Important 3 — restore-test deadlock | RESOLVED | Lifecycle-stage release observes the buffered worker outcome, uses a bounded/cancellable release, reports premature completion directly, and never waits indefinitely on `done`. `TestReleaseLifecycleStageReturnsPrematureWorkerOutcome` and the lifecycle-lock terminal-cleanup test are green. | -| Important 4 — complete native package gate | RESOLVED | Workflow and remediation plan both use the exact unfiltered command `go test ./internal/safeio ./internal/backup ./internal/authstorage -count=1`. Final logs prove all three packages executed natively. | -| Minor — non-executed gate classification | RESOLVED | Non-executed/skipped commands are `NOT_RUN` / `BLOCKED`; `FAIL` is reserved for commands that ran and failed. Historical results remain separately labelled. | -| Minor — explicit Windows StageArchive row | RESOLVED | `.artifacts/task-15/automated-gates.json` contains `windows_stagearchive_retained_capability` = PASS, bound to the final source and native backup result. | - -Additional failures exposed by the required unfiltered gate were fixed without narrowing the -workflow: Windows secret-bearing archive reservation is protected before use; StageArchive shares -one retained root capability across both staged files; claim/consume transitions serialize the -complete public validation and retained-handle operation while preserving ACL, hard-link identity, -reparse rejection, and no-delete invariants. - -## RED → GREEN record - -### Initial RED - -- Run `32122302381`: - https://github.com/mptyl/ThothII/actions/runs/32122302381 -- Source: `b31b27e5845ffd3adf311429367319beaba263c7`. -- Windows job: `95665197885`. -- Result: native `safeio`/`backup` failure, including the 10-minute restore lifecycle timeout; - `authstorage` was absent from the command. This established the RED for Important 2–4 and the - required native authority. -- Cleanup failure-injection tests added for Important 1 first exposed the short-circuit behavior - before the implementation was changed. - -### Final concurrency RED - -- Run `32140481263`: - https://github.com/mptyl/ThothII/actions/runs/32140481263 -- Source: `b48e9e9189dd0e8083db9bd0378704524e670edb`. -- Windows job: `95721724645`. -- Native results: backup PASS (`20.757s`), authstorage PASS (`104.180s`), safeio FAIL - (`63.502s`). The only failures were: - - `TestCanonicalPrivateClaimWaitsForRetainedRemoveOperation`: the concurrent claim returned - `false, unsafe file` before retained removal completed; - - `TestCanonicalPrivateClaimConsumeHasOneConcurrentWinner`: iteration 8 returned `unsafe file`. -- Diagnosis: the process mutex started below `validateClaimPaths`; a concurrent caller could fail - while reopening the retained no-delete directory before reaching the lock. - -### GREEN implementation and local gates - -The lock boundary was moved to the three public claim/read/remove APIs, covering validation, -relative operation, and handle close. The Unix implementation uses a no-op boundary and retains its -existing descriptor-relative semantics. - -Final-source local commands passed: - -```text -go test ./internal/safeio ./internal/backup ./internal/authstorage -count=1 -go test -race ./... -go vet ./... -go build -o /tmp/thothii-tht-host ./cmd/tht -GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go build -o /tmp/thothii-tht-windows.exe ./cmd/tht -GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go test -c ... ./internal/{safeio,backup,authstorage} -``` - -- Focused host package times: safeio `8.750s`, backup `8.378s`, authstorage `8.854s`. -- Race suite and vet: PASS. -- Host CLI: Mach-O arm64; Windows CLI and all three Windows test binaries: PE32+ x86-64. -- Cross-compilation remains compile-only and is not used as native proof. - -## Exact-source native certification - -- Run: `32141428407` -- URL: https://github.com/mptyl/ThothII/actions/runs/32141428407 -- Event/status/conclusion: `workflow_dispatch` / `completed` / `failure`. -- Head SHA: `10cd66fe6a5b484a4dc569326a228c1c5484a5d4` — exact final source match. -- Windows job: `Windows clone and Compose contract`, job `95724751282`: - https://github.com/mptyl/ThothII/actions/runs/32141428407/job/95724751282 -- Native step: `Run native Windows retained-capability tests` — **PASS**. -- Exact command: `go test ./internal/safeio ./internal/backup ./internal/authstorage -count=1`. -- Native package results: - - safeio PASS (`8.230s`); - - backup PASS (`5.195s`); - - authstorage PASS (`8.383s`). -- Job conclusion: `failure` only because the following `Verify Windows clone contract` baseline - step failed with a PowerShell `ParserError` at - `scripts/test-windows-clone-contract.ps1:208`; `$remoteYaml:` is not delimited before `:`. - -## Remaining branch/release blockers - -| Gate | Classification | Exact outcome | -|---|---|---| -| Windows native authentication packages | PASS | All three required packages executed on final source. | -| Windows clone contract | FAIL / baseline | Executed after native PASS; PowerShell parser error at line 208. | -| LF, Compose, docs, and TypeScript | FAIL / baseline CI contract | Job `95724751205`; unified Compose passed, then `test-no-deployment-coupling-scope.sh` failed because `TMPDIR` was unset. Downstream skipped commands are `NOT_RUN` / `BLOCKED`. | -| Linux Docker deployment and rollback | FAIL / infrastructure prerequisite | Job `95724751356`; executed smoke stopped because `rg` was unavailable. Cleanup proof passed; no new image manifest was generated. | -| Native Windows Docker Desktop/WSL2 startup | NOT_RUN / BLOCKED | Job `95724752028` was skipped by workflow conditions; no Docker/WSL2 command executed. | -| Harness/Ruff/other historical baseline gates | FAIL | Retained with their recorded source and results; not rewritten as final-source proof. | -| L2, real PSD/manual acceptance, provider readiness | PENDING | Required secrets, identity/access, or provider prerequisites remain unavailable. | - -The historical Docker image manifest remains bound to source -`74b062f1a737103524cbe706346cfd65f87cdfd1`; it was not reused as proof for the final source. - -## Principal source commits - -- `cd5f505` — exhaustive cleanup, Windows authority foundation, restore deadlock tests/fix, and - complete workflow/plan package command. -- `a0e05ad` through `b6396e6` — effective full-control DACL semantics, valid NT attributes/access, - self-relative descriptors, retained no-delete fixture ordering, and Windows installation fixture - protection. -- `824245d` — preserve existing lifecycle ACL trees instead of mutating inherited authority. -- `455fffb`, `2d1670e`, `c01482c`, `9fc1a15` — concurrent claim/consume and settled-loss handling. -- `6474118` — one retained StageArchive root capability shared across staged files. -- `feee4ee` — unified Windows path wrappers on the retained primitive. -- `b261dd4` — bounded private-root sharing contention handling. -- `b48e9e9` — deterministic retained-remove concurrency regression and claim-operation lock. -- `10cd66f` — final lock boundary includes public path validation; frozen source. - -## Files changed - -Source changes relative to the fix-round base: - -- `.github/workflows/deployment.yml`; -- `docs/superpowers/plans/2026-08-18-thothii-authentication-remediation.md`; -- `tools/tht/internal/authstorage/storage_test.go`; -- `tools/tht/internal/backup/{create.go,create_test.go,fixture_security_unix_test.go,fixture_security_windows_test.go,preflight.go,preflight_test.go,preflight_windows_test.go,restore.go,restore_test.go}`; -- `tools/tht/internal/safeio/{claim_unix.go,claim_windows.go,claim_windows_test.go,files.go,files_test.go,private_root_windows.go,private_windows.go,private_windows_test.go}`. - -Evidence/status changes are restricted to: - -- `.artifacts/task-15/automated-gates.json`; -- `.superpowers/sdd/2026-08-18-thothii-authentication-remediation/task-4-report.md`; -- `.superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-1-report.md`; -- `.superpowers/sdd/2026-08-16-thothii-authentication/task-15-report.md`; -- `PROJECT_STATE.md`. - -Machine-readable evidence SHA-256: -`5c110b7b2607693de078def441b10290c5a29024c83b7e5a0ced894b72b7507f`. - -## Git and protection status - -- The evidence commit contains only the five evidence/status files listed above; no source is - changed after frozen source `10cd66fe6a5b484a4dc569326a228c1c5484a5d4`. -- After the evidence commit and push, the intended status is synchronized - `feat/thoth-auth...origin/feat/thoth-auth` with only protected untracked `.playwright-cli/` and - `.thothctl/`. -- `AGENTS.md`, `CLAUDE.md`, and `docs/agents/` are untouched. No generated `tools/tht/tht` exists. -- Evidence commit SHA is reported externally after commit creation because a commit cannot contain - its own final hash. diff --git a/.superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-2-report.md b/.superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-2-report.md deleted file mode 100644 index 77f2fbbd..00000000 --- a/.superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-2-report.md +++ /dev/null @@ -1,133 +0,0 @@ -# Final-review fix round 2 report (sanitized) - -## Verdict - -- Base evidence head: `0f762ad6b67675356389cc546421a1c46ad5a736`. -- Frozen source: `2a9359071257f9b8a71d36ec2bbb25b161003f81` on `feat/thoth-auth`. -- Authentication remediation: **PASS**. -- Three original remediation Important findings: **RESOLVED**. -- Fix-round-2 bounded lifecycle Important: **ADDRESSED**. -- Fix-round-2 temporary Windows diagnostics Minor: **ADDRESSED**. -- Release readiness: **FAIL** for executed unrelated baseline gates, with unavailable - external/manual gates separately **PENDING**. -- Source and evidence are separate commits. The evidence-only phase changed no source or tests and - dispatched no workflow. - -## Finding disposition - -| Finding | Disposition | Evidence | -|---|---|---| -| Original Important — POSIX local-registry ownership | RESOLVED | Effective-UID ownership enforcement and its Node 24 coverage remain green at their recorded source. Fix round 2 did not alter this boundary. | -| Original Important — retained-capability StageArchive lifecycle | RESOLVED | Native Windows `internal/backup` passed on the exact source, preserving the retained-root staging and cleanup coverage. | -| Original Important — handle-relative Windows claim removal | RESOLVED | Native Windows `internal/safeio` and `internal/authstorage` passed on the exact source, including retained claim/consume coverage. | -| Fix-round-2 Important — fully bounded restore lifecycle test | ADDRESSED | Gate publication and release are context-aware; stage, outcome, admission, checkpoint, and verification waits are bounded; aborts cancel, safely release, bounded-join, then assert lock-free. The deterministic withheld-gate test proves prompt timeout/cancellation, worker join, and eventual lock release. | -| Fix-round-2 Minor — temporary Windows diagnostic matrix | ADDRESSED | `windowsRelativeOpenMatrix` and its diagnostic-only call/import were removed. Owner-only DACL shape, NT access normalization, full-control, cleanup, and retained no-delete tests remain. | - -The round-1 restore lifecycle finding was broadened by the scoped round-2 review: bounded release -alone was insufficient while stage publication, gate waits, and nearby outcome/admission waits -could still outlive a controller abort. The round-2 implementation closes that broader test -orchestration gap without changing production authentication semantics. - -## RED → GREEN record - -### RED - -The deterministic withheld-gate regression was introduced first and run without relying on a -global ten-minute package timeout: - -```text -go test ./internal/backup -run '^TestRestoreLifecycleCancellationJoinsWithWithheldGate$' -count=1 -``` - -It failed in approximately `0.64s` with: - -```text -cancelled restore worker did not join within the bounded deadline -``` - -This proved that cancellation did not yet unblock and join a worker retained at the lifecycle -gate. - -### GREEN and refactor - -- The gate uses a cancellation source shared by controller and worker. Both publication and - release are `select`-based and cancellation-aware. -- Shared bounded helpers cover stage, outcome, error, signal, release, and admission waits. -- Abort cleanup is ordered: cancel, cancel the controller gate when distinct, safely release a - pending gate, bounded-join the worker, then prove the lifecycle lock is free. -- Premature worker outcomes retain and surface their original error. -- The existing success, recovery, maintenance-barrier, stale-checkpoint, and verification - assertions remain active. - -Final local gates on the frozen source: - -```text -go test ./internal/backup -run '^(TestRestoreLifecycleCancellationJoinsWithWithheldGate|TestReleaseLifecycleStage|TestRestoreLifecycleLockExcludesCompetingTransactionsUntilTerminalCleanup|TestRestoreCannotApplyAStaleCheckpointOverAnInterleavedRestore|TestRestoreKeepsAdmissionBarrierActiveUntilVerificationCommits)$' -count=1 -go test ./internal/safeio ./internal/backup ./internal/authstorage -count=1 -go test ./... -count=1 -go test -race ./... -go vet ./... -go build -o /tmp/thothii-tht-host-fix-round-2 ./cmd/tht -GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go test -c ./internal/safeio -o /tmp/tht-safeio-fix-round-2-windows.test.exe -GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go test -c ./internal/backup -o /tmp/tht-backup-fix-round-2-windows.test.exe -GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go test -c ./internal/authstorage -o /tmp/tht-authstorage-fix-round-2-windows.test.exe -GOOS=windows GOARCH=amd64 CGO_ENABLED=0 go build -o /tmp/thothii-tht-fix-round-2-windows.exe ./cmd/tht -``` - -All commands passed. The final focused lifecycle run completed in `0.672s`; the full security -package run passed safeio, backup, and authstorage; race, vet, host build, Windows test-package -cross-compiles, and Windows CLI cross-compile also passed. Cross-compilation is recorded only as -compile evidence and is not used as native authority. - -## Exact-source native certification - -- Controller-authorized run: `32147345625` — - https://github.com/mptyl/ThothII/actions/runs/32147345625. -- Event/status/conclusion: `workflow_dispatch` / `completed` / `failure`. -- Head SHA: `2a9359071257f9b8a71d36ec2bbb25b161003f81`, exactly matching the frozen source. -- Windows job: `Windows clone and Compose contract`, job `95744249248` — - https://github.com/mptyl/ThothII/actions/runs/32147345625/job/95744249248. -- Native step: `Run native Windows retained-capability tests` — **PASS**. -- Exact unfiltered command: - `go test ./internal/safeio ./internal/backup ./internal/authstorage -count=1`. -- Native package results: - - `internal/safeio` PASS (`22.058s`); - - `internal/backup` PASS (`7.161s`); - - `internal/authstorage` PASS (`16.088s`). - -The Windows job failed only in the following baseline clone-contract step. PowerShell reported a -parser error at `scripts/test-windows-clone-contract.ps1:208` because `$remoteYaml:` is not a -delimited variable reference. This later failure does not alter the successful native Go step. - -## Separate release-readiness verdict - -| Gate | Classification | Exact outcome | -|---|---|---| -| Authentication remediation | PASS | Source and exact-source native three-package authority are green. | -| Windows clone contract | FAIL / baseline | Job `95744249248`; parser error at `scripts/test-windows-clone-contract.ps1:208`, after native PASS. | -| LF, Compose, docs, and TypeScript | FAIL / baseline CI contract | Job `95744249458`; unified Compose passed, then the existing unset-`TMPDIR` failure stopped the contract step. Downstream commands were skipped. | -| Linux Docker deployment and rollback | FAIL / infrastructure prerequisite | Job `95744249354`; the existing missing-`rg` prerequisite stopped the smoke before deployment. Cleanup passed and no new image manifest was generated. | -| Native Windows Docker Desktop/WSL2 startup | NOT_RUN / BLOCKED | Job `95744250450` was skipped by workflow conditions; no native Docker/WSL2 command ran. | -| L2, real PSD/manual acceptance, provider readiness | PENDING | Required secrets, identity/access, or provider prerequisites remain unavailable. | - -Executed failures remain `FAIL`; skipped commands are `NOT_RUN` / `BLOCKED`; unavailable external -gates remain `PENDING`. Therefore remediation PASS does not imply release readiness PASS. - -## Evidence and protection status - -- Machine-readable evidence: `.artifacts/task-15/automated-gates.json`; SHA-256 - `6c516db5c2064c4a4a2e5f25961b993cd4a8fe020bbbb822fbac7faa0c119599`. -- Current Task 4 report: - `.superpowers/sdd/2026-08-18-thothii-authentication-remediation/task-4-report.md`. -- Retained Task 15 report: - `.superpowers/sdd/2026-08-16-thothii-authentication/task-15-report.md`. -- Project snapshot: `PROJECT_STATE.md`. -- Historical Docker evidence remains bound to its recorded older source and is not reused as proof - for `2a9359071257f9b8a71d36ec2bbb25b161003f81`. -- `.playwright-cli/` and `.thothctl/` remain protected and untracked. No source/test file, - instruction file, workflow, or `docs/agents/` content changed in this evidence phase. -- The separate evidence commit SHA is reported after commit creation because a commit cannot - contain its own final hash. - -No credentials, tokens, internal endpoints, identities, registry names, raw environments, or -browser traces are retained in this report. diff --git a/.superpowers/sdd/2026-08-18-thothii-authentication-remediation/task-4-report.md b/.superpowers/sdd/2026-08-18-thothii-authentication-remediation/task-4-report.md deleted file mode 100644 index 6a6d3b8c..00000000 --- a/.superpowers/sdd/2026-08-18-thothii-authentication-remediation/task-4-report.md +++ /dev/null @@ -1,131 +0,0 @@ -# Task 4 authentication remediation recertification (sanitized) - -## Fix-round-2 recertification — remediation PASS - -- Exact source: `2a9359071257f9b8a71d36ec2bbb25b161003f81` on `feat/thoth-auth`. -- Authorized workflow: completed run `32147345625`, - https://github.com/mptyl/ThothII/actions/runs/32147345625, exact matching head SHA. -- Native job: `Windows clone and Compose contract`, job `95744249248`. -- Required native step: `Run native Windows retained-capability tests` — **PASS**. -- Exact unfiltered command: - `go test ./internal/safeio ./internal/backup ./internal/authstorage -count=1`. -- Package evidence: `internal/safeio` PASS (`22.058s`), `internal/backup` PASS (`7.161s`), - `internal/authstorage` PASS (`16.088s`). This includes explicit native Windows - StageArchive retained-capability and concurrent claim-consume coverage. -- The later `Verify Windows clone contract` step failed independently at - `scripts/test-windows-clone-contract.ps1:208`: PowerShell parsed `$remoteYaml:` as an invalid - variable reference. This baseline deployment-contract failure does not change the native Go - package result. -- The optional `Native Windows Docker Desktop/WSL2 startup` job was skipped by workflow - conditions. It is `NOT_RUN` / `BLOCKED`, because no Docker Desktop/WSL2 command executed. -- The workflow reached `completed` with conclusion `failure`: the native authentication step is - PASS, while the later clone-contract, LF/Compose, and Linux Docker baseline steps are FAIL. -- Existing LF/Compose job `95744249458` and Linux Docker job `95744249354` failures repeated - before downstream work. Skipped commands are `NOT_RUN` / `BLOCKED`, not executed failures. - External L2/PSD/provider gates remain `PENDING`. - -Finding disposition is explicit: the three original remediation Important findings remain -**RESOLVED**; the fix-round-2 lifecycle Important is **ADDRESSED**; and the temporary Windows -diagnostic-matrix Minor is **ADDRESSED**. Authentication remediation is **PASS**. This does not -change overall release readiness: executed baseline gates remain **FAIL**, while unavailable -external/manual gates remain **PENDING**. - -The section below is retained as historical evidence for the pre-fix frozen source. - -## Historical pre-fix result - -- Frozen source under test: `b31b27e5845ffd3adf311429367319beaba263c7` on `feat/thoth-auth`. -- Freeze check: PASS. No tracked source changed during certification. The only untracked paths - retained are `.playwright-cli/` and `.thothctl/`. -- Certification window: `2026-08-18T09:26Z` to `2026-08-18T09:48:36Z` (UTC; the start marker is - minute-precision because no earlier second-level operator timestamp was captured). -- Overall result: `FAIL` / `CHANGES_REQUIRED`. The three Important findings are not closed and - authentication is not implementation-complete or release-complete. - -## Local gate matrix - -| Gate | Result | Sanitized evidence | -|---|---|---| -| Go focused security tests | PASS | `safeio`, `backup`, and `authstorage`; 3 packages | -| Go race/vet/host build | PASS | 18 race-tested packages; vet and host CLI build exit 0 | -| Windows amd64 cross-compile | PASS | focused safeio/backup test binaries and CLI build; compile-only | -| POSIX registry ownership | PASS | Node 24 backend suite includes local-registry ownership coverage | -| Unix StageArchive retained capability | PASS | focused safeio/backup and race coverage passed on host | -| Backend Node 24 | PASS | 76 files / 1092 tests; typecheck and build passed | -| Frontend Node 24 | PASS | 61 files / 444 tests; typecheck and build passed | -| Authentication/F1 browser smoke | PASS | Node `v24.16.0`; filtered E2E 1 passed; sentinel scan passed | -| Harness pytest | FAIL | 951 passed, 1 failed, 4 skipped, 232 subtests; `test_f4_emits_column_types` could not find `workflow.yaml` from its test cwd | -| Ruff | FAIL | 192 errors; known baseline | -| Authentication docs smoke | PASS | required-term and forbidden-word checks passed | -| Shell syntax | PASS | `bash -n scripts/*.sh` | -| Default Compose contract | FAIL | required `THT_WORKSPACE_GIT_REMOTE` was unavailable | -| Unified Compose contract | FAIL | `compose.unified.yaml` is absent from the frozen source | -| Unified Docker smoke | FAIL | workflow attempted it on the frozen SHA but stopped before deployment because `rg` was unavailable; cleanup proof passed and no new image manifest was generated | -| L2 / PSD manual / provider readiness | PENDING | required external secrets, identities/access, or provider prerequisites unavailable/not reached | - -The first full backend Vitest attempt had one workspace-registry timeout. The focused test and a -fresh complete rerun passed, so the current backend result above is the fresh complete rerun. - -## Native Windows authority - -The authorized dispatch was bound to the frozen SHA: - -- Run: `32122302381` -- URL: https://github.com/mptyl/ThothII/actions/runs/32122302381 -- Head SHA: `b31b27e5845ffd3adf311429367319beaba263c7` -- Workflow conclusion: `failure` -- Job: `Windows clone and Compose contract`, job `95665197885` -- Job URL: https://github.com/mptyl/ThothII/actions/runs/32122302381/job/95665197885 -- Native step: `Run native Windows retained-capability tests` — `failure` -- Executed command: `go test ./internal/safeio ./internal/backup -count=1` -- Observed focused failures include `TestRemoveCanonicalPrivateClaimRetainsParentDuringDeletion` - and `TestRemoveCanonicalPrivateClaimPreservesOrphan`. -- The backup package timed out in - `TestRestoreLifecycleLockExcludesCompetingTransactionsUntilTerminalCleanup` after `10m0s`. -- Additional backup failures included retained-staging `unsafe file` results, Windows temporary-file - cleanup reporting that a file was still in use, and fixture cases that could not read external - secret declarations. The first two categories are remediation/security-boundary failures; the - fixture declaration failures are recorded as an accompanying CI-fixture issue. -- `internal/authstorage` was not requested by the frozen workflow step and therefore has no native - Windows execution evidence. Cross-compilation does not substitute for this gate. - -This native failure is the blocking gate. No source fix was attempted, and no later Docker smoke -was run locally after the failure. - -## Other workflow failures - -- `LF, Compose, docs, and TypeScript` (job `95665197839`) failed in - `Verify Compose and installation contracts` after the unified Compose contract itself passed. - `test-no-deployment-coupling-scope.sh` aborted on `TMPDIR: unbound variable`; this is classified - as a baseline/CI contract prerequisite, and later docs/TypeScript steps were skipped. -- `Linux Docker deployment and rollback` (job `95665197846`) failed before deployment because the - runner did not provide `rg` (`Task 13 smoke failed: rg is required`). The sanitized cleanup proof - passed and no Docker image manifest was generated. This is classified as an infrastructure - prerequisite failure, not as evidence of a remediation regression. - -## Evidence and provenance - -- Current machine-readable matrix: `.artifacts/task-15/automated-gates.json`; SHA-256 - `6c516db5c2064c4a4a2e5f25961b993cd4a8fe020bbbb822fbac7faa0c119599`. -- Current requested report: this file (SHA-256 recorded after the evidence commit if needed for - external indexing). -- Current fix-round report: - `.superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-2-report.md`. -- Historical Docker image manifest: `.artifacts/task-15/unified-docker-images.json`, unchanged - because no new immutable-source Docker smoke ran. Its retained historical SHA-256 is - `9c8dec4546909fd93799dbcf374bcb3a89bc46cfe0fd482472c0cbe757ddf5b6`, bound to historical source - `74b062f1a737103524cbe706346cfd65f87cdfd1`, not to this Task 4 candidate. -- The historical Task 15 report remains provenance for earlier source SHAs; its current addendum - records this recertification separately. - -No credentials, tokens, internal endpoints, provider identities, registry names, raw environments, -or browser traces are retained here. - -## Separate verdicts - -- Three Important findings: `CHANGES_REQUIRED`. Native Windows retained-capability authority - failed, and the frozen workflow omits the required `authstorage` package from its native command. -- Overall release readiness: `FAIL` with additional `PENDING` gates. The native Windows remediation - gate failed; the remote Docker attempt failed on a missing runner prerequisite; existing - Ruff/harness/Compose failures and external/manual prerequisites remain unresolved; and no - successful new unified Docker image evidence exists. diff --git a/.superpowers/sdd/adapter-final-fix-report.md b/.superpowers/sdd/adapter-final-fix-report.md deleted file mode 100644 index dfdd4c7e..00000000 --- a/.superpowers/sdd/adapter-final-fix-report.md +++ /dev/null @@ -1,137 +0,0 @@ -# Adapter Foundations final-review fix report - -Date: 2026-07-11 -Branch: `codex/portable-deployment` -Worktree: `/Users/mp/projects/ThothII/.worktrees/portable-deployment` -Binding findings: `.superpowers/sdd/adapter-final-review-findings.md` - -## Outcome - -All seven final-review findings are addressed as one coherent adapter-foundations change: - -1. HTTP vector reader and writer clients are independently optional. Capabilities reflect the - configured side; writer-only new and legacy configurations build successfully for targeted - writes; search without a reader raises public `VectorReadUnavailable`. -2. `VectorHealth` now reports read/write configured and reachable state independently, preserves - side-specific errors, and reports expected/observed embedding dimensions plus compatibility. - HTTP diagnostics cover read-only, write-only, both-up, and writer-down cases. Direct health - exposes its configured expected dimension without adding schema or migration work. -3. `ThothRestDwhAdapter` accepts `DatabaseIdentityConfig`, matching its resource contract. -4. Both vector adapters reject bools, floats, zero, and negative search limits using one exact - positive-integer guard. -5. Port tests explicitly cover public exports and frozen capability records. -6. A real `tht` subprocess test proves one legacy deprecation warning per config load on stderr - while JSON stdout remains parseable and uncontaminated. -7. The adapter plan and SDD progress explicitly constrain `build_vector_loader` to transitional - bulk sync and schedule its removal/migration in the local pgvector plan. Targeted memory and - solved-question writes remain on `build_vector_store(..., require_write=True)`. - -No pgvector schema or migration changes were made. - -## Files changed - -- `harness/tht/ports/vector.py` -- `harness/tht/ports/__init__.py` -- `harness/tht/adapters/vector/thoth_http.py` -- `harness/tht/adapters/vector/legacy_direct.py` -- `harness/tht/adapters/factory.py` -- `harness/tht/adapters/dwh/thoth_rest.py` -- `harness/tests/test_vector_port_contract.py` -- `harness/tests/test_adapter_factory.py` -- `harness/tests/test_config_resources.py` -- `harness/tests/test_config_legacy_compat.py` -- `harness/tests/test_adapter_command_regressions.py` -- `harness/tests/test_dwh_port_contract.py` -- `docs/superpowers/plans/2026-07-11-adapter-foundations.md` -- `.superpowers/sdd/progress.md` -- `.superpowers/sdd/adapter-final-fix-report.md` - -## TDD and verification evidence - -RED: - -```text -cd harness && .venv/bin/pytest tests/test_vector_port_contract.py \ - tests/test_adapter_factory.py tests/test_config_resources.py \ - tests/test_config_legacy_compat.py -q -``` - -Result: collection failed as expected because `VectorReadUnavailable` did not exist. After the -initial implementation, the same command exposed two expected contract/test-harness corrections: -dimension mismatch makes aggregate health unhealthy, and the installed CLI entry point is `tht` -rather than `python -m tht.cli`. - -GREEN, covering adapter/config/command regressions: - -```text -cd harness && .venv/bin/pytest tests/test_vector_port_contract.py \ - tests/test_adapter_factory.py tests/test_config_resources.py \ - tests/test_config_legacy_compat.py tests/test_adapter_command_regressions.py \ - tests/test_dwh_port_contract.py tests/test_memory_save_one.py \ - tests/test_solved_question.py tests/test_search_similar_kinds.py \ - tests/test_vector_dual_key.py -q -``` - -Result: `66 passed in 0.45s`. - -Docker availability: - -```text -docker info --format '{{.ServerVersion}}' -``` - -Result: `29.4.1` (available; command required Docker socket access). - -Full repository-default non-L2 harness suite, with Docker available for L0 tests: - -```text -cd harness && .venv/bin/pytest -q -``` - -Result: `433 passed, 5 deselected, 17 warnings in 9.14s`. The five deselections are the configured -L2/live-service tests. Warnings are existing legacy-workspace `FutureWarning` emissions. - -Scoped lint and diff hygiene: - -```text -cd harness && .venv/bin/ruff check tht/ports tht/adapters \ - tests/test_vector_port_contract.py tests/test_adapter_factory.py \ - tests/test_config_resources.py tests/test_config_legacy_compat.py \ - tests/test_adapter_command_regressions.py tests/test_dwh_port_contract.py -git diff --check -``` - -Result: `All checks passed!`; `git diff --check` produced no output. - -## Commit - -Commit subject: `fix(adapter): close final foundation review` - -The report is part of that same final commit. A Git object cannot contain its own SHA without -changing that SHA; the exact resulting commit ID is therefore recorded in the task handoff from -`git rev-parse HEAD` after creation. - -## Self-review - -- Reader/writer separation is preserved: search dereferences only `_reader`; hashes/upsert only - `_writer`; health probes each configured client independently and never substitutes one result - for the other. -- Writer failure contributes to aggregate `ok=False`, even when the reader succeeds. -- Dimension compatibility is derived only from configured embedding dimension and existing - `list_tables` metadata. Missing metadata remains `None`, not a guessed success/failure. -- The shared limit guard uses `type(limit) is int`, intentionally rejecting Python booleans and - numeric coercions before either adapter reaches its transport. -- Existing JSON/CLI behavior is preserved; the subprocess regression parses stdout as JSON and - counts exactly one deprecation marker on stderr. -- Scope remains adapter foundations. No vector DDL, schema initialization, or migration work was - introduced. - -## Concerns / follow-up - -- Write reachability uses the existing `list_tables` diagnostic on the separately authenticated - writer client. Deployments must allow that non-mutating diagnostic RPC to the writer credential; - failures are intentionally visible rather than hidden by reader success. -- Existing legacy-workspace tests emit 17 `FutureWarning`s in the full suite. This wave pins the - required production stderr behavior but does not migrate unrelated test fixtures. -- `build_vector_loader` remains transitional technical debt only for bulk sync, explicitly assigned - to `2026-07-11-local-pgvector-profile.md`. diff --git a/.superpowers/sdd/container-task-3-report.md b/.superpowers/sdd/container-task-3-report.md deleted file mode 100644 index faa90627..00000000 --- a/.superpowers/sdd/container-task-3-report.md +++ /dev/null @@ -1,206 +0,0 @@ -# Container Packaging Task 3 Report - -## Status - -Implemented the multi-stage core application image, non-root runtime, pinned Pi installation, -container entrypoint, context exclusions, and an in-image health smoke test. - -## TDD / Build Evidence - -Initial RED: - -```text -docker build -f docker/core.Dockerfile -t thothii-core:test . -ERROR: failed to build: resolve : lstat docker: no such file or directory -``` - -The first sandboxed attempt could not access the Docker socket; the authorized rerun reached the -builder and failed for the expected reason: the Dockerfile did not exist. - -GREEN build: - -```text -sh -n docker/core-entrypoint.sh docker/smoke/core-smoke.sh -docker build --progress=plain -f docker/core.Dockerfile -t thothii-core:test . -``` - -Result: shell syntax exited 0; Docker build exited 0. A final rebuild after tightening -`.dockerignore` also exited 0 and transferred only 17.60 kB of changed context (the initial clean -build transferred 1.02 MB). - -## Runtime and Entrypoints - -- Runtime user is `10001:10001` (`thoth`), never root. -- Runtime contains Node `v22.19.0` and Python `3.12.13`. Python 3.12 is intentional because the - harness declares `requires-python = ">=3.12"` and also satisfies the deployment floor of 3.11+. -- Pi is installed exactly as `@earendil-works/pi-coding-agent@0.80.3`; its build-time and runtime - version probes both reported `0.80.3`. -- `server` starts `/app/backend/dist/server.js`; `doctor` routes to `tht doctor`; `preprocess` - routes to the future-facing `tht preprocess` command; explicit `tht ...` and arbitrary CLI - arguments route to the installed `tht` binary. -- The gate extension's `typebox` runtime dependency is installed from the harness lockfile. - -## Smoke and Diagnostic Results - -```text -docker run --rm thothii-core:test doctor -config: error - configuration is invalid or unreadable -data_root: ok -``` - -Result: expected exit 1 for absent mounted workspace configuration, with no traceback and no -secret-bearing validation detail. - -```text -docker run --rm --entrypoint /app/docker/smoke/core-smoke.sh thothii-core:test -backend listening on http://127.0.0.1:8787 -v22.19.0 -Python 3.12.13 -core smoke: ok -``` - -Result: exit 0. The script asserted non-root execution, `tht --help`, `pi --version`, runtime -version floors, and `GET /health` through curl. Fastify's returned display address was loopback; -the inspected container environment is `HOST=0.0.0.0`, and the compiled server passes that value -to `app.listen`. - -```text -docker run --rm thothii-core:test tht --version -0.1.0 -``` - -Result: arbitrary `tht` entrypoint exited 0. - -An explicit runtime assertion checked UID 10001, exact Node and Pi versions, Python 3.11+, and the -absence of `/app/harness/.env` and `/app/harness/workspaces`; it exited 0. - -## Image Size and Containment Inspection - -```text -docker image inspect thothii-core:test --format '{{.Size}} {{json .Config.User}} {{json .Config.Env}}' -221419008 "10001:10001" [...runtime paths and version metadata only...] -``` - -Image size: **221,419,008 bytes** (about 211.2 MiB). - -`docker history --no-trunc thothii-core:test` was inspected. It contains only Dockerfile commands, -the pinned public package name/version, base-image metadata, and non-sensitive runtime variables; -no credentials or customer paths were found. An in-image filename scan found only -`/app/harness/.pi/settings.json` among `.env`, key/certificate, and settings-name candidates; that -tracked Pi file contains theme/startup preferences, not secrets. The build asserts `.env` and -workspace directories are absent. - -`.dockerignore` excludes VCS/agent state, all environment files except examples, package-manager -credential files, SSH/private-key and certificate formats, local virtualenvs/node_modules/caches, -backend runtime data, customer workspaces, sessions, artifacts, indexes, corpus, and deployment -mount content. - -## Self-review - -- `git diff --check` is clean. -- Entrypoint processes use `exec`, preserving container signal handling. -- Backend production dependencies are pruned; TypeScript build tools remain in the build stage. -- The writable `/data` root is owned by UID 10001; application payload remains root-owned and - read-only to the runtime user. -- CA certificates and curl are present for HTTPS integrations and health probing. -- No existing source, customer workspace, secret, or unrelated progress-ledger change is included - in the task commit. - -## Concerns - -- The `tht preprocess` command is deliberately a future-facing routing contract; its CLI group is - scheduled in the Evidence/preprocessing plan and is not implemented in the current harness. -- Python dependencies are range-resolved because the existing harness has no Python lockfile. The - Pi package, Node runtime, and package-lock-backed Node dependency sets are pinned/reproducible. -- The image was built and smoked on Docker Desktop arm64. The chosen official multi-arch base - images and Pi package are architecture-neutral at the package level, but amd64 still needs a CI - build/smoke before being advertised as verified. - -## Reproducibility Review Fix - -The original image pinned Pi's direct version in the Dockerfile but resolved its transitives at -build time, and pip resolved all harness dependencies from ranges. Both paths now consume committed -locks. - -### Lock generation - -Pi uses the minimal `docker/pi-runtime/package.json` and its committed npm v3 lock. It was generated -with: - -```text -npm install --package-lock-only --ignore-scripts --no-audit --no-fund \ - --prefix docker/pi-runtime -``` - -The package manifest specifies exact `@earendil-works/pi-coding-agent` version `0.80.3`; a lock -inspection confirmed that same resolved package version. Docker installs it with: - -```text -npm ci --omit=dev --ignore-scripts --no-audit --no-fund -``` - -The Python lock was generated directly from the harness production metadata plus one explicit, -pinned PEP 517 build-backend input—not from a host `pip freeze`: - -```text -uv pip compile harness/pyproject.toml docker/python-runtime/build-requirements.in \ - --universal \ - --python-version 3.12 \ - --no-emit-package tht \ - --generate-hashes \ - --custom-compile-command \ - 'uv pip compile harness/pyproject.toml docker/python-runtime/build-requirements.in --universal --python-version 3.12 --no-emit-package tht --generate-hashes --output-file docker/python-runtime/requirements.lock' \ - --output-file docker/python-runtime/requirements.lock -``` - -`pytest`, `ruff`, and `testcontainers` are absent. All production direct and transitive packages -are exact and hashed. `setuptools==80.9.0` is explicit so the local harness install can use -`--no-build-isolation` without an unpinned build-time resolution. Refresh instructions are in -`docker/LOCKS.md`. - -### No-cache rebuild and verification - -Final build command: - -```text -docker build --no-cache -f docker/core.Dockerfile -t thothii-core:test . -``` - -Result: exit 0. The logs showed Pi `0.80.3`, Node `v22.19.0`, a hash-enforced Python dependency -install, explicit `setuptools==80.9.0`, and a non-isolated local `tht` wheel build. No isolated -build-dependency download occurred. - -Fresh runtime checks: - -```text -docker run --rm --entrypoint /app/docker/smoke/core-smoke.sh thothii-core:test -backend listening on http://127.0.0.1:8787 -v22.19.0 -Python 3.12.13 -core smoke: ok - -docker run --rm thothii-core:test tht --version -0.1.0 - -/opt/venv/bin/pip check -No broken requirements found. -``` - -An in-container package inspection reconfirmed Pi `0.80.3`. Non-root UID, runtime version floors, -doctor's expected concise exit 1/no traceback, `/health`, and arbitrary `tht` routing all passed. - -The full filename containment scan found no `.env`, PEM, private-key, P12, or PFX file in `/app`; -`/app/harness/workspaces` remains absent. Image environment and `docker history --no-trunc` were -re-inspected and contain only public package/build commands and non-sensitive runtime metadata. - -Final locked image size: - -```text -220003986 10001:10001 -``` - -That is **220,003,986 bytes** (about 209.8 MiB), 1,415,022 bytes smaller than the original image. - -Remaining concern: the universal lock is resolved for Python 3.12 and includes hashes/markers for -all supported platforms, but only Linux arm64 has been built and smoked locally; amd64 remains a CI -verification gate. diff --git a/.superpowers/sdd/container-task-4-report.md b/.superpowers/sdd/container-task-4-report.md deleted file mode 100644 index 3ea43714..00000000 --- a/.superpowers/sdd/container-task-4-report.md +++ /dev/null @@ -1,82 +0,0 @@ -# Container Packaging Task 4 Report - -## Status - -Implemented and verified runtime-configured frontend packaging. - -## Changes - -- Added the browser runtime contract `window.__THOTHII_CONFIG__.backendBaseUrl`. -- Loaded `/config.js` before the Vite module entrypoint. -- Made runtime configuration take precedence while preserving `VITE_BACKEND_URL` and the - existing `http://localhost:8787` client default for development and tests. -- Added a multi-stage frontend image that builds with Node and serves static assets as - unprivileged UID/GID `101:101` with nginx on port 8080. -- Added startup-time `BACKEND_BASE_URL` substitution (default `/api`). -- Added `/api/` reverse proxying to `core:8787`, SPA fallback, no-cache runtime config, - and SSE-safe proxy settings (`proxy_buffering off`, `proxy_cache off`, one-hour read timeout). - -## TDD evidence - -- RED: `npx vitest run src/api/runtime-config.test.ts` failed because - `./runtime-config` did not exist. -- GREEN: targeted runtime config suite passed (3 tests after preserving the legacy client - default). - -## Verification - -- `cd frontend && npx vitest run --reporter=dot && npx tsc -b && npm run build` — exit 0 - (40 test files, 185 tests; TypeScript and Vite production build passed). -- `docker build -f docker/frontend.Dockerfile -t thothii-frontend:test .` — success. -- Image metadata reports `USER 101:101`. -- Two-container isolated-network smoke: - - `/config.js` returned `window.__THOTHII_CONFIG__ = { backendBaseUrl: "/api" };` - - `/api/health` proxied to the core image and returned `{"status":"ok"}`. - - an unknown nested route returned the SPA `index.html`. - - active nginx config contained `proxy_buffering off`, `proxy_cache off`, and - `proxy_read_timeout 1h`. - - `/config.js` returned `Cache-Control: no-store`. -- `sh -n docker/frontend-entrypoint.sh` and `git diff --check` — exit 0. - -## Secret-leakage inspection - -- `.dockerignore` excludes `.env*` (except examples), credentials/key formats, dependency - trees, build outputs, backend data, and deployment data. -- The runtime web root contained no `.env*`, `.pem`, `.key`, `.p12`, or `.pfx` files. -- Image history contained build/package instructions only; no secret build arguments or - credential values were introduced by this task. - -## Self-review / concerns - -- nginx resolves the `core` hostname at startup, matching the planned Compose service name; - standalone runs therefore need a reachable network alias named `core`. -- Existing frontend test warnings (React refs/act, MSW unmatched incidental requests, Vite - chunk-size warnings) remain; they did not fail the requested gates and are unrelated to - this task. -- `.superpowers/sdd/progress.md` was already modified by the orchestrator and was intentionally - excluded from this task's commit. - -## P1 review fixes - -Follow-up commit work addressed both review findings: - -- Runtime configuration is now produced with `jq -cn --arg`, so `BACKEND_BASE_URL` is encoded - by a real JSON serializer rather than interpolated into JavaScript by `sed`. -- The image includes `frontend-config-smoke`, which strips only the fixed assignment wrapper, - parses the remaining JSON with `jq`, requires exactly the `backendBaseUrl` key, and compares - the decoded value to the environment input. -- The hostile smoke passed with quotes, backslashes, a literal newline, ampersand, pipe, and - `"; globalThis.PWNED=true; //` in the value. A breakout would leave non-JSON trailing input - and fail parsing. -- Added `joinBackendPath`, shared by API fetch and EventSource creation. It removes duplicate - boundary slashes for relative and absolute bases while keeping empty and `/` bases rooted. - -Follow-up verification: - -- RED: six join cases failed with `joinBackendPath is not a function` before implementation. -- Targeted: runtime config, API client, and EventSource suites — 14 tests passed. -- Full frontend gate — exit 0 (40 test files, 191 tests, TypeScript, Vite build). -- Rebuilt `thothii-frontend:test` successfully. -- Hostile config image smoke — `frontend runtime config smoke: ok`. -- Rebuilt two-container smoke — default `/api` config, proxied `/api/health`, SPA fallback, - and SSE-safe nginx directives all passed. diff --git a/.superpowers/sdd/evidence-task-1-report.md b/.superpowers/sdd/evidence-task-1-report.md deleted file mode 100644 index c8c6dd18..00000000 --- a/.superpowers/sdd/evidence-task-1-report.md +++ /dev/null @@ -1,98 +0,0 @@ -# Evidence / Preprocessing Task 1 Report - -## Outcome - -Implemented the additive Evidence source port and canonical corpus records. Existing evidence, -search, vector, and session runtime code is unchanged. - -## Contract - -- `EvidenceSource` is a runtime-checkable protocol with `discover` and `acquire` operations. -- `SourceObject` and `AcquiredDocument` are frozen, reject extra fields, use independent metadata - defaults, and restrict metadata to Pydantic `JsonValue` values. -- `CanonicalDocument`, `CanonicalChunk`, and `CorpusManifest` are frozen and reject extra fields. -- Provenance includes stable source IDs, canonical URIs, fingerprints, modification time, and - content hashes. -- Pipeline versions are recorded on documents, chunks, and manifests. Manifests also carry schema - version, optional publish ID/vector generation, and paired embedding model/dimension fields. -- Credential-like metadata keys are rejected recursively. Credentials are not model fields and - therefore cannot enter serialized canonical artifacts through extras. - -## TDD evidence - -The initial focused run failed during collection because `tht.ports.evidence` and `tht.corpus` -did not exist. After implementation, the focused suite passed. - -## Verification - -- Focused models/protocol tests: 13 passed. -- Harness excluding Docker-backed L0 and the network-dependent wheel packaging test: 444 passed, - 5 deselected. -- Focused Ruff: passed. -- Full-repository Ruff remains blocked by 34 pre-existing findings outside the task files. -- An unrestricted `pytest -q` attempt reached 453 passed and 5 deselected, but reported 47 Docker - setup errors plus 4 Docker parity failures because the sandbox cannot access the Docker socket; - the wheel packaging test also failed because its isolated `uv build` needs unavailable network. - -## Concerns / follow-up - -- Pydantic's `frozen=True` prevents model field reassignment but does not recursively freeze list - and dict contents. `default_factory` prevents shared mutable defaults. Later pipeline stages should - treat these value objects as immutable and construct replacements rather than mutate collections. -- The adapter and normalization tasks should preserve the credential-free boundary by passing only - these records beyond acquisition. - -## Review hardening follow-up - -All six binding review areas were addressed in a separate TDD pass: - -- JSON metadata is recursively converted to immutable `FrozenDict`/tuple values while retaining - stable object/array JSON serialization. Manifest document and chunk collections are tuples. -- Secret-key matching now normalizes camelCase and punctuation. It rejects credential-specific - names (passwords, API keys, access/refresh tokens, client/private keys, session cookies and - authorization) recursively, while deliberate benign labels such as generic `token` and `secret` - remain valid. -- Canonical URIs require a scheme and reject userinfo or credential-bearing query parameters. -- Namespaced IDs, SHA-256 content hashes, timezone-aware UTC timestamps, embedding/vector - compatibility, unique IDs, chunk referential/provenance integrity, contiguous per-document - ordinals and pipeline-version consistency are validated. Nested Pydantic instances are always - revalidated so `model_copy(update=...)` cannot bypass a manifest boundary. -- Acquired arbitrary bytes have explicit base64 JSON encoding and validation, covered by a JSON - round-trip test. -- `EvidenceSourceError` classifies transient/retryable versus permanent failures and exposes only - recursively immutable, credential-screened JSON details. - -Follow-up verification: - -- Focused contract suite: 39 passed. -- Focused Ruff: passed. -- Harness excluding Docker-backed L0 and the network-dependent wheel packaging test: 470 passed, - 5 deselected. -- Fresh unrestricted harness attempt: 479 passed, 5 deselected; the same environmental boundary - remains (47 Docker socket setup errors, four Docker parity failures, one isolated `uv build` - network failure). - -## Final blocker follow-up - -The remaining four contract blockers were closed in a third TDD cycle: - -- `EvidenceSourceError` now always exposes the fixed public message/`args` value `evidence source - operation failed`; caller diagnostics are not retained. Category, details and args cannot be - reassigned, details remain recursively frozen and credential-screened, and an original exception - is available only when callers use standard exception chaining. -- Canonical document/chunk provenance stores only URI scheme, authority and path. Userinfo is - rejected; query strings and fragments are removed unconditionally, including AWS `X-Amz-*`, SAS - `sig`, and fragment token material. -- Binding model bases override Pydantic's unchecked `model_copy(update=...)`: merged values always - pass full field/model validation, so invalid copied records and top-level manifests fail. -- A canonical document/chunk `content_hash` must equal SHA-256 of the exact stored text encoded as - UTF-8. This establishes the normalization boundary explicitly: line-ending/frontmatter/text - normalization happens before model construction; the canonical models never rewrite content. - -Final follow-up verification: - -- Focused contract suite: 45 passed. -- Focused Ruff: passed. -- Harness excluding Docker-backed L0 and network-dependent packaging: 476 passed, 5 deselected. -- Fresh unrestricted harness attempt: 486 passed, 5 deselected, with the unchanged environmental - failures (47 Docker setup errors, four Docker parity failures, one isolated `uv build` failure). diff --git a/.superpowers/sdd/evidence-task-2-report.md b/.superpowers/sdd/evidence-task-2-report.md deleted file mode 100644 index 583ffda0..00000000 --- a/.superpowers/sdd/evidence-task-2-report.md +++ /dev/null @@ -1,65 +0,0 @@ -# Evidence Task 2 Report - -## Status - -Implemented filesystem and explicit-manifest HTTP Evidence source adapters, typed source -configuration with legacy compatibility, and factory construction. - -## Delivered behavior - -- Filesystem discovery is deterministic and rooted at a strict canonical directory. -- Symlink/path escapes are rejected before content is exposed. -- Discovery hashing and acquisition reads enforce a configurable byte limit. -- Filesystem fingerprints are content SHA-256 values; stable IDs derive from relative paths. -- HTTP accepts only explicit `http`/`https` manifest entries and keeps transport URLs private. -- HTTP provenance strips query strings/fragments, while config and adapter representations hide - signed or secret-bearing transport URLs. -- HTTP acquisition uses separate connect/read timeouts, streaming byte limits, bounded redirects, - private redirect rejection, and safe transient/permanent error classification. -- HTTP fingerprints prefer a deterministic ETag digest, then Last-Modified, then content SHA-256. -- `build_evidence_sources(cfg)` supports both typed `evidence.sources` entries and the legacy - `source_root` plus `evidence_dir` filesystem configuration. - -## TDD and verification - -- RED: focused tests initially failed during collection because the adapter package did not exist. -- GREEN: `15 passed` for filesystem, HTTP, and resource-config tests. -- Full harness: `548 passed, 5 deselected`. -- Changed-file Ruff: clean. -- Repository-wide Ruff remains non-clean due to 34 pre-existing findings in unrelated test files; - no unrelated lint files were modified. - -## Notes - -The approved `SourceObject` namespace grammar does not permit raw quoted ETags such as -`etag:"abc"`. The adapter therefore uses `etag:`: it preserves ETag-based -change identity without weakening the canonical contract or exposing validator contents. - -## Review hardening follow-up - -Four review findings were closed in a separate follow-up commit: - -- Filesystem access now anchors a persistent descriptor at the canonical root and walks each - component with `openat` semantics (`dir_fd`, `O_NOFOLLOW`, and `O_DIRECTORY`). The regular-file - check, bounded read, metadata, and hash all use the opened descriptor. Acquisition reopens by - the same path-safe mechanism and rejects a changed fingerprint. Deterministic tests swap both a - leaf and an ancestor to symlinks at open time. -- HTTP network policy defaults to public hosts only. Initial URLs and every redirect reject - userinfo, mixed public/private IPv4/IPv6 answers fail closed, and the connected peer must be a - public member of the previously validated DNS answer set before any body bytes are consumed. - Explicit `allow_private_hosts: true` is required for trusted private deployments and local tests. -- Every HTTP response is closed in a `finally` block, including redirects, status failures, - policy failures, oversized bodies, and mid-stream exceptions. -- ETag and Last-Modified values remain adapter-internal. Repeated discovery and acquisition send - conditional headers; a 304 reuses only previously verified cached bytes and identity. The LRU - content cache has an explicit byte bound (`max_cache_bytes`). Validators are not forwarded - across redirect origins. - -### Conditional cache binding correction - -The conditional cache now binds bytes and validators to both the canonical provenance key and the -exact final effective representation URL. Redirect traversal recomputes request headers per hop: -validators are sent only when that exact URL matches the cached final URL, never merely because a -redirect retains an origin. A same-origin path change therefore downloads and replaces the body. -The adapter accepts 304 only when the exact request carried a bound ETag or Last-Modified validator; -unsolicited and cross-origin 304 responses are permanent protocol errors. diff --git a/.superpowers/sdd/evidence-task-3-report.md b/.superpowers/sdd/evidence-task-3-report.md deleted file mode 100644 index db9ad616..00000000 --- a/.superpowers/sdd/evidence-task-3-report.md +++ /dev/null @@ -1,59 +0,0 @@ -# Evidence Task 3 — deterministic normalization and chunking - -## Outcome - -- Added pure `normalize(acquired, pipeline_version)` and `chunk(document, policy)` transforms. -- Normalization enforces UTF-8 (including UTF-8 BOM), a 10 MiB input ceiling, LF line endings, - NFC Unicode, safe YAML frontmatter extraction, canonical provenance URIs, and hashes the exact - canonical UTF-8 text stored on the document. -- Undecodable, unsupported-charset, oversized, and invalid-frontmatter inputs fail explicitly; - byte content is never truncated. -- Chunking uses a versioned immutable policy, paragraph/word boundaries with deterministic - character-count hard splits for long tokens, contiguous ordinals, provenance metadata, exact - per-chunk hashes, and IDs derived from document hash + ordinal + policy version. -- Empty documents produce no chunks. Non-ASCII, CRLF equivalence, repeatability, policy changes, - duplicate-content ordinal collisions, and max-character limits are covered by tests. - -## TDD evidence - -- Initial focused test run failed during collection because both transform modules were absent. -- The EOF-frontmatter edge test was separately observed failing before its implementation. -- Final focused verification: `12 passed`. - -## Verification - -- `cd harness && .venv/bin/pytest tests/test_corpus_normalize.py tests/test_corpus_chunk.py -q` - — **12 passed**. -- `cd harness && .venv/bin/pytest -q` — **573 passed, 5 deselected**. The sandboxed attempt could - not access Docker; the approved rerun with local Docker access passed. -- Targeted Ruff over all four implementation/test files — **clean**. -- Full `cd harness && .venv/bin/ruff check .` — reports **34 pre-existing errors** in unrelated - legacy tests (unused imports and existing E702 semicolon lines); none are in Task 3 files. - -## Concerns - -- The 10 MiB normalization ceiling is deliberately explicit and independent of adapter download - limits. If deployment policy needs a different ceiling, it should become a versioned pipeline - configuration before ingestion is wired. -- Character limits use Python Unicode code points (`len`), not UTF-8 bytes or tokenizer tokens; - this is recorded in the chunk-policy metadata and tested with non-ASCII content. - -## Review hardening follow-up - -- Chunk IDs now bind the canonical document identity, document content hash, ordinal, chunk hash, - and a canonical SHA-256 fingerprint of every `ChunkPolicy` field. Identical content in separate - documents and same-version policies with different limits cannot collide. -- Boundary-aware slicing now retains separators in the slices. Concatenating every chunk exactly - reconstructs the canonical document for repeated spaces, tabs, blank lines, Markdown hard - breaks, fenced code, whitespace-only input, Unicode, and overlong tokens; every slice remains - within `max_chars`. -- Frontmatter uses a bounded `SafeLoader` variant: duplicate keys, anchors/aliases, structures - deeper than 20 nodes, and documents larger than 1000 composed nodes are rejected. YAML parse, - JSON type, credential-safety, and resulting canonical-model errors attributable to frontmatter - map to `PermanentNormalizationError(reason="invalid_frontmatter")`; invalid pipeline policy - remains a programmer-facing `ValueError`. -- Follow-up TDD evidence: the expanded focused suite first reported 11 expected failures against - the prior implementation, then passed **45/45** across normalization, chunking, and manifest - invariants. -- Follow-up full verification: **586 passed, 5 deselected**. Targeted Ruff is clean. Full Ruff - continues to report the same **34 unrelated pre-existing** violations in legacy tests. diff --git a/.superpowers/sdd/evidence-task-4-report.md b/.superpowers/sdd/evidence-task-4-report.md deleted file mode 100644 index 8cfd32bb..00000000 --- a/.superpowers/sdd/evidence-task-4-report.md +++ /dev/null @@ -1,107 +0,0 @@ -# Evidence Task 4 — shared job envelope - -Status: complete - -## Delivered - -- Immutable `JobSpec`, `JobRun`, `JobReport`, per-stage state, sanitized error, and UTC - timestamp records. -- `run_job(spec, stages)` with a durable checkpoint at job start, before and after every stage, - and at terminal state. Successful stages are skipped when a prior run is resumed. -- Atomic JSON checkpoint/report replacement using a unique same-directory temporary file, - file `fsync`, atomic `os.replace`, and parent-directory `fsync`. -- Public reports contain fixed operational fields only. Workspace paths, stage return values, - exception messages, source content, credentials, and arbitrary metadata are not serialized. -- `WorkspaceJobLock` uses non-blocking kernel `flock` on a stable workspace/job-specific inode. - Locks are released by the kernel on process exit; lock files are never removed based on PID, - avoiding stale-lock and PID-reuse deletion races. Evidence and DWH use distinct lock files. -- Dry-run intent is immutable in the spec/report and exposed to every stage through `JobContext`. - -## TDD evidence - -Initial focused collection failed because `tht.jobs` did not exist. Tests then drove: - -- failure, sanitized reporting, resume, and idempotent successful-stage skipping; -- corrupt-checkpoint refusal before stage execution; -- JSON schema and path/secret/PII exclusion; -- dry-run propagation and ordered aware timestamps; -- multiprocessing exclusion, distinct Evidence/DWH jobs, traversal rejection, and recovery after - a lock-owning process crashes. - -Final focused result: - -```text -11 passed in 0.42s -``` - -## Verification - -```text -cd harness && .venv/bin/pytest -q -597 passed, 5 deselected, 17 warnings in 28.45s - -cd harness && .venv/bin/ruff check tht/jobs tests/test_job_runner.py tests/test_job_locking.py -All checks passed! -``` - -The full Ruff invocation was also run. It reports 34 pre-existing violations in unrelated legacy -tests; no Task 4 file is among them. L2 tests remain deselected by the repository configuration. - -## Operational notes - -- `fcntl.flock` intentionally targets the supported Linux/macOS deployment environments; it is not - a Windows locking implementation. -- The envelope does not publish or mutate an active corpus. Later pipeline stages must use - `JobContext.run_dir` for staging and perform their own final atomic publish only after validation. -- A dry run is an execution mode foundation: the runner exposes and records it; individual stages - remain responsible for suppressing external mutations. - -## Review hardening follow-up - -Four post-implementation findings were fixed test-first: - -1. Resume compatibility is now a canonical SHA-256 fingerprint over checkpoint schema version, - hashed workspace identity, job type, dry-run mode, explicit spec/pipeline versions, - configuration/input fingerprints, and the exact ordered explicit `stage_ids`. Any insertion, - removal, reorder, mode, identity, version, config, or input change rejects resume before a stage - executes. Omitting `resume_run_id` remains the explicit safe path for a new run. -2. Lock traversal now uses directory file descriptors with `O_DIRECTORY` and `O_NOFOLLOW`. - Lock files use `O_NOFOLLOW | O_CLOEXEC`; `fstat` requires a regular file owned by the current - UID with one link, and permissions are forced to `0600` (`0700` for private directories). - Pre-existing lock-file and lock-directory symlinks are rejected. -3. Stage failures now serialize only the fixed safe tuple `internal` / `stage_exception` / - `stage execution failed`. Neither exception class names nor messages are inspected for output; - a hostile exception-name/message regression test proves a terminal failed report is retained. -4. Job/run directory creation is no-follow, owner-checked, private, and durable. Each newly created - parent is fsynced, the run directory is fsynced before the first atomic file write, and the - existing file-fsync → replace → directory-fsync ordering has an explicit regression test. - -Follow-up verification: - -```text -focused job/lock suite: 27 passed in 0.45s -full harness suite: 613 passed, 5 deselected, 17 warnings in 29.65s -Task 4 scoped Ruff: All checks passed -``` - -Repository-wide Ruff continues to report the same 34 unrelated pre-existing legacy-test findings. - -## Final resume-integrity fix - -Resume is now read-only until the source checkpoint proves trustworthy. The runner loads the source -before allocating a new run ID or directory, validates the exact stage state/timestamp/error ledger, -rejects duplicate stage identifiers, and recomputes compatibility from every persisted compatibility -field plus the exact ordered persisted stage IDs. It first requires the stored fingerprint to match -that recomputation, then compares the trusted recomputation with the requested job fingerprint. - -Valid-JSON tampering tests cover removed, inserted/duplicated, reordered, and substituted stages; -input-field and stored-fingerprint changes; and invalid stage-state shapes. Every rejection occurs -before stage execution and asserts that the runs directory contains no orphan allocation. - -Final verification: - -```text -focused job/lock suite: 34 passed in 0.56s -full harness suite: 620 passed, 5 deselected, 17 warnings in 27.42s -Task 4 scoped Ruff: All checks passed -``` diff --git a/.superpowers/sdd/evidence-task-5-report.md b/.superpowers/sdd/evidence-task-5-report.md deleted file mode 100644 index e778b59d..00000000 --- a/.superpowers/sdd/evidence-task-5-report.md +++ /dev/null @@ -1,82 +0,0 @@ -# Evidence Task 5 report - -## Outcome - -Implemented an incremental Evidence corpus pipeline with immutable materialized generations, -generation-scoped vector records, and an fsynced atomic `ACTIVE` pointer. Runtime Evidence -artifact lookup reads the active canonical manifest and keeps a legacy source-tree fallback only -when no corpus has been published. - -The CLI is available as `tht preprocess evidence [--dry-run] [--resume RUN_ID] [--json]`. -JSON success and failure output is pristine and failure details are sanitized. - -## Safety and failure model - -- A workspace writer lock serializes preprocess writers; readers never take the lock. -- Generation directories, manifests, materialized files, locks, and `ACTIVE` reject symlink/path - escape cases and use owner-only durable writes. -- Vector records use generation-specific keys and metadata. The active manifest maps each active - document to its valid vector generation, allowing unchanged documents to retain their vectors. -- Runtime retrieval admits only active document IDs and their manifest-selected generations. - Removed documents and partial writes from failed generations are therefore unreachable. -- Embedding count and dimension checks occur before vector upsert; vector write count is checked - before staging/publish. Any failure leaves `ACTIVE` unchanged. -- Dry runs perform discovery/fingerprint planning only and never acquire, embed, write vectors, or - publish. Fully unchanged runs return the active generation without creating a replacement. -- Resume can safely retry idempotent generation-scoped upserts and publish an already staged, - compatibility-checked generation after a crash between staging and pointer replacement. - -## TDD evidence - -Initial focused collection failed because `tht.corpus.pipeline` and `tht.corpus.store` did not -exist. The implemented suite covers incremental skips, removals, model/policy rebuilds, acquire and -partial-vector failures, dry-run isolation, dimension validation, atomic reader snapshots, pointer -validation, symlink defense, and pristine CLI JSON. - -Fresh focused verification: - -```text -18 passed, 3 warnings in 0.39s -``` - -Command: - -```text -.venv/bin/pytest tests/test_corpus_pipeline.py tests/test_corpus_publish.py \ - tests/test_preprocess_cli.py tests/test_search_pack.py tests/test_session_documents.py -q -``` - -Scoped Ruff: `All checks passed!` - -Broader non-Docker/non-packaging run reached `560 passed, 5 deselected`; ten pre-existing HTTP -adapter tests could not bind localhost under the sandbox. The complete suite reached `570 passed, -5 deselected`, with the remaining failures/errors caused by denied Docker socket, localhost bind, -and offline wheel-build access. No task-focused test failed. - -## Remaining operational gate - -Live pgvector integration needs Docker or an authorized local pgvector endpoint. The compensation -strategy is logical isolation rather than destructive cleanup because the shared `VectorStore` -port intentionally exposes no delete/transaction API; unreachable failed generations can be -garbage-collected by a future maintenance job. - -## Review integration wave - -Added an enforceable `metadata_filter` vector-port contract and capability flags. Direct pgvector -places exact Evidence generation/document predicates in SQL before `LIMIT`; HTTP sends the same -filter to the RPC and deliberately does not use the legacy 404 fallback. The reader RPC script now -validates and applies that filter. Normal Evidence search and search-pack use an ACTIVE-aware -searcher that groups active documents by generation, executes complete server-filtered searches, -and merges the results. - -Added exact-generation Evidence cleanup to direct and HTTP writers plus the allowlisted writer RPC. -Pipeline failures compensate both staged filesystem state and vector writes; cleanup failures stay -sanitized and ACTIVE filtering remains the exposure boundary. Corpus-present session artifact -resolution now fails closed on corrupt/missing ACTIVE rather than falling through to source files. - -Focused review-wave verification: 45 passed, scoped Ruff clean. A mocked REST regression proves -the exact filter payload and fail-closed legacy 404 behavior. - -Still outstanding from the expanded review request: Task-4 JobRunner stage-by-stage integration, -published-generation retention/garbage collection, same-fd `dirfd` materialized-file reads, and -live local pgvector integration could not be completed in this wave. diff --git a/.superpowers/sdd/evidence-task-5b-report.md b/.superpowers/sdd/evidence-task-5b-report.md deleted file mode 100644 index ad0fcfdf..00000000 --- a/.superpowers/sdd/evidence-task-5b-report.md +++ /dev/null @@ -1,94 +0,0 @@ -# Evidence Task 5B implementation report - -## Status - -Integrated Evidence preprocessing with the Task 4 `JobRunner`. The CLI now accepts only a -32-character JobRunner run ID for `--resume`; generation IDs remain outputs. Runs persist the -exact ordered stages `discover`, `acquire_normalize_chunk`, `embed`, `vector_upsert`, -`stage_validate`, `publish`, and `retention_cleanup`. - -Successful-stage artifacts are copied into the new resume run before execution, allowing later -stages to continue without rediscovery, acquisition, normalization, chunking, or embedding. -Job compatibility includes workspace, configuration, discovered-input, pipeline, embedding, and -chunk-policy fingerprints. Generation-specific filesystem/vector compensation is retained, and a -compensated generation is rotated before retry. `ACTIVE` is mutated only by `publish`. - -Dry-run executes discovery/planning and makes every side-effecting stage a no-op. JSON output is -pristine and includes the JobRunner `run_id`, `resumed_from`, generation, plan, and publish status. - -## TDD evidence - -- RED: run-ID rejection and resume-artifact tests failed because generation IDs reached - configuration and resume runs had empty artifact directories. -- GREEN: the two regression tests passed after strict CLI validation and durable artifact carryover. -- Added pipeline job-plan and dry-run counting-fake coverage; both passed. - -## Fresh verification - -- Focused integration/search suite: `62 passed, 4 warnings`. -- Available harness suite excluding sandbox-blocked Docker, loopback HTTP-server, and networked - wheel-build tests: `559 passed, 5 deselected, 18 warnings`. -- Scoped Ruff: `All checks passed!`. -- `git diff --check`: clean. - -## Environment limitations and concerns - -The literal full harness invocation cannot complete in the managed sandbox: Docker socket access, -loopback HTTP test servers, and the `uv build` dependency resolution path are denied. It reached -`575 passed, 5 deselected` before those environment errors. The available-suite rerun above is -green. - -One pre-existing Pydantic serialization warning is exposed by the new end-to-end job test when -canonical metadata contains frozen tuple values; it does not contaminate CLI stdout. Retention is -an explicit stable no-op until a retention policy is configured. - -## Review fix wave — crash consistency and artifact integrity - -Addressed all five follow-up findings: - -- `JobRunner` now supports a test-only post-call/pre-checkpoint fault hook. Each stage seals a - canonical artifact manifest containing required flat filenames, SHA-256, byte size, producer - stage, and the full spec compatibility fingerprint. Resume validates the checkpoint and every - sealed artifact before allocating/copying a new run, rejecting missing, tampered, extra, nested, - or symlinked state. A sealed `running` stage is promoted after a simulated process crash; a - sealed `failed` stage is deliberately retried. -- Vector intent (exact record IDs and content hashes) is sealed before upsert. Execution reconciles - `existing_hashes` and writes only missing/mismatched rows. Crash-after-effect tests prove no - duplicate acquire, embed, or vector upsert. -- Raw upsert, stage, recovery-upsert, recovery-stage, and publish exceptions compensate the exact - generation. Compensation markers survive failed checkpoints; resume rotates the generation, - refreshes generation-bound artifacts, reconciles vectors, and stages idempotently. -- `CorpusStore.publish` is idempotent and failure-atomic. If replace succeeds but directory fsync - fails, it restores the previous `ACTIVE` value (or removes a newly created pointer), fsyncs the - rollback, and re-raises. Pipeline cleanup refuses to discard a generation referenced by ACTIVE. -- Added crash/resume coverage after all seven ordered stages; corrupt/missing plan, manifest, and - embeddings; unsafe extra paths; nonexistent run IDs; raw vector/stage failures; and post-replace - ACTIVE rollback. - -Fresh fix-wave verification: - -- Focused jobs/corpus/CLI/search suite: `82 passed, 17 warnings`. -- Available harness suite (same sandbox exclusions described above): - `579 passed, 5 deselected, 31 warnings`. -- Scoped Ruff and `git diff --check`: clean. - -## Final P1 fix — effect state and checkpoint-bound manifest roots - -- Stage checkpoints now distinguish `intent` from `completed`. Vector intent is atomically sealed - and checkpointed before upsert. A process-level `BaseException` after a partial multi-record - write leaves the stage `running/intent`; resume never promotes it and instead reconciles - `existing_hashes`, writing only the missing records. The completed state is persisted only after - reconciliation returns successfully. -- Every stage now persists its completed artifact state while still `running`, before the - post-call fault hook. The checkpoint binds the SHA-256 of canonical `artifact-manifest.json`, - effect state, exact producer stage, and exact required-file mapping. Resume validates this root - and all bindings before promotion or copying. -- Added process-interruption coverage proving the already-written vector record is not submitted - twice, remaining records are written, and publish completes only after reconciliation. Added - coordinated artifact/manifest, spec-binding, and producer-binding tamper rejection tests. - -Fresh verification: - -- Focused jobs/corpus/CLI/search suite: `86 passed, 18 warnings`. -- Available broad harness suite: `583 passed, 5 deselected, 32 warnings`. -- Scoped Ruff and `git diff --check`: clean. diff --git a/.superpowers/sdd/evidence-task-5c-report.md b/.superpowers/sdd/evidence-task-5c-report.md deleted file mode 100644 index 556bd02b..00000000 --- a/.superpowers/sdd/evidence-task-5c-report.md +++ /dev/null @@ -1,188 +0,0 @@ -# Evidence Task 5C report - -## Delivered - -- Added `vector.retain_published_generations` (default `3`, validation minimum `1`). -- Retention runs only after publication. It keeps ACTIVE, the newest configured generations, - and generations referenced by running or resumable failed job checkpoints. -- Cleanup deletes the exact Evidence generation from the vector store before removing its - immutable filesystem directory. Vector failures retain filesystem metadata for retry and - produce credential-free partial reports. -- Added idempotent `tht preprocess evidence gc [--dry-run] --json` reconciliation with pristine - JSON output. -- Materialized document reads now open generation/documents components with directory file - descriptors and `O_NOFOLLOW`, require a regular file owned by the process with one link, and - hash the bytes read from the same descriptor against the canonical manifest. -- HTTP generation deletion is pinned to `delete_vector_generation` with exact - table/kind/generation arguments. Legacy 404 responses fail closed with an actionable, - sanitized migration message. - -## Evidence - -- Focused retention, safe-read, CLI, and HTTP contract tests: `51 passed` (Docker-backed direct - parametrizations excluded from that focused invocation). -- Real Docker pgvector adapter suites: `33 passed`. -- Full harness suite, including Docker-backed tests: `668 passed, 5 deselected`. -- Changed-file Ruff: clean. -- `git diff --check`: clean. - -The five deselected tests are the repository's opt-in `l2` tests requiring external services; -they are not local pgvector tests. Test output retains pre-existing Pydantic serialization and -legacy-config deprecation warnings. - -## Review fix wave - -- Publication is now explicit and durable (`PUBLISHED` marker). Retention candidates require a - valid generation manifest and publication marker (ACTIVE remains backward-compatible), so - staged and malformed directories neither consume retention slots nor become deletion targets. -- The policy retains ACTIVE plus exactly `N-1` newest rollback publications, ordered by durable - publication time and generation id. Running and failed-resumable JobRunner checkpoints protect - every referenced plan generation. -- `VectorStore` now exposes exact Evidence generation inventory. Direct pgvector uses a constrained - `SELECT DISTINCT` over `kind='evidence'` and `metadata.vector_generation`; HTTP uses the - allowlisted `list_evidence_generations` RPC and fails closed on legacy 404. The writer RPC SQL, - revokes, and grants are packaged in `create_vector_writer_rpc.sql`. -- Explicit GC reconciles the union of published filesystem generations and vector-only orphans, - preserving vector-before-filesystem deletion and retry semantics. -- `run_as_job` holds the same corpus writer lock across checkpoint recovery, staging, publish, and - retention. Explicit GC already uses this lock, serializing candidate snapshots with publishers. -- Session artifact consumers no longer receive the corpus source path after validation. They get - an owned, read-only copy atomically written from the bytes read and hash-validated on the same - descriptor. - -Fresh verification after the fix wave: full harness `672 passed, 5 deselected`; Docker pgvector, -HTTP parity, and migration suites `43 passed`; exact direct inventory/delete integration `1 passed`; -changed-file Ruff and `git diff --check` clean. - -## Final hardening verification - -- Canonical generation validation is exact (`^gen:[0-9a-f]{32}$`) before HTTP/direct deletion; - malformed HTTP inventory rows fail closed rather than entering the GC candidate set. -- Added explicit protection coverage for running and failed-resumable JobRunner checkpoints, plus - a second-GC idempotence assertion for vector-only orphan reconciliation. -- Added deterministic concurrent locking coverage: a job paused after discovery retains the corpus - writer lock, explicit GC blocks, then completes after publication without deleting the active run. -- Added a descriptor-race regression: replacing the corpus pathname immediately after `read(2)` - leaves the atomically materialized session-owned copy byte-for-byte equal to the validated ACTIVE - document and its manifest hash. - -Final fresh evidence: Docker pgvector/HTTP/migration suites `48 passed`; full harness `680 passed, -5 external L2 deselected`; changed-file Ruff and `git diff --check` clean. - -## Integrated Task 5 dependency fixes - -- GC now distinguishes filesystem retention from vector dependencies. ACTIVE and the newest - `N-1` published manifests keep their directories; every exact generation in their - `document_generations` maps remains vector-protected even after its old publication directory is - evicted. Job-protected manifests receive the same dependency treatment. -- The real four-publication Docker lifecycle now includes an unchanged document whose vectors come - from the first generation. With retention `N=2`, only the final two publication directories remain - while the first generation's vectors remain searchable from ACTIVE and survive restart/explicit GC. -- Evidence lookup is always wrapped by the ACTIVE-aware searcher. With no corpus/ACTIVE, Evidence - returns no rows and search packs cannot expose legacy vectors; non-Evidence kinds are unchanged. -- Session artifact resolution holds the corpus writer lock, snapshots the active manifest once, and - materializes bytes using that exact `manifest_id`, preventing a concurrent publish/retain-1 GC from - changing or deleting the selected source generation. - -Focused unit tests, the updated real Docker lifecycle, changed-file Ruff, and `git diff --check` pass. -The final full harness invocation completed with exit code 0, including the concurrently added DWH -JobRunner tests. - -## Final ACTIVE search review fixes - -- `ActiveEvidenceSearcher` now treats default (`kinds=None`) and mixed-kind searches as explicit - split queries: non-Evidence kinds are queried separately, while Evidence is queried only with - ACTIVE manifest generation/document predicates applied server-side before every limit. -- Results are merged deterministically by descending similarity then stable id and truncated once - to the caller's global `top_n`. Pure non-Evidence searches retain their original delegate path. -- The corpus writer lock now covers manifest snapshot construction and all corresponding vector - queries, preventing retain-1 publication/GC from switching or deleting generations mid-search. -- Removed the public post-LIMIT `active_evidence_hits` helper; no public Evidence path performs - client filtering after limit. - -Focused default/mixed/no-ACTIVE/search-pack tests pass, the real Docker pgvector lifecycle passes, -and the final full harness plus scoped Ruff/diff invocation completed with exit code 0. - -## Workspace-scoped Evidence isolation - -- Evidence manifests, vector metadata, and record keys now carry the stable JobRunner workspace id - derived from the configured workspace identity (config stem), never credentials or absolute paths. -- Every ACTIVE server-side predicate includes `workspace_id`. Legacy unscoped rows therefore fail - closed and cannot appear in Evidence results. -- Vector generation inventory and deletion require the workspace namespace across the port, direct - pgvector adapter, HTTP client/adapter, and allowlisted RPC SQL. Legacy unscoped RPC overloads are - explicitly dropped during migration; destructive SQL matches collection, kind, generation, and - workspace together. -- GC recovers the persisted namespace from ACTIVE for explicit/restarted cleanup and can only list - or delete that workspace's generations. Real shared-pgvector coverage proves deleting a generation - for workspace A preserves the same generation in workspace B. -- `PipelineResult.model_dump` now serializes fields explicitly instead of `dataclasses.asdict`, - avoiding deepcopy of immutable `FrozenDict` metadata while preserving pristine JSON CLI output. - -Final focused verification: `89 passed` across corpus/CLI JSON, direct/HTTP parity, migrations, and -real Docker pgvector lifecycle; scoped Ruff and `git diff --check` clean. A contemporaneous full-suite -run reached unrelated Task 6 immutable-file tamper tests; those files were deliberately not changed. - -## Immutable corpus/workspace binding - -- A corpus root becomes bound to the workspace id persisted in its ACTIVE manifest. Job, non-job, - explicit GC, and ACTIVE search entry points compare the configured namespace before discovery, - vector access, staging, deletion, or ACTIVE mutation. -- Reusing the same paths after renaming a workspace now fails closed with a typed/sanitized message: - use a new corpus root or perform an intentional explicit rebuild. Unscoped legacy manifests also - fail this ownership check. -- Tests prove unchanged-document reuse cannot silently mix workspace A vectors into a workspace B - manifest, and that mismatched job, GC, and search paths perform no vector/filesystem mutations. - -Focused workspace-binding, search-pack, preprocess JSON, and scoped Ruff/diff tests pass. - -Compatibility follow-up: direct/internal `CorpusPipeline` instances now distinguish an omitted -workspace identity from an explicit config/job identity. An unbound instance adopts the persisted -ACTIVE owner (or `default` only for a brand-new direct corpus), preserving safe resume/GC tests and -the real pgvector lifecycle. Explicit config/job identities still fail closed on any mismatch. The -two reported regressions, workspace mismatch guards, real Docker lifecycle, scoped Ruff/diff, and -the full harness suite all pass. - -Final fail-closed follow-up: persisted ACTIVE ownership is now validated under the corpus lock before -every configured search delegate, including default, mixed, pack, and non-Evidence-only operations. -Malformed or missing `metadata.workspace_id` is intrinsically rejected even for unbound direct -callers; source discovery, vector operations, GC, files, and ACTIVE remain untouched. Focused tests, -real Docker lifecycle, scoped Ruff/diff, and the full harness regression run pass. - -Final lock/preflight follow-up: `CorpusPipeline.gc()` now acquires the corpus writer lock itself for -ownership validation through vector/filesystem cleanup. The store lock is thread-reentrant so nested -job retention is safe without weakening cross-thread/process exclusion; the CLI wrapper no longer -double-locks. Search find/pack performs locked corpus ownership preflight immediately after config -load, before DWH leasing, vector/searcher factories, embeddings, or schema work. Focused concurrency -and fail-closed tests, real Docker lifecycle, scoped Ruff/diff, and the full harness pass. - -## Compact public Evidence reports - -- Public `PipelineResult.model_dump()` is now a bounded operational envelope: terminal status, - run/resume/publication/generation/manifest identifiers, capped changed/unchanged/removed source - identifiers, and aggregate document/chunk counts. Full manifests, bodies, and metadata remain - internal/on disk and are never serialized to CLI stdout. -- `tht preprocess evidence` exits `1` for any durable terminal status other than `succeeded` in - both JSON and text modes. JSON stdout remains one pristine sanitized object; text mode emits one - compact stderr error without traceback, exception identity, evidence content, or credentials. -- Tests cover a real failed acquisition job, sensitive evidence content, capped thousand-item - summaries, bounded report size, and smoke-compatible changed/unchanged fields. - -Focused tests and scoped Ruff/diff pass. The contemporaneous full suite reaches an unrelated Task 6 -DWH snapshot fixture missing its newly required workspace identity. - -### Safe result representation and exact text totals - -- `PipelineResult.manifest` is explicitly excluded from dataclass representation and the custom - representation is fixed-size operational data only. It omits manifest ids, documents, chunks, - content, metadata, and errors; `str(result)` inherits the same safe representation. -- Text-mode Evidence success output reads the uncapped aggregate totals from `payload["counts"]` - rather than the intentionally capped identifier arrays. -- Regression coverage builds a thousand-document/chunk manifest containing content and - credential-like metadata secrets, checks bounded `repr`/`str`, and verifies exact totals above - the 100-item public-array cap. - -Focused Evidence verification passes (`67 passed`), and scoped Ruff is clean. The full harness run -is not green in this sandbox: Docker-backed tests cannot access the daemon, wheel packaging cannot -use the restricted build environment, and concurrent Task 6 DWH binding changes currently fail two -DWH tests. None of those failures touch the Evidence files in this follow-up. diff --git a/.superpowers/sdd/evidence-task-5d-report.md b/.superpowers/sdd/evidence-task-5d-report.md deleted file mode 100644 index 9d87b141..00000000 --- a/.superpowers/sdd/evidence-task-5d-report.md +++ /dev/null @@ -1,50 +0,0 @@ -# Evidence Task 5D — Real pgvector lifecycle gate - -## Status - -Complete. The Docker-backed L0 gate uses one persistent `pgvector/pgvector:pg16` -database and the production migrations, direct reader/writer `PgVectorStore`, -`CorpusStore`, `CorpusPipeline.run_as_job`/JobRunner, ACTIVE Evidence retrieval, -search-pack fusion, owned session artifact copy, retention, and explicit GC. - -## Lifecycle covered - -- Four real corpus publications with retention set to two generations. -- A higher-similarity stale vector proves ACTIVE metadata filtering happens before LIMIT - for normal Evidence retrieval and the search-pack fusion path. -- A removed source is absent from ACTIVE retrieval and cannot be copied to a session. -- An injected process death occurs after one real committed vector upsert. Resume uses the - real run ID, preserves that record, fills the missing records, and produces no duplicate keys. -- Database engines and direct store objects are disposed/recreated before persisted ACTIVE - retrieval is checked again. -- An exact canonical vector-only orphan generation is discovered and removed by explicit GC. -- Filesystem and vector inventories converge exactly to ACTIVE plus one rollback; a second GC - is a no-op. -- Owned session artifact bytes and SHA-256 match the ACTIVE canonical document. - -## Production bug found and fixed - -Production migration `003_roles.sql` intentionally restricted `vector_writer`, but omitted -the privileges used by the production generation lifecycle: `SELECT(metadata)` for inventory -and `DELETE` for cleanup on `vectors.evidence`. Consequently a real job published successfully -and then failed in `retention_cleanup` on its first run. - -Added versioned migration `004_evidence_generation_gc.sql` granting only those two Evidence -generation-management privileges. Runtime application code was not redesigned. - -## Verification - -- Target lifecycle: `1 passed` (Docker-backed). -- Full harness: `681 passed, 5 deselected`. -- Scoped Ruff: passed. -- `git diff --check`: passed. - -The existing Pydantic serialization and legacy-workspace deprecation warnings remain unchanged. - -## Follow-up assertion correction - -The removal phase now retains the removed canonical document ID/ref before publication and -asserts both fields are absent from post-resume ACTIVE Evidence hits. It reruns the real -search-pack fusion after removal, proves active fourth-generation content is positively -returned in both paths, and proves the removed content remains absent. The owned session -artifact lookup for the retained removed ID remains empty. diff --git a/.superpowers/sdd/evidence-task-6-report.md b/.superpowers/sdd/evidence-task-6-report.md deleted file mode 100644 index b1d347ab..00000000 --- a/.superpowers/sdd/evidence-task-6-report.md +++ /dev/null @@ -1,49 +0,0 @@ -# Evidence Task 6 — final fd-anchored DWH correction - -All DWH generation state below `.tht-dwh` is now accessed relative to the directory descriptor -retained by the shared/exclusive generation lease. ACTIVE reads, atomic temp writes, replacement, -fsync, and rollback use `openat`/`replaceat` operations. Generation staging, validation, -reconciliation, resume checks, retention classification, and recursive deletion likewise use owned -root/generations/candidate descriptors with `O_NOFOLLOW`; locked operations no longer reopen -generation paths through `workspace_root`. - -Portable reader snapshots are copied from validated generation file descriptors into private 0700 -process-owned temporary directories while the shared lease is held. This avoids Linux-only -`/proc/self/fd` paths and prevents a renamed/replaced `.tht-dwh` pathname from redirecting later -schema or LSH reads. Lease-scoped copies are removed on exit and standalone snapshots are removed -at process exit. - -Deterministic adversarial tests rename the DWH root after lease acquisition during ACTIVE reads, -ACTIVE publication, and retention cleanup. Each test proves the replacement tree is never read, -written, or deleted; the descriptor-pinned original either completes consistently or fails closed. -Existing owner binding, legacy rejection, crash reconciliation, resume, atomic rollback, retention, -and reader/writer exclusion behavior remains covered. - -## Final review correction - -Snapshot materialization now reads the manifest and every owned artifact exactly once through the -already-open generation descriptor, validates each hash against those exact bytes, and writes the -same byte objects to the private snapshot. A deterministic second-read mutation test proves hostile -pickle bytes can neither pass validation nor enter the snapshot. Reconciliation closes the ACTIVE -generation descriptor in a `finally` block on matches, mismatches, and exceptions. Pipeline-owned -snapshot directories are removed and deregistered after `run_job` on both successful and failed -runs, preventing repeated pipeline use from accumulating temporary directories or registry entries. - -The cleanup boundary now begins immediately after snapshot materialization. Resume checkpoint -validation and `JobSpec` construction are guarded by the same release routine as `run_job`, so -corrupt/mismatched resume state or constructor failure clears the pipeline holder, removes the -private directory, and restores the snapshot registry to its prior state before propagating. - -## Shipped preprocessing startup contract - -Local-vector preprocessing now uses a dedicated Compose override. Both one-shot jobs depend on a -successfully completed `vector-migrate`, whose transitive chain waits for database health and role -reconciliation. The generic preprocessing overlay remains independently renderable and contains no -local-vector services or password secrets. README commands include the local override and build the -job image before running. - -The real clean-project smoke no longer injects dependencies or manually starts, reconciles, or -migrates PostgreSQL. Its first shipped `compose run preprocess-evidence` demonstrably creates the -database, waits for health, runs reconciliation and migration, then runs the Evidence job. Unchanged -rerun, changed-source publish, DWH preprocessing, ACTIVE verification, and injected-failure cleanup -all pass through the same shipped dependency path. diff --git a/.superpowers/sdd/evidence-task-7-report.md b/.superpowers/sdd/evidence-task-7-report.md deleted file mode 100644 index 2c217cd2..00000000 --- a/.superpowers/sdd/evidence-task-7-report.md +++ /dev/null @@ -1,93 +0,0 @@ -# Evidence preprocessing Task 7 report - -Implemented the S3-compatible Evidence adapter, explicit preprocessing Compose overlay, and -operational gates. - -- S3 discovery uses bounded paginator pages, page size, and total objects; acquisition enforces a - byte ceiling and always closes streaming bodies. -- Provenance is canonical `s3://bucket/key`. Versioned objects use `s3-version:`; - unversioned objects use a hashed exact ETag, and acquisition refuses validator drift. -- The adapter uses boto3/botocore rather than custom signing. TLS verification is enabled by - default. Custom HTTP and private endpoints require independent explicit opt-ins; endpoint - userinfo is rejected and public custom endpoints are DNS-policy checked. -- Access, secret, and session credentials support file-secret resolution into masked `SecretStr` - config fields. They are never emitted in provenance, reports, errors, or Compose environment. -- `deploy/compose.preprocess.yaml` provides separate one-shot Evidence and DWH jobs and is inert - unless explicitly included with the `preprocess` profile. -- `scripts/preprocess-smoke.sh` verifies both services render without secret material and pins an - unchanged rerun plus a modified generation through deterministic pipeline tests. - -Verification: focused S3/HTTP/filesystem/config tests 34 passed; operational smoke 2 passed; core -image with locked boto3 extra built; full harness 702 passed, 5 deselected; scoped Ruff and diff -checks passed. - -Operational risk: custom S3-compatible endpoints remain part of the deployment trust boundary. -Private endpoint access must be explicitly enabled and should be restricted by container egress -policy in production. S3 list consistency semantics are provider-defined; version IDs are preferred -over ETags wherever bucket versioning is available. - -## Review correction - -The Compose overlay now uses committed, purpose-built Evidence and DWH workspace files with -job-specific dependencies. Its services create their lock roots and mount only the vector secrets -they consume. The operational smoke is a real isolated Compose project: real pgvector migrations, -a deterministic in-project embeddings endpoint, actual Evidence CLI JSON across initial/unchanged/ -mutated runs, exact ACTIVE verification, an actual DWH introspection job, and owned cleanup. - -S3 custom endpoints now fail closed unless declared trusted; HTTP and private loopback endpoints -need additional independent opt-ins. Boto uses forced path-style addressing. Custom endpoints reject -userinfo, query, fragment, and non-root paths. Buckets use strict DNS syntax; listed keys must remain -under prefix and within the S3 byte bound; validators must be nonempty/bounded. Because -ListObjectsV2 does not provide version IDs, discovery honestly fingerprints the exact ETag and -acquisition rejects ETag drift. - -Final correction verification: S3/config focused 20 passed; full harness 721 passed, 5 deselected; -real Compose smoke and image build passed; scoped Ruff, shell syntax, and diff checks passed. - -## Final security review correction - -Literal non-global IPv4/IPv6 endpoints now require the private-endpoint opt-in without claiming DNS -pinning for hostnames. Pagination uses explicit continuation requests and never fetches page -`max_pages + 1`. IP-shaped buckets, leading-slash prefixes, empty/overlong/control-character keys, -and absent validators fail closed. Acquisition accepts only the exact stored `SourceObject` and -compares the response ETag with the stored discovery validator. The real smoke snapshots generation -directory counts after every run and has an injected-failure cleanup mode; cleanup fails if Compose -down fails or any owned container, volume, or network remains. - -The canonical smoke correction counts only root-level `corpus/gen-<32 hex>` directories. It exposed -that the durable job path still published an empty unchanged generation; the pipeline now returns -the existing ACTIVE generation without staging a directory when compatibility and all source -fingerprints are unchanged. The smoke therefore proves directory deltas `+1`, `+0`, `+1`. -Failure injection runs a real exit-97 command after resources exist and reaches the EXIT trap. -Cleanup aggregates Compose-down, residual container/volume/network, and temp-directory failures -while preserving the original failure status. S3 prefixes are validated before any client request -for leading slash, UTF-8 byte length, controls, and DEL. - -## Canonical unchanged-run correction - -The durable job now persists a deterministic source snapshot keyed by source identity. Each entry -binds canonical URI, exact source fingerprint, UTC modification time, canonical immutable metadata, -and explicit media type and size contract fields. The manifest also binds document-to-source -provenance, supplied config/input fingerprints, compatibility, embedding settings, and pipeline and -chunk-policy versions. - -An unchanged run reuses ACTIVE only when ownership, bindings, the complete snapshot, document -provenance, materialized document hashes, and every required vector ID/content hash match exactly. -Snapshot changes rebuild only the affected sources; job input/config changes publish a new manifest -while retaining valid stable vector-generation dependencies. Missing or corrupt legacy contract -metadata, documents, or vectors fails closed and rebuilds. The Compose smoke now explicitly expects -the unchanged no-op to report `published=false` while proving generation deltas `+1`, `+0`, `+1`. - -## Corrupt ACTIVE reconstruction correction - -ACTIVE reuse now reconstructs each source contract from the persisted discovery snapshot and checks -the deterministic document identity, canonical URI, source fingerprint, UTC modification time, -source metadata, applicable media type, content hash, and pipeline identity against the owned -materialized document. The persisted document-source map carries the same exact binding. - -Chunks are recomputed under the current chunk policy and must match the manifest exactly in count, -order, IDs, ordinals, content, hashes, linkage, provenance, and policy metadata. Vector health must -report the configured dimension, and every recomputed chunk must have its generation-scoped vector -ID with the exact content hash. Missing, altered, or extra chunks and corrupt document or vector -contracts therefore disable the no-op and rebuild, while a valid unchanged run still performs no -source acquisition. diff --git a/.superpowers/sdd/model-provider-credential-report.md b/.superpowers/sdd/model-provider-credential-report.md deleted file mode 100644 index 7fcdc16c..00000000 --- a/.superpowers/sdd/model-provider-credential-report.md +++ /dev/null @@ -1,16 +0,0 @@ -# Model provider credential boundary - -The backend accepts only an absolute `THT_MODEL_API_KEY_FILE` reference. `PiProcessManager` reads -and validates it afresh before each hosted-provider spawn, rejects symlinks, non-regular/hard-linked, -empty, whitespace-containing, oversized, unreadable, or permissively-mode files, and accepts Docker -0444 secrets only beneath `/run/secrets`. Failures are sanitized and occur before child creation. - -Provider names are normalized and mapped to Pi-recognized variables. The child environment removes -the generic path, deprecated `PI_PROVIDER_API_KEY`, and all unselected known provider keys before -injecting only the selected key. Values never enter argv, settings, health, or diagnostics. Local -providers remain keyless and unknown hosted providers fail closed. - -The production Compose overlay mounts `model_api_key` read-only and points the backend at its file; -the deployment render smoke proves the value is absent from rendered configuration. Entrypoint, -root README, Pi configuration guide, environment example, and secrets operator guide document the -new contract and reject the legacy generic value variable. diff --git a/.superpowers/sdd/pgvector-final-fix-report.md b/.superpowers/sdd/pgvector-final-fix-report.md deleted file mode 100644 index e33ee639..00000000 --- a/.superpowers/sdd/pgvector-final-fix-report.md +++ /dev/null @@ -1,59 +0,0 @@ -# Local pgvector whole-plan final fix report - -## Outcome - -All four binding final-review findings are closed. - -1. `PgVectorStore.health()` checks namespace `USAGE` independently for reader and writer - before inspecting vector types. Real PostgreSQL tests revoke only schema `USAGE`, prove both - health sides false and operations unavailable, then grant it back and prove recovery. -2. Direct reader/writer passwords use workspace `password_file` references. Compose mounts the - two files read-only into core and exposes only `_FILE` paths. Rendered Compose and live - `docker inspect` checks prove secret contents are absent. -3. Direct search failures map to `VectorReadUnavailable`; hash/upsert failures map to - `VectorWriteUnavailable`. Messages are fixed and sanitized, original exceptions remain chained, - and upsert rollback is preserved. -4. The shared secret policy uses Linux `stat -c` with macOS `stat -f` fallback. Host files permit - only `0600`/`0400`; Docker's read-only `0444` is accepted only beneath `/run/secrets`. Tests and - operator docs pin this exact policy. - -## TDD evidence - -The new config, mode, schema-usage, unavailable-connection, and permission regressions failed -before their implementations. The first live secret-policy run also caught GNU `stat -f` accepting -an incompatible format invocation; detection now tries the native Linux form first. The next live -run caught smoke-generated rotation fixtures at `0644`; fixtures now model the documented host -policy. - -## Verification - -- Real direct pgvector + HTTP parity: `31 passed`. -- Full harness from `harness/`: `493 passed, 5 deselected`. -- Live `local-vector` rotation, restart persistence, inspect boundary, and backup/restore: pass. -- Core image vector migration discovery/status smoke: pass. -- External and local Compose deployment security contracts: pass. -- Config/port focused suite: `26 passed`. -- Secret policy, bootstrap rotation, and backup/restore safety scripts: pass. -- Changed Python Ruff, shell syntax, and `git diff --check`: pass. - -One attempted full-harness invocation from the repository root produced a path-dependent failure -in an existing test that opens `workflow.yaml` relative to CWD. It was immediately rerun using the -documented `cd harness && .venv/bin/pytest -q` command and passed completely. - -## Operational notes - -Workspace files contain file paths, never direct passwords. Secret contents necessarily exist in -the in-process validated `DatabaseConfig` used to establish PostgreSQL connections, but are not -serialized by doctor/Compose/inspect paths. Docker Desktop file-backed secrets may appear as bind -mounts; the safe runtime exception is therefore based on the read-only service mount location -`/run/secrets`, while source files remain owner-only on the host. - -## External-profile regression follow-up - -Local pgvector is now an explicit `deploy/compose.local-vector.yaml` overlay. The base Compose and -production external override contain no direct vector password declarations, mounts, or `_FILE` -variables, so external deployments do not resolve or require local password files. A real lifecycle -gate unsets all local secret-file variables, renders external config, builds and starts core, waits -for health, and inspects the live container for absence of local direct-vector secret paths. The -local overlay retains its live inspect assertion (paths present, values absent), rotation, restart -persistence, and transactional backup/restore drill. diff --git a/.superpowers/sdd/pgvector-task-1-report.md b/.superpowers/sdd/pgvector-task-1-report.md deleted file mode 100644 index f2025c83..00000000 --- a/.superpowers/sdd/pgvector-task-1-report.md +++ /dev/null @@ -1,95 +0,0 @@ -# Local pgvector Task 1 report - -## Status - -Implemented the direct `PgVectorStore` behind the transport-neutral `VectorStore` port. -The adapter uses separate optional reader and writer database configurations, derives -capabilities from configured authority, validates strict positive search limits, filters kinds -in SQL before limiting, and merges multi-collection results by cosine similarity. - -All collection identifiers are selected from the fixed `schema_records`, `evidence`, and -`memory` allowlist and composed with `psycopg2.sql.Identifier`. Values, vectors, kinds, hashes, -and limits remain bound parameters. Collection/kind mismatches fail with `VectorStoreError`. - -Upserts preserve the canonical metadata shape, use `record_key` conflict semantics, update the -transport hash and embedding, and leave semantic metadata fields intact. Health probes reader -and writer independently and reports observed `vector(N)` dimensions against the configured -embedding dimension. - -## Configuration and factory - -`pgvector_direct` now accepts explicit optional `reader` and `writer` `DatabaseConfig` entries. -The former `connection` entry remains supported as a deprecated read-only compatibility path. -`build_vector_store(..., require_write=True)` accepts writer-only direct configurations and -fails early when no explicit writer is present. - -The transitional `build_vector_loader` bulk-sync path remains in place. It uses an explicit -direct writer when present, or the legacy `connection`; it deliberately does not treat a new -reader-only credential as writable. No production schema migration was added. - -## TDD and verification - -- RED: the new tests initially failed at collection because `PgVectorStore` did not exist. -- Docker L0 pgvector tests: `11 passed`. -- Direct + HTTP parity/factory/config focus: `51 passed`. -- Full harness: `461 passed, 5 deselected`. -- Changed-file Ruff lint: clean. -- Changed-file Ruff format check: clean. -- `git diff --check`: clean. - -The repository-wide `ruff check .` still reports 34 pre-existing test-file findings outside -Task 1; none are in changed files. The full pytest suite emits 17 existing legacy-config -deprecation warnings. - -## Scope and concerns - -- Test fixtures create only the three existing vector tables needed to exercise the adapter; - migration/versioning remains Task 2. -- The legacy single `connection` form stays read-only through the public port, matching its - previous adapter behavior, while remaining available to the explicitly documented bulk-loader - transition. - -## Review fix wave - -The Task 1 review findings were addressed in a follow-up TDD cycle: - -- Search now validates requested kinds against the global known-kind set, intersects valid kinds - with each collection, and skips unrelated collections. A direct-versus-HTTP parity test covers - the multi-collection case. -- Health requires all three allowlisted tables, an `embedding vector(N)` column on every table, - the expected dimension on every table, and the appropriate read or write table privileges for - each configured side. Empty and partial schemas return deterministic, credential-free details; - unexpected database failures expose only their exception class. -- The Docker L0 fixture now provisions separate least-privilege reader and writer roles. Tests - prove the reader cannot insert, the writer cannot execute the cosine-search SELECT, and the - adapter still routes search to the reader and upsert/hash operations to the writer. Direct - upsert uses an atomic `INSERT ... ON CONFLICT DO NOTHING` followed by `UPDATE` for an existing - key, avoiding broad SELECT authority while retaining conflict-safe hash/upsert semantics. - -Fresh verification after the fix wave: - -- Docker L0 + HTTP port/search parity: `42 passed` (earlier checkpoint); the final L0 file has - `16 passed` including the stricter raw-role search denial. -- Expanded focused adapter/config suite: `56 passed`. -- Full harness: `466 passed, 5 deselected`. -- Changed-file Ruff lint/format and `git diff --check`: clean. - -## Sequence privilege health follow-up - -Writer health now resolves the real serial/identity sequence for the `id` column of every -required collection using `pg_get_serial_sequence`. It requires `USAGE` on each resolved -sequence, which is the privilege used by the adapter's implicit `nextval`; sequence `SELECT` is -not required because no adapter operation reads sequence state. - -The Docker fixture includes a writer role with complete table/hash-column authority but no -sequence grant. Its health is deterministically unhealthy and a new-key upsert fails. Granting -only sequence `USAGE` makes health green and the same port upsert succeeds. Sequence discovery is -guarded for partial schemas so a missing `id` column produces the existing sanitized schema -diagnostic instead of a PostgreSQL error. - -Fresh verification for this follow-up: - -- Docker pgvector L0 after formatting: `17 passed`. -- Expanded focused adapter/config/parity suite: `57 passed`. -- Full harness: `467 passed, 5 deselected`. -- Changed-file Ruff lint/format and `git diff --check`: clean. diff --git a/.superpowers/sdd/pgvector-task-2-report.md b/.superpowers/sdd/pgvector-task-2-report.md deleted file mode 100644 index 6fcac409..00000000 --- a/.superpowers/sdd/pgvector-task-2-report.md +++ /dev/null @@ -1,82 +0,0 @@ -# Local pgvector Task 2 report - -## Outcome - -Implemented ordered, idempotent production migrations and the `tht vector migrate` -interface, including `tht vector migrate --status --json` with pristine JSON output. - -## Implementation - -- `001_extensions.sql` installs pgvector. -- `002_schema_tables.sql` creates `vectors.schema_records`, `vectors.evidence`, and - `vectors.memory` with the `VectorWriteRecord` columns and `vector(768)` embeddings. -- `003_roles.sql` creates passwordless `NOLOGIN` reader/writer roles. Deployments inject - credentials (or grant these roles to separately-created login roles); no production secret - is stored in the repository. -- Reader authority is schema usage plus table `SELECT`. -- Writer authority is schema usage, table `INSERT`/`UPDATE`, narrow hash-probe column `SELECT`, - and sequence `USAGE`. It has no `DELETE`, broad row `SELECT`, DDL, or ownership authority. -- The migration runner discovers ordered SQL files, records SHA-256 checksums in - `public.tht_vector_migrations`, serializes runners with a transaction-scoped advisory lock, - and applies the full pending batch in one transaction. -- Status distinguishes applied, pending, and checksum-drifted migrations. Apply refuses drift. - A failed migration rolls back both prior migrations in that batch and ledger writes. - -## TDD evidence - -RED was observed with a real `pgvector/pgvector:pg16` testcontainer: 6 failures for the missing -module, missing command, and missing schema. - -GREEN verification: - -- Focused migration + direct adapter integration: `23 passed`. -- Full harness from the documented `harness/` cwd: `473 passed, 5 deselected`. -- Targeted Ruff (`tht` plus the new L0 test): clean. -- `git diff --check`: clean. - -The new L0 coverage exercises clean install, idempotent rerun, pristine JSON status, checksum -drift, transaction rollback, exact tables/columns/dimensions, role isolation, sequence authority, -and the real `PgVectorStore.health()` plus `VectorWriteRecord` upsert path. - -## Existing repository lint baseline - -The requested full `ruff check .` was run. It reports 34 pre-existing violations in unrelated -test files (unused imports and one-line semicolon statements). None are in Task 2 files; changing -them would exceed this task's scope. The complete harness test gate is green. - -## Self-review - -No unresolved Task 2 correctness concern found. One deliberate contract choice is worth noting: -writer `INSERT` and `UPDATE` are table-level because the approved direct adapter health probe uses -`has_table_privilege` for those authorities. Least privilege is retained by withholding broad -`SELECT`, `DELETE`, DDL, ownership, and credentials. - -## Review fix wave - -The post-implementation review found four production-boundary gaps. They are fixed as follows: - -- Migration SQL now ships inside the `tht` wheel (`tht/migrations/vector`) via explicit - setuptools package-data and is discovered through `importlib.resources`, rather than relying on - a source-checkout-relative directory. -- Both status and apply reject ledger versions absent from the installed manifest, including - nonnumeric future version labels. This treats a binary/database downgrade as drift instead of - silently reporting a healthy state. -- Migration files are ordered by parsed integer version; spellings such as `2` and `02` are - rejected as duplicate versions. -- Every migration transaction pins `search_path` locally to `pg_catalog, pg_temp`; catalog calls - and the ledger are schema-qualified. pgvector is installed into the locked `vectors` schema, - tables use `vectors.vector`, and `PgVectorStore` qualifies vector casts and the cosine operator. - A hostile admin default path with a writable shadow schema cannot redirect migration objects. -- The core image build asserts CLI discovery. Image verification now starts an ephemeral pgvector - database, runs the installed image's migration command, and compares pristine apply/status JSON. - -Additional verification after the fix wave: - -- Focused migration, adapter, hostile-path, and wheel suite: `27 passed`. -- Full harness: `477 passed, 5 deselected`. -- Production core image build: passed, including build-time CLI discovery. -- Core-image apply/status smoke against `pgvector/pgvector:pg16`: passed. -- Changed production and test files: Ruff clean; `git diff --check` clean. -- Full Ruff remains at the same 34 pre-existing unrelated test-file findings documented above. - -No dependency changed, so the committed Python requirements lock did not require regeneration. diff --git a/.superpowers/sdd/pgvector-task-3-report.md b/.superpowers/sdd/pgvector-task-3-report.md deleted file mode 100644 index 32199b42..00000000 --- a/.superpowers/sdd/pgvector-task-3-report.md +++ /dev/null @@ -1,133 +0,0 @@ -# Task 3 report — optional local pgvector profile - -## Status - -Implemented and verified the `local-vector` Compose profile. - -- `vector-db` uses pgvector 0.8.5 on PostgreSQL 16, pinned to the official multi-arch - manifest digest. -- `vector_data` is a project-scoped named volume and is not shared with application data. -- database readiness gates the packaged one-shot `vector-migrate` job; core declares the - migration completion dependency while remaining usable in the pre-existing external profile. -- bootstrap, migrator, reader, and writer identities are distinct. Bootstrap and migration - credentials are supplied as Compose secrets; the application receives only reader/writer - credentials. -- `deploy/workspaces/local-vector.yaml` selects `pgvector_direct` with separate reader and - writer connections. -- the base loopback port binding, `AUTH_MODE=none`, and `THOTH_PUBLIC_EXPOSURE=false` defaults - are unchanged. - -## Red/green evidence - -The initial Compose contract did not list `vector-db`, as required by the brief. The first real -smoke then failed migration 002 because bootstrap installed the vector extension in `public`. -The bootstrap was corrected to create the `vectors` schema under the migration owner and install -the extension there. A clean-volume rerun passed. - -## Verification - -- `./scripts/local-vector-smoke.sh`: PASS - - isolated generated Compose project and credentials - - clean migration plus idempotent status rerun - - reader/writer privilege health - - one-record upsert and similarity search - - restart of both `core` and `vector-db` - - persisted search result after restart - - project-only volume cleanup -- `./scripts/test-container-deployment.sh`: PASS -- `./scripts/test-backend-url-policy.sh`: PASS -- `docker compose --profile local-vector config --quiet`: PASS -- harness: 477 passed, 5 deselected -- backend: 84 passed; TypeScript typecheck PASS -- frontend: 226 passed; TypeScript typecheck PASS -- `git diff --check`: PASS - -## Self-review / concerns - -- Compose cannot make a dependency required only under one profile. The core dependency uses - `required: false` so the established `external` profile does not activate local infrastructure; - under `local-vector`, `compose up --wait` still fails if `vector-migrate` exits nonzero, and the - smoke verifies that successful migration precedes the healthy stack. -- Reader/writer passwords are injected into core environment variables because Compose service - attributes cannot be conditional by profile. Bootstrap and migrator credentials remain - file-backed secrets and are never exposed to core. -- The smoke intentionally refuses the operator project name `thothii` and removes only its unique - project namespace and volumes. - -## Follow-up hardening — credential reconciliation and cleanup ownership - -Review findings were resolved in a separate follow-up: - -- Replaced fresh-volume-only initialization with `vector-reconcile`, an idempotent one-shot that - runs after database health and before `vector-migrate`. It authenticates with only the bootstrap - admin secret, safely creates missing identities, reconciles role attributes and passwords on - existing volumes, restores memberships/ownership, and leaves vector data untouched. -- The migrator is explicitly `NOSUPERUSER NOCREATEDB NOCREATEROLE`. Schema/database ownership is - sufficient for all packaged migrations because reconciliation creates the two group roles first. -- The live smoke rotates migrator, reader, and writer secrets on the same populated volume, rejects - the old reader credential, reruns migrations, recreates core with the new runtime credentials, - and retrieves the record written before rotation and again after database/core restart. -- Smoke project names are no longer caller-controlled. Each run creates a unique namespace and - ownership token. Containers, networks, and volumes carry the ownership label; preflight refuses - any collision and cleanup verifies every discovered resource before `down --volumes`. -- Added a dynamic fake-Docker contract suite for caller override, collision, and mismatched cleanup - labels, plus a real-Docker collision probe using a unique labeled volume. - -Follow-up verification: - -- `./scripts/local-vector-smoke.sh`: PASS, including live secret rotation and persisted retrieval -- `./scripts/test-local-vector-smoke-safety.sh`: PASS -- `./scripts/test-local-vector-smoke-live-collision.sh`: PASS -- harness: 477 passed, 5 deselected -- backend: 84 passed; TypeScript typecheck PASS -- frontend: 226 passed; TypeScript typecheck PASS -- Compose security, backend URL, config, shell syntax, and diff checks: PASS - -Remaining operational constraint: the bootstrap admin secret must continue to match the PostgreSQL -bootstrap account stored in the volume. Runtime migrator/reader/writer rotation is supported without -data deletion; bootstrap-account password rotation is a distinct database-administration operation. - -## Final hardening — bootstrap account rotation - -The remaining operational constraint is now covered by -`scripts/vector-rotate-bootstrap-password.sh OLD_SECRET_FILE NEW_SECRET_FILE`: - -- It does not rely on `POSTGRES_PASSWORD_FILE` after initialization. -- It pre-stages the deployment-file replacement in the same directory, authenticates to the live - database with the explicit old file, and changes only the authenticated bootstrap role. -- Passwords are passed as connection parameters and rendered with psycopg2 SQL composition, so - shell and SQL metacharacters are not interpolated. -- A second connection must authenticate with the new password before the command succeeds. If that - verification fails, the still-open old connection restores the old database password. -- Only after verified database login does an atomic rename replace the current deployment secret. - Wrong-old authentication and verification failures leave deployment configuration unchanged. - -Final live smoke evidence on one existing `vector_data` volume: - -- wrong-old bootstrap rotation rejected; current deployment secret unchanged -- bootstrap password with quote characters rotated successfully -- old bootstrap login rejected and new login accepted -- `vector-reconcile`, packaged migrations, and core health passed afterward -- the vector record written before rotation remained searchable after rotation and after a further - database/core restart - -Final tests: - -- `./scripts/test-vector-bootstrap-rotation.sh`: PASS -- `./scripts/local-vector-smoke.sh`: PASS with negative and positive live bootstrap rotation -- existing local-vector collision/safety and Compose deployment contracts: PASS - -## Final identity and secret-policy alignment - -- `THT_VECTOR_BOOTSTRAP_USER` is now passed through core as well as vector-db and reconciliation, - so the rotation helper uses the authoritative configured role instead of defaulting to `postgres`. -- Rotation and reconciliation source the same raw-file `secret-policy.sh`: non-empty and no - whitespace, including trailing newlines. Rotation validates both files before Docker, - PostgreSQL, or atomic replacement staging; `test-vector-secret-policy.sh` pins empty, newline, - internal-space, and valid metacharacter cases. -- Fake-Docker tests prove a non-default identity reaches the helper path and whitespace rejection - performs no Docker call and creates no staged replacement. -- The real smoke runs the entire stack as `thoth_bootstrap_smoke`. Its whitespace-negative case - leaves the deployment file unchanged and proves the existing database login still succeeds; - non-default-account bootstrap rotation, reconciliation, migration, core health, restart, and - persisted retrieval all pass. diff --git a/.superpowers/sdd/pgvector-task-4-report.md b/.superpowers/sdd/pgvector-task-4-report.md deleted file mode 100644 index 38fe1607..00000000 --- a/.superpowers/sdd/pgvector-task-4-report.md +++ /dev/null @@ -1,94 +0,0 @@ -# Local pgvector Task 4 report - -## Outcome - -Implemented adapter parity gates and an operator-safe custom-format backup/restore workflow. - -- Direct and HTTP stores now share validation, configured-dimension rejection, and deterministic - similarity ordering with record ID as the tie-break. -- The parity fixture exercises identical records through real pgvector and the HTTP RPC contract: - kind filtering, ordering, hashes, replacement upserts, invalid collection/kind errors, and query - plus write dimensions. -- Backup explicitly allowlists the three vector tables and migration ledger, refuses overwrite, - writes through a partial file, and uses a custom compressed archive. -- Restore requires explicit active-source and target coordinates. It compares PostgreSQL system - identifier plus database OID (robust across DNS aliases), refuses the active database, checks for - an empty target unless force is explicit, and restores with exit-on-error. -- Passwords are accepted only through validated secret files, converted to private temporary - `PGPASSFILE`s, and never placed in command arguments or success/error logs. -- Role passwords/login identities are deliberately not dumped. The target must have the approved - passwordless group roles and pgvector extension reconciled before restore; archived ACLs restore - the reader/writer grants. - -## TDD and semantic alignment - -The first parity run exposed the intended HTTP differences: it accepted unknown collections and -wrong dimensions. Direct pgvector also had no stable order for equal cosine distance. The adapters -were aligned, and the final focused real-pgvector gate passed: **25 passed**. - -The first recovery run caught an incorrect probe username before restore. The second caught an -intersection between `pg_dump --schema` and the explicit public ledger table. The third confirmed -the archive contents but caught missing target group roles. Each defect was corrected and the -complete drill was rerun from a fresh generated project. - -## Live recovery smoke - -`./scripts/local-vector-smoke.sh --backup-restore`: **PASS**. - -- generated/owned source Compose project and source `vector_data` -- distinct restore container and distinct named restore volume -- migration and role health, secret rotation, restart persistence -- real custom backup, then deliberate mutation of the active source record -- same-database identity guard evaluated before restore -- restore into the separate target only -- restored hash equals the pre-mutation backup, proving retrieval parity -- migration ledger has all three applied versions -- all three restored embedding columns report `vectors.vector(768)` -- ownership-checked cleanup; the active operator project/volume is never addressed - -## Verification - -- parity + direct adapter: 25 passed -- full harness: 485 passed, 5 deselected -- changed Python files: Ruff clean -- shell syntax: clean -- `git diff --check`: clean -- full Ruff: unchanged repository baseline of 34 unrelated pre-existing test-file violations - -## Self-review and operational constraints - -The restore account must be able to read `pg_control_system()` for the robust cluster-identity -comparison and create/restore the selected objects. This is intentionally an administrative -recovery operation, not a runtime reader/writer action. `--force-nonempty` is explicit but still -uses `pg_restore --clean --if-exists`; operators should prefer a new database/volume and validate -migration status, health, and known retrieval before endpoint cutover. - -## Post-review hardening - -All five final review findings were addressed in a follow-up commit: - -- Restore now requires a physically separate PostgreSQL cluster and refuses any equal - `system_identifier`, independent of database OID or hostname. -- `pg_restore` combines `--single-transaction` with `--exit-on-error`. The live drill creates an - existing vector sentinel, deliberately fails late during a forced restore, and proves the - original sentinel row/hash remains unchanged before performing the successful restore. -- Backup uses a mode-0600 `mktemp` in the output directory, atomically renames it, and cleans only - that owned path. A fake-command test pins symlink-clobber resistance and preserves an adversarial - legacy `.partial` symlink and its target. -- HTTP parity now traverses the real `VectorRestClient` transport boundary. It asserts RPC URL/key - and kinds payloads, legacy 404 fallback, response conversion, malformed metadata tolerance, and - canonical `VectorRestError` to `VectorStoreError` mapping. -- The restored target runs role/secret reconciliation and a real `PgVectorStore` with separate - reader/writer logins. Health, known-record search, writer upsert, hash probe, schema/table/column/ - sequence authority, and 768-dimensional compatibility are therefore verified through the - production adapter. Reconciliation now restores group-role schema `USAGE`, which table-selected - archives cannot carry. - -### Atomic no-replace backup publication - -The final publication review is also closed. The private same-directory archive is published with -an atomic hard-link create rather than rename-overwrite semantics. If any process creates the final -file or symlink after preflight but before publication, `ln` fails with `EEXIST`, the backup exits -nonzero, the concurrent destination remains byte-for-byte intact, and the trap removes only the -randomly named temporary archive owned by this invocation. The fake `pg_dump` safety test creates -that destination immediately before returning and pins the failure and cleanup behavior. diff --git a/.superpowers/sdd/predeploy-fix-report.md b/.superpowers/sdd/predeploy-fix-report.md deleted file mode 100644 index 98968698..00000000 --- a/.superpowers/sdd/predeploy-fix-report.md +++ /dev/null @@ -1,929 +0,0 @@ -# Pre-deployment Fix Wave Report - -Date: 2026-07-14 -Worktree: `/home/chirone/ThothII/.worktrees/activity-log-cte-layout` -Base: `e5366d14a6da8fb331d94be60b8929cefb1fe3e0` - -## Outcome - -All three reviewed findings are implemented in one coherent backend/frontend wave: - -1. Resume leaves the prior selection, Zustand state, document panel, and EventSource untouched - until `POST /resume` succeeds. Cold Resume changes state and reconnects only after backend - clear/rebind; already-active same-session Resume preserves the existing binding; failure is a - no-op apart from the fixed toast. -2. SSE uses monotonically increasing per-session ids, cursor-filtered replay, native and manual - reconnect cursors, id continuity across `hub.clear`, and descriptor-id pending-gate - idempotence at both backend and frontend layers. -3. Generic Pi system events and readiness errors are projected through explicit public - allowlists. Sentinel URLs, paths, tokens, stderr, commands, and extra fields do not reach HTTP - or SSE. - -No harness, workflow, persistence, model, CTE viewer, CTE card, or shared Card file changed. - -## Interfaces - -- Frontend `resumeSession(id)` now returns - `Promise<{ id: string; alreadyActive: boolean }>` via `ResumeSessionResult`. -- Backend successful Resume always returns the same shape: - - running/waiting runtime: `{ id, alreadyActive: true }` - - validated cold runtime: `{ id, alreadyActive: false }` -- `SseHub.publish(sessionId, event, data): number` returns the assigned SSE id. -- `SseHub.subscribe(sessionId, send, { afterId, pending })` calls - `send(event, data, id)` for replay/live frames with `id > afterId`. -- `GET /sessions/:id/events` accepts native `Last-Event-ID` and manual - `?lastEventId=`; when both are valid it uses the greater cursor. -- Every emitted SSE frame is `id: \nevent: \ndata: \n\n`. -- Public readiness failure is exactly: - `Session services are not ready. Check configuration and connectivity, then try again.` -- Generic Pi system events are exactly `{ type: "system_event", event }`, and `event` must be a - non-empty string. - -## Files - -Backend production: - -- `backend/src/bridge/session-bridge.ts` -- `backend/src/routes/sessions.ts` -- `backend/src/sse/sse-hub.ts` - -Backend tests: - -- `backend/test/routes-sessions.test.ts` -- `backend/test/session-bridge.test.ts` -- `backend/test/sse-hub.test.ts` -- `backend/test/sse-route.test.ts` (new) - -Frontend production/support: - -- `frontend/src/api/sessions.ts` -- `frontend/src/api/types.ts` -- `frontend/src/shell/AppShell.tsx` -- `frontend/src/store/sessionStore.ts` -- `frontend/src/stream/useSessionStream.ts` -- `frontend/src/test/fakeEventSource.ts` - -Frontend tests: - -- `frontend/src/api/sessions.test.ts` -- `frontend/src/shell/AppShell.session-mgmt.test.tsx` -- `frontend/src/store/sessionStore.test.ts` -- `frontend/src/stream/useSessionStream.test.tsx` - -## TDD RED/GREEN evidence - -### 1. Backend Resume result and client-boundary allowlists - -RED command: - -```text -cd backend && npx vitest run test/routes-sessions.test.ts test/session-bridge.test.ts -``` - -RED output (exit 1): - -```text -Test Files 2 failed (2) -Tests 6 failed | 35 passed (41) - -expected { id: 's1' } to deeply equal { id: 's1', alreadyActive: false } -expected raw readiness URL/token/path to equal the fixed public message -expected three raw generic system events to equal [{ type: 'system_event', event: 'session_exit' }] -``` - -GREEN command: - -```text -cd backend && npx vitest run test/routes-sessions.test.ts test/session-bridge.test.ts -``` - -GREEN output (exit 0): - -```text -✓ test/session-bridge.test.ts (14 tests) -✓ test/routes-sessions.test.ts (27 tests) -Test Files 2 passed (2) -Tests 41 passed (41) -``` - -### 2. Backend exact-once SseHub and route framing - -RED command: - -```text -cd backend && npx vitest run test/sse-hub.test.ts test/sse-route.test.ts -``` - -RED output (exit 1): - -```text -Test Files 2 failed (2) -Tests 7 failed (7) - -expected [undefined, undefined, undefined] to deeply equal [1, 2, 3] -expected unconditional replay not to contain "one" / "two" -expected one buffered pending gate, received replay plus a second pending emission -``` - -GREEN command: - -```text -cd backend && npx vitest run test/sse-hub.test.ts test/sse-route.test.ts -``` - -GREEN output (exit 0): - -```text -✓ test/sse-hub.test.ts (4 tests) -✓ test/sse-route.test.ts (3 tests) -Test Files 2 passed (2) -Tests 7 passed (7) -``` - -### 3. Frontend cursor tracking and gate idempotence - -RED command: - -```text -cd frontend && npx vitest run src/stream/useSessionStream.test.tsx src/store/sessionStore.test.ts -``` - -RED output (exit 1): - -```text -Test Files 2 failed (2) -Tests 2 failed | 28 passed (30) - -expected /sessions/s1/events to be /sessions/s1/events?lastEventId=7 -expected duplicate gate pendingWidget to remain null, received gate-1 -``` - -GREEN command: - -```text -cd frontend && npx vitest run src/stream/useSessionStream.test.tsx src/store/sessionStore.test.ts -``` - -GREEN output (exit 0): - -```text -✓ src/store/sessionStore.test.ts (23 tests) -✓ src/stream/useSessionStream.test.tsx (7 tests) -Test Files 2 passed (2) -Tests 30 passed (30) -``` - -### 4. Frontend typed Resume and AppShell ordering/preservation - -Typed API RED command: - -```text -cd frontend && npx tsc -b -``` - -Typed API RED output (exit 1): - -```text -src/api/sessions.test.ts(43,9): error TS2322: Type 'void' is not assignable to type -'{ id: string; alreadyActive: boolean; }'. -``` - -Lifecycle RED command: - -```text -cd frontend && npx vitest run src/api/sessions.test.ts src/shell/AppShell.session-mgmt.test.tsx -``` - -Lifecycle RED output (exit 1): - -```text -✓ src/api/sessions.test.ts (9 tests) -❯ src/shell/AppShell.session-mgmt.test.tsx (15 tests | 4 failed) -Test Files 1 failed | 1 passed (2) -Tests 4 failed | 20 passed (24) - -already-active same-session Resume created two EventSources instead of one -deferred cold Resume closed the document panel before POST completion -failed same-session Resume closed the prior EventSource -failed Resume with no active session opened an EventSource -``` - -GREEN commands: - -```text -cd frontend && npx vitest run src/api/sessions.test.ts src/shell/AppShell.session-mgmt.test.tsx -cd frontend && npx tsc -b -``` - -GREEN output (exit 0): - -```text -✓ src/api/sessions.test.ts (9 tests) -✓ src/shell/AppShell.session-mgmt.test.tsx (15 tests) -Test Files 2 passed (2) -Tests 24 passed (24) -TypeScript: no output, exit 0 -``` - -The AppShell cold-reconnect test additionally proves that the old source accepts an event while -Resume is pending, the replacement URL carries `lastEventId=8`, the replacement receives one -post-resume transcript/activity row, and two deliveries of the same descriptor id yield one gate. - -## Affected verification - -Backend command: - -```text -cd backend && npx vitest run test/routes-sessions.test.ts test/session-bridge.test.ts \ - test/sse-hub.test.ts test/sse-route.test.ts test/health.test.ts test/e2e-f1.test.ts -``` - -Output (exit 0): - -```text -Test Files 6 passed (6) -Tests 52 passed (52) -``` - -Backend typecheck: - -```text -cd backend && npx tsc --noEmit -p . -``` - -Output: no output, exit 0. - -Frontend command: - -```text -cd frontend && npx vitest run src/api/sessions.test.ts src/store/sessionStore.test.ts \ - src/stream/useSessionStream.test.tsx src/shell/AppShell.session-mgmt.test.tsx \ - src/shell/CentralStatus.test.tsx src/shell/ModelActivityPanel.test.tsx \ - src/shell/f1-loop.test.tsx src/shell/AppShell.new-session.test.tsx -``` - -Output (exit 0): - -```text -Test Files 8 passed (8) -Tests 74 passed (74) -``` - -Frontend typecheck: - -```text -cd frontend && npx tsc -b -``` - -Output: no output, exit 0. - -## Full verification - -Backend full suite: - -```text -cd backend && npx vitest run -``` - -```text -Test Files 22 passed (22) -Tests 177 passed (177) -``` - -Frontend full suite: - -```text -cd frontend && npx vitest run -``` - -```text -Test Files 43 passed (43) -Tests 271 passed (271) -``` - -Backend production build: - -```text -cd backend && npm run build -> tsc -p tsconfig.json -exit 0 -``` - -Frontend production build: - -```text -cd frontend && npm run build -> tsc -b && vite build -✓ 4835 modules transformed. -✓ built in 8.25s -exit 0 -``` - -## Integrated re-review closure (2026-07-15) - -This section supersedes the earlier cold same-session assertion that the replacement URL carries -`lastEventId=8`. That behavior was correct only while the backend process and its in-memory id -sequence survived. A restarted backend begins a fresh sequence, so a successful cold Resume now -explicitly discards the browser's cursor before replacing the EventSource. - -All four integrated re-review findings are closed: - -1. `AppShell` passes a dedicated cursor-reset epoch to `useSessionStream`. A cold same-session - Resume increments it only after `alreadyActive: false`; a high cursor such as `901` is omitted - from the replacement URL and fresh low-id events/gates are consumed. An already-active - same-session Resume still preserves its source, cursor, and store. -2. `useSessionStream` no longer mutates the cursor ref during render. Effect setup resets cursor - state on session/reset-epoch changes, callbacks are guarded by a captured active-source - identity, and cleanup clears only its own active identity. A queued event from the replaced - source cannot write the new store or poison its next reconnect URL. -3. Backend Resume is serialized per session and rechecks runtime state inside the lock. Manifest, - readiness, and reopen validation precede the transport commit. Idle/failed replacement creates - and binds the new runtime before `hub.clear`, which occurs synchronously immediately before the - first `Resuming session` publish. Reopen/create failure returns exactly - `Session could not be resumed. Check configuration and connectivity, then try again.`, keeps the - prior hub buffer/subscribers attached, and does not expose exception sentinels. Concurrent calls - perform one cold start and the waiter returns `alreadyActive: true`. -4. `SseHub.forget(id)` removes subscribers, buffered events, and the last id. Permanent session - DELETE invokes it after disk deletion; ordinary close and Resume continue to use `clear`, which - preserves the id sequence. - -### Re-review files - -Production: - -- `backend/src/pi/pi-process-manager.ts` -- `backend/src/routes/sessions.ts` -- `backend/src/sse/sse-hub.ts` -- `frontend/src/shell/AppShell.tsx` -- `frontend/src/stream/useSessionStream.ts` - -Tests/support: - -- `backend/test/pi-process-manager.test.ts` -- `backend/test/routes-sessions.test.ts` -- `backend/test/sse-hub.test.ts` -- `frontend/src/shell/AppShell.session-mgmt.test.tsx` -- `frontend/src/stream/useSessionStream.test.tsx` -- `frontend/src/test/fakeEventSource.ts` - -### Re-review TDD RED/GREEN evidence - -Frontend RED command: - -```text -cd frontend && npx vitest run src/stream/useSessionStream.test.tsx \ - src/shell/AppShell.session-mgmt.test.tsx -``` - -RED output (exit 1): - -```text -Test Files 2 failed (2) -Tests 3 failed | 21 passed (24) - -reset epoch: expected the old source to close, received false -cold same-session: expected /sessions/s1/events, received ?lastEventId=901 -stale source: expected an empty transcript, received "stale session one" -``` - -Frontend GREEN command: - -```text -cd frontend && npx vitest run src/stream/useSessionStream.test.tsx \ - src/shell/AppShell.session-mgmt.test.tsx -cd frontend && npx tsc -b -``` - -GREEN output (exit 0): - -```text -Test Files 2 passed (2) -Tests 24 passed (24) -TypeScript: no output, exit 0 -``` - -Backend RED command: - -```text -cd backend && npx vitest run test/sse-hub.test.ts test/routes-sessions.test.ts -``` - -RED output (exit 1): - -```text -Test Files 2 failed (2) -Tests 8 failed | 28 passed (36) - -three Resume ordering assertions observed clear before reopen/create -reopen and create sentinels escaped as raw HTTP 500 responses -the concurrent waiter cold-started again instead of returning alreadyActive: true -SseHub.forget was absent and DELETE did not invoke permanent cleanup -``` - -Backend GREEN command: - -```text -cd backend && npx vitest run test/sse-hub.test.ts test/routes-sessions.test.ts -cd backend && npx tsc --noEmit -p . -``` - -GREEN output (exit 0): - -```text -Test Files 2 passed (2) -Tests 36 passed (36) -TypeScript: no output, exit 0 -``` - -The failure tests publish a post-failure probe through the same hub and prove that a subscriber -attached before either reopen or create rejection still receives it. The concurrency test overlaps -two same-id requests behind a deferred reopen and proves one manifest/readiness/reopen/create/clear -sequence. - -### Initial re-review verification (before independent-review hardening) - -```text -cd backend && npx vitest run -Test Files 22 passed (22) -Tests 182 passed (182) - -cd frontend && npx vitest run -Test Files 43 passed (43) -Tests 273 passed (273) - -cd backend && npm run build -> tsc -p tsconfig.json -exit 0 - -cd frontend && npm run build -> tsc -b && vite build -✓ 4835 modules transformed. -✓ built in 8.46s -exit 0 -``` - -`git diff --check` produced no output (exit 0). The frontend build retains its pre-existing -large-chunk warning; no new build or type errors were introduced. - -Final whitespace verification: - -```text -git diff --check -no output, exit 0 -``` - -## Self-review - -- Resume sequencing: reopen and runtime binding precede backend `clear` and HTTP success; frontend - state mutation and cursor-reset epoch follow it. Failure catch only emits fixed UI copy. -- Already active: same-session returns before reset/generation/manifest repaint; different session - resets the single-session store and binds the new id only after success. -- SSE exact-once: ids are transport identity, not content hashes; replay is strictly `id > cursor`; - `clear` retains the counter; pending gate matching uses only descriptor id. -- Cursor behavior: hook tracks `MessageEvent.lastEventId`, carries it only to an ordinary same-id - generation, and resets it on session-id/cold-runtime epoch change. Native EventSource reconnect - remains supported by the route header. -- Gate defense: the Zustand set survives pending clear but resets with the session store. -- Client boundary: raw `ensure.error` is unused in public responses; generic Pi system events are - reconstructed rather than spread; frontend type mirrors the two-field event. -- Scope: `git diff` contains no CTE/Card/harness/workflow/persistence/model changes. Four pre-existing - modified `.superpowers/sdd/{progress,task-2-report,task-3-report,task-4-report}.md` files are user - work and are excluded from staging. - -## Remaining concerns - -- The 200-event SSE ring limit remains intentional. A brand-new page can reconstruct only retained - backlog; an in-memory same-session reconnect is exact-once from its cursor. -- Per-session sequence counters remain in backend memory after `clear` by design so later in-process - cold same-id Resume cannot reuse ids. Permanent DELETE removes the counter via `forget`. -- The Delete-then-Resume adversarial route test proves the deleted session is not resurrected but - currently receives the runner's generic HTTP 500 when `sessionShow` can no longer find it. A - future API cleanup can normalize that missing-session response to 404 or 409. -- Frontend tests still print pre-existing MSW unhandled-request and React ref/`act` warnings even - though all 276 tests pass. The frontend production build still reports pre-existing large chunk - warnings. Neither warning class was introduced or expanded by this change. -- No live Pi/DWH smoke was run; this wave changes only REST/SSE/frontend lifecycle boundaries and - is covered by fake-Pi, live Fastify SSE, component, full-suite, typecheck, and production-build - gates. - -## Independent-review hardening - -The required independent review was run repeatedly against the uncommitted diff. Its first pass -found four Important lifecycle edges beyond the integrated findings: queued old-runtime callbacks, -post-spawn construction cleanup, concurrent frontend Resume completions, and the passive-effect -commit window. Its second pass confirmed those fixes and identified one remaining Important -retention issue in the new runtime-identity map. The final pass reported no Critical, Important, or -Minor findings and assessed the diff ready to merge. - -The resulting hardening is: - -- Runtime bridge callbacks are gated by the bound runtime identity. Replacement, close, and DELETE - invalidate the old identity, so queued old events cannot publish or call `failSession`. An active - runtime removed by the manager can still publish its complete public failure sequence; after the - terminal unmanaged `agent_end`, its binding is released and later events are rejected. -- `PiProcessManager` kills the spawned child and removes any registered map entry if either - spawn-boundary stderr setup or later RPC/bridge/map initialization throws. -- Resume completion compares against synchronously maintained current active-session identity. - Concurrent `alreadyActive: false` then `alreadyActive: true` results preserve the cold source, - cursor, store, and replayed gate. -- Stream source replacement uses a layout effect. A deterministic later-layout-effect test delivers - a queued old event inside the former commit-to-passive-cleanup window and proves it is ignored. -- Cursor tests cover both a restarted backend's fresh low ids and an in-process hub's preserved high - ids followed by a cursor-bearing ordinary reconnect. - -### Hardening TDD RED/GREEN evidence - -Backend identity/construction RED command: - -```text -cd backend && npx vitest run test/pi-process-manager.test.ts test/routes-sessions.test.ts -``` - -```text -Test Files 2 failed (2) -Tests 3 failed | 69 passed (72) - -post-spawn reader initialization did not kill the child -replaced and deleted runtime callbacks still called failSession/published -``` - -Additional spawn-boundary and terminal-release RED checks: - -```text -cd backend && npx vitest run test/pi-process-manager.test.ts \ - -t "spawn boundary initialization" -Tests 1 failed | 38 skipped (39) - -cd backend && npx vitest run test/routes-sessions.test.ts -t "terminal sequence" -Tests 1 failed | 34 skipped (35) -``` - -Frontend concurrency/layout RED command: - -```text -cd frontend && npx vitest run src/stream/useSessionStream.test.tsx \ - src/shell/AppShell.session-mgmt.test.tsx -``` - -```text -Test Files 2 failed (2) -Tests 2 failed | 25 passed (27) - -the later-layout-effect event wrote "commit-window stale text" -the false→true completion pair erased pending gate "cold-gate" -``` - -Final focused GREEN commands: - -```text -cd backend && npx vitest run test/pi-process-manager.test.ts \ - test/routes-sessions.test.ts test/sse-hub.test.ts -cd backend && npx tsc --noEmit -p . - -Test Files 3 passed (3) -Tests 79 passed (79) -TypeScript: no output, exit 0 - -cd frontend && npx vitest run src/stream/useSessionStream.test.tsx \ - src/shell/AppShell.session-mgmt.test.tsx -cd frontend && npx tsc -b - -Test Files 2 passed (2) -Tests 27 passed (27) -TypeScript: no output, exit 0 -``` - -### Final full verification after review hardening - -```text -cd backend && npx vitest run -Test Files 22 passed (22) -Tests 188 passed (188) - -cd frontend && npx vitest run -Test Files 43 passed (43) -Tests 276 passed (276) - -cd backend && npm run build -> tsc -p tsconfig.json -exit 0 - -cd frontend && npm run build -> tsc -b && vite build -✓ 4835 modules transformed. -✓ built in 8.47s -exit 0 -``` - -The final frontend run retains the repository's pre-existing MSW/ref/`act` warnings, and the build -retains the pre-existing large-chunk warning. No test, typecheck, or build failures remain. - -## Stale-bootstrap, lifecycle-lock, and competing-Resume hardening - -Date: 2026-07-15 -Base: `08b1f4909e8eb7538156cecc2e7a6cafb46ddfc7` - -This follow-up closes asynchronous identity/order and multi-client transport gaps found in the -pre-deployment review: - -- `PiProcessManager.teardownIfCurrent(id, runtime)` makes teardown an identity-checked operation. - Bootstrap re-checks identity after configuration/retrieval and before both the public - `Starting model` event and model start. Its failure continuation acquires the same session - lifecycle lock, claims only its own runtime identity, and holds serialization through persisted - failure and the public terminal sequence. A continuation left behind by Close or DELETE cannot - target a replacement or recreate forgotten SSE state. -- The former Resume-only promise tail is now a per-session lifecycle lock shared by Resume, Close, - and DELETE. Each route reads the current runtime inside the lock immediately before replacement - or removal and uses identity-checked teardown. Deferred route tests prove both orderings: - Resume then Close/Delete finishes removed with no post-removal bootstrap event; Close then Resume - creates only after Close completes; DELETE then Resume cannot recreate a deleted session. -- AppShell assigns each Resume invocation a monotonic token and records the latest target. A - completion for a different, superseding session id cannot reset the store, select a source, close - the panel, or repaint phase from a late manifest. Same-id invocations are per-target single-flight - operations through the POST and local binding commit: repeated pre-commit clicks update the - shared operation's latest token but issue no second POST or commit path. The operation becomes - joinable again before its manifest fetch, whose repaint remains token/id/selection guarded. Start - new, Stop, streamed session exit, and active-session deletion invalidate pending Resume work. - This prevents stale-source preservation and reverse/non-Resume intent overwrite without allowing - a slow manifest to suppress a later explicit rebind. -- `SseHub` subscriber registrations now carry idempotent transport-close callbacks. `clear` and - `forget` snapshot and actively close every response before discarding runtime transport state; - callback-driven unsubscription during that iteration is safe. The SSE route ends its response so - native EventSource reconnects with `Last-Event-ID`. Post-clear events retain monotonic ids and are - buffered for replay; `forget` additionally resets the id state. - -Production files: - -- `backend/src/pi/pi-process-manager.ts` -- `backend/src/routes/sessions.ts` -- `backend/src/sse/sse-hub.ts` -- `frontend/src/shell/AppShell.tsx` - -Regression tests: - -- `backend/test/pi-process-manager.test.ts` -- `backend/test/routes-sessions.test.ts` -- `backend/test/sse-hub.test.ts` -- `backend/test/sse-route.test.ts` -- `frontend/src/shell/AppShell.session-mgmt.test.tsx` - -### TDD RED/GREEN evidence - -Runtime identity API RED: - -```text -cd backend && npx vitest run test/pi-process-manager.test.ts -t "identity-checked teardown" - -Test Files 1 failed (1) -Tests 1 failed | 39 skipped (40) -TypeError: mgr.teardownIfCurrent is not a function -``` - -Runtime identity API GREEN: - -```text -Test Files 1 passed (1) -Tests 1 passed | 39 skipped (40) -``` - -Deferred bootstrap RED: - -```text -cd backend && npx vitest run test/routes-sessions.test.ts \ - -t "stale bootstrap|bootstrap that" - -Test Files 1 failed (1) -Tests 6 failed | 35 skipped (41) - -close/delete + replacement: stale continuation removed the replacement runtime -delete without replacement: stale continuation called failSession after forget -``` - -Deferred bootstrap GREEN: - -```text -Test Files 1 passed (1) -Tests 6 passed | 35 skipped (41) -``` - -Shared lifecycle ordering RED: - -```text -cd backend && npx vitest run test/routes-sessions.test.ts \ - -t "Resume followed|Close followed|Delete followed" - -Test Files 1 failed (1) -Tests 4 failed | 41 skipped (45) - -All four deferred assertions observed the competing route settle before the first lifecycle -operation released. -``` - -Bootstrap plus lifecycle GREEN: - -```text -cd backend && npx vitest run test/routes-sessions.test.ts \ - -t "Resume followed|Close followed|Delete followed|stale bootstrap|bootstrap that" - -Test Files 1 passed (1) -Tests 10 passed | 35 skipped (45) -``` - -Competing frontend Resume RED: - -```text -cd frontend && npx vitest run src/shell/AppShell.session-mgmt.test.tsx \ - -t "competing Resume|stale Resume manifest" - -Test Files 1 failed (1) -Tests 2 failed | 16 skipped (18) - -reverse POST completion opened a second, stale EventSource -late s1 manifest repainted the selected s3 phase from F3 to F7 -``` - -Competing and same-id Resume GREEN: - -```text -cd frontend && npx vitest run src/shell/AppShell.session-mgmt.test.tsx \ - -t "competing Resume|stale Resume manifest|false then true" - -Test Files 1 passed (1) -Tests 3 passed | 15 skipped (18) -``` - -### Independent-review hardening RED/GREEN - -The first final review reported no Critical findings and three Important edge cases: bootstrap -could start during an in-progress Close; bootstrap-owned failure was persisted twice; and an older -same-id result could overwrite newer state. The integrated reviewer also required non-Resume -navigation to invalidate pending Resume work. The final main review tightened the same-ID contract -to true single-flight so a second same-target click cannot preserve a dead pre-restart source. - -Backend review RED: - -```text -cd backend && npx vitest run test/routes-sessions.test.ts \ - -t "Close suppresses|bootstrap failure persists once" - -Test Files 1 failed (1) -Tests 2 failed | 45 skipped (47) - -deferred configure started Pi while closeSession was still pending -bootstrap/public failure called failSession twice -``` - -Backend review GREEN: - -```text -Test Files 1 passed (1) -Tests 2 passed | 45 skipped (47) -``` - -Same-id single-flight RED: - -```text -cd frontend && npx vitest run src/shell/AppShell.session-mgmt.test.tsx \ - -t "share one cold request" - -Test Files 1 failed (1) -Tests 1 failed | 18 skipped (19) - -two concurrent same-ID invocations issued two cold POSTs (three total including initial activation) -``` - -Non-Resume invalidation RED: - -```text -cd frontend && npx vitest run src/shell/AppShell.session-mgmt.test.tsx \ - -t "starting a new question invalidates" - -Test Files 1 failed (1) -Tests 1 failed | 19 skipped (20) - -the late Resume opened an EventSource after Start new returned to the landing state -``` - -Frontend review GREEN: - -```text -cd frontend && npx vitest run src/shell/AppShell.session-mgmt.test.tsx \ - -t "share one cold request|competing Resume|stale Resume manifest|starting a new question invalidates" - -Test Files 1 passed (1) -Tests 4 passed | 15 skipped (19) -``` - -Post-commit single-flight lifetime RED: - -```text -cd frontend && npx vitest run src/shell/AppShell.session-mgmt.test.tsx \ - -t "releases same-id single-flight" - -Test Files 1 failed (1) -Tests 1 failed | 19 skipped (20) - -s1 committed and waited on its manifest; after s3 superseded it, a new s1 Resume reused the old -operation and issued no second s1 POST (expected 2, received 1). -``` - -Same-id and manifest lifetime GREEN: - -```text -cd frontend && npx vitest run src/shell/AppShell.session-mgmt.test.tsx -t "same-id|manifest" -Test Files 1 passed (1) -Tests 4 passed | 16 skipped (20) - -cd frontend && npx tsc -b -no output, exit 0 -``` - -Multi-client SSE disconnect RED: - -```text -cd backend && npx vitest run test/sse-hub.test.ts test/sse-route.test.ts - -Test Files 2 failed (2) -Tests 3 failed | 6 passed (9) - -clear/forget invoked zero of two registered close callbacks, and two live HTTP SSE responses timed -out instead of reaching EOF after clear. -``` - -Multi-client SSE disconnect GREEN: - -```text -cd backend && npx vitest run test/sse-hub.test.ts test/sse-route.test.ts -Test Files 2 passed (2) -Tests 9 passed (9) - -cd backend && npx tsc --noEmit -p . -no output, exit 0 -``` - -The Hub tests use two subscribers whose close callbacks immediately unsubscribe themselves, proving -safe snapshot iteration and exactly-once closure. The live-route test opens two HTTP streams, proves -both receive EOF on clear, publishes a new event and gate, then reconnects after id 1 and replays -exactly ids 2 and 3. The forget test closes both subscribers and proves the next id resets to 1. - -Close now removes the observed runtime identity before awaiting persistence. Failure persistence is -claimed once per runtime and lifecycle-serialized; bootstrap's public `session_failed` cannot start -a duplicate. A per-target in-flight map owns the only same-ID POST and commit while its mutable -latest token keeps s1→s2→s1 ordering correct; it is removed immediately after the binding commit, -before awaiting the independently guarded manifest. One shared invalidation helper is called when -active deletion, streamed exit, Start new, or Stop begins. - -### Focused verification - -```text -cd backend && npx vitest run test/routes-sessions.test.ts test/pi-process-manager.test.ts \ - test/sse-hub.test.ts test/sse-route.test.ts -Test Files 4 passed (4) -Tests 96 passed (96) - -cd backend && npx tsc --noEmit -p . -no output, exit 0 - -cd frontend && npx vitest run src/shell/AppShell.session-mgmt.test.tsx \ - src/shell/AppShell.new-session.test.tsx src/stream/useSessionStream.test.tsx -Test Files 3 passed (3) -Tests 36 passed (36) - -cd frontend && npx tsc -b -no output, exit 0 -``` - -### Full verification - -```text -cd backend && npx vitest run -Test Files 22 passed (22) -Tests 202 passed (202) - -cd frontend && npx vitest run -Test Files 43 passed (43) -Tests 280 passed (280) - -cd backend && npm run build -> tsc -p tsconfig.json -exit 0 - -cd frontend && npm run build -> tsc -b && vite build -✓ 4835 modules transformed. -✓ built in 8.47s -exit 0 -``` - -The frontend suite/build retain the previously documented MSW, React ref/`act`, experimental type -stripping, and large-chunk warnings. No warning class was introduced by this wave. No harness, -workflow, persistence, SQL/CTE viewer, model-selection, or deployment file changed. The four -pre-existing modified `.superpowers/sdd/{progress,task-2-report,task-3-report,task-4-report}.md` -files remain excluded from staging. - -### Final independent-review verdict - -After the multi-client transport fix, the independent reviewer reported no Critical, Important, or -Minor findings. Its own focused verification passed 96 backend transport/lifecycle tests, 31 -frontend Resume/stream tests, both TypeScript checks, and `git diff --check`. Final assessment: -**Ready to deploy: Yes.** diff --git a/.superpowers/sdd/progress.md b/.superpowers/sdd/progress.md deleted file mode 100644 index d36bc8d2..00000000 --- a/.superpowers/sdd/progress.md +++ /dev/null @@ -1,53 +0,0 @@ -# DWH REST per-installation authentication SDD progress - -Plan: `docs/superpowers/plans/2026-08-20-dwh-rest-per-installation-auth.md` -Branch: `feat/dwh-rest-installation-auth` -Worktree: `/home/chirone/ThothII-next/.worktrees/dwh-rest-installation-auth` -Baseline: workspace docs PASS; Go unavailable on host (use containerized Go 1.26.5); default Compose pre-existing path-sensitive false positive under `/home/chirone`. - -Task 1: complete (commits 3bc84b0..1e82fd3, review clean after bounded-digest fix wave). -Task 1 plan note: the dotted-import grep was resolved by the exact module assertion in `09290a0`; `go list -m all` confirms the DWH module only. - -Task 2: complete (commits 541ef45, 971a0e6, e90a1a1; independent vet fix d2415b5; review clean after bounded-read, expiry/JSON, same-Store, and cross-Store synchronization fix waves). - -Task 3: complete (commits ebb360f, 055dcab, b1079bd; independent review clean after CLI grammar, metadata validation, and ambiguous-publication cleanup fix waves). - -Task 4: complete (commits 943f809, 419c344, 134dc19; independent review PASS after legacy multiplicity, fail-closed handler/socket, SGID 2750, O_RDONLY shared lock, OpenReadOnly, and safe socket-parent waves). Task 5 must precreate `.writer.lock` as `0640 root:dwh-auth`; add a non-owner group/cross-process integration proof when packaging permits. - -Task 5: complete (commits 87606c7, 62ec29f; independent review PASS after auth-subrequest header isolation and target-Nginx duplicate-header verification). No runtime installation or service/Nginx mutation performed. - -Task 6: complete (commits 1b18a0f, d0f7e04; independent review PASS after regex/duplicate bypass, exact-PID TCP, and bounded-cleanup hardening). Minor for final review: remove or rename the redundant legacy `negative_postgrest_bypass` fixture and align the historical fixture-count prose if useful. No active Nginx/runtime mutation performed. - -Task 7: complete (commits 7b9b8b3, 707c13d, f616aab, 7b86ea9, 7fe5316; Terra review PASS). Documentation and rollout-contract alignment completed; no runtime mutation performed. - -Task 8: complete at frozen SHA `6499d24892b4383ac492579303e766cfb51fe44e` (fix commits `09290a0`, `6499d24`; Terra review PASS). Focused, portability, scanner, DWH Go, and two full tools/tht matrix runs PASS; evidence recorded at `.artifacts/dwh-auth/source-verification.md`. Broad coupling remains `BASELINE_RED` debt; immutable paths remain unchanged. - -Task 9: PASS at frozen SHA `0c4ff3750d3ecd3fc514e50e511cf7475fbe0446`. Built and installed the exact local candidate, enabled and started `dwh-auth`, imported the protected legacy credential under public ID `legacy-shared`, and created `psd-mac-primary` with public key ID `oNPdOfoH7ypLtVb1`. Registry check, AF_UNIX-only listener, v1/legacy `204`, random/missing `401`, bounded journal scan, installed-file hashes, and `nginx -t` all PASS. Protected report: `/root/dwh-auth-provision/gate9-20260821T054514Z.report.md`, SHA-256 `65ce1e0d8be5f74355eca2b1dca901da16f2864f68eafb7dede9d23ef36b82d5`. Nginx was not changed or reloaded; the legacy key remains active; the old stack was not changed or stopped. Terra final review: PASS with no Critical or Important findings. - -Task 10: NOT STARTED and requires a second explicit authorization. Public Nginx cutover, Mac-key delivery/configuration, and legacy revocation have not occurred. Activity 1 remains `IN_DISCUSSION`; external deployment remains `SURVEY_NO_GO`. - -Final clarification (bookkeeping): initial authorization at `6499d24` stopped before installation because the protected legacy file was missing and a journal-scan finding remained. A secret-safe legacy file was prepared without emit/hash; Nginx metadata remained unchanged and `nginx -t` PASS. Fix commits `6fb4886`, `dee0f9c`, `0c4ff37` received Terra PASS, followed by a detached complete re-freeze PASS at full `0c4ff3750d3ecd3fc514e50e511cf7475fbe0446`. The owner then explicitly authorized Gate 9 at that exact SHA; Gate 9 completed as recorded above. Task 10 remains a separate gate. - -## Project A authentication runtime projection - -Plan: `docs/superpowers/plans/2026-08-21-project-a-server-auth-runtime-projection.md` -Plan commit: `64f46c7019e11a74dae35a7cdb447cd881e17061` -Implementation baseline: `64f46c7019e11a74dae35a7cdb447cd881e17061` -Runtime constraint: source, synthetic tests, and documentation only; Project A, `/srv`, Nginx, -the legacy stack, and shared services remain untouched. - -Task 1: complete (commits `8a8f2c2`, `f9e2950`, `de86760`; independent Terra review PASS after descriptor-relative rewrite, full-history validation, crash recovery, destructive replacement guards, deterministic failure seams, and interrupted-retention recovery). -Task 2: complete (commits `05f8615`, `da2f4a6`, `1e2c4e6`; independent review PASS after exact GID enforcement, bounded descriptor-bound namespace enumeration, strict trailing-slash parity, OIDC coverage, and complete one-retry `CURRENT` publication linearization). -Task 3: complete (commit `903c0b4`; independent Terra review PASS after retained-FD outer locking, cancellable runtime/canonical waits, public transaction-context propagation, deterministic swap/metadata/creator-race tests, and fail-closed pre/post-commit error handling). -Task 4: complete (commit `3d9a9f0`; independent Terra review PASS after moving the Linux/root restore gate before secret-bearing checkpoint creation). Exact focused Go, race, vet, Node 24 focused/full, TypeScript, projected Compose, secret-policy, Windows backup compile, and full serialized tools/tht gates PASS. Historical canonical/unified Compose failures were reproduced as baseline-only documentation/path coupling failures and were not weakened. -Task 5: complete (commit `ef7ae70`; independent Terra review PASS after read-only Vitest gate repair, -remote-Docker/context hardening, and adversarial canonical-mount/`sudo printenv` verifier fixes). -All 14 cross-layer acceptance cases, documentation verifiers, shell syntax, full Go/race/vet, -backend Node 24 Vitest/typecheck, projected Compose, and secret-policy gates PASS. The three known -default/canonical/unified Compose policy failures were reproduced at `a21e2c1` and remain -unmodified baseline debt. Project A has not been started; applying the descriptor or any runtime -root under `/srv/thothii` still requires a new explicit authorization. -Final Project A source review: PASS for `a21e2c1..ef7ae70`; independent Terra review found no -remaining Critical or Important issue after validating restore admission/order, backend path and -identity controls, transaction cancellation, lifecycle gating, portability, dependency scope, and -redaction. Pre-live stop boundary remains in force. diff --git a/.superpowers/sdd/task-2-report.md b/.superpowers/sdd/task-2-report.md deleted file mode 100644 index 8de7fa04..00000000 --- a/.superpowers/sdd/task-2-report.md +++ /dev/null @@ -1,325 +0,0 @@ -# Task 2 report — protected atomic registry - -## Scope and commit - -- Commit: `541ef45 feat: add protected DWH credential registry` -- Committed files only: - - `tools/dwh-auth/internal/securefile/securefile_linux.go` - - `tools/dwh-auth/internal/securefile/securefile_linux_test.go` - - `tools/dwh-auth/internal/registry/store.go` - - `tools/dwh-auth/internal/registry/store_test.go` -- No server, Nginx, systemd, Docker stack, real registry, secrets, or legacy ThothII files were - read or changed. Tests use `t.TempDir` and synthetic record digests only. - -## TDD evidence - -All Go commands ran in the required official `golang:1.26.5` container with only this linked -worktree bind-mounted at `/work`. The container image reports `go version go1.26.5 linux/amd64`. - -### RED - -Before either Task 2 production file existed, the focused command was run inside the container: - -```text -go test ./internal/securefile ./internal/registry -count=1 -``` - -It failed non-zero for the expected absent implementation symbols, including `undefined: OpenDir`, -`undefined: ReadSecret`, `undefined: Open`, `undefined: State`, `undefined: PublicRecord`, and -`undefined: Store`. - -### GREEN - -After the minimal implementation and formatting: - -```text -go test ./internal/securefile ./internal/registry -count=1 -``` - -Result: - -```text -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/securefile -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/registry -``` - -### Race verification - -The required race command completed successfully: - -```text -go test -race ./internal/securefile ./internal/registry -count=1 -``` - -Result: - -```text -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/securefile 1.027s -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/registry 1.179s -``` - -Additional scoped verification: - -```text -go vet ./internal/securefile ./internal/registry -go test ./... -count=1 -git diff --cached --check -``` - -The Task 2 vet command completed with no findings; all four DWH-auth packages passed the full -module test run; the staged-diff check completed with no output. - -## Delivered behavior - -- `securefile` is Linux-only and traverses absolute paths through descriptor-anchored - `syscall.Open`/`Openat` calls with `O_NOFOLLOW|O_CLOEXEC`; protected roots, child directories, - records, and secret files are regular/directories only and are checked against `Lstat` after - `Fstat`. -- Protected reads reject special, group-writable, or world-writable modes, cap record reads at - 4096 bytes, read at most one extra byte, and reject file-size changes or short/partial reads. - Secret ingress additionally requires exact `0600`. -- Secret output uses `O_CREAT|O_EXCL|O_NOFOLLOW`, exact `0600`, and an absolute protected parent. -- `registry.Open` creates protected `active` and `revoked` subdirectories under an existing safe - root. Record enumeration rejects unexpected entries, unsafe files, symlinks, oversized files, - bad filenames, malformed JSON, unknown JSON fields, duplicate JSON fields, and trailing JSON. -- `Add` validates Task 1 records, writes canonical JSON plus one newline through an exclusive - temporary file, sets final mode `0640`, syncs the file, renames under a protected per-root writer - lock, then syncs the directory. -- `Revoke` writes and syncs a valid revoked record before unlinking and syncing the active record. - `Find` checks revoked first; `List` resolves an active/revoked overlap to the revoked public - record. `FindLegacy` scans fail-closed and permits only the reserved legacy record state. -- `PublicRecord` deliberately omits `secret_sha256`; the redaction is regression-tested. - -## Security-test coverage - -- protected normal files and canonical record publication; -- symlinked roots, registry directories, records, and secret input; -- unsafe root/directory/record/secret modes; -- bounded/oversized record input; -- unknown, duplicate, trailing, and partial JSON; -- filename mismatch and multiple legacy-record integrity failures; -- revoked-state precedence when both active and revoked files exist; -- concurrent adds and concurrent reads during revocation, including the race detector. - -## Self-review - -Reviewed all syscall, path, mode, and error paths after the final race run: - -- Directory traversal never follows a supplied component; later operations use retained directory - descriptors, not re-opened untrusted prefixes. -- `Fstat` validates the opened object and `Lstat` must identify the same inode/device; the direct - child name grammar refuses separators, dot components, and NUL. -- File validation occurs before and after reads; mode/type/size checks fail closed. Directory - listing obtains a fresh `openat(dirfd, ".")` descriptor so scans do not share a mutable directory - offset. -- Writer serialization protects the check-then-rename no-replace sequence. Failed temporary - cleanup leaves an unexpected entry that later scans reject rather than silently accepting it. -- State-specific validation rejects revocation metadata in active records and requires it in - revoked records. Revoked files are consulted before active files so interruption after revoked - publication cannot reactivate a credential. -- All functionality uses only Go standard-library packages and Linux `syscall`; no CGO, SQLite, - or third-party module was added. - -## Concerns - -- The optional whole-module `go vet ./...` reports a pre-existing Task 1 test warning at - `internal/credential/credential_test.go:86` (`append` with no variadic values). The identical - line is present in approved HEAD `1e82fd3`, outside this task’s authorized files. Focused Task 2 - vet passes, and all module tests pass. -- The official image's login shell resets `PATH` and hides `/usr/local/go/bin`; all evidence uses - direct `go`/`gofmt` container entrypoints, which preserves the image’s Go 1.26.5 environment. -- The generic `apply_patch` helper intermittently failed before file access with a sandbox network - namespace error. Exact scoped corrections were applied through the shared worktree workflow; - this did not affect the final staged file set or verification evidence. - -## Review remediation — 2026-08-21 - -### Scope and fix commit - -- Review-fix commit: `971a0e6 fix: harden DWH credential registry reads`. -- Committed files only: - - `tools/dwh-auth/internal/registry/store.go` - - `tools/dwh-auth/internal/registry/store_test.go` -- The separate Task 1 vet correction is the independent preceding commit `d2415b5`; it is not - included in this Task 2 fix commit. No filesystem primitive, server, Nginx, service, registry, - secret, Docker stack, or legacy ThothII file was changed. - -### Strict TDD evidence - -All commands again used the official `golang:1.26.5` image with only this linked worktree mounted -at `/work`. - -#### RED - -The first focused command was run after the new regression tests and before production changes: - -```text -go test ./internal/securefile ./internal/registry -count=1 -``` - -It failed as intended. The three case-variant aliases (`SECRET_SHA256`, `Secret_SHA256`, and -`Schema_Version`) were accepted; past expiry returned active records from both `Find` and -`FindLegacy`; a revocation snapshot let readers return active data before publication; a temporary -file let `List`, `Check`, and `FindLegacy` observe false integrity failures; and the original -concurrent-read regression observed `ErrNotFound` during revocation. - -The deterministic exact-expiry test was then added before the clock implementation. Its focused -run failed as intended with: - -```text -internal/registry/store_test.go:572:10: store.now undefined -``` - -#### GREEN and verification - -After the minimum implementation and `gofmt`: - -```text -go test ./internal/securefile ./internal/registry -count=1 -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/securefile 0.014s -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/registry 0.684s - -go test -race ./internal/securefile ./internal/registry -count=1 -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/securefile 1.022s -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/registry 1.711s - -go vet ./internal/securefile ./internal/registry -``` - -The focused vet output was empty (success). Additional final checks passed: - -```text -go test ./... -count=1 -ok internal/credential -ok internal/record -ok internal/registry -ok internal/securefile - -go vet ./... -go test -race ./internal/registry -count=10 -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/registry 8.127s -git diff --cached --check -``` - -### Remediated security invariants - -- `Find` and `FindLegacy` now deny an active record when `ExpiresAt <= now.UTC()`, returning the - existing non-disclosing `ErrNotFound`. The unexported per-Store `now` function is the minimal - deterministic clock seam; past, exact-equality, and future cases are covered for v1 and legacy - records. A revoked record is still consulted before expiry and therefore remains authoritative. -- One per-Store `sync.RWMutex` creates an in-process consistent snapshot. `Add` and `Revoke` hold - it exclusively for their full writer-lock lifetime, including temporary-file publication and - revoked-then-active removal. `Find`, `FindLegacy`, `List`, and `Check` hold a shared lock; their - bodies delegate only to unlocked helpers, preventing nested-lock deadlocks. `Close` also takes - the exclusive lock before closing descriptors. -- Deterministic regression tests hold the writer path at the revocation publication/unlink and - temporary-file stages. They prove public readers wait, then see either the final revoked state or - a clean directory, eliminating the `Names`-to-load/unlink and temporary-entry false failures - within the Store contract. -- Before struct decoding, the outer record JSON object now requires exactly spelled keys from the - schema allowlist and rejects duplicate literal keys. `Decoder.DisallowUnknownFields`, recursive - duplicate detection, trailing-value rejection, record validation, filename matching, no-follow - reads, modes, and durability ordering remain intact. - -### Self-review and concerns - -- Reviewed the new lock boundaries, error returns, revoked-first ordering, clock fallback, - JSON-token consumption, and every unchanged `securefile` syscall/path/mode boundary. The change - adds only standard-library `sync`; it does not relax existing fail-closed behavior. -- The synchronized snapshot is intentionally per `Store`, matching the requested in-process - contract. The existing protected advisory lock continues to serialize writers across Store - instances/processes; no cross-process reader snapshot is claimed by this fix. -- The historical whole-module vet concern in the original Task 2 report is now resolved by the - independent Task 1 commit `d2415b5`; complete module vet passes in the final evidence above. - -## Cross-Store snapshot remediation — 2026-08-21 - -### Scope and TDD evidence - -This third Task 2 fix wave changes only the protected lock primitive and registry snapshot code: - -- `tools/dwh-auth/internal/securefile/securefile_linux.go` -- `tools/dwh-auth/internal/securefile/securefile_linux_test.go` -- `tools/dwh-auth/internal/registry/store.go` -- `tools/dwh-auth/internal/registry/store_test.go` - -All commands used the official `golang:1.26.5` image with only this linked worktree mounted at -`/work`. - -The test-only red patch initially tried to inspect the unexported `securefile.Dir.fd` through the -registry package and therefore did not compile. That assertion was removed without production -changes: the registry tests still create writer Store A and reader Store B through two independent -`Open(root)` calls, while the securefile test proves separate descriptors directly in its own -package. The subsequent behavioral RED run, before the production change, was: - -```text -go test ./internal/securefile ./internal/registry -count=1 -FAIL TestLockSharedAllowsReadersAndBlocksExclusiveWriter: Dir lacks shared advisory locking -FAIL TestCrossStoreReadersWaitAcrossRevokePublicationAndUnlink: - Find, List, Check, and FindLegacy completed during Store A's revocation snapshot -FAIL TestCrossStoreScanReadersWaitForWriterTemporaryFile: - Store B's List, Check, and FindLegacy observed `.tmp-regression` -``` - -### Delivered synchronization contract - -- `securefile.Dir.LockShared` now acquires `LOCK_SH` on the same protected, no-follow, exact-0600 - root lock file used by `Lock`, which continues to acquire `LOCK_EX`. The lock file is still - opened/created, mode-validated, inode-checked, and closed through the existing Linux syscall - path. -- Every public snapshot reader (`Find`, `FindLegacy`, `List`, and `Check`) takes its Store - `RLock`, then a shared advisory lock on root `.writer.lock`, and retains both through the whole - revoked/active lookup or directory scan/load. `Add` and `Revoke` retain Store `Lock`, then the - same root lock under `LOCK_EX`, over their full operation. -- The lock order is universally Store mutex then root advisory lock. Public methods delegate only - to unlocked helpers, so neither reader nor writer paths recursively acquire the Store mutex. - `Close` retains its exclusive Store mutex, preventing descriptor closure from racing any locked - reader or writer. -- Revoked-first precedence, expiry denial, exact JSON validation, no-follow checks, record modes, - temporary-file durability, and all previous behavior remain unchanged. - -### GREEN and repeated verification - -```text -go test ./internal/securefile ./internal/registry -count=1 -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/securefile 0.019s -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/registry 0.711s - -go test -race ./internal/securefile ./internal/registry -count=1 -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/securefile 1.032s -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/registry 1.748s - -go vet ./internal/securefile ./internal/registry -go test ./... -count=1 -ok internal/credential -ok internal/record -ok internal/registry -ok internal/securefile -go vet ./... - -go test -race ./internal/registry \ - -run 'TestCrossStoreReadersWaitAcrossRevokePublicationAndUnlink|TestCrossStoreScanReadersWaitForWriterTemporaryFile' -count=20 -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/registry 9.880s - -go test -race ./internal/securefile \ - -run TestLockSharedAllowsReadersAndBlocksExclusiveWriter -count=20 -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/securefile 1.063s - -git diff --check -``` - -Both vet commands and the whitespace check produced no output. The securefile regression opens -three protected directory descriptors, proves they are distinct, permits two independent shared -holders, proves a third descriptor cannot take `LOCK_EX|LOCK_NB`, then proves exclusive acquisition -succeeds after shared release. The registry regressions deterministically block Store B readers -while Store A holds the exclusive root lock and verify only final revoked/clean states afterward. - -### Self-review and concerns - -- Reviewed lock creation/reopen races, no-follow flags, exact lock-file mode validation, lock - release, descriptor lifetime, lock ordering, error wrapping, and the unlocked-helper call graph. - No public reader invokes another public reader or writer while holding a Store lock. -- Advisory synchronization necessarily covers cooperating registry Store instances/processes; - arbitrary external filesystem mutation remains fail-closed through the existing integrity - checks rather than being silently accepted. -- No known concerns within the registry's cooperating-process contract. diff --git a/.superpowers/sdd/task-3-report.md b/.superpowers/sdd/task-3-report.md deleted file mode 100644 index 52d8a10f..00000000 --- a/.superpowers/sdd/task-3-report.md +++ /dev/null @@ -1,129 +0,0 @@ -# Task 3 report — secret-safe dwh-auth administrative CLI - -## Scope - -- Added `tools/dwh-auth/internal/command/command.go`, its command tests, and - `tools/dwh-auth/cmd/dwh-auth/main.go`. -- The CLI accepts only the frozen Task 3 grammar: key create/import/list/status/revoke, - registry check, and the reserved serve invocation. -- Existing Task 1–2 APIs are consumed without modifying their files. -- No server, Nginx, systemd, Compose, portable `tht`, real registry, real secret, or legacy - stack was accessed or changed. - -## TDD evidence - -Tests were written before `Run` existed. In the official `golang:1.26.5` container, mounted -against only the dedicated worktree, the focused RED run was: - -```text -go test ./internal/command -count=1 -internal/command/command_test.go:214:10: undefined: Run -FAIL -``` - -After implementation and formatting: - -```text -go test ./internal/command -count=1 -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/command - -go test -race ./internal/command -count=1 -ok github.com/aritmolab/thothii/tools/dwh-auth/internal/command - -go test ./... -count=1 -ok internal/command, internal/credential, internal/record, internal/registry, internal/securefile - -go test -race ./... -count=1 -ok internal/command, internal/credential, internal/record, internal/registry, internal/securefile - -go vet ./... -``` - -## Contract coverage - -- Create generates the Task 1 canonical credential, writes it once through the protected - exclusive `0600` output primitive, syncs/closes it before registry publication, and emits - only `created key_id=... installation_id=... output=...`. -- Existing output is never overwritten. Publication failure attempts compensating removal; - cleanup uncertainty returns exit 4 and reports only the output path. -- Legacy import requires `--legacy-raw`, the reserved `legacy-shared` installation ID, and an - absolute exact-`0600` source. It verifies the opaque value, never changes its source, and - stores only its digest. -- List/status expose `PublicRecord` data only; JSON is written as pristine JSON with no digest. - Revoke requires a non-empty reason and reports only its public key ID. -- Relative paths, malformed/unknown flags, duplicate options, invalid IDs/metadata/expiry, - and missing required values return exit 2. Missing status/revoke keys return exit 3. - Registry/filesystem/integrity failures return exit 4. -- Diagnostics are fixed redacted strings. Tests use a sentinel secret and assert it is absent - from stdout/stderr, list/status/check JSON, import output, and unsafe/integrity failures. - -## Secret-redaction evidence - -The command never prints credential contents or digests. It does not use environment fallback, -interactive stdin, or `flag` diagnostics that echo argument values. The sentinel appears only -in synthetic temporary test input and an integrity-fixture file; all command output assertions -confirm it is absent. The registry’s existing `PublicRecord` contract omits `secret_sha256`. - -## Concerns - -- `serve` is grammar-reserved and returns a redacted exit-4 unavailable response; Task 4 owns - the Unix-socket service implementation and will wire this dispatch. -- The earlier concern about UTF-8 metadata hardening is superseded by `055dcab`: metadata now - rejects invalid UTF-8 and Unicode controls before generation/output. The nil-safe cleanup note - remains non-blocking and outside this review wave. - -## Review-fix wave - -Review findings were addressed in separate commit `055dcab`. Regression tests were added first. The focused RED run in the official Go 1.26.5 container failed on intentionally absent seams: - -```text -undefined: nowUTC -undefined: addRecord -undefined: closeStore -FAIL github.com/aritmolab/thothii/tools/dwh-auth/internal/command -``` - -The fix rejects embedded canonical v1 credentials in description/revocation reason without echoing metadata, validates UTF-8/Unicode controls and expiry against one captured UTC creation time before generation/output, reserves exactly `serve --registry-root ABS --socket ABS`, and makes publication cleanup depend on a definitive registry lookup. Output is retained after publication or close ambiguity, with path-only recovery guidance. - -Review-fix verification in Go 1.26.5: - -```text -go test ./internal/command -count=1 PASS -go test -race ./internal/command -count=1 PASS -go test ./... -count=1 PASS -go test -race ./... -count=1 PASS -go vet ./... PASS -git diff --check PASS -``` - -New tests cover synthetic canonical credentials embedded with prefix/suffix, invalid UTF-8, C1 Unicode controls, past/equal/future expiry, exact serve ordering, deterministic pre-/post-publication and close-failure seams, and sentinel absence from stdout/stderr/list/status JSON. - -## Cleanup snapshot review-fix wave - -The second re-review added two regression tests before implementation. The RED run in the -official Go 1.26.5 container showed the old `Find` proof incorrectly treated both cases as -cleanup-safe: - -```text -FAIL TestCreateRetainsOutputWhenSnapshotFindsUnrelatedIntegrityFailure - corrupt snapshot result = (4, "", "integrity failure\n") -FAIL TestCreateRetainsOutputWhenFailedPublicationRecordIsExpired - expired publication result = (4, "", "integrity failure\n") -``` - -Commit `b1079bd fix: retain DWH key output on ambiguous publication` replaces the `Find` proof -with a complete `Store.List()` snapshot. It removes generated output only when the snapshot -succeeds, the generated key ID is absent, and `Store.Close()` succeeds. Any unrelated integrity -error, active/revoked/expired record, or close error retains the output and emits only path-based -recovery guidance. The clean pre-publication failure path still removes the output. - -Final cleanup-wave verification in Go 1.26.5: - -```text -go test ./internal/command -count=1 PASS -go test -race ./internal/command -count=1 PASS -go test ./... -count=1 PASS -go test -race ./... -count=1 PASS -go vet ./... PASS -git diff --check PASS -``` diff --git a/.superpowers/sdd/task-4-report.md b/.superpowers/sdd/task-4-report.md deleted file mode 100644 index a7f811b0..00000000 --- a/.superpowers/sdd/task-4-report.md +++ /dev/null @@ -1,60 +0,0 @@ -# Task 4 — Unix-socket DWH verification report - -## Scope - -Implemented the standalone Linux verifier at `tools/dwh-auth/internal/service` and wired the exact command: - -```text -dwh-auth serve --registry-root ABSOLUTE_CANONICAL --socket ABSOLUTE_CANONICAL -``` - -The service accepts only `GET /verify`. It returns empty `204` responses with `X-DWH-Key-ID` for verified v1 or reserved legacy credentials; credential failures are generic empty `401` responses, and registry/integrity faults are empty `503` responses. Other paths/methods return empty `404`/`405`. - -## Security decisions - -- Exactly one `X-API-Key` header, maximum 128 bytes. -- Strict `thtdwh_v1.` parsing precedes legacy lookup; non-v1 values alone may use the reserved legacy record. -- Registry integrity is checked before every verification request, so unrelated malformed/unsafe records fail closed with `503`. -- Logs emit only timestamp, decision code, and (when safely parsed or verified) public key ID; test sentinels prove no key, digest, description, or query value is emitted. -- `serve` validates canonical absolute paths, performs startup `Store.Check`, and reports service startup errors as non-secret `integrity failure`. -- Socket collisions that are regular files, directories, symlinks, live sockets, or foreign-owned stale sockets are refused. Only an owned stale Unix socket after `ECONNREFUSED` can be reclaimed. -- Published sockets are mode `0660`; cancellation calls graceful shutdown and removes only a revalidated same-device/same-inode owned socket. A test seam proves a changed path is retained rather than unlinked. - -## Required supporting security fix - -Commit `943f809` (`fix: reject duplicate legacy DWH records`) tightens the Task 2 registry contract: a synthetically valid active plus revoked legacy pair is now an integrity failure. It is intentionally separate from the Task 4 commit. - -## TDD evidence - -RED was observed for the missing handler, listener/configuration API, CLI wiring, unrelated-registry corruption, active+revoked legacy state, and cleanup replacement race. Each increment was then implemented minimally and rerun GREEN. - -## Verification - -All commands were executed in official `golang:1.26.5`, with only this worktree mounted: - -```text -gofmt -w cmd internal/command internal/service -go test ./internal/service ./internal/command -count=1 -go test ./... -count=1 -go test -race ./... -count=1 -go vet ./... -git diff --check -``` - -All passed. A dependency scan also found no third-party Go dependencies. - -## Scope boundary - -No Nginx, systemd, real Unix socket, real registry, credential, legacy stack, or external service was changed. All test data was synthetic and temporary. - -## Follow-up hardening: runtime read-only registry and socket parent - -The Task 5 storage contract uses `root:dwh-auth` SGID directories (`2750`) and a service account with read-only group access. The original registry reader path was incompatible because shared locks were opened `O_RDWR` and lazily created as `0600`; secure-directory validation also rejected SGID. - -The runtime path now uses `registry.OpenReadOnly`: it opens only preprovisioned root, `active`, `revoked`, and `.writer.lock` paths, and rejects `Add`/`Revoke`. The administrative `Open` path bootstraps the lock through the exclusive writer path. Shared lock acquisition opens the existing `root:dwh-auth 0640` lock `O_RDONLY` with `LOCK_SH`; writer acquisition remains `O_RDWR` with `LOCK_EX`, preserving cross-process snapshot exclusion. Secure directories allow SGID but still reject setuid, sticky, group-write, and world-write bits. - -Task 5 must create `.writer.lock` as `0640 root:dwh-auth` alongside the `2750 root:dwh-auth` registry directories before the service starts. - -The socket parent must be a canonical non-symlink directory owned by the service EUID and not group/world writable. This removes the bind-to-chmod and path-replacement exposure from other principals. The remaining POSIX path race is bounded to trusted processes sharing the service EUID inside that non-contendible parent. - -Additional verification (official `golang:1.26.5`, worktree only): focused securefile/registry/service/command tests, full tests, full race tests, vet, plus ten race repetitions each for cross-store snapshot readers, `OpenReadOnly`, and listener tests: all PASS. diff --git a/.superpowers/sdd/task-5-report.md b/.superpowers/sdd/task-5-report.md deleted file mode 100644 index cebde76c..00000000 --- a/.superpowers/sdd/task-5-report.md +++ /dev/null @@ -1,71 +0,0 @@ -# Task 5 report — backend principal enforcement - -## RED - -Added backend route/auth tests before implementation. The initial focused run failed in -seven new assertions: `getPrincipal` did not exist, upstream requests still required the -legacy identity header, foreign session/SSE routes were not hidden, admin scope was not -enforced, new sessions had no trusted principal binding, and settings were global. - -## GREEN - -- Focused backend suite: `66 passed` across auth, sessions, SSE, and settings tests. -- Complete backend Vitest suite: `209 passed` across `22` files. -- `npx tsc --noEmit -p .`, `npm run build`, `git diff --check`, and changed Python - source Ruff all exit successfully. -- Harness targeted repository/local/migration tests and Python bytecode compilation exit - successfully. The new `tht session preferences get|set` commands are registered and - expose the expected Typer help. A direct local CLI preference smoke was not run because - the checked-in local workspace requires unavailable `THT_DB_HOST` configuration. - -## Route and child-process coverage - -- `GET /me` returns the request `PrincipalContext`; upstream accepts only the portal's - normalized `X-Thoth-*` identity tuple, with the legacy header ignored. Local mode uses - the same stable `THT_HOME`/`~/.thothii/identity.json` UUID contract as the harness. -- All session operations are principal-scoped: list (`mine` and admin-only `all`), show, - create, resume, close, delete, rename, group, archive, unarchive, documents, reviewer - response, steer, SQL preview/export, and SSE. Missing and foreign sessions are 404; - absent upstream identity is 401. SSE is authorized before response headers or hub - subscription, so a rejected request cannot attach to a live stream. -- New/resumed Pi runtimes and every route-spawned `tht` process receive - `THT_PRINCIPAL_ISSUER`, `THT_PRINCIPAL_SUBJECT`, optional display name, and admin flag. - The readiness `tht` child is also principal-bound. -- Settings use asynchronous repository-backed `tht session preferences get|set` in the - production runner, which isolates preferences by principal. The legacy settings file is - retained only as an injected-runner compatibility fallback for existing isolated tests. -- Repository/settings authorization failures map to 503 before model startup. SQL execution - errors remain 500 after authorization, preserving the prior API distinction. - -## Self-review and concerns - -- Confirmed the Task 4 portal emits lowercase `true`/`false` for the admin header; the - parser accepts that exact normalized form plus the repository's existing `1`/`0` - compatibility form, and rejects all other values. -- The harness principal resolver is the ownership authority; the backend never accepts an - owner supplied in request bodies. Its route guards use a repository-scoped `session show` - before every session resource operation. -- Existing dependency-injected route fakes without `sessionShow` retain a narrow test seam; - production `ThtRunner` always has that method, so deployed requests cannot bypass the - repository authorization check. - -## Review follow-up - -### RED - -Focused regressions initially failed exactly at the three review findings: stale ambient -display names survived into both `tht` and Pi child environments; mutation/document runner -methods dropped the selected workspace; and `expandLocalHome` did not exist. - -### GREEN - -- Child environments now remove all four `THT_PRINCIPAL_*` keys from their cloned base - environment before applying the exact request principal. Regression tests prove an absent - display name does not inherit a stale ambient value in either child path. -- `setName`, `setGroup`, `archive`, `unarchive`, and `documents` now take and retain an - optional workspace. The rename route regression proves `session show` authorization and - the mutation use the same non-default workspace. -- Local principal paths expand `~`/`~/...`; existing local home and identity file modes are - repaired to POSIX `0700`/`0600` when applicable, with Windows left unchanged. -- Focused suite: `74 passed`; full backend suite: `213 passed` across `22` files, followed by - TypeScript typecheck, production build, and diff check. diff --git a/.superpowers/sdd/task-6-report.md b/.superpowers/sdd/task-6-report.md deleted file mode 100644 index 38267610..00000000 --- a/.superpowers/sdd/task-6-report.md +++ /dev/null @@ -1,298 +0,0 @@ -# Task 6 — Frontend identity and administrator UX report - -## RED - -- Added API tests for the `/me` principal call and `mine`/`all` session-list scopes. -- Added component tests for regular-user scope, admin scope switching, owner labels, - administrator banner, foreign-owner delete confirmation, and foreign-owner archive - confirmation. -- Initial focused run: 7 expected failures (missing `getMe`, missing scope query, - missing owner label/admin controls, and missing foreign-action confirmation). -- The archive-confirmation regression was also run separately before its implementation - and failed because `window.confirm` was not called. - -## GREEN - -- `npx vitest run src/api/sessions.test.ts src/shell/NavSessions.test.tsx src/shell/AppShell.session-mgmt.test.tsx` - — passed (47 tests before the archive follow-up; the focused archive regression then passed). -- `npm test` — passed: 44 files / 305 tests. -- `npx tsc -b` — passed. -- `npm run build` — passed. -- `git diff --check` — passed. -- `npm run e2e` reached Playwright but could not run: the environment has no Chromium - executable at Playwright's configured cache path. No application test failure was reported. - -## Files changed - -- `frontend/src/api/types.ts`: typed principal and session scope contracts. -- `frontend/src/api/sessions.ts`: typed `/me` API call; scoped listing defaults to `mine`. -- `frontend/src/shell/AppShell.tsx`: identity query, admin-only session scope selector and - banner, owner-aware destructive action confirmations. -- `frontend/src/shell/NavSessions.tsx`: owner labels in the all-sessions view. -- `frontend/src/api/sessions.test.ts`, `frontend/src/shell/NavSessions.test.tsx`, and - `frontend/src/shell/AppShell.session-mgmt.test.tsx`: contract and UX coverage. - -## Self-review - -- Regular users remain fail-closed on `mine`; no administrator control renders without - `principal.isAdmin`. -- The all-sessions view includes owner labels (including `Unknown` for legacy records). -- Delete confirmation preserves the pre-existing select-all behavior and adds confirmation - for foreign/unknown owners. Foreign archive now also requires an explicit browser - confirmation; existing Stop & save already has its confirmation dialog. -- A read-only review found no critical, important, or minor issues. The archive guard was - added after that review in response to the requirement to cover every destructive rail - action, and has its own RED/GREEN regression plus the final full verification above. - -## Concerns - -- E2E remains environment-blocked until the Playwright Chromium browser is installed. -- Existing Vitest runs emit pre-existing MSW unmatched-request and dialog-ref warnings; all - assertions pass and this task does not modify those shared test/UI primitives. - -## Review remediation - -- A post-commit review correctly identified that matching `displayName` must never establish - ownership. The predicate now skips confirmation only when `session.author` exactly equals - `principal.subject`; all display-name matches and missing authors are conservative - cross-owner actions. -- Added RED/GREEN regressions where two principals share display name `Alice` but have distinct - subjects: both delete (with another session present, so select-all cannot mask the guard) and - archive require confirmation. -- Added `aria-pressed` to the My sessions / All sessions controls and asserts their selected state - before and after switching. -- Remediation verification: focused regressions passed; full frontend Vitest (44 files / 305 - tests), `npx tsc -b`, `npm run build`, and `git diff --check` all passed. - - ---- - -# DWH authentication Task 6 — Nginx and CI gate report - -## Scope - -Added only the two DWH-auth Nginx gates and the `dwh-auth-linux` deployment workflow job: - -- `scripts/test-dwh-auth-nginx-contract.sh` -- `scripts/test-dwh-auth-nginx-integration.sh` -- `.github/workflows/deployment.yml` - -This report deliberately remains unstaged. The pre-existing frontend Task 6 report above is -preserved rather than overwritten. - -## TDD RED - -The structural gate was written before any Task 5 template change. Those templates already met -the approved contract, so the behavioral RED was obtained by copying them into one exact temporary -root and removing only the effective `/dwh/` `auth_request` directive. The new checker failed as -required, with no credential material in output: - -```text -case=source_contract status=FAIL -``` - -The runtime gate was also first invoked before its file existed: - -```text -bash: scripts/test-dwh-auth-nginx-integration.sh: No such file or directory -``` - -The CI-job RED check found no `dwh-auth-linux` job in `deployment.yml`. No production template was -modified: the tests prove the existing Task 5 template contract instead of weakening it. - -## GREEN - -Shell syntax and workflow YAML were checked with: - -```text -bash -n scripts/test-dwh-auth-nginx-contract.sh scripts/test-dwh-auth-nginx-integration.sh -python3 -c import-yaml-and-safe-load -``` - -The structural gate passed its source contract plus these 13 real copied-and-mutated Nginx fixtures: - -```text -missing_auth_request -missing_proxy_method -missing_proxy_body -missing_proxy_header_isolation -missing_content_length_clear -missing_verifier_key_forward -missing_upstream_key_clear -missing_failure_mapping -public_verifier -tcp_authenticator -postgrest_bypass -failure_mapped_to_success -full_secret_rate_key -``` - -Each test mutates an effective, not comment-only, directive and requires the checker to reject it. -The source test and all 13 fixture tests emitted `case=... status=PASS`, followed by -`case=summary status=PASS`. - -The isolated Nginx 1.24 smoke passed these sanitized cases: - -```text -nginx_1_24 -build_dwh_auth -registry_setup -verifier_start -synthetic_upstreams -composite_nginx_config -nginx_start -auth_socket_unix_only -verifier_not_public -valid_v1 -valid_legacy -invalid_key -revoked_key -expired_key -duplicate_v1 -duplicate_legacy -stopped_verifier -header_and_path_isolation -summary -``` - -It builds with the pinned official Go 1.26.5 image when the host Go binary is absent, creates only -synthetic v1, legacy, revoked, and expired credentials in a `0700` `/tmp` root, runs both Nginx and -the verifier on explicit temporary Unix sockets, and uses a loopback-only marker backend. Its output -is strictly `case` and `status`; keys, values, and digests remain only in the exact temporary root -and are removed by the trap. - -`nginx -t` passed against the complete generated configuration. The marker proves that successful -`/dwh/?keep=exact&second=two` reaches the upstream unchanged, while neither the client API key nor -client or verifier `X-DWH-Key-ID` reaches it. A Unix forwarding probe proves that the verifier sees -only `X-API-Key`, with Cookie, Authorization, and spoofed audit ID absent. Duplicate v1 and ordinary -legacy headers return 401 through Nginx; a stopped verifier returns 503. - -The final local equivalent of the four CI commands passed: - -```text -Docker Go 1.26.5: go test -race ./... -count=1 and go vet ./... -bash scripts/test-dwh-auth-build-contract.sh -bash scripts/test-dwh-auth-nginx-contract.sh -bash scripts/test-dwh-auth-nginx-integration.sh -``` - -The Go race suite passed for command, credential, record, registry, securefile, and service; -`go vet` was silent; the build contract passed; both Nginx gates reached their summaries. - -## CI contract - -The new job uses `actions/checkout` with `persist-credentials: false`, pins Go 1.26.5 with cache -keyed on `tools/dwh-auth/go.mod`, installs `nginx-light`, and runs exactly the four required commands. -Existing jobs were not altered. - -## Self-review - -- The template tests parse normalized effective directives, so commented-out declarations cannot - satisfy the gate. -- The authentication socket is configured as `http://unix:...:/verify`, is observed by `ss -xl`, - and Nginx itself listens only on a temporary Unix socket; neither test starts a public listener. -- All spawned processes are registered by PID; cleanup signals only those PIDs and deletes only the - exact `mktemp` root after a guarded path check. -- The verifier, marker, registry, Nginx prefix, PID, logs, config, and sockets all reside beneath - that root. No `/etc`, systemd, active Nginx config, stack, legacy route, or real registry/key is - read or changed. -- Task 5 templates were not modified because the structural and runtime tests passed unchanged. - -## Concern - -The sandbox `apply_patch` helper repeatedly failed with `bwrap: loopback: Failed RTM_NEWADDR: -Operation not permitted`. A narrowly scoped fallback editor was used only for the workflow and the -Nginx-version assertion. Its first workflow insertion interpreted the action-reference at signs; -the two malformed values were immediately corrected and all final YAML, exact-string, syntax, and -four-command checks were rerun. No remaining product concern is known; the integration gate requires -Nginx 1.24 and Python 3, both supplied by the specified Ubuntu CI runner. - - ---- - -# DWH authentication Task 6 — review remediation wave - -## Review findings and RED evidence - -The three review findings were reproduced against the Task 6 commit before their corresponding -hardening was accepted. - -1. The contract checker originally selected only the first matching `/dwh/` location. A real copied - fixture appended this competing location without authentication: - -```nginx -location ~ ^/dwh/ { - proxy_pass http://127.0.0.1:3001; -} -``` - - The first run reached the new check and failed as required: - -```text -case=negative_postgrest_regex_bypass status=FAIL -``` - -2. The previous process stop sent TERM and immediately used an unbounded `wait`. A synthetic Python - child ignored TERM; the RED run used one exact short-lived watchdog only to prevent a test hang and - produced: - -```text -case=cleanup_term_ignored_bounded status=FAIL -``` - -3. The TCP detector has a positive-control regression. A scratch copy of the integration script - replaced its `ss -ltnpH` detector with `return 1`; its known loopback listener was then not - detected and the run failed with: - -```text -case=tcp_listener_detector_positive status=FAIL -``` - -All RED fixtures and the scratch script used an exact temporary path and were removed. No template, -service, workflow, key, or active Nginx configuration was changed. - -## GREEN changes - -- `location_declarations` consumes normalized, comment-stripped effective lines and `check_templates` - requires exactly one each of the only approved locations: verifier, unavailable named location, and - `/dwh/`. It therefore rejects both any extra intercepting location and a duplicate. The real regex - bypass and a new real duplicate `/dwh/` bypass fixture both pass by being rejected. -- `tcp_listener_for_pid` uses `ss -ltnpH` and a PID-bound match. The integration gate starts a - loopback-only synthetic listener, proves the detector sees that exact PID, stops and deregisters it, - then proves the verifier PID has no TCP listener while its Unix socket remains present. -- `stop_registered_pid` now sends TERM, polls for exit or zombie for a bounded deadline, sends KILL - if required, polls a second bounded deadline, and only reaps a direct child after terminal state is - proved. Explicit stops deregister their PID. The cleanup loop invokes that bounded operation only - for recorded PIDs and removes only its guarded temporary root. -- The synthetic child that ignores TERM is killed by the bounded path, must no longer answer to - `kill -0`, must not remain registered, and must finish within three seconds. Final gate output is - restricted to `case` and `status` lines. - -## GREEN verification - -```text -bash -n scripts/test-dwh-auth-nginx-contract.sh scripts/test-dwh-auth-nginx-integration.sh -Docker Go 1.26.5: go test -race ./... -count=1 and go vet ./... -bash scripts/test-dwh-auth-build-contract.sh -bash scripts/test-dwh-auth-nginx-contract.sh -gate contract: source plus 15 negative fixtures PASS, then summary PASS -bash scripts/test-dwh-auth-nginx-integration.sh -gate integration: 20 named cases PASS, then summary PASS -git diff --check -``` - -The integration cases include `cleanup_term_ignored_bounded`, -`tcp_listener_detector_positive`, `auth_socket_unix_only`, all existing credential decisions, -composite Nginx syntax, and stopped-verifier 503 behavior. Go race tests passed for command, -credential, record, registry, securefile, and service; vet and both diff checks were silent. - -## Self-review and concern - -The new location parser rejects comment-only and non-exact declarations because it operates on the -same normalized effective representation used by the rest of the contract. The TCP positive control -binds only `127.0.0.1` on a kernel-selected temporary port and is stopped through the same exact-PID -path under test. The bounded cleanup avoids arbitrary process lookup or broad signaling. - -The environment still intermittently rejects `apply_patch` with the sandbox loopback error noted in -the original report; only narrowly scoped fallback edits to the two authorized scripts were used and -all final gates were rerun. No remaining review concern is known. diff --git a/.superpowers/sdd/task-7-report.md b/.superpowers/sdd/task-7-report.md deleted file mode 100644 index 7ebb7998..00000000 --- a/.superpowers/sdd/task-7-report.md +++ /dev/null @@ -1,90 +0,0 @@ -# Task 7 — report - -## RED - -- Creato `scripts/test-verify-dwh-auth-docs.sh` con fixture positiva e fixture negative per - credenziale/digest sintetici, TLS insicuro, segreto in env/argv, mode world-readable, cattura - Nginx e coupling Compose. -- Eseguito `bash scripts/test-verify-dwh-auth-docs.sh` prima del verificatore: `case=verifier_missing status=FAIL`. - -## GREEN - -- Aggiunti manuali server, client, TLS, runbook PSD, collaudo ed evidenza sanitizzata; collegati - manuali locali/server, setup PSD, guida, indice e nav MkDocs. -- Eseguiti: `bash -n scripts/verify-dwh-auth-docs.sh scripts/test-verify-dwh-auth-docs.sh`, - `bash scripts/test-verify-dwh-auth-docs.sh`, `bash scripts/verify-dwh-auth-docs.sh`, - `bash scripts/test-verify-workspace-install-docs.sh`, `bash scripts/auth-docs-smoke.sh`. -- Tutti gli output finali sono PASS; il nuovo gate esercita una fixture positiva e nove negative. - -## Self-review - -- Verificati path/owner/mode: registry 2750, lock/record 0640, socket 0660. -- Verificata separazione: chiavi solo `rest_api`; PSD server `postgres_direct`; Mac/remoti REST; - nessun lifecycle Compose per `dwh-auth`. -- Verificati TLS `.it`/SAN, `.com` non coperto, `TLS_CA_FILE`, fingerprint fuori banda, rinnovo e - assenza di bypass. -- Verificati due gate Task 9–10, evidenze solo metadati e nessuna migrazione di sessioni/index/cache legacy. - -## Concern - -- Nessuna mutazione PSD/Nginx/systemd/registry o lettura di segreti è stata eseguita. I comandi del - runbook restano condizionati alle autorizzazioni separate dei Task 9 e 10. - - -## Review fix — RED/GREEN - -### RED review - -- La fixture `sudo nginx -T` ha prodotto il rifiuto `case=sudo_raw_nginx_capture status=FAIL` prima della correzione del gate. -- La fixture header legacy opaco ha prodotto `case=opaque_legacy_header_literal status=FAIL` prima della correzione del gate. -- Dopo avere riallineato le label UI nei manuali, `bash scripts/test-verify-workspace-install-docs.sh` ha prodotto `server-workspace-registry.md: curator flow missing registry rule`: il verifier cercava ancora le due label precedenti. Il test sulla base HEAD e il diff hanno confermato la causa. - -### GREEN review - -- Il gate DWH ora rifiuta anche header opaco, digest JSON quotato, `export` di API key, `curl --header` e `-H`, `sudo nginx -T`, raw diff e Compose; le mutation fixture coprono label, PSD direct/Mac REST/CA, socket e flag REST. -- Il runbook non prescrive raw diff o dump: solo checker strutturale e secret scan con metadati e PASS/FAIL. Il piano Task 10 adotta la stessa regola. -- Il template `psd-local` resta `rest_api` solo Mac/local/remota; il server PSD Project A resta `postgres_direct` con binding separato. La CA privata e `TLS_CA_FILE` sono obbligatori salvo trust approvato equivalente. -- Le procedure server ora coprono backup manifest protetto, restore, curl config 0600 senza segreto in argv/env/output, Unix 204/401, HTTPS 2xx/401, 503 bounded con trap, journal PASS/FAIL e retention alla disinstallazione. -- Il verifier workspace-install e entrambi i manuali registry usano ora le quattro label effettive: `Validate workspace source`, `Test workspace connections`, `Save entered secrets`, `Forget stored value`. - -### Final verification review - -- PASS: `bash scripts/test-verify-dwh-auth-docs.sh`. -- PASS: `bash scripts/verify-dwh-auth-docs.sh`. -- PASS: `bash scripts/test-verify-workspace-install-docs.sh` (fixture complete). -- PASS: `bash scripts/auth-docs-smoke.sh`. -- PASS: `bash -n scripts/verify-dwh-auth-docs.sh scripts/test-verify-dwh-auth-docs.sh` e `git diff --check`. - -### Review concern - -- Nessuna mutazione runtime e nessun segreto reale sono stati letti. I soli comandi server documentati restano soggetti ai gate autorizzativi Task 9 e Task 10. - -## Review fix wave 2 — RED/GREEN - -### RED wave 2 - -- Prima della correzione del proxy, `bash scripts/test-dwh-auth-build-contract.sh` ha fallito il contratto di preservazione path e `bash scripts/test-dwh-auth-nginx-integration.sh` ha chiuso con `case=header_and_path_isolation status=FAIL`: il prefisso `/dwh` arrivava a PostgREST invece di essere rimosso. -- Prima delle procedure finali, il gate docs ha rifiutato il path chiave non deterministico e la fixture curl con header legacy opaco ha dato `case=header_file_curl_synthetic status=FAIL` perché il valore non veniva confrontato esattamente. -- Le mutation fixture hanno catturato l'estrazione tar sul registro attivo e i rename non protetti. Dopo l'inasprimento finale del gate, la sorgente ha dato `dwh-auth docs: restore must stage/check then use guarded same-filesystem renames` finché mancava il controllo fail-closed del candidato. -- Il RED finale dello scanner journal è stato `dwh-auth docs: docs/install/dwh-auth-server.md lacks required topic: sys.argv[2:]`: il gate esige la lettura byte-esatta di v1 e legacy e un `journalctl` che fallisca chiuso. - -### GREEN wave 2 - -- Commit `f616aab fix: preserve PostgREST RPC path through DWH proxy`: `proxy_pass` termina con `/`; il contratto e l'integrazione verificano `/dwh/rpc/ping?x` verso `/rpc/ping?x`. -- Il runbook usa un singolo file chiave v1, header file `0600` passati solo con `curl --header @file`, socket 204 dual-key, HTTPS 2xx pre/post per v1 e 401 post-revoca per legacy `legacy-shared`. -- Restore protetto: staging sul filesystem `/var/lib`, check candidato, `mv -T --` guardato per ogni publish/rollback e pre-restore conservato. Backup/manifest restano root-only `0600` su storage cifrato approvato. -- Lo scanner journal esegue `journalctl` in un unico processo Python root, sopprime stderr, controlla return code e bytes esatti di entrambe le chiavi senza emettere journal o segreti; la shell mostra solo PASS/FAIL. -- Il verifier rifiuta `curl --config`, header in argv, raw Nginx/diff, TLS insicuro, segreti env, mode insicuri e Compose. Le fixture mutano path chiave, ID legacy, header/legacy probes, restore, journal, codici HTTPS e label UI. - -### Final verification wave 2 - -- PASS: `bash scripts/test-dwh-auth-build-contract.sh`. -- PASS: `bash scripts/test-dwh-auth-nginx-contract.sh`. -- PASS: `bash scripts/test-dwh-auth-nginx-integration.sh`. -- PASS: `bash scripts/test-verify-dwh-auth-docs.sh` e `bash scripts/verify-dwh-auth-docs.sh`. -- PASS: `bash scripts/test-verify-workspace-install-docs.sh` e `bash scripts/auth-docs-smoke.sh`. -- PASS: `bash -n` sugli otto gate shell e `git diff --check`. - -### Review concern wave 2 - -- Nessuna configurazione protetta, chiave reale, Nginx, systemd o stack PSD è stata letta o mutata. Le procedure privilegiate restano istruzioni condizionate ai Gate 9–10; la verifica degli owner/mode reali è un'attività del rollout autorizzato, non di questo task documentale. diff --git a/AGENTS.md b/AGENTS.md index 46840fc0..bcb509f9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,7 +7,9 @@ This file provides guidance to Codex (Codex.ai/code) when working with code in t Read [PROJECT_STATE.md](PROJECT_STATE.md) for the current-state snapshot: what was last built, pending manual gates, workspace/secret layout, and design-doc locations. This file holds the stable commands + architecture mental model; PROJECT_STATE.md holds the evolving -detail. Design history lives in `docs/superpowers/specs/` and `docs/superpowers/plans/`. +detail. Current architecture and contracts live in `docs/architecture/`, `docs/contracts/`, +and `docs/evidence.md`; durable design decisions live in `docs/adr/`. Git history is the source +for superseded designs and implementation plans. ## Commands diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 5c2a112c..00000000 --- a/CLAUDE.md +++ /dev/null @@ -1,87 +0,0 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -## Start here - -Read [PROJECT_STATE.md](PROJECT_STATE.md) for the current-state snapshot: what was last -built, pending manual gates, workspace/secret layout, and design-doc locations. This file -holds the stable commands + architecture mental model; PROJECT_STATE.md holds the evolving -detail. Design history lives in `docs/superpowers/specs/` and `docs/superpowers/plans/`. - -## Commands - -The repo has three independently-built layers. Run the **full stack** (real Pi + DWH, needs -VPN + `harness/.env` + `pi` on PATH) with `./scripts/run-stack.sh` (frontend :5173 → backend :8787). - -**harness/** (Python `tht` CLI + Pi gate extension) -- Install: `cd harness && python -m venv .venv && pip install -e ".[dev]"` (puts `tht` on PATH) -- Test: `.venv/bin/pytest -q` — `l2` (real GLM + remote DB) is opt-in via `addopts = -m 'not l2'`; `l0` (testcontainers) needs Docker -- Single test: `.venv/bin/pytest tests/test_session_mutations.py::test_set_name -v` (or `-k `); include e2e with `-m l2` -- Lint: `.venv/bin/ruff check .` (line-length 100) - -**backend/** (Fastify + TypeScript, vitest) -- Dev: `npm run dev` (tsx watch `src/server.ts`) · Build: `npm run build` (tsc → `dist/`) -- Test: `npx vitest run` · Single: `npx vitest run test/routes-sessions.test.ts -t "rename"` -- Typecheck: `npx tsc --noEmit -p .` (vitest does NOT type-check — run this before committing) - -**frontend/** (React 18 + Vite + vitest) -- Dev: `npm run dev` (Vite; set `VITE_BACKEND_URL`) · Build: `npm run build` -- Test: `npx vitest run` · Single: `npx vitest run src/shell/NavSessions.test.tsx` -- Typecheck: `npx tsc -b` · E2E: `npm run e2e` (Playwright) - -No ESLint on the TS layers — `tsc` is the gate. Tests use vitest + MSW (no network). - -## Architecture (the parts that need multiple files to see) - -``` -frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only) -``` - -- **The harness owns the workflow and all persistence.** `tht` (Python) is a deterministic - CLI; `harness/.pi/extensions/tht-gate.js` is a Pi extension that drives an **8-phase - NL→SQL workflow**. The single source of workflow truth is `harness/workflow.yaml`; the - orchestration rules the model must follow are `harness/.pi/skills/tht-sessione/SKILL.md`. - "Current phase" is computed by folding the decision ledger (`harness/tht/phase.py`), not - stored — read it before reasoning about phase logic. - -- **Persistence = phase documents, NOT chat.** A session is a directory under the workspace's - `sessions/` path: `session_manifest.yaml` + per-phase artifacts (`question.md`, - `schema_linking.json`, `sql_final.sql`, …) + `review_decisions.jsonl`. The contract - (SKILL.md): *"the persisted state is the truth — what is not recorded did not happen."* - There is no verbatim transcript store. A resumed Pi process rebuilds context from - `tht session show ` + the on-disk artifacts. - -- **The backend is a thin bridge with no database of its own.** `ThtRunner` shells `tht` - subcommands; `PiProcessManager` runs one Pi child per session and bridges its RPC stream; - `SessionBridge` maps Pi RPC events → client events (`ui_request`/`text_delta`/`info`); - `SseHub` fans them out over SSE to the browser. Persistence belongs to the HARNESS, which - selects the session repository from the workspace config (`harness/tht/session/repository.py`): - filesystem by default, **PostgreSQL when `session_storage` is configured** (server/portable - deployment). Settings flow through harness preferences (`tht session preferences`) with - `backend/data/settings.json` only as the file fallback for injected runners/tests. - -- **Human-in-the-loop gate contract.** The model proposes; a human reviewer decides at gates - via widgets (`reviewer_select` = single pick — a chosen option carrying a `decision` payload - auto-confirms/persists directly, an option without one only asks; `reviewer_decide` = multiselect, - each choice IS a decision; `reviewer_confirm` = artifact/phase gate). The frontend renders these - widget-descriptors (`src/widgets/` registry) and the live transcript is rebuilt in-memory - from the SSE stream (`src/store/sessionStore.ts`) — it is not persisted. - -## Project-specific gotchas - -- **`tht`'s `-c`/`--config` is a PER-COMMAND option** — it must follow the subcommand, never - precede it (`ThtRunner.buildArgv` enforces this; prepending caused live 500s). -- **`--json` output must be pristine** (only valid JSON on stdout) — used as a machine contract. -- **UI strings are English; document *content* stays the workspace language** (Italian for - `psd`) because it's the real data. Only chrome/labels are English. -- **Workspaces** (`harness/workspaces/*.yaml`) set the DB target and **absolute** - `paths.sessions/artifacts/indexes` — for `psd` these point at a *separate, uncommitted* repo - (`tht-workspace-psd/`). Secrets live ONLY in `harness/.env` (gitignored). -- **Settings are global** (workspace/provider/model/thinking, persisted via harness - preferences — `backend/data/settings.json` is only the fallback); the New-session form is - question-only. -- **Resume**: a resumable session re-enters at its last incomplete phase. The backend refuses - resume with 409 when `finalized` or `archived`, and `PiProcessManager.spawnFor` must send - `/riprendi-sessione ` (resume mode) vs `/nuova-domanda` (new) — sending the wrong prompt - silently turns a resume into a new question. diff --git a/CONTEXT.md b/CONTEXT.md index e244e670..a53c99e3 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -91,6 +91,159 @@ correzione successiva crea una nuova sessione derivata, collegata a quella prece dopo la finalizzazione. Non può modificare il ledger, gli artifact canonici o lo stato terminale della sessione. +## Evidence + +**Evidence Module** — Il modulo autonomo che possiede la preparazione delle Evidence e +la loro consultazione durante il workflow. La preparazione avviene fuori dalle singole +sessioni; il workflow usa soltanto contenuti già pubblicati. A runtime contribuisce agli +stage semantici esistenti, senza diventare uno stage visibile e senza modificare ledger, +artifact o stato del workflow. + +**Source Evidence** — Un documento originale del workspace, conservato senza modifiche +come riferimento umano e origine della successiva ristrutturazione. + +**Evidence Unit** — La più piccola unità semantica coerente, revisionabile e ricercabile +derivata da una sola Source Evidence. Possiede un identificatore stabile indipendente +dal kind, assegnato una volta nella forma `evidence:`; fonti diverse non vengono +fuse automaticamente. + +**Evidence kind** — La categoria semantica di una Evidence Unit, che ne determina i +campi specifici e ne orienta l'uso. Ogni unità ha un solo kind primario; i tipi iniziali +sono `glossary`, `domain`, `enum`, `example`, `mapping`, `normalization`, `formula` e +`reference`. + +**Glossary Evidence** — Una Evidence Unit che definisce il significato linguistico, i +sinonimi o le varianti di un termine. + +**Domain Evidence** — Una Evidence Unit che esprime una regola o un vincolo del dominio +non rappresentato da un kind più specifico. + +**Enum Evidence** — Una Evidence Unit che collega un insieme finito di valori +memorizzati ai relativi significati. + +**Example Evidence** — Una Evidence Unit che associa un input o una domanda alla sua +interpretazione o al risultato atteso. + +**Mapping Evidence** — Una Evidence Unit che collega un concetto logico agli elementi +del relativo schema fisico. + +**Normalization Evidence** — Una Evidence Unit che descrive la trasformazione di una +rappresentazione in una forma canonica. + +**Formula Evidence** — Una Evidence Unit che contiene una singola espressione PostgreSQL +componibile e ne dichiara gli input. Una query SQL completa non è una Formula Evidence. + +**Reference Evidence** — Una Evidence Unit che rappresenta un collegamento esterno da +restituire come contenuto autonomo, anziché come semplice provenienza. + +**Evidence purpose** — La destinazione dichiarata di una Evidence Unit nel workflow: +disambiguation, rewriting, schema linking o SQL generation. È distinta dall'Evidence +kind: il tipo descrive cosa contiene, il purpose quando può essere utile; durante la +ricerca il purpose richiesto è un filtro obbligatorio. Il recupero di esperienze e +soluzioni precedenti appartiene al Memory Module e non è un Evidence purpose. + +**Evidence Search Outcome** — Il risultato tipizzato di una consultazione del modulo +Evidence. Distingue una ricerca disponibile, che può legittimamente non trovare +corrispondenze, da un'indisponibilità tecnica che impedisce allo stage chiamante di +avanzare fino a un retry riuscito. + +**Evidence receipt** — La traccia minima di una consultazione disponibile conservata +nella sessione: stage semantico, purpose, generazione interrogata e identificatori delle +Evidence restituite. Non duplica il contenuto delle Evidence. + +**Curated Evidence** — Una o più Evidence Unit ristrutturate a partire da una Source +Evidence e conservate nel repository del workspace come proposte per la revisione +umana. Git conserva la versione precedente e rende visibile ogni modifica; una Curated +Evidence non è ancora contenuto autorevole del runtime. + +**Published Evidence** — Le Curated Evidence valide appartenenti alla revisione attiva +del workspace e alla generazione Evidence pubblicata. L'approvazione umana precede +l'attivazione, ma non viene duplicata come stato nel manifest. + +**Evidence Index** — La proiezione ricercabile e ricostruibile delle Published Evidence. +Accelera il recupero delle informazioni, ma non è una fonte di verità. + +**Evidence preparation** — Il processo di authoring che trasforma Source Evidence in +Curated Evidence mediante estrazione e normalizzazione deterministiche, una singola +ristrutturazione assistita dal modello e una validazione finale deterministica. Nella +prima versione accetta Markdown o testo UTF-8 e non acquisisce automaticamente il +contenuto di URL o documenti esterni. Prepara l'intero insieme delle modifiche in +un'area temporanea e lo applica atomicamente soltanto se tutti gli output sono validi; +non ritenta automaticamente una chiamata al modello fallita. + +**Supporting excerpt** — Un breve estratto presente nel Source Evidence che sostiene +una Evidence Unit. Il sistema ne verifica deterministicamente la presenza dopo la +normalizzazione meccanica; il curatore resta responsabile di verificarne la sufficienza +semantica. + +**Evidence resolution** — L'operazione esplicita con cui un curatore ritira una +Evidence Unit oppure la ricollega a un Source Evidence esistente. Aggiorna documento e +manifest insieme, lascia un diff Git revisionabile e non pubblica né crea commit. + +**Review item** — Un blocco di revisione descritto da codice stabile, messaggio umano e +campo opzionale. Finché viene mantenuto nell'Evidence Unit, ne impedisce la +pubblicazione; la sua storia è conservata da Git, non da uno stato interno all'item. + +**Retirement candidate** — Una Curated Evidence che il Source Evidence esistente non +sostiene più. Rimane visibile con un Review item e blocca la pubblicazione finché il +curatore non la elimina oppure la rende nuovamente coerente con il sorgente. + +**Evidence evaluation set** — Un piccolo insieme versionato di domande rappresentative +e relativi risultati attesi. La baseline è accettabile quando ogni domanda recupera +almeno un risultato atteso nei primi dieci risultati della fusione RRF; il risultato +nei primi cinque è informativo. Comprende almeno un caso lessicale, uno semantico e uno +misto e conserva, a fini diagnostici, le posizioni dense, BM25 e fused. + +**Candidate Evidence Generation** — Una generazione completa dell'Evidence Index che +può essere valutata ma non è ancora visibile alle sessioni. Diventa attiva soltanto se +supera l'Evidence evaluation set. + +**Evidence manifest** — Il file versionato e gestito dal sistema che collega ogni +Source Evidence al suo hash e alle Evidence Unit derivate. Conserva gli identificatori +stabili, permette l'elaborazione incrementale e segnala le unità rimaste orfane senza +cancellarle automaticamente. + +**Orphaned Evidence Unit** — Una Curated Evidence il cui Source Evidence non esiste più. +Rimane disponibile per la revisione, ma blocca la pubblicazione finché non viene +eliminata, ricollegata oppure ne viene ripristinato il sorgente. + +**Evidence Fragment** — Una proiezione ricercabile di una sezione semanticamente +coerente di una Published Evidence. La divisione segue intestazioni e confini di +paragrafo; formule, coppie valore/significato, mapping, regole e URL non vengono mai +tagliati. Il testo completo reso per il frammento usa il solo limite esistente +`max_chunk_chars`, pari per default a 4.000 caratteri; un elemento atomico troppo grande +produce un Review item bloccante. Qdrant indicizza i frammenti, mentre l'Evidence Module +li raggruppa per Evidence Unit. + +**Evidence Result** — La rappresentazione di una singola Evidence Unit restituita dalla +ricerca con metadati, migliori estratti, provenienza e riferimento al documento completo. + +**Hybrid Evidence retrieval** — La ricerca che combina in Qdrant una graduatoria +semantica dense e una graduatoria lessicale BM25 sparse mediante Reciprocal Rank +Fusion. I metadati tipizzati restringono o orientano i risultati senza creare una +collezione separata per ogni Evidence kind. + +**Evidence query text** — La rappresentazione deterministica condivisa dalla ricerca +dense e BM25: domanda originale, concetti, tabelle e colonne in ordine fisso. I campi +vuoti sono omessi; domanda e contesto ricevono soltanto normalizzazione Unicode NFC, +conversione degli a-capo e rimozione degli spazi esterni. Gli elementi contestuali sono +poi deduplicati e ordinati senza conversione delle maiuscole, mentre punteggiatura e +spazi interni della domanda non vengono riscritti. + +**Additive BM25 upgrade** — L'estensione non distruttiva della collezione semantica di +un workspace che conserva il vettore dense predefinito e aggiunge il solo vettore +sparse `bm25`. Soltanto gli Evidence Fragment ricevono valori BM25; Schema e Memory +mantengono invariati dati e ricerca dense. + +**Formula proposal** — Una formula individuata durante una sessione e conservata come +artefatto della sessione. Non diventa Published Evidence finché non viene importata, +revisionata e approvata nel repository del workspace. + +**Fail-closed Evidence retrieval** — Il comportamento per cui un indice assente, +incompatibile o non aggiornato produce nessuna Evidence e un avviso esplicito. Il +workflow può continuare, ma non usa mai silenziosamente contenuti di una revisione +precedente o di un altro workspace. + ## Catalogo dei metadati **Workspace Database** — Il database associato a un workspace, considerato nella sua diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 00000000..e466ce2c --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,327 @@ +--- +name: ThothII +description: "A calm, precise clinical analytics workbench for traceable and reviewable SQL workflows." +colors: + instrument-red: "oklch(55.87% 0.1881 23.2)" + instrument-red-hover: "oklch(50.95% 0.1812 24.1)" + porcelain-background: "oklch(99.18% 0.0011 17.2)" + porcelain-card: "oklch(99.85% 0.0006 17.2)" + warm-surface: "oklch(97.09% 0.0011 17.2)" + sunken-surface: "oklch(94.08% 0.0011 17.2)" + warm-graphite: "oklch(26.78% 0.0097 355.6)" + muted-graphite: "oklch(51.33% 0.0088 345.6)" + quiet-border: "oklch(90.93% 0.0035 354.7)" + success-mint: "oklch(75.77% 0.1581 165)" + warning-amber: "oklch(85.23% 0.1386 78.9)" + information-blue: "oklch(70.35% 0.1128 221.3)" +typography: + display: + fontFamily: "Fraunces, Source Serif Pro, Georgia, Times New Roman, serif" + fontSize: "3rem" + fontWeight: 600 + lineHeight: 1.03 + letterSpacing: "-0.025em" + headline: + fontFamily: "Fraunces, Source Serif Pro, Georgia, Times New Roman, serif" + fontSize: "1.875rem" + fontWeight: 600 + lineHeight: 1.15 + letterSpacing: "-0.015em" + title: + fontFamily: "Fraunces, Source Serif Pro, Georgia, Times New Roman, serif" + fontSize: "1.2rem" + fontWeight: 600 + lineHeight: 1.25 + letterSpacing: "-0.01em" + body: + fontFamily: "Manrope, -apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Arial, sans-serif" + fontSize: "0.9375rem" + fontWeight: 400 + lineHeight: 1.65 + letterSpacing: "normal" + control: + fontFamily: "Manrope, -apple-system, BlinkMacSystemFont, Segoe UI, system-ui, Arial, sans-serif" + fontSize: "0.875rem" + fontWeight: 600 + lineHeight: 1.25 + letterSpacing: "0.005em" + label: + fontFamily: "ui-monospace, SF Mono, Cascadia Code, Menlo, Consolas, monospace" + fontSize: "0.6875rem" + fontWeight: 600 + lineHeight: 1.25 + letterSpacing: "0.06em" +rounded: + xs: "4px" + sm: "6px" + md: "8px" + lg: "12px" + xl: "16px" + full: "9999px" +spacing: + xs: "4px" + sm: "8px" + md: "16px" + lg: "24px" + xl: "32px" +components: + button-primary: + backgroundColor: "{colors.instrument-red}" + textColor: "{colors.porcelain-background}" + typography: "{typography.control}" + rounded: "{rounded.md}" + padding: "0 14px" + height: "32px" + button-primary-hover: + backgroundColor: "{colors.instrument-red-hover}" + textColor: "{colors.porcelain-background}" + typography: "{typography.control}" + rounded: "{rounded.md}" + padding: "0 14px" + height: "32px" + button-secondary: + backgroundColor: "{colors.porcelain-card}" + textColor: "{colors.warm-graphite}" + typography: "{typography.control}" + rounded: "{rounded.md}" + padding: "0 14px" + height: "32px" + input-default: + backgroundColor: "{colors.porcelain-background}" + textColor: "{colors.warm-graphite}" + typography: "{typography.body}" + rounded: "{rounded.md}" + padding: "0 12px" + height: "40px" + card-default: + backgroundColor: "{colors.porcelain-card}" + textColor: "{colors.warm-graphite}" + rounded: "{rounded.lg}" + padding: "16px" + badge-primary: + backgroundColor: "{colors.instrument-red}" + textColor: "{colors.porcelain-background}" + typography: "{typography.control}" + rounded: "{rounded.sm}" + padding: "2px 8px" + height: "20px" +--- + +# Design System: ThothII + +## Overview + +**Creative North Star: "The Clinical Workbench"** + +ThothII should feel like a well-kept clinical workbench: warm enough for sustained reading, exact +enough for consequential review, and quiet enough that evidence, state, and decisions remain in the +foreground. The visual system is calm, precise, and trustworthy. It uses familiar product patterns, +restrained color, and deliberate density instead of decorative spectacle. + +The primary physical scene is an analyst reviewing persisted evidence and SQL on a large monitor in +a well-lit working environment. This makes the warm light theme the default. The supported dark +theme serves lower-light work without becoming a separate neon aesthetic. Both themes preserve the +same hierarchy and semantic roles. + +The system rejects generic SaaS ornament, conspicuous ripples, bounce or elastic motion, long +choreographed transitions, and effects that compete with the analytical task. Controls should feel +disciplined and tactile, never playful, sluggish, or visually unstable. + +**Key Characteristics:** + +- Warm, restrained surfaces with one scarce red accent. +- Editorial headings paired with highly legible operational body text. +- Dense information organized through hierarchy, rhythm, and progressive disclosure. +- Persisted artifacts and reviewer decisions presented as the visual source of truth. +- Fast state feedback with reduced-motion parity. + +**The Workbench Rule.** Every visual element must support inspection, action, state, or provenance. +Decoration without an operational purpose is forbidden. + +**The Persisted Truth Rule.** Persisted artifacts and reviewer decisions receive stronger hierarchy +than transient model narration. + +**The Density with Rhythm Rule.** Preserve information density, but vary spacing between groups so +users can scan structure without adding nested containers. + +## Colors + +The palette combines warm porcelain surfaces, warm graphite text, and an instrument red used only +for action, focus, and important state. OKLCH values in the frontmatter are normative because the +frontend uses OKLCH tokens directly. + +### Primary + +- **Instrument Red** (`instrument-red`): primary actions, current selection, focus identity, and + destructive meaning where the context already makes the action explicit. +- **Instrument Red Pressed** (`instrument-red-hover`): hover and active emphasis for the primary + action family. + +### Neutral + +- **Porcelain Background** (`porcelain-background`): the main canvas. +- **Porcelain Card** (`porcelain-card`): lifted panels, cards, and popovers. +- **Warm Surface** (`warm-surface`): sidebars, secondary controls, and muted regions. +- **Sunken Surface** (`sunken-surface`): selected rows, quiet emphasis, and inset regions. +- **Warm Graphite** (`warm-graphite`): primary text and high-confidence labels. +- **Muted Graphite** (`muted-graphite`): descriptions, timestamps, and secondary metadata. +- **Quiet Border** (`quiet-border`): structural boundaries, input outlines, and dividers. + +### Semantic + +- **Success Mint** (`success-mint`): completed and ready states. +- **Warning Amber** (`warning-amber`): waiting, attention, and in-progress states. +- **Information Blue** (`information-blue`): informational state when red would imply action. + +The dark theme keeps the same semantic mapping with neutral near-black surfaces and a slightly +lighter red accent. Do not introduce a second visual identity for dark mode. + +**The One Voice Rule.** Instrument Red should occupy no more than roughly ten percent of a screen. +Its rarity is what makes it authoritative. + +**The State Has a Name Rule.** Success, warning, information, and destructive colors are reserved +for their named states. Color is never the only state indicator. + +## Typography + +**Display Font:** Fraunces, with Source Serif Pro, Georgia, and Times New Roman fallbacks +**Body Font:** Manrope, with native system sans-serif fallbacks +**Label/Mono Font:** SF Mono or Cascadia Code, with Menlo and Consolas fallbacks + +**Character:** Fraunces gives persisted artifacts and key headings editorial authority. Manrope +keeps dense controls and prose calm and readable. The mono register separates machine identity, +metadata, SQL, identifiers, and micro-labels from natural-language content. + +### Hierarchy + +- **Display** (600, `3rem`, `1.03`): authentication and exceptional page-level statements only. +- **Headline** (600, `1.875rem`, `1.15`): major page or artifact titles. +- **Title** (600, `1.2rem`, `1.25`): panel and document section hierarchy. +- **Body** (400, `0.9375rem`, `1.65`): operational prose, with a target line length of 65 to 75 + characters where the surface controls width. +- **Control** (600, `0.875rem`, `1.25`): buttons, inputs, tabs, and compact actions. +- **Label** (600, `0.6875rem`, `0.06em` tracking): uppercase micro-labels, state metadata, and panel + headers. Labels use the mono family. + +Typography uses fixed sizes. Responsive changes happen at structural breakpoints, not through fluid +type scaling. Numeric data and identifiers use tabular numerals where comparison matters. + +**The Three Registers Rule.** Serif means authority, sans means interaction and reading, mono means +machine identity. Do not exchange these roles for novelty. + +**The Read Once Rule.** A heading, label, and body must be distinguishable on first glance through +size and weight. Do not repeat headings in explanatory copy. + +## Elevation + +The system is flat by default and layered when necessary. Borders mark structure. Warm, diffuse +shadows mark actual elevation for popovers, dialogs, and selected containers. Tonal layering should +solve most hierarchy before a shadow is introduced. + +### Shadow Vocabulary + +- **Contact Shadow** (`--shadow-xs`): a one-pixel contact shadow for controls and code blocks. +- **Panel Shadow** (`--shadow-sm`): a small two-stage shadow for cards that need separation from the + canvas. +- **Overlay Shadow** (`--shadow-md`): a broad, low-opacity shadow for dialogs and floating layers. + +Focus uses an explicit three-pixel ring. Waiting-for-input state may use a success-tinted ring, but +must retain a textual or structural cue. Motion for button state changes lasts `140ms` with +`cubic-bezier(0.22, 1, 0.36, 1)`. Dialog transitions last `100ms`. Activity pulses may run at +`1.5s`, and must be disabled under `prefers-reduced-motion`. + +**The Flat by Default Rule.** A resting surface has no shadow unless it is physically above another +surface. If every panel floats, none of them has hierarchy. + +**The Borders Structure, Shadows Elevate Rule.** Never use shadow as a substitute for grouping or a +border as a decorative accent. + +## Components + +Components are familiar, compact, and state-complete. Every interactive primitive must define +default, hover, focus, active, disabled, loading, and error behavior where those states apply. + +### Buttons + +- **Shape:** gently curved rectangle (`8px`) with a one-pixel transparent or structural border. +- **Primary:** Instrument Red, porcelain text, `32px` default height, and `14px` horizontal padding. +- **Hover / Focus:** shift to Instrument Red Pressed; show a three-pixel focus ring at 25 percent + opacity. Active state scales to `0.97` for `140ms` and removes elevation. +- **Secondary / Outline:** porcelain card surface, Quiet Border, Warm Graphite text, and a Warm + Surface hover. +- **Ghost:** transparent at rest, Warm Surface on hover. Use only where surrounding structure makes + the hit target obvious. + +### Badges and Status Indicators + +- **Style:** compact (`20px` height), gently curved (`6px`), and semibold. +- **State:** pair semantic color with text, icon, or position. A colored dot alone is insufficient + when the state affects workflow decisions. + +### Cards and Containers + +- **Corner Style:** softly rounded (`12px`), with `16px` default internal padding. +- **Background:** Porcelain Card over Porcelain Background or Warm Surface. +- **Shadow Strategy:** Panel Shadow only when the card must read as elevated. +- **Border:** one-pixel Quiet Border at partial opacity. +- **Nesting:** nested cards are forbidden. Use headings, dividers, spacing, or tonal regions. + +### Inputs and Fields + +- **Style:** `40px` height, `8px` corners, Porcelain Background, Quiet Border, and Manrope body text. +- **Focus:** three-pixel Instrument Red ring with a clear border shift. +- **Error / Disabled:** errors combine destructive color with explanatory text; disabled controls + retain readable contrast and use 50 percent opacity. + +### Navigation + +- **Style:** compact session rows use `8px` corners and restrained vertical padding. +- **Default / Hover / Active:** transparent at rest, Sunken Surface on hover, and the same surface + with stronger text weight when active. +- **Responsive:** collapse navigation structurally at the application breakpoint. Do not shrink + labels into illegibility. + +### Curated Evidence Documents + +Curated evidence follows a fixed reading order: title, compact type and purpose summary, scope, +typed content, supporting excerpts, review items, then collapsed technical provenance. Machine +metadata stays in invisible comments so GitHub Preview shows only the reviewable document. + +`applies_to` is rendered as “Ambito di applicazione” with separate bullet lists for concepts, +tables, and columns. Enum values also use lists. Tables are forbidden for metadata, scope, or any +one-dimensional collection; reserve tables for genuinely two-dimensional datasets. Long machine +identifiers use inline code. SQL uses fenced code. Supporting excerpts use blockquotes. + +**The Review Surface Rule.** The visible Markdown must be readable without understanding the +machine contract. Technical metadata belongs in progressive disclosure, not above the title. + +## Do's and Don'ts + +### Do: + +- **Do** make every state change unmistakable without interrupting flow. +- **Do** use Instrument Red only for primary action, current selection, focus identity, or explicit + destructive meaning. +- **Do** preserve information density with headings, rhythm, and progressive disclosure. +- **Do** keep keyboard focus explicit and pair color with text, shape, icon, or position. +- **Do** respect `prefers-reduced-motion` while preserving immediate non-kinetic feedback. +- **Do** use English for interface chrome and the workspace language for persisted document content. +- **Do** render curated metadata and scope as Markdown prose or lists, never as a frontmatter table. +- **Do** break long curated rules into paragraphs, labelled subsections, and lists at existing + punctuation boundaries while preserving the exact canonical text for machines. + +### Don't: + +- **Don't** add generic SaaS ornament, conspicuous ripples, bounce or elastic motion, long + choreographed transitions, or effects that compete with the analytical task. +- **Don't** make controls feel playful, sluggish, or visually unstable. +- **Don't** use gradient text, decorative glassmorphism, or full-saturation accents on inactive + states. +- **Don't** use a colored side stripe greater than one pixel on cards, callouts, list items, or + blockquotes. Use a full border, tonal background, icon, or heading instead. +- **Don't** nest cards or wrap every section in a container. +- **Don't** use a modal before exhausting inline or progressive alternatives. +- **Don't** use tables for `applies_to`, metadata, enum values, or other one-dimensional content. +- **Don't** use color as the sole carrier of success, warning, error, selection, or progress. +- **Don't** use display typography for buttons, labels, or data. +- **Don't** add em dashes to interface copy. Use commas, colons, semicolons, or parentheses. diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index e369408e..4b128b23 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -1,1295 +1,125 @@ # ThothII — Project State -> Starting-point snapshot for new sessions. -> **Requisito finale del progetto (owner, 2026-08-11):** al termine dell'ultima fase tecnica deve -> essere prodotto un documento unico che guidi l'utente passo-passo su (1) come preparare il -> repository dei workspace su Git secondo le regole del progetto, (2) come usare gli strumenti di -> ThothII per il repository (app + CLI `tht`), (3) come usare l'applicazione ThothII di base -> (sessioni, domande, gate). Il documento userà parole semplici ed esempi; i dettagli tecnici -> resteranno nei contratti esistenti. Esempio pratico completo: Policlinico San Donato. -> Last updated: 2026-08-21 (DWH per-installation authentication is active in dual-key mode; -> the owner deferred Mac acceptance and legacy revocation to the mandatory pre-Project-B gate, -> approved a clean replacement with no legacy-state migration and no new host account, authorized -> the read-only survey and Project A private preparation, and did not authorize either stopping the -> legacy stack or starting the new stack). -> Point a fresh session here ("read PROJECT_STATE.md") before substantial work. +Last updated: 2026-08-26. -### PSD server deployment program — design approved, execution PENDING (2026-08-20) +This file is the short operational snapshot. Stable commands and the architecture mental model +live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`, +`docs/contracts/`, `docs/adr/`, and `docs/evidence.md`. Superseded plans and reports are +available from Git history rather than duplicated in the working tree. -- **Approved design:** `docs/plans/2026-08-20-psd-server-deployment-program-design.md`; design - commit `3fd177b`. The owner approved a common read-only survey followed by two independently - accepted projects: A installs/proves a private local-auth stack through F1-F8; B starts only - after A PASS and integrates Supabase schema storage, Authentik OIDC, Nginx, the load balancer, - and the existing Aritmolab sidebar journey. -- **Executable entrypoint:** `docs/plans/2026-08-20-psd-server-deployment-program.md`, with separate - plans for the survey, Project A, and Project B. The server-local Sol agent must execute them with - `superpowers:executing-plans`, checkpointing every verified step and stopping on the documented - owner/secret/rollback boundaries. -- **Human gates:** `docs/testing/psd-server-project-a-manual.md` and - `docs/testing/psd-server-project-b-manual.md`; survey and Project A/B report templates are under - `docs/testing/evidence/`. Automated evidence never substitutes for the two explicit human PASS - decisions. -- **Workspace decision:** keep one `psd-clinical` descriptor in `tht-workspace-psd`; publish - `supported_transports: [rest_api, postgres_direct]`. Mac selects REST, server selects direct; - all installation bindings/secrets remain outside Git. -- **Data/runtime decision:** migrate configuration only. Legacy work sessions, Qdrant indexes, and - Ollama cache are not imported. Project A rebuilds internal Qdrant/Ollama and uses filesystem work - sessions. Project B uses the existing Supabase PostgreSQL database with isolated schema - `thoth_sessions`, dedicated migrator/runtime roles, forced RLS, and no PostgREST exposure. -- **Network/auth decision:** no SSH tunnel. Project A is loopback-only unless the surveyed load - balancer can prove an operator-only temporary endpoint. Project B preserves the real user flow - `Aritmolab homepage -> sidebar -> load balancer -> Nginx -> ThothII`, with direct ThothII-managed - OIDC and no second Nginx `auth_request`. -- **Clean-replacement amendment (owner, 2026-08-21):** no host `thothii` user or group is created. - The image retains its internal numeric UID/GID `10001:10001`; only its dedicated writable bind - trees may carry that unmapped numeric ownership. Old ThothII sessions/configuration are - disposable, but the exact legacy containers, images, source, and data remain intact until the - new Aritmolab journey passes Project B. Shared Omics/LocalLLM networks, ETL Evidence, DWH, - `dwh-auth`, Supabase, Authentik, Superset, and Aritmolab are never cleanup targets. Approved - design and executable amendment: `docs/superpowers/specs/2026-08-21-psd-clean-replacement-design.md` - and `docs/superpowers/plans/2026-08-21-psd-clean-replacement.md`. -- **State:** survey `SURVEY_NO_GO` for Project A private; Project A - `BLOCKED_BY_SURVEY_AND_MUTATION_GATE`; Project B `BLOCKED_BY_PROJECT_A_AND_PRE_B_GATE`. - Legacy retention and UID strategy are resolved. Remaining private-scope blockers are dedicated - read-only workspace access, a dedicated direct-DWH role/route, and sanitized Pi/LLM - metadata. The catalog-only survey proved the currently available `postgres` identity owns - `datawarehouse` and has full write/DDL privileges, so it must not be reused by the new core. - Pi metadata resolves to 0.80.3, `deepseek/deepseek-v4-pro`, thinking `high`, but the bounded - no-session/no-tool reachability probe is FAIL and must be diagnosed without exposing auth data. -- **Sequencing amendment (owner, 2026-08-21):** use two survey decisions. Project A private may - proceed only after `SURVEY_GO_PROJECT_A_PRIVATE` and a separate stop/start authorization. Mac - `rest_api` acceptance, the 48-hour/two-ETL observation, and revocation of `legacy-shared` are - mandatory before `SURVEY_GO_PROJECT_B`. Current authorization covers read-only survey and - preparation only; old-stack stop and new-stack start remain forbidden. -- **Runtime-projection preparation (2026-08-22):** the source contract and hermetic gates prepare - root-only canonical authentication plus a separate `10001:10001` read-only core projection. - This is implementation preparation and test evidence only. Project A has not been started; - applying its descriptor or any server runtime root still needs explicit authorization. +## Current product shape -### Authentication final-review fix round 2 — remediation PASS, release gates remain (2026-08-18) - -- Frozen source is `2a9359071257f9b8a71d36ec2bbb25b161003f81` on `feat/thoth-auth`. - Source and evidence are separate commits; generated local runtime-state directories remain - untracked and must not be staged. -- Local PASS on the frozen source: exact `safeio`/`backup`/`authstorage` tests, full Go race suite, - `go vet`, macOS host build, Windows amd64 package cross-compiles, and Windows CLI build. -- The lifecycle tests now use context-aware gate publication/release, bounded waits for stages, - outcomes and admission, and cancel plus bounded worker join before lock-release assertions. A - deterministic withheld-gate case proves timeout, cancellation, join, and eventual lock release. - The temporary Windows relative-open diagnostic matrix was removed without reducing DACL, NT - normalization, or retained no-delete assertions. -- Authorized exact-source workflow run `32147345625` completed on the exact frozen SHA and - executed the unfiltered native command - `go test ./internal/safeio ./internal/backup ./internal/authstorage -count=1`. The required step - passed: safeio `22.058s`, backup `7.161s`, authstorage `16.088s`. Native Windows StageArchive and - concurrent claim-consume evidence are therefore PASS, not inferred from cross-compilation. -- The same Windows job later failed the unrelated clone-contract script at - `scripts/test-windows-clone-contract.ps1:208` because `$remoteYaml:` is not a valid PowerShell - variable reference. LF/Compose and Linux Docker baseline failures also repeated. The optional - Windows Docker startup job was skipped without executing and is `NOT_RUN` / `BLOCKED`; the - overall completed run conclusion is `failure` because the baseline jobs remain red. -- Historical Node/auth/browser/docs PASS and harness/Ruff/Compose FAIL evidence remains bound to - its recorded source where not rerun. L2, PSD/manual, and provider prerequisites remain - `PENDING`; no new Docker image manifest was generated. -- Durable evidence: `.artifacts/task-15/automated-gates.json`, - `.superpowers/sdd/2026-08-18-thothii-authentication-remediation/task-4-report.md`, and - `.superpowers/sdd/2026-08-18-thothii-authentication-remediation/fix-round-2-report.md`. -- Current automated-gates SHA-256 is - `6c516db5c2064c4a4a2e5f25961b993cd4a8fe020bbbb822fbac7faa0c119599`; the historical Docker - manifest remains bound to its recorded older source and was not reused for this candidate. -- **State:** the three original remediation Important findings remain `RESOLVED`; the fix-round-2 - lifecycle Important is `ADDRESSED`; the Windows diagnostics Minor is `ADDRESSED`; authentication - remediation is `PASS`. Separately, release readiness remains `FAIL`, with L2, PSD/manual, and - provider gates `PENDING`, until unrelated deployment, runner, baseline, and external gates close. - -### P3 effective configuration and `.tht-dwh` — implementation complete, automated PASS, manual PASS (2026-08-13) - -- **Scope:** P3 (PRD D3): a versioned shared canonicalizer produces the non-secret effective - DWH/preprocessing configuration and a stable logical identity - (`workspace://@v1:`), used identically by the application sessions and the operator - CLI. `OWNER.json` writes are versioned; legacy roots remain readable; content-only/Evidence-only - changes keep the identity (no forced reconfiguration), while DWH-affecting changes fail closed - (never silently reusing the old generation). -- **Memory:** explicit workspace-global `paths.memory` root with a guarded migration command - (`tht memory migrate`) that copies and verifies exactly one legacy JSONL under the workspace - lock and fails closed on conflicts. -- **Revision-scoped records:** schema and Evidence Qdrant point IDs, payloads and queries include - `workspace_revision`; memory/solved stay workspace-wide. -- **Operator contract:** `tht` now carries `effectiveConfigIdentity`/`configFingerprint`/ - `inputFingerprint` in results; the operator config lease path is deterministic for the same - revision+identity. -- **Retained evidence:** `.artifacts/p3-integration/p3-da9428d84f152fe059d41a89436496b7/` - (15/15 checks PASS), bound to clean source commit - `3b0726472e15c157…`. -- **Manual gate:** P3 walkthrough in `docs/testing/p2-p6-manual-verification.md`; decision - **PASS** (owner approval 2026-08-13). -### P4 Qdrant collection lifecycle — implementation complete, automated PASS, manual PASS (2026-08-13) - -- **Scope:** P4 (PRD D4): one shared TypeScript collection manager owns the Qdrant collection - and payload-index contract; session admission self-heals a missing collection (1024/cosine + - the 8 required keyword payload indexes) and adds missing indexes, but never mutates an - incompatible collection (`semantic_index_incompatible`); the operator path keeps - `require_existing` semantics. -- **Host CLI:** `tht workspace vector inspect` (read-only contract report) and - `tht workspace vector rebuild --workspace --collection --confirm --destroy` - (guarded delete/recreate of only the descriptor-owned collection, with durable state before - deletion and verification after recreation; mismatched confirmation or missing `--destroy` - → exit 2). -- **Key files:** `backend/src/workspaces/qdrant-collection.ts` (+test), `backend/src/tht/tht-runner.ts` - (`qdrantEnsure` self-heal for admission; default `require_existing` elsewhere), - `backend/src/workspaces/runtime-config-lease.ts` (lease exposes `semanticQdrantUrl`), - `backend/src/workspace-maintenance.ts` + `preprocessing-service.ts` (`vector-inspect`/`vector-rebuild` - operator commands), `tools/tht/internal/workspaceops/operations.go` (+tests). -- **Automated acceptance:** PASS 11/11 (run `p4-466bbfdea9ef3111f36baa99fc2d64aa`, - report `.artifacts/p4-integration/p4-466bbfdea9ef3111f36baa99fc2d64aa/` retained via `--keep`, - bound to clean source commit `e056c19e6214254a9e3b2390e24c389920b84e95`): preflight, clean_state, - ownership, qdrant_up, self_heal_create_missing, self_heal_repairs_missing_index, - incompatible_refused, require_existing_refused, rebuild_recreates_contract, secret_scan, - cleanup_confinement. -- **Gates:** backend 666/666 + tsc clean; Go build+test 9/9; p4 runner unit tests 3/3; harness - 841 passed (only the two pre-existing debt failures unchanged). -- **Manual acceptance:** PASS (owner approval 2026-08-13) — walkthrough section P4 in - `docs/testing/p2-p6-manual-verification.md`. - - -### P5 curated FK annotations in Git — implementation complete, automated PASS, manual PASS (2026-08-13) - -- **Scope:** P5 (PRD D5): the canonical curated FK file is `/schema/annotations.yaml`, - a regular Git blob at the same commit as the descriptor. Absence is compatible (empty canonical set - + warning); symlinks, trees/gitlinks, cross-namespace paths, oversized (>16 MiB), non-UTF-8, and - malformed objects are refused at activation. Activation synchronizes the blob to the immutable - revision root `/data/sessions//revisions//artifacts/mschema/annotations.yaml` with a - restrictive mode and an adjacent ownership manifest (`workspace`, `commit`, `blobId`, - `contentDigest`, `destination`); re-sync is idempotent and re-verifies, and tampered destinations - fail closed. -- **Runtime root:** the backend renders `paths.annotations_root` for the pinned revision while - `paths.artifacts`/`indexes`/`memory`/`sessions` stay workspace-global (the binding-keyed DWH cache - at `artifacts.parent` is untouched); the harness resolves annotations from `annotations_root` with a - legacy fallback. -- **Review primitive:** `tht ... workspace schema accept --run --yes` is the only human FK - review path. It validates the current synced Git blob with the harness parser and records - `{ reviewedCandidatesDigest, annotationsDigest, workspaceRevision, blobId }`. Missing `--yes`, an - unknown run, an empty/malformed blob, or a non-matching candidate fails closed (`annotation_invalid`) - without recording a review. The P2 host-file `schema check --annotations --reviewed-candidates` - review write is superseded (read-only validation only). -- **Continuation gate:** `preprocess run` continues only when the accepted review's blob digest equals - the current revision's synced annotations digest and the DWH binding is compatible; otherwise it - records a new `manual_review_required` checkpoint. -- **Key files:** `backend/src/workspaces/annotations-sync.ts` (+test), `backend/src/workspaces/ - annotations.ts`, `backend/src/workspaces/git-repository.ts` (`annotationsObject`), - `backend/src/workspaces/registry.ts` (activation validation + sync), `backend/src/workspaces/ - preprocessing-service.ts` (`acceptSchema` + continuation gate), `backend/src/workspace-maintenance.ts` - (`schema-accept`), `tools/tht/internal/workspaceops/operations.go` (+tests), `harness/tht/ - config.py` + `cli/schema_cmd.py` (`paths.annotations_root`), `docs/contracts/ - workspace-preprocessing-cli.md`. -- **Gates:** backend **689/689** + tsc clean; Go build+test 9/9; harness focused schema/annotations - 52 passed. Full-suite re-run and the clean-state process goal are recorded at the acceptance gate. -- **Automated acceptance:** PASS 10/10 (run `p5-66b1f1e74f147a23c0a4bff04e6d2a4c`, report - `.artifacts/p5-integration/p5-66b1f1e74f147a23c0a4bff04e6d2a4c/` retained via `--keep`, bound to - clean source commit `9db0299063d5068198c05dc467d7f86dc34de85b`): preflight, clean_state, ownership, - activation_sync, accept_happy_path, revision_isolation, accept_negatives, continuation_gate, - secret_scan, cleanup_confinement. Runner: `scripts/p5-acceptance.sh` / - `backend/scripts/p5-acceptance.mjs` (+unit test `scripts/test-p5-acceptance.sh`). -- **Manual acceptance:** PASS (owner approval 2026-08-13) — walkthrough section P5 in - `docs/testing/p2-p6-manual-verification.md`. - -### P6 commit-addressed Evidence materialization — implementation complete, automated PASS, manual PASS (2026-08-13) - -- **Scope:** P6 (PRD D6): filesystem Evidence `/evidence` is materialized from the exact pinned - Git commit into the immutable revision content root `/snapshots///evidence` - at activation, with a sibling bounded manifest `/evidence.manifest.json` whose digest is chained - into `snapshot.json`. -- **Safety:** fixed Git plumbing (`ls-tree -r -z` + `cat-file blob`), no shell, no mobile checkout; - symlinks/gitlinks at any depth, traversal/absolute/duplicate/cross-namespace paths, and non-regular - modes are refused. Installation-local bounds (defaults): 4096 entries, 64 MiB total, 8 MiB per - file, 4096 path bytes, 1 MiB manifest; a size-sum preflight runs before writing and no partial root - is published. Re-activation reuses a valid root and fails closed on a tampered manifest. -- **Engine:** `evidencePolicy` no longer stops filesystem sources (`evidence_materialization_required` - retired); `preprocess evidence`/`preprocess run` operate on the materialized root. Evidence Qdrant - records remain revision-scoped; corpus ACTIVE is revision-qualified. HTTP/S3 Evidence is unchanged. -- **Retention:** materialized roots live inside the commit-addressed snapshot directory, so they are - retained while pinned and removed by the existing snapshot retention scan when unreferenced. -- **Key files:** `backend/src/workspaces/evidence-materialization.ts` (+test), - `backend/src/workspaces/git-repository.ts` (`evidenceTreeObjects`/`evidenceTreeId`/ - `evidenceBlobBytes`/`gitObjectSize`), `backend/src/workspaces/registry.ts` (activation staging + - integrity chain), `backend/src/workspaces/preprocessing-service.ts` (stop removal), - `backend/src/workspaces/types.ts` + `config.ts` (limits), `docs/contracts/ - workspace-preprocessing-cli.md`. -- **Gates:** backend **698/698** + tsc clean; Go build+test 9/9 (unchanged); harness focused suites - pass. Full-suite re-run and the clean-state process goal recorded at the acceptance gate. -- **Automated acceptance:** PASS 10/10 (run `p6-7a4c4d0ebb63399cfa9f674738b9e8fc`, report - `.artifacts/p6-integration/p6-7a4c4d0ebb63399cfa9f674738b9e8fc/` retained via `--keep`, bound to - clean source commit `124891bbfe8dc8270e8b58c4206150eb8bebeaa7`): preflight, clean_state, ownership, - activation_materialization, evidence_preprocess, revision_isolation, unsafe_tree_refused, - bound_refused, secret_scan, cleanup_confinement. Runner: `scripts/p6-acceptance.sh` / - `backend/scripts/p6-acceptance.mjs` (+unit test `scripts/test-p6-acceptance.sh`). -- **Manual acceptance:** PASS (owner approval 2026-08-13) — walkthrough section P6 in - `docs/testing/p2-p6-manual-verification.md`. - -### P7 PSD migration — plan + repository restructured + local validation PASS; owner-gated (2026-08-13) - -- **Plan:** `docs/superpowers/plans/2026-08-13-p7-psd-migration.md`. -- **Done (autonomous):** `/Users/mp/projects/tht-workspace-psd` restructured to the P1.1 layout and - committed (`thoth-workspaces.yaml` + `psd-clinical/workspace.yaml` schema v3 + `psd-clinical/ - evidence/` 36 `.md` + `psd-clinical/schema/annotations.yaml` 42 KB); legacy runtime dirs gitignored - and the old flat `psd.yaml` retired. A local `WorkspaceRegistry.bootstrap()` against a scratch bare - clone **activated `psd-clinical`** (descriptor valid, 36 Evidence materialized + manifest, 42 KB - annotations synced, `workspace-docs` generated) with no DWH/secret access. -- **Templates:** `deploy/psd/{workspace-bindings,operator,thothii-installation}.env.example` + - gitignored `secrets/`; operator checklist in `docs/install/psd-workspace-setup.md` (registered in - MkDocs nav). -- **Published (2026-08-13):** private repo `https://github.com/mptyl/tht-workspace-psd` (main = - `d4f9185`), consumed via SSH deploy key `thothii-psd` (read-write, passphrase-less, generated in - `deploy/psd/secrets/`). Real operator config is wired (gitignored): `deploy/psd/operator.env`, - `workspace-bindings.env`, `thothii-installation.yaml`, `connector-secrets.yaml` + `secrets/` - (DWH X-API-Key reused from the legacy `.env`; no CA — the DWH REST is public HTTPS). -- **Stack live:** started via `tht start` (project `thothii-70417a3e30ea`), all services - healthy, `qwen3-embedding:0.6b` present; the registry cloned + activated `psd-clinical` - (`ready`); `tht workspace inspect` returns `ok` with descriptor/catalog/runtime identities. - Gotcha recorded: `tht` uses a per-descriptor Compose project name, so the stack must be - started with `tht start` (not a raw `compose-with-preflight.sh up`). -- **Preprocessing live (2026-08-13):** with VPN active, `tht workspace preprocess run - --workspace psd-clinical` **succeeded** against the real PSD DWH — DWH introspection + LSH - (163 tables / 2275 columns), FK review (no new candidates: the 42 KB curated annotations are - authoritative), schema index (2438 records) and filesystem Evidence index (36 docs / 43 chunks). - Qdrant `psd-clinical` now holds **2482 revision-scoped points** (`schema_table` 164, - `schema_column` 2275, `evidence` 43; all carry `workspace_revision`). Rerun is idempotent - (Evidence `unchanged: 36`). -- **Fixes shipped during the live run** (real-DWH scale revealed them): (1) pruned ~95 GB of orphaned - acceptance-run Docker volumes; (2) raised `workspace-maintenance` tmpfs `/tmp` 64 MiB → 1 GiB - (PSD LSH snapshot is ~105 MB); (3) `vector rebuild` now recreates the 8 keyword payload indexes - (it only created dimensions/distance); (4) Qdrant upserts are chunked (256 points/batch) — a 2438- - record schema batch exceeded Qdrant's 32 MiB JSON limit; (5) frozen Evidence metadata lists now - stay lists (`FrozenList`) instead of tuples, preserving JSON shape; (6) embedding timeout 30 s → - 300 s and batch 32 → 16 for large CPU corpora. -- **Remaining:** live session smoke on `psd-clinical` (P8 L2) — create a session with a real - natural-language question and reach the first reviewer gate. - -### Final aggregate P2–P6 verification — automated PASS, manual PENDING (2026-08-13) - -- **Aggregate process goal:** one clean-state run exercises the complete DWH → FK → schema → - filesystem Evidence chain through `tht`/the operator surface, proves idempotency and - revision isolation, proves a second installation consumes the same Git workspace with its own - state, exercises unsafe-tree and bound negatives, and cleans only owned resources. -- **Automated acceptance:** PASS 12/12 (run `p2p6-ee542112c526ef0d4c25ddf6c8bc164b`, report - `.artifacts/p2p6-integration/p2p6-ee542112c526ef0d4c25ddf6c8bc164b/` retained via `--keep`, bound - to clean source commit `1dcf4051b0d9db8ae163e4d7c53871564ca3c564`): preflight, clean_state, - ownership, activation_materialization, dwh_chain, fk_schema_evidence_chain, revision_isolation, - second_installation, unsafe_tree_refused, bound_refused, secret_scan, cleanup_confinement. Runner: - `scripts/p2p6-acceptance.sh` / `backend/scripts/p2p6-acceptance.mjs` (+unit test). -- **Full suites + builds (design §10):** harness **873 passed / 4 deselected** (with color disabled; - the forced-color environment splits `--help` flags and trips the gate-CLI consistency test only); - backend **698/698** + tsc + build; frontend **364/364** + `tsc -b` + build; `tht` Go - build+test **9/9**; `git diff --check` clean. -- **Manual acceptance:** PENDING — "Final aggregate P2–P6 verification" in - `docs/testing/p2-p6-manual-verification.md`. - -### User-guide deliverable (owner requirement) — written, review PENDING (2026-08-13) - -- **`docs/guida-utente.md`** (Italian, simple words + examples) covers: (1) preparing the workspace - Git repository (catalog + schema-v3 descriptor + Evidence + curated annotations), (2) using the - ThothII tools for the repository (`tht` commands + read-only workspace management), and (3) - using the base ThothII application (sessions, questions, gates). It ends with a complete - Policlinico San Donato walkthrough and links to the technical contracts. -- Registered in the MkDocs nav (`mkdocs.yml`). Owner review PENDING. - -### P2 host preprocessing CLI — implementation complete, automated PASS, manual PENDING (2026-08-11) - -- **Scope:** P2 (PRD D2, based on the P1.1 registry contract): the installed native `tht` - binary is the only host interface for workspace preprocessing. Commands: `workspace inspect`, - `preprocess dwh`, `schema suggest-fks`, `schema check`, `index-schema`, `preprocess evidence`, - `preprocess run`, with the exact grammar, file-ingress bounds, result contract and exit codes in - `docs/contracts/workspace-preprocessing-cli.md`. -- **Operator:** `workspace-maintenance` is a profile-gated Compose service sharing the core image, - with no Pi auth/state, no backend/Pi/frontend listener, no Git credentials, and a compiled Node - entrypoint (`backend/src/workspace-maintenance.ts`) driving the existing harness engine through - pristine JSON machine interfaces (`schema_cmd.py`, `vector_cmd.py`, `preprocess_cmd.py`). -- **Boundaries honored:** FK review is digest-bound (candidate digest == persisted artifact; a - review accepted for the same candidate content counts); Qdrant collections are never created by - the product path (`require_existing` + pre-provisioned fixture, P4 owns lifecycle); filesystem - Evidence stops with `evidence_materialization_required` (P6); HTTP Evidence enforces an - installation private-host allowlist; `ssh_tunnel` stays fail-closed (P10); cross-revision DWH - reuse is explicitly P3. -- **Retained evidence:** `.artifacts/p2-integration/p2-b109757b26388a5ed6b1d173dee86584/` - (11/11 checks PASS), bound to clean source commit - `de5de36f9a4edfd4fbebf277822090871ccdd61f`. -- **Manual gate:** P2 walkthrough in `docs/testing/p2-p6-manual-verification.md`; the owner - approved P2 on 2026-08-11 (manual acceptance PASS). P3 and later start only after an explicit - new authorization. - -### P1.1 workspace-directory registry — automated integration PASS, manual PENDING (2026-08-11) - -- **Scope:** P1 correction (not preprocessing). Root curator-owned catalog `thoth-workspaces.yaml`; - one self-contained directory per workspace (`/workspace.yaml`, optional `/evidence/**`); - generated docs stay API-owned under `workspace-docs/`; internal immutable snapshots remain - flat (`//.yaml`) to preserve session pins and runtime trust. -- **Ownership:** the API may create a descriptor once when its catalog slot exists and the - descriptor Git object is absent at the exact base commit. Existing descriptors and curated - content are curator-owned and change only through Git commit/push then installation pull. - Update/delete publish payloads are refused as HTTP 409 `workspace_curator_owned`. Catalog and - Evidence are never written/staged/cleaned by the API. Explicit pull may produce one deterministic - docs-only follow-up commit that never touches curator bytes. -- **Schema/UI:** schema v3 remains the only descriptor schema; filesystem Evidence URI is exactly - `/evidence`. Browser workspace management is read-only for ready workspaces (Pull/Sync, - Validate, installation Test, Export, Evidence summary, curator Git guidance) and offers an - editable bootstrap form only for `configuration_required` catalog slots. -- **Retained evidence:** `.artifacts/p11-integration/p11-ac0b047024fb09eeca218512526a6b23/` - (`report.json` sha256 `44250145fede36de5de941262beb833c920e8c73366d987cdd738856aac6f6d6`), - 19/19 checks PASS, bound to clean source commit - `eac472011e465c24572d9a6bae14de0fb3e246c0` / tree `4fd15ec28d3b7967b7b8757158013307fb20f3b9`. -- **Verification:** backend Vitest **634 passed / 41 files** + tsc + build; frontend Vitest - **364 passed / 54 files** + tsc + build; harness focused Evidence/config pytest **39 passed**; - install-docs and schema-v3-only gates PASS; `workspace-registry-smoke.sh` and - `unified-deployment-smoke.sh` full Docker runs PASS with exact cleanup. -- **Known limitations:** P2–P6 plans/designs are unchanged and their old source paths are - inventoried for a later owner-approved adaptation plan. Windows Docker startup and native - PowerShell contract were not executed on a Windows host. P1's accepted historical evidence and - process artifacts remain untouched; the old P1 process commands are not rerunnable against the - superseding P1.1 repository contract. -- **Manual gate:** `.artifacts/manual-acceptance/p11/` prepared for the reviewer; - follow `docs/testing/p11-manual-acceptance.md`. The owner reviewed the walkthrough and - approved the implementation on 2026-08-11. +ThothII is a human-in-the-loop datamart builder with three independently built layers: ```text -P1.1 automated integration: PASS -P1.1 manual acceptance: PASS (owner approval 2026-08-11) +frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness → DWH (read-only) ``` -# P1 configuration process — ACCEPTED 2026-08-10 +The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence. +The backend is a process/RPC/SSE bridge without a database of its own. The frontend renders the +review gates and keeps the live transcript in memory. See +`docs/architecture/components.md` for the detailed component and data-flow map. -- Retained evidence: `.artifacts/p1-integration/p1-038bf31360180dc831220b33fbadcfe6/report.md` -- Final report hashes: `report.json` `f07d49097966de6f0307490089fdb2ae61379c04b7fc7177d3c44cf001e1b46a`; `report.md` `09b6a9e9ad9eed2b049e286af452e12fa1f3174ea8d633253542604470890c6c`. -- automated integration: PASS -- manual acceptance: PASS — explicitly approved by the project reviewer on 2026-08-10. -- The retained run is bound to clean source commit - `c7338969d7c7c1c396d9099b7ab2d309b70ab6cf` and tree - `5f7013904806054b9f587230be89f34cbc80f5fc`. Its hash-bound provenance contains exact - 43-file backend source and 39-file compiled `dist` manifests (manifest SHA-256 - `eb6d6c77c78d14c798b50d0be430ad124b8fd8965afbc4bb07d358089d23f49` and `9f9e8899f8aca882ff08d49ec6cd00caeea75691c8280095a9b88bc74939a30a`). -- The retained audit has exactly 15 PASS checks and 134 unique declared artifacts whose final - bytes match every SHA-256 declaration. It records 749 PASS command events and 1,664 production - child/network events, with listener shutdown and refusal checks recorded in the final ownership - artifact. Raw Git rejects configured executable diff drivers and other helper-bearing state. -- Manual production acceptance binds every regular compiled distribution file through an immutable - manifest and cached verified module bytes; imported dependency replacement is refused before - RUNNING. Snapshot rendering validates the bounded `snapshot.json`, expected digest, and Git blob - identity, refusing regular source replacement without publishing output. -- Final Task 8/9 focused suites pass (48/48 Task 8; 59/59 manual acceptance and renderer checks), - backend TypeScript/build pass, frontend tests/build pass. Historical harness pytest/Ruff debt - remains unrelated to this P1 work. +## Evidence restructuring — accepted -## Internal Qdrant + Ollama semantic infrastructure — LIVE 2026-08-08 +The evidence restructuring and PSD migration completed real acceptance on 2026-08-25. -- **Compose topology.** The mandatory application stack is `frontend`, `core`, `qdrant`, - `embedding`, and the one-shot `embedding-model-init`. Startup is CPU-first by default; Linux - hosts may opt into GPU exposure with `THOTH_ENABLE_EMBEDDING_GPU=1`. Qdrant is private on the - Compose network and persists `/qdrant/storage` in `qdrant-data`. Ollama persists its local model - cache in `embedding-models`, and `embedding-model-init` blocks `core` until - `qwen3-embedding:0.6b` is present. - -- **Semantic contract.** Internal semantic indexing is fixed to `qwen3-embedding:0.6b`, - `1024` dimensions, and cosine distance. Schema v3 is the only accepted workspace descriptor. - Schema v1 and v2 workspace descriptors are rejected before activation. Candidate snapshot - validation makes activation or a pull fail atomically and leaves the prior valid snapshot active; - there is no in-product migrator or automatic conversion. One workspace owns one Qdrant - collection, and - schema, Evidence, and Memory records coexist inside that collection with payload `kind` - separation. - -- **Final review runtime barriers.** Operational routes, retained session pins, and runtime - rendering now require schema version 3 before resolving bindings, readiness, diagnostics, or - Pi. Session admission verifies the exact internal Qdrant collection (dimensions, cosine - distance, and required keyword payload indexes) before Ollama and before manifest persistence. - The Qdrant adapter binds every search/list/delete filter to its constructed workspace identity - and rejects conflicting caller namespaces. -- **Boundary and persistence.** Only DWH and LLM remain external runtime application endpoints. - There are no active external vector or embedding endpoint instructions, bindings, or secrets in - the supported operator manuals. Qdrant remains a derived but persistent semantic index: the - canonical sources of truth stay the workspace Git descriptors, phase artifacts, and memory - registry/ledger. The Ollama model cache is recoverable for offline startup but is not the - canonical source of semantic content. -- **Backup and recovery.** `./scripts/vector-backup.sh --project-name --output ` - archives exactly one labeled `_qdrant-data` volume and preserves the prior `qdrant` - running state. `./scripts/vector-restore.sh --project-name --input - --confirm-project ` requires the exact repeated project confirmation, validates manifest - and archive safety before stopping `qdrant`, stages rollback content, restores semantic storage - in place, and restarts `qdrant` only if it was previously running. Recovery requires the registry - to already hold a reviewed v3 descriptor revision compatible with the restored collection; the - helper does not restore descriptors, rename collections, or repair a semantic-index - incompatibility. Backup and restore share one atomic Docker-daemon lock per Compose - project/Qdrant volume; contenders fail before volume resolution, and cleanup removes the lock - only when its ownership labels still match. -- **Verification recorded for Task 13 final audit.** On Apple M4 Pro - (`Darwin 25.5.0`, Docker Server `29.6.2 linux/arm64`), harness pytest passed - **827 passed / 4 deselected**; backend Vitest passed **477/477** plus TypeScript and build; - frontend Vitest passed **374/374** plus TypeScript and build; `git diff --check` passed. - Deployment contracts passed: - `test-default-compose.sh`, `test-unified-compose.sh`, `test-internal-semantic-compose.sh`, - `test-no-deployment-coupling.sh`, `test-compose-secret-policy.sh`, and - `verify-workspace-install-docs.sh --fixtures-only`. -- **Task 13 Docker smoke evidence.** CPU semantic smoke passed in **217.34s** and proved - offline Qdrant/Ollama persistence plus exact cleanup. Workspace registry smoke passed in - **42.06s** in fix round 1 with a per-run image tag derived from the unique Compose project, - and proves exact cleanup of compose containers, volumes, networks, and only that smoke image. - Unified deployment smoke passed in **125.57s**; update-only rollback smoke - passed in **85.40s**; Linux server deployment smoke passed in **55.99s**. The previously - observed `tht` rollback failure did not recur. -- **Task 13 image and manual-gate notes.** Verified pinned runtime images: - `qdrant/qdrant:v1.18.2@sha256:75eab8c4ba42096724fdcfde8b4de0b5713d529dde32f285a1f86fdcb2c9e50c` - and - `ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a`. - The workspace-registry smoke fix-round image used tag - `thothii-workspace-registry-smoke:thoth-workspace-registry-smoke-thoth-workspace-registry-smoke-10vi3a-19157`, - built manifest list `sha256:715b943057929418cad4aa71806d9edbaf823555d19bda6b875297617463fd4a` - with config `sha256:a566521981e08958aae9a12bfc7803bb5f3f835536b4bb8c39df8fcf26063161`, - and removed that exact reference during cleanup. Local GPU exposure (`THOTH_ENABLE_EMBEDDING_GPU=1`) and - Windows Docker Desktop startup were not manually executed in this run. -- **Task 13 known limitations.** Broad harness Ruff remains existing unrelated debt - (**220 errors**); touched harness files were verified Ruff-clean. The final active-reference - audit remains non-empty only in deterministic negative guards, retained off-repository migration - SQL, L2 compatibility fixtures, gitignored task notes, and historical reference notes. No active - schema-v3 operator manual or supported runtime deployment path retains external vector or - embedding endpoint coupling. -- **Final review fix verification.** Backend Vitest passed **477/477** plus TypeScript and build; - harness pytest passed **827 passed / 4 deselected** with the existing 74 warnings; touched Python - files are Ruff-clean. The complete `tht` Go suite, deterministic backup/restore safety test, - internal semantic Compose contract, no-deployment-coupling gate, CPU/offline semantic smoke, and - unified deployment smoke all pass after the final fix. The intermittent `tht` rollback failure was - traced to Docker Desktop alternating equivalent bind sources between `/private/...` and - `/host_mnt/private/...`. Exact state-v4 source hashes remain unchanged; only fresh bind - observations made by a Darwin `tht` carry non-serialized aliases for the rollback - comparison, so pre-fix recovery state remains readable and Linux `/host_mnt` paths remain - distinct. The rollback-only smoke passed twice consecutively after each fix revision, and the - subsequent full unified smoke passed with exact cleanup. +- The curated PSD revision contains 35 approved Evidence units and 60 review items. +- The PSD authoring repository publishes all 35 units using Curated unit schema v3. Its table-free + presentation uses hidden canonical metadata, wrapping Markdown scope lists, list-based enum + values, and collapsed technical provenance. Long domain rules now have a deterministic + human-readable presentation while retaining their exact canonical text for vector ingestion. + `tht evidence migrate ` performs the deterministic v1/v2 upgrade and older-v3 + presentation rewrite without model calls. The structured-rule PSD rewrite is currently local and + pending commit/publication. +- The accepted snapshot is + `psd-clinical-675990d90eae51da6f2bd51b1ae2609f245772ef-snapshot`. +- The active generation is `gen:f968b3bd7a553dbfef3cf47093698f2bc7f95f11`. +- Retrieval acceptance reached 20/20 Hit@10. +- A real session, `20301df7-cad7-403d-a4c1-9f35c9d07b66`, completed F1–F8 with five + receipts, three CTEs, and a final result of 78 patients. +- The durable acceptance record is + `docs/testing/evidence/evidence-restructuring-psd-acceptance-2026-08-25.md`. -# Historical archive -## Historical snapshots and archived reference notes +The canonical authoring, validation, publication, materialization, and preprocessing flow is +documented in `docs/evidence.md`. The governing contracts are +`docs/contracts/workspace-evidence-v3.md` and +`docs/contracts/workspace-preprocessing-cli.md`. -### Historical snapshot — Unified deployment release gate, Task 13 (2026-08-05) +## Workspace preprocessing and configuration -- **Release coverage.** `scripts/unified-deployment-smoke.sh` gates the two-service render/build, - frontend-to-core routing, embedded pinned Pi, Git registry bootstrap, offline recreation, valid - update, invalid-update retention, and the four persistent stores. `scripts/tht-update-smoke.sh` - independently exercises the bad-Pi update and automatic rollback path. - `scripts/server-deployment-smoke.sh` starts the server plus required session overlays with the - same smoke-built core/frontend images, disposable bind roots/secrets/session configuration, - upstream-auth checks, and fail-closed unavailable-session behavior. -- **Isolation and disclosure boundary.** Every run generates a unique temporary root, Compose - project, container/image names, transaction image tags, and run label. The rollback fixture uses - an immutable `hello-world` digest whose preflight exits successfully, guaranteeing the stopped - core state required by `tht` compensation. Cleanup includes stopped project containers in - its final ownership check immediately before teardown and removes only exact containers, - Compose resources, image references, control state, and temporary files. There is no global - prune. Failure diagnostics are bounded and sanitized, and all credentials/endpoints used by the - smokes are disposable fixtures rather than operator or repository secrets. Every public smoke - also has an internal 30-minute process-group supervisor with TERM/KILL of the complete group. -- **Cross-platform CI contract.** `.github/workflows/deployment.yml` uses immutable action commits, - pinned supported Node and Go versions, runs LF/Compose/secret/coupling/docs/TypeScript gates on - Linux, runs each Linux Docker smoke once under its own outer timeout, and copies the Windows - source into a path containing spaces before building/invoking native `tht` and rendering - Compose. The optional `windows_docker_startup` dispatch targets a labelled self-hosted Windows - Docker Desktop/WSL2 runner and performs bounded two-service startup and exact cleanup. No local - Windows or Windows Docker execution is claimed until that manual job is recorded. -- **Validation status.** Deterministic Phase A gates, backend **434/434** plus TypeScript, - frontend **386/386** plus TypeScript, and harness **862 passed / 5 L2 deselected** are green. - Review round 1 ran each Docker smoke exactly once without retry. Unified (`103.86s`) and - update-only (`46.45s`) passed build/start, core/Pi/registry/persistence setup and the stopped - candidate preflight, but `tht` stopped before mutation at its active-session inventory gate. - Round 2 replaces presence-only fixture checks with generated Compose renders plus the production - workspace resolver; this found and fixed missing explicit direct transport selections. The - clean-server preflight now atomically initializes the three hidden Pi-agent targets under the - writable parent bind while protected/tracked sources remain separate read-only mounts. Clean - empty-root render/setup and wrong-service/value/mount mutations are green. The corrected server - one-shot built and started both healthy services from an empty Pi-state root, then stopped at an - incorrectly addressed authenticated frontend hop. Fix round 3 adds the exact fourth private - non-admin claim and proves its nginx/backend transformation in a focused auth test. It also - centralizes schema-v2 registry descriptor resolution and secret-safe runtime rendering in - `ThtRunner`, preserving canonical revision identity and durable session roots for inventory, - create/resume/show, SQL, and Pi calls. The fresh update-only one-shot now passes mutation, - automatic `rolled_back` compensation, exact prior-image restoration, unchanged registry head - and mount identities, all four persistence sentinels, post-rollback doctor/workspace checks, - and exact labeled-resource cleanup. The one authorized server invocation was blocked at its - first Docker readiness call by the execution sandbox's socket permission before any Compose - resource could be created, so authenticated workspace/fail-closed session behavior remains an - explicit release gate. Native Windows PowerShell/Docker execution also remains pending. +The native host CLI `tht` is the operator surface. Workspace preprocessing runs through: -### Historical snapshot — Portable deployment decoupling (superseded 2026-08-08) - -- **Mandatory stack.** The supported Compose stack is exactly `frontend` plus `core`; use the - base file with `deploy/compose.local.yaml`, or with `deploy/compose.server.yaml` plus the - required public-server session overlay. `run-stack.sh` - invokes the base+local Compose command and the core image provides Pi, so no host Pi binary is - part of the launch contract. -- **External boundaries.** DWH, vector DB, embedding, LLM, and reverse-proxy services are - external configurable endpoints even when deployed on the same infrastructure. The two - superseded PSD/portal deployment overlays were removed. Workspace descriptors and migration - utilities remain separate from deployment runtime configuration. -- **Legacy PSD deployment ruling.** The PSD bootstrap was deleted because it generated the - retired overlay and was therefore deployment machinery, not a data migration utility. Its - remaining live contract checks were renamed for the generic local Compose profile. The coupling - gate rejects stale active deployment filenames and content while deliberately excluding - historical plans/specs, canonical workspace descriptors, and non-runtime migration helpers. -- **Fresh provider and secret contract.** Local, server, and standalone development mount the - protected Pi auth JSON plus tracked declarative model/settings files read-only under - `/home/thoth/.pi/agent`. The existing strict application bundle is a core-only Docker secret at - `/run/secrets/thothii.secrets`; operator env files contain only its absolute source path. - Provider readiness is exercised from a fresh Compose volume through model listing, configuration, - and sanitized credential status. -- **Install and scan closure.** Superseded copied one-service installation examples and the - provider-owned-network test are retired. Active manuals use the canonical base plus local/server - and optional overrides, while the category-based coupling scan covers runtime, Docker smoke, - install, operator, and positive deployment-test contracts and propagates scanner errors. - -### Historical snapshot — Portable Git workspace registry, pre-schema-v3 (superseded 2026-08-08) - -- **Source of truth and scope.** The canonical workspace repository is a generic Git remote, - configured only by `THT_WORKSPACE_GIT_REMOTE` and `THT_WORKSPACE_GIT_BRANCH` (there is no - committed PSD/Chirone remote or branch default). Both a local Docker installation and a server - persist its checkout, validated snapshots, state, and locks at `/data/workspace-registry`. - Connector endpoints, transport choices, and secret-file paths remain local bindings; secret - contents are never stored in Git, API responses, browser storage, diagnostics, or bundles. -- **Migration and session safety.** Schema-v2 descriptors are operational; legacy descriptors are - visible as `migration_required` until migrated by the documented operator workflow. New sessions - acquire a persistent revision lease before readiness and persist workspace ID plus immutable Git - revision. Retention hands that lease off only after an authoritative scan observes the manifest, - so a stale concurrent scan cannot prune the pinned snapshot. Resume resolves that historical - snapshot, while retention preserves every revision referenced by an open, closed, or failed - unarchived manifest. - Reconciliation runs only with a complete local installation list or an administrator's complete - server list, never from a remote user's partial view. -- **SSH connector boundary.** The current OpenSSH forward is owned by one bounded diagnostic and is - always cleaned up afterward. DWH/vector `ssh_tunnel` bindings therefore return - `workspace_not_activatable`, and new-session creation rejects them before persistence. Direct and - REST runtime connectors remain supported; Git remote access over SSH is unaffected. -- **Operator manuals.** Follow [the local manual](docs/install/local-workspace-registry.md) for - macOS/Windows/Linux Docker Desktop deployment and [the server manual](docs/install/server-workspace-registry.md) - for Gitea-compatible remotes, reverse proxy, migration, backup, and recovery. The release - workflow is Git review/push → installation pull → validate → local diagnostic test → browser-local - workspace/model/reasoning selection → revision-pinned session. -- **Verification recorded for this source branch.** `git diff --check` passed; backend Vitest - **371/371** and TypeScript passed; frontend Vitest **398/398** and TypeScript passed; the - harness document regression passed **10/10**. `./scripts/workspace-registry-smoke.sh` and the - executable installation-manual fixture verifier passed with Docker. A final unrestricted full - harness run remains a release command for the deployment environment; the earlier local - long-running harness run was intentionally cancelled before it produced a final result. - -### Historical snapshot — Session summary redesign (2026-07-23) - -- Session documents are projected at read time in outcome-first order: original question, - final SQL, persisted data preview, revised question, assumptions, one memory list, then - remaining technical documents. This applies to existing filesystem and repository-backed - sessions without rewriting their artifacts. -- Final SQL has an always-visible clipboard action. All prose, including original/revised - questions, assumptions, memory content, and remaining decisions, renders as Markdown. -- Memories are shown once, approved before declined. The generic decision list suppresses - memory ledger records plus `phase_approved`, `phase_auto_approved`, `table_approved`, - `table_promoted`, and `column_promoted`. -- The session summary has its own accessible pointer/keyboard resize separator, persists its - width independently from Model activity, reaches 50% when space permits, and preserves a - 512 px right-side minimum on narrower desktop layouts. -- Verification: harness **861 passed / 5 L2 deselected**, Pi gate **163/163**, full frontend - suite, TypeScript check, production build, Ruff, and `git diff --check` all passed. A live - pre-existing session returned the new canonical order and none of the suppressed decision - labels. Compose rebuilt and force-recreated both services; core image - `sha256:3566d1258b956f8ca96d3b5ff8f625503247a7fd3b0dffe77020ae03956403d8` - is healthy and frontend image - `sha256:8311ca1308b459ece7236bf143da7b1a226ff4082fed924e1b5a207c24b6ca29` - is running. Frontend and `/api/health` both returned HTTP 200. - -### Historical snapshot — Local Pi user auth + startup failure handling (2026-07-21) - -- The PSD Docker profile now bind-mounts the configurable host `PI_AUTH_FILE` read-only at - `/home/thoth/.pi/agent/auth.json`; on this Mac it resolves to the real user profile - `/Users/mp/.pi/agent/auth.json`. The container keeps its correct Linux identity - `HOME=/home/thoth` while Pi sees the user's independent `deepseek` and `zai` credentials. -- `deploy/pi/settings.json` is the non-secret model policy and exposes, in order, - `zai/glm-5.2`, `deepseek/deepseek-v4-flash`, `deepseek/deepseek-v4-pro`, and - `aritmolab/qwen3.6-35b-a3b`. The core image is aligned to Pi 0.80.3. -- New-session creation now validates the saved provider/model against Pi before persistence; - unavailable selections return sanitized `503 model_unavailable` without creating a manifest. - A synchronous runtime-construction failure after persistence marks that session `failed` and - returns the fixed startup-recovery message instead of leaving an ambiguous `open` session. -- Verification: backend 235/235, TypeScript clean, dedicated Compose auth/model contract green - with a demonstrated RED→GREEN cycle. Rebuilt core image - `sha256:a8b4dd9f016c2335e4da897073bc6d5bdf1e8ce9b60dcfcca171b6677f563228` - is healthy; live `/models` returned all four models; a real `deepseek-v4-pro` smoke reached its - first reviewer gate, deleted only its own session, and restored the exact prior settings. -- Deleted the three explicitly approved incomplete DeepSeek attempts: - `a390c8b8-0a91-4a37-967b-ce7ff9be9797`, `a2f974b2-4c48-4967-b4b6-afdbc2b2d541`, and - `f66e1959-3c71-4b10-8aa1-606992046b7e` (API delete 204, subsequent lookup 404 for each). - -### Historical snapshot — User-owned sessions cutover (2026-07-16) - -- **Target contract:** the public server runs `AUTH_MODE=upstream` with Task 4 portal identity - forwarding and Task 5 principal enforcement deployed together. Its session source of truth is - direct TLS-verified PostgreSQL `thoth_sessions`; local development remains loopback-only with - filesystem sessions under `THT_HOME`. The core never receives the migrator credential. -- **Deployment material:** copy `deploy/compose.session-server.yaml.example` and - `deploy/workspaces/server-sessions.yaml.example` into reviewed, untracked operator files. The - runtime password, migrator password, and CA are three separate Docker secret mounts; server - startup rejects public/local storage and incomplete server DB/TLS configuration. -- **Readiness behavior:** `/health` remains the unauthenticated process liveness endpoint. Any - route requiring unavailable session/preferences storage returns fixed HTTP 503 before starting - Pi; this is intentional and must not be hidden by changing liveness to a database check. -- **Manual cutover only:** schedule maintenance, drain Pi work, run the one-shot migrator and - require `pending=[]` and `drifted=[]`, then replace core and perform an authenticated storage - smoke. Archive/checksum the three reviewed legacy filesystem session directories before deleting - exactly those three with `docker/cutover-legacy-sessions.sh --delete`; no deletion has been run - from this repository task. Do not import their untrusted ownership. -- **Rollback:** PostgreSQL remains the single source of truth. Revert only to a compatible fixed - release; never re-enable filesystem persistence, restore the archive into production, or - dual-write during rollback. - -### Historical snapshot — Docker locale deployment, Profile A (superseded 2026-08-05) - -ThothII gira in Docker sul server co-locato, **embedded nel portale omics_portal** a `https://aritmolab.policlinicosandonato.it/datamart-builder` (backend invisibile, tutto same-origin via nginx del portale). - -- **2 container** su `compose.yaml`: `thothii-core` (Fastify + harness tht + Pi) + `thothii-frontend` (Vite + nginx-unprivileged). Rete `omics_portal_omics_network` (external) con alias `thothii-core`/`thothii-frontend`. -- **DB**: Postgres diretto `:5438` (stessa istanza: schema `datawarehouse` 163 tabelle + `vectors` pgvector). Ruoli dedicati `thoth_dwh_reader` (read-only) + `thoth_vector_rw` (read+write). Embeddings: Ollama `:11434`. -- **Secrets**: `deploy/thothii.env` (env_file, gitignored) + `THT_MODEL_API_KEY_FILE` (key modello, file 0600 — meccanismo provider-credentials di Codex) + bind-mount `~/.pi` (pi-config). -- **Backend**: merge di `codex/portable-deployment` (secret-bundle, provider-credentials, auth `upstream`, security hardening, CI multiarch). Setup Docker MIO tenuto (il modello Docker-secrets di Codex è in `deploy/` come alternativa inerte). -- **Portale** (repo `omics_portal`, branch `agent/patient-capabilities-datamart-ui`): `nginx.conf` rotte `/datamart-builder/api`+`/assets` + `auth_request`, template `datamart_builder.html` (mount `
` + tag `{% vite_assets %}`), vista `datamart_builder_api_auth`. Auth: `authentik Admins` bypass; utenti normali necessitano gruppo `omics-datamart-builder`. -- **Fix load-bearing**: `configPath` da `THT_CONFIG` (route senza workspace), `vite_assets` `mark_safe` (SPA bianca), `COPY harness/`+`cp workflow.yaml`+`pip install .` (pip 26 / tht module-relative), entrypoint `server` case. -- **Standalone/dev**: `docker-compose.dev.yml` (rete propria, porte host 8787/8090) + `scripts/docker-smoke.sh`. -- Piano dettagliato: `docs/superpowers/plans/2026-07-12-local-docker-deploy-implementation.md`. - -### Archived snapshot — Runtime incident fixes (2026-07-13) - -- The bind-mounted Pi profile came from host paths and did not trust `/app/harness`. - Pi 0.80 consequently loaded **zero** project extensions, prompts and skills, silently - sending `/nuova-domanda`/`/riprendi-sessione` to the model as plain text. The core - entrypoint now idempotently adds only `/app/harness` to the persistent - `/home/thoth/.pi/agent/trust.json`, preserving all existing decisions. -- The gate embeds the canonical `tht-sessione/SKILL.md` in the one-shot kickoff system - prompt and explicitly prohibits repository discovery. A live RPC `get_commands` must - show `torna`, `nuova-domanda`, `riprendi-sessione`, and `skill:tht-sessione` after deploy. -- Workspace identity is derived from the resolved config path, so - `config/tht.yaml -> workspaces/local.yaml` matches DWH artifact ownership (`local`). -- Direct pgvector now discovers the actual namespaces of the `vector` type and cosine - operator from PostgreSQL catalogs. This supports server layout `vectors.*` tables with - the extension installed in `public`. -- Live verification: session `2026-07-13-074712-dammi-la-lista-dei-pazienti-che-haoo-fat` - resumed directly at F1, ran `tht session show`, and completed `tht search pack` - (12 tables, 0 evidence, 2 solved) without repository exploration or adapter errors. - -### Archived snapshot — Workflow/UI regression fixes (2026-07-14) - -- **F1 Model Activity restored.** Session create/resume now preserves configured/persisted - thinking instead of forcing `off`. Pi's nested `thinking_delta` is bridged to a dedicated - named SSE `activity_delta`; EventSource subscribes to that name and the panel keeps it separate - from final assistant text. Reasoning remains in-memory and is not persisted to session artifacts. -- **F3 rewrite confirmation remains bypassed.** `rewrite_question` records approval and advances - automatically without a reviewer widget. The repeated prompt came from old running containers: - images had been rebuilt but services had not been recreated. -- **Join review is read-only and complete-set safe.** Join-only proposals render informational - cards with only `Continue` and `Other — specify`. Continue requires the exact complete id set; - all joins are persisted together by `decision add-join-set`, using an atomic ledger replacement - under a per-session cross-process writer lock. Other persists none of the rejected proposal. -- **CTE presentation fixed.** F6 CTE cards now structure purpose, rationale, tables, filters, keys, - and output columns with responsive wrapping/alignment. The Horizontal/Vertical switch is hidden - for a single SQL block (the per-CTE view), because it only affects multi-block layouts. -- **Latest render failure diagnosed and hardened.** Session - `2026-07-14-115847-estrai-i-pazienti-che-hanno-fatto-un-abl` sent an object in - `open_questions`, which React cannot render as a child. The v2 gate now enforces - `open_questions?: string[]`; the frontend also safely normalizes legacy malformed payloads. -- **Verification/deploy:** Python harness 798 passed / 5 L2 deselected; gate JS 126; backend 143; - frontend 250; TypeScript/build gates green. Compose rebuilt and force-recreated both services. - Running image ids: core `sha256:55acef2f12151ea97144c2f5e9164d63f2ca734bc2746fef553df94849e3fb3f`; - frontend `sha256:1043f79392420149655cc63d70461e2ca2005b2290a1e3e21dcf845ec3bd1c81`. - -### Archived snapshot — Pi-enabled model selector (2026-07-14) - -- **Pi is the allowlist authority.** `/models` reads the mounted Pi `enabledModels`, intersects - it with models currently available from Pi, and preserves the configured order. Enumeration - does not require `PI_PROVIDER`, does not inject generic/provider credentials, and fails closed - for missing or malformed scope. -- **Live scope:** exactly `deepseek/deepseek-v4-flash`, `zai/glm-5.2`, and - `local-qwen/qwen3.6-35b-a3b`. The live endpoint returned those three composite IDs once each and - in that order; `zai/glm-5v-turbo` and all other authenticated Pi models are hidden. -- **Validation/process smoke:** live settings updates returned 200 for DeepSeek Flash and local - Qwen, while hidden GLM-5V returned 400; a post-restore equality check confirmed the original app - settings were restored. The real `PiProcessManager` configure path succeeded for DeepSeek and - local Qwen without sending a prompt or starting a DWH operation; local Qwen required no hosted - provider key. Unknown and compound providers remain fail-closed in the verified backend suite. -- **Verification/deploy (`2026-07-14T19:24:22+02:00`):** backend **154/154** and frontend - **251/251** passed; both TypeScript gates and `git diff --check` were green. Compose built and - force-recreated only `core`; container start was `2026-07-14T17:24:01.992971976Z`, health was - `healthy`, and sanitized post-recreate logs contained only the backend listen line. Rebuilt image - ID and running container image ID both equal - `sha256:577f99754fd0731251c8ddd8608b1b8baee09d02fad66c759b23f8221083e676`. - -### Archived snapshot — Qwen connectivity + state-aware Resume recovery (2026-07-14) - -- **Pi turns have an explicit lifecycle.** The bridge tracks `idle`, `running`, `waiting`, - and `failed`; a reviewer gate is `waiting`, responses/steering return to `running`, and an - assistant provider error or unexpected Pi child exit becomes `failed`. Provider error details - are never forwarded to the client; the UI receives a fixed sanitized recovery message. -- **Resume preserves only active work.** `running`/`waiting` runtimes return as already active. - Every validated cold path—including recovery after a child has already exited—clears stale SSE - state before reopening and restarts from persisted provider/model/thinking with - `/riprendi-sessione`; `idle`/`failed` runtimes are torn down at that point. Failed validation does - not detach the existing stream. A successful Resume of the currently selected session also - closes and recreates its EventSource, so the replacement runtime cannot be left behind an old - same-ID stream. -- **Private Qwen routing is live.** Core is attached to both `omics_portal_omics_network` and - external `localllm_default`; frontend remains only on the portal network. The mounted Pi profile - resolves `local-qwen/qwen3.6-35b-a3b` at the sanitized base URL - `http://localllm-vllm:8000/v1`. A direct probe from core verified the model catalog and received - a non-empty real chat completion. -- **Verification/deploy (`2026-07-14T21:41:52+02:00`):** backend **168/168** and frontend - **256/256** passed; both TypeScript gates, both production builds, the Qwen Compose network - contract, and `git diff --check` exited 0. The initial deployment built and force-recreated - `core` and `frontend`; after the final crash-recovery review, only the affected `core` image was - rebuilt and force-recreated with no active Pi session. Core is healthy and its post-recreate - Qwen catalog/completion probe succeeded. Built and running image IDs match: core - `sha256:9867c2fa002b6f117da9a1d02c73b47bfe0372b8c1c137a5174c3e7ecb1e1db1`, frontend - `sha256:5a47f81bc887423e05cef8fd3feb075aa600cb33217247186124f60ca5a3005b`. - The frontend entry hash changed, so only `omics_portal-web-1` was restarted to invalidate its - indefinite Vite-manifest cache. The application-level Qwen smoke reached its first - `ui_request`, persisted the expected provider/model, deleted only its uniquely named smoke - session, restored the exact saved settings object, and left no smoke session or Pi runtime. The - supplied probe's success path left its keep-alive SSE reader open, so only that probe process was - terminated (exit 143), without touching backend, Pi, or unrelated runtimes. The same smoke then - exited 0 with `controller.abort()` in cleanup, preserving the gate, session cleanup, and exact - settings-restoration evidence. - -### Archived snapshot — Complete activity timeline + CTE spacing (2026-07-15) - -- **Model activity is complete from F1.** The left panel now records the submitted prompt before - session creation completes, then projects thinking, assistant output, sanitized tool lifecycle, - reviewer gates, status, and turn lifecycle in chronological order. It remains in-memory only; - closing and reopening the panel does not discard it, while Resume intentionally starts a fresh - live timeline. -- **Tool activity is a narrow public contract.** Only call id, tool name, and - `running`/`completed`/`failed` status cross Pi → backend → SSE. Updates, arguments, results, - commands, output, credentials, and raw errors stay server-side. Resume and SSE replay were also - hardened for process restarts, concurrent lifecycle requests, stale callbacks, cursor reset, and - multi-client reconnects. -- **CTE plan layout uses real Tailwind 3 spacing.** Shared cards use concrete 16 px default / 12 px - compact padding utilities; F6 CTE headers and content use responsive 16/20 px edge padding. - Semantic ordered steps, single-boundary divided filter/table lists, long-value wrapping, and a - plain top-divider rationale preserve the artifact content while improving scanability. -- **Final verification/deploy (`2026-07-15T02:18:54+02:00`, HEAD `56d73d2`):** backend - **204/204** and frontend **285/285** passed; both TypeScript gates and production builds exited - 0, and `git diff --check` was clean. The final lifecycle fixes make restarted-hub cursor replay - generation-aware and mutate frontend delete/resume state only for sessions actually deleted. - Compose rebuilt and force-recreated `core` and `frontend`; built and running image ids match: - core `sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` - (`healthy`), frontend - `sha256:dd7ef721b661b57b8c5422c47088e1a9a2e9821af021541cab372c9892fbd638` - (`running`). The entry changed from `index-DcZviApa.js` to `index-CuIt1NQg.js`, so only - `omics_portal-web-1` was restarted to refresh its indefinite manifest cache. -- **Final live local-Qwen smoke:** session - `2026-07-15-001922-final-no-thinking-activity-smoke-2026-07` observed **0** - `activity_delta` events while receiving **5** strictly allowlisted tool lifecycle events and the - first reviewer gate (`bash` running/completed and `reviewer_select` running). No forbidden tool - field crossed SSE. Cleanup closed with 200, deleted only that session with 204, restored the - exact settings object, and left no Pi runtime or smoke session. - -### Archived snapshot — Filtered Model activity projection (2026-07-15) - -- **Resolved contract.** `activityLog` still folds the complete in-memory prompt, thinking, - assistant, sanitized tool, reviewer-gate, status, and turn-lifecycle history. The left panel now - applies a default-deny rendering boundary and shows only prompt, thinking, status, and gate; - assistant text remains available to the central transcript, while tool, lifecycle, and unknown - future activity kinds do not render or move the panel scroll. -- **Frontend-only verification/deploy (`2026-07-15T03:44:02+02:00`, source HEAD `7208599`):** - frontend **287/287** passed; `npx tsc -b`, `npm run build`, and `git diff --check` exited 0. - Compose rebuilt and force-recreated only `frontend`; its image changed from - `sha256:dd7ef721b661b57b8c5422c47088e1a9a2e9821af021541cab372c9892fbd638` to - `sha256:75fc0b7786c75ded1b488e9fad3232aa061c82d89140c67ad62b5285954d1d36`, with container start - `2026-07-15T01:42:45.582760612Z`. The active Vite entry changed from - `index-CuIt1NQg.js` to `index-BdZpFO7j.js`, so only `omics_portal-web-1` was restarted at - `2026-07-15T01:42:53.715330158Z`; core retained image - `sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and start - `2026-07-15T00:18:37.040145636Z`. -- **Real local-Qwen no-COT smoke:** session - `2026-07-15-014328-activity-filter-smoke-2026-07-15t01-43-2` reached its first reviewer gate - with **0** `activity_delta` events and **3** tool lifecycle events, each containing exactly the - four public fields. The probe accepted the close response, deleted only that session with 204, - restored the exact saved settings object, confirmed the session absent, and left no Pi runtime. - -### Archived snapshot — Central activity log + compact CTE density (2026-07-15) - -- **Resolved UI contract.** The central working body now renders every chronological non-blank - assistant transcript line in one bounded accessible log, without user-entry echoes, - timer/spinner labels, or step messages. The left Model activity panel is a default-deny - projection of only thinking and status, while the complete raw activity fold and the existing - reviewer widgets, artifacts, and workflow state remain unchanged. F6 CTE headers, content, - table rows, and filter rows use 8 px vertical padding with 12 px lateral padding below `sm` and - 16 px from `sm` upward; divider top padding is 8 px. The final review amendment keeps historical - log rows at the full muted-foreground token so their normal-size text retains AA contrast. -- **Source verification (source HEAD - `09f9bdffe582ff3c66845ca219f60c11472a550a`).** Frontend tests passed **292/292** across - **43/43** files. `npx tsc -b`, `npm run build`, the Impeccable layout detector, and - `git diff --check` all exited 0; the detector returned `[]`. -- **Frontend-only deployment.** The pre-deploy frontend was image - `sha256:8c9aaee6453b26c69e4057d3f9e53aa791d449b88ba4a0669a25cd442667d582`, started - `2026-07-15T10:17:16.919166255Z`, serving `index-CveSyban.js`. Compose built and - force-recreated only `frontend` with `--no-deps`; the final running frontend is image - `sha256:39dc47d81abcb13466218500476fbb78d1be66a424f4293470437700c848c26d`, started - `2026-07-15T10:30:30.869532521Z`, serving `index-CihtpQJV.js`. Because the entry changed, - exactly `omics_portal-web-1` was restarted: it retained image - `sha256:4580cf2bc85f3656ee515996c24ed02d9096d24ed0d8afe384a2f8bd8cf4bac9` and moved from - start `2026-07-15T10:17:36.151611451Z` to `2026-07-15T10:30:46.369205604Z`. -- **Isolation and final state.** Core remained `running`/`healthy` with exactly its original image - `sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and start - `2026-07-15T00:18:37.040145636Z`; frontend and portal were `running` with no container - healthcheck. Pre/post `docker top` showed only core's supervisor and backend server, so no - unrelated Pi runtime existed to disturb. The count-only frontend sensitive/error pattern scan - was **0**. No live model smoke was run, and settings and sessions were intentionally untouched. - -### Archived snapshot — Resizable activity split + compact CTE rows (2026-07-15) - -- **Resolved UI contract.** `activityLog` remains the complete in-memory chronological fold. The - left Model activity panel default-denies every kind except prompt, thinking, and assistant, - labels those entries Question, Reasoning, and Response in source order, and hides status, tool, - gate, lifecycle, and unknown kinds. The desktop panel is pointer/keyboard resizable from 288–576 - px while preserving 512 px centrally, persists its global width in localStorage, and becomes an - overlay drawer below `lg` or whenever the measured app shell is narrower than 800 px. F6 CTE - cards retain their semantic structure and responsive grids; - lateral padding is 12/16 px, header/content edge padding is 8 px, internal section gaps are 12 - px, heading/divider spacing is 4 px, and table/filter rows use 4 px vertical padding with compact - line heights. -- **Source verification (`2026-07-15T15:22:17+02:00`, source HEAD - `f1af1f909b387ae12a10f1b534bf6a529ea42505`).** Frontend tests passed **295/295** across - **44/44** files. `npx tsc -b`, `npm run build`, and `git diff --check` exited 0; the Impeccable - layout detector returned `[]`. The local source build emitted Vite entry - `assets/index-PlvQqhNG.js`. -- **Frontend-only deployment.** The pre-deploy frontend was image - `sha256:39dc47d81abcb13466218500476fbb78d1be66a424f4293470437700c848c26d`, started - `2026-07-15T10:30:30.869532521Z`, serving `index-CihtpQJV.js`. Compose built and force-recreated - only `frontend` with `--no-deps`; the final running frontend is image - `sha256:0947211373784a860c7507d03c0fcf1f901ac3d1547cd0aa62b914d158f15bd5`, started - `2026-07-15T13:21:08.29613952Z`, serving `index-Dn7T524a.js`. Because the entry changed, exactly - `omics_portal-web-1` was restarted: it retained image - `sha256:4580cf2bc85f3656ee515996c24ed02d9096d24ed0d8afe384a2f8bd8cf4bac9` and moved from - start `2026-07-15T10:30:46.369205604Z` to `2026-07-15T13:21:21.968808414Z`. -- **Isolation and final state.** Core remained `running`/`healthy` with exactly its original image - `sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and start - `2026-07-15T00:18:37.040145636Z`; frontend and portal were `running` with no container - healthcheck. Pre/post `docker top` showed only core's supervisor and backend server, so no - unrelated Pi process existed and no Pi process was stopped or steered. The count-only frontend - sensitive/error pattern scan was **0**. No live model smoke was run; settings and sessions were - intentionally untouched. - -### Archived snapshot — Final activity-split fix (2026-07-15) - -- **Source and verification (`2026-07-15T15:56:57+02:00`).** Deployed source commit - `1f540fcb78ac9e552e56a21e47edf66e9872b323` (`1f540fc`). Frontend Vitest passed **298/298** - tests across **44/44** files; `npx tsc -b` and `npm run build` exited 0. The Impeccable detector - scoped to AppShell, ModelActivityPanel, index.css, and CtePlanViewer returned `[]`; `git diff - --check` exited 0. -- **Frontend-only deployment.** Compose built and force-recreated only `frontend` with `--no-deps`. - The frontend image changed from - `sha256:0947211373784a860c7507d03c0fcf1f901ac3d1547cd0aa62b914d158f15bd5` to - `sha256:6e14f55092b7e3aca9a396220394ae484147674d81b051771e394e59b73b1c88`; its active Vite entry - changed from `index-Dn7T524a.js` to `index-BIznZeLH.js`. Therefore exactly - `omics_portal-web-1` was restarted to refresh its manifest cache; it retained image - `sha256:4580cf2bc85f3656ee515996c24ed02d9096d24ed0d8afe384a2f8bd8cf4bac9` and started at - `2026-07-15T13:56:32.786693915Z`. -- **Isolation and final state.** Core retained image - `sha256:edd8f19ef269ee6f45ecaf464ba4053460d94378f9cdc967d2dbcb386f599147` and exact original - start `2026-07-15T00:18:37.040145636Z`, remaining `running`/`healthy`. Final frontend and portal - states are `running` (no healthcheck). Pre/post core process tables contained only the supervisor - and backend server, so Pi was preserved and no Pi process was stopped or steered. The count-only - frontend sensitive/error-pattern scan was **0**. No model smoke was run; settings and sessions - were intentionally untouched. - -## What ThothII is - -A **human-in-the-loop datamart builder**: it turns a natural-language question into -validated SQL (and optionally a dbt datamart) through a deterministic **8-phase -NL→SQL workflow**, where the model *proposes* and a human *reviewer decides* at gates. -The UI is meant to embed inside the Omics Portal (GSD design system) and is **English**. - -## Architecture — three layers - -``` -frontend (React, :5173) → backend (Fastify, :8787) → pi --mode rpc → tht / harness → DWH (read-only) +```sh +tht --installation /absolute/path/thothii-installation.yaml workspace preprocess evidence +tht --installation /absolute/path/thothii-installation.yaml workspace preprocess dwh ``` -- **harness/** — the Pi layer. A deterministic Python CLI **`tht`** + a Pi gate extension - (`.pi/extensions/tht-gate.js`) that runs the 8-phase workflow and emits/consumes - widget-descriptor JSON. **Owns all persistence.** Workflow truth is `harness/workflow.yaml`; - orchestration rules are `harness/.pi/skills/tht-sessione/SKILL.md`. -- **backend/** — Fastify + TypeScript. A **thin bridge**: proxies REST routes to the `tht` - CLI (`ThtRunner`), manages Pi processes (`PiProcessManager`, one child per session), - bridges Pi RPC events to SSE (`SessionBridge` + `SseHub`). No application database. -- **frontend/** — React 18 + base-ui + Tailwind + TanStack Query + Zustand. Chat-style - shell (`src/shell/AppShell.tsx`); the live transcript is rebuilt in-memory from the SSE - stream (`src/store/sessionStore.ts`), **not persisted**. +These commands use the profile-gated `workspace-maintenance` service. The former standalone +preprocessing Compose fixtures are retired. -### Persistence model (the load-bearing premise) -There is **no verbatim chat store**. Each workflow phase persists its own document into the -session directory, and that **IS** the persistence. A session = a directory under the -workspace's `sessions/` path containing `session_manifest.yaml` + phase artifacts -(`question.md`, `schema_linking.json`, `cte_plan.json`, `sql_final.sql`, -`validation_report.md`, `review_decisions.jsonl`, …). A fresh Pi process resumes by reading -`tht session show ` + the on-disk artifacts — never by replaying chat. +Workspace descriptors use schema v3. For PSD, workspace content and runtime roots point to the +separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside +Git and are supplied only through installation-local protected files. -## The 8 phases (harness/workflow.yaml) -F1 chiarimento · F2 memoria · F3 riscrittura (`question.md`) · F4 schema_linking -(`schema_linking.json`) · F5 sintesi · F6 cte (`cte_plan.json`, `cte_tests.json`) · -F7 sql_finale (`sql_final.sql`) · F8 datamart. Current phase is a fold over the decision -ledger (`harness/tht/phase.py`); statuses: `open` / `closed` / `finalized`. +## Active deployment work and manual gates -## How to run +### PSD server deployment program -**Full stack (real Pi + DWH):** from project root, -```bash -./scripts/run-stack.sh -# Opens frontend: http://localhost:5173 (proxies backend :8787) -``` -**Prereqs:** VPN on; `pi` on PATH (with a configured model, e.g., `pi model set claude-fable-5`); -`harness/.env` populated (see `.env.example`); `harness/config/tht.yaml` → workspace (psd recommended for testing). -All three layers' deps installed (`npm install` in each, `python -m venv + pip install -e ".[dev]"` in harness). +The approved design and executable entry point are: -**Individual dev:** -- backend: `cd backend && npm run dev` (tsx watch; env: `PORT`, `THT_HARNESS_DIR`, `THT_BIN`, `PI_BIN`, `AUTH_MODE`) -- frontend: `cd frontend && npm run dev` (Vite; `VITE_BACKEND_URL` → backend) -- harness install: `cd harness && python -m venv .venv && pip install -e ".[dev]"` → `tht` on PATH +- `docs/plans/2026-08-20-psd-server-deployment-program-design.md` +- `docs/plans/2026-08-20-psd-server-deployment-program.md` +- `docs/plans/2026-08-20-psd-server-survey.md` +- `docs/plans/2026-08-20-psd-server-project-a-standalone.md` +- `docs/plans/2026-08-20-psd-server-project-b-authentik.md` -## How to test (latest TS gates green 2026-07-15: backend 204 / frontend 292 (43 files); harness 798 / gate JS 126 last recorded 2026-07-14) -- harness: `cd harness && .venv/bin/pytest -q` (5 L2/real-DB tests are deselected by default) -- backend: `cd backend && npx vitest run` · typecheck `npx tsc --noEmit -p .` -- frontend: `cd frontend && npx vitest run` · typecheck `npx tsc -b` · e2e `npm run e2e` (Playwright) +Last recorded state: -## Config & workspaces -- Workspaces: `harness/workspaces/*.yaml` (`psd`, `tht-test`, `tht.example`). A workspace sets - the DB target and the **absolute** `paths.sessions/artifacts/indexes` (psd → a *separate* - repo `tht-workspace-psd/`, NOT committed here). -- Secrets live ONLY in `harness/.env` (gitignored; `THT_*` — DB, DWH REST, vector, SSL CA…). - See `harness/.env.example` for the variable list. -- App settings (global): `{ workspace, provider, model, thinking }`, persisted via harness - preferences (`tht session preferences get/set` → the configured session repository — - filesystem or Postgres in server mode). `backend/data/settings.json` (gitignored) remains - only the file fallback for injected runners/tests. The "New session" form is question-only; - these settings supply the rest. +- survey: `SURVEY_NO_GO`; +- Project A: `BLOCKED_BY_SURVEY_AND_MUTATION_GATE`; +- Project B: `BLOCKED_BY_PROJECT_A_AND_PRE_B_GATE`. -## Efficiency levers (NL→SQL workflow optimization, 2026-07-08) +The deployment is a clean replacement: legacy sessions, indexes, and application configuration +are not migration inputs. The existing stack remains intact until its documented mutation and +rollback gates are explicitly approved. Shared Omics/LocalLLM networks, ETL Evidence, DWH, +`dwh-auth`, Supabase, Authentik, Superset, and Aritmolab are outside cleanup scope. -Three deployed optimizations target model thinking time (the dominant cost, ~220s in F1 alone): +Human acceptance guides and sanitized report templates live under `docs/testing/` and +`docs/testing/evidence/`. The remediation checklist is +`docs/operations/psd-server-survey-remediation-checklist.md`. -1. **Join-graph via FK annotations + `tht schema suggest-fks`** - - DWH has no FK constraints declared. Annotations file (`tht-workspace-*/artifacts/mschema/annotations.yaml`) now stores curated logical FKs. - - Three ranking rules: mine from approved SQL (highest confidence), heuristics (`*_time_key → dim_time.day_key`), same-name PK discovery with `--assume` flag for disambiguity. - - **Psd workspace:** 228 FK suggestions already generated (139 tables); `tht schema suggest-fks --from-sql --assume cod_paz=dim_patient` for updates. - - **Activation:** automatic. The mschema renderer populates the `【Foreign keys】` section. F4 in SKILL.md now reads FK joins from there instead of the model re-deriving them. +### Authentication -2. **Context-pack consolidation at F1 kickoff (`tht search pack`)** - - Single embedding of the question, reused for schema + evidence + solved-question searches. - - Command: `tht search pack "" --session ` → `sessions//retrieval_pack.md` (tabelle candidate, relevant evidence, solved exemplars). - - Graceful degradation: if Ollama or vector store unreachable (no VPN), sections are empty but exit 0 — session continues with live searches. - - **Activation:** automatic at next session. SKILL.md F1 now prescribes as first call; reduces exploratory turns. +The local/OIDC authentication remediation passed its automated review on 2026-08-18. Release and +PSD mutation gates remain governed by: -3. **Phase-summary recap auto-construction from session ledger** - - `tht session show --json` includes the full decisions ledger; `tht phase meta --json` exports decision types per phase. - - Gate appends deterministic `【Decisioni registrate in questa fase】` section to v2 phase-summary artifacts. - - Model authors only `summary` + `checks`; the gate fills the recap table from persisted state → exact by construction. - - **Activation:** automatic at next session and Pi restart. SKILL.md Disciplina 6 updated: model keeps output brief, gate enriches from catalog + ledger. +- `docs/architecture/authentication.md`; +- `docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md`; +- `docs/operations/psd-dwh-auth-rollout.md`; +- `docs/testing/authentication-manual-acceptance.md`. -**Tests:** 358 Python + 111 JS gate, all pass. L2 (live DWH) verification on psd workspace recommended when time permits. +Do not infer authorization for server, Nginx, Authentik, database, credential, or cutover changes +from an automated PASS. -## Conventions & contracts (don't relearn the hard way) -- **`-c`/`--config` is a PER-COMMAND option in `tht`** — append it AFTER the subcommand, - never globally (`ThtRunner.buildArgv` handles this). -- **`--json` output must be pristine** (only valid JSON on stdout). -- **UI strings are English.** Document *content* stays in the workspace language (Italian - for psd) because it's the real data; only chrome/labels are English. -- **Settings are global**, not per-question. -- TDD throughout; tests assert real behavior, not mocks. Frequent, scoped commits. -- Global user rules (`~/.claude/CLAUDE.md`): think before coding, simplicity first, surgical - changes, goal-driven verification. +## Verification status -## Active memory — F8 promotion gate + solved-question recall — SHIPPED, L2 pending (2026-07-07) +- The P1.1 workspace-directory registry and P2–P6 preprocessing workstreams are implemented and + have automated coverage. +- Evidence restructuring has a real PSD acceptance PASS as recorded above. +- L2 tests requiring real providers or remote databases remain opt-in. +- Server deployment, release, and owner-operated acceptance steps remain pending wherever the + referenced runbooks require explicit approval. -Two additions to close the loop on reusable memory, on top of the existing `tht memory -search` (Phase 2) reuse: +Run the layer-specific checks documented in `AGENTS.md`. For release-sensitive changes, also run +the repository contract scripts in `scripts/` and build the MkDocs site. -- **F8 promotion gate.** `reviewer_memory_promote` (Phase 8, called with only the session - id): the gate computes candidates deterministically via `tht memory promote --preview - --json` (the 3 reusable decision types, already excluding previously promoted/declined - ones) and shows a pre-selected checklist. Selected → `tht memory save-one` persists to - the vectordb + records `memory_promoted`; deselected → `memory_promotion_declined` - (ledger detail `seq:`) so it is never re-proposed. `tht memory promote`/`save-one` were - added to the gate's anti-bypass FORBIDDEN list (model must go through the gate tool). -- **Solved-question exemplars.** New vector kind `solved_question` reusing the existing - `memory` pgvector table (no server-side DDL); `harness/tht/solved.py` does a one-row - upsert keyed by a hash of question+SQL. CLI: `tht memory solved-index` / `solved-search`. - `tht session finalize` auto-indexes the pair (best-effort: green line on upsert, cyan - "già aggiornata" on dedup no-op, yellow warning + the recovery command - `tht memory solved-index ` on failure). `SKILL.md` now prescribes calling - `solved-search` as reference-only context in F4 (schema linking), F6 (CTE plan) and F7 - (final SQL), and documents the finalize auto-index in "Session end". +## Operational invariants -**Pending L2 gate (not yet run — needs VPN + writer key):** one live end-to-end session on -workspace `psd` via `./scripts/run-stack.sh` to verify (a) the promotion checklist renders -pre-selected and persists selected/declined correctly, (b) finalize indexes the pair, -(c) `tht memory solved-search` returns it with sql + tables. - -**Fast-follow:** -- RestSearcher top-k dilution — **client-side DONE** (2026-07-07): `search_similar` manda - `kinds` alla RPC (filtro server-side esatto) con fallback automatico su server legacy - (404 → retry senza filtro, post-filter client). **Resta la migrazione server** della - funzione SQL `search_similar` (+`kinds text[] DEFAULT NULL`): istruzioni pronte in - `harness/docs/vector-rest-kinds-migration.md`; l'ordine di deploy è libero, ma fino - alla migrazione il filtro resta client-side e la diluizione persiste. -- ~~`tht memory solved-search` muore con traceback grezzo se il vectordb è irraggiungibile~~ - **DONE** (2026-07-07): degrada a warning di una riga su stderr, stdout puro (`[]` in - --json), exit 0 — copre VectorRestError/EmbeddingsError/OperationalError. - -## Review gates v2 — payload strutturati + viewer dedicati — COMPLETE (2026-07-07) - -Plan: `~/.claude/plans/prima-di-passare-ai-inherited-marshmallow.md`. Merged to `main` @ `2410f01` -(ff, pushed). Executed via subagent-driven-development (4 workstreams, task reviews, final -whole-branch review + fix wave). - -- **Contracts:** `artifact.data.schema_version: 2` for `cte_plan` / `cte_result` / `phase`, - built **deterministically by the gate** (catalog descriptions via `tht schema columns`; SQL - from `ctes/.sql`; preview rows persisted by `tht cte test`); the model contributes only - purpose/rationale/note. Non-v2 payloads fall through to the legacy renderers — old sessions - and `tools/replay/replay.json` keep working. -- **Harness (Python):** `CteTestRecord.preview_rows` (+ `_jsonable` coercer, ≤10 rows, cells - ≤200 chars); new read-only `tht cte info --session --json` (index/total from - `cte_plan.json`, same source as `next_cte`); `tht cte plan --doc -` writes - `cte_plan_doc.json` (chain documentation; `cte_plan.json` stays a load-bearing `list[str]`). -- **Gate (JS):** `gate/artifact-contracts.js` (soft validators → self-corrective `textResult`, - TypeBox untouched) + `gate/enrich.js` (pure, catalog lookups injected); - `prepareReviewerArguments` now coerces `artifact.data` too (GLM stringified-param - mitigation); `SKILL.md` Phase 5/6 + disciplines rewritten (plan via `reviewer_confirm - kind:"cte_plan"` with payload A; `cte_result` gates send THIN data only — never SQL/preview - as text). -- **Frontend:** `artifactV2.ts` types; `CtePlanViewer` (per-CTE cards + chain strip), - `CteResultViewer` (shiki SQL + AG Grid preview), `PhaseSummaryViewer` (checks + criteria - with the VALUES driving choices), `PreviewGrid` extracted from `ResultsPanel`, - `statusBadge.ts` shared success/warn/error tokens. -- **Replay:** v2 fixtures + `tools/replay/augment-review-gates.mjs`; `replay.json` regenerated; - offline visual pass ok (screenshots in the SDD scratch dir). -- **LIVE E2E (session `2026-07-07-011858`, GLM 5.2):** all 8 phases completed with the v2 - gates; session **finalized** (DWH validation battery green, needs VPN). -- **Bug found live + FIXED (`2410f01`):** infinite spinner at workflow end — the bridge dropped - Pi's `agent_end` (the ONLY end-of-turn signal) and `working` was released only by the next - gate, which the final turn doesn't have. Now: bridge maps `agent_end` → SSE - `system_event`; FE tracks `agentActive`; an unexpected Pi child exit notifies the client - (info error + synthetic `agent_end`). Memory: `pi-rpc-event-vocabulary`. -- **Open (non-blocking):** `tht.sqlcheck` maps table aliases by first occurrence (found and - worked around by the model in F6 — spawned as a separate task); one more live confirmation - that the spinner stops at F8 (the chain is unit-tested end to end). - -## F4 schema-linking column curation + look&feel v2 — COMPLETE (2026-07-06) - -Branches `feat/f4-schema-linking-column-curation` (PR #1) + `feat/frontend-lookfeel-v2`, landed -on `main` (`d942635` … `7491e8c`). The F4 gate (`reviewer_schema_linking`) presents -catalog-enriched tables/columns (descriptions from `tht schema columns`, hardened enrichment), -per-table columns modal (suggested pre-checked, suggested-first ordering + filter box); -decisions `column_promoted`/`column_excluded` + deterministic `tht session -sync-schema-linking` projection into `schema_linking.json`. Live-verified including the -clobber test (the model's joins write preserves curated columns). Look&feel v2: shadows/radii/ -mono labels, 70% gate modal, structured cards (colors untouched). Memory: -`thothii-visual-language-v2`. - -## Workflow contract hardening — COMPLETE (2026-07-01) - -Spec: `docs/superpowers/specs/2026-07-01-workflow-contract-hardening-design.md` · Plan: -`docs/superpowers/plans/2026-07-01-workflow-contract-hardening.md`. Merged to `main` @ `3dadc6f` -(pushed). Driven by analysis of Pi session `2026-06-30-165708` (GLM 5.2), where the model spent -~80% of its tool calls reverse-engineering the harness because `SKILL.md` mis-stated the -phase-advance contract — and Phase 6 was a hard dead-end. Three coordinated harness fixes (TDD): - -- **F6 CTE-approval dead-end FIXED.** The gate's `reviewer_confirm kind:"cte_result"` used to - register `cte_approved --subject phase:6`, which `decision_cmd` rejects (exit 5 — it needs a - real CTE name from the plan) → F6 could never close. New `tht cte next --session ` returns - the first unapproved plan CTE; the gate now approves **by name**. (`tht/cli/cte_cmd.py`, - `.pi/extensions/tht-gate.js`.) -- **`schema_linking.json` writer/validator.** `store.set_schema_linking` (validates against the - `SchemaLinking` model, THEN writes — no partial file) → CLI `tht session set-schema-linking - --file ` (exit 5 on bad JSON / ValidationError) → gate tool `write_schema_linking` - (stdin). Replaces the model hand-writing the F4 artifact + ad-hoc python validation. -- **`SKILL.md` corrected to match the code.** Only F2-empty / F6-skipped auto-advance - (`_AUTO_ADVANCE_PHASES={2,6}`); every substantive phase closes with `reviewer_confirm - kind:"phase"` (F7 is **two-step**: `kind:"sql"` records `sql_approved`, then `kind:"phase"` - advances). Fixed Discipline 2 + Phase 1/3/4, added a per-phase **cheat-sheet**, documented the - `SchemaLinking` shape. The old false "the reviewer_decide already advances" (F3) claim — the - exact cause of the observed thrash — is gone. - -Verified: harness pytest **281 passed** / 5 deselected, gate JS **34/34**, changed-files ruff -clean (the 36 `ruff check .` errors are pre-existing on `main`). Final whole-branch review -(opus): READY TO MERGE, no Critical/Important. Executed via subagent-driven-development -(implementer + task-review per task, final opus review). **DEFERRED (needs VPN): live F4/F6 -end-to-end** — resuming session `2026-06-30-165708` (stuck at F6) is the ideal live probe. - -## UI/UX redesign + Resume — COMPLETE (2026-06-30) - -Plan: **`~/.claude/plans/foamy-forging-dahl.md`**. Memory: `thothii-ui-redesign-inprogress.md`. -**All workstreams done and pushed to origin/main:** D + E @ `0eeb3f7`, B + C @ `b056ff3`, -F @ `cef9ae4`, A @ `0a13f71`, G @ `e8cdd00`(scope) + `cbb8e18`(results). Nothing pending from this -plan. Per-workstream detail below for reference. - -- **D — DONE** (`c12bdcd`): session display `name` = 3-5 Italian keywords via **YAKE** (no LLM), - derived in `tht session new` (CLI layer); `create_session` core unchanged (`name=None` default). - `yake` added to `harness/pyproject.toml`. TDD `tests/test_session_name.py`; harness 269 passed. -- **E — DONE** (`0eeb3f7`): rotating activity icon replaces the red dot in `CentralStatus` - (inline, clickable → opens the panel); `ModelActivityPanel` is a **5-line expandable - model-stream tail**; `WorkingSpinner` extracted to its own module; the separate spinner button - + orphaned `Transcript.tsx` removed. Frontend 87/87, tsc clean. **Live visual check DONE - (2026-06-30):** inline spinner opens the panel; 5-line collapsed tail; expand → full transcript. -- **B — DONE** (`b056ff3`): `WorkflowBar` is now colored **dots** F1..F8, no phase-name text - (amber-translucent=running, green=done, red=error, gray=pending; green connectors lead the active - dot). Each dot carries `data-state`. Error is lightweight: store `phaseError` set when an `info` - `level=error` arrives during the phase, cleared on the next `ui_request` (`sessionStore.ts`). - **All four states live-verified** via Playwright. -- **C — DONE** (`b056ff3`): right sidebar — single-line denser rows (inline status dot + name, - `py-1`), a 3-level type hierarchy via **`/impeccable`** (L1 `SESSIONS` red/bold/wide-tracking · - L2 section + group headers muted uppercase · L3 names normal-case), and the **"No group" label - removed** (ungrouped sessions render after the last group; guarded so the empty-state still - teaches when there are no groups). **Live-verified.** (Resume in `SessionMenu` stays with A1.) -- **Tests:** frontend **93/93** (was 87; +3 store `phaseError`, +2 `WorkflowBar` dot-state, +1 - AppShell no-"No group"), `tsc -b` clean. -- **F — DONE** (uncommitted; live check deferred to G): single-select answers **auto-confirm**. - `reviewer_select` options may carry a `decision` payload (`{type, subject, detail?, rationale?}`) - and an optional `advance`; picking such an option persists the decision directly via - `tht decision add` (shared `decisionAddArgs` helper, also used by `reviewer_decide`) — no redundant - `reviewer_decide`/`reviewer_confirm` gate. Options without a payload stay ask-only; back/exit/Other - never persist. Pure logic extracted to `resolveSelectOutcome`/`decisionAddArgs` (exported, unit- - tested). Contract docs updated: `reviewer_select` tool desc + `SKILL.md` (widget summary, - disciplines 2-3, Phase-1 single-pick) + the `CLAUDE.md` gate note. Gate JS **33/33**, harness 269. - **Live verification (model actually uses `reviewer_select`+decision, no follow-up gate, decision in - `review_decisions.jsonl`) deferred to G** — it is model-behavior-dependent. -- **A — DONE** (uncommitted): **A1** — `SessionMenu` gains a **Resume** item (gated to - `status!=="finalized" && !archived`), wired in `AppShell` to the existing `doResume` → `POST - /sessions/:id/resume`. 3 tests (`SessionMenu.test.tsx`); frontend **96/96**, tsc clean. **A2** — - diagnosis-first clean-room repro shows the **resume cold-start stall NO LONGER reproduces on pi - 0.79.4** (8/8 chained into the tool calls, fresh + partway; GLM 5.2 now narrates AND emits - `tht session show`+`read SKILL.md` in-turn). The earlier narrate-and-stop predates the pi upgrade. - Defense-in-depth applied: `RIPRENDI_KICKOFF` hardened to force the in-turn tool call (gate test + - live regression 2/2). The cross-model angle (weaker/older models) lives in **G**. -- **G — DONE** (`e8cdd00`+`cbb8e18`): cross-model behavior matrix via a committed clean-room harness - (`harness/scripts/model-matrix.mjs`). **Tier 1** — kickoff + resume first-turn: all *available* - models chain in-turn (`zai/glm-5.2`, `deepseek/deepseek-v4-{pro,flash}`, `aritmolab/qwen3.6-35b-a3b`, - `zai/glm-4.5-air`); the resume stall recurs on none (closes A's cross-model robustness). - `aritmolab/gemma4-26b-a4b` = **404 unavailable** at the endpoint (listed but not served) — infra - gap, not a workflow issue. **Tier 2** — F single-select auto-confirm verified live on `glm-5.2`: - answering the first `reviewer_select` persisted a `concept_clarified` decision **0→1** with **no - follow-up gate** (closes F's deferred live check). Full results: the G plan doc + memory - `thothii-cross-model-matrix`. No prompt hardening needed. - -**Status:** **All UI-redesign + resume workstreams done and pushed — D, E, B, C, F, A, G.** -Nothing pending from the plan. Optional nice-to-haves (not required): Tier-2 F/multiselect live for -the non-baseline models (cheap re-run with `harness/scripts/model-matrix.mjs` + the Tier-2 method), -and a one-off manual Playwright kebab→resume pass in the live UI. - -## Live verification + reviewer_select fix (2026-06-30, afternoon) - -Drove the real stack (Playwright → backend → real Pi → GLM 5.2 → DWH) end-to-end. - -- **F1 hang fix (`418187a`) VERIFIED LIVE.** Answered an F1 reviewer widget; Pi resumed (model - socket reopened) and the gate produced new output — vs the old silent hang. The transition - "silent hang → gate re-presents/advances" proves `ctx.ui.input` now resolves. -- **New bug found + fixed: reviewer_select `choices` vs `choice`.** The gate's `reviewer_select` - (and `reviewer_confirm` reject) read `resp.choice` (singular) but the frontend uniformly sends - `choices: [id]` (array) — so every single-select gate answered "Nessuna scelta ricevuta" and - re-proposed forever (multiselect was fine; it already read `choices`). Fix: a shared - `selectedChoice(resp)` helper (`harness/.pi/extensions/tht-gate.js`) reading the array; both - handlers use it. TDD: `gate/__tests__/gate_choice.test.js` RED→GREEN, full gate suite **28/28**. - VERIFIED LIVE: a single-select answer is now accepted and the workflow advances (2/4 → 3/4). -- **Resume cold-start STALL confirmed (open item #1).** On `/riprendi-sessione`, GLM 5.2 narrates - the bootstrap step then ends the turn without the tool call → Pi idle, unrecoverable from the UI. - Memory: `thothii-resume-cold-start-stall.md`. -- **GLM 5.2 F1 is slow (~3-4 min, ~50+ reads) but works** — looks stuck but isn't; don't hit - "Stop and save" (it `POST /close`s → kills Pi). Memory: `thothii-glm52-f1-slow-not-stuck.md`. - -## Earlier work — F1 reviewer-widget hang fix + multiselect guidance (committed 2026-06-30; authored 2026-06-29) - -Two fixes, **committed to `main`** (7 files): - -1. **Bug: every reviewer widget hung "stuck with no output" after the human answered** — F1 - disambiguation (and any gate) dead-ended. Root cause, confirmed from Pi's own source - (`@mariozechner/pi-coding-agent` `dist/modes/rpc/rpc-mode.js`, `createDialogPromise`): - `ctx.ui.input` assigns its OWN RPC id (`crypto.randomUUID`) and correlates - `extension_ui_response` on THAT id, silently dropping unknown ids. The gate puts a - different id (`u${Date.now()}`) inside the descriptor carried in `title`. `SessionBridge` - was replying with the **descriptor** id, so real Pi never resolved `ctx.ui.input` → the - model never continued. **Fix:** `SessionBridge` now stores Pi's top-level `m.id` - (`pendingPiId`) on the incoming request and replies `extension_ui_response{ id: pendingPiId, - value: }` (value still carries the descriptor id, so the gate's internal - `resp.id === descriptor.id` check holds). File: `backend/src/bridge/session-bridge.ts`. - Full write-up: memory `pi-ui-input-id-correlation.md`. - - **The test double was masking it:** `harness/tests/fake_pi/fake_pi_rpc.mjs` had forced - `m.id == descriptor.id`. Corrected to mirror real Pi (distinct `randomUUID` top-level id, - correlate on it, drop unknown ids); `test_fake_pi_contract.mjs` gained a negative - regression test ("respond with descriptor id → no follow-up"). - - TDD: `backend/test/session-bridge.test.ts` (unit) + `backend/test/e2e-f1.test.ts` - (integration — now asserts the model's follow-up arrives after the answer) went - RED→GREEN. - -2. **UX: multi-answer disambiguation** — `harness/.pi/skills/tht-sessione/SKILL.md` Phase 1 - now tells the model to use `reviewer_decide` (the existing multiselect/checkbox widget) - when an ambiguity admits several simultaneously-true answers, instead of single-pick - `reviewer_select`. Guidance-only — no new widget (`frontend MultiselectWidget` already - exists). - -Verified at commit time: backend `npx vitest run` **67/67 green**; `tsc --noEmit -p .` **OK**; -fake-pi contract `node --test test_fake_pi_contract.mjs` **2/2 green**. **Verified LIVE -2026-06-30** (see the top "Live verification" section). - -## Most recent feature — Session management (MERGED to main @ 2c21e46) -Full session management modeled on Claude's UI, all three layers: -- **Read-only "split view" panel** (left drawer, `SessionDocumentsPanel`) showing a session's - phase documents read-only (reuses `SqlViewer`/`SchemaLinkingViewer`/`MarkdownView`). -- **Rename / Move to group / Archive / Delete** via a kebab menu (`SessionMenu`) → REST → - `tht session set-name/set-group/archive/unarchive/delete`. Archive = a manifest `archived` - flag (not a dir move); groups = a manifest `group` field; delete = hard `rmtree` + confirm. -- Rail: collapsible group headers + "No group" + a separate **Archive** view. -- **Resume correctness:** read-only **guard** (HTTP 409 when `finalized` or `archived`); - `PiProcessManager.spawnFor` now has a `new`/`resume` mode (resume sends - `/riprendi-sessione `); a "Phase 0 — Resume" cold-start section in `SKILL.md`. -- Design docs: `docs/superpowers/specs/2026-06-29-session-management-design.md` + - `docs/superpowers/plans/2026-06-29-session-management.md`. - -### ⚠️ Open items / pending gates -1. **Resume cold-start stall — RESOLVED on pi 0.79.4 (workstream A, 2026-06-30).** The earlier - narrate-and-stop (GLM 5.2 narrating the bootstrap step then ending the turn without the tool - call) **no longer reproduces**: a clean-room repro of the backend's exact resume handshake - chained into `tht session show`+`read SKILL.md` in-turn **8/8** (fresh + partway sessions). The - pi upgrade is the likely fix. Defense-in-depth: `RIPRENDI_KICKOFF` hardened to force the in-turn - tool call (gate test + live 2/2). Memory: `thothii-resume-cold-start-stall.md`. **Remaining:** - the cross-model angle (older/weaker models) is folded into **G**; a full Playwright kebab→resume - pass through the live UI is still worth one manual run (item 2). -2. **Full Playwright live-stack verification (MANUAL, not yet run).** -3. **Minor backlog (non-blocking):** explicit id-traversal guard in `delete_session` - (today gated by `load_session`); `close_session` could reuse `_save_touched` (DRY); - delete-via-kebab integration test skipped (base-ui Menu portal not drivable in jsdom — - the dialog itself is unit-tested); a couple of test-file lint nits. -4. **DONE — F1 hang fix live-verified 2026-06-30** (see top section). The live verification - also surfaced + fixed the reviewer_select `choices` mismatch. -5. **Resolved: settings use `zai/glm-5.2`/medium** (not deepseek-flash); GLM 5.2 drives F1 - fine, just slowly (~3-4 min, ~50+ reads). To tell a truly stalled Pi from a merely-slow one, - check its sockets/children. Memory: `thothii-glm52-f1-slow-not-stuck.md`. -6. **DONE — Pi migrated to `@earendil-works/pi-coding-agent@0.80.3` (2026-07-04).** The old - scope `@mariozechner/pi-coding-agent` is frozen at 0.73.1; every release ≥0.74 lives under - the new scope `@earendil-works` (latest 0.80.3). The live `pi` (`~/.local/bin/pi`) was - repointed to 0.80.3. Validated by an API/RPC-surface diff (0.73.1→0.80.3: `ExtensionUIContext` - byte-identical, `rpc-types` additive-only, `createDialogPromise` + provider-registration API - unchanged) **plus** a live `model-matrix` smoke (GLM 5.2, `new`+`resume` both `CHAINED`, - full RPC event vocabulary incl. `extension_ui_request` intact). Notable: 0.80.3 adds - `ctx.mode: "tui"|"rpc"|"json"|"print"` (a proper mode discriminator; the earlier `0.79.4` - references above are superseded). Rollback: the old package is still on disk — repoint the - symlink to `@mariozechner/.../dist/cli.js`. Memory: `thothii-pi-earendil-migration.md`. - -## Where design history lives -- Specs: `docs/superpowers/specs/` · Plans: `docs/superpowers/plans/` -- SDD execution ledger (gitignored scratch): `.superpowers/sdd/progress.md` -- Auto-memory index: `~/.claude/projects/-Users-mp-projects-ThothII/memory/MEMORY.md` - (Pi RPC event vocabulary, ui.input id correlation, Omics Portal/GSD design system, - resume cold-start stall, GLM 5.2 F1 slow≠stuck). - -## Git -`main` @ `2410f01`, **pushed to `origin`** (github.com/mptyl/ThothII); working tree clean, no stashes. -Latest arc (2026-07-07): review-gates-v2 ff-merged — `9b4f6b9` (WS1 harness) · `3ad93cd` (WS2 -gate+SKILL) · `1c97289` (WS3 viewers) · `4042d0b`+`b089482` (WS4 replay) · `b59b57c` (review fix -wave) · `2410f01` (agent_end spinner fix). Branch `feat/review-gates-v2` still exists (local + -origin), fully merged. Before that (2026-07-05/06): F4 column curation (PR #1) + look&feel v2 — -`0d9e035` · `0a63ba9` · `d942635` · `9fe1c93` · `e9b2934` · `9403147` · `7491e8c`. -Older history (workflow hardening 2026-07-01, UI redesign 2026-06-30): see the sections above; -stale branches were pruned on 2026-06-30 (SHAs recoverable via reflog). +- `tht`'s `-c`/`--config` option follows the subcommand; it is not a global option. +- `--json` commands write pristine JSON to stdout. +- Persisted phase documents and the decision ledger are the source of session truth; chat is not. +- UI chrome is English; workspace document content retains the workspace language. +- The backend refuses resume for finalized or archived sessions. +- A resume must send `/riprendi-sessione `; a new session must send `/nuova-domanda`. +- DWH access is read-only. diff --git a/README.md b/README.md index 7a912f46..51b3a3ac 100644 --- a/README.md +++ b/README.md @@ -202,20 +202,18 @@ Startup mode adds bounded image build/two-service health startup, installation-a status, stopped-container-aware ownership checks, and exact cleanup. The ordinary hosted Windows job remains deterministic and does not claim Docker startup. -## Preprocessing jobs and S3 Evidence +## Workspace preprocessing and S3 Evidence -The included preprocessing services reuse the internal Qdrant/Ollama stack. Mount Evidence at -`/data/source/evidence`, then run the explicit preprocessing preset: +Run preprocessing through the native host CLI and the installation descriptor: ```sh -docker compose --env-file deploy/env/local.env \ - -f compose.yaml -f deploy/compose.local.yaml \ - -f deploy/compose.preprocess.yaml --profile preprocess run --rm preprocess-evidence +tht --installation /absolute/path/thothii-installation.yaml workspace preprocess evidence +tht --installation /absolute/path/thothii-installation.yaml workspace preprocess dwh ``` -Replace the final service with `preprocess-dwh` when required. The overlay makes each job wait for the internal Qdrant -service health checks and embedding model initialization; no separate semantic-service startup is -required. +The CLI starts the profile-gated `workspace-maintenance` service and enforces the workspace, +secret, Qdrant, and embedding contracts. See [Evidence](docs/evidence.md) and the +[workspace preprocessing CLI contract](docs/contracts/workspace-preprocessing-cli.md). S3 Evidence uses the optional `tht[s3]` dependency and canonical `s3://bucket/key` provenance. AWS endpoints are used when no custom URL is supplied. Every custom endpoint is an explicit egress diff --git a/backend/scripts/verify-workspace-descriptor-files.mjs b/backend/scripts/verify-workspace-descriptor-files.mjs index 6dcebc40..971bf230 100755 --- a/backend/scripts/verify-workspace-descriptor-files.mjs +++ b/backend/scripts/verify-workspace-descriptor-files.mjs @@ -15,28 +15,32 @@ const allowedKinds = new Set(["policy_text", "workspace_descriptor", "deployment // opener line through the closer line (including physical line endings). These // blocks are reviewed non-workspace runtime/config generation, not semantic proof. const reviewedExpandableBlocks = new Map([ - ["scripts/preprocess-smoke.sh", [ - { sha256: "fc530dc721c946644ab6552bbd46b7918d6c5f11f06f3495b6ea1fcda819b38d", rationale: "Generates the reviewed preprocess Compose override." }, + ["scripts/test-dwh-auth-nginx-integration.sh", [ + { sha256: "ead57234ad3520b5c7d4262b772957cbc7b9589da4f35fb17b160f948eb2ac7b", rationale: "Generates the reviewed isolated Nginx integration configuration." }, + ]], + ["scripts/test-install-tht.sh", [ + { sha256: "37f18ce7ce93cb8b84f3b3708462cc16d50fdc7bab22836c382dbacf8382f05f", rationale: "Generates the reviewed synthetic tht installer artifact." }, ]], ["scripts/test-server-pi-state-topology.sh", [ - { sha256: "6f746f7e8442b0a6ea0e216607a6a923d94b24cd8fa17fa2d1dac56e6f14f7ef", rationale: "Generates the isolated server topology test environment." }, + { sha256: "a9ab86c9408b22afb87f10f615570bc7ec09bc3e7eb8131f6d835afba9a6b1d8", rationale: "Generates the isolated server topology test environment, including its authentication configuration root." }, ]], ["scripts/test-vector-backup-restore-safety.sh", [ { sha256: "40b8a10a3c06aaa98e324fbf688b7d1f5cead330d7ba7eef98e06256d412a85a", rationale: "Generates the reviewed restore safety manifest." }, ]], ["scripts/test-windows-clone-contract.ps1", [ - { sha256: "80f4880576a0679cb58e7b92600e7a90550c93c254553a2d4b299539f9ff0bcf", rationale: "Generates reviewed Windows clone test configuration." }, - { sha256: "6166294bdc8a8bf6436ad402bcbf7cae0f3b67dc6051cecfcca79267a62b082c", rationale: "Same reviewed block in the repository-required CRLF checkout representation." }, - { sha256: "3216201d59400ed7d1ec23e536634b8235a2e78b336e45b4dc598624920f0057", rationale: "Generates reviewed Windows clone test configuration." }, - { sha256: "a4044bb38b27e8120e90d65a0695fe0afd7757c067ae8dd67f170edf569a1de0", rationale: "Same reviewed block in the repository-required CRLF checkout representation." }, + { sha256: "3204f772d33cad42bcac99191507051aefb2c91d2935bec6698b956e44f9bf45", rationale: "Generates reviewed Windows clone test configuration with its authentication configuration root." }, + { sha256: "f4814d842a7502b7ef30fd6b224d5cb17b0ffd6fb2367c41c49ac16587536d93", rationale: "Same reviewed block in the repository-required CRLF checkout representation." }, + { sha256: "6f25ce3b58cea47b74fe9319ed917d8089a2fb334bc0d469daa7e1f10865d870", rationale: "Generates the reviewed Windows Compose override for the canonical service topology." }, + { sha256: "45a3cf19f7ce697b858b63d27a4edc7fefa2414d0408e7b6d72a65c86d314f5b", rationale: "Same reviewed Compose override in the repository-required CRLF checkout representation." }, { sha256: "5d0d1a3fc45e99b3aacaf4ee5dd09a6bee1937784375dfe4bcfaa4ae32cfb9de", rationale: "Generates reviewed Windows clone test configuration." }, { sha256: "b903e5dae953ae1372f1a5276f12a92ed3dd632b897f3afe5e00c646d90a1b42", rationale: "Same reviewed block in the repository-required CRLF checkout representation." }, ]], ["scripts/unified-deployment-smoke.sh", [ - { sha256: "ca0c17d9ff8dc0fbe018fc1c5510eb33bc667a936fbe44a9be2d311390576825", rationale: "Generates reviewed Task 13 runtime configuration." }, - { sha256: "31ec00cc315b52da4a3bb6e3fba2d40aef29cdcd090bbc5d14c31f1aebbcfd04", rationale: "Generates reviewed Task 13 runtime configuration." }, - { sha256: "d92822815357ce3424e1a6eb43923df2b37b4fd93a3b5465ee9dfc69559ab0ed", rationale: "Generates reviewed Task 13 runtime configuration." }, - { sha256: "d6b8b7b951936c0452a485e9ee3b18a61251556581d6f7a2ce66f994b5700695", rationale: "Generates reviewed Task 13 runtime configuration." }, + { sha256: "1d60bf140165a8fabfa0c3729e776136904717e67becf3e0ab68c70d8e37847e", rationale: "Generates reviewed Task 13 runtime configuration." }, + { sha256: "36d3d8a2362dbdc4fad90948d6c227586d749f56b9a4bc5b6b5a91bcbec6407b", rationale: "Generates the reviewed local Task 13 Compose override." }, + { sha256: "c556f7d910d0788e219b042957e6b307cb9925b43920c680535d0d3a6dcbdb25", rationale: "Generates the reviewed local Task 13 installation descriptor." }, + { sha256: "526006fa6d48a8080b3834723630c64de5005a67243e944ebf1da15212b4d654", rationale: "Generates the reviewed server Task 13 Compose override." }, + { sha256: "c57ae2205c21ead0c2015a353aaabb948fa4ddd9b78a2cdcdb71f48cf2db742d", rationale: "Generates the reviewed projected-auth server Task 13 installation descriptor." }, ]], ["scripts/vector-backup.sh", [ { sha256: "571899db49dfdcec8107fbe1e0a86a61e7581979d3c4c248c20546843e275bcf", rationale: "Generates the reviewed backup manifest inside the helper command." }, diff --git a/backend/scripts/verify-workspace-descriptor-files.test.mjs b/backend/scripts/verify-workspace-descriptor-files.test.mjs index 97e4fe85..a66bc921 100644 --- a/backend/scripts/verify-workspace-descriptor-files.test.mjs +++ b/backend/scripts/verify-workspace-descriptor-files.test.mjs @@ -624,7 +624,6 @@ test("an in-band marker cannot authorize expandable content", async (t) => { test("current exact reviewed expandable blocks pass only at their trusted paths", async (t) => { const reviewedPaths = [ - "scripts/preprocess-smoke.sh", "scripts/test-server-pi-state-topology.sh", "scripts/test-vector-backup-restore-safety.sh", "scripts/test-windows-clone-contract.ps1", @@ -636,19 +635,6 @@ test("current exact reviewed expandable blocks pass only at their trusted paths" root: repositoryRoot, entries: reviewedPaths.map((path) => entry("deployment_script", path)), }); - - const root = await fixture(t); - const original = await readFile(join(repositoryRoot, "scripts/preprocess-smoke.sh"), "utf8"); - await put(root, "scripts/copied-preprocess.sh", original); - await assert.rejects( - verifyEntries({ root, entries: [entry("deployment_script", "scripts/copied-preprocess.sh")] }), - /exact-content reviewed allowlist/, - ); - await put(root, "scripts/preprocess-smoke.sh", original.replace('$tmp/smoke.yaml', '$tmp/other.yaml')); - await assert.rejects( - verifyEntries({ root, entries: [entry("deployment_script", "scripts/preprocess-smoke.sh")] }), - /exact-content reviewed allowlist/, - ); }); test("PowerShell tokenizer ignores opener text in comments and ordinary strings", async (t) => { diff --git a/backend/src/auth/runtime-projection.ts b/backend/src/auth/runtime-projection.ts index 5f83af29..24e4e645 100644 --- a/backend/src/auth/runtime-projection.ts +++ b/backend/src/auth/runtime-projection.ts @@ -581,7 +581,7 @@ function load(root: string): LoadedAuthConfig { ); return { value: selectedGeneration.value, - revision: `sha256:${selected.generation}`, + revision: selected.generation, sourcePath: join(generationsPath, selected.generation, "auth.yaml"), runtimeProjection: snapshot( selected.generation, diff --git a/backend/src/bridge/session-bridge.ts b/backend/src/bridge/session-bridge.ts index f1dbf2c2..69ea9559 100644 --- a/backend/src/bridge/session-bridge.ts +++ b/backend/src/bridge/session-bridge.ts @@ -4,6 +4,15 @@ const GENERIC_MODEL_FAILURE = "Model request failed. Check provider connectivity, then Resume the session."; const SUBSCRIPTION_MODEL_FAILURE = "The selected model is unavailable for the current subscription. Choose another model and start a new session."; +const PHASE_STARTED_NOTIFICATION_PREFIX = "__tht_phase_started__:"; + +function phaseStartedNotification(message: unknown): string | null { + if (typeof message !== "string" || !message.startsWith(PHASE_STARTED_NOTIFICATION_PREFIX)) { + return null; + } + const phase = message.slice(PHASE_STARTED_NOTIFICATION_PREFIX.length); + return /^F[1-8]$/.test(phase) ? phase : ""; +} function safeModelFailure(error: unknown): string { const detail = typeof error === "string" ? error : ""; @@ -36,7 +45,7 @@ export type ClientEvent = | { type: "activity_event"; activity: ToolActivity } | { type: "usage"; usage: TokenUsage } | { type: "info"; [k: string]: any } - | { type: "system_event"; event: string }; + | { type: "system_event"; event: string; phase?: string }; export type TurnState = "idle" | "running" | "waiting" | "failed"; @@ -71,7 +80,11 @@ export class SessionBridge { }); } } else if (m.type === "extension_ui_request" && m.method === "notify") { - this.fan({ type: "info", level: m.notifyType ?? "info", text: m.message ?? "" }); + const phase = phaseStartedNotification(m.message); + if (phase) this.fan({ type: "system_event", event: "phase_started", phase }); + else if (phase === null) { + this.fan({ type: "info", level: m.notifyType ?? "info", text: m.message ?? "" }); + } } else if (m.type === "message_update" && m.assistantMessageEvent?.type === "text_delta") { this.fan({ type: "text_delta", text: m.assistantMessageEvent.delta ?? "" }); } else if (m.type === "message_update" && m.assistantMessageEvent?.type === "thinking_delta") { diff --git a/backend/src/config.ts b/backend/src/config.ts index 4a4946e4..1b7e680a 100644 --- a/backend/src/config.ts +++ b/backend/src/config.ts @@ -176,7 +176,11 @@ function positiveDimension(value: string | undefined, fallback: number): number return parsed; } -export function loadConfig(env: Record): AppConfig { +export function loadConfig( + env: Record, + options: { surface?: "application" | "workspace-maintenance" } = {}, +): AppConfig { + const applicationSurface = options.surface !== "workspace-maintenance"; const defaultAuthConfigFile = "/run/thothii-auth/auth.yaml"; const authConfigFile = absoluteAuthPath(env.THT_AUTH_CONFIG_FILE ?? defaultAuthConfigFile, "file"); const authStateRoot = absoluteAuthPath(env.THT_AUTH_STATE_ROOT ?? "/data/auth", "state root"); @@ -211,13 +215,13 @@ export function loadConfig(env: Record): AppConfig { throw new Error(`unsupported AUTH_MODE=${requestedMode}; use none, mock, or upstream`); } const nodeEnvironment = env.NODE_ENV ?? process.env.NODE_ENV; - if ((requestedMode === "none" || requestedMode === "mock") + if (applicationSurface && (requestedMode === "none" || requestedMode === "mock") && nodeEnvironment !== "development" && nodeEnvironment !== "test") { throw new Error("production requires auth.yaml or AUTH_MODE=upstream"); } authMode = requestedMode as "none" | "mock" | "upstream"; } - const publicExposure = env.THOTH_PUBLIC_EXPOSURE === "true"; + const publicExposure = applicationSurface && env.THOTH_PUBLIC_EXPOSURE === "true"; if (publicExposure && authMode !== "oidc" && authMode !== "upstream") { throw new Error("public exposure requires AUTH_MODE=upstream or configured OIDC behind a trusted proxy"); } diff --git a/backend/src/pi/enabled-models.ts b/backend/src/pi/enabled-models.ts index 4f544451..5a5456a2 100644 --- a/backend/src/pi/enabled-models.ts +++ b/backend/src/pi/enabled-models.ts @@ -1,6 +1,6 @@ import { readFileSync } from "node:fs"; import { homedir } from "node:os"; -import { join } from "node:path"; +import { join, resolve } from "node:path"; export interface PiEnabledModelsResult { ids: string[]; @@ -43,7 +43,8 @@ function isExactCompositeId(value: unknown): value is string { export function loadPiEnabledModels(opts: LoadOptions): PiEnabledModelsResult { const warnings: string[] = []; const read = opts.read ?? ((path: string) => readFileSync(path, "utf8")); - const agentDir = opts.agentDir ?? join(homedir(), ".pi", "agent"); + const agentDir = opts.agentDir + ?? resolve(process.env.PI_CODING_AGENT_DIR ?? join(homedir(), ".pi", "agent")); const globalPath = join(agentDir, "settings.json"); const projectPath = join(opts.harnessDir, ".pi", "settings.json"); const globalSettings = readSettings(globalPath, false, read, warnings); diff --git a/backend/src/pi/managed-config.ts b/backend/src/pi/managed-config.ts index 42c12603..646189dd 100644 --- a/backend/src/pi/managed-config.ts +++ b/backend/src/pi/managed-config.ts @@ -56,6 +56,34 @@ export function validateDeclarativePiConfig(raw: string): void { assertDeclarativePiConfig(parsePiConfigJson(raw)); } +/** Return the selected provider's declarative apiKey value without knowing provider IDs in code. */ +export function configuredPiProviderApiKey( + raw: string | undefined, + provider: string | undefined, +): string | undefined { + if (raw === undefined || provider === undefined) return undefined; + const parsed = parsePiConfigJson(raw); + if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) { + throw new PiManagedConfigError(); + } + const providers = (parsed as { providers?: unknown }).providers; + if (!providers || typeof providers !== "object" || Array.isArray(providers)) { + throw new PiManagedConfigError(); + } + const entry = Object.entries(providers as Record) + .find(([id]) => id.trim().toLowerCase() === provider.trim().toLowerCase()); + if (!entry) return undefined; + const config = entry[1]; + if (!config || typeof config !== "object" || Array.isArray(config)) { + throw new PiManagedConfigError(); + } + assertDeclarativePiConfig(config); + const apiKey = (config as { apiKey?: unknown }).apiKey; + if (apiKey === undefined) return undefined; + if (typeof apiKey !== "string" || apiKey.length === 0) throw new PiManagedConfigError(); + return apiKey; +} + function configuredPiAgentDir(): string { return resolve(process.env.PI_CODING_AGENT_DIR ?? join(homedir(), ".pi", "agent")); } @@ -102,6 +130,7 @@ export function readConfiguredPiAgentFile( export interface PiRuntimeAgentSnapshot { agentDir: string; sessionDir: string; + models?: string; cleanup: () => void; } @@ -152,6 +181,7 @@ export function createPiRuntimeAgentSnapshot(): PiRuntimeAgentSnapshot { return { agentDir: snapshotDir, sessionDir: process.env.PI_CODING_AGENT_SESSION_DIR || join(sourceAgentDir, "sessions"), + models, cleanup: () => { if (cleaned) return; cleaned = true; diff --git a/backend/src/pi/management.ts b/backend/src/pi/management.ts index 114db34f..25efb0bf 100644 --- a/backend/src/pi/management.ts +++ b/backend/src/pi/management.ts @@ -9,8 +9,10 @@ import { } from "../settings/settings-store.js"; import type { PiModel } from "./list-models.js"; import { + configuredPiProviderApiKey, PI_MANAGED_CONFIG_ERROR_MESSAGE, isPiManagedConfigError, + readConfiguredPiAgentFile, } from "./managed-config.js"; import { createPiProviderSmoke, type PiProviderSmoke } from "./provider-smoke.js"; import { loadPiAuthProviders } from "./auth-providers.js"; @@ -119,6 +121,10 @@ export function createPiManagement(config: AppConfig, deps: PiManagementDeps): P authProviders: loadPiAuthProviders(), resolveCredentialValue: () => secretValue(config, "THT_MODEL_API_KEY"), credentialFile: config.modelApiKeyFile, + configuredApiKey: configuredPiProviderApiKey( + readConfiguredPiAgentFile("models.json", true), + provider, + ), }); } catch { return "missing"; diff --git a/backend/src/pi/pi-process-manager.ts b/backend/src/pi/pi-process-manager.ts index 226f967f..ee5d7b6f 100644 --- a/backend/src/pi/pi-process-manager.ts +++ b/backend/src/pi/pi-process-manager.ts @@ -7,7 +7,10 @@ import { buildPiChildEnv, canonicalPiProvider } from "./provider-credentials.js" import { loadPiAuthProviders } from "./auth-providers.js"; import { secretValue } from "../config/secret-bundle.js"; import { clearPrincipalEnvironment, principalEnvironment, type PrincipalContext } from "../auth/principal.js"; -import { createPiRuntimeAgentSnapshot } from "./managed-config.js"; +import { + configuredPiProviderApiKey, + createPiRuntimeAgentSnapshot, +} from "./managed-config.js"; export interface SessionRuntime { rpc: RpcClient; @@ -81,6 +84,7 @@ export class PiProcessManager { authProviders: this.loadAuthProviders(agent.agentDir), credentialValue: secretValue(this.cfg, "THT_MODEL_API_KEY"), credentialFile: this.cfg.modelApiKeyFile, + configuredApiKey: configuredPiProviderApiKey(agent.models, provider), additions: { THT_SESSION: sessionId, THT_AUTHOR: author }, }); env.PI_CODING_AGENT_DIR = agent.agentDir; diff --git a/backend/src/pi/provider-credentials.ts b/backend/src/pi/provider-credentials.ts index a01d6669..79e15625 100644 --- a/backend/src/pi/provider-credentials.ts +++ b/backend/src/pi/provider-credentials.ts @@ -42,9 +42,14 @@ const PROVIDER_KEY_ENV: Readonly> = { const COMPOUND_PROVIDERS = new Set([ "amazon-bedrock", "azure-openai-responses", "cloudflare-ai-gateway", "cloudflare-workers-ai", ]); -const LOCAL_PROVIDERS = new Set([ - "ollama", "lmstudio", "local", "aritmolab", "local-qwen", "faux", -]); + +function configuredCredentialEnv(apiKey: string | undefined): string | null | undefined { + if (apiKey === undefined) return undefined; + if (!apiKey.startsWith("$")) return null; + const matched = /^\$(?:\{([A-Z][A-Z0-9_]*(?:API_KEY|TOKEN))\}|([A-Z][A-Z0-9_]*(?:API_KEY|TOKEN)))$/.exec(apiKey); + if (!matched) throw new Error("model provider credential is unavailable"); + return matched[1] ?? matched[2]; +} export function canonicalPiProvider(provider: string | undefined): string | undefined { const value = provider?.trim().toLowerCase(); @@ -106,6 +111,8 @@ export function buildPiChildEnv(opts: { credentialFile?: string; additions?: NodeJS.ProcessEnv; credentialValue?: string; + /** Exact apiKey declaration from the selected provider in models.json. */ + configuredApiKey?: string; fsOps?: CredentialFsOps; /** * Providers pi can authenticate from its own auth store. For these, the single @@ -147,17 +154,22 @@ export function buildPiChildEnv(opts: { delete env.THT_SSL_CA_FILE; for (const name of PI_0803_CREDENTIAL_ENV_NAMES) delete env[name]; const provider = canonicalPiProvider(opts.provider); + const configuredEnv = configuredCredentialEnv(opts.configuredApiKey); + if (configuredEnv) delete env[configuredEnv]; if (provider && COMPOUND_PROVIDERS.has(provider)) { throw new Error( "compound credential bundles are unsupported by THT_MODEL_API_KEY_FILE; " + "dedicated provider configuration is required", ); } - if (provider && !LOCAL_PROVIDERS.has(provider)) { + if (provider) { // pi self-authenticates this provider from its own auth store; injecting the // single managed key here would force one provider's key onto another. if (opts.authProviders?.has(provider)) return env; - const envName = PROVIDER_KEY_ENV[provider]; + // A literal apiKey is entirely owned by models.json (commonly a non-secret + // placeholder for a local OpenAI-compatible endpoint) and needs no managed key. + if (configuredEnv === null) return env; + const envName = configuredEnv ?? PROVIDER_KEY_ENV[provider]; if (!envName || (!opts.credentialFile && opts.credentialValue === undefined)) { throw new Error("model provider credential is unavailable"); } @@ -169,7 +181,7 @@ export function buildPiChildEnv(opts: { } else if (opts.credentialFile) env[envName] = readCredential(opts.credentialFile, opts.fsOps ?? realFs); else throw new Error("model provider credential is unavailable"); - } else if (opts.credentialFile && !provider) { + } else if (opts.credentialFile) { throw new Error("model provider credential is unavailable"); } return env; @@ -181,11 +193,14 @@ export function piProviderCredentialStatus(opts: { credentialFile?: string; resolveCredentialValue?: () => string | undefined; authProviders?: ReadonlySet; + configuredApiKey?: string; fsOps?: CredentialFsOps; }): PiCredentialStatus { const provider = canonicalPiProvider(opts.provider); - if (!provider || LOCAL_PROVIDERS.has(provider)) return "missing"; + if (!provider) return "missing"; if (opts.authProviders?.has(provider)) return "present"; + const configuredEnv = configuredCredentialEnv(opts.configuredApiKey); + if (configuredEnv === null) return "missing"; try { buildPiChildEnv({ ambient: {}, @@ -193,6 +208,7 @@ export function piProviderCredentialStatus(opts: { credentialFile: opts.credentialFile, credentialValue: opts.resolveCredentialValue?.(), authProviders: opts.authProviders, + configuredApiKey: opts.configuredApiKey, fsOps: opts.fsOps, }); return "present"; diff --git a/backend/src/pi/provider-smoke.ts b/backend/src/pi/provider-smoke.ts index 976b0e3c..19c5cbd2 100644 --- a/backend/src/pi/provider-smoke.ts +++ b/backend/src/pi/provider-smoke.ts @@ -11,6 +11,7 @@ import { buildPiChildEnv, canonicalPiProvider } from "./provider-credentials.js" import type { PiReasoning } from "./management.js"; import { PiManagedConfigError, + configuredPiProviderApiKey, isPiManagedConfigError, parsePiConfigJson, readConfiguredPiAgentFile, @@ -65,11 +66,15 @@ export function createPiProviderSmoke( const canonicalProvider = canonicalPiProvider(provider); if (!canonicalProvider || timeoutMs <= 0) throw providerFailure(); const configuredAuthProviders = authProviders(); + const configuredModels = options.readModelsStore + ? options.readModelsStore() + : readConfiguredPiAgentFile("models.json", true); const env = buildPiChildEnv({ provider: canonicalProvider, authProviders: configuredAuthProviders, credentialValue: secretValue(config, "THT_MODEL_API_KEY"), credentialFile: config.modelApiKeyFile, + configuredApiKey: configuredPiProviderApiKey(configuredModels, canonicalProvider), }); clearPrincipalEnvironment(env); delete env.THT_DATA_ROOT; @@ -90,9 +95,6 @@ export function createPiProviderSmoke( ); writeDeclarativeAgentConfig(join(isolatedAgentDir, "auth.json"), authStore); } - const configuredModels = options.readModelsStore - ? options.readModelsStore() - : readConfiguredPiAgentFile("models.json", true); if (configuredModels !== undefined) { const modelsStore = selectedProviderModelsStore( configuredModels, diff --git a/backend/src/tht/tht-runner.ts b/backend/src/tht/tht-runner.ts index 8b681a9f..4be2e0b0 100644 --- a/backend/src/tht/tht-runner.ts +++ b/backend/src/tht/tht-runner.ts @@ -20,7 +20,7 @@ import { validateOperationalWorkspace, type WorkspaceDescriptor, } from "../workspaces/schema.js"; -import { reconcileCollection } from "../workspaces/qdrant-collection.js"; +import { reconcileCollection, type CollectionMode } from "../workspaces/qdrant-collection.js"; import type { WorkspaceSecretStore } from "../workspaces/secret-store.js"; export interface ThtConfig extends SecretBundleConfig { @@ -575,7 +575,7 @@ export class ThtRunner { async qdrantEnsure( workspace: WorkspaceDescriptor, timeoutSec: number, - mode: "self_heal" | "require_existing" = "require_existing", + mode: CollectionMode = "require_existing", ): Promise { let descriptor; try { diff --git a/backend/src/workspace-maintenance.ts b/backend/src/workspace-maintenance.ts index 81d6e01a..8a466eae 100644 --- a/backend/src/workspace-maintenance.ts +++ b/backend/src/workspace-maintenance.ts @@ -212,7 +212,7 @@ async function readSessionInventory(dataRoot: string, workspaceId: string): Prom } function createProductionService(): WorkspacePreprocessingService { - const config = loadConfig(process.env); + const config = loadConfig(process.env, { surface: "workspace-maintenance" }); const registry = new WorkspaceRegistry(config.workspaceRegistry); const workspaceSecretStore = new WorkspaceSecretStore({ root: config.workspaceSecretStoreRoot, @@ -307,6 +307,10 @@ function createProductionService(): WorkspacePreprocessingService { const result = await runner.qdrantEnsure(workspace, 30); return result.ok ? { ok: true as const } : { ok: false as const, code: result.code ?? "workspace_not_activatable" }; }, + evidencePreflight: async (workspace) => { + const result = await runner.qdrantEnsure(workspace, 30, "evidence_maintenance"); + return result.ok ? { ok: true as const } : { ok: false as const, code: result.code ?? "workspace_not_activatable" }; + }, }); } diff --git a/backend/src/workspaces/evidence-materialization.ts b/backend/src/workspaces/evidence/materialization.ts similarity index 96% rename from backend/src/workspaces/evidence-materialization.ts rename to backend/src/workspaces/evidence/materialization.ts index c9eea887..205b7493 100644 --- a/backend/src/workspaces/evidence-materialization.ts +++ b/backend/src/workspaces/evidence/materialization.ts @@ -9,7 +9,7 @@ import { writeFileSync, } from "node:fs"; import { dirname, isAbsolute, join } from "node:path"; -import { GitWorkspaceRepository } from "./git-repository.js"; +import type { GitWorkspaceRepository } from "../git-repository.js"; export interface EvidenceMaterializationLimits { maxEntries: number; @@ -81,7 +81,10 @@ function writeExclusiveNoFollow(path: string, contents: Buffer, mode: number): v } export interface MaterializeEvidenceTreeOptions { - repository: GitWorkspaceRepository; + repository: Pick< + GitWorkspaceRepository, + "evidenceTreeObjects" | "evidenceTreeId" | "gitObjectSize" | "evidenceBlobBytes" + >; revision: string; id: string; /** The workspace directory (e.g. `/`) that will receive `evidence/` and the manifest. */ diff --git a/backend/src/workspaces/evidence/preprocessing.ts b/backend/src/workspaces/evidence/preprocessing.ts new file mode 100644 index 00000000..7e09c894 --- /dev/null +++ b/backend/src/workspaces/evidence/preprocessing.ts @@ -0,0 +1,177 @@ +import { isIP } from "node:net"; +import type { WorkspaceDescriptor } from "../schema.js"; + +type EvidenceConfig = WorkspaceDescriptor["evidence"]; +type SemanticFailureCode = "workspace_not_activatable" | "semantic_index_incompatible"; + +export interface EvidenceJobState { + runId: string; + completedStages: string[]; + childRuns: Record; +} + +export interface EvidencePreprocessingDependencies { + runStage(argv: string[]): Promise>; + persistJob(): void; + evidencePreflight(): Promise<{ ok: true } | { ok: false; code: SemanticFailureCode }>; + requireRunId(value: unknown): string; + numberRecord(value: unknown): Record | undefined; +} + +export interface EvidencePreprocessingRequest { + evidence: EvidenceConfig; + job: EvidenceJobState; + dryRun?: boolean; + httpPrivateHostAllowlist?: readonly string[]; +} + +export interface EvidencePreprocessingOutcome { + status: "succeeded" | "unchanged" | "dry_run" | "failed"; + code: "ok" | "egress_policy_refused" | SemanticFailureCode; + runId?: string; + childRuns?: Record; + completedStages?: string[]; + counts?: Record; + warnings?: string[]; +} + +function isPrivateHost(hostname: string): boolean { + if (hostname === "localhost" || hostname === "metadata.google.internal") return true; + const address = isIP(hostname); + if (address === 4) { + if (/^127\./.test(hostname) || /^10\./.test(hostname) || /^192\.168\./.test(hostname)) { + return true; + } + if (/^169\.254\./.test(hostname) || /^0\./.test(hostname)) return true; + const match = /^172\.(\d+)\./.exec(hostname); + return Boolean(match && Number(match[1]) >= 16 && Number(match[1]) <= 31); + } + if (address === 6) { + const normalized = hostname.toLowerCase(); + return normalized === "::1" + || normalized.startsWith("fe80:") + || normalized.startsWith("fd") + || normalized.startsWith("fc"); + } + return hostname.endsWith(".internal"); +} + +function evidencePolicy( + evidence: EvidenceConfig, + httpPrivateHostAllowlist?: readonly string[], +): EvidencePreprocessingOutcome | undefined { + if (!evidence || evidence.source.type === "filesystem") return undefined; + if (evidence.source.type === "http") { + for (const value of evidence.source.uris) { + const host = new URL(value).hostname; + if ( + isPrivateHost(host) + && !(evidence.source.allow_private_hosts && httpPrivateHostAllowlist?.includes(host)) + ) { + return { status: "failed", code: "egress_policy_refused" }; + } + } + return undefined; + } + if ( + evidence.source.endpoint_url !== undefined + || evidence.source.credentials === "ambient" + || evidence.source.allow_private_endpoint + || evidence.source.allow_insecure_endpoint + ) { + return { status: "failed", code: "egress_policy_refused" }; + } + return undefined; +} + +function jobResult(job: EvidenceJobState): Pick< + EvidencePreprocessingOutcome, + "runId" | "childRuns" | "completedStages" +> { + return { + runId: job.runId, + childRuns: { ...job.childRuns }, + completedStages: [...job.completedStages], + }; +} + +async function runEvidenceStage( + request: EvidencePreprocessingRequest, + deps: EvidencePreprocessingDependencies, +): Promise { + const payload = await deps.runStage([ + "preprocess", + "evidence", + ...(request.dryRun ? ["--dry-run"] : []), + ...(request.job.childRuns.evidence + ? ["--resume", request.job.childRuns.evidence] + : []), + "--json", + "-c", + "/dev/fd/3", + ]); + if (typeof payload.run_id === "string") { + request.job.childRuns.evidence = deps.requireRunId(payload.run_id); + } + if (!request.dryRun && !request.job.completedStages.includes("evidence")) { + request.job.completedStages.push("evidence"); + } + deps.persistJob(); + return { + status: request.dryRun ? "dry_run" : "succeeded", + code: "ok", + ...jobResult(request.job), + counts: deps.numberRecord(payload.counts), + }; +} + +export async function preprocessEvidence( + request: EvidencePreprocessingRequest, + deps: EvidencePreprocessingDependencies, +): Promise { + if (!request.evidence) { + return { + status: "unchanged", + code: "ok", + warnings: ["workspace has no Evidence source"], + }; + } + const policy = evidencePolicy(request.evidence, request.httpPrivateHostAllowlist); + if (policy) return policy; + const preflight = await deps.evidencePreflight(); + if (!preflight.ok) { + return { status: "failed", code: preflight.code, runId: request.job.runId }; + } + if (request.job.completedStages.includes("evidence") && !request.dryRun) { + return { + status: "unchanged", + code: "ok", + runId: request.job.runId, + completedStages: [...request.job.completedStages], + }; + } + return await runEvidenceStage(request, deps); +} + +export async function continueEvidencePreprocessing( + request: Omit & { + priorCounts?: Record; + }, + deps: EvidencePreprocessingDependencies, +): Promise { + if (!request.evidence) { + return { + status: "succeeded", + code: "ok", + ...jobResult(request.job), + warnings: ["workspace has no Evidence source"], + ...(request.priorCounts ? { counts: request.priorCounts } : {}), + }; + } + const policy = evidencePolicy(request.evidence, request.httpPrivateHostAllowlist); + if (policy) return { ...policy, ...jobResult(request.job) }; + if (!request.job.completedStages.includes("evidence")) { + return await runEvidenceStage(request, deps); + } + return { status: "unchanged", code: "ok", ...jobResult(request.job) }; +} diff --git a/backend/src/workspaces/preprocessing-service.ts b/backend/src/workspaces/preprocessing-service.ts index c8180cab..f7dbc4f0 100644 --- a/backend/src/workspaces/preprocessing-service.ts +++ b/backend/src/workspaces/preprocessing-service.ts @@ -1,7 +1,12 @@ import { createHash, randomBytes } from "node:crypto"; import { readdirSync, readFileSync, rmSync, writeFileSync, mkdirSync } from "node:fs"; -import { isIP } from "node:net"; import { join } from "node:path"; +import { + continueEvidencePreprocessing, + preprocessEvidence as runEvidencePreprocessing, + type EvidencePreprocessingDependencies, + type EvidencePreprocessingOutcome, +} from "./evidence/preprocessing.js"; import type { WorkspaceDescriptor } from "./schema.js"; import { PreprocessingStateStore, @@ -68,6 +73,9 @@ export interface WorkspacePreprocessingServiceDeps { semanticPreflight(workspace: WorkspaceDescriptor): Promise< { ok: true } | { ok: false; code: "workspace_not_activatable" | "semantic_index_incompatible" } >; + evidencePreflight(workspace: WorkspaceDescriptor): Promise< + { ok: true } | { ok: false; code: "workspace_not_activatable" | "semantic_index_incompatible" } + >; httpPrivateHostAllowlist?: readonly string[]; } @@ -104,26 +112,6 @@ function baseResult( }; } -function isPrivateHost(hostname: string): boolean { - if (hostname === "localhost" || hostname === "metadata.google.internal") return true; - const address = isIP(hostname); - if (address === 4) { - if (/^127\./.test(hostname) || /^10\./.test(hostname) || /^192\.168\./.test(hostname)) return true; - if (/^169\.254\./.test(hostname) || /^0\./.test(hostname)) return true; - const match = /^172\.(\d+)\./.exec(hostname); - return Boolean(match && Number(match[1]) >= 16 && Number(match[1]) <= 31); - } - if (address === 6) { - const normalized = hostname.toLowerCase(); - return normalized === "::1" || normalized.startsWith("fe80:") || normalized.startsWith("fd") || normalized.startsWith("fc"); - } - return hostname.endsWith(".internal"); -} - -function noEvidenceWarning(workspace: WorkspaceDescriptor): string[] { - return workspace.evidence === undefined ? ["workspace has no Evidence source"] : []; -} - export class WorkspacePreprocessingService { constructor(private readonly deps: WorkspacePreprocessingServiceDeps) {} @@ -332,36 +320,16 @@ export class WorkspacePreprocessingService { async preprocessEvidence(options: { workspaceId: string; dryRun?: boolean; resumeRunId?: string }): Promise { const scope = await this.startRun(options.workspaceId, "preprocess evidence", options.resumeRunId); - if (scope.runtime.workspace.evidence === undefined) { - return baseResult(scope.runtime, "preprocess evidence", "unchanged", "ok", { - warnings: noEvidenceWarning(scope.runtime.workspace), - }); - } - const policy = this.evidencePolicy(scope.runtime.workspace); - if (policy !== undefined) return baseResult(scope.runtime, "preprocess evidence", policy.status, policy.code, { warnings: policy.warnings }); - const semantic = await this.deps.semanticPreflight(scope.runtime.workspace); - if (!semantic.ok) return baseResult(scope.runtime, "preprocess evidence", "failed", semantic.code, { runId: scope.job.runId }); - if (scope.job.completedStages.includes("evidence") && !options.dryRun) { - return baseResult(scope.runtime, "preprocess evidence", "unchanged", "ok", { - runId: scope.job.runId, - completedStages: [...scope.job.completedStages], - }); - } - const payload = await this.runJsonStage(scope.runtime, [ - "preprocess", "evidence", - ...(options.dryRun ? ["--dry-run"] : []), - ...(scope.job.childRuns.evidence ? ["--resume", scope.job.childRuns.evidence] : []), - "--json", "-c", "/dev/fd/3", - ]); - if (typeof payload.run_id === "string") scope.job.childRuns.evidence = this.requireRunId(payload.run_id); - if (!options.dryRun && !scope.job.completedStages.includes("evidence")) scope.job.completedStages.push("evidence"); - this.state(scope.runtime.workspaceId).writeJob(scope.job); - return baseResult(scope.runtime, "preprocess evidence", options.dryRun ? "dry_run" : "succeeded", "ok", { - runId: scope.job.runId, - childRuns: { ...scope.job.childRuns }, - completedStages: [...scope.job.completedStages], - counts: this.numberRecord(payload.counts), - }); + const outcome = await runEvidencePreprocessing( + { + evidence: scope.runtime.workspace.evidence, + job: scope.job, + dryRun: options.dryRun, + httpPrivateHostAllowlist: this.deps.httpPrivateHostAllowlist, + }, + this.evidenceDependencies(scope), + ); + return this.evidenceResult(scope, "preprocess evidence", outcome); } async run(options: { workspaceId: string; resumeRunId?: string }): Promise { @@ -401,59 +369,23 @@ export class WorkspacePreprocessingService { } const semantic = await this.deps.semanticPreflight(scope.runtime.workspace); if (!semantic.ok) return baseResult(scope.runtime, "preprocess run", "failed", semantic.code, { runId: scope.job.runId }); + let schemaCounts: Record | undefined; if (!scope.job.completedStages.includes("schema_index")) { const payload = await this.runJsonStage(scope.runtime, ["vector", "index-schema", "--json", "-c", "/dev/fd/3"]); scope.job.completedStages.push("schema_index"); this.state(scope.runtime.workspaceId).writeJob(scope.job); - const warnings = noEvidenceWarning(scope.runtime.workspace); - if (scope.runtime.workspace.evidence === undefined) { - return baseResult(scope.runtime, "preprocess run", "succeeded", "ok", { - runId: scope.job.runId, - childRuns: { ...scope.job.childRuns }, - completedStages: [...scope.job.completedStages], - counts: this.numberRecord(payload.counts), - warnings, - }); - } + schemaCounts = this.numberRecord(payload.counts); } - if (scope.runtime.workspace.evidence === undefined) { - return baseResult(scope.runtime, "preprocess run", "succeeded", "ok", { - runId: scope.job.runId, - childRuns: { ...scope.job.childRuns }, - completedStages: [...scope.job.completedStages], - warnings: noEvidenceWarning(scope.runtime.workspace), - }); - } - const policy = this.evidencePolicy(scope.runtime.workspace); - if (policy !== undefined) { - return baseResult(scope.runtime, "preprocess run", policy.status, policy.code, { - runId: scope.job.runId, - childRuns: { ...scope.job.childRuns }, - completedStages: [...scope.job.completedStages], - warnings: policy.warnings, - }); - } - if (!scope.job.completedStages.includes("evidence")) { - const payload = await this.runJsonStage(scope.runtime, [ - "preprocess", "evidence", - ...(scope.job.childRuns.evidence ? ["--resume", scope.job.childRuns.evidence] : []), - "--json", "-c", "/dev/fd/3", - ]); - if (typeof payload.run_id === "string") scope.job.childRuns.evidence = this.requireRunId(payload.run_id); - scope.job.completedStages.push("evidence"); - this.state(scope.runtime.workspaceId).writeJob(scope.job); - return baseResult(scope.runtime, "preprocess run", "succeeded", "ok", { - runId: scope.job.runId, - childRuns: { ...scope.job.childRuns }, - completedStages: [...scope.job.completedStages], - counts: this.numberRecord(payload.counts), - }); - } - return baseResult(scope.runtime, "preprocess run", "unchanged", "ok", { - runId: scope.job.runId, - childRuns: { ...scope.job.childRuns }, - completedStages: [...scope.job.completedStages], - }); + const outcome = await continueEvidencePreprocessing( + { + evidence: scope.runtime.workspace.evidence, + job: scope.job, + httpPrivateHostAllowlist: this.deps.httpPrivateHostAllowlist, + priorCounts: schemaCounts, + }, + this.evidenceDependencies(scope), + ); + return this.evidenceResult(scope, "preprocess run", outcome); } private async startRun(workspaceId: string, operation: string, resumeRunId?: string): Promise { @@ -476,6 +408,25 @@ export class WorkspacePreprocessingService { return new PreprocessingStateStore({ dataRoot: this.deps.dataRoot, workspaceId }); } + private evidenceDependencies(scope: RunScope): EvidencePreprocessingDependencies { + return { + runStage: async (argv) => await this.runJsonStage(scope.runtime, argv), + persistJob: () => this.state(scope.runtime.workspaceId).writeJob(scope.job), + evidencePreflight: async () => await this.deps.evidencePreflight(scope.runtime.workspace), + requireRunId: (value) => this.requireRunId(value), + numberRecord: (value) => this.numberRecord(value), + }; + } + + private evidenceResult( + scope: RunScope, + operation: "preprocess evidence" | "preprocess run", + outcome: EvidencePreprocessingOutcome, + ): WorkspaceOperationResult { + const { status, code, ...extra } = outcome; + return baseResult(scope.runtime, operation, status, code, extra); + } + private async runSuggestStage( scope: RunScope, fromSql: ReadonlyArray<{ name: string; sql: string }>, @@ -581,33 +532,4 @@ export class WorkspacePreprocessingService { return undefined; } - private evidencePolicy(workspace: WorkspaceDescriptor): { - status: WorkspaceOperationResult["status"]; - code: WorkspaceOperationResult["code"]; - warnings?: string[]; - } | undefined { - const evidence = workspace.evidence; - if (!evidence) return undefined; - // P6: filesystem Evidence is materialized from the pinned commit at activation, so the - // engine may proceed directly against the immutable revision content root. - if (evidence.source.type === "filesystem") return undefined; - if (evidence.source.type === "http") { - for (const value of evidence.source.uris) { - const host = new URL(value).hostname; - if (isPrivateHost(host) && !(evidence.source.allow_private_hosts && this.deps.httpPrivateHostAllowlist?.includes(host))) { - return { status: "failed", code: "egress_policy_refused" }; - } - } - return undefined; - } - if ( - evidence.source.endpoint_url !== undefined - || evidence.source.credentials === "ambient" - || evidence.source.allow_private_endpoint - || evidence.source.allow_insecure_endpoint - ) { - return { status: "failed", code: "egress_policy_refused" }; - } - return undefined; - } } diff --git a/backend/src/workspaces/qdrant-collection.ts b/backend/src/workspaces/qdrant-collection.ts index a934b423..94b8b4ff 100644 --- a/backend/src/workspaces/qdrant-collection.ts +++ b/backend/src/workspaces/qdrant-collection.ts @@ -3,12 +3,12 @@ export const QDRANT_REQUIRED_INDEXES = Object.freeze([ "record_kind", "vector_generation", "workspace_id", "workspace_revision", ]); -export type CollectionMode = "self_heal" | "require_existing"; +export type CollectionMode = "self_heal" | "require_existing" | "evidence_maintenance"; export interface CollectionCheck { ok: boolean; code?: "semantic_index_incompatible" | "workspace_not_activatable"; - state?: "ready" | "created" | "repaired"; + state?: "ready" | "created" | "repaired" | "upgraded"; } export interface ReconcileCollectionOptions { @@ -52,12 +52,43 @@ async function createCollection(opts: ReconcileCollectionOptions, request: typeo const res = await request(qdrantUrl(opts.baseUrl, `/collections/${encodeURIComponent(opts.collection)}`), { method: "PUT", headers: { "content-type": "application/json" }, - body: JSON.stringify({ vectors: { size: opts.dimensions, distance: qdrantDistance(opts.distance) } }), + body: JSON.stringify({ + vectors: { size: opts.dimensions, distance: qdrantDistance(opts.distance) }, + ...(opts.mode === "evidence_maintenance" ? { sparse_vectors: { bm25: { modifier: "idf" } } } : {}), + }), signal: opts.signal, }); if (!res.ok && res.status !== 409) throw new Error("qdrant collection creation failed"); } +type EvidenceSparseCompatibility = "compatible" | "upgradeable" | "incompatible"; + +function evidenceSparseCompatibility(info: any): EvidenceSparseCompatibility { + const sparseVectors = info?.config?.params?.sparse_vectors; + if (sparseVectors === undefined) return "upgradeable"; + if (!sparseVectors || typeof sparseVectors !== "object" || Array.isArray(sparseVectors)) { + return "incompatible"; + } + const bm25 = sparseVectors.bm25; + if (bm25 === undefined) return "upgradeable"; + return typeof bm25 === "object" && bm25 !== null && !Array.isArray(bm25) + && typeof bm25.modifier === "string" && bm25.modifier.toLowerCase() === "idf" + ? "compatible" : "incompatible"; +} + +async function createBm25Vector(opts: ReconcileCollectionOptions, request: typeof fetch): Promise { + const res = await request( + qdrantUrl(opts.baseUrl, `/collections/${encodeURIComponent(opts.collection)}/vectors/bm25`), + { + method: "PUT", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ sparse: { modifier: "idf" } }), + signal: opts.signal, + }, + ); + if (!res.ok && res.status !== 409) throw new Error("qdrant BM25 vector creation failed"); +} + async function createIndex(opts: ReconcileCollectionOptions, field: string, request: typeof fetch): Promise { const res = await request(qdrantUrl(opts.baseUrl, `/collections/${encodeURIComponent(opts.collection)}/index`), { method: "PUT", @@ -68,13 +99,14 @@ async function createIndex(opts: ReconcileCollectionOptions, field: string, requ if (!res.ok && res.status !== 409) throw new Error("qdrant index creation failed"); } -/** Reconcile a Qdrant collection: self-heal creates missing collections/indexes; require_existing - * only validates and refuses incompatible contracts (never mutates). */ +/** Reconcile a Qdrant collection. Only Evidence maintenance may add the BM25 sparse vector; + * session admission remains limited to the existing dense/index self-heal behavior. */ export async function reconcileCollection(opts: ReconcileCollectionOptions): Promise { const request = opts.request ?? fetch; + const allowsMutation = opts.mode === "self_heal" || opts.mode === "evidence_maintenance"; let info = await collectionInfo(opts, request); if (info === undefined) { - if (opts.mode !== "self_heal") return { ok: false, code: "semantic_index_incompatible" }; + if (!allowsMutation) return { ok: false, code: "semantic_index_incompatible" }; await createCollection(opts, request); // Tolerate an already-compatible concurrent creator: re-read the final state. info = await collectionInfo(opts, request); @@ -83,9 +115,22 @@ export async function reconcileCollection(opts: ReconcileCollectionOptions): Pro if (!vectorCompatibility(info, opts)) { return { ok: false, code: "semantic_index_incompatible" }; } + let bm25Added = false; + if (opts.mode === "evidence_maintenance") { + const sparse = evidenceSparseCompatibility(info); + if (sparse === "incompatible") return { ok: false, code: "semantic_index_incompatible" }; + if (sparse === "upgradeable") { + await createBm25Vector(opts, request); + info = await collectionInfo(opts, request); + if (!vectorCompatibility(info, opts) || evidenceSparseCompatibility(info) !== "compatible") { + return { ok: false, code: "semantic_index_incompatible" }; + } + bm25Added = true; + } + } const missing = await missingIndexes(opts, info); if (missing.length > 0) { - if (opts.mode !== "self_heal") return { ok: false, code: "semantic_index_incompatible" }; + if (!allowsMutation) return { ok: false, code: "semantic_index_incompatible" }; for (const field of missing) await createIndex(opts, field, request); // Qdrant payload indexes become visible asynchronously: poll until the // contract is complete or a bounded deadline passes (fail closed). @@ -101,5 +146,5 @@ export async function reconcileCollection(opts: ReconcileCollectionOptions): Pro } return { ok: false, code: "semantic_index_incompatible" }; } - return { ok: true, state: "ready" }; + return { ok: true, state: bm25Added ? "upgraded" : "ready" }; } diff --git a/backend/src/workspaces/registry.ts b/backend/src/workspaces/registry.ts index e69b4e87..f2f4fd1c 100644 --- a/backend/src/workspaces/registry.ts +++ b/backend/src/workspaces/registry.ts @@ -5,7 +5,7 @@ import { isAbsolute, join } from "node:path"; import { buildInstallationContract, renderWorkspaceDocs } from "./contracts.js"; import { parseAnnotationsYaml } from "./annotations.js"; import { syncAnnotations } from "./annotations-sync.js"; -import { materializeEvidenceTree } from "./evidence-materialization.js"; +import { materializeEvidenceTree } from "./evidence/materialization.js"; import { assertCatalogMatchesDescriptor, parseWorkspaceCatalogYaml, type WorkspaceCatalog, type WorkspaceCatalogEntry } from "./catalog.js"; import { GitWorkspaceRepository, diff --git a/backend/src/workspaces/runtime-renderer.ts b/backend/src/workspaces/runtime-renderer.ts index 23db59a1..4af55d8b 100644 --- a/backend/src/workspaces/runtime-renderer.ts +++ b/backend/src/workspaces/runtime-renderer.ts @@ -174,7 +174,10 @@ function renderEvidence( } return { - evidence: { sources: [renderedSource] }, + evidence: { + ...(workspace.evidence.schema_version === 2 ? { schema_version: 2 } : {}), + sources: [renderedSource], + }, vector: { max_chunk_chars: workspace.evidence.policy.max_chunk_chars, retain_published_generations: workspace.evidence.policy.retain_published_generations, diff --git a/backend/src/workspaces/schema.ts b/backend/src/workspaces/schema.ts index 666fae3e..66c5363c 100644 --- a/backend/src/workspaces/schema.ts +++ b/backend/src/workspaces/schema.ts @@ -115,6 +115,7 @@ export type EvidenceSource = }; export interface WorkspaceEvidence { + schema_version: 1 | 2; source: EvidenceSource; policy: EvidencePolicy; } @@ -246,10 +247,10 @@ const evidencePattern = z.string().refine(isSafeEvidencePattern, { const filesystemEvidenceSourceSchema = z.object({ type: z.literal("filesystem"), uri: z.string(), - patterns: z.array(evidencePattern).min(1).default(["**/*.md"]), + patterns: z.array(evidencePattern).min(1).optional(), max_bytes: positiveSafeInteger.default(10 * 1024 * 1024), }).strict().superRefine((source, context) => { - if (new Set(source.patterns).size !== source.patterns.length) { + if (source.patterns !== undefined && new Set(source.patterns).size !== source.patterns.length) { context.addIssue({ code: "custom", path: ["patterns"], message: "evidence patterns must not repeat" }); } }); @@ -323,12 +324,39 @@ const evidencePolicySchema = z.object({ retain_published_generations: positiveSafeInteger.default(3), }).strict(); const workspaceEvidenceSchema = z.object({ + schema_version: z.union([z.literal(1), z.literal(2)]).default(1), source: evidenceSourceSchema, policy: evidencePolicySchema.default({ max_chunk_chars: 4_000, retain_published_generations: 3, }), -}).strict(); +}).strict().superRefine((evidence, context) => { + if (evidence.schema_version !== 2 || evidence.source.type !== "filesystem") return; + const patterns = evidence.source.patterns ?? ["curated/**/*.md"]; + const selectsSource = patterns.some((pattern) => pattern === "source" || pattern.startsWith("source/")); + const selectsCurated = patterns.some((pattern) => pattern === "curated" || pattern.startsWith("curated/")); + if (selectsSource && selectsCurated) { + context.addIssue({ + code: "custom", + path: ["source", "patterns"], + message: "schema-versioned filesystem Evidence patterns cannot span source and curated", + }); + } else if (patterns.length !== 1 || patterns[0] !== "curated/**/*.md") { + context.addIssue({ + code: "custom", + path: ["source", "patterns"], + message: "schema-versioned filesystem Evidence patterns must be exactly curated/**/*.md", + }); + } +}).transform((evidence) => ({ + ...evidence, + source: evidence.source.type !== "filesystem" || evidence.source.patterns !== undefined + ? evidence.source + : { + ...evidence.source, + patterns: evidence.schema_version === 2 ? ["curated/**/*.md"] : ["**/*.md"], + }, +})); function unique(values: readonly T[], context: z.RefinementCtx, path: PropertyKey[]) { if (new Set(values).size !== values.length) { diff --git a/backend/test/auth-runtime-projection.test.ts b/backend/test/auth-runtime-projection.test.ts index 5a9a58f2..affa4ef6 100644 --- a/backend/test/auth-runtime-projection.test.ts +++ b/backend/test/auth-runtime-projection.test.ts @@ -70,6 +70,7 @@ const passwordHash = "$argon2id$v=19$m=65536,t=3,p=1$AAECAwQFBgcICQoLDA0ODw$DRo8ZSPI8G5OCvnFFapbVEjP69aDjy1Sw9i2743cPC4"; const userId = "6ba7b810-9dad-4ed1-80b4-00c04fd430c8"; const roots: string[] = []; +const linuxTest = test.runIf(process.platform === "linux"); afterEach(() => { fsHook.path = undefined; @@ -276,13 +277,13 @@ function expectDenied(operation: () => unknown): void { } } -test("loads ready projection as one immutable auth and local-users snapshot", () => { +linuxTest("loads ready projection as one immutable auth and local-users snapshot", () => { const root = projectionRoot(); const fixture = localProjectionFixture("synthetic-user", passwordHash); const generation = writeReadyProjection(root, fixture); const loaded = createProjectedAuthenticationConfigProvider(root).current(); - expect(loaded.revision).toBe(`sha256:${generation}`); + expect(loaded.revision).toBe(generation); expect(loaded.sourcePath).toBe( join(root, "generations", generation, "auth.yaml"), ); @@ -298,7 +299,7 @@ test("loads ready projection as one immutable auth and local-users snapshot", () ).toBe(true); }); -test("loads a complete OIDC projection without a users snapshot", () => { +linuxTest("loads a complete OIDC projection without a users snapshot", () => { const root = projectionRoot(); const generation = writeReadyOidcProjection(root); const loaded = createProjectedAuthenticationConfigProvider(root).current(); @@ -309,7 +310,7 @@ test("loads a complete OIDC projection without a users snapshot", () => { }); }); -test("loadConfig selects an immutable projected local provider and its in-memory registry", async () => { +linuxTest("loadConfig selects an immutable projected local provider and its in-memory registry", async () => { const root = projectionRoot(); writeReadyProjection(root, localProjectionFixture("projected-user", passwordHash)); @@ -323,7 +324,7 @@ test("loadConfig selects an immutable projected local provider and its in-memory expect(await registry?.findByUsername("PROJECTED-USER")).toMatchObject({ username: "projected-user" }); }); -test("loadConfig selects an immutable projected OIDC provider without direct-file fallback", () => { +linuxTest("loadConfig selects an immutable projected OIDC provider without direct-file fallback", () => { const root = projectionRoot(); writeReadyOidcProjection(root); @@ -338,7 +339,7 @@ test("loadConfig selects an immutable projected OIDC provider without direct-fil }); }); -test("rejects a trailing-slash runtime root", () => { +linuxTest("rejects a trailing-slash runtime root", () => { const root = projectionRoot(); writeReadyProjection( root, @@ -349,7 +350,7 @@ test("rejects a trailing-slash runtime root", () => { ); }); -test.each([ +linuxTest.each([ ["missing", undefined], [ "blocked", @@ -381,7 +382,7 @@ test.each([ ); }); -test.each([ +linuxTest.each([ "root traversal", "CURRENT symlink", "CURRENT hardlink", @@ -436,7 +437,7 @@ test.runIf(process.geteuid?.() === 0)( }, ); -test("rejects a foreign group with the correct owner", () => { +linuxTest("rejects a foreign group with the correct owner", () => { const root = projectionRoot(); writeReadyProjection( root, @@ -449,7 +450,7 @@ test("rejects a foreign group with the correct owner", () => { ); }); -test("enumerates closed namespaces without path-based readdirSync", () => { +linuxTest("enumerates closed namespaces without path-based readdirSync", () => { const root = projectionRoot(); const generation = writeReadyProjection( root, @@ -462,7 +463,7 @@ test("enumerates closed namespaces without path-based readdirSync", () => { ).toBe(generation); }); -test("rejects a symlinked runtime root", () => { +linuxTest("rejects a symlinked runtime root", () => { const root = projectionRoot(); writeReadyProjection( root, @@ -476,7 +477,7 @@ test("rejects a symlinked runtime root", () => { ); }); -test.each([ +linuxTest.each([ "root", "generations", "selected generation", @@ -514,7 +515,7 @@ test.each([ ); }); -test.each(["manifest", "generation", "size", "digest"])( +linuxTest.each(["manifest", "generation", "size", "digest"])( "rejects changed %s integrity data without secret disclosure", (kind) => { const root = projectionRoot(); @@ -542,7 +543,7 @@ test.each(["manifest", "generation", "size", "digest"])( }, ); -test("switches atomically to a later complete generation", () => { +linuxTest("switches atomically to a later complete generation", () => { const root = projectionRoot(); const first = writeReadyProjection( root, @@ -564,7 +565,7 @@ test("switches atomically to a later complete generation", () => { expect(provider.current().runtimeProjection?.generation).toBe(second); }); -test("retries once when CURRENT is atomically replaced between lstat and open", () => { +linuxTest("retries once when CURRENT is atomically replaced between lstat and open", () => { const root = projectionRoot(); const first = writeReadyProjection( root, @@ -591,7 +592,7 @@ test("retries once when CURRENT is atomically replaced between lstat and open", ).toBe(second); }); -test("retries once when CURRENT is replaced after the final identity read", () => { +linuxTest("retries once when CURRENT is replaced after the final identity read", () => { const root = projectionRoot(); const first = writeReadyProjection( root, @@ -628,7 +629,7 @@ test("retries once when CURRENT is replaced after the final identity read", () = expect(observations).toBe(3); }); -test("retries once when CURRENT is replaced between root descriptor and path observations", () => { +linuxTest("retries once when CURRENT is replaced between root descriptor and path observations", () => { const root = projectionRoot(); const first = writeReadyProjection( root, @@ -678,7 +679,7 @@ test("retries once when CURRENT is replaced between root descriptor and path obs expect(replacedCurrent).toBe(true); }); -test("rejects a second CURRENT replacement after the one permitted retry", () => { +linuxTest("rejects a second CURRENT replacement after the one permitted retry", () => { const root = projectionRoot(); const first = writeReadyProjection( root, @@ -717,7 +718,7 @@ test("rejects a second CURRENT replacement after the one permitted retry", () => ); }); -test("fails deterministically when generations is replaced during a load", () => { +linuxTest("fails deterministically when generations is replaced during a load", () => { const root = projectionRoot(); const generation = writeReadyProjection( root, @@ -748,7 +749,7 @@ test("fails deterministically when generations is replaced during a load", () => ); }); -test("fails deterministically when the selected generation directory is replaced during a load", () => { +linuxTest("fails deterministically when the selected generation directory is replaced during a load", () => { const root = projectionRoot(); const generation = writeReadyProjection( root, @@ -780,7 +781,7 @@ test("fails deterministically when the selected generation directory is replaced ); }); -test.each(["corrupt", "symlink"])( +linuxTest.each(["corrupt", "symlink"])( "rejects a %s retained predecessor generation", (kind) => { const root = projectionRoot(); @@ -819,7 +820,7 @@ test.each(["corrupt", "symlink"])( }, ); -test("has no direct-file fallback when CURRENT is absent", () => { +linuxTest("has no direct-file fallback when CURRENT is absent", () => { const root = projectionRoot(); const generation = writeReadyProjection( root, @@ -839,7 +840,7 @@ test("has no direct-file fallback when CURRENT is absent", () => { ); }); -test("in-flight snapshot authenticates A after selection B and deletion A, while a new load sees B", async () => { +linuxTest("in-flight snapshot authenticates A after selection B and deletion A, while a new load sees B", async () => { const root = projectionRoot(); const first = writeReadyProjection( root, diff --git a/backend/test/config.test.ts b/backend/test/config.test.ts index 68ba2742..17d8ab6f 100644 --- a/backend/test/config.test.ts +++ b/backend/test/config.test.ts @@ -97,6 +97,17 @@ test("loadConfig allows none and mock only outside production when auth.yaml is .toThrow("production requires auth.yaml or AUTH_MODE=upstream"); }); +test("workspace maintenance loads production configuration without an authentication surface", () => { + expect(loadConfig( + { NODE_ENV: "production", THOTH_PUBLIC_EXPOSURE: "true" }, + { surface: "workspace-maintenance" }, + )).toMatchObject({ + authMode: "none", + authentication: undefined, + publicExposure: false, + }); +}); + test("local Compose profiles explicitly select the development auth environment", () => { for (const profile of ["../../deploy/compose.local.yaml", "../../docker-compose.dev.yml"]) { expect(readFileSync(new URL(profile, import.meta.url), "utf8")).toMatch(/NODE_ENV:\s*development/); diff --git a/backend/test/enabled-models.test.ts b/backend/test/enabled-models.test.ts index df8842e7..2ae0232b 100644 --- a/backend/test/enabled-models.test.ts +++ b/backend/test/enabled-models.test.ts @@ -1,4 +1,4 @@ -import { expect, test } from "vitest"; +import { expect, test, vi } from "vitest"; import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; @@ -35,6 +35,20 @@ test("loads exact global enabledModels in configured order", () => { } finally { rmSync(f.root, { recursive: true, force: true }); } }); +test("uses the configured Pi agent directory when no explicit directory is passed", () => { + const f = fixture({ enabledModels: ["zai/glm-5.2"] }); + vi.stubEnv("PI_CODING_AGENT_DIR", f.agentDir); + try { + expect(loadPiEnabledModels({ harnessDir: f.harnessDir })).toMatchObject({ + ids: ["zai/glm-5.2"], + warnings: [], + }); + } finally { + vi.unstubAllEnvs(); + rmSync(f.root, { recursive: true, force: true }); + } +}); + test("project enabledModels overrides global enabledModels", () => { const f = fixture( { enabledModels: ["zai/glm-5.2", "zai/glm-5v-turbo"] }, diff --git a/backend/test/pi-managed-config.test.ts b/backend/test/pi-managed-config.test.ts new file mode 100644 index 00000000..c17ec3a1 --- /dev/null +++ b/backend/test/pi-managed-config.test.ts @@ -0,0 +1,28 @@ +import { expect, test } from "vitest"; +import { + PI_MANAGED_CONFIG_ERROR_MESSAGE, + configuredPiProviderApiKey, +} from "../src/pi/managed-config.js"; + +test("provider credential declarations are selected from models.json by provider ID", () => { + const raw = JSON.stringify({ + providers: { + hosted: { apiKey: "$HOSTED_API_KEY", models: [{ id: "one" }] }, + local: { apiKey: "local", models: [{ id: "two" }] }, + }, + }); + + expect(configuredPiProviderApiKey(raw, "HOSTED")).toBe("$HOSTED_API_KEY"); + expect(configuredPiProviderApiKey(raw, "local")).toBe("local"); + expect(configuredPiProviderApiKey(raw, "missing")).toBeUndefined(); +}); + +test.each([ + JSON.stringify({ providers: [] }), + JSON.stringify({ providers: { local: "invalid" } }), + JSON.stringify({ providers: { local: { apiKey: 42 } } }), + JSON.stringify({ providers: { local: { apiKey: "!must-not-run" } } }), +])("invalid declarative provider credential configuration fails closed", (raw) => { + expect(() => configuredPiProviderApiKey(raw, "local")) + .toThrow(PI_MANAGED_CONFIG_ERROR_MESSAGE); +}); diff --git a/backend/test/pi-management.test.ts b/backend/test/pi-management.test.ts index 452deae9..f0d79e8d 100644 --- a/backend/test/pi-management.test.ts +++ b/backend/test/pi-management.test.ts @@ -198,7 +198,10 @@ test("smoke uses the configured timeout and reports a sanitized timeout", async message: "Pi smoke check timed out", checkedAt: "2026-08-05T10:00:00.000Z", }); - expect(calls).toEqual([{ command: "/usr/local/bin/pi", args: ["--version"], timeout: 750 }]); + expect(calls).toHaveLength(1); + expect(calls[0]).toMatchObject({ command: "/usr/local/bin/pi", args: ["--version"] }); + expect(calls[0]!.timeout).toBeGreaterThan(0); + expect(calls[0]!.timeout).toBeLessThanOrEqual(750); }); // Catches a smoke endpoint that validates only the Pi binary/model catalogue and never makes a diff --git a/backend/test/pi-process-manager.test.ts b/backend/test/pi-process-manager.test.ts index 47fc5104..7a9d07a0 100644 --- a/backend/test/pi-process-manager.test.ts +++ b/backend/test/pi-process-manager.test.ts @@ -22,7 +22,7 @@ const SAFE_AUTH = '{"deepseek":{"type":"api_key","key":"safe-token"}}\n'; const SAFE_MODELS = [ "{", ' "providers": {', - ' "local-qwen": {"baseUrl":"http://model.invalid/v1","models":[{"id":"qwen"}]}', + ' "local-qwen": {"baseUrl":"http://model.invalid/v1","apiKey":"local","models":[{"id":"qwen"}]}', " }", "}", "", @@ -670,9 +670,22 @@ test.each([["OpenAI", "openai"], ["gemini", "google"]])( }, ); -test.each(["ollama", "local-qwen"])( - "local provider %s spawns without a model key and scrubs ambient credentials", +test.each(["installation-local", "private-compatible"])( + "provider %s configured with a literal apiKey spawns without a managed key", async (provider) => { + const root = mkdtempSync(path.join(tmpdir(), "tht-local-provider-")); + const agentDir = path.join(root, "agent"); + mkdirSync(agentDir, { mode: 0o700 }); + writeFileSync(path.join(agentDir, "models.json"), JSON.stringify({ + providers: { + [provider]: { + baseUrl: "http://model.invalid/v1", + apiKey: "local", + models: [{ id: "model" }], + }, + }, + }), { mode: 0o600 }); + vi.stubEnv("PI_CODING_AGENT_DIR", agentDir); vi.stubEnv("PI_PROVIDER_API_KEY", "ambient-secret"); vi.stubEnv("THT_MODEL_API_KEY_FILE", "/ambient/secret-path"); vi.stubEnv("OPENAI_API_KEY", "unselected-provider-secret"); @@ -690,6 +703,7 @@ test.each(["ollama", "local-qwen"])( } finally { mgr.teardown(`local-session-${provider}`); vi.unstubAllEnvs(); + rmSync(root, { recursive: true, force: true }); } }, ); diff --git a/backend/test/provider-credentials.test.ts b/backend/test/provider-credentials.test.ts index a3b06cb9..5d40a042 100644 --- a/backend/test/provider-credentials.test.ts +++ b/backend/test/provider-credentials.test.ts @@ -117,20 +117,41 @@ test("single-key providers scrub ambient compound companions before injecting th expect(env).not.toHaveProperty("CLOUDFLARE_GATEWAY_ID"); }); -test("local-qwen is an explicit local provider and needs no generic key", () => { +test("a provider with a literal apiKey in models.json needs no code-level provider exception", () => { const env = buildPiChildEnv({ ambient: { PI_PROVIDER_API_KEY: "must-not-leak", OPENAI_API_KEY: "must-not-leak", THT_MODEL_API_KEY_FILE: "/must/not/leak", }, - provider: "local-qwen", + provider: "installation-local", + configuredApiKey: "local", }); expect(env).not.toHaveProperty("PI_PROVIDER_API_KEY"); expect(env).not.toHaveProperty("OPENAI_API_KEY"); expect(env).not.toHaveProperty("THT_MODEL_API_KEY_FILE"); }); +test.each(["$PRIVATE_PROVIDER_API_KEY", "${PRIVATE_PROVIDER_API_KEY}"])( + "a custom provider credential target is derived from models.json: %s", + (configuredApiKey) => { + const env = buildPiChildEnv({ + ambient: { PRIVATE_PROVIDER_API_KEY: "stale" }, + provider: "private-provider", + configuredApiKey, + credentialValue: "selected-secret", + }); + expect(env.PRIVATE_PROVIDER_API_KEY).toBe("selected-secret"); + }, +); + +test("a custom provider cannot redirect a managed credential into a process-control variable", () => { + expect(() => buildPiChildEnv({ + ambient: {}, provider: "private-provider", configuredApiKey: "$PATH", + credentialValue: "selected-secret", + })).toThrow("model provider credential is unavailable"); +}); + test("credential status reports only present or missing without treating local providers as credentialed", () => { expect(piProviderCredentialStatus({ provider: "deepseek", @@ -139,7 +160,8 @@ test("credential status reports only present or missing without treating local p })).toBe("present"); expect(piProviderCredentialStatus({ provider: "deepseek" })).toBe("missing"); expect(piProviderCredentialStatus({ - provider: "local-qwen", + provider: "installation-local", + configuredApiKey: "local", resolveCredentialValue: () => "must-not-be-returned", })).toBe("missing"); }); @@ -159,7 +181,8 @@ test("credential status never resolves the generic secret for auth-store or loca resolveCredentialValue: unreadableSecret, })).toBe("present"); expect(piProviderCredentialStatus({ - provider: "local-qwen", + provider: "installation-local", + configuredApiKey: "local", resolveCredentialValue: unreadableSecret, })).toBe("missing"); expect(secretReads).toBe(0); diff --git a/backend/test/qdrant-collection.test.ts b/backend/test/qdrant-collection.test.ts index 917423f4..ea981691 100644 --- a/backend/test/qdrant-collection.test.ts +++ b/backend/test/qdrant-collection.test.ts @@ -28,6 +28,90 @@ const compatible = (size = 1024, distance = "Cosine", schema = payloadSchema) => payload_schema: schema, }); +test("Evidence maintenance adds an absent BM25 vector without changing the dense contract", async () => { + const info = compatible(); + const requests: Array<{ url: string; init?: any }> = []; + const request = async (url: string, init?: any) => { + requests.push({ url, init }); + if (init?.method === "PUT" && /\/vectors\/bm25$/.test(url)) { + expect(JSON.parse(String(init.body))).toEqual({ sparse: { modifier: "idf" } }); + (info.config.params as any).sparse_vectors = { bm25: { modifier: "idf" } }; + return { status: 200, ok: true, json: async () => ({}) } as any; + } + return { status: 200, ok: true, json: async () => ({ result: info }) } as any; + }; + + const result = await reconcileCollection({ + baseUrl: "http://qdrant:6333", collection: "c", dimensions: 1024, distance: "cosine", + mode: "evidence_maintenance", request, + }); + + expect(result).toEqual({ ok: true, state: "upgraded" }); + expect(info.config.params.vectors).toEqual({ size: 1024, distance: "Cosine" }); + expect(requests.filter(({ init }) => init?.method === "PUT")).toHaveLength(1); + expect(requests[1]?.url).toBe("http://qdrant:6333/collections/c/vectors/bm25"); +}); + +test("Evidence maintenance creates a missing collection with both required vector contracts", async () => { + let info: any; + let createdBody: any; + const request = async (url: string, init?: any) => { + if (init?.method === "PUT") { + createdBody = JSON.parse(String(init.body)); + info = { config: { params: createdBody }, payload_schema: payloadSchema }; + return { status: 200, ok: true, json: async () => ({}) } as any; + } + if (info === undefined) return { status: 404, ok: false, json: async () => ({}) } as any; + return { status: 200, ok: true, json: async () => ({ result: info }) } as any; + }; + + const result = await reconcileCollection({ + baseUrl: "http://qdrant:6333", collection: "c", dimensions: 1024, distance: "cosine", + mode: "evidence_maintenance", request, + }); + + expect(result).toEqual({ ok: true, state: "ready" }); + expect(createdBody).toEqual({ + vectors: { size: 1024, distance: "Cosine" }, + sparse_vectors: { bm25: { modifier: "idf" } }, + }); +}); + +test("Evidence maintenance refuses an incompatible BM25 definition without mutating", async () => { + const info = compatible(); + (info.config.params as any).sparse_vectors = { bm25: { modifier: "none" } }; + const requests: Array<{ url: string; init?: any }> = []; + const request = async (url: string, init?: any) => { + requests.push({ url, init }); + return { status: 200, ok: true, json: async () => ({ result: info }) } as any; + }; + + const result = await reconcileCollection({ + baseUrl: "http://qdrant:6333", collection: "c", dimensions: 1024, distance: "cosine", + mode: "evidence_maintenance", request, + }); + + expect(result).toEqual({ ok: false, code: "semantic_index_incompatible" }); + expect(requests.filter(({ init }) => init?.method === "PUT")).toEqual([]); +}); + +test("ordinary session reconciliation does not add BM25", async () => { + const info = compatible(); + const requests: Array<{ url: string; init?: any }> = []; + const request = async (url: string, init?: any) => { + requests.push({ url, init }); + return { status: 200, ok: true, json: async () => ({ result: info }) } as any; + }; + + const result = await reconcileCollection({ + baseUrl: "http://qdrant:6333", collection: "c", dimensions: 1024, distance: "cosine", + mode: "self_heal", request, + }); + + expect(result).toEqual({ ok: true, state: "ready" }); + expect(requests.filter(({ url }) => /\/vectors\/bm25$/.test(url))).toEqual([]); +}); + test("self-heal creates a missing compatible collection", async () => { const r = await reconcileCollection({ baseUrl: "http://qdrant:6333", collection: "c", dimensions: 1024, distance: "cosine", diff --git a/backend/test/session-bridge.test.ts b/backend/test/session-bridge.test.ts index f0a41341..87637866 100644 --- a/backend/test/session-bridge.test.ts +++ b/backend/test/session-bridge.test.ts @@ -42,6 +42,29 @@ test("real Pi thinking_delta becomes a dedicated activity_delta to the FE", () = expect(seen).toEqual([{ type: "activity_delta", text: "Valuto le ambiguità" }]); }); +test("a reserved phase notification becomes a structured phase_started event", () => { + const { rpc, fire } = fakeRpc(); + const bridge = new SessionBridge(rpc); + const seen: any[] = []; + bridge.onClientEvent((event) => seen.push(event)); + + fire({ + type: "extension_ui_request", + method: "notify", + notifyType: "info", + message: "__tht_phase_started__:F2", + }); + fire({ + type: "extension_ui_request", + method: "notify", + notifyType: "info", + message: "__tht_phase_started__:F9_DO_NOT_FORWARD", + }); + + expect(seen).toEqual([{ type: "system_event", event: "phase_started", phase: "F2" }]); + expect(JSON.stringify(seen)).not.toContain("DO_NOT_FORWARD"); +}); + test("assistant message_end exposes sanitized token usage with the configured context window", () => { const { rpc, fire } = fakeRpc(); const bridge = new SessionBridge(rpc); diff --git a/backend/test/windows-auth-storage.test.ts b/backend/test/windows-auth-storage.test.ts index ac8fed9a..c6fc8077 100644 --- a/backend/test/windows-auth-storage.test.ts +++ b/backend/test/windows-auth-storage.test.ts @@ -46,9 +46,9 @@ function realChildBridge( bridge: factory({ thtExecutable: pathStyle === "windows" ? "C:\\tht.exe" : launcher, spawnChild: (_executable, args, options) => spawn(launcher, [...args], options), - // Leave enough startup headroom for a real child under a busy CI host while retaining a - // sub-1.5-second bound from request start through final settlement. - deadlinesForTest: { timeoutMs: 750, terminationGraceMs: 50, finalSettlementMs: 500 }, + // Keep this stricter than the five-second production timeout without assuming that a + // real Node child can always start within 750 ms on a busy shared runner. + deadlinesForTest: { timeoutMs: 2_000, terminationGraceMs: 50, finalSettlementMs: 500 }, ...(mode === "stdin" ? { beforeInputForTest: async () => { await waitForMarker(marker, "stdin-closed"); @@ -585,6 +585,6 @@ describe("Windows auth-storage bridge", () => { await expect(outcome).resolves.toMatchObject({ message: "auth_session_store_invalid" }); await waitForMarker(marker, "terminated"); - expect(Date.now() - startedAt).toBeLessThan(1_500); + expect(Date.now() - startedAt).toBeLessThan(3_000); }, 5_000); }); diff --git a/backend/test/workspace-preprocessing-service.test.ts b/backend/test/workspace-preprocessing-service.test.ts index 2055b96b..482d9f18 100644 --- a/backend/test/workspace-preprocessing-service.test.ts +++ b/backend/test/workspace-preprocessing-service.test.ts @@ -161,6 +161,7 @@ function fixture(workspace = baseWorkspace) { runChild, listSessions: async () => [], semanticPreflight: async () => ({ ok: true }), + evidencePreflight: async () => ({ ok: true }), }); return { dataRoot, runChild, requests, service }; } @@ -283,6 +284,7 @@ test("index schema fails closed when semantic preflight refuses the collection", runChild, listSessions: async () => [], semanticPreflight: async () => ({ ok: false, code: "semantic_index_incompatible" }), + evidencePreflight: async () => ({ ok: true }), }); const result = await service.indexSchema({ workspaceId: "psd-clinical" }); @@ -309,6 +311,7 @@ test("filesystem Evidence proceeds after materialization and private HTTP hosts runChild: vi.fn(), listSessions: async () => [], semanticPreflight: async () => ({ ok: true }), + evidencePreflight: async () => ({ ok: true }), httpPrivateHostAllowlist: ["metadata.internal"], }); @@ -513,6 +516,7 @@ test("vector rebuild recreates the full collection contract including keyword in runChild: vi.fn(), listSessions: async () => [], semanticPreflight: async () => ({ ok: true }), + evidencePreflight: async () => ({ ok: true }), }); // replace global fetch used by vectorRebuild/reconcileCollection const original = globalThis.fetch; diff --git a/backend/test/workspace-registry-deployment.test.ts b/backend/test/workspace-registry-deployment.test.ts index 3c205843..632fbcec 100644 --- a/backend/test/workspace-registry-deployment.test.ts +++ b/backend/test/workspace-registry-deployment.test.ts @@ -293,6 +293,9 @@ test("Windows clone contract copies the shared complete schema v3 descriptor int expect(windows).toContain('thoth-workspaces.yaml'); expect(windows).toContain('$workspaceDestination = Join-Path $workspaceDirectory "workspace.yaml"'); expect(windows).toContain('Join-Path $workspaceEvidence "guide.md"'); + expect(windows).toContain('THT_AUTH_CONFIG_ROOT=$authConfigRoot'); + expect(windows).toContain('"core,embedding,embedding-model-init,frontend,qdrant"'); + expect(windows).toContain('"core,embedding,frontend,qdrant"'); expect(windows).not.toContain('schema_version: 3'); expect(descriptor).toMatchObject({ workspace: { diff --git a/backend/test/workspace-runtime-renderer.test.ts b/backend/test/workspace-runtime-renderer.test.ts index 8a151f38..d5d3c5ea 100644 --- a/backend/test/workspace-runtime-renderer.test.ts +++ b/backend/test/workspace-runtime-renderer.test.ts @@ -139,8 +139,14 @@ function evidenceSecretFile(name: string, contents: string): string { return path; } -function evidenceWorkspace(source: Record, policy?: Record) { - return parseWorkspaceYaml(`${canonicalEvidenceWorkspace}\nevidence:\n source: ${JSON.stringify(source)}${ +function evidenceWorkspace( + source: Record, + policy?: Record, + evidenceSchemaVersion?: number, +) { + return parseWorkspaceYaml(`${canonicalEvidenceWorkspace}\nevidence:${ + evidenceSchemaVersion === undefined ? "" : `\n schema_version: ${evidenceSchemaVersion}` + }\n source: ${JSON.stringify(source)}${ policy === undefined ? "" : `\n policy: ${JSON.stringify(policy)}` }\n`); } @@ -180,9 +186,10 @@ function evidenceRender( source: Record, evidenceBinding: RuntimeBindings["evidence"] = { missing: [], values: {} }, policy?: Record, + evidenceSchemaVersion?: number, ) { return renderRuntimeConfig( - evidenceWorkspace(source, policy), + evidenceWorkspace(source, policy, evidenceSchemaVersion), { ...directBindings, evidence: evidenceBinding }, paths, evidenceContext, @@ -195,15 +202,16 @@ test("renders filesystem Evidence below the immutable revision content root with const yaml = evidenceRender({ type: "filesystem", uri: "psd-clinical/evidence", - }); + }, undefined, undefined, 2); const rendered = parse(yaml); expect(rendered.runtime_identity.workspace_revision).toBe(evidenceRevision); expect(rendered.evidence).toEqual({ + schema_version: 2, sources: [{ type: "filesystem", root: `/srv/registry/snapshots/${evidenceRevision}/psd-clinical/evidence`, - patterns: ["**/*.md"], + patterns: ["curated/**/*.md"], max_bytes: 10_485_760, }], }); diff --git a/backend/test/workspaces-git-repository.test.ts b/backend/test/workspaces-git-repository.test.ts index 22004a00..659b921a 100644 --- a/backend/test/workspaces-git-repository.test.ts +++ b/backend/test/workspaces-git-repository.test.ts @@ -323,16 +323,40 @@ test("parallel contenders recover a stale lock file without overlapping critical const second = new WorkspaceRepositoryLock(locks); let active = 0; let maximum = 0; - const critical = async () => { + let firstEntered!: () => void; + const entered = new Promise((resolve) => { firstEntered = resolve; }); + let releaseFirst!: () => void; + const held = new Promise((resolve) => { releaseFirst = resolve; }); + const firstRun = first.run(async () => { + active += 1; + maximum = Math.max(maximum, active); + firstEntered(); + await held; + active -= 1; + }); + await entered; + const secondRun = second.run(async () => { active += 1; maximum = Math.max(maximum, active); await new Promise((resolve) => setTimeout(resolve, 25)); active -= 1; - }; + }); - const results = await Promise.allSettled([first.run(critical), second.run(critical)]); + let timeout!: ReturnType; + const contender = await Promise.race([ + secondRun.then( + () => ({ status: "fulfilled" as const }), + () => ({ status: "rejected" as const }), + ), + new Promise<{ status: "timed-out" }>((resolve) => { + timeout = setTimeout(() => resolve({ status: "timed-out" }), 2_000); + }), + ]); + clearTimeout(timeout); + releaseFirst(); + await firstRun; + await secondRun.catch(() => undefined); - expect(results.filter((result) => result.status === "fulfilled")).toHaveLength(1); - expect(results.filter((result) => result.status === "rejected")).toHaveLength(1); + expect(contender.status).toBe("rejected"); expect(maximum).toBe(1); }); diff --git a/backend/test/workspaces-schema.test.ts b/backend/test/workspaces-schema.test.ts index 3e93669e..07243da6 100644 --- a/backend/test/workspaces-schema.test.ts +++ b/backend/test/workspaces-schema.test.ts @@ -390,6 +390,78 @@ test("applies filesystem and policy defaults to the canonical descriptor", () => }); }); +test("defaults schema-versioned filesystem Evidence to curated documents only", () => { + const parsed = validateWorkspaceDescriptor({ + ...withEvidence({ + type: "filesystem", + uri: "psd-clinical/evidence", + }), + evidence: { + schema_version: 2, + source: { type: "filesystem", uri: "psd-clinical/evidence" }, + }, + }); + + expect(parsed.evidence).toMatchObject({ + schema_version: 2, + source: { patterns: ["curated/**/*.md"] }, + }); +}); + +test("rejects a schema-versioned Evidence layout that mixes source and curated runtime patterns", () => { + expectSafeEvidenceError({ + ...withEvidence({ + type: "filesystem", + uri: "psd-clinical/evidence", + }), + evidence: { + schema_version: 2, + source: { + type: "filesystem", + uri: "psd-clinical/evidence", + patterns: ["source/**/*.md", "curated/**/*.md"], + }, + }, + }, /source.*curated|curated.*source/i); +}); + +test("rejects a schema-versioned Evidence layout that acquires source documents at runtime", () => { + expectSafeEvidenceError({ + ...withEvidence({ + type: "filesystem", + uri: "psd-clinical/evidence", + }), + evidence: { + schema_version: 2, + source: { + type: "filesystem", + uri: "psd-clinical/evidence", + patterns: ["source/**/*.md"], + }, + }, + }, /curated/i); +}); + +test.each(["curated/**/*.yaml", "curated/**"])( + "rejects a schema-versioned Evidence layout that uses the non-canonical curated pattern %s", + (pattern) => { + expectSafeEvidenceError({ + ...withEvidence({ + type: "filesystem", + uri: "psd-clinical/evidence", + }), + evidence: { + schema_version: 2, + source: { + type: "filesystem", + uri: "psd-clinical/evidence", + patterns: [pattern], + }, + }, + }, /curated\/\*\*\/\*\.md/i); + }, +); + test("keeps evidence optional on schema v3", () => { expect(validateWorkspaceDescriptor(validWorkspaceObject())).not.toHaveProperty("evidence"); }); diff --git a/backend/test/workspaces/evidence/boundary.test.ts b/backend/test/workspaces/evidence/boundary.test.ts new file mode 100644 index 00000000..0c474cf5 --- /dev/null +++ b/backend/test/workspaces/evidence/boundary.test.ts @@ -0,0 +1,34 @@ +import { readdirSync, readFileSync } from "node:fs"; +import { basename, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import ts from "typescript"; +import { expect, test } from "vitest"; + +const evidenceRoot = fileURLToPath(new URL("../../../src/workspaces/evidence/", import.meta.url)); +const coreInfrastructure = new Set([ + "preprocessing-service", + "qdrant-collection", + "registry", +]); + +test("Evidence modules do not import core-owned registry or Qdrant lifecycle", () => { + const violations: Array<{ file: string; dependency: string }> = []; + for (const file of readdirSync(evidenceRoot).filter((name) => name.endsWith(".ts"))) { + const source = ts.createSourceFile( + file, + readFileSync(join(evidenceRoot, file), "utf8"), + ts.ScriptTarget.Latest, + true, + ts.ScriptKind.TS, + ); + for (const statement of source.statements) { + if (!ts.isImportDeclaration(statement) || !ts.isStringLiteral(statement.moduleSpecifier)) { + continue; + } + const dependency = basename(statement.moduleSpecifier.text).replace(/\.js$/, ""); + if (coreInfrastructure.has(dependency)) violations.push({ file, dependency }); + } + } + + expect(violations).toEqual([]); +}); diff --git a/backend/test/evidence-materialization.test.ts b/backend/test/workspaces/evidence/materialization.test.ts similarity index 95% rename from backend/test/evidence-materialization.test.ts rename to backend/test/workspaces/evidence/materialization.test.ts index 2bbfce5e..72490637 100644 --- a/backend/test/evidence-materialization.test.ts +++ b/backend/test/workspaces/evidence/materialization.test.ts @@ -4,9 +4,9 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { promisify } from "node:util"; import { afterEach, expect, test } from "vitest"; -import { GitWorkspaceRepository } from "../src/workspaces/git-repository.js"; -import { materializeEvidenceTree } from "../src/workspaces/evidence-materialization.js"; -import type { WorkspaceRegistryConfig } from "../src/workspaces/types.js"; +import { GitWorkspaceRepository } from "../../../src/workspaces/git-repository.js"; +import { materializeEvidenceTree } from "../../../src/workspaces/evidence/materialization.js"; +import type { WorkspaceRegistryConfig } from "../../../src/workspaces/types.js"; const runFile = promisify(execFile); const temporaryRoots: string[] = []; diff --git a/backend/test/workspaces/evidence/preprocessing.test.ts b/backend/test/workspaces/evidence/preprocessing.test.ts new file mode 100644 index 00000000..143ca260 --- /dev/null +++ b/backend/test/workspaces/evidence/preprocessing.test.ts @@ -0,0 +1,160 @@ +import { expect, test, vi } from "vitest"; +import { + continueEvidencePreprocessing, + preprocessEvidence, + type EvidenceJobState, + type EvidencePreprocessingDependencies, +} from "../../../src/workspaces/evidence/preprocessing.js"; +import type { WorkspaceDescriptor } from "../../../src/workspaces/schema.js"; + +type EvidenceConfig = NonNullable; + +const filesystemEvidence = { + source: { type: "filesystem", uri: "research/evidence" }, +} as EvidenceConfig; + +const privateHttpEvidence = { + source: { + type: "http", + uris: ["http://127.0.0.1/private.md"], + authentication: "none", + connect_timeout_ms: 1000, + read_timeout_ms: 2000, + max_bytes: 100, + max_redirects: 0, + allow_private_hosts: true, + max_cache_bytes: 100, + }, +} as EvidenceConfig; + +function job(overrides: Partial = {}): EvidenceJobState { + return { + runId: "a".repeat(32), + childRuns: {}, + completedStages: [], + ...overrides, + }; +} + +function dependencies(payload: Record = {}): EvidencePreprocessingDependencies & { + runStage: ReturnType; + persistJob: ReturnType; + evidencePreflight: ReturnType; +} { + return { + runStage: vi.fn(async () => payload), + persistJob: vi.fn(), + evidencePreflight: vi.fn(async () => ({ ok: true as const })), + requireRunId(value) { + if (typeof value !== "string" || !/^[0-9a-f]{32}$/.test(value)) { + throw new Error("child run id is invalid"); + } + return value; + }, + numberRecord(value) { + if (!value || typeof value !== "object" || Array.isArray(value)) return undefined; + return Object.fromEntries( + Object.entries(value as Record).map(([key, nested]) => [key, Number(nested)]), + ); + }, + }; +} + +test("Evidence maintenance preflights the additive BM25 contract before starting its stage", async () => { + const state = job({ childRuns: { evidence: "b".repeat(32) } }); + const deps = dependencies({ run_id: "c".repeat(32), counts: { added: 2 } }); + + const result = await preprocessEvidence( + { evidence: filesystemEvidence, job: state, dryRun: false }, + deps, + ); + + expect(deps.evidencePreflight).toHaveBeenCalledOnce(); + expect(deps.runStage).toHaveBeenCalledWith([ + "preprocess", "evidence", "--resume", "b".repeat(32), "--json", "-c", "/dev/fd/3", + ]); + expect(deps.persistJob).toHaveBeenCalledOnce(); + expect(state).toMatchObject({ + childRuns: { evidence: "c".repeat(32) }, + completedStages: ["evidence"], + }); + expect(result).toEqual({ + status: "succeeded", + code: "ok", + runId: "a".repeat(32), + childRuns: { evidence: "c".repeat(32) }, + completedStages: ["evidence"], + counts: { added: 2 }, + }); +}); + +test("owns Evidence egress refusal before shared semantic infrastructure", async () => { + const deps = dependencies(); + + const result = await preprocessEvidence( + { + evidence: privateHttpEvidence, + job: job(), + httpPrivateHostAllowlist: ["metadata.internal"], + }, + deps, + ); + + expect(result).toEqual({ status: "failed", code: "egress_policy_refused" }); + expect(deps.runStage).not.toHaveBeenCalled(); + expect(deps.persistJob).not.toHaveBeenCalled(); +}); + +test("projects aggregate no-Evidence and completed-stage outcomes without rerunning", async () => { + const deps = dependencies(); + const noEvidence = await continueEvidencePreprocessing( + { + evidence: undefined, + job: job({ completedStages: ["dwh", "schema_index"] }), + priorCounts: { added: 2 }, + }, + deps, + ); + const completed = await continueEvidencePreprocessing( + { + evidence: filesystemEvidence, + job: job({ completedStages: ["dwh", "schema_index", "evidence"] }), + }, + deps, + ); + + expect(noEvidence).toMatchObject({ + status: "succeeded", + code: "ok", + warnings: ["workspace has no Evidence source"], + counts: { added: 2 }, + }); + expect(completed).toMatchObject({ + status: "unchanged", + code: "ok", + completedStages: ["dwh", "schema_index", "evidence"], + }); + expect(deps.runStage).not.toHaveBeenCalled(); + expect(deps.persistJob).not.toHaveBeenCalled(); +}); + +test("preserves the narrow standalone projection for an already completed Evidence stage", async () => { + const deps = dependencies(); + + const result = await preprocessEvidence( + { + evidence: filesystemEvidence, + job: job({ childRuns: { evidence: "b".repeat(32) }, completedStages: ["evidence"] }), + }, + deps, + ); + + expect(result).toEqual({ + status: "unchanged", + code: "ok", + runId: "a".repeat(32), + completedStages: ["evidence"], + }); + expect(deps.runStage).not.toHaveBeenCalled(); + expect(deps.persistJob).not.toHaveBeenCalled(); +}); diff --git a/brain/codebase/datamart-builder-deployment-gotchas.md b/brain/codebase/datamart-builder-deployment-gotchas.md deleted file mode 100644 index 8dbceec4..00000000 --- a/brain/codebase/datamart-builder-deployment-gotchas.md +++ /dev/null @@ -1,14 +0,0 @@ -# Datamart Builder deployment gotchas - -- Il percorso pubblico attraversa due reverse proxy: nginx host → nginx Omics Portal → core ThothII. -- Per SSE, ogni livello deve disabilitare `proxy_buffering`, `proxy_request_buffering` e cache, usare HTTP/1.1, timeout lunghi e propagare `X-Accel-Buffering: no`. -- Il core aggiunge `X-Accel-Buffering: no` alla risposta EventSource; Omics Portal lo riaggiunge esplicitamente per i proxy a monte. -- Una sonda utile deve attraversare il portale autenticato e misurare l'arrivo degli header `200 text/event-stream`, non solo interrogare il core nel network Docker. -- La configurazione nginx host attiva vive in `/etc/nginx/sites-available/policlinicosandonato`; validare con `nginx -t` prima del reload. -- Django può tenere in memoria il manifest Vite per worker: dopo un rebuild del frontend riavviare anche i worker Omics Portal, altrimenti richieste diverse possono produrre hash asset vecchi e nuovi. -- Nel profilo server legacy, `vector_db` è una connessione pgvector RW condivisa; il factory può riusarla come writer solo con `profile=server`. La workstation senza writer REST deve restare read-only. -- Non convertire in-place `local.yaml` da chiavi legacy a risorse moderne durante un incident fix: cambia il binding del workspace e può invalidare gli snapshot DWH attivi. -- `session_storage` è configurazione di persistenza delle sessioni, non degli artefatti DWH: escluderla dall'impronta in `config_dwh_binding`, altrimenti l'attivazione di profili utente rende incompatibile un indice DWH già valido. -- Con profili per utente, un profilo privato vuoto deve essere inizializzato una sola volta dai settings legacy completi; trattare `{}` come settings effettivi fa creare sessioni senza provider/modello e lo spawn Pi fallisce prima di partire. - -- Regola operativa concordata: dopo ogni modifica o enhancement, ricostruire e ricreare tutti i container ThothII coinvolti prima dell'handoff, quindi verificare che siano healthy affinché l'utente possa testare live. Il push Git non aggiorna le immagini Docker automaticamente. diff --git a/brain/codebase/pi-model-selection.md b/brain/codebase/pi-model-selection.md deleted file mode 100644 index 37a489bf..00000000 --- a/brain/codebase/pi-model-selection.md +++ /dev/null @@ -1,11 +0,0 @@ -# Pi model selection - -- `get_available_models` restituisce tutti i modelli con autenticazione/configurazione disponibile; non applica `settings.json.enabledModels`. -- Il selettore web usa l'intersezione esatta tra catalogo RPC e `enabledModels`, preservando l'ordine configurato e confrontando `provider/model`. -- In produzione `PI_PROVIDER` può essere assente: il provider corrente vive in `/data/settings/settings.json`; l'enumerazione Pi deve quindi essere provider-neutral e usare il profilo montato per l'autenticazione. -- Lo spawn di enumerazione deve comunque passare da `buildPiChildEnv({})` per rimuovere credenziali ambientali e metadati dei secret. -- `local-qwen` è un provider custom locale esplicito; non richiede la chiave generica dei provider hosted. Provider sconosciuti e composti restano fail-closed. -- Il profilo Pi live è montato da `/home/chirone/thothii-data/pi-config` a `/home/thoth/.pi`; non leggere né stampare mai i valori di `auth.json`. -- Il provider live `local-qwen` usa `http://localllm-vllm:8000/v1` e il modello `qwen3.6-35b-a3b`; il file montato conserva owner/mode e non contiene modifiche alle credenziali. -- In Compose solo `core` entra nella rete esterna `localllm_default`, oltre alla rete del portale; `frontend` non deve avere accesso diretto alla rete del modello. -- La connettività è verificata dal namespace di `core`: catalogo `/v1/models`, modello atteso presente e completion reale non vuota, senza stampare il testo della risposta. diff --git a/brain/codebase/psd-dwh-transport.md b/brain/codebase/psd-dwh-transport.md deleted file mode 100644 index a4391f88..00000000 --- a/brain/codebase/psd-dwh-transport.md +++ /dev/null @@ -1,20 +0,0 @@ -# PSD DWH transport - -- Il workspace PSD supporta `rest_api` e `postgres_direct`; il trasporto è scelto dall'installazione. -- Il Mac usa REST/PostgREST; il ThothII installato sul server PSD usa PostgreSQL diretto. -- Il server deve usare un ruolo DWH dedicato: `USAGE` sullo schema e `SELECT` soltanto, verificati sui grant reali. -- La difesa read-only applicativa aggiunge: SQL strutturalmente SELECT-only, transazione `READ ONLY`, timeout, limite e rollback. -- La credenziale DWH REST `X-API-Key` non deve essere fornita al ThothII server. -- REST è il trasporto stabile per il Mac e per future installazioni remote che non possono aprire tunnel SSH. -- L'autenticazione REST target deve dare a ogni installazione un'identità separata, revocabile e auditabile; una chiave globale condivisa non scala. -- La rotazione della chiave globale esposta richiede una finestra dual-key che preservi il Mac prima della revoca. -- Il certificato REST corrente resta invariato: è self-issued e viene presentato anche dall'endpoint esterno; i client che non lo considerano già trusted richiedono `TLS_CA_FILE`. -- Il manuale deve coprire consegna e fingerprint della CA, scadenza/rinnovo coordinato e il fatto che i SAN correnti coprono `.it`, non `.com`. -- Non esiste un ambiente di test PSD: le rotazioni devono usare backup, dual-key, probe read-only e rollback sull'endpoint di produzione. -- Supabase Studio è un pannello amministrativo loopback, non un data-plane o un trasporto per ThothII. -- Le credenziali DWH, session storage e amministrazione Supabase devono restare separate. -- Il collegamento container→PostgreSQL deve usare un endpoint host/rete esplicito e ristretto; il loopback dell'host non è il loopback del container. -- Il nightly ETL PSD delle 03:00 usa PostgreSQL diretto; non è un consumer della route REST `/dwh/`. -- Un preprocessing REST Thoth genera `1 + 3T + Ct + Ce` richieste: una lista tabelle, tre RPC per tabella e due famiglie di campionamento testuale. -- Per PSD nel run 2026-08-13: `T=163` e `Ct+Ce=1180`, quindi 1670 richieste per ciclo; undici rerun spiegano 18.370 richieste. -- I repository server esistenti contengono materiale sensibile hardcoded: non copiarlo; inventariare, rimuovere dal tracking e ruotare i segreti coinvolti. diff --git a/brain/codebase/workflow-ui-contracts.md b/brain/codebase/workflow-ui-contracts.md deleted file mode 100644 index bae08bb0..00000000 --- a/brain/codebase/workflow-ui-contracts.md +++ /dev/null @@ -1,44 +0,0 @@ -# ThothII workflow UI contracts - -- Pi reasoning arrives as nested `message_update.assistantMessageEvent.type = thinking_delta`. - The backend maps it to the named SSE event `activity_delta`; the frontend EventSource must - explicitly subscribe to that name. Model activity is separate from final `text_delta` output. -- Session create/resume must preserve the configured or persisted thinking level. Forcing - `thinking: off` disables the upstream signal and makes the activity panel legitimately empty. -- A join-only `reviewer_decide` proposal is one complete, atomic join set. The read-only - `join-review` widget persists every proposed join on Continue; `Other — specify` persists none - and requires the model to propose the complete corrected set again. Ledger read, sequence - assignment, and atomic replacement share a per-session cross-process writer lock. -- A v2 phase summary accepts `open_questions?: string[]`. Validate this at the gate boundary and - normalize legacy malformed entries defensively in the viewer so one object cannot crash React. -- `SqlViewer`'s horizontal/vertical layout control is meaningful only with multiple SQL blocks; - hide it for the single CTE result shown by `CteResultViewer`. -- A Pi turn is `idle`, `running`, `waiting`, or `failed`. A reviewer gate/request moves it to - `waiting`; the reviewer response and steering move it back to `running`; provider failures and - unexpected Pi child exits mark it `failed` without forwarding raw failure detail. -- Resume preserves `running`/`waiting` runtimes. After manifest/readiness validation, every cold - path clears old SSE buffer/subscribers before reopen—including when a crashed child has already - left no runtime—and reuses persisted provider/model/thinking. Failed validation does not clear. -- A successful Resume of the already selected session increments the stream generation so React - closes the old EventSource and opens the same session URL again. Failed Resume must not reconnect. -- SSE endpoints are intentionally keep-alive. Browser cleanup and one-off probes must explicitly - close the EventSource or cancel/abort the response reader after their terminal event. -- `activityLog` remains the complete in-memory chronological fold. The left Model activity panel - default-denies every kind except prompt, thinking, and assistant, presenting them as Question, - Reasoning, and Response in source order; status, tool, gate, lifecycle, and unknown kinds stay - hidden. Its desktop width is pointer/keyboard resizable from 288–576 px while preserving 512 px - centrally, persists globally in localStorage, and becomes an overlay drawer below `lg`. -- F6 CTE cards keep their semantic structure, responsive grids, 12/16 px lateral padding, and 8 px - header/content edge padding. Internal section gaps are 12 px, headings/dividers use 4 px spacing, - and table/filter rows use 4 px vertical padding with compact line heights. -- Pi tool events may cross the backend/client boundary only as call id, tool name, and - `running`/`completed`/`failed` status. Tool updates, arguments, partial/final results, commands, - raw output, and raw errors remain server-side. -- The frontend uses Tailwind CSS 3.4. Shared primitives must use concrete Tailwind 3-compatible - spacing utilities; Tailwind 4 custom-spacing syntax can compile to no effective padding here. -- A native EventSource can retain a `Last-Event-ID` from an older backend process. If that cursor - is newer than every id produced by the current `SseHub` generation, treat it as stale and replay - the fresh generation from id 0 instead of suppressing all new low-id events. -- Batch delete mutates client selection, active-session, panel, and Resume intent only for ids whose - DELETE actually succeeded. A failed active delete preserves its live binding, and deleting an - unrelated session must not invalidate a concurrent Resume targeting another id. diff --git a/brain/index.md b/brain/index.md deleted file mode 100644 index 15288d32..00000000 --- a/brain/index.md +++ /dev/null @@ -1,6 +0,0 @@ -# Brain - -- [[codebase/datamart-builder-deployment-gotchas]] -- [[codebase/pi-model-selection]] -- [[codebase/psd-dwh-transport]] -- [[codebase/workflow-ui-contracts]] diff --git a/deploy/compose.preprocess.yaml b/deploy/compose.preprocess.yaml deleted file mode 100644 index 4873ba99..00000000 --- a/deploy/compose.preprocess.yaml +++ /dev/null @@ -1,44 +0,0 @@ -# Retired for operator use: this profile remains only as a non-public engine-fixture path. -# It exercises the legacy preprocessing fixtures and must not become a second operator interface. -services: - preprocess-evidence: - image: thothii-core:local - profiles: [preprocess] - build: - context: . - dockerfile: docker/core.Dockerfile - entrypoint: [sh, -ec] - command: ["mkdir -p /data/workspaces/preprocess-evidence && exec /app/docker/core-entrypoint.sh preprocess evidence --json -c /app/harness/workspaces/preprocess-evidence.yaml"] - environment: - THT_DATA_ROOT: /data - THT_SECRETS_FILE: /run/secrets/thothii.secrets - secrets: [{source: thothii_secrets, target: thothii.secrets}] - volumes: - - thoth_data:/data - - ./deploy/workspaces:/app/harness/workspaces:ro - restart: "no" - depends_on: - qdrant: - condition: service_healthy - embedding-model-init: - condition: service_completed_successfully - - preprocess-dwh: - image: thothii-core:local - profiles: [preprocess] - build: - context: . - dockerfile: docker/core.Dockerfile - entrypoint: [sh, -ec] - command: ["mkdir -p /data/workspaces/preprocess-dwh && exec /app/docker/core-entrypoint.sh preprocess dwh --steps introspect --json -c /app/harness/workspaces/preprocess-dwh.yaml"] - environment: - THT_DATA_ROOT: /data - THT_SECRETS_FILE: /run/secrets/thothii.secrets - secrets: [{source: thothii_secrets, target: thothii.secrets}] - volumes: - - thoth_data:/data - - ./deploy/workspaces:/app/harness/workspaces:ro - restart: "no" - -volumes: - thoth_data: diff --git a/deploy/pi/models.json b/deploy/pi/models.json index 75e48255..591d2da8 100644 --- a/deploy/pi/models.json +++ b/deploy/pi/models.json @@ -13,6 +13,34 @@ "maxTokens": 131072 } ] + }, + "local-qwen": { + "name": "Local Qwen", + "baseUrl": "https://ml-aritmolab.policlinicosandonato.it/v1", + "api": "openai-completions", + "apiKey": "local", + "models": [ + { + "id": "qwen3.6-35b-a3b", + "name": "Qwen3.6 35B A3B", + "reasoning": false, + "input": ["text"], + "cost": { + "input": 0, + "output": 0, + "cacheRead": 0, + "cacheWrite": 0 + }, + "contextWindow": 131072, + "maxTokens": 16384, + "compat": { + "supportsDeveloperRole": false, + "supportsReasoningEffort": false, + "supportsStore": false, + "maxTokensField": "max_tokens" + } + } + ] } } } diff --git a/deploy/pi/settings.json b/deploy/pi/settings.json index 08f6d1b4..6693eed8 100644 --- a/deploy/pi/settings.json +++ b/deploy/pi/settings.json @@ -4,6 +4,6 @@ "zai/glm-5.3", "deepseek/deepseek-v4-flash", "deepseek/deepseek-v4-pro", - "aritmolab/qwen3.6-35b-a3b" + "local-qwen/qwen3.6-35b-a3b" ] } diff --git a/deploy/workspaces/preprocess-dwh.yaml b/deploy/workspaces/preprocess-dwh.yaml deleted file mode 100644 index d2f3edff..00000000 --- a/deploy/workspaces/preprocess-dwh.yaml +++ /dev/null @@ -1,11 +0,0 @@ -language: en -dwh: - type: postgres_direct - connection: - host: "${THT_PREPROCESS_DWH_HOST:-dwh}" - port: "${THT_PREPROCESS_DWH_PORT:-5432}" - database: "${THT_PREPROCESS_DWH_DATABASE:-warehouse}" - schema: "${THT_PREPROCESS_DWH_SCHEMA:-public}" - user: "${THT_PREPROCESS_DWH_USER:-thoth_reader}" - password_file: "${THT_PREPROCESS_DWH_PASSWORD_FILE:-/run/secrets/preprocess-dwh-password}" -roots: {artifacts: artifacts, indexes: indexes, sessions: sessions} diff --git a/deploy/workspaces/preprocess-evidence.yaml b/deploy/workspaces/preprocess-evidence.yaml deleted file mode 100644 index 444b4844..00000000 --- a/deploy/workspaces/preprocess-evidence.yaml +++ /dev/null @@ -1,22 +0,0 @@ -language: en -dwh: - type: postgres_direct - connection: - host: "${THT_PREPROCESS_DWH_HOST:-unused}" - port: "${THT_PREPROCESS_DWH_PORT:-5432}" - database: "${THT_PREPROCESS_DWH_DATABASE:-unused}" - schema: "${THT_PREPROCESS_DWH_SCHEMA:-public}" - user: "${THT_PREPROCESS_DWH_USER:-unused}" - password_file: "${THT_PREPROCESS_DWH_PASSWORD_FILE:-/run/secrets/preprocess-dwh-password}" -vectors: - type: qdrant - base_url: http://qdrant:6333 - collection: preprocess-evidence -roots: {artifacts: artifacts, indexes: indexes, sessions: sessions} -evidence: {source_root: /data/source, evidence_dir: evidence} -embeddings: - provider: ollama_internal - base_url: http://embedding:11434 - model: qwen3-embedding:0.6b - dim: 1024 - batch_size: 32 diff --git a/docker/smoke/core-smoke.sh b/docker/smoke/core-smoke.sh index 82b8c745..50d0c78a 100755 --- a/docker/smoke/core-smoke.sh +++ b/docker/smoke/core-smoke.sh @@ -21,7 +21,7 @@ test ! -e /var/run/docker.sock touch /data/.core-smoke-writable rm /data/.core-smoke-writable -/app/docker/core-entrypoint.sh server & +AUTH_MODE=upstream /app/docker/core-entrypoint.sh server & server_pid=$! trap 'kill "$server_pid" 2>/dev/null || true; wait "$server_pid" 2>/dev/null || true' EXIT INT TERM diff --git a/docs/contracts/tht-pi.md b/docs/contracts/tht-pi.md deleted file mode 100644 index bc577cdd..00000000 --- a/docs/contracts/tht-pi.md +++ /dev/null @@ -1,245 +0,0 @@ -# `tht pi` lifecycle contract - -`tht` is the only component that drives Docker lifecycle operations. The `core` container -does not mount a Docker socket, and Pi is never updated in a running container. - -## Inspection and configuration - -```text -tht pi status -tht pi doctor -tht pi test -tht pi logs -tht pi configure -``` - -When `--installation` is omitted, `tht` first uses `THOTHII_INSTALLATION` and otherwise -discovers one valid `thothii-installation.yaml` in the current project tree, including an immediate -`deploy/*` directory. Use `--installation /absolute/path/thothii-installation.yaml` as an explicit -override when the descriptor is outside that tree or more than one installation is available. - -`status` executes the image-bundled `pi --version`. `doctor` compares that value with both the -container's `PI_VERSION` contract and the `io.thothii.pi.version` image label; a merely nonempty -version is not sufficient. `doctor` and `test` also require a healthy core, a successful Pi smoke, -valid settings, and an exact selected provider/model pair from the backend's available model -entries. `pi check` remains an alias for `pi test`. Logs are always a bounded, sanitized 200-line -snapshot; there is no follow mode. - -On a TTY, `pi configure` presents numbered provider, model, and thinking choices. Providers and -models come from the backend's closed model list, and the model choices are restricted to the -selected provider. In non-interactive use, all choices must be explicit: - -```text -tht pi configure \ - --provider zai --model glm-5.2 --thinking medium -``` - -The helper snapshots exact settings-file existence and raw bytes, applies the new values atomically, -and verifies the readback and rendered-configuration digest. A helper, readback, or digest failure -restores those exact bytes when the prior file existed; on a clean installation it removes the new -file and verifies the absent/default state. Empty prior files are supported. The command reports -the actual host path from `PI_AUTH_FILE`; credentials remain in that protected host file and must -never be passed as flags. - -Installation-managed Pi provider/model configuration is declarative only. Any JSON value beginning -with `!` is rejected recursively in the complete `models.json` before it can supply management -choices, and the exact selected provider/model and credential payload is checked again before the -isolated smoke files are written. The API returns only the fixed -`Pi provider/model configuration is invalid` message; rejected commands, paths, and secrets are -never included. Use `$NAME`/`${NAME}` environment references in `models.json`, or omit `apiKey` and -provide the selected credential through the protected `PI_AUTH_FILE`, `THT_MODEL_API_KEY_FILE`, or -`THT_SECRETS_FILE` contract. A literal leading exclamation mark uses Pi's `$!` escape. Direct -secret-file references are not a `models.json` feature: ThothII converts its managed key source to -the provider-native child environment, while `PI_AUTH_FILE` is mounted as Pi's protected credential -store. - -## Supported Compose entry points and current image - -Use `tht start`, `stop`, `status`, `logs`, and `doctor` for ordinary installation lifecycle -operations. All `tht` Compose commands automatically include the installation-specific -durable selector when it exists: - -```text -/.tht//current-image.yaml -``` - -This selector is part of the supported installation state: it keeps a verified Pi image selected -across a fresh `tht` process, stop/start, reconcile, and source checkout whose base image is -digest-pinned. Do not delete or hand-edit it. Direct raw `docker compose` lifecycle commands bypass -this protection and are unsupported. Advanced documented Compose rendering must use -`scripts/compose-with-preflight.sh` and include the same selector with `-f` when present; connector -secret overrides must never bypass that preflight wrapper. - -## Reloading Pi configuration - -Configuration reload is a separate lifecycle operation from an image update: - -```text -tht pi restart --yes [--drain] -``` - -`--yes` is required after reviewing the planned core recreation. Restart activates the durable -maintenance gate before it checks sessions. Without `--drain`, active open sessions refuse the -command. With `--drain`, the command polls the authenticated session inventory until no active -sessions remain; it never terminates sessions and the wait is bounded. - -Restart retains the exact current image and never builds, pulls, or upgrades an image. Before any -core mutation, it tags the captured running image ID with a transaction-scoped reference and -selects that reference through a lifecycle-only Compose override. A configured mutable tag moving -after capture therefore cannot change the restarted image. It recreates only `core` with -`--no-deps --force-recreate --no-build --pull never`; `frontend` and named volumes are not -recreated. Before reopening admission, it verifies health, the unchanged Pi version, the -provider/model/settings smoke, unchanged non-secret rendered configuration, the captured image -identity, and the complete persistence-mount fingerprint. - -Restart and update keep separate recovery state: - -```text -/.tht//restart-state.json -/.tht//update-state.json -``` - -The files are mode `0600` and share one installation lifecycle lock, so restart, update, and -rollback cannot race. Every mutating lifecycle command checks both files. Malformed or non-terminal -restart recovery state blocks update and rollback; malformed or incomplete update recovery state -blocks restart. A verified terminal restart state is cleaned up safely before a later mutation. -After core mutation, a restart failure leaves admission gated and preserves both -`restart-state.json` and its exact-image override; the operator must use status/logs and maintenance -recovery rather than deleting recovery material. - -## Updating Pi - -The normal update uses the repository's pinned version and build source automatically: - -```text -tht pi update -tht pi update --version 0.81.0 -``` - -With no `--version`, the command reads the single default `ARG PI_VERSION=` from -`docker/core.Dockerfile` in the selected project. The normal path confirms the explicit update -command, drains active sessions without terminating them, builds the candidate, recreates only -`core`, verifies it, and promotes it transactionally. - -Advanced registry updates remain available and require an immutable digest: - -```text -tht pi update \ - --version 0.81.0 --source build --yes --drain - -tht pi update \ - --version 0.81.0 --source pull \ - --image registry.example.invalid/thothii-core@sha256:<64-lowercase-hex-digits> --yes -``` - -`--source build` rebuilds only `core` with `PI_VERSION=`. `--source pull` requires an -immutable digest reference; mutable tags, URL forms, and credential-bearing references are -rejected. `--source` is never inferred. - -Before inventory, update activates the durable maintenance gate. Activation writes -`/data/settings/maintenance.json` in the mounted settings volume, closes admission, and waits for -all leases. A recreated candidate reads that marker at startup and therefore starts gated. The -loopback-only control endpoints cannot be reached through the frontend proxy and do not depend on -the configured authentication principal mode. Lost activation/deactivation responses are resolved -by querying gate status only when the original result is unknown. An explicit file or directory -durability failure is never converted to success by matching readback: the control API reports -`maintenance_durability_failed`, keeps or restores the safest durable marker state, and requires -recovery. - -Open, unarchived sessions stop an update. After an operator has completed or otherwise drained -their work, `--drain` makes the command poll the authenticated bare-array -`GET /sessions?scope=all` response until no active sessions remain. - -The configured `core.image` is never retagged or mutated. Each installation transaction creates -unique candidate and previous tags, including when two installations share a configured tag or -the configured image is digest-pinned. A temporary lifecycle-only Compose override selects those -tags for build, recreate, and rollback. Candidate build, pull, or candidate-tag failures happen -before `mutation_started` and therefore never recreate or roll back core. After verification, the -temporary candidate selector is atomically promoted to the durable current-image override. -Rollback atomically promotes the previous selector. Terminal cleanup removes only transaction -files and never deletes the durable selector. - -Only `core` is recreated, with `--no-deps --force-recreate`; `frontend` is not recreated and no -volume-replacement flags are used. Verification checks health; exact requested Pi version at all -three declared boundaries (the candidate executable, `PI_VERSION` environment, and -`io.thothii.pi.version` image label); the provider/model/settings smoke; unchanged -non-secret rendered configuration; and the complete persistence-mount fingerprint. - -## Recovery, rollback, and maintenance cleanup - -Recovery state and lock diagnostics live under: - -```text -/.tht//update-state.json -/.tht//restart-state.json -/.tht//*.lock.owner.json -``` - -Each recovery file is mode `0600`. Update state records transaction-scoped image identities, mount -fingerprints, target version/source, configuration digest, phase, and timestamp; restart state -records the retained image and its verification inputs. Neither file contains credentials, endpoint -values, secret paths, Compose output, or logs. A cross-platform OS advisory file lock serializes -both lifecycle operations; a crashed owner releases the lock automatically. Owner metadata is -diagnostic only and cannot wedge acquisition if empty, partial, or stale. - -Any post-candidate failure explicitly confirms or reactivates maintenance and rescans sessions -before compensation. Automatic rollback selects the transaction's previous image through the -lifecycle override and clears maintenance only after the previous image, configuration, mounts, -health, Pi smoke, and terminal recovery write are verified. Ambiguous compensation remains gated. -If the candidate core is stopped and cannot serve the maintenance endpoint, rollback proves that -state with Compose and writes the marker through a one-off previous-image `core` container sharing -the settings volume. It does not require the failed candidate, a host Node runtime, or the Docker -socket inside a container. The restored core is then recreated, verified, and rescanned before the -gate can open. - -For a failed update with `update-state.json`, first run: - -```text -tht pi rollback --yes -``` - -Rollback restores the image recorded in update state, but it checks restart state before making any -change. A failed, pending, or malformed restart state rejects rollback. A failed restart retains its -captured image and has no candidate image to roll back; first inspect status and logs, repair the -reported problem, then use maintenance recovery. - -Inspect and clean a stale durable gate with: - -```text -tht pi maintenance status -tht pi maintenance recover --yes -``` - -`maintenance recover` restores the captured restart image pin and lifecycle override when needed, -verifies and removes interrupted restart recovery material, and only then processes update state. -It completes an interrupted verified-image promotion, safely finalizes a preparation interrupted -before core mutation, and refuses other pending mutations. For terminal or absent recovery state, -it removes only a stale transaction override, verifies the running installation when the gate is -active, and only then removes the durable marker and reopens admission. It never removes -`current-image.yaml`. If rollback or recovery fails, leave the marker in place, preserve the -relevant recovery state and override, repair the reported Docker/configuration issue, and rerun -rollback or maintenance recovery. - -Missing confirmation, invalid arguments, active sessions, and an interrupted transaction exit -`2`. Docker and verification failures exit nonzero with concise, redacted guidance. Direct -read-only/log commands preserve the original Docker child exit code. - -## Go dependency security boundary - -The supported toolchain is Go `1.26.5`, released 2026-07-07, with module language version -`1.26.0`. The Docker builder is pinned by both patch tag and the multi-platform manifest-list -digest: - -```text -golang:1.26.5-bookworm@sha256:1ecb7edf62a0408027bd5729dfd6b1b8766e578e8df93995b225dfd0944eb651 -``` - -That manifest provides both `linux/amd64` and `linux/arm64/v8` builders. Go's official release -history is the authority for the patch level (`https://go.dev/doc/devel/release`); the Docker -Official Image is the authority for the builder (`https://hub.docker.com/_/golang`). -`golang.org/x/sys`, used by the Windows durable-replace implementation, is pinned to `v0.47.0`. -The directly used `github.com/sirupsen/logrus` is pinned to `v1.9.1`, which removes -GO-2025-4188 from the imported package set. The build contract verifies the exact toolchain, -dependencies, digest, and all five supported target builds (Windows amd64, Darwin amd64/arm64, and -Linux amd64/arm64). `go mod verify`, tests including the race detector, `go vet`, and -`govulncheck` are release gates. diff --git a/docs/contracts/workspace-evidence-v3.md b/docs/contracts/workspace-evidence-v3.md index da5115a6..0e20e9cb 100644 --- a/docs/contracts/workspace-evidence-v3.md +++ b/docs/contracts/workspace-evidence-v3.md @@ -47,6 +47,26 @@ evidence/ `source/`, the manifest, the evaluation set, and other support files are materialized for traceability but never acquired by v2 runtime preprocessing. +### Curated unit representation + +The `schema_version` inside each `curated/**/*.md` file is distinct from the workspace descriptor +version above. Unit schema v1 stores the complete typed unit in YAML frontmatter. Unit schema v2 +keeps short metadata in frontmatter and stores the typed payload in the body. Both remain readable +for compatibility. + +Unit schema v3 stores canonical machine metadata in an invisible `tht:metadata` comment and renders +the complete review surface as deterministic Markdown. It uses headings, paragraphs, wrapping +lists, fenced SQL, blockquotes, and a collapsed technical-details block. It never emits YAML +frontmatter or Markdown tables. Invisible `tht:` comments delimit typed fields. Parsers must reject +missing, duplicate, unknown, desynchronized, or unstructured body content; they must never silently +ignore it. Domain rules also retain their exact canonical text in an invisible `tht:raw-rule` +comment while presenting long prose as paragraphs, labelled subsections, and semicolon-derived +lists. Runtime chunking reads the parsed canonical rule, not this review-only presentation. + +Newly prepared units use v3. `tht evidence migrate ` upgrades v1 and v2 units and +canonicalizes an older v3 presentation locally without a model call, commit, publication, or +semantic change. + ### Example: filesystem ```yaml diff --git a/docs/evidence.md b/docs/evidence.md index 468a6f27..f5c9a82f 100644 --- a/docs/evidence.md +++ b/docs/evidence.md @@ -48,28 +48,57 @@ HTTP and S3 are separate adapters. They do not use the filesystem structure `sou ## What a curated unit must contain -Markdown units read by the legacy CLI loader use YAML frontmatter. The minimum fields are `id` and `title`; `tier`, `status`, `sources`, `tables`, and `concepts` describe the unit's context. +Canonical Curated Evidence v3 hides canonical machine metadata in an HTML comment and renders the +whole review surface as real Markdown. GitHub therefore shows no frontmatter table. The body layout +is deterministic for each Evidence kind: prose uses sections and paragraphs, scopes and enum values +use wrapping lists, formulas use fenced SQL, supporting excerpts use blockquotes, and unresolved +review items use dedicated blocks. Long domain rules are split into readable paragraphs, labelled +subsections, and lists at existing semicolon boundaries. Their exact original text remains canonical +in an invisible marker, so the formatting cannot change their meaning or bytes. + +Preprocessing parses the unit first and builds semantic chunks from the typed payload. The vector +store therefore receives the original rule text and not headings, list markers, or invisible +presentation metadata. ```markdown ---- -id: evidence:autonomia-batteria -title: Nominal battery range -tier: structural -status: reviewed -sources: - - source/domain/bicycle.md -tables: - - bicycle_model -concepts: - - concept:battery-range ---- + +# Fascia pediatrica -Verified definition of nominal range for an electric bicycle model. +> **Dominio** · Italiano +> +> **Scopi:** Disambiguazione -The rule must be atomic enough to cite without reconstructing an entire chapter. The text must distinguish the definition, conditions, and limits. +## Ambito di applicazione + +### Concetti + +- fascia pediatrica + +## Regola + +La fascia pediatrica comprende i pazienti con età inferiore a 18 anni. + +## Estratti di supporto + +> I pazienti sotto i 18 anni sono pediatrici. + +
+Dettagli tecnici e provenienza + +- **ID:** `evidence:fascia-pediatrica` +- **File sorgente:** `source/domain/paziente.md` + +
``` -Curated units must be atomic, readable by a second reviewer, and supported by the source. Provenance references must lead back to the original file and the passage that supports the claim. Do not put secrets, tokens, passwords, or credentials in metadata or URIs. +The actual files contain invisible `tht:` comments for canonical metadata and typed-field +boundaries. Removing, duplicating, or desynchronizing them makes validation fail closed instead of +silently ignoring content. Unit schemas v1 and v2 remain readable for compatibility, but newly +prepared units use v3. + +Curated units must be atomic, readable by a second reviewer, and supported by the source. +Provenance references must lead back to the original file and the passage that supports the claim. +Do not put secrets, tokens, passwords, or credentials in metadata or URIs. The modern canonical form also stores the Evidence kind, provenance, supporting excerpts, `source_file`, and `source_sha256`. The canonical contract rejects unknown fields, mutable metadata, and URIs containing credentials. Identifiers must remain stable even when a unit's kind changes. @@ -145,6 +174,9 @@ tht evidence prepare # Reprocess all sources with the installed pipeline. tht evidence prepare --upgrade +# Rewrite legacy v1/v2 units as table-free v3 Markdown without model calls. +tht evidence migrate + # Validate structure, manifest, links, and review items. tht evidence validate @@ -167,7 +199,9 @@ tht preprocess evidence --config --dry-run tht preprocess evidence --config --resume ``` -`evidence prepare`, `evidence validate`, and `evidence resolve` require the repository path. `preprocess evidence` uses the workspace configuration because it needs the embedding, vector store, retention policy, and artifact directory. +`evidence prepare`, `evidence migrate`, `evidence validate`, and `evidence resolve` require the +repository path. `preprocess evidence` uses the workspace configuration because it needs the +embedding, vector store, retention policy, and artifact directory. Exit codes are part of the operating contract: `evidence validate` returns `0` when the corpus is publishable, `1` for validation errors, and `3` when only review items or orphaned units remain. With `--json`, stdout must contain valid JSON only. diff --git a/docs/install/local-workspace-registry.md b/docs/install/local-workspace-registry.md deleted file mode 100644 index 9bf91723..00000000 --- a/docs/install/local-workspace-registry.md +++ /dev/null @@ -1,176 +0,0 @@ -# Local workspace repository installation (macOS, Windows, and Linux) - -This manual connects a local ThothII installation to one remote Git repository hosted by a Git -server such as GitHub, GitLab, or Gitea. ThothII is a read-only consumer: it fetches, -validates, and activates workspace revisions, but never edits, commits, pushes, or publishes them. - -## Architecture ownership contract - -| Component | Ownership | Operator contract | -| --- | --- | --- | -| DWH | External | Configure the external endpoint and complete its runtime credentials in Workspace management. | -| LLM | External | Configure the external endpoint and model policy during installation. | -| Qdrant | Internal | Compose runs the internal service and persists `qdrant-data`. | -| Ollama embedding | Internal | Compose runs the internal `qwen3-embedding:0.6b` service and model-init job. | - -## Semantic index ownership contract - -| Scope | Ownership rule | Isolation rule | -| --- | --- | --- | -| Workspace semantic index | Each workspace keeps exactly one Qdrant collection reserved for itself. | Schema, Evidence, and memory records share that one collection and are separated by the `kind` payload. | - -The mandatory semantic stack is CPU-first. Set `THOTH_ENABLE_EMBEDDING_GPU=1` only after the -documented GPU prerequisites are satisfied. The embedding contract is fixed at -`qwen3-embedding:0.6b`, 1024 dimensions, cosine distance. - -## Prerequisites - -- A working local installation described by [local.md](local.md). -- A remote Git repository and a read-only deploy credential for this ThothII installation. -- A separate authoring clone in which a workspace curator can edit and publish source revisions. -- `tht` built with `bash scripts/build-tht.sh`. - -## Prepare and publish a workspace source - -Create a local workspace in an ordinary source directory outside ThothII's data directories. The -canonical repository layout is: - -```text -thoth-workspaces.yaml -/workspace.yaml -/evidence/ # optional, repository-owned Evidence -/schema/annotations.yaml # optional curated annotations -``` - -The catalog lists `{id, name, description?}` and the descriptor at -`/workspace.yaml` must match that metadata. Use the examples in -`deploy/workspaces/` as authoring references. Do not store passwords, tokens, private keys, or -signed URLs in Git. - -Publishing is an author-side Git operation: validate the source, commit it, and push it from the -separate authoring clone to the configured branch. This is the only meaning of “publish” in the -workspace lifecycle. ThothII has no author identity and no Git write credential. - -## Use the workspace from the application - -After the installation is started, use Workspace management from the authenticated application: - -1. Run **Update workspace repository** to fetch and validate the configured Git branch into the - application-owned registry. The operation is all-or-nothing and does not modify the authoring - clone. -2. Confirm that the installation-owned `workspace-secrets` storage remains outside the source - repository and contains no credentials in the workspace descriptors. -3. Select the workspace and run **Validate workspace source** to verify the active descriptor, catalog, - Evidence, annotations, and runtime bindings. -4. Run **Test workspace connections** only with the approved read-only DWH/Evidence test configuration. - Results are redacted and the workspace source remains unchanged. - - -Schema v3 is the only accepted workspace descriptor. -Schema v1 and v2 workspace descriptors are rejected before activation. - - -## Configure the remote Git repository - -Copy `docs/install/examples/thothii-installation.local.yaml` to an operator-controlled absolute -path. Its `workspaceRepository` block records the remote, branch, and read-only access method. -Choose exactly one transport override: - -- SSH: `deploy/compose.git-ssh.yaml`, with a read-only deploy key and pinned `known_hosts` file. -- HTTPS: `deploy/compose.git-https.yaml`, with a read-only token in a Git credentials file and an - optional private CA file. - -The remote and branch are installation configuration. Git credentials remain protected -installation files and are never accepted by Workspace management or returned by its API. - -Example non-secret/operator paths: - -```dotenv -THT_WORKSPACE_GIT_REMOTE=git@git.example.com:organization/workspaces.git -THT_WORKSPACE_GIT_BRANCH=main -THT_WORKSPACE_INSTALLATION_ID=local -PI_AUTH_FILE=/absolute/path/to/operator/pi-auth.json -THT_SECRETS_FILE=/absolute/path/to/operator/thothii.secrets -THT_WORKSPACE_GIT_SSH_KEY_FILE=/absolute/path/to/operator/git-ssh-key -THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=/absolute/path/to/operator/git-known-hosts -``` - -Keep these files outside both the ThothII checkout and the workspace source repository. Protect -them with mode `0600` on macOS/Linux or an equivalent single-user ACL on Windows. - -## Start and update the installation - -Use only the installation-aware lifecycle: - -```bash -export THT_SOURCE_ROOT=/absolute/path/to/ThothII -THT_BIN=tht -INSTALLATION=/absolute/path/to/operator/thothii-installation.yaml -"$THT_BIN" --installation "$INSTALLATION" start -"$THT_BIN" --installation "$INSTALLATION" doctor -``` - -At startup ThothII clones or fetches the configured repository into its application-managed -`workspace-registry` volume. Later, **Update workspace repository** performs a server-side fetch -and fast-forward candidate checkout. It does not copy anything to the user's computer. - -## Complete runtime secrets in Workspace management - -### Chiavi DWH REST per installazione - -Se il binding selezionato è `rest_api`, la chiave DWH è una credenziale per questa installazione e si salva nel vault tramite **Save entered secrets** oppure in un file locale indicato da `API_KEY_FILE`. `postgres_direct` e `ssh_tunnel` non usano questa chiave. Per emissione, TLS, rotazione e verifica `/rpc/ping`, seguire [enrollment DWH REST](dwh-auth-client-enrollment.md). - -Open Workspace management after the first successful repository update. - -1. At the repository level, review the configured host, repository, branch, and current revision. -2. Select a workspace. Repository update does not require a selection; validation and connection - tests do. -3. Review the runtime fields derived from the selected DWH transport and Evidence authentication - mechanism. -4. Enter or rotate the required values and choose **Save entered secrets**. -5. Run **Validate workspace source** and then **Test workspace connections**. - -Secret fields are write-only. The GUI receives only configured/missing status. Values are -encrypted by the backend in the platform-neutral `workspace-secrets` volume. ThothII temporarily -materializes a restrictive file only while an existing file-oriented connector needs it, then -removes that file when the runtime lease ends. **Forget stored value** deletes the selected encrypted value. - -The workspace YAML stays environment-independent: it declares connector mechanisms, not host -paths or credentials. Installation trust material such as a Git CA or `known_hosts` remains an -operator concern; DWH and Evidence credentials are completed in the GUI. - -## Validation and activation behavior - -An update follows this sequence: - -1. Fetch the configured branch into a candidate checkout managed by ThothII. -2. Validate the catalog, every descriptor, repository-relative Evidence, and cross-workspace - invariants at the same Git commit. -3. If every workspace is valid, atomically mark that complete commit as active. -4. If any validation fails, report sanitized diagnostics and keep the previous active revision. - -The active checkout is read-only application state. Never edit files under -`/data/workspace-registry`. A source correction must be committed and pushed from the authoring -clone, then fetched again with **Update workspace repository**. - -## Backup, rotation, and recovery - -Back up the `workspace-registry`, `workspace-secrets`, `sessions`, `qdrant-data`, -`embedding-models`, `settings`, and `pi-state` volumes together. The encrypted vault is useless -without its generated master key, so preserve the entire `workspace-secrets` volume and protect -the backup as secret material. - -Rotate a runtime credential by saving its replacement in Workspace management and rerunning its -connection test. Rotate Git credentials in the installation files and restart `core`. To recover -from a bad remote revision, correct or revert it in the authoring repository and run the update; -until validation succeeds, the previous active snapshot remains available. - -## Troubleshooting - -| Symptom | Meaning and action | -| --- | --- | -| Repository unavailable | Check remote host, branch, read-only deploy credential, CA, and `known_hosts`. | -| Candidate rejected | Fix the reported source error in the authoring clone, commit, push, and update again. | -| Runtime configuration required | Select the workspace and complete each required secret field. | -| Connection test fails | Rotate the relevant secret or correct the non-secret endpoint in the source/installation as appropriate. | -| Active revision did not change | The candidate was invalid or was already active; inspect the repository status. | diff --git a/docs/install/local.md b/docs/install/local.md deleted file mode 100644 index 3dd98563..00000000 --- a/docs/install/local.md +++ /dev/null @@ -1,499 +0,0 @@ -# Install ThothII on a local PC or Mac - -This guide installs one loopback-only ThothII on the same Windows, macOS, or Linux computer that -runs Docker. The supported application is one Docker Compose distribution containing exactly -`frontend` and `core`; Pi is pinned inside `core`. DWH, vector database, embedding, and LLM remain -external configurable services even when they run on this computer. - -No host Pi, Node.js, Python, Go toolchain, Docker socket in core, or browser shell is required. -Commands that contain example paths must be changed to absolute paths on your computer. - -## Choose your platform - -- **macOS:** use Terminal and Docker Desktop. Apple Silicon and Intel are supported by the local - image build. -- **Windows PowerShell:** use Docker Desktop with its WSL2 engine, Git for Windows, the Windows - build launcher, and `tht-windows-amd64.exe`. -- **Windows WSL2 (recommended):** enable Docker Desktop integration for your Linux distribution, - clone under `/home/` rather than `/mnt/c`, and follow the Linux shell commands. -- **Linux PC:** use Docker Engine plus the Compose v2 plugin and the Linux `tht` binary. - -Windows users must also read [Windows and WSL2 line endings](windows-line-endings.md) before the -first build. - -## Prerequisites - -Install only: - -1. Git 2.39 or newer. -2. Docker Desktop on macOS/Windows, or Docker Engine on Linux. -3. Docker Compose v2 (`docker compose`, not legacy `docker-compose`). -4. About 10 GB of free disk for source, images, build cache, and initial volumes. -5. Network access to the workspace Git remote and configured DWH/vector/embedding/LLM endpoints. - -Verify the tools: - -```sh -git --version -docker version -docker compose version -docker run --rm hello-world -``` - -On Linux, add the operator to the Docker group only if local policy permits it; sign out and back -in afterward. A local installation needs no inbound firewall rule because ports bind only to -`127.0.0.1`. - -## Clone and verify LF - -Use a `git clone` command that disables automatic CRLF conversion for this checkout. - -macOS and Linux: - -```sh -git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git -cd ThothII -git config --local core.autocrlf false -bash scripts/verify-line-endings.sh -``` - -Windows PowerShell: - -```powershell -git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git -Set-Location ThothII -git config --local core.autocrlf false -& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh -``` - -Windows WSL2: - -```sh -mkdir -p "$HOME/src" && cd "$HOME/src" -git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git -cd ThothII -git config --local core.autocrlf false -bash scripts/verify-line-endings.sh -``` - -Stop if the verifier names any path. Do not build from a CRLF checkout. - -## Create the local operator files - -Copy the non-secret template. This untracked `.env` contains addresses and absolute source paths, -never secret values: - -```sh -cp deploy/env/local.env.example deploy/env/local.env -mkdir -p /absolute/path/to/thothii-operator/secrets -chmod 0700 /absolute/path/to/thothii-operator/secrets -``` - -Native Windows PowerShell performs the same setup without POSIX utilities. The ACL commands remove -inherited access from the new operator directory and grant full control only to the current Windows -identity. Stop if either `icacls.exe` command returns a nonzero exit code: - -```powershell -$OperatorDir = Join-Path $env:USERPROFILE 'thothii-operator' -$SecretsDir = Join-Path $OperatorDir 'secrets' -$CurrentUser = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name -if (Test-Path $OperatorDir) { throw 'Use a new operator directory or review its ACLs manually.' } -New-Item -ItemType Directory -Force -Path $OperatorDir, $SecretsDir | Out-Null -icacls.exe $OperatorDir /inheritance:r -if ($LASTEXITCODE -ne 0) { throw 'Could not remove inherited operator-directory ACLs.' } -icacls.exe $OperatorDir /grant:r "${CurrentUser}:(OI)(CI)F" -if ($LASTEXITCODE -ne 0) { throw 'Could not grant the current user the operator-directory ACL.' } -Copy-Item deploy/env/local.env.example deploy/env/local.env -Copy-Item docs/install/examples/thothii-installation.local.yaml ` - (Join-Path $OperatorDir 'thothii-installation.yaml') -``` - -Edit `deploy/env/local.env`. At minimum set the workspace Git remote, `PI_AUTH_FILE`, -`THT_SECRETS_FILE`, and external service endpoints. Create the Pi/application and Git transport -files under the protected operator directory and set mode `0600`. On Windows use a user-only ACL -instead. DWH and Evidence credentials are entered later through Workspace management and stored -in the backend's encrypted `workspace-secrets` volume. - -Do not paste credentials into this guide's commands, `.env`, workspace YAML, Git, URLs, image build -arguments, or the installation descriptor. Secret contents are mounted read-only under -`/run/secrets` (Pi's auth store has its own protected read-only mount) and must never be committed, -embedded, rendered, or logged. - -Follow [the local workspace repository guide](local-workspace-registry.md) to choose exactly one -read-only Git SSH/HTTPS override. A fresh install requires a valid private workspace repository; -the remote Git repository remains the source of truth. - -Copy the installation example to an operator-controlled file named exactly -`thothii-installation.yaml`, then replace all placeholders with absolute paths: - -```sh -cp docs/install/examples/thothii-installation.local.yaml \ - /absolute/path/to/thothii-operator/thothii-installation.yaml -``` - -For HTTPS, replace the SSH override in that file with `deploy/compose.git-https.yaml`. Add only -reviewed local overrides. Paths may contain spaces when correctly represented as YAML strings. - -Native Windows uses the same four fields. Use single-quoted absolute Windows paths so backslashes -remain literal YAML characters: - -```yaml -profile: local -projectDirectory: 'C:\Users\operator\src\ThothII' -envFile: 'C:\Users\operator\src\ThothII\deploy\env\local.env' -overrides: - - 'C:\Users\operator\src\ThothII\deploy\compose.git-ssh.yaml' -``` - -## Address external services - -An address is interpreted inside `core`. Therefore container 127.0.0.1 means the container itself, -not the Docker host. Keep external DWH and LLM addresses configurable in the installation; Qdrant -and embedding are internal services in the standard stack. - -- **Docker Desktop (macOS and Windows):** use `host.docker.internal`, for example - `http://host.docker.internal:11434`. -- **Linux:** if a service runs on the host, create an untracked override and include its absolute - path in `thothii-installation.yaml`: - -```yaml -services: - core: - extra_hosts: - - "host.docker.internal:host-gateway" -``` - -Then use `host.docker.internal` in the endpoint. `extra_hosts: host.docker.internal:host-gateway` -is a host routing aid, not a bundled service. Prefer a real DNS name for independently operated -services; retain TLS and authentication even when co-located. - -## Build ThothII and tht - -The canonical local Compose smoke uses the base file plus the local profile. Keep this exact -base+profile command available for install verification: - -~~~sh -docker compose --env-file deploy/env/local.env -f compose.yaml -f deploy/compose.local.yaml up --build -d -~~~ - -After the stack is ready, configure and check authentication with the single host CLI tht; see -the [local authentication guide](authentication-local.md). Authentication configuration is -installation-global and is checked before workspace tests. - -From the repository root, macOS/Linux/WSL2 users run: - -```sh -bash scripts/build-local.sh -bash scripts/build-tht.sh -``` - -Native PowerShell users run: - -```powershell -powershell -ExecutionPolicy Bypass -File scripts/build-local.ps1 -& "C:\Program Files\Git\bin\bash.exe" scripts/build-tht.sh -``` - -The second command uses Docker to create native operator binaries under `dist/tht`; users do -not need to know or install Go. Select `tht-darwin-arm64` or `-amd64` on macOS, -`tht-linux-amd64` or `-arm64` on Linux/WSL2, and `tht-windows-amd64.exe` on Windows. -Copy the selected file to the protected operator directory and, on macOS/Linux, run `chmod 0755` -on it. - -## Start and verify - -Set convenient variables (PowerShell users use `$THT_BIN` and `$INSTALLATION` with `& $THT_BIN`): -Every operator call has the form `tht --installation `. - -```sh -THT_BIN=/absolute/path/to/thothii-operator/tht -INSTALLATION=/absolute/path/to/thothii-operator/thothii-installation.yaml -"$THT_BIN" --installation "$INSTALLATION" update --check-only -"$THT_BIN" --installation "$INSTALLATION" start -"$THT_BIN" --installation "$INSTALLATION" status -"$THT_BIN" --installation "$INSTALLATION" doctor -``` - -Native PowerShell uses the same order: - -```powershell -$THT_BIN = 'C:\Users\operator\thothii-operator\tht.exe' -$INSTALLATION = 'C:\Users\operator\thothii-operator\thothii-installation.yaml' -& $THT_BIN --installation $INSTALLATION update --check-only -& $THT_BIN --installation $INSTALLATION start -& $THT_BIN --installation $INSTALLATION status -& $THT_BIN --installation $INSTALLATION doctor -``` - -Wait for both services, then check the same-origin frontend and direct loopback core: - -```sh -curl --fail http://127.0.0.1:8080/health -curl --fail http://127.0.0.1:8787/health -"$THT_BIN" --installation "$INSTALLATION" pi doctor -"$THT_BIN" --installation "$INSTALLATION" pi test -``` - -Native PowerShell must call `curl.exe` explicitly; Windows PowerShell may otherwise resolve `curl` -to `Invoke-WebRequest`: - -```powershell -curl.exe --fail --silent --show-error http://127.0.0.1:8080/health -curl.exe --fail --silent --show-error http://127.0.0.1:8787/health -& $THT_BIN --installation $INSTALLATION pi doctor -& $THT_BIN --installation $INSTALLATION pi test -``` - -Open . If a check fails, run `tht ... logs` or `pi logs`; these are -bounded and sanitize declared secrets. Do not publish either loopback port. - -## Update an installation - -Commit or back up local operator changes first and finish active sessions. A promoted Pi image is -selected by the durable, installation-specific `current-image.yaml` after every base/profile file. -Therefore rebuilding `thothii-core:local` followed by `update --check-only` does not reconcile a -previous `pi update`: the old promoted core would remain selected. - -Do not delete or edit the selector. `tht status` is the installation-aware selector test. If -the running core image is the base `thothii-core:local` image, no Pi update has promoted a durable -lifecycle image and an ordinary same-Pi-version source rebuild/start is supported. If status shows -a lifecycle image and the pulled Pi pin is unchanged, `pi update` would be a no-op and the procedure -must stop. A changed Pi pin uses transactional `pi update --source build` in either case. - -macOS, Linux, and WSL2: - -```sh -set -euo pipefail - -abort_update() { printf 'Source update stopped: %s\n' "$1" >&2; exit 1; } -require_clean_source() { - local source_state - if ! source_state="$(git status --porcelain --untracked-files=all)"; then - abort_update "git status failed" - fi - [[ -z "$source_state" ]] || abort_update "commit, remove, or back up every tracked/untracked source change" -} - -require_clean_source -if ! git pull --ff-only; then abort_update "git pull --ff-only failed"; fi -require_clean_source -if ! git config --local core.autocrlf false; then abort_update "could not set repository LF policy"; fi -if ! bash scripts/verify-line-endings.sh; then abort_update "the pulled checkout contains CRLF files"; fi -if ! SOURCE_REVISION="$(git rev-parse HEAD)"; then abort_update "could not record the pulled revision"; fi -if ! NEXT_PI_VERSION="$(sed -n 's/^ARG PI_VERSION=//p' docker/core.Dockerfile)"; then - abort_update "could not read the pulled Pi pin" -fi -[[ -n "$NEXT_PI_VERSION" && "$NEXT_PI_VERSION" != *$'\n'* ]] || abort_update "expected one pinned default PI_VERSION" -if ! INSTALLATION_STATUS="$("$THT_BIN" --installation "$INSTALLATION" status)"; then - abort_update "tht status failed" -fi -if ! RUNNING_PI_VERSION="$("$THT_BIN" --installation "$INSTALLATION" pi status)"; then - abort_update "tht pi status failed" -fi -RUNNING_PI_VERSION="${RUNNING_PI_VERSION#Pi version: }" -[[ -n "$RUNNING_PI_VERSION" ]] || abort_update "tht pi status returned no version" - -COMPACT_STATUS="${INSTALLATION_STATUS//[[:space:]]/}" -USES_BASE_CORE=false -if [[ "$COMPACT_STATUS" == *'"Image":"thothii-core:local"'* ]]; then - USES_BASE_CORE=true -fi -TRANSACTIONAL_PI_UPDATE=true -if [[ "$NEXT_PI_VERSION" == "$RUNNING_PI_VERSION" ]]; then - [[ "$USES_BASE_CORE" == true ]] || abort_update "same Pi version is selected by a durable lifecycle image" - TRANSACTIONAL_PI_UPDATE=false -fi - -if ! bash scripts/build-local.sh; then abort_update "the local image build failed"; fi -if ! bash scripts/build-tht.sh; then abort_update "the tht build failed"; fi -if ! "$THT_BIN" --installation "$INSTALLATION" update --check-only; then - abort_update "the installation render check failed" -fi -if [[ "$TRANSACTIONAL_PI_UPDATE" == true ]]; then - if ! "$THT_BIN" --installation "$INSTALLATION" pi update \ - --version "$NEXT_PI_VERSION" --source build --yes --drain; then - abort_update "the transactional core update failed" - fi -fi -if ! "$THT_BIN" --installation "$INSTALLATION" start; then abort_update "installation start failed"; fi -if ! curl --fail http://127.0.0.1:8080/health; then abort_update "frontend health check failed"; fi -if ! curl --fail http://127.0.0.1:8787/health; then abort_update "core health check failed"; fi -if ! FINAL_STATUS="$("$THT_BIN" --installation "$INSTALLATION" status)"; then abort_update "final status failed"; fi -if ! FINAL_PI_STATUS="$("$THT_BIN" --installation "$INSTALLATION" pi status)"; then abort_update "final pi status failed"; fi -[[ "${FINAL_PI_STATUS#Pi version: }" == "$NEXT_PI_VERSION" ]] || abort_update "running Pi version does not match the pulled pin" -if ! "$THT_BIN" --installation "$INSTALLATION" doctor; then abort_update "final doctor failed"; fi -require_clean_source -printf 'Built source revision: %s\n%s\n%s\n' "$SOURCE_REVISION" "$FINAL_STATUS" "$FINAL_PI_STATUS" -``` - -Native Windows PowerShell uses the same fail-closed version comparison and transactional promotion: - -```powershell -$ErrorActionPreference = 'Stop' -function Assert-NativeSuccess([string]$Step) { - if ($LASTEXITCODE -ne 0) { throw "$Step failed with exit code $LASTEXITCODE." } -} -function Assert-CleanSource { - $SourceState = @(git status --porcelain --untracked-files=all) - Assert-NativeSuccess 'git status' - if ($SourceState.Count -ne 0) { - throw 'Commit, remove, or back up every tracked/untracked source change.' - } -} - -Assert-CleanSource -git pull --ff-only -Assert-NativeSuccess 'source pull' -Assert-CleanSource -git config --local core.autocrlf false -Assert-NativeSuccess 'repository LF policy' -& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh -Assert-NativeSuccess 'pulled checkout LF verification' -$SourceRevision = git rev-parse HEAD -Assert-NativeSuccess 'source revision read' -$VersionLine = @(Select-String -Path docker/core.Dockerfile -Pattern '^ARG PI_VERSION=(.+)$') -if ($VersionLine.Count -ne 1) { throw 'Expected exactly one pinned default PI_VERSION.' } -$NextPiVersion = $VersionLine.Matches[0].Groups[1].Value -$InstallationStatus = @(& $THT_BIN --installation $INSTALLATION status) -Assert-NativeSuccess 'installation status' -$RunningPiStatus = (& $THT_BIN --installation $INSTALLATION pi status) -Assert-NativeSuccess 'Pi status' -$RunningPiVersion = $RunningPiStatus -replace '^Pi version:\s*', '' -if ([string]::IsNullOrWhiteSpace($RunningPiVersion)) { throw 'Pi status returned no version.' } -$Services = $InstallationStatus | ConvertFrom-Json -$CoreServices = @($Services | Where-Object { $_.Service -eq 'core' }) -if ($CoreServices.Count -ne 1) { throw 'Installation status did not identify exactly one core service.' } -$UsesBaseCore = $CoreServices[0].Image -eq 'thothii-core:local' -$TransactionalPiUpdate = $true -if ($NextPiVersion -eq $RunningPiVersion) { - if (-not $UsesBaseCore) { throw 'Same Pi version is selected by a durable lifecycle image.' } - $TransactionalPiUpdate = $false -} -powershell -ExecutionPolicy Bypass -File scripts/build-local.ps1 -Assert-NativeSuccess 'local image build' -& "C:\Program Files\Git\bin\bash.exe" scripts/build-tht.sh -Assert-NativeSuccess 'tht build' -& $THT_BIN --installation $INSTALLATION update --check-only -Assert-NativeSuccess 'installation render check' -if ($TransactionalPiUpdate) { - & $THT_BIN --installation $INSTALLATION pi update ` - --version $NextPiVersion --source build --yes --drain - Assert-NativeSuccess 'transactional core update' -} -& $THT_BIN --installation $INSTALLATION start -Assert-NativeSuccess 'installation start' -curl.exe --fail --silent --show-error http://127.0.0.1:8080/health -Assert-NativeSuccess 'frontend health check' -curl.exe --fail --silent --show-error http://127.0.0.1:8787/health -Assert-NativeSuccess 'core health check' -$FinalStatus = @(& $THT_BIN --installation $INSTALLATION status) -Assert-NativeSuccess 'final installation status' -$FinalPiStatus = (& $THT_BIN --installation $INSTALLATION pi status) -Assert-NativeSuccess 'final Pi status' -if (($FinalPiStatus -replace '^Pi version:\s*', '') -ne $NextPiVersion) { - throw 'Running Pi version does not match the pulled pin.' -} -& $THT_BIN --installation $INSTALLATION doctor -Assert-NativeSuccess 'final doctor' -Assert-CleanSource -Write-Output "Built source revision: $SourceRevision" -Write-Output $FinalStatus -Write-Output $FinalPiStatus -``` - -The revision is printed only after every source/build/start/health/installation-aware check passes -and a final porcelain check still reports no tracked or untracked source changes. For a changed Pi -pin, status reports the promoted lifecycle candidate; for a same-version installation with no -selector, status reports the rebuilt base core. `update --check-only` alone proves only that Compose -renders. -Review release notes before updating. See [Pi management](pi-management.md) for rollback; never -install a package in the running container. - -## Back up and restore - -Back up before source/Pi updates and test restoration periodically. First stop cleanly: - -```sh -"$THT_BIN" --installation "$INSTALLATION" stop -docker volume ls --format '{{.Name}}' | grep '^thothii-' -``` - -Identify the four exact volumes belonging to this installation: `settings`, `pi-state`, -`workspace-registry`, and `sessions`. Confirm their Compose project label with `docker volume -inspect`. For each exact volume, archive it to a protected backup directory: - -```sh -BACKUP_DIR=/absolute/path/to/backups/2026-08-05 -VOLUME=exact-installation-volume-name -mkdir -p "$BACKUP_DIR" -docker run --rm -v "$VOLUME:/source:ro" -v "$BACKUP_DIR:/backup" \ - alpine:3.22 tar -C /source -czf "/backup/$VOLUME.tgz" . -``` - -Native PowerShell can run the same read-only archive container: - -```powershell -$BackupDir = 'C:\Users\operator\thothii-backups\2026-08-05' -$Volume = 'exact-installation-volume-name' -New-Item -ItemType Directory -Force $BackupDir | Out-Null -docker run --rm -v "${Volume}:/source:ro" -v "${BackupDir}:/backup" ` - alpine:3.22 tar -C /source -czf "/backup/${Volume}.tgz" . -``` - -Also back up the installation descriptor, operator environment, generated overrides, and secret -files to separate encrypted/protected storage. Never commit them. Record image digests and the Git -revision. Do not back up while containers are running. - -Restore only while stopped and only into a new, verified-empty exact target volume. Test the -archive in a disposable installation first: - -```sh -TARGET_VOLUME=exact-empty-target-volume-name -ARCHIVE=/absolute/path/to/backups/2026-08-05/exact-volume-name.tgz -docker run --rm -v "$TARGET_VOLUME:/target" alpine:3.22 \ - sh -c 'test -z "$(ls -A /target)"' -docker run --rm -v "$TARGET_VOLUME:/target" -v "$(dirname "$ARCHIVE"):/backup:ro" \ - alpine:3.22 tar -C /target -xzf "/backup/$(basename "$ARCHIVE")" -``` - -Native PowerShell uses `Split-Path` to produce the read-only archive mount and archive name: - -```powershell -$TargetVolume = 'exact-empty-target-volume-name' -$Archive = 'C:\Users\operator\thothii-backups\2026-08-05\exact-volume-name.tgz' -$ArchiveDir = Split-Path -Parent $Archive -$ArchiveName = Split-Path -Leaf $Archive -docker run --rm -v "${TargetVolume}:/target" alpine:3.22 ` - sh -ceu 'test -z "$(ls -A /target)"' -if ($LASTEXITCODE -ne 0) { throw 'The restore target volume is not empty.' } -docker run --rm -v "${TargetVolume}:/target" -v "${ArchiveDir}:/backup:ro" ` - alpine:3.22 tar -C /target -xzf "/backup/${ArchiveName}" -if ($LASTEXITCODE -ne 0) { throw 'The volume restore failed.' } -``` - -Restore all four volumes from the same backup set, restore protected operator files separately, -then run `update --check-only`, `start`, `doctor`, registry status/diagnostics, and a known session -before normal use. Never merge an archive into a non-empty volume. - -## Data-preserving uninstall - -Run `tht stop`, retain the installation descriptor at the same absolute path, and make one -verified backup set. In Docker Desktop, remove only this installation's stopped `core` and -`frontend` containers and optional local images; leave its four named volumes. On Linux, use the -containers' exact Compose project labels to remove only those stopped containers. Do not prune -global Docker data. - -Do **not** run `docker compose down --volumes`: it deletes the application data this procedure is -meant to preserve. Keep the operator directory and protected secrets if you intend to reinstall. -Using the same descriptor path preserves the `tht` project identity and reconnects the same -named volumes after rebuilding the source checkout. - -## Next: workspaces and Pi - -Complete [local workspace-registry installation](local-workspace-registry.md), including Git trust, -bindings, pull, validation, diagnostics, and registry recovery. Then use [Pi management](pi-management.md) -for provider/model configuration, smoke testing, transactional update, and rollback. - -The Git-backed workspace registry is always the workspace source of truth. Local DWH, vector, -embedding, or LLM processes remain independent services and are never added to the mandatory -ThothII core. diff --git a/docs/install/pi-management.md b/docs/install/pi-management.md deleted file mode 100644 index 84ae9c44..00000000 --- a/docs/install/pi-management.md +++ /dev/null @@ -1,162 +0,0 @@ -# Pi management - -ThothII bundles Pi in the `core` image. Operators use the Pi Management page for safe application -defaults and the host-side `tht` CLI for lifecycle work. A local Pi installation is not -required. - -Run these commands from the root of the current ThothII checkout or worktree. `tht` discovers -the valid installation descriptor in that project tree, so it uses the `deploy/` files belonging to -the checkout from which you run it. Do not use `~/bin`: `~` is the user home directory, not the -project root. - -```sh -THT_BIN=tht -tht version --json -``` - -If `tht` is not on `PATH`, install the native host CLI using the installation procedure in -`local.md` or `server.md`, then set `THT_BIN` to that installed binary. For an installation -stored elsewhere, set `THOTHII_INSTALLATION` or pass -`--installation /thothii-installation.yaml` explicitly. - -## Choose application defaults - -Use the **Pi Management** page to select the supported provider, model, and reasoning default, then -choose **Save defaults**. The page shows credentials only as present or missing and can run bounded -diagnostics; it never accepts or displays a credential, opens a terminal, or updates an image. - -Alternatively, use the CLI from an administrator terminal: - -```sh -"$THT_BIN" pi configure -"$THT_BIN" pi configure --provider zai --model glm-5.2 --thinking medium -``` - -Use GUI Save defaults or CLI `pi configure`, not both for the same change. The CLI's interactive -choices are restricted to supported models; non-interactive use must supply all three values. Both -methods store application defaults in backend installation settings, not in the project policy file. - -Useful read-only checks are: - -```sh -"$THT_BIN" pi status -"$THT_BIN" pi doctor -"$THT_BIN" pi test -"$THT_BIN" pi check -"$THT_BIN" pi logs -``` - -`pi check` is an alias for `pi test`; logs are a sanitized, bounded snapshot with no follow mode. - -## Edit the provider catalog and enabled-model policy - -Edit these project-root files in source control, then review and deploy the change through the -normal project process: - -- `deploy/pi/models.json` is the provider catalog: provider endpoints and the models each provider - offers. -- `deploy/pi/settings.json` is the enabled-model policy only; it lists the models available to the - application and does not store application defaults. - -These files contain configuration, not credentials. Keep provider configuration declarative: Pi -management rejects executable `!command` values. Docker Compose mounts the selected configuration -and credential files read-only. - -## Store provider credentials - -`PI_AUTH_FILE` is a setting in the installation environment file (for example, -`deploy/env/local.env`). Its value is the absolute path of the protected host credential file that -this installation selects. Docker Compose mounts that selected file read-only for Pi. -Other declared protected material is likewise mounted read-only under `/run/secrets`. - -Set restrictive permissions on the host file (`0600` on macOS/Linux or a user-only ACL on Windows). -Never put its contents in installation YAML, Git, command arguments, the browser, screenshots, -tickets, rendered Compose output, or logs. Do not print the file while troubleshooting. - -## Reload changed configuration - -After changing the provider catalog, enabled-model policy, or selected credential file, reload the -running application with one confirmed restart: - -```sh -"$THT_BIN" pi restart --yes --drain -``` - -`--yes` confirms that core will be recreated. Without `--drain`, restart refuses active sessions; -with it, ThothII closes admission and waits for active sessions to finish without terminating them. -The wait is bounded. Restart retains the exact captured running image: it does not build, pull, or -upgrade an image. Before recreating core, it pins that image through transaction-scoped Compose -override material so a configured tag moving during the operation cannot change the selected -image, and Compose is explicitly told never to build or pull. It will restart only core, then -verifies health, Pi version, settings/model smoke, non-secret rendered configuration, and -persistence mounts before reopening admission. - -Use `pi restart --yes` when there are already no active sessions. Do not substitute `tht stop` -and `tht start` or raw Compose commands for this reload workflow. - -## Update the bundled Pi version - -`pi update` is for a new bundled Pi version; it is not a configuration reload. The simple command -uses the single `ARG PI_VERSION=...` pin in `docker/core.Dockerfile`, builds that version, waits for -active sessions to finish, and recreates only `core`: - -```sh -"$THT_BIN" pi update -``` - -To build a specific version, pass `--version`; source, confirmation, and drain are automatic for -this normal build path: - -```sh -"$THT_BIN" pi update --version 0.81.0 -``` - -A registry update must use an immutable digest, never a mutable tag: - -```sh -"$THT_BIN" pi update \ - --version 0.81.0 --source pull \ - --image registry.example.invalid/thothii-core@sha256:<64-lowercase-hex-digits> \ - --yes --drain -``` - -Update keeps new-session admission gated while it builds or pulls a candidate, recreates only -`core`, verifies it, and promotes the image only after success. It preserves the frontend and named -volumes. - -## Recover a failed lifecycle operation - -If a restart or update fails after core recreation, leave maintenance enabled and preserve the -reported recovery state and transaction override. Do not delete `.tht`, state files, -containers, or volumes. Inspect status and sanitized logs: - -```sh -"$THT_BIN" pi maintenance status -"$THT_BIN" pi status -"$THT_BIN" pi logs -``` - -For a failed update, restore its prior image: - -```sh -"$THT_BIN" pi rollback --yes -``` - -For a failed restart, use maintenance recovery instead of rollback. After repairing the reported -Docker, disk, or configuration problem, use the same command to complete either safe recovery path: - -```sh -"$THT_BIN" pi maintenance recover --yes -"$THT_BIN" pi doctor -"$THT_BIN" pi test -``` - -`pi rollback --yes` restores the prior update image. `pi maintenance recover --yes` checks both -restart and update recovery state before it can reopen admission. If either command fails, keep the -installation gated and collect only the sanitized diagnostics. - -## Direct support access - -Raw Compose access is unsupported because it can bypass the installation-specific environment and -durable image selector. For support, use the installation-aware `tht pi status`, -`tht pi doctor`, `tht pi test`, and `tht pi logs` commands. diff --git a/docs/install/psd-workspace-setup.md b/docs/install/psd-workspace-setup.md deleted file mode 100644 index 7d730139..00000000 --- a/docs/install/psd-workspace-setup.md +++ /dev/null @@ -1,65 +0,0 @@ -# Policlinico San Donato — setup workspace (nuova gestione) - -Authentication acceptance is documented in the [manual authentication matrix](../testing/authentication-manual-acceptance.md). -Use generic OIDC with Authentik as the certified group catalog, map only the exact TOT Users and -TOT Admin groups, then run **Validate workspace source**, `tht auth check`, `tht auth check --interactive`, -and **Test workspace connections** in that order. Browser callback E2E, native Windows execution, approved PSD -manual identities, external L2, and the two parked restore-lock preconditions remain pending the -Task 15/release gates. - -Guida operativa per collegare ThothII al DWH di PSD con il nuovo sistema (registry Git + descriptor -v3 + `tht`). - -## Stato storico Mac/local (2026-08-13) - -> Questo stato è storico per Mac/local; il server PSD Project A usa binding separato `postgres_direct` read-only. -> -> Per la rotazione della credenziale DWH, fare riferimento al [runbook PSD](../operations/psd-dwh-auth-rollout.md): non autorizza modifiche finché i due gate non sono approvati. Il ThothII PSD server resta `postgres_direct`; il Mac e i client remoti usano `rest_api` con una chiave per installazione. `postgres_direct` e `ssh_tunnel` non usano chiavi `dwh-auth`. - -- **Repository PSD pubblicato:** `https://github.com/mptyl/tht-workspace-psd` (privato), branch - `main`, commit `d4f9185`. Layout P1.1 già migrato e validato. -- **Deploy key SSH** (sola lettura, senza passphrase) in - `deploy/psd/secrets/git-ssh-key` e registrata sul repo come deploy key `thothii-psd`; il remote - Git usato dall'installazione è `git@github.com:mptyl/tht-workspace-psd.git`. -- **Config operatore pronta** (file reali gitignored in `deploy/psd/`): `operator.env`, - `thothii-installation.yaml` e i secret d'installazione in `secrets/` (pi-auth, secret bundle, - chiave SSH, known_hosts). L'API key DWH va completata nella gestione Workspace ed è conservata - nel vault cifrato del backend. Il certificato REST è self-issued/private: ogni Mac/local senza trust equivalente deve usare `TLS_CA_FILE` e verificare il fingerprint fuori banda, come in `docs/install/dwh-auth-tls.md`. -- **Stack avviato** (progetto `thothii-70417a3e30ea`, via `tht start`): `qdrant`, `embedding` - (con `qwen3-embedding:0.6b`), `core`, `frontend` sani. Il registry ha **clonato e attivato** - `psd-clinical` (stato `ready`). -- **`tht workspace inspect --workspace psd-clinical` = OK** (identità descrittore/catalogo - risolte); la configurazione runtime va completata e testata dalla GUI. -- **Bloccante residuo: VPN.** `supabase-aritmolab.policlinicosandonato.it` non risolve - (`NXDOMAIN`) → il preprocessing DWH e le sessioni live non possono ancora partire. - -## Avvio/arresto (canonico) - -Usare `tht` (stesso project name, quindi stessi volumi named): - -```bash -tht=dist/tht/tht-darwin-arm64 -"$tht" --installation "$(pwd)/deploy/psd/thothii-installation.yaml" start -"$tht" --installation "$(pwd)/deploy/psd/thothii-installation.yaml" workspace inspect --workspace psd-clinical --json -"$tht" --installation "$(pwd)/deploy/psd/thothii-installation.yaml" stop -``` - -> **Nota project name:** `tht` calcola un project name stabile dall'installation descriptor -> (`thothii-`); `docker compose` "a mano" usa invece `name: thothii` dal `compose.yaml`, quindi -> i volumi named non coinciderebbero. Perciò per lo stack si usa `tht start` (non -> `compose-with-preflight.sh up`). - -## Rimane: smoke live di una domanda (P8 L2) - -Il preprocessing è già completato. Resta solo: - -1. Aprire `http://localhost:8080` e selezionare `psd-clinical`. -2. Creare una sessione con una domanda reale in linguaggio naturale. -3. Seguire le 8 fasi fino al primo gate di revisione. - -## Cosa è già stato fatto - -- Ristrutturazione del repo PSD nel layout P1.1 + validazione locale. -- Pubblicazione GitHub + deploy key read-only + configurazione Git d'installazione. -- Avvio stack + attivazione registry + `tht inspect` verde. -- **Preprocessing live completato** su PSD: DWH → FK → schema → Evidence, idempotente. diff --git a/docs/install/reverse-proxy-caddy.md b/docs/install/reverse-proxy-caddy.md deleted file mode 100644 index ed675fa8..00000000 --- a/docs/install/reverse-proxy-caddy.md +++ /dev/null @@ -1,98 +0,0 @@ -# Put ThothII behind Caddy - -Choose exactly one authentication mode. Direct ThothII-managed OIDC and deprecated upstream -authentication are mutually exclusive proxy contracts; never combine their directives. - -## Direct ThothII-managed OIDC - -Use this mode when `auth.yaml` has `mode: oidc`. Caddy terminates TLS and proxies every request to -`frontend`; ThothII performs login, callback validation, session creation, and authorization. -Caddy must not apply `forward_auth` or another external authentication gateway. - -The public `/api/auth/oidc/login` and `/api/auth/oidc/callback` paths pass unchanged through the -same proxy as the rest of `/api`. The configured `publicUrl` must match the browser origin. - -```caddyfile -thoth.example.invalid { - reverse_proxy 127.0.0.1:8080 { - # No URI rewrite: OIDC login and callback paths reach frontend unchanged. - flush_interval -1 - header_up Host {host} - header_up X-Forwarded-Proto https - header_up X-Forwarded-Host {host} - } - - log { - output file /var/log/caddy/thoth-access.log - format json - } -} -``` - -After reload, run Workspace Validate for static validation, `tht auth check` for live, -non-interactive authentication diagnosis, and then Workspace Test for aggregate live validation. - -## Deprecated upstream migration mode - -Use this section only while the installation explicitly uses deprecated `upstream` mode. Do not -use it with `mode: oidc` or `mode: local`. Here an external authentication gateway owns login and -Caddy applies `forward_auth` before forwarding normalized private identity headers to `frontend`. - -Forwarding identity headers alone does not authenticate a user. The authentication gateway returns -2xx only after validating its own credential or session. Clear browser-supplied public and trusted -headers before the subrequest, and map identity only from the successful auth response. - -```caddyfile -thoth.example.invalid { - route { - request_header -X-Authenticated-User - request_header -X-Thoth-Principal-Issuer - request_header -X-Thoth-Principal-Subject - request_header -X-Thoth-Principal-Display-Name - request_header -X-Thoth-Is-Admin - request_header -X-Thoth-Trusted-Principal-Issuer - request_header -X-Thoth-Trusted-Principal-Subject - request_header -X-Thoth-Trusted-Principal-Display-Name - request_header -X-Thoth-Trusted-Is-Admin - - forward_auth auth-gateway:4180 { - uri /verify - copy_headers { - X-Thoth-Principal-Issuer>X-Thoth-Trusted-Principal-Issuer - X-Thoth-Principal-Subject>X-Thoth-Trusted-Principal-Subject - X-Thoth-Principal-Display-Name>X-Thoth-Trusted-Principal-Display-Name - X-Thoth-Is-Admin>X-Thoth-Trusted-Is-Admin - } - } - - reverse_proxy 127.0.0.1:8080 { - flush_interval -1 - header_up Host {host} - header_up X-Forwarded-Proto https - } - } -} -``` - -## Trust boundary - -Caddy is the only public listener and proxies only to loopback `frontend`, never directly to -`core`. Configure access logs to omit cookies, authorization data, query strings, and identity -headers. Keep Caddy keys and state outside ThothII source and operator directories. - -## Validate and reload - -Keep the public firewall closed while validating: - -```sh -curl --fail http://127.0.0.1:8080/health -caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile -sudo systemctl reload caddy -``` - -## Test authentication and SSE - -For direct OIDC, verify the login path redirects to the configured provider, the callback reaches -ThothII unchanged, forged identity headers grant nothing, and SSE is unbuffered. For deprecated -upstream mode, additionally verify the external gateway rejects unauthenticated traffic and only -its 2xx response can create trusted identity headers. diff --git a/docs/install/reverse-proxy-nginx.md b/docs/install/reverse-proxy-nginx.md deleted file mode 100644 index 2277112a..00000000 --- a/docs/install/reverse-proxy-nginx.md +++ /dev/null @@ -1,143 +0,0 @@ -# Put ThothII behind Nginx - -Choose exactly one authentication mode. Direct ThothII-managed OIDC and deprecated upstream -authentication are mutually exclusive proxy contracts; never combine their locations or headers. - -## Direct ThothII-managed OIDC - -Use this mode when `auth.yaml` has `mode: oidc`. Nginx terminates TLS and proxies every request -to loopback `frontend`. ThothII owns OIDC login, callback validation, browser sessions, and -authorization. No external `auth_request` or authentication gateway belongs in this server. - -The `location /` block below has a `proxy_pass` without a replacement URI, so public -`/api/auth/oidc/login` and `/api/auth/oidc/callback` are forwarded unchanged. The configured -`publicUrl` must match the browser origin. - -```nginx -server { - listen 80; - server_name thoth.example.invalid; - return 301 https://$host$request_uri; -} - -server { - listen 443 ssl; - server_name thoth.example.invalid; - - ssl_certificate /etc/nginx/tls/thoth/fullchain.pem; - ssl_certificate_key /etc/nginx/tls/thoth/privkey.pem; - ssl_protocols TLSv1.2 TLSv1.3; - - location / { - # No auth_request and no URI rewrite: ThothII receives OIDC paths unchanged. - proxy_set_header Host $host; - proxy_set_header X-Forwarded-Proto https; - proxy_set_header X-Forwarded-Host $host; - proxy_set_header X-Forwarded-For $remote_addr; - proxy_set_header Connection ""; - proxy_pass http://127.0.0.1:8080; - proxy_http_version 1.1; - proxy_buffering off; - proxy_cache off; - proxy_read_timeout 3600s; - add_header X-Accel-Buffering no always; - } -} -``` - -After reload, run Workspace Validate for static validation, `tht auth check` for live, -non-interactive authentication diagnosis, and then Workspace Test for aggregate live validation. - -## Deprecated upstream migration mode - -Use this section only while the installation explicitly uses deprecated `upstream` mode. Do not -use it with `mode: oidc` or `mode: local`. In this mode an external authentication gateway owns -login, and Nginx applies `auth_request` before forwarding normalized private identity headers. - -Forwarding identity headers alone does not authenticate a user. The authentication gateway returns -2xx only after validating its own credential or session. - -```nginx -server { - listen 443 ssl; - server_name thoth.example.invalid; - - ssl_certificate /etc/nginx/tls/thoth/fullchain.pem; - ssl_certificate_key /etc/nginx/tls/thoth/privkey.pem; - ssl_protocols TLSv1.2 TLSv1.3; - - location = /_authenticate { - internal; - proxy_pass http://auth-gateway:4180/verify; - proxy_pass_request_body off; - proxy_set_header Content-Length ""; - proxy_set_header X-Original-URI $request_uri; - proxy_set_header X-Original-Method $request_method; - proxy_set_header X-Authenticated-User ""; - proxy_set_header X-Thoth-Principal-Issuer ""; - proxy_set_header X-Thoth-Principal-Subject ""; - proxy_set_header X-Thoth-Principal-Display-Name ""; - proxy_set_header X-Thoth-Is-Admin ""; - proxy_set_header X-Thoth-Trusted-Principal-Issuer ""; - proxy_set_header X-Thoth-Trusted-Principal-Subject ""; - proxy_set_header X-Thoth-Trusted-Principal-Display-Name ""; - proxy_set_header X-Thoth-Trusted-Is-Admin ""; - } - - location / { - auth_request /_authenticate; - auth_request_set $thoth_principal_issuer - $upstream_http_x_thoth_principal_issuer; - auth_request_set $thoth_principal_subject - $upstream_http_x_thoth_principal_subject; - auth_request_set $thoth_principal_display_name - $upstream_http_x_thoth_principal_display_name; - auth_request_set $thoth_is_admin - $upstream_http_x_thoth_is_admin; - - proxy_set_header X-Authenticated-User ""; - proxy_set_header X-Thoth-Principal-Issuer ""; - proxy_set_header X-Thoth-Principal-Subject ""; - proxy_set_header X-Thoth-Principal-Display-Name ""; - proxy_set_header X-Thoth-Is-Admin ""; - proxy_set_header X-Thoth-Trusted-Principal-Issuer $thoth_principal_issuer; - proxy_set_header X-Thoth-Trusted-Principal-Subject $thoth_principal_subject; - proxy_set_header X-Thoth-Trusted-Principal-Display-Name $thoth_principal_display_name; - proxy_set_header X-Thoth-Trusted-Is-Admin $thoth_is_admin; - proxy_set_header Host $host; - proxy_set_header X-Forwarded-Proto https; - proxy_set_header X-Forwarded-Host $host; - proxy_set_header X-Forwarded-For $remote_addr; - proxy_set_header Connection ""; - proxy_pass http://127.0.0.1:8080; - proxy_http_version 1.1; - proxy_buffering off; - proxy_cache off; - proxy_read_timeout 3600s; - add_header X-Accel-Buffering no always; - } -} -``` - -## Trust boundary - -Nginx is the only public listener and proxies only to loopback `frontend`, never directly to -`core`. Keep private keys outside the ThothII tree. Do not log cookies, authorization headers, -OIDC callback query values, authentication bodies, or trusted identity headers. - -## Validate and reload - -Keep the public firewall closed while validating: - -```sh -curl --fail http://127.0.0.1:8080/health -sudo nginx -t -sudo systemctl reload nginx -``` - -## Test authentication and SSE - -For direct OIDC, verify login redirects to the configured provider, callback traffic reaches -ThothII unchanged, forged identity headers grant nothing, and SSE is unbuffered. For deprecated -upstream mode, additionally verify the external gateway rejects unauthenticated traffic and only -its 2xx response can create trusted identity headers. diff --git a/docs/install/server-workspace-registry.md b/docs/install/server-workspace-registry.md deleted file mode 100644 index 1b2ef6a6..00000000 --- a/docs/install/server-workspace-registry.md +++ /dev/null @@ -1,132 +0,0 @@ -# Server workspace repository installation - -This manual supplements [server.md](server.md). A server installation reads one remote Git -repository hosted by GitHub, GitLab, Gitea, Bitbucket, or another Git server. ThothII fetches and -validates complete revisions but never edits, commits, pushes, or publishes workspace source. - -## Architecture ownership contract - -| Component | Ownership | Operator contract | -| --- | --- | --- | -| DWH | External | Configure the external endpoint and complete runtime credentials through the authenticated GUI. | -| LLM | External | Configure the external endpoint and model policy under installation control. | -| Qdrant | Internal | Compose runs private Qdrant and persists `qdrant-data`; include it in Qdrant backup/restore. | -| Ollama embedding | Internal | Compose runs private Ollama with `qwen3-embedding:0.6b`. | - -## Semantic index ownership contract - -| Scope | Ownership rule | Isolation rule | -| --- | --- | --- | -| Workspace semantic index | Each workspace keeps exactly one Qdrant collection reserved for itself. | Schema, Evidence, and memory records share that one collection and are separated by the `kind` payload. | - -The fixed semantic contract is 1024 dimensions and cosine distance. DWH and LLM remain external; -Qdrant, Ollama, and `embedding-model-init` remain private internal services. - -## Service account, storage, and firewall - -Run the application as the documented unprivileged service account. Keep the source checkout, -operator files, application data, and workspace authoring clone separate: - -```text -/srv/thothii/app/ # ThothII source release -/srv/thothii/operator/ # installation descriptor and protected Git files -/srv/thothii/data/ # application data, encrypted workspace vault, sessions -/srv/workspace-authoring/ # optional curator clone; never mounted into ThothII -``` - -Expose only the authenticated same-origin reverse proxy. Keep `core`, Qdrant, and Ollama private. - -## Prepare and publish a workspace source - -Create a local workspace in the external authoring repository, which contains -`thoth-workspaces.yaml`, one -`/workspace.yaml` per catalog entry, optional repository-owned Evidence, and optional -curated schema annotations. It contains no credentials. - -Publishing belongs to the curator workflow outside ThothII: validate, review, commit, and push the -source revision to the configured protected branch. Grant the ThothII service only read access. - - -Schema v3 is the only accepted workspace descriptor. -Schema v1 and v2 workspace descriptors are rejected before activation. - - -## Configure the remote Git repository - -Copy `docs/install/examples/thothii-installation.server.yaml` to -`/srv/thothii/operator/thothii-installation.yaml`. Set `workspaceRepository.remote`, `.branch`, and -`.access`, then select exactly one Git transport override. The remote and credential are normally -repository-scoped read-only deploy credentials. - -For SSH, mount a private key and pinned known-hosts file. For HTTPS, mount a Git credentials file -and the required CA chain. These installation credentials are not editable in Workspace -management and are never exposed by the API. - -## Start and update the installation - -Use the installation-aware controller described by `server.md`: - -```bash -THT_BIN=/srv/thothii/operator/tht -INSTALLATION=/srv/thothii/operator/thothii-installation.yaml -"$THT_BIN" --installation "$INSTALLATION" start -"$THT_BIN" --installation "$INSTALLATION" doctor -``` - -The descriptor composes `compose.yaml`, `deploy/compose.server.yaml`, the server session-storage -override, and one read-only Git transport override. **Update workspace repository** fetches a -candidate on the server; it does not transfer workspace files to the operator workstation. - -## Complete runtime secrets in Workspace management - -### Chiavi DWH REST per installazione - -Un'installazione server che seleziona `rest_api` usa una chiave DWH nel vault cifrato o nel file `API_KEY_FILE`; `postgres_direct` e `ssh_tunnel` non usano chiavi `dwh-auth`. Il servizio `dwh-auth` del DWH ha lifecycle `systemd` separato e non appartiene al Compose di ThothII. Vedere [enrollment client](dwh-auth-client-enrollment.md) e [guida server DWH](dwh-auth-server.md). - -After repository activation, an authenticated user can: - -1. Review the configured repository identity and update it without selecting a workspace. -2. Select a workspace to see the DWH/Evidence credential fields required by its connector modes. -3. Blind-save or rotate values with **Save entered secrets**; returned responses contain status only. -4. Run **Validate workspace source** and then **Test workspace connections**. -5. Use **Forget stored value** for an obsolete value after dependent sessions and jobs have ended. - -The backend encrypts values in `/data/workspace-secrets`, including the installation-specific -master key. The server profile persists that directory inside `THT_DATA_ROOT`; no workspace YAML -path depends on Linux, macOS, or Windows. Plaintext exists only in a restrictive temporary file -for the duration of a diagnostic, session, or maintenance lease. - -Authorization is intentionally the current installation-wide authenticated-user policy. A future -role model or external secret manager can replace that policy without changing workspace source. - -## Validation and activation behavior - -Repository update is all-or-nothing: ThothII fetches the configured branch, validates catalog, -descriptors, Evidence paths, and cross-workspace invariants at one commit, then atomically activates -the complete candidate. A rejected candidate never replaces the previous active snapshot. The -application-owned checkout and snapshots are read-only runtime state. - -Validation proves descriptor and repository structure. **Test workspace connections** additionally -materializes the current runtime secrets and contacts only the selected workspace's configured -DWH/Evidence endpoints. Failure does not modify or publish workspace source. - -## Backup, rotation, and recovery - -Back up application data and Qdrant consistently. Qdrant backup/restore must cover `qdrant-data`; -application recovery must cover repository snapshots/state, sessions, settings, Pi state, and the -entire encrypted `/data/workspace-secrets` directory. Store backup encryption keys separately and -test restore procedures without production traffic. - -Rotate DWH/Evidence credentials through Workspace management. Rotate Git access by atomically -replacing its protected installation file and restarting `core`. Recover a bad source revision by -reverting or correcting it in the external authoring repository and updating again. - -## Troubleshooting - -| Symptom | Meaning and action | -| --- | --- | -| Git authentication failed | Verify repository-scoped read permission, branch, key/token, CA, and host-key pinning. | -| Candidate validation failed | Correct the source repository; the prior active commit remains in service. | -| Runtime configuration required | Select the workspace and complete all required write-only fields. | -| Secret store unavailable | Stop writes, preserve `/data/workspace-secrets`, and restore vault plus master key together. | -| Connection test failed | Rotate the indicated runtime credential or correct the relevant non-secret endpoint. | diff --git a/docs/install/server.md b/docs/install/server.md deleted file mode 100644 index ca1e67ad..00000000 --- a/docs/install/server.md +++ /dev/null @@ -1,568 +0,0 @@ -# Install ThothII on a Linux server - -Server authentication uses generic OIDC with the reverse proxy preserving the configured public -origin and callback path. Follow the [OIDC guide](authentication-oidc.md), [Authentik guide](authentik.md) -when applicable, and the [authentication acceptance matrix](../testing/authentication-manual-acceptance.md). -The host authentication CLI is `tht`. Use `tht --installation workspace inspect ---workspace --json` for the active workspace snapshot, `tht --installation -auth check` for live non-interactive authentication diagnosis, `auth check --interactive` for -device-flow identity validation, and `tht ... doctor --json` for the aggregate installation gate. - -This guide is for an installer with basic Linux administration and very basic Docker knowledge. -It deploys the same Compose distribution used on a local PC: the mandatory application is exactly -`frontend` plus `core`, and pinned Pi is inside `core`. The server does not need host Pi, Node.js, -Python, Go, a browser shell, or a Docker socket inside either container. - -Examples use `/srv/thothii` as an **example operator root**, `thoth.example.com` as a replaceable -DNS name, and systemd-based command names. Adapt them to local policy. Complete the server session -storage overlay and migration procedure before exposing a production installation. - -## Deployment contract - -- The generic Linux host and Docker Compose v2 are the deployment platform. No other - application's Compose project, network, path, or runtime is required. -- `frontend` is the only published service and defaults to `127.0.0.1:8080`; `core` has no host - port. A host Nginx or Caddy listener terminates TLS and sends all application traffic to - `frontend`, never directly to `core`. -- DWH, vector database, embedding service, and LLM are external configurable endpoints. This - remains true when they happen to run on the same physical server. -- Application, Git, connector, and session credentials are protected host files mounted read-only - under `/run/secrets`. Pi's protected auth JSON uses its dedicated read-only Pi mount. No secret - value belongs in Git, images, browser storage, environment values, rendered Compose, or logs. -- The Git-backed workspace registry is the source of truth. Installation-local bindings identify - endpoints and secret-file paths; they do not replace the reviewed Git workspace descriptors. -- `tht` is the operator CLI for start, stop, status, health, logs, Pi lifecycle, drain, and - rollback. Raw Compose lifecycle commands bypass installation state and are unsupported. - -Read [server workspace-registry installation](server-workspace-registry.md), -[Pi management](pi-management.md), and the session-server comments in -`deploy/compose.session-server.yaml.example` before the first public start. - -## Service account and directories - -The container runtime identity is fixed at UID/GID 10001. It does not require or permit creation of -a matching host account or group. Keep the number unmapped and use numeric ownership only for the -dedicated bind paths that the non-root container must read or write. If either lookup below finds a -host identity, stop and design an explicit remapping before installation. - -```sh -if getent passwd 10001 >/dev/null || getent group 10001 >/dev/null; then - printf '%s\n' 'UID/GID 10001 is already mapped; stop' >&2 - exit 1 -fi -operator_uid="$(id -u)" -operator_gid="$(id -g)" -test "$operator_uid" -ne 0 -id -nG | tr ' ' '\n' | grep -Fx docker >/dev/null -``` - -The invoking, pre-existing administrator owns source and operator files. It must already have the -site-approved Docker access required to run `tht`; this guide never changes group membership. -Docker-group membership is effectively host-root access and must remain limited to reviewed -administrators. Do not grant the operator direct write access to container runtime trees. - -Create explicit directories. `source` contains the clone; `operator` contains untracked path-only -configuration; the three writable trees are bind-mounted into `core`; `secrets` contains regular -files only. The parent is owned by the operator with numeric group 10001 so both the operator and -container can traverse it. Numeric ownership does not add entries to `/etc/passwd` or `/etc/group`. - -```sh -operator_uid="$(id -u)" -operator_gid="$(id -g)" -sudo install -d -o "$operator_uid" -g 10001 -m 0750 /srv/thothii -sudo install -d -o "$operator_uid" -g "$operator_gid" -m 0750 /srv/thothii/source -sudo install -d -o "$operator_uid" -g "$operator_gid" -m 0750 /srv/thothii/operator -sudo install -d -o 10001 -g "$operator_gid" -m 0750 /srv/thothii/secrets -sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/data -sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/pi-state -sudo install -d -o 10001 -g 10001 -m 0750 /srv/thothii/workspace-registry -sudo install -d -o root -g root -m 0700 /srv/thothii-backups -stat -c '%u:%g %a %n' \ - /srv/thothii /srv/thothii/source /srv/thothii/operator /srv/thothii/secrets \ - /srv/thothii/data /srv/thothii/pi-state /srv/thothii/workspace-registry \ - /srv/thothii-backups -``` - -Expected: the parent is `operator_uid:10001 750`; source/operator are -`operator_uid:operator_gid 750`; secrets are `10001:operator_gid 750`; the three runtime trees are -`10001:10001 750`; backups are `0:0 700`. Re-run the empty `getent` checks after creation. Do not -make `/srv/thothii` a shared application directory. - -## Projected server authentication: canonical root and runtime projection - -For a server descriptor that declares `authentication.runtimeProjection`, authentication has two -different roots. The **canonical authentication root** (`authentication.configDirectory`) is the -root-operated source of truth. It and its regular files are `root:root 0700/0600`. The **runtime -projection** is a separate Linux-only tree for the container reader: its root, `generations`, and -generation directories are `10001:10001 0700`; `CURRENT`, `manifest.json`, `auth.yaml`, and (for -local mode) `users.yaml` are `10001:10001 0600`. The publisher assigns the numeric IDs directly; -it does not create a host user or group for 10001. - -The runtime projection has only `CURRENT` and `generations/<64-lowercase-hex>/`. `CURRENT` selects -one complete immutable generation. A successful configure, user mutation, restore, or explicit -publish first blocks `CURRENT`, then verifies a new immutable generation, then makes it ready. -The selected generation and up to two predecessor generations are retained; no operator edits a -generation or `CURRENT` directly. A ready projection is usable only when its canonical revision is -equal to the current canonical authentication root. If the runtime projection is blocked, missing, -tampered, or unequal, `start`, `update --check-only`, `auth check`, and `doctor` fail closed before -admission or Compose lifecycle work. - -The runtime directory must be an absolute canonical path, distinct from the canonical root, and -must exactly equal `THT_AUTH_RUNTIME_ROOT` in the protected installation environment. The server -profile and numeric UID/GID values are validated before any projected mutation or publication. - -The descriptor loader adds `compose.auth-runtime-projection.yaml` automatically when -`runtimeProjection` is present; do not list that file under `overrides`. The automatic override -mounts the runtime projection **read-only and core-only** at `/run/thothii-auth`; the canonical -authentication root is never mounted. No other service receives that mount or -`THT_AUTH_RUNTIME_PROJECTION_ROOT`. The example descriptor uses -`/srv/example/thothii/auth-runtime` only as a replaceable path and contains no credential value. - -This source change is prepared and tested only: Project A has not been started. It does not -authorize a raw Compose lifecycle launch, an Nginx change, legacy-stack change, or mutation of -`/srv`. A later manual gate needs separate explicit authorization before applying any descriptor -or runtime root to a server. - -### Status, repair, and safe evidence - -Use the root-operated installation command; retain only its small redacted JSON result: - -```sh -sudo tht --installation "$INSTALLATION" auth status --json -``` - -`state: "ready"` and `equal: true` are required before a projected server can start. `state: -"blocked"`, `equal: false`, or a command refusal means that the runtime projection is blocked or -cannot be validated. Do not start the stack, inspect YAML, print an environment, or edit `CURRENT` -or a generation. Confirm the protected canonical root is available, then republish it with: - -```sh -sudo tht --installation "$INSTALLATION" auth publish -sudo tht --installation "$INSTALLATION" auth status --json -``` - -`auth publish` reconstructs the selected immutable generation from the canonical root; it never -uses an older runtime generation as authority. If publish fails, leave the projection blocked and -escalate using the sanitized command result plus descriptor path and timestamp only. Do not attach -passwords, hashes, YAML, raw environment output, `nginx -T`, or a secret-bearing diff to evidence. - -### Authentication restore - -An authentication-bearing restore first publishes a blocked selector, restores canonical -authentication, and publishes a verified candidate generation before any restart. If candidate or -recovery verification fails, the verified recovery checkpoint is republished when possible; an -unverified result remains blocked and prevents start. A restore without authentication entries -does not touch the runtime projection. This is in addition to the normal restore requirement that -browser sessions and pending OIDC state are cleared. - -## Firewall and network boundaries - -Set `THOTH_SERVER_BIND=127.0.0.1`. Permit inbound TCP 80/443 only to the TLS proxy; port 80 should -redirect to HTTPS. Do not open 8080 externally, and do not add a core port. If a separate proxy -host is used, replace loopback with a private, firewalled address and allow only that proxy source. - -Allow outbound DNS and HTTPS to the source/Git registries, plus only the configured ports for the -Git remote, DWH, vector database, embedding service, LLM, session PostgreSQL, and any approved -bastion. Docker's private `thothii` network carries only `frontend`↔`core` traffic. Do not attach -the mandatory stack to another application's network. - -After start, confirm the host listens as intended: - -```sh -sudo ss -lntp -``` - -Expected public listeners are the proxy on 80/443 and the frontend on loopback 8080. There must be -no host listener for core port 8787. - -## Address co-resident external services - -Endpoint values are resolved inside `core`. Therefore container 127.0.0.1 means the container -itself, not the Linux host. Prefer real DNS names with TLS, authentication, and firewall policy, -even for services on this physical server. - -When DNS is unavailable for a host-published service, create an untracked override such as -`/srv/thothii/operator/host-gateway.yaml` and add it to the installation descriptor: - -```yaml -services: - core: - extra_hosts: - - "host.docker.internal:host-gateway" -``` - -Use `host.docker.internal` in the endpoint binding. The `host-gateway` mapping supplies routing; -it does not bundle or trust the target service. A host service listening only on host -`127.0.0.1` is **not reachable** through this mapping. Bind that service to the ThothII Docker -bridge gateway address or to a dedicated private host interface—never to `0.0.0.0` merely to make -the check pass. A stable internal DNS record routed through an authenticated private listener is -the preferred alternative. - -After the first bounded start attempt, copy the exact core container name from `tht status` -into `CORE_NAME`, then derive—not guess—the network ID, Linux bridge interface, gateway, and -subnet. Compose networks normally use `br-`; an explicit -`com.docker.network.bridge.name` option takes precedence: - -```sh -CORE_NAME=replace-with-exact-core-container-name -NETWORK_ID=$(docker inspect --format '{{range .NetworkSettings.Networks}}{{.NetworkID}}{{end}}' "$CORE_NAME") -NETWORK_NAME=$(docker network inspect --format '{{.Name}}' "$NETWORK_ID") -BRIDGE=$(docker network inspect --format '{{index .Options "com.docker.network.bridge.name"}}' "$NETWORK_ID") -test -n "$BRIDGE" || BRIDGE="br-${NETWORK_ID%${NETWORK_ID#????????????}}" -GATEWAY=$(docker network inspect --format '{{(index .IPAM.Config 0).Gateway}}' "$NETWORK_ID") -SUBNET=$(docker network inspect --format '{{(index .IPAM.Config 0).Subnet}}' "$NETWORK_ID") -printf 'network=%s bridge=%s gateway=%s subnet=%s\n' "$NETWORK_NAME" "$BRIDGE" "$GATEWAY" "$SUBNET" -ip address show dev "$BRIDGE" -``` - -Bind the co-resident service to `$GATEWAY`. In the host firewall `INPUT` chain, allow its exact -TCP port only when source is `$SUBNET`, input interface is `$BRIDGE`, and destination is -`$GATEWAY`; reject other sources to that listener and persist the rules using the distribution's -firewall manager. Docker's `DOCKER-USER` chain governs forwarded/published traffic and does not -replace this host-input rule. Ask the firewall administrator to implement the equivalent policy -with nftables when iptables is not the site's source of truth. - -For an iptables-managed host, replace the port before applying these reviewed rules; the second -rule prevents any other interface/source from reaching that gateway listener: - -```sh -EXTERNAL_PORT=replace-with-exact-service-port -sudo iptables -I INPUT 1 -i "$BRIDGE" -s "$SUBNET" -d "$GATEWAY" -p tcp --dport "$EXTERNAL_PORT" -j ACCEPT -sudo iptables -I INPUT 2 -d "$GATEWAY" -p tcp --dport "$EXTERNAL_PORT" -j REJECT -``` - -Confirm reachability with `tht pi test` for the configured LLM/Pi path and with the -authenticated Workspace Diagnostics action for DWH, vector collection/embedding pairing, and -embedding endpoints. A timeout paired with `ss -lntp`, `ip address show dev "$BRIDGE"`, and the -firewall counters distinguishes a loopback bind from a subnet/interface rule failure. Do not add -a shell to the browser or mount the Docker socket into core for this diagnostic. - -Configure each boundary independently: - -- DWH: read-only runtime identity, database/schema, verified TLS, and direct or REST endpoint. -- Vector database: endpoint plus exact database/schema, collection, distance metric, and writer - policy declared by the reviewed workspace. -- Embedding service: endpoint and the collection/embedding pairing—model and dimensions must match - the existing collection. Co-residence does not permit silently changing that pairing. -- LLM: authenticated endpoint selected through deployment and Pi configuration. - -Never add those services to ThothII's mandatory Compose files. Follow -[the diagnostic protocol](../workspace-diagnostic-protocol.md) before enabling a workspace. - -## Prepare operator files and secrets - -Clone with LF line endings, then verify before every build: - -```sh -git -c core.autocrlf=false clone \ - https://github.example.invalid/your-org/ThothII.git /srv/thothii/source/ThothII -cd /srv/thothii/source/ThothII -git config --local core.autocrlf false -bash scripts/verify-line-endings.sh -sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh \ - /srv/thothii/pi-state 10001 10001 -``` - -The last command is a mandatory clean-install and restore preflight. The server profile bind-mounts -the writable Pi-state root and then overlays protected `auth.json` plus tracked `models.json` and -`settings.json` read-only below it. Docker requires those three hidden target files to exist under -the host parent bind before startup. The initializer creates them atomically with UID/GID 10001, -mode `0600`, rejects symlink roots or targets, and never overwrites existing contents. It is safe to rerun -after restoring `pi-state`; run it before any `tht start`, Compose render/start, or Pi update. - -Copy the path-only server environment and installation descriptor: - -```sh -cp deploy/env/server.env.example /srv/thothii/operator/server.env -cp docs/install/examples/thothii-installation.server.yaml \ - /srv/thothii/operator/thothii-installation.yaml -chmod 0600 /srv/thothii/operator/server.env \ - /srv/thothii/operator/thothii-installation.yaml -``` - -The invoking operator owns both placeholder files; use an editor that preserves ownership and mode, -or create replacements under `umask 0077` in the operator directory. -Replace every placeholder with an absolute path. Use exactly one Git transport override. For -HTTPS, replace `deploy/compose.git-ssh.yaml` with `deploy/compose.git-https.yaml`. Keep the required -session-server overlay. Optional host-gateway or pinned -image overrides go after them. - -Create each installation credential (Pi/application, Git, and session storage) as an independent -regular file in `/srv/thothii/secrets`, owned by -UID 10001, the invoking operator's numeric primary GID, and mode `0640`. Owner access lets the UID -10001 container read a file mounted under `/run/secrets`; group access lets the operator run `tht`. The -operator environment records only absolute `*_FILE` or `*_SOURCE` paths for those installation -credentials. DWH and Evidence values are entered later through Workspace management and persist -as ciphertext under `/data/workspace-secrets`; the frontend receives no secret values. Do not -print file contents while testing permissions. - -```sh -operator_gid="$(id -g)" -sudo find /srv/thothii/secrets -type f -exec chown "10001:$operator_gid" {} + -sudo find /srv/thothii/secrets -type f -exec chmod 0640 {} + -sudo find /srv/thothii/secrets -type f \( ! -uid 10001 -o ! -gid "$operator_gid" -o ! -perm 0640 \) -print -``` - -Configure the remote repository and exactly one read-only Git transport as described in -[server workspace repository installation](server-workspace-registry.md). After startup, complete -the selected workspace's DWH and Evidence credentials through Workspace management. Secret values -must never be pasted into `server.env`, the installation YAML, a URL, or a shell argument. - -## Build locally or select pinned images - -Choose one image source. For a source build, the repository's reproducible launcher builds the -same `core` and `frontend` images used by the local profile. It requires only Git, Docker, and -Compose; copy the reviewed path-only server environment to the launcher's untracked input first: - -```sh -cd /srv/thothii/source/ThothII -cp /srv/thothii/operator/server.env deploy/env/local.env -bash scripts/build-local.sh -``` - -The printed local-profile start command is not the server start command; use `tht` below. - -Alternatively, create a reviewed untracked override with release images pinned by immutable -digest. Mutable tags are not a production pin: - -```yaml -services: - core: - build: !reset null - image: registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits> - session-migrate: - build: !reset null - image: registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits> - frontend: - build: !reset null - image: registry.example.com/thothii/frontend@sha256:<64-lowercase-hex-digits> -``` - -Add that absolute file last in `overrides`. `core` and `session-migrate` must use the exact same -core digest; neither may retain a local build or `:local` image. Frontend uses its own exact digest. -Both images must come from one compatible release; the core image must retain the declared Pi -version labels checked by `tht pi doctor`. Pull access -belongs in the host Docker credential store, not in Compose or the installation descriptor. - -## Install tht - -Build the operator binaries with Docker. No Go installation or Go knowledge is required: - -```sh -cd /srv/thothii/source/ThothII -THT_THT_OUTPUT_DIRECTORY=/srv/thothii/operator/build-output \ - bash scripts/build-tht.sh -operator_gid="$(id -g)" -sudo install -o root -g "$operator_gid" -m 0750 \ - /srv/thothii/operator/build-output/tht-linux-amd64 \ - /srv/thothii/operator/tht -``` - -The source checkout remains controlled by the invoking operator. The explicit output directory is the only build -write boundary; the build script rejects relative or non-canonical output paths. After installation, -remove or retain `build-output` according to the site's reviewed artifact policy. - -Use `tht-linux-arm64` on an ARM64 server. Set these variables in the maintenance shell; do -not source `server.env` as shell code: - -```sh -THT_BIN=/srv/thothii/operator/tht -INSTALLATION=/srv/thothii/operator/thothii-installation.yaml -"$THT_BIN" --help -"$THT_BIN" --installation "$INSTALLATION" update --check-only -``` - -Every operator command includes the descriptor explicitly. This preserves the installation's -profile, overrides, project identity, and durable current-image selector. The general form is -`tht --installation /absolute/path/thothii-installation.yaml `. - -## Start and verify readiness - -Keep the TLS proxy stopped or firewalled during bootstrap. First stop the app, run the -installation-aware session migration, and inspect its pristine JSON. The command activates only -the `session-migrate` profile/service with `--no-deps --no-TTY`; it derives the migrator image from the -selected core image after all installation overrides, so this procedure is identical for source -and pinned modes. It exits nonzero unless both arrays are empty: - -```sh -"$THT_BIN" --installation "$INSTALLATION" stop -"$THT_BIN" --installation "$INSTALLATION" sessions migrate --yes -``` - -Successful output has this shape (the `applied` list may contain versions on first use): - -```json -{"applied":[],"drifted":[],"pending":[]} -``` - -Only after seeing `"pending":[]` and `"drifted":[]`, start and verify: - -```sh -"$THT_BIN" --installation "$INSTALLATION" start -"$THT_BIN" --installation "$INSTALLATION" status -"$THT_BIN" --installation "$INSTALLATION" doctor -curl --fail http://127.0.0.1:8080/health -"$THT_BIN" --installation "$INSTALLATION" pi doctor -"$THT_BIN" --installation "$INSTALLATION" pi test -``` - -`/health` proves process liveness. Readiness additionally requires both healthy services, a valid -Pi provider/model smoke, a successful Git registry pull with an active validated snapshot, valid -workspace diagnostics, and ready session PostgreSQL. Use the authenticated Workspace Management -page to pull and diagnose the reviewed workspace. A liveness response alone is not release -approval. - -After configuring the proxy, open in a browser. Verify an unauthenticated -request is denied or redirected by the real identity provider, an authorized user can load the -same-origin UI and `/api`, an unauthorized user is denied, and an administrator alone can open Pi -Management. Keep port 8080 inaccessible from other hosts. - -## Configure TLS and upstream authentication - -Choose [Nginx](reverse-proxy-nginx.md) or [Caddy](reverse-proxy-caddy.md). Both examples terminate -TLS and proxy only to loopback `frontend`. They preserve SSE and clear client-supplied identity -headers before authentication. - -The authentication gateway must validate a real login/session and return normalized issuer, -subject, display-name, and admin claims only after success. Merely forwarding those headers does -not authenticate anyone. Do not enable `AUTH_MODE=upstream` on a listener reachable around the -trusted proxy, and never expose `core`. - -## Operate Pi, drain, and roll back - -Configure only closed provider/model/reasoning choices. Credentials remain protected files: - -```sh -"$THT_BIN" --installation "$INSTALLATION" pi status -"$THT_BIN" --installation "$INSTALLATION" pi configure -"$THT_BIN" --installation "$INSTALLATION" pi doctor -"$THT_BIN" --installation "$INSTALLATION" pi logs -``` - -Before an update, announce maintenance and ask users to finish active work. `--drain` closes new -admission and waits until no active sessions remain; it does not discard sessions. Build-source -and registry-source examples are: - -```sh -"$THT_BIN" --installation "$INSTALLATION" pi update \ - --version 0.81.0 --source build --yes --drain -"$THT_BIN" --installation "$INSTALLATION" pi update \ - --version 0.81.0 --source pull \ - --image registry.example.com/thothii/core@sha256:<64-lowercase-hex-digits> \ - --yes --drain -``` - -The transaction recreates only `core`, preserves volumes, verifies health/configuration/Pi, and -automatically attempts rollback after a post-mutation failure. For interrupted or ambiguous state: - -```sh -"$THT_BIN" --installation "$INSTALLATION" pi maintenance status -"$THT_BIN" --installation "$INSTALLATION" pi rollback --yes -"$THT_BIN" --installation "$INSTALLATION" pi maintenance recover --yes -``` - -Leave maintenance active if rollback cannot be verified. Preserve `.tht//` -recovery state, repair the reported host/configuration issue, and rerun rollback or maintenance -recovery. Never delete or edit `current-image.yaml` or `update-state.json` to force progress. - -## Back up and restore - -Back up before source, workspace, session-schema, or Pi changes. Drain work, stop the installation, -record `git rev-parse HEAD`, image digests, and `tht status`, then archive the three bind trees -with numeric ownership. Do not include live secrets in this ordinary archive. - -```sh -"$THT_BIN" --installation "$INSTALLATION" stop -BACKUP=/srv/thothii-backups/2026-08-05 -sudo install -d -o root -g root -m 0700 "$BACKUP" -sudo tar --numeric-owner --xattrs --acls -C /srv/thothii -czf "$BACKUP/runtime-data.tgz" \ - data pi-state workspace-registry -sudo sh -ceu 'cd "$1"; sha256sum runtime-data.tgz > SHA256SUMS; sha256sum --check SHA256SUMS' sh "$BACKUP" -``` - -Back up the installation descriptor, path-only environment, generated overrides, source revision, -and secret files to separate encrypted access-controlled storage. Database-backed production -sessions require their own PostgreSQL-native consistent backup; the local bind tree is not a -substitute. Test both restore paths periodically. - -Restore only while stopped. Verify the checksum, extract first into a new empty root, inspect -ownership and expected registry layout, then retain the old trees by renaming them before placing -the restored set. This keeps the previous state recoverable: - -```sh -RESTORE=/srv/thothii-restore-2026-08-05 -sudo install -d -o root -g root -m 0700 "$RESTORE" -sudo sh -ceu 'cd "$1"; sha256sum --check SHA256SUMS' sh /srv/thothii-backups/2026-08-05 -sudo tar --numeric-owner --xattrs --acls -C "$RESTORE" \ - -xzf /srv/thothii-backups/2026-08-05/runtime-data.tgz -sudo test -d "$RESTORE/workspace-registry/repo" -sudo test -d "$RESTORE/workspace-registry/snapshots" -``` - -After placing the restored `pi-state` tree and before the first start, rerun -`sudo /srv/thothii/source/ThothII/scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001`. -It validates or recreates only the hidden regular mount targets; it does not alter restored Pi -state or any protected configuration source. - -During the reviewed restore window, move each old tree to a timestamped sibling, move the matching -restored tree into `/srv/thothii`, restore the PostgreSQL session backup from the same recovery -point, and keep the proxy closed. Run `update --check-only`, `start`, `doctor`, `pi test`, registry -status, workspace diagnostics, and a known historical session before reopening traffic. Never -merge an archive into a non-empty tree. - -## Diagnostics - -Begin with bounded, sanitized installation-aware commands: - -```sh -"$THT_BIN" --installation "$INSTALLATION" status -"$THT_BIN" --installation "$INSTALLATION" doctor -"$THT_BIN" --installation "$INSTALLATION" logs -"$THT_BIN" --installation "$INSTALLATION" pi status -"$THT_BIN" --installation "$INSTALLATION" pi doctor -"$THT_BIN" --installation "$INSTALLATION" pi test -"$THT_BIN" --installation "$INSTALLATION" pi logs -"$THT_BIN" --installation "$INSTALLATION" pi maintenance status -``` - -Use the authenticated Workspace Management status and diagnostic actions for Git revision, -degraded snapshot, bindings, DWH, vector, and embedding checks. Review proxy logs separately, but -configure both proxy and log shipping to exclude cookies, authorization data, identity payloads, -query strings, and secret values. Do not render Compose or print an environment as a diagnostic. - -Typical boundaries are: `doctor` for Docker/Compose/LF/volume/service health; `pi doctor` for image -and provider/model integrity; registry status for Git/snapshot health; workspace diagnostics for -external service identity; and the proxy/identity provider for login failures. - -## Data-preserving uninstall - -Drain and stop through `tht`, take and verify one final backup, and disable the TLS proxy -route. Set `THT_BACKUP_ROOT=/srv/thothii-backups` in `server.env`; the removal command verifies the -filesystem identity of that backup root, all three bind trees, and every declared secret before -and after removing anything. - -First run without confirmation. It displays the exact installation project, service, container -name, container ID, and stopped state, then exits without mutation. Check every target: - -```sh -"$THT_BIN" --installation "$INSTALLATION" stop -"$THT_BIN" --installation "$INSTALLATION" remove -``` - -If and only if both targets are the expected stopped `frontend` and `core` containers, confirm: - -```sh -"$THT_BIN" --installation "$INSTALLATION" remove --yes exact-core-id exact-frontend-id -``` - -Replace both example IDs with the values from the immediately preceding dry-run. The command -refuses confirmation if the current target set differs. The confirmed operation passes only those -previously displayed immutable container IDs to Docker, -uses no force or volume option, rejects running/replaced containers, and proves the preservation -paths still identify the same filesystem objects. Keep `/srv/thothii/data`, `pi-state`, -`workspace-registry`, `operator`, protected secrets, database backups, and the installation -descriptor if reinstallation is possible. Do not prune global Docker data. - -Do **not** run `docker compose down --volumes`; it deletes persistent application data. Reusing the -same protected descriptor path preserves the `tht` installation identity and allows a later -compatible source checkout to reconnect the retained state. diff --git a/docs/install/windows-line-endings.md b/docs/install/windows-line-endings.md deleted file mode 100644 index 5be4a31f..00000000 --- a/docs/install/windows-line-endings.md +++ /dev/null @@ -1,250 +0,0 @@ -# Windows and WSL2 line endings - -ThothII's containers execute shell scripts from the source checkout. Those files must stay LF, -even when the PC normally uses CRLF. The repository's `.gitattributes` is authoritative, but a -Windows Git setting or an old checkout can still leave incorrect bytes. Check line endings after -every clone and pull, before building an image. - -## Recommended WSL2 clone - -Use Docker Desktop with WSL2 integration. Clone inside the Linux filesystem, for example under -`/home//src`, rather than under `/mnt/c`. This avoids slow cross-filesystem builds, -permission surprises, and Windows tools rewriting files behind WSL. - -```sh -mkdir -p "$HOME/src" -cd "$HOME/src" -git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git -cd ThothII -git config --local core.autocrlf false -bash scripts/verify-line-endings.sh -``` - -Keep Docker Desktop's integration enabled for that WSL distribution. Run the Linux build scripts -and the Linux `tht` binary from the same WSL shell. - -## Repository-local LF policy - -Set the option in this repository only. Do not change a company-wide or personal Git policy just -for ThothII. - -```sh -git config --local core.autocrlf false -git config --local --get core.autocrlf -``` - -The second command must print `false`. `.gitattributes` keeps shell, YAML, Dockerfile, JSON, -TypeScript, Python, and Markdown files at LF; PowerShell files remain CRLF. - -For a native PowerShell clone, disable conversion during the first checkout and then store the -repository-local setting: - -```powershell -git -c core.autocrlf=false clone https://github.example.invalid/your-org/ThothII.git -Set-Location ThothII -git config --local core.autocrlf false -& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh -``` - -## Verify after clone or pull - -From WSL2, Git Bash, macOS, or Linux run: - -```sh -bash scripts/verify-line-endings.sh -``` - -Success exits with code 0 and prints no offending path. If it lists a file, do not build or start -ThothII. Correct the checkout first. Native PowerShell users can invoke the same script through -Git for Windows as shown above. - -## Recover an existing CRLF clone - -The safest recovery is to reclone into a new directory. First commit wanted work or copy it to a -backup outside both clones. Then clone with conversion disabled, run the verifier, and copy back -only reviewed changes. - -If a reviewed working tree must be repaired in place, Git must first normalize the index, export -that exact index to a separate repair directory, verify the exported bytes, and only then copy the -verified tracked files over the worktree. `git add --renormalize .` alone does not change existing -worktree bytes. - -> **WARNING — destructive worktree rewrite.** Make a backup outside the clone or commit every -> wanted tracked change before continuing. The copy step below overwrites tracked worktree bytes -> from the staged index export. Stop if the staged diff does not contain exactly the wanted content; -> untracked files are neither exported nor repaired. - -From WSL2, Git Bash, macOS, or Linux: - -```sh -set -euo pipefail - -abort_repair() { printf 'CRLF repair stopped: %s\n' "$1" >&2; exit 1; } -validate_index_export() { - git ls-files -s -z | while IFS= read -r -d '' entry; do - metadata="${entry%%$'\t'*}" - path="${entry#*$'\t'}" - mode="${metadata%% *}" - [[ "$path" != "$entry" ]] || exit 1 - case "$mode" in - 100644|100755) [[ -f "$REPAIR_DIR/$path" && ! -L "$REPAIR_DIR/$path" ]] || exit 1 ;; - 120000) [[ -L "$REPAIR_DIR/$path" ]] && readlink "$REPAIR_DIR/$path" >/dev/null || exit 1 ;; - *) printf 'Unsupported Git mode %s: %s\n' "$mode" "$path" >&2; exit 1 ;; - esac - done -} -validate_worktree_modes() { - git ls-files -s -z | while IFS= read -r -d '' entry; do - metadata="${entry%%$'\t'*}" - path="${entry#*$'\t'}" - mode="${metadata%% *}" - case "$mode" in - 100644|100755) [[ -f "$path" && ! -L "$path" ]] || exit 1 ;; - 120000) [[ -L "$path" ]] && readlink "$path" >/dev/null || exit 1 ;; - *) exit 1 ;; - esac - done -} -rewrite_index_entry() { - local mode="$1" path="$2" target temporary_link - case "$mode" in - 100644) - cp "$REPAIR_DIR/$path" "$path" && chmod a-x "$path" - ;; - 100755) - cp "$REPAIR_DIR/$path" "$path" && chmod a+x "$path" - ;; - 120000) - target="$(readlink "$REPAIR_DIR/$path")" || return 1 - temporary_link="${path}.thoth-lf-repair-link" - [[ ! -e "$temporary_link" && ! -L "$temporary_link" ]] || return 1 - ln -s "$target" "$temporary_link" || return 1 - rm -f "$path" || { rm -f "$temporary_link"; return 1; } - mv "$temporary_link" "$path" - ;; - *) return 1 ;; - esac -} - -if ! git status --short; then abort_repair "git status failed"; fi -if ! git config --local core.autocrlf false; then abort_repair "could not set repository LF policy"; fi -if ! git add --renormalize .; then abort_repair "index renormalization failed"; fi -if ! git diff --cached --check; then abort_repair "normalized index check failed"; fi -if ! git diff --cached; then abort_repair "normalized index review failed"; fi -REPAIR_DIR="$(cd .. && pwd -P)/ThothII-lf-repair" -if [[ -e "$REPAIR_DIR" ]]; then - abort_repair "choose a new empty LF repair directory: $REPAIR_DIR" -fi -if ! mkdir -p "$REPAIR_DIR"; then abort_repair "could not create LF repair directory"; fi -REPAIR_PREFIX="$REPAIR_DIR/" -if ! git checkout-index --all --force --prefix="$REPAIR_PREFIX"; then abort_repair "index export failed"; fi -if ! validate_index_export; then abort_repair "index export is missing entries or Git modes"; fi -if ! bash scripts/verify-line-endings.sh "$REPAIR_DIR"; then abort_repair "exported bytes failed LF verification"; fi -# WARNING: destructive copy; make a backup or commit wanted changes before this command. -if ! git ls-files -s -z | while IFS= read -r -d '' entry; do - metadata="${entry%%$'\t'*}" - path="${entry#*$'\t'}" - mode="${metadata%% *}" - rewrite_index_entry "$mode" "$path" || exit 1 -done; then - abort_repair "tracked-file rewrite failed; do not build from this worktree" -fi -if ! validate_worktree_modes; then abort_repair "repaired worktree does not match Git index modes"; fi -if ! bash scripts/verify-line-endings.sh; then abort_repair "repaired worktree failed LF verification"; fi -if ! git diff --cached --check; then abort_repair "repaired index check failed"; fi -``` - -Native Windows PowerShell runs the same Git operations and invokes the byte verifier through Git -for Windows: - -```powershell -$ErrorActionPreference = 'Stop' -function Assert-NativeSuccess([string]$Step) { - if ($LASTEXITCODE -ne 0) { throw "$Step failed with exit code $LASTEXITCODE." } -} -function ConvertFrom-IndexEntry([string]$Entry) { - if ($Entry -notmatch '^([0-9]{6}) [0-9a-f]+ [0-3]\t(.+)$') { - throw "Invalid Git index entry: $Entry" - } - [pscustomobject]@{ Mode = $Matches[1]; Path = $Matches[2] } -} - -git status --short -Assert-NativeSuccess 'git status' -git config --local core.autocrlf false -Assert-NativeSuccess 'repository LF policy' -git add --renormalize . -Assert-NativeSuccess 'index renormalization' -git diff --cached --check -Assert-NativeSuccess 'normalized index check' -git diff --cached -Assert-NativeSuccess 'normalized index review' -$RepairDir = Join-Path (Split-Path -Parent (Get-Location).Path) 'ThothII-lf-repair' -if (Test-Path $RepairDir) { throw 'Choose a new empty LF repair directory.' } -New-Item -ItemType Directory -Path $RepairDir | Out-Null -$RepairPrefix = $RepairDir.Replace('\', '/') + '/' -git -c core.symlinks=true checkout-index --all --force --prefix=$RepairPrefix -Assert-NativeSuccess 'index export' -$RawIndexEntries = @(git ls-files -s) -Assert-NativeSuccess 'index inventory' -$IndexEntries = @($RawIndexEntries | ForEach-Object { ConvertFrom-IndexEntry $_ }) -foreach ($Entry in $IndexEntries) { - $ExportPath = Join-Path $RepairDir $Entry.Path - $ExportItem = Get-Item -LiteralPath $ExportPath -Force -ErrorAction Stop - switch ($Entry.Mode) { - { $_ -in '100644', '100755' } { - if ($ExportItem.LinkType -eq 'SymbolicLink') { throw "Regular export became a symlink: $($Entry.Path)" } - } - '120000' { - if ($ExportItem.LinkType -ne 'SymbolicLink') { throw "Symlink export is not mode 120000: $($Entry.Path)" } - if ([string]::IsNullOrWhiteSpace([string]$ExportItem.Target)) { throw "Symlink target is empty: $($Entry.Path)" } - } - default { throw "Unsupported Git mode $($Entry.Mode): $($Entry.Path)" } - } -} -& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh $RepairDir -Assert-NativeSuccess 'exported byte LF verification' -# WARNING: destructive copy; make a backup or commit wanted changes before this command. -foreach ($Entry in $IndexEntries) { - $ExportPath = Join-Path $RepairDir $Entry.Path - switch ($Entry.Mode) { - { $_ -in '100644', '100755' } { - Copy-Item -LiteralPath $ExportPath -Destination $Entry.Path -Force -ErrorAction Stop - } - '120000' { - $LinkTarget = [string](Get-Item -LiteralPath $ExportPath -Force -ErrorAction Stop).Target - $TemporaryLink = "$($Entry.Path).thoth-lf-repair-link" - if (Test-Path -LiteralPath $TemporaryLink) { throw "Temporary symlink path exists: $TemporaryLink" } - New-Item -ItemType SymbolicLink -Path $TemporaryLink -Target $LinkTarget -ErrorAction Stop | Out-Null - Remove-Item -LiteralPath $Entry.Path -Force -ErrorAction Stop - Move-Item -LiteralPath $TemporaryLink -Destination $Entry.Path -ErrorAction Stop - } - default { throw "Unsupported Git mode $($Entry.Mode): $($Entry.Path)" } - } -} -foreach ($Entry in $IndexEntries) { - $WorktreeItem = Get-Item -LiteralPath $Entry.Path -Force -ErrorAction Stop - switch ($Entry.Mode) { - { $_ -in '100644', '100755' } { - if ($WorktreeItem.LinkType -eq 'SymbolicLink') { throw "Regular worktree entry became a symlink: $($Entry.Path)" } - } - '120000' { - if ($WorktreeItem.LinkType -ne 'SymbolicLink') { throw "Repaired worktree symlink is not mode 120000: $($Entry.Path)" } - if ([string]::IsNullOrWhiteSpace([string]$WorktreeItem.Target)) { throw "Repaired symlink target is empty: $($Entry.Path)" } - } - default { throw "Unsupported Git mode $($Entry.Mode): $($Entry.Path)" } - } -} -& "C:\Program Files\Git\bin\bash.exe" scripts/verify-line-endings.sh -Assert-NativeSuccess 'repaired worktree LF verification' -git diff --cached --check -Assert-NativeSuccess 'repaired index check' -``` - -The export inventory must contain every regular mode (`100644`/`100755`) and recreate every tracked -workspace compatibility symlink (`120000`). The first verifier proves the complete -export before any overwrite; every copy/link operation is fail-closed; the final verifier examines -the repaired worktree bytes. On native Windows, creating symlinks requires Developer Mode or an -elevated account; failure stops the rewrite. Review the staged diff again before committing, then -remove the separate repair directory only after inspecting it. The procedure intentionally avoids -`git reset --hard`; replacing the clone is easier to audit and safer for uncommitted work. diff --git a/docs/migrations/p1-to-p1-1-registry-layout.md b/docs/migrations/p1-to-p1-1-registry-layout.md deleted file mode 100644 index 75a272c3..00000000 --- a/docs/migrations/p1-to-p1-1-registry-layout.md +++ /dev/null @@ -1,38 +0,0 @@ -# P1 to P1.1 registry layout migration - -P1.1 is a repository-contract cutover. New ThothII builds reject the old flat layout and a -repository without `thoth-workspaces.yaml`, so migrate the registry in Git first and upgrade the -application only after that reviewed migration commit is pushed. - -## One reviewed migration commit - -Perform the layout move in a clean review clone and keep it in one reviewed Git commit: - -```sh -git mv workspaces/.yaml /workspace.yaml -git mv workspace-content//evidence /evidence -# create and review thoth-workspaces.yaml from descriptor metadata -``` - -For every workspace directory, preserve the existing descriptor bytes, move only the embedded -filesystem Evidence tree, and create `thoth-workspaces.yaml` with: - -- `schema_version: 1` -- the ordered `workspaces` list -- curator-owned `id`, `name`, and optional `description` copied from the reviewed descriptors - -Generated docs remain under `workspace-docs//`. Do not add an auto-migrator and do not let the -API rewrite the catalog or Evidence tree. - -## Cutover order - -1. Review the migration commit, including the new `thoth-workspaces.yaml` metadata. -2. Push that commit to the authoritative registry branch. -3. Upgrade ThothII only after that migration commit is pushed. -4. Pull the migrated registry into each installation before using workspace management. - -## Rollback - -Roll back the application revision and registry commit together. Do not point a P1.1 binary at the -old flat layout, and do not keep a migrated registry commit active while rolling the application -back to pre-P1.1 code. diff --git a/docs/operations/psd-dwh-auth-rollout.md b/docs/operations/psd-dwh-auth-rollout.md deleted file mode 100644 index 415c5e3e..00000000 --- a/docs/operations/psd-dwh-auth-rollout.md +++ /dev/null @@ -1,43 +0,0 @@ -# PSD — rollout controllato DWH REST - -Questo runbook rispecchia i Task 9–10 del [piano](../superpowers/plans/2026-08-20-dwh-rest-per-installation-auth.md). Gate A e la parte dual-key di Gate B sono stati eseguiti con autorizzazioni separate. L'emendamento del proprietario del 2026-08-21 rinvia collaudo Mac, osservazione e revoca a prima di Project B; non autorizza ulteriori mutazioni. - -## Invarianti - -- ThothII sul server PSD: `postgres_direct` read-only, senza chiave `dwh-auth`. -- Mac PSD e client remoti: `rest_api`, una chiave per installazione, HTTPS `.it` verificato. -- `dwh-auth` è `systemd` indipendente, non Compose; non fermare o sostituire il vecchio stack ora. -- Sessioni legacy, indici Qdrant e cache Ollama sono dati test: nessuna migrazione o backup per il cutover. Il vecchio stack resta comunque fino a cutover/rollback approvati. -- Usare solo `/dwh/rpc/ping`, mai risultati clinici o catture Nginx grezze. - -## Gate A — Task 9, servizio locale senza Nginx pubblico - -Richiedere prima autorizzazione per SHA congelato, target, rollback e impatto legacy. Senza consenso, fermarsi e registrare solo `IN_DISCUSSION`. - -1. Verificare in sola lettura architettura, gruppo `www-data`, nomi liberi, systemd, `nginx -t`, ping attuale e file legacy regolare `root:root` `0600`; non leggerlo, stamparlo o calcolarne hash. -2. Costruire con `bash scripts/build-dwh-auth.sh --output /tmp/dwh-auth-release`; registrare solo SHA sorgente e checksum binario. -3. Installare binario/unit/tmpfiles come nella [guida server](../install/dwh-auth-server.md): registry `root:dwh-auth` `2750`, lock/record `0640`, socket `dwh-auth:www-data` `0660`. -4. Importare una sola legacy `legacy-shared` dal file protetto e creare `psd-mac-primary` in nuovo file `0600` sotto `/root/dwh-auth-provision/`; mai segreti in argv, ambiente, log o evidenze. -5. Eseguire `dwh-auth check`, `systemd-analyze verify`, avviare l'unità e testare sul socket Unix con file header curl protetti `0600`: v1=204, legacy=204, casuale=401, assente=401. -6. Salvare solo ID pubblici, owner/mode, stato unit/socket, timestamp, checksum binario/config e rollback. Non modificare Nginx in questo gate. - -## Gate B — Task 10, Nginx e client - -Serve un secondo consenso: presentare file, backup, canale consegna Mac, osservazione ed esiti 2xx/401/503. - -1. Creare copie timestampate `root:root` `0600` di `/etc/nginx/sites-available/policlinicosandonato` e file coinvolti; non allegare configurazioni Nginx grezze alle evidenze. -2. Aggiungere solo `/etc/nginx/conf.d/dwh-auth-rate-limit.conf` e route DWH; preservare upstream `http://127.0.0.1:3001/`, mantenere byte-identiche le location vector e rimuovere la chiave prima di PostgREST. -3. Eseguire checker strutturale, scansione segreti con solo `PASS/FAIL` e metadati, installare candidati e `sudo nginx -t`. No raw diff: non eseguire o conservare raw diff, `nginx -T` o dump: il file legacy può contenere la chiave. Se uno fallisce, ripristinare backup prima di reload e registrare FAIL sanitizzato. -4. Dopo consenso fare reload, poi HTTPS `.it` con CA e file header curl protetti 0600: v1=2xx, legacy=2xx, casuale=401, assente=401 su `/dwh/rpc/ping`; guasto autenticatore=503, mai accesso permissivo. -5. **Deferred pre-Project-B:** consegnare al Mac chiave e CA separatamente, verificare fingerprint fuori banda, configurare vault GUI o `API_KEY_FILE`, poi **Validate workspace source** e **Test workspace connections**. -6. **Deferred pre-Project-B:** completare 48 ore di osservazione comprendenti due cicli ETL delle 03:00, quindi revocare `legacy-shared` con ragione `shared-credential-rotation`; v1=2xx post-revoca, legacy=401 post-revoca e journal limitato senza chiavi/digest. - -## Rollback e chiusura - -Durante dual-key il rollback ripristina solo route/servizio revisionati, verifica `nginx -t` e fa reload autorizzato. Non ripristina chiavi revocate, PostgreSQL, dati legacy o stack. Scatta per TLS, risposte inattese, salute degradata o assenza di consenso. - -Activity 1 resta `DEFERRED_PRE_PROJECT_B`: dual-key è attivo, ma PASS richiede ancora v1=2xx -post-revoca, legacy=401 post-revoca, servizio/Nginx validi, log sanitizzati, rollback leggibile e -accettazione owner. Il rinvio non blocca il survey e Project A privato; blocca Project B. Compilare -[evidenza](../testing/evidence/psd-dwh-auth-rollout-report-template.md) e -[collaudo](../testing/dwh-auth-manual-acceptance.md). diff --git a/docs/operations/psd-server-sol-orchestration-prompt.md b/docs/operations/psd-server-sol-orchestration-prompt.md deleted file mode 100644 index fc48d440..00000000 --- a/docs/operations/psd-server-sol-orchestration-prompt.md +++ /dev/null @@ -1,92 +0,0 @@ -# Prompt operativo per Sol — deploy ThothII su PSD - -## Ruolo - -Sei l'orchestratore del deploy di ThothII sul server PSD. Devi guidare il lavoro -in modo incrementale, verificabile e reversibile. Non assumere che una fase sia -completata: richiedi evidenze e applica i gate descritti nei piani. - -## Documenti normativi - -Leggi prima questi file, in quest'ordine: - -1. `AGENTS.md` -2. `PROJECT_STATE.md` -3. `docs/plans/2026-08-20-psd-server-deployment-program-design.md` -4. `docs/plans/2026-08-20-psd-server-deployment-program.md` -5. `docs/plans/2026-08-20-psd-server-survey.md` -6. `docs/plans/2026-08-20-psd-server-project-a-standalone.md` -7. `docs/plans/2026-08-20-psd-server-project-b-authentik.md` -8. `docs/testing/psd-server-project-a-manual.md` -9. `docs/testing/psd-server-project-b-manual.md` -10. `docs/testing/evidence/psd-server-survey-report-template.md` -11. `docs/testing/evidence/psd-server-project-a-report-template.md` -12. `docs/testing/evidence/psd-server-project-b-report-template.md` - -In caso di conflitto, prevalgono `AGENTS.md`, `PROJECT_STATE.md` e i piani -specifici delle fasi nell'ordine Survey, Project A, Project B. - -## Modelli e delega multi-agent - -Verifica quali modelli e quali primitive multi-agent sono realmente disponibili -nell'ambiente. Se disponibili: - -- **Sol** mantiene il controllo del piano, dei gate, delle decisioni architetturali, - della sicurezza, di Authentik, Nginx, Supabase e del rollback. -- **Terra** esegue esclusivamente survey e controlli read-only: host, Docker, - checkout, Compose, Nginx, TLS, Aritmolab, Authentik e PostgreSQL. -- **Luna** esegue comandi bounded e verifiche ripetibili: build, compose, `tht` - (`status`, `doctor`, `pi`), smoke test, preprocessing e migrazioni già - autorizzate dal piano. - -Se Terra o Luna non sono disponibili, lavora in sequenza con il modello -disponibile. Non simulare agenti inesistenti. - -Per ogni incarico delegato specifica sempre: obiettivo, comandi consentiti, -operazioni vietate, evidenze da raccogliere e formato della risposta: - -```text -status: PASS | FAIL | BLOCKED -facts: fatti osservati -commands: comandi eseguiti (senza segreti) -evidence: file o output redatti -risks: rischi residui -blockers: impedimenti -``` - -Non riportare password, token, cookie, client secret o variabili d'ambiente -sensibili nei log o nei report. - -Le attività read-only indipendenti possono essere eseguite in parallelo. Tutte -le mutazioni devono essere sequenziali, con checkpoint e verifica prima della -fase successiva. Non parallelizzare stop/start dello stack, build/recreate, -migrazioni, modifiche ad Authentik, Nginx o al bilanciatore. - -## Regole inderogabili - -1. Inizia soltanto con il survey read-only. -2. Non spegnere, modificare o rimuovere il vecchio ThothII prima del survey e - del backup verificabile. -3. Non procedere a Project A senza un report Survey `PASS`. -4. Project A usa autenticazione locale e deve essere provato completamente prima - di iniziare Project B. -5. Project B con Authentik parte solo dopo il `PASS` esplicito di Project A. -6. Il database Supabase esistente va riusato tramite schemi dedicati; non creare - un nuovo database per isolare il dataset. -7. Il workspace remoto `tht-workspace-psd` resta unico: REST sul Mac e - PostgreSQL diretto sul server. I segreti non vanno in Git. -8. Il link dalla sidebar di Aritmolab deve restare funzionante e il percorso - pubblico finale deve passare dal balancer e da Nginx. -9. Conserva sempre un rollback verso il vecchio stack e verso l'autenticazione - locale finché il cutover non è approvato. - -## Prima azione richiesta - -Leggi tutti i documenti normativi. Poi avvia **solo Task 1 — Survey** del piano -generale. Produci un report consolidato usando il relativo template, con esito -`GO` o `NO-GO`, senza eseguire modifiche persistenti. Fermati e segnala ogni -credenziale Authentik mancante, permesso insufficiente, ambiguità sul balancer o -discrepanza tra dominio osservato e configurazione di Aritmolab. - -Dopo il survey attendi l'approvazione del proprietario prima di eseguire -Project A. diff --git a/docs/operations/psd-server-survey-remediation-checklist.md b/docs/operations/psd-server-survey-remediation-checklist.md deleted file mode 100644 index ec99119a..00000000 --- a/docs/operations/psd-server-survey-remediation-checklist.md +++ /dev/null @@ -1,455 +0,0 @@ -# PSD Server Survey — Remediation Checklist - -## Purpose and authority - -Questo documento permette al proprietario e a Sol di discutere, decidere e chiudere uno alla volta -i blocker emersi dal survey read-only del server PSD. È il punto di ripresa operativo tra sessioni: -registra soltanto fatti sanitizzati, decisioni, responsabili e riferimenti a evidenze protette. - -Fonti normative: - -- `docs/operations/psd-server-sol-orchestration-prompt.md` -- `docs/plans/2026-08-20-psd-server-deployment-program.md` -- `docs/plans/2026-08-20-psd-server-survey.md` -- `docs/superpowers/specs/2026-08-20-psd-survey-remediation-checklist-design.md` - -Questo documento non autorizza modifiche a server, servizi, database, Nginx, load balancer, -Authentik, Aritmolab o repository esterni. - -## Program gate - -- Program result: `SURVEY_NO_GO` -- Survey report: `/var/tmp/thothii-psd-survey.fJh7DS/survey-report.md` -- Survey report SHA-256: `36461b6c7d1e44d055f24d6919e892b7017352ac9eb99ac67b9ae62f0614f8e7` -- Legacy stack: deve restare acceso e invariato durante la discussione di questa lista -- La preparazione statica di Project A privato è autorizzata; stop del legacy e start del nuovo - restano vietati fino a `SURVEY_GO_PROJECT_A_PRIVATE` e a un consenso di mutazione separato -- Project B remains forbidden fino ai PASS automatico, umano e del proprietario per Project A - -## Current activity and resume point - -- Current activity: `2` -- Title: Identify accountable owners for the private Project A scope -- Resume from: Activity 2, assign owner/authority for legacy rollback, direct DWH, workspace and Pi/LLM -- Discussion rule: una sola attività può essere `IN_DISCUSSION` -- Allowed states: `PENDING`, `IN_DISCUSSION`, `DEFERRED_PRE_PROJECT_B`, `BLOCKED`, `PASS` - -## How to use this checklist - -1. Leggere `Current activity` e `Resume from`. -2. Discutere soltanto l'attività corrente. -3. Non inserire password, token, cookie, chiavi private, stringhe di connessione, claim grezzi o - valori di secret. -4. Registrare solo percorsi protetti, owner, mode, timestamp, nomi o ID di oggetti, checksum ed - esiti sanitizzati. -5. Alla fine della discussione aggiornare stato, note, decisione, evidenze, blocker e prossimo - passo. -6. Spostare `Current activity` solo quando il gate dell'attività corrente è soddisfatto oppure il - proprietario decide esplicitamente di parcheggiarla come `BLOCKED`. -7. Non riaprire un'attività `PASS` salvo nuova evidenza che ne invalidi la decisione. - -## Activity summary - -| ID | Attività | Stato | Responsabile | Prossimo gate | -|---|---|---|---|---| -| 1 | Rotazione controllata della credenziale DWH esposta | `DEFERRED_PRE_PROJECT_B` | Proprietario del progetto | Chiusura obbligatoria prima di Project B | -| 2 | Assegnazione dei responsabili dei componenti condivisi | `IN_DISCUSSION` | Proprietario del progetto | Owner privati A e shared B distinti | -| 3 | Risoluzione del dominio pubblico `.it` oppure `.com` | `BLOCKED` | Unassigned | Necessario per Project B, non per A privato | -| 4 | Topologia e responsabilità del load balancer | `BLOCKED` | Unassigned | Necessario per Project B/route opzionale | -| 5 | Accesso read-only protetto ad Authentik | `BLOCKED` | Unassigned | Necessario per Project B, non per A privato | -| 6 | Accesso catalog-only protetto a PostgreSQL | `BLOCKED` | Unassigned | DWH direct read-only ancora da provare | -| 7 | Confine temporaneo e cleanup del vecchio ThothII | `PASS` | Proprietario del progetto | Retain fino a Project B PASS; cleanup esatto separato | -| 8 | Accesso Git read-only al workspace PSD | `BLOCKED` | Curator da confermare | Checkout/deploy key server mancanti | -| 9 | Metadati Pi e LLM verificabili | `BLOCKED` | Unassigned | Policy e reachability redatte mancanti | -| 10 | Conservazione evidenze e nuovo survey bounded | `PENDING` | Unassigned | Nuovo report e decisione proprietario | - -## Activity 1: Rotate or revoke the exposed DWH credential safely - -- Status: `DEFERRED_PRE_PROJECT_B` -- Accountable owner: Proprietario del progetto (confermato dall'utente) -- Objective: sostituire o revocare in modo controllato la credenziale DWH comparsa nell'output - interno del survey, senza interrompere consumer legittimi e senza esporne nuovamente il valore. -- Why this is required: la credenziale deve essere considerata compromessa; non può essere usata - come base affidabile per completare il survey o iniziare Project A. -- Ordered actions: - 1. identificare il team che gestisce la route DWH, il suo meccanismo di autenticazione e la - custodia del secret; - 2. determinare il tipo di credenziale senza leggerla o copiarla in questa checklist; - 3. inventariare i consumer tramite riferimenti di configurazione e secret object; - 4. scegliere una transizione a doppia credenziale oppure una finestra atomica con rollback; - 5. generare e distribuire il nuovo secret attraverso il meccanismo protetto approvato; - 6. verificare i consumer autorizzati, l'assenza del nuovo valore nei log e la continuità del - vecchio stack; - 7. revocare la vecchia credenziale e provare che non venga più accettata; - 8. registrare soltanto evidenze redatte. -- Required redacted evidence: - - owner e autorizzazione della rotazione; - - tipo e identificatore non sensibile della credenziale; - - percorso protetto o secret object, senza contenuto; - - elenco dei consumer aggiornati; - - timestamp e risultati dei test positivi e negativi; - - conferma di revoca della credenziale precedente; - - procedura di rollback e relativo esito. -- Discussion notes: - - la credenziale osservata è stata identificata, senza rileggerne il valore, come chiave statica - `X-API-Key` applicata da Nginx alla route DWH REST `/dwh/`; è distinta dalle password PostgreSQL; - - `/home/chirone/chirone/etl` documenta PostgREST come integrazione HTTP esterna, mentre i processi - ETL e Superset usano PostgreSQL diretto; - - il vecchio ThothII e Chirone WP3 usano PostgreSQL diretto nei profili locali/server; Supabase - Studio è un pannello amministrativo e non appartiene al data-plane applicativo; - - il design approvato resta invariato: il Mac usa `rest_api`, il nuovo ThothII sul server PSD usa - `postgres_direct` con ruolo DWH dedicato e realmente read-only; - - la ricognizione dei repository ha rilevato materiale sensibile hardcoded in file tracciati, - senza riportarne i valori. La bonifica e la rotazione dei segreti coinvolti restano obbligatorie. - - il censimento statico non trova consumer `/dwh/` in ETL, Superset o nel Chirone WP3 attivo: - usano PostgreSQL diretto. Il vecchio container `thothii-core-1` è configurato con - `transport: direct`; i due ThothII restano comunque tecnicamente capaci di usare REST; - - i log Nginx redatti provano 18.523 richieste `/dwh/` dal 23 luglio al 19 agosto 2026: - 18.481 hanno user-agent classificato `python-requests` e gli endpoint RPC corrispondono - prevalentemente a introspezione e campionamento Thoth. La sorgente è una sola, privata e - compatibile con un proxy/load balancer; non prova che esista un solo client finale; - - l'ipotesi che il traffico sia generato dal job ETL delle 03:00 è smentita: nella finestra - 02:30–03:30 Europe/Rome non compare nessuna richiesta `/dwh/`. Il 99,37% del traffico è - concentrato il 13 agosto tra le 17:38 e le 20:03; - - la firma del 13 agosto corrisponde a undici preprocessing Thoth: undici `list_tables`, e - per ciascuna esecuzione 163 chiamate a ognuno dei tre RPC per-tabella più 1.180 `top_values`, - cioè 1.670 richieste per ciclo. `PROJECT_STATE.md` registra proprio il preprocessing PSD live - del 13 agosto su 163 tabelle, con più rerun e correzioni emerse durante l'esecuzione; - - il DAG ETL `nightly_etl_orchestrator` è schedulato con `0 3 * * *`, ma scrive il DWH tramite - PostgreSQL/`psycopg2` diretto. Nel codice tracciato non chiama `/dwh/`, gli RPC Thoth o - `tht workspace preprocess`, né emerge un trigger indiretto verso Thoth; - - la configurazione Nginx nominalmente attiva accetta una sola chiave tramite confronto letterale - in un endpoint `auth_request`. Non esiste una mappa a più chiavi: la doppia credenziale richiede - un refactor, backup, `nginx -t`, reload e rollback in una fase di mutazione autorizzata. - - il proprietario conferma che il ThothII sul Mac deve continuare a usare REST e che sono previste - molte altre installazioni remote, senza tunnel SSH verso Supabase. `/dwh/` è quindi - un'interfaccia remota stabile e multi-client, non una compatibilità temporanea. - - il proprietario decide di mantenere il certificato TLS corrente. L'endpoint esterno REST - presenta lo stesso certificato self-issued di Nginx, valido fino al 21 giugno 2027 e con SAN - per `supabase-aritmolab.policlinicosandonato.it`; non copre un eventuale dominio `.com`; - - il manuale di installazione deve trattare `TLS_CA_FILE` come necessario per ogni client che - non abbia già quel certificato nel proprio trust store, spiegando consegna affidabile, - verifica del fingerprint, rinnovo e aggiornamento coordinato delle installazioni; - - non esiste un ambiente di test. La rotazione dovrà quindi usare una verifica production-safe: - backup, finestra dual-key, RPC `ping` senza dati clinici, test positivo/negativo e rollback. - - il proprietario approva un componente `dwh-auth` riutilizzabile ma opzionale, incluso nel - repository senza modificare il protocollo dei client portabili o il CLI `tht`; - - su PSD `dwh-auth` avrà un lifecycle `systemd` indipendente dallo stack ThothII e comunicherà - con Nginx tramite socket Unix. Lo stop o la sostituzione di ThothII non dovrà interrompere i - client REST; - - il registro sarà composto da file protetti, versionati e aggiornati atomicamente, con un file - per generazione della chiave. Conterrà digest SHA-256 di segreti casuali da almeno 256 bit e - metadati non sensibili, senza SQLite o nuove dipendenze runtime; - - le chiavi saranno assegnate alle installazioni, non alle persone. Saranno prive di scadenza - predefinita, con scadenza opzionale e revoca manuale; - - creazione e import leggeranno o scriveranno soltanto file protetti. Nessun segreto sarà - accettato come argomento, stampato o inserito in log, JSON, documenti o repository; - - la gestione sarà fail-closed: credenziali non valide riceveranno `401`, mentre guasti del - servizio o del registro saranno mappati a `503` senza fallback permissivo; - - il proprietario conferma che le sessioni, gli indici e le cache del vecchio ThothII erano solo - test e non richiedono migrazione o attività di salvaguardia dedicate. Restano necessari il - rollback della route DWH condivisa e il rispetto del gate esplicito prima di fermare lo stack; - - il proprietario ha revisionato e approvato la specifica scritta. Il piano eseguibile è in - `docs/superpowers/plans/2026-08-20-dwh-rest-per-installation-auth.md`; separa sviluppo e - verifica del componente dai due gate espliciti di mutazione PSD; - - la credenziale condivisa corrente, priva del nuovo identificativo pubblico, sarà l’unico record - temporaneo `legacy_raw` con ID `legacy-shared`. Dopo la revoca non saranno accettate chiavi - prive del formato versionato per installazione. -- Decision: il nuovo ThothII server userà PostgreSQL diretto; il Mac e le future installazioni - remote useranno REST. La chiave condivisa corrente non è un modello finale accettabile: la - rotazione esposta userà una transizione a doppia credenziale e l'architettura target assegnerà - un'identità revocabile distinta a ogni installazione. Supabase Studio non sarà usato come - trasporto. Il proprietario del progetto è accountable per coordinare la rotazione. Il meccanismo - target è il servizio server-only `dwh-auth`, indipendente dallo stack ThothII, con registro a file - e una chiave revocabile per installazione. -- Owner amendment 2026-08-21: Gate A e dual-key Gate B sono eseguiti; il Mac live test, le 48 ore - comprendenti due cicli ETL delle 03:00 e la revoca di `legacy-shared` sono rinviati al gate - obbligatorio prima di Project B. Il rinvio non equivale a PASS. -- Blockers: per chiudere Activity 1 restano il collaudo Mac, l'osservazione completa, la revoca, - v1 positivo post-revoca e legacy `401`. -- Next step: continuare Activity 2–10 per lo scope Project A privato; riaprire Activity 1 prima di - congelare il candidato Project B. - -## Activity 2: Identify accountable owners for shared components - -- Status: `IN_DISCUSSION` -- Accountable owner: Unassigned -- Objective: associare ogni componente condiviso a una persona o a un team con autorità di lettura, - modifica, approvazione e rollback. -- Why this is required: la leggibilità di una configurazione non implica autorità a modificarla. -- Ordered actions: - 1. identificare gli owner di DNS/load balancer, Nginx/certificati, Authentik, - Supabase/PostgreSQL, Aritmolab, workspace Git e backup legacy; - 2. registrare il canale di approvazione e la procedura di escalation; - 3. confermare separatamente chi può autorizzare Project A e Project B. -- Required redacted evidence: nomi dei team, ruoli, canali operativi e conferme di responsabilità; - nessun contatto personale sensibile. -- Discussion notes: Not discussed -- Decision: No decision recorded -- Blockers: nessun owner condiviso è stato ancora formalmente confermato. -- Next step: compilare ora la matrice distinguendo componenti necessari a Project A privato e - componenti shared/pubblici rinviabili a Project B. - -## Activity 3: Resolve the authoritative public origin - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: scegliere sulla base di evidenze l'unica origine pubblica finale tra il dominio `.it` - osservato e il dominio `.com` riportato nel piano. -- Why this is required: callback OIDC, certificato, cookie, Nginx, load balancer e sidebar devono - concordare sulla stessa origine HTTPS. -- Ordered actions: - 1. ottenere la dichiarazione autorevole dell'owner DNS/load balancer; - 2. verificare record DNS, route, backend, certificato e redirect; - 3. confrontare l'origine con la configurazione e la sidebar di Aritmolab; - 4. registrare l'origine approvata e le discrepanze da correggere in Project B. -- Required redacted evidence: hostname finale, record/route sanitizzati, SAN del certificato, - destinazione sidebar e approvazione dell'owner. -- Discussion notes: il survey ha osservato `.it`; il piano cita `.com`. La discrepanza è aperta. -- Decision: No decision recorded -- Blockers: owner DNS/load balancer non identificato e origine browser-visible non provata. -- Next step: ottenere la dichiarazione autorevole dopo l'assegnazione degli owner. - -## Activity 4: Establish the load-balancer contract - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: documentare il confine effettivo del load balancer e la procedura reversibile per le - route temporanea e finale. -- Why this is required: il survey locale non ha potuto provare owner, backend, health check, TLS, - source range o allowlist. -- Ordered actions: - 1. identificare superficie di configurazione e owner; - 2. registrare backend, porta, health check, punto TLS e source range verso Nginx; - 3. documentare deploy, validazione e rollback; - 4. stabilire se una route temporanea può essere limitata agli operatori; - 5. definire una prova positiva e una negativa dell'allowlist senza creare ancora la route. -- Required redacted evidence: nomi/ID delle route, backend e health check sanitizzati, ownership, - capacità di allowlist e procedura di rollback. -- Discussion notes: Not discussed -- Decision: No decision recorded -- Blockers: il load balancer non è ispezionabile dalla superficie locale autorizzata. -- Next step: coinvolgere l'owner identificato nell'Activity 2. - -## Activity 5: Provide protected read-only Authentik survey access - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: permettere un inventario Authentik bounded e read-only della versione installata. -- Why this is required: applicazioni, provider, flow, mapping, gruppi, service account, permessi API - e procedura di export non sono stati verificati. -- Ordered actions: - 1. identificare l'owner Authentik e la procedura di backup/export; - 2. predisporre una credenziale read-only o un'esecuzione assistita dall'owner; - 3. comunicare solo percorso, owner, mode e usabilità del secret; - 4. inventariare nomi/ID e convenzioni senza recuperare secret write-only; - 5. confrontare il comportamento con OpenAPI e documentazione della release installata. -- Required redacted evidence: versione, nomi/ID degli oggetti, permission set della credenziale, - riferimento all'export e risultati sanitizzati. -- Discussion notes: Not discussed -- Decision: No decision recorded -- Blockers: nessuna credenziale amministrativa/API utilizzabile è stata stabilita. -- Next step: ottenere dall'owner un meccanismo protetto post-rotazione. - -## Activity 6: Provide protected catalog-only PostgreSQL survey access - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: verificare database, schemi, ruoli, grant, migrazioni e PostgREST con sole query di - catalogo. -- Why this is required: il runtime DWH read-only, lo stato di `thoth_sessions` e i confini Supabase - non sono provati. -- Ordered actions: - 1. identificare DBA e procedura protetta di connessione; - 2. verificare database, utente corrente, schemi e owner; - 3. verificare i grant sullo schema `datawarehouse` senza write probe clinici; - 4. verificare stato di `thoth_sessions` e migration records; - 5. verificare gli schemi esposti da PostgREST; - 6. registrare TLS/CA, backup e convenzioni per ruoli migrator/runtime. -- Required redacted evidence: risultati catalogici bounded, nomi dei ruoli, attributi e grant, - schemi PostgREST, riferimento a backup e TLS; nessuna stringa di connessione. -- Discussion notes: il wrapper ETL `ConnectionFactory` ha aperto una sessione dichiarata - read-only e ha eseguito sole query aggregate a `pg_catalog`. Il database è PostgreSQL 15.8; - l'identità disponibile è `postgres`, owner dello schema `datawarehouse`, con `USAGE` e - `CREATE`. Su tutte le 163 relazioni catalogate possiede SELECT e anche tutti i privilegi di - scrittura/DDL tabellari verificati. Nessun nome tabella o dato clinico è stato raccolto. -- Decision: il meccanismo esistente è valido per il survey catalogico, ma è vietato come identità - runtime del nuovo core perché non è least-privilege né read-only. -- Blockers: il DBA deve fornire un ruolo dedicato con soli USAGE/SELECT e una route diretta - certificabile dal nuovo core. Il PostgREST DWH è loopback host su 127.0.0.1:3001 e non prova il - percorso PostgreSQL diretto richiesto da Project A. -- Next step: definire con il DBA ruolo, secret-file protetto, TLS/rete e query di grant da ripetere; - non creare il ruolo durante il survey. - -## Activity 7: Bind the disposable legacy boundary and cleanup exclusions - -- Status: `PASS` -- Accountable owner: Proprietario del progetto (confermato dall'utente) -- Objective: mantenere il vecchio ThothII solo come confine temporaneo di cutover e rimuovere - esclusivamente le sue risorse dopo il PASS reale di Aritmolab. -- Why this is required: Aritmolab usa oggi i container legacy, ma il proprietario ha dichiarato - sacrificabili sessioni, configurazione e dati del vecchio ThothII. -- Ordered actions: - 1. inventariare nuovamente container, image ID, source e data immediatamente prima dello stop; - 2. preservare invariati i due network condivisi e l'Evidence ETL esterna; - 3. chiudere la route e fermare solo i container legacy nel gate Project A autorizzato; - 4. conservare container, immagini, source e data fino al PASS automatico, umano e owner di B; - 5. rimuovere poi soltanto gli exact target sotto un'autorizzazione cleanup separata. -- Required redacted evidence: inventario esatto, owner, restart recipe, esclusioni shared, decisioni - Project A/B e manifest finale di cleanup; nessun contenuto di secret. -- Discussion notes: il legacy stack è ancora attivo e invariato. Compose project `thothii` usa - `/home/chirone/ThothII/compose.yaml`; i servizi sono `core` e `frontend`, senza named volume. - Il solo bind applicativo RW è `/home/chirone/thothii-data` (con i bind Pi annidati); Evidence è - un bind RO esterno. Una lettura tar verso `/dev/null` di source e data ha dato - `legacy_backup_readability=PASS`. Il source è circa 1.05 GB e il data bind circa 1.9 MB. - I dry-run Compose passano solo fornendo il path non segreto - `PI_AUTH_FILE=/home/chirone/thothii-data/pi-config/agent/auth.json` insieme a - `--env-file deploy/thothii.env -p thothii -f compose.yaml`; stop individua entrambi i container - e start è sintatticamente valido (non trova container arrestati mentre lo stack è ancora attivo). -- Decision: nessun backup legacy è richiesto. I container, immagini, source e data già presenti - restano il solo rollback temporaneo fino al PASS Project B; non si crea alcun utente host. Le - risorse esclusive potranno essere cancellate dopo il collaudo Aritmolab, mentre network shared, - ETL Evidence, Omics, LocalLLM, DWH, `dwh-auth`, Supabase, Authentik e Superset sono esclusi. -- Blockers: nessuno per questa decisione survey. Stop/start, route change e cleanup restano tre - autorizzazioni di mutazione separate e non sono autorizzati da questo PASS. -- Next step: usare il design e piano clean-replacement approvati; non eseguire ancora mutazioni. - -## Activity 8: Provide read-only PSD workspace Git access - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: verificare il repository remoto condiviso e il suo stato corrente dal server senza - capacità di push. -- Why this is required: SHA, descriptor, trasporti, Evidence, annotazioni e scope della deploy key - non sono stati osservati dal server. -- Ordered actions: - 1. identificare curator e owner della deploy key; - 2. fornire un riferimento protetto alla chiave server read-only; - 3. verificare remote, branch e SHA con modalità non interattiva; - 4. verificare catalogo, schema v3, trasporti, Evidence e annotazioni; - 5. provare che la credenziale server non possa effettuare push. -- Required redacted evidence: remote, branch, SHA, descriptor blob, stato Evidence/annotations e - attestazione read-only della deploy key. -- Discussion notes: Not discussed -- Decision: No decision recorded -- Blockers: nessun checkout workspace o deploy credential utilizzabile è stato localizzato. -- Next step: coinvolgere il curator e predisporre l'accesso server read-only. - -## Activity 9: Make Pi and LLM metadata verifiable - -- Status: `BLOCKED` -- Accountable owner: Unassigned -- Objective: verificare versione Pi, provider, modello, thinking level, riferimento credenziale e - reachability LLM senza esporre il secret. -- Why this is required: la root Pi legacy non è attraversabile dall'operatore del survey e i - default del nuovo source non provano la configurazione in esecuzione. -- Ordered actions: - 1. identificare owner della configurazione Pi/LLM; - 2. scegliere tra esecuzione assistita dall'owner e accesso read-only allowlisted; - 3. estrarre esclusivamente metadati non sensibili; - 4. eseguire un controllo bounded di reachability senza stampare credenziali; - 5. registrare anche il mismatch NVML/GPU come rischio separato, non come blocker CPU. -- Required redacted evidence: versione, provider, model ID, thinking level, endpoint sanitizzato, - percorso/mode della credenziale e risultato di reachability. -- Discussion notes: host `x86_64`; due GPU NVIDIA osservate, ma `nvidia-smi` non è utilizzabile per - mismatch driver/libreria NVML. Una lettura whitelist di `settings.json` ha rilevato Pi 0.80.3, - provider `deepseek`, modello `deepseek-v4-pro` e thinking `high`, senza leggere - `auth.json`. Un singolo probe senza tool, contesto o sessione ha prodotto solo - `pi_reachability=FAIL`. Il catalogo custom dichiara inoltre il solo provider `local-qwen`. -- Decision: i metadati sono verificati, ma reachability e coerenza default/catalogo non passano. -- Blockers: diagnosticare il FAIL senza esporre la credenziale e confermare il provider/modello - approvato per Project A; il mismatch NVML/GPU resta rischio separato, non un motivo per assumere - che il percorso CPU funzioni. -- Next step: eseguire un controllo assistito e sanitizzato della configurazione provider, quindi - ripetere una sola reachability probe bounded. - -## Read-only resume — 2026-08-21 - -- Host/source: Linux x86_64, Docker 29.1.1, Compose 2.40.3, application worktree clean at - `7118950416b3008a8182825de027c7f8b235de57`; Qdrant/Ollama images are local, while the required - embedding image/model is not yet proved local. -- Capacity: approximately 1.1 TB free on `/home` and 36 GB on `/`; several loopback candidate - ports are currently free. These facts do not reserve a path or port. -- Legacy: project `thothii` is still running and unchanged. Source is - `/home/chirone/ThothII` at `6ca4275`; the only RW application bind is - `/home/chirone/thothii-data`, plus the nested Pi binds. The source is about 1.05 GB and the data - bind about 1.9 MB. No backup was created. -- Recovery decision: legacy state is disposable; no backup is required. Existing stopped - containers, images, source and data remain only as the temporary Project A/B rollback boundary. -- Identity decision: neither UID nor GID 10001 maps to a host account. No host identity will be - created. The new image retains numeric `10001:10001`, confined to its distinct writable roots; - the existing operator owns source/configuration. Recheck both `getent` lookups before creating - paths and stop on any new mapping. -- Shared exclusion: never remove the Omics/LocalLLM networks or external ETL Evidence bind. -- Candidate paths: documented examples `/srv/thothii` and `/srv/thothii-backups` are absent and - therefore only candidates; they have not been created. `127.0.0.1:18080` è il candidato - frontend e risultava libero al momento del survey, ma non è riservato e va ricontrollato prima - dello start. Existing `/home/chirone/thothii-data` must not be reused. -- Workspace: the canonical private remote is documented as - `git@github.com:mptyl/tht-workspace-psd.git`; a non-interactive read-only remote query resolved - `main` at `bfbabf9f2defcf861a3225296eaff8c6d44c0ac9`. No server checkout or dedicated deploy-key - reference is present at the documented local paths, and the credential's inability to push is - not yet proved. -- Pi/LLM: legacy Pi version `0.80.3` is visible, but provider/model/thinking/credential reference - and bounded reachability remain unknown. -- Shared scope: `.it` resolves locally and `.com` does not, Nginx is valid/active, Authentik - 2026.2.1 and Supabase components are running. Owner, LB contract, Authentik inventory and - PostgreSQL catalog grants remain unproved and are not inferred. -- Decision: `SURVEY_NO_GO` for Project A private remains. The bounded work authorized now is - limited to planning/static preparation; no source clone, protected tree, backup, stop or start - has been performed. - -## Activity 10: Retain evidence and run the missing bounded survey checks - -- Status: `PENDING` -- Accountable owner: Unassigned -- Objective: conservare le evidenze protette, ripetere soltanto i controlli mancanti e produrre una - nuova decisione verificabile. -- Why this is required: il report corrente è `SURVEY_NO_GO` e non può essere promosso per inferenza. -- Ordered actions: - 1. scegliere il protected evidence root definitivo; - 2. trasferire la directory del survey senza modificarne i contenuti e verificare il digest; - 3. confermare che le Activity 1–9 siano `PASS` oppure abbiano una risoluzione proprietario - esplicitamente accettata; - 4. eseguire solo i controlli bounded mancanti del piano survey; - 5. aggiornare il report e verificarne checksum e secret hygiene; - 6. chiedere la decisione esplicita del proprietario. -- Required redacted evidence: percorso finale, digest, matrice Activity 1–9, nuovi risultati - bounded, report aggiornato e decisione firmata. -- Discussion notes: Not discussed -- Decision: No decision recorded -- Blockers: dipende dalla chiusura delle Activity 1–9 e dall'approvazione del retention root. -- Next step: avviare soltanto dopo la chiusura dei blocker precedenti. - -## Fresh survey and owner gates - -Un nuovo `SURVEY_GO_PROJECT_A_PRIVATE` richiede: - -- owner e autorità di stop/start/rollback identificati per il legacy e Project A; -- accessi read-only PostgreSQL, workspace Git e Pi/LLM verificati; -- identità DWH dimostrata read-only; -- inventario, restart recipe e cleanup exclusions legacy verificabili; -- risorse e percorsi della nuova installazione approvati; -- report redatto, secret-scan valido e checksum verificato; -- approvazione esplicita del proprietario. - -`SURVEY_GO_PROJECT_B` richiede inoltre: - -- collaudo Mac `rest_api` con chiave per installazione; -- 48 ore di osservazione comprendenti due cicli ETL delle 03:00; -- credenziale `legacy-shared` revocata, v1 positiva e legacy `401`; -- owner e autorità di modifica/rollback per tutti i componenti shared; -- origine pubblica unica e load-balancer contract provati; -- accesso read-only Authentik e inventario della release installata; -- ogni altro blocker pubblico/shared delle Activity 2–9 chiuso. - -Il PASS tecnico del survey non autorizza automaticamente Project A. L'autorizzazione deve essere -registrata separatamente. - -## Change log - -| Data | Attività | Modifica | Autore | -|---|---|---|---| -| 2026-08-20 | Initial | Creata checklist; Activity 1 aperta, Activity 2–10 pending | Sol | -| 2026-08-21 | Sequencing amendment | Activity 1 deferred pre-B; survey ripreso read-only; Project A private resta NO-GO | Owner/Sol | -| 2026-08-21 | Clean replacement | Activity 7 PASS; legacy disposable dopo B; nessun account host 10001 | Owner/Sol | diff --git a/docs/plans/2026-07-21-button-press-feedback-design.md b/docs/plans/2026-07-21-button-press-feedback-design.md deleted file mode 100644 index 0c63f6ce..00000000 --- a/docs/plans/2026-07-21-button-press-feedback-design.md +++ /dev/null @@ -1,19 +0,0 @@ -# Button Press Feedback Design - -## Context - -Buttons currently change on hover, but many provide little or no visible acknowledgment while the pointer is pressed. The interface mixes a shared Base UI button with native buttons, so changing only the shared component would leave inconsistent behavior. - -## Chosen interaction - -Apply one CSS press vocabulary to every enabled native button. During `:active`, the button compresses to `scale(0.97)`, loses raised shadow, and receives a restrained brightness change. The transition lasts 140 ms and uses an ease-out-quint curve (`cubic-bezier(0.22, 1, 0.36, 1)`). This reads as a physical press without bounce, ripple, layout movement, or JavaScript state. - -The existing one-pixel translation on the shared button is removed so shared and native buttons do not combine two motion patterns. - -## Accessibility - -Under `prefers-reduced-motion: reduce`, scale is disabled. The brightness and shadow change remain, providing a clear pressed state without kinetic motion. Disabled buttons receive no press treatment. - -## Verification - -A focused contract test checks the global enabled-button selector, scale, timing, easing, and reduced-motion override. The full frontend suite and typecheck guard regressions. Playwright then holds a real button in the active state and confirms its computed transform, followed by a visual screenshot/snapshot check. diff --git a/docs/plans/2026-07-21-button-press-feedback.md b/docs/plans/2026-07-21-button-press-feedback.md deleted file mode 100644 index fe01afdb..00000000 --- a/docs/plans/2026-07-21-button-press-feedback.md +++ /dev/null @@ -1,72 +0,0 @@ -# Button Press Feedback Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Give every enabled button immediate, consistent click acknowledgment while preserving a non-kinetic reduced-motion alternative. - -**Architecture:** Define the interaction once in the global Tailwind base layer so both Base UI and native buttons inherit it. Remove the shared button's older translation-only active state to avoid compounded transforms. Verify the CSS contract first, then exercise the real interaction in Playwright. - -**Tech Stack:** React 18, Tailwind CSS 3, Vitest, Playwright CLI. - ---- - -### Task 1: Specify the global press contract - -**Files:** -- Create: `frontend/src/button-press-feedback.test.ts` -- Test: `frontend/src/button-press-feedback.test.ts` - -**Step 1: Write the failing test** - -Read `src/index.css` and assert the enabled-button active selector, `scale(0.97)`, 140 ms duration, ease-out-quint curve, disabled exclusion, and reduced-motion transform override. Assert that `components/ui/button.tsx` no longer contains the legacy translation active class. - -**Step 2: Run test to verify it fails** - -Run: `npx vitest run src/button-press-feedback.test.ts` - -Expected: FAIL because the global press rules do not exist and the shared button still uses translation. - -### Task 2: Implement the press feedback - -**Files:** -- Modify: `frontend/src/index.css` -- Modify: `frontend/src/components/ui/button.tsx` -- Test: `frontend/src/button-press-feedback.test.ts` - -**Step 1: Add the minimal CSS** - -Add a global enabled-button transition and active state using only transform, filter, and shadow. Add a `prefers-reduced-motion` override that removes scale while preserving non-kinetic contrast feedback. - -**Step 2: Remove the legacy shared-button translation** - -Delete `active:not-aria-[haspopup]:translate-y-px` from the shared variant base string. - -**Step 3: Run the focused test** - -Run: `npx vitest run src/button-press-feedback.test.ts` - -Expected: PASS. - -### Task 3: Verify regressions and real-browser behavior - -**Files:** -- Verify: `frontend/src/index.css` -- Verify: `frontend/src/components/ui/button.tsx` - -**Step 1: Run frontend verification** - -Run: `npx vitest run` - -Run: `npx tsc -b` - -Expected: all tests pass and typecheck exits 0. - -**Step 2: Verify in Playwright** - -Open `http://localhost:5173`, hold pointer-down on an enabled button, and inspect its computed transform and filter before release. Repeat with reduced motion emulation and confirm transform remains `none` while contrast feedback remains. - -**Step 3: Review the final diff** - -Run: `git diff --check` and inspect `git diff --stat`. - -Expected: no whitespace errors and only the intended product/design, CSS, component, and test files changed. diff --git a/docs/plans/2026-08-08-internal-qdrant-ollama-design.md b/docs/plans/2026-08-08-internal-qdrant-ollama-design.md deleted file mode 100644 index b6965b36..00000000 --- a/docs/plans/2026-08-08-internal-qdrant-ollama-design.md +++ /dev/null @@ -1,210 +0,0 @@ -# Internal Qdrant and Ollama Architecture Design - -**Status:** approved on 2026-08-08 - -## Objective - -ThothII owns its semantic infrastructure. Every supported deployment includes a private Qdrant -service and a private Ollama embedding service. The analytical DWH remains external and read-only; -each workspace descriptor associates that DWH with one Qdrant collection used for database schema, -Evidence, and approved Memory records. - -## Decisions - -- Qdrant replaces pgvector as the only operational vector store. -- Ollama replaces workspace-selected external embedding endpoints. -- The default and required model is `qwen3-embedding:0.6b` with 1024-dimensional normalized dense - embeddings and cosine distance. -- One Qdrant collection belongs to one workspace. Schema, Evidence, and Memory points share that - collection and are separated by indexed payload field `kind`. -- Qdrant and Ollama are mandatory base-Compose services. They are not published on host ports and - are reachable only from the private Compose network. -- Existing schema-v1 and schema-v2 descriptors remain readable for migration, but they are not - activatable. The new operational contract is workspace schema v3. - -The model choice is based on the published Qwen model card: the 0.6B model supports more than 100 -languages, a 32K context window, Matryoshka dimensions up to 1024, and instruction-aware retrieval. -Ollama distributes a CPU-viable quantized build and can use an exposed GPU without changing the -application protocol. - -References: - -- -- -- -- -- - -## Target topology - -```text -browser -> frontend -> core -> external DWH - -> private Qdrant - -> private Ollama embedding -``` - -The base Compose project contains: - -- `frontend`: static React application and same-origin API proxy. -- `core`: Fastify, Pi, and the Python `tht` harness. -- `qdrant`: pinned Qdrant server with persistent `qdrant-data` volume. -- `embedding`: pinned Ollama server with persistent `embedding-models` volume. -- `embedding-model-init`: bounded one-shot service that pulls and verifies - `qwen3-embedding:0.6b`; `core` starts only after it succeeds. - -`qdrant` and `embedding` use `expose`, not `ports`. The core receives installation-owned internal -URLs: - -```text -THT_INTERNAL_QDRANT_URL=http://qdrant:6333 -THT_INTERNAL_EMBEDDING_URL=http://embedding:11434 -THT_INTERNAL_EMBEDDING_MODEL=qwen3-embedding:0.6b -THT_INTERNAL_EMBEDDING_DIMENSIONS=1024 -``` - -These are deployment facts, not workspace connector bindings. The runtime rejects non-loopback or -non-Compose-service hosts when these variables are overridden for development. - -An optional Linux GPU override exposes an available NVIDIA/AMD device to Ollama. The base profile -must remain CPU-safe. macOS Docker remains CPU-only because Docker Desktop cannot expose the Apple -GPU to an Ollama container. - -## Workspace schema v3 - -The workspace itself is the association between the external database and the internal collection: - -```yaml -workspace: - schema_version: 3 - id: psd-clinical - name: PSD Clinical - language: it - -dwh: - engine: postgres - database: postgres - schema: datawarehouse - supported_transports: [postgres_direct] - -semantic_index: - vector_store: - engine: qdrant - collection: psd-clinical - dimensions: 1024 - distance: cosine - embedding: - provider: ollama_internal - model: qwen3-embedding:0.6b - dimensions: 1024 - -llm_policy: - allowed: [zai/glm-5.2] -``` - -Invariants: - -- the collection name is an explicit portable identifier; -- active workspaces cannot share a collection; -- vector and embedding dimensions are both 1024; -- distance is `cosine`; -- provider and model are exactly the supported internal values; -- no vector transport, vector credential, embedding URL, or embedding credential may appear in a - schema-v3 descriptor or installation contract; -- DWH connectors remain installation-local and can still use the supported external DWH transports. - -Schema-v1/v2 pgvector descriptors are listed as `migration_required`. Migration creates a reviewed -schema-v3 document; it does not copy vector data implicitly. Existing semantic data is rebuilt from -the canonical schema documents, Evidence corpus, and Memory registry. - -## Qdrant data model - -Each point has a deterministic UUIDv5 derived from: - -```text -workspace_id + kind + record_key -``` - -The vector is the 1024-dimensional Ollama result. The payload is: - -```json -{ - "workspace_id": "psd-clinical", - "kind": "schema", - "source_id": "datawarehouse.patients", - "record_key": "schema:table:datawarehouse.patients", - "content_hash": "sha256:...", - "workspace_revision": "", - "generation": "", - "language": "it", - "text": "...", - "metadata": {} -} -``` - -`kind`, `source_id`, `content_hash`, `workspace_revision`, and `generation` receive keyword payload -indexes. Queries always filter by `workspace_id` and an explicit allowed `kind` set. Upsert is -idempotent. Evidence generation deletion is an exact filtered delete. Collection creation is also -idempotent and fails closed if an existing collection has incompatible dimensions or distance. - -## Harness integration - -The existing `VectorStore` port remains the workflow boundary. A `QdrantVectorStore` adapter maps -its operations to Qdrant REST endpoints while preserving current schema/Evidence/Memory call sites. -The existing Ollama embedding client is narrowed to the internal `/api/embed` contract and verifies: - -- configured model exists; -- output count matches input count; -- every vector has 1024 finite numeric values; -- no remote URL or API key is accepted. - -The JSONL Memory registry and persisted phase documents remain canonical. Qdrant remains a derived, -rebuildable semantic index. Schema, Evidence, and Memory ingestion all use the same point builder, -content hashing, and retry policy. - -## Readiness and failure behavior - -Readiness is layered: - -1. Compose waits for Qdrant health. -2. Compose waits for Ollama health and successful model initialization. -3. Workspace activation validates the schema-v3 contract. -4. Harness readiness ensures the Qdrant collection and checks its vector configuration. -5. Harness embeds a bounded probe and verifies 1024 dimensions. - -Failures are sanitized and fail closed: - -- unavailable Qdrant -> `workspace_not_activatable` before session persistence; -- unavailable or missing Ollama model -> `model_unavailable` before session persistence; -- collection mismatch -> `semantic_index_incompatible` without recreating or deleting data; -- embedding dimension mismatch -> no point write; -- partial batch failure -> operation reports failure and remains safe to retry. - -No health response, API response, or diagnostic log exposes DWH credentials or indexed text. - -## Deployment and migration - -The pgvector deployment path is retired: - -- remove local-vector Compose overlays and pgvector bootstrap/migration services; -- remove vector PostgreSQL role and password contracts; -- remove runtime support for vector REST/SSH and external embedding URLs; -- keep only the descriptor parser and migration code needed to recognize legacy workspaces; -- update local/server manuals, examples, smoke tests, CI coupling scans, backup instructions, and - release gates for four persistent stores plus Qdrant and Ollama volumes. - -Qdrant backup/restore uses collection snapshots or the persistent volume according to the operator -manual. Ollama model storage is a cache: it may be backed up for offline recovery but is not an -application source of truth. - -## Acceptance criteria - -- Base local and server Compose renders include healthy private `qdrant` and `embedding` services. -- A clean CPU-only installation downloads the model, creates a workspace collection, and embeds a - probe without external vector or embedding configuration. -- GPU override uses the same API and persistent model volume. -- Schema-v3 workspaces activate; schema-v1/v2 workspaces report `migration_required`. -- Two workspaces cannot claim the same Qdrant collection. -- Schema, Evidence, and Memory records coexist in one collection and remain filter-isolated. -- Existing workflow behavior and persisted session contracts remain unchanged. -- Tests reject all active pgvector deployment, external vector binding, and external embedding - configuration paths. diff --git a/docs/plans/2026-08-08-internal-qdrant-ollama.md b/docs/plans/2026-08-08-internal-qdrant-ollama.md deleted file mode 100644 index 075c2fa6..00000000 --- a/docs/plans/2026-08-08-internal-qdrant-ollama.md +++ /dev/null @@ -1,773 +0,0 @@ -# Internal Qdrant and Ollama Implementation Plan - -> **Historical nomenclature:** this plan predates the native host CLI convergence. The current -> operator command is `tht`; any older `thothctl` smoke-script or rollback wording below is retained -> only as historical evidence. - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Make Qdrant and Ollama mandatory internal ThothII services while keeping the analytical -DWH external and associating each workspace with one Qdrant collection for schema, Evidence, and -Memory embeddings. - -**Architecture:** Introduce workspace schema v3, preserve v1/v2 only as migration inputs, and keep -the existing harness `VectorStore` port behind a new Qdrant REST adapter. Base Compose owns Qdrant, -Ollama, their persistent volumes, and model initialization; workspace descriptors contain semantic -identity but no vector/embedding endpoints or credentials. - -**Tech Stack:** TypeScript/Fastify/Zod, Python 3.12/Pydantic/requests, React 18, Docker Compose, -Qdrant REST API, Ollama `/api/embed`, Vitest, pytest. - ---- - -## Guardrails - -- Apply `@superpowers:test-driven-development` to every behavior change: add one focused failing - test, observe the expected failure, implement the minimum, and rerun the focused test. -- Do not run broad suites until the corresponding code/config changes exist; this preserves the - requested ordering while still using TDD. -- Preserve the external DWH connector contract and session persistence model. -- Do not retain an operational fallback to pgvector or an external embedding endpoint. -- Do not delete or rewrite user workspace repositories or Qdrant data. Migration is descriptor-only; - semantic data is rebuilt explicitly. -- Commit after each task only when focused tests are green. - -### Task 1: Define workspace schema v3 - -**Files:** - -- Modify: `backend/src/workspaces/schema.ts` -- Modify: `backend/src/workspaces/types.ts` -- Modify: `backend/test/workspaces-schema.test.ts` -- Modify: `backend/test/workspaces-migrate-legacy.test.ts` -- Create: `backend/src/workspaces/migrate-v2-qdrant.ts` -- Create: `backend/test/workspaces-migrate-v2-qdrant.test.ts` - -**Step 1: Write the failing schema tests** - -Add tests proving that schema v3 accepts only this semantic shape: - -```ts -const semantic_index = { - vector_store: { - engine: "qdrant", - collection: "psd-clinical", - dimensions: 1024, - distance: "cosine", - }, - embedding: { - provider: "ollama_internal", - model: "qwen3-embedding:0.6b", - dimensions: 1024, - }, -}; -``` - -Add separate rejection cases for `pgvector`, `supported_transports`, external embedding providers, -non-1024 dimensions, non-cosine distance, and unknown fields. Assert v1/v2 remain parseable as -legacy descriptors but `isOperationalWorkspace()` returns false. - -**Step 2: Run the tests and verify RED** - -Run: - -```bash -cd backend -npx vitest run test/workspaces-schema.test.ts test/workspaces-migrate-v2-qdrant.test.ts -``` - -Expected: failure because schema version 3 and `migrateWorkspaceV2ToV3` do not exist. - -**Step 3: Implement the minimum schema and migration** - -Add `QdrantVectorStore`, `InternalEmbedding`, and `WorkspaceV3` types. Replace the operational type -guard with schema-v3-only semantics. Implement: - -```ts -export function migrateWorkspaceV2ToV3( - legacy: WorkspaceV2, - collection: string, -): WorkspaceV3 { - return validateOperationalWorkspace({ - workspace: { ...legacy.workspace, schema_version: 3 }, - dwh: legacy.dwh, - semantic_index: { - vector_store: { - engine: "qdrant", - collection, - dimensions: 1024, - distance: "cosine", - }, - embedding: { - provider: "ollama_internal", - model: "qwen3-embedding:0.6b", - dimensions: 1024, - }, - }, - llm_policy: legacy.llm_policy, - ...(legacy.diagnostics?.dwh_rest - ? { diagnostics: { dwh_rest: legacy.diagnostics.dwh_rest } } - : {}), - }); -} -``` - -Do not copy vector/embedding diagnostics or transports. - -**Step 4: Verify GREEN** - -Run the command from Step 2. Expected: all selected tests pass. - -**Step 5: Commit** - -```bash -git add backend/src/workspaces/schema.ts backend/src/workspaces/types.ts \ - backend/src/workspaces/migrate-v2-qdrant.ts backend/test/workspaces-schema.test.ts \ - backend/test/workspaces-migrate-legacy.test.ts backend/test/workspaces-migrate-v2-qdrant.test.ts -git commit -m "feat: define internal semantic workspace schema" -``` - -### Task 2: Make collection ownership unique in the Git registry - -**Files:** - -- Modify: `backend/src/workspaces/registry.ts` -- Modify: `backend/src/workspaces/migrate-legacy.ts` -- Modify: `backend/test/workspace-registry.test.ts` -- Modify: `backend/test/workspaces-migrate-legacy.test.ts` - -**Step 1: Write failing registry tests** - -Add fixtures with two schema-v3 workspaces claiming `collection: shared`. Assert snapshot activation -fails with `workspace_invalid` and retains the previous active snapshot. Assert v1/v2 entries are -listed as `migration_required` and cannot be acquired with `acquireSessionRevision()`. - -**Step 2: Verify RED** - -```bash -cd backend -npx vitest run test/workspace-registry.test.ts test/workspaces-migrate-legacy.test.ts \ - -t "collection|migration_required" -``` - -Expected: duplicate collections are currently accepted and v2 is currently operational. - -**Step 3: Implement uniqueness and migration state** - -During snapshot validation, build `Map` for operational descriptors and -raise a sanitized `workspace_invalid` error on a duplicate. Update migration output and CLI wording -to require an explicit target collection and schema v3. - -**Step 4: Verify GREEN and commit** - -```bash -cd backend -npx vitest run test/workspace-registry.test.ts test/workspaces-migrate-legacy.test.ts \ - -t "collection|migration_required" -cd .. -git add backend/src/workspaces/registry.ts backend/src/workspaces/migrate-legacy.ts \ - backend/test/workspace-registry.test.ts backend/test/workspaces-migrate-legacy.test.ts -git commit -m "feat: reserve one qdrant collection per workspace" -``` - -### Task 3: Remove external semantic bindings and render internal endpoints - -**Files:** - -- Modify: `backend/src/workspaces/contracts.ts` -- Modify: `backend/src/workspaces/bindings.ts` -- Modify: `backend/src/workspaces/runtime-renderer.ts` -- Modify: `backend/src/config.ts` -- Modify: `backend/test/workspaces-contracts.test.ts` -- Modify: `backend/test/workspaces-bindings.test.ts` -- Modify: `backend/test/workspace-runtime-renderer.test.ts` -- Modify: `backend/test/config.test.ts` - -**Step 1: Write failing contract tests** - -Assert schema-v3 installation contracts contain DWH variables only. Assert environment variables -matching `*_VECTOR_*`, `*_EMBEDDING_BASE_URL`, or semantic API-key suffixes are ignored/rejected. -Assert the rendered harness config always contains: - -```yaml -resources: - vector: - engine: qdrant - base_url: http://qdrant:6333 - collection: psd-clinical - embeddings: - provider: ollama_internal - base_url: http://embedding:11434 - model: qwen3-embedding:0.6b - dimensions: 1024 -``` - -**Step 2: Verify RED** - -```bash -cd backend -npx vitest run test/workspaces-contracts.test.ts test/workspaces-bindings.test.ts \ - test/workspace-runtime-renderer.test.ts test/config.test.ts -``` - -Expected: current contracts require external vector and embedding bindings. - -**Step 3: Implement internal runtime configuration** - -Add typed backend config fields with Compose defaults: - -```ts -internalQdrantUrl: "http://qdrant:6333" -internalEmbeddingUrl: "http://embedding:11434" -internalEmbeddingModel: "qwen3-embedding:0.6b" -internalEmbeddingDimensions: 1024 -``` - -Accept only `qdrant`, `embedding`, `localhost`, or loopback hosts. Keep these values out of Git -workspace descriptors, API payloads, and generated installation docs. Render them into the -ephemeral backend-owned harness config after descriptor validation. - -**Step 4: Verify GREEN and commit** - -Run Step 2, then: - -```bash -git add backend/src/config.ts backend/src/workspaces/contracts.ts backend/src/workspaces/bindings.ts \ - backend/src/workspaces/runtime-renderer.ts backend/test/config.test.ts \ - backend/test/workspaces-contracts.test.ts backend/test/workspaces-bindings.test.ts \ - backend/test/workspace-runtime-renderer.test.ts -git commit -m "feat: render private semantic service endpoints" -``` - -### Task 4: Narrow harness embedding configuration to internal Ollama - -**Files:** - -- Modify: `harness/tht/config.py` -- Modify: `harness/tht/config_compat.py` -- Modify: `harness/tht/vectorstore/embeddings.py` -- Modify: `harness/tht/cli/ollama_cmd.py` -- Modify: `harness/tests/test_config_resources.py` -- Create: `harness/tests/test_internal_embeddings.py` - -**Step 1: Write failing embedding tests** - -Use a fake `requests.Session` to prove `OllamaInternalEmbeddings.embed()` calls `/api/embed` with -model and batch input, returns 1024-dimensional finite vectors, and rejects count/dimension/NaN -mismatches. Add config tests rejecting external providers, API keys, and non-private base URLs. - -**Step 2: Verify RED** - -```bash -cd harness -.venv/bin/pytest tests/test_internal_embeddings.py tests/test_config_resources.py -q -``` - -Expected: `OllamaInternalEmbeddings` and internal-only config do not exist. - -**Step 3: Implement the client** - -Implement one bounded `/api/embed` request per batch: - -```python -response = self._session.post( - f"{self.base_url}/api/embed", - json={"model": self.model, "input": texts}, - timeout=self.timeout, -) -``` - -Validate response shape before returning any vector. Keep retry behavior bounded and sanitize URLs -and response bodies from raised errors. - -**Step 4: Verify GREEN and commit** - -```bash -cd harness -.venv/bin/pytest tests/test_internal_embeddings.py tests/test_config_resources.py -q -cd .. -git add harness/tht/config.py harness/tht/config_compat.py harness/tht/vectorstore/embeddings.py \ - harness/tht/cli/ollama_cmd.py harness/tests/test_config_resources.py \ - harness/tests/test_internal_embeddings.py -git commit -m "feat: use internal ollama embeddings" -``` - -### Task 5: Implement the Qdrant VectorStore adapter - -**Files:** - -- Create: `harness/tht/adapters/vector/qdrant.py` -- Modify: `harness/tht/adapters/vector/__init__.py` -- Modify: `harness/tht/ports/vector.py` -- Modify: `harness/tht/vectorstore/records.py` -- Modify: `harness/tht/vectorstore/store.py` -- Create: `harness/tests/test_qdrant_vector_store.py` -- Modify: `harness/tests/test_vector_port_contract.py` - -**Step 1: Write failing adapter tests** - -Test a real adapter against a deterministic fake HTTP server. Cover: - -- idempotent collection create with 1024/Cosine; -- mismatch fails without delete/recreate; -- keyword payload-index creation; -- deterministic UUIDv5 point IDs; -- upsert payload for `schema`, `evidence`, and `memory`; -- query filtered by workspace and allowed kinds; -- `existing_hashes`, exact Evidence generation list/delete, and health; -- sanitized timeouts and malformed responses. - -The point ID helper must satisfy: - -```python -def point_id(workspace_id: str, kind: str, record_key: str) -> str: - return str(uuid5(NAMESPACE_URL, f"thothii:{workspace_id}:{kind}:{record_key}")) -``` - -**Step 2: Verify RED** - -```bash -cd harness -.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -q -``` - -Expected: import failure for the Qdrant adapter. - -**Step 3: Implement minimal REST mappings** - -Use existing `requests` dependency and these endpoints: - -```text -GET /collections/{collection} -PUT /collections/{collection} -PUT /collections/{collection}/index -PUT /collections/{collection}/points?wait=true -POST /collections/{collection}/points/query -POST /collections/{collection}/points/scroll -POST /collections/{collection}/points/delete?wait=true -``` - -Every operation must include the workspace filter even though the collection is workspace-owned. -Map Qdrant scores and payloads back into existing `VectorHit` objects. - -**Step 4: Verify GREEN and commit** - -```bash -cd harness -.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py -q -cd .. -git add harness/tht/adapters/vector/qdrant.py harness/tht/adapters/vector/__init__.py \ - harness/tht/ports/vector.py harness/tht/vectorstore/records.py \ - harness/tht/vectorstore/store.py harness/tests/test_qdrant_vector_store.py \ - harness/tests/test_vector_port_contract.py -git commit -m "feat: add qdrant vector adapter" -``` - -### Task 6: Wire schema, Evidence, and Memory through Qdrant - -**Files:** - -- Modify: `harness/tht/vectorstore/reader.py` -- Modify: `harness/tht/cli/vector_cmd.py` -- Modify: `harness/tht/cli/memory_cmd.py` -- Modify: `harness/tht/corpus/pipeline.py` -- Modify: `harness/tht/search/evidence.py` -- Modify: `harness/tht/cli/schema_cmd.py` -- Modify: `harness/tests/test_memory_save_one.py` -- Modify: `harness/tests/test_search_pack.py` -- Create: `harness/tests/test_semantic_kind_isolation.py` - -**Step 1: Write failing integration tests** - -Use an in-memory fake implementing the `VectorStore` port. Assert: - -- schema records use `kind=schema`; -- corpus records use `kind=evidence` and exact generation; -- approved memories use `kind=memory`; -- search pack requests only its allowed kind set; -- all three paths share `workspace_id`, `workspace_revision`, hashing, and point-key construction; -- retries do not duplicate points. - -**Step 2: Verify RED** - -```bash -cd harness -.venv/bin/pytest tests/test_semantic_kind_isolation.py tests/test_memory_save_one.py \ - tests/test_search_pack.py -q -``` - -Expected: current factories select pgvector/HTTP adapters and payloads lack the v3 identity fields. - -**Step 3: Wire the adapter** - -Make schema-v3 `qdrant` the only operational vector factory branch. Reuse the current canonical -record builders; add only missing identity fields. Keep the JSONL Memory registry and filesystem -Evidence corpus as sources of truth. - -**Step 4: Verify GREEN and commit** - -Run Step 2, then commit the listed files with: - -```bash -git commit -m "feat: index semantic records in qdrant" -``` - -### Task 7: Add mandatory Qdrant and Ollama Compose services - -**Files:** - -- Modify: `compose.yaml` -- Create: `deploy/compose.embedding-gpu.yaml` -- Create: `docker/embedding-model-init.sh` -- Modify: `docker/core.Dockerfile` -- Modify: `deploy/env/local.env.example` -- Modify: `deploy/env/server.env.example` -- Modify: `scripts/run-stack.sh` -- Modify: `scripts/test-default-compose.sh` -- Modify: `scripts/test-unified-compose.sh` -- Create: `scripts/test-internal-semantic-compose.sh` - -**Step 1: Write failing Compose contract tests** - -Assert the rendered base profile has `core`, `frontend`, `qdrant`, `embedding`, and -`embedding-model-init`; private services have no published ports; persistent volumes exist; core -depends on Qdrant health and successful model init; no external vector/embedding binding is required. - -Also assert all service images use version plus immutable digest. Resolve and record supported -multi-architecture digests for Qdrant v1.18.x and Ollama v0.32.x during implementation: - -```bash -docker buildx imagetools inspect qdrant/qdrant:v1.18.2 -docker buildx imagetools inspect ollama/ollama:0.32.0 -``` - -**Step 2: Verify RED** - -```bash -./scripts/test-default-compose.sh -./scripts/test-unified-compose.sh -./scripts/test-internal-semantic-compose.sh -``` - -Expected: required services and volumes are absent. - -**Step 3: Implement the services** - -`embedding-model-init.sh` must wait with a bounded deadline, call `ollama pull` for the exact model, -and verify it appears in `/api/tags`. The Qdrant healthcheck uses its HTTP health endpoint. The CPU -base has no device reservation; the GPU override adds only the supported device stanza. - -**Step 4: Verify GREEN and commit** - -Run Step 2, then: - -```bash -git add compose.yaml deploy/compose.embedding-gpu.yaml docker/embedding-model-init.sh \ - docker/core.Dockerfile deploy/env/local.env.example deploy/env/server.env.example \ - scripts/run-stack.sh scripts/test-default-compose.sh scripts/test-unified-compose.sh \ - scripts/test-internal-semantic-compose.sh -git commit -m "feat: run qdrant and ollama inside thothii" -``` - -### Task 8: Retire pgvector deployment and external semantic connectors - -**Files:** - -- Delete: `deploy/compose.local-vector.yaml` -- Delete: `deploy/compose.preprocess-local-vector.yaml` -- Delete: `deploy/sql/20-vector-roles.sql` -- Delete: `deploy/vector/reconcile-roles.sh` -- Delete: `deploy/vector/rotate-bootstrap-password.py` -- Delete: `deploy/vector/secret-policy.sh` -- Delete: `deploy/vector/vector-db-entrypoint.sh` -- Delete: `scripts/local-vector-smoke.sh` -- Delete: `scripts/test-local-vector-smoke-safety.sh` -- Delete: `scripts/test-local-vector-smoke-live-collision.sh` -- Delete: `scripts/test-vector-bootstrap-rotation.sh` -- Delete: `scripts/test-vector-migration-image.sh` -- Delete: `scripts/test-vector-secret-policy.sh` -- Modify: `scripts/test-no-deployment-coupling.sh` -- Modify: `scripts/test-no-deployment-coupling-scope.sh` -- Modify: `scripts/test-compose-secret-policy.sh` -- Modify: `.github/workflows/deployment.yml` - -**Step 1: Write the failing coupling test** - -Teach the coupling gate to reject active `pgvector`, `local-vector`, `THT_VECTOR_*`, workspace -embedding URLs/API keys, and external vector transports while allowing historical specs and the -explicit descriptor migration module. - -**Step 2: Verify RED** - -```bash -./scripts/test-no-deployment-coupling-scope.sh -./scripts/test-no-deployment-coupling.sh -./scripts/test-compose-secret-policy.sh -``` - -Expected: active pgvector deployment paths are reported. - -**Step 3: Remove the retired paths and update CI** - -Remove only repository deployment machinery. Retain harness pgvector code temporarily only if it -is needed to read/export legacy data during migration; it must not be reachable from schema v3 or -Compose. Remove it in a follow-up task once migration fixtures no longer import it. - -**Step 4: Verify GREEN and commit** - -Run Step 2 and the workflow fixture tests, then commit all deletions and modifications: - -```bash -git add -A deploy scripts .github/workflows/deployment.yml -git commit -m "refactor: retire external vector deployment" -``` - -### Task 9: Update frontend workspace editing and examples - -**Files:** - -- Modify: `frontend/src/api/workspaces.ts` -- Modify: `frontend/src/shell/WorkspaceEditor.tsx` -- Modify: `frontend/src/shell/WorkspaceEditor.test.tsx` -- Modify: `frontend/src/shell/WorkspaceManager.test.tsx` -- Modify: `frontend/src/api/workspaces.test.ts` -- Modify: `frontend/src/workspaces/drafts.test.ts` -- Modify: `deploy/workspaces/example.yaml` -- Modify: `deploy/workspaces/psd.yaml.example` - -**Step 1: Write failing UI tests** - -Assert editor/preview show Qdrant collection and fixed internal embedding model, expose no vector -endpoint/credential fields, and publish schema v3. Assert legacy descriptors display a migration -banner and cannot be selected for a new session. - -**Step 2: Verify RED** - -```bash -cd frontend -npx vitest run src/shell/WorkspaceEditor.test.tsx src/shell/WorkspaceManager.test.tsx \ - src/api/workspaces.test.ts src/workspaces/drafts.test.ts -``` - -Expected: fixtures and controls still use pgvector/external embedding. - -**Step 3: Implement fixed semantic controls** - -Collection remains editable and validated. Engine, provider, model, dimensions, and distance render -as fixed architecture values. Remove external semantic diagnostics from drafts and publish payloads. - -**Step 4: Verify GREEN and commit** - -Run Step 2, then commit the listed files with: - -```bash -git commit -m "feat: edit qdrant workspace collections" -``` - -### Task 10: Add a real internal semantic smoke - -**Files:** - -- Create: `scripts/internal-semantic-smoke.sh` -- Modify: `scripts/unified-deployment-smoke.sh` -- Modify: `scripts/server-deployment-smoke.sh` -- Modify: `scripts/task13-runtime-fixture-check.ts` -- Modify: `scripts/test-task13-runtime-fixtures.sh` - -**Step 1: Write failing smoke fixture assertions** - -The fixture must require private Qdrant/Ollama services, model volume, Qdrant volume, fixed internal -URLs, and no host ports. It must reject wrong service names, external URLs, collection reuse, and -dimension changes. - -**Step 2: Verify RED** - -```bash -./scripts/test-task13-runtime-fixtures.sh local -./scripts/test-task13-runtime-fixtures.sh server -``` - -Expected: current fixture expects the two-service topology. - -**Step 3: Implement the live smoke** - -Using disposable volumes and a fixture workspace, start the stack on CPU, wait for the model, ensure -the collection, embed one record of each kind, query each kind with filters, restart offline, and -prove all points and the model remain available. Cleanup must remain exact and must not prune global -Docker resources. - -**Step 4: Verify GREEN and commit** - -```bash -./scripts/test-task13-runtime-fixtures.sh local -./scripts/test-task13-runtime-fixtures.sh server -./scripts/internal-semantic-smoke.sh -git add scripts/internal-semantic-smoke.sh scripts/unified-deployment-smoke.sh \ - scripts/server-deployment-smoke.sh scripts/task13-runtime-fixture-check.ts \ - scripts/test-task13-runtime-fixtures.sh -git commit -m "test: cover internal semantic services" -``` - -### Task 11: Update operator documentation and state - -**Files:** - -- Modify: `README.md` -- Modify: `AGENTS.md` -- Modify: `PROJECT_STATE.md` -- Modify: `docs/install/local-workspace-registry.md` -- Modify: `docs/install/server-workspace-registry.md` -- Modify: `docs/installazione-docker-4-contesti.md` -- Modify: `docs/workspace-diagnostic-protocol.md` -- Modify: `docs/gestione-memory.md` -- Modify: `deploy/secrets/README.md` -- Modify: `scripts/verify-workspace-install-docs.sh` -- Modify: `scripts/test-verify-workspace-install-docs.sh` - -**Step 1: Write failing documentation contract assertions** - -Require the four-service topology, CPU/GPU behavior, volume backup/restore, schema-v3 migration, -Qdrant collection ownership, and removal of external vector/embedding variables from active manuals. - -**Step 2: Verify RED** - -```bash -./scripts/test-verify-workspace-install-docs.sh -./scripts/verify-workspace-install-docs.sh --fixtures-only -``` - -Expected: manuals still describe external pgvector/embedding and a two-service mandatory stack. - -**Step 3: Update documentation** - -Document Qdrant as a derived but persistent index, Ollama model cache behavior, CPU-first startup, -optional GPU override, snapshot/restore, explicit legacy migration, and the fact that only the DWH -and LLM remain external application endpoints. - -**Step 4: Verify GREEN and commit** - -Run Step 2, then: - -```bash -git add README.md AGENTS.md PROJECT_STATE.md docs deploy/secrets/README.md \ - scripts/verify-workspace-install-docs.sh scripts/test-verify-workspace-install-docs.sh -git commit -m "docs: document internal semantic infrastructure" -``` - -### Task 12: Remove unreachable pgvector runtime code - -**Files:** - -- Delete: `harness/tht/adapters/vector/pgvector.py` -- Delete: `harness/tht/adapters/vector/legacy_direct.py` -- Delete: `harness/tht/adapters/vector/thoth_http.py` -- Delete: `harness/tht/vectorstore/rest_client.py` -- Delete: `harness/tht/vectorstore/rest_writer.py` -- Delete: `harness/tht/migrations/vector/001_extensions.sql` -- Delete: `harness/tht/migrations/vector/002_schema_tables.sql` -- Delete: `harness/tht/migrations/vector/003_roles.sql` -- Delete: `harness/tht/migrations/vector/004_evidence_generation_gc.sql` -- Modify: `harness/pyproject.toml` -- Modify/Delete: affected pgvector and migration tests under `harness/tests/l0/` - -**Step 1: Prove the code is unreachable** - -```bash -rg -n "PgVectorStore|ThothHttpVectorStore|LegacyDirectVectorStore|migrations/vector" \ - harness backend frontend compose.yaml deploy scripts docker docs \ - --glob '!docs/plans/**' --glob '!docs/superpowers/**' -``` - -Expected before cleanup: matches only in the files scheduled for deletion and legacy tests. If an -operational call site remains, stop and migrate it before deleting anything. - -**Step 2: Delete obsolete runtime and tests** - -Retain descriptor migration tests, but remove PostgreSQL vector runtime/migration packaging tests. -Remove `psycopg2-binary` only if the DWH/session PostgreSQL paths do not need it; otherwise keep it. - -**Step 3: Verify focused imports and packaging** - -```bash -cd harness -.venv/bin/pytest tests/test_qdrant_vector_store.py tests/test_vector_port_contract.py \ - tests/test_semantic_kind_isolation.py tests/test_vector_migration_packaging.py -q -python -m build -``` - -Expected: Qdrant tests pass and the wheel contains no pgvector migrations. Adjust the packaging test -to assert Qdrant has no SQL migration payload. - -**Step 4: Commit** - -```bash -git add -A harness -git commit -m "refactor: remove pgvector runtime" -``` - -### Task 13: Run complete verification - -**Files:** - -- Modify only if a genuine regression is discovered. - -**Step 1: Deterministic layer gates** - -```bash -cd harness && .venv/bin/pytest -q && .venv/bin/ruff check . -cd ../backend && npx vitest run && npx tsc --noEmit -p . && npm run build -cd ../frontend && npx vitest run && npx tsc -b && npm run build -cd .. && git diff --check -``` - -Expected: all gates pass. Existing unrelated Ruff debt must be reported separately if it remains; -new/modified files must be Ruff-clean. - -**Step 2: Deployment contracts** - -```bash -./scripts/test-default-compose.sh -./scripts/test-unified-compose.sh -./scripts/test-internal-semantic-compose.sh -./scripts/test-no-deployment-coupling.sh -./scripts/test-compose-secret-policy.sh -./scripts/verify-workspace-install-docs.sh --fixtures-only -``` - -Expected: all pass without external vector/embedding settings. - -**Step 3: Docker smokes** - -```bash -./scripts/internal-semantic-smoke.sh -./scripts/workspace-registry-smoke.sh -./scripts/unified-deployment-smoke.sh -./scripts/thothctl-update-smoke.sh -./scripts/server-deployment-smoke.sh -``` - -Expected: CPU semantic smoke passes, persistence survives offline restart, and every script proves -exact cleanup. Investigate the previously observed `thothctl` rollback failure independently if it -recurs; do not weaken the new semantic gate to hide it. - -**Step 4: Final audit** - -```bash -rg -n "pgvector|local-vector|THT_VECTOR_|EMBEDDING_BASE_URL|openai_compatible|ollama_compatible" \ - . --glob '!docs/plans/**' --glob '!docs/superpowers/**' --glob '!**/node_modules/**' \ - --glob '!**/.venv/**' --glob '!**/.git/**' -git status --short -``` - -Expected: no active operational references; only explicit legacy descriptor migration fixtures may -remain. Worktree contains only intentional changes. - -**Step 5: Commit verification metadata** - -Update `PROJECT_STATE.md` with exact counts, image digests, smoke durations, CPU hardware, and any -manual GPU/Windows gates. Commit only verified claims: - -```bash -git add PROJECT_STATE.md -git commit -m "docs: record qdrant ollama verification" -``` diff --git a/docs/plans/2026-08-10-rimozione-schema-v1-v2.md b/docs/plans/2026-08-10-rimozione-schema-v1-v2.md deleted file mode 100644 index 81ffa54d..00000000 --- a/docs/plans/2026-08-10-rimozione-schema-v1-v2.md +++ /dev/null @@ -1,447 +0,0 @@ -# Piano di implementazione: workspace descriptor esclusivamente schema v3 - -> **Per gli agenti esecutori:** SUB-SKILL OBBLIGATORIA: usare `superpowers:subagent-driven-development` (raccomandata) oppure `superpowers:executing-plans`, procedendo task per task con TDD e review tra i task. - -**Obiettivo:** rimuovere dal prodotto ogni capacità di leggere, migrare, rendere operativo o presentare workspace descriptor schema v1/v2. Il solo descriptor accettato diventa schema v3. Restano intatti i formati versionati non correlati e gli state file del registry già prodotti da versioni recenti con revisioni v3. - -**Architettura:** parser, registry, renderer, diagnostica, route e frontend convergono su un solo tipo `WorkspaceV3`. Il campo pubblico `WorkspaceRevision.state` scompare. Un decoder privato normalizza in memoria gli state file già scritti con `state: "operational"`, elimina quel campo prima di qualsiasi uso/API e rifiuta ogni combinazione non-v3 o incoerente. I build backend diventano clean-first, così la cancellazione dei migratori sorgente implica anche la loro assenza da `dist` e dall'immagine core. - -**Tech stack:** TypeScript 5, Zod 4, Fastify 5, React 18, Vitest, Node.js 22, Bash/PowerShell, Git e Docker Compose. - -**Stato:** piano revisionato dopo review indipendente. La sua approvazione non autorizza l'implementazione; attendere un esplicito ordine separato. - ---- - -## Decisioni confermate - -1. Nessun workspace v1/v2 reale deve essere preservato o migrato. -2. Eliminare `migrate-legacy.ts`, `migrate-v2-qdrant.ts` e le relative interfacce CLI. -3. Eliminare il campo `state` dal tipo/API `WorkspaceRevision` e da tutti i nuovi state/manifest del registry. -4. Descriptor v1/v2 presenti in Git o negli snapshot vengono rifiutati, senza conversione automatica. -5. Non toccare i documenti storici sotto `docs/superpowers/` e i vecchi piani; possono descrivere decisioni passate. -6. Non iniziare P2 finché P1 non dispone di nuova evidenza automatica e di una nuova decisione manuale esplicita. - -## Confini da non oltrepassare - -Questa rimozione riguarda soltanto il **workspace descriptor**. Non eliminare o rinominare: - -- `schemaVersion`/`schema_version` di bundle ZIP, report, job, ledger, manifest di sessione o artifact di fase; -- `RevisionLeaseRecord.state` (`creating`/`persisted`), maintenance state, process state o UI state non collegati a `WorkspaceRevision`; -- `migration_required` usato nei futuri piani P3–P6 per ownership DWH, punti semantici revisionless o altre migrazioni non-descriptor; -- `allowLegacy` del frontend sessioni, che significa “sessione senza revisione workspace” e non descriptor v1/v2; -- documenti storici o report conservati. - -L'unica compatibilità legacy mantenuta nel codice è il decoder privato degli state file già scritti con il campo revisionale `state: "operational"`. Non costituisce supporto a descriptor v1/v2. - -## Contratto v3-only - -- `WorkspaceDescriptor`, `CanonicalWorkspace` e `WorkspaceV3` rappresentano la stessa forma v3; mantenere gli alias soltanto quando migliorano la semantica dei confini. -- `parseWorkspaceYaml` e `validateWorkspaceDescriptor` accettano esclusivamente `workspace.schema_version === 3`. -- v1/v2 generano l'errore pubblico già sanitizzato `workspace_invalid`; non usare più il messaggio o lo stato `migration_required` per i descriptor. -- Un'attivazione Git contenente anche un solo descriptor non-v3 fallisce interamente e conserva il precedente active state. -- Le revisioni restituite dalle API contengono esattamente `id`, `commit`, `blob`, `snapshotPath`, senza `state`. -- Nuovi `active.json` e `snapshot.json` non contengono `state` nelle revisioni. - -## Compatibilità degli state file esistenti - -Definire due decoder stretti e distinti: - -```ts -interface StoredWorkspaceRevision { - id: string; - commit: string; - blob: string; - snapshotPath: string; - state?: "operational"; // solo input compatibile; mai restituito -} - -interface WorkspaceRevision { - id: string; - commit: string; - blob: string; - snapshotPath: string; -} -``` - -Regole: - -1. `active.json` accetta soltanto `{head,revisions}`; `snapshot.json` soltanto `{head,revisions,files}`. -2. Ogni revision object accetta soltanto i quattro campi correnti più l'opzionale vecchio `state: "operational"`. -3. `state: "migration_required"`, qualsiasi altro valore o campo sconosciuto è rifiutato. -4. Il decoder ricostruisce un nuovo oggetto `WorkspaceRevision`; non restituisce mai l'oggetto JSON originale. -5. Active state e snapshot manifest vengono confrontati dopo la normalizzazione. -6. L'integrità continua a validare path, commit, blob, digest, descriptor v3 e Evidence context. -7. La lettura non modifica snapshot storici. La successiva attivazione riscrive `active.json` nel formato corrente; tutti i nuovi snapshot sono state-free. -8. Un vecchio file già privo di `state` è naturalmente il formato corrente, ma il relativo descriptor deve comunque essere v3. - -## Mappa completa dei file - -### Backend produttivo - -- `backend/src/workspaces/schema.ts` -- `backend/src/workspaces/types.ts` -- `backend/src/workspaces/runtime-renderer.ts` -- `backend/src/workspaces/contracts.ts` -- `backend/src/workspaces/diagnostics.ts` -- `backend/src/workspaces/bindings.ts` -- `backend/src/workspaces/registry.ts` -- `backend/src/routes/workspaces.ts` -- `backend/src/routes/sessions.ts` -- `backend/src/routes/sql.ts` -- Eliminare `backend/src/workspaces/migrate-legacy.ts` -- Eliminare `backend/src/workspaces/migrate-v2-qdrant.ts` - -### Build e tooling P1 - -- `backend/package.json` -- Creare `backend/scripts/clean-dist.mjs` -- Creare un test Node per il clean build -- `backend/scripts/p1-manual-acceptance.mjs` -- `backend/scripts/p1-manual-acceptance.test.mjs` -- `backend/scripts/p1-render-snapshot.test.mjs` - -### Frontend - -- `frontend/src/api/workspaces.ts` -- `frontend/src/api/sessions.ts` -- `frontend/src/shell/SteerInput.tsx` -- `frontend/src/shell/WorkspaceManager.tsx` -- Test/fixture in `api`, `SteerInput`, `WorkspaceManager`, `NewSessionDialog`, `WorkspacePublishDialog` e `drafts`. - -### Deploy, fixture e verificatori - -- `scripts/workspace-registry-smoke.sh` -- Creare `scripts/fixtures/workspace-registry-smoke.yaml` -- `scripts/test-no-deployment-coupling-scope.sh` -- `scripts/test-windows-clone-contract.ps1` -- `scripts/verify-workspace-install-docs.sh` -- `scripts/test-verify-workspace-install-docs.sh` - -### Documentazione corrente - -- `README.md` -- sezione corrente di `PROJECT_STATE.md`, prima di `## Historical snapshots` -- `docs/workspace-diagnostic-protocol.md` -- `docs/install/local-workspace-registry.md` -- `docs/install/server-workspace-registry.md` - ---- - -### Task 0: Congelare scope e baseline prima delle modifiche - -**File:** nessuna modifica produttiva. - -- [ ] Registrare `BASE_SHA=$(git rev-parse HEAD)` e verificare che gli altri piani non vengano inclusi nei commit di implementazione. -- [ ] Salvare l'inventario iniziale dei simboli descriptor-legacy: - -```bash -git grep -nE 'WorkspaceV1|WorkspaceV2|LegacyWorkspace|migration_required|migrate-legacy|migrateWorkspaceV1ToV2|migrateWorkspaceV2ToV3' -- \ - backend/src backend/test backend/scripts frontend/src scripts README.md PROJECT_STATE.md docs/install docs/workspace-diagnostic-protocol.md -``` - -- [ ] Classificare ogni risultato come descriptor legacy, compatibility decoder previsto, contratto diverso o documento storico. -- [ ] Verificare nei registry/installazioni disponibili che i descriptor attivi siano v3; questa è una precondizione di deploy, non un migratore. -- [ ] Non procedere se il worktree contiene modifiche applicative non attribuibili a questo piano. - -### Task 1: Scrivere i test RED del contratto v3-only - -**File:** -- `backend/test/workspaces-schema.test.ts` -- `backend/test/workspace-registry.test.ts` -- `backend/test/routes-workspaces.test.ts` - -- [ ] Aggiungere test che `parseWorkspaceYaml`, `validateWorkspaceDescriptor` e le route validate/publish rifiutino esplicitamente v1 e v2. -- [ ] Aggiungere test registry per: - - bootstrap pulito con solo v1/v2: fallimento, nessun `active.json` pubblicato; - - repository misto v3+v2: attivazione atomica rifiutata; - - pull che introduce v1/v2: precedente active state ancora leggibile; - - retained snapshot contenente descriptor non-v3: rifiuto fail-closed; - - risposta API state-free. -- [ ] Eseguire: - -```bash -cd backend -npx vitest run test/workspaces-schema.test.ts test/workspace-registry.test.ts test/routes-workspaces.test.ts -``` - -Atteso: RED per i nuovi requisiti, non errori di fixture casuali. - -### Task 2: Rendere lo schema backend esclusivamente v3 - -**File:** -- `backend/src/workspaces/schema.ts` -- `backend/src/workspaces/types.ts` -- test del Task 1 - -- [ ] Eliminare `WorkspaceV1`, `WorkspaceV2`, `LegacyWorkspace`, relativi Zod schema e `migrateWorkspaceV1ToV2`. -- [ ] Rendere `WorkspaceDescriptorSchema = WorkspaceV3Schema`. -- [ ] Eliminare `validateCanonicalWorkspace`, aggiornando **tutti** i chiamanti in `routes/workspaces.ts`, incluso il chiamante attualmente oltre quelli elencati nel vecchio piano. -- [ ] Eliminare `isCanonicalWorkspace`/`isOperationalWorkspace` dopo aver sostituito i rami condizionali con validazione v3 diretta. -- [ ] Conservare test negativi v1/v2; non cancellare le sole prove che impediscono una regressione futura. -- [ ] Eseguire test focalizzati e typecheck. -- [ ] Commit: `refactor: make workspace descriptors schema v3 only`. - -### Task 3: Normalizzare in sicurezza active state e snapshot manifest - -**File:** -- `backend/src/workspaces/registry.ts` -- `backend/test/workspace-registry.test.ts` - -- [ ] Scrivere RED per state/manifest con: - - campo assente; - - vecchio `state: "operational"`; - - `state: "migration_required"`; - - valore sconosciuto; - - campo extra; - - active state e manifest con formati misti; - - snapshot attivo, storico e fallback offline. -- [ ] Rimuovere `state` da `WorkspaceRevision` e da tutti i nuovi writer. -- [ ] Sostituire cast e vecchie migrazioni con decoder stretti che restituiscono oggetti normalizzati state-free. -- [ ] Rimuovere `LegacyWorkspaceRevision`, `LegacyActiveState`, `LegacySnapshotManifest`, `deriveStateFromLegacyRevisions`, `migrateLegacyActiveState`, `migrateLegacySnapshotManifest`, `sameLegacyRevisions` e le condizioni operative basate su `state`. -- [ ] Mantenere tutti i controlli di integrità e far validare ogni YAML come v3. -- [ ] Provare che list/read/API non riemettono il vecchio campo anche immediatamente dopo un restart, prima di una nuova attivazione. -- [ ] Commit: `refactor: remove workspace revision state`. - -### Task 4: Eliminare i rami v1/v2 da renderer, contracts, bindings e diagnostica - -**File:** -- `backend/src/workspaces/runtime-renderer.ts` -- `backend/src/workspaces/contracts.ts` -- `backend/src/workspaces/diagnostics.ts` -- `backend/src/workspaces/bindings.ts` -- relativi test - -- [ ] Scrivere/aggiornare test RED che accettano v3 e rifiutano input non-v3 al confine, senza renderer/diagnoser legacy. -- [ ] Eliminare il renderer v2/pgvector e i rami v1. -- [ ] Eliminare variabili contract e diagnostica solamente v2. -- [ ] Semplificare bindings dopo la validazione v3, senza indebolire validazione secrets/trasporti. -- [ ] Eseguire i test focalizzati: - -```bash -cd backend -npx vitest run \ - test/workspace-runtime-renderer.test.ts \ - test/workspaces-contracts.test.ts \ - test/workspaces-diagnostics.test.ts \ - test/workspaces-bindings.test.ts \ - test/workspace-runtime-handoff.test.ts -``` - -- [ ] Commit: `refactor: remove legacy workspace runtime branches`. - -### Task 5: Rimuovere migratori senza perdere test di deployment non correlati - -**File:** -- Eliminare i due migratori e i test esclusivamente di migrazione. -- Creare/spostare in un test dedicato le prove deployment presenti in `workspaces-migrate-legacy.test.ts:81-114`. - -- [ ] Prima di eliminare `workspaces-migrate-legacy.test.ts`, spostare in un file con nome coerente: - - volume registry durevole e mount Git read-only; - - contratto Dockerfile; - - fallback offline smoke; - - self-test di cleanup dell'immagine per-run. -- [ ] Eliminare `migrate-legacy.ts`, `migrate-v2-qdrant.ts` e i test di trasformazione. -- [ ] Conservare un fixture v2 soltanto nei test negativi di rifiuto. -- [ ] Eseguire i nuovi test deployment e il typecheck. -- [ ] Commit: `refactor: remove workspace migration utilities`. - -### Task 6: Aggiornare tutte le route backend e il tooling P1 - -**File:** -- `backend/src/routes/workspaces.ts` -- `backend/src/routes/sessions.ts` -- `backend/src/routes/sql.ts` -- test route inclusi `routes-sql-meta.test.ts` -- `backend/scripts/p1-manual-acceptance.mjs` -- test manual/render P1 - -- [ ] Rimuovere filtri/gate `revision.state` da tutte le route. La garanzia deriva dal registry v3-only. -- [ ] Aggiornare mock/fixture `WorkspaceRevision` in tutti i test backend. -- [ ] Aggiornare il validatore del manifest P1 manuale affinché richieda esattamente la revisione state-free. -- [ ] Aggiornare i fixture `p1-manual-acceptance.test.mjs` e `p1-render-snapshot.test.mjs`. -- [ ] Aggiungere un test JS specifico che rifiuti manifest con revisioni malformate senza reintrodurre `migration_required`. -- [ ] Eseguire: - -```bash -cd backend -npx vitest run test/routes-workspaces.test.ts test/routes-sessions.test.ts test/routes-sql-meta.test.ts -cd .. -node --test --test-concurrency=1 \ - backend/scripts/p1-manual-acceptance.test.mjs \ - backend/scripts/p1-render-snapshot.test.mjs -``` - -- [ ] Commit: `refactor: remove workspace revision state consumers`. - -### Task 7: Rendere il build backend clean-first - -**File:** -- `backend/package.json` -- Creare `backend/scripts/clean-dist.mjs` -- Creare test Node del clean build - -- [ ] Scrivere RED: creare un file sentinella in `backend/dist/workspaces/`, eseguire il clean/build e verificare che non sopravviva. -- [ ] Implementare la pulizia con API Node multipiattaforma, non con `rm -rf` nella npm script. -- [ ] Fare eseguire il clean prima di `tsc` da `npm run build`. -- [ ] Verificare dopo il build: - -```bash -test ! -e backend/dist/workspaces/migrate-legacy.js -test ! -e backend/dist/workspaces/migrate-v2-qdrant.js -``` - -- [ ] Costruire l'immagine core in un contesto pulito e verificare che i due moduli non esistano nell'immagine. -- [ ] Verificare che i manifest di integrità P1 continuino a legare l'intero nuovo `dist`. -- [ ] Commit: `build: remove stale backend distribution files`. - -### Task 8: Aggiornare frontend e contratto API state-free - -**File:** -- `frontend/src/api/workspaces.ts` -- `frontend/src/api/sessions.ts` -- `frontend/src/shell/SteerInput.tsx` -- `frontend/src/shell/WorkspaceManager.tsx` -- test/fixture frontend correlati - -- [ ] Scrivere/aggiornare test per revisioni senza `state` e risposta non-v3 rifiutata al confine workspace. -- [ ] Eliminare `state` dal tipo e dal parser revisionale. -- [ ] Rimuovere gate/banner/filtro `migration_required` e anche la visualizzazione `record.revision.state`. -- [ ] Mantenere `allowLegacy` per sessioni senza revisione. -- [ ] Aggiornare fixture in: - - `api/workspaces.test.ts`, `api/sessions.test.ts`; - - `SteerInput.test.tsx`, `WorkspaceManager.test.tsx`; - - `NewSessionDialog.test.tsx`, `WorkspacePublishDialog.test.tsx`; - - `drafts.test.ts`, mantenendo il test negativo di schema non-3. -- [ ] Documentare che core e frontend devono essere aggiornati insieme; il parser nuovo non usa più `state`. -- [ ] Eseguire typecheck e suite frontend. -- [ ] Commit: `refactor: remove legacy workspace UI state`. - -### Task 9: Sostituire fixture e smoke con descriptor v3 completi - -**File:** -- `scripts/workspace-registry-smoke.sh` -- Creare `scripts/fixtures/workspace-registry-smoke.yaml` -- `scripts/test-no-deployment-coupling-scope.sh` -- `scripts/test-windows-clone-contract.ps1` -- test deployment spostati nel Task 5 - -- [ ] Creare un descriptor v3 completo `id: local`, collection `local`, embedding interno 1024/cosine, LLM policy e diagnostica DWH; omettere Evidence per non richiedere un tree Git nello smoke registry. -- [ ] Validare il fixture con il parser produttivo in un test backend. -- [ ] Copiare il fixture nello seed repository e rimuovere sia l'invocazione del migratore sia il build backend ormai inutile allo smoke. -- [ ] Nel test Windows non cambiare soltanto il numero di versione: fornire il contratto v3 completo mantenendo lo scopo path-with-spaces/clone. -- [ ] Aggiornare il fixture dello scope coupling senza indebolire l'assenza-gate. -- [ ] Eseguire test shell focalizzati e, con Docker disponibile, lo smoke reale senza retry. -- [ ] Commit: `test: replace legacy workspace deployment fixtures`. - -### Task 10: Aggiornare documentazione corrente e relativi verifier - -**File:** -- documenti/verifier indicati nella mappa - -- [ ] Aggiornare README e soltanto la sezione corrente di `PROJECT_STATE.md`; non riscrivere gli snapshot storici. -- [ ] Eliminare procedure di migrazione v1/v2 dai manuali local/server e dal protocollo diagnostico. -- [ ] Modificare `verify-workspace-install-docs.sh` perché richieda “schema v3 only” e l'assenza di `migration_required` nella documentazione corrente. -- [ ] Aggiornare i fixture negativi del test del verifier. -- [ ] Non cambiare gli usi di `migration_required` nei piani P3–P6 relativi a ownership/artifact diversi. -- [ ] Eseguire: - -```bash -bash scripts/test-verify-workspace-install-docs.sh -bash scripts/verify-workspace-install-docs.sh --fixtures-only -``` - -- [ ] Commit: `docs: make schema v3 the only workspace contract`. - -### Task 11: Eseguire absence gate e suite complete - -- [ ] Eseguire backend clean build, typecheck e test: - -```bash -cd backend -npm run build -npx tsc --noEmit -p . -npx vitest run -``` - -- [ ] Eseguire frontend: - -```bash -cd frontend -npx tsc -b -npx vitest run -npm run build -``` - -- [ ] Eseguire script/verifier interessati, incluso lo smoke Docker obbligatorio se l'ambiente dispone di Docker. Non lasciarlo “opzionale” in una consegna che modifica lo smoke. -- [ ] Eseguire `git diff --check`. -- [ ] Eseguire l'absence gate ristretto: - -```bash -git grep -nE 'WorkspaceV1|WorkspaceV2|LegacyWorkspace|migrateWorkspaceV1ToV2|migrateWorkspaceV2ToV3' -- \ - backend/src frontend/src scripts && exit 1 || true - -git grep -nE 'migration_required|migrate-legacy|migrate-v2-qdrant' -- \ - backend/src backend/scripts frontend/src scripts README.md docs/install docs/workspace-diagnostic-protocol.md && exit 1 || true - -test ! -e backend/dist/workspaces/migrate-legacy.js -test ! -e backend/dist/workspaces/migrate-v2-qdrant.js -``` - -Nota: trasformare questi esempi in uno script con allowlist esplicita; non affidarsi a `&& exit 1 || true`, che può mascherare errori di esecuzione. Lo script deve distinguere “nessun match” da errore Git/I/O. - -- [ ] Ispezionare il diff per assicurarsi che nessun formato non-descriptor sia stato modificato. - -### Task 12: Rigenerare l'evidenza automatica P1 - -- [ ] Partire dal commit sorgente finale pulito. -- [ ] Eseguire una sola integrazione completa, senza retry automatico: - -```bash -./scripts/p1-acceptance.sh integration --keep -``` - -- [ ] Verificare report JSON/Markdown, hash dichiarati, manifest sorgente/dist, secret scan, ownership cleanup e porte chiuse. -- [ ] Aggiornare `PROJECT_STATE.md` con il nuovo commit/tree/report e con stati distinti: - -```text -automated integration: PASS -manual acceptance: PENDING -``` - -- [ ] Committare soltanto lo stato tracciato, mai `.artifacts`. -- [ ] Non riusare l'evidenza precedente legata a `c733896`. - -### Task 13: Riaprire e chiudere il gate manuale P1 - -- [ ] Preparare un ambiente manuale nuovo: - -```bash -./scripts/p1-manual-acceptance.sh prepare -./scripts/p1-manual-acceptance.sh serve -``` - -- [ ] Il reviewer segue integralmente il nuovo `GUIDE.md`, verificando anche che revisioni/API/manifest siano state-free e che v1/v2 siano rifiutati senza mutazione. -- [ ] Arrestare il server e verificare porte/processi: - -```bash -./scripts/p1-manual-acceptance.sh stop -``` - -- [ ] Solo il reviewer crea `VERDICT.md` e decide PASS/FAIL. -- [ ] Se PASS, aggiornare `PROJECT_STATE.md` e committare `docs: record schema-v3-only P1 acceptance`. -- [ ] Pulire il lab soltanto dopo conferma del reviewer. -- [ ] **STOP:** non iniziare P2 finché il reviewer non approva esplicitamente il nuovo P1. - ---- - -## Criteri finali di accettazione - -1. Nessun descriptor v1/v2 viene parsato, pubblicato, attivato, renderizzato, diagnosticato o mostrato. -2. I vecchi state file di revisioni v3 con `state: "operational"` continuano a caricarsi, ma API e nuovi file sono state-free. -3. Descriptor non-v3 o state incoerenti falliscono senza sostituire il precedente active state. -4. Nessun migratore sopravvive in sorgenti, `dist`, immagine core, script o documentazione corrente. -5. I formati versionati non collegati ai workspace descriptor sono invariati. -6. Backend, frontend, verifier, smoke e build interessati sono verdi. -7. Una nuova integrazione P1 è PASS al commit finale. -8. La nuova acceptance manuale P1 è decisa esplicitamente dal reviewer. -9. P2 resta non iniziato fino a ulteriore autorizzazione. diff --git a/docs/plans/2026-08-14-read-only-workspace-runtime-secrets-design.md b/docs/plans/2026-08-14-read-only-workspace-runtime-secrets-design.md deleted file mode 100644 index 1e7083bd..00000000 --- a/docs/plans/2026-08-14-read-only-workspace-runtime-secrets-design.md +++ /dev/null @@ -1,189 +0,0 @@ -# Read-only Workspace Repository and Runtime Secrets Design - -**Date:** 2026-08-14 -**Status:** Approved - -## Purpose - -ThothII consumes workspaces from one administrator-configured Git repository. Workspace authors -prepare and publish source outside ThothII. The application fetches, validates, and activates -repository revisions, but never edits, commits, pushes, imports, or exports workspace source. - -Runtime credentials are intentionally absent from Git. After a workspace has been read, ThothII -derives the required credentials from its connector and authentication choices and lets an -authorized user complete them in the web application. The values are encrypted and persisted by -the backend; the browser retains neither workspace content nor secrets. - -## Ownership boundaries - -### Workspace source - -The workspace source is an ordinary directory maintained outside the ThothII runtime. It contains -the catalog, each `workspace.yaml`, curated evidence, annotations, and other repository-owned -content. Authors validate it using source-side tooling and publish it through their normal Git -workflow to GitHub, GitLab, Gitea, or another standards-compatible server. - -### ThothII installation - -The installation descriptor selects the Git remote, branch, and one read-only authentication -transport. SSH uses a read-only deploy key plus pinned known hosts. HTTPS uses a read-only deploy -token and may provide a private CA. Secret values remain outside versioned configuration. - -The installer performs a sanitized `git ls-remote` preflight. Credentials embedded in a remote URL -are rejected. The API exposes only a normalized repository identity: host, repository path, branch, -transport, active commit, and synchronization state. - -### ThothII runtime - -The local Git checkout, candidate validation area, immutable snapshots, and active state are -application-owned. They are read-only from the workspace-management API. A pull fetches a candidate -revision, validates the complete repository, and atomically activates it only if valid. A failed -candidate never replaces the last valid active revision. - -ThothII never generates or reconciles files back into the checkout and never invokes Git commit or -push. Generated operational artifacts live under application data, not in the source repository. - -## Repository synchronization states - -A repository refresh has these states: - -- `syncing`: fetching and validating a candidate revision; -- `active`: the candidate passed validation and became the active immutable revision; -- `invalid_candidate`: Git succeeded but repository validation failed; the previous revision stays active; -- `unavailable`: Git or authentication failed; the previous revision stays active; -- `empty`: no valid revision has ever been activated. - -Validation is atomic at repository-commit level. A malformed catalog, descriptor, evidence tree, or -cross-file reference rejects the complete candidate revision. - -## Runtime secret model - -### Requirement discovery - -The workspace descriptor contains connector type, authentication method, and non-secret logical -configuration. It never contains secret values or host filesystem paths. Connector adapters define -the secret fields required by each supported authentication method. For example: - -- PostgreSQL `username_password` requires `username` and `password`; -- REST `bearer` requires `api_key`; -- SSH tunnel authentication requires the connector password and SSH private key; -- Evidence HTTP signed URLs and static S3 credentials contribute their own secret requirements. - -Requirements have stable identifiers scoped by workspace and connector. Labels, descriptions, -input kinds, and required/optional status come from trusted application code rather than repository -HTML or executable metadata. - -### Persistent encrypted store - -The backend owns a `WorkspaceSecretStore` abstraction. The first implementation is a local encrypted -vault in application-managed persistent storage. Each secret is encrypted with authenticated -encryption and bound to its installation, workspace, connector, and field identifier as associated -data. Plaintext values never appear in Git, API responses, logs, error messages, diagnostics, or -browser storage. - -The installation bootstraps one vault key independently from workspace content. Deployment tooling -owns its platform-specific provisioning; the workspace schema and GUI never contain filesystem -paths. The storage interface allows a future Vault, cloud secret manager, or OS keychain provider -without changing workspace descriptors or API consumers. - -When an existing file-oriented harness connector needs a credential, the backend materializes it as -a restrictive temporary file in an application-owned runtime directory. Its lifetime is tied to the -diagnostic or runtime lease and it is removed on release. Persistent storage contains ciphertext -only. - -### Secret API - -For a selected workspace the API returns requirement metadata and status only: - -```json -{ - "workspaceId": "psd-clinical", - "state": "configuration_required", - "requirements": [ - { - "id": "dwh.password", - "connector": "dwh", - "label": "Database password", - "input": "password", - "required": true, - "configured": false - } - ] -} -``` - -A write request contains values only for the selected requirement identifiers. The response returns -status, never values. A delete operation forgets a configured value. Authorization is deliberately -deferred; the current authenticated application user may manage runtime workspace secrets. - -Workspace readiness is derived as follows: - -- `invalid`: repository structure or descriptor is invalid; -- `configuration_required`: structurally valid but required runtime values are missing; -- `ready`: required values exist but connectivity has not yet passed or is stale; -- `verified`: the most recent connector diagnostic passed for the active revision and current secret generation. - -Changing or deleting a secret invalidates the previous diagnostic result. - -## Browser behavior - -Workspace management is a two-level read-only interface occupying at least 60 percent of viewport -width and height. - -Level 1 explains the source/runtime separation and displays: - -- normalized repository host and path; -- configured branch and read-only transport; -- active revision and last synchronization result; -- `Update workspace repository`, which fetches, validates, and conditionally activates a revision; -- the workspace list, with selection required for workspace-specific actions. - -There is no Import bundle, Export bundle, Create, Edit, Delete, Publish, or conflict-resolution -operation. There are no browser-persisted workspace drafts or preferences. - -Level 2 for the selected workspace explains and displays: - -- immutable source identity and validation result; -- required runtime configuration grouped by connector; -- secret-entry controls whose values are write-only; -- `Save secrets`, `Forget` per configured value, and `Test workspace connection`; -- clear consequences for each button and a reminder that source changes must be committed and pushed - by an author outside ThothII before repository update. - -The browser keeps form values only in component memory and clears them after submission or dialog -close. It never receives saved secret values. - -## Compatibility and migration - -Existing Git author settings, publish endpoints, bundle endpoints, generated-document -reconciliation, bootstrap catalog slots, and browser draft storage are removed. Existing environment -bindings may be read during a bounded migration period only to seed non-secret connector values; -secret file paths are not part of the new public workspace contract. - -Session manifests continue to pin an immutable validated workspace revision. An already running -session keeps its acquired runtime lease; new or resumed work resolves the current encrypted secret -generation and fails closed when required credentials are unavailable. - -## Failure handling and security - -- Repository and vault errors use stable sanitized codes and never echo remotes with user info, - credential paths, secret identifiers that are not safe to disclose, or secret values. -- Vault writes are atomic and authenticated; corrupted ciphertext fails closed. -- Secret comparison uses no read API. Updating a secret is always a blind replacement. -- The backend applies request-size and field-count limits and rejects unknown requirement IDs. -- Temporary plaintext files use restrictive permissions, trusted directories, no-follow opens, and - deterministic cleanup. -- Git credentials are installation-only, read-only, and never sent to the frontend. - -## Verification - -Backend tests cover repository read-only behavior, atomic candidate activation, remote sanitization, -vault encryption and corruption, requirement discovery, blind secret writes/deletes, materialization -cleanup, readiness transitions, and absence of publish/bundle routes. - -Frontend tests cover the two-level explanation, viewport dimensions, repository identity, selection -gating, dynamic secret forms, write-only behavior, status changes, and absence of local-storage, -import, export, editing, and publishing controls. - -Deployment and CLI tests cover required remote/branch configuration, one read-only Git transport, -sanitized remote preflight, vault-key provisioning, and removal of Git author/write configuration. diff --git a/docs/plans/2026-08-14-read-only-workspace-runtime-secrets.md b/docs/plans/2026-08-14-read-only-workspace-runtime-secrets.md deleted file mode 100644 index d1709de6..00000000 --- a/docs/plans/2026-08-14-read-only-workspace-runtime-secrets.md +++ /dev/null @@ -1,401 +0,0 @@ -# Read-only Workspace Runtime Secrets Implementation Plan - -> **Historical nomenclature:** this plan predates the native host CLI convergence. References to -> `thothctl` and `tools/thothctl` describe the implementation snapshot from which this plan was -> written; current operator commands and paths use native `tht` and `tools/tht`. - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Make workspace consumption strictly read-only while adding installation-scoped Git identity and persistent GUI-managed runtime secrets. - -**Architecture:** Git remains the source of truth and is fetched into an application-owned checkout; complete candidate commits are validated before atomic activation and the backend has no Git write path. Runtime connector credentials are discovered from trusted connector contracts, stored as authenticated ciphertext by a backend vault, and materialized only for the lifetime of diagnostics or runtime leases. The browser exposes repository/readiness status and write-only secret forms without workspace persistence. - -**Tech Stack:** Fastify, TypeScript, Node.js crypto/filesystem, React 18, TanStack Query, Vitest, Go `thothctl`, Docker Compose. - ---- - -### Task 1: Freeze the Git repository boundary to read-only - -**Files:** -- Modify: `backend/src/workspaces/types.ts` -- Modify: `backend/src/workspaces/git-repository.ts` -- Modify: `backend/src/workspaces/registry.ts` -- Modify: `backend/test/workspaces-git-repository.test.ts` -- Modify: `backend/test/workspace-registry.test.ts` -- Modify: `backend/test/workspace-registry-deployment.test.ts` - -**Step 1: Write failing tests** - -Add tests proving that pull never configures a Git author, writes generated files, commits, or pushes; that a malformed candidate leaves the prior active snapshot intact; and that a missing catalog descriptor rejects the whole candidate instead of producing a bootstrap slot. - -**Step 2: Run the focused tests** - -Run: `cd backend && npx vitest run test/workspaces-git-repository.test.ts test/workspace-registry.test.ts test/workspace-registry-deployment.test.ts` - -Expected: FAIL on write/publish behavior and missing-descriptor semantics. - -**Step 3: Implement the read-only boundary** - -Remove `gitAuthorName`, `gitAuthorEmail`, mutation helpers, generated-document reconciliation, publish/conflict types, and bootstrap-slot activation. `pull()` must fetch, validate the complete commit in a candidate snapshot, and replace active state only after validation succeeds. - -**Step 4: Run focused tests** - -Run the command from Step 2. - -Expected: PASS. - -**Step 5: Commit** - -```bash -git add backend/src/workspaces backend/test/workspaces-git-repository.test.ts backend/test/workspace-registry.test.ts backend/test/workspace-registry-deployment.test.ts -git commit -m "refactor: make workspace repository strictly read only" -``` - -### Task 2: Remove publishing and bundle HTTP contracts - -**Files:** -- Modify: `backend/src/routes/workspaces.ts` -- Modify: `backend/test/routes-workspaces.test.ts` -- Modify: `backend/test/workspaces-runtime-v3-boundaries.test.ts` -- Modify: `backend/src/config.ts` -- Modify: `backend/test/workspaces-config.test.ts` - -**Step 1: Write failing route tests** - -Assert `POST /workspaces/publish`, `GET /workspaces/:id/export`, and `POST /workspaces/import` return 404 and that the backend no longer registers multipart or ZIP handling. Assert configuration no longer accepts Git author or bundle-limit settings as workspace-registry fields. - -**Step 2: Run tests and observe failure** - -Run: `cd backend && npx vitest run test/routes-workspaces.test.ts test/workspaces-config.test.ts test/workspaces-runtime-v3-boundaries.test.ts` - -Expected: FAIL because mutation and bundle routes still exist. - -**Step 3: Remove the mutation surface** - -Delete publish/import/export schemas and helpers, remove `multipart`, `yauzl`, and `yazl` usage from the route, and simplify safe workspace errors to read/validate/sync errors. - -**Step 4: Run tests** - -Run the command from Step 2 plus `cd backend && npx tsc --noEmit -p .`. - -Expected: PASS. - -**Step 5: Commit** - -```bash -git add backend/src backend/test package.json package-lock.json -git commit -m "refactor: remove workspace publishing and bundles" -``` - -### Task 3: Expose a sanitized installation repository identity - -**Files:** -- Modify: `backend/src/workspaces/git-repository.ts` -- Modify: `backend/src/routes/workspaces.ts` -- Modify: `backend/test/workspaces-git-repository.test.ts` -- Modify: `backend/test/routes-workspaces.test.ts` -- Modify: `tools/thothctl/internal/config/installation.go` -- Modify: `tools/thothctl/internal/config/installation_test.go` -- Modify: `deploy/psd/thothii-installation.yaml.example` -- Modify: `docs/install/examples/thothii-installation.local.yaml` -- Modify: `docs/install/examples/thothii-installation.server.yaml` - -**Step 1: Write failing parser and status tests** - -Cover HTTPS, SSH URL, and SCP-style remotes; reject embedded user-info for HTTPS; return only `host`, `repository`, `branch`, and `transport`; never return a token, key path, or raw credential-bearing URL. Add installation-descriptor tests for a required `workspaceRepository` block and exactly one read-only transport. - -**Step 2: Run focused tests** - -Run: `cd backend && npx vitest run test/workspaces-git-repository.test.ts test/routes-workspaces.test.ts && cd ../tools/thothctl && go test ./internal/config` - -Expected: FAIL because repository identity and typed installation configuration do not exist. - -**Step 3: Implement safe normalization and installation validation** - -Add the normalized identity to registry status. Extend `thothii-installation.yaml` with remote, branch, and SSH/HTTPS access metadata, validate it against the selected Compose override and environment without reading or returning secret values, and retain the existing environment rendering boundary. - -**Step 4: Run focused tests** - -Run the command from Step 2. - -Expected: PASS. - -**Step 5: Commit** - -```bash -git add backend tools/thothctl deploy docs/install/examples -git commit -m "feat: declare workspace repository in installation config" -``` - -### Task 4: Add the persistent encrypted workspace secret store - -**Files:** -- Create: `backend/src/workspaces/secret-store.ts` -- Create: `backend/test/workspace-secret-store.test.ts` -- Modify: `backend/src/config.ts` -- Modify: `backend/src/app.ts` -- Modify: `compose.yaml` -- Modify: `deploy/compose.local.yaml` -- Modify: `deploy/compose.server.yaml` - -**Step 1: Write failing vault tests** - -Test first-start initialization, atomic blind replacement, deletion, enumeration by configured ID only, AES-256-GCM ciphertext with installation/workspace/field associated data, corruption failure, restrictive files/directories, size limits, and absence of plaintext in persistent bytes. - -**Step 2: Run the vault test** - -Run: `cd backend && npx vitest run test/workspace-secret-store.test.ts` - -Expected: FAIL because `WorkspaceSecretStore` does not exist. - -**Step 3: Implement the vault** - -Create an injectable `WorkspaceSecretStore` backed by an application-managed data root. Persist a versioned encrypted document atomically, generate or load the installation vault key in the private control area, expose only `has`, `put`, `delete`, and scoped materialization operations, and never add a plaintext read API. - -**Step 4: Run tests and typecheck** - -Run: `cd backend && npx vitest run test/workspace-secret-store.test.ts && npx tsc --noEmit -p .` - -Expected: PASS. - -**Step 5: Commit** - -```bash -git add backend compose.yaml deploy -git commit -m "feat: persist encrypted workspace runtime secrets" -``` - -### Task 5: Derive connector requirements and integrate temporary materialization - -**Files:** -- Create: `backend/src/workspaces/secret-requirements.ts` -- Create: `backend/test/workspace-secret-requirements.test.ts` -- Modify: `backend/src/workspaces/bindings.ts` -- Modify: `backend/src/workspaces/runtime-config-lease.ts` -- Modify: `backend/src/tht/tht-runner.ts` -- Modify: `backend/src/app.ts` -- Modify: `backend/test/workspace-runtime-config-lease.test.ts` -- Modify: `backend/test/workspace-runtime-handoff.test.ts` -- Modify: `backend/test/workspaces-bindings.test.ts` - -**Step 1: Write failing requirement and lifecycle tests** - -Cover PostgreSQL password, REST bearer API key, unauthenticated REST, SSH private key/password, signed HTTP Evidence, and static S3 credentials. Assert temporary files are restrictive, live for exactly one diagnostic/runtime lease, disappear on release and error, and are never persisted in the encrypted vault document. - -**Step 2: Run focused tests** - -Run: `cd backend && npx vitest run test/workspace-secret-requirements.test.ts test/workspaces-bindings.test.ts test/workspace-runtime-config-lease.test.ts test/workspace-runtime-handoff.test.ts` - -Expected: FAIL because requirements still come from installation secret-file paths. - -**Step 3: Implement dynamic requirement resolution** - -Use the selected DWH transport and Evidence authentication contract to map trusted installation-contract suffixes to stable GUI requirement IDs. Overlay materialized temporary file paths only while resolving existing file-oriented connectors, and attach cleanup to every runtime lease. - -**Step 4: Run tests and typecheck** - -Run the command from Step 2 plus `cd backend && npx tsc --noEmit -p .`. - -Expected: PASS. - -**Step 5: Commit** - -```bash -git add backend/src backend/test -git commit -m "feat: resolve workspace secrets from connector requirements" -``` - -### Task 6: Add write-only workspace secret and readiness APIs - -**Files:** -- Modify: `backend/src/routes/workspaces.ts` -- Modify: `backend/src/app.ts` -- Modify: `backend/src/workspaces/types.ts` -- Modify: `backend/test/routes-workspaces.test.ts` - -**Step 1: Write failing API tests** - -Test `GET /workspaces/:id/runtime-configuration`, blind `PUT /workspaces/:id/secrets`, and `DELETE /workspaces/:id/secrets/:requirementId`. Assert strict bodies, limits, unknown-ID rejection, status-only responses, diagnostic invalidation, and `configuration_required`/`ready` state transitions. - -**Step 2: Run tests** - -Run: `cd backend && npx vitest run test/routes-workspaces.test.ts` - -Expected: FAIL because the routes do not exist. - -**Step 3: Implement the routes and readiness projection** - -Inject the secret store into workspace routes and runtime support. Compute per-workspace readiness from active descriptor, current requirement set, configured IDs, and diagnostic generation. Materialize values only inside the diagnostic request and always clean up. - -**Step 4: Run backend gates** - -Run: `cd backend && npx vitest run && npx tsc --noEmit -p . && npm run build`. - -Expected: PASS. - -**Step 5: Commit** - -```bash -git add backend -git commit -m "feat: manage runtime workspace secrets through the API" -``` - -### Task 7: Replace workspace management with the two-level read-only UI - -**Files:** -- Modify: `frontend/src/api/workspaces.ts` -- Modify: `frontend/src/api/workspaces.test.ts` -- Modify: `frontend/src/shell/WorkspaceManager.tsx` -- Modify: `frontend/src/shell/WorkspaceManager.test.tsx` -- Delete: `frontend/src/shell/WorkspacePublishDialog.tsx` -- Delete: corresponding publish-dialog tests -- Modify/Delete: `frontend/src/shell/WorkspaceEditor.tsx` and bootstrap-only tests as references permit -- Modify: `frontend/src/workspaces/drafts.ts` -- Modify: `frontend/src/workspaces/drafts.test.ts` - -**Step 1: Write failing UI/API tests** - -Assert the dialog uses at least 60% viewport width and height, shows general repository concepts and exact button consequences at level 1, gates workspace-specific controls on selection, renders requirement explanations and write-only fields at level 2, and has no create/edit/publish/import/export/bundle controls. - -**Step 2: Run focused tests** - -Run: `cd frontend && npx vitest run src/api/workspaces.test.ts src/shell/WorkspaceManager.test.tsx src/workspaces/drafts.test.ts` - -Expected: FAIL on the old draft/publish interface. - -**Step 3: Implement the read-only interface** - -Replace bootstrap editor state with repository status, selection, validation/readiness details, dynamic secret fields, blind save/forget actions, and connection test. Remove workspace draft persistence and clear secret field component state after submit/close. - -**Step 4: Run focused tests and typecheck** - -Run the command from Step 2 plus `cd frontend && npx tsc -b`. - -Expected: PASS. - -**Step 5: Commit** - -```bash -git add frontend -git commit -m "feat: add read-only workspace and secret management UI" -``` - -### Task 8: Remove browser-persisted workspace preferences - -**Files:** -- Modify: `frontend/src/workspaces/preferences.ts` -- Modify: `frontend/src/workspaces/preferences.test.ts` -- Modify: `frontend/src/api/sessions.ts` -- Modify: `frontend/src/api/sessions.test.ts` -- Modify: `frontend/src/shell/SteerInput.tsx` -- Modify: `frontend/src/shell/SteerInput.test.tsx` - -**Step 1: Write failing persistence-boundary tests** - -Assert workspace/model/thinking choices are kept only in current application memory or saved through the existing backend settings API, and that no workspace code calls `localStorage`. - -**Step 2: Run focused tests** - -Run: `cd frontend && npx vitest run src/workspaces/preferences.test.ts src/api/sessions.test.ts src/shell/SteerInput.test.tsx` - -Expected: FAIL because preferences still use browser storage. - -**Step 3: Implement ephemeral preferences** - -Replace the storage adapter with an in-memory external store seeded from backend settings. Preserve concurrent workspace-policy gates and session request determinism without persisting selections in the browser. - -**Step 4: Run frontend gates** - -Run: `cd frontend && npx vitest run && npx tsc -b && npm run build`. - -Expected: PASS. - -**Step 5: Commit** - -```bash -git add frontend -git commit -m "refactor: stop persisting workspace state in the browser" -``` - -### Task 9: Update deployment contracts and documentation - -**Files:** -- Modify: `compose.yaml` -- Modify: `deploy/compose.git-ssh.yaml` -- Modify: `deploy/compose.git-https.yaml` -- Modify: `deploy/workspace-registry.env.example` -- Modify: `deploy/psd/operator.env.example` -- Modify: `docs/install/local-workspace-registry.md` -- Modify: `docs/install/server-workspace-registry.md` -- Modify: `docs/guida-utente.md` -- Modify: `scripts/verify-workspace-install-docs.sh` -- Modify: `scripts/workspace-registry-smoke.sh` - -**Step 1: Update executable contract tests first** - -Require read-only Git wording and configuration, repository identity visibility, vault persistence, -and absence of author/push/bundle/browser-secret instructions. - -**Step 2: Run contract tests and observe failure** - -Run: `bash scripts/verify-workspace-install-docs.sh` - -Expected: FAIL against the old manuals and examples. - -**Step 3: Update deployment and manuals** - -Remove Git author settings and write-oriented documentation. Document installation Git bootstrap, -GUI runtime-secret completion, platform-neutral application storage, rotation/forget flows, and -candidate validation semantics. - -**Step 4: Run contract and Go gates** - -Run: `bash scripts/verify-workspace-install-docs.sh && cd tools/thothctl && go test ./...` - -Expected: PASS. - -**Step 5: Commit** - -```bash -git add compose.yaml deploy docs scripts tools/thothctl -git commit -m "docs: describe read-only workspace runtime configuration" -``` - -### Task 10: Full verification and deployed-container refresh - -**Files:** -- Modify only files needed to fix failures found by verification. - -**Step 1: Run static and unit gates** - -```bash -cd backend && npx vitest run && npx tsc --noEmit -p . && npm run build -cd ../frontend && npx vitest run && npx tsc -b && npm run build -cd ../harness && .venv/bin/pytest -q -cd ../tools/thothctl && go test ./... -``` - -Expected: all gates PASS. - -**Step 2: Run deployment contract gates** - -Run: `bash scripts/verify-workspace-install-docs.sh` and the focused workspace registry smoke appropriate to the configured installation. - -Expected: PASS without Git writes or secret disclosure. - -**Step 3: Inspect the final diff and secret scan** - -Run: `git diff --check`, inspect `git status --short`, and search active code/config for removed publish, bundle, Git author, and workspace-localStorage contracts. - -Expected: no whitespace errors, no accidental secrets, and only intended changes. - -**Step 4: Rebuild and restart affected services** - -Use the installation-aware `thothctl` lifecycle for the configured installation to rebuild/restart `core` and `frontend`, then verify health and repository status. Do not restart if no valid local installation descriptor is available; report that external gate explicitly. - -**Step 5: Commit verification fixes** - -```bash -git add -git commit -m "test: verify read-only workspace secret flow" -``` diff --git a/docs/plans/2026-08-18-evidence-canonica-design.md b/docs/plans/2026-08-18-evidence-canonica-design.md deleted file mode 100644 index 012f89c4..00000000 --- a/docs/plans/2026-08-18-evidence-canonica-design.md +++ /dev/null @@ -1,211 +0,0 @@ -# Evidence canonica — struttura tipizzata per disambiguazione, schema linking e SQL - -## Contesto e decisioni prese - -ThothII ha già due livelli separati che non si parlano: - -- **`EvidenceDoc`** (`harness/tht/evidence/model.py`) — runtime: frontmatter piatto - (`id/title/tier/status/tables/concepts/sources`) + `body` markdown libero. `tier` - distingue solo `structural|concept`, insufficiente rispetto ai 6 tipi reali del corpus. -- **Pipeline corpus** (`harness/tht/corpus/`) — canonizzazione *tecnica* (hash, - provenienza, chunking, vettorizzazione), agnostica rispetto al tipo di evidenza. - -Il corpus reale (`ChironeWp3/artifacts/evidence/`) ha una tassonomia implicita in 6 -directory (`00-glossario`, `10-domini-clinici`, `20-valori-enum`, `30-esempi-nlq`, -`40-mapping-semantico`, `50-metadati-normalizzazione`) ma: - -- frontmatter piatto, `tier` inadeguato; -- file già rotti (`--` invece di `---`, bullet `•⁠ ⁠` invece di `-`) che `EvidenceDoc.parse` - rifiuterebbe; -- al retrieval la struttura si perde: `tht search pack` proietta solo `title` + 400 char. - -**Decisioni (confermate nel brainstorming):** - -1. **Obiettivo**: strutturare il *contenuto* runtime, non toccare la pipeline corpus. -2. **Forma**: ibrido — frontmatter tipizzato + sezioni canoniche per `kind`. -3. **Tassonomia**: doppia — `kind` (contenuto) + `applies_to` (destinazioni: - `disambiguation`, `rewriting`, `schema_linking`, `sql_generation`, `memory`). -4. **Anchors come fonte primaria** per il value-grounding in F4; LSH solo fallback. -5. **Approccio**: contratto in `tht` + authoring sottile (niente app separata, niente - client LLM diretto in `tht`). -6. **Modello sorgente→canonizzato** (non sovrascrittura in place): file umani in - `source_root/evidence/`, canonizzati derivati in `artifacts/evidence/`, coerente con - l'esistente `tht evidence extract`. -7. **Review**: batch con diff aggregato (approvazione per file). -8. **Rielaborazione**: LLM-assistita via Pi (sessione di manutenzione + gate), con parte - deterministica in `tht`. - -## Modello canonico v2 - -### `CanonicalEvidence` (frontmatter tipizzato) - -```yaml ---- -schema_version: 2 -id: ev-dom-ablazione-see -title: Dominio Ablazione e SEE -kind: domain # glossario | domain | enum | example | mapping | normalization -applies_to: # destinazioni d'uso - - disambiguation # F1 - - schema_linking # F4 - - sql_generation # F6/F7 -status: reviewed -language: it -concepts: - - term: ablazione - synonyms: [ablazione transcatetere, SEE, studio elettrofisiologico] - - term: fibrillazione atriale - synonyms: [FA] -tables: - - name: datawarehouse.fact_studio_elettrofisiologico_endocavitario_ablazione - role: fact - columns: [ablazione_transcatetere, cod_paz, num] -anchors: - - column: datawarehouse.fact_see_ablazione_procedura_patologia.patologia - value: ablazione - match: exact -sources: [...] -# provenienza del derivato (aggiunta dal canonicalize, non dall'umano): -source_file: 10-domini-clinici/ablazione.md -source_fingerprint: sha256:... -canonicalized_at: 2026-08-18T... ---- -``` - -Il `body` resta markdown, ma con **titoli di sezione canonici per `kind`**: -- `glossario`: `## Definizione` -- `domain`: `## Cosa rappresenta`, `## Schema a stella`, `## Granularità`, `## Domande di business tipiche` -- `enum`: `## Valori ammessi` -- `example`: `## Domanda → SQL` + `## Nota clinica` -- `mapping`: `## Trasformazioni disponibili` -- `normalization`: `## Regole di normalizzazione` - -Le sezioni canoniche permettono al retrieval di estrarre solo la sezione rilevante per -fase invece di 400 caratteri generici. - -### `kind` vs `applies_to` - -- `kind` descrive il *contenuto* (i 6 tipi già impliciti). -- `applies_to` descrive *quando usarlo*. Esempi: `enum-tipo-intervento` è `kind: enum` ma - `applies_to: [disambiguation, sql_generation]`; un `example` è - `applies_to: [sql_generation, memory]`. - -## Modello a 3 livelli - -``` -source_root/evidence/*.md (umano, libero, può essere sporco) - │ tht evidence canonicalize (deterministico + LLM via Pi + review batch) - ▼ -artifacts/evidence/*.md (canonico, validato, derivato, rigenerabile) - │ tht evidence index (esistente) - ▼ -corpus / vector store (vettorializzato dal canonico, mai dal sorgente) -``` - -Il canonizzato porta `source_file` + `source_fingerprint`: se il sorgente cambia, la -canonizzazione ripropone il diff; se invariato, no-op. - -## Componenti da creare/modificare - -### 1. `harness/tht/evidence/model.py` (modifica) -- Aggiungere `CanonicalEvidence` (pydantic, v2) con `schema_version`, `kind`, - `applies_to`, `concepts[]` (`{term, synonyms[]}`), `tables[]` - (`{name, role, columns[]}`), `anchors[]` (`{column, value, match}`), `source_file`, - `source_fingerprint`, `canonicalized_at`. -- Validatore strict: `kind` ammesso, `applies_to` ammesso, `anchors[].column` deve - essere `schema.colonna` ben formato, `match` in `{exact, contains, regex, substring}`. -- `EvidenceDoc` v1 resta per leggere il sorgente e per retro-compatibilità dei vecchi - canonizzati; `CanonicalEvidence` è un modello separato, non una sottoclasse. -- Validatore che garantisce `source_fingerprint` = sha256 del contenuto sorgente. - -### 2. `harness/tht/evidence/lint.py` (nuovo, deterministico) -- `lint_source(path) -> list[Diagnostic]`: frontmatter rotto, `tier`/`status` non validi, - bullet Unicode, campi mancanti, `tables` non qualificate con schema, concetti senza - sinonimi, sezioni non canoniche per il `kind` atteso. -- Exit code 0 se nessun errore, 1 se warning, 2 se errori. Nessun LLM. - -### 3. `harness/tht/evidence/canonicalize.py` (nuovo) -- `plan(sources, artifacts) -> CanonicalizePlan`: confronta fingerprint dei sorgenti con - i canonizzati esistenti, produce la lista dei file da (ri)canonizzare. -- `render_proposal(source, llm_completions) -> CanonicalEvidence`: assembla il - canonizzato da parsing deterministico + campi semantici forniti da Pi. -- `apply(plan, approved_ids) -> None`: scrive solo i canonizzati approvati in - `artifacts/evidence/`, atomico per file, mantiene la gerarchia per dominio. -- `diff(source, candidate) -> str`: diff markdown per il gate. - -### 4. `harness/tht/vectorstore/records.py` (modifica) -- `evidence_records()`: metadata ora include `kind`, `applies_to`, `anchors` (non solo - `status/tier/tables/concepts`). -- Il `content` di ogni record resta title+body, ma per file con sezioni canoniche si - indicizza anche un record per sezione (id `evidence::`) così il retrieval - può restringere per sezione oltre che per documento. - -### 5. `harness/tht/search/__init__.py` + `cli/search_cmd.py` (modifica) -- `SearchResult` e `combined_search` accettano `applies_to` come filtro metadata. -- `tht search pack`: la sezione "Evidence rilevanti" ora proietta `kind` + sezione - canonica pertinente (non solo excerpt generico); usa `applies_to` per non mischiare - destinazioni. -- `tht search find --kind evidence --applies-to ` per il retrieval mirato. - -### 6. `harness/tht/cli/evidence_cmd.py` (modifica) -- Nuovi comandi: - - `tht evidence lint --source ` — diagnostica deterministica. - - `tht evidence canonicalize plan --source-root --json` — piano di - (ri)canonizzazione. - - `tht evidence canonicalize apply --plan --approved --json` — applica. -- `--json` sempre pristine (solo JSON su stdout), come da contratto di progetto. - -### 7. `harness/.pi/extensions/tht-gate.js` (modifica) -- Nuovo tool `reviewer_evidence_batch`: riceve il piano + i diff aggregati e presenta un - multiselect per file (approva/rifiuta/richiedi modifica). Le scelte approvate vengono - persistite tramite `tht evidence canonicalize apply`. -- Aggiungere `tht evidence canonicalize apply` alla FORBIDDEN anti-bypass list (il - modello non può applicare canonizzazioni senza review). - -### 8. Sessione di manutenzione Pi (nuova skill o sezione SKILL) -- Una skill dedicata `tht-evidence-canonicalize` (o un comando slash nel gate) descrive - la sessione di manutenzione: il modello legge i sorgenti marcati dal piano, propone i - campi semantici (`kind`, `applies_to`, `concepts` con sinonimi, `tables` con ruolo, - `anchors`) e li invia al `reviewer_evidence_batch`. -- Il gate è l'unico canale di review; il modello non scrive mai direttamente - `artifacts/evidence/`. - -### 9. `harness/.pi/skills/tht-sessione/SKILL.md` (modifica) -- Aggiornare i punti F1/F4/F6-F7 dove il modello legge evidence: spiegare che l'evidence - canonica espone `kind`/`applies_to`/`anchors`, che gli `anchors` sono la fonte primaria - per il value-grounding in F4, e che le sezioni canoniche vanno citate per destinazione. -- Aggiungere il nuovo tool `reviewer_evidence_batch` alla lista dei tool disponibili. - -## Migrazione del corpus (incrementale, v1+v2 convivono) - -- `CanonicalEvidence` (v2) convive con `EvidenceDoc` (v1): `lint` segnala i non-canonici - ma nulla si rompe; `load_evidence_dir` carica entrambi. -- Convertire a lotti, partendo da `20-valori-enum` e `10-domini-clinici` (impatto - maggiore su F1/F4, anchors più ricchi). -- Il canonizzato derivato in `artifacts/evidence/` sostituisce progressivamente il file - sorgente mirrorato; finché un sorgente non è canonizzato, `extract` continua a - rispecchiare il v1 come oggi. - -## Ordine di implementazione - -1. `CanonicalEvidence` + `lint` (TDD, nessun LLM). -2. `canonicalize.py` (plan/apply/diff) + comandi CLI (TDD). -3. `records.py` + filtro `applies_to` in `search` + proiezione pack (TDD). -4. Gate `reviewer_evidence_batch` + skill manutenzione Pi. -5. Aggiornamento `SKILL.md` tht-sessione. -6. Conversione primo lotto (`enum` + `domain`) con review batch. - -## Verifica - -- `cd harness && .venv/bin/pytest -q` — test nuovi per model/lint/canonicalize/records/search. -- `node --test` sul gate per `reviewer_evidence_batch` e anti-bypass. -- Su un campione del corpus: `tht evidence lint` riporta i file rotti (frontmatter `--`, - bullet Unicode) senza crash; `tht evidence canonicalize plan --json` produce piano - corretto; `apply` con fingerprint invariato è no-op. -- `tht search pack ""` mostra evidence con `kind`/sezione pertinente e - gli anchor presenti; `tht search find --kind evidence --applies-to schema_linking` - restituisce solo evidence pertinenti. -- Live: una sessione F4 su psd usa un anchor canonico per il value-grounding invece del - solo LSH (verificabile dal `reviewer_decide` con opzioni `value_grounded` provenienti - dall'evidence, non dal ranking LSH). -- `git diff --check` e typecheck/ruff puliti. diff --git a/docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md b/docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md deleted file mode 100644 index 930194a5..00000000 --- a/docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md +++ /dev/null @@ -1,408 +0,0 @@ -# ThothII Authentication Acceptance and PSD Deployment Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to execute this plan task-by-task. - -**Goal:** Validate local and OIDC authentication on macOS, deploy the exact feat/thoth-auth candidate to the Aritmolab/PSD server before merging it into main, and complete end-to-end acceptance with remote Authentik. - -**Architecture:** Test the candidate first as a standalone local installation. Then install the same immutable Git revision on the existing PSD installation with the installation-aware tht lifecycle, leaving main untouched. Authentik provides OIDC login and a mandatory direct groups claim; ThothII maps exact external groups to roles and validates mapped groups through the Authentik catalog API. - -**Tech Stack:** macOS, Docker Desktop, Docker Compose, native host `tht` plus Python workflow `tht`, local Argon2id authentication, generic OIDC Authorization Code + PKCE, Authentik, PSD workspace registry, reverse proxy/TLS. - ---- - -## Scope and release rules - -Do not merge feat/thoth-auth into main until every mandatory gate in Task 9 is PASS and the PSD owner accepts the evidence. - -Capture one candidate revision and reuse it everywhere: - -```bash -export CANDIDATE_SHA="$(git rev-parse HEAD)" -git fetch origin feat/thoth-auth -test "$CANDIDATE_SHA" = "$(git rev-parse origin/feat/thoth-auth)" -git show -s --format='%H%n%P%n%s' "$CANDIDATE_SHA" -git status --short --untracked-files=all -``` - -Never deploy a moving branch name without checking its resolved SHA. Never put passwords, OIDC client secrets, Authentik API tokens, cookies, authorization headers, raw ID tokens, or password hashes in Git, shell history, screenshots, logs, or evidence. - -Use protected operator values for , , , , , , , and . - -The Authentik contract is mandatory: a direct non-empty JSON array claim named groups; exact groups TOT Users and TOT Admin; mappings TOT Users -> user and TOT Admin -> admin; and a separate group-view-only API service account exposed only as THT_AUTHENTIK_API_TOKEN. Extra upstream groups are valid and silently ignored. - -## Task 0: Freeze the candidate and collect approvals - -**Files:** None; record results in the acceptance report in Task 9. - -Run from the candidate worktree: - -```bash -git diff --check -go test ./... -count=1 -go test -race ./... -go vet ./... -go build ./... -``` - -Expected: all commands pass, the candidate is pushed, and existing evidence is bound to the same SHA. A historical result from another revision is not evidence for this run. - -Before touching PSD, obtain the maintenance window, server access, public URL, Authentik provider details, protected secret locations, test identities for ordinary/admin/unmapped users, and permission to test PSD DWH/Evidence connections. - -## Task 1: Prepare and start the local macOS installation - -**Files:** - -- Read: docs/install/local.md -- Read: docs/install/authentication-local.md -- Read: docs/testing/authentication-manual-acceptance.md -- Use: an untracked local installation descriptor and protected secret/password files - -**Step 1: Verify prerequisites** - -```bash -docker version -docker compose version -bash scripts/verify-line-endings.sh -``` - -Expected: Docker Desktop and Compose are available and line-ending validation passes. - -**Step 2: Build and configure** - -```bash -bash scripts/build-local.sh -bash scripts/build-tht.sh -tht setup --profile local -``` - -For an existing installation, do not overwrite data; run tht --installation update --check-only instead of setup. - -**Step 3: Start and inspect** - -```bash -tht --installation start --build -tht --installation status -tht --installation doctor --json -curl --fail http://127.0.0.1:8080/health -curl --fail http://127.0.0.1:8787/health -``` - -Expected: core, frontend, qdrant, embedding, and the completed model initializer are healthy; doctor includes authentication after configuration and before services. - -## Task 2: Configure and test local login - -**Files:** - -- Read: docs/install/authentication-local.md -- Modify only protected installation state through tht auth configure and tht auth user - -**Step 1: Bootstrap the administrator** - -```bash -tht --installation auth configure \ - --mode local --public-url http://127.0.0.1:8080 \ - --admin-user --admin-display-name \ - --password-file -``` - -Remove the temporary password file immediately. Expected: non-secret auth.yaml is created and the user store contains Argon2id hashes, never plaintext passwords. - -**Step 2: Add and inspect a normal user** - -```bash -tht --installation auth user add --role user --display-name --password-file -tht --installation auth status --json -tht --installation auth check --json -``` - -Expected: pristine redacted JSON and no credential, hash, or session secret in output. - -**Step 3: Test browser authorization** - -At http://127.0.0.1:8080, in a private browser profile: - -1. Verify unauthenticated access reaches login and protected routes are denied. -2. Log in as the normal user and verify application/session routes work. -3. Verify Pi Management and other admin-only operations return HTTP 403 or are not exposed. -4. Log out and verify the session is invalidated. -5. Log in as the administrator and verify admin-only routes work. - -Expected: ordinary users authenticate without receiving admin permissions; administrators receive the configured admin permission set. - -**Step 4: Test account failure paths** - -Use auth user disable, enable, set-password, and logout-all user --yes on the test user. Test a wrong password and refresh the old browser session after logout-all. - -Expected: generic safe failures, disabled login rejection, re-enabled login success, and forced reauthentication. The last enabled administrator cannot be disabled or demoted. - -## Task 3: Test remembered sessions and local recovery - -**Files:** - -- Read: docs/architecture/authentication.md, Browser sessions -- Read: docs/install/authentication-local.md, Session behavior and recovery - -**Step 1: Test browser restart** - -Log in as the normal user with Remember me, close the browser completely, reopen it, and revisit the application. - -Expected: the session survives within the 7-day idle / 30-day absolute limits. Do not record the cookie. - -**Step 2: Test ThothII restart** - -```bash -tht --installation stop -tht --installation start -``` - -Expected: the remembered session remains valid after backend restart. - -**Step 3: Test invalidation** - -Change the test user password or role, and separately run auth user logout-all user --yes. Refresh after each operation. - -Expected: affected sessions are rejected and reauthentication is required; configuration revision changes invalidate all sessions. - -Go/no-go: do not proceed to PSD if local login, role separation, logout, or remembered-session behavior fails. - -## Task 4: Snapshot the current PSD installation - -**Files:** - -- Read: docs/install/server.md -- Read: docs/install/server-workspace-registry.md -- Use: protected server operator and backup locations - -**Step 1: Capture live state** - -```bash -THT_BIN= -INSTALLATION= -"$THT_BIN" --installation "$INSTALLATION" status -"$THT_BIN" --installation "$INSTALLATION" doctor -"$THT_BIN" --installation "$INSTALLATION" pi status -"$THT_BIN" --installation "$INSTALLATION" pi doctor -git -C /srv/thothii/source/ThothII status --short --untracked-files=all -git -C /srv/thothii/source/ThothII rev-parse HEAD -``` - -Save the live SHA as and capture image identities, workspace registry status, and maintenance/recovery state. Stop if the checkout is dirty or recovery is pending. - -**Step 2: Drain and back up** - -Announce maintenance, close the reverse proxy or show its maintenance page, drain active work, and stop through tht. Create the protected, checksummed backup specified in docs/install/server.md, including runtime trees and PSD PostgreSQL/session data where applicable. Back up credentials separately. Never run docker compose down --volumes. - -**Step 3: Check preconditions** - -```bash -git -C /srv/thothii/source/ThothII config --local core.autocrlf false -bash /srv/thothii/source/ThothII/scripts/verify-line-endings.sh -"$THT_BIN" --installation "$INSTALLATION" update --check-only -``` - -Expected: descriptor, protected secrets, Pi-state mount, workspace repository binding, and Compose render remain valid before source changes. - -## Task 5: Deploy the feature revision to PSD without merging main - -**Files:** - -- Server source checkout: /srv/thothii/source/ThothII -- Server operator binary: protected THT_BIN path -- Server installation descriptor and secret files: unchanged paths unless a reviewed auth update is required - -**Step 1: Select the exact candidate** - -```bash -git -C /srv/thothii/source/ThothII fetch origin feat/thoth-auth -git -C /srv/thothii/source/ThothII switch --detach -test "$(git -C /srv/thothii/source/ThothII rev-parse HEAD)" = "" -git -C /srv/thothii/source/ThothII status --short --untracked-files=all -``` - -Do not merge or rebase main. The running installation is intentionally based on the detached feature revision until acceptance completes. - -**Step 2: Build candidate artifacts** - -```bash -cd /srv/thothii/source/ThothII -bash scripts/build-local.sh -THT_THT_OUTPUT_DIRECTORY=/srv/thothii/operator/build-output bash scripts/build-tht.sh -``` - -Install the architecture-appropriate candidate tht only after its build succeeds. Keep the old operator binary recoverable. - -**Step 3: Start and verify the candidate** - -```bash -"$THT_BIN" --installation "$INSTALLATION" update --check-only -"$THT_BIN" --installation "$INSTALLATION" start --build -"$THT_BIN" --installation "$INSTALLATION" status -"$THT_BIN" --installation "$INSTALLATION" doctor --json -curl --fail http://127.0.0.1:8080/health -"$THT_BIN" --installation "$INSTALLATION" pi doctor -"$THT_BIN" --installation "$INSTALLATION" pi test -``` - -Expected: candidate frontend/core and internal services are healthy, no data volume was replaced, and the candidate SHA is recorded. Liveness alone is not release approval. - -## Task 6: Configure and validate remote Authentik - -**Files:** - -- Modify protected server authentication state through tht auth configure -- Modify protected secret entries THT_OIDC_CLIENT_SECRET and THT_AUTHENTIK_API_TOKEN -- Read: docs/install/authentik.md and docs/install/authentication-oidc.md - -**Step 1: Verify Authentik** - -Verify the OAuth2/OIDC callback exactly /api/auth/oidc/callback, scopes openid/profile/email, direct groups array mapping, exact groups TOT Users and TOT Admin, and a separate group-view-only catalog service account. Inspect a disposable identity without copying its token. - -**Step 2: Configure the ThothII mapping** - -```bash -tht --installation "$INSTALLATION" auth configure \ - --mode oidc --public-url \ - --issuer --client-id \ - --authentik-base-url \ - --user-group 'TOT Users' --admin-group 'TOT Admin' -``` - -Secrets are read from protected files, never command-line arguments. Confirm the non-secret mapping is: - -```yaml -authorization: - groupRoles: - TOT Users: [user] - TOT Admin: [admin] -``` - -**Step 3: Run static and live checks** - -```bash -tht --installation "$INSTALLATION" auth status --json -tht --installation "$INSTALLATION" auth check --json -tht --installation "$INSTALLATION" auth check --interactive -tht --installation "$INSTALLATION" doctor --json -``` - -Expected: configuration, discovery, issuer, JWKS, client-secret access, catalog access, and exact existence of every mapped group pass. Doctor lists authentication after configuration and before services. No output contains credentials or bearer tokens. - -A missing mapped group must fail with redacted oidc_mapped_group_missing. Unmapped groups produce neither error nor warning. Missing, indirect, malformed, or overage-style groups claims fail closed. - -**Step 4: Reload if required** - -If configuration requires process reload: - -```bash -"$THT_BIN" --installation "$INSTALLATION" pi restart --yes --drain -``` - -Repeat authentication, doctor, and health checks. Do not substitute raw Compose commands. - -## Task 7: Test PSD browser login and authorization - -**Files:** - -- Read: docs/testing/authentication-manual-acceptance.md -- Evidence: redacted report from Task 9 - -**Step 1: Ordinary user** - -In a private profile, authenticate with an identity in TOT Users but not TOT Admin. Verify callback success, application/session routes, denial of Pi Management/admin operations, opaque HttpOnly ThothII cookie, no bearer token in Web Storage, and logout invalidation. - -**Step 2: Administrator** - -Authenticate with TOT Admin. Verify Pi Management and allowed workspace-management operations. Access must derive from the exact mapped group, not a client-supplied header or browser-local flag. - -**Step 3: Unmapped and malformed groups** - -Authenticate with a valid token containing no mapped group. Expected: login may complete, but protected operations return 403 with no warning. Use a disposable provider mapping that omits or corrupts groups; expected: generic HTTP 401 oidc_callback_failed, with no internal claim details exposed. - -**Step 4: Provider outage/group drift** - -During a controlled window, make discovery/JWKS unavailable or rename a mapped group, run the CLI check, and restore it immediately. Expected: redacted fail-closed diagnostics followed by a successful check after restoration. Do not leave production broken. - -## Task 8: Test PSD workspace validation and real connections - -**Files:** - -- Read: docs/install/server-workspace-registry.md -- Use: authenticated PSD browser sessions - -**Step 1: Validate the workspace and authentication from the host CLI** - -```bash -"$THT_BIN" --installation "$INSTALLATION" \ - workspace inspect --workspace "$WORKSPACE_ID" --json -"$THT_BIN" --installation "$INSTALLATION" auth check --json -``` - -Expected: the workspace registry is ready, authentication readiness passes, and output is redacted while identifying the active workspace revision. - -**Step 2: Verify the application boundary** - -Open the configured public URL, authenticate with the approved identity, and verify that the application reaches the selected workspace without unexpected `401`/`403` responses. Keep DWH/Evidence connection tests read-only and use only the existing approved smoke question. - -**Step 3: Test ordinary-user authorization** - -Log in as TOT Users. Confirm inspection follows ordinary permissions while validation, secret mutation, and connection tests remain unavailable unless explicitly granted. - -**Step 4: Run a harmless end-to-end smoke** - -As an authorized PSD user, create or resume one harmless known-good session: - -```text -browser login -> same-origin API -> workspace readiness -> Pi/core -> result -> logout -``` - -Do not run mutating production queries. Preserve only a session ID and redacted outcome if approved. - -## Task 9: Close acceptance, rollback if needed, and decide merge readiness - -**Files:** - -- Create: docs/testing/evidence/2026-08-18-thothii-authentication-psd-acceptance.md or the approved external evidence location -- Read: docs/install/server.md and docs/contracts/tht-pi.md - -**Step 1: Mandatory gates** - -| Gate | Required evidence | -|---|---| -| Candidate identity | Local and server SHA exactly match pushed feat/thoth-auth | -| Local startup | macOS Compose, doctor, health, and Pi smoke pass | -| Local auth | Bootstrap, ordinary/admin roles, logout, bad password, disable/enable, logout-all pass | -| Local session | Remembered session survives browser and ThothII restart; revisions invalidate it | -| Server safety | Old SHA/image/status captured; backup checksummed; maintenance/drain completed | -| Candidate deploy | Server candidate status/doctor/health pass | -| Authentik | Direct groups claim, issuer/JWKS, secrets, catalog, and mapped groups pass | -| OIDC authorization | ordinary, admin, unmapped, malformed, logout, and outage cases pass | -| Workspace integration | `tht workspace inspect` and `tht auth check` pass; the authenticated application reaches the selected workspace | -| PSD smoke | One harmless known-good session completes | -| Hygiene | No secrets, tokens, cookies, hashes, or raw claims in evidence | - -**Step 2: Write the redacted report** - -Include candidate SHA, old SHA, timestamps, commands, browser cases, redacted diagnostic/HTTP codes, Authentik issuer/client/group names, workspace ID/revision, backup/checksum location, rollback decision, and unrelated CI failures. Never include secret values, raw tokens, cookies, or hashes. - -**Step 3: Roll back a failed candidate** - -1. Keep the proxy closed and preserve .tht// recovery state. -2. Do not use tht pi rollback as the whole-application rollback; it addresses only Pi lifecycle images. -3. Stop with tht. -4. Return the source checkout to , rebuild old application/operator artifacts, and start through the same descriptor. -5. Run update --check-only, status, doctor, health, Pi smoke, workspace diagnostics, and one harmless session. -6. For ambiguous recovery, leave maintenance active and follow pi maintenance status / pi maintenance recover --yes. Never delete volumes, selectors, or recovery files to force progress. - -Expected: the previous application serves again with prior data and workspace state intact. Record the failure and do not merge. - -**Step 4: Reopen traffic** - -After every gate passes, restore the reverse proxy, repeat one unauthenticated redirect and one authorized public login, and confirm only the proxy is externally reachable. - -**Step 5: Merge decision** - -Merge only after PSD owner acceptance, exact-SHA evidence, no unresolved auth/workspace/provider/deployment gate, and an accepted rollback path. If the merge creates a new commit, repeat Tasks 0, 5, 6, and 7 against the merge SHA. - -## Handoff checklist - -Deliver the redacted report, local result/SHA, PSD candidate SHA/images, Authentik provider and group mapping confirmation, workspace validation/connection results, backup/rollback status, and an explicit READY TO MERGE or NOT READY TO MERGE decision. diff --git a/docs/plans/2026-08-19-tht-documentation-convergence.md b/docs/plans/2026-08-19-tht-documentation-convergence.md deleted file mode 100644 index 7bbe506d..00000000 --- a/docs/plans/2026-08-19-tht-documentation-convergence.md +++ /dev/null @@ -1,92 +0,0 @@ -# Tht Documentation Convergence Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Align current documentation and documentation smoke checks with the converged native host CLI `tht`, while preserving historical references only where they describe past decisions or evidence. - -**Architecture:** Treat `tools/tht/cmd/tht/main.go` as the canonical host CLI surface for installation, authentication, diagnostics, lifecycle, and workspace operations. Keep the Python `harness/.venv/bin/tht` distinction explicit for the workflow runtime, and update current operator/test instructions to invoke the native `tht` with `--installation`. - -**Tech Stack:** Markdown documentation, shell smoke tests, Go CLI command surface, repository search-based verification. - ---- - -### Task 1: Classify current and historical legacy CLI references - -**Files:** -- Inspect: `README.md`, `PROJECT_STATE.md`, `AGENTS.md`, `docs/**`, `scripts/**` -- Reference: `tools/tht/cmd/tht/main.go` - -**Step 1:** Build a complete occurrence inventory with a case-insensitive search for the former host CLI name and classify every match. - -**Step 2:** Classify each occurrence as current operator documentation, documentation smoke expectation, executable/script contract, or historical design/evidence. - -**Step 3:** Record the classification in the implementation notes before editing. - -### Task 2: Update canonical operator and installation documentation - -**Files:** -- Modify: `README.md` -- Modify: `AGENTS.md` -- Modify: `PROJECT_STATE.md` -- Modify: `docs/guida-utente.md` -- Modify: `docs/contracts/workspace-preprocessing-cli.md` -- Rename/update: `docs/contracts/tht-pi.md` as the current `tht` Pi contract -- Modify: relevant installation and architecture pages that expose operator commands - -**Step 1:** Replace current host/operator invocations with `tht --installation ...`. - -**Step 2:** Document the distinction between the native host CLI `tht` and the Python harness CLI invoked by the backend/runtime. - -**Step 3:** Update command examples for `start`, `status`, `doctor`, `auth`, `workspace`, and `pi`. - -**Step 4:** Add a short historical note only where a document must explain the former name. - -### Task 3: Rewrite authentication acceptance and manual test instructions - -**Files:** -- Modify: `docs/testing/authentication-manual-acceptance.md` -- Modify: `docs/plans/2026-08-18-thothii-authentication-acceptance-and-psd-deployment.md` -- Modify: `docs/install/authentication-local.md` -- Modify: `docs/install/authentication-oidc.md` -- Modify: `docs/install/authentik.md` - -**Step 1:** Make `tht auth status`, `tht auth check`, `tht auth check --interactive`, and `tht doctor --json` the canonical terminal preflight. - -**Step 2:** Use `tht status`, `tht start`, and `tht workspace inspect --workspace psd-clinical --json` for PSD deployment checks. - -**Step 3:** Clarify that the P8 L2 gate is authentication-to-application integration through the first reviewer gate. - -**Step 4:** Retain the prior functional test suite as a baseline and add only the authentication boundary smoke required for this acceptance. - -### Task 4: Align documentation smoke tests - -**Files:** -- Modify: `scripts/auth-docs-smoke.sh` -- Modify: `scripts/test-auth-docs-smoke.sh` -- Inspect/update: any current smoke script whose user-facing command examples still require the legacy CLI name - -**Step 1:** Replace forbidden/current command assertions with `tht` equivalents. - -**Step 2:** Preserve negative checks for obsolete authentication CLI wording. - -**Step 3:** Run the positive and negative documentation fixtures. - -### Task 5: Preserve or annotate historical material - -**Files:** -- Inspect the historical discovery specification for context, without treating it as current operator documentation. -- Inspect: dated reports and archived acceptance scripts - -**Step 1:** Do not rewrite historical titles, commit evidence, or old implementation names solely to erase history. - -**Step 2:** Add a concise “historical nomenclature” note where an archived document could otherwise be mistaken for current instructions. - -### Task 6: Verify the convergence - -**Step 1:** Run `scripts/auth-docs-smoke.sh` and `scripts/test-auth-docs-smoke.sh`. - -**Step 2:** Search active documentation for remaining legacy CLI references. - -**Step 3:** Confirm every remaining match is either an explicit historical note, an ignored runtime directory name, or a non-document executable compatibility artifact. - -**Step 4:** Run `git diff --check` and report the exact files changed plus any intentionally retained historical references. diff --git a/docs/plans/2026-08-20-psd-server-deployment-program-design.md b/docs/plans/2026-08-20-psd-server-deployment-program-design.md deleted file mode 100644 index c9bc469d..00000000 --- a/docs/plans/2026-08-20-psd-server-deployment-program-design.md +++ /dev/null @@ -1,370 +0,0 @@ -# PSD Server Deployment Program — Design - -**Date:** 2026-08-20 - -**Status:** Approved by the owner - -**Owner sequencing amendment (2026-08-21):** the Mac `rest_api` acceptance and revocation of -`legacy-shared` are deferred to one mandatory pre-Project-B gate. This permits the bounded survey -and static, non-mutating Project A private preparation to proceed without changing the Mac. It does not -authorize stopping the legacy stack, starting the new stack, opening ingress, or beginning Project B. - -**Design-time application baseline:** `main` at `5c0dc8c` (execution must freeze and record the -then-current `origin/main` SHA) - -**Design-time workspace baseline:** `tht-workspace-psd/main` at `bfbabf9` (execution must freeze and -record the then-current remote SHA) - -## Purpose - -Replace the unused legacy ThothII installation on the PSD server with the current application, -recovering only useful configuration and rebuilding runtime state from canonical sources. Complete -the work as two independently accepted projects: - -1. deploy and prove ThothII with local authentication, a direct read-only PSD DWH connection, - internal Qdrant, and internal Ollama; -2. only after Project A passes, integrate the accepted installation with the server's Authentik, - Nginx, load balancer, and the existing Aritmolab sidebar link. - -The program must be executable by the Sol LLM from a terminal local to the server. It must test -each step, retain redacted evidence, stop on unsafe uncertainty, and include a separate human -manual-test document for each project. - -## Binding decisions - -- The legacy application may be unavailable for days. Service continuity is not a goal. -- The new source clone is prepared beside the old source directory. The old and new stacks are not - kept running simultaneously: inventory and backup happen first, the old stack is stopped, and - only then is the new stack started. -- Keep the old directory, configuration, containers, images, and data only as a temporary recovery - boundary until the real Aritmolab journey passes Project B. They are disposable after Project B - automated, human, and owner PASS. Do not migrate legacy application sessions, Qdrant data, - Ollama caches, or derived indexes. -- Do not create a host `thothii` user or group. Keep UID/GID `10001:10001` as the image's unmapped - numeric runtime identity and confine numeric ownership to the new installation's writable bind - trees. Stop if either number becomes mapped to a host account before installation. -- Recover configuration only: endpoints, non-secret policy, provider/model selection, relevant - paths, and references to protected credentials. Never copy an old setting without validating it - against the current contract. -- Use the canonical ThothII Compose distribution and reviewed installation-local overrides. Do not - modify an unrelated global Compose project or blindly adapt the legacy Compose file. -- Lifecycle operations use the installation-aware native `tht` CLI, not raw Compose commands. -- Never use `docker compose down --volumes`, global prune operations, broad recursive deletion, or - secret-bearing command arguments. - -## Repository and workspace model - -There are two Git sources with different responsibilities: - -- `ThothII` contains application code and non-secret deployment examples. -- `tht-workspace-psd` contains the shared, credential-free PSD workspace source. - -The workspace repository contains one logical workspace, `psd-clinical`. Its schema-v3 descriptor -will declare both supported DWH transports: - -```yaml -supported_transports: [rest_api, postgres_direct] -``` - -The Mac installation continues to select `rest_api`. The PSD server selects `postgres_direct`. -Endpoint values, user names, passwords, secret-file paths, machine paths, and Git credentials stay -in each installation's protected bindings and never enter the workspace repository. Evidence, -curated annotations, language, model policy, and semantic-index contract remain shared. - -Each installation owns its own Qdrant collection contents, Ollama model cache, preprocessing state, -and sessions even though both consume the same reviewed workspace commit. - -## Program structure - -The program consists of one non-mutating common survey followed by two independently gated -projects: - -```text -Common Survey PASS for Project A private scope - -> Project A automated PASS - -> Project A human PASS - -> Mac REST acceptance - -> 48-hour dual-key observation covering two 03:00 ETL cycles - -> legacy-shared revocation and negative proof - -> full pre-Project-B survey PASS - -> explicit Project B authorization - -> Project B automated PASS - -> Project B human PASS - -> final cutover acceptance -``` - -Project B must not start from a partial or assumed Project A result. - -## Common Survey - -The survey is a prerequisite, not a third implementation project. It runs before any server -mutation and produces a redacted report, topology map, change-scope inventory, unknowns list, and -GO/NO-GO decision. - -Sol inventories from the server-local terminal: - -- operating system, architecture, Docker and Compose versions, available CPU/RAM/disk, and clock; -- legacy ThothII source, SHA, dirty state, images, containers, networks, ports, volumes, mounts, - health, installation state, and recovery state; -- effective Compose rendering and ownership of every relevant file; -- DWH database/schema, direct listener, TLS, read-only role, and reachability from containers; -- Supabase/PostgreSQL topology, schema conventions, exposed PostgREST schemas, migration policy, - backup mechanism, and suitable role boundaries; -- Pi provider/model policy, credential references, external LLM reachability, and current versions; -- Nginx effective configuration, ThothII virtual host/location, upstream, forwarded headers, SSE - settings, certificate metadata, certificate generation/renewal, and rollback files; -- load-balancer routes, health checks, allowlist capability, TLS boundary, and configuration owner; -- Aritmolab deployment, networks, homepage, sidebar source, current ThothII destination, and release - procedure; -- Authentik version, deployment, current Aritmolab integration, provider conventions, group - conventions, backup/export procedure, API access, and credential locations; -- public DNS/origin that the final sidebar link must preserve; -- protected files by path, ownership, mode, and readability only, without printing their contents. - -The survey may use hashes, metadata, redacted renders, and permission checks. It must not emit -passwords, bearer tokens, API keys, cookies, OIDC client secrets, private keys, password hashes, or -raw identity tokens. If credential discovery fails, the owner may help locate the existing -Authentik credentials. - -## Project A — Standalone Server Acceptance - -### Runtime architecture - -Project A installs a clean five-service stack: - -```text -operator terminal or local headless browser - -> frontend - -> core + Pi + workflow harness - -> direct read-only PSD PostgreSQL DWH - -> internal Qdrant - -> internal Ollama (qwen3-embedding:0.6b, 1024 dimensions) - -> local filesystem work-session storage -``` - -The stack uses local authentication. It must not be publicly reachable. Core, Qdrant, and Ollama -remain private; the frontend binds to loopback unless the optional restricted test route below is -proved safe. - -### Preparation and cutover - -1. Freeze exact application and workspace SHAs and require clean source trees. -2. Publish and validate the multi-transport `psd-clinical` descriptor through the curator workflow. -3. Preserve the Mac REST binding unchanged; its live acceptance is deferred to the mandatory - pre-Project-B gate. -4. Prepare the new source clone and protected operator/runtime directories beside the old source. -5. Extract only approved configuration facts from the legacy installation. -6. Record and verify the exact restart recipe for the legacy stack. No data backup is required - because the owner declared legacy sessions and configuration disposable; the still-present - containers, images, source, and data are the temporary rollback boundary. -7. Stop the legacy stack without deleting its source, configuration, images, or data. -8. Build/install the current native `tht` and application images from the frozen source. -9. Configure local authentication, direct DWH bindings, Pi/provider access, and the Git workspace. -10. Start through `tht`, then validate health, authentication, workspace, workflow, and Pi. -11. Rebuild schema/Evidence preprocessing, Qdrant contents, and the Ollama model cache from - canonical sources. Prove the complete preprocessing rerun is idempotent. - -### Optional restricted test route - -If and only if the survey proves that the load balancer can enforce a test-operator allowlist -before a request reaches ThothII, Project A may use a temporary private hostname to exercise the -same network path as production: - -```text -authorized operator -> allowlisted load-balancer route -> Nginx -> frontend -> core/local auth -``` - -The route has a distinct hostname, no Aritmolab sidebar link, a certificate created by the existing -managed mechanism, correct forwarded-origin and SSE behavior, and a negative test from an -unauthorized source. Its local-auth `publicUrl` matches the private test origin. It is not a public -production route and must be removed after Project B. - -If isolation cannot be demonstrated, Sol must not approximate it or misdeclare -`THOTH_PUBLIC_EXPOSURE=false`; tests run against loopback from the local terminal instead. - -### Acceptance - -Project A requires: - -- exact source identity and reproducible build evidence; -- healthy frontend, core, Qdrant, embedding, and completed model initializer; -- local authentication checks including admin/user separation, wrong password, logout, - disable/enable, invalidation, and restart persistence; -- direct DWH connectivity with a demonstrably read-only runtime identity; -- active `psd-clinical` at the expected Git revision; -- compatible Qdrant collection/index contract and correct Ollama model/dimensions; -- complete and idempotent DWH, schema, annotation, and Evidence preprocessing; -- one harmless real PSD work session completed through F1-F8, ending in read-only validated SQL; -- persisted manifest, artifacts, reviewer decisions, and final SQL inspection; -- optional private-route positive and negative isolation evidence when that route is used; -- completed human manual-test report with an explicit PASS. - -The Mac row may be recorded only as `DEFERRED_PRE_PROJECT_B` under the dated owner amendment. It is -not part of the private server acceptance, but it must become PASS before Project B starts. - -Project A does not modify the production Aritmolab sidebar, production public route, or Authentik. - -## Project B — Authentik and Aritmolab Integration - -Project B begins only from the frozen, accepted Project A source, images, workspace revision, and -PASS report. It also requires the deferred Mac REST acceptance, the full 48-hour observation -window (including two scheduled 03:00 ETL cycles), revocation of `legacy-shared`, proof that the -legacy credential receives `401`, and the full pre-Project-B survey gate. - -### Final request and data flow - -```text -user - -> aritmolab.policlinicosandonato.com - -> Aritmolab homepage/sidebar - -> load balancer - -> Nginx/TLS - -> ThothII frontend and same-origin /api - -> ThothII OIDC Authorization Code + PKCE with Authentik - -> PostgreSQL thoth_sessions schema for owned work sessions - -> direct read-only datawarehouse schema for clinical queries - -> internal Qdrant and Ollama -``` - -Nginx terminates/proxies according to the observed deployment but does not add a second -`auth_request` in front of ThothII. ThothII performs generic OIDC directly. Nginx preserves the -public host and HTTPS scheme, forwards the callback path unchanged, and supports SSE without -buffering or premature timeouts. - -### Authentik configuration - -Before mutation, export or back up the relevant Authentik configuration. Locate existing protected -administrative/API credentials without exposing them. Create or adapt: - -- one OAuth2/OIDC provider and one ThothII application; -- the exact callback `/api/auth/oidc/callback`; -- `openid`, `profile`, and `email` scopes; -- a direct, non-empty JSON string array claim named `groups`; -- exact user/admin group mappings selected after the survey; -- a separate group-view-only service account/API token for ThothII diagnostics. - -Dedicated `TOT Users` and `TOT Admin` groups are the default unless the survey finds existing groups -with exactly the intended semantics and the owner approves their reuse. Additional groups are -ignored. Missing, malformed, indirect, or ambiguous configured groups fail closed. - -### Supabase session storage - -Do not create a separate PostgreSQL database. Use the server's existing Supabase PostgreSQL -database and isolate ThothII work sessions in the dedicated `thoth_sessions` schema. This schema is -distinct from the clinical `datawarehouse` schema. - -- A one-shot migrator role owns only the required schema migration privileges. -- Core receives only the restricted runtime role, never the migrator credential. -- Forced RLS and application ownership checks isolate sessions by Authentik principal. -- The schema stores principals/preferences, manifests, phase artifacts, review decisions, and - audit records. -- Chat and live SSE output remain ephemeral; semantic vectors remain in Qdrant; clinical data - remains in `datawarehouse`; browser authentication sessions remain in protected auth state. -- `thoth_sessions` must not be added to Supabase/PostgREST exposed schemas. -- The direct runtime connection uses the TLS/CA contract required by the current application. - -### Safe activation order - -1. Back up the accepted Project A operator/auth configuration and every external configuration to - be changed. -2. Prepare Authentik objects without exposing the new route. -3. Run and verify additive session-schema migrations; require no pending or drifted migrations. -4. Prepare OIDC secrets and non-secret configuration in protected installation state. -5. Validate Authentik discovery, issuer/JWKS, catalog access, and mapped groups. -6. Validate Nginx, certificate, load-balancer route, callback, forwarded headers, and SSE while - production traffic remains closed. -7. Start ThothII in OIDC/public server mode with PostgreSQL session storage. -8. Open the final load-balancer route. -9. Preserve or update the Aritmolab sidebar link so the established user journey remains intact. -10. Complete automated and human acceptance, then remove the Project A temporary route. - -### Acceptance - -Project B requires: - -- successful redacted static, live, and interactive authentication diagnostics; -- trusted certificate chain, correct public origin, callback, and proxy headers; -- proven load-balancer/Nginx routing and SSE operation; -- successful migration status, RLS/role tests, and proof that `thoth_sessions` is not REST-exposed; -- ordinary, administrator, unmapped, malformed-claim, logout, and controlled provider-failure - cases; -- login to Aritmolab followed by the sidebar link to ThothII without a second credential prompt; -- no raw OIDC token in browser storage, logs, diagnostics, or evidence; -- session ownership and administrator-boundary tests; -- one harmless F1-F8 PSD session under an OIDC identity; -- tested rollback and a completed human manual-test report with an explicit PASS. - -## Error handling and stop rules - -Every executable step follows: - -```text -precondition -> action -> verification -> redacted evidence -> checkpoint -``` - -Sol stops and requests owner help rather than improvising when it encounters: - -- a dirty or unidentified source checkout; -- uncertain ownership of Compose, Nginx, load-balancer, Aritmolab, or Authentik configuration; -- missing or insufficient credentials; -- an unsafe secret path or risk of secret disclosure; -- an unverifiable backup or rollback path; -- a DWH identity that is not demonstrably read-only; -- a change that would affect unrelated Nginx virtual hosts or other applications; -- an unprovable temporary-route restriction; -- Supabase migration drift, excessive roles, or unintended REST exposure; -- a material difference between the surveyed server and this design. - -Unchanged external state is not failure. Sol records the observation and continues only when the -current gate is satisfied. - -## Rollback boundaries - -- **Project A:** stop the new stack and restart the still-present legacy installation. No new - volume is copied into it and no promise is made to retain it after Project B PASS. -- **Project B ingress:** close the public route first, then restore the prior Nginx, - load-balancer, certificate reference, and sidebar configuration. -- **Project B application:** return to the accepted Project A local-auth operator configuration - while the public route remains closed. -- **Authentik:** initially disable new objects instead of deleting them; retain the pre-change - export until final acceptance. -- **Supabase:** migrations are additive. Rollback does not automatically drop `thoth_sessions` or - destroy evidence; destructive cleanup requires a separate explicit decision. -- **Workspace Git:** publish the multi-transport change as an isolated commit and retain the - previous revision. A rejected candidate never replaces the installation's last valid snapshot. - -## Documents and evidence - -The implementation-planning phase creates: - -1. a general execution program with cross-project gates; -2. a survey checklist and report template; -3. an executable Project A plan for Sol; -4. a plain-language Project A manual-test guide; -5. a Project A evidence/PASS template; -6. an executable Project B plan for Sol; -7. a plain-language Project B manual-test guide; -8. a Project B evidence/PASS template. - -Sol maintains a protected server-local progress journal and resumes from the last verified -checkpoint. Detailed topology and command output remain in a protected evidence directory on the -server. Only intentionally redacted reports and reusable templates may enter Git. - -The Project A human guide is terminal-first and may use a local headless browser. When the optional -private endpoint exists, it also includes operator browser checks. The Project B guide covers the -real Aritmolab homepage/sidebar, Authentik SSO, roles, logout, final workflow, and negative cases. - -## Known documentation reconciliation - -Some older server documentation describes vector and embedding services as external and the -mandatory stack as only frontend/core. Current `compose.yaml`, repository instructions, and project -state define Qdrant and Ollama as mandatory internal services. The execution plans must treat the -current code/Compose contract as authoritative and include a documentation correction rather than -following the stale statements. - -## Success condition - -The program is complete only when both projects have exact-source evidence, all automated gates -pass, both human manuals are completed with explicit PASS decisions, the Aritmolab sidebar reaches -the final ThothII URL through the established load balancer and Nginx, Authentik provides SSO, and a -real read-only PSD session completes F1-F8 under an authorized OIDC identity. diff --git a/docs/plans/2026-08-20-psd-server-deployment-program.md b/docs/plans/2026-08-20-psd-server-deployment-program.md deleted file mode 100644 index c8b5b692..00000000 --- a/docs/plans/2026-08-20-psd-server-deployment-program.md +++ /dev/null @@ -1,245 +0,0 @@ -# PSD Server Deployment Program Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Replace the legacy PSD ThothII installation, prove the replacement with local authentication, and then integrate the accepted release with Supabase, Authentik, Nginx, the load balancer, and the Aritmolab sidebar. - -**Architecture:** A read-only common survey freezes the actual server topology before any mutation. Project A installs a clean private stack and proves a complete PSD workflow; Project B begins only after a signed Project A PASS and performs the public OIDC/SSO cutover. Each project has an independent rollback boundary, human test guide, and evidence report. - -**Tech Stack:** Linux, Docker Engine, Docker Compose v2, native `tht`, Fastify/React/Pi, PostgreSQL/Supabase, Qdrant, Ollama, Nginx, Authentik OIDC, Aritmolab, load balancer. - ---- - -## Required reading and authority - -Read these files completely before starting: - -- `AGENTS.md` -- `PROJECT_STATE.md` -- `docs/plans/2026-08-20-psd-server-deployment-program-design.md` -- `docs/install/server.md` -- `docs/install/server-workspace-registry.md` -- `docs/install/authentication-local.md` -- `docs/install/authentication-oidc.md` -- `docs/install/authentik.md` -- `docs/contracts/workspace-preprocessing-cli.md` -- `docs/testing/authentication-manual-acceptance.md` - -The current `compose.yaml`, `deploy/compose.server.yaml`, repository instructions, and design are -authoritative where older server prose still describes Qdrant or Ollama as external. - -Run only from a terminal local to the server. Do not require SSH port forwarding. Do not print or -paste passwords, tokens, cookies, private keys, hashes, or raw identity claims. Commands that need -a credential must read a protected file or use an echo-free prompt. - -## Documents used during execution - -- Survey plan: `docs/plans/2026-08-20-psd-server-survey.md` -- Survey report: `docs/testing/evidence/psd-server-survey-report-template.md` -- Project A plan: `docs/plans/2026-08-20-psd-server-project-a-standalone.md` -- Project A human guide: `docs/testing/psd-server-project-a-manual.md` -- Project A report: `docs/testing/evidence/psd-server-project-a-report-template.md` -- Project B plan: `docs/plans/2026-08-20-psd-server-project-b-authentik.md` -- Project B human guide: `docs/testing/psd-server-project-b-manual.md` -- Project B report: `docs/testing/evidence/psd-server-project-b-report-template.md` - -The detailed evidence directory is a protected path on the server selected during the survey. The -repository receives only redacted reports after explicit owner review. - -## Owner-approved sequencing amendment — 2026-08-21 - -The Mac `rest_api` acceptance and revocation of `legacy-shared` move to a mandatory gate immediately -before Project B. The survey therefore records two distinct decisions: - -- `SURVEY_GO_PROJECT_A_PRIVATE`: technical prerequisite for requesting Project A private execution; -- `SURVEY_GO_PROJECT_B`: the complete shared-infrastructure decision, including Mac acceptance, - observation and legacy revocation. - -The amendment authorizes the read-only survey and static preparation of non-secret Project A -candidate facts and artifacts. While the current decision is `SURVEY_NO_GO`, it does not authorize -creating installation roots, cloning/building the candidate, creating protected configuration or -backup state, stopping the legacy stack, starting the new stack, changing public ingress, or -starting Project B. Those remain separate explicit gates after the scoped survey passes. - -### Task 1: Freeze the planning source - -**Files:** -- Read: `docs/plans/2026-08-20-psd-server-deployment-program-design.md` -- Record: protected server execution journal selected during the survey - -**Step 1: Verify the application checkout** - -Run: - -```bash -git status --short --branch -git rev-parse HEAD -git rev-parse origin/main -git show -s --format='%H%n%P%n%s' HEAD -``` - -Expected: the tree is clean and `HEAD` is the explicitly approved `origin/main` SHA. A newer SHA -than the design-time `5c0dc8c` is allowed only after recording and reviewing the intervening commits. - -**Step 2: Verify the plan files exist at that SHA** - -Run: - -```bash -test -f docs/plans/2026-08-20-psd-server-survey.md -test -f docs/plans/2026-08-20-psd-server-project-a-standalone.md -test -f docs/plans/2026-08-20-psd-server-project-b-authentik.md -git diff --check -``` - -Expected: every command exits zero. - -**Step 3: Record the immutable planning identity** - -Record the application SHA, plan commit, UTC timestamp, operator identity, and terminal-local access -method in the protected journal. Do not record CyberArk session secrets or screenshots. - -### Task 2: Execute and approve the common survey - -**Files:** -- Execute: `docs/plans/2026-08-20-psd-server-survey.md` -- Create from: `docs/testing/evidence/psd-server-survey-report-template.md` - -**Step 1: Execute every survey task without mutation** - -Expected: the survey identifies exact paths and owners for the old and new installations, Nginx, -load balancer, Aritmolab, Authentik, Supabase, the DWH, and protected credentials. - -**Step 2: Resolve every unknown** - -If an Authentik credential cannot be located, stop and ask the owner. If a configuration owner or -rollback boundary is unclear, stop; do not infer authority from file readability. - -**Step 3: Review the scoped survey GO/NO-GO** - -Expected: `SURVEY_GO_PROJECT_A_PRIVATE` requires a verified old-stack recovery path, an approved -new-installation root, enough resources, a direct read-only DWH path, workspace/model inputs, and -no unresolved mutation in the private Project A scope. Public-origin, load-balancer and Authentik -unknowns may remain explicitly deferred only while Project A is loopback-only and Task 10 is -omitted. `SURVEY_GO_PROJECT_B` retains the complete survey requirements. - -**Step 4: Checkpoint the survey** - -Hash the protected report and record only its path, SHA-256, timestamp, and GO result in the journal. - -### Task 3: Execute Project A - -**Files:** -- Execute: `docs/plans/2026-08-20-psd-server-project-a-standalone.md` -- Complete: `docs/testing/evidence/psd-server-project-a-report-template.md` - -**Step 1: Confirm the survey is GO for Project A private scope** - -Expected: the survey report hash matches the journal and no unresolved blocker remains inside the -Project A private scope. Before any stop/start, obtain a separate explicit owner authorization. - -**Step 2: Execute Project A task-by-task** - -Do not configure Authentik, change the production Aritmolab sidebar, or open the production route. - -**Step 3: Run the Project A human guide** - -Follow `docs/testing/psd-server-project-a-manual.md`. Record PASS/FAIL for every case; do not infer -manual PASS from automated output. The Mac REST row may be -`DEFERRED_PRE_PROJECT_B` only under the dated owner amendment. - -**Step 4: Close the Project A report** - -Expected: automated gates and the human guide are PASS; one harmless PSD session reached F8 and -produced validated read-only SQL; rollback remains available. The accepted report must list the -Mac REST item as an explicit deferred prerequisite rather than silently treating it as PASS. - -**Step 5: Obtain explicit owner approval** - -Record the approval and report digest. Project B remains forbidden without it. - -### Task 4: Freeze the Project B candidate - -**Files:** -- Read: accepted Project A report -- Record: protected server execution journal - -**Step 1: Recheck source and running images** - -Before freezing the candidate, close the pre-Project-B gate: validate the Mac installation with its -per-installation key, finish the 48-hour observation window including two 03:00 ETL cycles, revoke -`legacy-shared`, prove legacy `401` and v1 success, and obtain `SURVEY_GO_PROJECT_B`. - -Run the Project A plan's identity commands again. Record application SHA, workspace SHA, core image -ID, frontend image ID, Qdrant image digest, Ollama image digest, and local-auth configuration revision. - -Expected: all values match the accepted Project A report. - -**Step 2: Recheck rollback** - -Prove that the public route is still closed and the protected Project A configuration can be -selected without reconstructing it from memory. - -### Task 5: Execute Project B - -**Files:** -- Execute: `docs/plans/2026-08-20-psd-server-project-b-authentik.md` -- Complete: `docs/testing/evidence/psd-server-project-b-report-template.md` - -**Step 1: Execute Project B task-by-task** - -Keep public traffic closed until Authentik, Supabase migrations, ThothII diagnostics, Nginx, TLS, -and load-balancer preflight all pass. - -**Step 2: Run the Project B human guide** - -Follow `docs/testing/psd-server-project-b-manual.md` using approved ordinary and administrator -identities. The final path begins at the Aritmolab homepage and uses its existing sidebar link. - -**Step 3: Close the Project B report** - -Expected: SSO, roles, PostgreSQL ownership, the F1-F8 session, rollback rehearsal, and cleanup of the -temporary Project A endpoint all pass. - -### Task 6: Close the program - -**Files:** -- Modify: `PROJECT_STATE.md` -- Optionally create: reviewed redacted acceptance reports under `docs/testing/evidence/` - -**Step 1: Reconcile final state** - -Record final SHAs, image identities, workspace revision, Authentik object names/IDs (never secrets), -Supabase database/schema names, public origin, sidebar source revision, Nginx configuration identity, -and both report digests. - -**Step 2: Verify final negative boundaries** - -Expected: old stack stopped; Project A private endpoint removed; core/Qdrant/Ollama not externally -published; `thoth_sessions` absent from PostgREST exposed schemas; no secret appears in reports. - -**Step 3: Update project state** - -Add a dated factual section to `PROJECT_STATE.md`. Mark anything not actually run as PENDING. - -**Step 4: Run documentation checks** - -Run: - -```bash -git diff --check -bash scripts/auth-docs-smoke.sh -bash scripts/verify-workspace-install-docs.sh --fixtures-only -``` - -Expected: all checks pass. - -**Step 5: Commit only reviewed redacted documentation** - -```bash -git add PROJECT_STATE.md docs/testing/evidence -git diff --cached --check -git commit -m "docs: record PSD server deployment acceptance" -``` - -Expected: the commit contains no raw server inventory or secret material. diff --git a/docs/plans/2026-08-20-psd-server-project-a-standalone.md b/docs/plans/2026-08-20-psd-server-project-a-standalone.md deleted file mode 100644 index fd2e0882..00000000 --- a/docs/plans/2026-08-20-psd-server-project-a-standalone.md +++ /dev/null @@ -1,532 +0,0 @@ -# PSD Server Project A Standalone Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Install a clean PSD ThothII stack with local authentication, direct read-only DWH access, internal Qdrant/Ollama, rebuilt preprocessing, and one completed F1-F8 work session. - -**Architecture:** Keep the stopped legacy installation intact only as a temporary rollback boundary and deploy the current canonical five-service Compose stack from an adjacent clean clone. Create no host service identity: the image's UID/GID 10001 remains numeric and unmapped, confined to the new writable bind roots. A reviewed server-local override disables public exposure and uses filesystem work sessions; an optional load-balancer route is permitted only when it is operator-only and its denial boundary is proved first. - -**Tech Stack:** Git, Docker/Compose, native `tht`, local Argon2id authentication, PSD Supabase PostgreSQL direct transport, Qdrant, Ollama, Pi, Nginx/load-balancer test route where safe. - ---- - -## Preconditions - -- Common survey result is `SURVEY_GO_PROJECT_A_PRIVATE` and its digest is recorded. -- Every path below is replaced by the exact survey result before execution. -- No production Nginx/load-balancer/sidebar/Authentik change is in scope. -- The old stack remains running until its exact inventory and restart recipe are verified; old and - new stacks never run together. -- The server's workspace deploy credential remains read-only. A curator with write access publishes - the workspace change. -- Owner amendment 2026-08-21 declares legacy sessions/configuration disposable. Retain old resources - only until Project B proves the production Aritmolab journey, then delete them under a separate - exact cleanup authorization. No legacy backup is required. -- Do not create a host user or group for 10001. Immediately before preparing `/srv/thothii`, both - `getent passwd 10001` and `getent group 10001` must return no match. -- Owner amendment 2026-08-21 defers live Mac `rest_api` acceptance and `legacy-shared` revocation to - the mandatory pre-Project-B gate. It does not authorize stop/start; those require a later explicit - owner gate even after private preparation is complete. - -### Task 1: Freeze exact inputs - -**Files:** -- Read: protected survey report -- Read: `docs/plans/2026-08-20-psd-server-deployment-program-design.md` -- Record: protected Project A journal - -**Step 1: Record application identity** - -Run in the new planning checkout: - -```bash -git status --short --branch -git rev-parse HEAD -git rev-parse origin/main -git diff --check -``` - -Expected: clean and explicitly approved SHA. - -**Step 2: Record workspace remote identity** - -Use the surveyed read-only credential and run: - -```bash -git ls-remote refs/heads/main -``` - -Expected: one SHA recorded as the pre-change workspace revision. - -**Step 3: Check old-stack recoverability** - -Expected: exact old start/stop procedure, source SHA, Compose identity, volumes/binds, proxy closure -procedure, restart recipe, and shared-resource exclusions are present in the survey. Stop if any is -missing. - -### Task 2: Publish the multi-transport workspace revision - -**Files:** -- Modify in authorized curator clone: `psd-clinical/workspace.yaml` -- Verify: `thoth-workspaces.yaml` - -**Step 1: Create a clean curator branch** - -Run in a write-authorized clone, never in the application-managed registry checkout: - -```bash -git status --short --branch -git fetch origin main -git switch --create codex/psd-direct-transport origin/main -``` - -Expected: clean branch at the recorded remote SHA. - -**Step 2: Make the minimal descriptor change** - -Change exactly: - -```yaml -supported_transports: [rest_api] -``` - -to: - -```yaml -supported_transports: [rest_api, postgres_direct] -``` - -Do not duplicate the workspace or change its ID, collection, Evidence, annotations, model policy, -database, or schema. - -**Step 3: Review the descriptor-only diff** - -Run: - -```bash -git diff --check -git diff -- psd-clinical/workspace.yaml thoth-workspaces.yaml -``` - -Expected: one semantic line changed; catalog metadata remains identical. - -**Step 4: Validate with the current ThothII contract** - -Use a disposable installation/registry or the repository's current registry validation harness to -activate the candidate commit before publication. Expected: schema v3 accepts both transports, -Evidence and annotations materialize, and no secret is required for source validation. - -If no supported validator can be run in the curator environment, stop and request the owner to run -the established Mac validation; do not publish based only on YAML parsing. - -**Step 5: Commit and publish through curator review** - -```bash -git add psd-clinical/workspace.yaml -git diff --cached --check -git commit -m "feat: support direct PSD DWH transport" -git push --set-upstream origin codex/psd-direct-transport -``` - -Merge through the repository's normal review path. Record the resulting `main` SHA. - -**Step 6: Record the deferred Mac REST proof** - -Do not change the Mac during Project A. Record `DEFERRED_PRE_PROJECT_B`, the unchanged expected -transport `rest_api`, and the exact future diagnostics. The proof must become PASS before Project B, -after protected delivery/configuration of the per-installation key. - -### Task 3: Back up and stop the legacy installation - -**Files:** -- Create: surveyed protected legacy backup directory -- Record: Project A journal - -**Step 1: Capture final legacy state** - -Run the surveyed legacy status/doctor commands, record source SHA and image IDs, and confirm no -active user work. Do not use the new `tht` against an incompatible old descriptor. - -**Step 2: Close or maintenance-gate the old ThothII route** - -Change only the surveyed ThothII-specific route using its established mechanism. Validate Nginx and -load-balancer configuration before applying. Confirm external requests no longer reach the app. - -**Step 3: Record the disposable legacy boundary** - -Record exact container and image IDs plus filesystem device/inode/ownership/size for -`/home/chirone/ThothII` and `/home/chirone/thothii-data`. Do not archive their contents: the owner -declared them disposable. Prove that the external Evidence bind and both shared Docker networks -are excluded from any later cleanup manifest. - -**Step 4: Stop the old stack** - -Use its own supported controller. Expected: old containers stopped, not removed; volumes and bind -trees unchanged. - -**Step 5: Rehearse the restart command without executing it** - -Record the exact command, preconditions, port ownership, and route-restoration order. If it cannot -be stated unambiguously, stop before creating the new stack. - -### Task 4: Prepare the adjacent clean installation - -**Files:** -- Create: survey-selected new source root -- Create: survey-selected operator, secret, data, Pi-state, registry, and backup roots - -**Step 1: Create dedicated paths without creating identities** - -Follow `docs/install/server.md` numeric-ownership rules. Do not call `useradd`, `groupadd`, or -`usermod`. The existing operator owns source/operator files; only the new data, Pi-state, and -workspace-registry roots use unmapped numeric `10001:10001`. Stop if UID or GID 10001 resolves to a -host account, and do not change the image identity without a reviewed design amendment. - -**Step 2: Clone the frozen application source** - -```bash -git -c core.autocrlf=false clone /ThothII -git -C /ThothII config --local core.autocrlf false -git -C /ThothII switch --detach -git -C /ThothII status --short --branch -``` - -Expected: detached exact SHA, clean tree. - -**Step 3: Verify source and platform** - -```bash -cd /ThothII -bash scripts/verify-line-endings.sh -docker version -docker compose version -``` - -Expected: all pass. - -**Step 4: Prepare Pi state and build the operator** - -```bash -sudo scripts/prepare-server-pi-state.sh 10001 10001 -THT_THT_OUTPUT_DIRECTORY= bash scripts/build-tht.sh -``` - -Install only the binary matching the surveyed server architecture. Run `tht version --json` and -record its source identity. - -### Task 5: Create the protected Project A configuration - -**Files:** -- Create outside Git: `/server.env` -- Create outside Git: `/thothii-installation.yaml` -- Create outside Git: `/project-a-private.yaml` -- Create outside Git: `/auth.yaml` through `tht` - -**Step 1: Start from current examples** - -Copy `deploy/env/server.env.example` and `docs/install/examples/thothii-installation.server.yaml` -to the protected Project A operator root. Replace every placeholder with surveyed absolute paths. -Never source `server.env` as shell code. - -**Step 2: Add the private/local-session override** - -Create this reviewed override: - -```yaml -services: - core: - environment: - THOTH_PUBLIC_EXPOSURE: "false" - THT_SESSION_STORAGE: local - frontend: - ports: !override - - "127.0.0.1::8080" -``` - -Select an unused loopback port proved by `ss -lntp`. Do not publish core, Qdrant, or Ollama. - -**Step 3: Compose the installation descriptor** - -Use `profile: server`, the exact new source root/env/auth root, workspace remote/branch/read-only -access, Project A override, and exactly one Git transport override. Do not include the public -session-server overlay in Project A. - -For the projected local-auth descriptor, keep `authentication.configDirectory` as the canonical -root and add `runtimeProjection` with a distinct absolute runtime directory plus numeric `uid: 10001` -and `gid: 10001`. The descriptor loader includes the automatic runtime-projection override; do -not list it manually under `overrides`. The canonical root stays `root:root 0700/0600`; the -publisher owns the projection numerically as `10001:10001 0700/0600`. Only the projection is -mounted read-only into core. Do not create a host user/group, edit `CURRENT` or `generations`, or -apply these paths before the separately authorized start gate. - -**Step 4: Validate permissions and render** - -```bash - --installation update --check-only -``` - -Expected: Compose validates; only frontend has a loopback port; core declares public exposure false -and local session storage; Qdrant/Ollama are internal. - -**Step 5: Configure the local administrator** - -Create a temporary mode-0600 password file using an echo-free prompt, then run: - -```bash - --installation auth configure \ - --mode local --public-url \ - --admin-user --admin-display-name \ - --password-file -``` - -Remove the temporary input file after success and record that removal. Do not delete generated -`auth.yaml` or `users.yaml`. - -Before the later start gate, run the redacted projected status command and require `ready` plus -`equal: true`. A blocked result prevents start. `auth publish` is the only repair path: it rebuilds -from canonical authentication, including after a candidate or recovery restore outcome; it never -promotes a retained runtime generation on its own. - -### Task 6: Build and start the clean stack - -**Files:** -- Record: Project A evidence directory - -**Step 1: Build current images** - -```bash -cd /ThothII -bash scripts/build-local.sh -``` - -Expected: current core/frontend images build; pinned Qdrant/Ollama references resolve. - -**Step 2: Run preflight** - -```bash - --installation update --check-only - --installation pi doctor -``` - -Expected: no mutation error and no secret in output. - -**Step 3: Start through `tht`** - -```bash - --installation start --build - --installation status - --installation doctor --json - --installation pi test -``` - -Expected: frontend, core, Qdrant, embedding healthy and model initializer completed. Doctor has the -documented ordered checks and authentication PASS. - -**Step 4: Verify listener boundaries** - -Use `ss -lntp` and bounded Docker inspection. Expected: only the selected frontend loopback port is -host-published; no external core, Qdrant, or Ollama listener. - -### Task 7: Activate the workspace and direct DWH binding - -**Files:** -- Modify only through authenticated Workspace Management: encrypted workspace secret store - -**Step 1: Pull and inspect the reviewed workspace revision** - -```bash - --installation \ - workspace inspect --workspace psd-clinical --json -``` - -Expected: active workspace SHA equals the approved multi-transport revision. - -**Step 2: Configure runtime bindings** - -Through the authenticated Workspace Management API/UI, select `postgres_direct` and provide the -surveyed host, port, runtime user, password, and optional TLS CA. Secret values go to the encrypted -vault; they do not enter `server.env`, Git, shell arguments, or evidence. - -The generated contract names are: - -```text -THT_WS_PSD_CLINICAL_DWH_TRANSPORT -THT_WS_PSD_CLINICAL_DWH_HOST -THT_WS_PSD_CLINICAL_DWH_PORT -THT_WS_PSD_CLINICAL_DWH_USER -THT_WS_PSD_CLINICAL_DWH_PASSWORD_FILE -THT_WS_PSD_CLINICAL_DWH_TLS_CA_FILE -``` - -**Step 3: Validate and test connections** - -Run static validation, live connection test, workspace inspect, and `doctor --json`. Expected: DWH, -workspace, internal embedding, and Qdrant checks pass with redacted output. - -**Step 4: Re-prove read-only grants** - -Use the survey's catalog query through the exact configured identity. Expected: no DML/DDL grant on -`datawarehouse`. Stop if the runtime user is an owner, superuser, or write-capable role. - -### Task 8: Rebuild and verify semantic preprocessing - -**Files:** -- Create: Project A preprocessing evidence - -**Step 1: Inspect the empty/new collection state** - -```bash - --installation \ - workspace vector inspect --workspace psd-clinical --json -``` - -Expected: either a compatible empty collection or the documented missing-collection state. - -**Step 2: Create the descriptor-owned collection when missing** - -Use the guarded vector rebuild only for collection `psd-clinical`, with exact repeated confirmation -and `--destroy`. Do not run it against any other collection. - -**Step 3: Run complete preprocessing** - -```bash - --installation \ - workspace preprocess run --workspace psd-clinical --json -``` - -If it returns `manual_review_required`, inspect the exact run and curated annotations, obtain the -required human decision, run `workspace schema accept --workspace psd-clinical --run --yes`, -then resume the same run. Never auto-approve unknown FK changes. - -**Step 4: Verify collection contract and counts** - -Run vector inspection and record dimensions, cosine distance, required keyword indexes, and bounded -counts by payload kind/revision. Expected: all points carry the active workspace revision. - -**Step 5: Prove idempotency** - -Run the complete preprocessing command again. Expected: no new review, no duplicate logical points, -unchanged Evidence reported as unchanged, and the same effective configuration identity. - -### Task 9: Configure and test local users - -**Files:** -- Modify through `tht auth user`: protected local user registry - -**Step 1: Add an ordinary test user** - -Use an echo-free prompt or protected temporary password file: - -```bash - --installation auth user add \ - --role user --display-name --password-file -``` - -Remove the temporary input file after success. - -**Step 2: Run authentication diagnostics** - -```bash - --installation auth status --json - --installation auth check --json -``` - -Expected: pristine redacted JSON and PASS. - -**Step 3: Execute automated local-auth cases** - -Use same-origin requests or a local headless browser to prove ordinary/admin authorization, generic -wrong-password failure, disable/enable, password/role revision invalidation, logout-all, CSRF, and -remembered-session survival after core restart. Do not retain cookie jars after the test. - -### Task 10: Optionally add the private network-path test - -**Current scope boundary (owner, 2026-08-21):** omit this entire task and keep Project A -loopback-only. Any future use requires a separate shared-infrastructure authorization after the -public-origin and load-balancer activities pass; the Project A private survey decision alone is -insufficient. - -**Files:** -- Modify only surveyed test-specific load-balancer/Nginx files -- Create: test certificate through the existing managed mechanism - -**Step 1: Prove allowlist capability before proxying** - -Create a temporary hostname that returns a fixed maintenance response. From an approved operator -source expect success; from an unapproved source expect denial. Do not point it at ThothII yet. - -**Step 2: Validate and activate the test proxy** - -Configure Nginx with the same Host/HTTPS forwarding and SSE settings intended for production. Run -`nginx -t`, validate the load balancer, then reload through the established mechanism. - -**Step 3: Reconfigure local-auth public URL transactionally** - -If the exact private HTTPS origin differs from the loopback origin, use the supported authentication -configuration workflow and invalidate prior test sessions. Re-run auth and doctor checks. - -**Step 4: Prove both sides** - -Expected: authorized operator reaches the local login; unauthorized source remains denied before -ThothII. If this cannot be demonstrated, remove the test route and continue on loopback. - -### Task 11: Complete the F1-F8 acceptance session - -**Files:** -- Complete: `docs/testing/psd-server-project-a-manual.md` -- Create: protected session evidence - -**Step 1: Select the approved harmless question** - -Use a known read-only PSD question agreed by the owner. Record the wording in the protected report; -do not include patient-identifying values. - -**Step 2: Create the session as the ordinary local user** - -Use the private browser route when present; otherwise drive the same-origin frontend/API from the -server-local terminal/headless browser. Record only session ID and sanitized milestones. - -**Step 3: Review every gate** - -Complete F1-F8 without auto-confirming human decisions. Confirm persisted phase/artifact state after -each gate and resume once to prove recovery. - -**Step 4: Validate final SQL** - -Expected: finalized session, DWH validation PASS, SQL is read-only, and no clinical mutation occurs. - -**Step 5: Inspect persisted state** - -Confirm manifest, question, schema linking, Evidence, CTE plan/tests, final SQL, validation report, -and decision ledger exist in local filesystem session storage. Chat/SSE need not persist. - -### Task 12: Close Project A and preserve rollback - -**Files:** -- Complete: `docs/testing/evidence/psd-server-project-a-report-template.md` - -**Step 1: Run final diagnostics** - -Run status, doctor, auth check, workspace inspect, vector inspect, Pi test, and a bounded secret scan -of the intended report. - -**Step 2: Create a transactional new-installation backup** - -Use `tht backup --drain` with a protected explicit output. Verify its checksum. Do not include -secrets in the ordinary evidence archive. - -**Step 3: Complete human acceptance** - -Every private-server row in `docs/testing/psd-server-project-a-manual.md` must be PASS or explicitly -blocking. Only the Mac REST row may be `DEFERRED_PRE_PROJECT_B` under the dated owner amendment. - -**Step 4: Record the gate** - -Record exact SHAs/images, workspace revision, preprocessing identity/counts, session ID, report -digest, rollback status, and explicit `PROJECT_A_PRIVATE_PASS` or `PROJECT_A_FAIL`. A private PASS -does not authorize Project B while the deferred gate remains open. - -**Step 5: Stop on FAIL** - -On FAIL, stop the new stack and use the surveyed old-stack recovery plan if service restoration is -desired. Do not start Project B. diff --git a/docs/plans/2026-08-20-psd-server-project-b-authentik.md b/docs/plans/2026-08-20-psd-server-project-b-authentik.md deleted file mode 100644 index 2baf0e1e..00000000 --- a/docs/plans/2026-08-20-psd-server-project-b-authentik.md +++ /dev/null @@ -1,442 +0,0 @@ -# PSD Server Project B Authentik Integration Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Convert the accepted Project A installation to public OIDC mode, store owned work sessions in the existing Supabase database's `thoth_sessions` schema, and restore the established Aritmolab-sidebar user journey through the load balancer and Nginx. - -**Architecture:** Keep the same installation-descriptor path and Compose project so Project A Qdrant/Ollama volumes and bind state remain authoritative. Close ingress, snapshot Project A configuration, configure Authentik and Supabase, replace the protected installation configuration transactionally at the same paths, validate privately, then open the production route and sidebar link. - -**Tech Stack:** Authentik OAuth2/OIDC Authorization Code + PKCE, ThothII OIDC/session store, Supabase PostgreSQL schema migrations/RLS, Docker Compose, Nginx, load balancer, Aritmolab. - ---- - -## Preconditions - -- Project A automated and human reports are PASS and explicitly owner-approved. -- The Mac `rest_api` installation passes source validation and connection diagnostics with its - per-installation key. -- The dual-key observation has lasted at least 48 hours and includes two scheduled 03:00 ETL cycles. -- `legacy-shared` is revoked; v1 remains successful and the legacy credential is proven `401`. -- The current survey decision is `SURVEY_GO_PROJECT_B`, not only the private Project A decision. -- Application SHA, workspace SHA, images, Project A report digest, and rollback configuration match - the accepted evidence. -- The production route is closed before authentication/session-storage changes. -- All Authentik operations use the installed version's API/OpenAPI contract. Official current - references include [OAuth2/OIDC providers](https://docs.goauthentik.io/add-secure-apps/providers/oauth2/), - [provider property mappings](https://docs.goauthentik.io/add-secure-apps/providers/property-mappings/), - [application bindings](https://docs.goauthentik.io/add-secure-apps/applications/manage_apps/), and - [blueprint export](https://docs.goauthentik.io/customize/blueprints/export); installed-version - behavior wins over newer documentation. - -### Task 1: Freeze Project A and close ingress - -**Files:** -- Read: accepted Project A report -- Create: protected Project B transaction root - -**Step 1: Verify exact Project A state** - -Run status, doctor, auth check, workspace inspect, vector inspect, Pi status/test, Git SHA, and image -identity checks from Project A. Expected: all match the accepted report. - -**Step 2: Create a protected transaction root** - -Use `mktemp -d` under the survey-approved protected parent, mode `0700`. Record its path and do not -place it in Git. - -**Step 3: Close production and temporary ingress** - -Keep or restore a maintenance response at the production ThothII route. Disable the optional -Project A test route before changing authentication unless it is needed for a separately approved -private preflight. Confirm neither route reaches ThothII. - -**Step 4: Stop and back up Project A** - -```bash - --installation backup \ - --output /project-a-backup.tar --drain - --installation stop -``` - -Verify the archive using the controller's manifest/checksum contract. Preserve a copy of the exact -installation descriptor, env file, override files, auth directory, Nginx fragment, load-balancer -route, Aritmolab sidebar file/revision, and relevant Authentik export metadata. - -### Task 2: Decide exact Authentik names and roles - -**Files:** -- Create: protected `authentik-change-manifest.yaml` - -**Step 1: Select the final public origin** - -Use the live survey result, not historical `.it`/`.com` assumptions. Record exactly one HTTPS origin -and callback `/api/auth/oidc/callback`. - -**Step 2: Select exact groups** - -Default to dedicated `TOT Users` and `TOT Admin`. Reuse existing groups only if their membership -semantics match and the owner approves. Record exact case-sensitive names. - -**Step 3: Define least privilege** - -Map user group → `user`, admin group → `admin`. Define a separate service account/token with only -the installed Authentik permission needed to view exact group objects. No write, user-management, -directory-administration, or superuser permission. - -**Step 4: Obtain owner approval of the manifest** - -The manifest contains object names, slugs, intended bindings, callback, scopes, grant types, -credential destinations, and rollback action—but no secret values. Do not mutate Authentik before -approval. - -### Task 3: Export and prepare Authentik - -**Files:** -- Create: protected pre-change Authentik export -- Modify: Authentik objects named in the approved manifest - -**Step 1: Export relevant configuration** - -Use the installed version's supported blueprint/API export. A worker command such as -`ak export_blueprint` is valid only if present in that version. Protect export mode `0600`; remember -write-only provider secrets are not included, so backup their custody separately without printing. - -**Step 2: Verify API credential scope** - -Use a read-only call to list relevant groups/applications. Expected: administrative creation access -for the setup identity and a distinct path for the future group-view service account. Stop if the -credential is missing or ambiguous. - -**Step 3: Create or confirm exact groups** - -Create missing dedicated groups, or record approved existing group IDs. Do not bulk-copy LDAP or -unrelated Authentik memberships. - -**Step 4: Create the group-catalog service account** - -Grant only exact group-view permission. Create its token through the approved protected-secret -mechanism; write it directly to the ThothII secret destination without displaying it. - -**Step 5: Create the OIDC provider** - -Configure a confidential OAuth2/OIDC provider with the exact callback, issuer mode observed as -appropriate, Authorization Code, PKCE support, and Device Code only when required for -`tht auth check --interactive` and supported by the installed release. Do not enable implicit flow. - -**Step 6: Configure scopes and direct groups claim** - -Select `openid`, `profile`, and `email`. Inspect a disposable identity's decoded claim keys through -a protected verifier; retain only a redacted shape. Expected: ID token includes direct non-empty -`groups: [string, ...]`. - -If the installed default profile mapping already provides that exact claim, reuse it. Otherwise add -a provider scope/property mapping under the requested `profile` scope that returns: - -```python -return {"groups": [group.name for group in request.user.ak_groups.all()]} -``` - -Verify the installed mapping merge semantics before activation. Do not add a custom unrequested -scope because ThothII requests only `openid`, `profile`, and `email`. - -**Step 7: Create the Authentik application** - -Bind it to the provider. Configure display metadata according to local Aritmolab conventions. Do not -use an Authentik proxy provider or Nginx forward-auth for ThothII. - -**Step 8: Create and store the client secret** - -Write the client secret directly into the protected ThothII secret bundle key -`THT_OIDC_CLIENT_SECRET`. Store the service-account token as `THT_AUTHENTIK_API_TOKEN`. Never place -either value in the change manifest, shell history, Compose environment, or evidence. - -### Task 4: Prepare Supabase schema roles and backup - -**Files:** -- Read: `harness/tht/migrations/sessions/001_schema.sql` -- Read: `harness/tht/migrations/sessions/002_security.sql` -- Create: protected Supabase backup/evidence -- Create: runtime and migrator credential files - -**Step 1: Confirm the database/schema boundary** - -Expected: use the surveyed existing Supabase PostgreSQL database; clinical data remains in -`datawarehouse`; application sessions use schema `thoth_sessions`; no new database is created. - -**Step 2: Back up database metadata/data consistently** - -Use the existing Supabase/PostgreSQL backup procedure before DDL. Record backup ID, timestamp, -checksum, and restore command. Do not put a dump in the Git repository. - -**Step 3: Create or validate dedicated roles** - -Create one migrator login and one runtime login according to the migration contract. The runtime -role must not be superuser, owner, BYPASSRLS, CREATEROLE, CREATEDB, or a member of DWH write roles. -The migrator credential remains unavailable to core. - -**Step 4: Write protected credential files** - -Create separate mode-0640 runtime-password, migrator-password, and session-CA files with surveyed -ownership. Do not use command-line password arguments. - -**Step 5: Confirm PostgREST exclusion before migration** - -Record the exact exposed schema list. Expected: `thoth_sessions` absent. If the system exposes all -schemas implicitly, stop and resolve the boundary before migration. - -### Task 5: Prepare the stable Project B installation configuration - -**Files:** -- Modify at the same stable paths: operator env, installation descriptor, authentication directory -- Create: reviewed session-server override copied from `deploy/compose.session-server.yaml.example` -- Create: protected server-session workspace config copied from `deploy/workspaces/server-sessions.yaml.example` - -**Step 1: Preserve the Compose project name** - -The native controller derives the project name from the absolute installation-descriptor path. -Keep that exact path. Do not point Project B at a second descriptor path, because that would create -new Qdrant/Ollama named volumes instead of using the Project A accepted state. - -**Step 2: Stage Project B files beside the live files** - -Prepare new env/descriptor/override/auth inputs in the protected transaction root. Add the session -DB host/port/existing database name/runtime role/migrator role/TLS mode and three secret source -paths. Use `verify-full` where hostname/SAN permits; any `verify-ca` exception requires explicit -survey evidence and owner approval. - -**Step 3: Add the server-session override** - -Copy the current example to a reviewed local file and add it to the existing stable descriptor's -overrides before the Git transport override ordering required by the installation. Do not edit the -tracked example. - -**Step 4: Replace local auth state transactionally** - -With the stack stopped, move the complete Project A auth directory into the protected transaction -root, recreate an empty private directory at the same path/ownership/mode, and configure OIDC: - -```bash - --installation auth configure \ - --mode oidc --public-url \ - --issuer --client-id \ - --authentik-base-url \ - --user-group '' --admin-group '' -``` - -Expected: non-secret `auth.yaml` only; secrets resolved from the protected bundle. - -**Step 5: Atomically install staged path-only files** - -Use same-filesystem rename and preserve required ownership/mode. Keep the Project A originals in -the transaction root. Run `update --check-only`; on failure restore the originals immediately. - -### Task 6: Run and verify session migrations - -**Files:** -- Modify through one-shot migrator: existing database schema `thoth_sessions` - -**Step 1: Validate migration rendering** - -```bash - --installation update --check-only -``` - -Expected: core and `session-migrate` resolve the same core image; core lacks migrator password; -only the one-shot service sees it. - -**Step 2: Run migrations once** - -```bash - --installation sessions migrate --yes -``` - -Expected JSON: `"pending":[]` and `"drifted":[]`; `applied` may list `001` and `002` on first use. - -**Step 3: Run migration status/idempotency again** - -Run the same command. Expected: no new application and both pending/drifted remain empty. - -**Step 4: Verify database security** - -Through bounded catalog queries, prove forced RLS, policies on session tables, runtime role without -BYPASSRLS/ownership/DDL, migrator absent from core, and no runtime privileges on unrelated schemas. - -**Step 5: Recheck PostgREST exclusion** - -Expected: `thoth_sessions` still absent from exposed schemas and REST endpoints cannot address it. - -### Task 7: Validate Authentik and start privately - -**Files:** -- Record: Project B protected evidence - -**Step 1: Run static configuration validation** - -Run `update --check-only` and redacted `auth status --json`. Expected: mode OIDC, exact public origin, -issuer/client ID/group names, and no secret values. - -**Step 2: Start while public ingress remains closed** - -```bash - --installation start - --installation status - --installation auth check --json - --installation doctor --json - --installation pi test -``` - -Expected: OIDC discovery/issuer/JWKS, client-secret access, Authentik catalog token, exact mapped -groups, PostgreSQL session storage, workspace, services, workflow, and Pi pass. - -**Step 3: Run interactive device check when supported** - -```bash - --installation auth check --interactive -``` - -Expected: approved identity completes Device Authorization and direct groups claim validates. If -the installed provider does not support device flow, record PENDING rather than substituting a token. - -### Task 8: Prepare Nginx, TLS, load balancer, and sidebar - -**Files:** -- Modify only survey-approved ThothII Nginx fragment -- Modify only survey-approved load-balancer route -- Modify only exact Aritmolab sidebar source when its target must change - -**Step 1: Prepare direct-OIDC Nginx configuration** - -Follow `docs/install/reverse-proxy-nginx.md`, direct OIDC section. Required behavior: no -`auth_request`, no callback rewrite, frontend loopback upstream, Host and HTTPS forwarded headers, -HTTP/1.1, buffering/cache off, long SSE read timeout, and `X-Accel-Buffering: no`. - -**Step 2: Validate the managed certificate** - -Expected: SAN matches final hostname, validity is current, chain is trusted by approved clients, -private-key permissions match local policy, and renewal/generation ownership is recorded. Never -copy the key into ThothII. - -**Step 3: Validate Nginx without opening traffic** - -```bash -sudo nginx -t -curl --fail http://127.0.0.1:/health -``` - -Use local `--resolve`/Host tests only when they do not bypass the identity behavior being tested. - -**Step 4: Prepare the load-balancer route** - -Configure backend/health/TLS according to the surveyed owner procedure, initially disabled or -operator-only. Confirm it targets Nginx, never core/Qdrant/Ollama directly. - -**Step 5: Preserve the Aritmolab link contract** - -If the existing sidebar target already equals the final origin/path, leave source unchanged and -record proof. Otherwise make the smallest reviewed change, test it in Aritmolab's own test/build -system, and commit in that repository before deployment. - -### Task 9: Open ingress and run OIDC acceptance - -**Files:** -- Complete: `docs/testing/psd-server-project-b-manual.md` - -**Step 1: Reload Nginx through the established mechanism** - -Run `nginx -t` immediately before reload. Expected: reload succeeds and unrelated virtual hosts -remain healthy. - -**Step 2: Enable the final load-balancer route** - -Expected: HTTP redirects to HTTPS; TLS is valid; `/api/auth/oidc/login` redirects to the correct -Authentik provider; callback returns to the exact public origin. - -**Step 3: Test ordinary and admin identities** - -Start at Aritmolab, authenticate once, then use the sidebar. Expected: no second credential prompt; -ordinary user can use sessions but receives 403 for admin operations; admin has only documented -permissions. - -**Step 4: Test no-role and malformed cases** - -An identity with no mapped group authenticates but receives no application role/403. Missing, -malformed, indirect, or ambiguous group claims fail closed with generic browser errors and redacted -diagnostics. Do not retain raw claims. - -**Step 5: Test logout and restart** - -Verify ThothII logout revokes its own cookie. Document whether the Authentik SSO session remains and -therefore allows immediate re-login without credentials; do not claim global logout unless -configured and tested. Restart core and verify expected OIDC session behavior. - -### Task 10: Verify PostgreSQL ownership and complete F1-F8 - -**Files:** -- Create: protected Project B session evidence - -**Step 1: Create sessions under two identities** - -Expected: ordinary users see only their own sessions; cross-user access returns the documented -not-found boundary; admin behavior matches `session.read_all/manage_all` permissions. - -**Step 2: Verify RLS with the runtime path** - -Use application/API tests and bounded catalog evidence. Never disable RLS for diagnosis. - -**Step 3: Complete one harmless OIDC PSD session** - -Use the same approved read-only question or another owner-approved one. Complete F1-F8, validate -final SQL, resume once, and confirm session/artifacts/decisions are stored in `thoth_sessions`. - -**Step 4: Verify ephemeral boundaries** - -Expected: no chat transcript or SSE stream stored as session artifacts; no vectors in PostgreSQL; -Qdrant remains the semantic store. - -### Task 11: Test controlled failures and rollback - -**Files:** -- Record: protected rollback evidence - -**Step 1: Test a reversible provider/catalog failure** - -Use a controlled, owner-approved method such as a temporary disabled test credential or test object. -Expected: auth diagnostics and browser login fail closed, redacted, then pass after restoration. -Never break unrelated Authentik applications. - -**Step 2: Rehearse ingress-first rollback** - -Close the production route, validate Nginx restoration commands, and prove the protected Project A -configuration snapshot is complete. A full rollback need not destroy `thoth_sessions`. - -**Step 3: Verify additive database rollback boundary** - -Expected: rollback leaves schema/data intact for evidence and future recovery. No automatic DROP -SCHEMA or role deletion. - -### Task 12: Close Project B - -**Files:** -- Complete: `docs/testing/evidence/psd-server-project-b-report-template.md` - -**Step 1: Run final diagnostics** - -Run status, doctor, auth check, interactive check when supported, workspace inspect, vector inspect, -Pi test, Nginx validation, load-balancer health, Supabase migration/security checks, and sidebar test. - -**Step 2: Remove the Project A temporary endpoint** - -Remove only its load-balancer route, Nginx fragment, and managed certificate reference according to -their owners. Validate/reload and prove the hostname no longer routes. - -**Step 3: Complete the human guide and report** - -Every mandatory row must be PASS. Record exact source/image/workspace identities, Authentik object -names/IDs, Supabase database plus `thoth_sessions`, public origin, Aritmolab revision, report digest, -and `PROJECT_B_PASS` or `PROJECT_B_FAIL`. - -**Step 4: Handle FAIL safely** - -On FAIL, close ingress first. Restore Project A files at the same stable paths, move OIDC auth state -to protected evidence, restore the local-auth directory, validate, and start Project A privately. -Disable new Authentik objects; do not delete them or drop the session schema automatically. diff --git a/docs/plans/2026-08-20-psd-server-survey.md b/docs/plans/2026-08-20-psd-server-survey.md deleted file mode 100644 index 963ec60e..00000000 --- a/docs/plans/2026-08-20-psd-server-survey.md +++ /dev/null @@ -1,312 +0,0 @@ -# PSD Server Survey Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Produce a non-mutating, redacted survey of the PSD server that resolves every path, owner, network boundary, credential location, and rollback prerequisite needed by Projects A and B. - -**Architecture:** Collect bounded metadata from the terminal local to the server, retain raw output only in a protected directory, and summarize it in a redacted report. The survey makes no service, file, database, proxy, Authentik, or Git mutation. - -**Tech Stack:** Linux utilities, Docker/Compose inspection, Git, Nginx, OpenSSL, PostgreSQL/Supabase metadata queries, Authentik metadata/API discovery, Aritmolab source inspection. - ---- - -## Safety contract - -- Do not run `docker inspect` without a restrictive Go template; its default output can contain secrets. -- Do not run `docker compose config` into chat or a public log. Store raw output mode `0600`, then create a redacted derivative. -- Do not print process environments, secret-file contents, private keys, cookies, tokens, password hashes, or raw OIDC claims. -- Do not reload/restart services, fetch/pull Git, log in interactively, change file modes, or make API mutations. -- When a command needs privilege, use the server's approved CyberArk/local-terminal procedure. - -### Task 1: Create the protected survey workspace - -**Files:** -- Create: `/var/tmp/thothii-psd-survey./` -- Create: protected `survey-report.md` - -**Step 1: Create a private temporary root** - -Run: - -```bash -umask 0077 -PSD_SURVEY_ROOT="$(mktemp -d /var/tmp/thothii-psd-survey.XXXXXX)" -test -d "$PSD_SURVEY_ROOT" -chmod 0700 "$PSD_SURVEY_ROOT" -printf '%s\n' "$PSD_SURVEY_ROOT" -``` - -Expected: one new mode-0700 directory whose exact path is recorded in the operator journal. - -**Step 2: Copy the report template** - -Create `$PSD_SURVEY_ROOT/survey-report.md` from the headings in the design's Common Survey section. -Record only findings and references to protected raw files. - -### Task 2: Record host and Docker facts - -**Files:** -- Create: `$PSD_SURVEY_ROOT/host.txt` -- Create: `$PSD_SURVEY_ROOT/docker-projects.json` -- Create: `$PSD_SURVEY_ROOT/docker-containers.txt` - -**Step 1: Record bounded host metadata** - -Run each command with output redirected to `$PSD_SURVEY_ROOT/host.txt`: - -```bash -date -u '+%Y-%m-%dT%H:%M:%SZ' -uname -a -cat /etc/os-release -getconf LONG_BIT -nproc -free -h -df -hT -docker version -docker compose version -``` - -Expected: no credential content and enough capacity information to judge a parallel source tree and -a new five-service stack. - -**Step 2: Record Compose projects and bounded container identity** - -Run: - -```bash -docker compose ls --format json > "$PSD_SURVEY_ROOT/docker-projects.json" -docker ps -a --no-trunc --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-containers.txt" -docker network ls --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-networks.txt" -docker volume ls --format '{{json .}}' > "$PSD_SURVEY_ROOT/docker-volumes.txt" -``` - -Expected: inventory only. Do not inspect full container JSON. - -**Step 3: Identify candidate legacy ThothII containers** - -Use names, images, Compose project labels, published ports, and health from the bounded inventory. -For each candidate, query only these templates: - -```bash -docker inspect --format '{{.Name}} {{.Config.Image}} {{index .Config.Labels "com.docker.compose.project"}} {{index .Config.Labels "com.docker.compose.project.working_dir"}}' -docker inspect --format '{{json .NetworkSettings.Networks}}' -docker inspect --format '{{range .Mounts}}{{.Type}} {{.Source}} -> {{.Destination}} rw={{.RW}}{{println}}{{end}}' -``` - -Expected: exact source/Compose ownership and mounts without environment values. - -### Task 3: Survey the legacy ThothII installation - -**Files:** -- Create: `$PSD_SURVEY_ROOT/legacy-thothii.txt` -- Create: `$PSD_SURVEY_ROOT/legacy-compose.redacted.yaml` - -**Step 1: Resolve source and operator paths from evidence** - -Do not search broad filesystem roots. Derive paths from Compose labels, systemd units, Nginx -upstreams, and known operator documentation. Record uncertainty rather than guessing. - -**Step 2: Record source identity without fetching** - -Run in the identified source tree: - -```bash -git status --short --branch -git rev-parse HEAD -git remote -v -git log -5 --oneline --decorate -``` - -Expected: no mutation. Mark a dirty tree as NO-GO until the owner decides how to preserve it. - -**Step 3: Record lifecycle state with the legacy controller** - -If the old installation has a supported `tht`, run its bounded `status`, `doctor`, `pi status`, and -`pi maintenance status` commands. Otherwise record exact read-only Docker health and identify the -old lifecycle mechanism. Do not substitute current `tht` against an incompatible descriptor. - -**Step 4: Render and redact Compose safely** - -Store the raw render as mode `0600`. Replace secret-bearing scalar values with `[redacted]` before -using the derivative in analysis. Confirm the redacted render still shows service names, networks, -ports, volumes, image/build identities, and config file paths. - -### Task 4: Survey Nginx, TLS, and the load balancer - -**Files:** -- Create: `$PSD_SURVEY_ROOT/nginx.raw.txt` (protected) -- Create: `$PSD_SURVEY_ROOT/nginx-thothii.redacted.txt` -- Create: `$PSD_SURVEY_ROOT/tls-metadata.txt` -- Create: `$PSD_SURVEY_ROOT/load-balancer.md` - -**Step 1: Validate and capture Nginx without reload** - -Run: - -```bash -sudo nginx -t -sudo nginx -T > "$PSD_SURVEY_ROOT/nginx.raw.txt" 2>&1 -chmod 0600 "$PSD_SURVEY_ROOT/nginx.raw.txt" -``` - -Expected: configuration test passes. Do not reload Nginx. - -**Step 2: Extract only relevant directives** - -Create the redacted derivative containing the ThothII/Aritmolab `server_name`, `listen`, `location`, -`proxy_pass`, `proxy_set_header`, `proxy_buffering`, timeout, certificate path, and include-file -directives. Exclude unrelated virtual hosts and all authorization values. - -**Step 3: Record certificate metadata only** - -For each relevant public certificate—not its key—run: - -```bash -openssl x509 -in -noout -subject -issuer -serial -dates -ext subjectAltName -``` - -Expected: exact SAN/expiry/issuer and the observed generation/renewal mechanism. - -**Step 4: Map the load balancer** - -Record its owner, configuration surface, current Aritmolab backend, health check, TLS boundary, -source addresses seen by Nginx, and whether it can enforce a temporary hostname allowlist. Do not -create a route. If Sol cannot inspect it, name the human/team required for Project A/B gates. - -### Task 5: Survey Aritmolab and the sidebar integration - -**Files:** -- Create: `$PSD_SURVEY_ROOT/aritmolab.md` - -**Step 1: Resolve the Aritmolab source/deployment** - -Use Compose labels, Nginx paths, or the documented service unit. Record repository path, SHA, dirty -state, deployment command, container/network identity, and configuration owner. - -**Step 2: Locate the sidebar link** - -Use `rg` in the resolved source tree for the current ThothII URL, label, historical -`datamart-builder`, and sidebar/navigation definitions. Record exact files and line numbers. - -**Step 3: Resolve the real public origin** - -The owner reports `aritmolab.policlinicosandonato.com`; historical project state mentions a `.it` -origin and `/datamart-builder`. Record the live browser-visible origin and path from deployed -configuration. Do not choose between them without evidence. - -### Task 6: Survey Authentik - -**Files:** -- Create: `$PSD_SURVEY_ROOT/authentik.md` - -**Step 1: Identify deployment and version** - -Record Authentik containers/services, immutable image reference, version, base URL, database/Redis -dependencies, configuration owner, and backup/export procedure. Do not print container environments. - -**Step 2: Locate credential references** - -Record only file/secret-object paths, ownership, mode, and whether the local operator can use them. -If no usable administrative/API credential is found, stop and request owner help. - -**Step 3: Inventory relevant objects read-only** - -Using the installed version's API schema or admin interface, list only names/IDs for current -Aritmolab applications/providers, authorization flows, property mappings, groups, service accounts, -and policies that establish local conventions. Do not retrieve write-only secrets or raw tokens. - -**Step 4: Record version-specific constraints** - -Consult the official documentation matching the installed release for OAuth2/OIDC providers, -scope/property mappings, application bindings, blueprints/export, and API permission semantics. -Do not copy examples from a newer release without comparing the installed OpenAPI schema. - -### Task 7: Survey Supabase and the PSD DWH - -**Files:** -- Create: `$PSD_SURVEY_ROOT/supabase.md` -- Create: `$PSD_SURVEY_ROOT/dwh-readonly.txt` - -**Step 1: Map Supabase services without environments** - -Record PostgreSQL, pooler, PostgREST, gateway, and backup components; container networks and local -listeners; database name; TLS listener/CA; and the approved direct-connect route from ThothII core. - -**Step 2: Record exposed PostgREST schemas** - -Query only the explicit PostgREST schema setting through its known configuration mechanism. Do not -dump the whole environment. Confirm whether `thoth_sessions` already exists or is exposed. - -**Step 3: Inspect schemas and migration state** - -Through an approved administrative connection, run bounded catalog queries for existing schemas, -owners, and any `thoth_sessions` tables/migration records. Do not change them. - -**Step 4: Prove the intended DWH runtime identity is read-only** - -Connect using the protected runtime credential mechanism and query `current_database()`, -`current_user`, and grants for schema `datawarehouse`. Expected: USAGE/SELECT as required and no -INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, TRIGGER, CREATE, or ownership privileges. Do not run a -write probe against clinical tables. - -**Step 5: Identify session-schema roles** - -Record names or naming rules for a future migrator and runtime role. Do not create them. The final -design uses the existing database plus schema `thoth_sessions`, never a new database. - -### Task 8: Survey Git workspace and model boundaries - -**Files:** -- Create: `$PSD_SURVEY_ROOT/workspace-and-models.md` - -**Step 1: Inspect the workspace remote read-only** - -Record remote URL, branch, fetch credential path, current remote SHA, catalog entry, descriptor -schema version, current `supported_transports`, Evidence tree, annotations blob, and deploy-key -permissions. Do not push with the server's read-only deployment credential. - -**Step 2: Inspect Pi/LLM policy** - -Record selected provider/model/thinking level and credential references. Use bounded `tht pi status` -and `pi doctor` where compatible. Do not output provider keys. - -**Step 3: Check internal semantic capacity** - -Record CPU/GPU availability, free storage, and whether Docker can run the pinned Qdrant and Ollama -architectures. Do not pull images or models during the survey. - -### Task 9: Produce the survey decision - -**Files:** -- Modify: `$PSD_SURVEY_ROOT/survey-report.md` - -**Step 1: Complete the topology** - -Include exact component owners and flows for user → load balancer → Nginx → Aritmolab/sidebar → -ThothII, and core → Supabase DWH/auth session schema/Qdrant/Ollama/LLM/Authentik. - -**Step 2: List exact intended change files** - -Separate files owned by the new ThothII installation, workspace curator, Nginx, load balancer, -Aritmolab, Authentik, and Supabase. Mark shared files as owner-gated. - -**Step 3: State the scoped GO or NO-GO decisions** - -State both `SURVEY_GO_PROJECT_A_PRIVATE` and `SURVEY_GO_PROJECT_B`. The private decision requires -all paths, permissions, backup owners and rollback boundaries used by Project A; it may defer -public-origin, load-balancer, Authentik and Mac REST closeout facts that Project A does not mutate. -The Project B decision requires every shared/public fact plus the Mac acceptance, completed -observation window and revoked legacy credential. Each NO-GO must name concrete missing facts and -the person/system needed to resolve them. - -**Step 4: Hash and retain the report** - -Run: - -```bash -sha256sum "$PSD_SURVEY_ROOT/survey-report.md" > "$PSD_SURVEY_ROOT/survey-report.sha256" -sha256sum --check "$PSD_SURVEY_ROOT/survey-report.sha256" -``` - -Expected: checksum passes. Move the complete mode-0700 survey directory to the approved protected -evidence root without changing its contents; record the final path and digest in the journal. diff --git a/docs/prd/2026-08-09-workspace-preprocessing-prd.md b/docs/prd/2026-08-09-workspace-preprocessing-prd.md deleted file mode 100644 index 60fc54db..00000000 --- a/docs/prd/2026-08-09-workspace-preprocessing-prd.md +++ /dev/null @@ -1,543 +0,0 @@ -# PRD — Preprocessing per-workspace su ThothII (Qdrant + Git workspace registry) - -**Status:** PRD in revisione — decisioni D1–D9 chiuse il 2026-08-09; i piani P1–P10 partiranno solo dopo -revisione e conferma del proprietario -**Data:** 2026-08-09 -**Autore:** analisi dello stato attuale (branch `codex/git-workspace-registry`) + decisioni con il proprietario -**Uso:** riferimento stabile di requisiti e decisioni; da ogni punto nascerà un piano separato in -`docs/superpowers/plans/` (sez. 11) — questo documento non è un piano di lavoro - ---- - -> **Aggiornamento P1.1 (2026-08-11):** il layout del repository registry descritto nelle sezioni -> attive di questo PRD segue il contratto P1.1 accettato: catalogo di root `thoth-workspaces.yaml`, -> descriptor `/workspace.yaml`, evidence embedded `/evidence/`, annotazioni FK curate -> `/schema/annotations.yaml` (P5), docs generate `workspace-docs//`. I vecchi percorsi -> piatti (`workspaces/.yaml`, `workspace-content//evidence/`) sono superseded; le uniche -> occorrenze rimaste sono storiche (changelog/revisioni). Vedi -> `docs/superpowers/plans/2026-08-11-p2-p6-adaptation-to-p1-1-registry.md`. - ---- - -## 1. Contesto - -ThothII è passato da un indice semantico **pgvector sul server PSD** (descrizioni di tabelle/colonne, -evidence e memory embeddate nella stessa istanza Postgres del DWH, lettura via RPC `search_similar`, -scrittura via REST dedicato o loading diretto) a un'architettura con: - -- **infrastruttura semantica interna obbligatoria**: Qdrant + Ollama (`qwen3-embedding:0.6b`, 1024 dim, - cosine) come servizi Compose privati; una **collection Qdrant per workspace**, con schema/evidence/memory - separati dal payload `kind`; -- **workspace definito da un descriptor schema-v3 in un repo Git esterno** (id, DWH, collection, LLM - policy, diagnostics), con binding DWH locali all'installazione; -- **config harness renderizzata dal backend** a runtime (`runtime_identity` + `resources.vector` + - `resources.embeddings` + `roots` sotto `/sessions//`). - -La **macchina di preprocessing** (comandi, job a generazioni con publish atomico, adapter Qdrant, corpus -evidence, FK, memory) **esiste già ed è testata**: `tht preprocess evidence|dwh`, `tht vector init|index-schema`, -`tht schema introspect|suggest-fks|check`, `tht evidence extract|index`, `tht lsh build`, memory/solved. -Esistono job Compose fixture (`deploy/compose.preprocess.yaml` + `deploy/workspaces/preprocess-{dwh,evidence}.yaml`) -e uno smoke (`scripts/preprocess-smoke.sh`). - -### Il problema - -Il preprocessing **non è collegato al workspace reale del registry**: - -1. i job fixture usano una collection fissa (`preprocess-evidence`), un workspace_id derivato dal nome - file (`preprocess-evidence`) e roots sotto `/data/workspaces/preprocess-*`, che **non coincidono** con - quelli del runtime (`/sessions//`); -2. il backend **non espone alcun modo** di eseguire `tht preprocess` contro la config renderizzata di un - workspace (nessun endpoint, nessuno script, nessun comando documentato); -3. il **descriptor v3 e la config renderizzata non hanno la sezione `evidence`**: non c'è un posto canonico - dove dichiarare da dove arrivano le evidence di un workspace; -4. l'**ammissione sessione** (`ThtRunner.qdrantEnsure`) richiede la collection già esistente con 1024/cosine - e 8 payload keyword-index, ma **nessuno la crea esplicitamente** (`tht vector init` fallisce se manca); -5. le **generazioni `.tht-dwh` sono legate a un fingerprint della config completa** (`OWNER.json`: - workspace_id + config_fingerprint + input_fingerprint): se il preprocessing non usa la config identica a - quella renderizzata dal runtime, a runtime la generazione viene **rifiutata**; -6. la **cura FK** (`annotations.yaml`) è manuale e vive nel runtime artifacts; il registry **non sincronizza** - file dal repo ai roots runtime; -7. i **vecchi embedding pgvector (nomic 768d) non sono riusabili** (modello e dimensioni cambiati): serve - re-indicizzare i contenuti PSD. - -**Sintesi:** la parte "motore" è pronta; manca il **collegamento per-workspace** (config, esecuzione, -bootstrap, sorgente evidence) e la **documentazione operator**. - ---- - -## 2. Obiettivo - -Rendere l'attuale versione di ThothII (Qdrant + workspace su repo esterno) **configurabile e utilizzabile** -per un workspace reale, inclusa l'intera catena di preprocessing: **tabelle/colonne (catalogo + embedding), -FK (cura), evidence (sorgente → corpus → embedding), memory/solved**, con un flusso operator riproducibile, -documentato e verificato da smoke end-to-end. - -### Obiettivi secondari - -- O1. Un solo modo canonico di eseguire il preprocessing per un workspace (niente più fixture "speciali"). -- O2. Il preprocessing è **idempotente e ripristinabile**: rerun senza duplicati, publish atomico, GC. -- O3. Nessun segreto/endpoint entra nel repository registry né nei descriptor (invariante attuale preservato). -- O4. Il flusso è **documentato nei manuali operator** (`local/server-workspace-registry.md`) e coperto da - smoke automatici. -- O5. La **migrazione PSD** è definita (cosa si riusa, cosa si rigenera, cosa si esporta dal pgvector). -- O6. Ogni piano tecnico definisce, dove applicabile, un **goal automatico di processo completo**: da stato - pulito costruisce un ambiente isolato, simula il flusso end-to-end entro lo scope del piano e lo porta a - successo con un integration test riproducibile. -- O7. Dopo il successo automatico, un **percorso manuale separato** permette al reviewer di ripetere il - processo attraverso le interfacce reali, comprenderne l'architettura e approvare gli artefatti. - -## 3. Non-obiettivi (fuori scope di questo PRD) - -- Riscrivere il workflow NL→SQL o i gate (F1..F8) — restano invariati. -- Cambiare modello/architettura semantica (Qdrant/Ollama/1024/cosine) — già deciso e verificato. -- Rifare la UI di gestione workspace oltre a quanto già esiste. -- Il **deploy reale sul server PSD** (VPN, credenziali, portale, auth upstream): è un progetto operativo - separato che userà questo PRD come prerequisito tecnico. -- Supportare di nuovo pgvector o endpoint embedding esterni come percorso operativo. -- Il **comando di preprocessing avviabile dalla GUI**: è una release futura (fuori scope della release 0, - che è CLI sul host — vedi D2). - ---- - -## 4. Utenti - -| Utente | Esigenza | -| --- | --- | -| **Operatore/amministratore** (chi installa e cura un workspace) | Configurare DWH+evidence, eseguire il preprocessing, curare le FK, verificare lo stato, fare backup/restore. | -| **Autore ETL / curatore dominio** (es. il cliente PSD) | Mantenere evidence e annotazioni FK nel namespace del workspace nel repository registry con un flusso semplice. | -| **Reviewer umano** (usa l'app) | Vede search pack con tabelle/evidence/solved corretti: la qualità del retrieval dipende dal preprocessing. | -| **Sviluppatore ThothII** | Comandi/endpoint deterministici, testabili, senza sorprese di configurazione. | - ---- - -## 5. Scenario target (end-to-end) - -1. **Setup repo**: l'operatore usa un **unico repository Git registry** per tutti i workspace e crea - `/workspace.yaml` (schema-v3: DWH, collection, LLM policy) insieme al tree curato - `/evidence/`; descriptor e contenuti sono pubblicati nello stesso commit. -2. **Installazione**: `.env` + bindings `THT_WS_*` + secrets; `up` dello stack (frontend/core/qdrant/embedding). -3. **Registry**: pull → validazione → snapshot attivo; diagnostic DWH verdi. -4. **Preprocessing DWH**: introspezione (physical.yaml: tabelle/colonne/descrizioni/esempi/eligibility) + - LSH; generazione pubblicata sotto `.tht-dwh` del workspace. -5. **Cura FK**: `tht schema suggest-fks` → revisione umana → `annotations.yaml` (check senza orfani); - versionata dove deciso (vedi D5). -6. **Indice schema**: `tht vector index-schema` → record schema nella collection del workspace; la - collection, se inesistente, viene creata all'ammissione (self-heal) o dal primo write (vedi D4). -7. **Preprocessing evidence**: `tht preprocess evidence` → corpus generation (chunk+embed) nella collection - (kind `evidence`) + manifest ACTIVE nel corpus root del workspace. -8. **Memory/solved**: promozioni F8 (`memory promote/save-one`) e finalize (`solved-index`) scrivono nella - collection (kind `memory`). -9. **Uso**: nuova sessione → admission verde (collection+Ollama) → F1 `search pack` con tabelle, evidence e - solved del workspace; F4/F6 con FK curate. -10. **Operatività**: backup/restore volumi (Qdrant, corpus, `.tht-dwh`, registry), update, ripristino da - outage. - ---- - -## 6. Requisiti funzionali - -### RF1 — Configurazione per-workspace -- RF1.1 Un workspace del registry deve poter dichiarare **tutto ciò che serve al preprocessing** in un unico - posto canonico: DWH (già nel descriptor), **sorgente evidence completa** (protocollo/tipo, URI, parametri - non-secret), eventuali policy di chunk/retention. -- RF1.2 I segreti (password, API key, CA) restano fuori dal repo e dal descriptor (invariante attuale). -- RF1.3 La config harness usata dal preprocessing deve essere **derivata dalla stessa renderizzazione del - runtime** (stesso workspace_id, stessi roots, stessa configurazione effettiva). -- RF1.4 La configurazione (descriptor + bindings + config renderizzata) deve **prevedere i tre trasporti - DWH**: `postgres_direct`, `rest_api`, `ssh_tunnel`. PSD usa `rest_api`; altri database potranno usare - direct o tunnel. -- RF1.5 Il registry usa **un unico repository Git** per più workspace. Per una sorgente evidence - `filesystem`, l'URI è relativa alla root del repository ed è confinata lessicalmente a - `/`; path assoluti, traversal (`..`) e riferimenti al namespace di un - altro workspace sono invalidi. Descriptor e sorgente devono essere risolti dalla **stessa revisione Git**. - -### RF2 — Preprocessing DWH (tabelle/colonne) -- RF2.1 Comando/azione per eseguire `introspect` + `lsh` per un workspace del registry, contro la sua config - effettiva, con output JSON e resume. -- RF2.2 La CLI di preprocessing raggiunge il DWH **con il trasporto dichiarato dal workspace** (direct, - REST o tunnel SSH), come il runtime. -- RF2.3 Il catalogo risultante (`physical.yaml`) alimenta: cache `tht schema introspect`, render mschema - (F1/F4), record schema per l'embedding (RF4). -- RF2.4 Refreshing esplicito quando il DWH cambia (`--refresh`/nuova generazione), senza invalidare le - sessioni esistenti (generazioni + ACTIVE pointer, già implementato). - -### RF3 — FK -- RF3.1 Flusso curato per-workspace: `tht schema suggest-fks` (+ `--from-sql`, `--assume`, `--write`), - revisione umana, `tht schema check` (zero orfani). -- RF3.2 Le FK curate devono essere **disponibili a runtime** (sezione `【Foreign keys】` del render mschema, - usata da F4/F6) e **versionate nel repository registry** (D5). -- RF3.3 Nessuna FK derivata dal modello: il modello usa solo la lista curata (contratto SKILL invariato). - -### RF4 — Indice semantico schema + bootstrap collection -- RF4.1 `tht vector index-schema` embedda i record schema (tabella+colonna, con descrizioni/esempi/sinonimi) - nella collection del workspace (kind `schema`), idempotente (hash → upsert solo del cambiato). -- RF4.2 **Bootstrap della collection**: se inesistente all'ammissione sessione, il runtime la crea - (self-heal) con 1024/cosine + i payload keyword-index richiesti (`content_hash, document_id, kind, - record_key, record_kind, vector_generation, workspace_id, workspace_revision`). -- RF4.3 La **CLI deve poter cancellare e ricreare** la collection di un workspace (rebuild esplicito con - guardie di sicurezza e conferma). -- RF4.4 Prima di una sessione, l'ammissione resta invariata (collection compatibile + Ollama). - - -### RF5 — Evidence -- RF5.1 Sorgente evidence dichiarabile per-workspace nel descriptor (protocollo/tipo + URI). Per PSD è un - **tree di file `.md` versionato nell'unico repository registry**, sotto - `psd/evidence/`; in generale ogni workspace usa - `/evidence/`. HTTP manifest e S3 restano opzioni del motore per sorgenti - esterne. -- RF5.2 `tht preprocess evidence` per-workspace: discover → acquire → normalize/chunk → embed → upsert - (kind `evidence`, payload `document_id`/`vector_generation`) → publish ACTIVE nel corpus root del workspace, - con resume e dry-run (già implementato nel motore). -- RF5.3 GC/retention delle generazioni evidence (filesystem + punti Qdrant) con le policy esistenti. -- RF5.4 A runtime la ricerca evidence è filtrata dalla generazione ACTIVE e dal workspace_id (già - implementato: `ActiveEvidenceSearcher`); il flusso RF5 deve garantire che il corpus ACTIVE appartenga al - workspace giusto. - -### RF6 — Memory e domande risolte -- RF6.1 `memory promote/save-one` (F8) e `memory solved-index` (finalize) scrivono nella collection del - workspace (kind `memory`/`solved_question`) — verificare end-to-end con Qdrant e risolvere i TODO residui - in `memory_cmd.py`. -- RF6.2 Il registro JSONL resta la fonte canonica; Qdrant è proiezione di ricerca (invariante attuale). - -### RF7 — Migrazione PSD -- RF7.1 Definire cosa si **riusa** (physical.yaml, annotations.yaml con le ~228 FK curate, le 895 evidence - `.md`), cosa si **rigenera** (tutti gli embedding, modello diverso) e cosa si **esporta** dal pgvector del - server prima della dismissione. -- RF7.2 La migrazione è un'operazione documentata e rieseguibile, non un one-shot nel codice. - -### RF8 — Operatività e documentazione -- RF8.1 Manuali operator aggiornati con la sequenza completa per-workspace (config → preprocess → cura → - verifica → uso → backup/restore). -- RF8.2 La documentazione di progetto spiega **cos'è `.tht-dwh`** (generazioni, `OWNER.json`, `ACTIVE`, - vincolo di fingerprint) in modo comprensibile per l'operatore (D3). -- RF8.3 Smoke end-to-end automatico (workspace nuovo → tutto il ciclo → sessione reale → cleanup) che - sostituisce/completa `preprocess-smoke.sh` (oggi solo fixture). -- RF8.4 Backup/restore coprono Qdrant (già `vector-backup.sh`/`vector-restore.sh`), corpus, `.tht-dwh` e - registry. -- RF8.5 Ogni piano successivo traduce il proprio risultato operativo in un **process goal** verificabile da - un integration test completo per quello scope; eventuali interventi umani iniziali o intermedi sono - ammessi solo se inevitabili, espliciti, documentati e riprendibili. -- RF8.6 Ogni process goal automatico riuscito è seguito, quando utile, da un walkthrough manuale su un - ambiente nuovo e separato; automazione e accettazione umana producono evidenze distinte. - ---- - -## 7. Requisiti non funzionali - -- **RNF1 Sicurezza**: nessun segreto in repo/descriptor/config renderizzata/log; la CLI di preprocessing - (D2) non espone credenziali, non le logga e non le scrive negli artefatti. -- **RNF2 Determinismo/idempotenza**: rerun del preprocessing = zero duplicati (hash content), publish - atomico, generazioni immutabili (già nel motore). -- **RNF3 Robustezza**: degradazione controllata (workspace senza evidence o senza collection funziona, con - warning); errori sanitizzati; nessun fallimento che corrompa la generazione attiva. -- **RNF4 Isolamento per-workspace**: ogni filtro Qdrant legato a workspace_id; rifiuto di namespace - conflittuali (già implementato nell'adapter). -- **RNF5 Compatibilità**: il preprocessing deve funzionare con la config renderizzata dal backend - (fingerprint `OWNER.json` compatibile) — è il vincolo chiave di design (vedi D3). -- **RNF6 Performance**: introspezione ~minuti (non nel path di sessione), embedding batch, LSH boundato; - il retrieval a runtime non cambia i costi attuali. -- **RNF7 Manutenibilità**: nessun fork dei fixture; un solo percorso canonico (O1). -- **RNF8 Integration-first**: il successo di un piano tecnico richiede un'esecuzione completa da ambiente - pulito, senza retry automatici che mascherino errori; ogni fallimento viene diagnosticato, corretto alla - radice e seguito da una nuova esecuzione completa. -- **RNF9 Evidenza e cleanup**: ogni ambiente simulato ha identità/ownership esplicita, risorse univoche, - segreti fittizi, report machine-readable e leggibile, scansione anti-secret e cleanup confinato alle sole - risorse possedute dal run. - ---- - -## 8. Standard di esecuzione e verifica — integration-first - -Questo standard si applica a P1 e, **ovunque sia tecnicamente significativo**, a tutti i piani successivi. -Un piano che non possa applicarlo deve motivare esplicitamente l'eccezione e definire il verifier più vicino -possibile al processo reale. - -### S1 — Goal automatico di processo - -- Ogni piano definisce il **processo completo entro il proprio scope**, con punto iniziale pulito, input, - componenti attraversati, risultato osservabile e criteri di successo. -- Il goal non è "far passare alcuni test", ma **simulare con successo il processo operativo** che la feature - deve rendere possibile. Per P1 il confine completo è Git → registry → API → snapshot/docs → render → - `tht config check`; estrazione evidence e Qdrant appartengono ai piani successivi. -- Durante l'esecuzione il goal resta aperto fino a una prova integrale verde. Se l'ambiente agentico supporta - goal persistenti, l'esecutore lo registra all'inizio e lo completa soltanto dopo l'evidenza finale. - -### S2 — Ambiente di integrazione isolato - -- Il test costruisce dipendenze controllate sotto `.artifacts///`: repository Git simulati, - checkout, roots runtime, secret fixture, richieste/risposte, log e output. -- Ogni run usa identità e nomi univoci e un manifest di ownership; non usa credenziali, repository o dati - reali salvo quando il piano dichiara esplicitamente un gate L2. -- I servizi reali appartenenti allo scope vengono attraversati tramite le loro interfacce normali; quelli - esterni o non ancora nello scope sono sostituiti da fixture fedeli e deterministiche. - -### S3 — Contratto di successo - -- Il test parte da stato pulito, esegue il processo una volta senza retry automatici, termina con exit code - zero e produce `report.json` più un report leggibile. -- Un fallimento richiede diagnosi della causa, test regressivo/correzione e una nuova esecuzione completa da - stato pulito; ripetere alla cieca non costituisce progresso verso il goal. -- Il gate finale comprende determinismo/idempotenza pertinenti, scansione anti-secret, verifica degli - artefatti e prova del cleanup confinato. Gli artefatti possono essere conservati con `--keep` per review. - -### S4 — Interventi umani inevitabili - -- Passi umani iniziali o intermedi sono ammessi solo quando non simulabili in modo affidabile (per esempio - accesso approvato a un sistema reale o review di contenuto curato). -- Ogni passo umano dichiara precondizioni, istruzioni, evidenza richiesta, criterio di decisione e checkpoint - di ripresa; l'automazione copre e verifica tutto ciò che precede e segue il checkpoint. -- Un intervento umano non può essere sostituito da un'assunzione silenziosa né rendere non riproducibile il - resto del processo. - -### S5 — Walkthrough manuale successivo - -- Dopo il goal automatico verde, il reviewer ripete il processo in un **ambiente nuovo e separato**, usando - le interfacce reali e una guida passo-passo che spiega componente, stato letto, artefatto prodotto e - invariante verificata. -- Il walkthrough serve a comprensione architetturale e accettazione; non sostituisce l'integration test e non - ne riusa lo stato già mutato. -- Lo stato di consegna distingue almeno `automated integration: PASS` e `manual acceptance: PENDING/PASS`. - Un piano non è pienamente accettato finché l'eventuale gate manuale richiesto non è stato deciso dal reviewer. - -### S6 — Contenuto obbligatorio dei piani - -Ogni piano tecnico riporta, adattandoli al proprio scope: - -1. **Automated process goal** e comando unico di esecuzione; -2. topologia dell'ambiente simulato e confini delle dipendenze; -3. asserzioni del full integration test e contratto del report; -4. checkpoint umani inevitabili, oppure dichiarazione esplicita che non ve ne sono; -5. walkthrough/gate manuale successivo, quando utile; -6. evidenze di completamento, retention degli artefatti e cleanup esatto. - ---- - -## 9. Criteri di accettazione (bozza) - -1. Da un repository registry vuoto si arriva a una sessione funzionante seguendo **solo i manuali aggiornati**, - senza toccare file fixture. -2. La CLI di preprocessing funziona **sia sul PC/Mac dell'utente sia sul server che ospita il DWH** - (stesso comando, config derivata dal workspace). -3. La configurazione di un workspace dichiara e usa uno dei **tre trasporti DWH** (`postgres_direct`, - `rest_api`, `ssh_tunnel`); PSD usa `rest_api`. -4. Il goal automatico P1 costruisce da zero repository Git simulati e ambiente isolato, attraversa con - HTTP reale il processo Git → registry → validate/publish/read/export → snapshot/docs → render → - `tht config check`, supera casi positivi e negativi senza retry e produce report/artefatti secret-free. -5. Solo dopo il punto 4, un ambiente manuale nuovo avvia il backend su `127.0.0.1:8791` e permette al - reviewer di ripetere ogni chiamata e ispezionare commit, snapshot, ZIP e config renderizzate seguendo una - guida; il gate resta `PENDING` finché il reviewer non lo approva. -6. `search pack` di una domanda reale restituisce tabelle (con descrizioni), evidence della generazione - ACTIVE e solved dello stesso workspace; F4/F6 mostrano le FK curate. -7. `qdrantEnsure`/`ollamaEnsure` verdi all'ammissione; la collection ha esattamente 1024/cosine + gli 8 - keyword-index. -8. Rerun del preprocessing: `unchanged` (nessun duplicato); modifica di un'evidence → nuova generazione, - ACTIVE aggiornato, vecchie generazioni in GC. -9. Smoke end-to-end automatico verde in CI con cleanup esatto (stile `preprocess-smoke.sh`). -10. Migrazione PSD documentata e provata almeno in dry-run (re-introspection o riuso catalogo + re-embedding). -11. Ogni piano tecnico successivo include un process goal automatico completo per il proprio scope e un - walkthrough manuale quando utile, oppure documenta l'inevitabile eccezione umana secondo S4. - ---- - -## 10. Decisioni chiuse (2026-08-09) - -> La sezione nasceva come "punti di discussione"; le decisioni sono state prese con il proprietario del -> prodotto il 2026-08-09. Ogni punto resta il riferimento del proprio piano (sez. 11). Le opzioni scartate -> sono omesse; la motivazione della scelta è inclusa in ogni punto. - -### D1 — Config per-workspace: **c) misto, con sorgente completa nel descriptor** -- Il descriptor v3 guadagna una sezione `evidence` che configura **tutta la lettura della sorgente**: - protocollo/tipo, URI/sorgente, eventuali parametri non-secret. -- Si usa **un unico repository Git registry** per tutti i workspace. Ogni workspace possiede il proprio tree - versionato sotto `/evidence/`; per PSD il path canonico è - `psd/evidence/`. -- Per `filesystem`, l'URI del descriptor è repo-relative, confinata al namespace dello stesso workspace e - risolta dalla stessa revisione Git del descriptor. Sono vietati path assoluti, traversal e riferimenti al - contenuto di un altro workspace; il controllo reale di symlink/containment durante la materializzazione - appartiene a P6. -- Eventuali segreti (HTTP autenticato, S3) restano in overlay d'installazione — invariante: nessun segreto - nel repo/descriptor. -- P1 applica lo standard integration-first: prima persegue un goal automatico Git→registry→HTTP→render→ - harness sotto `.artifacts/p1-integration//`, poi offre un walkthrough manuale separato sotto - `.artifacts/manual-acceptance/p1/`. Non include ancora estrazione, embedding o verifica degli artefatti - `artifacts/evidence` (P2+P6). - -### D2 — Esecuzione: **CLI sul host in release 0; GUI in release futura** -- **Release 0**: una **CLI installata con ThothII sul host** — sia il PC/Mac dell'utente sia il server che - ospita il DWH — che esegue **tutta la catena di preprocessing** (DWH introspect+LSH, FK, index-schema, - evidence e quanto serve) per un workspace del registry. -- La CLI deriva la config dal descriptor+bindings con la stessa identità del runtime → soddisfa il vincolo - D3 senza dipendere dal backend. -- **Release futura (fuori scope)**: comando avviabile dalla GUI (endpoint backend da progettare poi). - -### D3 — Fingerprint `.tht-dwh`: **accettare il vincolo + documentarlo** -- Le generazioni DWH restano legate alla config effettiva (workspace_id + config_fingerprint + - input_fingerprint in `OWNER.json`). -- La CLI (D2) gira con la config derivata dal descriptor+bindings, quindi identica alla runtime. -- **La documentazione di progetto deve spiegare chiaramente cos'è `.tht-dwh`** (directory delle generazioni - catalogo/LSH, `OWNER.json`, `ACTIVE` pointer, perché il fingerprint protegge da artefatti di un'altra - config) — oggi non è chiaro. - -### D4 — Bootstrap collection: **self-heal all'ammissione + CLI delete/recreate** -- Se la collection non esiste all'ammissione sessione, il runtime la crea (1024/cosine + payload - keyword-index) — self-heal. -- La **CLI deve poter cancellare e ricreare le collection** (rebuild esplicito, con guardie di sicurezza). - -### D5 — Versioning artifacts curati: **c) misto** -- `annotations.yaml` (cura FK, cura umana) **versionata nel repository registry** e sincronizzata ai roots - runtime (il registry copia il file negli snapshots → sync). -- `physical.yaml` (derivato dall'introspezione) rigenerato localmente, non versionato. - -### D6 — Evidence: **a) nell'unico repository registry, con namespace per-workspace** -- Ogni workspace contiene il proprio tree versionato sotto - `/evidence/`; le dimensioni non sono un vincolo. -- P6 materializza il tree dalla **stessa revisione Git** del descriptor, verifica il containment reale - (inclusi i symlink) e lo rende disponibile al preprocessing senza usare un checkout mobile. -- HTTP/S3 restano opzioni future per sorgenti esterne (il motore le supporta già). - -### D7 — Migrazione PSD: **inclusa, con accesso al server** -- Il piano P7 copre: riuso di physical.yaml + annotations.yaml + evidence `.md` dal repository registry; export - dal pgvector del server PSD (accesso disponibile); re-embedding con `qwen3-embedding:0.6b`; dry-run - documentato. - -### D8 — Verifica end-to-end: **integration-first + walkthrough manuale; remote Git libero; DWH multi-trasporto** -- Ogni fase adotta lo standard della sez. 8: prima un process goal automatico da ambiente pulito, poi — - quando utile o richiesto — un walkthrough manuale su stato separato. P1 è il primo riferimento concreto. -- Il livello finale del PRD resta: smoke automatico su workspace sintetico (CI) + gate manuale L2 su PSD. -- Il namespace PSD sarà **prima alimentato nel repository registry** (descriptor + evidence + annotations - nello stesso flusso Git), **poi** usato da ThothII. Accesso al server PSD disponibile. -- **Remote Git**: lo creiamo noi, nessun vincolo tecnico (consigliato GitHub via HTTPS; SSH resta - possibile se servirà). -- **Trasporto DWH**: PSD via **REST** (come oggi); **la configurazione deve prevedere le tre modalità** — - `rest_api`, `postgres_direct`, `ssh_tunnel` — perché altri database potrebbero richiedere accesso TCP - diretto o via tunnel. Oggi `ssh_tunnel` è solo diagnostico a runtime: va reso operativo dove serve - (vedi P10). - -### D9 — Retention/GC: **confermata** -- Default invariati (`retain_published_generations: 3`, chunk 4000 char), configurabili per-workspace via - la sezione `evidence`/policy del descriptor (D1). - -## 11. Mappa dei piani (uno per punto del PRD) - -Questo PRD non diventa un unico piano: **ogni decisione/requisito produce un piano separato (P1–P10)** in -`docs/superpowers/plans/`, eseguibile in sequenza o come workstream indipendenti. Il PRD resta il -riferimento stabile (requisiti + decisioni); ogni piano cita il punto di origine e i criteri di -accettazione applicabili (sez. 9) e adotta lo standard integration-first (sez. 8). - -| Piano | Punto PRD | Contenuto sintetico | Dipende da | -| --- | --- | --- | --- | -| P1 | D1 | Descriptor v3: sezione `evidence` (protocollo/tipo, URI repo-relative sotto `/evidence/`) + policy e isolamento namespace; goal automatico Git→registry→HTTP→render→harness, seguito da walkthrough manuale | — | -| P2 | D2 | **CLI di preprocessing sul host (release 0)**: comando per-workspace che esegue l'intera catena (DWH, FK, index-schema, evidence) con la config derivata da descriptor+bindings; funziona su PC/Mac utente e server DWH | P1 | -| P3 | D3 | Vincolo fingerprint `.tht-dwh` (test: preprocess con config identica alla runtime) + **documentazione di progetto su cos'è `.tht-dwh`** | P2 | -| P4 | D4 | Bootstrap collection: **self-heal all'ammissione** (creazione 1024/cosine + keyword-index) + **comandi CLI delete/recreate** con guardie | — | -| P5 | D5 | `annotations.yaml` versionata nel repository registry + sync registry → roots runtime | P1 | -| P6 | D6 | Materializzazione del tree `/evidence/` dalla revisione Git fissata → preprocess; containment reale e protezione da symlink escape | P1 | -| P7 | D7 | Migrazione PSD: riuso catalogo/annotations/evidence, **export pgvector (accesso server)**, re-embedding, dry-run | P1–P6 | -| P8 | D8 | Verifica end-to-end: smoke CI + gate L2 su PSD (**namespace PSD nel repository registry alimentato prima dell'uso**; remote Git a scelta) | P1–P7 | -| P9 | D9 | GC/retention per-workspace: policy configurabili, default invariati | P1 | -| P10 | D8/RF1.4 | Trasporti DWH operativi: rendere `ssh_tunnel` utilizzabile a runtime e nella CLI di preprocessing (oggi solo diagnostico); verifica dei tre trasporti (direct, REST, tunnel) | P1, P2 | - -### Standard di verifica obbligatorio del futuro piano P1 - -P1 è il primo piano che applica integralmente la sez. 8 e deve contenere due task/gate distinti e ordinati. - -#### 1. Automated integration goal — complete P1 configuration process - -Un comando unico (nome definitivo nel piano, interfaccia indicativa -`./scripts/p1-acceptance.sh integration --keep`) costruisce da zero: - -```text -.artifacts/p1-integration// -├── ownership.json -├── remote.git/ # remote bare locale -├── author/ # clone curatore + tree evidence -├── installation/ # checkout, snapshot e stato registry -├── runtime-data/ -├── fixture-secrets/ -├── requests/ # payload HTTP positivi/negativi -├── responses/ -├── rendered/ -├── exports/ -├── logs/ -├── report.json -└── report.md -``` - -Il test attraversa le interfacce reali appartenenti a P1: Git reale locale, backend Fastify su una porta -loopback temporanea, route HTTP validate/publish/pull/read/export, snapshot/docs/contract, renderer di -produzione e `tht config check`. Verifica anche stessa revisione Git per descriptor/tree, determinismo, -path/protocolli/secret fields invalidi, assenza di leak e cleanup confinato. Non usa frontend, Docker, DWH, -Qdrant o Ollama perché non appartengono allo scope P1. - -L'esecuzione non applica retry automatici. In caso di errore l'esecutore diagnostica, aggiunge la copertura -regressiva necessaria, corregge e rilancia l'intero scenario da una nuova root pulita. Il goal è raggiunto -solo con exit code zero e report integralmente verde; con `--keep` le evidenze restano disponibili. - -#### 2. Manual acceptance gate — descriptor and rendered configuration artifacts - -Dopo il goal automatico verde, il piano prepara uno stato nuovo e indipendente sotto: - -```text -.artifacts/manual-acceptance/p1/ -├── remote.git/ -├── author/ -├── installation/ -├── runtime-data/ -├── requests/ -├── responses/ -├── output/ -├── logs/ -└── GUIDE.md -``` - -Un helper esegue soltanto `prepare/serve/stop/cleanup`; `serve` avvia il backend reale sull'host, senza -Docker e senza frontend, vincolato a `127.0.0.1:8791`. Il reviewer segue `GUIDE.md` ed esegue personalmente -le chiamate HTTP, i comandi Git, l'export ZIP, il doppio rendering, il confronto e `tht config check`, poi -prova i casi invalidi e decide il gate. - -Il gate verifica manualmente: sezione `evidence`; pubblicazione/rilettura; commit e snapshot immutabile; -workspace docs/contract; config harness; assenza di segreti; sicurezza protocollo/path e isolamento -cross-workspace; output deterministico. Gli artefatti restano fino alla decisione e il cleanup rimuove solo -la root posseduta dal test. - -P1 **non** dichiara di aver generato o validato `artifacts/evidence`: estrazione e mirroring richiedono -P2+P6; record Qdrant, embedding, generazioni ACTIVE e retention appartengono ai piani successivi. Dopo il -goal automatico lo stato è `automated integration: PASS / manual acceptance: PENDING`; P1 diventa pienamente -accettato soltanto dopo la decisione del reviewer. - -Ordine consigliato: **P1 → P2 → P3** (catena config/esecuzione), **P4** e **P5/P6** in parallelo dopo P1, -poi **P7 → P8**; P9 può essere assorbito in P1 o restare autonomo; **P10** dopo P1+P2 (necessario solo se un -workspace target richiede davvero il tunnel — per PSD non serve, usa REST). - -Ogni piano segue la prassi del repo: TDD, commit scoping, verifica layer (pytest/vitest/tsc/build) e lo -standard della sez. 8; gate deployment e smoke Docker si aggiungono quando appartengono allo scope. Lo stato -traccia separatamente implementazione, automated integration e manual acceptance. - ---- - -## 12. Storico revisioni - -| Versione | Data | Contenuto | -| --- | --- | --- | -| v0.1 | 2026-08-09 | Bozza da analisi dello stato attuale (gap preprocessing per-workspace) | -| v0.2 | 2026-08-09 | Decisioni D1–D9 chiuse con il proprietario; mappa piani P1–P10; requisiti RF1–RF8 aggiornati (evidence nel descriptor, CLI sul host, self-heal collection, multi-trasporto DWH) | -| v0.3 | 2026-08-09 | Revisione di coerenza (numerazioni, riferimenti incrociati, header di stato) — pronto per revisione del proprietario | -| v0.4 | 2026-08-09 | D1/D6: repository registry unico, namespace `workspace-content//evidence/` *(percorso storico P1, superseded da P1.1)*, pin alla stessa revisione Git e gate manuale P1 con remote locale usa-e-getta sotto `.artifacts/` | -| v0.5 | 2026-08-09 | Standard integration-first per P1–P10: process goal automatico completo da ambiente simulato e pulito, gestione esplicita degli interventi umani inevitabili e walkthrough manuale successivo su stato separato | - ---- - -## 13. Riferimenti - -- Stato attuale: `PROJECT_STATE.md` (sezioni "Internal Qdrant + Ollama semantic infrastructure", snapshot - registry) e `AGENTS.md`. -- Design architettura semantica: `docs/plans/2026-08-08-internal-qdrant-ollama-design.md` e relativo piano. -- Registry: `docs/superpowers/specs/2026-08-03-git-workspace-registry-design.md`, manuali - `docs/install/local-workspace-registry.md` / `server-workspace-registry.md`. -- Motore preprocessing: `harness/tht/cli/preprocess_cmd.py`, `harness/tht/corpus/pipeline.py`, - `harness/tht/jobs/dwh_pipeline.py`, `harness/tht/adapters/vector/qdrant.py`, - `harness/tht/vectorstore/records.py`, `harness/tht/cli/{vector,schema,evidence,memory}_cmd.py`. -- Fixture attuali: `deploy/compose.preprocess.yaml`, `deploy/workspaces/preprocess-{dwh,evidence}.yaml`, - `scripts/preprocess-smoke.sh`. -- Ammissione runtime: `backend/src/tht/tht-runner.ts` (`qdrantEnsure`/`ollamaEnsure`), - `backend/src/workspaces/runtime-renderer.ts`. diff --git a/docs/reports/2026-08-15-tht-command-audit.md b/docs/reports/2026-08-15-tht-command-audit.md deleted file mode 100644 index d2cc3a00..00000000 --- a/docs/reports/2026-08-15-tht-command-audit.md +++ /dev/null @@ -1,171 +0,0 @@ -# Audit critico dei comandi `tht` - -Data: 2026-08-15 - -## Scopo - -Questo audit valuta tutti i comandi terminali registrati dall'attuale CLI Python `tht` prima di -unificare la CLI di ThothII sotto un solo eseguibile pubblico. La valutazione incrocia: - -- il contratto del workflow Pi in `harness/.pi/skills/tht-sessione/SKILL.md`; -- le invocazioni reali del gate in `harness/.pi/extensions/tht-gate.js`; -- le invocazioni del backend in `backend/src/tht/tht-runner.ts`; -- i job operatore in `backend/src/workspaces/preprocessing-service.ts`; -- il migratore in `docker/session-migrate.sh`; -- test e documentazione esistenti. - -L'inventario autorevole contiene 76 comandi Typer più il comando callback `doctor`: 77 comandi -terminali complessivi. - -## Legenda - -- **WF — intoccabile workflow**: chiamato da Pi, dal gate o dal contratto delle otto fasi. Va - conservato con semantica, output JSON ed exit code compatibili. Non deve necessariamente apparire - nell'help ordinario dell'utente. -- **PL — intoccabile piattaforma**: chiamato dal backend, dai job workspace o dal deployment. Anche - questo è un contratto interno, non necessariamente un comando da mostrare all'utente. -- **ADV — mantenere avanzato**: non è nel flusso automatico, ma offre una capacità amministrativa o - di recupero che sarebbe imprudente perdere. Va nascosto dall'help base. -- **ACCORPA**: la capacità serve, ma non merita un comando autonomo. -- **RIMUOVI**: il comando non ha chiamanti reali ed è duplicato, superato, pericoloso o incompleto. - L'eventuale logica riutilizzata da altri flussi resta una libreria interna. - -## Risultato sintetico - -| Esito | Numero | Conseguenza | -|---|---:|---| -| WF o PL, intoccabili | 55 | Conservare il contratto; nascondere i primitivi tecnici dall'help base | -| ADV o ACCORPA | 8 | Conservare la capacità riducendo la superficie UX | -| RIMUOVI | 14 | Eliminare il comando dalla nuova CLI | -| **Totale** | **77** | Una sola CLI pubblica molto più semplice, senza riscrivere il workflow vivo | - -## Matrice completa - -### Diagnostica, configurazione e dipendenze - -| Comando | Valutazione | -|---|---| -| `config check` | **ACCORPA** in `tht doctor`: la validazione della configurazione serve, ma due preflight distinti confondono l'utente. | -| `doctor` | **ACCORPA/MANTIENI pubblico** come unico `tht doctor`, includendo controlli host, Compose, storage e configurazione runtime. | -| `db ping` | **PL**: il backend lo usa per rifiutare correttamente una nuova sessione quando il DWH non è raggiungibile o non è read-only. Interno. | -| `db fetch-ca` | **ACCORPA** in `tht setup` o nella configurazione workspace: utile per TLS, ma non giustifica un comando isolato. | -| `ollama ensure` | **PL**: preflight automatico dell'embedder usato dal backend. Interno. | - -### Fasi e decision ledger - -| Comando | Valutazione | -|---|---| -| `phase advance` | **WF**: il gate lo usa per avanzare solo dopo la decisione umana. Primitivo anti-bypass, quindi interno. | -| `phase meta` | **WF**: fornisce al gate la definizione data-driven delle fasi e dei tipi di decisione. Interno. | -| `phase reopen` | **WF**: è il percorso canonico per tornare a una fase precedente e invalidare deterministicamente gli artefatti successivi. | -| `phase show` | **WF**: il gate lo usa per calcolare la fase corrente. Interno. | -| `decision add` | **WF**: persistenza fondamentale delle decisioni del reviewer. Solo gate, non shell utente. | -| `decision add-batch` | **WF**: scrittura atomica delle decisioni multiple. Evita ledger parziali. | -| `decision add-join-set` | **WF**: sostituzione atomica dell'intero insieme di join. | -| `decision list` | **RIMUOVI**: nessun chiamante; `session show --json` contiene già il ledger necessario. | -| `decision retract` | **RIMUOVI** dalla CLI: nessun flusso vivo lo invoca e `phase reopen` è il percorso di correzione supportato. La semantica tombstone può restare nel dominio finché utile. | - -### Sessioni - -| Comando | Valutazione | -|---|---| -| `session archive` | **PL**: usato dalla gestione sessioni del backend. | -| `session check` | **WF**: gate oggettivo della fase 5; verifica decisioni e schema linking. | -| `session close` | **PL**: usato dal backend. | -| `session delete` | **PL**: usato dal backend con i relativi controlli applicativi. | -| `session documents` | **WF/PL**: ricostruisce il contesto persistito e alimenta sia Pi sia la GUI. | -| `session fail` | **PL**: usato dal backend per rappresentare il fallimento terminale. | -| `session finalize` | **WF**: chiusura deterministica della fase finale e indicizzazione della domanda risolta. | -| `session list` | **PL**: alimenta la lista sessioni della GUI. | -| `session migrate` | **PL**: eseguito dal servizio one-shot di migrazione server; resta interno dietro `tht sessions migrate`. | -| `session new` | **WF/PL**: crea la persistenza iniziale della domanda; il backend dipende dal JSON restituito. | -| `session preferences get` | **PL**: lettura delle preferenze applicative. Interno. | -| `session preferences set` | **PL**: scrittura delle preferenze applicative. Interno. | -| `session reopen` | **PL**: riapertura dello stato terminale esposta dalla gestione sessioni. | -| `session retrieval-pack` | **WF**: legge il retrieval pack già persistito per il kickoff di Pi. Distinto da `search pack`, che lo costruisce. | -| `session set-group` | **PL**: rinomina il raggruppamento dalla GUI. | -| `session set-name` | **PL**: rinomina la sessione dalla GUI. | -| `session set-question` | **WF**: persiste deterministicamente domanda riscritta e assunzioni. Solo gate. | -| `session set-schema-linking` | **WF**: valida e scrive `schema_linking.json`. Solo gate. | -| `session show` | **WF/PL**: fonte compatta dello stato persistito per resume, gate e backend. | -| `session sync-schema-linking` | **WF**: riproietta deterministicamente il ledger nello schema linking. | -| `session unarchive` | **PL**: usato dalla gestione sessioni del backend. | - -### Schema e retrieval - -| Comando | Valutazione | -|---|---| -| `schema check` | **PL**: validazione delle annotazioni curate nel workflow workspace. | -| `schema columns` | **WF**: il gate usa il catalogo colonne per validare e correggere il linking. | -| `schema introspect` | **WF**: fallback previsto dal contratto quando manca lo schema fisico; la modalità refresh resta manutenzione. | -| `schema render` | **WF**: produce il contesto mschema usato dal modello. | -| `schema suggest-fks` | **PL**: comando del flusso operatore per le annotazioni FK curate. | -| `search find` | **WF**: ricerca mirata di evidence, valori e formule durante le fasi. | -| `search pack` | **WF/PL**: costruisce e persiste il contesto iniziale F1; usato anche dal backend. | - -### CTE, SQL e datamart - -| Comando | Valutazione | -|---|---| -| `cte info` | **WF**: restituisce SQL persistito, posizione nel piano e ultimo test. | -| `cte list` | **RIMUOVI**: nessun chiamante o test; `cte plan`, `cte info` e `session documents` coprono il bisogno. | -| `cte next` | **WF**: il gate determina il prossimo CTE da revisionare. | -| `cte plan` | **WF**: persiste l'ordine completo dei CTE. | -| `cte save` | **WF**: tool deterministico di scrittura usato dal gate. | -| `cte test` | **WF**: verifica read-only dei CTE prevista esplicitamente dal contratto. | -| `sql validate` | **WF**: validazione strutturale e read-only prima dell'esecuzione. | -| `sql preview` | **WF/PL**: preview controllata usata dal modello e dalla GUI. | -| `sql set-final` | **WF**: unica scrittura canonica di `sql_final.sql` attraverso il repository di sessione. | -| `sql export` | **PL**: esportazione richiesta dalla GUI. | -| `sql explain` | **RIMUOVI**: nessun chiamante, test o requisito nel workflow corrente. Si reintroduce solo con un vero passo di analisi del piano. | -| `sql save` | **RIMUOVI**: duplica `set-final` ed `export` e permette un percorso di scrittura non usato. | -| `datamart generate` | **WF**: fase 8 del workflow. | - -### Memory - -| Comando | Valutazione | -|---|---| -| `memory promote` | **WF**: preview dei candidati di promozione usata dal gate. | -| `memory save-one` | **WF**: persistenza atomica della singola memory approvata. | -| `memory search` | **WF**: recupero delle memory riutilizzabili nella fase 2. | -| `memory solved-index` | **WF**: recupero manuale previsto se l'indicizzazione al finalize fallisce. | -| `memory solved-search` | **WF**: recupero di domande risolte simili nelle fasi successive. | -| `memory list` | **ADV**: mantenere per amministrare record errati, ma fuori dall'help base. | -| `memory show` | **ADV**: mantenere insieme a `list` per ispezione puntuale. | -| `memory update` | **ADV**: mantenere per correggere il merito di una memory senza alterarne la provenienza. | -| `memory delete` | **ADV**: mantenere come rimedio selettivo; richiede conferma esplicita nella nuova CLI. | -| `memory index` | **ADV**: utile come riparazione/full-resync, ma va presentato come manutenzione e non come uso normale. | -| `memory clear` | **RIMUOVI**: distruzione globale non usata; confligge con una UX sicura di backup/ripristino. | -| `memory migrate` | **RIMUOVI**: migrazione legacy una tantum senza dati di produzione da preservare. | - -### Preprocessing, evidence e indici - -| Comando | Valutazione | -|---|---| -| `preprocess dwh` | **PL**: pipeline canonica usata da `tht workspace preprocess dwh/run`. | -| `preprocess evidence` | **PL**: pipeline canonica usata da `tht workspace preprocess evidence/run`. | -| `vector index-schema` | **PL**: indicizzazione schema usata dal workflow workspace. | -| `evidence extract` | **RIMUOVI**: primitivo superato dalla pipeline versionata `preprocess evidence`. Conservare soltanto la logica riusata. | -| `evidence index` | **RIMUOVI**: primitivo superato dalla stessa pipeline versionata. | -| `lsh build` | **RIMUOVI** come comando: è già uno step di `preprocess dwh`; il builder resta interno. | -| `lsh query` | **RIMUOVI**: probe visuale senza chiamanti, test o documentazione operativa. La ricerca applicativa passa da `search find`. | -| `vector init` | **RIMUOVI**: il controllo di Qdrant/embedder è ormai coperto dal reconciler di collezione, da `ollama ensure` e dal nuovo `tht doctor`. | - -### Formule di concetto - -| Comando | Valutazione | -|---|---| -| `formula save` | **RIMUOVI** dalla CLI corrente: nessun chiamante, test o flusso di approvazione lo usa. Conservare il formato/store e la lettura tramite `search find --kind formula`. | -| `formula list` | **RIMUOVI**: stesso sottosistema incompleto. Un futuro flusso di curation dovrà progettare insieme creazione, approvazione, elenco e modifica. | - -## Conseguenza per la nuova CLI unica - -La semplificazione migliore non consiste nel rinominare tutti i 55 contratti vivi o nel mostrarli -all'utente. Consiste nel mantenere un unico eseguibile `tht` con due livelli di visibilità: - -1. l'help ordinario mostra soltanto setup, lifecycle, backup/restore, Pi e workspace; -2. i contratti WF/PL restano invocabili dallo stesso eseguibile, ma sono interni/nascosti e usati da - backend, gate e job one-shot. - -In questo modo l'utente vede una CLI piccola, mentre il workflow non subisce una riscrittura inutile -e rischiosa. Non serve un secondo eseguibile né un alias `thothctl`. diff --git a/docs/reports/2026-08-15-tht-command-maintain-erase-enhance.md b/docs/reports/2026-08-15-tht-command-maintain-erase-enhance.md deleted file mode 100644 index 9f2095d9..00000000 --- a/docs/reports/2026-08-15-tht-command-maintain-erase-enhance.md +++ /dev/null @@ -1,148 +0,0 @@ -# Proposta maintain-erase-enhance per i comandi `tht` - -Data: 2026-08-15 - -## Criterio - -- **MAINTAIN**: il comando resta disponibile senza modifiche sostanziali. Come richiesto, non viene - aggiunta una motivazione. -- **ERASE**: il comando viene eliminato dalla nuova CLI; la motivazione indica la duplicazione, il - superamento o l'assenza di un utilizzo reale. -- **ENHANCE**: la capacità viene mantenuta, ma il comando viene migliorato, accorpato o reso più - sicuro. La proposta indica l'intervento. - -La proposta copre tutti i 77 comandi terminali dell'attuale CLI Python. - -## Sintesi - -| Proposta | Numero | -|---|---:| -| MAINTAIN | 55 | -| ENHANCE | 8 | -| ERASE | 14 | -| **Totale** | **77** | - -## Lista completa - -### Diagnostica, configurazione e dipendenze - -| Comando | Proposta | -|---|---| -| `config check` | **ENHANCE** — incorporare la validazione nel comando pubblico `tht doctor`, mantenendo una funzione interna riutilizzabile e l'output strutturato. Evita due preflight sovrapposti. | -| `doctor` | **ENHANCE** — farne l'unica diagnostica multilivello: installazione, descriptor, Compose, storage, configurazione runtime, DWH, Pi, Qdrant ed embedder. Deve offrire output umano e `--json`, senza mutare lo stato. | -| `db ping` | **MAINTAIN** | -| `db fetch-ca` | **ENHANCE** — integrarlo nel setup guidato del workspace, mostrando endpoint e fingerprint prima della conferma. Può restare disponibile come operazione TLS avanzata, ma non come passaggio manuale obbligatorio. | -| `ollama ensure` | **MAINTAIN** | - -### Fasi e decision ledger - -| Comando | Proposta | -|---|---| -| `phase advance` | **MAINTAIN** | -| `phase meta` | **MAINTAIN** | -| `phase reopen` | **MAINTAIN** | -| `phase show` | **MAINTAIN** | -| `decision add` | **MAINTAIN** | -| `decision add-batch` | **MAINTAIN** | -| `decision add-join-set` | **MAINTAIN** | -| `decision list` | **ERASE** — non ha chiamanti reali e duplica il ledger già restituito da `session show --json`. | -| `decision retract` | **ERASE** — non è invocato dal workflow corrente; `phase reopen` è il percorso supportato per correggere e invalidare deterministicamente le decisioni. La semantica tombstone può restare nel dominio. | - -### Sessioni - -| Comando | Proposta | -|---|---| -| `session archive` | **MAINTAIN** | -| `session check` | **MAINTAIN** | -| `session close` | **MAINTAIN** | -| `session delete` | **MAINTAIN** | -| `session documents` | **MAINTAIN** | -| `session fail` | **MAINTAIN** | -| `session finalize` | **MAINTAIN** | -| `session list` | **MAINTAIN** | -| `session migrate` | **MAINTAIN** | -| `session new` | **MAINTAIN** | -| `session preferences get` | **MAINTAIN** | -| `session preferences set` | **MAINTAIN** | -| `session reopen` | **MAINTAIN** | -| `session retrieval-pack` | **MAINTAIN** | -| `session set-group` | **MAINTAIN** | -| `session set-name` | **MAINTAIN** | -| `session set-question` | **MAINTAIN** | -| `session set-schema-linking` | **MAINTAIN** | -| `session show` | **MAINTAIN** | -| `session sync-schema-linking` | **MAINTAIN** | -| `session unarchive` | **MAINTAIN** | - -### Schema e retrieval - -| Comando | Proposta | -|---|---| -| `schema check` | **MAINTAIN** | -| `schema columns` | **MAINTAIN** | -| `schema introspect` | **MAINTAIN** | -| `schema render` | **MAINTAIN** | -| `schema suggest-fks` | **MAINTAIN** | -| `search find` | **MAINTAIN** | -| `search pack` | **MAINTAIN** | - -### CTE, SQL e datamart - -| Comando | Proposta | -|---|---| -| `cte info` | **MAINTAIN** | -| `cte list` | **ERASE** — non ha chiamanti o test e sovrappone informazioni già disponibili con `cte plan`, `cte info` e `session documents`. | -| `cte next` | **MAINTAIN** | -| `cte plan` | **MAINTAIN** | -| `cte save` | **MAINTAIN** | -| `cte test` | **MAINTAIN** | -| `sql validate` | **MAINTAIN** | -| `sql preview` | **MAINTAIN** | -| `sql set-final` | **MAINTAIN** | -| `sql export` | **MAINTAIN** | -| `sql explain` | **ERASE** — non è usato né testato dal workflow attuale. Va reintrodotto soltanto se l'analisi del piano diventa un passo esplicito del processo. | -| `sql save` | **ERASE** — duplica `sql set-final` e `sql export` e introduce un percorso di scrittura non utilizzato. | -| `datamart generate` | **MAINTAIN** | - -### Memory - -| Comando | Proposta | -|---|---| -| `memory promote` | **MAINTAIN** | -| `memory save-one` | **MAINTAIN** | -| `memory search` | **MAINTAIN** | -| `memory solved-index` | **MAINTAIN** | -| `memory solved-search` | **MAINTAIN** | -| `memory list` | **ENHANCE** — trasformarlo in una vista amministrativa paginata, con filtri, provenienza, stato e output `--json`; non mostrarlo nell'help base. | -| `memory show` | **ENHANCE** — mostrare provenienza immutabile, decisione sorgente, stato dell'indice e riferimenti necessari a una correzione consapevole. | -| `memory update` | **ENHANCE** — limitare l'aggiornamento ai campi modificabili, mostrare un diff prima della conferma e impedire modifiche alla provenienza. | -| `memory delete` | **ENHANCE** — richiedere identificatore esatto e conferma esplicita, mostrare l'impatto e verificare la rimozione coerente da registro e indice. | -| `memory index` | **ENHANCE** — riposizionarlo come comando di repair: prima rileva il drift, poi ricostruisce soltanto con conferma e verifica finale. Non deve sembrare un'operazione ordinaria. | -| `memory clear` | **ERASE** — cancellazione globale non usata e troppo facile da eseguire per errore; backup/ripristino e cancellazione selettiva sono percorsi più sicuri. | -| `memory migrate` | **ERASE** — migrazione legacy una tantum; non esistono dati di produzione da preservare e la nuova architettura può partire direttamente dal formato corrente. | - -### Preprocessing, evidence e indici - -| Comando | Proposta | -|---|---| -| `preprocess dwh` | **MAINTAIN** | -| `preprocess evidence` | **MAINTAIN** | -| `vector index-schema` | **MAINTAIN** | -| `evidence extract` | **ERASE** — è un primitivo superato dalla pipeline versionata `preprocess evidence`; l'eventuale logica condivisa resta interna. | -| `evidence index` | **ERASE** — è un secondo primitivo superato dalla stessa pipeline, che già gestisce materializzazione, indicizzazione, versionamento e resume. | -| `lsh build` | **ERASE** — la costruzione LSH è già uno step di `preprocess dwh`; mantenere due ingressi permette esecuzioni parziali incoerenti. | -| `lsh query` | **ERASE** — probe visuale senza chiamanti, test o documentazione operativa; il workflow usa `search find`. | -| `vector init` | **ERASE** — il controllo di Qdrant ed embedder è già coperto dal reconciler della collezione, da `ollama ensure` e dal nuovo `tht doctor`. | - -### Formule di concetto - -| Comando | Proposta | -|---|---| -| `formula save` | **ERASE** — non ha chiamanti, test o un flusso di approvazione completo. Il formato e lo store possono restare disponibili alla ricerca finché non viene progettata una vera curation. | -| `formula list` | **ERASE** — appartiene allo stesso sottosistema incompleto; un futuro flusso deve progettare insieme creazione, approvazione, elenco, modifica e cancellazione. | - -## Impatto sulla UX - -I 55 comandi `MAINTAIN` comprendono molti contratti macchina intoccabili. Mantenerli non implica -mostrarli tutti nell'help principale. La futura CLI unica può conservare gli stessi percorsi per -backend, gate e job, mostrando all'utente soltanto i gruppi operativi di primo livello. diff --git a/docs/reports/l2-run-report-2026-06-27.md b/docs/reports/l2-run-report-2026-06-27.md deleted file mode 100644 index c65008d8..00000000 --- a/docs/reports/l2-run-report-2026-06-27.md +++ /dev/null @@ -1,126 +0,0 @@ -# L2 Run Report — 2026-06-27 (sessione cardioversione + ablazione) - -> Esito della prima sessione L2 end-to-end dopo il porting CLI+skill (Onda -1→4 + -> Skill + 0b). Sessione non-deterministica, esito informativo non bloccante per il -> "done" del porting codice (come da piano L2.1, riga 1187). - -## Setup al momento del run - -- Pi: `@earendil-works/pi-coding-agent`, provider `zai`, model `glm-5.2` (default). -- Harness: Onda -1→4 + Skill committate; Onda 0b (workspace cliente `tht-workspace-psd`, - indice LSH 75737 valori, evidence 35). -- `.env` popolato, VPN OK, DWH REST 200, Ollama UP. -- Pre-run fix applicati in questa sessione: `_YamlModel.to_yaml` (bloccava `session new`), - `config/tht.yaml` symlink al workspace cliente (il gate chiama `tht` senza `-c`). - -## Come è partita la sessione - -Lancio `pi --mode rpc` + `/nuova-domanda ""`. -**Nota critica su `--mode rpc`:** la TUI interattiva di Pi (`pi` senza `--mode`) **non -renderizza** i widget `extension_ui_request` del gate (gestiti solo in `modes/rpc/`). -La modalità RPC emette i widget come JSONL su stdio per un client esterno — che non -esiste ancora in ThothII. Il run è stato possibile solo perché il modello, non vedendo -UI, ha operato via shell/tool fino al blocco fatale (vedi bug #4). - -## Cosa ha fatto il modello (transcript: 117 eventi, 365KB) - -Sessione Pi: `~/.pi/agent/sessions/--Users-mp-projects-ThothII-harness--/2026-06-27T13-44-51...jsonl`. -Sessione tht: `tht-workspace-psd/sessions/2026-06-27-134553-crea-una-lista...` (status: open, F1). - -Il modello ha lavorato molto e correttamente nel dominio: -- Ha creato la sessione, caricato la skill, iniziato F1. -- Ha eseguito ricerche semantiche (evidence + LSH), individuato le tabelle centrali - (`fact_cardioversione_elettrica`, `fact_see_ablazione`, `dim_patient`). -- Ha letto evidence molto pertinenti (esempio NLQ, glossario coorti/universi), costruendo - un quadro dominio corretto e verificato sulle tabelle reali. - -Il workflow **non è avanzato oltre F1**: nessuna decisione registrata -(`review_decisions.jsonl` assente). Il modello si è arenato su bug di porting (sotto). - -## Bug di porting emersi (4, di cui 1 fatale) - -### #1 — `tht` non nel PATH del processo Pi [basso] -Il gate chiama `execFileSync("tht", args, {cwd: ctx.cwd})`. L'eseguibile nel venv non è nel -PATH di Pi. Il modello ha creato un wrapper in `~/.local/bin` (workaround). -**Fix root-cause:** installare `tht` in una dir nel PATH (pip install -e . con entry point -globale, o symlink `/usr/local/bin/tht -> harness/.venv/bin/tht`). - -### #2 — `phase show` non passava il config [medio, FIXATO] -`phase_cmd._cfg()` non passava il config a `_load_config_or_exit()`, quindi falliva con -"File non trovato config/tht.yaml" quando non c'era `THT_WORKSPACE` env. -**Fix applicato (dal modello in sessione, validato e pulito):** `_cfg()` ora risolve -`THT_WORKSPACE`/`THT_CONFIG` env, poi fallback a `config/tht.yaml` (stessa convenzione di -`CONFIG_OPT`). Testato: `phase show` funziona. Suite 165 passed. - -### #3 — `session check` signature inconsistente [basso, da verificare] -Il modello ha notato che `session check` prende la sessione come argomento posizionale, -non `--session` come gli altri cmd. Da verificare e allineare. - -### #4 — `ctx.sendRaw is not a function` [FATALE, blocca tutti i widget] -Il gate `tht-gate.js:164` emette i widget con `ctx.sendRaw({type:"extension_ui_request"...})`. -Il runtime Pi installato **non espone `ctx.sendRaw`** sul context delle extensions. Il -modello l'ha verificato leggendo le type definitions (`ExtensionContext` espone `ui`, -`mode`, `hasUI`, `cwd`, `abort` — ma non `sendRaw`). **Nessun widget può essere emesso in -nessuna modalità.** Questo ha fermato il workflow a F1. - -## Analisi del bug #4 (mismatch architetturale, non un typo) - -Il commento nel gate stesso (righe 2-6) dice: *"REWRITE of the reference implementation... -replaces ctx.ui.* blocking primitives"*. Il porting ha **sostituito** i dialog nativi di Pi -con `ctx.sendRaw`, presumendo un'API widget-descriptor diretta che **questa versione di Pi -non espone alle extensions**. Verifiche sul runtime installato: - -- `ctx.sendRaw`: **non esiste** (0 refs in `core/extensions/`). -- `extension_ui_request`: emesso **solo dal runtime** (`modes/rpc/rpc-mode.js`), come - traduzione dei dialog nativi (`ctx.ui.select` → `{method:"select"}`), non come API per - le extensions. -- Canali disponibili in RPC mode per ricevere una decisione umana (enum chiuso): - `ctx.ui.select` / `confirm` / `input` / `editor` (+ `notify` one-way). I `method` di - `extension_ui_request` sono: select/confirm/input/editor/notify/setWidget/setStatus/ - set_editor_text. **Nessun method custom** per widget-descriptor. -- `ctx.ui.custom` (usato da ChironeWp3 per il multiselect TUI): in RPC mode è un **no-op** - (`return undefined`, commento: "Custom UI not supported in RPC mode"). - -### Conseguenza per i 6 widget della spec §4 -- `select` (scelta singola) → ✅ `ctx.ui.select` -- `confirm` (approvazione) → ✅ `ctx.ui.confirm` -- `input` (testo libero / Altro) → ✅ `ctx.ui.input` -- `multiselect` (scelta multipla — **critico per F4 schema-linking**) → ❌ nessun canale - in RPC. Solo `ctx.ui.custom` lo faceva (TUI only). - -Il multiselect è il vero ostacolo. F4 richiede di promuovere/escludere **più** tabelle/ -colonne in una volta. - -## Opzioni per il design del gate (decisione architetturale APERTA) - -Il design del gate influenza tutta la relazione harness↔backend↔FE. Non è un fix da -inserire in coda a una sessione di porting; merita brainstorming dedicato. Opzioni: - -- **A — Torna a `ctx.ui.*` nativi.** Il gate riscrive `emitAndWait` su `ctx.ui.select`/ - `confirm`/`input` (come ChironeWp3 originale). Multiselect F4 emulato (serie di select, o - un input). Funziona con il Pi installato ora. Perde i widget-descriptor ricchi spec §4; - il FE riceve method nativi, la mappatura kind→method va nel backend/FE. -- **B — Versione di Pi con API widget custom.** Verificare se un Pi più recente/preview - espone `ctx.sendRaw` o un method custom. Se sì, il gate attuale funziona. Rischio: - inseguire un'API magari non pubblica; aggiornare Pi può rompere provider/auth. -- **C — Wrapper ibrido (valutato, NON realizzabile).** Registrare un handler che intercetti - gli `extension_ui_request` nativi e li arricchisca nel formato widget-descriptor spec §4 - prima di mandarli al client RPC. **Scartato:** non c'è canale per widget-descriptor custom - in RPC (method è enum chiuso, `ctx.ui.custom` è no-op in RPC). - -## Artefatti prodotti - -- `tht-workspace-psd/sessions/2026-06-27-134553-.../` (manifest + question.md, F1, open). -- Sessione Pi transcript (vedi sopra) — fonte primaria per il debug. -- Fix codice: `_cfg()` in `phase_cmd.py` (bug #2), applicato pulito. - -## Conclusione - -**La sessione L2 ha colto bug di porting reali — ha fatto il suo lavoro.** Il loop -skill→LLM→gate funziona nel dominio (ricerche, evidence, quadro corretto) ma si blocca a -F1 sul bug fatale #4. Tre dei quattro bug sono risolvibili a basso costo (#1, #2 fixato, -#3); il #4 è una decisione di design del gate che richiede brainstorming prima del codice. - -**Stato del porting codice (Onda -1→4 + Skill + 0b):** completo e verificato a livello L0/L1 -(165 passed) + L2 value-grounding (PASS). I bug L2 emersi sono incrementi di qualità, non -regressioni del porting — L2 coglie ciò che L0/L1 per design non possono. diff --git a/docs/skill-tht-sessione.md b/docs/skill-tht-sessione.md deleted file mode 100644 index 9c2d70c4..00000000 --- a/docs/skill-tht-sessione.md +++ /dev/null @@ -1,8 +0,0 @@ -# Testo della skill `tht-sessione` - -Questa pagina pubblica il testo completo della skill operativa usata dall'harness Pi. -La sorgente è [harness/.pi/skills/tht-sessione/SKILL.md](../harness/.pi/skills/tht-sessione/SKILL.md). - -Il blocco seguente viene incluso direttamente dal file sorgente durante il build MkDocs: non è una copia manuale. - ---8<-- "harness/.pi/skills/tht-sessione/SKILL.md" diff --git a/docs/superpowers/2026-06-27-stato-e-ripresa.md b/docs/superpowers/2026-06-27-stato-e-ripresa.md deleted file mode 100644 index a3cfdd56..00000000 --- a/docs/superpowers/2026-06-27-stato-e-ripresa.md +++ /dev/null @@ -1,46 +0,0 @@ -# ThothII — Stato e come riprendere - -**Aggiornato:** 2026-06-27 (fine sessione) -**Branch corrente:** `main` (HEAD `b909f3e`) - -## Cosa è fatto (tutto su `main`) - -1. **Harness** (`harness/`) — completo e RPC-ready. Piano: [docs/superpowers/plans/2026-06-27-harness-rpc-readiness.md](plans/2026-06-27-harness-rpc-readiness.md). -2. **Backend** (`backend/`) — completo. Spec: [docs/superpowers/specs/2026-06-27-backend-design.md](specs/2026-06-27-backend-design.md). Piano: [docs/superpowers/plans/2026-06-27-backend-implementation.md](plans/2026-06-27-backend-implementation.md). - -Entrambi sviluppati con subagent-driven development (un implementer + review per task, più review finale del branch). I branch `feat/harness-rpc-readiness` e `feat/backend-implementation` sono già merged in `main` (si possono cancellare: `git branch -d feat/harness-rpc-readiness feat/backend-implementation`). - -## Verifica che sia tutto verde (primo comando di domani) - -```bash -cd ~/projects/ThothII/backend && npm test && npm run build -cd ../harness && source .venv/bin/activate && pytest -q --ignore=tests/l2 && npm test -``` -Atteso: backend 32 test + tsc OK; harness 233 pytest + 24 node:test. - -## Il contratto chiave (per non perderlo) - -Lo spike ha scoperto che il gate non poteva usare `ctx.sendRaw` (inesistente in Pi). Il **wire contract reale**: il gate emette via `ctx.ui.input(JSON.stringify(descriptor))` → Pi manda `{type:"extension_ui_request", id, method:"input", title:""}` → il client risponde `{type:"extension_ui_response", id, value:""}`. Il backend (`SessionBridge`) decodifica `title`→descriptor e ri-codifica la risposta in `value`. Il contratto verso il frontend (`ui_request`/`ui_response`) NON cambia. - -## Due strade per riprendere (scelta rimandata) - -### A) Validazione end-to-end con Pi reale (rischio residuo più importante) -I test sono tutti deterministici contro `fake-pi-rpc`; il loop con **Pi reale** non è mai stato eseguito. Richiede VPN attiva + `pi` (GLM 5.2) configurato + `harness/.env` popolato + il symlink `harness/config/tht.yaml`. -Due assunzioni da provare dal vivo: (1) il comando RPC `prompt` attiva l'`input` hook del gate; (2) Pi serializza verbatim il descriptor nel `title` di `ctx.ui.input`. -Come: avviare il backend (`cd backend && npm run dev`), fare `POST /sessions` con una domanda reale, aprire l'SSE `GET /sessions/:id/events`, verificare che arrivi il widget F1 e che la risposta avanzi. Se fallisce, il fix è localizzato a `emitAndWait` (gate) + il `fake-pi-rpc`. - -### B) Frontend (terzo progetto) -Brainstorming → spec → piano del **frontend** (React/Next/ShadCn/AGGrid), come da architettura [docs/superpowers/specs/2026-06-25-thothii-architecture-design.md](specs/2026-06-25-thothii-architecture-design.md) §6. Consuma SOLO la REST+SSE del backend (contratto già stabile). Avviare con la skill `superpowers:brainstorming`. - -## Backlog non bloccante (da chiudere quando si tocca l'area) - -- **`GET /models`** torna `[]` di default (seam `listModels` pronto): cablare `pi --list-models` o uno spawn Pi effimero perché la tendina FE si popoli. -- **`resume()`** manda il kickoff `/nuova-domanda` invece di `/riprendi-sessione`: per il Pi reale `spawnFor` deve poter scegliere il kickoff di ripresa (altrimenti la sessione ripresa riparte da F1). -- **Multi-workspace lato Pi/gate**: il backend è workspace-aware, ma il gate usa il symlink `config/tht.yaml` (mono-workspace MVP). Per il multi-workspace reale il gate dovrebbe accettare un env `THT_CONFIG`/workspace. -- **`SseHub`**: i `Set` per-sessione vuoti non vengono rimossi (leak minore su processi long-running). -- **Copertura test** da estendere: route `steer`/`close`/404, handler SSE, percorsi `notify`/`oidc`; test gate per `/riprendi-sessione` RPC e i path di re-present di `emitAndWait` (cancel/id-mismatch/JSON-invalido). -- `GET /sessions/:id/events` su id inattivo apre uno stream 200 vuoto invece di 404. - -## Nota operativa - -Il ledger di esecuzione subagent-driven è in `.git/sdd/progress.md` (non versionato): contiene la cronologia task→commit→review. Utile se serve ricostruire perché una scelta è stata fatta. diff --git a/docs/superpowers/plans/2026-06-25-harness-implementation.md b/docs/superpowers/plans/2026-06-25-harness-implementation.md deleted file mode 100644 index f59bf21b..00000000 --- a/docs/superpowers/plans/2026-06-25-harness-implementation.md +++ /dev/null @@ -1,1697 +0,0 @@ -# ThothII — Harness Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build `harness/` — the self-contained Pi layer (CLI `nsp` Python + gate extension JS + skills markdown + `.pi/`) that runs the 8-phase NL→SQL workflow emitting/consuming widget-descriptor JSON, derived from ChironeWp3 as a validated-and-adapted starting point (not assumed reliable). - -**Architecture:** Port the relevant ChironeWp3 files into `harness/` task-by-task, then adapt each to the ThothII contract: `workflow.yaml` as the single source of workflow truth (F2), an "effective decisions as of pointer" view + `teardown_to_phase` for correct rollback (D15), a per-step task-document generator with enforced bounds for a 35B/<200k model (D16), the dual vector API key + `memory save-one` (D11), free-text interpretation (D13), value/formula grounding (D14), and the gate rewritten to emit/consume the 6-widget descriptor taxonomy (D2/D4). - -**Tech Stack:** Python ≥3.12 (typer, pydantic v2, pyyaml, sqlalchemy 2.0, psycopg2-binary, datasketch, sqlglot, pytest, testcontainers[postgres]); Node/JS for the Pi gate extension; Ollama embeddings (nomic-embed-text-v2-moe); Pi + GLM 5.2 for L2 tests. Testing: two levels (L1 fake + L2 real), see Testing Strategy below. - -**Reference spec:** `docs/superpowers/specs/2026-06-25-thothii-architecture-design.md` (all D1–D16 and §4.*). - -**Source of ported code:** `/Users/mp/projects/ThothII/ChironeWp3/` (read-only reference; never modify it). - ---- - -## Testing Strategy — three levels (L0 testcontainers, L1 fake, L2 real) - -This plan uses a **three-level test strategy**. The split is mandatory because the skill→LLM→tool-call loop cannot be exercised without a real model, and the DB-touching ported code needs a real database to validate (the "not assumed reliable" principle is hollow without it). - -### ⚠️ The honest headline: the core of the system has NO automated regression coverage - -The **skill→LLM→gate loop** — the heart of the system — is covered **only by L2** (manual, non-deterministic, slow, requires credentials+VPN). **There is no automated regression protection on it.** This is a deliberate, conscious choice, not an accidental side-effect: the LLM is non-deterministic and requires a configured Pi + network, so it cannot live in the fast automated loop. **Consequence: agentic-behavior regressions surface at pre-release L2 runs, not at commit.** Accept this and plan L2 runs before any release. A partial automated net for this gap would be a **fake-Pi** that mocks the Pi runtime and lets the gate glue run in CI; that is documented as a **cross-cutting follow-up** (built alongside the backend plan, not in this plan). - -### L0 — testcontainers, local Postgres, runs on every local run (CI) - -**What:** integrity tests of ported DB-touching modules against a real Postgres in a Docker container (testcontainers). No LLM, no remote network. Docker is available on the dev machine, so L0 is always-on locally. - -**Dependencies:** Docker (present in the dev env). No credentials, no VPN. - -**Coverage:** the ported logic that speaks to a DB — `db/connection` (read-only enforcement, exit code 2 if writable), `db/introspect`, `db/sampling`, `db/execute`, `search/RRF` + LSH with real data, `vectorstore/store` direct-load path. This is where "not assumed reliable" gains real teeth for the data layer. - -### L1 — fake data, deterministic, runs on every local run (CI) - -**What:** logic-pure tests with fake data (`tmp_path`, fixtures, mocks). No DB, no LLM, no network. - -**Coverage (honest):** -- Python logic-pure: `workflow.yaml` loading, `effective_decisions`, `teardown_to_phase`, `generate_task_doc`, `aggregate_lsh_multi` (on fake hits), formula store read/write, `decision_retracted`, CLI contract tests (valid input → well-formed JSON). -- **Gate builder functions (pure, in JS, tested in JS)**: the widget-descriptor builders produce the correct JSON given params. Tested in-language (node:test/vitest), no Python↔JS bridge, no Python mirror. - -**Honest limitation (load-bearing):** L1 can test the gate **builders** (pure functions) but **NOT the gate glue** — registration, emission via `ctx.sendRaw`, the no-limbo loop, the anti-bypass hooks. The glue depends on the Pi runtime (`pi.registerTool`, `ctx.sendRaw`, `emitAndWait`) and cannot run without either a real Pi or a fake-Pi mock. **The glue is tested only at L2.** Fuzzy/negative tests on the glue (malformed tool params, out-of-order calls, orphan responses) likewise live at L2; only fuzzy tests on the pure builders live in L1. - -### L2 — real LLM + real remote DB, manual/pre-release, NOT automated - -**What:** end-to-end sessions with GLM 5.2 + the real Chirone datawarehouse and pgvector, reached via REST over VPN. Plus the gate-glue validation (the part L1 cannot reach). - -**Dependencies (all required, skip if missing):** -- LLM: Pi configured locally with GLM 5.2 (already ready). -- DB: the remote Supabase endpoints (see "L2 connection params" below), reachable via VPN. -- `harness/.env` populated with the API keys + CA path. - -**Modes:** -- **Human-in-the-loop (default):** the test runs the harness against GLM 5.2 and the reviewer answers each gate widget via the terminal, prompted by the test. Used for exploratory validation and for exercising the gate glue (which L1 cannot). -- **Pre-scripted answers (opt-in):** for specific cases (free-text "Altro", value grounding "ablazione", rollback), the test supplies canned reviewer answers automatically and asserts the outcome — no human typing. Used to test exact behaviors deterministically with the real model. - -**Coverage (honest):** validates the assumption L1 cannot — that GLM 5.2 produces tool calls the gate accepts, that the skill's prompts lead to the expected interaction shape, that the gate glue handles real tool-call sequences (incl. Altro/Rifiuta/rollback), that value grounding and formula approval surface correctly on the real schema, that `memory save-one` upserts to the real pgvector. Closes the skill→LLM→gate loop AND exercises the gate glue. - -**Honest limitation:** L2 is non-deterministic (the model may behave differently across runs) and slow/costly. It is a pre-release safety net, not a regression gate. A small, curated set of scenarios. - -### L2 connection params (from the operator) - -- **DWH (read-only):** `https://supabase-aritmolab.policlinicosandonato.it/dwh/` — PostgREST, schema `datawarehouse` + `datawarehouse_marts`, role `dwh_reader`. Header `X-API-Key: ${THOTH_DWH_API_KEY}`. -- **pgvector reader:** `https://supabase-aritmolab.policlinicosandonato.it/vector/v1/` — role `vector_reader` (`search_similar`, `list_tables`). Header `X-API-Key: ${THOTH_VEC_API_KEY}` (same value as `THOTH_DWH_API_KEY` — shared reader key). -- **pgvector writer (upsert-only):** same URL as reader, different key → role `vector_writer` (`existing_vector_hashes`, `upsert_vector_records`, no DELETE). Header `X-API-Key: ${THOTH_VEC_WRITE_API_KEY}`. -- **TLS:** self-signed, CA = leaf. On the operator's Mac, a local copy of the CA bundle must be present; `${THOTH_SSL_CA}` points to its local path. (Server path `/etc/nginx/ssl/policlinicosandonato.it.fullchain.crt` is unreachable from outside.) -- **Workspace:** `chirone-test`. -- **Test question (L2):** "dammi la lista dei pazienti che hanno fatto un'ablazione nel 2025" — exercises D14 value grounding + formula on the "ablazione" multi-column case. - -### API key handling (security) - -- All keys live **only** in `harness/.env` (gitignored). Never in code, never in the plan, never committed. -- `.env.example` is committed with variable names and empty values. -- L2 tests load `.env` via `python-dotenv` at startup; if any required var is missing/empty, the L2 test is **skipped with a clear message** (not failed), so L0/L1 never break for missing credentials. -- Tests never log key values. URLs in logs are fine; secrets are masked. -- ⚠️ The operator should rotate the key that appeared in chat transcripts. - -### Test marker convention - -- **L0 tests:** marker `@pytest.mark.l0`. Run always locally (Docker present). Auto-skip if Docker unavailable (rare). File naming: `tests/l0/test_*.py`. -- **L1 tests:** no marker (default, run always). File naming: `test_*.py` under `tests/`. Gate builder tests live in JS (node:test/vitest) under `harness/.pi/extensions/gate/__tests__/` and run via `npm test` alongside pytest. -- **L2 tests:** marker `@pytest.mark.l2`. Skipped automatically when `.env` incomplete. File naming: `tests/l2/test_*.py`. Default run: `pytest -m "not l2"` (L0+L1 only); pre-release: `pytest -m l2` (L2 only). - ---- - -## File Structure - -``` -harness/ -├── .pi/ ← Pi project config (ported + adapted) -│ ├── settings.json -│ ├── prompts/ ← /nuova-domanda, /riprendi-sessione -│ ├── skills/nsp-sessione/ ← SKILL.md + sub-files (adapted: task-doc, D13/D14) -│ └── extensions/ -│ ├── nsp-gate.js ← REWRITTEN: 6-widget descriptor emit/consume (glue, verified at L2) -│ ├── reserved-labels.mjs ← ported -│ └── gate/ -│ ├── builders.js ← NEW: pure widget-builder functions (L1, JS-tested) -│ └── __tests__/ ← node:test golden + fuzzy (L1) -│ ├── builders.test.js -│ └── golden/ -├── nsp/ ← Python package (ported + adapted) -│ ├── __init__.py -│ ├── config.py ← ported + vector_write_rest (D11) -│ ├── workspace.py ← NEW: YAML load + ${VAR} expand (D3 confine) -│ ├── workflow.py ← NEW: reads workflow.yaml (F2) -│ ├── phase.py ← REWRITTEN: data-driven + effective_decisions (D15) -│ ├── teardown.py ← NEW: teardown_to_phase (D15) -│ ├── taskdoc.py ← NEW: per-step task document generator (D16) -│ ├── decisions.py ← ported + decision_retracted (D15) -│ ├── cli/ -│ │ ├── __init__.py ← Typer app `nsp` -│ │ ├── _guards.py ← ported (require_vector_write_allowed etc.) -│ │ ├── workspace_cmd.py ← NEW: nsp workspace list/show -│ │ ├── phase_cmd.py ← adapted: + phase meta --json (F2) -│ │ ├── decision_cmd.py ← adapted: + retraction (D15) -│ │ ├── memory_cmd.py ← adapted: + save-one (D11) -│ │ ├── search_cmd.py ← adapted: + --kind formula (D14b) -│ │ ├── session_cmd.py ← adapted: + consistency check (D15) -│ │ └── (sql_cmd, cte_cmd, db_cmd, lsh_cmd, vector_cmd, evidence_cmd, schema_cmd — ported) -│ ├── db/ ← ported (connection, introspect, sampling, execute) -│ ├── rest/ ← ported (client, execute) -│ ├── search/ ← ported (combined_search, RRF) + formula retrieval (D14b) -│ ├── mschema/ ← ported (models, render, eligibility) -│ ├── vectorstore/ ← ported + dual-key rest_client (D11) -│ ├── evidence/ ← ported + formula kind (D14b) -│ ├── memory.py ← ported -│ └── session/ ← ported (models, store, artifacts) -├── workflow.yaml ← NEW: single source of workflow truth (F2) -├── workspaces/ ← workspace YAML definitions (D3) -│ └── chirone.example.yaml -├── sessions/ ← session persistence (FS, append-only ledger) -├── tests/ -│ ├── conftest.py ← L2 skip-when-no-.env fixtures -│ ├── golden/ ← golden JSON for widget builders (L1) -│ ├── fixtures/ -│ │ └── scenarios/ ← static .jsonl fixtures (synthesized once by build-scenarios) -│ ├── test_workflow.py ← L1 -│ ├── test_phase_effective.py ← L1 -│ ├── test_teardown.py ← L1 -│ ├── test_taskdoc.py ← L1 -│ ├── test_workspace.py ← L1 -│ ├── test_vector_dual_key.py ← L1 -│ ├── test_memory_save_one.py ← L1 (mocked writer) -│ ├── test_freetext_interpretation.py ← L1 (rationale contract) -│ ├── test_db_connection.py ← L0 Livello B (testcontainers: read-only enforcement) -│ ├── test_rest_client.py ← L1 Livello B (mock transport: RPC call) -│ ├── test_mschema_render.py ← L1 Livello B (3 formats on fake mschema) -│ ├── test_mschema_eligibility.py ← L1 Livello B (eligibility rules on fake columns) -│ ├── test_db_introspect.py ← L0 Livello B (testcontainers: known schema) -│ ├── test_db_sampling.py ← L0 Livello B (testcontainers: known data) -│ ├── test_rrf.py ← L0 Livello B (testcontainers: RRF fusion with real data) -│ ├── test_cli_contract.py ← L1 Livello B (each CLI: valid input → well-formed JSON) -│ ├── l0/ ← testcontainers tests (need Docker) -│ │ └── __init__.py -│ └── l2/ ← L2 (real GLM 5.2 + remote DB, @pytest.mark.l2) -│ └── __init__.py -│ └── l2/ -│ ├── test_session_ablazione.py ← L2 (GLM 5.2 + real DWH, human or scripted) -│ ├── test_value_grounding_real.py ← L2 ("ablazione" on real schema) -│ └── test_memory_save_one_real.py ← L2 (real pgvector writer upsert) -├── scripts/ -│ ├── create_vector_writer_rpc.sql ← ported (writer RPC allowlist) -│ └── create_vector_reader_rpc.sql ← NEW (reader RPC, D11 §5.4 note) -├── pyproject.toml -└── .env.example -``` - ---- - -## Phase A — Foundation (cross-cutting infrastructure first) - -Rationale: every feature depends on these. `workflow.yaml` + `effective_decisions` + `teardown` + `taskdoc` are the substrate the gate and the D11/D13/D14 features build on. - -### Task A1: Scaffold the harness project - -**Files:** -- Create: `harness/pyproject.toml` -- Create: `harness/.env.example` -- Create: `harness/nsp/__init__.py` -- Create: `harness/.gitignore` - -- [ ] **Step 1: Create `pyproject.toml`** - -```toml -[project] -name = "nsp" -version = "0.1.0" -description = "ThothII harness — deterministic CLI for the NL→SQL workflow" -requires-python = ">=3.12" -dependencies = [ - "typer>=0.12", - "pydantic>=2.0", - "pyyaml>=6.0", - "sqlalchemy>=2.0", - "psycopg2-binary>=2.9", - "datasketch>=1.6", - "sqlglot>=23.0", - "python-dotenv>=1.0", - "rich>=13.0", - "requests>=2.31", - "tqdm>=4.66", -] - -[project.scripts] -nsp = "nsp.cli:app" - -[project.optional-dependencies] -dev = [ - "pytest>=8.0", - "testcontainers[postgres]>=4.0", - "ruff>=0.5", -] - -[tool.ruff] -line-length = 100 - -[tool.pytest.ini_options] -testpaths = ["tests"] -``` - -- [ ] **Step 2: Create `.env.example`** (port the structure from `ChironeWp3/.env.example`, renamed to `THOTH_*` per §5.4) - -```bash -# ThothII profile: server (full rebuild) | workstation (read REST + optional upsert) -THOTH_PROFILE=server - -# Workspace DB (relational, direct transport) — example: chirone -THOTH_DB_HOST= -THOTH_DB_PORT=5432 -THOTH_DB_USER= -THOTH_DB_PASSWORD= - -# Vector REST — LETTURA (rpc search_similar) sul Supabase remoto -THOTH_VEC_REST_URL=https://host/vector/v1/ -THOTH_VEC_API_KEY= - -# Vector REST — SCRITTURA controllata (upsert only) da client remoti autorizzati -THOTH_VEC_WRITE_API_KEY= - -# Vector direct loading (server-only) -THOTH_VEC_HOST=localhost -THOTH_VEC_PORT=5438 -THOTH_VEC_USER=postgres -THOTH_VEC_PASSWORD= - -# Embeddings (Ollama) -THOTH_OLLAMA_URL=http://localhost:11434 - -# TLS (optional, internal CA) -THOTH_SSL_CA= -``` - -- [ ] **Step 3: Create `.gitignore`** - -``` -__pycache__/ -*.pyc -.env -.venv/ -sessions/ -indexes/ -*.egg-info/ -dist/ -build/ -``` - -- [ ] **Step 4: Create empty `nsp/__init__.py` and install** - -```bash -cd /Users/mp/projects/ThothII/harness -python -m venv .venv && source .venv/bin/activate -pip install -e ".[dev]" -``` - -- [ ] **Step 5: Verify `nsp` command is not yet wired (expected: import error) then commit** - -```bash -nsp --help 2>&1 | head -3 -``` -Expected: error (no `cli` module yet) — this is fine, we add it in Task A4. - -```bash -git add harness/ -git commit -m "feat(harness): scaffold project (pyproject, env, gitignore)" -``` - ---- - -### Task A2: Port config + create `workspace.py` (D3 confine) - -**Files:** -- Create: `harness/nsp/config.py` (ported + adapted from `ChironeWp3/src/psdwp3/config.py`) -- Create: `harness/nsp/workspace.py` -- Create: `harness/workspaces/chirone.example.yaml` -- Test: `harness/tests/test_workspace.py` - -- [ ] **Step 1: Port `config.py`** — copy `ChironeWp3/src/psdwp3/config.py` → `harness/nsp/config.py`. Rename the package prefix `psdwp3` → `nsp` everywhere. Verify it still defines: `DatabaseConfig`, `RestConfig`, `PathsConfig`, `EligibilityConfig`, `LshConfig`, `EvidenceSourcesConfig`, `EmbeddingsConfig`, `VectorConfig`, `SearchConfig`, `ExecutionConfig`, and the `Config` class with `vector_rest` + `vector_write_rest` (both `RestConfig | None`, see ChironeWp3 config.py:154-159) + `_expand_env` (lines 16-32). - -- [ ] **Step 2: Create `workspaces/chirone.example.yaml`** per spec §5.1 (relational + vector_db with rest/write_rest + evidence + embeddings + execution). - -- [ ] **Step 3: Write failing test for workspace loading** - -```python -# tests/test_workspace.py -from pathlib import Path -from nsp.workspace import load_workspace, WorkspaceError - -def test_load_workspace_expands_env_vars(monkeypatch, tmp_path): - monkeypatch.setenv("THOTH_VEC_API_KEY", "secret-reader") - monkeypatch.setenv("THOTH_VEC_WRITE_API_KEY", "secret-writer") - monkeypatch.setenv("THOTH_VEC_REST_URL", "https://example/vector/v1/") - yaml = tmp_path / "w.yaml" - yaml.write_text( - "name: test\n" - "relational:\n db_type: postgres\n transport: rest\n" - " rest: { base_url: 'https://dwh/', api_key: '${THOTH_VEC_API_KEY}' }\n" - "vector_db:\n collection: test_docs\n dim: 768\n" - " rest: { base_url: '${THOTH_VEC_REST_URL}', api_key: '${THOTH_VEC_API_KEY}' }\n" - " write_rest: { base_url: '${THOTH_VEC_REST_URL}', api_key: '${THOTH_VEC_WRITE_API_KEY}' }\n" - "embeddings:\n provider: ollama\n base_url: 'http://localhost:11434'\n model: m\n dim: 768\n" - ) - ws = load_workspace(yaml) - assert ws.vector_db.rest.api_key == "secret-reader" - assert ws.vector_db.write_rest.api_key == "secret-writer" - -def test_load_workspace_missing_env_raises(monkeypatch, tmp_path): - monkeypatch.delenv("THOTH_VEC_API_KEY", raising=False) - yaml = tmp_path / "w.yaml" - yaml.write_text( - "name: test\nrelational: { db_type: postgres, transport: rest, rest: { base_url: 'https://dwh/', api_key: '${THOTH_VEC_API_KEY}' } }\n" - "vector_db: { collection: c, dim: 768, rest: { base_url: 'https://v/', api_key: '${THOTH_VEC_API_KEY}' } }\n" - "embeddings: { provider: ollama, base_url: 'http://x', model: m, dim: 768 }\n" - ) - try: - load_workspace(yaml) - assert False, "should have raised" - except WorkspaceError as e: - assert "THOTH_VEC_API_KEY" in str(e) -``` - -- [ ] **Step 4: Run test, verify fail** - -Run: `cd harness && pytest tests/test_workspace.py -v` -Expected: FAIL — `ModuleNotFoundError: nsp.workspace` - -- [ ] **Step 5: Implement `workspace.py`** - -```python -# nsp/workspace.py -"""Workspace YAML loading — the single boundary for workspace configuration (spec D3). -Reads workspaces/.yaml, expands ${VAR} from env, validates via the Config model. -Future migration to a DB store would replace only this module. -""" -from __future__ import annotations -import os -from pathlib import Path -import yaml -from nsp.config import Config, ConfigError - -class WorkspaceError(Exception): - pass - -def _expand_str(s: str) -> str: - """Expand ${VAR} occurrences in s. Raises if a referenced var is unset.""" - PREFIX = "${" - SUFFIX = "}" - out: list[str] = [] - i = 0 - while i < len(s): - start = s.find(PREFIX, i) - if start == -1: - out.append(s[i:]) - break - out.append(s[i:start]) - end = s.find(SUFFIX, start + len(PREFIX)) - if end == -1: - raise WorkspaceError(f'Sintassi non valida (manca "}}"): {s[start:]}') - var = s[start + len(PREFIX) : end] - if var not in os.environ: - raise WorkspaceError(f"Variabile d'ambiente non definita: {var}") - out.append(os.environ[var]) - i = end + len(SUFFIX) - return "".join(out) - -def _expand_env(obj): - if isinstance(obj, str): - return _expand_str(obj) - if isinstance(obj, dict): - return {k: _expand_env(v) for k, v in obj.items()} - if isinstance(obj, list): - return [_expand_env(v) for v in obj] - return obj - -def load_workspace(path: str | Path) -> Config: - raw = yaml.safe_load(Path(path).read_text()) - try: - return Config.model_validate(_expand_env(raw)) - except ConfigError as e: - raise WorkspaceError(str(e)) from e -``` - -- [ ] **Step 6: Run test, verify pass** - -Run: `pytest tests/test_workspace.py -v` -Expected: 2 PASS - -- [ ] **Step 7: Commit** - -```bash -git add harness/nsp/config.py harness/nsp/workspace.py harness/workspaces/chirone.example.yaml harness/tests/test_workspace.py -git commit -m "feat(harness): port config + workspace.py YAML loader (D3)" -``` - ---- - -### Task A3: Create `workflow.yaml` + `workflow.py` (F2) — single source of truth - -**Files:** -- Create: `harness/workflow.yaml` -- Create: `harness/nsp/workflow.py` -- Test: `harness/tests/test_workflow.py` - -- [ ] **Step 1: Create `workflow.yaml`** (the 8-phase definition from spec §5.3) - -```yaml -# harness/workflow.yaml — single source of workflow truth (spec F2) -schema_version: 1 - -phases: - - id: F1 - name: chiaramento - advance: kind:phase - prerequisites: [] - artifacts_out: [] - - id: F2 - name: memoria - advance: auto_if_empty - prerequisites: [] - artifacts_out: [] - - id: F3 - name: riscrittura - advance: kind:phase - prerequisites: - - decision_exists: question_rewritten - artifacts_out: [question.md] - - id: F4 - name: schema_linking - advance: reviewer_decide - prerequisites: [] - artifacts_out: [schema_linking.json] - - id: F5 - name: sintesi - advance: kind:phase - prerequisites: - - file_validates: [schema_linking.json, SchemaLinking] - artifacts_out: [] - - id: F6 - name: cte - advance: auto_if_empty_or_skipped - prerequisites: - - any: - - decision_subject_exists: [phase_skipped, "phase:6"] - - all_ctes_approved: true - artifacts_out: [cte_plan.json, ctes/, cte_tests.json] - - id: F7 - name: sql_finale - advance: kind:phase - prerequisites: - - decision_exists: sql_approved - artifacts_out: [sql_final.sql] - - id: F8 - name: datamart - advance: reviewer_decide - prerequisites: - - any: - - decision_exists: datamart_requested - - decision_exists: datamart_declined - artifacts_out: [] - -decision_min_phase: auto -max_phase: auto -``` - -- [ ] **Step 2: Write failing test for workflow loading** - -```python -# tests/test_workflow.py -from nsp.workflow import load_workflow, PhaseSpec - -def test_workflow_loads_8_phases(): - wf = load_workflow() - assert len(wf.phases) == 8 - assert wf.phases[0].id == "F1" - assert wf.max_phase == 8 - assert wf.phase_by_num(1).name == "chiaramento" - -def test_decision_min_phase_derived(): - wf = load_workflow() - # question_rewritten is a prerequisite of F3 → min phase 3 - assert wf.decision_min_phase("question_rewritten") == 3 - # sql_approved is a prerequisite of F7 → min phase 7 - assert wf.decision_min_phase("sql_approved") == 7 - # unknown type → phase 1 (default) - assert wf.decision_min_phase("nonexistent_type") == 1 - -def test_phase_name_lookup(): - wf = load_workflow() - assert wf.phase_name(6) == "cte" - assert wf.phase_name(8) == "datamart" # the JS drift bug — F8 must be present - -def test_artifacts_out_per_phase(): - wf = load_workflow() - assert "schema_linking.json" in wf.phase_by_num(4).artifacts_out - assert "sql_final.sql" in wf.phase_by_num(7).artifacts_out -``` - -- [ ] **Step 3: Run test, verify fail** - -Run: `pytest tests/test_workflow.py -v` -Expected: FAIL — `ModuleNotFoundError: nsp.workflow` - -- [ ] **Step 4: Implement `workflow.py`** - -```python -# nsp/workflow.py -"""Reads workflow.yaml — the SINGLE source of workflow truth (spec F2, §5.3). -phase.py, the gate, and the skill all read from here. No more duplicated constants. -""" -from __future__ import annotations -from dataclasses import dataclass, field -from pathlib import Path -import yaml - -_WF_PATH = Path(__file__).resolve().parent.parent / "workflow.yaml" - -@dataclass -class PhaseSpec: - id: str - num: int - name: str - advance: str - prerequisites: list - artifacts_out: list[str] = field(default_factory=list) - -@dataclass -class Workflow: - schema_version: int - phases: list[PhaseSpec] - _decision_min_map: dict[str, int] = field(default_factory=dict) - - @property - def max_phase(self) -> int: - return len(self.phases) - - def phase_by_num(self, n: int) -> PhaseSpec: - return self.phases[n - 1] - - def phase_name(self, n: int) -> str: - return self.phase_by_num(n).name if 1 <= n <= self.max_phase else "?" - - def decision_min_phase(self, decision_type: str) -> int: - # A decision type's min phase = the earliest phase whose prerequisites - # reference it (via decision_exists/decision_subject_exists), else 1. - return self._decision_min_map.get(decision_type, 1) - -def _collect_decision_mins(phases: list[PhaseSpec]) -> dict[str, int]: - """Scan prerequisites for decision_exists / decision_subject_exists mentions.""" - mins: dict[str, int] = {} - def scan(node, phase_num: int): - if isinstance(node, dict): - for k, v in node.items(): - if k in ("decision_exists", "decision_subject_exists"): - if isinstance(v, list): - dtype = v[0] - else: - dtype = v - if dtype not in mins or phase_num < mins[dtype]: - mins[dtype] = phase_num - else: - scan(v, phase_num) - elif isinstance(node, list): - for item in node: - scan(item, phase_num) - for p in phases: - scan(p.prerequisites, p.num) - return mins - -def load_workflow(path: Path | str = _WF_PATH) -> Workflow: - raw = yaml.safe_load(Path(path).read_text()) - phases = [] - for i, p in enumerate(raw["phases"], start=1): - phases.append(PhaseSpec( - id=p["id"], num=i, name=p["name"], - advance=p["advance"], prerequisites=p.get("prerequisites", []), - artifacts_out=p.get("artifacts_out", []), - )) - return Workflow( - schema_version=raw.get("schema_version", 1), - phases=phases, - _decision_min_map=_collect_decision_mins(phases), - ) -``` - -- [ ] **Step 5: Run test, verify pass** - -Run: `pytest tests/test_workflow.py -v` -Expected: 4 PASS - -- [ ] **Step 6: Commit** - -```bash -git add harness/workflow.yaml harness/nsp/workflow.py harness/tests/test_workflow.py -git commit -m "feat(harness): workflow.yaml as single source of truth + workflow.py loader (F2)" -``` - ---- - -### Task A4: Port `decisions.py` + add `decision_retracted` (D15) - -**Files:** -- Create: `harness/nsp/decisions.py` (ported + adapted) -- Test: `harness/tests/test_decisions_retract.py` - -- [ ] **Step 1: Port `decisions.py`** — copy `ChironeWp3/src/psdwp3/session/decisions.py`. Rename package. Add `decision_retracted` to the `DecisionType` Literal. The `DecisionRecord` model gains an optional `retracts: int | None` field (the `decision_seq` being retracted, for step-level rollback). - -- [ ] **Step 2: Write failing test for retraction** - -```python -# tests/test_decisions_retract.py -from pathlib import Path -from nsp.decisions import append_decision, list_decisions, DecisionRecord - -def test_retracted_decision_in_audit_but_marked(tmp_path): - session = tmp_path / "s1" - session.mkdir() - append_decision(session, DecisionRecord(seq=1, ts="t", type="table_promoted", - subject="phase:4", detail="t1", rationale="r")) - append_decision(session, DecisionRecord(seq=2, ts="t", type="decision_retracted", - subject="phase:4", detail="retract", rationale="wrong", retracts=1)) - all_decisions = list_decisions(session) - assert len(all_decisions) == 2 # both in audit - assert all_decisions[1].retracts == 1 -``` - -- [ ] **Step 3: Run, verify fail, implement the `retracts` field + ensure `append_decision` writes it, verify pass.** (The ported `append_decision` already writes all fields; just ensure the new field is serialized. If using pydantic `model_dump`, add `retracts: int | None = None`.) - -Run: `pytest tests/test_decisions_retract.py -v` -Expected after implement: PASS - -- [ ] **Step 4: Commit** - -```bash -git add harness/nsp/decisions.py harness/tests/test_decisions_retract.py -git commit -m "feat(harness): port decisions.py + decision_retracted for step rollback (D15)" -``` - ---- - -### Task A5: Rewrite `phase.py` — data-driven + `effective_decisions` (D15 core fix) - -**Files:** -- Create: `harness/nsp/phase.py` (rewritten) -- Port: `harness/nsp/session/{models,store,artifacts}.py` (needed for SchemaLinking validation) -- Test: `harness/tests/test_phase_effective.py` - -This is the single most important architectural fix (spec §4.8). `phase.py` reads from `workflow.yaml`, and ALL helpers consult `effective_decisions()` instead of raw `list_decisions()`. - -- [ ] **Step 1: Port session models/store/artifacts** — copy `ChironeWp3/src/psdwp3/session/{models.py, store.py, artifacts.py}` → `harness/nsp/session/`. Rename package. These are needed for `SchemaLinking` validation in `advance_problems`. - -- [ ] **Step 2: Write failing test for effective_decisions** - -```python -# tests/test_phase_effective.py -from pathlib import Path -from nsp.phase import current_phase, effective_decisions -from nsp.decisions import append_decision, DecisionRecord - -def _d(session, seq, dtype, subject, **kw): - append_decision(session, DecisionRecord(seq=seq, ts="t", type=dtype, subject=subject, - detail=kw.get("detail", ""), rationale=kw.get("rationale", ""), - retracts=kw.get("retracts"))) - -def test_effective_decisions_excludes_pre_reopen_tail(tmp_path): - s = tmp_path / "s"; s.mkdir() - _d(s, 1, "phase_approved", "phase:1") - _d(s, 2, "phase_approved", "phase:2") - _d(s, 3, "phase_reopened", "phase:1") # rollback to phase 1 - _d(s, 4, "table_promoted", "phase:4") # stale: produced after reopen but for phase 4 (not current) - # After reopen to phase:1, the effective view truncates everything after the last phase_reopened. - eff = effective_decisions(s) - # The reopen at seq 3 means decisions after it (seq 4) are NOT effective, - # and the reopen itself sets the pointer. Effective = decisions up to and incl. reopen, - # then re-walked. The stale table_promoted at seq 4 must be excluded. - types = [d.type for d in eff] - assert "table_promoted" not in types - -def test_current_phase_after_reopen(tmp_path): - s = tmp_path / "s"; s.mkdir() - _d(s, 1, "phase_approved", "phase:1") - _d(s, 2, "phase_approved", "phase:2") - _d(s, 3, "phase_reopened", "phase:1") - assert current_phase(s) == 1 - _d(s, 4, "phase_approved", "phase:1") # re-approve after reopen - assert current_phase(s) == 2 - -def test_retracted_decision_excluded_from_effective(tmp_path): - s = tmp_path / "s"; s.mkdir() - _d(s, 1, "table_promoted", "phase:4", detail="t1") - _d(s, 2, "decision_retracted", "phase:4", retracts=1) - eff = effective_decisions(s) - types = [d.type for d in eff] - assert "table_promoted" not in types # retracted → excluded -``` - -- [ ] **Step 3: Run, verify fail** - -Run: `pytest tests/test_phase_effective.py -v` -Expected: FAIL - -- [ ] **Step 4: Implement `phase.py`** — the core. Port the fold logic from `ChironeWp3/src/psdwp3/session/phase.py` but: - - Read `max_phase`, `phase_name`, `decision_min_phase`, `advance_problems` rules from `workflow.yaml` via `load_workflow()`. - - Add `effective_decisions(session)`: replay the ledger; when hitting a `phase_reopened phase:N`, truncate everything after it in the effective view AND re-walk from phase N. When hitting a `decision_retracted`, exclude the retracted seq. - - Rewrite `advance_problems(phase)`, `approved_ctes`, `substantive_count_current_phase`, `auto_advance_eligible` to consult `effective_decisions()` instead of `list_decisions()`. - -Reference skeleton (the `effective_decisions` core — the load-bearing part): - -```python -# nsp/phase.py (key function) -from nsp.decisions import list_decisions -from nsp.workflow import load_workflow - -def effective_decisions(session_dir) -> list: - """The canonical effective view. ALL helpers MUST use this, not list_decisions(). - Replays the ledger: phase_reopened truncates the tail; decision_retracted excludes the target. - """ - all_d = list_decisions(session_dir) - retracted_seqs = {d.retracts for d in all_d if d.type == "decision_retracted" and d.retracts} - # Find the LAST phase_reopened; everything strictly after it is stale. - last_reopen_idx = -1 - for i, d in enumerate(all_d): - if d.type == "phase_reopened": - last_reopen_idx = i - if last_reopen_idx >= 0: - effective = all_d[: last_reopen_idx + 1] - else: - effective = list(all_d) - # Exclude retracted decisions (by seq), and the retraction records themselves. - return [d for d in effective if d.seq not in retracted_seqs and d.type != "decision_retracted"] -``` - -Implement `current_phase(session_dir)` as the fold over `effective_decisions(session_dir)` (not over raw list). Implement `advance_problems(phase, session_dir)` by evaluating the `prerequisites` list from `workflow.yaml` for that phase against `effective_decisions`. Reuse the prerequisite predicate types: `decision_exists`, `decision_subject_exists`, `file_validates`, `all_ctes_approved`, `any`, `all`. - -- [ ] **Step 5: Run, verify pass** - -Run: `pytest tests/test_phase_effective.py -v` -Expected: 3 PASS - -- [ ] **Step 6: Commit** - -```bash -git add harness/nsp/phase.py harness/nsp/session/ harness/tests/test_phase_effective.py -git commit -m "feat(harness): rewrite phase.py data-driven + effective_decisions (D15 core, F2)" -``` - ---- - -### Task A6: Create `teardown.py` — `teardown_to_phase` (D15 teardown) - -**Files:** -- Create: `harness/nsp/teardown.py` -- Test: `harness/tests/test_teardown.py` - -- [ ] **Step 1: Write failing test** - -```python -# tests/test_teardown.py -from pathlib import Path -from nsp.teardown import teardown_to_phase, TeardownReport - -def test_teardown_to_phase_4_deletes_phase5plus_artifacts(tmp_path): - s = tmp_path / "sess"; s.mkdir() - (s / "schema_linking.json").write_text("{}") # F4 artifact - (s / "cte_plan.json").write_text("[]") # F6 artifact - (s / "ctes").mkdir() - (s / "ctes" / "x.sql").write_text("SELECT 1") # F6 artifact - (s / "sql_final.sql").write_text("SELECT 1") # F7 artifact - report = teardown_to_phase(s, target_phase=4) - assert (s / "schema_linking.json").exists() # F4 preserved (target is 4) - assert not (s / "cte_plan.json").exists() # F6 deleted (>4) - assert not (s / "ctes").exists() # F6 dir deleted - assert not (s / "sql_final.sql").exists() # F7 deleted - assert "cte_plan.json" in report.deleted_files - assert "sql_final.sql" in report.deleted_files - -def test_teardown_records_deleted_in_ledger(tmp_path): - # teardown itself appends a phase_reopened decision (the rollback marker) - # — verifying that is covered by test_phase_effective; here we check the report. - s = tmp_path / "sess"; s.mkdir() - (s / "sql_final.sql").write_text("SELECT 1") - report = teardown_to_phase(s, target_phase=4) - assert len(report.deleted_files) >= 1 -``` - -- [ ] **Step 2: Run, verify fail** - -Run: `pytest tests/test_teardown.py -v` - -- [ ] **Step 3: Implement `teardown.py`** - -```python -# nsp/teardown.py -"""Artifact teardown on rollback (spec D15, §4.8). -Deletes every artifact whose producing phase > target, using the artifacts_out map -from workflow.yaml. Recomputes derived state. Fixes the orphaned-CTE-blocks-finalize bug. -""" -from __future__ import annotations -from dataclasses import dataclass, field -from pathlib import Path -from nsp.workflow import load_workflow - -@dataclass -class TeardownReport: - target_phase: int - deleted_files: list[str] = field(default_factory=list) - -def teardown_to_phase(session_dir: str | Path, target_phase: int) -> TeardownReport: - session_dir = Path(session_dir) - wf = load_workflow() - report = TeardownReport(target_phase=target_phase) - for phase in wf.phases: - if phase.num <= target_phase: - continue - for artifact in phase.artifacts_out: - target = session_dir / artifact.rstrip("/") - if artifact.endswith("/"): - # directory artifact (e.g. ctes/) - if target.exists(): - for f in target.glob("*"): - f.unlink() - report.deleted_files.append(f.name) - target.rmdir() - else: - if target.exists(): - target.unlink() - report.deleted_files.append(artifact) - return report -``` - -- [ ] **Step 4: Run, verify pass** - -Run: `pytest tests/test_teardown.py -v` -Expected: 2 PASS - -- [ ] **Step 5: Commit** - -```bash -git add harness/nsp/teardown.py harness/tests/test_teardown.py -git commit -m "feat(harness): teardown_to_phase — artifact teardown on rollback (D15)" -``` - ---- - -### Task A7: Create `taskdoc.py` — per-step task document generator (D16) - -**Files:** -- Create: `harness/nsp/taskdoc.py` -- Test: `harness/tests/test_taskdoc.py` - -- [ ] **Step 1: Write failing test** - -```python -# tests/test_taskdoc.py -from pathlib import Path -from nsp.taskdoc import generate_task_doc, TaskDoc - -def test_task_doc_includes_question_and_schema_scope_not_full_physical(tmp_path): - s = tmp_path / "sess"; s.mkdir() - (s / "question.md").write_text("# Domanda\nQuanti pazienti?\n## Assunzioni\n- a") - (s / "schema_linking.json").write_text('{"candidates":[{"kind":"table","name":"pazienti"}],"joins":[]}') - # A fatal-sized artifact that must NOT appear in the task doc - (s.parent / "physical.yaml").write_text("x: " + "y" * 800_000) - doc = generate_task_doc(session_dir=s, phase=7, promoted_tables=["pazienti"]) - assert "Quanti pazienti?" in doc.body - assert "pazienti" in doc.body - assert len(doc.body) < 100_000 # bounded — no fatal full-schema read - assert "physical.yaml" not in doc.body # never embedded - -def test_task_doc_byte_budget_enforced(tmp_path): - s = tmp_path / "sess"; s.mkdir() - (s / "question.md").write_text("q") - doc = generate_task_doc(session_dir=s, phase=1, promoted_tables=[]) - assert doc.byte_budget_ok is True -``` - -- [ ] **Step 2: Run, verify fail** - -Run: `pytest tests/test_taskdoc.py -v` - -- [ ] **Step 3: Implement `taskdoc.py`** - -```python -# nsp/taskdoc.py -"""Per-step task document generator (spec D16, §4.9). -Emits a single compact document per phase/step, derived from prior artifacts, -with enforced byte budget (target <20k tokens). NEVER embeds full physical.yaml. -""" -from __future__ import annotations -from dataclasses import dataclass -from pathlib import Path - -MAX_BODY_BYTES = 80_000 # ~20k tokens - -@dataclass -class TaskDoc: - phase: int - body: str - byte_budget_ok: bool - -def generate_task_doc(session_dir: Path | str, phase: int, promoted_tables: list[str] | None = None) -> TaskDoc: - session_dir = Path(session_dir) - parts: list[str] = [] - q = session_dir / "question.md" - if q.exists(): - parts.append("## Domanda\n" + q.read_text()) - sl = session_dir / "schema_linking.json" - if sl.exists() and phase >= 4: - parts.append("## Schema linking (deciso)\n```json\n" + sl.read_text() + "\n```") - # Task header for the phase - parts.append(f"## Task: fase {phase}") - body = "\n\n".join(parts) - return TaskDoc(phase=phase, body=body, byte_budget_ok=len(body.encode()) <= MAX_BODY_BYTES) -``` - -- [ ] **Step 4: Run, verify pass** - -Run: `pytest tests/test_taskdoc.py -v` -Expected: 2 PASS - -- [ ] **Step 5: Commit** - -```bash -git add harness/nsp/taskdoc.py harness/tests/test_taskdoc.py -git commit -m "feat(harness): taskdoc per-step generator with byte budget (D16)" -``` - ---- - -### Task A8: Port the CLI skeleton + `nsp phase meta --json` (F2) - -**Files:** -- Create: `harness/nsp/cli/__init__.py` (Typer app) -- Create: `harness/nsp/cli/phase_cmd.py` (adapted) -- Create: `harness/nsp/cli/workspace_cmd.py` -- Port the command modules (sql, cte, db, lsh, vector, evidence, schema, memory, search, session, decision) — adapt imports -- Test: `harness/tests/test_cli_phase_meta.py` - -**Honest scope:** this task ports the CLI structure + `phase meta` and verifies that ONE command behaves. It does NOT verify the 11 ported command modules — those get contract tests in **Task A9 (Livello B)**. Porting them here without tests would assume them reliable, which the spec forbids. - -- [ ] **Step 1: Port CLI modules** — copy `ChironeWp3/src/psdwp3/cli/*.py` → `harness/nsp/cli/`. Rename package imports. The gate will need `nsp phase meta --json` to avoid mirroring constants. - -- [ ] **Step 2: Write failing test for `nsp phase meta --json`** - -```python -# tests/test_cli_phase_meta.py -import json -from typer.testing import CliRunner -from nsp.cli import app - -runner = CliRunner() - -def test_phase_meta_json_returns_workflow_data(): - result = runner.invoke(app, ["phase", "meta", "--json"]) - assert result.exit_code == 0 - data = json.loads(result.stdout) - assert data["max_phase"] == 8 - assert len(data["phases"]) == 8 - assert data["phases"][7]["name"] == "datamart" # F8 present (fixes the JS drift) - assert "advance" in data["phases"][0] -``` - -- [ ] **Step 3: Run, verify fail, then implement `phase meta` in `phase_cmd.py`** - -The command reads from `load_workflow()` and dumps `{max_phase, phases: [{num,name,advance,artifacts_out}]}` as JSON. - -- [ ] **Step 4: Run, verify pass; commit** - -```bash -git add harness/nsp/cli/ harness/tests/test_cli_phase_meta.py -git commit -m "feat(harness): port CLI structure + nsp phase meta --json (F2, kills JS/Python drift)" -``` - ---- - -### Task A9: Contract tests for ported load-bearing modules — L0 (testcontainers) + L1 (fake data) - -**Rationale:** spec §1 forbids assuming ported code is reliable. The DB-touching modules get **L0 tests** (testcontainers — real Postgres in container, where "not assumed reliable" gains real teeth); the logic modules get **L1 tests** (fake data). Docker is available on the dev machine, so L0 runs locally on every `pytest`. - -**L0 tests (testcontainers, marker `@pytest.mark.l0`):** - -- [ ] **Step 1: `tests/l0/test_db_connection.py`** — read-only enforcement (exit code 2 if writable role), `ping` succeeds on read-only role. - -- [ ] **Step 2: `tests/l0/test_db_introspect.py` + `test_db_sampling.py`** — create a known schema + known rows in the container, verify `introspect` returns them with columns + comments, and `unique_values_for_lsh` returns the expected most-frequent values. - -- [ ] **Step 3: `tests/l0/test_rrf.py`** — load real-ish data into the container (LSH index built from sampled values + a small vector set), verify `combined_search`/`rrf_fuse` fuses them with stable, sensible ranking. This is the one that truly exercises the ported RRF/LSH pipeline. - -**L1 tests (fake data, no marker):** - -- [ ] **Step 4: `tests/test_rest_client.py`** — mock `requests`, verify one RPC call carries the `X-API-Key` header and parses the response. - -- [ ] **Step 5: `tests/test_mschema_render.py`** — the 3 formats (markdown, mschema-text ThothAI, schema-dict) from a fake `PhysicalSchema`. Catches port breaks in the render layer. - -- [ ] **Step 6: `tests/test_mschema_eligibility.py`** — eligibility rules on fake columns (wide-text excluded above threshold). - -- [ ] **Step 7: `tests/test_cli_contract.py`** — one test per ported CLI command (sql, cte, schema, session, decision, lsh, vector, evidence, memory, search, db): valid input → exit 0 + parseable JSON. - -- [ ] **Step 8: Register `l0` marker in `pyproject.toml`** and run all A9 tests. - -```toml -[tool.pytest.ini_options] -testpaths = ["tests"] -markers = [ - "l0: testcontainers tests (need Docker, run locally)", - "l2: end-to-end tests requiring real GLM 5.2 + remote DB (skipped when .env incomplete)", -] -addopts = "-m 'not l2'" # L0 runs by default (Docker present); L2 opt-in -``` - -```bash -cd harness && pytest tests/l0/ tests/test_rest_client.py tests/test_mschema_render.py \ - tests/test_mschema_eligibility.py tests/test_cli_contract.py -v -``` - -- [ ] **Step 9: Fix porting bugs the tests surface; commit** - -```bash -git add harness/tests/l0/ harness/tests/test_rest_client.py harness/tests/test_mschema_*.py \ - harness/tests/test_cli_contract.py harness/nsp/{db,rest,mschema,search}/ harness/pyproject.toml -git commit -m "test(harness): L0 testcontainers + L1 contract tests for ported modules (spec §1)" -``` - ---- - -## Phase B — Feature deviations (D11, D13, D14) - -### Task B1: Dual vector API key — port `vectorstore` + verify writer config (D11) - -**Files:** -- Port: `harness/nsp/vectorstore/{rest_client.py, rest_writer.py, store.py, reader.py, embeddings.py, records.py}` -- Port: `harness/nsp/cli/_guards.py` -- Port: `harness/scripts/create_vector_writer_rpc.sql` -- Create: `harness/scripts/create_vector_reader_rpc.sql` -- Test: `harness/tests/test_vector_dual_key.py` - -- [ ] **Step 1: Port the vectorstore modules** (rest_client, rest_writer, store, reader, embeddings, records) and `_guards.py` from ChironeWp3. These already implement the dual-key model per spec §5.4 — port them, rename package, verify `has_vector_write_rest` checks `.api_key.strip()`. - -- [ ] **Step 2: Write failing test for dual-key client construction** - -```python -# tests/test_vector_dual_key.py -from nsp.config import RestConfig -from nsp.vectorstore.rest_client import VectorRestClient - -def test_reader_and_writer_use_separate_keys(): - reader = VectorRestClient(RestConfig(base_url="https://v/", api_key="K-READ")) - writer = VectorRestClient(RestConfig(base_url="https://v/", api_key="K-WRITE")) - assert reader.api_key == "K-READ" - assert writer.api_key == "K-WRITE" - -def test_has_vector_write_rest_false_for_empty_key(): - from nsp.cli._guards import has_vector_write_rest - from nsp.config import Config, RestConfig - cfg = Config(vector_write_rest=RestConfig(base_url="x", api_key=" ")) - assert has_vector_write_rest(cfg) is False -``` - -- [ ] **Step 3: Run, verify fail, adjust ported code so `VectorRestClient` exposes `api_key` (it stores the config; add a property if missing), verify pass.** - -- [ ] **Step 4: Create `create_vector_reader_rpc.sql`** — author the reader RPC (`search_similar`, `list_tables`) mirroring the writer allowlist pattern, granted to a `vector_reader` role. (The reader RPCs lived server-side in Supabase and were never in the ChironeWp3 repo — spec §5.4 note. Author them now.) - -- [ ] **Step 5: Commit** - -```bash -git add harness/nsp/vectorstore/ harness/nsp/cli/_guards.py harness/scripts/ harness/tests/test_vector_dual_key.py -git commit -m "feat(harness): port vectorstore dual-key + reader RPC (D11, §5.4)" -``` - ---- - -### Task B2: `nsp memory save-one` — targeted upsert via writer key (D11) - -**Files:** -- Modify: `harness/nsp/cli/memory_cmd.py` (add `save-one`) -- Modify: `harness/nsp/memory.py` (add `memory_vector_record_for_decision`) -- Test: `harness/tests/test_memory_save_one.py` - -- [ ] **Step 1: Write failing test** - -```python -# tests/test_memory_save_one.py -from pathlib import Path -from unittest.mock import patch, MagicMock -from typer.testing import CliRunner -from nsp.cli import app - -runner = CliRunner() - -def test_save_one_calls_upsert_with_single_record(tmp_path, monkeypatch): - session = tmp_path / "s"; session.mkdir() - # build a minimal registry + a decision_seq to save - from nsp.decisions import append_decision, DecisionRecord - append_decision(session, DecisionRecord(seq=7, ts="t", type="table_promoted", - subject="phase:4", detail="pazienti", rationale="r")) - mock_writer = MagicMock() - mock_writer.upsert_records.return_value = 1 - with patch("nsp.cli.memory_cmd.open_store", return_value=mock_writer): - result = runner.invoke(app, ["memory", "save-one", "--session", str(session), "--decision-seq", "7"]) - assert result.exit_code == 0 - mock_writer.sync.assert_not_called() # must be a single upsert, NOT full resync - mock_writer.upsert_records.assert_called_once() - args = mock_writer.upsert_records.call_args - assert len(args[0][1]) == 1 # exactly one row - -def test_save_one_refused_without_writer_key(tmp_path, monkeypatch): - monkeypatch.setenv("THOTH_PROFILE", "workstation") - session = tmp_path / "s"; session.mkdir() - # config with no write_rest - result = runner.invoke(app, ["memory", "save-one", "--session", str(session), "--decision-seq", "1"]) - assert result.exit_code == 4 # require_vector_write_allowed gate -``` - -- [ ] **Step 2: Run, verify fail** - -- [ ] **Step 3: Implement `save-one`** in `memory_cmd.py`: - - Guard: `require_vector_write_allowed(cfg, "memory save-one")`. - - Load the registry, find the memory record for `decision_seq`. - - Build a single `VectorRecord` (reuse `memory_vector_records` but filter to one). - - Call `writer.existing_hashes(kinds={"memory"})` + embed (if hash changed) + `writer.upsert_records(table="memory", rows=[one])`. - - Do NOT call `writer.sync` (that's the full-resync path). - -- [ ] **Step 4: Run, verify pass; commit** - -```bash -git add harness/nsp/cli/memory_cmd.py harness/nsp/memory.py harness/tests/test_memory_save_one.py -git commit -m "feat(harness): nsp memory save-one — targeted upsert via writer key (D11)" -``` - ---- - -### Task B3: Value & Schema Linking — `value_grounded` decision + LSH multi-column (D14a) - -**Files:** -- Modify: `harness/nsp/decisions.py` (add `value_grounded`) -- Modify: `harness/nsp/search/__init__.py` (stop collapsing multi-column to single best) -- Modify: `harness/nsp/session/models.py` (`Candidate.grounded_values`) -- Test: `harness/tests/test_value_grounding.py` - -- [ ] **Step 1: Port `search/__init__.py`, `lshindex/`, `db/sampling.py`** from ChironeWp3. Then modify. - -- [ ] **Step 2: Write failing test for multi-column LSH exposure** - -```python -# tests/test_value_grounding.py -from nsp.search import aggregate_lsh_multi - -def test_value_in_multiple_columns_returns_all(): - hits = [ - {"table": "t", "column": "c1", "value": "ablazione", "score": 0.9}, - {"table": "t", "column": "c2", "value": "ablazione", "score": 0.7}, - ] - result = aggregate_lsh_multi(hits) - # NOT collapsed to single best — both columns exposed - cols = {h["column"] for h in result["t"]} - assert cols == {"c1", "c2"} -``` - -- [ ] **Step 3: Run, verify fail, then implement `aggregate_lsh_multi`** (replaces the collapsing `_aggregate_lsh` behavior — keep the old name as a thin wrapper to the new one if other code needs single-best). Add `value_grounded` to DecisionType. Add `grounded_values: list[dict] = []` to `Candidate` in `session/models.py`. - -- [ ] **Step 4: Run, verify pass; commit** - -```bash -git add harness/nsp/search/ harness/nsp/lshindex/ harness/nsp/db/sampling.py harness/nsp/decisions.py harness/nsp/session/models.py harness/tests/test_value_grounding.py -git commit -m "feat(harness): value grounding — multi-column LSH + value_grounded (D14a)" -``` - ---- - -### Task B4: SQL formula evidence — formula kind + retrieval + approval (D14b) - -**Files:** -- Modify: `harness/nsp/evidence/model.py` (activate `tier: concept` / add formula fields) -- Create: `harness/nsp/evidence/formula_store.py` -- Modify: `harness/nsp/cli/search_cmd.py` (`--kind formula`) -- Modify: `harness/nsp/decisions.py` (add `concept_formula_approved`, `concept_formula_rejected`) -- Test: `harness/tests/test_formula.py` - -- [ ] **Step 1: Port `evidence/` modules** from ChironeWp3. - -- [ ] **Step 2: Write failing test for formula retrieval** - -```python -# tests/test_formula.py -from pathlib import Path -from nsp.evidence.formula_store import ConceptFormula, save_formula, retrieve_formula - -def test_formula_retrieval_by_concept(tmp_path): - f = ConceptFormula(concept="fascia pediatrica", columns=["data_nascita"], - sql="CASE WHEN ... END", status="reviewed", sources=["s1"]) - save_formula(tmp_path, f) - results = retrieve_formula(tmp_path, "fascia pediatrica") - assert len(results) == 1 - assert results[0].sql.startswith("CASE WHEN") - assert results[0].concept == "fascia pediatrica" -``` - -- [ ] **Step 3: Run, verify fail, implement `formula_store.py`** (frontmatter YAML + SQL body, concept→formula units). Add `--kind formula` to `search_cmd` that calls `retrieve_formula`. Add the two new decision types. - -- [ ] **Step 4: Run, verify pass; commit** - -```bash -git add harness/nsp/evidence/ harness/nsp/cli/search_cmd.py harness/nsp/decisions.py harness/tests/test_formula.py -git commit -m "feat(harness): SQL formula evidence — concept→formula units + retrieval + approval decisions (D14b)" -``` - ---- - -### Task B5: Free-text interpretation guidance (D13) - -**Files:** -- Modify: `harness/nsp/cli/decision_cmd.py` (ensure free-text from "Altro"/"Rifiuta" is captured in `rationale`) -- Modify: `.pi/skills/nsp-sessione/SKILL.md` (+ sub-files) — port + add the D13 §4.6 guidance -- Test: `harness/tests/test_freetext_interpretation.py` - -- [ ] **Step 1: Write failing test** — a golden-style test that, given a `ui_response` with `control: freetext` (Altro text), verifies the harness records the user's text in the decision `rationale` (not discarded). - -```python -# tests/test_freetext_interpretation.py -from pathlib import Path -from nsp.decisions import append_decision, list_decisions, DecisionRecord - -def test_altro_freetext_recorded_in_rationale(tmp_path): - s = tmp_path / "sess"; s.mkdir() - # The gate, on receiving Altro text, appends a decision whose rationale carries the user's words. - append_decision(s, DecisionRecord(seq=1, ts="t", type="table_promoted", - subject="phase:4", detail="ablazione", - rationale="Utente (Altro): 'ablazione' va cercato anche in patologia, non solo nel flag")) - decisions = list_decisions(s) - assert "patologia" in decisions[0].rationale # user text preserved, not discarded -``` - -- [ ] **Step 2: Port `SKILL.md` + sub-files** from ChironeWp3. Add a section "Interpretazione del testo libero" per spec §4.6: instructs the model to (1) evaluate Altro/Rifiuta/steering text in context, (2) act on it, (3) if ambiguous, re-ask instead of defaulting, (4) record the text in the decision rationale. - -- [ ] **Step 3: Run, verify pass (the test asserts the recording contract; the actual model behavior is enforced by the skill prose + gate, validated by fake-pi golden tests in Task C3); commit** - -```bash -git add harness/.pi/skills/ harness/nsp/cli/decision_cmd.py harness/tests/test_freetext_interpretation.py -git commit -m "feat(harness): free-text interpretation guidance + rationale capture (D13)" -``` - ---- - -## Phase C — Gate → widget-descriptor (D2/D4) - -The gate has two parts with **very different testability**: -- **Builders** (pure functions that turn tool-call params into widget-descriptor JSON) — fully testable in L1, **in JS, in-language**. -- **Glue** (Pi runtime integration: `pi.registerTool`, `ctx.sendRaw`, `emitAndWait`, anti-bypass hooks, no-limbo loop) — **NOT testable in L1**; it depends on the Pi runtime. Tested at L2. - -This phase therefore has: **C1 (builders, L1, JS)** + **C2 (glue implementation, no L1 test — verified at L2)**. There is no fake-LLM driver and no `build-scenarios` in this plan (see cross-cutting follow-up). - -### Task C1: Pure widget-builder functions + JS tests (L1) - -**Files:** -- Create: `harness/.pi/extensions/gate/builders.js` (pure functions, no Pi context) -- Create: `harness/.pi/extensions/gate/__tests__/builders.test.js` (node:test, in-language) -- Create: `harness/.pi/extensions/gate/__tests__/golden/*.json` (golden widget descriptors) -- Create: `harness/package.json` (for `npm test` → `node --test`) - -This task extracts widget-descriptor **construction** into pure, testable functions and tests them in JS directly — no Python↔JS bridge, no Python mirror (a mirror would drift; the golden tests would validate the mirror, not the real gate). - -- [ ] **Step 1: Create `package.json`** for the gate JS tests. - -```json -{ - "name": "thothii-harness-gate", - "private": true, - "scripts": { "test": "node --test .pi/extensions/gate/__tests__/" } -} -``` - -- [ ] **Step 2: Define the pure builder API in `builders.js`.** Each builder takes plain params and returns a plain widget-descriptor object. No `ctx`, no I/O. - -```javascript -// gate/builders.js — pure functions, no Pi context. Each returns a ui_request descriptor (spec §4.1). -function buildSelectRequest({ id, phase, title, options, intro = null, allowOther = true, recommended = null }) { /* ... */ } -function buildMultiselectRequest({ id, phase, title, options, content = null, allowEmpty = false, allowOther = true }) { /* ... */ } -function buildArtifactGate({ id, phase, title, artifact /* {kind, data, version} */, action /* {kind, prompt} */ }) { /* ... */ } -function buildInfoRequest({ phase, level, text }) { /* ... */ } -function buildFreetextRequest({ id, phase, title }) { /* ... */ } -function withChildLinkage(option, widgetSpec) { /* sets option.opens = widgetSpec; returns option */ } -module.exports = { buildSelectRequest, buildMultiselectRequest, buildArtifactGate, buildInfoRequest, buildFreetextRequest, withChildLinkage }; -``` - -- [ ] **Step 3: Write JS golden tests** — one golden file per widget type, one test per widget. Assert exact shape. - -```javascript -// gate/__tests__/builders.test.js -const test = require("node:test"); -const assert = require("node:assert"); -const fs = require("node:fs"); -const path = require("node:path"); -const { buildSelectRequest, buildMultiselectRequest, buildArtifactGate } = require("../builders.js"); - -const GOLDEN = path.join(__dirname, "golden"); - -test("select F1 matches golden", () => { - const result = buildSelectRequest({ id: "u1", phase: "F1", title: "Disambigua 'ablazione'", - options: [{ id: "o1", label: "procedura" }, { id: "o2", label: "patologia" }] }); - const golden = JSON.parse(fs.readFileSync(path.join(GOLDEN, "select_F1.json"))); - assert.equal(result.widget, "select"); - assert.deepEqual(result.reserved, golden.reserved); // back/exit/other - assert.deepEqual(result.options, golden.options); - assert.ok(result.schema_version); -}); - -test("artifact-gate F5 schema matches golden", () => { - const result = buildArtifactGate({ id: "u2", phase: "F5", title: "Schema-linking", - artifact: { kind: "schema_linking", data: { candidates: [] }, version: 1 }, - action: { kind: "confirm", prompt: "Confermi?" } }); - assert.equal(result.widget, "artifact-gate"); - assert.equal(result.artifact.kind, "schema_linking"); - assert.equal(result.action.kind, "confirm"); -}); - -test("multiselect F4 carries content + allowEmpty", () => { - const result = buildMultiselectRequest({ id: "u3", phase: "F4", title: "Tabelle", - options: [], content: { candidates: [] }, allowEmpty: false }); - assert.equal(result.widget, "multiselect"); - assert.equal(result.allow_empty, false); - assert.ok("content" in result); -}); - -test("Altro option carries freetext linkage", () => { - const result = buildSelectRequest({ id: "u4", phase: "F1", title: "x", options: [] }); - const other = result.options.find(o => o.id === "other"); - assert.equal(other.opens.widget, "freetext"); -}); -``` - -- [ ] **Step 4: Create the golden files**, run `npm test`, verify pass. - -```bash -cd harness && npm test -``` - -- [ ] **Step 5: Fuzzy tests on builders (L1, JS)** — malformed params (missing `title`, non-list `options`, empty `options` with `allowEmpty:false`). Assert builders throw clearly or return a well-formed error descriptor, never silently produce a broken widget. - -```javascript -test("select with missing title throws clearly", () => { - assert.throws(() => buildSelectRequest({ id: "u", phase: "F1", options: [] }), /title/); -}); -test("multiselect allowEmpty:false with zero options throws", () => { - assert.throws(() => buildMultiselectRequest({ id: "u", phase: "F4", title: "x", options: [], allowEmpty: false }), /allow_empty|options/); -}); -``` - -- [ ] **Step 6: Commit** - -```bash -git add harness/package.json harness/.pi/extensions/gate/ -git commit -m "feat(harness): pure widget-builder functions + JS golden/fuzzy tests (D2, L1)" -``` - ---- - -### Task C2: Gate glue — rewrite `nsp-gate.js` (implementation; verification at L2) - -**Files:** -- Create: `harness/.pi/extensions/nsp-gate.js` (rewritten from ChironeWp3) -- Port: `harness/.pi/extensions/reserved-labels.mjs` -- **No L1 test** — the glue depends on the Pi runtime and is verified at L2 (Task D4). - -**Honest scope:** this task implements the glue but cannot unit-test it in L1. The glue's correctness is validated end-to-end at L2 with real Pi + GLM 5.2. This is the load-bearing limitation stated in the Testing Strategy headline. Do NOT attempt to mock Pi here — that is the fake-Pi follow-up (cross-cutting), not part of this plan. - -- [ ] **Step 1: Port `reserved-labels.mjs`** (BACK/EXIT/OTHER labels, `isReserved`/`stripReserved`). - -- [ ] **Step 2: Port the anti-bypass hooks verbatim** from the original `nsp-gate.js`: `tool_call` hook blocking `nsp phase advance|reopen`, `nsp decision add`, `nsp cte plan`; protected-file list (`review_decisions.jsonl`, `session_manifest.yaml`, `cte_plan.json`); input-lock; `before_agent_start` kickoff. These are load-bearing (spec D4) and unchanged. - -- [ ] **Step 3: Wire tools to builders + emission.** Each of the 4 tools (`reviewer_select`, `reviewer_decide`, `reviewer_confirm`, `rewrite_question`) calls the matching builder from C1, then emits via `ctx.sendRaw({type:"extension_ui_request", ...})` and awaits the correlated response by `id`. Preserve: no-limbo invariant (Esc/cancel re-presents — never returns undefined), recommended-option marker, reading `SCHEMA_LINKING_PHASE`/`PHASE_NAMES` from `nsp phase meta --json` (no mirroring). - -```javascript -// nsp-gate.js (the glue) — wires C1 builders to the Pi runtime -const { buildSelectRequest, buildMultiselectRequest, buildArtifactGate } = require("./gate/builders.js"); - -module.exports = function (pi) { - pi.registerTool({ - name: "reviewer_select", /* ... */, - async execute(id, params, signal, onUpdate, ctx) { - const widget = buildSelectRequest({ id, phase: await currentPhase(ctx), ...params }); - const resp = await emitAndWait(ctx, widget); // extension_ui_request → wait → response - return handleSelectResponse(resp, params); // incl. Altro→freetext linkage, Back→reopen - } - }); - // ... reviewer_decide, reviewer_confirm, rewrite_question, anti-bypass hooks, kickoff injection -}; -``` - -- [ ] **Step 4: Manual smoke (optional, pre-L2)** — launch `pi --mode rpc` from `harness/` once, confirm the extension loads and `/nuova-domanda` triggers a tool registration (no crash). This is a sanity check, not a regression test. - -- [ ] **Step 5: Commit** (verification happens at L2 in Task D4) - -```bash -git add harness/.pi/extensions/nsp-gate.js harness/.pi/extensions/reserved-labels.mjs -git commit -m "feat(harness): rewrite nsp-gate.js glue wired to builders (D2/D4) — verified at L2" -``` - ---- - -## Phase D — End-to-end validation (L1 smoke + L2 real) - -### Task D1: L1 session-coherence smoke test (pure logic, CI) - -**Files:** -- Create: `harness/tests/test_session_coherence_smoke.py` - -A pure-logic test (no LLM, no DB) that constructs a plausible ledger by hand and asserts the invariants D1-the-L2-version would check. This is the CI-runnable proxy for "full session coherence." - -- [ ] **Step 1: Write the test** — build a synthetic ledger (decisions appended directly, not via LLM) representing a full F1→F8 walk + a rollback F6→F4 + re-derive. Assert: `current_phase` folds correctly at each step, `effective_decisions` excludes the stale tail after rollback, `teardown_to_phase(4)` deletes the right artifacts, `generate_task_doc` stays under byte budget at every phase, and `nsp session consistency` (the post-rollback coherence check) passes. - -```python -# tests/test_session_coherence_smoke.py -from pathlib import Path -from nsp.decisions import append_decision, DecisionRecord -from nsp.phase import current_phase, effective_decisions -from nsp.teardown import teardown_to_phase -from nsp.taskdoc import generate_task_doc - -def test_full_walk_then_rollback_stays_coherent(tmp_path): - s = tmp_path / "sess"; s.mkdir() - # build a synthetic full-session ledger - for n in range(1, 9): - append_decision(s, DecisionRecord(seq=n, ts="t", type="phase_approved", subject=f"phase:{n}", - detail="", rationale="")) - assert current_phase(s) == 9 # max_phase + 1 - # rollback to F4 - append_decision(s, DecisionRecord(seq=9, ts="t", type="phase_reopened", subject="phase:4", - detail="", rationale="")) - report = teardown_to_phase(s, target_phase=4) - assert current_phase(s) == 4 - assert "sql_final.sql" in report.deleted_files or "cte_plan.json" in report.deleted_files - # task docs stay bounded across phases - for ph in range(1, 9): - doc = generate_task_doc(session_dir=s, phase=ph, promoted_tables=["dim_paziente"]) - assert doc.byte_budget_ok, f"phase {ph} task doc over budget" -``` - -- [ ] **Step 2: Run, verify pass; commit** - -```bash -git add harness/tests/test_session_coherence_smoke.py -git commit -m "test(harness): L1 session-coherence smoke (full walk + rollback, pure logic)" -``` - ---- - -### Task D2: `.pi/` config + prompts + theme ported - -**Files:** -- Create: `harness/.pi/settings.json` -- Create: `harness/.pi/themes/` (port theme) -- Create: `harness/.pi/prompts/nuova-domanda.md`, `riprendi-sessione.md` -- No test (config files) - -- [ ] **Step 1: Port** `.pi/settings.json`, theme, and the two prompt files from ChironeWp3. Verify `pi --mode rpc` launched with `cwd=harness/` finds the `.pi/` directory. - -- [ ] **Step 2: Commit** - -```bash -git add harness/.pi/ -git commit -m "feat(harness): port .pi/ config, prompts, theme" -``` - ---- - -### Task D3: L2 conftest + skip-when-no-.env guard - -**Files:** -- Create: `harness/tests/conftest.py` -- Create: `harness/tests/l2/__init__.py` -- Modify: `harness/pyproject.toml` (register `l2` marker) - -- [ ] **Step 1: Register the `l2` marker** in `pyproject.toml`: - -```toml -[tool.pytest.ini_options] -testpaths = ["tests"] -markers = [ - "l2: end-to-end tests requiring real GLM 5.2 + remote DB (skipped when .env incomplete)", -] -addopts = "-m 'not l2'" # CI default: skip L2 unless explicitly requested -``` - -- [ ] **Step 2: Write `conftest.py`** with a session fixture that loads `.env` and a `l2_env` fixture that skips if any required var is missing. - -```python -# tests/conftest.py -import os -from pathlib import Path -import pytest -from dotenv import load_dotenv - -REQUIRED_L2 = ["THOTH_DWH_API_KEY", "THOTH_VEC_API_KEY", "THOTH_VEC_WRITE_API_KEY", "THOTH_SSL_CA"] - -@pytest.fixture(scope="session", autouse=True) -def _load_env(): - load_dotenv(Path(__file__).resolve().parent.parent / ".env") - -@pytest.fixture(scope="session") -def l2_env(): - missing = [v for v in REQUIRED_L2 if not os.environ.get(v, "").strip()] - if missing: - pytest.skip(f"L2 skipped — missing env vars: {', '.join(missing)} (populate harness/.env)") - return True -``` - -- [ ] **Step 3: Commit** - -```bash -git add harness/tests/conftest.py harness/tests/l2/__init__.py harness/pyproject.toml -git commit -m "test(harness): L2 marker + skip-when-no-.env guard (Testing Strategy)" -``` - ---- - -### Task D4: L2 full session — "ablazione" with GLM 5.2 (human-in-the-loop) - -**Files:** -- Create: `harness/tests/l2/test_session_ablazione.py` -- Create: `harness/workspaces/chirone-test.yaml` - -This is the **end-to-end validation that L1 cannot do**: GLM 5.2 driving the real harness against the real DWH + pgvector, on the "ablazione" question (which exercises D14 value grounding + formula on a multi-column case). Default mode = human answers via terminal; scripted-answers mode for repeatability. - -- [ ] **Step 1: Create `workspaces/chirone-test.yaml`** pointing at the remote endpoints per the L2 connection params. Secrets via `${THOTH_*}`. - -```yaml -name: chirone-test -description: "Chirone DWH — L2 test workspace" -relational: - db_type: postgres - transport: rest - rest: - base_url: https://supabase-aritmolab.policlinicosandonato.it/dwh/ - api_key: ${THOTH_DWH_API_KEY} - ssl_ca: ${THOTH_SSL_CA} -vector_db: - collection: chirone_docs - dim: 768 - rest: - base_url: https://supabase-aritmolab.policlinicosandonato.it/vector/v1/ - api_key: ${THOTH_VEC_API_KEY} - ssl_ca: ${THOTH_SSL_CA} - write_rest: - base_url: https://supabase-aritmolab.policlinicosandonato.it/vector/v1/ - api_key: ${THOTH_VEC_WRITE_API_KEY} - ssl_ca: ${THOTH_SSL_CA} -evidence: - source_root: ${EVIDENCE_ROOT} - evidence_dir: evidence/chirone -embeddings: - provider: ollama - base_url: ${THOTH_OLLAMA_URL} - model: nomic-embed-text-v2-moe - dim: 768 - batch_size: 64 -execution: - allow: [cte_test, explain, preview, aggregate, export] - max_preview_rows: 10 - statement_timeout_ms: 5000 - forbidden_functions: [set_config, dblink, dblink_exec, lo_import] -``` - -- [ ] **Step 2: Write the L2 test** — spawns Pi (`pi --mode rpc`, cwd=harness/) with GLM 5.2, runs `/nuova-domanda "dammi la lista dei pazienti che hanno fatto un'ablazione nel 2025"`, and either (a) prompts the reviewer in the terminal for each gate widget (human mode) or (b) feeds canned answers (scripted mode via `--answers-file`). Asserts: a session is produced, it reaches a finalized state, the `value_grounded` and `concept_formula_approved` decisions appear (D14 exercised), `sql_final.sql` is present and read-only-validates against the DWH. - -```python -# tests/l2/test_session_ablazione.py -import subprocess, json -from pathlib import Path -import pytest - -@pytest.mark.l2 -def test_ablazione_session_human_in_loop(l2_env, tmp_path): - """L2: GLM 5.2 + real DWH. Reviewer answers gates via terminal. - Run with: pytest -m l2 tests/l2/test_session_ablazione.py -s - """ - session_dir = tmp_path / "sess" - proc = subprocess.run( - ["pi", "--mode", "rpc"], - cwd="harness/", - # the test harness feeds /nuova-domanda and relays terminal I/O for reviewer answers - ... - timeout=600, - ) - assert (session_dir / "sql_final.sql").exists() - ledger = (session_dir / "review_decisions.jsonl").read_text().splitlines() - types = {json.loads(l)["type"] for l in ledger} - assert "value_grounded" in types or "concept_formula_approved" in types # D14 exercised -``` - -- [ ] **Step 3: Run manually** (`pytest -m l2 -s`), confirm the session finalizes and D14 surfaces. This is non-deterministic and slow — it's a pre-release check, not CI. **Commit the test + workspace; the operator runs it before release.** - -```bash -git add harness/tests/l2/test_session_ablazione.py harness/workspaces/chirone-test.yaml -git commit -m "test(harness): L2 ablazione session with GLM 5.2 + real DWH (D14, human/scripted)" -``` - ---- - -### Task D5: L2 specific behaviors — value grounding (real) + memory save-one (real) - -**Files:** -- Create: `harness/tests/l2/test_value_grounding_real.py` -- Create: `harness/tests/l2/test_memory_save_one_real.py` - -Targeted L2 tests that close specific gaps L1 leaves open: real value grounding on the live schema, and real `memory save-one` upsert to pgvector. - -- [ ] **Step 1: `test_value_grounding_real.py`** — runs `nsp search "ablazione" --kind values --workspace chirone-test`, asserts multiple columns are returned (not collapsed), and that a grounding widget would surface them. Validates D14a on the real schema. - -- [ ] **Step 2: `test_memory_save_one_real.py`** — runs `nsp memory save-one` against `chirone-test` (writer key), asserts the upsert returns `upserted >= 0` (idempotent) and a subsequent `search_similar` finds the memory. Validates D11 end-to-end. - -- [ ] **Step 3: Run manually (`pytest -m l2 -s tests/l2/`); commit** - -```bash -git add harness/tests/l2/test_value_grounding_real.py harness/tests/l2/test_memory_save_one_real.py -git commit -m "test(harness): L2 value grounding (real schema) + memory save-one (real pgvector)" -``` - ---- - -### Task D6: Documentation — README + workflow editing + testing guide - -**Files:** -- Create: `harness/README.md` -- Create: `harness/docs/workflow-editing.md` -- Create: `harness/docs/testing.md` - -- [ ] **Step 1: Write `README.md`** — install, configure `.env` + `workspaces/`, run `nsp`, run L0+L1 tests, run L2 tests (with the CA-bundle copy step). - -- [ ] **Step 2: Write `docs/workflow-editing.md`** — how to edit `workflow.yaml` (add/reorder/merge/skip phases), referencing spec §5.3. - -- [ ] **Step 3: Write `docs/testing.md`** — explain the L0/L1/L2 split honestly: what each level covers and does NOT cover (in plain language: L0 = DB-touching code vs real Postgres in container; L1 = pure logic + gate builders, no DB no LLM; L2 = full system with real GLM 5.2 + remote DB, manual pre-release). State the headline limitation plainly: the LLM→gate conversation has no automated regression coverage. How to run each level (`pytest` for L0+L1, `pytest -m l2` for L2 needing `.env` + VPN + CA bundle). The security note on keys (`.env` gitignored, never logged). - -- [ ] **Step 4: Commit** - -```bash -git add harness/README.md harness/docs/ -git commit -m "docs(harness): README + workflow editing + testing guide (L1/L2)" -``` - ---- - -## Self-Review Notes - -**Spec coverage check** (spec section → task): -- §1 (ChironeWp3 as starting point, NOT assumed reliable): enforced by a **3-tier porting classification**: - - **Tier A (modified by ThothII → full regression test):** phase.py (A5), decisions.py (A4), vectorstore/_guards dual-key (B1), memory_cmd save-one (B2), search multi-column (B3), evidence formula (B4), phase_cmd meta (A8). Each has a focused L1 test for the changed behavior. - - **Tier B (ported unchanged but load-bearing → contract test):** db/connection + introspect + sampling + RRF (**L0 testcontainers**, A9 steps 1-3 — real Postgres, where "not reliable" is tested for real), rest/client + mschema/render + mschema/eligibility + 11 CLI commands (**L1 fake data**, A9 steps 4-7). The L0/L1 split inside Tier B reflects whether the module touches a DB. - - **Tier C (ported unchanged, non-load-bearing → smoke OK):** fetch_ca, output/formatting helpers. Acceptable with smoke. - This makes "not assumed reliable" honest for the load-bearing surface. ✓ -- D1 (3 projects): this plan = harness only; backend/frontend are separate plans. ✓ -- D2/D4 (widget-descriptor gate): **builders** in C1 (L1, JS, in-language — golden + fuzzy); **glue** in C2 (implementation only — verified at L2, NOT in L1, because it depends on the Pi runtime). The limitation is documented in the Testing Strategy headline. ✓ -- D3 (workspace YAML): Task A2. ✓ -- D5 (FS persistence, ledger): Task A4 (decisions port), session models in A5. ✓ -- D6 (auth): out of scope for harness (auth is backend). ✓ (noted) -- D7 (backend SQL read-only): out of scope for harness. ✓ -- D8 (nsp stays Python): all harness tasks are Python (gate glue is JS, per the source). ✓ -- D10 (testing): **three levels** — L0 testcontainers (A9 steps 1-3, runs locally), L1 fake/logic + gate builders (A2-A8, B1-B5, C1, D1; runs locally), L2 real GLM 5.2 + remote DB (D4, D5; pre-release). Honest headline in Testing Strategy: the skill→LLM→gate loop has NO automated regression coverage. ✓ -- D11 (dual key + save-one): B1 (L1 dual-key), B2 (L1 save-one mocked), D5 (L2 save-one real). ✓ -- D12 (deployment model B): harness runs locally — `.env.example` (THOTH_PROFILE), L2 against remote REST over VPN. ✓ -- D13 (free-text interpretation): B5 (L1 rationale contract), D4 (L2 with real model — the glue that records rationale is L2-only). ✓ -- D14a (value grounding): B3 (L1 aggregate_lsh_multi on fake hits), D5 (L2 real schema). ✓ -- D14b (formula): B4 (L1 formula store). ✓ (L2 formula approval rides on D4) -- D15 (rollback): A4 (retract), A5 (effective view), A6 (teardown), D1 (L1 coherence smoke), D4 (L2 glue behavior). ✓ -- D16 (context minimization): A7 (taskdoc, L1). ✓ (per-phase reset in Pi noted as execution detail) - -**Type consistency:** `effective_decisions`, `teardown_to_phase`, `generate_task_doc`, `load_workflow`, `save_formula`, `retrieve_formula`, `aggregate_lsh_multi`, `decision_retracted`, `value_grounded`, `concept_formula_approved/rejected`, builder functions (`buildSelectRequest` etc.) — names match across tasks. ✓ - -**Testing honesty (the load-bearing facts):** -- **The skill→LLM→gate loop has NO automated regression coverage.** It is exercised only at L2 (manual, non-deterministic, pre-release). This is a conscious choice; plan L2 runs before release. -- **The gate glue cannot be unit-tested in L1** — it depends on the Pi runtime (`pi.registerTool`, `ctx.sendRaw`). Only the pure builders are L1-tested (in JS, in-language). -- L0 (testcontainers) tests the DB-touching ported code against a real Postgres — this is where "not assumed reliable" is actually enforced for the data layer. -- L2 tests (D4, D5) are non-deterministic, slow, require credentials + VPN + CA bundle. Skipped automatically when `.env` incomplete. - -**Known follow-ups (not in this plan):** -- **Cross-cutting: a fake-Pi runtime mock** that lets the gate glue run in L1/CI. When built (likely alongside the backend plan, which also needs a fake-Pi to test its RPC client), it would close the gap "gate glue has no L1 test" AND enable a `build-scenarios`/`fake-LLM` replay harness for deterministic gate-behavior tests. This is the single highest-value testing investment not in this plan. It is deferred because it is substantial and shared with the backend. -- The SSE/REST surface the backend will build on (harness speaks JSONL via Pi + `nsp --json`). -- `session consistency` check integration into `nsp session check` (Task A5 stubs the effective view; the full assertion grows during execution). -- Per-phase context reset mechanism in Pi (D16 §4.9 step 3) — depends on Pi's context-management capability, verified when running the gate against real Pi in Task D4. - -This plan is self-contained: at the end, `harness/` runs the full 8-phase workflow emitting/consuming widget-descriptor JSON, validated by L0 (testcontainers) + L1 (logic + gate builders) on every local run, plus L2 (GLM 5.2 + real DB) pre-release. Ready for the backend to spawn and drive. diff --git a/docs/superpowers/plans/2026-06-27-backend-implementation.md b/docs/superpowers/plans/2026-06-27-backend-implementation.md deleted file mode 100644 index d41e0d1e..00000000 --- a/docs/superpowers/plans/2026-06-27-backend-implementation.md +++ /dev/null @@ -1,1024 +0,0 @@ -# Backend Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Costruire il backend ThothII: un orchestratore/traduttore Node+Fastify+TS che avvia Pi in RPC (un processo per sessione), fa da ponte JSONL↔(SSE/REST) verso il frontend, delega a `tht` l'esecuzione del SQL finale, e applica auth pluggabile. - -**Architecture:** Backend senza stato persistente proprio (verità su disco harness). Un `RpcClient` (framing LF-only) per ogni processo Pi gestito dal `PiProcessManager` (uno per sessione attiva). Il `SessionBridge` traduce `extension_ui_request`↔`ui_request` correlando per `id` e mantiene il widget pendente per il re-emit su riconnessione SSE. Le interazioni con il modello (prompt/steer/set_model/set_thinking/get_available_models) usano i comandi RPC nativi di Pi. Il SQL finale è delegato a `tht sql preview/export` (nessun client DB nel backend). Tutto testabile in CI contro il **fake-pi-rpc** consegnato dal Piano Harness. - -**Tech Stack:** Node ≥ 20, TypeScript (ESM), Fastify 5, vitest (test), tsx (dev run). Pi = `@earendil-works/pi-coding-agent` (`pi --mode rpc`). CLI `tht` (Python, già installato nel venv harness). - -## Global Constraints - -- **Prerequisito:** il Piano Harness (`2026-06-27-harness-rpc-readiness.md`) deve essere completato — il backend dipende da: gate RPC-ready, `THT_SESSION` id injection, `tht sql preview --json/--offset`, `tht session list/show --json`, campi manifest, `fake-pi-rpc`. -- **Framing RPC: LF-only JSONL** — `JSON.stringify(v)+"\n"`; lettura split su `\n`, strip `\r` finale. MAI `readline`. Riferimento: `@earendil-works/pi-coding-agent/dist/modes/rpc/jsonl.js`. -- **WIRE CONTRACT (corretto post-spike, Piano Harness Task 1):** il gate usa l'API UI nativa di Pi. Sul wire arriva `{type:"extension_ui_request", id, method:"input", title:""}`; il backend **decodifica `title` → descriptor**, lo espone al FE come `ui_request`, e risponde `{type:"extension_ui_response", id, value:""}` (o `{id, cancelled:true}`). `ctx.sendRaw` non esiste; non esistono eventi `{ui_request:…}` nidificati. Il contratto verso il FE (`ui_request`/`ui_response`, architettura §4) resta invariato — la traduzione native↔FE è responsabilità del `SessionBridge`. -- **Deployment MVP:** localhost, mono-operatore (D12-B). Auth default `none` (utente `dev@local`). -- **Il backend NON tocca il DB**: ogni esecuzione SQL passa da `tht` (BE-2). Nessun driver `pg`/REST nel backend. -- **Posizione harness:** path configurabile (`THT_HARNESS_DIR`, default `../harness` rispetto al backend); `tht` invocato dal venv harness; spawn di Pi con `cwd = THT_HARNESS_DIR`. -- **Nessun segreto nel codice**: credenziali via `.env`/ambiente, mai committate. -- **Test in CI senza Pi/LLM reali**: si usa `fake-pi-rpc` (Piano Harness, `harness/tests/fake_pi/fake_pi_rpc.mjs`). L2 con Pi reale resta separato e informativo. - ---- - -### Task 1: Scaffold del progetto backend - -**Files:** -- Create: `backend/package.json`, `backend/tsconfig.json`, `backend/vitest.config.ts` -- Create: `backend/src/config.ts`, `backend/src/server.ts`, `backend/src/app.ts` -- Test: `backend/test/health.test.ts` - -**Interfaces:** -- Produces: `buildApp(config: AppConfig): FastifyInstance` (registra le route, non ascolta); `loadConfig(env): AppConfig` con `{ port, harnessDir, thtBin, piBin, authMode, defaults: {provider?, model?, thinking?}, maxPiProcesses }`. - -- [ ] **Step 1: package.json + tsconfig + vitest config** - -```json -// backend/package.json -{ - "name": "thothii-backend", - "private": true, - "type": "module", - "scripts": { - "dev": "tsx watch src/server.ts", - "build": "tsc -p tsconfig.json", - "test": "vitest run", - "start": "node dist/server.js" - }, - "dependencies": { "fastify": "^5.0.0" }, - "devDependencies": { "typescript": "^5.6.0", "tsx": "^4.19.0", "vitest": "^2.1.0", "@types/node": "^22.0.0" } -} -``` -```json -// backend/tsconfig.json -{ "compilerOptions": { "target": "ES2022", "module": "ES2022", "moduleResolution": "Bundler", - "strict": true, "outDir": "dist", "rootDir": "src", "esModuleInterop": true, "skipLibCheck": true }, - "include": ["src"] } -``` -```typescript -// backend/vitest.config.ts -import { defineConfig } from "vitest/config"; -export default defineConfig({ test: { environment: "node", include: ["test/**/*.test.ts"] } }); -``` - -- [ ] **Step 2: Scrivere il test health** - -```typescript -// backend/test/health.test.ts -import { test, expect } from "vitest"; -import { buildApp } from "../src/app.js"; -import { loadConfig } from "../src/config.js"; - -test("GET /health ritorna ok", async () => { - const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/tmp/h" })); - const res = await app.inject({ method: "GET", url: "/health" }); - expect(res.statusCode).toBe(200); - expect(res.json()).toEqual({ status: "ok" }); -}); -``` - -- [ ] **Step 3: Eseguire (deve fallire)** - -Run: `cd backend && npm install && npm test` -Expected: FAIL — `Cannot find module '../src/app.js'`. - -- [ ] **Step 4: Implementare config + app + server** - -```typescript -// backend/src/config.ts -export interface AppConfig { - port: number; harnessDir: string; thtBin: string; piBin: string; - authMode: "none" | "mock" | "oidc"; - defaults: { provider?: string; model?: string; thinking?: string }; - maxPiProcesses: number; -} -export function loadConfig(env: Record): AppConfig { - return { - port: Number(env.PORT ?? 8787), - harnessDir: env.THT_HARNESS_DIR ?? "../harness", - thtBin: env.THT_BIN ?? "tht", - piBin: env.PI_BIN ?? "pi", - authMode: (env.AUTH_MODE as AppConfig["authMode"]) ?? "none", - defaults: { provider: env.PI_PROVIDER, model: env.PI_MODEL, thinking: env.PI_THINKING }, - maxPiProcesses: Number(env.MAX_PI_PROCESSES ?? 4), - }; -} -``` -```typescript -// backend/src/app.ts -import Fastify, { type FastifyInstance } from "fastify"; -import type { AppConfig } from "./config.js"; -export function buildApp(_config: AppConfig): FastifyInstance { - const app = Fastify({ logger: false }); - app.get("/health", async () => ({ status: "ok" })); - return app; -} -``` -```typescript -// backend/src/server.ts -import { buildApp } from "./app.js"; -import { loadConfig } from "./config.js"; -const config = loadConfig(process.env); -const app = buildApp(config); -app.listen({ port: config.port, host: "127.0.0.1" }) - .then((addr) => console.log(`backend listening on ${addr}`)); -``` - -- [ ] **Step 5: Eseguire (deve passare) + commit** - -Run: `cd backend && npm test` -Expected: PASS. -```bash -git add backend/package.json backend/tsconfig.json backend/vitest.config.ts backend/src backend/test -git commit -m "feat(backend): scaffold Fastify+TS + /health" -``` - ---- - -### Task 2: LineSplitter — lettura JSONL LF-only - -**Files:** -- Create: `backend/src/rpc/line-splitter.ts` -- Test: `backend/test/line-splitter.test.ts` - -**Interfaces:** -- Produces: `attachJsonlReader(stream: Readable, onLine: (line: string) => void): () => void` — split su `\n`, strip `\r` finale, gestione chunk parziali; ritorna funzione di detach. - -- [ ] **Step 1: Scrivere il test** - -```typescript -import { test, expect } from "vitest"; -import { Readable } from "node:stream"; -import { attachJsonlReader } from "../src/rpc/line-splitter.js"; - -test("riassembla righe spezzate tra chunk, split solo su \\n", async () => { - const lines: string[] = []; - const s = new Readable({ read() {} }); - attachJsonlReader(s, (l) => lines.push(l)); - s.push('{"a":1}\n{"b":'); s.push('2}\r\n{"u":"
 in stringa"}\n'); s.push(null); - await new Promise((r) => s.on("end", r)); - expect(lines).toEqual(['{"a":1}', '{"b":2}', '{"u":"
 in stringa"}']); -}); -``` - -- [ ] **Step 2: Eseguire (deve fallire)** - -Run: `cd backend && npm test -- line-splitter` -Expected: FAIL — modulo assente. - -- [ ] **Step 3: Implementare (port di jsonl.js)** - -```typescript -// backend/src/rpc/line-splitter.ts -import { StringDecoder } from "node:string_decoder"; -import type { Readable } from "node:stream"; -export function attachJsonlReader(stream: Readable, onLine: (line: string) => void): () => void { - const decoder = new StringDecoder("utf8"); - let buffer = ""; - const emit = (line: string) => onLine(line.endsWith("\r") ? line.slice(0, -1) : line); - const onData = (chunk: Buffer | string) => { - buffer += typeof chunk === "string" ? chunk : decoder.write(chunk); - for (let nl; (nl = buffer.indexOf("\n")) !== -1; ) { - emit(buffer.slice(0, nl)); buffer = buffer.slice(nl + 1); - } - }; - const onEnd = () => { buffer += decoder.end(); if (buffer) { emit(buffer); buffer = ""; } }; - stream.on("data", onData); stream.on("end", onEnd); - return () => { stream.off("data", onData); stream.off("end", onEnd); }; -} -``` - -- [ ] **Step 4: Eseguire (deve passare) + commit** - -Run: `cd backend && npm test -- line-splitter` -Expected: PASS. -```bash -git add backend/src/rpc/line-splitter.ts backend/test/line-splitter.test.ts -git commit -m "feat(backend): LF-only JSONL line splitter" -``` - ---- - -### Task 3: RpcClient — spawn, invio comandi, correlazione risposte - -**Files:** -- Create: `backend/src/rpc/rpc-client.ts` -- Test: `backend/test/rpc-client.test.ts` - -**Interfaces:** -- Consumes: `attachJsonlReader` (Task 2); `harness/tests/fake_pi/fake_pi_rpc.mjs` (Piano Harness). -- Produces: `class RpcClient`: - - `constructor(child: ChildProcessWithoutNullStreams)` - - `send(cmd: object): void` — scrive `JSON.stringify(cmd)+"\n"` su stdin - - `request(cmd: object & {type:string}): Promise` — invia con `id` generato, risolve sulla `{type:"response"}` correlata - - `on(event: "event", cb: (evt: any) => void)` — emette ogni messaggio NON-response (eventi: `extension_ui_request`, `text_delta`, `agent_end`, …) - - `nextId(): string` - -- [ ] **Step 1: Scrivere il test (contro fake-pi-rpc)** - -```typescript -import { test, expect } from "vitest"; -import { spawn } from "node:child_process"; -import path from "node:path"; -import { RpcClient } from "../src/rpc/rpc-client.js"; - -const FAKE = path.resolve("../harness/tests/fake_pi/fake_pi_rpc.mjs"); -const SCRIPT = path.resolve("../harness/tests/fake_pi/scripts/f1_disambiguation.json"); - -test("request(get_available_models) correla la response", async () => { - const child = spawn("node", [FAKE, SCRIPT]); - const rpc = new RpcClient(child as any); - const res = await rpc.request({ type: "get_available_models" }); - expect(res.data.models[0].provider).toBe("zai"); - child.stdin.end(); -}); - -test("prompt emette un evento extension_ui_request", async () => { - const child = spawn("node", [FAKE, SCRIPT]); - const rpc = new RpcClient(child as any); - const got = new Promise((resolve) => rpc.on("event", (e) => e.type === "extension_ui_request" && resolve(e))); - rpc.send({ type: "prompt", message: "/nuova-domanda \"x\"" }); - const evt = await got; - expect(evt.method).toBe("input"); // shape nativa di Pi - expect(JSON.parse(evt.title).widget).toBe("select"); // descriptor nel title - child.stdin.end(); -}); -``` - -- [ ] **Step 2: Eseguire (deve fallire)** - -Run: `cd backend && npm test -- rpc-client` -Expected: FAIL — modulo assente. - -- [ ] **Step 3: Implementare RpcClient** - -```typescript -// backend/src/rpc/rpc-client.ts -import type { ChildProcessWithoutNullStreams } from "node:child_process"; -import { attachJsonlReader } from "./line-splitter.js"; -type Listener = (evt: any) => void; -export class RpcClient { - private seq = 0; - private pending = new Map void>(); - private listeners = new Set(); - constructor(private child: ChildProcessWithoutNullStreams) { - attachJsonlReader(child.stdout, (line) => { - if (!line) return; - let msg: any; try { msg = JSON.parse(line); } catch { return; } - if (msg.type === "response" && msg.id && this.pending.has(msg.id)) { - this.pending.get(msg.id)!(msg); this.pending.delete(msg.id); return; - } - for (const l of this.listeners) l(msg); - }); - } - nextId(): string { return `c${++this.seq}`; } - send(cmd: object): void { this.child.stdin.write(JSON.stringify(cmd) + "\n"); } - request(cmd: object & { type: string }): Promise { - const id = this.nextId(); - return new Promise((resolve) => { this.pending.set(id, resolve); this.send({ ...cmd, id }); }); - } - on(_event: "event", cb: Listener): void { this.listeners.add(cb); } -} -``` - -- [ ] **Step 4: Eseguire (deve passare) + commit** - -Run: `cd backend && npm test -- rpc-client` -Expected: PASS (2 test). -```bash -git add backend/src/rpc/rpc-client.ts backend/test/rpc-client.test.ts -git commit -m "feat(backend): RpcClient (spawn/send/request/event) over JSONL" -``` - ---- - -### Task 4: SessionBridge — traduzione widget-descriptor ↔ RPC + widget pendente - -**Files:** -- Create: `backend/src/bridge/session-bridge.ts` -- Test: `backend/test/session-bridge.test.ts` - -**Interfaces:** -- Consumes: `RpcClient` (Task 3). -- Produces: `class SessionBridge`: - - `constructor(rpc: RpcClient)` - - `onClientEvent(cb: (e: ClientEvent) => void)` — emette verso il FE: `{type:"ui_request"|"info"|"text_delta"|"system_event", ...}` - - `respond(uiResponse: object & {id:string}): void` — invia a Pi `{type:"extension_ui_response", id, value: JSON.stringify(uiResponse)}` (shape NATIVA: il payload va in `value`) - - `steer(text: string): void` — invia `{type:"steer", message: text}` - - `pendingWidget(): object | null` — l'ultimo descriptor non ancora risposto (per re-emit) -- **Decodifica della shape nativa (WIRE CONTRACT):** un evento `{type:"extension_ui_request", id, method:"input", title}` → il bridge fa `JSON.parse(title)` → descriptor, lo espone come `ui_request`; `method:"notify"` → `info`; altri `method` (setStatus/setWidget) ignorati in MVP. -- Tipi: `ClientEvent = {type:"ui_request", ui_request:object} | {type:"text_delta", text:string} | {type:"info",...} | {type:"system_event",...}`. - -- [ ] **Step 1: Scrivere il test** - -```typescript -import { test, expect, vi } from "vitest"; -import { SessionBridge } from "../src/bridge/session-bridge.js"; - -function fakeRpc() { - const sent: any[] = []; let evcb: any; - return { rpc: { send: (c:any)=>sent.push(c), on: (_:any,cb:any)=>{evcb=cb}, request: vi.fn() } as any, - sent, fire: (m:any)=>evcb(m) }; -} - -test("extension_ui_request nativo (method:input, title=json) diventa ui_request ed è il pendente", () => { - const { rpc, fire } = fakeRpc(); - const b = new SessionBridge(rpc); - const seen: any[] = []; b.onClientEvent((e)=>seen.push(e)); - const descriptor = { id:"u1", widget:"select" }; - fire({ type:"extension_ui_request", id:"u1", method:"input", title: JSON.stringify(descriptor) }); - expect(seen[0]).toEqual({ type:"ui_request", ui_request: descriptor }); - expect(b.pendingWidget()).toEqual(descriptor); -}); - -test("respond invia extension_ui_response con payload in value e azzera il pendente", () => { - const { rpc, sent, fire } = fakeRpc(); - const b = new SessionBridge(rpc); - fire({ type:"extension_ui_request", id:"u1", method:"input", title: JSON.stringify({ id:"u1", widget:"select" }) }); - b.respond({ id:"u1", choices:["a"] }); - expect(sent.at(-1)).toEqual({ type:"extension_ui_response", id:"u1", value: JSON.stringify({ id:"u1", choices:["a"] }) }); - expect(b.pendingWidget()).toBeNull(); -}); - -test("steer invia un comando steer", () => { - const { rpc, sent } = fakeRpc(); - new SessionBridge(rpc).steer("considera solo il 2024"); - expect(sent.at(-1)).toEqual({ type:"steer", message:"considera solo il 2024" }); -}); -``` - -- [ ] **Step 2: Eseguire (deve fallire)** - -Run: `cd backend && npm test -- session-bridge` -Expected: FAIL — modulo assente. - -- [ ] **Step 3: Implementare SessionBridge** - -```typescript -// backend/src/bridge/session-bridge.ts -import type { RpcClient } from "../rpc/rpc-client.js"; -export type ClientEvent = - | { type: "ui_request"; ui_request: any } - | { type: "text_delta"; text: string } - | { type: "info"; [k: string]: any } - | { type: "system_event"; [k: string]: any }; - -export class SessionBridge { - private pending: any = null; - private cbs = new Set<(e: ClientEvent) => void>(); - constructor(private rpc: RpcClient) { - rpc.on("event", (m) => { - if (m.type === "extension_ui_request" && m.method === "input") { - let descriptor: any; try { descriptor = JSON.parse(m.title); } catch { return; } - this.pending = descriptor; - this.fan({ type: "ui_request", ui_request: descriptor }); - } else if (m.type === "extension_ui_request" && m.method === "notify") { - this.fan({ type: "info", level: m.notifyType ?? "info", text: m.message ?? "" }); - } else if (m.type === "text_delta") { - this.fan({ type: "text_delta", text: m.text ?? "" }); - } else if (m.type === "system_event") { - this.fan(m as ClientEvent); - } - // altri method nativi (setStatus/setWidget) e altri eventi Pi (agent_end, tool_call) non inoltrati in MVP - }); - } - private fan(e: ClientEvent) { for (const cb of this.cbs) cb(e); } - onClientEvent(cb: (e: ClientEvent) => void) { this.cbs.add(cb); } - respond(uiResponse: object & { id: string }) { - this.rpc.send({ type: "extension_ui_response", id: uiResponse.id, value: JSON.stringify(uiResponse) }); - if (this.pending && uiResponse.id === this.pending.id) this.pending = null; - } - steer(text: string) { this.rpc.send({ type: "steer", message: text }); } - pendingWidget() { return this.pending; } -} -``` - -- [ ] **Step 4: Eseguire (deve passare) + commit** - -Run: `cd backend && npm test -- session-bridge` -Expected: PASS (3 test). -```bash -git add backend/src/bridge/session-bridge.ts backend/test/session-bridge.test.ts -git commit -m "feat(backend): SessionBridge widget-descriptor <-> RPC + pending widget" -``` - ---- - -### Task 5: ThtRunner — wrapper dei comandi `tht … --json` - -**Files:** -- Create: `backend/src/tht/tht-runner.ts` -- Test: `backend/test/tht-runner.test.ts` - -**Interfaces:** -- Produces: `class ThtRunner` (config: `{thtBin, harnessDir, configPath}`): - - `sessionNew(opts): Promise<{id:string}>` → `tht session new [--provider…] --json` - - `sessionList(): Promise` → `tht session list --json` - - `sessionShow(id): Promise` → `tht session show --json` - - `sqlPreview(id, {limit, offset}): Promise<{columns,rows,execution_ms,truncated}>` - - `sqlExport(id): Promise<{path:string}>` - - `run(args: string[]): Promise<{code:number, stdout:string, stderr:string}>` (primitiva, iniettabile per test) - -- [ ] **Step 1: Scrivere il test (run iniettabile)** - -```typescript -import { test, expect } from "vitest"; -import { ThtRunner } from "../src/tht/tht-runner.js"; - -test("sessionNew parsa l'id dal JSON", async () => { - const r = new ThtRunner({ thtBin: "tht", harnessDir: "/h", configPath: "config/tht.yaml" }); - r.run = async () => ({ code: 0, stdout: '{"id":"2026-06-27-100000-x"}', stderr: "" }); - expect(await r.sessionNew({ question: "q" })).toEqual({ id: "2026-06-27-100000-x" }); -}); - -test("run con exit != 0 propaga errore con stderr", async () => { - const r = new ThtRunner({ thtBin: "tht", harnessDir: "/h", configPath: "config/tht.yaml" }); - r.run = async () => ({ code: 1, stdout: "", stderr: "ERRORE: boom" }); - await expect(r.sessionList()).rejects.toThrow(/boom/); -}); -``` - -- [ ] **Step 2: Eseguire (deve fallire)** - -Run: `cd backend && npm test -- tht-runner` -Expected: FAIL — modulo assente. - -- [ ] **Step 3: Implementare ThtRunner** - -```typescript -// backend/src/tht/tht-runner.ts -import { spawn } from "node:child_process"; -export interface ThtConfig { thtBin: string; harnessDir: string; configPath: string; } -export interface SessionRow { id: string; status: string; question: string; summary: string | null; - created_at: string; updated_at: string | null; author: string | null; } -export class ThtRunner { - constructor(private cfg: ThtConfig) {} - run(args: string[]): Promise<{ code: number; stdout: string; stderr: string }> { - return new Promise((resolve) => { - const ch = spawn(this.cfg.thtBin, ["-c", this.cfg.configPath, ...args], { cwd: this.cfg.harnessDir }); - let stdout = "", stderr = ""; - ch.stdout.on("data", (d) => (stdout += d)); ch.stderr.on("data", (d) => (stderr += d)); - ch.on("close", (code) => resolve({ code: code ?? 0, stdout, stderr })); - }); - } - private async json(args: string[]): Promise { - const { code, stdout, stderr } = await this.run(args); - if (code !== 0) throw new Error(`tht ${args.join(" ")} exit ${code}: ${stderr.trim()}`); - return JSON.parse(stdout) as T; - } - async sessionNew(o: { question: string; provider?: string; model?: string; thinking?: string; name?: string }) { - const a = ["session", "new", o.question]; - for (const [f, v] of [["--provider", o.provider], ["--model", o.model], ["--thinking", o.thinking], ["--name", o.name]] as const) - if (v) a.push(f, v); - a.push("--json"); - return this.json<{ id: string }>(a); - } - sessionList() { return this.json(["session", "list", "--json"]); } - sessionShow(id: string) { return this.json(["session", "show", id, "--json"]); } - sqlPreview(id: string, p: { limit?: number; offset?: number }) { - const a = ["sql", "preview", `sessions/${id}/sql_final.sql`, "--session", id, "--json"]; - if (p.limit != null) a.push("--limit", String(p.limit)); - if (p.offset) a.push("--offset", String(p.offset)); - return this.json<{ columns: string[]; rows: unknown[][]; execution_ms: number; truncated: boolean }>(a); - } - async sqlExport(id: string) { - const { code, stdout, stderr } = await this.run(["sql", "export", "--session", id]); - if (code !== 0) throw new Error(`tht sql export exit ${code}: ${stderr.trim()}`); - return { path: stdout.trim() }; - } -} -``` - -- [ ] **Step 4: Eseguire (deve passare) + commit** - -Run: `cd backend && npm test -- tht-runner` -Expected: PASS (2 test). -```bash -git add backend/src/tht/tht-runner.ts backend/test/tht-runner.test.ts -git commit -m "feat(backend): ThtRunner wrapper for tht --json (sessions, sql preview/export)" -``` - ---- - -### Task 6: PiProcessManager — un Pi per sessione, spawn/teardown/resume + settings - -**Files:** -- Create: `backend/src/pi/pi-process-manager.ts` -- Test: `backend/test/pi-process-manager.test.ts` - -**Interfaces:** -- Consumes: `RpcClient` (Task 3), `SessionBridge` (Task 4), `AppConfig` (Task 1). -- Produces: `class PiProcessManager`: - - `spawnFor(sessionId: string, opts: {provider?, model?, thinking?, name?, author?}): Promise` — spawn `pi --mode rpc` con `cwd=harnessDir`, env `{...process.env, THT_SESSION: sessionId, THT_AUTHOR: author, PATH: :PATH}`, `--approve`; dopo lo spawn applica `set_model`/`set_thinking_level` via RPC; manda `prompt "/nuova-domanda …"` per avviare il workflow. - - `get(sessionId): SessionRuntime | undefined` - - `teardown(sessionId): void` - - `count(): number` — rispetta `maxPiProcesses` (errore esplicito oltre il cap) -- `SessionRuntime = { rpc: RpcClient; bridge: SessionBridge; child: ChildProcess }` -- Iniettabile: `spawnFn` (default `spawn`) per test senza Pi reale. - -- [ ] **Step 1: Scrivere il test (spawnFn iniettato = fake-pi-rpc)** - -```typescript -import { test, expect } from "vitest"; -import { spawn } from "node:child_process"; -import path from "node:path"; -import { PiProcessManager } from "../src/pi/pi-process-manager.js"; -import { loadConfig } from "../src/config.js"; - -const FAKE = path.resolve("../harness/tests/fake_pi/fake_pi_rpc.mjs"); -const SCRIPT = path.resolve("../harness/tests/fake_pi/scripts/f1_disambiguation.json"); - -test("spawnFor avvia un runtime e il bridge emette il widget F1", async () => { - const cfg = loadConfig({ THT_HARNESS_DIR: "../harness" }); - const mgr = new PiProcessManager(cfg, { spawnFn: () => spawn("node", [FAKE, SCRIPT]) as any }); - const rt = await mgr.spawnFor("2026-06-27-100000-x", {}); - const widget = await new Promise((res) => rt.bridge.onClientEvent((e) => e.type === "ui_request" && res(e))); - expect(widget.ui_request.widget).toBe("select"); - mgr.teardown("2026-06-27-100000-x"); - expect(mgr.count()).toBe(0); -}); - -test("oltre maxPiProcesses solleva errore", async () => { - const cfg = { ...loadConfig({}), maxPiProcesses: 1 }; - const mgr = new PiProcessManager(cfg, { spawnFn: () => spawn("node", [FAKE, SCRIPT]) as any }); - await mgr.spawnFor("a", {}); - await expect(mgr.spawnFor("b", {})).rejects.toThrow(/max/i); - mgr.teardown("a"); -}); -``` - -- [ ] **Step 2: Eseguire (deve fallire)** - -Run: `cd backend && npm test -- pi-process-manager` -Expected: FAIL — modulo assente. - -- [ ] **Step 3: Implementare PiProcessManager** - -```typescript -// backend/src/pi/pi-process-manager.ts -import { spawn as nodeSpawn, type ChildProcessWithoutNullStreams } from "node:child_process"; -import type { AppConfig } from "../config.js"; -import { RpcClient } from "../rpc/rpc-client.js"; -import { SessionBridge } from "../bridge/session-bridge.js"; - -export interface SessionRuntime { rpc: RpcClient; bridge: SessionBridge; child: ChildProcessWithoutNullStreams; } -type SpawnFn = (cfg: AppConfig, sessionId: string, env: NodeJS.ProcessEnv) => ChildProcessWithoutNullStreams; - -export class PiProcessManager { - private runtimes = new Map(); - private spawnFn: SpawnFn; - constructor(private cfg: AppConfig, opts?: { spawnFn?: (...a: any[]) => ChildProcessWithoutNullStreams }) { - this.spawnFn = opts?.spawnFn - ? () => opts.spawnFn!() - : (cfg, sessionId, env) => nodeSpawn(cfg.piBin, ["--mode", "rpc", "--approve"], { cwd: cfg.harnessDir, env }); - } - count() { return this.runtimes.size; } - get(id: string) { return this.runtimes.get(id); } - async spawnFor(sessionId: string, o: { provider?: string; model?: string; thinking?: string; author?: string }) { - if (this.runtimes.size >= this.cfg.maxPiProcesses) throw new Error("max Pi processes reached"); - const env = { ...process.env, THT_SESSION: sessionId, THT_AUTHOR: o.author ?? "dev@local" }; - const child = this.spawnFn(this.cfg, sessionId, env); - const rpc = new RpcClient(child); const bridge = new SessionBridge(rpc); - const rt: SessionRuntime = { rpc, bridge, child }; - this.runtimes.set(sessionId, rt); - child.on("exit", () => this.runtimes.delete(sessionId)); - const provider = o.provider ?? this.cfg.defaults.provider; - const model = o.model ?? this.cfg.defaults.model; - const thinking = o.thinking ?? this.cfg.defaults.thinking; - if (provider && model) await rpc.request({ type: "set_model", provider, modelId: model }); - if (thinking) await rpc.request({ type: "set_thinking_level", level: thinking }); - rpc.send({ type: "prompt", message: `/nuova-domanda "kickoff"` }); - return rt; - } - teardown(id: string) { const rt = this.runtimes.get(id); if (rt) { rt.child.kill(); this.runtimes.delete(id); } } -} -``` - -> Nota PATH (rischio L2 #1): in produzione `env.PATH` deve includere `harness/.venv/bin` perché Pi spawni `tht`. Aggiungere alla composizione `env`: `PATH: \`${harnessVenvBin}:${process.env.PATH}\``. Coperto in Task 11 (wiring reale). - -- [ ] **Step 4: Eseguire (deve passare) + commit** - -Run: `cd backend && npm test -- pi-process-manager` -Expected: PASS (2 test). -```bash -git add backend/src/pi/pi-process-manager.ts backend/test/pi-process-manager.test.ts -git commit -m "feat(backend): PiProcessManager (one Pi per session, cap, set_model/thinking)" -``` - ---- - -### Task 7: SSE hub + re-emit del widget pendente - -**Files:** -- Create: `backend/src/sse/sse-hub.ts` -- Test: `backend/test/sse-hub.test.ts` - -**Interfaces:** -- Produces: `class SseHub`: - - `subscribe(sessionId, send: (event: string, data: object) => void, pending?: object | null): () => void` — alla sottoscrizione, se `pending` è presente, invia subito `send("ui_request", {ui_request: pending})`; ritorna unsubscribe. - - `publish(sessionId, event: string, data: object): void` — a tutti i subscriber della sessione. - -- [ ] **Step 1: Scrivere il test** - -```typescript -import { test, expect } from "vitest"; -import { SseHub } from "../src/sse/sse-hub.js"; - -test("re-emette il widget pendente alla sottoscrizione", () => { - const hub = new SseHub(); const sent: any[] = []; - hub.subscribe("s1", (ev, data) => sent.push({ ev, data }), { id: "u1", widget: "select" }); - expect(sent[0]).toEqual({ ev: "ui_request", data: { ui_request: { id: "u1", widget: "select" } } }); -}); - -test("publish raggiunge i subscriber e unsubscribe li stacca", () => { - const hub = new SseHub(); const sent: any[] = []; - const off = hub.subscribe("s1", (ev, data) => sent.push({ ev, data })); - hub.publish("s1", "text_delta", { text: "x" }); - off(); hub.publish("s1", "text_delta", { text: "y" }); - expect(sent).toEqual([{ ev: "text_delta", data: { text: "x" } }]); -}); -``` - -- [ ] **Step 2: Eseguire (deve fallire)** - -Run: `cd backend && npm test -- sse-hub` -Expected: FAIL — modulo assente. - -- [ ] **Step 3: Implementare SseHub** - -```typescript -// backend/src/sse/sse-hub.ts -type Send = (event: string, data: object) => void; -export class SseHub { - private subs = new Map>(); - subscribe(sessionId: string, send: Send, pending?: object | null): () => void { - if (!this.subs.has(sessionId)) this.subs.set(sessionId, new Set()); - this.subs.get(sessionId)!.add(send); - if (pending) send("ui_request", { ui_request: pending }); - return () => this.subs.get(sessionId)?.delete(send); - } - publish(sessionId: string, event: string, data: object): void { - for (const s of this.subs.get(sessionId) ?? []) s(event, data); - } -} -``` - -- [ ] **Step 4: Eseguire (deve passare) + commit** - -Run: `cd backend && npm test -- sse-hub` -Expected: PASS (2 test). -```bash -git add backend/src/sse/sse-hub.ts backend/test/sse-hub.test.ts -git commit -m "feat(backend): SSE hub with pending-widget re-emit on (re)subscribe" -``` - ---- - -### Task 8: Auth middleware pluggabile (none/mock/oidc) - -**Files:** -- Create: `backend/src/auth/auth.ts` -- Test: `backend/test/auth.test.ts` - -**Interfaces:** -- Produces: `authPreHandler(mode)`→ Fastify preHandler che imposta `req.user = {id}`: `none`→`dev@local`; `mock`→header `x-mock-user`; `oidc`→verifica bearer (stub MVP: `501` se non configurato). `getUser(req): {id:string}`. - -- [ ] **Step 1: Scrivere il test** - -```typescript -import { test, expect } from "vitest"; -import Fastify from "fastify"; -import { authPreHandler, getUser } from "../src/auth/auth.js"; - -test("mode none assegna dev@local", async () => { - const app = Fastify(); app.addHook("preHandler", authPreHandler("none")); - app.get("/me", async (req) => getUser(req)); - expect((await app.inject({ method: "GET", url: "/me" })).json()).toEqual({ id: "dev@local" }); -}); - -test("mode mock legge l'header", async () => { - const app = Fastify(); app.addHook("preHandler", authPreHandler("mock")); - app.get("/me", async (req) => getUser(req)); - const res = await app.inject({ method: "GET", url: "/me", headers: { "x-mock-user": "alice" } }); - expect(res.json()).toEqual({ id: "alice" }); -}); -``` - -- [ ] **Step 2: Eseguire (deve fallire)** - -Run: `cd backend && npm test -- auth` -Expected: FAIL — modulo assente. - -- [ ] **Step 3: Implementare auth** - -```typescript -// backend/src/auth/auth.ts -import type { FastifyRequest, FastifyReply } from "fastify"; -export function authPreHandler(mode: "none" | "mock" | "oidc") { - return async (req: FastifyRequest, reply: FastifyReply) => { - if (mode === "none") (req as any).user = { id: "dev@local" }; - else if (mode === "mock") (req as any).user = { id: (req.headers["x-mock-user"] as string) ?? "mock" }; - else { reply.code(501); throw new Error("OIDC non configurato (MVP: usa none/mock)"); } - }; -} -export function getUser(req: FastifyRequest): { id: string } { return (req as any).user ?? { id: "dev@local" }; } -``` - -- [ ] **Step 4: Eseguire (deve passare) + commit** - -Run: `cd backend && npm test -- auth` -Expected: PASS (2 test). -```bash -git add backend/src/auth/auth.ts backend/test/auth.test.ts -git commit -m "feat(backend): pluggable auth (none/mock/oidc seam)" -``` - ---- - -### Task 9: Route sessioni + SSE + response/steer (wiring) - -**Files:** -- Modify: `backend/src/app.ts` (registra le route, costruisce i singleton) -- Create: `backend/src/routes/sessions.ts` -- Test: `backend/test/routes-sessions.test.ts` - -**Interfaces:** -- Consumes: `PiProcessManager` (6), `ThtRunner` (5), `SseHub` (7), `auth` (8). -- Produces (route, tutte sotto `authPreHandler`): - - `POST /sessions {workspace, question, provider?, model?, thinking?, name?}` → `tht session new` (con author dall'auth) → `mgr.spawnFor(id)` → `{id}` - - `GET /sessions` → `tht session list --json` - - `GET /sessions/:id` → `tht session show --json` - - `GET /sessions/:id/events` → SSE; sottoscrive `SseHub` con `bridge.pendingWidget()`; collega `bridge.onClientEvent` → `hub.publish` - - `POST /sessions/:id/response {ui_response}` → `bridge.respond` - - `POST /sessions/:id/steer {text}` → `bridge.steer` - - `POST /sessions/:id/close` → `mgr.teardown` - -- [ ] **Step 1: Scrivere il test (inietta ThtRunner + spawnFn fake)** - -```typescript -import { test, expect } from "vitest"; -import { spawn } from "node:child_process"; -import path from "node:path"; -import { buildApp } from "../src/app.js"; -import { loadConfig } from "../src/config.js"; - -const FAKE = path.resolve("../harness/tests/fake_pi/fake_pi_rpc.mjs"); -const SCRIPT = path.resolve("../harness/tests/fake_pi/scripts/f1_disambiguation.json"); - -test("POST /sessions crea e avvia, GET /sessions lista", async () => { - const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), { - thtRunner: { sessionNew: async () => ({ id: "s1" }), sessionList: async () => [{ id: "s1" }] } as any, - spawnFn: () => spawn("node", [FAKE, SCRIPT]) as any, - }); - const created = await app.inject({ method: "POST", url: "/sessions", payload: { workspace: "w", question: "q" } }); - expect(created.json()).toEqual({ id: "s1" }); - const list = await app.inject({ method: "GET", url: "/sessions" }); - expect(list.json()).toEqual([{ id: "s1" }]); -}); - -test("POST /sessions/:id/response inoltra al bridge (no error)", async () => { - const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), { - thtRunner: { sessionNew: async () => ({ id: "s1" }) } as any, - spawnFn: () => spawn("node", [FAKE, SCRIPT]) as any, - }); - await app.inject({ method: "POST", url: "/sessions", payload: { workspace: "w", question: "q" } }); - const res = await app.inject({ method: "POST", url: "/sessions/s1/response", - payload: { ui_response: { id: "u1", choices: ["a"] } } }); - expect(res.statusCode).toBe(204); -}); -``` - -- [ ] **Step 2: Eseguire (deve fallire)** - -Run: `cd backend && npm test -- routes-sessions` -Expected: FAIL — `buildApp` non accetta deps / route assenti. - -- [ ] **Step 3: Rendere buildApp iniettabile e registrare le route** - -In `app.ts` accettare `deps?: { thtRunner?, spawnFn? }`, costruire `ThtRunner`/`PiProcessManager`/`SseHub`, applicare `authPreHandler(config.authMode)`, e registrare `sessionRoutes`. - -```typescript -// backend/src/routes/sessions.ts (estratto load-bearing) -import type { FastifyInstance } from "fastify"; -import type { PiProcessManager } from "../pi/pi-process-manager.js"; -import type { ThtRunner } from "../tht/tht-runner.js"; -import type { SseHub } from "../sse/sse-hub.js"; -import { getUser } from "../auth/auth.js"; - -export function sessionRoutes(app: FastifyInstance, d: { mgr: PiProcessManager; tht: ThtRunner; hub: SseHub }) { - app.post("/sessions", async (req, reply) => { - const b = req.body as any; - const { id } = await d.tht.sessionNew({ question: b.question, provider: b.provider, model: b.model, thinking: b.thinking, name: b.name }); - const rt = await d.mgr.spawnFor(id, { provider: b.provider, model: b.model, thinking: b.thinking, author: getUser(req).id }); - rt.bridge.onClientEvent((e) => d.hub.publish(id, e.type, e)); - return { id }; - }); - app.get("/sessions", async () => d.tht.sessionList()); - app.get("/sessions/:id", async (req) => d.tht.sessionShow((req.params as any).id)); - app.post("/sessions/:id/response", async (req, reply) => { - const id = (req.params as any).id; const rt = d.mgr.get(id); - if (!rt) return reply.code(404).send({ error: "sessione non attiva" }); - rt.bridge.respond((req.body as any).ui_response); return reply.code(204).send(); - }); - app.post("/sessions/:id/steer", async (req, reply) => { - const rt = d.mgr.get((req.params as any).id); - if (!rt) return reply.code(404).send({ error: "sessione non attiva" }); - rt.bridge.steer((req.body as any).text); return reply.code(204).send(); - }); - app.post("/sessions/:id/close", async (req) => { d.mgr.teardown((req.params as any).id); return { closed: true }; }); - app.get("/sessions/:id/events", (req, reply) => { - const id = (req.params as any).id; const rt = d.mgr.get(id); - reply.raw.writeHead(200, { "Content-Type": "text/event-stream", "Cache-Control": "no-cache", Connection: "keep-alive" }); - const send = (event: string, data: object) => reply.raw.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`); - const off = d.hub.subscribe(id, send, rt?.bridge.pendingWidget() ?? null); - req.raw.on("close", off); - }); -} -``` - -- [ ] **Step 4: Eseguire (deve passare) + commit** - -Run: `cd backend && npm test -- routes-sessions` -Expected: PASS (2 test). -```bash -git add backend/src/app.ts backend/src/routes/sessions.ts backend/test/routes-sessions.test.ts -git commit -m "feat(backend): session routes + SSE + response/steer wiring" -``` - ---- - -### Task 10: Route SQL (preview/export) + models + workspaces - -**Files:** -- Create: `backend/src/routes/sql.ts`, `backend/src/routes/meta.ts` -- Modify: `backend/src/app.ts` (registra) -- Test: `backend/test/routes-sql-meta.test.ts` - -**Interfaces:** -- Produces: - - `POST /sessions/:id/sql/preview {limit?, offset?}` → `tht.sqlPreview` → `{columns, rows, execution_ms, truncated}` - - `POST /sessions/:id/sql/export` → `tht.sqlExport` → `{path}` - - `GET /models` → `rpc.request({type:"get_available_models"})` da un Pi effimero, OR set statico dai default se nessun Pi attivo (MVP: legge da un processo Pi effimero via `mgr`); ritorna `{models:[...]}` - - `GET /workspaces` → lista da `harnessDir/workspaces/*.yaml` (solo nomi + path, no secret) - -- [ ] **Step 1: Scrivere il test** - -```typescript -import { test, expect } from "vitest"; -import { buildApp } from "../src/app.js"; -import { loadConfig } from "../src/config.js"; - -test("POST sql/preview ritorna le righe da ThtRunner", async () => { - const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), { - thtRunner: { sqlPreview: async () => ({ columns: ["a"], rows: [[1]], execution_ms: 2, truncated: false }) } as any, - }); - const res = await app.inject({ method: "POST", url: "/sessions/s1/sql/preview", payload: { limit: 10, offset: 0 } }); - expect(res.json()).toEqual({ columns: ["a"], rows: [[1]], execution_ms: 2, truncated: false }); -}); - -test("GET /workspaces elenca gli yaml senza secret", async () => { - const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), { thtRunner: {} as any }); - const res = await app.inject({ method: "GET", url: "/workspaces" }); - expect(res.statusCode).toBe(200); - expect(Array.isArray(res.json())).toBe(true); -}); -``` - -- [ ] **Step 2: Eseguire (deve fallire)** - -Run: `cd backend && npm test -- routes-sql-meta` -Expected: FAIL — route assenti. - -- [ ] **Step 3: Implementare le route** - -```typescript -// backend/src/routes/sql.ts -import type { FastifyInstance } from "fastify"; -import type { ThtRunner } from "../tht/tht-runner.js"; -export function sqlRoutes(app: FastifyInstance, d: { tht: ThtRunner }) { - app.post("/sessions/:id/sql/preview", async (req) => { - const b = (req.body ?? {}) as any; - return d.tht.sqlPreview((req.params as any).id, { limit: b.limit, offset: b.offset }); - }); - app.post("/sessions/:id/sql/export", async (req) => d.tht.sqlExport((req.params as any).id)); -} -``` -```typescript -// backend/src/routes/meta.ts -import type { FastifyInstance } from "fastify"; -import { readdirSync } from "node:fs"; -import { join } from "node:path"; -export function metaRoutes(app: FastifyInstance, d: { harnessDir: string; listModels: () => Promise }) { - app.get("/workspaces", async () => { - const dir = join(d.harnessDir, "workspaces"); - return readdirSync(dir).filter((f) => f.endsWith(".yaml")) - .map((f) => ({ name: f.replace(/\.ya?ml$/, ""), file: f })); - }); - app.get("/models", async () => d.listModels()); -} -``` -In `app.ts`, `listModels` fa spawn di un Pi effimero, `rpc.request({type:"get_available_models"})`, poi teardown; in caso di errore ritorna `{ models: [] }`. - -- [ ] **Step 4: Eseguire (deve passare) + commit** - -Run: `cd backend && npm test -- routes-sql-meta` -Expected: PASS (2 test). -```bash -git add backend/src/routes/sql.ts backend/src/routes/meta.ts backend/src/app.ts backend/test/routes-sql-meta.test.ts -git commit -m "feat(backend): sql preview/export + models + workspaces routes" -``` - ---- - -### Task 11: Resume + PATH venv + smoke end-to-end F1 (fake-pi-rpc) - -**Files:** -- Modify: `backend/src/pi/pi-process-manager.ts` (env PATH venv; `resume(sessionId)` legge il manifest via ThtRunner) -- Modify: `backend/src/routes/sessions.ts` (`POST /sessions/:id/resume`) -- Test: `backend/test/e2e-f1.test.ts` - -**Interfaces:** -- Produces: `mgr.resume(sessionId, tht)` — rilegge `provider/model/thinking` da `tht.sessionShow` e fa `spawnFor` con quei valori; PATH dello spawn reale include `harnessDir/.venv/bin`. - -- [ ] **Step 1: Scrivere lo smoke end-to-end** - -```typescript -import { test, expect } from "vitest"; -import { spawn } from "node:child_process"; -import path from "node:path"; -import { buildApp } from "../src/app.js"; -import { loadConfig } from "../src/config.js"; - -const FAKE = path.resolve("../harness/tests/fake_pi/fake_pi_rpc.mjs"); -const SCRIPT = path.resolve("../harness/tests/fake_pi/scripts/f1_disambiguation.json"); - -test("loop F1: crea sessione → SSE riceve il widget → risponde → 204", async () => { - const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), { - thtRunner: { sessionNew: async () => ({ id: "s1" }) } as any, - spawnFn: () => spawn("node", [FAKE, SCRIPT]) as any, - }); - await app.listen({ port: 0, host: "127.0.0.1" }); - const base = `http://127.0.0.1:${(app.server.address() as any).port}`; - await fetch(`${base}/sessions`, { method: "POST", headers: { "content-type": "application/json" }, - body: JSON.stringify({ workspace: "w", question: "q" }) }); - // SSE: leggi il primo evento ui_request - const es = await fetch(`${base}/sessions/s1/events`); - const reader = es.body!.getReader(); const chunk = await reader.read(); - const text = new TextDecoder().decode(chunk.value); - expect(text).toContain("ui_request"); - await reader.cancel(); - const resp = await fetch(`${base}/sessions/s1/response`, { method: "POST", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ ui_response: { id: "u1", choices: ["a"] } }) }); - expect(resp.status).toBe(204); - await app.close(); -}); -``` - -- [ ] **Step 2: Eseguire (deve fallire)** - -Run: `cd backend && npm test -- e2e-f1` -Expected: FAIL inizialmente (race sull'ordine widget/SSE o resume assente). Diagnosticare con systematic-debugging se necessario. - -- [ ] **Step 3: Implementare resume + PATH venv + fix ordine eventi** - -- In `pi-process-manager.ts` (spawn reale): `PATH: \`${join(cfg.harnessDir, ".venv/bin")}:${process.env.PATH}\``. -- Aggiungere `async resume(id, tht)` che legge `sessionShow(id)` → `{provider, model, thinking}` e chiama `spawnFor`. -- Garantire che la route `POST /sessions` colleghi `bridge.onClientEvent → hub.publish` PRIMA di mandare il `prompt`, così l'evento widget non si perde; il `pendingWidget()` + re-emit (Task 7) copre comunque il caso SSE sottoscritto dopo. -- Aggiungere `POST /sessions/:id/resume` → `mgr.resume(id, tht)` + ricollegamento bridge→hub. - -- [ ] **Step 4: Eseguire (deve passare) + commit** - -Run: `cd backend && npm test` -Expected: PASS (tutta la suite). -```bash -git add backend/src/pi/pi-process-manager.ts backend/src/routes/sessions.ts backend/test/e2e-f1.test.ts -git commit -m "feat(backend): resume + venv PATH + end-to-end F1 smoke (fake-pi-rpc)" -``` - -- [ ] **Step 5: Validazione L2 con Pi reale (informativo, non-CI)** - -Con harness configurato (`.env` + VPN) e `config/tht.yaml` valido: avviare `npm run dev`, fare `POST /sessions` con una domanda reale, aprire l'SSE e confermare che il widget F1 arriva dal Pi reale e che la risposta avanza il workflow. Annotare l'esito (questo è il primo loop end-to-end reale BE↔harness). - ---- - -## Self-Review - -**Spec coverage** (vs `2026-06-27-backend-design.md`): -- BE-1 (un Pi per sessione, resume): Task 6 + Task 11 ✓ -- BE-2 (delega SQL): Task 5 (ThtRunner) + Task 10 (route) ✓ -- BE-3 (re-emit widget pendente): Task 7 + Task 4 (`pendingWidget`) ✓ -- BE-4 (test fake-Pi + unit TS): tutti i task usano vitest; fake-pi-rpc in Task 3/6/9/11 ✓ -- BE-5 (id pre-creato): Task 9 (`tht session new` poi `spawnFor` con `THT_SESSION`) ✓ (dipende dal Piano Harness Task 5) -- BE-6 (model/thinking/provider per-sessione, persistiti, resume): Task 6 (`set_model`/`set_thinking`) + Task 11 (resume legge manifest) ✓ -- BE-7 (settings Pi MVP): `--name`/provider/model/thinking via `sessionNew` (Task 5) + `GET /models` (Task 10); `--approve` nello spawn (Task 6); `quietStartup`/`trust`/`systemPrompt` sono harness-side (Piano Harness Task 9) ✓ -- API §4 (tutti gli endpoint): Task 9 (sessions/events/response/steer/close) + Task 10 (sql/models/workspaces) + Task 1 (health) ✓ -- Auth D6: Task 8 ✓ -- Componenti §5 (PiProcessManager, RpcClient, SessionBridge, SseHub, ThtRunner, auth, config): Task 1–9 ✓ -- Rischio PATH `tht` (§9): Task 6 (nota) + Task 11 (fix) ✓ - -**Placeholder scan:** nessun TBD. Le note implementative (PATH venv, `listModels` effimero) sono concretizzate nel task che le richiede (11, 10). - -**Type consistency:** `ThtRunner.sqlPreview` ritorna `{columns, rows, execution_ms, truncated}` usato identico in Task 10. `SessionRuntime = {rpc, bridge, child}` coerente tra Task 6 e Task 9. `SessionBridge.respond(uiResponse)`/`steer(text)`/`pendingWidget()` coerenti tra Task 4, 7, 9. Comandi RPC (`set_model {provider, modelId}`, `set_thinking_level {level}`, `steer {message}`, `prompt {message}`, `get_available_models`) coerenti con `rpc-types.d.ts` di Pi. Envelope `{type:"extension_ui_request", ui_request}` coerente con il gate (Piano Harness Task 4) e il fake-pi-rpc (Piano Harness Task 10). - -**Nota di sequenza:** Task 1–8 sono indipendenti dal Pi reale (CI puro con fake-pi-rpc). Le validazioni L2 (Task 11 Step 5) richiedono harness configurato + VPN. diff --git a/docs/superpowers/plans/2026-06-27-frontend-implementation.md b/docs/superpowers/plans/2026-06-27-frontend-implementation.md deleted file mode 100644 index 8fc9bcc9..00000000 --- a/docs/superpowers/plans/2026-06-27-frontend-implementation.md +++ /dev/null @@ -1,860 +0,0 @@ -# Frontend Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Costruire il frontend ThothII: una SPA Vite+React+TS che presenta il workflow NL→SQL human-in-the-loop renderizzando i widget-descriptor via SSE e raccogliendo le decisioni del revisore via REST, consumando solo il contratto già implementato del backend. - -**Architecture:** SPA client-only su localhost. TanStack Query per le REST cacheable, uno store Zustand per lo stato live della sessione, un hook `useSessionStream` che apre l'SSE (`EventSource`) e alimenta lo store. Un widget registry mappa `kind`→renderer con fallback universale. I viewer (schema-linking Mermaid, SQL via shiki, risultati AGGrid) vivono nella sidebar destra del layout a 4 zone. Build a slice verticali con il loop F1 chiuso il prima possibile. - -**Tech Stack:** Vite 6, React 18, TypeScript 5.6, @tanstack/react-query 5, zustand 5, Tailwind 3.4 + ShadCn, ag-grid-react/community 32, shiki 1, mermaid 11; test: Vitest 2 + @testing-library/react 16 + jsdom + MSW 2; e2e: @playwright/test 1. - -## Global Constraints - -- **Client-only SPA** (FE-1): niente SSR/route server. Tutto gira nel browser su localhost e consuma il backend Fastify separato. -- **Base URL backend:** da `import.meta.env.VITE_BACKEND_URL`, default `http://localhost:8787`. Mai hard-coded altrove. -- **Contratto FE↔BE invariato.** SSE eventi: `{type:"ui_request",ui_request}` | `{type:"text_delta",text}` | `{type:"info",level?,text}` | `{type:"system_event",event,...}`. REST: `GET /workspaces|/models|/sessions|/sessions/:id`; `POST /sessions|/sessions/:id/response|/steer|/sql/preview|/sql/export|/close|/resume`. Widget kind: `info|select|multiselect|freetext|artifact-gate|artifact` + fallback. -- **SSE via `EventSource` nativo** (FE-3): GET, nessun header (auth=`none` MVP). Non introdurre SSE su fetch in questo MVP. -- **TDD** (FE-5): ogni task scrive prima il test (Vitest+RTL); MSW mocka le REST, un mock di `EventSource` simula l'SSE. Nessun test tocca il backend reale tranne il Playwright e2e (Task finale). -- **Widget isolati** (FE-4): ogni renderer è un file con props `{descriptor, onRespond}`; aggiungere un widget = registrarlo, senza toccare store/stream/registry. -- **Invariante no-limbo:** nessun widget può "chiudere senza rispondere"; Esc/cancel non è una risposta valida. -- **TypeScript strict**; tutti i tipi del contratto vivono in `src/api/types.ts` (unica fonte). Niente `any` se non al confine del fallback. -- **Working dir:** tutti i comandi da `/Users/mp/projects/ThothII/frontend`. Il backend (per l'e2e) è in `../backend`, il fake-pi-rpc in `../harness/tests/fake_pi/`. - ---- - -### Task 1: Scaffold Vite + React + TS + Tailwind/ShadCn + Vitest - -**Files:** -- Create: `frontend/package.json`, `frontend/vite.config.ts`, `frontend/tsconfig.json`, `frontend/vitest.config.ts`, `frontend/index.html`, `frontend/tailwind.config.ts`, `frontend/postcss.config.js`, `frontend/src/main.tsx`, `frontend/src/App.tsx`, `frontend/src/index.css`, `frontend/src/test/setup.ts` -- Test: `frontend/src/App.test.tsx` - -**Interfaces:** -- Produces: un'app montabile; `App` componente root; `npm test` esegue Vitest (jsdom); `npm run dev` serve la SPA; `npm run build` (tsc + vite build) pulito. - -- [ ] **Step 1: package.json + config** - -```json -// frontend/package.json -{ - "name": "thothii-frontend", - "private": true, - "type": "module", - "scripts": { - "dev": "vite", - "build": "tsc -b && vite build", - "preview": "vite preview", - "test": "vitest run", - "test:watch": "vitest", - "e2e": "playwright test" - }, - "dependencies": { - "react": "^18.3.1", "react-dom": "^18.3.1", - "@tanstack/react-query": "^5.59.0", "zustand": "^5.0.0" - }, - "devDependencies": { - "vite": "^6.0.0", "@vitejs/plugin-react": "^4.3.0", "typescript": "^5.6.0", - "vitest": "^2.1.0", "jsdom": "^25.0.0", - "@testing-library/react": "^16.0.0", "@testing-library/jest-dom": "^6.5.0", "@testing-library/user-event": "^14.5.0", - "msw": "^2.4.0", - "tailwindcss": "^3.4.0", "postcss": "^8.4.0", "autoprefixer": "^10.4.0", - "@types/react": "^18.3.0", "@types/react-dom": "^18.3.0" - } -} -``` -```ts -// frontend/vite.config.ts -import { defineConfig } from "vite"; -import react from "@vitejs/plugin-react"; -export default defineConfig({ plugins: [react()] }); -``` -```ts -// frontend/vitest.config.ts -import { defineConfig } from "vitest/config"; -import react from "@vitejs/plugin-react"; -export default defineConfig({ - plugins: [react()], - test: { environment: "jsdom", globals: true, setupFiles: ["./src/test/setup.ts"], include: ["src/**/*.test.{ts,tsx}"] }, -}); -``` -```json -// frontend/tsconfig.json -{ "compilerOptions": { "target": "ES2022", "useDefineForClassFields": true, "lib": ["ES2022","DOM","DOM.Iterable"], - "module": "ESNext", "moduleResolution": "Bundler", "jsx": "react-jsx", "strict": true, "noEmit": true, - "esModuleInterop": true, "skipLibCheck": true, "types": ["vitest/globals","@testing-library/jest-dom"] }, - "include": ["src"] } -``` -`tailwind.config.ts` (`content: ["./index.html","./src/**/*.{ts,tsx}"]`), `postcss.config.js` (tailwind+autoprefixer), `index.html` (root div + `/src/main.tsx`), `src/index.css` (`@tailwind base/components/utilities`). - -- [ ] **Step 2: Write the failing test** - -```tsx -// frontend/src/App.test.tsx -import { render, screen } from "@testing-library/react"; -import { App } from "./App"; -test("App renders the ThothII title", () => { - render(); - expect(screen.getByText(/ThothII/i)).toBeInTheDocument(); -}); -``` - -- [ ] **Step 3: Run (fail)** - -Run: `cd frontend && npm install && npm test` -Expected: FAIL — `Cannot find module './App'`. - -- [ ] **Step 4: Implement App + main + setup** - -```tsx -// frontend/src/App.tsx -export function App() { - return
ThothII
; -} -``` -```tsx -// frontend/src/main.tsx -import { StrictMode } from "react"; -import { createRoot } from "react-dom/client"; -import { App } from "./App"; -import "./index.css"; -createRoot(document.getElementById("root")!).render(); -``` -```ts -// frontend/src/test/setup.ts -import "@testing-library/jest-dom/vitest"; -``` - -- [ ] **Step 5: Run (pass) + ShadCn init + commit** - -Run: `cd frontend && npm test` → PASS. Then init ShadCn (`npx shadcn@latest init -d`) and add the base components used later: `npx shadcn@latest add button checkbox radio-group textarea card dialog badge sonner`. Verify `npm run build` clean. -```bash -git add frontend -git commit -m "feat(frontend): scaffold Vite+React+TS+Tailwind/ShadCn + Vitest" -``` - ---- - -### Task 2: API types + REST client - -**Files:** -- Create: `frontend/src/api/types.ts`, `frontend/src/api/client.ts`, `frontend/src/api/sessions.ts`, `frontend/src/api/workspaces.ts`, `frontend/src/api/models.ts`, `frontend/src/api/sql.ts` -- Create (test infra): `frontend/src/test/msw.ts` -- Test: `frontend/src/api/sessions.test.ts` - -**Interfaces:** -- Produces (canonical contract types — every later task imports these): -```ts -export interface WidgetOption { id: string; label: string; meta?: Record; selected?: boolean; recommended?: boolean; opens?: WidgetDescriptor; } -export interface WidgetDescriptor { - id: string; schema_version?: number; session_id?: string; phase?: string; - title?: string; intro?: string; - widget: "info" | "select" | "multiselect" | "freetext" | "artifact-gate" | "artifact" | string; - options?: WidgetOption[]; reserved?: string[]; allow_empty?: boolean; - artifact?: { kind: string; content?: string; [k: string]: unknown }; - level?: "info" | "warning" | "error"; text?: string; - [k: string]: unknown; -} -export interface UiResponse { id: string; kind?: string; choices?: string[]; text?: string; decision?: { type: string }; control?: string; } -export type StreamEvent = - | { type: "ui_request"; ui_request: WidgetDescriptor } - | { type: "text_delta"; text: string } - | { type: "info"; level?: "info" | "warning" | "error"; text: string } - | { type: "system_event"; event: string; [k: string]: unknown }; -export interface SessionSummary { id: string; status: string; question: string; summary: string | null; created_at: string; updated_at: string | null; author: string | null; } -export interface PreviewResult { columns: string[]; rows: unknown[][]; execution_ms: number; truncated: boolean; limit: number; offset: number; } -``` -- Produces (functions): `createSession(input): Promise<{id:string}>`, `listSessions(): Promise`, `getSession(id): Promise`, `postResponse(id, uiResponse): Promise`, `postSteer(id, text): Promise`, `closeSession(id): Promise`, `resumeSession(id): Promise`, `listWorkspaces(): Promise<{name:string;file:string}[]>`, `listModels(): Promise<{models:unknown[]}>`, `sqlPreview(id,{limit?,offset?}): Promise`, `sqlExport(id): Promise<{path:string}>`. Base: `apiFetch(path, init?)` in `client.ts` using `VITE_BACKEND_URL`. - -- [ ] **Step 1: MSW test infra + failing test** - -```ts -// frontend/src/test/msw.ts -import { setupServer } from "msw/node"; -export const server = setupServer(); -``` -Register in `src/test/setup.ts`: `beforeAll(()=>server.listen()); afterEach(()=>server.resetHandlers()); afterAll(()=>server.close());` (import `server` + vitest globals). -```ts -// frontend/src/api/sessions.test.ts -import { http, HttpResponse } from "msw"; -import { server } from "../test/msw"; -import { createSession, listSessions } from "./sessions"; - -test("createSession POSTs and returns the id", async () => { - server.use(http.post("http://localhost:8787/sessions", () => HttpResponse.json({ id: "s1" }))); - expect(await createSession({ workspace: "w", question: "q" })).toEqual({ id: "s1" }); -}); -test("listSessions GETs the array", async () => { - server.use(http.get("http://localhost:8787/sessions", () => HttpResponse.json([{ id: "s1", status: "open", question: "q", summary: null, created_at: "t", updated_at: null, author: null }]))); - const rows = await listSessions(); - expect(rows[0].id).toBe("s1"); -}); -``` - -- [ ] **Step 2: Run (fail)** - -Run: `cd frontend && npm test -- sessions` -Expected: FAIL — modules absent. - -- [ ] **Step 3: Implement client + api modules** - -```ts -// frontend/src/api/client.ts -const BASE = import.meta.env.VITE_BACKEND_URL ?? "http://localhost:8787"; -export async function apiFetch(path: string, init?: RequestInit): Promise { - const res = await fetch(`${BASE}${path}`, { headers: { "content-type": "application/json" }, ...init }); - if (!res.ok) throw new Error(`${res.status} ${await res.text().catch(() => "")}`); - return res.status === 204 ? (undefined as T) : ((await res.json()) as T); -} -export { BASE }; -``` -```ts -// frontend/src/api/sessions.ts -import { apiFetch } from "./client"; -import type { SessionSummary, UiResponse } from "./types"; -export const createSession = (i: { workspace: string; question: string; provider?: string; model?: string; thinking?: string; name?: string }) => - apiFetch<{ id: string }>("/sessions", { method: "POST", body: JSON.stringify(i) }); -export const listSessions = () => apiFetch("/sessions"); -export const getSession = (id: string) => apiFetch(`/sessions/${id}`); -export const postResponse = (id: string, uiResponse: UiResponse) => apiFetch(`/sessions/${id}/response`, { method: "POST", body: JSON.stringify({ ui_response: uiResponse }) }); -export const postSteer = (id: string, text: string) => apiFetch(`/sessions/${id}/steer`, { method: "POST", body: JSON.stringify({ text }) }); -export const closeSession = (id: string) => apiFetch(`/sessions/${id}/close`, { method: "POST" }); -export const resumeSession = (id: string) => apiFetch(`/sessions/${id}/resume`, { method: "POST" }); -``` -`workspaces.ts` (`listWorkspaces`), `models.ts` (`listModels`), `sql.ts` (`sqlPreview`, `sqlExport`) follow the same pattern with their endpoints; `types.ts` holds the interfaces above. - -- [ ] **Step 4: Run (pass) + commit** - -Run: `cd frontend && npm test -- sessions` → PASS; `npm run build` clean. -```bash -git add frontend/src/api frontend/src/test -git commit -m "feat(frontend): contract types + REST client (MSW-tested)" -``` - ---- - -### Task 3: Session store (Zustand) - -**Files:** -- Create: `frontend/src/store/sessionStore.ts` -- Test: `frontend/src/store/sessionStore.test.ts` - -**Interfaces:** -- Consumes: `StreamEvent`, `WidgetDescriptor` (Task 2). -- Produces: `useSessionStore` (Zustand) with state `{ pendingWidget: WidgetDescriptor|null; transcript: {role:"assistant";text:string}[]; toasts: {level:string;text:string}[]; lastSystemEvent: StreamEvent|null }` and actions `applyEvent(e: StreamEvent): void`, `clearPending(): void`, `resetSession(): void`. `applyEvent`: `ui_request`→set pendingWidget; `text_delta`→append to the current assistant transcript entry (create if none/after a widget); `info`→push toast; `system_event`→set lastSystemEvent. - -- [ ] **Step 1: Failing test** - -```ts -// frontend/src/store/sessionStore.test.ts -import { useSessionStore } from "./sessionStore"; -beforeEach(() => useSessionStore.getState().resetSession()); - -test("ui_request sets pendingWidget", () => { - useSessionStore.getState().applyEvent({ type: "ui_request", ui_request: { id: "u1", widget: "select" } }); - expect(useSessionStore.getState().pendingWidget?.id).toBe("u1"); -}); -test("text_delta accumulates into transcript", () => { - const s = useSessionStore.getState(); - s.applyEvent({ type: "text_delta", text: "Ana" }); - s.applyEvent({ type: "text_delta", text: "lisi" }); - expect(useSessionStore.getState().transcript.at(-1)?.text).toBe("Analisi"); -}); -test("info pushes a toast", () => { - useSessionStore.getState().applyEvent({ type: "info", level: "warning", text: "attenzione" }); - expect(useSessionStore.getState().toasts.at(-1)).toEqual({ level: "warning", text: "attenzione" }); -}); -test("clearPending removes the widget", () => { - useSessionStore.getState().applyEvent({ type: "ui_request", ui_request: { id: "u1", widget: "select" } }); - useSessionStore.getState().clearPending(); - expect(useSessionStore.getState().pendingWidget).toBeNull(); -}); -``` - -- [ ] **Step 2: Run (fail)** — `npm test -- sessionStore` → module absent. - -- [ ] **Step 3: Implement** - -```ts -// frontend/src/store/sessionStore.ts -import { create } from "zustand"; -import type { StreamEvent, WidgetDescriptor } from "../api/types"; -interface Entry { role: "assistant"; text: string } -interface SessionState { - pendingWidget: WidgetDescriptor | null; transcript: Entry[]; toasts: { level: string; text: string }[]; lastSystemEvent: StreamEvent | null; - applyEvent: (e: StreamEvent) => void; clearPending: () => void; resetSession: () => void; -} -const empty = { pendingWidget: null, transcript: [] as Entry[], toasts: [] as { level: string; text: string }[], lastSystemEvent: null }; -export const useSessionStore = create((set) => ({ - ...empty, - applyEvent: (e) => set((st) => { - if (e.type === "ui_request") return { pendingWidget: e.ui_request }; - if (e.type === "text_delta") { - const t = [...st.transcript]; - const last = t.at(-1); - if (last && !st.pendingWidget) t[t.length - 1] = { role: "assistant", text: last.text + e.text }; - else t.push({ role: "assistant", text: e.text }); - return { transcript: t }; - } - if (e.type === "info") return { toasts: [...st.toasts, { level: e.level ?? "info", text: e.text }] }; - if (e.type === "system_event") return { lastSystemEvent: e }; - return {}; - }), - clearPending: () => set({ pendingWidget: null }), - resetSession: () => set({ ...empty }), -})); -``` - -- [ ] **Step 4: Run (pass) + commit** - -Run: `npm test -- sessionStore` → PASS. -```bash -git add frontend/src/store -git commit -m "feat(frontend): Zustand session store + applyEvent" -``` - ---- - -### Task 4: `useSessionStream` (SSE → store) - -**Files:** -- Create: `frontend/src/stream/useSessionStream.ts` -- Create (test): `frontend/src/test/fakeEventSource.ts` -- Test: `frontend/src/stream/useSessionStream.test.tsx` - -**Interfaces:** -- Consumes: `useSessionStore.applyEvent` (Task 3), `BASE` (Task 2). -- Produces: `useSessionStream(sessionId: string | null): { connected: boolean }` — when `sessionId` is set, opens `new EventSource(\`${BASE}/sessions/${id}/events\`)`, parses each `message` `data` as JSON `StreamEvent`, calls `applyEvent`; closes on unmount / id change. Uses the global `EventSource` (overridable in tests via a fake). - -- [ ] **Step 1: Fake EventSource + failing test** - -```ts -// frontend/src/test/fakeEventSource.ts -export class FakeEventSource { - static instances: FakeEventSource[] = []; - onmessage: ((e: { data: string }) => void) | null = null; - onopen: (() => void) | null = null; - onerror: (() => void) | null = null; - closed = false; - constructor(public url: string) { FakeEventSource.instances.push(this); } - emit(obj: unknown) { this.onmessage?.({ data: JSON.stringify(obj) }); } - close() { this.closed = true; } -} -``` -```tsx -// frontend/src/stream/useSessionStream.test.tsx -import { renderHook } from "@testing-library/react"; -import { act } from "react"; -import { FakeEventSource } from "../test/fakeEventSource"; -import { useSessionStream } from "./useSessionStream"; -import { useSessionStore } from "../store/sessionStore"; - -beforeEach(() => { FakeEventSource.instances = []; (globalThis as any).EventSource = FakeEventSource; useSessionStore.getState().resetSession(); }); - -test("opens an EventSource for the session and feeds events to the store", () => { - renderHook(() => useSessionStream("s1")); - const es = FakeEventSource.instances[0]; - expect(es.url).toContain("/sessions/s1/events"); - act(() => es.emit({ type: "ui_request", ui_request: { id: "u1", widget: "select" } })); - expect(useSessionStore.getState().pendingWidget?.id).toBe("u1"); -}); -test("closes the stream on unmount", () => { - const { unmount } = renderHook(() => useSessionStream("s1")); - const es = FakeEventSource.instances[0]; - unmount(); - expect(es.closed).toBe(true); -}); -``` - -- [ ] **Step 2: Run (fail)** — module absent. - -- [ ] **Step 3: Implement** - -```ts -// frontend/src/stream/useSessionStream.ts -import { useEffect, useState } from "react"; -import { BASE } from "../api/client"; -import { useSessionStore } from "../store/sessionStore"; -import type { StreamEvent } from "../api/types"; -export function useSessionStream(sessionId: string | null) { - const [connected, setConnected] = useState(false); - const applyEvent = useSessionStore((s) => s.applyEvent); - useEffect(() => { - if (!sessionId) return; - const es = new EventSource(`${BASE}/sessions/${sessionId}/events`); - es.onopen = () => setConnected(true); - es.onerror = () => setConnected(false); - es.onmessage = (ev) => { try { applyEvent(JSON.parse(ev.data) as StreamEvent); } catch { /* ignore malformed */ } }; - return () => { es.close(); setConnected(false); }; - }, [sessionId, applyEvent]); - return { connected }; -} -``` - -- [ ] **Step 4: Run (pass) + commit** - -Run: `npm test -- useSessionStream` → PASS. -```bash -git add frontend/src/stream frontend/src/test/fakeEventSource.ts -git commit -m "feat(frontend): useSessionStream (EventSource -> store)" -``` - ---- - -### Task 5: Widget registry + fallback - -**Files:** -- Create: `frontend/src/widgets/registry.ts`, `frontend/src/widgets/FallbackWidget.tsx`, `frontend/src/widgets/types.ts` -- Test: `frontend/src/widgets/registry.test.tsx` - -**Interfaces:** -- Consumes: `WidgetDescriptor`, `UiResponse` (Task 2). -- Produces: `WidgetProps = { descriptor: WidgetDescriptor; onRespond: (r: UiResponse) => void }` (`widgets/types.ts`); `register(kind: string, comp: React.FC): void`; `resolve(kind: string): React.FC` (returns `FallbackWidget` for unknown). `FallbackWidget` renders the descriptor JSON + a freetext box that responds with `{id, control:"freetext", text}`. - -- [ ] **Step 1: Failing test** - -```tsx -// frontend/src/widgets/registry.test.tsx -import { render, screen } from "@testing-library/react"; -import { register, resolve } from "./registry"; -import type { WidgetProps } from "./types"; - -test("resolve returns the registered renderer", () => { - const Dummy = (_: WidgetProps) =>
dummy
; - register("dummy", Dummy); - expect(resolve("dummy")).toBe(Dummy); -}); -test("resolve falls back for unknown kind and shows the JSON", () => { - const Comp = resolve("totally-unknown"); - render( {}} />); - expect(screen.getByText(/Widget non supportato/i)).toBeInTheDocument(); -}); -``` - -- [ ] **Step 2: Run (fail)** — modules absent. - -- [ ] **Step 3: Implement** - -```tsx -// frontend/src/widgets/types.ts -import type { WidgetDescriptor, UiResponse } from "../api/types"; -export type WidgetProps = { descriptor: WidgetDescriptor; onRespond: (r: UiResponse) => void }; -``` -```tsx -// frontend/src/widgets/FallbackWidget.tsx -import { useState } from "react"; -import type { WidgetProps } from "./types"; -export function FallbackWidget({ descriptor, onRespond }: WidgetProps) { - const [text, setText] = useState(""); - return ( -
-

Widget non supportato (kind: {descriptor.widget}) — rispondi manualmente

- 
{JSON.stringify(descriptor, null, 2)}
-