diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index 45ce7d08..3adc627b 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -23,9 +23,9 @@ - **Retained evidence:** `.artifacts/p2-integration/p2-b109757b26388a5ed6b1d173dee86584/` (11/11 checks PASS), bound to clean source commit `de5de36f9a4edfd4fbebf277822090871ccdd61f`. -- **Manual gate:** P2 walkthrough in `docs/testing/p2-p6-manual-verification.md`; decision - **PENDING** and recorded independently. P3 and later start only after an explicit new - authorization. +- **Manual gate:** P2 walkthrough in `docs/testing/p2-p6-manual-verification.md`; the owner + approved P2 on 2026-08-11 (manual acceptance PASS). P3 and later start only after an explicit + new authorization. ### P1.1 workspace-directory registry — automated integration PASS, manual PENDING (2026-08-11) diff --git a/docs/superpowers/plans/2026-08-11-p3-effective-config-and-tht-dwh.md b/docs/superpowers/plans/2026-08-11-p3-effective-config-and-tht-dwh.md new file mode 100644 index 00000000..2ef120da --- /dev/null +++ b/docs/superpowers/plans/2026-08-11-p3-effective-config-and-tht-dwh.md @@ -0,0 +1,150 @@ +# P3 Effective Configuration and `.tht-dwh` Ownership — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Use superpowers:test-driven-development for every behavior change and superpowers:verification-before-completion before any completion claim. + +**Goal:** Make the effective DWH/preprocessing configuration reproducible and versioned across the operator and session paths, key reusable DWH generations by a stable logical identity instead of random temporary config files, scope schema/Evidence state to the pinned revision, add an explicit workspace-global memory root with a safe migration, and document `.tht-dwh` for operators. + +**Architecture:** A versioned shared canonicalizer (TS, shared by the backend session renderer and the compiled operator entrypoint) produces the non-secret effective DWH/preprocessing configuration and its stable logical config-source identity; the harness consumes the same canonical form when writing `OWNER.json`. DWH cache roots are keyed by the versioned effective DWH binding; revision-scoped runtime roots receive verified physical/LSH snapshots from that cache. An explicit `paths.memory` root becomes workspace-global; schema/Evidence Qdrant records and queries carry `workspace_revision` while memory/solved stay workspace-wide. Existing `OWNER.json` schema-v1 roots and legacy memory JSONL are read-compatible and migrated explicitly under the workspace lock; no in-place reinterpretation. + +**Tech Stack:** TypeScript 5, Node 22, Python 3.12 (harness), Pydantic 2, Qdrant, Vitest, pytest, Bash. Host interface remains `thothctl` (Go) unchanged in grammar; only its effective-config identity benefits. + +**Source contract:** `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` §5 (P3), PRD D3, and the P1.1 registry contract. + +--- + +## P3 completion contract + +P3 is complete only when all of the following are true: + +1. A **versioned shared canonicalizer** (`effective-config` TS module, plus the matching harness canonical form) derives the non-secret effective DWH/preprocessing configuration deterministically. Both the session runtime renderer (`ThtRunner`) and the operator entrypoint (`workspace-maintenance`) consume the same canonicalizer and produce byte-identical effective DWH bindings for the same workspace revision. +2. **Stable logical config-source identity** replaces dependence on random temporary config filenames: the operator config lease path and the `OWNER.json` `input_fingerprint` derive from the same canonical logical identity, so two runs over the same revision reuse the same DWH generation. +3. **Content-only revisions never invalidate DWH generations**: `session_storage` and `runtime_identity` remain excluded from the effective-config fingerprint; a Git content-only Evidence/annotation commit does not force introspection. A semantically identical revision (same effective DWH binding) reuses the existing generation. +4. **Effective-config changes fail closed**: a changed endpoint/transport/database/schema/root-affecting policy produces a different binding identity; the harness refuses to reuse a generation owned by a different effective configuration (`effective_config_mismatch` semantics) and the operator surfaces a stable code. +5. **`OWNER.json` schema-v1 compatibility**: existing schema-v1 roots remain readable; an explicit, reviewed migration upgrades them to the versioned identity without in-place reinterpretation. Migration is documented and tested; conflicting legacy roots fail closed. +6. **Explicit workspace-global memory root**: the rendered harness config includes `paths.memory` = `/sessions//memory`. All memory commands, locks, registry JSONL, and Qdrant projection rebuilds use that root. A migration copies and verifies one legacy canonical JSONL under the workspace lock before rebuilding the projection; conflicting legacy registries fail closed. +7. **Revision-scoped schema/Evidence state**: Qdrant schema and Evidence point IDs and queries include `workspace_revision`; `annotations`, corpus `ACTIVE`, and schema/Evidence Qdrant records are revision-scoped. Memory/solved records and queries remain workspace-wide. +8. **Reusable DWH cache layout**: a workspace cache keyed by the versioned effective DWH binding holds verified `physical.yaml`/LSH generations; each pinned revision's runtime root receives verified snapshots from that cache. No pinned runtime consumes the mutable cache root directly. +9. **Documentation**: `.tht-dwh`, immutable generations, `OWNER.json`, `ACTIVE`, input vs config fingerprints, safe migration, regeneration, and recovery are explained in the operator manuals and a dedicated `docs/contracts/tht-dwh.md`. +10. A clean-state P3 automated process goal passes without retry, retains machine/human reports and a secret scan, and proves exact cleanup; the P3 manual walkthrough remains PENDING until the owner decides. +11. No P4 work (collection lifecycle/rebuild), P5 (Git annotations), or P6 (Evidence materialization) is performed. + +## Frozen decisions + +- Canonicalizer version starts at `1`; the canonical serialization is a deterministic JSON document of the non-secret effective DWH/preprocessing configuration (DWH resource, roots, vector/embedding contract, evidence policy) with a fixed key order. +- Logical config-source identity = `@` (versioned), replacing the P2 `_config_source` fallback that used a temporary file path. +- `input_fingerprint` = sha256 of the logical config-source identity; `config_fingerprint` = sha256 of the canonical effective-config document (versioned). +- `paths.memory` is rendered by the backend for both session and operator; legacy configs without it continue to resolve memory beneath `artifacts/memory` only through the explicit migration path. +- Qdrant schema/Evidence record key and query filter include `workspace_revision`; memory/solved keep `workspace_id` only. +- No automatic remote migration or in-place reinterpretation of legacy roots; operators run the documented migration under the workspace lock. + +## Target file map + +**Shared canonicalizer (TS)** +- Create `backend/src/workspaces/effective-config.ts`, `effective-config.test.ts`: versioned canonical serialization, logical identity, fingerprint helpers. +- Modify `backend/src/workspaces/runtime-config-lease.ts`: lease naming/identity from the logical identity (deterministic, no random names for the same revision); publish the canonical effective-config identity in the lease manifest. +- Modify `backend/src/workspace-maintenance.ts` and `backend/src/workspaces/preprocessing-service.ts`: consume the canonicalizer; stable `effective_config_mismatch` surfacing. +- Tests: `backend/test/workspace-runtime-config-lease.test.ts`, `workspace-preprocessing-service.test.ts`, `workspace-maintenance.test.ts`, plus new `effective-config.test.ts`. + +**Harness (Python)** +- Modify `harness/tht/config.py`: canonical effective-config document and logical identity; `paths.memory` explicit root. +- Modify `harness/tht/jobs/dwh_pipeline.py`: `config_dwh_binding` consumes the canonical identity; `OWNER.json` schema-v1 compatibility reader + migration guard. +- Modify `harness/tht/cli/memory_cmd.py` (and locks/registry): explicit `paths.memory` root; migration of one legacy canonical JSONL under the workspace lock. +- Modify `harness/tht/vectorstore/records.py` and `harness/tht/adapters/vector/qdrant.py`: `workspace_revision` in schema/Evidence point IDs and queries. +- Tests: `harness/tests/test_dwh_preprocess_job.py`, `test_lsh_job_resume.py`, `test_memory_*.py`, `test_qdrant_*.py`, `test_registry_evidence_config.py`. + +**Docs** +- Create `docs/contracts/tht-dwh.md`. +- Modify `docs/install/local-workspace-registry.md`, `docs/install/server-workspace-registry.md`, `docs/testing/p2-p6-manual-verification.md` (P3 section). +- Modify after evidence exists: `PROJECT_STATE.md`. + +**Acceptance** +- Create `scripts/p3-acceptance.sh`, `scripts/test-p3-acceptance.sh`, `backend/scripts/p3-acceptance.mjs`, `backend/scripts/p3-acceptance.test.mjs` (pattern: P2 acceptance runner). + +--- + +### Task 1: Versioned shared canonicalizer and logical identity (TS) + +**Files:** create `backend/src/workspaces/effective-config.ts` + test; modify `backend/src/workspaces/runtime-config-lease.ts`, `workspace-maintenance.ts`, `preprocessing-service.ts` + tests. + +1. Write failing tests: canonical document is deterministic (byte-identical for equal inputs, key-ordered, versioned); logical identity is `@sha256:<64hex>`; fingerprint helpers produce `sha256:` values; `session_storage`/`runtime_identity` are excluded; a content-only revision (descriptor change without DWH-affecting fields) yields the same identity; a DWH endpoint/transport/database/schema/root-affecting change yields a different identity; the operator lease path for the same revision is deterministic (no random component for the same logical identity). +2. Implement the canonicalizer: fixed key order, non-secret fields only (never binding file contents, endpoints are allowed as non-secret identity inputs), versioned envelope. +3. Wire the operator lease manifest to carry `effectiveConfigIdentity`; replace the random lease-name component for the same revision with the deterministic identity suffix (retaining uniqueness across revisions). +4. Commit: `feat: versioned effective-config canonicalizer (P3)`. + +### Task 2: Harness canonical form and `OWNER.json` versioned identity + +**Files:** modify `harness/tht/config.py`, `harness/tht/jobs/dwh_pipeline.py` + tests. + +1. Failing tests: `config_dwh_binding` derives `config_fingerprint`/`input_fingerprint` from the versioned canonical document and logical identity; the schema-v1 `OWNER.json` shape remains readable; a versioned root is written with the new fields; a semantically identical revision reuses the generation; a changed endpoint fails closed (never reuses the old generation); `session_storage`/`runtime_identity` exclusion is preserved (content-only commit does not invalidate). +2. Implement: canonical JSON document + logical identity in the harness (mirror of Task 1, versioned); `OWNER.json` compatibility reader accepting schema-v1 keys and the versioned shape; explicit `effective_config_mismatch` refusal when a root belongs to a different canonical identity. +3. Commit: `feat: versioned OWNER.json identity and compatibility (P3)`. + +### Task 3: DWH cache keyed by effective binding + revision-scoped runtime roots + +**Files:** modify `harness/tht/jobs/dwh_pipeline.py`, runtime root selection (`harness/tht/config.py` roots), `backend/src/workspaces/runtime-config-lease.ts` + tests. + +1. Failing tests: the workspace DWH cache is keyed by the versioned effective binding; each pinned revision's runtime root receives verified `physical.yaml`/LSH snapshots from the cache; no pinned runtime reads the mutable cache root directly; a content-only commit changes the revision runtime root but reuses the cache; a binding change creates a new cache root and fails closed on reuse. +2. Implement cache/root separation and verified snapshot handoff (hash-verified copies under the revision runtime root). +3. Commit: `feat: effective-binding DWH cache with revision-scoped runtime roots (P3)`. + +### Task 4: Explicit workspace-global memory root + legacy migration + +**Files:** modify `harness/tht/config.py`, `harness/tht/cli/memory_cmd.py`, memory registry/lock; `backend/src/workspaces/runtime-config-lease.ts` (render `paths.memory`); tests. + +1. Failing tests: rendered config includes `paths.memory` = `/sessions//memory`; memory commands/locks/registry JSONL use it; migration copies and verifies exactly one legacy canonical JSONL under the workspace lock and rebuilds the Qdrant projection; two conflicting legacy registries fail closed; no in-place reinterpretation. +2. Implement the memory root plumbing and the guarded migration. +3. Commit: `feat: explicit workspace memory root with guarded migration (P3)`. + +### Task 5: Revision-scoped Qdrant schema/Evidence records + +**Files:** modify `harness/tht/vectorstore/records.py`, `harness/tht/adapters/vector/qdrant.py`, schema-index and evidence writers + tests. + +1. Failing tests: schema and Evidence point IDs include `workspace_revision`; schema/Evidence queries filter by `workspace_revision`; memory/solved identities and queries remain workspace-wide; existing workspace-global points are not silently reinterpreted (explicit re-index after migration is required). +2. Implement the revision-scoped record keys/filters. +3. Commit: `feat: revision-scoped schema and Evidence vector records (P3)`. + +### Task 6: Proofs — operator/session identity, reuse, fail-closed + +**Files:** extend `backend/test/workspace-runtime-config-lease.test.ts`, `workspace-preprocessing-service.test.ts`, `harness/tests/test_dwh_preprocess_job.py`. + +1. Failing tests (cross-layer): for the same workspace revision, the operator-rendered effective DWH binding is byte-identical to the session-rendered one; a semantically identical revision reuses the DWH generation (rerun `unchanged`); a changed endpoint/transport/database/schema/root-affecting policy fails closed with `effective_config_mismatch` (never silently reusing artifacts). +2. Implement any gap the tests expose (expect the canonicalizer to be the single shared source). +3. Commit: `test: prove operator/session effective-config identity (P3)`. + +### Task 7: Documentation — `.tht-dwh`, generations, fingerprints, migration, recovery + +**Files:** create `docs/contracts/tht-dwh.md`; modify install manuals and `docs/testing/p2-p6-manual-verification.md` (P3 section). + +1. Write the contract doc explaining: what `.tht-dwh` is, immutable generations, `OWNER.json` (schema-v1 vs versioned), `ACTIVE`, input vs config fingerprints, why a fingerprint protects against artifacts of another configuration, the safe migration procedure, regeneration, and recovery. +2. Update the operator manuals with the P3 migration step and the P3 walkthrough section (decision PENDING). +3. Commit: `docs: explain .tht-dwh and P3 migration (P3)`. + +### Task 8: Clean-state P3 automated process goal + +**Files:** create `scripts/p3-acceptance.sh`, `scripts/test-p3-acceptance.sh`, `backend/scripts/p3-acceptance.mjs`, `backend/scripts/p3-acceptance.test.mjs` (pattern: P2 acceptance runner, owned root `.artifacts/p3-integration/p3-/`). + +1. Write RED runner tests (ownership, run-id, cleanup, --keep, injected failure, report bounds, no retry). +2. Implement the clean-state scenario: build fixtures (P1.1 registry + REST DWH + HTTP Evidence + pre-provisioned Qdrant + embedding stub); run `thothctl` product commands; prove: identical operator/session effective binding, content-only revision reuse (`unchanged`), DWH-affecting change fails closed, memory migration + rebuild, revision-scoped Qdrant records, no P4/P5/P6 scope, secret scan, exact cleanup. +3. Run the runner tests, then one clean integration run without retry; retain the report (`report.md` ends with `P3 automated integration: PASS` / `P3 manual acceptance: PENDING`). +4. Commit: `test: add P3 effective-config process acceptance`. + +### Task 9: Final verification and owner handoff + +1. Full backend Vitest + tsc + build; full frontend Vitest + tsc + build (unchanged expectations); harness pytest (excluding the documented pre-existing debt) + Ruff on touched files; Go build/tests (unchanged grammar); compose config checks; docs gates; one final clean P3 acceptance run. +2. Update `PROJECT_STATE.md` (P3 implementation complete, automated PASS, manual PENDING) only after evidence exists. +3. Stop. No P4 work begins without a new explicit authorization. + +--- + +## Owner checkpoint + +After Task 9 the implementation session stops. The owner executes the P3 walkthrough in +`docs/testing/p2-p6-manual-verification.md` and records the decision. P4 (Qdrant collection +lifecycle), P5 (Git FK annotations), and P6 (Evidence materialization) start only after explicit +authorization. + +## Explicit exclusions + +- No collection create/repair/rebuild (P4), no Git annotation synchronization (P5), no Evidence + materialization (P6), no changes to the host `thothctl` grammar, no migration of real PSD + content, no changes to accepted P1/P1.1/P2 evidence.