ThothII/docs/superpowers/plans/2026-08-11-p3-effective-config-and-tht-dwh.md

P3 Effective Configuration and .tht-dwh Ownership — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Use superpowers:test-driven-development for every behavior change and superpowers:verification-before-completion before any completion claim.

Goal: Make the effective DWH/preprocessing configuration reproducible and versioned across the operator and session paths, key reusable DWH generations by a stable logical identity instead of random temporary config files, scope schema/Evidence state to the pinned revision, add an explicit workspace-global memory root with a safe migration, and document .tht-dwh for operators.

Architecture: A versioned shared canonicalizer (TS, shared by the backend session renderer and the compiled operator entrypoint) produces the non-secret effective DWH/preprocessing configuration and its stable logical config-source identity; the harness consumes the same canonical form when writing OWNER.json. DWH cache roots are keyed by the versioned effective DWH binding; revision-scoped runtime roots receive verified physical/LSH snapshots from that cache. An explicit paths.memory root becomes workspace-global; schema/Evidence Qdrant records and queries carry workspace_revision while memory/solved stay workspace-wide. Existing OWNER.json schema-v1 roots and legacy memory JSONL are read-compatible and migrated explicitly under the workspace lock; no in-place reinterpretation.

Tech Stack: TypeScript 5, Node 22, Python 3.12 (harness), Pydantic 2, Qdrant, Vitest, pytest, Bash. Host interface remains thothctl (Go) unchanged in grammar; only its effective-config identity benefits.

Source contract: docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md §5 (P3), PRD D3, and the P1.1 registry contract.

---

P3 completion contract

P3 is complete only when all of the following are true:

1. A versioned shared canonicalizer (effective-config TS module, plus the matching harness canonical form) derives the non-secret effective DWH/preprocessing configuration deterministically. Both the session runtime renderer (ThtRunner) and the operator entrypoint (workspace-maintenance) consume the same canonicalizer and produce byte-identical effective DWH bindings for the same workspace revision.
2. Stable logical config-source identity replaces dependence on random temporary config filenames: the operator config lease path and the OWNER.json input_fingerprint derive from the same canonical logical identity, so two runs over the same revision reuse the same DWH generation.
3. Content-only revisions never invalidate DWH generations: session_storage and runtime_identity remain excluded from the effective-config fingerprint; a Git content-only Evidence/annotation commit does not force introspection. A semantically identical revision (same effective DWH binding) reuses the existing generation.
4. Effective-config changes fail closed: a changed endpoint/transport/database/schema/root-affecting policy produces a different binding identity; the harness refuses to reuse a generation owned by a different effective configuration (effective_config_mismatch semantics) and the operator surfaces a stable code.
5. OWNER.json schema-v1 compatibility: existing schema-v1 roots remain readable; an explicit, reviewed migration upgrades them to the versioned identity without in-place reinterpretation. Migration is documented and tested; conflicting legacy roots fail closed.
6. Explicit workspace-global memory root: the rendered harness config includes paths.memory = <dataRoot>/sessions/<workspace-id>/memory. All memory commands, locks, registry JSONL, and Qdrant projection rebuilds use that root. A migration copies and verifies one legacy canonical JSONL under the workspace lock before rebuilding the projection; conflicting legacy registries fail closed.
7. Revision-scoped schema/Evidence state: Qdrant schema and Evidence point IDs and queries include workspace_revision; annotations, corpus ACTIVE, and schema/Evidence Qdrant records are revision-scoped. Memory/solved records and queries remain workspace-wide.
8. Reusable DWH cache layout: a workspace cache keyed by the versioned effective DWH binding holds verified physical.yaml/LSH generations; each pinned revision's runtime root receives verified snapshots from that cache. No pinned runtime consumes the mutable cache root directly.
9. Documentation: .tht-dwh, immutable generations, OWNER.json, ACTIVE, input vs config fingerprints, safe migration, regeneration, and recovery are explained in the operator manuals and a dedicated docs/contracts/tht-dwh.md.
10. A clean-state P3 automated process goal passes without retry, retains machine/human reports and a secret scan, and proves exact cleanup; the P3 manual walkthrough remains PENDING until the owner decides.
11. No P4 work (collection lifecycle/rebuild), P5 (Git annotations), or P6 (Evidence materialization) is performed.

Frozen decisions

• Canonicalizer version starts at 1; the canonical serialization is a deterministic JSON document of the non-secret effective DWH/preprocessing configuration (DWH resource, roots, vector/embedding contract, evidence policy) with a fixed key order.
• Logical config-source identity = <workspace-id>@<canonical-effective-config-sha256> (versioned), replacing the P2 _config_source fallback that used a temporary file path.
• input_fingerprint = sha256 of the logical config-source identity; config_fingerprint = sha256 of the canonical effective-config document (versioned).
• paths.memory is rendered by the backend for both session and operator; legacy configs without it continue to resolve memory beneath artifacts/memory only through the explicit migration path.
• Qdrant schema/Evidence record key and query filter include workspace_revision; memory/solved keep workspace_id only.
• No automatic remote migration or in-place reinterpretation of legacy roots; operators run the documented migration under the workspace lock.

Target file map

Shared canonicalizer (TS)

• Create backend/src/workspaces/effective-config.ts, effective-config.test.ts: versioned canonical serialization, logical identity, fingerprint helpers.
• Modify backend/src/workspaces/runtime-config-lease.ts: lease naming/identity from the logical identity (deterministic, no random names for the same revision); publish the canonical effective-config identity in the lease manifest.
• Modify backend/src/workspace-maintenance.ts and backend/src/workspaces/preprocessing-service.ts: consume the canonicalizer; stable effective_config_mismatch surfacing.
• Tests: backend/test/workspace-runtime-config-lease.test.ts, workspace-preprocessing-service.test.ts, workspace-maintenance.test.ts, plus new effective-config.test.ts.

Harness (Python)

• Modify harness/tht/config.py: canonical effective-config document and logical identity; paths.memory explicit root.
• Modify harness/tht/jobs/dwh_pipeline.py: config_dwh_binding consumes the canonical identity; OWNER.json schema-v1 compatibility reader + migration guard.
• Modify harness/tht/cli/memory_cmd.py (and locks/registry): explicit paths.memory root; migration of one legacy canonical JSONL under the workspace lock.
• Modify harness/tht/vectorstore/records.py and harness/tht/adapters/vector/qdrant.py: workspace_revision in schema/Evidence point IDs and queries.
• Tests: harness/tests/test_dwh_preprocess_job.py, test_lsh_job_resume.py, test_memory_*.py, test_qdrant_*.py, test_registry_evidence_config.py.

Docs

• Create docs/contracts/tht-dwh.md.
• Modify docs/install/local-workspace-registry.md, docs/install/server-workspace-registry.md, docs/testing/p2-p6-manual-verification.md (P3 section).
• Modify after evidence exists: PROJECT_STATE.md.

Acceptance

• Create scripts/p3-acceptance.sh, scripts/test-p3-acceptance.sh, backend/scripts/p3-acceptance.mjs, backend/scripts/p3-acceptance.test.mjs (pattern: P2 acceptance runner).

---

Task 1: Versioned shared canonicalizer and logical identity (TS)

Files: create backend/src/workspaces/effective-config.ts + test; modify backend/src/workspaces/runtime-config-lease.ts, workspace-maintenance.ts, preprocessing-service.ts + tests.

1. Write failing tests: canonical document is deterministic (byte-identical for equal inputs, key-ordered, versioned); logical identity is <workspace-id>@sha256:<64hex>; fingerprint helpers produce sha256: values; session_storage/runtime_identity are excluded; a content-only revision (descriptor change without DWH-affecting fields) yields the same identity; a DWH endpoint/transport/database/schema/root-affecting change yields a different identity; the operator lease path for the same revision is deterministic (no random component for the same logical identity).
2. Implement the canonicalizer: fixed key order, non-secret fields only (never binding file contents, endpoints are allowed as non-secret identity inputs), versioned envelope.
3. Wire the operator lease manifest to carry effectiveConfigIdentity; replace the random lease-name component for the same revision with the deterministic identity suffix (retaining uniqueness across revisions).
4. Commit: feat: versioned effective-config canonicalizer (P3).

Task 2: Harness canonical form and OWNER.json versioned identity

Files: modify harness/tht/config.py, harness/tht/jobs/dwh_pipeline.py + tests.

1. Failing tests: config_dwh_binding derives config_fingerprint/input_fingerprint from the versioned canonical document and logical identity; the schema-v1 OWNER.json shape remains readable; a versioned root is written with the new fields; a semantically identical revision reuses the generation; a changed endpoint fails closed (never reuses the old generation); session_storage/runtime_identity exclusion is preserved (content-only commit does not invalidate).
2. Implement: canonical JSON document + logical identity in the harness (mirror of Task 1, versioned); OWNER.json compatibility reader accepting schema-v1 keys and the versioned shape; explicit effective_config_mismatch refusal when a root belongs to a different canonical identity.
3. Commit: feat: versioned OWNER.json identity and compatibility (P3).

Task 3: DWH cache keyed by effective binding + revision-scoped runtime roots

Files: modify harness/tht/jobs/dwh_pipeline.py, runtime root selection (harness/tht/config.py roots), backend/src/workspaces/runtime-config-lease.ts + tests.

1. Failing tests: the workspace DWH cache is keyed by the versioned effective binding; each pinned revision's runtime root receives verified physical.yaml/LSH snapshots from the cache; no pinned runtime reads the mutable cache root directly; a content-only commit changes the revision runtime root but reuses the cache; a binding change creates a new cache root and fails closed on reuse.
2. Implement cache/root separation and verified snapshot handoff (hash-verified copies under the revision runtime root).
3. Commit: feat: effective-binding DWH cache with revision-scoped runtime roots (P3).

Task 4: Explicit workspace-global memory root + legacy migration

Files: modify harness/tht/config.py, harness/tht/cli/memory_cmd.py, memory registry/lock; backend/src/workspaces/runtime-config-lease.ts (render paths.memory); tests.

1. Failing tests: rendered config includes paths.memory = <dataRoot>/sessions/<id>/memory; memory commands/locks/registry JSONL use it; migration copies and verifies exactly one legacy canonical JSONL under the workspace lock and rebuilds the Qdrant projection; two conflicting legacy registries fail closed; no in-place reinterpretation.
2. Implement the memory root plumbing and the guarded migration.
3. Commit: feat: explicit workspace memory root with guarded migration (P3).

Task 5: Revision-scoped Qdrant schema/Evidence records

Files: modify harness/tht/vectorstore/records.py, harness/tht/adapters/vector/qdrant.py, schema-index and evidence writers + tests.

1. Failing tests: schema and Evidence point IDs include workspace_revision; schema/Evidence queries filter by workspace_revision; memory/solved identities and queries remain workspace-wide; existing workspace-global points are not silently reinterpreted (explicit re-index after migration is required).
2. Implement the revision-scoped record keys/filters.
3. Commit: feat: revision-scoped schema and Evidence vector records (P3).

Task 6: Proofs — operator/session identity, reuse, fail-closed

Files: extend backend/test/workspace-runtime-config-lease.test.ts, workspace-preprocessing-service.test.ts, harness/tests/test_dwh_preprocess_job.py.

1. Failing tests (cross-layer): for the same workspace revision, the operator-rendered effective DWH binding is byte-identical to the session-rendered one; a semantically identical revision reuses the DWH generation (rerun unchanged); a changed endpoint/transport/database/schema/root-affecting policy fails closed with effective_config_mismatch (never silently reusing artifacts).
2. Implement any gap the tests expose (expect the canonicalizer to be the single shared source).
3. Commit: test: prove operator/session effective-config identity (P3).

Task 7: Documentation — .tht-dwh, generations, fingerprints, migration, recovery

Files: create docs/contracts/tht-dwh.md; modify install manuals and docs/testing/p2-p6-manual-verification.md (P3 section).

1. Write the contract doc explaining: what .tht-dwh is, immutable generations, OWNER.json (schema-v1 vs versioned), ACTIVE, input vs config fingerprints, why a fingerprint protects against artifacts of another configuration, the safe migration procedure, regeneration, and recovery.
2. Update the operator manuals with the P3 migration step and the P3 walkthrough section (decision PENDING).
3. Commit: docs: explain .tht-dwh and P3 migration (P3).

Task 8: Clean-state P3 automated process goal

Files: create scripts/p3-acceptance.sh, scripts/test-p3-acceptance.sh, backend/scripts/p3-acceptance.mjs, backend/scripts/p3-acceptance.test.mjs (pattern: P2 acceptance runner, owned root .artifacts/p3-integration/p3-<run-id>/).

1. Write RED runner tests (ownership, run-id, cleanup, --keep, injected failure, report bounds, no retry).
2. Implement the clean-state scenario: build fixtures (P1.1 registry + REST DWH + HTTP Evidence + pre-provisioned Qdrant + embedding stub); run thothctl product commands; prove: identical operator/session effective binding, content-only revision reuse (unchanged), DWH-affecting change fails closed, memory migration + rebuild, revision-scoped Qdrant records, no P4/P5/P6 scope, secret scan, exact cleanup.
3. Run the runner tests, then one clean integration run without retry; retain the report (report.md ends with P3 automated integration: PASS / P3 manual acceptance: PENDING).
4. Commit: test: add P3 effective-config process acceptance.

Task 9: Final verification and owner handoff

1. Full backend Vitest + tsc + build; full frontend Vitest + tsc + build (unchanged expectations); harness pytest (excluding the documented pre-existing debt) + Ruff on touched files; Go build/tests (unchanged grammar); compose config checks; docs gates; one final clean P3 acceptance run.
2. Update PROJECT_STATE.md (P3 implementation complete, automated PASS, manual PENDING) only after evidence exists.
3. Stop. No P4 work begins without a new explicit authorization.

---

Owner checkpoint

After Task 9 the implementation session stops. The owner executes the P3 walkthrough in docs/testing/p2-p6-manual-verification.md and records the decision. P4 (Qdrant collection lifecycle), P5 (Git FK annotations), and P6 (Evidence materialization) start only after explicit authorization.

Explicit exclusions

• No collection create/repair/rebuild (P4), no Git annotation synchronization (P5), no Evidence materialization (P6), no changes to the host thothctl grammar, no migration of real PSD content, no changes to accepted P1/P1.1/P2 evidence.