docs: record P2 manual acceptance and plan P3 effective configuration
This commit is contained in:
@@ -0,0 +1,150 @@
|
||||
# P3 Effective Configuration and `.tht-dwh` Ownership — Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Use superpowers:test-driven-development for every behavior change and superpowers:verification-before-completion before any completion claim.
|
||||
|
||||
**Goal:** Make the effective DWH/preprocessing configuration reproducible and versioned across the operator and session paths, key reusable DWH generations by a stable logical identity instead of random temporary config files, scope schema/Evidence state to the pinned revision, add an explicit workspace-global memory root with a safe migration, and document `.tht-dwh` for operators.
|
||||
|
||||
**Architecture:** A versioned shared canonicalizer (TS, shared by the backend session renderer and the compiled operator entrypoint) produces the non-secret effective DWH/preprocessing configuration and its stable logical config-source identity; the harness consumes the same canonical form when writing `OWNER.json`. DWH cache roots are keyed by the versioned effective DWH binding; revision-scoped runtime roots receive verified physical/LSH snapshots from that cache. An explicit `paths.memory` root becomes workspace-global; schema/Evidence Qdrant records and queries carry `workspace_revision` while memory/solved stay workspace-wide. Existing `OWNER.json` schema-v1 roots and legacy memory JSONL are read-compatible and migrated explicitly under the workspace lock; no in-place reinterpretation.
|
||||
|
||||
**Tech Stack:** TypeScript 5, Node 22, Python 3.12 (harness), Pydantic 2, Qdrant, Vitest, pytest, Bash. Host interface remains `thothctl` (Go) unchanged in grammar; only its effective-config identity benefits.
|
||||
|
||||
**Source contract:** `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` §5 (P3), PRD D3, and the P1.1 registry contract.
|
||||
|
||||
---
|
||||
|
||||
## P3 completion contract
|
||||
|
||||
P3 is complete only when all of the following are true:
|
||||
|
||||
1. A **versioned shared canonicalizer** (`effective-config` TS module, plus the matching harness canonical form) derives the non-secret effective DWH/preprocessing configuration deterministically. Both the session runtime renderer (`ThtRunner`) and the operator entrypoint (`workspace-maintenance`) consume the same canonicalizer and produce byte-identical effective DWH bindings for the same workspace revision.
|
||||
2. **Stable logical config-source identity** replaces dependence on random temporary config filenames: the operator config lease path and the `OWNER.json` `input_fingerprint` derive from the same canonical logical identity, so two runs over the same revision reuse the same DWH generation.
|
||||
3. **Content-only revisions never invalidate DWH generations**: `session_storage` and `runtime_identity` remain excluded from the effective-config fingerprint; a Git content-only Evidence/annotation commit does not force introspection. A semantically identical revision (same effective DWH binding) reuses the existing generation.
|
||||
4. **Effective-config changes fail closed**: a changed endpoint/transport/database/schema/root-affecting policy produces a different binding identity; the harness refuses to reuse a generation owned by a different effective configuration (`effective_config_mismatch` semantics) and the operator surfaces a stable code.
|
||||
5. **`OWNER.json` schema-v1 compatibility**: existing schema-v1 roots remain readable; an explicit, reviewed migration upgrades them to the versioned identity without in-place reinterpretation. Migration is documented and tested; conflicting legacy roots fail closed.
|
||||
6. **Explicit workspace-global memory root**: the rendered harness config includes `paths.memory` = `<dataRoot>/sessions/<workspace-id>/memory`. All memory commands, locks, registry JSONL, and Qdrant projection rebuilds use that root. A migration copies and verifies one legacy canonical JSONL under the workspace lock before rebuilding the projection; conflicting legacy registries fail closed.
|
||||
7. **Revision-scoped schema/Evidence state**: Qdrant schema and Evidence point IDs and queries include `workspace_revision`; `annotations`, corpus `ACTIVE`, and schema/Evidence Qdrant records are revision-scoped. Memory/solved records and queries remain workspace-wide.
|
||||
8. **Reusable DWH cache layout**: a workspace cache keyed by the versioned effective DWH binding holds verified `physical.yaml`/LSH generations; each pinned revision's runtime root receives verified snapshots from that cache. No pinned runtime consumes the mutable cache root directly.
|
||||
9. **Documentation**: `.tht-dwh`, immutable generations, `OWNER.json`, `ACTIVE`, input vs config fingerprints, safe migration, regeneration, and recovery are explained in the operator manuals and a dedicated `docs/contracts/tht-dwh.md`.
|
||||
10. A clean-state P3 automated process goal passes without retry, retains machine/human reports and a secret scan, and proves exact cleanup; the P3 manual walkthrough remains PENDING until the owner decides.
|
||||
11. No P4 work (collection lifecycle/rebuild), P5 (Git annotations), or P6 (Evidence materialization) is performed.
|
||||
|
||||
## Frozen decisions
|
||||
|
||||
- Canonicalizer version starts at `1`; the canonical serialization is a deterministic JSON document of the non-secret effective DWH/preprocessing configuration (DWH resource, roots, vector/embedding contract, evidence policy) with a fixed key order.
|
||||
- Logical config-source identity = `<workspace-id>@<canonical-effective-config-sha256>` (versioned), replacing the P2 `_config_source` fallback that used a temporary file path.
|
||||
- `input_fingerprint` = sha256 of the logical config-source identity; `config_fingerprint` = sha256 of the canonical effective-config document (versioned).
|
||||
- `paths.memory` is rendered by the backend for both session and operator; legacy configs without it continue to resolve memory beneath `artifacts/memory` only through the explicit migration path.
|
||||
- Qdrant schema/Evidence record key and query filter include `workspace_revision`; memory/solved keep `workspace_id` only.
|
||||
- No automatic remote migration or in-place reinterpretation of legacy roots; operators run the documented migration under the workspace lock.
|
||||
|
||||
## Target file map
|
||||
|
||||
**Shared canonicalizer (TS)**
|
||||
- Create `backend/src/workspaces/effective-config.ts`, `effective-config.test.ts`: versioned canonical serialization, logical identity, fingerprint helpers.
|
||||
- Modify `backend/src/workspaces/runtime-config-lease.ts`: lease naming/identity from the logical identity (deterministic, no random names for the same revision); publish the canonical effective-config identity in the lease manifest.
|
||||
- Modify `backend/src/workspace-maintenance.ts` and `backend/src/workspaces/preprocessing-service.ts`: consume the canonicalizer; stable `effective_config_mismatch` surfacing.
|
||||
- Tests: `backend/test/workspace-runtime-config-lease.test.ts`, `workspace-preprocessing-service.test.ts`, `workspace-maintenance.test.ts`, plus new `effective-config.test.ts`.
|
||||
|
||||
**Harness (Python)**
|
||||
- Modify `harness/tht/config.py`: canonical effective-config document and logical identity; `paths.memory` explicit root.
|
||||
- Modify `harness/tht/jobs/dwh_pipeline.py`: `config_dwh_binding` consumes the canonical identity; `OWNER.json` schema-v1 compatibility reader + migration guard.
|
||||
- Modify `harness/tht/cli/memory_cmd.py` (and locks/registry): explicit `paths.memory` root; migration of one legacy canonical JSONL under the workspace lock.
|
||||
- Modify `harness/tht/vectorstore/records.py` and `harness/tht/adapters/vector/qdrant.py`: `workspace_revision` in schema/Evidence point IDs and queries.
|
||||
- Tests: `harness/tests/test_dwh_preprocess_job.py`, `test_lsh_job_resume.py`, `test_memory_*.py`, `test_qdrant_*.py`, `test_registry_evidence_config.py`.
|
||||
|
||||
**Docs**
|
||||
- Create `docs/contracts/tht-dwh.md`.
|
||||
- Modify `docs/install/local-workspace-registry.md`, `docs/install/server-workspace-registry.md`, `docs/testing/p2-p6-manual-verification.md` (P3 section).
|
||||
- Modify after evidence exists: `PROJECT_STATE.md`.
|
||||
|
||||
**Acceptance**
|
||||
- Create `scripts/p3-acceptance.sh`, `scripts/test-p3-acceptance.sh`, `backend/scripts/p3-acceptance.mjs`, `backend/scripts/p3-acceptance.test.mjs` (pattern: P2 acceptance runner).
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Versioned shared canonicalizer and logical identity (TS)
|
||||
|
||||
**Files:** create `backend/src/workspaces/effective-config.ts` + test; modify `backend/src/workspaces/runtime-config-lease.ts`, `workspace-maintenance.ts`, `preprocessing-service.ts` + tests.
|
||||
|
||||
1. Write failing tests: canonical document is deterministic (byte-identical for equal inputs, key-ordered, versioned); logical identity is `<workspace-id>@sha256:<64hex>`; fingerprint helpers produce `sha256:` values; `session_storage`/`runtime_identity` are excluded; a content-only revision (descriptor change without DWH-affecting fields) yields the same identity; a DWH endpoint/transport/database/schema/root-affecting change yields a different identity; the operator lease path for the same revision is deterministic (no random component for the same logical identity).
|
||||
2. Implement the canonicalizer: fixed key order, non-secret fields only (never binding file contents, endpoints are allowed as non-secret identity inputs), versioned envelope.
|
||||
3. Wire the operator lease manifest to carry `effectiveConfigIdentity`; replace the random lease-name component for the same revision with the deterministic identity suffix (retaining uniqueness across revisions).
|
||||
4. Commit: `feat: versioned effective-config canonicalizer (P3)`.
|
||||
|
||||
### Task 2: Harness canonical form and `OWNER.json` versioned identity
|
||||
|
||||
**Files:** modify `harness/tht/config.py`, `harness/tht/jobs/dwh_pipeline.py` + tests.
|
||||
|
||||
1. Failing tests: `config_dwh_binding` derives `config_fingerprint`/`input_fingerprint` from the versioned canonical document and logical identity; the schema-v1 `OWNER.json` shape remains readable; a versioned root is written with the new fields; a semantically identical revision reuses the generation; a changed endpoint fails closed (never reuses the old generation); `session_storage`/`runtime_identity` exclusion is preserved (content-only commit does not invalidate).
|
||||
2. Implement: canonical JSON document + logical identity in the harness (mirror of Task 1, versioned); `OWNER.json` compatibility reader accepting schema-v1 keys and the versioned shape; explicit `effective_config_mismatch` refusal when a root belongs to a different canonical identity.
|
||||
3. Commit: `feat: versioned OWNER.json identity and compatibility (P3)`.
|
||||
|
||||
### Task 3: DWH cache keyed by effective binding + revision-scoped runtime roots
|
||||
|
||||
**Files:** modify `harness/tht/jobs/dwh_pipeline.py`, runtime root selection (`harness/tht/config.py` roots), `backend/src/workspaces/runtime-config-lease.ts` + tests.
|
||||
|
||||
1. Failing tests: the workspace DWH cache is keyed by the versioned effective binding; each pinned revision's runtime root receives verified `physical.yaml`/LSH snapshots from the cache; no pinned runtime reads the mutable cache root directly; a content-only commit changes the revision runtime root but reuses the cache; a binding change creates a new cache root and fails closed on reuse.
|
||||
2. Implement cache/root separation and verified snapshot handoff (hash-verified copies under the revision runtime root).
|
||||
3. Commit: `feat: effective-binding DWH cache with revision-scoped runtime roots (P3)`.
|
||||
|
||||
### Task 4: Explicit workspace-global memory root + legacy migration
|
||||
|
||||
**Files:** modify `harness/tht/config.py`, `harness/tht/cli/memory_cmd.py`, memory registry/lock; `backend/src/workspaces/runtime-config-lease.ts` (render `paths.memory`); tests.
|
||||
|
||||
1. Failing tests: rendered config includes `paths.memory` = `<dataRoot>/sessions/<id>/memory`; memory commands/locks/registry JSONL use it; migration copies and verifies exactly one legacy canonical JSONL under the workspace lock and rebuilds the Qdrant projection; two conflicting legacy registries fail closed; no in-place reinterpretation.
|
||||
2. Implement the memory root plumbing and the guarded migration.
|
||||
3. Commit: `feat: explicit workspace memory root with guarded migration (P3)`.
|
||||
|
||||
### Task 5: Revision-scoped Qdrant schema/Evidence records
|
||||
|
||||
**Files:** modify `harness/tht/vectorstore/records.py`, `harness/tht/adapters/vector/qdrant.py`, schema-index and evidence writers + tests.
|
||||
|
||||
1. Failing tests: schema and Evidence point IDs include `workspace_revision`; schema/Evidence queries filter by `workspace_revision`; memory/solved identities and queries remain workspace-wide; existing workspace-global points are not silently reinterpreted (explicit re-index after migration is required).
|
||||
2. Implement the revision-scoped record keys/filters.
|
||||
3. Commit: `feat: revision-scoped schema and Evidence vector records (P3)`.
|
||||
|
||||
### Task 6: Proofs — operator/session identity, reuse, fail-closed
|
||||
|
||||
**Files:** extend `backend/test/workspace-runtime-config-lease.test.ts`, `workspace-preprocessing-service.test.ts`, `harness/tests/test_dwh_preprocess_job.py`.
|
||||
|
||||
1. Failing tests (cross-layer): for the same workspace revision, the operator-rendered effective DWH binding is byte-identical to the session-rendered one; a semantically identical revision reuses the DWH generation (rerun `unchanged`); a changed endpoint/transport/database/schema/root-affecting policy fails closed with `effective_config_mismatch` (never silently reusing artifacts).
|
||||
2. Implement any gap the tests expose (expect the canonicalizer to be the single shared source).
|
||||
3. Commit: `test: prove operator/session effective-config identity (P3)`.
|
||||
|
||||
### Task 7: Documentation — `.tht-dwh`, generations, fingerprints, migration, recovery
|
||||
|
||||
**Files:** create `docs/contracts/tht-dwh.md`; modify install manuals and `docs/testing/p2-p6-manual-verification.md` (P3 section).
|
||||
|
||||
1. Write the contract doc explaining: what `.tht-dwh` is, immutable generations, `OWNER.json` (schema-v1 vs versioned), `ACTIVE`, input vs config fingerprints, why a fingerprint protects against artifacts of another configuration, the safe migration procedure, regeneration, and recovery.
|
||||
2. Update the operator manuals with the P3 migration step and the P3 walkthrough section (decision PENDING).
|
||||
3. Commit: `docs: explain .tht-dwh and P3 migration (P3)`.
|
||||
|
||||
### Task 8: Clean-state P3 automated process goal
|
||||
|
||||
**Files:** create `scripts/p3-acceptance.sh`, `scripts/test-p3-acceptance.sh`, `backend/scripts/p3-acceptance.mjs`, `backend/scripts/p3-acceptance.test.mjs` (pattern: P2 acceptance runner, owned root `.artifacts/p3-integration/p3-<run-id>/`).
|
||||
|
||||
1. Write RED runner tests (ownership, run-id, cleanup, --keep, injected failure, report bounds, no retry).
|
||||
2. Implement the clean-state scenario: build fixtures (P1.1 registry + REST DWH + HTTP Evidence + pre-provisioned Qdrant + embedding stub); run `thothctl` product commands; prove: identical operator/session effective binding, content-only revision reuse (`unchanged`), DWH-affecting change fails closed, memory migration + rebuild, revision-scoped Qdrant records, no P4/P5/P6 scope, secret scan, exact cleanup.
|
||||
3. Run the runner tests, then one clean integration run without retry; retain the report (`report.md` ends with `P3 automated integration: PASS` / `P3 manual acceptance: PENDING`).
|
||||
4. Commit: `test: add P3 effective-config process acceptance`.
|
||||
|
||||
### Task 9: Final verification and owner handoff
|
||||
|
||||
1. Full backend Vitest + tsc + build; full frontend Vitest + tsc + build (unchanged expectations); harness pytest (excluding the documented pre-existing debt) + Ruff on touched files; Go build/tests (unchanged grammar); compose config checks; docs gates; one final clean P3 acceptance run.
|
||||
2. Update `PROJECT_STATE.md` (P3 implementation complete, automated PASS, manual PENDING) only after evidence exists.
|
||||
3. Stop. No P4 work begins without a new explicit authorization.
|
||||
|
||||
---
|
||||
|
||||
## Owner checkpoint
|
||||
|
||||
After Task 9 the implementation session stops. The owner executes the P3 walkthrough in
|
||||
`docs/testing/p2-p6-manual-verification.md` and records the decision. P4 (Qdrant collection
|
||||
lifecycle), P5 (Git FK annotations), and P6 (Evidence materialization) start only after explicit
|
||||
authorization.
|
||||
|
||||
## Explicit exclusions
|
||||
|
||||
- No collection create/repair/rebuild (P4), no Git annotation synchronization (P5), no Evidence
|
||||
materialization (P6), no changes to the host `thothctl` grammar, no migration of real PSD
|
||||
content, no changes to accepted P1/P1.1/P2 evidence.
|
||||
Reference in New Issue
Block a user