mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: record P2 manual acceptance and plan P3 effective configuration

Browse Source
This commit is contained in:
marcopan
2026-08-12 14:44:27 +02:00
parent 3c5c2e8afd
commit 13f74de4fa
2 changed files with 153 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PROJECT_STATE.md
+3 -3
View File
@@ -23,9 +23,9 @@
- **Retained evidence:** `.artifacts/p2-integration/p2-b109757b26388a5ed6b1d173dee86584/`
(11/11 checks PASS), bound to clean source commit
`de5de36f9a4edfd4fbebf277822090871ccdd61f`.
- **Manual gate:** P2 walkthrough in `docs/testing/p2-p6-manual-verification.md`; decision
**PENDING** and recorded independently. P3 and later start only after an explicit new
authorization.
- **Manual gate:** P2 walkthrough in `docs/testing/p2-p6-manual-verification.md`; the owner
approved P2 on 2026-08-11 (manual acceptance PASS). P3 and later start only after an explicit
new authorization.
### P1.1 workspace-directory registry — automated integration PASS, manual PENDING (2026-08-11)
docs/superpowers/plans/2026-08-11-p3-effective-config-and-tht-dwh.md
+150
View File
@@ -0,0 +1,150 @@
# P3 Effective Configuration and `.tht-dwh` Ownership — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Use superpowers:test-driven-development for every behavior change and superpowers:verification-before-completion before any completion claim.
**Goal:** Make the effective DWH/preprocessing configuration reproducible and versioned across the operator and session paths, key reusable DWH generations by a stable logical identity instead of random temporary config files, scope schema/Evidence state to the pinned revision, add an explicit workspace-global memory root with a safe migration, and document `.tht-dwh` for operators.
**Architecture:** A versioned shared canonicalizer (TS, shared by the backend session renderer and the compiled operator entrypoint) produces the non-secret effective DWH/preprocessing configuration and its stable logical config-source identity; the harness consumes the same canonical form when writing `OWNER.json`. DWH cache roots are keyed by the versioned effective DWH binding; revision-scoped runtime roots receive verified physical/LSH snapshots from that cache. An explicit `paths.memory` root becomes workspace-global; schema/Evidence Qdrant records and queries carry `workspace_revision` while memory/solved stay workspace-wide. Existing `OWNER.json` schema-v1 roots and legacy memory JSONL are read-compatible and migrated explicitly under the workspace lock; no in-place reinterpretation.
**Tech Stack:** TypeScript 5, Node 22, Python 3.12 (harness), Pydantic 2, Qdrant, Vitest, pytest, Bash. Host interface remains `thothctl` (Go) unchanged in grammar; only its effective-config identity benefits.
**Source contract:** `docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md` §5 (P3), PRD D3, and the P1.1 registry contract.
---
## P3 completion contract
P3 is complete only when all of the following are true:
1. A **versioned shared canonicalizer** (`effective-config` TS module, plus the matching harness canonical form) derives the non-secret effective DWH/preprocessing configuration deterministically. Both the session runtime renderer (`ThtRunner`) and the operator entrypoint (`workspace-maintenance`) consume the same canonicalizer and produce byte-identical effective DWH bindings for the same workspace revision.
2. **Stable logical config-source identity** replaces dependence on random temporary config filenames: the operator config lease path and the `OWNER.json` `input_fingerprint` derive from the same canonical logical identity, so two runs over the same revision reuse the same DWH generation.
3. **Content-only revisions never invalidate DWH generations**: `session_storage` and `runtime_identity` remain excluded from the effective-config fingerprint; a Git content-only Evidence/annotation commit does not force introspection. A semantically identical revision (same effective DWH binding) reuses the existing generation.
4. **Effective-config changes fail closed**: a changed endpoint/transport/database/schema/root-affecting policy produces a different binding identity; the harness refuses to reuse a generation owned by a different effective configuration (`effective_config_mismatch` semantics) and the operator surfaces a stable code.
5. **`OWNER.json` schema-v1 compatibility**: existing schema-v1 roots remain readable; an explicit, reviewed migration upgrades them to the versioned identity without in-place reinterpretation. Migration is documented and tested; conflicting legacy roots fail closed.
6. **Explicit workspace-global memory root**: the rendered harness config includes `paths.memory` = `<dataRoot>/sessions/<workspace-id>/memory`. All memory commands, locks, registry JSONL, and Qdrant projection rebuilds use that root. A migration copies and verifies one legacy canonical JSONL under the workspace lock before rebuilding the projection; conflicting legacy registries fail closed.
7. **Revision-scoped schema/Evidence state**: Qdrant schema and Evidence point IDs and queries include `workspace_revision`; `annotations`, corpus `ACTIVE`, and schema/Evidence Qdrant records are revision-scoped. Memory/solved records and queries remain workspace-wide.
8. **Reusable DWH cache layout**: a workspace cache keyed by the versioned effective DWH binding holds verified `physical.yaml`/LSH generations; each pinned revision's runtime root receives verified snapshots from that cache. No pinned runtime consumes the mutable cache root directly.
9. **Documentation**: `.tht-dwh`, immutable generations, `OWNER.json`, `ACTIVE`, input vs config fingerprints, safe migration, regeneration, and recovery are explained in the operator manuals and a dedicated `docs/contracts/tht-dwh.md`.
10. A clean-state P3 automated process goal passes without retry, retains machine/human reports and a secret scan, and proves exact cleanup; the P3 manual walkthrough remains PENDING until the owner decides.
11. No P4 work (collection lifecycle/rebuild), P5 (Git annotations), or P6 (Evidence materialization) is performed.
## Frozen decisions
- Canonicalizer version starts at `1`; the canonical serialization is a deterministic JSON document of the non-secret effective DWH/preprocessing configuration (DWH resource, roots, vector/embedding contract, evidence policy) with a fixed key order.
- Logical config-source identity = `<workspace-id>@<canonical-effective-config-sha256>` (versioned), replacing the P2 `_config_source` fallback that used a temporary file path.
- `input_fingerprint` = sha256 of the logical config-source identity; `config_fingerprint` = sha256 of the canonical effective-config document (versioned).
- `paths.memory` is rendered by the backend for both session and operator; legacy configs without it continue to resolve memory beneath `artifacts/memory` only through the explicit migration path.
- Qdrant schema/Evidence record key and query filter include `workspace_revision`; memory/solved keep `workspace_id` only.
- No automatic remote migration or in-place reinterpretation of legacy roots; operators run the documented migration under the workspace lock.
## Target file map
**Shared canonicalizer (TS)**
- Create `backend/src/workspaces/effective-config.ts`, `effective-config.test.ts`: versioned canonical serialization, logical identity, fingerprint helpers.
- Modify `backend/src/workspaces/runtime-config-lease.ts`: lease naming/identity from the logical identity (deterministic, no random names for the same revision); publish the canonical effective-config identity in the lease manifest.
- Modify `backend/src/workspace-maintenance.ts` and `backend/src/workspaces/preprocessing-service.ts`: consume the canonicalizer; stable `effective_config_mismatch` surfacing.
- Tests: `backend/test/workspace-runtime-config-lease.test.ts`, `workspace-preprocessing-service.test.ts`, `workspace-maintenance.test.ts`, plus new `effective-config.test.ts`.
**Harness (Python)**
- Modify `harness/tht/config.py`: canonical effective-config document and logical identity; `paths.memory` explicit root.
- Modify `harness/tht/jobs/dwh_pipeline.py`: `config_dwh_binding` consumes the canonical identity; `OWNER.json` schema-v1 compatibility reader + migration guard.
- Modify `harness/tht/cli/memory_cmd.py` (and locks/registry): explicit `paths.memory` root; migration of one legacy canonical JSONL under the workspace lock.
- Modify `harness/tht/vectorstore/records.py` and `harness/tht/adapters/vector/qdrant.py`: `workspace_revision` in schema/Evidence point IDs and queries.
- Tests: `harness/tests/test_dwh_preprocess_job.py`, `test_lsh_job_resume.py`, `test_memory_*.py`, `test_qdrant_*.py`, `test_registry_evidence_config.py`.
**Docs**
- Create `docs/contracts/tht-dwh.md`.
- Modify `docs/install/local-workspace-registry.md`, `docs/install/server-workspace-registry.md`, `docs/testing/p2-p6-manual-verification.md` (P3 section).
- Modify after evidence exists: `PROJECT_STATE.md`.
**Acceptance**
- Create `scripts/p3-acceptance.sh`, `scripts/test-p3-acceptance.sh`, `backend/scripts/p3-acceptance.mjs`, `backend/scripts/p3-acceptance.test.mjs` (pattern: P2 acceptance runner).
---
### Task 1: Versioned shared canonicalizer and logical identity (TS)
**Files:** create `backend/src/workspaces/effective-config.ts` + test; modify `backend/src/workspaces/runtime-config-lease.ts`, `workspace-maintenance.ts`, `preprocessing-service.ts` + tests.
1. Write failing tests: canonical document is deterministic (byte-identical for equal inputs, key-ordered, versioned); logical identity is `<workspace-id>@sha256:<64hex>`; fingerprint helpers produce `sha256:` values; `session_storage`/`runtime_identity` are excluded; a content-only revision (descriptor change without DWH-affecting fields) yields the same identity; a DWH endpoint/transport/database/schema/root-affecting change yields a different identity; the operator lease path for the same revision is deterministic (no random component for the same logical identity).
2. Implement the canonicalizer: fixed key order, non-secret fields only (never binding file contents, endpoints are allowed as non-secret identity inputs), versioned envelope.
3. Wire the operator lease manifest to carry `effectiveConfigIdentity`; replace the random lease-name component for the same revision with the deterministic identity suffix (retaining uniqueness across revisions).
4. Commit: `feat: versioned effective-config canonicalizer (P3)`.
### Task 2: Harness canonical form and `OWNER.json` versioned identity
**Files:** modify `harness/tht/config.py`, `harness/tht/jobs/dwh_pipeline.py` + tests.
1. Failing tests: `config_dwh_binding` derives `config_fingerprint`/`input_fingerprint` from the versioned canonical document and logical identity; the schema-v1 `OWNER.json` shape remains readable; a versioned root is written with the new fields; a semantically identical revision reuses the generation; a changed endpoint fails closed (never reuses the old generation); `session_storage`/`runtime_identity` exclusion is preserved (content-only commit does not invalidate).
2. Implement: canonical JSON document + logical identity in the harness (mirror of Task 1, versioned); `OWNER.json` compatibility reader accepting schema-v1 keys and the versioned shape; explicit `effective_config_mismatch` refusal when a root belongs to a different canonical identity.
3. Commit: `feat: versioned OWNER.json identity and compatibility (P3)`.
### Task 3: DWH cache keyed by effective binding + revision-scoped runtime roots
**Files:** modify `harness/tht/jobs/dwh_pipeline.py`, runtime root selection (`harness/tht/config.py` roots), `backend/src/workspaces/runtime-config-lease.ts` + tests.
1. Failing tests: the workspace DWH cache is keyed by the versioned effective binding; each pinned revision's runtime root receives verified `physical.yaml`/LSH snapshots from the cache; no pinned runtime reads the mutable cache root directly; a content-only commit changes the revision runtime root but reuses the cache; a binding change creates a new cache root and fails closed on reuse.
2. Implement cache/root separation and verified snapshot handoff (hash-verified copies under the revision runtime root).
3. Commit: `feat: effective-binding DWH cache with revision-scoped runtime roots (P3)`.
### Task 4: Explicit workspace-global memory root + legacy migration
**Files:** modify `harness/tht/config.py`, `harness/tht/cli/memory_cmd.py`, memory registry/lock; `backend/src/workspaces/runtime-config-lease.ts` (render `paths.memory`); tests.
1. Failing tests: rendered config includes `paths.memory` = `<dataRoot>/sessions/<id>/memory`; memory commands/locks/registry JSONL use it; migration copies and verifies exactly one legacy canonical JSONL under the workspace lock and rebuilds the Qdrant projection; two conflicting legacy registries fail closed; no in-place reinterpretation.
2. Implement the memory root plumbing and the guarded migration.
3. Commit: `feat: explicit workspace memory root with guarded migration (P3)`.
### Task 5: Revision-scoped Qdrant schema/Evidence records
**Files:** modify `harness/tht/vectorstore/records.py`, `harness/tht/adapters/vector/qdrant.py`, schema-index and evidence writers + tests.
1. Failing tests: schema and Evidence point IDs include `workspace_revision`; schema/Evidence queries filter by `workspace_revision`; memory/solved identities and queries remain workspace-wide; existing workspace-global points are not silently reinterpreted (explicit re-index after migration is required).
2. Implement the revision-scoped record keys/filters.
3. Commit: `feat: revision-scoped schema and Evidence vector records (P3)`.
### Task 6: Proofs — operator/session identity, reuse, fail-closed
**Files:** extend `backend/test/workspace-runtime-config-lease.test.ts`, `workspace-preprocessing-service.test.ts`, `harness/tests/test_dwh_preprocess_job.py`.
1. Failing tests (cross-layer): for the same workspace revision, the operator-rendered effective DWH binding is byte-identical to the session-rendered one; a semantically identical revision reuses the DWH generation (rerun `unchanged`); a changed endpoint/transport/database/schema/root-affecting policy fails closed with `effective_config_mismatch` (never silently reusing artifacts).
2. Implement any gap the tests expose (expect the canonicalizer to be the single shared source).
3. Commit: `test: prove operator/session effective-config identity (P3)`.
### Task 7: Documentation — `.tht-dwh`, generations, fingerprints, migration, recovery
**Files:** create `docs/contracts/tht-dwh.md`; modify install manuals and `docs/testing/p2-p6-manual-verification.md` (P3 section).
1. Write the contract doc explaining: what `.tht-dwh` is, immutable generations, `OWNER.json` (schema-v1 vs versioned), `ACTIVE`, input vs config fingerprints, why a fingerprint protects against artifacts of another configuration, the safe migration procedure, regeneration, and recovery.
2. Update the operator manuals with the P3 migration step and the P3 walkthrough section (decision PENDING).
3. Commit: `docs: explain .tht-dwh and P3 migration (P3)`.
### Task 8: Clean-state P3 automated process goal
**Files:** create `scripts/p3-acceptance.sh`, `scripts/test-p3-acceptance.sh`, `backend/scripts/p3-acceptance.mjs`, `backend/scripts/p3-acceptance.test.mjs` (pattern: P2 acceptance runner, owned root `.artifacts/p3-integration/p3-<run-id>/`).
1. Write RED runner tests (ownership, run-id, cleanup, --keep, injected failure, report bounds, no retry).
2. Implement the clean-state scenario: build fixtures (P1.1 registry + REST DWH + HTTP Evidence + pre-provisioned Qdrant + embedding stub); run `thothctl` product commands; prove: identical operator/session effective binding, content-only revision reuse (`unchanged`), DWH-affecting change fails closed, memory migration + rebuild, revision-scoped Qdrant records, no P4/P5/P6 scope, secret scan, exact cleanup.
3. Run the runner tests, then one clean integration run without retry; retain the report (`report.md` ends with `P3 automated integration: PASS` / `P3 manual acceptance: PENDING`).
4. Commit: `test: add P3 effective-config process acceptance`.
### Task 9: Final verification and owner handoff
1. Full backend Vitest + tsc + build; full frontend Vitest + tsc + build (unchanged expectations); harness pytest (excluding the documented pre-existing debt) + Ruff on touched files; Go build/tests (unchanged grammar); compose config checks; docs gates; one final clean P3 acceptance run.
2. Update `PROJECT_STATE.md` (P3 implementation complete, automated PASS, manual PENDING) only after evidence exists.
3. Stop. No P4 work begins without a new explicit authorization.
---
## Owner checkpoint
After Task 9 the implementation session stops. The owner executes the P3 walkthrough in
`docs/testing/p2-p6-manual-verification.md` and records the decision. P4 (Qdrant collection
lifecycle), P5 (Git FK annotations), and P6 (Evidence materialization) start only after explicit
authorization.
## Explicit exclusions
- No collection create/repair/rebuild (P4), no Git annotation synchronization (P5), no Evidence
materialization (P6), no changes to the host `thothctl` grammar, no migration of real PSD
content, no changes to accepted P1/P1.1/P2 evidence.