2343 lines
155 KiB
Markdown
2343 lines
155 KiB
Markdown
# P3 Effective Configuration Fingerprint and Revision Isolation Implementation Plan
|
||
|
||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** Make DWH-derived preprocessing safely reusable across semantically equivalent Git revisions while keeping curated artifacts, corpus state, and schema/Evidence vectors revision-pinned and Memory workspace-global, with explicit compatibility migrations from the P2 layout.
|
||
|
||
**Architecture:** The Python harness owns one versioned canonical effective-DWH binding computed from an allowlist of non-secret, output-affecting values. Both the P2 dedicated one-shot maintenance process and session runtime consume the same rendered config and ask the harness for that binding; neither hashes temporary YAML paths independently. `thothctl` launches only the profile-gated `workspace-maintenance` Compose job: that process owns client-addressed durable run creation/replay, owner-qualified quiescence, the exclusive reader gate, and the inherited P2 writer-lock capability without requiring Fastify or stopping `core`. A binding-keyed workspace-global DWH cache feeds verified immutable snapshots into revision-qualified runtime roots, while explicit migrations copy and verify schema-v1 owners and legacy Memory state without reinterpreting or merging it in place. Qdrant partitions identity by `(workspace_id, workspace_revision)` for schema/Evidence and by `workspace_id` alone for Memory/solved records.
|
||
|
||
**Tech Stack:** Python 3.12, Pydantic v2, Typer, pytest, Node.js 22, TypeScript, Vitest, repo-owned Node-API v8 `workspace-fs-at` with a closed typed lock API and wrapper-internal exact `fs-ext@2.1.1` for `flock(2)` only, Go 1.26.5 `thothctl` (matching `go.mod` `toolchain go1.26.5`), Docker Compose, Qdrant, Ollama, Bash, Git, YAML/JSON, SHA-256, UUIDv5, POSIX `*at`/`flock`/`fsync`/atomic rename.
|
||
|
||
---
|
||
|
||
## Scope, dependency gate, and invariants
|
||
|
||
This is **P3 / D3 only**. P2 must already be implemented, automatically green, manually accepted, and present in this worktree. P3 extends—without renaming, wrapping in a parallel tree, or duplicating—P2's exact frozen handoff:
|
||
|
||
- `backend/src/workspaces/runtime-config-lease.ts` / `WorkspaceRuntimeConfigLeaseFactory.acquireSession` and `.acquireMaintenance`;
|
||
- `backend/native/workspace-fs-at/{workspace_fs_at.cc,binding.gyp}`, `backend/src/native/workspace-fs-at-binding.d.ts`, `backend/src/workspaces/workspace-fs-at.ts`, and `backend/scripts/build-workspace-fs-at.mjs` / the repo-owned Node-API v8 `WorkspaceFsAtV1` seam with typed `openat`/`mkdirat`/no-follow `fstatat`/directory-fsync/close ownership, exact `LockFileName = "writer.lock" | "session-readers.lock"`, `openOrCreateLockAt(...)`, and `flockOwnedLock(handle, ownership, wait)`; exact `fs-ext@2.1.1` remains wrapper-internal and `flock(2)`-only;
|
||
- `backend/src/workspaces/workspace-lock-root-lease.ts` / `CanonicalWorkspaceLockRootInput`, `BorrowedVerifiedWorkspaceLockRootLease`, `VerifiedWorkspaceLockRootLease`, `WorkspaceSessionReadersLockLease`, `VerifiedWorkspaceLockRootLease.acquireSessionReadersShared()`, `VerifiedWorkspaceLockRootLeaseFactory.acquire`/`.acquireOrProvision`, and the retained root identity, all consuming only that typed FD-relative seam;
|
||
- `backend/src/workspaces/preprocessing-state.ts` / `PreprocessingStateStore`, `BorrowedWorkspaceSessionReadersExclusiveLockLease`, `WorkspaceWriterLockCapability.runUnderSessionReadersExclusive(...)`, `OrderedWorkspaceWriterCapabilitySet`, `runUnderWorkspaceWriterLock`, `runUnderOrderedWorkspaceWriterLocks`, and the exact inherited writer FD 3/root FD 4 contract;
|
||
- `backend/src/workspaces/registry-publication.ts` / the exact `RegistryAddressedRequestV1` bootstrap/pull union, `RegistryAddressedPlanV1` bootstrap/pull union, `RegistryAddressedPublicationStateV1` bootstrap/pull union, `RegistryBootstrapRecoveryIdentityV1`, `RegistryBootstrapRecoveryScanLimitsV1`, `RegistryActiveSnapshotV1`, and exact `RegistryEnsureBootstrapAddressedResultV1` (`already_active` snapshot or `bootstrap_terminal` result), phases `request_claimed` through `terminal_durable`, `AddressedWorkspacePublicationLeaseV1`, `CapabilityAwareRegistryPublicationParticipant`, `CapabilityAwareRegistryPublicationSynchronizer`, and `CapabilityAwareRegistryPublicationLifecycleOwner.run`;
|
||
- `backend/src/workspaces/registry.ts` / `WorkspaceRegistry.ensureBootstrapAddressed` as the sole bounded automatic bootstrap selector and `WorkspaceRegistry.publishAddressed` for explicit addressed mutation;
|
||
- `backend/src/routes/workspaces.ts` and `backend/src/app.ts` / inspect, lazy bootstrap, and status routed only through `ensureBootstrapAddressed`, pull and author publication routed only through `publishAddressed`, with no old `bootstrap`/`pull`/`activate`/direct-pointer escape;
|
||
- `backend/src/workspaces/preprocessing-service.ts` / `WorkspacePreprocessingService.execute`;
|
||
- the single `backend/src/workspace-maintenance.ts` / exported `main` entrypoint;
|
||
- `tools/thothctl/internal/workspaceops/operations.go` / `ParseWorkspaceCommand` and `Run`;
|
||
- P2 tests `workspace-fs-at-native.test.ts`, `workspace-lock-root-lease.test.ts`, `workspace-session-readers-lock.test.ts`, `workspace-runtime-config-lease.test.ts`, `workspace-preprocessing-state.test.ts`, `workspace-registry-addressed-publication.test.ts`, `workspace-registry-addressed-process.test.ts`, `routes-workspaces.test.ts`, `app.test.ts`, `workspace-preprocessing-service.test.ts`, and `workspace-maintenance.test.ts`;
|
||
- `scripts/p2-acceptance.sh`.
|
||
|
||
No `backend/src/workspace-maintenance/` directory, `WorkspaceMaintenanceOperator`, `tools/thothctl/internal/workspace` package, `workspace.Parse`, or `workspace.Run` may be introduced: those names are nonexistent and conflict with the P2 handoff.
|
||
|
||
Do not start P3 against the current pre-P2 tree. Do not implement P4 collection creation/rebuild, P5 Git annotation synchronization, P6 filesystem Evidence materialization, PSD migration, SSH runtime transport, GUI/API preprocessing, or aggregate P2–P6 verification.
|
||
|
||
Preserve these invariants throughout:
|
||
|
||
1. P2 reads/writes the existing `OWNER.json` schema-v1 contract unchanged until the explicit P3 migration code and tests exist. Capture a valid P2 schema-v1 fixture before changing the writer.
|
||
2. The reusable digest excludes secrets, credential values/files, Git commit, random config filename, `runtime_identity`, `session_storage`, vector/embedding/search/execution settings, and revision-qualified output paths.
|
||
3. The digest includes every non-secret value that can change physical/LSH output: DWH transport and canonical endpoint identity, database/schema and non-secret login identity, examples/introspection, eligibility and LSH policies, plus an explicit artifact-layout policy version.
|
||
4. A content-only Git commit reuses a cache. A changed endpoint, transport, database, schema, or included policy gets another cache and never falls back to the old cache.
|
||
5. Runtime layout is exactly:
|
||
|
||
```text
|
||
/data/sessions/<workspace-id>/
|
||
sessions/ # workspace-global session manifests
|
||
memory/ # workspace-global canonical registry
|
||
preprocessing/dwh-cache/<binding-key>/ # reusable immutable generations
|
||
revisions/<40-hex>/readiness/<binding-key>/READY.json
|
||
revisions/<40-hex>/dwh-snapshots/<binding-key>/{artifacts,indexes}/
|
||
revisions/<40-hex>/artifacts/ # annotations and revision-owned artifacts
|
||
revisions/<40-hex>/indexes/ # revision-owned indexes
|
||
revisions/<40-hex>/corpus/ # corpus generations + ACTIVE
|
||
```
|
||
|
||
6. `paths.memory` is authoritative when present. Only a config that lacks it may use the compatibility fallback `paths.artifacts/memory`.
|
||
7. Schema/Evidence Qdrant reads, hashes, writes, lists, and deletes require a 40-hex revision in point identity and filter. Memory/solved records deliberately omit revision from identity and filters.
|
||
8. P3 is additive until workspace layout enablement. `preprocessing/layout-version.json` is one workspace-global version marker and never names a revision or binding. Readiness is immutable and keyed by the exact pair `(40-hex Git commit, 64-hex effective-DWH cache key)` at `revisions/<commit>/readiness/<binding-key>/READY.json`, but **at most one effective binding may ever become READY for a Git revision**. Before preparation or activation, securely enumerate that revision's readiness directory: an existing valid READY for the selected key may only byte-match; any valid READY for another key makes activation fail `effective_config_mismatch` before cache/snapshot/semantic writes. A changed effective binding therefore requires a new content commit/revision. Once the global marker exists, a session probes its harness-owned effective binding, selects only that exact readiness generation and binding-qualified DWH snapshot, and fails `migration_required` before Pi/session child spawn when it is absent. Prepare maintenance is available for an unready revision only when no different binding is already READY there, through the trusted future resolver and never through an active fallback.
|
||
9. Every mutating migration/activation child runs under P2's exact opaque `WorkspaceWriterLockCapability`; its sole `spawnChild` method passes the actual locked writer open file description as FD 3 and the same retained verified root directory open file description as FD 4. P3 extends P2's one closed request union in place. Before content access, the child validates FD 4 as the expected service-owned root, opens `preprocessing/writer.lock` relative to FD 4, proves that inode is FD 3, and proves FD 3 is the already-held exclusive lock. P3 introduces no root brand, verified-root string, second capability, direct spawn, or ambient/path authorization seam. Source/destination bytes are reverified, publication is atomic/idempotent, and legacy filesystem sources remain for rollback. Conflicting Memory registries or cache destinations fail closed.
|
||
10. The public non-activating `semantic-revision` command is inventory/rebuild-readiness-only: it persists exact legacy IDs/digests and verifies that current schema/Evidence sources are rebuildable, but performs **zero Qdrant replacement upserts and zero deletes**. Semantic publish-before-delete occurs only inside one quiesced activation transaction: after durable admission blocking and exclusive acquisition of the no-follow reader gate, reverify the persisted inventory and sources, publish and read back every replacement, publish/verify the global reader-layout marker on initial enablement, delete only unchanged exact legacy IDs, and publish that commit+binding READY last. `core` may remain running because the durable marker prevents new admission and the exclusive gate proves zero active session readers. No successful non-activation command can create a mixed legacy/replacement reader interval.
|
||
11. The four ordinary operations (`migrate_dwh_cache`, `migrate_memory`, `migrate_semantic_revision`, and `activate_revision_layout`) keep the selected-workspace **writer-first** lifecycle: the sole P2 factory `acquire`s the existing root, transfers it into `runUnderWorkspaceWriterLock`, then the still-live exact capability owns ordinary job/quiescence, exclusive reader-gate acquisition, participants/locked children, publication, reverse release, and root close. The client pre-generates the run ID; process death retains state and owner quiescence for exact same-ID dead-`core` resume. Only activation may publish Qdrant replacements, switch reader mode/publish the sole binding READY, or delete legacy IDs.
|
||
12. `registry_pull` is the exact repository-first P2 callback exception and never enters the ordinary writer-first function. It invokes only `WorkspaceRegistry.publishAddressed` with a P2 `registry_pull` create/resume literal whose validated installation boundary supplies `installationIdentitySha256`, `repositoryIdentitySha256`, and `remoteRefIdentitySha256`, and whose create branch also supplies `expectedBaseCommit`. Inspect, lazy list/bootstrap, and status instead invoke only `WorkspaceRegistry.ensureBootstrapAddressed`. Its continuously repository-locked selector first revalidates active state: a valid snapshot returns P2's exact `already_active` result without job scanning, state writes, or network; only absence enters bounded exact-identity zero-create/one-resume selection. Corrupt or incompatible active state, corrupt/mismatched/pull/multiple nonterminal jobs, and every identity/base mismatch fail closed before network or state mutation. Under `repository.lock`, P2 durably advances `request_claimed` → `target_advertised` → `target_fetched` → `planned`, then `acquireOrProvision`s every complete changed-ID root through the sole typed `WorkspaceFsAtV1` seam and passes all roots once to `runUnderOrderedWorkspaceWriterLocks`. The exact same callback-scoped set flows through the production lifecycle owner, per-workspace addressed participant leases, every synchronizer, pointer publication, and `terminal_durable`; reverse invalidation/release occurs before repository release. The same owner and callbacks remain valid for P2's no-active-base `registry_bootstrap` plan/result variant. Same-ID recovery is phase-exact: before a durable advertisement it may repeat advertisement; at/after `target_advertised` the recorded OID is permanent, exact-OID fetch/ref reconciliation is the only allowed recovery, and after `target_fetched` no network or target reselection occurs. `migrate_dwh_cache` remains pre-READY preparation and writes no global marker, semantic point, or READY.
|
||
13. Maintenance control is profile-independent and host-unpublished: `workspaceops.Run` invokes only the selected installation's existing profile-gated `docker compose run --rm --no-deps --no-TTY workspace-maintenance` job with bounded canonical stdin/result. It never calls Fastify, an HTTP/internal route, `compose exec core`, a frontend proxy, host Python/Node/Pi/`tht`, or a core stop/start command. Request/result bounds, lock order, no-follow files, deadlines, state transitions, and failure mapping are the exact contract frozen in Task 8.
|
||
14. Every admitted session owns P2's exact `WorkspaceSessionReadersLockLease` from the final quiescence recheck until all Pi/session children and their stdout/stderr streams have settled. `WorkspaceReaderLeaseFactory.acquireForSession` delegates only to `VerifiedWorkspaceLockRootLease.acquireSessionReadersShared()`, and the resulting owner is transferred into `PiProcessManager` for exactly-once teardown. Maintenance publishes/drains quiescence first, then `WorkspaceReaderLeaseFactory.acquireForMaintenance` delegates only to `WorkspaceWriterLockCapability.runUnderSessionReadersExclusive(...)` around the full quiesced action through terminal durability and matching owner clear; its borrowed lease never escapes the callback. P3 never opens/flocks directly, imports `WorkspaceFsAtV1`, `fs-ext`, or the raw addon for reader acquisition, exposes a directory handle/path/FD/lock name/flags, casts a handle/name, or adds another lock wrapper. `--json` remains pristine and all public errors use bounded stable codes, especially `effective_config_mismatch`, `migration_required`, `preprocessing_conflict`, `workspace_quiesced`, `registry_bootstrap_recovery_conflict`, and `semantic_index_incompatible`.
|
||
|
||
## Automated process goal
|
||
|
||
At implementation start, register this persistent goal if the agent runtime supports goals:
|
||
|
||
> From a clean P3 source tree and clean Docker/fixture namespace, use only the released `thothctl` host interface to launch dedicated maintenance jobs that atomically create/resume client-addressed runs, durably block admission, drain shared reader leases, and hold the exclusive reader gate plus the exact P2 capability's writer FD 3 and retained-root FD 4 without Fastify or stopping `core`. Migrate a valid P2 schema-v1 DWH owner and legacy Memory registry; prove `dwh-cache` materializes the selected binding-qualified revision snapshot without READY activation; prove every non-activating command preserves one unmixed reader mode; then use one activation to publish/verify revision-scoped schema/Evidence replacements, switch the global layout, delete exact verified legacy IDs, publish immutable commit+binding READY, and owner-clear quiescence. Pull content-only revision B, block it until its exact READY, and reuse the cache while isolating revision state. Change binding X to Y at READY B, prove B admission and same-revision activation are refused without mutation, publish/pull C, and prove new v2 Y cache → C/Y snapshot → immutable C/Y READY. Restore installation binding X and admit historical B/X, then restore Y and admit C/Y, proving immutable B/X bytes rather than pinned-revision-only operability; retain global Memory/solved, reject conflicts, survive dead-core resume and SIGKILL, scan fixture secrets, and remove exactly owned resources.
|
||
|
||
Keep the goal open until `./scripts/p3-acceptance.sh integration --keep` succeeds once from clean state with no automatic retry and its report has been verified. Focused tests are progress evidence, not completion of the process goal.
|
||
|
||
### Controlled topology and report contract
|
||
|
||
The P3 process test extends P2's private fixture topology: a bare local Git remote with revisions A/B/C, installation descriptor and fixture secret files, the real dedicated `workspace-maintenance` image/job, an optionally running or deliberately unavailable `core`, real Qdrant, real internal Ollama, and the P2 controlled REST-DWH fixture. No production credential, registry, volume, or network is permitted. The process owns a unique Compose project and `.artifacts/p3-effective-config/<run-id>/ownership.json`.
|
||
|
||
`report.json` and `report.md` must record the clean source commit/tree, built image digest, fixture Git commits, workspace ID/revisions, safe canonical binding hashes, owner schema/digests, DWH ACTIVE generation and file digests, commit+binding readiness paths and binding-qualified snapshot identities, Qdrant counts by safe kind/revision, Memory registry/projection counts, command event results, secret-scan result, cleanup inventory/result, and overall PASS/FAIL. They must never contain endpoint credentials, fixture secret values, signed URLs, rendered config, raw child stderr, or unbounded logs. `--keep` retains the owned run for review; `cleanup --run <run-id>` later removes only resources in its ownership manifest while retaining the sanitized reports and a final cleanup report.
|
||
|
||
There are no unavoidable human steps inside automation. Manual acceptance is a separate clean environment after automated PASS.
|
||
|
||
---
|
||
|
||
### Task 1: Freeze P2 schema-v1 ownership and baseline the dependency
|
||
|
||
**Files:**
|
||
- Create: `harness/tests/fixtures/dwh-owner-v1/README.md`
|
||
- Create: `harness/tests/fixtures/dwh-owner-v1/OWNER.json`
|
||
- Create: `harness/tests/fixtures/dwh-owner-v1/ACTIVE`
|
||
- Create: `harness/tests/fixtures/dwh-owner-v1/generations/11111111111111111111111111111111/{physical.yaml,analytics_lsh.pkl,analytics_minhashes.pkl,analytics_meta.json,generation-manifest.json}`
|
||
- Modify: `harness/tests/test_dwh_preprocess_job.py`
|
||
|
||
**Step 1: Verify the P2 checkpoint before editing.**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
git status --short
|
||
./scripts/p2-acceptance.sh integration --keep
|
||
(cd backend && npm run build:native && npx vitest run \
|
||
test/workspace-fs-at-native.test.ts test/workspace-lock-root-lease.test.ts \
|
||
test/workspace-session-readers-lock.test.ts \
|
||
test/workspace-registry-addressed-publication.test.ts \
|
||
test/workspace-registry-addressed-process.test.ts test/routes-workspaces.test.ts \
|
||
test/app.test.ts && npx tsc --noEmit -p .)
|
||
cd harness && .venv/bin/pytest -q tests/test_dwh_preprocess_job.py
|
||
```
|
||
|
||
Expected: clean tracked tree before the retained P2 report is created; P2 report says `automated integration: PASS`; the repo-owned `workspace-fs-at` addon/root lease and automatic bootstrap recovery route/app/process suites pass with TypeScript green; and the existing DWH suite passes. If P2 has not been manually accepted, stop for the P2 checkpoint.
|
||
|
||
**Step 2: Create the fixture through the unmodified P2 operational writer.**
|
||
|
||
Use the P2 controlled config to run `tht preprocess dwh`, copy only the bounded owner/ACTIVE/generation files above, replace source-specific values with deterministic fixture values, recompute all declared SHA-256 values, and document the exact generation command in `README.md`. Do not hand-wave a structurally plausible owner.
|
||
|
||
**Step 3: Write the RED compatibility test.**
|
||
|
||
Add `test_schema_v1_fixture_is_a_valid_p2_owner_and_remains_readable` and `test_p2_writer_still_emits_schema_v1_before_explicit_migration` to `test_dwh_preprocess_job.py`. The first loads the fixture with the current reader and verifies all files/ACTIVE; the second asserts the P2 writer still emits `schema_version == 1` at this checkpoint.
|
||
|
||
**Step 4: Run the focused tests.**
|
||
|
||
```bash
|
||
cd harness && .venv/bin/pytest -q \
|
||
tests/test_dwh_preprocess_job.py::test_schema_v1_fixture_is_a_valid_p2_owner_and_remains_readable \
|
||
tests/test_dwh_preprocess_job.py::test_p2_writer_still_emits_schema_v1_before_explicit_migration
|
||
```
|
||
|
||
Expected: PASS. This is a characterization task, not production behavior change.
|
||
|
||
**Step 5: Commit the fixture checkpoint.**
|
||
|
||
```bash
|
||
git add harness/tests/fixtures/dwh-owner-v1 harness/tests/test_dwh_preprocess_job.py
|
||
git commit -m "test: preserve P2 DWH owner compatibility fixture"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 2: Add the harness-owned versioned effective-DWH canonicalizer
|
||
|
||
**Files:**
|
||
- Create: `harness/tht/effective_dwh.py`
|
||
- Create: `harness/tests/test_effective_dwh_binding.py`
|
||
- Modify: `harness/tht/config.py`
|
||
- Modify: `harness/tht/cli/config_cmd.py`
|
||
- Modify: `harness/tht/jobs/dwh_pipeline.py`
|
||
|
||
**Step 1: Write RED tests for the exact canonical contract.**
|
||
|
||
Create tests named:
|
||
|
||
- `test_binding_is_canonical_versioned_secret_free_and_source_stable`
|
||
- `test_content_revision_temp_path_session_and_semantic_changes_do_not_change_binding`
|
||
- `test_each_dwh_output_affecting_field_changes_binding`
|
||
- `test_missing_registry_identity_cannot_claim_a_v2_cache`
|
||
- `test_effective_dwh_json_is_pristine_and_safe`
|
||
|
||
Use parameterized mutations for transport, direct host/port/user, REST canonical base URL, database, schema, examples, eligibility, every LSH field, language if it affects generated descriptions, and `artifact_layout_version`. Include obvious fixture passwords/API keys and assert neither secret nor secret-file path appears in canonical JSON, CLI output, exceptions, or `repr`.
|
||
|
||
Run:
|
||
|
||
```bash
|
||
cd harness && .venv/bin/pytest -q tests/test_effective_dwh_binding.py
|
||
```
|
||
|
||
Expected: RED because `tht.effective_dwh` and `tht config effective-dwh` do not exist.
|
||
|
||
**Step 2: Implement only the canonical contract.**
|
||
|
||
In `effective_dwh.py`, define immutable models/constants and functions with these public names:
|
||
|
||
```python
|
||
EFFECTIVE_DWH_SCHEMA_VERSION = 2
|
||
CANONICALIZER_VERSION = "effective-dwh-v1"
|
||
ARTIFACT_LAYOUT_VERSION = "dwh-cache-v1"
|
||
|
||
class EffectiveDwhBinding(BaseModel):
|
||
schema_version: Literal[2]
|
||
workspace_id: str
|
||
logical_source_identity: str
|
||
canonicalizer_version: Literal["effective-dwh-v1"]
|
||
effective_config_sha256: str
|
||
input_fingerprint: str
|
||
|
||
|
||
def canonical_effective_dwh_config(cfg: Config) -> dict[str, object]: ...
|
||
def effective_dwh_binding(cfg: Config) -> EffectiveDwhBinding: ...
|
||
def effective_dwh_binding_json(cfg: Config) -> str: ... # sort_keys, compact, trailing newline
|
||
def effective_dwh_cache_key(binding: EffectiveDwhBinding) -> str: ... # 64 lowercase hex
|
||
def effective_dwh_cache_root(cfg: Config) -> Path: ...
|
||
def legacy_schema_v1_binding(cfg: Config) -> dict[str, str]: ...
|
||
```
|
||
|
||
`canonical_effective_dwh_config` must build an explicit allowlist, normalize URLs/host case/default ports without DNS/network access, and reject query/userinfo/fragments. Never start with `cfg.model_dump()` and subtract fields. `effective_dwh_cache_root` appends the binding key to the configured cache **base**; it must reject missing registry identity and any symlink/path escape.
|
||
|
||
Add `Config._logical_source_identity` if needed, but keep `_config_source` as the compatibility source used by `legacy_schema_v1_binding`. Registry configs set both to `workspace://<id>`; legacy file configs retain their resolved filename only for schema-v1 compatibility.
|
||
|
||
Add `tht config effective-dwh --json -c <config>` in `config_cmd.py`. It emits only the safe binding, cache key, and layout version.
|
||
|
||
Replace `dwh_pipeline.config_dwh_binding` internals with a compatibility wrapper that delegates to `effective_dwh_binding` only after the new owner path is enabled in Task 4; until then it must keep schema-v1 behavior so the characterization test remains green.
|
||
|
||
**Step 3: Run RED/GREEN tests.**
|
||
|
||
```bash
|
||
cd harness && .venv/bin/pytest -q tests/test_effective_dwh_binding.py tests/test_config_resources.py
|
||
```
|
||
|
||
Expected: PASS; exact stdout from the JSON test parses as one object and contains no secret fixture.
|
||
|
||
**Step 4: Lint the touched Python.**
|
||
|
||
```bash
|
||
cd harness && .venv/bin/ruff check tht/effective_dwh.py tht/config.py tht/cli/config_cmd.py tht/jobs/dwh_pipeline.py tests/test_effective_dwh_binding.py
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
**Step 5: Commit.**
|
||
|
||
```bash
|
||
git add harness/tht/effective_dwh.py harness/tht/config.py harness/tht/cli/config_cmd.py \
|
||
harness/tht/jobs/dwh_pipeline.py harness/tests/test_effective_dwh_binding.py
|
||
git commit -m "feat: define canonical effective DWH binding"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 3: Add global layout and immutable commit+binding readiness models without activation
|
||
|
||
**Files:**
|
||
- Modify: `backend/src/workspaces/runtime-renderer.ts`
|
||
- Create: `backend/src/workspaces/revision-layout.ts`
|
||
- Modify: `backend/test/workspace-runtime-renderer.test.ts`
|
||
- Create: `backend/test/workspace-revision-layout.test.ts`
|
||
- Modify: `harness/tht/config.py`
|
||
- Modify: `harness/tht/paths.py`
|
||
- Create: `harness/tht/dwh_snapshot.py`
|
||
- Create: `harness/tests/test_dwh_snapshot.py`
|
||
- Modify: `harness/tests/test_config_resources.py`
|
||
- Modify: `harness/tests/test_portable_paths.py`
|
||
|
||
This task is deliberately additive. Session and ordinary maintenance continue to render the exact P2
|
||
workspace-global roots. No marker is written and no runtime consumer changes behavior here.
|
||
|
||
**Step 1: Write RED model, secure-reader, and resolver tests.**
|
||
|
||
Add exact future layout support for:
|
||
|
||
```text
|
||
/data/sessions/<workspace-id>/
|
||
preprocessing/layout-version.json # workspace-global version only
|
||
preprocessing/dwh-cache/<binding-key>/
|
||
revisions/<40-hex>/readiness/<binding-key>/READY.json
|
||
revisions/<40-hex>/dwh-snapshots/<binding-key>/artifacts/ACTIVE
|
||
revisions/<40-hex>/dwh-snapshots/<binding-key>/artifacts/generations/<generation>/physical.yaml
|
||
revisions/<40-hex>/dwh-snapshots/<binding-key>/artifacts/generations/<generation>/snapshot-manifest.json
|
||
revisions/<40-hex>/dwh-snapshots/<binding-key>/indexes/generations/<generation>/analytics_lsh.pkl
|
||
revisions/<40-hex>/dwh-snapshots/<binding-key>/indexes/generations/<generation>/analytics_minhashes.pkl
|
||
revisions/<40-hex>/dwh-snapshots/<binding-key>/indexes/generations/<generation>/analytics_meta.json
|
||
revisions/<40-hex>/corpus/
|
||
memory/
|
||
sessions/
|
||
```
|
||
|
||
Freeze `LayoutVersionMarkerV1` as an exact-key schema containing only `schemaVersion: 1`,
|
||
`layoutVersion: "revision-layout-v1"`, workspace ID, and enabled UTC time. It never contains an
|
||
active revision or binding. Freeze `RevisionReadyManifestV1` as an exact-key schema containing
|
||
workspace ID, 40-hex revision, descriptor blob, complete effective DWH binding plus its binding SHA
|
||
and 64-hex cache key, the binding-qualified DWH snapshot generation+manifest digest, Memory registry
|
||
digest/status, semantic inventory/replacement/deletion digests and status, preparing outer run ID,
|
||
and ready UTC time. Its canonical bytes are published exclusively at
|
||
`revisions/<revision>/readiness/<cache-key>/READY.json`; an existing file must byte-match after strict
|
||
reverification and is never replaced. Thus readiness is an immutable generation for one exact
|
||
`(commit, effective binding)` pair, and the sole session admission proof is the manifest selected by
|
||
the session's harness-reported cache key. The secure readiness-directory reader additionally enforces
|
||
at most one READY key per revision: a second valid key, an unsafe entry, or ambiguous directory state
|
||
fails closed. A same-commit installation binding change cannot create a sibling READY or snapshot;
|
||
activation returns `effective_config_mismatch` before mutation and the changed binding must be paired
|
||
with a new content commit/revision.
|
||
|
||
Freeze these shared synchronous TypeScript exports in
|
||
`backend/src/workspaces/revision-layout.ts`:
|
||
|
||
```ts
|
||
import type {
|
||
BorrowedVerifiedWorkspaceLockRootLease,
|
||
CanonicalWorkspaceId,
|
||
Revision40,
|
||
Sha256Hex,
|
||
} from "./workspace-lock-root-lease.js";
|
||
export type WorkspaceLayout = "p2-global" | "revision-layout-v1";
|
||
export type RevisionLayoutState =
|
||
| { layout: "p2-global"; ready: null }
|
||
| { layout: "revision-layout-v1"; ready: RevisionReadyManifestV1 | null };
|
||
export function futureWorkspaceLayoutPaths(
|
||
rootLease: BorrowedVerifiedWorkspaceLockRootLease,
|
||
workspaceId: CanonicalWorkspaceId,
|
||
workspaceRevision: Revision40,
|
||
): RuntimePaths;
|
||
export function bindingQualifiedWorkspaceLayoutPaths(
|
||
futurePaths: RuntimePaths, effectiveDwhCacheKey: Sha256Hex,
|
||
): RuntimePaths;
|
||
export function workspaceRuntimePaths(
|
||
rootLease: BorrowedVerifiedWorkspaceLockRootLease,
|
||
workspaceId: CanonicalWorkspaceId,
|
||
workspaceRevision: Revision40,
|
||
state: RevisionLayoutState,
|
||
): RuntimePaths;
|
||
export function readRevisionLayoutState(
|
||
rootLease: BorrowedVerifiedWorkspaceLockRootLease,
|
||
expected: {
|
||
workspaceId: CanonicalWorkspaceId; workspaceRevision: Revision40; descriptorBlob: Revision40;
|
||
effectiveDwhCacheKey: Sha256Hex;
|
||
},
|
||
): RevisionLayoutState;
|
||
```
|
||
|
||
`readRevisionLayoutState` is deliberately synchronous inside an already-active P2 root-lease borrow,
|
||
so `WorkspaceRuntimeConfigLeaseFactory.acquireSession` and `.acquireMaintenance` retain their released
|
||
signatures and no raw root string crosses the handoff. It is called only after the fixed read-only
|
||
harness binding probe has returned the exact cache key. The module uses P2's installation-bound
|
||
root-relative safe reader; it never derives/reopens an ambient path. It opens every component and leaf
|
||
with `O_NOFOLLOW`, `fstat`, owner/mode/nlink/regular-file and post-read identity checks, bounds both JSON
|
||
files, and closes every relative child FD while leaving the borrowed root owned by its caller. An absent global marker returns `p2-global`. A present invalid
|
||
marker fails closed. With a valid global marker, absence of the exact
|
||
`readiness/<expected-cache-key>/READY.json` returns `{layout: "revision-layout-v1", ready: null}` when
|
||
the exact key is absent; it never selects newest, the sole other-key entry, or a revision-only READY,
|
||
so session admission maps the absence to `migration_required`. The activation/prepare guard separately
|
||
reports `effective_config_mismatch` when that sole other-key READY exists. Multiple READY keys, an
|
||
unsafe entry, or a present invalid, descriptor-, revision-, or binding-mismatched selected READY fail
|
||
closed as corrupted state. No asynchronous wrapper, `readFile`, or symlink-following convenience API
|
||
is permitted.
|
||
|
||
`futureWorkspaceLayoutPaths` is the trusted migration-destination resolver. It validates the
|
||
already verified workspace root, workspace ID, and 40-hex revision, then derives binding-independent
|
||
future roots without reading the global marker, READY, current rendered config, or any P2
|
||
compatibility fallback. `bindingQualifiedWorkspaceLayoutPaths` validates the harness-provided 64-hex
|
||
cache key and derives only the exact cache, DWH snapshot, and READY generation beneath those trusted
|
||
roots. Neither accepts a destination path from argv/stdin. `workspaceRuntimePaths` may return P2 paths
|
||
only when the global marker is absent; when layout v1 is enabled it uses the selected binding-qualified
|
||
resolver and never falls back because READY is absent.
|
||
|
||
In `harness/tht/dwh_snapshot.py`, freeze:
|
||
|
||
```python
|
||
@dataclass(frozen=True)
|
||
class DwhArtifactSnapshot:
|
||
generation_id: str
|
||
physical: Path
|
||
analytics_lsh: Path
|
||
analytics_minhashes: Path
|
||
analytics_meta: Path
|
||
manifest_sha256: str
|
||
|
||
|
||
def resolve_revision_dwh_snapshot(cfg: Config) -> DwhArtifactSnapshot: ...
|
||
def resolve_effective_dwh_inputs(cfg: Config) -> DwhArtifactSnapshot | None: ...
|
||
```
|
||
|
||
`resolve_revision_dwh_snapshot` securely validates the exact `ACTIVE`, generation directory,
|
||
manifest, modes/link counts, revision identity, and every declared digest. `resolve_effective_dwh_inputs`
|
||
is the only compatibility branch: when `paths.dwh_snapshot` is absent it returns `None`, telling
|
||
existing consumers to use unchanged P2 paths. No caller constructs a snapshot path itself.
|
||
|
||
Run:
|
||
|
||
```bash
|
||
(cd backend && npx vitest run \
|
||
test/workspace-runtime-renderer.test.ts \
|
||
test/workspace-revision-layout.test.ts)
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_config_resources.py tests/test_portable_paths.py tests/test_dwh_snapshot.py)
|
||
```
|
||
|
||
Expected: RED on missing models/resolver, while every existing P2 rendering assertion stays green.
|
||
|
||
**Step 2: Implement additive models only.**
|
||
|
||
Extend `RuntimePaths`/`PathsConfig` with optional `memory`, `corpus`, `dwh_cache`, and
|
||
`dwh_snapshot`. Keep P2 fallback semantics exactly for ordinary active rendering. Implement the
|
||
pure trusted future resolver and synchronous secure state reader, but do not call either from
|
||
`ThtRunner` or `WorkspaceRuntimeConfigLeaseFactory` yet.
|
||
|
||
**Step 3: Run GREEN tests and the P2 regression boundary.**
|
||
|
||
```bash
|
||
(cd backend && npx vitest run \
|
||
test/workspace-runtime-renderer.test.ts \
|
||
test/workspace-revision-layout.test.ts \
|
||
test/workspace-runtime-config-lease.test.ts \
|
||
test/workspace-preprocessing-service.test.ts \
|
||
test/workspace-maintenance.test.ts && npx tsc --noEmit -p .)
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_config_resources.py tests/test_portable_paths.py tests/test_dwh_snapshot.py \
|
||
tests/test_dwh_preprocess_job.py tests/test_lsh_job_resume.py)
|
||
```
|
||
|
||
Expected: PASS; session and maintenance still render byte-identical P2 YAML and equal schema-v1
|
||
`config_dwh_binding()`.
|
||
|
||
**Step 4: Commit.**
|
||
|
||
```bash
|
||
git add backend/src/workspaces/runtime-renderer.ts backend/src/workspaces/revision-layout.ts \
|
||
backend/test/workspace-runtime-renderer.test.ts backend/test/workspace-revision-layout.test.ts \
|
||
harness/tht/config.py harness/tht/paths.py harness/tht/dwh_snapshot.py \
|
||
harness/tests/test_config_resources.py harness/tests/test_portable_paths.py \
|
||
harness/tests/test_dwh_snapshot.py
|
||
git commit -m "feat: add inactive revision layout models"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 4: Add schema-v2 OWNER reading/writing without weakening schema-v1 validation
|
||
|
||
**Files:**
|
||
- Create: `harness/tht/dwh_owner.py`
|
||
- Create: `harness/tests/test_dwh_owner_v2.py`
|
||
- Modify: `harness/tht/jobs/dwh_pipeline.py`
|
||
- Modify: `harness/tests/test_dwh_preprocess_job.py`
|
||
|
||
**Step 1: Write RED owner tests.**
|
||
|
||
Cover new empty binding-key cache schema 2; complete `EffectiveDwhBinding`; canonical
|
||
`binding_sha256`; strict exact keys/modes/UID/nlink; malformed ACTIVE/manifest; v1 compatibility
|
||
read-only; content-only revision reuse; and mismatch refusal without sibling-key fallback.
|
||
|
||
```bash
|
||
(cd harness && .venv/bin/pytest -q tests/test_dwh_owner_v2.py tests/test_dwh_preprocess_job.py)
|
||
```
|
||
|
||
Expected: RED because v2 owner code does not exist.
|
||
|
||
**Step 2: Implement strict dual readers and a v2-only writer.**
|
||
|
||
```python
|
||
class OwnerSchemaV1(BaseModel): ...
|
||
class OwnerSchemaV2(BaseModel): ...
|
||
|
||
|
||
def read_owner_at(root_fd: int) -> OwnerSchemaV1 | OwnerSchemaV2: ...
|
||
def validate_owner_at(root_fd: int, expected: EffectiveDwhBinding) -> OwnerSchemaV2: ...
|
||
def write_owner_v2_at(root_fd: int, binding: EffectiveDwhBinding) -> None: ...
|
||
```
|
||
|
||
Select only by literal `schema_version`; never reinterpret v1 as v2. New v2 cache generations bind
|
||
the complete v2 owner digest. Existing P2 runtime roots remain selected until Task 9 activation.
|
||
|
||
**Step 3: Run focused/regression suites and lint.**
|
||
|
||
```bash
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_dwh_owner_v2.py tests/test_dwh_preprocess_job.py tests/test_search_pack.py && \
|
||
.venv/bin/ruff check tht/dwh_owner.py tht/jobs/dwh_pipeline.py \
|
||
tests/test_dwh_owner_v2.py tests/test_dwh_preprocess_job.py)
|
||
```
|
||
|
||
Expected: PASS, including the untouched valid schema-v1 fixture.
|
||
|
||
**Step 4: Commit.**
|
||
|
||
```bash
|
||
git add harness/tht/dwh_owner.py harness/tht/jobs/dwh_pipeline.py \
|
||
harness/tests/test_dwh_owner_v2.py harness/tests/test_dwh_preprocess_job.py
|
||
git commit -m "feat: version DWH cache ownership"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 5: Implement verified schema-v1 migration, prepare-mode DWH builds, and immutable binding snapshots
|
||
|
||
**Files:**
|
||
- Create: `harness/tht/dwh_migration.py`
|
||
- Create: `harness/tests/test_dwh_owner_migration.py`
|
||
- Modify: `harness/tht/dwh_snapshot.py`
|
||
- Modify: `harness/tht/effective_dwh.py`
|
||
- Modify: `harness/tht/jobs/dwh_pipeline.py`
|
||
- Modify: `harness/tht/cli/config_cmd.py`
|
||
- Modify: `harness/tht/cli/preprocess_cmd.py`
|
||
|
||
**Step 1: Write RED migration/lock tests.**
|
||
|
||
Cover verified v1 copy with source preservation; prepare-mode introspection/LSH build when the exact
|
||
binding-key cache is absent or the v1 binding does not match; v2-only owner/manifests; idempotent full
|
||
reverification; mismatch/corruption/collision/partial destination refusal; copy/build/fsync/rename
|
||
fault injection; and exact binding-qualified snapshot layout/digests. Assert the migration source is
|
||
exactly the legacy
|
||
`<verified-workspace-root>/.tht-dwh` and destinations are exactly the cache/snapshot roots returned by
|
||
`futureWorkspaceLayoutPaths`, even while active P2 rendering still points at compatibility roots.
|
||
Reject a config-derived fallback destination and every caller-supplied destination. Require P2's
|
||
actual inherited writer FD 3 and retained-root FD 4: missing/closed descriptors, another writer/root inode, a cross-root pair, or a forged environment marker
|
||
without both descriptors, and ordinary direct internal CLI all fail `preprocessing_conflict`; SIGKILL releases
|
||
the parent Node lock and leaves no readable partial. Add the complete changed-binding boundary test: READY for revision B/binding X exists; installation
|
||
binding changes to Y; session admission refuses B/X and activation fails `effective_config_mismatch`
|
||
before DWH access, cache/snapshot creation, semantic calls, or READY publication. After a content-only
|
||
commit C is published and pulled, prepare mode builds and verifies Y's new v2 owner/cache and C/Y
|
||
binding-qualified snapshot; immutable B/X READY and snapshot remain byte-identical; C/Y READY is later
|
||
published. While installation binding Y is selected, B admission remains intentionally refused although B/X bytes are immutable. The test then explicitly restores installation binding X and admits B/X, restores Y, and admits C/Y; revision pinning alone is never claimed to restore a historical installation binding.
|
||
|
||
```bash
|
||
(cd harness && .venv/bin/pytest -q tests/test_dwh_owner_migration.py)
|
||
```
|
||
|
||
Expected: RED.
|
||
|
||
**Step 2: Implement migration with the fixed kernel-lock capability.**
|
||
|
||
```python
|
||
@dataclass(frozen=True)
|
||
class DwhMigrationReport: ...
|
||
|
||
|
||
def migrate_schema_v1_cache(cfg: Config) -> DwhMigrationReport: ...
|
||
def prepare_effective_dwh_cache(cfg: Config) -> DwhMigrationReport: ...
|
||
def materialize_revision_dwh_snapshot(cfg: Config) -> DwhArtifactSnapshot: ...
|
||
```
|
||
|
||
Both functions begin with P2's `require_workspace_writer_lock(cfg)`, which validates the retained root directory, opens the canonical lock relative to FD 4, proves that inode is FD 3, and calls `fcntl.flock(3, LOCK_EX|LOCK_NB)`. There is no callback, boolean, path, independently reopened root,
|
||
or environment-marker seam.
|
||
|
||
Derive the sole P2 migration source as `<verified-workspace-root>/.tht-dwh`; never accept a caller
|
||
path. The backend first performs the fixed read-only effective-binding probe, then acquires
|
||
maintenance with `layoutIntent: "prepare-revision-layout-v1"` and binds the returned key through
|
||
`bindingQualifiedWorkspaceLayoutPaths`. Python re-derives and asserts the configured cache and
|
||
snapshot roots equal `/preprocessing/dwh-cache/<binding-key>` and
|
||
`/revisions/<revision>/dwh-snapshots/<binding-key>/...`, never `effective_dwh_cache_root` under an
|
||
active P2 fallback. If the exact legacy schema-v1 binding matches, verify every generation/digest and
|
||
ACTIVE, copy to a temporary sibling of that trusted binding-key cache, write schema-v2
|
||
owner/manifests, re-read through strict v2 code, rename, and fsync. Preserve source bytes.
|
||
|
||
If no exact v2 cache exists and the legacy binding does not match, activation must call
|
||
`prepare_effective_dwh_cache`: run the existing DWH introspection and LSH pipeline in explicit
|
||
`prepare-revision-layout-v1` mode, writing only a temporary sibling of that binding-key cache; publish
|
||
`OWNER.json` schema v2, generation manifest and ACTIVE in the existing safe order; strictly re-read
|
||
the complete owner/binding/digests; then rename/fsync the cache. Prepare mode is allowed for an unready
|
||
commit+binding pair, requires the exact FD 3/FD 4 pair, cannot read or fall back to any old cache, and on failure removes
|
||
or quarantines only its unpublished temporary sibling while leaving legacy and other binding caches
|
||
unchanged. Existing exact destinations must fully reverify and return `already_prepared`; different
|
||
bytes at the same binding key fail closed.
|
||
|
||
`materialize_revision_dwh_snapshot` writes the exact Task 3 binding-qualified generation layout. It
|
||
copies only the validated cache ACTIVE generation, writes and revalidates
|
||
`snapshot-manifest.json`, fsyncs, renames, then publishes that binding snapshot's `ACTIVE` last. It
|
||
never symlinks the revision to the cache and never overwrites another binding snapshot.
|
||
|
||
Add fixed internal `tht config migrate-dwh-cache --json -c <config>`, `tht preprocess dwh
|
||
--prepare-revision-layout-v1 --json -c <config>`, and `tht config materialize-dwh-snapshot --json -c
|
||
<config>`. Direct invocation of any mutating operation without the actual inherited writer FD 3 and retained-root FD 4 fails
|
||
before source inspection or DWH access.
|
||
|
||
**Step 3: Run focused tests and the P2 regression boundary.**
|
||
|
||
```bash
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_dwh_owner_migration.py tests/test_dwh_owner_v2.py \
|
||
tests/test_dwh_preprocess_job.py tests/test_search_pack.py tests/test_dwh_snapshot.py && \
|
||
.venv/bin/ruff check tht/dwh_migration.py tht/dwh_snapshot.py tht/effective_dwh.py \
|
||
tht/jobs/dwh_pipeline.py tht/cli/config_cmd.py tht/cli/preprocess_cmd.py \
|
||
tests/test_dwh_owner_migration.py)
|
||
```
|
||
|
||
Expected: PASS; v1 source and P2 active paths remain byte-identical.
|
||
|
||
**Step 4: Commit.**
|
||
|
||
```bash
|
||
git add harness/tht/dwh_migration.py harness/tht/dwh_snapshot.py harness/tht/effective_dwh.py \
|
||
harness/tht/jobs/dwh_pipeline.py harness/tht/cli/config_cmd.py \
|
||
harness/tht/cli/preprocess_cmd.py harness/tests/test_dwh_owner_migration.py
|
||
git commit -m "feat: migrate legacy DWH caches explicitly"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 6: Add revision semantic identities, inventory/readiness preparation, and activation-only publish/delete
|
||
|
||
**Files:**
|
||
- Modify: `harness/tht/vectorstore/records.py`
|
||
- Modify: `harness/tht/adapters/vector/qdrant.py`
|
||
- Modify: `harness/tht/ports/vector.py`
|
||
- Modify: `harness/tht/adapters/factory.py`
|
||
- Modify: `harness/tht/search/evidence.py`
|
||
- Create: `harness/tht/semantic_migration.py`
|
||
- Modify: `harness/tht/cli/vector_cmd.py`
|
||
- Create: `harness/tests/test_semantic_revision_migration.py`
|
||
- Modify: `harness/tests/test_qdrant_vector_store.py`
|
||
- Modify: `harness/tests/test_semantic_kind_isolation.py`
|
||
- Modify: `harness/tests/test_qdrant_cli_commands.py`
|
||
- Modify: `harness/tests/test_corpus_pipeline.py`
|
||
- Modify: `harness/tests/test_memory_save_one.py`
|
||
- Modify: `harness/tests/test_solved_question.py`
|
||
|
||
**Step 1: Write RED scoped identity/filter tests.**
|
||
|
||
Test every operation: schema/Evidence IDs and filters include exact workspace+40-hex revision;
|
||
Memory/solved remain workspace-only; mixed searches partition and merge deterministically; caller
|
||
namespace conflicts fail before network; and revisionless points report `migration_required`.
|
||
Runtime selection stays in P2 legacy mode until Task 9 publishes the global layout version and the
|
||
exact A+binding READY. The public non-activating semantic command may only inventory legacy points
|
||
and prove canonical schema/Evidence inputs are rebuildable; it must not call Qdrant upsert or delete.
|
||
Only Task 9 activation may write/verify revision-scoped targets. After global enablement, an unready
|
||
revision/binding never falls back to P2.
|
||
|
||
**Step 2: Write RED publish-before-delete/resume tests.**
|
||
|
||
Freeze semantic state phases in the P2 `PreprocessingStateStore` record:
|
||
`legacy_inventory_persisted`, `semantic_sources_ready`, `replacement_published`,
|
||
`replacement_verified`, `legacy_delete_complete`. The non-activating command can reach only the first
|
||
two; the last three are activation-only. The inventory contains each exact legacy schema/Evidence point ID and the
|
||
SHA-256 of canonical bounded payload bytes, plus the inventory digest and source artifact/corpus
|
||
digests.
|
||
|
||
Tests must prove:
|
||
|
||
- after every successful non-activation command, a newly admitted legacy-mode test session observes
|
||
no revision-scoped replacement points and Qdrant's upsert/delete call counts remain exactly zero;
|
||
- only activation can invoke schema/Evidence replacement upserts; readback verification completes
|
||
before any delete call;
|
||
- embed/upsert/verification failure deletes zero legacy IDs;
|
||
- unavailable Evidence canonical source returns `migration_required` and deletes zero IDs;
|
||
- resume after replacement verification does not re-embed verified points;
|
||
- final deletion addresses only inventory IDs and first re-reads each legacy payload digest;
|
||
- changed/missing legacy payload stops with conflict and does not broaden deletion;
|
||
- crash during deletion resumes the exact remaining set;
|
||
- Memory/solved and other workspaces/revisions are never listed or deleted.
|
||
|
||
```bash
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_qdrant_vector_store.py tests/test_semantic_kind_isolation.py \
|
||
tests/test_qdrant_cli_commands.py tests/test_corpus_pipeline.py \
|
||
tests/test_memory_save_one.py tests/test_solved_question.py \
|
||
tests/test_semantic_revision_migration.py)
|
||
```
|
||
|
||
Expected: RED.
|
||
|
||
**Step 3: Implement the exact contracts.**
|
||
|
||
```python
|
||
def point_id(
|
||
workspace_id: str,
|
||
kind: str,
|
||
record_key: str,
|
||
workspace_revision: str | None = None,
|
||
) -> str: ...
|
||
|
||
@dataclass(frozen=True)
|
||
class LegacySemanticPoint:
|
||
point_id: str
|
||
payload_sha256: str
|
||
kind: Literal["schema", "evidence"]
|
||
|
||
|
||
def inventory_legacy_semantic_points(cfg: Config) -> SemanticLegacyInventory: ...
|
||
def verify_semantic_rebuild_readiness(cfg: Config, inventory: SemanticLegacyInventory) -> SemanticReadinessReport: ...
|
||
def publish_semantic_replacements(cfg: Config, inventory: SemanticLegacyInventory) -> SemanticMigrationReport: ...
|
||
def verify_semantic_replacements(cfg: Config, inventory: SemanticLegacyInventory) -> SemanticMigrationReport: ...
|
||
def delete_confirmed_legacy_semantic_points(cfg: Config, inventory: SemanticLegacyInventory) -> SemanticMigrationReport: ...
|
||
```
|
||
|
||
Use UUIDv5 input `thothii:<workspace>:<revision>:<semantic-kind>:<record-key>` for schema/Evidence
|
||
and the existing workspace-global form for Memory. Apply identity filters to search, hashes, scroll,
|
||
list, and delete. Preparation begins by verifying the inherited writer FD 3 and retained-root FD 4 pair. The fixed Python `inventory`
|
||
phase returns a bounded exact legacy inventory; `WorkspacePreprocessingService.execute` persists
|
||
those exact bytes and digest through `PreprocessingStateStore`. The fixed `readiness` phase opens and
|
||
hashes the verified DWH snapshot and canonical Evidence corpus/source, proves they are complete and
|
||
rebuildable, and returns only bounded counts/digests. It has no vector-store writer dependency and
|
||
must be proven incapable of Qdrant upsert/delete. A successful released `semantic-revision` command
|
||
stops after persisting `semantic_sources_ready`, durably completes and owner-clears quiescence while
|
||
still holding the exclusive reader gate, then admits a test legacy reader that sees exactly the
|
||
pre-command legacy set.
|
||
|
||
Only activation may invoke fixed `publish` and `verify` phases. With durable quiescence installed
|
||
and the exclusive reader gate held, they reverify the persisted inventory/readiness bytes, rebuild current-revision schema from the binding-qualified DWH
|
||
snapshot and Evidence from the canonical corpus/source, then upsert/read back every replacement
|
||
identity/hash. Only after the backend durably records `replacement_verified` and publishes/verifies
|
||
the initial global reader marker may fixed `delete-confirmed` receive the exact persisted inventory
|
||
over bounded child stdin, re-read each legacy digest, and delete those IDs. Python never imports or
|
||
impersonates the TypeScript store, and no phase accepts an arbitrary state path. No phase creates or
|
||
deletes the collection.
|
||
|
||
**Step 4: Run tests and lint.**
|
||
|
||
```bash
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_qdrant_vector_store.py tests/test_semantic_kind_isolation.py \
|
||
tests/test_qdrant_cli_commands.py tests/test_corpus_pipeline.py \
|
||
tests/test_memory_save_one.py tests/test_solved_question.py \
|
||
tests/test_semantic_revision_migration.py && \
|
||
.venv/bin/ruff check tht/vectorstore/records.py tht/adapters/vector/qdrant.py \
|
||
tht/ports/vector.py tht/adapters/factory.py tht/search/evidence.py \
|
||
tht/semantic_migration.py tht/cli/vector_cmd.py \
|
||
tests/test_semantic_revision_migration.py)
|
||
```
|
||
|
||
Expected: PASS; failure-before-delete assertions observe zero deletion calls.
|
||
|
||
**Step 5: Commit.**
|
||
|
||
```bash
|
||
git add harness/tht/vectorstore/records.py harness/tht/adapters/vector/qdrant.py \
|
||
harness/tht/ports/vector.py harness/tht/adapters/factory.py harness/tht/search/evidence.py \
|
||
harness/tht/semantic_migration.py harness/tht/cli/vector_cmd.py \
|
||
harness/tests/test_semantic_revision_migration.py \
|
||
harness/tests/test_qdrant_vector_store.py harness/tests/test_semantic_kind_isolation.py \
|
||
harness/tests/test_qdrant_cli_commands.py harness/tests/test_corpus_pipeline.py \
|
||
harness/tests/test_memory_save_one.py harness/tests/test_solved_question.py
|
||
git commit -m "feat: migrate semantic state publish before delete"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 7: Add the explicit Memory root and migrate one canonical registry without activating it
|
||
|
||
**Files:**
|
||
- Create: `harness/tht/memory_migration.py`
|
||
- Create: `harness/tests/test_memory_migration.py`
|
||
- Modify: `harness/tht/cli/memory_cmd.py`
|
||
- Modify: `harness/tht/memory.py`
|
||
- Modify: `harness/tht/solved.py`
|
||
- Modify: `harness/tests/test_memory_promotion.py`
|
||
- Modify: `harness/tests/test_memory_save_one.py`
|
||
- Modify: `harness/tests/test_solved_search_cli.py`
|
||
- Modify: `harness/tests/test_repository_memory_sql_paths.py`
|
||
|
||
**Step 1: Write RED path/migration tests.**
|
||
|
||
Cover all Memory commands; solved global identity; compatibility fallback only when `paths.memory`
|
||
is absent; exact known P2/revision candidates; zero/one/equivalent/conflicting registries; unsafe
|
||
file cases; fault injection; idempotence; source preservation; and rebuild of only Memory/solved
|
||
projections. Assert every known legacy source separately from the exact future destination
|
||
`<verified-workspace-root>/memory` returned by `futureWorkspaceLayoutPaths`; a P2
|
||
`paths.artifacts/memory` fallback must never become the destination. The P2-rendered active config
|
||
still lacks explicit `paths.memory`, so ordinary runtime behavior remains on its legacy root here.
|
||
|
||
```bash
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_memory_migration.py tests/test_memory_promotion.py \
|
||
tests/test_memory_save_one.py tests/test_solved_search_cli.py)
|
||
```
|
||
|
||
Expected: RED.
|
||
|
||
**Step 2: Centralize future and fallback paths.**
|
||
|
||
```python
|
||
def memory_root(cfg: Config) -> Path: ...
|
||
def registry_path(cfg: Config) -> Path: ...
|
||
class MemoryRegistryLock: ...
|
||
```
|
||
|
||
Every command uses these helpers. Explicit `paths.memory` selects the future global root; absence
|
||
selects exactly the P2 fallback. Do not change Memory IDs or merge semantics.
|
||
|
||
**Step 3: Implement migration with the actual writer FD 3 and retained-root FD 4.**
|
||
|
||
```python
|
||
@dataclass(frozen=True)
|
||
class MemoryMigrationReport: ...
|
||
|
||
|
||
def migrate_memory_root(cfg: Config) -> MemoryMigrationReport: ...
|
||
def rebuild_global_memory_projection(cfg: Config) -> dict[str, int]: ...
|
||
```
|
||
|
||
Derive source candidates from the verified workspace root, never caller paths. The backend passes a
|
||
`layoutIntent: "prepare-revision-layout-v1"` maintenance config derived only from
|
||
`futureWorkspaceLayoutPaths`; Python re-derives and asserts that `paths.memory` is exactly the future
|
||
workspace-global `/memory` destination before writing or rebuilding projections. Begin with
|
||
`require_workspace_writer_lock(cfg)`. Validate/canonicalize every `MemoryRecord`; conflicting
|
||
semantic content fails without merge. Publish the future canonical JSONL atomically and reverify,
|
||
then rebuild only global Memory/solved projections from that exact registry. Add fixed internal
|
||
`tht memory migrate-root --json -c <config>`; missing/closed/substituted FD fails.
|
||
|
||
**Step 4: Run tests and lint.**
|
||
|
||
```bash
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_memory_migration.py tests/test_memory_promotion.py tests/test_memory_save_one.py \
|
||
tests/test_solved_search_cli.py tests/test_solved_question.py \
|
||
tests/test_repository_memory_sql_paths.py && \
|
||
.venv/bin/ruff check tht/memory_migration.py tht/memory.py tht/solved.py \
|
||
tht/cli/memory_cmd.py tests/test_memory_migration.py)
|
||
```
|
||
|
||
Expected: PASS; P2 fallback paths remain green.
|
||
|
||
**Step 5: Commit.**
|
||
|
||
```bash
|
||
git add harness/tht/memory_migration.py harness/tht/memory.py harness/tht/solved.py \
|
||
harness/tht/cli/memory_cmd.py harness/tests/test_memory_migration.py \
|
||
harness/tests/test_memory_promotion.py harness/tests/test_memory_save_one.py \
|
||
harness/tests/test_solved_search_cli.py harness/tests/test_repository_memory_sql_paths.py
|
||
git commit -m "feat: migrate workspace-global memory state"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 8: Expose registry pull and durable quiesced migrations through the exact P2 one-shot API
|
||
|
||
**Files:**
|
||
- Reuse unchanged from accepted P2: `backend/src/workspaces/workspace-fs-at.ts`
|
||
- Reuse unchanged from accepted P2: `backend/src/native/workspace-fs-at-binding.d.ts`
|
||
- Re-run unchanged P2 ownership test: `backend/test/workspace-fs-at-native.test.ts`
|
||
- Reuse unchanged from accepted P2: `backend/src/workspaces/workspace-lock-root-lease.ts`
|
||
- Re-run unchanged P2 ownership tests: `backend/test/workspace-lock-root-lease.test.ts`, `backend/test/workspace-session-readers-lock.test.ts`
|
||
- Modify only for the P3 child-request union (not reader locking): `backend/src/workspaces/preprocessing-state.ts`
|
||
- Modify: `backend/src/workspaces/preprocessing-service.ts`
|
||
- Modify: `backend/src/workspace-maintenance.ts`
|
||
- Modify: `backend/src/workspaces/registry.ts`
|
||
- Modify: `backend/src/workspaces/registry-publication.ts`
|
||
- Create: `backend/src/workspaces/registry-pull-job.ts`
|
||
- Create: `backend/test/registry-pull-job-imports.compile.ts`
|
||
- Modify: `backend/test/workspace-registry.test.ts`
|
||
- Modify: `backend/test/workspace-registry-addressed-publication.test.ts`
|
||
- Modify: `backend/test/workspace-registry-addressed-process.test.ts`
|
||
- Modify: `backend/test/fixtures/workspace-lock-root-worker.mjs`
|
||
- Modify: `backend/test/fixtures/workspace-registry-addressed-worker.mjs`
|
||
- Create: `backend/src/workspaces/workspace-reader-lease.ts`
|
||
- Modify: `backend/src/routes/workspaces.ts`
|
||
- Modify: `backend/src/routes/sessions.ts`
|
||
- Modify: `backend/src/app.ts`
|
||
- Modify: `backend/test/app.test.ts`
|
||
- Modify: `backend/src/pi/pi-process-manager.ts`
|
||
- Modify: `backend/test/workspace-preprocessing-state.test.ts`
|
||
- Modify: `backend/test/workspace-preprocessing-service.test.ts`
|
||
- Modify: `backend/test/workspace-maintenance.test.ts`
|
||
- Create: `backend/test/workspace-reader-lease.test.ts`
|
||
- Modify: `backend/test/routes-workspaces.test.ts`
|
||
- Modify: `backend/test/routes-sessions.test.ts`
|
||
- Modify: `backend/test/pi-process-manager.test.ts`
|
||
- Create: `harness/tht/locked_child_stdin.py`
|
||
- Create: `harness/tht/layout_markers.py`
|
||
- Modify: `harness/tht/cli/config_cmd.py`
|
||
- Modify: `harness/tht/cli/vector_cmd.py`
|
||
- Modify: `harness/tht/semantic_migration.py`
|
||
- Create: `harness/tests/test_locked_child_stdin.py`
|
||
- Create: `harness/tests/test_layout_marker_commands.py`
|
||
- Create: `harness/tests/test_p3_internal_cli.py`
|
||
- Modify: `harness/tests/test_semantic_revision_migration.py`
|
||
- Modify: `harness/tests/test_qdrant_cli_commands.py`
|
||
- Modify: `tools/thothctl/internal/workspaceops/operations.go`
|
||
- Modify: `tools/thothctl/internal/workspaceops/operations_test.go`
|
||
- Modify: `tools/thothctl/cmd/thothctl/main.go`
|
||
- Modify: `tools/thothctl/cmd/thothctl/main_test.go`
|
||
- Modify: `deploy/compose.git-https.yaml`
|
||
- Modify: `deploy/compose.git-ssh.yaml`
|
||
- Modify: `scripts/generate-connector-secrets-override.sh`
|
||
- Modify: `scripts/test-preprocess-compose-config.sh`
|
||
|
||
**Step 1: Write RED public-command, exact-P2 registry, route, provisioning, and compile-import tests.**
|
||
|
||
Add these released public commands, with no aliases:
|
||
|
||
```text
|
||
thothctl --installation <abs> workspace registry pull
|
||
--workspace <id> [--resume <32-hex-outer-run-id>] [--json]
|
||
thothctl --installation <abs> workspace migrate dwh-cache
|
||
--workspace <id> [--resume <32-hex-outer-run-id>] [--json]
|
||
thothctl --installation <abs> workspace migrate memory
|
||
--workspace <id> [--resume <32-hex-outer-run-id>] [--json]
|
||
thothctl --installation <abs> workspace migrate semantic-revision
|
||
--workspace <id> --yes [--resume <32-hex-outer-run-id>] [--json]
|
||
```
|
||
|
||
Before a fresh call launches Compose, `thothctl` obtains exactly 16 bytes from `crypto/rand`, formats
|
||
one 32-lowercase-hex outer run ID, builds the canonical request/digest below, and retains that ID in
|
||
every result or error. For these four released pull/migration commands, `--resume` uses exactly the
|
||
caller-supplied ID; no newest-run, digest, marker, or automatic selection exists. P2's separate
|
||
empty-registry inspect/lazy-list/status bootstrap path remains the sole automatic-recovery exception via
|
||
`ensureBootstrapAddressed`. The four ordinary operations reject wrong workspace, operation, active
|
||
revision/descriptor, selected binding, or request digest before mutation. `registry_pull` is different:
|
||
the outer request addresses the P2 job, and all base/target authority comes from the exact durable P2
|
||
`registry_pull` state. An ambiguous Compose exit/timeout/truncated result returns the known ID and exact
|
||
`--resume` instruction; it never allocates a replacement ID.
|
||
|
||
Freeze `RegistryPullCommand` and operation literal `registry_pull`. It alone receives writable registry
|
||
storage plus the selected HTTPS/SSH Git transport capability and calls only
|
||
`WorkspaceRegistry.publishAddressed`. It never acquires the selected workspace writer first. Every
|
||
ordinary migration gets a read-only registry and no Git credential. Rendered Compose tests inspect
|
||
mounts, environment, profile, image ID, and command; `inspect` cannot pull an existing registry.
|
||
|
||
The RED registry matrix must consume P2's exact request/plan/state/callback surface rather than shadow
|
||
it. Test both `registry_bootstrap` (no base; changed set is all target IDs) and `registry_pull` (exact
|
||
base; changed set is the lexical symmetric base/target identity difference). Test every phase in order:
|
||
`request_claimed`, `target_advertised`, `target_fetched`, `planned`, `participants_prepared`,
|
||
`publication_intent_durable`, `target_published`, `terminal_durable`. Test the exact
|
||
`addressed-publication-jobs/<run-id>.json` path and reject the removed pull-only job-directory spelling.
|
||
Require repository lock before the claim/network/pin sequence; `acquireOrProvision` for every complete
|
||
changed ID while the repository lock is held; one call to `runUnderOrderedWorkspaceWriterLocks`; the
|
||
same callback-scoped capability set in the production lifecycle owner and every synchronizer; the same
|
||
addressed lease objects in participants; no reacquisition; and all-or-nothing pointer publication.
|
||
|
||
Root tests include newly added and never-used removed IDs with absent leaves, concurrent creators,
|
||
symlink substitution, wrong owner/mode, parent replacement, failure before publication, retained unused
|
||
leaf, and successful same-ID retry. Re-run P2's unchanged native seam and
|
||
`workspace-session-readers-lock.test.ts` production ownership tests: exact
|
||
`LockFileName = "writer.lock" | "session-readers.lock"`, wrapper-only typed open/flock ownership,
|
||
`VerifiedWorkspaceLockRootLease.acquireSessionReadersShared()`, exact
|
||
`WorkspaceSessionReadersLockLease` transfer/close, and
|
||
`WorkspaceWriterLockCapability.runUnderSessionReadersExclusive(...)`. P3 RED tests compile the adapter
|
||
against those exact types, spy that each adapter method calls only its matching P2 method, and exercise
|
||
real contention through the production P2 methods rather than a shadow opener. Compile/source fences
|
||
reject any P3 `WorkspaceFsAtV1` dependency for reader acquisition, raw-addon/`fs-ext` import, direct
|
||
open/flock, directory handle, path, numeric FD, lock name/flags, cast, wrapper lease, public constructor,
|
||
or second lock factory.
|
||
|
||
Route/app tests prove inspect, lazy list bootstrap, and status call only `ensureBootstrapAddressed`;
|
||
explicit pull and author-publication activation call only `publishAddressed`. Across every bootstrap
|
||
preterminal kill point, each actual product caller automatically resumes the sole exact-identity
|
||
nonterminal run under continuously held `repository.lock`; zero nonterminal jobs creates one fresh run.
|
||
Add a two-/three-process barrier test in which inspect, lazy list, and status all observe initial absence:
|
||
the first locked caller publishes exactly one bootstrap, while queued callers revalidate and receive the
|
||
same exact P2 `already_active` snapshot result without scanning zero into another create, state writes,
|
||
or network. Ordinary calls made after active state exists take the same branch. Corrupt/incompatible
|
||
active state and corrupt/mismatched/pull/multiple nonterminals return the exact fail-closed code before
|
||
network/state and never choose newest/mtime/lexical/OID. Each route/service adapter consumes P2's exact
|
||
`RegistryEnsureBootstrapAddressedResultV1` rather than assuming a terminal bootstrap result, and compile
|
||
imports pin its `RegistryActiveSnapshotV1` snapshot. The removed public/internal `bootstrap`, `pull`,
|
||
`activate`, and direct active-pointer writer are absent. Session routes use `canonicalInput(id)` plus
|
||
existing-leaf `acquire`, never provisioning. Application construction injects the sole P2
|
||
`WorkspaceFsAtV1` into the installation-bound root factory and production lifecycle owner once;
|
||
`WorkspaceReaderLeaseFactory` receives neither that wrapper nor any filesystem operand.
|
||
|
||
For the actual production pull call, add compile assertions on both literal branches and runtime spies
|
||
through the released create and resume commands. Create must pass the validated boundary's
|
||
`installationIdentitySha256`, `repositoryIdentitySha256`, `remoteRefIdentitySha256`, and
|
||
`expectedBaseCommit`; resume must pass all three identity digests and omit only
|
||
`expectedBaseCommit`. Omission, substitution, cross-installation/repository/ref reuse, and create-base
|
||
mismatch must fail before the first advertisement/network call and before addressed state creation or
|
||
transition. Assert zero calls to the network and state-mutation spies, and prove the host request cannot
|
||
supply or override any of these four values.
|
||
|
||
Create `backend/test/registry-pull-job-imports.compile.ts` with one `import type` declaration from the
|
||
single exact pull-job module below and references to all six released names. In that same compile gate,
|
||
import P2's exact `LockFileName` from `workspace-fs-at.js` and exact
|
||
`RegistryActiveSnapshotV1`/`RegistryEnsureBootstrapAddressedResultV1` from
|
||
`registry-publication.js`; no alias or alternate module path is permitted:
|
||
|
||
```ts
|
||
import type {
|
||
RegistryPullPublicJobRequestV1,
|
||
RegistryPullPhaseV1,
|
||
RegistryPullAddressedJobRequestV1,
|
||
RegistryPullParticipantStateV1,
|
||
RegistryPullSynchronizerStateV1,
|
||
RegistryPullJobStateV1,
|
||
} from "../src/workspaces/registry-pull-job.js";
|
||
import type { LockFileName } from "../src/workspaces/workspace-fs-at.js";
|
||
import type {
|
||
Revision40,
|
||
Sha256Hex,
|
||
} from "../src/workspaces/workspace-lock-root-lease.js";
|
||
import type {
|
||
RegistryActiveSnapshotV1,
|
||
RegistryAddressedPublicationPhaseV1,
|
||
RegistryAddressedRequestV1,
|
||
RegistryEnsureBootstrapAddressedResultV1,
|
||
RegistryPullAddressedPublicationStateV1,
|
||
} from "../src/workspaces/registry-publication.js";
|
||
|
||
type Equal<A, B> =
|
||
(<T>() => T extends A ? 1 : 2) extends (<T>() => T extends B ? 1 : 2)
|
||
? (<T>() => T extends B ? 1 : 2) extends (<T>() => T extends A ? 1 : 2)
|
||
? true
|
||
: false
|
||
: false;
|
||
type Assert<T extends true> = T;
|
||
type AllRegistryPullExports = readonly [
|
||
RegistryPullPublicJobRequestV1,
|
||
RegistryPullPhaseV1,
|
||
RegistryPullAddressedJobRequestV1,
|
||
RegistryPullParticipantStateV1,
|
||
RegistryPullSynchronizerStateV1,
|
||
RegistryPullJobStateV1,
|
||
];
|
||
type RegistryPullCreateRequestV1 = Extract<
|
||
RegistryPullAddressedJobRequestV1,
|
||
{ readonly mode: "create" }
|
||
>;
|
||
type RegistryPullResumeRequestV1 = Extract<
|
||
RegistryPullAddressedJobRequestV1,
|
||
{ readonly mode: "resume" }
|
||
>;
|
||
type ExactP2HandoffSymbols = readonly [
|
||
Assert<Equal<LockFileName, "writer.lock" | "session-readers.lock">>,
|
||
RegistryActiveSnapshotV1,
|
||
RegistryEnsureBootstrapAddressedResultV1,
|
||
Assert<Equal<
|
||
Extract<RegistryEnsureBootstrapAddressedResultV1, { readonly kind: "already_active" }>["snapshot"],
|
||
RegistryActiveSnapshotV1
|
||
>>,
|
||
];
|
||
type ExactP2Parity = readonly [
|
||
Assert<Equal<RegistryPullPhaseV1, RegistryAddressedPublicationPhaseV1>>,
|
||
Assert<Equal<
|
||
RegistryPullAddressedJobRequestV1,
|
||
Extract<RegistryAddressedRequestV1, { readonly operation: "registry_pull" }>
|
||
>>,
|
||
Assert<Equal<
|
||
Pick<
|
||
RegistryPullCreateRequestV1,
|
||
| "installationIdentitySha256"
|
||
| "repositoryIdentitySha256"
|
||
| "remoteRefIdentitySha256"
|
||
| "expectedBaseCommit"
|
||
>,
|
||
{
|
||
readonly installationIdentitySha256: Sha256Hex;
|
||
readonly repositoryIdentitySha256: Sha256Hex;
|
||
readonly remoteRefIdentitySha256: Sha256Hex;
|
||
readonly expectedBaseCommit: Revision40;
|
||
}
|
||
>>,
|
||
Assert<Equal<
|
||
Pick<
|
||
RegistryPullResumeRequestV1,
|
||
| "installationIdentitySha256"
|
||
| "repositoryIdentitySha256"
|
||
| "remoteRefIdentitySha256"
|
||
>,
|
||
{
|
||
readonly installationIdentitySha256: Sha256Hex;
|
||
readonly repositoryIdentitySha256: Sha256Hex;
|
||
readonly remoteRefIdentitySha256: Sha256Hex;
|
||
}
|
||
>>,
|
||
Assert<Equal<Extract<keyof RegistryPullResumeRequestV1, "expectedBaseCommit">, never>>,
|
||
Assert<Equal<RegistryPullJobStateV1, RegistryPullAddressedPublicationStateV1>>,
|
||
];
|
||
export type { AllRegistryPullExports, ExactP2HandoffSymbols, ExactP2Parity };
|
||
```
|
||
|
||
Run:
|
||
|
||
```bash
|
||
(cd tools/thothctl && test "$(go env GOVERSION)" = "go1.26.5" && \
|
||
go test ./internal/workspaceops ./cmd/thothctl \
|
||
-run 'Workspace|RegistryPull|Migrate|Resume|Capability|DedicatedJob' -v)
|
||
(cd backend && npx vitest run \
|
||
test/workspace-lock-root-lease.test.ts test/workspace-session-readers-lock.test.ts \
|
||
test/workspace-preprocessing-state.test.ts \
|
||
test/workspace-preprocessing-service.test.ts test/workspace-maintenance.test.ts \
|
||
test/workspace-registry.test.ts test/workspace-registry-addressed-publication.test.ts \
|
||
test/workspace-registry-addressed-process.test.ts test/workspace-reader-lease.test.ts \
|
||
test/routes-workspaces.test.ts test/routes-sessions.test.ts test/pi-process-manager.test.ts)
|
||
(cd backend && npx tsc --noEmit --target ES2022 --module ES2022 \
|
||
--moduleResolution Bundler --strict --skipLibCheck \
|
||
test/registry-pull-job-imports.compile.ts)
|
||
```
|
||
|
||
Expected: RED on new command members, reader lifecycle, P3 callback participants, root-provisioning
|
||
coverage, route regression, and the not-yet-created exact export module; existing P2 names themselves
|
||
compile unchanged.
|
||
|
||
**Step 2: Export one exact RegistryPull type surface without forking P2 state.**
|
||
|
||
Create exactly `backend/src/workspaces/registry-pull-job.ts`. It is the sole module that exports the six
|
||
`RegistryPull*` job names consumed by P4-P6. It imports P2's brands and addressed types; it declares no
|
||
second workspace/revision/SHA/run-ID brand and no second durable job shape:
|
||
|
||
```ts
|
||
// backend/src/workspaces/registry-pull-job.ts
|
||
import type {
|
||
CanonicalWorkspaceId,
|
||
Sha256Hex,
|
||
} from "./workspace-lock-root-lease.js";
|
||
import type {
|
||
RegistryAddressedPublicationPhaseV1,
|
||
RegistryAddressedRequestV1,
|
||
RegistryPullAddressedPublicationStateV1,
|
||
RegistryRunId32,
|
||
} from "./registry-publication.js";
|
||
|
||
export interface RegistryPullPublicJobRequestV1 {
|
||
readonly mode: "create" | "resume";
|
||
readonly operation: "registry_pull";
|
||
readonly requestSha256: Sha256Hex;
|
||
readonly runId: RegistryRunId32;
|
||
readonly schemaVersion: 1;
|
||
readonly workspaceId: CanonicalWorkspaceId; // result selector, never the changed set
|
||
}
|
||
export type RegistryPullPhaseV1 = RegistryAddressedPublicationPhaseV1;
|
||
export type RegistryPullAddressedJobRequestV1 = Extract<
|
||
RegistryAddressedRequestV1,
|
||
{ readonly operation: "registry_pull" }
|
||
>;
|
||
export interface RegistryPullParticipantStateV1 {
|
||
readonly workspaceId: CanonicalWorkspaceId;
|
||
readonly baseWorkspaceIdentitySha256: Sha256Hex | null;
|
||
readonly targetWorkspaceIdentitySha256: Sha256Hex | null;
|
||
readonly preparedSha256: Sha256Hex | null;
|
||
}
|
||
export interface RegistryPullSynchronizerStateV1 {
|
||
readonly synchronizerId: string;
|
||
readonly preparedSha256: Sha256Hex | null;
|
||
}
|
||
export type RegistryPullJobStateV1 = RegistryPullAddressedPublicationStateV1;
|
||
```
|
||
|
||
`RegistryPullPhaseV1`, `RegistryPullAddressedJobRequestV1`, and `RegistryPullJobStateV1` are deliberate
|
||
exact aliases to P2's released types, not parallel brands or persisted wrappers. Thus the pull state is
|
||
field-for-field P2: `jobArtifactPath`; operation/base fields; `installationIdentitySha256`,
|
||
`repositoryIdentitySha256`, and `remoteRefIdentitySha256`; nullable `advertisedTargetCommit`, `immutableTargetRef`,
|
||
`fetchedTargetCommit`, target/changed-plan fields and digests before their phases; and all publication,
|
||
terminal, and prior-state digests. `RegistryPullParticipantStateV1` and
|
||
`RegistryPullSynchronizerStateV1` are bounded callback receipts used to compute P2's aggregate digests;
|
||
they never create another job artifact or add fields to `RegistryPullJobStateV1`.
|
||
|
||
The compile-import test is a release gate and its bidirectional `Equal` assertions are the exact P2 symbol/field parity fence. Add an AST/source-boundary test that all six names have one
|
||
exporting module, that consumers import that exact `.js` path, and that none is redeclared in
|
||
`preprocessing-state.ts`, `registry-publication.ts`, or P4-P6. No barrel, compatibility export, or P5/P6
|
||
alias is permitted.
|
||
|
||
The public process request remains canonical compact JSON plus LF. `requestSha256` hashes exactly
|
||
`{"operation":"...","runId":"...","schemaVersion":1,"workspaceId":"..."}\n`; the stdin object has
|
||
exactly `mode,operation,requestSha256,runId,schemaVersion,workspaceId` in lexicographic order and is at
|
||
most 4,096 bytes. Go and TypeScript share golden vectors. Reject duplicate/reordered/unknown keys, BOM,
|
||
noncanonical whitespace/escaping, trailing bytes, invalid UTF-8, bad identities, and oversize input
|
||
before state access.
|
||
|
||
The state locations are exactly:
|
||
|
||
```text
|
||
/data/sessions/<workspace-id>/preprocessing/
|
||
writer.lock
|
||
session-readers.lock
|
||
maintenance-quiescence.json
|
||
jobs/<32-hex-run-id>.json # four ordinary operations only
|
||
<registry-root>/
|
||
repository.lock
|
||
addressed-publication-jobs/<32-hex-run-id>.json # exact P2 addressed state
|
||
/data/sessions/<changed-id>/preprocessing/
|
||
maintenance-quiescence.json # addressed callback owner
|
||
session-readers.lock
|
||
```
|
||
|
||
Do not create a pull-only job directory, a P3 registry state directory, or another active-pointer writer.
|
||
|
||
**Step 3: Implement writer-first ordinary ownership and consume P2's repository-first callback.**
|
||
|
||
`thothctl` invokes no backend transport. Fresh and resume commands launch only P2's selected,
|
||
immutable-image-pinned, profile-gated dedicated job:
|
||
|
||
```text
|
||
docker compose ... run --rm --no-deps --no-TTY --name thoth-workspace-maintenance-<run-id> workspace-maintenance --request-json-stdin
|
||
```
|
||
|
||
The job is runnable with `core` absent. No path uses `compose exec core`, curl/HTTP/internal routes,
|
||
Fastify, frontend, host Python/Node/Pi/`tht`, or a core stop/start command.
|
||
|
||
The four ordinary operations retain this exact order:
|
||
|
||
```text
|
||
VerifiedWorkspaceLockRootLeaseFactory.acquire(existing canonical selected ID)
|
||
-> transfer into runUnderWorkspaceWriterLock
|
||
-> exact WorkspaceWriterLockCapability callback (writer FD 3 + retained root FD 4)
|
||
-> ordinary job create/replay, owner quiescence publication, and reader drain
|
||
-> WorkspaceReaderLeaseFactory.acquireForMaintenance(writerCapability, full quiesced callback)
|
||
-> P2 runUnderSessionReadersExclusive holds LOCK_EX across final recheck, children, and publication
|
||
-> terminal durability and owner clear inside that same callback
|
||
-> callback settlement invalidates/closes reader borrow, then writer/root close
|
||
```
|
||
|
||
They must neither call `acquireOrProvision` nor enter `publishAddressed`. Implement
|
||
`WorkspaceReaderLeaseFactory` as a zero-filesystem adapter over P2's exact owner methods—without a P3
|
||
verified-root alias or wrapper lease:
|
||
|
||
```ts
|
||
import type {
|
||
VerifiedWorkspaceLockRootLease,
|
||
WorkspaceSessionReadersLockLease,
|
||
} from "./workspace-lock-root-lease.js";
|
||
import type {
|
||
BorrowedWorkspaceSessionReadersExclusiveLockLease,
|
||
WorkspaceWriterLockCapability,
|
||
} from "./preprocessing-state.js";
|
||
|
||
export class WorkspaceReaderLeaseFactory {
|
||
acquireForSession(
|
||
rootLease: VerifiedWorkspaceLockRootLease,
|
||
): Promise<WorkspaceSessionReadersLockLease> {
|
||
return rootLease.acquireSessionReadersShared();
|
||
}
|
||
|
||
acquireForMaintenance<T>(
|
||
writerCapability: WorkspaceWriterLockCapability,
|
||
action: (lease: BorrowedWorkspaceSessionReadersExclusiveLockLease) => Promise<T>,
|
||
): Promise<T> {
|
||
return writerCapability.runUnderSessionReadersExclusive(action);
|
||
}
|
||
}
|
||
```
|
||
|
||
There is no P3 `WorkspaceReaderLease`, `release()` wrapper, constructor, lock opener, or copied lifetime
|
||
logic. `acquireForSession` calls only P2's consuming `acquireSessionReadersShared()` and returns its exact
|
||
`WorkspaceSessionReadersLockLease`. `acquireForMaintenance` returns the exact result of
|
||
`writerCapability.runUnderSessionReadersExclusive(action)`; it never returns an owned maintenance lease,
|
||
and the exact `BorrowedWorkspaceSessionReadersExclusiveLockLease` may exist only inside `action`.
|
||
`WorkspaceReaderLeaseFactory` has no constructor dependency and never imports or receives
|
||
`WorkspaceFsAtV1`, a directory/regular-file handle, path, FD, lock name, flags, or mode. P2 alone opens,
|
||
flocks, invalidates, and closes the lock.
|
||
|
||
The session route calls only `canonicalInput(id)` then existing-leaf `acquire(input)`, checks
|
||
quiescence, calls `acquireForSession(rootLease)`, and rechecks quiescence while holding that exact P2
|
||
shared owner. `RuntimeOptions.sessionReadersLease` is typed as `WorkspaceSessionReadersLockLease`.
|
||
`PiProcessManager` synchronously calls `transfer()` at the ownership handoff and from then on closes its
|
||
owned lease exactly once only after every Pi/session child **and every stdout/stderr/read stream** has
|
||
settled. The route closes an untransferred owner on every pre-handoff failure; after transfer its source
|
||
close is the P2 idempotent no-op and the manager owns cleanup across configure/start failure, `exit`,
|
||
`close`, replacement, explicit teardown, shutdown, cancellation, and stream error.
|
||
|
||
For ordinary maintenance, publish the owner-qualified quiescence marker and drain existing readers
|
||
first, then call `acquireForMaintenance(writerCapability, async (readerBorrow) => { ... })` exactly once.
|
||
That callback contains the complete quiesced action: final drain/admission-state recheck, every locked
|
||
child/participant, publication, terminal durability, and matching owner clear. It calls
|
||
`readerBorrow.assertLive()` at its protected boundaries and settles only after child stdout/stderr
|
||
collection settles. The P2 exclusive owner remains held for the callback's resolve/reject lifetime; the
|
||
borrow invalidates before return, and no existing reader is killed.
|
||
|
||
Adapter tests use compile-time exact P2 imports and runtime spies to prove one call to each matching P2
|
||
method, exact returned/result identity, no other call, and no maintenance-borrow escape. Production tests
|
||
reuse `workspace-session-readers-lock.test.ts`: acquire a real shared owner, transfer it into a fixture
|
||
`PiProcessManager`, keep a child plus stdout/stderr drains open, and prove the production exclusive
|
||
callback contends until all child/stream teardown and exactly-once close. Conversely, hold the production
|
||
exclusive callback around the full quiesced action and prove new shared admission contends, captured
|
||
borrows fail after settlement, callback failure closes once, and release permits admission. Source/type
|
||
fences reject a direct wrapper/raw-addon/`fs-ext` import, directory or file handle, path, numeric FD, lock
|
||
name/flags/mode, cast, direct open/flock, path fallback, wrapper lease, or alternate factory.
|
||
|
||
`registry_pull` has no selected-workspace wrapper. Its production path first enters the validated
|
||
installation/registry boundary and obtains one immutable identity tuple. The boundary derives and
|
||
revalidates all four values against the retained installation descriptor, owned registry volume,
|
||
canonical repository, configured remote/ref, and current active snapshot; the host request cannot
|
||
supply or override any of them:
|
||
|
||
```ts
|
||
const {
|
||
installationIdentitySha256,
|
||
repositoryIdentitySha256,
|
||
remoteRefIdentitySha256,
|
||
expectedBaseCommit,
|
||
} = await validatedInstallationRegistryBoundary.deriveRegistryPullIdentity();
|
||
|
||
const result = await registry.publishAddressed(
|
||
request.mode === "create"
|
||
? ({
|
||
mode: "create",
|
||
operation: "registry_pull",
|
||
runId: request.runId,
|
||
requestSha256: request.requestSha256,
|
||
installationIdentitySha256,
|
||
repositoryIdentitySha256,
|
||
remoteRefIdentitySha256,
|
||
expectedBaseCommit,
|
||
} satisfies Extract<
|
||
RegistryPullAddressedJobRequestV1,
|
||
{ readonly mode: "create" }
|
||
>)
|
||
: ({
|
||
mode: "resume",
|
||
operation: "registry_pull",
|
||
runId: request.runId,
|
||
requestSha256: request.requestSha256,
|
||
installationIdentitySha256,
|
||
repositoryIdentitySha256,
|
||
remoteRefIdentitySha256,
|
||
} satisfies Extract<
|
||
RegistryPullAddressedJobRequestV1,
|
||
{ readonly mode: "resume" }
|
||
>),
|
||
);
|
||
```
|
||
|
||
These are the actual production call literals, not test-only examples: their two `satisfies` clauses are
|
||
compile gates against P2's exact request union, while runtime command/service tests spy on this call for
|
||
both modes. A create base mismatch, or any create/resume installation/repository/remote identity mismatch
|
||
with validated active state or a durable addressed record, fails before creating/transitioning addressed
|
||
state and before advertisement or other network. Do not redeclare P2's callback. Consume it exactly:
|
||
`CapabilityAwareRegistryPublicationLifecycleOwner.run({ plan, capabilities, participants,
|
||
synchronizers, action })`. Its `plan` is the P2 `RegistryAddressedPlanV1` union, so P3 participants and
|
||
synchronizers handle both `RegistryBootstrapAddressedPlanV1` and `RegistryPullAddressedPlanV1` by
|
||
discriminating `plan.operation`. Each participant receives exactly
|
||
`AddressedWorkspacePublicationLeaseV1`, including P2's borrowed retained root, writer capability,
|
||
quiescence, and reader lease. Each synchronizer receives the exact same
|
||
`OrderedWorkspaceWriterCapabilitySet`. No callback calls root acquisition, writer-lock acquisition,
|
||
reader acquisition, registry publication, or a nested lifecycle owner.
|
||
|
||
For both addressed variants, P2 holds `repository.lock`, computes the exact complete changed IDs, calls
|
||
`acquireOrProvision(canonicalInput(id))` for each ID, and passes the full root array once to
|
||
`runUnderOrderedWorkspaceWriterLocks`. Missing added and never-used removed leaves are securely
|
||
provisioned under the retained parent FD with P2's fixed UID/mode/fsync/identity rules and retained on
|
||
failure. The factory consumes only P2's `WorkspaceFsAtV1` wrapper over
|
||
`backend/native/workspace-fs-at/workspace_fs_at.cc`; literal `openat`/`mkdirat`/no-follow `fstatat`,
|
||
directory fsync, owned close, the closed `"writer.lock" | "session-readers.lock"` open surface, typed
|
||
flock modes, flags, errors, Node 22 build, and Darwin/Linux behavior remain the exact P2 contract. The
|
||
wrapper's module-private synchronous numeric borrow is the only bridge to exact `fs-ext@2.1.1` and is
|
||
used internally for typed flock; P3 cannot import either underlying module, accept/return/borrow/cast an
|
||
FD, cast an owned handle/name, accept flags, implement another mkdir/open/flock factory, or introduce a
|
||
path fallback. Bootstrap requires no
|
||
active base, null base fields, `[]` base workspaces, and all target IDs. Pull requires the exact active
|
||
base and lexical symmetric base/target workspace-identity difference. Empty changed sets still use the
|
||
same lifecycle and publish/terminal protocol without inventing a selected lock.
|
||
|
||
Product bootstrap recovery is automatic rather than a new public selector. Inspect, lazy list, and
|
||
status call only `ensureBootstrapAddressed(identity)` and consume P2's exact discriminated result union.
|
||
Under one continuously held `repository.lock`, the selector first reads and validates active state. A
|
||
valid compatible snapshot returns the exact `already_active` branch with that snapshot and performs no
|
||
job scan, state create/transition, or network; corrupt/incompatible active state fails closed. Only
|
||
validated absence no-follow scans one lexically sorted `addressed-publication-jobs/` snapshot with P2's
|
||
exact 4,096-entry, 1,048,576-byte-per-artifact, and 67,108,864-byte-total bounds and validates every
|
||
record before selection. Zero nonterminals creates one fresh ID; exactly one matching
|
||
`registry_bootstrap` nonterminal resumes that exact ID before any network. Unknown/corrupt/churning
|
||
entries, identity mismatch, nonterminal pull, or multiple nonterminals fail
|
||
`registry_bootstrap_recovery_conflict`; terminal jobs are ignored for automatic selection, and no
|
||
mtime/newest/lexical-last/OID/remote-head heuristic exists. A queued inspect/list/status caller never
|
||
acts on its pre-lock observation: after acquiring the lock it returns the new `already_active` snapshot
|
||
published by the winning caller. The bootstrap request digest excludes run ID/mode and binds schema,
|
||
operation, installation, repository, and remote-ref identity exactly.
|
||
|
||
Fresh addressed create durably writes `request_claimed` before `ls-remote`, fetch, or any network call.
|
||
Then it advertises once and records the OID at `target_advertised`; exact-OID fetch creates only
|
||
`refs/thoth/addressed-runs/<run-id>/target`, verifies it equals the advertisement, and records
|
||
`target_fetched`; only then may manifest reading and the full plan produce `planned`. All writes use the
|
||
exact P2 sibling-write/file-fsync/rename/parent-fsync state transition and `priorStateSha256` chain.
|
||
|
||
Same-ID recovery is split at the durable pin and tested literally:
|
||
|
||
1. Before `request_claimed` rename is durable, no network was allowed; the same request may recreate
|
||
that claim.
|
||
2. From durable `request_claimed` until `target_advertised` is durable, no target was promised; resume
|
||
repeats advertisement and may observe a newer OID.
|
||
3. At/after durable `target_advertised`, that OID is permanent. If the immutable run ref is absent,
|
||
resume retries fetch of only that exact OID; if it already resolves to that OID, resume performs no
|
||
network and advances; a different OID is corruption.
|
||
4. At/after `target_fetched`, resume never advertises, fetches, consults remote target selection, or
|
||
accepts another OID. A terminal job replays its stored result even if a later independent pull moved
|
||
remote-tracking refs. Drift checks apply only while reconciling a nonterminal run.
|
||
|
||
After `planned`, participant preparation and aggregate digests precede
|
||
`publication_intent_durable`. Publication stages immutable target records, atomically renames and
|
||
parent-fsyncs the installation-wide active pointer, rereads exact bytes, advances
|
||
`target_published`, and persists `terminal_durable` before callback settlement. Before pointer rename,
|
||
active is exact base (or absent for bootstrap). At/after it, same-ID resume accepts only exact recorded
|
||
base/target projections, converges all-base or mixed to target, recognizes all-target lost
|
||
acknowledgement, and refuses every third identity. It reuses the same full callback capability set.
|
||
Owner markers clear only after terminal durability while readers remain exclusive; then callback
|
||
borrows are invalidated and readers/quiescence/writers/roots release in reverse lexical order before
|
||
repository release.
|
||
|
||
Kill/process tests cover before claim rename, after claim fsync, during/after advertisement before its
|
||
durable record, after advertisement fsync, during/after exact fetch, after immutable-ref creation before
|
||
`target_fetched`, after its fsync, after `planned`, before/after pointer file fsync/rename/parent fsync,
|
||
after target reread before phase persistence, and after terminal before clear. Assert exact same-ID
|
||
behavior at every boundary, no post-pin target reselection, no post-`target_fetched` network, terminal
|
||
replay despite a later tracking-ref move, and deterministic base/target/mixed recovery. Repeat every
|
||
bootstrap preterminal boundary through actual inspect, lazy-list, and status callers and prove automatic
|
||
same-ID selection occurs before network; exercise zero/one/multiple/pull/mismatched/corrupt/churning job
|
||
sets and all three P2 scan bounds. Repository/ref/installation identity change, a missing/changed pinned
|
||
object, third active identity, wrong owner/digest, unsafe state, or nonterminal clear fails closed without
|
||
publication, and automatic conflicts preserve exact `registry_bootstrap_recovery_conflict`.
|
||
|
||
**Step 4: Extend P2's closed locked-child union in place and implement the one-shot lifecycle.**
|
||
|
||
In `backend/src/workspaces/preprocessing-state.ts`, preserve P2's three variants and extend the same
|
||
exported alias—never create a parallel spawner—with these exact P3 contracts:
|
||
|
||
```ts
|
||
type P3LockedOperation =
|
||
| "p3_migrate_dwh_cache" | "p3_prepare_dwh_cache" | "p3_materialize_dwh_snapshot"
|
||
| "p3_migrate_memory_root" | "p3_rebuild_memory_projection"
|
||
| "p3_inventory_semantic_legacy" | "p3_check_semantic_readiness"
|
||
| "p3_publish_semantic_replacements" | "p3_verify_semantic_replacements"
|
||
| "p3_delete_confirmed_semantic_legacy" | "p3_prepare_layout_markers"
|
||
| "p3_publish_layout_version" | "p3_publish_revision_ready"
|
||
| "p3_verify_revision_readiness";
|
||
import type {
|
||
CanonicalWorkspaceId,
|
||
Revision40,
|
||
Sha256Hex,
|
||
WorkspaceLockRootIdentityV1,
|
||
} from "./workspace-lock-root-lease.js";
|
||
interface P3LockedChildContextV1 {
|
||
schemaVersion: 1;
|
||
workspaceId: CanonicalWorkspaceId;
|
||
rootIdentity: WorkspaceLockRootIdentityV1; // from the consumed P2 lease; never argv/stdin
|
||
workspaceRevision: Revision40;
|
||
descriptorBlob: Revision40;
|
||
effectiveDwhBindingSha256: Sha256Hex;
|
||
effectiveDwhCacheKey: Sha256Hex;
|
||
outerRunId: string; // exactly 32 lowercase hex
|
||
configLease: MaintenanceRuntimeConfigLease; // opaque owned lease, not a caller path
|
||
}
|
||
interface P3LockedChildStdinV1 {
|
||
artifactKind: "semantic-legacy-inventory-v1" | "revision-ready-input-v1";
|
||
contentBase64: string;
|
||
contentByteLength: number;
|
||
contentSha256: Sha256Hex;
|
||
descriptorBlob: Revision40;
|
||
effectiveDwhBindingSha256: Sha256Hex;
|
||
effectiveDwhCacheKey: Sha256Hex;
|
||
outerRunId: string; // exactly 32 lowercase hex
|
||
schemaVersion: 1;
|
||
stateArtifact: "semantic-legacy-inventory.json" | "revision-ready-input.json";
|
||
workspaceId: CanonicalWorkspaceId;
|
||
workspaceRevision: Revision40;
|
||
}
|
||
type P3LockedChildRequest =
|
||
| { kind: "p3_migrate_dwh_cache"; context: P3LockedChildContextV1 }
|
||
| { kind: "p3_prepare_dwh_cache"; context: P3LockedChildContextV1 }
|
||
| { kind: "p3_materialize_dwh_snapshot"; context: P3LockedChildContextV1 }
|
||
| { kind: "p3_migrate_memory_root"; context: P3LockedChildContextV1 }
|
||
| { kind: "p3_rebuild_memory_projection"; context: P3LockedChildContextV1 }
|
||
| { kind: "p3_inventory_semantic_legacy"; context: P3LockedChildContextV1 }
|
||
| { kind: "p3_check_semantic_readiness"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 }
|
||
| { kind: "p3_publish_semantic_replacements"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 }
|
||
| { kind: "p3_verify_semantic_replacements"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 }
|
||
| { kind: "p3_delete_confirmed_semantic_legacy"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 }
|
||
| { kind: "p3_prepare_layout_markers"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 }
|
||
| { kind: "p3_publish_layout_version"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 }
|
||
| { kind: "p3_publish_revision_ready"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 }
|
||
| { kind: "p3_verify_revision_readiness"; context: P3LockedChildContextV1; stdin: P3LockedChildStdinV1 };
|
||
export type WorkspaceLockedChildRequest =
|
||
| DwhLockedChildRequest | SchemaLockedChildRequest | EvidenceLockedChildRequest
|
||
| P3LockedChildRequest;
|
||
interface P3ResultBaseV1 {
|
||
schemaVersion: 1; workspaceId: CanonicalWorkspaceId; workspaceRevision: Revision40;
|
||
effectiveDwhCacheKey: Sha256Hex; status: "succeeded" | "unchanged" | "not_ready";
|
||
}
|
||
type P3LockedChildParsedResult =
|
||
| (P3ResultBaseV1 & { operation: "p3_migrate_dwh_cache" | "p3_prepare_dwh_cache";
|
||
ownerSha256: Sha256Hex; generationId: string; generationManifestSha256: Sha256Hex })
|
||
| (P3ResultBaseV1 & { operation: "p3_materialize_dwh_snapshot";
|
||
snapshotGenerationId: string; snapshotManifestSha256: Sha256Hex })
|
||
| (P3ResultBaseV1 & { operation: "p3_migrate_memory_root";
|
||
registrySha256: Sha256Hex; recordCount: number })
|
||
| (P3ResultBaseV1 & { operation: "p3_rebuild_memory_projection";
|
||
registrySha256: Sha256Hex; memoryPointCount: number; solvedPointCount: number })
|
||
| (P3ResultBaseV1 & { operation: "p3_inventory_semantic_legacy";
|
||
inventorySha256: Sha256Hex; inventoryByteLength: number; pointCount: number;
|
||
artifactKind: "semantic-legacy-inventory-v1"; contentBase64: string })
|
||
| (P3ResultBaseV1 & { operation: "p3_check_semantic_readiness";
|
||
inventorySha256: Sha256Hex; schemaSourceSha256: Sha256Hex;
|
||
evidenceSourceSha256: Sha256Hex; schemaCount: number; evidenceCount: number })
|
||
| (P3ResultBaseV1 & { operation: "p3_publish_semantic_replacements" |
|
||
"p3_verify_semantic_replacements" | "p3_delete_confirmed_semantic_legacy";
|
||
inventorySha256: Sha256Hex; replacementSetSha256: Sha256Hex;
|
||
schemaCount: number; evidenceCount: number; deletedLegacyCount: number })
|
||
| (P3ResultBaseV1 & { operation: "p3_prepare_layout_markers";
|
||
layoutMarkerInputSha256: Sha256Hex; readyInputSha256: Sha256Hex })
|
||
| (P3ResultBaseV1 & { operation: "p3_publish_layout_version";
|
||
layoutMarkerInputSha256: Sha256Hex; layoutMarkerSha256: Sha256Hex })
|
||
| (P3ResultBaseV1 & { operation: "p3_publish_revision_ready";
|
||
readyInputSha256: Sha256Hex; readyManifestSha256: Sha256Hex })
|
||
| (P3ResultBaseV1 & { operation: "p3_verify_revision_readiness";
|
||
readyInputSha256: Sha256Hex; readyManifestSha256: Sha256Hex });
|
||
```
|
||
|
||
Only semantic operations accept `semantic-legacy-inventory-v1`; all four marker/readiness variants
|
||
accept only `revision-ready-input-v1`. Marker preparation returns canonical bounded layout/READY input
|
||
digests but writes neither marker. The coordinator then calls the two exact capability variants in
|
||
transaction order: `p3_publish_layout_version` before confirmed legacy delete and
|
||
`p3_publish_revision_ready` after deletion. Both publishers mutate only through inherited FD 4 while
|
||
FD 3 remains verified/held; readiness verification rereads the exact published bytes. Node never
|
||
publishes either marker by pathname.
|
||
|
||
The child stdin wire is frozen byte-for-byte. It is one UTF-8 compact JSON object with **exactly** the
|
||
12 keys in lexicographic order `artifactKind,contentBase64,contentByteLength,contentSha256,descriptorBlob,effectiveDwhBindingSha256,effectiveDwhCacheKey,outerRunId,schemaVersion,stateArtifact,workspaceId,workspaceRevision`, no BOM, duplicate/reordered/unknown key, optional whitespace, alternate escaping, or trailing byte, followed by exactly one LF. `contentBase64` is strict padded RFC 4648 base64 of the canonical persisted artifact bytes; `contentByteLength` and `contentSha256` describe the decoded bytes, never the JSON/base64 text. The remaining identity fields must equal the locked request context, rendered config, and exact identity fields inside the decoded artifact.
|
||
|
||
Inventory content is at most 716,800 decoded bytes and readiness content at most 65,536. With the
|
||
inherited canonical ASCII workspace-ID bound of 128 bytes and all fixed-width identities above, the
|
||
exact largest inventory stdin is `4*ceil(716800/3) + 745 = 956,481` bytes; the largest readiness stdin
|
||
is 88,118 bytes. Both are below the fixed 1,048,576-byte stdin cap. Node computes the envelope once
|
||
from state-store bytes, validates it before spawn, and writes those exact bytes without
|
||
`JSON.stringify(Uint8Array)` or a second serialization. No variant can represent raw argv, executable,
|
||
env, cwd, stdio, FD, config path, workspace destination, or arbitrary state path.
|
||
|
||
The capability's exhaustive switch is the sole argv builder and emits exactly these internal argv
|
||
after the `tht` executable (where `CONFIG` comes only from the opaque lease):
|
||
|
||
```text
|
||
p3_migrate_dwh_cache config migrate-dwh-cache --json -c CONFIG
|
||
p3_prepare_dwh_cache preprocess dwh --prepare-revision-layout-v1 --json -c CONFIG
|
||
p3_materialize_dwh_snapshot config materialize-dwh-snapshot --json -c CONFIG
|
||
p3_migrate_memory_root memory migrate-root --json -c CONFIG
|
||
p3_rebuild_memory_projection memory rebuild-projection --json -c CONFIG
|
||
p3_inventory_semantic_legacy vector semantic-inventory --json -c CONFIG
|
||
p3_check_semantic_readiness vector semantic-readiness --json -c CONFIG
|
||
p3_publish_semantic_replacements vector semantic-publish --json -c CONFIG
|
||
p3_verify_semantic_replacements vector semantic-verify --json -c CONFIG
|
||
p3_delete_confirmed_semantic_legacy vector semantic-delete-confirmed --json -c CONFIG
|
||
p3_prepare_layout_markers config prepare-layout-markers --json -c CONFIG
|
||
p3_publish_layout_version config publish-layout-version --json -c CONFIG
|
||
p3_publish_revision_ready config publish-revision-ready --json -c CONFIG
|
||
p3_verify_revision_readiness config verify-revision-readiness --json -c CONFIG
|
||
```
|
||
|
||
Implement `harness/tht/locked_child_stdin.py` now, not in Task 9, with one bounded binary-reader API
|
||
whose caller supplies only a compile-time literal expected artifact kind/state-artifact and decoded
|
||
limit. It requires actual writer FD 3 and retained-root FD 4 first; streams at most 1,048,576 stdin bytes; enforces the canonical
|
||
wire above; strict-base64 decodes; checks decoded length/digest; derives workspace, revision,
|
||
descriptor, and both binding identities from `Config`; and verifies outer-run/artifact identities
|
||
inside the decoded exact-key artifact. All four stdin-consuming semantic wrappers and all four marker/readiness
|
||
wrappers must call it; inventory has no stdin. Python negatives cover zero/two LF, BOM, invalid UTF-8, empty/truncated/oversize JSON,
|
||
duplicate/missing/unknown/reordered keys, whitespace/noncanonical escaping, bool-for-integer, invalid
|
||
base64/padding, decoded oversize, length/digest mismatch, wrong artifact/state/run/workspace/revision/
|
||
descriptor/binding, content-identity mismatch, and missing/substituted/cross-root FD 3 or FD 4 before content use.
|
||
|
||
Implement `harness/tht/layout_markers.py` and register all four exact Typer targets in
|
||
`harness/tht/cli/config_cmd.py` in this task. `config prepare-layout-markers` accepts only the frozen
|
||
`revision-ready-input-v1` envelope, validates the single-READY-per-revision rule, and returns the
|
||
canonical bounded layout/READY input digests without publication. `config
|
||
verify-revision-readiness` accepts the same contract and securely reopens the coordinator-published
|
||
marker and selected READY, checks exact bytes/digests/identities and rejects another READY key. Unit
|
||
tests cover help/registration, happy paths, strict stdin negatives, symlink/hardlink/replacement,
|
||
other-key READY, and no-write preparation. A table-driven Python test proves all fourteen argv targets
|
||
above resolve to the intended Typer command. A built-image test runs each target's fixed prefix with
|
||
`--help` in the selected `workspace-maintenance` image and compares the fourteen-target set exactly, so coordinator tests
|
||
cannot go GREEN against an unimplemented image command.
|
||
|
||
All mutating commands inherit only the actual writer FD 3 and retained-root FD 4 from the exact P2 capability. Inventory content remains capped
|
||
at 716,800 decoded bytes, its RFC 4648 base64 is exactly at most 955,736 bytes, `pointCount` is bounded
|
||
`0..716800`, and canonical workspace ID is ASCII at most 128 bytes. With the exact inventory-result
|
||
keys/types above, compact lexicographically keyed JSON, maximum-width values, and exactly one LF, the
|
||
proved worst-case child stdout is `955736 + 581 = 956,317` bytes. Therefore inventory child stdout is
|
||
bounded to **1,048,576 bytes** (matching P2's 1 MiB boundary), not 921,600; every other variant remains
|
||
262,144 and stderr 65,536. A real bounded-collector round-trip test emits a 716,800-byte inventory,
|
||
observes exactly 956,317 serialized bytes at maximum-width fields, strictly parses/decodes it, and
|
||
compares all original bytes/digests; 1,048,576 total bytes is accepted and byte 1,048,577 terminates
|
||
the process group without retaining overflow. Timeouts are 60 seconds for inventory/readiness
|
||
verification, 300 seconds for migrations/snapshot/semantic verify/delete, 900 seconds for Memory
|
||
projection, and 1,800 seconds for DWH prepare and semantic publish. Overflow/timeout terminates the
|
||
child process group and returns a stable failed result.
|
||
|
||
Parse exactly `P3LockedChildParsedResult` above; validate base identity, literal operation, SHA-256,
|
||
generation ID, integer counts, and status, and reject unknown keys, impossible per-operation fields,
|
||
identity mismatch, output other than the one canonical JSON object plus one LF, invalid
|
||
base64/decoded length, and public unsafe fields. Inventory `contentBase64` is decoded, hash/length
|
||
checked, persisted by `PreprocessingStateStore`, then removed before any public result encoding.
|
||
|
||
Add TypeScript compile-time `satisfies Record<P3LockedOperation,...>` and runtime `assertNever` tests,
|
||
plus AST/source-boundary tests that fail on `spawn`, `exec`, `fork`, or raw argv outside the
|
||
capability. Exercise every variant's exact argv/stdin/result/timeout; missing/substituted/cross-root FD 3 or FD 4,
|
||
cross-workspace/revision/binding context, wrong artifact, post-settlement use, and arbitrary
|
||
path/argv constructions fail before spawn. Coordinator tests monkeypatch all general process spawn
|
||
APIs to throw and prove every migration still succeeds only through `capability.spawnChild`.
|
||
|
||
|
||
The exhaustive `WorkspaceWriterLockCapability.spawnChild` switch remains P2's sole process producer.
|
||
Every P3 variant passes the locked writer open description as child FD 3 and the same capability-owned
|
||
retained root directory open description as child FD 4. Before reading config/stdin or touching a
|
||
source/destination, the Python shared child guard verifies FD 4's expected root identity and ownership, opens
|
||
`preprocessing/writer.lock` relative to FD 4 with no-follow semantics, proves it is FD 3's inode, and
|
||
proves FD 3 is already exclusively held. Missing, closed, renumbered, independently locked, substituted,
|
||
cross-root, cross-workspace, or post-callback FD use fails `preprocessing_conflict`. No P3 raw root path, verified-root alias, capability wrapper/brand, environment marker, or direct helper spawn exists.
|
||
|
||
On an ordinary failure/signal, retain state and owner quiescence, return the exact run ID, and require
|
||
same-ID resume; never wait for or restart `core`. Resume skips only phases whose exact persisted inputs
|
||
and outputs reverify. After every successful non-activation operation, owner-clear while the exclusive
|
||
reader lease remains held, then test a fresh admission. Before initial activation no revision-scoped
|
||
semantic replacement exists. After enablement a session selects only its currently probed exact
|
||
commit+binding READY; a second run cannot steal or clear the workspace.
|
||
|
||
**Step 5: Run type/symbol/process gates and commit the complete release surface.**
|
||
|
||
Run:
|
||
|
||
```bash
|
||
(cd tools/thothctl && test "$(go env GOVERSION)" = "go1.26.5" && go test ./...)
|
||
(cd backend && npm run build:native && npx vitest run \
|
||
test/workspace-fs-at-native.test.ts test/workspace-lock-root-lease.test.ts \
|
||
test/workspace-session-readers-lock.test.ts \
|
||
test/workspace-preprocessing-state.test.ts test/workspace-preprocessing-service.test.ts \
|
||
test/workspace-maintenance.test.ts test/workspace-registry.test.ts \
|
||
test/workspace-registry-addressed-publication.test.ts \
|
||
test/workspace-registry-addressed-process.test.ts test/workspace-reader-lease.test.ts \
|
||
test/routes-workspaces.test.ts test/routes-sessions.test.ts test/app.test.ts \
|
||
test/pi-process-manager.test.ts && npx tsc --noEmit -p . && npm run build)
|
||
(cd backend && npx tsc --noEmit --target ES2022 --module ES2022 \
|
||
--moduleResolution Bundler --strict --skipLibCheck \
|
||
test/registry-pull-job-imports.compile.ts)
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_locked_child_stdin.py tests/test_layout_marker_commands.py \
|
||
tests/test_p3_internal_cli.py tests/test_semantic_revision_migration.py \
|
||
tests/test_qdrant_cli_commands.py)
|
||
bash scripts/test-preprocess-compose-config.sh
|
||
bash scripts/test-compose-secret-policy.sh
|
||
```
|
||
|
||
Expected: PASS with no Go toolchain download. The four ordinary operations retain writer-first order,
|
||
request bytes, phase transitions, fixed child argv/results, admission semantics, and existing-root
|
||
`acquire`. The P2 native addon and typed wrapper pass their Linux/Darwin contract and remain the sole
|
||
FD-relative provisioning/open/flock seam. P3's zero-filesystem adapter calls only
|
||
`acquireSessionReadersShared()` and `runUnderSessionReadersExclusive(...)`; the exact shared owner is
|
||
transferred into `PiProcessManager` through child/stream teardown, and the exclusive callback encloses
|
||
the full quiesced action. No P3 wrapper/handle/path/FD/name/flags/direct-open factory exists. The pull path
|
||
calls only `publishAddressed`; its actual create literal passes all three validated identity digests plus
|
||
`expectedBaseCommit`, its resume literal passes all three digests, and compile/runtime mismatch tests
|
||
fail before network/state. It proves all eight exact P2 phases, pre-advertisement and
|
||
post-advertisement same-ID recovery, exact-OID immutable-ref recovery, no network at/after
|
||
`target_fetched`, bootstrap/pull callback parity, complete lexical ownership, missing-root provisioning,
|
||
same participant/capability objects, no reentry, exact state path, all-or-nothing publication,
|
||
base/target/mixed reconciliation, terminal-before-clear recovery, and reverse release. Inspect, lazy
|
||
list, and status call only `ensureBootstrapAddressed` and consume its full result union: bounded
|
||
exact-identity automatic same-ID bootstrap recovery works at every preterminal kill point, and queued or
|
||
ordinary active callers converge through the exact `already_active` snapshot with one publication and
|
||
no second network/state work; ambiguous/corrupt/incompatible cases fail before network/state. Routes
|
||
retain the sole bootstrap/pull/activation mutation lifecycle. Every mutating child validates both writer
|
||
FD 3 and retained-root FD 4. The standalone compile command imports all six released job types from
|
||
`../src/workspaces/registry-pull-job.js`, exact `LockFileName` from `workspace-fs-at.js`, and exact
|
||
`RegistryActiveSnapshotV1`/`RegistryEnsureBootstrapAddressedResultV1` from
|
||
`registry-publication.js`.
|
||
|
||
Run a final symbol fence before commit:
|
||
|
||
```bash
|
||
python3 - <<'PY'
|
||
from pathlib import Path
|
||
roots = [Path("backend/src"), Path("backend/test")]
|
||
text = "\n".join(p.read_text() for root in roots for p in root.rglob("*.ts"))
|
||
for stale in (
|
||
"RegistryPullBa" + "seTargetPlanV1",
|
||
"RegistryAddressedW" + "orkspaceLeaseOwner",
|
||
"acquireCo" + "mpleteSet",
|
||
"releaseCo" + "mpleteSet",
|
||
"pullAndPubl" + "ishAddressed",
|
||
"registry-" + "pull-jobs",
|
||
"VerifiedWo" + "rkspaceRoot",
|
||
"WorkspaceLockFi" + "leNameV1",
|
||
"RegistryValidatedActive" + "SnapshotV1",
|
||
"RegistryBootstrapEnsure" + "AddressedResultV1",
|
||
"interface Workspace" + "ReaderLease",
|
||
):
|
||
assert stale not in text, stale
|
||
module = Path("backend/src/workspaces/registry-pull-job.ts").read_text()
|
||
for name in (
|
||
"RegistryPullPublicJobRequestV1",
|
||
"RegistryPullPhaseV1",
|
||
"RegistryPullAddressedJobRequestV1",
|
||
"RegistryPullParticipantStateV1",
|
||
"RegistryPullSynchronizerStateV1",
|
||
"RegistryPullJobStateV1",
|
||
):
|
||
assert f"export " in module and name in module, name
|
||
print("P3 registry symbol fence: PASS")
|
||
PY
|
||
```
|
||
|
||
Expected: `P3 registry symbol fence: PASS`.
|
||
|
||
Commit all owning files, including the exact export and compile-import test:
|
||
|
||
```bash
|
||
git add backend/src/workspaces/preprocessing-state.ts \
|
||
backend/src/workspaces/preprocessing-service.ts backend/src/workspace-maintenance.ts \
|
||
backend/src/workspaces/registry.ts backend/src/workspaces/registry-publication.ts \
|
||
backend/src/workspaces/registry-pull-job.ts \
|
||
backend/test/registry-pull-job-imports.compile.ts \
|
||
backend/test/workspace-registry.test.ts \
|
||
backend/test/workspace-registry-addressed-publication.test.ts \
|
||
backend/test/workspace-registry-addressed-process.test.ts \
|
||
backend/test/fixtures/workspace-lock-root-worker.mjs \
|
||
backend/test/fixtures/workspace-registry-addressed-worker.mjs \
|
||
backend/src/workspaces/workspace-reader-lease.ts \
|
||
backend/src/routes/workspaces.ts backend/src/routes/sessions.ts backend/src/app.ts \
|
||
backend/test/app.test.ts backend/src/pi/pi-process-manager.ts \
|
||
backend/test/workspace-preprocessing-state.test.ts \
|
||
backend/test/workspace-preprocessing-service.test.ts backend/test/workspace-maintenance.test.ts \
|
||
backend/test/workspace-reader-lease.test.ts backend/test/routes-workspaces.test.ts \
|
||
backend/test/routes-sessions.test.ts backend/test/pi-process-manager.test.ts \
|
||
harness/tht/locked_child_stdin.py harness/tht/layout_markers.py \
|
||
harness/tht/cli/config_cmd.py harness/tht/cli/vector_cmd.py harness/tht/semantic_migration.py \
|
||
harness/tests/test_locked_child_stdin.py harness/tests/test_layout_marker_commands.py \
|
||
harness/tests/test_p3_internal_cli.py harness/tests/test_semantic_revision_migration.py \
|
||
harness/tests/test_qdrant_cli_commands.py \
|
||
tools/thothctl/internal/workspaceops/operations.go \
|
||
tools/thothctl/internal/workspaceops/operations_test.go \
|
||
tools/thothctl/cmd/thothctl/main.go tools/thothctl/cmd/thothctl/main_test.go \
|
||
deploy/compose.git-https.yaml deploy/compose.git-ssh.yaml \
|
||
scripts/generate-connector-secrets-override.sh scripts/test-preprocess-compose-config.sh
|
||
git commit -m "feat: add exact addressed registry pull and resumable maintenance"
|
||
```
|
||
---
|
||
|
||
### Task 9: Prepare one exact binding generation per commit and activate the revision layout
|
||
|
||
**Files:**
|
||
- Modify: `backend/src/workspaces/revision-layout.ts`
|
||
- Modify: `backend/src/workspaces/runtime-config-lease.ts`
|
||
- Modify: `backend/src/workspaces/runtime-renderer.ts`
|
||
- Modify: `backend/src/tht/tht-runner.ts`
|
||
- Modify: `backend/src/workspaces/preprocessing-state.ts`
|
||
- Modify: `backend/src/workspaces/preprocessing-service.ts`
|
||
- Modify: `backend/src/workspace-maintenance.ts`
|
||
- Reuse unchanged from Task 8: `backend/src/workspaces/workspace-reader-lease.ts`
|
||
- Modify: `backend/src/pi/pi-process-manager.ts`
|
||
- Modify: `backend/test/workspace-revision-layout.test.ts`
|
||
- Modify: `backend/test/workspace-runtime-renderer.test.ts`
|
||
- Modify: `backend/test/workspace-runtime-config-lease.test.ts`
|
||
- Modify: `backend/test/workspace-runtime-handoff.test.ts`
|
||
- Create: `backend/test/workspace-effective-config-equivalence.test.ts`
|
||
- Modify: `backend/test/workspace-preprocessing-service.test.ts`
|
||
- Modify: `backend/test/workspace-maintenance.test.ts`
|
||
- Modify: `backend/test/workspace-reader-lease.test.ts`
|
||
- Modify: `backend/test/pi-process-manager.test.ts`
|
||
- Modify: `harness/tht/layout_markers.py`
|
||
- Modify: `harness/tht/cli/config_cmd.py`
|
||
- Modify: `harness/tests/test_layout_marker_commands.py`
|
||
- Modify: `harness/tests/test_p3_internal_cli.py`
|
||
- Modify: `harness/tht/cli/schema_cmd.py`
|
||
- Modify: `harness/tht/cli/lsh_cmd.py`
|
||
- Modify: `harness/tht/cli/preprocess_cmd.py`
|
||
- Modify: `harness/tht/jobs/dwh_pipeline.py`
|
||
- Modify: `harness/tht/taskdoc.py`
|
||
- Modify: `harness/tht/search/evidence.py`
|
||
- Modify: `harness/tht/cli/memory_cmd.py`
|
||
- Modify: `harness/tht/memory.py`
|
||
- Modify: `harness/tht/solved.py`
|
||
- Modify: `harness/tests/test_dwh_snapshot.py`
|
||
- Modify: `harness/tests/test_dwh_owner_migration.py`
|
||
- Modify: `harness/tests/test_dwh_preprocess_job.py`
|
||
- Modify: `harness/tests/test_lsh_job_resume.py`
|
||
- Modify: `harness/tests/test_search_pack.py`
|
||
- Modify: `harness/tests/test_schema_fk_annotations.py`
|
||
- Modify: `harness/tests/test_qdrant_cli_commands.py`
|
||
- Modify: `harness/tests/test_corpus_pipeline.py`
|
||
- Modify: `harness/tests/test_memory_migration.py`
|
||
- Modify: `harness/tests/test_memory_promotion.py`
|
||
- Modify: `harness/tests/test_memory_save_one.py`
|
||
- Modify: `harness/tests/test_solved_search_cli.py`
|
||
- Modify: `tools/thothctl/internal/workspaceops/operations.go`, `operations_test.go`
|
||
- Modify: `tools/thothctl/cmd/thothctl/main.go`, `main_test.go`
|
||
|
||
**Step 1: Write the RED complete-consumer, admission, and transaction tests.**
|
||
|
||
Add the exact released host command:
|
||
|
||
```text
|
||
thothctl --installation <abs> workspace migrate activate-revision-layout
|
||
--workspace <id> --yes [--resume <32-hex-outer-run-id>] [--json]
|
||
```
|
||
|
||
Fresh pre-generates one outer run ID and launches Task 8's single dedicated job with the canonical
|
||
create request. The one-shot atomically creates or exactly replays that ID/digest and can produce no
|
||
second run. Every retry after a confirmed nonterminal result uses that exact `--resume`; an ambiguous
|
||
host result exposes the generated ID with `run_durability_unconfirmed`. Freeze durable phases:
|
||
|
||
```text
|
||
run_addressed -> quiescence_published -> reader_gate_exclusive -> effective_binding_probed ->
|
||
future_paths_bound -> dwh_cache_prepared -> dwh_snapshot_ready -> memory_ready ->
|
||
semantic_inventory_persisted -> semantic_sources_ready ->
|
||
semantic_replacements_published -> semantic_replacements_verified ->
|
||
layout_version_published -> legacy_delete_complete -> ready_published ->
|
||
terminal_durable -> quiescence_owner_cleared
|
||
```
|
||
|
||
Transitions are closed, monotonic, and artifact-digest bound. Resume re-verifies a phase before
|
||
skipping it. It must resume after a crash during reader drain, partial legacy deletion, either atomic
|
||
marker rename, terminal-state fsync, and before owner-qualified quiescence clear, including while
|
||
`core` is unavailable.
|
||
|
||
Initial revision A plus effective binding X requires all of these before its READY publication:
|
||
|
||
1. matching durable quiescence/run ownership, safe bounded session inventory, and an exclusive
|
||
`session-readers.lock` lease proving zero active readers while new admission is blocked;
|
||
2. P2 writer lock held continuously inside maintenance and active revision/descriptor/config
|
||
revalidated;
|
||
3. fixed read-only harness binding probe returns X, and every later child/result repeats X's binding
|
||
SHA/cache key;
|
||
4. matching schema-v1 DWH is migrated, or otherwise prepare-mode DWH builds, then strictly reverifies
|
||
the trusted X cache and materializes the exact A/X binding-qualified snapshot;
|
||
5. canonical Memory is migrated/reverified at the future global root;
|
||
6. exact legacy semantic inventory and source-readiness bytes are persisted, then all A replacements
|
||
are published and read back while the exclusive reader lease remains held;
|
||
7. global `layout-version.json` is published atomically (or strictly reverified if present);
|
||
8. only then, exact digest-confirmed legacy semantic deletion completes;
|
||
9. exact `revisions/A/readiness/X/READY.json` is exclusively published last and strictly re-read.
|
||
|
||
The layout marker never names A or X. For later revision B, released `workspace registry pull`
|
||
publishes B through P2's repository-first addressed callback and its complete changed-set capability set. With layout v1 already global, B/X
|
||
has no READY: the fixed binding probe selects X and session admission fails `migration_required`, while
|
||
prepare maintenance returns only trusted B/X paths. Activation reuses the verified X cache, creates a
|
||
B/X snapshot and readiness generation without rewriting the marker; A/X and B/X coexist.
|
||
|
||
Also test a changed installation binding Y while commit B is already READY for X. The session probe
|
||
selects Y, refuses B/X cache/snapshot/READY, and fails `migration_required`; activation securely finds
|
||
the existing B/X READY and returns `effective_config_mismatch` **before** DWH prepare/migration,
|
||
snapshot writes, semantic inventory/publish/delete, or marker/READY writes. B/X READY, snapshot,
|
||
semantic points, and owner remain byte-identical. The fixture then makes and publishes content-only
|
||
commit C and invokes released registry pull. C/Y is unready and isolated; activation now chooses the
|
||
Task 5 prepare-mode DWH build (not legacy migration or X fallback), publishes/verifies the new
|
||
schema-v2 Y cache and C/Y snapshot, publishes/read-backs C semantic replacements while quiesced, and
|
||
exclusively creates `revisions/C/readiness/Y/READY.json`. With Y still selected, C/Y is admitted and
|
||
B admission remains intentionally refused; B/X READY, snapshot, semantic points, and owner bytes remain
|
||
unchanged but revision pinning alone does not restore binding X. The test then explicitly restores the
|
||
installation binding to X, probes X and admits historical B/X; restores the installation binding to Y,
|
||
probes Y and admits C/Y. Repeat the refusal -> newly published successor commit -> changed-binding
|
||
activation for endpoint, transport, database, schema, and one included policy mutation, using a
|
||
distinct fresh revision for every newly selected binding; no test may activate B/Y or any second READY
|
||
binding at one revision.
|
||
|
||
Inventory every physical/LSH consumer and fail the test if it references `.tht-dwh`,
|
||
`artifacts/mschema/physical.yaml`, or `indexes/lsh` outside `dwh_snapshot.py`. Explicitly exercise
|
||
`schema_cmd.physical_path`, schema introspection/check/index, `lsh_cmd` build/read, preprocess DWH,
|
||
`dwh_pipeline`, `taskdoc`/search-pack generation, and mschema readers through
|
||
`resolve_revision_dwh_snapshot`. Exercise all Memory commands through `memory_root/registry_path` and
|
||
Evidence preprocessing/search through explicit `paths.corpus`.
|
||
|
||
Add fault tests at every phase and prove throughout the publish/delete/READY interval that the durable
|
||
marker blocks the new-admission race, the exclusive reader gate proves all shared readers drained,
|
||
persistent inventory was accepted, and no session child remains. Replacement publish/readback must
|
||
precede the first legacy delete. READY must follow the final delete. On failure/SIGKILL the OS locks
|
||
release but durable owner state and quiescence remain, sessions stay refused, and only exact same-ID
|
||
resume continues even with dead `core`. No failure clears quiescence or rolls back to P2 once the
|
||
global marker exists; wrong owner/digest can neither resume nor clear.
|
||
|
||
Run:
|
||
|
||
```bash
|
||
(cd backend && npx vitest run \
|
||
test/workspace-revision-layout.test.ts \
|
||
test/workspace-runtime-renderer.test.ts \
|
||
test/workspace-runtime-config-lease.test.ts \
|
||
test/workspace-runtime-handoff.test.ts \
|
||
test/workspace-effective-config-equivalence.test.ts \
|
||
test/workspace-preprocessing-service.test.ts \
|
||
test/workspace-maintenance.test.ts test/workspace-reader-lease.test.ts \
|
||
test/routes-sessions.test.ts test/pi-process-manager.test.ts)
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_dwh_snapshot.py tests/test_dwh_owner_migration.py \
|
||
tests/test_dwh_preprocess_job.py tests/test_lsh_job_resume.py tests/test_search_pack.py \
|
||
tests/test_schema_fk_annotations.py tests/test_qdrant_cli_commands.py \
|
||
tests/test_corpus_pipeline.py tests/test_memory_migration.py \
|
||
tests/test_memory_promotion.py tests/test_memory_save_one.py tests/test_solved_search_cli.py)
|
||
(cd tools/thothctl && test "$(go env GOVERSION)" = "go1.26.5" && \
|
||
go test ./internal/workspaceops ./cmd/thothctl -run 'Activate|RegistryPull|Resume|Quiesc|Reader|DedicatedJob' -v)
|
||
```
|
||
|
||
Expected: RED on activation, readiness gating, and remaining direct paths.
|
||
|
||
**Step 2: Implement synchronous marker-gated session rendering and trusted prepare rendering.**
|
||
|
||
Keep P2's exact synchronous signatures:
|
||
|
||
```ts
|
||
export class WorkspaceRuntimeConfigLeaseFactory {
|
||
acquireSession(snapshotPath: string): RuntimeConfigLease;
|
||
acquireMaintenance(input: MaintenanceRuntimeInput): RuntimeConfigLease;
|
||
}
|
||
```
|
||
|
||
Extend `MaintenanceRuntimeInput` with exact `layoutIntent: "active" |
|
||
"prepare-revision-layout-v1"`; keep both factory signatures synchronous and do not add a TypeScript
|
||
effective-DWH canonicalizer. In layout v1 the returned `RuntimeConfigLease` begins as a bounded
|
||
binding-probe lease: its YAML has the exact descriptor/DWH inputs and trusted future base roots, may be
|
||
used only for fixed `tht config effective-dwh --json -c CONFIG`, and cannot spawn Pi or any mutating
|
||
child. The existing async `ThtRunner` startup/coordinator path runs that read-only probe with 65,536
|
||
stdout bytes, 16,384 stderr bytes and a 30-second timeout, validates the exact `EffectiveDwhBinding`
|
||
and key, then calls a new synchronous lease method
|
||
`selectEffectiveDwh(binding): BoundRuntimeConfigLease`. That method invokes
|
||
`readRevisionLayoutState(..., effectiveDwhCacheKey)` and
|
||
`bindingQualifiedWorkspaceLayoutPaths`; it never selects newest/only READY. This adds no alternate
|
||
factory entrypoint and preserves P2's exact `acquireSession(snapshotPath)` and
|
||
`acquireMaintenance(input)` signatures.
|
||
|
||
Session behavior after selection is exact:
|
||
|
||
- no global marker: byte-identical P2 roots and normal P2 handoff;
|
||
- valid marker + exact commit+binding READY: render its binding-qualified cache/snapshot and revision
|
||
roots, reverify READY identity, then and only then spawn Pi/session work;
|
||
- valid marker + absent exact READY: `migration_required` before Pi/session or semantic child spawn,
|
||
even if another READY exists for that commit;
|
||
- invalid marker/selected READY or binding mismatch: fail closed.
|
||
|
||
Active maintenance follows the same rule. Prepare maintenance is the only exception: before READY and
|
||
only after the secure directory guard proves that revision has no READY for another binding, it uses
|
||
the probed binding key with the trusted resolvers and renders only:
|
||
|
||
```text
|
||
sessions = /data/sessions/<id>/sessions
|
||
memory = /data/sessions/<id>/memory
|
||
dwh_cache = /data/sessions/<id>/preprocessing/dwh-cache/<binding-key>
|
||
dwh_snapshot = /data/sessions/<id>/revisions/<revision>/dwh-snapshots/<binding-key>
|
||
ready = /data/sessions/<id>/revisions/<revision>/readiness/<binding-key>/READY.json
|
||
artifacts = /data/sessions/<id>/revisions/<revision>/artifacts
|
||
indexes = /data/sessions/<id>/revisions/<revision>/indexes
|
||
corpus = /data/sessions/<id>/revisions/<revision>/corpus
|
||
```
|
||
|
||
It never consults P2 `memory`, `.tht-dwh`, artifacts, indexes, corpus, another binding cache/snapshot,
|
||
or a sibling READY as a destination. Tests prove the probe lease cannot escape to a session/mutating
|
||
spawn and that session and maintenance selection use byte-identical binding-qualified YAML.
|
||
|
||
**Step 3: Implement the quiesced activation transaction in the flat coordinator.**
|
||
|
||
`workspaceops.Run` launches only Task 8's dedicated job. Inside that one-shot,
|
||
`WorkspacePreprocessingService.execute` obtains `CanonicalWorkspaceLockRootInput` from the sole P2
|
||
factory, uses existing-root `acquire`, transfers the exact `VerifiedWorkspaceLockRootLease`, and enters
|
||
one `runUnderWorkspaceWriterLock(rootLease, async (writerCapability) => ...)` action **before** ordinary
|
||
job or quiescence. Inside that still-live writer callback, publish the durable owner marker, drain
|
||
readers, then call Task 8's exact
|
||
`readerLeaseFactory.acquireForMaintenance(writerCapability, async (readerBorrow) => ...)`. The complete
|
||
quiesced activation—including final recheck, all children/publication, terminal durability, and matching
|
||
owner clear—settles inside that callback; `readerBorrow` never escapes. The writer action consumes and
|
||
closes the root only after the adapter has returned and P2 has invalidated/closed the exclusive reader.
|
||
Each fixed internal migration request runs only through `writerCapability.spawnChild`; thus the actual locked open file
|
||
description is child FD 3 and the same retained root description is FD 4; neither is reopened. Revalidate target snapshot and inventory at
|
||
every child boundary.
|
||
|
||
Activation is the sole semantic cutover and one continuous quiesced transaction. It binds the exact
|
||
commit+effective key, then **before DWH/Memory/semantic mutation** securely enumerates the revision's
|
||
readiness directory. An empty directory permits preparation; the selected READY may only byte-match
|
||
for idempotent resume; one READY at another key returns `effective_config_mismatch`; multiple/unsafe
|
||
entries fail closed. Only after this guard does activation migrate or prepare-build/reverify the cache
|
||
and binding snapshot; reuse/recheck Memory and persisted semantic inventory/readiness; publish all
|
||
replacement points and read back their exact identities/digests; atomically publish/reverify the
|
||
global marker for initial enablement; delete only unchanged exact legacy inventory IDs; and
|
||
exclusively publish the selected commit+binding READY last. For later **revisions** it strictly verifies
|
||
the unchanged global marker and does not broaden deletion. All exclusive files use sibling write +
|
||
file fsync + rename + parent fsync and synchronous no-follow reread. Different bytes at the same
|
||
`(commit,key)` fail closed; a changed binding never writes a sibling generation for that commit and
|
||
must use a newly published revision.
|
||
|
||
Only after READY and terminal run state are durably re-read does the same one-shot owner remove its
|
||
byte-matching quiescence marker while still holding the exclusive reader gate, then release the gate.
|
||
No HTTP/backend acknowledgment and no core lifecycle action exists. Failure before the global marker
|
||
leaves P2 rendering but stays durably quiesced; failure after the marker leaves the revision unready
|
||
and stays quiesced. Process death releases OS locks but exact same-ID resume is the only recovery path.
|
||
|
||
**Step 4: Switch every consumer only through the shared resolvers.**
|
||
|
||
Remove every direct path found by Step 1. P3 configs require a valid revision snapshot; legacy configs
|
||
without `dwh_snapshot` retain their compatibility branch only when the global marker is absent.
|
||
Schema/LSH reads never probe cache/old roots after layout enablement. Memory and corpus use explicit
|
||
config paths. Schema/Evidence vector operations select strict revision scope only from a ready P3
|
||
config.
|
||
|
||
**Step 5: Prove A to B transition and effective-binding equivalence.**
|
||
|
||
For A/B with identical DWH inputs, run `tht config effective-dwh --json` against session and
|
||
maintenance leases and assert byte-identical binding/cache key, distinct revision roots, and stable
|
||
`workspace://<id>`. Use the released command, not raw Git or the backend route:
|
||
|
||
```bash
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace registry pull \
|
||
--workspace "$WORKSPACE" --json
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace migrate dwh-cache \
|
||
--workspace "$WORKSPACE" --json
|
||
```
|
||
|
||
Assert B/X session admission is `migration_required` before B/X READY. The released `dwh-cache`
|
||
command resolves only B/X, materializes/reverifies the binding-qualified B/X snapshot for pre-READY
|
||
preparation/P5 validation, and proves no layout marker, READY, or semantic point changed; then activate
|
||
and admit B/X. Then, without changing commit B, change effective binding to Y:
|
||
admission fails and activation returns `effective_config_mismatch` before
|
||
`p3_prepare_dwh_cache`, cache/snapshot/semantic/marker writes, or READY publication. Publish and pull
|
||
content-only revision C, prove C/Y is unready, then run the full released activation path:
|
||
`p3_prepare_dwh_cache` creates the new v2 Y owner/cache, C/Y snapshot, semantic verification, and
|
||
immutable C/Y READY. With Y selected, admit C/Y and continue refusing B; prove B/X bytes unchanged.
|
||
Explicitly restore installation binding X and admit B/X, then restore Y and admit C/Y. Repeat the
|
||
complete refusal -> fresh successor revision -> new-binding activation path for endpoint, transport,
|
||
database, schema, and one included policy, never reusing an already-READY revision for the next
|
||
binding. Hold the registry addressed callback-scoped ordered set open to prove every changed-workspace writer
|
||
capability contends without reacquisition; hold each ordinary child open for its selected writer lock;
|
||
hold multiple session reader leases to prove drain, and rely on durable admission blocking plus the exclusive reader gate for
|
||
semantic cutover while `core` may keep running.
|
||
|
||
**Step 6: Run GREEN activation and full P2 regression gates.**
|
||
|
||
```bash
|
||
(cd backend && npx vitest run \
|
||
test/workspace-revision-layout.test.ts \
|
||
test/workspace-runtime-renderer.test.ts \
|
||
test/workspace-runtime-config-lease.test.ts \
|
||
test/workspace-runtime-handoff.test.ts \
|
||
test/workspace-effective-config-equivalence.test.ts \
|
||
test/workspace-preprocessing-state.test.ts \
|
||
test/workspace-preprocessing-service.test.ts \
|
||
test/workspace-maintenance.test.ts test/workspace-reader-lease.test.ts \
|
||
test/routes-sessions.test.ts test/pi-process-manager.test.ts test/tht-runner.test.ts && npx tsc --noEmit -p . && npm run build)
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_effective_dwh_binding.py tests/test_dwh_snapshot.py tests/test_dwh_owner_v2.py \
|
||
tests/test_dwh_owner_migration.py tests/test_dwh_preprocess_job.py \
|
||
tests/test_locked_child_stdin.py tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py \
|
||
tests/test_lsh_job_resume.py tests/test_search_pack.py tests/test_schema_fk_annotations.py \
|
||
tests/test_qdrant_vector_store.py tests/test_semantic_kind_isolation.py \
|
||
tests/test_qdrant_cli_commands.py tests/test_corpus_pipeline.py \
|
||
tests/test_memory_migration.py tests/test_memory_promotion.py \
|
||
tests/test_memory_save_one.py tests/test_solved_question.py tests/test_solved_search_cli.py)
|
||
(cd tools/thothctl && test "$(go env GOVERSION)" = "go1.26.5" && go test ./...)
|
||
./scripts/p2-acceptance.sh integration --keep
|
||
```
|
||
|
||
Expected: every focused gate passes; fresh P2 acceptance still passes before global enablement;
|
||
A/X, B/X, same-revision Y refusal, C/Y READY, explicit X restore/B admission, and Y restore/C
|
||
admission tests pass. New admission cannot cross quiescence publication; semantic delete and READY
|
||
publication occur only under the exclusive reader gate, regardless of whether `core` is running.
|
||
|
||
**Step 7: Commit.**
|
||
|
||
```bash
|
||
git add backend/src/workspaces/revision-layout.ts \
|
||
backend/src/workspaces/runtime-config-lease.ts backend/src/workspaces/runtime-renderer.ts \
|
||
backend/src/tht/tht-runner.ts backend/src/workspaces/preprocessing-state.ts \
|
||
backend/src/workspaces/preprocessing-service.ts backend/src/workspace-maintenance.ts \
|
||
backend/src/pi/pi-process-manager.ts \
|
||
backend/test/workspace-revision-layout.test.ts backend/test/workspace-runtime-renderer.test.ts \
|
||
backend/test/workspace-runtime-config-lease.test.ts backend/test/workspace-runtime-handoff.test.ts \
|
||
backend/test/workspace-effective-config-equivalence.test.ts \
|
||
backend/test/workspace-preprocessing-service.test.ts backend/test/workspace-maintenance.test.ts \
|
||
backend/test/workspace-reader-lease.test.ts backend/test/pi-process-manager.test.ts \
|
||
harness/tht/cli/schema_cmd.py harness/tht/cli/lsh_cmd.py harness/tht/cli/preprocess_cmd.py \
|
||
harness/tht/jobs/dwh_pipeline.py harness/tht/taskdoc.py harness/tht/search/evidence.py \
|
||
harness/tht/cli/memory_cmd.py harness/tht/memory.py harness/tht/solved.py \
|
||
harness/tht/layout_markers.py harness/tht/cli/config_cmd.py \
|
||
harness/tests/test_layout_marker_commands.py harness/tests/test_p3_internal_cli.py \
|
||
harness/tests/test_dwh_snapshot.py harness/tests/test_dwh_owner_migration.py \
|
||
harness/tests/test_dwh_preprocess_job.py harness/tests/test_lsh_job_resume.py \
|
||
harness/tests/test_search_pack.py harness/tests/test_schema_fk_annotations.py \
|
||
harness/tests/test_qdrant_cli_commands.py harness/tests/test_corpus_pipeline.py \
|
||
harness/tests/test_memory_migration.py harness/tests/test_memory_promotion.py \
|
||
harness/tests/test_memory_save_one.py harness/tests/test_solved_search_cli.py \
|
||
tools/thothctl/internal/workspaceops/operations.go \
|
||
tools/thothctl/internal/workspaceops/operations_test.go \
|
||
tools/thothctl/cmd/thothctl/main.go tools/thothctl/cmd/thothctl/main_test.go
|
||
git commit -m "feat: activate revision-isolated workspace layout"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 10: Document `.tht-dwh`, migration, recovery, and the P3 manual walkthrough
|
||
|
||
**Files:**
|
||
- Modify: `docs/install/local-workspace-registry.md`
|
||
- Modify: `docs/install/server-workspace-registry.md`
|
||
- Modify: `docs/testing/p2-p6-manual-verification.md`
|
||
- Create: `docs/architecture/effective-dwh-cache.md`
|
||
- Modify: `docs/index.md`
|
||
|
||
**Step 1: Write documentation contract tests first.**
|
||
|
||
Create or extend `scripts/test-verify-workspace-install-docs.sh` assertions requiring both manuals and the architecture page to name:
|
||
|
||
- effective config vs input fingerprint;
|
||
- logical source identity and excluded revision/temp filename/secrets;
|
||
- `.tht-dwh`, immutable `generations/`, `OWNER.json` v1/v2, `ACTIVE`;
|
||
- cache key, revision snapshot roots, and why mismatch fails closed;
|
||
- exact inspect/registry-pull/migrate/`--resume`/preprocess/reindex/recovery commands, including that explicit `--resume` is for pull/migration while empty-registry bootstrap recovery is automatic through inspect, lazy list, and status;
|
||
- the repo-owned Node-API v8 `workspace-fs-at` seam for anchored missing-root provisioning, its closed typed writer/reader lock-name and flock surface, its Darwin/Linux durability behavior, and that exact `fs-ext@2.1.1` is wrapper-internal and `flock(2)` only with no P3 raw FD/import/cast/path fallback;
|
||
- client-generated addressed run IDs/digests, the pull create/resume identity/base fields derived inside the validated boundary, bounded exact-identity `ensureBootstrapAddressed` active-state/zero/one/multiple/corrupt rules (including the `already_active` no-network/no-state branch) and `registry_bootstrap_recovery_conflict`, no-follow durable owner quiescence, shared session reader leases, exclusive drain, dead-core same-ID resume, and owner-only clear;
|
||
- global layout-version marker vs immutable commit+effective-binding READY, with at most one READY binding per revision and a required new content commit for changed binding;
|
||
- Memory canonical JSONL vs Qdrant projection;
|
||
- backup scope for cache, revisions, corpus, Memory, Qdrant, and registry;
|
||
- warning not to edit OWNER/ACTIVE or copy an unverified generation manually.
|
||
|
||
Run:
|
||
|
||
```bash
|
||
./scripts/test-verify-workspace-install-docs.sh
|
||
```
|
||
|
||
Expected: RED on missing P3 content.
|
||
|
||
**Step 2: Write operator-facing architecture and recovery content.**
|
||
|
||
Explain safe recovery choices: inspect; inspect/lazy list/status always enter the continuously repository-locked selector, return its validated `already_active` snapshot without job scan/network/state when active state exists (including queued callers after another caller publishes), and only for validated absence automatically resume the sole exact-identity nonterminal bootstrap or create one, while incompatible active state or multiple, pull, mismatched, corrupt, churning, or over-bound job sets stop with `registry_bootstrap_recovery_conflict` and no operator-selected run ID; preserve legacy filesystem sources; run non-activating DWH/Memory preparation; use `semantic-revision` only to inventory legacy points and prove rebuild readiness, explicitly documenting that it writes no replacement Qdrant point and deletes nothing. `workspace migrate dwh-cache` must be documented as pre-READY materialization of the selected cache plus binding-qualified pulled-revision snapshot, usable before P5 acceptance but incapable of publishing layout/READY or admitting that revision. Perform replacement publish/readback, global reader-mode switch, exact legacy deletion, and immutable commit+binding READY only in one addressed one-shot run after durable admission blocking and exclusive reader-gate acquisition. The CLI never calls Fastify/HTTP or stops `core`; blocked admission plus zero shared readers makes publication safe while `core` may run. Use the client-generated run ID with exact `--resume` after interruption or ambiguous response, including dead-core recovery; SIGKILL releases OS locks but retains owner marker/run state. Prepare a changed binding only after publishing/pulling a new content commit; never activate a second binding READY for one revision. While Y is selected, B/X bytes remain immutable but B admission is refused; historical B/X is demonstrated only by restoring installation X, then Y is restored for C/Y. Rebuild Memory projection from canonical JSONL; never hand-clear quiescence or weaken digest checks. Evidence that cannot be rebuilt stays `migration_required` and its legacy points are not deleted. A global marker without the current exact commit+binding READY intentionally blocks sessions while prepare mode remains available only if that revision has no other READY binding. State that schema-v1 remains readable only to verify/migrate, not writable by P3. Explain that absent changed-set roots are provisioned only by P2's owned `WorkspaceFsAtV1` Node-API seam using literal FD-relative syscalls, with Linux directory-fsync success and Darwin success-or-exact fail-closed behavior. For readers, P3 calls only `VerifiedWorkspaceLockRootLease.acquireSessionReadersShared()` and `WorkspaceWriterLockCapability.runUnderSessionReadersExclusive(...)` through the zero-filesystem adapter; it never opens/flocks directly or receives a wrapper, directory handle, path, numeric FD, lock name, flags, or mode, and exact `fs-ext@2.1.1` remains P2-wrapper-internal and `flock(2)` only.
|
||
|
||
**Step 3: Replace the P3 placeholder in the living manual with an independently runnable clean walkthrough.**
|
||
|
||
Use variables `INSTALLATION`, `THOTHCTL`, `WORKSPACE`, and a new fixture/private Git remote. Include these exact operator calls:
|
||
|
||
```bash
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace inspect \
|
||
--workspace "$WORKSPACE" --json
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace migrate dwh-cache \
|
||
--workspace "$WORKSPACE" --json
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace migrate memory \
|
||
--workspace "$WORKSPACE" --json
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace migrate semantic-revision \
|
||
--workspace "$WORKSPACE" --yes --json
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace migrate activate-revision-layout \
|
||
--workspace "$WORKSPACE" --yes --json
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace preprocess dwh \
|
||
--workspace "$WORKSPACE" --json
|
||
|
||
# After committing/pushing content-only B, use the released product pull, never raw registry mutation:
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace registry pull \
|
||
--workspace "$WORKSPACE" --json
|
||
# Pre-READY: materialize the pulled revision's selected cache/snapshot; do not activate READY.
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace migrate dwh-cache \
|
||
--workspace "$WORKSPACE" --json
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace migrate activate-revision-layout \
|
||
--workspace "$WORKSPACE" --yes --json
|
||
|
||
# Exact crash recovery pattern: export RUN_ID from the returned 32-hex runId first.
|
||
RUN_ID="${RUN_ID:?export RUN_ID as the returned 32-hex outer run ID}"
|
||
[[ "$RUN_ID" =~ ^[0-9a-f]{32}$ ]]
|
||
"$THOTHCTL" --installation "$INSTALLATION" workspace migrate activate-revision-layout \
|
||
--workspace "$WORKSPACE" --yes --resume "$RUN_ID" --json
|
||
```
|
||
|
||
Then walk the reviewer through:
|
||
|
||
1. during the initial empty-registry bootstrap, inject one preterminal death for each of inspect, lazy list, and status, then prove the next actual caller automatically resumes the same exact-identity run before network; prove multiple/mismatched/corrupt jobs stop with `registry_bootstrap_recovery_conflict`; after terminal recovery, record revision A and compare the safe operator/session effective binding;
|
||
2. inspect migrated v2 OWNER, ACTIVE, binding-qualified immutable snapshot manifests/digests, the workspace-global `layout-version.json`, and exact A/binding `readiness/<key>/READY.json` without printing config/secrets;
|
||
3. after each non-activating command, prove owner-clear permits a session that sees no mixed legacy/replacement state; specifically prove `dwh-cache` materializes the selected binding-qualified snapshot but no READY/layout/semantic bytes, and `semantic-revision` makes zero Qdrant upsert/delete calls;
|
||
4. hold two live sessions with shared reader leases, start activation, prove the durable marker refuses a racing new admission, let both readers drain without killing them, and prove publication starts only after the exclusive reader lease establishes zero active readers; keep `core` running and show no Fastify/HTTP or stop/start command occurs;
|
||
5. inject SIGKILL during partial deletion and after READY rename, make `core` unavailable, then use the returned exact `--resume` ID to reach success without re-embedding or broad deletion; prove the OS lock released, the durable owner marker remained, and wrong owner/digest could not clear it;
|
||
6. commit/push content-only revision B, invoke released `workspace registry pull`, run `workspace migrate dwh-cache`, prove the B/X cache and binding-qualified snapshot exist pre-READY and are usable for later P5 validation while B admission still fails, then activate and prove the same cache generation but distinct revision roots and schema/Evidence points;
|
||
7. prove A/X READY remains, and Memory/solved results remain visible at B with global identity;
|
||
8. hold activation/mutation open and prove concurrent registry publication is refused; no existing reader is killed to reach exclusive ownership;
|
||
9. while commit B is READY for X, change the fixture binding to Y; prove B admission and same-revision activation are refused with zero mutation while B/X bytes remain immutable. Publish/pull C, run pre-READY `dwh-cache`, activate C/Y, and admit C/Y. Then explicitly restore installation X and admit B/X; restore Y and admit C/Y. Repeat on distinct successor revisions for endpoint, transport, database, schema, and one included policy;
|
||
10. create a conflicting legacy Memory registry, prove durable owner quiescence, remove only the conflict, and resume the same ID;
|
||
11. scan captured output for fixture secret canaries and perform exact cleanup.
|
||
|
||
For every step explain component, state read, artifact produced, invariant, evidence to retain, and PASS/FAIL criterion. Set `automated integration: PENDING` until Task 12; keep `manual acceptance: PENDING` and `Decision: PENDING` until reviewer action. The manual environment must not reuse automated state.
|
||
|
||
**Step 4: Run docs checks.**
|
||
|
||
```bash
|
||
./scripts/test-verify-workspace-install-docs.sh
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
**Step 5: Commit.**
|
||
|
||
```bash
|
||
git add docs/install/local-workspace-registry.md docs/install/server-workspace-registry.md \
|
||
docs/testing/p2-p6-manual-verification.md docs/architecture/effective-dwh-cache.md \
|
||
docs/index.md scripts/test-verify-workspace-install-docs.sh
|
||
git commit -m "docs: explain effective DWH cache ownership"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 11: Build the clean-state P3 acceptance process and self-tests
|
||
|
||
**Files:**
|
||
- Create: `scripts/p3-acceptance.sh`
|
||
- Create: `scripts/test-p3-acceptance.sh`
|
||
- Create: `backend/scripts/p3-acceptance.mjs`
|
||
- Modify: `.gitignore` only if `.artifacts/p3-effective-config/` is not already covered
|
||
|
||
**Step 1: Write RED acceptance-harness tests.**
|
||
|
||
`test-p3-acceptance.sh` must test, without requiring the full expensive success run:
|
||
|
||
- exact CLI grammar: `integration [--keep]` and `cleanup --run <id>`;
|
||
- dirty tracked source refusal before build/start;
|
||
- unique run/project IDs and exclusive ownership manifest creation;
|
||
- no retry loop or recursive self-invocation;
|
||
- trap cleanup on pre-service and post-service injected failures;
|
||
- refusal to clean an unowned/mismatched project, volume, network, path, or symlink;
|
||
- `--keep` retains owned resources and reports; cleanup removes owned resources only and is idempotent;
|
||
- report schema and finalization on PASS and injected FAIL;
|
||
- secret canaries absent from stdout/stderr/report/public retained files;
|
||
- Qdrant/Ollama listener shutdown and Compose resource absence after normal cleanup.
|
||
|
||
Run:
|
||
|
||
```bash
|
||
./scripts/test-p3-acceptance.sh
|
||
```
|
||
|
||
Expected: RED because the runner does not exist.
|
||
|
||
**Step 2: Implement the process runner.**
|
||
|
||
Follow the hardened P1/P2 runner conventions: trusted fixed toolchain discovery, sanitized environment, no shell-evaluated fixture data, bounded subprocess capture, exact ownership labels, raw-Git safety, single attempt, atomic report writes, SHA-256 artifact manifest, and signal-safe cleanup. Reuse P2 fixture setup functions rather than copying them if they are already factored into sourceable non-executable helpers.
|
||
|
||
The full process assertions are:
|
||
|
||
1. clean source commit/tree, exact `go1.26.5`, and no pre-existing owned resources;
|
||
2. revision A P2 schema-v1 fixture and legacy Memory/schema/Evidence state detected as `migration_required`, with byte-identical P2 session/maintenance roots;
|
||
3. the synchronous no-follow layout reader rejects symlinks/replacement/malformed state, and the trusted future resolver selects exact migration destinations without active fallback;
|
||
4. every released operation pre-generates a 32-hex ID and launches only the immutable-image-pinned, profile-gated `workspace-maintenance` Compose job with canonical 4,096-byte-bounded request stdin and bounded exact JSON result; no Fastify/HTTP/internal route, `compose exec core`, frontend, host Python/Node/Pi/`tht`, or core stop/start is invoked. The four ordinary operations each use the selected-workspace P2 existing-root/writer-first lifecycle: shared session ownership is acquired only by `acquireSessionReadersShared()` and transferred into `PiProcessManager` through all child/stream teardown, while the full quiesced maintenance action is enclosed only by `runUnderSessionReadersExclusive(...)` via the zero-filesystem adapter. `registry_pull` uses only the repository-first P2 addressed callback; its actual create request passes validated installation/repository/remote identities plus expected base and its resume request passes all three identities, with compile/runtime mismatch-before-network/state proof. It owns the complete lexical changed set, provisions absent roots and opens/flocks writer/reader gates only through P2's typed repo-owned `WorkspaceFsAtV1` seam (with native Linux/Darwin tests green, `fs-ext` wrapper-internal, and no P3 raw FD/import/cast/path factory), uses the exact addressed state artifact, and resumes the same ID by its durable pin phase while `core` is dead;
|
||
5. every P3 request variant is exhaustively built by the opaque capability with fixed argv, the canonical exact-key base64 stdin wire, strict Python parser, bounded result and the actual writer FD 3/retained-root FD 4 pair; every argv target, including all four marker/readiness commands, exists in the built image; maximum 716,800-byte inventory round-trips through the 1,048,576-byte collector; DWH/Memory retain sources and reverify destinations; arbitrary spawn/argv/path, malformed stdin, missing/substituted/cross-root/post-callback/nested/concurrent capability misuse, and SIGKILL fail safely;
|
||
6. after each successful non-activation command, terminal durability plus owner-only clear permits admission into one unmixed mode or expected `migration_required`; `dwh-cache` materializes the selected pre-READY cache and binding-qualified revision snapshot with zero READY/layout/semantic writes, and `semantic-revision` persists inventory/readiness with exactly zero replacement upserts/deletes;
|
||
7. durable marker publication blocks a racing admission; multiple existing shared reader leases drain without being killed; the exclusive no-follow OS lease proves zero active readers and is held while activation publishes/verifies A/X replacements, publishes the global reader marker, deletes only unchanged exact legacy IDs, and publishes immutable A/X READY last while `core` may remain running;
|
||
8. success durably records terminal state before the matching owner clears quiescence and releases exclusive ownership. SIGKILL releases OS locks but retains marker/run state; dead-core same-ID resume succeeds; wrong owner/digest and a second run cannot resume, mutate, or clear;
|
||
9. every schema/LSH/search-pack/mschema reader resolves the selected binding snapshot, every Memory command uses the global root, and no direct old-root/sibling-binding probe remains after enablement;
|
||
10. released registry pull alone receives writable registry/Git capability; repository lock precedes durable `request_claimed`, `target_advertised`, exact-OID `target_fetched`, and immutable exact A→B `planned`; complete changed IDs use P2 `acquireOrProvision` backed only by `WorkspaceFsAtV1` and one callback-scoped root/writer/quiescence/readers set lexically; bootstrap and pull callbacks remain field-for-field P2; inspect/lazy-list/status use only bounded `ensureBootstrapAddressed`, whose locked active recheck returns the exact `already_active` snapshot without scan/network/state and whose absent branch performs automatic zero/create or one/resume selection. A three-caller initial-absence race publishes exactly once and all callers converge; incompatible active state or pull/multiple/mismatched/corrupt/churning/over-bound job sets fail closed before network/state; participants cannot reenter; active publication is all-or-nothing; and B/X sessions fail before B/X READY while ordinary prepare resolves only B/X;
|
||
11. resumed B/X activation reuses X cache, publishes distinct revision state and immutable B/X READY, preserves A/X READY, and retains global Memory/solved retrieval;
|
||
12. ordinary marker-before-exclusive ordering, new-admission race refusal, multi-reader drain, writer-before-reader order, typed shared/exclusive reader locking, publish-before-delete, partial-delete/after-READY SIGKILL resume, dead-core recovery, and owner-only clear are proved. Separately, dead-core registry SIGKILL spans before/after claim, advertisement, immutable-ref fetch, `target_fetched`, active rename/fsync, and terminal-before-clear. Same-ID pre-advertisement recovery may re-advertise; post-advertisement recovery keeps the recorded OID; at/after `target_fetched` it never uses the network. For bootstrap, every preterminal death is resumed through each actual inspect/lazy-list/status caller by the sole matching exact-identity run before any fresh advertisement; queued callers recheck and return the resulting exact `already_active` snapshot; and every incompatible-active/ambiguity/corruption/bound failure is stable and network/state-free. Exact mixed base/target reconciles, a third installation/repository/remote identity, create base mismatch, or missing/changed pinned target is rejected before network/state, and terminal replay succeeds even if a later independent pull moved tracking refs;
|
||
13. after pull, `migrate dwh-cache` proves pre-READY cache plus binding-qualified snapshot materialization usable before P5 acceptance with no READY activation. At READY B/X, binding Y first refuses B admission and same-revision activation with zero mutation; then C is published/pulled, C/Y is prepared and activated, and C/Y is admitted while B/X bytes remain unchanged. Historical B/X is proved only by explicitly restoring installation X and admitting B/X, followed by restoring Y and admitting C/Y; transport, database, schema, and one root-affecting policy repeat on distinct successors with no second READY at one revision;
|
||
14. corrupted owner, conflicting Memory registry, caller namespace conflict, unsafe marker/READY, partial migration, wrong `--resume`, and interrupted activation fail closed with stable codes and no secret leakage;
|
||
15. safe reports pass secret scan and declared hashes; cleanup removes exactly owned Docker/filesystem state.
|
||
|
||
**Step 3: Run the runner self-tests.**
|
||
|
||
```bash
|
||
./scripts/test-p3-acceptance.sh
|
||
```
|
||
|
||
Expected: PASS.
|
||
|
||
**Step 4: Commit.**
|
||
|
||
```bash
|
||
git add scripts/p3-acceptance.sh scripts/test-p3-acceptance.sh backend/scripts/p3-acceptance.mjs .gitignore
|
||
git commit -m "test: add P3 clean-state process goal"
|
||
```
|
||
|
||
---
|
||
|
||
### Task 12: Focused verification, full clean process run, retained report, and checkpoint
|
||
|
||
**Files:**
|
||
- Create: `docs/reports/2026-08-10-p3-effective-config-checkpoint.md`
|
||
- Modify: `docs/testing/p2-p6-manual-verification.md`
|
||
- Modify: `PROJECT_STATE.md`
|
||
- Create: `scripts/lint-plan-shell-fences.py`
|
||
- Create: `scripts/test-lint-plan-shell-fences.py`
|
||
|
||
**Step 1: Verify the source is clean before the evidence run.**
|
||
|
||
```bash
|
||
git status --short
|
||
git rev-parse HEAD
|
||
git rev-parse 'HEAD^{tree}'
|
||
```
|
||
|
||
Expected: empty status and recorded commit/tree. If documentation/report changes are still uncommitted, commit them before running; the acceptance runner must not accept a dirty tracked tree.
|
||
|
||
**Step 2: Run all P3-focused gates.**
|
||
|
||
```bash
|
||
(cd harness && .venv/bin/pytest -q \
|
||
tests/test_effective_dwh_binding.py tests/test_dwh_snapshot.py tests/test_dwh_owner_v2.py \
|
||
tests/test_dwh_owner_migration.py tests/test_dwh_preprocess_job.py \
|
||
tests/test_config_resources.py tests/test_portable_paths.py \
|
||
tests/test_locked_child_stdin.py tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py \
|
||
tests/test_lsh_job_resume.py \
|
||
tests/test_search_pack.py tests/test_schema_fk_annotations.py \
|
||
tests/test_qdrant_vector_store.py tests/test_semantic_kind_isolation.py \
|
||
tests/test_semantic_revision_migration.py tests/test_qdrant_cli_commands.py \
|
||
tests/test_corpus_pipeline.py tests/test_memory_migration.py \
|
||
tests/test_memory_promotion.py tests/test_memory_save_one.py \
|
||
tests/test_solved_question.py tests/test_solved_search_cli.py \
|
||
tests/test_repository_memory_sql_paths.py)
|
||
(cd harness && .venv/bin/ruff check \
|
||
tht/effective_dwh.py tht/config.py tht/cli/config_cmd.py tht/jobs/dwh_pipeline.py \
|
||
tht/paths.py tht/dwh_snapshot.py tht/dwh_owner.py tht/dwh_migration.py \
|
||
tht/cli/preprocess_cmd.py tht/vectorstore/records.py tht/adapters/vector/qdrant.py \
|
||
tht/ports/vector.py tht/adapters/factory.py tht/search/evidence.py \
|
||
tht/semantic_migration.py tht/cli/vector_cmd.py tht/memory_migration.py \
|
||
tht/cli/memory_cmd.py tht/memory.py tht/solved.py tht/cli/schema_cmd.py \
|
||
tht/cli/lsh_cmd.py tht/taskdoc.py tht/locked_child_stdin.py tht/layout_markers.py \
|
||
tests/test_dwh_preprocess_job.py tests/test_effective_dwh_binding.py \
|
||
tests/test_config_resources.py tests/test_portable_paths.py tests/test_dwh_snapshot.py \
|
||
tests/test_dwh_owner_v2.py tests/test_dwh_owner_migration.py tests/test_search_pack.py \
|
||
tests/test_qdrant_vector_store.py tests/test_semantic_kind_isolation.py \
|
||
tests/test_qdrant_cli_commands.py tests/test_corpus_pipeline.py \
|
||
tests/test_memory_save_one.py tests/test_solved_question.py \
|
||
tests/test_semantic_revision_migration.py tests/test_memory_migration.py \
|
||
tests/test_memory_promotion.py tests/test_solved_search_cli.py \
|
||
tests/test_repository_memory_sql_paths.py tests/test_locked_child_stdin.py \
|
||
tests/test_layout_marker_commands.py tests/test_p3_internal_cli.py tests/test_lsh_job_resume.py \
|
||
tests/test_schema_fk_annotations.py)
|
||
(cd backend && npx vitest run \
|
||
test/workspace-revision-layout.test.ts test/workspace-runtime-renderer.test.ts \
|
||
test/workspace-runtime-config-lease.test.ts test/workspace-runtime-handoff.test.ts \
|
||
test/workspace-effective-config-equivalence.test.ts test/workspace-lock-root-lease.test.ts \
|
||
test/workspace-preprocessing-state.test.ts test/workspace-preprocessing-service.test.ts \
|
||
test/workspace-maintenance.test.ts test/workspace-registry.test.ts \
|
||
test/workspace-registry-addressed-publication.test.ts \
|
||
test/workspace-registry-addressed-process.test.ts test/workspace-reader-lease.test.ts \
|
||
test/routes-workspaces.test.ts test/routes-sessions.test.ts \
|
||
test/pi-process-manager.test.ts test/tht-runner.test.ts && \
|
||
npx tsc --noEmit -p . && npm run build)
|
||
(cd backend && npx tsc --noEmit --target ES2022 --module ES2022 \
|
||
--moduleResolution Bundler --strict --skipLibCheck \
|
||
test/registry-pull-job-imports.compile.ts)
|
||
(cd tools/thothctl && test "$(go env GOVERSION)" = "go1.26.5" && go test ./...)
|
||
./scripts/test-verify-workspace-install-docs.sh
|
||
./scripts/test-p3-acceptance.sh
|
||
python3 scripts/test-lint-plan-shell-fences.py
|
||
python3 scripts/lint-plan-shell-fences.py \
|
||
docs/superpowers/plans/2026-08-10-p2-host-workspace-preprocessing-cli.md \
|
||
docs/superpowers/plans/2026-08-10-p3-effective-config-fingerprint.md
|
||
```
|
||
|
||
The plan-shell linter extracts every fenced `bash`/`sh` block byte-for-byte, rejects shell blocks
|
||
containing angle-bracket placeholders, and runs `bash -n` on each block independently. Deliberate
|
||
metavariables may remain only in `text` fences; runnable shell must use quoted environment checks such
|
||
as `${RUN_ID:?message}` and an explicit regex validation. Its self-test includes unterminated quotes,
|
||
redirection-shaped `<placeholder>`, heredocs, and both P2/P3 documents.
|
||
|
||
Expected: every command exits 0. Record exact counts and versions, including Go 1.26.5, in the checkpoint report; no offline gate may download another Go toolchain. Do not run the full unrelated repository suites here; aggregate/full-stack verification belongs after P6 unless a touched-layer regression requires it.
|
||
|
||
**Step 3: Run the complete process once, without retry.**
|
||
|
||
```bash
|
||
./scripts/p3-acceptance.sh integration --keep
|
||
```
|
||
|
||
Expected: exit 0, one new `.artifacts/p3-effective-config/<run-id>/report.json`, matching `report.md`, ownership manifest, `overall: PASS`, `automated integration: PASS`, secret scan PASS, cleanup test PASS, and all fifteen assertions above PASS. Do not rerun a failure blindly: diagnose, add a regression test, fix, commit to regain clean state, then perform a new complete run with a new run ID.
|
||
|
||
**Step 4: Independently validate retained evidence.**
|
||
|
||
Use a small bounded script to parse `report.json`, verify every declared SHA-256, compare source commit/tree to current HEAD, confirm no undeclared public files, scan report/stdout/stderr/public manifests for every fixture canary, and query Docker by exact ownership labels. Then exercise retained cleanup:
|
||
|
||
```bash
|
||
RUN_ID="${RUN_ID:?export RUN_ID as the retained 32-hex acceptance run ID}"
|
||
[[ "$RUN_ID" =~ ^[0-9a-f]{32}$ ]]
|
||
./scripts/p3-acceptance.sh cleanup --run "$RUN_ID"
|
||
./scripts/p3-acceptance.sh cleanup --run "$RUN_ID"
|
||
```
|
||
|
||
Expected: first cleanup removes exactly declared containers/networks/volumes/temp roots and writes cleanup PASS; second reports idempotent `already_clean`; sanitized reports remain. Confirm no owned listener accepts connections.
|
||
|
||
**Step 5: Write and commit the automated checkpoint report.**
|
||
|
||
`docs/reports/2026-08-10-p3-effective-config-checkpoint.md` must record commit/tree, tool versions, focused command counts, retained report path and SHA-256, automated result, cleanup result, known unrelated debt (if any), and `manual acceptance: PENDING`. Update the P3 manual header and `PROJECT_STATE.md` with the same retained evidence and explicitly state that P4 is blocked pending reviewer approval.
|
||
|
||
```bash
|
||
git add docs/reports/2026-08-10-p3-effective-config-checkpoint.md \
|
||
docs/testing/p2-p6-manual-verification.md PROJECT_STATE.md \
|
||
scripts/lint-plan-shell-fences.py scripts/test-lint-plan-shell-fences.py
|
||
git commit -m "docs: record P3 automated verification"
|
||
```
|
||
|
||
Because that documentation commit is after the retained source commit, do not claim the report is bound to the doc commit; record both precisely. If policy requires a report bound to final docs too, run a fresh clean acceptance once and replace the retained reference rather than editing provenance.
|
||
|
||
**Step 6: Stop for the human checkpoint.**
|
||
|
||
Send the reviewer:
|
||
|
||
- retained report path and hashes;
|
||
- concise architecture summary;
|
||
- focused verification results;
|
||
- exact P3 manual walkthrough section;
|
||
- explicit choices `PASS`, `FAIL with notes`, or `DEFER`.
|
||
|
||
Do not begin P4. After the reviewer runs the walkthrough in a new environment and explicitly approves, update `Decision: PASS`, `manual acceptance: PASS`, `PROJECT_STATE.md`, and the checkpoint report in one scoped documentation commit:
|
||
|
||
```bash
|
||
git add docs/testing/p2-p6-manual-verification.md \
|
||
docs/reports/2026-08-10-p3-effective-config-checkpoint.md PROJECT_STATE.md
|
||
git commit -m "docs: record P3 manual acceptance"
|
||
```
|
||
|
||
Only that explicit decision completes P3 and unblocks P4.
|