From 16ee92c8f980b363a82cb0e2777c354ed25519fa Mon Sep 17 00:00:00 2001 From: mptyl Date: Mon, 10 Aug 2026 18:59:31 +0200 Subject: [PATCH] docs: design P2-P6 workspace preprocessing --- ...10-p2-p6-workspace-preprocessing-design.md | 271 ++++++++++++++++++ docs/testing/p2-p6-manual-verification.md | 128 +++++++++ 2 files changed, 399 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md create mode 100644 docs/testing/p2-p6-manual-verification.md diff --git a/docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md b/docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md new file mode 100644 index 00000000..d026dc81 --- /dev/null +++ b/docs/superpowers/specs/2026-08-10-p2-p6-workspace-preprocessing-design.md @@ -0,0 +1,271 @@ +# P2–P6 Registry-Aware Workspace Preprocessing Design + +**Status:** Reviewed execution design; P1 automated and manual acceptance PASS +**Date:** 2026-08-10 +**Source:** `docs/prd/2026-08-09-workspace-preprocessing-prd.md` D2–D6 + +## 1. Goal and delivery protocol + +Connect the existing preprocessing engine to a revision-pinned schema-v3 Git workspace without +requiring Python, Node, or Pi on the host. Delivery remains on +`codex/git-workspace-registry`, with separately reviewable commits and a hard user checkpoint after +each of P2, P3, P4, P5, and P6. + +Each plan has its own clean-state automated process goal. Focused tests run while implementing each +plan; the aggregate P2–P6 Docker/process smoke and full repository verification run only after P6. +`docs/testing/p2-p6-manual-verification.md` is the single living manual walkthrough and records one +independent section and decision for each plan. + +## 2. Chosen architecture + +The existing native `thothctl` binary becomes the only host interface. New `workspace` commands use +the installation descriptor to reconstruct the exact Compose project and launch a one-shot +maintenance process from the selected `core` image. The process uses the installation's registry, +sessions, Qdrant, embedding, Git credentials, connector bindings, and secret mounts. It does not +call a running backend HTTP server and does not require a host language runtime. + +```text +thothctl --installation ... workspace + -> docker compose ... run --rm workspace-maintenance + -> compiled Node operator entrypoint in the core image + -> WorkspaceRegistry + runtime renderer + installation bindings + -> restrictive temporary harness config + -> existing tht commands + -> DWH / artifacts / Qdrant / internal Ollama +``` + +The operator entrypoint shares production classes with the backend, but is a separate process and +interface. It writes pristine JSON to stdout, bounded sanitized diagnostics to stderr, and never +returns a secret value or rendered secret-bearing transport URL. + +## 3. Common identity and persistence + +Every operation binds these values before doing work: + +- workspace ID; +- exact 40-hex active Git commit; +- exact canonical descriptor snapshot; +- installation-local bindings resolved under configured secret roots; +- runtime roots beneath `/data/sessions/`; +- internal Qdrant/Ollama contract; +- the existing harness DWH ownership binding in P2 and its compatible, versioned P3 refinement. + +Mutable preprocessing state and outputs remain beneath the workspace runtime boundary. Operator job +state is an atomic, versioned JSON document under +`/data/sessions//preprocessing/`. It records the operation, revision, existing +harness ownership binding, child run IDs, completed stages, manual checkpoint, and terminal status. +P2 resume is same-revision only. P5 deliberately upgrades FK review to a controlled revision +transition and does not pretend an old same-revision resume token remains valid after a Git push. + +P3 separates reusable DWH-derived state (keyed by the revision-independent effective DWH binding) +from revision-scoped curated/semantic state. Schema/Evidence points and reads bind +`workspace_revision`; their point identities include that revision. Memory/solved records remain +workspace-wide. The harness config gains an explicit workspace-global `paths.memory` root rendered +as `/data/sessions//memory`; legacy configs without it continue to resolve memory +beneath `artifacts/memory` until explicitly migrated. All memory commands, locking, registry JSONL, +and projection rebuild use the explicit root when present. Revision-scoped artifact/corpus roots +therefore cannot split the canonical memory registry. DWH generations may be safely reused when +their existing effective DWH binding is unchanged. + +## 4. P2 — host preprocessing CLI + +### Command contract + +The initial public family is intentionally small: + +```text +thothctl ... workspace inspect --workspace [--json] +thothctl ... workspace preprocess dwh --workspace [--resume ] [--json] +thothctl ... workspace schema suggest-fks --workspace [tht-safe options] [--json] +thothctl ... workspace schema check --workspace [--json] +thothctl ... workspace index-schema --workspace [--json] +thothctl ... workspace preprocess evidence --workspace [--dry-run] [--resume ] [--json] +thothctl ... workspace preprocess run --workspace [--resume ] [--json] +``` + +`preprocess run` orders DWH introspection/LSH, FK review, schema indexing, and Evidence. When newly +suggested FK changes need human review it records `manual_review_required` and exits without +indexing schema or Evidence. A later explicit resume continues only after `schema check` succeeds. +A fixture with already-curated FK can complete without a human pause. + +P2 proves the complete chain with controlled REST DWH and HTTP Evidence fixtures. A filesystem +Evidence source is parsed and rendered but the operational command stops with the stable code +`evidence_materialization_required`; commit-addressed filesystem consumption belongs to P6. + +The command never edits or pushes the registry repository. Curators use an ordinary review clone. +P2 retains the existing engine's generation publication, idempotency, dry-run, and child resume +semantics rather than adding a second preprocessing engine. + +P2 includes the missing machine interfaces in the harness: JSON forms for FK suggest/check and +schema indexing, Evidence job identity from `runtime_identity.workspace_id` rather than a temporary +config filename, and bounded safe ingress/egress for `--from-sql` and annotation export. Host input +is a readable regular non-symlink file, is size-bounded by `thothctl`, and is streamed over the +one-shot process stdin rather than mounted as an arbitrary host directory. + +The one-shot job is a dedicated Compose profile/service, not `compose run core`. It has no Pi auth +mount, no writable Pi state, and an entrypoint that does not run Pi trust initialization. Git and +connector override generators attach only the credentials required by the selected workspace +operation to this service. + +## 5. P3 — effective configuration and `.tht-dwh` ownership + +P2 initially consumes the existing harness schema-v1 ownership contract unchanged. P3 makes that +contract reproducible across the operator and session paths without silently invalidating existing +generations. The current exclusion of `session_storage` and `runtime_identity` is preserved: a Git +content-only commit must not force DWH introspection when the effective DWH configuration is +unchanged. + +P3 introduces a versioned shared canonicalizer for the non-secret effective DWH/preprocessing +configuration and a stable logical config-source identity. It replaces dependence on random +temporary config filenames while retaining a compatibility reader and explicit migration for +existing `OWNER.json` schema-v1 roots. It also adds the explicit workspace-global memory root and +a migration that copies and verifies one legacy canonical JSONL under the workspace lock before +rebuilding its Qdrant projection; conflicting legacy registries fail closed. No in-place +reinterpretation is allowed. + +Reusable DWH cache roots are keyed by the versioned effective DWH binding. Revision-scoped runtime +roots receive verified physical/LSH snapshots from that cache, while annotations, corpus ACTIVE, +and schema/Evidence Qdrant records remain revision-specific. Qdrant schema/Evidence point IDs and +queries include `workspace_revision`; memory/solved identities and queries remain workspace-wide. + +P3 proves that the operator and a session render the same effective DWH binding, that semantically +identical revisions reuse it, and that a changed endpoint/transport/database/schema/root-affecting +policy fails closed. Documentation explains `.tht-dwh`, immutable generations, `OWNER.json`, +`ACTIVE`, input versus config fingerprints, safe migration, regeneration, and recovery. + +## 6. P4 — Qdrant collection lifecycle + +A shared TypeScript collection manager owns Qdrant collection and payload-index reconciliation. +Session admission and the operator path both call it. + +Self-heal may: + +- create a missing collection with exactly 1024 dimensions and cosine distance; +- create any missing required keyword index; +- tolerate an already-compatible concurrent creator and re-read final state. + +Self-heal never mutates incompatible dimensions, distance, or index types. Those return +`semantic_index_incompatible`. + +The host CLI adds guarded collection inspection and rebuild. Rebuild requires the exact workspace +ID, exact collection name repeated as confirmation, and an explicit destructive flag. It uses a +cross-process quiescence protocol rather than trusting the one-shot job: `thothctl` acquires the +installation lifecycle lock, asks the running backend to durably activate maintenance, verifies +the complete session inventory is closed/finalized/archived, and polls a new loopback-only internal +quiescence endpoint until both admission leases and `PiProcessManager.count()` are zero. It then +stops `core`, rechecks that the container is stopped, and starts the dedicated maintenance service. +The maintenance marker prevents a racing restart from admitting work. This backend-mediated drain +is an explicit exception to the ordinary preprocessing path's no-backend-HTTP rule. + +The job also verifies no preprocessing lock is held. It deletes only the descriptor-owned +collection, recreates the complete contract, verifies it, and emits JSON. No prefix matching or +global Qdrant mutation is allowed. A durable rebuild state is written before deletion; success +restarts core and clears maintenance only after health verification. Failure after deletion leaves +maintenance active and provides an explicit recovery/recreate command rather than claiming +rollback of lost vector data. + +## 7. P5 — curated FK annotations in Git + +The canonical path is fixed, not descriptor-configurable: + +```text +workspace-content//schema/annotations.yaml +``` + +The registry validates that the object is a regular Git blob at the same commit as the descriptor. +Absence remains compatible and produces an empty canonical annotation set plus a warning until a +curator publishes one. Symlinks, submodules, trees at the file path, cross-namespace paths, and +malformed annotations are rejected. + +On activation and before preprocessing/session use, the exact blob is read with fixed Git argv, +validated by the harness annotation parser, and atomically synchronized to the immutable +revision-qualified runtime root: + +```text +/data/sessions//revisions//artifacts/mschema/annotations.yaml +``` + +P3 changes operator and session rendering so `artifacts` and `indexes` select that exact revision +root; the shared session-manifest root remains `/data/sessions//sessions`. Existing +workspace-global artifact roots are treated as legacy input and require the explicit P3 migration; +there is no mutable compatibility symlink or pointer used by pinned runtimes. + +The synchronized file has restrictive mode and an adjacent ownership manifest containing +workspace, commit, blob ID, content digest, and destination. A newer active revision writes a +different directory, so a pinned historical runtime continues to receive its own revision. +`physical.yaml` remains generated locally and is never published. + +The annotation blob is bounded (16 MiB), UTF-8, and parsed before synchronization. The +preprocessing CLI never pushes curated content. P2 same-revision local review is superseded in P5 +by an explicit controlled transition: after commit/push/pull, the operator reviews the current Git +blob against the recorded candidate, runs `workspace schema accept --run --yes`, and records +the accepted candidate/current-blob digests and new revision. Continuation requires that exact +accepted blob and compatible reusable DWH binding; otherwise it starts a new run. An empty file or +`schema check` alone is not evidence of human review. + +## 8. P6 — commit-addressed Evidence materialization + +For filesystem Evidence, the registry materializes exactly +`workspace-content//evidence` from the pinned commit into an immutable revision content root. +It does not consume the mobile registry checkout and does not resolve against author files. + +Materialization uses fixed Git plumbing to enumerate object type, mode, path, object ID, and bytes. +It rejects every symlink at any depth, gitlink/submodule, device/FIFO/socket, unsupported mode, +absolute/traversing/non-normalized path, cross-workspace namespace, duplicate normalized path, +oversized individual source object, or object identity change. No archive is extracted by a shell. + +Files are written with no-follow/exclusive semantics beneath a fresh owned staging directory. +Every file is hashed and recorded in a bounded manifest. Installation-local non-secret limits bound +entry count, cumulative bytes, path/segment bytes, and manifest bytes; conservative defaults are +documented and may be raised deliberately for large repositories. Materialization streams blobs +and performs a disk-space preflight, so per-file-valid adversarial trees cannot exhaust memory or +inodes silently. The complete tree and manifest are fsynced and atomically renamed only after all +checks pass. A subsequent consumer revalidates destination ownership and the manifest before +reuse. Partial staging is removed without following links. + +The runtime renderer receives the verified immutable content root, after which the existing +filesystem Evidence adapter may discover only beneath that root. Corpus ACTIVE and Evidence vector +records are revision-scoped, and retrieval requires the pinned revision. Retention keeps +materialized and derived roots for every retained/pinned workspace revision and removes only +unreferenced, manifest-owned roots. P6 also makes P2's filesystem Evidence path operational and +removes the temporary stable stop. + +## 9. Error and output contract + +Stable public codes include: + +- `workspace_not_found` / `workspace_not_activatable`; +- `binding_missing`; +- `preprocessing_conflict`; +- `preprocessing_resume_mismatch`; +- `manual_review_required`; +- `evidence_materialization_required` (P2–P5 only); +- `effective_config_mismatch`; +- `semantic_index_incompatible`; +- `annotation_invalid`; +- `evidence_materialization_unsafe`. + +JSON output contains status, stable code, workspace ID, revision, operation/run ID, completed stage +names, counts, and safe artifact identities. It excludes endpoint credentials, secret contents, +query-bearing signed URLs, raw child stderr, and arbitrary exception strings. + +One writer lock per workspace serializes preprocessing, annotation synchronization, collection +rebuild, and materialization publication where they could conflict. Read-only inspection remains +concurrent. + +## 10. Verification and manual acceptance + +Each Px provides one clean-state process goal with unique ownership under `.artifacts/p-...`, +no retry, fixture-only secrets, machine/readable reports, secret scan, exact cleanup, and retained +`--keep` mode. Focused unit/integration/type/lint tests are evidence for that Px. After each Px the +user receives a report and an independently runnable manual section, and work stops for explicit +authorization. + +After P6, one aggregate test starts from a new Git registry and installation, executes P2 through P6 +with real local Git, REST fixtures, Qdrant, and Ollama, proves DWH/FK/schema/Evidence outputs, +repeats for idempotency, proves a second installation can consume the same Git workspace with its +own local state, exercises negative security cases, and cleans only owned resources. Only then are +the full harness/backend/frontend suites and builds run. + +A GUI/backend preprocessing endpoint, real PSD migration, SSH runtime transport, and policy-driven +long-term GC beyond the existing engine remain outside P2–P6. diff --git a/docs/testing/p2-p6-manual-verification.md b/docs/testing/p2-p6-manual-verification.md new file mode 100644 index 00000000..8b782500 --- /dev/null +++ b/docs/testing/p2-p6-manual-verification.md @@ -0,0 +1,128 @@ +# P2–P6 Manual Verification Walkthrough + +> Living document. Each section is completed with exact released commands and artifacts during its +> corresponding plan. Automated integration and manual acceptance use separate clean state. + +## Global rules + +- Use a new temporary operator root and a new private fixture Git remote for each Px. +- Never use production PSD credentials in a retained report or screenshot. +- Keep descriptor/content in Git; keep endpoints, bindings, credentials, and certificates in the + installation-local protected directory. +- Do not print secret files, rendered signed URLs, Compose environments, or unbounded logs. +- Record the ThothII commit, workspace commit, installation descriptor path, Compose project name, + command exit status, and report path. +- A focused manual PASS does not replace the automated process goal. + +## P2 — Host preprocessing CLI + +**Status:** instructions to be finalized by P2 implementation; not yet runnable. + +Manual goal: from a clean local installation, use only `thothctl` on the host to inspect one +registry workspace and execute the controlled REST-DWH/HTTP-Evidence preprocessing path without a +host Python or Node runtime. + +Checks to fill during P2: + +1. installation/render preflight; +2. workspace inspection and exact revision display; +3. DWH preprocessing and resume; +4. FK machine output and manual-review checkpoint; +5. schema check/index; +6. HTTP Evidence dry-run and real run; +7. idempotent rerun; +8. filesystem Evidence stable deferred error; +9. secret scan and exact cleanup. + +Decision: **PENDING**. + +## P3 — Effective config and `.tht-dwh` + +**Status:** instructions to be finalized by P3 implementation; not yet runnable. + +Manual goal: compare operator and session effective DWH identities, inspect `OWNER.json` and +`ACTIVE` without exposing secrets, prove safe reuse after a content-only revision, and prove +fail-closed behavior after a DWH-affecting change. + +Checks to fill during P3: + +1. canonical fingerprint comparison; +2. stable logical config-source identity; +3. schema-v1 ownership compatibility/migration; +4. DWH cache reuse across equivalent revisions; +5. revision-scoped schema/Evidence state; +6. mismatch rejection and recovery. + +Decision: **PENDING**. + +## P4 — Qdrant bootstrap and guarded rebuild + +**Status:** instructions to be finalized by P4 implementation; not yet runnable. + +Manual goal: prove admission creates a missing compatible collection and indexes, refuses an +incompatible collection, and permits destructive rebuild only under durable maintenance with no +active readers/jobs and exact repeated confirmation. + +Checks to fill during P4: + +1. missing-collection self-heal; +2. missing-index self-heal; +3. dimensions/distance/index-type refusal; +4. confirmation mismatch refusal; +5. active-reader/job refusal; +6. successful drained rebuild; +7. interrupted rebuild recovery with maintenance retained. + +Decision: **PENDING**. + +## P5 — Curated FK annotations in Git + +**Status:** instructions to be finalized by P5 implementation; not yet runnable. + +Manual goal: curate `workspace-content//schema/annotations.yaml` in an author clone, publish it, +pull the new revision, explicitly accept the reviewed blob, and prove atomic revision-correct sync +without changing `physical.yaml` in Git. + +Checks to fill during P5: + +1. candidate/export review; +2. Git commit and exact blob identity; +3. pull and controlled revision transition; +4. explicit acceptance record; +5. synchronized destination and ownership manifest; +6. malformed/oversized/symlink/cross-namespace refusal; +7. pinned historical revision isolation. + +Decision: **PENDING**. + +## P6 — Commit-addressed Evidence materialization + +**Status:** instructions to be finalized by P6 implementation; not yet runnable. + +Manual goal: materialize filesystem Evidence from the pinned Git commit, inspect its bounded +manifest, preprocess/index it, retrieve only the pinned revision, and exercise unsafe-tree and +aggregate-limit failures without partial publication. + +Checks to fill during P6: + +1. exact commit/tree/object identities; +2. successful atomic materialization; +3. manifest and file digest verification; +4. filesystem Evidence dry-run/run/idempotency; +5. revision-filtered Qdrant retrieval and corpus ACTIVE; +6. nested symlink/gitlink/traversal/special-file refusal; +7. file-count/total-byte/path/manifest limit refusal; +8. retention while pinned and owned cleanup after release. + +Decision: **PENDING**. + +## Final aggregate P2–P6 verification + +**Status:** runnable only after P6. + +The final manual pass will start with a new registry and two independent installations. It will +run the complete DWH → FK → schema → filesystem Evidence chain, prove idempotency and revision +isolation, confirm the second installation uses its own secrets/state, and compare its observations +to the retained aggregate automated report. + +Decision: **PENDING**.