docs: design P2-P6 workspace preprocessing
This commit is contained in:
@@ -0,0 +1,271 @@
|
||||
# P2–P6 Registry-Aware Workspace Preprocessing Design
|
||||
|
||||
**Status:** Reviewed execution design; P1 automated and manual acceptance PASS
|
||||
**Date:** 2026-08-10
|
||||
**Source:** `docs/prd/2026-08-09-workspace-preprocessing-prd.md` D2–D6
|
||||
|
||||
## 1. Goal and delivery protocol
|
||||
|
||||
Connect the existing preprocessing engine to a revision-pinned schema-v3 Git workspace without
|
||||
requiring Python, Node, or Pi on the host. Delivery remains on
|
||||
`codex/git-workspace-registry`, with separately reviewable commits and a hard user checkpoint after
|
||||
each of P2, P3, P4, P5, and P6.
|
||||
|
||||
Each plan has its own clean-state automated process goal. Focused tests run while implementing each
|
||||
plan; the aggregate P2–P6 Docker/process smoke and full repository verification run only after P6.
|
||||
`docs/testing/p2-p6-manual-verification.md` is the single living manual walkthrough and records one
|
||||
independent section and decision for each plan.
|
||||
|
||||
## 2. Chosen architecture
|
||||
|
||||
The existing native `thothctl` binary becomes the only host interface. New `workspace` commands use
|
||||
the installation descriptor to reconstruct the exact Compose project and launch a one-shot
|
||||
maintenance process from the selected `core` image. The process uses the installation's registry,
|
||||
sessions, Qdrant, embedding, Git credentials, connector bindings, and secret mounts. It does not
|
||||
call a running backend HTTP server and does not require a host language runtime.
|
||||
|
||||
```text
|
||||
thothctl --installation ... workspace <command>
|
||||
-> docker compose ... run --rm workspace-maintenance <fixed argv>
|
||||
-> compiled Node operator entrypoint in the core image
|
||||
-> WorkspaceRegistry + runtime renderer + installation bindings
|
||||
-> restrictive temporary harness config
|
||||
-> existing tht commands
|
||||
-> DWH / artifacts / Qdrant / internal Ollama
|
||||
```
|
||||
|
||||
The operator entrypoint shares production classes with the backend, but is a separate process and
|
||||
interface. It writes pristine JSON to stdout, bounded sanitized diagnostics to stderr, and never
|
||||
returns a secret value or rendered secret-bearing transport URL.
|
||||
|
||||
## 3. Common identity and persistence
|
||||
|
||||
Every operation binds these values before doing work:
|
||||
|
||||
- workspace ID;
|
||||
- exact 40-hex active Git commit;
|
||||
- exact canonical descriptor snapshot;
|
||||
- installation-local bindings resolved under configured secret roots;
|
||||
- runtime roots beneath `/data/sessions/<workspace-id>`;
|
||||
- internal Qdrant/Ollama contract;
|
||||
- the existing harness DWH ownership binding in P2 and its compatible, versioned P3 refinement.
|
||||
|
||||
Mutable preprocessing state and outputs remain beneath the workspace runtime boundary. Operator job
|
||||
state is an atomic, versioned JSON document under
|
||||
`/data/sessions/<workspace-id>/preprocessing/`. It records the operation, revision, existing
|
||||
harness ownership binding, child run IDs, completed stages, manual checkpoint, and terminal status.
|
||||
P2 resume is same-revision only. P5 deliberately upgrades FK review to a controlled revision
|
||||
transition and does not pretend an old same-revision resume token remains valid after a Git push.
|
||||
|
||||
P3 separates reusable DWH-derived state (keyed by the revision-independent effective DWH binding)
|
||||
from revision-scoped curated/semantic state. Schema/Evidence points and reads bind
|
||||
`workspace_revision`; their point identities include that revision. Memory/solved records remain
|
||||
workspace-wide. The harness config gains an explicit workspace-global `paths.memory` root rendered
|
||||
as `/data/sessions/<workspace-id>/memory`; legacy configs without it continue to resolve memory
|
||||
beneath `artifacts/memory` until explicitly migrated. All memory commands, locking, registry JSONL,
|
||||
and projection rebuild use the explicit root when present. Revision-scoped artifact/corpus roots
|
||||
therefore cannot split the canonical memory registry. DWH generations may be safely reused when
|
||||
their existing effective DWH binding is unchanged.
|
||||
|
||||
## 4. P2 — host preprocessing CLI
|
||||
|
||||
### Command contract
|
||||
|
||||
The initial public family is intentionally small:
|
||||
|
||||
```text
|
||||
thothctl ... workspace inspect --workspace <id> [--json]
|
||||
thothctl ... workspace preprocess dwh --workspace <id> [--resume <id>] [--json]
|
||||
thothctl ... workspace schema suggest-fks --workspace <id> [tht-safe options] [--json]
|
||||
thothctl ... workspace schema check --workspace <id> [--json]
|
||||
thothctl ... workspace index-schema --workspace <id> [--json]
|
||||
thothctl ... workspace preprocess evidence --workspace <id> [--dry-run] [--resume <id>] [--json]
|
||||
thothctl ... workspace preprocess run --workspace <id> [--resume <id>] [--json]
|
||||
```
|
||||
|
||||
`preprocess run` orders DWH introspection/LSH, FK review, schema indexing, and Evidence. When newly
|
||||
suggested FK changes need human review it records `manual_review_required` and exits without
|
||||
indexing schema or Evidence. A later explicit resume continues only after `schema check` succeeds.
|
||||
A fixture with already-curated FK can complete without a human pause.
|
||||
|
||||
P2 proves the complete chain with controlled REST DWH and HTTP Evidence fixtures. A filesystem
|
||||
Evidence source is parsed and rendered but the operational command stops with the stable code
|
||||
`evidence_materialization_required`; commit-addressed filesystem consumption belongs to P6.
|
||||
|
||||
The command never edits or pushes the registry repository. Curators use an ordinary review clone.
|
||||
P2 retains the existing engine's generation publication, idempotency, dry-run, and child resume
|
||||
semantics rather than adding a second preprocessing engine.
|
||||
|
||||
P2 includes the missing machine interfaces in the harness: JSON forms for FK suggest/check and
|
||||
schema indexing, Evidence job identity from `runtime_identity.workspace_id` rather than a temporary
|
||||
config filename, and bounded safe ingress/egress for `--from-sql` and annotation export. Host input
|
||||
is a readable regular non-symlink file, is size-bounded by `thothctl`, and is streamed over the
|
||||
one-shot process stdin rather than mounted as an arbitrary host directory.
|
||||
|
||||
The one-shot job is a dedicated Compose profile/service, not `compose run core`. It has no Pi auth
|
||||
mount, no writable Pi state, and an entrypoint that does not run Pi trust initialization. Git and
|
||||
connector override generators attach only the credentials required by the selected workspace
|
||||
operation to this service.
|
||||
|
||||
## 5. P3 — effective configuration and `.tht-dwh` ownership
|
||||
|
||||
P2 initially consumes the existing harness schema-v1 ownership contract unchanged. P3 makes that
|
||||
contract reproducible across the operator and session paths without silently invalidating existing
|
||||
generations. The current exclusion of `session_storage` and `runtime_identity` is preserved: a Git
|
||||
content-only commit must not force DWH introspection when the effective DWH configuration is
|
||||
unchanged.
|
||||
|
||||
P3 introduces a versioned shared canonicalizer for the non-secret effective DWH/preprocessing
|
||||
configuration and a stable logical config-source identity. It replaces dependence on random
|
||||
temporary config filenames while retaining a compatibility reader and explicit migration for
|
||||
existing `OWNER.json` schema-v1 roots. It also adds the explicit workspace-global memory root and
|
||||
a migration that copies and verifies one legacy canonical JSONL under the workspace lock before
|
||||
rebuilding its Qdrant projection; conflicting legacy registries fail closed. No in-place
|
||||
reinterpretation is allowed.
|
||||
|
||||
Reusable DWH cache roots are keyed by the versioned effective DWH binding. Revision-scoped runtime
|
||||
roots receive verified physical/LSH snapshots from that cache, while annotations, corpus ACTIVE,
|
||||
and schema/Evidence Qdrant records remain revision-specific. Qdrant schema/Evidence point IDs and
|
||||
queries include `workspace_revision`; memory/solved identities and queries remain workspace-wide.
|
||||
|
||||
P3 proves that the operator and a session render the same effective DWH binding, that semantically
|
||||
identical revisions reuse it, and that a changed endpoint/transport/database/schema/root-affecting
|
||||
policy fails closed. Documentation explains `.tht-dwh`, immutable generations, `OWNER.json`,
|
||||
`ACTIVE`, input versus config fingerprints, safe migration, regeneration, and recovery.
|
||||
|
||||
## 6. P4 — Qdrant collection lifecycle
|
||||
|
||||
A shared TypeScript collection manager owns Qdrant collection and payload-index reconciliation.
|
||||
Session admission and the operator path both call it.
|
||||
|
||||
Self-heal may:
|
||||
|
||||
- create a missing collection with exactly 1024 dimensions and cosine distance;
|
||||
- create any missing required keyword index;
|
||||
- tolerate an already-compatible concurrent creator and re-read final state.
|
||||
|
||||
Self-heal never mutates incompatible dimensions, distance, or index types. Those return
|
||||
`semantic_index_incompatible`.
|
||||
|
||||
The host CLI adds guarded collection inspection and rebuild. Rebuild requires the exact workspace
|
||||
ID, exact collection name repeated as confirmation, and an explicit destructive flag. It uses a
|
||||
cross-process quiescence protocol rather than trusting the one-shot job: `thothctl` acquires the
|
||||
installation lifecycle lock, asks the running backend to durably activate maintenance, verifies
|
||||
the complete session inventory is closed/finalized/archived, and polls a new loopback-only internal
|
||||
quiescence endpoint until both admission leases and `PiProcessManager.count()` are zero. It then
|
||||
stops `core`, rechecks that the container is stopped, and starts the dedicated maintenance service.
|
||||
The maintenance marker prevents a racing restart from admitting work. This backend-mediated drain
|
||||
is an explicit exception to the ordinary preprocessing path's no-backend-HTTP rule.
|
||||
|
||||
The job also verifies no preprocessing lock is held. It deletes only the descriptor-owned
|
||||
collection, recreates the complete contract, verifies it, and emits JSON. No prefix matching or
|
||||
global Qdrant mutation is allowed. A durable rebuild state is written before deletion; success
|
||||
restarts core and clears maintenance only after health verification. Failure after deletion leaves
|
||||
maintenance active and provides an explicit recovery/recreate command rather than claiming
|
||||
rollback of lost vector data.
|
||||
|
||||
## 7. P5 — curated FK annotations in Git
|
||||
|
||||
The canonical path is fixed, not descriptor-configurable:
|
||||
|
||||
```text
|
||||
workspace-content/<workspace-id>/schema/annotations.yaml
|
||||
```
|
||||
|
||||
The registry validates that the object is a regular Git blob at the same commit as the descriptor.
|
||||
Absence remains compatible and produces an empty canonical annotation set plus a warning until a
|
||||
curator publishes one. Symlinks, submodules, trees at the file path, cross-namespace paths, and
|
||||
malformed annotations are rejected.
|
||||
|
||||
On activation and before preprocessing/session use, the exact blob is read with fixed Git argv,
|
||||
validated by the harness annotation parser, and atomically synchronized to the immutable
|
||||
revision-qualified runtime root:
|
||||
|
||||
```text
|
||||
/data/sessions/<workspace-id>/revisions/<commit>/artifacts/mschema/annotations.yaml
|
||||
```
|
||||
|
||||
P3 changes operator and session rendering so `artifacts` and `indexes` select that exact revision
|
||||
root; the shared session-manifest root remains `/data/sessions/<workspace-id>/sessions`. Existing
|
||||
workspace-global artifact roots are treated as legacy input and require the explicit P3 migration;
|
||||
there is no mutable compatibility symlink or pointer used by pinned runtimes.
|
||||
|
||||
The synchronized file has restrictive mode and an adjacent ownership manifest containing
|
||||
workspace, commit, blob ID, content digest, and destination. A newer active revision writes a
|
||||
different directory, so a pinned historical runtime continues to receive its own revision.
|
||||
`physical.yaml` remains generated locally and is never published.
|
||||
|
||||
The annotation blob is bounded (16 MiB), UTF-8, and parsed before synchronization. The
|
||||
preprocessing CLI never pushes curated content. P2 same-revision local review is superseded in P5
|
||||
by an explicit controlled transition: after commit/push/pull, the operator reviews the current Git
|
||||
blob against the recorded candidate, runs `workspace schema accept --run <id> --yes`, and records
|
||||
the accepted candidate/current-blob digests and new revision. Continuation requires that exact
|
||||
accepted blob and compatible reusable DWH binding; otherwise it starts a new run. An empty file or
|
||||
`schema check` alone is not evidence of human review.
|
||||
|
||||
## 8. P6 — commit-addressed Evidence materialization
|
||||
|
||||
For filesystem Evidence, the registry materializes exactly
|
||||
`workspace-content/<id>/evidence` from the pinned commit into an immutable revision content root.
|
||||
It does not consume the mobile registry checkout and does not resolve against author files.
|
||||
|
||||
Materialization uses fixed Git plumbing to enumerate object type, mode, path, object ID, and bytes.
|
||||
It rejects every symlink at any depth, gitlink/submodule, device/FIFO/socket, unsupported mode,
|
||||
absolute/traversing/non-normalized path, cross-workspace namespace, duplicate normalized path,
|
||||
oversized individual source object, or object identity change. No archive is extracted by a shell.
|
||||
|
||||
Files are written with no-follow/exclusive semantics beneath a fresh owned staging directory.
|
||||
Every file is hashed and recorded in a bounded manifest. Installation-local non-secret limits bound
|
||||
entry count, cumulative bytes, path/segment bytes, and manifest bytes; conservative defaults are
|
||||
documented and may be raised deliberately for large repositories. Materialization streams blobs
|
||||
and performs a disk-space preflight, so per-file-valid adversarial trees cannot exhaust memory or
|
||||
inodes silently. The complete tree and manifest are fsynced and atomically renamed only after all
|
||||
checks pass. A subsequent consumer revalidates destination ownership and the manifest before
|
||||
reuse. Partial staging is removed without following links.
|
||||
|
||||
The runtime renderer receives the verified immutable content root, after which the existing
|
||||
filesystem Evidence adapter may discover only beneath that root. Corpus ACTIVE and Evidence vector
|
||||
records are revision-scoped, and retrieval requires the pinned revision. Retention keeps
|
||||
materialized and derived roots for every retained/pinned workspace revision and removes only
|
||||
unreferenced, manifest-owned roots. P6 also makes P2's filesystem Evidence path operational and
|
||||
removes the temporary stable stop.
|
||||
|
||||
## 9. Error and output contract
|
||||
|
||||
Stable public codes include:
|
||||
|
||||
- `workspace_not_found` / `workspace_not_activatable`;
|
||||
- `binding_missing`;
|
||||
- `preprocessing_conflict`;
|
||||
- `preprocessing_resume_mismatch`;
|
||||
- `manual_review_required`;
|
||||
- `evidence_materialization_required` (P2–P5 only);
|
||||
- `effective_config_mismatch`;
|
||||
- `semantic_index_incompatible`;
|
||||
- `annotation_invalid`;
|
||||
- `evidence_materialization_unsafe`.
|
||||
|
||||
JSON output contains status, stable code, workspace ID, revision, operation/run ID, completed stage
|
||||
names, counts, and safe artifact identities. It excludes endpoint credentials, secret contents,
|
||||
query-bearing signed URLs, raw child stderr, and arbitrary exception strings.
|
||||
|
||||
One writer lock per workspace serializes preprocessing, annotation synchronization, collection
|
||||
rebuild, and materialization publication where they could conflict. Read-only inspection remains
|
||||
concurrent.
|
||||
|
||||
## 10. Verification and manual acceptance
|
||||
|
||||
Each Px provides one clean-state process goal with unique ownership under `.artifacts/p<id>-...`,
|
||||
no retry, fixture-only secrets, machine/readable reports, secret scan, exact cleanup, and retained
|
||||
`--keep` mode. Focused unit/integration/type/lint tests are evidence for that Px. After each Px the
|
||||
user receives a report and an independently runnable manual section, and work stops for explicit
|
||||
authorization.
|
||||
|
||||
After P6, one aggregate test starts from a new Git registry and installation, executes P2 through P6
|
||||
with real local Git, REST fixtures, Qdrant, and Ollama, proves DWH/FK/schema/Evidence outputs,
|
||||
repeats for idempotency, proves a second installation can consume the same Git workspace with its
|
||||
own local state, exercises negative security cases, and cleans only owned resources. Only then are
|
||||
the full harness/backend/frontend suites and builds run.
|
||||
|
||||
A GUI/backend preprocessing endpoint, real PSD migration, SSH runtime transport, and policy-driven
|
||||
long-term GC beyond the existing engine remain outside P2–P6.
|
||||
Reference in New Issue
Block a user