diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index 08030c53..45ce7d08 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -3,6 +3,30 @@ > Starting-point snapshot for new sessions. Last updated: 2026-08-10 (final verification). > Point a fresh session here ("read PROJECT_STATE.md") before substantial work. +### P2 host preprocessing CLI — implementation complete, automated PASS, manual PENDING (2026-08-11) + +- **Scope:** P2 (PRD D2, based on the P1.1 registry contract): the installed native `thothctl` + binary is the only host interface for workspace preprocessing. Commands: `workspace inspect`, + `preprocess dwh`, `schema suggest-fks`, `schema check`, `index-schema`, `preprocess evidence`, + `preprocess run`, with the exact grammar, file-ingress bounds, result contract and exit codes in + `docs/contracts/workspace-preprocessing-cli.md`. +- **Operator:** `workspace-maintenance` is a profile-gated Compose service sharing the core image, + with no Pi auth/state, no backend/Pi/frontend listener, no Git credentials, and a compiled Node + entrypoint (`backend/src/workspace-maintenance.ts`) driving the existing harness engine through + pristine JSON machine interfaces (`schema_cmd.py`, `vector_cmd.py`, `preprocess_cmd.py`). +- **Boundaries honored:** FK review is digest-bound (candidate digest == persisted artifact; a + review accepted for the same candidate content counts); Qdrant collections are never created by + the product path (`require_existing` + pre-provisioned fixture, P4 owns lifecycle); filesystem + Evidence stops with `evidence_materialization_required` (P6); HTTP Evidence enforces an + installation private-host allowlist; `ssh_tunnel` stays fail-closed (P10); cross-revision DWH + reuse is explicitly P3. +- **Retained evidence:** `.artifacts/p2-integration/p2-b109757b26388a5ed6b1d173dee86584/` + (11/11 checks PASS), bound to clean source commit + `de5de36f9a4edfd4fbebf277822090871ccdd61f`. +- **Manual gate:** P2 walkthrough in `docs/testing/p2-p6-manual-verification.md`; decision + **PENDING** and recorded independently. P3 and later start only after an explicit new + authorization. + ### P1.1 workspace-directory registry — automated integration PASS, manual PENDING (2026-08-11) - **Scope:** P1 correction (not preprocessing). Root curator-owned catalog `thoth-workspaces.yaml`; diff --git a/docs/install/local-workspace-registry.md b/docs/install/local-workspace-registry.md index 13fc9c3b..1fea6358 100644 --- a/docs/install/local-workspace-registry.md +++ b/docs/install/local-workspace-registry.md @@ -19,6 +19,17 @@ Compose stack. Never put credentials in workspace YAML, Git, browser drafts, dia | Qdrant | Internal | Mandatory private Compose semantic service; persistent `qdrant-data` volume. | | Ollama embedding | Internal | Mandatory private Compose semantic service for `qwen3-embedding:0.6b`. | + +## Host preprocessing (P2) + +The installed native `thothctl` is the only host entrypoint for workspace preprocessing +(introspection+LSH, FK review, schema indexing, HTTP Evidence). Use +`thothctl --installation workspace --workspace [--json]` +per `docs/contracts/workspace-preprocessing-cli.md` and the P2 walkthrough in +`docs/testing/p2-p6-manual-verification.md`. Preprocessing runs through the profile-gated +`workspace-maintenance` Compose service; it never starts a backend/Pi/frontend listener and never +attaches Git credentials. + ## Prerequisites - macOS: Docker Desktop, Git, and sufficient volume disk space. Git Credential Manager is useful diff --git a/docs/install/server-workspace-registry.md b/docs/install/server-workspace-registry.md index 4b11d96b..c96911d9 100644 --- a/docs/install/server-workspace-registry.md +++ b/docs/install/server-workspace-registry.md @@ -14,6 +14,17 @@ proxy; never publish the core port directly. | Qdrant | Internal | Mandatory private Compose semantic service; persistent `qdrant-data` volume. | | Ollama embedding | Internal | Mandatory private Compose semantic service for `qwen3-embedding:0.6b`. | + +## Host preprocessing (P2) + +The installed native `thothctl` is the only host entrypoint for workspace preprocessing +(introspection+LSH, FK review, schema indexing, HTTP Evidence). Use +`thothctl --installation workspace --workspace [--json]` +per `docs/contracts/workspace-preprocessing-cli.md` and the P2 walkthrough in +`docs/testing/p2-p6-manual-verification.md`. Preprocessing runs through the profile-gated +`workspace-maintenance` Compose service; it never starts a backend/Pi/frontend listener and never +attaches Git credentials. + ## Service account, storage, and firewall Create a dedicated host service account and an operator root such as `/srv/thothii`. The core diff --git a/docs/testing/p2-p6-manual-verification.md b/docs/testing/p2-p6-manual-verification.md index 741b0e4a..dca9a92b 100644 --- a/docs/testing/p2-p6-manual-verification.md +++ b/docs/testing/p2-p6-manual-verification.md @@ -16,25 +16,45 @@ ## P2 — Host preprocessing CLI -**Status:** instructions to be finalized by P2 implementation; not yet runnable. +**Status:** P2 implementation complete; automated integration PASS; manual acceptance PENDING. Manual goal: from a clean local installation, use only `thothctl` on the host to inspect one registry workspace and execute the controlled REST-DWH/HTTP-Evidence preprocessing path without a -host Python or Node runtime. +host Python or Node runtime. Use a fresh operator root and a fresh fixture Git remote; never reuse +the automated `.artifacts/p2-integration/**` state. -Checks to fill during P2: +Commands (contract: `docs/contracts/workspace-preprocessing-cli.md`): -1. installation/render preflight; -2. workspace inspection and exact revision display; -3. DWH preprocessing and resume; -4. FK machine output and manual-review checkpoint; -5. schema check/index; -6. HTTP Evidence dry-run and real run; -7. idempotent rerun; -8. filesystem Evidence stable deferred error; -9. secret scan and exact cleanup. +```bash +thothctl --installation /thothii-installation.yaml workspace inspect --workspace --json +thothctl --installation /thothii-installation.yaml workspace preprocess dwh --workspace --json +thothctl --installation /thothii-installation.yaml workspace preprocess dwh --workspace --resume --json +thothctl --installation /thothii-installation.yaml workspace schema suggest-fks --workspace --from-sql .sql --output .yaml --json +thothctl --installation /thothii-installation.yaml workspace schema check --workspace --annotations .yaml --reviewed-candidates --json +thothctl --installation /thothii-installation.yaml workspace index-schema --workspace --json +thothctl --installation /thothii-installation.yaml workspace preprocess evidence --workspace --dry-run --json +thothctl --installation /thothii-installation.yaml workspace preprocess evidence --workspace --json +thothctl --installation /thothii-installation.yaml workspace preprocess run --workspace --json +``` -Decision: **PENDING**. +Checks: + +1. installation/render preflight (`inspect` returns exact revision + catalog/descriptor digests); +2. DWH introspection+LSH succeeds, rerun is `unchanged`, `--resume ` is `unchanged`/`succeeded`; +3. `schema suggest-fks` returns pristine JSON with `suggestedFksYaml` and a `manual_review_required` + block (exit 3) when candidates exist; the suggested YAML digest equals the reported digest; +4. `schema check --annotations --reviewed-candidates ` succeeds after review; +5. `index-schema` counts against a pre-provisioned compatible collection and rerun is `unchanged`; +6. HTTP Evidence `--dry-run` returns `dry_run`, the real run publishes, rerun is `unchanged`, an input + mutation produces a new generation/ACTIVE; +7. filesystem Evidence returns a stable `evidence_materialization_required` block with no partial + corpus/vector publication; +8. negatives: missing workspace (`workspace_not_activatable`), resume of a nonexistent run + (`preprocessing_resume_mismatch`), invalid annotations digest (`annotation_invalid`), no-Evidence + skip warning, no collection creation, no backend/Pi/frontend listener; +9. secret scan over retained artifacts and exact owned-resource cleanup. + +Decision: **PENDING** (independent manual gate; automation never records PASS). ## P3 — Effective config and `.tht-dwh` diff --git a/scripts/verify-workspace-install-docs.sh b/scripts/verify-workspace-install-docs.sh index 7b4d0dbd..051a8aaa 100755 --- a/scripts/verify-workspace-install-docs.sh +++ b/scripts/verify-workspace-install-docs.sh @@ -176,9 +176,11 @@ verify_compose_internal_semantic_contract() { import sys, yaml, pathlib doc = yaml.safe_load(pathlib.Path(sys.argv[1]).read_text()) services = doc["services"] -expected = {"core", "frontend", "qdrant", "embedding", "embedding-model-init"} +expected = {"core", "frontend", "qdrant", "embedding", "embedding-model-init", "workspace-maintenance"} if set(services) != expected: raise SystemExit(f"compose.yaml services mismatch: {sorted(services)}") +if doc["services"]["workspace-maintenance"].get("profiles") != ["workspace-maintenance"]: + raise SystemExit("workspace-maintenance must be profile-gated and absent from default startup") core = services["core"] env = core["environment"] for key, value in {