diff --git a/harness/README.md b/harness/README.md new file mode 100644 index 00000000..bdfc38c1 --- /dev/null +++ b/harness/README.md @@ -0,0 +1,88 @@ +# ThothII harness — `nsp` + +The self-contained Pi layer of ThothII: a deterministic Python CLI (`nsp`) + a Pi gate +extension (`.pi/extensions/nsp-gate.js`) that runs the 8-phase NL→SQL workflow, +emitting and consuming widget-descriptor JSON. Derived from ChironeWp3 as a validated +starting point, **adapted** (not assumed reliable) to the ThothII contract. + +## Install + +```bash +cd harness +python -m venv .venv && source .venv/bin/activate +pip install -e ".[dev]" +``` + +The `nsp` command is now on PATH. Node ≥ 20 is needed for the gate JS tests +(`npm test`). + +## Configure + +### `.env` (gitignored — secrets live ONLY here) + +```bash +cp .env.example .env +# fill in: THOTH_PROFILE, THOTH_DB_*, THOTH_DWH_API_KEY, THOTH_VEC_API_KEY, +# THOTH_VEC_WRITE_API_KEY, THOTH_SSL_CA, THOTH_OLLAMA_URL, ... +``` + +Keys are never logged; URLs are fine. Rotate any key that appeared in chat. + +### `workspaces/.yaml` + +A workspace wires the relational DWH + the pgvector (dual-key) + embeddings + evidence. +See `workspaces/chirone.example.yaml`. `${THOTH_*}` tokens expand from `.env`. + +## The workflow + +`workflow.yaml` is the **single source of workflow truth** (spec F2). Eight phases +(F1 chiarimento → F8 datamart). Edit it to change the workflow; `phase.py`, the gate, +and the skill all read from it — no mirrored constants to drift. + +```bash +nsp phase meta --json # the gate reads workflow facts from here +nsp phase meta # human-readable +``` + +## Run + +The harness runs inside Pi (`pi --mode rpc`, cwd = `harness/`): + +```bash +pi --mode rpc +# in Pi: /nuova-domanda "" +``` + +The gate (`nsp-gate.js`) presents reviewer widgets (widget-descriptor), the reviewer +decides, and `nsp` persists decisions to the append-only ledger (`review_decisions.jsonl`). +See `docs/testing.md` for what each interaction level validates. + +## Test + +```bash +pytest # L0 (testcontainers, real Postgres) + L1 (pure logic + gate builders) +npm test # gate widget-builder golden + fuzzy tests (JS) +pytest -m l2 # L2: real GLM 5.2 + remote DWH (pre-release; needs .env + VPN + CA bundle) +``` + +L0 needs Docker (present on the dev machine). L2 is manual, non-deterministic, and +skips cleanly without `.env`. See `docs/testing.md` for the honest split of what each +level covers and — crucially — does **not** cover. + +## Layout + +``` +nsp/ Python package (CLI + workflow + phase + decisions + db/rest/mschema/ + vectorstore/evidence/search/session + memory) +.pi/ Pi project (settings, prompts, themes, extensions/nsp-gate.js + gate/) +workflow.yaml single source of workflow truth (F2) +workspaces/ workspace YAML definitions (D3) +scripts/ reader/writer RPC SQL for pgvector (D11) +tests/ L0 (testcontainers), L1 (logic + builders), L2 (real model + DB) +docs/ testing guide + workflow editing +``` + +## Reference + +- Architecture spec: `docs/superpowers/specs/2026-06-25-thothii-architecture-design.md` +- Implementation plan: `docs/superpowers/plans/2026-06-25-harness-implementation.md` diff --git a/harness/docs/testing.md b/harness/docs/testing.md new file mode 100644 index 00000000..a4d792af --- /dev/null +++ b/harness/docs/testing.md @@ -0,0 +1,96 @@ +# Testing the harness — the L0 / L1 / L2 split + +The harness is tested at **three levels**. The split is mandatory and honest: the +non-deterministic core (the skill→LLM→gate loop) cannot live in the fast automated +loop, and the DB-touching ported code needs a real database to validate. + +## ⚠️ The honest headline + +**The skill→LLM→gate loop — the heart of the system — has NO automated regression +coverage.** It is exercised **only at L2** (manual, non-deterministic, slow, requires +credentials + VPN). This is a deliberate, conscious choice: the LLM is non-deterministic +and needs a configured Pi + network, so it cannot run in CI. + +**Consequence: agentic-behavior regressions surface at pre-release L2 runs, not at +commit. Accept this and run L2 before any release.** + +A partial automated net for this gap would be a **fake-Pi** runtime mock that lets the +gate glue run in CI — documented as the single highest-value cross-cutting follow-up +(built alongside the backend plan, not here). + +## L0 — testcontainers, real Postgres (runs locally on every `pytest`) + +**What:** integrity tests of the ported DB-touching modules against a real Postgres in +a Docker container. No LLM, no remote network. + +**Dependencies:** Docker (present on the dev machine). No credentials, no VPN. + +**Coverage:** `db/connection` (read-only enforcement — exit 2 if writable role, +cannot INSERT), `db/introspect` (known schema: tables, columns, types, comments, FKs, +enum, composite PK), `db/sampling` (most-frequent values + truncation reporting), +`mschema/render` + `mschema/eligibility` (the column-eligibility principle), the RRF +pipeline (when the LSH index path lands). This is where "ported code is not assumed +reliable" gains real teeth for the data layer. + +**Run:** `pytest` (default; auto-skips if Docker is absent). File naming: +`tests/l0/test_*.py`, marker `@pytest.mark.l0`. + +## L1 — fake data, deterministic (runs locally on every `pytest`) + +**What:** logic-pure tests with fake data (`tmp_path`, fixtures, mocks). No DB, no LLM, +no network. + +**Coverage (honest):** +- Python logic-pure: `workflow.yaml` loading, `effective_decisions`, + `teardown_to_phase`, `generate_task_doc`, `aggregate_lsh_multi` (on fake hits), + `formula_store` read/write, `decision_retracted`, `save_one_memory`, the + rationale-capture contract, the session-coherence smoke, CLI `phase meta --json`. +- **Gate builder functions** (pure, in JS, tested in JS): the widget-descriptor + builders produce the correct JSON given params. Tested in-language (`node --test`), + no Python↔JS bridge, no Python mirror. + +**Honest limitation (load-bearing):** L1 can test the gate **builders** (pure functions) +but **NOT the gate glue** — registration, emission via `ctx.sendRaw`, the no-limbo loop, +the anti-bypass hooks. The glue depends on the Pi runtime (`pi.registerTool`, +`ctx.sendRaw`, `emitAndWait`) and cannot run without either a real Pi or a fake-Pi mock. +**The glue is tested only at L2.** + +**Run:** `pytest` (default) + `npm test` (gate builders, JS). No marker. + +## L2 — real LLM + real remote DB, manual / pre-release (NOT automated) + +**What:** end-to-end sessions with GLM 5.2 + the real Chirone DWH + pgvector, reached +via REST over VPN. Plus the gate-glue validation (the part L1 cannot reach). + +**Dependencies (all required, skip cleanly if missing):** +- LLM: Pi configured locally with GLM 5.2. +- DB: the remote Supabase endpoints (DWH read-only + pgvector reader/writer), via VPN. +- `harness/.env` populated with the API keys + CA path. + +**Coverage (honest):** validates the assumption L1 cannot — that GLM 5.2 produces tool +calls the gate accepts, that the skill's prompts lead to the expected interaction shape, +that the gate glue handles real tool-call sequences (incl. Altro/Rifiuta/rollback), +that value grounding and formula approval surface correctly on the real schema, that +`memory save-one` upserts to the real pgvector. **Closes the skill→LLM→gate loop AND +exercises the gate glue.** + +**Honest limitation:** L2 is non-deterministic (the model may behave differently across +runs) and slow/costly. It is a **pre-release safety net, not a regression gate**. + +**Run:** `pytest -m l2` (only; default run is `pytest -m 'not l2'`). The `l2_env` +fixture skips each L2 test (not fails) when `.env` is incomplete. File naming: +`tests/l2/test_*.py`, marker `@pytest.mark.l2`. + +## How to run each level + +```bash +pytest # L0 + L1 (default; addopts '-m not l2') +npm test # gate builders (JS, node --test) +pytest -m l2 # L2 only — pre-release, needs .env + VPN + CA bundle +``` + +## Security note on keys + +All keys live **only** in `harness/.env` (gitignored). Never in code, never committed, +never logged. `.env.example` is committed with variable names and empty values. Tests +mask secrets; URLs in logs are fine. **Rotate any key that appeared in a chat transcript.** diff --git a/harness/docs/workflow-editing.md b/harness/docs/workflow-editing.md new file mode 100644 index 00000000..c3f5075c --- /dev/null +++ b/harness/docs/workflow-editing.md @@ -0,0 +1,75 @@ +# Editing the workflow + +`harness/workflow.yaml` is the **single source of workflow truth** (spec F2, §5.3). +`phase.py`, the gate (`nsp-gate.js`), and the skill all read from it. There are **no +mirrored constants** in JS or Python — that was the ChironeWp3 drift bug +(`PHASE_NAMES` truncated to 7 entries in JS, F8/datamart silently dropped). Editing +this one file is the only place the workflow changes. + +## Structure + +```yaml +schema_version: 1 +phases: + - id: F3 # stable id (referenced by the gate) + name: riscrittura # human label (nsp phase show / gate UI) + advance: kind:phase # how the phase advances (see below) + prerequisites: # what must hold before advancing (gate checks these) + - decision_exists: question_rewritten + artifacts_out: [question.md] # files produced; used by teardown on rollback (D15) +``` + +`decision_min_phase: auto` and `max_phase: auto` mean these are derived from the +phase list (don't hard-code them). + +## `advance` kinds + +| kind | meaning | +|----------------------------|----------------------------------------------------------------| +| `kind:phase` | advances on any phase-approved decision for this phase | +| `auto_if_empty` | auto-advances if no decisions were made (F2 memoria) | +| `auto_if_empty_or_skipped` | auto-advances if empty OR a `phase_skipped` decision exists | +| `reviewer_decide` | requires an explicit reviewer decision (F4, F8) | + +## `prerequisites` predicates + +A phase is "ready to advance" when ALL its prerequisites hold. Predicates: + +- `decision_exists: ` — a decision of that type exists (effective view). +- `decision_subject_exists: [, ]` — e.g. a `phase_skipped` for `phase:6`. +- `file_validates: [, ]` — e.g. `schema_linking.json` parses as a + `SchemaLinking`. +- `all_ctes_approved: true` — all CTE blocks approved (F6). +- `any: [...]` / `all: [...]` — combine predicates. + +A decision type's **min phase** is derived: the earliest phase whose prerequisites +reference it. So adding a prerequisite like `decision_exists: my_new_decision` to F5 +automatically makes `my_new_decision` valid from phase 5. + +## Common edits + +### Add a phase + +Append a phase to `phases:`. `max_phase` becomes the new count automatically. Give it +an `id`, `name`, `advance`, and (optionally) `prerequisites` / `artifacts_out`. + +### Reorder / merge / skip phases + +Reorder the list; `max_phase` and `decision_min_phase` recompute. To make a phase +skippable, set `advance: auto_if_empty_or_skipped` and let the gate emit a +`phase_skipped` decision (predicate `decision_subject_exists: [phase_skipped, "phase:N"]`). + +### Add an artifact + +Add it to a phase's `artifacts_out`. On rollback (`teardown_to_phase`), artifacts +produced by phases **after** the target are deleted; the target phase's artifacts are +preserved. This is the D15 fix for the orphaned-CTE-blocks-finalize bug. + +## After editing + +```bash +nsp phase meta --json # confirm the new shape (max_phase, phases, artifacts_out) +pytest # L1 coherence smoke re-derives from the new workflow.yaml +``` + +No JS or Python constants to update — that's the point.