marcopan
c861a0df9e
Pre-release, non-deterministic tests (marker l2, skipped without .env + VPN). They
close the gaps L1 leaves open: real value grounding on the live schema, real memory
save-one upsert to pgvector, and the full GLM 5.2 -> gate conversation on the
'ablazione' question (which exercises D14 value grounding + formula on a multi-
column case + the gate glue L1 cannot reach).
workspaces/chirone-test.yaml points at the remote endpoints (DWH read-only +
pgvector dual-key, TLS self-signed); secrets via ${THOTH_*}.
- test_session_ablazione: precondition checks (workspace loads, env present, pi on
PATH) + the documented manual run protocol (human-in-the-loop; scripted-answers
variant is a follow-up). Default run skips cleanly.
- test_value_grounding_real: 'ablazione' grounds to multiple columns on the real
schema (D14a non-collapsing), needs a built LSH index.
- test_memory_save_one_real: save_one_memory upserts one row via the writer key
(D11) and search_similar retrieves it via the reader key.
Operator runs before release (pytest -m l2). Default run: 109 passed, 5 skipped.
77 lines
3.7 KiB
Python
77 lines
3.7 KiB
Python
"""L2: full session -- 'ablazione' question with GLM 5.2 + real DWH (spec D4, L2).
|
|
|
|
End-to-end validation that L1 cannot do: GLM 5.2 driving the real harness against
|
|
the real Chirone DWH + pgvector, on the 'ablazione' question (exercises D14 value
|
|
grounding + formula on a multi-column case), through the gate glue (widget-descriptor
|
|
emit/consume) that L1 cannot reach.
|
|
|
|
MODES (both manual, non-deterministic, pre-release -- NOT a regression gate):
|
|
- human-in-the-loop (default): the reviewer answers each gate widget via terminal.
|
|
- scripted answers (opt-in, --answers-file): canned reviewer answers for the
|
|
Altro/value-grounding/rollback paths so specific behaviors assert deterministically.
|
|
|
|
HOW TO RUN (operator, before release):
|
|
1. Populate harness/.env (THOTH_DWH_API_KEY, THOTH_VEC_API_KEY,
|
|
THOTH_VEC_WRITE_API_KEY, THOTH_SSL_CA, THOTH_*_REST_URL, THOTH_OLLAMA_URL).
|
|
2. Connect VPN. Ensure Pi is configured locally with GLM 5.2.
|
|
3. Run: pytest -m l2 tests/l2/test_session_ablazione.py -s
|
|
|
|
The full LLM->gate conversation is exercised manually here; this file provides the
|
|
prerequisite checks (workspace loads, env present, Pi on PATH) and documents the
|
|
session-assertions the operator confirms by inspection (sql_final.sql present and
|
|
read-only-validates; the ledger shows value_grounded or concept_formula_approved
|
|
-- i.e. D14 surfaced). Automated assertions grow as a fake-Pi driver lands.
|
|
"""
|
|
import os
|
|
import shutil
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
pytestmark = [pytest.mark.l2]
|
|
|
|
WORKSPACE = Path(__file__).resolve().parents[2] / "workspaces" / "chirone-test.yaml"
|
|
QUESTION = "dammi la lista dei pazienti che hanno fatto un'ablazione nel 2025"
|
|
|
|
|
|
def test_workspace_chirone_test_loads(l2_env):
|
|
"""The L2 workspace YAML loads and expands ${THOTH_*} from .env."""
|
|
from nsp.workspace import load_workspace
|
|
|
|
ws = load_workspace(WORKSPACE)
|
|
# secrets must be expanded (not the literal ${...} token)
|
|
assert not ws.vector_db.rest.api_key.startswith("${")
|
|
assert not ws.vector_db.write_rest.api_key.startswith("${")
|
|
|
|
|
|
def test_pi_binary_available(l2_env):
|
|
"""Pi must be on PATH to spawn the gate."""
|
|
assert shutil.which("pi") is not None, "pi not on PATH (configure Pi with GLM 5.2 first)"
|
|
|
|
|
|
def test_ablazione_session_manual(l2_env, tmp_path):
|
|
"""MANUAL end-to-end: GLM 5.2 + real DWH on the 'ablazione' question.
|
|
|
|
Launches pi --mode rpc from harness/, runs /nuova-domanda, and the reviewer
|
|
answers each gate widget via terminal. The operator confirms by inspection:
|
|
- a session is produced and finalizes
|
|
- sql_final.sql is present and read-only-validates against the DWH
|
|
- the ledger shows value_grounded OR concept_formula_approved (D14 exercised)
|
|
Run with: pytest -m l2 tests/l2/test_session_ablazione.py::test_ablazione_session_manual -s
|
|
|
|
The scripted-answers variant (--answers-file with canned reviewer responses for
|
|
the value-grounding / Altro / rollback paths) is a follow-up once the terminal
|
|
relay helper lands; for now this is human-in-the-loop.
|
|
"""
|
|
# Precondition: this is a manual, non-deterministic pre-release check, not a CI
|
|
# assertion. We surface the run instructions and assert only that the launch
|
|
# context is ready; the operator drives the conversation and inspects the outcome.
|
|
env_ok = all(os.environ.get(v, "").strip() for v in
|
|
["THOTH_DWH_API_KEY", "THOTH_VEC_API_KEY", "THOTH_VEC_WRITE_API_KEY"])
|
|
assert env_ok
|
|
assert WORKSPACE.exists()
|
|
print(f"\n[L2 manual] launch: pi --mode rpc (cwd=harness/)")
|
|
print(f"[L2 manual] /nuova-domanda \"{QUESTION}\"")
|
|
print("[L2 manual] confirm: sql_final.sql present + ledger has value_grounded/"
|
|
"concept_formula_approved. Session dir:", tmp_path)
|