marcopanandClaude Fable 5
c4951e2aa5
Audit findings 6.1-6.4 + the audit's remediation plan itself (docs/superpowers/plans/2026-07-20-full-audit-remediation-plan.md). - ruff: 34 → 0 (unused imports/f-strings auto-fixed; E702 semicolon lines split in test files; one unused local dropped). Suite still 819 green. - CLAUDE.md + PROJECT_STATE.md no longer claim "no database / settings in settings.json": the harness selects filesystem OR PostgreSQL session storage (repository.py, server mode), and settings flow through harness preferences with the JSON file as fallback only. - tools/replay: stub /me (SPA boot was parsing the SPA's own HTML as JSON) and /runtime/prewarm. - failSession best-effort persistence now logs its failure server-side instead of vanishing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
78 lines
3.7 KiB
Python
78 lines
3.7 KiB
Python
"""L2: full session -- 'ablazione' question with GLM 5.2 + real DWH (spec D4, L2).
|
|
|
|
End-to-end validation that L1 cannot do: GLM 5.2 driving the real harness against
|
|
the real Chirone DWH + pgvector, on the 'ablazione' question (exercises D14 value
|
|
grounding + formula on a multi-column case), through the gate glue (widget-descriptor
|
|
emit/consume) that L1 cannot reach.
|
|
|
|
MODES (both manual, non-deterministic, pre-release -- NOT a regression gate):
|
|
- human-in-the-loop (default): the reviewer answers each gate widget via terminal.
|
|
- scripted answers (opt-in, --answers-file): canned reviewer answers for the
|
|
Altro/value-grounding/rollback paths so specific behaviors assert deterministically.
|
|
|
|
HOW TO RUN (operator, before release):
|
|
1. Populate harness/.env (THT_DWH_API_KEY, THT_VEC_API_KEY,
|
|
THT_VEC_WRITE_API_KEY, THT_SSL_CA, THT_*_REST_URL, THT_OLLAMA_URL).
|
|
2. Connect VPN. Ensure Pi is configured locally with GLM 5.2.
|
|
3. Run: pytest -m l2 tests/l2/test_session_ablazione.py -s
|
|
|
|
The full LLM->gate conversation is exercised manually here; this file provides the
|
|
prerequisite checks (workspace loads, env present, Pi on PATH) and documents the
|
|
session-assertions the operator confirms by inspection (sql_final.sql present and
|
|
read-only-validates; the ledger shows value_grounded or concept_formula_approved
|
|
-- i.e. D14 surfaced). Automated assertions grow as a fake-Pi driver lands.
|
|
"""
|
|
import os
|
|
import shutil
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
pytestmark = [pytest.mark.l2]
|
|
|
|
WORKSPACE = Path(__file__).resolve().parents[2] / "workspaces" / "tht-test.yaml"
|
|
QUESTION = "dammi la lista dei pazienti che hanno fatto un'ablazione nel 2025"
|
|
|
|
|
|
def test_workspace_chirone_test_loads(l2_env):
|
|
"""The L2 workspace YAML loads and expands ${THT_*} from .env."""
|
|
from tht.workspace import load_workspace
|
|
|
|
ws = load_workspace(WORKSPACE)
|
|
# secrets must be expanded (not the literal ${...} token)
|
|
assert not ws.rest.api_key.startswith("${")
|
|
assert not ws.vector_rest.api_key.startswith("${")
|
|
assert not ws.vector_write_rest.api_key.startswith("${")
|
|
|
|
|
|
def test_pi_binary_available(l2_env):
|
|
"""Pi must be on PATH to spawn the gate."""
|
|
assert shutil.which("pi") is not None, "pi not on PATH (configure Pi with GLM 5.2 first)"
|
|
|
|
|
|
def test_ablazione_session_manual(l2_env, tmp_path):
|
|
"""MANUAL end-to-end: GLM 5.2 + real DWH on the 'ablazione' question.
|
|
|
|
Launches pi --mode rpc from harness/, runs /nuova-domanda, and the reviewer
|
|
answers each gate widget via terminal. The operator confirms by inspection:
|
|
- a session is produced and finalizes
|
|
- sql_final.sql is present and read-only-validates against the DWH
|
|
- the ledger shows value_grounded OR concept_formula_approved (D14 exercised)
|
|
Run with: pytest -m l2 tests/l2/test_session_ablazione.py::test_ablazione_session_manual -s
|
|
|
|
The scripted-answers variant (--answers-file with canned reviewer responses for
|
|
the value-grounding / Altro / rollback paths) is a follow-up once the terminal
|
|
relay helper lands; for now this is human-in-the-loop.
|
|
"""
|
|
# Precondition: this is a manual, non-deterministic pre-release check, not a CI
|
|
# assertion. We surface the run instructions and assert only that the launch
|
|
# context is ready; the operator drives the conversation and inspects the outcome.
|
|
env_ok = all(os.environ.get(v, "").strip() for v in
|
|
["THT_DWH_API_KEY", "THT_VEC_API_KEY", "THT_VEC_WRITE_API_KEY"])
|
|
assert env_ok
|
|
assert WORKSPACE.exists()
|
|
print("\n[L2 manual] launch: pi --mode rpc (cwd=harness/)")
|
|
print(f"[L2 manual] /nuova-domanda \"{QUESTION}\"")
|
|
print("[L2 manual] confirm: sql_final.sql present + ledger has value_grounded/"
|
|
"concept_formula_approved. Session dir:", tmp_path)
|