Commit Graph
1296 Commits
Author SHA1 Message Date
marcopan de61034a8d feat(harness): riscrittura skill tht-sessione (F1-F8 + 4 sottomoduli)
Skill ex-novo che riflette Thoth (non copia di ChironeWp3):
- vocabolario widget-descriptor (reviewer_select/decide/confirm) invece di 'dialog native'
- D11 save-one in F2 (upsert mirato vs full resync)
- D14a value_grounded (LSH multi-colonna non collassa) + D14b concept_formula in F4
- D13 free-text e D15 rollback nelle discipline trasversali
- memory vive SOLO nel vectordb (drop registry, spec 5): save-one/promote senza registry
- F8 datamart onesto (stub NotImplementedError)

Sottomoduli cte/memoria/rewriting/sql-generation portati adattando nsp->tht, con
i vincoli precisi trasferiti fedelmente (max 5 memorie, solo 3 tipi riusabili,
CTE solo WITH senza SELECT, dim_time join non aritmetica, sql_final pulito).

Verifica: zero residui nsp/chirone/psd nella skill; ogni 'tht <cmd>' citato e'
registrato (correzione: 'tht formula retrieve' era inesistente -> riformulato in
'ricerca nelle evidence'). Suite: 165 passed.
2026-06-27 14:24:20 +02:00
marcopan 91a374492c feat(harness): port sql/cte/datamart/lsh cmd (Onda 4) — CLI completa F1→F8
Ultima onda CLI. 4 cmd portati con rename + grep-per-file (3 residui nsp nei messaggi
fixati). Nessun drift costanti phase in questi cmd.

La CLI tht e' ora COMPLETA: 14 gruppi di comandi (phase config schema session vector
memory search evidence db decision sql cte datamart lsh). tht --help li list tutti.
Suite: 165 passed.

Il loop skill->LLM->gate ora ha tutti i comandi che la skill chiamera'. Resta:
skill riscritta (S), setup pre-sessione (0b), sessione L2 manuale.
2026-06-27 14:16:35 +02:00
marcopan 99b01b0407 feat(harness): port memory/search/evidence/db/decision cmd (Onda 3.2)
5 cmd foglia portati con rename + drift fix:
- memory_cmd: portato col modello registry INTATTO (TODO marker per il drop registry
  decisione spec 5 — task separato, richiede L2 per validare il rewrite su vectordb)
- search_cmd: creata search_app sub-app (era funzione standalone in ChironeWp3),
  registrata come 'tht search find'
- evidence_cmd, db_cmd, decision_cmd: port verbatim

Drift fix decision_cmd: DECISION_MIN_PHASE.get(type,1) -> load_workflow().decision_min_phase(type).
Check grep-per-file: ~15 residui nsp/PSD_SSL_CA nei messaggi utente fixati (nsp <cmd>
-> tht <cmd>, nsp.yaml -> workspace yaml, PSD_SSL_CA -> THT_SSL_CA).

Suite: 165 passed. tht --help ora mostra 10 sottocomandi.
2026-06-27 14:14:48 +02:00
marcopan 159207a8f1 feat(harness): arricchisci metadata memory (subject/detail/rationale) — Onda 3.1 TDD
Correzione del gap ereditato (resosi NECESSARIO dal drop del registry, spec 5): il
metadata del VectorRecord memory ora porta subject/detail/rationale oltre a
type/session_id/tables/concepts. pack_metadata li serializza nel jsonb via
**record.metadata. search_similar proietta metadata completo -> la F2 ricostruisce
la decisione direttamente dall'hit, senza lookup registro.

L1: 4 test (subject/detail/rationale presenti, campi esistenti preservati,
no cross-contamination multi-record, save_one_memory propaga il metadata alla riga).
Suite: 165 passed.
2026-06-27 14:10:39 +02:00
marcopan 40b3bac6c6 feat(harness): port vector_cmd (Onda 2) — helper vector condivisi
Esporta make_embedder/open_store/open_searcher/require_vector_cfg usati da
memory/search/evidence cmd (Onda 3). 3 residui nsp nei messaggi fixati (grep-per-file).
2026-06-27 14:08:54 +02:00
marcopan 3120bdc192 feat(harness): port config_cmd + schema_cmd + session_cmd (Onda 1.3-1.4, radici CLI)
Le 3 radici intra-CLI, portate con rename psdwp3->tht + fix path import (session.phase
-> phase). Espongono gli helper condivisi: CONFIG_OPT (config_cmd), _load_config_or_exit/
physical_path/annotations_path (schema_cmd), session_dir/load_session_or_exit (session_cmd).

Drift phase fix in session_cmd (7 siti): MAX_PHASE -> wf.max_phase, PHASE_NAMES ->
wf.phase_name(), SCHEMA_LINKING_PHASE -> wf.schema_linking_phase(). Le vecchie costanti
non esistono piu' in tht.phase (sono metodi su Workflow dalla Onda -1/F2).

Check grep-per-file applicato: trovati e fixati 4 residui (config/nsp.yaml, 2x
'nsp schema introspect', 'Sessioni PsdWp3') che il sed psdwp3->tht non tocca. Lesson
del fix precedente applicata.

session_cmd importa sql_cmd lazy (dentro finalize_cmd) -> non si rompe finche' sql_cmd
(Onda 4) non sara' portato.

Suite: 161 passed. tht --help ora mostra phase, config, schema, session.
2026-06-27 14:08:08 +02:00
marcopan 16ec5a5138 fix(harness): 'datawarehouse Chirone' residuo in VENDORED.md (stessa causa Onda 0)
Stesso bug del precedente: VENDORED.md e' arrivato con Onda 0 (dopo l'Onda -1 che
aveva pulito i riferimenti cliente). Neutralizzato a 'il datawarehouse' (coerente
con le altre neutralizzazioni). Sweep completo tht/ ora vuoto per chirone/psdwp3/
policlinico/sandonato.
2026-06-27 14:02:39 +02:00
marcopan 1b8a13b864 fix(harness): nsp residuo nei moduli Onda 0 (lshindex msg + VENDORED.md)
User review ha trovato 2 residui 'nsp' sfuggiti al renaming: erano nei moduli
portati in Onda 0 (DOPO l'Onda -1 che aveva pulito), in messaggi utente/docstring
non in import. L'import-smoke di Onda 0 non li catturava (verifica solo import, non
stringhe). Corretti a tht: 'tht lsh build' (lshindex:61), 'tht schema introspect'
(VENDORED.md:25).

Lesson: dopo ogni port di file sorgente, grep di nsp su quel file, non solo import-smoke.
Suite: 161 passed, zero residui nsp nel codice.
2026-06-27 13:59:51 +02:00
marcopan a312746fc8 feat(harness): require_phase_or_exit + phase advance/reopen/show (Onda 1.2)
require_phase_or_exit: guard riscritto vs Workflow (load_workflow().phase_name invece
della costante PHASE_NAMES drift). Exit 1 se la sessione e' sotto soglia. Usato da
cte/decision/datamart cmd.

Comandi phase (portati + adattati al modello ThothII, non copia cieca):
- advance: persiste phase_approved; --auto exit 6 se la fase non e' completa
  (contratto col gate)
- reopen: persiste phase_reopened + teardown_to_phase degli artefatti oltre il target
- show: stato sessione (fase corrente, ultime decisioni)

session_dir helper tenuto qui (mirror di session_cmd) per evitare circular import.
_cfg() fa fallback a THT_WORKSPACE env finche' _load_config_or_exit (Onda 1.4) non
sara' portato.

L1: 4 test require_phase_or_exit (allow at/above, exit below, message con nome fase
dal workflow). Suite: 161 passed.
2026-06-27 13:15:12 +02:00
marcopan 37bb074efe feat(harness): Workflow.schema_linking_phase() (Onda 1.1, TDD)
Aggiunge il metodo che i cmd CLI useranno al posto della vecchia costante
SCHEMA_LINKING_PHASE (drift fix Onda 1). Ritorna il num della fase il cui
artifacts_out contiene schema_linking.json, default 5 se nessuna la dichiara.

L1: 4 test (fase reale F4, posizione arbitraria, default 5, artefatti multipli).
Suite: 157 passed.
2026-06-27 13:09:51 +02:00
marcopan 31552782c0 test(harness): L1 characterization tests per Onda 0 (sqlcheck, ctetest, execute)
44 test L1 sui 3 moduli backend con logica non banale (opzione 2 della user review):

- sqlcheck.validate_sql (16 test): parse/single-statement, read-only enforcement
  (INSERT/UPDATE/DELETE/CREATE/DROP/ALTER/TRUNCATE/GRANT rifiutati, WITH/UNION ok),
  forbidden functions (dblink default blacklist, custom set, allowed not flagged),
  object-existence (tabella inesistente, CTE non flaggata, perimetro promoted warning,
  colonna inesistente con alias). Documenta una limitazione reale: le funzioni
  aggregate specializzate (count/sum/coalesce) NON sono catturate dal name-matcher
  perche' sqlglot modella .name come argomento, non come nome funzione.

- ctetest (14 test): has_trailing_select (semantica controintuitiva: True = violazione),
  last_cte_name, build_test_sql, ledger I/O (load/append roundtrip, JSON-array e
  JSONL tolleranti, corrupt-ledger raise).

- execute._inject_limit (6 test): LIMIT iniettato quando assente (limit+1 per
  troncamento), rispettato quando presente, non iniettato su non-query, UNION/WITH ok.

Suite: 153 passed (109 + 44). Bonus: __psd_probe__ -> __tht_probe__ (riferimento
cliente neutralizzato in ctetest).
2026-06-27 13:04:34 +02:00
marcopan d857004f47 docs(plan): allinea piano alle 3 correzioni spec (drop registry, repo workspace, LSH)
- Task 3.2: nuovo Step 1b — drop funzioni registry da memory_cmd + tht/memory.py
  (load/save/update/delete/promote/reusable_promotions); promotion (F5) riscritta
  come upsert batch al vectordb; memory list/delete su vectordb (no registry).
- Skill F2/F5: drop riferimenti a 'memory promote + memory index' con registry.
- Onda 0b riscritta: repo workspace per-cliente separato (no copia in harness/),
  LSH scarica-tutti-i-valori-distinti (no 'campiona'), indice nel repo workspace cliente.
2026-06-27 12:52:58 +02:00
marcopan 20e3820bd7 docs(spec): drop registry memory + evidence repo separato + LSH scarica-tutto
Tre correzioni da user review:

5. Memory SOLO pgvector, niente registry. Il registry.jsonl di ChironeWp3 è vestigiale:
   una volta che il metadata del vectordb ha subject/detail/rationale (decisione 6), il
   registry non serve. Promotion (F5) = upsert diretto a vectordb. tht/memory.py non
   porta le 6 funzioni registry. 'Cancellare' = metadata.status='superseded' (audit
   trail; il writer e' upsert-only). Multi-workstation OK per costruzione.

7. Evidence: repo workspace separato per-cliente, NON dentro ThothII. ThothII e'
   generico; un repo tht-workspace-<cliente>/ contiene evidence/ + workspace YAML +
   indici LSH. Deploy = checkout ThothII + checkout workspace-cliente. Niente copia
   in harness/.

8. LSH: scarica TUTTI i valori distinti (non 'campiona'), costruisce MinHash+LSH
   dentro harness come preprocessing. Indice per-cliente (nel repo workspace cliente).

Sezione 3 punto 2 (F2) riscritta: save-one diretto, drop reference a promote/index
con registry. Arricchimento metadata ora 'obbligatorio' (non opzionale): senza
registry, il vectordb e' l'unica fonte. Residui nsp nei path spec corretti a tht.
2026-06-27 12:50:15 +02:00
marcopan ea6412fafc feat(harness): port backend Onda 0 — vendor, lshindex, sqlcheck, execute, rest/exec, ctetest, report, datamart
8 moduli leaf portati verbatim da ChironeWp3 con rename psdwp3→tht:
- vendor/thoth_lsh (MinHash/LSH, leaf puro datasketch+tqdm) + VENDORED.md
- lshindex/ (build/save/load/query, dipende vendor + LshConfig)
- sqlcheck/ (validate_sql, leaf ExecutionConfig+mschema)
- execute/ + execute/warnings (run_controlled/explain, leaf sqlglot+sqlalchemy)
- rest/execute + rest/explain (REST variants, dipendono execute+rest.client)
- ctetest (CTE test records, leaf sqlglot+pydantic)
- report (validation report rendering, dipende execute+sqlcheck)
- datamart (stub NotImplementedError)

Verifica: import smoke catena completa OK, pytest 109 passed. Deps (datasketch, sqlglot,
sqlalchemy, pydantic, requests, tqdm) già in pyproject. VENDORED.md neutralizzato
(riferimenti PsdWp3→Thoth).
2026-06-27 10:34:23 +02:00
marcopan fc5fbe6b65 refactor(harness): renaming prodotto tht (Onda -1)
Thoth (tht) è il prodotto, PSD è il cliente. Nessun riferimento al contesto
clinico nel codice.

Rinomine:
- comando+package nsp→tht (dir nsp/→tht/, 46 import, pyproject entry point)
- gate nsp-gate.js→tht-gate.js (+ rewrite token, relayIfNspFails→relayIfThtFails)
- workspace chirone.{example,test}.yaml→tht.{example,test}.yaml (generici)
- env THOTH_→THT_ (19 var) + NSP_ stragglers (NSP_HARNESS_ROOT, NSP_SESSION)
- commenti/docstring chirone/psdwp3/policlinico neutralizzati ('the reference
  implementation', 'the DWH')

Aggiunto [tool.setuptools.packages.find] include=['tht*'] (necessario: l'auto-
discovery rompeva con tht/ + workspaces/ come top-level multipli).

.env operatore aggiornato in-place (prefissi THT_, valori preservati, gitignored).

Verifica: pytest 109 passed, npm test 14 pass, tht phase meta --json OK, zero
residui nsp/THOTH_/NSP_/chirone nel package.
2026-06-27 10:33:16 +02:00
marcopan 0dcc0246dc docs(plan): tht porting CLI + skill — implementation plan
Piano da spec 2026-06-27-cli-port-completo-skill-riscritta-design.md.
Struttura in onde: -1 (renaming tht isolato), 0 (backend), 0b (evidence+LSH setup),
1 (radici CLI + phase drift), 2 (vector), 3 (foglia + metadata memory), 4 (SQL/CTE),
S (skill riscritta), L2 (sessione manuale).

TDD per logica nuova (schema_linking_phase, require_phase_or_exit, arricchimento
metadata memory); port+smoke+commit per i port verbatim (logica gia' validata in
ChironeWp3). pytest verde (109 passed) a ogni task come gate di regressione.

Self-review: copertura spec completa (11 decisioni), nessun placeholder, type
consistency verificata. Gap residui onesti: circularita' session_cmd<->sql_cmd
(risolto con import lazy), RPC server mancanti (fuori piano codice), L2 manuale.
2026-06-27 10:13:44 +02:00
marcopan 46dec04299 docs(spec): renaming prodotto tht come Onda -1 isolata
Renaming richiesto in user review: Thoth (tht) e' il prodotto, PSD e' il cliente.
Nessun riferimento al contesto clinico nel codice.

Decisioni 8-10:
8. Rinomine: nsp->tht (comando+package+46 import), nsp-sessione->tht-sessione,
   nsp-gate.js->tht-gate.js, chirone.*->tht.{example,test}.yaml (generici; il deploy
   cliente crea il suo psd.yaml non-committato), THOTH_*->THT_* env.
9. Neutralizzazione riferimenti chirone/psd/policlinico/sandonato nei commenti/
   docstring (resi generici o rimossi). Il contesto cliente vive SOLO nei file di
   config reali (.env gitignored, workspace cliente non-committato).
10. Onda -1 isolata PRIMA del porting CLI: pytest resta 109 passed (rename verificato
    da solo), poi il porting avviene col nome nuovo (niente doppio lavoro).

Ordine esecuzione aggiornato a 7 step (Onda -1 prima di tutto). Self-review:
corretti i residui incoerenti di nsp/chirone nello spec (righe che usavano ancora
i nomi vecchi dove dovevano essere tht). Residui rimasti sono legittimi (descrivono
il renaming o il path sorgente one-shot della copia evidence).
2026-06-27 10:00:54 +02:00
marcopan 19c646bb21 docs(spec): indipendenza ChironeWp3 + registro memory locale + evidence in ThothII
Tre decisioni su dipendenze implicite (domanda user review):

4. Indipendenza da ChironeWp3 (proprieta' architetturale): quando ThothII e' pronto,
   il server non deve avere ChironeWp3 — solo Supabase (RPC SECURITY DEFINER nel DB,
   indipendenti dal codice app) + cartella evidence (dentro ThothII). Verificato:
   codice ThothII non ha riferimenti ChironeWp3/psdwp3, .env non punta a path chirone.

5. Registro memory: locale per-workstation (registry.jsonl in harness/artifacts/memory/
   su ciascuna). La F2 legge dal vectordb condiviso e, grazie all'arricchimento metadata
   (decisione 6), ricostruisce la decisione senza lookup registro. Il registro serve
   solo per la F5 (promozione: locale + indicizza condiviso). Multi-workstation OK.

7. Evidence: dentro ThothII. La cartella (229 markdown statici curati, 11M, nessun ETL)
   si sposta in harness/evidence/. ThothII self-contained. Nota: revisionare per PII
   prima di committare.

Onda 0b aggiornata: cp evidence in harness/evidence/ invece di puntare path esterno.
2026-06-27 09:51:01 +02:00
marcopan 5e9553fde8 docs(spec): arricchisci metadata memory vectordb (no lookup registro)
Gap trovato in user review: la tabella vectors.memory aveva metadata
{type,session_id,tables,concepts} — mancavano subject/detail/rationale strutturati,
quindi l'hit vettoriale non bastava per applicare la memoria. ChironeWp3 faceva
lookup nel registro canonico via mem_id; stesso difetto ereditato in ThothII.

Decisione: arricchire il metadata del VectorRecord memory con subject/detail/rationale
(in memory_vector_records, nsp/memory.py). pack_metadata (rest_writer.py:26) li
serializza gia' nel jsonb via **record.metadata. Nessuna modifica al writer RPC,
nessuna modifica allo schema DB. search_similar proietta gia' metadata completo ->
la F2 ricostruisce la decisione direttamente dall'hit, senza lookup registro.

Momento ideale: tabella memory vuota, niente re-indicizzazione. Aggiunto come task
esplicito in Onda 3 (dove si porta memory_cmd). Registro globale resta source-of-truth
per la promozione (F5), ma la F2 legge solo dal vectordb.
2026-06-27 09:43:56 +02:00
marcopan 9f14a13594 docs(spec): aggiungi Onda 0b — setup pre-sessione evidence + LSH build
Buco trovato prima della user review: lo spec claims D14 value-grounding ed
evidence-based F4, ma non setup né evidence né l'indice LSH. Senza, la sessione
L2 girerebbe degradata (solo segnali vettoriali) e i claim sarebbero falsi.

Onda 0b (dopo Onda 0 + 4, prima della sessione L2):
- Evidence: cablare THOTH_DOCS_ROOT=/Users/mp/Chirone/chirone/etl/docs nel .env +
  blocco evidence nel chirone-test.yaml. La cartella esiste già.
- LSH: nsp lsh build sul workspace chirone-test (one-shot, richiede VPN + Ollama).
  Verifica: nsp search ritorna match multi-colonna + test_value_grounding_real
  smette di skip-piare.

Ordine esecuzione aggiornato (6 step), D14a value-grounding marcato 'sì (se Onda 0b)',
nsp lsh build tolto dal fuori-scope (ora dentro).
2026-06-27 09:38:38 +02:00
marcopan 96499e70d6 docs(spec): porting CLI completo + riscrittura skill nsp-sessione
Design approvato in brainstorming per sbloccare il loop skill→LLM→gate (oggi
non testabile: 11/12 cmd CLI mancanti + skill assente).

Decisioni chiave:
- Scope F1→F8 completo (tutti i cmd + 3 cluster backend + skill riscritta)
- Drift phase.py: riscrittura diretta dei 12 siti cmd che usano le vecchie
  costanti (MAX_PHASE/PHASE_NAMES/SCHEMA_LINKING_PHASE/DECISION_MIN_PHASE) ->
  load_workflow() + metodi Workflow. Niente wrapper.
- Skill: riscrittura completa ex-novo che riflette ThothII (widget-descriptor,
  D11 save-one, D13 free-text, D14 value-grounding/formula, D15 rollback),
  prendendo spunto da ChironeWp3 ma non copiandola.

5 onde topologiche (backend -> radici CLI -> vector -> foglia -> SQL/CTE).
Test: pytest verde a ogni onda + import smoke; sessione L2 manuale su domanda
complessa (cardioversione + ablazione same-year) che esercita F4 complesso,
D14, F6 CTE.

Spec self-reviewdato: corretta ambiguità su save-one (discriminante = comando,
non profilo dedotto dal modello) e nota onesta su F2 (memory azzerata = D11
validato in seconda sessione).
2026-06-27 09:32:42 +02:00
marcopan e58f6c092e fix(harness): workspace + write-URL + L2 tests per accesso REST reale
Bug trovato provando la connessione reale col .env: il write endpoint vive su un
PATH DEDICATO /vector/write/v1/ (non /vector/v1/), e il modello Config ha write_rest/
vector_write_rest a TOP-LEVEL (non nidificati in vector_db).

- .env.example: aggiunge THOTH_VEC_WRITE_REST_URL (path dedicato del writer, con
  avviso che le due chiavi valgono su path separati).
- workspaces/chirone-test.yaml: riscritto allineato a chirone.example.yaml + config.py
  (vector_rest/vector_write_rest top-level; write_rest punta a THOTH_VEC_WRITE_REST_URL).
- tests/l2/*: corretti gli accessi strutturali (ws.vector_write_rest invece di
  ws.vector_db.write_rest; ws.vector_rest invece di ws.vector_db.rest).
  test_value_grounding_real skip-when-import-fails su nsp.lshindex (modulo deferred da B3).

Verificato end-to-end: save_one_memory (embeddings -> writer REST /vector/write/v1/
-> upsert pgvector -> read-back reader) PASSED. Suite L0+L1: 109 passed. Suite L2:
4 passed, 1 skipped (lshindex deferred).

Nota operativa: THOTH_SSL_CA va lasciato VUOTO sulla workstation (cert GoDaddy
pubblico in certifi). I campi direct-transport (THOTH_DB_*, THOTH_VEC_PASSWORD)
sono obbligatori per il modello ma inutilizzati in transport=rest: riempiti con
dummy nel .env locale (come faceva ChironeWp3).
2026-06-27 08:20:36 +02:00
marcopan 276717005d fix(harness): drop THOTH_SSL_CA from REQUIRED_L2 + isolate profile in workspace test
Two fixes found while unblocking the L2 setup:

1. conftest: THOTH_SSL_CA is NOT an L2 prerequisite. The DWH endpoint presents a
   public cert (*.policlinicosandonato.it, signed by GoDaddy), already in the
   certifi bundle, so the REST clients validate TLS with verify=True -- no CA file
   needed. The ssl_ca line was commented out in ChironeWp3's nsp.yaml too.

2. test_workspace: the profile-default assertion collided with the operator's real
   harness/.env once load_dotenv (D3) started injecting THOTH_PROFILE into the
   process env. The test now dels THOTH_PROFILE to assert the actual *default*
   (server), regardless of what the operator set in .env.

Suite: 109 passed.
2026-06-27 06:45:00 +02:00
marcopan 50d5f9c9cf docs(harness): README + workflow editing + testing guide (D6)
README: install, configure (.env + workspaces/), the workflow, run inside Pi, the
three-level test commands, layout, references.

docs/workflow-editing.md: how to edit workflow.yaml (add/reorder/merge/skip phases,
advance kinds, prerequisite predicates, decision_min_phase derivation, artifacts_out
+ teardown) -- referencing spec §5.3. Emphasizes no mirrored constants (the F2 point).

docs/testing.md: the honest L0/L1/L2 split in plain language -- what each covers and
does NOT. States the headline plainly: the skill->LLM->gate loop has NO automated
regression coverage (L2 only, pre-release). Documents the fake-Pi follow-up as the
gap-closer. Security note on keys (.env gitignored, never logged, rotate leaked keys).
2026-06-26 23:20:06 +02:00
marcopan c861a0df9e test(harness): L2 tests -- ablazione session + value grounding + memory save-one (D4, D5)
Pre-release, non-deterministic tests (marker l2, skipped without .env + VPN). They
close the gaps L1 leaves open: real value grounding on the live schema, real memory
save-one upsert to pgvector, and the full GLM 5.2 -> gate conversation on the
'ablazione' question (which exercises D14 value grounding + formula on a multi-
column case + the gate glue L1 cannot reach).

workspaces/chirone-test.yaml points at the remote endpoints (DWH read-only +
pgvector dual-key, TLS self-signed); secrets via ${THOTH_*}.

- test_session_ablazione: precondition checks (workspace loads, env present, pi on
  PATH) + the documented manual run protocol (human-in-the-loop; scripted-answers
  variant is a follow-up). Default run skips cleanly.
- test_value_grounding_real: 'ablazione' grounds to multiple columns on the real
  schema (D14a non-collapsing), needs a built LSH index.
- test_memory_save_one_real: save_one_memory upserts one row via the writer key
  (D11) and search_similar retrieves it via the reader key.

Operator runs before release (pytest -m l2). Default run: 109 passed, 5 skipped.
2026-06-26 23:18:34 +02:00
marcopan 806bc510d7 test(harness): L2 marker + skip-when-no-.env guard (D3, Testing Strategy)
conftest loads harness/.env once (session, autouse) via python-dotenv, and exposes
an l2_env fixture that SKIPS (not fails) when any L2 prerequisite var is missing/
empty: THOTH_DWH_API_KEY, THOTH_VEC_API_KEY, THOTH_VEC_WRITE_API_KEY, THOTH_SSL_CA.
So the default run (pytest = L0+L1, addopts '-m not l2') stays green without .env;
only pytest -m l2 (pre-release, with .env + VPN) exercises them. l0/l2 markers were
registered in A9. tests/l2/ package created for the L2 tests (D4, D5).
2026-06-26 23:16:40 +02:00
marcopan d747f89d12 feat(harness): port .pi/ config, prompts, theme (D2)
settings.json (theme: thothii-mono), the two slash-command prompts
(/nuova-domanda, /riprendi-sessione), and the theme JSON. Renamed PsdWp3 -> ThothII
in prompt prose and the theme name. No tests (config files). pi --mode rpc launched
with cwd=harness/ finds the .pi/ directory + the extensions (nsp-gate.js, gate/).
2026-06-26 23:15:44 +02:00
marcopan d5c0fffc2a test(harness): L1 session-coherence smoke -- full walk + rollback (D1)
Pure-logic smoke (no LLM, no DB) that builds a synthetic ledger by hand and asserts
the Phase-A substrate stays coherent: full F1->F8 walk reaches terminal phase
(max+1); rollback truncates the effective view (stale phase-7 decision excluded
after reopen to F4) and resets current_phase; teardown deletes artifacts beyond the
target while preserving the target phase's; re-approve after rollback advances
correctly; taskdoc stays under byte budget across all phases; decision_retraction
excludes the retracted seq + the marker itself from effective_decisions.

This is the CI-runnable coherence net for the L2 session test (which exercises the
LLM->gate loop that L1 cannot).
2026-06-26 23:14:57 +02:00
marcopan 60b6e38096 feat(harness): rewrite nsp-gate.js glue wired to widget-descriptor builders (D2/D4)
Rewrite of ChironeWp3's gate extension. The pure widget-descriptor CONSTRUCTION
is in ./gate/builders.js (L1-tested, C1); this file is the GLUE -- it depends on
the Pi runtime (pi.on, pi.registerTool, ctx.sendRaw) and is verified end-to-end at
L2 (Task D4), NOT unit-tested here. A fake-Pi runtime mock (cross-cutting
follow-up) would let it run in CI.

PRESERVED VERBATIM (load-bearing runtime glue, spec D4):
- anti-bypass tool_call hook: FORBIDDEN (nsp phase advance|reopen, decision add,
  cte plan) + PROTECTED_FILES (review_decisions.jsonl, session_manifest.yaml,
  cte_plan.json)
- input lock + the input hook: /nuova-domanda|/riprendi-sessione entry detection,
  free-input block, the `!`-prefixed steer channel
- before_agent_start kickoff injection + the two kickoff payloads (model prose)
- agent_end prose safety net (nudges the model back to reviewer_* tools)
- session_start state reset
- exit-code contracts with the CLI (5 = gate refusal, 6 = needs human,
  7 = not-ready silent no-op)
- textResult / nsp() / relayIfNspFails / advanceIfReady helpers

TWO CORRECTIVE CHANGES vs source:
1. F2 single source: workflow facts (max_phase, phase names, schema-linking phase)
   come from `nsp phase meta --json`, NOT from JS-mirrored constants. The source's
   PHASE_NAMES array (truncated to 7) is gone; F8/datamart can no longer drift.
2. D2/D4 widget-descriptor: reviewer interaction is emitted as a widget-descriptor
   (built by ./gate/builders.js) and awaited by id via emitAndWait + the
   extension_ui_response dispatcher. This replaces the source's blocking native TUI
   primitives (ctx.ui.select/custom) and introduces the correlation-by-id layer
   ChironeWp3 never had.

Four tools wired: reviewer_select, reviewer_decide (persists via nsp decision add),
reviewer_confirm (gate; privileged action on approve), rewrite_question. Plus the
/torna slash command for rollback. No-limbo invariant preserved: cancel/undefined
re-presents the widget; real escapes are always in the descriptor's reserved field.
2026-06-26 23:12:51 +02:00
marcopan 7971d73628 feat(harness): pure widget-builder functions + JS golden/fuzzy tests (D2, L1)
The gate's widget-descriptor CONSTRUCTION, extracted into pure testable functions.
Each builder turns plain params into a ui_request descriptor (spec §4.1 taxonomy):
buildSelectRequest, buildMultiselectRequest, buildArtifactGate, buildInfoRequest,
buildFreetextRequest, withChildLinkage. No Pi context, no I/O -- the part of the
gate fully testable in L1 (in JS, in-language, no Python mirror).

Validation in the builders (not just happy-path): select requires title + array
options; multiselect allow_empty:false with zero options throws (a broken widget);
artifact-gate requires an artifact with a kind + a valid action.kind
(confirm/approve_reject/view_only); info level must be info/warning/error. The
Altro escape hatch with freetext linkage is always injected on blocking pick
widgets (no-limbo invariant).

L1: 14 node:test cases -- 3 golden files (select_F1, multiselect_F4,
artifact_gate_F5) pin the exact descriptor shape; fuzzy tests assert bad params
throw clearly rather than silently producing a broken widget.

package.json wires 'npm test' -> node --test (runs alongside pytest). The gate
GLUE (emission, anti-bypass, no-limbo loop) is C2, verified at L2.
2026-06-26 23:07:56 +02:00
marcopan e6eeb2ae3e test(harness): free-text rationale-capture contract (D13, §4.6)
D13 instructs the model to evaluate Altro/Rifiuta/steering free text in context,
act on it, re-ask if ambiguous, and record the user's words in the decision
rationale. The actual interpretation is model behavior enforced by the skill prose
+ gate, validated at L2; this test pins the RECORDING contract the gate relies on:
free text from 'Altro' round-trips into the decision rationale and survives
persistence, never silently discarded.

L1: test_freetext_interpretation (4 tests) -- Altro text preserved, persistence
roundtrip (exact), multiline steering intact, empty rationale allowed.

The skill prose ('Interpretazione del testo libero') ports with the .pi/ skill
in Phase D.
2026-06-26 23:04:22 +02:00
marcopan f104c015a1 feat(harness): SQL formula evidence -- concept->formula units + retrieval (D14b)
New evidence/formula_store.py: ConceptFormula (concept, columns, sql, status,
sources) as a frontmatter-YAML + SQL-body unit, stored one-file-per-formula under
<formulas>/<slug>-<n>.sql.md. save_formula is append-only (competing drafts and
reviewed versions coexist); retrieve_formula(concept) returns all of them so the
gate can surface candidates and let the reviewer choose.

concept_formula_approved / concept_formula_rejected added to DecisionType
(records the reviewer's choice; approved formulas travel with schema-linking).

L1: test_formula (7 tests) -- retrieval by concept, save/reload roundtrip (SQL
body preserved, frontmatter well-formed), multiple formulas per concept, empty
on no-match / missing dir, decision-type existence, default draft status.

Deferred: --kind formula on nsp search (needs search_cmd porting) wires
retrieve_formula into the CLI; lands with the search command.
2026-06-26 23:03:29 +02:00
marcopan c0285e55a0 feat(harness): value grounding -- multi-column LSH + value_grounded (D14a)
Ports search/__init__.py (combined_search/RRF/aggregate) renamed psdwp3->nsp.

New aggregate_lsh_multi (the D14a deviation): groups LSH hits by table keeping
EVERY column where a value appears -- NOT collapsed to a single best column.
The old _aggregate_lsh hid alternative groundings (e.g. 'ablazione' matching both
a boolean flag and a free-text patologia field). aggregate_lsh_multi exposes all
columns so the value-grounding widget lets the reviewer choose the anchor(s).
Within one (table, column) the best-scored value is kept; columns ordered by score.

value_grounded added to DecisionType (records the reviewer's anchor choice).

L1: test_value_grounding (6 tests) -- multi-column exposure, grouping, within-column
best-value, ordering, empty, and the value_grounded decision-type existence.

Deferred: lshindex/ (needs vendor/thoth_lsh) and the L0 test_rrf.py land with the
nsp lsh build command + index-building path; not needed for the pure L1 core here.
2026-06-26 23:02:11 +02:00
marcopan 9fa1e3e498 feat(harness): nsp memory save-one core — targeted upsert via writer key (D11)
The D11 deviation is a single-row pgvector upsert, not a full vectorstore resync.
Ports memory.py + session/{store,artifacts} + textutil (deps of memory), renamed
psdwp3->nsp. Decision import paths rewired from nsp.session.decisions to nsp.decisions
(our A4 port lives at the top level). session/models.py left UNCHANGED to preserve
the Phase-A ThothII additions (D12/D15 author/summary, D14a grounded_values,
D14b concept_formulas).

New in memory.py:
- memory_vector_record_for_decision(records, decision_seq): the single VectorRecord
  for a chosen decision (reuses memory_vector_records, filtered to one).
- save_one_memory(records, decision_seq, writer, embedder): embeds one record and
  calls writer.upsert_records('memory', [row]) -- NEVER writer.sync (that's the
  full-resync, server-side-only path). Returns the upsert count.

L1: test_memory_save_one (5 tests) pins the contract -- single row, one upsert
call, sync never called, None/0 for unknown seq.

Deferred: the full nsp memory save-one CLI command (config/session loading + the
workstation write-guard) lands when memory_cmd.py is ported alongside the other
CLI commands. The pure D11 core is what L1 can honestly cover here.
2026-06-26 23:00:06 +02:00
marcopan 796a39d893 feat(harness): port vectorstore dual-key + reader RPC (D11, §5.4)
Ports vectorstore/{rest_client,rest_writer,store,reader,embeddings,records},
evidence/model (leaf dep of records), and cli/_guards (require_vector_write_allowed
workstation write-guard). Renamed psdwp3->nsp, verbatim.

VectorRestClient gains an api_key property so reader/writer clients carry their
distinct keys visibly (spec D11: vector_reader / vector_writer on the same endpoint).

scripts/create_vector_reader_rpc.sql is NEW: the reader RPCs (search_similar,
list_tables) lived server-side in Supabase and were never versioned. Authored now
mirroring the writer allowlist pattern (table allowlist, security definer, revoke
from anon/authenticated, grant to vector_reader only). Writer RPC ported verbatim.

L1: test_vector_dual_key (7 tests) pins the dual-key construction + the workstation
write-guard (exit 4 without writer key).
2026-06-26 22:55:40 +02:00
marcopan eb3bde90e2 test(harness): L0 testcontainers + L1 contract tests for ported db/mschema/rest (A9, spec §1)
Ports the leaf data-layer modules and validates them:
- mschema/ (models, eligibility, merge, render), db/ (connection, sampling,
  introspect, fetch_ca), rest/client.py -- renamed psdwp3->nsp, verbatim.
- L0 (testcontainers, real Postgres): db connection read-only enforcement
  (psd_ro cannot CREATE/INSERT), introspect against a known schema (tables,
  columns, types, comments, FKs, enum, composite PK), sampling most-frequent
  values + truncation reporting. 15 tests, ~4s.
- L1 (fake data): rest/client RPC contract (mocked transport -- X-API-Key
  header, payloads, base_url slash handling, HTTP/network error surfacing),
  mschema/render 3 formats (markdown, mschema-text, schema-dict) +
  eligibility rules (wide_text excluded, short_text/numeric/enum/temporal/
  boolean eligible, annotation override wins). 25 tests.

pyproject registers l0/l2 markers + addopts '-m not l2' (L2 opt-in).

Deferred to their dependency-porting tasks: test_rrf.py (search needs
vectorstore, B3) and the 11 CLI contract tests (need _guards/session, wired
when each command lands). 'Not assumed reliable' now has real teeth for the
data layer; CLI/search contracts follow.
2026-06-26 22:53:08 +02:00
marcopan 5f24bd1adc feat(harness): port CLI skeleton + nsp phase meta --json (F2, kills JS/Python drift)
nsp.cli app registers phase_app (the gate's workflow-fact source). phase meta
--json emits {schema_version, max_phase, phases:[{num,id,name,advance,artifacts_out}]}
from load_workflow() -- the single source of truth. F8/datamart is present (the
exact JS-drift bug in ChironeWp3, PHASE_NAMES truncated to 7, is structurally gone).

Scope: the 12 other command groups land in their porting tasks (A9 ports
db/mschema/_guards; B1 vectorstore; B3 search/lshindex). Eager-importing them now
would break the app on unported deps -- deferred to keep the suite green at each commit.
2026-06-26 22:47:20 +02:00
marcopan a579cb88e5 feat(harness): taskdoc per-step generator with byte budget (D16)
Genera un documento compatto per fase, derivato da artefatti + effective_decisions,
con byte budget enforced (target <20k token per un 35B/<200k). Mai incorpora
physical.yaml (~190k token, fatale). D15+D16 complementari: il brief delle decisioni
e' effective-aware, quindi post-rollback riflette lo stato corretto (le stale di
fasi > current_phase sono escluse).
6 tests (question+schema, no physical.yaml, budget ok/violato, header fase,
stale-excluded post-rollback). 32 total passing.
2026-06-26 22:38:24 +02:00
marcopan 4b584934c8 feat(harness): teardown_to_phase -- artifact teardown on rollback (D15)
Cancella ogni artefatto la cui fase produttrice > target, usando artifacts_out di
workflow.yaml. Risolve il bug latente di ChironeWp3: ctes/*.sql orfani (non piu'
nel piano dopo un re-derive) restavano su disco e bloccavano finalize.
Da chiamare insieme all'append di phase_reopened per mantenere stato coerente.
6 tests (target 4/1/7, empty, missing, orfani CTE). 26 total passing.
2026-06-26 22:37:18 +02:00
marcopan 7d0562b82d feat(harness): phase.py rewrite + effective_decisions (D15 core fix, F2)
The single most important architectural fix vs ChironeWp3: ALL helpers consult
effective_decisions() instead of raw list_decisions(), so the reopen-aware view is
consistent everywhere (fixes the bug where approved_ctes/advance_problems conflated
stale pre-reopen decisions with new ones).

Model (corrected during TDD):
- current_phase folds the audit (excluding retracted) with guard 'n == cur' --
  already reopen-aware (old phase_approved:N after reopen to M<N don't advance).
- effective_decisions = decisions whose phase <= current_phase. A sql_approved at
  phase 7 is stale when current_phase=4 after a rollback to F4, even if in the ledger.
  Rollback to F4 does NOT invalidate decisions of phases 1-3 (they stay effective).
- decision_retracted markers excluded (audit-only).

Also: session/models.py ported (SchemaLinking + Candidate with grounded_values D14a
+ concept_formulas D14b). MAX_PHASE/PHASE_NAMES read from workflow.yaml via
load_workflow() (no duplication). Strada 2: ladder if-phase-N kept for now,
generic prerequisites evaluator (F2 full) deferred.

7 phase tests + 20 total passing.
2026-06-26 22:35:43 +02:00
marcopan fbcd694ace feat(harness): decisions.py ported + decision_retracted for step rollback (D15)
- ported from ChironeWp3 (22 DecisionType, append-only jsonl, monotonic seq)
- added decision_retracted type + retracts field for step-level rollback (D15):
  the retracted decision stays in the audit log, effective_decisions() (Task A5)
  will exclude it from the active view
- 5 tests: retract marker + monotonic seq + retracts default + empty session +
  literal includes retracted. All 13 harness tests pass.
2026-06-26 22:30:16 +02:00
marcopan 4a1272fa0a feat(harness): workflow.yaml as single source of truth + workflow.py loader (F2)
- workflow.yaml: 8-phase definition, data-driven prerequisites, no hardcoded ladder
- workflow.py: load_workflow() reader; max_phase=len(phases), decision_min_phase
  derived from prerequisites scan (no duplication)
- 6 tests: phase count, decision_min_phase derivation, name lookup (incl. F8
  presence -- the JS drift bug structurally impossible now), artifacts_out,
  advance strategy, schema_version. All 8 harness tests pass.
2026-06-26 22:29:17 +02:00
marcopan 2e34ba33e0 docs(spec): allinea §5.1 workspace YAML al Config reale (decisione B, post Task A2)
La prima stesura usava una struttura 'ideale' (relational/vector_db.collection/
embeddings.provider) che non combaciava col modello Config portato da ChironeWp3.
Allineato alla struttura reale (database/rest/vector_rest/vector_write_rest/
vector_db top-level). Aggiunta nota di allineamento + modello delle key D11.
2026-06-26 22:27:05 +02:00
marcopan 6ba3336775 feat(harness): config.py + workspace.py (D3 confine) + chirone.example.yaml
- config.py ported from ChironeWp3, PSD_PROFILE -> THOTH_PROFILE; dual vector key
  (vector_rest/vector_write_rest top-level) preserved verbatim
- workspace.py: thin boundary wrapper (D3) around load_config
- workspaces/chirone.example.yaml: aligned to the real Config shape
- tests/test_workspace.py: 2 tests (env expand + dual key; missing env raises)
- .env.example: added DWH + DOCS_ROOT vars
All tests pass (2/2).
2026-06-26 22:25:00 +02:00
marcopan 72a2c060d4 feat(harness): scaffold project (pyproject, env, gitignore) 2026-06-26 22:17:10 +02:00
marcopan 561e7aef08 chore: init ThothII repo — gitignore references, baseline docs (PRD, spec, harness plan) 2026-06-26 22:00:43 +02:00