Commit Graph
57 Commits
Author SHA1 Message Date
User 6dbf93fff9 feat: harden runtime readiness and session workflow 2026-07-14 10:27:25 +02:00
marcopan 4cf0adbdd2 fix(preprocess): initialize fresh DWH writers safely 2026-07-12 07:07:31 +02:00
marcopan 22e806a41b fix(preprocess): restrict DWH root claims to writers 2026-07-12 07:00:51 +02:00
marcopan 8110793f61 fix(evidence): make result summaries safe 2026-07-12 06:50:23 +02:00
marcopan b43075289e fix(preprocess): bind DWH artifacts to workspace 2026-07-12 06:46:06 +02:00
marcopan 0a2421c513 fix(evidence): bound reports and fail failed jobs 2026-07-12 06:45:19 +02:00
marcopan efcb0deb31 fix(evidence): lock GC and preflight search ownership 2026-07-12 06:07:30 +02:00
marcopan 6bb158233f fix(preprocess): lease DWH generation reads 2026-07-12 05:58:12 +02:00
marcopan db35ddb041 fix(evidence): bind corpus to workspace identity 2026-07-12 05:57:23 +02:00
marcopan 78ff360882 fix(preprocess): reconcile durable DWH generations 2026-07-12 05:51:49 +02:00
marcopan 24f6a5daa6 fix(preprocess): publish DWH artifacts atomically 2026-07-12 05:43:57 +02:00
marcopan 3a9808f126 feat(preprocess): add resumable DWH jobs 2026-07-12 05:34:38 +02:00
marcopan f3b49f41c8 feat(evidence): add safe retention and materialized reads 2026-07-12 05:02:47 +02:00
marcopan 1d5f8c76a7 feat(preprocess): resume evidence jobs by run id 2026-07-12 04:39:32 +02:00
marcopan 981b149249 fix(evidence): enforce active vector generations 2026-07-12 04:33:07 +02:00
marcopan b96d4f13b9 feat(preprocess): publish incremental Evidence corpus 2026-07-12 04:26:24 +02:00
marcopan 3588a7749b fix(vector): harden packaged migrations 2026-07-12 01:32:33 +02:00
marcopan c0d50e9b08 feat(vector): version pgvector schema 2026-07-12 01:21:29 +02:00
marcopan 7f3f6f7ce7 fix(storage): harden portable path diagnostics 2026-07-11 21:25:03 +02:00
marcopan eacb139e22 feat(storage): resolve portable workspace roots 2026-07-11 21:21:06 +02:00
marcopan 45d57756aa fix(core): restore adapter command behavior 2026-07-11 21:04:53 +02:00
marcopan a35efa16de fix(core): preserve adapter command contracts 2026-07-11 21:00:50 +02:00
marcopan dbbab6d005 refactor(core): route integrations through adapter factory 2026-07-11 20:52:41 +02:00
marcopanandClaude Fable 5 e24b41b156 feat(opt): three efficiency levers for NL→SQL workflow
Lever 1: Join-graph via FK logics in annotations + suggest-fks command
  - TableAnnotation.foreign_keys field stores curated logical FKs (DWH has no FK constraints)
  - tht schema suggest-fks: mine from approved SQL, heuristics (time_key → dim_time),
    same-name discovery + explicit --assume flag for multi-owner PKs
  - mschema renders 【Foreign keys】 section populated; validation in merge.py
  - SKILL.md F4 now reads FKs from mschema-text, no custom data_time_key logic

Lever 2: Context-pack consolidation at kickoff (tht search pack)
  - Single embedding of question, reused for schema + evidence + solved searches
  - One command: tht search pack <question> --session <id> → retrieval_pack.md
  - Graceful degradation when Ollama/vector store unreachable (exit 0, empty sections)
  - SKILL.md F1 prescribes as first call; reduces model thinking turns via pre-retrieval

Lever 3: Phase-summary recap v2 auto-construction from session ledger
  - tht session show --json includes full decisions ledger
  - tht phase meta --json exports 'emits' (substantive decision types per phase)
  - Gate appends deterministic 【Decisioni registrate in questa fase】 section (appendLedgerSection)
  - Model authors only summary + checks; recap table comes from persisted state (exact by construction)
  - SKILL.md Disciplina 6: brief model output, gate fills the rest

Tests: 358 Python (including 10 FK + 3 pack + 1 session-ledger tests) + 111 JS gate tests, all pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 17:43:08 +02:00
marcopanandClaude Fable 5 34aeda006e perf(f1): cache-guard schema introspect + F1 toolbox in skill (~-5 min per session)
Transcript analysis (session 2026-07-06-175012, GLM 5.2) showed F1 at 567s:
182s wasted on a useless `tht schema introspect` (re-introspecting the remote
DWH although physical.yaml was already materialized) plus ~220s of model
thinking inflated by ~7 exploratory turns (--help/find/cat). The actual
searches cost ~15s; reviewer gates (~145s, untouched) are the quality contract.

- schema_cmd.py: introspect now exits 0 with "OK (cache)" in ~1s when
  physical.yaml exists; --refresh forces the real re-introspection.
  Deterministic cross-model guarantee, verified live on psd (163 tables, 1.2s).
- SKILL.md: F1 toolbox (only `tht search find` + `tht schema render`; no
  introspect/--help/filesystem browsing; batch all searches in one turn);
  F4 step 1 is render-only with a one-shot introspect fallback.
- tht-gate.js: `tht schema introspect ... --refresh` added to FORBIDDEN
  (maintenance stays shell-only, never in-session).
- tests: 4 new pytest cases (cache hit placement proven with fake credentials,
  refresh bypass, corrupt-catalog fall-through, render fallback message) and
  2 gate anti-bypass JS cases.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 16:02:45 +02:00
marcopanandClaude Fable 5 a26f16ad79 feat(vector): server-side kinds filter for search_similar (legacy fallback) + graceful solved-search degrade
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 14:23:28 +02:00
marcopanandClaude Fable 5 8d427a2ccb fix(gate): actionable recovery for mid-promotion failures; truthful solved-index copy; fast-follow notes
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 14:07:00 +02:00
marcopanandClaude Fable 5 4c4099df09 fix(finalize): report the unchanged (no-upsert) solved-question case instead of staying silent
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 13:50:53 +02:00
marcopanandClaude Fable 5 2742a1fc42 feat(finalize): auto-index the question->SQL pair (best-effort, never blocks)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 13:48:02 +02:00
marcopanandClaude Fable 5 1f429b6b35 feat(cli): tht memory solved-index / solved-search (question->SQL exemplars)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-07 13:43:52 +02:00
marcopan 9b4f6b9804 feat(harness): persist CTE preview rows, add tht cte info, --doc for cte plan
WS1 of review-gates-v2: gives the JS gate (WS2) deterministic data to build the
cte_result v2 payload.

- CteTestRecord gains optional preview_rows (JSON-coerced, truncated cells);
  test_cmd populates it from the bounded result rows.
- New read-only `tht cte info <name> --session <id> [--json]`: plan
  index/total, persisted .sql, cte_plan_doc.json entry (if any), last
  CteTestRecord. Exits 1 with a clean stderr message on missing
  session/plan/name/sql.
- `tht cte plan --doc -` validates a chain-doc JSON (ctes[].name must match
  --name, same order) and writes it to cte_plan_doc.json; cte_plan.json stays
  a plain list[str] (load-bearing for tht.phase.next_cte). --doc is optional.
2026-07-07 00:23:50 +02:00
marcopanandMarco Pancotti d942635085 fix(f4): harden schema-columns enrichment + ValidationError guard + staged-helper note 2026-07-06 23:25:00 +02:00
marcopanandMarco Pancotti 3ed570ee60 feat(tht): promoted_columns_for helper (curated column set) 2026-07-06 23:25:00 +02:00
00365cc6e6 feat(tht): sync-schema-linking projects F4 ledger into schema_linking.json
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 23:25:00 +02:00
marcopanandMarco Pancotti 9339a8272b feat(tht): schema columns reader (name/description/type/pk) from catalog 2026-07-06 23:25:00 +02:00
marcopanandClaude Opus 4.8 00777922d8 feat(cli): 'tht session set-schema-linking' (file/stdin, validated)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 16:44:36 +02:00
marcopanandClaude Opus 4.8 47a52170ce feat(cte): add 'tht cte next' — first unapproved plan CTE
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-01 16:29:52 +02:00
marcopanandClaude Opus 4.8 c12bdcd257 feat(harness): derive a 3-5 keyword session name (YAKE, no LLM)
New sessions get a concise Italian-keyword `name` instead of the truncated
question. `tht session new` (when no --name is given) derives it via a new
`_extract_name` helper using YAKE (pure-Python, unsupervised, Italian, no LLM),
dropping generic query verbs and keeping the top keywords in reading order;
falls back to `_summarize` if YAKE is unavailable. `create_session` core keeps
its `name=None` default — the policy lives at the CLI layer.

TDD: tests/test_session_name.py (unit + CliRunner integration). Full harness
suite 269 passed; ruff clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-30 15:54:54 +02:00
marcopan 7e19f81173 feat(harness): tht ollama ensure CLI command (hard-fail preflight) 2026-06-29 19:25:23 +02:00
marcopan 9c39ea129e fix(harness): ensure_ollama never raises on start/probe failure (stage server) 2026-06-29 19:19:26 +02:00
marcopan 5f5cac05d5 feat(harness): EmbeddingsConfig bin/start_cmd + ensure_ollama preflight orchestration 2026-06-29 19:15:28 +02:00
marcopan 03053b0716 feat(harness): tht session documents --json read command 2026-06-29 12:25:40 +02:00
marcopanandClaude Opus 4.8 26337d0e32 feat(harness): tht session set-name/set-group/archive/unarchive/delete commands
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 12:20:43 +02:00
marcopan 60fa665d73 feat(harness): manifest archived/group fields + session mutation helpers 2026-06-29 12:16:52 +02:00
marcopanandClaude Sonnet 4.6 d630cac3ab feat(task-10): SQL routes + /workspaces + /models + fix sql preview path bug
Harness:
- preview_cmd FILE positional arg made optional; when omitted with --session,
  path is derived via _session_sql_file (mirrors export_cmd) — fixes the
  deferred Task-5 bug where the backend passed sessions/<id>/sql_final.sql
  relative to harnessDir, which broke for workspace-dependent paths.
- New pytest: test_preview_session_no_file_resolves_sql_final

Backend:
- ThtRunner.sqlPreview: drop positional file arg; use --session only
- New routes/sql.ts: POST /sessions/:id/sql/preview + /export
- New routes/meta.ts: GET /workspaces (yaml scan) + GET /models (injectable
  seam + graceful fallback to {models:[]})
- app.ts: register sqlRoutes + metaRoutes; add listModels to BuildAppDeps
- tht-runner.test.ts: add sqlPreview argv assertion (no file path)
- test/routes-sql-meta.test.ts: 9 tests (sql preview/export + meta routes)

Tests: harness 233 passed; backend 29 passed; build clean.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-27 21:31:35 +02:00
marcopanandClaude Sonnet 4.6 5d9a0bb548 feat(harness): manifest provider/model/thinking/name + session new options (BE-6/7)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-27 20:29:40 +02:00
marcopanandClaude Sonnet 4.6 a10a872a99 feat(cli): tht session list --json + session show --json (Task 7)
- Add _list_sessions(sessions_root) pure helper: scans sessions dir,
  returns list[dict] with id/status/question/summary/created_at/
  updated_at/author, sorted by created_at desc.
- Add `tht session list` command: --json emits pristine JSON array,
  human mode prints one line per session.
- Add --json flag to `tht session show`: emits manifest
  (model_dump by_alias) + phase (current_phase) + has_schema_linking.
- Tests: 8 tests in test_session_list_json.py (TDD red→green).
- Full suite: 230 passed (--ignore=tests/l2).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-27 20:23:46 +02:00
marcopanandClaude Opus 4.8 85e2fdc00a fix(harness): correct truncated signalling for sql preview offset>0
When offset>0 the wrapper added an outer LIMIT N, so run_controlled's
_inject_limit bailed (a LIMIT IS present) and truncated was always False —
AGGrid could never detect more rows. Fix: for offset>0 probe with LIMIT (N+1)
OFFSET M, then compute truncated = len(rows) > N in do_run and slice back to N.
offset==0 path unchanged (delegates to extracted _run_transport helper). JSON
still reports the user's requested limit N and correct truncated. Adds 3 tests
exercising the real do_run offset>0 path (N+1 -> truncated True, N -> False,
offset==0 verbatim). 222/222 passing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 20:18:58 +02:00
marcopanandClaude Sonnet 4.6 7d9769cfff feat(harness): tht sql preview --json + --offset for AGGrid paging (BE-2)
Add inject_limit_offset (tht/execute/limit.py) — pure subquery wrapper that
applies LIMIT/OFFSET non-destructively without clobbering user-supplied LIMITs.
Wire offset param into do_run (pre-processing when offset>0) and add --offset /
--json flags to preview_cmd; JSON mode emits pristine stdout with columns, rows,
execution_ms, truncated, limit, offset. 5 new tests (4 unit + 1 JSON-purity),
219/219 total passing (no regressions).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-27 20:13:20 +02:00
marcopanandClaude Opus 4.8 c4d130828f fix(harness): remediation difetti review — gate↔CLI, D15, D7/D6, D14, robustezza
Implementazione del piano di remediation progressiva sui difetti emersi
dall'analisi dell'harness. Tutto verificato: 214 test Python (incl. L0 su
Postgres reale), 14 test JS del gate, ruff pulito.

Blocco 1 (CRITICA, integrazione gate↔CLI):
- phase advance: gate usa --auto + exit 6; reviewer_confirm kind:phase fa
  advance esplicito che applica i prerequisiti (prima non avanzava per le
  fasi a conferma umana).
- cte plan riceve i --name dal gate (param names); set-question con id
  posizionale; skill `tht search find`; nuovo comando `tht memory save-one`
  con dedup hash client-side in save_one_memory.

Blocco 2 (D15, stato post-rollback):
- campo `phase` su DecisionRecord + effective_decisions phase-aware per i
  subject "a nome" (cte_approved ecc.); _compute_promotions e finalize sulla
  vista effective; finalize confronta col piano CTE effettivo, non glob;
  `decision add --retracts` + comando `decision retract`.

Blocco 3 (D7 read-only + D6 manifest):
- assert_read_only su tutti e quattro i codepath (direct + REST);
- manifest author/summary/updated_at/updated_by/schema_version popolati +
  helper touch_manifest sulle mutazioni.

Blocco 4-5 (D14a/D14b):
- decision_min_phase data-driven via `emits:` in workflow.yaml;
- formula evidence: status auto, search_formulas, gruppo CLI `tht formula`,
  `search find --kind formula`, load_evidence_dir salta i .sql.md.

Blocco 6 (robustezza):
- taskdoc slice promoted_tables + bound enforced; report escaping/bound +
  rsplit note; filtro kind reader REST/direct; conteggio upserted robusto;
  guard REST run_query non-list; LSH disallineato -> LshIndexError.

Blocco 7 (pulizia):
- dead code gate e KIND_TO_TABLE morto rimossi; doc Postgres-only
  (README + connection.py).

Blocco 0 (parziale): test di compatibilità firma gate↔CLI
(tests/integration). Rinviati: fake-Pi runtime completo, artifact-gate da
disco (#23), parità eligibility REST/direct (#28), unificazione
reserved-labels (#30), memory_rejected da deselezione (#33).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-27 17:16:51 +02:00