mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(harness): cross-model behavior matrix (G) — harness + results

Browse Source 
Tier 1 (clean-room first-turn harness, harness/scripts/model-matrix.mjs):
kickoff + resume chain in-turn on ALL available models — zai/glm-5.2,
deepseek/deepseek-v4-{pro,flash}, aritmolab/qwen3.6-35b-a3b, zai/glm-4.5-air.
The resume cold-start stall recurs on none (closes A's cross-model robustness).
aritmolab/gemma4-26b-a4b is a 404 at the endpoint (listed but not served) — an
availability gap classified as MODEL_ERROR, not a workflow issue.

Tier 2 (live, baseline zai/glm-5.2): F single-select auto-confirm verified
end-to-end — answering the first reviewer_select persisted a concept_clarified
decision (review_decisions.jsonl 0->1) with no follow-up confirmation gate.
Closes F's deferred live check.

No prompt hardening needed. Results in the G plan doc + memory. Throwaway psd
sessions used and deleted; real sessions untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
marcopan
2026-06-30 18:46:55 +02:00
co-authored by Claude Opus 4.8
parent e8cdd0037c
commit cbb8e184f1
3 changed files with 201 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
PROJECT_STATE.md
+12 -5
View File
@@ -122,12 +122,19 @@ all pushed to origin. A (Resume menu + stall diagnosis/hardening) DONE, uncommit
`tht session show`+`read SKILL.md` in-turn). The earlier narrate-and-stop predates the pi upgrade.
Defense-in-depth applied: `RIPRENDI_KICKOFF` hardened to force the in-turn tool call (gate test +
live regression 2/2). The cross-model angle (weaker/older models) lives in **G**.
- **G (last, separate):** cross-model behavior matrix (Qwen3.6 / GLM 5.2 / Deepseek V4 / others) —
also the home for **F's live check** and **A's cross-model resume robustness**.
- **G — DONE** (uncommitted): cross-model behavior matrix via a committed clean-room harness
(`harness/scripts/model-matrix.mjs`). **Tier 1** — kickoff + resume first-turn: all *available*
models chain in-turn (`zai/glm-5.2`, `deepseek/deepseek-v4-{pro,flash}`, `aritmolab/qwen3.6-35b-a3b`,
`zai/glm-4.5-air`); the resume stall recurs on none (closes A's cross-model robustness).
`aritmolab/gemma4-26b-a4b` = **404 unavailable** at the endpoint (listed but not served) — infra
gap, not a workflow issue. **Tier 2** — F single-select auto-confirm verified live on `glm-5.2`:
answering the first `reviewer_select` persisted a `concept_clarified` decision **0→1** with **no
follow-up gate** (closes F's deferred live check). Full results: the G plan doc + memory
`thothii-cross-model-matrix`. No prompt hardening needed.
**Status:** A-G core work done (D, E, B, C, F, A). **Remaining:** **G** (cross-model matrix,
incl. F's live single-select check + A's resume-on-weaker-models), plus a one-off manual Playwright
kebab→resume pass through the live UI (open item 2).
**Status:** **All UI-redesign + resume workstreams done — D, E, B, C, F, A, G.** D/E/B/C/F/A pushed;
**G uncommitted** (harness + docs + PROJECT_STATE). Remaining nice-to-haves: Tier-2 F/multiselect for
the non-baseline models (cheap re-run), and a one-off manual Playwright kebab→resume pass in the UI.
## Live verification + reviewer_select fix (2026-06-30, afternoon)