ThothII/docs/superpowers/plans/2026-06-30-cross-model-behavior-matrix.md

Plan — Workstream G: cross-model behavior matrix

Scoping doc (not yet executed). The UI/UX redesign workstreams D, E, B, C, F, A are done and on main. G is the final, separate piece: prove the model-facing workflow is robust across any configured model, not just GLM 5.2 — and close the two live checks deferred into G:

• F's live check — does the model actually emit a single-pick reviewer_select carrying a decision payload (auto-confirm), with no redundant follow-up gate, and the decision landing in review_decisions.jsonl?
• A's resume robustness — does each model chain into the resume bootstrap tool calls in-turn (no narrate-and-stop), as GLM 5.2 now does on pi 0.79.4?

Why this is its own workstream

A2 already showed model behavior is the variable that matters (the resume stall was a model/pi artifact, not a code bug). The gate contract (kickoffs, reviewer_* widgets, SKILL discipline) is model-agnostic by design, but only GLM 5.2 has been exercised live. Weaker/non-thinking models are the realistic failure surface: turn-dropping, narrate-and-stop, stringified-array tool args, ignoring the auto-confirm decision payload, choosing the wrong widget.

Models in play (from backend /models, 2026-06-30)

Plan name	id(s)	tier / notes
GLM 5.2	zai/glm-5.2	baseline (verified live); thinking
Deepseek V4	deepseek/deepseek-v4-pro, …-flash	pro + a cheaper/faster flash
Qwen3.6	aritmolab/qwen3.6-35b-a3b	local AritmoLab endpoint; MoE
(others)	zai/glm-4.7, glm-5.1, gemma4-26b	breadth / regression coverage

Prioritise the three named primaries first (GLM 5.2 baseline, Deepseek V4 pro, Qwen3.6), then add one non-thinking / smaller model (gemma4 or glm-4.5-air) to probe the weak end.

Method — two tiers (cheap signal first, full run sparingly)

Tier 1 — clean-room first-turn harness (fast, ~15-30s/run, cheap). Generalise the A2 repro-driver.mjs (the proven clean-room driver: spawn pi --mode rpc with the backend env, send set_model {provider,modelId} → set_thinking_level {level} → prompt, capture raw JSONL, classify the first turn). Parameterise over (model, thinking, scenario) and record: CHAINED_INTO_TOOLCALL vs STALLED_TURN_ENDED_IDLE, time-to-first-tool, the first tool name, and the assistant-text tail. Scenarios that only need the FIRST turn:

• new-question kickoff — first tool should be read (SKILL.md). (kickoff robustness)
• resume kickoff — first tools should be tht session show + read SKILL.md in-turn. (A)

Tier 2 — full live run via Playwright (slow, ~3-4 min/F1, run sparingly). Only for the end-to-end widget/persistence checks that need a real gate round-trip. Drive ONE canonical session per model partway through F1 and assert:

• single-select (F) — the model emits reviewer_select whose chosen option carries a decision; answering it persists ONE decision (review_decisions.jsonl) and shows no follow-up confirmation widget; an "Altro"/"back" answer is still handled (non-persisting).
• multiselect — a genuinely multi-answer ambiguity uses reviewer_decide (checkboxes).
• thinking vs non-thinking pacing — note latency / whether non-thinking models skip steps.

What to record

A matrix (model × scenario) → {verdict, time-to-first-tool, widget/decision correctness, notes}. Capture in PROJECT_STATE + a new thothii-cross-model-matrix memory. For every model that misbehaves, harden the model-facing prompt (kickoffs in tht-gate.js, discipline in SKILL.md) and re-run — never special-case per model in code; keep the contract uniform.

Deliverables

1. A reusable matrix harness (promote the A2 driver out of scratchpad into e.g. harness/scripts/model-matrix.mjs, committed).
2. The filled results matrix (PROJECT_STATE + memory).
3. Prompt hardening PRs where a model diverges, each re-verified.
4. A short "supported models" note (which models drive the workflow reliably; which to avoid).

Results (executed 2026-06-30)

Harness committed at harness/scripts/model-matrix.mjs. Throwaway sessions used + deleted; the two real psd sessions were never driven.

Tier 1 — kickoff + resume first-turn (medium thinking), all available models CHAINED in-turn:

model	new (→read SKILL)	resume (→tht session show+read)
zai/glm-5.2	✅ 10.9s	✅ 14.5s
deepseek/deepseek-v4-pro	✅ 8.9s	✅ 7.8s
deepseek/deepseek-v4-flash	✅ 5.9s	✅ 6.4s
aritmolab/qwen3.6-35b-a3b	✅ 6.5s	✅ 7.0s
zai/glm-4.5-air	✅ 18.3s	✅ 11.3s
aritmolab/gemma4-26b-a4b	⚠️ MODEL_ERROR — 404 "model does not exist" at the endpoint (in the registry but not served); not a workflow issue

→ The kickoff contract is model-agnostic across the available fleet; the resume cold-start stall does not recur on any model (closes A's cross-model robustness). No prompt hardening needed.

Tier 2 — F single-select auto-confirm, live on baseline zai/glm-5.2: F1 reached the first reviewer_select ("cosa significa 'fibrillazione atriale'?") at ~321s; answering a concrete option drove review_decisions.jsonl 0 → 1 (a full concept_clarified decision persisted directly) with no follow-up confirmation gate. → F's auto-confirm contract verified end-to-end.

Supported-models note: GLM 5.2 (baseline), Deepseek V4 (pro + flash), Qwen3.6 35B, and the lighter GLM-4.5-air all drive the workflow reliably. gemma4-26b-a4b is listed but not served by the AritmoLab endpoint (404) — exclude until the endpoint provides it. Tier-2 per-model F/ multiselect behavior beyond the baseline remains a cheap future add (re-run with each model).

Risks / notes

• psd is the active workspace (real client data). Use throwaway sessions + delete after (the A2 pattern); never drive turns on the user's real sessions.
• Cost/time: Tier 1 is cheap; cap Tier 2 to one full F1 per model. Local AritmoLab models (Qwen3.6, gemma4) may need the AritmoLab endpoint reachable (VPN) and may differ on tool-call formatting — prepareReviewerArguments already parses stringified-array args, a known quirk.
• Non-thinking models may need thinking:"low"/none; sweep the thinking level as a variable.
• Strictly separate from the shipped workstreams: G changes prompts/docs only, never the gate's control flow.