diff --git a/docs/superpowers/specs/2026-07-06-f4-schema-linking-column-curation-design.md b/docs/superpowers/specs/2026-07-06-f4-schema-linking-column-curation-design.md new file mode 100644 index 00000000..4c00f9c7 --- /dev/null +++ b/docs/superpowers/specs/2026-07-06-f4-schema-linking-column-curation-design.md @@ -0,0 +1,220 @@ +# F4 schema-linking — per-table column curation — design + +Date: 2026-07-06 +Status: approved (brainstorming), pending plan + +The F4 "Schema linking: tabelle/colonne/esclusioni" reviewer gate currently lets the +reviewer only promote/exclude **tables**; columns are chosen by the model and never +surfaced for review. This feature lets the reviewer, per promoted table, curate the +**columns** to use — over the table's full catalog column list — with the model's +suggested columns pre-selected. The curated selection is **persisted** (into +`schema_linking.json` + the decision ledger) and guides SQL generation downstream. +Excluded tables get a read-only column view. + +Decisions locked in brainstorming: +- **Approach A** — a dedicated structured `schema-linking` gate widget (not a flat + `reviewer_decide`). +- **Option 1 (soft)** — the curated column set is persisted and the model is + *instructed* to honor it for output columns; **no hard SQL validator**. Hard + enforcement is an explicit follow-up (needs live model + VPN). +- **Staged commit** — column checkboxes are staged in widget state and sent once, on + the gate's Confirm. +- **Interaction** — the F4 gate stays **inline** (as today's `reviewer_decide`); each + table row opens **one** columns modal (no nested modals). + +## Scope + +| In scope | Out of scope (explicit) | +|---|---| +| Only the F4 **tabelle/colonne/esclusioni** gate (replay gate 7) | The F4 **join proposte** gate (gate 8) — unchanged | +| Table rows with description + rationale under the name | Hard SQL enforcement (reject SQL using non-approved columns) — **Option 2, follow-up** | +| Per-table columns modal (name + description, checkbox), suggested pre-selected + bold | Any change to the Pi RPC wire protocol beyond the new descriptor/response payloads | +| Persist curated columns into `schema_linking.json` + `review_decisions.jsonl` | Reviewer editing of column *descriptions* (read-only from catalog) | +| Excluded tables: read-only column modal | Column curation for tables the reviewer excludes | + +--- + +## 1. Data contract + +### 1a. Gate descriptor (harness → frontend) + +A new widget kind replaces the flat `reviewer_decide` for this gate only: + +```jsonc +{ + "id": "", + "widget": "schema-linking", + "title": "F4 — Schema linking: tabelle/colonne/esclusioni", + "tables": [ + { + "id": "t-ablazione", + "name": "fact_studio_elettrofisiologico_endocavitario_ablazione", + "kind": "promote", // "promote" | "exclude" + "recommended": true, // pre-enacted (checkbox checked) when true + "description": "", + "rationale": "", + "columns": [ + { "name": "cod_paz", "description": "", "type": "bigint", "pk": true, "suggested": true }, + { "name": "num", "description": "", "type": "bigint", "pk": false, "suggested": false } + ] + } + ], + "reserved": ["back", "exit", "other"] +} +``` + +- `columns` is populated **deterministically** from the catalog + `artifacts/mschema/physical.yaml` (per table: `comment`; per column: + `type`, `pk`, `comment`). The **model** supplies only: promote/exclude `kind`, + `rationale`, and the `suggested` set. `tht-gate.js` merges model-proposal + + catalog into the descriptor, so the model transcribes little (fewer tokens, no + invented column names). +- For `kind: "exclude"`, `columns[].suggested` is ignored by the UI (read-only view). + +### 1b. Gate response (frontend → harness) + +Structured response, sent once on gate Confirm: + +```jsonc +{ + "id": "", + "kind": "schema-linking", + "tables": [ + { "id": "t-ablazione", "enacted": true, "columns": ["cod_paz", "num", "data_time_key", "ablazione_transcatetere"] }, + { "id": "t-impianto", "enacted": true, "columns": ["cod_paz", "num", "data_time_key"] }, + { "id": "x-sostituzione", "enacted": true } // exclude enacted; no columns key + ] +} +``` + +- `enacted` mirrors the row checkbox (the reviewer may decline a recommended + promote/exclude). +- `columns` is present only for enacted `promote` tables; it is the reviewer's final + selection (suggested ∖ deselected ∪ newly selected). + +--- + +## 2. Interaction model + +The gate renders **inline** (same surface as today's `reviewer_decide`), as a card +listing every table. Column detail lives **only** in the modal (requirement 3: no +column rows in the list). + +### Promote row +- Line 1: table `name` (prominent). +- Below: `description` (muted) then `rationale` (the "why include"). +- A `PROMUOVI` badge. +- A row checkbox = enact this promote decision (checked by default when + `recommended`). +- A **"Colonne k/n"** button (k = currently selected, n = total) → opens the columns + modal. + +### Columns modal (promote) +- List of the table's fields. **First cell = checkbox.** +- `suggested` fields are **pre-selected**. +- **Selected field ⇒ its name + description render bold**; deselected ⇒ normal. This + applies to both pre-selected fields and fields the reviewer selects later; on + deselection they return to normal (requirement 1). +- Selection is **staged** in the widget's state (not committed until the gate's + Confirm). Closing the modal keeps the staged selection; the row's "Colonne k/n" + count updates. + +### Exclude row +- Line 1: table `name`. +- Below: `description` + exclusion `rationale` (requirement 2). +- An `ESCLUDI` badge; row checkbox = enact the exclude decision. +- A **"Colonne"** button → opens the columns modal in **read-only** mode: checkboxes + disabled, all fields rendered normal (no preselection, no bold), purely + informational (requirement 2). + +### Confirm +- The gate's **Confirm** builds the structured response (§1b) from staged state and + calls `onRespond`. Reserved controls (`back`/`exit`/`other`) behave as today. + +--- + +## 3. Frontend components + +| File | Change | +|---|---| +| `frontend/src/widgets/SchemaLinkingGateWidget.tsx` | **New.** Inline card; staged state `Map }>` initialized from `recommended` (enacted) and `suggested` (selected). Builds the §1b response on Confirm. | +| `frontend/src/widgets/SchemaColumnsDialog.tsx` | **New.** `components/ui/dialog` modal listing `columns`; prop `readOnly` for excluded tables; bold-when-selected rule; writes staged selection back to the parent on toggle. | +| `frontend/src/widgets/registry.ts` | Register `schema-linking` → `SchemaLinkingGateWidget`. | +| `frontend/src/widgets/types.ts` | Descriptor + option types for the new payload. | +| `frontend/src/api/types.ts` | Extend `UiResponse` with the structured `schema-linking` shape (`tables[]`). | +| `frontend/src/shell/WidgetHost.tsx` | Pass the structured response through (it already forwards `UiResponse`; verify `setLastUserEntry` summary handles the new kind). | + +Reuse the v2 visual vocabulary: cards `border-border/70 + shadow-sm`, `.thot-label` +(mono) for the section/field micro-labels, `rounded-*` radii, first-action-primary +button convention. No new color tokens. + +--- + +## 4. Harness changes + +| File | Change | +|---|---| +| `harness/.pi/extensions/tht-gate.js` | Emit the `schema-linking` descriptor (merge model proposal + catalog columns). Add a response handler for `widget:"schema-linking"`: per enacted table → `tht decision add` (`table_promoted`/`table_excluded`, as today); per selected column of a promoted table → new `column_promoted` decision (`subject = "table.col"`); suggested-but-deselected → `column_excluded`. Then invoke the deterministic reconcile (below). | +| catalog reader (`tht`) | A deterministic reader returning a table's columns + comments from `physical.yaml` (`[{name, description, type, pk}]`). **Verify whether an existing `tht schema …`/search command already exposes this; add a thin command if not.** | +| `schema_linking.json` reconcile (`tht`) | A deterministic `tht` command that sets a promoted table's column candidates to exactly the reviewer's selection (selected → `decision: "promoted"`; deselected suggested → dropped or `"excluded"`; newly selected → added). Makes the curation **authoritative** rather than relying on model transcription. | +| `harness/tht/decisions.py` | Add `column_promoted` / `column_excluded` to `DecisionType`. | +| `harness/workflow.yaml` | Add `column_promoted` / `column_excluded` to F4 `emits`. | +| `harness/.pi/skills/tht-sessione/SKILL.md` | F4: describe the column-curation step + the new gate. F6/F7: instruct the model to treat `schema_linking.json` promoted columns as the reviewer-approved **output** set, while remaining free to use other columns as **join keys / filter predicates** (Option 1, soft). | + +Sequencing of "model proposes → gate reconciles `schema_linking.json`" is a detail to +pin down in the plan (the reconcile must run after the tables exist in the artifact). + +--- + +## 5. Downstream (Option 1, soft) + +- `harness/tht/cli/sql_cmd.py`: `promoted_tables_for` stays. Optionally add a + `promoted_columns_for` helper (read promoted column candidates) for **future** use; + it is **not** wired into validation now — no SQL is rejected on column grounds. +- The only behavioral coupling now is via `SKILL.md` guidance + the curated + `schema_linking.json` the model reads when composing CTEs/SQL. + +--- + +## 6. Testing & replay (no VPN) + +- **Augmented replay fixture.** A script reads `physical.yaml` for the 6 tables of the + recorded psd session and rewrites replay gate 7 as a `schema-linking` descriptor + with real `columns`. This makes the new widget **visually verifiable in replay** + (server on :5333) before any harness wiring exists. Touches + `tools/replay/extract.mjs` + regenerates `tools/replay/replay.json`. +- **Frontend unit (vitest + MSW):** widget staging (suggested pre-selected, count), + bold-toggle on select/deselect, read-only for excluded, structured response shape on + Confirm, reserved controls. +- **Harness unit:** `tht-gate.js` response handler (structured response → the right + `tht decision add` calls + reconcile invocation); pytest for the catalog reader and + the `schema_linking.json` reconcile command. +- **End-to-end with the live model is deferred** (no VPN); the soft SQL guidance can + only be spot-checked in replay. + +--- + +## 7. Phased delivery (one spec, plan sequenced in phases) + +1. **Frontend + contract + replay fixture** — new widget, columns modal, structured + `UiResponse`, registry; augmented replay fixture. Fully verifiable offline. +2. **Harness** — gate emission (catalog merge + `suggested`), structured-response + handler, ledger types, `schema_linking.json` reconcile, `tht` commands, `SKILL.md` + guidance. + +Follow-up (separate spec, when VPN is available): **Option 2** hard SQL enforcement. + +--- + +## 8. Risks & open decisions + +- **Interaction shape** (§2): inline gate + single columns modal is chosen over + gate-as-modal + nested modal to avoid nested-dialog complexity. Revisit only if the + reviewer explicitly wants the gate itself modal. +- **Reconcile ordering** (§4): the deterministic `schema_linking.json` reconcile must + run after the table candidates exist; exact sequencing pinned in the plan. +- **Deselecting a key column** (e.g. `cod_paz`): under Option 1 this does not block — + `SKILL.md` notes join keys remain usable even when not selected as output columns. + A soft UI hint ("key column") is possible but not required for v1. +- **Catalog reader existence** (§4): confirm whether `tht` already exposes + `physical.yaml` columns; add a thin command only if missing.