ThothII/docs/superpowers/specs/2026-07-06-f4-schema-linking-column-curation-design.md

F4 schema-linking — per-table column curation — design

Date: 2026-07-06 Status: approved (brainstorming), pending plan

The F4 "Schema linking: tabelle/colonne/esclusioni" reviewer gate currently lets the reviewer only promote/exclude tables; columns are chosen by the model and never surfaced for review. This feature lets the reviewer, per promoted table, curate the columns to use — over the table's full catalog column list — with the model's suggested columns pre-selected. The curated selection is persisted (into schema_linking.json + the decision ledger) and guides SQL generation downstream. Excluded tables get a read-only column view.

Decisions locked in brainstorming:

• Approach A — a dedicated structured schema-linking gate widget (not a flat reviewer_decide).
• Option 1 (soft) — the curated column set is persisted and the model is instructed to honor it for output columns; no hard SQL validator. Hard enforcement is an explicit follow-up (needs live model + VPN).
• Staged commit — column checkboxes are staged in widget state and sent once, on the gate's Confirm.
• Interaction — the F4 gate stays inline (as today's reviewer_decide); each table row opens one columns modal (no nested modals).

Scope

In scope	Out of scope (explicit)
Only the F4 tabelle/colonne/esclusioni gate (replay gate 7)	The F4 join proposte gate (gate 8) — unchanged
Table rows with description + rationale under the name	Hard SQL enforcement (reject SQL using non-approved columns) — Option 2, follow-up
Per-table columns modal (name + description, checkbox), suggested pre-selected + bold	Any change to the Pi RPC wire protocol beyond the new descriptor/response payloads
Persist curated columns into schema_linking.json + review_decisions.jsonl	Reviewer editing of column descriptions (read-only from catalog)
Excluded tables: read-only column modal	Column curation for tables the reviewer excludes

---

1. Data contract

1a. Gate descriptor (harness → frontend)

A new widget kind replaces the flat reviewer_decide for this gate only:

{
  "id": "<pi ui id>",
  "widget": "schema-linking",
  "title": "F4 — Schema linking: tabelle/colonne/esclusioni",
  "tables": [
    {
      "id": "t-ablazione",
      "name": "fact_studio_elettrofisiologico_endocavitario_ablazione",
      "kind": "promote",            // "promote" | "exclude"
      "recommended": true,          // pre-enacted (checkbox checked) when true
      "description": "<table comment from physical.yaml>",
      "rationale": "<model's promote/exclude motivation>",
      "columns": [
        { "name": "cod_paz", "description": "<column comment>", "type": "bigint", "pk": true, "suggested": true },
        { "name": "num",     "description": "<column comment>", "type": "bigint", "pk": false, "suggested": false }
      ]
    }
  ],
  "reserved": ["back", "exit", "other"]
}

• columns is populated deterministically from the catalog artifacts/mschema/physical.yaml (per table: comment; per column: type, pk, comment). The model supplies only: promote/exclude kind, rationale, and the suggested set. tht-gate.js merges model-proposal + catalog into the descriptor, so the model transcribes little (fewer tokens, no invented column names).
• For kind: "exclude", columns[].suggested is ignored by the UI (read-only view).

1b. Gate response (frontend → harness)

Structured response, sent once on gate Confirm:

{
  "id": "<same ui id>",
  "kind": "schema-linking",
  "tables": [
    { "id": "t-ablazione", "enacted": true,  "columns": ["cod_paz", "num", "data_time_key", "ablazione_transcatetere"] },
    { "id": "t-impianto",  "enacted": true,  "columns": ["cod_paz", "num", "data_time_key"] },
    { "id": "x-sostituzione", "enacted": true }          // exclude enacted; no columns key
  ]
}

• enacted mirrors the row checkbox (the reviewer may decline a recommended promote/exclude).
• columns is present only for enacted promote tables; it is the reviewer's final selection (suggested ∖ deselected ∪ newly selected).

---

2. Interaction model

The gate renders inline (same surface as today's reviewer_decide), as a card listing every table. Column detail lives only in the modal (requirement 3: no column rows in the list).

Promote row

• Line 1: table name (prominent).
• Below: description (muted) then rationale (the "why include").
• A PROMUOVI badge.
• A row checkbox = enact this promote decision (checked by default when recommended).
• A "Colonne k/n" button (k = currently selected, n = total) → opens the columns modal.

Columns modal (promote)

• List of the table's fields. First cell = checkbox.
• suggested fields are pre-selected.
• Selected field ⇒ its name + description render bold; deselected ⇒ normal. This applies to both pre-selected fields and fields the reviewer selects later; on deselection they return to normal (requirement 1).
• Selection is staged in the widget's state (not committed until the gate's Confirm). Closing the modal keeps the staged selection; the row's "Colonne k/n" count updates.

Exclude row

• Line 1: table name.
• Below: description + exclusion rationale (requirement 2).
• An ESCLUDI badge; row checkbox = enact the exclude decision.
• A "Colonne" button → opens the columns modal in read-only mode: checkboxes disabled, all fields rendered normal (no preselection, no bold), purely informational (requirement 2).

Confirm

• The gate's Confirm builds the structured response (§1b) from staged state and calls onRespond. Reserved controls (back/exit/other) behave as today.

---

3. Frontend components

File	Change
frontend/src/widgets/SchemaLinkingGateWidget.tsx	New. Inline card; staged state Map<tableId, { enacted: boolean; selected: Set<string> }> initialized from recommended (enacted) and suggested (selected). Builds the §1b response on Confirm.
frontend/src/widgets/SchemaColumnsDialog.tsx	New. components/ui/dialog modal listing columns; prop readOnly for excluded tables; bold-when-selected rule; writes staged selection back to the parent on toggle.
frontend/src/widgets/registry.ts	Register schema-linking → SchemaLinkingGateWidget.
frontend/src/widgets/types.ts	Descriptor + option types for the new payload.
frontend/src/api/types.ts	Extend UiResponse with the structured schema-linking shape (tables[]).
frontend/src/shell/WidgetHost.tsx	Pass the structured response through (it already forwards UiResponse; verify setLastUserEntry summary handles the new kind).

Reuse the v2 visual vocabulary: cards border-border/70 + shadow-sm, .thot-label (mono) for the section/field micro-labels, rounded-* radii, first-action-primary button convention. No new color tokens.

---

4. Harness changes

File	Change
harness/.pi/extensions/tht-gate.js	Emit the schema-linking descriptor (merge model proposal + catalog columns). Add a response handler for widget:"schema-linking": per enacted table → tht decision add (table_promoted/table_excluded, as today); per selected column of a promoted table → new column_promoted decision (subject = "table.col"); suggested-but-deselected → column_excluded. Then invoke the deterministic reconcile (below).
catalog reader (tht)	A deterministic reader returning a table's columns + comments from physical.yaml ([{name, description, type, pk}]). Verify whether an existing tht schema …/search command already exposes this; add a thin command if not.
schema_linking.json reconcile (tht)	A deterministic tht command that sets a promoted table's column candidates to exactly the reviewer's selection (selected → decision: "promoted"; deselected suggested → dropped or "excluded"; newly selected → added). Makes the curation authoritative rather than relying on model transcription.
harness/tht/decisions.py	Add column_promoted / column_excluded to DecisionType.
harness/workflow.yaml	Add column_promoted / column_excluded to F4 emits.
harness/.pi/skills/tht-sessione/SKILL.md	F4: describe the column-curation step + the new gate. F6/F7: instruct the model to treat schema_linking.json promoted columns as the reviewer-approved output set, while remaining free to use other columns as join keys / filter predicates (Option 1, soft).

Sequencing of "model proposes → gate reconciles schema_linking.json" is a detail to pin down in the plan (the reconcile must run after the tables exist in the artifact).

---

5. Downstream (Option 1, soft)

• harness/tht/cli/sql_cmd.py: promoted_tables_for stays. Optionally add a promoted_columns_for helper (read promoted column candidates) for future use; it is not wired into validation now — no SQL is rejected on column grounds.
• The only behavioral coupling now is via SKILL.md guidance + the curated schema_linking.json the model reads when composing CTEs/SQL.

---

6. Testing & replay (no VPN)

• Augmented replay fixture. A script reads physical.yaml for the 6 tables of the recorded psd session and rewrites replay gate 7 as a schema-linking descriptor with real columns. This makes the new widget visually verifiable in replay (server on :5333) before any harness wiring exists. Touches tools/replay/extract.mjs + regenerates tools/replay/replay.json.
• Frontend unit (vitest + MSW): widget staging (suggested pre-selected, count), bold-toggle on select/deselect, read-only for excluded, structured response shape on Confirm, reserved controls.
• Harness unit: tht-gate.js response handler (structured response → the right tht decision add calls + reconcile invocation); pytest for the catalog reader and the schema_linking.json reconcile command.
• End-to-end with the live model is deferred (no VPN); the soft SQL guidance can only be spot-checked in replay.

---

7. Phased delivery (one spec, plan sequenced in phases)

1. Frontend + contract + replay fixture — new widget, columns modal, structured UiResponse, registry; augmented replay fixture. Fully verifiable offline.
2. Harness — gate emission (catalog merge + suggested), structured-response handler, ledger types, schema_linking.json reconcile, tht commands, SKILL.md guidance.

Follow-up (separate spec, when VPN is available): Option 2 hard SQL enforcement.

---

8. Risks & open decisions

• Interaction shape (§2): inline gate + single columns modal is chosen over gate-as-modal + nested modal to avoid nested-dialog complexity. Revisit only if the reviewer explicitly wants the gate itself modal.
• Reconcile ordering (§4): the deterministic schema_linking.json reconcile must run after the table candidates exist; exact sequencing pinned in the plan.
• Deselecting a key column (e.g. cod_paz): under Option 1 this does not block — SKILL.md notes join keys remain usable even when not selected as output columns. A soft UI hint ("key column") is possible but not required for v1.
• Catalog reader existence (§4): confirm whether tht already exposes physical.yaml columns; add a thin command only if missing.