ThothII/docs/superpowers/specs/2026-07-06-f4-schema-linking-column-curation-design.md

# F4 schema-linking — per-table column curation — design

Date: 2026-07-06
Status: approved (brainstorming), pending plan

The F4 "Schema linking: tabelle/colonne/esclusioni" reviewer gate currently lets the
reviewer only promote/exclude **tables**; columns are chosen by the model and never
surfaced for review. This feature lets the reviewer, per promoted table, curate the
**columns** to use — over the table's full catalog column list — with the model's
suggested columns pre-selected. The curated selection is **persisted** (into
`schema_linking.json` + the decision ledger) and guides SQL generation downstream.
Excluded tables get a read-only column view.

Decisions locked in brainstorming:
- **Approach A** — a dedicated structured `schema-linking` gate widget (not a flat
  `reviewer_decide`).
- **Option 1 (soft)** — the curated column set is persisted and the model is
  *instructed* to honor it for output columns; **no hard SQL validator**. Hard
  enforcement is an explicit follow-up (needs live model + VPN).
- **Staged commit** — column checkboxes are staged in widget state and sent once, on
  the gate's Confirm.
- **Interaction** — the F4 gate stays **inline** (as today's `reviewer_decide`); each
  table row opens **one** columns modal (no nested modals).

## Scope

| In scope | Out of scope (explicit) |
|---|---|
| Only the F4 **tabelle/colonne/esclusioni** gate (replay gate 7) | The F4 **join proposte** gate (gate 8) — unchanged |
| Table rows with description + rationale under the name | Hard SQL enforcement (reject SQL using non-approved columns) — **Option 2, follow-up** |
| Per-table columns modal (name + description, checkbox), suggested pre-selected + bold | Any change to the Pi RPC wire protocol beyond the new descriptor/response payloads |
| Persist curated columns into `schema_linking.json` + `review_decisions.jsonl` | Reviewer editing of column *descriptions* (read-only from catalog) |
| Excluded tables: read-only column modal | Column curation for tables the reviewer excludes |

---

## 1. Data contract

### 1a. Gate descriptor (harness → frontend)

A new widget kind replaces the flat `reviewer_decide` for this gate only:

```jsonc
{
  "id": "<pi ui id>",
  "widget": "schema-linking",
  "title": "F4 — Schema linking: tabelle/colonne/esclusioni",
  "tables": [
    {
      "id": "t-ablazione",
      "name": "fact_studio_elettrofisiologico_endocavitario_ablazione",
      "kind": "promote",            // "promote" | "exclude"
      "recommended": true,          // pre-enacted (checkbox checked) when true
      "description": "<table comment from physical.yaml>",
      "rationale": "<model's promote/exclude motivation>",
      "columns": [
        { "name": "cod_paz", "description": "<column comment>", "type": "bigint", "pk": true, "suggested": true },
        { "name": "num",     "description": "<column comment>", "type": "bigint", "pk": false, "suggested": false }
      ]
    }
  ],
  "reserved": ["back", "exit", "other"]
}
```

- `columns` is populated **deterministically** from the catalog
  `artifacts/mschema/physical.yaml` (per table: `comment`; per column:
  `type`, `pk`, `comment`). The **model** supplies only: promote/exclude `kind`,
  `rationale`, and the `suggested` set. `tht-gate.js` merges model-proposal +
  catalog into the descriptor, so the model transcribes little (fewer tokens, no
  invented column names).
- For `kind: "exclude"`, `columns[].suggested` is ignored by the UI (read-only view).

### 1b. Gate response (frontend → harness)

Structured response, sent once on gate Confirm:

```jsonc
{
  "id": "<same ui id>",
  "kind": "schema-linking",
  "tables": [
    { "id": "t-ablazione", "enacted": true,  "columns": ["cod_paz", "num", "data_time_key", "ablazione_transcatetere"] },
    { "id": "t-impianto",  "enacted": true,  "columns": ["cod_paz", "num", "data_time_key"] },
    { "id": "x-sostituzione", "enacted": true }          // exclude enacted; no columns key
  ]
}
```

- `enacted` mirrors the row checkbox (the reviewer may decline a recommended
  promote/exclude).
- `columns` is present only for enacted `promote` tables; it is the reviewer's final
  selection (suggested ∖ deselected ∪ newly selected).

---

## 2. Interaction model

The gate renders **inline** (same surface as today's `reviewer_decide`), as a card
listing every table. Column detail lives **only** in the modal (requirement 3: no
column rows in the list).

### Promote row
- Line 1: table `name` (prominent).
- Below: `description` (muted) then `rationale` (the "why include").
- A `PROMUOVI` badge.
- A row checkbox = enact this promote decision (checked by default when
  `recommended`).
- A **"Colonne k/n"** button (k = currently selected, n = total) → opens the columns
  modal.

### Columns modal (promote)
- List of the table's fields. **First cell = checkbox.**
- `suggested` fields are **pre-selected**.
- **Selected field ⇒ its name + description render bold**; deselected ⇒ normal. This
  applies to both pre-selected fields and fields the reviewer selects later; on
  deselection they return to normal (requirement 1).
- Selection is **staged** in the widget's state (not committed until the gate's
  Confirm). Closing the modal keeps the staged selection; the row's "Colonne k/n"
  count updates.

### Exclude row
- Line 1: table `name`.
- Below: `description` + exclusion `rationale` (requirement 2).
- An `ESCLUDI` badge; row checkbox = enact the exclude decision.
- A **"Colonne"** button → opens the columns modal in **read-only** mode: checkboxes
  disabled, all fields rendered normal (no preselection, no bold), purely
  informational (requirement 2).

### Confirm
- The gate's **Confirm** builds the structured response (§1b) from staged state and
  calls `onRespond`. Reserved controls (`back`/`exit`/`other`) behave as today.

---

## 3. Frontend components

| File | Change |
|---|---|
| `frontend/src/widgets/SchemaLinkingGateWidget.tsx` | **New.** Inline card; staged state `Map<tableId, { enacted: boolean; selected: Set<string> }>` initialized from `recommended` (enacted) and `suggested` (selected). Builds the §1b response on Confirm. |
| `frontend/src/widgets/SchemaColumnsDialog.tsx` | **New.** `components/ui/dialog` modal listing `columns`; prop `readOnly` for excluded tables; bold-when-selected rule; writes staged selection back to the parent on toggle. |
| `frontend/src/widgets/registry.ts` | Register `schema-linking` → `SchemaLinkingGateWidget`. |
| `frontend/src/widgets/types.ts` | Descriptor + option types for the new payload. |
| `frontend/src/api/types.ts` | Extend `UiResponse` with the structured `schema-linking` shape (`tables[]`). |
| `frontend/src/shell/WidgetHost.tsx` | Pass the structured response through (it already forwards `UiResponse`; verify `setLastUserEntry` summary handles the new kind). |

Reuse the v2 visual vocabulary: cards `border-border/70 + shadow-sm`, `.thot-label`
(mono) for the section/field micro-labels, `rounded-*` radii, first-action-primary
button convention. No new color tokens.

---

## 4. Harness changes

| File | Change |
|---|---|
| `harness/.pi/extensions/tht-gate.js` | Emit the `schema-linking` descriptor (merge model proposal + catalog columns). Add a response handler for `widget:"schema-linking"`: per enacted table → `tht decision add` (`table_promoted`/`table_excluded`, as today); per selected column of a promoted table → new `column_promoted` decision (`subject = "table.col"`); suggested-but-deselected → `column_excluded`. Then invoke the deterministic reconcile (below). |
| catalog reader (`tht`) | A deterministic reader returning a table's columns + comments from `physical.yaml` (`[{name, description, type, pk}]`). **Verify whether an existing `tht schema …`/search command already exposes this; add a thin command if not.** |
| `schema_linking.json` reconcile (`tht`) | A deterministic `tht` command that sets a promoted table's column candidates to exactly the reviewer's selection (selected → `decision: "promoted"`; deselected suggested → dropped or `"excluded"`; newly selected → added). Makes the curation **authoritative** rather than relying on model transcription. |
| `harness/tht/decisions.py` | Add `column_promoted` / `column_excluded` to `DecisionType`. |
| `harness/workflow.yaml` | Add `column_promoted` / `column_excluded` to F4 `emits`. |
| `harness/.pi/skills/tht-sessione/SKILL.md` | F4: describe the column-curation step + the new gate. F6/F7: instruct the model to treat `schema_linking.json` promoted columns as the reviewer-approved **output** set, while remaining free to use other columns as **join keys / filter predicates** (Option 1, soft). |

Sequencing of "model proposes → gate reconciles `schema_linking.json`" is a detail to
pin down in the plan (the reconcile must run after the tables exist in the artifact).

---

## 5. Downstream (Option 1, soft)

- `harness/tht/cli/sql_cmd.py`: `promoted_tables_for` stays. Optionally add a
  `promoted_columns_for` helper (read promoted column candidates) for **future** use;
  it is **not** wired into validation now — no SQL is rejected on column grounds.
- The only behavioral coupling now is via `SKILL.md` guidance + the curated
  `schema_linking.json` the model reads when composing CTEs/SQL.

---

## 6. Testing & replay (no VPN)

- **Augmented replay fixture.** A script reads `physical.yaml` for the 6 tables of the
  recorded psd session and rewrites replay gate 7 as a `schema-linking` descriptor
  with real `columns`. This makes the new widget **visually verifiable in replay**
  (server on :5333) before any harness wiring exists. Touches
  `tools/replay/extract.mjs` + regenerates `tools/replay/replay.json`.
- **Frontend unit (vitest + MSW):** widget staging (suggested pre-selected, count),
  bold-toggle on select/deselect, read-only for excluded, structured response shape on
  Confirm, reserved controls.
- **Harness unit:** `tht-gate.js` response handler (structured response → the right
  `tht decision add` calls + reconcile invocation); pytest for the catalog reader and
  the `schema_linking.json` reconcile command.
- **End-to-end with the live model is deferred** (no VPN); the soft SQL guidance can
  only be spot-checked in replay.

---

## 7. Phased delivery (one spec, plan sequenced in phases)

1. **Frontend + contract + replay fixture** — new widget, columns modal, structured
   `UiResponse`, registry; augmented replay fixture. Fully verifiable offline.
2. **Harness** — gate emission (catalog merge + `suggested`), structured-response
   handler, ledger types, `schema_linking.json` reconcile, `tht` commands, `SKILL.md`
   guidance.

Follow-up (separate spec, when VPN is available): **Option 2** hard SQL enforcement.

---

## 8. Risks & open decisions

- **Interaction shape** (§2): inline gate + single columns modal is chosen over
  gate-as-modal + nested modal to avoid nested-dialog complexity. Revisit only if the
  reviewer explicitly wants the gate itself modal.
- **Reconcile ordering** (§4): the deterministic `schema_linking.json` reconcile must
  run after the table candidates exist; exact sequencing pinned in the plan.
- **Deselecting a key column** (e.g. `cod_paz`): under Option 1 this does not block —
  `SKILL.md` notes join keys remain usable even when not selected as output columns.
  A soft UI hint ("key column") is possible but not required for v1.
- **Catalog reader existence** (§4): confirm whether `tht` already exposes
  `physical.yaml` columns; add a thin command only if missing.