docs: design for F4 schema-linking per-table column curation
Reviewer curates, per promoted table, which columns to use over the full catalog list (suggested pre-selected + bold); selection persists into schema_linking.json + the decision ledger and softly guides SQL generation (Option 1). Dedicated structured schema-linking gate widget (Approach A), staged commit, inline gate + single columns modal. Hard SQL enforcement is an explicit follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
committed by
Marco Pancotti
Marco Pancotti
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
e11ffecfc4
commit
94d171ddf1
@@ -0,0 +1,220 @@
|
||||
# F4 schema-linking — per-table column curation — design
|
||||
|
||||
Date: 2026-07-06
|
||||
Status: approved (brainstorming), pending plan
|
||||
|
||||
The F4 "Schema linking: tabelle/colonne/esclusioni" reviewer gate currently lets the
|
||||
reviewer only promote/exclude **tables**; columns are chosen by the model and never
|
||||
surfaced for review. This feature lets the reviewer, per promoted table, curate the
|
||||
**columns** to use — over the table's full catalog column list — with the model's
|
||||
suggested columns pre-selected. The curated selection is **persisted** (into
|
||||
`schema_linking.json` + the decision ledger) and guides SQL generation downstream.
|
||||
Excluded tables get a read-only column view.
|
||||
|
||||
Decisions locked in brainstorming:
|
||||
- **Approach A** — a dedicated structured `schema-linking` gate widget (not a flat
|
||||
`reviewer_decide`).
|
||||
- **Option 1 (soft)** — the curated column set is persisted and the model is
|
||||
*instructed* to honor it for output columns; **no hard SQL validator**. Hard
|
||||
enforcement is an explicit follow-up (needs live model + VPN).
|
||||
- **Staged commit** — column checkboxes are staged in widget state and sent once, on
|
||||
the gate's Confirm.
|
||||
- **Interaction** — the F4 gate stays **inline** (as today's `reviewer_decide`); each
|
||||
table row opens **one** columns modal (no nested modals).
|
||||
|
||||
## Scope
|
||||
|
||||
| In scope | Out of scope (explicit) |
|
||||
|---|---|
|
||||
| Only the F4 **tabelle/colonne/esclusioni** gate (replay gate 7) | The F4 **join proposte** gate (gate 8) — unchanged |
|
||||
| Table rows with description + rationale under the name | Hard SQL enforcement (reject SQL using non-approved columns) — **Option 2, follow-up** |
|
||||
| Per-table columns modal (name + description, checkbox), suggested pre-selected + bold | Any change to the Pi RPC wire protocol beyond the new descriptor/response payloads |
|
||||
| Persist curated columns into `schema_linking.json` + `review_decisions.jsonl` | Reviewer editing of column *descriptions* (read-only from catalog) |
|
||||
| Excluded tables: read-only column modal | Column curation for tables the reviewer excludes |
|
||||
|
||||
---
|
||||
|
||||
## 1. Data contract
|
||||
|
||||
### 1a. Gate descriptor (harness → frontend)
|
||||
|
||||
A new widget kind replaces the flat `reviewer_decide` for this gate only:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "<pi ui id>",
|
||||
"widget": "schema-linking",
|
||||
"title": "F4 — Schema linking: tabelle/colonne/esclusioni",
|
||||
"tables": [
|
||||
{
|
||||
"id": "t-ablazione",
|
||||
"name": "fact_studio_elettrofisiologico_endocavitario_ablazione",
|
||||
"kind": "promote", // "promote" | "exclude"
|
||||
"recommended": true, // pre-enacted (checkbox checked) when true
|
||||
"description": "<table comment from physical.yaml>",
|
||||
"rationale": "<model's promote/exclude motivation>",
|
||||
"columns": [
|
||||
{ "name": "cod_paz", "description": "<column comment>", "type": "bigint", "pk": true, "suggested": true },
|
||||
{ "name": "num", "description": "<column comment>", "type": "bigint", "pk": false, "suggested": false }
|
||||
]
|
||||
}
|
||||
],
|
||||
"reserved": ["back", "exit", "other"]
|
||||
}
|
||||
```
|
||||
|
||||
- `columns` is populated **deterministically** from the catalog
|
||||
`artifacts/mschema/physical.yaml` (per table: `comment`; per column:
|
||||
`type`, `pk`, `comment`). The **model** supplies only: promote/exclude `kind`,
|
||||
`rationale`, and the `suggested` set. `tht-gate.js` merges model-proposal +
|
||||
catalog into the descriptor, so the model transcribes little (fewer tokens, no
|
||||
invented column names).
|
||||
- For `kind: "exclude"`, `columns[].suggested` is ignored by the UI (read-only view).
|
||||
|
||||
### 1b. Gate response (frontend → harness)
|
||||
|
||||
Structured response, sent once on gate Confirm:
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"id": "<same ui id>",
|
||||
"kind": "schema-linking",
|
||||
"tables": [
|
||||
{ "id": "t-ablazione", "enacted": true, "columns": ["cod_paz", "num", "data_time_key", "ablazione_transcatetere"] },
|
||||
{ "id": "t-impianto", "enacted": true, "columns": ["cod_paz", "num", "data_time_key"] },
|
||||
{ "id": "x-sostituzione", "enacted": true } // exclude enacted; no columns key
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- `enacted` mirrors the row checkbox (the reviewer may decline a recommended
|
||||
promote/exclude).
|
||||
- `columns` is present only for enacted `promote` tables; it is the reviewer's final
|
||||
selection (suggested ∖ deselected ∪ newly selected).
|
||||
|
||||
---
|
||||
|
||||
## 2. Interaction model
|
||||
|
||||
The gate renders **inline** (same surface as today's `reviewer_decide`), as a card
|
||||
listing every table. Column detail lives **only** in the modal (requirement 3: no
|
||||
column rows in the list).
|
||||
|
||||
### Promote row
|
||||
- Line 1: table `name` (prominent).
|
||||
- Below: `description` (muted) then `rationale` (the "why include").
|
||||
- A `PROMUOVI` badge.
|
||||
- A row checkbox = enact this promote decision (checked by default when
|
||||
`recommended`).
|
||||
- A **"Colonne k/n"** button (k = currently selected, n = total) → opens the columns
|
||||
modal.
|
||||
|
||||
### Columns modal (promote)
|
||||
- List of the table's fields. **First cell = checkbox.**
|
||||
- `suggested` fields are **pre-selected**.
|
||||
- **Selected field ⇒ its name + description render bold**; deselected ⇒ normal. This
|
||||
applies to both pre-selected fields and fields the reviewer selects later; on
|
||||
deselection they return to normal (requirement 1).
|
||||
- Selection is **staged** in the widget's state (not committed until the gate's
|
||||
Confirm). Closing the modal keeps the staged selection; the row's "Colonne k/n"
|
||||
count updates.
|
||||
|
||||
### Exclude row
|
||||
- Line 1: table `name`.
|
||||
- Below: `description` + exclusion `rationale` (requirement 2).
|
||||
- An `ESCLUDI` badge; row checkbox = enact the exclude decision.
|
||||
- A **"Colonne"** button → opens the columns modal in **read-only** mode: checkboxes
|
||||
disabled, all fields rendered normal (no preselection, no bold), purely
|
||||
informational (requirement 2).
|
||||
|
||||
### Confirm
|
||||
- The gate's **Confirm** builds the structured response (§1b) from staged state and
|
||||
calls `onRespond`. Reserved controls (`back`/`exit`/`other`) behave as today.
|
||||
|
||||
---
|
||||
|
||||
## 3. Frontend components
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `frontend/src/widgets/SchemaLinkingGateWidget.tsx` | **New.** Inline card; staged state `Map<tableId, { enacted: boolean; selected: Set<string> }>` initialized from `recommended` (enacted) and `suggested` (selected). Builds the §1b response on Confirm. |
|
||||
| `frontend/src/widgets/SchemaColumnsDialog.tsx` | **New.** `components/ui/dialog` modal listing `columns`; prop `readOnly` for excluded tables; bold-when-selected rule; writes staged selection back to the parent on toggle. |
|
||||
| `frontend/src/widgets/registry.ts` | Register `schema-linking` → `SchemaLinkingGateWidget`. |
|
||||
| `frontend/src/widgets/types.ts` | Descriptor + option types for the new payload. |
|
||||
| `frontend/src/api/types.ts` | Extend `UiResponse` with the structured `schema-linking` shape (`tables[]`). |
|
||||
| `frontend/src/shell/WidgetHost.tsx` | Pass the structured response through (it already forwards `UiResponse`; verify `setLastUserEntry` summary handles the new kind). |
|
||||
|
||||
Reuse the v2 visual vocabulary: cards `border-border/70 + shadow-sm`, `.thot-label`
|
||||
(mono) for the section/field micro-labels, `rounded-*` radii, first-action-primary
|
||||
button convention. No new color tokens.
|
||||
|
||||
---
|
||||
|
||||
## 4. Harness changes
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `harness/.pi/extensions/tht-gate.js` | Emit the `schema-linking` descriptor (merge model proposal + catalog columns). Add a response handler for `widget:"schema-linking"`: per enacted table → `tht decision add` (`table_promoted`/`table_excluded`, as today); per selected column of a promoted table → new `column_promoted` decision (`subject = "table.col"`); suggested-but-deselected → `column_excluded`. Then invoke the deterministic reconcile (below). |
|
||||
| catalog reader (`tht`) | A deterministic reader returning a table's columns + comments from `physical.yaml` (`[{name, description, type, pk}]`). **Verify whether an existing `tht schema …`/search command already exposes this; add a thin command if not.** |
|
||||
| `schema_linking.json` reconcile (`tht`) | A deterministic `tht` command that sets a promoted table's column candidates to exactly the reviewer's selection (selected → `decision: "promoted"`; deselected suggested → dropped or `"excluded"`; newly selected → added). Makes the curation **authoritative** rather than relying on model transcription. |
|
||||
| `harness/tht/decisions.py` | Add `column_promoted` / `column_excluded` to `DecisionType`. |
|
||||
| `harness/workflow.yaml` | Add `column_promoted` / `column_excluded` to F4 `emits`. |
|
||||
| `harness/.pi/skills/tht-sessione/SKILL.md` | F4: describe the column-curation step + the new gate. F6/F7: instruct the model to treat `schema_linking.json` promoted columns as the reviewer-approved **output** set, while remaining free to use other columns as **join keys / filter predicates** (Option 1, soft). |
|
||||
|
||||
Sequencing of "model proposes → gate reconciles `schema_linking.json`" is a detail to
|
||||
pin down in the plan (the reconcile must run after the tables exist in the artifact).
|
||||
|
||||
---
|
||||
|
||||
## 5. Downstream (Option 1, soft)
|
||||
|
||||
- `harness/tht/cli/sql_cmd.py`: `promoted_tables_for` stays. Optionally add a
|
||||
`promoted_columns_for` helper (read promoted column candidates) for **future** use;
|
||||
it is **not** wired into validation now — no SQL is rejected on column grounds.
|
||||
- The only behavioral coupling now is via `SKILL.md` guidance + the curated
|
||||
`schema_linking.json` the model reads when composing CTEs/SQL.
|
||||
|
||||
---
|
||||
|
||||
## 6. Testing & replay (no VPN)
|
||||
|
||||
- **Augmented replay fixture.** A script reads `physical.yaml` for the 6 tables of the
|
||||
recorded psd session and rewrites replay gate 7 as a `schema-linking` descriptor
|
||||
with real `columns`. This makes the new widget **visually verifiable in replay**
|
||||
(server on :5333) before any harness wiring exists. Touches
|
||||
`tools/replay/extract.mjs` + regenerates `tools/replay/replay.json`.
|
||||
- **Frontend unit (vitest + MSW):** widget staging (suggested pre-selected, count),
|
||||
bold-toggle on select/deselect, read-only for excluded, structured response shape on
|
||||
Confirm, reserved controls.
|
||||
- **Harness unit:** `tht-gate.js` response handler (structured response → the right
|
||||
`tht decision add` calls + reconcile invocation); pytest for the catalog reader and
|
||||
the `schema_linking.json` reconcile command.
|
||||
- **End-to-end with the live model is deferred** (no VPN); the soft SQL guidance can
|
||||
only be spot-checked in replay.
|
||||
|
||||
---
|
||||
|
||||
## 7. Phased delivery (one spec, plan sequenced in phases)
|
||||
|
||||
1. **Frontend + contract + replay fixture** — new widget, columns modal, structured
|
||||
`UiResponse`, registry; augmented replay fixture. Fully verifiable offline.
|
||||
2. **Harness** — gate emission (catalog merge + `suggested`), structured-response
|
||||
handler, ledger types, `schema_linking.json` reconcile, `tht` commands, `SKILL.md`
|
||||
guidance.
|
||||
|
||||
Follow-up (separate spec, when VPN is available): **Option 2** hard SQL enforcement.
|
||||
|
||||
---
|
||||
|
||||
## 8. Risks & open decisions
|
||||
|
||||
- **Interaction shape** (§2): inline gate + single columns modal is chosen over
|
||||
gate-as-modal + nested modal to avoid nested-dialog complexity. Revisit only if the
|
||||
reviewer explicitly wants the gate itself modal.
|
||||
- **Reconcile ordering** (§4): the deterministic `schema_linking.json` reconcile must
|
||||
run after the table candidates exist; exact sequencing pinned in the plan.
|
||||
- **Deselecting a key column** (e.g. `cod_paz`): under Option 1 this does not block —
|
||||
`SKILL.md` notes join keys remain usable even when not selected as output columns.
|
||||
A soft UI hint ("key column") is possible but not required for v1.
|
||||
- **Catalog reader existence** (§4): confirm whether `tht` already exposes
|
||||
`physical.yaml` columns; add a thin command only if missing.
|
||||
Reference in New Issue
Block a user