ThothII/docs/superpowers/plans/2026-07-06-f4-schema-linking-column-curation-harness.md

F4 schema-linking column curation — Plan 2 (harness + persistence + live)

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking. Depends on Plan 1 (the schema-linking frontend widget + UiResponse.tables contract) being merged.

Goal: Make the F4 gate emit the schema-linking descriptor (real catalog columns, suggested pre-flagged), record the reviewer's per-table column curation as authoritative ledger decisions, project them deterministically into schema_linking.json, and instruct the model to honor the curated columns downstream — verified end-to-end on the live stack.

Architecture: A new Pi reviewer tool reviewer_schema_linking (in tht-gate.js) enriches the model's table proposal with catalog columns (tht schema columns), emits the descriptor via the existing emitAndWait, then records table_promoted/table_excluded + column_promoted/column_excluded decisions and calls a deterministic projection (tht session sync-schema-linking) that rebuilds schema_linking.json from the ledger. The backend is already response-shape-agnostic (verify only). Downstream honoring is soft (Option 1): SKILL.md guidance, no hard SQL validator.

Tech Stack: Python 3 (typer CLI, pydantic models), Node ESM Pi extension (node --test), Fastify backend (vitest). Live stack via ./scripts/run-stack.sh (needs VPN + harness/.env + pi on PATH).

Global Constraints

• tht's -c/--config is a per-command option — it follows the subcommand, never precedes it.
• --json output must be pristine (only valid JSON on stdout) — it is a machine contract consumed by the extension.
• A decision type is only accepted in a phase listed in that phase's emits in workflow.yaml (drives decision_min_phase / require_phase_or_exit). New types must be added to F4 emits or they are rejected.
• The extension's privileged tht calls go through tht(ctx, args) = execFileSync("tht", args, {cwd: ctx.cwd}); the anti-bypass hook blocks only the model's bash calls, not the extension.
• SchemaLinking pydantic model has extra:"forbid" — the projected dict must match its fields exactly.
• UI chrome English; data content (table/column names, Italian descriptions) as-is. No em dashes in copy you write.
• Harness Python tests: harness/.venv/bin/pytest -q (mark l2 for live GLM+DB is opt-in). JS gate tests: cd harness && npm test (node --test .pi/extensions/gate/__tests__/*.test.js). Backend: cd backend && npx vitest run + npx tsc --noEmit -p ..

---

File Structure

File	Change
harness/tht/cli/schema_cmd.py (modify)	New tht schema columns <table> --json reader over physical.yaml.
harness/tht/decisions.py (modify)	Add column_promoted, column_excluded to DecisionType.
harness/workflow.yaml (modify)	Add the two types to F4 emits.
harness/tht/session/store.py (modify)	sync_schema_linking(session_id, sessions_root) projection from the ledger.
harness/tht/cli/session_cmd.py (modify)	tht session sync-schema-linking <id> command.
harness/.pi/extensions/gate/builders.js (modify)	buildSchemaLinkingRequest(...).
harness/.pi/extensions/tht-gate.js (modify)	reviewer_schema_linking tool; extend prepareReviewerArguments for stringified tables.
harness/.pi/skills/tht-sessione/SKILL.md (modify)	Phase 4: use reviewer_schema_linking; honor curated columns downstream.
harness/tht/cli/sql_cmd.py (modify)	promoted_columns_for helper (persisted set; surfaced, not enforced).
harness/tests/… + backend/test/… (tests)	pytest + node:test + a backend passthrough test.

Non-goal (documented, not built): an AST-level "output column diverges from curated set" warning in sqlcheck. Rationale: the final projection references CTE-qualified columns, not base table.column, so mapping back through CTEs is fragile; the human F7 gate is the backstop. We persist and surface the curated set instead.

---

Task 1: Catalog reader — tht schema columns

Files:

• Modify: harness/tht/cli/schema_cmd.py
• Test: harness/tests/test_schema_columns_cmd.py (create)

Interfaces:

• Produces: tht schema columns <table> --json printing {"table": str, "description": str, "columns": [{"name": str, "description": str, "type": str, "pk": bool}]}.

• Step 1: Write the failing test

Create harness/tests/test_schema_columns_cmd.py:

import json
from datetime import datetime
from typer.testing import CliRunner
from tht.cli import app  # the root Typer app that mounts schema_app as "schema"
from tht.mschema.models import PhysicalSchema, TablePhysical, ColumnPhysical


def _write_catalog(tmp_path):
    phys = PhysicalSchema(
        database="d", schema="s", introspected_at=datetime(2026, 1, 1),
        tables={
            "dim_patient": TablePhysical(
                comment="Anagrafica",
                columns={
                    "cod_paz": ColumnPhysical(type="bigint", pk=True, comment="Codice paziente"),
                    "nome": ColumnPhysical(type="text", comment="Nome"),
                },
            )
        },
    )
    out = tmp_path / "artifacts" / "mschema" / "physical.yaml"
    phys.to_yaml(out)
    return out


def test_schema_columns_json(tmp_path, monkeypatch):
    _write_catalog(tmp_path)
    cfg = tmp_path / "workspace.yaml"
    cfg.write_text(
        "database: {transport: none, database: d, schema: s}\n"
        f"paths: {{artifacts: {tmp_path/'artifacts'}, indexes: {tmp_path/'i'}, sessions: {tmp_path/'s'}}}\n"
    )
    res = CliRunner().invoke(app, ["schema", "columns", "dim_patient", "--json", "-c", str(cfg)])
    assert res.exit_code == 0, res.output
    data = json.loads(res.output)
    assert data["table"] == "dim_patient"
    assert data["description"] == "Anagrafica"
    assert {"name": "cod_paz", "description": "Codice paziente", "type": "bigint", "pk": True} in data["columns"]

Note: the config file shape must match load_config. Read one existing harness/workspaces/*.yaml and an existing schema test (e.g. any test invoking schema render) first and mirror their config fixture exactly; adjust the cfg.write_text block to the real minimal config. If a shared config fixture helper exists in harness/tests/conftest.py, use it.

• Step 2: Run test to verify it fails

Run: cd harness && .venv/bin/pytest tests/test_schema_columns_cmd.py -q Expected: FAIL — no such command columns.

• Step 3: Implement the command

In harness/tht/cli/schema_cmd.py, add after render_cmd:

@schema_app.command("columns")
def columns_cmd(
    table: str = typer.Argument(..., help="Nome tabella (chiave in physical.yaml)."),
    json_out: bool = typer.Option(False, "--json", help="Emetti JSON puro su stdout."),
    config: Path = CONFIG_OPT,
) -> None:
    """Elenca nome/descrizione/tipo/pk delle colonne di una tabella dal catalogo."""
    import json as _json

    from tht.mschema.models import PhysicalSchema

    cfg = _load_config_or_exit(config)
    phys_file = physical_path(cfg)
    if not phys_file.exists():
        typer.secho(
            f"ERRORE: {phys_file} non trovato. Esegui prima `tht schema introspect`.",
            fg=typer.colors.RED, err=True,
        )
        raise typer.Exit(code=1)
    physical = PhysicalSchema.from_yaml(phys_file)
    tbl = physical.tables.get(table)
    if tbl is None:
        typer.secho(f"ERRORE: tabella non nel catalogo: {table}", fg=typer.colors.RED, err=True)
        raise typer.Exit(code=1)
    payload = {
        "table": table,
        "description": tbl.comment,
        "columns": [
            {"name": name, "description": col.comment, "type": col.type, "pk": col.pk}
            for name, col in tbl.columns.items()
        ],
    }
    if json_out:
        typer.echo(_json.dumps(payload, ensure_ascii=False))
        return
    typer.echo(f"{table}: {tbl.comment}")
    for c in payload["columns"]:
        typer.echo(f"  {'*' if c['pk'] else ' '} {c['name']} ({c['type']}) — {c['description']}")

• Step 4: Run test to verify it passes

Run: cd harness && .venv/bin/pytest tests/test_schema_columns_cmd.py -q Expected: PASS.

• Step 5: Lint + commit

Run: cd harness && .venv/bin/ruff check tht/cli/schema_cmd.py

git add harness/tht/cli/schema_cmd.py harness/tests/test_schema_columns_cmd.py
git commit -m "feat(tht): schema columns reader (name/description/type/pk) from catalog"

---

Task 2: Ledger decision types + workflow emits

Files:

• Modify: harness/tht/decisions.py
• Modify: harness/workflow.yaml
• Test: harness/tests/test_column_decisions.py (create)

Interfaces:

• Produces: column_promoted, column_excluded accepted by append_decision and by F4's phase-eligibility.

• Step 1: Write the failing test

Create harness/tests/test_column_decisions.py:

from tht.decisions import DecisionType
import typing


def test_column_decision_types_exist():
    allowed = set(typing.get_args(DecisionType))
    assert "column_promoted" in allowed
    assert "column_excluded" in allowed


def test_f4_emits_column_types():
    import yaml
    from pathlib import Path
    wf = yaml.safe_load(Path("workflow.yaml").read_text())
    f4 = next(p for p in wf["phases"] if p["id"] == "F4")
    assert "column_promoted" in f4["emits"]
    assert "column_excluded" in f4["emits"]

• Step 2: Run to verify it fails

Run: cd harness && .venv/bin/pytest tests/test_column_decisions.py -q Expected: FAIL.

• Step 3: Add the types

In harness/tht/decisions.py, add to the DecisionType Literal (next to column_corrected):

    "column_promoted",
    "column_excluded",

In harness/workflow.yaml, F4 emits — append the two types:

    emits: [table_promoted, table_excluded, column_promoted, column_excluded, column_corrected, join_modified,
            evidence_accepted, evidence_rejected, value_grounded,
            concept_formula_approved, concept_formula_rejected]

• Step 4: Run to verify it passes + full suite unaffected

Run: cd harness && .venv/bin/pytest tests/test_column_decisions.py -q && .venv/bin/pytest -q Expected: new tests PASS; suite green (excluding opt-in l2).

• Step 5: Commit

git add harness/tht/decisions.py harness/workflow.yaml harness/tests/test_column_decisions.py
git commit -m "feat(tht): column_promoted/column_excluded decision types (F4)"

---

Task 3: Deterministic projection — tht session sync-schema-linking

Files:

• Modify: harness/tht/session/store.py
• Modify: harness/tht/cli/session_cmd.py
• Test: harness/tests/test_sync_schema_linking.py (create)

Interfaces:

• Consumes: effective_decisions (tht.phase), set_schema_linking (Task uses existing), DecisionRecord.

• Produces: sync_schema_linking(session_id, sessions_root) -> Path and tht session sync-schema-linking <id>.

• Step 1: Write the failing test

Create harness/tests/test_sync_schema_linking.py:

import json
from tht.session.store import sync_schema_linking, set_schema_linking
from tht.decisions import append_decision


def _new_session(tmp_path):
    # Mirror the minimal session fixture used by other store tests (conftest helper
    # if present). Must create session_manifest.yaml with a `question`.
    from tht.session.store import create_session
    from tht.config import DatabaseConfig  # adjust import to the real DatabaseConfig
    db = DatabaseConfig(transport="none", database="d", db_schema="s")
    m = create_session("domanda X", db, tmp_path)
    return m.id


def test_projection_from_ledger(tmp_path):
    sid = _new_session(tmp_path)
    sdir = tmp_path / sid
    # seed a prior schema_linking with a join to prove joins are preserved
    set_schema_linking(sid, {"question": "domanda X", "candidates": [], "joins": [
        {"from": "dim_patient.cod_paz", "to": "fact_x.cod_paz"}], "excluded": []}, tmp_path)
    append_decision(sdir, type="table_promoted", subject="dim_patient")
    append_decision(sdir, type="column_promoted", subject="dim_patient.cod_paz")
    append_decision(sdir, type="column_promoted", subject="dim_patient.nome")
    append_decision(sdir, type="table_excluded", subject="fact_sost")

    sync_schema_linking(sid, tmp_path)

    data = json.loads((sdir / "schema_linking.json").read_text())
    tabs = {(c["kind"], c["name"], c["decision"]) for c in data["candidates"]}
    assert ("table", "dim_patient", "promoted") in tabs
    assert ("column", "dim_patient.cod_paz", "promoted") in tabs
    assert ("column", "dim_patient.nome", "promoted") in tabs
    assert {"kind": "table", "name": "fact_sost"} in [
        {"kind": e["kind"], "name": e["name"]} for e in data["excluded"]]
    assert data["joins"], "existing joins must be preserved"

Note: read harness/tests/conftest.py and an existing store test (e.g. test_session_mutations.py) first; reuse their session-creation fixture and the real DatabaseConfig constructor signature instead of the placeholder import above.

• Step 2: Run to verify it fails

Run: cd harness && .venv/bin/pytest tests/test_sync_schema_linking.py -q Expected: FAIL — sync_schema_linking undefined.

• Step 3: Implement the projection

In harness/tht/session/store.py, add:

def sync_schema_linking(session_id: str, sessions_root: Path) -> Path:
    """Project the effective F4 ledger decisions into schema_linking.json.

    candidates/excluded are rebuilt from table_promoted/table_excluded +
    column_promoted/column_excluded (last decision per subject wins). question,
    joins, concept_formulas and open_questions are preserved from the existing
    file when present. The reviewer's curation is thus authoritative and
    deterministic (no model transcription)."""
    from tht.phase import effective_decisions

    session_dir = sessions_root / session_id
    manifest = load_session(session_id, sessions_root)

    existing: dict = {}
    sl_path = session_dir / "schema_linking.json"
    if sl_path.exists():
        existing = json.loads(sl_path.read_text())

    # last decision per subject wins (handles a re-run of the gate).
    latest: dict[str, str] = {}
    for d in effective_decisions(session_dir):
        if d.type in ("table_promoted", "table_excluded", "column_promoted", "column_excluded"):
            latest[d.subject] = d.type

    candidates: list[dict] = []
    excluded: list[dict] = []
    for subject, dtype in latest.items():
        is_column = "." in subject
        kind = "column" if is_column else "table"
        if dtype in ("table_promoted", "column_promoted"):
            candidates.append({"kind": kind, "name": subject, "decision": "promoted"})
        else:
            excluded.append({"kind": kind, "name": subject})

    data = {
        "question": existing.get("question") or manifest.question,
        "candidates": candidates,
        "joins": existing.get("joins", []),
        "excluded": excluded,
        "open_questions": existing.get("open_questions", []),
        "concept_formulas": existing.get("concept_formulas", []),
    }
    return set_schema_linking(session_id, data, sessions_root)

In harness/tht/cli/session_cmd.py, add a command (near set_schema_linking_cmd):

@session_app.command("sync-schema-linking")
def sync_schema_linking_cmd(
    session_id: str = typer.Argument(...),
    config: Path = CONFIG_OPT,
) -> None:
    """Riproietta schema_linking.json dalle decisioni F4 del ledger (deterministico)."""
    from tht.session.store import sync_schema_linking

    cfg = _load_config_or_exit(config)
    load_session_or_exit(cfg, session_id)
    path = sync_schema_linking(session_id, cfg.paths.sessions)
    typer.secho(f"OK: schema_linking.json riproiettato ({path}).", fg=typer.colors.GREEN)

• Step 4: Run to verify it passes

Run: cd harness && .venv/bin/pytest tests/test_sync_schema_linking.py -q Expected: PASS.

• Step 5: Lint + commit

Run: cd harness && .venv/bin/ruff check tht/session/store.py tht/cli/session_cmd.py

git add harness/tht/session/store.py harness/tht/cli/session_cmd.py harness/tests/test_sync_schema_linking.py
git commit -m "feat(tht): sync-schema-linking projects F4 ledger into schema_linking.json"

---

Task 4: Descriptor builder — buildSchemaLinkingRequest

Files:

• Modify: harness/.pi/extensions/gate/builders.js
• Test: harness/.pi/extensions/gate/__tests__/builders.test.js (append)

Interfaces:

• Produces: buildSchemaLinkingRequest({ id, phase, title, tables }) -> { type:"ui_request", widget:"schema-linking", tables, reserved }.

• Step 1: Write the failing test

Append to harness/.pi/extensions/gate/__tests__/builders.test.js (match its existing node:test import style):

test("buildSchemaLinkingRequest carries tables + reserved", () => {
  const { buildSchemaLinkingRequest } = require("../builders.js");
  const out = buildSchemaLinkingRequest({
    id: "u7", phase: "F4", title: "F4 — Schema linking",
    tables: [{ id: "t1", name: "dim_patient", kind: "promote", columns: [] }],
  });
  assert.equal(out.widget, "schema-linking");
  assert.equal(out.id, "u7");
  assert.equal(out.tables.length, 1);
  assert.deepEqual(out.reserved, ["back", "exit", "other"]);
});

• Step 2: Run to verify it fails

Run: cd harness && node --test .pi/extensions/gate/__tests__/builders.test.js Expected: FAIL — buildSchemaLinkingRequest is not a function.

• Step 3: Implement the builder

In harness/.pi/extensions/gate/builders.js, add before module.exports:

// Build a `schema-linking` ui_request: table rows (promote/exclude) each carrying
// their full catalog columns; the frontend renders rows + a per-table columns
// modal (suggested pre-selected). `tables` is validated shallowly here.
function buildSchemaLinkingRequest({ id, phase, title, tables }) {
  requireString(title, "title", "schema-linking");
  const tabs = requireArray(tables, "tables", "schema-linking");
  return {
    type: "ui_request",
    id,
    phase,
    schema_version: SCHEMA_VERSION,
    widget: "schema-linking",
    title,
    tables: [...tabs],
    reserved: RESERVED,
  };
}

Add buildSchemaLinkingRequest, to the module.exports object.

• Step 4: Run to verify it passes

Run: cd harness && node --test .pi/extensions/gate/__tests__/builders.test.js Expected: PASS.

• Step 5: Commit

git add harness/.pi/extensions/gate/builders.js harness/.pi/extensions/gate/__tests__/builders.test.js
git commit -m "feat(gate): buildSchemaLinkingRequest descriptor builder"

---

Task 5: Reviewer tool reviewer_schema_linking + stringified tables

Files:

• Modify: harness/.pi/extensions/tht-gate.js
• Test: harness/.pi/extensions/gate/__tests__/gate_stringified_params.test.js (append) + a new roundtrip test file.

Interfaces:

• Consumes: buildSchemaLinkingRequest (Task 4), emitAndWait, decisionAddArgs, tht, relayIfThtFails (all in tht-gate.js), tht schema columns --json (Task 1), tht session sync-schema-linking (Task 3).

• Produces: Pi tool reviewer_schema_linking; prepareReviewerArguments parses a stringified tables array.

• Step 1: Extend prepareReviewerArguments (stringified tables)

In harness/.pi/extensions/tht-gate.js, inside prepareReviewerArguments, add next to the options block:

	if (typeof args.tables === "string") {
		try {
			const parsed = JSON.parse(args.tables);
			if (Array.isArray(parsed)) args.tables = parsed;
		} catch {
			/* not JSON */
		}
	}

• Step 2: Write the failing stringified test

Append to harness/.pi/extensions/gate/__tests__/gate_stringified_params.test.js (mirror its existing imports of prepareReviewerArguments):

test("prepareReviewerArguments parses stringified tables array", () => {
  const out = prepareReviewerArguments({ tables: JSON.stringify([{ id: "t1", name: "x" }]) });
  assert.ok(Array.isArray(out.tables));
  assert.equal(out.tables[0].id, "t1");
});

Run: cd harness && node --test .pi/extensions/gate/__tests__/gate_stringified_params.test.js Expected: FAIL before Step 1's edit, PASS after. (Apply Step 1, then rerun to confirm PASS.)

• Step 3: Import the builder

In tht-gate.js, add buildSchemaLinkingRequest, to the existing import { … } from "./gate/builders.js"; block (alongside buildMultiselectRequest).

• Step 4: Register the tool

In tht-gate.js, after the reviewer_decide pi.registerTool({...}) block, add:

	pi.registerTool({
		name: "reviewer_schema_linking",
		label: "Schema linking: tabelle/colonne/esclusioni (reviewer)",
		description:
			"F4: presenta al reviewer le tabelle da promuovere/escludere con la loro descrizione " +
			"e, per ogni tabella, le colonne dal catalogo (le suggerite pre-selezionate). Il reviewer " +
			"cura le colonne di ogni tabella promossa. PERSISTE table_promoted/table_excluded e " +
			"column_promoted/column_excluded, poi riproietta schema_linking.json in modo deterministico " +
			"(tht session sync-schema-linking). `tables[]`: {id, name, kind: 'promote'|'exclude', " +
			"rationale?, suggested_columns?: string[]}. Le colonne complete arrivano dal catalogo, non dal modello.",
		parameters: Type.Object({
			session: Type.String(),
			title: Type.String(),
			tables: Type.Array(
				Type.Object({
					id: Type.String(),
					name: Type.String(),
					kind: Type.Union([Type.Literal("promote"), Type.Literal("exclude")]),
					rationale: Type.Optional(Type.String()),
					suggested_columns: Type.Optional(Type.Array(Type.String())),
					recommended: Type.Optional(Type.Boolean()),
				}),
			),
			advance: Type.Optional(Type.Boolean()),
		}),
		prepareArguments: prepareReviewerArguments,
		async execute(_id, params, _signal, _onUpdate, ctx) {
			lockActive = true;
			const { session, title, tables, advance } = params;
			const phase = phaseId(ctx, currentPhase(ctx, session));

			// Enrich each table with its full catalog columns (deterministic source).
			const enriched = tables.map((t) => {
				const cat = JSON.parse(tht(ctx, ["schema", "columns", t.name, "--json"]));
				const suggested = new Set(t.suggested_columns ?? []);
				return {
					id: t.id,
					name: t.name,
					kind: t.kind,
					recommended: t.recommended ?? true,
					description: cat.description ?? "",
					rationale: t.rationale ?? "",
					columns: cat.columns.map((c) => ({ ...c, suggested: suggested.has(c.name) })),
				};
			});

			const widget = buildSchemaLinkingRequest({
				id: `u${Date.now()}`,
				phase,
				title,
				tables: enriched,
			});
			const resp = await emitAndWait(ctx, widget);
			if (resp.control === "back") return textResult("Il reviewer vuole tornare indietro.");
			if (resp.control === "exit") return textResult("Il reviewer vuole uscire.");
			if (resp.control === "freetext")
				return textResult(`Altro (reviewer): ${resp.text}. Riformula tenendone conto.`);

			const byId = new Map(enriched.map((t) => [t.id, t]));
			let n = 0;
			for (const rt of resp.tables ?? []) {
				const t = byId.get(rt.id);
				if (!t || !rt.enacted) continue;
				if (t.kind === "promote") {
					const e1 = relayIfThtFails(ctx, decisionAddArgs(session, {
						type: "table_promoted", subject: t.name, detail: t.description, rationale: t.rationale,
					}), "");
					if (e1) return e1;
					n++;
					const sel = new Set(rt.columns ?? []);
					for (const c of t.columns) {
						const type = sel.has(c.name)
							? "column_promoted"
							: (c.suggested ? "column_excluded" : null);
						if (!type) continue;
						const e2 = relayIfThtFails(ctx, decisionAddArgs(session, {
							type, subject: `${t.name}.${c.name}`, detail: c.description ?? "",
						}), "");
						if (e2) return e2;
					}
				} else {
					const e3 = relayIfThtFails(ctx, decisionAddArgs(session, {
						type: "table_excluded", subject: t.name, detail: t.description, rationale: t.rationale,
					}), "");
					if (e3) return e3;
					n++;
				}
			}

			// Deterministic projection of the ledger into schema_linking.json.
			const eSync = relayIfThtFails(ctx, ["session", "sync-schema-linking", session], "");
			if (eSync) return eSync;

			if (advance) advanceIfReady(ctx, session);
			return textResult(`Schema linking registrato dal reviewer (${n} tabelle + colonne curate).`);
		},
	});

• Step 5: Write a roundtrip test

Create harness/.pi/extensions/gate/__tests__/gate_schema_linking.test.js, modeled on gate_roundtrip.test.js (read it first for the fake_pi_runtime harness + how it stubs ctx.ui.input, execFileSync/tht). The test must:

• stub tht schema columns dim_patient --json → {"table":"dim_patient","description":"Anagrafica","columns":[{"name":"cod_paz","description":"Codice","type":"bigint","pk":true},{"name":"nome","description":"Nome","type":"text","pk":false}]};
• stub ctx.ui.input to return JSON.stringify({ id:<descriptor.id>, kind:"schema-linking", tables:[{ id:"t-pat", enacted:true, columns:["cod_paz"] }] });
• assert the recorded tht decision add calls include table_promoted dim_patient, column_promoted dim_patient.cod_paz, column_excluded dim_patient.nome, and that session sync-schema-linking was invoked.

Use the same capture mechanism gate_roundtrip.test.js uses to intercept tht(...) argv.

• Step 6: Run the gate tests

Run: cd harness && npm test Expected: all gate tests PASS, including the new roundtrip + stringified cases.

• Step 7: Commit

git add harness/.pi/extensions/tht-gate.js harness/.pi/extensions/gate/__tests__/gate_stringified_params.test.js harness/.pi/extensions/gate/__tests__/gate_schema_linking.test.js
git commit -m "feat(gate): reviewer_schema_linking tool with per-column curation + deterministic sync"

---

Task 6: SKILL.md Phase 4 guidance

Files:

• Modify: harness/.pi/skills/tht-sessione/SKILL.md

• Step 1: Read the current Phase 4 section

Read the ## Phase 4 — Schema linking section (after Phase 3, ~line 200+) and the phase-map row (line 40). Anchor edits there.

• Step 2: Replace the tables/columns/exclusions step

Where Phase 4 currently instructs a reviewer_decide for tables/columns/exclusions, replace with (keep the joins gate and reviewer_confirm kind:"phase" as they are):

1. Propose tables to promote/exclude with **`reviewer_schema_linking`**: pass
   `tables[]` as `{ id, name, kind: "promote"|"exclude", rationale, suggested_columns }`.
   Do NOT list every column yourself — the gate loads the full column set (with
   descriptions) from the catalog and pre-selects your `suggested_columns`. The
   reviewer curates the columns per promoted table. The tool records
   `table_promoted`/`table_excluded` + `column_promoted`/`column_excluded` and
   re-projects `schema_linking.json` deterministically (you do NOT hand-write the
   tables/columns part with `write_schema_linking`).

• Step 3: Add the downstream honoring rule

In the phase-map row for F4 and in the Phase 6/7 sections, add:

The promoted columns in `schema_linking.json` are the reviewer-approved OUTPUT
columns: project exactly those in the final SELECT. You remain free to reference
other columns as join keys or filter predicates when the query requires them.

• Step 4: Sanity + commit

No automated test (prose). Re-read the edited section for consistency with the tool's actual behavior.

git add harness/.pi/skills/tht-sessione/SKILL.md
git commit -m "docs(skill): Phase 4 uses reviewer_schema_linking; honor curated output columns"

---

Task 7: Backend passthrough verification (no prod change)

Files:

• Test: backend/test/schema-linking-passthrough.test.ts (create)

Rationale: SessionBridge.respond(uiResponse) sends { type:"extension_ui_response", id, value: JSON.stringify(uiResponse) } — it is already shape-agnostic, so the structured { tables: [...] } reaches Pi untouched. This task pins that with a test; no production code changes.

• Step 1: Write the test

Create backend/test/schema-linking-passthrough.test.ts, modeled on the existing bridge test (read backend/test/ for the RpcClient stub pattern):

import { describe, it, expect, vi } from "vitest";
import { SessionBridge } from "../src/bridge/session-bridge";

describe("schema-linking response passthrough", () => {
  it("forwards the structured tables payload to Pi verbatim", () => {
    const send = vi.fn();
    const rpc = { on: vi.fn(), send } as any;
    const bridge = new SessionBridge(rpc);
    // simulate a pending Pi ui request id
    (bridge as any).pendingPiId = "pi-123";
    const resp = { id: "u7", kind: "schema-linking", tables: [{ id: "t1", enacted: true, columns: ["cod_paz"] }] };
    bridge.respond(resp);
    expect(send).toHaveBeenCalledWith({
      type: "extension_ui_response",
      id: "pi-123",
      value: JSON.stringify(resp),
    });
  });
});

Note: adjust the private-field priming (pendingPiId) to how respond actually reads the pending id (read session-bridge.ts:respond); if respond no-ops without a pending id, set it as the real code expects.

• Step 2: Run + typecheck + commit

Run: cd backend && npx vitest run test/schema-linking-passthrough.test.ts && npx tsc --noEmit -p .

git add backend/test/schema-linking-passthrough.test.ts
git commit -m "test(backend): pin schema-linking structured response passthrough"

---

Task 8: promoted_columns_for helper (surface, not enforce)

Files:

• Modify: harness/tht/cli/sql_cmd.py
• Test: harness/tests/test_promoted_columns_for.py (create)

Rationale: expose the curated column set as a persisted, queryable helper (used by SKILL surfacing / future work). The AST divergence warning is a documented non-goal (fragile through CTEs).

• Step 1: Write the failing test

Create harness/tests/test_promoted_columns_for.py:

import json
from tht.cli.sql_cmd import promoted_columns_for


def test_promoted_columns_for(tmp_path):
    sid = "sess1"
    sdir = tmp_path / sid
    sdir.mkdir(parents=True)
    (sdir / "schema_linking.json").write_text(json.dumps({
        "question": "q",
        "candidates": [
            {"kind": "table", "name": "dim_patient", "decision": "promoted"},
            {"kind": "column", "name": "dim_patient.cod_paz", "decision": "promoted"},
            {"kind": "column", "name": "dim_patient.nome", "decision": "excluded"},
        ],
        "joins": [], "excluded": [],
    }))

    class Cfg:
        class paths:  # noqa: N801
            sessions = tmp_path
    assert promoted_columns_for(Cfg, sid) == {"dim_patient.cod_paz"}

• Step 2: Run to verify it fails

Run: cd harness && .venv/bin/pytest tests/test_promoted_columns_for.py -q Expected: FAIL — promoted_columns_for undefined.

• Step 3: Implement

In harness/tht/cli/sql_cmd.py, add next to promoted_tables_for:

def promoted_columns_for(cfg, session_id: str | None) -> set[str] | None:
    if session_id is None:
        return None
    linking_path = cfg.paths.sessions / session_id / "schema_linking.json"
    if not linking_path.exists():
        return None
    from tht.session.models import SchemaLinking

    linking = SchemaLinking.model_validate(json.loads(linking_path.read_text()))
    return {
        c.name for c in linking.candidates
        if c.kind == "column" and c.decision == "promoted"
    }

• Step 4: Run + lint + commit

Run: cd harness && .venv/bin/pytest tests/test_promoted_columns_for.py -q && .venv/bin/ruff check tht/cli/sql_cmd.py

git add harness/tht/cli/sql_cmd.py harness/tests/test_promoted_columns_for.py
git commit -m "feat(tht): promoted_columns_for helper (curated column set)"

---

Task 9: Live end-to-end verification

Files: none (verification procedure). Requires VPN + harness/.env + pi on PATH.

• Step 1: Rebuild + start the full stack

Run: ./scripts/run-stack.sh (frontend :5173 → backend :8787). Confirm both are up.

• Step 2: Drive an F4 session to the schema-linking gate

In the browser (:5173), start a new session with a question that promotes several tables (reuse the psd ablation question). Advance F1 to F4. At F4 the gate now renders as the schema-linking widget (table rows + description + rationale + "Colonne k/n").

• Step 3: Curate columns and confirm

• For a promoted table, open the modal: suggested columns are pre-checked + bold.

• Deselect one suggested column and select one previously-unsuggested column; confirm the bold updates and the row count changes.

• Confirm the gate.

• Step 4: Verify persistence on disk

Run (psd sessions path):

SID=<the session id>
cd /Users/mp/projects/tht-workspace-psd/sessions/$SID
grep -E '"type": ?"(table_promoted|table_excluded|column_promoted|column_excluded)"' review_decisions.jsonl
node -e 'const j=require("./schema_linking.json");console.log(j.candidates.filter(c=>c.kind==="column"&&c.decision==="promoted").map(c=>c.name))'

Expected: the ledger shows the per-column decisions matching your clicks; schema_linking.json promoted columns equal exactly the curated set (deselected suggested column absent, newly selected column present).

• Step 5: Verify downstream honoring (soft)

Continue the session through F5/F6 to F7 (final SQL). Inspect sql_final.sql:

• The final SELECT projects the curated output columns (the deselected column is not in the output; the added column is).
• Join keys / filter predicates outside the curated set are still allowed (no rejection).

Record the outcome. If the model ignores the curated set, tighten the Task 6 SKILL.md wording and re-run (this is the only step that exercises the model's honoring).

• Step 6: Regression sweep

Run all offline suites:

cd harness && .venv/bin/pytest -q && npm test
cd ../backend && npx vitest run && npx tsc --noEmit -p .
cd ../frontend && npx vitest run && npx tsc -b

Expected: all green.

---

Self-Review

Spec coverage (harness slice of the design doc):

• §1a descriptor built by harness (catalog merge + suggested) → Tasks 1, 4, 5. ✅
• §1b structured response consumed → Task 5 handler; §4 backend passthrough → Task 7. ✅
• §4 ledger types → Task 2; deterministic schema_linking.json reconcile → Task 3; catalog reader → Task 1; SKILL.md → Task 6. ✅
• §5 downstream soft (Option 1) → Task 6 guidance + Task 8 helper; hard enforcement remains out of scope (documented non-goal). ✅
• §6 live verification (now possible with VPN) → Task 9. ✅

Placeholder scan: code is concrete. Three tasks (1, 3, 5) explicitly instruct reading a neighboring fixture/test first to mirror exact config/harness shapes (test scaffolding that genuinely depends on existing conventions) rather than guessing them; the production code in every task is complete.

Type consistency: decision types column_promoted/column_excluded identical across decisions.py, workflow.yaml, the gate handler, and sync projection. tht schema columns JSON shape ({table, description, columns:[{name,description,type,pk}]}) produced in Task 1 and consumed verbatim in Task 5. Response shape { id, kind:"schema-linking", tables:[{id,enacted,columns?}] } matches Plan 1 and the Task 5/7 consumers. sync_schema_linking(session_id, sessions_root) signature identical in store.py, the CLI command, and the gate's session sync-schema-linking shell call.

Ordering note for implementer: Task 5's gate handler depends on Tasks 1 (schema columns), 3 (sync-schema-linking), and 4 (builder) existing; keep the task order. Task 9 depends on Plan 1 being merged so the frontend can render the descriptor.