mptyl/ThothII
1
0
Fork 0
Code Issues 25 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat: protect sensitive catalog samples

Browse Source
This commit is contained in:
Codex
2026-08-30 12:14:23 +02:00
parent 6278ee9d81
commit 0736983bc5
28 changed files with 1162 additions and 128 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTEXT.md
+11 -6
View File
@@ -362,13 +362,18 @@ richiede autenticazione. Non appartiene al workspace ed è indipendente dalla co
richiesta LiteLLM per conto del backend. Non è un servizio HTTP, non possiede il lifecycle della
Description Generation Run e non è una CLI esposta agli utenti.
**Catalog Sample** — Un input transitorio composto da un massimo di cinque righe reali e dai
valori di esempio bounded letti da una Catalog Table per la generazione delle descrizioni. È
trattato come dato non fidato, non viene persistito e non diventa Catalog Metadata.
**Catalog Sample** — Un input transitorio composto da un massimo di cinque righe e da valori di
esempio bounded di una Catalog Table per la generazione delle descrizioni. Può contenere valori
reali oppure sintetici in base al Sensitive Data Flag della Catalog Column; non viene persistito
e non diventa Catalog Metadata.
**Sensitive Data Policy** — L'insieme di regole che classifica i valori sorgente protetti per
l'uso nei processi AI e ne prescrive l'esclusione o l'anonimizzazione. Non coincide con la sola
classificazione dei dati personali.
**Sensitive Data Flag** — La scelta binaria umana applicata a una Catalog Column: `true` protegge
i valori sorgente e `false` ne consente l'invio al modello. Il valore predefinito è `false`, anche
per le nuove colonne.
**Sensitive Data Policy** — La regola che applica il Sensitive Data Flag ai Catalog Sample:
valori sintetici per una colonna protetta, valori reali per una colonna non protetta. L'AI può
suggerire il flag dai soli metadati tecnici, ma soltanto l'utente lo imposta.
_Avoid_: PII filter, sample filter
**Introspection Capability** — Una categoria di struttura fisica che una Database Binding
PROJECT_STATE.md
+15 -6
View File
@@ -104,16 +104,18 @@ one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before loca
sessions still consume the existing workspace configuration in this slice: database-management
records do not yet change the NL→SQL handoff. The accepted design is recorded in
`docs/plans/2026-08-26-metadata-catalog-from-thothai.md`, the snapshot contract under
`docs/contracts/`, and ADRs 0001–0010.
`docs/contracts/`, and ADRs 0001–0011.
Semantic aliases, value descriptions, synonyms, concepts, and logical relationships remain
deferred to their dedicated slices.
AI Description Generation preserves ThothAI's use of real source samples: up to five source rows
and five representative non-null example values may be sent transiently to the configured model
provider. The UI and operator documentation disclose this behavior. A required follow-up
improvement is a Sensitive Data Policy that classifies protected fields and excludes or anonymizes
their values before model calls.
AI Description Generation uses the catalog's human-owned Sensitive Data Flag. The flag defaults to
`false`, including for newly synchronized columns. An administrator may request an AI proposal based
only on structural metadata, but it remains an unsaved draft until the human reviews and saves it.
For unprotected columns, up to five source rows and five representative non-null values may be sent
transiently to the configured model provider. Protected columns are omitted from source reads and
replaced in the prompt by deterministic plausible values derived only from their metadata. Existing
descriptions are not regenerated when a flag changes.
The accepted AI-description design is recorded in
`docs/plans/2026-08-28-ai-catalog-description-generation.md`, with the formal specification in the
@@ -141,6 +143,13 @@ point the next required design gate is to compare the catalog snapshot with the
preprocessing/schema-linking contracts and plan the cutover; this follow-up must not be treated as
optional cleanup or silently omitted.
**Deferred follow-up — Sensitive Data Policy in schema-linking.** The policy is first delivered
and tested in catalog description generation. Its enforcement for core schema-linking remains
out of scope until the current tickets are closed and the owner has completed the acceptance test.
At that gate, resume the design: `tht` must receive a read-only projection of the current Sensitive
Data Flags and exclude values from columns marked sensitive from every LSH result before it is
given to Pi. Do not start this integration before the owner gives final approval after that test.
## Active deployment work and manual gates
### PSD server deployment program
README.md
+10 -6
View File
@@ -305,12 +305,16 @@ Configuration changes take effect after restart and do not use Pi settings or wo
`llm_policy`.
Before enabling Description Generation, approve the selected model provider for bounded source-data
disclosure. A request may send up to five real source rows and up to five representative distinct,
non-null example values for relevant columns. Samples are transient and are not stored in generation
runs, run logs, application logs, API responses, or catalog metadata; prompt and sample snapshots are
not retained. Automated Sensitive Data Policy filtering and anonymization are not currently provided.
A future Sensitive Data Policy is required to classify protected fields and exclude or anonymize
their values before model calls.
disclosure. Every catalog column has a **Sensitive** flag that defaults to `false`. Administrators can
request an AI proposal based only on structural metadata, then must review and save the resulting
checkboxes themselves. The proposal never reads column contents and is not persisted automatically.
For unprotected columns, a request may send up to five real source rows and five representative
distinct, non-null example values. Protected columns are omitted from source reads and replaced in the
prompt by deterministic plausible values derived only from column metadata. Samples are transient and
are not stored in generation runs, run logs, application logs, API responses, or catalog metadata;
prompt and sample snapshots are not retained. A flag change applies to later generations and does not
regenerate existing descriptions.
Description Generation is an interactive Database Management operation, not a user-facing CLI.
The installation runs at most one sequential generation at a time. The run drawer exposes safe
backend/src/app.ts
+7
View File
@@ -56,6 +56,7 @@ import { metadataGenerationModelRoutes } from "./routes/metadata-generation-mode
import { catalogDescriptionConsolidationRoutes } from "./routes/catalog-description-consolidation.js";
import { PythonModelCompleter, type ModelCompleter } from "./catalog/model-completer.js";
import { DescriptionGenerationWorker } from "./catalog/description-generation-worker.js";
import { SensitiveDataSuggester } from "./catalog/sensitive-data-suggester.js";
import {
PostgresDescriptionSourceSampler,
type DescriptionSourceSampler,
@@ -179,6 +180,11 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
catalogOperationCoordinator,
descriptionSourceSampler,
);
const sensitiveDataSuggester = new SensitiveDataSuggester(
catalogRepository,
metadataGenerationModels,
modelCompleter,
);
const catalogService = deps?.catalogService ?? new CatalogService(
catalogRepository,
workspaceRegistry,
@@ -438,6 +444,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
catalogDescriptionGenerationRoutes(app, {
repository: catalogRepository,
worker: descriptionGenerationWorker,
sensitiveDataSuggester,
});
settingsRoutes(app, { cfg: config, listModels, getSettings });
piManagementRoutes(app, { service: piManagement });
backend/src/catalog/description-generation-worker.ts
+49 -23
View File
@@ -12,6 +12,7 @@ import type {
DescriptionSourceSampleValue,
DescriptionTargetSourceSample,
} from "./description-source-sampler.js";
import { syntheticSampleValue } from "./synthetic-sample-value.js";
import {
DescriptionGenerationRunActiveError,
type CatalogColumn,
@@ -306,35 +307,55 @@ function sourceSampleFor(
): PromptSourceSample | undefined {
const targetId = target.kind === "column" ? target.column.id : target.table.id;
const sample = samples.find((candidate) => candidate.targetId === targetId);
if (!sample) return undefined;
const relevantColumns = new Set(
target.kind === "column" ? [target.column.name] : target.columns.map((column) => column.name),
);
const rows = sample.rows.slice(0, budget.rows).map((row) => {
const columns = target.kind === "column" ? [target.column] : target.columns;
const sourceRows = sample?.rows ?? [];
const hasSensitiveColumns = columns.some((column) => column.sensitive);
const hasNonSensitiveColumns = columns.some((column) => !column.sensitive);
const realRowCount = hasNonSensitiveColumns
? Math.min(sourceRows.length, budget.rows)
: 0;
const rowCount = hasSensitiveColumns ? MAX_SAMPLE_ROWS_PER_REQUEST : realRowCount;
const rows = Array.from({ length: rowCount }, (_, rowIndex) => {
const row = rowIndex < realRowCount ? sourceRows[rowIndex] : undefined;
const sourceFields = new Map((row?.fields ?? []).map((field) => [field.name, field.value]));
const fields: Array<{ name: string; value: PromptSampleValue }> = [];
const seen = new Set<string>();
for (const field of row.fields) {
if (!relevantColumns.has(field.name) || seen.has(field.name)) continue;
seen.add(field.name);
for (const column of columns) {
const value = column.sensitive
? syntheticSampleValue(column, rowIndex + 1)
: sourceFields.get(column.name);
if (value === undefined) continue;
fields.push({
name: boundedJsonText(field.name, MAX_IDENTIFIER_JSON_BYTES),
value: promptSampleValue(field.value),
name: boundedJsonText(column.name, MAX_IDENTIFIER_JSON_BYTES),
value: promptSampleValue(value),
});
if (fields.length === MAX_SAMPLE_FIELDS_PER_ROW) break;
}
return { fields };
});
budget.rows -= rows.length;
budget.rows -= realRowCount;
const representativeValues: PromptSourceSample["representativeValues"] = [];
const seenColumns = new Set<string>();
let remainingRepresentativeValues = budget.representativeValues;
for (const examples of sample.representativeValues) {
if (remainingRepresentativeValues === 0) break;
if (!relevantColumns.has(examples.column) || seenColumns.has(examples.column)) continue;
seenColumns.add(examples.column);
let remainingRealRepresentativeValues = budget.representativeValues;
let remainingSyntheticRepresentativeValues = MAX_REPRESENTATIVE_VALUES_PER_REQUEST;
for (const column of columns) {
if (seenColumns.has(column.name)) continue;
const remainingRepresentativeValues = column.sensitive
? remainingSyntheticRepresentativeValues
: remainingRealRepresentativeValues;
if (remainingRepresentativeValues === 0) continue;
const sourceExamples = sample?.representativeValues.find(
(examples) => examples.column === column.name,
);
const exampleValues = column.sensitive
? Array.from(
{ length: Math.min(Math.max(rowCount, 1), remainingRepresentativeValues) },
(_, index) => syntheticSampleValue(column, index + 1),
)
: sourceExamples?.values ?? [];
seenColumns.add(column.name);
const values: Array<Exclude<PromptSampleValue, null>> = [];
const seenValues = new Set<string>();
for (const value of examples.values) {
for (const value of exampleValues) {
const normalized = promptSampleValue(value);
if (normalized === null) continue;
const key = JSON.stringify([typeof normalized, normalized]);
@@ -345,11 +366,15 @@ function sourceSampleFor(
}
if (values.length > 0) {
representativeValues.push({
column: boundedJsonText(examples.column, MAX_IDENTIFIER_JSON_BYTES),
column: boundedJsonText(column.name, MAX_IDENTIFIER_JSON_BYTES),
values,
});
remainingRepresentativeValues -= values.length;
budget.representativeValues -= values.length;
if (column.sensitive) {
remainingSyntheticRepresentativeValues -= values.length;
} else {
remainingRealRepresentativeValues -= values.length;
budget.representativeValues -= values.length;
}
}
if (representativeValues.length === MAX_SAMPLE_COLUMNS) break;
}
@@ -933,8 +958,9 @@ export class DescriptionGenerationWorker {
targetId: target.kind === "column" ? target.column.id : target.table.id,
tableName: target.table.name,
columnNames: target.kind === "column"
? [target.column.name]
: target.columns.map((column) => column.name),
? target.column.sensitive ? [] : [target.column.name]
: target.columns.filter((column) => !column.sensitive)
.map((column) => column.name),
})),
signal,
);
backend/src/catalog/memory-repository.ts
+10 -1
View File
@@ -202,10 +202,18 @@ export class MemoryCatalogRepository implements CatalogRepository {
expectedVersion: number,
description: string | null,
generatedDescription: string | null,
sensitive?: boolean,
): Promise<CatalogColumn | undefined> {
const current = await this.getColumn(databaseId, tableId, columnId);
if (!current || current.version !== expectedVersion) return undefined;
const updated = { ...current, description, generatedDescription, version: current.version + 1, updatedAt: new Date().toISOString() };
const updated = {
...current,
description,
generatedDescription,
sensitive: sensitive ?? current.sensitive,
version: current.version + 1,
updatedAt: new Date().toISOString(),
};
this.columns.set(columnId, updated);
return structuredClone(updated);
}
@@ -621,6 +629,7 @@ export class MemoryCatalogRepository implements CatalogRepository {
sourceComment: observed.sourceComment,
description: null,
generatedDescription: null,
sensitive: false,
lastSyncedDatabaseVersion: expectedDatabaseVersion,
lastSyncedAt: now,
version: 1,
backend/src/catalog/migrate.ts
+2
View File
@@ -8,6 +8,7 @@ import * as catalogTablesMigration from "./migrations/002_catalog_tables.js";
import * as catalogSchemaSyncMigration from "./migrations/003_catalog_schema_sync.js";
import * as catalogRuntimeSequencePrivilegesMigration from "./migrations/004_catalog_runtime_sequence_privileges.js";
import * as descriptionGenerationRunsMigration from "./migrations/005_description_generation_runs.js";
import * as sensitiveDataFlagMigration from "./migrations/006_sensitive_data_flag.js";
const connectionString = process.env.THT_CATALOG_MIGRATOR_DATABASE_URL;
const host = process.env.THT_CATALOG_DB_HOST;
@@ -38,6 +39,7 @@ const provider: MigrationProvider = {
"003_catalog_schema_sync": catalogSchemaSyncMigration,
"004_catalog_runtime_sequence_privileges": catalogRuntimeSequencePrivilegesMigration,
"005_description_generation_runs": descriptionGenerationRunsMigration,
"006_sensitive_data_flag": sensitiveDataFlagMigration,
};
},
};
backend/src/catalog/migrations/006_sensitive_data_flag.ts
+12
View File
@@ -0,0 +1,12 @@
import type { Kysely } from "kysely";
import type { CatalogDatabase } from "../repository.js";
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.alterTable("catalog_columns")
.addColumn("sensitive", "boolean", (column) => column.notNull().defaultTo(false))
.execute();
}
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.alterTable("catalog_columns").dropColumn("sensitive").execute();
}
backend/src/catalog/repository.ts
+4
View File
@@ -108,6 +108,7 @@ interface CatalogColumnTable {
sourceComment: string | null;
description: string | null;
generatedDescription: string | null;
sensitive: Generated<boolean>;
lastSyncedDatabaseVersion: number | null;
lastSyncedAt: Timestamp | null;
version: Generated<number>;
@@ -290,6 +291,7 @@ function serializeColumn(row: Selectable<CatalogColumnTable>, foreignKeyCount =
sourceComment: row.sourceComment,
description: row.description,
generatedDescription: row.generatedDescription,
sensitive: row.sensitive,
lastSyncedDatabaseVersion: row.lastSyncedDatabaseVersion,
lastSyncedAt: row.lastSyncedAt === null ? null : new Date(row.lastSyncedAt).toISOString(),
version: row.version,
@@ -561,6 +563,7 @@ export class KyselyCatalogRepository implements CatalogRepository {
expectedVersion: number,
description: string | null,
generatedDescription: string | null,
sensitive?: boolean,
): Promise<CatalogColumn | undefined> {
const belongs = await this.db.selectFrom("catalogTables").select("id")
.where("id", "=", tableId).where("databaseId", "=", databaseId).executeTakeFirst();
@@ -568,6 +571,7 @@ export class KyselyCatalogRepository implements CatalogRepository {
const row = await this.db.updateTable("catalogColumns").set({
description,
generatedDescription,
...(sensitive === undefined ? {} : { sensitive }),
version: sql`version + 1`,
updatedAt: sql`now()`,
}).where("id", "=", columnId).where("tableId", "=", tableId)
backend/src/catalog/sensitive-data-suggester.ts
+103
View File
@@ -0,0 +1,103 @@
import { z } from "zod";
import type { MetadataGenerationModels } from "./metadata-generation-models.js";
import type { ModelCompleter } from "./model-completer.js";
import type { CatalogRepository } from "./types.js";
const MAX_COLUMNS = 10_000;
const responseSchema = z.object({
suggestions: z.array(z.object({
columnId: z.uuid(),
sensitive: z.boolean(),
}).strict()).max(MAX_COLUMNS),
}).strict();
export interface SensitiveDataSuggestion {
columnId: string;
sensitive: boolean;
}
export class SensitiveDataSuggestionTargetNotFoundError extends Error {
constructor() {
super("database not found");
this.name = "SensitiveDataSuggestionTargetNotFoundError";
}
}
export class SensitiveDataSuggestionInvalidResponseError extends Error {
constructor() {
super("sensitive-data suggestion response is invalid");
this.name = "SensitiveDataSuggestionInvalidResponseError";
}
}
export class SensitiveDataSuggester {
constructor(
private readonly repository: CatalogRepository,
private readonly models: MetadataGenerationModels,
private readonly completer: ModelCompleter,
) {}
async suggest(
databaseId: string,
modelId: string,
signal: AbortSignal,
): Promise<readonly SensitiveDataSuggestion[]> {
const database = await this.repository.get(databaseId);
if (!database) throw new SensitiveDataSuggestionTargetNotFoundError();
const tables = await this.repository.listTables(databaseId);
const columns = (await Promise.all(tables.map(async (table) => ({
table,
columns: await this.repository.listColumns(databaseId, table.id),
})))).flatMap(({ table, columns: tableColumns }) => tableColumns.map((column) => ({
columnId: column.id,
table: table.name,
column: column.name,
dataType: column.dataType,
nullable: column.isNullable,
primaryKey: column.isPrimaryKey,
foreignKey: column.isForeignKey,
})));
if (columns.length === 0) return [];
if (columns.length > MAX_COLUMNS) throw new SensitiveDataSuggestionInvalidResponseError();
const content = await this.completer.complete({
model: this.models.resolve(modelId),
signal,
messages: [
{
role: "system",
content: [
"Classify whether each database column is likely to contain sensitive source values.",
"Use only the supplied structural metadata. Return strict JSON with this exact shape:",
'{"suggestions":[{"columnId":"uuid","sensitive":true}]}',
"Return every supplied column exactly once. Do not add explanations or markdown.",
].join("\n"),
},
{
role: "user",
content: JSON.stringify({
database: database.databaseName,
schema: database.schema,
columns,
}),
},
],
});
try {
const parsed = responseSchema.parse(JSON.parse(content));
const expected = new Set(columns.map((column) => column.columnId));
const received = new Set(parsed.suggestions.map((suggestion) => suggestion.columnId));
if (received.size !== parsed.suggestions.length
|| received.size !== expected.size
|| [...received].some((columnId) => !expected.has(columnId))) {
throw new SensitiveDataSuggestionInvalidResponseError();
}
return parsed.suggestions;
} catch (error) {
if (error instanceof SensitiveDataSuggestionInvalidResponseError) throw error;
throw new SensitiveDataSuggestionInvalidResponseError();
}
}
}
backend/src/catalog/synthetic-sample-value.ts
+67
View File
@@ -0,0 +1,67 @@
import type { DescriptionSourceSampleValue } from "./description-source-sampler.js";
const FIRST_NAMES = ["marta", "luca", "elena", "paolo", "giulia"] as const;
const LAST_NAMES = ["rossi", "bianchi", "conti", "romano", "ferrari"] as const;
const NUMERIC_TYPE = /(int|numeric|decimal|real|double|float|money)/;
const BOOLEAN_TYPE = /(bool)/;
function nameAt(index: number): string {
const offset = Math.max(0, index - 1);
return `${FIRST_NAMES[offset % FIRST_NAMES.length]} ${LAST_NAMES[offset % LAST_NAMES.length]}`;
}
export function syntheticSampleValue(
column: { name: string; dataType: string },
index: number,
): DescriptionSourceSampleValue {
const ordinal = Math.max(1, index);
const name = column.name.toLocaleLowerCase("en-US");
const type = column.dataType.toLocaleLowerCase("en-US");
const person = nameAt(ordinal).split(" ");
const safeName = name.replace(/[^a-z0-9]+/g, "_").replace(/^_+|_+$/g, "") || "value";
if (type.endsWith("[]") || type.startsWith("_") || /\barray\b/.test(type)) {
if (NUMERIC_TYPE.test(type)) {
return `{${1000 + ordinal},${1001 + ordinal}}`;
}
if (BOOLEAN_TYPE.test(type)) return `{${ordinal % 2 === 1},${ordinal % 2 !== 1}}`;
return `{${safeName}_${String(ordinal).padStart(3, "0")},${safeName}_${String(ordinal + 1).padStart(3, "0")}}`;
}
if (/^jsonb?$/.test(type)) {
return JSON.stringify({ example: `${safeName}_${String(ordinal).padStart(3, "0")}`, sequence: ordinal });
}
if (/(uuid|uniqueidentifier)/.test(type)) {
return `00000000-0000-4000-8000-${String(ordinal).padStart(12, "0")}`;
}
if (/\bcidr\b/.test(type)) return "192.0.2.0/24";
if (/\binet\b/.test(type)) return `192.0.2.${((ordinal - 1) % 254) + 1}`;
if (/(timestamp|datetime)/.test(type)) {
return `2024-01-${String(Math.min(ordinal, 28)).padStart(2, "0")}T10:30:00.000Z`;
}
if (/\bdate\b/.test(type)) {
return `198${ordinal % 10}-01-${String(Math.min(ordinal, 28)).padStart(2, "0")}`;
}
if (/\btime\b/.test(type)) {
return `10:30:${String(ordinal % 60).padStart(2, "0")}`;
}
if (/\binterval\b/.test(type)) {
return `${ordinal} days ${String(ordinal % 24).padStart(2, "0")}:00:00`;
}
if (/\bbytea\b/.test(type)) return `\\x${ordinal.toString(16).padStart(8, "0")}`;
if (BOOLEAN_TYPE.test(type)) return ordinal % 2 === 1;
if (NUMERIC_TYPE.test(type)) return 1000 + ordinal;
if (/e[-_]?mail/.test(name)) {
return `${person[0]}.${person[1]}@example.com`;
}
if (/(phone|mobile|cell|telefono|telefono_mobile|tel_)/.test(name)) {
return `+39 02 5550 ${String(1000 + ordinal).padStart(4, "0")}`;
}
if (/(first_?name|given_?name|nome)/.test(name)) return person[0]!;
if (/(last_?name|family_?name|surname|cognome)/.test(name)) return person[1]!;
if (/(full_?name|patient_?name|person_?name)/.test(name)) return nameAt(ordinal);
if (/(birth|dob|data_nascita)/.test(name)) {
return `198${ordinal % 10}-01-${String(Math.min(ordinal, 28)).padStart(2, "0")}`;
}
return `${safeName}_${String(ordinal).padStart(3, "0")}`;
}
backend/src/catalog/types.ts
+2
View File
@@ -88,6 +88,7 @@ export interface CatalogColumn {
sourceComment: string | null;
description: string | null;
generatedDescription: string | null;
sensitive: boolean;
lastSyncedDatabaseVersion: number | null;
lastSyncedAt: string | null;
version: number;
@@ -349,6 +350,7 @@ export interface CatalogRepository {
expectedVersion: number,
description: string | null,
generatedDescription: string | null,
sensitive?: boolean,
): Promise<CatalogColumn | undefined>;
consolidateGeneratedDescriptions(
databaseId: string,
backend/src/routes/catalog-description-generation.ts
+41 -1
View File
@@ -11,6 +11,12 @@ import {
type DescriptionGenerationWorker,
} from "../catalog/description-generation-worker.js";
import { MetadataGenerationModelUnavailableError } from "../catalog/metadata-generation-models.js";
import { ModelCompletionProviderError } from "../catalog/model-completer.js";
import {
SensitiveDataSuggester,
SensitiveDataSuggestionInvalidResponseError,
SensitiveDataSuggestionTargetNotFoundError,
} from "../catalog/sensitive-data-suggester.js";
import {
CatalogOperationInProgressError,
CatalogUnavailableError,
@@ -22,6 +28,7 @@ import {
const idSchema = z.uuid();
const modelIdSchema = z.string().regex(/^[a-z][a-z0-9._-]{0,63}$/);
const suggestionSchema = z.object({ modelId: modelIdSchema }).strict();
const selectedTargetIdsSchema = z.array(idSchema).min(1);
const startSchema = z.discriminatedUnion("scope", [
z.object({
@@ -116,6 +123,19 @@ function safeError(reply: FastifyReply, error: unknown) {
message: "The selected metadata-generation model is unavailable.",
});
}
if (error instanceof SensitiveDataSuggestionTargetNotFoundError) {
return reply.code(404).send({
code: "database_not_found",
message: "Database configuration was not found.",
});
}
if (error instanceof SensitiveDataSuggestionInvalidResponseError
|| error instanceof ModelCompletionProviderError) {
return reply.code(502).send({
code: "sensitive_data_suggestion_failed",
message: "Sensitive-data suggestions could not be prepared.",
});
}
if (error instanceof DescriptionGenerationDuplicateTargetIdsError) {
return reply.code(400).send({
code: "description_generation_target_ids_duplicate",
@@ -172,8 +192,28 @@ function safeError(reply: FastifyReply, error: unknown) {
export function catalogDescriptionGenerationRoutes(
app: FastifyInstance,
deps: { repository: CatalogRepository; worker: DescriptionGenerationWorker },
deps: {
repository: CatalogRepository;
worker: DescriptionGenerationWorker;
sensitiveDataSuggester: SensitiveDataSuggester;
},
): void {
app.post("/catalog/databases/:databaseId/sensitive-data-suggestions", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
const input = suggestionSchema.parse(request.body);
const suggestions = await deps.sensitiveDataSuggester.suggest(
databaseId,
input.modelId,
new AbortController().signal,
);
return { suggestions };
} catch (error) {
return safeError(reply, error);
}
});
app.post("/catalog/databases/:databaseId/description-generation-runs", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
backend/src/routes/catalog-schema.ts
+2
View File
@@ -18,6 +18,7 @@ const metadataSchema = z.object({
version: z.number().int().positive(),
description: z.string().max(20_000).nullable(),
generatedDescription: z.string().max(20_000).nullable(),
sensitive: z.boolean().optional(),
}).strict();
const createRunSchema = z.object({
version: z.number().int().positive(),
@@ -117,6 +118,7 @@ export function catalogSchemaRoutes(
input.version,
normalized(input.description),
normalized(input.generatedDescription),
input.sensitive,
);
if (!updated) return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
return updated;
backend/test/catalog-description-generation-routes.test.ts
+173
View File
@@ -141,6 +141,83 @@ async function waitForTerminalRun(app: ReturnType<typeof buildApp>, runId: strin
throw new Error(`Description Generation Run ${runId} did not finish`);
}
test("suggests sensitive flags from structural metadata without persisting them", async () => {
const modelCompleter = {
complete: vi.fn(async () => JSON.stringify({
suggestions: [{ columnId: expect.any(String), sensitive: true }],
})),
};
const { app, repository, database, column } = await setup(modelCompleter);
modelCompleter.complete.mockResolvedValueOnce(JSON.stringify({
suggestions: [{ columnId: column.id, sensitive: true }],
}));
try {
const response = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/sensitive-data-suggestions`,
payload: { modelId: configuredModel.id },
});
expect(response.statusCode).toBe(200);
expect(response.json()).toEqual({
suggestions: [{ columnId: column.id, sensitive: true }],
});
expect(await repository.getColumn(database.id, column.tableId, column.id))
.toMatchObject({ sensitive: false });
const request = modelCompleter.complete.mock.calls[0]![0] as ModelCompletionRequest;
const prompt = request.messages.map((message) => message.content).join("\n");
expect(prompt).toContain("patients");
expect(prompt).toContain("birth_date");
expect(prompt).toContain("date");
expect(prompt).not.toContain("Patient date of birth");
expect(prompt).not.toContain("test-provider-secret");
} finally {
await app.close();
}
});
test.each(["malformed", "incomplete", "duplicate"] as const)(
"fails safely when sensitive-data suggestions are %s",
async (kind) => {
const modelCompleter: ModelCompleter = {
complete: vi.fn(async () => "unused"),
};
const { app, repository, database, column } = await setup(modelCompleter);
const rawResponse = kind === "malformed"
? "RAW_PROVIDER_RESPONSE_DO_NOT_EXPOSE_{"
: kind === "incomplete"
? JSON.stringify({ suggestions: [] })
: JSON.stringify({
suggestions: [
{ columnId: column.id, sensitive: true },
{ columnId: column.id, sensitive: true },
],
});
vi.mocked(modelCompleter.complete).mockResolvedValueOnce(rawResponse);
try {
const response = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/sensitive-data-suggestions`,
payload: { modelId: configuredModel.id },
});
expect(response.statusCode).toBe(502);
expect(response.json()).toEqual({
code: "sensitive_data_suggestion_failed",
message: "Sensitive-data suggestions could not be prepared.",
});
expect(response.body).not.toContain(rawResponse);
expect(await repository.getColumn(database.id, column.tableId, column.id))
.toMatchObject({ sensitive: false });
} finally {
await app.close();
}
},
);
interface SseFrame {
id?: string;
event?: string;
@@ -398,6 +475,102 @@ test("keeps real source samples transient across the Fastify API and application
}
});
test("never exposes a protected source value to the model, persistence, logs, or browser APIs", async () => {
let selectedColumnId = "";
const protectedValue = "PROTECTED_SOURCE_VALUE_8f4c2a";
const modelCompleter: ModelCompleter = {
complete: vi.fn(async () => JSON.stringify({
results: [{
targetId: selectedColumnId,
outcome: "generated",
description: "Data di nascita del paziente.",
}],
})),
};
const descriptionSourceSampler: DescriptionSourceSampler = {
sample: vi.fn(async (_database, targets) => [{
targetId: targets[0]!.targetId,
tableName: targets[0]!.tableName,
rows: [{ fields: [{ name: "birth_date", value: protectedValue }] }],
representativeValues: [{ column: "birth_date", values: [protectedValue] }],
}]),
};
const { app, repository, database, table, column } = await setup(
modelCompleter,
{},
"it",
descriptionSourceSampler,
);
selectedColumnId = column.id;
await repository.updateColumnMetadata(
database.id,
table.id,
column.id,
column.version,
column.description,
column.generatedDescription,
true,
);
const logSpies = [
vi.spyOn(app.log, "info"),
vi.spyOn(app.log, "warn"),
vi.spyOn(app.log, "error"),
];
try {
const start = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: {
modelId: configuredModel.id,
scope: "selected_columns",
targetIds: [column.id],
},
});
expect(start.statusCode).toBe(202);
await waitForTerminalRun(app, start.json().id);
expect(descriptionSourceSampler.sample).toHaveBeenCalledWith(
expect.anything(),
[expect.objectContaining({ targetId: column.id, columnNames: [] })],
expect.any(AbortSignal),
);
const completionRequest = vi.mocked(modelCompleter.complete).mock.calls[0]![0];
const providerPayload = JSON.stringify(completionRequest.messages);
expect(providerPayload).not.toContain(protectedValue);
expect(providerPayload).toContain("1981-01-01");
const apiResponses = await Promise.all([
app.inject({ method: "GET", url: `/catalog/description-generation-runs/${start.json().id}` }),
app.inject({
method: "GET",
url: `/catalog/description-generation-runs/${start.json().id}/events-list`,
}),
app.inject({ method: "GET", url: `/catalog/databases/${database.id}` }),
app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables` }),
app.inject({
method: "GET",
url: `/catalog/databases/${database.id}/tables/${table.id}/columns`,
}),
]);
expect(apiResponses.every((response) => response.statusCode === 200)).toBe(true);
expect(apiResponses.map((response) => response.body).join("\n")).not.toContain(protectedValue);
const persisted = JSON.stringify({
run: await repository.getDescriptionGenerationRun(start.json().id),
events: await repository.listDescriptionGenerationEvents(start.json().id),
database: await repository.get(database.id),
table: await repository.getTable(database.id, table.id),
column: await repository.getColumn(database.id, table.id, column.id),
});
expect(persisted).not.toContain(protectedValue);
expect(JSON.stringify(logSpies.flatMap((spy) => spy.mock.calls))).not.toContain(protectedValue);
} finally {
for (const spy of logSpies) spy.mockRestore();
await app.close();
}
});
test("wires the production sampler to the same injected CatalogPostgresAccess instance", async () => {
let selectedColumnId = "";
const sampleSecret = "PRODUCTION_WIRING_SAMPLE_f2986a";
backend/test/catalog-description-generation-worker.test.ts
+201 -14
View File
@@ -222,7 +222,7 @@ test("adds only bounded transient source samples to the model request", async ()
tables: [{ name: "patients", sourceComment: null }],
columns: [{
tableName: "patients",
name: "status",
name: "patient_email",
ordinalPosition: 1,
dataType: "text",
isNullable: true,
@@ -243,9 +243,18 @@ test("adds only bounded transient source samples to the model request", async ()
});
const table = (await repository.listTables(database.id))[0]!;
const columns = await repository.listColumns(database.id, table.id);
const column = columns.find((candidate) => candidate.name === "status")!;
const column = columns.find((candidate) => candidate.name === "patient_email")!;
const ward = columns.find((candidate) => candidate.name === "ward")!;
const sampleSecret = "ONLY_IN_TRANSIENT_SAMPLE_7f29c8";
await repository.updateColumnMetadata(
database.id,
table.id,
column.id,
column.version,
column.description,
column.generatedDescription,
true,
);
const sampleSecret = "real.patient@hospital.invalid";
const sourceSampler: DescriptionSourceSampler = {
sample: vi.fn(async () => [{
targetId: column.id,
@@ -265,11 +274,14 @@ test("adds only bounded transient source samples to the model request", async ()
rows: [
{ fields: [{ name: ward.name, value: "row-4" }] },
{ fields: [{ name: ward.name, value: "row-5" }] },
{ fields: [{ name: ward.name, value: "row-6-must-be-omitted" }] },
{ fields: [{ name: ward.name, value: "row-6" }] },
{ fields: [{ name: ward.name, value: "row-7" }] },
{ fields: [{ name: ward.name, value: "row-8" }] },
{ fields: [{ name: ward.name, value: "row-9-must-be-omitted" }] },
],
representativeValues: [{
column: ward.name,
values: ["ward-1", "ward-2", "ward-3-must-be-omitted"],
values: ["ward-1", "ward-2", "ward-3", "ward-4", "ward-5", "ward-6-must-be-omitted"],
}],
}]),
};
@@ -321,7 +333,7 @@ test("adds only bounded transient source samples to the model request", async ()
expect(sourceSampler.sample).toHaveBeenCalledWith(
expect.objectContaining({ id: database.id, binding: database.binding }),
[
{ targetId: column.id, tableName: table.name, columnNames: [column.name] },
{ targetId: column.id, tableName: table.name, columnNames: [] },
{ targetId: ward.id, tableName: table.name, columnNames: [ward.name] },
],
expect.any(AbortSignal),
@@ -338,21 +350,29 @@ test("adds only bounded transient source samples to the model request", async ()
targetContext.sourceSample?.representativeValues.flatMap((entry) => entry.values) ?? []
),
);
expect(sampledRows).toHaveLength(5);
expect(representativeValues).toHaveLength(5);
expect(context.targets[0].sourceSample.rows).toHaveLength(3);
expect(context.targets[1].sourceSample.rows).toHaveLength(2);
expect(sampledRows).toHaveLength(10);
expect(representativeValues).toHaveLength(10);
expect(context.targets[0].sourceSample.rows).toHaveLength(5);
expect(context.targets[1].sourceSample.rows).toHaveLength(5);
expect(context.targets[0].sourceSample.representativeValues).toEqual([{
column: column.name,
values: [sampleSecret, "two", "three"],
values: [
"marta.rossi@example.com",
"luca.bianchi@example.com",
"elena.conti@example.com",
"paolo.romano@example.com",
"giulia.ferrari@example.com",
],
}]);
expect(context.targets[1].sourceSample.representativeValues).toEqual([{
column: ward.name,
values: ["ward-1", "ward-2"],
values: ["ward-1", "ward-2", "ward-3", "ward-4", "ward-5"],
}]);
expect(userMessage).toContain(sampleSecret);
expect(userMessage).not.toContain(sampleSecret);
expect(userMessage).not.toMatch(/synthetic|fake|fittizi/i);
expect(userMessage).toContain("marta.rossi@example.com");
expect(userMessage).not.toMatch(
/row-6-must-be-omitted|ward-3-must-be-omitted/,
/row-9-must-be-omitted|ward-6-must-be-omitted/,
);
const persisted = JSON.stringify({
@@ -366,6 +386,173 @@ test("adds only bounded transient source samples to the model request", async ()
expect(persisted).not.toContain(sampleSecret);
});
test("gives every sensitive column synthetic context without consuming the real sample budget", async () => {
const repository = new MemoryCatalogRepository();
const database = await repository.create({
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
});
await repository.applySchemaSync(database.id, database.version, "all", [], {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: [{ name: "patients", sourceComment: null }],
columns: [{
tableName: "patients",
name: "patient_email",
ordinalPosition: 1,
dataType: "text",
isNullable: true,
defaultExpression: null,
primaryKeyPosition: null,
sourceComment: null,
}, {
tableName: "patients",
name: "patient_phone",
ordinalPosition: 2,
dataType: "text",
isNullable: true,
defaultExpression: null,
primaryKeyPosition: null,
sourceComment: null,
}, {
tableName: "patients",
name: "ward",
ordinalPosition: 3,
dataType: "text",
isNullable: true,
defaultExpression: null,
primaryKeyPosition: null,
sourceComment: null,
}],
relationships: [],
});
const table = (await repository.listTables(database.id))[0]!;
const columns = await repository.listColumns(database.id, table.id);
const email = columns.find((column) => column.name === "patient_email")!;
const phone = columns.find((column) => column.name === "patient_phone")!;
const ward = columns.find((column) => column.name === "ward")!;
for (const column of [email, phone]) {
await repository.updateColumnMetadata(
database.id,
table.id,
column.id,
column.version,
column.description,
column.generatedDescription,
true,
);
}
const wardValues = ["ward-a", "ward-b", "ward-c", "ward-d", "ward-e"];
const sourceSampler: DescriptionSourceSampler = {
sample: vi.fn(async (_database, targets) => targets.map((target) => {
if (target.columnNames.length === 0) {
return {
targetId: target.targetId,
tableName: target.tableName,
rows: [],
representativeValues: [],
};
}
const columnName = target.columnNames[0]!;
return {
targetId: target.targetId,
tableName: target.tableName,
rows: wardValues.map((value) => ({ fields: [{ name: columnName, value }] })),
representativeValues: [{ column: columnName, values: wardValues }],
};
})),
};
const completer: ModelCompleter = {
complete: vi.fn(async (request) => {
const context = JSON.parse(request.messages[1]!.content.split("\n").slice(1).join("\n"));
return JSON.stringify({
results: context.targets.map((target: { targetId: string }) => ({
targetId: target.targetId,
outcome: "generated",
description: "Descrizione generata.",
})),
});
}),
};
const models: MetadataGenerationModels = {
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
resolve: () => ({
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "test-provider-secret",
}),
};
const worker = new DescriptionGenerationWorker(
repository,
{
read: vi.fn(async () => ({
workspace: { workspace: { language: "it" } },
revision: {},
})),
} as unknown as WorkspaceRegistry,
models,
completer,
new CatalogOperationCoordinator(),
sourceSampler,
);
const run = await worker.start(
database.id,
"openai-mini",
"selected_columns",
[email.id, phone.id, ward.id],
);
await worker.waitForRun(run.id);
expect(sourceSampler.sample).toHaveBeenCalledWith(
expect.objectContaining({ id: database.id }),
[
{ targetId: email.id, tableName: table.name, columnNames: [] },
{ targetId: phone.id, tableName: table.name, columnNames: [] },
{ targetId: ward.id, tableName: table.name, columnNames: [ward.name] },
],
expect.any(AbortSignal),
);
const request = vi.mocked(completer.complete).mock.calls[0]![0] as ModelCompletionRequest;
const context = JSON.parse(request.messages[1]!.content.split("\n").slice(1).join("\n"));
const targets = new Map(
context.targets.map((target: { targetId: string }) => [target.targetId, target]),
);
expect(targets.get(email.id)).toMatchObject({
sourceSample: {
rows: expect.arrayContaining([
{ fields: [{ name: email.name, value: "marta.rossi@example.com" }] },
]),
representativeValues: [{
column: email.name,
values: expect.arrayContaining(["marta.rossi@example.com"]),
}],
},
});
expect(targets.get(phone.id)).toMatchObject({
sourceSample: {
rows: expect.arrayContaining([
{ fields: [{ name: phone.name, value: "+39 02 5550 1001" }] },
]),
representativeValues: [{
column: phone.name,
values: expect.arrayContaining(["+39 02 5550 1001"]),
}],
},
});
expect(targets.get(ward.id)).toMatchObject({
sourceSample: {
rows: wardValues.map((value) => ({ fields: [{ name: ward.name, value }] })),
representativeValues: [{ column: ward.name, values: wardValues }],
},
});
});
test("continues metadata-only with one safe warning when source sampling is unavailable", async () => {
const repository = new MemoryCatalogRepository();
const database = await repository.create({
backend/test/catalog-description-generation.integration.test.ts
+2
View File
@@ -11,6 +11,7 @@ import { up as upDatabases } from "../src/catalog/migrations/001_workspace_datab
import { up as upTables } from "../src/catalog/migrations/002_catalog_tables.js";
import { up as upSchemaSync } from "../src/catalog/migrations/003_catalog_schema_sync.js";
import { up as upDescriptionGeneration } from "../src/catalog/migrations/005_description_generation_runs.js";
import { up as upSensitiveDataFlag } from "../src/catalog/migrations/006_sensitive_data_flag.js";
import { KyselyCatalogRepository, type CatalogDatabase } from "../src/catalog/repository.js";
import { loadConfig } from "../src/config.js";
import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
@@ -43,6 +44,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
await upDatabases(db);
await upTables(db);
await upSchemaSync(db);
await upSensitiveDataFlag(db);
await upDescriptionGeneration(db);
const repository = new KyselyCatalogRepository(db);
const database = await repository.create({
backend/test/catalog-description-source-sampler.test.ts
+28
View File
@@ -92,6 +92,34 @@ test("samples at most five source rows and five distinct non-null examples in a
expect(end).toHaveBeenCalledOnce();
});
test("does not issue a SELECT when a protected target has no source columns", async () => {
const query = vi.fn(async () => ({ rows: [] }));
const end = vi.fn(async () => undefined);
const access: CatalogPostgresAccess = {
connect: vi.fn(async () => ({ query, end }) as CatalogDatabaseClient),
};
const sampler = new PostgresDescriptionSourceSampler(access);
const samples = await sampler.sample(database, [{
targetId: target.targetId,
tableName: target.tableName,
columnNames: [],
}], new AbortController().signal);
expect(samples).toEqual([{
targetId: target.targetId,
tableName: target.tableName,
rows: [],
representativeValues: [],
}]);
expect(query.mock.calls).toEqual([
["BEGIN TRANSACTION READ ONLY", []],
["ROLLBACK", []],
]);
expect(query.mock.calls.some(([sql]) => String(sql).startsWith("SELECT"))).toBe(false);
expect(end).toHaveBeenCalledOnce();
});
test("rolls back and closes the source connection when sampling fails", async () => {
const query = vi.fn(async (sql: string) => {
if (sql.startsWith("SELECT")) throw new Error("distinctive-source-secret");
backend/test/catalog-repository.integration.test.ts
+43 -2
View File
@@ -10,6 +10,7 @@ import { up as upTables } from "../src/catalog/migrations/002_catalog_tables.js"
import { up as upSchemaSync } from "../src/catalog/migrations/003_catalog_schema_sync.js";
import { up as upRuntimeSequencePrivileges } from "../src/catalog/migrations/004_catalog_runtime_sequence_privileges.js";
import { up as upDescriptionGeneration } from "../src/catalog/migrations/005_description_generation_runs.js";
import { up as upSensitiveDataFlag } from "../src/catalog/migrations/006_sensitive_data_flag.js";
const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
@@ -23,6 +24,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per wo
await upDatabases(db);
await upTables(db);
await upSchemaSync(db);
await upSensitiveDataFlag(db);
await sql`CREATE ROLE thothii_catalog_runtime`.execute(db);
await upRuntimeSequencePrivileges(db);
const sequencePrivilege = await sql<{ allowed: boolean }>`
@@ -75,11 +77,47 @@ test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per wo
};
expect(await repository.applySchemaSync(created.id, 1, "columns", [], fullColumnsSnapshot))
.toMatchObject({ created: 4, deleted: 0 });
expect((await repository.listColumns(created.id, patients.id)).map((column) => column.name))
.toEqual(["id", "name"]);
expect((await repository.listColumns(created.id, patients.id)).map((column) => ({
name: column.name,
sensitive: column.sensitive,
}))).toEqual([
{ name: "id", sensitive: false },
{ name: "name", sensitive: false },
]);
expect((await repository.listColumns(created.id, visits.id)).map((column) => column.name))
.toEqual(["id", "patient_id"]);
const patientName = (await repository.listColumns(created.id, patients.id))
.find((column) => column.name === "name")!;
expect(await repository.updateColumnMetadata(
created.id,
patients.id,
patientName.id,
patientName.version,
patientName.description,
patientName.generatedDescription,
true,
)).toMatchObject({ sensitive: true });
const refreshedColumnsSnapshot: ObservedSchemaSnapshot = {
...fullColumnsSnapshot,
schemaVersion: 2,
columns: fullColumnsSnapshot.columns.map((column) => column.tableName === "patients"
&& column.name === "name"
? { ...column, sourceComment: "Sensitive patient name" }
: column),
};
expect(await repository.applySchemaSync(
created.id,
1,
"columns",
[patients.id],
refreshedColumnsSnapshot,
)).toMatchObject({ updated: 1 });
expect(await repository.getColumn(created.id, patients.id, patientName.id)).toMatchObject({
sensitive: true,
sourceComment: "Sensitive patient name",
});
const reducedColumnsSnapshot: ObservedSchemaSnapshot = {
...fullColumnsSnapshot,
columns: fullColumnsSnapshot.columns.filter((column) => column.name === "id"),
@@ -135,6 +173,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository performs scoped metadata cl
await upDatabases(db);
await upTables(db);
await upSchemaSync(db);
await upSensitiveDataFlag(db);
const repository = new KyselyCatalogRepository(db);
const database = await repository.create({
workspaceId: "cleanup-test",
@@ -212,6 +251,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository atomically consolidates sel
await upDatabases(db);
await upTables(db);
await upSchemaSync(db);
await upSensitiveDataFlag(db);
const repository = new KyselyCatalogRepository(db);
const database = await repository.create({
workspaceId: "consolidation-test",
@@ -310,6 +350,7 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists globally exclusive
await upDatabases(db);
await upTables(db);
await upSchemaSync(db);
await upSensitiveDataFlag(db);
await upDescriptionGeneration(db);
const repository = new KyselyCatalogRepository(db);
const firstDatabase = await repository.create({
backend/test/catalog-schema-routes.test.ts
+17 -3
View File
@@ -234,16 +234,30 @@ test("keeps generated descriptions editable and preserves them across synchroniz
});
expect(editedTable.json()).toMatchObject({ description: null, generatedDescription: "Generated table draft" });
const idColumn = (await repository.listColumns(database.id, patients.id))[0];
expect(idColumn.sensitive).toBe(false);
const editedColumn = await app.inject({
method: "PATCH", url: `/catalog/databases/${database.id}/tables/${patients.id}/columns/${idColumn.id}`,
payload: { version: idColumn.version, description: "Reviewed key", generatedDescription: "Generated key draft" },
payload: {
version: idColumn.version,
description: "Reviewed key",
generatedDescription: "Generated key draft",
sensitive: true,
},
});
expect(editedColumn.json()).toMatchObject({
description: "Reviewed key",
generatedDescription: "Generated key draft",
sensitive: true,
});
expect(editedColumn.json()).toMatchObject({ description: "Reviewed key", generatedDescription: "Generated key draft" });
const second = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
await waitFor(repository, second.json().id, "succeeded");
expect(await repository.getTable(database.id, patients.id)).toMatchObject({ generatedDescription: "Generated table draft" });
expect(await repository.getColumn(database.id, patients.id, idColumn.id)).toMatchObject({ description: "Reviewed key", generatedDescription: "Generated key draft" });
expect(await repository.getColumn(database.id, patients.id, idColumn.id)).toMatchObject({
description: "Reviewed key",
generatedDescription: "Generated key draft",
sensitive: true,
});
});
test("consolidates non-empty generated table descriptions and reports skipped selections", async () => {
backend/test/catalog-synthetic-sample-value.test.ts
+29
View File
@@ -0,0 +1,29 @@
import { expect, test } from "vitest";
import { syntheticSampleValue } from "../src/catalog/synthetic-sample-value.js";
test.each([
["text[]", "tags", 1, "{tags_001,tags_002}"],
["json", "payload", 2, '{"example":"payload_002","sequence":2}'],
["jsonb", "attributes", 3, '{"example":"attributes_003","sequence":3}'],
["inet", "client_ip", 4, "192.0.2.4"],
["cidr", "network", 5, "192.0.2.0/24"],
["time without time zone", "opening_time", 6, "10:30:06"],
["interval", "duration", 7, "7 days 07:00:00"],
["bytea", "digest", 8, "\\x00000008"],
] as const)(
"creates a deterministic PostgreSQL-shaped value for %s",
(dataType, name, index, expected) => {
const column = { name, dataType };
expect(syntheticSampleValue(column, index)).toBe(expected);
expect(syntheticSampleValue(column, index)).toBe(expected);
},
);
test("uses the PostgreSQL type before birth-related name hints", () => {
expect(syntheticSampleValue({ name: "birth_year", dataType: "integer" }, 2)).toBe(1002);
expect(syntheticSampleValue({ name: "birth_date", dataType: "date" }, 2))
.toBe("1982-01-02");
expect(syntheticSampleValue({ name: "birth_recorded_at", dataType: "timestamp" }, 2))
.toBe("2024-01-02T10:30:00.000Z");
});
docs/adr/0011-gate-source-samples-with-a-sensitive-data-flag.md
+19
View File
@@ -0,0 +1,19 @@
---
status: accepted
---
# Gate source samples with a Sensitive Data Flag
Each Catalog Column has one human-set `sensitive` boolean, defaulting to `false`. An AI may prefill
draft suggestions from structural metadata only, but the user decides and only the boolean is
persisted; there is no rationale, history, audit ledger, fingerprint, review state, or retroactive
regeneration of existing descriptions.
Description generation extends ADR-0010 by sending bounded real values when `sensitive` is false
and deterministic plausible synthetic values when it is true, without identifying the synthetic
values to the model. The false default deliberately favors the expected stable schemas and the
minority of protected columns: a new column remains eligible for real sampling until a user marks
it sensitive.
Applying the same flag to LSH value grounding is deferred until the current catalog tickets and
owner acceptance test are complete; `PROJECT_STATE.md` records that required follow-up gate.
docs/operations/database-management.md
+17 -7
View File
@@ -48,9 +48,17 @@ the connection binding, or secrets. Deleting a table cascades to its columns and
Generated descriptions can be requested for selected tables, selected columns, every eligible
target, or targets with a missing generated description. The backend accepts one installation-wide
run and processes targets sequentially. It reads at most five source rows and five representative
non-null values per relevant source through a read-only connector, then sends that bounded sample
transiently to the configured model provider.
run and processes targets sequentially. Every catalog column has a **Sensitive** flag, which defaults
to `false`, including after a newly discovered column is synchronized. Before generation, an
administrator can ask the configured model to suggest flags from structural metadata only (database,
schema, table and column names, data types, nullability, primary keys, and foreign keys). Suggestions
remain an unsaved draft until a human reviews and saves them.
For a column with `sensitive=false`, the worker may read at most five source rows and five
representative non-null values through a read-only connector. For `sensitive=true`, the source query
does not request that column's values; deterministic plausible values derived only from its name and
type take their place in the model prompt. The prompt does not identify those values as synthetic, so
the model can still describe the field as if it had received representative data.
Each successful result is persisted immediately. Stop terminates the active helper but retains
earlier results. A helper has at most one provider retry; three consecutively exhausted technical
@@ -58,10 +66,12 @@ batches fail the run. Stale queued/running work is marked interrupted at startup
unlocked only when no local worker/helper is live. There is no automatic resume and no public
description-generation CLI.
Review generated text before copying it into the curated **Description** field. The sampling rule
is a deliberate data-disclosure boundary: do not use this facility for fields whose values must
not be sent to the configured provider until a Sensitive Data Policy is in place.
Review generated text before copying it into the curated **Description** field. Because the flag
defaults to `false`, an administrator must review the classification and mark protected fields before
starting generation. Changing a flag affects future generations only; existing generated or curated
descriptions are not regenerated. Real and substituted samples remain transient and are not persisted
or returned to the browser.
The decisions behind this surface are [ADRs 0001–0010](../adr/0001-postgres-metadata-catalog.md)
The decisions behind this surface are [ADRs 0001–0011](../adr/0001-postgres-metadata-catalog.md)
and the detailed acceptance record is
[AI catalog description generation acceptance](../testing/2026-08-29-ai-catalog-description-generation-acceptance.md).
frontend/src/api/catalog-databases.ts
+18 -1
View File
@@ -89,6 +89,7 @@ export interface CatalogColumn {
sourceComment: string | null;
description: string | null;
generatedDescription: string | null;
sensitive: boolean;
lastSyncedDatabaseVersion: number | null;
lastSyncedAt: string | null;
version: number;
@@ -96,6 +97,15 @@ export interface CatalogColumn {
updatedAt: string;
}
export interface SensitiveDataSuggestion {
columnId: string;
sensitive: boolean;
}
export interface SensitiveDataSuggestions {
suggestions: SensitiveDataSuggestion[];
}
export interface CatalogRelationshipColumn {
position: number;
sourceColumnId: string;
@@ -341,11 +351,18 @@ export const updateCatalogColumnMetadata = (
version: number,
description: string | null,
generatedDescription: string | null,
sensitive?: boolean,
) => apiFetch<CatalogColumn>(
`/catalog/databases/${encodeURIComponent(databaseId)}/tables/${encodeURIComponent(tableId)}/columns/${encodeURIComponent(columnId)}`,
{ method: "PATCH", body: JSON.stringify({ version, description, generatedDescription }) },
{ method: "PATCH", body: JSON.stringify({ version, description, generatedDescription, sensitive }) },
);
export const suggestSensitiveFields = (databaseId: string, modelId: string) =>
apiFetch<SensitiveDataSuggestions>(
`/catalog/databases/${encodeURIComponent(databaseId)}/sensitive-data-suggestions`,
{ method: "POST", body: JSON.stringify({ modelId }) },
);
export const listCatalogRelationships = (databaseId: string) =>
apiFetch<CatalogRelationship[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/relationships`);
frontend/src/shell/DatabaseManagementPage.test.tsx
+148 -47
View File
@@ -286,10 +286,13 @@ test("discloses bounded transient source samples in database-wide and selected g
name: "Metadata generation source data disclosure",
});
expect(within(disclosure).getByText(
"Description generation may send up to five real source rows and up to five representative non-null example values to the selected model provider.",
"Description generation may send up to five source rows and up to five representative non-null example values to the selected model provider. Values from columns marked sensitive are replaced with plausible synthetic values before the request.",
)).toBeVisible();
expect(within(disclosure).getByText(
"Samples are transient and are not stored in run logs or catalog metadata. Automated Sensitive Data Policy filtering and anonymization are not currently provided; they are planned for future work.",
"Values from unmarked columns may be sent unchanged. Samples are transient and are not stored in run logs or catalog metadata.",
)).toBeVisible();
expect(within(disclosure).getByText(
"Sensitive-field suggestions use structural metadata only and remain unsaved until you choose Save sensitive fields.",
)).toBeVisible();
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
@@ -333,7 +336,7 @@ test.each(synchronizationScopes)(
});
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox"));
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: synchronizationScopes[0].label })).toBeEnabled();
@@ -365,7 +368,7 @@ test("starts Generate Missing for one configured database without confirmation",
renderPage({ rows: [makeDatabase()] });
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox"));
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate Missing" }));
@@ -396,7 +399,7 @@ test("confirms Generate All replacement, supports cancel, and sends the database
renderPage({ rows: [makeDatabase()] });
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox"));
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate All" }));
@@ -434,7 +437,7 @@ test("keeps the database selected and safely explains when no descriptions are e
renderPage({ rows: [makeDatabase()] });
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox"));
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate Missing" }));
@@ -493,7 +496,7 @@ test.each([
]) client.setQueryData(queryKey, []);
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox"));
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", {
name: scope === "all" ? "Generate All" : "Generate Missing",
@@ -568,7 +571,7 @@ test.each([
for (const database of rows.slice(0, selectedRows)) {
const row = await screen.findByRole("row", { name: new RegExp(database.workspaceName) });
await user.click(within(row).getByRole("checkbox"));
await user.click(within(row).getByRole("checkbox", { name: /toggle row selection/i }));
}
await user.click(screen.getByRole("button", { name: "Actions" }));
@@ -599,13 +602,13 @@ test("disables database-wide generation while a description generation is active
renderPage({ rows: [makeDatabase()] });
let databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox"));
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate Missing" }));
expect(await screen.findByRole("complementary", { name: "Description generation" })).toBeVisible();
databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox"));
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate All" }))
.toHaveAttribute("aria-disabled", "true");
@@ -630,8 +633,8 @@ test("selected database Actions confirms and deletes catalog tables for the full
const firstRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
const secondRow = await screen.findByRole("row", { name: /Radiology/ });
await user.click(within(firstRow).getByRole("checkbox"));
await user.click(within(secondRow).getByRole("checkbox"));
await user.click(within(firstRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(secondRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Delete all tables" })).toBeEnabled();
@@ -642,10 +645,10 @@ test("selected database Actions confirms and deletes catalog tables for the full
expect(screen.getByText(/database configurations and source databases are unchanged/i)).toBeVisible();
expect(cleanupBody).toBeUndefined();
await user.click(within(secondRow).getByRole("checkbox"));
await user.click(within(secondRow).getByRole("checkbox", { name: /toggle row selection/i }));
expect(screen.queryByText(/Delete all catalog tables for/)).not.toBeInTheDocument();
expect(cleanupBody).toBeUndefined();
await user.click(within(secondRow).getByRole("checkbox"));
await user.click(within(secondRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Delete all tables" }));
await waitFor(() => expect(screen.getByRole("button", { name: "Delete catalog tables" })).toHaveFocus());
@@ -681,7 +684,7 @@ test("presents completed synchronization steps as success and skips unneeded con
});
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox"));
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Synchronize all columns" }));
@@ -1047,6 +1050,7 @@ const patientIdColumn: CatalogColumn = {
sourceComment: "Patient identifier",
description: null,
generatedDescription: null,
sensitive: false,
lastSyncedDatabaseVersion: 3,
lastSyncedAt: "2026-08-27T10:00:00Z",
version: 1,
@@ -1106,7 +1110,7 @@ test("selected table Actions exposes cleanup commands and sends synchronization
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
const tableRow = await screen.findByRole("row", { name: /patients/ });
await user.click(within(tableRow).getByRole("checkbox"));
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
const synchronizeColumns = await screen.findByRole("menuitem", { name: "Synchronize columns" });
@@ -1413,8 +1417,8 @@ test("moves selected generated table descriptions and reports copied and skipped
await user.click(screen.getByRole("tab", { name: "Tables" }));
const patientsRow = await screen.findByRole("row", { name: /patients/ });
const visitsRow = await screen.findByRole("row", { name: /visits/ });
await user.click(within(patientsRow).getByRole("checkbox"));
await user.click(within(visitsRow).getByRole("checkbox"));
await user.click(within(patientsRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(visitsRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", {
name: "Move generated description to Description",
@@ -1451,6 +1455,7 @@ test("selects columns, moves generated descriptions, and refreshes only the affe
sourceComment: "Patient identifier",
description: "Old identifier",
generatedDescription: "Generated identifier",
sensitive: false,
lastSyncedDatabaseVersion: 3,
lastSyncedAt: "2026-08-27T10:00:00Z",
version: 1,
@@ -1491,7 +1496,7 @@ test("selects columns, moves generated descriptions, and refreshes only the affe
const tableQuery = ["catalog-tables", patientsTable.databaseId] as const;
const selectableRow = async (name: RegExp) => {
const rows = await screen.findAllByRole("row", { name });
const row = rows.find((candidate) => within(candidate).queryByRole("checkbox"));
const row = rows.find((candidate) => within(candidate).queryByRole("checkbox", { name: /toggle row selection/i }));
expect(row).toBeDefined();
return row!;
};
@@ -1501,8 +1506,8 @@ test("selects columns, moves generated descriptions, and refreshes only the affe
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const idRow = await selectableRow(/Patient identifier/);
const nameRow = await selectableRow(/Keep curated patient name/);
await user.click(within(idRow).getByRole("checkbox"));
await user.click(within(nameRow).getByRole("checkbox"));
await user.click(within(idRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(nameRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", {
name: "Move generated description to Description",
@@ -1549,9 +1554,9 @@ test("starts one selected column with the configured default model", async () =>
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const columnRow = (await screen.findAllByRole("row", { name: /Patient identifier/ }))
.find((row) => within(row).queryByRole("checkbox"));
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
expect(columnRow).toBeDefined();
await user.click(within(columnRow!).getByRole("checkbox"));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
@@ -1566,6 +1571,101 @@ test("starts one selected column with the configured default model", async () =>
expect(await screen.findByText("Description generation started for 1 column")).toBeVisible();
});
test("reviews AI-sensitive-field suggestions as an editable draft and saves only changed columns", async () => {
const user = userEvent.setup();
const idColumn = { ...patientIdColumn, sensitive: false };
const nameColumn = {
...patientIdColumn,
id: "eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee",
name: "name",
ordinalPosition: 2,
primaryKeyPosition: null,
isPrimaryKey: false,
description: "Patient name",
generatedDescription: "Name of the patient",
sensitive: false,
};
let columns = [idColumn, nameColumn];
let suggestionBody: unknown;
const patches: Array<{ columnId: string; body: unknown }> = [];
server.use(
http.get("/api/catalog/metadata-generation/models", () => HttpResponse.json({
models: [{ id: "local-qwen", label: "Local Qwen" }],
default: "local-qwen",
})),
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
http.get(
"/api/catalog/databases/:databaseId/tables/:tableId/columns",
() => HttpResponse.json(columns),
),
http.post(
"/api/catalog/databases/:databaseId/sensitive-data-suggestions",
async ({ request }) => {
suggestionBody = await request.json();
return HttpResponse.json({
suggestions: [
{ columnId: idColumn.id, sensitive: true },
{ columnId: nameColumn.id, sensitive: true },
{ columnId: "ffffffff-ffff-4fff-8fff-ffffffffffff", sensitive: true },
],
});
},
),
http.patch(
"/api/catalog/databases/:databaseId/tables/:tableId/columns/:columnId",
async ({ params, request }) => {
const body = await request.json() as {
version: number;
description: string | null;
generatedDescription: string | null;
sensitive: boolean;
};
patches.push({ columnId: String(params.columnId), body });
const current = columns.find((column) => column.id === params.columnId)!;
const updated = { ...current, ...body, version: current.version + 1 };
columns = columns.map((column) => column.id === updated.id ? updated : column);
return HttpResponse.json(updated);
},
),
);
renderPage({ rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })] });
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const idSensitive = await screen.findByRole("checkbox", { name: "Sensitive data for id" });
const nameSensitive = await screen.findByRole("checkbox", { name: "Sensitive data for name" });
expect(idSensitive).not.toBeChecked();
expect(nameSensitive).not.toBeChecked();
await user.click(screen.getByRole("button", { name: "Suggest sensitive fields" }));
await waitFor(() => expect(suggestionBody).toEqual({ modelId: "local-qwen" }));
await waitFor(() => {
expect(screen.getByRole("checkbox", { name: "Sensitive data for id" })).toBeChecked();
expect(screen.getByRole("checkbox", { name: "Sensitive data for name" })).toBeChecked();
});
expect(patches).toHaveLength(0);
await user.click(screen.getByRole("checkbox", { name: "Sensitive data for name" }));
expect(screen.getByRole("checkbox", { name: "Sensitive data for name" })).not.toBeChecked();
await user.click(screen.getByRole("button", { name: "Save sensitive fields" }));
await waitFor(() => expect(patches).toEqual([{
columnId: idColumn.id,
body: {
version: idColumn.version,
description: idColumn.description,
generatedDescription: idColumn.generatedDescription,
sensitive: true,
},
}]));
expect(await screen.findByText("Sensitive fields saved")).toBeVisible();
expect(screen.getByRole("checkbox", { name: "Sensitive data for id" })).toBeChecked();
expect(screen.getByRole("checkbox", { name: "Sensitive data for name" })).not.toBeChecked();
});
test("polls a description run and renders its events in sequence order", async () => {
const user = userEvent.setup();
const queuedRun = makeDescriptionGenerationRun();
@@ -1617,8 +1717,8 @@ test("polls a description run and renders its events in sequence order", async (
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const columnRow = (await screen.findAllByRole("row", { name: /Patient identifier/ }))
.find((row) => within(row).queryByRole("checkbox"));
await user.click(within(columnRow!).getByRole("checkbox"));
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
@@ -1699,8 +1799,8 @@ test("refreshes Catalog Tables and Catalog Columns after column generation compl
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const columnRow = (await screen.findAllByRole("row", { name: /Patient identifier/ }))
.find((row) => within(row).queryByRole("checkbox"));
await user.click(within(columnRow!).getByRole("checkbox"));
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
@@ -1708,7 +1808,7 @@ test("refreshes Catalog Tables and Catalog Columns after column generation compl
await waitFor(() => expect(columnReads).toBe(2));
await waitFor(() => expect(tableReads).toBe(2));
expect(within((await screen.findAllByRole("row", { name: /Stable patient identifier/ }))
.find((row) => within(row).queryByRole("checkbox"))!).getByText("Stable patient identifier")).toBeVisible();
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }))!).getByText("Stable patient identifier")).toBeVisible();
expect(client.getQueryState(unrelatedColumnsQuery)?.isInvalidated).toBe(true);
});
@@ -1732,8 +1832,8 @@ test("keeps the selected column and shows a safe message when generation cannot
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const columnRow = (await screen.findAllByRole("row", { name: /Patient identifier/ }))
.find((row) => within(row).queryByRole("checkbox"));
await user.click(within(columnRow!).getByRole("checkbox"));
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
@@ -1799,8 +1899,8 @@ test("shows a basic failed run without exposing private model or provider fields
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const columnRow = (await screen.findAllByRole("row", { name: /Patient identifier/ }))
.find((row) => within(row).queryByRole("checkbox"));
await user.click(within(columnRow!).getByRole("checkbox"));
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
@@ -1851,10 +1951,10 @@ test("offers generation and consolidation for multiple selected columns", async
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const idRow = (await screen.findAllByRole("row", { name: /Patient identifier/ }))
.find((row) => within(row).queryByRole("checkbox"));
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
const nameRow = (await screen.findAllByRole("row", { name: /Patient name/ }))
.find((row) => within(row).queryByRole("checkbox"));
await user.click(within(idRow!).getByRole("checkbox"));
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(idRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate description" })).toBeEnabled();
expect(screen.getByRole("menuitem", {
@@ -1862,7 +1962,7 @@ test("offers generation and consolidation for multiple selected columns", async
})).toBeEnabled();
await user.keyboard("{Escape}");
await user.click(within(nameRow!).getByRole("checkbox"));
await user.click(within(nameRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
const generate = await screen.findByRole("menuitem", { name: "Generate descriptions" });
expect(generate).toBeEnabled();
@@ -1901,7 +2001,7 @@ test.each([
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
const tableRow = await screen.findByRole("row", { name: /patients/ });
await user.click(within(tableRow).getByRole("checkbox"));
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate description" }))
.toHaveAttribute("aria-disabled", "true");
@@ -1909,8 +2009,8 @@ test.each([
await user.click(screen.getByRole("button", { name: "Clear" }));
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const columnRow = (await screen.findAllByRole("row", { name: /Patient identifier/ }))
.find((row) => within(row).queryByRole("checkbox"));
await user.click(within(columnRow!).getByRole("checkbox"));
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate description" }))
@@ -1943,8 +2043,8 @@ test("disables another generation start while the selected-column run is active"
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const columnRow = (await screen.findAllByRole("row", { name: /Patient identifier/ }))
.find((row) => within(row).queryByRole("checkbox"));
await user.click(within(columnRow!).getByRole("checkbox"));
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
expect(await screen.findByRole("complementary", { name: "Description generation" })).toBeVisible();
@@ -1956,7 +2056,7 @@ test("disables another generation start while the selected-column run is active"
await user.click(screen.getByRole("button", { name: "Back to tables" }));
const tableRow = await screen.findByRole("row", { name: /patients/ });
await user.click(within(tableRow).getByRole("checkbox"));
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate description" }))
.toHaveAttribute("aria-disabled", "true");
@@ -1982,7 +2082,7 @@ test("disables selected description consolidation without database.manage", asyn
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
const tableRow = await screen.findByRole("row", { name: /patients/ });
await user.click(within(tableRow).getByRole("checkbox"));
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", {
@@ -2011,7 +2111,7 @@ test("shows an operation conflict without clearing selected descriptions or refr
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
const tableRow = await screen.findByRole("row", { name: /patients/ });
await user.click(within(tableRow).getByRole("checkbox"));
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", {
name: "Move generated description to Description",
@@ -2045,7 +2145,7 @@ test("selected table Actions confirms and deletes incoming and outgoing relation
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
const tableRow = await screen.findByRole("row", { name: /patients/ });
await user.click(within(tableRow).getByRole("checkbox"));
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Delete all relationships" }));
@@ -2053,10 +2153,10 @@ test("selected table Actions confirms and deletes incoming and outgoing relation
expect(screen.getByText(/incoming and outgoing relationships/i)).toBeVisible();
expect(cleanupBody).toBeUndefined();
await user.click(within(tableRow).getByRole("checkbox"));
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
expect(screen.queryByText("Delete all relationships for 1 table?")).not.toBeInTheDocument();
expect(cleanupBody).toBeUndefined();
await user.click(within(tableRow).getByRole("checkbox"));
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Delete all relationships" }));
await user.click(screen.getByRole("button", { name: "Delete relationships" }));
@@ -2133,6 +2233,7 @@ test("navigates from a table to columns and from the database to physical relati
sourceComment: "Patient identifier",
description: null,
generatedDescription: null,
sensitive: false,
lastSyncedDatabaseVersion: 3,
lastSyncedAt: "2026-08-27T10:00:00Z",
version: 1,
frontend/src/shell/database-management/DatabaseColumns.tsx
+122 -6
View File
@@ -3,7 +3,7 @@ import { Menu } from "@base-ui/react/menu";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { AgGridReact } from "ag-grid-react";
import type { ColDef, ICellRendererParams } from "ag-grid-community";
import { ChevronDown, KeyRound, Link2, Pencil, RefreshCw, Save, X } from "lucide-react";
import { ChevronDown, KeyRound, Link2, Pencil, RefreshCw, Save, Sparkles, X } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../../components/ui/button";
import { ApiError, apiErrorMessage } from "../../api/client";
@@ -11,6 +11,7 @@ import {
consolidateCatalogDescriptions,
listCatalogColumns,
startDescriptionGenerationRun,
suggestSensitiveFields,
updateCatalogColumnMetadata,
type CatalogColumn,
type CatalogTable,
@@ -31,7 +32,9 @@ interface Props {
interface GridContext {
canManage: boolean;
busy: boolean;
onEdit: (column: CatalogColumn, origin: HTMLButtonElement) => void;
onSensitiveChange: (column: CatalogColumn, sensitive: boolean) => void;
}
function KeyCell({ data }: ICellRendererParams<CatalogColumn>) {
@@ -55,6 +58,23 @@ function ActionCell({ data, context }: ICellRendererParams<CatalogColumn, unknow
);
}
function SensitiveCell({ data, context }: ICellRendererParams<CatalogColumn, unknown, GridContext>) {
if (!data || !context) return null;
return (
<div className="flex h-full items-center justify-center" onClick={(event) => event.stopPropagation()}>
<input
type="checkbox"
className="size-4 accent-primary outline-none focus-visible:ring-3 focus-visible:ring-ring/50"
checked={data.sensitive}
disabled={!context.canManage || context.busy}
aria-label={`Sensitive data for ${data.name}`}
onChange={(event) => context.onSensitiveChange(data, event.target.checked)}
onClick={(event) => event.stopPropagation()}
/>
</div>
);
}
export function DatabaseColumns({
databaseId,
table,
@@ -77,6 +97,8 @@ export function DatabaseColumns({
const [editingId, setEditingId] = useState<string | null>(null);
const [description, setDescription] = useState("");
const [generatedDescription, setGeneratedDescription] = useState("");
const [sensitiveDrafts, setSensitiveDrafts] = useState<Record<string, boolean>>({});
const [sensitiveAction, setSensitiveAction] = useState<"suggest" | "save" | null>(null);
const [baseline, setBaseline] = useState("");
const [version, setVersion] = useState<number | null>(null);
const [stale, setStale] = useState(false);
@@ -86,7 +108,17 @@ export function DatabaseColumns({
const originRef = useRef<HTMLButtonElement | null>(null);
const active = editingId ? data.find((column) => column.id === editingId) : undefined;
const fingerprint = JSON.stringify([description, generatedDescription]);
const dirty = Boolean(editingId && fingerprint !== baseline);
const editorDirty = Boolean(editingId && fingerprint !== baseline);
const changedSensitiveColumns = data.filter((column) => (
Object.hasOwn(sensitiveDrafts, column.id)
&& sensitiveDrafts[column.id] !== column.sensitive
));
const dirty = editorDirty || changedSensitiveColumns.length > 0;
const displayedColumns = useMemo(() => data.map((column) => (
Object.hasOwn(sensitiveDrafts, column.id)
? { ...column, sensitive: sensitiveDrafts[column.id]! }
: column
)), [data, sensitiveDrafts]);
useEffect(() => { onNavigationStateChange({ dirty, busy }); }, [busy, dirty, onNavigationStateChange]);
useEffect(() => {
@@ -107,7 +139,7 @@ export function DatabaseColumns({
};
const closeEditor = () => {
if (busy) return;
if (dirty && !window.confirm("Discard unsaved column metadata?")) return;
if (editorDirty && !window.confirm("Discard unsaved column metadata?")) return;
setEditingId(null);
window.setTimeout(() => originRef.current?.focus(), 0);
};
@@ -178,8 +210,74 @@ export function DatabaseColumns({
} finally { setBusy(false); }
};
const changeSensitive = (column: CatalogColumn, sensitive: boolean) => {
const persisted = data.find((candidate) => candidate.id === column.id);
if (!persisted) return;
setSensitiveDrafts((current) => {
const next = { ...current };
if (sensitive === persisted.sensitive) delete next[column.id];
else next[column.id] = sensitive;
return next;
});
};
const suggestSensitive = async () => {
if (!selectedMetadataModel) return;
setBusy(true);
setSensitiveAction("suggest");
try {
const result = await suggestSensitiveFields(databaseId, selectedMetadataModel);
const currentById = new Map(data.map((column) => [column.id, column]));
const next: Record<string, boolean> = {};
for (const suggestion of result.suggestions) {
const column = currentById.get(suggestion.columnId);
if (column && suggestion.sensitive !== column.sensitive) {
next[column.id] = suggestion.sensitive;
}
}
setSensitiveDrafts(next);
toast.success("Sensitive field suggestions ready for review");
} catch (error) {
toast.error(apiErrorMessage(error));
} finally {
setSensitiveAction(null);
setBusy(false);
}
};
const saveSensitive = async () => {
if (changedSensitiveColumns.length === 0) return;
setBusy(true);
setSensitiveAction("save");
try {
const updated = await Promise.all(changedSensitiveColumns.map((column) => (
updateCatalogColumnMetadata(
databaseId,
table.id,
column.id,
column.version,
column.description,
column.generatedDescription,
sensitiveDrafts[column.id],
)
)));
const updatedById = new Map(updated.map((column) => [column.id, column]));
queryClient.setQueryData<CatalogColumn[]>(queryKey, (current = []) => current.map(
(column) => updatedById.get(column.id) ?? column,
));
setSensitiveDrafts({});
toast.success("Sensitive fields saved");
} catch (error) {
toast.error(apiErrorMessage(error));
} finally {
setSensitiveAction(null);
setBusy(false);
}
};
const columns = useMemo<ColDef<CatalogColumn>[]>(() => [
{ field: "ordinalPosition", headerName: "#", width: 64, maxWidth: 64, filter: "agNumberColumnFilter" },
{ field: "sensitive", headerName: "Sensitive", minWidth: 110, width: 110, sortable: false, filter: false, resizable: false, cellRenderer: SensitiveCell },
{ field: "name", headerName: "Name", minWidth: 190, flex: 1, cellClass: "font-mono text-xs" },
{ field: "dataType", headerName: "Type", minWidth: 150, flex: 0.8, cellClass: "font-mono text-xs" },
{ headerName: "Keys", minWidth: 125, width: 125, sortable: false, filter: false, cellRenderer: KeyCell },
@@ -189,7 +287,12 @@ export function DatabaseColumns({
{ field: "description", headerName: "Description", minWidth: 230, flex: 1.2, valueFormatter: ({ value }) => value ?? "" },
{ colId: "actions", headerName: "", width: 64, maxWidth: 64, pinned: "right", sortable: false, filter: false, resizable: false, cellRenderer: ActionCell },
], []);
const context = useMemo<GridContext>(() => ({ canManage, onEdit: edit }), [canManage, data]);
const context = useMemo<GridContext>(() => ({
canManage,
busy,
onEdit: edit,
onSensitiveChange: changeSensitive,
}), [busy, canManage, data]);
if (editingId && active) {
return (
@@ -213,7 +316,7 @@ export function DatabaseColumns({
</div>
<div className="mt-5 flex justify-end gap-2 border-t border-border pt-4">
<Button type="button" variant="outline" disabled={busy} onClick={closeEditor}>Cancel</Button>
<Button type="button" disabled={!canManage || !dirty || busy || stale} onClick={() => void save()}><Save />{busy ? "Saving…" : "Save metadata"}</Button>
<Button type="button" disabled={!canManage || !editorDirty || busy || stale} onClick={() => void save()}><Save />{busy ? "Saving…" : "Save metadata"}</Button>
</div>
</div>
);
@@ -255,6 +358,19 @@ export function DatabaseColumns({
<span className="thot-label whitespace-nowrap">Catalog columns</span>
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search columns" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
<Button
type="button"
variant="outline"
disabled={!canManage || !selectedMetadataModel || descriptionGenerationActive || busy}
onClick={() => void suggestSensitive()}
>
<Sparkles />{sensitiveAction === "suggest" ? "Suggesting…" : "Suggest sensitive fields"}
</Button>
{changedSensitiveColumns.length > 0 ? (
<Button type="button" disabled={!canManage || busy} onClick={() => void saveSensitive()}>
<Save />{sensitiveAction === "save" ? "Saving…" : "Save sensitive fields"}
</Button>
) : null}
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
<Button type="button" disabled={!canManage || busy} onClick={onSync}><RefreshCw />Sync columns</Button>
</>
@@ -264,7 +380,7 @@ export function DatabaseColumns({
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
<AgGridReact<CatalogColumn>
ref={gridRef}
rowData={data}
rowData={displayedColumns}
columnDefs={columns}
context={context}
loading={isLoading}
frontend/src/shell/database-management/MetadataGenerationModelSelector.tsx
+9 -5
View File
@@ -61,13 +61,17 @@ export function MetadataGenerationModelSelector({
className="space-y-1 text-xs font-normal leading-4 text-muted-foreground"
>
<p>
Description generation may send up to five real source rows and up to five representative
non-null example values to the selected model provider.
Description generation may send up to five source rows and up to five representative
non-null example values to the selected model provider. Values from columns marked
sensitive are replaced with plausible synthetic values before the request.
</p>
<p>
Samples are transient and are not stored in run logs or catalog metadata. Automated
Sensitive Data Policy filtering and anonymization are not currently provided; they are
planned for future work.
Values from unmarked columns may be sent unchanged. Samples are transient and are not
stored in run logs or catalog metadata.
</p>
<p>
Sensitive-field suggestions use structural metadata only and remain unsaved until you
choose Save sensitive fields.
</p>
</div>
</div>
mkdocs.yml
+1
View File
@@ -84,6 +84,7 @@ nav:
- 0008 Manual catalog cleanup: adr/0008-allow-manual-catalog-metadata-cleanup.md
- 0009 Sequential description generation: adr/0009-use-one-sequential-description-generation-run.md
- 0010 Bounded source samples: adr/0010-allow-bounded-real-source-samples-for-description-generation.md
- 0011 Sensitive data flag: adr/0011-gate-source-samples-with-a-sensitive-data-flag.md
- AI catalog description acceptance: testing/2026-08-29-ai-catalog-description-generation-acceptance.md
- Design records:
- Metadata catalog design: plans/2026-08-26-metadata-catalog-from-thothai.md