Compare commits

..
Author SHA1 Message Date
Codex 7630762927 docs: research open-core conversion patterns 2026-09-02 18:15:53 +02:00
113 changed files with 1413 additions and 9213 deletions
+23 -96
View File
@@ -244,29 +244,6 @@ incompatibile o non aggiornato produce nessuna Evidence e un avviso esplicito. I
workflow può continuare, ma non usa mai silenziosamente contenuti di una revisione
precedente o di un altro workspace.
## Configurazione dei modelli
**Workspace Descriptor** — La dichiarazione versionata dell'identità e dello scope del
Workspace Database e delle Evidence di un workspace. Non definisce modelli, selezioni di
modello o Database Binding specifiche di un'installazione.
**Installation Model Catalog** — L'insieme dichiarativo, proprio di un'installazione, dei
modelli disponibili, dei loro Model Usage e dei relativi default. È l'unica autorità per i
modelli di sessione, generazione dei metadati ed embedding e non appartiene a un workspace.
_Avoid_: Model Catalog, Metadata Generation Model Configuration
**Model Usage** — Lo scopo per cui un modello dell'Installation Model Catalog può essere
usato: `session`, `metadata_generation` oppure `embedding`. L'ammissibilità e il default
dipendono dall'uso, non dal workspace.
**Model Selection** — La scelta runtime, a livello di installazione, di un modello del
catalogo per uno specifico Model Usage. Riferisce l'identità canonica del modello senza
ridefinirne provider, endpoint o capacità.
**Model Runtime Projection** — La rappresentazione derivata e non autoritativa
dell'Installation Model Catalog richiesta da uno specifico runtime. Può essere rigenerata
integralmente dalla configurazione dell'installazione.
## Catalogo dei metadati
**Workspace Database** — Il database che appartiene a un solo workspace e non può essere
@@ -319,40 +296,8 @@ Metadata Catalog. Non è creata o modificata manualmente, ma può essere rimossa
Metadata Cleanup.
_Avoid_: denormalized FK, relationship string
**Logical Relationship** — Una relazione modificabile fra due Catalog Column che non corrisponde
necessariamente a un vincolo fisico. Può essere Generated o Manual e rimane distinta dalla Catalog
Relationship osservata nel database.
**Generated Relationship** — Una Logical Relationship ricavata dai nomi delle colonne, dalle
primary key e dalla compatibilità dei tipi mediante regole deterministiche, senza LLM, embedding o
campionamento dei dati. Una ricostruzione non riattiva una Generated Relationship cancellata
logicamente, ma può ricrearne una cancellata fisicamente.
**Manual Relationship** — Una Logical Relationship aggiunta dall'utente. La ricostruzione delle
Generated Relationship non la modifica.
**Logical Relationship Deletion** — L'esclusione persistente di una Logical Relationship che ne
conserva l'identità per impedirne la ricreazione automatica finché esistono entrambe le Catalog
Column alle quali è collegata.
**Permanent Relationship Deletion** — La rimozione completa di una Logical Relationship. Una
ricostruzione successiva può ricrearla quando soddisfa nuovamente le regole di inferenza. Anche il
cleanup distruttivo di una tabella o colonna endpoint rimuove permanentemente le relative esclusioni.
**Relationship Reconstruction** — L'operazione amministrativa esplicita che scopre e aggiunge le
Generated Relationship mancanti. Conserva le Manual Relationship e le relationship già presenti e
non riattiva quelle cancellate logicamente.
**Relationship Restore** — La riattivazione esplicita di una Logical Relationship cancellata
logicamente.
**Effective Relationship Map** — La vista unificata delle Catalog Relationship fisiche e delle
Logical Relationship, con origine e stato espliciti. È l'interfaccia usata dall'amministrazione e
dalla comprensione dello schema, non un ulteriore modello persistito.
**Effective Relationship Snapshot** — La proiezione runtime immutabile delle relationship attive
contenute nell'Effective Relationship Map. È derivata dal Metadata Catalog per una singola sessione
e viene eliminata insieme alla relativa configurazione runtime.
**Logical Relationship** — Una relazione semantica curata o inferita che non corrisponde
necessariamente a un vincolo fisico. Ha ownership e lifecycle distinti da Catalog Relationship.
**Description** — Il testo curato e consolidato che descrive una Catalog Table o Catalog Column
per gli usi downstream.
@@ -408,56 +353,38 @@ con la binding corrente.
distinti dai fatti strutturali governati dalla sincronizzazione. Possono essere popolati dall'AI,
da un'importazione o da una modifica amministrativa senza cambiare il database esterno.
**Metadata Generation Model Configuration** — La configurazione a livello di setup applicativo
che elenca i modelli selezionabili, il default e i riferimenti agli eventuali segreti per la sola
generazione dei metadati. Un modello keyless è ammesso solo con un endpoint esplicito che non
richiede autenticazione. Non appartiene al workspace ed è indipendente dalla configurazione Pi.
**Model Completion Helper** — Il processo Python interno ed effimero che esegue una singola
richiesta LiteLLM per conto del backend. Non è un servizio HTTP, non possiede il lifecycle della
Description Generation Run e non è una CLI esposta agli utenti.
**Catalog Sample** — Un input transitorio composto da un massimo di cinque righe e da valori di
esempio bounded di una Catalog Table per la generazione delle descrizioni. Può contenere valori
reali oppure sintetici in base alla Source Value Disclosure Decision; non viene persistito e non
diventa Catalog Metadata.
reali oppure sintetici in base al Sensitive Data Flag della Catalog Column; non viene persistito
e non diventa Catalog Metadata.
**Sensitive Data Flag** — La classificazione binaria umana applicata a una Catalog Column. Può
essere impostata liberamente dall'amministratore anche in contrasto con una valutazione automatica.
**Sensitive Data Flag** — La scelta binaria umana applicata a una Catalog Column: `true` protegge
i valori sorgente e `false` ne consente l'invio al modello. Il valore predefinito è `false`, anche
per le nuove colonne.
**Local Sensitivity Assessment** — La valutazione locale, non autoritativa e priva di LLM di una
Catalog Column, basata su metadati e contenuto sorgente, con esito `sensitive`, `non_sensitive`
oppure `unknown`.
_Avoid_: AI suggestion, automatic flag
**Local NER Detector** — Il componente NLP opzionale e CPU-only che esamina soltanto testo ancora
ambiguo e restituisce evidenze al Local Sensitivity Assessment. Non decide lo stato della colonna,
non usa un LLM generativo e non persiste valori sorgente.
_Avoid_: AI classifier, local LLM fallback
**Model Data Boundary** — La qualificazione amministrativa di un modello come `internal` oppure
`external` rispetto al confine entro cui i valori sorgente possono essere comunicati.
_Avoid_: local model, remote model
**Source Value Disclosure Decision** — L'unica decisione effettiva che stabilisce se un modello
riceve valori sorgente reali oppure sostituti sintetici, combinando Model Data Boundary e Sensitive
Data Flag.
_Avoid_: sample filter, export flag
**Sensitive Data Policy** — L'insieme versionato di regole locali generali e specifiche che produce
una Local Sensitivity Assessment. Un singolo riscontro blocca l'intera colonna e qualsiasi valore
testuale più lungo di 500 caratteri rende sensibile la colonna.
**Sensitive Data Policy** — La regola che applica il Sensitive Data Flag ai Catalog Sample:
valori sintetici per una colonna protetta, valori reali per una colonna non protetta. L'AI può
suggerire il flag dai soli metadati tecnici di un database, delle tabelle o delle colonne
esplicitamente selezionate; le richieste ampie vengono divise in batch bounded, ma soltanto
l'utente imposta i flag dopo aver rivisto la proposta completa.
_Avoid_: PII filter, sample filter
**Sensitivity Analysis Run** — Il tentativo amministrativo esplicito e tracciato che valuta una
selezione di colonne mediante la Sensitive Data Policy. Conserva stato, copertura e conteggi
aggregati, ma non valori sorgente né esiti per colonna.
_Avoid_: Sensitive Data Suggestion Run, AI analysis
**Sensitive Data Suggestion Run** — Il tentativo amministrativo tracciato con cui il modello
propone Sensitive Data Flag dai soli metadati strutturali. Conserva stato e conteggi aggregati,
ma non i suggerimenti per colonna, che restano una proposta transitoria fino al salvataggio umano.
**Sensitivity Review Draft** — La proposta transitoria che associa alle colonne selezionate una
Local Sensitivity Assessment e le relative evidenze sanificate. Non modifica il Sensitive Data Flag
finché l'amministratore non salva le proprie decisioni e viene scartata al reload.
_Avoid_: automatic flag
**Sensitivity Analysis Event** — Una riga testuale ordinata e sanificata che registra l'avvio,
l'esito o l'errore di una Sensitivity Analysis Run senza conservare contenuti sorgente, output grezzi
del detector o proposte per colonna.
_Avoid_: Sensitive Data Suggestion Event
**Sensitive Data Suggestion Event** — Una riga testuale ordinata e sanitizzata che registra
l'avvio, l'esito o l'errore di una Sensitive Data Suggestion Run senza conservare prompt,
risposte grezze del provider o proposte per colonna.
**Introspection Capability** — Una categoria di struttura fisica che una Database Binding
può osservare, come tabelle, colonne, relazioni, indici o enum. Una capability non disponibile
+4 -40
View File
@@ -12,10 +12,6 @@ colors:
muted-graphite: "oklch(51.33% 0.0088 345.6)"
quiet-border: "oklch(90.93% 0.0035 354.7)"
success-mint: "oklch(75.77% 0.1581 165)"
navigation-active: "oklch(92.5% 0.052 23.2)"
navigation-active-hover: "oklch(89.5% 0.071 23.2)"
navigation-active-foreground: "oklch(36.5% 0.11 23.2)"
navigation-active-border: "oklch(60% 0.135 23.2)"
warning-amber: "oklch(85.23% 0.1386 78.9)"
information-blue: "oklch(70.35% 0.1128 221.3)"
typography:
@@ -156,8 +152,8 @@ frontend uses OKLCH tokens directly.
### Primary
- **Instrument Red** (`instrument-red`): primary actions, focus identity, and destructive meaning
where the context already makes the action explicit.
- **Instrument Red** (`instrument-red`): primary actions, current selection, focus identity, and
destructive meaning where the context already makes the action explicit.
- **Instrument Red Pressed** (`instrument-red-hover`): hover and active emphasis for the primary
action family.
@@ -174,9 +170,6 @@ frontend uses OKLCH tokens directly.
### Semantic
- **Success Mint** (`success-mint`): completed and ready states.
- **Navigation Active** (`navigation-active`): the one application surface currently in the
foreground. It shares Instrument Red's hue but uses a lighter, lower-chroma fill, so location is
visible without carrying the full weight of a primary action.
- **Warning Amber** (`warning-amber`): waiting, attention, and in-progress states.
- **Information Blue** (`information-blue`): informational state when red would imply action.
@@ -279,44 +272,15 @@ default, hover, focus, active, disabled, loading, and error behavior where those
- **Focus:** three-pixel Instrument Red ring with a clear border shift.
- **Error / Disabled:** errors combine destructive color with explanatory text; disabled controls
retain readable contrast and use 50 percent opacity.
- **Metadata catalog model:** Database Management keeps one compact, installation-level
metadata-generation LLM selector in the application header. The selection persists across
database, table, column, and relationship views; when no usable profile is configured, the
disabled control explains: “No metadata-generation LLM model is configured for this installation.”
### Navigation
- **Style:** compact session rows use `8px` corners and restrained vertical padding.
- **Default / Hover / Active:** porcelain at rest, Sunken Surface on hover, and a muted Navigation
Active red with a defined border when current. Exactly one top-level navigation control is current.
- **Administrative controls:** the admin-only Administration accordion groups Database management,
a structural divider, Workspace management, and Pi management in that order. Its trigger exposes
expanded state and starts collapsed by default, while non-admin users do not receive the accordion
or its navigation actions.
- **Default / Hover / Active:** transparent at rest, Sunken Surface on hover, and the same surface
with stronger text weight when active.
- **Responsive:** collapse navigation structurally at the application breakpoint. Do not shrink
labels into illegibility.
### Tabs
- **Shape:** compact label tabs sit on a shared baseline with rounded top corners and a two-pixel
lower edge. Inactive labels retain a complete Quiet Border and Porcelain Card surface, so every
label reads as a tab before interaction; hover feedback reinforces clickability.
- **Current:** the selected tab uses the muted Navigation Active red for its fill, text, and defined border.
It must expose `aria-selected`, participate in a labelled `tablist`/`tabpanel`, and be the only
tab in the roving keyboard tab order.
- **Keyboard:** Left/Right move between adjacent tabs with wrapping; Home/End select the first or
last tab.
### Tooltips
- **Row actions:** icon-action tooltips open three pixels below the trigger and align to its trailing
edge, so they never cover the icon row. They use a dark slate surface, porcelain text, and a
defined border rather than the light popover treatment.
- **Interaction:** tooltip layers never receive pointer events. They appear on hover and keyboard
focus with a short ease-out transition, while the icon button keeps its complete accessible name.
- **Scope:** this treatment is shared by database, table, column, and relationship row actions.
Toolbar and navigation hints may use separate collision-aware placement.
### Curated Evidence Documents
Curated evidence follows a fixed reading order: title, compact type and purpose summary, scope,
+15 -28
View File
@@ -1,6 +1,6 @@
# ThothII — Project State
Last updated: 2026-08-31.
Last updated: 2026-08-27.
This file is the short operational snapshot. Stable commands and the architecture mental model
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
@@ -90,27 +90,16 @@ relationships. Curated and generated descriptions are editable; generated descri
and Database Management can generate or consolidate them for selected tables, selected columns,
all targets, or only targets whose Generated Description is missing.
Relationship Management is now reachable directly from each configured Fleet database. One
Relationship Map shows read-only Physical Relationships together with Generated and Manual Logical
Relationships, with Active, Excluded, and All filters. Administrators can add a single-column
relationship, run deterministic name/PK/type inference, exclude or restore a logical relationship,
or delete it permanently. Exclusion retains a tombstone that a rebuild cannot reactivate; permanent
deletion allows a later rebuild to infer the same endpoints again. Inference uses no LLM, embedding,
or source values. It supports normalized table-qualified names, unique non-generic PK names,
composite-PK source columns, and the `*time_key -> dim_time.<single PK>` warehouse convention while
ignoring bare generic names. Explicit table/column metadata cleanup remains a destructive boundary: it removes
the attached logical relationships and exclusions and requires a full schema synchronization before
inference or runtime publication can continue.
The previous Database Management renderer remains a temporary comparison fallback for development
and staging only: `?db-ui=legacy` is honored in Vite development or when
`VITE_DB_MANAGEMENT_LEGACY=true`; it is not a production presentation. The standalone Fleet Ledger
prototype on port `5173` also remains temporary until owner acceptance of the integrated surface,
after which both migration aids can be removed.
Schema refresh is one durable asynchronous engine with database-table, selected-table-column,
relationship, and full-database actions. Database-level menus expose only the table, relationship,
and full scopes; selecting tables exposes column synchronization plus manual column and relationship cleanup for that subset. Database selections
Schema refresh is one durable asynchronous engine with database-table, database-column,
selected-table-column, relationship, and full-database actions. Database-level menus expose the
table, all-column, relationship, and full scopes separately; selecting tables exposes column
synchronization plus manual column and relationship cleanup for that subset. Database selections
also expose manual table and relationship cleanup. Cleanup selections are atomic and share the
one-active-operation-per-database exclusion with synchronization. Runs have leases and
restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before
@@ -126,15 +115,13 @@ SSH is not yet enabled for NL→SQL session runtime.
The catalog runs in the internal `catalog-db` PostgreSQL service. Kysely migrations are an explicit
one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before local startup. Runtime
sessions now consume the Catalog's active effective relationship map through an immutable JSON
snapshot tied to the runtime-config lease. The harness uses that snapshot as its exclusive
relationship source while retaining Git-pinned annotations for descriptive metadata; legacy
runtimes without a snapshot keep the previous merge behavior. The accepted design is recorded in
sessions still consume the existing workspace configuration in this slice: database-management
records do not yet change the NL→SQL handoff. The accepted design is recorded in
`docs/plans/2026-08-26-metadata-catalog-from-thothai.md`, the snapshot contract under
`docs/contracts/`, and ADRs 0001–0012.
`docs/contracts/`, and ADRs 0001–0011.
Semantic aliases, value descriptions, synonyms, and concepts remain deferred to their dedicated
slices.
Semantic aliases, value descriptions, synonyms, concepts, and logical relationships remain
deferred to their dedicated slices.
AI Description Generation uses the catalog's human-owned Sensitive Data Flag. The flag defaults to
`false`, including for newly synchronized columns. An administrator may request an AI proposal based
@@ -171,11 +158,11 @@ credential. The Python client supplies only its fixed non-secret compatibility p
The AritmoLab entry also sets `disableThinking: true`, mapped to the endpoint's chat-template flag,
because its default reasoning prose would violate the worker's exact JSON response contract.
Logical relationship integration with core schema-linking is complete: session creation and resume
materialize the active physical/generated/manual map, retrieval-pack generation and Pi receive the
same runtime config, and snapshot validation fails closed on a declared missing, invalid, or orphaned
endpoint. Broader publication of other Catalog metadata to schema-linking remains a separate future
slice.
Integration of the completed metadata catalog with core schema-linking is explicitly deferred
until the database, table, column, relationship, and synchronization slices are complete. At that
point the next required design gate is to compare the catalog snapshot with the current DWH
preprocessing/schema-linking contracts and plan the cutover; this follow-up must not be treated as
optional cleanup or silently omitted.
**Deferred follow-up — Sensitive Data Policy in schema-linking.** The policy is first delivered
and tested in catalog description generation. Its enforcement for core schema-linking remains
+2 -22
View File
@@ -59,13 +59,10 @@ import { DescriptionGenerationWorker } from "./catalog/description-generation-wo
import { SensitiveDataSuggester } from "./catalog/sensitive-data-suggester.js";
import { SensitiveDataSuggestionRunner } from "./catalog/sensitive-data-suggestion-runner.js";
import {
ConcreteDescriptionSourceSampler,
PostgresDescriptionSourceSampler,
type DescriptionSourceSampler,
} from "./catalog/description-source-sampler.js";
import { catalogDescriptionGenerationRoutes } from "./routes/catalog-description-generation.js";
import { CatalogLogicalRelationshipService } from "./catalog/logical-relationship-service.js";
import { catalogLogicalRelationshipRoutes } from "./routes/catalog-logical-relationships.js";
import { EffectiveRelationshipSnapshotProvider } from "./catalog/effective-relationship-snapshot.js";
export interface BuildAppDeps {
thtRunner?: ThtRunner;
@@ -82,8 +79,6 @@ export interface BuildAppDeps {
catalogService?: CatalogService;
catalogPostgresAccess?: CatalogPostgresAccess;
catalogTableService?: CatalogTableService;
catalogLogicalRelationshipService?: CatalogLogicalRelationshipService;
effectiveRelationshipSnapshotProvider?: EffectiveRelationshipSnapshotProvider;
catalogSchemaIntrospector?: CatalogSchemaIntrospector;
catalogSyncWorker?: CatalogSyncWorker;
catalogOperationCoordinator?: CatalogOperationCoordinator;
@@ -177,7 +172,7 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
{ connectTimeoutMs: config.workspaceDiagnosticTimeoutMs },
);
const descriptionSourceSampler = deps?.descriptionSourceSampler
?? new ConcreteDescriptionSourceSampler(catalogPostgresAccess, workspaceSecretStore);
?? new PostgresDescriptionSourceSampler(catalogPostgresAccess);
const descriptionGenerationWorker = new DescriptionGenerationWorker(
catalogRepository,
workspaceRegistry,
@@ -205,16 +200,6 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
catalogOperationCoordinator,
);
const catalogTableService = deps?.catalogTableService ?? new CatalogTableService(catalogRepository);
const catalogLogicalRelationshipService = deps?.catalogLogicalRelationshipService
?? new CatalogLogicalRelationshipService(catalogRepository);
const effectiveRelationships = deps?.effectiveRelationshipSnapshotProvider
?? (config.catalogDatabase === undefined
? undefined
: new EffectiveRelationshipSnapshotProvider(
catalogRepository,
catalogLogicalRelationshipService,
catalogOperationCoordinator,
));
const catalogSchemaIntrospector = deps?.catalogSchemaIntrospector ?? new ConcreteCatalogSchemaIntrospector(
catalogPostgresAccess,
workspaceSecretStore,
@@ -409,7 +394,6 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
legacyWorkspaceMode: config.legacyWorkspaceMode,
workspaceRuntimeSupport,
maintenanceBarrier,
effectiveRelationships,
});
app.post("/internal/maintenance/activate", async (req, reply) => {
try {
@@ -458,10 +442,6 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
worker: catalogSyncWorker,
operations: catalogOperationCoordinator,
});
catalogLogicalRelationshipRoutes(app, {
service: catalogLogicalRelationshipService,
operations: catalogOperationCoordinator,
});
catalogDescriptionConsolidationRoutes(app, {
repository: catalogRepository,
operations: catalogOperationCoordinator,
@@ -1,7 +1,7 @@
import { z } from "zod";
import type { WorkspaceRegistry } from "../workspaces/registry.js";
import type { MetadataGenerationModels, ResolvedMetadataGenerationModel } from "./metadata-generation-models.js";
import type { ModelCompleter, ModelCompletionMessage, ModelCompletionResult } from "./model-completer.js";
import type { ModelCompleter, ModelCompletionMessage } from "./model-completer.js";
import {
ModelCompletionCancelledError,
ModelCompletionProviderError,
@@ -38,7 +38,6 @@ const MAX_SAMPLE_FIELDS_PER_ROW = 4;
const MAX_SAMPLE_COLUMNS = 4;
const MAX_REPRESENTATIVE_VALUES_PER_REQUEST = 5;
const MAX_TARGET_SAMPLE_JSON_BYTES = 8 * 1024;
const MAX_COMPLETION_ATTEMPTS_PER_BATCH = 2;
const generatedOutcomeSchema = z.object({
targetId: z.uuid(),
outcome: z.literal("generated"),
@@ -56,9 +55,7 @@ const completionResponseSchema = z.object({
results: z.array(outcomeSchema).min(1).max(MAX_TARGETS_PER_BATCH),
}).strict();
class InvalidModelJsonError extends Error {}
class InvalidModelSchemaError extends Error {}
class MissingModelTargetsError extends Error {}
class InvalidModelOutcomeError extends Error {}
class DescriptionGenerationBatchError extends Error {
constructor(
readonly failure: unknown,
@@ -146,9 +143,6 @@ interface DescriptionGenerationCounters {
generated: number;
nonGeneratable: number;
failed: number;
inputTokens: number;
cacheReadTokens: number;
outputTokens: number;
consecutiveTechnicalFailures: number;
}
@@ -158,9 +152,6 @@ function persistedCounters(counters: DescriptionGenerationCounters) {
generated: counters.generated,
nonGeneratable: counters.nonGeneratable,
failed: counters.failed,
inputTokens: counters.inputTokens,
cacheReadTokens: counters.cacheReadTokens,
outputTokens: counters.outputTokens,
};
}
@@ -634,47 +625,36 @@ function messagesFor(
}
function parseOutcomes(content: string, expectedTargetIds: readonly string[]): Map<string, ParsedOutcome> {
const trimmed = content.trim();
const fenced = /^```(?:json)?[ \t]*\r?\n([\s\S]*?)\r?\n```$/iu.exec(trimmed);
let parsed: unknown;
try {
parsed = JSON.parse(fenced?.[1] ?? trimmed);
} catch {
throw new InvalidModelJsonError();
}
const response = completionResponseSchema.safeParse(parsed);
if (!response.success) throw new InvalidModelSchemaError();
const outcomes = response.data.results;
const expected = new Set(expectedTargetIds);
if (outcomes.length !== expectedTargetIds.length || expected.size !== expectedTargetIds.length) {
throw new MissingModelTargetsError();
}
const mapped = new Map<string, ParsedOutcome>();
for (const outcome of outcomes) {
if (!expected.has(outcome.targetId) || mapped.has(outcome.targetId)) {
throw new MissingModelTargetsError();
const trimmed = content.trim();
const fenced = /^```(?:json)?[ \t]*\r?\n([\s\S]*?)\r?\n```$/iu.exec(trimmed);
const outcomes = completionResponseSchema.parse(JSON.parse(fenced?.[1] ?? trimmed)).results;
const expected = new Set(expectedTargetIds);
if (outcomes.length !== expectedTargetIds.length || expected.size !== expectedTargetIds.length) {
throw new InvalidModelOutcomeError();
}
mapped.set(outcome.targetId, outcome.outcome === "generated"
? { ...outcome, description: outcome.description.trim() }
: outcome);
const mapped = new Map<string, ParsedOutcome>();
for (const outcome of outcomes) {
if (!expected.has(outcome.targetId) || mapped.has(outcome.targetId)) {
throw new InvalidModelOutcomeError();
}
mapped.set(outcome.targetId, outcome.outcome === "generated"
? { ...outcome, description: outcome.description.trim() }
: outcome);
}
if (mapped.size !== expected.size) throw new InvalidModelOutcomeError();
return mapped;
} catch (error) {
if (error instanceof InvalidModelOutcomeError) throw error;
throw new InvalidModelOutcomeError();
}
if (mapped.size !== expected.size) throw new MissingModelTargetsError();
return mapped;
}
function safeFailure(error: unknown): string {
if (error instanceof DescriptionGenerationFailureStreakError) return error.message;
const failure = error instanceof DescriptionGenerationBatchError ? error.failure : error;
if (failure instanceof ModelCompletionProviderError) return "The model provider request failed.";
if (failure instanceof InvalidModelJsonError) return "The model response was not valid JSON.";
if (failure instanceof InvalidModelSchemaError) {
return "The model response did not match the required schema.";
}
if (failure instanceof MissingModelTargetsError) {
return "The model response was missing one or more requested targets.";
}
if (failure instanceof InvalidModelOutcomeError) return "The model response was invalid.";
return "Description generation failed.";
}
@@ -927,9 +907,6 @@ export class DescriptionGenerationWorker {
generated: 0,
nonGeneratable: 0,
failed: 0,
inputTokens: 0,
cacheReadTokens: 0,
outputTokens: 0,
consecutiveTechnicalFailures: 0,
};
await this.processTargets(run, database, plan.columnTargets, model, counters, signal);
@@ -996,41 +973,20 @@ export class DescriptionGenerationWorker {
);
}
throwIfCancelled(signal);
const expectedTargetIds = batch.map((target) => (
target.kind === "column" ? target.column.id : target.table.id
));
const messages = messagesFor(database, batch, run.language, sourceSamples);
let outcomes: Map<string, ParsedOutcome> | undefined;
let terminalFailure: unknown;
for (let attempt = 1; attempt <= MAX_COMPLETION_ATTEMPTS_PER_BATCH; attempt += 1) {
try {
const completion = await this.completer.complete({ model, messages, signal });
const result: ModelCompletionResult = typeof completion === "string"
? { content: completion, usage: { input: 0, cacheRead: 0, output: 0 } }
: completion;
counters.inputTokens += result.usage.input;
counters.cacheReadTokens += result.usage.cacheRead;
counters.outputTokens += result.usage.output;
throwIfCancelled(signal);
outcomes = parseOutcomes(result.content, expectedTargetIds);
break;
} catch (error) {
if (error instanceof ModelCompletionCancelledError) throw error;
terminalFailure = error;
if (attempt < MAX_COMPLETION_ATTEMPTS_PER_BATCH) {
await this.appendEvent(
run.id,
"warning",
`${safeFailure(error)} Retrying batch (attempt ${attempt + 1} of ${MAX_COMPLETION_ATTEMPTS_PER_BATCH}).`,
);
}
}
}
if (!outcomes) {
const batchError = new DescriptionGenerationBatchError(
terminalFailure,
batch.map(failureTarget),
);
let outcomes: Map<string, ParsedOutcome>;
try {
const content = await this.completer.complete({
model,
messages: messagesFor(database, batch, run.language, sourceSamples),
signal,
});
throwIfCancelled(signal);
outcomes = parseOutcomes(content, batch.map((target) => (
target.kind === "column" ? target.column.id : target.table.id
)));
} catch (error) {
if (error instanceof ModelCompletionCancelledError) throw error;
const batchError = new DescriptionGenerationBatchError(error, batch.map(failureTarget));
counters.processed += batch.length;
counters.failed += batch.length;
counters.consecutiveTechnicalFailures += 1;
@@ -1053,10 +1009,7 @@ export class DescriptionGenerationWorker {
const targetId = target.kind === "column" ? target.column.id : target.table.id;
const outcome = outcomes.get(targetId);
if (!outcome) {
throw new DescriptionGenerationBatchError(
new MissingModelTargetsError(),
[failureTarget(target)],
);
throw new DescriptionGenerationBatchError(new InvalidModelOutcomeError(), [failureTarget(target)]);
}
const generatedDescription = outcome.outcome === "generated"
? outcome.description
+43 -132
View File
@@ -1,8 +1,5 @@
import { readFile } from "node:fs/promises";
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
import type { CatalogPostgresAccess } from "./postgres-access.js";
import { CATALOG_SECRET_IDS } from "./secrets.js";
import { CatalogConnectorError, type WorkspaceDatabase } from "./types.js";
import type { WorkspaceDatabase } from "./types.js";
const MAX_SOURCE_ROWS = 5;
const MAX_REPRESENTATIVE_VALUES = 5;
@@ -82,82 +79,15 @@ function distinctKey(value: Exclude<DescriptionSourceSampleValue, null>): string
return `${typeof value}:${String(value)}`;
}
function columnsFor(target: DescriptionSourceSamplingTarget): string[] {
return [...new Set(target.columnNames)].slice(0, MAX_SOURCE_COLUMNS_PER_TARGET);
}
function normalizedSample(
target: DescriptionSourceSamplingTarget,
columnNames: readonly string[],
sourceRows: readonly Record<string, unknown>[],
): DescriptionTargetSourceSample {
const rows = sourceRows.slice(0, MAX_SOURCE_ROWS).map((row) => ({
fields: columnNames.flatMap((name) => {
const value = normalizeValue(row[name]);
return value === undefined ? [] : [{ name, value }];
}),
}));
const valuesByColumn = new Map<string, Exclude<DescriptionSourceSampleValue, null>[]>();
const seenByColumn = new Map<string, Set<string>>();
let representativeValueCount = 0;
for (const row of rows) {
for (const field of row.fields) {
if (representativeValueCount === MAX_REPRESENTATIVE_VALUES) break;
if (field.value === null) continue;
const seen = seenByColumn.get(field.name) ?? new Set<string>();
const key = distinctKey(field.value);
if (seen.has(key)) continue;
seen.add(key);
seenByColumn.set(field.name, seen);
const values = valuesByColumn.get(field.name) ?? [];
values.push(field.value);
valuesByColumn.set(field.name, values);
representativeValueCount += 1;
}
if (representativeValueCount === MAX_REPRESENTATIVE_VALUES) break;
}
return {
targetId: target.targetId,
tableName: target.tableName,
rows,
representativeValues: columnNames.flatMap((column) => {
const values = valuesByColumn.get(column);
return values && values.length > 0 ? [{ column, values }] : [];
}),
};
}
function samplingSql(
database: WorkspaceDatabase,
target: DescriptionSourceSamplingTarget,
columns: readonly string[],
): string {
const projections = columns.map((columnName) => {
const identifier = quoteIdentifier(columnName);
return `LEFT((${identifier})::text, ${MAX_SOURCE_VALUE_BYTES}) AS ${identifier}`;
});
return [
`SELECT ${projections.join(", ")}`,
`FROM ${quoteIdentifier(database.schema)}.${quoteIdentifier(target.tableName)}`,
`LIMIT ${MAX_SOURCE_ROWS}`,
].join(" ");
}
/** Bounded source sampler that follows the database's PostgreSQL-wire or REST binding. */
export class ConcreteDescriptionSourceSampler implements DescriptionSourceSampler {
constructor(
private readonly access: CatalogPostgresAccess,
private readonly secretStore?: Pick<WorkspaceSecretStore, "materialize">,
) {}
/** PostgreSQL-wire sampler. REST bindings remain unsupported by CatalogPostgresAccess. */
export class PostgresDescriptionSourceSampler implements DescriptionSourceSampler {
constructor(private readonly access: CatalogPostgresAccess) {}
async sample(
database: WorkspaceDatabase,
targets: readonly DescriptionSourceSamplingTarget[],
signal: AbortSignal,
): Promise<readonly DescriptionTargetSourceSample[]> {
if (database.binding.transport === "rest_api") {
return await this.sampleRest(database, targets, signal);
}
const client = await this.access.connect(database, signal);
let transactionOpen = false;
try {
@@ -165,7 +95,7 @@ export class ConcreteDescriptionSourceSampler implements DescriptionSourceSample
transactionOpen = true;
const samples: DescriptionTargetSourceSample[] = [];
for (const target of targets) {
const columnNames = columnsFor(target);
const columnNames = [...new Set(target.columnNames)].slice(0, MAX_SOURCE_COLUMNS_PER_TARGET);
if (columnNames.length === 0) {
samples.push({
targetId: target.targetId,
@@ -185,7 +115,44 @@ export class ConcreteDescriptionSourceSampler implements DescriptionSourceSample
"LIMIT $2",
].join(" ");
const result = await client.query(sql, [MAX_SOURCE_VALUE_BYTES, MAX_SOURCE_ROWS]);
samples.push(normalizedSample(target, columnNames, result.rows));
const rows = result.rows.slice(0, MAX_SOURCE_ROWS).map((row) => ({
fields: columnNames.flatMap((name) => {
const value = normalizeValue(row[name]);
return value === undefined ? [] : [{ name, value }];
}),
}));
const valuesByColumn = new Map<
string,
Exclude<DescriptionSourceSampleValue, null>[]
>();
const seenByColumn = new Map<string, Set<string>>();
let representativeValueCount = 0;
for (const row of rows) {
for (const field of row.fields) {
if (representativeValueCount === MAX_REPRESENTATIVE_VALUES) break;
if (field.value === null) continue;
const seen = seenByColumn.get(field.name) ?? new Set<string>();
const key = distinctKey(field.value);
if (seen.has(key)) continue;
seen.add(key);
seenByColumn.set(field.name, seen);
const values = valuesByColumn.get(field.name) ?? [];
values.push(field.value);
valuesByColumn.set(field.name, values);
representativeValueCount += 1;
}
if (representativeValueCount === MAX_REPRESENTATIVE_VALUES) break;
}
const representativeValues = columnNames.flatMap((column) => {
const values = valuesByColumn.get(column);
return values && values.length > 0 ? [{ column, values }] : [];
});
samples.push({
targetId: target.targetId,
tableName: target.tableName,
rows,
representativeValues,
});
}
return samples;
} finally {
@@ -193,60 +160,4 @@ export class ConcreteDescriptionSourceSampler implements DescriptionSourceSample
await client.end().catch(() => undefined);
}
}
private async sampleRest(
database: WorkspaceDatabase,
targets: readonly DescriptionSourceSamplingTarget[],
signal: AbortSignal,
): Promise<readonly DescriptionTargetSourceSample[]> {
if (!this.secretStore) throw new CatalogConnectorError("REST source sampling is not configured");
const auth = database.binding.restAuth ?? "bearer";
const materialized = this.secretStore.materialize(
database.workspaceId,
auth === "none" ? [] : [CATALOG_SECRET_IDS.apiKey],
);
try {
const headers: Record<string, string> = { "content-type": "application/json" };
if (auth !== "none") {
const credentialFile = materialized.files.get(CATALOG_SECRET_IDS.apiKey);
if (!credentialFile) throw new CatalogConnectorError("REST API key is not configured");
const credential = (await readFile(credentialFile, "utf8")).trim();
if (auth === "bearer") headers.authorization = `Bearer ${credential}`;
else headers["x-api-key"] = credential;
}
const baseUrl = database.binding.baseUrl?.replace(/\/+$/, "");
if (!baseUrl) throw new CatalogConnectorError("Database binding is incomplete");
const samples: DescriptionTargetSourceSample[] = [];
for (const target of targets) {
const columnNames = columnsFor(target);
if (columnNames.length === 0) {
samples.push(normalizedSample(target, columnNames, []));
continue;
}
const response = await fetch(`${baseUrl}/rpc/run_query`, {
method: "POST",
headers,
body: JSON.stringify({ query_text: samplingSql(database, target, columnNames) }),
signal,
});
if (!response.ok) throw new CatalogConnectorError("REST source sampling failed");
const body: unknown = await response.json();
if (!Array.isArray(body)
|| body.some((row) => !row || typeof row !== "object" || Array.isArray(row))) {
throw new CatalogConnectorError("REST source sampling response is invalid");
}
samples.push(normalizedSample(
target,
columnNames,
body as Array<Record<string, unknown>>,
));
}
return samples;
} catch (error) {
if (error instanceof CatalogConnectorError) throw error;
throw new CatalogConnectorError("REST source sampling failed");
} finally {
materialized.release();
}
}
}
@@ -1,93 +0,0 @@
import type { CatalogRelationship, CatalogRepository } from "./types.js";
export interface EffectiveRelationshipSnapshotReader {
list(databaseId: string): Promise<CatalogRelationship[]>;
}
export interface EffectiveRelationshipSnapshotCoordinator {
run<T>(databaseId: string, operation: () => Promise<T>): Promise<T>;
}
export class EffectiveRelationshipSnapshotStaleError extends Error {}
interface EffectiveRelationship {
sourceTable: string;
sourceColumns: string[];
targetTable: string;
targetColumns: string[];
origin: CatalogRelationship["origin"];
}
interface EffectiveRelationshipSnapshot {
schemaVersion: 1;
workspaceId: string;
relationships: EffectiveRelationship[];
}
const originRank: Record<CatalogRelationship["origin"], number> = {
physical: 0,
manual: 1,
generated: 2,
};
function endpointKey(relationship: EffectiveRelationship): string {
return [
relationship.sourceTable,
relationship.sourceColumns.join("\u0000"),
relationship.targetTable,
relationship.targetColumns.join("\u0000"),
].join("\u0001");
}
function compareRelationships(left: EffectiveRelationship, right: EffectiveRelationship): number {
return endpointKey(left).localeCompare(endpointKey(right))
|| originRank[left.origin] - originRank[right.origin];
}
/**
* Adapter from the mutable Catalog model to the immutable relationship contract consumed by the
* harness. The returned JSON is a deterministic projection, never an authored second store.
*/
export class EffectiveRelationshipSnapshotProvider {
constructor(
private readonly repository: Pick<CatalogRepository, "get" | "getByWorkspace">,
private readonly relationships: EffectiveRelationshipSnapshotReader,
private readonly operations: EffectiveRelationshipSnapshotCoordinator,
) {}
async render(workspaceId: string): Promise<string | undefined> {
const database = await this.repository.getByWorkspace(workspaceId);
if (!database) return undefined;
return await this.operations.run(database.id, async () => {
const current = await this.repository.get(database.id);
if (!current || current.schemaSyncedVersion !== current.version) {
throw new EffectiveRelationshipSnapshotStaleError(
"effective relationship snapshot requires a current full schema synchronization",
);
}
const projected = (await this.relationships.list(database.id))
.filter((relationship) => relationship.status === "active")
.map((relationship): EffectiveRelationship => ({
sourceTable: relationship.sourceTableName,
sourceColumns: relationship.columns.map((column) => column.sourceColumnName),
targetTable: relationship.targetTableName,
targetColumns: relationship.columns.map((column) => column.targetColumnName),
origin: relationship.origin,
}))
.sort(compareRelationships);
const seen = new Set<string>();
const snapshot: EffectiveRelationshipSnapshot = {
schemaVersion: 1,
workspaceId,
relationships: projected.filter((relationship) => {
const key = endpointKey(relationship);
if (seen.has(key)) return false;
seen.add(key);
return true;
}),
};
return `${JSON.stringify(snapshot, null, 2)}\n`;
});
}
}
@@ -1,260 +0,0 @@
import type {
CatalogLogicalRelationship,
CatalogLogicalRelationshipCandidate,
CatalogLogicalRelationshipContext,
CatalogLogicalRelationshipEndpoint,
CatalogRelationship,
CatalogRepository,
} from "./types.js";
export class LogicalRelationshipDatabaseNotFoundError extends Error {}
export class LogicalRelationshipDuplicateError extends Error {}
export class LogicalRelationshipNotFoundError extends Error {}
export class LogicalRelationshipReadOnlyError extends Error {}
export class LogicalRelationshipSchemaStaleError extends Error {}
export class LogicalRelationshipTargetNotUniqueError extends Error {}
export class LogicalRelationshipTypeIncompatibleError extends Error {}
export class LogicalRelationshipColumnNotFoundError extends Error {
constructor(readonly field: "sourceColumnId" | "targetColumnId") {
super(`Catalog column '${field}' was not found`);
}
}
export interface RebuildGeneratedRelationshipsResult {
added: number;
alreadyPresent: number;
excluded: number;
ambiguous: number;
}
function identifierTokens(value: string): string[] {
return value
.replace(/([a-z0-9])([A-Z])/g, "$1_$2")
.toLowerCase()
.split(/[^a-z0-9]+/)
.filter(Boolean);
}
function singularWord(value: string): string {
if (value.length > 4 && value.endsWith("ies")) return `${value.slice(0, -3)}y`;
if (value.length > 4 && /(ches|shes|xes|zes|ses)$/.test(value)) return value.slice(0, -2);
if (value.length > 3 && value.endsWith("s") && !/(ss|us)$/.test(value)) return value.slice(0, -1);
return value;
}
function tableAliases(tableName: string): string[] {
const tokens = identifierTokens(tableName);
if (tokens.length === 0) return [];
const normalized = tokens.join("_");
const singular = [...tokens];
singular[singular.length - 1] = singularWord(singular[singular.length - 1]);
return [...new Set([normalized, singular.join("_")])];
}
const GENERIC_PRIMARY_KEY_NAMES = new Set(["id", "key", "code", "pk"]);
function nameMatches(
source: CatalogLogicalRelationshipEndpoint,
target: CatalogLogicalRelationshipEndpoint,
): boolean {
const sourceName = identifierTokens(source.columnName).join("_");
const targetName = identifierTokens(target.columnName).join("_");
if (!sourceName || !targetName) return false;
const expected = new Set<string>();
if (!GENERIC_PRIMARY_KEY_NAMES.has(targetName)) expected.add(targetName);
for (const alias of tableAliases(target.tableName)) {
expected.add(`${alias}_${targetName}`);
expected.add(`${alias.replaceAll("_", "")}${targetName.replaceAll("_", "")}`);
if (targetName === "id" || targetName === "pk") expected.add(alias);
}
return expected.has(sourceName);
}
function canonicalDataType(value: string): string {
const normalized = value.trim().toLowerCase().replace(/\s+/g, " ");
const arraySuffix = normalized.endsWith("[]") ? "[]" : "";
const base = arraySuffix ? normalized.slice(0, -2) : normalized;
const withoutModifier = base.replace(/\([^)]*\)/g, "").trim();
const aliases: Record<string, string> = {
int2: "smallint",
smallserial: "smallint",
int4: "integer",
int: "integer",
serial: "integer",
int8: "bigint",
bigserial: "bigint",
decimal: "numeric",
varchar: "text",
"character varying": "text",
bool: "boolean",
"timestamp without time zone": "timestamp",
"timestamp with time zone": "timestamptz",
"time without time zone": "time",
"time with time zone": "timetz",
};
return `${aliases[withoutModifier] ?? withoutModifier}${arraySuffix}`;
}
function typesCompatible(left: string, right: string): boolean {
return canonicalDataType(left) === canonicalDataType(right);
}
function pairKey(sourceColumnId: string, targetColumnId: string): string {
return `${sourceColumnId}\u0000${targetColumnId}`;
}
function relationshipPair(relationship: CatalogLogicalRelationship): CatalogLogicalRelationshipCandidate {
return {
sourceColumnId: relationship.columns[0].sourceColumnId,
targetColumnId: relationship.columns[0].targetColumnId,
};
}
function relationshipSortKey(relationship: CatalogRelationship): string {
const sourceColumns = relationship.columns.map((column) => column.sourceColumnName).join(",");
const targetColumns = relationship.columns.map((column) => column.targetColumnName).join(",");
return [
relationship.sourceTableName,
sourceColumns,
relationship.targetTableName,
targetColumns,
relationship.origin,
].join("\u0000");
}
export class CatalogLogicalRelationshipService {
constructor(private readonly repository: CatalogRepository) {}
async list(databaseId: string): Promise<CatalogRelationship[]> {
if (!(await this.repository.get(databaseId))) throw new LogicalRelationshipDatabaseNotFoundError();
const relationships: CatalogRelationship[] = [
...await this.repository.listRelationships(databaseId),
...await this.repository.listLogicalRelationships(databaseId),
];
return relationships.sort((left, right) => relationshipSortKey(left).localeCompare(relationshipSortKey(right)));
}
async addManual(
databaseId: string,
sourceColumnId: string,
targetColumnId: string,
): Promise<CatalogLogicalRelationship> {
const context = await this.requiredContext(databaseId);
const source = context.endpoints.find((endpoint) => endpoint.columnId === sourceColumnId);
if (!source) throw new LogicalRelationshipColumnNotFoundError("sourceColumnId");
const target = context.endpoints.find((endpoint) => endpoint.columnId === targetColumnId);
if (!target) throw new LogicalRelationshipColumnNotFoundError("targetColumnId");
if (sourceColumnId === targetColumnId
|| target.primaryKeyPosition === null
|| target.tablePrimaryKeyColumnCount !== 1) {
throw new LogicalRelationshipTargetNotUniqueError();
}
if (!typesCompatible(source.dataType, target.dataType)) {
throw new LogicalRelationshipTypeIncompatibleError();
}
const key = pairKey(sourceColumnId, targetColumnId);
if (context.physicalPairs.some((pair) => pairKey(pair.sourceColumnId, pair.targetColumnId) === key)
|| context.logicalRelationships.some((relationship) => {
const pair = relationshipPair(relationship);
return pairKey(pair.sourceColumnId, pair.targetColumnId) === key;
})) throw new LogicalRelationshipDuplicateError();
const created = await this.repository.insertLogicalRelationship(
databaseId,
sourceColumnId,
targetColumnId,
false,
);
if (!created) throw new LogicalRelationshipDuplicateError();
return created;
}
async rebuildGenerated(databaseId: string): Promise<RebuildGeneratedRelationshipsResult> {
const context = await this.requiredContext(databaseId);
const targets = context.endpoints.filter((endpoint) => (
endpoint.primaryKeyPosition !== null && endpoint.tablePrimaryKeyColumnCount === 1
));
const physical = new Set(context.physicalPairs.map((pair) => pairKey(pair.sourceColumnId, pair.targetColumnId)));
const active = new Set<string>();
const excluded = new Set<string>();
for (const relationship of context.logicalRelationships) {
const pair = relationshipPair(relationship);
(relationship.status === "excluded" ? excluded : active)
.add(pairKey(pair.sourceColumnId, pair.targetColumnId));
}
const pending: CatalogLogicalRelationshipCandidate[] = [];
let alreadyPresent = 0;
let excludedCount = 0;
let ambiguous = 0;
const dimTimeTargets = targets.filter((target) => (
identifierTokens(target.tableName).join("_") === "dim_time"
));
const sources = context.endpoints.filter((endpoint) => (
endpoint.primaryKeyPosition === null || endpoint.tablePrimaryKeyColumnCount > 1
));
for (const source of sources) {
const sourceName = identifierTokens(source.columnName).join("_");
const isTimeKey = sourceName.endsWith("time_key")
&& identifierTokens(source.tableName).join("_") !== "dim_time";
const candidates = isTimeKey ? dimTimeTargets : targets;
const matches = candidates.filter((target) => (
target.columnId !== source.columnId
&& typesCompatible(source.dataType, target.dataType)
&& (isTimeKey || nameMatches(source, target))
));
if (matches.length > 1) {
ambiguous += 1;
continue;
}
if (matches.length === 0) continue;
const candidate = { sourceColumnId: source.columnId, targetColumnId: matches[0].columnId };
const key = pairKey(candidate.sourceColumnId, candidate.targetColumnId);
if (physical.has(key) || active.has(key)) {
alreadyPresent += 1;
} else if (excluded.has(key)) {
excludedCount += 1;
} else {
pending.push(candidate);
}
}
const added = await this.repository.insertGeneratedLogicalRelationships(databaseId, pending);
alreadyPresent += pending.length - added;
return { added, alreadyPresent, excluded: excludedCount, ambiguous };
}
async setStatus(
databaseId: string,
relationshipId: string,
status: CatalogLogicalRelationship["status"],
): Promise<CatalogLogicalRelationship> {
if (!(await this.repository.get(databaseId))) throw new LogicalRelationshipDatabaseNotFoundError();
const updated = await this.repository.setLogicalRelationshipStatus(databaseId, relationshipId, status);
if (updated) return updated;
await this.assertNotPhysical(databaseId, relationshipId);
throw new LogicalRelationshipNotFoundError();
}
async deletePermanently(databaseId: string, relationshipId: string): Promise<void> {
if (!(await this.repository.get(databaseId))) throw new LogicalRelationshipDatabaseNotFoundError();
if (await this.repository.deleteLogicalRelationship(databaseId, relationshipId)) return;
await this.assertNotPhysical(databaseId, relationshipId);
throw new LogicalRelationshipNotFoundError();
}
private async requiredContext(databaseId: string): Promise<CatalogLogicalRelationshipContext> {
const database = await this.repository.get(databaseId);
if (!database) throw new LogicalRelationshipDatabaseNotFoundError();
if (database.schemaSyncedVersion !== database.version) {
throw new LogicalRelationshipSchemaStaleError();
}
const context = await this.repository.getLogicalRelationshipContext(databaseId);
if (!context) throw new LogicalRelationshipDatabaseNotFoundError();
return context;
}
private async assertNotPhysical(databaseId: string, relationshipId: string): Promise<void> {
if ((await this.repository.listRelationships(databaseId)).some((relationship) => relationship.id === relationshipId)) {
throw new LogicalRelationshipReadOnlyError();
}
}
}
+5 -158
View File
@@ -14,10 +14,7 @@ import {
type CatalogDatabaseMetadataDeleteTarget,
type CatalogMetadataDeleteCounts,
type CatalogMetrics,
type CatalogLogicalRelationship,
type CatalogLogicalRelationshipCandidate,
type CatalogLogicalRelationshipContext,
type CatalogPhysicalRelationship,
type CatalogRelationship,
type CatalogSchemaDiff,
type CatalogSyncCounts,
type CatalogSyncEvent,
@@ -52,8 +49,7 @@ export class MemoryCatalogRepository implements CatalogRepository {
private readonly records = new Map<string, WorkspaceDatabase>();
private readonly tables = new Map<string, CatalogTable>();
private readonly columns = new Map<string, CatalogColumn>();
private readonly relationships = new Map<string, CatalogPhysicalRelationship>();
private readonly logicalRelationships = new Map<string, CatalogLogicalRelationship>();
private readonly relationships = new Map<string, CatalogRelationship>();
private readonly descriptionGenerationRuns = new Map<string, DescriptionGenerationRun>();
private readonly descriptionGenerationEvents = new Map<string, DescriptionGenerationEvent[]>();
private readonly sensitiveDataSuggestionRuns = new Map<string, SensitiveDataSuggestionRun>();
@@ -175,9 +171,6 @@ export class MemoryCatalogRepository implements CatalogRepository {
for (const [relationshipId, relationship] of this.relationships) {
if (relationship.databaseId === id) this.relationships.delete(relationshipId);
}
for (const [relationshipId, relationship] of this.logicalRelationships) {
if (relationship.databaseId === id) this.logicalRelationships.delete(relationshipId);
}
for (const [runId, run] of this.descriptionGenerationRuns) {
if (run.databaseId !== id) continue;
this.descriptionGenerationRuns.delete(runId);
@@ -335,10 +328,7 @@ export class MemoryCatalogRepository implements CatalogRepository {
processed: 0,
generated: 0,
nonGeneratable: 0,
failed: 0,
inputTokens: 0,
cacheReadTokens: 0,
outputTokens: 0,
failed: 0,
createdAt: now,
startedAt: null,
updatedAt: now,
@@ -447,10 +437,7 @@ export class MemoryCatalogRepository implements CatalogRepository {
status: "running",
total: 0,
suggestedSensitive: 0,
suggestedNonSensitive: 0,
inputTokens: 0,
cacheReadTokens: 0,
outputTokens: 0,
suggestedNonSensitive: 0,
createdAt: now,
startedAt: now,
updatedAt: now,
@@ -540,138 +527,12 @@ export class MemoryCatalogRepository implements CatalogRepository {
.map((event) => structuredClone(event));
}
async listRelationships(databaseId: string): Promise<CatalogPhysicalRelationship[]> {
async listRelationships(databaseId: string): Promise<CatalogRelationship[]> {
return [...this.relationships.values()].filter((relationship) => relationship.databaseId === databaseId)
.sort((a, b) => `${a.sourceTableName}.${a.constraintName}`.localeCompare(`${b.sourceTableName}.${b.constraintName}`))
.map((relationship) => structuredClone(relationship));
}
async listLogicalRelationships(databaseId: string): Promise<CatalogLogicalRelationship[]> {
return [...this.logicalRelationships.values()]
.filter((relationship) => relationship.databaseId === databaseId)
.sort((a, b) => {
const left = `${a.sourceTableName}.${a.columns[0].sourceColumnName}.${a.targetTableName}.${a.columns[0].targetColumnName}`;
const right = `${b.sourceTableName}.${b.columns[0].sourceColumnName}.${b.targetTableName}.${b.columns[0].targetColumnName}`;
return left.localeCompare(right);
})
.map((relationship) => structuredClone(relationship));
}
async getLogicalRelationshipContext(
databaseId: string,
): Promise<CatalogLogicalRelationshipContext | undefined> {
if (!this.records.has(databaseId)) return undefined;
const tables = [...this.tables.values()].filter((table) => table.databaseId === databaseId);
const tableById = new Map(tables.map((table) => [table.id, table]));
const columns = [...this.columns.values()].filter((column) => tableById.has(column.tableId));
const primaryKeyCounts = new Map<string, number>();
for (const column of columns) {
if (column.primaryKeyPosition !== null) {
primaryKeyCounts.set(column.tableId, (primaryKeyCounts.get(column.tableId) ?? 0) + 1);
}
}
return {
endpoints: columns.map((column) => ({
columnId: column.id,
columnName: column.name,
tableId: column.tableId,
tableName: tableById.get(column.tableId)!.name,
dataType: column.dataType,
primaryKeyPosition: column.primaryKeyPosition,
tablePrimaryKeyColumnCount: primaryKeyCounts.get(column.tableId) ?? 0,
})),
physicalPairs: [...this.relationships.values()]
.filter((relationship) => relationship.databaseId === databaseId)
.flatMap((relationship) => relationship.columns.map((column) => ({
sourceColumnId: column.sourceColumnId,
targetColumnId: column.targetColumnId,
}))),
logicalRelationships: await this.listLogicalRelationships(databaseId),
};
}
async insertLogicalRelationship(
databaseId: string,
sourceColumnId: string,
targetColumnId: string,
generated: boolean,
): Promise<CatalogLogicalRelationship | undefined> {
if ([...this.logicalRelationships.values()].some((relationship) => (
relationship.databaseId === databaseId
&& relationship.columns[0].sourceColumnId === sourceColumnId
&& relationship.columns[0].targetColumnId === targetColumnId
))) return undefined;
const sourceColumn = this.columns.get(sourceColumnId);
const targetColumn = this.columns.get(targetColumnId);
const sourceTable = sourceColumn ? this.tables.get(sourceColumn.tableId) : undefined;
const targetTable = targetColumn ? this.tables.get(targetColumn.tableId) : undefined;
if (!sourceColumn || !targetColumn || !sourceTable || !targetTable
|| sourceTable.databaseId !== databaseId || targetTable.databaseId !== databaseId
|| sourceColumnId === targetColumnId) return undefined;
const now = new Date().toISOString();
const relationship: CatalogLogicalRelationship = {
id: randomUUID(),
databaseId,
constraintName: null,
sourceTableId: sourceTable.id,
sourceTableName: sourceTable.name,
targetTableId: targetTable.id,
targetTableName: targetTable.name,
updateRule: null,
deleteRule: null,
deferrable: false,
initiallyDeferred: false,
columns: [{
position: 1,
sourceColumnId,
sourceColumnName: sourceColumn.name,
targetColumnId,
targetColumnName: targetColumn.name,
}],
lastSyncedDatabaseVersion: null,
lastSyncedAt: null,
createdAt: now,
updatedAt: now,
origin: generated ? "generated" : "manual",
status: "active",
};
this.logicalRelationships.set(relationship.id, relationship);
return structuredClone(relationship);
}
async insertGeneratedLogicalRelationships(
databaseId: string,
candidates: readonly CatalogLogicalRelationshipCandidate[],
): Promise<number> {
let added = 0;
for (const candidate of candidates) {
if (await this.insertLogicalRelationship(
databaseId,
candidate.sourceColumnId,
candidate.targetColumnId,
true,
)) added += 1;
}
return added;
}
async setLogicalRelationshipStatus(
databaseId: string,
relationshipId: string,
status: CatalogLogicalRelationship["status"],
): Promise<CatalogLogicalRelationship | undefined> {
const current = this.logicalRelationships.get(relationshipId);
if (!current || current.databaseId !== databaseId) return undefined;
const updated = { ...current, status, updatedAt: new Date().toISOString() };
this.logicalRelationships.set(relationshipId, updated);
return structuredClone(updated);
}
async deleteLogicalRelationship(databaseId: string, relationshipId: string): Promise<boolean> {
const current = this.logicalRelationships.get(relationshipId);
return Boolean(current?.databaseId === databaseId && this.logicalRelationships.delete(relationshipId));
}
async deleteDatabaseMetadata(
databaseIds: readonly string[],
target: CatalogDatabaseMetadataDeleteTarget,
@@ -713,12 +574,6 @@ export class MemoryCatalogRepository implements CatalogRepository {
const columns = [...this.columns.values()].filter((column) => selected.has(column.tableId));
const deletedColumnIds = new Set(columns.map((column) => column.id));
for (const column of columns) this.columns.delete(column.id);
for (const relationship of [...this.logicalRelationships.values()]) {
const pair = relationship.columns[0];
if (deletedColumnIds.has(pair.sourceColumnId) || deletedColumnIds.has(pair.targetColumnId)) {
this.logicalRelationships.delete(relationship.id);
}
}
for (const [relationshipId, relationship] of this.relationships) {
if (relationship.databaseId !== databaseId) continue;
this.relationships.set(relationshipId, {
@@ -1002,8 +857,6 @@ export class MemoryCatalogRepository implements CatalogRepository {
lastSyncedAt: now,
createdAt: current?.createdAt ?? now,
updatedAt: comparable === nextComparable ? (current?.updatedAt ?? now) : now,
origin: "physical",
status: "active",
});
if (!current) created += 1;
else if (comparable !== nextComparable) updated += 1;
@@ -1220,12 +1073,6 @@ export class MemoryCatalogRepository implements CatalogRepository {
this.relationships.delete(relationship.id);
}
}
for (const relationship of [...this.logicalRelationships.values()]) {
const pair = relationship.columns[0];
if (pair.sourceColumnId === columnId || pair.targetColumnId === columnId) {
this.logicalRelationships.delete(relationship.id);
}
}
}
private markCatalogIncomplete(databaseIds: readonly string[]): void {
-4
View File
@@ -10,8 +10,6 @@ import * as catalogRuntimeSequencePrivilegesMigration from "./migrations/004_cat
import * as descriptionGenerationRunsMigration from "./migrations/005_description_generation_runs.js";
import * as sensitiveDataFlagMigration from "./migrations/006_sensitive_data_flag.js";
import * as sensitiveDataSuggestionRunsMigration from "./migrations/007_sensitive_data_suggestion_runs.js";
import * as catalogLogicalRelationshipsMigration from "./migrations/008_catalog_logical_relationships.js";
import * as aiTokenUsageMigration from "./migrations/009_ai_token_usage.js";
const connectionString = process.env.THT_CATALOG_MIGRATOR_DATABASE_URL;
const host = process.env.THT_CATALOG_DB_HOST;
@@ -44,8 +42,6 @@ const provider: MigrationProvider = {
"005_description_generation_runs": descriptionGenerationRunsMigration,
"006_sensitive_data_flag": sensitiveDataFlagMigration,
"007_sensitive_data_suggestion_runs": sensitiveDataSuggestionRunsMigration,
"008_catalog_logical_relationships": catalogLogicalRelationshipsMigration,
"009_ai_token_usage": aiTokenUsageMigration,
};
},
};
@@ -1,35 +0,0 @@
import { type Kysely, sql } from "kysely";
import type { CatalogDatabase } from "../repository.js";
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.createTable("catalog_logical_relationships")
.addColumn("id", "uuid", (column) => column.primaryKey())
.addColumn("database_id", "uuid", (column) => column.notNull()
.references("workspace_databases.id").onDelete("cascade"))
.addColumn("source_column_id", "uuid", (column) => column.notNull()
.references("catalog_columns.id").onDelete("cascade"))
.addColumn("target_column_id", "uuid", (column) => column.notNull()
.references("catalog_columns.id").onDelete("cascade"))
.addColumn("generated", "boolean", (column) => column.notNull().defaultTo(false))
.addColumn("deleted_at", "timestamptz")
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addUniqueConstraint(
"catalog_logical_relationships_endpoint_key",
["database_id", "source_column_id", "target_column_id"],
)
.addCheckConstraint(
"catalog_logical_relationships_distinct_columns_check",
sql`source_column_id <> target_column_id`,
)
.execute();
await db.schema.createIndex("catalog_logical_relationships_database_deleted_idx")
.on("catalog_logical_relationships")
.columns(["database_id", "deleted_at"])
.execute();
}
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.dropTable("catalog_logical_relationships").execute();
}
@@ -1,20 +0,0 @@
import type { Kysely } from "kysely";
import type { CatalogDatabase } from "../repository.js";
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
for (const table of ["description_generation_runs", "sensitive_data_suggestion_runs"] as const) {
await db.schema.alterTable(table)
.addColumn("input_tokens", "integer", (col) => col.notNull().defaultTo(0))
.addColumn("cache_read_tokens", "integer", (col) => col.notNull().defaultTo(0))
.addColumn("output_tokens", "integer", (col) => col.notNull().defaultTo(0))
.execute();
}
}
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
for (const table of ["sensitive_data_suggestion_runs", "description_generation_runs"] as const) {
await db.schema.alterTable(table)
.dropColumn("input_tokens").dropColumn("cache_read_tokens").dropColumn("output_tokens")
.execute();
}
}
+5 -7
View File
@@ -4,7 +4,7 @@ import type { ResolvedMetadataGenerationModel } from "./metadata-generation-mode
const MAX_HELPER_OUTPUT_BYTES = 64 * 1024;
const helperOutputSchema = z.discriminatedUnion("ok", [
z.object({ ok: z.literal(true), content: z.string(), usage: z.object({ input: z.number().int().nonnegative(), cacheRead: z.number().int().nonnegative(), output: z.number().int().nonnegative() }).strict().optional() }).strict(),
z.object({ ok: z.literal(true), content: z.string() }).strict(),
z.object({ ok: z.literal(false), error: z.literal("provider_failure") }).strict(),
]);
@@ -18,12 +18,10 @@ export interface ModelCompletionRequest {
messages: readonly ModelCompletionMessage[];
signal: AbortSignal;
}
export interface ModelCompletionUsage { input: number; cacheRead: number; output: number; }
export interface ModelCompletionResult { content: string; usage: ModelCompletionUsage; }
/** The provider boundary used by Description Generation. */
export interface ModelCompleter {
complete(request: ModelCompletionRequest): Promise<string | ModelCompletionResult>;
complete(request: ModelCompletionRequest): Promise<string>;
}
export class ModelCompletionProviderError extends Error {
@@ -49,7 +47,7 @@ export class PythonModelCompleter implements ModelCompleter {
terminationGraceMs?: number;
}) {}
async complete(request: ModelCompletionRequest): Promise<ModelCompletionResult> {
async complete(request: ModelCompletionRequest): Promise<string> {
if (request.signal.aborted) throw new ModelCompletionCancelledError();
const payload = {
model: `${request.model.provider}/${request.model.model}`,
@@ -64,7 +62,7 @@ export class PythonModelCompleter implements ModelCompleter {
...(request.model.disableThinking === true ? { disable_thinking: true } : {}),
};
return await new Promise<ModelCompletionResult>((resolve, reject) => {
return await new Promise<string>((resolve, reject) => {
const child = spawn(
this.options.pythonExecutable,
["-m", this.options.helperModule ?? "tht.internal.litellm_completion"],
@@ -137,7 +135,7 @@ export class PythonModelCompleter implements ModelCompleter {
if (!output.ok) return fail();
settled = true;
cleanup();
resolve({ content: output.content, usage: output.usage ?? { input: 0, cacheRead: 0, output: 0 } });
resolve(output.content);
} catch {
fail();
}
+4 -188
View File
@@ -23,10 +23,7 @@ import {
type CatalogDatabaseMetadataDeleteTarget,
type CatalogMetadataDeleteCounts,
type CatalogMetrics,
type CatalogLogicalRelationship,
type CatalogLogicalRelationshipCandidate,
type CatalogLogicalRelationshipContext,
type CatalogPhysicalRelationship,
type CatalogRelationship,
type CatalogSchemaDiff,
type CatalogSyncCounts,
type CatalogSyncEvent,
@@ -148,17 +145,6 @@ interface CatalogRelationshipColumnTable {
targetColumnId: string;
}
interface CatalogLogicalRelationshipTable {
id: string;
databaseId: string;
sourceColumnId: string;
targetColumnId: string;
generated: Generated<boolean>;
deletedAt: Timestamp | null;
createdAt: Timestamp;
updatedAt: Timestamp;
}
interface DescriptionGenerationRunTable {
id: string;
databaseId: string;
@@ -171,9 +157,6 @@ interface DescriptionGenerationRunTable {
generated: number;
nonGeneratable: number;
failed: number;
inputTokens: number;
cacheReadTokens: number;
outputTokens: number;
createdAt: Timestamp;
startedAt: Timestamp | null;
updatedAt: Timestamp;
@@ -198,9 +181,6 @@ interface SensitiveDataSuggestionRunTable {
total: number;
suggestedSensitive: number;
suggestedNonSensitive: number;
inputTokens: number;
cacheReadTokens: number;
outputTokens: number;
createdAt: Timestamp;
startedAt: Timestamp;
updatedAt: Timestamp;
@@ -261,7 +241,6 @@ export interface CatalogDatabase {
catalogColumns: CatalogColumnTable;
catalogRelationships: CatalogRelationshipTable;
catalogRelationshipColumns: CatalogRelationshipColumnTable;
catalogLogicalRelationships: CatalogLogicalRelationshipTable;
descriptionGenerationRuns: DescriptionGenerationRunTable;
descriptionGenerationEvents: DescriptionGenerationEventTable;
sensitiveDataSuggestionRuns: SensitiveDataSuggestionRunTable;
@@ -822,9 +801,6 @@ export class KyselyCatalogRepository implements CatalogRepository {
generated: 0,
nonGeneratable: 0,
failed: 0,
inputTokens: 0,
cacheReadTokens: 0,
outputTokens: 0,
startedAt: null,
finishedAt: null,
errorSummary: null,
@@ -952,9 +928,6 @@ export class KyselyCatalogRepository implements CatalogRepository {
total: 0,
suggestedSensitive: 0,
suggestedNonSensitive: 0,
inputTokens: 0,
cacheReadTokens: 0,
outputTokens: 0,
finishedAt: null,
errorSummary: null,
}).returningAll().executeTakeFirstOrThrow();
@@ -1049,7 +1022,7 @@ export class KyselyCatalogRepository implements CatalogRepository {
return rows.map(serializeSensitiveDataSuggestionEvent);
}
async listRelationships(databaseId: string): Promise<CatalogPhysicalRelationship[]> {
async listRelationships(databaseId: string): Promise<CatalogRelationship[]> {
const rows = await this.db.selectFrom("catalogRelationships as relationship")
.innerJoin("catalogTables as sourceTable", "sourceTable.id", "relationship.sourceTableId")
.innerJoin("catalogTables as targetTable", "targetTable.id", "relationship.targetTableId")
@@ -1076,7 +1049,7 @@ export class KyselyCatalogRepository implements CatalogRepository {
.where("pair.relationshipId", "in", rows.map((row) => row.id))
.orderBy("pair.relationshipId").orderBy("pair.position")
.execute();
const byRelationship = new Map<string, CatalogPhysicalRelationship["columns"]>();
const byRelationship = new Map<string, CatalogRelationship["columns"]>();
for (const pair of pairs) {
const items = byRelationship.get(pair.relationshipId) ?? [];
items.push(pair);
@@ -1088,160 +1061,9 @@ export class KyselyCatalogRepository implements CatalogRepository {
lastSyncedAt: row.lastSyncedAt === null ? null : new Date(row.lastSyncedAt).toISOString(),
createdAt: new Date(row.createdAt).toISOString(),
updatedAt: new Date(row.updatedAt).toISOString(),
origin: "physical" as const,
status: "active" as const,
}));
}
async listLogicalRelationships(databaseId: string): Promise<CatalogLogicalRelationship[]> {
const rows = await this.db.selectFrom("catalogLogicalRelationships as relationship")
.innerJoin("catalogColumns as sourceColumn", "sourceColumn.id", "relationship.sourceColumnId")
.innerJoin("catalogTables as sourceTable", "sourceTable.id", "sourceColumn.tableId")
.innerJoin("catalogColumns as targetColumn", "targetColumn.id", "relationship.targetColumnId")
.innerJoin("catalogTables as targetTable", "targetTable.id", "targetColumn.tableId")
.select([
"relationship.id", "relationship.databaseId", "relationship.generated",
"relationship.deletedAt", "relationship.createdAt", "relationship.updatedAt",
"sourceTable.id as sourceTableId", "sourceTable.name as sourceTableName",
"sourceColumn.id as sourceColumnId", "sourceColumn.name as sourceColumnName",
"targetTable.id as targetTableId", "targetTable.name as targetTableName",
"targetColumn.id as targetColumnId", "targetColumn.name as targetColumnName",
])
.where("relationship.databaseId", "=", databaseId)
.orderBy("sourceTable.name")
.orderBy("sourceColumn.name")
.orderBy("targetTable.name")
.orderBy("targetColumn.name")
.execute();
return rows.map((row) => ({
id: row.id,
databaseId: row.databaseId,
constraintName: null,
sourceTableId: row.sourceTableId,
sourceTableName: row.sourceTableName,
targetTableId: row.targetTableId,
targetTableName: row.targetTableName,
updateRule: null,
deleteRule: null,
deferrable: false,
initiallyDeferred: false,
columns: [{
position: 1,
sourceColumnId: row.sourceColumnId,
sourceColumnName: row.sourceColumnName,
targetColumnId: row.targetColumnId,
targetColumnName: row.targetColumnName,
}],
lastSyncedDatabaseVersion: null,
lastSyncedAt: null,
createdAt: new Date(row.createdAt).toISOString(),
updatedAt: new Date(row.updatedAt).toISOString(),
origin: row.generated ? "generated" : "manual",
status: row.deletedAt === null ? "active" : "excluded",
}));
}
async getLogicalRelationshipContext(
databaseId: string,
): Promise<CatalogLogicalRelationshipContext | undefined> {
if (!(await selectOne(this.db, databaseId))) return undefined;
const columns = await this.db.selectFrom("catalogColumns as column")
.innerJoin("catalogTables as table", "table.id", "column.tableId")
.select([
"column.id as columnId", "column.name as columnName", "column.dataType",
"column.primaryKeyPosition", "table.id as tableId", "table.name as tableName",
])
.where("table.databaseId", "=", databaseId)
.orderBy("table.name")
.orderBy("column.ordinalPosition")
.execute();
const primaryKeyCounts = new Map<string, number>();
for (const column of columns) {
if (column.primaryKeyPosition !== null) {
primaryKeyCounts.set(column.tableId, (primaryKeyCounts.get(column.tableId) ?? 0) + 1);
}
}
const physicalPairs = await this.db.selectFrom("catalogRelationshipColumns as pair")
.innerJoin("catalogRelationships as relationship", "relationship.id", "pair.relationshipId")
.select(["pair.sourceColumnId", "pair.targetColumnId"])
.where("relationship.databaseId", "=", databaseId)
.execute();
return {
endpoints: columns.map((column) => ({
...column,
tablePrimaryKeyColumnCount: primaryKeyCounts.get(column.tableId) ?? 0,
})),
physicalPairs,
logicalRelationships: await this.listLogicalRelationships(databaseId),
};
}
async insertLogicalRelationship(
databaseId: string,
sourceColumnId: string,
targetColumnId: string,
generated: boolean,
): Promise<CatalogLogicalRelationship | undefined> {
const id = randomUUID();
const inserted = await this.db.insertInto("catalogLogicalRelationships").values({
id, databaseId, sourceColumnId, targetColumnId, generated,
}).onConflict((conflict) => conflict
.columns(["databaseId", "sourceColumnId", "targetColumnId"])
.doNothing())
.returning("id")
.executeTakeFirst();
if (!inserted) return undefined;
return (await this.listLogicalRelationships(databaseId)).find((item) => item.id === id);
}
async insertGeneratedLogicalRelationships(
databaseId: string,
candidates: readonly CatalogLogicalRelationshipCandidate[],
): Promise<number> {
if (candidates.length === 0) return 0;
return await this.db.transaction().execute(async (trx) => {
let added = 0;
for (const candidate of candidates) {
const inserted = await trx.insertInto("catalogLogicalRelationships").values({
id: randomUUID(),
databaseId,
sourceColumnId: candidate.sourceColumnId,
targetColumnId: candidate.targetColumnId,
generated: true,
}).onConflict((conflict) => conflict
.columns(["databaseId", "sourceColumnId", "targetColumnId"])
.doNothing())
.returning("id")
.executeTakeFirst();
if (inserted) added += 1;
}
return added;
});
}
async setLogicalRelationshipStatus(
databaseId: string,
relationshipId: string,
status: CatalogLogicalRelationship["status"],
): Promise<CatalogLogicalRelationship | undefined> {
const updated = await this.db.updateTable("catalogLogicalRelationships")
.set({ deletedAt: status === "excluded" ? sql`now()` : null, updatedAt: sql`now()` })
.where("databaseId", "=", databaseId)
.where("id", "=", relationshipId)
.returning("id")
.executeTakeFirst();
if (!updated) return undefined;
return (await this.listLogicalRelationships(databaseId)).find((item) => item.id === relationshipId);
}
async deleteLogicalRelationship(databaseId: string, relationshipId: string): Promise<boolean> {
const result = await this.db.deleteFrom("catalogLogicalRelationships")
.where("databaseId", "=", databaseId)
.where("id", "=", relationshipId)
.executeTakeFirst();
return result.numDeletedRows > 0n;
}
async deleteDatabaseMetadata(
databaseIds: readonly string[],
target: CatalogDatabaseMetadataDeleteTarget,
@@ -1799,13 +1621,7 @@ export class UnavailableCatalogRepository implements CatalogRepository {
async updateSensitiveDataSuggestionRun(): Promise<SensitiveDataSuggestionRun | undefined> { return this.fail(); }
async appendSensitiveDataSuggestionEvent(): Promise<SensitiveDataSuggestionEvent> { return this.fail(); }
async listSensitiveDataSuggestionEvents(): Promise<SensitiveDataSuggestionEvent[]> { return this.fail(); }
async listRelationships(): Promise<CatalogPhysicalRelationship[]> { return this.fail(); }
async listLogicalRelationships(): Promise<CatalogLogicalRelationship[]> { return this.fail(); }
async getLogicalRelationshipContext(): Promise<CatalogLogicalRelationshipContext | undefined> { return this.fail(); }
async insertLogicalRelationship(): Promise<CatalogLogicalRelationship | undefined> { return this.fail(); }
async insertGeneratedLogicalRelationships(): Promise<number> { return this.fail(); }
async setLogicalRelationshipStatus(): Promise<CatalogLogicalRelationship | undefined> { return this.fail(); }
async deleteLogicalRelationship(): Promise<boolean> { return this.fail(); }
async listRelationships(): Promise<CatalogRelationship[]> { return this.fail(); }
async deleteDatabaseMetadata(): Promise<CatalogMetadataDeleteCounts | undefined> { return this.fail(); }
async deleteTableMetadata(): Promise<CatalogMetadataDeleteCounts | undefined> { return this.fail(); }
async planSchemaSync(): Promise<CatalogSchemaDiff> { return this.fail(); }
@@ -1,6 +1,6 @@
import { z } from "zod";
import type { MetadataGenerationModels } from "./metadata-generation-models.js";
import type { ModelCompleter, ModelCompletionMessage, ModelCompletionResult, ModelCompletionUsage } from "./model-completer.js";
import type { ModelCompleter, ModelCompletionMessage } from "./model-completer.js";
import type {
CatalogColumn,
CatalogRepository,
@@ -201,8 +201,6 @@ export class SensitiveDataSuggester {
targetIds: readonly string[],
signal: AbortSignal,
onPrepared?: (total: number) => void | Promise<void>,
onProgress?: (processed: number, suggestions: readonly SensitiveDataSuggestion[]) => void | Promise<void>,
onUsage?: (usage: ModelCompletionUsage) => void | Promise<void>,
): Promise<readonly SensitiveDataSuggestion[]> {
const database = await this.repository.get(databaseId);
if (!database) throw new SensitiveDataSuggestionTargetNotFoundError("database");
@@ -214,16 +212,11 @@ export class SensitiveDataSuggester {
for (const batch of batchesFor(database, columns)) {
let received: Map<string, { columnId: string; sensitive: boolean }> | undefined;
for (let attempt = 0; attempt < 2 && !received; attempt += 1) {
const completion = await this.completer.complete({
const content = await this.completer.complete({
model,
signal,
messages: [systemMessage, { role: "user", content: userContent(database, batch) }],
});
const result: ModelCompletionResult = typeof completion === "string"
? { content: completion, usage: { input: 0, cacheRead: 0, output: 0 } }
: completion;
await onUsage?.(result.usage);
const content = result.content;
try {
const parsed = responseSchema.parse(JSON.parse(content));
const expected = new Set(batch.map((column) => column.columnId));
@@ -247,7 +240,6 @@ export class SensitiveDataSuggester {
currentSensitive: column.currentSensitive,
sensitive: received!.get(column.columnId)!.sensitive,
})));
await onProgress?.(suggestions.length, suggestions.slice(-batch.length));
}
return suggestions;
}
@@ -10,7 +10,6 @@ import type {
SensitiveDataSuggestionRun,
SensitiveDataSuggestionScope,
} from "./types.js";
import type { ModelCompletionUsage } from "./model-completer.js";
const interruptedMessage = "Sensitive-field suggestion generation was interrupted by backend restart.";
const failedMessage = "Sensitive-field suggestion generation failed.";
@@ -74,31 +73,6 @@ export class SensitiveDataSuggestionRunner {
});
if (!prepared) throw new Error("Sensitive Data Suggestion Run disappeared");
},
async (processed, batch) => {
const suggestedSensitive = batch.filter((suggestion) => suggestion.sensitive).length;
const suggestedNonSensitive = batch.length - suggestedSensitive;
const current = await this.repository.getSensitiveDataSuggestionRun(started.id);
if (!current) throw new Error("Sensitive Data Suggestion Run disappeared");
const progress = await this.repository.updateSensitiveDataSuggestionRun(started.id, {
suggestedSensitive: current.suggestedSensitive + suggestedSensitive,
suggestedNonSensitive: current.suggestedNonSensitive + suggestedNonSensitive,
});
if (!progress) throw new Error("Sensitive Data Suggestion Run disappeared");
await this.repository.appendSensitiveDataSuggestionEvent(
started.id,
"info",
`Classified ${processed} of ${progress.total} columns.`,
);
},
async (usage: ModelCompletionUsage) => {
const current = await this.repository.getSensitiveDataSuggestionRun(started.id);
if (!current) throw new Error("Sensitive Data Suggestion Run disappeared");
await this.repository.updateSensitiveDataSuggestionRun(started.id, {
inputTokens: current.inputTokens + usage.input,
cacheReadTokens: current.cacheReadTokens + usage.cacheRead,
outputTokens: current.outputTokens + usage.output,
});
},
);
const suggestedSensitive = suggestions.filter((suggestion) => suggestion.sensitive).length;
const suggestedNonSensitive = suggestions.length - suggestedSensitive;
+2 -65
View File
@@ -2,7 +2,6 @@ import { buildInstallationContract } from "../workspaces/contracts.js";
import { resolveBinding } from "../workspaces/bindings.js";
import type { WorkspaceRegistry } from "../workspaces/registry.js";
import type { WorkspaceDescriptor } from "../workspaces/schema.js";
import { discoverWorkspaceSecretRequirements } from "../workspaces/secret-requirements.js";
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
import { createConcreteDiagnosticAdapters } from "../workspaces/diagnostics.js";
import { CatalogOperationCoordinator } from "./operation-coordinator.js";
@@ -26,21 +25,6 @@ export interface CatalogListItem extends Omit<WorkspaceDatabase, "id"> {
workspaceName: string;
workspaceDescription?: string;
workspaceAvailable: boolean;
workspaceRevision: { commit: string; blob: string } | null;
workspaceEvidence: {
sourceType: "filesystem" | "http" | "s3" | null;
state:
| "not_declared"
| "materialized_current_revision"
| "configuration_required"
| "configured_unverified"
| "workspace_unavailable";
};
runtimeBinding: {
transport: DatabaseBinding["transport"];
configurationState: "ready" | "configuration_required";
sessionTransportSupported: boolean;
} | null;
configured: boolean;
secrets: Record<CatalogSecretName, boolean>;
}
@@ -84,43 +68,6 @@ function secretState(store: WorkspaceSecretStore, workspaceId: string): Record<C
))) as Record<CatalogSecretName, boolean>;
}
function workspaceRuntimeState(
store: WorkspaceSecretStore,
workspace: WorkspaceDescriptor,
secretRoots: readonly string[],
): NonNullable<CatalogListItem["runtimeBinding"]> {
const requirements = discoverWorkspaceSecretRequirements(workspace, process.env);
const secretVariables = new Set(requirements.map(({ variable }) => variable));
const effective = resolveBinding(workspace, "DWH", process.env, secretRoots);
const configurationRequired = requirements.some(({ id, required }) => (
required && !store.has(workspace.workspace.id, id)
)) || effective.missing.some((variable) => !secretVariables.has(variable));
return {
transport: effective.transport,
configurationState: configurationRequired ? "configuration_required" : "ready",
sessionTransportSupported: effective.transport !== "ssh_tunnel",
};
}
function workspaceEvidenceState(
store: WorkspaceSecretStore,
workspace: WorkspaceDescriptor,
): CatalogListItem["workspaceEvidence"] {
const source = workspace.evidence?.source;
if (!source) return { sourceType: null, state: "not_declared" };
if (source.type === "filesystem") {
return { sourceType: source.type, state: "materialized_current_revision" };
}
const configurationRequired = discoverWorkspaceSecretRequirements(workspace, process.env)
.some(({ connector, id, required }) => (
connector === "evidence" && required && !store.has(workspace.workspace.id, id)
));
return {
sourceType: source.type,
state: configurationRequired ? "configuration_required" : "configured_unverified",
};
}
export class CatalogService {
private readonly adapters = createConcreteDiagnosticAdapters();
@@ -144,8 +91,7 @@ export class CatalogService {
const byWorkspace = new Map(configured.map((database) => [database.workspaceId, database]));
const active = await Promise.all(workspaces.map(async (entry) => {
const database = byWorkspace.get(entry.id);
const { workspace } = await this.registry.readPinned(entry.id, entry.revision.commit);
const runtimeDatabaseBinding = yamlBinding(workspace, this.secretRoots);
const { workspace } = await this.registry.read(entry.id);
const base = database ?? {
workspaceId: entry.id,
engine: "postgres" as const,
@@ -154,7 +100,7 @@ export class CatalogService {
version: 0,
createdAt: "",
updatedAt: "",
binding: runtimeDatabaseBinding,
binding: yamlBinding(workspace, this.secretRoots),
connectionStatus: "untested" as const,
};
return {
@@ -162,12 +108,6 @@ export class CatalogService {
workspaceName: entry.name,
workspaceDescription: entry.description,
workspaceAvailable: true,
workspaceRevision: {
commit: entry.revision.commit,
blob: entry.revision.blob,
},
workspaceEvidence: workspaceEvidenceState(this.secretStore, workspace),
runtimeBinding: workspaceRuntimeState(this.secretStore, workspace, this.secretRoots),
configured: database !== undefined,
secrets: secretState(this.secretStore, entry.id),
};
@@ -180,9 +120,6 @@ export class CatalogService {
workspaceName: database.workspaceId,
workspaceDescription: "Workspace is no longer present in the repository catalog.",
workspaceAvailable: false,
workspaceRevision: null,
workspaceEvidence: { sourceType: null, state: "workspace_unavailable" },
runtimeBinding: null,
configured: true,
secrets: secretState(this.secretStore, database.workspaceId),
}));
+2 -78
View File
@@ -128,7 +128,7 @@ export interface CatalogRelationshipColumn {
targetColumnName: string;
}
export interface CatalogPhysicalRelationship {
export interface CatalogRelationship {
id: string;
databaseId: string;
constraintName: string;
@@ -145,52 +145,6 @@ export interface CatalogPhysicalRelationship {
lastSyncedAt: string | null;
createdAt: string;
updatedAt: string;
origin: "physical";
status: "active";
}
export interface CatalogLogicalRelationship {
id: string;
databaseId: string;
constraintName: null;
sourceTableId: string;
sourceTableName: string;
targetTableId: string;
targetTableName: string;
updateRule: null;
deleteRule: null;
deferrable: false;
initiallyDeferred: false;
columns: [CatalogRelationshipColumn];
lastSyncedDatabaseVersion: null;
lastSyncedAt: null;
createdAt: string;
updatedAt: string;
origin: "generated" | "manual";
status: "active" | "excluded";
}
export type CatalogRelationship = CatalogPhysicalRelationship | CatalogLogicalRelationship;
export interface CatalogLogicalRelationshipEndpoint {
columnId: string;
columnName: string;
tableId: string;
tableName: string;
dataType: string;
primaryKeyPosition: number | null;
tablePrimaryKeyColumnCount: number;
}
export interface CatalogLogicalRelationshipContext {
endpoints: CatalogLogicalRelationshipEndpoint[];
physicalPairs: Array<{ sourceColumnId: string; targetColumnId: string }>;
logicalRelationships: CatalogLogicalRelationship[];
}
export interface CatalogLogicalRelationshipCandidate {
sourceColumnId: string;
targetColumnId: string;
}
export type CatalogDatabaseMetadataDeleteTarget = "tables" | "relationships";
@@ -234,9 +188,6 @@ export interface DescriptionGenerationRun {
generated: number;
nonGeneratable: number;
failed: number;
inputTokens: number;
cacheReadTokens: number;
outputTokens: number;
createdAt: string;
startedAt: string | null;
updatedAt: string;
@@ -253,9 +204,6 @@ export interface DescriptionGenerationRunUpdate {
startedAt?: string | null;
finishedAt?: string | null;
errorSummary?: string | null;
inputTokens?: number;
cacheReadTokens?: number;
outputTokens?: number;
}
export interface DescriptionGenerationEvent {
@@ -278,9 +226,6 @@ export interface SensitiveDataSuggestionRun {
total: number;
suggestedSensitive: number;
suggestedNonSensitive: number;
inputTokens: number;
cacheReadTokens: number;
outputTokens: number;
createdAt: string;
startedAt: string;
updatedAt: string;
@@ -295,9 +240,6 @@ export interface SensitiveDataSuggestionRunUpdate {
suggestedNonSensitive?: number;
finishedAt?: string | null;
errorSummary?: string | null;
inputTokens?: number;
cacheReadTokens?: number;
outputTokens?: number;
}
export interface SensitiveDataSuggestionEvent {
@@ -514,25 +456,7 @@ export interface CatalogRepository {
runId: string,
afterSequence?: number,
): Promise<SensitiveDataSuggestionEvent[]>;
listRelationships(databaseId: string): Promise<CatalogPhysicalRelationship[]>;
listLogicalRelationships(databaseId: string): Promise<CatalogLogicalRelationship[]>;
getLogicalRelationshipContext(databaseId: string): Promise<CatalogLogicalRelationshipContext | undefined>;
insertLogicalRelationship(
databaseId: string,
sourceColumnId: string,
targetColumnId: string,
generated: boolean,
): Promise<CatalogLogicalRelationship | undefined>;
insertGeneratedLogicalRelationships(
databaseId: string,
candidates: readonly CatalogLogicalRelationshipCandidate[],
): Promise<number>;
setLogicalRelationshipStatus(
databaseId: string,
relationshipId: string,
status: CatalogLogicalRelationship["status"],
): Promise<CatalogLogicalRelationship | undefined>;
deleteLogicalRelationship(databaseId: string, relationshipId: string): Promise<boolean>;
listRelationships(databaseId: string): Promise<CatalogRelationship[]>;
deleteDatabaseMetadata(
databaseIds: readonly string[],
target: CatalogDatabaseMetadataDeleteTarget,
@@ -101,9 +101,6 @@ function publicRun(run: DescriptionGenerationRun) {
generated: run.generated,
nonGeneratable: run.nonGeneratable,
failed: run.failed,
inputTokens: run.inputTokens,
cacheReadTokens: run.cacheReadTokens,
outputTokens: run.outputTokens,
createdAt: run.createdAt,
startedAt: run.startedAt,
updatedAt: run.updatedAt,
@@ -132,9 +129,6 @@ function publicSensitiveDataSuggestionRun(run: SensitiveDataSuggestionRun) {
total: run.total,
suggestedSensitive: run.suggestedSensitive,
suggestedNonSensitive: run.suggestedNonSensitive,
inputTokens: run.inputTokens,
cacheReadTokens: run.cacheReadTokens,
outputTokens: run.outputTokens,
createdAt: run.createdAt,
startedAt: run.startedAt,
updatedAt: run.updatedAt,
@@ -1,154 +0,0 @@
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { z } from "zod";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import {
CatalogLogicalRelationshipService,
LogicalRelationshipColumnNotFoundError,
LogicalRelationshipDatabaseNotFoundError,
LogicalRelationshipDuplicateError,
LogicalRelationshipNotFoundError,
LogicalRelationshipReadOnlyError,
LogicalRelationshipSchemaStaleError,
LogicalRelationshipTargetNotUniqueError,
LogicalRelationshipTypeIncompatibleError,
} from "../catalog/logical-relationship-service.js";
import type { CatalogOperationCoordinator } from "../catalog/operation-coordinator.js";
import { CatalogOperationInProgressError, CatalogUnavailableError } from "../catalog/types.js";
const idSchema = z.uuid();
const createSchema = z.object({
sourceColumnId: idSchema,
targetColumnId: idSchema,
}).strict();
const statusSchema = z.object({ status: z.enum(["active", "excluded"]) }).strict();
const emptySchema = z.object({}).strict();
function manage(request: FastifyRequest, reply: FastifyReply) {
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
}
function safeError(reply: FastifyReply, error: unknown) {
if (error instanceof CatalogUnavailableError) {
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
}
if (error instanceof CatalogOperationInProgressError) {
return reply.code(409).send({
code: "database_operation_in_progress",
message: "A database operation is already in progress.",
});
}
if (error instanceof LogicalRelationshipDatabaseNotFoundError) {
return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
}
if (error instanceof LogicalRelationshipColumnNotFoundError) {
return reply.code(404).send({
code: "column_not_found",
message: "Catalog column was not found.",
field: error.field,
});
}
if (error instanceof LogicalRelationshipNotFoundError) {
return reply.code(404).send({ code: "relationship_not_found", message: "Relationship was not found." });
}
if (error instanceof LogicalRelationshipDuplicateError) {
return reply.code(409).send({ code: "relationship_duplicate", message: "Relationship already exists." });
}
if (error instanceof LogicalRelationshipReadOnlyError) {
return reply.code(409).send({ code: "relationship_read_only", message: "Physical relationships are read-only." });
}
if (error instanceof LogicalRelationshipSchemaStaleError) {
return reply.code(409).send({
code: "relationship_schema_stale",
message: "Synchronize the current database schema before managing logical relationships.",
});
}
if (error instanceof LogicalRelationshipTargetNotUniqueError) {
return reply.code(422).send({
code: "relationship_target_not_unique",
message: "Target column must be the only primary-key column of its table.",
field: "targetColumnId",
});
}
if (error instanceof LogicalRelationshipTypeIncompatibleError) {
return reply.code(422).send({
code: "relationship_type_incompatible",
message: "Source and target column types are incompatible.",
field: "targetColumnId",
});
}
if (error instanceof z.ZodError) {
return reply.code(400).send({
code: "relationship_request_invalid",
message: "Relationship request is invalid.",
});
}
return reply.code(500).send({
code: "relationship_operation_failed",
message: "Relationship operation failed.",
});
}
export function catalogLogicalRelationshipRoutes(
app: FastifyInstance,
deps: {
service: CatalogLogicalRelationshipService;
operations: CatalogOperationCoordinator;
},
): void {
app.get("/catalog/databases/:databaseId/relationships", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
return await deps.service.list(databaseId);
} catch (error) { return safeError(reply, error); }
});
app.post("/catalog/databases/:databaseId/relationships", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
const input = createSchema.parse(request.body);
const relationship = await deps.operations.run(databaseId, async () => (
await deps.service.addManual(databaseId, input.sourceColumnId, input.targetColumnId)
));
return reply.code(201).send(relationship);
} catch (error) { return safeError(reply, error); }
});
app.post("/catalog/databases/:databaseId/relationships/rebuild-generated", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
emptySchema.parse(request.body ?? {});
return await deps.operations.run(databaseId, async () => (
await deps.service.rebuildGenerated(databaseId)
));
} catch (error) { return safeError(reply, error); }
});
app.patch("/catalog/databases/:databaseId/relationships/:relationshipId", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const params = request.params as { databaseId?: unknown; relationshipId?: unknown };
const databaseId = idSchema.parse(params.databaseId);
const relationshipId = idSchema.parse(params.relationshipId);
const input = statusSchema.parse(request.body);
return await deps.operations.run(databaseId, async () => (
await deps.service.setStatus(databaseId, relationshipId, input.status)
));
} catch (error) { return safeError(reply, error); }
});
app.delete("/catalog/databases/:databaseId/relationships/:relationshipId", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const params = request.params as { databaseId?: unknown; relationshipId?: unknown };
const databaseId = idSchema.parse(params.databaseId);
const relationshipId = idSchema.parse(params.relationshipId);
await deps.operations.run(databaseId, async () => {
await deps.service.deletePermanently(databaseId, relationshipId);
});
return reply.code(204).send();
} catch (error) { return safeError(reply, error); }
});
}
+11
View File
@@ -131,6 +131,17 @@ export function catalogSchemaRoutes(
} catch (error) { return safeError(reply, error); }
});
app.get("/catalog/databases/:databaseId/relationships", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
if (!(await deps.repository.get(databaseId))) {
return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
}
return await deps.repository.listRelationships(databaseId);
} catch (error) { return safeError(reply, error); }
});
app.post("/catalog/databases/metadata-cleanup", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
+8 -35
View File
@@ -11,7 +11,6 @@ import type { WorkspaceRegistry } from "../workspaces/registry.js";
import { validateOperationalWorkspace, type WorkspaceDescriptor } from "../workspaces/schema.js";
import type { MaintenanceBarrier } from "../runtime/maintenance-gate.js";
import { hasPermission, isPrincipalContext, requirePermission } from "../auth/authorization.js";
import type { EffectiveRelationshipSnapshotProvider } from "../catalog/effective-relationship-snapshot.js";
const BOOTSTRAP_FAILURE_MESSAGE =
"Session startup failed. Check configuration and connectivity, then Resume the session.";
@@ -41,8 +40,6 @@ export function sessionRoutes(
/** Fail-closed installation/runtime transport capability check. */
workspaceRuntimeSupport: (workspace: WorkspaceDescriptor) => boolean;
maintenanceBarrier: MaintenanceBarrier;
/** Optional only for narrow route-test stubs and installations without a Catalog database. */
effectiveRelationships?: EffectiveRelationshipSnapshotProvider;
},
) {
const lifecycleTails = new Map<string, Promise<void>>();
@@ -87,21 +84,11 @@ export function sessionRoutes(
isAdmin: hasPermission(principal, permission),
});
const optionsWithRuntimeConfig = async (
runner: any,
workspaceConfigPath: string | undefined,
workspaceId: string | undefined,
options: any,
) => {
if (!workspaceConfigPath || typeof runner.acquireWorkspaceRuntime !== "function") return options;
const effectiveRelationships = workspaceId && d.effectiveRelationships
? await d.effectiveRelationships.render(workspaceId)
: undefined;
return {
...options,
runtimeConfig: runner.acquireWorkspaceRuntime(workspaceConfigPath, effectiveRelationships),
};
};
const optionsWithRuntimeConfig = (runner: any, workspaceConfigPath: string | undefined, options: any) => (
workspaceConfigPath && typeof runner.acquireWorkspaceRuntime === "function"
? { ...options, runtimeConfig: runner.acquireWorkspaceRuntime(workspaceConfigPath) }
: options
);
const maintenanceReply = (reply: any) => reply.code(503).send({
code: "maintenance",
@@ -461,12 +448,7 @@ export function sessionRoutes(
let runtimeOptions = options;
let rt: ReturnType<PiProcessManager["createFor"]> | undefined;
try {
runtimeOptions = await optionsWithRuntimeConfig(
runner,
workspaceConfigPath,
workspaceId,
options,
);
runtimeOptions = optionsWithRuntimeConfig(runner, workspaceConfigPath, options);
rt = d.mgr.createFor(id, runtimeOptions);
bindRuntime(id, rt, runner, workspaceConfigPath);
} catch (error) {
@@ -483,11 +465,7 @@ export function sessionRoutes(
info(id, "Session created");
bootstrap(
id, rt, runner, workspaceConfigPath, d.mgr.configure(rt, runtimeOptions),
runner.searchPack(
b.question,
id,
(runtimeOptions as any).runtimeConfig?.path ?? workspaceConfigPath,
),
runner.searchPack(b.question, id, workspaceConfigPath),
() => d.mgr.start(id, rt, runtimeOptions),
);
return { id };
@@ -661,12 +639,7 @@ export function sessionRoutes(
if (boundRuntimes.get(id) === current) boundRuntimes.delete(id);
d.mgr.teardownIfCurrent(id, current);
}
runtimeOptions = await optionsWithRuntimeConfig(
runner,
workspaceConfigPath,
saved.workspace_id,
options,
);
runtimeOptions = optionsWithRuntimeConfig(runner, workspaceConfigPath, options);
rt = d.mgr.createFor(id, runtimeOptions);
bindRuntime(id, rt, runner, workspaceConfigPath);
} catch {
+12 -27
View File
@@ -213,37 +213,23 @@ export class ThtRunner {
}
/** Render one immutable canonical registry revision into a backend-owned harness config. */
acquireWorkspaceRuntime(
workspaceConfigPath: string,
effectiveRelationships?: string,
): RuntimeConfigLease {
const effectiveRelationshipsPath = effectiveRelationships === undefined
? undefined
: this.createRuntimeSnapshot(effectiveRelationships);
let rendered: ReturnType<typeof renderWorkspaceRuntimeFromSnapshotPath>;
try {
rendered = renderWorkspaceRuntimeFromSnapshotPath({
snapshotPath: workspaceConfigPath,
harnessDir: this.cfg.harnessDir,
configPath: this.cfg.configPath,
dataRoot: this.cfg.dataRoot ?? (() => {
throw new Error("registry workspace runtime requires an absolute data root");
})(),
secretRoots: this.cfg.secretRoots ?? [],
semanticRuntime: this.cfg.semanticRuntime ?? DEFAULT_SEMANTIC_RUNTIME,
workspaceSecretStore: this.cfg.workspaceSecretStore,
effectiveRelationshipsPath,
});
} catch (error) {
if (effectiveRelationshipsPath) this.cleanupRuntimeSnapshot(effectiveRelationshipsPath);
throw error;
}
acquireWorkspaceRuntime(workspaceConfigPath: string): RuntimeConfigLease {
const rendered = renderWorkspaceRuntimeFromSnapshotPath({
snapshotPath: workspaceConfigPath,
harnessDir: this.cfg.harnessDir,
configPath: this.cfg.configPath,
dataRoot: this.cfg.dataRoot ?? (() => {
throw new Error("registry workspace runtime requires an absolute data root");
})(),
secretRoots: this.cfg.secretRoots ?? [],
semanticRuntime: this.cfg.semanticRuntime ?? DEFAULT_SEMANTIC_RUNTIME,
workspaceSecretStore: this.cfg.workspaceSecretStore,
});
let path: string;
try {
path = this.createRuntimeSnapshot(rendered.renderedConfig);
} catch (error) {
rendered.releaseSecrets();
if (effectiveRelationshipsPath) this.cleanupRuntimeSnapshot(effectiveRelationshipsPath);
throw error;
}
let released = false;
@@ -255,7 +241,6 @@ export class ThtRunner {
if (released) return;
released = true;
this.cleanupRuntimeSnapshot(path);
if (effectiveRelationshipsPath) this.cleanupRuntimeSnapshot(effectiveRelationshipsPath);
rendered.releaseSecrets();
},
};
+3 -24
View File
@@ -243,12 +243,7 @@ function readSnapshotWorkspace(snapshotPath: string): {
}
}
function runtimePaths(
dataRoot: string,
workspaceId: string,
workspaceRevision?: string,
effectiveRelationshipsPath?: string,
): RuntimePaths {
function runtimePaths(dataRoot: string, workspaceId: string, workspaceRevision?: string): RuntimePaths {
if (!isAbsolute(dataRoot)) throw new Error("registry workspace runtime requires an absolute data root");
const root = join(dataRoot, "sessions", workspaceId);
return {
@@ -259,9 +254,6 @@ function runtimePaths(
...(workspaceRevision === undefined
? {}
: { annotations_root: join(dataRoot, "sessions", workspaceId, "revisions", workspaceRevision, "artifacts") }),
...(effectiveRelationshipsPath === undefined
? {}
: { effective_relationships: effectiveRelationshipsPath }),
};
}
@@ -309,7 +301,6 @@ function renderWorkspaceRuntimeFromWorkspace(options: {
dataRoot: string;
secretRoots: readonly string[];
semanticRuntime: SemanticRuntimeConfig;
effectiveRelationshipsPath?: string;
workspaceSecretStore?: WorkspaceSecretStore;
}): RenderedWorkspaceRuntime {
const secretLease = options.workspaceSecretStore === undefined
@@ -334,12 +325,7 @@ function renderWorkspaceRuntimeFromWorkspace(options: {
workspaceId: options.workspaceId,
workspaceRevision: options.workspaceRevision,
revisionContentRoot: options.revisionContentRoot,
runtimePaths: runtimePaths(
options.dataRoot,
options.workspaceId,
options.workspaceRevision,
options.effectiveRelationshipsPath,
),
runtimePaths: runtimePaths(options.dataRoot, options.workspaceId, options.workspaceRevision),
installationOverlay: overlay,
bindings,
bindingDigest: stableBindingDigest(bindings),
@@ -348,12 +334,7 @@ function renderWorkspaceRuntimeFromWorkspace(options: {
renderedConfig: renderRuntimeConfig(
options.workspace,
bindings,
runtimePaths(
options.dataRoot,
options.workspaceId,
options.workspaceRevision,
options.effectiveRelationshipsPath,
),
runtimePaths(options.dataRoot, options.workspaceId, options.workspaceRevision),
context,
overlay,
options.semanticRuntime,
@@ -372,7 +353,6 @@ export function renderWorkspaceRuntimeFromSnapshotPath(options: {
dataRoot: string;
secretRoots: readonly string[];
semanticRuntime: SemanticRuntimeConfig;
effectiveRelationshipsPath?: string;
workspaceSecretStore?: WorkspaceSecretStore;
}): RenderedWorkspaceRuntime {
const snapshot = readSnapshotWorkspace(options.snapshotPath);
@@ -386,7 +366,6 @@ export function renderWorkspaceRuntimeFromSnapshotPath(options: {
dataRoot: options.dataRoot,
secretRoots: options.secretRoots,
semanticRuntime: options.semanticRuntime,
effectiveRelationshipsPath: options.effectiveRelationshipsPath,
workspaceSecretStore: options.workspaceSecretStore,
});
}
@@ -12,8 +12,6 @@ export interface RuntimePaths {
memory: string;
/** Revision-qualified root for curated FK annotations (P5); optional for legacy callers. */
annotations_root?: string;
/** Immutable Catalog projection used as the exclusive runtime relationship source. */
effective_relationships?: string;
}
export interface RuntimeIdentity {
+4 -63
View File
@@ -13,10 +13,7 @@ import type { WorkspaceRegistry, WorkspaceRevision } from "../src/workspaces/reg
import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
const roots: string[] = [];
afterEach(() => {
vi.unstubAllEnvs();
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
});
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
const workspace: WorkspaceDescriptor = {
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
@@ -39,7 +36,6 @@ function setup(
catalogOperationCoordinator?: CatalogOperationCoordinator;
catalogPostgresAccess?: CatalogPostgresAccess;
} = {},
workspaceDescriptor: WorkspaceDescriptor = workspace,
) {
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-secret-"));
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-secret-runtime-"));
@@ -49,8 +45,7 @@ function setup(
const registry = {
list: vi.fn(async () => [revision]),
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
read: vi.fn(async () => ({ workspace: workspaceDescriptor, revision })),
readPinned: vi.fn(async () => ({ workspace: workspaceDescriptor, workspaceConfigPath: revision.snapshotPath })),
read: vi.fn(async () => ({ workspace, revision })),
} as unknown as WorkspaceRegistry;
const app = buildApp(loadConfig({
THT_HARNESS_DIR: "/missing",
@@ -101,31 +96,10 @@ const fleetSnapshot: ObservedSchemaSnapshot = {
};
test("lists every YAML workspace and creates its one database configuration", async () => {
const { app, secretStore } = setup();
const { app } = setup();
const initial = await app.inject({ method: "GET", url: "/catalog/databases" });
expect(initial.statusCode).toBe(200);
expect(initial.json()).toMatchObject([{
workspaceId: "psd-clinical",
configured: false,
databaseName: "warehouse",
workspaceRevision: { commit: revision.commit, blob: revision.blob },
workspaceEvidence: { sourceType: null, state: "not_declared" },
runtimeBinding: {
transport: "postgres_direct",
configurationState: "configuration_required",
sessionTransportSupported: true,
},
}]);
vi.stubEnv("THT_WS_PSD_CLINICAL_DWH_HOST", "runtime-db.internal");
vi.stubEnv("THT_WS_PSD_CLINICAL_DWH_PORT", "5432");
vi.stubEnv("THT_WS_PSD_CLINICAL_DWH_USER", "runtime-reader");
vi.stubEnv("THT_WS_PSD_CLINICAL_DWH_TRANSPORT", "postgres_direct");
secretStore.putMany("psd-clinical", { "dwh.password": "runtime-password" });
const runtimeReady = await app.inject({ method: "GET", url: "/catalog/databases" });
expect(runtimeReady.json()).toMatchObject([{
runtimeBinding: { configurationState: "ready", sessionTransportSupported: true },
}]);
expect(initial.json()).toMatchObject([{ workspaceId: "psd-clinical", configured: false, databaseName: "warehouse" }]);
const created = await app.inject({ method: "POST", url: "/catalog/databases", payload: direct });
expect(created.statusCode).toBe(201);
@@ -136,39 +110,6 @@ test("lists every YAML workspace and creates its one database configuration", as
expect(listed.json()).toMatchObject([{ configured: true, binding: { transport: "postgres_direct", host: "db.internal" } }]);
});
test("projects remote Evidence credential state without conflating catalog secrets", async () => {
const evidenceWorkspace: WorkspaceDescriptor = {
...workspace,
evidence: {
schema_version: 2,
source: {
type: "http",
uris: ["https://evidence.example.test/guide.md"],
authentication: "signed_urls_file",
connect_timeout_ms: 5_000,
read_timeout_ms: 30_000,
max_bytes: 10 * 1024 * 1024,
max_redirects: 5,
allow_private_hosts: false,
max_cache_bytes: 64 * 1024 * 1024,
},
policy: { max_chunk_chars: 4_000, retain_published_generations: 3 },
},
};
const { app, secretStore } = setup({}, {}, evidenceWorkspace);
const missing = await app.inject({ method: "GET", url: "/catalog/databases" });
expect(missing.json()).toMatchObject([{
workspaceEvidence: { sourceType: "http", state: "configuration_required" },
}]);
secretStore.putMany("psd-clinical", { "evidence.signed_urls": "https://signed.example.test/evidence" });
const configured = await app.inject({ method: "GET", url: "/catalog/databases" });
expect(configured.json()).toMatchObject([{
workspaceEvidence: { sourceType: "http", state: "configured_unverified" },
}]);
});
test("lists orphaned records and takes the REST diagnostic path from workspace YAML", async () => {
const { app, repository } = setup();
await repository.create({
@@ -218,12 +218,6 @@ test("suggests sensitive flags from structural metadata without persisting them"
runId: responseBody.run.id,
sequence: 2,
level: "info",
message: "Classified 1 of 1 columns.",
},
{
runId: responseBody.run.id,
sequence: 3,
level: "info",
message: "Sensitive-field suggestion generation completed for 1 column.",
},
]);
@@ -677,7 +671,6 @@ test("generates one selected Catalog Column from a single JSON code fence", asyn
errorSummary: null,
});
expect(Object.keys(start.json()).sort()).toEqual([
"cacheReadTokens",
"createdAt",
"databaseId",
"errorSummary",
@@ -685,11 +678,9 @@ test("generates one selected Catalog Column from a single JSON code fence", asyn
"finishedAt",
"generated",
"id",
"inputTokens",
"language",
"modelId",
"nonGeneratable",
"outputTokens",
"processed",
"scope",
"startedAt",
@@ -1281,7 +1272,7 @@ test("Stop aborts source sampling before any model request", async () => {
test("an isolated exhausted technical batch failure allows completion with errors", async () => {
const modelCompleter: ModelCompleter = {
complete: vi.fn(async (request) => {
if (vi.mocked(modelCompleter.complete).mock.calls.length <= 2) {
if (vi.mocked(modelCompleter.complete).mock.calls.length === 1) {
throw new ModelCompletionProviderError();
}
const context = JSON.parse(request.messages[1]!.content.split("\n").slice(1).join("\n"));
@@ -1329,7 +1320,7 @@ test("an isolated exhausted technical batch failure allows completion with error
failed: 10,
errorSummary: "Description generation completed with errors.",
});
expect(modelCompleter.complete).toHaveBeenCalledTimes(3);
expect(modelCompleter.complete).toHaveBeenCalledTimes(2);
const updated = new Map(
(await repository.listColumns(database.id, table.id)).map((column) => [column.id, column]),
);
@@ -1355,10 +1346,10 @@ test("success resets the technical-failure streak and the third later failure st
const modelCompleter: ModelCompleter = {
complete: vi.fn(async (request) => {
const call = vi.mocked(modelCompleter.complete).mock.calls.length;
if ([1, 2, 4, 5, 6, 7, 8, 9].includes(call)) {
if ([1, 2, 4, 5, 6].includes(call)) {
throw Object.assign(new ModelCompletionProviderError(), { message: sensitiveDiagnostic });
}
if (call > 9) throw new Error("a later batch must not start");
if (call > 6) throw new Error("a later batch must not start");
const context = JSON.parse(request.messages[1]!.content.split("\n").slice(1).join("\n"));
return JSON.stringify({
results: context.targets.map((target: { targetId: string }) => ({
@@ -1398,25 +1389,25 @@ test("success resets the technical-failure streak and the third later failure st
expect(run).toMatchObject({
status: "failed",
total: 61,
processed: 50,
processed: 60,
generated: 10,
nonGeneratable: 0,
failed: 40,
failed: 50,
errorSummary: "Description generation stopped after three consecutive technical batch failures.",
});
expect(modelCompleter.complete).toHaveBeenCalledTimes(9);
expect(modelCompleter.complete).toHaveBeenCalledTimes(6);
const requests = vi.mocked(modelCompleter.complete).mock.calls.map(([request]) => request);
expect(new Set(requests.map((request) => request.signal)).size).toBe(1);
expect(new Set(requests.map((request) => request.model.id))).toEqual(new Set([configuredModel.id]));
const updated = new Map(
(await repository.listColumns(database.id, table.id)).map((column) => [column.id, column]),
);
targetIds.slice(10, 20).forEach((targetId) => {
targetIds.slice(20, 30).forEach((targetId) => {
expect(updated.get(targetId)?.generatedDescription).toBe("Successful reset batch.");
});
expect(updated.get(targetIds[50]!)?.generatedDescription).toBeNull();
expect(updated.get(targetIds[60]!)?.generatedDescription).toBeNull();
const events = await repository.listDescriptionGenerationEvents(run.id);
expect(events.filter((event) => event.message.includes("model provider request failed"))).toHaveLength(8);
expect(events.filter((event) => event.message.includes("model provider request failed"))).toHaveLength(5);
expect(events.at(-1)).toEqual(expect.objectContaining({
level: "error",
message: "Description generation stopped after three consecutive technical batch failures.",
@@ -1669,7 +1660,6 @@ test("Description Generation history is newest-first, bounded, and exposes only
expect(response.statusCode).toBe(200);
expect(response.json().map((run: { id: string }) => run.id)).toEqual(ids.slice(1).reverse());
expect(Object.keys(response.json()[0]).sort()).toEqual([
"cacheReadTokens",
"createdAt",
"databaseId",
"errorSummary",
@@ -1677,11 +1667,9 @@ test("Description Generation history is newest-first, bounded, and exposes only
"finishedAt",
"generated",
"id",
"inputTokens",
"language",
"modelId",
"nonGeneratable",
"outputTokens",
"processed",
"scope",
"startedAt",
@@ -1957,7 +1945,7 @@ test("retains completed batch writes when a later batch response is malformed",
});
const { run } = await waitForTerminalRun(app, start.json().id);
expect(modelCompleter.complete).toHaveBeenCalledTimes(3);
expect(modelCompleter.complete).toHaveBeenCalledTimes(2);
expect(run).toMatchObject({
status: "completed_with_errors",
total: 11,
@@ -1984,14 +1972,9 @@ test("retains completed batch writes when a later batch response is malformed",
expect(events.slice(2, 12).map((event) => event.message)).toEqual(
orderedIds.slice(0, 10).map((targetId) => `Generated description for Catalog Column ${targetId}.`),
);
expect(events.find((event) => event.level === "warning" && event.message.includes("Retrying batch"))).toEqual(
expect.objectContaining({
message: "The model response did not match the required schema. Retrying batch (attempt 2 of 2).",
}),
);
expect(events.find((event) => event.level === "error")).toEqual(expect.objectContaining({
level: "error",
message: `The model response did not match the required schema. Affected Catalog Column target: ${orderedIds[10]}.`,
message: `The model response was invalid. Affected Catalog Column target: ${orderedIds[10]}.`,
}));
} finally {
await app.close();
@@ -2203,7 +2186,7 @@ test("Generate Missing skips prior partial results and includes null, empty, and
const metadata = JSON.parse(request.messages[1]!.content.split("\n").slice(1).join("\n"));
if (mode === "partial") {
partialCall += 1;
if (partialCall === 2 || partialCall === 3) throw new ModelCompletionProviderError();
if (partialCall === 2) throw new ModelCompletionProviderError();
return JSON.stringify({
results: metadata.targets.map((target: { targetId: string }) => ({
targetId: target.targetId,
@@ -2529,58 +2512,6 @@ test("localizes valid non-generatable Catalog Column results in English", async
}
});
test("retries invalid JSON once and completes the batch when the second response is valid", async () => {
const modelCompleter: ModelCompleter = {
complete: vi.fn(async (request) => {
if (vi.mocked(modelCompleter.complete).mock.calls.length === 1) {
return { content: "not-json", usage: { input: 11, cacheRead: 3, output: 2 } };
}
const context = JSON.parse(request.messages[1]!.content.split("\n").slice(1).join("\n"));
return {
content: JSON.stringify({
results: context.targets.map((target: { targetId: string }) => ({
targetId: target.targetId,
outcome: "generated",
description: "Generated after the application retry.",
})),
}),
usage: { input: 7, cacheRead: 1, output: 5 },
};
}),
};
const { app, repository, database, table, column } = await setup(modelCompleter);
try {
const start = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: configuredModel.id, scope: "selected_columns", targetIds: [column.id] },
});
const { run } = await waitForTerminalRun(app, start.json().id);
expect(run).toMatchObject({
status: "completed",
processed: 1,
generated: 1,
failed: 0,
inputTokens: 18,
cacheReadTokens: 4,
outputTokens: 7,
});
expect(modelCompleter.complete).toHaveBeenCalledTimes(2);
expect(await repository.getColumn(database.id, table.id, column.id)).toMatchObject({
generatedDescription: "Generated after the application retry.",
});
expect(await repository.listDescriptionGenerationEvents(run.id)).toEqual(expect.arrayContaining([
expect.objectContaining({
level: "warning",
message: "The model response was not valid JSON. Retrying batch (attempt 2 of 2).",
}),
]));
} finally {
await app.close();
}
});
test("fails safely when the provider fails and redacts provider diagnostics", async () => {
const sensitiveDiagnostic = "test-provider-secret private prompt raw provider payload";
const modelCompleter: ModelCompleter = {
@@ -2597,7 +2528,6 @@ test("fails safely when the provider fails and redacts provider diagnostics", as
});
const { run } = await waitForTerminalRun(app, start.json().id);
expect(modelCompleter.complete).toHaveBeenCalledTimes(2);
expect(run).toMatchObject({
status: "completed_with_errors",
processed: 1,
@@ -2635,7 +2565,7 @@ test.each([
["duplicate mappings", (targetIds: readonly string[]) => ({ results: [
{ targetId: targetIds[0], outcome: "generated", description: "First valid value" },
{ targetId: targetIds[0], outcome: "non_generatable" },
] }), "The model response was missing one or more requested targets."],
] })],
["unknown mappings", (targetIds: readonly string[]) => ({ results: [
{ targetId: targetIds[0], outcome: "non_generatable" },
{
@@ -2643,15 +2573,15 @@ test.each([
outcome: "generated",
description: "Unknown target value",
},
] }), "The model response was missing one or more requested targets."],
] })],
["missing mappings", (targetIds: readonly string[]) => ({ results: [
{ targetId: targetIds[0], outcome: "generated", description: "Only one result" },
] }), "The model response was missing one or more requested targets."],
] })],
["malformed mappings", (targetIds: readonly string[]) => ({ results: [
{ targetId: targetIds[0], outcome: "generated", description: "First valid value" },
{ targetId: targetIds[1], outcome: "generated", description: " " },
] }), "The model response did not match the required schema."],
] as const)("rejects %s without applying any result from the batch", async (_name, responseFor, failureMessage) => {
] })],
] as const)("rejects %s without applying any result from the batch", async (_name, responseFor) => {
let selectedColumnIds: string[] = [];
const modelCompleter: ModelCompleter = {
complete: vi.fn(async () => JSON.stringify(responseFor(selectedColumnIds))),
@@ -2695,7 +2625,6 @@ test.each([
});
const { run } = await waitForTerminalRun(app, start.json().id);
expect(modelCompleter.complete).toHaveBeenCalledTimes(2);
expect(run).toMatchObject({
status: "completed_with_errors",
processed: 2,
@@ -2715,7 +2644,7 @@ test.each([
expect((await repository.listDescriptionGenerationEvents(run.id)).find((event) => event.level === "error")).toEqual(
expect.objectContaining({
level: "error",
message: `${failureMessage} Affected Catalog Column targets: ${selectedColumnIds.join(", ")}.`,
message: `The model response was invalid. Affected Catalog Column targets: ${selectedColumnIds.join(", ")}.`,
}),
);
} finally {
@@ -13,7 +13,6 @@ import { up as upSchemaSync } from "../src/catalog/migrations/003_catalog_schema
import { up as upDescriptionGeneration } from "../src/catalog/migrations/005_description_generation_runs.js";
import { up as upSensitiveDataFlag } from "../src/catalog/migrations/006_sensitive_data_flag.js";
import { up as upSensitiveSuggestionRuns } from "../src/catalog/migrations/007_sensitive_data_suggestion_runs.js";
import { up as upAiTokenUsage } from "../src/catalog/migrations/009_ai_token_usage.js";
import { KyselyCatalogRepository, type CatalogDatabase } from "../src/catalog/repository.js";
import { loadConfig } from "../src/config.js";
import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
@@ -49,7 +48,6 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
await upSensitiveDataFlag(db);
await upDescriptionGeneration(db);
await upSensitiveSuggestionRuns(db);
await upAiTokenUsage(db);
const repository = new KyselyCatalogRepository(db);
const database = await repository.create({
workspaceId: "psd-clinical",
@@ -126,7 +124,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
description: "Elenco dei pazienti e dei loro dati clinici.",
}] });
}
if (call === 5) {
if (call === 4) {
return JSON.stringify({ results: [
{
targetId: birthDate.id,
@@ -140,14 +138,14 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
},
] });
}
if (call === 6) {
if (call === 5) {
return JSON.stringify({ results: [{
targetId: table.id,
outcome: "generated",
description: "Descrizione rigenerata della tabella pazienti.",
}] });
}
if (call === 7) {
if (call === 6) {
return JSON.stringify({ results: [{
targetId: birthDate.id,
outcome: "generated",
@@ -342,7 +340,7 @@ test.skipIf(!dockerAvailable)("Fastify persists Description Generation success a
expect(await repository.getColumn(database.id, table.id, birthDate.id)).toMatchObject({
generatedDescription: "Descrizione recuperata della data di nascita.",
});
expect(modelCompleter.complete).toHaveBeenCalledTimes(7);
expect(modelCompleter.complete).toHaveBeenCalledTimes(6);
expect(JSON.stringify(vi.mocked(modelCompleter.complete).mock.calls)).toContain(persistedSampleSecret);
const runIds = [
@@ -1,9 +1,6 @@
import { expect, test, vi } from "vitest";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
ConcreteDescriptionSourceSampler,
PostgresDescriptionSourceSampler,
type DescriptionSourceSamplingTarget,
} from "../src/catalog/description-source-sampler.js";
import type {
@@ -11,8 +8,6 @@ import type {
CatalogPostgresAccess,
} from "../src/catalog/postgres-access.js";
import type { WorkspaceDatabase } from "../src/catalog/types.js";
import type { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
import { CATALOG_SECRET_IDS } from "../src/catalog/secrets.js";
const database: WorkspaceDatabase = {
id: "11111111-1111-4111-8111-111111111111",
@@ -56,7 +51,7 @@ test("samples at most five source rows and five distinct non-null examples in a
const access: CatalogPostgresAccess = {
connect: vi.fn(async () => ({ query, end }) as CatalogDatabaseClient),
};
const sampler = new ConcreteDescriptionSourceSampler(access);
const sampler = new PostgresDescriptionSourceSampler(access);
const controller = new AbortController();
const samples = await sampler.sample(database, [target], controller.signal);
@@ -97,71 +92,13 @@ test("samples at most five source rows and five distinct non-null examples in a
expect(end).toHaveBeenCalledOnce();
});
test("samples source rows through the configured REST run_query binding", async () => {
const root = mkdtempSync(join(tmpdir(), "tht-source-rest-"));
const credentialFile = join(root, "api-key");
writeFileSync(credentialFile, "test-api-key\n", { mode: 0o600 });
const release = vi.fn();
const secretStore = {
materialize: vi.fn(() => ({
files: new Map([[CATALOG_SECRET_IDS.apiKey, credentialFile]]),
release,
})),
} as unknown as WorkspaceSecretStore;
const fetchMock = vi.fn(async () => new Response(JSON.stringify([
{ 'status"code': "active", ward: null },
{ 'status"code': "pending", ward: "A" },
]), { status: 200, headers: { "content-type": "application/json" } }));
vi.stubGlobal("fetch", fetchMock);
const access: CatalogPostgresAccess = {
connect: vi.fn(async () => { throw new Error("PostgreSQL access must not be used"); }),
};
const sampler = new ConcreteDescriptionSourceSampler(access, secretStore);
const restDatabase: WorkspaceDatabase = {
...database,
binding: {
transport: "rest_api",
baseUrl: "https://dwh.example.test/root/",
restPath: "/health",
restAuth: "x-api-key",
},
};
try {
await expect(sampler.sample(restDatabase, [target], new AbortController().signal)).resolves.toEqual([{
targetId: target.targetId,
tableName: target.tableName,
rows: [
{ fields: [{ name: 'status"code', value: "active" }, { name: "ward", value: null }] },
{ fields: [{ name: 'status"code', value: "pending" }, { name: "ward", value: "A" }] },
],
representativeValues: [
{ column: 'status"code', values: ["active", "pending"] },
{ column: "ward", values: ["A"] },
],
}]);
expect(access.connect).not.toHaveBeenCalled();
expect(fetchMock).toHaveBeenCalledWith("https://dwh.example.test/root/rpc/run_query", expect.objectContaining({
method: "POST",
headers: { "content-type": "application/json", "x-api-key": "test-api-key" },
body: JSON.stringify({
query_text: 'SELECT LEFT(("status""code")::text, 256) AS "status""code", LEFT(("ward")::text, 256) AS "ward" FROM "clinical""data"."patient""facts" LIMIT 5',
}),
}));
expect(release).toHaveBeenCalledOnce();
} finally {
vi.unstubAllGlobals();
rmSync(root, { recursive: true, force: true });
}
});
test("does not issue a SELECT when a protected target has no source columns", async () => {
const query = vi.fn(async () => ({ rows: [] }));
const end = vi.fn(async () => undefined);
const access: CatalogPostgresAccess = {
connect: vi.fn(async () => ({ query, end }) as CatalogDatabaseClient),
};
const sampler = new ConcreteDescriptionSourceSampler(access);
const sampler = new PostgresDescriptionSourceSampler(access);
const samples = await sampler.sample(database, [{
targetId: target.targetId,
@@ -192,7 +129,7 @@ test("rolls back and closes the source connection when sampling fails", async ()
const access: CatalogPostgresAccess = {
connect: vi.fn(async () => ({ query, end }) as CatalogDatabaseClient),
};
const sampler = new ConcreteDescriptionSourceSampler(access);
const sampler = new PostgresDescriptionSourceSampler(access);
const controller = new AbortController();
await expect(sampler.sample(database, [target], controller.signal)).rejects.toThrow();
@@ -1,207 +0,0 @@
import { expect, test } from "vitest";
import {
CatalogLogicalRelationshipService,
LogicalRelationshipDuplicateError,
LogicalRelationshipSchemaStaleError,
LogicalRelationshipTargetNotUniqueError,
LogicalRelationshipTypeIncompatibleError,
} from "../src/catalog/logical-relationship-service.js";
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
import type { ObservedSchemaSnapshot } from "../src/catalog/types.js";
const column = (
tableName: string,
name: string,
ordinalPosition: number,
dataType: string,
primaryKeyPosition: number | null,
) => ({
tableName, name, ordinalPosition, dataType, primaryKeyPosition,
isNullable: false, defaultExpression: null, sourceComment: null,
});
function schema(
tableNames: string[],
columns: ObservedSchemaSnapshot["columns"],
relationships: ObservedSchemaSnapshot["relationships"] = [],
): ObservedSchemaSnapshot {
return {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: tableNames.map((name) => ({ name, sourceComment: null })),
columns,
relationships,
};
}
async function setup(snapshot: ObservedSchemaSnapshot) {
const repository = new MemoryCatalogRepository();
const database = await repository.create({
workspaceId: "relationships", engine: "postgres", databaseName: "warehouse", schema: "public",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
});
await repository.applySchemaSync(database.id, database.version, "all", [], snapshot);
const service = new CatalogLogicalRelationshipService(repository);
const context = (await repository.getLogicalRelationshipContext(database.id))!;
const endpoint = (tableName: string, columnName: string) => context.endpoints.find((item) => (
item.tableName === tableName && item.columnName === columnName
))!;
return { repository, database, service, endpoint };
}
test("keeps excluded relationships across rebuild and recreates hard-deleted relationships", async () => {
const { database, service, endpoint } = await setup(schema(
["users", "orders"],
[column("users", "id", 1, "bigint", 1), column("orders", "id", 1, "bigint", 1),
column("orders", "user_id", 2, "bigint", null)],
));
const source = endpoint("orders", "user_id");
const target = endpoint("users", "id");
const manual = await service.addManual(database.id, source.columnId, target.columnId);
expect(manual).toMatchObject({ origin: "manual", status: "active" });
expect(await service.setStatus(database.id, manual.id, "excluded"))
.toMatchObject({ status: "excluded" });
await expect(service.rebuildGenerated(database.id)).resolves.toEqual({
added: 0, alreadyPresent: 0, excluded: 1, ambiguous: 0,
});
expect((await service.list(database.id)).filter((item) => item.origin !== "physical"))
.toMatchObject([{ id: manual.id, origin: "manual", status: "excluded" }]);
await service.deletePermanently(database.id, manual.id);
await expect(service.rebuildGenerated(database.id)).resolves.toEqual({
added: 1, alreadyPresent: 0, excluded: 0, ambiguous: 0,
});
expect((await service.list(database.id)).filter((item) => item.origin !== "physical"))
.toMatchObject([{ origin: "generated", status: "active" }]);
});
test("normalizes snake, kebab, and camel names without fuzzy matching", async () => {
const { database, service } = await setup(schema(
["users", "events"],
[column("users", "id", 1, "bigint", 1), column("events", "id", 1, "bigint", 1),
column("events", "user_id", 2, "bigint", null),
column("events", "user-id", 3, "bigint", null),
column("events", "userId", 4, "bigint", null),
column("events", "userid", 5, "bigint", null),
column("events", "unrelated", 6, "bigint", null)],
));
await expect(service.rebuildGenerated(database.id)).resolves.toEqual({
added: 4, alreadyPresent: 0, excluded: 0, ambiguous: 0,
});
});
test("skips generic bare primary-key names while retaining table-qualified matches", async () => {
const { database, service } = await setup(schema(
["users", "accounts", "events"],
[column("users", "id", 1, "bigint", 1), column("accounts", "id", 1, "bigint", 1),
column("events", "event_key", 1, "bigint", 1), column("events", "id", 2, "bigint", null),
column("events", "user_id", 3, "bigint", null)],
));
await expect(service.rebuildGenerated(database.id)).resolves.toEqual({
added: 1, alreadyPresent: 0, excluded: 0, ambiguous: 0,
});
});
test("infers foreign keys from composite-primary-key sources", async () => {
const { database, service } = await setup(schema(
["users", "groups", "memberships"],
[column("users", "id", 1, "bigint", 1), column("groups", "id", 1, "bigint", 1),
column("memberships", "user_id", 1, "bigint", 1),
column("memberships", "group_id", 2, "bigint", 2)],
));
await expect(service.rebuildGenerated(database.id)).resolves.toEqual({
added: 2, alreadyPresent: 0, excluded: 0, ambiguous: 0,
});
});
test("maps time-key columns to the single primary key of dim_time", async () => {
const { database, service } = await setup(schema(
["dim_time", "admissions"],
[column("dim_time", "day_key", 1, "integer", 1),
column("admissions", "id", 1, "bigint", 1),
column("admissions", "admission_time_key", 2, "integer", null),
column("admissions", "discharge_time_key", 3, "integer", null)],
));
await expect(service.rebuildGenerated(database.id)).resolves.toEqual({
added: 2, alreadyPresent: 0, excluded: 0, ambiguous: 0,
});
});
test("skips ambiguous targets and physical foreign-key pairs", async () => {
const ambiguous = await setup(schema(
["user", "users", "events"],
[column("user", "id", 1, "bigint", 1), column("users", "id", 1, "bigint", 1),
column("events", "id", 1, "bigint", 1), column("events", "user_id", 2, "bigint", null)],
));
await expect(ambiguous.service.rebuildGenerated(ambiguous.database.id)).resolves.toEqual({
added: 0, alreadyPresent: 0, excluded: 0, ambiguous: 1,
});
const physical = await setup(schema(
["users", "orders"],
[column("users", "id", 1, "bigint", 1), column("orders", "id", 1, "bigint", 1),
column("orders", "user_id", 2, "bigint", null)],
[{
constraintName: "orders_user_id_fkey", sourceTableName: "orders", targetTableName: "users",
updateRule: "NO ACTION", deleteRule: "NO ACTION", deferrable: false, initiallyDeferred: false,
columns: [{ position: 1, sourceColumnName: "user_id", targetColumnName: "id" }],
}],
));
await expect(physical.service.rebuildGenerated(physical.database.id)).resolves.toEqual({
added: 0, alreadyPresent: 1, excluded: 0, ambiguous: 0,
});
await expect(physical.service.addManual(
physical.database.id,
physical.endpoint("orders", "user_id").columnId,
physical.endpoint("users", "id").columnId,
)).rejects.toBeInstanceOf(LogicalRelationshipDuplicateError);
});
test("validates target uniqueness and canonical type compatibility for manual relationships", async () => {
const { database, service, endpoint } = await setup(schema(
["users", "composite", "events"],
[column("users", "id", 1, "bigint", 1),
column("composite", "left_id", 1, "bigint", 1), column("composite", "right_id", 2, "bigint", 2),
column("events", "id", 1, "bigint", 1), column("events", "user_id", 2, "integer", null),
column("events", "composite_id", 3, "bigint", null)],
));
await expect(service.addManual(
database.id, endpoint("events", "composite_id").columnId, endpoint("composite", "left_id").columnId,
)).rejects.toBeInstanceOf(LogicalRelationshipTargetNotUniqueError);
await expect(service.addManual(
database.id, endpoint("events", "user_id").columnId, endpoint("users", "id").columnId,
)).rejects.toBeInstanceOf(LogicalRelationshipTypeIncompatibleError);
});
test("rebuild is additive when an existing generated relationship stops matching", async () => {
const initial = schema(
["users", "orders"],
[column("users", "id", 1, "bigint", 1), column("orders", "id", 1, "bigint", 1),
column("orders", "user_id", 2, "bigint", null)],
);
const { repository, database, service } = await setup(initial);
await service.rebuildGenerated(database.id);
const changed = structuredClone(initial);
changed.columns.find((item) => item.tableName === "orders" && item.name === "user_id")!.dataType = "text";
await repository.applySchemaSync(database.id, database.version, "columns", [], changed);
await expect(service.rebuildGenerated(database.id)).resolves.toEqual({
added: 0, alreadyPresent: 0, excluded: 0, ambiguous: 0,
});
expect((await service.list(database.id)).filter((item) => item.origin === "generated")).toHaveLength(1);
});
test("refuses inference until the current database version has a full schema sync", async () => {
const repository = new MemoryCatalogRepository();
const database = await repository.create({
workspaceId: "unsynced-relationships",
engine: "postgres",
databaseName: "warehouse",
schema: "public",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
});
const service = new CatalogLogicalRelationshipService(repository);
await expect(service.rebuildGenerated(database.id))
.rejects.toBeInstanceOf(LogicalRelationshipSchemaStaleError);
});
@@ -12,8 +12,6 @@ import { up as upRuntimeSequencePrivileges } from "../src/catalog/migrations/004
import { up as upDescriptionGeneration } from "../src/catalog/migrations/005_description_generation_runs.js";
import { up as upSensitiveDataFlag } from "../src/catalog/migrations/006_sensitive_data_flag.js";
import { up as upSensitiveSuggestionRuns } from "../src/catalog/migrations/007_sensitive_data_suggestion_runs.js";
import { up as upLogicalRelationships } from "../src/catalog/migrations/008_catalog_logical_relationships.js";
import { up as upAiTokenUsage } from "../src/catalog/migrations/009_ai_token_usage.js";
const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
@@ -28,10 +26,6 @@ test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per wo
await upTables(db);
await upSchemaSync(db);
await upSensitiveDataFlag(db);
await upLogicalRelationships(db);
await upDescriptionGeneration(db);
await upSensitiveSuggestionRuns(db);
await upAiTokenUsage(db);
await sql`CREATE ROLE thothii_catalog_runtime`.execute(db);
await upRuntimeSequencePrivileges(db);
const sequencePrivilege = await sql<{ allowed: boolean }>`
@@ -208,7 +202,6 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository performs scoped metadata cl
await upTables(db);
await upSchemaSync(db);
await upSensitiveDataFlag(db);
await upLogicalRelationships(db);
const repository = new KyselyCatalogRepository(db);
const database = await repository.create({
workspaceId: "cleanup-test",
@@ -287,7 +280,6 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository atomically consolidates sel
await upTables(db);
await upSchemaSync(db);
await upSensitiveDataFlag(db);
await upLogicalRelationships(db);
const repository = new KyselyCatalogRepository(db);
const database = await repository.create({
workspaceId: "consolidation-test",
@@ -387,10 +379,8 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
await upTables(db);
await upSchemaSync(db);
await upSensitiveDataFlag(db);
await upLogicalRelationships(db);
await upDescriptionGeneration(db);
await upSensitiveSuggestionRuns(db);
await upAiTokenUsage(db);
const repository = new KyselyCatalogRepository(db);
const firstDatabase = await repository.create({
workspaceId: "generation-one",
@@ -620,63 +610,3 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository persists description and se
await container.stop();
}
}, 60_000);
test.skipIf(!dockerAvailable)("PostgreSQL repository persists logical relationship lifecycle and tombstones", async () => {
const container = await new PostgreSqlContainer("postgres:17.6-bookworm").start();
const db = new Kysely<CatalogDatabase>({
dialect: new PostgresDialect({ pool: new Pool({ connectionString: container.getConnectionUri() }) }),
plugins: [new CamelCasePlugin()],
});
try {
await upDatabases(db);
await upTables(db);
await upSchemaSync(db);
await upSensitiveDataFlag(db);
await upLogicalRelationships(db);
const repository = new KyselyCatalogRepository(db);
const database = await repository.create({
workspaceId: "logical-relationships",
engine: "postgres",
databaseName: "warehouse",
schema: "public",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
});
await repository.applySchemaSync(database.id, database.version, "all", [], {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: [{ name: "users", sourceComment: null }, { name: "orders", sourceComment: null }],
columns: [
{ tableName: "users", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
{ tableName: "orders", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
{ tableName: "orders", name: "user_id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
],
relationships: [],
});
const context = (await repository.getLogicalRelationshipContext(database.id))!;
const source = context.endpoints.find((item) => item.tableName === "orders" && item.columnName === "user_id")!;
const target = context.endpoints.find((item) => item.tableName === "users" && item.columnName === "id")!;
const created = await repository.insertLogicalRelationship(database.id, source.columnId, target.columnId, false);
expect(created).toMatchObject({ origin: "manual", status: "active" });
await expect(repository.insertLogicalRelationship(database.id, source.columnId, target.columnId, true))
.resolves.toBeUndefined();
expect(await repository.setLogicalRelationshipStatus(database.id, created!.id, "excluded"))
.toMatchObject({ status: "excluded" });
await expect(repository.insertGeneratedLogicalRelationships(database.id, [{
sourceColumnId: source.columnId, targetColumnId: target.columnId,
}])).resolves.toBe(0);
await expect(repository.deleteLogicalRelationship(database.id, created!.id)).resolves.toBe(true);
await expect(repository.insertGeneratedLogicalRelationships(database.id, [{
sourceColumnId: source.columnId, targetColumnId: target.columnId,
}])).resolves.toBe(1);
expect(await repository.listLogicalRelationships(database.id))
.toMatchObject([{ origin: "generated", status: "active" }]);
await db.deleteFrom("catalogColumns").where("id", "=", source.columnId).execute();
expect(await repository.listLogicalRelationships(database.id)).toEqual([]);
} finally {
await db.destroy();
await container.stop();
}
}, 60_000);
-123
View File
@@ -122,7 +122,6 @@ async function setup(env: Record<string, string> = {}) {
list: vi.fn(async () => [revision]),
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
read: vi.fn(async () => ({ workspace, revision })),
readPinned: vi.fn(async () => ({ workspace, workspaceConfigPath: revision.snapshotPath })),
} as unknown as WorkspaceRegistry;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test", ...env }), {
thtRunner: {} as never,
@@ -606,125 +605,3 @@ test("rejects a missing database without partially cleaning valid selections", a
expect(response.statusCode).toBe(404);
expect(await repository.listTables(database.id)).toHaveLength(2);
});
test("serves one relationship map and supports the manual relationship lifecycle", async () => {
const { app, repository, database } = await setup();
await seedCatalog(repository, database);
const tables = await repository.listTables(database.id);
const patients = tables.find((table) => table.name === "patients")!;
const visits = tables.find((table) => table.name === "visits")!;
const patientId = (await repository.listColumns(database.id, patients.id)).find((item) => item.name === "id")!;
const visitId = (await repository.listColumns(database.id, visits.id)).find((item) => item.name === "id")!;
const created = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/relationships`,
payload: { sourceColumnId: visitId.id, targetColumnId: patientId.id },
});
expect(created.statusCode).toBe(201);
expect(created.json()).toMatchObject({ origin: "manual", status: "active", constraintName: null });
const listed = (await app.inject({
method: "GET", url: `/catalog/databases/${database.id}/relationships`,
})).json();
expect(listed.map((item: { origin: string }) => item.origin).sort()).toEqual(["manual", "physical"]);
const relationshipId = created.json().id;
const excluded = await app.inject({
method: "PATCH",
url: `/catalog/databases/${database.id}/relationships/${relationshipId}`,
payload: { status: "excluded" },
});
expect(excluded.statusCode).toBe(200);
expect(excluded.json()).toMatchObject({ origin: "manual", status: "excluded" });
const restored = await app.inject({
method: "PATCH",
url: `/catalog/databases/${database.id}/relationships/${relationshipId}`,
payload: { status: "active" },
});
expect(restored.json()).toMatchObject({ status: "active" });
expect((await app.inject({
method: "DELETE", url: `/catalog/databases/${database.id}/relationships/${relationshipId}`,
})).statusCode).toBe(204);
});
test("rejects generated inference while the Catalog schema is not current", async () => {
const { app, database } = await setup();
const response = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/relationships/rebuild-generated`,
});
expect(response.statusCode).toBe(409);
expect(response.json()).toEqual({
code: "relationship_schema_stale",
message: "Synchronize the current database schema before managing logical relationships.",
});
});
test("rebuilds generated relationships and returns the exact summary", async () => {
const { app, repository, database } = await setup();
const observed = snapshot();
observed.relationships = [];
await seedCatalog(repository, database, observed);
const first = await app.inject({
method: "POST", url: `/catalog/databases/${database.id}/relationships/rebuild-generated`,
});
expect(first.statusCode).toBe(200);
expect(first.json()).toEqual({ added: 1, alreadyPresent: 0, excluded: 0, ambiguous: 0 });
const generated = (await repository.listLogicalRelationships(database.id))[0]!;
await app.inject({
method: "PATCH",
url: `/catalog/databases/${database.id}/relationships/${generated.id}`,
payload: { status: "excluded" },
});
const second = await app.inject({
method: "POST", url: `/catalog/databases/${database.id}/relationships/rebuild-generated`,
});
expect(second.json()).toEqual({ added: 0, alreadyPresent: 0, excluded: 1, ambiguous: 0 });
});
test("validates relationship requests and keeps physical relationships read-only", async () => {
const { app, repository, database } = await setup();
await seedCatalog(repository, database);
const physical = (await repository.listRelationships(database.id))[0]!;
const invalid = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/relationships`,
payload: { sourceColumnId: physical.columns[0].sourceColumnId, targetColumnId: physical.columns[0].targetColumnId, generated: true },
});
expect(invalid.statusCode).toBe(400);
expect(invalid.json()).toMatchObject({ code: "relationship_request_invalid" });
const readOnly = await app.inject({
method: "PATCH",
url: `/catalog/databases/${database.id}/relationships/${physical.id}`,
payload: { status: "excluded" },
});
expect(readOnly.statusCode).toBe(409);
expect(readOnly.json()).toMatchObject({ code: "relationship_read_only" });
});
test("requires database.manage for relationship map mutations", async () => {
const { app, repository, database } = await setup({ AUTH_MODE: "upstream" });
const observed = snapshot();
observed.relationships = [];
await seedCatalog(repository, database, observed);
const response = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/relationships/rebuild-generated`,
headers: {
"x-thoth-principal-issuer": "portal",
"x-thoth-principal-subject": "catalog-reader",
"x-thoth-is-admin": "0",
},
});
expect(response.statusCode).toBe(403);
expect(response.json()).toEqual({ code: "auth_forbidden", error: "This operation is not permitted" });
});
@@ -54,7 +54,6 @@ async function setup() {
list: vi.fn(async () => [revision]),
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
read: vi.fn(async () => ({ workspace, revision })),
readPinned: vi.fn(async () => ({ workspace, workspaceConfigPath: revision.snapshotPath })),
} as unknown as WorkspaceRegistry;
const secretStore = new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" });
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
@@ -1,196 +0,0 @@
import { describe, expect, test, vi } from "vitest";
import { EffectiveRelationshipSnapshotProvider } from "../src/catalog/effective-relationship-snapshot.js";
import type { CatalogRelationship, WorkspaceDatabase } from "../src/catalog/types.js";
const timestamp = "2026-08-31T10:00:00.000Z";
function database(): WorkspaceDatabase {
return {
id: "database-1",
workspaceId: "psd",
engine: "postgres",
databaseName: "warehouse",
schema: "public",
version: 1,
createdAt: timestamp,
updatedAt: timestamp,
binding: { transport: "postgres_direct" },
connectionStatus: "reachable",
schemaSyncedVersion: 1,
schemaSyncedAt: timestamp,
};
}
const operations = {
async run<T>(_databaseId: string, operation: () => Promise<T>): Promise<T> {
return await operation();
},
};
function relationship(options: {
id: string;
origin: "physical" | "generated" | "manual";
status?: "active" | "excluded";
sourceTable?: string;
sourceColumns?: string[];
targetTable?: string;
targetColumns?: string[];
}): CatalogRelationship {
const sourceColumns = options.sourceColumns ?? ["user_id"];
const targetColumns = options.targetColumns ?? ["id"];
const columns = sourceColumns.map((sourceColumnName, position) => ({
position: position + 1,
sourceColumnId: `source-${position}`,
sourceColumnName,
targetColumnId: `target-${position}`,
targetColumnName: targetColumns[position],
}));
const common = {
id: options.id,
databaseId: "database-1",
sourceTableId: "source-table",
sourceTableName: options.sourceTable ?? "orders",
targetTableId: "target-table",
targetTableName: options.targetTable ?? "users",
columns,
createdAt: timestamp,
updatedAt: timestamp,
};
if (options.origin === "physical") {
return {
...common,
constraintName: `fk_${options.id}`,
updateRule: "NO ACTION",
deleteRule: "NO ACTION",
deferrable: false,
initiallyDeferred: false,
lastSyncedDatabaseVersion: 1,
lastSyncedAt: timestamp,
origin: "physical",
status: "active",
};
}
return {
...common,
constraintName: null,
updateRule: null,
deleteRule: null,
deferrable: false,
initiallyDeferred: false,
lastSyncedDatabaseVersion: null,
lastSyncedAt: null,
origin: options.origin,
status: options.status ?? "active",
columns: [columns[0]!],
};
}
describe("EffectiveRelationshipSnapshotProvider", () => {
test("returns no projection for a workspace without a Catalog database", async () => {
const list = vi.fn();
const provider = new EffectiveRelationshipSnapshotProvider(
{
get: vi.fn().mockResolvedValue(undefined),
getByWorkspace: vi.fn().mockResolvedValue(undefined),
},
{ list },
operations,
);
await expect(provider.render("legacy")).resolves.toBeUndefined();
expect(list).not.toHaveBeenCalled();
});
test("renders a deterministic active map with physical precedence", async () => {
const duplicateGenerated = relationship({ id: "generated", origin: "generated" });
const physical = relationship({ id: "physical", origin: "physical" });
const compositePhysical = relationship({
id: "physical-composite",
origin: "physical",
sourceTable: "order_lines",
sourceColumns: ["order_id", "tenant_id"],
targetTable: "orders",
targetColumns: ["id", "tenant_id"],
});
const manual = relationship({
id: "manual",
origin: "manual",
sourceTable: "invoices",
sourceColumns: ["customer_id"],
targetTable: "customers",
targetColumns: ["id"],
});
const excluded = relationship({
id: "excluded",
origin: "generated",
status: "excluded",
sourceTable: "invoices",
});
const list = vi.fn().mockResolvedValue([
manual,
duplicateGenerated,
excluded,
physical,
compositePhysical,
]);
const provider = new EffectiveRelationshipSnapshotProvider(
{
get: vi.fn().mockResolvedValue(database()),
getByWorkspace: vi.fn().mockResolvedValue(database()),
},
{ list },
operations,
);
const rendered = await provider.render("psd");
expect(rendered?.endsWith("\n")).toBe(true);
expect(JSON.parse(rendered!)).toEqual({
schemaVersion: 1,
workspaceId: "psd",
relationships: [
{
sourceTable: "invoices",
sourceColumns: ["customer_id"],
targetTable: "customers",
targetColumns: ["id"],
origin: "manual",
},
{
sourceTable: "order_lines",
sourceColumns: ["order_id", "tenant_id"],
targetTable: "orders",
targetColumns: ["id", "tenant_id"],
origin: "physical",
},
{
sourceTable: "orders",
sourceColumns: ["user_id"],
targetTable: "users",
targetColumns: ["id"],
origin: "physical",
},
],
});
expect(list).toHaveBeenCalledWith("database-1");
});
test("fails closed when the Catalog schema is absent or stale", async () => {
const stale = database();
delete stale.schemaSyncedVersion;
delete stale.schemaSyncedAt;
const list = vi.fn();
const provider = new EffectiveRelationshipSnapshotProvider(
{
get: vi.fn().mockResolvedValue(stale),
getByWorkspace: vi.fn().mockResolvedValue(stale),
},
{ list },
operations,
);
await expect(provider.render("psd")).rejects.toThrow(
"effective relationship snapshot requires a current full schema synchronization",
);
expect(list).not.toHaveBeenCalled();
});
});
@@ -127,7 +127,7 @@ test("resolves only a configured selection for the later generation boundary", (
expect(() => models.resolve("unknown-model")).toThrow(MetadataGenerationModelUnavailableError);
});
test("loads DeepSeek models, GLM, and an explicit keyless Qwen endpoint from installation setup", () => {
test("loads DeepSeek, GLM, and an explicit keyless Qwen endpoint from installation setup", () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: glm-53
models:
@@ -135,10 +135,6 @@ test("loads DeepSeek models, GLM, and an explicit keyless Qwen endpoint from ins
label: DeepSeek V4 Pro
litellm: {provider: deepseek, model: deepseek-v4-pro}
apiKeyEnv: DEEPSEEK_API_KEY
- id: deepseek-v4-flash
label: DeepSeek V4 Flash
litellm: {provider: deepseek, model: deepseek-v4-flash}
apiKeyEnv: DEEPSEEK_API_KEY
- id: glm-53
label: GLM 5.3
litellm:
@@ -160,7 +156,6 @@ test("loads DeepSeek models, GLM, and an explicit keyless Qwen endpoint from ins
expect(models.catalog()).toEqual({
models: [
{ id: "deepseek-v4-pro", label: "DeepSeek V4 Pro" },
{ id: "deepseek-v4-flash", label: "DeepSeek V4 Flash" },
{ id: "glm-53", label: "GLM 5.3" },
{ id: "qwen-36", label: "Qwen 3.6" },
],
+2 -2
View File
@@ -64,7 +64,7 @@ sys.stdout.write(json.dumps({"ok": True, "content": "Descrizione italiana"}))
signal: new AbortController().signal,
});
expect(content).toEqual({ content: "Descrizione italiana", usage: { input: 0, cacheRead: 0, output: 0 } });
expect(content).toBe("Descrizione italiana");
const captured = JSON.parse(readFileSync(join(roots[0]!, "request.json"), "utf8"));
expect(captured.request).toEqual({
model: "openai/gpt-4.1-mini",
@@ -100,7 +100,7 @@ sys.stdout.write(json.dumps({"ok": True, "content": "Descrizione Qwen"}))
},
messages: [{ role: "user", content: "Describe invented metadata." }],
signal: new AbortController().signal,
})).resolves.toEqual({ content: "Descrizione Qwen", usage: { input: 0, cacheRead: 0, output: 0 } });
})).resolves.toBe("Descrizione Qwen");
expect(JSON.parse(readFileSync(join(roots[0]!, "request.json"), "utf8"))).toEqual({
model: "openai/qwen3.6-35b-a3b",
-57
View File
@@ -517,63 +517,6 @@ test("creates a session from the active immutable workspace revision", async ()
}));
});
test("hands one effective relationship snapshot to both retrieval and Pi", async () => {
const effective = JSON.stringify({
schemaVersion: 1,
workspaceId: "default",
relationships: [],
});
const render = vi.fn(async () => effective);
const acquireWorkspaceRuntime = vi.fn((_workspace: string, relationships?: string) => ({
path: "/runtime/with-relationships.yaml",
workspaceId: "default",
workspaceRevision: "e".repeat(40),
release: vi.fn(),
}));
const searchPack = vi.fn(async () => {});
const createFor = vi.fn(() => ({ bridge: { onClientEvent: () => {} } }));
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "../harness" }), {
thtRunner: {
sessionNew: async () => ({ id: "effective-map" }),
acquireWorkspaceRuntime,
searchPack,
} as any,
effectiveRelationshipSnapshotProvider: { render } as any,
readiness: { ensure: async () => ({ ok: true }) } as any,
mgr: {
get: () => undefined,
createFor,
configure: async () => {},
start: () => {},
} as any,
getSettings: () => ({ workspace: "default", thinking: "low" }) as any,
});
const response = await app.inject({
method: "POST",
url: "/sessions",
payload: { question: "Which users placed orders?", workspaceId: "default" },
});
expect(response.statusCode).toBe(200);
expect(render).toHaveBeenCalledWith("default");
expect(acquireWorkspaceRuntime).toHaveBeenCalledWith(
expect.stringContaining("/default.yaml"),
effective,
);
expect(createFor).toHaveBeenCalledWith(
"effective-map",
expect.objectContaining({
runtimeConfig: expect.objectContaining({ path: "/runtime/with-relationships.yaml" }),
}),
);
expect(searchPack).toHaveBeenCalledWith(
"Which users placed orders?",
"effective-map",
"/runtime/with-relationships.yaml",
);
});
test("rejects an SSH-only workspace before persisting or starting a session", async () => {
const sessionNew = vi.fn(async () => ({ id: "must-not-exist" }));
const ensure = vi.fn(async () => ({ ok: true }));
@@ -197,25 +197,6 @@ test("ThtRunner uses a vault secret only for the lifetime of its runtime lease",
expect(existsSync(rendered.database.password_file)).toBe(false);
});
test("ThtRunner binds and cleans the effective relationship snapshot with its runtime lease", async () => {
const f = await fixture();
const runner = runnerFor(f);
const relationships = JSON.stringify({
schemaVersion: 1,
workspaceId: "psd-clinical",
relationships: [],
});
const lease = runner.acquireWorkspaceRuntime(f.revision.snapshotPath, relationships);
const rendered = parse(readFileSync(lease.path, "utf8")) as {
paths: { effective_relationships: string };
};
expect(readFileSync(rendered.paths.effective_relationships, "utf8")).toBe(relationships);
lease.release();
expect(existsSync(rendered.paths.effective_relationships)).toBe(false);
});
test("separate runtime leases hand off byte-identical revision Evidence configs accepted by tht", async () => {
const f = await fixture();
const runner = runnerFor(f);
@@ -17,12 +17,6 @@ metadataGeneration:
provider: deepseek
model: deepseek-v4-pro
apiKeyEnv: DEEPSEEK_API_KEY
- id: deepseek-v4-flash
label: DeepSeek V4 Flash
litellm:
provider: deepseek
model: deepseek-v4-flash
apiKeyEnv: DEEPSEEK_API_KEY
- id: glm-53
label: GLM 5.3
litellm:
@@ -1,36 +0,0 @@
# Use the Catalog as the logical relationship authority
ThothII stores database-declared foreign keys and user-managed Logical Relationships in separate
Catalog models, as required by ADR-0006, but exposes them through one effective relationship map.
Physical relationships remain read-only and are refreshed from the database. Logical relationships
are either Generated by deterministic column-name inference or Manual; inference does not call an
AI model and does not inspect source values.
A generated rebuild is additive. It preserves active and Manual relationships, never reactivates a
logically deleted relationship, and may recreate a relationship only after permanent deletion. A
logical deletion is therefore represented by retaining the relationship with an exclusion marker;
a permanent deletion removes it. Inference accepts only unambiguous, type-compatible matches to a
single-column primary key and skips all other candidates. It recognizes normalized table-qualified
names such as `user_id -> users.id`, exact non-generic primary-key names with one owner, and the
warehouse convention `*time_key -> dim_time.<single PK>`. A source column may participate in a
composite primary key; bare generic names such as `id`, `key`, `code`, and `pk` are not evidence by
themselves.
An exclusion is durable while both Catalog Column endpoints exist. Explicit metadata cleanup of an
endpoint table or column is a destructive boundary: it permanently removes every relationship and
exclusion attached to that endpoint, invalidates the synchronized-catalog marker, and requires a
full schema synchronization. The newly imported endpoints may then be inferred again. Preserving an
exclusion across endpoint destruction would require a second denormalized name-based identity, which
this design deliberately avoids.
The installation-local Catalog is the sole writable authority for Logical Relationships. Git-pinned
workspace annotations remain authoritative for descriptive metadata but their embedded foreign keys
are legacy compatibility data. The backend materializes the active effective map as an immutable,
deterministic runtime snapshot when a session starts or resumes. When that snapshot is present, the
harness uses it as the exclusive relationship source and ignores relationships embedded in both the
physical schema artifact and workspace annotations; a missing or invalid declared snapshot fails
closed. Legacy runtimes without a snapshot retain the previous merge behavior.
The snapshot is a projection, not another authored store. Its lifetime is tied to the runtime-config
lease, physical relationships take precedence over duplicate Logical Relationships, composite-key
column order is retained, and one Pi process observes one stable map for its complete lifetime.
@@ -1,67 +0,0 @@
# Use one Installation Model Catalog with runtime projections
ThothII currently declares model availability independently in installation
`metadataGeneration`, Pi configuration, and workspace `llm_policy` and embedding settings. The
Installation Model Catalog in `thothii-installation.yaml` becomes the sole authored authority for
session, metadata-generation, and embedding models, including their allowed usages and per-usage
defaults. Workspace descriptors retain database identity and scope plus Evidence concerns, but no
model policy or selection; installation-local Database Bindings remain separate from them.
Pi, the backend metadata-generation helper, and the embedding runtime consume generated Model
Runtime Projections of that catalog. Runtime Model Selection stores only a canonical catalog model
identity and use-specific controls such as thinking level; endpoint, provider, capabilities, and
credential references remain catalog facts, while secret values remain in protected secret stores.
The metadata-generation helper continues to use LiteLLM independently of Pi, as established by
ADR-0009: unifying model declaration does not unify execution lifecycles.
The migration is intentionally fail-closed. Workspace schema v4 removes `llm_policy` and the
entire redundant `semantic_index`; collection identity is derived from the workspace identity,
while vector-store and embedding facts come from the installation. A deterministic migration
rewrites existing descriptors. The
installation loader replaces `metadataGeneration` with `modelCatalog` and rejects the legacy form
with an actionable migration error rather than keeping two live sources. Existing Pi
`models.json` and enabled-model settings become generated artifacts and are never edited as
authoritative configuration.
Catalog identities use the canonical `provider/model` form; runtime-specific upstream names are
adapter facts, not additional ThothII identities. Installation descriptors use schema version 2
and model-free workspace descriptors use schema version 4. Removing a model never substitutes it
inside an existing session: an unresolvable resume fails explicitly. Published semantic indexes
record the embedding identity and dimensions that produced them and require explicit
reprocessing when those facts change.
Model eligibility is expressed by the presence of a `session` or `metadataGeneration` block,
without a duplicate usages list. The installation declares one active embedding identity and its
dimensions rather than a selectable embedding catalog. Runtime projections are regenerated
deterministically and atomically at start, so they require no persisted digest and are excluded
from installation backups; restore regenerates them from the validated installation descriptor.
A provider owns one endpoint, one explicit authentication mode, and only the runtime adapters it
needs. Authentication is either a protected secret-environment reference, Pi-owned authentication
for session-only built-in models, or explicit keyless operation for an explicit endpoint; models
cannot override it. Pi built-in model facts are not copied into the installation. A session default
and the single embedding definition are required, while metadata generation and its default may be
omitted together. The schema deliberately excludes unused abstractions and future properties until
runtime behavior requires them.
The catalog session default replaces `PI_PROVIDER`, `PI_MODEL`, and persisted installation model
defaults as configuration sources. A user or session selection is only a canonical catalog
reference, and an existing session keeps that reference without silently switching models. A
generated Compose projection supplies the catalog-derived embedding values and projection mounts
to every affected service, so neither the base Compose files nor `operator.env` repeat model facts.
Workspace v3-to-v4 migration is a deterministic removal of `llm_policy` and `semantic_index`.
Installation migration instead inspects the legacy installation metadata block and both Pi model
files: it emits a v2 candidate only when their identities and settings can be reconciled without
guessing. Conflicts produce an actionable report and leave every source untouched.
## Considered Options
- A separate catalog file referenced by the installation was rejected because it adds path,
permission, backup, and atomic-update coordination without a current need for cross-installation
sharing.
- Pi `models.json` was rejected as the authority because it is a Pi-specific projection that does
not express all ThothII usages, built-in providers, metadata-generation controls, or embedding
facts.
- Transitional dual reading was rejected because it would preserve the configuration discrepancy
this decision is intended to eliminate.
@@ -1,52 +0,0 @@
---
status: accepted
---
# Assess sensitive columns locally from source content
The Sensitive Data Flag remains a human-owned boolean. An explicit, selection-scoped sensitivity
analysis may propose changes by inspecting both catalog metadata and source values, but it never
writes the flag. The administrator may accept, reject, or reverse every proposal.
One TypeScript `SensitivityClassifier` is the only component allowed to produce the column-level
assessment `sensitive`, `non_sensitive`, or `unknown`. It applies a versioned Sensitive Data Policy
and consumes values through database-independent streaming adapters. Database-specific code may
read and normalize bounded values, but it may not decide sensitivity.
The classifier first applies deterministic metadata rules, value validators, checksums,
dictionaries, and length rules. A single validated sensitive match makes the whole column
`sensitive`; any textual value longer than 500 characters is such a match. It attempts a complete
scan, but after five seconds per table it continues by sampling within the remaining run budget. A
completed scan with no finding may produce `non_sensitive`; an incomplete scan with no finding
produces `unknown`.
Ambiguous text may additionally be sent to an optional local NER detector only while time remains.
The detector runs on CPU, receives no tools or network access, does not persist source values, and
returns evidence rather than the column decision. The initial supported detector is
[`fastino/gliner2-privacy-filter-PII-multi`](https://huggingface.co/fastino/gliner2-privacy-filter-PII-multi),
used through the Apache-2.0 GLiNER2 Python library with a pinned model revision. Its model weights
and GLiNER2 code are Apache-2.0, and its mDeBERTa base model is MIT. It is trained for seven
languages including Italian and can run on CPU without using the installation's GPUs.
The NER detector is an optional installation asset because its weights and runtime are materially
larger than the deterministic TypeScript engine. It is invoked only for otherwise unresolved text,
never for values already classified by a decisive rule. If it is disabled, unavailable, times out,
or returns no qualifying evidence before the deadline, the classifier follows the same coverage
rule and may return `unknown`.
No generative LLM, internal or external, participates in sensitivity assessment. A fallback to an
installation-local LLM is unnecessary while a permissively licensed local NER implementation is
available, and would reintroduce queue latency, non-deterministic judgments, prompt-injection
surface, and contention with normal inference. Introducing such a fallback would require a new
decision based on evidence that the NER path is unusable.
This decision supersedes ADR-0011 only where that ADR assigns draft sensitivity suggestions to an
AI using structural metadata. ADR-0011's human authority, transient draft, explicit scope, and
Sensitive Data Flag remain in force. How description generation uses the flag is outside this
decision.
The selected model's published evaluation is not an Italian production acceptance test. Before
enabling the NER profile by default, ThothII must pin the artifacts, generate a dependency/license
inventory, and pass a CPU benchmark plus a labeled Italian corpus representative of the target
databases. Failure of those gates disables NER; it does not silently select another model or an
LLM.
-1
View File
@@ -108,7 +108,6 @@ not supported by this installation until the declarative contract is extended ge
| Model | Level | Why | Visibility |
|---|---|---|---|
| `deepseek/deepseek-v4-pro` | Built-in Pi | Known public API, already in the build | All projects |
| `deepseek/deepseek-v4-flash` | Built-in Pi | Known public API, already in the build | All projects |
| `zai/glm-5.3` | `deploy/pi/models.json` | Custom OpenAI-compatible endpoint | ThothII installation |
| `local-qwen/qwen3.6-35b-a3b` | `deploy/pi/models.json` | Locally configured OpenAI-compatible endpoint | ThothII installation |
+3 -16
View File
@@ -6,7 +6,7 @@ session workflow.
## What the catalog owns
For each YAML workspace, an administrator may configure at most one Metadata Catalog binding. It holds
For each YAML workspace, an administrator may create at most one database configuration. It holds
the database name, schema, connection binding, write-only encrypted secrets, observed physical
schema, optional curated descriptions, generated descriptions, and durable operation history.
@@ -27,18 +27,6 @@ It uses `GET /catalog/metrics` without `databaseId` for installation totals and
the current database. Choose a selection-scoped operation from the action selector and then press
**Run**; unavailable operations remain listed with an explanation. Row-specific actions are the icon
controls in the final column, and each navigation or action icon has an immediate conceptual tooltip.
An unconfigured workspace exposes **Configure catalog** directly on its row; there is no global
database-creation action and the selected workspace cannot be changed in the configuration form.
The master grid keeps three independent states visible:
- **Revision / Evidence** comes from the active immutable workspace revision. Filesystem Evidence is
materialized with that revision; remote Evidence is reported as configured-but-unverified or as
requiring credentials.
- **NL→SQL runtime** is calculated from the workspace DWH/Evidence requirements and runtime secret
store. It also reports transports, such as SSH, that are diagnostic-only and unsupported by sessions.
- **Metadata Catalog** reports whether the installation-local catalog configuration exists, then shows
its separately versioned connection-test or synchronization state.
Configuration, object details, metadata editors, synchronization history, description history,
sensitive-field review, and suggestion-run history open in right-side drawers backed by the
@@ -54,9 +42,8 @@ remains available only until the integrated Fleet Ledger surface passes owner ac
## Configure and test a database
1. Open **Database Management** and find the repository workspace marked **Not configured**.
2. Choose **Configure catalog** on that row. Configure its PostgreSQL catalog binding with
`postgres_direct`, `rest_api`, or `ssh_tunnel` and
1. Open **Database Management** and choose a workspace.
2. Create its PostgreSQL configuration. Choose `postgres_direct`, `rest_api`, or `ssh_tunnel` and
complete the binding fields that the chosen transport requires.
3. Enter secrets only when replacing them. They remain write-only and are never returned by the
application.
-43
View File
@@ -1,43 +0,0 @@
# ThothII Docker refresh
The active local `psd-local` stack uses the operator environment generated for the installation:
`deploy/psd/operator.env`
It is not `deploy/thothii.env` (that file is empty) and `deploy/env/local.env` is only an example
path referenced by the generic launcher. The running stack also uses these Compose overlays:
```text
compose.yaml
deploy/compose.local.yaml
deploy/compose.git-ssh.yaml
deploy/psd/connector-secrets.yaml
```
Refresh the stack from the repository root with:
```bash
compose_psd=(
docker compose
--env-file deploy/psd/operator.env
-p thothii-18998cca7b0a
-f compose.yaml
-f deploy/compose.local.yaml
-f deploy/compose.git-ssh.yaml
-f deploy/psd/connector-secrets.yaml
)
"${compose_psd[@]}" config --quiet
"${compose_psd[@]}" build core frontend
"${compose_psd[@]}" stop core frontend
"${compose_psd[@]}" up -d catalog-db
"${compose_psd[@]}" run --rm catalog-migrate
"${compose_psd[@]}" up -d --remove-orphans
```
Building before the stop keeps the existing application available if an image fails to compile.
Stopping only `core` and `frontend` prevents the old backend from using a newly migrated catalog;
the database, Qdrant, embedding service, named volumes, and installation state remain in place.
The installation secret files referenced by that env file live under `deploy/psd/secrets/` and
must never be committed or printed.
@@ -109,13 +109,12 @@ canonico; i percorsi e i comportamenti descrivono il sorgente disponibile il 202
conserva il valore esistente e la sostituzione è un'azione esplicita. Delete rimuove anche i
segreti associati.
34. La pagina usa AG Grid come master e un form React come detail, con sezioni Database, Connection
e TLS/SSH condizionali. Non esiste un'azione globale `Add database`: ogni riga `unconfigured`
offre `Configure catalog`, apre il form già vincolato a quello specifico workspace YAML e crea il
record soltanto al Save; `workspace_id` non è selezionabile né modificabile.
35. La grid mostra separatamente revisione/Evidence del workspace, binding runtime NL→SQL e
configurazione del Metadata Catalog, oltre a database, schema, endpoint e ultimo aggiornamento.
Su schermi piccoli il dettaglio occupa il pannello completo. Il cambio riga con modifiche non
salvate e Delete richiedono conferma, senza conferma testuale tipizzata.
e TLS/SSH condizionali. La toolbar offre `Add database`; le righe `unconfigured` offrono
`Configure`. Entrambe selezionano esclusivamente workspace YAML senza un database e creano il
record soltanto al Save; `workspace_id` diventa immutabile dopo la creazione.
35. La grid mostra workspace, database, schema, transport, endpoint, stato connessione e ultimo
aggiornamento. Su schermi piccoli il dettaglio occupa il pannello completo. Il cambio riga con
modifiche non salvate e Delete richiedono conferma, senza conferma testuale tipizzata.
36. La Database Binding conserva `connection_status`, `tested_version`, `last_tested_at`, un codice
errore e un messaggio breve sanificato. Non conserva stack trace, DSN, credenziali o output grezzo
del driver.
@@ -179,12 +178,12 @@ canonico; i percorsi e i comportamenti descrivono il sorgente disponibile il 202
e Schema Sync. Il renderer e il runtime delle sessioni NL→SQL restano fuori scope e continuano a
rifiutarla finché non verrà deciso il relativo cutover.
55. I menu di azione a livello Workspace Database espongono separatamente `Synchronize tables`,
`Synchronize relationships` e `Synchronize all`. Su una selezione di
`Synchronize all columns`, `Synchronize relationships` e `Synchronize all`. Su una selezione di
database lo scope scelto viene avviato per ogni database idoneo; non viene sostituito
implicitamente con una sincronizzazione completa.
56. Lo scope Columns è disponibile dalla grid Tables e limita la riconciliazione alle tabelle
selezionate; la pagina Columns non espone azioni di sincronizzazione. La grid Tables espone
`Synchronize columns` sulle tabelle selezionate.
56. Per lo scope Columns, `tableIds` vuoto significa tutte le Catalog Table correnti del Workspace
Database; `tableIds` valorizzato limita invece la riconciliazione alle tabelle indicate. La grid
Tables espone `Synchronize columns` sulle tabelle selezionate.
## Correzione del modello mentale corrente
@@ -570,9 +569,9 @@ Le griglie che dispongono di azioni massive usano checkbox e una toolbar contest
menu `Actions` e cancellazione della selezione. La selezione identifica ID espliciti, può essere
accumulata attraverso i filtri e viene azzerata dopo successo, nuova sincronizzazione o uscita
dalla pagina; un'azione è all-or-nothing se un elemento non è idoneo. I menu a livello database
espongono gli scope fisici come azioni distinte: `Synchronize tables`, `Synchronize relationships`
e `Synchronize all`. La grid Tables espone invece `Synchronize columns` per le tabelle selezionate;
la pagina Columns non espone sincronizzazione. Le selezioni database aggiungono `Delete all tables` e
espongono gli scope fisici come azioni distinte: `Synchronize tables`, `Synchronize all columns`,
`Synchronize relationships` e `Synchronize all`. La grid Tables espone invece `Synchronize
columns` per le tabelle selezionate. Le selezioni database aggiungono `Delete all tables` e
`Delete all relationships`; le selezioni tabelle aggiungono `Delete all columns` e `Delete all
relationships`. Queste operazioni sono atomiche, richiedono conferma e non modificano database
esterno, binding, configurazione o segreti. Test connection resta un'azione distinta; griglie senza
@@ -1,245 +0,0 @@
# Installation Model Catalog
Status: accepted design; implementation not started.
## Outcome
`thothii-installation.yaml` is the only operator-authored source for models used by interactive
sessions, metadata generation, and embedding. Runtime-specific files are deterministic projections,
not additional configuration sources. Workspace descriptors contain database and Evidence concerns
and no model, provider, allowlist, default, embedding, or vector-store configuration.
This design does not merge execution lifecycles. Pi continues to run interactive sessions, the
short-lived LiteLLM helper continues to perform metadata generation, and the internal Ollama service
continues to provide embeddings. They share model declaration, not execution machinery.
## Canonical installation shape
The following example covers all currently required cases: a Pi built-in model, an authenticated
custom endpoint, a keyless internal endpoint, metadata generation, and the single embedding model.
```yaml
schemaVersion: 2
profile: server
projectDirectory: /srv/thothii
envFile: /srv/thothii/operator.env
workspaceRepository:
remote: git@git.example.com:organization/workspaces.git
branch: main
access: ssh
modelCatalog:
defaults:
session: zai/glm-5.3
metadataGeneration: local-qwen/qwen3.6-35b-a3b
embedding:
id: ollama/qwen3-embedding:0.6b
dimensions: 1024
providers:
deepseek:
authentication:
mode: pi_auth
session:
mode: pi_builtin
models:
deepseek-v4-pro:
session: {}
deepseek-v4-flash:
session: {}
zai:
endpoint:
baseUrl: https://api.z.ai/api/coding/paas/v4
authentication:
mode: secret_env
apiKeyEnv: ZAI_API_KEY
session:
mode: openai_compatible
metadataGeneration:
litellmProvider: openai
models:
glm-5.3:
label: GLM-5.3
session:
reasoning: true
contextWindow: 200000
maxTokens: 131072
metadataGeneration: {}
local-qwen:
endpoint:
baseUrl: https://ml-aritmolab.policlinicosandonato.it/v1
authentication:
mode: none
session:
mode: openai_compatible
metadataGeneration:
litellmProvider: openai
models:
qwen3.6-35b-a3b:
label: Qwen3.6 35B A3B
session:
reasoning: false
contextWindow: 131072
maxTokens: 16384
compatibility:
supportsDeveloperRole: false
supportsReasoningEffort: false
supportsStore: false
maxTokensField: max_tokens
metadataGeneration:
disableThinking: true
authentication:
configDirectory: /srv/thothii/auth-canonical
runtimeProjection:
directory: /srv/thothii/auth-runtime
uid: 10001
gid: 10001
```
The catalog uses maps instead of repeated IDs. The canonical identity of a model is always derived
as `<provider-key>/<model-key>`. `upstreamModel` may be added to a model only when the endpoint uses
a different identifier. `label` is optional and falls back to the canonical identity.
Model eligibility is not repeated in an `usages` array. A `session` block makes the model eligible
for sessions; a `metadataGeneration` block makes it eligible for metadata generation. The embedding
is a single required installation value rather than a list plus default.
## Provider and authentication rules
A provider owns one endpoint, one authentication mode, and zero or one adapter for each runtime.
Model entries cannot override provider endpoint or credentials. If the same upstream service needs
different endpoints or credentials, the installation declares two provider identities.
Supported session modes are intentionally closed:
- `pi_builtin`: Pi already owns the model's technical descriptor; the model's `session` block is
empty and ThothII does not copy context-window or compatibility facts.
- `openai_compatible`: ThothII generates a Pi custom-provider descriptor; each session model supplies
the technical values required by Pi.
Metadata generation uses the provider-level `litellmProvider`. A model-level
`metadataGeneration.disableThinking: true` is permitted only for an explicit compatible endpoint.
There is no generic adapter or plugin abstraction in schema version 2.
Exactly one provider authentication mode is allowed:
- `secret_env` requires an approved API-key environment reference present in the protected secret
bundle. Secret values never enter YAML, generated files, logs, arguments, or API responses.
- `pi_auth` is valid only for session-only `pi_builtin` providers and resolves through Pi's protected
authentication projection.
- `none` is valid only for an explicit endpoint. Runtime projections may supply a fixed non-secret
compatibility placeholder when a client library requires a non-empty key.
## Defaults and selections
`defaults.session` and `embedding` are required. `defaults.metadataGeneration` is required exactly
when at least one model has a `metadataGeneration` block; metadata generation may otherwise be
absent and its UI controls are disabled.
`modelCatalog.defaults.session` is the only configured session-model default. `PI_PROVIDER`,
`PI_MODEL`, and provider/model fields in installation-default settings are removed. A user choice is
a Model Selection containing only the canonical model identity and runtime controls such as thinking
level. A session manifest pins the selected canonical identity.
Removing the currently selected model causes new-session selection to fall back to the catalog
default with an explicit administrative warning. An existing session is never silently moved to a
different model; resume fails with `model_unavailable` when its pinned identity can no longer be
resolved.
## Generated runtime projections
Before Compose starts, `tht` strictly validates schema version 2 and generates installation-local
artifacts below `deploy/<installation-id>/generated/`:
- a normalized catalog JSON consumed defensively by the backend;
- Pi `models.json` for custom providers;
- Pi `settings.json`, combining fixed product settings with the session-eligible canonical IDs;
- a Compose override that mounts the projections and supplies embedding identity and dimensions to
core, preprocessing, and `embedding-model-init`.
Generation is deterministic and published only after every candidate artifact validates. A failed
generation aborts start before Compose is invoked. `tht doctor` recomputes expected bytes and reports
differences; no digest manifest or separate apply command exists. When projection bytes change,
`tht start` recreates the affected services so they cannot continue with an older bind mount.
Generated projections are not backed up. Restore validates the canonical installation descriptor,
regenerates every projection, and only then starts services. Base Compose files and `operator.env`
must contain no model identities, defaults, endpoints, or dimensions.
## Workspace schema v4
Workspace schema v4 removes both top-level `llm_policy` and `semantic_index`. The entire latter
block is redundant today: its engine and distance are product constants, its collection duplicates
the workspace ID, and its model and dimensions are installation facts.
The runtime derives:
- Qdrant collection identity from the workspace ID;
- engine and distance from the supported product contract;
- embedding identity and dimensions from the Installation Model Catalog.
The published index generation records the canonical embedding identity and dimensions that created
it. A mismatch makes the index explicitly incompatible and requires operator-triggered
preprocessing. No existing index is deleted or rebuilt automatically.
The v3-to-v4 workspace migration is deterministic: set `workspace.schema_version` to `4`, remove
`llm_policy`, and remove `semantic_index`. It does not alter database, Evidence, diagnostics, or
binding data.
## Installation migration
Legacy installation migration must inspect all three former sources:
1. `metadataGeneration` in `thothii-installation.yaml`;
2. `deploy/pi/models.json`;
3. `deploy/pi/settings.json`.
The migrator emits a version-2 candidate only when it can reconcile identities, endpoints,
credentials, and runtime-specific facts without guessing. Ambiguous aliases such as `glm-53`,
`zai/glm-5.3`, and `openai/glm-5.3` are not silently equated. A conflict produces a field-level
report and leaves every input unchanged for operator resolution.
After migration, the strict loader rejects `metadataGeneration`, workspace `llm_policy`, workspace
`semantic_index`, legacy Pi source files, unknown fields, duplicate YAML keys, invalid defaults, and
incompatible authentication/adapter combinations with an actionable `migration_required` or
validation error.
## Final simplicity audit
The accepted design removes every configuration duplication that can be removed without inference:
- one authored installation file instead of an installation block plus two Pi files;
- one canonical `provider/model` identity instead of display IDs and runtime IDs;
- per-use blocks instead of a duplicated usages list;
- one embedding entry instead of a selectable embedding catalog;
- one catalog session default instead of environment and settings defaults;
- no model or vector-store fields in workspace descriptors;
- provider-level credentials instead of per-model credentials;
- no generic runtime-plugin abstraction;
- no persisted digest, apply command, or backup of generated projections.
The remaining generated files are necessary boundary adapters, not configuration concepts. Making
the backend parse the authoring YAML independently would remove one file but restore two semantic
validators. Hard-coding embedding values in Compose would remove one projection but restore a model
source outside the catalog. Inferring authentication from missing fields would save one YAML key but
turn a safe explicit choice into ambiguity. These apparent simplifications are therefore rejected.
No further reduction was found that preserves one authority, strict validation, explicit security,
session determinism, and model-free workspaces.
## Implementation surface
Implementation must update the host `tht` installation loader, setup and lifecycle projection,
doctor, backup/restore, Compose mounts and embedding inputs, backend catalog/settings/session model
resolution, workspace schema and migration, runtime rendering and diagnostics, frontend workspace
drafts and model filtering, examples, fixtures, and documentation. Existing session manifests remain
readable and keep their pinned provider/model identity; only resume resolution changes to the new
catalog.
This document authorizes design only. Software implementation begins only after a separate explicit
request.
@@ -1,299 +0,0 @@
# Visualizzatore ERD nel browser: proposta solo open source
Data dell'audit: 31 agosto 2026.
## Decisione
Per ThothII sceglierei **AntV X6 + ELK.js** come fondazione del visualizzatore ERD.
- **AntV X6** è MIT, non ha una distinta edizione Professional e include nel progetto OSS
le funzioni che servono davvero a un diagramma complesso: nodi e porte personalizzabili,
pan/zoom, selezione, minimappa, history, clipboard, tastiera, routing, virtual rendering ed
export SVG/PNG/JPEG.
- **ELK.js** è il motore di layout automatico, EPL-2.0 e senza tier commerciale. Il layout
layered gestisce porte, vincoli sull'ordine delle porte, archi multipli, self-loop e routing
ortogonale: proprietà importanti per foreign key composite e schemi affollati.
- La combinazione non dipende da esempi o componenti a pagamento. Il lavoro applicativo che
resta da fare — semantica ER, livelli di dettaglio, filtri e adattatore X6/ELK — appartiene
davvero al dominio di ThothII e non è una funzione nascosta dietro un piano Pro.
La seconda scelta è **React Flow + ELK.js**. È probabilmente l'integrazione più naturale con
l'attuale frontend React e ha una migliore storia documentata per accessibilità. Il runtime è
MIT e non viene tecnicamente depotenziato, ma diversi pattern avanzati già implementati
(auto-layout, edge routing, undo/redo, copy/paste, expand/collapse) sono pubblicati come esempi
con licenza React Flow Pro. Non è quindi la scelta più lineare se il criterio prioritario è
evitare anche una dipendenza progettuale da materiale Professional.
## Cosa significa «open source senza limitazioni Professional»
L'audit distingue tre casi:
1. **OSS completo**: licenza OSI e nessun tier commerciale che trattenga funzioni essenziali.
2. **OSS con caveat**: il motore è aperto, ma esistono esempi premium, API instabili o limiti
architetturali rilevanti. Può essere usato, purché il limite sia accettato esplicitamente.
3. **Escluso**: licenza proprietaria oppure core aperto con funzioni necessarie al viewer
complesso riservate alla versione commerciale.
La valutazione riguarda sia la licenza sia ciò che si può effettivamente costruire senza
acquistare un prodotto complementare. Le note sulle licenze sono tecniche, non consulenza legale.
## Matrice di selezione
| Soluzione | Licenza OSS | Tier Pro rilevante | Adeguatezza a ERD complessi | Verdetto |
|---|---|---|---|---|
| **AntV X6 + ELK.js** | MIT + EPL-2.0 | Nessuno individuato | Porte per colonna, minimappa, routing, export, virtual rendering e layout avanzato | **Consigliata** |
| **React Flow + ELK.js** | MIT + EPL-2.0 | Esempi/pattern avanzati Pro, non runtime separato | Ottima UX React, nodi HTML, minimappa e accessibilità; export SVG non nativo | **Valida con caveat** |
| **maxGraph** | Apache-2.0 | Nessuno | Molto completo, SVG, porte, folding e layout; API imperativa e integrazione React costosa | **Valida con caveat** |
| **Cytoscape.js + ELK** | MIT | Nessuno | Molto scalabile come grafo, ma meno adatto a tabelle ricche e porte per campo | **Alternativa di nicchia** |
| **Liam ERD/CLI** | Apache-2.0 | Nessuno | Viewer ERD già rifinito e self-hostable; il package embeddable è interno e instabile | **Reference o app separata** |
| **Graphviz + Viz.js** | EPL-2.0 + MIT | Nessuno | Layout ed SVG eccellenti; non offre da solo un explorer applicativo | **Renderer di export** |
| **Mermaid.js + Panzoom** | MIT + MIT/ISC | Mermaid Chart è separato | Facile, ma insufficiente per interazioni e modelli ER ricchi | **Solo preview semplice** |
| **Sprotty/GLSP** | EPL-2.0 | Nessuno | Potente e completo, ma è un framework di modeling più pesante del necessario | **Non prioritario** |
| **JointJS** | Core MPL-2.0 | Molte funzioni utili sono in JointJS+ | Lo split commerciale intercetta proprio le necessità del viewer | **Esclusa** |
| **GoJS** | Proprietaria | Licenza di deployment | Completa tecnicamente, ma non open source | **Esclusa** |
| **yFiles** | Proprietaria | Licenza commerciale | Completa tecnicamente, ma non open source | **Esclusa** |
## 1. Scelta principale: AntV X6 + ELK.js
### Funzioni disponibili nell'open source
X6 offre un canvas diagrammatico basato su SVG con supporto anche a nodi HTML/React. Il core
espone porte, archi personalizzabili, self-loop e multiedge. I plugin distribuiti dal progetto
comprendono Scroller, MiniMap, Selection, History, Clipboard, Keyboard, DnD, Stencil, Snapline,
Transform ed Export. La documentazione copre inoltre router `orth`, `manhattan` ed `er`, zoom e
panning. Non è emersa un'edizione X6 Pro che renda a pagamento queste capacità.
Fonti primarie: [repository e licenza X6](https://github.com/antvis/X6),
[porte](https://x6.antv.antgroup.com/en/tutorial/basic/port),
[router](https://x6.antv.antgroup.com/en/api/registry/router),
[minimappa](https://x6.antv.antgroup.com/en/tutorial/plugins/minimap),
[scroller](https://x6.antv.antgroup.com/en/tutorial/plugins/scroller) ed
[export](https://x6.antv.antgroup.com/en/tutorial/plugins/export).
ELK non è un renderer: calcola la geometria del grafo. L'algoritmo layered supporta porte e
relativi vincoli, archi ortogonali, self-loop e archi multipli. Va eseguito in un Web Worker per
non bloccare l'interfaccia sui modelli grandi. L'adattatore dovrà passare a ELK le porte delle
colonne e usare anche le `edge sections` e i bend point restituiti, non soltanto le coordinate
dei nodi.
Fonti primarie: [ELK.js e licenza](https://github.com/kieler/elkjs) e
[ELK layered](https://eclipse.dev/elk/reference/algorithms/org-eclipse-elk-layered.html).
### Limiti reali
- X6 è più imperativo di React Flow: conviene incapsularlo in un singolo componente React con
un adapter stabile, evitando di spargere istanze e listener nel resto dell'applicazione.
- L'accessibilità per screen reader non è documentata al livello di React Flow; tab order,
focus, descrizioni ARIA e navigazione da tastiera richiedono test e lavoro applicativo.
- Un nodo React/HTML può introdurre `foreignObject` nell'SVG. Per un export portabile e
stampabile è meglio produrre un SVG separato dallo stesso modello e dalla geometria ELK.
Questi sono costi tecnici, non limitazioni commerciali.
## 2. Seconda scelta: React Flow + ELK.js
`@xyflow/react` è MIT. Custom nodes React, handle multipli, pan/zoom, fit view, Controls,
MiniMap, selezione e funzioni di accessibilità sono nel runtime OSS. React Flow Pro vende
supporto, template ed esempi con relativo codice sorgente; non esiste una libreria runtime Pro
che sblocchi il canvas.
Il caveat è comunque concreto: auto-layout, edge routing, undo/redo, copy/paste ed
expand/collapse compaiono nel catalogo degli esempi Pro e quel codice usa la **xyflow Pro
License**, non la MIT del core. Le stesse funzioni possono essere sviluppate sopra le API OSS,
ma non si può considerare tutto il materiale ufficiale liberamente riutilizzabile.
Fonti primarie: [package React Flow](https://github.com/xyflow/xyflow/blob/main/packages/react/package.json),
[funzioni del core](https://reactflow.dev/index),
[catalogo degli esempi Pro](https://reactflow.dev/examples/pro-examples),
[auto-layout](https://reactflow.dev/examples/layout/auto-layout) e
[termini Pro](https://reactflow.dev/pro).
È una buona scelta se si privilegiano velocità d'integrazione React, componenti HTML e
accessibilità. Non è la prima scelta di questo audit perché l'utente ha chiesto espressamente
di minimizzare la distanza fra ciò che è open source e l'offerta Professional.
Altro limite: React Flow combina nodi DOM e archi SVG. L'esempio ufficiale di download usa
`html-to-image`, quindi il diagramma interattivo non si traduce automaticamente in un SVG puro.
## 3. Altre soluzioni interamente open source
### maxGraph
maxGraph, successore TypeScript di mxGraph, è Apache-2.0 e non ha un piano Pro. Offre rendering
SVG, connection constraints/porte, loop e multigraph, layout, routing, outline, grouping e
folding. Può quindi sostenere un editor ERD ricco.
Lo terrei come terza scelta: l'API è imperativa, non esiste un wrapper React ufficiale, la
virtualizzazione non è documentata e gran parte dell'accessibilità va costruita. Il progetto è
ancora pre-1.0. Il costo di integrazione e manutenzione sarebbe maggiore di X6.
Fonti: [licenza e repository](https://github.com/maxGraph/maxGraph),
[guida Vite/TypeScript](https://maxgraph.github.io/maxGraph/docs/getting-started/) e
[gestione della complessità](https://maxgraph.github.io/maxGraph/docs/usage/group-and-complexity-management/).
### Cytoscape.js
Cytoscape.js e molte estensioni sono MIT, senza tier professionale. È ottimo per grandi grafi,
filtri, selezioni e algoritmi di rete. Il rendering Canvas è però meno naturale per schede-tabella
ricche, testo selezionabile e handle collegati alle singole colonne. Inoltre l'adapter
`cytoscape.js-elk` non passa le porte a ELK né usa le route degli archi: non risolve da solo il
problema delle foreign key per colonna.
È appropriato per una vista di dipendenze ad alto livello, non come renderer ERD principale.
L'estensione `cytoscape-svg` è GPL-3.0; in un prodotto che non vuole assorbire quel vincolo è
preferibile una pipeline Graphviz/Viz.js o un generatore SVG proprio.
Fonti: [Cytoscape.js](https://github.com/cytoscape/cytoscape.js),
[adapter ELK](https://github.com/cytoscape/cytoscape.js-elk) e
[cytoscape-svg](https://github.com/kinimesi/cytoscape-svg).
### Liam ERD
Liam ERD è Apache-2.0, self-hostable e già orientato al problema: pan/zoom, ricerca, filtro,
command palette e modalità `TABLE_NAME`, `KEY_ONLY`, `ALL_FIELDS`. La documentazione dichiara
supporto a schemi con oltre cento tabelle. La CLI può generare un'app Vite statica.
Non userei però direttamente `@liam-hq/erd-core` dentro ThothII: il maintainer lo definisce una
dipendenza interna, ne sconsiglia l'uso diretto e avverte che l'API può cambiare; il package è
ancora 0.x. Inoltre occorre verificare la compatibilità con la versione React di ThothII e la
fedeltà delle foreign key composite. Liam è quindi un ottimo benchmark UX, una CLI per una vista
separata o una base da forkare consapevolmente, non l'interfaccia stabile su cui fondare il
prodotto.
Fonti: [repository Liam](https://github.com/liam-hq/liam),
[README di erd-core](https://github.com/liam-hq/liam/blob/main/frontend/packages/erd-core/README.md),
[funzioni UI](https://liambx.com/docs/ui-features) e [CLI](https://liambx.com/docs/cli).
### Sprotty/GLSP
Sprotty e Eclipse GLSP sono EPL-2.0 e offrono un'infrastruttura seria per diagrammi SVG,
layout e protocolli client/server. Sono pensati per strumenti di modeling estensibili, con
dependency injection e un'architettura più ampia di un viewer. Restano una soluzione OSS valida,
ma sproporzionata per questa esigenza salvo che ThothII evolva in un vero editor di modelli.
Fonti: [Sprotty](https://github.com/eclipse-sprotty/sprotty) ed
[Eclipse GLSP](https://eclipse.dev/glsp/documentation/overview/).
## 4. Renderer complementari, non fondazioni del viewer
### Graphviz e Viz.js
Graphviz è EPL-2.0; `@viz-js/viz`, il port WebAssembly utilizzabile nel browser, è MIT. Graphviz
produce SVG di alta qualità, supporta label HTML-like e porte e rimane molto utile per export,
stampa o una vista read-only. Non fornisce però da solo ricerca, filtri, livelli di dettaglio,
editing o gestione dello stato applicativo.
Lo userei come renderer di export alternativo, non come UI primaria. Se il layout a schermo
deve corrispondere esattamente all'export, è preferibile generare l'SVG direttamente dalla
geometria ELK invece di mantenere due motori di layout indipendenti.
Fonti: [licenza Graphviz](https://graphviz.org/license/),
[formati SVG](https://graphviz.org/docs/outputs/svg/) e
[Viz.js](https://github.com/mdaines/viz-js).
### Mermaid.js e librerie di pan/zoom
Mermaid.js è MIT e non ha feature del renderer open source bloccate. **Mermaid Chart** è invece
un servizio commerciale distinto e il suo piano gratuito ha limiti: non va confuso con la
libreria self-hosted.
Si può aggiungere navigazione a un SVG Mermaid con `@panzoom/panzoom` (MIT) o `d3-zoom` (ISC),
entrambi senza tier Pro. Questo migliora l'esperienza corrente, ma non supera i limiti strutturali
del diagramma ER Mermaid: interazioni a livello di tabella, controllo ridotto delle porte e del
routing, assenza di semantic zoom e difficoltà crescente su schemi molto grandi.
Mermaid resta adatto a preview e documentazione, non al viewer finale.
Fonti: [Mermaid.js](https://github.com/mermaid-js/mermaid),
[distinzione da Mermaid Chart](https://mermaid.ai/open-source/ecosystem/mermaid-chart.html),
[Panzoom](https://github.com/timmywil/panzoom) e [d3-zoom](https://github.com/d3/d3-zoom).
### Python
SchemaSpy ed ERAlchemy sono open source e possono generare documentazione o grafi attraverso
Graphviz, ma non sostituiscono il componente interattivo React. Python è utile lato backend per
normalizzare metadati o produrre artefatti batch; pan/zoom, selezione, ricerca e dettagli restano
responsabilità del browser. Aggiungere un servizio Python solo per disegnare l'ERD non porta un
vantaggio architetturale a ThothII.
## 5. Soluzioni escluse
### JointJS / JointJS+
Il core JointJS è MPL-2.0, ma JointJS+ è commerciale e comprende proprio molte funzioni che
servirebbero qui: PaperScroller, Navigator/minimappa, selection, clipboard, keyboard, undo/redo,
toolbar, export e layout aggiuntivi. Sarebbe tecnicamente possibile ricostruirle sul core, ma la
distanza fra OSS e Professional è troppo grande rispetto al criterio richiesto.
Fonti: [licenza](https://www.jointjs.com/license),
[confronto delle funzioni](https://www.jointjs.com/features) e
[prezzi](https://www.jointjs.com/pricing).
### GoJS e yFiles
Sono prodotti completi e maturi, ma non open source. GoJS richiede una licenza per il deployment
e mostra una filigrana senza chiave; yFiles usa licenze proprietarie, con evaluation temporanea e
licenza necessaria per la distribuzione. Non soddisfano il requisito, quindi non entrano nella
shortlist.
Fonti: [licensing GoJS](https://gojs.net/latest/intro/deployment.html) e
[licensing yFiles](https://docs.yworks.com/yfiles-html/dguide/deployment/licensing.html).
## 6. Architettura proposta per ThothII
Il frontend usa React 18 e oggi `SchemaLinkingViewer.tsx` genera Mermaid con un limite di 45
elementi. La resa corrente perde informazione: accorpa più foreign key fra la stessa coppia di
tabelle, omette i self-reference e usa cardinalità generiche. Il catalogo espone già tabelle,
colonne, chiavi primarie e relazioni con coppie ordinate di colonne, quindi può alimentare un
modello più fedele.
Propongo questa separazione:
1. **Modello renderer-neutral**: `TableNode`, `ColumnPort` e `RelationEdge` con nome del
constraint e lista ordinata delle coppie sorgente/destinazione. Non appiattire le FK composite.
2. **Snapshot API**: un endpoint coerente, per esempio
`GET /catalog/databases/:databaseId/schema-diagram`, che restituisca tabelle, colonne,
relazioni e versione del catalogo in una sola lettura, evitando la richiesta colonne N+1.
3. **Layout worker**: ELK.js in Web Worker, con porte per colonna, port constraints, route degli
archi e cache per versione del database e filtri correnti.
4. **Renderer interattivo**: X6 incapsulato in `DatabaseRelationshipDiagram.tsx`, affiancato
alla vista tabellare con un toggle List/Diagram e con riuso del drawer dei dettagli.
5. **Export**: generatore SVG separato basato sullo stesso modello e sulla geometria ELK;
Graphviz/Viz.js può essere un fallback per layout alternativi o documentazione batch.
Il catalogo non registra oggi tutti i vincoli UNIQUE né il tipo MATCH delle FK. Senza questi
dati non sempre è possibile distinguere correttamente 1:1 da 1:N. Il renderer non deve inventare
la cardinalità: deve mostrare una notazione neutra finché il metadato necessario non viene
raccolto.
### Funzioni necessarie per schemi complessi
- semantic zoom: solo nomi tabella, poi PK/FK, infine tutti i campi;
- ricerca e filtri per schema, tabella, colonna e tipo di relazione;
- modalità focus con vicinato a uno o due hop;
- evidenziazione della relazione e delle due colonne al passaggio/selezione;
- minimappa, fit selection, navigazione da tastiera e pannello dettagli;
- distinzione visiva di PK, FK, nullable, composite key, self-loop e relazioni parallele;
- layout automatico ricalcolabile, ma posizioni manuali persistibili;
- rendering progressivo o virtuale e fallback tabellare sempre disponibile;
- export dell'intero schema e dell'area filtrata in SVG/PNG.
## 7. Proof of concept consigliato
Un POC breve dovrebbe usare **X6 + ELK.js** su tre dataset sintetici: circa 30, 150 e 500
tabelle, includendo FK composite, più FK tra la stessa coppia, self-loop, tabelle isolate e hub
ad alto grado.
I criteri di accettazione dovrebbero misurare:
- tempo di layout nel worker e tempo al primo frame interattivo;
- fluidità di pan/zoom e uso memoria sul caso da 500 tabelle;
- correttezza di porte, archi paralleli, self-loop e FK composite;
- leggibilità nelle tre soglie di semantic zoom;
- navigazione completa da tastiera e comportamento con screen reader;
- qualità e portabilità dell'SVG esportato;
- assenza di codice, esempi o componenti soggetti a licenza commerciale.
Se X6 non raggiunge il livello di accessibilità richiesto senza un costo eccessivo, il confronto
finale va fatto con React Flow + lo stesso adapter ELK e lo stesso modello dati. In questo modo
si cambia renderer senza rifare API, semantica delle relazioni o pipeline di export.
@@ -1,246 +0,0 @@
# Gestione delle relationship: da ThothAI a ThothII
**Stato:** analisi e direzione funzionale/UX confermate nel *grill with docs*; non costituisce ancora un piano di implementazione.
**Revisione esaminata:** commit ThothII `f586152636b1fd653b0bc1d40b54be7f89bbd2bb`. I sorgenti legacy di ThothAI citati sotto sono versionati nello stesso repository, sotto `Thoth/ThothAI/`.
**Ambito:** import delle foreign key fisiche, inferenza di relationship logiche, modifica umana, pubblicazione verso il workflow NL→SQL e principali gap tra i due sistemi.
## Sintesi fattuale
1. In ThothAI le foreign key dichiarate nel database venivano importate e una procedura separata proponeva relationship basate sul nome dei campi e su una verifica dei valori. Entrambi i percorsi scrivevano però nello stesso modello `Relationship` (`Thoth/ThothAI/backend/thoth_core/dbmanagement.py:484-640`; `Thoth/ThothAI/backend/thoth_core/admin_models/admin_sqldb.py:571-826`; `Thoth/ThothAI/backend/thoth_core/models.py:488-530`).
2. Il ricordo di una relationship inferita persistita come `generated` non trova riscontro nel modello esaminato: `Relationship` contiene solo quattro foreign key verso tabelle e colonne, senza provenienza, stato, confidenza o flag `generated`. La procedura di inferenza calcola localmente pattern e tasso di validazione, ma non li salva (`Thoth/ThothAI/backend/thoth_core/models.py:488-530`; `Thoth/ThothAI/backend/thoth_core/admin_models/admin_sqldb.py:697-801`).
3. L'amministratore Django di ThothAI permetteva CRUD manuale sul medesimo insieme di relationship e verificava che gli endpoint appartenessero allo stesso database e alle tabelle selezionate; non distingueva visivamente o semanticamente relationship fisiche, inferite e manuali (`Thoth/ThothAI/backend/thoth_core/admin_models/admin_relationship.py:19-313`).
4. ThothII oggi separa già i due concetti: il catalogo backend conserva solo le foreign key fisiche dichiarate, mentre il core/harness possiede annotazioni di foreign key logiche curate e un comando che suggerisce candidate per nome, primary key e SQL osservato (`CONTEXT.md:291-300`; `docs/adr/0006-separate-physical-and-logical-relationships.md:3-8`; `harness/tht/cli/schema_cmd.py:203-299`).
5. I due mondi ThothII non sono ancora integrati: i record di Database Management non modificano il workflow NL→SQL e l'integrazione catalogo→core/schema-linking è ancora un gate di design esplicitamente differito (`PROJECT_STATE.md:116-124`; `PROJECT_STATE.md:161-165`).
## Evidenze ThothAI
### Foreign key ufficiali
Il percorso di import legge le foreign key dal database, limita l'import alle tabelle già presenti nel catalogo, crea le colonne mancanti, crea o riusa un record `Relationship` e aggiorna anche le stringhe denormalizzate `pk_field`/`fk_field` sulle colonne (`Thoth/ThothAI/backend/thoth_core/dbmanagement.py:484-640`, in particolare `:501-513`, `:526-591` e `:606-607`).
**Fatto:** una foreign key fisica diventa quindi un record applicativo, non rimane soltanto un fatto letto al momento dal database.
### Relationship inferite
La routine legacy dichiara sei famiglie di confronto tra il nome della colonna candidata e quello della primary key: corrispondenza esatta, snake case, kebab case, camel case, concatenazione e solo nome tabella (`Thoth/ThothAI/backend/thoth_core/admin_models/admin_sqldb.py:571-689`).
Per una candidata, la routine:
- legge fino a 20 valori distinti e non nulli dalla colonna candidata;
- verifica ogni valore contro la primary key bersaglio;
- accetta la candidata quando almeno il 70% dei valori esaminati trova riscontro;
- crea o recupera un normale `Relationship` e aggiorna i campi denormalizzati delle tabelle (`Thoth/ThothAI/backend/thoth_core/admin_models/admin_sqldb.py:697-801`).
**Correzione documentale:** un commento parla di campionamento casuale, ma la query mostrata non contiene un ordinamento casuale; il comportamento verificabile dal codice è “fino a 20 valori distinti e non nulli”, non un campione statisticamente casuale (`Thoth/ThothAI/backend/thoth_core/admin_models/admin_sqldb.py:697-716`).
**Rischio legacy:** nomi di tabelle e colonne sono interpolati direttamente in SQL in questo percorso. Portare la logica letteralmente in ThothII riprodurrebbe un problema di quoting/sicurezza e richiederebbe inoltre una policy esplicita per l'accesso ai valori del DWH (`Thoth/ThothAI/backend/thoth_core/admin_models/admin_sqldb.py:709-750`).
### Un solo modello per tre origini
Il modello `Relationship` legacy contiene soltanto `source_table`, `target_table`, `source_column` e `target_column`, più metodi di rappresentazione/aggiornamento. Non contiene campi per origine, algoritmo, evidenza, confidenza, approvazione o disabilitazione, né un vincolo di unicità dichiarato nel modello (`Thoth/ThothAI/backend/thoth_core/models.py:488-530`).
L'admin consente aggiunta, modifica e cancellazione ordinarie e valida la coerenza tra database, tabelle e colonne (`Thoth/ThothAI/backend/thoth_core/admin_models/admin_relationship.py:30-38`, `:125-157`, `:218-245`).
**Conclusione fattuale:** relationship importate, inferite e create manualmente convergono nello stesso tipo persistito. Dal record finale non è possibile ricostruirne con certezza l'origine. Il termine `generated`, se usato nell'interfaccia o nel linguaggio operativo dell'epoca, non era una qualificazione persistita dal modello esaminato.
### Uso nella comprensione dello schema
La generazione M-Schema conserva colonne PK/FK e ricostruisce la sezione `【Foreign keys】` analizzando le stringhe denormalizzate `fk_field` (`Thoth/ThothAI/frontend/sql_generator/helpers/main_helpers/main_generate_mschema.py:25-61`, `:94-101`, `:160-187`).
**Fatto:** le relationship curate nel catalogo legacy influenzavano la rappresentazione dello schema consumata dal generatore SQL, anche se tramite una proiezione denormalizzata.
## Stato attuale di ThothII
### Catalogo backend: relationship fisiche
Il modello di dominio corrente distingue esplicitamente:
- **Physical Relationship:** vincolo dichiarato nel database;
- **Catalog Relationship:** copia persistita di quel fatto fisico, non creabile o modificabile manualmente ma eliminabile per cleanup;
- **Logical Relationship:** relazione semantica curata o inferita, con ownership e lifecycle separati (`CONTEXT.md:291-300`; `docs/adr/0006-separate-physical-and-logical-relationships.md:3-8`).
La migrazione del catalogo crea `catalog_relationships` e le coppie ordinate in `catalog_relationship_columns`. L'identità della relazione fisica è `(source_table_id, constraint_name)`; non esistono campi di origine, stato o confidenza (`backend/src/catalog/migrations/003_catalog_schema_sync.ts:42-74`). Il tipo TypeScript è solo strutturale e le descrizioni generate riguardano esclusivamente tabelle o colonne (`backend/src/catalog/types.ts:123-152`).
L'introspezione PostgreSQL legge soltanto `pg_constraint` con `contype = 'f'` e mantiene l'ordine delle coppie per le chiavi composite (`backend/src/catalog/schema-introspector.ts:110-171`; `backend/src/catalog/schema-introspector.ts:360-416`; `docs/contracts/catalog-schema-snapshot.md:40-53`). La sincronizzazione autoritativa crea, aggiorna o elimina le copie fisiche in base allo snapshot (`backend/src/catalog/repository.ts:1165-1218`; `backend/src/catalog/repository.ts:1316-1393`).
Le API e la UI espongono lettura, sync e cleanup in massa, non CRUD logico per singola relationship (`backend/src/routes/catalog-schema.ts:25-37`; `backend/src/routes/catalog-schema.ts:134-150`; `frontend/src/api/catalog-databases.ts:452-489`; `frontend/src/shell/database-management/DatabaseRelationships.tsx:114-123`; `frontend/src/shell/database-management/DatabaseRelationships.tsx:160-225`).
**Effetto rilevante:** il cleanup è intenzionalmente reversibile tramite una sync successiva e può lasciare temporaneamente un catalogo incompleto (`docs/adr/0008-allow-manual-catalog-metadata-cleanup.md:7-16`). Un test di integrazione ammette anche una relationship rimasta senza coppie di colonne dopo la cancellazione delle colonne catalogate (`backend/test/catalog-repository.integration.test.ts:194-264`, in particolare `:246-252`). Un futuro consumer non può quindi assumere che ogni stato intermedio del catalogo sia pubblicabile così com'è.
### Core/harness: relationship logiche e suggerimenti
Il modello M-Schema del core possiede già `TableAnnotation.foreign_keys`, descritte come foreign key logiche curate e unite alle foreign key fisiche (`harness/tht/mschema/models.py:35-39`; `harness/tht/mschema/models.py:76-84`). Il renderer fonde i due insiemi e li presenta insieme nella sezione `【Foreign keys】` (`harness/tht/mschema/render.py:9-20`; `harness/tht/mschema/render.py:46-80`).
Il comando `schema suggest-fks` costruisce candidate da:
- uguaglianze trovate in SQL, quando esattamente un lato è una primary key;
- convenzione speciale `*time_key → dim_time.<PK singola>`;
- stesso nome tra colonna e primary key a proprietario univoco;
- assunzioni esplicite per disambiguare;
- esclusione di nomi PK generici come `id`, `key` e `code` e dei proprietari ambigui (`harness/tht/cli/schema_cmd.py:203-299`; `harness/tht/mschema/fkmine.py:1-58`).
Il comando può produrre un documento candidato e, con `--write`, aggiungere annotazioni mancanti in modo idempotente; il messaggio stesso chiede revisione manuale (`harness/tht/cli/schema_cmd.py:406-429`; `harness/tht/cli/schema_cmd.py:519-540`; `harness/tests/test_schema_fk_annotations.py:131-146`; `harness/tests/test_schema_fk_annotations.py:454-470`).
**Fatto:** ThothII non parte da zero sull'inferenza. Possiede già un motore più conservativo, basato su PK univoche e SQL osservato, ma non conserva per ogni relazione origine, evidenza, frequenza o confidenza. Il miner conta le occorrenze internamente, ma il modello candidato non promuove quel conteggio a lifecycle persistito (`harness/tht/mschema/fkmine.py:1-58`; `harness/tht/mschema/models.py:35-39`).
**Rischi strutturali già visibili:**
- `ForeignKey` ammette liste di colonne, ma non valida che source e target abbiano la stessa cardinalità; il renderer usa `zip`, quindi una relazione malformata può essere troncata silenziosamente (`harness/tht/mschema/models.py:35-39`; `harness/tht/mschema/render.py:76-78`).
- il merge evita duplicati rispetto alle FK fisiche iniziali, ma non aggiorna l'insieme `seen` dopo aver aggiunto un'annotazione; due annotazioni logiche uguali possono sopravvivere al merge (`harness/tht/mschema/render.py:9-20`).
- il catalogo fisico supporta coppie composite ordinate, mentre le euristiche correnti e legacy sono sostanzialmente unary. La semantica delle candidate composite resta da decidere (`backend/src/catalog/migrations/003_catalog_schema_sync.ts:56-74`; `harness/tht/cli/schema_cmd.py:203-299`).
### Revisione e pubblicazione correnti
Le annotazioni canoniche del workspace sono un blob Git. Il flusso pubblico produce candidate, verifica le annotazioni e richiede un'accettazione umana esplicita dopo commit/push/pull; l'accettazione registra digest del candidato e delle annotazioni, revisione e blob (`docs/contracts/workspace-preprocessing-cli.md:89-106`; `backend/src/workspaces/preprocessing-service.ts:202-297`). Il preprocessing successivo procede solo se il digest accettato coincide con le annotazioni correnti (`backend/src/workspaces/preprocessing-service.ts:335-388`).
**Limite fattuale:** il record di review conserva digest, revisione e blob, ma non attore, motivazione o decisioni per singola candidata (`backend/src/workspaces/preprocessing-state.ts:67-73`).
Il runtime usa le annotazioni quando renderizza lo schema, ma la ricerca vettoriale indicizza record di tabelle e colonne senza contenuto esplicito delle relationship (`harness/tht/vectorstore/records.py:106-138`; `harness/tht/cli/search_cmd.py:233-273`). Inoltre la vista colonne usata in F4 continua a leggere i commenti fisici, non le annotazioni (`harness/tht/cli/schema_cmd.py:592-621`).
La review dei join durante una sessione è distinta dalla curatela globale: il reviewer conferma l'insieme dei join oppure chiede una revisione completa; non modifica la mappa canonica delle relationship (`harness/.pi/skills/tht-sessione/SKILL.md:301-341`; `frontend/src/widgets/JoinReviewWidget.tsx:5-84`). Il modello di sessione registra join come due stringhe e una decisione opzionale, senza ID stabile della relationship o coppie ordinate strutturate (`harness/tht/session/models.py:147-170`).
## Delta e rischi da sottoporre al grill
| Tema | Fatto documentato | Delta/rischio aperto |
|---|---|---|
| Origine | ThothAI perdeva l'origine; ThothII separa fisico e logico a livello concettuale | Decidere quale provenienza debba essere persistita per manuale, euristica, SQL osservato e import fisico |
| `generated` | Non era un flag del modello ThothAI; in ThothII “Generated Description” è già un termine del catalogo (`CONTEXT.md:302-311`) | Usare `generated` anche per relationship potrebbe creare ambiguità terminologica |
| Cancellazione utente | In ThothAI era CRUD sul record unico; in ThothII il cleanup fisico viene ricostruito dalla sync | “Eliminare” può significare cancellare una relazione logica, sopprimere un fatto fisico per il core oppure pulire temporaneamente la copia catalogata: sono operazioni diverse |
| Inferenza | ThothAI usava sei pattern e valori DWH; ThothII usa PK univoche, nomi e SQL osservato | Stabilire se sostituire, integrare o non portare il campionamento dei valori; servono policy di dati sensibili, query read-only, quoting e limiti |
| Approvazione | ThothII dispone di review Git/digest dell'intero artefatto | Mancano decisioni per candidata, motivazione, attore, sticky rejection, versione algoritmo e gestione dello stale |
| Compositi | Il catalogo fisico conserva coppie ordinate; le annotazioni accettano liste | Mancano invarianti forti e una strategia di inferenza/review per join compositi |
| Pubblicazione | Catalogo management e runtime core sono oggi separati | Va stabilito se pubblicare tutto, una selezione esplicita o una revisione immutabile; il catalogo può essere incompleto durante cleanup/sync |
| UI e ownership | UI catalogo fisico read-only; curatela logica via CLI/Git; review join per sessione separata | Va scelto chi cura la mappa e in quale superficie, senza confondere amministrazione globale e correzione della singola sessione |
| Retrieval | Le relationship entrano nel render M-Schema ma non nei record vettoriali | Va deciso se e come influenzano selezione tabelle, ranking e descrizioni, oltre al rendering finale |
| Drift | Sync fisica è autoritativa; annotazioni sono una revisione separata | Servono semantiche per endpoint rinominati/eliminati, candidate stale, orphan e riapprovazione |
Le analisi già presenti nel repository trattano l'integrazione catalogo→core, la selezione pubblicabile e la riparazione degli indici come questioni ancora aperte; le loro proposte non sono decisioni implementate (`docs/research/2026-08-23-thothii-metadata-publication-qdrant-seams.md:20-43`; `docs/research/2026-08-23-thothii-metadata-publication-qdrant-seams.md:239-301`). Anche il piano di migrazione rinvia il lifecycle/admin delle relationship logiche (`docs/plans/2026-08-26-metadata-catalog-from-thothai.md:681-691`).
## Direzione confermata nel grill
Il 31 agosto 2026 è stato concordato il seguente flusso minimo:
1. Le foreign key dichiarate continuano a essere sincronizzate come Catalog Relationship fisiche.
2. Le foreign key ipotetiche sono salvate una sola volta come Logical Relationship fra colonna
sorgente e colonna destinazione; le tabelle sono ricavate dalle colonne e la UI le presenta nel
relativo contesto tabella.
3. Una relationship inferita è marcata `generated`; una relationship aggiunta dall'utente non lo è.
4. La cancellazione logica conserva il record e impedisce a una ricostruzione di riattivarlo.
5. La cancellazione fisica rimuove il record; una ricostruzione può ricrearlo se viene nuovamente
inferito.
6. La ricostruzione è additiva: conserva le relationship attive già presenti, non rimuove quelle
non più inferibili e non modifica le relationship manuali.
7. L'inferenza usa nomi, primary key e compatibilità dei tipi. Non campiona valori del DWH.
8. La relationship è l'unica fonte di verità: non viene duplicata in stringhe `fk_field` sulle
colonne.
9. I nomi vengono confrontati senza distinzione fra maiuscole/minuscole e normalizzando snake case,
kebab case e camel case. La regola riconosce anche casi come `user_id → users.id`, richiede tipi
compatibili e una sola destinazione possibile; riconosce inoltre un nome PK non generico con un
unico proprietario e la convenzione `*time_key → dim_time.<PK singola>`. Le colonne sorgenti
possono appartenere a PK composite, mentre nomi generici isolati come `id`, `key`, `code` e `pk`
non costituiscono evidenza. I casi ambigui vengono ignorati e non si usa fuzzy matching o un LLM.
10. La ricostruzione parte da un'azione amministrativa esplicita `Rebuild generated relationships`,
separata dalla sincronizzazione dello schema.
11. Le relationship cancellate logicamente restano consultabili tramite filtro e possono essere
riattivate con un'azione `Restore`.
Non restano decisioni di dominio aperte per il flusso minimo. La progettazione UX e il seam tecnico
sono descritti nelle sezioni seguenti.
## Lacuna UX emersa nel grill
La vista Fleet `Relationships` esiste, ma nella UI di produzione non ha oggi un punto di ingresso
raggiungibile. La riga del database espone sincronizzazione, tabelle, dettagli, modifica e rimozione,
ma non le relationship; inoltre i tab `Overview / Tables / Relationships` appartengono soltanto alla
presentazione legacy. Il test di navigazione esistente esercita anch'esso la presentazione legacy,
non quella Fleet (`frontend/src/shell/database-management/DatabaseGrid.tsx:122-180`;
`frontend/src/shell/database-management/DatabaseForm.tsx:442-475`;
`frontend/src/shell/database-management/DatabaseForm.tsx:517-546`;
`frontend/src/shell/DatabaseManagementPage.test.tsx:2635-2703`).
Se aperta programmaticamente, la vista mostra soltanto `Physical relationships` in sola lettura. La
toolbar contiene ricerca, conteggio, `Refresh`, un selettore azione con esecuzione esplicita e
`Sync history`; la griglia offre unicamente `Details`, che apre il drawer della relationship fisica.
Sono assenti ingresso visibile, aggiunta manuale, ricostruzione delle generated relationship, origine,
stato attivo/cancellato, cancellazione logica, cancellazione fisica e ripristino
(`frontend/src/shell/database-management/DatabaseRelationships.tsx:114-225`).
I pattern Fleet già consolidati da riutilizzare sono:
- una sola griglia nel livello corrente, con breadcrumb e controllo Back;
- azioni di pagina nel selettore `Choose an action…` con `Run action` e motivo visibile quando
indisponibili;
- azioni della singola riga nella colonna finale fissata a destra;
- form e dettagli in un drawer modeless che restituisce il focus al controllo di origine;
- conferme distruttive inline o nel drawer, non tramite una nuova pagina;
- toast per accettazione o errore e feedback persistente soltanto per le operazioni lunghe;
- card di errore con Retry ed empty state che indica la prossima azione possibile.
ThothAI non offre un modello UX da copiare. L'inferenza è nascosta fra 21 bulk action della lista
database, non mostra avanzamento in tempo reale e restituisce soltanto messaggi a fine richiesta. Il
CRUD manuale vive in un'altra schermata Django Admin, non distingue origine o stato, offre soltanto
la cancellazione fisica e il form di aggiunta ha una validazione server strutturalmente incoerente
con le select popolate dal browser
(`Thoth/ThothAI/backend/thoth_core/admin_models/admin_sqldb.py:252-274`;
`Thoth/ThothAI/backend/thoth_core/admin_models/admin_relationship.py:58-119`;
`Thoth/ThothAI/backend/thoth_core/admin_models/admin_relationship.py:218-313`).
## UX confermata
Il 31 agosto 2026 sono state confermate le seguenti scelte:
1. La colonna Actions della riga database espone un accesso diretto `Relationships`, accanto a
`Tables`. La vista conserva breadcrumb e controllo `Back to databases` esistenti.
2. La pagina presenta una sola griglia `Relationship map`, contenente relationship `Physical`,
`Generated` e `Manual`. Le colonne sono Source table, Source column, Target table, Target column,
Origin, Status e Actions. Constraint e regole update/delete rimangono nel drawer delle FK fisiche.
3. Un filtro visibile seleziona `Active`, `Excluded` o `All`; il valore predefinito è `Active`. Le FK
fisiche sono consultabili ma non modificabili da questa vista.
4. `Add relationship` è il pulsante primario visibile nella toolbar. Apre il drawer standard con i
quattro campi Source table, Source column, Target table e Target column e i comandi `Cancel` e
`Add relationship`. Il flusso minimo gestisce una sola coppia di colonne e mostra la validazione
accanto al campo interessato.
5. `Rebuild generated relationships` entra nell'attuale selettore `Choose an action…`, insieme a
`Synchronize physical relationships` e `Synchronize full schema`, con esecuzione esplicita tramite
`Run action`. `Refresh` ricarica la griglia; `Sync history` resta riservato alla sincronizzazione
fisica.
6. Il drawer di dettaglio contiene le azioni sulle relationship logiche. `Exclude` realizza la
cancellazione logica, `Delete permanently` quella fisica e `Restore` riattiva una relationship
esclusa. La conferma avviene nel drawer e spiega rispettivamente che la ricostruzione non
riattiverà un record escluso e potrà invece ricreare un record eliminato definitivamente.
7. La ricostruzione non introduce un nuovo sistema di job o di storico. Durante l'esecuzione mostra
`Rebuilding…`; al termine un toast riporta added, already present, excluded e ambiguous. Add,
Exclude, Delete permanently e Restore producono toast specifici; gli errori mantengono aperto il
contesto corrente. L'empty state propone di ricostruire dai nomi o aggiungere manualmente.
8. L'inferenza non usa AI. È codice deterministico nel backend del catalogo, basato su normalizzazione
dei nomi, primary key/unicità, compatibilità dei tipi e assenza di ambiguità. Non usa LLM,
embedding o campionamento dei dati. L'AI consuma la mappa risultante per comprendere lo schema,
ma non la costruisce.
## Seam tecnico confermato
Il Catalog PostgreSQL è l'unica fonte modificabile delle Logical Relationship. Le relationship
fisiche e logiche restano in modelli distinti, coerentemente con ADR-0006, mentre un servizio profondo
espone a API e UI una sola mappa discriminata per origine e stato.
All'avvio o alla ripresa di una sessione, il backend materializza le sole relationship attive in una
snapshot JSON canonica e immutabile, collegata alla stessa lease della configurazione runtime. La
snapshot comprende anche le FK fisiche e conserva l'ordine delle coppie composite. Se due record hanno
gli stessi endpoint, la FK fisica ha precedenza.
Quando la snapshot è presente, l'harness la usa come fonte esclusiva delle relationship e continua a
leggere dalle annotazioni Git-pinned soltanto descrizioni, sinonimi, concetti e altri metadati. Non
scrive `annotations.yaml` e non interroga direttamente il Catalog. Una snapshot dichiarata ma assente,
invalida o incoerente con lo schema fisico fallisce esplicitamente; un runtime legacy che non dichiara
la snapshot mantiene il precedente comportamento di compatibilità.
Questa proiezione non è un secondo store: non può essere modificata, viene eliminata insieme alla
configurazione runtime e una sessione Pi vede una mappa stabile per tutta la propria vita. La decisione
duratura è registrata in ADR-0012.
La proiezione e la ricostruzione sono ammesse soltanto dopo una sincronizzazione completa della
versione corrente del database e vengono serializzate con le mutazioni del catalogo. Un catalogo mai
sincronizzato, reso stale da una modifica della configurazione o invalidato da metadata cleanup non
può quindi diventare accidentalmente la fonte esclusiva del runtime. Il cleanup esplicito di una
tabella o colonna endpoint è anche il confine distruttivo del tombstone: rimuove definitivamente la
relationship esclusa, perché conservarla richiederebbe una seconda identità testuale denormalizzata.
@@ -1,219 +0,0 @@
# Licenza del core e compatibilità del repository
Data della verifica: 2026-09-02
Ticket: [Valutare licenza del core e compatibilità legale del repository](https://git.tylconsulting.it/mptyl/ThothII/issues/22)
> Questa è un'analisi tecnica delle licenze e della provenienza osservabile nel repository, non
> consulenza legale. Prima della prima distribuzione pubblica o commerciale, un professionista
> qualificato nella giurisdizione di TYL Consulting e dei clienti deve validare confini dei lavori
> derivati, titolarità, brevetti, marchi, EULA e obblighi di redistribuzione.
## Decisione proposta
Adottare **Apache License 2.0** per la Community Edition e una licenza commerciale separata per
l'Enterprise Edition. Apache-2.0 è preferibile a MIT per un prodotto enterprise perché contiene
una concessione brevettuale esplicita con clausola di cessazione, disciplina i contributi, tutela i
marchi e descrive esplicitamente come non derivati i lavori separabili o che si limitano a collegarsi
alle interfacce. Rimane una licenza permissiva: consente uso commerciale, modifica e distribuzione
proprietaria delle modifiche nel rispetto degli obblighi della licenza.
**La pubblicazione non è ancora pronta.** Due problemi devono essere risolti prima di applicare
Apache-2.0 al repository:
1. `yake==0.7.3` è importato direttamente dal runtime Python e il file `LICENSE` dell'artefatto
ufficiale contiene AGPL-3.0, benché i metadati PyPI dichiarino in modo incoerente LGPL/GPL.
Va sostituito con una dipendenza permissiva, rimosso, oppure coperto da una licenza commerciale
verificata. L'isolamento va considerato solo dopo parere legale; nel codice attuale non c'è
isolamento, ma un normale `import yake`.
2. [`harness/tht/vendor/VENDORED.md`](../../harness/tht/vendor/VENDORED.md) dichiara Apache-2.0 per
`thoth_lsh.py`, mentre l'artefatto upstream `thoth-dbmanager==0.7.4` su PyPI contiene una licenza
MIT e il relativo copyright. Occorre correggere la provenienza e conservare il testo MIT, oppure
documentare formalmente una nuova licenza da parte di tutti i titolari effettivi.
## MIT, Apache-2.0 e copyleft
| Tema | MIT | Apache-2.0 | GPLv3 / AGPLv3 |
| --- | --- | --- | --- |
| Uso commerciale e codice proprietario | Sì; obbligo essenziale di conservare copyright e licenza | Sì; consente termini diversi sulle modifiche, preservando gli obblighi Apache | Il codice coperto e le opere combinate distribuite devono rispettare il copyleft |
| Brevetti | Nessuna concessione brevettuale espressa nel testo | Concessione espressa sui claim necessariamente violati dal contributo; termina in caso di determinata causa brevettuale | GPLv3/AGPLv3 contengono condizioni brevettuali proprie |
| Redistribuzione | Conservare copyright e testo MIT nelle copie o porzioni sostanziali | Fornire la licenza, marcare i file modificati, conservare notice pertinenti e riprodurre `NOTICE` quando presente | Fornire il Corresponding Source e non imporre ulteriori restrizioni incompatibili |
| Moduli proprietari | Consentiti; il testo non definisce il confine fra opere | Consentiti; la definizione esclude lavori separabili o che si limitano a collegarsi alle interfacce | Confine fattuale e giuridico. L'aggregazione di opere indipendenti è distinta da un programma combinato |
| Uso via rete | Nessun obbligo specifico | Nessun obbligo specifico | AGPLv3 §13 richiede offrire il Corresponding Source agli utenti remoti di una versione modificata |
| Contributi | Nessuna disciplina esplicita nel breve testo | §5: contributi intenzionalmente sottoposti sono Apache-2.0 salvo dichiarazione o accordo separato | Contributi al lavoro coperto devono poter essere distribuiti sotto la licenza applicabile |
| Marchi | Non disciplinati | §6 non concede diritti sui marchi salvo uso descrittivo e `NOTICE` | Non sono automaticamente una licenza del marchio |
Fonti primarie: [testo MIT dell'OSI](https://opensource.org/license/mit/),
[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0),
[GPLv3](https://www.gnu.org/licenses/gpl-3.0.html),
[AGPLv3](https://www.gnu.org/licenses/agpl-3.0.html) e
[FAQ GNU su plugin e aggregazione](https://www.gnu.org/licenses/gpl-faq.html#GPLAndPlugins).
### Perché non scegliere MIT
MIT renderebbe semplice l'adozione e il riuso proprietario, ma non dà a compratori e contributori
la stessa chiarezza di Apache-2.0 sui brevetti, sui contributi e sui marchi. La brevità non compensa
questa minore precisione per un prodotto venduto a IT department enterprise. MIT resta compatibile
con l'obiettivo open-core, ma Apache-2.0 gestisce meglio il rischio del progetto.
### Perché non scegliere copyleft per il core
Il copyleft aumenterebbe la reciprocità, ma rende più delicato il confine con l'Enterprise Edition e
può creare una revisione legale più onerosa per i clienti. AGPL è inoltre poco allineata alla scelta
commerciale già fissata: installazione interamente nel trust boundary del cliente e funzionalità
Enterprise proprietarie. Se in futuro la priorità diventasse impedire fork proprietari del core,
GPL/AGPL andrebbero valutate in una decisione distinta, non introdotte accidentalmente da una
dipendenza.
## Confine Community / Enterprise
Il confine raccomandato è tecnico oltre che contrattuale:
- Community Edition in repository pubblico, licenza Apache-2.0, workflow fondamentale completo e
interfacce di estensione pubbliche;
- Enterprise Edition in repository, package e immagini separati, con EULA commerciale e SBOM
autonomo;
- integrazione attraverso API/versioned contracts o process boundary. Anche se Apache-2.0 consente
linking e opere separabili, questa separazione riduce errori di packaging e rende auditabile il
perimetro venduto;
- una distribuzione Enterprise può aggregare componenti Community, ma deve continuare a fornire
Apache-2.0 e tutti i notice della Community. L'EULA non deve revocare ai clienti i diritti
Apache sui componenti Community;
- marchio e logo `ThothII` richiedono una policy separata: la licenza del codice non deve essere
usata come licenza del brand.
Il giudizio finale sul fatto che un plugin concreto sia un'opera indipendente o derivata dipende
dalla sua implementazione e dalla legge applicabile: è un punto da sottoporre al legale quando
esisterà il primo modulo Enterprise.
## Inventario del repository
L'inventario deriva dai file tracciati nel commit `ae053961`, dai lockfile e, per i casi anomali,
dagli artefatti ufficiali dei registry. I metadati dei registry non sostituiscono i testi di licenza
inclusi nelle distribuzioni: l'incoerenza di YAKE lo dimostra.
### Stato della licenza propria
- Non esistono `LICENSE`, `NOTICE`, `COPYING`, `AUTHORS` o `THIRD_PARTY_NOTICES` alla radice.
- I manifest `backend/package.json`, `frontend/package.json`, `harness/package.json`,
`docker/pi-runtime/package.json`, `tools/html-to-pptx/package.json` e
`harness/pyproject.toml` non dichiarano la licenza del progetto.
- La cronologia Git contiene più identità (`mptyl`, `Marco Pancotti`, `User`, `Codex`, Gitea
Actions) e trailer `Co-authored-by` di strumenti AI, ma nessun `Signed-off-by`. La dichiarazione
del proprietario è che il codice appartiene interamente a TYL Consulting; prima del rilascio
occorre collegare documentalmente gli alias al titolare e conservare i termini applicabili agli
output generati.
### JavaScript / TypeScript
Cinque lockfile npm sono presenti e nessun manifest dichiara la licenza del package ThothII.
| Superficie | Package bloccati | Risultato dei metadati nel lock |
| --- | ---: | --- |
| backend | 366 | MIT 296, ISC 27, Apache-2.0 19, BSD-3-Clause 16, BlueOak 4, senza campo 3, Unlicense 1 |
| runtime Pi | 140 | MIT 67, Apache-2.0 46, BSD-3-Clause 13, ISC 8, BlueOak 5, 0BSD 1 |
| frontend | 874 | prevalentemente MIT; inoltre OFL-1.1, CC-BY-4.0, Python-2.0, BSD, ISC e licenze alternative |
| harness (dev) | 1 | MIT |
| exporter PPTX | 22 | prevalentemente MIT; `jszip` offre l'alternativa MIT oppure GPL-3.0-or-later |
Elementi che richiedono notice o verifica esplicita:
- `@fontsource-variable/geist@5.2.9` è OFL-1.1 e viene distribuito nel frontend; OFL richiede che
copyright e licenza accompagnino ogni copia del font. Vedi
[OFL 1.1](https://openfontlicense.org/open-font-license-official-text/).
- `dompurify@3.4.11` offre `MPL-2.0 OR Apache-2.0`: documentare la scelta Apache-2.0.
- `jszip@3.10.1` offre `MIT OR GPL-3.0-or-later`: documentare la scelta MIT.
- `khroma`, `buildcheck`, `cpu-features` e `ssh2` non hanno il campo `license` nei lock/registry
esaminati: lo scanner di release deve leggere la licenza inclusa nei tarball, non assumere MIT.
- `@earendil-works/pi-coding-agent@0.80.3`, AG Grid Community, Fastify, Kysely e OpenID Client
risultano MIT nei metadati ufficiali npm.
Fonti: lockfile locali e metadata del [registry npm](https://registry.npmjs.org/).
### Python
`harness/uv.lock` blocca 85 package, incluso il workspace `tht`; i metadata ufficiali PyPI degli 84
package esterni riportano soprattutto MIT, Apache e BSD. Le eccezioni sono:
- **YAKE 0.7.3: blocker.** Il file `LICENSE` nel source distribution ufficiale è AGPL-3.0 e offre
separatamente una licenza commerciale. I metadata dichiarano invece `LGPLv3` e classificano
GPLv3. Il codice lo importa in
[`harness/tht/session/store.py`](../../harness/tht/session/store.py). Fonti:
[tag upstream v0.7.3](https://github.com/INESCTEC/yake/blob/v0.7.3/LICENSE) e
[release PyPI](https://pypi.org/project/yake/0.7.3/).
- `psycopg2-binary==2.9.12`: LGPLv3 con eccezioni specifiche nel file `LICENSE` dell'artefatto.
- `certifi==2026.7.22`: MPL-2.0 per il bundle di CA modificato.
- `tqdm==4.70.0`: licenza composita con parti MPL-2.0 e MIT, non una semplice alternativa;
preservare il suo `LICENCE`.
- `fsspec==2026.7.0`: il metadata PyPI è vuoto, ma il source distribution include BSD-3-Clause.
`docs/requirements.lock` contiene 31 dipendenze di build documentale; fra queste `certifi` e
`pathspec` riportano MPL-2.0. Sono build-time, ma vanno comunque incluse nella scansione del processo
di release se vengono ridistribuite in immagini o toolchain.
### Go
`tools/tht/go.mod` ha 12 dipendenze dichiarate e il grafo scaricato contiene solo famiglie
permissive osservate nei testi root: Apache-2.0, MIT, BSD-2-Clause, BSD-3-Clause e ISC. Alcuni moduli
Apache includono anche `NOTICE`, che deve essere propagato nell'immagine/binario distribuito.
`tools/dwh-auth/go.mod` non dichiara dipendenze esterne. `go.sum` è presente solo per `tools/tht`.
### Codice vendorizzato e asset
- `harness/tht/vendor/thoth_lsh.py` è copiato e modificato da `thoth-dbmanager==0.7.4`.
L'artefatto PyPI ufficiale contiene MIT, mentre il repository lo marca Apache-2.0. È necessario
risolvere la discrepanza e includere l'attribuzione corretta. Fonti:
[metadata PyPI](https://pypi.org/pypi/thoth-dbmanager/0.7.4/json) e
[`VENDORED.md`](../../harness/tht/vendor/VENDORED.md).
- `frontend/prototypes/database-management/assets/thoth-cosmic-archive.png` è documentato come
output del tool ImageGen. Prima di pubblicarlo sotto Apache-2.0, conservare evidenza del contratto
applicabile e della decisione del titolare di includerlo nella Community Edition.
- `presentations/thothii-congresso.pptx` è generato dalle sorgenti HTML/JSON locali e contiene PNG
rasterizzati. L'HTML carica Fraunces e Manrope da Google Fonts: conservare le rispettive licenze
font e decidere se la presentazione appartiene davvero al source release del prodotto.
- Il solo font binario individuato nel bundle applicativo arriva da `@fontsource-variable/geist` ed
è OFL-1.1. Non sono presenti altri font, PDF, archivi o WASM tracciati; è presente un PNG e un PPTX.
### Immagini container e modelli
Il deploy usa immagini pin per Node, Python, Go, nginx-unprivileged, PostgreSQL, Qdrant e Ollama.
Il bootstrap scarica dinamicamente `qwen3-embedding:0.6b`. I lockfile applicativi non inventariano i
package OS delle immagini né la licenza dei pesi scaricati. Prima della distribuzione occorre
generare un SBOM per ogni immagine finale, archiviare i notice dei base image e acquisire la licenza
del modello esatto; il nome del modello senza digest non è una prova sufficiente.
## Contributi e copyright
Per l'open-core proposto non è necessario acquisire il copyright di ogni contributore se
l'Enterprise resta un'opera separata. Apache-2.0 §5 fornisce un default inbound=outbound, ma non è
un trasferimento di titolarità.
Raccomandazione iniziale:
1. `CONTRIBUTING.md` con Apache-2.0 inbound=outbound e Developer Certificate of Origin;
2. sign-off obbligatorio e verifica automatica;
3. CLA soltanto se TYL vuole in futuro dual-license **gli stessi file** Community, cambiare licenza
unilateralmente o ottenere concessioni brevettuali/di relicensing ulteriori;
4. registro di provenienza per codice, asset, modelli e dipendenze vendorizzate;
5. policy marchio separata dalla licenza del codice.
Fonti primarie: [Apache-2.0 §5](https://www.apache.org/licenses/LICENSE-2.0),
[Developer Certificate of Origin](https://developercertificate.org/) e
[Apache Contributor Agreements](https://www.apache.org/licenses/contributor-agreements.html).
## Gate prima della pubblicazione
1. Sostituire YAKE o ottenere e verificare una licenza commerciale compatibile con entrambe le
edizioni; la sostituzione è preferibile perché YAKE serve soltanto a derivare il nome sessione.
2. Risolvere la provenienza MIT/Apache di `thoth_lsh.py` e correggere `VENDORED.md`.
3. Fissare per iscritto l'elenco di directory Community; spostare Enterprise in artefatti separati.
4. Aggiungere `LICENSE` Apache-2.0, copyright TYL, `NOTICE`, `THIRD_PARTY_NOTICES` e campi SPDX nei
manifest pubblicati.
5. Generare SBOM e license report dai lockfile e dalle immagini; fallire la release su licenze non
ammesse, metadata mancanti o modelli senza licenza acquisita.
6. Verificare titolarità degli alias Git e condizioni degli output AI; decidere se prototype e
presentazione fanno parte del rilascio.
7. Far validare da un legale: EULA Enterprise, separazione dei moduli, politica contributi/CLA,
marchio, notice e dipendenze LGPL/MPL/OFL.
Con questi gate chiusi, Apache-2.0 offre il miglior equilibrio fra reputazione, adozione enterprise,
chiarezza brevettuale e possibilità di vendere capacità superiori in moduli proprietari separati.
@@ -1,218 +0,0 @@
# Local sensitive-column classifier: TypeScript or Python
**Date:** 2026-09-02
**Scope:** library/runtime choice for a local classifier without a generative LLM. Synthetic data generation and
model-boundary policy for description generation are deliberately out of scope.
## Recommendation
Implement the mandatory classifier **inside the TypeScript backend**, with no learned model in its
mandatory path. The agreed policy is dominated by deterministic work:
type/declared-size rules, bounded string normalization, patterns, checksums, exact dictionaries,
context terms, and "one match makes the column sensitive". TypeScript is fully adequate for that
work and keeps the classifier in the process which already owns the catalog, source connectors,
run lifecycle, cancellation, and the human override.
Use small, focused dependencies rather than a general NLP framework:
- `validator` for syntax/checksum-oriented recognizers such as email, IP, MAC, IBAN, BIC, payment
cards, and locale-aware Italian tax IDs, passports, identity cards and VAT numbers; its official
API exposes these validators individually, so unused validators need not be imported
([validator.js source and API](https://github.com/validatorjs/validator.js/)).
- `libphonenumber-js/max` for international telephone parsing and digit-pattern validation. The
project's `max` metadata is intentionally more precise than its default/minimal metadata
([libphonenumber-js validation documentation](https://github.com/catamphetamine/libphonenumber-js/blob/master/README.md#isvalid-boolean)).
- ThothII-owned recognizers for identifiers not covered adequately by a selected validator (for
example the Italian driving licence) and organization-specific identifiers, each with a version,
tests, context words, and checksum/semantic validation where the identifier defines one.
- A safe regular-expression boundary for administrator-authored rules. `node-re2` offers a mostly
`RegExp`-compatible, ReDoS-resistant engine and a set API for matching many patterns, but is a
native addon that downloads or builds a binary during installation. That packaging cost must be
tested against the project images before adoption
([node-re2 README](https://github.com/uhop/node-re2/blob/master/README.md)). If arbitrary regular
expressions are not exposed, ThothII can instead keep a reviewed built-in pattern set and bound
every inspected value.
Do **not** add NLP.js, spaCy, Presidio, or a transformer merely to implement regexes and dictionaries.
They do not make deterministic identifiers more semantically understandable. In particular, NLP.js
documents useful Italian tokenization plus enum/regex/built-in entities, but its advertised built-ins
are mostly the same formatted values already covered above; it does not advertise a ready-made
Italian PII model for arbitrary people or clinical concepts
([NLP.js official README](https://github.com/axa-group/nlp.js/)).
Add an **optional, non-generative statistical NER pack** based on
[`fastino/gliner2-privacy-filter-PII-multi`](https://huggingface.co/fastino/gliner2-privacy-filter-PII-multi).
The selected model and its GLiNER2 runtime are Apache-2.0, its mDeBERTa base is MIT, and the model
is trained across seven languages including Italian. Run the official implementation in one warmed,
CPU-only Python worker; do not spawn it once per column or table. It returns entity evidence only.
The TypeScript classifier remains the sole owner of the final column assessment and invokes NER
only for bounded text that deterministic rules did not resolve and only while the scan deadline has
time remaining.
This is a positive Q29 decision, not a placeholder for later library selection. The production gate
is limited to pinning artifacts, producing an SBOM/license inventory, and measuring the selected
model on PSD fixtures and target CPUs. If the optional pack is absent or misses its deadline, the
normal coverage rule produces `unknown`; ThothII does not substitute a generative LLM.
## Why the TypeScript baseline is sufficient
The classifier is not being asked to infer an open-ended privacy judgment from prose. It executes a
versioned policy and produces evidence. Its mandatory recognizers can be expressed as:
| Rule family | Implementation | Model needed? |
| --- | --- | --- |
| declared `text`/CLOB/unbounded string or declared maximum `>500` | catalog metadata rule | no |
| an observed value longer than 500 characters | length rule, ideally detected in SQL before transferring the value | no |
| email, IP/MAC, URL, UUID, payment card, IBAN/BIC, phone | pattern plus format/checksum validator | no |
| Italian fiscal code/VAT/passport/identity card/driving licence | locale-aware validator where available, otherwise a reviewed country recognizer; pattern, context and checksum where defined | no |
| credentials and technical secrets | anchored prefixes, token shapes, entropy/character-class heuristics, and organization allow/deny rules | no |
| hospital- or customer-specific terms/codes | normalized exact dictionary or phrase matching | no |
| person/location/organization in otherwise unstructured short text | statistical NER is useful but probabilistic | yes, optional |
| clinical meaning in unstructured short Italian text | domain NER or a governed terminology; generic NER is not enough | optional and domain-specific |
This is not speculative parity with a Python implementation. Presidio's own supported-entity table
shows that its global email, IBAN, credit-card and similar recognizers use pattern matching,
validation, context and checksums, while its Italian fiscal-code/VAT/passport/identity-card/licence
coverage is likewise rule-based
([Presidio supported entities](https://presidio.dataprivacystack.org/supported_entities/)). Those
mechanisms are portable; Presidio provides tested implementations and orchestration, not a unique
Python-language capability.
For large dictionaries, exact normalized token/phrase matching is still deterministic NLP and does
not require a learned model. The policy format should describe the rule rather than the library,
for example `kind: email`, `kind: checksum_id`, `kind: regex`, `kind: phrase_set`, and
`kind: max_length`. This makes a future engine change possible without changing persisted policy or
assessment evidence.
## What Python/Presidio would materially add
Presidio is the strongest Python option if ThothII later needs a broader recognizer ecosystem. Its
Analyzer composes rule-based recognizers, regexes, validation, NER and contextual enhancement; it
also exposes custom recognizers, batch processing and decision tracing
([Presidio Analyzer architecture](https://presidio.dataprivacystack.org/analyzer/)). Its current
catalog includes Italian identifiers and its YAML registry supports custom patterns and language-
specific context. Recent source also contains a `NoOpNlpEngine`, so a deterministic-only Presidio
deployment can avoid loading spaCy; this removes model cost but not the Python runtime/process
boundary
([Presidio `NoOpNlpEngine` integration](https://github.com/data-privacy-stack/presidio/blob/main/presidio-analyzer/presidio_analyzer/recognizer_registry/recognizer_registry.py)).
`presidio-structured` is conceptually close because it maps tabular columns/JSON keys to detected
entities, but its documented strategies select the most common, highest-confidence, or a mixed
entity. ThothII would still need its own deadline/sampling/`unknown` semantics and human-override
lifecycle; Presidio also lists improved mixed free-text/structured support as future work
([Presidio Structured](https://presidio.dataprivacystack.org/structured/)).
That benefit does not currently justify using it as ThothII's mandatory engine:
- The backend currently has no PII/NLP dependencies and is an ESM TypeScript service
(`backend/package.json`); the existing Python environment belongs to the separate harness/core
layer and also has no Presidio or spaCy dependency (`harness/pyproject.toml`). A Python analyzer therefore means a new
ownership and deployment boundary, not simply another import.
- Presidio is Python-based and can be run as a package or REST container. Its REST API intentionally
has no built-in authentication, so a sidecar would have to remain on a protected internal network
and be guarded by infrastructure
([Presidio FAQ, deployment](https://presidio.dataprivacystack.org/faq/#how-can-i-deploy-presidio-into-my-environment)).
- Presidio defaults to English. Adding Italian requires both an Italian NLP pipeline and adapted
language-specific recognizers/context; language-agnostic regexes alone do not solve that setup
([Presidio multilingual guidance](https://github.com/data-privacy-stack/presidio/blob/main/docs/analyzer/languages.md)).
- Presidio itself warns that automated detection cannot guarantee finding all sensitive data and
documents an unavoidable false-positive/false-negative trade-off
([Presidio FAQ](https://presidio.dataprivacystack.org/faq/#what-is-presidio)). It cannot turn a
negative sample into proof that a column is safe.
## Learned NLP is not an LLM, but it is still a model
A spaCy NER pipeline does not generate text and is not an LLM. It is a statistical token classifier:
it predicts spans such as `PERSON`, `LOCATION` and `ORGANIZATION`. spaCy explicitly notes that the
predictions depend on the training examples and will not always be correct
([spaCy NER documentation](https://spacy.io/usage/linguistic-features#named-entities)). That makes it
a useful additional **positive detector**, not the authority which declares a sampled column safe.
spaCy publishes Italian CPU pipelines with an NER component, but they are trained on news/media
data rather than clinical notes. More importantly, all three official packages (`it_core_news_sm`,
`md`, and `lg`) are licensed **CC BY-NC-SA 3.0**, so they are a no-go for a product which can be
used commercially; using the MIT-licensed spaCy runtime does not change the weights' license
([spaCy Italian pipelines](https://spacy.io/models/it),
[`sm` metadata](https://raw.githubusercontent.com/explosion/spacy-models/master/meta/it_core_news_sm-3.8.0.json),
[`md` metadata](https://raw.githubusercontent.com/explosion/spacy-models/master/meta/it_core_news_md-3.8.0.json),
[`lg` metadata](https://raw.githubusercontent.com/explosion/spacy-models/master/meta/it_core_news_lg-3.8.0.json)). A general Italian model also recognizes
names and places, not all sensitive clinical meaning. Training a suitable model would require a
labeled, representative and legally usable corpus; Presidio's own model guidance likewise states
that a labeled PII dataset is required to train a new model
([Presidio spaCy/Stanza guidance](https://github.com/data-privacy-stack/presidio/blob/main/docs/analyzer/nlp_engines/spacy_stanza.md#training-your-own-model)).
Consequently, the absence of a local LLM is irrelevant to sensitivity assessment. Installations
with a local LLM may let description generation receive broader source content under a separate
model-boundary policy, but **the sensitivity classifier does not call that LLM**. The optional NER
pack is a distinct local classifier capability and is never inferred from the description-model
catalog.
## Licensing and redistribution gate
Framework code, model weights, and training data are three separate licensing layers. A permissive
runtime does not grant ThothII permission to bundle weights trained from restricted data. For an
open-source product that may be deployed commercially, the default rule should therefore be:
**reject `NC`, research-only, custom restrictive, missing, or ambiguous model licenses**. Pinning an
approved artifact must also pin its model card, upstream model, training datasets, required notices,
and checksums. This is a technical compatibility screen, not legal advice.
| Candidate | Framework code | Weights and training-data evidence | Commercial bundling in ThothII | Decision |
| --- | --- | --- | --- | --- |
| Presidio, deterministic recognizers only | MIT; its license permits use and redistribution with the notice ([Presidio license](https://github.com/data-privacy-stack/presidio/blob/main/LICENSE)) | No general Italian model is required with `NoOpNlpEngine`; third-party dependencies still retain their notices | Compatible, but adds a Python process without adding unique detection capability for the agreed baseline | **GO legally; NO-GO architecturally for v1** |
| spaCy runtime | MIT ([spaCy license](https://github.com/explosion/spaCy/blob/master/LICENSE)) | Runtime only; no permission is conferred on downloaded pipelines | Compatible if used with separately approved weights | **GO for code only** |
| Official spaCy Italian `sm`/`md`/`lg` | spaCy code is MIT | Every current package declares CC BY-NC-SA 3.0; the metadata traces part of training to UD Italian ISDT under the same non-commercial license ([`sm`](https://raw.githubusercontent.com/explosion/spacy-models/master/meta/it_core_news_sm-3.8.0.json), [`md`](https://raw.githubusercontent.com/explosion/spacy-models/master/meta/it_core_news_md-3.8.0.json), [`lg`](https://raw.githubusercontent.com/explosion/spacy-models/master/meta/it_core_news_lg-3.8.0.json)) | `NC` excludes the intended commercial deployments; `SA` adds redistribution obligations | **NO-GO** |
| Stanza runtime | Apache-2.0 ([Stanza license](https://github.com/stanfordnlp/stanza/blob/main/LICENSE)) | Stanford says its language packs are ODC-By only to the extent it owns the rights and explicitly tells users to inspect training-data licenses. Italian NER uses FBK's KIND data; KIND annotations are CC BY-NC 4.0 or CC BY-NC-SA 4.0 ([Stanza model licensing note](https://stanfordnlp.github.io/stanza/performance.html), [Italian NER source](https://stanfordnlp.github.io/stanza/ner_models.html), [KIND license](https://github.com/dhfbk/KIND#license)) | The Italian model's non-commercial source data fails the product requirement | **GO for code; NO-GO for Italian weights** |
| Flair runtime and official multilingual NER | MIT ([Flair repository and license](https://github.com/flairNLP/flair)) | The official `ner-multi` model card neither includes Italian among its four trained languages nor declares a model license; a maintainer issue asking whether bundled weights are MIT was closed without an answer ([model card](https://huggingface.co/flair/ner-multi), [license issue](https://github.com/flairNLP/flair/issues/1487)) | Runtime is usable, but these weights are unsuitable and their redistribution terms are not established | **GO for code; NO-GO for these weights** |
| `osiria/flare-it-ner` | Runs in Transformers/PyTorch; the card labels the weights MIT | Trained on CC BY 4.0 WikiNER plus an additional manually annotated custom dataset whose redistribution terms are not identified in the model card ([model card](https://huggingface.co/osiria/flare-it-ner), [WikiNER dataset](https://figshare.com/articles/dataset/Learning_multilingual_named_entity_recognition_from_Wikipedia/5462500)) | The weights look promising, but an MIT label alone does not resolve the undocumented custom training-data rights | **NO-GO until provenance is explicit** |
| Transformers.js / ONNX runtime | Apache-2.0 ([Transformers.js license](https://github.com/huggingface/transformers.js/blob/main/LICENSE)) | ONNX conversion does not replace the source model's license or cure training-data restrictions; each exact model needs its own audit | Compatible runtime, but there is no blanket approval for arbitrary Hub/ONNX weights | **GO for code; weights case by case** |
| `Laibniz/italian-ner-pii-browser-uncased` | Packaged for Transformers.js as quantized ONNX; model card says Apache-2.0 | It is a conversion of an Osiria model and therefore inherits the same undocumented custom-data provenance ([model card](https://huggingface.co/Laibniz/italian-ner-pii-browser-uncased)) | CPU/browser packaging does not cure the upstream licensing gap | **NO-GO** |
| `Ar86Bat/multilang-pii-ner` | XLM-R/Transformers; model card labels weights MIT | Trained on `ai4privacy/open-pii-masking-500k-ai4privacy`, whose card declares CC BY 4.0 ([model card](https://huggingface.co/Ar86Bat/multilang-pii-ner), [dataset card](https://huggingface.co/datasets/ai4privacy/open-pii-masking-500k-ai4privacy)) | Commercial redistribution appears compatible with MIT notice plus CC BY attribution; an ONNX export, dependency audit, and PSD evaluation are still required | **CONDITIONAL GO as optional PII NER** |
| `urchade/gliner_multi_pii-v1` | GLiNER code and model are Apache-2.0 | Its published multilingual synthetic dataset, including Italian, is Apache-2.0 ([model card](https://huggingface.co/urchade/gliner_multi_pii-v1), [dataset card](https://huggingface.co/datasets/urchade/synthetic-pii-ner-mistral-v1)) | The cleanest publicly inspectable license chain, but the published SPY comparison reports substantially lower recall than the selected Fastino model | **GO legally; reserve candidate** |
| `fastino/gliner2-privacy-filter-PII-multi` | GLiNER2 code is Apache-2.0; the direct local dependencies are permissively licensed | Weights are Apache-2.0 and the mDeBERTa base is MIT. The authors report a purpose-built synthetic corpus spanning seven languages, including Italian; the corpus itself is not published, so training is not fully reproducible ([model card](https://huggingface.co/fastino/gliner2-privacy-filter-PII-multi), [paper](https://arxiv.org/abs/2605.09973), [base model](https://huggingface.co/microsoft/mdeberta-v3-base)) | Free local/commercial use and redistribution are allowed subject to Apache notices. It has the best reported exact-span F1 and recall among the compared open detectors on SPY, although that evaluation is not Italian-specific | **SELECTED optional NER pack** |
| `openai/privacy-filter` | Code and weights are Apache-2.0 and it runs locally through Python or Transformers.js | The publisher explicitly permits commercial deployment, but documents the model as primarily English and warns that non-English performance may drop ([model card](https://huggingface.co/openai/privacy-filter), [source](https://github.com/openai/privacy-filter)) | Clean license and direct TypeScript path, but insufficient Italian evidence and a larger one-billion-parameter artifact make it a weaker fit | **GO legally; NO-GO as primary Italian detector** |
| OpenMed Italian PII models | Cards label the weights Apache-2.0 | They are trained on `ai4privacy/pii-masking-400k`, whose license limits use to academic/non-commercial purposes unless a commercial agreement is obtained ([representative model](https://huggingface.co/OpenMed/OpenMed-PII-Italian-BioClinicalBERT-Base-110M-v1), [dataset license](https://huggingface.co/datasets/ai4privacy/pii-masking-400k/blob/main/LICENSE)) | An Apache label on derivative weights does not remove the explicit upstream non-commercial restriction | **NO-GO** |
The search therefore found a usable, free-of-charge, permissively licensed solution. Fastino
GLiNER2-PII is selected because it combines an Apache code/weight grant, explicit Italian training,
CPU operation, configurable entity labels, and materially higher published recall than the older
Urchade model. This does not prove Italian clinical accuracy: the published SPY evaluation is
English and the training corpus is synthetic. Those are quality and reproducibility limitations,
not a reason to reintroduce a generative LLM into the classifier.
## Fit with current ThothII design
The current backend already owns source sampling and the human-owned `Sensitive Data Flag`; source
values are transient and existing description sampling is bounded (`PROJECT_STATE.md` and
`backend/src/catalog/description-source-sampler.ts`). The new
classifier should therefore be a backend module behind the single effective sensitivity-decision
point already being designed, while preserving these agreed semantics:
1. `sensitive`, `non_sensitive`, and `unknown` are assessment outcomes; `unknown` leaves the
human-owned Sensitive Data Flag unchanged. Its effect on later description generation is out of
scope here.
2. Any positive rule match stops the column scan and yields `sensitive`.
3. A negative time-bounded sample yields `unknown`, never `non_sensitive`.
4. The administrator may set the final binary flag either way; the automated result remains a
proposal with rule/version/coverage evidence and no stored source value.
5. Learned NER, if present, can add positive evidence but cannot convert `unknown` to
`non_sensitive`.
The five-second scan budget should be enforced at the database-read orchestration layer and not
inside a recognizer library. An in-process deterministic engine maximizes the portion of that budget
available to database reads and avoids model cold-start/IPC costs. It should stream bounded batches,
check cancellation/deadline between batches, and stop at the first match.
## Acceptance gate for enabling the selected NER pack
Build a versioned, value-free fixture corpus representing at least: Italian/general identifiers,
false-positive lookalikes, bounded short notes with people/places, credential formats, null/high-
cardinality columns, and PSD-specific clinical codes. Measure per category rather than one aggregate
score. Pin the GLiNER2 package and model revision/checksum, build an SBOM including transitive
dependencies, retain required notices, prohibit model downloads at inference time, and measure warm
latency and memory on the target CPU. The detector is enabled only where those checks pass.
The acceptance test may tune label descriptions and per-label confidence thresholds, but it does
not reopen the architecture or silently select another Hub model. If quality or performance is
unacceptable, the optional NER pack remains disabled and unresolved sampled columns remain
`unknown`. Replacing the chosen model or adding a local-LLM fallback requires a new documented
decision.
@@ -0,0 +1,256 @@
# Prodotti open-core comparabili e meccanismi di conversione per ThothII
Ricerca originale: 2026-09-02
Issue: [Studiare prodotti open-core comparabili e i loro meccanismi di conversione](https://git.tylconsulting.it/mptyl/ThothII/issues/25)
## Domanda
Come separano core gratuito e capacità Enterprise prodotti open-core comparabili nei data tool e
developer tool, quale bisogno induce la conversione e quali anti-pattern hanno danneggiato adozione
o fiducia? Quali lezioni sono applicabili a ThothII come Customer-Hosted Installation?
## Risposta in breve
Il pattern più solido non è «far pagare il risultato migliore», ma lasciare aperto un prodotto che
completa davvero il lavoro fondamentale e far pagare il costo organizzativo del suo uso su scala:
identità e autorizzazioni, policy, audit di conformità, esercizio production-grade, integrazioni
governate e responsabilità contrattuale.
Metabase è il confronto più vicino a ThothII: nell'edizione open include interrogazione, generazione
SQL, funzioni AI, modello portato dal cliente, MCP, Agent API e CLI; Pro aggiunge permessi granulari,
SSO e auditing, mentre Enterprise differenzia soprattutto deployment air-gapped, procurement e
SLA. GitLab e Grafana confermano lo stesso schema nei developer tool: il lavoro quotidiano resta
nel core e la conversione è indotta dalla crescita dell'organizzazione o dal rischio operativo.
Per ThothII ne segue una separazione consigliata:
- **Community Edition open source:** percorso completo domanda → artifact SQL revisionato, Evidence,
catalogo, gate umani, BYOM/embedding, connettori fondamentali, CLI e contratti di estensione,
installazione singola sicura e utilizzabile in produzione;
- **Enterprise Edition commerciale:** federazione delle identità e group sync, RBAC/policy avanzati,
audit esportabile e resistente alle manomissioni, segregazione dei ruoli, HA/DR/backup e upgrade
governati, pacchetti air-gap, integrazioni ETL/CI/CD e secret manager certificate, supporto e SLA.
La qualità del workflow F1–F8, la leggibilità degli artifact e la sicurezza minima non devono essere
paywall. Una Community Edition incapace di provare il vantaggio distintivo di ThothII non costruisce
né reputazione né un futuro funnel commerciale.
## Metodo e limiti
La ricerca usa documentazione, licenze e pagine di prezzo ufficiali correnti al 2 settembre 2026.
Le aziende non pubblicano dati di funnel sufficienti a dimostrare quali singole feature causino una
conversione. I «trigger» sotto sono quindi inferenze esplicite ricavate dal confine dei piani, dal
buyer descritto e dal meccanismo di prezzo; non sono tassi di conversione osservati.
## Confronto
| Prodotto | Core gratuito | Confine commerciale | Trigger di conversione inferito | Lezione per ThothII |
| --- | --- | --- | --- | --- |
| **Metabase** | Open Source Edition in AGPL. Il piano open include query builder, editor SQL, AI, BYOM, MCP, Agent API e CLI. | Pro aggiunge SSO, permessi row/column, controlli e auditing; Enterprise aggiunge air-gap, procurement, success engineer e SLA. | Dati condivisi fra più gruppi o tenant, requisiti normativi, deployment mission-critical. | Analogo più forte: aprire il motore AI/SQL completo e monetizzare governo e affidabilità. |
| **GitLab** | Community Edition MIT; il prodotto Free conserva repository e workflow DevOps fondamentali. | Enterprise Edition ha codice dedicato e feature Premium/Ultimate; Premium punta a organizzazioni in crescita, Ultimate a sicurezza e compliance. | Coordinamento su scala, controlli di processo, security e compliance. | Tenere stabile il core e far emergere il valore Enterprise quando crescono attori, ambienti e rischio. |
| **Grafana** | Grafana OSS è AGPL e resta un prodotto completo di visualizzazione e dashboard. | Enterprise aggiunge RBAC, SAML/team sync, permessi sui data source, audit, Vault, plugin premium e supporto 24/7. | Accesso a dati sensibili, integrazione con stack aziendale, conformità e responsabilità operativa. | Vendere controllo e integrazioni certificate, non la capacità fondamentale di interrogare e capire i dati. |
| **Mattermost** | Team Edition è MIT e self-hosted; offre il lavoro collaborativo fondamentale. | Un'edizione commerciale self-hosted abilita feature tramite licenza; Entry è il fallback gratuito con limiti e omissioni. | SSO, compliance, HA, supporto e dimensione del deployment. | Il singolo artefatto installabile può semplificare upgrade e trial, ma i limiti quantitativi e il riposizionamento della Community creano confusione. |
| **Airbyte** | Il protocollo è MIT; Core e connector sono oggi ELv2, utilizzabili internamente ma con restrizione alla rivendita come servizio. Il motore di replica resta gratuito. | Offerte commerciali aggiungono governance, supporto/SLA e deployment controllati; l'offerta corrente Enterprise Flex comprende anche un control plane gestito. | Sovranità, governance multi-workspace, secret manager, operatività e supporto. | Utile per il confine funzionale, non come modello di licenza o architettura: ELv2 non è una licenza open source OSI e Flex non è interamente customer-hosted. |
### Metabase: il comparabile più vicino
Metabase pubblica l'Open Source Edition in AGPL e distribuisce la parte Enterprise con licenza
commerciale; nel repository le due famiglie di codice sono separate anche per directory
([licenze Metabase](https://www.metabase.com/license/),
[LICENSE del repository](https://github.com/metabase/metabase/blob/master/LICENSE.txt)). La
[matrice corrente dei piani](https://www.metabase.com/pricing/compare-plans) mantiene in Open Source
le funzioni che dimostrano il valore del prodotto: domande AI, BYOM, MCP, Agent API, CLI, query
builder ed editor SQL. Sposta invece in Pro permessi row/column, SSO, controlli AI, analytics d'uso
e auditing.
La [pagina prezzi](https://www.metabase.com/pricing) esplicita anche due scelte economiche utili:
Open Source ha utenti illimitati; Pro combina un minimo mensile con prezzo per utente; Enterprise
parte da 20.000 USD/anno e risponde a procurement, air-gap e SLA. Inoltre Pro ed Enterprise hanno
lo stesso insieme di feature principali: il salto Enterprise vende soprattutto modalità di
fornitura e responsabilità. Il pricing non va copiato, ma il segnale è pertinente: il cliente
regolato non paga per una query «più intelligente», paga perché il sistema può essere adottato e
presidiato secondo le regole dell'organizzazione.
### GitLab: separazione tecnica e crescita organizzativa
GitLab dichiara la Community Edition sotto MIT e la Enterprise Edition sotto licenza più
restrittiva ([licensing](https://docs.gitlab.com/development/licensing/)). In un unico codebase, il
codice Enterprise risiede in `ee/`; senza licenza, una build EE deve comportarsi come CE
([linee guida EE](https://docs.gitlab.com/development/ee_features/)). Questo riduce il rischio che
l'installazione perda la funzione fondamentale quando una licenza scade e consente di testare
esplicitamente sia la modalità Community sia quella Enterprise.
Il [pricing corrente](https://about.gitlab.com/pricing/) presenta Premium a 29 USD per utente/mese
per organizzazioni in crescita e Ultimate a prezzo personalizzato per sicurezza e compliance. Il
trigger non è l'accesso al repository o alla CI di base: è il passaggio da un team che produce a
un'organizzazione che deve coordinare, dimostrare controlli e ridurre rischio. Per ThothII il pattern
tecnico è replicabile, mentre il prezzo per seat lo è meno: pochi specialisti possono produrre un
datamart di valore per un'intera organizzazione, quindi il numero degli utenti non misura bene il
valore generato.
### Grafana: integrazioni, governance e supporto
Grafana OSS è passato da Apache-2.0 ad AGPLv3 per le versioni correnti
([FAQ di licensing](https://grafana.com/licensing/)). Grafana Enterprise è una build commerciale
che aggiunge RBAC, permessi sui data source, SAML e sincronizzazione dei team, audit, integrazione
Vault, reporting, plugin premium e supporto continuativo
([feature Enterprise](https://grafana.com/docs/grafana/latest/introduction/grafana-enterprise/)).
Sono capacità che acquistano valore quando dashboard e data source diventano infrastruttura
aziendale condivisa.
L'[attivazione Enterprise](https://grafana.com/docs/grafana/latest/administration/enterprise-licensing/)
usa una licenza legata all'istanza e agli utenti attivi; per ambienti senza Internet sono previste
modalità concordate con il vendor. La stessa documentazione mostra un buon principio di scadenza:
diverse configurazioni di sicurezza già applicate continuano a funzionare, mentre non è possibile
crearene o modificarne di nuove. ThothII dovrebbe essere ancora più conservativo: una scadenza non
deve rendere illeggibili gli artifact, invalidare decisioni persistite o interrompere un workflow
Community già consentito.
### Mattermost: distribuzione semplice, ma confine instabile
Mattermost usa due pattern contemporaneamente: Team Edition è una piattaforma self-hosted MIT;
l'Enterprise Edition è un binario commerciale che può funzionare gratuitamente in modalità Entry
e sblocca le funzioni a pagamento con una licenza
([edizioni](https://docs.mattermost.com/product-overview/editions-and-offerings.html),
[subscription self-hosted](https://docs.mattermost.com/product-overview/self-hosted-subscriptions.html)).
È un precedente utile per consegnare a un cliente un solo pacchetto Enterprise che degrada in modo
prevedibile a un set gratuito.
La documentazione corrente racconta però anche il costo di un confine non stabile: Team Edition era
stata usata molto oltre il pubblico previsto e alcune integrazioni SSO sovrapponevano capacità
commerciali; il prodotto ha quindi rimosso SSO dalla Team Edition e l'ha riposizionata per piccoli
team. Per ThothII è preferibile fissare prima del lancio una promessa Community durevole e non
ritirare in seguito capacità che gli utenti hanno ragionevolmente considerato parte del core.
### Airbyte: completezza del motore, ma ambiguità terminologica
Airbyte lascia gratuito il motore di replica e un ampio ecosistema di connector; l'offerta
Enterprise ha aggiunto nel tempo multi-tenancy, RBAC, PII masking, secret manager esterni, air-gap,
API/Terraform e supporto ([annuncio Self-Managed Enterprise](https://airbyte.com/blog/1-0-enterprise-ga),
[pricing corrente](https://airbyte.com/pricing)). È un altro esempio di core che svolge davvero il
lavoro e di conversione legata all'industrializzazione.
Il limite come modello per ThothII è duplice. La [pagina licensing corrente](https://airbyte.com/why-open-source)
definisce MIT il protocollo, ma ELv2 Core e connector: ELv2 vieta di rivendere il prodotto come
servizio e quindi introduce una restrizione di campo d'uso. Secondo la
[Open Source Definition](https://opensource.org/osd), una licenza open source non può imporre tale
restrizione. Inoltre Enterprise Flex oggi comprende un control plane gestito, incompatibile con la
decisione ThothII che domande, prompt, metadati e risultati non escano dal trust boundary del
cliente. Airbyte è quindi un comparabile di packaging, non una licenza o topologia da copiare.
## Anti-pattern osservabili
### 1. Rendere incompleto il lavoro fondamentale
I comparabili più credibili non riservano all'Enterprise l'azione che genera la prima prova di
valore: Metabase non blocca AI/SQL/BYOM; GitLab non blocca source control e CI di base; Grafana non
blocca dashboard e interrogazione; Airbyte non blocca il motore di replica. Per ThothII, mettere a
pagamento fasi F1–F8, qualità SQL, Evidence o review fondamentale trasformerebbe la Community in una
demo e ridurrebbe la reputazione che l'apertura deve costruire.
### 2. Mettere dietro licenza la sicurezza minima
SSO federato, group sync, ruoli personalizzati e audit di conformità sono boundary Enterprise
ricorrenti. Non ne segue che una Community possa essere insicura. ThothII Community deve conservare
autenticazione e autorizzazione di base, secret references, accesso DWH read-only, decision ledger,
provenienza degli artifact e log operativi minimi. Enterprise può aggiungere segregazione dei
compiti, policy obbligatorie, retention, export SIEM e audit resistente alle manomissioni.
### 3. Usare «open source» per una licenza source-available
MIT, Apache-2.0, AGPL e le altre licenze approvate OSI permettono uso per qualsiasi scopo. Una
clausola che proibisce di offrire il software come servizio può essere commercialmente legittima,
ma cambia la categoria. Presentare ELv2 o BSL come semplice open source crea ambiguità proprio sul
diritto che utenti e integratori devono poter valutare rapidamente. Se ThothII sceglie MIT o
Apache-2.0 per il core, deve accettare anche fork e offerte commerciali concorrenti; la protezione
economica deve venire da marchio, roadmap, moduli Enterprise, certificazioni e servizio.
### 4. Cambiare licenza dopo aver chiesto investimento all'ecosistema
HashiCorp passò i prodotti core da MPL-2.0 a BSL nel 2023, descrivendo BSL come source-available
([annuncio ufficiale](https://www.hashicorp.com/en/blog/hashicorp-adopts-business-source-license)).
Quattro settimane dopo, OpenTofu pubblicò il fork di Terraform e avviò l'ingresso nella Linux
Foundation ([cronologia OpenTofu](https://opentofu.org/blog/the-opentofu-fork-is-now-available/)).
Elastic cambiò da Apache-2.0 a SSPL/ELv2 nel 2021 e nel 2024 aggiunse nuovamente AGPL, una licenza
OSI, per la parte gratuita ([FAQ Elastic](https://www.elastic.co/pricing/faq/licensing/)); OpenSearch
documenta di essere nato dal precedente codice Apache-2.0
([FAQ OpenSearch](https://opensearch.org/faq/)). Questi fatti non misurano il danno commerciale, ma
dimostrano un esito verificabile: un cambio tardivo può frammentare progetto, nome ed ecosistema.
### 5. Applicare metriche di prezzo non allineate al valore
GitLab, Metabase e Grafana possono usare utenti o utenti attivi perché la platea operativa cresce
con l'adozione. ThothII ha invece pochi operatori IT/data e artifact consumati da molti processi
aziendali. Un prezzo puramente per seat può ostacolare collaborazione e sottovalutare installazioni
ad alto valore; un prezzo per token è ancora meno coerente quando modello ed embedding sono BYOM.
La ricerca suggerisce di valutare installazione, ambienti/workspace governati e livello di supporto
come unità principali, lasciando la decisione economica definitiva al ticket di packaging/prezzo.
### 6. Far fallire il core alla scadenza della licenza
Un contratto annuale deve disabilitare soltanto capacità commerciali future o modificabili. Dati,
artifact, revisioni e funzioni Community devono restare leggibili ed esportabili; non devono esistere
ransom lock-in o dipendenze online obbligatorie in un ambiente air-gapped. Questo riduce il rischio
per il buyer e rende credibile la promessa customer-hosted.
## Boundary raccomandato per ThothII
### Community Edition open source
- workflow deterministico F1–F8 completo e gate human-in-the-loop;
- catalogo, Evidence, schema linking e generazione del Transformation Package;
- configurazione BYOM e provider di embedding scelti dal cliente;
- connettori fondamentali e accesso read-only;
- persistenza, provenance e decision ledger necessari a riprendere e verificare il lavoro;
- frontend, backend, CLI, contratti degli artifact e SDK/extension points;
- installazione customer-hosted singola, backup/export manuale documentato e aggiornamenti
Community;
- sicurezza di base sufficiente a un uso reale, senza limiti artificiali a utenti, domande, token o
qualità del risultato.
### Enterprise Edition commerciale
- SAML/OIDC enterprise, directory/group sync, SCIM e integrazione con identity manager;
- RBAC personalizzato, segregazione dei ruoli, approval policy e policy pack obbligatori;
- audit immutabile/esportabile, retention, integrazione SIEM e report di conformità;
- HA, disaster recovery, backup/restore automatizzati, LTS, upgrade orchestrati e immagini firmate;
- pacchetti air-gap, mirror degli artifact e gestione offline delle licenze;
- secret manager e connettori certificati, con manutenzione/SLA;
- promozione degli artifact fra ambienti e integrazioni ETL, CI/CD e cataloghi aziendali;
- fleet/multi-installation management dentro il boundary del cliente;
- onboarding, training, supporto con SLA e responsabilità contrattuale.
### Separazione tecnica iniziale
Se TYL Consulting vuole che il codice Enterprise rimanga proprietario, il modello più chiaro è:
1. repository pubblico Community con build e test che non dipendono da codice privato;
2. contratti stabili per moduli, policy, identity provider, connettori e operator tooling;
3. repository privato Enterprise che produce moduli o una distribuzione comprendente il core;
4. una modalità senza licenza che conserva esattamente il comportamento Community;
5. license file offline firmato, senza telemetria obbligatoria né chiamata periodica a TYL;
6. test automatici del downgrade e della leggibilità/esportabilità degli artifact.
Il modello Metabase del codice commerciale visibile nello stesso repository è valido soltanto se si
accetta che l'Enterprise sia source-available. Non corrisponde invece all'obiettivo di mantenere quel
codice non pubblico. Il modello GitLab della directory dedicata è utile come disciplina di boundary,
ma non obbliga ThothII ad adottarne struttura o complessità.
## Decisioni che questa ricerca non prende
- MIT contro Apache-2.0: entrambe sono licenze open source permissive e consentono uso commerciale;
la scelta finale richiede revisione legale di brevetti, notice, dipendenze e contributi.
- prezzo e metrica contrattuale definitivi;
- lista iniziale dei connettori certificati;
- scelta fra plugin Enterprise, distribuzione unica o immagini separate;
- dettagli del meccanismo di licenza offline.
## Conclusione
Per ThothII l'open core sostenibile è un **core aperto completo nel risultato** e un'Enterprise
completa nella **responsabilità organizzativa**. Il primo deve permettere a un IT department di
provare e usare realmente il percorso domanda → SQL governato; la seconda deve rendere quel percorso
adottabile in un'organizzazione regolata, multi-ruolo e mission-critical. Questo boundary sostiene
insieme reputazione, conversione e fiducia senza dipendere da SaaS, consumo token o degradazione
artificiale della qualità.
@@ -17,7 +17,6 @@ The installation-level model catalog contained these entries, with `glm-53` as t
| Selection ID | LiteLLM route | Authentication |
| --- | --- | --- |
| `deepseek-v4-pro` | `deepseek/deepseek-v4-pro` | Protected `DEEPSEEK_API_KEY` value already used by core |
| `deepseek-v4-flash` | `deepseek/deepseek-v4-flash` | Protected `DEEPSEEK_API_KEY` value already used by core |
| `glm-53` | `openai/glm-5.3` on the Z.AI coding endpoint | Protected `ZAI_API_KEY` value already used by core |
| `qwen-36` | `openai/qwen3.6-35b-a3b` on the VPN-only AritmoLab endpoint | No operator API key |
@@ -1,428 +0,0 @@
# Piano di test: metadati di schema, campi sensibili e descrizioni AI
- Data: 2026-08-31
- Stato: proposto
- Baseline analizzata: `f586152` sul branch `test/database-baseline`
## 1. Obiettivo
Validare insieme le tre capacità recentemente introdotte in Database Management:
1. acquisizione autorevole dei metadati fisici di uno schema esterno;
2. proposta e revisione umana dei campi sensibili dal punto di vista privacy;
3. generazione, revisione e consolidamento delle descrizioni di tabelle e colonne.
Il piano è risk-based: perdita o corruzione di metadati, lettura o invio di valori protetti e
applicazione di una selezione al target sbagliato sono rischi bloccanti. La qualità linguistica dei
testi AI è invece valutata separatamente dal contratto tecnico, perché l'output del modello è una
proposta soggetta a revisione umana.
Questo documento non costituisce una certificazione normativa o GDPR: verifica i controlli tecnici
e il flusso operativo implementati dal prodotto.
### 1.1 Assunzione operativa: ambiente completamente sacrificabile
Per indicazione esplicita del proprietario, l'installazione sul Mac è esclusivamente di sviluppo e
test. Non contiene produzione e non esiste alcun requisito di conservazione dello stato locale.
- `catalog-db`, migrazioni, configurazioni locali, run, eventi, flag, descrizioni curate e generate
possono essere cancellati e ricreati tutte le volte necessarie;
- il database reale già collegato è la fonte autorevole dalla quale ricostruire il catalogo ed è la
sorgente primaria per validare schema, classificazione privacy e generazione delle descrizioni;
- reset completi, failure injection, dati incoerenti deliberati e prove distruttive sul catalogo
locale sono ammessi senza backup o procedura di rollback dell'ambiente;
- compatibilità con stato locale preesistente, sessioni legacy e vecchie revisioni del catalogo non
è un gate di questo piano;
- le prove di atomicità, idempotenza e preservazione dei campi restano necessarie perché verificano
il comportamento del prodotto dentro un ciclo di test, non perché debbano proteggere il Mac.
Il database sorgente resta normalmente read-only: quando una prova richiede DDL o mutazioni fra
scan e conferma si usa uno schema di test esplicitamente scrivibile o la fixture supplementare. La
disponibilità a buttare lo stato locale non elimina il rischio di inviare dati personali a un
provider esterno: proprio questa non-esfiltrazione è uno degli esiti principali da collaudare.
## 2. Terminologia e risultato atteso
I tre campi testuali del catalogo hanno autorità diverse e non devono essere confusi:
| Campo | Origine e autorità | Comportamento atteso |
| --- | --- | --- |
| `sourceComment` | Commento fisico acquisito dalla sorgente | Si aggiorna con la sincronizzazione e non è modificabile come contenuto curato. |
| `generatedDescription` | Proposta prodotta dall'AI o corretta nel catalogo | È salvata separatamente, può essere rigenerata esplicitamente e non sovrascrive gli altri due campi. |
| `description` | Descrizione curata dall'amministratore | Cambia solo tramite modifica esplicita o consolidamento di una proposta selezionata. |
Nel seguito, “commento generato” indica `generatedDescription`. La generazione non deve mai
modificare `sourceComment`; il passaggio a `description` avviene solo con il consolidamento umano.
## 3. Riferimenti e baseline
Il comportamento da verificare deriva da:
- stato corrente del progetto in `PROJECT_STATE.md`;
- [contratto Catalog Schema Snapshot](../contracts/catalog-schema-snapshot.md);
- [piano del Metadata Catalog](../plans/2026-08-26-metadata-catalog-from-thothai.md);
- [specifica della generazione descrizioni](../plans/2026-08-28-ai-catalog-description-generation-spec.md);
- [ADR 0007: sincronizzazione autorevole durevole](../adr/0007-durable-authoritative-schema-synchronization.md);
- [ADR 0009: un solo run sequenziale di generazione](../adr/0009-use-one-sequential-description-generation-run.md);
- [ADR 0010: campioni reali limitati](../adr/0010-allow-bounded-real-source-samples-for-description-generation.md);
- [ADR 0011: Sensitive Data Flag](../adr/0011-gate-source-samples-with-a-sensitive-data-flag.md);
- [accettazione AI del 2026-08-29](./2026-08-29-ai-catalog-description-generation-acceptance.md).
L'accettazione del 2026-08-29 è una baseline utile ma non chiude il gate attuale: precede i commit
`0736983` e `cb40c09`, inviava campioni reali inventati e documentava che un valore campione era
stato ripreso nella descrizione. Deve quindi essere ripetuta sul comportamento privacy corrente.
## 4. Perimetro
### Incluso
- trasporti `postgres_direct`, `rest_api` e `ssh_tunnel`;
- RPC REST tipizzato `POST /rpc/schema_snapshot` e fallback singolo read-only su
`POST /rpc/run_query`;
- metadati di tabelle, colonne, commenti sorgente, tipi, default, nullabilità, posizioni PK e coppie
FK ordinate;
- scope di sincronizzazione `tables`, `columns`, `relationships` e `all`;
- run durevoli, conferma delle differenze distruttive, cancellazione, recovery, eventi SSE e
fallback polling;
- Sensitive Data Flag, suggerimenti AI strutturali, review draft e storico operativo;
- scope di generazione `selected_columns`, `selected_tables`, `all` e `missing`;
- campionamento read-only, valori sintetici per colonne protette, batching, retry, stop, Unlock,
storico e consolidamento;
- uso controllato del database reale già collegato, inclusi schema e valori non sensibili, per il
collaudo end-to-end con fake provider e provider configurati;
- permessi, minimizzazione dei dati, redazione di log/errori e resistenza a input ostili.
### Escluso o rinviato
- applicazione del Sensitive Data Flag allo schema-linking/LSH del core, esplicitamente rinviata;
- sostituzione di `annotations.yaml`, pubblicazione Qdrant e cutover del runtime NL→SQL;
- alias, sinonimi, concetti, value descriptions e relazioni logiche;
- audit delle decisioni umane sul flag: per disegno si conserva solo il booleano corrente;
- DDL o scritture sul database sorgente;
- conservazione di cataloghi, run, descrizioni o sessioni locali precedenti al reset;
- compatibilità all'indietro con formati o migrazioni legacy non appartenenti alla baseline corrente.
## 5. Priorità e strategia
| Priorità | Significato | Esempi |
| --- | --- | --- |
| P0 | Gate bloccante di sicurezza o integrità | Nessuna lettura/invio di valori protetti; applicazione atomica; scope esatto; segreti non esposti. |
| P1 | Contratto funzionale necessario al rilascio | Trasporti, stati dei run, recovery, batching, review e consolidamento. |
| P2 | Qualità, UX e robustezza non distruttiva | Copy, focus, benchmark del modello, carico e degrado SSE. |
Le prove sono distribuite su quattro livelli:
1. **Contratto/unità**: repository in memoria, finti connector e Model Completer; nessuna rete.
2. **Integrazione**: catalogo PostgreSQL ricreabile via Docker, database reale come sorgente,
fixture supplementare, fake REST/SSH e Model Completer osservabile.
3. **UI/E2E locale**: component test con MSW e almeno un flusso Playwright sullo stack locale.
4. **Accettazione L2**: provider configurato realmente e database reale dopo review dei flag.
I test deterministici verificano il contratto. Le prove con un modello reale verificano
compatibilità e qualità, ma non sostituiscono i gate P0.
## 6. Ambiente e dati di test
### 6.1 Ambiente minimo
- stack locale con `catalog-db` eliminabile, migrazioni ripetibili e backend/frontend della stessa
baseline;
- database reale già collegato come sorgente primaria, con utenza capace di `SELECT` ma non di
`INSERT`, `UPDATE`, `DELETE`, DDL o cambio di schema;
- PostgreSQL effimero supplementare solo per le mutazioni controllate che non devono essere fatte
sul database reale;
- server REST fake in grado di servire snapshot valido, capability `unavailable`, 404, risposta
parziale/malformata e fallback `run_query`;
- server OpenSSH effimero con `known_hosts` esatto e varianti host key errata/assente;
- Model Completer fake che registra transitoriamente i messaggi e restituisce esiti programmabili;
- browser senza segreti in Web Storage e raccolta di log backend/SSE/API per le scansioni canary.
Le prove PostgreSQL di integrazione non devono risultare `skip`: la disponibilità di Docker è una
precondizione del gate. Il catalogo locale può essere azzerato prima di ogni wave senza snapshot o
backup del suo stato precedente.
### 6.2 Database reale e fixture supplementare `catalog_qa`
La prima baseline viene acquisita dal database reale collegato. Il test registra soltanto inventario
strutturale, conteggi e risultati sanitizzati; poi svuota il catalogo locale e dimostra di poterlo
ricostruire dalla stessa sorgente. Non è richiesto preservare alcun metadato locale precedente.
La fixture `catalog_qa` integra il database reale soltanto quando servono casi controllabili o
mutazioni che la sorgente reale non contiene. Deve includere almeno:
- `customers`: UUID PK, nome, email, codice fiscale, telefono, data di nascita, indirizzo e note;
- `orders`: FK verso `customers`, importo numerico, stato, timestamp, default e campi nullable;
- `order_lines`: PK composta e relazione composta ordinata;
- `clinical_events`: campi sanitari evidenti e tabella partizionata;
- `products`: SKU pubblico, categoria, prezzo e flag booleano;
- `empty_table`: nessuna riga ma struttura valida;
- `wide_entity`: almeno 23 colonne e metadati lunghi, per forzare batch `10 + 10 + 3` e il limite
dimensionale del messaggio;
- identificatori quotati, commenti Unicode/italiani, commenti null e oggetti fuori dallo schema.
Usare valori canary inventati e univoci, per esempio:
- `PRIV_EMAIL_CANARY_...`, `PRIV_TAX_CANARY_...`, `PRIV_HEALTH_CANARY_...` nelle colonne protette;
- `PUBLIC_SKU_CANARY_...` in una colonna esplicitamente non sensibile;
- `SECRET_API_CANARY_...` solo nel secret store del test.
I canary protetti non devono comparire nei messaggi al modello, nel catalogo, negli eventi, nelle
API, nel DOM o nei log. Il canary pubblico può apparire nel messaggio al provider entro i limiti
documentati e dopo la disclosure esplicita dell'utente. Sul database reale la stessa proprietà va
provata soprattutto osservando la proiezione SQL e il payload transitorio: i valori protetti non
devono essere letti, e nessun valore grezzo deve entrare nell'evidenza conservata.
### 6.3 Mutazioni della sorgente
Preparare tre revisioni dello schema:
- **A — iniziale**: struttura completa e commenti sorgente valorizzati;
- **B — distruttiva**: rimozione di una tabella, una colonna e una FK, più aggiunta di una nuova
colonna sensibile per nome;
- **C — race di conferma**: modifica ulteriore fra piano distruttivo e conferma, per provare il
re-scan.
Queste revisioni possono vivere nella fixture o in uno schema reale esplicitamente dichiarato
scrivibile. Fra una prova e l'altra è consentito eliminare completamente il catalogo locale,
riapplicare le migrazioni e ripartire dal database reale.
## 7. Casi di test — acquisizione dei metadati
| ID | P | Livello | Scenario | Risultato atteso |
| --- | --- | --- | --- | --- |
| MET-01 | P0 | API/Integrazione | Avvio senza binding raggiungibile, con versione database obsoleta o senza `database.manage`. | Il run non parte; risposta sicura e catalogo invariato. I segreti restano write-only. |
| MET-02 | P0 | Integrazione | Reset completo del catalogo e `all` sul database reale collegato, senza mock del client `pg_catalog`; ripetizione sulla fixture A per gli edge case assenti. | Il catalogo viene ricostruito da zero con snapshot esatta di tabelle, colonne, `sourceComment`, tipo, default, nullabilità, PK e FK ordinate; stato `succeeded`; nessuna scrittura alla sorgente. |
| MET-03 | P1 | Contratto/Integrazione | Stessa fixture via RPC REST tipizzato. | `schemaVersion: 1` e capability sono validate strettamente; risultato normalizzato uguale a MET-02. `unavailable` non è interpretato come collezione vuota. |
| MET-04 | P0 | Contratto/Integrazione | `/schema_snapshot` assente, fallback `run_query`; poi fallback assente, parziale, non JSON o con campi extra/mancanti. | Il fallback usa una sola query read-only. Ogni errore o snapshot invalida fallisce senza modifiche parziali; nessun fallback nasconde un errore operativo diverso da capability assente. |
| MET-05 | P1 | Integrazione | Accesso `ssh_tunnel` con host key corretta, errata e assente; errore durante apertura/chiusura. | Parità con MET-02 nel caso valido; fail-closed negli altri casi; processi, lease e file-segreto sempre rilasciati. |
| MET-06 | P0 | API | Esecuzione separata di `tables`, `columns`, `relationships` e `all`, con e senza selezione tabelle. | Ogni scope è autorevole solo nel proprio confine; nessun record fuori scope cambia o viene eliminato. Selezioni duplicate/inesistenti sono rifiutate atomicamente. |
| MET-07 | P0 | API/Integrazione | Ripetizione idempotente della fixture A dopo modifica di `description`, `generatedDescription` e `sensitive`. | Nessun diff fisico spurio; contenuti curati, proposte AI e flag delle entità ancora presenti sono preservati. Una nuova colonna nasce con `sensitive=false`. |
| MET-08 | P0 | API/Integrazione | Passaggio A→B, conferma assente/errata/scaduta e passaggio A→B→C prima della conferma. | Stato `awaiting_confirmation`, piano visibile e nessuna applicazione anticipata. La conferma valida provoca re-scan; se il diff cambia viene emesso un nuovo piano/token e il vecchio non applica nulla. Apply atomica oppure zero modifiche. |
| MET-09 | P0 | API/Integrazione | Timeout, disconnessione, capability incompleta o eccezione durante scan/apply. | Stato terminale coerente, errore sanitizzato, catalogo precedente intatto e lock rilasciato. Nessun segreto, SQL sensibile o stack trace nelle API/eventi. |
| MET-10 | P1 | Worker | Cancel in `queued`, `running`, `awaiting_confirmation` e `applying`; retry, restart con run attivo e lease scaduto. | Cancel è efficace solo prima di apply ed è rifiutato durante apply; retry crea un nuovo run. Recovery non duplica l'apply, marca correttamente i run interrotti, rilascia il lock e rimuove snapshot/diff/token interni non più necessari. |
| MET-11 | P0 | API | Due operazioni sullo stesso database: sync, cleanup, connection test, edit o generazione; in parallelo, operazioni su database diversi. | Una sola operazione possiede il database; conflitto 409 sicuro sullo stesso target. Nessun lock cross-database non previsto e nessuna release del token altrui. |
| MET-12 | P1 | UI | Avvio dai menu database/tabella, visualizzazione piano, conferma, history drawer, chiusura drawer, perdita SSE e polling. | Scope e selezione inviati sono esatti; azioni non eleggibili restano visibili con motivo; chiudere il drawer non ferma il run; replay/polling deduplicano gli eventi e aggiornano griglie/KPI. |
| MET-13 | P1 | E2E | Cleanup manuale di tabelle/colonne/relazioni e successiva sincronizzazione. | Cleanup modifica solo il catalogo; la sorgente resta invariata; un sync autorevole ripristina gli oggetti ancora presenti in sorgente. |
| MET-14 | P0 | Integrazione | Cambio binding/versione fra scan e apply ed errore iniettato a metà transazione. | La freshness viene ricontrollata sotto lock; il run fallisce senza righe parziali e conserva integralmente il catalogo precedente. |
| MET-15 | P1 | API/Worker | Replay SSE con `Last-Event-ID`/`after`, polling concorrente e retention oltre 30 giorni. | Cursori monotoni e nessun duplicato; gli eventi scaduti vengono potati senza corrompere run e stato finale. |
## 8. Casi di test — identificazione e protezione dei campi sensibili
| ID | P | Livello | Scenario | Risultato atteso |
| --- | --- | --- | --- | --- |
| PRV-01 | P0 | Repository/API | Prima sincronizzazione, re-sync e aggiunta di una colonna. | Il default è `false`; il valore umano delle colonne esistenti è preservato; la nuova colonna è esplicitamente da riesaminare ma non riceve uno stato audit inventato. |
| PRV-02 | P0 | API | Suggerimento per un database, tabelle selezionate e colonne selezionate; database multipli, target duplicati o mancanti. | Il provider riceve esattamente le colonne dello scope. Input ambigui sono rifiutati prima della chiamata e nessun flag cambia. |
| PRV-03 | P0 | Contratto | Ispezione del messaggio al classifier. | Sono presenti solo database, schema, tabella, colonna, tipo, nullabilità, PK e FK. Non compaiono righe, valori, commenti, descrizioni, flag corrente o segreti. |
| PRV-04 | P1 | Contratto | `wide_entity`, identificatori lunghi e limite byte. | Ordine deterministico, batch massimi di dieci colonne e rispetto del limite messaggio; una singola colonna non rappresentabile fallisce prima del provider con errore sicuro. |
| PRV-05 | P0 | API | Risposta valida, fenced/prosa, JSON malformato, target mancante/duplicato/ignoto e provider failure. | Ogni colonna richiesta compare una sola volta. Una classificazione invalida viene ritentata una volta; dopo esaurimento si ottiene errore sanitizzato e nessuna modifica. |
| PRV-06 | P0 | UI/API | Apertura draft, modifica manuale, chiusura/reload e salvataggio. | La proposta non è persistita prima di Save; reload la scarta. Il reviewer può invertire scelte; si salvano solo colonne cambiate con versione ottimistica; un conflitto richiede reload. |
| PRV-07 | P1 | Repository/UI | Tentativi completati, falliti e attivi al restart. | Ogni tentativo ha un run distinto con scope, modello, contatori ed eventi sanitizzati; startup marca `interrupted` i run attivi. Storico newest-first senza target ID, proposte, prompt, output grezzo o diagnostica provider. |
| PRV-08 | P0 | Integrazione | Generazione descrizioni su target con canary protetti. | Le colonne protette sono assenti dalla proiezione SQL, non semplicemente filtrate dopo la lettura. Se non rimangono colonne leggibili non viene eseguita una `SELECT`. Nessun canary protetto esce dal processo. |
| PRV-09 | P0 | Contratto/Integrazione | Tabella mista con colonne sensibili e pubbliche. | Per le sensibili il prompt contiene valori plausibili, deterministici e limitati derivati dai soli metadati, nello stesso formato dei campioni e senza etichettarli al modello come sintetici. Per le pubbliche: massimo cinque righe e cinque valori rappresentativi, valori troncati e transazione read-only chiusa con rollback. |
| PRV-10 | P1 | API | Cambio `false→true→false` dopo una descrizione già generata. | Il testo esistente non viene rigenerato retroattivamente. Solo le generazioni future cambiano fonte del contesto; tornando `false` il campionamento reale torna eleggibile. |
| PRV-11 | P0 | API/UI | Utente senza `database.manage`, modello non configurato, catalogo/provider indisponibile e richiesta interrotta. | Controlli nascosti/disabilitati in UI e rifiuto server-side; errori non espongono dettagli. Un tentativo fallito compare nello storico senza trasformarsi in audit della decisione umana. |
| PRV-12 | P1 | L2 | Corpus strutturale etichettato con identificatori personali, credenziali/token, salute, finanza, localizzazione e controlli non sensibili/ambigui, in inglese e italiano. | Si misurano precisione, recall e falsi negativi per modello. I campi critici mancati sono sottoposti al product owner; la soglia quantitativa va ratificata prima di diventare gate, perché il classifier è advisory e human-in-the-loop. |
| PRV-13 | P0 | UI/E2E | Modifica di un flag nella review senza Save e tentativo immediato di generare descrizioni. | Gate di rilascio da formalizzare: la generazione deve essere bloccata finché il draft non è salvato o scartato. In alternativa la UI deve dichiarare inequivocabilmente che verrà usato il valore persistito; non è accettabile mostrare “protetto” e campionare come non protetto. |
## 9. Casi di test — generazione e consolidamento dei commenti
| ID | P | Livello | Scenario | Risultato atteso |
| --- | --- | --- | --- | --- |
| GEN-01 | P0 | Config/API | Modelli validi, default, endpoint anonimo esplicito, secret ref mancante/errato e nessun modello. | Al browser arrivano solo ID, label e default. Nessuna chiave, provider payload o configurazione privata è esposta; feature disabilitata in modo comprensibile se non configurata. |
| GEN-02 | P0 | API | `selected_columns`, `selected_tables`, `all`, `missing`, target duplicati/inesistenti e zero eleggibili. | Scope esatto; `missing` include null/vuoto/whitespace e salta proposte esistenti; `all` sostituisce solo dopo conferma esplicita; input invalido non crea run. |
| GEN-03 | P1 | Worker | 23 colonne più tabelle, metadati e campioni lunghi. | Colonne prima delle tabelle per lo scope globale; richieste omogenee, sequenziali, massimo dieci target e sotto i limiti byte; contesto tabella aggiornato dopo le colonne. |
| GEN-04 | P0 | Contratto | JSON puro, un solo code fence JSON completo, prosa extra, payload multipli, target mancante/duplicato/ignoto, descrizione vuota o oltre limite. | Sono accettati solo i primi due formati validi. Una mappatura ambigua non applica alcun risultato del batch e conta come errore tecnico. |
| GEN-05 | P1 | API | Esito `generated` e `non_generatable` in workspace italiano/inglese. | Testo generato trimmato e salvato; il testo standard non generabile è localizzato dall'applicazione, non copiato dal provider. Gli errori tecnici non scrivono tale testo. |
| GEN-06 | P0 | Worker | Errore transiente, errore esaurito isolato, tre batch falliti consecutivi e successo fra due errori. | Helper con al massimo un retry e nessun fallback modello. Errori isolati portano a `completed_with_errors`; tre consecutivi a `failed`; un successo azzera il contatore. |
| GEN-07 | P0 | Integrazione | Successi seguiti da cancel, interruption o failure. | Ogni risultato valido è persistito subito e resta disponibile; il target fallito non riceve dati ambigui. “Generate Missing” consente il recupero naturale senza resume automatico. |
| GEN-08 | P0 | API | Secondo run durante un run attivo e modifica/sync/cleanup/consolidamento sul database posseduto. | Un solo run di generazione attivo nell'installazione; 409 chiaro al secondo Start. Il database target resta riservato e le operazioni incompatibili non alterano selezione o dati. |
| GEN-09 | P0 | Integrazione/Worker/UI | Stop in queued/running con helper attivo e con una `SELECT` sorgente deliberatamente bloccata/lenta. | Helper e query/connessione vengono terminati entro 5 secondi, stato `cancelled`, nessuna chiamata modello dopo Stop, risultati precedenti conservati ed eventi consultabili. Un test con sampler fake non è sufficiente. |
| GEN-10 | P0 | Worker/API | Restart con run `queued/running`; Unlock con worker/helper vivo e con run realmente stale; race Unlock/Start. | Startup marca `interrupted` senza replay. Unlock è rifiutato se esiste lavoro locale vivo e non può liberare la reservation di un nuovo run. |
| GEN-11 | P1 | Integrazione | Sorgente campioni indisponibile ma metadati validi. | La generazione prosegue metadata-only con un solo warning sicuro; nessun tentativo alternativo espone dettagli di connessione. |
| GEN-12 | P1 | API/UI | SSE disconnesso, replay da sequence, polling concorrente e riapertura storico. | Eventi persistiti, ordinati e deduplicati; history newest-first; contatori/stato finali coerenti; nessun prompt, campione, risposta completa o stack trace. |
| GEN-13 | P0 | UI/API | Revisione manuale della proposta e consolidamento selettivo di tabelle/colonne, inclusi target vuoti/stale. | Si copia solo `generatedDescription` non vuota dei target risolti; conteggio `copied/skipped` corretto; operazione atomica; `sourceComment` invariato. |
| GEN-14 | P0 | UI/Sicurezza | Tutti gli scope di generazione da database, tabelle e colonne selezionate, utente senza permesso, metadata contenente istruzioni ostili. | Prima di ogni Start, inclusi `selected_tables` e `selected_columns`, compare la disclosure “fino a cinque righe e cinque valori”. Il server applica comunque l'autorizzazione. Metadati e valori sono trattati come dati non fidati e l'output resta nel contratto JSON. |
| GEN-15 | P1 | L2 | Run reale sul modello di default e almeno un endpoint alternativo supportato, usando il database reale dopo la review dei flag e un role read-only. | Descrizioni nella lingua workspace, coerenti con struttura/commenti e senza fatti inventati critici; almeno una colonna e una tabella consolidate. Nessun valore protetto o segreto compare in request osservabile, eventi, API, persistenza o log. |
| GEN-16 | P1 | Integrazione | Fastify→worker→processo Python→LiteLLM→endpoint OpenAI-compatible locale di cattura. | Routing provider/model, header API key, `disableThinking`, singolo retry, timeout, limite output e payload sono corretti; chiave e diagnostica non risalgono a stdout, API o log applicativi. |
## 10. Percorso E2E prioritario
Il caso `E2E-01` deve attraversare le tre feature senza sostituire i test di contratto:
1. azzerare `catalog-db`, riapplicare le migrazioni e verificare che il role del database reale sia
realmente read-only;
2. eseguire `all` sul database reale e confrontare catalogo e snapshot sorgente; usare la fixture A
in una seconda esecuzione per gli edge case mancanti;
3. richiedere suggerimenti privacy sull'intero database;
4. modificare almeno una proposta e salvare i flag revisionati;
5. avviare `missing` dopo la disclosure, usando un Model Completer osservabile;
6. dimostrare che i canary protetti non sono letti né inviati e che il canary pubblico rispetta i
limiti;
7. correggere una `generatedDescription` e consolidare una tabella e una colonna;
8. applicare la fixture B, controllare il piano distruttivo e introdurre C prima della conferma;
9. confermare dopo il re-scan e verificare atomicità, preservazione di descrizioni/flag delle entità
superstiti e `sensitive=false` sulla nuova colonna;
10. perdere la connessione SSE, riaprire entrambi gli storici e verificare replay, polling e KPI.
Il percorso deve essere eseguito con fake provider dopo ogni reset rilevante e, in forma ridotta,
con il provider configurato sul database reale dopo che i flag sono stati revisionati e salvati.
Non è richiesto ripristinare lo stato locale precedente al test.
## 11. Valutazione qualitativa dei modelli
La qualità non deve confondersi con la sicurezza: anche un classifier perfetto non autorizza
l'esfiltrazione di un canary protetto e una descrizione elegante non rende valido un payload
ambiguo.
### 11.1 Classificazione privacy
Per ogni modello registrare matrice di confusione, precisione, recall e falsi negativi per categoria.
Eseguire almeno tre iterazioni sul corpus fisso per rilevare instabilità. Fino alla ratifica di una
soglia da parte del product owner, il risultato è un gate di review: ogni falso negativo su
credenziali/token, identificatori fiscali, dati sanitari o finanziari richiede accettazione esplicita
o correzione prima del rilascio operativo.
### 11.2 Descrizioni generate
Valutare ogni testo da 0 a 2 su:
- correttezza rispetto a struttura e commenti sorgente;
- specificità e utilità per un revisore;
- lingua e chiarezza;
- assenza di istruzioni seguite dai dati non fidati o fatti inventati;
- assenza di valori protetti e segreti.
Soglia proposta da ratificare: almeno 8/10, nessun punteggio 0 su correttezza o sicurezza. Un valore
esplicitamente non sensibile, reale o inventato, può essere ripreso entro il perimetro dichiarato;
un valore protetto non può mai esserlo.
## 12. Copertura esistente e gap da chiudere
| Area | Evidenza automatica già presente | Gap principale |
| --- | --- | --- |
| Snapshot e sincronizzazione | `backend/test/catalog-schema-introspector.test.ts`, `catalog-schema-routes.test.ts`, `catalog-table-introspector.test.ts`, `catalog-repository.integration.test.ts` | Introspezione `pg_catalog` realmente end-to-end, parità live dei tre trasporti e un unico E2E con re-scan distruttivo. |
| Privacy | `catalog-description-generation-routes.test.ts`, `catalog-description-generation-worker.test.ts`, `catalog-description-source-sampler.test.ts`, `catalog-synthetic-sample-value.test.ts` | Prova canary integrata query→prompt→API/log e benchmark reale post-ADR 0011. |
| Generazione | `catalog-description-generation-routes.test.ts`, `catalog-description-generation-worker.test.ts`, `catalog-description-generation.integration.test.ts` e test del helper | Accettazione reale aggiornata, cancellazione di una query PostgreSQL bloccata e integrazione ermetica fino all'endpoint LiteLLM locale. |
| UI | `DatabaseManagementPage.test.tsx`, `DescriptionGenerationDrawer.test.tsx`, `SensitiveDataSuggestionHistoryDrawer.test.tsx` | L'E2E Playwright corrente verifica soprattutto il layout, non il workflow funzionale. |
Nuovi asset consigliati:
- `backend/test/catalog-metadata-privacy-workflow.integration.test.ts`;
- `backend/test/fixtures/catalog-privacy-schema.sql` e snapshot REST equivalenti;
- integrazione a due PostgreSQL per le query reali `pg_catalog` e lo stop del sampler;
- endpoint OpenAI-compatible locale di cattura per GEN-16;
- `frontend/e2e/database-management-workflow.spec.ts`;
- corpus strutturale versionato per PRV-12, senza valori business;
- nuovo report di accettazione L2 che sostituisca il gate privacy del 2026-08-29.
## 13. Ordine di esecuzione
### Wave 1 — contratto rapido
- parser snapshot, introspector, scope e diff;
- classifier strutturale, batching e validazione output;
- sampler, valori sintetici, prompt bounds e parser descrizioni;
- autorizzazione, redazione e race del coordinator.
### Wave 2 — integrazione PostgreSQL
- reset totale di `catalog-db`, bootstrap delle migrazioni e ricostruzione dal database reale;
- migrazioni e vincoli del repository;
- atomicità sincronizzazione/consolidamento;
- run ed eventi persistiti, restart e cancellation;
- `E2E-01` con fake provider e canary.
### Wave 3 — frontend e stack locale
- component test MSW;
- Playwright funzionale, perdita SSE e polling;
- verifica disclosure, review draft, history e consolidamento.
### Wave 4 — accettazione L2
- provider reale sul database collegato dopo classificazione e review dei flag;
- benchmark PRV-12 e rubric GEN-15;
- scansione finale di canary e segreti;
- approvazione del product owner.
Comandi di regressione:
```bash
cd backend && npx vitest run
cd backend && npx tsc --noEmit -p .
cd backend && npm run build
cd frontend && npx vitest run
cd frontend && npx tsc -b
cd frontend && npm run build
cd frontend && npm run e2e
./scripts/build-docs.sh
```
Il report deve evidenziare esplicitamente eventuali test Docker/L2 saltati; un `skip` non equivale a
PASS del relativo gate.
## 14. Evidenze da conservare
Per ogni esecuzione registrare:
- commit, configurazione pubblica dei modelli, versione fixture e trasporto;
- ID e stato finale dei run, contatori ed eventi sanitizzati;
- snapshot catalogo prima/dopo e piano distruttivo con metadati e conteggi sanitizzati, senza valori
grezzi delle righe sorgente;
- report test/JUnit, screenshot dei gate UI e risultato della scansione canary;
- matrice di confusione privacy e rubric delle descrizioni per le prove L2;
- difetti con ID del caso, severità, riproducibilità e decisione finale.
Non allegare prompt completi, righe campione, output grezzi del provider, chiavi, digest o frammenti
di segreti. Il Model Completer spy deve verificare in memoria le asserzioni e scartare il payload al
termine del test.
Non serve conservare backup del catalogo locale, run precedenti o descrizioni generate durante una
wave: l'evidenza è il report sanitizzato e la capacità di ricostruire nuovamente il risultato dalla
sorgente reale.
## 15. Criteri di ingresso e uscita
### Ingresso
- baseline unica per backend/frontend e procedura verificata per eliminare e ricreare `catalog-db`;
- accesso al database reale collegato e inventario delle tabelle/colonne da includere nella review;
- fixture e canary supplementari approvati per i soli edge case controllati;
- role sorgente read-only verificato con una scrittura deliberatamente negata;
- fake connector/provider disponibili e log collection attiva;
- per L2, secret reference configurato senza materializzare il valore nell'evidenza.
### Uscita
- 100% dei casi P0 e P1 applicabili superati; nessun difetto Sev-1/Sev-2 aperto;
- almeno due ricostruzioni complete e coerenti del catalogo a partire dal database reale dopo reset
indipendenti;
- zero comparsa dei canary protetti e dei segreti fuori dalla sorgente/secret store del test;
- nessuna modifica parziale dopo errori, cancel o conferme stale;
- parità normalizzata dei trasporti supportati e nessuna integration PostgreSQL richiesta saltata;
- stati, contatori, eventi e history coerenti dopo success, partial failure, stop e restart;
- `sourceComment`, `generatedDescription` e `description` mantengono l'autorità prevista;
- accettazione L2 sul database collegato approvata dal product owner e
build/test/typecheck/documentazione verdi;
- ogni deviazione P2 o soglia qualitativa non ancora ratificata è documentata con owner e data.
## 16. Rischi residui da rendere espliciti
- `sensitive=false` è il default e il prodotto non conserva uno stato “review completata”: dopo ogni
reset, classificazione strutturale e salvataggio umano dei flag devono precedere qualunque run con
provider reale. Il catalogo è ricostruibile; un invio errato a un provider non lo è.
- Il flag protegge i valori campionati. Nomi, `sourceComment` e descrizioni sono metadati inviabili
al modello e possono contenere testo libero: se nel database reale includono PII serve una
decisione aggiuntiva di redazione, non una diversa aspettativa di test.
- La UI deve risolvere il caso di un flag modificato ma non salvato prima della generazione e deve
mostrare la disclosure anche per tabelle/colonne selezionate; il piano considera entrambi P0.
- L'AbortSignal corrente va provato contro una query PostgreSQL realmente bloccata: la sola
cancellazione del helper non dimostra che la lettura sorgente sia interrompibile.
- Lo storico dei Sensitive Data Suggestion Run è operativo, non un audit delle decisioni umane.
- La policy privacy non è ancora applicata allo schema-linking/LSH; nessun risultato di questo piano
deve essere presentato come copertura di quel percorso.
- Un provider reale resta non deterministico: il rilascio deve dipendere dai gate tecnici e dalla
review umana, non dalla ripetizione byte-identica delle descrizioni.
- L'esclusione fra operazioni e Unlock è in parte locale al processo. Se il deployment ammetterà più
repliche backend, servirà un gate aggiuntivo con due istanze contro lo stesso catalogo; non va
dedotta sicurezza multi-replica dai test single-process.
File diff suppressed because it is too large Load Diff
+49 -323
View File
@@ -1,4 +1,4 @@
import { expect, test, type Locator, type Page, type Route } from "@playwright/test";
import { expect, test, type Page, type Route } from "@playwright/test";
import { createAuthenticationStack } from "./fixtures/auth-stack.mjs";
test.describe.configure({ mode: "serial" });
@@ -12,13 +12,6 @@ const database = {
workspaceId: "psd-clinical",
workspaceName: "Policlinico San Donato",
workspaceAvailable: true,
workspaceRevision: { commit: "a".repeat(40), blob: "b".repeat(40) },
workspaceEvidence: { sourceType: "filesystem", state: "materialized_current_revision" },
runtimeBinding: {
transport: "postgres_direct",
configurationState: "ready",
sessionTransportSupported: true,
},
configured: true,
engine: "postgres",
databaseName: "warehouse",
@@ -44,22 +37,6 @@ const database = {
},
};
const unconfiguredDatabase = {
...database,
id: undefined,
workspaceId: "research-lab",
workspaceName: "Research laboratory",
configured: false,
databaseName: "research",
schema: "analytics",
version: 0,
createdAt: "",
updatedAt: "",
binding: { transport: "postgres_direct", port: 5432 },
connectionStatus: "untested",
testedVersion: undefined,
};
const table = {
id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
databaseId,
@@ -72,28 +49,6 @@ const table = {
updatedAt: "2026-08-27T09:00:00Z",
};
const descriptionRun = {
id: "77777777-7777-4777-8777-777777777777",
databaseId,
scope: "selected_columns",
modelId: "glm-53",
language: "it",
status: "completed",
total: 5,
processed: 5,
generated: 4,
nonGeneratable: 1,
failed: 0,
inputTokens: 1200,
cacheReadTokens: 80,
outputTokens: 420,
createdAt: "2026-08-31T09:00:00Z",
startedAt: "2026-08-31T09:00:01Z",
updatedAt: "2026-08-31T09:00:04Z",
finishedAt: "2026-08-31T09:00:04Z",
errorSummary: null,
};
function metrics(requestedDatabaseId: string | null) {
return {
scope: requestedDatabaseId ? "database" : "global",
@@ -155,8 +110,7 @@ async function expectContextPanelGeometry(page: Page, accessibleName: string) {
const managerCenter = managerBox.x + managerBox.width / 2;
const panelCenter = panelBox.x + panelBox.width / 2;
expect(Math.abs(panelCenter - managerCenter)).toBeLessThanOrEqual(1);
const expectedPanelWidth = Math.min(1200, managerBox.width - (managerBox.width <= 768 ? 24 : 32));
expect(Math.abs(panelBox.width - expectedPanelWidth)).toBeLessThanOrEqual(1);
expect(Math.abs(panelBox.width / managerBox.width - 0.6)).toBeLessThanOrEqual(0.005);
expect(panelBox.x).toBeGreaterThanOrEqual(managerBox.x - 1);
expect(panelBox.x + panelBox.width).toBeLessThanOrEqual(managerBox.x + managerBox.width + 1);
expect(panelBox.x + panelBox.width).toBeLessThanOrEqual(sessionRailBox.x + 1);
@@ -164,112 +118,7 @@ async function expectContextPanelGeometry(page: Page, accessibleName: string) {
expect(panelBox.y + panelBox.height).toBeLessThanOrEqual(managerBox.y + managerBox.height + 1);
}
async function expectCurrentNavigation(page: Page, accessibleName: string) {
const navigation = page.getByRole("complementary", { name: "Session navigation" });
const button = navigation.getByRole("button", { name: accessibleName, exact: true });
const available = navigation.locator('[data-navigation-state="available"]').first();
await expect(navigation.locator('[data-navigation-state="current"]')).toHaveCount(1);
await expect(button).toHaveAttribute("data-navigation-state", "current");
await expect(button).toHaveAttribute("aria-current", "page");
expect(await button.getAttribute("class")).toContain("bg-[oklch(var(--nav-active))]");
const [style, availableStyle] = await Promise.all([
button.evaluate((element) => ({
backgroundColor: getComputedStyle(element).backgroundColor,
borderColor: getComputedStyle(element).borderTopColor,
})),
available.evaluate((element) => ({
backgroundColor: getComputedStyle(element).backgroundColor,
borderColor: getComputedStyle(element).borderTopColor,
})),
]);
expect(style.backgroundColor).not.toBe(availableStyle.backgroundColor);
expect(style.borderColor).not.toBe(availableStyle.borderColor);
const tokens = await page.evaluate(() => {
const root = getComputedStyle(document.documentElement);
const read = (name: string) => root.getPropertyValue(name).trim().split(/\s+/).map(Number);
return { primary: read("--primary"), active: read("--nav-active") };
});
expect(tokens.active[2]).toBeCloseTo(tokens.primary[2], 1);
expect(tokens.active[1]).toBeLessThan(tokens.primary[1]);
expect(tokens.active[0]).toBeGreaterThan(tokens.primary[0]);
}
async function expectSelectedSessionScopeTab(page: Page, accessibleName: string) {
const tablist = page.getByRole("tablist", { name: "Session scope" });
const selected = tablist.getByRole("tab", { name: accessibleName, exact: true });
const inactive = tablist.locator('[role="tab"][aria-selected="false"]');
await expect(tablist.locator('[role="tab"][aria-selected="true"]')).toHaveCount(1);
await expect(selected).toHaveAttribute("data-tab-state", "active");
expect(await selected.getAttribute("class")).toContain("bg-[oklch(var(--nav-active))]");
const [selectedStyle, inactiveStyle] = await Promise.all([
selected.evaluate((element) => ({
backgroundColor: getComputedStyle(element).backgroundColor,
borderColor: getComputedStyle(element).borderBottomColor,
})),
inactive.evaluate((element) => ({
backgroundColor: getComputedStyle(element).backgroundColor,
borderColor: getComputedStyle(element).borderBottomColor,
borderWidth: getComputedStyle(element).borderTopWidth,
})),
]);
expect(selectedStyle.backgroundColor).not.toBe(inactiveStyle.backgroundColor);
expect(selectedStyle.borderColor).not.toBe(inactiveStyle.borderColor);
expect(inactiveStyle.borderWidth).toBe("1px");
expect(inactiveStyle.backgroundColor).not.toMatch(/^(?:transparent|rgba\(0, 0, 0, 0\))$/);
expect(inactiveStyle.borderColor).not.toMatch(/^(?:transparent|rgba\(0, 0, 0, 0\))$/);
}
function cssLightness(color: string): number {
const oklab = color.match(/^okl(?:ab|ch)\(([\d.]+)(%)?/);
if (oklab) {
const value = Number(oklab[1]);
return oklab[2] ? value / 100 : value;
}
const channels = color.match(/[\d.]+/g)?.slice(0, 3).map(Number) ?? [];
if (channels.length !== 3) return Number.NaN;
return color.startsWith("color(srgb") ? Math.max(...channels) : Math.max(...channels) / 255;
}
async function expectFleetRowTooltipBelow(trigger: Locator, expectedContent: string) {
await trigger.hover();
await expect.poll(() => trigger.evaluate((element) => {
const owner = element.closest<HTMLElement>("[data-tooltip]");
return owner ? getComputedStyle(owner, "::after").visibility : "missing";
})).toBe("visible");
await expect.poll(() => trigger.evaluate((element) => {
const owner = element.closest<HTMLElement>("[data-tooltip]");
return owner ? getComputedStyle(owner, "::after").opacity : "missing";
})).toBe("1");
const state = await trigger.evaluate((element) => {
const owner = element.closest<HTMLElement>("[data-tooltip]");
if (!owner) throw new Error("Row action tooltip owner is missing");
const style = getComputedStyle(owner, "::after");
return {
content: style.content.replace(/^['\"]|['\"]$/g, ""),
ownerHeight: owner.getBoundingClientRect().height,
top: Number.parseFloat(style.top),
right: Number.parseFloat(style.right),
opacity: style.opacity,
pointerEvents: style.pointerEvents,
backgroundColor: style.backgroundColor,
color: style.color,
};
});
expect(state.content).toBe(expectedContent);
expect(state.top - state.ownerHeight).toBeCloseTo(3, 1);
expect(state.right).toBe(0);
expect(state.opacity).toBe("1");
expect(state.pointerEvents).toBe("none");
expect(cssLightness(state.backgroundColor)).toBeLessThan(0.35);
expect(cssLightness(state.color)).toBeGreaterThan(0.9);
}
async function expectContextPanelHeaderUsesPrimary(
async function expectContextPanelHeaderMatchesNewSession(
page: Page,
accessibleName: string,
) {
@@ -278,22 +127,26 @@ async function expectContextPanelHeaderUsesPrimary(
.getByRole("main", { name: "Database management" });
const panel = manager.getByRole("dialog", { name: accessibleName });
const header = panel.locator(':scope > [data-catalog-panel-region="header"]');
await expect(header).toHaveCount(1);
const newSession = page
.getByRole("complementary", { name: "Session navigation" })
.getByRole("button", { name: "New session", exact: true });
const currentNavigation = page.locator('[data-navigation-state="current"]');
const [headerStyle, navigationStyle] = await Promise.all([
await expect(header).toHaveCount(1);
await expect(newSession).toBeVisible();
const [headerStyle, actionStyle] = await Promise.all([
header.evaluate((element) => ({
backgroundColor: getComputedStyle(element).backgroundColor,
backgroundImage: getComputedStyle(element).backgroundImage,
})),
currentNavigation.evaluate((element) => ({
newSession.evaluate((element) => ({
backgroundColor: getComputedStyle(element).backgroundColor,
})),
]);
expect(headerStyle.backgroundImage).toBe("none");
expect(headerStyle.backgroundColor).not.toBe("rgba(0, 0, 0, 0)");
expect(headerStyle.backgroundColor).not.toBe(navigationStyle.backgroundColor);
expect(headerStyle.backgroundColor).toBe(actionStyle.backgroundColor);
}
async function expectWorkAreaPanelGeometry(page: Page, accessibleName: string) {
@@ -320,8 +173,7 @@ async function expectWorkAreaPanelGeometry(page: Page, accessibleName: string) {
expect(Math.abs(
panelBox.x + panelBox.width / 2 - (workAreaBox.x + workAreaBox.width / 2),
)).toBeLessThanOrEqual(1);
const expectedPanelWidth = Math.min(1200, workAreaBox.width - (workAreaBox.width <= 768 ? 24 : 32));
expect(Math.abs(panelBox.width - expectedPanelWidth)).toBeLessThanOrEqual(1);
expect(Math.abs(panelBox.width / workAreaBox.width - 0.6)).toBeLessThanOrEqual(0.005);
expect(panelBox.x).toBeGreaterThanOrEqual(workAreaBox.x - 1);
expect(panelBox.x + panelBox.width).toBeLessThanOrEqual(workAreaBox.x + workAreaBox.width + 1);
expect(panelBox.x + panelBox.width).toBeLessThanOrEqual(sessionRailBox.x + 1);
@@ -329,7 +181,7 @@ async function expectWorkAreaPanelGeometry(page: Page, accessibleName: string) {
expect(panelBox.y + panelBox.height).toBeLessThanOrEqual(workAreaBox.y + workAreaBox.height + 1);
}
async function expectWorkAreaPanelHeaderUsesPrimary(
async function expectWorkAreaPanelHeaderMatchesNewSession(
page: Page,
accessibleName: string,
) {
@@ -337,19 +189,21 @@ async function expectWorkAreaPanelHeaderUsesPrimary(
.getByTestId("conversation-column")
.getByRole("dialog", { name: accessibleName });
const header = panel.locator(':scope > [data-work-area-panel-region="header"]');
const currentNavigation = page.locator('[data-navigation-state="current"]');
const [headerStyle, navigationStyle] = await Promise.all([
const newSession = page
.getByRole("complementary", { name: "Session navigation" })
.getByRole("button", { name: "New session", exact: true });
const [headerStyle, actionStyle] = await Promise.all([
header.evaluate((element) => ({
backgroundColor: getComputedStyle(element).backgroundColor,
backgroundImage: getComputedStyle(element).backgroundImage,
})),
currentNavigation.evaluate((element) => ({
newSession.evaluate((element) => ({
backgroundColor: getComputedStyle(element).backgroundColor,
})),
]);
expect(headerStyle.backgroundImage).toBe("none");
expect(headerStyle.backgroundColor).not.toBe(navigationStyle.backgroundColor);
expect(headerStyle.backgroundColor).toBe(actionStyle.backgroundColor);
}
test.beforeAll(async () => {
@@ -364,24 +218,12 @@ test.afterAll(async () => {
test("context panels stay inside the manager and the Tables grid sits in a sidebar-colored frame", async ({ page }) => {
const unexpectedCatalogRequests: string[] = [];
const responses: Record<string, unknown> = {
"GET /api/catalog/databases": [database, unconfiguredDatabase],
"GET /api/catalog/databases": [database],
"GET /api/catalog/metadata-generation/models": {
models: [
{ id: "deepseek-v4-pro", label: "DeepSeek V4 Pro" },
{ id: "deepseek-v4-flash", label: "DeepSeek V4 Flash" },
{ id: "glm-53", label: "GLM 5.3" },
{ id: "qwen-36", label: "AritmoLab Qwen 3.6 35B A3B" },
],
default: "glm-53",
models: [],
default: null,
},
"GET /api/catalog/description-generation-runs?limit=50": [descriptionRun],
[`GET /api/catalog/description-generation-runs/${descriptionRun.id}`]: descriptionRun,
[`GET /api/catalog/description-generation-runs/${descriptionRun.id}/events-list?after=0`]: [{
sequence: 1,
level: "info",
message: "Description generation completed.",
createdAt: "2026-08-31T09:00:04Z",
}],
"GET /api/catalog/description-generation-runs?limit=50": [],
"GET /api/catalog/sensitive-data-suggestion-runs?limit=50": [],
[`GET /api/catalog/databases/${databaseId}/tables`]: [table],
};
@@ -410,41 +252,10 @@ test("context panels stay inside the manager and the Tables grid sits in a sideb
await page.goto(stack.publicUrl);
await signInAsAdmin(page);
await expectCurrentNavigation(page, "New session");
await expectSelectedSessionScopeTab(page, "My sessions");
await page.getByRole("tab", { name: "All sessions", exact: true }).click();
await expectSelectedSessionScopeTab(page, "All sessions");
await page.getByRole("tab", { name: "My sessions", exact: true }).click();
await expectSelectedSessionScopeTab(page, "My sessions");
const adminNavigationRail = page.getByRole("complementary", { name: "Session navigation" });
const administration = adminNavigationRail.getByRole("button", { name: "Administration", exact: true });
await expect(administration).toHaveCSS("font-family", /Manrope/);
await expect(administration).toHaveAttribute("aria-expanded", "false");
await expect(adminNavigationRail.getByRole("region", { name: "Administration" })).toHaveCount(0);
await administration.click();
await expect(administration).toHaveAttribute("aria-expanded", "true");
const administrationPanel = adminNavigationRail.getByRole("region", { name: "Administration" });
await expect(administrationPanel).toBeVisible();
expect(await administrationPanel.locator(":scope > *").evaluateAll((elements) => elements.map((element) => (
element.getAttribute("role") === "separator" ? "separator" : element.textContent?.trim()
)))).toEqual([
"Database management",
"separator",
"Workspace management",
"Pi management",
]);
await administration.click();
await expect(administration).toHaveAttribute("aria-expanded", "false");
await expect(adminNavigationRail.getByRole("button", { name: "Database management", exact: true })).toHaveCount(0);
await administration.click();
await expect(administration).toHaveAttribute("aria-expanded", "true");
await adminNavigationRail
await page
.getByRole("button", { name: "Database management", exact: true })
.click();
await expectCurrentNavigation(page, "Database management");
await expect(
page.getByRole("complementary", { name: "Session navigation" }),
@@ -457,42 +268,27 @@ test("context panels stay inside the manager and the Tables grid sits in a sideb
await expect(
page.getByRole("button", { name: "Back to workspace", exact: true }),
).toHaveCount(0);
await expect(page.getByRole("button", { name: /Add database/i })).toHaveCount(0);
await expect(page.getByRole("columnheader", { name: /Revision \/ Evidence/ })).toBeVisible();
await expect(page.getByRole("columnheader", { name: /NL→SQL runtime/ })).toBeVisible();
await expect(page.getByRole("columnheader", { name: /Metadata Catalog/ })).toBeVisible();
const metadataModelSelector = page.getByRole("combobox", {
name: "Metadata-generation LLM model",
});
await expect(metadataModelSelector).toHaveCount(1);
await expect(metadataModelSelector).toHaveValue("glm-53");
await expect(metadataModelSelector.locator("option")).toHaveText([
"DeepSeek V4 Pro",
"DeepSeek V4 Flash",
"GLM 5.3",
"AritmoLab Qwen 3.6 35B A3B",
]);
await page
.getByRole("button", { name: "Configure catalog for Research laboratory", exact: true })
.click();
const configurePanel = page.getByRole("dialog", { name: "Configure catalog" });
await expectContextPanelGeometry(page, "Configure catalog");
const configuredWorkspace = configurePanel.getByLabel("Workspace", { exact: true });
await expect(configuredWorkspace).toHaveValue("Research laboratory");
await expect(configuredWorkspace).toHaveAttribute("readonly", "");
await configurePanel.getByRole("button", { name: "Close database panel" }).click();
await page
.getByRole("button", { name: "Edit Policlinico San Donato", exact: true })
.click();
await expectContextPanelGeometry(page, "Edit database");
await expectContextPanelHeaderUsesPrimary(page, "Edit database");
await expectContextPanelHeaderMatchesNewSession(page, "Edit database");
await page
.getByRole("dialog", { name: "Edit database" })
.getByRole("button", { name: "Close database panel" })
.click();
await page
.getByRole("button", { name: "Description history", exact: true })
.click();
await expectContextPanelGeometry(page, "Description generation");
await expectContextPanelHeaderMatchesNewSession(page, "Description generation");
await page
.getByRole("dialog", { name: "Description generation" })
.getByRole("button", { name: /close/i })
.click();
await page.setViewportSize({ width: 1280, height: 800 });
await page
.getByRole("button", { name: "Edit Policlinico San Donato", exact: true })
@@ -504,12 +300,12 @@ test("context panels stay inside the manager and the Tables grid sits in a sideb
.click();
await page.setViewportSize({ width: 1910, height: 911 });
const databaseTablesAction = page.getByRole("button", {
name: "View tables for Policlinico San Donato",
exact: true,
});
await expectFleetRowTooltipBelow(databaseTablesAction, "Tables");
await databaseTablesAction.click();
await page
.getByRole("button", {
name: "View tables for Policlinico San Donato",
exact: true,
})
.click();
await expect(
page.getByRole("region", {
@@ -517,61 +313,6 @@ test("context panels stay inside the manager and the Tables grid sits in a sideb
}),
).toBeVisible();
await page
.getByRole("button", { name: "Description history", exact: true })
.click();
await expectContextPanelGeometry(page, "Description generation");
await expectContextPanelHeaderUsesPrimary(page, "Description generation");
const descriptionPanel = page.getByRole("dialog", { name: "Description generation" });
const descriptionBody = descriptionPanel.locator(
':scope > [data-catalog-panel-region="body"]',
);
const progressTable = descriptionPanel.getByRole("table", {
name: "Description generation progress",
});
const eventLog = descriptionPanel.getByRole("log", {
name: "Description generation events",
});
const recentRuns = descriptionPanel.getByRole("region", {
name: "Description generation history",
});
await expect(progressTable.getByRole("columnheader")).toHaveCount(5);
await expect(eventLog).toBeVisible();
await expect(recentRuns).toBeVisible();
const [bodyBox, eventLogBox, recentRunsBox] = await Promise.all([
descriptionBody.boundingBox(),
eventLog.boundingBox(),
recentRuns.boundingBox(),
]);
expect(bodyBox).not.toBeNull();
expect(eventLogBox).not.toBeNull();
expect(recentRunsBox).not.toBeNull();
if (bodyBox && eventLogBox && recentRunsBox) {
const eventBottomGutter = bodyBox.y + bodyBox.height - eventLogBox.y - eventLogBox.height;
const recentBottomGutter = bodyBox.y + bodyBox.height - recentRunsBox.y - recentRunsBox.height;
expect(eventBottomGutter).toBeGreaterThanOrEqual(10);
expect(recentBottomGutter).toBeGreaterThanOrEqual(10);
expect(eventLogBox.x + eventLogBox.width).toBeLessThan(recentRunsBox.x);
}
await page.setViewportSize({ width: 720, height: 800 });
const [compactManagerBox, compactDescriptionPanelBox] = await Promise.all([
page.getByTestId("conversation-column").getByRole("main", {
name: "Database management",
}).boundingBox(),
descriptionPanel.boundingBox(),
]);
expect(compactManagerBox).not.toBeNull();
expect(compactDescriptionPanelBox).not.toBeNull();
if (compactManagerBox && compactDescriptionPanelBox) {
expect(compactManagerBox.width - compactDescriptionPanelBox.width).toBeGreaterThanOrEqual(20);
}
expect(await descriptionBody.evaluate((element) => getComputedStyle(element).display)).not.toBe("grid");
await page.setViewportSize({ width: 1910, height: 911 });
await descriptionPanel.getByRole("button", { name: /close/i }).click();
const applicationBar = page.locator(
'main[aria-label="Database management"] .thot-fleet-ledger__header',
);
@@ -585,10 +326,6 @@ test("context panels stay inside the manager and the Tables grid sits in a sideb
await expect(
applicationBar.getByRole("group", { name: "Database management actions" }),
).toBeVisible();
await expect(
applicationBar.getByRole("combobox", { name: "Metadata-generation LLM model" }),
).toHaveValue("glm-53");
await expect(metadataModelSelector).toHaveCount(1);
await expect(applicationBar).not.toContainText("Thoth catalog · Fleet ledger");
await expect(applicationBar).not.toContainText("Inspect physical metadata");
@@ -618,14 +355,11 @@ test("context panels stay inside the manager and the Tables grid sits in a sideb
await expect(page.getByRole("row", { name: /patients/ })).toBeVisible();
const editPatientsAction = page.getByRole("button", {
name: "Edit description for patients",
exact: true,
});
await expectFleetRowTooltipBelow(editPatientsAction, "Edit metadata");
await editPatientsAction.click();
await page
.getByRole("button", { name: "Edit description for patients", exact: true })
.click();
await expectContextPanelGeometry(page, "Review table description");
await expectContextPanelHeaderUsesPrimary(page, "Review table description");
await expectContextPanelHeaderMatchesNewSession(page, "Review table description");
const tableEditor = page.getByRole("dialog", { name: "Review table description" });
const tableEditorFooter = tableEditor.locator(
':scope > [data-catalog-panel-region="footer"]',
@@ -766,10 +500,6 @@ test("Workspace and Pi management share the centered work-area panel without cov
await page.goto(stack.publicUrl);
await signInAsAdmin(page);
const administration = page.getByRole("button", { name: "Administration", exact: true });
await expect(administration).toHaveAttribute("aria-expanded", "false");
await administration.click();
await expect(administration).toHaveAttribute("aria-expanded", "true");
for (const viewport of [
{ width: 1910, height: 911 },
@@ -781,29 +511,25 @@ test("Workspace and Pi management share the centered work-area panel without cov
exact: true,
});
await workspaceTrigger.click();
await expectCurrentNavigation(page, "Workspace management");
await expectWorkAreaPanelGeometry(page, "Workspace management");
await expectWorkAreaPanelHeaderUsesPrimary(page, "Workspace management");
await expectWorkAreaPanelHeaderMatchesNewSession(page, "Workspace management");
const piTrigger = page.getByRole("button", { name: "Pi management", exact: true });
await piTrigger.click();
await expectCurrentNavigation(page, "Pi management");
await expect(
page.getByRole("dialog", { name: "Workspace management" }),
).toHaveCount(0);
await expectWorkAreaPanelGeometry(page, "Pi management");
await expectWorkAreaPanelHeaderUsesPrimary(page, "Pi management");
await expectWorkAreaPanelHeaderMatchesNewSession(page, "Pi management");
await page
.getByRole("dialog", { name: "Pi management" })
.getByRole("button", { name: "Close Pi management" })
.click();
await expect(piTrigger).toBeFocused();
await expectCurrentNavigation(page, "New session");
}
await page.setViewportSize({ width: 720, height: 800 });
await page.getByRole("button", { name: "Workspace management", exact: true }).click();
await expectCurrentNavigation(page, "Workspace management");
const compactWorkArea = page.getByTestId("conversation-column");
const compactPanel = compactWorkArea.getByRole("dialog", { name: "Workspace management" });
const [compactWorkAreaBox, compactPanelBox] = await Promise.all([
@@ -813,7 +539,7 @@ test("Workspace and Pi management share the centered work-area panel without cov
expect(compactWorkAreaBox).not.toBeNull();
expect(compactPanelBox).not.toBeNull();
if (compactWorkAreaBox && compactPanelBox) {
expect(Math.abs(compactPanelBox.width - compactWorkAreaBox.width + 24)).toBeLessThanOrEqual(1);
expect(Math.abs(compactPanelBox.width - compactWorkAreaBox.width + 16)).toBeLessThanOrEqual(1);
expect(Math.abs(
compactPanelBox.x + compactPanelBox.width / 2
- (compactWorkAreaBox.x + compactWorkAreaBox.width / 2),
@@ -269,6 +269,7 @@ export function FleetSelectionActions({ props, direction = "row" }: { props: Var
if (state.inventoryLevel === "databases") actions = [
{ id: "databases-test", group: "Connection", label: "Test connections", icon: <TestTube2 size={14} />, disabled, onClick: run(`Testing ${count} selected connection${count === 1 ? "" : "s"}`) },
{ id: "databases-sync-tables", group: "Synchronization", label: "Synchronize tables", icon: <RefreshCw size={14} />, disabled, onClick: run("Table synchronization started", "running") },
{ id: "databases-sync-columns", group: "Synchronization", label: "Synchronize all columns", icon: <Columns3 size={14} />, disabled, onClick: run("Column synchronization started", "running") },
{ id: "databases-sync-relationships", group: "Synchronization", label: "Synchronize relationships", icon: <Activity size={14} />, disabled, onClick: run("Relationship synchronization started", "running") },
{ id: "databases-sync-all", group: "Synchronization", label: "Synchronize all", icon: <RefreshCw size={14} />, disabled, onClick: run("Full schema synchronization started", "running") },
{ id: "databases-generate-all", group: "Descriptions", label: "Generate all descriptions", icon: <Sparkles size={14} />, disabled, onClick: run("Generate all confirmation opened") },
@@ -278,16 +279,16 @@ export function FleetSelectionActions({ props, direction = "row" }: { props: Var
{ id: "databases-delete-relationships", group: "Cleanup", label: "Delete all relationships", icon: <Trash2 size={14} />, executeLabel: "Review deletion", destructive: true, disabled, onClick: run("Delete relationships confirmation opened", "changes") },
];
if (state.inventoryLevel === "tables") actions = [
{ id: "tables-generate", group: "Descriptions", label: "Generate table descriptions", icon: <Sparkles size={14} />, disabled, onClick: run(`Description generation started for ${count} table${count === 1 ? "" : "s"}`) },
{ id: "tables-generate", group: "Descriptions", label: "Generate descriptions", icon: <Sparkles size={14} />, disabled, onClick: run(`Description generation started for ${count} table${count === 1 ? "" : "s"}`) },
{ id: "tables-sync-columns", group: "Synchronization", label: "Synchronize columns", icon: <RefreshCw size={14} />, disabled, onClick: run("Selected table column synchronization started", "running") },
{ id: "tables-consolidate", group: "Descriptions", label: "Copy generated descriptions to Description", icon: <Save size={14} />, disabled, onClick: run("Generated descriptions copied to curated Description") },
{ id: "tables-consolidate", group: "Descriptions", label: "Move generated to Description", icon: <Save size={14} />, disabled, onClick: run("Generated descriptions moved to curated Description") },
{ id: "tables-suggest-sensitive", group: "Sensitive data", label: "Suggest sensitive fields", icon: <ShieldCheck size={14} />, executeLabel: "Open review", disabled, onClick: () => onOpenPanel("sensitive-review") },
{ id: "tables-delete-columns", group: "Cleanup", label: "Delete all columns", icon: <Trash2 size={14} />, executeLabel: "Review deletion", destructive: true, disabled, onClick: run("Delete table columns confirmation opened", "changes") },
{ id: "tables-delete-relationships", group: "Cleanup", label: "Delete all relationships", icon: <Trash2 size={14} />, executeLabel: "Review deletion", destructive: true, disabled, onClick: run("Delete table relationships confirmation opened", "changes") },
];
if (state.inventoryLevel === "columns") actions = [
{ id: "columns-generate", group: "Descriptions", label: "Generate column descriptions", icon: <Sparkles size={14} />, disabled, onClick: run(`Description generation started for ${count} column${count === 1 ? "" : "s"}`) },
{ id: "columns-consolidate", group: "Descriptions", label: "Copy generated descriptions to Description", icon: <Save size={14} />, disabled, onClick: run("Generated column descriptions copied") },
{ id: "columns-generate", group: "Descriptions", label: "Generate descriptions", icon: <Sparkles size={14} />, disabled, onClick: run(`Description generation started for ${count} column${count === 1 ? "" : "s"}`) },
{ id: "columns-consolidate", group: "Descriptions", label: "Move generated to Description", icon: <Save size={14} />, disabled, onClick: run("Generated column descriptions consolidated") },
{ id: "columns-suggest-sensitive", group: "Sensitive data", label: "Suggest sensitive fields", icon: <ShieldCheck size={14} />, executeLabel: "Open review", disabled, onClick: () => onOpenPanel("sensitive-review") },
{ id: "columns-save-sensitive", group: "Sensitive data", label: `Save sensitive fields (${Object.keys(state.sensitiveDrafts).length})`, icon: <LockKeyhole size={14} />, executeLabel: "Save changes", disabled: Object.keys(state.sensitiveDrafts).length === 0, onClick: onSaveSensitive },
];
+1 -105
View File
@@ -1,15 +1,7 @@
import { http, HttpResponse } from "msw";
import { expect, test } from "vitest";
import { server } from "../test/msw";
import {
createCatalogRelationship,
deleteCatalogRelationship,
getCatalogMetrics,
rebuildGeneratedRelationships,
setCatalogRelationshipStatus,
type CatalogMetrics,
type CatalogRelationship,
} from "./catalog-databases";
import { getCatalogMetrics, type CatalogMetrics } from "./catalog-databases";
const databaseId = "d4baf0f5-b9c3-4dc5-aad2-2f5676a36e64";
@@ -66,99 +58,3 @@ test("propagates an unsuccessful catalog metrics response", async () => {
await expect(getCatalogMetrics()).rejects.toMatchObject({ status: 503 });
});
const logicalRelationship: CatalogRelationship = {
id: "11111111-1111-4111-8111-111111111111",
databaseId,
constraintName: null,
sourceTableId: "22222222-2222-4222-8222-222222222222",
sourceTableName: "orders",
targetTableId: "33333333-3333-4333-8333-333333333333",
targetTableName: "users",
updateRule: null,
deleteRule: null,
deferrable: false,
initiallyDeferred: false,
columns: [{
position: 1,
sourceColumnId: "44444444-4444-4444-8444-444444444444",
sourceColumnName: "user_id",
targetColumnId: "55555555-5555-4555-8555-555555555555",
targetColumnName: "id",
}],
origin: "manual",
status: "active",
lastSyncedDatabaseVersion: null,
lastSyncedAt: null,
createdAt: "2026-08-31T12:00:00.000Z",
updatedAt: "2026-08-31T12:00:00.000Z",
};
test("creates a manual relationship from the selected source and target columns", async () => {
let requestBody: unknown;
server.use(http.post("/api/catalog/databases/:databaseId/relationships", async ({ request }) => {
requestBody = await request.json();
return HttpResponse.json(logicalRelationship, { status: 201 });
}));
await expect(createCatalogRelationship(
databaseId,
logicalRelationship.columns[0].sourceColumnId,
logicalRelationship.columns[0].targetColumnId,
)).resolves.toEqual(logicalRelationship);
expect(requestBody).toEqual({
sourceColumnId: logicalRelationship.columns[0].sourceColumnId,
targetColumnId: logicalRelationship.columns[0].targetColumnId,
});
});
test("rebuilds generated relationships for one database", async () => {
let calls = 0;
server.use(http.post(
"/api/catalog/databases/:databaseId/relationships/rebuild-generated",
() => {
calls += 1;
return HttpResponse.json({ added: 12, alreadyPresent: 7, excluded: 3, ambiguous: 2 });
},
));
await expect(rebuildGeneratedRelationships(databaseId)).resolves.toEqual({
added: 12,
alreadyPresent: 7,
excluded: 3,
ambiguous: 2,
});
expect(calls).toBe(1);
});
test("changes whether a logical relationship is active or excluded", async () => {
let requestBody: unknown;
server.use(http.patch(
"/api/catalog/databases/:databaseId/relationships/:relationshipId",
async ({ request }) => {
requestBody = await request.json();
return HttpResponse.json({ ...logicalRelationship, status: "excluded" });
},
));
await expect(setCatalogRelationshipStatus(
databaseId,
logicalRelationship.id,
"excluded",
)).resolves.toMatchObject({ status: "excluded" });
expect(requestBody).toEqual({ status: "excluded" });
});
test("permanently deletes a logical relationship", async () => {
let calls = 0;
server.use(http.delete(
"/api/catalog/databases/:databaseId/relationships/:relationshipId",
() => {
calls += 1;
return new HttpResponse(null, { status: 204 });
},
));
await expect(deleteCatalogRelationship(databaseId, logicalRelationship.id)).resolves.toBeUndefined();
expect(calls).toBe(1);
});
+3 -67
View File
@@ -3,12 +3,6 @@ import { joinBackendPath } from "./runtime-config";
export type DatabaseTransport = "postgres_direct" | "rest_api" | "ssh_tunnel";
export type ConnectionStatus = "untested" | "reachable" | "failed";
export type WorkspaceEvidenceState =
| "not_declared"
| "materialized_current_revision"
| "configuration_required"
| "configured_unverified"
| "workspace_unavailable";
export type CatalogSecretName =
| "password"
| "apiKey"
@@ -39,16 +33,6 @@ export interface CatalogDatabase {
workspaceName: string;
workspaceDescription?: string;
workspaceAvailable: boolean;
workspaceRevision: { commit: string; blob: string } | null;
workspaceEvidence: {
sourceType: "filesystem" | "http" | "s3" | null;
state: WorkspaceEvidenceState;
};
runtimeBinding: {
transport: DatabaseTransport;
configurationState: "ready" | "configuration_required";
sessionTransportSupported: boolean;
} | null;
configured: boolean;
engine: "postgres";
databaseName: string;
@@ -157,9 +141,6 @@ export interface SensitiveDataSuggestionRun {
total: number;
suggestedSensitive: number;
suggestedNonSensitive: number;
inputTokens?: number;
cacheReadTokens?: number;
outputTokens?: number;
createdAt: string;
startedAt: string | null;
updatedAt: string;
@@ -183,37 +164,25 @@ export interface CatalogRelationshipColumn {
targetColumnName: string;
}
export type CatalogRelationshipOrigin = "physical" | "generated" | "manual";
export type CatalogRelationshipStatus = "active" | "excluded";
export interface CatalogRelationship {
id: string;
databaseId: string;
constraintName: string | null;
constraintName: string;
sourceTableId: string;
sourceTableName: string;
targetTableId: string;
targetTableName: string;
updateRule: string | null;
deleteRule: string | null;
updateRule: string;
deleteRule: string;
deferrable: boolean;
initiallyDeferred: boolean;
columns: CatalogRelationshipColumn[];
origin: CatalogRelationshipOrigin;
status: CatalogRelationshipStatus;
lastSyncedDatabaseVersion: number | null;
lastSyncedAt: string | null;
createdAt: string;
updatedAt: string;
}
export interface CatalogRelationshipRebuildResult {
added: number;
alreadyPresent: number;
excluded: number;
ambiguous: number;
}
export type CatalogDatabaseMetadataDeleteTarget = "tables" | "relationships";
export type CatalogTableMetadataDeleteTarget = "columns" | "relationships";
@@ -300,9 +269,6 @@ export interface DescriptionGenerationRun {
generated: number;
nonGeneratable: number;
failed: number;
inputTokens?: number;
cacheReadTokens?: number;
outputTokens?: number;
createdAt: string;
startedAt: string | null;
updatedAt: string;
@@ -486,36 +452,6 @@ export const listSensitiveDataSuggestionEvents = (runId: string, after = 0) =>
export const listCatalogRelationships = (databaseId: string) =>
apiFetch<CatalogRelationship[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/relationships`);
export const createCatalogRelationship = (
databaseId: string,
sourceColumnId: string,
targetColumnId: string,
) => apiFetch<CatalogRelationship>(
`/catalog/databases/${encodeURIComponent(databaseId)}/relationships`,
{ method: "POST", body: JSON.stringify({ sourceColumnId, targetColumnId }) },
);
export const rebuildGeneratedRelationships = (databaseId: string) =>
apiFetch<CatalogRelationshipRebuildResult>(
`/catalog/databases/${encodeURIComponent(databaseId)}/relationships/rebuild-generated`,
{ method: "POST" },
);
export const setCatalogRelationshipStatus = (
databaseId: string,
relationshipId: string,
status: CatalogRelationshipStatus,
) => apiFetch<CatalogRelationship>(
`/catalog/databases/${encodeURIComponent(databaseId)}/relationships/${encodeURIComponent(relationshipId)}`,
{ method: "PATCH", body: JSON.stringify({ status }) },
);
export const deleteCatalogRelationship = (databaseId: string, relationshipId: string) =>
apiFetch<void>(
`/catalog/databases/${encodeURIComponent(databaseId)}/relationships/${encodeURIComponent(relationshipId)}`,
{ method: "DELETE" },
);
export const deleteCatalogDatabaseMetadata = (
databaseIds: string[],
target: CatalogDatabaseMetadataDeleteTarget,
-9
View File
@@ -152,15 +152,6 @@ test.each([
["sensitive_data_suggestion_history_request_invalid", "Sensitive suggestion history parameters are invalid."],
["sensitive_data_suggestion_history_failed", "Sensitive suggestion history could not be loaded."],
["sensitive_data_suggestion_run_not_found", "The sensitive suggestion run was not found."],
["relationship_not_found", "The relationship no longer exists. Refresh and try again."],
["relationship_duplicate", "This relationship already exists."],
["relationship_target_not_unique", "The target column must be the only primary-key column of its table."],
["relationship_type_incompatible", "Source and target column types are not compatible."],
["relationship_read_only", "Physical relationships are read-only."],
["relationship_request_invalid", "The relationship request is invalid."],
["relationship_operation_failed", "The relationship operation failed."],
["relationship_schema_stale", "Synchronize the current database schema before managing logical relationships."],
["column_not_found", "The selected catalog column was not found. Refresh and try again."],
])("maps the catalog error code %s to safe local copy", async (code, message) => {
const fetchSpy = vi.spyOn(globalThis, "fetch").mockResolvedValue(
new Response(JSON.stringify({ code, message: "provider detail must not be trusted" }), {
-12
View File
@@ -35,9 +35,6 @@ const safeErrorCodes = new Set([
"sensitive_data_suggestion_run_not_found",
"schema_sync_conflict", "schema_introspection_failed", "schema_request_invalid",
"schema_operation_failed", "sync_run_not_found", "table_stale", "column_stale",
"relationship_not_found", "relationship_duplicate", "relationship_target_not_unique",
"relationship_type_incompatible", "relationship_read_only", "relationship_request_invalid",
"relationship_operation_failed", "relationship_schema_stale", "column_not_found",
]);
type SafeErrorPayload = {
@@ -107,15 +104,6 @@ const localCodeMessages: Record<string, string> = {
sync_run_not_found: "The synchronization run was not found.",
table_stale: "Table metadata changed. Reload and try again.",
column_stale: "Column metadata changed. Reload and try again.",
relationship_not_found: "The relationship no longer exists. Refresh and try again.",
relationship_duplicate: "This relationship already exists.",
relationship_target_not_unique: "The target column must be the only primary-key column of its table.",
relationship_type_incompatible: "Source and target column types are not compatible.",
relationship_read_only: "Physical relationships are read-only.",
relationship_request_invalid: "The relationship request is invalid.",
relationship_operation_failed: "The relationship operation failed.",
relationship_schema_stale: "Synchronize the current database schema before managing logical relationships.",
column_not_found: "The selected catalog column was not found. Refresh and try again.",
};
const localStatusMessages: Record<number, string> = {
+1 -1
View File
@@ -78,7 +78,7 @@ export function LoginPage({ config, onAuthenticated, onRetry }: LoginPageProps)
<div className="mx-auto grid min-h-[calc(100vh-4rem)] max-w-5xl items-center gap-12 lg:grid-cols-[minmax(0,1fr)_26rem]">
<section className="hidden max-w-xl lg:block">
<h1 className="max-w-lg font-heading text-5xl font-semibold leading-[1.03] tracking-tight sm:text-6xl">
From intent to SQL, with a human in the loop
From intent to SQL, with Human In The Loop
</h1>
<p className="mt-6 max-w-md text-base leading-7 text-muted-foreground">AI generates, you guide and approve.</p>
</section>
+1 -3
View File
@@ -4,7 +4,7 @@ import { cva, type VariantProps } from "class-variance-authority"
import { cn } from "@/lib/utils"
const buttonVariants = cva(
"group/button inline-flex shrink-0 items-center justify-center rounded-md border border-transparent bg-clip-padding font-sans text-sm font-semibold tracking-[0.005em] whitespace-nowrap transition-all outline-none select-none focus-visible:ring-3 focus-visible:ring-ring/25 disabled:pointer-events-none disabled:opacity-50 aria-invalid:border-destructive aria-invalid:ring-3 aria-invalid:ring-destructive/20 dark:aria-invalid:border-destructive/50 dark:aria-invalid:ring-destructive/40 [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4",
"group/button inline-flex shrink-0 items-center justify-center rounded-md border border-transparent bg-clip-padding text-sm font-semibold tracking-[0.005em] whitespace-nowrap transition-all outline-none select-none focus-visible:ring-3 focus-visible:ring-ring/25 disabled:pointer-events-none disabled:opacity-50 aria-invalid:border-destructive aria-invalid:ring-3 aria-invalid:ring-destructive/20 dark:aria-invalid:border-destructive/50 dark:aria-invalid:ring-destructive/40 [&_svg]:pointer-events-none [&_svg]:shrink-0 [&_svg:not([class*='size-'])]:size-4",
{
variants: {
variant: {
@@ -13,8 +13,6 @@ const buttonVariants = cva(
"border-border bg-card shadow-xs hover:bg-muted hover:text-foreground aria-expanded:bg-muted aria-expanded:text-foreground dark:border-input dark:bg-input/30 dark:hover:bg-input/50",
secondary:
"bg-secondary text-secondary-foreground hover:bg-[color-mix(in_oklch,oklch(var(--secondary)),oklch(var(--foreground))_5%)] aria-expanded:bg-secondary aria-expanded:text-secondary-foreground",
navigationActive:
"border-[oklch(var(--nav-active-border))] bg-[oklch(var(--nav-active))] text-[oklch(var(--nav-active-foreground))] shadow-xs hover:bg-[oklch(var(--nav-active-hover))] hover:text-[oklch(var(--nav-active-foreground))] focus-visible:border-[oklch(var(--nav-active-border))] focus-visible:ring-[oklch(var(--nav-active-border)/0.32)]",
ghost:
"hover:bg-muted hover:text-foreground aria-expanded:bg-muted aria-expanded:text-foreground dark:hover:bg-muted/50",
destructive:
-10
View File
@@ -47,12 +47,6 @@
--success: 0.7577 0.1581 165.0;
--warning: 0.8523 0.1386 78.9;
--info: 0.7035 0.1128 221.3;
/* Current navigation shares Instrument Red's hue, with lower chroma and
higher lightness so selection reads as a muted location signal. */
--nav-active: 0.9250 0.0520 23.2;
--nav-active-hover: 0.8950 0.0710 23.2;
--nav-active-foreground: 0.3650 0.1100 23.2;
--nav-active-border: 0.6000 0.1350 23.2;
--radius: 0.5rem;
--sidebar: 0.9709 0.0011 17.2;
--sidebar-foreground: 0.2678 0.0097 355.6;
@@ -87,10 +81,6 @@
--border: 0.3715 0 90;
--input: 0.3715 0 90;
--ring: 0.6897 0.1779 16.9;
--nav-active: 0.3350 0.0550 20.3;
--nav-active-hover: 0.3750 0.0650 20.3;
--nav-active-foreground: 0.8900 0.0700 20.3;
--nav-active-border: 0.6800 0.1350 20.3;
--sidebar: 0.2350 0 90;
--sidebar-foreground: 0.9310 0 90;
--sidebar-primary: 0.6023 0.1848 20.3;
+14 -40
View File
@@ -1,4 +1,4 @@
import { act, render, screen, waitFor, within } from "@testing-library/react";
import { act, render, screen, waitFor } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { http, HttpResponse } from "msw";
@@ -45,51 +45,24 @@ beforeEach(() => {
});
describe("authenticated shell permissions", () => {
test("hides administrative navigation from a session user", async () => {
test("shows only read-safe workspace chrome to a session user", async () => {
renderShell({ subject: "user-1", isAdmin: false, roles: ["user"], permissions: ["session.use"] });
expect(await screen.findByRole("button", { name: "New session" })).toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Administration" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Database management" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Workspace management" })).not.toBeInTheDocument();
expect(await screen.findByRole("button", { name: "Workspace management" })).toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Pi management" })).not.toBeInTheDocument();
expect(screen.queryByRole("tab", { name: "All sessions" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "All sessions" })).not.toBeInTheDocument();
});
test("shows the ordered administrative accordion only to an admin", async () => {
const user = userEvent.setup();
test("shows management and all-session chrome only for exact permissions", async () => {
renderShell({
subject: "admin-1",
isAdmin: true,
roles: ["admin"],
permissions: ["session.use", "session.read_all", "workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage"],
permissions: ["session.use", "session.read_all", "workspace.manage", "workspace.secrets.manage", "pi.manage"],
});
const sessionNavigation = screen.getByRole("complementary", { name: "Session navigation" });
const trigger = await within(sessionNavigation).findByRole("button", { name: "Administration" });
expect(trigger).toHaveClass("font-sans");
expect(trigger).toHaveAttribute("aria-expanded", "false");
expect(within(sessionNavigation).queryByRole("region", { name: "Administration" })).not.toBeInTheDocument();
trigger.focus();
await user.keyboard("{Enter}");
expect(trigger).toHaveAttribute("aria-expanded", "true");
const panel = within(sessionNavigation).getByRole("region", { name: "Administration" });
const database = within(panel).getByRole("button", { name: "Database management" });
const separator = within(panel).getByRole("separator");
const workspace = within(panel).getByRole("button", { name: "Workspace management" });
const pi = within(panel).getByRole("button", { name: "Pi management" });
expect(panel.children[0]).toBe(database);
expect(panel.children[1]).toBe(separator);
expect(panel.children[2]).toBe(workspace);
expect(panel.children[3]).toBe(pi);
expect(screen.getByRole("tab", { name: "All sessions" })).toBeInTheDocument();
await user.keyboard(" ");
expect(trigger).toHaveAttribute("aria-expanded", "false");
expect(within(sessionNavigation).queryByRole("region", { name: "Administration" })).not.toBeInTheDocument();
expect(within(sessionNavigation).queryByRole("button", { name: "Database management" })).not.toBeInTheDocument();
expect(await screen.findByRole("button", { name: "Pi management" })).toBeInTheDocument();
expect(screen.getByRole("button", { name: "All sessions" })).toBeInTheDocument();
});
test("keeps identity but hides logout outside local authentication", async () => {
@@ -205,14 +178,15 @@ describe("authenticated shell permissions", () => {
subject: "admin-1", isAdmin: true, roles: ["admin"],
permissions: ["session.use", "session.read_all", "workspace.manage", "pi.manage"],
});
expect(await screen.findByRole("button", { name: "Administration" })).toBeInTheDocument();
expect(screen.getByRole("tab", { name: "All sessions" })).toBeInTheDocument();
expect(await screen.findByRole("button", { name: "Pi management" })).toBeInTheDocument();
expect(screen.getByRole("button", { name: "All sessions" })).toBeInTheDocument();
act(() => clearAuthState());
expect(screen.queryByRole("button", { name: "Administration" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Workspace management" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Pi management" })).not.toBeInTheDocument();
expect(screen.queryByRole("tab", { name: "All sessions" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "All sessions" })).not.toBeInTheDocument();
await userEvent.click(screen.getByRole("button", { name: "Workspace management" }));
expect(await screen.findByRole("heading", { name: "Workspace management" })).toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Update workspace repository" })).not.toBeInTheDocument();
});
test("remounts the real shell so user-A panel and transcript state cannot survive user-B", async () => {
@@ -17,10 +17,6 @@ function renderShell() {
);
}
async function expandAdministration() {
await userEvent.click(screen.getByRole("button", { name: "Administration" }));
}
beforeEach(() => {
clearAuthState();
setAuthState({
@@ -96,19 +92,10 @@ test("uses Fleet Ledger by default and gates the legacy fallback to non-producti
test("replaces the core conversation while keeping the session navigation", async () => {
renderShell();
await expandAdministration();
const conversationColumn = screen.getByTestId("conversation-column");
const sessionNavigation = screen.getByRole("complementary", { name: "Session navigation" });
const newSession = within(sessionNavigation).getByRole("button", { name: "New session" });
const databaseManagement = within(sessionNavigation).getByRole("button", { name: "Database management" });
expect(newSession).toHaveAttribute("aria-current", "page");
expect(newSession).toHaveAttribute("data-navigation-state", "current");
expect(newSession).toHaveClass("bg-[oklch(var(--nav-active))]");
expect(databaseManagement).toHaveAttribute("data-navigation-state", "available");
expect(databaseManagement).not.toHaveAttribute("aria-current");
await userEvent.click(databaseManagement);
const manager = screen.getByRole("main", { name: "Database management" });
@@ -116,13 +103,8 @@ test("replaces the core conversation while keeping the session navigation", asyn
expect(conversationColumn).toContainElement(manager);
expect(screen.getByRole("complementary", { name: "Session navigation" })).toBe(sessionNavigation);
expect(sessionNavigation).toBeVisible();
expect(newSession).toBeVisible();
expect(newSession).toHaveAttribute("data-navigation-state", "available");
expect(newSession).not.toHaveAttribute("aria-current");
expect(newSession).not.toHaveClass("bg-[oklch(var(--nav-active))]");
expect(within(sessionNavigation).getByRole("button", { name: "New session" })).toBeVisible();
expect(databaseManagement).toHaveAttribute("aria-current", "page");
expect(databaseManagement).toHaveAttribute("data-navigation-state", "current");
expect(databaseManagement).toHaveClass("bg-[oklch(var(--nav-active))]");
expect(screen.queryByRole("button", { name: "Back to workspace" })).not.toBeInTheDocument();
expect(screen.queryByRole("textbox", { name: /new question/i })).not.toBeInTheDocument();
});
@@ -153,7 +135,6 @@ test("keeps a live core session connected while returning from database manageme
})),
);
renderShell();
await expandAdministration();
const session = await screen.findByTestId("session-item-s1");
await userEvent.click(session);
@@ -176,7 +157,7 @@ test("keeps a live core session connected while returning from database manageme
expect(FakeEventSource.instances).toHaveLength(1);
});
test("hides the complete administrative navigation from a regular user", () => {
test("keeps read-safe workspace access but hides privileged management entries", () => {
clearAuthState();
setAuthState({
issuer: "test",
@@ -190,9 +171,7 @@ test("hides the complete administrative navigation from a regular user", () => {
renderShell();
expect(screen.getByRole("button", { name: "New session" })).toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Administration" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Workspace management" })).not.toBeInTheDocument();
expect(screen.getByRole("button", { name: "Workspace management" })).toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Database management" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Pi management" })).not.toBeInTheDocument();
});
@@ -203,13 +182,6 @@ test("does not leave database management without confirming a dirty form", async
workspaceId: "psd-clinical",
workspaceName: "Policlinico San Donato",
workspaceAvailable: true,
workspaceRevision: { commit: "a".repeat(40), blob: "b".repeat(40) },
workspaceEvidence: { sourceType: "filesystem", state: "materialized_current_revision" },
runtimeBinding: {
transport: "postgres_direct",
configurationState: "ready",
sessionTransportSupported: true,
},
configured: true,
engine: "postgres",
databaseName: "warehouse",
@@ -230,7 +202,6 @@ test("does not leave database management without confirming a dirty form", async
}])));
const confirm = vi.spyOn(window, "confirm").mockReturnValue(false);
renderShell();
await expandAdministration();
await userEvent.click(screen.getByRole("button", { name: "Database management" }));
await userEvent.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
@@ -156,11 +156,6 @@ test("model selector shows the three Pi-enabled models and stores the selected p
});
test("opens Workspace management from the right sidebar without interrupting the shell", async () => {
setAuthState({
issuer: "test", subject: "admin", roles: ["admin"],
permissions: ["session.use", "workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage"],
isAdmin: true, csrfToken: null, session: null,
});
server.use(
http.get("/api/workspace-registry/status", () =>
HttpResponse.json({ branch: "main", ahead: 0, behind: 0, degraded: false })),
@@ -168,31 +163,15 @@ test("opens Workspace management from the right sidebar without interrupting the
);
renderShell();
await userEvent.click(screen.getByRole("button", { name: "Administration" }));
await userEvent.click(screen.getByRole("button", { name: "Workspace management" }));
expect(await screen.findByRole("heading", { name: "Workspace management" })).toBeVisible();
const dialog = screen.getByRole("dialog", { name: "Workspace management" });
const workArea = screen.getByTestId("conversation-column");
const sessionNavigation = screen.getByRole("complementary", { name: "Session navigation" });
const workspaceManagement = within(sessionNavigation).getByRole("button", { name: "Workspace management" });
const newSession = within(sessionNavigation).getByRole("button", { name: "New session" });
expect(workArea).toContainElement(dialog);
expect(sessionNavigation).not.toContainElement(dialog);
expect(screen.getByTestId("app-shell")).toHaveAttribute("data-activity-layout", "closed");
expect(workspaceManagement).toHaveAttribute("aria-current", "page");
expect(workspaceManagement).toHaveAttribute("aria-expanded", "true");
expect(workspaceManagement).toHaveAttribute("data-navigation-state", "current");
expect(workspaceManagement).toHaveClass("bg-[oklch(var(--nav-active))]");
expect(newSession).toHaveAttribute("data-navigation-state", "available");
expect(newSession).not.toHaveAttribute("aria-current");
await userEvent.click(screen.getByRole("button", { name: "Close workspace management" }));
await waitFor(() => expect(screen.queryByRole("dialog", { name: "Workspace management" })).not.toBeInTheDocument());
expect(newSession).toHaveAttribute("aria-current", "page");
expect(newSession).toHaveAttribute("data-navigation-state", "current");
expect(workspaceManagement).toHaveAttribute("aria-expanded", "false");
expect(workspaceManagement).toHaveAttribute("data-navigation-state", "available");
});
test("does not block the shell when the DWH is unavailable at startup", async () => {
@@ -205,8 +184,7 @@ test("does not block the shell when the DWH is unavailable at startup", async ()
);
renderShell();
expect(await screen.findByRole("button", { name: "New session" })).toBeVisible();
expect(screen.queryByRole("button", { name: "Administration" })).not.toBeInTheDocument();
expect(await screen.findByRole("button", { name: "Workspace management" })).toBeVisible();
expect(screen.queryByRole("heading", { name: "Connection unavailable" })).not.toBeInTheDocument();
expect(healthChecks).toBe(0);
});
@@ -16,7 +16,7 @@ const regularUser: AuthenticatedUser = {
};
const adminUser: AuthenticatedUser = {
...regularUser, subject: "alice-id", roles: ["admin"] as const,
permissions: ["session.use", "session.read_all", "workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage"] as const, isAdmin: true,
permissions: ["session.use", "session.read_all", "pi.manage"] as const, isAdmin: true,
};
function wrap(user: AuthenticatedUser = regularUser) {
@@ -95,8 +95,7 @@ test("regular users load only their sessions and never see administrator control
wrap();
await screen.findByText("Attiva uno");
expect(scope).toBe("mine");
expect(screen.queryByRole("button", { name: "Administration" })).not.toBeInTheDocument();
expect(screen.queryByRole("tab", { name: "All sessions" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "All sessions" })).not.toBeInTheDocument();
expect(screen.queryByText(/administrator view/i)).not.toBeInTheDocument();
});
@@ -116,25 +115,17 @@ test("Pi management preserves the open session summary and the model activity ti
http.post("/api/sessions/:id/resume", () => resumeResult("s1")),
http.get("/api/sessions/:id", () => HttpResponse.json({ phase: 4 })),
);
wrap(adminUser);
wrap();
await user.click(await screen.findByText("Attiva uno"));
expect(await screen.findByRole("complementary", { name: "Session summary" })).toHaveTextContent("Domanda originale");
await user.click(screen.getByRole("button", { name: "Administration" }));
await user.click(screen.getByRole("button", { name: "Pi management" }));
expect(await screen.findByRole("heading", { name: "Pi management" })).toBeVisible();
const dialog = screen.getByRole("dialog", { name: "Pi management" });
const workArea = screen.getByTestId("conversation-column");
const sessionNavigation = screen.getByRole("complementary", { name: "Session navigation" });
const piManagement = within(sessionNavigation).getByRole("button", { name: "Pi management" });
const newSession = within(sessionNavigation).getByRole("button", { name: "New session" });
expect(workArea).toContainElement(dialog);
expect(sessionNavigation).not.toContainElement(dialog);
expect(piManagement).toHaveAttribute("aria-current", "page");
expect(piManagement).toHaveAttribute("aria-expanded", "true");
expect(piManagement).toHaveAttribute("data-navigation-state", "current");
expect(newSession).toHaveAttribute("data-navigation-state", "available");
expect(newSession).not.toHaveAttribute("aria-current");
const preservedSummary = screen.getByRole("complementary", { name: "Session summary" });
expect(preservedSummary).not.toHaveAttribute("aria-hidden", "true");
expect(preservedSummary).toHaveTextContent("Attiva uno");
@@ -142,10 +133,6 @@ test("Pi management preserves the open session summary and the model activity ti
await waitFor(() => {
expect(screen.getByRole("complementary", { name: "Session summary" })).toHaveTextContent("Domanda originale");
});
expect(newSession).toHaveAttribute("aria-current", "page");
expect(newSession).toHaveAttribute("data-navigation-state", "current");
expect(piManagement).toHaveAttribute("aria-expanded", "false");
expect(piManagement).toHaveAttribute("data-navigation-state", "available");
await user.click(screen.getByRole("button", { name: "Resume" }));
await screen.findByRole("button", { name: "Show model activity" });
@@ -173,66 +160,17 @@ test("administrators can explicitly switch to all sessions and see owners", asyn
}),
);
wrap(adminUser);
const tablist = await screen.findByRole("tablist", { name: "Session scope" });
const mySessions = within(tablist).getByRole("tab", { name: "My sessions" });
const allSessions = within(tablist).getByRole("tab", { name: "All sessions" });
expect(mySessions).toHaveAttribute("aria-selected", "true");
expect(mySessions).toHaveAttribute("aria-controls", "session-scope-panel");
expect(mySessions).toHaveAttribute("tabindex", "0");
expect(mySessions).toHaveAttribute("data-tab-state", "active");
expect(mySessions).toHaveClass("bg-[oklch(var(--nav-active))]");
expect(allSessions).toHaveAttribute("aria-selected", "false");
expect(allSessions).toHaveAttribute("aria-controls", "session-scope-panel");
expect(allSessions).toHaveAttribute("tabindex", "-1");
expect(allSessions).toHaveAttribute("data-tab-state", "inactive");
expect(allSessions).toHaveClass("border-border", "bg-card");
expect(allSessions).not.toHaveClass("border-transparent", "bg-transparent");
expect(screen.getByRole("tabpanel")).toHaveAttribute("aria-labelledby", "session-scope-mine-tab");
await userEvent.click(allSessions);
await screen.findByRole("button", { name: "All sessions" });
expect(screen.getByRole("button", { name: "My sessions" })).toHaveAttribute("aria-pressed", "true");
expect(screen.getByRole("button", { name: "All sessions" })).toHaveAttribute("aria-pressed", "false");
await userEvent.click(screen.getByRole("button", { name: "All sessions" }));
await waitFor(() => expect(scope).toBe("all"));
expect(mySessions).toHaveAttribute("aria-selected", "false");
expect(mySessions).toHaveAttribute("data-tab-state", "inactive");
expect(allSessions).toHaveAttribute("aria-selected", "true");
expect(allSessions).toHaveAttribute("data-tab-state", "active");
expect(allSessions).toHaveClass("bg-[oklch(var(--nav-active))]");
expect(screen.getByRole("tabpanel")).toHaveAttribute("aria-labelledby", "session-scope-all-tab");
expect(screen.getByRole("button", { name: "My sessions" })).toHaveAttribute("aria-pressed", "false");
expect(screen.getByRole("button", { name: "All sessions" })).toHaveAttribute("aria-pressed", "true");
expect(await screen.findByText("Administrator view: all sessions")).toBeInTheDocument();
expect(screen.getByText("Owner: Bob")).toBeInTheDocument();
});
test("session scope tabs support roving keyboard navigation", async () => {
let scope = "";
server.use(http.get("/api/sessions", ({ request }) => {
scope = new URL(request.url).searchParams.get("scope") ?? "";
return HttpResponse.json(LIST);
}));
wrap(adminUser);
const user = userEvent.setup();
const tablist = await screen.findByRole("tablist", { name: "Session scope" });
const mySessions = within(tablist).getByRole("tab", { name: "My sessions" });
const allSessions = within(tablist).getByRole("tab", { name: "All sessions" });
mySessions.focus();
await user.keyboard("{ArrowRight}");
await waitFor(() => expect(scope).toBe("all"));
expect(allSessions).toHaveFocus();
expect(allSessions).toHaveAttribute("aria-selected", "true");
expect(mySessions).toHaveAttribute("tabindex", "-1");
await user.keyboard("{ArrowLeft}");
await waitFor(() => expect(scope).toBe("mine"));
expect(mySessions).toHaveFocus();
expect(mySessions).toHaveAttribute("aria-selected", "true");
await user.keyboard("{End}");
expect(allSessions).toHaveFocus();
expect(allSessions).toHaveAttribute("aria-selected", "true");
await user.keyboard("{Home}");
expect(mySessions).toHaveFocus();
expect(mySessions).toHaveAttribute("aria-selected", "true");
});
test("administrator confirms before deleting a same-named user's session", async () => {
let deletes = 0;
server.use(
@@ -249,7 +187,7 @@ test("administrator confirms before deleting a same-named user's session", async
}),
);
wrap(adminUser);
await userEvent.click(await screen.findByRole("tab", { name: "All sessions" }));
await userEvent.click(await screen.findByRole("button", { name: "All sessions" }));
await screen.findByText("Owner: Alice");
await userEvent.click(screen.getByRole("checkbox", { name: "Select Attiva uno" }));
await userEvent.click(screen.getByRole("button", { name: "Delete 1 selected sessions" }));
@@ -275,7 +213,7 @@ test("administrator confirms before archiving a same-named user's session", asyn
}),
);
wrap(adminUser);
await userEvent.click(await screen.findByRole("tab", { name: "All sessions" }));
await userEvent.click(await screen.findByRole("button", { name: "All sessions" }));
await screen.findByText("Owner: Alice");
await userEvent.click(screen.getByRole("button", { name: "Session actions" }));
await userEvent.click(await screen.findByText("Archive"));
+94 -219
View File
@@ -17,9 +17,8 @@ import { StopConfirmDialog } from "./StopConfirmDialog";
import { SteerInput, ComposerFooter } from "./SteerInput";
import { WorkflowBar } from "./WorkflowBar";
import { DatabaseManagementPage } from "./DatabaseManagementPage";
import { Pencil, ArrowLeft, ArrowRight, ChevronDown, Trash2 } from "lucide-react";
import { Accordion } from "@base-ui/react/accordion";
import { Button, buttonVariants } from "../components/ui/button";
import { Pencil, ArrowLeft, ArrowRight, Trash2 } from "lucide-react";
import { Button } from "../components/ui/button";
import { Checkbox } from "../components/ui/checkbox";
import { Toaster } from "../components/ui/sonner";
import { toast } from "sonner";
@@ -35,7 +34,7 @@ import type { SessionScope, SessionSummary } from "../api/types";
import { useAuthGeneration, useAuthUser } from "../auth/authState";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import type { CSSProperties, KeyboardEvent } from "react";
import type { CSSProperties } from "react";
import { captureAuthOperation, isAuthOperationCurrent, StaleAuthOperationError, type AuthOperationGuard } from "../auth/authOperation";
interface AppShellProps {
@@ -44,31 +43,6 @@ interface AppShellProps {
type ActiveSurface = "core" | "database-management";
type ActiveManagementPanel = "workspace" | "pi" | null;
const SESSION_SCOPES: readonly SessionScope[] = ["mine", "all"];
const APP_NAVIGATION_STORAGE_KEY = "thothii:app-navigation";
function readPersistedNavigation(): { surface: ActiveSurface; panel: ActiveManagementPanel } {
if (import.meta.env.MODE === "test") return { surface: "core", panel: null };
try {
const parsed = JSON.parse(window.localStorage.getItem(APP_NAVIGATION_STORAGE_KEY) ?? "null") as {
surface?: unknown;
panel?: unknown;
} | null;
return {
surface: parsed?.surface === "database-management" ? "database-management" : "core",
panel: parsed?.panel === "workspace" || parsed?.panel === "pi" ? parsed.panel : null,
};
} catch {
return { surface: "core", panel: null };
}
}
function sessionScopeTabClass(selected: boolean): string {
const base = "relative -mb-px flex h-8 w-full cursor-pointer items-center justify-center rounded-t-md border border-b-2 px-2 text-xs font-semibold tracking-[0.005em] outline-none focus-visible:z-10 focus-visible:ring-3 focus-visible:ring-[oklch(var(--nav-active-border)/0.28)]";
return selected
? `${base} border-[oklch(var(--nav-active-border))] bg-[oklch(var(--nav-active))] text-[oklch(var(--nav-active-foreground))] hover:bg-[oklch(var(--nav-active-hover))]`
: `${base} border-border bg-card text-muted-foreground hover:border-muted-foreground/45 hover:bg-muted/55 hover:text-foreground`;
}
type DatabaseManagementPresentation = "legacy" | "fleet";
@@ -155,13 +129,8 @@ export function AppShell({ canLogout }: AppShellProps) {
const [creatingSession, setCreatingSession] = useState(false);
const [awaitingQuestion, setAwaitingQuestion] = useState(false);
const [sessionScope, setSessionScope] = useState<SessionScope>("mine");
const sessionScopeTabRefs = useRef<Record<SessionScope, HTMLButtonElement | null>>({
mine: null,
all: null,
});
const principal = authenticatedUser;
const permissions = authenticatedUser?.permissions ?? [];
const isAdmin = authenticatedUser?.isAdmin === true;
const canReadAllSessions = permissions.includes("session.read_all");
const canManageWorkspace = permissions.includes("workspace.manage");
const canManageWorkspaceSecrets = permissions.includes("workspace.secrets.manage");
@@ -183,8 +152,7 @@ export function AppShell({ canLogout }: AppShellProps) {
const queryClient = useQueryClient();
const [showActivity, setShowActivity] = useState(false);
const persistedNavigation = readPersistedNavigation();
const [activeSurface, setActiveSurface] = useState<ActiveSurface>(persistedNavigation.surface);
const [activeSurface, setActiveSurface] = useState<ActiveSurface>("core");
const databaseManagementPresentation = resolveDatabaseManagementPresentation({
isDevelopment: import.meta.env.DEV,
mode: import.meta.env.MODE,
@@ -195,8 +163,7 @@ export function AppShell({ canLogout }: AppShellProps) {
const updateDatabaseNavigationState = useCallback((state: { dirty: boolean; busy: boolean }) => {
databaseNavigationRef.current = state;
}, []);
const [activeManagementPanel, setActiveManagementPanel] = useState<ActiveManagementPanel>(persistedNavigation.panel);
const [adminNavigationValue, setAdminNavigationValue] = useState<string[]>([]);
const [activeManagementPanel, setActiveManagementPanel] = useState<ActiveManagementPanel>(null);
const [activeOpen, setActiveOpen] = useState(true);
const [archiveOpen, setArchiveOpen] = useState(false);
const [renameTarget, setRenameTarget] = useState<SessionSummary | null>(null);
@@ -206,10 +173,6 @@ export function AppShell({ canLogout }: AppShellProps) {
const [collapsedGroups, setCollapsedGroups] = useState<Record<string, boolean>>({});
const [renameGroupTarget, setRenameGroupTarget] = useState<string | null>(null);
const operationEpochRef = useRef(0);
useEffect(() => {
if (import.meta.env.MODE === "test") return;
window.localStorage.setItem(APP_NAVIGATION_STORAGE_KEY, JSON.stringify({ surface: activeSurface, panel: activeManagementPanel }));
}, [activeManagementPanel, activeSurface]);
useEffect(() => () => { operationEpochRef.current += 1; }, []);
const groups = useMemo(
@@ -273,22 +236,6 @@ export function AppShell({ canLogout }: AppShellProps) {
setSelectedSessionIds(selected ? new Set(sessions.map((session) => session.id)) : new Set());
}
function handleSessionScopeTabKeyDown(
event: KeyboardEvent<HTMLButtonElement>,
current: SessionScope,
) {
const currentIndex = SESSION_SCOPES.indexOf(current);
let target: SessionScope | undefined;
if (event.key === "ArrowRight") target = SESSION_SCOPES[(currentIndex + 1) % SESSION_SCOPES.length];
else if (event.key === "ArrowLeft") target = SESSION_SCOPES[(currentIndex - 1 + SESSION_SCOPES.length) % SESSION_SCOPES.length];
else if (event.key === "Home") target = SESSION_SCOPES[0];
else if (event.key === "End") target = SESSION_SCOPES.at(-1);
if (!target) return;
event.preventDefault();
setSessionScope(target);
sessionScopeTabRefs.current[target]?.focus();
}
function canLeaveDatabaseManagement(): boolean {
if (activeSurface !== "database-management") return true;
if (databaseNavigationRef.current.busy) {
@@ -680,12 +627,6 @@ export function AppShell({ canLogout }: AppShellProps) {
await logoutUser();
}
const currentNavigation = activeSurface === "database-management"
? "database"
: activeManagementPanel ?? "core";
const adminNavigationOpen = adminNavigationValue.includes("administration");
const managementNavigationCurrent = currentNavigation !== "core";
return (
<div
ref={containerRef}
@@ -777,7 +718,7 @@ export function AppShell({ canLogout }: AppShellProps) {
<>
{activeSession?.question && (
<header className="sticky top-0 z-10 -mx-6 -mt-8 border-b border-border/60 bg-background/95 px-6 pb-3 pt-8 backdrop-blur supports-[backdrop-filter]:bg-background/80">
<p className="thot-label mb-0.5 text-muted-foreground">Question</p>
<p className="thot-label mb-0.5 text-muted-foreground">Domanda</p>
<h2 className="font-heading text-[0.95rem] font-semibold leading-snug text-foreground break-words">
{activeSession.question}
</h2>
@@ -849,7 +790,7 @@ export function AppShell({ canLogout }: AppShellProps) {
<p className="thot-label mt-1.5">
Datamart Builder with
<br />
Human-in-the-loop review
Human In The Loop
</p>
{authenticatedUser && (
<div className="mt-4 flex items-center justify-between gap-2 border-t border-border/70 pt-3 text-left">
@@ -867,180 +808,115 @@ export function AppShell({ canLogout }: AppShellProps) {
<div className="flex flex-col gap-2 px-4 pb-3">
<Button
variant={currentNavigation === "core" ? "navigationActive" : "outline"}
variant="default"
size="sm"
className="w-full"
aria-current={currentNavigation === "core" ? "page" : undefined}
data-navigation-state={currentNavigation === "core" ? "current" : "available"}
onClick={startNewSession}
>
New session
</Button>
{isAdmin && (
<Accordion.Root
value={adminNavigationValue}
onValueChange={setAdminNavigationValue}
className="rounded-lg border border-border/80 bg-card/45 p-1.5"
<Button
variant="outline"
size="sm"
className="w-full"
onClick={() => {
if (!canLeaveDatabaseManagement()) return;
setActiveSurface("core");
setActiveManagementPanel("workspace");
}}
>
Workspace management
</Button>
{canManageDatabase && (
<Button
variant={activeSurface === "database-management" ? "secondary" : "outline"}
size="sm"
className="w-full"
aria-current={activeSurface === "database-management" ? "page" : undefined}
onClick={() => {
setActiveManagementPanel(null);
setActiveSurface("database-management");
}}
>
<Accordion.Item value="administration">
<Accordion.Header className="m-0">
<Accordion.Trigger
className={buttonVariants({
variant: !adminNavigationOpen && managementNavigationCurrent ? "navigationActive" : "outline",
size: "sm",
className: "w-full justify-between px-2.5",
})}
data-navigation-state={!adminNavigationOpen && managementNavigationCurrent ? "current" : "available"}
>
<span>Administration</span>
<ChevronDown
aria-hidden="true"
data-icon="inline-end"
className={`transition-transform duration-150 motion-reduce:transition-none ${adminNavigationOpen ? "rotate-180" : ""}`}
/>
</Accordion.Trigger>
</Accordion.Header>
<Accordion.Panel className="grid gap-1.5 px-0.5 pt-1.5">
<Button
variant={currentNavigation === "database" ? "navigationActive" : "outline"}
size="sm"
className="w-full"
aria-current={currentNavigation === "database" ? "page" : undefined}
data-navigation-state={currentNavigation === "database" ? "current" : canManageDatabase ? "available" : "unavailable"}
disabled={!canManageDatabase}
title={canManageDatabase ? undefined : "Database management permission is required"}
onClick={() => {
setActiveManagementPanel(null);
setActiveSurface("database-management");
}}
>
Database management
</Button>
<div role="separator" aria-orientation="horizontal" className="mx-1 h-px bg-border/90" />
<Button
variant={currentNavigation === "workspace" ? "navigationActive" : "outline"}
size="sm"
className="w-full"
aria-current={currentNavigation === "workspace" ? "page" : undefined}
aria-expanded={activeManagementPanel === "workspace"}
data-navigation-state={currentNavigation === "workspace" ? "current" : "available"}
onClick={() => {
if (!canLeaveDatabaseManagement()) return;
setActiveSurface("core");
setActiveManagementPanel("workspace");
}}
>
Workspace management
</Button>
<Button
variant={currentNavigation === "pi" ? "navigationActive" : "outline"}
size="sm"
className="w-full"
aria-current={currentNavigation === "pi" ? "page" : undefined}
aria-expanded={activeManagementPanel === "pi"}
data-navigation-state={currentNavigation === "pi" ? "current" : canManagePi ? "available" : "unavailable"}
disabled={!canManagePi}
title={canManagePi ? undefined : "Pi management permission is required"}
onClick={() => {
if (!canLeaveDatabaseManagement()) return;
setActiveSurface("core");
setActiveManagementPanel("pi");
}}
>
Pi management
</Button>
</Accordion.Panel>
</Accordion.Item>
</Accordion.Root>
Database management
</Button>
)}
{canManagePi && (
<Button
variant="outline"
size="sm"
className="w-full"
onClick={() => {
if (!canLeaveDatabaseManagement()) return;
setActiveSurface("core");
setActiveManagementPanel("pi");
}}
>
Pi management
</Button>
)}
</div>
{canReadAllSessions && (
<div className="px-4">
<div
role="tablist"
aria-label="Session scope"
aria-orientation="horizontal"
className="grid grid-cols-2 border-b border-border"
>
<button
ref={(node) => { sessionScopeTabRefs.current.mine = node; }}
id="session-scope-mine-tab"
type="button"
role="tab"
aria-selected={sessionScope === "mine"}
aria-controls="session-scope-panel"
tabIndex={sessionScope === "mine" ? 0 : -1}
data-tab-state={sessionScope === "mine" ? "active" : "inactive"}
className={sessionScopeTabClass(sessionScope === "mine")}
<div className="px-4 pb-3">
<div className="grid grid-cols-2 gap-1 rounded-lg bg-muted p-1" aria-label="Session scope">
<Button
variant={sessionScope === "mine" ? "secondary" : "ghost"}
size="xs"
aria-pressed={sessionScope === "mine"}
onClick={() => setSessionScope("mine")}
onKeyDown={(event) => handleSessionScopeTabKeyDown(event, "mine")}
>
My sessions
</button>
<button
ref={(node) => { sessionScopeTabRefs.current.all = node; }}
id="session-scope-all-tab"
type="button"
role="tab"
aria-selected={showingAllSessions}
aria-controls="session-scope-panel"
tabIndex={showingAllSessions ? 0 : -1}
data-tab-state={showingAllSessions ? "active" : "inactive"}
className={sessionScopeTabClass(showingAllSessions)}
</Button>
<Button
variant={showingAllSessions ? "secondary" : "ghost"}
size="xs"
aria-pressed={showingAllSessions}
onClick={() => setSessionScope("all")}
onKeyDown={(event) => handleSessionScopeTabKeyDown(event, "all")}
>
All sessions
</button>
</Button>
</div>
{showingAllSessions && (
<p className="mt-2 text-xs font-medium text-amber-700 dark:text-amber-400">
Administrator view: all sessions
</p>
)}
</div>
)}
<div
id={canReadAllSessions ? "session-scope-panel" : undefined}
role={canReadAllSessions ? "tabpanel" : undefined}
aria-labelledby={canReadAllSessions ? `session-scope-${sessionScope}-tab` : undefined}
className="flex min-h-0 flex-1 flex-col"
>
{canReadAllSessions && showingAllSessions && (
<p className="mx-4 mb-2 mt-2 text-xs font-medium text-amber-700 dark:text-amber-400">
Administrator view: all sessions
</p>
{/* L1 — rail title */}
<div className="px-4 pb-1.5 pt-1">
<span className="thot-label text-[0.8rem] font-bold tracking-[0.18em] text-primary">
Sessions
</span>
</div>
<div className="flex items-center justify-between px-4 pb-2">
<label className="flex cursor-pointer items-center gap-2 text-xs font-medium text-muted-foreground hover:text-foreground">
<Checkbox
checked={allSessionsSelected}
aria-label="Select all sessions"
disabled={sessions.length === 0}
onCheckedChange={(selected) => toggleAllSessions(selected === true)}
/>
<span>Select all</span>
</label>
{selectedSessions.length > 0 && (
<Button
variant="destructive"
size="xs"
aria-label={`Delete ${selectedSessions.length} selected sessions`}
onClick={() => {
requestDelete(selectedSessions);
}}
>
<Trash2 />
Delete ({selectedSessions.length})
</Button>
)}
{/* L1 — rail title */}
<div className="px-4 pb-1.5 pt-1">
<span className="thot-label text-[0.8rem] font-bold tracking-[0.18em] text-primary">
Sessions
</span>
</div>
<div className="flex items-center justify-between px-4 pb-2">
<label className="flex cursor-pointer items-center gap-2 text-xs font-medium text-muted-foreground hover:text-foreground">
<Checkbox
checked={allSessionsSelected}
aria-label="Select all sessions"
disabled={sessions.length === 0}
onCheckedChange={(selected) => toggleAllSessions(selected === true)}
/>
<span>Select all</span>
</label>
{selectedSessions.length > 0 && (
<Button
variant="destructive"
size="xs"
aria-label={`Delete ${selectedSessions.length} selected sessions`}
onClick={() => {
requestDelete(selectedSessions);
}}
>
<Trash2 />
Delete ({selectedSessions.length})
</Button>
)}
</div>
<div className="flex-1 overflow-y-auto px-2 pb-4">
</div>
<div className="flex-1 overflow-y-auto px-2 pb-4">
{/* L2 — section toggle */}
<button
type="button"
@@ -1126,7 +1002,6 @@ export function AppShell({ canLogout }: AppShellProps) {
showOwner={showingAllSessions}
/>
)}
</div>
</div>
</aside>
)}
+132 -324
View File
@@ -29,13 +29,6 @@ function makeDatabase(overrides: Partial<CatalogDatabase> = {}): CatalogDatabase
workspaceId: "psd-clinical",
workspaceName: "Policlinico San Donato",
workspaceAvailable: true,
workspaceRevision: { commit: "a".repeat(40), blob: "b".repeat(40) },
workspaceEvidence: { sourceType: "filesystem", state: "materialized_current_revision" },
runtimeBinding: {
transport: "postgres_direct",
configurationState: "ready",
sessionTransportSupported: true,
},
configured: true,
engine: "postgres",
databaseName: "warehouse",
@@ -73,9 +66,6 @@ const orphan = makeDatabase({
workspaceId: "retired",
workspaceName: "Retired workspace",
workspaceAvailable: false,
workspaceRevision: null,
workspaceEvidence: { sourceType: null, state: "workspace_unavailable" },
runtimeBinding: null,
});
function renderPage({
@@ -197,11 +187,12 @@ function registerCompletedSyncRun(run: CatalogSyncRun) {
const synchronizationScopes = [
{ scope: "tables", label: "Synchronize tables" },
{ scope: "columns", label: "Synchronize all columns" },
{ scope: "relationships", label: "Synchronize relationships" },
{ scope: "all", label: "Synchronize all" },
] as const;
test("renders Fleet Ledger with real metrics, conceptual row tooltips, and persistent model selection", async () => {
test("renders Fleet Ledger with real metrics, conceptual row tooltips, and contextual model selection", async () => {
const user = userEvent.setup();
server.use(
http.get("/api/catalog/metrics", () => HttpResponse.json({
@@ -232,13 +223,10 @@ test("renders Fleet Ledger with real metrics, conceptual row tooltips, and persi
expect(screen.queryByText("Thoth catalog · Fleet ledger")).not.toBeInTheDocument();
expect(screen.queryByText("Inspect physical metadata, curate descriptions and control catalog operations.")).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Back to workspace" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: /Add database/i })).not.toBeInTheDocument();
expect(screen.getByRole("columnheader", { name: /Catalog status/ })).toBeVisible();
const summary = screen.getByRole("region", { name: "Fleet summary" });
expect(await within(summary).findByText("2,275")).toBeVisible();
expect(within(summary).getByText("75%")).toBeVisible();
const modelSelector = await screen.findByRole("combobox", { name: "Metadata-generation LLM model" });
expect(modelSelector).toHaveValue("local-qwen");
expect(screen.queryByRole("combobox", { name: "Metadata description model" })).not.toBeInTheDocument();
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
const tablesAction = within(databaseRow).getByRole("button", { name: "View tables for Policlinico San Donato" });
@@ -258,143 +246,12 @@ test("renders Fleet Ledger with real metrics, conceptual row tooltips, and persi
await user.click(within(restoredDatabaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
const actionPicker = screen.getByRole("combobox", { name: "Batch action" });
expect(within(actionPicker).getByRole("option", { name: /^Synchronize all(?:,|$)/ })).toBeVisible();
expect(within(actionPicker).queryByRole("option", { name: "Synchronize all columns" })).not.toBeInTheDocument();
await user.selectOptions(actionPicker, "generate-missing");
expect(screen.getAllByRole("combobox", { name: "Metadata-generation LLM model" })).toHaveLength(1);
expect(modelSelector).toHaveValue("local-qwen");
expect(await screen.findByRole("combobox", { name: "Metadata description model" })).toHaveValue("local-qwen");
});
test("shows synchronization, description, and sensitive-data states independently", async () => {
const configured = makeDatabase({ connectionStatus: "reachable", testedVersion: 3 });
const needsRuntimeConfiguration = makeDatabase({
...unconfigured,
workspaceEvidence: { sourceType: "http", state: "configuration_required" },
runtimeBinding: {
transport: "rest_api",
configurationState: "configuration_required",
sessionTransportSupported: true,
},
});
renderPage({ rows: [configured, needsRuntimeConfiguration, orphan], presentation: "fleet" });
const configuredRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
expect(within(configuredRow).getByText("Not synchronized")).toBeVisible();
expect(within(configuredRow).getByText(/Descriptions ·/)).toBeVisible();
expect(within(configuredRow).getByText(/Sensitive ·/)).toBeVisible();
const needsConfigurationRow = screen.getByRole("row", { name: /Research laboratory/ });
expect(within(needsConfigurationRow).getByText("Not synchronized")).toBeVisible();
expect(within(needsConfigurationRow).getByText(/Descriptions ·/)).toBeVisible();
const orphanRow = screen.getByRole("row", { name: /Retired workspace/ });
expect(within(orphanRow).getByText("Not synchronized")).toBeVisible();
});
test("shows the configured access type and endpoint in catalog status", async () => {
const direct = makeDatabase();
const rest = makeDatabase({
id: "33333333-3333-4333-8333-333333333333",
workspaceId: "rest-lab",
workspaceName: "REST laboratory",
binding: {
transport: "rest_api",
baseUrl: "https://203.0.113.18/dwh/",
restPath: "/health",
restAuth: "x-api-key",
},
});
const ssh = makeDatabase({
id: "44444444-4444-4444-8444-444444444444",
workspaceId: "ssh-lab",
workspaceName: "SSH laboratory",
binding: {
transport: "ssh_tunnel",
sshHost: "bastion.example.test",
sshPort: 2222,
sshUsername: "operator",
sshTargetHost: "postgres.internal",
sshTargetPort: 5432,
username: "reader",
},
});
renderPage({ rows: [direct, rest, ssh], presentation: "fleet" });
const directRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
expect(within(directRow).getByText("Direct PostgreSQL connection")).toBeVisible();
expect(within(directRow).getByText("db.internal:5432")).toBeVisible();
const restRow = screen.getByRole("row", { name: /REST laboratory/ });
expect(within(restRow).getByText("REST")).toBeVisible();
expect(within(restRow).getByText("203.0.113.18")).toBeVisible();
const sshRow = screen.getByRole("row", { name: /SSH laboratory/ });
expect(within(sshRow).getByText("SSH")).toBeVisible();
expect(within(sshRow).getByText("bastion.example.test:2222")).toBeVisible();
});
test("keeps Fleet Ledger visible when a catalog response omits the Evidence projection", async () => {
const legacyDatabase = makeDatabase();
delete (legacyDatabase as Partial<CatalogDatabase>).workspaceEvidence;
renderPage({ rows: [legacyDatabase], presentation: "fleet" });
expect(await screen.findByRole("heading", { name: "Database management" })).toBeVisible();
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
expect(within(databaseRow).getByText("Not synchronized")).toBeVisible();
});
test("opens the relationship map directly from a Fleet database and restores focus on return", async () => {
const user = userEvent.setup();
server.use(
http.get("/api/catalog/databases/:databaseId/relationships", () => HttpResponse.json([])),
);
renderPage({
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
presentation: "fleet",
});
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
const relationshipsAction = within(databaseRow).getByRole("button", {
name: "View relationships for Policlinico San Donato",
});
expect(relationshipsAction.parentElement).toHaveAttribute("data-tooltip", "Relationships");
expect(relationshipsAction).not.toHaveAttribute("title");
await user.click(relationshipsAction);
expect(await screen.findByRole("region", {
name: "Relationships for Policlinico San Donato",
})).toBeVisible();
expect(screen.getByText("Relationship map")).toBeVisible();
await user.click(screen.getByRole("button", { name: "Back to databases" }));
await waitFor(() => expect(screen.getByRole("button", {
name: "View relationships for Policlinico San Donato",
})).toHaveFocus());
});
test("offers catalog configuration directly on an unconfigured Fleet workspace", async () => {
const user = userEvent.setup();
renderPage({ rows: [unconfigured], presentation: "fleet" });
const databaseRow = await screen.findByRole("row", { name: /Research laboratory/ });
expect(within(databaseRow).queryByRole("button", {
name: "View relationships for Research laboratory",
})).not.toBeInTheDocument();
expect(within(databaseRow).queryByRole("button", {
name: "View tables for Research laboratory",
})).not.toBeInTheDocument();
await user.click(within(databaseRow).getByRole("button", {
name: "Configure catalog for Research laboratory",
}));
const drawer = await screen.findByRole("dialog", { name: "Configure catalog" });
expect(within(drawer).getByLabelText("Workspace")).toHaveAttribute("readonly");
expect(within(drawer).getByLabelText("Workspace")).toHaveValue("Research laboratory");
});
test("keeps the table back action in the Fleet Ledger breadcrumb while browsing columns", async () => {
test("keeps only the local back action while browsing Fleet Ledger columns", async () => {
const user = userEvent.setup();
server.use(
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
@@ -407,17 +264,9 @@ test("keeps the table back action in the Fleet Ledger breadcrumb while browsing
name: "View tables for Policlinico San Donato",
}));
expect(await screen.findByRole("button", { name: "Back to databases" })).toBeVisible();
expect(screen.getByRole("combobox", { name: "Table action" })).toBeDisabled();
expect(screen.queryByRole("button", { name: "Sync history" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Description history" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Sensitive history" })).not.toBeInTheDocument();
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const tableBackAction = await screen.findByRole("button", { name: "Back to tables" });
expect(tableBackAction).toBeVisible();
expect(screen.getByRole("combobox", { name: "Column action" })).toBeDisabled();
expect(screen.getByRole("navigation", { name: "Database hierarchy" }).parentElement)
.toContainElement(tableBackAction);
expect(await screen.findByRole("button", { name: "Back to tables" })).toBeVisible();
await waitFor(() => {
expect(screen.queryByRole("button", { name: "Back to databases" })).not.toBeInTheDocument();
});
@@ -427,32 +276,6 @@ test("keeps the table back action in the Fleet Ledger breadcrumb while browsing
expect(screen.queryByRole("button", { name: "Back to tables" })).not.toBeInTheDocument();
});
test("uses the database back-action background treatment for the table back action", async () => {
const user = userEvent.setup();
server.use(
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
http.get("/api/catalog/databases/:databaseId/tables/:tableId/columns", () => HttpResponse.json([])),
);
renderPage({ rows: [makeDatabase()], presentation: "fleet" });
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("button", {
name: "View tables for Policlinico San Donato",
}));
const databaseBackAction = await screen.findByRole("button", { name: "Back to databases" });
expect(databaseBackAction).toHaveClass("thot-fleet-back-action");
const backgroundTreatment = (element: HTMLElement) => element.className
.split(/\s+/)
.filter((className) => /(^|:)bg-/.test(className))
.sort();
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
const tableBackAction = await screen.findByRole("button", { name: "Back to tables" });
expect(tableBackAction).toHaveClass("thot-fleet-back-action");
expect(backgroundTreatment(tableBackAction)).toEqual(backgroundTreatment(databaseBackAction));
});
test("reopens database synchronization history when no run is active", async () => {
const user = userEvent.setup();
const completed: CatalogSyncRun = {
@@ -470,7 +293,7 @@ test("reopens database synchronization history when no run is active", async ()
renderPage({ rows: [makeDatabase()], presentation: "fleet" });
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByText("Policlinico San Donato"));
await user.click(within(databaseRow).getByRole("button", { name: "View synchronization history for Policlinico San Donato" }));
const drawer = await screen.findByRole("dialog", { name: "Schema synchronization" });
expect(drawer).toBeVisible();
@@ -489,7 +312,7 @@ test("selects the configured metadata-description model by default", async () =>
})));
renderPage();
const selector = await screen.findByRole("combobox", { name: "Metadata-generation LLM model" });
const selector = await screen.findByRole("combobox", { name: "Metadata description model" });
await waitFor(() => expect(selector).toHaveValue("local-qwen"));
});
@@ -497,15 +320,15 @@ test("keeps both run-history buttons visible beside the metadata-description sel
const user = userEvent.setup();
renderPage();
const selector = await screen.findByRole("combobox", { name: "Metadata-generation LLM model" });
const selector = await screen.findByRole("combobox", { name: "Metadata description model" });
const toolbar = screen.getByRole("group", { name: "Database management controls" });
const metadataControls = screen.getByRole("group", { name: "Metadata description controls" });
const actions = screen.getByRole("group", { name: "Database management actions" });
const descriptionHistoryButton = within(metadataControls).getByRole("button", {
name: "View description generation history",
name: "Run descriptions generation history",
});
const suggestionHistoryButton = within(metadataControls).getByRole("button", {
name: "View sensitive suggestion history",
name: "Run sensitive suggestions history",
});
expect(toolbar).toHaveClass("sm:items-end", "sm:justify-between");
@@ -513,15 +336,15 @@ test("keeps both run-history buttons visible beside the metadata-description sel
expect(metadataControls).toContainElement(selector);
expect(descriptionHistoryButton).toBeVisible();
expect(descriptionHistoryButton).toBeEnabled();
expect(descriptionHistoryButton).toHaveTextContent("View description generation history");
expect(descriptionHistoryButton).toHaveTextContent("Run descriptions generation history");
expect(descriptionHistoryButton).toHaveClass("disabled:opacity-70");
expect(suggestionHistoryButton).toBeVisible();
expect(suggestionHistoryButton).toBeEnabled();
expect(suggestionHistoryButton).toHaveTextContent("View sensitive suggestion history");
expect(suggestionHistoryButton).toHaveTextContent("Run sensitive suggestions history");
expect(toolbar.lastElementChild).toBe(actions);
expect(actions).toHaveClass("sm:justify-end");
expect(actions).toContainElement(screen.getByRole("button", { name: "Refresh" }));
expect(within(actions).queryByRole("button", { name: /Add database/i })).not.toBeInTheDocument();
expect(actions).toContainElement(screen.getByRole("button", { name: "Add database" }));
expect(screen.queryByRole("note", {
name: "Metadata generation source data disclosure",
})).not.toBeInTheDocument();
@@ -548,7 +371,7 @@ test("changes the metadata-description model in page-local state", async () => {
})));
renderPage();
const selector = await screen.findByRole("combobox", { name: "Metadata-generation LLM model" });
const selector = await screen.findByRole("combobox", { name: "Metadata description model" });
await waitFor(() => expect(selector).toHaveValue("local-qwen"));
await user.selectOptions(selector, "openai-mini");
@@ -562,10 +385,10 @@ test("explains application setup when no metadata-description model is configure
})));
renderPage();
const selector = await screen.findByRole("combobox", { name: "Metadata-generation LLM model" });
const selector = await screen.findByRole("combobox", { name: "Metadata description model" });
expect(selector).toBeDisabled();
expect(await screen.findByText(
"No metadata-generation LLM model is configured for this installation.",
"Configure a metadata-generation model in application setup to enable description generation.",
)).toBeVisible();
});
@@ -576,11 +399,11 @@ test("uses a safe disabled setup state when metadata-description models are unav
}, { status: 503 })));
renderPage();
const selector = await screen.findByRole("combobox", { name: "Metadata-generation LLM model" });
const selector = await screen.findByRole("combobox", { name: "Metadata description model" });
await waitFor(() => expect(selector).toHaveDisplayValue("Model choices unavailable"));
expect(selector).toBeDisabled();
expect(screen.getByText(
"Metadata-generation LLM model choices are unavailable.",
"Configure a metadata-generation model in application setup to enable description generation.",
)).toBeVisible();
expect(screen.queryByText("internal model registry details")).not.toBeInTheDocument();
});
@@ -594,11 +417,11 @@ test.each([null, "missing-model"])(
})));
renderPage();
const selector = await screen.findByRole("combobox", { name: "Metadata-generation LLM model" });
const selector = await screen.findByRole("combobox", { name: "Metadata description model" });
await waitFor(() => expect(selector).toHaveDisplayValue("No usable model configured"));
expect(selector).toBeDisabled();
expect(screen.getByText(
"No metadata-generation LLM model is configured for this installation.",
"Configure a metadata-generation model in application setup to enable description generation.",
)).toBeVisible();
},
);
@@ -618,7 +441,7 @@ test("renders only the public metadata-generation model contract", async () => {
})));
renderPage();
const selector = await screen.findByRole("combobox", { name: "Metadata-generation LLM model" });
const selector = await screen.findByRole("combobox", { name: "Metadata description model" });
await waitFor(() => expect(selector).toHaveValue("approved-model"));
expect(within(selector).getAllByRole("option").map((option) => option.textContent))
.toEqual(["Approved model"]);
@@ -635,9 +458,8 @@ test("starts with a full-width list and applies the row action matrix", async ()
expect(screen.getByRole("button", { name: "Delete Policlinico San Donato" })).toBeEnabled();
expect(screen.getByRole("button", { name: "View Research laboratory" })).toBeEnabled();
expect(screen.getByRole("button", { name: "Configure catalog for Research laboratory" })).toBeEnabled();
expect(screen.queryByRole("button", { name: "Edit Research laboratory" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Delete Research laboratory" })).not.toBeInTheDocument();
expect(screen.getByRole("button", { name: "Edit Research laboratory" })).toBeEnabled();
expect(screen.getByRole("button", { name: "Delete Research laboratory" })).toBeDisabled();
expect(screen.getByRole("button", { name: "View Retired workspace" })).toBeEnabled();
expect(screen.getByRole("button", { name: "Edit Retired workspace" })).toBeDisabled();
@@ -671,11 +493,10 @@ test.each(synchronizationScopes)(
await user.click(screen.getByRole("menuitem", { name: label }));
await waitFor(() => expect(startBody).toEqual({ version: 3, scope, tableIds: [] }));
expect(screen.getByText("1 selected")).toBeVisible();
},
);
test("discloses source sampling before starting Generate missing descriptions", async () => {
test("discloses source sampling before starting Generate Missing", async () => {
const user = userEvent.setup();
const queuedRun = makeDescriptionGenerationRun({ scope: "missing", total: 3 });
let startBody: unknown;
@@ -696,7 +517,7 @@ test("discloses source sampling before starting Generate missing descriptions",
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate missing descriptions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate Missing" }));
expect(screen.getByText("Generate missing descriptions for Policlinico San Donato?")).toBeVisible();
expect(screen.getByRole("note", {
@@ -706,7 +527,7 @@ test("discloses source sampling before starting Generate missing descriptions",
);
expect(startBody).toBeUndefined();
await user.click(screen.getByRole("button", { name: "Generate missing descriptions" }));
await user.click(screen.getByRole("button", { name: "Generate Missing" }));
await waitFor(() => expect(startBody).toEqual({
modelId: "local-qwen",
scope: "missing",
@@ -715,7 +536,7 @@ test("discloses source sampling before starting Generate missing descriptions",
expect(await screen.findByRole("dialog", { name: "Description generation" })).toBeVisible();
});
test("keeps Fleet Generate missing descriptions behind the source-data disclosure", async () => {
test("keeps Fleet Generate Missing behind the source-data disclosure", async () => {
const user = userEvent.setup();
let generationStarts = 0;
server.use(
@@ -737,7 +558,7 @@ test("keeps Fleet Generate missing descriptions behind the source-data disclosur
"generate-missing",
);
await waitFor(() => expect(screen.getByRole("combobox", {
name: "Metadata-generation LLM model",
name: "Metadata description model",
})).toHaveValue("local-qwen"));
await user.click(screen.getByRole("button", { name: "Run action" }));
@@ -753,7 +574,7 @@ test("keeps Fleet Generate missing descriptions behind the source-data disclosur
expect(generationStarts).toBe(0);
});
test("confirms generating all descriptions, supports cancel, and sends the database-wide scope", async () => {
test("confirms Generate All replacement, supports cancel, and sends the database-wide scope", async () => {
const user = userEvent.setup();
const queuedRun = makeDescriptionGenerationRun({ scope: "all", total: 6 });
let startBody: unknown;
@@ -774,7 +595,7 @@ test("confirms generating all descriptions, supports cancel, and sends the datab
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate all descriptions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate All" }));
expect(screen.getByText("Replace generated descriptions for Policlinico San Donato?")).toBeVisible();
expect(screen.getByText(/existing generated descriptions for eligible tables and columns will be replaced/i)).toBeVisible();
@@ -788,8 +609,8 @@ test("confirms generating all descriptions, supports cancel, and sends the datab
expect(startBody).toBeUndefined();
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate all descriptions" }));
await user.click(screen.getByRole("button", { name: "Generate all descriptions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate All" }));
await user.click(screen.getByRole("button", { name: "Generate All" }));
await waitFor(() => expect(startBody).toEqual({
modelId: "local-qwen",
@@ -815,8 +636,8 @@ test("keeps the database selected and safely explains when no descriptions are e
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate missing descriptions" }));
await user.click(screen.getByRole("button", { name: "Generate missing descriptions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate Missing" }));
await user.click(screen.getByRole("button", { name: "Generate Missing" }));
expect(await screen.findByText(
"No eligible catalog tables or columns need description generation.",
@@ -876,10 +697,10 @@ test.each([
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", {
name: scope === "all" ? "Generate all descriptions" : "Generate missing descriptions",
name: scope === "all" ? "Generate All" : "Generate Missing",
}));
await user.click(screen.getByRole("button", {
name: scope === "all" ? "Generate all descriptions" : "Generate missing descriptions",
name: scope === "all" ? "Generate All" : "Generate Missing",
}));
expect(await screen.findByRole("heading", {
@@ -954,9 +775,9 @@ test.each([
}
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate all descriptions" }))
expect(await screen.findByRole("menuitem", { name: "Generate All" }))
.toHaveAttribute("aria-disabled", "true");
expect(screen.getByRole("menuitem", { name: "Generate missing descriptions" }))
expect(screen.getByRole("menuitem", { name: "Generate Missing" }))
.toHaveAttribute("aria-disabled", "true");
});
@@ -983,16 +804,16 @@ test("disables database-wide generation while a description generation is active
let databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate missing descriptions" }));
await user.click(screen.getByRole("button", { name: "Generate missing descriptions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate Missing" }));
await user.click(screen.getByRole("button", { name: "Generate Missing" }));
expect(await screen.findByRole("dialog", { name: "Description generation" })).toBeVisible();
databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate all descriptions" }))
expect(await screen.findByRole("menuitem", { name: "Generate All" }))
.toHaveAttribute("aria-disabled", "true");
expect(screen.getByRole("menuitem", { name: "Generate missing descriptions" }))
expect(screen.getByRole("menuitem", { name: "Generate Missing" }))
.toHaveAttribute("aria-disabled", "true");
});
@@ -1045,16 +866,15 @@ test("selected database Actions confirms and deletes catalog tables for the full
],
target: "tables",
}));
expect(screen.getByText("2 selected")).toBeVisible();
await user.click(screen.getByRole("button", { name: "Clear" }));
expect(screen.queryByText("2 selected")).not.toBeInTheDocument();
await waitFor(() => expect(screen.queryByText("2 selected")).not.toBeInTheDocument());
await waitFor(() => expect(screen.getByRole("textbox", { name: "Search databases" })).toHaveFocus());
});
test("presents completed database synchronization steps as success and skips unneeded confirmation", async () => {
test("presents completed synchronization steps as success and skips unneeded confirmation", async () => {
const user = userEvent.setup();
const run = {
...makeSyncRun("all"),
counts: { tables: 163, columns: 2_275, relationships: 18 },
...makeSyncRun("columns"),
counts: { tables: 163, columns: 2_275 },
};
registerCompletedSyncRun(run);
server.use(http.post("/api/catalog/databases/:databaseId/sync-runs", () => (
@@ -1067,7 +887,7 @@ test("presents completed database synchronization steps as success and skips unn
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Synchronize all" }));
await user.click(await screen.findByRole("menuitem", { name: "Synchronize all columns" }));
expect(await screen.findByRole("heading", { name: "Succeeded" })).toBeVisible();
const steps = screen.getByRole("list", { name: "Synchronization steps" });
@@ -1084,7 +904,7 @@ test("presents completed database synchronization steps as success and skips unn
expect(steps.querySelectorAll(".text-primary")).toHaveLength(0);
});
test("database Overview exposes the database synchronization scopes", async () => {
test("database Overview exposes every synchronization scope", async () => {
const user = userEvent.setup();
let startBody: unknown;
const run = makeSyncRun("relationships");
@@ -1131,7 +951,7 @@ test("replaces the list with View and returns focus to the originating action",
expect(screen.getByRole("button", { name: "View Policlinico San Donato" })).toBeVisible();
});
test("configures the catalog for the selected workspace and locks that workspace", async () => {
test("opens an unconfigured row as Edit while creating its first saved configuration", async () => {
const user = userEvent.setup();
let postBody: unknown;
const saved = makeDatabase({
@@ -1151,16 +971,16 @@ test("configures the catalog for the selected workspace and locks that workspace
);
renderPage({ rows: () => rows });
await user.click(await screen.findByRole("button", { name: "Configure catalog for Research laboratory" }));
await user.click(await screen.findByRole("button", { name: "Edit Research laboratory" }));
expect(screen.getByRole("heading", { name: "Configure catalog" })).toBeVisible();
expect(screen.getByLabelText("Workspace")).toHaveAttribute("readonly");
expect(screen.getByRole("heading", { name: "Edit database" })).toBeVisible();
expect(screen.getByLabelText("Workspace")).toBeDisabled();
await user.selectOptions(screen.getByLabelText("Transport"), "rest_api");
await user.type(screen.getByLabelText("Base URL"), "https://psd.example/api");
expect(screen.getByLabelText("Diagnostic endpoint")).toHaveValue("/health");
expect(screen.getByLabelText("Diagnostic endpoint")).toHaveAttribute("readonly");
await user.click(screen.getByRole("button", { name: "Save catalog configuration" }));
await user.click(screen.getByRole("button", { name: "Save database" }));
await waitFor(() => expect(postBody).toMatchObject({
workspaceId: "lab",
@@ -1169,7 +989,7 @@ test("configures the catalog for the selected workspace and locks that workspace
expect(await screen.findByRole("heading", { name: "Edit database" })).toBeVisible();
});
test("has no global database creation path and scopes configuration to the chosen row", async () => {
test("global Add offers only unconfigured workspaces and lets the operator choose one", async () => {
const user = userEvent.setup();
const second = makeDatabase({
...unconfigured,
@@ -1179,13 +999,14 @@ test("has no global database creation path and scopes configuration to the chose
});
renderPage({ rows: [makeDatabase(), unconfigured, second] });
expect(screen.queryByRole("button", { name: /Add database/i })).not.toBeInTheDocument();
await user.click(await screen.findByRole("button", { name: "Configure catalog for Radiology" }));
await user.click(await screen.findByRole("button", { name: "Add database" }));
expect(screen.getByRole("heading", { name: "Configure catalog" })).toBeVisible();
const workspace = screen.getByLabelText("Workspace");
expect(workspace).toHaveAttribute("readonly");
expect(workspace).toHaveValue("Radiology");
expect(screen.getByRole("heading", { name: "Add database" })).toBeVisible();
expect(screen.getByRole("button", { name: "Add database" })).toBeVisible();
const selector = screen.getByLabelText("Workspace");
expect(selector).toBeEnabled();
expect(screen.queryByRole("option", { name: "Policlinico San Donato" })).not.toBeInTheDocument();
await user.selectOptions(selector, "radiology");
expect(screen.getByDisplayValue("radiology_dwh")).toBeVisible();
});
@@ -1243,8 +1064,7 @@ test("uses an in-page destructive form and returns the YAML workspace to Not con
await waitFor(() => expect(deleteCalled).toBe(true));
expect(deleteVersion).toBe("3");
expect(await screen.findByRole("button", { name: "Configure catalog for Policlinico San Donato" })).toBeEnabled();
expect(screen.queryByRole("button", { name: "Delete Policlinico San Donato" })).not.toBeInTheDocument();
expect(await screen.findByRole("button", { name: "Delete Policlinico San Donato" })).toBeDisabled();
expect(screen.getByText("Not configured")).toBeVisible();
});
@@ -1472,7 +1292,7 @@ test("filters catalog tables as the operator types", async () => {
});
});
test("selected table Actions expose table commands and send synchronization ids", async () => {
test("selected table Actions exposes cleanup commands and sends synchronization ids", async () => {
const user = userEvent.setup();
let startBody: unknown;
const run = makeSyncRun("columns", [patientsTable.id]);
@@ -1494,11 +1314,10 @@ test("selected table Actions expose table commands and send synchronization ids"
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
const synchronizeColumns = await screen.findByRole("menuitem", { name: "Synchronize columns for selected tables" });
const synchronizeColumns = await screen.findByRole("menuitem", { name: "Synchronize columns" });
expect(synchronizeColumns).toBeEnabled();
expect(screen.getByRole("menuitem", { name: "Synchronize database tables" })).toBeEnabled();
expect(screen.getByRole("menuitem", { name: "Remove synchronized columns" })).toBeEnabled();
expect(screen.queryByRole("menuitem", { name: "Delete all relationships" })).not.toBeInTheDocument();
expect(screen.getByRole("menuitem", { name: "Delete all columns" })).toBeEnabled();
expect(screen.getByRole("menuitem", { name: "Delete all relationships" })).toBeEnabled();
await user.click(synchronizeColumns);
await waitFor(() => expect(startBody).toEqual({
@@ -1547,10 +1366,10 @@ test("starts description generation for multiple selected tables", async () => {
await user.click(within(await screen.findByRole("row", { name: /visits/ })).getByRole("checkbox"));
await user.click(screen.getByRole("button", { name: "Actions" }));
const generate = await screen.findByRole("menuitem", { name: "Generate table description" });
const generate = await screen.findByRole("menuitem", { name: "Generate descriptions" });
expect(generate).toBeEnabled();
expect(screen.getByRole("menuitem", {
name: "Copy generated descriptions to Description",
name: "Move generated description to Description",
})).toBeEnabled();
await user.click(generate);
@@ -1566,37 +1385,6 @@ test("starts description generation for multiple selected tables", async () => {
expect(screen.getByText("2 selected")).toBeVisible();
});
test("generates descriptions for every column in selected tables", async () => {
const user = userEvent.setup();
const queuedRun = makeDescriptionGenerationRun({ scope: "selected_columns", total: 1 });
let startBody: unknown;
server.use(
http.get("/api/catalog/metadata-generation/models", () => HttpResponse.json({ models: [{ id: "local-qwen", label: "Local Qwen" }], default: "local-qwen" })),
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
http.get(`/api/catalog/databases/:databaseId/tables/${patientsTable.id}/columns`, () => HttpResponse.json([patientIdColumn])),
http.post("/api/catalog/databases/:databaseId/description-generation-runs", async ({ request }) => {
startBody = await request.json();
return HttpResponse.json(queuedRun, { status: 202 });
}),
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(queuedRun)),
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
);
renderPage({ rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })] });
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(within(await screen.findByRole("row", { name: /patients/ })).getByRole("checkbox"));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate descriptions for all columns" }));
await waitFor(() => expect(startBody).toEqual({
modelId: "local-qwen",
scope: "selected_columns",
targetIds: [patientIdColumn.id],
}));
expect(await screen.findByText("Column description generation started for 1 column")).toBeVisible();
});
test("observes an active run from another browser and reopens a terminal run from history", async () => {
const user = userEvent.setup();
const activeRun = makeDescriptionGenerationRun({
@@ -1629,9 +1417,9 @@ test("observes an active run from another browser and reopens a terminal run fro
renderPage();
const observe = await screen.findByRole("button", {
name: "View description generation history",
name: "Run descriptions generation history",
});
expect(observe).toHaveTextContent("View description generation history");
expect(observe).toHaveTextContent("Run descriptions generation history");
expect(requestedLimit).toBe("50");
await user.click(observe);
@@ -1655,9 +1443,9 @@ test("keeps the sensitive suggestion history label stable while showing active s
renderPage();
const historyButton = await screen.findByRole("button", {
name: "View sensitive suggestion history",
name: "Run sensitive suggestions history",
});
expect(historyButton).toHaveTextContent("View sensitive suggestion history");
expect(historyButton).toHaveTextContent("Run sensitive suggestions history");
await waitFor(() => expect(historyButton).toHaveAttribute(
"title",
"Sensitive suggestion generation is active",
@@ -1693,7 +1481,7 @@ test.each([
client.setQueryData(unrelatedColumnKey, []);
await user.click(await screen.findByRole("button", {
name: "View description generation history",
name: "Run descriptions generation history",
}));
await waitFor(() => expect(client.getQueryState(tableKey)?.isInvalidated).toBe(true));
@@ -1732,7 +1520,7 @@ test("keeps selected tables when description generation cannot start", async ()
await user.click(within(await screen.findByRole("row", { name: /patients/ })).getByRole("checkbox"));
await user.click(within(await screen.findByRole("row", { name: /visits/ })).getByRole("checkbox"));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate table description" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate descriptions" }));
expect(await screen.findByText("A description generation run is already active.")).toBeVisible();
expect(screen.getByText("2 selected")).toBeVisible();
@@ -1802,7 +1590,7 @@ test("refreshes Catalog Tables and Catalog Columns after table generation comple
await user.click(within(await screen.findByRole("row", { name: /patients/ })).getByRole("checkbox"));
await user.click(within(await screen.findByRole("row", { name: /visits/ })).getByRole("checkbox"));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate table description" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate descriptions" }));
expect(await screen.findByRole("heading", { name: "Completed" })).toBeVisible();
await waitFor(() => expect(tableReads).toBe(2));
@@ -1858,7 +1646,7 @@ test("moves selected generated table descriptions and reports copied and skipped
await user.click(within(visitsRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", {
name: "Copy generated descriptions to Description",
name: "Move generated description to Description",
}));
await waitFor(() => expect(consolidationBody).toEqual({
@@ -1946,9 +1734,8 @@ test("selects columns, moves generated descriptions, and refreshes only the affe
await user.click(within(idRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(nameRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Synchronize columns for this table" })).toBeEnabled();
await user.click(await screen.findByRole("menuitem", {
name: "Copy generated descriptions to Description",
name: "Move generated description to Description",
}));
await waitFor(() => expect(consolidationBody).toEqual({
@@ -1991,18 +1778,12 @@ test("starts one selected column with the configured default model", async () =>
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
expect(screen.queryByRole("button", { name: "Sync columns" })).not.toBeInTheDocument();
expect(screen.getByRole("columnheader", { name: /Select/i })).toBeVisible();
expect(await screen.findByRole("checkbox", { name: "Sensitive data for id" })).toHaveAttribute(
"title",
"Mark this column as sensitive; this does not select it for actions.",
);
const columnRow = (await screen.findAllByRole("row", { name: /Patient identifier/ }))
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
expect(columnRow).toBeDefined();
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate column description" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
await waitFor(() => expect(startBody).toEqual({
modelId: "local-qwen",
@@ -2356,7 +2137,7 @@ test("polls a description run and renders its events in sequence order", async (
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate column description" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
const drawer = await screen.findByRole("dialog", { name: "Description generation" });
expect(within(drawer).getByText(/local-qwen/)).toBeVisible();
@@ -2438,7 +2219,7 @@ test("refreshes Catalog Tables and Catalog Columns after column generation compl
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate column description" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
expect(await screen.findByRole("heading", { name: "Completed" })).toBeVisible();
await waitFor(() => expect(columnReads).toBe(2));
@@ -2471,7 +2252,7 @@ test("keeps the selected column and shows a safe message when generation cannot
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate column description" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
expect(await screen.findByText("A description generation run is already active.")).toBeVisible();
expect(screen.getByText("1 selected")).toBeVisible();
@@ -2538,7 +2319,7 @@ test("shows a basic failed run without exposing private model or provider fields
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate column description" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
const drawer = await screen.findByRole("dialog", { name: "Description generation" });
expect(await within(drawer).findByRole("heading", { name: "Failed" })).toBeVisible();
@@ -2592,18 +2373,18 @@ test("offers generation and consolidation for multiple selected columns", async
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(idRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate column description" })).toBeEnabled();
expect(await screen.findByRole("menuitem", { name: "Generate description" })).toBeEnabled();
expect(screen.getByRole("menuitem", {
name: "Copy generated descriptions to Description",
name: "Move generated description to Description",
})).toBeEnabled();
await user.keyboard("{Escape}");
await user.click(within(nameRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
const generate = await screen.findByRole("menuitem", { name: "Generate column descriptions" });
const generate = await screen.findByRole("menuitem", { name: "Generate descriptions" });
expect(generate).toBeEnabled();
expect(await screen.findByRole("menuitem", {
name: "Copy generated descriptions to Description",
name: "Move generated description to Description",
})).toBeEnabled();
await user.click(generate);
@@ -2639,7 +2420,7 @@ test.each([
const tableRow = await screen.findByRole("row", { name: /patients/ });
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate descriptions for all columns" }))
expect(await screen.findByRole("menuitem", { name: "Generate description" }))
.toHaveAttribute("aria-disabled", "true");
await user.keyboard("{Escape}");
await user.click(screen.getByRole("button", { name: "Clear" }));
@@ -2649,7 +2430,7 @@ test.each([
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate column description" }))
expect(await screen.findByRole("menuitem", { name: "Generate description" }))
.toHaveAttribute("aria-disabled", "true");
});
@@ -2682,11 +2463,11 @@ test("disables another generation start while the selected-column run is active"
.find((row) => within(row).queryByRole("checkbox", { name: /toggle row selection/i }));
await user.click(within(columnRow!).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate column description" }));
await user.click(await screen.findByRole("menuitem", { name: "Generate description" }));
expect(await screen.findByRole("dialog", { name: "Description generation" })).toBeVisible();
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate column description" }))
expect(await screen.findByRole("menuitem", { name: "Generate description" }))
.toHaveAttribute("aria-disabled", "true");
expect(screen.getByText("1 selected")).toBeVisible();
@@ -2694,7 +2475,7 @@ test("disables another generation start while the selected-column run is active"
const tableRow = await screen.findByRole("row", { name: /patients/ });
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: "Generate descriptions for all columns" }))
expect(await screen.findByRole("menuitem", { name: "Generate description" }))
.toHaveAttribute("aria-disabled", "true");
});
@@ -2722,7 +2503,7 @@ test("disables selected description consolidation without database.manage", asyn
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", {
name: "Copy generated descriptions to Description",
name: "Move generated description to Description",
})).toHaveAttribute("aria-disabled", "true");
expect(consolidationCalls).toBe(0);
});
@@ -2750,7 +2531,7 @@ test("shows an operation conflict without clearing selected descriptions or refr
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", {
name: "Copy generated descriptions to Description",
name: "Move generated description to Description",
}));
expect(await screen.findByText("A database operation is already in progress.")).toBeVisible();
@@ -2758,22 +2539,52 @@ test("shows an operation conflict without clearing selected descriptions or refr
expect(tableReads).toBe(1);
});
test("selected table Actions do not expose relationship cleanup", async () => {
test("selected table Actions confirms and deletes incoming and outgoing relationships", async () => {
const user = userEvent.setup();
let cleanupBody: unknown;
server.use(
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
http.post("/api/catalog/databases/:databaseId/tables/metadata-cleanup", async ({ request }) => {
cleanupBody = await request.json();
return HttpResponse.json({ tables: 0, columns: 0, relationships: 2 });
}),
);
renderPage({
const { client } = renderPage({
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
});
const unselectedColumnQuery = [
"catalog-columns",
patientsTable.databaseId,
"bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
] as const;
client.setQueryData(unselectedColumnQuery, []);
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
const tableRow = await screen.findByRole("row", { name: /patients/ });
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(screen.queryByRole("menuitem", { name: "Delete all relationships" })).not.toBeInTheDocument();
expect(screen.queryByRole("menuitem", { name: "Clear catalog relationships" })).not.toBeInTheDocument();
await user.click(await screen.findByRole("menuitem", { name: "Delete all relationships" }));
expect(screen.getByText("Delete all relationships for 1 table?")).toBeVisible();
expect(screen.getByText(/incoming and outgoing relationships/i)).toBeVisible();
expect(cleanupBody).toBeUndefined();
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
expect(screen.queryByText("Delete all relationships for 1 table?")).not.toBeInTheDocument();
expect(cleanupBody).toBeUndefined();
await user.click(within(tableRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Delete all relationships" }));
await user.click(screen.getByRole("button", { name: "Delete relationships" }));
await waitFor(() => expect(cleanupBody).toEqual({
tableIds: [patientsTable.id],
target: "relationships",
}));
await waitFor(() => expect(screen.queryByText("1 selected")).not.toBeInTheDocument());
expect(client.getQueryState(unselectedColumnQuery)?.isInvalidated).toBe(true);
await waitFor(() => expect(screen.getByRole("textbox", { name: "Search tables" })).toHaveFocus());
});
test("navigates purely from a database to its tables and edits review metadata", async () => {
@@ -2866,8 +2677,6 @@ test("navigates from a table to columns and from the database to physical relati
deferrable: false,
initiallyDeferred: false,
columns: [{ position: 1, sourceColumnId: "1", sourceColumnName: "patient_id", targetColumnId: idColumn.id, targetColumnName: "id" }],
origin: "physical",
status: "active",
lastSyncedDatabaseVersion: 3,
lastSyncedAt: "2026-08-27T10:00:00Z",
createdAt: "2026-08-27T10:00:00Z",
@@ -2946,10 +2755,9 @@ test("opens the durable job drawer and confirms its exact destructive plan", asy
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
});
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox", { name: /toggle row selection/i }));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Synchronize tables" }));
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "Sync tables" }));
const synchronizationDrawer = await screen.findByRole("dialog", {
name: "Schema synchronization",
+116 -160
View File
@@ -6,7 +6,7 @@ import {
useState,
} from "react";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { History, RefreshCw } from "lucide-react";
import { History, Plus, RefreshCw } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../components/ui/button";
import { ApiError, apiErrorMessage } from "../api/client";
@@ -80,20 +80,6 @@ const SYNC_STARTED_MESSAGES: Record<CatalogSyncScope, string> = {
relationships: "Relationship synchronization started",
all: "Full schema synchronization started",
};
const DATABASE_SCREEN_STORAGE_KEY = "thothii:database-management-screen";
function readPersistedDatabaseScreen(): DatabaseScreen {
if (import.meta.env.MODE === "test") return { kind: "list" };
try {
const parsed = JSON.parse(window.localStorage.getItem(DATABASE_SCREEN_STORAGE_KEY) ?? "null") as { kind?: unknown; workspaceId?: unknown } | null;
if ((parsed?.kind === "tables" || parsed?.kind === "relationships") && typeof parsed.workspaceId === "string") {
return { kind: parsed.kind, workspaceId: parsed.workspaceId };
}
} catch {
// Ignore malformed persisted navigation and return to the safe list view.
}
return { kind: "list" };
}
interface Props {
canManage: boolean;
@@ -170,13 +156,12 @@ export function DatabaseManagementPage({
(run) => run.status === "running",
);
const [screen, setScreen] = useState<DatabaseScreen>(readPersistedDatabaseScreen);
const [screen, setScreen] = useState<DatabaseScreen>({ kind: "list" });
const [selectedMetadataModel, setSelectedMetadataModel] = useState("");
const [draft, setDraft] = useState<DatabaseFormDraft | null>(null);
const [baseline, setBaseline] = useState("");
const [formSource, setFormSource] = useState<FormSource | null>(null);
const [search, setSearch] = useState("");
const [selectedDatabaseId, setSelectedDatabaseId] = useState<string | null>(null);
const [busyAction, setBusyAction] = useState<DatabaseBusyAction>(null);
const [stale, setStale] = useState(false);
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
@@ -186,11 +171,6 @@ export function DatabaseManagementPage({
busy: false,
});
const [tableNestedNavigationActive, setTableNestedNavigationActive] = useState(false);
const [tableNestedNavigationBack, setTableNestedNavigationBack] = useState<(() => void) | null>(null);
const handleTableNestedNavigationChange = useCallback((active: boolean, onBack?: () => void) => {
setTableNestedNavigationActive(active);
setTableNestedNavigationBack(() => (active ? onBack ?? null : null));
}, []);
const [activeSyncRun, setActiveSyncRun] = useState<CatalogSyncRun | null>(null);
const [syncHistoryDatabaseId, setSyncHistoryDatabaseId] = useState<string | null>(null);
const [syncDrawerOpen, setSyncDrawerOpen] = useState(false);
@@ -204,12 +184,6 @@ export function DatabaseManagementPage({
suggestions: SensitiveDataSuggestion[];
} | null>(null);
useEffect(() => {
if (import.meta.env.MODE === "test") return;
const persisted = screen.kind === "tables" || screen.kind === "relationships" ? screen : { kind: "list" };
window.localStorage.setItem(DATABASE_SCREEN_STORAGE_KEY, JSON.stringify(persisted));
}, [screen]);
const originRef = useRef<HTMLElement | null>(null);
const searchInputRef = useRef<HTMLInputElement>(null);
const formHeadingRef = useRef<HTMLHeadingElement>(null);
@@ -235,7 +209,11 @@ export function DatabaseManagementPage({
() => queryClient.invalidateQueries({ queryKey: ["catalog-metrics"] }),
[queryClient],
);
const editable = screen.kind === "configure" || screen.kind === "edit";
const availableWorkspaces = useMemo(
() => rows.filter((row) => row.workspaceAvailable && !row.configured),
[rows],
);
const editable = screen.kind === "add" || screen.kind === "edit";
const configurationDirty = Boolean(
editable
&& draft
@@ -243,9 +221,8 @@ export function DatabaseManagementPage({
);
const dirty = Boolean(editable && draft && (configurationDirty || hasSecretChanges(draft)));
const busy = busyAction !== null;
const catalogChildVisible = screen.kind === "tables" || screen.kind === "relationships";
const navigationDirty = catalogChildVisible ? tablesNavigationState.dirty : dirty;
const navigationBusy = catalogChildVisible ? tablesNavigationState.busy : busy;
const navigationDirty = screen.kind === "tables" ? tablesNavigationState.dirty : dirty;
const navigationBusy = screen.kind === "tables" ? tablesNavigationState.busy : busy;
useEffect(() => {
onNavigationStateChange?.({ dirty: navigationDirty, busy: navigationBusy });
@@ -286,14 +263,9 @@ export function DatabaseManagementPage({
}, [queryClient]);
const restoreListFocus = useCallback(() => {
const originLabel = originRef.current?.getAttribute("aria-label");
window.setTimeout(() => {
if (originRef.current?.isConnected) {
originRef.current.focus();
} else if (originLabel) {
const matchingAction = [...document.querySelectorAll<HTMLButtonElement>("button")]
.find((button) => button.getAttribute("aria-label") === originLabel);
(matchingAction ?? searchInputRef.current)?.focus();
} else {
searchInputRef.current?.focus();
}
@@ -310,7 +282,6 @@ export function DatabaseManagementPage({
setPartialSecretFailure(null);
setTablesNavigationState({ dirty: false, busy: false });
setTableNestedNavigationActive(false);
setTableNestedNavigationBack(null);
restoreListFocus();
}, [restoreListFocus]);
@@ -336,6 +307,7 @@ export function DatabaseManagementPage({
mode: DatabaseFormMode,
row: CatalogDatabase,
origin: HTMLElement,
workspaceLocked = false,
) => {
const nextDraft = draftFrom(row);
originRef.current = origin;
@@ -348,6 +320,7 @@ export function DatabaseManagementPage({
setScreen({
kind: mode,
workspaceId: row.workspaceId,
...(mode === "add" ? { workspaceLocked } : {}),
});
}, []);
@@ -356,7 +329,7 @@ export function DatabaseManagementPage({
}, [openForm]);
const editRow = useCallback((row: CatalogDatabase, origin: HTMLButtonElement) => {
openForm(row.configured ? "edit" : "configure", row, origin);
openForm(row.configured ? "edit" : "add", row, origin, !row.configured);
}, [openForm]);
const deleteRow = useCallback((row: CatalogDatabase, origin: HTMLButtonElement) => {
@@ -373,21 +346,17 @@ export function DatabaseManagementPage({
setPartialSecretFailure(null);
setTablesNavigationState({ dirty: false, busy: false });
setTableNestedNavigationActive(false);
setTableNestedNavigationBack(null);
setScreen({ kind: "tables", workspaceId: row.workspaceId });
}, []);
const openRelationships = useCallback((row: CatalogDatabase, origin?: HTMLElement) => {
const openRelationships = useCallback((row: CatalogDatabase) => {
if (!row.configured || !row.id) return;
if (origin) originRef.current = origin;
setDraft(null);
setBaseline("");
setFormSource(null);
setStale(false);
setPartialSecretFailure(null);
setTablesNavigationState({ dirty: false, busy: false });
setTableNestedNavigationActive(false);
setTableNestedNavigationBack(null);
setScreen({ kind: "relationships", workspaceId: row.workspaceId });
}, []);
@@ -403,6 +372,25 @@ export function DatabaseManagementPage({
setScreen({ kind: "view", workspaceId: row.workspaceId });
}, []);
const addDatabase = useCallback((origin: HTMLElement) => {
const workspace = availableWorkspaces[0];
if (!workspace || !canManage) return;
openForm("add", workspace, origin, false);
}, [availableWorkspaces, canManage, openForm]);
const changeWorkspace = useCallback((workspaceId: string) => {
const workspace = availableWorkspaces.find((row) => row.workspaceId === workspaceId);
if (!workspace) return;
if (dirty && !window.confirm("Discard changes and choose another workspace?")) return;
const nextDraft = draftFrom(workspace);
setDraft(nextDraft);
setBaseline(configurationFingerprint(nextDraft));
setFormSource({ id: workspace.id, version: workspace.version });
setStale(false);
setPartialSecretFailure(null);
setScreen({ kind: "add", workspaceId, workspaceLocked: false });
}, [availableWorkspaces, dirty]);
const changeField = useCallback((field: "databaseName" | "schema", value: string) => {
setDraft((current) => current ? { ...current, [field]: value } : current);
}, []);
@@ -636,7 +624,7 @@ export function DatabaseManagementPage({
showList();
toast.info("This database configuration has already been deleted");
} else if (screen.kind === "edit" && !latest.configured) {
setScreen({ kind: "configure", workspaceId: latest.workspaceId });
setScreen({ kind: "add", workspaceId: latest.workspaceId, workspaceLocked: true });
}
} catch (error) {
toast.error(apiErrorMessage(error));
@@ -681,40 +669,22 @@ export function DatabaseManagementPage({
}, [queryClient]);
const openDescriptionGenerationHistory = useCallback(() => {
const scopedRuns = activeRow ? descriptionGenerationRuns.filter((item) => item.databaseId === activeRow.id) : descriptionGenerationRuns;
const scopedActiveRun = observedActiveDescriptionGenerationRun
&& (!activeRow || observedActiveDescriptionGenerationRun.databaseId === activeRow.id)
? observedActiveDescriptionGenerationRun
: undefined;
const scopedSelectedRun = activeDescriptionGenerationRun
&& (!activeRow || activeDescriptionGenerationRun.databaseId === activeRow.id)
? activeDescriptionGenerationRun
: undefined;
const run = scopedActiveRun
?? scopedRuns[0]
?? scopedSelectedRun;
const run = observedActiveDescriptionGenerationRun
?? descriptionGenerationRuns[0]
?? activeDescriptionGenerationRun;
if (run) setActiveDescriptionGenerationRun(run);
setSensitiveDataSuggestionHistoryDrawerOpen(false);
setDescriptionGenerationDrawerOpen(true);
}, [activeDescriptionGenerationRun, activeRow, descriptionGenerationRuns, observedActiveDescriptionGenerationRun]);
}, [activeDescriptionGenerationRun, descriptionGenerationRuns, observedActiveDescriptionGenerationRun]);
const openSensitiveDataSuggestionHistory = useCallback(() => {
const scopedRuns = activeRow ? sensitiveDataSuggestionRuns.filter((item) => item.databaseId === activeRow.id) : sensitiveDataSuggestionRuns;
const scopedActiveRun = observedActiveSensitiveDataSuggestionRun
&& (!activeRow || observedActiveSensitiveDataSuggestionRun.databaseId === activeRow.id)
? observedActiveSensitiveDataSuggestionRun
: undefined;
const scopedSelectedRun = activeSensitiveDataSuggestionRun
&& (!activeRow || activeSensitiveDataSuggestionRun.databaseId === activeRow.id)
? activeSensitiveDataSuggestionRun
: undefined;
const run = scopedActiveRun
?? scopedRuns[0]
?? scopedSelectedRun;
const run = observedActiveSensitiveDataSuggestionRun
?? sensitiveDataSuggestionRuns[0]
?? activeSensitiveDataSuggestionRun;
if (run) setActiveSensitiveDataSuggestionRun(run);
setDescriptionGenerationDrawerOpen(false);
setSensitiveDataSuggestionHistoryDrawerOpen(true);
}, [activeRow, activeSensitiveDataSuggestionRun, observedActiveSensitiveDataSuggestionRun, sensitiveDataSuggestionRuns]);
}, [activeSensitiveDataSuggestionRun, observedActiveSensitiveDataSuggestionRun, sensitiveDataSuggestionRuns]);
const descriptionGenerationTerminated = useCallback(async (run: DescriptionGenerationRun) => {
await Promise.all([
@@ -779,7 +749,7 @@ export function DatabaseManagementPage({
scope,
);
rememberDescriptionGenerationRun(run);
toast.success(`${scope === "all" ? "Generate all descriptions" : "Generate missing descriptions"} started for ${database.workspaceName}`);
toast.success(`${scope === "all" ? "Generate All" : "Generate Missing"} started for ${database.workspaceName}`);
} catch (error) {
toast.error(apiErrorMessage(error));
throw error;
@@ -886,11 +856,6 @@ export function DatabaseManagementPage({
}
}, [invalidateCatalogMetrics, queryClient]);
const clearActiveCatalogTables = useCallback(async () => {
if (!activeRow?.id) return;
await deleteSelectedMetadata([activeRow], "tables");
}, [activeRow, deleteSelectedMetadata]);
const syncDatabase = useCallback(async (scope: CatalogSyncScope) => {
if (!activeRow?.id || !activeRow.configured) return;
try {
@@ -937,7 +902,6 @@ export function DatabaseManagementPage({
isLoading={metadataModelsQuery.isLoading}
selectedModel={selectedMetadataModel}
onSelectedModelChange={setSelectedMetadataModel}
variant="compact"
/>
);
@@ -979,7 +943,6 @@ export function DatabaseManagementPage({
/>
<DescriptionGenerationDrawer
open={descriptionGenerationDrawerOpen}
databaseId={activeRow?.id ?? null}
run={activeDescriptionGenerationRun}
modelLabel={metadataModels.find(
(model) => model.id === activeDescriptionGenerationRun?.modelId,
@@ -990,7 +953,6 @@ export function DatabaseManagementPage({
/>
<SensitiveDataSuggestionHistoryDrawer
open={sensitiveDataSuggestionHistoryDrawerOpen}
databaseId={activeRow?.id ?? null}
run={activeSensitiveDataSuggestionRun}
modelLabel={metadataModels.find(
(model) => model.id === activeSensitiveDataSuggestionRun?.modelId,
@@ -1018,6 +980,7 @@ export function DatabaseManagementPage({
eyebrow="Catalog configuration"
title={databaseFormTitle(
fleetFormMode,
fleetFormMode === "add" && Boolean(screen.kind === "add" && screen.workspaceLocked),
"drawer",
)}
description={`${activeRow.workspaceName} · ${activeRow.workspaceId}`}
@@ -1030,6 +993,8 @@ export function DatabaseManagementPage({
presentation="drawer"
mode={fleetFormMode}
row={activeRow}
availableWorkspaces={availableWorkspaces}
workspaceLocked={fleetFormMode === "add" && Boolean(screen.kind === "add" && screen.workspaceLocked)}
draft={draft}
dirty={dirty}
canManage={canManage}
@@ -1040,6 +1005,7 @@ export function DatabaseManagementPage({
partialSecretFailure={partialSecretFailure}
headingRef={formHeadingRef}
onBack={backToList}
onWorkspaceChange={changeWorkspace}
onFieldChange={changeField}
onTransportChange={changeTransport}
onBindingChange={changeBinding}
@@ -1054,8 +1020,6 @@ export function DatabaseManagementPage({
onOpenRelationships={() => openRelationships(activeRow)}
onSync={(scope) => void syncDatabase(scope)}
onOpenSync={() => openSync(activeRow)}
onOpenDescriptionHistory={openDescriptionGenerationHistory}
onOpenSensitiveHistory={openSensitiveDataSuggestionHistory}
activeSyncRun={currentActiveRun}
/>
</FleetLedgerDrawer>
@@ -1073,13 +1037,13 @@ export function DatabaseManagementPage({
onOpenOverview={() => openOverview(activeRow)}
onOpenRelationships={() => openRelationships(activeRow)}
onNavigationStateChange={setTablesNavigationState}
onNestedNavigationChange={handleTableNestedNavigationChange}
onNestedNavigationChange={setTableNestedNavigationActive}
onRunStarted={rememberSyncRun}
onOpenSync={() => openSync(activeRow)}
onDescriptionGenerationRunStarted={rememberDescriptionGenerationRun}
onSuggestSensitive={suggestActiveDatabaseSensitiveFields}
onClearCatalogTables={clearActiveCatalogTables}
onCatalogMetricsChanged={invalidateCatalogMetrics}
metadataModelControl={fleetMetadataModelControl}
/>
) : screen.kind === "relationships" && relationshipsVisible ? (
<DatabaseRelationships
@@ -1091,10 +1055,6 @@ export function DatabaseManagementPage({
onOpenTables={() => openTables(activeRow)}
onRunStarted={rememberSyncRun}
onOpenSync={() => openSync(activeRow)}
onOpenDescriptionHistory={openDescriptionGenerationHistory}
onOpenSensitiveHistory={openSensitiveDataSuggestionHistory}
onCatalogMetricsChanged={invalidateCatalogMetrics}
onNavigationStateChange={setTablesNavigationState}
/>
) : isError && rows.length === 0 ? (
<div className="grid h-full place-items-center p-6">
@@ -1118,19 +1078,17 @@ export function DatabaseManagementPage({
onSearchChange={setSearch}
onView={viewRow}
onOpenTables={(row, origin) => openTables(row, origin)}
onOpenRelationships={(row, origin) => openRelationships(row, origin)}
onEdit={editRow}
onDelete={deleteRow}
onOpenSync={openSync}
onSelectionChange={(selected) => setSelectedDatabaseId(selected.length === 1 ? selected[0].id ?? null : null)}
onTestSelected={testSelected}
onSyncSelected={syncSelected}
selectedMetadataModel={selectedMetadataModelAvailable ? selectedMetadataModel : null}
descriptionGenerationActive={descriptionGenerationActive}
onGenerateDescriptions={generateDatabaseDescriptions}
onSuggestSensitive={suggestDatabaseSensitiveFields}
sensitiveDataSuggestionRuns={sensitiveDataSuggestionRuns}
onDeleteMetadataSelected={deleteSelectedMetadata}
metadataModelControl={fleetMetadataModelControl}
onRefresh={refreshList}
/>
);
@@ -1160,46 +1118,28 @@ export function DatabaseManagementPage({
)}
actions={(
<>
{fleetMetadataModelControl}
<Button
type="button"
variant="outline"
size="sm"
className="whitespace-nowrap"
aria-label="View schema synchronization logs"
title="View schema synchronization progress and history"
disabled={!canManage || (!selectedDatabaseId && !activeSyncRun?.databaseId)}
onClick={() => openSync(selectedDatabaseId ? rows.find((row) => row.id === selectedDatabaseId) : undefined)}
>
<RefreshCw aria-hidden="true" />
{currentActiveRun ? "Schema sync active" : "Schema sync logs"}
<Button type="button" variant="outline" size="sm" className="thot-fleet-nav-action thot-fleet-nav-action--below" data-tooltip="Description history" data-fleet-action="secondary" onClick={openDescriptionGenerationHistory}>
<History aria-hidden="true" /> Description history
{observedActiveDescriptionGenerationRun ? <span className="size-1.5 rounded-full bg-warning" aria-hidden="true" /> : null}
</Button>
<Button
type="button"
variant="outline"
size="sm"
className="whitespace-nowrap"
aria-label="View AI description generation logs"
title="View AI description generation progress and history"
disabled={!canManage}
onClick={openDescriptionGenerationHistory}
>
<History aria-hidden="true" />
{descriptionGenerationActive ? "AI descriptions active" : "AI description logs"}
</Button>
<Button
type="button"
variant="outline"
size="sm"
className="whitespace-nowrap"
aria-label="View sensitive analysis progress"
title="View sensitive analysis progress and history"
disabled={!canManage}
onClick={openSensitiveDataSuggestionHistory}
>
<History aria-hidden="true" />
{observedActiveSensitiveDataSuggestionRun ? "Sensitive analysis active" : "Sensitive analysis history"}
<Button type="button" variant="outline" size="sm" className="thot-fleet-nav-action thot-fleet-nav-action--below" data-tooltip="Sensitive history" data-fleet-action="secondary" onClick={openSensitiveDataSuggestionHistory}>
<History aria-hidden="true" /> Sensitive history
{observedActiveSensitiveDataSuggestionRun ? <span className="size-1.5 rounded-full bg-warning" aria-hidden="true" /> : null}
</Button>
{(screen.kind === "list" || formVisible) ? (
<Button
type="button"
size="sm"
className="thot-fleet-nav-action thot-fleet-nav-action--below"
data-tooltip="Add database"
data-fleet-action="primary"
disabled={!canManage || availableWorkspaces.length === 0}
title={availableWorkspaces.length === 0 ? "Every available workspace is already configured" : undefined}
onClick={(event) => addDatabase(event.currentTarget)}
>
<Plus aria-hidden="true" /> Add database
</Button>
) : null}
</>
)}
/>
@@ -1219,17 +1159,9 @@ export function DatabaseManagementPage({
navigation={(
<FleetLedgerBreadcrumb
items={fleetBreadcrumb}
back={screen.kind === "relationships"
back={screen.kind === "relationships" || (screen.kind === "tables" && !tableNestedNavigationActive)
? { label: "Back to databases", onBack: showList, disabled: navigationBusy }
: screen.kind === "tables" && tableNestedNavigationActive
? {
label: "Back to tables",
onBack: () => tableNestedNavigationBack?.(),
disabled: navigationBusy || !tableNestedNavigationBack,
}
: screen.kind === "tables"
? { label: "Back to databases", onBack: showList, disabled: navigationBusy }
: undefined}
: undefined}
/>
)}
children={<FleetLedgerContent busy={isLoading || isFetching}>{fleetContent}</FleetLedgerContent>}
@@ -1273,16 +1205,40 @@ export function DatabaseManagementPage({
selectedModel={selectedMetadataModel}
onSelectedModelChange={setSelectedMetadataModel}
/>
{screen.kind === "list" ? <>
<Button type="button" variant="outline" className="whitespace-nowrap disabled:opacity-70" aria-label="View description generation history" disabled={!canManage} title="View description generation history" onClick={openDescriptionGenerationHistory}>
<History /> View description generation history
</Button>
<Button type="button" variant="outline" className="whitespace-nowrap disabled:opacity-70" aria-label="View sensitive suggestion history" disabled={!canManage} title={observedActiveSensitiveDataSuggestionRun ? "Sensitive suggestion generation is active" : "View sensitive suggestion history"} onClick={openSensitiveDataSuggestionHistory}>
<History />
{observedActiveSensitiveDataSuggestionRun ? <span aria-hidden="true" className="size-2 rounded-full bg-primary" /> : null}
View sensitive suggestion history
</Button>
</> : null}
<Button
type="button"
variant="outline"
className="whitespace-nowrap disabled:opacity-70"
aria-label="Run descriptions generation history"
disabled={!canManage}
title={observedActiveDescriptionGenerationRun
? "Description generation is active"
: "View description generation history"}
onClick={openDescriptionGenerationHistory}
>
<History />
Run descriptions generation history
{observedActiveDescriptionGenerationRun ? (
<span aria-hidden="true" className="size-1.5 rounded-full bg-primary" />
) : null}
</Button>
<Button
type="button"
variant="outline"
className="whitespace-nowrap disabled:opacity-70"
aria-label="Run sensitive suggestions history"
disabled={!canManage}
title={observedActiveSensitiveDataSuggestionRun
? "Sensitive suggestion generation is active"
: "View sensitive suggestion history"}
onClick={openSensitiveDataSuggestionHistory}
>
<History />
Run sensitive suggestions history
{observedActiveSensitiveDataSuggestionRun ? (
<span aria-hidden="true" className="size-1.5 rounded-full bg-primary" />
) : null}
</Button>
</div>
{screen.kind === "list" ? (
<div
@@ -1293,6 +1249,14 @@ export function DatabaseManagementPage({
<Button type="button" variant="outline" disabled={isFetching} onClick={() => void refreshList()}>
<RefreshCw className={isFetching ? "animate-spin" : ""} /> Refresh
</Button>
<Button
type="button"
disabled={!canManage || availableWorkspaces.length === 0}
title={availableWorkspaces.length === 0 ? "Every available workspace is already configured" : undefined}
onClick={(event) => addDatabase(event.currentTarget)}
>
<Plus /> Add database
</Button>
</div>
) : null}
</div>
@@ -1329,12 +1293,10 @@ export function DatabaseManagementPage({
onEdit={editRow}
onDelete={deleteRow}
onOpenSync={openSync}
onSelectionChange={(selected) => setSelectedDatabaseId(selected.length === 1 ? selected[0].id ?? null : null)}
onTestSelected={testSelected}
onSyncSelected={syncSelected}
selectedMetadataModel={selectedMetadataModelAvailable ? selectedMetadataModel : null}
descriptionGenerationActive={descriptionGenerationActive}
sensitiveDataSuggestionRuns={sensitiveDataSuggestionRuns}
onGenerateDescriptions={generateDatabaseDescriptions}
onSuggestSensitive={suggestDatabaseSensitiveFields}
onDeleteMetadataSelected={deleteSelectedMetadata}
@@ -1346,6 +1308,8 @@ export function DatabaseManagementPage({
<DatabaseForm
mode={screen.kind as DatabaseFormMode}
row={activeRow}
availableWorkspaces={availableWorkspaces}
workspaceLocked={screen.kind === "add" && Boolean(screen.workspaceLocked)}
draft={draft}
dirty={dirty}
canManage={canManage}
@@ -1356,6 +1320,7 @@ export function DatabaseManagementPage({
partialSecretFailure={partialSecretFailure}
headingRef={formHeadingRef}
onBack={backToList}
onWorkspaceChange={changeWorkspace}
onFieldChange={changeField}
onTransportChange={changeTransport}
onBindingChange={changeBinding}
@@ -1370,8 +1335,6 @@ export function DatabaseManagementPage({
onOpenRelationships={() => openRelationships(activeRow)}
onSync={(scope) => void syncDatabase(scope)}
onOpenSync={() => openSync(activeRow)}
onOpenDescriptionHistory={openDescriptionGenerationHistory}
onOpenSensitiveHistory={openSensitiveDataSuggestionHistory}
activeSyncRun={currentActiveRun}
/>
) : null}
@@ -1391,7 +1354,6 @@ export function DatabaseManagementPage({
onOpenSync={() => openSync(activeRow)}
onDescriptionGenerationRunStarted={rememberDescriptionGenerationRun}
onSuggestSensitive={suggestActiveDatabaseSensitiveFields}
onClearCatalogTables={clearActiveCatalogTables}
onCatalogMetricsChanged={invalidateCatalogMetrics}
/>
) : null}
@@ -1405,10 +1367,6 @@ export function DatabaseManagementPage({
onOpenTables={() => openTables(activeRow)}
onRunStarted={rememberSyncRun}
onOpenSync={() => openSync(activeRow)}
onOpenDescriptionHistory={openDescriptionGenerationHistory}
onOpenSensitiveHistory={openSensitiveDataSuggestionHistory}
onCatalogMetricsChanged={invalidateCatalogMetrics}
onNavigationStateChange={setTablesNavigationState}
/>
) : null}
</div>
@@ -1423,7 +1381,6 @@ export function DatabaseManagementPage({
/>
<DescriptionGenerationDrawer
open={descriptionGenerationDrawerOpen}
databaseId={activeRow?.id ?? null}
run={activeDescriptionGenerationRun}
modelLabel={metadataModels.find(
(model) => model.id === activeDescriptionGenerationRun?.modelId,
@@ -1434,7 +1391,6 @@ export function DatabaseManagementPage({
/>
<SensitiveDataSuggestionHistoryDrawer
open={sensitiveDataSuggestionHistoryDrawerOpen}
databaseId={activeRow?.id ?? null}
run={activeSensitiveDataSuggestionRun}
modelLabel={metadataModels.find(
(model) => model.id === activeSensitiveDataSuggestionRun?.modelId,
+1 -1
View File
@@ -356,7 +356,7 @@ test("shows an explicit recoverable incomplete state when no provider model is a
renderManagement();
const incomplete = await screen.findByRole("alert", { name: "Pi configuration incomplete" });
expect(incomplete).toHaveTextContent("No provider or model options are available");
expect(incomplete).toHaveTextContent("No enabled provider and model choices are available");
expect(incomplete).toHaveTextContent("Check the host-managed Pi model configuration");
expect(screen.queryByText("Loading Pi management…")).not.toBeInTheDocument();
expect(screen.getByRole("button", { name: "Save defaults" })).toBeDisabled();
+2 -2
View File
@@ -78,7 +78,7 @@ function ReadinessRail({ ready, configured, credentials, smokeState }: {
state={credentials === "present" ? "ready" : "attention"}
/>
<RailItem
label="Configuration test"
label="Saved-config test"
value={smokeState === "passed" ? "Saved configuration test passed" : smokeState === "failed" ? "Saved configuration test failed" : "Saved configuration test not run"}
state={smokeState === "passed" ? "ready" : smokeState === "failed" ? "attention" : "idle"}
/>
@@ -505,7 +505,7 @@ export function PiManagement({ open, onClose }: { open: boolean; onClose: () =>
) : (
<div className="mt-4">
<div role="alert" aria-label="Pi configuration incomplete" className="rounded-md border border-amber-500/30 bg-amber-500/10 p-4 text-sm text-amber-900 dark:text-amber-200">
<p className="font-semibold">No provider or model options are available.</p>
<p className="font-semibold">No enabled provider and model choices are available.</p>
<p className="mt-1">Check the host-managed Pi model configuration, then retry this panel.</p>
</div>
<div className="mt-4 flex flex-wrap gap-2">
+1 -1
View File
@@ -296,7 +296,7 @@ export function ComposerFooter() {
</FooterSelect>
<span
className="whitespace-nowrap font-mono text-[0.7rem] tabular-nums text-muted-foreground"
aria-label="Input, cached, and output tokens"
aria-label="Input non-cached, cached, and output tokens"
>
{formatTokenUsage(tokenUsage)}
</span>
@@ -249,7 +249,6 @@ export function CatalogSyncDrawer({
key={item.id}
type="button"
aria-current={item.id === run?.id ? "true" : undefined}
data-status={item.state}
className="flex w-full items-center justify-between gap-3 px-3 py-2 text-left text-sm hover:bg-muted/50 aria-[current=true]:bg-primary/8"
onClick={() => onRunChange(item)}
>
@@ -285,12 +284,7 @@ export function CatalogSyncDrawer({
open
ariaLabel="Schema synchronization"
eyebrow="Schema synchronization"
title={run ? (
<span className="inline-flex items-center gap-2">
{!terminal(run) ? <LoaderCircle className="size-5 animate-spin" aria-hidden="true" /> : null}
{stateLabel(run)[0].toUpperCase()}{stateLabel(run).slice(1)}
</span>
) : "Synchronization history"}
title={run ? `${stateLabel(run)[0].toUpperCase()}${stateLabel(run).slice(1)}` : "Synchronization history"}
description={run ? `${run.scope} · ${elapsed(run, now)}` : undefined}
onClose={onClose}
closeLabel="Close synchronization drawer"
@@ -1,4 +1,4 @@
import { useEffect, useMemo, useRef, useState } from "react";
import { useEffect, useMemo, useRef, useState, type ReactNode } from "react";
import { Menu } from "@base-ui/react/menu";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { AgGridReact } from "ag-grid-react";
@@ -10,11 +10,9 @@ import { ApiError, apiErrorMessage } from "../../api/client";
import {
consolidateCatalogDescriptions,
listCatalogColumns,
startCatalogSync,
startDescriptionGenerationRun,
updateCatalogColumnMetadata,
type CatalogColumn,
type CatalogSyncRun,
type CatalogTable,
type DescriptionGenerationRun,
type SensitiveDataSuggestionRequest,
@@ -22,21 +20,21 @@ import {
import type { DatabaseNavigationState } from "./model";
import { FleetActionSelector, type FleetActionOption } from "./FleetActionSelector";
import { FleetLedgerDrawer } from "./FleetLedgerShell";
import { NO_METADATA_GENERATION_LLM_MODEL_MESSAGE } from "./MetadataGenerationModelSelector";
interface Props {
databaseId: string;
table: CatalogTable;
databaseVersion: number;
canManage: boolean;
selectedMetadataModel: string | null;
descriptionGenerationActive: boolean;
onDescriptionGenerationRunStarted: (run: DescriptionGenerationRun) => void;
onCatalogSyncRunStarted?: (run: CatalogSyncRun) => void;
onNavigationStateChange: (state: DatabaseNavigationState) => void;
onSync: () => void;
onSuggestSensitive: (selection: SensitiveDataSuggestionRequest, scopeLabel: string) => Promise<void>;
bindingReady?: boolean;
catalogOperationActive?: boolean;
onCatalogMetricsChanged?: () => void | Promise<void>;
metadataModelControl?: ReactNode;
presentation?: "legacy" | "fleet";
}
@@ -81,7 +79,6 @@ function SensitiveCell({ data, context }: ICellRendererParams<CatalogColumn, unk
checked={data.sensitive}
disabled={!context.canManage || context.busy}
aria-label={`Sensitive data for ${data.name}`}
title="Mark this column as sensitive; this does not select it for actions."
onChange={(event) => context.onSensitiveChange(data, event.target.checked)}
onClick={(event) => event.stopPropagation()}
/>
@@ -92,16 +89,17 @@ function SensitiveCell({ data, context }: ICellRendererParams<CatalogColumn, unk
export function DatabaseColumns({
databaseId,
table,
databaseVersion,
canManage,
selectedMetadataModel,
descriptionGenerationActive,
onDescriptionGenerationRunStarted,
onCatalogSyncRunStarted,
onNavigationStateChange,
onSync,
onSuggestSensitive,
bindingReady = true,
catalogOperationActive = false,
onCatalogMetricsChanged,
metadataModelControl,
presentation = "legacy",
}: Props) {
const queryClient = useQueryClient();
@@ -291,33 +289,16 @@ export function DatabaseColumns({
};
type FleetColumnAction =
| "sync-columns"
| "generate-descriptions"
| "consolidate-descriptions"
| "suggest-sensitive"
| "sync-columns"
| "save-sensitive";
const fleetActions: readonly FleetActionOption<FleetColumnAction>[] = [
{
id: "sync-columns",
label: "Synchronize columns for this table",
group: "Synchronization",
scopeLabel: "Selected table",
runLabel: "Synchronize",
disabled: !canManage || selectedIds.length === 0 || catalogOperationActive || busy,
disabledReason: !canManage
? "You do not have permission to synchronize the catalog."
: selectedIds.length === 0
? "Select at least one column."
: catalogOperationActive
? "Wait for the active catalog operation to finish."
: busy
? "Another action is running."
: undefined,
},
{
id: "generate-descriptions",
label: "Generate column descriptions",
label: "Generate descriptions",
group: "Descriptions",
runLabel: "Generate",
disabled: !canManage || selectedIds.length === 0 || !selectedMetadataModel || descriptionGenerationActive || catalogOperationActive || busy,
@@ -326,7 +307,7 @@ export function DatabaseColumns({
: selectedIds.length === 0
? "Select at least one column."
: !selectedMetadataModel
? NO_METADATA_GENERATION_LLM_MODEL_MESSAGE
? "Choose a metadata model first."
: descriptionGenerationActive || catalogOperationActive
? "Wait for the active catalog operation to finish."
: busy
@@ -335,7 +316,7 @@ export function DatabaseColumns({
},
{
id: "consolidate-descriptions",
label: "Copy generated descriptions to Description",
label: "Move generated to Description",
group: "Descriptions",
runLabel: "Move",
disabled: !canManage || selectedIds.length === 0 || catalogOperationActive || busy,
@@ -360,23 +341,40 @@ export function DatabaseColumns({
: selectedIds.length === 0
? "Select at least one column."
: !selectedMetadataModel
? NO_METADATA_GENERATION_LLM_MODEL_MESSAGE
? "Choose a metadata model first."
: descriptionGenerationActive || catalogOperationActive
? "Wait for the active catalog operation to finish."
: busy
? "Another action is running."
: undefined,
},
{
id: "sync-columns",
label: "Synchronize columns",
group: "Synchronization",
scopeLabel: "All columns in this table",
runLabel: "Synchronize",
disabled: !canManage || !bindingReady || catalogOperationActive || busy,
disabledReason: !canManage
? "You do not have permission to synchronize the catalog."
: !bindingReady
? "Test the current database binding first."
: catalogOperationActive
? "Wait for the active catalog operation to finish."
: busy
? "Another action is running."
: undefined,
},
{
id: "save-sensitive",
label: "Save sensitive field changes",
label: "Save sensitive-field changes",
group: "Sensitive data",
runLabel: "Save",
disabled: !canManage || changedSensitiveColumns.length === 0 || catalogOperationActive || busy,
disabledReason: !canManage
? "You do not have permission to update sensitive fields."
: changedSensitiveColumns.length === 0
? "No sensitive field changes to save."
? "No sensitive-field changes are waiting to be saved."
: catalogOperationActive
? "Wait for the active catalog operation to finish."
: busy
@@ -386,10 +384,10 @@ export function DatabaseColumns({
];
const runFleetAction = async (action: FleetColumnAction) => {
if (action === "sync-columns") await synchronizeColumns();
else if (action === "generate-descriptions") await generateDescriptions();
if (action === "generate-descriptions") await generateDescriptions();
else if (action === "consolidate-descriptions") await consolidateDescriptions();
else if (action === "suggest-sensitive") await suggestSensitive();
else if (action === "sync-columns") onSync();
else await saveSensitive();
};
@@ -398,25 +396,9 @@ export function DatabaseColumns({
setSelectedIds([]);
};
const synchronizeColumns = async () => {
if (selectedIds.length === 0) return;
setBusy(true);
try {
const run = await startCatalogSync(databaseId, databaseVersion, "columns", [table.id]);
onCatalogSyncRunStarted?.(run);
gridRef.current?.api.deselectAll();
setSelectedIds([]);
toast.success("Column synchronization started");
} catch (error) {
toast.error(apiErrorMessage(error));
} finally {
setBusy(false);
}
};
const columns = useMemo<ColDef<CatalogColumn>[]>(() => [
{ field: "ordinalPosition", headerName: "#", width: 64, maxWidth: 64, filter: "agNumberColumnFilter" },
{ field: "sensitive", headerName: "Sensitive", headerTooltip: "Mark this column as sensitive; this does not select it for actions.", minWidth: 110, width: 110, sortable: false, filter: false, resizable: false, cellRenderer: SensitiveCell },
{ field: "sensitive", headerName: "Sensitive", minWidth: 110, width: 110, sortable: false, filter: false, resizable: false, cellRenderer: SensitiveCell },
{ field: "name", headerName: "Name", minWidth: 190, flex: 1, cellClass: "font-mono text-xs" },
{ field: "dataType", headerName: "Type", minWidth: 150, flex: 0.8, cellClass: "font-mono text-xs" },
{ headerName: "Keys", minWidth: 125, width: 125, sortable: false, filter: false, cellRenderer: KeyCell },
@@ -439,7 +421,7 @@ export function DatabaseColumns({
<p className="font-semibold text-destructive">Catalog columns could not be loaded.</p>
<p className="mt-1 leading-5 text-muted-foreground">{apiErrorMessage(error)}</p>
<Button type="button" variant="outline" className="mt-4" disabled={isFetching} onClick={() => void refetch()}>
<RefreshCw className={isFetching ? "animate-spin" : ""} aria-hidden="true" /> Retry loading columns
<RefreshCw className={isFetching ? "animate-spin" : ""} aria-hidden="true" /> Retry columns
</Button>
</div>
</div>
@@ -486,6 +468,7 @@ export function DatabaseColumns({
busyLabel={sensitiveAction === "suggest" ? "Suggesting…" : sensitiveAction === "save" ? "Saving…" : "Running…"}
onRun={runFleetAction}
onClear={clearSelection}
renderContext={(action) => action === "generate-descriptions" || action === "suggest-sensitive" ? metadataModelControl : null}
label="Column action"
/>
</>
@@ -497,9 +480,8 @@ export function DatabaseColumns({
<Menu.Portal>
<Menu.Positioner side="bottom" align="start" sideOffset={4}>
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || !selectedIds.length || catalogOperationActive || busy} onClick={() => void synchronizeColumns()}>Synchronize columns for this table</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || !selectedMetadataModel || descriptionGenerationActive || busy} onClick={() => void generateDescriptions()}>Generate {selectedIds.length === 1 ? "column description" : "column descriptions"}</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || busy} onClick={() => void consolidateDescriptions()}>Copy generated descriptions to Description</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || !selectedMetadataModel || descriptionGenerationActive || busy} onClick={() => void generateDescriptions()}>Generate {selectedIds.length === 1 ? "description" : "descriptions"}</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || busy} onClick={() => void consolidateDescriptions()}>Move generated description to Description</Menu.Item>
</Menu.Popup>
</Menu.Positioner>
</Menu.Portal>
@@ -519,6 +501,7 @@ export function DatabaseColumns({
busyLabel={sensitiveAction === "save" ? "Saving…" : "Running…"}
onRun={runFleetAction}
onClear={clearSelection}
renderContext={(action) => action === "generate-descriptions" || action === "suggest-sensitive" ? metadataModelControl : null}
label="Column action"
/>
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search columns" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
@@ -532,6 +515,7 @@ export function DatabaseColumns({
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
{changedSensitiveColumns.length > 0 ? <Button type="button" disabled={!canManage || busy} onClick={() => void saveSensitive()}><Save />{sensitiveAction === "save" ? "Saving…" : "Save sensitive fields"}</Button> : null}
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
<Button type="button" disabled={!canManage || busy} onClick={onSync}><RefreshCw />Sync columns</Button>
</>
)
)}
@@ -548,17 +532,6 @@ export function DatabaseColumns({
defaultColDef={{ sortable: true, filter: true, resizable: true }}
getRowId={({ data: row }) => row.id}
rowSelection={{ mode: "multiRow", selectAll: "filtered", enableClickSelection: false }}
selectionColumnDef={{
headerName: "Select",
headerTooltip: "Select columns for actions",
width: 72,
minWidth: 72,
maxWidth: 72,
pinned: "left",
sortable: false,
resizable: false,
suppressMovable: true,
}}
onSelectionChanged={({ api }) => setSelectedIds(api.getSelectedRows().map((column) => column.id))}
rowHeight={44}
headerHeight={38}
@@ -14,9 +14,6 @@ const database: CatalogDatabase = {
workspaceId: "psd-clinical",
workspaceName: "Policlinico San Donato",
workspaceAvailable: true,
workspaceRevision: { commit: "a".repeat(40), blob: "b".repeat(40) },
workspaceEvidence: { sourceType: "filesystem", state: "materialized_current_revision" },
runtimeBinding: { transport: "postgres_direct", configurationState: "ready", sessionTransportSupported: true },
configured: true,
engine: "postgres",
databaseName: "warehouse",
@@ -110,12 +107,12 @@ test("Fleet columns distinguish a failed query from an empty catalog and retry i
presentation="fleet"
databaseId={database.id!}
table={table}
databaseVersion={database.version}
canManage
selectedMetadataModel={null}
descriptionGenerationActive={false}
onDescriptionGenerationRunStarted={noop}
onNavigationStateChange={noop}
onSync={noop}
onSuggestSensitive={noopAsync}
/>,
);
@@ -125,7 +122,7 @@ test("Fleet columns distinguish a failed query from an empty catalog and retry i
expect(alert).toHaveTextContent("The database catalog is unavailable.");
expect(screen.queryByText(/No columns synchronized/i)).not.toBeInTheDocument();
await user.click(screen.getByRole("button", { name: "Retry loading columns" }));
await user.click(screen.getByRole("button", { name: "Retry columns" }));
await waitFor(() => expect(requests).toBe(2));
await waitFor(() => expect(screen.queryByText("Catalog columns could not be loaded.")).not.toBeInTheDocument());
});
@@ -12,6 +12,7 @@ import { Button } from "../../components/ui/button";
import type {
CatalogDatabase,
CatalogSecretName,
CatalogSyncScope,
CatalogSyncRun,
DatabaseBinding,
DatabaseTransport,
@@ -22,7 +23,7 @@ import type {
DatabaseFormMode,
} from "./model";
import { statusLabel } from "./model";
import { DatabaseSyncMenu, type DatabaseSyncScope } from "./DatabaseSyncMenu";
import { DatabaseSyncMenu } from "./DatabaseSyncMenu";
const inputClass = "h-9 w-full rounded-md border border-input bg-card px-3 text-sm outline-none transition focus:border-primary/60 focus:ring-3 focus:ring-ring/15 read-only:bg-muted/40 read-only:text-muted-foreground disabled:bg-muted disabled:text-muted-foreground";
const labelClass = "grid min-w-0 gap-1.5 text-xs font-medium text-foreground";
@@ -112,6 +113,8 @@ function SecretField({
interface DatabaseFormProps {
mode: DatabaseFormMode;
row: CatalogDatabase;
availableWorkspaces: CatalogDatabase[];
workspaceLocked: boolean;
draft: DatabaseFormDraft;
dirty: boolean;
canManage: boolean;
@@ -122,6 +125,7 @@ interface DatabaseFormProps {
partialSecretFailure: string | null;
headingRef: RefObject<HTMLHeadingElement>;
onBack: () => void;
onWorkspaceChange: (workspaceId: string) => void;
onFieldChange: (field: "databaseName" | "schema", value: string) => void;
onTransportChange: (transport: DatabaseTransport) => void;
onBindingChange: <K extends keyof DatabaseBinding>(key: K, value: DatabaseBinding[K]) => void;
@@ -134,19 +138,18 @@ interface DatabaseFormProps {
onRetrySecrets: () => void;
onOpenTables: (origin: HTMLButtonElement) => void;
onOpenRelationships: () => void;
onSync: (scope: DatabaseSyncScope) => void;
onSync: (scope: CatalogSyncScope) => void;
onOpenSync: () => void;
onOpenDescriptionHistory?: () => void;
onOpenSensitiveHistory?: () => void;
activeSyncRun?: CatalogSyncRun;
presentation?: "page" | "drawer";
}
export function databaseFormTitle(
mode: DatabaseFormMode,
workspaceLocked: boolean,
presentation: "page" | "drawer" = "page",
): string {
if (mode === "configure") return "Configure catalog";
if (mode === "add") return workspaceLocked ? "Edit database" : "Add database";
if (mode === "edit") return "Edit database";
if (mode === "delete") return presentation === "drawer" ? "Remove configuration" : "Delete database";
return "Database details";
@@ -155,6 +158,8 @@ export function databaseFormTitle(
export function DatabaseForm({
mode,
row,
availableWorkspaces,
workspaceLocked,
draft,
dirty,
canManage,
@@ -165,6 +170,7 @@ export function DatabaseForm({
partialSecretFailure,
headingRef,
onBack,
onWorkspaceChange,
onFieldChange,
onTransportChange,
onBindingChange,
@@ -179,14 +185,12 @@ export function DatabaseForm({
onOpenRelationships,
onSync,
onOpenSync,
onOpenDescriptionHistory,
onOpenSensitiveHistory,
activeSyncRun,
presentation = "page",
}: DatabaseFormProps) {
const title = databaseFormTitle(mode, presentation);
const title = databaseFormTitle(mode, workspaceLocked, presentation);
const readOnly = mode === "view" || mode === "delete";
const editable = mode === "configure" || mode === "edit";
const editable = mode === "add" || mode === "edit";
const busy = busyAction !== null;
const testDisabled = !canManage
|| !row.configured
@@ -264,9 +268,27 @@ export function DatabaseForm({
) : null}
<div className="grid gap-4 md:grid-cols-2">
<Field label="Workspace">
<input className={inputClass} value={row.workspaceName} readOnly />
</Field>
{mode === "add" ? (
<Field label="Workspace">
<select
className={inputClass}
aria-label="Workspace"
value={draft.workspaceId}
disabled={workspaceLocked || busy}
onChange={(event) => onWorkspaceChange(event.target.value)}
>
{availableWorkspaces.map((workspace) => (
<option key={workspace.workspaceId} value={workspace.workspaceId}>
{workspace.workspaceName}
</option>
))}
</select>
</Field>
) : (
<Field label="Workspace">
<input className={inputClass} value={row.workspaceName} readOnly />
</Field>
)}
<Field label="Workspace ID">
<input className={`${inputClass} font-mono text-xs`} value={row.workspaceId} readOnly />
</Field>
@@ -281,7 +303,7 @@ export function DatabaseForm({
disabled={readOnly || busy}
onChange={(event) => onTransportChange(event.target.value as DatabaseTransport)}
>
<option value="postgres_direct">Direct PostgreSQL connection</option>
<option value="postgres_direct">Direct PostgreSQL</option>
<option value="rest_api">REST API</option>
<option value="ssh_tunnel">SSH tunnel</option>
</select>
@@ -398,13 +420,13 @@ export function DatabaseForm({
<TestTube2 /> {busyAction === "test" ? "Testing…" : "Test connection"}
</Button>
) : null}
{mode === "configure" || mode === "edit" ? (
{mode === "add" || mode === "edit" ? (
<Button type="submit" disabled={saveDisabled}>
<Save />
{busyAction === "save"
? "Saving…"
: mode === "configure"
? "Save catalog configuration"
: mode === "add"
? workspaceLocked ? "Save database" : "Add database"
: "Save changes"}
</Button>
) : null}
@@ -437,16 +459,6 @@ export function DatabaseForm({
{activeSyncRun ? <RefreshCw className="animate-spin" /> : <History />} Sync history
</Button>
) : null}
{mode === "view" && row.configured && onOpenDescriptionHistory ? (
<Button type="button" variant="outline" size="sm" onClick={onOpenDescriptionHistory}>
<History /> Description history
</Button>
) : null}
{mode === "view" && row.configured && onOpenSensitiveHistory ? (
<Button type="button" variant="outline" size="sm" onClick={onOpenSensitiveHistory}>
<History /> Sensitive history
</Button>
) : null}
{mode === "view" ? (
<DatabaseSyncMenu
disabled={!canManage || !bindingReady || busy || Boolean(activeSyncRun)}
@@ -454,8 +466,6 @@ export function DatabaseForm({
onSelect={onSync}
/>
) : null}
{mode === "view" && row.configured && onOpenDescriptionHistory ? <Button type="button" variant="outline" onClick={onOpenDescriptionHistory}><History /> Description history</Button> : null}
{mode === "view" && row.configured && onOpenSensitiveHistory ? <Button type="button" variant="outline" onClick={onOpenSensitiveHistory}><History /> Sensitive history</Button> : null}
</div>
</div>
{row.lastTestedAt ? (
@@ -1,5 +1,4 @@
import { useEffect, useMemo, useRef, useState, type RefObject } from "react";
import { useQueries } from "@tanstack/react-query";
import { useEffect, useMemo, useRef, useState, type ReactNode, type RefObject } from "react";
import { Menu } from "@base-ui/react/menu";
import { AgGridReact } from "ag-grid-react";
import {
@@ -10,19 +9,17 @@ import {
} from "ag-grid-community";
import "ag-grid-community/styles/ag-grid.css";
import "ag-grid-community/styles/ag-theme-alpine.css";
import { ChevronDown, Eye, History, Info, Link2, Pencil, RefreshCw, Rows3, Sparkles, Trash2, X } from "lucide-react";
import { ChevronDown, Eye, History, Info, Pencil, RefreshCw, Rows3, Sparkles, Trash2, X } from "lucide-react";
import { Button } from "../../components/ui/button";
import { getCatalogMetrics } from "../../api/catalog-databases";
import type {
CatalogDatabase,
CatalogMetrics,
SensitiveDataSuggestionRun,
CatalogDatabaseMetadataDeleteTarget,
CatalogSyncScope,
DescriptionGenerationScope,
} from "../../api/catalog-databases";
import { databaseSyncItemClass, databaseSyncScopes, type DatabaseSyncScope } from "./DatabaseSyncMenu";
import { statusLabel } from "./model";
import { databaseSyncItemClass, databaseSyncScopes } from "./DatabaseSyncMenu";
import { FleetActionSelector, type FleetActionOption } from "./FleetActionSelector";
import { NO_METADATA_GENERATION_LLM_MODEL_MESSAGE } from "./MetadataGenerationModelSelector";
ModuleRegistry.registerModules([AllCommunityModule]);
@@ -36,16 +33,13 @@ interface DatabaseGridProps {
onSearchChange: (value: string) => void;
onView: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
onOpenTables?: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
onOpenRelationships?: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
onEdit: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
onDelete: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
onOpenSync: (row: CatalogDatabase) => void;
onSelectionChange?: (rows: CatalogDatabase[]) => void;
onTestSelected: (rows: CatalogDatabase[]) => Promise<void>;
onSyncSelected: (rows: CatalogDatabase[], scope: DatabaseSyncScope) => Promise<void>;
onSyncSelected: (rows: CatalogDatabase[], scope: CatalogSyncScope) => Promise<void>;
selectedMetadataModel: string | null;
descriptionGenerationActive: boolean;
sensitiveDataSuggestionRuns?: SensitiveDataSuggestionRun[];
onGenerateDescriptions: (
rows: CatalogDatabase[],
scope: Extract<DescriptionGenerationScope, "all" | "missing">,
@@ -55,6 +49,7 @@ interface DatabaseGridProps {
rows: CatalogDatabase[],
target: CatalogDatabaseMetadataDeleteTarget,
) => Promise<void>;
metadataModelControl?: ReactNode;
onRefresh?: () => void | Promise<void>;
presentation?: "legacy" | "fleet";
}
@@ -64,7 +59,6 @@ interface DatabaseGridContext {
presentation: "legacy" | "fleet";
onView: DatabaseGridProps["onView"];
onOpenTables?: DatabaseGridProps["onOpenTables"];
onOpenRelationships?: DatabaseGridProps["onOpenRelationships"];
onEdit: DatabaseGridProps["onEdit"];
onDelete: DatabaseGridProps["onDelete"];
onOpenSync: DatabaseGridProps["onOpenSync"];
@@ -73,6 +67,7 @@ interface DatabaseGridContext {
type DatabaseFleetAction =
| "test"
| "sync-tables"
| "sync-columns"
| "sync-relationships"
| "sync-all"
| "generate-all"
@@ -96,96 +91,18 @@ function useCompactViewport(): boolean {
return compact;
}
type StateTone = "neutral" | "info" | "success" | "warning" | "danger";
function StatusPill({ row }: { row: CatalogDatabase }) {
const tone = !row.workspaceAvailable
? "border-destructive/30 bg-destructive/10 text-destructive"
: !row.configured
? "border-border bg-muted text-muted-foreground"
: row.connectionStatus === "reachable"
? "border-emerald-300/70 bg-emerald-50 text-emerald-800 dark:bg-emerald-950/30 dark:text-emerald-300"
: row.connectionStatus === "failed"
? "border-destructive/30 bg-destructive/10 text-destructive"
: "border-amber-300/70 bg-amber-50 text-amber-900 dark:bg-amber-950/30 dark:text-amber-300";
const stateToneClass: Record<StateTone, string> = {
neutral: "bg-muted-foreground/55",
info: "bg-sky-500",
success: "bg-emerald-500",
warning: "bg-amber-500",
danger: "bg-destructive",
};
function StateCell({
label,
detail,
tone,
detailClassName = "text-muted-foreground",
}: {
label: string;
detail: string;
tone: StateTone;
detailClassName?: string;
}) {
return (
<div className="flex h-full min-w-0 flex-col justify-center gap-0.5 py-1 leading-tight">
<span className="flex min-w-0 items-center gap-1.5 text-xs font-semibold text-foreground">
<span className={`size-1.5 shrink-0 rounded-full ${stateToneClass[tone]}`} aria-hidden="true" />
<span className="truncate">{label}</span>
</span>
<span className={`truncate pl-3 text-[11px] ${detailClassName}`}>{detail}</span>
</div>
);
}
function endpoint(host: string | undefined, port: number | undefined): string {
if (!host) return "Endpoint not configured";
return port ? `${host}:${port}` : host;
}
function accessSummary(row: CatalogDatabase): { label: string; detail: string } {
const transport = row.configured
? row.binding.transport
: row.runtimeBinding?.transport ?? row.binding.transport;
if (transport === "rest_api") {
if (!row.binding.baseUrl) return { label: "REST", detail: "Endpoint not configured" };
try {
return { label: "REST", detail: new URL(row.binding.baseUrl).host };
} catch {
return { label: "REST", detail: row.binding.baseUrl };
}
}
if (transport === "ssh_tunnel") {
return { label: "SSH", detail: endpoint(row.binding.sshHost, row.binding.sshPort) };
}
return {
label: "Direct PostgreSQL connection",
detail: endpoint(row.binding.host, row.binding.port),
};
}
function coverageStatus(total: number, complete: number): { label: string; detail: string; tone: StateTone } {
if (total === 0) return { label: "Not started", detail: "No catalog objects", tone: "danger" };
if (complete >= total) return { label: "Complete", detail: `${complete}/${total}`, tone: "success" };
if (complete > 0) return { label: "Partial", detail: `${complete}/${total}`, tone: "warning" };
return { label: "Not started", detail: `0/${total}`, tone: "danger" };
}
function CatalogStatusCells({ row, metrics, sensitiveRun }: { row: CatalogDatabase; metrics?: CatalogMetrics; sensitiveRun?: SensitiveDataSuggestionRun }) {
const access = accessSummary(row);
const synchronized = row.schemaSyncedVersion === row.version;
const syncStatus = !row.schemaSyncedVersion
? { label: "Not synchronized", detail: "Schema not synchronized", tone: "danger" as const }
: synchronized
? { label: "Synchronized", detail: "Current schema", tone: "success" as const }
: { label: "Partial", detail: "Schema revision changed", tone: "warning" as const };
const descriptions = coverageStatus(metrics?.descriptionTargets ?? 0, metrics?.describedTargets ?? 0);
const sensitiveProcessed = sensitiveRun ? sensitiveRun.suggestedSensitive + sensitiveRun.suggestedNonSensitive : 0;
const sensitive = !sensitiveRun
? { label: "Not started", detail: "No analysis has been run", tone: "danger" as const }
: sensitiveRun.status === "completed" && sensitiveProcessed >= sensitiveRun.total
? { label: "Complete", detail: `${sensitiveProcessed}/${sensitiveRun.total}`, tone: "success" as const }
: ["failed", "interrupted"].includes(sensitiveRun.status)
? { label: "Incomplete", detail: `${sensitiveProcessed}/${sensitiveRun.total}`, tone: "danger" as const }
: { label: "Partial", detail: `${sensitiveProcessed}/${sensitiveRun.total}`, tone: "warning" as const };
return (
<div className="grid h-full min-w-[540px] grid-cols-4 gap-3 py-1">
<StateCell label={access.label} detail={access.detail} tone="info" />
<StateCell label={syncStatus.label} detail={syncStatus.detail} tone={syncStatus.tone} />
<StateCell label={descriptions.label} detail={`Descriptions · ${descriptions.detail}`} tone={descriptions.tone} />
<StateCell label={sensitive.label} detail={`Sensitive · ${sensitive.detail}`} tone={sensitive.tone} />
</div>
);
return <div className="flex items-center gap-1.5"><span className={`inline-flex max-w-full truncate rounded-full border px-2 py-0.5 text-[11px] font-semibold ${tone}`}>{statusLabel(row)}</span>{row.activeSyncRun ? <span className="inline-flex rounded-full border border-primary/30 bg-primary/8 px-2 py-0.5 text-[11px] font-semibold text-primary">Syncing</span> : null}</div>;
}
function DatabaseActionsCell({
@@ -194,51 +111,19 @@ function DatabaseActionsCell({
}: ICellRendererParams<CatalogDatabase, unknown, DatabaseGridContext>) {
if (!data || !context) return null;
if (data.workspaceAvailable && !data.configured) {
const configureReasonId = `database-configure-reason-${data.workspaceId}`;
return (
<div className="flex h-full items-center justify-end gap-1.5" onClick={(event) => event.stopPropagation()}>
<Button
type="button"
variant="ghost"
size="icon-lg"
className={context.presentation === "fleet" ? "thot-fleet-icon-action" : undefined}
data-tooltip={context.presentation === "fleet" ? "Details" : undefined}
title={context.presentation === "legacy" ? `View ${data.workspaceName}` : undefined}
aria-label={`View ${data.workspaceName}`}
onClick={(event) => context.onView(data, event.currentTarget)}
>
{context.presentation === "fleet" ? <Info aria-hidden="true" /> : <Eye aria-hidden="true" />}
</Button>
<Button
type="button"
size="sm"
className="whitespace-nowrap"
aria-label={`Configure catalog for ${data.workspaceName}`}
aria-describedby={!context.canManage ? configureReasonId : undefined}
disabled={!context.canManage}
onClick={(event) => context.onEdit(data, event.currentTarget)}
>
Configure catalog
</Button>
{!context.canManage ? (
<span id={configureReasonId} className="sr-only">
Database management permission is required.
</span>
) : null}
</div>
);
}
const editDisabled = !context.canManage || !data.workspaceAvailable;
const deleteDisabled = !context.canManage || !data.configured;
const editLabel = `Edit ${data.workspaceName}`;
const editReasonId = `database-edit-reason-${data.workspaceId}`;
const deleteReasonId = `database-delete-reason-${data.workspaceId}`;
const relationshipsReasonId = `database-relationships-reason-${data.workspaceId}`;
return (
<div className="flex h-full items-center justify-end gap-0.5" onClick={(event) => event.stopPropagation()}>
{data.activeSyncRun || (context.presentation === "fleet" && data.configured && data.id) ? (
<Button type="button" variant="ghost" size="icon-lg" className={context.presentation === "fleet" ? "thot-fleet-icon-action" : undefined} data-tooltip={context.presentation === "fleet" ? "Synchronization history" : undefined} title={context.presentation === "legacy" ? `View synchronization for ${data.workspaceName}` : undefined} aria-label={`View synchronization history for ${data.workspaceName}`} onClick={() => context.onOpenSync(data)}>
{data.activeSyncRun ? <RefreshCw className="animate-spin" aria-hidden="true" /> : <History aria-hidden="true" />}
</Button>
) : null}
{context.presentation === "fleet" && context.onOpenTables ? (
<span className="thot-fleet-icon-action inline-flex" data-tooltip="Tables">
<Button
@@ -253,26 +138,6 @@ function DatabaseActionsCell({
</Button>
</span>
) : null}
{context.presentation === "fleet" && context.onOpenRelationships ? (
<span className="thot-fleet-icon-action inline-flex" data-tooltip="Relationships">
<Button
type="button"
variant="ghost"
size="icon-lg"
aria-label={`View relationships for ${data.workspaceName}`}
aria-describedby={!data.configured || !data.id ? relationshipsReasonId : undefined}
disabled={!data.configured || !data.id}
onClick={(event) => context.onOpenRelationships?.(data, event.currentTarget)}
>
<Link2 aria-hidden="true" />
</Button>
{!data.configured || !data.id ? (
<span id={relationshipsReasonId} className="sr-only">
Save this database configuration before managing relationships.
</span>
) : null}
</span>
) : null}
<Button
type="button"
variant="ghost"
@@ -328,41 +193,20 @@ export function DatabaseGrid({
onSearchChange,
onView,
onOpenTables,
onOpenRelationships,
onEdit,
onDelete,
onOpenSync,
onSelectionChange,
onTestSelected,
onSyncSelected,
selectedMetadataModel,
descriptionGenerationActive,
sensitiveDataSuggestionRuns = [],
onGenerateDescriptions,
onSuggestSensitive,
onDeleteMetadataSelected,
metadataModelControl,
onRefresh,
presentation = "legacy",
}: DatabaseGridProps) {
const metricQueries = useQueries({
queries: rows.filter((row) => row.id).map((row) => ({
queryKey: ["catalog-metrics", row.id],
queryFn: () => getCatalogMetrics(row.id!),
enabled: canManage,
staleTime: 10_000,
})),
});
const metricsByDatabase = useMemo(() => {
const result: Record<string, CatalogMetrics> = {};
rows.filter((row) => row.id).forEach((row, index) => {
const metrics = metricQueries[index]?.data;
if (metrics && row.id) result[row.id] = metrics;
});
return result;
}, [metricQueries, rows]);
const sensitiveRunsByDatabase = useMemo(() => new Map(
rows.filter((row) => row.id).map((row) => [row.id!, sensitiveDataSuggestionRuns.find((run) => run.databaseId === row.id)]),
), [rows, sensitiveDataSuggestionRuns]);
const compact = useCompactViewport();
const gridRef = useRef<AgGridReact<CatalogDatabase>>(null);
const actionsTriggerRef = useRef<HTMLButtonElement>(null);
@@ -374,8 +218,8 @@ export function DatabaseGrid({
Extract<DescriptionGenerationScope, "all" | "missing"> | null
>(null);
const context = useMemo<DatabaseGridContext>(
() => ({ canManage, presentation, onView, onOpenTables, onOpenRelationships, onEdit, onDelete, onOpenSync }),
[canManage, presentation, onView, onOpenTables, onOpenRelationships, onEdit, onDelete, onOpenSync],
() => ({ canManage, presentation, onView, onOpenTables, onEdit, onDelete, onOpenSync }),
[canManage, presentation, onView, onOpenTables, onEdit, onDelete, onOpenSync],
);
const columnDefs = useMemo<ColDef<CatalogDatabase>[]>(() => {
@@ -386,18 +230,49 @@ export function DatabaseGrid({
flex: compact ? undefined : 1.35,
width: compact ? 112 : undefined,
};
const status: ColDef<CatalogDatabase> = {
headerName: "Catalog status",
minWidth: 580,
flex: 2.5,
cellRenderer: ({ data }: ICellRendererParams<CatalogDatabase>) => {
const databaseId = data?.id;
return data
? <CatalogStatusCells row={data} metrics={databaseId ? metricsByDatabase[databaseId] : undefined} sensitiveRun={databaseId ? sensitiveRunsByDatabase.get(databaseId) : undefined} />
: null;
const database: ColDef<CatalogDatabase> = {
field: "databaseName", headerName: "Database", minWidth: 145, flex: 1,
};
const schema: ColDef<CatalogDatabase> = {
field: "schema", headerName: "Schema", minWidth: 140, flex: 0.9,
};
const transport: ColDef<CatalogDatabase> = {
field: "binding.transport",
headerName: "Transport",
minWidth: 135,
flex: 0.9,
valueFormatter: ({ value }) => String(value ?? "").replaceAll("_", " "),
};
const endpoint: ColDef<CatalogDatabase> = {
headerName: "Endpoint",
minWidth: 190,
flex: 1.2,
valueGetter: ({ data }) => {
if (!data) return "";
if (data.binding.transport === "rest_api") return data.binding.baseUrl ?? "";
if (data.binding.transport === "ssh_tunnel") return data.binding.sshHost ?? "";
return data.binding.host ? `${data.binding.host}:${data.binding.port ?? 5432}` : "";
},
};
const actionsWidth = presentation === "fleet" ? 232 : 210;
const status: ColDef<CatalogDatabase> = {
headerName: "Status",
minWidth: compact ? 96 : 130,
width: compact ? 96 : undefined,
flex: compact ? undefined : 0.8,
cellClass: compact ? "thot-database-status-cell" : undefined,
valueGetter: ({ data }) => (data ? statusLabel(data) : ""),
cellRenderer: ({ data }: ICellRendererParams<CatalogDatabase>) => (
data ? <StatusPill row={data} /> : null
),
};
const updated: ColDef<CatalogDatabase> = {
field: "updatedAt",
headerName: "Updated",
minWidth: 170,
flex: 0.9,
valueFormatter: ({ value }) => value ? new Date(String(value)).toLocaleString() : "",
};
const actionsWidth = presentation === "fleet" ? 196 : compact ? 148 : 164;
const actions: ColDef<CatalogDatabase> = {
colId: "actions",
headerName: "Actions",
@@ -415,8 +290,10 @@ export function DatabaseGrid({
cellRenderer: DatabaseActionsCell,
};
return [workspace, status, actions];
}, [compact, metricsByDatabase, presentation, sensitiveRunsByDatabase]);
return compact
? [workspace, status, database, schema, transport, endpoint, updated, actions]
: [workspace, database, schema, transport, endpoint, status, updated, actions];
}, [compact, presentation]);
const configuredCount = rows.filter((row) => row.configured).length;
const hiddenSelected = selectedRows.filter((row) => {
@@ -455,14 +332,14 @@ export function DatabaseGrid({
: undefined);
const generationReason = permissionReason
?? (selectedRows.length !== 1 ? "Select exactly one database" : undefined)
?? (!selectedMetadataModel ? NO_METADATA_GENERATION_LLM_MODEL_MESSAGE : undefined)
?? (!selectedMetadataModel ? "Choose a metadata model in the generation flow" : undefined)
?? (descriptionGenerationActive ? "Wait for active description generation" : undefined)
?? (selectedRows.some((row) => !row.configured || !row.id || row.activeSyncRun)
? "The selected database must be configured and idle"
: undefined);
const sensitiveReason = permissionReason
?? (selectedRows.length !== 1 ? "Select exactly one database" : undefined)
?? (!selectedMetadataModel ? NO_METADATA_GENERATION_LLM_MODEL_MESSAGE : undefined)
?? (!selectedMetadataModel ? "Choose a metadata model in the suggestion flow" : undefined)
?? (descriptionGenerationActive ? "Wait for the active metadata operation" : undefined)
?? (selectedRows.some((row) => !row.configured || !row.id || row.activeSyncRun)
? "The selected database must be configured and idle"
@@ -475,6 +352,7 @@ export function DatabaseGrid({
return [
{ id: "test", label: "Test connections", group: "Connection", disabled: !canTestSelection, disabledReason: testReason },
{ id: "sync-tables", label: "Synchronize tables", group: "Synchronization", disabled: !canSyncSelection, disabledReason: syncReason },
{ id: "sync-columns", label: "Synchronize all columns", group: "Synchronization", disabled: !canSyncSelection, disabledReason: syncReason },
{ id: "sync-relationships", label: "Synchronize relationships", group: "Synchronization", disabled: !canSyncSelection, disabledReason: syncReason },
{ id: "sync-all", label: "Synchronize all", group: "Synchronization", disabled: !canSyncSelection, disabledReason: syncReason },
{ id: "generate-missing", label: "Generate missing descriptions", group: "Descriptions", disabled: !canGenerateDescriptions, disabledReason: generationReason },
@@ -521,10 +399,15 @@ export function DatabaseGrid({
const perform = async (
kind: "test" | "sync" | "generate" | "suggest" | "delete",
operation: () => Promise<void>,
clearSelection = true,
) => {
setAction(kind);
try {
await operation();
if (clearSelection) {
gridRef.current?.api.deselectAll();
setSelectedRows([]);
}
setPendingDelete(null);
setPendingGenerationScope(null);
if (kind === "delete") window.setTimeout(() => searchInputRef.current?.focus(), 0);
@@ -548,11 +431,11 @@ export function DatabaseGrid({
return;
}
if (selectedAction.startsWith("sync-")) {
const scope = selectedAction.slice("sync-".length) as DatabaseSyncScope;
const scope = selectedAction.slice("sync-".length) as CatalogSyncScope;
await perform("sync", () => onSyncSelected(selectedRows, scope));
return;
}
await perform("suggest", () => onSuggestSensitive(selectedRows));
await perform("suggest", () => onSuggestSensitive(selectedRows), false);
};
return (
@@ -608,7 +491,7 @@ export function DatabaseGrid({
() => onGenerateDescriptions(selectedRows, pendingGenerationScope),
)}
>
{pendingGenerationScope === "all" ? "Generate all descriptions" : "Generate missing descriptions"}
{pendingGenerationScope === "all" ? "Generate All" : "Generate Missing"}
</Button>
</div>
) : pendingDelete ? (
@@ -655,6 +538,12 @@ export function DatabaseGrid({
busyLabel={action === "suggest" ? "Suggesting…" : "Running…"}
selectRef={actionSelectRef}
onRun={runFleetAction}
renderContext={(selectedAction) => (
metadataModelControl
&& ["generate-all", "generate-missing", "suggest-sensitive"].includes(selectedAction)
? <div className="thot-fleet-metadata-model">{metadataModelControl}</div>
: null
)}
onClear={() => {
gridRef.current?.api.deselectAll();
setSelectedRows([]);
@@ -688,14 +577,14 @@ export function DatabaseGrid({
disabled={!canGenerateDescriptions}
onClick={() => setPendingGenerationScope("all")}
>
Generate all descriptions
Generate All
</Menu.Item>
<Menu.Item
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
disabled={!canGenerateDescriptions}
onClick={() => setPendingGenerationScope("missing")}
>
Generate missing descriptions
Generate Missing
</Menu.Item>
<Menu.Separator className="my-1 h-px bg-border" />
<Menu.Item
@@ -721,7 +610,7 @@ export function DatabaseGrid({
variant="outline"
disabled={!canManage || action !== null || descriptionGenerationActive}
title={descriptionGenerationActive ? "Wait for the active description generation to finish" : undefined}
onClick={() => void perform("suggest", () => onSuggestSensitive(selectedRows))}
onClick={() => void perform("suggest", () => onSuggestSensitive(selectedRows), false)}
>
<Sparkles />{action === "suggest" ? "Suggesting…" : "Suggest sensitive fields"}
</Button>
@@ -742,7 +631,7 @@ export function DatabaseGrid({
</Button>
) : null}</>}
<span className="whitespace-nowrap text-xs tabular-nums text-muted-foreground">
{configuredCount}/{rows.length} catalogs configured
{configuredCount}/{rows.length}
</span>
{isFetching && !isLoading ? (
<span className="text-xs text-muted-foreground" role="status">Refreshing</span>
@@ -769,11 +658,9 @@ export function DatabaseGrid({
onSelectionChanged={({ api }) => {
if (pendingDelete && action === null) setPendingDelete(null);
if (pendingGenerationScope && action === null) setPendingGenerationScope(null);
const rows = api.getSelectedRows();
setSelectedRows(rows);
onSelectionChange?.(rows);
setSelectedRows(api.getSelectedRows());
}}
rowHeight={52}
rowHeight={44}
headerHeight={38}
animateRows={false}
/>
@@ -1,379 +0,0 @@
import { render, screen, waitFor, within } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { http, HttpResponse } from "msw";
import { expect, test, vi } from "vitest";
import type {
CatalogColumn,
CatalogDatabase,
CatalogRelationship,
CatalogTable,
} from "../../api/catalog-databases";
import { Toaster } from "../../components/ui/sonner";
import { server } from "../../test/msw";
import { DatabaseRelationships } from "./DatabaseRelationships";
const databaseId = "11111111-1111-4111-8111-111111111111";
const database: CatalogDatabase = {
id: databaseId,
workspaceId: "psd",
workspaceName: "Policlinico San Donato",
workspaceAvailable: true,
workspaceRevision: { commit: "a".repeat(40), blob: "b".repeat(40) },
workspaceEvidence: { sourceType: "filesystem", state: "materialized_current_revision" },
runtimeBinding: { transport: "postgres_direct", configurationState: "ready", sessionTransportSupported: true },
configured: true,
engine: "postgres",
databaseName: "analytics",
schema: "public",
binding: { transport: "postgres_direct", port: 5432 },
connectionStatus: "reachable",
testedVersion: 3,
version: 3,
createdAt: "2026-08-31T12:00:00.000Z",
updatedAt: "2026-08-31T12:00:00.000Z",
activeSyncRun: undefined,
secrets: {
password: false,
apiKey: false,
sshPrivateKey: false,
sshPrivateKeyPassphrase: false,
sshKnownHosts: false,
tlsCa: false,
},
};
function relationship(
id: string,
origin: CatalogRelationship["origin"],
status: CatalogRelationship["status"],
sourceTableName: string,
): CatalogRelationship {
const physical = origin === "physical";
return {
id,
databaseId,
constraintName: physical ? `${sourceTableName}_user_id_fkey` : null,
sourceTableId: `${id}-source-table`,
sourceTableName,
targetTableId: `${id}-target-table`,
targetTableName: "users",
updateRule: physical ? "NO ACTION" : null,
deleteRule: physical ? "CASCADE" : null,
deferrable: false,
initiallyDeferred: false,
columns: [{
position: 1,
sourceColumnId: `${id}-source-column`,
sourceColumnName: "user_id",
targetColumnId: `${id}-target-column`,
targetColumnName: "id",
}],
origin,
status,
lastSyncedDatabaseVersion: physical ? 3 : null,
lastSyncedAt: physical ? "2026-08-31T12:00:00.000Z" : null,
createdAt: "2026-08-31T12:00:00.000Z",
updatedAt: "2026-08-31T12:00:00.000Z",
};
}
function renderRelationships(rows: CatalogRelationship[], canManage = true) {
server.use(http.get(
"/api/catalog/databases/:databaseId/relationships",
() => HttpResponse.json(rows),
));
const client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
const callbacks = {
onBackToDatabases: vi.fn(),
onOpenOverview: vi.fn(),
onOpenTables: vi.fn(),
onRunStarted: vi.fn(),
onOpenSync: vi.fn(),
};
const view = render(
<QueryClientProvider client={client}>
<DatabaseRelationships
presentation="fleet"
database={database}
canManage={canManage}
{...callbacks}
/>
<Toaster duration={Infinity} />
</QueryClientProvider>,
);
return { ...view, client, callbacks };
}
test("shows one relationship map and filters active, excluded, and all relationships", async () => {
const user = userEvent.setup();
renderRelationships([
relationship("physical", "physical", "active", "visits"),
relationship("generated", "generated", "active", "orders"),
relationship("manual", "manual", "active", "payments"),
relationship("excluded", "generated", "excluded", "legacy_orders"),
]);
expect(await screen.findByText("visits")).toBeVisible();
expect(screen.getByText("orders")).toBeVisible();
expect(screen.getByText("payments")).toBeVisible();
expect(screen.queryByText("legacy_orders")).not.toBeInTheDocument();
expect(screen.getByText("Physical")).toBeVisible();
expect(screen.getByText("Generated")).toBeVisible();
expect(screen.getByText("Manual")).toBeVisible();
await user.selectOptions(screen.getByRole("combobox", { name: "Relationship status" }), "excluded");
expect(await screen.findByText("legacy_orders")).toBeVisible();
await waitFor(() => expect(screen.queryByText("visits")).not.toBeInTheDocument());
await user.selectOptions(screen.getByRole("combobox", { name: "Relationship status" }), "all");
expect(await screen.findByText("visits")).toBeVisible();
expect(screen.getByText("legacy_orders")).toBeVisible();
});
test("adds a manual relationship from four catalog selections", async () => {
const user = userEvent.setup();
const orders: CatalogTable = {
id: "22222222-2222-4222-8222-222222222222",
databaseId,
name: "orders",
sourceComment: null,
description: null,
generatedDescription: null,
version: 1,
createdAt: "2026-08-31T12:00:00.000Z",
updatedAt: "2026-08-31T12:00:00.000Z",
};
const users: CatalogTable = {
...orders,
id: "33333333-3333-4333-8333-333333333333",
name: "users",
};
const sourceColumn: CatalogColumn = {
id: "44444444-4444-4444-8444-444444444444",
tableId: orders.id,
name: "user_id",
ordinalPosition: 1,
dataType: "bigint",
isNullable: false,
defaultExpression: null,
primaryKeyPosition: null,
isPrimaryKey: false,
isForeignKey: false,
foreignKeyCount: 0,
sourceComment: null,
description: null,
generatedDescription: null,
sensitive: false,
lastSyncedDatabaseVersion: 3,
lastSyncedAt: "2026-08-31T12:00:00.000Z",
version: 1,
createdAt: "2026-08-31T12:00:00.000Z",
updatedAt: "2026-08-31T12:00:00.000Z",
};
const targetColumn: CatalogColumn = {
...sourceColumn,
id: "55555555-5555-4555-8555-555555555555",
tableId: users.id,
name: "id",
primaryKeyPosition: 1,
isPrimaryKey: true,
};
let requestBody: unknown;
server.use(
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([orders, users])),
http.get(
"/api/catalog/databases/:databaseId/tables/:tableId/columns",
({ params }) => HttpResponse.json(params.tableId === orders.id ? [sourceColumn] : [targetColumn]),
),
http.post("/api/catalog/databases/:databaseId/relationships", async ({ request }) => {
requestBody = await request.json();
return HttpResponse.json(relationship("created", "manual", "active", "orders"), { status: 201 });
}),
);
renderRelationships([]);
await user.click(await screen.findByRole("button", { name: "Add relationship" }));
const drawer = await screen.findByRole("dialog", { name: "Add relationship" });
expect(drawer).toHaveAttribute("aria-modal", "false");
const submit = within(drawer).getByRole("button", { name: "Add relationship" });
await user.selectOptions(screen.getByRole("combobox", { name: "Source table" }), orders.id);
await user.selectOptions(screen.getByRole("combobox", { name: "Source column" }), sourceColumn.id);
await user.selectOptions(screen.getByRole("combobox", { name: "Target table" }), users.id);
await user.selectOptions(screen.getByRole("combobox", { name: "Target column" }), targetColumn.id);
expect(submit).toBeEnabled();
await user.click(submit);
await waitFor(() => expect(requestBody).toEqual({
sourceColumnId: sourceColumn.id,
targetColumnId: targetColumn.id,
}));
expect(await screen.findByText("Relationship added.")).toBeVisible();
await waitFor(() => expect(screen.queryByRole("dialog", { name: "Add relationship" })).not.toBeInTheDocument());
});
test("rebuilds generated relationships with progress and a result summary", async () => {
const user = userEvent.setup();
let finish: (() => void) | undefined;
server.use(http.post(
"/api/catalog/databases/:databaseId/relationships/rebuild-generated",
async () => {
await new Promise<void>((resolve) => { finish = resolve; });
return HttpResponse.json({ added: 12, alreadyPresent: 7, excluded: 3, ambiguous: 2 });
},
));
renderRelationships([]);
await user.selectOptions(
await screen.findByRole("combobox", { name: "Relationship action" }),
"rebuild-generated",
);
await user.click(screen.getByRole("button", { name: "Run action" }));
expect(await screen.findByRole("button", { name: "Rebuilding…" })).toBeDisabled();
finish?.();
expect(await screen.findByText(
"Rebuild complete: 12 added, 7 already present, 3 excluded, 2 ambiguous.",
)).toBeVisible();
});
test("reports when rebuilding finds no new relationships", async () => {
const user = userEvent.setup();
server.use(http.post(
"/api/catalog/databases/:databaseId/relationships/rebuild-generated",
() => HttpResponse.json({ added: 0, alreadyPresent: 0, excluded: 0, ambiguous: 0 }),
));
renderRelationships([]);
await user.selectOptions(
await screen.findByRole("combobox", { name: "Relationship action" }),
"rebuild-generated",
);
await user.click(screen.getByRole("button", { name: "Run action" }));
expect(await screen.findByText("Rebuild complete: no new relationships found.")).toBeVisible();
});
test("excludes a generated relationship after an explanatory drawer confirmation", async () => {
const user = userEvent.setup();
const generated = relationship("generated", "generated", "active", "orders");
let requestBody: unknown;
server.use(http.patch(
"/api/catalog/databases/:databaseId/relationships/:relationshipId",
async ({ request }) => {
requestBody = await request.json();
return HttpResponse.json({ ...generated, status: "excluded" });
},
));
renderRelationships([generated]);
await user.click(await screen.findByRole("button", {
name: "View relationship orders.user_id to users.id",
}));
const drawer = await screen.findByRole("dialog", { name: "Relationship details" });
expect(within(drawer).getByText("Generated")).toBeVisible();
await user.click(within(drawer).getByRole("button", { name: "Exclude" }));
expect(within(drawer).getByText(
"The relationship will move to Excluded. Rebuilds will not recreate it, and you can restore it later.",
)).toBeVisible();
const confirmExclude = within(drawer).getByRole("button", { name: "Exclude relationship" });
await waitFor(() => expect(confirmExclude).toHaveFocus());
await user.click(confirmExclude);
await waitFor(() => expect(requestBody).toEqual({ status: "excluded" }));
expect(await screen.findByText(
"Relationship excluded. Future rebuilds will leave it excluded.",
)).toBeVisible();
});
test("keeps physical relationships read-only in the details drawer", async () => {
const user = userEvent.setup();
renderRelationships([relationship("physical", "physical", "active", "visits")]);
await user.click(await screen.findByRole("button", {
name: "View relationship visits.user_id to users.id",
}));
const drawer = await screen.findByRole("dialog", { name: "Relationship details" });
expect(within(drawer).getByText("This information is synchronized from the source schema and is read-only.")).toBeVisible();
expect(within(drawer).queryByRole("button", { name: "Exclude" })).not.toBeInTheDocument();
expect(within(drawer).queryByRole("button", { name: "Restore" })).not.toBeInTheDocument();
expect(within(drawer).queryByRole("button", { name: "Delete permanently" })).not.toBeInTheDocument();
});
test("permanently deletes a logical relationship after warning that rebuild may recreate it", async () => {
const user = userEvent.setup();
const manual = relationship("manual", "manual", "active", "payments");
let deletes = 0;
server.use(http.delete(
"/api/catalog/databases/:databaseId/relationships/:relationshipId",
() => {
deletes += 1;
return new HttpResponse(null, { status: 204 });
},
));
renderRelationships([manual]);
await user.click(await screen.findByRole("button", {
name: "View relationship payments.user_id to users.id",
}));
const drawer = await screen.findByRole("dialog", { name: "Relationship details" });
await user.click(within(drawer).getByRole("button", { name: "Delete permanently" }));
expect(within(drawer).getByText(
"The relationship record will be erased. A future rebuild may infer it again.",
)).toBeVisible();
await user.click(within(drawer).getByRole("button", { name: "Delete permanently" }));
await waitFor(() => expect(deletes).toBe(1));
expect(await screen.findByText(
"Relationship deleted permanently. A future rebuild may infer it again.",
)).toBeVisible();
});
test("restores an excluded logical relationship", async () => {
const user = userEvent.setup();
const excluded = relationship("excluded", "generated", "excluded", "legacy_orders");
let requestBody: unknown;
server.use(http.patch(
"/api/catalog/databases/:databaseId/relationships/:relationshipId",
async ({ request }) => {
requestBody = await request.json();
return HttpResponse.json({ ...excluded, status: "active" });
},
));
renderRelationships([excluded]);
await user.selectOptions(
await screen.findByRole("combobox", { name: "Relationship status" }),
"excluded",
);
await user.click(await screen.findByRole("button", {
name: "View relationship legacy_orders.user_id to users.id",
}));
const drawer = await screen.findByRole("dialog", { name: "Relationship details" });
await user.click(within(drawer).getByRole("button", { name: "Restore" }));
await waitFor(() => expect(requestBody).toEqual({ status: "active" }));
expect(await screen.findByText("Relationship restored.")).toBeVisible();
});
test("keeps relationship context open when a logical mutation fails", async () => {
const user = userEvent.setup();
const generated = relationship("generated-error", "generated", "active", "orders");
server.use(http.patch(
"/api/catalog/databases/:databaseId/relationships/:relationshipId",
() => HttpResponse.json({ code: "relationship_not_found" }, { status: 404 }),
));
renderRelationships([generated]);
await user.click(await screen.findByRole("button", {
name: "View relationship orders.user_id to users.id",
}));
const drawer = await screen.findByRole("dialog", { name: "Relationship details" });
await user.click(within(drawer).getByRole("button", { name: "Exclude" }));
await user.click(within(drawer).getByRole("button", { name: "Exclude relationship" }));
expect(await screen.findByText("The relationship no longer exists. Refresh and try again.")).toBeVisible();
expect(screen.getByRole("dialog", { name: "Relationship details" })).toBeVisible();
});
@@ -1,26 +1,19 @@
import { useEffect, useMemo, useRef, useState } from "react";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { useMemo, useRef, useState, type ReactNode } from "react";
import { useQuery } from "@tanstack/react-query";
import { AgGridReact } from "ag-grid-react";
import type { ColDef, ICellRendererParams } from "ag-grid-community";
import { ArrowLeft, History, Info, Play, Plus, RefreshCw } from "lucide-react";
import { ArrowLeft, History, Info, Play, RefreshCw } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../../components/ui/button";
import { ApiError, apiErrorMessage } from "../../api/client";
import { apiErrorMessage } from "../../api/client";
import {
createCatalogRelationship,
deleteCatalogRelationship,
listCatalogRelationships,
listCatalogColumns,
listCatalogTables,
rebuildGeneratedRelationships,
setCatalogRelationshipStatus,
startCatalogSync,
type CatalogDatabase,
type CatalogRelationship,
type CatalogSyncRun,
} from "../../api/catalog-databases";
import { FleetLedgerDrawer } from "./FleetLedgerShell";
import type { DatabaseNavigationState } from "./model";
interface Props {
database: CatalogDatabase;
@@ -30,10 +23,7 @@ interface Props {
onOpenTables: () => void;
onRunStarted: (run: CatalogSyncRun) => void;
onOpenSync: () => void;
onOpenDescriptionHistory?: () => void;
onOpenSensitiveHistory?: () => void;
onCatalogMetricsChanged?: () => void | Promise<void>;
onNavigationStateChange?: (state: DatabaseNavigationState) => void;
metadataModelControl?: ReactNode;
presentation?: "legacy" | "fleet";
}
@@ -42,19 +32,6 @@ interface RelationshipGridContext {
onView: (relationship: CatalogRelationship, origin: HTMLButtonElement) => void;
}
type RelationshipStatusFilter = "active" | "excluded" | "all";
function titleCase(value: string): string {
return value.charAt(0).toUpperCase() + value.slice(1);
}
function relationshipLabel(relationship: CatalogRelationship): string {
const pair = relationship.columns[0];
const source = `${relationship.sourceTableName}${pair ? `.${pair.sourceColumnName}` : ""}`;
const target = `${relationship.targetTableName}${pair ? `.${pair.targetColumnName}` : ""}`;
return `${source} to ${target}`;
}
function RelationshipActionsCell({ data, context }: ICellRendererParams<CatalogRelationship, unknown, RelationshipGridContext>) {
if (!data || !context || context.presentation !== "fleet") return null;
return (
@@ -65,7 +42,7 @@ function RelationshipActionsCell({ data, context }: ICellRendererParams<CatalogR
size="icon-lg"
className="thot-fleet-icon-action"
data-tooltip="Details"
aria-label={`View relationship ${relationshipLabel(data)}`}
aria-label={`View relationship ${data.constraintName}`}
onClick={(event) => context.onView(data, event.currentTarget)}
>
<Info aria-hidden="true" />
@@ -82,86 +59,32 @@ export function DatabaseRelationships({
onOpenTables,
onRunStarted,
onOpenSync,
onOpenDescriptionHistory,
onOpenSensitiveHistory,
onCatalogMetricsChanged,
onNavigationStateChange,
presentation = "legacy",
}: Props) {
const databaseId = database.id!;
const queryClient = useQueryClient();
const { data = [], error, isError, isLoading, isFetching, refetch } = useQuery({
queryKey: ["catalog-relationships", databaseId],
queryFn: () => listCatalogRelationships(databaseId),
retry: false,
});
const [search, setSearch] = useState("");
const [statusFilter, setStatusFilter] = useState<RelationshipStatusFilter>("active");
const [busy, setBusy] = useState(false);
const [selectedAction, setSelectedAction] = useState<"" | "rebuild-generated" | "relationships" | "all">("");
const [selectedAction, setSelectedAction] = useState<"" | "relationships" | "all">("");
const [activeRelationshipId, setActiveRelationshipId] = useState<string | null>(null);
const [addOpen, setAddOpen] = useState(false);
const [sourceTableId, setSourceTableId] = useState("");
const [sourceColumnId, setSourceColumnId] = useState("");
const [targetTableId, setTargetTableId] = useState("");
const [targetColumnId, setTargetColumnId] = useState("");
const [addError, setAddError] = useState<string | null>(null);
const [targetColumnError, setTargetColumnError] = useState<string | null>(null);
const [pendingMutation, setPendingMutation] = useState<"exclude" | "delete" | null>(null);
const originRef = useRef<HTMLButtonElement | null>(null);
const statusFilterRef = useRef<HTMLSelectElement | null>(null);
const bindingReady = database.connectionStatus === "reachable" && database.testedVersion === database.version;
const activeRelationship = activeRelationshipId
? data.find((relationship) => relationship.id === activeRelationshipId)
: undefined;
const visibleRelationships = useMemo(() => statusFilter === "all"
? data
: data.filter((relationship) => relationship.status === statusFilter), [data, statusFilter]);
const tablesQuery = useQuery({
queryKey: ["catalog-tables", databaseId],
queryFn: () => listCatalogTables(databaseId),
enabled: addOpen,
retry: false,
});
const sourceColumnsQuery = useQuery({
queryKey: ["catalog-columns", databaseId, sourceTableId],
queryFn: () => listCatalogColumns(databaseId, sourceTableId),
enabled: addOpen && Boolean(sourceTableId),
retry: false,
});
const targetColumnsQuery = useQuery({
queryKey: ["catalog-columns", databaseId, targetTableId],
queryFn: () => listCatalogColumns(databaseId, targetTableId),
enabled: addOpen && Boolean(targetTableId),
retry: false,
});
const targetColumns = (targetColumnsQuery.data ?? []).filter((column) => column.isPrimaryKey);
useEffect(() => {
if (!pendingMutation) return;
const timer = window.setTimeout(() => document
.getElementById("relationship-mutation-confirm-button")?.focus(), 0);
return () => window.clearTimeout(timer);
}, [pendingMutation]);
useEffect(() => {
onNavigationStateChange?.({ dirty: false, busy });
return () => onNavigationStateChange?.({ dirty: false, busy: false });
}, [busy, onNavigationStateChange]);
const selectedActionUnavailable = !selectedAction
? null
: !canManage
? "You do not have permission to manage catalog relationships."
: database.activeSyncRun
? "Wait for the active synchronization to finish."
: selectedAction !== "rebuild-generated" && !bindingReady
? "Test the current database binding first."
? "You do not have permission to synchronize the catalog."
: !bindingReady
? "Test the current database binding first."
: database.activeSyncRun
? "Wait for the active synchronization to finish."
: null;
const addUnavailable = !canManage
? "You do not have permission to add catalog relationships."
: database.activeSyncRun
? "Wait for the active synchronization to finish."
: busy
? "Wait for the current relationship operation to finish."
: null;
const synchronize = async (scope: "relationships" | "all") => {
setBusy(true);
@@ -174,146 +97,28 @@ export function DatabaseRelationships({
const openRelationship = (relationship: CatalogRelationship, origin: HTMLButtonElement) => {
originRef.current = origin;
setPendingMutation(null);
setAddOpen(false);
setActiveRelationshipId(relationship.id);
};
const closeRelationship = () => {
if (busy) return;
setPendingMutation(null);
setActiveRelationshipId(null);
window.setTimeout(() => {
if (originRef.current?.isConnected) originRef.current.focus();
else statusFilterRef.current?.focus();
}, 0);
};
const changeRelationshipStatus = async (status: "active" | "excluded") => {
if (!activeRelationship || activeRelationship.origin === "physical") return;
setBusy(true);
try {
await setCatalogRelationshipStatus(databaseId, activeRelationship.id, status);
await queryClient.invalidateQueries({ queryKey: ["catalog-relationships", databaseId], exact: true });
await onCatalogMetricsChanged?.();
setPendingMutation(null);
setActiveRelationshipId(null);
window.setTimeout(() => {
if (originRef.current?.isConnected) originRef.current.focus();
else statusFilterRef.current?.focus();
}, 0);
toast.success(status === "excluded"
? "Relationship excluded. Future rebuilds will leave it excluded."
: "Relationship restored.");
} catch (error) {
toast.error(apiErrorMessage(error));
} finally {
setBusy(false);
}
};
const permanentlyDeleteRelationship = async () => {
if (!activeRelationship || activeRelationship.origin === "physical") return;
setBusy(true);
try {
await deleteCatalogRelationship(databaseId, activeRelationship.id);
await queryClient.invalidateQueries({ queryKey: ["catalog-relationships", databaseId], exact: true });
await onCatalogMetricsChanged?.();
setPendingMutation(null);
setActiveRelationshipId(null);
window.setTimeout(() => {
if (originRef.current?.isConnected) originRef.current.focus();
else statusFilterRef.current?.focus();
}, 0);
toast.success("Relationship deleted permanently. A future rebuild may infer it again.");
} catch (error) {
toast.error(apiErrorMessage(error));
} finally {
setBusy(false);
}
};
const resetAddForm = () => {
setSourceTableId("");
setSourceColumnId("");
setTargetTableId("");
setTargetColumnId("");
setAddError(null);
setTargetColumnError(null);
};
const closeAdd = () => {
if (busy) return;
setAddOpen(false);
resetAddForm();
};
const addRelationship = async () => {
if (!sourceColumnId || !targetColumnId) return;
setBusy(true);
setAddError(null);
setTargetColumnError(null);
try {
await createCatalogRelationship(databaseId, sourceColumnId, targetColumnId);
await queryClient.invalidateQueries({ queryKey: ["catalog-relationships", databaseId], exact: true });
await onCatalogMetricsChanged?.();
setAddOpen(false);
resetAddForm();
toast.success("Relationship added.");
} catch (error) {
if (error instanceof ApiError && error.code === "relationship_target_not_unique") {
setTargetColumnError("The target column must be the only primary-key column of its table.");
} else if (error instanceof ApiError && error.code === "relationship_type_incompatible") {
setTargetColumnError("Source and target column types are not compatible.");
} else if (error instanceof ApiError && error.code === "relationship_duplicate") {
setAddError("This relationship already exists.");
} else {
setAddError(apiErrorMessage(error));
}
toast.error(apiErrorMessage(error));
} finally {
setBusy(false);
}
window.setTimeout(() => originRef.current?.focus(), 0);
};
const runSelectedAction = async () => {
if (!selectedAction) return;
if (selectedAction === "rebuild-generated") {
setBusy(true);
try {
const result = await rebuildGeneratedRelationships(databaseId);
await queryClient.invalidateQueries({ queryKey: ["catalog-relationships", databaseId], exact: true });
await onCatalogMetricsChanged?.();
const noFindings = result.added === 0
&& result.alreadyPresent === 0
&& result.excluded === 0
&& result.ambiguous === 0;
toast.success(noFindings
? "Rebuild complete: no new relationships found."
: `Rebuild complete: ${result.added} added, ${result.alreadyPresent} already present, ${result.excluded} excluded, ${result.ambiguous} ambiguous.`);
} catch (error) {
toast.error(apiErrorMessage(error));
} finally {
setBusy(false);
}
} else {
await synchronize(selectedAction);
}
await synchronize(selectedAction);
setSelectedAction("");
};
const columns = useMemo<ColDef<CatalogRelationship>[]>(() => [
...(presentation === "legacy" ? [{ field: "constraintName", headerName: "Constraint", minWidth: 220, flex: 1, cellClass: "font-mono text-xs" } satisfies ColDef<CatalogRelationship>] : []),
{ field: "constraintName", headerName: "Constraint", minWidth: 220, flex: 1, cellClass: "font-mono text-xs" },
{ field: "sourceTableName", headerName: "Source table", minWidth: 200, flex: 1 },
{ headerName: "Source column", minWidth: 180, flex: 1, valueGetter: ({ data: row }) => row?.columns.map((pair) => pair.sourceColumnName).join(", ") ?? "" },
{ headerName: "Source columns", minWidth: 200, flex: 1, valueGetter: ({ data: row }) => row?.columns.map((pair) => pair.sourceColumnName).join(", ") ?? "" },
{ field: "targetTableName", headerName: "Target table", minWidth: 200, flex: 1 },
{ headerName: "Target column", minWidth: 180, flex: 1, valueGetter: ({ data: row }) => row?.columns.map((pair) => pair.targetColumnName).join(", ") ?? "" },
{ field: "origin", headerName: "Origin", minWidth: 120, width: 120, valueFormatter: ({ value }) => titleCase(String(value ?? "")) },
{ field: "status", headerName: "Status", minWidth: 110, width: 110, valueFormatter: ({ value }) => titleCase(String(value ?? "")) },
...(presentation === "legacy" ? [
{ field: "updateRule", headerName: "On update", minWidth: 125, width: 125 },
{ field: "deleteRule", headerName: "On delete", minWidth: 125, width: 125 },
] satisfies ColDef<CatalogRelationship>[] : []),
{ headerName: "Target columns", minWidth: 200, flex: 1, valueGetter: ({ data: row }) => row?.columns.map((pair) => pair.targetColumnName).join(", ") ?? "" },
{ field: "updateRule", headerName: "On update", minWidth: 125, width: 125 },
{ field: "deleteRule", headerName: "On delete", minWidth: 125, width: 125 },
...(presentation === "fleet" ? [{ colId: "actions", headerName: "Actions", width: 64, minWidth: 64, maxWidth: 64, pinned: "right" as const, lockPinned: true, sortable: false, filter: false, resizable: false, suppressMovable: true, suppressHeaderMenuButton: true, cellRenderer: RelationshipActionsCell }] : []),
], [presentation]);
const context = useMemo<RelationshipGridContext>(() => ({ presentation, onView: openRelationship }), [data, presentation]);
@@ -352,27 +157,14 @@ export function DatabaseRelationships({
</div> : null}
<div className={presentation === "fleet" ? "thot-fleet-grid-surface flex min-h-0 flex-1 flex-col overflow-hidden bg-card" : "mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5"}>
<div className={`${presentation === "fleet" ? "thot-fleet-grid-toolbar " : ""}flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2`}>
<span className="thot-label whitespace-nowrap">Relationship map</span>
<label className="sr-only" htmlFor="relationship-status-filter">Relationship status</label>
<select
ref={statusFilterRef}
id="relationship-status-filter"
className="h-8 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15"
aria-label="Relationship status"
value={statusFilter}
onChange={(event) => setStatusFilter(event.target.value as RelationshipStatusFilter)}
>
<option value="active">Active</option>
<option value="excluded">Excluded</option>
<option value="all">All</option>
</select>
<span className="thot-label whitespace-nowrap">Physical relationships</span>
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search relationships" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
<span className="text-xs tabular-nums text-muted-foreground">{visibleRelationships.length}</span>
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
{presentation === "fleet" ? (
<div className="flex flex-wrap items-center gap-2">
<div className="thot-fleet-action-selector__scope" aria-live="polite">
<strong>{selectedAction === "relationships" ? "Physical relationships in this database" : "This database"}</strong>
<strong>{selectedAction === "all" ? "This database" : selectedAction === "relationships" ? "All relationships in this database" : "This database"}</strong>
<span>Action scope</span>
</div>
<label className="sr-only" htmlFor="relationship-action">Relationship action</label>
@@ -381,212 +173,45 @@ export function DatabaseRelationships({
className="thot-fleet-action-selector__select"
value={selectedAction}
disabled={busy}
onChange={(event) => setSelectedAction(event.target.value as "" | "rebuild-generated" | "relationships" | "all")}
onChange={(event) => setSelectedAction(event.target.value as "" | "relationships" | "all")}
>
<option value="">Choose an action…</option>
<optgroup label="Logical relationships">
<option value="rebuild-generated">Rebuild generated relationships</option>
</optgroup>
<optgroup label="Synchronization">
<option value="relationships">Synchronize physical relationships</option>
<option value="all">Synchronize the full database schema</option>
<option value="relationships">Synchronize all relationships in this database</option>
<option value="all">Synchronize the full schema for this database</option>
</optgroup>
</select>
<Button type="button" size="sm" disabled={!selectedAction || Boolean(selectedActionUnavailable) || busy} aria-describedby={selectedActionUnavailable ? "relationship-action-reason" : undefined} onClick={() => void runSelectedAction()}><Play aria-hidden="true" />{busy ? selectedAction === "rebuild-generated" ? "Rebuilding…" : "Running…" : "Run action"}</Button>
<Button type="button" size="sm" disabled={!selectedAction || Boolean(selectedActionUnavailable) || busy} aria-describedby={selectedActionUnavailable ? "relationship-action-reason" : undefined} onClick={() => void runSelectedAction()}><Play aria-hidden="true" />{busy ? "Running…" : "Run action"}</Button>
<Button type="button" size="sm" variant="outline" className="thot-fleet-nav-action thot-fleet-nav-action--below" data-tooltip="Synchronization history" onClick={onOpenSync}>
{database.activeSyncRun ? <RefreshCw className="animate-spin" aria-hidden="true" /> : <History aria-hidden="true" />} Sync history
</Button>
{onOpenDescriptionHistory ? <Button type="button" size="sm" variant="outline" onClick={onOpenDescriptionHistory}><History aria-hidden="true" /> Description history</Button> : null}
{onOpenSensitiveHistory ? <Button type="button" size="sm" variant="outline" onClick={onOpenSensitiveHistory}><History aria-hidden="true" /> Sensitive history</Button> : null}
<Button
type="button"
size="sm"
disabled={Boolean(addUnavailable)}
aria-describedby={addUnavailable ? "relationship-add-reason" : undefined}
onClick={() => {
setActiveRelationshipId(null);
setPendingMutation(null);
setAddOpen(true);
}}
>
<Plus aria-hidden="true" /> Add relationship
</Button>
{addUnavailable ? <span id="relationship-add-reason" className="sr-only">{addUnavailable}</span> : null}
{selectedActionUnavailable ? <span id="relationship-action-reason" className="basis-full text-xs text-muted-foreground" role="status">{selectedActionUnavailable}</span> : null}
</div>
) : <Button type="button" disabled={!canManage || !bindingReady || busy || Boolean(database.activeSyncRun)} title={!bindingReady ? "Test the current database binding before synchronizing relationships" : undefined} onClick={() => void synchronize("relationships")}><RefreshCw />Synchronize relationships</Button>}
) : <Button type="button" disabled={!canManage || !bindingReady || busy || Boolean(database.activeSyncRun)} title={!bindingReady ? "Test the current database binding before synchronizing relationships" : undefined} onClick={() => void synchronize("relationships")}><RefreshCw />Sync relationships</Button>}
</div>
{!bindingReady ? <div className="border-b border-border bg-muted/35 px-4 py-3 text-sm text-muted-foreground">{presentation === "fleet" ? "Test the current database binding in database details before synchronizing relationships." : "Test the current database binding from Overview before synchronizing relationships."}</div> : null}
{fleetQueryError ?? <div className="relative min-h-[280px] flex-1">
<div className={`thot-database-grid ${presentation === "fleet" ? "thot-fleet-ledger-grid " : ""}ag-theme-alpine absolute inset-0 h-full w-full`}>
<AgGridReact<CatalogRelationship> rowData={visibleRelationships} columnDefs={columns} context={context} loading={isLoading} quickFilterText={search} defaultColDef={{ sortable: true, filter: true, resizable: true }} getRowId={({ data: row }) => row.id} rowHeight={44} headerHeight={38} animateRows={false} overlayNoRowsTemplate={statusFilter === "active" ? "No relationships yet. Rebuild from column names or add one manually." : statusFilter === "excluded" ? "No excluded relationships." : "No relationships for this database."} />
<AgGridReact<CatalogRelationship> rowData={data} columnDefs={columns} context={context} loading={isLoading} quickFilterText={search} defaultColDef={{ sortable: true, filter: true, resizable: true }} getRowId={({ data: row }) => row.id} rowHeight={44} headerHeight={38} animateRows={false} overlayNoRowsTemplate="No physical relationships synchronized for this database." />
</div>
</div>}
</div>
{presentation === "fleet" && addOpen ? (
<FleetLedgerDrawer
open
eyebrow="Logical relationship"
title="Add relationship"
description="Connect one source column to one primary-key target column."
onClose={closeAdd}
closeLabel="Close relationship form"
busy={busy}
footer={(
<>
<Button type="button" variant="outline" disabled={busy} onClick={closeAdd}>Cancel</Button>
<Button
type="button"
disabled={busy || !sourceTableId || !sourceColumnId || !targetTableId || !targetColumnId}
onClick={() => void addRelationship()}
>
{busy ? "Adding…" : "Add relationship"}
</Button>
</>
)}
>
<div className="grid gap-4 text-sm sm:grid-cols-2">
<label className="grid gap-1.5">
<span className="font-semibold">Source table</span>
<select
className="h-9 rounded-md border border-input bg-background px-3 outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15"
aria-label="Source table"
value={sourceTableId}
disabled={busy || tablesQuery.isLoading}
onChange={(event) => {
setSourceTableId(event.target.value);
setSourceColumnId("");
setAddError(null);
}}
>
<option value="">Select source table…</option>
{(tablesQuery.data ?? []).map((table) => <option key={table.id} value={table.id}>{table.name}</option>)}
</select>
</label>
<label className="grid gap-1.5">
<span className="font-semibold">Source column</span>
<select
className="h-9 rounded-md border border-input bg-background px-3 outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15"
aria-label="Source column"
value={sourceColumnId}
disabled={busy || !sourceTableId || sourceColumnsQuery.isLoading}
onChange={(event) => {
setSourceColumnId(event.target.value);
setAddError(null);
setTargetColumnError(null);
}}
>
<option value="">Select source column…</option>
{(sourceColumnsQuery.data ?? []).map((column) => <option key={column.id} value={column.id}>{column.name}</option>)}
</select>
</label>
<label className="grid gap-1.5">
<span className="font-semibold">Target table</span>
<select
className="h-9 rounded-md border border-input bg-background px-3 outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15"
aria-label="Target table"
value={targetTableId}
disabled={busy || tablesQuery.isLoading}
onChange={(event) => {
setTargetTableId(event.target.value);
setTargetColumnId("");
setAddError(null);
setTargetColumnError(null);
}}
>
<option value="">Select target table…</option>
{(tablesQuery.data ?? []).map((table) => <option key={table.id} value={table.id}>{table.name}</option>)}
</select>
</label>
<label className="grid gap-1.5">
<span className="font-semibold">Target column</span>
<select
className="h-9 rounded-md border border-input bg-background px-3 outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15"
aria-label="Target column"
aria-invalid={Boolean(targetColumnError)}
aria-describedby={targetColumnError ? "target-column-error" : undefined}
value={targetColumnId}
disabled={busy || !targetTableId || targetColumnsQuery.isLoading}
onChange={(event) => {
setTargetColumnId(event.target.value);
setAddError(null);
setTargetColumnError(null);
}}
>
<option value="">Select target column…</option>
{targetColumns.map((column) => <option key={column.id} value={column.id}>{column.name}</option>)}
</select>
{targetColumnError ? <span id="target-column-error" className="text-xs text-destructive" role="alert">{targetColumnError}</span> : null}
</label>
{tablesQuery.isError || sourceColumnsQuery.isError || targetColumnsQuery.isError ? (
<p className="sm:col-span-2 text-sm text-destructive" role="alert">Catalog tables or columns could not be loaded.</p>
) : null}
{addError ? <p className="sm:col-span-2 text-sm text-destructive" role="alert">{addError}</p> : null}
</div>
</FleetLedgerDrawer>
) : null}
{presentation === "fleet" && activeRelationship ? (
<FleetLedgerDrawer
open
eyebrow={`${titleCase(activeRelationship.origin)} relationship`}
eyebrow="Physical relationship"
title="Relationship details"
description={activeRelationship.origin === "physical"
? "This information is synchronized from the source schema and is read-only."
: activeRelationship.origin === "generated"
? "This relationship was inferred from catalog column names."
: "This relationship was added manually."}
description="This information is synchronized from the source schema and is read-only."
onClose={closeRelationship}
closeLabel="Close relationship details"
busy={busy}
footer={pendingMutation ? (
<div
className="flex min-w-0 flex-1 flex-wrap items-center gap-2"
role="group"
aria-labelledby="relationship-mutation-confirmation"
onKeyDown={(event) => {
if (event.key === "Escape" && !busy) setPendingMutation(null);
}}
>
<p id="relationship-mutation-confirmation" className="mr-auto max-w-xl text-sm text-muted-foreground">
{pendingMutation === "exclude"
? "The relationship will move to Excluded. Rebuilds will not recreate it, and you can restore it later."
: "The relationship record will be erased. A future rebuild may infer it again."}
</p>
<Button type="button" variant="ghost" disabled={busy} onClick={() => setPendingMutation(null)}>Cancel</Button>
<Button
id="relationship-mutation-confirm-button"
type="button"
variant={pendingMutation === "delete" ? "destructive" : "default"}
disabled={busy}
onClick={() => void (pendingMutation === "exclude"
? changeRelationshipStatus("excluded")
: permanentlyDeleteRelationship())}
>
{pendingMutation === "exclude" ? "Exclude relationship" : "Delete permanently"}
</Button>
</div>
) : (
<>
<Button type="button" variant="outline" disabled={busy} onClick={closeRelationship}>Close</Button>
{activeRelationship.origin !== "physical" && activeRelationship.status === "active" ? (
<Button type="button" variant="outline" disabled={!canManage || busy} onClick={() => setPendingMutation("exclude")}>Exclude</Button>
) : null}
{activeRelationship.origin !== "physical" && activeRelationship.status === "excluded" ? (
<Button type="button" disabled={!canManage || busy} onClick={() => void changeRelationshipStatus("active")}>Restore</Button>
) : null}
{activeRelationship.origin !== "physical" ? (
<Button type="button" variant="destructive" disabled={!canManage || busy} onClick={() => setPendingMutation("delete")}>Delete permanently</Button>
) : null}
</>
)}
footer={<Button type="button" variant="outline" onClick={closeRelationship}>Close</Button>}
>
<dl className="grid gap-4 text-sm sm:grid-cols-2">
<div><dt className="thot-label">Origin</dt><dd className="mt-1">{titleCase(activeRelationship.origin)}</dd></div>
<div><dt className="thot-label">Status</dt><dd className="mt-1">{titleCase(activeRelationship.status)}</dd></div>
{activeRelationship.constraintName ? <div className="sm:col-span-2"><dt className="thot-label">Constraint</dt><dd className="mt-1 break-words font-mono text-xs">{activeRelationship.constraintName}</dd></div> : null}
<div className="sm:col-span-2"><dt className="thot-label">Constraint</dt><dd className="mt-1 break-words font-mono text-xs">{activeRelationship.constraintName}</dd></div>
<div><dt className="thot-label">Source table</dt><dd className="mt-1 break-words">{activeRelationship.sourceTableName}</dd></div>
<div><dt className="thot-label">Target table</dt><dd className="mt-1 break-words">{activeRelationship.targetTableName}</dd></div>
{activeRelationship.origin === "physical" ? <><div><dt className="thot-label">On update</dt><dd className="mt-1">{activeRelationship.updateRule ?? "Not specified"}</dd></div><div><dt className="thot-label">On delete</dt><dd className="mt-1">{activeRelationship.deleteRule ?? "Not specified"}</dd></div></> : null}
<div><dt className="thot-label">On update</dt><dd className="mt-1">{activeRelationship.updateRule ?? "Not specified"}</dd></div>
<div><dt className="thot-label">On delete</dt><dd className="mt-1">{activeRelationship.deleteRule ?? "Not specified"}</dd></div>
<div className="sm:col-span-2">
<dt className="thot-label">Column mapping</dt>
<dd className="mt-2 overflow-hidden rounded-md border border-border">
@@ -3,15 +3,14 @@ import { ChevronDown, RefreshCw } from "lucide-react";
import { buttonVariants } from "../../components/ui/button";
import type { CatalogSyncScope } from "../../api/catalog-databases";
export type DatabaseSyncScope = Exclude<CatalogSyncScope, "columns">;
export const databaseSyncItemClass = [
"flex cursor-default select-none items-center rounded-md px-3 py-2 text-sm outline-none",
"data-[highlighted]:bg-muted data-[disabled]:opacity-45",
].join(" ");
export const databaseSyncScopes: ReadonlyArray<{ scope: DatabaseSyncScope; label: string }> = [
export const databaseSyncScopes: ReadonlyArray<{ scope: CatalogSyncScope; label: string }> = [
{ scope: "tables", label: "Synchronize tables" },
{ scope: "columns", label: "Synchronize all columns" },
{ scope: "relationships", label: "Synchronize relationships" },
{ scope: "all", label: "Synchronize all" },
];
@@ -19,7 +18,7 @@ export const databaseSyncScopes: ReadonlyArray<{ scope: DatabaseSyncScope; label
interface DatabaseSyncMenuProps {
disabled: boolean;
disabledReason?: string;
onSelect: (scope: DatabaseSyncScope) => void;
onSelect: (scope: CatalogSyncScope) => void;
}
export function DatabaseSyncMenu({ disabled, disabledReason, onSelect }: DatabaseSyncMenuProps) {
@@ -1,16 +1,15 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { useEffect, useMemo, useRef, useState, type ReactNode } from "react";
import { Menu } from "@base-ui/react/menu";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { AgGridReact } from "ag-grid-react";
import type { ColDef, ICellRendererParams } from "ag-grid-community";
import { ArrowLeft, ChevronDown, Columns3, Pencil, RefreshCw, Save, Sparkles, Trash2, X } from "lucide-react";
import { ArrowLeft, ChevronDown, Columns3, History, Pencil, RefreshCw, Save, Sparkles, Trash2, X } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../../components/ui/button";
import { ApiError, apiErrorMessage } from "../../api/client";
import {
consolidateCatalogDescriptions,
deleteCatalogTableMetadata,
listCatalogColumns,
listCatalogTables,
startCatalogSync,
startDescriptionGenerationRun,
@@ -26,7 +25,6 @@ import type { DatabaseNavigationState } from "./model";
import { DatabaseColumns } from "./DatabaseColumns";
import { FleetActionSelector, type FleetActionOption } from "./FleetActionSelector";
import { FleetLedgerDrawer } from "./FleetLedgerShell";
import { NO_METADATA_GENERATION_LLM_MODEL_MESSAGE } from "./MetadataGenerationModelSelector";
interface Props {
database: CatalogDatabase;
@@ -38,13 +36,13 @@ interface Props {
onOpenOverview: () => void;
onOpenRelationships: () => void;
onNavigationStateChange: (state: DatabaseNavigationState) => void;
onNestedNavigationChange?: (active: boolean, onBack?: () => void) => void;
onNestedNavigationChange?: (active: boolean) => void;
onRunStarted: (run: CatalogSyncRun) => void;
onOpenSync: () => void;
onClearCatalogTables?: () => Promise<void>;
onDescriptionGenerationRunStarted: (run: DescriptionGenerationRun) => void;
onSuggestSensitive: (selection: SensitiveDataSuggestionRequest, scopeLabel: string) => Promise<void>;
onCatalogMetricsChanged?: () => void | Promise<void>;
metadataModelControl?: ReactNode;
presentation?: "legacy" | "fleet";
}
@@ -95,10 +93,10 @@ export function DatabaseTables({
onNestedNavigationChange,
onRunStarted,
onOpenSync,
onClearCatalogTables,
onDescriptionGenerationRunStarted,
onSuggestSensitive,
onCatalogMetricsChanged,
metadataModelControl,
presentation = "legacy",
}: Props) {
const databaseId = database.id!;
@@ -133,22 +131,9 @@ export function DatabaseTables({
const nestedNavigationActive = presentation === "fleet" && Boolean(activeTable) && tableSection === "columns";
const bindingReady = database.connectionStatus === "reachable" && database.testedVersion === database.version;
const currentRun = activeRun ?? database.activeSyncRun;
const clearAllCatalogTables = async () => {
if (!onClearCatalogTables || !window.confirm("Clear all catalog tables, columns, and relationships for this database? The source database will not be changed.")) return;
await onClearCatalogTables();
};
const leaveTable = useCallback(() => {
if (navigationBusy) return;
if (dirty && !window.confirm("Discard unsaved metadata?")) return;
setActiveTableId(null);
setTableSection("overview");
window.setTimeout(() => originRef.current?.focus(), 0);
}, [dirty, navigationBusy]);
useEffect(() => { onNavigationStateChange({ dirty, busy: navigationBusy }); }, [dirty, navigationBusy, onNavigationStateChange]);
useEffect(() => {
onNestedNavigationChange?.(nestedNavigationActive, nestedNavigationActive ? leaveTable : undefined);
}, [leaveTable, nestedNavigationActive, onNestedNavigationChange]);
useEffect(() => { onNestedNavigationChange?.(nestedNavigationActive); }, [nestedNavigationActive, onNestedNavigationChange]);
useEffect(() => () => { onNestedNavigationChange?.(false); }, [onNestedNavigationChange]);
useEffect(() => {
if (!activeTableId || !activeTable || editorVersion === activeTable.version || busy) return;
@@ -180,6 +165,13 @@ export function DatabaseTables({
setStaleBannerOpen(true);
setColumnNavigation({ dirty: false, busy: false });
};
const leaveTable = () => {
if (navigationBusy) return;
if (dirty && !window.confirm("Discard unsaved metadata?")) return;
setActiveTableId(null);
setTableSection("overview");
window.setTimeout(() => originRef.current?.focus(), 0);
};
const navigateDatabase = (target: "databases" | "overview" | "relationships") => {
if (navigationBusy) return;
if (dirty && !window.confirm("Discard unsaved metadata?")) return;
@@ -286,28 +278,6 @@ export function DatabaseTables({
toast.error(apiErrorMessage(error));
} finally { setBusy(null); }
};
const generateColumnDescriptions = async () => {
if (selectedIds.length === 0 || !selectedMetadataModel) return;
setBusy("generate");
try {
const columns = (await Promise.all(selectedIds.map((tableId) => listCatalogColumns(databaseId, tableId)))).flat();
const columnIds = columns.map((column) => column.id);
if (columnIds.length === 0) {
toast.error("The selected tables have no synchronized catalog columns.");
return;
}
const run = await startDescriptionGenerationRun(
databaseId,
selectedMetadataModel,
"selected_columns",
columnIds,
);
onDescriptionGenerationRunStarted(run);
toast.success(`Column description generation started for ${columnIds.length} column${columnIds.length === 1 ? "" : "s"}`);
} catch (error) {
toast.error(apiErrorMessage(error));
} finally { setBusy(null); }
};
const suggestSensitive = async () => {
if (selectedIds.length === 0) return;
setBusy("suggest");
@@ -325,17 +295,17 @@ export function DatabaseTables({
type FleetTableAction =
| "generate-descriptions"
| "generate-column-descriptions"
| "sync-tables"
| "sync-columns"
| "consolidate-descriptions"
| "suggest-sensitive"
| "clear-columns";
| "clear-columns"
| "clear-relationships";
const fleetActions: readonly FleetActionOption<FleetTableAction>[] = [
{
id: "generate-descriptions",
label: "Generate table description",
label: "Generate descriptions",
group: "Descriptions",
runLabel: "Generate",
disabled: !canManage || selectedIds.length === 0 || !selectedMetadataModel || descriptionGenerationActive || busy !== null || Boolean(currentRun),
@@ -344,25 +314,7 @@ export function DatabaseTables({
: selectedIds.length === 0
? "Select at least one table."
: !selectedMetadataModel
? NO_METADATA_GENERATION_LLM_MODEL_MESSAGE
: descriptionGenerationActive || Boolean(currentRun)
? "Wait for the active catalog operation to finish."
: busy !== null
? "Another action is running."
: undefined,
},
{
id: "generate-column-descriptions",
label: "Generate column descriptions",
group: "Descriptions",
runLabel: "Generate",
disabled: !canManage || selectedIds.length === 0 || !selectedMetadataModel || descriptionGenerationActive || busy !== null || Boolean(currentRun),
disabledReason: !canManage
? "You do not have permission to generate descriptions."
: selectedIds.length === 0
? "Select at least one table."
: !selectedMetadataModel
? NO_METADATA_GENERATION_LLM_MODEL_MESSAGE
? "Choose a metadata model first."
: descriptionGenerationActive || Boolean(currentRun)
? "Wait for the active catalog operation to finish."
: busy !== null
@@ -371,7 +323,7 @@ export function DatabaseTables({
},
{
id: "consolidate-descriptions",
label: "Copy generated descriptions to Description",
label: "Move generated to Description",
group: "Descriptions",
runLabel: "Move",
disabled: !canManage || selectedIds.length === 0 || busy !== null || Boolean(currentRun),
@@ -385,24 +337,22 @@ export function DatabaseTables({
},
{
id: "sync-tables",
label: "Synchronize database tables",
label: "Synchronize tables",
group: "Synchronization",
scopeLabel: "Selected database",
scopeLabel: "All tables in this database",
runLabel: "Synchronize",
disabled: !canManage || selectedIds.length === 0 || !bindingReady || busy !== null || Boolean(currentRun),
disabled: !canManage || !bindingReady || busy !== null || Boolean(currentRun),
disabledReason: !canManage
? "You do not have permission to synchronize the catalog."
: selectedIds.length === 0
? "Select at least one table."
: !bindingReady
? "Test the current database binding first."
: busy !== null || Boolean(currentRun)
? "Wait for the active catalog operation to finish."
: undefined,
: !bindingReady
? "Test the current database binding first."
: busy !== null || Boolean(currentRun)
? "Wait for the active catalog operation to finish."
: undefined,
},
{
id: "sync-columns",
label: "Synchronize columns for selected tables",
label: "Synchronize columns",
group: "Synchronization",
runLabel: "Synchronize",
disabled: !canManage || selectedIds.length === 0 || !bindingReady || busy !== null || Boolean(currentRun),
@@ -427,7 +377,7 @@ export function DatabaseTables({
: selectedIds.length === 0
? "Select at least one table."
: !selectedMetadataModel
? NO_METADATA_GENERATION_LLM_MODEL_MESSAGE
? "Choose a metadata model first."
: descriptionGenerationActive || Boolean(currentRun)
? "Wait for the active catalog operation to finish."
: busy !== null
@@ -436,7 +386,22 @@ export function DatabaseTables({
},
{
id: "clear-columns",
label: "Remove synchronized columns",
label: "Clear catalog columns",
group: "Catalog cleanup",
tone: "destructive",
runLabel: "Review cleanup",
disabled: !canManage || selectedIds.length === 0 || busy !== null || Boolean(currentRun),
disabledReason: !canManage
? "You do not have permission to clear catalog metadata."
: selectedIds.length === 0
? "Select at least one table."
: busy !== null || Boolean(currentRun)
? "Wait for the active catalog operation to finish."
: undefined,
},
{
id: "clear-relationships",
label: "Clear catalog relationships",
group: "Catalog cleanup",
tone: "destructive",
runLabel: "Review cleanup",
@@ -453,12 +418,12 @@ export function DatabaseTables({
const runFleetAction = async (action: FleetTableAction) => {
if (action === "generate-descriptions") await generateDescriptions();
else if (action === "generate-column-descriptions") await generateColumnDescriptions();
else if (action === "sync-tables") await synchronize("tables");
else if (action === "consolidate-descriptions") await consolidateDescriptions();
else if (action === "sync-columns") await synchronize("columns", selectedIds);
else if (action === "suggest-sensitive") await suggestSensitive();
else if (action === "clear-columns") setPendingDelete("columns");
else setPendingDelete("relationships");
};
const clearSelection = () => {
@@ -496,7 +461,7 @@ export function DatabaseTables({
const databaseHeader = (
<div className="px-4 pt-5 sm:px-5">
<Button type="button" variant="ghost" className="-ml-2 mb-4" disabled={navigationBusy} onClick={() => navigateDatabase("databases")}><ArrowLeft />Back to databases</Button>
<div className="flex flex-wrap items-start justify-between gap-4 border-b border-border pb-5">
<div className="flex flex-wrap items-start justify-between gap-4 border-b border-border pb-5">
<div>
<p className="thot-label mb-1">Database management</p>
<h2 ref={headingRef} tabIndex={-1} className="font-heading text-2xl font-semibold tracking-tight outline-none">{database.workspaceName}</h2>
@@ -504,7 +469,6 @@ export function DatabaseTables({
</div>
<div className="flex gap-2">
{currentRun ? <Button type="button" variant="outline" onClick={onOpenSync}><RefreshCw className="animate-spin" />View sync</Button> : null}
{onClearCatalogTables ? <Button type="button" variant="outline" className="text-destructive" disabled={!canManage || busy !== null || Boolean(currentRun)} onClick={() => void clearAllCatalogTables()}>Clear catalog tables</Button> : null}
<Button type="button" disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)} onClick={() => void synchronize("all")}><RefreshCw />Sync all</Button>
</div>
</div>
@@ -521,6 +485,17 @@ export function DatabaseTables({
return (
<section aria-label={`Columns for ${activeTable.name}`} className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border bg-muted/25 px-3 py-2">
<Button
type="button"
variant="outline"
className="thot-fleet-icon-action border-primary/35 bg-primary/8 shadow-sm"
disabled={navigationBusy}
data-tooltip="Tables"
aria-label="Back to tables"
onClick={leaveTable}
>
<ArrowLeft aria-hidden="true" /> Back to tables
</Button>
<div className="min-w-0">
<p className="thot-label">Physical table</p>
<h3 className="truncate font-heading text-lg font-semibold">{activeTable.name}</h3>
@@ -529,16 +504,17 @@ export function DatabaseTables({
<DatabaseColumns
databaseId={databaseId}
table={activeTable}
databaseVersion={database.version}
canManage={canManage}
selectedMetadataModel={selectedMetadataModel}
descriptionGenerationActive={descriptionGenerationActive}
onDescriptionGenerationRunStarted={onDescriptionGenerationRunStarted}
onCatalogSyncRunStarted={onRunStarted}
onNavigationStateChange={setColumnNavigation}
onSync={() => void synchronize("columns", [activeTable.id])}
onSuggestSensitive={onSuggestSensitive}
bindingReady={bindingReady}
catalogOperationActive={Boolean(currentRun)}
onCatalogMetricsChanged={onCatalogMetricsChanged}
metadataModelControl={metadataModelControl}
presentation="fleet"
/>
</section>
@@ -581,7 +557,7 @@ export function DatabaseTables({
rowHeight={44}
headerHeight={38}
animateRows={false}
overlayNoRowsTemplate="No catalog tables. Synchronize the schema from the database view."
overlayNoRowsTemplate="No catalog tables. Test the binding, then sync tables."
/>
</div>
</div>}
@@ -642,13 +618,12 @@ export function DatabaseTables({
<DatabaseColumns
databaseId={databaseId}
table={activeTable}
databaseVersion={database.version}
canManage={canManage}
selectedMetadataModel={selectedMetadataModel}
descriptionGenerationActive={descriptionGenerationActive}
onDescriptionGenerationRunStarted={onDescriptionGenerationRunStarted}
onCatalogSyncRunStarted={onRunStarted}
onNavigationStateChange={setColumnNavigation}
onSync={() => void synchronize("columns", [activeTable.id])}
onSuggestSensitive={onSuggestSensitive}
onCatalogMetricsChanged={onCatalogMetricsChanged}
/>
@@ -699,11 +674,19 @@ export function DatabaseTables({
<div className="mr-auto min-w-56">
<p id="table-cleanup-confirmation" className="text-sm font-semibold">
{presentation === "fleet"
? `Remove synchronized columns for ${selectedIds.length} table${selectedIds.length === 1 ? "" : "s"}?`
: `Remove synchronized columns for ${selectedIds.length} table${selectedIds.length === 1 ? "" : "s"}?`}
? `Clear catalog ${pendingDelete} for ${selectedIds.length} table${selectedIds.length === 1 ? "" : "s"}?`
: pendingDelete === "columns"
? `Delete all catalog columns for ${selectedIds.length} table${selectedIds.length === 1 ? "" : "s"}?`
: `Delete all relationships for ${selectedIds.length} table${selectedIds.length === 1 ? "" : "s"}?`}
</p>
<p className="text-xs text-muted-foreground">
Only synchronized catalog columns for these tables will be removed. The source database is not modified.
{presentation === "fleet"
? pendingDelete === "columns"
? "Only synchronized catalog columns for these tables will be cleared. The source database is not modified."
: "Only synchronized catalog relationships for these tables will be cleared. The source database is not modified."
: pendingDelete === "columns"
? "Only columns belonging to the selected catalog tables will be removed."
: "All incoming and outgoing relationships for the selected catalog tables will be removed."}
</p>
</div>
<Button type="button" variant="ghost" disabled={busy !== null} onClick={closeDeleteConfirmation}>Cancel</Button>
@@ -714,7 +697,7 @@ export function DatabaseTables({
disabled={busy !== null}
onClick={() => void deleteMetadata(pendingDelete)}
>
<Trash2 />Remove synchronized columns
<Trash2 />{presentation === "fleet" ? "Clear catalog" : pendingDelete === "columns" ? "Delete columns" : "Delete relationships"}
</Button>
</div>
) : (
@@ -729,6 +712,7 @@ export function DatabaseTables({
selectRef={fleetActionSelectRef}
onRun={runFleetAction}
onClear={clearSelection}
renderContext={(action) => action === "generate-descriptions" || action === "suggest-sensitive" ? metadataModelControl : null}
label="Table action"
/>
</>
@@ -741,13 +725,12 @@ export function DatabaseTables({
<Menu.Portal>
<Menu.Positioner side="bottom" align="start" sideOffset={4}>
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || !selectedMetadataModel || descriptionGenerationActive || busy !== null || Boolean(currentRun)} onClick={() => void generateDescriptions()}>Generate table description</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || !selectedMetadataModel || descriptionGenerationActive || busy !== null || Boolean(currentRun)} onClick={() => void generateColumnDescriptions()}>Generate descriptions for all columns</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)} onClick={() => void synchronize("tables")}>Synchronize database tables</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)} onClick={() => void synchronize("columns", selectedIds)}>Synchronize columns for selected tables</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || busy !== null || Boolean(currentRun)} onClick={() => void consolidateDescriptions()}>Copy generated descriptions to Description</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || !selectedMetadataModel || descriptionGenerationActive || busy !== null || Boolean(currentRun)} onClick={() => void generateDescriptions()}>Generate {selectedIds.length === 1 ? "description" : "descriptions"}</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)} onClick={() => void synchronize("columns", selectedIds)}>Synchronize columns</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canManage || busy !== null || Boolean(currentRun)} onClick={() => void consolidateDescriptions()}>Move generated description to Description</Menu.Item>
<Menu.Separator className="my-1 h-px bg-border" />
<Menu.Item className="rounded-md px-3 py-2 text-sm text-destructive outline-none data-[highlighted]:bg-destructive/10 data-[disabled]:opacity-45" disabled={!canManage || busy !== null || Boolean(currentRun)} onClick={() => setPendingDelete("columns")}>Remove synchronized columns</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm text-destructive outline-none data-[highlighted]:bg-destructive/10 data-[disabled]:opacity-45" disabled={!canManage || busy !== null || Boolean(currentRun)} onClick={() => setPendingDelete("columns")}>Delete all columns</Menu.Item>
<Menu.Item className="rounded-md px-3 py-2 text-sm text-destructive outline-none data-[highlighted]:bg-destructive/10 data-[disabled]:opacity-45" disabled={!canManage || busy !== null || Boolean(currentRun)} onClick={() => setPendingDelete("relationships")}>Delete all relationships</Menu.Item>
</Menu.Popup>
</Menu.Positioner>
</Menu.Portal>
@@ -768,6 +751,7 @@ export function DatabaseTables({
selectRef={fleetActionSelectRef}
onRun={runFleetAction}
onClear={clearSelection}
renderContext={(action) => action === "generate-descriptions" || action === "suggest-sensitive" ? metadataModelControl : null}
label="Table action"
/>
<input ref={searchInputRef} className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search tables" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
@@ -780,11 +764,17 @@ export function DatabaseTables({
<input ref={searchInputRef} className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search tables" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
<span aria-live="polite" className="whitespace-nowrap text-xs tabular-nums text-muted-foreground">{search.trim() ? `${displayedCount} of ${data.length}` : displayedCount}</span>
<Button type="button" variant="outline" disabled={isFetching || busy !== null} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
<Button type="button" disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)} title={!bindingReady ? "Test the current database binding before synchronizing tables" : undefined} onClick={() => void synchronize("tables")}><RefreshCw className={busy === "sync" ? "animate-spin" : ""} />Sync tables</Button>
</>
)
)}
{presentation === "fleet" ? (
<Button type="button" size="sm" variant="outline" className="thot-fleet-nav-action thot-fleet-nav-action--below" data-tooltip="Synchronization history" onClick={onOpenSync}>
{currentRun ? <RefreshCw className="animate-spin" aria-hidden="true" /> : <History aria-hidden="true" />} Sync history
</Button>
) : null}
</div>
{!bindingReady ? <div className="border-b border-border bg-muted/35 px-4 py-3 text-sm text-muted-foreground">Test the current database binding from Overview before synchronizing columns.</div> : null}
{!bindingReady ? <div className="border-b border-border bg-muted/35 px-4 py-3 text-sm text-muted-foreground">Test the current database binding from Overview before synchronizing tables.</div> : null}
{fleetQueryError ?? <div className="relative min-h-[280px] flex-1">
<div className={`thot-database-grid ${presentation === "fleet" ? "thot-fleet-ledger-grid " : ""}ag-theme-alpine absolute inset-0 h-full w-full`}>
<AgGridReact<CatalogTable>
@@ -805,7 +795,7 @@ export function DatabaseTables({
rowHeight={44}
headerHeight={38}
animateRows={false}
overlayNoRowsTemplate="No catalog tables. Synchronize the schema from the database view."
overlayNoRowsTemplate="No catalog tables. Test the binding, then sync tables."
/>
</div>
</div>}
@@ -109,9 +109,6 @@ test("replays ordered SSE events, reconnects from the latest sequence, and keeps
renderDrawer();
const log = await screen.findByRole("log", { name: "Description generation events" });
const progress = screen.getByRole("table", { name: "Description generation progress" });
expect(within(progress).getAllByRole("columnheader")).toHaveLength(5);
expect(log).toHaveClass("thot-catalog-drawer__event-log");
expect(await within(log).findByText("Run queued")).toBeVisible();
await waitFor(() => expect(FakeEventSource.instances).toHaveLength(1));
expect(FakeEventSource.instances[0].url).toBe(
@@ -251,33 +248,6 @@ test("reopens a terminal run from newest-first persisted history", async () => {
expect(within(drawer).getByText("3", { selector: "[data-count='generated']" })).toBeVisible();
});
test("styles a successfully completed run with the success background", async () => {
const completedRun: DescriptionGenerationRun = {
...runningRun,
status: "completed",
total: 1,
processed: 1,
generated: 1,
updatedAt: "2026-08-28T08:02:00Z",
finishedAt: "2026-08-28T08:02:00Z",
};
server.use(
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([completedRun])),
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(completedRun)),
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
);
renderDrawer({ initialRun: completedRun });
const history = await screen.findByRole("region", {
name: "Description generation history",
});
const completedHistoryItem = within(history).getByRole("button", {
name: /Completed.*missing/i,
});
expect(completedHistoryItem).toHaveAttribute("data-status", "completed");
expect(completedHistoryItem).toHaveClass("bg-[oklch(var(--success)/0.1)]");
});
test("offers Unlock only for an apparently stale active run and explains the interruption", async () => {
const user = userEvent.setup();
const staleRun: DescriptionGenerationRun = {
@@ -1,4 +1,4 @@
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { useCallback, useEffect, useRef, useState } from "react";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { LoaderCircle, LockOpen, Square } from "lucide-react";
import { toast } from "sonner";
@@ -18,7 +18,6 @@ import { FleetLedgerDrawer } from "./FleetLedgerShell";
interface Props {
open: boolean;
databaseId?: string | null;
run: DescriptionGenerationRun | null;
modelLabel: string;
onClose: () => void;
@@ -63,7 +62,6 @@ function timestamp(value: string | null): string {
export function DescriptionGenerationDrawer({
open,
databaseId = null,
run: initialRun,
modelLabel,
onClose,
@@ -102,15 +100,11 @@ export function DescriptionGenerationDrawer({
? 3_000
: false,
});
const visibleHistory = useMemo(
() => (historyQuery.data ?? []).filter((item) => !databaseId || item.databaseId === databaseId),
[databaseId, historyQuery.data],
);
useEffect(() => {
if (!open || initialRun || !visibleHistory[0]) return;
onRunUpdate(visibleHistory[0]);
}, [initialRun, onRunUpdate, open, visibleHistory]);
if (!open || initialRun || !historyQuery.data?.[0]) return;
onRunUpdate(historyQuery.data[0]);
}, [historyQuery.data, initialRun, onRunUpdate, open]);
const mergeEvents = useCallback((incoming: DescriptionGenerationEvent[]) => {
if (incoming.length === 0) return;
@@ -300,16 +294,6 @@ export function DescriptionGenerationDrawer({
}
const apparentlyStale = isApparentlyStale(run);
const finalError = safeFinalError(run.errorSummary);
const modelDisplay = modelLabel && modelLabel !== run.modelId
? `${modelLabel} (${run.modelId})`
: run.modelId;
const progressCounters = [
["total", "Total"],
["processed", "Processed"],
["generated", "Generated"],
["nonGeneratable", "Not generated"],
["failed", "Failed"],
] as const;
const footer = ["queued", "running"].includes(run.status) ? (
<div className="w-full">
{apparentlyStale ? (
@@ -351,80 +335,55 @@ export function DescriptionGenerationDrawer({
open
ariaLabel="Description generation"
eyebrow="Description generation"
title={(
<span className="inline-flex items-center gap-2">
{!isTerminal(run) ? <LoaderCircle className="size-5 animate-spin" aria-hidden="true" /> : null}
{statusLabel(run)}
</span>
)}
description={run.scope.replaceAll("_", " ")}
title={statusLabel(run)}
description={`${modelLabel}${modelLabel !== run.modelId ? ` (${run.modelId})` : ""} · ${run.scope.replaceAll("_", " ")}`}
onClose={onClose}
closeLabel="Close description generation"
busy={action !== null}
bodyClassName="thot-catalog-drawer__history-layout thot-catalog-drawer__run-layout"
bodyClassName="thot-catalog-drawer__history-layout"
footer={footer}
>
<div className="thot-catalog-drawer__run-main">
{runQuery.isError ? (
{runQuery.isError ? (
<div role="alert" className="rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
{apiErrorMessage(runQuery.error)}
</div>
) : null}
<div className={`grid gap-4 sm:grid-cols-2${runQuery.isError ? " mt-4" : ""}`}>
<section aria-label="AI usage">
<h3 className="thot-label mb-2">AI usage</h3>
<dl className="grid grid-cols-[auto_1fr] gap-x-4 gap-y-1 text-xs">
<dt>Model</dt><dd className="truncate text-right" title={modelDisplay}>{modelDisplay}</dd>
<dt>Input tokens</dt><dd className="text-right tabular-nums">{(run.inputTokens ?? 0).toLocaleString("en-US")}</dd>
<dt>Cache tokens</dt><dd className="text-right tabular-nums">{(run.cacheReadTokens ?? 0).toLocaleString("en-US")}</dd>
<dt>Output tokens</dt><dd className="text-right tabular-nums">{(run.outputTokens ?? 0).toLocaleString("en-US")}</dd>
</dl>
</section>
<section aria-label="Description generation timestamps">
<h3 className="thot-label mb-2">Timestamps</h3>
<dl className="grid grid-cols-[auto_1fr] gap-x-4 gap-y-1 text-xs">
<dt className="text-muted-foreground">Created</dt><dd className="text-right tabular-nums">{timestamp(run.createdAt)}</dd>
<dt className="text-muted-foreground">Started</dt><dd className="text-right tabular-nums">{timestamp(run.startedAt)}</dd>
<dt className="text-muted-foreground">Updated</dt><dd className="text-right tabular-nums">{timestamp(run.updatedAt)}</dd>
<dt className="text-muted-foreground">Finished</dt><dd className="text-right tabular-nums">{timestamp(run.finishedAt)}</dd>
</dl>
</section>
</div>
<section className="mt-4" aria-label="Description generation counters">
<section aria-label="Description generation counters">
<h3 className="thot-label mb-2">Progress</h3>
<div className="overflow-x-auto rounded-md border border-border bg-muted/20">
<table className="w-full min-w-[28rem] table-fixed" aria-label="Description generation progress">
<thead>
<tr className="border-b border-border bg-muted/45">
{progressCounters.map(([name, label]) => (
<th key={name} scope="col" className="border-l border-border px-2 py-1.5 text-left text-[10px] font-semibold uppercase tracking-wide text-muted-foreground first:border-l-0">
{label}
</th>
))}
</tr>
</thead>
<tbody>
<tr>
{progressCounters.map(([name]) => (
<td key={name} data-count={name} className="border-l border-border px-2 py-1.5 text-base font-semibold tabular-nums first:border-l-0">
{run[name]}
</td>
))}
</tr>
</tbody>
</table>
<div className="grid grid-cols-2 gap-2">
{([
["total", "Total"],
["processed", "Processed"],
["generated", "Generated"],
["nonGeneratable", "Not generated"],
["failed", "Failed"],
] as const).map(([name, label]) => (
<div key={name} className="rounded-md border border-border bg-muted/25 px-3 py-2">
<p className="text-[11px] font-semibold uppercase tracking-wide text-muted-foreground">{label}</p>
<p data-count={name} className="mt-1 text-lg font-semibold tabular-nums">{run[name]}</p>
</div>
))}
</div>
</section>
<section className="mt-5" aria-label="Description generation timestamps">
<h3 className="thot-label mb-2">Timestamps</h3>
<dl className="grid grid-cols-[auto_1fr] gap-x-4 gap-y-1.5 text-sm">
<dt className="text-muted-foreground">Created</dt><dd className="text-right">{timestamp(run.createdAt)}</dd>
<dt className="text-muted-foreground">Started</dt><dd className="text-right">{timestamp(run.startedAt)}</dd>
<dt className="text-muted-foreground">Updated</dt><dd className="text-right">{timestamp(run.updatedAt)}</dd>
<dt className="text-muted-foreground">Finished</dt><dd className="text-right">{timestamp(run.finishedAt)}</dd>
</dl>
</section>
{finalError ? (
<div className="mt-3 rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
<div className="mt-5 rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
{finalError}
</div>
) : null}
<section className="thot-catalog-drawer__events mt-3" aria-label="Description generation log">
<section className="mt-5" aria-label="Description generation log">
<div className="mb-2 flex items-center justify-between">
<h3 className="thot-label">Events</h3>
<span className="text-xs tabular-nums text-muted-foreground">{events.length}</span>
@@ -432,7 +391,7 @@ export function DescriptionGenerationDrawer({
<div
role="log"
aria-label="Description generation events"
className="thot-catalog-drawer__event-log overflow-y-auto rounded-md bg-zinc-950 p-3 font-mono text-xs leading-5 text-zinc-200"
className="max-h-72 overflow-y-auto rounded-md bg-zinc-950 p-3 font-mono text-xs leading-5 text-zinc-200"
>
{events.length === 0 ? <p className="text-zinc-500">Waiting for events…</p> : events.map((event) => (
<p key={event.sequence} className={event.level === "error" ? "text-red-300" : event.level === "warning" ? "text-amber-300" : undefined}>
@@ -442,13 +401,12 @@ export function DescriptionGenerationDrawer({
</div>
{eventQuery.isError ? <p role="alert" className="mt-2 text-sm text-destructive">{apiErrorMessage(eventQuery.error)}</p> : null}
</section>
</div>
<section className="thot-catalog-drawer__history" aria-label="Description generation history">
<section className="thot-catalog-drawer__history mt-5" aria-label="Description generation history">
<div className="mb-2 flex items-center justify-between gap-3">
<h3 className="thot-label">Recent runs</h3>
<span className="text-xs tabular-nums text-muted-foreground">
{visibleHistory.length}
{historyQuery.data?.length ?? 0}
</span>
</div>
{historyQuery.isError ? (
@@ -457,18 +415,17 @@ export function DescriptionGenerationDrawer({
</p>
) : historyQuery.isLoading ? (
<p className="text-sm text-muted-foreground">Loading run history…</p>
) : visibleHistory.length === 0 ? (
) : (historyQuery.data ?? []).length === 0 ? (
<p className="text-sm text-muted-foreground">No description generation runs yet.</p>
) : (
<div className="thot-catalog-drawer__history-list divide-y divide-border overflow-y-auto rounded-md border border-border">
{visibleHistory.map((item) => (
<div className="max-h-56 divide-y divide-border overflow-y-auto rounded-md border border-border">
{(historyQuery.data ?? []).map((item) => (
<button
key={item.id}
type="button"
aria-label={`${statusLabel(item)} · ${item.scope.replaceAll("_", " ")}`}
aria-current={item.id === run.id ? "true" : undefined}
data-status={item.status}
className={`flex w-full items-start justify-between gap-3 px-3 py-2 text-left text-sm hover:bg-muted/50 focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/25${item.status === "completed" ? " bg-[oklch(var(--success)/0.1)]" : ""}`}
className="flex w-full items-start justify-between gap-3 px-3 py-2 text-left text-sm hover:bg-muted/50 focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/25"
onClick={() => onRunUpdate(item)}
>
<span className="font-medium">{statusLabel(item)}</span>
@@ -8,26 +8,13 @@ const actions = [
{ id: "sync", label: "Synchronize tables", group: "Synchronization", scopeLabel: "All tables in this database" },
{
id: "generate",
label: "Generate column descriptions",
label: "Generate descriptions",
group: "Descriptions",
disabled: true,
disabledReason: "Choose a metadata model first",
},
] as const;
test("disables the picker when no rows are selected", () => {
render(
<FleetActionSelector
actions={actions}
selectedCount={0}
onRun={vi.fn()}
onClear={vi.fn()}
/>,
);
expect(screen.getByRole("combobox", { name: "Batch action" })).toBeDisabled();
});
test("keeps unavailable commands discoverable and explains why Run is disabled", async () => {
const user = userEvent.setup();
render(
@@ -63,7 +50,6 @@ test("runs an enabled command explicitly and resets the picker", async () => {
await user.selectOptions(picker, "sync");
expect(screen.getByText("All tables in this database")).toBeVisible();
expect(screen.getByText("Action scope")).toBeVisible();
expect(screen.getByRole("button", { name: "Run action" })).toHaveClass("bg-destructive");
await user.click(screen.getByRole("button", { name: "Run action" }));
expect(onRun).toHaveBeenCalledWith("sync");
@@ -44,6 +44,7 @@ export function FleetActionSelector<ActionId extends string>({
selectedCount,
hiddenSelectedCount = 0,
busy = false,
busyLabel = "Running…",
selectRef,
onRun,
onClear,
@@ -101,7 +102,7 @@ export function FleetActionSelector<ActionId extends string>({
className="thot-fleet-action-selector__select"
aria-label={label}
value={selectedActionId}
disabled={busy || selectedCount === 0}
disabled={busy}
onChange={(event) => setSelectedActionId(event.target.value as ActionId | "")}
>
<option value="">Choose an action…</option>
@@ -122,14 +123,13 @@ export function FleetActionSelector<ActionId extends string>({
<Button
type="button"
size="sm"
variant="destructive"
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
variant={selectedAction?.tone === "destructive" ? "destructive" : "default"}
disabled={!selectedAction || selectedAction.disabled || busy}
aria-describedby={unavailableReason ? `${hintId}-reason` : undefined}
onClick={() => void run()}
>
<Play aria-hidden="true" />
Run action
{busy ? busyLabel : selectedAction?.runLabel ?? "Run action"}
</Button>
<Button type="button" size="sm" variant="ghost" disabled={busy} onClick={onClear}>
<X aria-hidden="true" /> Clear
@@ -1,36 +0,0 @@
import { readFileSync, readdirSync } from "node:fs";
import { expect, test } from "vitest";
const sourceDirectory = "src/shell/database-management";
const styles = readFileSync(`${sourceDirectory}/FleetLedgerShell.css`, "utf8");
test("uses operational body text for every available-command list", () => {
const availableCommandLists = readdirSync(sourceDirectory)
.filter((fileName) => fileName.endsWith(".tsx") && !fileName.includes(".test."))
.flatMap((fileName) => {
const source = readFileSync(`${sourceDirectory}/${fileName}`, "utf8");
const matches = source.match(
/<select\b[\s\S]*?<option value="">Choose an action…<\/option>[\s\S]*?<\/select>/g,
);
return (matches ?? []).map((markup) => ({ fileName, markup }));
});
expect(availableCommandLists.map(({ fileName }) => fileName).sort()).toEqual([
"DatabaseRelationships.tsx",
"FleetActionSelector.tsx",
]);
for (const { markup } of availableCommandLists) {
expect(markup).toContain('className="thot-fleet-action-selector__select"');
}
expect(styles).toMatch(
/\.thot-fleet-action-selector__select\s*\{[^}]*font-size:\s*0\.9375rem;/s,
);
expect(styles).toMatch(
/\.thot-fleet-action-selector\s*\{[^}]*flex:\s*0 0 auto;/s,
);
expect(styles).toMatch(
/\.thot-fleet-action-selector__select\s*\{[^}]*width:\s*max-content;/s,
);
});
@@ -274,7 +274,7 @@
background: oklch(var(--card));
}
.thot-fleet-back-action {
.thot-fleet-ledger__navigation > button:first-child {
min-height: 2.25rem;
border: 1px solid oklch(var(--primary) / 0.28);
color: oklch(var(--primary));
@@ -283,7 +283,7 @@
font-weight: 700;
}
.thot-fleet-back-action:not(:disabled):hover {
.thot-fleet-ledger__navigation > button:first-child:not(:disabled):hover {
border-color: oklch(var(--primary) / 0.48);
color: oklch(var(--primary));
background: oklch(var(--primary) / 0.14);
@@ -381,8 +381,7 @@
.thot-fleet-action-selector {
display: flex;
min-width: 0;
max-width: 100%;
flex: 0 0 auto;
flex: 1;
flex-wrap: wrap;
gap: 0.5rem;
align-items: center;
@@ -403,17 +402,15 @@
}
.thot-fleet-action-selector__select {
flex: 0 0 auto;
width: max-content;
min-width: 0;
max-width: 100%;
width: min(100%, 22rem);
min-width: 13rem;
height: 2rem;
padding: 0 2rem 0 0.625rem;
border: 1px solid oklch(var(--input));
border-radius: var(--radius);
color: oklch(var(--foreground));
background: oklch(var(--background));
font-size: 0.9375rem;
font-size: 0.8125rem;
outline: none;
}
@@ -434,6 +431,24 @@
line-height: 1.4;
}
.thot-fleet-metadata-model {
min-width: min(100%, 14rem);
flex: 0 1 16rem;
}
.thot-fleet-metadata-model > div {
min-width: 0;
gap: 0.25rem;
}
.thot-fleet-metadata-model label {
gap: 0.25rem;
}
.thot-fleet-metadata-model select {
height: 2rem;
}
.thot-fleet-ledger__selection-summary {
min-width: 0;
overflow: hidden;
@@ -462,11 +477,11 @@
.thot-fleet-ledger__drawer {
position: absolute;
z-index: 50;
inset-block: clamp(0.625rem, 2vh, 1rem);
inset-block: 0.75rem;
inset-inline-start: 50%;
inset-inline-end: auto;
display: flex;
width: min(75rem, calc(100% - 2rem));
width: 60%;
min-width: 0;
flex-direction: column;
overflow: hidden;
@@ -482,10 +497,10 @@
.thot-fleet-ledger .thot-fleet-ledger__context-panel {
position: absolute;
inset-block: clamp(0.625rem, 2vh, 1rem);
inset-block: 0.75rem;
inset-inline-start: 50%;
inset-inline-end: auto;
width: min(75rem, calc(100% - 2rem));
width: 60%;
max-width: none;
min-width: 0;
transform: translateX(-50%);
@@ -540,20 +555,6 @@
box-shadow: inset 0 -1px 0 oklch(var(--primary-foreground) / 0.18);
}
.thot-sensitive-run--success .thot-fleet-ledger__drawer-header,
.thot-sensitive-run--success .thot-catalog-drawer__header {
color: oklch(var(--primary-foreground));
background-color: oklch(var(--success));
border-color: oklch(var(--primary-foreground) / 0.18);
}
.thot-sensitive-run--failure .thot-fleet-ledger__drawer-header,
.thot-sensitive-run--failure .thot-catalog-drawer__header {
color: oklch(var(--destructive-foreground));
background-color: oklch(var(--destructive));
border-color: oklch(var(--destructive-foreground) / 0.18);
}
.thot-fleet-ledger__drawer-header .thot-fleet-ledger__drawer-title,
.thot-fleet-ledger__drawer-header .thot-fleet-ledger__drawer-description,
.thot-fleet-ledger__drawer-header .thot-fleet-ledger__eyebrow,
@@ -581,30 +582,6 @@
box-shadow: inset 3px 0 0 oklch(var(--primary));
}
.thot-catalog-drawer__history button[data-status="completed"],
.thot-catalog-drawer__history button[data-status="succeeded"] {
background: oklch(var(--success) / 0.1);
}
.thot-catalog-drawer__history button[aria-current="true"][data-status="completed"],
.thot-catalog-drawer__history button[aria-current="true"][data-status="succeeded"] {
background: oklch(var(--success) / 0.1);
box-shadow: inset 0 0 0 1px oklch(var(--success) / 0.5);
}
.thot-catalog-drawer__event-log {
min-height: 8rem;
max-height: 18rem;
}
.thot-catalog-drawer__history-list {
max-height: 14rem;
}
.thot-catalog-drawer__run-layout > .thot-catalog-drawer__history {
margin-top: 1.25rem;
}
.thot-fleet-grid-surface {
position: relative;
}
@@ -613,13 +590,6 @@
background: oklch(var(--muted) / 0.32);
}
@media (min-width: 48rem) {
.thot-fleet-grid-toolbar > .thot-fleet-action-selector {
flex: 0 0 auto;
min-width: 0;
}
}
.thot-fleet-ledger-grid {
--ag-header-background-color: oklch(var(--muted) / 0.84);
--ag-odd-row-background-color: oklch(var(--card));
@@ -642,14 +612,6 @@
background: oklch(var(--muted) / 0.12);
}
/* Keep selected rows readable without overpowering the status indicators. */
.thot-fleet-ledger-grid .ag-row.ag-row-selected,
.thot-fleet-ledger-grid .ag-row.ag-row-selected .ag-cell,
.thot-fleet-ledger-grid .ag-row.ag-row-selected .ag-grid-pinned-left-cells,
.thot-fleet-ledger-grid .ag-row.ag-row-selected .ag-grid-pinned-right-cells {
background-color: oklch(93% 0.002 17.2) !important;
}
.thot-fleet-icon-action {
position: relative;
}
@@ -661,29 +623,24 @@
.thot-fleet-icon-action::after {
position: absolute;
z-index: 80;
inset-inline-end: 0;
inset-block-start: calc(100% + 3px);
inset-inline-end: calc(100% + 5px);
inset-block-start: 50%;
width: max-content;
max-width: 12rem;
padding: 5px 8px;
border: 1px solid oklch(42% 0.018 258);
border-radius: 6px;
color: oklch(96% 0.006 258);
background: oklch(24% 0.014 258);
box-shadow: 0 8px 20px -10px oklch(12% 0.02 258 / 0.72);
padding: 0.3rem 0.45rem;
border: 1px solid oklch(var(--border));
border-radius: calc(var(--radius) - 0.125rem);
color: oklch(var(--popover-foreground));
background: oklch(var(--popover));
box-shadow: var(--shadow-sm);
content: attr(data-tooltip);
font-size: 0.6875rem;
font-weight: 600;
line-height: 1.25;
text-align: center;
white-space: normal;
font-weight: 650;
line-height: 1.2;
opacity: 0;
pointer-events: none;
transform: translateY(-2px);
transition:
opacity 90ms linear,
transform 120ms cubic-bezier(0.25, 1, 0.5, 1),
visibility 0ms linear 120ms;
transform: translate(2px, -50%);
transition: opacity 60ms cubic-bezier(0.22, 1, 0.36, 1), transform 60ms cubic-bezier(0.22, 1, 0.36, 1);
visibility: hidden;
}
@@ -720,8 +677,7 @@
.thot-fleet-icon-action:focus-visible::after,
.thot-fleet-icon-action:focus-within::after {
opacity: 1;
transform: translateY(0);
transition-delay: 60ms;
transform: translate(0, -50%);
visibility: visible;
}
@@ -789,10 +745,6 @@
padding: 0;
}
.thot-fleet-ledger__drawer-body.thot-catalog-drawer__run-layout {
padding: clamp(0.75rem, 1.5cqi, 1.25rem);
}
.thot-fleet-ledger__drawer-footer {
display: flex;
flex-wrap: wrap;
@@ -833,8 +785,8 @@
@container fleet-surface (max-width: 47.999rem) {
.thot-fleet-ledger__drawer,
.thot-fleet-ledger .thot-fleet-ledger__context-panel {
inset-block: 0.625rem;
width: calc(100% - 1.5rem);
inset-block: 0.5rem;
width: calc(100% - 1rem);
}
}
@@ -865,11 +817,11 @@
}
}
@container work-area-panel (min-width: 48rem) {
@media (min-width: 40rem) {
.thot-catalog-drawer__history-layout {
display: grid;
grid-template-columns: minmax(0, 1fr) clamp(12.5rem, 25cqi, 15rem);
gap: 0 clamp(0.75rem, 2cqi, 1.25rem);
grid-template-columns: minmax(0, 1fr) 13.5rem;
gap: 0 1rem;
align-content: start;
}
@@ -885,45 +837,6 @@
align-self: start;
margin-top: 0;
}
.thot-catalog-drawer__run-layout {
height: 100%;
min-height: 0;
grid-template-rows: minmax(0, 1fr);
align-content: stretch;
overflow: hidden;
}
.thot-catalog-drawer__run-layout > .thot-catalog-drawer__run-main {
display: flex;
min-height: 0;
flex-direction: column;
grid-column: 1;
grid-row: 1;
}
.thot-catalog-drawer__run-layout > .thot-catalog-drawer__history {
position: static;
display: flex;
min-height: 0;
flex-direction: column;
align-self: stretch;
grid-row: 1;
}
.thot-catalog-drawer__run-layout .thot-catalog-drawer__events {
display: flex;
min-height: 10rem;
flex: 1;
flex-direction: column;
}
.thot-catalog-drawer__run-layout .thot-catalog-drawer__event-log,
.thot-catalog-drawer__run-layout .thot-catalog-drawer__history-list {
min-height: 0;
max-height: none;
flex: 1;
}
}
@media (max-width: 47.999rem) {
@@ -965,6 +878,9 @@
width: 100%;
}
.thot-fleet-action-selector__select {
width: 100%;
}
}
@media (max-height: 44rem) and (min-width: 48rem) {
@@ -233,7 +233,7 @@ export function FleetLedgerBreadcrumb({
type="button"
variant="ghost"
size="sm"
className="thot-fleet-nav-action thot-fleet-back-action"
className="thot-fleet-nav-action"
data-tooltip={back.label.replace(/^Back to\s+/i, "")}
onClick={back.onBack}
disabled={back.disabled}
@@ -1,27 +0,0 @@
import { readFileSync } from "node:fs";
import { resolve } from "node:path";
import { describe, expect, test } from "vitest";
const pageSource = readFileSync(resolve(__dirname, "../DatabaseManagementPage.tsx"), "utf8");
const componentSource = (name: string) => readFileSync(resolve(__dirname, name), "utf8");
describe("history action placement", () => {
test("workspace header does not own database history actions", () => {
const fleetHeader = pageSource.slice(pageSource.indexOf('title="Database management"'), pageSource.indexOf("kpis={"));
expect(fleetHeader).not.toContain("Description history");
expect(fleetHeader).not.toContain("Sensitive history");
});
test("database tables do not own metadata history actions", () => {
for (const file of ["DatabaseTables.tsx"]) {
const source = componentSource(file);
expect(source, file).not.toContain("Description history");
expect(source, file).not.toContain("Sensitive history");
}
for (const file of ["DatabaseForm.tsx", "DatabaseRelationships.tsx"]) {
const source = componentSource(file);
expect(source, file).toContain("Description history");
expect(source, file).toContain("Sensitive history");
}
});
});
@@ -1,9 +1,5 @@
import { useId } from "react";
import type { MetadataGenerationModels } from "../../api/catalog-databases";
export const NO_METADATA_GENERATION_LLM_MODEL_MESSAGE =
"No metadata-generation LLM model is configured for this installation.";
interface Props {
canManage: boolean;
data?: MetadataGenerationModels;
@@ -11,7 +7,6 @@ interface Props {
isLoading: boolean;
selectedModel: string;
onSelectedModelChange: (modelId: string) => void;
variant?: "default" | "compact";
}
export function MetadataGenerationModelSelector({
@@ -21,35 +16,21 @@ export function MetadataGenerationModelSelector({
isLoading,
selectedModel,
onSelectedModelChange,
variant = "default",
}: Props) {
const helpId = useId();
const models = data?.models ?? [];
const configuredDefault = data?.default ?? "";
const hasUsableDefault = configuredDefault !== ""
&& models.some((model) => model.id === configuredDefault);
const noUsableModel = !isLoading && !isError && Boolean(data) && !hasUsableDefault;
const unavailableMessage = !isLoading && isError
? "Metadata-generation LLM model choices are unavailable."
: noUsableModel
? NO_METADATA_GENERATION_LLM_MODEL_MESSAGE
: null;
const compact = variant === "compact";
const unavailable = noUsableModel || (!isLoading && isError);
return (
<div className={compact
? "flex min-w-0 flex-wrap items-center gap-x-2 gap-y-1"
: "grid min-w-56 max-w-lg gap-1.5"}
>
<label className={compact
? "flex min-w-0 items-center gap-2 text-xs font-medium text-foreground"
: "grid gap-1.5 text-xs font-medium text-foreground"}
>
<span className={compact ? "whitespace-nowrap" : undefined}>Metadata-generation LLM model</span>
<div className="grid min-w-56 max-w-lg gap-1.5">
<label className="grid gap-1.5 text-xs font-medium text-foreground">
<span>Metadata description model</span>
<select
className={`${compact ? "h-8 min-w-52" : "h-9 w-full"} rounded-md border border-input bg-card px-3 text-sm outline-none transition focus:border-primary/60 focus:ring-3 focus:ring-ring/15 disabled:bg-muted disabled:text-muted-foreground`}
aria-label="Metadata-generation LLM model"
aria-describedby={unavailableMessage ? helpId : undefined}
className="h-9 w-full rounded-md border border-input bg-card px-3 text-sm outline-none transition focus:border-primary/60 focus:ring-3 focus:ring-ring/15 disabled:bg-muted disabled:text-muted-foreground"
aria-label="Metadata description model"
aria-describedby={unavailable ? "metadata-generation-model-help" : undefined}
value={selectedModel}
disabled={!canManage || isLoading || isError || !hasUsableDefault}
onChange={(event) => onSelectedModelChange(event.target.value)}
@@ -64,9 +45,9 @@ export function MetadataGenerationModelSelector({
))}
</select>
</label>
{unavailableMessage ? (
<p id={helpId} className="max-w-72 text-xs font-normal leading-4 text-muted-foreground">
{unavailableMessage}
{unavailable ? (
<p id="metadata-generation-model-help" className="max-w-72 text-xs font-normal leading-4 text-muted-foreground">
Configure a metadata-generation model in application setup to enable description generation.
</p>
) : null}
</div>
@@ -1,40 +0,0 @@
import { readFileSync } from "node:fs";
import { resolve } from "node:path";
import { describe, expect, test } from "vitest";
const sourceRoot = resolve(__dirname);
describe("Recent runs success styling contract", () => {
test("every history renderer exposes its run status to the shared styles", () => {
const renderers = [
"DescriptionGenerationDrawer.tsx",
"SensitiveDataSuggestionHistoryDrawer.tsx",
"CatalogSyncDrawer.tsx",
];
for (const renderer of renderers) {
const source = readFileSync(resolve(sourceRoot, renderer), "utf8");
expect(source, renderer).toMatch(/data-status=\{item\.(?:status|state)\}/);
}
});
test("shared history styles cover both successful status vocabularies", () => {
const styles = readFileSync(resolve(sourceRoot, "FleetLedgerShell.css"), "utf8");
expect(styles).toContain('button[data-status="completed"]');
expect(styles).toContain('button[data-status="succeeded"]');
expect(styles).toMatch(/oklch\(var\(--success\)/);
});
test("run drawers keep responsive gutters and switch columns from their own width", () => {
const styles = readFileSync(resolve(sourceRoot, "FleetLedgerShell.css"), "utf8");
expect(styles).toContain("width: min(75rem, calc(100% - 2rem));");
expect(styles).toContain(
".thot-fleet-ledger__drawer-body.thot-catalog-drawer__run-layout",
);
expect(styles).toContain("padding: clamp(0.75rem, 1.5cqi, 1.25rem);");
expect(styles).toContain("@container work-area-panel (min-width: 48rem)");
expect(styles).toContain(".thot-catalog-drawer__history-list");
expect(styles).toContain(".thot-catalog-drawer__event-log");
});
});
@@ -97,11 +97,6 @@ test("shows sensitive suggestion results, safe events, and lets operators inspec
const drawer = await screen.findByRole("dialog", { name: "Sensitive suggestion history" });
expect(within(drawer).getByRole("heading", { name: "Running" })).toBeVisible();
const progress = within(drawer).getByRole("table", { name: "Sensitive suggestion progress" });
expect(within(progress).getAllByRole("columnheader")).toHaveLength(3);
expect(within(drawer).getByRole("log", { name: "Sensitive suggestion events" })).toHaveClass(
"thot-catalog-drawer__event-log",
);
expect(await within(drawer).findByText("Classifying columns")).toBeVisible();
expect(within(drawer).queryByRole("button", { name: "Stop" })).not.toBeInTheDocument();
expect(within(drawer).queryByRole("button", { name: "Unlock stale run" })).not.toBeInTheDocument();
@@ -1,5 +1,4 @@
import { useEffect, useMemo } from "react";
import { LoaderCircle } from "lucide-react";
import { useEffect } from "react";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { apiErrorMessage } from "../../api/client";
import {
@@ -12,7 +11,6 @@ import { FleetLedgerDrawer } from "./FleetLedgerShell";
interface Props {
open: boolean;
databaseId?: string | null;
run: SensitiveDataSuggestionRun | null;
modelLabel: string;
onClose: () => void;
@@ -31,12 +29,6 @@ function isTerminal(run?: SensitiveDataSuggestionRun): boolean {
return Boolean(run && ["completed", "failed", "interrupted"].includes(run.status));
}
function outcomeClass(run: SensitiveDataSuggestionRun): string {
if (run.status === "completed") return "thot-sensitive-run--success";
if (["failed", "interrupted"].includes(run.status)) return "thot-sensitive-run--failure";
return "";
}
function safeFinalError(summary: string | null): string | null {
const value = summary?.trim();
if (!value) return null;
@@ -54,7 +46,6 @@ function timestamp(value: string | null): string {
export function SensitiveDataSuggestionHistoryDrawer({
open,
databaseId = null,
run: initialRun,
modelLabel,
onClose,
@@ -87,15 +78,11 @@ export function SensitiveDataSuggestionHistoryDrawer({
? 3_000
: false,
});
const visibleHistory = useMemo(
() => (historyQuery.data ?? []).filter((item) => !databaseId || item.databaseId === databaseId),
[databaseId, historyQuery.data],
);
useEffect(() => {
if (!open || initialRun || !visibleHistory[0]) return;
onRunUpdate(visibleHistory[0]);
}, [initialRun, onRunUpdate, open, visibleHistory]);
if (!open || initialRun || !historyQuery.data?.[0]) return;
onRunUpdate(historyQuery.data[0]);
}, [historyQuery.data, initialRun, onRunUpdate, open]);
useEffect(() => {
if (!run) return;
@@ -140,92 +127,56 @@ export function SensitiveDataSuggestionHistoryDrawer({
const events = eventQuery.data ?? [];
const finalError = safeFinalError(run.errorSummary);
const modelDisplay = modelLabel && modelLabel !== run.modelId
? `${modelLabel} (${run.modelId})`
: run.modelId;
const progressCounters = [
["total", "Total columns"],
["suggestedSensitive", "Sensitive"],
["suggestedNonSensitive", "Not sensitive"],
] as const;
return (
<FleetLedgerDrawer
open
className={outcomeClass(run)}
ariaLabel="Sensitive suggestion history"
eyebrow="Sensitive data suggestions"
title={(
<span className="inline-flex items-center gap-2">
{!isTerminal(run) ? <LoaderCircle className="size-5 animate-spin" aria-hidden="true" /> : null}
{statusLabel(run)}
</span>
)}
description={run.scope.replaceAll("_", " ")}
title={statusLabel(run)}
description={`${modelLabel}${modelLabel !== run.modelId ? ` (${run.modelId})` : ""} · ${run.scope.replaceAll("_", " ")}`}
onClose={onClose}
closeLabel="Close sensitive suggestion history"
bodyClassName="thot-catalog-drawer__history-layout thot-catalog-drawer__run-layout"
bodyClassName="thot-catalog-drawer__history-layout"
>
<div className="thot-catalog-drawer__run-main">
{runQuery.isError ? (
{runQuery.isError ? (
<div role="alert" className="rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
{apiErrorMessage(runQuery.error)}
</div>
) : null}
<div className={`grid gap-4 sm:grid-cols-2${runQuery.isError ? " mt-4" : ""}`}>
<section aria-label="AI usage">
<h3 className="thot-label mb-2">AI usage</h3>
<dl className="grid grid-cols-[auto_1fr] gap-x-4 gap-y-1 text-xs">
<dt>Model</dt><dd className="truncate text-right" title={modelDisplay}>{modelDisplay}</dd>
<dt>Input tokens</dt><dd className="text-right tabular-nums">{(run.inputTokens ?? 0).toLocaleString("en-US")}</dd>
<dt>Cache tokens</dt><dd className="text-right tabular-nums">{(run.cacheReadTokens ?? 0).toLocaleString("en-US")}</dd>
<dt>Output tokens</dt><dd className="text-right tabular-nums">{(run.outputTokens ?? 0).toLocaleString("en-US")}</dd>
</dl>
</section>
<section aria-label="Sensitive suggestion timestamps">
<h3 className="thot-label mb-2">Timestamps</h3>
<dl className="grid grid-cols-[auto_1fr] gap-x-4 gap-y-1 text-xs">
<dt className="text-muted-foreground">Created</dt><dd className="text-right tabular-nums">{timestamp(run.createdAt)}</dd>
<dt className="text-muted-foreground">Started</dt><dd className="text-right tabular-nums">{timestamp(run.startedAt)}</dd>
<dt className="text-muted-foreground">Updated</dt><dd className="text-right tabular-nums">{timestamp(run.updatedAt)}</dd>
<dt className="text-muted-foreground">Finished</dt><dd className="text-right tabular-nums">{timestamp(run.finishedAt)}</dd>
</dl>
</section>
</div>
<section className="mt-4" aria-label="Sensitive suggestion counters">
<section aria-label="Sensitive suggestion counters">
<h3 className="thot-label mb-2">Results</h3>
<div className="overflow-x-auto rounded-md border border-border bg-muted/20">
<table className="w-full min-w-[20rem] table-fixed" aria-label="Sensitive suggestion progress">
<thead>
<tr className="border-b border-border bg-muted/45">
{progressCounters.map(([name, label]) => (
<th key={name} scope="col" className="border-l border-border px-2 py-1.5 text-left text-[10px] font-semibold uppercase tracking-wide text-muted-foreground first:border-l-0">
{label}
</th>
))}
</tr>
</thead>
<tbody>
<tr>
{progressCounters.map(([name]) => (
<td key={name} data-count={name} className="border-l border-border px-2 py-1.5 text-base font-semibold tabular-nums first:border-l-0">
{run[name]}
</td>
))}
</tr>
</tbody>
</table>
<div className="grid grid-cols-2 gap-2">
{([
["total", "Total columns"],
["suggestedSensitive", "Sensitive"],
["suggestedNonSensitive", "Not sensitive"],
] as const).map(([name, label]) => (
<div key={name} className="rounded-md border border-border bg-muted/25 px-3 py-2">
<p className="text-[11px] font-semibold uppercase tracking-wide text-muted-foreground">{label}</p>
<p data-count={name} className="mt-1 text-lg font-semibold tabular-nums">{run[name]}</p>
</div>
))}
</div>
</section>
<section className="mt-5" aria-label="Sensitive suggestion timestamps">
<h3 className="thot-label mb-2">Timestamps</h3>
<dl className="grid grid-cols-[auto_1fr] gap-x-4 gap-y-1.5 text-sm">
<dt className="text-muted-foreground">Created</dt><dd className="text-right">{timestamp(run.createdAt)}</dd>
<dt className="text-muted-foreground">Started</dt><dd className="text-right">{timestamp(run.startedAt)}</dd>
<dt className="text-muted-foreground">Updated</dt><dd className="text-right">{timestamp(run.updatedAt)}</dd>
<dt className="text-muted-foreground">Finished</dt><dd className="text-right">{timestamp(run.finishedAt)}</dd>
</dl>
</section>
{finalError ? (
<div className="mt-3 rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
<div className="mt-5 rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
{finalError}
</div>
) : null}
<section className="thot-catalog-drawer__events mt-3" aria-label="Sensitive suggestion log">
<section className="mt-5" aria-label="Sensitive suggestion log">
<div className="mb-2 flex items-center justify-between">
<h3 className="thot-label">Events</h3>
<span className="text-xs tabular-nums text-muted-foreground">{events.length}</span>
@@ -233,7 +184,7 @@ export function SensitiveDataSuggestionHistoryDrawer({
<div
role="log"
aria-label="Sensitive suggestion events"
className="thot-catalog-drawer__event-log overflow-y-auto rounded-md bg-zinc-950 p-3 font-mono text-xs leading-5 text-zinc-200"
className="max-h-72 overflow-y-auto rounded-md bg-zinc-950 p-3 font-mono text-xs leading-5 text-zinc-200"
>
{events.length === 0 ? <p className="text-zinc-500">Waiting for events…</p> : events.map((event) => (
<p key={event.sequence} className={event.level === "error" ? "text-red-300" : event.level === "warning" ? "text-amber-300" : undefined}>
@@ -243,30 +194,28 @@ export function SensitiveDataSuggestionHistoryDrawer({
</div>
{eventQuery.isError ? <p role="alert" className="mt-2 text-sm text-destructive">{apiErrorMessage(eventQuery.error)}</p> : null}
</section>
</div>
<section className="thot-catalog-drawer__history" aria-label="Sensitive suggestion run history">
<section className="thot-catalog-drawer__history mt-5" aria-label="Sensitive suggestion run history">
<div className="mb-2 flex items-center justify-between gap-3">
<h3 className="thot-label">Recent runs</h3>
<span className="text-xs tabular-nums text-muted-foreground">
{visibleHistory.length}
{historyQuery.data?.length ?? 0}
</span>
</div>
{historyQuery.isError ? (
<p role="alert" className="text-sm text-destructive">{apiErrorMessage(historyQuery.error)}</p>
) : historyQuery.isLoading ? (
<p className="text-sm text-muted-foreground">Loading suggestion history…</p>
) : visibleHistory.length === 0 ? (
) : (historyQuery.data ?? []).length === 0 ? (
<p className="text-sm text-muted-foreground">No sensitive suggestion runs yet.</p>
) : (
<div className="thot-catalog-drawer__history-list divide-y divide-border overflow-y-auto rounded-md border border-border">
{visibleHistory.map((item) => (
<div className="max-h-56 divide-y divide-border overflow-y-auto rounded-md border border-border">
{(historyQuery.data ?? []).map((item) => (
<button
key={item.id}
type="button"
aria-label={`${statusLabel(item)} · ${item.scope.replaceAll("_", " ")}`}
aria-current={item.id === run.id ? "true" : undefined}
data-status={item.status}
className="flex w-full items-start justify-between gap-3 px-3 py-2 text-left text-sm hover:bg-muted/50 focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/25"
onClick={() => onRunUpdate(item)}
>
@@ -4,7 +4,7 @@ import type {
DatabaseConfiguration,
} from "../../api/catalog-databases";
export type DatabaseFormMode = "configure" | "view" | "edit" | "delete";
export type DatabaseFormMode = "add" | "view" | "edit" | "delete";
export type DatabaseScreen =
| { kind: "list" }
@@ -13,6 +13,7 @@ export type DatabaseScreen =
| {
kind: DatabaseFormMode;
workspaceId: string;
workspaceLocked?: boolean;
};
export type DatabaseBusyAction = "save" | "test" | "delete" | "retry-secrets" | "reload" | null;
+1 -1
View File
@@ -176,7 +176,7 @@ test("compacts table, filter, detail, and rationale rows", () => {
expect(within(rationale!).getByText("Rationale")).toHaveClass("mb-1");
});
test("shows 'No dependencies' for a CTE with an empty depends_on", () => {
test("shows 'no dependencies' for a CTE with an empty depends_on", () => {
render(<CtePlanViewer plan={plan} />);
expect(screen.getByText(/no dependencies/i)).toBeInTheDocument();
});
+1 -1
View File
@@ -93,7 +93,7 @@ function CteCard({ cte, total }: { cte: CtePlanCte; total: number }) {
))}
</div>
) : (
<p className="text-xs text-muted-foreground">No dependencies</p>
<p className="text-xs text-muted-foreground">no dependencies</p>
)}
</section>
@@ -52,7 +52,7 @@ test("(b) clicking toggle switches to table listing a promoted table name", asyn
test("(c) oversized linking (>45) shows cap notice", () => {
render(<SchemaLinkingViewer linking={bigLinking} />);
expect(
screen.getByText(/too many nodes to display in the graph/i)
screen.getByText(/too many elements for the graph/i)
).toBeInTheDocument();
});
+1 -1
View File
@@ -147,7 +147,7 @@ export function SchemaLinkingViewer({ linking }: { linking: SchemaLinking }) {
{oversized && (
<p className="text-sm text-amber-600">
Too many nodes to display in the graph. Use the table instead.
Too many elements for the graph, use the table
</p>
)}

Some files were not shown because too many files have changed in this diff Show More