feat: add AI catalog description generation

This commit is contained in:
Codex
2026-08-29 16:42:56 +02:00
parent b0afba81ca
commit 376dd5a09d
76 changed files with 14860 additions and 102 deletions
+6 -4
View File
@@ -76,12 +76,14 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
There is no verbatim transcript store. A resumed Pi process rebuilds context from
`tht session show <id>` + the on-disk artifacts.
- **The backend is a thin bridge with no database.** `ThtRunner` shells the Python workflow `tht`
subcommands inside `core`;
- **The backend bridges sessions and owns the installation-local metadata catalog.** `ThtRunner`
shells the Python workflow `tht` subcommands inside `core`;
`PiProcessManager` runs one Pi child per session and bridges its RPC stream;
`SessionBridge` maps Pi RPC events → client events (`ui_request`/`text_delta`/`info`);
`SseHub` fans them out over SSE to the browser. App settings live in a JSON file
(`backend/data/settings.json`), not a DB.
`SseHub` fans them out over SSE to the browser. The separate PostgreSQL catalog stores database
metadata and sequential AI description-generation runs. Description generation samples the DWH
through read-only connectors and calls a short-lived Python LiteLLM helper; it does not use Pi or
expose a public CLI command. App settings still live in `backend/data/settings.json`.
- **Human-in-the-loop gate contract.** The model proposes; a human reviewer decides at gates
via widgets (`reviewer_select` = single pick — a chosen option carrying a `decision` payload
+40
View File
@@ -306,6 +306,11 @@ per gli usi downstream.
essere consolidata come Description. Rimane distinta dal commento osservato nel database.
_Avoid_: generated comment, source comment
**Description Consolidation** — L'azione amministrativa esplicita che copia la Generated
Description di Catalog Table o Catalog Column selezionate nella relativa Description. Opera sulla
selezione corrente, conserva la Generated Description e non modifica il commento osservato o il
database esterno.
**Table Synchronization** — La riconciliazione esplicita che rende le Catalog Table di un
Workspace Database uguali alle Physical Table osservate: crea quelle nuove, aggiorna i metadati
di origine ed elimina definitivamente quelle assenti. Non modifica mai il database esterno.
@@ -318,6 +323,23 @@ un unico snapshot completo tramite Synchronize All.
**Catalog Sync Run** — L'esecuzione durevole in background di una Schema Synchronization, con
scope, stato, avanzamento e log propri. Al massimo un run per Workspace Database può essere attivo.
**Description Generation Run** — L'esecuzione asincrona e sequenziale che usa il modello scelto
per produrre Generated Description di Catalog Table o Catalog Column. Al massimo una run è attiva
nell'intera installazione e ogni risultato valido viene salvato appena disponibile. Dopo
un'interruzione il recupero è manuale tramite una nuova generazione dei soli elementi mancanti.
**Description Generation Event** — Una riga testuale ordinata che registra avanzamento, risultato
o errore di una Description Generation Run e alimenta il log visibile all'amministratore.
**Non-generatable Description** — L'esito valido con cui il modello dichiara di non disporre di
informazioni sufficienti per descrivere il target. Produce una Generated Description standard
nella lingua del workspace e non rappresenta un timeout, un errore del provider o una risposta
non valida.
**Description Generation Unlock** — Il recupero amministrativo che marca come interrotta una
Description Generation Run registrata come attiva quando il backend non ha alcun processo di
generazione vivo. Non è un meccanismo di lock distribuito.
**Catalog Metadata Cleanup** — La rimozione amministrativa esplicita di Catalog Table, Catalog
Column o Catalog Relationship selezionate. Non modifica il Workspace Database, la Database Binding
o i segreti, e può lasciare il Metadata Catalog intenzionalmente incompleto fino alla prossima
@@ -331,6 +353,24 @@ con la binding corrente.
distinti dai fatti strutturali governati dalla sincronizzazione. Possono essere popolati dall'AI,
da un'importazione o da una modifica amministrativa senza cambiare il database esterno.
**Metadata Generation Model Configuration** — La configurazione a livello di setup applicativo
che elenca i modelli selezionabili, il default e i riferimenti agli eventuali segreti per la sola
generazione dei metadati. Un modello keyless è ammesso solo con un endpoint esplicito che non
richiede autenticazione. Non appartiene al workspace ed è indipendente dalla configurazione Pi.
**Model Completion Helper** — Il processo Python interno ed effimero che esegue una singola
richiesta LiteLLM per conto del backend. Non è un servizio HTTP, non possiede il lifecycle della
Description Generation Run e non è una CLI esposta agli utenti.
**Catalog Sample** — Un input transitorio composto da un massimo di cinque righe reali e dai
valori di esempio bounded letti da una Catalog Table per la generazione delle descrizioni. È
trattato come dato non fidato, non viene persistito e non diventa Catalog Metadata.
**Sensitive Data Policy** — L'insieme di regole che classifica i valori sorgente protetti per
l'uso nei processi AI e ne prescrive l'esclusione o l'anonimizzazione. Non coincide con la sola
classificazione dei dati personali.
_Avoid_: PII filter, sample filter
**Introspection Capability** — Una categoria di struttura fisica che una Database Binding
può osservare, come tabelle, colonne, relazioni, indici o enum. Una capability non disponibile
è distinta da una capability osservata che non ha restituito elementi.
+34 -4
View File
@@ -79,7 +79,8 @@ hand, but administrators can explicitly clear catalog tables, columns, or relati
touching the source database, binding, configuration, or secrets. Table deletion cascades through
columns and relationships; table-scoped relationship cleanup includes incoming and outgoing
relationships. Curated and generated descriptions are editable; generated descriptions start null
and AI generation/consolidation is deferred.
and Database Management can generate or consolidate them for selected tables, selected columns,
all targets, or only targets whose Generated Description is missing.
Schema refresh is one durable asynchronous engine with database-table, database-column,
selected-table-column, relationship, and full-database actions. Database-level menus expose the
@@ -103,10 +104,36 @@ one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before loca
sessions still consume the existing workspace configuration in this slice: database-management
records do not yet change the NL→SQL handoff. The accepted design is recorded in
`docs/plans/2026-08-26-metadata-catalog-from-thothai.md`, the snapshot contract under
`docs/contracts/`, and ADRs 0001–0008.
`docs/contracts/`, and ADRs 0001–0010.
Semantic aliases, value descriptions, synonyms, concepts, AI metadata generation/consolidation,
and logical relationships remain deferred to their dedicated slices.
Semantic aliases, value descriptions, synonyms, concepts, and logical relationships remain
deferred to their dedicated slices.
AI Description Generation preserves ThothAI's use of real source samples: up to five source rows
and five representative non-null example values may be sent transiently to the configured model
provider. The UI and operator documentation disclose this behavior. A required follow-up
improvement is a Sensitive Data Policy that classifies protected fields and excludes or anonymizes
their values before model calls.
The accepted AI-description design is recorded in
`docs/plans/2026-08-28-ai-catalog-description-generation.md`, with the formal specification in the
adjacent `-spec.md` document and Gitea issue #4. Gitea issues #5–#11 deliver the implementation.
The runtime deliberately keeps ThothAI's simple operating model: one installation-wide sequential
run owned by the backend, one short-lived Python/LiteLLM completion helper per request, and
persistence limited to the run, its safe ordered text events, and each Generated Description as
soon as it succeeds. The helper performs at most one provider retry and never falls back to another
model. Stop terminates the current helper and retains prior results; three consecutive exhausted
technical batches fail the run. Startup marks stale queued/running work interrupted, and Unlock is
available only when no local start, worker, or helper is live. Runs remain inspectable through a
live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation
CLI. ADRs 0009–0010 record the runtime and source-sampling decisions.
Metadata-generation setup accepts the protected `DEEPSEEK_API_KEY` and `ZAI_API_KEY` references.
It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is
explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator
credential. The Python client supplies only its fixed non-secret compatibility placeholder.
The AritmoLab entry also sets `disableThinking: true`, mapped to the endpoint's chat-template flag,
because its default reasoning prose would violate the worker's exact JSON response contract.
Integration of the completed metadata catalog with core schema-linking is explicitly deferred
until the database, table, column, relationship, and synchronization slices are complete. At that
@@ -159,6 +186,9 @@ from an automated PASS.
- The P1.1 workspace-directory registry and P2–P6 preprocessing workstreams are implemented and
have automated coverage.
- Evidence restructuring has a real PSD acceptance PASS as recorded above.
- AI Description Generation has automated coverage across installation setup, model selection,
generation/consolidation scopes, bounded sampling, cancellation/recovery, history, SSE/polling,
and the LiteLLM helper boundary.
- L2 tests requiring real providers or remote databases remain opt-in.
- Server deployment, release, and owner-operated acceptance steps remain pending wherever the
referenced runbooks require explicit approval.
+29 -1
View File
@@ -17,6 +17,8 @@ From a fresh clone, run these commands from the repository root:
```sh
cp deploy/env/local.env.example deploy/env/local.env
# Copy docs/install/examples/thothii-installation.local.yaml to a protected operator path,
# replace its placeholders, chmod it 600, and set that exact THT_INSTALLATION_CONFIG_SOURCE.
# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
./scripts/run-stack.sh
```
@@ -29,7 +31,8 @@ application rollout:
```sh
cp deploy/env/server.env.example deploy/env/server.env
# Edit all absolute storage, Pi/secret/session files, and endpoint paths.
# Prepare a mode-600 thothii-installation.yaml from the server example and set its exact
# path as THT_INSTALLATION_CONFIG_SOURCE. Edit all remaining storage/secret/endpoint paths.
sudo scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
@@ -293,6 +296,31 @@ Copy `deploy/secrets/thothii.secrets.example` to a protected host file, include
keys, and set its absolute path as `THT_SECRETS_FILE` in the operator env. Keep Pi's native
provider auth in the separate protected file named by `PI_AUTH_FILE`.
Description Generation is configured independently in the protected installation descriptor under
`metadataGeneration`. Set `THT_INSTALLATION_CONFIG_SOURCE` to that exact host file; Compose mounts
it read-only into `core` and supplies the fixed runtime `THT_INSTALLATION_CONFIG_FILE` path. Each
keyed model stores only an audited `apiKeyEnv` reference. The referenced value stays in the secret
bundle; a model may omit `apiKeyEnv` only when it declares an explicit endpoint that accepts
unauthenticated requests. The browser receives only model IDs, labels, and the configured default.
Configuration changes take effect after restart and do not use Pi settings or workspace
`llm_policy`.
Before enabling Description Generation, approve the selected model provider for bounded source-data
disclosure. A request may send up to five real source rows and up to five representative distinct,
non-null example values for relevant columns. Samples are transient and are not stored in generation
runs, run logs, application logs, API responses, or catalog metadata; prompt and sample snapshots are
not retained. Automated Sensitive Data Policy filtering and anonymization are not currently provided.
A future Sensitive Data Policy is required to classify protected fields and exclude or anonymize
their values before model calls.
Description Generation is an interactive Database Management operation, not a user-facing CLI.
The installation runs at most one sequential generation at a time. The run drawer exposes safe
ordered events through SSE with polling fallback, Stop terminates the current helper while keeping
already stored results, and Run history retains terminal runs for inspection. A backend restart
marks queued or running work interrupted instead of resuming it; use Generate Missing to continue.
Unlock is reserved for a stale recorded run and is rejected while a local start, worker, or helper
is still live.
The bundle is mounted read-only as `/run/secrets/thothii.secrets` and must be mode `0600` or
`0400` on the host. Docker's runtime `0444` mode is accepted only beneath `/run/secrets`; see
[`deploy/secrets/README.md`](deploy/secrets/README.md). A PEM CA chain is deliberately not a
+51 -2
View File
@@ -2,7 +2,7 @@ import Fastify, { type FastifyInstance, type FastifyRequest } from "fastify";
import cors from "@fastify/cors";
import cookie from "@fastify/cookie";
import rateLimit from "@fastify/rate-limit";
import { join } from "node:path";
import { dirname, isAbsolute, join } from "node:path";
import { tmpdir } from "node:os";
import type { AppConfig } from "./config.js";
import { ThtRunner } from "./tht/tht-runner.js";
@@ -48,6 +48,19 @@ import { catalogTableRoutes } from "./routes/catalog-tables.js";
import { ConcreteCatalogSchemaIntrospector, type CatalogSchemaIntrospector } from "./catalog/schema-introspector.js";
import { CatalogSyncWorker } from "./catalog/sync-worker.js";
import { catalogSchemaRoutes } from "./routes/catalog-schema.js";
import {
loadMetadataGenerationModels,
type MetadataGenerationModels,
} from "./catalog/metadata-generation-models.js";
import { metadataGenerationModelRoutes } from "./routes/metadata-generation-models.js";
import { catalogDescriptionConsolidationRoutes } from "./routes/catalog-description-consolidation.js";
import { PythonModelCompleter, type ModelCompleter } from "./catalog/model-completer.js";
import { DescriptionGenerationWorker } from "./catalog/description-generation-worker.js";
import {
PostgresDescriptionSourceSampler,
type DescriptionSourceSampler,
} from "./catalog/description-source-sampler.js";
import { catalogDescriptionGenerationRoutes } from "./routes/catalog-description-generation.js";
export interface BuildAppDeps {
thtRunner?: ThtRunner;
@@ -67,6 +80,9 @@ export interface BuildAppDeps {
catalogSchemaIntrospector?: CatalogSchemaIntrospector;
catalogSyncWorker?: CatalogSyncWorker;
catalogOperationCoordinator?: CatalogOperationCoordinator;
metadataGenerationModels?: MetadataGenerationModels;
modelCompleter?: ModelCompleter;
descriptionSourceSampler?: DescriptionSourceSampler;
workspaceRuntimeSupport?: (workspace: WorkspaceDescriptor) => boolean;
maintenanceBarrier?: MaintenanceBarrier;
piManagement?: PiManagementService;
@@ -141,10 +157,28 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
const workspaceRegistry = deps?.workspaceRegistry ?? new WorkspaceRegistry(config.workspaceRegistry);
const catalogRepository = deps?.catalogRepository ?? createCatalogRepository(config.catalogDatabase);
const catalogOperationCoordinator = deps?.catalogOperationCoordinator ?? new CatalogOperationCoordinator();
const metadataGenerationModels = deps?.metadataGenerationModels ?? loadMetadataGenerationModels({
installationFile: config.installationConfigFile,
secretsFile: config.secretsFile,
});
const modelCompleter = deps?.modelCompleter ?? new PythonModelCompleter({
pythonExecutable: isAbsolute(config.thtBin) ? join(dirname(config.thtBin), "python") : "python3",
cwd: config.harnessDir,
});
const catalogPostgresAccess = deps?.catalogPostgresAccess ?? new ConcreteCatalogPostgresAccess(
workspaceSecretStore,
{ connectTimeoutMs: config.workspaceDiagnosticTimeoutMs },
);
const descriptionSourceSampler = deps?.descriptionSourceSampler
?? new PostgresDescriptionSourceSampler(catalogPostgresAccess);
const descriptionGenerationWorker = new DescriptionGenerationWorker(
catalogRepository,
workspaceRegistry,
metadataGenerationModels,
modelCompleter,
catalogOperationCoordinator,
descriptionSourceSampler,
);
const catalogService = deps?.catalogService ?? new CatalogService(
catalogRepository,
workspaceRegistry,
@@ -166,10 +200,12 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
config.catalogSyncTimeoutMs,
);
app.addHook("onReady", async () => { await catalogSyncWorker.initialize(); });
app.addHook("onReady", async () => { await descriptionGenerationWorker.initialize(); });
if (!deps?.catalogRepository && catalogRepository.close) {
app.addHook("onClose", async () => { await catalogRepository.close?.(); });
}
app.addHook("onClose", async () => { await catalogSyncWorker.stop(); });
app.addHook("onClose", async () => { await descriptionGenerationWorker.stop(); });
const workspaceDiagnoser = deps?.workspaceDiagnoser
?? createProductionWorkspaceDiagnoser(config.workspaceDiagnosticTimeoutMs, undefined, {
internalQdrantUrl: config.internalQdrantUrl,
@@ -384,12 +420,25 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
secretStore: workspaceSecretStore,
});
catalogDatabaseRoutes(app, { repository: catalogRepository, service: catalogService, operations: catalogOperationCoordinator });
catalogTableRoutes(app, { repository: catalogRepository, service: catalogTableService });
catalogTableRoutes(app, {
repository: catalogRepository,
service: catalogTableService,
operations: catalogOperationCoordinator,
});
catalogSchemaRoutes(app, {
repository: catalogRepository,
worker: catalogSyncWorker,
operations: catalogOperationCoordinator,
});
catalogDescriptionConsolidationRoutes(app, {
repository: catalogRepository,
operations: catalogOperationCoordinator,
});
metadataGenerationModelRoutes(app, metadataGenerationModels);
catalogDescriptionGenerationRoutes(app, {
repository: catalogRepository,
worker: descriptionGenerationWorker,
});
settingsRoutes(app, { cfg: config, listModels, getSettings });
piManagementRoutes(app, { service: piManagement });
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,163 @@
import type { CatalogPostgresAccess } from "./postgres-access.js";
import type { WorkspaceDatabase } from "./types.js";
const MAX_SOURCE_ROWS = 5;
const MAX_REPRESENTATIVE_VALUES = 5;
const MAX_SOURCE_COLUMNS_PER_TARGET = 8;
const MAX_SOURCE_VALUE_BYTES = 256;
export type DescriptionSourceSampleValue = string | number | boolean | null;
export interface DescriptionSourceSampleField {
name: string;
value: DescriptionSourceSampleValue;
}
export interface DescriptionSourceSampleRow {
fields: readonly DescriptionSourceSampleField[];
}
export interface DescriptionSourceRepresentativeValues {
column: string;
values: readonly Exclude<DescriptionSourceSampleValue, null>[];
}
export interface DescriptionTargetSourceSample {
targetId: string;
tableName: string;
rows: readonly DescriptionSourceSampleRow[];
representativeValues: readonly DescriptionSourceRepresentativeValues[];
}
export interface DescriptionSourceSamplingTarget {
targetId: string;
tableName: string;
columnNames: readonly string[];
}
/** Optional, transient source context for one model-completion batch. */
export interface DescriptionSourceSampler {
sample(
database: WorkspaceDatabase,
targets: readonly DescriptionSourceSamplingTarget[],
signal: AbortSignal,
): Promise<readonly DescriptionTargetSourceSample[]>;
}
function quoteIdentifier(identifier: string): string {
return `"${identifier.replaceAll('"', '""')}"`;
}
function boundedUtf8(value: string, maxBytes: number): string {
const normalized = value
.normalize("NFC")
.replace(/\r\n?/g, "\n")
.replace(/[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/g, " ");
if (Buffer.byteLength(normalized, "utf8") <= maxBytes) return normalized;
let result = "";
let bytes = 0;
for (const character of normalized) {
const characterBytes = Buffer.byteLength(character, "utf8");
if (bytes + characterBytes > maxBytes) break;
result += character;
bytes += characterBytes;
}
return result;
}
function normalizeValue(value: unknown): DescriptionSourceSampleValue | undefined {
if (value === null) return null;
if (typeof value === "string") return boundedUtf8(value, MAX_SOURCE_VALUE_BYTES);
if (typeof value === "boolean") return value;
if (typeof value === "number") return Number.isFinite(value) ? value : undefined;
if (typeof value === "bigint") return boundedUtf8(String(value), MAX_SOURCE_VALUE_BYTES);
if (value instanceof Date && !Number.isNaN(value.valueOf())) return value.toISOString();
return undefined;
}
function distinctKey(value: Exclude<DescriptionSourceSampleValue, null>): string {
return `${typeof value}:${String(value)}`;
}
/** PostgreSQL-wire sampler. REST bindings remain unsupported by CatalogPostgresAccess. */
export class PostgresDescriptionSourceSampler implements DescriptionSourceSampler {
constructor(private readonly access: CatalogPostgresAccess) {}
async sample(
database: WorkspaceDatabase,
targets: readonly DescriptionSourceSamplingTarget[],
signal: AbortSignal,
): Promise<readonly DescriptionTargetSourceSample[]> {
const client = await this.access.connect(database, signal);
let transactionOpen = false;
try {
await client.query("BEGIN TRANSACTION READ ONLY", []);
transactionOpen = true;
const samples: DescriptionTargetSourceSample[] = [];
for (const target of targets) {
const columnNames = [...new Set(target.columnNames)].slice(0, MAX_SOURCE_COLUMNS_PER_TARGET);
if (columnNames.length === 0) {
samples.push({
targetId: target.targetId,
tableName: target.tableName,
rows: [],
representativeValues: [],
});
continue;
}
const projections = columnNames.map((columnName) => {
const identifier = quoteIdentifier(columnName);
return `LEFT((${identifier})::text, $1) AS ${identifier}`;
});
const sql = [
`SELECT ${projections.join(", ")}`,
`FROM ${quoteIdentifier(database.schema)}.${quoteIdentifier(target.tableName)}`,
"LIMIT $2",
].join(" ");
const result = await client.query(sql, [MAX_SOURCE_VALUE_BYTES, MAX_SOURCE_ROWS]);
const rows = result.rows.slice(0, MAX_SOURCE_ROWS).map((row) => ({
fields: columnNames.flatMap((name) => {
const value = normalizeValue(row[name]);
return value === undefined ? [] : [{ name, value }];
}),
}));
const valuesByColumn = new Map<
string,
Exclude<DescriptionSourceSampleValue, null>[]
>();
const seenByColumn = new Map<string, Set<string>>();
let representativeValueCount = 0;
for (const row of rows) {
for (const field of row.fields) {
if (representativeValueCount === MAX_REPRESENTATIVE_VALUES) break;
if (field.value === null) continue;
const seen = seenByColumn.get(field.name) ?? new Set<string>();
const key = distinctKey(field.value);
if (seen.has(key)) continue;
seen.add(key);
seenByColumn.set(field.name, seen);
const values = valuesByColumn.get(field.name) ?? [];
values.push(field.value);
valuesByColumn.set(field.name, values);
representativeValueCount += 1;
}
if (representativeValueCount === MAX_REPRESENTATIVE_VALUES) break;
}
const representativeValues = columnNames.flatMap((column) => {
const values = valuesByColumn.get(column);
return values && values.length > 0 ? [{ column, values }] : [];
});
samples.push({
targetId: target.targetId,
tableName: target.tableName,
rows,
representativeValues,
});
}
return samples;
} finally {
if (transactionOpen) await client.query("ROLLBACK", []).catch(() => undefined);
await client.end().catch(() => undefined);
}
}
}
+174
View File
@@ -2,7 +2,10 @@ import { randomUUID } from "node:crypto";
import {
CatalogConflictError,
CatalogConnectorError,
DescriptionGenerationRunActiveError,
type CatalogColumn,
type CatalogDescriptionConsolidationCounts,
type CatalogDescriptionTarget,
type CatalogDatabaseMetadataDeleteTarget,
type CatalogMetadataDeleteCounts,
type CatalogRelationship,
@@ -19,6 +22,10 @@ import {
type DatabaseTestResult,
type ObservedCatalogTable,
type ObservedSchemaSnapshot,
type DescriptionGenerationEvent,
type DescriptionGenerationRun,
type DescriptionGenerationRunUpdate,
type DescriptionGenerationScope,
type TableSyncRepositoryResult,
type WorkspaceDatabase,
} from "./types.js";
@@ -33,6 +40,8 @@ export class MemoryCatalogRepository implements CatalogRepository {
private readonly tables = new Map<string, CatalogTable>();
private readonly columns = new Map<string, CatalogColumn>();
private readonly relationships = new Map<string, CatalogRelationship>();
private readonly descriptionGenerationRuns = new Map<string, DescriptionGenerationRun>();
private readonly descriptionGenerationEvents = new Map<string, DescriptionGenerationEvent[]>();
private readonly syncRuns = new Map<string, CatalogSyncRun>();
private readonly syncEvents = new Map<string, CatalogSyncEvent[]>();
@@ -122,6 +131,11 @@ export class MemoryCatalogRepository implements CatalogRepository {
for (const [relationshipId, relationship] of this.relationships) {
if (relationship.databaseId === id) this.relationships.delete(relationshipId);
}
for (const [runId, run] of this.descriptionGenerationRuns) {
if (run.databaseId !== id) continue;
this.descriptionGenerationRuns.delete(runId);
this.descriptionGenerationEvents.delete(runId);
}
return this.records.delete(id);
}
async listTables(databaseId: string): Promise<CatalogTable[]> {
@@ -196,6 +210,166 @@ export class MemoryCatalogRepository implements CatalogRepository {
return structuredClone(updated);
}
async consolidateGeneratedDescriptions(
databaseId: string,
target: CatalogDescriptionTarget,
targetIds: readonly string[],
): Promise<CatalogDescriptionConsolidationCounts | undefined> {
const selectedTargetIds = [...new Set(targetIds)];
if (!this.records.has(databaseId) || selectedTargetIds.length === 0) {
return undefined;
}
const now = new Date().toISOString();
if (target === "tables") {
const targets = selectedTargetIds.map((id) => this.tables.get(id));
if (targets.some((table) => !table || table.databaseId !== databaseId)) return undefined;
const copied = targets.filter((table) => Boolean(table!.generatedDescription?.trim()));
for (const table of copied) {
this.tables.set(table!.id, {
...table!,
description: table!.generatedDescription,
version: table!.version + 1,
updatedAt: now,
});
}
return { copied: copied.length, skipped: targets.length - copied.length };
}
const targets = selectedTargetIds.map((id) => this.columns.get(id));
if (targets.some((column) => (
!column || this.tables.get(column.tableId)?.databaseId !== databaseId
))) return undefined;
const copied = targets.filter((column) => Boolean(column!.generatedDescription?.trim()));
for (const column of copied) {
this.columns.set(column!.id, {
...column!,
description: column!.generatedDescription,
version: column!.version + 1,
updatedAt: now,
});
}
return { copied: copied.length, skipped: targets.length - copied.length };
}
async createDescriptionGenerationRun(
databaseId: string,
scope: DescriptionGenerationScope,
modelId: string,
language: DescriptionGenerationRun["language"],
total: number,
): Promise<DescriptionGenerationRun> {
if ([...this.descriptionGenerationRuns.values()].some((run) => (
run.status === "queued" || run.status === "running"
))) {
throw new DescriptionGenerationRunActiveError("A description generation run is already active");
}
const now = new Date().toISOString();
const run: DescriptionGenerationRun = {
id: randomUUID(),
databaseId,
scope,
modelId,
language,
status: "queued",
total,
processed: 0,
generated: 0,
nonGeneratable: 0,
failed: 0,
createdAt: now,
startedAt: null,
updatedAt: now,
finishedAt: null,
errorSummary: null,
};
this.descriptionGenerationRuns.set(run.id, run);
return structuredClone(run);
}
async getDescriptionGenerationRun(runId: string): Promise<DescriptionGenerationRun | undefined> {
const run = this.descriptionGenerationRuns.get(runId);
return run ? structuredClone(run) : undefined;
}
async listDescriptionGenerationRuns(limit = 50): Promise<DescriptionGenerationRun[]> {
return [...this.descriptionGenerationRuns.values()]
.sort((a, b) => b.createdAt.localeCompare(a.createdAt) || b.id.localeCompare(a.id))
.slice(0, limit)
.map((run) => structuredClone(run));
}
async getActiveDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined> {
const run = [...this.descriptionGenerationRuns.values()]
.filter((candidate) => candidate.status === "queued" || candidate.status === "running")
.sort((a, b) => b.createdAt.localeCompare(a.createdAt))[0];
return run ? structuredClone(run) : undefined;
}
async interruptActiveDescriptionGenerationRuns(
errorSummary: string,
): Promise<DescriptionGenerationRun[]> {
const interrupted: DescriptionGenerationRun[] = [];
for (const run of this.descriptionGenerationRuns.values()) {
if (run.status !== "queued" && run.status !== "running") continue;
const now = new Date().toISOString();
const updated: DescriptionGenerationRun = {
...run,
status: "interrupted",
updatedAt: now,
finishedAt: now,
errorSummary,
};
this.descriptionGenerationRuns.set(run.id, updated);
interrupted.push(structuredClone(updated));
}
return interrupted;
}
async updateDescriptionGenerationRun(
runId: string,
update: DescriptionGenerationRunUpdate,
): Promise<DescriptionGenerationRun | undefined> {
const current = this.descriptionGenerationRuns.get(runId);
if (!current) return undefined;
const updated = {
...current,
...structuredClone(update),
updatedAt: new Date().toISOString(),
};
this.descriptionGenerationRuns.set(runId, updated);
return structuredClone(updated);
}
async appendDescriptionGenerationEvent(
runId: string,
level: DescriptionGenerationEvent["level"],
message: string,
): Promise<DescriptionGenerationEvent> {
if (!this.descriptionGenerationRuns.has(runId)) {
throw new CatalogConflictError("Description Generation Run does not exist");
}
const events = this.descriptionGenerationEvents.get(runId) ?? [];
const event: DescriptionGenerationEvent = {
runId,
sequence: events.length + 1,
level,
message,
createdAt: new Date().toISOString(),
};
events.push(event);
this.descriptionGenerationEvents.set(runId, events);
return structuredClone(event);
}
async listDescriptionGenerationEvents(
runId: string,
afterSequence = 0,
): Promise<DescriptionGenerationEvent[]> {
return (this.descriptionGenerationEvents.get(runId) ?? [])
.filter((event) => event.sequence > afterSequence)
.map((event) => structuredClone(event));
}
async listRelationships(databaseId: string): Promise<CatalogRelationship[]> {
return [...this.relationships.values()].filter((relationship) => relationship.databaseId === databaseId)
.sort((a, b) => `${a.sourceTableName}.${a.constraintName}`.localeCompare(`${b.sourceTableName}.${b.constraintName}`))
@@ -0,0 +1,235 @@
import {
closeSync, constants, fstatSync, lstatSync, openSync, readFileSync,
type Stats,
} from "node:fs";
import { parseAllDocuments } from "yaml";
import { z } from "zod";
import {
loadSecretBundle,
METADATA_GENERATION_SECRET_KEYS,
} from "../config/secret-bundle.js";
const MAX_INSTALLATION_BYTES = 1024 * 1024;
const RUNTIME_INSTALLATION_FILE = "/run/thothii-installation/thothii-installation.yaml";
const modelId = z.string().regex(/^[a-z][a-z0-9._-]{0,63}$/);
const apiKeyEnvironment = z.enum(METADATA_GENERATION_SECRET_KEYS);
const endpointSchema = z.object({
baseUrl: z.string().min(1).max(2048).refine((value) => {
try {
const url = new URL(value);
return (url.protocol === "http:" || url.protocol === "https:")
&& url.username === "" && url.password === "" && url.search === "" && url.hash === "";
} catch {
return false;
}
}),
apiVersion: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/).optional(),
}).strict();
const configuredModelSchema = z.object({
id: modelId,
label: z.string().min(1).max(128).refine((value) => value.trim() === value && !/\p{Cc}/u.test(value)),
litellm: z.object({
provider: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/),
model: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$/),
disableThinking: z.literal(true).optional(),
endpoint: endpointSchema.optional(),
}).strict(),
apiKeyEnv: apiKeyEnvironment.optional(),
}).strict().superRefine((value, context) => {
if (value.apiKeyEnv === undefined && value.litellm.endpoint === undefined) {
context.addIssue({
code: z.ZodIssueCode.custom,
path: ["apiKeyEnv"],
message: "keyless models require an explicit endpoint",
});
}
if (value.litellm.disableThinking === true && value.litellm.endpoint === undefined) {
context.addIssue({
code: z.ZodIssueCode.custom,
path: ["litellm", "disableThinking"],
message: "thinking may be disabled only for an explicit endpoint",
});
}
});
const metadataGenerationSchema = z.object({
default: modelId.optional(),
models: z.array(configuredModelSchema).max(64).default([]),
}).strict();
const installationSchema = z.object({
metadataGeneration: metadataGenerationSchema.optional(),
}).passthrough();
export interface MetadataGenerationModelChoice {
id: string;
label: string;
}
export interface MetadataGenerationModelCatalog {
models: MetadataGenerationModelChoice[];
default: string | null;
}
export interface ResolvedMetadataGenerationModel {
readonly id: string;
readonly provider: string;
readonly model: string;
readonly disableThinking?: true;
readonly endpoint?: Readonly<{ baseUrl: string; apiVersion?: string }>;
readonly apiKeyEnv?: string;
readonly apiKey?: string;
}
export class MetadataGenerationModelUnavailableError extends Error {
constructor() {
super("metadata-generation model is unavailable");
this.name = "MetadataGenerationModelUnavailableError";
}
}
/** The complete interface callers need: safe discovery plus fail-closed runtime resolution. */
export interface MetadataGenerationModels {
catalog(): MetadataGenerationModelCatalog;
resolve(selection: string): ResolvedMetadataGenerationModel;
}
class RestartLoadedMetadataGenerationModels implements MetadataGenerationModels {
readonly #models: ReadonlyMap<string, ResolvedMetadataGenerationModel>;
readonly #catalog: MetadataGenerationModelCatalog;
constructor(
models: ReadonlyMap<string, ResolvedMetadataGenerationModel> = new Map(),
defaultModel: string | null = null,
choices: MetadataGenerationModelChoice[] = [],
) {
this.#models = models;
this.#catalog = {
models: choices.map((choice) => ({ ...choice })),
default: defaultModel,
};
}
catalog(): MetadataGenerationModelCatalog {
return {
models: this.#catalog.models.map((choice) => ({ ...choice })),
default: this.#catalog.default,
};
}
resolve(selection: string): ResolvedMetadataGenerationModel {
const model = typeof selection === "string" ? this.#models.get(selection) : undefined;
if (!model) throw new MetadataGenerationModelUnavailableError();
return model;
}
}
function invalid(message = "metadata-generation configuration is invalid"): Error {
return new Error(message);
}
function protectedInstallationStat(file: string, info: Stats): boolean {
const mode = info.mode & 0o777;
if (!info.isFile() || info.isSymbolicLink() || info.nlink !== 1
|| info.size < 1 || info.size > MAX_INSTALLATION_BYTES) return false;
if (file === RUNTIME_INSTALLATION_FILE && info.uid === 0 && mode === 0o444) return true;
return info.uid === (process.getuid?.() ?? info.uid) && (mode === 0o400 || mode === 0o600);
}
function readProtectedInstallation(file: string): string {
let descriptor: number | undefined;
try {
const before = lstatSync(file);
if (!protectedInstallationStat(file, before)) throw new Error("unavailable");
descriptor = openSync(file, constants.O_RDONLY | constants.O_NOFOLLOW);
const opened = fstatSync(descriptor);
if (!protectedInstallationStat(file, opened)
|| before.dev !== opened.dev || before.ino !== opened.ino) throw new Error("unavailable");
const source = readFileSync(descriptor, "utf8");
const after = fstatSync(descriptor);
const current = lstatSync(file);
if (!protectedInstallationStat(file, after) || !protectedInstallationStat(file, current)
|| opened.dev !== after.dev || opened.ino !== after.ino
|| opened.dev !== current.dev || opened.ino !== current.ino) throw new Error("unavailable");
return source;
} finally {
if (descriptor !== undefined) try { closeSync(descriptor); } catch { /* sanitized below */ }
}
}
function readInstallation(file: string): unknown {
try {
const documents = parseAllDocuments(readProtectedInstallation(file), { uniqueKeys: true });
if (documents.length !== 1) throw invalid("metadata-generation installation must contain one YAML document");
const document = documents[0];
if (document.errors.length > 0 || document.warnings.length > 0) {
throw invalid("metadata-generation installation contains invalid YAML");
}
return document.toJSON();
} catch (error) {
if (error instanceof Error && error.message.startsWith("metadata-generation")) throw error;
throw invalid("metadata-generation installation is unavailable");
}
}
export function loadMetadataGenerationModels(options: {
installationFile?: string;
secretsFile?: string;
}): MetadataGenerationModels {
if (!options.installationFile) return new RestartLoadedMetadataGenerationModels();
const installation = installationSchema.safeParse(readInstallation(options.installationFile));
if (!installation.success) throw invalid();
const configured = installation.data.metadataGeneration;
if (!configured || configured.models.length === 0) {
if (configured?.default !== undefined) throw invalid("metadata-generation default does not identify a configured model");
return new RestartLoadedMetadataGenerationModels();
}
if (!configured.default) throw invalid("metadata-generation default is required when models are configured");
const seen = new Set<string>();
for (const model of configured.models) {
if (seen.has(model.id)) throw invalid(`metadata-generation model id "${model.id}" is duplicated`);
seen.add(model.id);
}
if (!seen.has(configured.default)) {
throw invalid(`metadata-generation default "${configured.default}" is not configured`);
}
const requiresSecrets = configured.models.some((model) => model.apiKeyEnv !== undefined);
let secrets: ReadonlyMap<string, string> = new Map();
if (requiresSecrets) {
if (!options.secretsFile) throw invalid("metadata-generation keyed models require THT_SECRETS_FILE");
try {
secrets = loadSecretBundle(options.secretsFile);
} catch {
throw invalid("metadata-generation secrets are unavailable");
}
}
const models = new Map<string, ResolvedMetadataGenerationModel>();
for (const configuredModel of configured.models) {
let apiKey: string | undefined;
if (configuredModel.apiKeyEnv !== undefined) {
apiKey = secrets.get(configuredModel.apiKeyEnv);
if (!apiKey) {
throw invalid(`metadata-generation model "${configuredModel.id}" secret "${configuredModel.apiKeyEnv}" is missing`);
}
if (apiKey.length > 16 * 1024 || /\s/u.test(apiKey)) {
throw invalid(`metadata-generation model "${configuredModel.id}" secret "${configuredModel.apiKeyEnv}" is unusable`);
}
}
models.set(configuredModel.id, Object.freeze({
id: configuredModel.id,
provider: configuredModel.litellm.provider,
model: configuredModel.litellm.model,
...(configuredModel.litellm.disableThinking === true ? { disableThinking: true as const } : {}),
...(configuredModel.litellm.endpoint === undefined
? {}
: { endpoint: Object.freeze({ ...configuredModel.litellm.endpoint }) }),
...(configuredModel.apiKeyEnv === undefined
? {}
: { apiKeyEnv: configuredModel.apiKeyEnv, apiKey }),
}));
}
return new RestartLoadedMetadataGenerationModels(
models,
configured.default,
configured.models.map(({ id, label }) => ({ id, label })),
);
}
+2
View File
@@ -7,6 +7,7 @@ import * as initialMigration from "./migrations/001_workspace_databases.js";
import * as catalogTablesMigration from "./migrations/002_catalog_tables.js";
import * as catalogSchemaSyncMigration from "./migrations/003_catalog_schema_sync.js";
import * as catalogRuntimeSequencePrivilegesMigration from "./migrations/004_catalog_runtime_sequence_privileges.js";
import * as descriptionGenerationRunsMigration from "./migrations/005_description_generation_runs.js";
const connectionString = process.env.THT_CATALOG_MIGRATOR_DATABASE_URL;
const host = process.env.THT_CATALOG_DB_HOST;
@@ -36,6 +37,7 @@ const provider: MigrationProvider = {
"002_catalog_tables": catalogTablesMigration,
"003_catalog_schema_sync": catalogSchemaSyncMigration,
"004_catalog_runtime_sequence_privileges": catalogRuntimeSequencePrivilegesMigration,
"005_description_generation_runs": descriptionGenerationRunsMigration,
};
},
};
@@ -0,0 +1,87 @@
import { sql, type Kysely } from "kysely";
import type { CatalogDatabase } from "../repository.js";
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.createTable("description_generation_runs")
.addColumn("id", "uuid", (column) => column.primaryKey())
.addColumn("database_id", "uuid", (column) => column.notNull()
.references("workspace_databases.id").onDelete("cascade"))
.addColumn("scope", "text", (column) => column.notNull())
.addColumn("model_id", "text", (column) => column.notNull())
.addColumn("language", "text", (column) => column.notNull())
.addColumn("status", "text", (column) => column.notNull())
.addColumn("total", "integer", (column) => column.notNull())
.addColumn("processed", "integer", (column) => column.notNull().defaultTo(0))
.addColumn("generated", "integer", (column) => column.notNull().defaultTo(0))
.addColumn("non_generatable", "integer", (column) => column.notNull().defaultTo(0))
.addColumn("failed", "integer", (column) => column.notNull().defaultTo(0))
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addColumn("started_at", "timestamptz")
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addColumn("finished_at", "timestamptz")
.addColumn("error_summary", "text")
.addCheckConstraint(
"description_generation_runs_scope_check",
sql`scope in ('selected_columns', 'selected_tables', 'all', 'missing')`,
)
.addCheckConstraint(
"description_generation_runs_model_id_check",
sql`model_id ~ '^[a-z][a-z0-9._-]{0,63}$'`,
)
.addCheckConstraint(
"description_generation_runs_language_check",
sql`language in ('en', 'it')`,
)
.addCheckConstraint(
"description_generation_runs_status_check",
sql`status in (
'queued', 'running', 'completed', 'completed_with_errors',
'cancelled', 'failed', 'interrupted'
)`,
)
.addCheckConstraint(
"description_generation_runs_counters_check",
sql`total > 0
and processed between 0 and total
and generated >= 0
and non_generatable >= 0
and failed >= 0
and generated + non_generatable + failed <= processed`,
)
.addCheckConstraint(
"description_generation_runs_error_summary_check",
sql`error_summary is null or char_length(error_summary) between 1 and 2000`,
)
.execute();
await db.schema.createIndex("description_generation_runs_database_created_idx")
.on("description_generation_runs")
.columns(["database_id", "created_at"])
.execute();
await sql`CREATE UNIQUE INDEX description_generation_runs_one_active
ON description_generation_runs ((true))
WHERE status IN ('queued', 'running')`.execute(db);
await db.schema.createTable("description_generation_events")
.addColumn("run_id", "uuid", (column) => column.notNull()
.references("description_generation_runs.id").onDelete("cascade"))
.addColumn("sequence", "integer", (column) => column.notNull())
.addColumn("level", "text", (column) => column.notNull())
.addColumn("message", "text", (column) => column.notNull())
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addPrimaryKeyConstraint("description_generation_events_pkey", ["run_id", "sequence"])
.addCheckConstraint("description_generation_events_sequence_check", sql`sequence > 0`)
.addCheckConstraint(
"description_generation_events_level_check",
sql`level in ('info', 'warning', 'error')`,
)
.addCheckConstraint(
"description_generation_events_message_check",
sql`char_length(message) between 1 and 2000`,
)
.execute();
}
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.dropTable("description_generation_events").execute();
await db.schema.dropTable("description_generation_runs").execute();
}
+150
View File
@@ -0,0 +1,150 @@
import { spawn } from "node:child_process";
import { z } from "zod";
import type { ResolvedMetadataGenerationModel } from "./metadata-generation-models.js";
const MAX_HELPER_OUTPUT_BYTES = 64 * 1024;
const helperOutputSchema = z.discriminatedUnion("ok", [
z.object({ ok: z.literal(true), content: z.string() }).strict(),
z.object({ ok: z.literal(false), error: z.literal("provider_failure") }).strict(),
]);
export interface ModelCompletionMessage {
role: "system" | "user";
content: string;
}
export interface ModelCompletionRequest {
model: ResolvedMetadataGenerationModel;
messages: readonly ModelCompletionMessage[];
signal: AbortSignal;
}
/** The provider boundary used by Description Generation. */
export interface ModelCompleter {
complete(request: ModelCompletionRequest): Promise<string>;
}
export class ModelCompletionProviderError extends Error {
constructor() {
super("model completion failed");
this.name = "ModelCompletionProviderError";
}
}
export class ModelCompletionCancelledError extends Error {
constructor() {
super("model completion cancelled");
this.name = "ModelCompletionCancelledError";
}
}
export class PythonModelCompleter implements ModelCompleter {
constructor(private readonly options: {
pythonExecutable: string;
cwd: string;
helperModule?: string;
timeoutMs?: number;
terminationGraceMs?: number;
}) {}
async complete(request: ModelCompletionRequest): Promise<string> {
if (request.signal.aborted) throw new ModelCompletionCancelledError();
const payload = {
model: `${request.model.provider}/${request.model.model}`,
...(request.model.apiKey === undefined ? {} : { api_key: request.model.apiKey }),
messages: request.messages.map((message) => ({ ...message })),
...(request.model.endpoint?.baseUrl === undefined
? {}
: { api_base: request.model.endpoint.baseUrl }),
...(request.model.endpoint?.apiVersion === undefined
? {}
: { api_version: request.model.endpoint.apiVersion }),
...(request.model.disableThinking === true ? { disable_thinking: true } : {}),
};
return await new Promise<string>((resolve, reject) => {
const child = spawn(
this.options.pythonExecutable,
["-m", this.options.helperModule ?? "tht.internal.litellm_completion"],
{
cwd: this.options.cwd,
stdio: ["pipe", "pipe", "pipe"],
},
);
let stdout = "";
let settled = false;
let timeout: ReturnType<typeof setTimeout> | undefined;
let killFallback: ReturnType<typeof setTimeout> | undefined;
let terminatingWith: Error | undefined;
const cleanup = () => {
if (timeout) clearTimeout(timeout);
if (killFallback) clearTimeout(killFallback);
request.signal.removeEventListener("abort", cancel);
};
const fail = (error: Error = new ModelCompletionProviderError()) => {
if (settled) return;
settled = true;
cleanup();
reject(error);
};
const terminate = (error: Error) => {
if (settled || terminatingWith) return;
terminatingWith = error;
if (timeout) clearTimeout(timeout);
try {
child.kill("SIGTERM");
} catch {
fail(error);
return;
}
killFallback = setTimeout(() => {
if (settled || child.exitCode !== null || child.signalCode !== null) return;
try {
child.kill("SIGKILL");
} catch {
fail(error);
}
}, this.options.terminationGraceMs ?? 250);
};
const cancel = () => {
terminate(new ModelCompletionCancelledError());
};
timeout = setTimeout(() => {
terminate(new ModelCompletionProviderError());
}, this.options.timeoutMs ?? 120_000);
request.signal.addEventListener("abort", cancel, { once: true });
if (request.signal.aborted) cancel();
child.stdout.setEncoding("utf8");
child.stdout.on("data", (chunk: string) => {
if (terminatingWith) return;
stdout += chunk;
if (Buffer.byteLength(stdout, "utf8") > MAX_HELPER_OUTPUT_BYTES) {
terminate(new ModelCompletionProviderError());
}
});
// Helper and provider diagnostics are deliberately not copied into application logs.
child.stderr.resume();
child.once("error", () => fail(terminatingWith ?? new ModelCompletionProviderError()));
child.once("close", (code) => {
if (settled) return;
if (terminatingWith) return fail(terminatingWith);
try {
if (code !== 0) return fail();
const output = helperOutputSchema.parse(JSON.parse(stdout));
if (!output.ok) return fail();
settled = true;
cleanup();
resolve(output.content);
} catch {
fail();
}
});
child.stdin.once("error", () => {
if (terminatingWith) return;
terminate(new ModelCompletionProviderError());
});
child.stdin.end(JSON.stringify(payload));
});
}
}
+17 -5
View File
@@ -1,22 +1,34 @@
import { CatalogOperationInProgressError } from "./types.js";
/** Serializes connection tests, synchronization, and metadata cleanup for each catalog database. */
/** Serializes mutable catalog operations for each Workspace Database. */
export class CatalogOperationCoordinator {
private readonly active = new Set<string>();
private readonly active = new Map<
string,
{ token: symbol; owner: "catalog_operation" | "description_generation" }
>();
reserve(databaseId: string): () => void {
reserve(
databaseId: string,
owner: "catalog_operation" | "description_generation" = "catalog_operation",
): () => void {
if (this.active.has(databaseId)) {
throw new CatalogOperationInProgressError("A database operation is already in progress");
}
this.active.add(databaseId);
const token = Symbol(databaseId);
this.active.set(databaseId, { token, owner });
let released = false;
return () => {
if (released) return;
released = true;
this.active.delete(databaseId);
if (this.active.get(databaseId)?.token === token) this.active.delete(databaseId);
};
}
/** Administrative recovery for a reservation whose owning local operation is no longer live. */
releaseStale(databaseId: string, owner: "description_generation"): void {
if (this.active.get(databaseId)?.owner === owner) this.active.delete(databaseId);
}
async run<T>(databaseId: string, operation: () => Promise<T>): Promise<T> {
const release = this.reserve(databaseId);
try {
+5 -1
View File
@@ -180,13 +180,17 @@ export class ConcreteCatalogPostgresAccess implements CatalogPostgresAccess {
materialized.release();
};
const abort = () => { void close(); };
signal.addEventListener("abort", abort, { once: true });
try {
if (signal.aborted) throw new CatalogConnectorError("PostgreSQL connector aborted");
const passwordFile = materialized.files.get(CATALOG_SECRET_IDS.password);
if (!passwordFile) throw new CatalogConnectorError("Database password is not configured");
const password = await readFile(passwordFile, "utf8");
if (signal.aborted) throw new CatalogConnectorError("PostgreSQL connector aborted");
const tlsCaFile = materialized.files.get(CATALOG_SECRET_IDS.tlsCa);
const tlsCa = tlsCaFile ? await readFile(tlsCaFile, "utf8") : undefined;
if (signal.aborted) throw new CatalogConnectorError("PostgreSQL connector aborted");
let host: string;
let port: number;
@@ -243,8 +247,8 @@ export class ConcreteCatalogPostgresAccess implements CatalogPostgresAccess {
connectionTimeoutMillis: this.connectTimeoutMs,
...(stream ? { stream: () => stream } : {}),
});
signal.addEventListener("abort", abort, { once: true });
await client.connect();
if (signal.aborted) throw new CatalogConnectorError("PostgreSQL connector aborted");
return {
query: async (sql, values) => await client!.query(sql, [...values]),
end: close,
+260
View File
@@ -15,7 +15,10 @@ import {
CatalogConflictError,
CatalogConnectorError,
CatalogUnavailableError,
DescriptionGenerationRunActiveError,
type CatalogColumn,
type CatalogDescriptionConsolidationCounts,
type CatalogDescriptionTarget,
type CatalogDatabaseMetadataDeleteTarget,
type CatalogMetadataDeleteCounts,
type CatalogRelationship,
@@ -31,6 +34,10 @@ import {
type DatabaseBinding,
type DatabaseConfigurationInput,
type DatabaseTestResult,
type DescriptionGenerationEvent,
type DescriptionGenerationRun,
type DescriptionGenerationRunUpdate,
type DescriptionGenerationScope,
type ObservedCatalogTable,
type ObservedSchemaSnapshot,
type TableSyncRepositoryResult,
@@ -131,6 +138,33 @@ interface CatalogRelationshipColumnTable {
targetColumnId: string;
}
interface DescriptionGenerationRunTable {
id: string;
databaseId: string;
scope: DescriptionGenerationScope;
modelId: string;
language: DescriptionGenerationRun["language"];
status: DescriptionGenerationRun["status"];
total: number;
processed: number;
generated: number;
nonGeneratable: number;
failed: number;
createdAt: Timestamp;
startedAt: Timestamp | null;
updatedAt: Timestamp;
finishedAt: Timestamp | null;
errorSummary: string | null;
}
interface DescriptionGenerationEventTable {
runId: string;
sequence: number;
level: DescriptionGenerationEvent["level"];
message: string;
createdAt: Timestamp;
}
interface CatalogSyncRunTable {
id: string;
databaseId: string;
@@ -176,6 +210,8 @@ export interface CatalogDatabase {
catalogColumns: CatalogColumnTable;
catalogRelationships: CatalogRelationshipTable;
catalogRelationshipColumns: CatalogRelationshipColumnTable;
descriptionGenerationRuns: DescriptionGenerationRunTable;
descriptionGenerationEvents: DescriptionGenerationEventTable;
catalogSyncRuns: CatalogSyncRunTable;
catalogSyncEvents: CatalogSyncEventTable;
}
@@ -281,6 +317,25 @@ function serializeSyncEvent(row: Selectable<CatalogSyncEventTable>): CatalogSync
return { ...row, id: Number(row.id), data: row.data ?? {}, createdAt: new Date(row.createdAt).toISOString() };
}
function serializeDescriptionGenerationRun(
row: Selectable<DescriptionGenerationRunTable>,
): DescriptionGenerationRun {
const stamp = (value: Date | string | null) => value === null ? null : new Date(value).toISOString();
return {
...row,
createdAt: new Date(row.createdAt).toISOString(),
startedAt: stamp(row.startedAt),
updatedAt: new Date(row.updatedAt).toISOString(),
finishedAt: stamp(row.finishedAt),
};
}
function serializeDescriptionGenerationEvent(
row: Selectable<DescriptionGenerationEventTable>,
): DescriptionGenerationEvent {
return { ...row, createdAt: new Date(row.createdAt).toISOString() };
}
function bindingValues(databaseId: string, binding: DatabaseBinding) {
return {
databaseId,
@@ -520,6 +575,202 @@ export class KyselyCatalogRepository implements CatalogRepository {
return row ? await this.getColumn(databaseId, tableId, row.id) : undefined;
}
async consolidateGeneratedDescriptions(
databaseId: string,
target: CatalogDescriptionTarget,
targetIds: readonly string[],
): Promise<CatalogDescriptionConsolidationCounts | undefined> {
const selectedTargetIds = [...new Set(targetIds)];
if (selectedTargetIds.length === 0) return undefined;
return await this.db.transaction().execute(async (trx) => {
const database = await trx.selectFrom("workspaceDatabases").select("id")
.where("id", "=", databaseId).forUpdate().executeTakeFirst();
if (!database) return undefined;
if (target === "tables") {
const rows = await trx.selectFrom("catalogTables")
.select(["id", "generatedDescription"])
.where("databaseId", "=", databaseId)
.where("id", "in", selectedTargetIds)
.orderBy("id")
.forUpdate()
.execute();
if (rows.length !== selectedTargetIds.length) return undefined;
const copiedIds = rows
.filter((row) => Boolean(row.generatedDescription?.trim()))
.map((row) => row.id);
if (copiedIds.length > 0) {
await trx.updateTable("catalogTables").set({
description: sql`generated_description`,
version: sql`version + 1`,
updatedAt: sql`now()`,
}).where("id", "in", copiedIds).execute();
}
return {
copied: copiedIds.length,
skipped: selectedTargetIds.length - copiedIds.length,
};
}
const tableRows = await trx.selectFrom("catalogTables").select("id")
.where("databaseId", "=", databaseId).execute();
const rows = tableRows.length === 0 ? [] : await trx.selectFrom("catalogColumns")
.select(["id", "generatedDescription"])
.where("tableId", "in", tableRows.map((table) => table.id))
.where("id", "in", selectedTargetIds)
.orderBy("id")
.forUpdate()
.execute();
if (rows.length !== selectedTargetIds.length) return undefined;
const copiedIds = rows
.filter((row) => Boolean(row.generatedDescription?.trim()))
.map((row) => row.id);
if (copiedIds.length > 0) {
await trx.updateTable("catalogColumns").set({
description: sql`generated_description`,
version: sql`version + 1`,
updatedAt: sql`now()`,
}).where("id", "in", copiedIds).execute();
}
return {
copied: copiedIds.length,
skipped: selectedTargetIds.length - copiedIds.length,
};
});
}
async createDescriptionGenerationRun(
databaseId: string,
scope: DescriptionGenerationScope,
modelId: string,
language: DescriptionGenerationRun["language"],
total: number,
): Promise<DescriptionGenerationRun> {
try {
const row = await this.db.insertInto("descriptionGenerationRuns").values({
id: randomUUID(),
databaseId,
scope,
modelId,
language,
status: "queued",
total,
processed: 0,
generated: 0,
nonGeneratable: 0,
failed: 0,
startedAt: null,
finishedAt: null,
errorSummary: null,
}).returningAll().executeTakeFirstOrThrow();
return serializeDescriptionGenerationRun(row);
} catch (error: any) {
if (error?.code === "23505"
&& error?.constraint === "description_generation_runs_one_active") {
throw new DescriptionGenerationRunActiveError(
"A description generation run is already active",
);
}
throw error;
}
}
async getDescriptionGenerationRun(
runId: string,
): Promise<DescriptionGenerationRun | undefined> {
const row = await this.db.selectFrom("descriptionGenerationRuns")
.selectAll()
.where("id", "=", runId)
.executeTakeFirst();
return row ? serializeDescriptionGenerationRun(row) : undefined;
}
async listDescriptionGenerationRuns(limit = 50): Promise<DescriptionGenerationRun[]> {
const rows = await this.db.selectFrom("descriptionGenerationRuns")
.selectAll()
.orderBy("createdAt", "desc")
.orderBy("id", "desc")
.limit(limit)
.execute();
return rows.map(serializeDescriptionGenerationRun);
}
async getActiveDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined> {
const row = await this.db.selectFrom("descriptionGenerationRuns")
.selectAll()
.where("status", "in", ["queued", "running"])
.orderBy("createdAt", "desc")
.executeTakeFirst();
return row ? serializeDescriptionGenerationRun(row) : undefined;
}
async interruptActiveDescriptionGenerationRuns(
errorSummary: string,
): Promise<DescriptionGenerationRun[]> {
const rows = await this.db.updateTable("descriptionGenerationRuns")
.set({
status: "interrupted",
finishedAt: sql`now()`,
updatedAt: sql`now()`,
errorSummary,
})
.where("status", "in", ["queued", "running"])
.returningAll()
.execute();
return rows.map(serializeDescriptionGenerationRun);
}
async updateDescriptionGenerationRun(
runId: string,
update: DescriptionGenerationRunUpdate,
): Promise<DescriptionGenerationRun | undefined> {
const values: any = { ...update, updatedAt: sql`now()` };
const row = await this.db.updateTable("descriptionGenerationRuns")
.set(values)
.where("id", "=", runId)
.returningAll()
.executeTakeFirst();
return row ? serializeDescriptionGenerationRun(row) : undefined;
}
async appendDescriptionGenerationEvent(
runId: string,
level: DescriptionGenerationEvent["level"],
message: string,
): Promise<DescriptionGenerationEvent> {
return await this.db.transaction().execute(async (trx) => {
const run = await trx.selectFrom("descriptionGenerationRuns")
.select("id")
.where("id", "=", runId)
.forUpdate()
.executeTakeFirst();
if (!run) throw new CatalogConflictError("Description Generation Run does not exist");
const current = await trx.selectFrom("descriptionGenerationEvents")
.select(sql<number>`coalesce(max(sequence), 0)::int`.as("sequence"))
.where("runId", "=", runId)
.executeTakeFirst();
const row = await trx.insertInto("descriptionGenerationEvents").values({
runId,
sequence: Number(current?.sequence ?? 0) + 1,
level,
message,
}).returningAll().executeTakeFirstOrThrow();
return serializeDescriptionGenerationEvent(row);
});
}
async listDescriptionGenerationEvents(
runId: string,
afterSequence = 0,
): Promise<DescriptionGenerationEvent[]> {
const rows = await this.db.selectFrom("descriptionGenerationEvents")
.selectAll()
.where("runId", "=", runId)
.where("sequence", ">", afterSequence)
.orderBy("sequence")
.execute();
return rows.map(serializeDescriptionGenerationEvent);
}
async listRelationships(databaseId: string): Promise<CatalogRelationship[]> {
const rows = await this.db.selectFrom("catalogRelationships as relationship")
.innerJoin("catalogTables as sourceTable", "sourceTable.id", "relationship.sourceTableId")
@@ -1102,6 +1353,15 @@ export class UnavailableCatalogRepository implements CatalogRepository {
async listColumns(): Promise<CatalogColumn[]> { return this.fail(); }
async getColumn(): Promise<CatalogColumn | undefined> { return this.fail(); }
async updateColumnMetadata(): Promise<CatalogColumn | undefined> { return this.fail(); }
async consolidateGeneratedDescriptions(): Promise<CatalogDescriptionConsolidationCounts | undefined> { return this.fail(); }
async createDescriptionGenerationRun(): Promise<DescriptionGenerationRun> { return this.fail(); }
async getDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined> { return this.fail(); }
async listDescriptionGenerationRuns(): Promise<DescriptionGenerationRun[]> { return this.fail(); }
async getActiveDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined> { return this.fail(); }
async interruptActiveDescriptionGenerationRuns(): Promise<DescriptionGenerationRun[]> { return this.fail(); }
async updateDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined> { return this.fail(); }
async appendDescriptionGenerationEvent(): Promise<DescriptionGenerationEvent> { return this.fail(); }
async listDescriptionGenerationEvents(): Promise<DescriptionGenerationEvent[]> { return this.fail(); }
async listRelationships(): Promise<CatalogRelationship[]> { return this.fail(); }
async deleteDatabaseMetadata(): Promise<CatalogMetadataDeleteCounts | undefined> { return this.fail(); }
async deleteTableMetadata(): Promise<CatalogMetadataDeleteCounts | undefined> { return this.fail(); }
+90
View File
@@ -135,6 +135,7 @@ export interface CatalogRelationship {
export type CatalogDatabaseMetadataDeleteTarget = "tables" | "relationships";
export type CatalogTableMetadataDeleteTarget = "columns" | "relationships";
export type CatalogDescriptionTarget = "tables" | "columns";
export interface CatalogMetadataDeleteCounts {
tables: number;
@@ -142,6 +143,63 @@ export interface CatalogMetadataDeleteCounts {
relationships: number;
}
export interface CatalogDescriptionConsolidationCounts {
copied: number;
skipped: number;
}
export type DescriptionGenerationScope =
| "selected_columns"
| "selected_tables"
| "all"
| "missing";
export type DescriptionGenerationStatus =
| "queued"
| "running"
| "completed"
| "completed_with_errors"
| "cancelled"
| "failed"
| "interrupted";
export interface DescriptionGenerationRun {
id: string;
databaseId: string;
scope: DescriptionGenerationScope;
modelId: string;
language: "en" | "it";
status: DescriptionGenerationStatus;
total: number;
processed: number;
generated: number;
nonGeneratable: number;
failed: number;
createdAt: string;
startedAt: string | null;
updatedAt: string;
finishedAt: string | null;
errorSummary: string | null;
}
export interface DescriptionGenerationRunUpdate {
status?: DescriptionGenerationStatus;
processed?: number;
generated?: number;
nonGeneratable?: number;
failed?: number;
startedAt?: string | null;
finishedAt?: string | null;
errorSummary?: string | null;
}
export interface DescriptionGenerationEvent {
runId: string;
sequence: number;
level: "info" | "warning" | "error";
message: string;
createdAt: string;
}
export interface ObservedRelationshipColumn {
position: number;
sourceColumnName: string;
@@ -292,6 +350,37 @@ export interface CatalogRepository {
description: string | null,
generatedDescription: string | null,
): Promise<CatalogColumn | undefined>;
consolidateGeneratedDescriptions(
databaseId: string,
target: CatalogDescriptionTarget,
targetIds: readonly string[],
): Promise<CatalogDescriptionConsolidationCounts | undefined>;
createDescriptionGenerationRun(
databaseId: string,
scope: DescriptionGenerationScope,
modelId: string,
language: DescriptionGenerationRun["language"],
total: number,
): Promise<DescriptionGenerationRun>;
getDescriptionGenerationRun(runId: string): Promise<DescriptionGenerationRun | undefined>;
listDescriptionGenerationRuns(limit?: number): Promise<DescriptionGenerationRun[]>;
getActiveDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined>;
interruptActiveDescriptionGenerationRuns(
errorSummary: string,
): Promise<DescriptionGenerationRun[]>;
updateDescriptionGenerationRun(
runId: string,
update: DescriptionGenerationRunUpdate,
): Promise<DescriptionGenerationRun | undefined>;
appendDescriptionGenerationEvent(
runId: string,
level: DescriptionGenerationEvent["level"],
message: string,
): Promise<DescriptionGenerationEvent>;
listDescriptionGenerationEvents(
runId: string,
afterSequence?: number,
): Promise<DescriptionGenerationEvent[]>;
listRelationships(databaseId: string): Promise<CatalogRelationship[]>;
deleteDatabaseMetadata(
databaseIds: readonly string[],
@@ -347,6 +436,7 @@ export interface CatalogRepository {
}
export class CatalogConflictError extends Error {}
export class DescriptionGenerationRunActiveError extends CatalogConflictError {}
export class CatalogUnavailableError extends Error {}
export class CatalogOperationInProgressError extends Error {}
export class CatalogConnectorError extends Error {}
+9
View File
@@ -31,6 +31,7 @@ export interface AppConfig {
ollamaEnsureTimeoutMs: number;
piManagementTimeoutMs: number;
secretsFile?: string;
installationConfigFile?: string;
piAuthFile?: string;
secretFiles: Readonly<Record<string, string | undefined>>;
modelApiKeyFile?: string;
@@ -322,6 +323,13 @@ export function loadConfig(
secretsFile.trim() !== secretsFile || secretsFile.length === 0 || secretsFile.includes("\0")
|| !path.isAbsolute(secretsFile)
)) throw new Error("secret bundle configuration is invalid");
const installationConfigFile = env.THT_INSTALLATION_CONFIG_FILE;
if (installationConfigFile !== undefined && (
installationConfigFile.trim() !== installationConfigFile
|| installationConfigFile.length === 0
|| installationConfigFile.includes("\0")
|| !path.isAbsolute(installationConfigFile)
)) throw new Error("installation configuration is invalid");
const piAuthFile = env.THT_PI_AUTH_FILE;
if (piAuthFile !== undefined && (
piAuthFile.trim() !== piAuthFile || piAuthFile.length === 0 || piAuthFile.includes("\0")
@@ -404,6 +412,7 @@ export function loadConfig(
ollamaEnsureTimeoutMs: Number(env.OLLAMA_ENSURE_TIMEOUT_MS ?? 60000),
piManagementTimeoutMs: piManagementTimeout(env.PI_MANAGEMENT_TIMEOUT_MS),
secretsFile,
installationConfigFile,
piAuthFile,
secretFiles,
modelApiKeyFile,
+7
View File
@@ -8,12 +8,19 @@ import {
isUsableAuthenticationSecret,
} from "../auth/secret-policy.js";
/** Credential names that metadata-generation model entries may reference. */
export const METADATA_GENERATION_SECRET_KEYS = Object.freeze([
"THT_METADATA_API_KEY", "ANTHROPIC_API_KEY", "AZURE_API_KEY", "GEMINI_API_KEY",
"DEEPSEEK_API_KEY", "OPENAI_API_KEY", "OPENROUTER_API_KEY", "ZAI_API_KEY",
] as const);
/** Keys accepted by the deployment bundle. Keep this list intentionally explicit. */
export const SECRET_BUNDLE_KEYS = Object.freeze([
"THT_MODEL_API_KEY", "THT_DWH_API_KEY", "THT_VEC_API_KEY", "THT_VEC_WRITE_API_KEY",
"THT_CA", "THT_SSL_CA", "THT_VECTOR_BOOTSTRAP_PASSWORD", "THT_VECTOR_MIGRATOR_PASSWORD",
"THT_VECTOR_READER_PASSWORD", "THT_VECTOR_WRITER_PASSWORD", "PI_PROVIDER_API_KEY",
"THT_OIDC_CLIENT_SECRET", "THT_AUTHENTIK_API_TOKEN",
...METADATA_GENERATION_SECRET_KEYS,
] as const);
const ALLOWED = new Set<string>(SECRET_BUNDLE_KEYS);
@@ -0,0 +1,72 @@
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { z } from "zod";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import type { CatalogOperationCoordinator } from "../catalog/operation-coordinator.js";
import {
CatalogOperationInProgressError,
CatalogUnavailableError,
type CatalogRepository,
} from "../catalog/types.js";
const idSchema = z.uuid();
const consolidationSchema = z.object({
target: z.enum(["tables", "columns"]),
targetIds: z.array(idSchema).min(1).max(10_000),
}).strict();
function manage(request: FastifyRequest, reply: FastifyReply) {
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
}
function safeError(reply: FastifyReply, error: unknown) {
if (error instanceof CatalogUnavailableError) {
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
}
if (error instanceof CatalogOperationInProgressError) {
return reply.code(409).send({
code: "database_operation_in_progress",
message: "A database operation is already in progress.",
});
}
if (error instanceof z.ZodError) {
return reply.code(400).send({
code: "description_consolidation_invalid",
message: "Description consolidation request is invalid.",
});
}
return reply.code(500).send({
code: "description_consolidation_failed",
message: "Description consolidation failed.",
});
}
export function catalogDescriptionConsolidationRoutes(
app: FastifyInstance,
deps: { repository: CatalogRepository; operations: CatalogOperationCoordinator },
): void {
app.post("/catalog/databases/:databaseId/descriptions/consolidate", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
const input = consolidationSchema.parse(request.body);
const targetIds = [...new Set(input.targetIds)];
const result = await deps.operations.run(
databaseId,
async () => await deps.repository.consolidateGeneratedDescriptions(
databaseId,
input.target,
targetIds,
),
);
if (!result) {
return reply.code(404).send({
code: "catalog_target_not_found",
message: "The database or one or more selected catalog targets were not found.",
});
}
return result;
} catch (error) {
return safeError(reply, error);
}
});
}
@@ -0,0 +1,347 @@
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { z } from "zod";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import {
DescriptionGenerationDuplicateTargetIdsError,
DescriptionGenerationNoEligibleTargetsError,
DescriptionGenerationRunLiveError,
DescriptionGenerationTargetIdsRequiredError,
DescriptionGenerationTargetNotFoundError,
DescriptionGenerationWorkspaceUnavailableError,
type DescriptionGenerationWorker,
} from "../catalog/description-generation-worker.js";
import { MetadataGenerationModelUnavailableError } from "../catalog/metadata-generation-models.js";
import {
CatalogOperationInProgressError,
CatalogUnavailableError,
DescriptionGenerationRunActiveError,
type CatalogRepository,
type DescriptionGenerationEvent,
type DescriptionGenerationRun,
} from "../catalog/types.js";
const idSchema = z.uuid();
const modelIdSchema = z.string().regex(/^[a-z][a-z0-9._-]{0,63}$/);
const selectedTargetIdsSchema = z.array(idSchema).min(1);
const startSchema = z.discriminatedUnion("scope", [
z.object({
modelId: modelIdSchema,
scope: z.literal("selected_columns"),
targetIds: selectedTargetIdsSchema,
}).strict(),
z.object({
modelId: modelIdSchema,
scope: z.literal("selected_tables"),
targetIds: selectedTargetIdsSchema,
}).strict(),
z.object({ modelId: modelIdSchema, scope: z.literal("all") }).strict(),
z.object({ modelId: modelIdSchema, scope: z.literal("missing") }).strict(),
]);
const eventQuerySchema = z.object({
after: z.coerce.number().int().nonnegative().default(0),
}).strict();
const historyQuerySchema = z.object({
limit: z.coerce.number().int().min(1).max(100).default(50),
}).strict();
const terminalStatuses = new Set<DescriptionGenerationRun["status"]>([
"completed",
"completed_with_errors",
"cancelled",
"failed",
"interrupted",
]);
function manage(request: FastifyRequest, reply: FastifyReply) {
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
}
function publicEvent(event: DescriptionGenerationEvent) {
return {
sequence: event.sequence,
level: event.level,
message: event.message,
createdAt: event.createdAt,
};
}
function publicRun(run: DescriptionGenerationRun) {
return {
id: run.id,
databaseId: run.databaseId,
scope: run.scope,
modelId: run.modelId,
language: run.language,
status: run.status,
total: run.total,
processed: run.processed,
generated: run.generated,
nonGeneratable: run.nonGeneratable,
failed: run.failed,
createdAt: run.createdAt,
startedAt: run.startedAt,
updatedAt: run.updatedAt,
finishedAt: run.finishedAt,
errorSummary: run.errorSummary,
};
}
function safeError(reply: FastifyReply, error: unknown) {
if (error instanceof CatalogUnavailableError) {
return reply.code(503).send({
code: "catalog_unavailable",
message: "Database catalog is unavailable.",
});
}
if (error instanceof DescriptionGenerationRunActiveError) {
return reply.code(409).send({
code: "description_generation_run_active",
message: "A Description Generation Run is already active.",
});
}
if (error instanceof DescriptionGenerationRunLiveError) {
return reply.code(409).send({
code: "description_generation_run_live",
message: "A local Description Generation worker or helper is still running.",
});
}
if (error instanceof CatalogOperationInProgressError) {
return reply.code(409).send({
code: "database_operation_in_progress",
message: "A database operation is already in progress.",
});
}
if (error instanceof MetadataGenerationModelUnavailableError) {
return reply.code(409).send({
code: "metadata_generation_model_unavailable",
message: "The selected metadata-generation model is unavailable.",
});
}
if (error instanceof DescriptionGenerationDuplicateTargetIdsError) {
return reply.code(400).send({
code: "description_generation_target_ids_duplicate",
message: "Description generation target IDs must be unique.",
});
}
if (error instanceof DescriptionGenerationTargetIdsRequiredError) {
return reply.code(400).send({
code: "description_generation_request_invalid",
message: "At least one description generation target ID is required.",
});
}
if (error instanceof DescriptionGenerationNoEligibleTargetsError) {
return reply.code(409).send({
code: "description_generation_no_eligible_targets",
message: error.scope === "all"
? "No Catalog Tables or Catalog Columns are available for description generation."
: "No Catalog Tables or Catalog Columns have a missing Generated Description.",
});
}
if (error instanceof DescriptionGenerationTargetNotFoundError) {
const code = error.target === "database"
? "database_not_found"
: error.target === "table"
? "catalog_table_not_found"
: "catalog_column_not_found";
const message = error.target === "database"
? "Database configuration was not found."
: error.target === "table"
? "One or more selected Catalog Tables were not found."
: "One or more selected Catalog Columns were not found.";
return reply.code(404).send({
code,
message,
});
}
if (error instanceof DescriptionGenerationWorkspaceUnavailableError) {
return reply.code(409).send({
code: "workspace_configuration_unavailable",
message: "The database workspace configuration is unavailable.",
});
}
if (error instanceof z.ZodError) {
return reply.code(400).send({
code: "description_generation_request_invalid",
message: "Description generation request is invalid.",
});
}
return reply.code(500).send({
code: "description_generation_failed",
message: "Description generation failed.",
});
}
export function catalogDescriptionGenerationRoutes(
app: FastifyInstance,
deps: { repository: CatalogRepository; worker: DescriptionGenerationWorker },
): void {
app.post("/catalog/databases/:databaseId/description-generation-runs", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
const input = startSchema.parse(request.body);
const targetIds = "targetIds" in input ? input.targetIds : [];
const run = await deps.worker.start(databaseId, input.modelId, input.scope, targetIds);
return reply.code(202).send(publicRun(run));
} catch (error) {
return safeError(reply, error);
}
});
app.get("/catalog/description-generation-runs", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const { limit } = historyQuerySchema.parse(request.query);
return (await deps.repository.listDescriptionGenerationRuns(limit)).map(publicRun);
} catch (error) {
return safeError(reply, error);
}
});
app.get("/catalog/description-generation-runs/:runId", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
const run = await deps.repository.getDescriptionGenerationRun(runId);
return run ? publicRun(run) : reply.code(404).send({
code: "description_generation_run_not_found",
message: "Description Generation Run was not found.",
});
} catch (error) {
return safeError(reply, error);
}
});
app.post("/catalog/description-generation-runs/:runId/cancel", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
const run = await deps.worker.cancel(runId);
return run ? publicRun(run) : reply.code(404).send({
code: "description_generation_run_not_found",
message: "Description Generation Run was not found.",
});
} catch (error) {
return safeError(reply, error);
}
});
app.post("/catalog/description-generation-runs/unlock", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const run = await deps.worker.unlock();
return run ? publicRun(run) : reply.code(404).send({
code: "description_generation_run_not_found",
message: "No stale active Description Generation Run was found.",
});
} catch (error) {
return safeError(reply, error);
}
});
app.get("/catalog/description-generation-runs/:runId/events", async (request, reply) => {
if (!manage(request, reply)) return reply;
let unsubscribe: (() => void) | undefined;
let hijacked = false;
try {
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
let after = eventQuerySchema.parse(request.query).after;
const headerCursor = Number(request.headers["last-event-id"]);
if (Number.isInteger(headerCursor) && headerCursor >= 0) after = Math.max(after, headerCursor);
if (!(await deps.repository.getDescriptionGenerationRun(runId))) {
return reply.code(404).send({
code: "description_generation_run_not_found",
message: "Description Generation Run was not found.",
});
}
const buffered: DescriptionGenerationEvent[] = [];
let ready = false;
let closed = false;
let lastRunSnapshot = "";
let delivery = Promise.resolve();
const close = () => {
if (closed) return;
closed = true;
unsubscribe?.();
if (!reply.raw.destroyed) reply.raw.end();
};
const writeRun = (run: DescriptionGenerationRun) => {
if (closed) return;
const snapshot = JSON.stringify(publicRun(run));
if (snapshot === lastRunSnapshot) return;
lastRunSnapshot = snapshot;
reply.raw.write(`event: run\ndata: ${snapshot}\n\n`);
if (terminalStatuses.has(run.status)) close();
};
const enqueue = (event: DescriptionGenerationEvent) => {
delivery = delivery.then(async () => {
if (closed || event.sequence <= after) return;
after = event.sequence;
reply.raw.write(
`id: ${event.sequence}\nevent: log\ndata: ${JSON.stringify(publicEvent(event))}\n\n`,
);
const run = await deps.repository.getDescriptionGenerationRun(runId);
if (run) writeRun(run);
}).catch(close);
};
unsubscribe = deps.worker.subscribeEvents(runId, (event) => {
if (ready) enqueue(event);
else buffered.push(event);
});
const persisted = await deps.repository.listDescriptionGenerationEvents(runId, after);
reply.hijack();
hijacked = true;
reply.raw.writeHead(200, {
"content-type": "text/event-stream; charset=utf-8",
"cache-control": "no-cache, no-transform",
connection: "keep-alive",
"x-accel-buffering": "no",
});
reply.raw.once("close", close);
request.raw.once("aborted", close);
for (const event of persisted) {
if (event.sequence <= after) continue;
after = event.sequence;
reply.raw.write(
`id: ${event.sequence}\nevent: log\ndata: ${JSON.stringify(publicEvent(event))}\n\n`,
);
}
ready = true;
buffered.sort((a, b) => a.sequence - b.sequence).forEach(enqueue);
let pending = delivery;
await pending;
while (pending !== delivery) {
pending = delivery;
await pending;
}
const current = await deps.repository.getDescriptionGenerationRun(runId);
if (current) writeRun(current);
return reply;
} catch (error) {
unsubscribe?.();
if (hijacked) {
if (!reply.raw.destroyed) reply.raw.end();
return reply;
}
return safeError(reply, error);
}
});
app.get("/catalog/description-generation-runs/:runId/events-list", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
const { after } = eventQuerySchema.parse(request.query);
if (!(await deps.repository.getDescriptionGenerationRun(runId))) {
return reply.code(404).send({
code: "description_generation_run_not_found",
message: "Description Generation Run was not found.",
});
}
return (await deps.repository.listDescriptionGenerationEvents(runId, after)).map(publicEvent);
} catch (error) {
return safeError(reply, error);
}
});
}
+25 -16
View File
@@ -43,7 +43,13 @@ function safeError(reply: FastifyReply, error: unknown) {
if (error instanceof CatalogUnavailableError) {
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
}
if (error instanceof CatalogConflictError || error instanceof CatalogOperationInProgressError) {
if (error instanceof CatalogOperationInProgressError) {
return reply.code(409).send({
code: "database_operation_in_progress",
message: "A database operation is already in progress.",
});
}
if (error instanceof CatalogConflictError) {
return reply.code(409).send({ code: "schema_sync_conflict", message: error.message });
}
if (error instanceof CatalogConnectorError) {
@@ -98,21 +104,24 @@ export function catalogSchemaRoutes(
const tableId = idSchema.parse(params.tableId);
const columnId = idSchema.parse(params.columnId);
const input = metadataSchema.parse(request.body);
const current = await deps.repository.getColumn(databaseId, tableId, columnId);
if (!current) return reply.code(404).send({ code: "column_not_found", message: "Catalog column was not found." });
if (current.version !== input.version) {
return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
}
const updated = await deps.repository.updateColumnMetadata(
databaseId,
tableId,
columnId,
input.version,
normalized(input.description),
normalized(input.generatedDescription),
);
if (!updated) return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
return updated;
const operation = async () => {
const current = await deps.repository.getColumn(databaseId, tableId, columnId);
if (!current) return reply.code(404).send({ code: "column_not_found", message: "Catalog column was not found." });
if (current.version !== input.version) {
return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
}
const updated = await deps.repository.updateColumnMetadata(
databaseId,
tableId,
columnId,
input.version,
normalized(input.description),
normalized(input.generatedDescription),
);
if (!updated) return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
return updated;
};
return deps.operations ? await deps.operations.run(databaseId, operation) : await operation();
} catch (error) { return safeError(reply, error); }
});
+22 -15
View File
@@ -2,6 +2,7 @@ import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { z } from "zod";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import type { CatalogTableService } from "../catalog/table-service.js";
import type { CatalogOperationCoordinator } from "../catalog/operation-coordinator.js";
import {
CatalogConnectorError,
CatalogOperationInProgressError,
@@ -38,7 +39,11 @@ function safeError(reply: FastifyReply, error: unknown) {
export function catalogTableRoutes(
app: FastifyInstance,
deps: { repository: CatalogRepository; service: CatalogTableService },
deps: {
repository: CatalogRepository;
service: CatalogTableService;
operations: CatalogOperationCoordinator;
},
): void {
app.get("/catalog/databases/:databaseId/tables", async (request, reply) => {
if (!manage(request, reply)) return reply;
@@ -59,20 +64,22 @@ export function catalogTableRoutes(
const tableId = idSchema.parse(params.tableId);
const input = updateSchema.parse(request.body);
const { version, description } = input;
const current = await deps.repository.getTable(databaseId, tableId);
if (!current) return reply.code(404).send({ code: "table_not_found", message: "Catalog table was not found." });
if (current.version !== version) {
return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
}
const updated = await deps.service.updateMetadata(
databaseId,
tableId,
version,
description,
input.generatedDescription === undefined ? current.generatedDescription : input.generatedDescription,
);
if (!updated) return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
return updated;
return await deps.operations.run(databaseId, async () => {
const current = await deps.repository.getTable(databaseId, tableId);
if (!current) return reply.code(404).send({ code: "table_not_found", message: "Catalog table was not found." });
if (current.version !== version) {
return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
}
const updated = await deps.service.updateMetadata(
databaseId,
tableId,
version,
description,
input.generatedDescription === undefined ? current.generatedDescription : input.generatedDescription,
);
if (!updated) return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
return updated;
});
} catch (error) { return safeError(reply, error); }
});
@@ -0,0 +1,13 @@
import type { FastifyInstance } from "fastify";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import type { MetadataGenerationModels } from "../catalog/metadata-generation-models.js";
export function metadataGenerationModelRoutes(
app: FastifyInstance,
models: MetadataGenerationModels,
): void {
app.get("/catalog/metadata-generation/models", async (request, reply) => {
if (!isPrincipalContext(requirePermission(request, reply, "database.manage"))) return reply;
return models.catalog();
});
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,456 @@
import { expect, test, vi } from "vitest";
import { DescriptionGenerationWorker } from "../src/catalog/description-generation-worker.js";
import type { DescriptionSourceSampler } from "../src/catalog/description-source-sampler.js";
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
import type { MetadataGenerationModels } from "../src/catalog/metadata-generation-models.js";
import { ModelCompletionCancelledError } from "../src/catalog/model-completer.js";
import type { ModelCompleter, ModelCompletionRequest } from "../src/catalog/model-completer.js";
import { CatalogOperationCoordinator } from "../src/catalog/operation-coordinator.js";
import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
test("serializes Unlock with Start so stale recovery cannot release a new reservation", async () => {
let lookupStarted!: () => void;
const started = new Promise<void>((resolve) => { lookupStarted = resolve; });
let releaseLookup!: () => void;
const gate = new Promise<void>((resolve) => { releaseLookup = resolve; });
const repository = {
getActiveDescriptionGenerationRun: vi.fn(async () => {
lookupStarted();
await gate;
return undefined;
}),
} as unknown as MemoryCatalogRepository;
const resolveModel = vi.fn();
const worker = new DescriptionGenerationWorker(
repository,
{} as WorkspaceRegistry,
{
catalog: () => ({ models: [], default: "" }),
resolve: resolveModel,
} as MetadataGenerationModels,
{} as ModelCompleter,
new CatalogOperationCoordinator(),
{ sample: vi.fn(async () => []) },
);
const unlocking = worker.unlock();
await started;
await expect(worker.start(
"11111111-1111-4111-8111-111111111111",
"openai-mini",
"missing",
[],
)).rejects.toThrow("already active");
expect(resolveModel).not.toHaveBeenCalled();
releaseLookup();
await expect(unlocking).resolves.toBeUndefined();
});
test("exposes an awaitable background job and absorbs provider promise rejection", async () => {
const repository = new MemoryCatalogRepository();
const database = await repository.create({
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
});
await repository.applySchemaSync(database.id, database.version, "all", [], {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: [{ name: "patients", sourceComment: null }],
columns: [{
tableName: "patients",
name: "birth_date",
ordinalPosition: 1,
dataType: "date",
isNullable: true,
defaultExpression: null,
primaryKeyPosition: null,
sourceComment: null,
}],
relationships: [],
});
const table = (await repository.listTables(database.id))[0]!;
const column = (await repository.listColumns(database.id, table.id))[0]!;
let rejectCompletion!: (error: Error) => void;
const pendingCompletion = new Promise<string>((_resolve, reject) => { rejectCompletion = reject; });
const completer: ModelCompleter = {
complete: vi.fn(async () => await pendingCompletion),
};
const models: MetadataGenerationModels = {
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
resolve: () => ({
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "test-provider-secret",
}),
};
const operations = new CatalogOperationCoordinator();
const sourceSampler: DescriptionSourceSampler = {
sample: vi.fn(async () => []),
};
const worker = new DescriptionGenerationWorker(
repository,
{
read: vi.fn(async () => ({
workspace: { workspace: { language: "it" } },
revision: {},
})),
} as unknown as WorkspaceRegistry,
models,
completer,
operations,
sourceSampler,
);
const run = await worker.start(database.id, "openai-mini", "selected_columns", [column.id]);
let settled = false;
const waiting = worker.waitForRun(run.id).then(() => { settled = true; });
await new Promise((resolve) => setTimeout(resolve, 0));
expect(settled).toBe(false);
rejectCompletion(new Error("test-provider-secret private prompt raw response"));
await expect(waiting).resolves.toBeUndefined();
expect(await repository.getDescriptionGenerationRun(run.id)).toMatchObject({
status: "completed_with_errors",
failed: 1,
errorSummary: "Description generation completed with errors.",
});
const events = await repository.listDescriptionGenerationEvents(run.id);
expect(JSON.stringify(events)).not.toMatch(/test-provider-secret|private prompt|raw response/);
const release = operations.reserve(database.id);
release();
await expect(worker.start(
database.id,
"openai-mini",
"selected_columns",
[],
)).rejects.toThrow("at least one target ID is required");
});
test("marks an active run interrupted when the backend worker stops", async () => {
const repository = new MemoryCatalogRepository();
const database = await repository.create({
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
});
await repository.applySchemaSync(database.id, database.version, "all", [], {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: [{ name: "patients", sourceComment: null }],
columns: [{
tableName: "patients",
name: "status",
ordinalPosition: 1,
dataType: "text",
isNullable: true,
defaultExpression: null,
primaryKeyPosition: null,
sourceComment: null,
}],
relationships: [],
});
const table = (await repository.listTables(database.id))[0]!;
const column = (await repository.listColumns(database.id, table.id))[0]!;
const completer: ModelCompleter = {
complete: vi.fn(async (request) => await new Promise<string>((_resolve, reject) => {
const cancel = () => reject(new ModelCompletionCancelledError());
if (request.signal.aborted) cancel();
else request.signal.addEventListener("abort", cancel, { once: true });
})),
};
const worker = new DescriptionGenerationWorker(
repository,
{
read: vi.fn(async () => ({
workspace: { workspace: { language: "it" } },
revision: {},
})),
} as unknown as WorkspaceRegistry,
{
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
resolve: () => ({
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "test-provider-secret",
}),
},
completer,
new CatalogOperationCoordinator(),
{ sample: vi.fn(async () => []) },
);
const run = await worker.start(database.id, "openai-mini", "selected_columns", [column.id]);
await vi.waitFor(() => expect(completer.complete).toHaveBeenCalledOnce());
await worker.stop();
expect(await repository.getDescriptionGenerationRun(run.id)).toMatchObject({
status: "interrupted",
errorSummary: "Description generation was interrupted by backend shutdown.",
});
expect(await repository.listDescriptionGenerationEvents(run.id)).toContainEqual(
expect.objectContaining({
level: "warning",
message: "Description generation was interrupted by backend shutdown.",
}),
);
});
test("adds only bounded transient source samples to the model request", async () => {
const repository = new MemoryCatalogRepository();
const database = await repository.create({
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
});
await repository.applySchemaSync(database.id, database.version, "all", [], {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: [{ name: "patients", sourceComment: null }],
columns: [{
tableName: "patients",
name: "status",
ordinalPosition: 1,
dataType: "text",
isNullable: true,
defaultExpression: null,
primaryKeyPosition: null,
sourceComment: null,
}, {
tableName: "patients",
name: "ward",
ordinalPosition: 2,
dataType: "text",
isNullable: true,
defaultExpression: null,
primaryKeyPosition: null,
sourceComment: null,
}],
relationships: [],
});
const table = (await repository.listTables(database.id))[0]!;
const columns = await repository.listColumns(database.id, table.id);
const column = columns.find((candidate) => candidate.name === "status")!;
const ward = columns.find((candidate) => candidate.name === "ward")!;
const sampleSecret = "ONLY_IN_TRANSIENT_SAMPLE_7f29c8";
const sourceSampler: DescriptionSourceSampler = {
sample: vi.fn(async () => [{
targetId: column.id,
tableName: table.name,
rows: [
{ fields: [{ name: column.name, value: sampleSecret }] },
{ fields: [{ name: column.name, value: "row-2" }] },
{ fields: [{ name: column.name, value: "row-3" }] },
],
representativeValues: [{
column: column.name,
values: [sampleSecret, sampleSecret, "two", "three"],
}],
}, {
targetId: ward.id,
tableName: table.name,
rows: [
{ fields: [{ name: ward.name, value: "row-4" }] },
{ fields: [{ name: ward.name, value: "row-5" }] },
{ fields: [{ name: ward.name, value: "row-6-must-be-omitted" }] },
],
representativeValues: [{
column: ward.name,
values: ["ward-1", "ward-2", "ward-3-must-be-omitted"],
}],
}]),
};
const completer: ModelCompleter = {
complete: vi.fn(async () => JSON.stringify({
results: [{
targetId: column.id,
outcome: "generated",
description: "Stato amministrativo del paziente.",
}, {
targetId: ward.id,
outcome: "generated",
description: "Reparto associato al paziente.",
}],
})),
};
const models: MetadataGenerationModels = {
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
resolve: () => ({
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "test-provider-secret",
}),
};
const worker = new DescriptionGenerationWorker(
repository,
{
read: vi.fn(async () => ({
workspace: { workspace: { language: "it" } },
revision: {},
})),
} as unknown as WorkspaceRegistry,
models,
completer,
new CatalogOperationCoordinator(),
sourceSampler,
);
const run = await worker.start(
database.id,
"openai-mini",
"selected_columns",
[column.id, ward.id],
);
await worker.waitForRun(run.id);
expect(sourceSampler.sample).toHaveBeenCalledWith(
expect.objectContaining({ id: database.id, binding: database.binding }),
[
{ targetId: column.id, tableName: table.name, columnNames: [column.name] },
{ targetId: ward.id, tableName: table.name, columnNames: [ward.name] },
],
expect.any(AbortSignal),
);
const request = vi.mocked(completer.complete).mock.calls[0]![0] as ModelCompletionRequest;
expect(request.messages[0]?.content).toContain("untrusted");
const userMessage = request.messages[1]!.content;
const context = JSON.parse(userMessage.slice(userMessage.indexOf("\n") + 1));
const sampledRows = context.targets.flatMap(
(targetContext: { sourceSample?: { rows: unknown[] } }) => targetContext.sourceSample?.rows ?? [],
);
const representativeValues = context.targets.flatMap(
(targetContext: { sourceSample?: { representativeValues: Array<{ values: unknown[] }> } }) => (
targetContext.sourceSample?.representativeValues.flatMap((entry) => entry.values) ?? []
),
);
expect(sampledRows).toHaveLength(5);
expect(representativeValues).toHaveLength(5);
expect(context.targets[0].sourceSample.rows).toHaveLength(3);
expect(context.targets[1].sourceSample.rows).toHaveLength(2);
expect(context.targets[0].sourceSample.representativeValues).toEqual([{
column: column.name,
values: [sampleSecret, "two", "three"],
}]);
expect(context.targets[1].sourceSample.representativeValues).toEqual([{
column: ward.name,
values: ["ward-1", "ward-2"],
}]);
expect(userMessage).toContain(sampleSecret);
expect(userMessage).not.toMatch(
/row-6-must-be-omitted|ward-3-must-be-omitted/,
);
const persisted = JSON.stringify({
run: await repository.getDescriptionGenerationRun(run.id),
events: await repository.listDescriptionGenerationEvents(run.id),
database: await repository.get(database.id),
table: await repository.getTable(database.id, table.id),
column: await repository.getColumn(database.id, table.id, column.id),
ward: await repository.getColumn(database.id, table.id, ward.id),
});
expect(persisted).not.toContain(sampleSecret);
});
test("continues metadata-only with one safe warning when source sampling is unavailable", async () => {
const repository = new MemoryCatalogRepository();
const database = await repository.create({
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
binding: {
transport: "rest_api",
baseUrl: "https://dwh.example.test",
restPath: "/rpc/run_query",
restAuth: "none",
},
});
await repository.applySchemaSync(database.id, database.version, "all", [], {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: [{ name: "patients", sourceComment: null }],
columns: [{
tableName: "patients",
name: "status",
ordinalPosition: 1,
dataType: "text",
isNullable: true,
defaultExpression: null,
primaryKeyPosition: null,
sourceComment: null,
}],
relationships: [],
});
const table = (await repository.listTables(database.id))[0]!;
const column = (await repository.listColumns(database.id, table.id))[0]!;
const samplingFailureSecret = "UNAVAILABLE_SAMPLE_DETAIL_48b1f1";
const sourceSampler: DescriptionSourceSampler = {
sample: vi.fn(async () => { throw new Error(samplingFailureSecret); }),
};
const completer: ModelCompleter = {
complete: vi.fn(async () => JSON.stringify({
results: [{
targetId: column.id,
outcome: "generated",
description: "Stato del paziente.",
}],
})),
};
const models: MetadataGenerationModels = {
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
resolve: () => ({
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "test-provider-secret",
}),
};
const worker = new DescriptionGenerationWorker(
repository,
{
read: vi.fn(async () => ({
workspace: { workspace: { language: "it" } },
revision: {},
})),
} as unknown as WorkspaceRegistry,
models,
completer,
new CatalogOperationCoordinator(),
sourceSampler,
);
const run = await worker.start(database.id, "openai-mini", "selected_columns", [column.id]);
await worker.waitForRun(run.id);
expect(await repository.getDescriptionGenerationRun(run.id)).toMatchObject({
status: "completed",
processed: 1,
generated: 1,
failed: 0,
});
const request = vi.mocked(completer.complete).mock.calls[0]![0] as ModelCompletionRequest;
expect(request.messages[1]?.content).not.toMatch(/sourceSample|UNAVAILABLE_SAMPLE_DETAIL/);
const events = await repository.listDescriptionGenerationEvents(run.id);
expect(events.filter((event) => event.level === "warning")).toEqual([
expect.objectContaining({
message: "Source samples unavailable for this batch; generation continued with catalog metadata only.",
}),
]);
expect(JSON.stringify(events)).not.toContain(samplingFailureSecret);
});
@@ -0,0 +1,384 @@
import { spawnSync } from "node:child_process";
import { PostgreSqlContainer } from "@testcontainers/postgresql";
import { CamelCasePlugin, Kysely, PostgresDialect } from "kysely";
import { Pool } from "pg";
import { expect, test, vi } from "vitest";
import { buildApp } from "../src/app.js";
import type { DescriptionSourceSampler } from "../src/catalog/description-source-sampler.js";
import type { MetadataGenerationModels } from "../src/catalog/metadata-generation-models.js";
import { ModelCompletionProviderError, type ModelCompleter } from "../src/catalog/model-completer.js";
import { up as upDatabases } from "../src/catalog/migrations/001_workspace_databases.js";
import { up as upTables } from "../src/catalog/migrations/002_catalog_tables.js";
import { up as upSchemaSync } from "../src/catalog/migrations/003_catalog_schema_sync.js";
import { up as upDescriptionGeneration } from "../src/catalog/migrations/005_description_generation_runs.js";
import { KyselyCatalogRepository, type CatalogDatabase } from "../src/catalog/repository.js";
import { loadConfig } from "../src/config.js";
import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
async function terminalRun(app: ReturnType<typeof buildApp>, runId: string) {
for (let attempt = 0; attempt < 200; attempt += 1) {
const response = await app.inject({
method: "GET",
url: `/catalog/description-generation-runs/${runId}`,
});
const run = response.json();
if (["completed", "completed_with_errors", "cancelled", "failed", "interrupted"].includes(run.status)) {
return run;
}
await new Promise((resolve) => setTimeout(resolve, 5));
}
throw new Error(`Description Generation Run ${runId} did not finish`);
}
test.skipIf(!dockerAvailable)("Fastify persists Description Generation success and failure through PostgreSQL", async () => {
const container = await new PostgreSqlContainer("postgres:17.6-bookworm").start();
const db = new Kysely<CatalogDatabase>({
dialect: new PostgresDialect({ pool: new Pool({ connectionString: container.getConnectionUri() }) }),
plugins: [new CamelCasePlugin()],
});
let app: ReturnType<typeof buildApp> | undefined;
try {
await upDatabases(db);
await upTables(db);
await upSchemaSync(db);
await upDescriptionGeneration(db);
const repository = new KyselyCatalogRepository(db);
const database = await repository.create({
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
});
await repository.applySchemaSync(database.id, database.version, "all", [], {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: [{ name: "patients", sourceComment: "Clinical patients" }],
columns: [
{
tableName: "patients",
name: "birth_date",
ordinalPosition: 1,
dataType: "date",
isNullable: true,
defaultExpression: null,
primaryKeyPosition: null,
sourceComment: "Patient date of birth",
},
{
tableName: "patients",
name: "status",
ordinalPosition: 2,
dataType: "text",
isNullable: true,
defaultExpression: null,
primaryKeyPosition: null,
sourceComment: "Patient status",
},
],
relationships: [],
});
const table = (await repository.listTables(database.id))[0]!;
const columns = await repository.listColumns(database.id, table.id);
const birthDate = columns.find((column) => column.name === "birth_date")!;
const status = columns.find((column) => column.name === "status")!;
const curatedTable = (await repository.updateTableMetadata(
database.id,
table.id,
table.version,
"Elenco curato dei pazienti.",
null,
))!;
const curatedStatus = (await repository.updateColumnMetadata(
database.id,
table.id,
status.id,
status.version,
"Stato curato del paziente.",
null,
))!;
let call = 0;
const modelCompleter: ModelCompleter = {
complete: vi.fn(async () => {
call += 1;
if (call === 1) {
return JSON.stringify({ results: [
{
targetId: birthDate.id,
outcome: "generated",
description: "Data di nascita del paziente.",
},
{ targetId: status.id, outcome: "non_generatable" },
] });
}
if (call === 2) {
return JSON.stringify({ results: [{
targetId: table.id,
outcome: "generated",
description: "Elenco dei pazienti e dei loro dati clinici.",
}] });
}
if (call === 4) {
return JSON.stringify({ results: [
{
targetId: birthDate.id,
outcome: "generated",
description: "Descrizione rigenerata della data di nascita.",
},
{
targetId: status.id,
outcome: "generated",
description: "Descrizione rigenerata dello stato.",
},
] });
}
if (call === 5) {
return JSON.stringify({ results: [{
targetId: table.id,
outcome: "generated",
description: "Descrizione rigenerata della tabella pazienti.",
}] });
}
if (call === 6) {
return JSON.stringify({ results: [{
targetId: birthDate.id,
outcome: "generated",
description: "Descrizione recuperata della data di nascita.",
}] });
}
throw new ModelCompletionProviderError();
}),
};
const models: MetadataGenerationModels = {
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
resolve: () => ({
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "test-provider-secret",
}),
};
const registry = {
list: vi.fn(async () => []),
read: vi.fn(async () => ({
workspace: { workspace: { language: "it" } },
revision: {},
})),
} as unknown as WorkspaceRegistry;
const persistedSampleSecret = "POSTGRES_TRANSIENT_SAMPLE_6a0d7b";
const descriptionSourceSampler: DescriptionSourceSampler = {
sample: vi.fn(async (_database, targets) => targets.map((target) => ({
targetId: target.targetId,
tableName: target.tableName,
rows: [{
fields: target.columnNames.slice(0, 1).map((name) => ({
name,
value: persistedSampleSecret,
})),
}],
representativeValues: target.columnNames.slice(0, 1).map((column) => ({
column,
values: [persistedSampleSecret],
})),
}))),
};
app = buildApp(loadConfig({ NODE_ENV: "test", THT_HARNESS_DIR: "/missing" }), {
thtRunner: {} as never,
workspaceRegistry: registry,
workspaceDiagnoser: vi.fn(),
catalogRepository: repository,
metadataGenerationModels: models,
modelCompleter,
descriptionSourceSampler,
});
const successfulStart = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: {
modelId: "openai-mini",
scope: "selected_columns",
targetIds: [status.id, birthDate.id],
},
});
expect(successfulStart.statusCode).toBe(202);
expect(await terminalRun(app, successfulStart.json().id)).toMatchObject({
status: "completed",
total: 2,
processed: 2,
generated: 1,
nonGeneratable: 1,
failed: 0,
});
expect(await repository.getColumn(database.id, table.id, birthDate.id)).toMatchObject({
generatedDescription: "Data di nascita del paziente.",
version: birthDate.version + 1,
});
const generatedStatus = (await repository.getColumn(database.id, table.id, status.id))!;
expect(generatedStatus).toMatchObject({
description: "Stato curato del paziente.",
generatedDescription: "Non generabile",
version: curatedStatus.version + 1,
});
const tableStart = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "selected_tables", targetIds: [table.id] },
});
expect(tableStart.statusCode).toBe(202);
expect(await terminalRun(app, tableStart.json().id)).toMatchObject({
scope: "selected_tables",
status: "completed",
processed: 1,
generated: 1,
nonGeneratable: 0,
failed: 0,
});
expect(await repository.getTable(database.id, table.id)).toMatchObject({
description: "Elenco curato dei pazienti.",
generatedDescription: "Elenco dei pazienti e dei loro dati clinici.",
version: curatedTable.version + 1,
});
const failedStart = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "selected_columns", targetIds: [status.id] },
});
expect(failedStart.statusCode).toBe(202);
const failedRun = await terminalRun(app, failedStart.json().id);
expect(failedRun).toMatchObject({
status: "completed_with_errors",
processed: 1,
generated: 0,
failed: 1,
errorSummary: "Description generation completed with errors.",
});
expect(await repository.getColumn(database.id, table.id, status.id)).toMatchObject({
description: "Stato curato del paziente.",
generatedDescription: "Non generabile",
version: generatedStatus.version,
});
const events = await app.inject({
method: "GET",
url: `/catalog/description-generation-runs/${failedRun.id}/events-list`,
});
expect(events.statusCode).toBe(200);
expect(events.json().find((event: { level: string }) => event.level === "error")).toEqual(expect.objectContaining({
level: "error",
message: `The model provider request failed. Affected Catalog Column target: ${status.id}.`,
}));
expect(events.body).not.toMatch(/test-provider-secret|gpt-4\.1-mini|raw provider/i);
const allStart = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "all" },
});
expect(allStart.statusCode).toBe(202);
const allRun = await terminalRun(app, allStart.json().id);
expect(allRun).toMatchObject({
scope: "all",
status: "completed",
total: 3,
processed: 3,
generated: 3,
nonGeneratable: 0,
failed: 0,
});
expect(await repository.getColumn(database.id, table.id, birthDate.id)).toMatchObject({
generatedDescription: "Descrizione rigenerata della data di nascita.",
});
expect(await repository.getColumn(database.id, table.id, status.id)).toMatchObject({
generatedDescription: "Descrizione rigenerata dello stato.",
});
expect(await repository.getTable(database.id, table.id)).toMatchObject({
generatedDescription: "Descrizione rigenerata della tabella pazienti.",
});
const allEvents = await app.inject({
method: "GET",
url: `/catalog/description-generation-runs/${allRun.id}/events-list`,
});
expect(allEvents.json()[0]).toEqual(expect.objectContaining({
message: "Description generation queued (scope: all).",
}));
const regeneratedBirthDate = (await repository.getColumn(
database.id, table.id, birthDate.id,
))!;
await repository.updateColumnMetadata(
database.id,
table.id,
birthDate.id,
regeneratedBirthDate.version,
regeneratedBirthDate.description,
" ",
);
const missingStart = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/description-generation-runs`,
payload: { modelId: "openai-mini", scope: "missing" },
});
expect(missingStart.statusCode).toBe(202);
expect(await terminalRun(app, missingStart.json().id)).toMatchObject({
scope: "missing",
status: "completed",
total: 1,
processed: 1,
generated: 1,
nonGeneratable: 0,
failed: 0,
});
expect(await repository.getColumn(database.id, table.id, birthDate.id)).toMatchObject({
generatedDescription: "Descrizione recuperata della data di nascita.",
});
expect(modelCompleter.complete).toHaveBeenCalledTimes(6);
expect(JSON.stringify(vi.mocked(modelCompleter.complete).mock.calls)).toContain(persistedSampleSecret);
const runIds = [
successfulStart.json().id,
tableStart.json().id,
failedStart.json().id,
allStart.json().id,
missingStart.json().id,
];
const durableState = JSON.stringify({
runs: await Promise.all(runIds.map(async (runId) => (
await repository.getDescriptionGenerationRun(runId)
))),
events: await Promise.all(runIds.map(async (runId) => (
await repository.listDescriptionGenerationEvents(runId)
))),
database: await repository.get(database.id),
table: await repository.getTable(database.id, table.id),
columns: await repository.listColumns(database.id, table.id),
});
expect(durableState).not.toContain(persistedSampleSecret);
const apiResponses = await Promise.all([
...runIds.flatMap((runId) => [
app!.inject({ method: "GET", url: `/catalog/description-generation-runs/${runId}` }),
app!.inject({
method: "GET",
url: `/catalog/description-generation-runs/${runId}/events-list`,
}),
]),
app.inject({ method: "GET", url: `/catalog/databases/${database.id}` }),
app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables` }),
app.inject({
method: "GET",
url: `/catalog/databases/${database.id}/tables/${table.id}/columns`,
}),
]);
expect(apiResponses.map((response) => response.body).join("\n")).not.toContain(
persistedSampleSecret,
);
} finally {
if (app) await app.close();
await db.destroy();
await container.stop();
}
}, 60_000);
@@ -0,0 +1,111 @@
import { expect, test, vi } from "vitest";
import {
PostgresDescriptionSourceSampler,
type DescriptionSourceSamplingTarget,
} from "../src/catalog/description-source-sampler.js";
import type {
CatalogDatabaseClient,
CatalogPostgresAccess,
} from "../src/catalog/postgres-access.js";
import type { WorkspaceDatabase } from "../src/catalog/types.js";
const database: WorkspaceDatabase = {
id: "11111111-1111-4111-8111-111111111111",
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: 'clinical"data',
version: 1,
createdAt: "2026-08-28T08:00:00Z",
updatedAt: "2026-08-28T08:00:00Z",
connectionStatus: "reachable",
binding: {
transport: "postgres_direct",
host: "db.internal",
port: 5432,
username: "reader",
},
};
const target: DescriptionSourceSamplingTarget = {
targetId: "22222222-2222-4222-8222-222222222222",
tableName: 'patient"facts',
columnNames: ['status"code', "ward"],
};
test("samples at most five source rows and five distinct non-null examples in a read-only transaction", async () => {
const query = vi.fn(async (sql: string) => {
if (!sql.startsWith("SELECT")) return { rows: [] };
return {
rows: [
{ 'status"code': "active", ward: null },
{ 'status"code': "pending", ward: "A" },
{ 'status"code': "closed", ward: "A" },
{ 'status"code': "transferred", ward: "B" },
{ 'status"code': "unknown", ward: "C" },
{ 'status"code': "must-not-be-sampled", ward: "D" },
],
};
});
const end = vi.fn(async () => undefined);
const access: CatalogPostgresAccess = {
connect: vi.fn(async () => ({ query, end }) as CatalogDatabaseClient),
};
const sampler = new PostgresDescriptionSourceSampler(access);
const controller = new AbortController();
const samples = await sampler.sample(database, [target], controller.signal);
expect(samples).toEqual([{
targetId: target.targetId,
tableName: target.tableName,
rows: [
{ fields: [{ name: 'status"code', value: "active" }, { name: "ward", value: null }] },
{ fields: [{ name: 'status"code', value: "pending" }, { name: "ward", value: "A" }] },
{ fields: [{ name: 'status"code', value: "closed" }, { name: "ward", value: "A" }] },
{ fields: [{ name: 'status"code', value: "transferred" }, { name: "ward", value: "B" }] },
{ fields: [{ name: 'status"code', value: "unknown" }, { name: "ward", value: "C" }] },
],
representativeValues: [
{
column: 'status"code',
values: ["active", "pending", "closed", "transferred"],
},
{ column: "ward", values: ["A"] },
],
}]);
expect(access.connect).toHaveBeenCalledWith(database, controller.signal);
expect(samples[0]!.representativeValues.flatMap((entry) => entry.values)).toHaveLength(5);
expect(query.mock.calls).toEqual([
["BEGIN TRANSACTION READ ONLY", []],
[
'SELECT LEFT(("status""code")::text, $1) AS "status""code", LEFT(("ward")::text, $1) AS "ward" FROM "clinical""data"."patient""facts" LIMIT $2',
[256, 5],
],
["ROLLBACK", []],
]);
expect(query.mock.calls.map(([sql]) => String(sql).split(" ")[0])).toEqual([
"BEGIN",
"SELECT",
"ROLLBACK",
]);
expect(end).toHaveBeenCalledOnce();
});
test("rolls back and closes the source connection when sampling fails", async () => {
const query = vi.fn(async (sql: string) => {
if (sql.startsWith("SELECT")) throw new Error("distinctive-source-secret");
return { rows: [] };
});
const end = vi.fn(async () => undefined);
const access: CatalogPostgresAccess = {
connect: vi.fn(async () => ({ query, end }) as CatalogDatabaseClient),
};
const sampler = new PostgresDescriptionSourceSampler(access);
const controller = new AbortController();
await expect(sampler.sample(database, [target], controller.signal)).rejects.toThrow();
expect(query).toHaveBeenCalledWith("ROLLBACK", []);
expect(end).toHaveBeenCalledOnce();
});
@@ -18,3 +18,23 @@ test("reserves duplicate database ids only once for a batch operation", async ()
expect(await coordinator.runMany(["database-a", "database-a"], async () => "completed"))
.toBe("completed");
});
test("stale generation recovery releases only its own reservation token", () => {
const coordinator = new CatalogOperationCoordinator();
const releaseOtherOperation = coordinator.reserve("database-a");
coordinator.releaseStale("database-a", "description_generation");
expect(() => coordinator.reserve("database-a")).toThrow(
"A database operation is already in progress",
);
releaseOtherOperation();
const releaseStaleGeneration = coordinator.reserve("database-a", "description_generation");
coordinator.releaseStale("database-a", "description_generation");
const releaseNewOperation = coordinator.reserve("database-a");
releaseStaleGeneration();
expect(() => coordinator.reserve("database-a")).toThrow(
"A database operation is already in progress",
);
releaseNewOperation();
});
@@ -169,3 +169,23 @@ test("connects pg through OpenSSH, supplies askpass, and releases all secret lea
expect(child.kill).toHaveBeenCalledWith("SIGTERM");
expect(leasedPaths.some(existsSync)).toBe(false);
});
test("fails before creating a transport when the sampling signal is already aborted", async () => {
const store = secretStore();
store.putMany("psd-clinical", {
[CATALOG_SECRET_IDS.password]: "db-password",
[CATALOG_SECRET_IDS.sshPrivateKey]: "PRIVATE KEY\n",
[CATALOG_SECRET_IDS.sshKnownHosts]: "bastion.internal ssh-ed25519 AAAATEST\n",
});
const spawnSsh = vi.fn(() => fakeChild());
const createClient = vi.fn();
const access = new ConcreteCatalogPostgresAccess(store, { spawnSsh, createClient });
const controller = new AbortController();
controller.abort();
await expect(access.connect(sshDatabase(), controller.signal)).rejects.toThrow(
"PostgreSQL connector aborted",
);
expect(spawnSsh).not.toHaveBeenCalled();
expect(createClient).not.toHaveBeenCalled();
});
@@ -9,6 +9,7 @@ import { up as upDatabases } from "../src/catalog/migrations/001_workspace_datab
import { up as upTables } from "../src/catalog/migrations/002_catalog_tables.js";
import { up as upSchemaSync } from "../src/catalog/migrations/003_catalog_schema_sync.js";
import { up as upRuntimeSequencePrivileges } from "../src/catalog/migrations/004_catalog_runtime_sequence_privileges.js";
import { up as upDescriptionGeneration } from "../src/catalog/migrations/005_description_generation_runs.js";
const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
@@ -200,3 +201,285 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository performs scoped metadata cl
await container.stop();
}
}, 60_000);
test.skipIf(!dockerAvailable)("PostgreSQL repository atomically consolidates selected table and column descriptions", async () => {
const container = await new PostgreSqlContainer("postgres:17.6-bookworm").start();
const db = new Kysely<CatalogDatabase>({
dialect: new PostgresDialect({ pool: new Pool({ connectionString: container.getConnectionUri() }) }),
plugins: [new CamelCasePlugin()],
});
try {
await upDatabases(db);
await upTables(db);
await upSchemaSync(db);
const repository = new KyselyCatalogRepository(db);
const database = await repository.create({
workspaceId: "consolidation-test",
engine: "postgres",
databaseName: "warehouse",
schema: "public",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
});
const snapshot: ObservedSchemaSnapshot = {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: [
{ name: "patients", sourceComment: null },
{ name: "visits", sourceComment: null },
],
columns: [
{ tableName: "visits", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
{ tableName: "visits", name: "patient_id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
],
relationships: [],
};
await repository.applySchemaSync(database.id, database.version, "all", [], snapshot);
const tables = await repository.listTables(database.id);
const patients = tables.find((table) => table.name === "patients")!;
const visits = tables.find((table) => table.name === "visits")!;
await repository.updateTableMetadata(
database.id, patients.id, patients.version, "Old patients", "Generated patients",
);
await repository.updateTableMetadata(
database.id, visits.id, visits.version, "Keep visits", " ",
);
expect(await repository.consolidateGeneratedDescriptions(
database.id, "tables", [patients.id, visits.id],
)).toEqual({ copied: 1, skipped: 1 });
expect(await repository.getTable(database.id, patients.id)).toMatchObject({
description: "Generated patients",
generatedDescription: "Generated patients",
version: patients.version + 2,
});
expect(await repository.getTable(database.id, visits.id)).toMatchObject({
description: "Keep visits",
generatedDescription: " ",
version: visits.version + 1,
});
const columns = await repository.listColumns(database.id, visits.id);
const id = columns.find((column) => column.name === "id")!;
const patientId = columns.find((column) => column.name === "patient_id")!;
await repository.updateColumnMetadata(
database.id, visits.id, id.id, id.version, "Old id", "Generated id",
);
await repository.updateColumnMetadata(
database.id, visits.id, patientId.id, patientId.version, "Keep patient reference", null,
);
expect(await repository.consolidateGeneratedDescriptions(
database.id, "columns", [id.id, patientId.id],
)).toEqual({ copied: 1, skipped: 1 });
expect(await repository.getColumn(database.id, visits.id, id.id)).toMatchObject({
description: "Generated id",
generatedDescription: "Generated id",
version: id.version + 2,
});
expect(await repository.getColumn(database.id, visits.id, patientId.id)).toMatchObject({
description: "Keep patient reference",
generatedDescription: null,
version: patientId.version + 1,
});
const currentVisits = (await repository.getTable(database.id, visits.id))!;
await repository.updateTableMetadata(
database.id, visits.id, currentVisits.version, "Still curated visits", "Generated visits",
);
expect(await repository.consolidateGeneratedDescriptions(database.id, "tables", [
visits.id,
"99999999-9999-4999-8999-999999999999",
])).toBeUndefined();
expect(await repository.getTable(database.id, visits.id)).toMatchObject({
description: "Still curated visits",
generatedDescription: "Generated visits",
});
} finally {
await db.destroy();
await container.stop();
}
}, 60_000);
test.skipIf(!dockerAvailable)("PostgreSQL repository persists globally exclusive Description Generation Runs and ordered events", async () => {
const container = await new PostgreSqlContainer("postgres:17.6-bookworm").start();
const db = new Kysely<CatalogDatabase>({
dialect: new PostgresDialect({ pool: new Pool({ connectionString: container.getConnectionUri() }) }),
plugins: [new CamelCasePlugin()],
});
try {
await upDatabases(db);
await upTables(db);
await upSchemaSync(db);
await upDescriptionGeneration(db);
const repository = new KyselyCatalogRepository(db);
const firstDatabase = await repository.create({
workspaceId: "generation-one",
engine: "postgres",
databaseName: "warehouse_one",
schema: "public",
binding: { transport: "postgres_direct", host: "one.internal", port: 5432, username: "reader" },
});
const secondDatabase = await repository.create({
workspaceId: "generation-two",
engine: "postgres",
databaseName: "warehouse_two",
schema: "public",
binding: { transport: "postgres_direct", host: "two.internal", port: 5432, username: "reader" },
});
await repository.applySchemaSync(firstDatabase.id, firstDatabase.version, "all", [], {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: [{ name: "patients", sourceComment: "Clinical patients" }],
columns: [{
tableName: "patients",
name: "birth_date",
ordinalPosition: 1,
dataType: "date",
isNullable: true,
defaultExpression: null,
primaryKeyPosition: null,
sourceComment: "Patient date of birth",
}],
relationships: [],
});
const table = (await repository.listTables(firstDatabase.id))[0]!;
const column = (await repository.listColumns(firstDatabase.id, table.id))[0]!;
const run = await repository.createDescriptionGenerationRun(
firstDatabase.id,
"selected_columns",
"openai-mini",
"it",
1,
);
expect(run).toMatchObject({
databaseId: firstDatabase.id,
scope: "selected_columns",
modelId: "openai-mini",
language: "it",
status: "queued",
total: 1,
processed: 0,
generated: 0,
nonGeneratable: 0,
failed: 0,
startedAt: null,
finishedAt: null,
errorSummary: null,
});
await expect(repository.createDescriptionGenerationRun(
secondDatabase.id,
"selected_columns",
"openai-mini",
"en",
1,
)).rejects.toThrow("A description generation run is already active");
await repository.updateDescriptionGenerationRun(run.id, {
status: "running",
startedAt: new Date().toISOString(),
});
await expect(repository.createDescriptionGenerationRun(
secondDatabase.id,
"selected_columns",
"openai-mini",
"en",
1,
)).rejects.toThrow("A description generation run is already active");
await repository.appendDescriptionGenerationEvent(run.id, "info", "Description generation queued.");
await repository.appendDescriptionGenerationEvent(run.id, "info", "Description generation started.");
expect(await repository.listDescriptionGenerationEvents(run.id, 1)).toEqual([
expect.objectContaining({
runId: run.id,
sequence: 2,
level: "info",
message: "Description generation started.",
createdAt: expect.any(String),
}),
]);
const updatedColumn = await repository.updateColumnMetadata(
firstDatabase.id,
table.id,
column.id,
column.version,
column.description,
"Data di nascita del paziente.",
);
expect(updatedColumn).toMatchObject({
generatedDescription: "Data di nascita del paziente.",
version: column.version + 1,
});
expect(await repository.updateDescriptionGenerationRun(run.id, {
status: "completed",
processed: 1,
generated: 1,
startedAt: new Date().toISOString(),
finishedAt: new Date().toISOString(),
})).toMatchObject({
status: "completed",
processed: 1,
generated: 1,
});
const next = await repository.createDescriptionGenerationRun(
secondDatabase.id,
"missing",
"openai-mini",
"en",
1,
);
expect(await repository.getDescriptionGenerationRun(next.id)).toMatchObject({
scope: "missing",
total: 1,
});
await repository.updateDescriptionGenerationRun(next.id, {
status: "running",
startedAt: new Date().toISOString(),
});
expect(await repository.updateDescriptionGenerationRun(next.id, {
status: "failed",
processed: 1,
failed: 1,
finishedAt: new Date().toISOString(),
errorSummary: "The model provider request failed.",
})).toMatchObject({
status: "failed",
processed: 1,
failed: 1,
errorSummary: "The model provider request failed.",
});
const allRun = await repository.createDescriptionGenerationRun(
firstDatabase.id,
"all",
"openai-mini",
"it",
2,
);
expect(await repository.getDescriptionGenerationRun(allRun.id)).toMatchObject({
scope: "all",
total: 2,
});
expect(await repository.getActiveDescriptionGenerationRun()).toMatchObject({ id: allRun.id });
expect((await repository.listDescriptionGenerationRuns(2)).map((candidate) => candidate.id)).toEqual([
allRun.id,
next.id,
]);
expect(await repository.interruptActiveDescriptionGenerationRuns(
"Description generation was interrupted by backend restart.",
)).toEqual([
expect.objectContaining({
id: allRun.id,
status: "interrupted",
finishedAt: expect.any(String),
errorSummary: "Description generation was interrupted by backend restart.",
}),
]);
expect(await repository.getActiveDescriptionGenerationRun()).toBeUndefined();
} finally {
await db.destroy();
await container.stop();
}
}, 60_000);
+207 -3
View File
@@ -5,6 +5,7 @@ import { afterEach, expect, test, vi } from "vitest";
import { buildApp } from "../src/app.js";
import { loadConfig } from "../src/config.js";
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
import { CatalogOperationCoordinator } from "../src/catalog/operation-coordinator.js";
import type { CatalogSchemaIntrospector } from "../src/catalog/schema-introspector.js";
import type { CatalogSyncRun, ObservedSchemaSnapshot } from "../src/catalog/types.js";
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
@@ -95,7 +96,7 @@ async function waitFor(repository: MemoryCatalogRepository, runId: string, state
throw new Error(`Run ${runId} did not reach ${state}`);
}
async function setup() {
async function setup(env: Record<string, string> = {}) {
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-schema-secret-"));
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-schema-runtime-"));
roots.push(secretRoot, runtimeRoot);
@@ -116,21 +117,23 @@ async function setup() {
return structuredClone(observed);
});
const introspector: CatalogSchemaIntrospector = { scan };
const operations = new CatalogOperationCoordinator();
const registry = {
list: vi.fn(async () => [revision]),
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
read: vi.fn(async () => ({ workspace, revision })),
} as unknown as WorkspaceRegistry;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test", ...env }), {
thtRunner: {} as never,
workspaceRegistry: registry,
workspaceSecretStore: new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" }),
catalogRepository: repository,
catalogSchemaIntrospector: introspector,
catalogOperationCoordinator: operations,
workspaceDiagnoser: vi.fn(),
});
return {
app, repository, database: (await repository.get(created.id))!, scan,
app, repository, database: (await repository.get(created.id))!, scan, operations,
setObserved(next: ObservedSchemaSnapshot) { observed = next; },
};
}
@@ -243,6 +246,207 @@ test("keeps generated descriptions editable and preserves them across synchroniz
expect(await repository.getColumn(database.id, patients.id, idColumn.id)).toMatchObject({ description: "Reviewed key", generatedDescription: "Generated key draft" });
});
test("consolidates non-empty generated table descriptions and reports skipped selections", async () => {
const { app, repository, database, scan } = await setup();
await seedCatalog(repository, database);
const tables = await repository.listTables(database.id);
const patients = tables.find((table) => table.name === "patients")!;
const visits = tables.find((table) => table.name === "visits")!;
await repository.updateTableMetadata(
database.id,
patients.id,
patients.version,
"Curated patients",
"Generated patients",
);
await repository.updateTableMetadata(
database.id,
visits.id,
visits.version,
"Keep curated visits",
null,
);
const response = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
payload: { target: "tables", targetIds: [patients.id, visits.id] },
});
expect(response.statusCode).toBe(200);
expect(response.json()).toEqual({ copied: 1, skipped: 1 });
expect(await repository.getTable(database.id, patients.id)).toMatchObject({
description: "Generated patients",
generatedDescription: "Generated patients",
version: patients.version + 2,
});
expect(await repository.getTable(database.id, visits.id)).toMatchObject({
description: "Keep curated visits",
generatedDescription: null,
version: visits.version + 1,
});
expect(scan).not.toHaveBeenCalled();
});
test("consolidates non-empty generated column descriptions and preserves curated text for empty proposals", async () => {
const { app, repository, database, scan } = await setup();
await seedCatalog(repository, database);
const visits = (await repository.listTables(database.id)).find((table) => table.name === "visits")!;
const columns = await repository.listColumns(database.id, visits.id);
const id = columns.find((column) => column.name === "id")!;
const patientId = columns.find((column) => column.name === "patient_id")!;
await repository.updateColumnMetadata(
database.id,
visits.id,
id.id,
id.version,
"Curated visit identifier",
"Generated visit identifier",
);
await repository.updateColumnMetadata(
database.id,
visits.id,
patientId.id,
patientId.version,
"Keep curated patient reference",
"",
);
const response = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
payload: { target: "columns", targetIds: [id.id, patientId.id] },
});
expect(response.statusCode).toBe(200);
expect(response.json()).toEqual({ copied: 1, skipped: 1 });
expect(await repository.getColumn(database.id, visits.id, id.id)).toMatchObject({
description: "Generated visit identifier",
generatedDescription: "Generated visit identifier",
version: id.version + 2,
});
expect(await repository.getColumn(database.id, visits.id, patientId.id)).toMatchObject({
description: "Keep curated patient reference",
generatedDescription: "",
version: patientId.version + 1,
});
expect(scan).not.toHaveBeenCalled();
});
test("rejects description consolidation while the Workspace Database is reserved", async () => {
const { app, repository, database, operations } = await setup();
await seedCatalog(repository, database);
const table = (await repository.listTables(database.id))[0]!;
const edited = await repository.updateTableMetadata(
database.id,
table.id,
table.version,
"Existing curated text",
"Generated text",
);
const release = operations.reserve(database.id);
try {
const response = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
payload: { target: "tables", targetIds: [table.id] },
});
expect(response.statusCode).toBe(409);
expect(response.json()).toEqual({
code: "database_operation_in_progress",
message: "A database operation is already in progress.",
});
expect(await repository.getTable(database.id, table.id)).toMatchObject({
description: "Existing curated text",
generatedDescription: "Generated text",
version: edited!.version,
});
} finally {
release();
}
});
test("requires database.manage for description consolidation", async () => {
const { app, repository, database } = await setup({ AUTH_MODE: "upstream" });
await seedCatalog(repository, database);
const table = (await repository.listTables(database.id))[0]!;
const response = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
headers: {
"x-thoth-principal-issuer": "portal",
"x-thoth-principal-subject": "catalog-reader",
"x-thoth-is-admin": "0",
},
payload: { target: "tables", targetIds: [table.id] },
});
expect(response.statusCode).toBe(403);
expect(response.json()).toEqual({ code: "auth_forbidden", error: "This operation is not permitted" });
});
test("strictly validates description consolidation database and target ids", async () => {
const { app, repository, database } = await setup();
await seedCatalog(repository, database);
const table = (await repository.listTables(database.id))[0]!;
const responses = await Promise.all([
app.inject({
method: "POST",
url: "/catalog/databases/not-a-uuid/descriptions/consolidate",
payload: { target: "tables", targetIds: [table.id] },
}),
app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
payload: { target: "tables", targetIds: ["not-a-uuid"] },
}),
app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
payload: { target: "tables", targetIds: [table.id], unexpected: true },
}),
]);
expect(responses.map((response) => response.statusCode)).toEqual([400, 400, 400]);
for (const response of responses) {
expect(response.json()).toEqual({
code: "description_consolidation_invalid",
message: "Description consolidation request is invalid.",
});
}
});
test("rejects a missing consolidation target without copying valid selections", async () => {
const { app, repository, database } = await setup();
await seedCatalog(repository, database);
const table = (await repository.listTables(database.id))[0]!;
const edited = await repository.updateTableMetadata(
database.id,
table.id,
table.version,
"Existing curated text",
"Generated text",
);
const response = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
payload: {
target: "tables",
targetIds: [table.id, "99999999-9999-4999-8999-999999999999"],
},
});
expect(response.statusCode).toBe(404);
expect(await repository.getTable(database.id, table.id)).toMatchObject({
description: "Existing curated text",
generatedDescription: "Generated text",
version: edited!.version,
});
});
test("waits for confirmation and rescans before applying destructive changes", async () => {
const { app, repository, database, scan, setObserved } = await setup();
const first = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
+8
View File
@@ -290,6 +290,14 @@ test("loadConfig accepts only an absolute generic model key file", () => {
.toThrow(/model credential configuration is invalid/);
});
test("loadConfig accepts only an absolute runtime installation descriptor path", () => {
expect(loadConfig({
THT_INSTALLATION_CONFIG_FILE: "/run/thothii-installation/thothii-installation.yaml",
}).installationConfigFile).toBe("/run/thothii-installation/thothii-installation.yaml");
expect(() => loadConfig({ THT_INSTALLATION_CONFIG_FILE: "host/thothii-installation.yaml" }))
.toThrow("installation configuration is invalid");
});
test("loadConfig accepts a file-backed catalog role and rejects partial catalog configuration", () => {
expect(loadConfig({
THT_CATALOG_DB_HOST: "catalog-db",
@@ -0,0 +1,271 @@
import { chmodSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, expect, test, vi } from "vitest";
import { buildApp } from "../src/app.js";
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
import {
loadMetadataGenerationModels,
MetadataGenerationModelUnavailableError,
} from "../src/catalog/metadata-generation-models.js";
import { loadConfig } from "../src/config.js";
import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
const roots: string[] = [];
afterEach(() => {
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
});
function metadataConfiguration(
metadataGeneration: string,
secrets = "OPENAI_API_KEY=raw-provider-secret\n",
) {
const root = mkdtempSync(join(tmpdir(), "thothii-metadata-models-"));
roots.push(root);
const installationFile = join(root, "thothii-installation.yaml");
const secretsFile = join(root, "thothii.secrets");
writeFileSync(installationFile, metadataGeneration, { mode: 0o600 });
writeFileSync(secretsFile, secrets, { mode: 0o600 });
chmodSync(installationFile, 0o600);
chmodSync(secretsFile, 0o600);
return { installationFile, secretsFile };
}
function appFor(installationFile: string, secretsFile: string) {
const config = loadConfig({
NODE_ENV: "test",
THT_HARNESS_DIR: "/missing",
THT_INSTALLATION_CONFIG_FILE: installationFile,
THT_SECRETS_FILE: secretsFile,
PI_PROVIDER: "unrelated-pi-provider",
PI_MODEL: "unrelated-pi-model",
});
return buildApp(config, {
thtRunner: {} as never,
workspaceRegistry: { list: vi.fn(async () => []) } as unknown as WorkspaceRegistry,
workspaceDiagnoser: vi.fn(),
catalogRepository: new MemoryCatalogRepository(),
});
}
test("exposes only safe metadata-generation choices and their configured default", async () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm:
provider: openai
model: gpt-4.1-mini
endpoint:
baseUrl: https://api.openai.example/v1
apiVersion: "2026-08-01"
apiKeyEnv: OPENAI_API_KEY
`);
const app = appFor(installationFile, secretsFile);
const response = await app.inject({ method: "GET", url: "/catalog/metadata-generation/models" });
expect(response.statusCode).toBe(200);
expect(response.json()).toEqual({
models: [{ id: "openai-mini", label: "OpenAI Mini" }],
default: "openai-mini",
});
expect(response.body).not.toMatch(/openai\/gpt|gpt-4\.1|api\.openai|OPENAI_API_KEY|raw-provider-secret/);
await app.close();
});
test("rejects an unprotected installation descriptor", () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm: {provider: openai, model: gpt-4.1-mini}
apiKeyEnv: OPENAI_API_KEY
`);
chmodSync(installationFile, 0o644);
expect(() => loadMetadataGenerationModels({ installationFile, secretsFile }))
.toThrow("metadata-generation installation is unavailable");
});
test("returns an empty safe catalog when no metadata-generation model is configured", async () => {
const { installationFile, secretsFile } = metadataConfiguration("profile: local\n");
const app = appFor(installationFile, secretsFile);
const response = await app.inject({ method: "GET", url: "/catalog/metadata-generation/models" });
expect(response.statusCode).toBe(200);
expect(response.json()).toEqual({ models: [], default: null });
await app.close();
});
test("resolves only a configured selection for the later generation boundary", () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm:
provider: openai
model: gpt-4.1-mini
endpoint: {baseUrl: https://api.openai.example/v1, apiVersion: "2026-08-01"}
apiKeyEnv: OPENAI_API_KEY
`);
const models = loadMetadataGenerationModels({ installationFile, secretsFile });
expect(models.resolve("openai-mini")).toEqual({
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
endpoint: { baseUrl: "https://api.openai.example/v1", apiVersion: "2026-08-01" },
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "raw-provider-secret",
});
expect(() => models.resolve("unknown-model")).toThrow(MetadataGenerationModelUnavailableError);
});
test("loads DeepSeek, GLM, and an explicit keyless Qwen endpoint from installation setup", () => {
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
default: glm-53
models:
- id: deepseek-v4-pro
label: DeepSeek V4 Pro
litellm: {provider: deepseek, model: deepseek-v4-pro}
apiKeyEnv: DEEPSEEK_API_KEY
- id: glm-53
label: GLM 5.3
litellm:
provider: openai
model: glm-5.3
endpoint: {baseUrl: https://api.z.ai/api/coding/paas/v4}
apiKeyEnv: ZAI_API_KEY
- id: qwen-36
label: Qwen 3.6
litellm:
provider: openai
model: qwen3.6-35b-a3b
disableThinking: true
endpoint: {baseUrl: https://models.internal.example/v1}
`, "DEEPSEEK_API_KEY=deepseek-secret\nZAI_API_KEY=zai-secret\n");
const models = loadMetadataGenerationModels({ installationFile, secretsFile });
expect(models.catalog()).toEqual({
models: [
{ id: "deepseek-v4-pro", label: "DeepSeek V4 Pro" },
{ id: "glm-53", label: "GLM 5.3" },
{ id: "qwen-36", label: "Qwen 3.6" },
],
default: "glm-53",
});
expect(models.resolve("deepseek-v4-pro")).toMatchObject({
apiKeyEnv: "DEEPSEEK_API_KEY",
apiKey: "deepseek-secret",
});
expect(models.resolve("qwen-36")).toEqual({
id: "qwen-36",
provider: "openai",
model: "qwen3.6-35b-a3b",
disableThinking: true,
endpoint: { baseUrl: "https://models.internal.example/v1" },
});
});
test("loads an explicit keyless endpoint without a secret bundle", () => {
const { installationFile } = metadataConfiguration(`metadataGeneration:
default: qwen-36
models:
- id: qwen-36
label: Qwen 3.6
litellm:
provider: openai
model: qwen3.6-35b-a3b
disableThinking: true
endpoint: {baseUrl: https://models.internal.example/v1}
`);
expect(loadMetadataGenerationModels({ installationFile }).resolve("qwen-36")).toEqual({
id: "qwen-36",
provider: "openai",
model: "qwen3.6-35b-a3b",
disableThinking: true,
endpoint: { baseUrl: "https://models.internal.example/v1" },
});
});
test.each([
["invalid YAML", "metadataGeneration: [\n", "OPENAI_API_KEY=secret\n", /invalid YAML/],
["duplicate ids", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
- {id: openai-mini, label: Two, litellm: {provider: openai, model: gpt-4.1}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /model id "openai-mini" is duplicated/],
["missing default", `metadataGeneration:
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /default is required/],
["unknown default", `metadataGeneration:
default: absent
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /default "absent" is not configured/],
["malformed settings", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: "open ai", model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
["malformed endpoint", `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: One
litellm: {provider: openai, model: gpt-4.1-mini, endpoint: {baseUrl: not-a-url}}
apiKeyEnv: OPENAI_API_KEY
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
["keyless hosted model without endpoint", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}}
`, "", /configuration is invalid/],
["disable thinking without endpoint", `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: One
litellm: {provider: openai, model: gpt-4.1-mini, disableThinking: true}
apiKeyEnv: OPENAI_API_KEY
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
["unallowed secret reference", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: THT_DWH_API_KEY}
`, "THT_DWH_API_KEY=secret\n", /configuration is invalid/],
["missing referenced secret", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "THT_DWH_API_KEY=secret\n", /secret "OPENAI_API_KEY" is missing/],
["unusable referenced secret", `metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`, "OPENAI_API_KEY=secret with whitespace\n", /secret "OPENAI_API_KEY" is unusable/],
] as const)("rejects %s metadata-generation configuration", (_name, yaml, secrets, expected) => {
const { installationFile, secretsFile } = metadataConfiguration(yaml, secrets);
expect(() => loadMetadataGenerationModels({ installationFile, secretsFile })).toThrow(expected);
});
test("rejects a missing secret-bundle declaration for configured models", () => {
const { installationFile } = metadataConfiguration(`metadataGeneration:
default: openai-mini
models:
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
`);
expect(() => loadMetadataGenerationModels({ installationFile }))
.toThrow("metadata-generation keyed models require THT_SECRETS_FILE");
});
+256
View File
@@ -0,0 +1,256 @@
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, expect, test } from "vitest";
import {
ModelCompletionProviderError,
PythonModelCompleter,
} from "../src/catalog/model-completer.js";
const roots: string[] = [];
afterEach(() => {
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
});
function helper(source: string, options: { terminationGraceMs?: number } = {}) {
const root = mkdtempSync(join(tmpdir(), "thothii-model-completer-"));
roots.push(root);
writeFileSync(join(root, "fake_completion_helper.py"), source, "utf8");
return new PythonModelCompleter({
pythonExecutable: "python3",
cwd: root,
helperModule: "fake_completion_helper",
timeoutMs: 5_000,
...options,
});
}
async function waitUntil(predicate: () => boolean, timeoutMs = 2_000): Promise<void> {
const deadline = Date.now() + timeoutMs;
while (!predicate()) {
if (Date.now() >= deadline) throw new Error("condition was not met before timeout");
await new Promise((resolve) => setTimeout(resolve, 10));
}
}
test("uses the short-lived Python helper stdin/stdout protocol without process arguments", async () => {
const completer = helper(`
import json
import pathlib
import sys
request = json.loads(sys.stdin.read())
pathlib.Path("request.json").write_text(
json.dumps({"request": request, "argv": sys.argv}, sort_keys=True),
encoding="utf-8",
)
sys.stdout.write(json.dumps({"ok": True, "content": "Descrizione italiana"}))
`);
const content = await completer.complete({
model: {
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
endpoint: { baseUrl: "https://models.example.test/v1", apiVersion: "2026-08-01" },
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "test-provider-secret",
},
messages: [
{ role: "system", content: "Return one description." },
{ role: "user", content: "Private metadata prompt." },
],
signal: new AbortController().signal,
});
expect(content).toBe("Descrizione italiana");
const captured = JSON.parse(readFileSync(join(roots[0]!, "request.json"), "utf8"));
expect(captured.request).toEqual({
model: "openai/gpt-4.1-mini",
api_key: "test-provider-secret",
messages: [
{ role: "system", content: "Return one description." },
{ role: "user", content: "Private metadata prompt." },
],
api_base: "https://models.example.test/v1",
api_version: "2026-08-01",
});
expect(JSON.stringify(captured.argv)).not.toMatch(/test-provider-secret|Private metadata prompt/);
});
test("omits api_key for an explicitly configured keyless endpoint", async () => {
const completer = helper(`
import json
import pathlib
import sys
request = json.loads(sys.stdin.read())
pathlib.Path("request.json").write_text(json.dumps(request, sort_keys=True), encoding="utf-8")
sys.stdout.write(json.dumps({"ok": True, "content": "Descrizione Qwen"}))
`);
await expect(completer.complete({
model: {
id: "qwen-36",
provider: "openai",
model: "qwen3.6-35b-a3b",
disableThinking: true,
endpoint: { baseUrl: "https://models.internal.example/v1" },
},
messages: [{ role: "user", content: "Describe invented metadata." }],
signal: new AbortController().signal,
})).resolves.toBe("Descrizione Qwen");
expect(JSON.parse(readFileSync(join(roots[0]!, "request.json"), "utf8"))).toEqual({
model: "openai/qwen3.6-35b-a3b",
messages: [{ role: "user", content: "Describe invented metadata." }],
api_base: "https://models.internal.example/v1",
disable_thinking: true,
});
});
test("normalizes helper failures and rejects non-pristine stdout without leaking diagnostics", async () => {
const secret = "test-provider-secret";
const prompt = "private metadata prompt";
const completers = [
helper(`
import json
import sys
request = json.loads(sys.stdin.read())
print(request["api_key"] + " " + request["messages"][0]["content"], file=sys.stderr)
sys.stdout.write(json.dumps({"ok": False, "error": "provider_failure"}))
`),
helper(`
import json
import sys
sys.stdin.read()
sys.stdout.write(json.dumps({"ok": True, "content": "first"}) + "\\n" + json.dumps({"ok": True, "content": "second"}))
`),
];
for (const completer of completers) {
let failure: unknown;
try {
await completer.complete({
model: {
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY",
apiKey: secret,
},
messages: [{ role: "user", content: prompt }],
signal: new AbortController().signal,
});
} catch (error) {
failure = error;
}
expect(failure).toBeInstanceOf(ModelCompletionProviderError);
expect(String(failure)).not.toMatch(new RegExp(`${secret}|${prompt}`));
}
});
test("aborting a completion terminates its Python helper and returns a cancellation error", async () => {
const completer = helper(`
import os
import pathlib
import signal
import sys
import time
sys.stdin.read()
def terminate(_signum, _frame):
pathlib.Path("terminated.txt").write_text("SIGTERM", encoding="utf-8")
raise SystemExit(0)
signal.signal(signal.SIGTERM, terminate)
pathlib.Path("pid.txt").write_text(str(os.getpid()), encoding="utf-8")
while True:
time.sleep(0.05)
`);
const controller = new AbortController();
const completion = completer.complete({
model: {
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "test-provider-secret",
},
messages: [{ role: "user", content: "Private metadata prompt." }],
signal: controller.signal,
});
const observed = completion.then(
() => undefined,
(error: unknown) => error,
);
const root = roots[0]!;
await waitUntil(() => existsSync(join(root, "pid.txt")));
const pid = Number(readFileSync(join(root, "pid.txt"), "utf8"));
controller.abort();
await expect(observed).resolves.toMatchObject({ name: "ModelCompletionCancelledError" });
await waitUntil(() => {
try {
process.kill(pid, 0);
return false;
} catch {
return true;
}
});
expect(readFileSync(join(root, "terminated.txt"), "utf8")).toBe("SIGTERM");
});
test("aborting escalates to SIGKILL when the Python helper does not exit after SIGTERM", async () => {
const completer = helper(`
import os
import pathlib
import signal
import sys
import time
sys.stdin.read()
def ignore_term(_signum, _frame):
pathlib.Path("sigterm.txt").write_text("received", encoding="utf-8")
signal.signal(signal.SIGTERM, ignore_term)
pathlib.Path("pid.txt").write_text(str(os.getpid()), encoding="utf-8")
while True:
time.sleep(0.05)
`, { terminationGraceMs: 25 });
const controller = new AbortController();
const observed = completer.complete({
model: {
id: "openai-mini",
provider: "openai",
model: "gpt-4.1-mini",
apiKeyEnv: "OPENAI_API_KEY",
apiKey: "test-provider-secret",
},
messages: [{ role: "user", content: "Private metadata prompt." }],
signal: controller.signal,
}).then(
() => undefined,
(error: unknown) => error,
);
const root = roots[0]!;
await waitUntil(() => existsSync(join(root, "pid.txt")));
const pid = Number(readFileSync(join(root, "pid.txt"), "utf8"));
controller.abort();
await expect(observed).resolves.toMatchObject({ name: "ModelCompletionCancelledError" });
expect(readFileSync(join(root, "sigterm.txt"), "utf8")).toBe("received");
await waitUntil(() => {
try {
process.kill(pid, 0);
return false;
} catch {
return true;
}
});
});
+15 -1
View File
@@ -2,7 +2,12 @@ import { afterEach, expect, test } from "vitest";
import { chmodSync, mkdtempSync, renameSync, rmSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { loadSecretBundle, loadSecretBundleWithFs, secretValue } from "../src/config/secret-bundle.js";
import {
loadSecretBundle,
loadSecretBundleWithFs,
METADATA_GENERATION_SECRET_KEYS,
secretValue,
} from "../src/config/secret-bundle.js";
const dirs: string[] = [];
afterEach(() => { for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); });
@@ -22,6 +27,15 @@ test("parses comments, blank lines and values containing equals", () => {
]));
});
test("existing secret consumers accept a bundle containing an allowed metadata-model key", () => {
const file = bundle("THT_DWH_API_KEY=dwh-secret\nOPENAI_API_KEY=metadata-secret\n");
expect(secretValue({ secretsFile: file }, "THT_DWH_API_KEY")).toBe("dwh-secret");
});
test.each(METADATA_GENERATION_SECRET_KEYS)("accepts audited metadata-model key %s", (name) => {
expect(loadSecretBundle(bundle(`${name}=metadata-secret\n`)).get(name)).toBe("metadata-secret");
});
test("accepts the fixed OIDC and Authentik secret references", () => {
const file = bundle("THT_OIDC_CLIENT_SECRET=oidc-secret\nTHT_AUTHENTIK_API_TOKEN=authentik-token\n");
expect(loadSecretBundle(file)).toEqual(new Map([
+8
View File
@@ -22,6 +22,7 @@ services:
THT_WORKSPACE_SECRET_RUNTIME_ROOT: /tmp/thothii-workspace-secrets
THT_WORKSPACE_SECRET_ROOTS: /run/secrets
THT_SECRETS_FILE: /run/secrets/thothii.secrets
THT_INSTALLATION_CONFIG_FILE: /run/thothii-installation/thothii-installation.yaml
THT_PI_AUTH_FILE: /home/thoth/.pi/agent/auth.json
THT_AUTH_CONFIG_FILE: /run/thothii-auth/auth.yaml
THT_AUTH_STATE_ROOT: /data/auth
@@ -53,6 +54,9 @@ services:
- source: thothii_secrets
target: thothii.secrets
- catalog_runtime_password
configs:
- source: thothii_installation_config
target: /run/thothii-installation/thothii-installation.yaml
healthcheck:
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"]
interval: 15s
@@ -271,3 +275,7 @@ secrets:
file: "${THT_CATALOG_RUNTIME_PASSWORD_SOURCE:-./deploy/secrets/catalog-runtime-password}"
catalog_migrator_password:
file: "${THT_CATALOG_MIGRATOR_PASSWORD_SOURCE:-./deploy/secrets/catalog-migrator-password}"
configs:
thothii_installation_config:
file: "${THT_INSTALLATION_CONFIG_SOURCE:?set THT_INSTALLATION_CONFIG_SOURCE}"
+1
View File
@@ -6,6 +6,7 @@ THOTH_CORE_HTTP_PORT=8787
MAX_PI_PROCESSES=4
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
THT_INSTALLATION_CONFIG_SOURCE=/absolute/path/to/thothii-installation.yaml
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
THT_CATALOG_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/catalog-runtime-password
THT_CATALOG_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/catalog-migrator-password
+1
View File
@@ -5,6 +5,7 @@ THOTH_HTTP_PORT=8080
MAX_PI_PROCESSES=4
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
THT_INSTALLATION_CONFIG_SOURCE=/absolute/path/to/thothii-installation.yaml
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
THT_CATALOG_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/catalog-runtime-password
THT_CATALOG_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/catalog-migrator-password
+1
View File
@@ -8,6 +8,7 @@ THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=<abs>/deploy/psd/secrets/git-known-hosts
# App
THT_SECRETS_FILE=<abs>/deploy/psd/secrets/thothii.secrets
THT_INSTALLATION_CONFIG_SOURCE=<abs>/deploy/psd/thothii-installation.yaml
PI_AUTH_FILE=<abs>/deploy/psd/secrets/pi-auth.json
THT_AUTH_CONFIG_ROOT=<abs>/deploy/psd/auth
# DWH and Evidence credentials are entered later in Workspace management and stored encrypted
@@ -8,6 +8,32 @@ workspaceRepository:
remote: git@github.com:mptyl/tht-workspace-psd.git
branch: main
access: ssh
metadataGeneration:
default: glm-53
models:
- id: deepseek-v4-pro
label: DeepSeek V4 Pro
litellm:
provider: deepseek
model: deepseek-v4-pro
apiKeyEnv: DEEPSEEK_API_KEY
- id: glm-53
label: GLM 5.3
litellm:
provider: openai
model: glm-5.3
endpoint:
baseUrl: https://api.z.ai/api/coding/paas/v4
apiKeyEnv: ZAI_API_KEY
- id: qwen-36
label: AritmoLab Qwen 3.6 35B A3B
litellm:
provider: openai
model: qwen3.6-35b-a3b
disableThinking: true
endpoint:
baseUrl: https://ml-aritmolab.policlinicosandonato.it/v1
# Qwen omette apiKeyEnv: l'endpoint VPN non autentica le richieste.
authentication:
configDirectory: "<abs>/projects/ThothII/deploy/psd/auth"
overrides:
+18 -5
View File
@@ -9,11 +9,24 @@ chmod 600 deploy/secrets/thothii.secrets
```
The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). The supported
keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`,
`THT_OIDC_CLIENT_SECRET`, and `THT_AUTHENTIK_API_TOKEN`. The two authentication keys are fixed
empty entries for local authentication and must be populated only in a protected OIDC installation.
Other configured values must be non-empty and contain no whitespace. Do not put secrets
in the root `.env`, workspace YAML, URLs, logs, or rendered Compose output.
installation keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`,
`THT_OIDC_CLIENT_SECRET`, and `THT_AUTHENTIK_API_TOKEN`. Metadata-generation models may reference
exactly one of `THT_METADATA_API_KEY`, `ANTHROPIC_API_KEY`, `AZURE_API_KEY`, `GEMINI_API_KEY`,
`DEEPSEEK_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY`, or `ZAI_API_KEY` through their
descriptor `apiKeyEnv`. An entry for an explicitly configured endpoint that accepts unauthenticated
requests may omit `apiKeyEnv`; hosted/default endpoints must always reference a key.
OpenAI-compatible Qwen endpoints that emit reasoning in `content` can additionally set
`litellm.disableThinking: true`; the adapter sends the server's bounded chat-template flag so the
strict JSON result remains parseable.
The two authentication keys must be omitted until they have non-empty values in a protected OIDC
installation. Other configured values must be non-empty and contain no
whitespace. Do not put secrets in the root `.env`, installation YAML, workspace YAML, URLs, logs,
or rendered Compose output.
`THT_MODEL_API_KEY` remains the generic Pi child credential. Metadata generation is a separate
backend-owned runtime and reads only the key named by its own `metadataGeneration.models[].apiKeyEnv`
(when present);
it does not read Pi settings, `PI_AUTH_FILE`, or workspace `llm_policy`.
Do not add vector or embedding endpoint credentials to the bundle. Active operator manuals use
internal Qdrant and Ollama services, so vector/embedding runtime endpoint secrets are not part of
+9 -3
View File
@@ -5,12 +5,18 @@
# Hosted model provider (single-key providers only).
# THT_MODEL_API_KEY=replace-me
# Description Generation only. The selected name must match apiKeyEnv in metadataGeneration.
# Allowed names: THT_METADATA_API_KEY, ANTHROPIC_API_KEY, AZURE_API_KEY, GEMINI_API_KEY,
# DEEPSEEK_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, or ZAI_API_KEY.
# OPENAI_API_KEY=replace-me
# A model with an explicit unauthenticated endpoint omits apiKeyEnv and needs no bundle entry.
# External DWH adapter.
# THT_DWH_API_KEY=replace-me
# Optional CA material/path understood by the configured adapter.
# THT_CA=/run/secrets/ca-chain.pem
# OIDC/Authentik references. Keep these fixed keys empty until OIDC is configured.
THT_OIDC_CLIENT_SECRET=
THT_AUTHENTIK_API_TOKEN=
# OIDC/Authentik references. Uncomment only with non-empty values when configured.
# THT_OIDC_CLIENT_SECRET=replace-me
# THT_AUTHENTIK_API_TOKEN=replace-me
@@ -0,0 +1,32 @@
---
status: accepted
---
# Use one sequential description-generation run
AI description generation is an infrequent, administrator-triggered catalog operation. The
backend therefore owns one installation-wide asynchronous run and processes its model requests
sequentially. A second generation start is rejected while that run is active. The existing
catalog-operation coordinator also reserves the target Workspace Database so synchronization,
cleanup, and catalog edits cannot overlap the run.
Each model completion is performed by a short-lived internal Python process using LiteLLM. The
backend sends a structured request on stdin, reads pristine structured output from stdout, and
keeps diagnostics on stderr. This helper is neither an HTTP service nor a user-facing CLI. Using
Pi for this operation would couple a deterministic batch task to interactive session lifecycle and
gate behavior without adding product value; the helper reuses Python already present in the core
image while keeping the integration small.
Installation model entries normally reference a protected API key. A reference may be omitted
only for an explicit unauthenticated endpoint; the helper uses a fixed non-secret client
placeholder because OpenAI-compatible SDKs require a non-empty client value even when the server
ignores authentication.
For an explicitly configured Qwen-compatible endpoint, `disableThinking: true` maps to the narrow
chat-template option that prevents reasoning prose from surrounding the required JSON result.
Persistence is deliberately limited to one run record, ordered text events, and the Generated
Description written to each target as soon as it succeeds. There are no durable per-target jobs,
leases, invocation records, automatic resume, or distributed locks. On backend startup, any run
still recorded as queued or running becomes interrupted. An administrator continues by starting
Generate Missing, and may use Unlock only when no live generation process exists. Cancellation
stops the loop and terminates the current helper process.
@@ -0,0 +1,15 @@
---
status: accepted
---
# Allow bounded real source samples for description generation
Description generation may send up to five real rows and up to five distinct non-null example
values for relevant columns to the configured model provider, preserving the useful behavior of
ThothAI. Samples are read only for the current request, treated as untrusted data, bounded before
prompt construction, and never stored in generation runs, events, or catalog metadata.
This first slice makes that behavior explicit in the UI and operator documentation. A follow-up
Sensitive Data Policy is required to classify protected fields and exclude or anonymize their
values before model calls. Until that policy exists, administrators must regard generation as a
controlled disclosure of sampled source data to the selected provider.
+1 -1
View File
@@ -33,7 +33,7 @@ The production role expansion from `backend/src/auth/config.ts` is exact:
| Role | Permissions |
|---|---|
| `user` | `session.use` |
| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, `workspace.manage`, `workspace.secrets.manage`, `pi.manage`, `auth.diagnostics.read` |
| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, `workspace.manage`, `workspace.secrets.manage`, `database.manage`, `pi.manage`, `auth.diagnostics.read` |
`admin` therefore includes the ordinary `session.use` permission. No other role or permission
label is part of the production catalog.
+18 -2
View File
@@ -22,7 +22,9 @@ flowchart LR
THT --> VDB["Qdrant / vector store"]
BE --> CFG["settings.json\nworkspace registry"]
BE --> CAT["catalog-db\nPostgreSQL + Kysely"]
BE -->|catalog Test + Table Sync| DWH
BE -->|catalog Test + Sync + bounded AI sampling| DWH
BE -->|one request per subprocess| LLMHELPER["LiteLLM helper\nPython, short-lived"]
LLMHELPER -->|configured model| PROVIDER["AI provider"]
FE -.->|renders widgets| EXT
```
@@ -31,7 +33,7 @@ Dipendenze principali:
| Module | Depends on | Responsibility |
| --- | --- | --- |
| `frontend/` | Backend REST and SSE APIs | UI, gate widgets, and in-memory transcript |
| `backend/src/` | Pi, `tht`, configuration, workspace registry, catalog PostgreSQL, and read-only DWH connectors | Transport, session lifecycle, catalog CRUD, connection tests, table introspection, and APIs |
| `backend/src/` | Pi, `tht`, configuration, workspace registry, catalog PostgreSQL, read-only DWH connectors, and the internal LiteLLM helper | Transport, session lifecycle, catalog CRUD, connection tests, table introspection, sequential AI description generation, and APIs |
| `harness/.pi/` | Pi and `tht phase` | Workflow orchestration and human-in-the-loop gates |
| `harness/tht/` | Filesystem, DWH, and vector store | Persistence, CLI, Evidence, schema, and preprocessing |
| workspace repository | `source/`, `curated/`, manifest, and artifacts | Versioned Evidence source and session output |
@@ -65,6 +67,20 @@ sequenceDiagram
The backend uses `ThtRunner` for CLI subprocesses, `PiProcessManager` for one Pi process per session, `SessionBridge` to adapt RPC events, and `SseHub` to distribute them to clients.
## Catalog description generation
Catalog description generation is a backend-owned administrative operation, separate from the
Pi session workflow and from the public `tht` CLI. The frontend starts one run for selected catalog
tables or columns. A single installation-wide worker processes targets sequentially, reads at most
the configured bounded sample from the source DWH through its read-only connection, and invokes a
short-lived Python LiteLLM helper once per target. The selected model comes from the installation
descriptor; its API key remains in the protected installation secret bundle.
Each result is written immediately to `Generated Description`. Run state and sanitized activity
events are stored in `catalog-db` and exposed to the drawer through REST and SSE. An administrator
may later copy selected generated descriptions into `Description`. There is no parallel run queue,
automatic retry policy, or second orchestration subsystem.
## Main backend classes
The diagram shows the classes that form the bridge between the browser, Pi, and `tht`. Fastify routes receive requests and delegate to these services.
+12 -3
View File
@@ -12,6 +12,8 @@ flowchart LR
USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
FE --> BE["Backend\nFastify"]
BE --> CATALOG["Metadata catalog\nPostgreSQL"]
BE --> MODEL["Configured AI model\nvia short-lived LiteLLM helper"]
BE -->|bounded read-only samples| DWH
BE --> PI["Pi\nRPC per sessione"]
PI --> THT["tht and harness\nworkflow and persistence"]
THT --> DWH["DWH\nread only"]
@@ -50,12 +52,19 @@ A session is a directory under `sessions/` (the workspace defines the path): `se
configurations stored in PostgreSQL through Kysely.
- `CatalogTableService` reconciles persisted Catalog Tables with a successful external schema scan;
`ConcreteCatalogTableIntrospector` isolates direct PostgreSQL, typed REST, and SSH-tunnel access.
- `DescriptionGenerationWorker` admits one installation-wide run and processes its table or column
targets sequentially.
- `PostgresDescriptionSourceSampler` reads bounded real rows and five representative examples from
the configured DWH connection; `ProcessModelCompleter` invokes the short-lived Python LiteLLM
helper with the installation-selected model.
Application settings remain in `backend/data/settings.json`; session state remains in harness phase
documents. PostgreSQL stores only the administrative database catalog, bindings, observed tables,
and curated descriptions. Connector secrets remain write-only in the encrypted workspace secret
store. Catalog SSH support is limited to connection tests and table synchronization; it does not
change the session runtime binding contract.
curated and generated descriptions, description-generation runs, and sanitized run events.
Connector secrets remain write-only in the encrypted workspace secret store; model credentials
remain in the protected installation secret bundle. Catalog SSH support is limited to connection
tests, table synchronization, and bounded description-generation sampling; it does not change the
session runtime binding contract.
## Human-in-the-loop gate contract
@@ -1,4 +1,5 @@
# Copy this file to an operator-controlled path named exactly thothii-installation.yaml.
# Copy this file to a protected operator-controlled path named exactly thothii-installation.yaml
# and set mode 0600 (or 0400) before using it as THT_INSTALLATION_CONFIG_SOURCE.
# Replace every absolute placeholder. Select exactly one Git transport override.
profile: local
projectDirectory: "/absolute/path/to/ThothII"
@@ -7,6 +8,17 @@ workspaceRepository:
remote: git@git.example.com:organization/workspaces.git
branch: main
access: ssh
metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm:
provider: openai
model: gpt-4.1-mini
apiKeyEnv: OPENAI_API_KEY
# apiKeyEnv may be omitted only for an explicit endpoint that accepts
# unauthenticated requests.
authentication:
configDirectory: "/absolute/path/to/thothii-auth"
overrides:
@@ -1,4 +1,5 @@
# Copy this file to a protected operator path named exactly thothii-installation.yaml.
# Copy this file to a protected operator path named exactly thothii-installation.yaml
# and set mode 0600 (or 0400) before using it as THT_INSTALLATION_CONFIG_SOURCE.
# Replace every absolute placeholder. Select exactly one Git transport override.
profile: server
projectDirectory: "/absolute/path/to/ThothII"
@@ -7,6 +8,19 @@ workspaceRepository:
remote: git@git.example.com:organization/workspaces.git
branch: main
access: ssh
metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm:
provider: openai
model: gpt-4.1-mini
endpoint:
baseUrl: https://api.openai.example/v1
apiKeyEnv: OPENAI_API_KEY
# apiKeyEnv may be omitted only for an explicit endpoint that accepts
# unauthenticated requests.
authentication:
# Root-operated source of truth; it is never mounted into core.
configDirectory: "/srv/example/thothii/auth-canonical"
+17
View File
@@ -42,6 +42,10 @@ cp deploy/env/local.env.example deploy/env/local.env
cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets
chmod 600 deploy/secrets/thothii.secrets
# Copy docs/install/examples/thothii-installation.local.yaml to a protected operator path,
# replace every placeholder, chmod it 600, then set that exact path as
# THT_INSTALLATION_CONFIG_SOURCE in deploy/env/local.env.
docker compose --env-file deploy/env/local.env \
-f compose.yaml -f deploy/compose.local.yaml up --build -d
```
@@ -50,6 +54,7 @@ Set these values in `deploy/env/local.env`:
- `PI_AUTH_FILE`
- `THT_SECRETS_FILE`
- `THT_INSTALLATION_CONFIG_SOURCE` (the exact protected host `thothii-installation.yaml`)
- `THT_WORKSPACE_GIT_REMOTE`
- DWH endpoint
- LLM endpoint
@@ -64,8 +69,20 @@ The documented and supported bundle keys are:
```dotenv
THT_MODEL_API_KEY=...
THT_DWH_API_KEY=...
OPENAI_API_KEY=...
```
`THT_MODEL_API_KEY` remains Pi-only. A metadata-generation model instead references one audited
bundle name from `deploy/secrets/README.md` through `metadataGeneration.models[].apiKeyEnv`.
`apiKeyEnv` may be omitted only for an explicit endpoint that accepts unauthenticated requests;
hosted/default endpoints remain keyed.
Provider/model/endpoint settings stay in the protected installation descriptor; raw keys do not.
Compose mounts exactly `THT_INSTALLATION_CONFIG_SOURCE` into `core` as a read-only config and sets
the backend-only runtime path `THT_INSTALLATION_CONFIG_FILE` to
`/run/thothii-installation/thothii-installation.yaml`. Do not set the runtime path in the host env.
Descriptor and bundle changes are loaded only after application restart.
A private PEM CA remains outside the bundle and must be mounted through a reviewed Compose override.
## Preprocessing
@@ -0,0 +1,249 @@
# AI-generated descriptions for Catalog Tables and Catalog Columns
## Problem Statement
ThothII already stores a Generated Description separately from the curated Description for Catalog
Tables and Catalog Columns, but administrators cannot populate it with AI. ThothAI provides the
useful core workflow—generate table and column comments from schema context and small real-data
samples—but its execution, configuration, and interaction model cannot be copied directly into
ThothII.
Administrators need an asynchronous workflow integrated into Database Management. They must be
able to choose an installation-approved model, generate descriptions for selected or missing
targets, observe understandable progress, stop or recover a stuck operation, review generated
text, and explicitly consolidate it. The solution must retain ThothAI's practical simplicity and
must not introduce a general job platform, model gateway, distributed scheduler, or competing
user-facing CLI.
## Solution
Add Description Generation to Database Management as one installation-wide, sequential background
run owned by the Fastify backend. The browser starts a run and remains responsive while the backend
processes bounded requests one at a time. Each completion is delegated to a short-lived internal
Python helper using LiteLLM. Models, their default, and any API-key secret references are declared in
application setup YAML and are independent of both workspaces and Pi configuration.
Each valid result is written immediately to the target's Generated Description. A minimal run row
and ordered text events provide status, counters, history, and a live log. A stopped or crashed run
is not resumed automatically; completed results remain in place and Generate Missing supplies the
simple recovery path. An Unlock action marks a stale recorded run interrupted only when no helper
or backend generation loop is alive.
Prompts use catalog context and, when available, no more than five real rows and five representative
non-null examples. Samples are transient and never logged or persisted. A valid inability to infer
a description produces a standard application-localized value such as `Non generabile`; provider,
timeout, and response-validation failures remain technical errors.
Generated text remains separate from Description until an administrator uses the existing
checkbox selection and Actions control to consolidate it. Consolidation retains Generated
Description and never writes comments to the external Workspace Database.
## User Stories
1. As an installation operator, I want to declare the models allowed for metadata generation in setup YAML, so that model availability is controlled centrally.
2. As an installation operator, I want to declare one default metadata-generation model, so that administrators begin with a safe operational choice.
3. As an installation operator, I want each model to reference its own API-key secret, so that credentials are not stored in workspaces or browser-visible settings.
4. As an installation operator, I want metadata-generation models to remain independent of Pi models, so that changing this workflow cannot disrupt the core NL-to-SQL experience.
5. As an installation operator, I want invalid model setup to fail validation clearly, so that the application does not start with ambiguous provider behavior.
6. As a Catalog Administrator, I want generation controls to explain when no model is configured, so that I know why the action is unavailable.
7. As a Catalog Administrator, I want to select an approved model from a selector initialized to the setup default, so that I control which model performs the work.
8. As a Catalog Administrator, I want to generate descriptions for selected Catalog Tables, so that I can work on a focused part of the catalog.
9. As a Catalog Administrator, I want to generate descriptions for selected Catalog Columns, so that I can work on individual fields without regenerating a whole table.
10. As a Catalog Administrator, I want to generate all eligible descriptions for a Workspace Database, so that I can initialize a catalog in one operation.
11. As a Catalog Administrator, I want to generate only missing descriptions, so that I can continue interrupted work without replacing completed proposals.
12. As a Catalog Administrator, I want a full run to process Catalog Columns before their Catalog Tables, so that table descriptions can benefit from column descriptions.
13. As a Catalog Administrator, I want generation to run asynchronously after I start it, so that the browser remains usable and progress is not tied to one HTTP request.
14. As a Catalog Administrator, I want only one Description Generation Run active in the installation, so that provider traffic and operational behavior remain predictable.
15. As a Catalog Administrator, I want a second start attempt to return a clear conflict, so that I cannot accidentally overlap generation runs.
16. As a Catalog Administrator, I want synchronization, cleanup, consolidation, and edits for the target Workspace Database blocked during generation, so that the simple sequential run sees stable catalog state.
17. As a Catalog Administrator, I want to see the run's model, scope, status, counters, and timestamps, so that I understand what is happening.
18. As a Catalog Administrator, I want a chronological text log, so that I can follow completed targets and diagnose errors.
19. As a Catalog Administrator, I want live log updates with a polling fallback, so that temporary SSE problems do not hide run progress.
20. As a Catalog Administrator, I want completed and interrupted runs to remain inspectable, so that I can understand prior activity.
21. As a Catalog Administrator, I want to stop an active run, so that I can halt an incorrect or unexpectedly costly operation.
22. As a Catalog Administrator, I want stopping a run to terminate its current model helper and prevent later targets from starting, so that stop has prompt operational effect.
23. As a Catalog Administrator, I want valid results completed before a stop or failure to remain saved, so that useful work is not discarded.
24. As a Catalog Administrator, I want a run left active by a backend restart to become interrupted, so that the UI does not claim nonexistent work is still running.
25. As a Catalog Administrator, I want to unlock a stale active run when no generation process is alive, so that an erroneous recorded lock cannot block future work.
26. As a Catalog Administrator, I want Unlock rejected while a live generation process exists, so that recovery cannot create an overlapping run.
27. As a Catalog Administrator, I want Generate Missing to continue after interruption, so that recovery does not require a special resume mechanism.
28. As a Catalog Administrator, I want one retry for a transient model failure, so that a brief provider fault does not immediately lose a batch.
29. As a Catalog Administrator, I want the run to fail after three consecutive technical failures, so that a broken provider does not generate an unbounded stream of attempts.
30. As a Catalog Administrator, I want a successful request to reset the consecutive-failure count, so that isolated errors do not prematurely stop a useful run.
31. As a Catalog Administrator, I want a completed-with-errors result when isolated batches fail but the run reaches its end, so that partial problems remain visible.
32. As a Catalog Administrator, I want no automatic fallback to a different model, so that the selected model remains truthful and predictable.
33. As a Catalog Administrator, I want malformed or ambiguous model output rejected without writing it, so that descriptions cannot be assigned to the wrong target.
34. As a Catalog Administrator, I want an inability to infer a description represented by standard localized text, so that every valid outcome is understandable in the workspace language.
35. As a Catalog Administrator, I want technical failures kept distinct from non-generatable outcomes, so that provider problems are not mistaken for catalog knowledge.
36. As a Catalog Administrator, I want generated prose written in the workspace language, so that it matches the catalog's intended audience.
37. As a Catalog Administrator, I want generated text stored separately from curated Description, so that AI output remains a reviewable proposal.
38. As a Catalog Administrator, I want to edit a Generated Description manually, so that I can improve a proposal before consolidation.
39. As a Catalog Administrator, I want to select one or more tables or columns and run “Move generated description to Description” from the existing Actions control, so that review remains integrated into the current grids.
40. As a Catalog Administrator, I want consolidation to retain the Generated Description, so that I can still see the proposal from which the curated text was copied.
41. As a Catalog Administrator, I want selected records without a Generated Description skipped and reported, so that the bulk action does not erase curated text.
42. As a Catalog Administrator, I want consolidation and generation to modify only the Metadata Catalog, so that no external database comment is changed.
43. As a Catalog Administrator, I want prompts to use schema facts and existing catalog text, so that generated descriptions are grounded in available metadata.
44. As a Catalog Administrator, I want prompts to use at most five real source rows and five representative values when available, so that the model has useful examples without unbounded disclosure.
45. As a Catalog Administrator, I want to be warned that real source samples are sent to the selected provider, so that I can make an informed disclosure decision.
46. As a Catalog Administrator, I want sampled rows and values excluded from persistence and logs, so that operational history does not become a secondary data store.
47. As a security operator, I want API keys, prompts, samples, and complete provider payloads redacted from logs, so that diagnostics do not leak secrets or source data.
48. As a support operator, I want concise per-target and per-batch event messages, so that failures can be diagnosed without provider-specific internals.
49. As an authorized administrator, I want all generation, cancellation, unlock, and consolidation actions protected by database-management permission, so that ordinary users cannot mutate catalog metadata.
50. As an unauthorized user, I want generation controls hidden or disabled and API calls rejected, so that frontend visibility is not treated as authorization.
51. As an operator, I want setup changes to take effect after an application restart, so that configuration lifecycle remains simple and explicit.
52. As a product owner, I want the first release to avoid queues, parallel calls, distributed locks, and automatic resume, so that effort remains focused on generating and reviewing useful descriptions.
## Implementation Decisions
- The Fastify backend owns one installation-wide Description Generation Run and its sequential
processing loop. It does not delegate lifecycle ownership to Pi or Python.
- A Description Generation Run has one of `queued`, `running`, `completed`,
`completed_with_errors`, `cancelled`, `failed`, or `interrupted`. It stores the Workspace
Database, requested scope, selected model identifier, workspace language, progress counters,
timestamps, and an optional final error summary.
- Ordered Description Generation Events store timestamp, severity, and safe human-readable text.
No durable per-target jobs, model invocation rows, prompt snapshots, sample snapshots, leases,
heartbeats, registry revisions, or provenance chains are introduced.
- Starting a run schedules an in-process background loop and returns the run immediately. The API
exposes start, run/history lookup, event listing and streaming, cancellation, stale-run unlock,
and the safe list of configured model choices. There are no retry-item or resume endpoints.
- One in-memory generation manager enforces the installation-wide active-run rule. The existing
Catalog Operation Coordinator reserves the target Workspace Database for the duration of the
run, without being generalized into a new operation framework.
- On backend startup, persisted `queued` or `running` Description Generation Runs become
`interrupted`. The application performs no automatic replay or resume.
- Unlock succeeds only when no live generation loop or helper child exists. It marks the stale run
interrupted and releases the local reservation; it is not a distributed lock recovery protocol.
- Each model completion uses a short-lived Python helper backed by LiteLLM. Structured input is
supplied over stdin, structured output alone is emitted on stdout, diagnostics use stderr, and
the helper can be terminated by cancellation.
- A completion request contains no more than ten targets. Requests run one at a time. The helper
performs at most one retry for a transient technical failure.
- Three consecutive model-request failures fail the run. A successful request resets that count.
Isolated exhausted failures may be logged and skipped, producing `completed_with_errors` if the
run later reaches its end.
- Every valid generated or non-generatable result is applied immediately to Generated Description.
Earlier writes are retained after cancellation, interruption, or later failure.
- A response must identify requested targets unambiguously and classify each returned result as
generated or non-generatable. Duplicate, unknown, missing, or malformed mappings cause a
technical request failure and no result from that ambiguous response is applied.
- The parser also tolerates one JSON object enclosed by one complete `json` code fence, because
some supported models add that formatting despite the prompt. Any prose outside the fence,
multiple payloads, or malformed/ambiguous mappings remain invalid.
- The application supplies localized standard non-generatable text. Provider wording is not used
as the standard value, and technical errors never write that value.
- A full-database run generates eligible Catalog Columns before Catalog Tables. Generate Missing
excludes targets whose Generated Description is already non-empty; all-generation may replace
existing generated proposals only after the initiating action makes that scope explicit.
- Model choices are declared under a metadata-generation section in installation setup YAML. Each
choice has a stable identifier, display label, LiteLLM provider/model settings, optional endpoint
settings, and an optional environment-secret reference for its API key. The reference may be
omitted only when an explicit endpoint is configured for unauthenticated access. One identifier
is the default.
- An explicit endpoint may set `disableThinking: true`; the helper translates it only to the
Qwen-compatible chat-template switch needed to keep the response within the strict JSON contract.
- Metadata-generation setup is separate from application settings for Pi and from workspace
`llm_policy`. Raw keys never enter setup YAML, the catalog database, API responses, process
arguments, or event text. Configuration reload is restart-only.
- If setup defines no usable model, the safe model-list response is empty and the UI disables
generation with an explanation. The backend still rejects direct generation attempts.
- Prompt construction treats schema names, comments, descriptions, and values as untrusted data.
It requests output in the workspace language and separates instructions from catalog content.
- A request may contain up to five real source rows and up to five representative distinct,
non-null values for relevant columns. Inputs are bounded before prompt construction and are not
persisted or logged.
- The UI discloses that real data can be sent to the selected provider. A future Sensitive Data
Policy will classify values and exclude or anonymize protected data; that policy is not silently
approximated in this slice.
- The generation UI reuses Database Management's table and column selections, model selector,
Actions control, run drawer conventions, SSE delivery, and polling fallback where practical.
Visual parity with Catalog Sync Run logs is not required.
- The consolidation action copies each selected, non-empty Generated Description into Description
in a catalog transaction, retains Generated Description, skips empty proposals, and reports
copied and skipped counts. It never writes to the external Workspace Database.
- Generation, cancellation, unlock, and consolidation require the existing database-management
permission and are validated by the backend independently of UI state.
- No user-facing generation CLI is added. The Python process is an internal completion adapter,
not an operator surface or a long-lived service.
## Testing Decisions
- Tests assert externally observable behavior rather than private loop structure, process timing,
or LiteLLM implementation details.
- The primary and highest test seam is the Fastify catalog API with a test PostgreSQL catalog and
an injected fake Model Completer. It verifies complete paths through authorization, run
persistence, sequential processing, event delivery, Generated Description updates, and final
status without contacting a real provider.
- API tests cover each generation scope, column-before-table order, the ten-target request bound,
model validation, one-active-run conflict, target-database exclusion, cancellation, startup
interruption, Unlock safeguards, Generate Missing, partial success, consecutive failure
handling, non-generatable localization, malformed responses, redacted events, and permissions.
- Catalog repository integration tests verify the migration, run and event ordering, active-run
constraint, immediate description writes, history queries, startup interruption, and bulk
consolidation behavior against PostgreSQL.
- The Python helper has a small black-box contract suite using a simulated LiteLLM adapter. It
verifies stdin/stdout framing, pristine stdout, stderr diagnostics, normalized success and
failure output, one transient retry, secret redaction, and termination behavior.
- Setup-validation tests cover duplicate model identifiers, missing or unknown defaults, malformed
provider settings, missing secret references, safe public model projection, and strict separation
from Pi and workspace model settings.
- Database Management tests use the existing browser-level component seam with MSW. They verify
model selection and default, selected/all/missing actions, disabled state without models, running
progress and logs, polling recovery, cancellation, Unlock visibility, terminal summaries,
generated-text refresh, and selected consolidation with copied/skipped counts.
- Existing Catalog Sync Run route, repository, SSE, and drawer tests are prior art for asynchronous
status and event behavior. Existing catalog table/column editing and Database Management tests
are prior art for optimistic catalog updates, permissions, selection, and action controls.
- One required manual acceptance gate, outside deterministic CI, uses the installation's configured
default model and a disposable PostgreSQL database containing only invented data. Its application
credentials are read-only. It generates Italian text for one Catalog Column and one Catalog
Table, verifies their Generated Description, inspects the safe activity log, confirms that no key
or sample value is exposed, and consolidates one selected result. If the configured secret is not
available, acceptance stops without exposing or requesting the key in conversation.
- Delivery includes a strict MkDocs build executed through repository-managed, reproducible
documentation dependencies rather than globally installed Python packages. A readable direct
dependency file is retained, a complete transitive lock is generated with `uv`, and one canonical
repository command performs the strict build from that lock.
- Successful real-provider acceptance is recorded in a short sanitized report under
`docs/testing/`. It identifies the model and checks performed but contains no credentials,
prompts, source samples, complete provider payloads, or generated database values.
- No tests are added for worker queues, parallel generation, distributed locking, multi-replica
recovery, automatic resume, cost accounting, or model fallback because those behaviors are out
of scope.
## Out of Scope
- Reusing Pi to execute Description Generation or changing Pi's model configuration.
- A shared Installation Model Registry, model gateway, long-lived Python sidecar, or provider
management platform.
- A user-facing generation CLI.
- Parallel model calls, worker queues, adaptive rate limiting, distributed locks, leases,
heartbeats, automatic resume, or multi-replica execution.
- Durable target jobs, invocation history, prompts, samples, token usage, cost accounting,
provenance chains, target snapshots, or advanced retention controls.
- Automatic retry or resume of individual targets beyond one technical helper retry and a new
Generate Missing run.
- Automatic fallback to a different model.
- Writing generated text into comments of the external Workspace Database.
- Generating logical relationships or other catalog metadata beyond Catalog Table and Catalog
Column descriptions.
- Implementing the Sensitive Data Policy. Its definition and exclusion/anonymization behavior are
a required follow-up improvement.
- Generalizing the log viewer across unrelated metadata operations. That broader concern remains
related to Gitea issue #2.
## Further Notes
- The design deliberately follows ThothAI's proven simple workflow while adapting it to ThothII's
asynchronous browser interaction, setup ownership, and existing Generated Description model.
- The source-sampling disclosure is a release requirement, not merely documentation for operators.
- `completed_with_errors` is reserved for a run that reaches the end after isolated technical
failures. Three consecutive failures end the run as `failed`.
- Successful values are their own recovery record: after interruption, Generate Missing naturally
skips them without needing replay state.
- The implementation is available without a feature flag once the catalog migration and valid
setup are present. With no configured model, the feature remains visibly unavailable rather than
partially initialized.
- The final manual gate is intentionally narrow: one real-provider run covers one Catalog Column
and one Catalog Table, generated Italian text, safe events, and one consolidation. Automated
tests remain the evidence for All, Missing, Stop, restart interruption, Unlock, and failure paths.
@@ -0,0 +1,173 @@
# AI catalog description generation
Status: simplified design, API, persistence, test seams, and delivery tickets accepted.
## Objective
Bring ThothAI's useful AI comment-generation workflow into the ThothII Metadata Catalog without
turning it into a general job platform. Administrators can generate editable descriptions for
catalog tables and columns, inspect progress, stop a run, recover a stale run, and explicitly copy
approved generated text into the curated Description field.
The implementation is UI/API only. There is no user-facing generation command.
## ThothAI behavior retained
- Generate descriptions for selected tables, selected columns, missing descriptions, or all
eligible targets.
- Generate columns before their containing table when running the full workflow, so table prompts
can benefit from the resulting column descriptions.
- Process bounded batches of at most ten targets, one model request at a time.
- Include schema context, existing catalog text, up to five real source rows, and up to five
representative non-null values when available.
- Keep generated text separate from the curated Description until an administrator consolidates
it.
- Use the existing table and column checkboxes plus the Actions selector to copy Generated
Description into Description for one or more selected records. The generated value is retained.
- Store a localized standard value such as `Non generabile` when a valid model response says that
a description cannot be inferred.
Unlike ThothAI, every generation action is asynchronous from the browser's perspective and exposes
a persistent, readable activity log.
## Minimal architecture
The Fastify backend owns the run lifecycle and sequential loop. It starts one short-lived Python
helper for each model completion. The helper uses LiteLLM, accepts structured input on stdin,
returns structured output on stdout, and writes diagnostics only to stderr.
This is preferred over reusing Pi. Pi remains the interactive NL-to-SQL orchestration surface,
whereas description generation is a bounded batch transformation with no conversational state or
human gate. A LiteLLM helper avoids inventing a Pi session protocol for a task that needs one
request and one structured response.
There is no Python daemon, model gateway, queue service, worker pool, or generation CLI. Python is
already a core implementation language in ThothII's harness and core image; this helper does not
introduce a new runtime family.
## Run lifecycle and exclusion
- At most one Description Generation Run may be queued or running in the installation.
- Start returns immediately after creating the run and scheduling the in-process backend loop.
- Requests are sequential; there is no parallel provider traffic.
- The target Workspace Database is reserved through the existing in-memory catalog-operation
coordinator. Synchronization, cleanup, consolidation, and direct catalog edits for that database
are rejected while generation is active.
- A second generation start is rejected with a conflict response.
- Stop terminates the current helper process, stops further targets, and marks the run cancelled.
- Backend startup marks any queued or running generation row interrupted. It does not resume work.
- Generate Missing is the normal manual continuation mechanism because successful values were
already saved.
- Unlock is available only when the backend has no live generation process; it marks a stale
recorded run interrupted and clears the local reservation.
This is intentionally a single-process policy. Multi-replica coordination is out of scope.
## Persistence
Persist only:
- a Description Generation Run with database, scope, selected model, language, status, counters,
timestamps, and an optional final error summary;
- ordered Description Generation Events containing timestamp, level, and human-readable text;
- each successful or non-generatable result directly in the target's Generated Description.
Do not add per-target job rows, invocation history, prompt or sample snapshots, provider cost
accounting, leases, heartbeats, registry revisions, or generated-description provenance. The event
log is operational evidence, not a replay mechanism.
## Model setup
Selectable models and their default belong to application setup YAML, not to a workspace. Each
entry supplies a stable display identifier, LiteLLM provider/model information, optional endpoint
settings, and—unless that explicit endpoint is unauthenticated—a reference to an installation
secret containing the API key. Keyless entries without an explicit endpoint are invalid. Raw keys must not be
stored in the YAML, database, frontend, events, or process arguments.
An explicit endpoint may opt into `disableThinking: true` when its Qwen-compatible chat template
would otherwise place reasoning text around the required JSON result.
This metadata-generation configuration is independent of the existing Pi provider/model settings
and workspace `llm_policy`. A setup change takes effect after application restart. If no model is
configured, generation controls are disabled with an explanatory message.
The browser receives only the selectable identifiers and labels. The selected value defaults to
the setup default and is validated again by the backend when a run starts.
## Prompt inputs and outputs
Targets are grouped in model requests of at most ten. Prompts distinguish instructions from
untrusted schema names, comments, descriptions, and sampled values. A response must map every
returned result to a requested target and classify it as generated or non-generatable. Missing,
duplicate, unknown, or malformed target results make that request a technical failure rather than
silently writing ambiguous text.
One complete `json` code fence around the object is tolerated for model compatibility; prose
outside it, multiple payloads, and ambiguous mappings are still rejected.
For a complete database run, eligible columns are processed before tables. A table request can use
the current Generated Description or Description of its columns. The output language is the
workspace language; the standard non-generatable text is localized by the application rather than
trusted to arbitrary model wording.
Up to five source rows and five representative examples may be sent to the provider and are never
persisted. Delivery must call out this disclosure. A follow-up Sensitive Data Policy will define
which values are excluded or anonymized.
## Errors, retry, and logs
The helper performs at most one retry for a transient technical provider failure. A final failed
request produces an error event and increments the consecutive-error count. The run stops as
failed after three consecutive technical failures; any successful request resets the count. There
is no automatic fallback to another model.
Valid non-generatable outcomes are results, not technical errors. Successful results from earlier
requests remain stored when a later request fails or the run is stopped.
The UI shows status, counters, selected model, start/end times, and a chronological text log. Live
delivery may reuse the existing SSE infrastructure with polling as fallback; exact visual parity
with synchronization logs is not required. Logs must not contain API keys, prompts, source sample
values, or full provider payloads.
## Explicitly deferred complexity
- shared model registry or cutover of Pi configuration;
- long-lived Python sidecar or internal HTTP model gateway;
- generic catalog-operation kernel;
- durable target items, invocation records, target snapshots, or provenance chains;
- distributed locks, leases, heartbeats, worker queues, automatic resume, or multi-replica support;
- parallel calls, adaptive rate limiting, cost estimation, advanced metrics, or model fallback;
- user-facing generation CLI;
- automatic writeback to comments in the external database;
- Sensitive Data Policy implementation, which remains a required improvement after this slice.
## Delivery tracking
The accepted specification is Gitea issue #4 and the implementation is split into issues #5–#11.
Each ticket is a bounded vertical slice with explicit Gitea dependencies. Implementation proceeds
from the unblocked frontier, using a fresh subagent context for each ticket; integration and final
verification remain centralized so later slices cannot silently reopen the deferred platform
features above.
Issue #4 remains open until delivery completes four final gates: the stale Compose service-set
contract is corrected in its own commit; documentation dependencies are repository-managed and a
strict MkDocs build passes; one narrow real-provider acceptance run succeeds against non-sensitive
test data; and a separate, non-blocking Sensitive Data Policy design ticket is linked as required
follow-up work.
The documentation toolchain retains a readable direct-dependency input, adds a complete lock
generated with `uv`, and exposes one canonical strict-build command. Real-provider acceptance uses
the installation's configured default model and a disposable PostgreSQL database seeded only with
invented values and accessed read-only by the application. A missing protected model secret stops
the gate without disclosing it. The successful gate is captured in a sanitized report under
`docs/testing/` without prompts, samples, full generated values, payloads, or credentials.
Delivery is organized as four reviewable commits: the stale Compose contract correction, the
reproducible documentation toolchain, the AI-description feature, and—only after acceptance—the
sanitized acceptance report. A failed real-provider gate does not invalidate already verified
commits, but issue #4 remains open and no acceptance report claims success. Application defects are
fixed and reverified; missing configuration or provider unavailability is recorded and retried.
After every gate passes, the existing `codex/db-management` branch is pushed to its configured
origin without introducing a new pull-request workflow, then issue #4 is closed with links to the
delivery evidence. The separate Sensitive Data Policy issue is created as non-blocking follow-up,
linked to #4, and labeled `enhancement` plus `ready-for-human` because its design requires a future
`grill-with-docs` before agent implementation.
+121
View File
@@ -132,6 +132,13 @@ export interface CatalogMetadataDeleteCounts {
relationships: number;
}
export type CatalogDescriptionTarget = "tables" | "columns";
export interface CatalogDescriptionConsolidationCounts {
copied: number;
skipped: number;
}
export type CatalogSyncScope = "tables" | "columns" | "relationships" | "all";
export type CatalogSyncState = "queued" | "running" | "awaiting_confirmation" | "applying"
| "succeeded" | "failed" | "cancelled" | "interrupted";
@@ -176,8 +183,113 @@ export interface CatalogSyncEvent {
createdAt: string;
}
export interface MetadataGenerationModel {
id: string;
label: string;
}
export interface MetadataGenerationModels {
models: MetadataGenerationModel[];
default: string | null;
}
export type DescriptionGenerationScope = "selected_columns" | "selected_tables" | "all" | "missing";
export type DescriptionGenerationStatus = "queued" | "running" | "completed"
| "completed_with_errors" | "cancelled" | "failed" | "interrupted";
export interface DescriptionGenerationRun {
id: string;
databaseId: string;
scope: DescriptionGenerationScope;
modelId: string;
language: "en" | "it";
status: DescriptionGenerationStatus;
total: number;
processed: number;
generated: number;
nonGeneratable: number;
failed: number;
createdAt: string;
startedAt: string | null;
updatedAt: string;
finishedAt: string | null;
errorSummary: string | null;
}
export interface DescriptionGenerationEvent {
sequence: number;
level: "info" | "warning" | "error";
message: string;
createdAt: string;
}
export const listCatalogDatabases = () => apiFetch<CatalogDatabase[]>("/catalog/databases");
export const listMetadataGenerationModels = () =>
apiFetch<MetadataGenerationModels>("/catalog/metadata-generation/models");
export function startDescriptionGenerationRun(
databaseId: string,
modelId: string,
scope: Extract<DescriptionGenerationScope, "selected_columns" | "selected_tables">,
targetIds: string[],
): Promise<DescriptionGenerationRun>;
export function startDescriptionGenerationRun(
databaseId: string,
modelId: string,
scope: Extract<DescriptionGenerationScope, "all" | "missing">,
): Promise<DescriptionGenerationRun>;
export function startDescriptionGenerationRun(
databaseId: string,
modelId: string,
scope: DescriptionGenerationScope,
targetIds?: string[],
): Promise<DescriptionGenerationRun> {
const body = scope === "all" || scope === "missing"
? { modelId, scope }
: { modelId, scope, targetIds: targetIds ?? [] };
return apiFetch<DescriptionGenerationRun>(
`/catalog/databases/${encodeURIComponent(databaseId)}/description-generation-runs`,
{ method: "POST", body: JSON.stringify(body) },
);
}
export const getDescriptionGenerationRun = (runId: string) =>
apiFetch<DescriptionGenerationRun>(
`/catalog/description-generation-runs/${encodeURIComponent(runId)}`,
);
export const listDescriptionGenerationRuns = (limit = 50) =>
apiFetch<DescriptionGenerationRun[]>(
`/catalog/description-generation-runs?limit=${encodeURIComponent(String(limit))}`,
);
export const listDescriptionGenerationEvents = (runId: string, after = 0) =>
apiFetch<DescriptionGenerationEvent[]>(
`/catalog/description-generation-runs/${encodeURIComponent(runId)}/events-list?after=${after}`,
);
export const cancelDescriptionGenerationRun = (runId: string) =>
apiFetch<DescriptionGenerationRun>(
`/catalog/description-generation-runs/${encodeURIComponent(runId)}/cancel`,
{ method: "POST" },
);
export const unlockDescriptionGenerationRun = () =>
apiFetch<DescriptionGenerationRun>(
"/catalog/description-generation-runs/unlock",
{ method: "POST" },
);
export function descriptionGenerationEventsUrl(runId: string, after = 0): string {
const url = joinBackendPath(
BASE,
`/catalog/description-generation-runs/${encodeURIComponent(runId)}/events?after=${after}`,
);
assertSameOriginRequestUrl(url);
return url;
}
export const createCatalogDatabase = (input: DatabaseConfiguration) =>
apiFetch<CatalogDatabase>("/catalog/databases", { method: "POST", body: JSON.stringify(input) });
@@ -254,6 +366,15 @@ export const deleteCatalogTableMetadata = (
{ method: "POST", body: JSON.stringify({ tableIds, target }) },
);
export const consolidateCatalogDescriptions = (
databaseId: string,
target: CatalogDescriptionTarget,
targetIds: string[],
) => apiFetch<CatalogDescriptionConsolidationCounts>(
`/catalog/databases/${encodeURIComponent(databaseId)}/descriptions/consolidate`,
{ method: "POST", body: JSON.stringify({ target, targetIds }) },
);
export const startCatalogSync = (
databaseId: string,
version: number,
@@ -0,0 +1,76 @@
import { http, HttpResponse } from "msw";
import { server } from "../test/msw";
import {
cancelDescriptionGenerationRun,
descriptionGenerationEventsUrl,
listDescriptionGenerationRuns,
unlockDescriptionGenerationRun,
type DescriptionGenerationRun,
} from "./catalog-databases";
const historicalRun: DescriptionGenerationRun = {
id: "77777777-7777-4777-8777-777777777777",
databaseId: "11111111-1111-4111-8111-111111111111",
scope: "missing",
modelId: "local-qwen",
language: "it",
status: "interrupted",
total: 4,
processed: 2,
generated: 2,
nonGeneratable: 0,
failed: 0,
createdAt: "2026-08-28T08:00:00Z",
startedAt: "2026-08-28T08:00:01Z",
updatedAt: "2026-08-28T08:02:00Z",
finishedAt: "2026-08-28T08:02:00Z",
errorSummary: null,
};
test("lists newest-first persisted description-generation history with a bounded limit", async () => {
let requestedLimit: string | null = null;
server.use(http.get("/api/catalog/description-generation-runs", ({ request }) => {
requestedLimit = new URL(request.url).searchParams.get("limit");
return HttpResponse.json([historicalRun]);
}));
await expect(listDescriptionGenerationRuns(50)).resolves.toEqual([historicalRun]);
expect(requestedLimit).toBe("50");
});
test("stops a description-generation run with the run-scoped body-less endpoint", async () => {
let contentType: string | null = "unset";
server.use(http.post(
"/api/catalog/description-generation-runs/:runId/cancel",
({ request }) => {
contentType = request.headers.get("content-type");
return HttpResponse.json({ ...historicalRun, status: "cancelled" });
},
));
await expect(cancelDescriptionGenerationRun(historicalRun.id)).resolves.toMatchObject({
id: historicalRun.id,
status: "cancelled",
});
expect(contentType).toBeNull();
});
test("unlocks the installation-wide stale description-generation run without a request body", async () => {
let contentType: string | null = "unset";
server.use(http.post(
"/api/catalog/description-generation-runs/unlock",
({ request }) => {
contentType = request.headers.get("content-type");
return HttpResponse.json(historicalRun);
},
));
await expect(unlockDescriptionGenerationRun()).resolves.toEqual(historicalRun);
expect(contentType).toBeNull();
});
test("builds the same-origin description-generation SSE replay URL from an event sequence", () => {
expect(descriptionGenerationEventsUrl(historicalRun.id, 17)).toBe(
`/api/catalog/description-generation-runs/${historicalRun.id}/events?after=17`,
);
});
+21
View File
@@ -143,6 +143,27 @@ test("keeps only a known safe bounded JSON error payload", async () => {
}
});
test.each([
["description_generation_target_ids_duplicate", "Description generation target IDs must be unique."],
["description_generation_no_eligible_targets", "No eligible catalog tables or columns need description generation."],
["catalog_table_not_found", "One or more selected catalog tables were not found."],
])("maps the description-generation error code %s to safe local copy", async (code, message) => {
const fetchSpy = vi.spyOn(globalThis, "fetch").mockResolvedValue(
new Response(JSON.stringify({ code, message: "provider detail must not be trusted" }), {
status: 400,
headers: { "content-type": "application/json" },
}),
);
try {
const failure = await apiFetch<unknown>("/description-generation-error")
.catch((error: unknown) => error) as ApiError;
expect(failure).toMatchObject({ code, message, payload: { code } });
} finally {
fetchSpy.mockRestore();
}
});
test("derives local messages without retaining a malicious known-code message", async () => {
const fetchSpy = vi.spyOn(globalThis, "fetch").mockResolvedValue(
new Response(JSON.stringify({
+22 -1
View File
@@ -15,7 +15,14 @@ const safeErrorCodes = new Set([
"semantic_index_incompatible", "pi_management_forbidden", "pi_management_unavailable",
"pi_management_invalid_config", "pi_management_write_failed",
"catalog_unavailable", "database_conflict", "database_invalid", "database_not_found",
"database_stale", "database_operation_failed",
"database_stale", "database_operation_failed", "database_operation_in_progress",
"catalog_target_not_found", "description_consolidation_invalid", "description_consolidation_failed",
"description_generation_request_invalid", "description_generation_run_active",
"description_generation_failed", "description_generation_run_not_found",
"description_generation_no_eligible_targets",
"description_generation_target_ids_duplicate", "metadata_generation_model_unavailable",
"catalog_column_not_found", "catalog_table_not_found",
"workspace_configuration_unavailable",
"schema_sync_conflict", "schema_introspection_failed", "schema_request_invalid",
"schema_operation_failed", "sync_run_not_found", "table_stale", "column_stale",
]);
@@ -56,6 +63,20 @@ const localCodeMessages: Record<string, string> = {
database_not_found: "The database configuration was not found.",
database_stale: "The database configuration changed. Reload and try again.",
database_operation_failed: "The database operation failed.",
database_operation_in_progress: "A database operation is already in progress.",
catalog_target_not_found: "One or more selected catalog targets were not found.",
description_consolidation_invalid: "The description consolidation request is invalid.",
description_consolidation_failed: "Description consolidation failed.",
description_generation_request_invalid: "The description generation request is invalid.",
description_generation_run_active: "A description generation run is already active.",
description_generation_failed: "Description generation failed.",
description_generation_run_not_found: "The description generation run was not found.",
description_generation_no_eligible_targets: "No eligible catalog tables or columns need description generation.",
description_generation_target_ids_duplicate: "Description generation target IDs must be unique.",
metadata_generation_model_unavailable: "The selected description model is unavailable.",
catalog_column_not_found: "The selected catalog column was not found.",
catalog_table_not_found: "One or more selected catalog tables were not found.",
workspace_configuration_unavailable: "The database workspace configuration is unavailable.",
schema_sync_conflict: "A schema synchronization is already active or no longer current.",
schema_introspection_failed: "The database schema could not be read safely.",
schema_request_invalid: "The schema request is invalid.",
File diff suppressed because it is too large Load Diff
+160 -16
View File
@@ -6,7 +6,7 @@ import {
useState,
} from "react";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { Plus, RefreshCw } from "lucide-react";
import { History, Plus, RefreshCw } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../components/ui/button";
import { ApiError, apiErrorMessage } from "../api/client";
@@ -15,8 +15,11 @@ import {
deleteCatalogDatabase,
deleteCatalogDatabaseMetadata,
listCatalogDatabases,
listDescriptionGenerationRuns,
listMetadataGenerationModels,
replaceCatalogDatabaseSecrets,
startCatalogSync,
startDescriptionGenerationRun,
testCatalogDatabase,
updateCatalogDatabase,
type CatalogDatabase,
@@ -26,12 +29,15 @@ import {
type CatalogSyncRun,
type DatabaseBinding,
type DatabaseTransport,
type DescriptionGenerationRun,
} from "../api/catalog-databases";
import { DatabaseGrid } from "./database-management/DatabaseGrid";
import { DatabaseForm } from "./database-management/DatabaseForm";
import { DatabaseTables } from "./database-management/DatabaseTables";
import { DatabaseRelationships } from "./database-management/DatabaseRelationships";
import { CatalogSyncDrawer } from "./database-management/CatalogSyncDrawer";
import { MetadataGenerationModelSelector } from "./database-management/MetadataGenerationModelSelector";
import { DescriptionGenerationDrawer } from "./database-management/DescriptionGenerationDrawer";
import {
configurationFingerprint,
configurationFromDraft,
@@ -46,6 +52,8 @@ import {
} from "./database-management/model";
const DATABASE_QUERY_KEY = ["catalog-databases"] as const;
const DESCRIPTION_GENERATION_HISTORY_QUERY_KEY = ["description-generation-runs", 50] as const;
const DESCRIPTION_GENERATION_ACTIVE_STATUSES = ["queued", "running"] as const;
const SYNC_STARTED_MESSAGES: Record<CatalogSyncScope, string> = {
tables: "Table synchronization started",
columns: "Column synchronization started",
@@ -68,6 +76,12 @@ function isStaleError(error: unknown): boolean {
return error instanceof ApiError && error.code === "database_stale";
}
function isDescriptionGenerationActive(run?: DescriptionGenerationRun | null): boolean {
return Boolean(run && DESCRIPTION_GENERATION_ACTIVE_STATUSES.some(
(status) => status === run.status,
));
}
export function DatabaseManagementPage({
canManage,
canManageSecrets,
@@ -86,8 +100,31 @@ export function DatabaseManagementPage({
retry: false,
});
const rows = data ?? [];
const metadataModelsQuery = useQuery({
queryKey: ["metadata-generation-models"],
queryFn: listMetadataGenerationModels,
retry: false,
});
const metadataModels = metadataModelsQuery.data?.models ?? [];
const configuredMetadataDefault = metadataModelsQuery.data?.default ?? "";
const defaultMetadataModel = configuredMetadataDefault
&& metadataModels.some((model) => model.id === configuredMetadataDefault)
? configuredMetadataDefault
: "";
const descriptionGenerationRunsQuery = useQuery({
queryKey: DESCRIPTION_GENERATION_HISTORY_QUERY_KEY,
queryFn: () => listDescriptionGenerationRuns(50),
enabled: canManage,
retry: false,
refetchInterval: 5_000,
});
const descriptionGenerationRuns = descriptionGenerationRunsQuery.data ?? [];
const observedActiveDescriptionGenerationRun = descriptionGenerationRuns.find(
(run) => isDescriptionGenerationActive(run),
);
const [screen, setScreen] = useState<DatabaseScreen>({ kind: "list" });
const [selectedMetadataModel, setSelectedMetadataModel] = useState("");
const [draft, setDraft] = useState<DatabaseFormDraft | null>(null);
const [baseline, setBaseline] = useState("");
const [formSource, setFormSource] = useState<FormSource | null>(null);
@@ -102,11 +139,17 @@ export function DatabaseManagementPage({
});
const [activeSyncRun, setActiveSyncRun] = useState<CatalogSyncRun | null>(null);
const [syncDrawerOpen, setSyncDrawerOpen] = useState(false);
const [activeDescriptionGenerationRun, setActiveDescriptionGenerationRun] = useState<DescriptionGenerationRun | null>(null);
const [descriptionGenerationDrawerOpen, setDescriptionGenerationDrawerOpen] = useState(false);
const originRef = useRef<HTMLElement | null>(null);
const searchInputRef = useRef<HTMLInputElement>(null);
const formHeadingRef = useRef<HTMLHeadingElement>(null);
useEffect(() => {
setSelectedMetadataModel(defaultMetadataModel);
}, [defaultMetadataModel]);
const activeRow = screen.kind === "list"
? undefined
: rows.find((row) => row.workspaceId === screen.workspaceId);
@@ -556,6 +599,36 @@ export function DatabaseManagementPage({
setSyncDrawerOpen(true);
}, [updateTrackedSyncRun]);
const rememberDescriptionGenerationRun = useCallback((run: DescriptionGenerationRun) => {
setActiveDescriptionGenerationRun(run);
queryClient.setQueryData<DescriptionGenerationRun[]>(
DESCRIPTION_GENERATION_HISTORY_QUERY_KEY,
(current = []) => [run, ...current.filter((item) => item.id !== run.id)],
);
setDescriptionGenerationDrawerOpen(true);
}, [queryClient]);
const openDescriptionGenerationHistory = useCallback(() => {
const run = observedActiveDescriptionGenerationRun
?? descriptionGenerationRuns[0]
?? activeDescriptionGenerationRun;
if (!run) return;
setActiveDescriptionGenerationRun(run);
setDescriptionGenerationDrawerOpen(true);
}, [activeDescriptionGenerationRun, descriptionGenerationRuns, observedActiveDescriptionGenerationRun]);
const descriptionGenerationTerminated = useCallback(async (run: DescriptionGenerationRun) => {
await Promise.all([
queryClient.invalidateQueries({
queryKey: ["catalog-tables", run.databaseId],
exact: true,
}),
queryClient.invalidateQueries({
queryKey: ["catalog-columns", run.databaseId],
}),
]);
}, [queryClient]);
const openSync = useCallback((row?: CatalogDatabase) => {
const run = row?.activeSyncRun ?? activeSyncRun;
if (!run) return;
@@ -590,6 +663,26 @@ export function DatabaseManagementPage({
}
}, [queryClient, rememberSyncRun]);
const generateDatabaseDescriptions = useCallback(async (
selected: CatalogDatabase[],
scope: "all" | "missing",
) => {
const database = selected.length === 1 ? selected[0] : undefined;
if (!database?.id || !selectedMetadataModel) return;
try {
const run = await startDescriptionGenerationRun(
database.id,
selectedMetadataModel,
scope,
);
rememberDescriptionGenerationRun(run);
toast.success(`${scope === "all" ? "Generate All" : "Generate Missing"} started for ${database.workspaceName}`);
} catch (error) {
toast.error(apiErrorMessage(error));
throw error;
}
}, [rememberDescriptionGenerationRun, selectedMetadataModel]);
const deleteSelectedMetadata = useCallback(async (
selected: CatalogDatabase[],
target: CatalogDatabaseMetadataDeleteTarget,
@@ -640,6 +733,16 @@ export function DatabaseManagementPage({
: activeRow?.activeSyncRun && ["queued", "running", "awaiting_confirmation", "applying"].includes(activeRow.activeSyncRun.state)
? activeRow.activeSyncRun
: undefined;
const selectedMetadataModelAvailable = Boolean(
selectedMetadataModel
&& metadataModels.some((model) => model.id === selectedMetadataModel),
);
const descriptionGenerationActive = isDescriptionGenerationActive(
observedActiveDescriptionGenerationRun ?? activeDescriptionGenerationRun,
);
const descriptionGenerationHistoryAvailable = Boolean(
descriptionGenerationRuns.length > 0 || activeDescriptionGenerationRun,
);
return (
<main aria-label="Database management" className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
@@ -651,21 +754,46 @@ export function DatabaseManagementPage({
One database configuration for each repository workspace.
</p>
</div>
{screen.kind === "list" ? (
<div className="flex items-center gap-2">
<Button type="button" variant="outline" disabled={isFetching} onClick={() => void refreshList()}>
<RefreshCw className={isFetching ? "animate-spin" : ""} /> Refresh
</Button>
<Button
type="button"
disabled={!canManage || availableWorkspaces.length === 0}
title={availableWorkspaces.length === 0 ? "Every available workspace is already configured" : undefined}
onClick={(event) => addDatabase(event.currentTarget)}
>
<Plus /> Add database
</Button>
</div>
) : null}
<div className="flex flex-wrap items-end justify-end gap-3">
<MetadataGenerationModelSelector
canManage={canManage}
data={metadataModelsQuery.data}
isError={metadataModelsQuery.isError}
isLoading={metadataModelsQuery.isLoading}
selectedModel={selectedMetadataModel}
onSelectedModelChange={setSelectedMetadataModel}
/>
<Button
type="button"
variant="outline"
aria-label={observedActiveDescriptionGenerationRun
? "Observe active description generation"
: "View description generation history"}
disabled={!canManage || !descriptionGenerationHistoryAvailable}
title={!descriptionGenerationHistoryAvailable && !descriptionGenerationRunsQuery.isLoading
? "No description generation runs yet"
: undefined}
onClick={openDescriptionGenerationHistory}
>
<History />
{observedActiveDescriptionGenerationRun ? "Active run" : "Run history"}
</Button>
{screen.kind === "list" ? (
<div className="flex items-center gap-2">
<Button type="button" variant="outline" disabled={isFetching} onClick={() => void refreshList()}>
<RefreshCw className={isFetching ? "animate-spin" : ""} /> Refresh
</Button>
<Button
type="button"
disabled={!canManage || availableWorkspaces.length === 0}
title={availableWorkspaces.length === 0 ? "Every available workspace is already configured" : undefined}
onClick={(event) => addDatabase(event.currentTarget)}
>
<Plus /> Add database
</Button>
</div>
) : null}
</div>
</header>
<div className="relative flex min-h-0 flex-1">
@@ -701,6 +829,9 @@ export function DatabaseManagementPage({
onOpenSync={openSync}
onTestSelected={testSelected}
onSyncSelected={syncSelected}
selectedMetadataModel={selectedMetadataModelAvailable ? selectedMetadataModel : null}
descriptionGenerationActive={descriptionGenerationActive}
onGenerateDescriptions={generateDatabaseDescriptions}
onDeleteMetadataSelected={deleteSelectedMetadata}
/>
)}
@@ -746,12 +877,15 @@ export function DatabaseManagementPage({
database={activeRow}
canManage={canManage}
activeRun={currentActiveRun}
selectedMetadataModel={selectedMetadataModelAvailable ? selectedMetadataModel : null}
descriptionGenerationActive={descriptionGenerationActive}
onBackToDatabases={showList}
onOpenOverview={() => openOverview(activeRow)}
onOpenRelationships={() => openRelationships(activeRow)}
onNavigationStateChange={setTablesNavigationState}
onRunStarted={rememberSyncRun}
onOpenSync={() => openSync(activeRow)}
onDescriptionGenerationRunStarted={rememberDescriptionGenerationRun}
/>
) : null}
@@ -776,6 +910,16 @@ export function DatabaseManagementPage({
onRunUpdate={updateTrackedSyncRun}
onCatalogChanged={() => void catalogChanged()}
/>
<DescriptionGenerationDrawer
open={descriptionGenerationDrawerOpen}
run={activeDescriptionGenerationRun}
modelLabel={metadataModels.find(
(model) => model.id === activeDescriptionGenerationRun?.modelId,
)?.label ?? activeDescriptionGenerationRun?.modelId ?? ""}
onClose={() => setDescriptionGenerationDrawerOpen(false)}
onRunUpdate={setActiveDescriptionGenerationRun}
onTerminal={(run) => void descriptionGenerationTerminated(run)}
/>
</main>
);
}
@@ -1,16 +1,20 @@
import { useEffect, useMemo, useRef, useState } from "react";
import { Menu } from "@base-ui/react/menu";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { AgGridReact } from "ag-grid-react";
import type { ColDef, ICellRendererParams } from "ag-grid-community";
import { KeyRound, Link2, Pencil, RefreshCw, Save } from "lucide-react";
import { ChevronDown, KeyRound, Link2, Pencil, RefreshCw, Save, X } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../../components/ui/button";
import { ApiError, apiErrorMessage } from "../../api/client";
import {
consolidateCatalogDescriptions,
listCatalogColumns,
startDescriptionGenerationRun,
updateCatalogColumnMetadata,
type CatalogColumn,
type CatalogTable,
type DescriptionGenerationRun,
} from "../../api/catalog-databases";
import type { DatabaseNavigationState } from "./model";
@@ -18,6 +22,9 @@ interface Props {
databaseId: string;
table: CatalogTable;
canManage: boolean;
selectedMetadataModel: string | null;
descriptionGenerationActive: boolean;
onDescriptionGenerationRunStarted: (run: DescriptionGenerationRun) => void;
onNavigationStateChange: (state: DatabaseNavigationState) => void;
onSync: () => void;
}
@@ -48,7 +55,16 @@ function ActionCell({ data, context }: ICellRendererParams<CatalogColumn, unknow
);
}
export function DatabaseColumns({ databaseId, table, canManage, onNavigationStateChange, onSync }: Props) {
export function DatabaseColumns({
databaseId,
table,
canManage,
selectedMetadataModel,
descriptionGenerationActive,
onDescriptionGenerationRunStarted,
onNavigationStateChange,
onSync,
}: Props) {
const queryClient = useQueryClient();
const queryKey = ["catalog-columns", databaseId, table.id] as const;
const { data = [], isLoading, isFetching, refetch } = useQuery({
@@ -57,6 +73,7 @@ export function DatabaseColumns({ databaseId, table, canManage, onNavigationStat
retry: false,
});
const [search, setSearch] = useState("");
const [selectedIds, setSelectedIds] = useState<string[]>([]);
const [editingId, setEditingId] = useState<string | null>(null);
const [description, setDescription] = useState("");
const [generatedDescription, setGeneratedDescription] = useState("");
@@ -65,6 +82,7 @@ export function DatabaseColumns({ databaseId, table, canManage, onNavigationStat
const [stale, setStale] = useState(false);
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
const [busy, setBusy] = useState(false);
const gridRef = useRef<AgGridReact<CatalogColumn>>(null);
const originRef = useRef<HTMLButtonElement | null>(null);
const active = editingId ? data.find((column) => column.id === editingId) : undefined;
const fingerprint = JSON.stringify([description, generatedDescription]);
@@ -131,6 +149,34 @@ export function DatabaseColumns({ databaseId, table, canManage, onNavigationStat
setStaleBannerOpen(true);
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(false); }
};
const consolidateDescriptions = async () => {
setBusy(true);
try {
const counts = await consolidateCatalogDescriptions(databaseId, "columns", selectedIds);
await queryClient.invalidateQueries({ queryKey, exact: true });
gridRef.current?.api.deselectAll();
setSelectedIds([]);
toast.success(`Copied ${counts.copied} description${counts.copied === 1 ? "" : "s"}; skipped ${counts.skipped}`);
} catch (error) {
toast.error(apiErrorMessage(error));
} finally { setBusy(false); }
};
const generateDescriptions = async () => {
if (selectedIds.length === 0 || !selectedMetadataModel) return;
setBusy(true);
try {
const run = await startDescriptionGenerationRun(
databaseId,
selectedMetadataModel,
"selected_columns",
selectedIds,
);
onDescriptionGenerationRunStarted(run);
toast.success(`Description generation started for ${selectedIds.length} column${selectedIds.length === 1 ? "" : "s"}`);
} catch (error) {
toast.error(apiErrorMessage(error));
} finally { setBusy(false); }
};
const columns = useMemo<ColDef<CatalogColumn>[]>(() => [
{ field: "ordinalPosition", headerName: "#", width: 64, maxWidth: 64, filter: "agNumberColumnFilter" },
@@ -176,15 +222,62 @@ export function DatabaseColumns({ databaseId, table, canManage, onNavigationStat
return (
<div className="flex min-h-0 flex-1 flex-col">
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
<span className="thot-label whitespace-nowrap">Catalog columns</span>
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search columns" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
<Button type="button" disabled={!canManage || busy} onClick={onSync}><RefreshCw />Sync columns</Button>
{selectedIds.length > 0 ? (
<>
<span className="text-sm font-semibold">{selectedIds.length} selected</span>
<Menu.Root>
<Menu.Trigger className="inline-flex h-8 items-center justify-center gap-2 rounded-md border border-input bg-background px-3 text-sm font-medium hover:bg-muted disabled:pointer-events-none disabled:opacity-50" disabled={busy}>Actions <ChevronDown className="size-4" /></Menu.Trigger>
<Menu.Portal>
<Menu.Positioner side="bottom" align="start" sideOffset={4}>
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
<Menu.Item
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
disabled={!canManage || !selectedMetadataModel || descriptionGenerationActive || busy}
onClick={() => void generateDescriptions()}
>
Generate {selectedIds.length === 1 ? "description" : "descriptions"}
</Menu.Item>
<Menu.Item
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
disabled={!canManage || busy}
onClick={() => void consolidateDescriptions()}
>
Move generated description to Description
</Menu.Item>
</Menu.Popup>
</Menu.Positioner>
</Menu.Portal>
</Menu.Root>
<Button type="button" variant="ghost" onClick={() => { gridRef.current?.api.deselectAll(); setSelectedIds([]); }}><X />Clear</Button>
</>
) : (
<>
<span className="thot-label whitespace-nowrap">Catalog columns</span>
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search columns" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
<Button type="button" disabled={!canManage || busy} onClick={onSync}><RefreshCw />Sync columns</Button>
</>
)}
</div>
<div className="relative min-h-[280px] flex-1">
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
<AgGridReact<CatalogColumn> rowData={data} columnDefs={columns} context={context} loading={isLoading} quickFilterText={search} defaultColDef={{ sortable: true, filter: true, resizable: true }} getRowId={({ data: row }) => row.id} rowHeight={44} headerHeight={38} animateRows={false} overlayNoRowsTemplate="No columns synchronized for this table." />
<AgGridReact<CatalogColumn>
ref={gridRef}
rowData={data}
columnDefs={columns}
context={context}
loading={isLoading}
quickFilterText={search}
defaultColDef={{ sortable: true, filter: true, resizable: true }}
getRowId={({ data: row }) => row.id}
rowSelection={{ mode: "multiRow", selectAll: "filtered", enableClickSelection: false }}
onSelectionChanged={({ api }) => setSelectedIds(api.getSelectedRows().map((column) => column.id))}
rowHeight={44}
headerHeight={38}
animateRows={false}
overlayNoRowsTemplate="No columns synchronized for this table."
/>
</div>
</div>
</div>
@@ -15,6 +15,7 @@ import type {
CatalogDatabase,
CatalogDatabaseMetadataDeleteTarget,
CatalogSyncScope,
DescriptionGenerationScope,
} from "../../api/catalog-databases";
import { statusLabel } from "./model";
import { databaseSyncItemClass, databaseSyncScopes } from "./DatabaseSyncMenu";
@@ -35,6 +36,12 @@ interface DatabaseGridProps {
onOpenSync: (row: CatalogDatabase) => void;
onTestSelected: (rows: CatalogDatabase[]) => Promise<void>;
onSyncSelected: (rows: CatalogDatabase[], scope: CatalogSyncScope) => Promise<void>;
selectedMetadataModel: string | null;
descriptionGenerationActive: boolean;
onGenerateDescriptions: (
rows: CatalogDatabase[],
scope: Extract<DescriptionGenerationScope, "all" | "missing">,
) => Promise<void>;
onDeleteMetadataSelected: (
rows: CatalogDatabase[],
target: CatalogDatabaseMetadataDeleteTarget,
@@ -154,14 +161,18 @@ export function DatabaseGrid({
onOpenSync,
onTestSelected,
onSyncSelected,
selectedMetadataModel,
descriptionGenerationActive,
onGenerateDescriptions,
onDeleteMetadataSelected,
}: DatabaseGridProps) {
const compact = useCompactViewport();
const gridRef = useRef<AgGridReact<CatalogDatabase>>(null);
const actionsTriggerRef = useRef<HTMLButtonElement>(null);
const [selectedRows, setSelectedRows] = useState<CatalogDatabase[]>([]);
const [action, setAction] = useState<"test" | "sync" | "delete" | null>(null);
const [action, setAction] = useState<"test" | "sync" | "generate" | "delete" | null>(null);
const [pendingDelete, setPendingDelete] = useState<CatalogDatabaseMetadataDeleteTarget | null>(null);
const [pendingGenerateAll, setPendingGenerateAll] = useState(false);
const context = useMemo<DatabaseGridContext>(
() => ({ canManage, onView, onEdit, onDelete, onOpenSync }),
[canManage, onView, onEdit, onDelete, onOpenSync],
@@ -247,6 +258,11 @@ export function DatabaseGrid({
}).length;
const canTestSelection = canManage && selectedRows.length > 0 && selectedRows.every((row) => row.configured && row.id && !row.activeSyncRun);
const canSyncSelection = canManage && selectedRows.length > 0 && selectedRows.every((row) => row.configured && row.id && row.connectionStatus === "reachable" && row.testedVersion === row.version && !row.activeSyncRun);
const canGenerateDescriptions = canManage
&& selectedRows.length === 1
&& Boolean(selectedMetadataModel)
&& !descriptionGenerationActive
&& selectedRows.every((row) => row.configured && row.id && !row.activeSyncRun);
const canDeleteMetadataSelection = canManage && selectedRows.length > 0
&& selectedRows.every((row) => row.configured && row.id && !row.activeSyncRun);
useEffect(() => {
@@ -254,18 +270,30 @@ export function DatabaseGrid({
const timer = window.setTimeout(() => document.getElementById("database-cleanup-confirm-button")?.focus(), 0);
return () => window.clearTimeout(timer);
}, [pendingDelete]);
useEffect(() => {
if (!pendingGenerateAll) return;
const timer = window.setTimeout(() => document.getElementById("database-generate-all-confirm-button")?.focus(), 0);
return () => window.clearTimeout(timer);
}, [pendingGenerateAll]);
const closeDeleteConfirmation = () => {
setPendingDelete(null);
window.setTimeout(() => actionsTriggerRef.current?.focus(), 0);
};
const perform = async (kind: "test" | "sync" | "delete", operation: () => Promise<void>) => {
const closeGenerateAllConfirmation = () => {
setPendingGenerateAll(false);
window.setTimeout(() => actionsTriggerRef.current?.focus(), 0);
};
const perform = async (kind: "test" | "sync" | "generate" | "delete", operation: () => Promise<void>) => {
setAction(kind);
try {
await operation();
gridRef.current?.api.deselectAll();
setSelectedRows([]);
setPendingDelete(null);
setPendingGenerateAll(false);
if (kind === "delete") window.setTimeout(() => searchInputRef.current?.focus(), 0);
} catch {
// The page-level operation owns safe error feedback. Preserve the selection for retry.
} finally { setAction(null); }
};
@@ -273,7 +301,38 @@ export function DatabaseGrid({
<section aria-label="Workspace databases" className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
{selectedRows.length > 0 ? (
pendingDelete ? (
pendingGenerateAll ? (
<div
role="group"
aria-labelledby="database-generate-all-confirmation"
className="flex min-w-0 flex-1 flex-wrap items-center gap-2"
onKeyDown={(event) => {
if (event.key === "Escape" && action === null) closeGenerateAllConfirmation();
}}
>
<div className="mr-auto min-w-56">
<p id="database-generate-all-confirmation" className="text-sm font-semibold">
Replace generated descriptions for {selectedRows[0]?.workspaceName}?
</p>
<p className="text-xs text-muted-foreground">
Existing generated descriptions for eligible tables and columns will be replaced.
</p>
</div>
<Button type="button" variant="ghost" disabled={action !== null} onClick={closeGenerateAllConfirmation}>Cancel</Button>
<Button
id="database-generate-all-confirm-button"
type="button"
variant="destructive"
disabled={action !== null}
onClick={() => void perform(
"generate",
() => onGenerateDescriptions(selectedRows, "all"),
)}
>
Generate All
</Button>
</div>
) : pendingDelete ? (
<div
role="group"
aria-labelledby="database-cleanup-confirmation"
@@ -327,6 +386,24 @@ export function DatabaseGrid({
</Menu.Item>
))}
<Menu.Separator className="my-1 h-px bg-border" />
<Menu.Item
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
disabled={!canGenerateDescriptions}
onClick={() => setPendingGenerateAll(true)}
>
Generate All
</Menu.Item>
<Menu.Item
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
disabled={!canGenerateDescriptions}
onClick={() => void perform(
"generate",
() => onGenerateDescriptions(selectedRows, "missing"),
)}
>
Generate Missing
</Menu.Item>
<Menu.Separator className="my-1 h-px bg-border" />
<Menu.Item
className="rounded-md px-3 py-2 text-sm text-destructive outline-none data-[highlighted]:bg-destructive/10 data-[disabled]:opacity-45"
disabled={!canDeleteMetadataSelection}
@@ -383,6 +460,7 @@ export function DatabaseGrid({
selectionColumnDef={{ width: 44, maxWidth: 44, pinned: "left" }}
onSelectionChanged={({ api }) => {
if (pendingDelete && action === null) setPendingDelete(null);
if (pendingGenerateAll && action === null) setPendingGenerateAll(false);
setSelectedRows(api.getSelectedRows());
}}
rowHeight={44}
@@ -8,14 +8,17 @@ import { toast } from "sonner";
import { Button } from "../../components/ui/button";
import { ApiError, apiErrorMessage } from "../../api/client";
import {
consolidateCatalogDescriptions,
deleteCatalogTableMetadata,
listCatalogTables,
startCatalogSync,
startDescriptionGenerationRun,
updateCatalogTableMetadata,
type CatalogDatabase,
type CatalogSyncRun,
type CatalogTable,
type CatalogTableMetadataDeleteTarget,
type DescriptionGenerationRun,
} from "../../api/catalog-databases";
import type { DatabaseNavigationState } from "./model";
import { DatabaseColumns } from "./DatabaseColumns";
@@ -24,12 +27,15 @@ interface Props {
database: CatalogDatabase;
canManage: boolean;
activeRun?: CatalogSyncRun;
selectedMetadataModel: string | null;
descriptionGenerationActive: boolean;
onBackToDatabases: () => void;
onOpenOverview: () => void;
onOpenRelationships: () => void;
onNavigationStateChange: (state: DatabaseNavigationState) => void;
onRunStarted: (run: CatalogSyncRun) => void;
onOpenSync: () => void;
onDescriptionGenerationRunStarted: (run: DescriptionGenerationRun) => void;
}
interface TableGridContext {
@@ -55,12 +61,15 @@ export function DatabaseTables({
database,
canManage,
activeRun,
selectedMetadataModel,
descriptionGenerationActive,
onBackToDatabases,
onOpenOverview,
onOpenRelationships,
onNavigationStateChange,
onRunStarted,
onOpenSync,
onDescriptionGenerationRunStarted,
}: Props) {
const databaseId = database.id!;
const queryClient = useQueryClient();
@@ -77,7 +86,7 @@ export function DatabaseTables({
const [editorVersion, setEditorVersion] = useState<number | null>(null);
const [stale, setStale] = useState(false);
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
const [busy, setBusy] = useState<"sync" | "save" | "delete" | null>(null);
const [busy, setBusy] = useState<"sync" | "save" | "delete" | "consolidate" | "generate" | null>(null);
const [pendingDelete, setPendingDelete] = useState<CatalogTableMetadataDeleteTarget | null>(null);
const [columnNavigation, setColumnNavigation] = useState<DatabaseNavigationState>({ dirty: false, busy: false });
const gridRef = useRef<AgGridReact<CatalogTable>>(null);
@@ -203,6 +212,34 @@ export function DatabaseTables({
toast.error(apiErrorMessage(error));
} finally { setBusy(null); }
};
const consolidateDescriptions = async () => {
setBusy("consolidate");
try {
const counts = await consolidateCatalogDescriptions(databaseId, "tables", selectedIds);
await queryClient.invalidateQueries({ queryKey, exact: true });
gridRef.current?.api.deselectAll();
setSelectedIds([]);
toast.success(`Copied ${counts.copied} description${counts.copied === 1 ? "" : "s"}; skipped ${counts.skipped}`);
} catch (error) {
toast.error(apiErrorMessage(error));
} finally { setBusy(null); }
};
const generateDescriptions = async () => {
if (selectedIds.length === 0 || !selectedMetadataModel) return;
setBusy("generate");
try {
const run = await startDescriptionGenerationRun(
databaseId,
selectedMetadataModel,
"selected_tables",
selectedIds,
);
onDescriptionGenerationRunStarted(run);
toast.success(`Description generation started for ${selectedIds.length} table${selectedIds.length === 1 ? "" : "s"}`);
} catch (error) {
toast.error(apiErrorMessage(error));
} finally { setBusy(null); }
};
const columns = useMemo<ColDef<CatalogTable>[]>(() => [
{ field: "name", headerName: "Name", minWidth: 250, flex: 1, cellClass: "font-mono text-xs" },
@@ -257,7 +294,16 @@ export function DatabaseTables({
</nav>
</div>
{tableSection === "columns" ? (
<DatabaseColumns databaseId={databaseId} table={activeTable} canManage={canManage} onNavigationStateChange={setColumnNavigation} onSync={() => void synchronize("columns", [activeTable.id])} />
<DatabaseColumns
databaseId={databaseId}
table={activeTable}
canManage={canManage}
selectedMetadataModel={selectedMetadataModel}
descriptionGenerationActive={descriptionGenerationActive}
onDescriptionGenerationRunStarted={onDescriptionGenerationRunStarted}
onNavigationStateChange={setColumnNavigation}
onSync={() => void synchronize("columns", [activeTable.id])}
/>
) : (
<div className="min-h-0 overflow-y-auto px-4 py-5">
<p className="text-sm text-muted-foreground">Only review metadata can be changed.</p>
@@ -334,6 +380,13 @@ export function DatabaseTables({
<Menu.Portal>
<Menu.Positioner side="bottom" align="start" sideOffset={4}>
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
<Menu.Item
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
disabled={!canManage || !selectedMetadataModel || descriptionGenerationActive || busy !== null || Boolean(currentRun)}
onClick={() => void generateDescriptions()}
>
Generate {selectedIds.length === 1 ? "description" : "descriptions"}
</Menu.Item>
<Menu.Item
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)}
@@ -341,6 +394,13 @@ export function DatabaseTables({
>
Synchronize columns
</Menu.Item>
<Menu.Item
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
disabled={!canManage || busy !== null || Boolean(currentRun)}
onClick={() => void consolidateDescriptions()}
>
Move generated description to Description
</Menu.Item>
<Menu.Separator className="my-1 h-px bg-border" />
<Menu.Item
className="rounded-md px-3 py-2 text-sm text-destructive outline-none data-[highlighted]:bg-destructive/10 data-[disabled]:opacity-45"
@@ -0,0 +1,338 @@
import { act, render, screen, waitFor, within } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { http, HttpResponse } from "msw";
import { useState } from "react";
import type {
DescriptionGenerationEvent,
DescriptionGenerationRun,
} from "../../api/catalog-databases";
import { Toaster } from "../../components/ui/sonner";
import { FakeEventSource } from "../../test/fakeEventSource";
import { server } from "../../test/msw";
import { DescriptionGenerationDrawer } from "./DescriptionGenerationDrawer";
const runningRun: DescriptionGenerationRun = {
id: "77777777-7777-4777-8777-777777777777",
databaseId: "11111111-1111-4111-8111-111111111111",
scope: "missing",
modelId: "local-qwen",
language: "it",
status: "running",
total: 4,
processed: 1,
generated: 1,
nonGeneratable: 0,
failed: 0,
createdAt: "2026-08-28T08:00:00Z",
startedAt: "2026-08-28T08:00:01Z",
updatedAt: new Date().toISOString(),
finishedAt: null,
errorSummary: null,
};
function event(
sequence: number,
message: string,
level: DescriptionGenerationEvent["level"] = "info",
): DescriptionGenerationEvent {
return {
sequence,
level,
message,
createdAt: `2026-08-28T08:00:0${sequence}Z`,
};
}
function renderDrawer({
initialRun = runningRun,
onTerminal = vi.fn(),
}: {
initialRun?: DescriptionGenerationRun;
onTerminal?: (run: DescriptionGenerationRun) => void;
} = {}) {
const client = new QueryClient({
defaultOptions: { queries: { retry: false, gcTime: Infinity } },
});
function Harness() {
const [run, setRun] = useState(initialRun);
return (
<DescriptionGenerationDrawer
open
run={run}
modelLabel="Local Qwen"
onClose={() => undefined}
onRunUpdate={setRun}
onTerminal={onTerminal}
/>
);
}
return {
client,
...render(
<QueryClientProvider client={client}>
<Harness />
<Toaster duration={Infinity} />
</QueryClientProvider>,
),
};
}
const nativeEventSource = globalThis.EventSource;
beforeEach(() => {
FakeEventSource.instances = [];
(globalThis as unknown as { EventSource: typeof EventSource }).EventSource = (
FakeEventSource as unknown as typeof EventSource
);
});
afterEach(() => {
(globalThis as unknown as { EventSource: typeof EventSource }).EventSource = nativeEventSource;
});
test("replays ordered SSE events, reconnects from the latest sequence, and keeps polling as a deduplicated fallback", async () => {
let eventReads = 0;
server.use(
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([runningRun])),
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(runningRun)),
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => {
eventReads += 1;
return HttpResponse.json(eventReads === 1 ? [event(1, "Run queued")] : [
event(2, "First description saved"),
event(3, "Polling recovered this event"),
]);
}),
);
renderDrawer();
const log = await screen.findByRole("log", { name: "Description generation events" });
expect(await within(log).findByText("Run queued")).toBeVisible();
await waitFor(() => expect(FakeEventSource.instances).toHaveLength(1));
expect(FakeEventSource.instances[0].url).toBe(
`/api/catalog/description-generation-runs/${runningRun.id}/events?after=1`,
);
act(() => {
FakeEventSource.instances[0].emitNamed("log", event(2, "First description saved"), "2");
FakeEventSource.instances[0].emitNamed("log", event(2, "First description saved"), "2");
FakeEventSource.instances[0].onerror?.();
});
expect(await within(log).findByText("Polling recovered this event")).toBeVisible();
await waitFor(() => expect(FakeEventSource.instances).toHaveLength(2));
expect(FakeEventSource.instances[1].url).toBe(
`/api/catalog/description-generation-runs/${runningRun.id}/events?after=3`,
);
act(() => FakeEventSource.instances[0].onerror?.());
await new Promise((resolve) => window.setTimeout(resolve, 250));
expect(FakeEventSource.instances).toHaveLength(2);
expect(FakeEventSource.instances[1].closed).toBe(false);
act(() => {
FakeEventSource.instances[1].emitNamed("log", event(3, "Polling recovered this event"), "3");
FakeEventSource.instances[1].emitNamed("log", event(4, "Replay continued in order"), "4");
});
await waitFor(() => expect(within(log).getAllByText("First description saved")).toHaveLength(1));
expect(within(log).getAllByText("Polling recovered this event")).toHaveLength(1);
expect(within(log).getByText("Replay continued in order")).toBeVisible();
expect(log.textContent!.indexOf("Run queued")).toBeLessThan(
log.textContent!.indexOf("First description saved"),
);
expect(log.textContent!.indexOf("First description saved")).toBeLessThan(
log.textContent!.indexOf("Polling recovered this event"),
);
});
test("stops queued or running work without hiding earlier results or events", async () => {
const user = userEvent.setup();
const cancelledRun: DescriptionGenerationRun = {
...runningRun,
status: "cancelled",
processed: 2,
generated: 2,
updatedAt: "2026-08-28T08:03:00Z",
finishedAt: "2026-08-28T08:03:00Z",
};
let cancelledRunId: string | undefined;
server.use(
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([cancelledRun])),
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(runningRun)),
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([
event(1, "First description remains saved"),
])),
http.post("/api/catalog/description-generation-runs/:runId/cancel", ({ params }) => {
cancelledRunId = String(params.runId);
return HttpResponse.json(cancelledRun);
}),
);
renderDrawer();
const drawer = await screen.findByRole("complementary", { name: "Description generation" });
expect(await within(drawer).findByText("First description remains saved")).toBeVisible();
await user.click(within(drawer).getByRole("button", { name: "Stop" }));
expect(await within(drawer).findByRole("heading", { name: "Cancelled" })).toBeVisible();
expect(cancelledRunId).toBe(runningRun.id);
expect(within(drawer).getByText("2", { selector: "[data-count='generated']" })).toBeVisible();
expect(within(drawer).getByText("First description remains saved")).toBeVisible();
});
test.each([
"completed",
"completed_with_errors",
"failed",
"cancelled",
"interrupted",
] satisfies DescriptionGenerationRun["status"][])(
"notifies catalog consumers when a run becomes terminal with %s",
async (status) => {
const onTerminal = vi.fn();
const terminalRun: DescriptionGenerationRun = {
...runningRun,
status,
finishedAt: "2026-08-28T08:04:00Z",
};
server.use(
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([terminalRun])),
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(terminalRun)),
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
);
renderDrawer({ initialRun: terminalRun, onTerminal });
await waitFor(() => expect(onTerminal).toHaveBeenCalledOnce());
expect(onTerminal).toHaveBeenCalledWith(terminalRun);
},
);
test("reopens a terminal run from newest-first persisted history", async () => {
const user = userEvent.setup();
const completedRun: DescriptionGenerationRun = {
...runningRun,
id: "88888888-8888-4888-8888-888888888888",
scope: "selected_tables",
status: "completed",
total: 3,
processed: 3,
generated: 3,
createdAt: "2026-08-28T07:00:00Z",
updatedAt: "2026-08-28T07:02:00Z",
finishedAt: "2026-08-28T07:02:00Z",
};
server.use(
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([
runningRun,
completedRun,
])),
http.get("/api/catalog/description-generation-runs/:runId", ({ params }) => (
HttpResponse.json(params.runId === completedRun.id ? completedRun : runningRun)
)),
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
);
renderDrawer();
const history = await screen.findByRole("region", {
name: "Description generation history",
});
await user.click(within(history).getByRole("button", {
name: /Completed.*selected tables/i,
}));
const drawer = screen.getByRole("complementary", { name: "Description generation" });
expect(await within(drawer).findByRole("heading", { name: "Completed" })).toBeVisible();
expect(within(drawer).getByText("3", { selector: "[data-count='generated']" })).toBeVisible();
});
test("offers Unlock only for an apparently stale active run and explains the interruption", async () => {
const user = userEvent.setup();
const staleRun: DescriptionGenerationRun = {
...runningRun,
updatedAt: "2020-01-01T00:00:00Z",
};
const interruptedRun: DescriptionGenerationRun = {
...staleRun,
status: "interrupted",
finishedAt: "2026-08-28T08:05:00Z",
};
server.use(
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([staleRun])),
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(staleRun)),
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
http.post("/api/catalog/description-generation-runs/unlock", () => (
HttpResponse.json(interruptedRun)
)),
);
renderDrawer({ initialRun: staleRun });
const drawer = await screen.findByRole("complementary", { name: "Description generation" });
expect(within(drawer).getByText(
"Unlock interrupts apparently stale work and marks the run interrupted. A live worker will prevent it.",
)).toBeVisible();
await user.click(within(drawer).getByRole("button", { name: "Unlock stale run" }));
expect(await within(drawer).findByRole("heading", { name: "Interrupted" })).toBeVisible();
});
test("does not offer Unlock while an active run still appears current", async () => {
server.use(
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([runningRun])),
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(runningRun)),
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
);
renderDrawer();
const drawer = await screen.findByRole("complementary", { name: "Description generation" });
expect(within(drawer).queryByRole("button", { name: "Unlock stale run" })).not.toBeInTheDocument();
});
test("explains when Unlock is refused because a live worker still exists", async () => {
const user = userEvent.setup();
const staleRun: DescriptionGenerationRun = {
...runningRun,
updatedAt: "2020-01-01T00:00:00Z",
};
server.use(
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([staleRun])),
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(staleRun)),
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
http.post("/api/catalog/description-generation-runs/unlock", () => HttpResponse.json({
code: "description_generation_run_active",
message: "Private worker detail must not be shown.",
}, { status: 409 })),
);
renderDrawer({ initialRun: staleRun });
const drawer = await screen.findByRole("complementary", { name: "Description generation" });
await user.click(within(drawer).getByRole("button", { name: "Unlock stale run" }));
expect(await screen.findByText(
"Unlock was refused because a live description worker is still running.",
)).toBeVisible();
expect(screen.queryByText("Private worker detail must not be shown.")).not.toBeInTheDocument();
expect(within(drawer).getByRole("heading", { name: "Running" })).toBeVisible();
});
test("does not render stack traces or sensitive generation details from a final summary", async () => {
const failedRun: DescriptionGenerationRun = {
...runningRun,
status: "failed",
finishedAt: "2026-08-28T08:05:00Z",
errorSummary: "Traceback: provider payload included prompt and source sample\n at helper.py:42",
};
server.use(
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([failedRun])),
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(failedRun)),
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
);
renderDrawer({ initialRun: failedRun });
expect(await screen.findByText(
"The run ended before all targets were processed. Review the event log for safe details.",
)).toBeVisible();
expect(screen.queryByText(/Traceback|provider payload|source sample/i)).not.toBeInTheDocument();
});
@@ -0,0 +1,413 @@
import { useCallback, useEffect, useRef, useState } from "react";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { LoaderCircle, LockOpen, Square, X } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../../components/ui/button";
import { ApiError, apiErrorMessage } from "../../api/client";
import {
cancelDescriptionGenerationRun,
descriptionGenerationEventsUrl,
getDescriptionGenerationRun,
listDescriptionGenerationEvents,
listDescriptionGenerationRuns,
type DescriptionGenerationEvent,
type DescriptionGenerationRun,
unlockDescriptionGenerationRun,
} from "../../api/catalog-databases";
interface Props {
open: boolean;
run: DescriptionGenerationRun | null;
modelLabel: string;
onClose: () => void;
onRunUpdate: (run: DescriptionGenerationRun) => void;
onTerminal: (run: DescriptionGenerationRun) => void;
}
const APPARENTLY_STALE_AFTER_MS = 5 * 60 * 1_000;
const SAFE_FINAL_ERROR = "The run ended before all targets were processed. Review the event log for safe details.";
function statusLabel(run: DescriptionGenerationRun): string {
const label = run.status.replaceAll("_", " ");
return `${label[0].toUpperCase()}${label.slice(1)}`;
}
function isTerminal(run?: DescriptionGenerationRun): boolean {
return Boolean(run && [
"completed", "completed_with_errors", "cancelled", "failed", "interrupted",
].includes(run.status));
}
function isApparentlyStale(run: DescriptionGenerationRun): boolean {
if (!["queued", "running"].includes(run.status)) return false;
const updatedAt = new Date(run.updatedAt).getTime();
return Number.isFinite(updatedAt) && Date.now() - updatedAt >= APPARENTLY_STALE_AFTER_MS;
}
function safeFinalError(summary: string | null): string | null {
const value = summary?.trim();
if (!value) return null;
if (
value.length > 500
|| /[\r\n]/.test(value)
|| /traceback|stack trace|(?:^|\s)at\s+\S+[:(]|prompt|provider\s+payload|source\s+sample|bearer\s+\S+|api[_ -]?key|sk-[a-z0-9_-]+/i.test(value)
) return SAFE_FINAL_ERROR;
return value;
}
function timestamp(value: string | null): string {
return value ? new Date(value).toLocaleString() : "Not available";
}
export function DescriptionGenerationDrawer({
open,
run: initialRun,
modelLabel,
onClose,
onRunUpdate,
onTerminal,
}: Props) {
const queryClient = useQueryClient();
const runId = initialRun?.id ?? null;
const [events, setEvents] = useState<DescriptionGenerationEvent[]>([]);
const [action, setAction] = useState<"stop" | "unlock" | null>(null);
const lastSequence = useRef(0);
const terminalRuns = useRef(new Set<string>());
const finalEventsRun = useRef<string | null>(null);
const runQuery = useQuery({
queryKey: ["description-generation-run", runId],
queryFn: () => getDescriptionGenerationRun(runId!),
enabled: Boolean(runId),
initialData: initialRun ?? undefined,
retry: false,
refetchInterval: (query) => isTerminal(query.state.data) ? false : 1_000,
});
const run = runQuery.data;
const eventQuery = useQuery({
queryKey: ["description-generation-events", runId],
queryFn: () => listDescriptionGenerationEvents(runId!, lastSequence.current),
enabled: Boolean(runId),
retry: false,
refetchInterval: run && isTerminal(run) ? false : 1_500,
});
const historyQuery = useQuery({
queryKey: ["description-generation-runs", 50],
queryFn: () => listDescriptionGenerationRuns(50),
enabled: open,
retry: false,
refetchInterval: (query) => query.state.data?.some((item) => !isTerminal(item))
? 3_000
: false,
});
const mergeEvents = useCallback((incoming: DescriptionGenerationEvent[]) => {
if (incoming.length === 0) return;
setEvents((current) => {
const bySequence = new Map(current.map((event) => [event.sequence, event]));
for (const event of incoming) bySequence.set(event.sequence, event);
const merged = [...bySequence.values()].sort((left, right) => left.sequence - right.sequence);
lastSequence.current = merged.at(-1)?.sequence ?? lastSequence.current;
return merged;
});
}, []);
useEffect(() => {
setEvents([]);
lastSequence.current = 0;
finalEventsRun.current = null;
}, [runId]);
useEffect(() => {
mergeEvents(eventQuery.data ?? []);
}, [eventQuery.data, mergeEvents]);
const live = Boolean(run && !isTerminal(run));
useEffect(() => {
if (
!runId
|| !open
|| !live
|| !eventQuery.isFetched
|| typeof EventSource === "undefined"
) return;
let source: EventSource | null = null;
let reconnectTimer: number | null = null;
let disposed = false;
const connect = () => {
if (disposed) return;
const polledSequence = (eventQuery.data ?? []).reduce(
(latest, event) => Math.max(latest, event.sequence),
0,
);
lastSequence.current = Math.max(lastSequence.current, polledSequence);
let currentSource: EventSource;
try {
currentSource = new EventSource(descriptionGenerationEventsUrl(runId, lastSequence.current));
} catch {
return;
}
source = currentSource;
const isCurrent = () => !disposed && source === currentSource;
const log = (message: MessageEvent<string>) => {
if (!isCurrent()) return;
try {
mergeEvents([JSON.parse(message.data) as DescriptionGenerationEvent]);
} catch { /* ordered polling remains authoritative */ }
};
const update = (message: MessageEvent<string>) => {
if (!isCurrent()) return;
try {
queryClient.setQueryData(
["description-generation-run", runId],
JSON.parse(message.data) as DescriptionGenerationRun,
);
} catch { /* status polling remains authoritative */ }
};
const message = (incoming: MessageEvent<string>) => {
if (!isCurrent()) return;
try {
const parsed = JSON.parse(incoming.data) as DescriptionGenerationEvent | DescriptionGenerationRun;
if ("sequence" in parsed) mergeEvents([parsed]);
else if ("status" in parsed) queryClient.setQueryData(["description-generation-run", runId], parsed);
} catch { /* polling remains authoritative */ }
};
currentSource.addEventListener("log", log as EventListener);
currentSource.addEventListener("run", update as EventListener);
currentSource.onmessage = message;
currentSource.onerror = () => {
if (!isCurrent() || reconnectTimer !== null) return;
currentSource.close();
source = null;
void eventQuery.refetch().finally(() => {
if (disposed || reconnectTimer !== null) return;
reconnectTimer = window.setTimeout(() => {
reconnectTimer = null;
connect();
}, 200);
});
};
};
connect();
return () => {
disposed = true;
source?.close();
if (reconnectTimer !== null) window.clearTimeout(reconnectTimer);
};
}, [eventQuery.isFetched, eventQuery.refetch, live, mergeEvents, open, queryClient, runId]);
useEffect(() => {
if (!run) return;
onRunUpdate(run);
queryClient.setQueryData<DescriptionGenerationRun[]>(
["description-generation-runs", 50],
(current) => {
if (!current) return [run];
return current.some((item) => item.id === run.id)
? current.map((item) => item.id === run.id ? run : item)
: [run, ...current];
},
);
}, [onRunUpdate, queryClient, run]);
useEffect(() => {
if (!run || !isTerminal(run) || finalEventsRun.current === run.id) return;
finalEventsRun.current = run.id;
void eventQuery.refetch();
}, [eventQuery.refetch, run]);
useEffect(() => {
if (!run || !isTerminal(run) || terminalRuns.current.has(run.id)) return;
terminalRuns.current.add(run.id);
onTerminal(run);
}, [onTerminal, run]);
const stop = async () => {
if (!run || isTerminal(run)) return;
setAction("stop");
try {
const next = await cancelDescriptionGenerationRun(run.id);
queryClient.setQueryData(["description-generation-run", run.id], next);
onRunUpdate(next);
await historyQuery.refetch();
} catch (error) {
toast.error(apiErrorMessage(error));
} finally {
setAction(null);
}
};
const unlock = async () => {
if (!run || !isApparentlyStale(run)) return;
setAction("unlock");
try {
const next = await unlockDescriptionGenerationRun();
queryClient.setQueryData(["description-generation-run", next.id], next);
onRunUpdate(next);
await historyQuery.refetch();
} catch (error) {
toast.error(error instanceof ApiError && error.status === 409
? "Unlock was refused because a live description worker is still running."
: apiErrorMessage(error));
} finally {
setAction(null);
}
};
if (!open || !run) return null;
const apparentlyStale = isApparentlyStale(run);
const finalError = safeFinalError(run.errorSummary);
return (
<aside
aria-label="Description generation"
className="fixed inset-y-2 right-0 z-50 flex w-full max-w-[400px] flex-col border-l border-border bg-background shadow-2xl sm:inset-y-4"
>
<div className="flex items-start justify-between gap-4 border-b border-border px-5 py-4">
<div>
<p className="thot-label">Description generation</p>
<h2 className="mt-1 font-heading text-xl font-semibold">{statusLabel(run)}</h2>
<p className="mt-1 text-sm text-muted-foreground">
{modelLabel}{modelLabel !== run.modelId ? ` (${run.modelId})` : ""} · {run.scope.replaceAll("_", " ")}
</p>
</div>
<Button type="button" variant="ghost" size="icon-lg" aria-label="Close description generation" onClick={onClose}>
<X aria-hidden="true" />
</Button>
</div>
<div className="min-h-0 flex-1 overflow-y-auto px-5 py-5">
{runQuery.isError ? (
<div role="alert" className="rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
{apiErrorMessage(runQuery.error)}
</div>
) : null}
<section aria-label="Description generation counters">
<h3 className="thot-label mb-2">Progress</h3>
<div className="grid grid-cols-2 gap-2">
{([
["total", "Total"],
["processed", "Processed"],
["generated", "Generated"],
["nonGeneratable", "Not generated"],
["failed", "Failed"],
] as const).map(([name, label]) => (
<div key={name} className="rounded-md border border-border bg-muted/25 px-3 py-2">
<p className="text-[11px] font-semibold uppercase tracking-wide text-muted-foreground">{label}</p>
<p data-count={name} className="mt-1 text-lg font-semibold tabular-nums">{run[name]}</p>
</div>
))}
</div>
</section>
<section className="mt-5" aria-label="Description generation timestamps">
<h3 className="thot-label mb-2">Timestamps</h3>
<dl className="grid grid-cols-[auto_1fr] gap-x-4 gap-y-1.5 text-sm">
<dt className="text-muted-foreground">Created</dt><dd className="text-right">{timestamp(run.createdAt)}</dd>
<dt className="text-muted-foreground">Started</dt><dd className="text-right">{timestamp(run.startedAt)}</dd>
<dt className="text-muted-foreground">Updated</dt><dd className="text-right">{timestamp(run.updatedAt)}</dd>
<dt className="text-muted-foreground">Finished</dt><dd className="text-right">{timestamp(run.finishedAt)}</dd>
</dl>
</section>
{finalError ? (
<div className="mt-5 rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
{finalError}
</div>
) : null}
<section className="mt-5" aria-label="Description generation log">
<div className="mb-2 flex items-center justify-between">
<h3 className="thot-label">Events</h3>
<span className="text-xs tabular-nums text-muted-foreground">{events.length}</span>
</div>
<div
role="log"
aria-label="Description generation events"
className="max-h-72 overflow-y-auto rounded-md bg-zinc-950 p-3 font-mono text-xs leading-5 text-zinc-200"
>
{events.length === 0 ? <p className="text-zinc-500">Waiting for events…</p> : events.map((event) => (
<p key={event.sequence} className={event.level === "error" ? "text-red-300" : event.level === "warning" ? "text-amber-300" : undefined}>
<span className="mr-2 text-zinc-500">{new Date(event.createdAt).toLocaleTimeString()}</span>{event.message}
</p>
))}
</div>
{eventQuery.isError ? <p role="alert" className="mt-2 text-sm text-destructive">{apiErrorMessage(eventQuery.error)}</p> : null}
</section>
<section className="mt-5" aria-label="Description generation history">
<div className="mb-2 flex items-center justify-between gap-3">
<h3 className="thot-label">Recent runs</h3>
<span className="text-xs tabular-nums text-muted-foreground">
{historyQuery.data?.length ?? 0}
</span>
</div>
{historyQuery.isError ? (
<p role="alert" className="text-sm text-destructive">
{apiErrorMessage(historyQuery.error)}
</p>
) : historyQuery.isLoading ? (
<p className="text-sm text-muted-foreground">Loading run history…</p>
) : (historyQuery.data ?? []).length === 0 ? (
<p className="text-sm text-muted-foreground">No description generation runs yet.</p>
) : (
<div className="max-h-56 divide-y divide-border overflow-y-auto rounded-md border border-border">
{(historyQuery.data ?? []).map((item) => (
<button
key={item.id}
type="button"
aria-label={`${statusLabel(item)} · ${item.scope.replaceAll("_", " ")}`}
aria-current={item.id === run.id ? "true" : undefined}
className="flex w-full items-start justify-between gap-3 px-3 py-2 text-left text-sm hover:bg-muted/50 focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/25"
onClick={() => onRunUpdate(item)}
>
<span className="font-medium">{statusLabel(item)}</span>
<span className="shrink-0 text-right text-xs text-muted-foreground">
{item.processed}/{item.total}<br />{new Date(item.createdAt).toLocaleString()}
</span>
</button>
))}
</div>
)}
</section>
</div>
{["queued", "running"].includes(run.status) ? (
<div className="border-t border-border px-5 py-4">
{apparentlyStale ? (
<p
id="description-generation-unlock-help"
className="mb-3 rounded-md border border-amber-500/35 bg-amber-500/8 px-3 py-2 text-xs leading-5 text-muted-foreground"
>
Unlock interrupts apparently stale work and marks the run interrupted. A live worker will prevent it.
</p>
) : null}
<div className="flex flex-wrap justify-end gap-2">
{apparentlyStale ? (
<Button
type="button"
variant="outline"
aria-label="Unlock stale run"
aria-describedby="description-generation-unlock-help"
disabled={action !== null}
onClick={() => void unlock()}
>
{action === "unlock" ? <LoaderCircle className="animate-spin" /> : <LockOpen />}
{action === "unlock" ? "Unlocking…" : "Unlock"}
</Button>
) : null}
<Button
type="button"
variant="destructive"
disabled={action !== null}
onClick={() => void stop()}
>
{action === "stop" ? <LoaderCircle className="animate-spin" /> : <Square />}
{action === "stop" ? "Stopping…" : "Stop"}
</Button>
</div>
</div>
) : null}
</aside>
);
}
@@ -0,0 +1,75 @@
import type { MetadataGenerationModels } from "../../api/catalog-databases";
interface Props {
canManage: boolean;
data?: MetadataGenerationModels;
isError: boolean;
isLoading: boolean;
selectedModel: string;
onSelectedModelChange: (modelId: string) => void;
}
export function MetadataGenerationModelSelector({
canManage,
data,
isError,
isLoading,
selectedModel,
onSelectedModelChange,
}: Props) {
const models = data?.models ?? [];
const configuredDefault = data?.default ?? "";
const hasUsableDefault = configuredDefault !== ""
&& models.some((model) => model.id === configuredDefault);
const noUsableModel = !isLoading && !isError && Boolean(data) && !hasUsableDefault;
const unavailable = noUsableModel || (!isLoading && isError);
const describedBy = [
"metadata-generation-source-data-disclosure",
unavailable ? "metadata-generation-model-help" : null,
].filter(Boolean).join(" ");
return (
<div className="grid min-w-56 max-w-lg gap-1.5">
<label className="grid gap-1.5 text-xs font-medium text-foreground">
<span>Metadata description model</span>
<select
className="h-9 w-full rounded-md border border-input bg-card px-3 text-sm outline-none transition focus:border-primary/60 focus:ring-3 focus:ring-ring/15 disabled:bg-muted disabled:text-muted-foreground"
aria-label="Metadata description model"
aria-describedby={describedBy}
value={selectedModel}
disabled={!canManage || isLoading || isError || !hasUsableDefault}
onChange={(event) => onSelectedModelChange(event.target.value)}
>
{isLoading ? <option value="">Loading models…</option> : null}
{!isLoading && isError ? <option value="">Model choices unavailable</option> : null}
{noUsableModel ? (
<option value="">{models.length === 0 ? "No model configured" : "No usable model configured"}</option>
) : null}
{models.map((model) => (
<option key={model.id} value={model.id}>{model.label}</option>
))}
</select>
</label>
{unavailable ? (
<p id="metadata-generation-model-help" className="max-w-72 text-xs font-normal leading-4 text-muted-foreground">
Configure a metadata-generation model in application setup to enable description generation.
</p>
) : null}
<div
id="metadata-generation-source-data-disclosure"
role="note"
aria-label="Metadata generation source data disclosure"
className="space-y-1 text-xs font-normal leading-4 text-muted-foreground"
>
<p>
Description generation may send up to five real source rows and up to five representative
non-null example values to the selected model provider.
</p>
<p>
Samples are transient and are not stored in run logs or catalog metadata. Automated
Sensitive Data Policy filtering and anonymization are not currently provided; they are
planned for future work.
</p>
</div>
</div>
);
}
+2
View File
@@ -11,4 +11,6 @@ export const server = setupServer(
})),
http.get("/api/health/dwh", () => HttpResponse.json({ ok: true, detail: "ok" })),
http.get("/api/catalog/databases", () => HttpResponse.json([])),
http.get("/api/catalog/metadata-generation/models", () => HttpResponse.json({ models: [], default: null })),
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([])),
);
+1
View File
@@ -17,6 +17,7 @@ dependencies = [
"tqdm>=4.66",
"yake>=0.4",
"portalocker>=2.10",
"litellm>=1.98,<2",
]
[project.scripts]
@@ -0,0 +1,421 @@
from __future__ import annotations
import json
import os
import subprocess
import sys
from pathlib import Path
from types import SimpleNamespace
import pytest
from tht.internal.litellm_completion import handle_request
HARNESS_ROOT = Path(__file__).resolve().parents[1]
HELPER_MODULE = "tht.internal.litellm_completion"
def _run_raw_helper(
tmp_path: Path, raw_request: str, fake_litellm: str
) -> subprocess.CompletedProcess[str]:
provider_path = tmp_path / "provider"
provider_path.mkdir()
(provider_path / "litellm.py").write_text(fake_litellm, encoding="utf-8")
env = os.environ.copy()
env["PYTHONPATH"] = os.pathsep.join((str(provider_path), str(HARNESS_ROOT)))
return subprocess.run(
[sys.executable, "-m", HELPER_MODULE],
input=raw_request,
text=True,
capture_output=True,
cwd=HARNESS_ROOT,
env=env,
timeout=10,
check=False,
)
def _run_helper(tmp_path: Path, request: object, fake_litellm: str) -> subprocess.CompletedProcess[str]:
return _run_raw_helper(tmp_path, json.dumps(request), fake_litellm)
def _valid_request() -> dict[str, object]:
return {
"model": "openai/gpt-4.1-mini",
"api_key": "sk-secret",
"messages": [{"role": "user", "content": "private prompt"}],
}
def test_subprocess_emits_one_pristine_success_line_and_silences_provider_output(tmp_path: Path):
secret = "sk-contract-secret"
prompt = "private patient prompt"
request = {
"model": "openai/gpt-4.1-mini",
"api_key": secret,
"messages": [
{"role": "system", "content": "Return one description."},
{"role": "user", "content": prompt},
],
"api_base": "https://models.example.test/v1",
"api_version": "2026-08-01-preview",
}
fake_litellm = f'''
import sys
suppress_debug_info = False
set_verbose = True
turn_off_message_logging = False
log_raw_request_response = True
print("provider import noise")
print("provider import diagnostic", file=sys.stderr)
def completion(**kwargs):
assert suppress_debug_info is True
assert set_verbose is False
assert turn_off_message_logging is True
assert log_raw_request_response is False
assert kwargs == {{
"model": "openai/gpt-4.1-mini",
"api_key": {secret!r},
"messages": [
{{"role": "system", "content": "Return one description."}},
{{"role": "user", "content": {prompt!r}}},
],
"api_base": "https://models.example.test/v1",
"api_version": "2026-08-01-preview",
"num_retries": 1,
"stream": False,
}}
print(repr(kwargs))
print(repr(kwargs), file=sys.stderr)
return {{"choices": [{{"message": {{"content": "Descrizione italiana"}}}}]}}
'''
result = _run_helper(tmp_path, request, fake_litellm)
assert result.returncode == 0
assert result.stdout == '{"ok":true,"content":"Descrizione italiana"}\n'
assert result.stderr == ""
assert secret not in result.stdout + result.stderr
assert prompt not in result.stdout + result.stderr
def test_subprocess_rejects_an_unknown_request_key_before_loading_provider(tmp_path: Path):
request = {**_valid_request(), "unexpected": True}
result = _run_helper(tmp_path, request, 'raise AssertionError("provider was loaded")')
assert result.returncode == 0
assert result.stdout == '{"ok":false,"error":"invalid_request"}\n'
assert result.stderr == "completion helper: invalid request\n"
@pytest.mark.parametrize(
"raw_request",
[
"{not-json",
"[]",
json.dumps({"model": "openai/gpt-4.1-mini", "api_key": "sk-secret"}),
json.dumps({**_valid_request(), "model": "bad model"}),
json.dumps({**_valid_request(), "model": "a" * 257}),
json.dumps({**_valid_request(), "api_key": " "}),
json.dumps({**_valid_request(), "api_key": "x" * (16 * 1024 + 1)}),
json.dumps(
{
**_valid_request(),
"messages": [{"role": "assistant", "content": "not allowed"}],
}
),
json.dumps(
{
**_valid_request(),
"messages": [{"role": "user", "content": "ok", "name": "extra"}],
}
),
json.dumps({**_valid_request(), "messages": []}),
json.dumps(
{
**_valid_request(),
"messages": [{"role": "user", "content": "ok"}] * 33,
}
),
json.dumps(
{
**_valid_request(),
"messages": [{"role": "user"}],
}
),
json.dumps(
{
**_valid_request(),
"messages": [{"role": "user", "content": 1}],
}
),
json.dumps(
{
**_valid_request(),
"messages": [{"role": "user", "content": "x" * (64 * 1024 + 1)}],
}
),
json.dumps(
{
**_valid_request(),
"messages": [
{"role": "user", "content": "x" * (48 * 1024)},
{"role": "system", "content": "y" * (48 * 1024)},
{"role": "user", "content": "z" * (48 * 1024)},
],
}
),
json.dumps({**_valid_request(), "api_base": "file:///tmp/provider"}),
json.dumps({**_valid_request(), "api_base": "https://user@example.test/v1"}),
json.dumps({**_valid_request(), "api_base": "https://example.test/v1?key=value"}),
json.dumps({**_valid_request(), "api_version": "2026 preview"}),
json.dumps({**_valid_request(), "disable_thinking": False}),
(
'{"model":"openai/first","model":"openai/second","api_key":"sk-secret",'
'"messages":[{"role":"user","content":"private prompt"}]}'
),
(
'{"model":"openai/gpt-4.1-mini","api_key":"sk-secret",'
'"messages":[{"role":"user","content":"private prompt"}],"number":'
+ "1" * 5000
+ "}"
),
json.dumps(
{
**_valid_request(),
"messages": [{"role": "user", "content": "x" * (256 * 1024)}],
}
),
],
ids=[
"malformed-json",
"non-object",
"missing-required-key",
"invalid-model",
"long-model",
"empty-secret",
"long-secret",
"invalid-role",
"message-extra-key",
"empty-messages",
"too-many-messages",
"message-missing-key",
"non-string-content",
"long-message",
"aggregate-content-too-long",
"invalid-url-scheme",
"url-credentials",
"url-query",
"invalid-api-version",
"invalid-disable-thinking",
"duplicate-key",
"oversized-number",
"oversized-stdin",
],
)
def test_subprocess_rejects_malformed_or_oversized_requests_without_loading_provider(
tmp_path: Path, raw_request: str
):
result = _run_raw_helper(
tmp_path,
raw_request,
'raise AssertionError("provider was loaded with private prompt and sk-secret")',
)
assert result.returncode == 0
assert result.stdout == '{"ok":false,"error":"invalid_request"}\n'
assert result.stderr == "completion helper: invalid request\n"
assert "private prompt" not in result.stdout + result.stderr
assert "sk-secret" not in result.stdout + result.stderr
def test_subprocess_normalizes_provider_failure_without_leaking_provider_output(tmp_path: Path):
secret = "sk-provider-canary"
prompt = "provider prompt canary"
request = {
**_valid_request(),
"api_key": secret,
"messages": [{"role": "user", "content": prompt}],
}
fake_litellm = '''
import os
import sys
def completion(**kwargs):
leaked = repr(kwargs).encode("utf-8")
print(repr(kwargs))
print(repr(kwargs), file=sys.stderr)
os.write(1, leaked + b"\\n")
os.write(2, leaked + b"\\n")
os.write(1, b"x" * (1024 * 1024))
os.write(2, b"x" * (1024 * 1024))
raise RuntimeError(repr(kwargs))
'''
result = _run_helper(tmp_path, request, fake_litellm)
assert result.returncode == 0
assert result.stdout == '{"ok":false,"error":"provider_failure"}\n'
assert result.stderr == "completion helper: provider failure\n"
assert len(result.stderr.encode("utf-8")) <= 256
assert secret not in result.stdout + result.stderr
assert prompt not in result.stdout + result.stderr
def test_subprocess_normalizes_an_invalid_response_to_one_safe_line(tmp_path: Path):
secret = "sk-invalid-response"
prompt = "invalid response prompt"
request = {
**_valid_request(),
"api_key": secret,
"messages": [{"role": "user", "content": prompt}],
}
fake_litellm = '''
import os
def completion(**kwargs):
os.write(1, repr(kwargs).encode("utf-8"))
os.write(2, repr(kwargs).encode("utf-8"))
return {"choices": [{"message": {"content": ""}}]}
'''
result = _run_helper(tmp_path, request, fake_litellm)
assert result.returncode == 0
assert result.stdout == '{"ok":false,"error":"invalid_response"}\n'
assert result.stderr == "completion helper: invalid response\n"
assert secret not in result.stdout + result.stderr
assert prompt not in result.stdout + result.stderr
def test_subprocess_escapes_multiline_content_without_adding_output_frames(tmp_path: Path):
content = 'Prima riga\nSeconda "riga" — fine'
fake_litellm = f'''
def completion(**kwargs):
return {{"choices": [{{"message": {{"content": {content!r}}}}}]}}
'''
result = _run_helper(tmp_path, _valid_request(), fake_litellm)
assert result.returncode == 0
assert result.stdout == json.dumps(
{"ok": True, "content": content}, ensure_ascii=False, separators=(",", ":")
) + "\n"
assert result.stdout.count("\n") == 1
assert result.stderr == ""
def test_injected_completion_configures_one_retry_without_fallback_or_secret_export():
secret = "sk-injected-canary"
request = {
**_valid_request(),
"api_key": secret,
"messages": [
{"role": "system", "content": "System instruction"},
{"role": "user", "content": "Private prompt"},
],
}
calls: list[dict[str, object]] = []
def completion(**kwargs: object) -> object:
assert secret not in os.environ.values()
calls.append(kwargs)
return SimpleNamespace(
choices=[SimpleNamespace(message=SimpleNamespace(content="Validated content"))]
)
result = handle_request(request, completion=completion)
assert result == {"ok": True, "content": "Validated content"}
assert calls == [{
"model": "openai/gpt-4.1-mini",
"api_key": secret,
"messages": [
{"role": "system", "content": "System instruction"},
{"role": "user", "content": "Private prompt"},
],
"num_retries": 1,
"stream": False,
}]
def test_injected_completion_uses_non_secret_client_placeholder_for_keyless_endpoint():
calls: list[dict[str, object]] = []
def completion(**kwargs: object) -> object:
calls.append(kwargs)
return {"choices": [{"message": {"content": "Descrizione Qwen"}}]}
request = {
"model": "openai/qwen3.6-35b-a3b",
"messages": [{"role": "user", "content": "Invented metadata"}],
"api_base": "https://models.internal.example/v1",
"disable_thinking": True,
}
assert handle_request(request, completion=completion) == {
"ok": True,
"content": "Descrizione Qwen",
}
assert calls == [{
"model": "openai/qwen3.6-35b-a3b",
"api_key": "not-required",
"messages": [{"role": "user", "content": "Invented metadata"}],
"num_retries": 1,
"stream": False,
"api_base": "https://models.internal.example/v1",
"extra_body": {"chat_template_kwargs": {"enable_thinking": False}},
}]
def test_injected_provider_failure_is_not_retried():
attempts = 0
def completion(**kwargs: object) -> object:
nonlocal attempts
attempts += 1
raise RuntimeError(repr(kwargs))
result = handle_request(_valid_request(), completion=completion)
assert result == {"ok": False, "error": "provider_failure"}
assert attempts == 1
@pytest.mark.parametrize(
"response",
[
None,
{},
{"choices": []},
{
"choices": [
{"message": {"content": "first"}},
{"message": {"content": "second"}},
]
},
{"choices": [{"message": {}}]},
{"choices": [{"message": {"content": None}}]},
{"choices": [{"message": {"content": " "}}]},
{"choices": [{"message": {"content": "x" * (64 * 1024 + 1)}}]},
],
ids=[
"none",
"missing-choices",
"empty-choices",
"ambiguous-choices",
"missing-content",
"non-string-content",
"empty-content",
"oversized-content",
],
)
def test_injected_completion_rejects_invalid_response_shapes(response: object):
result = handle_request(_valid_request(), completion=lambda **_: response)
assert result == {"ok": False, "error": "invalid_response"}
@@ -0,0 +1,9 @@
import tomllib
from pathlib import Path
def test_runtime_dependencies_constrain_litellm_to_the_supported_major():
pyproject = Path(__file__).resolve().parents[1] / "pyproject.toml"
project = tomllib.loads(pyproject.read_text(encoding="utf-8"))["project"]
assert "litellm>=1.98,<2" in project["dependencies"]
+1
View File
@@ -0,0 +1 @@
"""Internal process adapters that are not part of the ``tht`` CLI surface."""
+316
View File
@@ -0,0 +1,316 @@
"""One-shot stdin/stdout adapter around LiteLLM chat completion."""
from __future__ import annotations
import contextlib
import json
import os
import re
import sys
from collections.abc import Callable, Iterator, Mapping, Sequence
from dataclasses import dataclass
from typing import Any
from urllib.parse import urlsplit
MAX_STDIN_BYTES = 256 * 1024
MAX_API_KEY_BYTES = 16 * 1024
MAX_MESSAGES = 32
MAX_MESSAGE_CONTENT_BYTES = 64 * 1024
MAX_TOTAL_CONTENT_BYTES = 128 * 1024
MAX_RESPONSE_CONTENT_BYTES = 64 * 1024
MAX_STDERR_BYTES = 256
KEYLESS_API_KEY_PLACEHOLDER = "not-required"
_REQUIRED_KEYS = frozenset({"model", "messages"})
_OPTIONAL_KEYS = frozenset({"api_key", "api_base", "api_version", "disable_thinking"})
_MESSAGE_KEYS = frozenset({"role", "content"})
_MODEL_PATTERN = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}\Z")
_API_VERSION_PATTERN = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}\Z")
_DIAGNOSTICS = {
"invalid_request": "completion helper: invalid request\n",
"provider_failure": "completion helper: provider failure\n",
"invalid_response": "completion helper: invalid response\n",
}
Completion = Callable[..., object]
Result = dict[str, object]
class _InvalidRequest(Exception):
pass
class _InvalidResponse(Exception):
pass
@dataclass(frozen=True, slots=True, repr=False)
class _ValidatedRequest:
model: str
api_key: str | None
messages: tuple[tuple[str, str], ...]
api_base: str | None
api_version: str | None
disable_thinking: bool
def _utf8_size(value: str, error: type[Exception]) -> int:
try:
return len(value.encode("utf-8"))
except UnicodeEncodeError as exc:
raise error from exc
def _object_without_duplicates(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
value: dict[str, Any] = {}
for key, item in pairs:
if key in value:
raise _InvalidRequest
value[key] = item
return value
def _reject_nonstandard_number(_: str) -> None:
raise _InvalidRequest
def _read_json_request() -> object:
raw = sys.stdin.buffer.read(MAX_STDIN_BYTES + 1)
if not raw or len(raw) > MAX_STDIN_BYTES:
raise _InvalidRequest
try:
return json.loads(
raw.decode("utf-8"),
object_pairs_hook=_object_without_duplicates,
parse_constant=_reject_nonstandard_number,
)
except (UnicodeError, ValueError, RecursionError, _InvalidRequest) as exc:
raise _InvalidRequest from exc
def _validate_api_base(value: object) -> str:
if type(value) is not str or not value or _utf8_size(value, _InvalidRequest) > 2048:
raise _InvalidRequest
if any(character.isspace() or ord(character) < 32 for character in value):
raise _InvalidRequest
try:
parsed = urlsplit(value)
port = parsed.port
except ValueError as exc:
raise _InvalidRequest from exc
if (
parsed.scheme.lower() not in {"http", "https"}
or not parsed.hostname
or parsed.username is not None
or parsed.password is not None
or parsed.query
or parsed.fragment
or (port is not None and not 1 <= port <= 65535)
):
raise _InvalidRequest
return value
def _validate_request(value: object) -> _ValidatedRequest:
if type(value) is not dict:
raise _InvalidRequest
keys = set(value)
if not _REQUIRED_KEYS.issubset(keys) or not keys.issubset(_REQUIRED_KEYS | _OPTIONAL_KEYS):
raise _InvalidRequest
model = value["model"]
if type(model) is not str or _MODEL_PATTERN.fullmatch(model) is None:
raise _InvalidRequest
api_key = None
if "api_key" in value:
candidate = value["api_key"]
if (
type(candidate) is not str
or not candidate
or _utf8_size(candidate, _InvalidRequest) > MAX_API_KEY_BYTES
or any(character.isspace() or ord(character) < 32 for character in candidate)
):
raise _InvalidRequest
api_key = candidate
raw_messages = value["messages"]
if type(raw_messages) is not list or not 1 <= len(raw_messages) <= MAX_MESSAGES:
raise _InvalidRequest
messages: list[tuple[str, str]] = []
total_content_bytes = 0
for raw_message in raw_messages:
if type(raw_message) is not dict or set(raw_message) != _MESSAGE_KEYS:
raise _InvalidRequest
role = raw_message["role"]
content = raw_message["content"]
if type(role) is not str or role not in {"system", "user"}:
raise _InvalidRequest
if type(content) is not str or not content.strip():
raise _InvalidRequest
content_bytes = _utf8_size(content, _InvalidRequest)
if content_bytes > MAX_MESSAGE_CONTENT_BYTES:
raise _InvalidRequest
total_content_bytes += content_bytes
if total_content_bytes > MAX_TOTAL_CONTENT_BYTES:
raise _InvalidRequest
messages.append((role, content))
api_base = None
if "api_base" in value:
api_base = _validate_api_base(value["api_base"])
api_version = None
if "api_version" in value:
candidate = value["api_version"]
if type(candidate) is not str or _API_VERSION_PATTERN.fullmatch(candidate) is None:
raise _InvalidRequest
api_version = candidate
disable_thinking = False
if "disable_thinking" in value:
if value["disable_thinking"] is not True:
raise _InvalidRequest
disable_thinking = True
return _ValidatedRequest(
model=model,
api_key=api_key,
messages=tuple(messages),
api_base=api_base,
api_version=api_version,
disable_thinking=disable_thinking,
)
def _litellm_completion(**kwargs: object) -> object:
import litellm
litellm.suppress_debug_info = True
litellm.set_verbose = False
litellm.turn_off_message_logging = True
litellm.log_raw_request_response = False
litellm.redact_messages_in_exceptions = True
litellm.redact_user_api_key_info = True
return litellm.completion(**kwargs)
@contextlib.contextmanager
def _silence_provider_output() -> Iterator[None]:
with open(os.devnull, "w", encoding="utf-8") as sink:
sys.stdout.flush()
sys.stderr.flush()
saved_stdout = os.dup(1)
saved_stderr = os.dup(2)
try:
os.dup2(sink.fileno(), 1)
os.dup2(sink.fileno(), 2)
with contextlib.redirect_stdout(sink), contextlib.redirect_stderr(sink):
yield
finally:
os.dup2(saved_stdout, 1)
os.dup2(saved_stderr, 2)
os.close(saved_stdout)
os.close(saved_stderr)
def _provider_kwargs(request: _ValidatedRequest) -> dict[str, object]:
kwargs: dict[str, object] = {
"model": request.model,
# OpenAI-compatible SDKs require a non-empty client value even when the
# explicitly configured endpoint does not authenticate requests.
"api_key": request.api_key or KEYLESS_API_KEY_PLACEHOLDER,
"messages": [
{"role": role, "content": content} for role, content in request.messages
],
"num_retries": 1,
"stream": False,
}
if request.api_base is not None:
kwargs["api_base"] = request.api_base
if request.api_version is not None:
kwargs["api_version"] = request.api_version
if request.disable_thinking:
kwargs["extra_body"] = {"chat_template_kwargs": {"enable_thinking": False}}
return kwargs
def _field(value: object, name: str) -> object:
if isinstance(value, Mapping):
if name not in value:
raise _InvalidResponse
return value[name]
try:
return getattr(value, name)
except (AttributeError, TypeError) as exc:
raise _InvalidResponse from exc
def _response_content(response: object) -> str:
choices = _field(response, "choices")
if (
not isinstance(choices, Sequence)
or isinstance(choices, (str, bytes, bytearray))
or len(choices) != 1
):
raise _InvalidResponse
message = _field(choices[0], "message")
content = _field(message, "content")
if (
type(content) is not str
or not content.strip()
or _utf8_size(content, _InvalidResponse) > MAX_RESPONSE_CONTENT_BYTES
):
raise _InvalidResponse
return content
def handle_request(request: object, *, completion: Completion | None = None) -> Result:
"""Validate one request and normalize one injected or LiteLLM completion."""
try:
validated = _validate_request(request)
except _InvalidRequest:
return {"ok": False, "error": "invalid_request"}
provider = completion or _litellm_completion
try:
with _silence_provider_output():
response = provider(**_provider_kwargs(validated))
except Exception: # noqa: BLE001 - provider failures cross a redacted process boundary
return {"ok": False, "error": "provider_failure"}
try:
with _silence_provider_output():
content = _response_content(response)
except Exception: # noqa: BLE001 - arbitrary provider objects are untrusted response data
return {"ok": False, "error": "invalid_response"}
return {"ok": True, "content": content}
def _write_json(payload: Mapping[str, object]) -> None:
sys.stdout.write(json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + "\n")
def _write_diagnostic(result: Mapping[str, object]) -> None:
error = result.get("error")
if not isinstance(error, str):
return
diagnostic = _DIAGNOSTICS.get(error, "completion helper: failure\n")
encoded = diagnostic.encode("utf-8")[:MAX_STDERR_BYTES]
sys.stderr.write(encoded.decode("utf-8", errors="ignore"))
def main() -> None:
try:
request = _read_json_request()
except _InvalidRequest:
result: Result = {"ok": False, "error": "invalid_request"}
else:
result = handle_request(request)
_write_json(result)
if result.get("ok") is False:
_write_diagnostic(result)
if __name__ == "__main__":
main()
+2365
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -83,7 +83,7 @@ architecture = (root / "docs/architecture/authentication.md").read_text()
user_row = "| `user` | `session.use` |"
admin_row = (
"| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, "
"`workspace.manage`, `workspace.secrets.manage`, `pi.manage`, `auth.diagnostics.read` |"
"`workspace.manage`, `workspace.secrets.manage`, `database.manage`, `pi.manage`, `auth.diagnostics.read` |"
)
if user_row not in architecture or admin_row not in architecture:
raise SystemExit("auth docs smoke: role-to-permission map is not exact")
+282 -2
View File
@@ -14,6 +14,7 @@ import (
"sort"
"strings"
"sync"
"unicode"
"github.com/aritmolab/thothii/tools/tht/internal/safeio"
"github.com/compose-spec/compose-go/v2/dotenv"
@@ -25,6 +26,10 @@ const installationFileName = "thothii-installation.yaml"
const maxEnvironmentFileBytes = 1 << 20
const maxMetadataSecretBundleBytes = 64 << 10
const maxMetadataGenerationModels = 64
const maxSecretSources = 32
var dotenvParseMu sync.Mutex
@@ -35,9 +40,34 @@ type descriptor struct {
EnvFile string `yaml:"envFile"`
WorkspaceRepository workspaceRepositoryDescriptor `yaml:"workspaceRepository"`
Authentication authenticationDescriptor `yaml:"authentication"`
MetadataGeneration metadataGenerationDescriptor `yaml:"metadataGeneration"`
Overrides []string `yaml:"overrides"`
}
type metadataGenerationDescriptor struct {
Default string `yaml:"default"`
Models []metadataGenerationModelDescriptor `yaml:"models"`
}
type metadataGenerationModelDescriptor struct {
ID string `yaml:"id"`
Label string `yaml:"label"`
LiteLLM liteLLMDescriptor `yaml:"litellm"`
APIKeyEnv string `yaml:"apiKeyEnv"`
}
type liteLLMDescriptor struct {
Provider string `yaml:"provider"`
Model string `yaml:"model"`
DisableThinking bool `yaml:"disableThinking"`
Endpoint *metadataEndpointDescriptor `yaml:"endpoint"`
}
type metadataEndpointDescriptor struct {
BaseURL string `yaml:"baseUrl"`
APIVersion string `yaml:"apiVersion"`
}
type authenticationDescriptor struct {
ConfigDirectory string `yaml:"configDirectory"`
RuntimeProjection *runtimeProjectionDescriptor `yaml:"runtimeProjection"`
@@ -75,6 +105,36 @@ type Authentication struct {
RuntimeProjection *RuntimeProjection
}
// MetadataGenerationEndpoint contains optional provider endpoint settings for one LiteLLM model.
type MetadataGenerationEndpoint struct {
BaseURL string
APIVersion string
}
// MetadataGenerationLiteLLM identifies the provider/model pair used by the internal completion helper.
type MetadataGenerationLiteLLM struct {
Provider string
Model string
DisableThinking bool
Endpoint *MetadataGenerationEndpoint
}
// MetadataGenerationModel is one selectable installation-owned metadata-generation model.
// APIKeyEnv is an optional reference only; credential values never enter Installation.
// An empty value is allowed only for a model with an explicit keyless endpoint.
type MetadataGenerationModel struct {
ID string
Label string
LiteLLM MetadataGenerationLiteLLM
APIKeyEnv string
}
// MetadataGeneration is the installation-owned model list and its default selection.
type MetadataGeneration struct {
Default string
Models []MetadataGenerationModel
}
// Installation is a validated local Compose installation. It intentionally contains paths, not
// environment values or secret content.
type Installation struct {
@@ -84,6 +144,7 @@ type Installation struct {
EnvFile string
WorkspaceRepository WorkspaceRepository
Authentication Authentication
MetadataGeneration MetadataGeneration
Overrides []string
}
@@ -137,6 +198,29 @@ func Load(path string) (Installation, error) {
GID: raw.Authentication.RuntimeProjection.GID,
}
}
metadataGeneration := MetadataGeneration{
Default: raw.MetadataGeneration.Default,
Models: make([]MetadataGenerationModel, 0, len(raw.MetadataGeneration.Models)),
}
for _, rawModel := range raw.MetadataGeneration.Models {
model := MetadataGenerationModel{
ID: rawModel.ID,
Label: rawModel.Label,
LiteLLM: MetadataGenerationLiteLLM{
Provider: rawModel.LiteLLM.Provider,
Model: rawModel.LiteLLM.Model,
DisableThinking: rawModel.LiteLLM.DisableThinking,
},
APIKeyEnv: rawModel.APIKeyEnv,
}
if rawModel.LiteLLM.Endpoint != nil {
model.LiteLLM.Endpoint = &MetadataGenerationEndpoint{
BaseURL: rawModel.LiteLLM.Endpoint.BaseURL,
APIVersion: rawModel.LiteLLM.Endpoint.APIVersion,
}
}
metadataGeneration.Models = append(metadataGeneration.Models, model)
}
installation := Installation{
Path: path,
Profile: raw.Profile,
@@ -147,13 +231,17 @@ func Load(path string) (Installation, error) {
Branch: raw.WorkspaceRepository.Branch,
Access: raw.WorkspaceRepository.Access,
},
Authentication: authentication,
Overrides: make([]string, 0, len(raw.Overrides)),
Authentication: authentication,
MetadataGeneration: metadataGeneration,
Overrides: make([]string, 0, len(raw.Overrides)),
}
values, err := installation.environmentValues()
if err != nil {
return Installation{}, errors.New("installation secret declarations could not be read")
}
if err := installation.validateMetadataGeneration(values); err != nil {
return Installation{}, err
}
if values["THT_AUTH_CONFIG_ROOT"] != installation.AuthenticationDirectory() {
return Installation{}, errors.New("authentication.configDirectory must match THT_AUTH_CONFIG_ROOT")
}
@@ -189,6 +277,198 @@ func Load(path string) (Installation, error) {
return installation, nil
}
func (i Installation) validateMetadataGeneration(values map[string]string) error {
if len(i.MetadataGeneration.Models) > maxMetadataGenerationModels {
return fmt.Errorf(
"metadataGeneration.models must contain at most %d entries",
maxMetadataGenerationModels,
)
}
seen := make(map[string]struct{}, len(i.MetadataGeneration.Models))
for index, model := range i.MetadataGeneration.Models {
if err := validateMetadataGenerationModel(index, model); err != nil {
return err
}
if _, exists := seen[model.ID]; exists {
return fmt.Errorf("duplicate metadataGeneration model id %q", model.ID)
}
seen[model.ID] = struct{}{}
}
if len(i.MetadataGeneration.Models) > 0 && i.MetadataGeneration.Default == "" {
return errors.New("metadataGeneration.default is required when models are configured")
}
if i.MetadataGeneration.Default != "" {
if _, exists := seen[i.MetadataGeneration.Default]; !exists {
return fmt.Errorf(
"metadataGeneration.default %q does not identify a configured model",
i.MetadataGeneration.Default,
)
}
}
if len(i.MetadataGeneration.Models) == 0 {
return nil
}
if values["THT_INSTALLATION_CONFIG_SOURCE"] != i.Path {
return errors.New("metadataGeneration requires THT_INSTALLATION_CONFIG_SOURCE to match the installation file")
}
requiresSecrets := false
for _, model := range i.MetadataGeneration.Models {
if model.APIKeyEnv != "" {
requiresSecrets = true
break
}
}
secrets := map[string]string{}
if requiresSecrets {
bundlePath := values["THT_SECRETS_FILE"]
if bundlePath == "" {
return errors.New("metadataGeneration keyed models require THT_SECRETS_FILE")
}
var err error
secrets, err = readMetadataGenerationSecrets(bundlePath)
if err != nil {
return err
}
}
for _, model := range i.MetadataGeneration.Models {
if model.APIKeyEnv == "" {
continue
}
value, exists := secrets[model.APIKeyEnv]
if !exists {
return fmt.Errorf(
"metadataGeneration model %q secret %q is missing from THT_SECRETS_FILE",
model.ID,
model.APIKeyEnv,
)
}
if !usableMetadataGenerationSecret(value) {
return fmt.Errorf(
"metadataGeneration model %q secret %q is unusable",
model.ID,
model.APIKeyEnv,
)
}
}
return nil
}
var metadataModelIDPattern = regexp.MustCompile(`^[a-z][a-z0-9._-]{0,63}$`)
var metadataProviderPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`)
var metadataProviderModelPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$`)
var metadataAPIVersionPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`)
var metadataSecretBundleKeyPattern = regexp.MustCompile(`^[A-Z][A-Z0-9_]{0,127}$`)
var metadataAPIKeyEnvironments = map[string]struct{}{
"THT_METADATA_API_KEY": {},
"ANTHROPIC_API_KEY": {},
"AZURE_API_KEY": {},
"GEMINI_API_KEY": {},
"DEEPSEEK_API_KEY": {},
"OPENAI_API_KEY": {},
"OPENROUTER_API_KEY": {},
"ZAI_API_KEY": {},
}
var metadataSecretBundleKeys = map[string]struct{}{
"THT_MODEL_API_KEY": {},
"THT_DWH_API_KEY": {},
"THT_VEC_API_KEY": {},
"THT_VEC_WRITE_API_KEY": {},
"THT_CA": {},
"THT_SSL_CA": {},
"THT_VECTOR_BOOTSTRAP_PASSWORD": {},
"THT_VECTOR_MIGRATOR_PASSWORD": {},
"THT_VECTOR_READER_PASSWORD": {},
"THT_VECTOR_WRITER_PASSWORD": {},
"PI_PROVIDER_API_KEY": {},
"THT_OIDC_CLIENT_SECRET": {},
"THT_AUTHENTIK_API_TOKEN": {},
"THT_METADATA_API_KEY": {},
"ANTHROPIC_API_KEY": {},
"AZURE_API_KEY": {},
"GEMINI_API_KEY": {},
"DEEPSEEK_API_KEY": {},
"OPENAI_API_KEY": {},
"OPENROUTER_API_KEY": {},
"ZAI_API_KEY": {},
}
func validateMetadataGenerationModel(index int, model MetadataGenerationModel) error {
prefix := fmt.Sprintf("metadataGeneration.models[%d]", index)
if !metadataModelIDPattern.MatchString(model.ID) {
return fmt.Errorf("%s.id is invalid", prefix)
}
if len(model.Label) == 0 || len(model.Label) > 128 || strings.TrimSpace(model.Label) != model.Label ||
strings.IndexFunc(model.Label, unicode.IsControl) >= 0 {
return fmt.Errorf("%s.label is invalid", prefix)
}
if !metadataProviderPattern.MatchString(model.LiteLLM.Provider) {
return fmt.Errorf("%s.litellm.provider is invalid", prefix)
}
if !metadataProviderModelPattern.MatchString(model.LiteLLM.Model) {
return fmt.Errorf("%s.litellm.model is invalid", prefix)
}
if model.APIKeyEnv == "" {
if model.LiteLLM.Endpoint == nil {
return fmt.Errorf("%s.apiKeyEnv is required unless an explicit keyless endpoint is configured", prefix)
}
} else {
if !metadataSecretBundleKeyPattern.MatchString(model.APIKeyEnv) {
return fmt.Errorf("%s.apiKeyEnv is invalid", prefix)
}
if _, allowed := metadataAPIKeyEnvironments[model.APIKeyEnv]; !allowed {
return fmt.Errorf("%s.apiKeyEnv is invalid", prefix)
}
}
if endpoint := model.LiteLLM.Endpoint; endpoint != nil {
parsed, err := url.Parse(endpoint.BaseURL)
if err != nil || strings.TrimSpace(endpoint.BaseURL) != endpoint.BaseURL ||
(parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Hostname() == "" ||
parsed.User != nil || parsed.RawQuery != "" || parsed.Fragment != "" {
return fmt.Errorf("%s.litellm.endpoint.baseUrl is invalid", prefix)
}
if endpoint.APIVersion != "" && !metadataAPIVersionPattern.MatchString(endpoint.APIVersion) {
return fmt.Errorf("%s.litellm.endpoint.apiVersion is invalid", prefix)
}
}
if model.LiteLLM.DisableThinking && model.LiteLLM.Endpoint == nil {
return fmt.Errorf("%s.litellm.disableThinking requires an explicit endpoint", prefix)
}
return nil
}
func readMetadataGenerationSecrets(path string) (map[string]string, error) {
contents, err := safeio.ReadCanonicalPrivateRegular(path, maxMetadataSecretBundleBytes)
if err != nil {
return nil, errors.New("metadataGeneration secrets in THT_SECRETS_FILE are unavailable")
}
values := make(map[string]string)
for _, raw := range strings.Split(string(contents), "\n") {
line := strings.TrimSuffix(raw, "\r")
if strings.TrimSpace(line) == "" || strings.HasPrefix(strings.TrimSpace(line), "#") {
continue
}
key, value, found := strings.Cut(line, "=")
if !found || !metadataSecretBundleKeyPattern.MatchString(key) {
return nil, errors.New("metadataGeneration secrets in THT_SECRETS_FILE are invalid")
}
if _, allowed := metadataSecretBundleKeys[key]; !allowed {
return nil, errors.New("metadataGeneration secrets in THT_SECRETS_FILE are invalid")
}
if _, duplicate := values[key]; duplicate {
return nil, errors.New("metadataGeneration secrets in THT_SECRETS_FILE contain duplicate keys")
}
values[key] = value
}
return values, nil
}
func usableMetadataGenerationSecret(value string) bool {
return len(value) > 0 && len(value) <= 16*1024 && strings.TrimSpace(value) == value &&
strings.IndexFunc(value, func(character rune) bool {
return unicode.IsSpace(character) || unicode.IsControl(character)
}) < 0
}
// AuthenticationDirectory returns the descriptor-owned, non-secret authentication root.
func (i Installation) AuthenticationDirectory() string { return i.Authentication.ConfigDirectory }
@@ -0,0 +1,454 @@
package config
import (
"fmt"
"os"
"path/filepath"
"runtime"
"strconv"
"strings"
"testing"
)
func TestLoadAcceptsMetadataGenerationModels(t *testing.T) {
installationPath, _, envFile, _ := writeInstallation(t, "local")
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o600); err != nil {
t.Fatal(err)
}
appendFile(t, envFile, strings.Join([]string{
"THT_SECRETS_FILE=" + strconv.Quote(secretBundle),
"THT_INSTALLATION_CONFIG_SOURCE=" + strconv.Quote(installationPath),
}, "\n")+"\n")
appendFile(t, installationPath, `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm:
provider: openai
model: gpt-4.1-mini
endpoint:
baseUrl: https://api.openai.example/v1
apiVersion: "2026-08-01"
apiKeyEnv: OPENAI_API_KEY
`)
installation, err := Load(installationPath)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if installation.MetadataGeneration.Default != "openai-mini" {
t.Fatalf("metadata default = %q", installation.MetadataGeneration.Default)
}
if len(installation.MetadataGeneration.Models) != 1 {
t.Fatalf("metadata models = %#v", installation.MetadataGeneration.Models)
}
model := installation.MetadataGeneration.Models[0]
if model.ID != "openai-mini" || model.Label != "OpenAI Mini" ||
model.LiteLLM.Provider != "openai" || model.LiteLLM.Model != "gpt-4.1-mini" ||
model.LiteLLM.Endpoint == nil || model.LiteLLM.Endpoint.BaseURL != "https://api.openai.example/v1" ||
model.LiteLLM.Endpoint.APIVersion != "2026-08-01" || model.APIKeyEnv != "OPENAI_API_KEY" {
t.Fatalf("metadata model = %#v", model)
}
if strings.Contains(strings.TrimSpace(model.APIKeyEnv), "provider-secret") {
t.Fatal("installation model exposed the credential value")
}
}
func TestLoadAcceptsMixedKeyedAndKeylessMetadataGenerationModels(t *testing.T) {
installationPath, _, envFile, _ := writeInstallation(t, "local")
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
if err := os.WriteFile(
secretBundle,
[]byte("DEEPSEEK_API_KEY=deepseek-secret\nZAI_API_KEY=zai-secret\n"),
0o600,
); err != nil {
t.Fatal(err)
}
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
appendFile(t, installationPath, `metadataGeneration:
default: glm-53
models:
- id: deepseek-v4-pro
label: DeepSeek V4 Pro
litellm: {provider: deepseek, model: deepseek-v4-pro}
apiKeyEnv: DEEPSEEK_API_KEY
- id: glm-53
label: GLM 5.3
litellm:
provider: openai
model: glm-5.3
endpoint: {baseUrl: https://api.z.ai/api/coding/paas/v4}
apiKeyEnv: ZAI_API_KEY
- id: qwen-36
label: Qwen 3.6
litellm:
provider: openai
model: qwen3.6-35b-a3b
disableThinking: true
endpoint: {baseUrl: https://models.internal.example/v1}
`)
installation, err := Load(installationPath)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
if len(installation.MetadataGeneration.Models) != 3 {
t.Fatalf("metadata models = %#v", installation.MetadataGeneration.Models)
}
qwen := installation.MetadataGeneration.Models[2]
if qwen.APIKeyEnv != "" || !qwen.LiteLLM.DisableThinking || qwen.LiteLLM.Endpoint == nil {
t.Fatalf("keyless qwen model = %#v", qwen)
}
}
func TestLoadAcceptsOnlyExplicitKeylessMetadataGenerationModelWithoutBundle(t *testing.T) {
installationPath, _, envFile, _ := writeInstallation(t, "local")
appendFile(t, envFile, "THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
appendFile(t, installationPath, `metadataGeneration:
default: qwen-36
models:
- id: qwen-36
label: Qwen 3.6
litellm:
provider: openai
model: qwen3.6-35b-a3b
disableThinking: true
endpoint: {baseUrl: https://models.internal.example/v1}
`)
if _, err := Load(installationPath); err != nil {
t.Fatalf("Load() error = %v", err)
}
}
func TestLoadRejectsTooManyMetadataGenerationModels(t *testing.T) {
installationPath, _, _, _ := writeInstallation(t, "local")
var configuration strings.Builder
configuration.WriteString("metadataGeneration:\n default: model-0\n models:\n")
for index := 0; index <= maxMetadataGenerationModels; index++ {
fmt.Fprintf(&configuration, ` - id: model-%d
label: Model %d
litellm: {provider: openai, model: gpt-4.1-mini}
apiKeyEnv: OPENAI_API_KEY
`, index, index)
}
appendFile(t, installationPath, configuration.String())
_, err := Load(installationPath)
want := fmt.Sprintf(
"metadataGeneration.models must contain at most %d entries",
maxMetadataGenerationModels,
)
if err == nil || !strings.Contains(err.Error(), want) {
t.Fatalf("Load() error = %v, want %q", err, want)
}
}
func TestLoadRejectsDuplicateMetadataGenerationModelIDs(t *testing.T) {
installationPath, _, envFile, _ := writeInstallation(t, "local")
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o600); err != nil {
t.Fatal(err)
}
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
appendFile(t, installationPath, `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm: {provider: openai, model: gpt-4.1-mini}
apiKeyEnv: OPENAI_API_KEY
- id: openai-mini
label: Duplicate
litellm: {provider: openai, model: gpt-4.1}
apiKeyEnv: OPENAI_API_KEY
`)
_, err := Load(installationPath)
if err == nil || !strings.Contains(err.Error(), `duplicate metadataGeneration model id "openai-mini"`) {
t.Fatalf("Load() error = %v, want actionable duplicate-id error", err)
}
}
func TestLoadRejectsMissingOrUnknownMetadataGenerationDefault(t *testing.T) {
for _, test := range []struct {
name string
defaultYAML string
want string
}{
{
name: "missing",
want: "metadataGeneration.default is required when models are configured",
},
{
name: "unknown",
defaultYAML: " default: unavailable\n",
want: `metadataGeneration.default "unavailable" does not identify a configured model`,
},
} {
t.Run(test.name, func(t *testing.T) {
installationPath, _, envFile, _ := writeInstallation(t, "local")
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o600); err != nil {
t.Fatal(err)
}
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
appendFile(t, installationPath, "metadataGeneration:\n"+test.defaultYAML+` models:
- id: openai-mini
label: OpenAI Mini
litellm: {provider: openai, model: gpt-4.1-mini}
apiKeyEnv: OPENAI_API_KEY
`)
_, err := Load(installationPath)
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Load() error = %v, want %q", err, test.want)
}
})
}
}
func TestLoadRejectsMalformedMetadataGenerationModelSettings(t *testing.T) {
validPrefix := ` - id: openai-mini
label: OpenAI Mini
litellm:
`
for _, test := range []struct {
name string
model string
want string
}{
{
name: "unstable id",
model: strings.Replace(validPrefix, "openai-mini", "OpenAI Mini", 1) +
" provider: openai\n model: gpt-4.1-mini\n apiKeyEnv: OPENAI_API_KEY\n",
want: "metadataGeneration.models[0].id is invalid",
},
{
name: "blank label",
model: strings.Replace(validPrefix, "OpenAI Mini", `" "`, 1) +
" provider: openai\n model: gpt-4.1-mini\n apiKeyEnv: OPENAI_API_KEY\n",
want: "metadataGeneration.models[0].label is invalid",
},
{
name: "provider",
model: validPrefix + " provider: open ai\n model: gpt-4.1-mini\n apiKeyEnv: OPENAI_API_KEY\n",
want: "metadataGeneration.models[0].litellm.provider is invalid",
},
{
name: "model",
model: validPrefix + " provider: openai\n model: \" gpt-4.1-mini\"\n apiKeyEnv: OPENAI_API_KEY\n",
want: "metadataGeneration.models[0].litellm.model is invalid",
},
{
name: "endpoint",
model: validPrefix + " provider: openai\n model: gpt-4.1-mini\n" +
" endpoint:\n baseUrl: https://operator@api.example/v1\n apiKeyEnv: OPENAI_API_KEY\n",
want: "metadataGeneration.models[0].litellm.endpoint.baseUrl is invalid",
},
{
name: "api version",
model: validPrefix + " provider: openai\n model: gpt-4.1-mini\n" +
" endpoint:\n baseUrl: https://api.example/v1\n apiVersion: \"bad version\"\n apiKeyEnv: OPENAI_API_KEY\n",
want: "metadataGeneration.models[0].litellm.endpoint.apiVersion is invalid",
},
} {
t.Run(test.name, func(t *testing.T) {
installationPath, _, envFile, _ := writeInstallation(t, "local")
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o600); err != nil {
t.Fatal(err)
}
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
appendFile(t, installationPath, "metadataGeneration:\n default: openai-mini\n models:\n"+test.model)
_, err := Load(installationPath)
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Load() error = %v, want %q", err, test.want)
}
})
}
}
func TestLoadRejectsMissingOrUnusableMetadataGenerationSecrets(t *testing.T) {
for _, test := range []struct {
name string
apiKeyEnvYAML string
bundle string
declareBundle bool
want string
}{
{
name: "missing reference",
apiKeyEnvYAML: "",
bundle: "OPENAI_API_KEY=provider-secret\n",
declareBundle: true,
want: "metadataGeneration.models[0].apiKeyEnv is required unless an explicit keyless endpoint is configured",
},
{
name: "malformed reference",
apiKeyEnvYAML: " apiKeyEnv: openai-api-key\n",
bundle: "OPENAI_API_KEY=provider-secret\n",
declareBundle: true,
want: "metadataGeneration.models[0].apiKeyEnv is invalid",
},
{
name: "unallowed reference",
apiKeyEnvYAML: " apiKeyEnv: THT_DWH_API_KEY\n",
bundle: "THT_DWH_API_KEY=dwh-secret\n",
declareBundle: true,
want: "metadataGeneration.models[0].apiKeyEnv is invalid",
},
{
name: "bundle not declared",
apiKeyEnvYAML: " apiKeyEnv: OPENAI_API_KEY\n",
bundle: "OPENAI_API_KEY=provider-secret\n",
want: "metadataGeneration keyed models require THT_SECRETS_FILE",
},
{
name: "reference absent from bundle",
apiKeyEnvYAML: " apiKeyEnv: OPENAI_API_KEY\n",
bundle: "THT_DWH_API_KEY=dwh-secret\n",
declareBundle: true,
want: "metadataGeneration model \"openai-mini\" secret \"OPENAI_API_KEY\" is missing",
},
{
name: "unusable value",
apiKeyEnvYAML: " apiKeyEnv: OPENAI_API_KEY\n",
bundle: "OPENAI_API_KEY=secret with whitespace\n",
declareBundle: true,
want: "metadataGeneration model \"openai-mini\" secret \"OPENAI_API_KEY\" is unusable",
},
} {
t.Run(test.name, func(t *testing.T) {
installationPath, _, envFile, _ := writeInstallation(t, "local")
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
if err := os.WriteFile(secretBundle, []byte(test.bundle), 0o600); err != nil {
t.Fatal(err)
}
environment := "THT_INSTALLATION_CONFIG_SOURCE=" + strconv.Quote(installationPath) + "\n"
if test.declareBundle {
environment += "THT_SECRETS_FILE=" + strconv.Quote(secretBundle) + "\n"
}
appendFile(t, envFile, environment)
appendFile(t, installationPath, `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm: {provider: openai, model: gpt-4.1-mini}
`+test.apiKeyEnvYAML)
_, err := Load(installationPath)
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Load() error = %v, want %q", err, test.want)
}
if strings.Contains(strings.ToLower(err.Error()), "secret with whitespace") ||
strings.Contains(strings.ToLower(err.Error()), "provider-secret") {
t.Fatalf("Load() exposed secret content: %v", err)
}
})
}
}
func TestLoadRejectsUnprotectedMetadataGenerationSecretBundle(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("POSIX mode assertion; Windows ACL coverage lives in safeio")
}
installationPath, _, envFile, _ := writeInstallation(t, "local")
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o644); err != nil {
t.Fatal(err)
}
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
appendFile(t, installationPath, `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm: {provider: openai, model: gpt-4.1-mini}
apiKeyEnv: OPENAI_API_KEY
`)
_, err := Load(installationPath)
if err == nil || !strings.Contains(err.Error(), "metadataGeneration secrets in THT_SECRETS_FILE are unavailable") {
t.Fatalf("Load() error = %v, want protected-bundle error", err)
}
}
func TestLoadRejectsUnknownMetadataGenerationSecretBundleKeys(t *testing.T) {
installationPath, _, envFile, _ := writeInstallation(t, "local")
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
if err := os.WriteFile(
secretBundle,
[]byte("OPENAI_API_KEY=provider-secret\nUNRECOGNIZED_API_KEY=unknown-secret\n"),
0o600,
); err != nil {
t.Fatal(err)
}
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
appendFile(t, installationPath, `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm: {provider: openai, model: gpt-4.1-mini}
apiKeyEnv: OPENAI_API_KEY
`)
_, err := Load(installationPath)
if err == nil || !strings.Contains(err.Error(), "metadataGeneration secrets in THT_SECRETS_FILE are invalid") {
t.Fatalf("Load() error = %v, want invalid-bundle error", err)
}
if strings.Contains(err.Error(), "UNRECOGNIZED_API_KEY") || strings.Contains(err.Error(), "unknown-secret") {
t.Fatalf("Load() exposed rejected bundle content: %v", err)
}
}
func TestLoadRequiresConfiguredMetadataGenerationToUseTheSameMountedDescriptor(t *testing.T) {
for _, configuredSource := range []string{"", "/different/thothii-installation.yaml"} {
installationPath, _, envFile, _ := writeInstallation(t, "local")
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o600); err != nil {
t.Fatal(err)
}
environment := "THT_SECRETS_FILE=" + strconv.Quote(secretBundle) + "\n"
if configuredSource != "" {
environment += "THT_INSTALLATION_CONFIG_SOURCE=" + strconv.Quote(configuredSource) + "\n"
}
appendFile(t, envFile, environment)
appendFile(t, installationPath, `metadataGeneration:
default: openai-mini
models:
- id: openai-mini
label: OpenAI Mini
litellm: {provider: openai, model: gpt-4.1-mini}
apiKeyEnv: OPENAI_API_KEY
`)
_, err := Load(installationPath)
if err == nil || !strings.Contains(err.Error(), "metadataGeneration requires THT_INSTALLATION_CONFIG_SOURCE to match the installation file") {
t.Fatalf("Load() source %q error = %v", configuredSource, err)
}
}
}
func appendFile(t *testing.T, path, contents string) {
t.Helper()
file, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0)
if err != nil {
t.Fatal(err)
}
defer file.Close()
if _, err := file.WriteString(contents); err != nil {
t.Fatal(err)
}
}
+1
View File
@@ -343,6 +343,7 @@ func render(root, descriptorPath string, value answers) ([]byte, []byte, error)
"THT_WORKSPACE_GIT_BRANCH=" + dotenvValue(value.workspaceBranch),
"THT_WORKSPACE_INSTALLATION_ID=" + dotenvValue(value.installationID),
"THT_AUTH_CONFIG_ROOT=" + dotenvValue(descriptor.Authentication.ConfigDirectory),
"THT_INSTALLATION_CONFIG_SOURCE=" + dotenvValue(descriptorPath),
"THT_SECRETS_FILE=" + dotenvValue(value.secretsFile),
"PI_AUTH_FILE=" + dotenvValue(value.piAuthFile),
"THOTH_HTTP_PORT=8080", "THOTH_CORE_HTTP_PORT=8787", "MAX_PI_PROCESSES=4",
@@ -0,0 +1,31 @@
package setup
import (
"os"
"strconv"
"strings"
"testing"
)
func TestEnsureFilesProjectsItsInstallationYAMLToTheApplication(t *testing.T) {
root := newProject(t, "metadata generation config")
setNonInteractiveAnswers(t, newExternalSecrets(t, root))
result, err := EnsureFiles(Request{
ProjectRoot: root, InstallationID: "metadata", Profile: "local", NonInteractive: true,
}, strings.NewReader(""), ioDiscard{})
if err != nil {
t.Fatal(err)
}
environment, err := os.ReadFile(result.EnvironmentPath)
if err != nil {
t.Fatal(err)
}
want := "THT_INSTALLATION_CONFIG_SOURCE=" + strconv.Quote(result.DescriptorPath)
if !strings.Contains(string(environment), want) {
t.Fatalf("generated environment does not project installation YAML: want %q", want)
}
if strings.Contains(string(environment), "PI_MODEL") || strings.Contains(string(environment), "llm_policy") {
t.Fatal("metadata-generation projection was coupled to Pi or workspace policy")
}
}