feat: add AI catalog description generation
This commit is contained in:
@@ -76,12 +76,14 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
|
||||
There is no verbatim transcript store. A resumed Pi process rebuilds context from
|
||||
`tht session show <id>` + the on-disk artifacts.
|
||||
|
||||
- **The backend is a thin bridge with no database.** `ThtRunner` shells the Python workflow `tht`
|
||||
subcommands inside `core`;
|
||||
- **The backend bridges sessions and owns the installation-local metadata catalog.** `ThtRunner`
|
||||
shells the Python workflow `tht` subcommands inside `core`;
|
||||
`PiProcessManager` runs one Pi child per session and bridges its RPC stream;
|
||||
`SessionBridge` maps Pi RPC events → client events (`ui_request`/`text_delta`/`info`);
|
||||
`SseHub` fans them out over SSE to the browser. App settings live in a JSON file
|
||||
(`backend/data/settings.json`), not a DB.
|
||||
`SseHub` fans them out over SSE to the browser. The separate PostgreSQL catalog stores database
|
||||
metadata and sequential AI description-generation runs. Description generation samples the DWH
|
||||
through read-only connectors and calls a short-lived Python LiteLLM helper; it does not use Pi or
|
||||
expose a public CLI command. App settings still live in `backend/data/settings.json`.
|
||||
|
||||
- **Human-in-the-loop gate contract.** The model proposes; a human reviewer decides at gates
|
||||
via widgets (`reviewer_select` = single pick — a chosen option carrying a `decision` payload
|
||||
|
||||
+40
@@ -306,6 +306,11 @@ per gli usi downstream.
|
||||
essere consolidata come Description. Rimane distinta dal commento osservato nel database.
|
||||
_Avoid_: generated comment, source comment
|
||||
|
||||
**Description Consolidation** — L'azione amministrativa esplicita che copia la Generated
|
||||
Description di Catalog Table o Catalog Column selezionate nella relativa Description. Opera sulla
|
||||
selezione corrente, conserva la Generated Description e non modifica il commento osservato o il
|
||||
database esterno.
|
||||
|
||||
**Table Synchronization** — La riconciliazione esplicita che rende le Catalog Table di un
|
||||
Workspace Database uguali alle Physical Table osservate: crea quelle nuove, aggiorna i metadati
|
||||
di origine ed elimina definitivamente quelle assenti. Non modifica mai il database esterno.
|
||||
@@ -318,6 +323,23 @@ un unico snapshot completo tramite Synchronize All.
|
||||
**Catalog Sync Run** — L'esecuzione durevole in background di una Schema Synchronization, con
|
||||
scope, stato, avanzamento e log propri. Al massimo un run per Workspace Database può essere attivo.
|
||||
|
||||
**Description Generation Run** — L'esecuzione asincrona e sequenziale che usa il modello scelto
|
||||
per produrre Generated Description di Catalog Table o Catalog Column. Al massimo una run è attiva
|
||||
nell'intera installazione e ogni risultato valido viene salvato appena disponibile. Dopo
|
||||
un'interruzione il recupero è manuale tramite una nuova generazione dei soli elementi mancanti.
|
||||
|
||||
**Description Generation Event** — Una riga testuale ordinata che registra avanzamento, risultato
|
||||
o errore di una Description Generation Run e alimenta il log visibile all'amministratore.
|
||||
|
||||
**Non-generatable Description** — L'esito valido con cui il modello dichiara di non disporre di
|
||||
informazioni sufficienti per descrivere il target. Produce una Generated Description standard
|
||||
nella lingua del workspace e non rappresenta un timeout, un errore del provider o una risposta
|
||||
non valida.
|
||||
|
||||
**Description Generation Unlock** — Il recupero amministrativo che marca come interrotta una
|
||||
Description Generation Run registrata come attiva quando il backend non ha alcun processo di
|
||||
generazione vivo. Non è un meccanismo di lock distribuito.
|
||||
|
||||
**Catalog Metadata Cleanup** — La rimozione amministrativa esplicita di Catalog Table, Catalog
|
||||
Column o Catalog Relationship selezionate. Non modifica il Workspace Database, la Database Binding
|
||||
o i segreti, e può lasciare il Metadata Catalog intenzionalmente incompleto fino alla prossima
|
||||
@@ -331,6 +353,24 @@ con la binding corrente.
|
||||
distinti dai fatti strutturali governati dalla sincronizzazione. Possono essere popolati dall'AI,
|
||||
da un'importazione o da una modifica amministrativa senza cambiare il database esterno.
|
||||
|
||||
**Metadata Generation Model Configuration** — La configurazione a livello di setup applicativo
|
||||
che elenca i modelli selezionabili, il default e i riferimenti agli eventuali segreti per la sola
|
||||
generazione dei metadati. Un modello keyless è ammesso solo con un endpoint esplicito che non
|
||||
richiede autenticazione. Non appartiene al workspace ed è indipendente dalla configurazione Pi.
|
||||
|
||||
**Model Completion Helper** — Il processo Python interno ed effimero che esegue una singola
|
||||
richiesta LiteLLM per conto del backend. Non è un servizio HTTP, non possiede il lifecycle della
|
||||
Description Generation Run e non è una CLI esposta agli utenti.
|
||||
|
||||
**Catalog Sample** — Un input transitorio composto da un massimo di cinque righe reali e dai
|
||||
valori di esempio bounded letti da una Catalog Table per la generazione delle descrizioni. È
|
||||
trattato come dato non fidato, non viene persistito e non diventa Catalog Metadata.
|
||||
|
||||
**Sensitive Data Policy** — L'insieme di regole che classifica i valori sorgente protetti per
|
||||
l'uso nei processi AI e ne prescrive l'esclusione o l'anonimizzazione. Non coincide con la sola
|
||||
classificazione dei dati personali.
|
||||
_Avoid_: PII filter, sample filter
|
||||
|
||||
**Introspection Capability** — Una categoria di struttura fisica che una Database Binding
|
||||
può osservare, come tabelle, colonne, relazioni, indici o enum. Una capability non disponibile
|
||||
è distinta da una capability osservata che non ha restituito elementi.
|
||||
|
||||
+34
-4
@@ -79,7 +79,8 @@ hand, but administrators can explicitly clear catalog tables, columns, or relati
|
||||
touching the source database, binding, configuration, or secrets. Table deletion cascades through
|
||||
columns and relationships; table-scoped relationship cleanup includes incoming and outgoing
|
||||
relationships. Curated and generated descriptions are editable; generated descriptions start null
|
||||
and AI generation/consolidation is deferred.
|
||||
and Database Management can generate or consolidate them for selected tables, selected columns,
|
||||
all targets, or only targets whose Generated Description is missing.
|
||||
|
||||
Schema refresh is one durable asynchronous engine with database-table, database-column,
|
||||
selected-table-column, relationship, and full-database actions. Database-level menus expose the
|
||||
@@ -103,10 +104,36 @@ one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before loca
|
||||
sessions still consume the existing workspace configuration in this slice: database-management
|
||||
records do not yet change the NL→SQL handoff. The accepted design is recorded in
|
||||
`docs/plans/2026-08-26-metadata-catalog-from-thothai.md`, the snapshot contract under
|
||||
`docs/contracts/`, and ADRs 0001–0008.
|
||||
`docs/contracts/`, and ADRs 0001–0010.
|
||||
|
||||
Semantic aliases, value descriptions, synonyms, concepts, AI metadata generation/consolidation,
|
||||
and logical relationships remain deferred to their dedicated slices.
|
||||
Semantic aliases, value descriptions, synonyms, concepts, and logical relationships remain
|
||||
deferred to their dedicated slices.
|
||||
|
||||
AI Description Generation preserves ThothAI's use of real source samples: up to five source rows
|
||||
and five representative non-null example values may be sent transiently to the configured model
|
||||
provider. The UI and operator documentation disclose this behavior. A required follow-up
|
||||
improvement is a Sensitive Data Policy that classifies protected fields and excludes or anonymizes
|
||||
their values before model calls.
|
||||
|
||||
The accepted AI-description design is recorded in
|
||||
`docs/plans/2026-08-28-ai-catalog-description-generation.md`, with the formal specification in the
|
||||
adjacent `-spec.md` document and Gitea issue #4. Gitea issues #5–#11 deliver the implementation.
|
||||
The runtime deliberately keeps ThothAI's simple operating model: one installation-wide sequential
|
||||
run owned by the backend, one short-lived Python/LiteLLM completion helper per request, and
|
||||
persistence limited to the run, its safe ordered text events, and each Generated Description as
|
||||
soon as it succeeds. The helper performs at most one provider retry and never falls back to another
|
||||
model. Stop terminates the current helper and retains prior results; three consecutive exhausted
|
||||
technical batches fail the run. Startup marks stale queued/running work interrupted, and Unlock is
|
||||
available only when no local start, worker, or helper is live. Runs remain inspectable through a
|
||||
live SSE log with ordered polling fallback; there is no automatic resume or user-facing generation
|
||||
CLI. ADRs 0009–0010 record the runtime and source-sampling decisions.
|
||||
|
||||
Metadata-generation setup accepts the protected `DEEPSEEK_API_KEY` and `ZAI_API_KEY` references.
|
||||
It also accepts a model with no secret reference only when its OpenAI-compatible endpoint is
|
||||
explicit; this covers the VPN-only AritmoLab Qwen 3.6 server without creating a fake operator
|
||||
credential. The Python client supplies only its fixed non-secret compatibility placeholder.
|
||||
The AritmoLab entry also sets `disableThinking: true`, mapped to the endpoint's chat-template flag,
|
||||
because its default reasoning prose would violate the worker's exact JSON response contract.
|
||||
|
||||
Integration of the completed metadata catalog with core schema-linking is explicitly deferred
|
||||
until the database, table, column, relationship, and synchronization slices are complete. At that
|
||||
@@ -159,6 +186,9 @@ from an automated PASS.
|
||||
- The P1.1 workspace-directory registry and P2–P6 preprocessing workstreams are implemented and
|
||||
have automated coverage.
|
||||
- Evidence restructuring has a real PSD acceptance PASS as recorded above.
|
||||
- AI Description Generation has automated coverage across installation setup, model selection,
|
||||
generation/consolidation scopes, bounded sampling, cancellation/recovery, history, SSE/polling,
|
||||
and the LiteLLM helper boundary.
|
||||
- L2 tests requiring real providers or remote databases remain opt-in.
|
||||
- Server deployment, release, and owner-operated acceptance steps remain pending wherever the
|
||||
referenced runbooks require explicit approval.
|
||||
|
||||
@@ -17,6 +17,8 @@ From a fresh clone, run these commands from the repository root:
|
||||
|
||||
```sh
|
||||
cp deploy/env/local.env.example deploy/env/local.env
|
||||
# Copy docs/install/examples/thothii-installation.local.yaml to a protected operator path,
|
||||
# replace its placeholders, chmod it 600, and set that exact THT_INSTALLATION_CONFIG_SOURCE.
|
||||
# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
|
||||
./scripts/run-stack.sh
|
||||
```
|
||||
@@ -29,7 +31,8 @@ application rollout:
|
||||
|
||||
```sh
|
||||
cp deploy/env/server.env.example deploy/env/server.env
|
||||
# Edit all absolute storage, Pi/secret/session files, and endpoint paths.
|
||||
# Prepare a mode-600 thothii-installation.yaml from the server example and set its exact
|
||||
# path as THT_INSTALLATION_CONFIG_SOURCE. Edit all remaining storage/secret/endpoint paths.
|
||||
sudo scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001
|
||||
docker compose --env-file deploy/env/server.env \
|
||||
-f compose.yaml -f deploy/compose.server.yaml \
|
||||
@@ -293,6 +296,31 @@ Copy `deploy/secrets/thothii.secrets.example` to a protected host file, include
|
||||
keys, and set its absolute path as `THT_SECRETS_FILE` in the operator env. Keep Pi's native
|
||||
provider auth in the separate protected file named by `PI_AUTH_FILE`.
|
||||
|
||||
Description Generation is configured independently in the protected installation descriptor under
|
||||
`metadataGeneration`. Set `THT_INSTALLATION_CONFIG_SOURCE` to that exact host file; Compose mounts
|
||||
it read-only into `core` and supplies the fixed runtime `THT_INSTALLATION_CONFIG_FILE` path. Each
|
||||
keyed model stores only an audited `apiKeyEnv` reference. The referenced value stays in the secret
|
||||
bundle; a model may omit `apiKeyEnv` only when it declares an explicit endpoint that accepts
|
||||
unauthenticated requests. The browser receives only model IDs, labels, and the configured default.
|
||||
Configuration changes take effect after restart and do not use Pi settings or workspace
|
||||
`llm_policy`.
|
||||
|
||||
Before enabling Description Generation, approve the selected model provider for bounded source-data
|
||||
disclosure. A request may send up to five real source rows and up to five representative distinct,
|
||||
non-null example values for relevant columns. Samples are transient and are not stored in generation
|
||||
runs, run logs, application logs, API responses, or catalog metadata; prompt and sample snapshots are
|
||||
not retained. Automated Sensitive Data Policy filtering and anonymization are not currently provided.
|
||||
A future Sensitive Data Policy is required to classify protected fields and exclude or anonymize
|
||||
their values before model calls.
|
||||
|
||||
Description Generation is an interactive Database Management operation, not a user-facing CLI.
|
||||
The installation runs at most one sequential generation at a time. The run drawer exposes safe
|
||||
ordered events through SSE with polling fallback, Stop terminates the current helper while keeping
|
||||
already stored results, and Run history retains terminal runs for inspection. A backend restart
|
||||
marks queued or running work interrupted instead of resuming it; use Generate Missing to continue.
|
||||
Unlock is reserved for a stale recorded run and is rejected while a local start, worker, or helper
|
||||
is still live.
|
||||
|
||||
The bundle is mounted read-only as `/run/secrets/thothii.secrets` and must be mode `0600` or
|
||||
`0400` on the host. Docker's runtime `0444` mode is accepted only beneath `/run/secrets`; see
|
||||
[`deploy/secrets/README.md`](deploy/secrets/README.md). A PEM CA chain is deliberately not a
|
||||
|
||||
+51
-2
@@ -2,7 +2,7 @@ import Fastify, { type FastifyInstance, type FastifyRequest } from "fastify";
|
||||
import cors from "@fastify/cors";
|
||||
import cookie from "@fastify/cookie";
|
||||
import rateLimit from "@fastify/rate-limit";
|
||||
import { join } from "node:path";
|
||||
import { dirname, isAbsolute, join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { AppConfig } from "./config.js";
|
||||
import { ThtRunner } from "./tht/tht-runner.js";
|
||||
@@ -48,6 +48,19 @@ import { catalogTableRoutes } from "./routes/catalog-tables.js";
|
||||
import { ConcreteCatalogSchemaIntrospector, type CatalogSchemaIntrospector } from "./catalog/schema-introspector.js";
|
||||
import { CatalogSyncWorker } from "./catalog/sync-worker.js";
|
||||
import { catalogSchemaRoutes } from "./routes/catalog-schema.js";
|
||||
import {
|
||||
loadMetadataGenerationModels,
|
||||
type MetadataGenerationModels,
|
||||
} from "./catalog/metadata-generation-models.js";
|
||||
import { metadataGenerationModelRoutes } from "./routes/metadata-generation-models.js";
|
||||
import { catalogDescriptionConsolidationRoutes } from "./routes/catalog-description-consolidation.js";
|
||||
import { PythonModelCompleter, type ModelCompleter } from "./catalog/model-completer.js";
|
||||
import { DescriptionGenerationWorker } from "./catalog/description-generation-worker.js";
|
||||
import {
|
||||
PostgresDescriptionSourceSampler,
|
||||
type DescriptionSourceSampler,
|
||||
} from "./catalog/description-source-sampler.js";
|
||||
import { catalogDescriptionGenerationRoutes } from "./routes/catalog-description-generation.js";
|
||||
|
||||
export interface BuildAppDeps {
|
||||
thtRunner?: ThtRunner;
|
||||
@@ -67,6 +80,9 @@ export interface BuildAppDeps {
|
||||
catalogSchemaIntrospector?: CatalogSchemaIntrospector;
|
||||
catalogSyncWorker?: CatalogSyncWorker;
|
||||
catalogOperationCoordinator?: CatalogOperationCoordinator;
|
||||
metadataGenerationModels?: MetadataGenerationModels;
|
||||
modelCompleter?: ModelCompleter;
|
||||
descriptionSourceSampler?: DescriptionSourceSampler;
|
||||
workspaceRuntimeSupport?: (workspace: WorkspaceDescriptor) => boolean;
|
||||
maintenanceBarrier?: MaintenanceBarrier;
|
||||
piManagement?: PiManagementService;
|
||||
@@ -141,10 +157,28 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
|
||||
const workspaceRegistry = deps?.workspaceRegistry ?? new WorkspaceRegistry(config.workspaceRegistry);
|
||||
const catalogRepository = deps?.catalogRepository ?? createCatalogRepository(config.catalogDatabase);
|
||||
const catalogOperationCoordinator = deps?.catalogOperationCoordinator ?? new CatalogOperationCoordinator();
|
||||
const metadataGenerationModels = deps?.metadataGenerationModels ?? loadMetadataGenerationModels({
|
||||
installationFile: config.installationConfigFile,
|
||||
secretsFile: config.secretsFile,
|
||||
});
|
||||
const modelCompleter = deps?.modelCompleter ?? new PythonModelCompleter({
|
||||
pythonExecutable: isAbsolute(config.thtBin) ? join(dirname(config.thtBin), "python") : "python3",
|
||||
cwd: config.harnessDir,
|
||||
});
|
||||
const catalogPostgresAccess = deps?.catalogPostgresAccess ?? new ConcreteCatalogPostgresAccess(
|
||||
workspaceSecretStore,
|
||||
{ connectTimeoutMs: config.workspaceDiagnosticTimeoutMs },
|
||||
);
|
||||
const descriptionSourceSampler = deps?.descriptionSourceSampler
|
||||
?? new PostgresDescriptionSourceSampler(catalogPostgresAccess);
|
||||
const descriptionGenerationWorker = new DescriptionGenerationWorker(
|
||||
catalogRepository,
|
||||
workspaceRegistry,
|
||||
metadataGenerationModels,
|
||||
modelCompleter,
|
||||
catalogOperationCoordinator,
|
||||
descriptionSourceSampler,
|
||||
);
|
||||
const catalogService = deps?.catalogService ?? new CatalogService(
|
||||
catalogRepository,
|
||||
workspaceRegistry,
|
||||
@@ -166,10 +200,12 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
|
||||
config.catalogSyncTimeoutMs,
|
||||
);
|
||||
app.addHook("onReady", async () => { await catalogSyncWorker.initialize(); });
|
||||
app.addHook("onReady", async () => { await descriptionGenerationWorker.initialize(); });
|
||||
if (!deps?.catalogRepository && catalogRepository.close) {
|
||||
app.addHook("onClose", async () => { await catalogRepository.close?.(); });
|
||||
}
|
||||
app.addHook("onClose", async () => { await catalogSyncWorker.stop(); });
|
||||
app.addHook("onClose", async () => { await descriptionGenerationWorker.stop(); });
|
||||
const workspaceDiagnoser = deps?.workspaceDiagnoser
|
||||
?? createProductionWorkspaceDiagnoser(config.workspaceDiagnosticTimeoutMs, undefined, {
|
||||
internalQdrantUrl: config.internalQdrantUrl,
|
||||
@@ -384,12 +420,25 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
|
||||
secretStore: workspaceSecretStore,
|
||||
});
|
||||
catalogDatabaseRoutes(app, { repository: catalogRepository, service: catalogService, operations: catalogOperationCoordinator });
|
||||
catalogTableRoutes(app, { repository: catalogRepository, service: catalogTableService });
|
||||
catalogTableRoutes(app, {
|
||||
repository: catalogRepository,
|
||||
service: catalogTableService,
|
||||
operations: catalogOperationCoordinator,
|
||||
});
|
||||
catalogSchemaRoutes(app, {
|
||||
repository: catalogRepository,
|
||||
worker: catalogSyncWorker,
|
||||
operations: catalogOperationCoordinator,
|
||||
});
|
||||
catalogDescriptionConsolidationRoutes(app, {
|
||||
repository: catalogRepository,
|
||||
operations: catalogOperationCoordinator,
|
||||
});
|
||||
metadataGenerationModelRoutes(app, metadataGenerationModels);
|
||||
catalogDescriptionGenerationRoutes(app, {
|
||||
repository: catalogRepository,
|
||||
worker: descriptionGenerationWorker,
|
||||
});
|
||||
settingsRoutes(app, { cfg: config, listModels, getSettings });
|
||||
piManagementRoutes(app, { service: piManagement });
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,163 @@
|
||||
import type { CatalogPostgresAccess } from "./postgres-access.js";
|
||||
import type { WorkspaceDatabase } from "./types.js";
|
||||
|
||||
const MAX_SOURCE_ROWS = 5;
|
||||
const MAX_REPRESENTATIVE_VALUES = 5;
|
||||
const MAX_SOURCE_COLUMNS_PER_TARGET = 8;
|
||||
const MAX_SOURCE_VALUE_BYTES = 256;
|
||||
|
||||
export type DescriptionSourceSampleValue = string | number | boolean | null;
|
||||
|
||||
export interface DescriptionSourceSampleField {
|
||||
name: string;
|
||||
value: DescriptionSourceSampleValue;
|
||||
}
|
||||
|
||||
export interface DescriptionSourceSampleRow {
|
||||
fields: readonly DescriptionSourceSampleField[];
|
||||
}
|
||||
|
||||
export interface DescriptionSourceRepresentativeValues {
|
||||
column: string;
|
||||
values: readonly Exclude<DescriptionSourceSampleValue, null>[];
|
||||
}
|
||||
|
||||
export interface DescriptionTargetSourceSample {
|
||||
targetId: string;
|
||||
tableName: string;
|
||||
rows: readonly DescriptionSourceSampleRow[];
|
||||
representativeValues: readonly DescriptionSourceRepresentativeValues[];
|
||||
}
|
||||
|
||||
export interface DescriptionSourceSamplingTarget {
|
||||
targetId: string;
|
||||
tableName: string;
|
||||
columnNames: readonly string[];
|
||||
}
|
||||
|
||||
/** Optional, transient source context for one model-completion batch. */
|
||||
export interface DescriptionSourceSampler {
|
||||
sample(
|
||||
database: WorkspaceDatabase,
|
||||
targets: readonly DescriptionSourceSamplingTarget[],
|
||||
signal: AbortSignal,
|
||||
): Promise<readonly DescriptionTargetSourceSample[]>;
|
||||
}
|
||||
|
||||
function quoteIdentifier(identifier: string): string {
|
||||
return `"${identifier.replaceAll('"', '""')}"`;
|
||||
}
|
||||
|
||||
function boundedUtf8(value: string, maxBytes: number): string {
|
||||
const normalized = value
|
||||
.normalize("NFC")
|
||||
.replace(/\r\n?/g, "\n")
|
||||
.replace(/[\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f]/g, " ");
|
||||
if (Buffer.byteLength(normalized, "utf8") <= maxBytes) return normalized;
|
||||
let result = "";
|
||||
let bytes = 0;
|
||||
for (const character of normalized) {
|
||||
const characterBytes = Buffer.byteLength(character, "utf8");
|
||||
if (bytes + characterBytes > maxBytes) break;
|
||||
result += character;
|
||||
bytes += characterBytes;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
function normalizeValue(value: unknown): DescriptionSourceSampleValue | undefined {
|
||||
if (value === null) return null;
|
||||
if (typeof value === "string") return boundedUtf8(value, MAX_SOURCE_VALUE_BYTES);
|
||||
if (typeof value === "boolean") return value;
|
||||
if (typeof value === "number") return Number.isFinite(value) ? value : undefined;
|
||||
if (typeof value === "bigint") return boundedUtf8(String(value), MAX_SOURCE_VALUE_BYTES);
|
||||
if (value instanceof Date && !Number.isNaN(value.valueOf())) return value.toISOString();
|
||||
return undefined;
|
||||
}
|
||||
|
||||
function distinctKey(value: Exclude<DescriptionSourceSampleValue, null>): string {
|
||||
return `${typeof value}:${String(value)}`;
|
||||
}
|
||||
|
||||
/** PostgreSQL-wire sampler. REST bindings remain unsupported by CatalogPostgresAccess. */
|
||||
export class PostgresDescriptionSourceSampler implements DescriptionSourceSampler {
|
||||
constructor(private readonly access: CatalogPostgresAccess) {}
|
||||
|
||||
async sample(
|
||||
database: WorkspaceDatabase,
|
||||
targets: readonly DescriptionSourceSamplingTarget[],
|
||||
signal: AbortSignal,
|
||||
): Promise<readonly DescriptionTargetSourceSample[]> {
|
||||
const client = await this.access.connect(database, signal);
|
||||
let transactionOpen = false;
|
||||
try {
|
||||
await client.query("BEGIN TRANSACTION READ ONLY", []);
|
||||
transactionOpen = true;
|
||||
const samples: DescriptionTargetSourceSample[] = [];
|
||||
for (const target of targets) {
|
||||
const columnNames = [...new Set(target.columnNames)].slice(0, MAX_SOURCE_COLUMNS_PER_TARGET);
|
||||
if (columnNames.length === 0) {
|
||||
samples.push({
|
||||
targetId: target.targetId,
|
||||
tableName: target.tableName,
|
||||
rows: [],
|
||||
representativeValues: [],
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const projections = columnNames.map((columnName) => {
|
||||
const identifier = quoteIdentifier(columnName);
|
||||
return `LEFT((${identifier})::text, $1) AS ${identifier}`;
|
||||
});
|
||||
const sql = [
|
||||
`SELECT ${projections.join(", ")}`,
|
||||
`FROM ${quoteIdentifier(database.schema)}.${quoteIdentifier(target.tableName)}`,
|
||||
"LIMIT $2",
|
||||
].join(" ");
|
||||
const result = await client.query(sql, [MAX_SOURCE_VALUE_BYTES, MAX_SOURCE_ROWS]);
|
||||
const rows = result.rows.slice(0, MAX_SOURCE_ROWS).map((row) => ({
|
||||
fields: columnNames.flatMap((name) => {
|
||||
const value = normalizeValue(row[name]);
|
||||
return value === undefined ? [] : [{ name, value }];
|
||||
}),
|
||||
}));
|
||||
const valuesByColumn = new Map<
|
||||
string,
|
||||
Exclude<DescriptionSourceSampleValue, null>[]
|
||||
>();
|
||||
const seenByColumn = new Map<string, Set<string>>();
|
||||
let representativeValueCount = 0;
|
||||
for (const row of rows) {
|
||||
for (const field of row.fields) {
|
||||
if (representativeValueCount === MAX_REPRESENTATIVE_VALUES) break;
|
||||
if (field.value === null) continue;
|
||||
const seen = seenByColumn.get(field.name) ?? new Set<string>();
|
||||
const key = distinctKey(field.value);
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
seenByColumn.set(field.name, seen);
|
||||
const values = valuesByColumn.get(field.name) ?? [];
|
||||
values.push(field.value);
|
||||
valuesByColumn.set(field.name, values);
|
||||
representativeValueCount += 1;
|
||||
}
|
||||
if (representativeValueCount === MAX_REPRESENTATIVE_VALUES) break;
|
||||
}
|
||||
const representativeValues = columnNames.flatMap((column) => {
|
||||
const values = valuesByColumn.get(column);
|
||||
return values && values.length > 0 ? [{ column, values }] : [];
|
||||
});
|
||||
samples.push({
|
||||
targetId: target.targetId,
|
||||
tableName: target.tableName,
|
||||
rows,
|
||||
representativeValues,
|
||||
});
|
||||
}
|
||||
return samples;
|
||||
} finally {
|
||||
if (transactionOpen) await client.query("ROLLBACK", []).catch(() => undefined);
|
||||
await client.end().catch(() => undefined);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2,7 +2,10 @@ import { randomUUID } from "node:crypto";
|
||||
import {
|
||||
CatalogConflictError,
|
||||
CatalogConnectorError,
|
||||
DescriptionGenerationRunActiveError,
|
||||
type CatalogColumn,
|
||||
type CatalogDescriptionConsolidationCounts,
|
||||
type CatalogDescriptionTarget,
|
||||
type CatalogDatabaseMetadataDeleteTarget,
|
||||
type CatalogMetadataDeleteCounts,
|
||||
type CatalogRelationship,
|
||||
@@ -19,6 +22,10 @@ import {
|
||||
type DatabaseTestResult,
|
||||
type ObservedCatalogTable,
|
||||
type ObservedSchemaSnapshot,
|
||||
type DescriptionGenerationEvent,
|
||||
type DescriptionGenerationRun,
|
||||
type DescriptionGenerationRunUpdate,
|
||||
type DescriptionGenerationScope,
|
||||
type TableSyncRepositoryResult,
|
||||
type WorkspaceDatabase,
|
||||
} from "./types.js";
|
||||
@@ -33,6 +40,8 @@ export class MemoryCatalogRepository implements CatalogRepository {
|
||||
private readonly tables = new Map<string, CatalogTable>();
|
||||
private readonly columns = new Map<string, CatalogColumn>();
|
||||
private readonly relationships = new Map<string, CatalogRelationship>();
|
||||
private readonly descriptionGenerationRuns = new Map<string, DescriptionGenerationRun>();
|
||||
private readonly descriptionGenerationEvents = new Map<string, DescriptionGenerationEvent[]>();
|
||||
private readonly syncRuns = new Map<string, CatalogSyncRun>();
|
||||
private readonly syncEvents = new Map<string, CatalogSyncEvent[]>();
|
||||
|
||||
@@ -122,6 +131,11 @@ export class MemoryCatalogRepository implements CatalogRepository {
|
||||
for (const [relationshipId, relationship] of this.relationships) {
|
||||
if (relationship.databaseId === id) this.relationships.delete(relationshipId);
|
||||
}
|
||||
for (const [runId, run] of this.descriptionGenerationRuns) {
|
||||
if (run.databaseId !== id) continue;
|
||||
this.descriptionGenerationRuns.delete(runId);
|
||||
this.descriptionGenerationEvents.delete(runId);
|
||||
}
|
||||
return this.records.delete(id);
|
||||
}
|
||||
async listTables(databaseId: string): Promise<CatalogTable[]> {
|
||||
@@ -196,6 +210,166 @@ export class MemoryCatalogRepository implements CatalogRepository {
|
||||
return structuredClone(updated);
|
||||
}
|
||||
|
||||
async consolidateGeneratedDescriptions(
|
||||
databaseId: string,
|
||||
target: CatalogDescriptionTarget,
|
||||
targetIds: readonly string[],
|
||||
): Promise<CatalogDescriptionConsolidationCounts | undefined> {
|
||||
const selectedTargetIds = [...new Set(targetIds)];
|
||||
if (!this.records.has(databaseId) || selectedTargetIds.length === 0) {
|
||||
return undefined;
|
||||
}
|
||||
const now = new Date().toISOString();
|
||||
if (target === "tables") {
|
||||
const targets = selectedTargetIds.map((id) => this.tables.get(id));
|
||||
if (targets.some((table) => !table || table.databaseId !== databaseId)) return undefined;
|
||||
const copied = targets.filter((table) => Boolean(table!.generatedDescription?.trim()));
|
||||
for (const table of copied) {
|
||||
this.tables.set(table!.id, {
|
||||
...table!,
|
||||
description: table!.generatedDescription,
|
||||
version: table!.version + 1,
|
||||
updatedAt: now,
|
||||
});
|
||||
}
|
||||
return { copied: copied.length, skipped: targets.length - copied.length };
|
||||
}
|
||||
|
||||
const targets = selectedTargetIds.map((id) => this.columns.get(id));
|
||||
if (targets.some((column) => (
|
||||
!column || this.tables.get(column.tableId)?.databaseId !== databaseId
|
||||
))) return undefined;
|
||||
const copied = targets.filter((column) => Boolean(column!.generatedDescription?.trim()));
|
||||
for (const column of copied) {
|
||||
this.columns.set(column!.id, {
|
||||
...column!,
|
||||
description: column!.generatedDescription,
|
||||
version: column!.version + 1,
|
||||
updatedAt: now,
|
||||
});
|
||||
}
|
||||
return { copied: copied.length, skipped: targets.length - copied.length };
|
||||
}
|
||||
|
||||
async createDescriptionGenerationRun(
|
||||
databaseId: string,
|
||||
scope: DescriptionGenerationScope,
|
||||
modelId: string,
|
||||
language: DescriptionGenerationRun["language"],
|
||||
total: number,
|
||||
): Promise<DescriptionGenerationRun> {
|
||||
if ([...this.descriptionGenerationRuns.values()].some((run) => (
|
||||
run.status === "queued" || run.status === "running"
|
||||
))) {
|
||||
throw new DescriptionGenerationRunActiveError("A description generation run is already active");
|
||||
}
|
||||
const now = new Date().toISOString();
|
||||
const run: DescriptionGenerationRun = {
|
||||
id: randomUUID(),
|
||||
databaseId,
|
||||
scope,
|
||||
modelId,
|
||||
language,
|
||||
status: "queued",
|
||||
total,
|
||||
processed: 0,
|
||||
generated: 0,
|
||||
nonGeneratable: 0,
|
||||
failed: 0,
|
||||
createdAt: now,
|
||||
startedAt: null,
|
||||
updatedAt: now,
|
||||
finishedAt: null,
|
||||
errorSummary: null,
|
||||
};
|
||||
this.descriptionGenerationRuns.set(run.id, run);
|
||||
return structuredClone(run);
|
||||
}
|
||||
|
||||
async getDescriptionGenerationRun(runId: string): Promise<DescriptionGenerationRun | undefined> {
|
||||
const run = this.descriptionGenerationRuns.get(runId);
|
||||
return run ? structuredClone(run) : undefined;
|
||||
}
|
||||
|
||||
async listDescriptionGenerationRuns(limit = 50): Promise<DescriptionGenerationRun[]> {
|
||||
return [...this.descriptionGenerationRuns.values()]
|
||||
.sort((a, b) => b.createdAt.localeCompare(a.createdAt) || b.id.localeCompare(a.id))
|
||||
.slice(0, limit)
|
||||
.map((run) => structuredClone(run));
|
||||
}
|
||||
|
||||
async getActiveDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined> {
|
||||
const run = [...this.descriptionGenerationRuns.values()]
|
||||
.filter((candidate) => candidate.status === "queued" || candidate.status === "running")
|
||||
.sort((a, b) => b.createdAt.localeCompare(a.createdAt))[0];
|
||||
return run ? structuredClone(run) : undefined;
|
||||
}
|
||||
|
||||
async interruptActiveDescriptionGenerationRuns(
|
||||
errorSummary: string,
|
||||
): Promise<DescriptionGenerationRun[]> {
|
||||
const interrupted: DescriptionGenerationRun[] = [];
|
||||
for (const run of this.descriptionGenerationRuns.values()) {
|
||||
if (run.status !== "queued" && run.status !== "running") continue;
|
||||
const now = new Date().toISOString();
|
||||
const updated: DescriptionGenerationRun = {
|
||||
...run,
|
||||
status: "interrupted",
|
||||
updatedAt: now,
|
||||
finishedAt: now,
|
||||
errorSummary,
|
||||
};
|
||||
this.descriptionGenerationRuns.set(run.id, updated);
|
||||
interrupted.push(structuredClone(updated));
|
||||
}
|
||||
return interrupted;
|
||||
}
|
||||
|
||||
async updateDescriptionGenerationRun(
|
||||
runId: string,
|
||||
update: DescriptionGenerationRunUpdate,
|
||||
): Promise<DescriptionGenerationRun | undefined> {
|
||||
const current = this.descriptionGenerationRuns.get(runId);
|
||||
if (!current) return undefined;
|
||||
const updated = {
|
||||
...current,
|
||||
...structuredClone(update),
|
||||
updatedAt: new Date().toISOString(),
|
||||
};
|
||||
this.descriptionGenerationRuns.set(runId, updated);
|
||||
return structuredClone(updated);
|
||||
}
|
||||
|
||||
async appendDescriptionGenerationEvent(
|
||||
runId: string,
|
||||
level: DescriptionGenerationEvent["level"],
|
||||
message: string,
|
||||
): Promise<DescriptionGenerationEvent> {
|
||||
if (!this.descriptionGenerationRuns.has(runId)) {
|
||||
throw new CatalogConflictError("Description Generation Run does not exist");
|
||||
}
|
||||
const events = this.descriptionGenerationEvents.get(runId) ?? [];
|
||||
const event: DescriptionGenerationEvent = {
|
||||
runId,
|
||||
sequence: events.length + 1,
|
||||
level,
|
||||
message,
|
||||
createdAt: new Date().toISOString(),
|
||||
};
|
||||
events.push(event);
|
||||
this.descriptionGenerationEvents.set(runId, events);
|
||||
return structuredClone(event);
|
||||
}
|
||||
|
||||
async listDescriptionGenerationEvents(
|
||||
runId: string,
|
||||
afterSequence = 0,
|
||||
): Promise<DescriptionGenerationEvent[]> {
|
||||
return (this.descriptionGenerationEvents.get(runId) ?? [])
|
||||
.filter((event) => event.sequence > afterSequence)
|
||||
.map((event) => structuredClone(event));
|
||||
}
|
||||
|
||||
async listRelationships(databaseId: string): Promise<CatalogRelationship[]> {
|
||||
return [...this.relationships.values()].filter((relationship) => relationship.databaseId === databaseId)
|
||||
.sort((a, b) => `${a.sourceTableName}.${a.constraintName}`.localeCompare(`${b.sourceTableName}.${b.constraintName}`))
|
||||
|
||||
@@ -0,0 +1,235 @@
|
||||
import {
|
||||
closeSync, constants, fstatSync, lstatSync, openSync, readFileSync,
|
||||
type Stats,
|
||||
} from "node:fs";
|
||||
import { parseAllDocuments } from "yaml";
|
||||
import { z } from "zod";
|
||||
import {
|
||||
loadSecretBundle,
|
||||
METADATA_GENERATION_SECRET_KEYS,
|
||||
} from "../config/secret-bundle.js";
|
||||
|
||||
const MAX_INSTALLATION_BYTES = 1024 * 1024;
|
||||
const RUNTIME_INSTALLATION_FILE = "/run/thothii-installation/thothii-installation.yaml";
|
||||
const modelId = z.string().regex(/^[a-z][a-z0-9._-]{0,63}$/);
|
||||
const apiKeyEnvironment = z.enum(METADATA_GENERATION_SECRET_KEYS);
|
||||
const endpointSchema = z.object({
|
||||
baseUrl: z.string().min(1).max(2048).refine((value) => {
|
||||
try {
|
||||
const url = new URL(value);
|
||||
return (url.protocol === "http:" || url.protocol === "https:")
|
||||
&& url.username === "" && url.password === "" && url.search === "" && url.hash === "";
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}),
|
||||
apiVersion: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/).optional(),
|
||||
}).strict();
|
||||
const configuredModelSchema = z.object({
|
||||
id: modelId,
|
||||
label: z.string().min(1).max(128).refine((value) => value.trim() === value && !/\p{Cc}/u.test(value)),
|
||||
litellm: z.object({
|
||||
provider: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/),
|
||||
model: z.string().regex(/^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$/),
|
||||
disableThinking: z.literal(true).optional(),
|
||||
endpoint: endpointSchema.optional(),
|
||||
}).strict(),
|
||||
apiKeyEnv: apiKeyEnvironment.optional(),
|
||||
}).strict().superRefine((value, context) => {
|
||||
if (value.apiKeyEnv === undefined && value.litellm.endpoint === undefined) {
|
||||
context.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: ["apiKeyEnv"],
|
||||
message: "keyless models require an explicit endpoint",
|
||||
});
|
||||
}
|
||||
if (value.litellm.disableThinking === true && value.litellm.endpoint === undefined) {
|
||||
context.addIssue({
|
||||
code: z.ZodIssueCode.custom,
|
||||
path: ["litellm", "disableThinking"],
|
||||
message: "thinking may be disabled only for an explicit endpoint",
|
||||
});
|
||||
}
|
||||
});
|
||||
const metadataGenerationSchema = z.object({
|
||||
default: modelId.optional(),
|
||||
models: z.array(configuredModelSchema).max(64).default([]),
|
||||
}).strict();
|
||||
const installationSchema = z.object({
|
||||
metadataGeneration: metadataGenerationSchema.optional(),
|
||||
}).passthrough();
|
||||
|
||||
export interface MetadataGenerationModelChoice {
|
||||
id: string;
|
||||
label: string;
|
||||
}
|
||||
|
||||
export interface MetadataGenerationModelCatalog {
|
||||
models: MetadataGenerationModelChoice[];
|
||||
default: string | null;
|
||||
}
|
||||
|
||||
export interface ResolvedMetadataGenerationModel {
|
||||
readonly id: string;
|
||||
readonly provider: string;
|
||||
readonly model: string;
|
||||
readonly disableThinking?: true;
|
||||
readonly endpoint?: Readonly<{ baseUrl: string; apiVersion?: string }>;
|
||||
readonly apiKeyEnv?: string;
|
||||
readonly apiKey?: string;
|
||||
}
|
||||
|
||||
export class MetadataGenerationModelUnavailableError extends Error {
|
||||
constructor() {
|
||||
super("metadata-generation model is unavailable");
|
||||
this.name = "MetadataGenerationModelUnavailableError";
|
||||
}
|
||||
}
|
||||
|
||||
/** The complete interface callers need: safe discovery plus fail-closed runtime resolution. */
|
||||
export interface MetadataGenerationModels {
|
||||
catalog(): MetadataGenerationModelCatalog;
|
||||
resolve(selection: string): ResolvedMetadataGenerationModel;
|
||||
}
|
||||
|
||||
class RestartLoadedMetadataGenerationModels implements MetadataGenerationModels {
|
||||
readonly #models: ReadonlyMap<string, ResolvedMetadataGenerationModel>;
|
||||
readonly #catalog: MetadataGenerationModelCatalog;
|
||||
|
||||
constructor(
|
||||
models: ReadonlyMap<string, ResolvedMetadataGenerationModel> = new Map(),
|
||||
defaultModel: string | null = null,
|
||||
choices: MetadataGenerationModelChoice[] = [],
|
||||
) {
|
||||
this.#models = models;
|
||||
this.#catalog = {
|
||||
models: choices.map((choice) => ({ ...choice })),
|
||||
default: defaultModel,
|
||||
};
|
||||
}
|
||||
|
||||
catalog(): MetadataGenerationModelCatalog {
|
||||
return {
|
||||
models: this.#catalog.models.map((choice) => ({ ...choice })),
|
||||
default: this.#catalog.default,
|
||||
};
|
||||
}
|
||||
|
||||
resolve(selection: string): ResolvedMetadataGenerationModel {
|
||||
const model = typeof selection === "string" ? this.#models.get(selection) : undefined;
|
||||
if (!model) throw new MetadataGenerationModelUnavailableError();
|
||||
return model;
|
||||
}
|
||||
}
|
||||
|
||||
function invalid(message = "metadata-generation configuration is invalid"): Error {
|
||||
return new Error(message);
|
||||
}
|
||||
|
||||
function protectedInstallationStat(file: string, info: Stats): boolean {
|
||||
const mode = info.mode & 0o777;
|
||||
if (!info.isFile() || info.isSymbolicLink() || info.nlink !== 1
|
||||
|| info.size < 1 || info.size > MAX_INSTALLATION_BYTES) return false;
|
||||
if (file === RUNTIME_INSTALLATION_FILE && info.uid === 0 && mode === 0o444) return true;
|
||||
return info.uid === (process.getuid?.() ?? info.uid) && (mode === 0o400 || mode === 0o600);
|
||||
}
|
||||
|
||||
function readProtectedInstallation(file: string): string {
|
||||
let descriptor: number | undefined;
|
||||
try {
|
||||
const before = lstatSync(file);
|
||||
if (!protectedInstallationStat(file, before)) throw new Error("unavailable");
|
||||
descriptor = openSync(file, constants.O_RDONLY | constants.O_NOFOLLOW);
|
||||
const opened = fstatSync(descriptor);
|
||||
if (!protectedInstallationStat(file, opened)
|
||||
|| before.dev !== opened.dev || before.ino !== opened.ino) throw new Error("unavailable");
|
||||
const source = readFileSync(descriptor, "utf8");
|
||||
const after = fstatSync(descriptor);
|
||||
const current = lstatSync(file);
|
||||
if (!protectedInstallationStat(file, after) || !protectedInstallationStat(file, current)
|
||||
|| opened.dev !== after.dev || opened.ino !== after.ino
|
||||
|| opened.dev !== current.dev || opened.ino !== current.ino) throw new Error("unavailable");
|
||||
return source;
|
||||
} finally {
|
||||
if (descriptor !== undefined) try { closeSync(descriptor); } catch { /* sanitized below */ }
|
||||
}
|
||||
}
|
||||
|
||||
function readInstallation(file: string): unknown {
|
||||
try {
|
||||
const documents = parseAllDocuments(readProtectedInstallation(file), { uniqueKeys: true });
|
||||
if (documents.length !== 1) throw invalid("metadata-generation installation must contain one YAML document");
|
||||
const document = documents[0];
|
||||
if (document.errors.length > 0 || document.warnings.length > 0) {
|
||||
throw invalid("metadata-generation installation contains invalid YAML");
|
||||
}
|
||||
return document.toJSON();
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.message.startsWith("metadata-generation")) throw error;
|
||||
throw invalid("metadata-generation installation is unavailable");
|
||||
}
|
||||
}
|
||||
|
||||
export function loadMetadataGenerationModels(options: {
|
||||
installationFile?: string;
|
||||
secretsFile?: string;
|
||||
}): MetadataGenerationModels {
|
||||
if (!options.installationFile) return new RestartLoadedMetadataGenerationModels();
|
||||
const installation = installationSchema.safeParse(readInstallation(options.installationFile));
|
||||
if (!installation.success) throw invalid();
|
||||
const configured = installation.data.metadataGeneration;
|
||||
if (!configured || configured.models.length === 0) {
|
||||
if (configured?.default !== undefined) throw invalid("metadata-generation default does not identify a configured model");
|
||||
return new RestartLoadedMetadataGenerationModels();
|
||||
}
|
||||
if (!configured.default) throw invalid("metadata-generation default is required when models are configured");
|
||||
|
||||
const seen = new Set<string>();
|
||||
for (const model of configured.models) {
|
||||
if (seen.has(model.id)) throw invalid(`metadata-generation model id "${model.id}" is duplicated`);
|
||||
seen.add(model.id);
|
||||
}
|
||||
if (!seen.has(configured.default)) {
|
||||
throw invalid(`metadata-generation default "${configured.default}" is not configured`);
|
||||
}
|
||||
const requiresSecrets = configured.models.some((model) => model.apiKeyEnv !== undefined);
|
||||
let secrets: ReadonlyMap<string, string> = new Map();
|
||||
if (requiresSecrets) {
|
||||
if (!options.secretsFile) throw invalid("metadata-generation keyed models require THT_SECRETS_FILE");
|
||||
try {
|
||||
secrets = loadSecretBundle(options.secretsFile);
|
||||
} catch {
|
||||
throw invalid("metadata-generation secrets are unavailable");
|
||||
}
|
||||
}
|
||||
const models = new Map<string, ResolvedMetadataGenerationModel>();
|
||||
for (const configuredModel of configured.models) {
|
||||
let apiKey: string | undefined;
|
||||
if (configuredModel.apiKeyEnv !== undefined) {
|
||||
apiKey = secrets.get(configuredModel.apiKeyEnv);
|
||||
if (!apiKey) {
|
||||
throw invalid(`metadata-generation model "${configuredModel.id}" secret "${configuredModel.apiKeyEnv}" is missing`);
|
||||
}
|
||||
if (apiKey.length > 16 * 1024 || /\s/u.test(apiKey)) {
|
||||
throw invalid(`metadata-generation model "${configuredModel.id}" secret "${configuredModel.apiKeyEnv}" is unusable`);
|
||||
}
|
||||
}
|
||||
models.set(configuredModel.id, Object.freeze({
|
||||
id: configuredModel.id,
|
||||
provider: configuredModel.litellm.provider,
|
||||
model: configuredModel.litellm.model,
|
||||
...(configuredModel.litellm.disableThinking === true ? { disableThinking: true as const } : {}),
|
||||
...(configuredModel.litellm.endpoint === undefined
|
||||
? {}
|
||||
: { endpoint: Object.freeze({ ...configuredModel.litellm.endpoint }) }),
|
||||
...(configuredModel.apiKeyEnv === undefined
|
||||
? {}
|
||||
: { apiKeyEnv: configuredModel.apiKeyEnv, apiKey }),
|
||||
}));
|
||||
}
|
||||
return new RestartLoadedMetadataGenerationModels(
|
||||
models,
|
||||
configured.default,
|
||||
configured.models.map(({ id, label }) => ({ id, label })),
|
||||
);
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import * as initialMigration from "./migrations/001_workspace_databases.js";
|
||||
import * as catalogTablesMigration from "./migrations/002_catalog_tables.js";
|
||||
import * as catalogSchemaSyncMigration from "./migrations/003_catalog_schema_sync.js";
|
||||
import * as catalogRuntimeSequencePrivilegesMigration from "./migrations/004_catalog_runtime_sequence_privileges.js";
|
||||
import * as descriptionGenerationRunsMigration from "./migrations/005_description_generation_runs.js";
|
||||
|
||||
const connectionString = process.env.THT_CATALOG_MIGRATOR_DATABASE_URL;
|
||||
const host = process.env.THT_CATALOG_DB_HOST;
|
||||
@@ -36,6 +37,7 @@ const provider: MigrationProvider = {
|
||||
"002_catalog_tables": catalogTablesMigration,
|
||||
"003_catalog_schema_sync": catalogSchemaSyncMigration,
|
||||
"004_catalog_runtime_sequence_privileges": catalogRuntimeSequencePrivilegesMigration,
|
||||
"005_description_generation_runs": descriptionGenerationRunsMigration,
|
||||
};
|
||||
},
|
||||
};
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
import { sql, type Kysely } from "kysely";
|
||||
import type { CatalogDatabase } from "../repository.js";
|
||||
|
||||
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||
await db.schema.createTable("description_generation_runs")
|
||||
.addColumn("id", "uuid", (column) => column.primaryKey())
|
||||
.addColumn("database_id", "uuid", (column) => column.notNull()
|
||||
.references("workspace_databases.id").onDelete("cascade"))
|
||||
.addColumn("scope", "text", (column) => column.notNull())
|
||||
.addColumn("model_id", "text", (column) => column.notNull())
|
||||
.addColumn("language", "text", (column) => column.notNull())
|
||||
.addColumn("status", "text", (column) => column.notNull())
|
||||
.addColumn("total", "integer", (column) => column.notNull())
|
||||
.addColumn("processed", "integer", (column) => column.notNull().defaultTo(0))
|
||||
.addColumn("generated", "integer", (column) => column.notNull().defaultTo(0))
|
||||
.addColumn("non_generatable", "integer", (column) => column.notNull().defaultTo(0))
|
||||
.addColumn("failed", "integer", (column) => column.notNull().defaultTo(0))
|
||||
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addColumn("started_at", "timestamptz")
|
||||
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addColumn("finished_at", "timestamptz")
|
||||
.addColumn("error_summary", "text")
|
||||
.addCheckConstraint(
|
||||
"description_generation_runs_scope_check",
|
||||
sql`scope in ('selected_columns', 'selected_tables', 'all', 'missing')`,
|
||||
)
|
||||
.addCheckConstraint(
|
||||
"description_generation_runs_model_id_check",
|
||||
sql`model_id ~ '^[a-z][a-z0-9._-]{0,63}$'`,
|
||||
)
|
||||
.addCheckConstraint(
|
||||
"description_generation_runs_language_check",
|
||||
sql`language in ('en', 'it')`,
|
||||
)
|
||||
.addCheckConstraint(
|
||||
"description_generation_runs_status_check",
|
||||
sql`status in (
|
||||
'queued', 'running', 'completed', 'completed_with_errors',
|
||||
'cancelled', 'failed', 'interrupted'
|
||||
)`,
|
||||
)
|
||||
.addCheckConstraint(
|
||||
"description_generation_runs_counters_check",
|
||||
sql`total > 0
|
||||
and processed between 0 and total
|
||||
and generated >= 0
|
||||
and non_generatable >= 0
|
||||
and failed >= 0
|
||||
and generated + non_generatable + failed <= processed`,
|
||||
)
|
||||
.addCheckConstraint(
|
||||
"description_generation_runs_error_summary_check",
|
||||
sql`error_summary is null or char_length(error_summary) between 1 and 2000`,
|
||||
)
|
||||
.execute();
|
||||
await db.schema.createIndex("description_generation_runs_database_created_idx")
|
||||
.on("description_generation_runs")
|
||||
.columns(["database_id", "created_at"])
|
||||
.execute();
|
||||
await sql`CREATE UNIQUE INDEX description_generation_runs_one_active
|
||||
ON description_generation_runs ((true))
|
||||
WHERE status IN ('queued', 'running')`.execute(db);
|
||||
|
||||
await db.schema.createTable("description_generation_events")
|
||||
.addColumn("run_id", "uuid", (column) => column.notNull()
|
||||
.references("description_generation_runs.id").onDelete("cascade"))
|
||||
.addColumn("sequence", "integer", (column) => column.notNull())
|
||||
.addColumn("level", "text", (column) => column.notNull())
|
||||
.addColumn("message", "text", (column) => column.notNull())
|
||||
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addPrimaryKeyConstraint("description_generation_events_pkey", ["run_id", "sequence"])
|
||||
.addCheckConstraint("description_generation_events_sequence_check", sql`sequence > 0`)
|
||||
.addCheckConstraint(
|
||||
"description_generation_events_level_check",
|
||||
sql`level in ('info', 'warning', 'error')`,
|
||||
)
|
||||
.addCheckConstraint(
|
||||
"description_generation_events_message_check",
|
||||
sql`char_length(message) between 1 and 2000`,
|
||||
)
|
||||
.execute();
|
||||
}
|
||||
|
||||
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||
await db.schema.dropTable("description_generation_events").execute();
|
||||
await db.schema.dropTable("description_generation_runs").execute();
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import { z } from "zod";
|
||||
import type { ResolvedMetadataGenerationModel } from "./metadata-generation-models.js";
|
||||
|
||||
const MAX_HELPER_OUTPUT_BYTES = 64 * 1024;
|
||||
const helperOutputSchema = z.discriminatedUnion("ok", [
|
||||
z.object({ ok: z.literal(true), content: z.string() }).strict(),
|
||||
z.object({ ok: z.literal(false), error: z.literal("provider_failure") }).strict(),
|
||||
]);
|
||||
|
||||
export interface ModelCompletionMessage {
|
||||
role: "system" | "user";
|
||||
content: string;
|
||||
}
|
||||
|
||||
export interface ModelCompletionRequest {
|
||||
model: ResolvedMetadataGenerationModel;
|
||||
messages: readonly ModelCompletionMessage[];
|
||||
signal: AbortSignal;
|
||||
}
|
||||
|
||||
/** The provider boundary used by Description Generation. */
|
||||
export interface ModelCompleter {
|
||||
complete(request: ModelCompletionRequest): Promise<string>;
|
||||
}
|
||||
|
||||
export class ModelCompletionProviderError extends Error {
|
||||
constructor() {
|
||||
super("model completion failed");
|
||||
this.name = "ModelCompletionProviderError";
|
||||
}
|
||||
}
|
||||
|
||||
export class ModelCompletionCancelledError extends Error {
|
||||
constructor() {
|
||||
super("model completion cancelled");
|
||||
this.name = "ModelCompletionCancelledError";
|
||||
}
|
||||
}
|
||||
|
||||
export class PythonModelCompleter implements ModelCompleter {
|
||||
constructor(private readonly options: {
|
||||
pythonExecutable: string;
|
||||
cwd: string;
|
||||
helperModule?: string;
|
||||
timeoutMs?: number;
|
||||
terminationGraceMs?: number;
|
||||
}) {}
|
||||
|
||||
async complete(request: ModelCompletionRequest): Promise<string> {
|
||||
if (request.signal.aborted) throw new ModelCompletionCancelledError();
|
||||
const payload = {
|
||||
model: `${request.model.provider}/${request.model.model}`,
|
||||
...(request.model.apiKey === undefined ? {} : { api_key: request.model.apiKey }),
|
||||
messages: request.messages.map((message) => ({ ...message })),
|
||||
...(request.model.endpoint?.baseUrl === undefined
|
||||
? {}
|
||||
: { api_base: request.model.endpoint.baseUrl }),
|
||||
...(request.model.endpoint?.apiVersion === undefined
|
||||
? {}
|
||||
: { api_version: request.model.endpoint.apiVersion }),
|
||||
...(request.model.disableThinking === true ? { disable_thinking: true } : {}),
|
||||
};
|
||||
|
||||
return await new Promise<string>((resolve, reject) => {
|
||||
const child = spawn(
|
||||
this.options.pythonExecutable,
|
||||
["-m", this.options.helperModule ?? "tht.internal.litellm_completion"],
|
||||
{
|
||||
cwd: this.options.cwd,
|
||||
stdio: ["pipe", "pipe", "pipe"],
|
||||
},
|
||||
);
|
||||
let stdout = "";
|
||||
let settled = false;
|
||||
let timeout: ReturnType<typeof setTimeout> | undefined;
|
||||
let killFallback: ReturnType<typeof setTimeout> | undefined;
|
||||
let terminatingWith: Error | undefined;
|
||||
const cleanup = () => {
|
||||
if (timeout) clearTimeout(timeout);
|
||||
if (killFallback) clearTimeout(killFallback);
|
||||
request.signal.removeEventListener("abort", cancel);
|
||||
};
|
||||
const fail = (error: Error = new ModelCompletionProviderError()) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
cleanup();
|
||||
reject(error);
|
||||
};
|
||||
const terminate = (error: Error) => {
|
||||
if (settled || terminatingWith) return;
|
||||
terminatingWith = error;
|
||||
if (timeout) clearTimeout(timeout);
|
||||
try {
|
||||
child.kill("SIGTERM");
|
||||
} catch {
|
||||
fail(error);
|
||||
return;
|
||||
}
|
||||
killFallback = setTimeout(() => {
|
||||
if (settled || child.exitCode !== null || child.signalCode !== null) return;
|
||||
try {
|
||||
child.kill("SIGKILL");
|
||||
} catch {
|
||||
fail(error);
|
||||
}
|
||||
}, this.options.terminationGraceMs ?? 250);
|
||||
};
|
||||
const cancel = () => {
|
||||
terminate(new ModelCompletionCancelledError());
|
||||
};
|
||||
timeout = setTimeout(() => {
|
||||
terminate(new ModelCompletionProviderError());
|
||||
}, this.options.timeoutMs ?? 120_000);
|
||||
request.signal.addEventListener("abort", cancel, { once: true });
|
||||
if (request.signal.aborted) cancel();
|
||||
|
||||
child.stdout.setEncoding("utf8");
|
||||
child.stdout.on("data", (chunk: string) => {
|
||||
if (terminatingWith) return;
|
||||
stdout += chunk;
|
||||
if (Buffer.byteLength(stdout, "utf8") > MAX_HELPER_OUTPUT_BYTES) {
|
||||
terminate(new ModelCompletionProviderError());
|
||||
}
|
||||
});
|
||||
// Helper and provider diagnostics are deliberately not copied into application logs.
|
||||
child.stderr.resume();
|
||||
child.once("error", () => fail(terminatingWith ?? new ModelCompletionProviderError()));
|
||||
child.once("close", (code) => {
|
||||
if (settled) return;
|
||||
if (terminatingWith) return fail(terminatingWith);
|
||||
try {
|
||||
if (code !== 0) return fail();
|
||||
const output = helperOutputSchema.parse(JSON.parse(stdout));
|
||||
if (!output.ok) return fail();
|
||||
settled = true;
|
||||
cleanup();
|
||||
resolve(output.content);
|
||||
} catch {
|
||||
fail();
|
||||
}
|
||||
});
|
||||
child.stdin.once("error", () => {
|
||||
if (terminatingWith) return;
|
||||
terminate(new ModelCompletionProviderError());
|
||||
});
|
||||
child.stdin.end(JSON.stringify(payload));
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -1,22 +1,34 @@
|
||||
import { CatalogOperationInProgressError } from "./types.js";
|
||||
|
||||
/** Serializes connection tests, synchronization, and metadata cleanup for each catalog database. */
|
||||
/** Serializes mutable catalog operations for each Workspace Database. */
|
||||
export class CatalogOperationCoordinator {
|
||||
private readonly active = new Set<string>();
|
||||
private readonly active = new Map<
|
||||
string,
|
||||
{ token: symbol; owner: "catalog_operation" | "description_generation" }
|
||||
>();
|
||||
|
||||
reserve(databaseId: string): () => void {
|
||||
reserve(
|
||||
databaseId: string,
|
||||
owner: "catalog_operation" | "description_generation" = "catalog_operation",
|
||||
): () => void {
|
||||
if (this.active.has(databaseId)) {
|
||||
throw new CatalogOperationInProgressError("A database operation is already in progress");
|
||||
}
|
||||
this.active.add(databaseId);
|
||||
const token = Symbol(databaseId);
|
||||
this.active.set(databaseId, { token, owner });
|
||||
let released = false;
|
||||
return () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
this.active.delete(databaseId);
|
||||
if (this.active.get(databaseId)?.token === token) this.active.delete(databaseId);
|
||||
};
|
||||
}
|
||||
|
||||
/** Administrative recovery for a reservation whose owning local operation is no longer live. */
|
||||
releaseStale(databaseId: string, owner: "description_generation"): void {
|
||||
if (this.active.get(databaseId)?.owner === owner) this.active.delete(databaseId);
|
||||
}
|
||||
|
||||
async run<T>(databaseId: string, operation: () => Promise<T>): Promise<T> {
|
||||
const release = this.reserve(databaseId);
|
||||
try {
|
||||
|
||||
@@ -180,13 +180,17 @@ export class ConcreteCatalogPostgresAccess implements CatalogPostgresAccess {
|
||||
materialized.release();
|
||||
};
|
||||
const abort = () => { void close(); };
|
||||
signal.addEventListener("abort", abort, { once: true });
|
||||
|
||||
try {
|
||||
if (signal.aborted) throw new CatalogConnectorError("PostgreSQL connector aborted");
|
||||
const passwordFile = materialized.files.get(CATALOG_SECRET_IDS.password);
|
||||
if (!passwordFile) throw new CatalogConnectorError("Database password is not configured");
|
||||
const password = await readFile(passwordFile, "utf8");
|
||||
if (signal.aborted) throw new CatalogConnectorError("PostgreSQL connector aborted");
|
||||
const tlsCaFile = materialized.files.get(CATALOG_SECRET_IDS.tlsCa);
|
||||
const tlsCa = tlsCaFile ? await readFile(tlsCaFile, "utf8") : undefined;
|
||||
if (signal.aborted) throw new CatalogConnectorError("PostgreSQL connector aborted");
|
||||
let host: string;
|
||||
let port: number;
|
||||
|
||||
@@ -243,8 +247,8 @@ export class ConcreteCatalogPostgresAccess implements CatalogPostgresAccess {
|
||||
connectionTimeoutMillis: this.connectTimeoutMs,
|
||||
...(stream ? { stream: () => stream } : {}),
|
||||
});
|
||||
signal.addEventListener("abort", abort, { once: true });
|
||||
await client.connect();
|
||||
if (signal.aborted) throw new CatalogConnectorError("PostgreSQL connector aborted");
|
||||
return {
|
||||
query: async (sql, values) => await client!.query(sql, [...values]),
|
||||
end: close,
|
||||
|
||||
@@ -15,7 +15,10 @@ import {
|
||||
CatalogConflictError,
|
||||
CatalogConnectorError,
|
||||
CatalogUnavailableError,
|
||||
DescriptionGenerationRunActiveError,
|
||||
type CatalogColumn,
|
||||
type CatalogDescriptionConsolidationCounts,
|
||||
type CatalogDescriptionTarget,
|
||||
type CatalogDatabaseMetadataDeleteTarget,
|
||||
type CatalogMetadataDeleteCounts,
|
||||
type CatalogRelationship,
|
||||
@@ -31,6 +34,10 @@ import {
|
||||
type DatabaseBinding,
|
||||
type DatabaseConfigurationInput,
|
||||
type DatabaseTestResult,
|
||||
type DescriptionGenerationEvent,
|
||||
type DescriptionGenerationRun,
|
||||
type DescriptionGenerationRunUpdate,
|
||||
type DescriptionGenerationScope,
|
||||
type ObservedCatalogTable,
|
||||
type ObservedSchemaSnapshot,
|
||||
type TableSyncRepositoryResult,
|
||||
@@ -131,6 +138,33 @@ interface CatalogRelationshipColumnTable {
|
||||
targetColumnId: string;
|
||||
}
|
||||
|
||||
interface DescriptionGenerationRunTable {
|
||||
id: string;
|
||||
databaseId: string;
|
||||
scope: DescriptionGenerationScope;
|
||||
modelId: string;
|
||||
language: DescriptionGenerationRun["language"];
|
||||
status: DescriptionGenerationRun["status"];
|
||||
total: number;
|
||||
processed: number;
|
||||
generated: number;
|
||||
nonGeneratable: number;
|
||||
failed: number;
|
||||
createdAt: Timestamp;
|
||||
startedAt: Timestamp | null;
|
||||
updatedAt: Timestamp;
|
||||
finishedAt: Timestamp | null;
|
||||
errorSummary: string | null;
|
||||
}
|
||||
|
||||
interface DescriptionGenerationEventTable {
|
||||
runId: string;
|
||||
sequence: number;
|
||||
level: DescriptionGenerationEvent["level"];
|
||||
message: string;
|
||||
createdAt: Timestamp;
|
||||
}
|
||||
|
||||
interface CatalogSyncRunTable {
|
||||
id: string;
|
||||
databaseId: string;
|
||||
@@ -176,6 +210,8 @@ export interface CatalogDatabase {
|
||||
catalogColumns: CatalogColumnTable;
|
||||
catalogRelationships: CatalogRelationshipTable;
|
||||
catalogRelationshipColumns: CatalogRelationshipColumnTable;
|
||||
descriptionGenerationRuns: DescriptionGenerationRunTable;
|
||||
descriptionGenerationEvents: DescriptionGenerationEventTable;
|
||||
catalogSyncRuns: CatalogSyncRunTable;
|
||||
catalogSyncEvents: CatalogSyncEventTable;
|
||||
}
|
||||
@@ -281,6 +317,25 @@ function serializeSyncEvent(row: Selectable<CatalogSyncEventTable>): CatalogSync
|
||||
return { ...row, id: Number(row.id), data: row.data ?? {}, createdAt: new Date(row.createdAt).toISOString() };
|
||||
}
|
||||
|
||||
function serializeDescriptionGenerationRun(
|
||||
row: Selectable<DescriptionGenerationRunTable>,
|
||||
): DescriptionGenerationRun {
|
||||
const stamp = (value: Date | string | null) => value === null ? null : new Date(value).toISOString();
|
||||
return {
|
||||
...row,
|
||||
createdAt: new Date(row.createdAt).toISOString(),
|
||||
startedAt: stamp(row.startedAt),
|
||||
updatedAt: new Date(row.updatedAt).toISOString(),
|
||||
finishedAt: stamp(row.finishedAt),
|
||||
};
|
||||
}
|
||||
|
||||
function serializeDescriptionGenerationEvent(
|
||||
row: Selectable<DescriptionGenerationEventTable>,
|
||||
): DescriptionGenerationEvent {
|
||||
return { ...row, createdAt: new Date(row.createdAt).toISOString() };
|
||||
}
|
||||
|
||||
function bindingValues(databaseId: string, binding: DatabaseBinding) {
|
||||
return {
|
||||
databaseId,
|
||||
@@ -520,6 +575,202 @@ export class KyselyCatalogRepository implements CatalogRepository {
|
||||
return row ? await this.getColumn(databaseId, tableId, row.id) : undefined;
|
||||
}
|
||||
|
||||
async consolidateGeneratedDescriptions(
|
||||
databaseId: string,
|
||||
target: CatalogDescriptionTarget,
|
||||
targetIds: readonly string[],
|
||||
): Promise<CatalogDescriptionConsolidationCounts | undefined> {
|
||||
const selectedTargetIds = [...new Set(targetIds)];
|
||||
if (selectedTargetIds.length === 0) return undefined;
|
||||
return await this.db.transaction().execute(async (trx) => {
|
||||
const database = await trx.selectFrom("workspaceDatabases").select("id")
|
||||
.where("id", "=", databaseId).forUpdate().executeTakeFirst();
|
||||
if (!database) return undefined;
|
||||
if (target === "tables") {
|
||||
const rows = await trx.selectFrom("catalogTables")
|
||||
.select(["id", "generatedDescription"])
|
||||
.where("databaseId", "=", databaseId)
|
||||
.where("id", "in", selectedTargetIds)
|
||||
.orderBy("id")
|
||||
.forUpdate()
|
||||
.execute();
|
||||
if (rows.length !== selectedTargetIds.length) return undefined;
|
||||
const copiedIds = rows
|
||||
.filter((row) => Boolean(row.generatedDescription?.trim()))
|
||||
.map((row) => row.id);
|
||||
if (copiedIds.length > 0) {
|
||||
await trx.updateTable("catalogTables").set({
|
||||
description: sql`generated_description`,
|
||||
version: sql`version + 1`,
|
||||
updatedAt: sql`now()`,
|
||||
}).where("id", "in", copiedIds).execute();
|
||||
}
|
||||
return {
|
||||
copied: copiedIds.length,
|
||||
skipped: selectedTargetIds.length - copiedIds.length,
|
||||
};
|
||||
}
|
||||
|
||||
const tableRows = await trx.selectFrom("catalogTables").select("id")
|
||||
.where("databaseId", "=", databaseId).execute();
|
||||
const rows = tableRows.length === 0 ? [] : await trx.selectFrom("catalogColumns")
|
||||
.select(["id", "generatedDescription"])
|
||||
.where("tableId", "in", tableRows.map((table) => table.id))
|
||||
.where("id", "in", selectedTargetIds)
|
||||
.orderBy("id")
|
||||
.forUpdate()
|
||||
.execute();
|
||||
if (rows.length !== selectedTargetIds.length) return undefined;
|
||||
const copiedIds = rows
|
||||
.filter((row) => Boolean(row.generatedDescription?.trim()))
|
||||
.map((row) => row.id);
|
||||
if (copiedIds.length > 0) {
|
||||
await trx.updateTable("catalogColumns").set({
|
||||
description: sql`generated_description`,
|
||||
version: sql`version + 1`,
|
||||
updatedAt: sql`now()`,
|
||||
}).where("id", "in", copiedIds).execute();
|
||||
}
|
||||
return {
|
||||
copied: copiedIds.length,
|
||||
skipped: selectedTargetIds.length - copiedIds.length,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
async createDescriptionGenerationRun(
|
||||
databaseId: string,
|
||||
scope: DescriptionGenerationScope,
|
||||
modelId: string,
|
||||
language: DescriptionGenerationRun["language"],
|
||||
total: number,
|
||||
): Promise<DescriptionGenerationRun> {
|
||||
try {
|
||||
const row = await this.db.insertInto("descriptionGenerationRuns").values({
|
||||
id: randomUUID(),
|
||||
databaseId,
|
||||
scope,
|
||||
modelId,
|
||||
language,
|
||||
status: "queued",
|
||||
total,
|
||||
processed: 0,
|
||||
generated: 0,
|
||||
nonGeneratable: 0,
|
||||
failed: 0,
|
||||
startedAt: null,
|
||||
finishedAt: null,
|
||||
errorSummary: null,
|
||||
}).returningAll().executeTakeFirstOrThrow();
|
||||
return serializeDescriptionGenerationRun(row);
|
||||
} catch (error: any) {
|
||||
if (error?.code === "23505"
|
||||
&& error?.constraint === "description_generation_runs_one_active") {
|
||||
throw new DescriptionGenerationRunActiveError(
|
||||
"A description generation run is already active",
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async getDescriptionGenerationRun(
|
||||
runId: string,
|
||||
): Promise<DescriptionGenerationRun | undefined> {
|
||||
const row = await this.db.selectFrom("descriptionGenerationRuns")
|
||||
.selectAll()
|
||||
.where("id", "=", runId)
|
||||
.executeTakeFirst();
|
||||
return row ? serializeDescriptionGenerationRun(row) : undefined;
|
||||
}
|
||||
|
||||
async listDescriptionGenerationRuns(limit = 50): Promise<DescriptionGenerationRun[]> {
|
||||
const rows = await this.db.selectFrom("descriptionGenerationRuns")
|
||||
.selectAll()
|
||||
.orderBy("createdAt", "desc")
|
||||
.orderBy("id", "desc")
|
||||
.limit(limit)
|
||||
.execute();
|
||||
return rows.map(serializeDescriptionGenerationRun);
|
||||
}
|
||||
|
||||
async getActiveDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined> {
|
||||
const row = await this.db.selectFrom("descriptionGenerationRuns")
|
||||
.selectAll()
|
||||
.where("status", "in", ["queued", "running"])
|
||||
.orderBy("createdAt", "desc")
|
||||
.executeTakeFirst();
|
||||
return row ? serializeDescriptionGenerationRun(row) : undefined;
|
||||
}
|
||||
|
||||
async interruptActiveDescriptionGenerationRuns(
|
||||
errorSummary: string,
|
||||
): Promise<DescriptionGenerationRun[]> {
|
||||
const rows = await this.db.updateTable("descriptionGenerationRuns")
|
||||
.set({
|
||||
status: "interrupted",
|
||||
finishedAt: sql`now()`,
|
||||
updatedAt: sql`now()`,
|
||||
errorSummary,
|
||||
})
|
||||
.where("status", "in", ["queued", "running"])
|
||||
.returningAll()
|
||||
.execute();
|
||||
return rows.map(serializeDescriptionGenerationRun);
|
||||
}
|
||||
|
||||
async updateDescriptionGenerationRun(
|
||||
runId: string,
|
||||
update: DescriptionGenerationRunUpdate,
|
||||
): Promise<DescriptionGenerationRun | undefined> {
|
||||
const values: any = { ...update, updatedAt: sql`now()` };
|
||||
const row = await this.db.updateTable("descriptionGenerationRuns")
|
||||
.set(values)
|
||||
.where("id", "=", runId)
|
||||
.returningAll()
|
||||
.executeTakeFirst();
|
||||
return row ? serializeDescriptionGenerationRun(row) : undefined;
|
||||
}
|
||||
|
||||
async appendDescriptionGenerationEvent(
|
||||
runId: string,
|
||||
level: DescriptionGenerationEvent["level"],
|
||||
message: string,
|
||||
): Promise<DescriptionGenerationEvent> {
|
||||
return await this.db.transaction().execute(async (trx) => {
|
||||
const run = await trx.selectFrom("descriptionGenerationRuns")
|
||||
.select("id")
|
||||
.where("id", "=", runId)
|
||||
.forUpdate()
|
||||
.executeTakeFirst();
|
||||
if (!run) throw new CatalogConflictError("Description Generation Run does not exist");
|
||||
const current = await trx.selectFrom("descriptionGenerationEvents")
|
||||
.select(sql<number>`coalesce(max(sequence), 0)::int`.as("sequence"))
|
||||
.where("runId", "=", runId)
|
||||
.executeTakeFirst();
|
||||
const row = await trx.insertInto("descriptionGenerationEvents").values({
|
||||
runId,
|
||||
sequence: Number(current?.sequence ?? 0) + 1,
|
||||
level,
|
||||
message,
|
||||
}).returningAll().executeTakeFirstOrThrow();
|
||||
return serializeDescriptionGenerationEvent(row);
|
||||
});
|
||||
}
|
||||
|
||||
async listDescriptionGenerationEvents(
|
||||
runId: string,
|
||||
afterSequence = 0,
|
||||
): Promise<DescriptionGenerationEvent[]> {
|
||||
const rows = await this.db.selectFrom("descriptionGenerationEvents")
|
||||
.selectAll()
|
||||
.where("runId", "=", runId)
|
||||
.where("sequence", ">", afterSequence)
|
||||
.orderBy("sequence")
|
||||
.execute();
|
||||
return rows.map(serializeDescriptionGenerationEvent);
|
||||
}
|
||||
|
||||
async listRelationships(databaseId: string): Promise<CatalogRelationship[]> {
|
||||
const rows = await this.db.selectFrom("catalogRelationships as relationship")
|
||||
.innerJoin("catalogTables as sourceTable", "sourceTable.id", "relationship.sourceTableId")
|
||||
@@ -1102,6 +1353,15 @@ export class UnavailableCatalogRepository implements CatalogRepository {
|
||||
async listColumns(): Promise<CatalogColumn[]> { return this.fail(); }
|
||||
async getColumn(): Promise<CatalogColumn | undefined> { return this.fail(); }
|
||||
async updateColumnMetadata(): Promise<CatalogColumn | undefined> { return this.fail(); }
|
||||
async consolidateGeneratedDescriptions(): Promise<CatalogDescriptionConsolidationCounts | undefined> { return this.fail(); }
|
||||
async createDescriptionGenerationRun(): Promise<DescriptionGenerationRun> { return this.fail(); }
|
||||
async getDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined> { return this.fail(); }
|
||||
async listDescriptionGenerationRuns(): Promise<DescriptionGenerationRun[]> { return this.fail(); }
|
||||
async getActiveDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined> { return this.fail(); }
|
||||
async interruptActiveDescriptionGenerationRuns(): Promise<DescriptionGenerationRun[]> { return this.fail(); }
|
||||
async updateDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined> { return this.fail(); }
|
||||
async appendDescriptionGenerationEvent(): Promise<DescriptionGenerationEvent> { return this.fail(); }
|
||||
async listDescriptionGenerationEvents(): Promise<DescriptionGenerationEvent[]> { return this.fail(); }
|
||||
async listRelationships(): Promise<CatalogRelationship[]> { return this.fail(); }
|
||||
async deleteDatabaseMetadata(): Promise<CatalogMetadataDeleteCounts | undefined> { return this.fail(); }
|
||||
async deleteTableMetadata(): Promise<CatalogMetadataDeleteCounts | undefined> { return this.fail(); }
|
||||
|
||||
@@ -135,6 +135,7 @@ export interface CatalogRelationship {
|
||||
|
||||
export type CatalogDatabaseMetadataDeleteTarget = "tables" | "relationships";
|
||||
export type CatalogTableMetadataDeleteTarget = "columns" | "relationships";
|
||||
export type CatalogDescriptionTarget = "tables" | "columns";
|
||||
|
||||
export interface CatalogMetadataDeleteCounts {
|
||||
tables: number;
|
||||
@@ -142,6 +143,63 @@ export interface CatalogMetadataDeleteCounts {
|
||||
relationships: number;
|
||||
}
|
||||
|
||||
export interface CatalogDescriptionConsolidationCounts {
|
||||
copied: number;
|
||||
skipped: number;
|
||||
}
|
||||
|
||||
export type DescriptionGenerationScope =
|
||||
| "selected_columns"
|
||||
| "selected_tables"
|
||||
| "all"
|
||||
| "missing";
|
||||
export type DescriptionGenerationStatus =
|
||||
| "queued"
|
||||
| "running"
|
||||
| "completed"
|
||||
| "completed_with_errors"
|
||||
| "cancelled"
|
||||
| "failed"
|
||||
| "interrupted";
|
||||
|
||||
export interface DescriptionGenerationRun {
|
||||
id: string;
|
||||
databaseId: string;
|
||||
scope: DescriptionGenerationScope;
|
||||
modelId: string;
|
||||
language: "en" | "it";
|
||||
status: DescriptionGenerationStatus;
|
||||
total: number;
|
||||
processed: number;
|
||||
generated: number;
|
||||
nonGeneratable: number;
|
||||
failed: number;
|
||||
createdAt: string;
|
||||
startedAt: string | null;
|
||||
updatedAt: string;
|
||||
finishedAt: string | null;
|
||||
errorSummary: string | null;
|
||||
}
|
||||
|
||||
export interface DescriptionGenerationRunUpdate {
|
||||
status?: DescriptionGenerationStatus;
|
||||
processed?: number;
|
||||
generated?: number;
|
||||
nonGeneratable?: number;
|
||||
failed?: number;
|
||||
startedAt?: string | null;
|
||||
finishedAt?: string | null;
|
||||
errorSummary?: string | null;
|
||||
}
|
||||
|
||||
export interface DescriptionGenerationEvent {
|
||||
runId: string;
|
||||
sequence: number;
|
||||
level: "info" | "warning" | "error";
|
||||
message: string;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export interface ObservedRelationshipColumn {
|
||||
position: number;
|
||||
sourceColumnName: string;
|
||||
@@ -292,6 +350,37 @@ export interface CatalogRepository {
|
||||
description: string | null,
|
||||
generatedDescription: string | null,
|
||||
): Promise<CatalogColumn | undefined>;
|
||||
consolidateGeneratedDescriptions(
|
||||
databaseId: string,
|
||||
target: CatalogDescriptionTarget,
|
||||
targetIds: readonly string[],
|
||||
): Promise<CatalogDescriptionConsolidationCounts | undefined>;
|
||||
createDescriptionGenerationRun(
|
||||
databaseId: string,
|
||||
scope: DescriptionGenerationScope,
|
||||
modelId: string,
|
||||
language: DescriptionGenerationRun["language"],
|
||||
total: number,
|
||||
): Promise<DescriptionGenerationRun>;
|
||||
getDescriptionGenerationRun(runId: string): Promise<DescriptionGenerationRun | undefined>;
|
||||
listDescriptionGenerationRuns(limit?: number): Promise<DescriptionGenerationRun[]>;
|
||||
getActiveDescriptionGenerationRun(): Promise<DescriptionGenerationRun | undefined>;
|
||||
interruptActiveDescriptionGenerationRuns(
|
||||
errorSummary: string,
|
||||
): Promise<DescriptionGenerationRun[]>;
|
||||
updateDescriptionGenerationRun(
|
||||
runId: string,
|
||||
update: DescriptionGenerationRunUpdate,
|
||||
): Promise<DescriptionGenerationRun | undefined>;
|
||||
appendDescriptionGenerationEvent(
|
||||
runId: string,
|
||||
level: DescriptionGenerationEvent["level"],
|
||||
message: string,
|
||||
): Promise<DescriptionGenerationEvent>;
|
||||
listDescriptionGenerationEvents(
|
||||
runId: string,
|
||||
afterSequence?: number,
|
||||
): Promise<DescriptionGenerationEvent[]>;
|
||||
listRelationships(databaseId: string): Promise<CatalogRelationship[]>;
|
||||
deleteDatabaseMetadata(
|
||||
databaseIds: readonly string[],
|
||||
@@ -347,6 +436,7 @@ export interface CatalogRepository {
|
||||
}
|
||||
|
||||
export class CatalogConflictError extends Error {}
|
||||
export class DescriptionGenerationRunActiveError extends CatalogConflictError {}
|
||||
export class CatalogUnavailableError extends Error {}
|
||||
export class CatalogOperationInProgressError extends Error {}
|
||||
export class CatalogConnectorError extends Error {}
|
||||
|
||||
@@ -31,6 +31,7 @@ export interface AppConfig {
|
||||
ollamaEnsureTimeoutMs: number;
|
||||
piManagementTimeoutMs: number;
|
||||
secretsFile?: string;
|
||||
installationConfigFile?: string;
|
||||
piAuthFile?: string;
|
||||
secretFiles: Readonly<Record<string, string | undefined>>;
|
||||
modelApiKeyFile?: string;
|
||||
@@ -322,6 +323,13 @@ export function loadConfig(
|
||||
secretsFile.trim() !== secretsFile || secretsFile.length === 0 || secretsFile.includes("\0")
|
||||
|| !path.isAbsolute(secretsFile)
|
||||
)) throw new Error("secret bundle configuration is invalid");
|
||||
const installationConfigFile = env.THT_INSTALLATION_CONFIG_FILE;
|
||||
if (installationConfigFile !== undefined && (
|
||||
installationConfigFile.trim() !== installationConfigFile
|
||||
|| installationConfigFile.length === 0
|
||||
|| installationConfigFile.includes("\0")
|
||||
|| !path.isAbsolute(installationConfigFile)
|
||||
)) throw new Error("installation configuration is invalid");
|
||||
const piAuthFile = env.THT_PI_AUTH_FILE;
|
||||
if (piAuthFile !== undefined && (
|
||||
piAuthFile.trim() !== piAuthFile || piAuthFile.length === 0 || piAuthFile.includes("\0")
|
||||
@@ -404,6 +412,7 @@ export function loadConfig(
|
||||
ollamaEnsureTimeoutMs: Number(env.OLLAMA_ENSURE_TIMEOUT_MS ?? 60000),
|
||||
piManagementTimeoutMs: piManagementTimeout(env.PI_MANAGEMENT_TIMEOUT_MS),
|
||||
secretsFile,
|
||||
installationConfigFile,
|
||||
piAuthFile,
|
||||
secretFiles,
|
||||
modelApiKeyFile,
|
||||
|
||||
@@ -8,12 +8,19 @@ import {
|
||||
isUsableAuthenticationSecret,
|
||||
} from "../auth/secret-policy.js";
|
||||
|
||||
/** Credential names that metadata-generation model entries may reference. */
|
||||
export const METADATA_GENERATION_SECRET_KEYS = Object.freeze([
|
||||
"THT_METADATA_API_KEY", "ANTHROPIC_API_KEY", "AZURE_API_KEY", "GEMINI_API_KEY",
|
||||
"DEEPSEEK_API_KEY", "OPENAI_API_KEY", "OPENROUTER_API_KEY", "ZAI_API_KEY",
|
||||
] as const);
|
||||
|
||||
/** Keys accepted by the deployment bundle. Keep this list intentionally explicit. */
|
||||
export const SECRET_BUNDLE_KEYS = Object.freeze([
|
||||
"THT_MODEL_API_KEY", "THT_DWH_API_KEY", "THT_VEC_API_KEY", "THT_VEC_WRITE_API_KEY",
|
||||
"THT_CA", "THT_SSL_CA", "THT_VECTOR_BOOTSTRAP_PASSWORD", "THT_VECTOR_MIGRATOR_PASSWORD",
|
||||
"THT_VECTOR_READER_PASSWORD", "THT_VECTOR_WRITER_PASSWORD", "PI_PROVIDER_API_KEY",
|
||||
"THT_OIDC_CLIENT_SECRET", "THT_AUTHENTIK_API_TOKEN",
|
||||
...METADATA_GENERATION_SECRET_KEYS,
|
||||
] as const);
|
||||
|
||||
const ALLOWED = new Set<string>(SECRET_BUNDLE_KEYS);
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
|
||||
import { z } from "zod";
|
||||
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||
import type { CatalogOperationCoordinator } from "../catalog/operation-coordinator.js";
|
||||
import {
|
||||
CatalogOperationInProgressError,
|
||||
CatalogUnavailableError,
|
||||
type CatalogRepository,
|
||||
} from "../catalog/types.js";
|
||||
|
||||
const idSchema = z.uuid();
|
||||
const consolidationSchema = z.object({
|
||||
target: z.enum(["tables", "columns"]),
|
||||
targetIds: z.array(idSchema).min(1).max(10_000),
|
||||
}).strict();
|
||||
|
||||
function manage(request: FastifyRequest, reply: FastifyReply) {
|
||||
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
|
||||
}
|
||||
|
||||
function safeError(reply: FastifyReply, error: unknown) {
|
||||
if (error instanceof CatalogUnavailableError) {
|
||||
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
|
||||
}
|
||||
if (error instanceof CatalogOperationInProgressError) {
|
||||
return reply.code(409).send({
|
||||
code: "database_operation_in_progress",
|
||||
message: "A database operation is already in progress.",
|
||||
});
|
||||
}
|
||||
if (error instanceof z.ZodError) {
|
||||
return reply.code(400).send({
|
||||
code: "description_consolidation_invalid",
|
||||
message: "Description consolidation request is invalid.",
|
||||
});
|
||||
}
|
||||
return reply.code(500).send({
|
||||
code: "description_consolidation_failed",
|
||||
message: "Description consolidation failed.",
|
||||
});
|
||||
}
|
||||
|
||||
export function catalogDescriptionConsolidationRoutes(
|
||||
app: FastifyInstance,
|
||||
deps: { repository: CatalogRepository; operations: CatalogOperationCoordinator },
|
||||
): void {
|
||||
app.post("/catalog/databases/:databaseId/descriptions/consolidate", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
|
||||
const input = consolidationSchema.parse(request.body);
|
||||
const targetIds = [...new Set(input.targetIds)];
|
||||
const result = await deps.operations.run(
|
||||
databaseId,
|
||||
async () => await deps.repository.consolidateGeneratedDescriptions(
|
||||
databaseId,
|
||||
input.target,
|
||||
targetIds,
|
||||
),
|
||||
);
|
||||
if (!result) {
|
||||
return reply.code(404).send({
|
||||
code: "catalog_target_not_found",
|
||||
message: "The database or one or more selected catalog targets were not found.",
|
||||
});
|
||||
}
|
||||
return result;
|
||||
} catch (error) {
|
||||
return safeError(reply, error);
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,347 @@
|
||||
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
|
||||
import { z } from "zod";
|
||||
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||
import {
|
||||
DescriptionGenerationDuplicateTargetIdsError,
|
||||
DescriptionGenerationNoEligibleTargetsError,
|
||||
DescriptionGenerationRunLiveError,
|
||||
DescriptionGenerationTargetIdsRequiredError,
|
||||
DescriptionGenerationTargetNotFoundError,
|
||||
DescriptionGenerationWorkspaceUnavailableError,
|
||||
type DescriptionGenerationWorker,
|
||||
} from "../catalog/description-generation-worker.js";
|
||||
import { MetadataGenerationModelUnavailableError } from "../catalog/metadata-generation-models.js";
|
||||
import {
|
||||
CatalogOperationInProgressError,
|
||||
CatalogUnavailableError,
|
||||
DescriptionGenerationRunActiveError,
|
||||
type CatalogRepository,
|
||||
type DescriptionGenerationEvent,
|
||||
type DescriptionGenerationRun,
|
||||
} from "../catalog/types.js";
|
||||
|
||||
const idSchema = z.uuid();
|
||||
const modelIdSchema = z.string().regex(/^[a-z][a-z0-9._-]{0,63}$/);
|
||||
const selectedTargetIdsSchema = z.array(idSchema).min(1);
|
||||
const startSchema = z.discriminatedUnion("scope", [
|
||||
z.object({
|
||||
modelId: modelIdSchema,
|
||||
scope: z.literal("selected_columns"),
|
||||
targetIds: selectedTargetIdsSchema,
|
||||
}).strict(),
|
||||
z.object({
|
||||
modelId: modelIdSchema,
|
||||
scope: z.literal("selected_tables"),
|
||||
targetIds: selectedTargetIdsSchema,
|
||||
}).strict(),
|
||||
z.object({ modelId: modelIdSchema, scope: z.literal("all") }).strict(),
|
||||
z.object({ modelId: modelIdSchema, scope: z.literal("missing") }).strict(),
|
||||
]);
|
||||
const eventQuerySchema = z.object({
|
||||
after: z.coerce.number().int().nonnegative().default(0),
|
||||
}).strict();
|
||||
const historyQuerySchema = z.object({
|
||||
limit: z.coerce.number().int().min(1).max(100).default(50),
|
||||
}).strict();
|
||||
const terminalStatuses = new Set<DescriptionGenerationRun["status"]>([
|
||||
"completed",
|
||||
"completed_with_errors",
|
||||
"cancelled",
|
||||
"failed",
|
||||
"interrupted",
|
||||
]);
|
||||
|
||||
function manage(request: FastifyRequest, reply: FastifyReply) {
|
||||
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
|
||||
}
|
||||
|
||||
function publicEvent(event: DescriptionGenerationEvent) {
|
||||
return {
|
||||
sequence: event.sequence,
|
||||
level: event.level,
|
||||
message: event.message,
|
||||
createdAt: event.createdAt,
|
||||
};
|
||||
}
|
||||
|
||||
function publicRun(run: DescriptionGenerationRun) {
|
||||
return {
|
||||
id: run.id,
|
||||
databaseId: run.databaseId,
|
||||
scope: run.scope,
|
||||
modelId: run.modelId,
|
||||
language: run.language,
|
||||
status: run.status,
|
||||
total: run.total,
|
||||
processed: run.processed,
|
||||
generated: run.generated,
|
||||
nonGeneratable: run.nonGeneratable,
|
||||
failed: run.failed,
|
||||
createdAt: run.createdAt,
|
||||
startedAt: run.startedAt,
|
||||
updatedAt: run.updatedAt,
|
||||
finishedAt: run.finishedAt,
|
||||
errorSummary: run.errorSummary,
|
||||
};
|
||||
}
|
||||
|
||||
function safeError(reply: FastifyReply, error: unknown) {
|
||||
if (error instanceof CatalogUnavailableError) {
|
||||
return reply.code(503).send({
|
||||
code: "catalog_unavailable",
|
||||
message: "Database catalog is unavailable.",
|
||||
});
|
||||
}
|
||||
if (error instanceof DescriptionGenerationRunActiveError) {
|
||||
return reply.code(409).send({
|
||||
code: "description_generation_run_active",
|
||||
message: "A Description Generation Run is already active.",
|
||||
});
|
||||
}
|
||||
if (error instanceof DescriptionGenerationRunLiveError) {
|
||||
return reply.code(409).send({
|
||||
code: "description_generation_run_live",
|
||||
message: "A local Description Generation worker or helper is still running.",
|
||||
});
|
||||
}
|
||||
if (error instanceof CatalogOperationInProgressError) {
|
||||
return reply.code(409).send({
|
||||
code: "database_operation_in_progress",
|
||||
message: "A database operation is already in progress.",
|
||||
});
|
||||
}
|
||||
if (error instanceof MetadataGenerationModelUnavailableError) {
|
||||
return reply.code(409).send({
|
||||
code: "metadata_generation_model_unavailable",
|
||||
message: "The selected metadata-generation model is unavailable.",
|
||||
});
|
||||
}
|
||||
if (error instanceof DescriptionGenerationDuplicateTargetIdsError) {
|
||||
return reply.code(400).send({
|
||||
code: "description_generation_target_ids_duplicate",
|
||||
message: "Description generation target IDs must be unique.",
|
||||
});
|
||||
}
|
||||
if (error instanceof DescriptionGenerationTargetIdsRequiredError) {
|
||||
return reply.code(400).send({
|
||||
code: "description_generation_request_invalid",
|
||||
message: "At least one description generation target ID is required.",
|
||||
});
|
||||
}
|
||||
if (error instanceof DescriptionGenerationNoEligibleTargetsError) {
|
||||
return reply.code(409).send({
|
||||
code: "description_generation_no_eligible_targets",
|
||||
message: error.scope === "all"
|
||||
? "No Catalog Tables or Catalog Columns are available for description generation."
|
||||
: "No Catalog Tables or Catalog Columns have a missing Generated Description.",
|
||||
});
|
||||
}
|
||||
if (error instanceof DescriptionGenerationTargetNotFoundError) {
|
||||
const code = error.target === "database"
|
||||
? "database_not_found"
|
||||
: error.target === "table"
|
||||
? "catalog_table_not_found"
|
||||
: "catalog_column_not_found";
|
||||
const message = error.target === "database"
|
||||
? "Database configuration was not found."
|
||||
: error.target === "table"
|
||||
? "One or more selected Catalog Tables were not found."
|
||||
: "One or more selected Catalog Columns were not found.";
|
||||
return reply.code(404).send({
|
||||
code,
|
||||
message,
|
||||
});
|
||||
}
|
||||
if (error instanceof DescriptionGenerationWorkspaceUnavailableError) {
|
||||
return reply.code(409).send({
|
||||
code: "workspace_configuration_unavailable",
|
||||
message: "The database workspace configuration is unavailable.",
|
||||
});
|
||||
}
|
||||
if (error instanceof z.ZodError) {
|
||||
return reply.code(400).send({
|
||||
code: "description_generation_request_invalid",
|
||||
message: "Description generation request is invalid.",
|
||||
});
|
||||
}
|
||||
return reply.code(500).send({
|
||||
code: "description_generation_failed",
|
||||
message: "Description generation failed.",
|
||||
});
|
||||
}
|
||||
|
||||
export function catalogDescriptionGenerationRoutes(
|
||||
app: FastifyInstance,
|
||||
deps: { repository: CatalogRepository; worker: DescriptionGenerationWorker },
|
||||
): void {
|
||||
app.post("/catalog/databases/:databaseId/description-generation-runs", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
|
||||
const input = startSchema.parse(request.body);
|
||||
const targetIds = "targetIds" in input ? input.targetIds : [];
|
||||
const run = await deps.worker.start(databaseId, input.modelId, input.scope, targetIds);
|
||||
return reply.code(202).send(publicRun(run));
|
||||
} catch (error) {
|
||||
return safeError(reply, error);
|
||||
}
|
||||
});
|
||||
|
||||
app.get("/catalog/description-generation-runs", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const { limit } = historyQuerySchema.parse(request.query);
|
||||
return (await deps.repository.listDescriptionGenerationRuns(limit)).map(publicRun);
|
||||
} catch (error) {
|
||||
return safeError(reply, error);
|
||||
}
|
||||
});
|
||||
|
||||
app.get("/catalog/description-generation-runs/:runId", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||
const run = await deps.repository.getDescriptionGenerationRun(runId);
|
||||
return run ? publicRun(run) : reply.code(404).send({
|
||||
code: "description_generation_run_not_found",
|
||||
message: "Description Generation Run was not found.",
|
||||
});
|
||||
} catch (error) {
|
||||
return safeError(reply, error);
|
||||
}
|
||||
});
|
||||
|
||||
app.post("/catalog/description-generation-runs/:runId/cancel", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||
const run = await deps.worker.cancel(runId);
|
||||
return run ? publicRun(run) : reply.code(404).send({
|
||||
code: "description_generation_run_not_found",
|
||||
message: "Description Generation Run was not found.",
|
||||
});
|
||||
} catch (error) {
|
||||
return safeError(reply, error);
|
||||
}
|
||||
});
|
||||
|
||||
app.post("/catalog/description-generation-runs/unlock", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const run = await deps.worker.unlock();
|
||||
return run ? publicRun(run) : reply.code(404).send({
|
||||
code: "description_generation_run_not_found",
|
||||
message: "No stale active Description Generation Run was found.",
|
||||
});
|
||||
} catch (error) {
|
||||
return safeError(reply, error);
|
||||
}
|
||||
});
|
||||
|
||||
app.get("/catalog/description-generation-runs/:runId/events", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
let unsubscribe: (() => void) | undefined;
|
||||
let hijacked = false;
|
||||
try {
|
||||
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||
let after = eventQuerySchema.parse(request.query).after;
|
||||
const headerCursor = Number(request.headers["last-event-id"]);
|
||||
if (Number.isInteger(headerCursor) && headerCursor >= 0) after = Math.max(after, headerCursor);
|
||||
if (!(await deps.repository.getDescriptionGenerationRun(runId))) {
|
||||
return reply.code(404).send({
|
||||
code: "description_generation_run_not_found",
|
||||
message: "Description Generation Run was not found.",
|
||||
});
|
||||
}
|
||||
|
||||
const buffered: DescriptionGenerationEvent[] = [];
|
||||
let ready = false;
|
||||
let closed = false;
|
||||
let lastRunSnapshot = "";
|
||||
let delivery = Promise.resolve();
|
||||
const close = () => {
|
||||
if (closed) return;
|
||||
closed = true;
|
||||
unsubscribe?.();
|
||||
if (!reply.raw.destroyed) reply.raw.end();
|
||||
};
|
||||
const writeRun = (run: DescriptionGenerationRun) => {
|
||||
if (closed) return;
|
||||
const snapshot = JSON.stringify(publicRun(run));
|
||||
if (snapshot === lastRunSnapshot) return;
|
||||
lastRunSnapshot = snapshot;
|
||||
reply.raw.write(`event: run\ndata: ${snapshot}\n\n`);
|
||||
if (terminalStatuses.has(run.status)) close();
|
||||
};
|
||||
const enqueue = (event: DescriptionGenerationEvent) => {
|
||||
delivery = delivery.then(async () => {
|
||||
if (closed || event.sequence <= after) return;
|
||||
after = event.sequence;
|
||||
reply.raw.write(
|
||||
`id: ${event.sequence}\nevent: log\ndata: ${JSON.stringify(publicEvent(event))}\n\n`,
|
||||
);
|
||||
const run = await deps.repository.getDescriptionGenerationRun(runId);
|
||||
if (run) writeRun(run);
|
||||
}).catch(close);
|
||||
};
|
||||
unsubscribe = deps.worker.subscribeEvents(runId, (event) => {
|
||||
if (ready) enqueue(event);
|
||||
else buffered.push(event);
|
||||
});
|
||||
const persisted = await deps.repository.listDescriptionGenerationEvents(runId, after);
|
||||
|
||||
reply.hijack();
|
||||
hijacked = true;
|
||||
reply.raw.writeHead(200, {
|
||||
"content-type": "text/event-stream; charset=utf-8",
|
||||
"cache-control": "no-cache, no-transform",
|
||||
connection: "keep-alive",
|
||||
"x-accel-buffering": "no",
|
||||
});
|
||||
reply.raw.once("close", close);
|
||||
request.raw.once("aborted", close);
|
||||
for (const event of persisted) {
|
||||
if (event.sequence <= after) continue;
|
||||
after = event.sequence;
|
||||
reply.raw.write(
|
||||
`id: ${event.sequence}\nevent: log\ndata: ${JSON.stringify(publicEvent(event))}\n\n`,
|
||||
);
|
||||
}
|
||||
ready = true;
|
||||
buffered.sort((a, b) => a.sequence - b.sequence).forEach(enqueue);
|
||||
let pending = delivery;
|
||||
await pending;
|
||||
while (pending !== delivery) {
|
||||
pending = delivery;
|
||||
await pending;
|
||||
}
|
||||
const current = await deps.repository.getDescriptionGenerationRun(runId);
|
||||
if (current) writeRun(current);
|
||||
return reply;
|
||||
} catch (error) {
|
||||
unsubscribe?.();
|
||||
if (hijacked) {
|
||||
if (!reply.raw.destroyed) reply.raw.end();
|
||||
return reply;
|
||||
}
|
||||
return safeError(reply, error);
|
||||
}
|
||||
});
|
||||
|
||||
app.get("/catalog/description-generation-runs/:runId/events-list", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||
const { after } = eventQuerySchema.parse(request.query);
|
||||
if (!(await deps.repository.getDescriptionGenerationRun(runId))) {
|
||||
return reply.code(404).send({
|
||||
code: "description_generation_run_not_found",
|
||||
message: "Description Generation Run was not found.",
|
||||
});
|
||||
}
|
||||
return (await deps.repository.listDescriptionGenerationEvents(runId, after)).map(publicEvent);
|
||||
} catch (error) {
|
||||
return safeError(reply, error);
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -43,7 +43,13 @@ function safeError(reply: FastifyReply, error: unknown) {
|
||||
if (error instanceof CatalogUnavailableError) {
|
||||
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
|
||||
}
|
||||
if (error instanceof CatalogConflictError || error instanceof CatalogOperationInProgressError) {
|
||||
if (error instanceof CatalogOperationInProgressError) {
|
||||
return reply.code(409).send({
|
||||
code: "database_operation_in_progress",
|
||||
message: "A database operation is already in progress.",
|
||||
});
|
||||
}
|
||||
if (error instanceof CatalogConflictError) {
|
||||
return reply.code(409).send({ code: "schema_sync_conflict", message: error.message });
|
||||
}
|
||||
if (error instanceof CatalogConnectorError) {
|
||||
@@ -98,21 +104,24 @@ export function catalogSchemaRoutes(
|
||||
const tableId = idSchema.parse(params.tableId);
|
||||
const columnId = idSchema.parse(params.columnId);
|
||||
const input = metadataSchema.parse(request.body);
|
||||
const current = await deps.repository.getColumn(databaseId, tableId, columnId);
|
||||
if (!current) return reply.code(404).send({ code: "column_not_found", message: "Catalog column was not found." });
|
||||
if (current.version !== input.version) {
|
||||
return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
|
||||
}
|
||||
const updated = await deps.repository.updateColumnMetadata(
|
||||
databaseId,
|
||||
tableId,
|
||||
columnId,
|
||||
input.version,
|
||||
normalized(input.description),
|
||||
normalized(input.generatedDescription),
|
||||
);
|
||||
if (!updated) return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
|
||||
return updated;
|
||||
const operation = async () => {
|
||||
const current = await deps.repository.getColumn(databaseId, tableId, columnId);
|
||||
if (!current) return reply.code(404).send({ code: "column_not_found", message: "Catalog column was not found." });
|
||||
if (current.version !== input.version) {
|
||||
return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
|
||||
}
|
||||
const updated = await deps.repository.updateColumnMetadata(
|
||||
databaseId,
|
||||
tableId,
|
||||
columnId,
|
||||
input.version,
|
||||
normalized(input.description),
|
||||
normalized(input.generatedDescription),
|
||||
);
|
||||
if (!updated) return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
|
||||
return updated;
|
||||
};
|
||||
return deps.operations ? await deps.operations.run(databaseId, operation) : await operation();
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
|
||||
import { z } from "zod";
|
||||
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||
import type { CatalogTableService } from "../catalog/table-service.js";
|
||||
import type { CatalogOperationCoordinator } from "../catalog/operation-coordinator.js";
|
||||
import {
|
||||
CatalogConnectorError,
|
||||
CatalogOperationInProgressError,
|
||||
@@ -38,7 +39,11 @@ function safeError(reply: FastifyReply, error: unknown) {
|
||||
|
||||
export function catalogTableRoutes(
|
||||
app: FastifyInstance,
|
||||
deps: { repository: CatalogRepository; service: CatalogTableService },
|
||||
deps: {
|
||||
repository: CatalogRepository;
|
||||
service: CatalogTableService;
|
||||
operations: CatalogOperationCoordinator;
|
||||
},
|
||||
): void {
|
||||
app.get("/catalog/databases/:databaseId/tables", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
@@ -59,20 +64,22 @@ export function catalogTableRoutes(
|
||||
const tableId = idSchema.parse(params.tableId);
|
||||
const input = updateSchema.parse(request.body);
|
||||
const { version, description } = input;
|
||||
const current = await deps.repository.getTable(databaseId, tableId);
|
||||
if (!current) return reply.code(404).send({ code: "table_not_found", message: "Catalog table was not found." });
|
||||
if (current.version !== version) {
|
||||
return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
|
||||
}
|
||||
const updated = await deps.service.updateMetadata(
|
||||
databaseId,
|
||||
tableId,
|
||||
version,
|
||||
description,
|
||||
input.generatedDescription === undefined ? current.generatedDescription : input.generatedDescription,
|
||||
);
|
||||
if (!updated) return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
|
||||
return updated;
|
||||
return await deps.operations.run(databaseId, async () => {
|
||||
const current = await deps.repository.getTable(databaseId, tableId);
|
||||
if (!current) return reply.code(404).send({ code: "table_not_found", message: "Catalog table was not found." });
|
||||
if (current.version !== version) {
|
||||
return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
|
||||
}
|
||||
const updated = await deps.service.updateMetadata(
|
||||
databaseId,
|
||||
tableId,
|
||||
version,
|
||||
description,
|
||||
input.generatedDescription === undefined ? current.generatedDescription : input.generatedDescription,
|
||||
);
|
||||
if (!updated) return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
|
||||
return updated;
|
||||
});
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
import type { FastifyInstance } from "fastify";
|
||||
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||
import type { MetadataGenerationModels } from "../catalog/metadata-generation-models.js";
|
||||
|
||||
export function metadataGenerationModelRoutes(
|
||||
app: FastifyInstance,
|
||||
models: MetadataGenerationModels,
|
||||
): void {
|
||||
app.get("/catalog/metadata-generation/models", async (request, reply) => {
|
||||
if (!isPrincipalContext(requirePermission(request, reply, "database.manage"))) return reply;
|
||||
return models.catalog();
|
||||
});
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,456 @@
|
||||
import { expect, test, vi } from "vitest";
|
||||
import { DescriptionGenerationWorker } from "../src/catalog/description-generation-worker.js";
|
||||
import type { DescriptionSourceSampler } from "../src/catalog/description-source-sampler.js";
|
||||
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
|
||||
import type { MetadataGenerationModels } from "../src/catalog/metadata-generation-models.js";
|
||||
import { ModelCompletionCancelledError } from "../src/catalog/model-completer.js";
|
||||
import type { ModelCompleter, ModelCompletionRequest } from "../src/catalog/model-completer.js";
|
||||
import { CatalogOperationCoordinator } from "../src/catalog/operation-coordinator.js";
|
||||
import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
|
||||
|
||||
test("serializes Unlock with Start so stale recovery cannot release a new reservation", async () => {
|
||||
let lookupStarted!: () => void;
|
||||
const started = new Promise<void>((resolve) => { lookupStarted = resolve; });
|
||||
let releaseLookup!: () => void;
|
||||
const gate = new Promise<void>((resolve) => { releaseLookup = resolve; });
|
||||
const repository = {
|
||||
getActiveDescriptionGenerationRun: vi.fn(async () => {
|
||||
lookupStarted();
|
||||
await gate;
|
||||
return undefined;
|
||||
}),
|
||||
} as unknown as MemoryCatalogRepository;
|
||||
const resolveModel = vi.fn();
|
||||
const worker = new DescriptionGenerationWorker(
|
||||
repository,
|
||||
{} as WorkspaceRegistry,
|
||||
{
|
||||
catalog: () => ({ models: [], default: "" }),
|
||||
resolve: resolveModel,
|
||||
} as MetadataGenerationModels,
|
||||
{} as ModelCompleter,
|
||||
new CatalogOperationCoordinator(),
|
||||
{ sample: vi.fn(async () => []) },
|
||||
);
|
||||
|
||||
const unlocking = worker.unlock();
|
||||
await started;
|
||||
await expect(worker.start(
|
||||
"11111111-1111-4111-8111-111111111111",
|
||||
"openai-mini",
|
||||
"missing",
|
||||
[],
|
||||
)).rejects.toThrow("already active");
|
||||
expect(resolveModel).not.toHaveBeenCalled();
|
||||
|
||||
releaseLookup();
|
||||
await expect(unlocking).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
test("exposes an awaitable background job and absorbs provider promise rejection", async () => {
|
||||
const repository = new MemoryCatalogRepository();
|
||||
const database = await repository.create({
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||
});
|
||||
await repository.applySchemaSync(database.id, database.version, "all", [], {
|
||||
schemaVersion: 1,
|
||||
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||
tables: [{ name: "patients", sourceComment: null }],
|
||||
columns: [{
|
||||
tableName: "patients",
|
||||
name: "birth_date",
|
||||
ordinalPosition: 1,
|
||||
dataType: "date",
|
||||
isNullable: true,
|
||||
defaultExpression: null,
|
||||
primaryKeyPosition: null,
|
||||
sourceComment: null,
|
||||
}],
|
||||
relationships: [],
|
||||
});
|
||||
const table = (await repository.listTables(database.id))[0]!;
|
||||
const column = (await repository.listColumns(database.id, table.id))[0]!;
|
||||
let rejectCompletion!: (error: Error) => void;
|
||||
const pendingCompletion = new Promise<string>((_resolve, reject) => { rejectCompletion = reject; });
|
||||
const completer: ModelCompleter = {
|
||||
complete: vi.fn(async () => await pendingCompletion),
|
||||
};
|
||||
const models: MetadataGenerationModels = {
|
||||
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
|
||||
resolve: () => ({
|
||||
id: "openai-mini",
|
||||
provider: "openai",
|
||||
model: "gpt-4.1-mini",
|
||||
apiKeyEnv: "OPENAI_API_KEY",
|
||||
apiKey: "test-provider-secret",
|
||||
}),
|
||||
};
|
||||
const operations = new CatalogOperationCoordinator();
|
||||
const sourceSampler: DescriptionSourceSampler = {
|
||||
sample: vi.fn(async () => []),
|
||||
};
|
||||
const worker = new DescriptionGenerationWorker(
|
||||
repository,
|
||||
{
|
||||
read: vi.fn(async () => ({
|
||||
workspace: { workspace: { language: "it" } },
|
||||
revision: {},
|
||||
})),
|
||||
} as unknown as WorkspaceRegistry,
|
||||
models,
|
||||
completer,
|
||||
operations,
|
||||
sourceSampler,
|
||||
);
|
||||
|
||||
const run = await worker.start(database.id, "openai-mini", "selected_columns", [column.id]);
|
||||
let settled = false;
|
||||
const waiting = worker.waitForRun(run.id).then(() => { settled = true; });
|
||||
await new Promise((resolve) => setTimeout(resolve, 0));
|
||||
expect(settled).toBe(false);
|
||||
|
||||
rejectCompletion(new Error("test-provider-secret private prompt raw response"));
|
||||
await expect(waiting).resolves.toBeUndefined();
|
||||
expect(await repository.getDescriptionGenerationRun(run.id)).toMatchObject({
|
||||
status: "completed_with_errors",
|
||||
failed: 1,
|
||||
errorSummary: "Description generation completed with errors.",
|
||||
});
|
||||
const events = await repository.listDescriptionGenerationEvents(run.id);
|
||||
expect(JSON.stringify(events)).not.toMatch(/test-provider-secret|private prompt|raw response/);
|
||||
|
||||
const release = operations.reserve(database.id);
|
||||
release();
|
||||
|
||||
await expect(worker.start(
|
||||
database.id,
|
||||
"openai-mini",
|
||||
"selected_columns",
|
||||
[],
|
||||
)).rejects.toThrow("at least one target ID is required");
|
||||
});
|
||||
|
||||
test("marks an active run interrupted when the backend worker stops", async () => {
|
||||
const repository = new MemoryCatalogRepository();
|
||||
const database = await repository.create({
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||
});
|
||||
await repository.applySchemaSync(database.id, database.version, "all", [], {
|
||||
schemaVersion: 1,
|
||||
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||
tables: [{ name: "patients", sourceComment: null }],
|
||||
columns: [{
|
||||
tableName: "patients",
|
||||
name: "status",
|
||||
ordinalPosition: 1,
|
||||
dataType: "text",
|
||||
isNullable: true,
|
||||
defaultExpression: null,
|
||||
primaryKeyPosition: null,
|
||||
sourceComment: null,
|
||||
}],
|
||||
relationships: [],
|
||||
});
|
||||
const table = (await repository.listTables(database.id))[0]!;
|
||||
const column = (await repository.listColumns(database.id, table.id))[0]!;
|
||||
const completer: ModelCompleter = {
|
||||
complete: vi.fn(async (request) => await new Promise<string>((_resolve, reject) => {
|
||||
const cancel = () => reject(new ModelCompletionCancelledError());
|
||||
if (request.signal.aborted) cancel();
|
||||
else request.signal.addEventListener("abort", cancel, { once: true });
|
||||
})),
|
||||
};
|
||||
const worker = new DescriptionGenerationWorker(
|
||||
repository,
|
||||
{
|
||||
read: vi.fn(async () => ({
|
||||
workspace: { workspace: { language: "it" } },
|
||||
revision: {},
|
||||
})),
|
||||
} as unknown as WorkspaceRegistry,
|
||||
{
|
||||
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
|
||||
resolve: () => ({
|
||||
id: "openai-mini",
|
||||
provider: "openai",
|
||||
model: "gpt-4.1-mini",
|
||||
apiKeyEnv: "OPENAI_API_KEY",
|
||||
apiKey: "test-provider-secret",
|
||||
}),
|
||||
},
|
||||
completer,
|
||||
new CatalogOperationCoordinator(),
|
||||
{ sample: vi.fn(async () => []) },
|
||||
);
|
||||
|
||||
const run = await worker.start(database.id, "openai-mini", "selected_columns", [column.id]);
|
||||
await vi.waitFor(() => expect(completer.complete).toHaveBeenCalledOnce());
|
||||
await worker.stop();
|
||||
|
||||
expect(await repository.getDescriptionGenerationRun(run.id)).toMatchObject({
|
||||
status: "interrupted",
|
||||
errorSummary: "Description generation was interrupted by backend shutdown.",
|
||||
});
|
||||
expect(await repository.listDescriptionGenerationEvents(run.id)).toContainEqual(
|
||||
expect.objectContaining({
|
||||
level: "warning",
|
||||
message: "Description generation was interrupted by backend shutdown.",
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
test("adds only bounded transient source samples to the model request", async () => {
|
||||
const repository = new MemoryCatalogRepository();
|
||||
const database = await repository.create({
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||
});
|
||||
await repository.applySchemaSync(database.id, database.version, "all", [], {
|
||||
schemaVersion: 1,
|
||||
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||
tables: [{ name: "patients", sourceComment: null }],
|
||||
columns: [{
|
||||
tableName: "patients",
|
||||
name: "status",
|
||||
ordinalPosition: 1,
|
||||
dataType: "text",
|
||||
isNullable: true,
|
||||
defaultExpression: null,
|
||||
primaryKeyPosition: null,
|
||||
sourceComment: null,
|
||||
}, {
|
||||
tableName: "patients",
|
||||
name: "ward",
|
||||
ordinalPosition: 2,
|
||||
dataType: "text",
|
||||
isNullable: true,
|
||||
defaultExpression: null,
|
||||
primaryKeyPosition: null,
|
||||
sourceComment: null,
|
||||
}],
|
||||
relationships: [],
|
||||
});
|
||||
const table = (await repository.listTables(database.id))[0]!;
|
||||
const columns = await repository.listColumns(database.id, table.id);
|
||||
const column = columns.find((candidate) => candidate.name === "status")!;
|
||||
const ward = columns.find((candidate) => candidate.name === "ward")!;
|
||||
const sampleSecret = "ONLY_IN_TRANSIENT_SAMPLE_7f29c8";
|
||||
const sourceSampler: DescriptionSourceSampler = {
|
||||
sample: vi.fn(async () => [{
|
||||
targetId: column.id,
|
||||
tableName: table.name,
|
||||
rows: [
|
||||
{ fields: [{ name: column.name, value: sampleSecret }] },
|
||||
{ fields: [{ name: column.name, value: "row-2" }] },
|
||||
{ fields: [{ name: column.name, value: "row-3" }] },
|
||||
],
|
||||
representativeValues: [{
|
||||
column: column.name,
|
||||
values: [sampleSecret, sampleSecret, "two", "three"],
|
||||
}],
|
||||
}, {
|
||||
targetId: ward.id,
|
||||
tableName: table.name,
|
||||
rows: [
|
||||
{ fields: [{ name: ward.name, value: "row-4" }] },
|
||||
{ fields: [{ name: ward.name, value: "row-5" }] },
|
||||
{ fields: [{ name: ward.name, value: "row-6-must-be-omitted" }] },
|
||||
],
|
||||
representativeValues: [{
|
||||
column: ward.name,
|
||||
values: ["ward-1", "ward-2", "ward-3-must-be-omitted"],
|
||||
}],
|
||||
}]),
|
||||
};
|
||||
const completer: ModelCompleter = {
|
||||
complete: vi.fn(async () => JSON.stringify({
|
||||
results: [{
|
||||
targetId: column.id,
|
||||
outcome: "generated",
|
||||
description: "Stato amministrativo del paziente.",
|
||||
}, {
|
||||
targetId: ward.id,
|
||||
outcome: "generated",
|
||||
description: "Reparto associato al paziente.",
|
||||
}],
|
||||
})),
|
||||
};
|
||||
const models: MetadataGenerationModels = {
|
||||
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
|
||||
resolve: () => ({
|
||||
id: "openai-mini",
|
||||
provider: "openai",
|
||||
model: "gpt-4.1-mini",
|
||||
apiKeyEnv: "OPENAI_API_KEY",
|
||||
apiKey: "test-provider-secret",
|
||||
}),
|
||||
};
|
||||
const worker = new DescriptionGenerationWorker(
|
||||
repository,
|
||||
{
|
||||
read: vi.fn(async () => ({
|
||||
workspace: { workspace: { language: "it" } },
|
||||
revision: {},
|
||||
})),
|
||||
} as unknown as WorkspaceRegistry,
|
||||
models,
|
||||
completer,
|
||||
new CatalogOperationCoordinator(),
|
||||
sourceSampler,
|
||||
);
|
||||
|
||||
const run = await worker.start(
|
||||
database.id,
|
||||
"openai-mini",
|
||||
"selected_columns",
|
||||
[column.id, ward.id],
|
||||
);
|
||||
await worker.waitForRun(run.id);
|
||||
|
||||
expect(sourceSampler.sample).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ id: database.id, binding: database.binding }),
|
||||
[
|
||||
{ targetId: column.id, tableName: table.name, columnNames: [column.name] },
|
||||
{ targetId: ward.id, tableName: table.name, columnNames: [ward.name] },
|
||||
],
|
||||
expect.any(AbortSignal),
|
||||
);
|
||||
const request = vi.mocked(completer.complete).mock.calls[0]![0] as ModelCompletionRequest;
|
||||
expect(request.messages[0]?.content).toContain("untrusted");
|
||||
const userMessage = request.messages[1]!.content;
|
||||
const context = JSON.parse(userMessage.slice(userMessage.indexOf("\n") + 1));
|
||||
const sampledRows = context.targets.flatMap(
|
||||
(targetContext: { sourceSample?: { rows: unknown[] } }) => targetContext.sourceSample?.rows ?? [],
|
||||
);
|
||||
const representativeValues = context.targets.flatMap(
|
||||
(targetContext: { sourceSample?: { representativeValues: Array<{ values: unknown[] }> } }) => (
|
||||
targetContext.sourceSample?.representativeValues.flatMap((entry) => entry.values) ?? []
|
||||
),
|
||||
);
|
||||
expect(sampledRows).toHaveLength(5);
|
||||
expect(representativeValues).toHaveLength(5);
|
||||
expect(context.targets[0].sourceSample.rows).toHaveLength(3);
|
||||
expect(context.targets[1].sourceSample.rows).toHaveLength(2);
|
||||
expect(context.targets[0].sourceSample.representativeValues).toEqual([{
|
||||
column: column.name,
|
||||
values: [sampleSecret, "two", "three"],
|
||||
}]);
|
||||
expect(context.targets[1].sourceSample.representativeValues).toEqual([{
|
||||
column: ward.name,
|
||||
values: ["ward-1", "ward-2"],
|
||||
}]);
|
||||
expect(userMessage).toContain(sampleSecret);
|
||||
expect(userMessage).not.toMatch(
|
||||
/row-6-must-be-omitted|ward-3-must-be-omitted/,
|
||||
);
|
||||
|
||||
const persisted = JSON.stringify({
|
||||
run: await repository.getDescriptionGenerationRun(run.id),
|
||||
events: await repository.listDescriptionGenerationEvents(run.id),
|
||||
database: await repository.get(database.id),
|
||||
table: await repository.getTable(database.id, table.id),
|
||||
column: await repository.getColumn(database.id, table.id, column.id),
|
||||
ward: await repository.getColumn(database.id, table.id, ward.id),
|
||||
});
|
||||
expect(persisted).not.toContain(sampleSecret);
|
||||
});
|
||||
|
||||
test("continues metadata-only with one safe warning when source sampling is unavailable", async () => {
|
||||
const repository = new MemoryCatalogRepository();
|
||||
const database = await repository.create({
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
binding: {
|
||||
transport: "rest_api",
|
||||
baseUrl: "https://dwh.example.test",
|
||||
restPath: "/rpc/run_query",
|
||||
restAuth: "none",
|
||||
},
|
||||
});
|
||||
await repository.applySchemaSync(database.id, database.version, "all", [], {
|
||||
schemaVersion: 1,
|
||||
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||
tables: [{ name: "patients", sourceComment: null }],
|
||||
columns: [{
|
||||
tableName: "patients",
|
||||
name: "status",
|
||||
ordinalPosition: 1,
|
||||
dataType: "text",
|
||||
isNullable: true,
|
||||
defaultExpression: null,
|
||||
primaryKeyPosition: null,
|
||||
sourceComment: null,
|
||||
}],
|
||||
relationships: [],
|
||||
});
|
||||
const table = (await repository.listTables(database.id))[0]!;
|
||||
const column = (await repository.listColumns(database.id, table.id))[0]!;
|
||||
const samplingFailureSecret = "UNAVAILABLE_SAMPLE_DETAIL_48b1f1";
|
||||
const sourceSampler: DescriptionSourceSampler = {
|
||||
sample: vi.fn(async () => { throw new Error(samplingFailureSecret); }),
|
||||
};
|
||||
const completer: ModelCompleter = {
|
||||
complete: vi.fn(async () => JSON.stringify({
|
||||
results: [{
|
||||
targetId: column.id,
|
||||
outcome: "generated",
|
||||
description: "Stato del paziente.",
|
||||
}],
|
||||
})),
|
||||
};
|
||||
const models: MetadataGenerationModels = {
|
||||
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
|
||||
resolve: () => ({
|
||||
id: "openai-mini",
|
||||
provider: "openai",
|
||||
model: "gpt-4.1-mini",
|
||||
apiKeyEnv: "OPENAI_API_KEY",
|
||||
apiKey: "test-provider-secret",
|
||||
}),
|
||||
};
|
||||
const worker = new DescriptionGenerationWorker(
|
||||
repository,
|
||||
{
|
||||
read: vi.fn(async () => ({
|
||||
workspace: { workspace: { language: "it" } },
|
||||
revision: {},
|
||||
})),
|
||||
} as unknown as WorkspaceRegistry,
|
||||
models,
|
||||
completer,
|
||||
new CatalogOperationCoordinator(),
|
||||
sourceSampler,
|
||||
);
|
||||
|
||||
const run = await worker.start(database.id, "openai-mini", "selected_columns", [column.id]);
|
||||
await worker.waitForRun(run.id);
|
||||
|
||||
expect(await repository.getDescriptionGenerationRun(run.id)).toMatchObject({
|
||||
status: "completed",
|
||||
processed: 1,
|
||||
generated: 1,
|
||||
failed: 0,
|
||||
});
|
||||
const request = vi.mocked(completer.complete).mock.calls[0]![0] as ModelCompletionRequest;
|
||||
expect(request.messages[1]?.content).not.toMatch(/sourceSample|UNAVAILABLE_SAMPLE_DETAIL/);
|
||||
const events = await repository.listDescriptionGenerationEvents(run.id);
|
||||
expect(events.filter((event) => event.level === "warning")).toEqual([
|
||||
expect.objectContaining({
|
||||
message: "Source samples unavailable for this batch; generation continued with catalog metadata only.",
|
||||
}),
|
||||
]);
|
||||
expect(JSON.stringify(events)).not.toContain(samplingFailureSecret);
|
||||
});
|
||||
@@ -0,0 +1,384 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { PostgreSqlContainer } from "@testcontainers/postgresql";
|
||||
import { CamelCasePlugin, Kysely, PostgresDialect } from "kysely";
|
||||
import { Pool } from "pg";
|
||||
import { expect, test, vi } from "vitest";
|
||||
import { buildApp } from "../src/app.js";
|
||||
import type { DescriptionSourceSampler } from "../src/catalog/description-source-sampler.js";
|
||||
import type { MetadataGenerationModels } from "../src/catalog/metadata-generation-models.js";
|
||||
import { ModelCompletionProviderError, type ModelCompleter } from "../src/catalog/model-completer.js";
|
||||
import { up as upDatabases } from "../src/catalog/migrations/001_workspace_databases.js";
|
||||
import { up as upTables } from "../src/catalog/migrations/002_catalog_tables.js";
|
||||
import { up as upSchemaSync } from "../src/catalog/migrations/003_catalog_schema_sync.js";
|
||||
import { up as upDescriptionGeneration } from "../src/catalog/migrations/005_description_generation_runs.js";
|
||||
import { KyselyCatalogRepository, type CatalogDatabase } from "../src/catalog/repository.js";
|
||||
import { loadConfig } from "../src/config.js";
|
||||
import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
|
||||
|
||||
const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
|
||||
|
||||
async function terminalRun(app: ReturnType<typeof buildApp>, runId: string) {
|
||||
for (let attempt = 0; attempt < 200; attempt += 1) {
|
||||
const response = await app.inject({
|
||||
method: "GET",
|
||||
url: `/catalog/description-generation-runs/${runId}`,
|
||||
});
|
||||
const run = response.json();
|
||||
if (["completed", "completed_with_errors", "cancelled", "failed", "interrupted"].includes(run.status)) {
|
||||
return run;
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
}
|
||||
throw new Error(`Description Generation Run ${runId} did not finish`);
|
||||
}
|
||||
|
||||
test.skipIf(!dockerAvailable)("Fastify persists Description Generation success and failure through PostgreSQL", async () => {
|
||||
const container = await new PostgreSqlContainer("postgres:17.6-bookworm").start();
|
||||
const db = new Kysely<CatalogDatabase>({
|
||||
dialect: new PostgresDialect({ pool: new Pool({ connectionString: container.getConnectionUri() }) }),
|
||||
plugins: [new CamelCasePlugin()],
|
||||
});
|
||||
let app: ReturnType<typeof buildApp> | undefined;
|
||||
try {
|
||||
await upDatabases(db);
|
||||
await upTables(db);
|
||||
await upSchemaSync(db);
|
||||
await upDescriptionGeneration(db);
|
||||
const repository = new KyselyCatalogRepository(db);
|
||||
const database = await repository.create({
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||
});
|
||||
await repository.applySchemaSync(database.id, database.version, "all", [], {
|
||||
schemaVersion: 1,
|
||||
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||
tables: [{ name: "patients", sourceComment: "Clinical patients" }],
|
||||
columns: [
|
||||
{
|
||||
tableName: "patients",
|
||||
name: "birth_date",
|
||||
ordinalPosition: 1,
|
||||
dataType: "date",
|
||||
isNullable: true,
|
||||
defaultExpression: null,
|
||||
primaryKeyPosition: null,
|
||||
sourceComment: "Patient date of birth",
|
||||
},
|
||||
{
|
||||
tableName: "patients",
|
||||
name: "status",
|
||||
ordinalPosition: 2,
|
||||
dataType: "text",
|
||||
isNullable: true,
|
||||
defaultExpression: null,
|
||||
primaryKeyPosition: null,
|
||||
sourceComment: "Patient status",
|
||||
},
|
||||
],
|
||||
relationships: [],
|
||||
});
|
||||
const table = (await repository.listTables(database.id))[0]!;
|
||||
const columns = await repository.listColumns(database.id, table.id);
|
||||
const birthDate = columns.find((column) => column.name === "birth_date")!;
|
||||
const status = columns.find((column) => column.name === "status")!;
|
||||
const curatedTable = (await repository.updateTableMetadata(
|
||||
database.id,
|
||||
table.id,
|
||||
table.version,
|
||||
"Elenco curato dei pazienti.",
|
||||
null,
|
||||
))!;
|
||||
const curatedStatus = (await repository.updateColumnMetadata(
|
||||
database.id,
|
||||
table.id,
|
||||
status.id,
|
||||
status.version,
|
||||
"Stato curato del paziente.",
|
||||
null,
|
||||
))!;
|
||||
let call = 0;
|
||||
const modelCompleter: ModelCompleter = {
|
||||
complete: vi.fn(async () => {
|
||||
call += 1;
|
||||
if (call === 1) {
|
||||
return JSON.stringify({ results: [
|
||||
{
|
||||
targetId: birthDate.id,
|
||||
outcome: "generated",
|
||||
description: "Data di nascita del paziente.",
|
||||
},
|
||||
{ targetId: status.id, outcome: "non_generatable" },
|
||||
] });
|
||||
}
|
||||
if (call === 2) {
|
||||
return JSON.stringify({ results: [{
|
||||
targetId: table.id,
|
||||
outcome: "generated",
|
||||
description: "Elenco dei pazienti e dei loro dati clinici.",
|
||||
}] });
|
||||
}
|
||||
if (call === 4) {
|
||||
return JSON.stringify({ results: [
|
||||
{
|
||||
targetId: birthDate.id,
|
||||
outcome: "generated",
|
||||
description: "Descrizione rigenerata della data di nascita.",
|
||||
},
|
||||
{
|
||||
targetId: status.id,
|
||||
outcome: "generated",
|
||||
description: "Descrizione rigenerata dello stato.",
|
||||
},
|
||||
] });
|
||||
}
|
||||
if (call === 5) {
|
||||
return JSON.stringify({ results: [{
|
||||
targetId: table.id,
|
||||
outcome: "generated",
|
||||
description: "Descrizione rigenerata della tabella pazienti.",
|
||||
}] });
|
||||
}
|
||||
if (call === 6) {
|
||||
return JSON.stringify({ results: [{
|
||||
targetId: birthDate.id,
|
||||
outcome: "generated",
|
||||
description: "Descrizione recuperata della data di nascita.",
|
||||
}] });
|
||||
}
|
||||
throw new ModelCompletionProviderError();
|
||||
}),
|
||||
};
|
||||
const models: MetadataGenerationModels = {
|
||||
catalog: () => ({ models: [{ id: "openai-mini", label: "OpenAI Mini" }], default: "openai-mini" }),
|
||||
resolve: () => ({
|
||||
id: "openai-mini",
|
||||
provider: "openai",
|
||||
model: "gpt-4.1-mini",
|
||||
apiKeyEnv: "OPENAI_API_KEY",
|
||||
apiKey: "test-provider-secret",
|
||||
}),
|
||||
};
|
||||
const registry = {
|
||||
list: vi.fn(async () => []),
|
||||
read: vi.fn(async () => ({
|
||||
workspace: { workspace: { language: "it" } },
|
||||
revision: {},
|
||||
})),
|
||||
} as unknown as WorkspaceRegistry;
|
||||
const persistedSampleSecret = "POSTGRES_TRANSIENT_SAMPLE_6a0d7b";
|
||||
const descriptionSourceSampler: DescriptionSourceSampler = {
|
||||
sample: vi.fn(async (_database, targets) => targets.map((target) => ({
|
||||
targetId: target.targetId,
|
||||
tableName: target.tableName,
|
||||
rows: [{
|
||||
fields: target.columnNames.slice(0, 1).map((name) => ({
|
||||
name,
|
||||
value: persistedSampleSecret,
|
||||
})),
|
||||
}],
|
||||
representativeValues: target.columnNames.slice(0, 1).map((column) => ({
|
||||
column,
|
||||
values: [persistedSampleSecret],
|
||||
})),
|
||||
}))),
|
||||
};
|
||||
app = buildApp(loadConfig({ NODE_ENV: "test", THT_HARNESS_DIR: "/missing" }), {
|
||||
thtRunner: {} as never,
|
||||
workspaceRegistry: registry,
|
||||
workspaceDiagnoser: vi.fn(),
|
||||
catalogRepository: repository,
|
||||
metadataGenerationModels: models,
|
||||
modelCompleter,
|
||||
descriptionSourceSampler,
|
||||
});
|
||||
|
||||
const successfulStart = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/description-generation-runs`,
|
||||
payload: {
|
||||
modelId: "openai-mini",
|
||||
scope: "selected_columns",
|
||||
targetIds: [status.id, birthDate.id],
|
||||
},
|
||||
});
|
||||
expect(successfulStart.statusCode).toBe(202);
|
||||
expect(await terminalRun(app, successfulStart.json().id)).toMatchObject({
|
||||
status: "completed",
|
||||
total: 2,
|
||||
processed: 2,
|
||||
generated: 1,
|
||||
nonGeneratable: 1,
|
||||
failed: 0,
|
||||
});
|
||||
expect(await repository.getColumn(database.id, table.id, birthDate.id)).toMatchObject({
|
||||
generatedDescription: "Data di nascita del paziente.",
|
||||
version: birthDate.version + 1,
|
||||
});
|
||||
const generatedStatus = (await repository.getColumn(database.id, table.id, status.id))!;
|
||||
expect(generatedStatus).toMatchObject({
|
||||
description: "Stato curato del paziente.",
|
||||
generatedDescription: "Non generabile",
|
||||
version: curatedStatus.version + 1,
|
||||
});
|
||||
|
||||
const tableStart = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/description-generation-runs`,
|
||||
payload: { modelId: "openai-mini", scope: "selected_tables", targetIds: [table.id] },
|
||||
});
|
||||
expect(tableStart.statusCode).toBe(202);
|
||||
expect(await terminalRun(app, tableStart.json().id)).toMatchObject({
|
||||
scope: "selected_tables",
|
||||
status: "completed",
|
||||
processed: 1,
|
||||
generated: 1,
|
||||
nonGeneratable: 0,
|
||||
failed: 0,
|
||||
});
|
||||
expect(await repository.getTable(database.id, table.id)).toMatchObject({
|
||||
description: "Elenco curato dei pazienti.",
|
||||
generatedDescription: "Elenco dei pazienti e dei loro dati clinici.",
|
||||
version: curatedTable.version + 1,
|
||||
});
|
||||
|
||||
const failedStart = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/description-generation-runs`,
|
||||
payload: { modelId: "openai-mini", scope: "selected_columns", targetIds: [status.id] },
|
||||
});
|
||||
expect(failedStart.statusCode).toBe(202);
|
||||
const failedRun = await terminalRun(app, failedStart.json().id);
|
||||
expect(failedRun).toMatchObject({
|
||||
status: "completed_with_errors",
|
||||
processed: 1,
|
||||
generated: 0,
|
||||
failed: 1,
|
||||
errorSummary: "Description generation completed with errors.",
|
||||
});
|
||||
expect(await repository.getColumn(database.id, table.id, status.id)).toMatchObject({
|
||||
description: "Stato curato del paziente.",
|
||||
generatedDescription: "Non generabile",
|
||||
version: generatedStatus.version,
|
||||
});
|
||||
const events = await app.inject({
|
||||
method: "GET",
|
||||
url: `/catalog/description-generation-runs/${failedRun.id}/events-list`,
|
||||
});
|
||||
expect(events.statusCode).toBe(200);
|
||||
expect(events.json().find((event: { level: string }) => event.level === "error")).toEqual(expect.objectContaining({
|
||||
level: "error",
|
||||
message: `The model provider request failed. Affected Catalog Column target: ${status.id}.`,
|
||||
}));
|
||||
expect(events.body).not.toMatch(/test-provider-secret|gpt-4\.1-mini|raw provider/i);
|
||||
|
||||
const allStart = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/description-generation-runs`,
|
||||
payload: { modelId: "openai-mini", scope: "all" },
|
||||
});
|
||||
expect(allStart.statusCode).toBe(202);
|
||||
const allRun = await terminalRun(app, allStart.json().id);
|
||||
expect(allRun).toMatchObject({
|
||||
scope: "all",
|
||||
status: "completed",
|
||||
total: 3,
|
||||
processed: 3,
|
||||
generated: 3,
|
||||
nonGeneratable: 0,
|
||||
failed: 0,
|
||||
});
|
||||
expect(await repository.getColumn(database.id, table.id, birthDate.id)).toMatchObject({
|
||||
generatedDescription: "Descrizione rigenerata della data di nascita.",
|
||||
});
|
||||
expect(await repository.getColumn(database.id, table.id, status.id)).toMatchObject({
|
||||
generatedDescription: "Descrizione rigenerata dello stato.",
|
||||
});
|
||||
expect(await repository.getTable(database.id, table.id)).toMatchObject({
|
||||
generatedDescription: "Descrizione rigenerata della tabella pazienti.",
|
||||
});
|
||||
const allEvents = await app.inject({
|
||||
method: "GET",
|
||||
url: `/catalog/description-generation-runs/${allRun.id}/events-list`,
|
||||
});
|
||||
expect(allEvents.json()[0]).toEqual(expect.objectContaining({
|
||||
message: "Description generation queued (scope: all).",
|
||||
}));
|
||||
|
||||
const regeneratedBirthDate = (await repository.getColumn(
|
||||
database.id, table.id, birthDate.id,
|
||||
))!;
|
||||
await repository.updateColumnMetadata(
|
||||
database.id,
|
||||
table.id,
|
||||
birthDate.id,
|
||||
regeneratedBirthDate.version,
|
||||
regeneratedBirthDate.description,
|
||||
" ",
|
||||
);
|
||||
const missingStart = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/description-generation-runs`,
|
||||
payload: { modelId: "openai-mini", scope: "missing" },
|
||||
});
|
||||
expect(missingStart.statusCode).toBe(202);
|
||||
expect(await terminalRun(app, missingStart.json().id)).toMatchObject({
|
||||
scope: "missing",
|
||||
status: "completed",
|
||||
total: 1,
|
||||
processed: 1,
|
||||
generated: 1,
|
||||
nonGeneratable: 0,
|
||||
failed: 0,
|
||||
});
|
||||
expect(await repository.getColumn(database.id, table.id, birthDate.id)).toMatchObject({
|
||||
generatedDescription: "Descrizione recuperata della data di nascita.",
|
||||
});
|
||||
expect(modelCompleter.complete).toHaveBeenCalledTimes(6);
|
||||
expect(JSON.stringify(vi.mocked(modelCompleter.complete).mock.calls)).toContain(persistedSampleSecret);
|
||||
|
||||
const runIds = [
|
||||
successfulStart.json().id,
|
||||
tableStart.json().id,
|
||||
failedStart.json().id,
|
||||
allStart.json().id,
|
||||
missingStart.json().id,
|
||||
];
|
||||
const durableState = JSON.stringify({
|
||||
runs: await Promise.all(runIds.map(async (runId) => (
|
||||
await repository.getDescriptionGenerationRun(runId)
|
||||
))),
|
||||
events: await Promise.all(runIds.map(async (runId) => (
|
||||
await repository.listDescriptionGenerationEvents(runId)
|
||||
))),
|
||||
database: await repository.get(database.id),
|
||||
table: await repository.getTable(database.id, table.id),
|
||||
columns: await repository.listColumns(database.id, table.id),
|
||||
});
|
||||
expect(durableState).not.toContain(persistedSampleSecret);
|
||||
const apiResponses = await Promise.all([
|
||||
...runIds.flatMap((runId) => [
|
||||
app!.inject({ method: "GET", url: `/catalog/description-generation-runs/${runId}` }),
|
||||
app!.inject({
|
||||
method: "GET",
|
||||
url: `/catalog/description-generation-runs/${runId}/events-list`,
|
||||
}),
|
||||
]),
|
||||
app.inject({ method: "GET", url: `/catalog/databases/${database.id}` }),
|
||||
app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables` }),
|
||||
app.inject({
|
||||
method: "GET",
|
||||
url: `/catalog/databases/${database.id}/tables/${table.id}/columns`,
|
||||
}),
|
||||
]);
|
||||
expect(apiResponses.map((response) => response.body).join("\n")).not.toContain(
|
||||
persistedSampleSecret,
|
||||
);
|
||||
} finally {
|
||||
if (app) await app.close();
|
||||
await db.destroy();
|
||||
await container.stop();
|
||||
}
|
||||
}, 60_000);
|
||||
@@ -0,0 +1,111 @@
|
||||
import { expect, test, vi } from "vitest";
|
||||
import {
|
||||
PostgresDescriptionSourceSampler,
|
||||
type DescriptionSourceSamplingTarget,
|
||||
} from "../src/catalog/description-source-sampler.js";
|
||||
import type {
|
||||
CatalogDatabaseClient,
|
||||
CatalogPostgresAccess,
|
||||
} from "../src/catalog/postgres-access.js";
|
||||
import type { WorkspaceDatabase } from "../src/catalog/types.js";
|
||||
|
||||
const database: WorkspaceDatabase = {
|
||||
id: "11111111-1111-4111-8111-111111111111",
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: 'clinical"data',
|
||||
version: 1,
|
||||
createdAt: "2026-08-28T08:00:00Z",
|
||||
updatedAt: "2026-08-28T08:00:00Z",
|
||||
connectionStatus: "reachable",
|
||||
binding: {
|
||||
transport: "postgres_direct",
|
||||
host: "db.internal",
|
||||
port: 5432,
|
||||
username: "reader",
|
||||
},
|
||||
};
|
||||
|
||||
const target: DescriptionSourceSamplingTarget = {
|
||||
targetId: "22222222-2222-4222-8222-222222222222",
|
||||
tableName: 'patient"facts',
|
||||
columnNames: ['status"code', "ward"],
|
||||
};
|
||||
|
||||
test("samples at most five source rows and five distinct non-null examples in a read-only transaction", async () => {
|
||||
const query = vi.fn(async (sql: string) => {
|
||||
if (!sql.startsWith("SELECT")) return { rows: [] };
|
||||
return {
|
||||
rows: [
|
||||
{ 'status"code': "active", ward: null },
|
||||
{ 'status"code': "pending", ward: "A" },
|
||||
{ 'status"code': "closed", ward: "A" },
|
||||
{ 'status"code': "transferred", ward: "B" },
|
||||
{ 'status"code': "unknown", ward: "C" },
|
||||
{ 'status"code': "must-not-be-sampled", ward: "D" },
|
||||
],
|
||||
};
|
||||
});
|
||||
const end = vi.fn(async () => undefined);
|
||||
const access: CatalogPostgresAccess = {
|
||||
connect: vi.fn(async () => ({ query, end }) as CatalogDatabaseClient),
|
||||
};
|
||||
const sampler = new PostgresDescriptionSourceSampler(access);
|
||||
const controller = new AbortController();
|
||||
|
||||
const samples = await sampler.sample(database, [target], controller.signal);
|
||||
|
||||
expect(samples).toEqual([{
|
||||
targetId: target.targetId,
|
||||
tableName: target.tableName,
|
||||
rows: [
|
||||
{ fields: [{ name: 'status"code', value: "active" }, { name: "ward", value: null }] },
|
||||
{ fields: [{ name: 'status"code', value: "pending" }, { name: "ward", value: "A" }] },
|
||||
{ fields: [{ name: 'status"code', value: "closed" }, { name: "ward", value: "A" }] },
|
||||
{ fields: [{ name: 'status"code', value: "transferred" }, { name: "ward", value: "B" }] },
|
||||
{ fields: [{ name: 'status"code', value: "unknown" }, { name: "ward", value: "C" }] },
|
||||
],
|
||||
representativeValues: [
|
||||
{
|
||||
column: 'status"code',
|
||||
values: ["active", "pending", "closed", "transferred"],
|
||||
},
|
||||
{ column: "ward", values: ["A"] },
|
||||
],
|
||||
}]);
|
||||
expect(access.connect).toHaveBeenCalledWith(database, controller.signal);
|
||||
expect(samples[0]!.representativeValues.flatMap((entry) => entry.values)).toHaveLength(5);
|
||||
expect(query.mock.calls).toEqual([
|
||||
["BEGIN TRANSACTION READ ONLY", []],
|
||||
[
|
||||
'SELECT LEFT(("status""code")::text, $1) AS "status""code", LEFT(("ward")::text, $1) AS "ward" FROM "clinical""data"."patient""facts" LIMIT $2',
|
||||
[256, 5],
|
||||
],
|
||||
["ROLLBACK", []],
|
||||
]);
|
||||
expect(query.mock.calls.map(([sql]) => String(sql).split(" ")[0])).toEqual([
|
||||
"BEGIN",
|
||||
"SELECT",
|
||||
"ROLLBACK",
|
||||
]);
|
||||
expect(end).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
test("rolls back and closes the source connection when sampling fails", async () => {
|
||||
const query = vi.fn(async (sql: string) => {
|
||||
if (sql.startsWith("SELECT")) throw new Error("distinctive-source-secret");
|
||||
return { rows: [] };
|
||||
});
|
||||
const end = vi.fn(async () => undefined);
|
||||
const access: CatalogPostgresAccess = {
|
||||
connect: vi.fn(async () => ({ query, end }) as CatalogDatabaseClient),
|
||||
};
|
||||
const sampler = new PostgresDescriptionSourceSampler(access);
|
||||
const controller = new AbortController();
|
||||
|
||||
await expect(sampler.sample(database, [target], controller.signal)).rejects.toThrow();
|
||||
|
||||
expect(query).toHaveBeenCalledWith("ROLLBACK", []);
|
||||
expect(end).toHaveBeenCalledOnce();
|
||||
});
|
||||
@@ -18,3 +18,23 @@ test("reserves duplicate database ids only once for a batch operation", async ()
|
||||
expect(await coordinator.runMany(["database-a", "database-a"], async () => "completed"))
|
||||
.toBe("completed");
|
||||
});
|
||||
|
||||
test("stale generation recovery releases only its own reservation token", () => {
|
||||
const coordinator = new CatalogOperationCoordinator();
|
||||
const releaseOtherOperation = coordinator.reserve("database-a");
|
||||
|
||||
coordinator.releaseStale("database-a", "description_generation");
|
||||
expect(() => coordinator.reserve("database-a")).toThrow(
|
||||
"A database operation is already in progress",
|
||||
);
|
||||
releaseOtherOperation();
|
||||
|
||||
const releaseStaleGeneration = coordinator.reserve("database-a", "description_generation");
|
||||
coordinator.releaseStale("database-a", "description_generation");
|
||||
const releaseNewOperation = coordinator.reserve("database-a");
|
||||
releaseStaleGeneration();
|
||||
expect(() => coordinator.reserve("database-a")).toThrow(
|
||||
"A database operation is already in progress",
|
||||
);
|
||||
releaseNewOperation();
|
||||
});
|
||||
|
||||
@@ -169,3 +169,23 @@ test("connects pg through OpenSSH, supplies askpass, and releases all secret lea
|
||||
expect(child.kill).toHaveBeenCalledWith("SIGTERM");
|
||||
expect(leasedPaths.some(existsSync)).toBe(false);
|
||||
});
|
||||
|
||||
test("fails before creating a transport when the sampling signal is already aborted", async () => {
|
||||
const store = secretStore();
|
||||
store.putMany("psd-clinical", {
|
||||
[CATALOG_SECRET_IDS.password]: "db-password",
|
||||
[CATALOG_SECRET_IDS.sshPrivateKey]: "PRIVATE KEY\n",
|
||||
[CATALOG_SECRET_IDS.sshKnownHosts]: "bastion.internal ssh-ed25519 AAAATEST\n",
|
||||
});
|
||||
const spawnSsh = vi.fn(() => fakeChild());
|
||||
const createClient = vi.fn();
|
||||
const access = new ConcreteCatalogPostgresAccess(store, { spawnSsh, createClient });
|
||||
const controller = new AbortController();
|
||||
controller.abort();
|
||||
|
||||
await expect(access.connect(sshDatabase(), controller.signal)).rejects.toThrow(
|
||||
"PostgreSQL connector aborted",
|
||||
);
|
||||
expect(spawnSsh).not.toHaveBeenCalled();
|
||||
expect(createClient).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
@@ -9,6 +9,7 @@ import { up as upDatabases } from "../src/catalog/migrations/001_workspace_datab
|
||||
import { up as upTables } from "../src/catalog/migrations/002_catalog_tables.js";
|
||||
import { up as upSchemaSync } from "../src/catalog/migrations/003_catalog_schema_sync.js";
|
||||
import { up as upRuntimeSequencePrivileges } from "../src/catalog/migrations/004_catalog_runtime_sequence_privileges.js";
|
||||
import { up as upDescriptionGeneration } from "../src/catalog/migrations/005_description_generation_runs.js";
|
||||
|
||||
const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
|
||||
|
||||
@@ -200,3 +201,285 @@ test.skipIf(!dockerAvailable)("PostgreSQL repository performs scoped metadata cl
|
||||
await container.stop();
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
test.skipIf(!dockerAvailable)("PostgreSQL repository atomically consolidates selected table and column descriptions", async () => {
|
||||
const container = await new PostgreSqlContainer("postgres:17.6-bookworm").start();
|
||||
const db = new Kysely<CatalogDatabase>({
|
||||
dialect: new PostgresDialect({ pool: new Pool({ connectionString: container.getConnectionUri() }) }),
|
||||
plugins: [new CamelCasePlugin()],
|
||||
});
|
||||
try {
|
||||
await upDatabases(db);
|
||||
await upTables(db);
|
||||
await upSchemaSync(db);
|
||||
const repository = new KyselyCatalogRepository(db);
|
||||
const database = await repository.create({
|
||||
workspaceId: "consolidation-test",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "public",
|
||||
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||
});
|
||||
const snapshot: ObservedSchemaSnapshot = {
|
||||
schemaVersion: 1,
|
||||
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||
tables: [
|
||||
{ name: "patients", sourceComment: null },
|
||||
{ name: "visits", sourceComment: null },
|
||||
],
|
||||
columns: [
|
||||
{ tableName: "visits", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
|
||||
{ tableName: "visits", name: "patient_id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
|
||||
],
|
||||
relationships: [],
|
||||
};
|
||||
await repository.applySchemaSync(database.id, database.version, "all", [], snapshot);
|
||||
const tables = await repository.listTables(database.id);
|
||||
const patients = tables.find((table) => table.name === "patients")!;
|
||||
const visits = tables.find((table) => table.name === "visits")!;
|
||||
await repository.updateTableMetadata(
|
||||
database.id, patients.id, patients.version, "Old patients", "Generated patients",
|
||||
);
|
||||
await repository.updateTableMetadata(
|
||||
database.id, visits.id, visits.version, "Keep visits", " ",
|
||||
);
|
||||
|
||||
expect(await repository.consolidateGeneratedDescriptions(
|
||||
database.id, "tables", [patients.id, visits.id],
|
||||
)).toEqual({ copied: 1, skipped: 1 });
|
||||
expect(await repository.getTable(database.id, patients.id)).toMatchObject({
|
||||
description: "Generated patients",
|
||||
generatedDescription: "Generated patients",
|
||||
version: patients.version + 2,
|
||||
});
|
||||
expect(await repository.getTable(database.id, visits.id)).toMatchObject({
|
||||
description: "Keep visits",
|
||||
generatedDescription: " ",
|
||||
version: visits.version + 1,
|
||||
});
|
||||
|
||||
const columns = await repository.listColumns(database.id, visits.id);
|
||||
const id = columns.find((column) => column.name === "id")!;
|
||||
const patientId = columns.find((column) => column.name === "patient_id")!;
|
||||
await repository.updateColumnMetadata(
|
||||
database.id, visits.id, id.id, id.version, "Old id", "Generated id",
|
||||
);
|
||||
await repository.updateColumnMetadata(
|
||||
database.id, visits.id, patientId.id, patientId.version, "Keep patient reference", null,
|
||||
);
|
||||
|
||||
expect(await repository.consolidateGeneratedDescriptions(
|
||||
database.id, "columns", [id.id, patientId.id],
|
||||
)).toEqual({ copied: 1, skipped: 1 });
|
||||
expect(await repository.getColumn(database.id, visits.id, id.id)).toMatchObject({
|
||||
description: "Generated id",
|
||||
generatedDescription: "Generated id",
|
||||
version: id.version + 2,
|
||||
});
|
||||
expect(await repository.getColumn(database.id, visits.id, patientId.id)).toMatchObject({
|
||||
description: "Keep patient reference",
|
||||
generatedDescription: null,
|
||||
version: patientId.version + 1,
|
||||
});
|
||||
|
||||
const currentVisits = (await repository.getTable(database.id, visits.id))!;
|
||||
await repository.updateTableMetadata(
|
||||
database.id, visits.id, currentVisits.version, "Still curated visits", "Generated visits",
|
||||
);
|
||||
expect(await repository.consolidateGeneratedDescriptions(database.id, "tables", [
|
||||
visits.id,
|
||||
"99999999-9999-4999-8999-999999999999",
|
||||
])).toBeUndefined();
|
||||
expect(await repository.getTable(database.id, visits.id)).toMatchObject({
|
||||
description: "Still curated visits",
|
||||
generatedDescription: "Generated visits",
|
||||
});
|
||||
} finally {
|
||||
await db.destroy();
|
||||
await container.stop();
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
test.skipIf(!dockerAvailable)("PostgreSQL repository persists globally exclusive Description Generation Runs and ordered events", async () => {
|
||||
const container = await new PostgreSqlContainer("postgres:17.6-bookworm").start();
|
||||
const db = new Kysely<CatalogDatabase>({
|
||||
dialect: new PostgresDialect({ pool: new Pool({ connectionString: container.getConnectionUri() }) }),
|
||||
plugins: [new CamelCasePlugin()],
|
||||
});
|
||||
try {
|
||||
await upDatabases(db);
|
||||
await upTables(db);
|
||||
await upSchemaSync(db);
|
||||
await upDescriptionGeneration(db);
|
||||
const repository = new KyselyCatalogRepository(db);
|
||||
const firstDatabase = await repository.create({
|
||||
workspaceId: "generation-one",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse_one",
|
||||
schema: "public",
|
||||
binding: { transport: "postgres_direct", host: "one.internal", port: 5432, username: "reader" },
|
||||
});
|
||||
const secondDatabase = await repository.create({
|
||||
workspaceId: "generation-two",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse_two",
|
||||
schema: "public",
|
||||
binding: { transport: "postgres_direct", host: "two.internal", port: 5432, username: "reader" },
|
||||
});
|
||||
await repository.applySchemaSync(firstDatabase.id, firstDatabase.version, "all", [], {
|
||||
schemaVersion: 1,
|
||||
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||
tables: [{ name: "patients", sourceComment: "Clinical patients" }],
|
||||
columns: [{
|
||||
tableName: "patients",
|
||||
name: "birth_date",
|
||||
ordinalPosition: 1,
|
||||
dataType: "date",
|
||||
isNullable: true,
|
||||
defaultExpression: null,
|
||||
primaryKeyPosition: null,
|
||||
sourceComment: "Patient date of birth",
|
||||
}],
|
||||
relationships: [],
|
||||
});
|
||||
const table = (await repository.listTables(firstDatabase.id))[0]!;
|
||||
const column = (await repository.listColumns(firstDatabase.id, table.id))[0]!;
|
||||
|
||||
const run = await repository.createDescriptionGenerationRun(
|
||||
firstDatabase.id,
|
||||
"selected_columns",
|
||||
"openai-mini",
|
||||
"it",
|
||||
1,
|
||||
);
|
||||
expect(run).toMatchObject({
|
||||
databaseId: firstDatabase.id,
|
||||
scope: "selected_columns",
|
||||
modelId: "openai-mini",
|
||||
language: "it",
|
||||
status: "queued",
|
||||
total: 1,
|
||||
processed: 0,
|
||||
generated: 0,
|
||||
nonGeneratable: 0,
|
||||
failed: 0,
|
||||
startedAt: null,
|
||||
finishedAt: null,
|
||||
errorSummary: null,
|
||||
});
|
||||
await expect(repository.createDescriptionGenerationRun(
|
||||
secondDatabase.id,
|
||||
"selected_columns",
|
||||
"openai-mini",
|
||||
"en",
|
||||
1,
|
||||
)).rejects.toThrow("A description generation run is already active");
|
||||
await repository.updateDescriptionGenerationRun(run.id, {
|
||||
status: "running",
|
||||
startedAt: new Date().toISOString(),
|
||||
});
|
||||
await expect(repository.createDescriptionGenerationRun(
|
||||
secondDatabase.id,
|
||||
"selected_columns",
|
||||
"openai-mini",
|
||||
"en",
|
||||
1,
|
||||
)).rejects.toThrow("A description generation run is already active");
|
||||
|
||||
await repository.appendDescriptionGenerationEvent(run.id, "info", "Description generation queued.");
|
||||
await repository.appendDescriptionGenerationEvent(run.id, "info", "Description generation started.");
|
||||
expect(await repository.listDescriptionGenerationEvents(run.id, 1)).toEqual([
|
||||
expect.objectContaining({
|
||||
runId: run.id,
|
||||
sequence: 2,
|
||||
level: "info",
|
||||
message: "Description generation started.",
|
||||
createdAt: expect.any(String),
|
||||
}),
|
||||
]);
|
||||
|
||||
const updatedColumn = await repository.updateColumnMetadata(
|
||||
firstDatabase.id,
|
||||
table.id,
|
||||
column.id,
|
||||
column.version,
|
||||
column.description,
|
||||
"Data di nascita del paziente.",
|
||||
);
|
||||
expect(updatedColumn).toMatchObject({
|
||||
generatedDescription: "Data di nascita del paziente.",
|
||||
version: column.version + 1,
|
||||
});
|
||||
expect(await repository.updateDescriptionGenerationRun(run.id, {
|
||||
status: "completed",
|
||||
processed: 1,
|
||||
generated: 1,
|
||||
startedAt: new Date().toISOString(),
|
||||
finishedAt: new Date().toISOString(),
|
||||
})).toMatchObject({
|
||||
status: "completed",
|
||||
processed: 1,
|
||||
generated: 1,
|
||||
});
|
||||
|
||||
const next = await repository.createDescriptionGenerationRun(
|
||||
secondDatabase.id,
|
||||
"missing",
|
||||
"openai-mini",
|
||||
"en",
|
||||
1,
|
||||
);
|
||||
expect(await repository.getDescriptionGenerationRun(next.id)).toMatchObject({
|
||||
scope: "missing",
|
||||
total: 1,
|
||||
});
|
||||
await repository.updateDescriptionGenerationRun(next.id, {
|
||||
status: "running",
|
||||
startedAt: new Date().toISOString(),
|
||||
});
|
||||
expect(await repository.updateDescriptionGenerationRun(next.id, {
|
||||
status: "failed",
|
||||
processed: 1,
|
||||
failed: 1,
|
||||
finishedAt: new Date().toISOString(),
|
||||
errorSummary: "The model provider request failed.",
|
||||
})).toMatchObject({
|
||||
status: "failed",
|
||||
processed: 1,
|
||||
failed: 1,
|
||||
errorSummary: "The model provider request failed.",
|
||||
});
|
||||
|
||||
const allRun = await repository.createDescriptionGenerationRun(
|
||||
firstDatabase.id,
|
||||
"all",
|
||||
"openai-mini",
|
||||
"it",
|
||||
2,
|
||||
);
|
||||
expect(await repository.getDescriptionGenerationRun(allRun.id)).toMatchObject({
|
||||
scope: "all",
|
||||
total: 2,
|
||||
});
|
||||
expect(await repository.getActiveDescriptionGenerationRun()).toMatchObject({ id: allRun.id });
|
||||
expect((await repository.listDescriptionGenerationRuns(2)).map((candidate) => candidate.id)).toEqual([
|
||||
allRun.id,
|
||||
next.id,
|
||||
]);
|
||||
|
||||
expect(await repository.interruptActiveDescriptionGenerationRuns(
|
||||
"Description generation was interrupted by backend restart.",
|
||||
)).toEqual([
|
||||
expect.objectContaining({
|
||||
id: allRun.id,
|
||||
status: "interrupted",
|
||||
finishedAt: expect.any(String),
|
||||
errorSummary: "Description generation was interrupted by backend restart.",
|
||||
}),
|
||||
]);
|
||||
expect(await repository.getActiveDescriptionGenerationRun()).toBeUndefined();
|
||||
} finally {
|
||||
await db.destroy();
|
||||
await container.stop();
|
||||
}
|
||||
}, 60_000);
|
||||
|
||||
@@ -5,6 +5,7 @@ import { afterEach, expect, test, vi } from "vitest";
|
||||
import { buildApp } from "../src/app.js";
|
||||
import { loadConfig } from "../src/config.js";
|
||||
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
|
||||
import { CatalogOperationCoordinator } from "../src/catalog/operation-coordinator.js";
|
||||
import type { CatalogSchemaIntrospector } from "../src/catalog/schema-introspector.js";
|
||||
import type { CatalogSyncRun, ObservedSchemaSnapshot } from "../src/catalog/types.js";
|
||||
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||
@@ -95,7 +96,7 @@ async function waitFor(repository: MemoryCatalogRepository, runId: string, state
|
||||
throw new Error(`Run ${runId} did not reach ${state}`);
|
||||
}
|
||||
|
||||
async function setup() {
|
||||
async function setup(env: Record<string, string> = {}) {
|
||||
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-schema-secret-"));
|
||||
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-schema-runtime-"));
|
||||
roots.push(secretRoot, runtimeRoot);
|
||||
@@ -116,21 +117,23 @@ async function setup() {
|
||||
return structuredClone(observed);
|
||||
});
|
||||
const introspector: CatalogSchemaIntrospector = { scan };
|
||||
const operations = new CatalogOperationCoordinator();
|
||||
const registry = {
|
||||
list: vi.fn(async () => [revision]),
|
||||
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
|
||||
read: vi.fn(async () => ({ workspace, revision })),
|
||||
} as unknown as WorkspaceRegistry;
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test", ...env }), {
|
||||
thtRunner: {} as never,
|
||||
workspaceRegistry: registry,
|
||||
workspaceSecretStore: new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" }),
|
||||
catalogRepository: repository,
|
||||
catalogSchemaIntrospector: introspector,
|
||||
catalogOperationCoordinator: operations,
|
||||
workspaceDiagnoser: vi.fn(),
|
||||
});
|
||||
return {
|
||||
app, repository, database: (await repository.get(created.id))!, scan,
|
||||
app, repository, database: (await repository.get(created.id))!, scan, operations,
|
||||
setObserved(next: ObservedSchemaSnapshot) { observed = next; },
|
||||
};
|
||||
}
|
||||
@@ -243,6 +246,207 @@ test("keeps generated descriptions editable and preserves them across synchroniz
|
||||
expect(await repository.getColumn(database.id, patients.id, idColumn.id)).toMatchObject({ description: "Reviewed key", generatedDescription: "Generated key draft" });
|
||||
});
|
||||
|
||||
test("consolidates non-empty generated table descriptions and reports skipped selections", async () => {
|
||||
const { app, repository, database, scan } = await setup();
|
||||
await seedCatalog(repository, database);
|
||||
const tables = await repository.listTables(database.id);
|
||||
const patients = tables.find((table) => table.name === "patients")!;
|
||||
const visits = tables.find((table) => table.name === "visits")!;
|
||||
await repository.updateTableMetadata(
|
||||
database.id,
|
||||
patients.id,
|
||||
patients.version,
|
||||
"Curated patients",
|
||||
"Generated patients",
|
||||
);
|
||||
await repository.updateTableMetadata(
|
||||
database.id,
|
||||
visits.id,
|
||||
visits.version,
|
||||
"Keep curated visits",
|
||||
null,
|
||||
);
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
|
||||
payload: { target: "tables", targetIds: [patients.id, visits.id] },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(response.json()).toEqual({ copied: 1, skipped: 1 });
|
||||
expect(await repository.getTable(database.id, patients.id)).toMatchObject({
|
||||
description: "Generated patients",
|
||||
generatedDescription: "Generated patients",
|
||||
version: patients.version + 2,
|
||||
});
|
||||
expect(await repository.getTable(database.id, visits.id)).toMatchObject({
|
||||
description: "Keep curated visits",
|
||||
generatedDescription: null,
|
||||
version: visits.version + 1,
|
||||
});
|
||||
expect(scan).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test("consolidates non-empty generated column descriptions and preserves curated text for empty proposals", async () => {
|
||||
const { app, repository, database, scan } = await setup();
|
||||
await seedCatalog(repository, database);
|
||||
const visits = (await repository.listTables(database.id)).find((table) => table.name === "visits")!;
|
||||
const columns = await repository.listColumns(database.id, visits.id);
|
||||
const id = columns.find((column) => column.name === "id")!;
|
||||
const patientId = columns.find((column) => column.name === "patient_id")!;
|
||||
await repository.updateColumnMetadata(
|
||||
database.id,
|
||||
visits.id,
|
||||
id.id,
|
||||
id.version,
|
||||
"Curated visit identifier",
|
||||
"Generated visit identifier",
|
||||
);
|
||||
await repository.updateColumnMetadata(
|
||||
database.id,
|
||||
visits.id,
|
||||
patientId.id,
|
||||
patientId.version,
|
||||
"Keep curated patient reference",
|
||||
"",
|
||||
);
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
|
||||
payload: { target: "columns", targetIds: [id.id, patientId.id] },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(response.json()).toEqual({ copied: 1, skipped: 1 });
|
||||
expect(await repository.getColumn(database.id, visits.id, id.id)).toMatchObject({
|
||||
description: "Generated visit identifier",
|
||||
generatedDescription: "Generated visit identifier",
|
||||
version: id.version + 2,
|
||||
});
|
||||
expect(await repository.getColumn(database.id, visits.id, patientId.id)).toMatchObject({
|
||||
description: "Keep curated patient reference",
|
||||
generatedDescription: "",
|
||||
version: patientId.version + 1,
|
||||
});
|
||||
expect(scan).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
test("rejects description consolidation while the Workspace Database is reserved", async () => {
|
||||
const { app, repository, database, operations } = await setup();
|
||||
await seedCatalog(repository, database);
|
||||
const table = (await repository.listTables(database.id))[0]!;
|
||||
const edited = await repository.updateTableMetadata(
|
||||
database.id,
|
||||
table.id,
|
||||
table.version,
|
||||
"Existing curated text",
|
||||
"Generated text",
|
||||
);
|
||||
const release = operations.reserve(database.id);
|
||||
try {
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
|
||||
payload: { target: "tables", targetIds: [table.id] },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(409);
|
||||
expect(response.json()).toEqual({
|
||||
code: "database_operation_in_progress",
|
||||
message: "A database operation is already in progress.",
|
||||
});
|
||||
expect(await repository.getTable(database.id, table.id)).toMatchObject({
|
||||
description: "Existing curated text",
|
||||
generatedDescription: "Generated text",
|
||||
version: edited!.version,
|
||||
});
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
});
|
||||
|
||||
test("requires database.manage for description consolidation", async () => {
|
||||
const { app, repository, database } = await setup({ AUTH_MODE: "upstream" });
|
||||
await seedCatalog(repository, database);
|
||||
const table = (await repository.listTables(database.id))[0]!;
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
|
||||
headers: {
|
||||
"x-thoth-principal-issuer": "portal",
|
||||
"x-thoth-principal-subject": "catalog-reader",
|
||||
"x-thoth-is-admin": "0",
|
||||
},
|
||||
payload: { target: "tables", targetIds: [table.id] },
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(403);
|
||||
expect(response.json()).toEqual({ code: "auth_forbidden", error: "This operation is not permitted" });
|
||||
});
|
||||
|
||||
test("strictly validates description consolidation database and target ids", async () => {
|
||||
const { app, repository, database } = await setup();
|
||||
await seedCatalog(repository, database);
|
||||
const table = (await repository.listTables(database.id))[0]!;
|
||||
|
||||
const responses = await Promise.all([
|
||||
app.inject({
|
||||
method: "POST",
|
||||
url: "/catalog/databases/not-a-uuid/descriptions/consolidate",
|
||||
payload: { target: "tables", targetIds: [table.id] },
|
||||
}),
|
||||
app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
|
||||
payload: { target: "tables", targetIds: ["not-a-uuid"] },
|
||||
}),
|
||||
app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
|
||||
payload: { target: "tables", targetIds: [table.id], unexpected: true },
|
||||
}),
|
||||
]);
|
||||
|
||||
expect(responses.map((response) => response.statusCode)).toEqual([400, 400, 400]);
|
||||
for (const response of responses) {
|
||||
expect(response.json()).toEqual({
|
||||
code: "description_consolidation_invalid",
|
||||
message: "Description consolidation request is invalid.",
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
test("rejects a missing consolidation target without copying valid selections", async () => {
|
||||
const { app, repository, database } = await setup();
|
||||
await seedCatalog(repository, database);
|
||||
const table = (await repository.listTables(database.id))[0]!;
|
||||
const edited = await repository.updateTableMetadata(
|
||||
database.id,
|
||||
table.id,
|
||||
table.version,
|
||||
"Existing curated text",
|
||||
"Generated text",
|
||||
);
|
||||
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/descriptions/consolidate`,
|
||||
payload: {
|
||||
target: "tables",
|
||||
targetIds: [table.id, "99999999-9999-4999-8999-999999999999"],
|
||||
},
|
||||
});
|
||||
|
||||
expect(response.statusCode).toBe(404);
|
||||
expect(await repository.getTable(database.id, table.id)).toMatchObject({
|
||||
description: "Existing curated text",
|
||||
generatedDescription: "Generated text",
|
||||
version: edited!.version,
|
||||
});
|
||||
});
|
||||
|
||||
test("waits for confirmation and rescans before applying destructive changes", async () => {
|
||||
const { app, repository, database, scan, setObserved } = await setup();
|
||||
const first = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
|
||||
|
||||
@@ -290,6 +290,14 @@ test("loadConfig accepts only an absolute generic model key file", () => {
|
||||
.toThrow(/model credential configuration is invalid/);
|
||||
});
|
||||
|
||||
test("loadConfig accepts only an absolute runtime installation descriptor path", () => {
|
||||
expect(loadConfig({
|
||||
THT_INSTALLATION_CONFIG_FILE: "/run/thothii-installation/thothii-installation.yaml",
|
||||
}).installationConfigFile).toBe("/run/thothii-installation/thothii-installation.yaml");
|
||||
expect(() => loadConfig({ THT_INSTALLATION_CONFIG_FILE: "host/thothii-installation.yaml" }))
|
||||
.toThrow("installation configuration is invalid");
|
||||
});
|
||||
|
||||
test("loadConfig accepts a file-backed catalog role and rejects partial catalog configuration", () => {
|
||||
expect(loadConfig({
|
||||
THT_CATALOG_DB_HOST: "catalog-db",
|
||||
|
||||
@@ -0,0 +1,271 @@
|
||||
import { chmodSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, expect, test, vi } from "vitest";
|
||||
import { buildApp } from "../src/app.js";
|
||||
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
|
||||
import {
|
||||
loadMetadataGenerationModels,
|
||||
MetadataGenerationModelUnavailableError,
|
||||
} from "../src/catalog/metadata-generation-models.js";
|
||||
import { loadConfig } from "../src/config.js";
|
||||
import type { WorkspaceRegistry } from "../src/workspaces/registry.js";
|
||||
|
||||
const roots: string[] = [];
|
||||
|
||||
afterEach(() => {
|
||||
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function metadataConfiguration(
|
||||
metadataGeneration: string,
|
||||
secrets = "OPENAI_API_KEY=raw-provider-secret\n",
|
||||
) {
|
||||
const root = mkdtempSync(join(tmpdir(), "thothii-metadata-models-"));
|
||||
roots.push(root);
|
||||
const installationFile = join(root, "thothii-installation.yaml");
|
||||
const secretsFile = join(root, "thothii.secrets");
|
||||
writeFileSync(installationFile, metadataGeneration, { mode: 0o600 });
|
||||
writeFileSync(secretsFile, secrets, { mode: 0o600 });
|
||||
chmodSync(installationFile, 0o600);
|
||||
chmodSync(secretsFile, 0o600);
|
||||
return { installationFile, secretsFile };
|
||||
}
|
||||
|
||||
function appFor(installationFile: string, secretsFile: string) {
|
||||
const config = loadConfig({
|
||||
NODE_ENV: "test",
|
||||
THT_HARNESS_DIR: "/missing",
|
||||
THT_INSTALLATION_CONFIG_FILE: installationFile,
|
||||
THT_SECRETS_FILE: secretsFile,
|
||||
PI_PROVIDER: "unrelated-pi-provider",
|
||||
PI_MODEL: "unrelated-pi-model",
|
||||
});
|
||||
return buildApp(config, {
|
||||
thtRunner: {} as never,
|
||||
workspaceRegistry: { list: vi.fn(async () => []) } as unknown as WorkspaceRegistry,
|
||||
workspaceDiagnoser: vi.fn(),
|
||||
catalogRepository: new MemoryCatalogRepository(),
|
||||
});
|
||||
}
|
||||
|
||||
test("exposes only safe metadata-generation choices and their configured default", async () => {
|
||||
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm:
|
||||
provider: openai
|
||||
model: gpt-4.1-mini
|
||||
endpoint:
|
||||
baseUrl: https://api.openai.example/v1
|
||||
apiVersion: "2026-08-01"
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`);
|
||||
const app = appFor(installationFile, secretsFile);
|
||||
|
||||
const response = await app.inject({ method: "GET", url: "/catalog/metadata-generation/models" });
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(response.json()).toEqual({
|
||||
models: [{ id: "openai-mini", label: "OpenAI Mini" }],
|
||||
default: "openai-mini",
|
||||
});
|
||||
expect(response.body).not.toMatch(/openai\/gpt|gpt-4\.1|api\.openai|OPENAI_API_KEY|raw-provider-secret/);
|
||||
await app.close();
|
||||
});
|
||||
|
||||
test("rejects an unprotected installation descriptor", () => {
|
||||
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm: {provider: openai, model: gpt-4.1-mini}
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`);
|
||||
chmodSync(installationFile, 0o644);
|
||||
|
||||
expect(() => loadMetadataGenerationModels({ installationFile, secretsFile }))
|
||||
.toThrow("metadata-generation installation is unavailable");
|
||||
});
|
||||
|
||||
test("returns an empty safe catalog when no metadata-generation model is configured", async () => {
|
||||
const { installationFile, secretsFile } = metadataConfiguration("profile: local\n");
|
||||
const app = appFor(installationFile, secretsFile);
|
||||
|
||||
const response = await app.inject({ method: "GET", url: "/catalog/metadata-generation/models" });
|
||||
|
||||
expect(response.statusCode).toBe(200);
|
||||
expect(response.json()).toEqual({ models: [], default: null });
|
||||
await app.close();
|
||||
});
|
||||
|
||||
test("resolves only a configured selection for the later generation boundary", () => {
|
||||
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm:
|
||||
provider: openai
|
||||
model: gpt-4.1-mini
|
||||
endpoint: {baseUrl: https://api.openai.example/v1, apiVersion: "2026-08-01"}
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`);
|
||||
const models = loadMetadataGenerationModels({ installationFile, secretsFile });
|
||||
|
||||
expect(models.resolve("openai-mini")).toEqual({
|
||||
id: "openai-mini",
|
||||
provider: "openai",
|
||||
model: "gpt-4.1-mini",
|
||||
endpoint: { baseUrl: "https://api.openai.example/v1", apiVersion: "2026-08-01" },
|
||||
apiKeyEnv: "OPENAI_API_KEY",
|
||||
apiKey: "raw-provider-secret",
|
||||
});
|
||||
expect(() => models.resolve("unknown-model")).toThrow(MetadataGenerationModelUnavailableError);
|
||||
});
|
||||
|
||||
test("loads DeepSeek, GLM, and an explicit keyless Qwen endpoint from installation setup", () => {
|
||||
const { installationFile, secretsFile } = metadataConfiguration(`metadataGeneration:
|
||||
default: glm-53
|
||||
models:
|
||||
- id: deepseek-v4-pro
|
||||
label: DeepSeek V4 Pro
|
||||
litellm: {provider: deepseek, model: deepseek-v4-pro}
|
||||
apiKeyEnv: DEEPSEEK_API_KEY
|
||||
- id: glm-53
|
||||
label: GLM 5.3
|
||||
litellm:
|
||||
provider: openai
|
||||
model: glm-5.3
|
||||
endpoint: {baseUrl: https://api.z.ai/api/coding/paas/v4}
|
||||
apiKeyEnv: ZAI_API_KEY
|
||||
- id: qwen-36
|
||||
label: Qwen 3.6
|
||||
litellm:
|
||||
provider: openai
|
||||
model: qwen3.6-35b-a3b
|
||||
disableThinking: true
|
||||
endpoint: {baseUrl: https://models.internal.example/v1}
|
||||
`, "DEEPSEEK_API_KEY=deepseek-secret\nZAI_API_KEY=zai-secret\n");
|
||||
|
||||
const models = loadMetadataGenerationModels({ installationFile, secretsFile });
|
||||
|
||||
expect(models.catalog()).toEqual({
|
||||
models: [
|
||||
{ id: "deepseek-v4-pro", label: "DeepSeek V4 Pro" },
|
||||
{ id: "glm-53", label: "GLM 5.3" },
|
||||
{ id: "qwen-36", label: "Qwen 3.6" },
|
||||
],
|
||||
default: "glm-53",
|
||||
});
|
||||
expect(models.resolve("deepseek-v4-pro")).toMatchObject({
|
||||
apiKeyEnv: "DEEPSEEK_API_KEY",
|
||||
apiKey: "deepseek-secret",
|
||||
});
|
||||
expect(models.resolve("qwen-36")).toEqual({
|
||||
id: "qwen-36",
|
||||
provider: "openai",
|
||||
model: "qwen3.6-35b-a3b",
|
||||
disableThinking: true,
|
||||
endpoint: { baseUrl: "https://models.internal.example/v1" },
|
||||
});
|
||||
});
|
||||
|
||||
test("loads an explicit keyless endpoint without a secret bundle", () => {
|
||||
const { installationFile } = metadataConfiguration(`metadataGeneration:
|
||||
default: qwen-36
|
||||
models:
|
||||
- id: qwen-36
|
||||
label: Qwen 3.6
|
||||
litellm:
|
||||
provider: openai
|
||||
model: qwen3.6-35b-a3b
|
||||
disableThinking: true
|
||||
endpoint: {baseUrl: https://models.internal.example/v1}
|
||||
`);
|
||||
|
||||
expect(loadMetadataGenerationModels({ installationFile }).resolve("qwen-36")).toEqual({
|
||||
id: "qwen-36",
|
||||
provider: "openai",
|
||||
model: "qwen3.6-35b-a3b",
|
||||
disableThinking: true,
|
||||
endpoint: { baseUrl: "https://models.internal.example/v1" },
|
||||
});
|
||||
});
|
||||
|
||||
test.each([
|
||||
["invalid YAML", "metadataGeneration: [\n", "OPENAI_API_KEY=secret\n", /invalid YAML/],
|
||||
["duplicate ids", `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
|
||||
- {id: openai-mini, label: Two, litellm: {provider: openai, model: gpt-4.1}, apiKeyEnv: OPENAI_API_KEY}
|
||||
`, "OPENAI_API_KEY=secret\n", /model id "openai-mini" is duplicated/],
|
||||
["missing default", `metadataGeneration:
|
||||
models:
|
||||
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
|
||||
`, "OPENAI_API_KEY=secret\n", /default is required/],
|
||||
["unknown default", `metadataGeneration:
|
||||
default: absent
|
||||
models:
|
||||
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
|
||||
`, "OPENAI_API_KEY=secret\n", /default "absent" is not configured/],
|
||||
["malformed settings", `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- {id: openai-mini, label: One, litellm: {provider: "open ai", model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
|
||||
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
|
||||
["malformed endpoint", `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: One
|
||||
litellm: {provider: openai, model: gpt-4.1-mini, endpoint: {baseUrl: not-a-url}}
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
|
||||
["keyless hosted model without endpoint", `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}}
|
||||
`, "", /configuration is invalid/],
|
||||
["disable thinking without endpoint", `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: One
|
||||
litellm: {provider: openai, model: gpt-4.1-mini, disableThinking: true}
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`, "OPENAI_API_KEY=secret\n", /configuration is invalid/],
|
||||
["unallowed secret reference", `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: THT_DWH_API_KEY}
|
||||
`, "THT_DWH_API_KEY=secret\n", /configuration is invalid/],
|
||||
["missing referenced secret", `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
|
||||
`, "THT_DWH_API_KEY=secret\n", /secret "OPENAI_API_KEY" is missing/],
|
||||
["unusable referenced secret", `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
|
||||
`, "OPENAI_API_KEY=secret with whitespace\n", /secret "OPENAI_API_KEY" is unusable/],
|
||||
] as const)("rejects %s metadata-generation configuration", (_name, yaml, secrets, expected) => {
|
||||
const { installationFile, secretsFile } = metadataConfiguration(yaml, secrets);
|
||||
expect(() => loadMetadataGenerationModels({ installationFile, secretsFile })).toThrow(expected);
|
||||
});
|
||||
|
||||
test("rejects a missing secret-bundle declaration for configured models", () => {
|
||||
const { installationFile } = metadataConfiguration(`metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- {id: openai-mini, label: One, litellm: {provider: openai, model: gpt-4.1-mini}, apiKeyEnv: OPENAI_API_KEY}
|
||||
`);
|
||||
|
||||
expect(() => loadMetadataGenerationModels({ installationFile }))
|
||||
.toThrow("metadata-generation keyed models require THT_SECRETS_FILE");
|
||||
});
|
||||
@@ -0,0 +1,256 @@
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, expect, test } from "vitest";
|
||||
import {
|
||||
ModelCompletionProviderError,
|
||||
PythonModelCompleter,
|
||||
} from "../src/catalog/model-completer.js";
|
||||
|
||||
const roots: string[] = [];
|
||||
|
||||
afterEach(() => {
|
||||
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function helper(source: string, options: { terminationGraceMs?: number } = {}) {
|
||||
const root = mkdtempSync(join(tmpdir(), "thothii-model-completer-"));
|
||||
roots.push(root);
|
||||
writeFileSync(join(root, "fake_completion_helper.py"), source, "utf8");
|
||||
return new PythonModelCompleter({
|
||||
pythonExecutable: "python3",
|
||||
cwd: root,
|
||||
helperModule: "fake_completion_helper",
|
||||
timeoutMs: 5_000,
|
||||
...options,
|
||||
});
|
||||
}
|
||||
|
||||
async function waitUntil(predicate: () => boolean, timeoutMs = 2_000): Promise<void> {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (!predicate()) {
|
||||
if (Date.now() >= deadline) throw new Error("condition was not met before timeout");
|
||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
}
|
||||
}
|
||||
|
||||
test("uses the short-lived Python helper stdin/stdout protocol without process arguments", async () => {
|
||||
const completer = helper(`
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
request = json.loads(sys.stdin.read())
|
||||
pathlib.Path("request.json").write_text(
|
||||
json.dumps({"request": request, "argv": sys.argv}, sort_keys=True),
|
||||
encoding="utf-8",
|
||||
)
|
||||
sys.stdout.write(json.dumps({"ok": True, "content": "Descrizione italiana"}))
|
||||
`);
|
||||
|
||||
const content = await completer.complete({
|
||||
model: {
|
||||
id: "openai-mini",
|
||||
provider: "openai",
|
||||
model: "gpt-4.1-mini",
|
||||
endpoint: { baseUrl: "https://models.example.test/v1", apiVersion: "2026-08-01" },
|
||||
apiKeyEnv: "OPENAI_API_KEY",
|
||||
apiKey: "test-provider-secret",
|
||||
},
|
||||
messages: [
|
||||
{ role: "system", content: "Return one description." },
|
||||
{ role: "user", content: "Private metadata prompt." },
|
||||
],
|
||||
signal: new AbortController().signal,
|
||||
});
|
||||
|
||||
expect(content).toBe("Descrizione italiana");
|
||||
const captured = JSON.parse(readFileSync(join(roots[0]!, "request.json"), "utf8"));
|
||||
expect(captured.request).toEqual({
|
||||
model: "openai/gpt-4.1-mini",
|
||||
api_key: "test-provider-secret",
|
||||
messages: [
|
||||
{ role: "system", content: "Return one description." },
|
||||
{ role: "user", content: "Private metadata prompt." },
|
||||
],
|
||||
api_base: "https://models.example.test/v1",
|
||||
api_version: "2026-08-01",
|
||||
});
|
||||
expect(JSON.stringify(captured.argv)).not.toMatch(/test-provider-secret|Private metadata prompt/);
|
||||
});
|
||||
|
||||
test("omits api_key for an explicitly configured keyless endpoint", async () => {
|
||||
const completer = helper(`
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
request = json.loads(sys.stdin.read())
|
||||
pathlib.Path("request.json").write_text(json.dumps(request, sort_keys=True), encoding="utf-8")
|
||||
sys.stdout.write(json.dumps({"ok": True, "content": "Descrizione Qwen"}))
|
||||
`);
|
||||
|
||||
await expect(completer.complete({
|
||||
model: {
|
||||
id: "qwen-36",
|
||||
provider: "openai",
|
||||
model: "qwen3.6-35b-a3b",
|
||||
disableThinking: true,
|
||||
endpoint: { baseUrl: "https://models.internal.example/v1" },
|
||||
},
|
||||
messages: [{ role: "user", content: "Describe invented metadata." }],
|
||||
signal: new AbortController().signal,
|
||||
})).resolves.toBe("Descrizione Qwen");
|
||||
|
||||
expect(JSON.parse(readFileSync(join(roots[0]!, "request.json"), "utf8"))).toEqual({
|
||||
model: "openai/qwen3.6-35b-a3b",
|
||||
messages: [{ role: "user", content: "Describe invented metadata." }],
|
||||
api_base: "https://models.internal.example/v1",
|
||||
disable_thinking: true,
|
||||
});
|
||||
});
|
||||
|
||||
test("normalizes helper failures and rejects non-pristine stdout without leaking diagnostics", async () => {
|
||||
const secret = "test-provider-secret";
|
||||
const prompt = "private metadata prompt";
|
||||
const completers = [
|
||||
helper(`
|
||||
import json
|
||||
import sys
|
||||
request = json.loads(sys.stdin.read())
|
||||
print(request["api_key"] + " " + request["messages"][0]["content"], file=sys.stderr)
|
||||
sys.stdout.write(json.dumps({"ok": False, "error": "provider_failure"}))
|
||||
`),
|
||||
helper(`
|
||||
import json
|
||||
import sys
|
||||
sys.stdin.read()
|
||||
sys.stdout.write(json.dumps({"ok": True, "content": "first"}) + "\\n" + json.dumps({"ok": True, "content": "second"}))
|
||||
`),
|
||||
];
|
||||
|
||||
for (const completer of completers) {
|
||||
let failure: unknown;
|
||||
try {
|
||||
await completer.complete({
|
||||
model: {
|
||||
id: "openai-mini",
|
||||
provider: "openai",
|
||||
model: "gpt-4.1-mini",
|
||||
apiKeyEnv: "OPENAI_API_KEY",
|
||||
apiKey: secret,
|
||||
},
|
||||
messages: [{ role: "user", content: prompt }],
|
||||
signal: new AbortController().signal,
|
||||
});
|
||||
} catch (error) {
|
||||
failure = error;
|
||||
}
|
||||
expect(failure).toBeInstanceOf(ModelCompletionProviderError);
|
||||
expect(String(failure)).not.toMatch(new RegExp(`${secret}|${prompt}`));
|
||||
}
|
||||
});
|
||||
|
||||
test("aborting a completion terminates its Python helper and returns a cancellation error", async () => {
|
||||
const completer = helper(`
|
||||
import os
|
||||
import pathlib
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.stdin.read()
|
||||
|
||||
def terminate(_signum, _frame):
|
||||
pathlib.Path("terminated.txt").write_text("SIGTERM", encoding="utf-8")
|
||||
raise SystemExit(0)
|
||||
|
||||
signal.signal(signal.SIGTERM, terminate)
|
||||
pathlib.Path("pid.txt").write_text(str(os.getpid()), encoding="utf-8")
|
||||
while True:
|
||||
time.sleep(0.05)
|
||||
`);
|
||||
const controller = new AbortController();
|
||||
const completion = completer.complete({
|
||||
model: {
|
||||
id: "openai-mini",
|
||||
provider: "openai",
|
||||
model: "gpt-4.1-mini",
|
||||
apiKeyEnv: "OPENAI_API_KEY",
|
||||
apiKey: "test-provider-secret",
|
||||
},
|
||||
messages: [{ role: "user", content: "Private metadata prompt." }],
|
||||
signal: controller.signal,
|
||||
});
|
||||
const observed = completion.then(
|
||||
() => undefined,
|
||||
(error: unknown) => error,
|
||||
);
|
||||
const root = roots[0]!;
|
||||
await waitUntil(() => existsSync(join(root, "pid.txt")));
|
||||
const pid = Number(readFileSync(join(root, "pid.txt"), "utf8"));
|
||||
|
||||
controller.abort();
|
||||
|
||||
await expect(observed).resolves.toMatchObject({ name: "ModelCompletionCancelledError" });
|
||||
await waitUntil(() => {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return false;
|
||||
} catch {
|
||||
return true;
|
||||
}
|
||||
});
|
||||
expect(readFileSync(join(root, "terminated.txt"), "utf8")).toBe("SIGTERM");
|
||||
});
|
||||
|
||||
test("aborting escalates to SIGKILL when the Python helper does not exit after SIGTERM", async () => {
|
||||
const completer = helper(`
|
||||
import os
|
||||
import pathlib
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.stdin.read()
|
||||
|
||||
def ignore_term(_signum, _frame):
|
||||
pathlib.Path("sigterm.txt").write_text("received", encoding="utf-8")
|
||||
|
||||
signal.signal(signal.SIGTERM, ignore_term)
|
||||
pathlib.Path("pid.txt").write_text(str(os.getpid()), encoding="utf-8")
|
||||
while True:
|
||||
time.sleep(0.05)
|
||||
`, { terminationGraceMs: 25 });
|
||||
const controller = new AbortController();
|
||||
const observed = completer.complete({
|
||||
model: {
|
||||
id: "openai-mini",
|
||||
provider: "openai",
|
||||
model: "gpt-4.1-mini",
|
||||
apiKeyEnv: "OPENAI_API_KEY",
|
||||
apiKey: "test-provider-secret",
|
||||
},
|
||||
messages: [{ role: "user", content: "Private metadata prompt." }],
|
||||
signal: controller.signal,
|
||||
}).then(
|
||||
() => undefined,
|
||||
(error: unknown) => error,
|
||||
);
|
||||
const root = roots[0]!;
|
||||
await waitUntil(() => existsSync(join(root, "pid.txt")));
|
||||
const pid = Number(readFileSync(join(root, "pid.txt"), "utf8"));
|
||||
|
||||
controller.abort();
|
||||
|
||||
await expect(observed).resolves.toMatchObject({ name: "ModelCompletionCancelledError" });
|
||||
expect(readFileSync(join(root, "sigterm.txt"), "utf8")).toBe("received");
|
||||
await waitUntil(() => {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return false;
|
||||
} catch {
|
||||
return true;
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -2,7 +2,12 @@ import { afterEach, expect, test } from "vitest";
|
||||
import { chmodSync, mkdtempSync, renameSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { loadSecretBundle, loadSecretBundleWithFs, secretValue } from "../src/config/secret-bundle.js";
|
||||
import {
|
||||
loadSecretBundle,
|
||||
loadSecretBundleWithFs,
|
||||
METADATA_GENERATION_SECRET_KEYS,
|
||||
secretValue,
|
||||
} from "../src/config/secret-bundle.js";
|
||||
|
||||
const dirs: string[] = [];
|
||||
afterEach(() => { for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true, force: true }); });
|
||||
@@ -22,6 +27,15 @@ test("parses comments, blank lines and values containing equals", () => {
|
||||
]));
|
||||
});
|
||||
|
||||
test("existing secret consumers accept a bundle containing an allowed metadata-model key", () => {
|
||||
const file = bundle("THT_DWH_API_KEY=dwh-secret\nOPENAI_API_KEY=metadata-secret\n");
|
||||
expect(secretValue({ secretsFile: file }, "THT_DWH_API_KEY")).toBe("dwh-secret");
|
||||
});
|
||||
|
||||
test.each(METADATA_GENERATION_SECRET_KEYS)("accepts audited metadata-model key %s", (name) => {
|
||||
expect(loadSecretBundle(bundle(`${name}=metadata-secret\n`)).get(name)).toBe("metadata-secret");
|
||||
});
|
||||
|
||||
test("accepts the fixed OIDC and Authentik secret references", () => {
|
||||
const file = bundle("THT_OIDC_CLIENT_SECRET=oidc-secret\nTHT_AUTHENTIK_API_TOKEN=authentik-token\n");
|
||||
expect(loadSecretBundle(file)).toEqual(new Map([
|
||||
|
||||
@@ -22,6 +22,7 @@ services:
|
||||
THT_WORKSPACE_SECRET_RUNTIME_ROOT: /tmp/thothii-workspace-secrets
|
||||
THT_WORKSPACE_SECRET_ROOTS: /run/secrets
|
||||
THT_SECRETS_FILE: /run/secrets/thothii.secrets
|
||||
THT_INSTALLATION_CONFIG_FILE: /run/thothii-installation/thothii-installation.yaml
|
||||
THT_PI_AUTH_FILE: /home/thoth/.pi/agent/auth.json
|
||||
THT_AUTH_CONFIG_FILE: /run/thothii-auth/auth.yaml
|
||||
THT_AUTH_STATE_ROOT: /data/auth
|
||||
@@ -53,6 +54,9 @@ services:
|
||||
- source: thothii_secrets
|
||||
target: thothii.secrets
|
||||
- catalog_runtime_password
|
||||
configs:
|
||||
- source: thothii_installation_config
|
||||
target: /run/thothii-installation/thothii-installation.yaml
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"]
|
||||
interval: 15s
|
||||
@@ -271,3 +275,7 @@ secrets:
|
||||
file: "${THT_CATALOG_RUNTIME_PASSWORD_SOURCE:-./deploy/secrets/catalog-runtime-password}"
|
||||
catalog_migrator_password:
|
||||
file: "${THT_CATALOG_MIGRATOR_PASSWORD_SOURCE:-./deploy/secrets/catalog-migrator-password}"
|
||||
|
||||
configs:
|
||||
thothii_installation_config:
|
||||
file: "${THT_INSTALLATION_CONFIG_SOURCE:?set THT_INSTALLATION_CONFIG_SOURCE}"
|
||||
|
||||
Vendored
+1
@@ -6,6 +6,7 @@ THOTH_CORE_HTTP_PORT=8787
|
||||
MAX_PI_PROCESSES=4
|
||||
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
|
||||
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
|
||||
THT_INSTALLATION_CONFIG_SOURCE=/absolute/path/to/thothii-installation.yaml
|
||||
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
|
||||
THT_CATALOG_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/catalog-runtime-password
|
||||
THT_CATALOG_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/catalog-migrator-password
|
||||
|
||||
Vendored
+1
@@ -5,6 +5,7 @@ THOTH_HTTP_PORT=8080
|
||||
MAX_PI_PROCESSES=4
|
||||
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
|
||||
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
|
||||
THT_INSTALLATION_CONFIG_SOURCE=/absolute/path/to/thothii-installation.yaml
|
||||
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
|
||||
THT_CATALOG_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/catalog-runtime-password
|
||||
THT_CATALOG_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/catalog-migrator-password
|
||||
|
||||
@@ -8,6 +8,7 @@ THT_WORKSPACE_GIT_KNOWN_HOSTS_FILE=<abs>/deploy/psd/secrets/git-known-hosts
|
||||
|
||||
# App
|
||||
THT_SECRETS_FILE=<abs>/deploy/psd/secrets/thothii.secrets
|
||||
THT_INSTALLATION_CONFIG_SOURCE=<abs>/deploy/psd/thothii-installation.yaml
|
||||
PI_AUTH_FILE=<abs>/deploy/psd/secrets/pi-auth.json
|
||||
THT_AUTH_CONFIG_ROOT=<abs>/deploy/psd/auth
|
||||
# DWH and Evidence credentials are entered later in Workspace management and stored encrypted
|
||||
|
||||
@@ -8,6 +8,32 @@ workspaceRepository:
|
||||
remote: git@github.com:mptyl/tht-workspace-psd.git
|
||||
branch: main
|
||||
access: ssh
|
||||
metadataGeneration:
|
||||
default: glm-53
|
||||
models:
|
||||
- id: deepseek-v4-pro
|
||||
label: DeepSeek V4 Pro
|
||||
litellm:
|
||||
provider: deepseek
|
||||
model: deepseek-v4-pro
|
||||
apiKeyEnv: DEEPSEEK_API_KEY
|
||||
- id: glm-53
|
||||
label: GLM 5.3
|
||||
litellm:
|
||||
provider: openai
|
||||
model: glm-5.3
|
||||
endpoint:
|
||||
baseUrl: https://api.z.ai/api/coding/paas/v4
|
||||
apiKeyEnv: ZAI_API_KEY
|
||||
- id: qwen-36
|
||||
label: AritmoLab Qwen 3.6 35B A3B
|
||||
litellm:
|
||||
provider: openai
|
||||
model: qwen3.6-35b-a3b
|
||||
disableThinking: true
|
||||
endpoint:
|
||||
baseUrl: https://ml-aritmolab.policlinicosandonato.it/v1
|
||||
# Qwen omette apiKeyEnv: l'endpoint VPN non autentica le richieste.
|
||||
authentication:
|
||||
configDirectory: "<abs>/projects/ThothII/deploy/psd/auth"
|
||||
overrides:
|
||||
|
||||
@@ -9,11 +9,24 @@ chmod 600 deploy/secrets/thothii.secrets
|
||||
```
|
||||
|
||||
The file uses strict `KEY=VALUE` lines (comments and blank lines are allowed). The supported
|
||||
keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`,
|
||||
`THT_OIDC_CLIENT_SECRET`, and `THT_AUTHENTIK_API_TOKEN`. The two authentication keys are fixed
|
||||
empty entries for local authentication and must be populated only in a protected OIDC installation.
|
||||
Other configured values must be non-empty and contain no whitespace. Do not put secrets
|
||||
in the root `.env`, workspace YAML, URLs, logs, or rendered Compose output.
|
||||
installation keys are `THT_MODEL_API_KEY`, `THT_DWH_API_KEY`, `THT_CA`, `THT_SSL_CA`,
|
||||
`THT_OIDC_CLIENT_SECRET`, and `THT_AUTHENTIK_API_TOKEN`. Metadata-generation models may reference
|
||||
exactly one of `THT_METADATA_API_KEY`, `ANTHROPIC_API_KEY`, `AZURE_API_KEY`, `GEMINI_API_KEY`,
|
||||
`DEEPSEEK_API_KEY`, `OPENAI_API_KEY`, `OPENROUTER_API_KEY`, or `ZAI_API_KEY` through their
|
||||
descriptor `apiKeyEnv`. An entry for an explicitly configured endpoint that accepts unauthenticated
|
||||
requests may omit `apiKeyEnv`; hosted/default endpoints must always reference a key.
|
||||
OpenAI-compatible Qwen endpoints that emit reasoning in `content` can additionally set
|
||||
`litellm.disableThinking: true`; the adapter sends the server's bounded chat-template flag so the
|
||||
strict JSON result remains parseable.
|
||||
The two authentication keys must be omitted until they have non-empty values in a protected OIDC
|
||||
installation. Other configured values must be non-empty and contain no
|
||||
whitespace. Do not put secrets in the root `.env`, installation YAML, workspace YAML, URLs, logs,
|
||||
or rendered Compose output.
|
||||
|
||||
`THT_MODEL_API_KEY` remains the generic Pi child credential. Metadata generation is a separate
|
||||
backend-owned runtime and reads only the key named by its own `metadataGeneration.models[].apiKeyEnv`
|
||||
(when present);
|
||||
it does not read Pi settings, `PI_AUTH_FILE`, or workspace `llm_policy`.
|
||||
|
||||
Do not add vector or embedding endpoint credentials to the bundle. Active operator manuals use
|
||||
internal Qdrant and Ollama services, so vector/embedding runtime endpoint secrets are not part of
|
||||
|
||||
@@ -5,12 +5,18 @@
|
||||
# Hosted model provider (single-key providers only).
|
||||
# THT_MODEL_API_KEY=replace-me
|
||||
|
||||
# Description Generation only. The selected name must match apiKeyEnv in metadataGeneration.
|
||||
# Allowed names: THT_METADATA_API_KEY, ANTHROPIC_API_KEY, AZURE_API_KEY, GEMINI_API_KEY,
|
||||
# DEEPSEEK_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY, or ZAI_API_KEY.
|
||||
# OPENAI_API_KEY=replace-me
|
||||
# A model with an explicit unauthenticated endpoint omits apiKeyEnv and needs no bundle entry.
|
||||
|
||||
# External DWH adapter.
|
||||
# THT_DWH_API_KEY=replace-me
|
||||
|
||||
# Optional CA material/path understood by the configured adapter.
|
||||
# THT_CA=/run/secrets/ca-chain.pem
|
||||
|
||||
# OIDC/Authentik references. Keep these fixed keys empty until OIDC is configured.
|
||||
THT_OIDC_CLIENT_SECRET=
|
||||
THT_AUTHENTIK_API_TOKEN=
|
||||
# OIDC/Authentik references. Uncomment only with non-empty values when configured.
|
||||
# THT_OIDC_CLIENT_SECRET=replace-me
|
||||
# THT_AUTHENTIK_API_TOKEN=replace-me
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# Use one sequential description-generation run
|
||||
|
||||
AI description generation is an infrequent, administrator-triggered catalog operation. The
|
||||
backend therefore owns one installation-wide asynchronous run and processes its model requests
|
||||
sequentially. A second generation start is rejected while that run is active. The existing
|
||||
catalog-operation coordinator also reserves the target Workspace Database so synchronization,
|
||||
cleanup, and catalog edits cannot overlap the run.
|
||||
|
||||
Each model completion is performed by a short-lived internal Python process using LiteLLM. The
|
||||
backend sends a structured request on stdin, reads pristine structured output from stdout, and
|
||||
keeps diagnostics on stderr. This helper is neither an HTTP service nor a user-facing CLI. Using
|
||||
Pi for this operation would couple a deterministic batch task to interactive session lifecycle and
|
||||
gate behavior without adding product value; the helper reuses Python already present in the core
|
||||
image while keeping the integration small.
|
||||
|
||||
Installation model entries normally reference a protected API key. A reference may be omitted
|
||||
only for an explicit unauthenticated endpoint; the helper uses a fixed non-secret client
|
||||
placeholder because OpenAI-compatible SDKs require a non-empty client value even when the server
|
||||
ignores authentication.
|
||||
For an explicitly configured Qwen-compatible endpoint, `disableThinking: true` maps to the narrow
|
||||
chat-template option that prevents reasoning prose from surrounding the required JSON result.
|
||||
|
||||
Persistence is deliberately limited to one run record, ordered text events, and the Generated
|
||||
Description written to each target as soon as it succeeds. There are no durable per-target jobs,
|
||||
leases, invocation records, automatic resume, or distributed locks. On backend startup, any run
|
||||
still recorded as queued or running becomes interrupted. An administrator continues by starting
|
||||
Generate Missing, and may use Unlock only when no live generation process exists. Cancellation
|
||||
stops the loop and terminates the current helper process.
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
status: accepted
|
||||
---
|
||||
|
||||
# Allow bounded real source samples for description generation
|
||||
|
||||
Description generation may send up to five real rows and up to five distinct non-null example
|
||||
values for relevant columns to the configured model provider, preserving the useful behavior of
|
||||
ThothAI. Samples are read only for the current request, treated as untrusted data, bounded before
|
||||
prompt construction, and never stored in generation runs, events, or catalog metadata.
|
||||
|
||||
This first slice makes that behavior explicit in the UI and operator documentation. A follow-up
|
||||
Sensitive Data Policy is required to classify protected fields and exclude or anonymize their
|
||||
values before model calls. Until that policy exists, administrators must regard generation as a
|
||||
controlled disclosure of sampled source data to the selected provider.
|
||||
@@ -33,7 +33,7 @@ The production role expansion from `backend/src/auth/config.ts` is exact:
|
||||
| Role | Permissions |
|
||||
|---|---|
|
||||
| `user` | `session.use` |
|
||||
| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, `workspace.manage`, `workspace.secrets.manage`, `pi.manage`, `auth.diagnostics.read` |
|
||||
| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, `workspace.manage`, `workspace.secrets.manage`, `database.manage`, `pi.manage`, `auth.diagnostics.read` |
|
||||
|
||||
`admin` therefore includes the ordinary `session.use` permission. No other role or permission
|
||||
label is part of the production catalog.
|
||||
|
||||
@@ -22,7 +22,9 @@ flowchart LR
|
||||
THT --> VDB["Qdrant / vector store"]
|
||||
BE --> CFG["settings.json\nworkspace registry"]
|
||||
BE --> CAT["catalog-db\nPostgreSQL + Kysely"]
|
||||
BE -->|catalog Test + Table Sync| DWH
|
||||
BE -->|catalog Test + Sync + bounded AI sampling| DWH
|
||||
BE -->|one request per subprocess| LLMHELPER["LiteLLM helper\nPython, short-lived"]
|
||||
LLMHELPER -->|configured model| PROVIDER["AI provider"]
|
||||
FE -.->|renders widgets| EXT
|
||||
```
|
||||
|
||||
@@ -31,7 +33,7 @@ Dipendenze principali:
|
||||
| Module | Depends on | Responsibility |
|
||||
| --- | --- | --- |
|
||||
| `frontend/` | Backend REST and SSE APIs | UI, gate widgets, and in-memory transcript |
|
||||
| `backend/src/` | Pi, `tht`, configuration, workspace registry, catalog PostgreSQL, and read-only DWH connectors | Transport, session lifecycle, catalog CRUD, connection tests, table introspection, and APIs |
|
||||
| `backend/src/` | Pi, `tht`, configuration, workspace registry, catalog PostgreSQL, read-only DWH connectors, and the internal LiteLLM helper | Transport, session lifecycle, catalog CRUD, connection tests, table introspection, sequential AI description generation, and APIs |
|
||||
| `harness/.pi/` | Pi and `tht phase` | Workflow orchestration and human-in-the-loop gates |
|
||||
| `harness/tht/` | Filesystem, DWH, and vector store | Persistence, CLI, Evidence, schema, and preprocessing |
|
||||
| workspace repository | `source/`, `curated/`, manifest, and artifacts | Versioned Evidence source and session output |
|
||||
@@ -65,6 +67,20 @@ sequenceDiagram
|
||||
|
||||
The backend uses `ThtRunner` for CLI subprocesses, `PiProcessManager` for one Pi process per session, `SessionBridge` to adapt RPC events, and `SseHub` to distribute them to clients.
|
||||
|
||||
## Catalog description generation
|
||||
|
||||
Catalog description generation is a backend-owned administrative operation, separate from the
|
||||
Pi session workflow and from the public `tht` CLI. The frontend starts one run for selected catalog
|
||||
tables or columns. A single installation-wide worker processes targets sequentially, reads at most
|
||||
the configured bounded sample from the source DWH through its read-only connection, and invokes a
|
||||
short-lived Python LiteLLM helper once per target. The selected model comes from the installation
|
||||
descriptor; its API key remains in the protected installation secret bundle.
|
||||
|
||||
Each result is written immediately to `Generated Description`. Run state and sanitized activity
|
||||
events are stored in `catalog-db` and exposed to the drawer through REST and SSE. An administrator
|
||||
may later copy selected generated descriptions into `Description`. There is no parallel run queue,
|
||||
automatic retry policy, or second orchestration subsystem.
|
||||
|
||||
## Main backend classes
|
||||
|
||||
The diagram shows the classes that form the bridge between the browser, Pi, and `tht`. Fastify routes receive requests and delegate to these services.
|
||||
|
||||
@@ -12,6 +12,8 @@ flowchart LR
|
||||
USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
|
||||
FE --> BE["Backend\nFastify"]
|
||||
BE --> CATALOG["Metadata catalog\nPostgreSQL"]
|
||||
BE --> MODEL["Configured AI model\nvia short-lived LiteLLM helper"]
|
||||
BE -->|bounded read-only samples| DWH
|
||||
BE --> PI["Pi\nRPC per sessione"]
|
||||
PI --> THT["tht and harness\nworkflow and persistence"]
|
||||
THT --> DWH["DWH\nread only"]
|
||||
@@ -50,12 +52,19 @@ A session is a directory under `sessions/` (the workspace defines the path): `se
|
||||
configurations stored in PostgreSQL through Kysely.
|
||||
- `CatalogTableService` reconciles persisted Catalog Tables with a successful external schema scan;
|
||||
`ConcreteCatalogTableIntrospector` isolates direct PostgreSQL, typed REST, and SSH-tunnel access.
|
||||
- `DescriptionGenerationWorker` admits one installation-wide run and processes its table or column
|
||||
targets sequentially.
|
||||
- `PostgresDescriptionSourceSampler` reads bounded real rows and five representative examples from
|
||||
the configured DWH connection; `ProcessModelCompleter` invokes the short-lived Python LiteLLM
|
||||
helper with the installation-selected model.
|
||||
|
||||
Application settings remain in `backend/data/settings.json`; session state remains in harness phase
|
||||
documents. PostgreSQL stores only the administrative database catalog, bindings, observed tables,
|
||||
and curated descriptions. Connector secrets remain write-only in the encrypted workspace secret
|
||||
store. Catalog SSH support is limited to connection tests and table synchronization; it does not
|
||||
change the session runtime binding contract.
|
||||
curated and generated descriptions, description-generation runs, and sanitized run events.
|
||||
Connector secrets remain write-only in the encrypted workspace secret store; model credentials
|
||||
remain in the protected installation secret bundle. Catalog SSH support is limited to connection
|
||||
tests, table synchronization, and bounded description-generation sampling; it does not change the
|
||||
session runtime binding contract.
|
||||
|
||||
## Human-in-the-loop gate contract
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# Copy this file to an operator-controlled path named exactly thothii-installation.yaml.
|
||||
# Copy this file to a protected operator-controlled path named exactly thothii-installation.yaml
|
||||
# and set mode 0600 (or 0400) before using it as THT_INSTALLATION_CONFIG_SOURCE.
|
||||
# Replace every absolute placeholder. Select exactly one Git transport override.
|
||||
profile: local
|
||||
projectDirectory: "/absolute/path/to/ThothII"
|
||||
@@ -7,6 +8,17 @@ workspaceRepository:
|
||||
remote: git@git.example.com:organization/workspaces.git
|
||||
branch: main
|
||||
access: ssh
|
||||
metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm:
|
||||
provider: openai
|
||||
model: gpt-4.1-mini
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
# apiKeyEnv may be omitted only for an explicit endpoint that accepts
|
||||
# unauthenticated requests.
|
||||
authentication:
|
||||
configDirectory: "/absolute/path/to/thothii-auth"
|
||||
overrides:
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# Copy this file to a protected operator path named exactly thothii-installation.yaml.
|
||||
# Copy this file to a protected operator path named exactly thothii-installation.yaml
|
||||
# and set mode 0600 (or 0400) before using it as THT_INSTALLATION_CONFIG_SOURCE.
|
||||
# Replace every absolute placeholder. Select exactly one Git transport override.
|
||||
profile: server
|
||||
projectDirectory: "/absolute/path/to/ThothII"
|
||||
@@ -7,6 +8,19 @@ workspaceRepository:
|
||||
remote: git@git.example.com:organization/workspaces.git
|
||||
branch: main
|
||||
access: ssh
|
||||
metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm:
|
||||
provider: openai
|
||||
model: gpt-4.1-mini
|
||||
endpoint:
|
||||
baseUrl: https://api.openai.example/v1
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
# apiKeyEnv may be omitted only for an explicit endpoint that accepts
|
||||
# unauthenticated requests.
|
||||
authentication:
|
||||
# Root-operated source of truth; it is never mounted into core.
|
||||
configDirectory: "/srv/example/thothii/auth-canonical"
|
||||
|
||||
@@ -42,6 +42,10 @@ cp deploy/env/local.env.example deploy/env/local.env
|
||||
cp deploy/secrets/thothii.secrets.example deploy/secrets/thothii.secrets
|
||||
chmod 600 deploy/secrets/thothii.secrets
|
||||
|
||||
# Copy docs/install/examples/thothii-installation.local.yaml to a protected operator path,
|
||||
# replace every placeholder, chmod it 600, then set that exact path as
|
||||
# THT_INSTALLATION_CONFIG_SOURCE in deploy/env/local.env.
|
||||
|
||||
docker compose --env-file deploy/env/local.env \
|
||||
-f compose.yaml -f deploy/compose.local.yaml up --build -d
|
||||
```
|
||||
@@ -50,6 +54,7 @@ Set these values in `deploy/env/local.env`:
|
||||
|
||||
- `PI_AUTH_FILE`
|
||||
- `THT_SECRETS_FILE`
|
||||
- `THT_INSTALLATION_CONFIG_SOURCE` (the exact protected host `thothii-installation.yaml`)
|
||||
- `THT_WORKSPACE_GIT_REMOTE`
|
||||
- DWH endpoint
|
||||
- LLM endpoint
|
||||
@@ -64,8 +69,20 @@ The documented and supported bundle keys are:
|
||||
```dotenv
|
||||
THT_MODEL_API_KEY=...
|
||||
THT_DWH_API_KEY=...
|
||||
OPENAI_API_KEY=...
|
||||
```
|
||||
|
||||
`THT_MODEL_API_KEY` remains Pi-only. A metadata-generation model instead references one audited
|
||||
bundle name from `deploy/secrets/README.md` through `metadataGeneration.models[].apiKeyEnv`.
|
||||
`apiKeyEnv` may be omitted only for an explicit endpoint that accepts unauthenticated requests;
|
||||
hosted/default endpoints remain keyed.
|
||||
Provider/model/endpoint settings stay in the protected installation descriptor; raw keys do not.
|
||||
|
||||
Compose mounts exactly `THT_INSTALLATION_CONFIG_SOURCE` into `core` as a read-only config and sets
|
||||
the backend-only runtime path `THT_INSTALLATION_CONFIG_FILE` to
|
||||
`/run/thothii-installation/thothii-installation.yaml`. Do not set the runtime path in the host env.
|
||||
Descriptor and bundle changes are loaded only after application restart.
|
||||
|
||||
A private PEM CA remains outside the bundle and must be mounted through a reviewed Compose override.
|
||||
|
||||
## Preprocessing
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
# AI-generated descriptions for Catalog Tables and Catalog Columns
|
||||
|
||||
## Problem Statement
|
||||
|
||||
ThothII already stores a Generated Description separately from the curated Description for Catalog
|
||||
Tables and Catalog Columns, but administrators cannot populate it with AI. ThothAI provides the
|
||||
useful core workflow—generate table and column comments from schema context and small real-data
|
||||
samples—but its execution, configuration, and interaction model cannot be copied directly into
|
||||
ThothII.
|
||||
|
||||
Administrators need an asynchronous workflow integrated into Database Management. They must be
|
||||
able to choose an installation-approved model, generate descriptions for selected or missing
|
||||
targets, observe understandable progress, stop or recover a stuck operation, review generated
|
||||
text, and explicitly consolidate it. The solution must retain ThothAI's practical simplicity and
|
||||
must not introduce a general job platform, model gateway, distributed scheduler, or competing
|
||||
user-facing CLI.
|
||||
|
||||
## Solution
|
||||
|
||||
Add Description Generation to Database Management as one installation-wide, sequential background
|
||||
run owned by the Fastify backend. The browser starts a run and remains responsive while the backend
|
||||
processes bounded requests one at a time. Each completion is delegated to a short-lived internal
|
||||
Python helper using LiteLLM. Models, their default, and any API-key secret references are declared in
|
||||
application setup YAML and are independent of both workspaces and Pi configuration.
|
||||
|
||||
Each valid result is written immediately to the target's Generated Description. A minimal run row
|
||||
and ordered text events provide status, counters, history, and a live log. A stopped or crashed run
|
||||
is not resumed automatically; completed results remain in place and Generate Missing supplies the
|
||||
simple recovery path. An Unlock action marks a stale recorded run interrupted only when no helper
|
||||
or backend generation loop is alive.
|
||||
|
||||
Prompts use catalog context and, when available, no more than five real rows and five representative
|
||||
non-null examples. Samples are transient and never logged or persisted. A valid inability to infer
|
||||
a description produces a standard application-localized value such as `Non generabile`; provider,
|
||||
timeout, and response-validation failures remain technical errors.
|
||||
|
||||
Generated text remains separate from Description until an administrator uses the existing
|
||||
checkbox selection and Actions control to consolidate it. Consolidation retains Generated
|
||||
Description and never writes comments to the external Workspace Database.
|
||||
|
||||
## User Stories
|
||||
|
||||
1. As an installation operator, I want to declare the models allowed for metadata generation in setup YAML, so that model availability is controlled centrally.
|
||||
2. As an installation operator, I want to declare one default metadata-generation model, so that administrators begin with a safe operational choice.
|
||||
3. As an installation operator, I want each model to reference its own API-key secret, so that credentials are not stored in workspaces or browser-visible settings.
|
||||
4. As an installation operator, I want metadata-generation models to remain independent of Pi models, so that changing this workflow cannot disrupt the core NL-to-SQL experience.
|
||||
5. As an installation operator, I want invalid model setup to fail validation clearly, so that the application does not start with ambiguous provider behavior.
|
||||
6. As a Catalog Administrator, I want generation controls to explain when no model is configured, so that I know why the action is unavailable.
|
||||
7. As a Catalog Administrator, I want to select an approved model from a selector initialized to the setup default, so that I control which model performs the work.
|
||||
8. As a Catalog Administrator, I want to generate descriptions for selected Catalog Tables, so that I can work on a focused part of the catalog.
|
||||
9. As a Catalog Administrator, I want to generate descriptions for selected Catalog Columns, so that I can work on individual fields without regenerating a whole table.
|
||||
10. As a Catalog Administrator, I want to generate all eligible descriptions for a Workspace Database, so that I can initialize a catalog in one operation.
|
||||
11. As a Catalog Administrator, I want to generate only missing descriptions, so that I can continue interrupted work without replacing completed proposals.
|
||||
12. As a Catalog Administrator, I want a full run to process Catalog Columns before their Catalog Tables, so that table descriptions can benefit from column descriptions.
|
||||
13. As a Catalog Administrator, I want generation to run asynchronously after I start it, so that the browser remains usable and progress is not tied to one HTTP request.
|
||||
14. As a Catalog Administrator, I want only one Description Generation Run active in the installation, so that provider traffic and operational behavior remain predictable.
|
||||
15. As a Catalog Administrator, I want a second start attempt to return a clear conflict, so that I cannot accidentally overlap generation runs.
|
||||
16. As a Catalog Administrator, I want synchronization, cleanup, consolidation, and edits for the target Workspace Database blocked during generation, so that the simple sequential run sees stable catalog state.
|
||||
17. As a Catalog Administrator, I want to see the run's model, scope, status, counters, and timestamps, so that I understand what is happening.
|
||||
18. As a Catalog Administrator, I want a chronological text log, so that I can follow completed targets and diagnose errors.
|
||||
19. As a Catalog Administrator, I want live log updates with a polling fallback, so that temporary SSE problems do not hide run progress.
|
||||
20. As a Catalog Administrator, I want completed and interrupted runs to remain inspectable, so that I can understand prior activity.
|
||||
21. As a Catalog Administrator, I want to stop an active run, so that I can halt an incorrect or unexpectedly costly operation.
|
||||
22. As a Catalog Administrator, I want stopping a run to terminate its current model helper and prevent later targets from starting, so that stop has prompt operational effect.
|
||||
23. As a Catalog Administrator, I want valid results completed before a stop or failure to remain saved, so that useful work is not discarded.
|
||||
24. As a Catalog Administrator, I want a run left active by a backend restart to become interrupted, so that the UI does not claim nonexistent work is still running.
|
||||
25. As a Catalog Administrator, I want to unlock a stale active run when no generation process is alive, so that an erroneous recorded lock cannot block future work.
|
||||
26. As a Catalog Administrator, I want Unlock rejected while a live generation process exists, so that recovery cannot create an overlapping run.
|
||||
27. As a Catalog Administrator, I want Generate Missing to continue after interruption, so that recovery does not require a special resume mechanism.
|
||||
28. As a Catalog Administrator, I want one retry for a transient model failure, so that a brief provider fault does not immediately lose a batch.
|
||||
29. As a Catalog Administrator, I want the run to fail after three consecutive technical failures, so that a broken provider does not generate an unbounded stream of attempts.
|
||||
30. As a Catalog Administrator, I want a successful request to reset the consecutive-failure count, so that isolated errors do not prematurely stop a useful run.
|
||||
31. As a Catalog Administrator, I want a completed-with-errors result when isolated batches fail but the run reaches its end, so that partial problems remain visible.
|
||||
32. As a Catalog Administrator, I want no automatic fallback to a different model, so that the selected model remains truthful and predictable.
|
||||
33. As a Catalog Administrator, I want malformed or ambiguous model output rejected without writing it, so that descriptions cannot be assigned to the wrong target.
|
||||
34. As a Catalog Administrator, I want an inability to infer a description represented by standard localized text, so that every valid outcome is understandable in the workspace language.
|
||||
35. As a Catalog Administrator, I want technical failures kept distinct from non-generatable outcomes, so that provider problems are not mistaken for catalog knowledge.
|
||||
36. As a Catalog Administrator, I want generated prose written in the workspace language, so that it matches the catalog's intended audience.
|
||||
37. As a Catalog Administrator, I want generated text stored separately from curated Description, so that AI output remains a reviewable proposal.
|
||||
38. As a Catalog Administrator, I want to edit a Generated Description manually, so that I can improve a proposal before consolidation.
|
||||
39. As a Catalog Administrator, I want to select one or more tables or columns and run “Move generated description to Description” from the existing Actions control, so that review remains integrated into the current grids.
|
||||
40. As a Catalog Administrator, I want consolidation to retain the Generated Description, so that I can still see the proposal from which the curated text was copied.
|
||||
41. As a Catalog Administrator, I want selected records without a Generated Description skipped and reported, so that the bulk action does not erase curated text.
|
||||
42. As a Catalog Administrator, I want consolidation and generation to modify only the Metadata Catalog, so that no external database comment is changed.
|
||||
43. As a Catalog Administrator, I want prompts to use schema facts and existing catalog text, so that generated descriptions are grounded in available metadata.
|
||||
44. As a Catalog Administrator, I want prompts to use at most five real source rows and five representative values when available, so that the model has useful examples without unbounded disclosure.
|
||||
45. As a Catalog Administrator, I want to be warned that real source samples are sent to the selected provider, so that I can make an informed disclosure decision.
|
||||
46. As a Catalog Administrator, I want sampled rows and values excluded from persistence and logs, so that operational history does not become a secondary data store.
|
||||
47. As a security operator, I want API keys, prompts, samples, and complete provider payloads redacted from logs, so that diagnostics do not leak secrets or source data.
|
||||
48. As a support operator, I want concise per-target and per-batch event messages, so that failures can be diagnosed without provider-specific internals.
|
||||
49. As an authorized administrator, I want all generation, cancellation, unlock, and consolidation actions protected by database-management permission, so that ordinary users cannot mutate catalog metadata.
|
||||
50. As an unauthorized user, I want generation controls hidden or disabled and API calls rejected, so that frontend visibility is not treated as authorization.
|
||||
51. As an operator, I want setup changes to take effect after an application restart, so that configuration lifecycle remains simple and explicit.
|
||||
52. As a product owner, I want the first release to avoid queues, parallel calls, distributed locks, and automatic resume, so that effort remains focused on generating and reviewing useful descriptions.
|
||||
|
||||
## Implementation Decisions
|
||||
|
||||
- The Fastify backend owns one installation-wide Description Generation Run and its sequential
|
||||
processing loop. It does not delegate lifecycle ownership to Pi or Python.
|
||||
- A Description Generation Run has one of `queued`, `running`, `completed`,
|
||||
`completed_with_errors`, `cancelled`, `failed`, or `interrupted`. It stores the Workspace
|
||||
Database, requested scope, selected model identifier, workspace language, progress counters,
|
||||
timestamps, and an optional final error summary.
|
||||
- Ordered Description Generation Events store timestamp, severity, and safe human-readable text.
|
||||
No durable per-target jobs, model invocation rows, prompt snapshots, sample snapshots, leases,
|
||||
heartbeats, registry revisions, or provenance chains are introduced.
|
||||
- Starting a run schedules an in-process background loop and returns the run immediately. The API
|
||||
exposes start, run/history lookup, event listing and streaming, cancellation, stale-run unlock,
|
||||
and the safe list of configured model choices. There are no retry-item or resume endpoints.
|
||||
- One in-memory generation manager enforces the installation-wide active-run rule. The existing
|
||||
Catalog Operation Coordinator reserves the target Workspace Database for the duration of the
|
||||
run, without being generalized into a new operation framework.
|
||||
- On backend startup, persisted `queued` or `running` Description Generation Runs become
|
||||
`interrupted`. The application performs no automatic replay or resume.
|
||||
- Unlock succeeds only when no live generation loop or helper child exists. It marks the stale run
|
||||
interrupted and releases the local reservation; it is not a distributed lock recovery protocol.
|
||||
- Each model completion uses a short-lived Python helper backed by LiteLLM. Structured input is
|
||||
supplied over stdin, structured output alone is emitted on stdout, diagnostics use stderr, and
|
||||
the helper can be terminated by cancellation.
|
||||
- A completion request contains no more than ten targets. Requests run one at a time. The helper
|
||||
performs at most one retry for a transient technical failure.
|
||||
- Three consecutive model-request failures fail the run. A successful request resets that count.
|
||||
Isolated exhausted failures may be logged and skipped, producing `completed_with_errors` if the
|
||||
run later reaches its end.
|
||||
- Every valid generated or non-generatable result is applied immediately to Generated Description.
|
||||
Earlier writes are retained after cancellation, interruption, or later failure.
|
||||
- A response must identify requested targets unambiguously and classify each returned result as
|
||||
generated or non-generatable. Duplicate, unknown, missing, or malformed mappings cause a
|
||||
technical request failure and no result from that ambiguous response is applied.
|
||||
- The parser also tolerates one JSON object enclosed by one complete `json` code fence, because
|
||||
some supported models add that formatting despite the prompt. Any prose outside the fence,
|
||||
multiple payloads, or malformed/ambiguous mappings remain invalid.
|
||||
- The application supplies localized standard non-generatable text. Provider wording is not used
|
||||
as the standard value, and technical errors never write that value.
|
||||
- A full-database run generates eligible Catalog Columns before Catalog Tables. Generate Missing
|
||||
excludes targets whose Generated Description is already non-empty; all-generation may replace
|
||||
existing generated proposals only after the initiating action makes that scope explicit.
|
||||
- Model choices are declared under a metadata-generation section in installation setup YAML. Each
|
||||
choice has a stable identifier, display label, LiteLLM provider/model settings, optional endpoint
|
||||
settings, and an optional environment-secret reference for its API key. The reference may be
|
||||
omitted only when an explicit endpoint is configured for unauthenticated access. One identifier
|
||||
is the default.
|
||||
- An explicit endpoint may set `disableThinking: true`; the helper translates it only to the
|
||||
Qwen-compatible chat-template switch needed to keep the response within the strict JSON contract.
|
||||
- Metadata-generation setup is separate from application settings for Pi and from workspace
|
||||
`llm_policy`. Raw keys never enter setup YAML, the catalog database, API responses, process
|
||||
arguments, or event text. Configuration reload is restart-only.
|
||||
- If setup defines no usable model, the safe model-list response is empty and the UI disables
|
||||
generation with an explanation. The backend still rejects direct generation attempts.
|
||||
- Prompt construction treats schema names, comments, descriptions, and values as untrusted data.
|
||||
It requests output in the workspace language and separates instructions from catalog content.
|
||||
- A request may contain up to five real source rows and up to five representative distinct,
|
||||
non-null values for relevant columns. Inputs are bounded before prompt construction and are not
|
||||
persisted or logged.
|
||||
- The UI discloses that real data can be sent to the selected provider. A future Sensitive Data
|
||||
Policy will classify values and exclude or anonymize protected data; that policy is not silently
|
||||
approximated in this slice.
|
||||
- The generation UI reuses Database Management's table and column selections, model selector,
|
||||
Actions control, run drawer conventions, SSE delivery, and polling fallback where practical.
|
||||
Visual parity with Catalog Sync Run logs is not required.
|
||||
- The consolidation action copies each selected, non-empty Generated Description into Description
|
||||
in a catalog transaction, retains Generated Description, skips empty proposals, and reports
|
||||
copied and skipped counts. It never writes to the external Workspace Database.
|
||||
- Generation, cancellation, unlock, and consolidation require the existing database-management
|
||||
permission and are validated by the backend independently of UI state.
|
||||
- No user-facing generation CLI is added. The Python process is an internal completion adapter,
|
||||
not an operator surface or a long-lived service.
|
||||
|
||||
## Testing Decisions
|
||||
|
||||
- Tests assert externally observable behavior rather than private loop structure, process timing,
|
||||
or LiteLLM implementation details.
|
||||
- The primary and highest test seam is the Fastify catalog API with a test PostgreSQL catalog and
|
||||
an injected fake Model Completer. It verifies complete paths through authorization, run
|
||||
persistence, sequential processing, event delivery, Generated Description updates, and final
|
||||
status without contacting a real provider.
|
||||
- API tests cover each generation scope, column-before-table order, the ten-target request bound,
|
||||
model validation, one-active-run conflict, target-database exclusion, cancellation, startup
|
||||
interruption, Unlock safeguards, Generate Missing, partial success, consecutive failure
|
||||
handling, non-generatable localization, malformed responses, redacted events, and permissions.
|
||||
- Catalog repository integration tests verify the migration, run and event ordering, active-run
|
||||
constraint, immediate description writes, history queries, startup interruption, and bulk
|
||||
consolidation behavior against PostgreSQL.
|
||||
- The Python helper has a small black-box contract suite using a simulated LiteLLM adapter. It
|
||||
verifies stdin/stdout framing, pristine stdout, stderr diagnostics, normalized success and
|
||||
failure output, one transient retry, secret redaction, and termination behavior.
|
||||
- Setup-validation tests cover duplicate model identifiers, missing or unknown defaults, malformed
|
||||
provider settings, missing secret references, safe public model projection, and strict separation
|
||||
from Pi and workspace model settings.
|
||||
- Database Management tests use the existing browser-level component seam with MSW. They verify
|
||||
model selection and default, selected/all/missing actions, disabled state without models, running
|
||||
progress and logs, polling recovery, cancellation, Unlock visibility, terminal summaries,
|
||||
generated-text refresh, and selected consolidation with copied/skipped counts.
|
||||
- Existing Catalog Sync Run route, repository, SSE, and drawer tests are prior art for asynchronous
|
||||
status and event behavior. Existing catalog table/column editing and Database Management tests
|
||||
are prior art for optimistic catalog updates, permissions, selection, and action controls.
|
||||
- One required manual acceptance gate, outside deterministic CI, uses the installation's configured
|
||||
default model and a disposable PostgreSQL database containing only invented data. Its application
|
||||
credentials are read-only. It generates Italian text for one Catalog Column and one Catalog
|
||||
Table, verifies their Generated Description, inspects the safe activity log, confirms that no key
|
||||
or sample value is exposed, and consolidates one selected result. If the configured secret is not
|
||||
available, acceptance stops without exposing or requesting the key in conversation.
|
||||
- Delivery includes a strict MkDocs build executed through repository-managed, reproducible
|
||||
documentation dependencies rather than globally installed Python packages. A readable direct
|
||||
dependency file is retained, a complete transitive lock is generated with `uv`, and one canonical
|
||||
repository command performs the strict build from that lock.
|
||||
- Successful real-provider acceptance is recorded in a short sanitized report under
|
||||
`docs/testing/`. It identifies the model and checks performed but contains no credentials,
|
||||
prompts, source samples, complete provider payloads, or generated database values.
|
||||
- No tests are added for worker queues, parallel generation, distributed locking, multi-replica
|
||||
recovery, automatic resume, cost accounting, or model fallback because those behaviors are out
|
||||
of scope.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Reusing Pi to execute Description Generation or changing Pi's model configuration.
|
||||
- A shared Installation Model Registry, model gateway, long-lived Python sidecar, or provider
|
||||
management platform.
|
||||
- A user-facing generation CLI.
|
||||
- Parallel model calls, worker queues, adaptive rate limiting, distributed locks, leases,
|
||||
heartbeats, automatic resume, or multi-replica execution.
|
||||
- Durable target jobs, invocation history, prompts, samples, token usage, cost accounting,
|
||||
provenance chains, target snapshots, or advanced retention controls.
|
||||
- Automatic retry or resume of individual targets beyond one technical helper retry and a new
|
||||
Generate Missing run.
|
||||
- Automatic fallback to a different model.
|
||||
- Writing generated text into comments of the external Workspace Database.
|
||||
- Generating logical relationships or other catalog metadata beyond Catalog Table and Catalog
|
||||
Column descriptions.
|
||||
- Implementing the Sensitive Data Policy. Its definition and exclusion/anonymization behavior are
|
||||
a required follow-up improvement.
|
||||
- Generalizing the log viewer across unrelated metadata operations. That broader concern remains
|
||||
related to Gitea issue #2.
|
||||
|
||||
## Further Notes
|
||||
|
||||
- The design deliberately follows ThothAI's proven simple workflow while adapting it to ThothII's
|
||||
asynchronous browser interaction, setup ownership, and existing Generated Description model.
|
||||
- The source-sampling disclosure is a release requirement, not merely documentation for operators.
|
||||
- `completed_with_errors` is reserved for a run that reaches the end after isolated technical
|
||||
failures. Three consecutive failures end the run as `failed`.
|
||||
- Successful values are their own recovery record: after interruption, Generate Missing naturally
|
||||
skips them without needing replay state.
|
||||
- The implementation is available without a feature flag once the catalog migration and valid
|
||||
setup are present. With no configured model, the feature remains visibly unavailable rather than
|
||||
partially initialized.
|
||||
- The final manual gate is intentionally narrow: one real-provider run covers one Catalog Column
|
||||
and one Catalog Table, generated Italian text, safe events, and one consolidation. Automated
|
||||
tests remain the evidence for All, Missing, Stop, restart interruption, Unlock, and failure paths.
|
||||
@@ -0,0 +1,173 @@
|
||||
# AI catalog description generation
|
||||
|
||||
Status: simplified design, API, persistence, test seams, and delivery tickets accepted.
|
||||
|
||||
## Objective
|
||||
|
||||
Bring ThothAI's useful AI comment-generation workflow into the ThothII Metadata Catalog without
|
||||
turning it into a general job platform. Administrators can generate editable descriptions for
|
||||
catalog tables and columns, inspect progress, stop a run, recover a stale run, and explicitly copy
|
||||
approved generated text into the curated Description field.
|
||||
|
||||
The implementation is UI/API only. There is no user-facing generation command.
|
||||
|
||||
## ThothAI behavior retained
|
||||
|
||||
- Generate descriptions for selected tables, selected columns, missing descriptions, or all
|
||||
eligible targets.
|
||||
- Generate columns before their containing table when running the full workflow, so table prompts
|
||||
can benefit from the resulting column descriptions.
|
||||
- Process bounded batches of at most ten targets, one model request at a time.
|
||||
- Include schema context, existing catalog text, up to five real source rows, and up to five
|
||||
representative non-null values when available.
|
||||
- Keep generated text separate from the curated Description until an administrator consolidates
|
||||
it.
|
||||
- Use the existing table and column checkboxes plus the Actions selector to copy Generated
|
||||
Description into Description for one or more selected records. The generated value is retained.
|
||||
- Store a localized standard value such as `Non generabile` when a valid model response says that
|
||||
a description cannot be inferred.
|
||||
|
||||
Unlike ThothAI, every generation action is asynchronous from the browser's perspective and exposes
|
||||
a persistent, readable activity log.
|
||||
|
||||
## Minimal architecture
|
||||
|
||||
The Fastify backend owns the run lifecycle and sequential loop. It starts one short-lived Python
|
||||
helper for each model completion. The helper uses LiteLLM, accepts structured input on stdin,
|
||||
returns structured output on stdout, and writes diagnostics only to stderr.
|
||||
|
||||
This is preferred over reusing Pi. Pi remains the interactive NL-to-SQL orchestration surface,
|
||||
whereas description generation is a bounded batch transformation with no conversational state or
|
||||
human gate. A LiteLLM helper avoids inventing a Pi session protocol for a task that needs one
|
||||
request and one structured response.
|
||||
|
||||
There is no Python daemon, model gateway, queue service, worker pool, or generation CLI. Python is
|
||||
already a core implementation language in ThothII's harness and core image; this helper does not
|
||||
introduce a new runtime family.
|
||||
|
||||
## Run lifecycle and exclusion
|
||||
|
||||
- At most one Description Generation Run may be queued or running in the installation.
|
||||
- Start returns immediately after creating the run and scheduling the in-process backend loop.
|
||||
- Requests are sequential; there is no parallel provider traffic.
|
||||
- The target Workspace Database is reserved through the existing in-memory catalog-operation
|
||||
coordinator. Synchronization, cleanup, consolidation, and direct catalog edits for that database
|
||||
are rejected while generation is active.
|
||||
- A second generation start is rejected with a conflict response.
|
||||
- Stop terminates the current helper process, stops further targets, and marks the run cancelled.
|
||||
- Backend startup marks any queued or running generation row interrupted. It does not resume work.
|
||||
- Generate Missing is the normal manual continuation mechanism because successful values were
|
||||
already saved.
|
||||
- Unlock is available only when the backend has no live generation process; it marks a stale
|
||||
recorded run interrupted and clears the local reservation.
|
||||
|
||||
This is intentionally a single-process policy. Multi-replica coordination is out of scope.
|
||||
|
||||
## Persistence
|
||||
|
||||
Persist only:
|
||||
|
||||
- a Description Generation Run with database, scope, selected model, language, status, counters,
|
||||
timestamps, and an optional final error summary;
|
||||
- ordered Description Generation Events containing timestamp, level, and human-readable text;
|
||||
- each successful or non-generatable result directly in the target's Generated Description.
|
||||
|
||||
Do not add per-target job rows, invocation history, prompt or sample snapshots, provider cost
|
||||
accounting, leases, heartbeats, registry revisions, or generated-description provenance. The event
|
||||
log is operational evidence, not a replay mechanism.
|
||||
|
||||
## Model setup
|
||||
|
||||
Selectable models and their default belong to application setup YAML, not to a workspace. Each
|
||||
entry supplies a stable display identifier, LiteLLM provider/model information, optional endpoint
|
||||
settings, and—unless that explicit endpoint is unauthenticated—a reference to an installation
|
||||
secret containing the API key. Keyless entries without an explicit endpoint are invalid. Raw keys must not be
|
||||
stored in the YAML, database, frontend, events, or process arguments.
|
||||
An explicit endpoint may opt into `disableThinking: true` when its Qwen-compatible chat template
|
||||
would otherwise place reasoning text around the required JSON result.
|
||||
|
||||
This metadata-generation configuration is independent of the existing Pi provider/model settings
|
||||
and workspace `llm_policy`. A setup change takes effect after application restart. If no model is
|
||||
configured, generation controls are disabled with an explanatory message.
|
||||
|
||||
The browser receives only the selectable identifiers and labels. The selected value defaults to
|
||||
the setup default and is validated again by the backend when a run starts.
|
||||
|
||||
## Prompt inputs and outputs
|
||||
|
||||
Targets are grouped in model requests of at most ten. Prompts distinguish instructions from
|
||||
untrusted schema names, comments, descriptions, and sampled values. A response must map every
|
||||
returned result to a requested target and classify it as generated or non-generatable. Missing,
|
||||
duplicate, unknown, or malformed target results make that request a technical failure rather than
|
||||
silently writing ambiguous text.
|
||||
One complete `json` code fence around the object is tolerated for model compatibility; prose
|
||||
outside it, multiple payloads, and ambiguous mappings are still rejected.
|
||||
|
||||
For a complete database run, eligible columns are processed before tables. A table request can use
|
||||
the current Generated Description or Description of its columns. The output language is the
|
||||
workspace language; the standard non-generatable text is localized by the application rather than
|
||||
trusted to arbitrary model wording.
|
||||
|
||||
Up to five source rows and five representative examples may be sent to the provider and are never
|
||||
persisted. Delivery must call out this disclosure. A follow-up Sensitive Data Policy will define
|
||||
which values are excluded or anonymized.
|
||||
|
||||
## Errors, retry, and logs
|
||||
|
||||
The helper performs at most one retry for a transient technical provider failure. A final failed
|
||||
request produces an error event and increments the consecutive-error count. The run stops as
|
||||
failed after three consecutive technical failures; any successful request resets the count. There
|
||||
is no automatic fallback to another model.
|
||||
|
||||
Valid non-generatable outcomes are results, not technical errors. Successful results from earlier
|
||||
requests remain stored when a later request fails or the run is stopped.
|
||||
|
||||
The UI shows status, counters, selected model, start/end times, and a chronological text log. Live
|
||||
delivery may reuse the existing SSE infrastructure with polling as fallback; exact visual parity
|
||||
with synchronization logs is not required. Logs must not contain API keys, prompts, source sample
|
||||
values, or full provider payloads.
|
||||
|
||||
## Explicitly deferred complexity
|
||||
|
||||
- shared model registry or cutover of Pi configuration;
|
||||
- long-lived Python sidecar or internal HTTP model gateway;
|
||||
- generic catalog-operation kernel;
|
||||
- durable target items, invocation records, target snapshots, or provenance chains;
|
||||
- distributed locks, leases, heartbeats, worker queues, automatic resume, or multi-replica support;
|
||||
- parallel calls, adaptive rate limiting, cost estimation, advanced metrics, or model fallback;
|
||||
- user-facing generation CLI;
|
||||
- automatic writeback to comments in the external database;
|
||||
- Sensitive Data Policy implementation, which remains a required improvement after this slice.
|
||||
|
||||
## Delivery tracking
|
||||
|
||||
The accepted specification is Gitea issue #4 and the implementation is split into issues #5–#11.
|
||||
Each ticket is a bounded vertical slice with explicit Gitea dependencies. Implementation proceeds
|
||||
from the unblocked frontier, using a fresh subagent context for each ticket; integration and final
|
||||
verification remain centralized so later slices cannot silently reopen the deferred platform
|
||||
features above.
|
||||
|
||||
Issue #4 remains open until delivery completes four final gates: the stale Compose service-set
|
||||
contract is corrected in its own commit; documentation dependencies are repository-managed and a
|
||||
strict MkDocs build passes; one narrow real-provider acceptance run succeeds against non-sensitive
|
||||
test data; and a separate, non-blocking Sensitive Data Policy design ticket is linked as required
|
||||
follow-up work.
|
||||
|
||||
The documentation toolchain retains a readable direct-dependency input, adds a complete lock
|
||||
generated with `uv`, and exposes one canonical strict-build command. Real-provider acceptance uses
|
||||
the installation's configured default model and a disposable PostgreSQL database seeded only with
|
||||
invented values and accessed read-only by the application. A missing protected model secret stops
|
||||
the gate without disclosing it. The successful gate is captured in a sanitized report under
|
||||
`docs/testing/` without prompts, samples, full generated values, payloads, or credentials.
|
||||
|
||||
Delivery is organized as four reviewable commits: the stale Compose contract correction, the
|
||||
reproducible documentation toolchain, the AI-description feature, and—only after acceptance—the
|
||||
sanitized acceptance report. A failed real-provider gate does not invalidate already verified
|
||||
commits, but issue #4 remains open and no acceptance report claims success. Application defects are
|
||||
fixed and reverified; missing configuration or provider unavailability is recorded and retried.
|
||||
|
||||
After every gate passes, the existing `codex/db-management` branch is pushed to its configured
|
||||
origin without introducing a new pull-request workflow, then issue #4 is closed with links to the
|
||||
delivery evidence. The separate Sensitive Data Policy issue is created as non-blocking follow-up,
|
||||
linked to #4, and labeled `enhancement` plus `ready-for-human` because its design requires a future
|
||||
`grill-with-docs` before agent implementation.
|
||||
@@ -132,6 +132,13 @@ export interface CatalogMetadataDeleteCounts {
|
||||
relationships: number;
|
||||
}
|
||||
|
||||
export type CatalogDescriptionTarget = "tables" | "columns";
|
||||
|
||||
export interface CatalogDescriptionConsolidationCounts {
|
||||
copied: number;
|
||||
skipped: number;
|
||||
}
|
||||
|
||||
export type CatalogSyncScope = "tables" | "columns" | "relationships" | "all";
|
||||
export type CatalogSyncState = "queued" | "running" | "awaiting_confirmation" | "applying"
|
||||
| "succeeded" | "failed" | "cancelled" | "interrupted";
|
||||
@@ -176,8 +183,113 @@ export interface CatalogSyncEvent {
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export interface MetadataGenerationModel {
|
||||
id: string;
|
||||
label: string;
|
||||
}
|
||||
|
||||
export interface MetadataGenerationModels {
|
||||
models: MetadataGenerationModel[];
|
||||
default: string | null;
|
||||
}
|
||||
|
||||
export type DescriptionGenerationScope = "selected_columns" | "selected_tables" | "all" | "missing";
|
||||
export type DescriptionGenerationStatus = "queued" | "running" | "completed"
|
||||
| "completed_with_errors" | "cancelled" | "failed" | "interrupted";
|
||||
|
||||
export interface DescriptionGenerationRun {
|
||||
id: string;
|
||||
databaseId: string;
|
||||
scope: DescriptionGenerationScope;
|
||||
modelId: string;
|
||||
language: "en" | "it";
|
||||
status: DescriptionGenerationStatus;
|
||||
total: number;
|
||||
processed: number;
|
||||
generated: number;
|
||||
nonGeneratable: number;
|
||||
failed: number;
|
||||
createdAt: string;
|
||||
startedAt: string | null;
|
||||
updatedAt: string;
|
||||
finishedAt: string | null;
|
||||
errorSummary: string | null;
|
||||
}
|
||||
|
||||
export interface DescriptionGenerationEvent {
|
||||
sequence: number;
|
||||
level: "info" | "warning" | "error";
|
||||
message: string;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export const listCatalogDatabases = () => apiFetch<CatalogDatabase[]>("/catalog/databases");
|
||||
|
||||
export const listMetadataGenerationModels = () =>
|
||||
apiFetch<MetadataGenerationModels>("/catalog/metadata-generation/models");
|
||||
|
||||
export function startDescriptionGenerationRun(
|
||||
databaseId: string,
|
||||
modelId: string,
|
||||
scope: Extract<DescriptionGenerationScope, "selected_columns" | "selected_tables">,
|
||||
targetIds: string[],
|
||||
): Promise<DescriptionGenerationRun>;
|
||||
export function startDescriptionGenerationRun(
|
||||
databaseId: string,
|
||||
modelId: string,
|
||||
scope: Extract<DescriptionGenerationScope, "all" | "missing">,
|
||||
): Promise<DescriptionGenerationRun>;
|
||||
export function startDescriptionGenerationRun(
|
||||
databaseId: string,
|
||||
modelId: string,
|
||||
scope: DescriptionGenerationScope,
|
||||
targetIds?: string[],
|
||||
): Promise<DescriptionGenerationRun> {
|
||||
const body = scope === "all" || scope === "missing"
|
||||
? { modelId, scope }
|
||||
: { modelId, scope, targetIds: targetIds ?? [] };
|
||||
return apiFetch<DescriptionGenerationRun>(
|
||||
`/catalog/databases/${encodeURIComponent(databaseId)}/description-generation-runs`,
|
||||
{ method: "POST", body: JSON.stringify(body) },
|
||||
);
|
||||
}
|
||||
|
||||
export const getDescriptionGenerationRun = (runId: string) =>
|
||||
apiFetch<DescriptionGenerationRun>(
|
||||
`/catalog/description-generation-runs/${encodeURIComponent(runId)}`,
|
||||
);
|
||||
|
||||
export const listDescriptionGenerationRuns = (limit = 50) =>
|
||||
apiFetch<DescriptionGenerationRun[]>(
|
||||
`/catalog/description-generation-runs?limit=${encodeURIComponent(String(limit))}`,
|
||||
);
|
||||
|
||||
export const listDescriptionGenerationEvents = (runId: string, after = 0) =>
|
||||
apiFetch<DescriptionGenerationEvent[]>(
|
||||
`/catalog/description-generation-runs/${encodeURIComponent(runId)}/events-list?after=${after}`,
|
||||
);
|
||||
|
||||
export const cancelDescriptionGenerationRun = (runId: string) =>
|
||||
apiFetch<DescriptionGenerationRun>(
|
||||
`/catalog/description-generation-runs/${encodeURIComponent(runId)}/cancel`,
|
||||
{ method: "POST" },
|
||||
);
|
||||
|
||||
export const unlockDescriptionGenerationRun = () =>
|
||||
apiFetch<DescriptionGenerationRun>(
|
||||
"/catalog/description-generation-runs/unlock",
|
||||
{ method: "POST" },
|
||||
);
|
||||
|
||||
export function descriptionGenerationEventsUrl(runId: string, after = 0): string {
|
||||
const url = joinBackendPath(
|
||||
BASE,
|
||||
`/catalog/description-generation-runs/${encodeURIComponent(runId)}/events?after=${after}`,
|
||||
);
|
||||
assertSameOriginRequestUrl(url);
|
||||
return url;
|
||||
}
|
||||
|
||||
export const createCatalogDatabase = (input: DatabaseConfiguration) =>
|
||||
apiFetch<CatalogDatabase>("/catalog/databases", { method: "POST", body: JSON.stringify(input) });
|
||||
|
||||
@@ -254,6 +366,15 @@ export const deleteCatalogTableMetadata = (
|
||||
{ method: "POST", body: JSON.stringify({ tableIds, target }) },
|
||||
);
|
||||
|
||||
export const consolidateCatalogDescriptions = (
|
||||
databaseId: string,
|
||||
target: CatalogDescriptionTarget,
|
||||
targetIds: string[],
|
||||
) => apiFetch<CatalogDescriptionConsolidationCounts>(
|
||||
`/catalog/databases/${encodeURIComponent(databaseId)}/descriptions/consolidate`,
|
||||
{ method: "POST", body: JSON.stringify({ target, targetIds }) },
|
||||
);
|
||||
|
||||
export const startCatalogSync = (
|
||||
databaseId: string,
|
||||
version: number,
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
import { http, HttpResponse } from "msw";
|
||||
import { server } from "../test/msw";
|
||||
import {
|
||||
cancelDescriptionGenerationRun,
|
||||
descriptionGenerationEventsUrl,
|
||||
listDescriptionGenerationRuns,
|
||||
unlockDescriptionGenerationRun,
|
||||
type DescriptionGenerationRun,
|
||||
} from "./catalog-databases";
|
||||
|
||||
const historicalRun: DescriptionGenerationRun = {
|
||||
id: "77777777-7777-4777-8777-777777777777",
|
||||
databaseId: "11111111-1111-4111-8111-111111111111",
|
||||
scope: "missing",
|
||||
modelId: "local-qwen",
|
||||
language: "it",
|
||||
status: "interrupted",
|
||||
total: 4,
|
||||
processed: 2,
|
||||
generated: 2,
|
||||
nonGeneratable: 0,
|
||||
failed: 0,
|
||||
createdAt: "2026-08-28T08:00:00Z",
|
||||
startedAt: "2026-08-28T08:00:01Z",
|
||||
updatedAt: "2026-08-28T08:02:00Z",
|
||||
finishedAt: "2026-08-28T08:02:00Z",
|
||||
errorSummary: null,
|
||||
};
|
||||
|
||||
test("lists newest-first persisted description-generation history with a bounded limit", async () => {
|
||||
let requestedLimit: string | null = null;
|
||||
server.use(http.get("/api/catalog/description-generation-runs", ({ request }) => {
|
||||
requestedLimit = new URL(request.url).searchParams.get("limit");
|
||||
return HttpResponse.json([historicalRun]);
|
||||
}));
|
||||
|
||||
await expect(listDescriptionGenerationRuns(50)).resolves.toEqual([historicalRun]);
|
||||
expect(requestedLimit).toBe("50");
|
||||
});
|
||||
|
||||
test("stops a description-generation run with the run-scoped body-less endpoint", async () => {
|
||||
let contentType: string | null = "unset";
|
||||
server.use(http.post(
|
||||
"/api/catalog/description-generation-runs/:runId/cancel",
|
||||
({ request }) => {
|
||||
contentType = request.headers.get("content-type");
|
||||
return HttpResponse.json({ ...historicalRun, status: "cancelled" });
|
||||
},
|
||||
));
|
||||
|
||||
await expect(cancelDescriptionGenerationRun(historicalRun.id)).resolves.toMatchObject({
|
||||
id: historicalRun.id,
|
||||
status: "cancelled",
|
||||
});
|
||||
expect(contentType).toBeNull();
|
||||
});
|
||||
|
||||
test("unlocks the installation-wide stale description-generation run without a request body", async () => {
|
||||
let contentType: string | null = "unset";
|
||||
server.use(http.post(
|
||||
"/api/catalog/description-generation-runs/unlock",
|
||||
({ request }) => {
|
||||
contentType = request.headers.get("content-type");
|
||||
return HttpResponse.json(historicalRun);
|
||||
},
|
||||
));
|
||||
|
||||
await expect(unlockDescriptionGenerationRun()).resolves.toEqual(historicalRun);
|
||||
expect(contentType).toBeNull();
|
||||
});
|
||||
|
||||
test("builds the same-origin description-generation SSE replay URL from an event sequence", () => {
|
||||
expect(descriptionGenerationEventsUrl(historicalRun.id, 17)).toBe(
|
||||
`/api/catalog/description-generation-runs/${historicalRun.id}/events?after=17`,
|
||||
);
|
||||
});
|
||||
@@ -143,6 +143,27 @@ test("keeps only a known safe bounded JSON error payload", async () => {
|
||||
}
|
||||
});
|
||||
|
||||
test.each([
|
||||
["description_generation_target_ids_duplicate", "Description generation target IDs must be unique."],
|
||||
["description_generation_no_eligible_targets", "No eligible catalog tables or columns need description generation."],
|
||||
["catalog_table_not_found", "One or more selected catalog tables were not found."],
|
||||
])("maps the description-generation error code %s to safe local copy", async (code, message) => {
|
||||
const fetchSpy = vi.spyOn(globalThis, "fetch").mockResolvedValue(
|
||||
new Response(JSON.stringify({ code, message: "provider detail must not be trusted" }), {
|
||||
status: 400,
|
||||
headers: { "content-type": "application/json" },
|
||||
}),
|
||||
);
|
||||
|
||||
try {
|
||||
const failure = await apiFetch<unknown>("/description-generation-error")
|
||||
.catch((error: unknown) => error) as ApiError;
|
||||
expect(failure).toMatchObject({ code, message, payload: { code } });
|
||||
} finally {
|
||||
fetchSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
test("derives local messages without retaining a malicious known-code message", async () => {
|
||||
const fetchSpy = vi.spyOn(globalThis, "fetch").mockResolvedValue(
|
||||
new Response(JSON.stringify({
|
||||
|
||||
@@ -15,7 +15,14 @@ const safeErrorCodes = new Set([
|
||||
"semantic_index_incompatible", "pi_management_forbidden", "pi_management_unavailable",
|
||||
"pi_management_invalid_config", "pi_management_write_failed",
|
||||
"catalog_unavailable", "database_conflict", "database_invalid", "database_not_found",
|
||||
"database_stale", "database_operation_failed",
|
||||
"database_stale", "database_operation_failed", "database_operation_in_progress",
|
||||
"catalog_target_not_found", "description_consolidation_invalid", "description_consolidation_failed",
|
||||
"description_generation_request_invalid", "description_generation_run_active",
|
||||
"description_generation_failed", "description_generation_run_not_found",
|
||||
"description_generation_no_eligible_targets",
|
||||
"description_generation_target_ids_duplicate", "metadata_generation_model_unavailable",
|
||||
"catalog_column_not_found", "catalog_table_not_found",
|
||||
"workspace_configuration_unavailable",
|
||||
"schema_sync_conflict", "schema_introspection_failed", "schema_request_invalid",
|
||||
"schema_operation_failed", "sync_run_not_found", "table_stale", "column_stale",
|
||||
]);
|
||||
@@ -56,6 +63,20 @@ const localCodeMessages: Record<string, string> = {
|
||||
database_not_found: "The database configuration was not found.",
|
||||
database_stale: "The database configuration changed. Reload and try again.",
|
||||
database_operation_failed: "The database operation failed.",
|
||||
database_operation_in_progress: "A database operation is already in progress.",
|
||||
catalog_target_not_found: "One or more selected catalog targets were not found.",
|
||||
description_consolidation_invalid: "The description consolidation request is invalid.",
|
||||
description_consolidation_failed: "Description consolidation failed.",
|
||||
description_generation_request_invalid: "The description generation request is invalid.",
|
||||
description_generation_run_active: "A description generation run is already active.",
|
||||
description_generation_failed: "Description generation failed.",
|
||||
description_generation_run_not_found: "The description generation run was not found.",
|
||||
description_generation_no_eligible_targets: "No eligible catalog tables or columns need description generation.",
|
||||
description_generation_target_ids_duplicate: "Description generation target IDs must be unique.",
|
||||
metadata_generation_model_unavailable: "The selected description model is unavailable.",
|
||||
catalog_column_not_found: "The selected catalog column was not found.",
|
||||
catalog_table_not_found: "One or more selected catalog tables were not found.",
|
||||
workspace_configuration_unavailable: "The database workspace configuration is unavailable.",
|
||||
schema_sync_conflict: "A schema synchronization is already active or no longer current.",
|
||||
schema_introspection_failed: "The database schema could not be read safely.",
|
||||
schema_request_invalid: "The schema request is invalid.",
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -6,7 +6,7 @@ import {
|
||||
useState,
|
||||
} from "react";
|
||||
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { Plus, RefreshCw } from "lucide-react";
|
||||
import { History, Plus, RefreshCw } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
import { Button } from "../components/ui/button";
|
||||
import { ApiError, apiErrorMessage } from "../api/client";
|
||||
@@ -15,8 +15,11 @@ import {
|
||||
deleteCatalogDatabase,
|
||||
deleteCatalogDatabaseMetadata,
|
||||
listCatalogDatabases,
|
||||
listDescriptionGenerationRuns,
|
||||
listMetadataGenerationModels,
|
||||
replaceCatalogDatabaseSecrets,
|
||||
startCatalogSync,
|
||||
startDescriptionGenerationRun,
|
||||
testCatalogDatabase,
|
||||
updateCatalogDatabase,
|
||||
type CatalogDatabase,
|
||||
@@ -26,12 +29,15 @@ import {
|
||||
type CatalogSyncRun,
|
||||
type DatabaseBinding,
|
||||
type DatabaseTransport,
|
||||
type DescriptionGenerationRun,
|
||||
} from "../api/catalog-databases";
|
||||
import { DatabaseGrid } from "./database-management/DatabaseGrid";
|
||||
import { DatabaseForm } from "./database-management/DatabaseForm";
|
||||
import { DatabaseTables } from "./database-management/DatabaseTables";
|
||||
import { DatabaseRelationships } from "./database-management/DatabaseRelationships";
|
||||
import { CatalogSyncDrawer } from "./database-management/CatalogSyncDrawer";
|
||||
import { MetadataGenerationModelSelector } from "./database-management/MetadataGenerationModelSelector";
|
||||
import { DescriptionGenerationDrawer } from "./database-management/DescriptionGenerationDrawer";
|
||||
import {
|
||||
configurationFingerprint,
|
||||
configurationFromDraft,
|
||||
@@ -46,6 +52,8 @@ import {
|
||||
} from "./database-management/model";
|
||||
|
||||
const DATABASE_QUERY_KEY = ["catalog-databases"] as const;
|
||||
const DESCRIPTION_GENERATION_HISTORY_QUERY_KEY = ["description-generation-runs", 50] as const;
|
||||
const DESCRIPTION_GENERATION_ACTIVE_STATUSES = ["queued", "running"] as const;
|
||||
const SYNC_STARTED_MESSAGES: Record<CatalogSyncScope, string> = {
|
||||
tables: "Table synchronization started",
|
||||
columns: "Column synchronization started",
|
||||
@@ -68,6 +76,12 @@ function isStaleError(error: unknown): boolean {
|
||||
return error instanceof ApiError && error.code === "database_stale";
|
||||
}
|
||||
|
||||
function isDescriptionGenerationActive(run?: DescriptionGenerationRun | null): boolean {
|
||||
return Boolean(run && DESCRIPTION_GENERATION_ACTIVE_STATUSES.some(
|
||||
(status) => status === run.status,
|
||||
));
|
||||
}
|
||||
|
||||
export function DatabaseManagementPage({
|
||||
canManage,
|
||||
canManageSecrets,
|
||||
@@ -86,8 +100,31 @@ export function DatabaseManagementPage({
|
||||
retry: false,
|
||||
});
|
||||
const rows = data ?? [];
|
||||
const metadataModelsQuery = useQuery({
|
||||
queryKey: ["metadata-generation-models"],
|
||||
queryFn: listMetadataGenerationModels,
|
||||
retry: false,
|
||||
});
|
||||
const metadataModels = metadataModelsQuery.data?.models ?? [];
|
||||
const configuredMetadataDefault = metadataModelsQuery.data?.default ?? "";
|
||||
const defaultMetadataModel = configuredMetadataDefault
|
||||
&& metadataModels.some((model) => model.id === configuredMetadataDefault)
|
||||
? configuredMetadataDefault
|
||||
: "";
|
||||
const descriptionGenerationRunsQuery = useQuery({
|
||||
queryKey: DESCRIPTION_GENERATION_HISTORY_QUERY_KEY,
|
||||
queryFn: () => listDescriptionGenerationRuns(50),
|
||||
enabled: canManage,
|
||||
retry: false,
|
||||
refetchInterval: 5_000,
|
||||
});
|
||||
const descriptionGenerationRuns = descriptionGenerationRunsQuery.data ?? [];
|
||||
const observedActiveDescriptionGenerationRun = descriptionGenerationRuns.find(
|
||||
(run) => isDescriptionGenerationActive(run),
|
||||
);
|
||||
|
||||
const [screen, setScreen] = useState<DatabaseScreen>({ kind: "list" });
|
||||
const [selectedMetadataModel, setSelectedMetadataModel] = useState("");
|
||||
const [draft, setDraft] = useState<DatabaseFormDraft | null>(null);
|
||||
const [baseline, setBaseline] = useState("");
|
||||
const [formSource, setFormSource] = useState<FormSource | null>(null);
|
||||
@@ -102,11 +139,17 @@ export function DatabaseManagementPage({
|
||||
});
|
||||
const [activeSyncRun, setActiveSyncRun] = useState<CatalogSyncRun | null>(null);
|
||||
const [syncDrawerOpen, setSyncDrawerOpen] = useState(false);
|
||||
const [activeDescriptionGenerationRun, setActiveDescriptionGenerationRun] = useState<DescriptionGenerationRun | null>(null);
|
||||
const [descriptionGenerationDrawerOpen, setDescriptionGenerationDrawerOpen] = useState(false);
|
||||
|
||||
const originRef = useRef<HTMLElement | null>(null);
|
||||
const searchInputRef = useRef<HTMLInputElement>(null);
|
||||
const formHeadingRef = useRef<HTMLHeadingElement>(null);
|
||||
|
||||
useEffect(() => {
|
||||
setSelectedMetadataModel(defaultMetadataModel);
|
||||
}, [defaultMetadataModel]);
|
||||
|
||||
const activeRow = screen.kind === "list"
|
||||
? undefined
|
||||
: rows.find((row) => row.workspaceId === screen.workspaceId);
|
||||
@@ -556,6 +599,36 @@ export function DatabaseManagementPage({
|
||||
setSyncDrawerOpen(true);
|
||||
}, [updateTrackedSyncRun]);
|
||||
|
||||
const rememberDescriptionGenerationRun = useCallback((run: DescriptionGenerationRun) => {
|
||||
setActiveDescriptionGenerationRun(run);
|
||||
queryClient.setQueryData<DescriptionGenerationRun[]>(
|
||||
DESCRIPTION_GENERATION_HISTORY_QUERY_KEY,
|
||||
(current = []) => [run, ...current.filter((item) => item.id !== run.id)],
|
||||
);
|
||||
setDescriptionGenerationDrawerOpen(true);
|
||||
}, [queryClient]);
|
||||
|
||||
const openDescriptionGenerationHistory = useCallback(() => {
|
||||
const run = observedActiveDescriptionGenerationRun
|
||||
?? descriptionGenerationRuns[0]
|
||||
?? activeDescriptionGenerationRun;
|
||||
if (!run) return;
|
||||
setActiveDescriptionGenerationRun(run);
|
||||
setDescriptionGenerationDrawerOpen(true);
|
||||
}, [activeDescriptionGenerationRun, descriptionGenerationRuns, observedActiveDescriptionGenerationRun]);
|
||||
|
||||
const descriptionGenerationTerminated = useCallback(async (run: DescriptionGenerationRun) => {
|
||||
await Promise.all([
|
||||
queryClient.invalidateQueries({
|
||||
queryKey: ["catalog-tables", run.databaseId],
|
||||
exact: true,
|
||||
}),
|
||||
queryClient.invalidateQueries({
|
||||
queryKey: ["catalog-columns", run.databaseId],
|
||||
}),
|
||||
]);
|
||||
}, [queryClient]);
|
||||
|
||||
const openSync = useCallback((row?: CatalogDatabase) => {
|
||||
const run = row?.activeSyncRun ?? activeSyncRun;
|
||||
if (!run) return;
|
||||
@@ -590,6 +663,26 @@ export function DatabaseManagementPage({
|
||||
}
|
||||
}, [queryClient, rememberSyncRun]);
|
||||
|
||||
const generateDatabaseDescriptions = useCallback(async (
|
||||
selected: CatalogDatabase[],
|
||||
scope: "all" | "missing",
|
||||
) => {
|
||||
const database = selected.length === 1 ? selected[0] : undefined;
|
||||
if (!database?.id || !selectedMetadataModel) return;
|
||||
try {
|
||||
const run = await startDescriptionGenerationRun(
|
||||
database.id,
|
||||
selectedMetadataModel,
|
||||
scope,
|
||||
);
|
||||
rememberDescriptionGenerationRun(run);
|
||||
toast.success(`${scope === "all" ? "Generate All" : "Generate Missing"} started for ${database.workspaceName}`);
|
||||
} catch (error) {
|
||||
toast.error(apiErrorMessage(error));
|
||||
throw error;
|
||||
}
|
||||
}, [rememberDescriptionGenerationRun, selectedMetadataModel]);
|
||||
|
||||
const deleteSelectedMetadata = useCallback(async (
|
||||
selected: CatalogDatabase[],
|
||||
target: CatalogDatabaseMetadataDeleteTarget,
|
||||
@@ -640,6 +733,16 @@ export function DatabaseManagementPage({
|
||||
: activeRow?.activeSyncRun && ["queued", "running", "awaiting_confirmation", "applying"].includes(activeRow.activeSyncRun.state)
|
||||
? activeRow.activeSyncRun
|
||||
: undefined;
|
||||
const selectedMetadataModelAvailable = Boolean(
|
||||
selectedMetadataModel
|
||||
&& metadataModels.some((model) => model.id === selectedMetadataModel),
|
||||
);
|
||||
const descriptionGenerationActive = isDescriptionGenerationActive(
|
||||
observedActiveDescriptionGenerationRun ?? activeDescriptionGenerationRun,
|
||||
);
|
||||
const descriptionGenerationHistoryAvailable = Boolean(
|
||||
descriptionGenerationRuns.length > 0 || activeDescriptionGenerationRun,
|
||||
);
|
||||
|
||||
return (
|
||||
<main aria-label="Database management" className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
|
||||
@@ -651,21 +754,46 @@ export function DatabaseManagementPage({
|
||||
One database configuration for each repository workspace.
|
||||
</p>
|
||||
</div>
|
||||
{screen.kind === "list" ? (
|
||||
<div className="flex items-center gap-2">
|
||||
<Button type="button" variant="outline" disabled={isFetching} onClick={() => void refreshList()}>
|
||||
<RefreshCw className={isFetching ? "animate-spin" : ""} /> Refresh
|
||||
</Button>
|
||||
<Button
|
||||
type="button"
|
||||
disabled={!canManage || availableWorkspaces.length === 0}
|
||||
title={availableWorkspaces.length === 0 ? "Every available workspace is already configured" : undefined}
|
||||
onClick={(event) => addDatabase(event.currentTarget)}
|
||||
>
|
||||
<Plus /> Add database
|
||||
</Button>
|
||||
</div>
|
||||
) : null}
|
||||
<div className="flex flex-wrap items-end justify-end gap-3">
|
||||
<MetadataGenerationModelSelector
|
||||
canManage={canManage}
|
||||
data={metadataModelsQuery.data}
|
||||
isError={metadataModelsQuery.isError}
|
||||
isLoading={metadataModelsQuery.isLoading}
|
||||
selectedModel={selectedMetadataModel}
|
||||
onSelectedModelChange={setSelectedMetadataModel}
|
||||
/>
|
||||
<Button
|
||||
type="button"
|
||||
variant="outline"
|
||||
aria-label={observedActiveDescriptionGenerationRun
|
||||
? "Observe active description generation"
|
||||
: "View description generation history"}
|
||||
disabled={!canManage || !descriptionGenerationHistoryAvailable}
|
||||
title={!descriptionGenerationHistoryAvailable && !descriptionGenerationRunsQuery.isLoading
|
||||
? "No description generation runs yet"
|
||||
: undefined}
|
||||
onClick={openDescriptionGenerationHistory}
|
||||
>
|
||||
<History />
|
||||
{observedActiveDescriptionGenerationRun ? "Active run" : "Run history"}
|
||||
</Button>
|
||||
{screen.kind === "list" ? (
|
||||
<div className="flex items-center gap-2">
|
||||
<Button type="button" variant="outline" disabled={isFetching} onClick={() => void refreshList()}>
|
||||
<RefreshCw className={isFetching ? "animate-spin" : ""} /> Refresh
|
||||
</Button>
|
||||
<Button
|
||||
type="button"
|
||||
disabled={!canManage || availableWorkspaces.length === 0}
|
||||
title={availableWorkspaces.length === 0 ? "Every available workspace is already configured" : undefined}
|
||||
onClick={(event) => addDatabase(event.currentTarget)}
|
||||
>
|
||||
<Plus /> Add database
|
||||
</Button>
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<div className="relative flex min-h-0 flex-1">
|
||||
@@ -701,6 +829,9 @@ export function DatabaseManagementPage({
|
||||
onOpenSync={openSync}
|
||||
onTestSelected={testSelected}
|
||||
onSyncSelected={syncSelected}
|
||||
selectedMetadataModel={selectedMetadataModelAvailable ? selectedMetadataModel : null}
|
||||
descriptionGenerationActive={descriptionGenerationActive}
|
||||
onGenerateDescriptions={generateDatabaseDescriptions}
|
||||
onDeleteMetadataSelected={deleteSelectedMetadata}
|
||||
/>
|
||||
)}
|
||||
@@ -746,12 +877,15 @@ export function DatabaseManagementPage({
|
||||
database={activeRow}
|
||||
canManage={canManage}
|
||||
activeRun={currentActiveRun}
|
||||
selectedMetadataModel={selectedMetadataModelAvailable ? selectedMetadataModel : null}
|
||||
descriptionGenerationActive={descriptionGenerationActive}
|
||||
onBackToDatabases={showList}
|
||||
onOpenOverview={() => openOverview(activeRow)}
|
||||
onOpenRelationships={() => openRelationships(activeRow)}
|
||||
onNavigationStateChange={setTablesNavigationState}
|
||||
onRunStarted={rememberSyncRun}
|
||||
onOpenSync={() => openSync(activeRow)}
|
||||
onDescriptionGenerationRunStarted={rememberDescriptionGenerationRun}
|
||||
/>
|
||||
) : null}
|
||||
|
||||
@@ -776,6 +910,16 @@ export function DatabaseManagementPage({
|
||||
onRunUpdate={updateTrackedSyncRun}
|
||||
onCatalogChanged={() => void catalogChanged()}
|
||||
/>
|
||||
<DescriptionGenerationDrawer
|
||||
open={descriptionGenerationDrawerOpen}
|
||||
run={activeDescriptionGenerationRun}
|
||||
modelLabel={metadataModels.find(
|
||||
(model) => model.id === activeDescriptionGenerationRun?.modelId,
|
||||
)?.label ?? activeDescriptionGenerationRun?.modelId ?? ""}
|
||||
onClose={() => setDescriptionGenerationDrawerOpen(false)}
|
||||
onRunUpdate={setActiveDescriptionGenerationRun}
|
||||
onTerminal={(run) => void descriptionGenerationTerminated(run)}
|
||||
/>
|
||||
</main>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,16 +1,20 @@
|
||||
import { useEffect, useMemo, useRef, useState } from "react";
|
||||
import { Menu } from "@base-ui/react/menu";
|
||||
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { AgGridReact } from "ag-grid-react";
|
||||
import type { ColDef, ICellRendererParams } from "ag-grid-community";
|
||||
import { KeyRound, Link2, Pencil, RefreshCw, Save } from "lucide-react";
|
||||
import { ChevronDown, KeyRound, Link2, Pencil, RefreshCw, Save, X } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
import { Button } from "../../components/ui/button";
|
||||
import { ApiError, apiErrorMessage } from "../../api/client";
|
||||
import {
|
||||
consolidateCatalogDescriptions,
|
||||
listCatalogColumns,
|
||||
startDescriptionGenerationRun,
|
||||
updateCatalogColumnMetadata,
|
||||
type CatalogColumn,
|
||||
type CatalogTable,
|
||||
type DescriptionGenerationRun,
|
||||
} from "../../api/catalog-databases";
|
||||
import type { DatabaseNavigationState } from "./model";
|
||||
|
||||
@@ -18,6 +22,9 @@ interface Props {
|
||||
databaseId: string;
|
||||
table: CatalogTable;
|
||||
canManage: boolean;
|
||||
selectedMetadataModel: string | null;
|
||||
descriptionGenerationActive: boolean;
|
||||
onDescriptionGenerationRunStarted: (run: DescriptionGenerationRun) => void;
|
||||
onNavigationStateChange: (state: DatabaseNavigationState) => void;
|
||||
onSync: () => void;
|
||||
}
|
||||
@@ -48,7 +55,16 @@ function ActionCell({ data, context }: ICellRendererParams<CatalogColumn, unknow
|
||||
);
|
||||
}
|
||||
|
||||
export function DatabaseColumns({ databaseId, table, canManage, onNavigationStateChange, onSync }: Props) {
|
||||
export function DatabaseColumns({
|
||||
databaseId,
|
||||
table,
|
||||
canManage,
|
||||
selectedMetadataModel,
|
||||
descriptionGenerationActive,
|
||||
onDescriptionGenerationRunStarted,
|
||||
onNavigationStateChange,
|
||||
onSync,
|
||||
}: Props) {
|
||||
const queryClient = useQueryClient();
|
||||
const queryKey = ["catalog-columns", databaseId, table.id] as const;
|
||||
const { data = [], isLoading, isFetching, refetch } = useQuery({
|
||||
@@ -57,6 +73,7 @@ export function DatabaseColumns({ databaseId, table, canManage, onNavigationStat
|
||||
retry: false,
|
||||
});
|
||||
const [search, setSearch] = useState("");
|
||||
const [selectedIds, setSelectedIds] = useState<string[]>([]);
|
||||
const [editingId, setEditingId] = useState<string | null>(null);
|
||||
const [description, setDescription] = useState("");
|
||||
const [generatedDescription, setGeneratedDescription] = useState("");
|
||||
@@ -65,6 +82,7 @@ export function DatabaseColumns({ databaseId, table, canManage, onNavigationStat
|
||||
const [stale, setStale] = useState(false);
|
||||
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const gridRef = useRef<AgGridReact<CatalogColumn>>(null);
|
||||
const originRef = useRef<HTMLButtonElement | null>(null);
|
||||
const active = editingId ? data.find((column) => column.id === editingId) : undefined;
|
||||
const fingerprint = JSON.stringify([description, generatedDescription]);
|
||||
@@ -131,6 +149,34 @@ export function DatabaseColumns({ databaseId, table, canManage, onNavigationStat
|
||||
setStaleBannerOpen(true);
|
||||
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(false); }
|
||||
};
|
||||
const consolidateDescriptions = async () => {
|
||||
setBusy(true);
|
||||
try {
|
||||
const counts = await consolidateCatalogDescriptions(databaseId, "columns", selectedIds);
|
||||
await queryClient.invalidateQueries({ queryKey, exact: true });
|
||||
gridRef.current?.api.deselectAll();
|
||||
setSelectedIds([]);
|
||||
toast.success(`Copied ${counts.copied} description${counts.copied === 1 ? "" : "s"}; skipped ${counts.skipped}`);
|
||||
} catch (error) {
|
||||
toast.error(apiErrorMessage(error));
|
||||
} finally { setBusy(false); }
|
||||
};
|
||||
const generateDescriptions = async () => {
|
||||
if (selectedIds.length === 0 || !selectedMetadataModel) return;
|
||||
setBusy(true);
|
||||
try {
|
||||
const run = await startDescriptionGenerationRun(
|
||||
databaseId,
|
||||
selectedMetadataModel,
|
||||
"selected_columns",
|
||||
selectedIds,
|
||||
);
|
||||
onDescriptionGenerationRunStarted(run);
|
||||
toast.success(`Description generation started for ${selectedIds.length} column${selectedIds.length === 1 ? "" : "s"}`);
|
||||
} catch (error) {
|
||||
toast.error(apiErrorMessage(error));
|
||||
} finally { setBusy(false); }
|
||||
};
|
||||
|
||||
const columns = useMemo<ColDef<CatalogColumn>[]>(() => [
|
||||
{ field: "ordinalPosition", headerName: "#", width: 64, maxWidth: 64, filter: "agNumberColumnFilter" },
|
||||
@@ -176,15 +222,62 @@ export function DatabaseColumns({ databaseId, table, canManage, onNavigationStat
|
||||
return (
|
||||
<div className="flex min-h-0 flex-1 flex-col">
|
||||
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
|
||||
<span className="thot-label whitespace-nowrap">Catalog columns</span>
|
||||
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search columns" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
|
||||
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
|
||||
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
|
||||
<Button type="button" disabled={!canManage || busy} onClick={onSync}><RefreshCw />Sync columns</Button>
|
||||
{selectedIds.length > 0 ? (
|
||||
<>
|
||||
<span className="text-sm font-semibold">{selectedIds.length} selected</span>
|
||||
<Menu.Root>
|
||||
<Menu.Trigger className="inline-flex h-8 items-center justify-center gap-2 rounded-md border border-input bg-background px-3 text-sm font-medium hover:bg-muted disabled:pointer-events-none disabled:opacity-50" disabled={busy}>Actions <ChevronDown className="size-4" /></Menu.Trigger>
|
||||
<Menu.Portal>
|
||||
<Menu.Positioner side="bottom" align="start" sideOffset={4}>
|
||||
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
|
||||
<Menu.Item
|
||||
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
|
||||
disabled={!canManage || !selectedMetadataModel || descriptionGenerationActive || busy}
|
||||
onClick={() => void generateDescriptions()}
|
||||
>
|
||||
Generate {selectedIds.length === 1 ? "description" : "descriptions"}
|
||||
</Menu.Item>
|
||||
<Menu.Item
|
||||
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
|
||||
disabled={!canManage || busy}
|
||||
onClick={() => void consolidateDescriptions()}
|
||||
>
|
||||
Move generated description to Description
|
||||
</Menu.Item>
|
||||
</Menu.Popup>
|
||||
</Menu.Positioner>
|
||||
</Menu.Portal>
|
||||
</Menu.Root>
|
||||
<Button type="button" variant="ghost" onClick={() => { gridRef.current?.api.deselectAll(); setSelectedIds([]); }}><X />Clear</Button>
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<span className="thot-label whitespace-nowrap">Catalog columns</span>
|
||||
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search columns" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
|
||||
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
|
||||
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
|
||||
<Button type="button" disabled={!canManage || busy} onClick={onSync}><RefreshCw />Sync columns</Button>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
<div className="relative min-h-[280px] flex-1">
|
||||
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
|
||||
<AgGridReact<CatalogColumn> rowData={data} columnDefs={columns} context={context} loading={isLoading} quickFilterText={search} defaultColDef={{ sortable: true, filter: true, resizable: true }} getRowId={({ data: row }) => row.id} rowHeight={44} headerHeight={38} animateRows={false} overlayNoRowsTemplate="No columns synchronized for this table." />
|
||||
<AgGridReact<CatalogColumn>
|
||||
ref={gridRef}
|
||||
rowData={data}
|
||||
columnDefs={columns}
|
||||
context={context}
|
||||
loading={isLoading}
|
||||
quickFilterText={search}
|
||||
defaultColDef={{ sortable: true, filter: true, resizable: true }}
|
||||
getRowId={({ data: row }) => row.id}
|
||||
rowSelection={{ mode: "multiRow", selectAll: "filtered", enableClickSelection: false }}
|
||||
onSelectionChanged={({ api }) => setSelectedIds(api.getSelectedRows().map((column) => column.id))}
|
||||
rowHeight={44}
|
||||
headerHeight={38}
|
||||
animateRows={false}
|
||||
overlayNoRowsTemplate="No columns synchronized for this table."
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -15,6 +15,7 @@ import type {
|
||||
CatalogDatabase,
|
||||
CatalogDatabaseMetadataDeleteTarget,
|
||||
CatalogSyncScope,
|
||||
DescriptionGenerationScope,
|
||||
} from "../../api/catalog-databases";
|
||||
import { statusLabel } from "./model";
|
||||
import { databaseSyncItemClass, databaseSyncScopes } from "./DatabaseSyncMenu";
|
||||
@@ -35,6 +36,12 @@ interface DatabaseGridProps {
|
||||
onOpenSync: (row: CatalogDatabase) => void;
|
||||
onTestSelected: (rows: CatalogDatabase[]) => Promise<void>;
|
||||
onSyncSelected: (rows: CatalogDatabase[], scope: CatalogSyncScope) => Promise<void>;
|
||||
selectedMetadataModel: string | null;
|
||||
descriptionGenerationActive: boolean;
|
||||
onGenerateDescriptions: (
|
||||
rows: CatalogDatabase[],
|
||||
scope: Extract<DescriptionGenerationScope, "all" | "missing">,
|
||||
) => Promise<void>;
|
||||
onDeleteMetadataSelected: (
|
||||
rows: CatalogDatabase[],
|
||||
target: CatalogDatabaseMetadataDeleteTarget,
|
||||
@@ -154,14 +161,18 @@ export function DatabaseGrid({
|
||||
onOpenSync,
|
||||
onTestSelected,
|
||||
onSyncSelected,
|
||||
selectedMetadataModel,
|
||||
descriptionGenerationActive,
|
||||
onGenerateDescriptions,
|
||||
onDeleteMetadataSelected,
|
||||
}: DatabaseGridProps) {
|
||||
const compact = useCompactViewport();
|
||||
const gridRef = useRef<AgGridReact<CatalogDatabase>>(null);
|
||||
const actionsTriggerRef = useRef<HTMLButtonElement>(null);
|
||||
const [selectedRows, setSelectedRows] = useState<CatalogDatabase[]>([]);
|
||||
const [action, setAction] = useState<"test" | "sync" | "delete" | null>(null);
|
||||
const [action, setAction] = useState<"test" | "sync" | "generate" | "delete" | null>(null);
|
||||
const [pendingDelete, setPendingDelete] = useState<CatalogDatabaseMetadataDeleteTarget | null>(null);
|
||||
const [pendingGenerateAll, setPendingGenerateAll] = useState(false);
|
||||
const context = useMemo<DatabaseGridContext>(
|
||||
() => ({ canManage, onView, onEdit, onDelete, onOpenSync }),
|
||||
[canManage, onView, onEdit, onDelete, onOpenSync],
|
||||
@@ -247,6 +258,11 @@ export function DatabaseGrid({
|
||||
}).length;
|
||||
const canTestSelection = canManage && selectedRows.length > 0 && selectedRows.every((row) => row.configured && row.id && !row.activeSyncRun);
|
||||
const canSyncSelection = canManage && selectedRows.length > 0 && selectedRows.every((row) => row.configured && row.id && row.connectionStatus === "reachable" && row.testedVersion === row.version && !row.activeSyncRun);
|
||||
const canGenerateDescriptions = canManage
|
||||
&& selectedRows.length === 1
|
||||
&& Boolean(selectedMetadataModel)
|
||||
&& !descriptionGenerationActive
|
||||
&& selectedRows.every((row) => row.configured && row.id && !row.activeSyncRun);
|
||||
const canDeleteMetadataSelection = canManage && selectedRows.length > 0
|
||||
&& selectedRows.every((row) => row.configured && row.id && !row.activeSyncRun);
|
||||
useEffect(() => {
|
||||
@@ -254,18 +270,30 @@ export function DatabaseGrid({
|
||||
const timer = window.setTimeout(() => document.getElementById("database-cleanup-confirm-button")?.focus(), 0);
|
||||
return () => window.clearTimeout(timer);
|
||||
}, [pendingDelete]);
|
||||
useEffect(() => {
|
||||
if (!pendingGenerateAll) return;
|
||||
const timer = window.setTimeout(() => document.getElementById("database-generate-all-confirm-button")?.focus(), 0);
|
||||
return () => window.clearTimeout(timer);
|
||||
}, [pendingGenerateAll]);
|
||||
const closeDeleteConfirmation = () => {
|
||||
setPendingDelete(null);
|
||||
window.setTimeout(() => actionsTriggerRef.current?.focus(), 0);
|
||||
};
|
||||
const perform = async (kind: "test" | "sync" | "delete", operation: () => Promise<void>) => {
|
||||
const closeGenerateAllConfirmation = () => {
|
||||
setPendingGenerateAll(false);
|
||||
window.setTimeout(() => actionsTriggerRef.current?.focus(), 0);
|
||||
};
|
||||
const perform = async (kind: "test" | "sync" | "generate" | "delete", operation: () => Promise<void>) => {
|
||||
setAction(kind);
|
||||
try {
|
||||
await operation();
|
||||
gridRef.current?.api.deselectAll();
|
||||
setSelectedRows([]);
|
||||
setPendingDelete(null);
|
||||
setPendingGenerateAll(false);
|
||||
if (kind === "delete") window.setTimeout(() => searchInputRef.current?.focus(), 0);
|
||||
} catch {
|
||||
// The page-level operation owns safe error feedback. Preserve the selection for retry.
|
||||
} finally { setAction(null); }
|
||||
};
|
||||
|
||||
@@ -273,7 +301,38 @@ export function DatabaseGrid({
|
||||
<section aria-label="Workspace databases" className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
|
||||
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
|
||||
{selectedRows.length > 0 ? (
|
||||
pendingDelete ? (
|
||||
pendingGenerateAll ? (
|
||||
<div
|
||||
role="group"
|
||||
aria-labelledby="database-generate-all-confirmation"
|
||||
className="flex min-w-0 flex-1 flex-wrap items-center gap-2"
|
||||
onKeyDown={(event) => {
|
||||
if (event.key === "Escape" && action === null) closeGenerateAllConfirmation();
|
||||
}}
|
||||
>
|
||||
<div className="mr-auto min-w-56">
|
||||
<p id="database-generate-all-confirmation" className="text-sm font-semibold">
|
||||
Replace generated descriptions for {selectedRows[0]?.workspaceName}?
|
||||
</p>
|
||||
<p className="text-xs text-muted-foreground">
|
||||
Existing generated descriptions for eligible tables and columns will be replaced.
|
||||
</p>
|
||||
</div>
|
||||
<Button type="button" variant="ghost" disabled={action !== null} onClick={closeGenerateAllConfirmation}>Cancel</Button>
|
||||
<Button
|
||||
id="database-generate-all-confirm-button"
|
||||
type="button"
|
||||
variant="destructive"
|
||||
disabled={action !== null}
|
||||
onClick={() => void perform(
|
||||
"generate",
|
||||
() => onGenerateDescriptions(selectedRows, "all"),
|
||||
)}
|
||||
>
|
||||
Generate All
|
||||
</Button>
|
||||
</div>
|
||||
) : pendingDelete ? (
|
||||
<div
|
||||
role="group"
|
||||
aria-labelledby="database-cleanup-confirmation"
|
||||
@@ -327,6 +386,24 @@ export function DatabaseGrid({
|
||||
</Menu.Item>
|
||||
))}
|
||||
<Menu.Separator className="my-1 h-px bg-border" />
|
||||
<Menu.Item
|
||||
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
|
||||
disabled={!canGenerateDescriptions}
|
||||
onClick={() => setPendingGenerateAll(true)}
|
||||
>
|
||||
Generate All
|
||||
</Menu.Item>
|
||||
<Menu.Item
|
||||
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
|
||||
disabled={!canGenerateDescriptions}
|
||||
onClick={() => void perform(
|
||||
"generate",
|
||||
() => onGenerateDescriptions(selectedRows, "missing"),
|
||||
)}
|
||||
>
|
||||
Generate Missing
|
||||
</Menu.Item>
|
||||
<Menu.Separator className="my-1 h-px bg-border" />
|
||||
<Menu.Item
|
||||
className="rounded-md px-3 py-2 text-sm text-destructive outline-none data-[highlighted]:bg-destructive/10 data-[disabled]:opacity-45"
|
||||
disabled={!canDeleteMetadataSelection}
|
||||
@@ -383,6 +460,7 @@ export function DatabaseGrid({
|
||||
selectionColumnDef={{ width: 44, maxWidth: 44, pinned: "left" }}
|
||||
onSelectionChanged={({ api }) => {
|
||||
if (pendingDelete && action === null) setPendingDelete(null);
|
||||
if (pendingGenerateAll && action === null) setPendingGenerateAll(false);
|
||||
setSelectedRows(api.getSelectedRows());
|
||||
}}
|
||||
rowHeight={44}
|
||||
|
||||
@@ -8,14 +8,17 @@ import { toast } from "sonner";
|
||||
import { Button } from "../../components/ui/button";
|
||||
import { ApiError, apiErrorMessage } from "../../api/client";
|
||||
import {
|
||||
consolidateCatalogDescriptions,
|
||||
deleteCatalogTableMetadata,
|
||||
listCatalogTables,
|
||||
startCatalogSync,
|
||||
startDescriptionGenerationRun,
|
||||
updateCatalogTableMetadata,
|
||||
type CatalogDatabase,
|
||||
type CatalogSyncRun,
|
||||
type CatalogTable,
|
||||
type CatalogTableMetadataDeleteTarget,
|
||||
type DescriptionGenerationRun,
|
||||
} from "../../api/catalog-databases";
|
||||
import type { DatabaseNavigationState } from "./model";
|
||||
import { DatabaseColumns } from "./DatabaseColumns";
|
||||
@@ -24,12 +27,15 @@ interface Props {
|
||||
database: CatalogDatabase;
|
||||
canManage: boolean;
|
||||
activeRun?: CatalogSyncRun;
|
||||
selectedMetadataModel: string | null;
|
||||
descriptionGenerationActive: boolean;
|
||||
onBackToDatabases: () => void;
|
||||
onOpenOverview: () => void;
|
||||
onOpenRelationships: () => void;
|
||||
onNavigationStateChange: (state: DatabaseNavigationState) => void;
|
||||
onRunStarted: (run: CatalogSyncRun) => void;
|
||||
onOpenSync: () => void;
|
||||
onDescriptionGenerationRunStarted: (run: DescriptionGenerationRun) => void;
|
||||
}
|
||||
|
||||
interface TableGridContext {
|
||||
@@ -55,12 +61,15 @@ export function DatabaseTables({
|
||||
database,
|
||||
canManage,
|
||||
activeRun,
|
||||
selectedMetadataModel,
|
||||
descriptionGenerationActive,
|
||||
onBackToDatabases,
|
||||
onOpenOverview,
|
||||
onOpenRelationships,
|
||||
onNavigationStateChange,
|
||||
onRunStarted,
|
||||
onOpenSync,
|
||||
onDescriptionGenerationRunStarted,
|
||||
}: Props) {
|
||||
const databaseId = database.id!;
|
||||
const queryClient = useQueryClient();
|
||||
@@ -77,7 +86,7 @@ export function DatabaseTables({
|
||||
const [editorVersion, setEditorVersion] = useState<number | null>(null);
|
||||
const [stale, setStale] = useState(false);
|
||||
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
|
||||
const [busy, setBusy] = useState<"sync" | "save" | "delete" | null>(null);
|
||||
const [busy, setBusy] = useState<"sync" | "save" | "delete" | "consolidate" | "generate" | null>(null);
|
||||
const [pendingDelete, setPendingDelete] = useState<CatalogTableMetadataDeleteTarget | null>(null);
|
||||
const [columnNavigation, setColumnNavigation] = useState<DatabaseNavigationState>({ dirty: false, busy: false });
|
||||
const gridRef = useRef<AgGridReact<CatalogTable>>(null);
|
||||
@@ -203,6 +212,34 @@ export function DatabaseTables({
|
||||
toast.error(apiErrorMessage(error));
|
||||
} finally { setBusy(null); }
|
||||
};
|
||||
const consolidateDescriptions = async () => {
|
||||
setBusy("consolidate");
|
||||
try {
|
||||
const counts = await consolidateCatalogDescriptions(databaseId, "tables", selectedIds);
|
||||
await queryClient.invalidateQueries({ queryKey, exact: true });
|
||||
gridRef.current?.api.deselectAll();
|
||||
setSelectedIds([]);
|
||||
toast.success(`Copied ${counts.copied} description${counts.copied === 1 ? "" : "s"}; skipped ${counts.skipped}`);
|
||||
} catch (error) {
|
||||
toast.error(apiErrorMessage(error));
|
||||
} finally { setBusy(null); }
|
||||
};
|
||||
const generateDescriptions = async () => {
|
||||
if (selectedIds.length === 0 || !selectedMetadataModel) return;
|
||||
setBusy("generate");
|
||||
try {
|
||||
const run = await startDescriptionGenerationRun(
|
||||
databaseId,
|
||||
selectedMetadataModel,
|
||||
"selected_tables",
|
||||
selectedIds,
|
||||
);
|
||||
onDescriptionGenerationRunStarted(run);
|
||||
toast.success(`Description generation started for ${selectedIds.length} table${selectedIds.length === 1 ? "" : "s"}`);
|
||||
} catch (error) {
|
||||
toast.error(apiErrorMessage(error));
|
||||
} finally { setBusy(null); }
|
||||
};
|
||||
|
||||
const columns = useMemo<ColDef<CatalogTable>[]>(() => [
|
||||
{ field: "name", headerName: "Name", minWidth: 250, flex: 1, cellClass: "font-mono text-xs" },
|
||||
@@ -257,7 +294,16 @@ export function DatabaseTables({
|
||||
</nav>
|
||||
</div>
|
||||
{tableSection === "columns" ? (
|
||||
<DatabaseColumns databaseId={databaseId} table={activeTable} canManage={canManage} onNavigationStateChange={setColumnNavigation} onSync={() => void synchronize("columns", [activeTable.id])} />
|
||||
<DatabaseColumns
|
||||
databaseId={databaseId}
|
||||
table={activeTable}
|
||||
canManage={canManage}
|
||||
selectedMetadataModel={selectedMetadataModel}
|
||||
descriptionGenerationActive={descriptionGenerationActive}
|
||||
onDescriptionGenerationRunStarted={onDescriptionGenerationRunStarted}
|
||||
onNavigationStateChange={setColumnNavigation}
|
||||
onSync={() => void synchronize("columns", [activeTable.id])}
|
||||
/>
|
||||
) : (
|
||||
<div className="min-h-0 overflow-y-auto px-4 py-5">
|
||||
<p className="text-sm text-muted-foreground">Only review metadata can be changed.</p>
|
||||
@@ -334,6 +380,13 @@ export function DatabaseTables({
|
||||
<Menu.Portal>
|
||||
<Menu.Positioner side="bottom" align="start" sideOffset={4}>
|
||||
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
|
||||
<Menu.Item
|
||||
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
|
||||
disabled={!canManage || !selectedMetadataModel || descriptionGenerationActive || busy !== null || Boolean(currentRun)}
|
||||
onClick={() => void generateDescriptions()}
|
||||
>
|
||||
Generate {selectedIds.length === 1 ? "description" : "descriptions"}
|
||||
</Menu.Item>
|
||||
<Menu.Item
|
||||
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
|
||||
disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)}
|
||||
@@ -341,6 +394,13 @@ export function DatabaseTables({
|
||||
>
|
||||
Synchronize columns
|
||||
</Menu.Item>
|
||||
<Menu.Item
|
||||
className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45"
|
||||
disabled={!canManage || busy !== null || Boolean(currentRun)}
|
||||
onClick={() => void consolidateDescriptions()}
|
||||
>
|
||||
Move generated description to Description
|
||||
</Menu.Item>
|
||||
<Menu.Separator className="my-1 h-px bg-border" />
|
||||
<Menu.Item
|
||||
className="rounded-md px-3 py-2 text-sm text-destructive outline-none data-[highlighted]:bg-destructive/10 data-[disabled]:opacity-45"
|
||||
|
||||
@@ -0,0 +1,338 @@
|
||||
import { act, render, screen, waitFor, within } from "@testing-library/react";
|
||||
import userEvent from "@testing-library/user-event";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import { http, HttpResponse } from "msw";
|
||||
import { useState } from "react";
|
||||
import type {
|
||||
DescriptionGenerationEvent,
|
||||
DescriptionGenerationRun,
|
||||
} from "../../api/catalog-databases";
|
||||
import { Toaster } from "../../components/ui/sonner";
|
||||
import { FakeEventSource } from "../../test/fakeEventSource";
|
||||
import { server } from "../../test/msw";
|
||||
import { DescriptionGenerationDrawer } from "./DescriptionGenerationDrawer";
|
||||
|
||||
const runningRun: DescriptionGenerationRun = {
|
||||
id: "77777777-7777-4777-8777-777777777777",
|
||||
databaseId: "11111111-1111-4111-8111-111111111111",
|
||||
scope: "missing",
|
||||
modelId: "local-qwen",
|
||||
language: "it",
|
||||
status: "running",
|
||||
total: 4,
|
||||
processed: 1,
|
||||
generated: 1,
|
||||
nonGeneratable: 0,
|
||||
failed: 0,
|
||||
createdAt: "2026-08-28T08:00:00Z",
|
||||
startedAt: "2026-08-28T08:00:01Z",
|
||||
updatedAt: new Date().toISOString(),
|
||||
finishedAt: null,
|
||||
errorSummary: null,
|
||||
};
|
||||
|
||||
function event(
|
||||
sequence: number,
|
||||
message: string,
|
||||
level: DescriptionGenerationEvent["level"] = "info",
|
||||
): DescriptionGenerationEvent {
|
||||
return {
|
||||
sequence,
|
||||
level,
|
||||
message,
|
||||
createdAt: `2026-08-28T08:00:0${sequence}Z`,
|
||||
};
|
||||
}
|
||||
|
||||
function renderDrawer({
|
||||
initialRun = runningRun,
|
||||
onTerminal = vi.fn(),
|
||||
}: {
|
||||
initialRun?: DescriptionGenerationRun;
|
||||
onTerminal?: (run: DescriptionGenerationRun) => void;
|
||||
} = {}) {
|
||||
const client = new QueryClient({
|
||||
defaultOptions: { queries: { retry: false, gcTime: Infinity } },
|
||||
});
|
||||
|
||||
function Harness() {
|
||||
const [run, setRun] = useState(initialRun);
|
||||
return (
|
||||
<DescriptionGenerationDrawer
|
||||
open
|
||||
run={run}
|
||||
modelLabel="Local Qwen"
|
||||
onClose={() => undefined}
|
||||
onRunUpdate={setRun}
|
||||
onTerminal={onTerminal}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
client,
|
||||
...render(
|
||||
<QueryClientProvider client={client}>
|
||||
<Harness />
|
||||
<Toaster duration={Infinity} />
|
||||
</QueryClientProvider>,
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
const nativeEventSource = globalThis.EventSource;
|
||||
|
||||
beforeEach(() => {
|
||||
FakeEventSource.instances = [];
|
||||
(globalThis as unknown as { EventSource: typeof EventSource }).EventSource = (
|
||||
FakeEventSource as unknown as typeof EventSource
|
||||
);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
(globalThis as unknown as { EventSource: typeof EventSource }).EventSource = nativeEventSource;
|
||||
});
|
||||
|
||||
test("replays ordered SSE events, reconnects from the latest sequence, and keeps polling as a deduplicated fallback", async () => {
|
||||
let eventReads = 0;
|
||||
server.use(
|
||||
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([runningRun])),
|
||||
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(runningRun)),
|
||||
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => {
|
||||
eventReads += 1;
|
||||
return HttpResponse.json(eventReads === 1 ? [event(1, "Run queued")] : [
|
||||
event(2, "First description saved"),
|
||||
event(3, "Polling recovered this event"),
|
||||
]);
|
||||
}),
|
||||
);
|
||||
renderDrawer();
|
||||
|
||||
const log = await screen.findByRole("log", { name: "Description generation events" });
|
||||
expect(await within(log).findByText("Run queued")).toBeVisible();
|
||||
await waitFor(() => expect(FakeEventSource.instances).toHaveLength(1));
|
||||
expect(FakeEventSource.instances[0].url).toBe(
|
||||
`/api/catalog/description-generation-runs/${runningRun.id}/events?after=1`,
|
||||
);
|
||||
|
||||
act(() => {
|
||||
FakeEventSource.instances[0].emitNamed("log", event(2, "First description saved"), "2");
|
||||
FakeEventSource.instances[0].emitNamed("log", event(2, "First description saved"), "2");
|
||||
FakeEventSource.instances[0].onerror?.();
|
||||
});
|
||||
|
||||
expect(await within(log).findByText("Polling recovered this event")).toBeVisible();
|
||||
await waitFor(() => expect(FakeEventSource.instances).toHaveLength(2));
|
||||
expect(FakeEventSource.instances[1].url).toBe(
|
||||
`/api/catalog/description-generation-runs/${runningRun.id}/events?after=3`,
|
||||
);
|
||||
|
||||
act(() => FakeEventSource.instances[0].onerror?.());
|
||||
await new Promise((resolve) => window.setTimeout(resolve, 250));
|
||||
expect(FakeEventSource.instances).toHaveLength(2);
|
||||
expect(FakeEventSource.instances[1].closed).toBe(false);
|
||||
|
||||
act(() => {
|
||||
FakeEventSource.instances[1].emitNamed("log", event(3, "Polling recovered this event"), "3");
|
||||
FakeEventSource.instances[1].emitNamed("log", event(4, "Replay continued in order"), "4");
|
||||
});
|
||||
|
||||
await waitFor(() => expect(within(log).getAllByText("First description saved")).toHaveLength(1));
|
||||
expect(within(log).getAllByText("Polling recovered this event")).toHaveLength(1);
|
||||
expect(within(log).getByText("Replay continued in order")).toBeVisible();
|
||||
expect(log.textContent!.indexOf("Run queued")).toBeLessThan(
|
||||
log.textContent!.indexOf("First description saved"),
|
||||
);
|
||||
expect(log.textContent!.indexOf("First description saved")).toBeLessThan(
|
||||
log.textContent!.indexOf("Polling recovered this event"),
|
||||
);
|
||||
});
|
||||
|
||||
test("stops queued or running work without hiding earlier results or events", async () => {
|
||||
const user = userEvent.setup();
|
||||
const cancelledRun: DescriptionGenerationRun = {
|
||||
...runningRun,
|
||||
status: "cancelled",
|
||||
processed: 2,
|
||||
generated: 2,
|
||||
updatedAt: "2026-08-28T08:03:00Z",
|
||||
finishedAt: "2026-08-28T08:03:00Z",
|
||||
};
|
||||
let cancelledRunId: string | undefined;
|
||||
server.use(
|
||||
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([cancelledRun])),
|
||||
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(runningRun)),
|
||||
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([
|
||||
event(1, "First description remains saved"),
|
||||
])),
|
||||
http.post("/api/catalog/description-generation-runs/:runId/cancel", ({ params }) => {
|
||||
cancelledRunId = String(params.runId);
|
||||
return HttpResponse.json(cancelledRun);
|
||||
}),
|
||||
);
|
||||
renderDrawer();
|
||||
|
||||
const drawer = await screen.findByRole("complementary", { name: "Description generation" });
|
||||
expect(await within(drawer).findByText("First description remains saved")).toBeVisible();
|
||||
await user.click(within(drawer).getByRole("button", { name: "Stop" }));
|
||||
|
||||
expect(await within(drawer).findByRole("heading", { name: "Cancelled" })).toBeVisible();
|
||||
expect(cancelledRunId).toBe(runningRun.id);
|
||||
expect(within(drawer).getByText("2", { selector: "[data-count='generated']" })).toBeVisible();
|
||||
expect(within(drawer).getByText("First description remains saved")).toBeVisible();
|
||||
});
|
||||
|
||||
test.each([
|
||||
"completed",
|
||||
"completed_with_errors",
|
||||
"failed",
|
||||
"cancelled",
|
||||
"interrupted",
|
||||
] satisfies DescriptionGenerationRun["status"][])(
|
||||
"notifies catalog consumers when a run becomes terminal with %s",
|
||||
async (status) => {
|
||||
const onTerminal = vi.fn();
|
||||
const terminalRun: DescriptionGenerationRun = {
|
||||
...runningRun,
|
||||
status,
|
||||
finishedAt: "2026-08-28T08:04:00Z",
|
||||
};
|
||||
server.use(
|
||||
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([terminalRun])),
|
||||
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(terminalRun)),
|
||||
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
|
||||
);
|
||||
|
||||
renderDrawer({ initialRun: terminalRun, onTerminal });
|
||||
|
||||
await waitFor(() => expect(onTerminal).toHaveBeenCalledOnce());
|
||||
expect(onTerminal).toHaveBeenCalledWith(terminalRun);
|
||||
},
|
||||
);
|
||||
|
||||
test("reopens a terminal run from newest-first persisted history", async () => {
|
||||
const user = userEvent.setup();
|
||||
const completedRun: DescriptionGenerationRun = {
|
||||
...runningRun,
|
||||
id: "88888888-8888-4888-8888-888888888888",
|
||||
scope: "selected_tables",
|
||||
status: "completed",
|
||||
total: 3,
|
||||
processed: 3,
|
||||
generated: 3,
|
||||
createdAt: "2026-08-28T07:00:00Z",
|
||||
updatedAt: "2026-08-28T07:02:00Z",
|
||||
finishedAt: "2026-08-28T07:02:00Z",
|
||||
};
|
||||
server.use(
|
||||
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([
|
||||
runningRun,
|
||||
completedRun,
|
||||
])),
|
||||
http.get("/api/catalog/description-generation-runs/:runId", ({ params }) => (
|
||||
HttpResponse.json(params.runId === completedRun.id ? completedRun : runningRun)
|
||||
)),
|
||||
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
|
||||
);
|
||||
renderDrawer();
|
||||
|
||||
const history = await screen.findByRole("region", {
|
||||
name: "Description generation history",
|
||||
});
|
||||
await user.click(within(history).getByRole("button", {
|
||||
name: /Completed.*selected tables/i,
|
||||
}));
|
||||
|
||||
const drawer = screen.getByRole("complementary", { name: "Description generation" });
|
||||
expect(await within(drawer).findByRole("heading", { name: "Completed" })).toBeVisible();
|
||||
expect(within(drawer).getByText("3", { selector: "[data-count='generated']" })).toBeVisible();
|
||||
});
|
||||
|
||||
test("offers Unlock only for an apparently stale active run and explains the interruption", async () => {
|
||||
const user = userEvent.setup();
|
||||
const staleRun: DescriptionGenerationRun = {
|
||||
...runningRun,
|
||||
updatedAt: "2020-01-01T00:00:00Z",
|
||||
};
|
||||
const interruptedRun: DescriptionGenerationRun = {
|
||||
...staleRun,
|
||||
status: "interrupted",
|
||||
finishedAt: "2026-08-28T08:05:00Z",
|
||||
};
|
||||
server.use(
|
||||
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([staleRun])),
|
||||
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(staleRun)),
|
||||
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
|
||||
http.post("/api/catalog/description-generation-runs/unlock", () => (
|
||||
HttpResponse.json(interruptedRun)
|
||||
)),
|
||||
);
|
||||
renderDrawer({ initialRun: staleRun });
|
||||
|
||||
const drawer = await screen.findByRole("complementary", { name: "Description generation" });
|
||||
expect(within(drawer).getByText(
|
||||
"Unlock interrupts apparently stale work and marks the run interrupted. A live worker will prevent it.",
|
||||
)).toBeVisible();
|
||||
await user.click(within(drawer).getByRole("button", { name: "Unlock stale run" }));
|
||||
|
||||
expect(await within(drawer).findByRole("heading", { name: "Interrupted" })).toBeVisible();
|
||||
});
|
||||
|
||||
test("does not offer Unlock while an active run still appears current", async () => {
|
||||
server.use(
|
||||
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([runningRun])),
|
||||
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(runningRun)),
|
||||
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
|
||||
);
|
||||
renderDrawer();
|
||||
|
||||
const drawer = await screen.findByRole("complementary", { name: "Description generation" });
|
||||
expect(within(drawer).queryByRole("button", { name: "Unlock stale run" })).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
test("explains when Unlock is refused because a live worker still exists", async () => {
|
||||
const user = userEvent.setup();
|
||||
const staleRun: DescriptionGenerationRun = {
|
||||
...runningRun,
|
||||
updatedAt: "2020-01-01T00:00:00Z",
|
||||
};
|
||||
server.use(
|
||||
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([staleRun])),
|
||||
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(staleRun)),
|
||||
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
|
||||
http.post("/api/catalog/description-generation-runs/unlock", () => HttpResponse.json({
|
||||
code: "description_generation_run_active",
|
||||
message: "Private worker detail must not be shown.",
|
||||
}, { status: 409 })),
|
||||
);
|
||||
renderDrawer({ initialRun: staleRun });
|
||||
|
||||
const drawer = await screen.findByRole("complementary", { name: "Description generation" });
|
||||
await user.click(within(drawer).getByRole("button", { name: "Unlock stale run" }));
|
||||
|
||||
expect(await screen.findByText(
|
||||
"Unlock was refused because a live description worker is still running.",
|
||||
)).toBeVisible();
|
||||
expect(screen.queryByText("Private worker detail must not be shown.")).not.toBeInTheDocument();
|
||||
expect(within(drawer).getByRole("heading", { name: "Running" })).toBeVisible();
|
||||
});
|
||||
|
||||
test("does not render stack traces or sensitive generation details from a final summary", async () => {
|
||||
const failedRun: DescriptionGenerationRun = {
|
||||
...runningRun,
|
||||
status: "failed",
|
||||
finishedAt: "2026-08-28T08:05:00Z",
|
||||
errorSummary: "Traceback: provider payload included prompt and source sample\n at helper.py:42",
|
||||
};
|
||||
server.use(
|
||||
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([failedRun])),
|
||||
http.get("/api/catalog/description-generation-runs/:runId", () => HttpResponse.json(failedRun)),
|
||||
http.get("/api/catalog/description-generation-runs/:runId/events-list", () => HttpResponse.json([])),
|
||||
);
|
||||
renderDrawer({ initialRun: failedRun });
|
||||
|
||||
expect(await screen.findByText(
|
||||
"The run ended before all targets were processed. Review the event log for safe details.",
|
||||
)).toBeVisible();
|
||||
expect(screen.queryByText(/Traceback|provider payload|source sample/i)).not.toBeInTheDocument();
|
||||
});
|
||||
@@ -0,0 +1,413 @@
|
||||
import { useCallback, useEffect, useRef, useState } from "react";
|
||||
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { LoaderCircle, LockOpen, Square, X } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
import { Button } from "../../components/ui/button";
|
||||
import { ApiError, apiErrorMessage } from "../../api/client";
|
||||
import {
|
||||
cancelDescriptionGenerationRun,
|
||||
descriptionGenerationEventsUrl,
|
||||
getDescriptionGenerationRun,
|
||||
listDescriptionGenerationEvents,
|
||||
listDescriptionGenerationRuns,
|
||||
type DescriptionGenerationEvent,
|
||||
type DescriptionGenerationRun,
|
||||
unlockDescriptionGenerationRun,
|
||||
} from "../../api/catalog-databases";
|
||||
|
||||
interface Props {
|
||||
open: boolean;
|
||||
run: DescriptionGenerationRun | null;
|
||||
modelLabel: string;
|
||||
onClose: () => void;
|
||||
onRunUpdate: (run: DescriptionGenerationRun) => void;
|
||||
onTerminal: (run: DescriptionGenerationRun) => void;
|
||||
}
|
||||
|
||||
const APPARENTLY_STALE_AFTER_MS = 5 * 60 * 1_000;
|
||||
const SAFE_FINAL_ERROR = "The run ended before all targets were processed. Review the event log for safe details.";
|
||||
|
||||
function statusLabel(run: DescriptionGenerationRun): string {
|
||||
const label = run.status.replaceAll("_", " ");
|
||||
return `${label[0].toUpperCase()}${label.slice(1)}`;
|
||||
}
|
||||
|
||||
function isTerminal(run?: DescriptionGenerationRun): boolean {
|
||||
return Boolean(run && [
|
||||
"completed", "completed_with_errors", "cancelled", "failed", "interrupted",
|
||||
].includes(run.status));
|
||||
}
|
||||
|
||||
function isApparentlyStale(run: DescriptionGenerationRun): boolean {
|
||||
if (!["queued", "running"].includes(run.status)) return false;
|
||||
const updatedAt = new Date(run.updatedAt).getTime();
|
||||
return Number.isFinite(updatedAt) && Date.now() - updatedAt >= APPARENTLY_STALE_AFTER_MS;
|
||||
}
|
||||
|
||||
function safeFinalError(summary: string | null): string | null {
|
||||
const value = summary?.trim();
|
||||
if (!value) return null;
|
||||
if (
|
||||
value.length > 500
|
||||
|| /[\r\n]/.test(value)
|
||||
|| /traceback|stack trace|(?:^|\s)at\s+\S+[:(]|prompt|provider\s+payload|source\s+sample|bearer\s+\S+|api[_ -]?key|sk-[a-z0-9_-]+/i.test(value)
|
||||
) return SAFE_FINAL_ERROR;
|
||||
return value;
|
||||
}
|
||||
|
||||
function timestamp(value: string | null): string {
|
||||
return value ? new Date(value).toLocaleString() : "Not available";
|
||||
}
|
||||
|
||||
export function DescriptionGenerationDrawer({
|
||||
open,
|
||||
run: initialRun,
|
||||
modelLabel,
|
||||
onClose,
|
||||
onRunUpdate,
|
||||
onTerminal,
|
||||
}: Props) {
|
||||
const queryClient = useQueryClient();
|
||||
const runId = initialRun?.id ?? null;
|
||||
const [events, setEvents] = useState<DescriptionGenerationEvent[]>([]);
|
||||
const [action, setAction] = useState<"stop" | "unlock" | null>(null);
|
||||
const lastSequence = useRef(0);
|
||||
const terminalRuns = useRef(new Set<string>());
|
||||
const finalEventsRun = useRef<string | null>(null);
|
||||
const runQuery = useQuery({
|
||||
queryKey: ["description-generation-run", runId],
|
||||
queryFn: () => getDescriptionGenerationRun(runId!),
|
||||
enabled: Boolean(runId),
|
||||
initialData: initialRun ?? undefined,
|
||||
retry: false,
|
||||
refetchInterval: (query) => isTerminal(query.state.data) ? false : 1_000,
|
||||
});
|
||||
const run = runQuery.data;
|
||||
const eventQuery = useQuery({
|
||||
queryKey: ["description-generation-events", runId],
|
||||
queryFn: () => listDescriptionGenerationEvents(runId!, lastSequence.current),
|
||||
enabled: Boolean(runId),
|
||||
retry: false,
|
||||
refetchInterval: run && isTerminal(run) ? false : 1_500,
|
||||
});
|
||||
const historyQuery = useQuery({
|
||||
queryKey: ["description-generation-runs", 50],
|
||||
queryFn: () => listDescriptionGenerationRuns(50),
|
||||
enabled: open,
|
||||
retry: false,
|
||||
refetchInterval: (query) => query.state.data?.some((item) => !isTerminal(item))
|
||||
? 3_000
|
||||
: false,
|
||||
});
|
||||
|
||||
const mergeEvents = useCallback((incoming: DescriptionGenerationEvent[]) => {
|
||||
if (incoming.length === 0) return;
|
||||
setEvents((current) => {
|
||||
const bySequence = new Map(current.map((event) => [event.sequence, event]));
|
||||
for (const event of incoming) bySequence.set(event.sequence, event);
|
||||
const merged = [...bySequence.values()].sort((left, right) => left.sequence - right.sequence);
|
||||
lastSequence.current = merged.at(-1)?.sequence ?? lastSequence.current;
|
||||
return merged;
|
||||
});
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
setEvents([]);
|
||||
lastSequence.current = 0;
|
||||
finalEventsRun.current = null;
|
||||
}, [runId]);
|
||||
|
||||
useEffect(() => {
|
||||
mergeEvents(eventQuery.data ?? []);
|
||||
}, [eventQuery.data, mergeEvents]);
|
||||
|
||||
const live = Boolean(run && !isTerminal(run));
|
||||
useEffect(() => {
|
||||
if (
|
||||
!runId
|
||||
|| !open
|
||||
|| !live
|
||||
|| !eventQuery.isFetched
|
||||
|| typeof EventSource === "undefined"
|
||||
) return;
|
||||
|
||||
let source: EventSource | null = null;
|
||||
let reconnectTimer: number | null = null;
|
||||
let disposed = false;
|
||||
|
||||
const connect = () => {
|
||||
if (disposed) return;
|
||||
const polledSequence = (eventQuery.data ?? []).reduce(
|
||||
(latest, event) => Math.max(latest, event.sequence),
|
||||
0,
|
||||
);
|
||||
lastSequence.current = Math.max(lastSequence.current, polledSequence);
|
||||
let currentSource: EventSource;
|
||||
try {
|
||||
currentSource = new EventSource(descriptionGenerationEventsUrl(runId, lastSequence.current));
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
source = currentSource;
|
||||
const isCurrent = () => !disposed && source === currentSource;
|
||||
const log = (message: MessageEvent<string>) => {
|
||||
if (!isCurrent()) return;
|
||||
try {
|
||||
mergeEvents([JSON.parse(message.data) as DescriptionGenerationEvent]);
|
||||
} catch { /* ordered polling remains authoritative */ }
|
||||
};
|
||||
const update = (message: MessageEvent<string>) => {
|
||||
if (!isCurrent()) return;
|
||||
try {
|
||||
queryClient.setQueryData(
|
||||
["description-generation-run", runId],
|
||||
JSON.parse(message.data) as DescriptionGenerationRun,
|
||||
);
|
||||
} catch { /* status polling remains authoritative */ }
|
||||
};
|
||||
const message = (incoming: MessageEvent<string>) => {
|
||||
if (!isCurrent()) return;
|
||||
try {
|
||||
const parsed = JSON.parse(incoming.data) as DescriptionGenerationEvent | DescriptionGenerationRun;
|
||||
if ("sequence" in parsed) mergeEvents([parsed]);
|
||||
else if ("status" in parsed) queryClient.setQueryData(["description-generation-run", runId], parsed);
|
||||
} catch { /* polling remains authoritative */ }
|
||||
};
|
||||
currentSource.addEventListener("log", log as EventListener);
|
||||
currentSource.addEventListener("run", update as EventListener);
|
||||
currentSource.onmessage = message;
|
||||
currentSource.onerror = () => {
|
||||
if (!isCurrent() || reconnectTimer !== null) return;
|
||||
currentSource.close();
|
||||
source = null;
|
||||
void eventQuery.refetch().finally(() => {
|
||||
if (disposed || reconnectTimer !== null) return;
|
||||
reconnectTimer = window.setTimeout(() => {
|
||||
reconnectTimer = null;
|
||||
connect();
|
||||
}, 200);
|
||||
});
|
||||
};
|
||||
};
|
||||
|
||||
connect();
|
||||
return () => {
|
||||
disposed = true;
|
||||
source?.close();
|
||||
if (reconnectTimer !== null) window.clearTimeout(reconnectTimer);
|
||||
};
|
||||
}, [eventQuery.isFetched, eventQuery.refetch, live, mergeEvents, open, queryClient, runId]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!run) return;
|
||||
onRunUpdate(run);
|
||||
queryClient.setQueryData<DescriptionGenerationRun[]>(
|
||||
["description-generation-runs", 50],
|
||||
(current) => {
|
||||
if (!current) return [run];
|
||||
return current.some((item) => item.id === run.id)
|
||||
? current.map((item) => item.id === run.id ? run : item)
|
||||
: [run, ...current];
|
||||
},
|
||||
);
|
||||
}, [onRunUpdate, queryClient, run]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!run || !isTerminal(run) || finalEventsRun.current === run.id) return;
|
||||
finalEventsRun.current = run.id;
|
||||
void eventQuery.refetch();
|
||||
}, [eventQuery.refetch, run]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!run || !isTerminal(run) || terminalRuns.current.has(run.id)) return;
|
||||
terminalRuns.current.add(run.id);
|
||||
onTerminal(run);
|
||||
}, [onTerminal, run]);
|
||||
|
||||
const stop = async () => {
|
||||
if (!run || isTerminal(run)) return;
|
||||
setAction("stop");
|
||||
try {
|
||||
const next = await cancelDescriptionGenerationRun(run.id);
|
||||
queryClient.setQueryData(["description-generation-run", run.id], next);
|
||||
onRunUpdate(next);
|
||||
await historyQuery.refetch();
|
||||
} catch (error) {
|
||||
toast.error(apiErrorMessage(error));
|
||||
} finally {
|
||||
setAction(null);
|
||||
}
|
||||
};
|
||||
|
||||
const unlock = async () => {
|
||||
if (!run || !isApparentlyStale(run)) return;
|
||||
setAction("unlock");
|
||||
try {
|
||||
const next = await unlockDescriptionGenerationRun();
|
||||
queryClient.setQueryData(["description-generation-run", next.id], next);
|
||||
onRunUpdate(next);
|
||||
await historyQuery.refetch();
|
||||
} catch (error) {
|
||||
toast.error(error instanceof ApiError && error.status === 409
|
||||
? "Unlock was refused because a live description worker is still running."
|
||||
: apiErrorMessage(error));
|
||||
} finally {
|
||||
setAction(null);
|
||||
}
|
||||
};
|
||||
|
||||
if (!open || !run) return null;
|
||||
const apparentlyStale = isApparentlyStale(run);
|
||||
const finalError = safeFinalError(run.errorSummary);
|
||||
return (
|
||||
<aside
|
||||
aria-label="Description generation"
|
||||
className="fixed inset-y-2 right-0 z-50 flex w-full max-w-[400px] flex-col border-l border-border bg-background shadow-2xl sm:inset-y-4"
|
||||
>
|
||||
<div className="flex items-start justify-between gap-4 border-b border-border px-5 py-4">
|
||||
<div>
|
||||
<p className="thot-label">Description generation</p>
|
||||
<h2 className="mt-1 font-heading text-xl font-semibold">{statusLabel(run)}</h2>
|
||||
<p className="mt-1 text-sm text-muted-foreground">
|
||||
{modelLabel}{modelLabel !== run.modelId ? ` (${run.modelId})` : ""} · {run.scope.replaceAll("_", " ")}
|
||||
</p>
|
||||
</div>
|
||||
<Button type="button" variant="ghost" size="icon-lg" aria-label="Close description generation" onClick={onClose}>
|
||||
<X aria-hidden="true" />
|
||||
</Button>
|
||||
</div>
|
||||
<div className="min-h-0 flex-1 overflow-y-auto px-5 py-5">
|
||||
{runQuery.isError ? (
|
||||
<div role="alert" className="rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
|
||||
{apiErrorMessage(runQuery.error)}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
<section aria-label="Description generation counters">
|
||||
<h3 className="thot-label mb-2">Progress</h3>
|
||||
<div className="grid grid-cols-2 gap-2">
|
||||
{([
|
||||
["total", "Total"],
|
||||
["processed", "Processed"],
|
||||
["generated", "Generated"],
|
||||
["nonGeneratable", "Not generated"],
|
||||
["failed", "Failed"],
|
||||
] as const).map(([name, label]) => (
|
||||
<div key={name} className="rounded-md border border-border bg-muted/25 px-3 py-2">
|
||||
<p className="text-[11px] font-semibold uppercase tracking-wide text-muted-foreground">{label}</p>
|
||||
<p data-count={name} className="mt-1 text-lg font-semibold tabular-nums">{run[name]}</p>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section className="mt-5" aria-label="Description generation timestamps">
|
||||
<h3 className="thot-label mb-2">Timestamps</h3>
|
||||
<dl className="grid grid-cols-[auto_1fr] gap-x-4 gap-y-1.5 text-sm">
|
||||
<dt className="text-muted-foreground">Created</dt><dd className="text-right">{timestamp(run.createdAt)}</dd>
|
||||
<dt className="text-muted-foreground">Started</dt><dd className="text-right">{timestamp(run.startedAt)}</dd>
|
||||
<dt className="text-muted-foreground">Updated</dt><dd className="text-right">{timestamp(run.updatedAt)}</dd>
|
||||
<dt className="text-muted-foreground">Finished</dt><dd className="text-right">{timestamp(run.finishedAt)}</dd>
|
||||
</dl>
|
||||
</section>
|
||||
|
||||
{finalError ? (
|
||||
<div className="mt-5 rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
|
||||
{finalError}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
<section className="mt-5" aria-label="Description generation log">
|
||||
<div className="mb-2 flex items-center justify-between">
|
||||
<h3 className="thot-label">Events</h3>
|
||||
<span className="text-xs tabular-nums text-muted-foreground">{events.length}</span>
|
||||
</div>
|
||||
<div
|
||||
role="log"
|
||||
aria-label="Description generation events"
|
||||
className="max-h-72 overflow-y-auto rounded-md bg-zinc-950 p-3 font-mono text-xs leading-5 text-zinc-200"
|
||||
>
|
||||
{events.length === 0 ? <p className="text-zinc-500">Waiting for events…</p> : events.map((event) => (
|
||||
<p key={event.sequence} className={event.level === "error" ? "text-red-300" : event.level === "warning" ? "text-amber-300" : undefined}>
|
||||
<span className="mr-2 text-zinc-500">{new Date(event.createdAt).toLocaleTimeString()}</span>{event.message}
|
||||
</p>
|
||||
))}
|
||||
</div>
|
||||
{eventQuery.isError ? <p role="alert" className="mt-2 text-sm text-destructive">{apiErrorMessage(eventQuery.error)}</p> : null}
|
||||
</section>
|
||||
|
||||
<section className="mt-5" aria-label="Description generation history">
|
||||
<div className="mb-2 flex items-center justify-between gap-3">
|
||||
<h3 className="thot-label">Recent runs</h3>
|
||||
<span className="text-xs tabular-nums text-muted-foreground">
|
||||
{historyQuery.data?.length ?? 0}
|
||||
</span>
|
||||
</div>
|
||||
{historyQuery.isError ? (
|
||||
<p role="alert" className="text-sm text-destructive">
|
||||
{apiErrorMessage(historyQuery.error)}
|
||||
</p>
|
||||
) : historyQuery.isLoading ? (
|
||||
<p className="text-sm text-muted-foreground">Loading run history…</p>
|
||||
) : (historyQuery.data ?? []).length === 0 ? (
|
||||
<p className="text-sm text-muted-foreground">No description generation runs yet.</p>
|
||||
) : (
|
||||
<div className="max-h-56 divide-y divide-border overflow-y-auto rounded-md border border-border">
|
||||
{(historyQuery.data ?? []).map((item) => (
|
||||
<button
|
||||
key={item.id}
|
||||
type="button"
|
||||
aria-label={`${statusLabel(item)} · ${item.scope.replaceAll("_", " ")}`}
|
||||
aria-current={item.id === run.id ? "true" : undefined}
|
||||
className="flex w-full items-start justify-between gap-3 px-3 py-2 text-left text-sm hover:bg-muted/50 focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/25"
|
||||
onClick={() => onRunUpdate(item)}
|
||||
>
|
||||
<span className="font-medium">{statusLabel(item)}</span>
|
||||
<span className="shrink-0 text-right text-xs text-muted-foreground">
|
||||
{item.processed}/{item.total}<br />{new Date(item.createdAt).toLocaleString()}
|
||||
</span>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
</section>
|
||||
</div>
|
||||
{["queued", "running"].includes(run.status) ? (
|
||||
<div className="border-t border-border px-5 py-4">
|
||||
{apparentlyStale ? (
|
||||
<p
|
||||
id="description-generation-unlock-help"
|
||||
className="mb-3 rounded-md border border-amber-500/35 bg-amber-500/8 px-3 py-2 text-xs leading-5 text-muted-foreground"
|
||||
>
|
||||
Unlock interrupts apparently stale work and marks the run interrupted. A live worker will prevent it.
|
||||
</p>
|
||||
) : null}
|
||||
<div className="flex flex-wrap justify-end gap-2">
|
||||
{apparentlyStale ? (
|
||||
<Button
|
||||
type="button"
|
||||
variant="outline"
|
||||
aria-label="Unlock stale run"
|
||||
aria-describedby="description-generation-unlock-help"
|
||||
disabled={action !== null}
|
||||
onClick={() => void unlock()}
|
||||
>
|
||||
{action === "unlock" ? <LoaderCircle className="animate-spin" /> : <LockOpen />}
|
||||
{action === "unlock" ? "Unlocking…" : "Unlock"}
|
||||
</Button>
|
||||
) : null}
|
||||
<Button
|
||||
type="button"
|
||||
variant="destructive"
|
||||
disabled={action !== null}
|
||||
onClick={() => void stop()}
|
||||
>
|
||||
{action === "stop" ? <LoaderCircle className="animate-spin" /> : <Square />}
|
||||
{action === "stop" ? "Stopping…" : "Stop"}
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
) : null}
|
||||
</aside>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
import type { MetadataGenerationModels } from "../../api/catalog-databases";
|
||||
|
||||
interface Props {
|
||||
canManage: boolean;
|
||||
data?: MetadataGenerationModels;
|
||||
isError: boolean;
|
||||
isLoading: boolean;
|
||||
selectedModel: string;
|
||||
onSelectedModelChange: (modelId: string) => void;
|
||||
}
|
||||
|
||||
export function MetadataGenerationModelSelector({
|
||||
canManage,
|
||||
data,
|
||||
isError,
|
||||
isLoading,
|
||||
selectedModel,
|
||||
onSelectedModelChange,
|
||||
}: Props) {
|
||||
const models = data?.models ?? [];
|
||||
const configuredDefault = data?.default ?? "";
|
||||
const hasUsableDefault = configuredDefault !== ""
|
||||
&& models.some((model) => model.id === configuredDefault);
|
||||
const noUsableModel = !isLoading && !isError && Boolean(data) && !hasUsableDefault;
|
||||
const unavailable = noUsableModel || (!isLoading && isError);
|
||||
const describedBy = [
|
||||
"metadata-generation-source-data-disclosure",
|
||||
unavailable ? "metadata-generation-model-help" : null,
|
||||
].filter(Boolean).join(" ");
|
||||
return (
|
||||
<div className="grid min-w-56 max-w-lg gap-1.5">
|
||||
<label className="grid gap-1.5 text-xs font-medium text-foreground">
|
||||
<span>Metadata description model</span>
|
||||
<select
|
||||
className="h-9 w-full rounded-md border border-input bg-card px-3 text-sm outline-none transition focus:border-primary/60 focus:ring-3 focus:ring-ring/15 disabled:bg-muted disabled:text-muted-foreground"
|
||||
aria-label="Metadata description model"
|
||||
aria-describedby={describedBy}
|
||||
value={selectedModel}
|
||||
disabled={!canManage || isLoading || isError || !hasUsableDefault}
|
||||
onChange={(event) => onSelectedModelChange(event.target.value)}
|
||||
>
|
||||
{isLoading ? <option value="">Loading models…</option> : null}
|
||||
{!isLoading && isError ? <option value="">Model choices unavailable</option> : null}
|
||||
{noUsableModel ? (
|
||||
<option value="">{models.length === 0 ? "No model configured" : "No usable model configured"}</option>
|
||||
) : null}
|
||||
{models.map((model) => (
|
||||
<option key={model.id} value={model.id}>{model.label}</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
{unavailable ? (
|
||||
<p id="metadata-generation-model-help" className="max-w-72 text-xs font-normal leading-4 text-muted-foreground">
|
||||
Configure a metadata-generation model in application setup to enable description generation.
|
||||
</p>
|
||||
) : null}
|
||||
<div
|
||||
id="metadata-generation-source-data-disclosure"
|
||||
role="note"
|
||||
aria-label="Metadata generation source data disclosure"
|
||||
className="space-y-1 text-xs font-normal leading-4 text-muted-foreground"
|
||||
>
|
||||
<p>
|
||||
Description generation may send up to five real source rows and up to five representative
|
||||
non-null example values to the selected model provider.
|
||||
</p>
|
||||
<p>
|
||||
Samples are transient and are not stored in run logs or catalog metadata. Automated
|
||||
Sensitive Data Policy filtering and anonymization are not currently provided; they are
|
||||
planned for future work.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -11,4 +11,6 @@ export const server = setupServer(
|
||||
})),
|
||||
http.get("/api/health/dwh", () => HttpResponse.json({ ok: true, detail: "ok" })),
|
||||
http.get("/api/catalog/databases", () => HttpResponse.json([])),
|
||||
http.get("/api/catalog/metadata-generation/models", () => HttpResponse.json({ models: [], default: null })),
|
||||
http.get("/api/catalog/description-generation-runs", () => HttpResponse.json([])),
|
||||
);
|
||||
|
||||
@@ -17,6 +17,7 @@ dependencies = [
|
||||
"tqdm>=4.66",
|
||||
"yake>=0.4",
|
||||
"portalocker>=2.10",
|
||||
"litellm>=1.98,<2",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
|
||||
@@ -0,0 +1,421 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from tht.internal.litellm_completion import handle_request
|
||||
|
||||
HARNESS_ROOT = Path(__file__).resolve().parents[1]
|
||||
HELPER_MODULE = "tht.internal.litellm_completion"
|
||||
|
||||
|
||||
def _run_raw_helper(
|
||||
tmp_path: Path, raw_request: str, fake_litellm: str
|
||||
) -> subprocess.CompletedProcess[str]:
|
||||
provider_path = tmp_path / "provider"
|
||||
provider_path.mkdir()
|
||||
(provider_path / "litellm.py").write_text(fake_litellm, encoding="utf-8")
|
||||
|
||||
env = os.environ.copy()
|
||||
env["PYTHONPATH"] = os.pathsep.join((str(provider_path), str(HARNESS_ROOT)))
|
||||
return subprocess.run(
|
||||
[sys.executable, "-m", HELPER_MODULE],
|
||||
input=raw_request,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
cwd=HARNESS_ROOT,
|
||||
env=env,
|
||||
timeout=10,
|
||||
check=False,
|
||||
)
|
||||
|
||||
|
||||
def _run_helper(tmp_path: Path, request: object, fake_litellm: str) -> subprocess.CompletedProcess[str]:
|
||||
return _run_raw_helper(tmp_path, json.dumps(request), fake_litellm)
|
||||
|
||||
|
||||
def _valid_request() -> dict[str, object]:
|
||||
return {
|
||||
"model": "openai/gpt-4.1-mini",
|
||||
"api_key": "sk-secret",
|
||||
"messages": [{"role": "user", "content": "private prompt"}],
|
||||
}
|
||||
|
||||
|
||||
def test_subprocess_emits_one_pristine_success_line_and_silences_provider_output(tmp_path: Path):
|
||||
secret = "sk-contract-secret"
|
||||
prompt = "private patient prompt"
|
||||
request = {
|
||||
"model": "openai/gpt-4.1-mini",
|
||||
"api_key": secret,
|
||||
"messages": [
|
||||
{"role": "system", "content": "Return one description."},
|
||||
{"role": "user", "content": prompt},
|
||||
],
|
||||
"api_base": "https://models.example.test/v1",
|
||||
"api_version": "2026-08-01-preview",
|
||||
}
|
||||
fake_litellm = f'''
|
||||
import sys
|
||||
|
||||
suppress_debug_info = False
|
||||
set_verbose = True
|
||||
turn_off_message_logging = False
|
||||
log_raw_request_response = True
|
||||
|
||||
print("provider import noise")
|
||||
print("provider import diagnostic", file=sys.stderr)
|
||||
|
||||
def completion(**kwargs):
|
||||
assert suppress_debug_info is True
|
||||
assert set_verbose is False
|
||||
assert turn_off_message_logging is True
|
||||
assert log_raw_request_response is False
|
||||
assert kwargs == {{
|
||||
"model": "openai/gpt-4.1-mini",
|
||||
"api_key": {secret!r},
|
||||
"messages": [
|
||||
{{"role": "system", "content": "Return one description."}},
|
||||
{{"role": "user", "content": {prompt!r}}},
|
||||
],
|
||||
"api_base": "https://models.example.test/v1",
|
||||
"api_version": "2026-08-01-preview",
|
||||
"num_retries": 1,
|
||||
"stream": False,
|
||||
}}
|
||||
print(repr(kwargs))
|
||||
print(repr(kwargs), file=sys.stderr)
|
||||
return {{"choices": [{{"message": {{"content": "Descrizione italiana"}}}}]}}
|
||||
'''
|
||||
|
||||
result = _run_helper(tmp_path, request, fake_litellm)
|
||||
|
||||
assert result.returncode == 0
|
||||
assert result.stdout == '{"ok":true,"content":"Descrizione italiana"}\n'
|
||||
assert result.stderr == ""
|
||||
assert secret not in result.stdout + result.stderr
|
||||
assert prompt not in result.stdout + result.stderr
|
||||
|
||||
|
||||
def test_subprocess_rejects_an_unknown_request_key_before_loading_provider(tmp_path: Path):
|
||||
request = {**_valid_request(), "unexpected": True}
|
||||
|
||||
result = _run_helper(tmp_path, request, 'raise AssertionError("provider was loaded")')
|
||||
|
||||
assert result.returncode == 0
|
||||
assert result.stdout == '{"ok":false,"error":"invalid_request"}\n'
|
||||
assert result.stderr == "completion helper: invalid request\n"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"raw_request",
|
||||
[
|
||||
"{not-json",
|
||||
"[]",
|
||||
json.dumps({"model": "openai/gpt-4.1-mini", "api_key": "sk-secret"}),
|
||||
json.dumps({**_valid_request(), "model": "bad model"}),
|
||||
json.dumps({**_valid_request(), "model": "a" * 257}),
|
||||
json.dumps({**_valid_request(), "api_key": " "}),
|
||||
json.dumps({**_valid_request(), "api_key": "x" * (16 * 1024 + 1)}),
|
||||
json.dumps(
|
||||
{
|
||||
**_valid_request(),
|
||||
"messages": [{"role": "assistant", "content": "not allowed"}],
|
||||
}
|
||||
),
|
||||
json.dumps(
|
||||
{
|
||||
**_valid_request(),
|
||||
"messages": [{"role": "user", "content": "ok", "name": "extra"}],
|
||||
}
|
||||
),
|
||||
json.dumps({**_valid_request(), "messages": []}),
|
||||
json.dumps(
|
||||
{
|
||||
**_valid_request(),
|
||||
"messages": [{"role": "user", "content": "ok"}] * 33,
|
||||
}
|
||||
),
|
||||
json.dumps(
|
||||
{
|
||||
**_valid_request(),
|
||||
"messages": [{"role": "user"}],
|
||||
}
|
||||
),
|
||||
json.dumps(
|
||||
{
|
||||
**_valid_request(),
|
||||
"messages": [{"role": "user", "content": 1}],
|
||||
}
|
||||
),
|
||||
json.dumps(
|
||||
{
|
||||
**_valid_request(),
|
||||
"messages": [{"role": "user", "content": "x" * (64 * 1024 + 1)}],
|
||||
}
|
||||
),
|
||||
json.dumps(
|
||||
{
|
||||
**_valid_request(),
|
||||
"messages": [
|
||||
{"role": "user", "content": "x" * (48 * 1024)},
|
||||
{"role": "system", "content": "y" * (48 * 1024)},
|
||||
{"role": "user", "content": "z" * (48 * 1024)},
|
||||
],
|
||||
}
|
||||
),
|
||||
json.dumps({**_valid_request(), "api_base": "file:///tmp/provider"}),
|
||||
json.dumps({**_valid_request(), "api_base": "https://user@example.test/v1"}),
|
||||
json.dumps({**_valid_request(), "api_base": "https://example.test/v1?key=value"}),
|
||||
json.dumps({**_valid_request(), "api_version": "2026 preview"}),
|
||||
json.dumps({**_valid_request(), "disable_thinking": False}),
|
||||
(
|
||||
'{"model":"openai/first","model":"openai/second","api_key":"sk-secret",'
|
||||
'"messages":[{"role":"user","content":"private prompt"}]}'
|
||||
),
|
||||
(
|
||||
'{"model":"openai/gpt-4.1-mini","api_key":"sk-secret",'
|
||||
'"messages":[{"role":"user","content":"private prompt"}],"number":'
|
||||
+ "1" * 5000
|
||||
+ "}"
|
||||
),
|
||||
json.dumps(
|
||||
{
|
||||
**_valid_request(),
|
||||
"messages": [{"role": "user", "content": "x" * (256 * 1024)}],
|
||||
}
|
||||
),
|
||||
],
|
||||
ids=[
|
||||
"malformed-json",
|
||||
"non-object",
|
||||
"missing-required-key",
|
||||
"invalid-model",
|
||||
"long-model",
|
||||
"empty-secret",
|
||||
"long-secret",
|
||||
"invalid-role",
|
||||
"message-extra-key",
|
||||
"empty-messages",
|
||||
"too-many-messages",
|
||||
"message-missing-key",
|
||||
"non-string-content",
|
||||
"long-message",
|
||||
"aggregate-content-too-long",
|
||||
"invalid-url-scheme",
|
||||
"url-credentials",
|
||||
"url-query",
|
||||
"invalid-api-version",
|
||||
"invalid-disable-thinking",
|
||||
"duplicate-key",
|
||||
"oversized-number",
|
||||
"oversized-stdin",
|
||||
],
|
||||
)
|
||||
def test_subprocess_rejects_malformed_or_oversized_requests_without_loading_provider(
|
||||
tmp_path: Path, raw_request: str
|
||||
):
|
||||
result = _run_raw_helper(
|
||||
tmp_path,
|
||||
raw_request,
|
||||
'raise AssertionError("provider was loaded with private prompt and sk-secret")',
|
||||
)
|
||||
|
||||
assert result.returncode == 0
|
||||
assert result.stdout == '{"ok":false,"error":"invalid_request"}\n'
|
||||
assert result.stderr == "completion helper: invalid request\n"
|
||||
assert "private prompt" not in result.stdout + result.stderr
|
||||
assert "sk-secret" not in result.stdout + result.stderr
|
||||
|
||||
|
||||
def test_subprocess_normalizes_provider_failure_without_leaking_provider_output(tmp_path: Path):
|
||||
secret = "sk-provider-canary"
|
||||
prompt = "provider prompt canary"
|
||||
request = {
|
||||
**_valid_request(),
|
||||
"api_key": secret,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
}
|
||||
fake_litellm = '''
|
||||
import os
|
||||
import sys
|
||||
|
||||
def completion(**kwargs):
|
||||
leaked = repr(kwargs).encode("utf-8")
|
||||
print(repr(kwargs))
|
||||
print(repr(kwargs), file=sys.stderr)
|
||||
os.write(1, leaked + b"\\n")
|
||||
os.write(2, leaked + b"\\n")
|
||||
os.write(1, b"x" * (1024 * 1024))
|
||||
os.write(2, b"x" * (1024 * 1024))
|
||||
raise RuntimeError(repr(kwargs))
|
||||
'''
|
||||
|
||||
result = _run_helper(tmp_path, request, fake_litellm)
|
||||
|
||||
assert result.returncode == 0
|
||||
assert result.stdout == '{"ok":false,"error":"provider_failure"}\n'
|
||||
assert result.stderr == "completion helper: provider failure\n"
|
||||
assert len(result.stderr.encode("utf-8")) <= 256
|
||||
assert secret not in result.stdout + result.stderr
|
||||
assert prompt not in result.stdout + result.stderr
|
||||
|
||||
|
||||
def test_subprocess_normalizes_an_invalid_response_to_one_safe_line(tmp_path: Path):
|
||||
secret = "sk-invalid-response"
|
||||
prompt = "invalid response prompt"
|
||||
request = {
|
||||
**_valid_request(),
|
||||
"api_key": secret,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
}
|
||||
fake_litellm = '''
|
||||
import os
|
||||
|
||||
def completion(**kwargs):
|
||||
os.write(1, repr(kwargs).encode("utf-8"))
|
||||
os.write(2, repr(kwargs).encode("utf-8"))
|
||||
return {"choices": [{"message": {"content": ""}}]}
|
||||
'''
|
||||
|
||||
result = _run_helper(tmp_path, request, fake_litellm)
|
||||
|
||||
assert result.returncode == 0
|
||||
assert result.stdout == '{"ok":false,"error":"invalid_response"}\n'
|
||||
assert result.stderr == "completion helper: invalid response\n"
|
||||
assert secret not in result.stdout + result.stderr
|
||||
assert prompt not in result.stdout + result.stderr
|
||||
|
||||
|
||||
def test_subprocess_escapes_multiline_content_without_adding_output_frames(tmp_path: Path):
|
||||
content = 'Prima riga\nSeconda "riga" — fine'
|
||||
fake_litellm = f'''
|
||||
def completion(**kwargs):
|
||||
return {{"choices": [{{"message": {{"content": {content!r}}}}}]}}
|
||||
'''
|
||||
|
||||
result = _run_helper(tmp_path, _valid_request(), fake_litellm)
|
||||
|
||||
assert result.returncode == 0
|
||||
assert result.stdout == json.dumps(
|
||||
{"ok": True, "content": content}, ensure_ascii=False, separators=(",", ":")
|
||||
) + "\n"
|
||||
assert result.stdout.count("\n") == 1
|
||||
assert result.stderr == ""
|
||||
|
||||
|
||||
def test_injected_completion_configures_one_retry_without_fallback_or_secret_export():
|
||||
secret = "sk-injected-canary"
|
||||
request = {
|
||||
**_valid_request(),
|
||||
"api_key": secret,
|
||||
"messages": [
|
||||
{"role": "system", "content": "System instruction"},
|
||||
{"role": "user", "content": "Private prompt"},
|
||||
],
|
||||
}
|
||||
calls: list[dict[str, object]] = []
|
||||
|
||||
def completion(**kwargs: object) -> object:
|
||||
assert secret not in os.environ.values()
|
||||
calls.append(kwargs)
|
||||
return SimpleNamespace(
|
||||
choices=[SimpleNamespace(message=SimpleNamespace(content="Validated content"))]
|
||||
)
|
||||
|
||||
result = handle_request(request, completion=completion)
|
||||
|
||||
assert result == {"ok": True, "content": "Validated content"}
|
||||
assert calls == [{
|
||||
"model": "openai/gpt-4.1-mini",
|
||||
"api_key": secret,
|
||||
"messages": [
|
||||
{"role": "system", "content": "System instruction"},
|
||||
{"role": "user", "content": "Private prompt"},
|
||||
],
|
||||
"num_retries": 1,
|
||||
"stream": False,
|
||||
}]
|
||||
|
||||
|
||||
def test_injected_completion_uses_non_secret_client_placeholder_for_keyless_endpoint():
|
||||
calls: list[dict[str, object]] = []
|
||||
|
||||
def completion(**kwargs: object) -> object:
|
||||
calls.append(kwargs)
|
||||
return {"choices": [{"message": {"content": "Descrizione Qwen"}}]}
|
||||
|
||||
request = {
|
||||
"model": "openai/qwen3.6-35b-a3b",
|
||||
"messages": [{"role": "user", "content": "Invented metadata"}],
|
||||
"api_base": "https://models.internal.example/v1",
|
||||
"disable_thinking": True,
|
||||
}
|
||||
|
||||
assert handle_request(request, completion=completion) == {
|
||||
"ok": True,
|
||||
"content": "Descrizione Qwen",
|
||||
}
|
||||
assert calls == [{
|
||||
"model": "openai/qwen3.6-35b-a3b",
|
||||
"api_key": "not-required",
|
||||
"messages": [{"role": "user", "content": "Invented metadata"}],
|
||||
"num_retries": 1,
|
||||
"stream": False,
|
||||
"api_base": "https://models.internal.example/v1",
|
||||
"extra_body": {"chat_template_kwargs": {"enable_thinking": False}},
|
||||
}]
|
||||
|
||||
|
||||
def test_injected_provider_failure_is_not_retried():
|
||||
attempts = 0
|
||||
|
||||
def completion(**kwargs: object) -> object:
|
||||
nonlocal attempts
|
||||
attempts += 1
|
||||
raise RuntimeError(repr(kwargs))
|
||||
|
||||
result = handle_request(_valid_request(), completion=completion)
|
||||
|
||||
assert result == {"ok": False, "error": "provider_failure"}
|
||||
assert attempts == 1
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"response",
|
||||
[
|
||||
None,
|
||||
{},
|
||||
{"choices": []},
|
||||
{
|
||||
"choices": [
|
||||
{"message": {"content": "first"}},
|
||||
{"message": {"content": "second"}},
|
||||
]
|
||||
},
|
||||
{"choices": [{"message": {}}]},
|
||||
{"choices": [{"message": {"content": None}}]},
|
||||
{"choices": [{"message": {"content": " "}}]},
|
||||
{"choices": [{"message": {"content": "x" * (64 * 1024 + 1)}}]},
|
||||
],
|
||||
ids=[
|
||||
"none",
|
||||
"missing-choices",
|
||||
"empty-choices",
|
||||
"ambiguous-choices",
|
||||
"missing-content",
|
||||
"non-string-content",
|
||||
"empty-content",
|
||||
"oversized-content",
|
||||
],
|
||||
)
|
||||
def test_injected_completion_rejects_invalid_response_shapes(response: object):
|
||||
result = handle_request(_valid_request(), completion=lambda **_: response)
|
||||
|
||||
assert result == {"ok": False, "error": "invalid_response"}
|
||||
@@ -0,0 +1,9 @@
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def test_runtime_dependencies_constrain_litellm_to_the_supported_major():
|
||||
pyproject = Path(__file__).resolve().parents[1] / "pyproject.toml"
|
||||
project = tomllib.loads(pyproject.read_text(encoding="utf-8"))["project"]
|
||||
|
||||
assert "litellm>=1.98,<2" in project["dependencies"]
|
||||
@@ -0,0 +1 @@
|
||||
"""Internal process adapters that are not part of the ``tht`` CLI surface."""
|
||||
@@ -0,0 +1,316 @@
|
||||
"""One-shot stdin/stdout adapter around LiteLLM chat completion."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from collections.abc import Callable, Iterator, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
MAX_STDIN_BYTES = 256 * 1024
|
||||
MAX_API_KEY_BYTES = 16 * 1024
|
||||
MAX_MESSAGES = 32
|
||||
MAX_MESSAGE_CONTENT_BYTES = 64 * 1024
|
||||
MAX_TOTAL_CONTENT_BYTES = 128 * 1024
|
||||
MAX_RESPONSE_CONTENT_BYTES = 64 * 1024
|
||||
MAX_STDERR_BYTES = 256
|
||||
KEYLESS_API_KEY_PLACEHOLDER = "not-required"
|
||||
|
||||
_REQUIRED_KEYS = frozenset({"model", "messages"})
|
||||
_OPTIONAL_KEYS = frozenset({"api_key", "api_base", "api_version", "disable_thinking"})
|
||||
_MESSAGE_KEYS = frozenset({"role", "content"})
|
||||
_MODEL_PATTERN = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}\Z")
|
||||
_API_VERSION_PATTERN = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}\Z")
|
||||
_DIAGNOSTICS = {
|
||||
"invalid_request": "completion helper: invalid request\n",
|
||||
"provider_failure": "completion helper: provider failure\n",
|
||||
"invalid_response": "completion helper: invalid response\n",
|
||||
}
|
||||
|
||||
Completion = Callable[..., object]
|
||||
Result = dict[str, object]
|
||||
|
||||
|
||||
class _InvalidRequest(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class _InvalidResponse(Exception):
|
||||
pass
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True, repr=False)
|
||||
class _ValidatedRequest:
|
||||
model: str
|
||||
api_key: str | None
|
||||
messages: tuple[tuple[str, str], ...]
|
||||
api_base: str | None
|
||||
api_version: str | None
|
||||
disable_thinking: bool
|
||||
|
||||
|
||||
def _utf8_size(value: str, error: type[Exception]) -> int:
|
||||
try:
|
||||
return len(value.encode("utf-8"))
|
||||
except UnicodeEncodeError as exc:
|
||||
raise error from exc
|
||||
|
||||
|
||||
def _object_without_duplicates(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
|
||||
value: dict[str, Any] = {}
|
||||
for key, item in pairs:
|
||||
if key in value:
|
||||
raise _InvalidRequest
|
||||
value[key] = item
|
||||
return value
|
||||
|
||||
|
||||
def _reject_nonstandard_number(_: str) -> None:
|
||||
raise _InvalidRequest
|
||||
|
||||
|
||||
def _read_json_request() -> object:
|
||||
raw = sys.stdin.buffer.read(MAX_STDIN_BYTES + 1)
|
||||
if not raw or len(raw) > MAX_STDIN_BYTES:
|
||||
raise _InvalidRequest
|
||||
try:
|
||||
return json.loads(
|
||||
raw.decode("utf-8"),
|
||||
object_pairs_hook=_object_without_duplicates,
|
||||
parse_constant=_reject_nonstandard_number,
|
||||
)
|
||||
except (UnicodeError, ValueError, RecursionError, _InvalidRequest) as exc:
|
||||
raise _InvalidRequest from exc
|
||||
|
||||
|
||||
def _validate_api_base(value: object) -> str:
|
||||
if type(value) is not str or not value or _utf8_size(value, _InvalidRequest) > 2048:
|
||||
raise _InvalidRequest
|
||||
if any(character.isspace() or ord(character) < 32 for character in value):
|
||||
raise _InvalidRequest
|
||||
try:
|
||||
parsed = urlsplit(value)
|
||||
port = parsed.port
|
||||
except ValueError as exc:
|
||||
raise _InvalidRequest from exc
|
||||
if (
|
||||
parsed.scheme.lower() not in {"http", "https"}
|
||||
or not parsed.hostname
|
||||
or parsed.username is not None
|
||||
or parsed.password is not None
|
||||
or parsed.query
|
||||
or parsed.fragment
|
||||
or (port is not None and not 1 <= port <= 65535)
|
||||
):
|
||||
raise _InvalidRequest
|
||||
return value
|
||||
|
||||
|
||||
def _validate_request(value: object) -> _ValidatedRequest:
|
||||
if type(value) is not dict:
|
||||
raise _InvalidRequest
|
||||
keys = set(value)
|
||||
if not _REQUIRED_KEYS.issubset(keys) or not keys.issubset(_REQUIRED_KEYS | _OPTIONAL_KEYS):
|
||||
raise _InvalidRequest
|
||||
|
||||
model = value["model"]
|
||||
if type(model) is not str or _MODEL_PATTERN.fullmatch(model) is None:
|
||||
raise _InvalidRequest
|
||||
|
||||
api_key = None
|
||||
if "api_key" in value:
|
||||
candidate = value["api_key"]
|
||||
if (
|
||||
type(candidate) is not str
|
||||
or not candidate
|
||||
or _utf8_size(candidate, _InvalidRequest) > MAX_API_KEY_BYTES
|
||||
or any(character.isspace() or ord(character) < 32 for character in candidate)
|
||||
):
|
||||
raise _InvalidRequest
|
||||
api_key = candidate
|
||||
|
||||
raw_messages = value["messages"]
|
||||
if type(raw_messages) is not list or not 1 <= len(raw_messages) <= MAX_MESSAGES:
|
||||
raise _InvalidRequest
|
||||
messages: list[tuple[str, str]] = []
|
||||
total_content_bytes = 0
|
||||
for raw_message in raw_messages:
|
||||
if type(raw_message) is not dict or set(raw_message) != _MESSAGE_KEYS:
|
||||
raise _InvalidRequest
|
||||
role = raw_message["role"]
|
||||
content = raw_message["content"]
|
||||
if type(role) is not str or role not in {"system", "user"}:
|
||||
raise _InvalidRequest
|
||||
if type(content) is not str or not content.strip():
|
||||
raise _InvalidRequest
|
||||
content_bytes = _utf8_size(content, _InvalidRequest)
|
||||
if content_bytes > MAX_MESSAGE_CONTENT_BYTES:
|
||||
raise _InvalidRequest
|
||||
total_content_bytes += content_bytes
|
||||
if total_content_bytes > MAX_TOTAL_CONTENT_BYTES:
|
||||
raise _InvalidRequest
|
||||
messages.append((role, content))
|
||||
|
||||
api_base = None
|
||||
if "api_base" in value:
|
||||
api_base = _validate_api_base(value["api_base"])
|
||||
|
||||
api_version = None
|
||||
if "api_version" in value:
|
||||
candidate = value["api_version"]
|
||||
if type(candidate) is not str or _API_VERSION_PATTERN.fullmatch(candidate) is None:
|
||||
raise _InvalidRequest
|
||||
api_version = candidate
|
||||
|
||||
disable_thinking = False
|
||||
if "disable_thinking" in value:
|
||||
if value["disable_thinking"] is not True:
|
||||
raise _InvalidRequest
|
||||
disable_thinking = True
|
||||
|
||||
return _ValidatedRequest(
|
||||
model=model,
|
||||
api_key=api_key,
|
||||
messages=tuple(messages),
|
||||
api_base=api_base,
|
||||
api_version=api_version,
|
||||
disable_thinking=disable_thinking,
|
||||
)
|
||||
|
||||
|
||||
def _litellm_completion(**kwargs: object) -> object:
|
||||
import litellm
|
||||
|
||||
litellm.suppress_debug_info = True
|
||||
litellm.set_verbose = False
|
||||
litellm.turn_off_message_logging = True
|
||||
litellm.log_raw_request_response = False
|
||||
litellm.redact_messages_in_exceptions = True
|
||||
litellm.redact_user_api_key_info = True
|
||||
return litellm.completion(**kwargs)
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _silence_provider_output() -> Iterator[None]:
|
||||
with open(os.devnull, "w", encoding="utf-8") as sink:
|
||||
sys.stdout.flush()
|
||||
sys.stderr.flush()
|
||||
saved_stdout = os.dup(1)
|
||||
saved_stderr = os.dup(2)
|
||||
try:
|
||||
os.dup2(sink.fileno(), 1)
|
||||
os.dup2(sink.fileno(), 2)
|
||||
with contextlib.redirect_stdout(sink), contextlib.redirect_stderr(sink):
|
||||
yield
|
||||
finally:
|
||||
os.dup2(saved_stdout, 1)
|
||||
os.dup2(saved_stderr, 2)
|
||||
os.close(saved_stdout)
|
||||
os.close(saved_stderr)
|
||||
|
||||
|
||||
def _provider_kwargs(request: _ValidatedRequest) -> dict[str, object]:
|
||||
kwargs: dict[str, object] = {
|
||||
"model": request.model,
|
||||
# OpenAI-compatible SDKs require a non-empty client value even when the
|
||||
# explicitly configured endpoint does not authenticate requests.
|
||||
"api_key": request.api_key or KEYLESS_API_KEY_PLACEHOLDER,
|
||||
"messages": [
|
||||
{"role": role, "content": content} for role, content in request.messages
|
||||
],
|
||||
"num_retries": 1,
|
||||
"stream": False,
|
||||
}
|
||||
if request.api_base is not None:
|
||||
kwargs["api_base"] = request.api_base
|
||||
if request.api_version is not None:
|
||||
kwargs["api_version"] = request.api_version
|
||||
if request.disable_thinking:
|
||||
kwargs["extra_body"] = {"chat_template_kwargs": {"enable_thinking": False}}
|
||||
return kwargs
|
||||
|
||||
|
||||
def _field(value: object, name: str) -> object:
|
||||
if isinstance(value, Mapping):
|
||||
if name not in value:
|
||||
raise _InvalidResponse
|
||||
return value[name]
|
||||
try:
|
||||
return getattr(value, name)
|
||||
except (AttributeError, TypeError) as exc:
|
||||
raise _InvalidResponse from exc
|
||||
|
||||
|
||||
def _response_content(response: object) -> str:
|
||||
choices = _field(response, "choices")
|
||||
if (
|
||||
not isinstance(choices, Sequence)
|
||||
or isinstance(choices, (str, bytes, bytearray))
|
||||
or len(choices) != 1
|
||||
):
|
||||
raise _InvalidResponse
|
||||
message = _field(choices[0], "message")
|
||||
content = _field(message, "content")
|
||||
if (
|
||||
type(content) is not str
|
||||
or not content.strip()
|
||||
or _utf8_size(content, _InvalidResponse) > MAX_RESPONSE_CONTENT_BYTES
|
||||
):
|
||||
raise _InvalidResponse
|
||||
return content
|
||||
|
||||
|
||||
def handle_request(request: object, *, completion: Completion | None = None) -> Result:
|
||||
"""Validate one request and normalize one injected or LiteLLM completion."""
|
||||
try:
|
||||
validated = _validate_request(request)
|
||||
except _InvalidRequest:
|
||||
return {"ok": False, "error": "invalid_request"}
|
||||
|
||||
provider = completion or _litellm_completion
|
||||
try:
|
||||
with _silence_provider_output():
|
||||
response = provider(**_provider_kwargs(validated))
|
||||
except Exception: # noqa: BLE001 - provider failures cross a redacted process boundary
|
||||
return {"ok": False, "error": "provider_failure"}
|
||||
|
||||
try:
|
||||
with _silence_provider_output():
|
||||
content = _response_content(response)
|
||||
except Exception: # noqa: BLE001 - arbitrary provider objects are untrusted response data
|
||||
return {"ok": False, "error": "invalid_response"}
|
||||
return {"ok": True, "content": content}
|
||||
|
||||
|
||||
def _write_json(payload: Mapping[str, object]) -> None:
|
||||
sys.stdout.write(json.dumps(payload, ensure_ascii=False, separators=(",", ":")) + "\n")
|
||||
|
||||
|
||||
def _write_diagnostic(result: Mapping[str, object]) -> None:
|
||||
error = result.get("error")
|
||||
if not isinstance(error, str):
|
||||
return
|
||||
diagnostic = _DIAGNOSTICS.get(error, "completion helper: failure\n")
|
||||
encoded = diagnostic.encode("utf-8")[:MAX_STDERR_BYTES]
|
||||
sys.stderr.write(encoded.decode("utf-8", errors="ignore"))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
try:
|
||||
request = _read_json_request()
|
||||
except _InvalidRequest:
|
||||
result: Result = {"ok": False, "error": "invalid_request"}
|
||||
else:
|
||||
result = handle_request(request)
|
||||
_write_json(result)
|
||||
if result.get("ok") is False:
|
||||
_write_diagnostic(result)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Generated
+2365
File diff suppressed because it is too large
Load Diff
@@ -83,7 +83,7 @@ architecture = (root / "docs/architecture/authentication.md").read_text()
|
||||
user_row = "| `user` | `session.use` |"
|
||||
admin_row = (
|
||||
"| `admin` | `session.use`, `session.read_all`, `session.manage_all`, `settings.manage`, "
|
||||
"`workspace.manage`, `workspace.secrets.manage`, `pi.manage`, `auth.diagnostics.read` |"
|
||||
"`workspace.manage`, `workspace.secrets.manage`, `database.manage`, `pi.manage`, `auth.diagnostics.read` |"
|
||||
)
|
||||
if user_row not in architecture or admin_row not in architecture:
|
||||
raise SystemExit("auth docs smoke: role-to-permission map is not exact")
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"sort"
|
||||
"strings"
|
||||
"sync"
|
||||
"unicode"
|
||||
|
||||
"github.com/aritmolab/thothii/tools/tht/internal/safeio"
|
||||
"github.com/compose-spec/compose-go/v2/dotenv"
|
||||
@@ -25,6 +26,10 @@ const installationFileName = "thothii-installation.yaml"
|
||||
|
||||
const maxEnvironmentFileBytes = 1 << 20
|
||||
|
||||
const maxMetadataSecretBundleBytes = 64 << 10
|
||||
|
||||
const maxMetadataGenerationModels = 64
|
||||
|
||||
const maxSecretSources = 32
|
||||
|
||||
var dotenvParseMu sync.Mutex
|
||||
@@ -35,9 +40,34 @@ type descriptor struct {
|
||||
EnvFile string `yaml:"envFile"`
|
||||
WorkspaceRepository workspaceRepositoryDescriptor `yaml:"workspaceRepository"`
|
||||
Authentication authenticationDescriptor `yaml:"authentication"`
|
||||
MetadataGeneration metadataGenerationDescriptor `yaml:"metadataGeneration"`
|
||||
Overrides []string `yaml:"overrides"`
|
||||
}
|
||||
|
||||
type metadataGenerationDescriptor struct {
|
||||
Default string `yaml:"default"`
|
||||
Models []metadataGenerationModelDescriptor `yaml:"models"`
|
||||
}
|
||||
|
||||
type metadataGenerationModelDescriptor struct {
|
||||
ID string `yaml:"id"`
|
||||
Label string `yaml:"label"`
|
||||
LiteLLM liteLLMDescriptor `yaml:"litellm"`
|
||||
APIKeyEnv string `yaml:"apiKeyEnv"`
|
||||
}
|
||||
|
||||
type liteLLMDescriptor struct {
|
||||
Provider string `yaml:"provider"`
|
||||
Model string `yaml:"model"`
|
||||
DisableThinking bool `yaml:"disableThinking"`
|
||||
Endpoint *metadataEndpointDescriptor `yaml:"endpoint"`
|
||||
}
|
||||
|
||||
type metadataEndpointDescriptor struct {
|
||||
BaseURL string `yaml:"baseUrl"`
|
||||
APIVersion string `yaml:"apiVersion"`
|
||||
}
|
||||
|
||||
type authenticationDescriptor struct {
|
||||
ConfigDirectory string `yaml:"configDirectory"`
|
||||
RuntimeProjection *runtimeProjectionDescriptor `yaml:"runtimeProjection"`
|
||||
@@ -75,6 +105,36 @@ type Authentication struct {
|
||||
RuntimeProjection *RuntimeProjection
|
||||
}
|
||||
|
||||
// MetadataGenerationEndpoint contains optional provider endpoint settings for one LiteLLM model.
|
||||
type MetadataGenerationEndpoint struct {
|
||||
BaseURL string
|
||||
APIVersion string
|
||||
}
|
||||
|
||||
// MetadataGenerationLiteLLM identifies the provider/model pair used by the internal completion helper.
|
||||
type MetadataGenerationLiteLLM struct {
|
||||
Provider string
|
||||
Model string
|
||||
DisableThinking bool
|
||||
Endpoint *MetadataGenerationEndpoint
|
||||
}
|
||||
|
||||
// MetadataGenerationModel is one selectable installation-owned metadata-generation model.
|
||||
// APIKeyEnv is an optional reference only; credential values never enter Installation.
|
||||
// An empty value is allowed only for a model with an explicit keyless endpoint.
|
||||
type MetadataGenerationModel struct {
|
||||
ID string
|
||||
Label string
|
||||
LiteLLM MetadataGenerationLiteLLM
|
||||
APIKeyEnv string
|
||||
}
|
||||
|
||||
// MetadataGeneration is the installation-owned model list and its default selection.
|
||||
type MetadataGeneration struct {
|
||||
Default string
|
||||
Models []MetadataGenerationModel
|
||||
}
|
||||
|
||||
// Installation is a validated local Compose installation. It intentionally contains paths, not
|
||||
// environment values or secret content.
|
||||
type Installation struct {
|
||||
@@ -84,6 +144,7 @@ type Installation struct {
|
||||
EnvFile string
|
||||
WorkspaceRepository WorkspaceRepository
|
||||
Authentication Authentication
|
||||
MetadataGeneration MetadataGeneration
|
||||
Overrides []string
|
||||
}
|
||||
|
||||
@@ -137,6 +198,29 @@ func Load(path string) (Installation, error) {
|
||||
GID: raw.Authentication.RuntimeProjection.GID,
|
||||
}
|
||||
}
|
||||
metadataGeneration := MetadataGeneration{
|
||||
Default: raw.MetadataGeneration.Default,
|
||||
Models: make([]MetadataGenerationModel, 0, len(raw.MetadataGeneration.Models)),
|
||||
}
|
||||
for _, rawModel := range raw.MetadataGeneration.Models {
|
||||
model := MetadataGenerationModel{
|
||||
ID: rawModel.ID,
|
||||
Label: rawModel.Label,
|
||||
LiteLLM: MetadataGenerationLiteLLM{
|
||||
Provider: rawModel.LiteLLM.Provider,
|
||||
Model: rawModel.LiteLLM.Model,
|
||||
DisableThinking: rawModel.LiteLLM.DisableThinking,
|
||||
},
|
||||
APIKeyEnv: rawModel.APIKeyEnv,
|
||||
}
|
||||
if rawModel.LiteLLM.Endpoint != nil {
|
||||
model.LiteLLM.Endpoint = &MetadataGenerationEndpoint{
|
||||
BaseURL: rawModel.LiteLLM.Endpoint.BaseURL,
|
||||
APIVersion: rawModel.LiteLLM.Endpoint.APIVersion,
|
||||
}
|
||||
}
|
||||
metadataGeneration.Models = append(metadataGeneration.Models, model)
|
||||
}
|
||||
installation := Installation{
|
||||
Path: path,
|
||||
Profile: raw.Profile,
|
||||
@@ -147,13 +231,17 @@ func Load(path string) (Installation, error) {
|
||||
Branch: raw.WorkspaceRepository.Branch,
|
||||
Access: raw.WorkspaceRepository.Access,
|
||||
},
|
||||
Authentication: authentication,
|
||||
Overrides: make([]string, 0, len(raw.Overrides)),
|
||||
Authentication: authentication,
|
||||
MetadataGeneration: metadataGeneration,
|
||||
Overrides: make([]string, 0, len(raw.Overrides)),
|
||||
}
|
||||
values, err := installation.environmentValues()
|
||||
if err != nil {
|
||||
return Installation{}, errors.New("installation secret declarations could not be read")
|
||||
}
|
||||
if err := installation.validateMetadataGeneration(values); err != nil {
|
||||
return Installation{}, err
|
||||
}
|
||||
if values["THT_AUTH_CONFIG_ROOT"] != installation.AuthenticationDirectory() {
|
||||
return Installation{}, errors.New("authentication.configDirectory must match THT_AUTH_CONFIG_ROOT")
|
||||
}
|
||||
@@ -189,6 +277,198 @@ func Load(path string) (Installation, error) {
|
||||
return installation, nil
|
||||
}
|
||||
|
||||
func (i Installation) validateMetadataGeneration(values map[string]string) error {
|
||||
if len(i.MetadataGeneration.Models) > maxMetadataGenerationModels {
|
||||
return fmt.Errorf(
|
||||
"metadataGeneration.models must contain at most %d entries",
|
||||
maxMetadataGenerationModels,
|
||||
)
|
||||
}
|
||||
seen := make(map[string]struct{}, len(i.MetadataGeneration.Models))
|
||||
for index, model := range i.MetadataGeneration.Models {
|
||||
if err := validateMetadataGenerationModel(index, model); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, exists := seen[model.ID]; exists {
|
||||
return fmt.Errorf("duplicate metadataGeneration model id %q", model.ID)
|
||||
}
|
||||
seen[model.ID] = struct{}{}
|
||||
}
|
||||
if len(i.MetadataGeneration.Models) > 0 && i.MetadataGeneration.Default == "" {
|
||||
return errors.New("metadataGeneration.default is required when models are configured")
|
||||
}
|
||||
if i.MetadataGeneration.Default != "" {
|
||||
if _, exists := seen[i.MetadataGeneration.Default]; !exists {
|
||||
return fmt.Errorf(
|
||||
"metadataGeneration.default %q does not identify a configured model",
|
||||
i.MetadataGeneration.Default,
|
||||
)
|
||||
}
|
||||
}
|
||||
if len(i.MetadataGeneration.Models) == 0 {
|
||||
return nil
|
||||
}
|
||||
if values["THT_INSTALLATION_CONFIG_SOURCE"] != i.Path {
|
||||
return errors.New("metadataGeneration requires THT_INSTALLATION_CONFIG_SOURCE to match the installation file")
|
||||
}
|
||||
requiresSecrets := false
|
||||
for _, model := range i.MetadataGeneration.Models {
|
||||
if model.APIKeyEnv != "" {
|
||||
requiresSecrets = true
|
||||
break
|
||||
}
|
||||
}
|
||||
secrets := map[string]string{}
|
||||
if requiresSecrets {
|
||||
bundlePath := values["THT_SECRETS_FILE"]
|
||||
if bundlePath == "" {
|
||||
return errors.New("metadataGeneration keyed models require THT_SECRETS_FILE")
|
||||
}
|
||||
var err error
|
||||
secrets, err = readMetadataGenerationSecrets(bundlePath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
for _, model := range i.MetadataGeneration.Models {
|
||||
if model.APIKeyEnv == "" {
|
||||
continue
|
||||
}
|
||||
value, exists := secrets[model.APIKeyEnv]
|
||||
if !exists {
|
||||
return fmt.Errorf(
|
||||
"metadataGeneration model %q secret %q is missing from THT_SECRETS_FILE",
|
||||
model.ID,
|
||||
model.APIKeyEnv,
|
||||
)
|
||||
}
|
||||
if !usableMetadataGenerationSecret(value) {
|
||||
return fmt.Errorf(
|
||||
"metadataGeneration model %q secret %q is unusable",
|
||||
model.ID,
|
||||
model.APIKeyEnv,
|
||||
)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
var metadataModelIDPattern = regexp.MustCompile(`^[a-z][a-z0-9._-]{0,63}$`)
|
||||
var metadataProviderPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`)
|
||||
var metadataProviderModelPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}$`)
|
||||
var metadataAPIVersionPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`)
|
||||
var metadataSecretBundleKeyPattern = regexp.MustCompile(`^[A-Z][A-Z0-9_]{0,127}$`)
|
||||
var metadataAPIKeyEnvironments = map[string]struct{}{
|
||||
"THT_METADATA_API_KEY": {},
|
||||
"ANTHROPIC_API_KEY": {},
|
||||
"AZURE_API_KEY": {},
|
||||
"GEMINI_API_KEY": {},
|
||||
"DEEPSEEK_API_KEY": {},
|
||||
"OPENAI_API_KEY": {},
|
||||
"OPENROUTER_API_KEY": {},
|
||||
"ZAI_API_KEY": {},
|
||||
}
|
||||
var metadataSecretBundleKeys = map[string]struct{}{
|
||||
"THT_MODEL_API_KEY": {},
|
||||
"THT_DWH_API_KEY": {},
|
||||
"THT_VEC_API_KEY": {},
|
||||
"THT_VEC_WRITE_API_KEY": {},
|
||||
"THT_CA": {},
|
||||
"THT_SSL_CA": {},
|
||||
"THT_VECTOR_BOOTSTRAP_PASSWORD": {},
|
||||
"THT_VECTOR_MIGRATOR_PASSWORD": {},
|
||||
"THT_VECTOR_READER_PASSWORD": {},
|
||||
"THT_VECTOR_WRITER_PASSWORD": {},
|
||||
"PI_PROVIDER_API_KEY": {},
|
||||
"THT_OIDC_CLIENT_SECRET": {},
|
||||
"THT_AUTHENTIK_API_TOKEN": {},
|
||||
"THT_METADATA_API_KEY": {},
|
||||
"ANTHROPIC_API_KEY": {},
|
||||
"AZURE_API_KEY": {},
|
||||
"GEMINI_API_KEY": {},
|
||||
"DEEPSEEK_API_KEY": {},
|
||||
"OPENAI_API_KEY": {},
|
||||
"OPENROUTER_API_KEY": {},
|
||||
"ZAI_API_KEY": {},
|
||||
}
|
||||
|
||||
func validateMetadataGenerationModel(index int, model MetadataGenerationModel) error {
|
||||
prefix := fmt.Sprintf("metadataGeneration.models[%d]", index)
|
||||
if !metadataModelIDPattern.MatchString(model.ID) {
|
||||
return fmt.Errorf("%s.id is invalid", prefix)
|
||||
}
|
||||
if len(model.Label) == 0 || len(model.Label) > 128 || strings.TrimSpace(model.Label) != model.Label ||
|
||||
strings.IndexFunc(model.Label, unicode.IsControl) >= 0 {
|
||||
return fmt.Errorf("%s.label is invalid", prefix)
|
||||
}
|
||||
if !metadataProviderPattern.MatchString(model.LiteLLM.Provider) {
|
||||
return fmt.Errorf("%s.litellm.provider is invalid", prefix)
|
||||
}
|
||||
if !metadataProviderModelPattern.MatchString(model.LiteLLM.Model) {
|
||||
return fmt.Errorf("%s.litellm.model is invalid", prefix)
|
||||
}
|
||||
if model.APIKeyEnv == "" {
|
||||
if model.LiteLLM.Endpoint == nil {
|
||||
return fmt.Errorf("%s.apiKeyEnv is required unless an explicit keyless endpoint is configured", prefix)
|
||||
}
|
||||
} else {
|
||||
if !metadataSecretBundleKeyPattern.MatchString(model.APIKeyEnv) {
|
||||
return fmt.Errorf("%s.apiKeyEnv is invalid", prefix)
|
||||
}
|
||||
if _, allowed := metadataAPIKeyEnvironments[model.APIKeyEnv]; !allowed {
|
||||
return fmt.Errorf("%s.apiKeyEnv is invalid", prefix)
|
||||
}
|
||||
}
|
||||
if endpoint := model.LiteLLM.Endpoint; endpoint != nil {
|
||||
parsed, err := url.Parse(endpoint.BaseURL)
|
||||
if err != nil || strings.TrimSpace(endpoint.BaseURL) != endpoint.BaseURL ||
|
||||
(parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Hostname() == "" ||
|
||||
parsed.User != nil || parsed.RawQuery != "" || parsed.Fragment != "" {
|
||||
return fmt.Errorf("%s.litellm.endpoint.baseUrl is invalid", prefix)
|
||||
}
|
||||
if endpoint.APIVersion != "" && !metadataAPIVersionPattern.MatchString(endpoint.APIVersion) {
|
||||
return fmt.Errorf("%s.litellm.endpoint.apiVersion is invalid", prefix)
|
||||
}
|
||||
}
|
||||
if model.LiteLLM.DisableThinking && model.LiteLLM.Endpoint == nil {
|
||||
return fmt.Errorf("%s.litellm.disableThinking requires an explicit endpoint", prefix)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func readMetadataGenerationSecrets(path string) (map[string]string, error) {
|
||||
contents, err := safeio.ReadCanonicalPrivateRegular(path, maxMetadataSecretBundleBytes)
|
||||
if err != nil {
|
||||
return nil, errors.New("metadataGeneration secrets in THT_SECRETS_FILE are unavailable")
|
||||
}
|
||||
values := make(map[string]string)
|
||||
for _, raw := range strings.Split(string(contents), "\n") {
|
||||
line := strings.TrimSuffix(raw, "\r")
|
||||
if strings.TrimSpace(line) == "" || strings.HasPrefix(strings.TrimSpace(line), "#") {
|
||||
continue
|
||||
}
|
||||
key, value, found := strings.Cut(line, "=")
|
||||
if !found || !metadataSecretBundleKeyPattern.MatchString(key) {
|
||||
return nil, errors.New("metadataGeneration secrets in THT_SECRETS_FILE are invalid")
|
||||
}
|
||||
if _, allowed := metadataSecretBundleKeys[key]; !allowed {
|
||||
return nil, errors.New("metadataGeneration secrets in THT_SECRETS_FILE are invalid")
|
||||
}
|
||||
if _, duplicate := values[key]; duplicate {
|
||||
return nil, errors.New("metadataGeneration secrets in THT_SECRETS_FILE contain duplicate keys")
|
||||
}
|
||||
values[key] = value
|
||||
}
|
||||
return values, nil
|
||||
}
|
||||
|
||||
func usableMetadataGenerationSecret(value string) bool {
|
||||
return len(value) > 0 && len(value) <= 16*1024 && strings.TrimSpace(value) == value &&
|
||||
strings.IndexFunc(value, func(character rune) bool {
|
||||
return unicode.IsSpace(character) || unicode.IsControl(character)
|
||||
}) < 0
|
||||
}
|
||||
|
||||
// AuthenticationDirectory returns the descriptor-owned, non-secret authentication root.
|
||||
func (i Installation) AuthenticationDirectory() string { return i.Authentication.ConfigDirectory }
|
||||
|
||||
|
||||
@@ -0,0 +1,454 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestLoadAcceptsMetadataGenerationModels(t *testing.T) {
|
||||
installationPath, _, envFile, _ := writeInstallation(t, "local")
|
||||
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
|
||||
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
appendFile(t, envFile, strings.Join([]string{
|
||||
"THT_SECRETS_FILE=" + strconv.Quote(secretBundle),
|
||||
"THT_INSTALLATION_CONFIG_SOURCE=" + strconv.Quote(installationPath),
|
||||
}, "\n")+"\n")
|
||||
appendFile(t, installationPath, `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm:
|
||||
provider: openai
|
||||
model: gpt-4.1-mini
|
||||
endpoint:
|
||||
baseUrl: https://api.openai.example/v1
|
||||
apiVersion: "2026-08-01"
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`)
|
||||
|
||||
installation, err := Load(installationPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
if installation.MetadataGeneration.Default != "openai-mini" {
|
||||
t.Fatalf("metadata default = %q", installation.MetadataGeneration.Default)
|
||||
}
|
||||
if len(installation.MetadataGeneration.Models) != 1 {
|
||||
t.Fatalf("metadata models = %#v", installation.MetadataGeneration.Models)
|
||||
}
|
||||
model := installation.MetadataGeneration.Models[0]
|
||||
if model.ID != "openai-mini" || model.Label != "OpenAI Mini" ||
|
||||
model.LiteLLM.Provider != "openai" || model.LiteLLM.Model != "gpt-4.1-mini" ||
|
||||
model.LiteLLM.Endpoint == nil || model.LiteLLM.Endpoint.BaseURL != "https://api.openai.example/v1" ||
|
||||
model.LiteLLM.Endpoint.APIVersion != "2026-08-01" || model.APIKeyEnv != "OPENAI_API_KEY" {
|
||||
t.Fatalf("metadata model = %#v", model)
|
||||
}
|
||||
if strings.Contains(strings.TrimSpace(model.APIKeyEnv), "provider-secret") {
|
||||
t.Fatal("installation model exposed the credential value")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadAcceptsMixedKeyedAndKeylessMetadataGenerationModels(t *testing.T) {
|
||||
installationPath, _, envFile, _ := writeInstallation(t, "local")
|
||||
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
|
||||
if err := os.WriteFile(
|
||||
secretBundle,
|
||||
[]byte("DEEPSEEK_API_KEY=deepseek-secret\nZAI_API_KEY=zai-secret\n"),
|
||||
0o600,
|
||||
); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
|
||||
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
|
||||
appendFile(t, installationPath, `metadataGeneration:
|
||||
default: glm-53
|
||||
models:
|
||||
- id: deepseek-v4-pro
|
||||
label: DeepSeek V4 Pro
|
||||
litellm: {provider: deepseek, model: deepseek-v4-pro}
|
||||
apiKeyEnv: DEEPSEEK_API_KEY
|
||||
- id: glm-53
|
||||
label: GLM 5.3
|
||||
litellm:
|
||||
provider: openai
|
||||
model: glm-5.3
|
||||
endpoint: {baseUrl: https://api.z.ai/api/coding/paas/v4}
|
||||
apiKeyEnv: ZAI_API_KEY
|
||||
- id: qwen-36
|
||||
label: Qwen 3.6
|
||||
litellm:
|
||||
provider: openai
|
||||
model: qwen3.6-35b-a3b
|
||||
disableThinking: true
|
||||
endpoint: {baseUrl: https://models.internal.example/v1}
|
||||
`)
|
||||
|
||||
installation, err := Load(installationPath)
|
||||
if err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
if len(installation.MetadataGeneration.Models) != 3 {
|
||||
t.Fatalf("metadata models = %#v", installation.MetadataGeneration.Models)
|
||||
}
|
||||
qwen := installation.MetadataGeneration.Models[2]
|
||||
if qwen.APIKeyEnv != "" || !qwen.LiteLLM.DisableThinking || qwen.LiteLLM.Endpoint == nil {
|
||||
t.Fatalf("keyless qwen model = %#v", qwen)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadAcceptsOnlyExplicitKeylessMetadataGenerationModelWithoutBundle(t *testing.T) {
|
||||
installationPath, _, envFile, _ := writeInstallation(t, "local")
|
||||
appendFile(t, envFile, "THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
|
||||
appendFile(t, installationPath, `metadataGeneration:
|
||||
default: qwen-36
|
||||
models:
|
||||
- id: qwen-36
|
||||
label: Qwen 3.6
|
||||
litellm:
|
||||
provider: openai
|
||||
model: qwen3.6-35b-a3b
|
||||
disableThinking: true
|
||||
endpoint: {baseUrl: https://models.internal.example/v1}
|
||||
`)
|
||||
|
||||
if _, err := Load(installationPath); err != nil {
|
||||
t.Fatalf("Load() error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRejectsTooManyMetadataGenerationModels(t *testing.T) {
|
||||
installationPath, _, _, _ := writeInstallation(t, "local")
|
||||
var configuration strings.Builder
|
||||
configuration.WriteString("metadataGeneration:\n default: model-0\n models:\n")
|
||||
for index := 0; index <= maxMetadataGenerationModels; index++ {
|
||||
fmt.Fprintf(&configuration, ` - id: model-%d
|
||||
label: Model %d
|
||||
litellm: {provider: openai, model: gpt-4.1-mini}
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`, index, index)
|
||||
}
|
||||
appendFile(t, installationPath, configuration.String())
|
||||
|
||||
_, err := Load(installationPath)
|
||||
want := fmt.Sprintf(
|
||||
"metadataGeneration.models must contain at most %d entries",
|
||||
maxMetadataGenerationModels,
|
||||
)
|
||||
if err == nil || !strings.Contains(err.Error(), want) {
|
||||
t.Fatalf("Load() error = %v, want %q", err, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRejectsDuplicateMetadataGenerationModelIDs(t *testing.T) {
|
||||
installationPath, _, envFile, _ := writeInstallation(t, "local")
|
||||
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
|
||||
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
|
||||
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
|
||||
appendFile(t, installationPath, `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm: {provider: openai, model: gpt-4.1-mini}
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
- id: openai-mini
|
||||
label: Duplicate
|
||||
litellm: {provider: openai, model: gpt-4.1}
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`)
|
||||
|
||||
_, err := Load(installationPath)
|
||||
if err == nil || !strings.Contains(err.Error(), `duplicate metadataGeneration model id "openai-mini"`) {
|
||||
t.Fatalf("Load() error = %v, want actionable duplicate-id error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRejectsMissingOrUnknownMetadataGenerationDefault(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
defaultYAML string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "missing",
|
||||
want: "metadataGeneration.default is required when models are configured",
|
||||
},
|
||||
{
|
||||
name: "unknown",
|
||||
defaultYAML: " default: unavailable\n",
|
||||
want: `metadataGeneration.default "unavailable" does not identify a configured model`,
|
||||
},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
installationPath, _, envFile, _ := writeInstallation(t, "local")
|
||||
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
|
||||
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
|
||||
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
|
||||
appendFile(t, installationPath, "metadataGeneration:\n"+test.defaultYAML+` models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm: {provider: openai, model: gpt-4.1-mini}
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`)
|
||||
|
||||
_, err := Load(installationPath)
|
||||
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("Load() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRejectsMalformedMetadataGenerationModelSettings(t *testing.T) {
|
||||
validPrefix := ` - id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm:
|
||||
`
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
model string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "unstable id",
|
||||
model: strings.Replace(validPrefix, "openai-mini", "OpenAI Mini", 1) +
|
||||
" provider: openai\n model: gpt-4.1-mini\n apiKeyEnv: OPENAI_API_KEY\n",
|
||||
want: "metadataGeneration.models[0].id is invalid",
|
||||
},
|
||||
{
|
||||
name: "blank label",
|
||||
model: strings.Replace(validPrefix, "OpenAI Mini", `" "`, 1) +
|
||||
" provider: openai\n model: gpt-4.1-mini\n apiKeyEnv: OPENAI_API_KEY\n",
|
||||
want: "metadataGeneration.models[0].label is invalid",
|
||||
},
|
||||
{
|
||||
name: "provider",
|
||||
model: validPrefix + " provider: open ai\n model: gpt-4.1-mini\n apiKeyEnv: OPENAI_API_KEY\n",
|
||||
want: "metadataGeneration.models[0].litellm.provider is invalid",
|
||||
},
|
||||
{
|
||||
name: "model",
|
||||
model: validPrefix + " provider: openai\n model: \" gpt-4.1-mini\"\n apiKeyEnv: OPENAI_API_KEY\n",
|
||||
want: "metadataGeneration.models[0].litellm.model is invalid",
|
||||
},
|
||||
{
|
||||
name: "endpoint",
|
||||
model: validPrefix + " provider: openai\n model: gpt-4.1-mini\n" +
|
||||
" endpoint:\n baseUrl: https://operator@api.example/v1\n apiKeyEnv: OPENAI_API_KEY\n",
|
||||
want: "metadataGeneration.models[0].litellm.endpoint.baseUrl is invalid",
|
||||
},
|
||||
{
|
||||
name: "api version",
|
||||
model: validPrefix + " provider: openai\n model: gpt-4.1-mini\n" +
|
||||
" endpoint:\n baseUrl: https://api.example/v1\n apiVersion: \"bad version\"\n apiKeyEnv: OPENAI_API_KEY\n",
|
||||
want: "metadataGeneration.models[0].litellm.endpoint.apiVersion is invalid",
|
||||
},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
installationPath, _, envFile, _ := writeInstallation(t, "local")
|
||||
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
|
||||
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
|
||||
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
|
||||
appendFile(t, installationPath, "metadataGeneration:\n default: openai-mini\n models:\n"+test.model)
|
||||
|
||||
_, err := Load(installationPath)
|
||||
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("Load() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRejectsMissingOrUnusableMetadataGenerationSecrets(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
apiKeyEnvYAML string
|
||||
bundle string
|
||||
declareBundle bool
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "missing reference",
|
||||
apiKeyEnvYAML: "",
|
||||
bundle: "OPENAI_API_KEY=provider-secret\n",
|
||||
declareBundle: true,
|
||||
want: "metadataGeneration.models[0].apiKeyEnv is required unless an explicit keyless endpoint is configured",
|
||||
},
|
||||
{
|
||||
name: "malformed reference",
|
||||
apiKeyEnvYAML: " apiKeyEnv: openai-api-key\n",
|
||||
bundle: "OPENAI_API_KEY=provider-secret\n",
|
||||
declareBundle: true,
|
||||
want: "metadataGeneration.models[0].apiKeyEnv is invalid",
|
||||
},
|
||||
{
|
||||
name: "unallowed reference",
|
||||
apiKeyEnvYAML: " apiKeyEnv: THT_DWH_API_KEY\n",
|
||||
bundle: "THT_DWH_API_KEY=dwh-secret\n",
|
||||
declareBundle: true,
|
||||
want: "metadataGeneration.models[0].apiKeyEnv is invalid",
|
||||
},
|
||||
{
|
||||
name: "bundle not declared",
|
||||
apiKeyEnvYAML: " apiKeyEnv: OPENAI_API_KEY\n",
|
||||
bundle: "OPENAI_API_KEY=provider-secret\n",
|
||||
want: "metadataGeneration keyed models require THT_SECRETS_FILE",
|
||||
},
|
||||
{
|
||||
name: "reference absent from bundle",
|
||||
apiKeyEnvYAML: " apiKeyEnv: OPENAI_API_KEY\n",
|
||||
bundle: "THT_DWH_API_KEY=dwh-secret\n",
|
||||
declareBundle: true,
|
||||
want: "metadataGeneration model \"openai-mini\" secret \"OPENAI_API_KEY\" is missing",
|
||||
},
|
||||
{
|
||||
name: "unusable value",
|
||||
apiKeyEnvYAML: " apiKeyEnv: OPENAI_API_KEY\n",
|
||||
bundle: "OPENAI_API_KEY=secret with whitespace\n",
|
||||
declareBundle: true,
|
||||
want: "metadataGeneration model \"openai-mini\" secret \"OPENAI_API_KEY\" is unusable",
|
||||
},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
installationPath, _, envFile, _ := writeInstallation(t, "local")
|
||||
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
|
||||
if err := os.WriteFile(secretBundle, []byte(test.bundle), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
environment := "THT_INSTALLATION_CONFIG_SOURCE=" + strconv.Quote(installationPath) + "\n"
|
||||
if test.declareBundle {
|
||||
environment += "THT_SECRETS_FILE=" + strconv.Quote(secretBundle) + "\n"
|
||||
}
|
||||
appendFile(t, envFile, environment)
|
||||
appendFile(t, installationPath, `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm: {provider: openai, model: gpt-4.1-mini}
|
||||
`+test.apiKeyEnvYAML)
|
||||
|
||||
_, err := Load(installationPath)
|
||||
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("Load() error = %v, want %q", err, test.want)
|
||||
}
|
||||
if strings.Contains(strings.ToLower(err.Error()), "secret with whitespace") ||
|
||||
strings.Contains(strings.ToLower(err.Error()), "provider-secret") {
|
||||
t.Fatalf("Load() exposed secret content: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRejectsUnprotectedMetadataGenerationSecretBundle(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("POSIX mode assertion; Windows ACL coverage lives in safeio")
|
||||
}
|
||||
installationPath, _, envFile, _ := writeInstallation(t, "local")
|
||||
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
|
||||
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
|
||||
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
|
||||
appendFile(t, installationPath, `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm: {provider: openai, model: gpt-4.1-mini}
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`)
|
||||
|
||||
_, err := Load(installationPath)
|
||||
if err == nil || !strings.Contains(err.Error(), "metadataGeneration secrets in THT_SECRETS_FILE are unavailable") {
|
||||
t.Fatalf("Load() error = %v, want protected-bundle error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRejectsUnknownMetadataGenerationSecretBundleKeys(t *testing.T) {
|
||||
installationPath, _, envFile, _ := writeInstallation(t, "local")
|
||||
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
|
||||
if err := os.WriteFile(
|
||||
secretBundle,
|
||||
[]byte("OPENAI_API_KEY=provider-secret\nUNRECOGNIZED_API_KEY=unknown-secret\n"),
|
||||
0o600,
|
||||
); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
appendFile(t, envFile, "THT_SECRETS_FILE="+strconv.Quote(secretBundle)+"\n"+
|
||||
"THT_INSTALLATION_CONFIG_SOURCE="+strconv.Quote(installationPath)+"\n")
|
||||
appendFile(t, installationPath, `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm: {provider: openai, model: gpt-4.1-mini}
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`)
|
||||
|
||||
_, err := Load(installationPath)
|
||||
if err == nil || !strings.Contains(err.Error(), "metadataGeneration secrets in THT_SECRETS_FILE are invalid") {
|
||||
t.Fatalf("Load() error = %v, want invalid-bundle error", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "UNRECOGNIZED_API_KEY") || strings.Contains(err.Error(), "unknown-secret") {
|
||||
t.Fatalf("Load() exposed rejected bundle content: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRequiresConfiguredMetadataGenerationToUseTheSameMountedDescriptor(t *testing.T) {
|
||||
for _, configuredSource := range []string{"", "/different/thothii-installation.yaml"} {
|
||||
installationPath, _, envFile, _ := writeInstallation(t, "local")
|
||||
secretBundle := filepath.Join(filepath.Dir(installationPath), "thothii.secrets")
|
||||
if err := os.WriteFile(secretBundle, []byte("OPENAI_API_KEY=provider-secret\n"), 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
environment := "THT_SECRETS_FILE=" + strconv.Quote(secretBundle) + "\n"
|
||||
if configuredSource != "" {
|
||||
environment += "THT_INSTALLATION_CONFIG_SOURCE=" + strconv.Quote(configuredSource) + "\n"
|
||||
}
|
||||
appendFile(t, envFile, environment)
|
||||
appendFile(t, installationPath, `metadataGeneration:
|
||||
default: openai-mini
|
||||
models:
|
||||
- id: openai-mini
|
||||
label: OpenAI Mini
|
||||
litellm: {provider: openai, model: gpt-4.1-mini}
|
||||
apiKeyEnv: OPENAI_API_KEY
|
||||
`)
|
||||
|
||||
_, err := Load(installationPath)
|
||||
if err == nil || !strings.Contains(err.Error(), "metadataGeneration requires THT_INSTALLATION_CONFIG_SOURCE to match the installation file") {
|
||||
t.Fatalf("Load() source %q error = %v", configuredSource, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func appendFile(t *testing.T, path, contents string) {
|
||||
t.Helper()
|
||||
file, err := os.OpenFile(path, os.O_APPEND|os.O_WRONLY, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer file.Close()
|
||||
if _, err := file.WriteString(contents); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
@@ -343,6 +343,7 @@ func render(root, descriptorPath string, value answers) ([]byte, []byte, error)
|
||||
"THT_WORKSPACE_GIT_BRANCH=" + dotenvValue(value.workspaceBranch),
|
||||
"THT_WORKSPACE_INSTALLATION_ID=" + dotenvValue(value.installationID),
|
||||
"THT_AUTH_CONFIG_ROOT=" + dotenvValue(descriptor.Authentication.ConfigDirectory),
|
||||
"THT_INSTALLATION_CONFIG_SOURCE=" + dotenvValue(descriptorPath),
|
||||
"THT_SECRETS_FILE=" + dotenvValue(value.secretsFile),
|
||||
"PI_AUTH_FILE=" + dotenvValue(value.piAuthFile),
|
||||
"THOTH_HTTP_PORT=8080", "THOTH_CORE_HTTP_PORT=8787", "MAX_PI_PROCESSES=4",
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
package setup
|
||||
|
||||
import (
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestEnsureFilesProjectsItsInstallationYAMLToTheApplication(t *testing.T) {
|
||||
root := newProject(t, "metadata generation config")
|
||||
setNonInteractiveAnswers(t, newExternalSecrets(t, root))
|
||||
|
||||
result, err := EnsureFiles(Request{
|
||||
ProjectRoot: root, InstallationID: "metadata", Profile: "local", NonInteractive: true,
|
||||
}, strings.NewReader(""), ioDiscard{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
environment, err := os.ReadFile(result.EnvironmentPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := "THT_INSTALLATION_CONFIG_SOURCE=" + strconv.Quote(result.DescriptorPath)
|
||||
if !strings.Contains(string(environment), want) {
|
||||
t.Fatalf("generated environment does not project installation YAML: want %q", want)
|
||||
}
|
||||
if strings.Contains(string(environment), "PI_MODEL") || strings.Contains(string(environment), "llm_policy") {
|
||||
t.Fatal("metadata-generation projection was coupled to Pi or workspace policy")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user