feat: implement metadata catalog database management

This commit is contained in:
Codex
2026-08-27 22:43:54 +02:00
parent 705af3aeb2
commit 79c4c925b5
86 changed files with 12566 additions and 135 deletions
+14
View File
@@ -1,5 +1,19 @@
# AGENTS.md
## Agent skills
### Issue tracker
Issues for this repository live in the self-hosted Gitea repository at `https://git.tylconsulting.it/mptyl/ThothII`; use its web UI or authenticated Gitea API. See `docs/agents/issue-tracker.md`.
### Triage labels
Use the canonical labels `needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, and `wontfix`. See `docs/agents/triage-labels.md`.
### Domain docs
This is a single-context repository with root `CONTEXT.md` and `docs/adr/`. See `docs/agents/domain.md`.
This file provides guidance to Codex (Codex.ai/code) when working with code in this repository.
## Start here
+73 -11
View File
@@ -246,10 +246,23 @@ precedente o di un altro workspace.
## Catalogo dei metadati
**Workspace Database** — Il database associato in modo uno-a-uno a un workspace,
considerato nella sua interezza fisica: tutte le tabelle, le colonne e le relazioni
disponibili. La sua struttura fisica viene acquisita interrogando il database; il
Metadata Catalog non crea né possiede l'identità del workspace.
**Workspace Database** — Il database che appartiene a un solo workspace e non può essere
condiviso con altri workspace; un workspace può averne al massimo uno. È considerato nella
coppia composta dal database PostgreSQL e da un solo schema: tutte le tabelle, le colonne e
le relazioni catalogate appartengono a quello schema. Il Metadata Catalog conserva
l'associazione, ma non crea né possiede l'identità del workspace.
**Database Binding** — La configurazione specifica di un'installazione che seleziona un
trasporto e fornisce i riferimenti necessari a raggiungere un Workspace Database. Non è una
seconda identità del database e non viene condivisa automaticamente fra installazioni.
**Thoth REST Connector** — Il trasporto REST tipizzato con cui ThothII interroga ed
introspeziona un Workspace Database attraverso il contratto RPC DWH supportato. Non è un
client configurabile per API REST arbitrarie.
**Orphaned Workspace Database** — Un Workspace Database il cui workspace non è più presente
nel catalogo autorevole. Rimane conservato per il recupero amministrativo, ma non può essere
usato dal workflow finché non viene riassegnato a un workspace esistente.
**Metadata Catalog** — Il contesto amministrativo che raccoglie e cura i metadati di un
Workspace Database. Non definisce quali elementi partecipano al workflow SQL.
@@ -257,12 +270,61 @@ Workspace Database. Non definisce quali elementi partecipano al workflow SQL.
**Database Profile** — L'insieme curato di scope, descrizioni e metadati semantici
associato a un Workspace Database.
**Physical Schema Snapshot** — L'inventario della struttura fisica osservata in un
Workspace Database durante una specifica introspezione. Non è un progetto dello schema
né un'autorizzazione a modificarne la struttura.
**Physical Table** — Una tabella osservata nello schema esterno di un Workspace Database.
La sua identità e il suo nome appartengono al database esterno, non al Metadata Catalog.
**AI Proposal** — Un contenuto generato con l'ausilio dell'AI che non è ancora stato
approvato come contenuto canonico.
**Catalog Table** — La rappresentazione persistita di una Physical Table nel Metadata Catalog.
La sua appartenenza e identità fisica derivano esclusivamente dall'introspezione; soltanto i suoi
Catalog Metadata possono essere curati amministrativamente.
_Avoid_: SqlTable, managed table
**Publication** — Una versione approvata e immutabile dei contenuti del Metadata
Catalog resa disponibile ai suoi consumatori.
**Physical Column** — Una colonna osservata in una Physical Table, inclusi nome, posizione,
tipo e appartenenza a chiavi dichiarate. La sua identità e i suoi fatti strutturali appartengono
al database esterno.
**Catalog Column** — La rappresentazione persistita di una Physical Column nel Metadata Catalog.
I fatti osservati sono governati dalla sincronizzazione; Description e Generated Description
sono metadati amministrativi modificabili.
_Avoid_: SqlColumn, managed column
**Physical Relationship** — Un vincolo foreign key dichiarato nel database esterno. La sua
identità comprende il vincolo e la sequenza ordinata delle coppie di colonne che lo compongono.
**Catalog Relationship** — La rappresentazione persistita di una Physical Relationship nel
Metadata Catalog. È governata esclusivamente dall'introspezione e non è creata o modificata
manualmente.
_Avoid_: denormalized FK, relationship string
**Logical Relationship** — Una relazione semantica curata o inferita che non corrisponde
necessariamente a un vincolo fisico. Ha ownership e lifecycle distinti da Catalog Relationship.
**Description** — Il testo curato e consolidato che descrive una Catalog Table o Catalog Column
per gli usi downstream.
**Generated Description** — Una proposta modificabile sottoposta a revisione umana prima di
essere consolidata come Description. Rimane distinta dal commento osservato nel database.
_Avoid_: generated comment, source comment
**Table Synchronization** — La riconciliazione esplicita che rende le Catalog Table di un
Workspace Database uguali alle Physical Table osservate: crea quelle nuove, aggiorna i metadati
di origine ed elimina definitivamente quelle assenti. Non modifica mai il database esterno.
_Avoid_: table import
**Schema Synchronization** — La riconciliazione esplicita e autorevole di tabelle, colonne e
Catalog Relationship di un Workspace Database. Può operare su uno scope specifico oppure su
un unico snapshot completo tramite Synchronize All.
**Catalog Sync Run** — L'esecuzione durevole in background di una Schema Synchronization, con
scope, stato, avanzamento e log propri. Al massimo un run per Workspace Database può essere attivo.
**Catalog Freshness** — La corrispondenza fra uno scope sincronizzato e la versione corrente
della Database Binding. Uno scope rimane consultabile ma è stale finché non viene sincronizzato
con la binding corrente.
**Catalog Metadata** — I campi mutabili che descrivono database, tabelle, colonne e relazioni,
distinti dai fatti strutturali governati dalla sincronizzazione. Possono essere popolati dall'AI,
da un'importazione o da una modifica amministrativa senza cambiare il database esterno.
**Introspection Capability** — Una categoria di struttura fisica che una Database Binding
può osservare, come tabelle, colonne, relazioni, indici o enum. Una capability non disponibile
è distinta da una capability osservata che non ha restituito elementi.
+49 -3
View File
@@ -1,6 +1,6 @@
# ThothII — Project State
Last updated: 2026-08-26.
Last updated: 2026-08-27.
This file is the short operational snapshot. Stable commands and the architecture mental model
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
@@ -16,8 +16,9 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
```
The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence.
The backend is a process/RPC/SSE bridge without a database of its own. The frontend renders the
review gates and keeps the live transcript in memory. See
The backend remains a process/RPC/SSE bridge for sessions and now also owns an isolated PostgreSQL
metadata catalog for administrative database configuration. The frontend renders the review gates
and keeps the live transcript in memory. See
`docs/architecture/components.md` for the detailed component and data-flow map.
## Evidence restructuring — accepted
@@ -62,6 +63,51 @@ Workspace descriptors use schema v3. For PSD, workspace content and runtime root
separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside
Git and are supplied only through installation-local protected files.
## Database management
The database, table, and authoritative physical-schema catalog slices are implemented. Database
management opens a responsive AG Grid master-detail surface, lists every YAML workspace, creates
at most one PostgreSQL database configuration per workspace, edits direct PostgreSQL, REST API, or
SSH-tunnel installation bindings, replaces write-only encrypted secrets, and tests supported
connector bindings.
Configured databases use pure hierarchical navigation through `Overview`, `Tables`, and
`Relationships`; a selected table has `Overview` and `Columns`. Physical membership, source
comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are
immutable projections of the external schema. Curated and generated descriptions are editable;
generated descriptions start null and AI generation/consolidation is deferred.
Schema refresh is one durable asynchronous engine with database-table, database-column,
selected-table-column, relationship, and full-database actions. Database-level menus expose the
table, all-column, relationship, and full scopes separately; selecting tables exposes column
synchronization for that subset. Runs have one-active-job-per-database exclusion, leases and
restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before
apply, retained history, and a live SSE log with polling fallback. Null metadata renders blank
rather than as a placeholder.
Direct PostgreSQL and strict known-host-verified OpenSSH use `pg_catalog`. REST bindings use the
typed full-snapshot `POST /rpc/schema_snapshot` contract when available. Servers such as the
current PSD endpoint that exposes only `POST /rpc/run_query` use one catalog-owned read-only query
to return the exact same strict v1 snapshot in a single round trip. Both paths remain fail-closed:
an absent capability, query error, partial result, or invalid snapshot applies no catalog changes.
SSH is not yet enabled for NL→SQL session runtime.
The catalog runs in the internal `catalog-db` PostgreSQL service. Kysely migrations are an explicit
one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before local startup. Runtime
sessions still consume the existing workspace configuration in this slice: database-management
records do not yet change the NL→SQL handoff. The accepted design is recorded in
`docs/plans/2026-08-26-metadata-catalog-from-thothai.md`, the snapshot contract under
`docs/contracts/`, and ADRs 0001–0007.
Semantic aliases, value descriptions, synonyms, concepts, AI metadata generation/consolidation,
and logical relationships remain deferred to their dedicated slices.
Integration of the completed metadata catalog with core schema-linking is explicitly deferred
until the database, table, column, relationship, and synchronization slices are complete. At that
point the next required design gate is to compare the catalog snapshot with the current DWH
preprocessing/schema-linking contracts and plan the cutover; this follow-up must not be treated as
optional cleanup or silently omitted.
## Active deployment work and manual gates
### PSD server deployment program
+25 -11
View File
@@ -1,15 +1,16 @@
# ThothII
ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/`tht`
core. The portable deployment runs exactly two application services; data services remain
external in this profile, except for the mandatory internal semantic services bundled in Compose.
core. The portable deployment runs two application services plus the installation-local metadata
catalog; DWH and LLM services remain external. Semantic services are bundled in Compose.
Authentication is configured through the single host CLI tht: see the [local authentication guide](docs/install/authentication-local.md),
[generic OIDC guide](docs/install/authentication-oidc.md), and [manual acceptance matrix](docs/testing/authentication-manual-acceptance.md).
## Docker Compose: local startup
Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external,
Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`,
`catalog-db`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external,
configurable endpoints—even when they are co-located with ThothII.
From a fresh clone, run these commands from the repository root:
@@ -17,17 +18,28 @@ From a fresh clone, run these commands from the repository root:
```sh
cp deploy/env/local.env.example deploy/env/local.env
# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
docker compose --env-file deploy/env/local.env \
-f compose.yaml -f deploy/compose.local.yaml up --build -d
./scripts/run-stack.sh
```
`./scripts/run-stack.sh` runs this same base+local command in the foreground. The core image
contains its Pi runtime; no host `pi` executable is used. For a server installation:
The launcher builds the core, starts `catalog-db`, runs the explicit one-shot Kysely migrations,
then runs the base+local stack in the foreground. Migrations never run implicitly in backend
startup. The core image contains its Pi runtime; no host `pi` executable is used. For a server
installation, build the image, start the catalog, and run the same migration service before the
application rollout:
```sh
cp deploy/env/server.env.example deploy/env/server.env
# Edit all absolute storage, Pi/secret/session files, and endpoint paths.
sudo scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example build core
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example up -d catalog-db
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example run --rm catalog-migrate
docker compose --env-file deploy/env/server.env \
-f compose.yaml -f deploy/compose.server.yaml \
-f deploy/compose.session-server.yaml.example up --build -d
@@ -101,10 +113,12 @@ schema, Evidence, and Memory records share that collection and stay separated by
`kind`.
<!-- workspace-descriptor-contract:end -->
Connector `ssh_tunnel` bindings are diagnostic-only in this release: their bounded probe always
cleans up the loopback forward and returns `workspace_not_activatable`; session creation is rejected
before persistence. Git registry access over SSH is unaffected. Use direct or REST connector
transport for runtime sessions.
For NL→SQL runtime sessions, connector `ssh_tunnel` bindings remain diagnostic-only: their bounded
probe cleans up the loopback forward and returns `workspace_not_activatable`; session creation is
rejected before persistence. Database management is a separate boundary and supports a strict
OpenSSH tunnel for **Test connection** and **Sync tables**, using a private key, optional passphrase,
mandatory `known_hosts`, and optional PostgreSQL TLS CA/server name. Git registry access over SSH is
unaffected. Use direct or REST connector transport for runtime sessions.
`docker-compose.dev.yml` is deliberately local: both published ports bind to `127.0.0.1`,
`THT_SESSION_STORAGE=local`, and `THT_HOME=/data/local-home`. Do not set
+2049
View File
File diff suppressed because it is too large Load Diff
+3
View File
@@ -6,6 +6,7 @@
"dev": "tsx watch src/server.ts",
"prebuild": "node scripts/clean-dist.mjs",
"build": "tsc -p tsconfig.json",
"catalog:migrate": "node dist/catalog/migrate.js",
"test": "vitest run",
"start": "node dist/server.js",
"test:schema-v3-verifier": "python3 -I -B scripts/test_revision_state_policy.py && node --test scripts/verify-workspace-descriptor-files.test.mjs scripts/revision-state-policy.test.mjs"
@@ -16,12 +17,14 @@
"@fastify/rate-limit": "11.2.0",
"@types/pg": "^8.20.3",
"fastify": "^5.0.0",
"kysely": "^0.29.5",
"openid-client": "6.8.5",
"pg": "^8.22.0",
"yaml": "^2.9.0",
"zod": "^4.4.3"
},
"devDependencies": {
"@testcontainers/postgresql": "^12.1.0",
"@types/node": "24.13.3",
"tsx": "^4.19.0",
"typescript": "^5.6.0",
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env node
import { readFileSync } from "node:fs";
const path = process.env.THT_SSH_PASSPHRASE_FILE;
if (!path) process.exit(1);
process.stdout.write(readFileSync(path));
+52
View File
@@ -37,6 +37,17 @@ import { supportsSessionRuntime } from "./workspaces/bindings.js";
import { resolveRuntimeBindingsWithWorkspaceSecrets } from "./workspaces/secret-requirements.js";
import type { WorkspaceDescriptor } from "./workspaces/schema.js";
import { WorkspaceSecretStore } from "./workspaces/secret-store.js";
import { createCatalogRepository } from "./catalog/repository.js";
import type { CatalogRepository } from "./catalog/types.js";
import { CatalogService } from "./catalog/service.js";
import { catalogDatabaseRoutes } from "./routes/catalog-databases.js";
import { CatalogOperationCoordinator } from "./catalog/operation-coordinator.js";
import { ConcreteCatalogPostgresAccess, type CatalogPostgresAccess } from "./catalog/postgres-access.js";
import { CatalogTableService } from "./catalog/table-service.js";
import { catalogTableRoutes } from "./routes/catalog-tables.js";
import { ConcreteCatalogSchemaIntrospector, type CatalogSchemaIntrospector } from "./catalog/schema-introspector.js";
import { CatalogSyncWorker } from "./catalog/sync-worker.js";
import { catalogSchemaRoutes } from "./routes/catalog-schema.js";
export interface BuildAppDeps {
thtRunner?: ThtRunner;
@@ -49,6 +60,13 @@ export interface BuildAppDeps {
workspaceRegistry?: WorkspaceRegistry;
workspaceDiagnoser?: WorkspaceDiagnoser;
workspaceSecretStore?: WorkspaceSecretStore;
catalogRepository?: CatalogRepository;
catalogService?: CatalogService;
catalogPostgresAccess?: CatalogPostgresAccess;
catalogTableService?: CatalogTableService;
catalogSchemaIntrospector?: CatalogSchemaIntrospector;
catalogSyncWorker?: CatalogSyncWorker;
catalogOperationCoordinator?: CatalogOperationCoordinator;
workspaceRuntimeSupport?: (workspace: WorkspaceDescriptor) => boolean;
maintenanceBarrier?: MaintenanceBarrier;
piManagement?: PiManagementService;
@@ -121,6 +139,37 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
const mgr = deps?.mgr ?? new PiProcessManager(config, deps?.spawnFn ? { spawnFn: deps.spawnFn } : undefined);
const hub = deps?.hub ?? new SseHub();
const workspaceRegistry = deps?.workspaceRegistry ?? new WorkspaceRegistry(config.workspaceRegistry);
const catalogRepository = deps?.catalogRepository ?? createCatalogRepository(config.catalogDatabase);
const catalogOperationCoordinator = deps?.catalogOperationCoordinator ?? new CatalogOperationCoordinator();
const catalogPostgresAccess = deps?.catalogPostgresAccess ?? new ConcreteCatalogPostgresAccess(
workspaceSecretStore,
{ connectTimeoutMs: config.workspaceDiagnosticTimeoutMs },
);
const catalogService = deps?.catalogService ?? new CatalogService(
catalogRepository,
workspaceRegistry,
workspaceSecretStore,
config.workspaceRegistry.secretRoots,
config.workspaceDiagnosticTimeoutMs,
catalogPostgresAccess,
catalogOperationCoordinator,
);
const catalogTableService = deps?.catalogTableService ?? new CatalogTableService(catalogRepository);
const catalogSchemaIntrospector = deps?.catalogSchemaIntrospector ?? new ConcreteCatalogSchemaIntrospector(
catalogPostgresAccess,
workspaceSecretStore,
);
const catalogSyncWorker = deps?.catalogSyncWorker ?? new CatalogSyncWorker(
catalogRepository,
catalogSchemaIntrospector,
catalogOperationCoordinator,
config.catalogSyncTimeoutMs,
);
app.addHook("onReady", async () => { await catalogSyncWorker.initialize(); });
if (!deps?.catalogRepository && catalogRepository.close) {
app.addHook("onClose", async () => { await catalogRepository.close?.(); });
}
app.addHook("onClose", async () => { await catalogSyncWorker.stop(); });
const workspaceDiagnoser = deps?.workspaceDiagnoser
?? createProductionWorkspaceDiagnoser(config.workspaceDiagnosticTimeoutMs, undefined, {
internalQdrantUrl: config.internalQdrantUrl,
@@ -334,6 +383,9 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
authDiagnoser,
secretStore: workspaceSecretStore,
});
catalogDatabaseRoutes(app, { repository: catalogRepository, service: catalogService, operations: catalogOperationCoordinator });
catalogTableRoutes(app, { repository: catalogRepository, service: catalogTableService });
catalogSchemaRoutes(app, { repository: catalogRepository, worker: catalogSyncWorker });
settingsRoutes(app, { cfg: config, listModels, getSettings });
piManagementRoutes(app, { service: piManagement });
+1 -1
View File
@@ -38,7 +38,7 @@ const MAX_MAPPED_GROUPS = 128;
const ROLES = ["user", "admin"] as const;
export const PERMISSION_CATALOG: readonly Permission[] = [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
];
const invalid = (): Error => new Error("authentication configuration is invalid");
+1 -1
View File
@@ -33,7 +33,7 @@ const EMPTY_HKDF_SALT = Buffer.alloc(0);
const ROLES = ["user", "admin"] as const;
const PERMISSIONS = [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
] as const satisfies readonly Permission[];
const invalid = (): Error => new Error("auth_session_store_invalid");
+1 -1
View File
@@ -5,7 +5,7 @@ export type Role = "user" | "admin";
export type Permission =
| "session.use" | "session.read_all" | "session.manage_all"
| "settings.manage" | "workspace.manage" | "workspace.secrets.manage"
| "pi.manage" | "auth.diagnostics.read";
| "database.manage" | "pi.manage" | "auth.diagnostics.read";
export interface AuthenticationSessionConfig {
regularTtlSeconds: number;
+692
View File
@@ -0,0 +1,692 @@
import { randomUUID } from "node:crypto";
import {
CatalogConflictError,
CatalogConnectorError,
type CatalogColumn,
type CatalogRelationship,
type CatalogSchemaDiff,
type CatalogSyncCounts,
type CatalogSyncEvent,
type CatalogSyncRun,
type CatalogSyncRunUpdate,
type CatalogSyncScope,
type CatalogTable,
type CatalogRepository,
type DatabaseConfigurationInput,
type DatabaseTestResult,
type ObservedCatalogTable,
type ObservedSchemaSnapshot,
type TableSyncRepositoryResult,
type WorkspaceDatabase,
} from "./types.js";
function clone(value: WorkspaceDatabase): WorkspaceDatabase {
return structuredClone(value);
}
/** Deterministic repository used by route tests and local contract consumers. */
export class MemoryCatalogRepository implements CatalogRepository {
private readonly records = new Map<string, WorkspaceDatabase>();
private readonly tables = new Map<string, CatalogTable>();
private readonly columns = new Map<string, CatalogColumn>();
private readonly relationships = new Map<string, CatalogRelationship>();
private readonly syncRuns = new Map<string, CatalogSyncRun>();
private readonly syncEvents = new Map<string, CatalogSyncEvent[]>();
async list(): Promise<WorkspaceDatabase[]> {
return [...this.records.values()].sort((a, b) => a.workspaceId.localeCompare(b.workspaceId)).map(clone);
}
async get(id: string): Promise<WorkspaceDatabase | undefined> {
const value = this.records.get(id);
return value ? clone(value) : undefined;
}
async getByWorkspace(workspaceId: string): Promise<WorkspaceDatabase | undefined> {
const value = [...this.records.values()].find((record) => record.workspaceId === workspaceId);
return value ? clone(value) : undefined;
}
async create(input: DatabaseConfigurationInput): Promise<WorkspaceDatabase> {
if ([...this.records.values()].some((record) => record.workspaceId === input.workspaceId)) {
throw new CatalogConflictError("Workspace database already exists");
}
const now = new Date().toISOString();
const record: WorkspaceDatabase = {
id: randomUUID(),
...structuredClone(input),
version: 1,
createdAt: now,
updatedAt: now,
connectionStatus: "untested",
};
this.records.set(record.id, record);
return clone(record);
}
async update(id: string, expectedVersion: number, input: DatabaseConfigurationInput): Promise<WorkspaceDatabase | undefined> {
const current = this.records.get(id);
if (!current || current.version !== expectedVersion) return undefined;
if ([...this.records.values()].some((record) => record.id !== id && record.workspaceId === input.workspaceId)) {
throw new CatalogConflictError("Workspace database already exists");
}
const updated: WorkspaceDatabase = {
...current,
...structuredClone(input),
version: current.version + 1,
updatedAt: new Date().toISOString(),
connectionStatus: "untested",
testedVersion: undefined,
lastTestedAt: undefined,
lastErrorCode: undefined,
lastErrorMessage: undefined,
schemaSyncedVersion: undefined,
schemaSyncedAt: undefined,
};
this.records.set(id, updated);
return clone(updated);
}
async recordTest(id: string, expectedVersion: number, result: DatabaseTestResult): Promise<WorkspaceDatabase | undefined> {
const current = this.records.get(id);
if (!current || current.version !== expectedVersion) return undefined;
const updated = {
...current,
connectionStatus: result.connectionStatus,
testedVersion: result.testedVersion,
lastTestedAt: result.lastTestedAt,
lastErrorCode: result.errorCode,
lastErrorMessage: result.errorMessage,
};
this.records.set(id, updated);
return clone(updated);
}
async touch(id: string, expectedVersion: number): Promise<WorkspaceDatabase | undefined> {
const current = this.records.get(id);
if (!current || current.version !== expectedVersion) return undefined;
return await this.update(id, expectedVersion, {
workspaceId: current.workspaceId,
engine: current.engine,
databaseName: current.databaseName,
schema: current.schema,
binding: current.binding,
});
}
async delete(id: string, expectedVersion: number): Promise<boolean> {
const current = this.records.get(id);
if (!current || current.version !== expectedVersion) return false;
for (const [tableId, table] of this.tables) {
if (table.databaseId === id) {
this.tables.delete(tableId);
for (const [columnId, column] of this.columns) if (column.tableId === tableId) this.columns.delete(columnId);
}
}
for (const [relationshipId, relationship] of this.relationships) {
if (relationship.databaseId === id) this.relationships.delete(relationshipId);
}
return this.records.delete(id);
}
async listTables(databaseId: string): Promise<CatalogTable[]> {
return [...this.tables.values()]
.filter((table) => table.databaseId === databaseId)
.sort((a, b) => a.name.localeCompare(b.name))
.map((table) => structuredClone(table));
}
async getTable(databaseId: string, tableId: string): Promise<CatalogTable | undefined> {
const table = this.tables.get(tableId);
return table?.databaseId === databaseId ? structuredClone(table) : undefined;
}
async updateTableDescription(
databaseId: string,
tableId: string,
expectedVersion: number,
description: string | null,
): Promise<CatalogTable | undefined> {
const current = this.tables.get(tableId);
if (!current || current.databaseId !== databaseId || current.version !== expectedVersion) return undefined;
const updated = {
...current,
description,
version: current.version + 1,
updatedAt: new Date().toISOString(),
};
this.tables.set(tableId, updated);
return structuredClone(updated);
}
async updateTableMetadata(
databaseId: string,
tableId: string,
expectedVersion: number,
description: string | null,
generatedDescription: string | null,
): Promise<CatalogTable | undefined> {
const current = this.tables.get(tableId);
if (!current || current.databaseId !== databaseId || current.version !== expectedVersion) return undefined;
const updated = {
...current, description, generatedDescription, version: current.version + 1,
updatedAt: new Date().toISOString(),
};
this.tables.set(tableId, updated);
return structuredClone(updated);
}
async listColumns(databaseId: string, tableId: string): Promise<CatalogColumn[]> {
const table = this.tables.get(tableId);
if (!table || table.databaseId !== databaseId) return [];
return [...this.columns.values()].filter((column) => column.tableId === tableId)
.sort((a, b) => a.ordinalPosition - b.ordinalPosition).map((column) => structuredClone(column));
}
async getColumn(databaseId: string, tableId: string, columnId: string): Promise<CatalogColumn | undefined> {
const table = this.tables.get(tableId);
const column = this.columns.get(columnId);
return table?.databaseId === databaseId && column?.tableId === tableId ? structuredClone(column) : undefined;
}
async updateColumnMetadata(
databaseId: string,
tableId: string,
columnId: string,
expectedVersion: number,
description: string | null,
generatedDescription: string | null,
): Promise<CatalogColumn | undefined> {
const current = await this.getColumn(databaseId, tableId, columnId);
if (!current || current.version !== expectedVersion) return undefined;
const updated = { ...current, description, generatedDescription, version: current.version + 1, updatedAt: new Date().toISOString() };
this.columns.set(columnId, updated);
return structuredClone(updated);
}
async listRelationships(databaseId: string): Promise<CatalogRelationship[]> {
return [...this.relationships.values()].filter((relationship) => relationship.databaseId === databaseId)
.sort((a, b) => `${a.sourceTableName}.${a.constraintName}`.localeCompare(`${b.sourceTableName}.${b.constraintName}`))
.map((relationship) => structuredClone(relationship));
}
async planSchemaSync(
databaseId: string,
scope: CatalogSyncScope,
tableIds: readonly string[],
snapshot: ObservedSchemaSnapshot,
): Promise<CatalogSchemaDiff> {
this.assertSnapshotCapability(scope, snapshot);
const tables = await this.listTables(databaseId);
const selectedTableIds = new Set(tableIds);
const selectedTables = scope === "columns" && selectedTableIds.size > 0
? tables.filter((table) => selectedTableIds.has(table.id))
: tables;
const observedTables = new Set(snapshot.tables.map((table) => table.name));
const observedColumns = new Set(snapshot.columns.map((column) => `${column.tableName}\u0000${column.name}`));
const observedRelationships = new Set(
snapshot.relationships.map((relationship) => `${relationship.sourceTableName}\u0000${relationship.constraintName}`),
);
const deletedTableIds = new Set(tables.filter((table) => !observedTables.has(table.name)).map((table) => table.id));
return {
deletedTables: scope === "tables" || scope === "all"
? tables.filter((table) => !observedTables.has(table.name)).map((table) => table.name).sort()
: [],
deletedColumns: scope === "tables" || scope === "columns" || scope === "all"
? [...this.columns.values()]
.filter((column) => scope === "tables" ? deletedTableIds.has(column.tableId) : selectedTables.some((table) => table.id === column.tableId))
.filter((column) => {
const table = tables.find((candidate) => candidate.id === column.tableId);
return table && (scope === "tables" || !observedColumns.has(`${table.name}\u0000${column.name}`));
})
.map((column) => ({
tableName: tables.find((table) => table.id === column.tableId)?.name ?? "",
columnName: column.name,
}))
.sort((a, b) => `${a.tableName}.${a.columnName}`.localeCompare(`${b.tableName}.${b.columnName}`))
: [],
deletedRelationships: scope === "tables" || scope === "relationships" || scope === "all"
? [...this.relationships.values()]
.filter((relationship) => relationship.databaseId === databaseId)
.filter((relationship) => scope === "tables"
? deletedTableIds.has(relationship.sourceTableId) || deletedTableIds.has(relationship.targetTableId)
: !observedRelationships.has(`${relationship.sourceTableName}\u0000${relationship.constraintName}`))
.map((relationship) => ({
sourceTableName: relationship.sourceTableName,
constraintName: relationship.constraintName,
}))
.sort((a, b) => `${a.sourceTableName}.${a.constraintName}`.localeCompare(`${b.sourceTableName}.${b.constraintName}`))
: [],
};
}
async applySchemaSync(
databaseId: string,
expectedDatabaseVersion: number,
scope: CatalogSyncScope,
tableIds: readonly string[],
snapshot: ObservedSchemaSnapshot,
): Promise<CatalogSyncCounts | undefined> {
const database = this.records.get(databaseId);
if (!database || database.version !== expectedDatabaseVersion) return undefined;
this.assertSnapshotCapability(scope, snapshot);
const tableBackup = structuredClone([...this.tables.entries()]);
const columnBackup = structuredClone([...this.columns.entries()]);
const relationshipBackup = structuredClone([...this.relationships.entries()]);
const now = new Date().toISOString();
let created = 0;
let updated = 0;
let deleted = 0;
try {
if (scope === "tables" || scope === "all") {
const observedByName = new Map(snapshot.tables.map((table) => [table.name, table]));
const existing = await this.listTables(databaseId);
for (const table of existing) {
const observed = observedByName.get(table.name);
if (!observed) {
this.deleteTable(table.id);
deleted += 1;
} else {
const changed = table.sourceComment !== observed.sourceComment;
this.tables.set(table.id, {
...table,
sourceComment: observed.sourceComment,
lastSyncedDatabaseVersion: expectedDatabaseVersion,
lastSyncedAt: now,
version: changed ? table.version + 1 : table.version,
updatedAt: changed ? now : table.updatedAt,
});
if (changed) updated += 1;
observedByName.delete(table.name);
}
}
for (const observed of observedByName.values()) {
const id = randomUUID();
this.tables.set(id, {
id,
databaseId,
name: observed.name,
sourceComment: observed.sourceComment,
description: null,
generatedDescription: null,
lastSyncedDatabaseVersion: expectedDatabaseVersion,
lastSyncedAt: now,
version: 1,
createdAt: now,
updatedAt: now,
});
created += 1;
}
}
if (scope === "columns" || scope === "all") {
const currentTables = await this.listTables(databaseId);
const selectedIds = scope === "all" || tableIds.length === 0
? new Set(currentTables.map((table) => table.id))
: new Set(tableIds);
const selectedTables = currentTables.filter((table) => selectedIds.has(table.id));
if (scope === "columns" && selectedTables.length !== selectedIds.size) {
throw new CatalogConnectorError("One or more selected tables no longer exist");
}
const selectedNames = new Set(selectedTables.map((table) => table.name));
const tableByName = new Map(selectedTables.map((table) => [table.name, table]));
const observedByKey = new Map(
snapshot.columns
.filter((column) => selectedNames.has(column.tableName))
.map((column) => [`${column.tableName}\u0000${column.name}`, column]),
);
for (const column of [...this.columns.values()].filter((candidate) => selectedIds.has(candidate.tableId))) {
const table = selectedTables.find((candidate) => candidate.id === column.tableId);
if (!table) continue;
const key = `${table.name}\u0000${column.name}`;
const observed = observedByKey.get(key);
if (!observed) {
this.deleteColumn(column.id);
deleted += 1;
} else {
const changed = column.ordinalPosition !== observed.ordinalPosition
|| column.dataType !== observed.dataType
|| column.isNullable !== observed.isNullable
|| column.defaultExpression !== observed.defaultExpression
|| column.primaryKeyPosition !== observed.primaryKeyPosition
|| column.sourceComment !== observed.sourceComment;
this.columns.set(column.id, {
...column,
ordinalPosition: observed.ordinalPosition,
dataType: observed.dataType,
isNullable: observed.isNullable,
defaultExpression: observed.defaultExpression,
primaryKeyPosition: observed.primaryKeyPosition,
isPrimaryKey: observed.primaryKeyPosition !== null,
sourceComment: observed.sourceComment,
lastSyncedDatabaseVersion: expectedDatabaseVersion,
lastSyncedAt: now,
version: changed ? column.version + 1 : column.version,
updatedAt: changed ? now : column.updatedAt,
});
if (changed) updated += 1;
observedByKey.delete(key);
}
}
for (const observed of observedByKey.values()) {
const table = tableByName.get(observed.tableName);
if (!table) continue;
const id = randomUUID();
this.columns.set(id, {
id,
tableId: table.id,
name: observed.name,
ordinalPosition: observed.ordinalPosition,
dataType: observed.dataType,
isNullable: observed.isNullable,
defaultExpression: observed.defaultExpression,
primaryKeyPosition: observed.primaryKeyPosition,
isPrimaryKey: observed.primaryKeyPosition !== null,
isForeignKey: false,
foreignKeyCount: 0,
sourceComment: observed.sourceComment,
description: null,
generatedDescription: null,
lastSyncedDatabaseVersion: expectedDatabaseVersion,
lastSyncedAt: now,
version: 1,
createdAt: now,
updatedAt: now,
});
created += 1;
}
}
if (scope === "relationships" || scope === "all") {
const tables = await this.listTables(databaseId);
const tableByName = new Map(tables.map((table) => [table.name, table]));
const existing = [...this.relationships.values()].filter((relationship) => relationship.databaseId === databaseId);
const existingByKey = new Map(existing.map((relationship) => [`${relationship.sourceTableName}\u0000${relationship.constraintName}`, relationship]));
const observedKeys = new Set(snapshot.relationships.map((relationship) => `${relationship.sourceTableName}\u0000${relationship.constraintName}`));
for (const relationship of existing) {
if (!observedKeys.has(`${relationship.sourceTableName}\u0000${relationship.constraintName}`)) {
this.relationships.delete(relationship.id);
deleted += 1;
}
}
for (const observed of snapshot.relationships) {
const sourceTable = tableByName.get(observed.sourceTableName);
const targetTable = tableByName.get(observed.targetTableName);
if (!sourceTable || !targetTable) {
throw new CatalogConnectorError(`Relationship ${observed.constraintName} refers to an unknown table`);
}
const pairs = observed.columns.map((pair) => {
const source = [...this.columns.values()].find((column) => column.tableId === sourceTable.id && column.name === pair.sourceColumnName);
const target = [...this.columns.values()].find((column) => column.tableId === targetTable.id && column.name === pair.targetColumnName);
if (!source || !target) {
throw new CatalogConnectorError(`Relationship ${observed.constraintName} refers to an unknown column`);
}
return {
position: pair.position,
sourceColumnId: source.id,
sourceColumnName: source.name,
targetColumnId: target.id,
targetColumnName: target.name,
};
}).sort((a, b) => a.position - b.position);
const key = `${observed.sourceTableName}\u0000${observed.constraintName}`;
const current = existingByKey.get(key);
const comparable = current && JSON.stringify({
target: current.targetTableName,
update: current.updateRule,
delete: current.deleteRule,
deferrable: current.deferrable,
deferred: current.initiallyDeferred,
columns: current.columns.map((pair) => [pair.position, pair.sourceColumnName, pair.targetColumnName]),
});
const nextComparable = JSON.stringify({
target: observed.targetTableName,
update: observed.updateRule,
delete: observed.deleteRule,
deferrable: observed.deferrable,
deferred: observed.initiallyDeferred,
columns: pairs.map((pair) => [pair.position, pair.sourceColumnName, pair.targetColumnName]),
});
const id = current?.id ?? randomUUID();
this.relationships.set(id, {
id,
databaseId,
constraintName: observed.constraintName,
sourceTableId: sourceTable.id,
sourceTableName: sourceTable.name,
targetTableId: targetTable.id,
targetTableName: targetTable.name,
updateRule: observed.updateRule,
deleteRule: observed.deleteRule,
deferrable: observed.deferrable,
initiallyDeferred: observed.initiallyDeferred,
columns: pairs,
lastSyncedDatabaseVersion: expectedDatabaseVersion,
lastSyncedAt: now,
createdAt: current?.createdAt ?? now,
updatedAt: comparable === nextComparable ? (current?.updatedAt ?? now) : now,
});
if (!current) created += 1;
else if (comparable !== nextComparable) updated += 1;
}
this.refreshForeignKeyFlags(databaseId);
}
if (scope === "all") {
this.records.set(databaseId, { ...database, schemaSyncedVersion: expectedDatabaseVersion, schemaSyncedAt: now });
}
return {
tables: (await this.listTables(databaseId)).length,
columns: [...this.columns.values()].filter((column) => this.tables.get(column.tableId)?.databaseId === databaseId).length,
relationships: (await this.listRelationships(databaseId)).length,
created,
updated,
deleted,
};
} catch (error) {
this.tables.clear();
this.columns.clear();
this.relationships.clear();
for (const [id, table] of tableBackup) this.tables.set(id, table);
for (const [id, column] of columnBackup) this.columns.set(id, column);
for (const [id, relationship] of relationshipBackup) this.relationships.set(id, relationship);
throw error;
}
}
async createSyncRun(
databaseId: string,
scope: CatalogSyncScope,
tableIds: readonly string[],
requestedDatabaseVersion: number,
): Promise<CatalogSyncRun> {
const activeStates = new Set<CatalogSyncRun["state"]>(["queued", "running", "awaiting_confirmation", "applying"]);
if ([...this.syncRuns.values()].some((run) => run.databaseId === databaseId && activeStates.has(run.state))) {
throw new CatalogConflictError("A schema synchronization is already active for this database");
}
const now = new Date().toISOString();
const run: CatalogSyncRun = {
id: randomUUID(), databaseId, scope, tableIds: [...tableIds], state: "queued", phase: "queued",
requestedDatabaseVersion, observedSnapshot: null, plannedDiff: null, confirmationToken: null,
counts: {}, errorCode: null, errorMessage: null, cancelRequested: false,
createdAt: now, startedAt: null, updatedAt: now, finishedAt: null, heartbeatAt: null,
leaseOwner: null, leaseExpiresAt: null,
};
this.syncRuns.set(run.id, run);
return structuredClone(run);
}
async getSyncRun(runId: string): Promise<CatalogSyncRun | undefined> {
const run = this.syncRuns.get(runId);
return run ? structuredClone(run) : undefined;
}
async claimSyncRun(runId: string, workerId: string, leaseExpiresAt: string): Promise<CatalogSyncRun | undefined> {
const run = this.syncRuns.get(runId);
if (!run || run.state !== "queued") return undefined;
if (run.leaseOwner && run.leaseOwner !== workerId && run.leaseExpiresAt && run.leaseExpiresAt > new Date().toISOString()) {
return undefined;
}
return await this.updateSyncRun(runId, {
state: "running",
startedAt: run.startedAt ?? new Date().toISOString(),
heartbeatAt: new Date().toISOString(),
leaseOwner: workerId,
leaseExpiresAt,
});
}
async listSyncRuns(databaseId: string, limit = 20): Promise<CatalogSyncRun[]> {
return [...this.syncRuns.values()].filter((run) => run.databaseId === databaseId)
.sort((a, b) => b.createdAt.localeCompare(a.createdAt)).slice(0, limit).map((run) => structuredClone(run));
}
async updateSyncRun(runId: string, update: CatalogSyncRunUpdate): Promise<CatalogSyncRun | undefined> {
const current = this.syncRuns.get(runId);
if (!current) return undefined;
const updated = { ...current, ...structuredClone(update), updatedAt: new Date().toISOString() };
this.syncRuns.set(runId, updated);
return structuredClone(updated);
}
async requestSyncRunCancellation(runId: string): Promise<CatalogSyncRun | undefined> {
const run = this.syncRuns.get(runId);
if (!run) return undefined;
if (!["queued", "running", "awaiting_confirmation"].includes(run.state)) return structuredClone(run);
return await this.updateSyncRun(runId, { cancelRequested: true });
}
async appendSyncEvent(
runId: string,
level: CatalogSyncEvent["level"],
eventType: string,
message: string,
data: Record<string, unknown> = {},
): Promise<CatalogSyncEvent> {
const events = this.syncEvents.get(runId) ?? [];
const event: CatalogSyncEvent = {
id: [...this.syncEvents.values()].reduce((count, values) => count + values.length, 0) + 1,
runId, sequence: events.length + 1, level, eventType, message, data: structuredClone(data),
createdAt: new Date().toISOString(),
};
events.push(event);
this.syncEvents.set(runId, events);
return structuredClone(event);
}
async listSyncEvents(runId: string, afterSequence = 0): Promise<CatalogSyncEvent[]> {
return (this.syncEvents.get(runId) ?? []).filter((event) => event.sequence > afterSequence).map((event) => structuredClone(event));
}
async pruneSyncEvents(before: string): Promise<void> {
for (const [runId, events] of this.syncEvents) {
this.syncEvents.set(runId, events.filter((event) => event.createdAt >= before));
}
}
async interruptActiveSyncRuns(): Promise<void> {
for (const run of this.syncRuns.values()) {
if (["queued", "running", "awaiting_confirmation", "applying"].includes(run.state)) {
await this.updateSyncRun(run.id, {
state: "interrupted", phase: "completed", finishedAt: new Date().toISOString(),
errorCode: "SYNC_INTERRUPTED", errorMessage: "Synchronization was interrupted by a service restart",
});
}
}
}
async reconcileTables(
databaseId: string,
expectedDatabaseVersion: number,
observed: readonly ObservedCatalogTable[],
confirmedDeletedNames: readonly string[],
): Promise<TableSyncRepositoryResult | undefined> {
const database = this.records.get(databaseId);
if (!database || database.version !== expectedDatabaseVersion) return undefined;
const existing = await this.listTables(databaseId);
const observedNames = new Set(observed.map((table) => table.name));
const deletedNames = existing.filter((table) => !observedNames.has(table.name)).map((table) => table.name).sort();
const confirmation = [...new Set(confirmedDeletedNames)].sort();
if (deletedNames.length > 0 && JSON.stringify(deletedNames) !== JSON.stringify(confirmation)) {
return { kind: "confirmation_required", deletedNames };
}
const byName = new Map(existing.map((table) => [table.name, table]));
let createdCount = 0;
let updatedCount = 0;
for (const observedTable of observed) {
const current = byName.get(observedTable.name);
const now = new Date().toISOString();
if (!current) {
const created: CatalogTable = {
id: randomUUID(),
databaseId,
name: observedTable.name,
sourceComment: observedTable.sourceComment,
description: null,
generatedDescription: null,
lastSyncedDatabaseVersion: expectedDatabaseVersion,
lastSyncedAt: now,
version: 1,
createdAt: now,
updatedAt: now,
};
this.tables.set(created.id, created);
createdCount += 1;
} else if (current.sourceComment !== observedTable.sourceComment) {
this.tables.set(current.id, {
...current,
sourceComment: observedTable.sourceComment,
version: current.version + 1,
updatedAt: now,
});
updatedCount += 1;
}
}
for (const table of existing) {
if (deletedNames.includes(table.name)) this.deleteTable(table.id);
}
return {
kind: "applied",
createdCount,
updatedCount,
deletedCount: deletedNames.length,
tables: await this.listTables(databaseId),
};
}
private assertSnapshotCapability(scope: CatalogSyncScope, snapshot: ObservedSchemaSnapshot): void {
const required = scope === "all" ? ["tables", "columns", "relationships"] as const : [scope] as const;
for (const capability of required) {
if (snapshot.capabilities[capability] !== "available") {
throw new CatalogConnectorError(`Schema introspection capability '${capability}' is unavailable`);
}
}
}
private deleteTable(tableId: string): void {
this.tables.delete(tableId);
for (const column of [...this.columns.values()]) if (column.tableId === tableId) this.deleteColumn(column.id);
for (const relationship of [...this.relationships.values()]) {
if (relationship.sourceTableId === tableId || relationship.targetTableId === tableId) {
this.relationships.delete(relationship.id);
}
}
}
private deleteColumn(columnId: string): void {
this.columns.delete(columnId);
for (const relationship of [...this.relationships.values()]) {
if (relationship.columns.some((pair) => pair.sourceColumnId === columnId || pair.targetColumnId === columnId)) {
this.relationships.delete(relationship.id);
}
}
}
private refreshForeignKeyFlags(databaseId: string): void {
const counts = new Map<string, number>();
for (const relationship of this.relationships.values()) {
if (relationship.databaseId !== databaseId) continue;
for (const pair of relationship.columns) counts.set(pair.sourceColumnId, (counts.get(pair.sourceColumnId) ?? 0) + 1);
}
for (const column of this.columns.values()) {
if (this.tables.get(column.tableId)?.databaseId !== databaseId) continue;
const foreignKeyCount = counts.get(column.id) ?? 0;
this.columns.set(column.id, { ...column, isForeignKey: foreignKeyCount > 0, foreignKeyCount });
}
}
async available(): Promise<boolean> { return true; }
}
+51
View File
@@ -0,0 +1,51 @@
import { CamelCasePlugin, Kysely, PostgresDialect } from "kysely";
import { Migrator, type MigrationProvider } from "kysely/migration";
import { Pool } from "pg";
import { readFile } from "node:fs/promises";
import type { CatalogDatabase } from "./repository.js";
import * as initialMigration from "./migrations/001_workspace_databases.js";
import * as catalogTablesMigration from "./migrations/002_catalog_tables.js";
import * as catalogSchemaSyncMigration from "./migrations/003_catalog_schema_sync.js";
import * as catalogRuntimeSequencePrivilegesMigration from "./migrations/004_catalog_runtime_sequence_privileges.js";
const connectionString = process.env.THT_CATALOG_MIGRATOR_DATABASE_URL;
const host = process.env.THT_CATALOG_DB_HOST;
const database = process.env.THT_CATALOG_DB_NAME;
const user = process.env.THT_CATALOG_MIGRATOR_USER;
const passwordFile = process.env.THT_CATALOG_MIGRATOR_PASSWORD_FILE;
if (!connectionString && (!host || !database || !user || !passwordFile)) {
throw new Error("catalog migrator database configuration is required");
}
const pool = new Pool(connectionString ? { connectionString, max: 1 } : {
host,
port: Number(process.env.THT_CATALOG_DB_PORT ?? 5432),
database,
user,
password: async () => (await readFile(passwordFile!, "utf8")).trim(),
max: 1,
});
const db = new Kysely<CatalogDatabase>({
dialect: new PostgresDialect({ pool }),
plugins: [new CamelCasePlugin()],
});
const provider: MigrationProvider = {
async getMigrations() {
return {
"001_workspace_databases": initialMigration,
"002_catalog_tables": catalogTablesMigration,
"003_catalog_schema_sync": catalogSchemaSyncMigration,
"004_catalog_runtime_sequence_privileges": catalogRuntimeSequencePrivilegesMigration,
};
},
};
try {
const result = await new Migrator({ db, provider }).migrateToLatest();
for (const item of result.results ?? []) {
process.stdout.write(`${item.migrationName}: ${item.status}\n`);
}
if (result.error) throw result.error;
} finally {
await db.destroy();
}
@@ -0,0 +1,58 @@
import { sql, type Kysely } from "kysely";
import type { CatalogDatabase } from "../repository.js";
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.createTable("workspace_databases")
.addColumn("id", "uuid", (column) => column.primaryKey())
.addColumn("workspace_id", "text", (column) => column.notNull().unique())
.addColumn("engine", "text", (column) => column.notNull())
.addColumn("database_name", "text", (column) => column.notNull())
.addColumn("schema_name", "text", (column) => column.notNull())
.addColumn("version", "integer", (column) => column.notNull().defaultTo(1))
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addCheckConstraint("workspace_databases_engine_check", sql`engine = 'postgres'`)
.addCheckConstraint("workspace_databases_version_check", sql`version > 0`)
.execute();
await db.schema.createTable("database_bindings")
.addColumn("database_id", "uuid", (column) => column.primaryKey()
.references("workspace_databases.id").onDelete("cascade"))
.addColumn("transport", "text", (column) => column.notNull())
.addColumn("host", "text")
.addColumn("port", "integer")
.addColumn("username", "text")
.addColumn("base_url", "text")
.addColumn("rest_path", "text")
.addColumn("rest_auth", "text")
.addColumn("tls_servername", "text")
.addColumn("ssh_host", "text")
.addColumn("ssh_port", "integer")
.addColumn("ssh_username", "text")
.addColumn("ssh_target_host", "text")
.addColumn("ssh_target_port", "integer")
.addColumn("connection_status", "text", (column) => column.notNull().defaultTo("untested"))
.addColumn("tested_version", "integer")
.addColumn("last_tested_at", "timestamptz")
.addColumn("last_error_code", "text")
.addColumn("last_error_message", "text")
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addCheckConstraint("database_bindings_transport_check", sql`transport in ('postgres_direct', 'rest_api', 'ssh_tunnel')`)
.addCheckConstraint("database_bindings_status_check", sql`connection_status in ('untested', 'reachable', 'failed')`)
.addCheckConstraint("database_bindings_port_check", sql`port is null or port between 1 and 65535`)
.addCheckConstraint("database_bindings_ssh_port_check", sql`ssh_port is null or ssh_port between 1 and 65535`)
.addCheckConstraint("database_bindings_ssh_target_port_check", sql`ssh_target_port is null or ssh_target_port between 1 and 65535`)
.addCheckConstraint("database_bindings_transport_fields_check", sql`
(transport = 'postgres_direct' and host is not null and port is not null and username is not null)
or (transport = 'rest_api' and base_url is not null and rest_path is not null and rest_auth is not null)
or (transport = 'ssh_tunnel' and username is not null and ssh_host is not null
and ssh_port is not null and ssh_username is not null and ssh_target_host is not null
and ssh_target_port is not null)
`)
.execute();
}
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.dropTable("database_bindings").execute();
await db.schema.dropTable("workspace_databases").execute();
}
@@ -0,0 +1,27 @@
import { sql, type Kysely } from "kysely";
import type { CatalogDatabase } from "../repository.js";
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.createTable("catalog_tables")
.addColumn("id", "uuid", (column) => column.primaryKey())
.addColumn("database_id", "uuid", (column) => column.notNull()
.references("workspace_databases.id").onDelete("cascade"))
.addColumn("name", "text", (column) => column.notNull())
.addColumn("source_comment", "text")
.addColumn("description", "text")
.addColumn("generated_description", "text")
.addColumn("version", "integer", (column) => column.notNull().defaultTo(1))
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addUniqueConstraint("catalog_tables_database_name_key", ["database_id", "name"])
.addCheckConstraint("catalog_tables_name_check", sql`char_length(name) between 1 and 128`)
.addCheckConstraint("catalog_tables_version_check", sql`version > 0`)
.execute();
await db.schema.createIndex("catalog_tables_database_id_idx")
.on("catalog_tables").column("database_id").execute();
}
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.dropTable("catalog_tables").execute();
}
@@ -0,0 +1,136 @@
import { sql, type Kysely } from "kysely";
import type { CatalogDatabase } from "../repository.js";
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.alterTable("workspace_databases")
.addColumn("schema_synced_version", "integer")
.addColumn("schema_synced_at", "timestamptz")
.execute();
await db.schema.alterTable("catalog_tables")
.addColumn("last_synced_database_version", "integer")
.addColumn("last_synced_at", "timestamptz")
.execute();
await db.schema.createTable("catalog_columns")
.addColumn("id", "uuid", (column) => column.primaryKey())
.addColumn("table_id", "uuid", (column) => column.notNull()
.references("catalog_tables.id").onDelete("cascade"))
.addColumn("name", "text", (column) => column.notNull())
.addColumn("ordinal_position", "integer", (column) => column.notNull())
.addColumn("data_type", "text", (column) => column.notNull())
.addColumn("is_nullable", "boolean", (column) => column.notNull())
.addColumn("default_expression", "text")
.addColumn("primary_key_position", "integer")
.addColumn("source_comment", "text")
.addColumn("description", "text")
.addColumn("generated_description", "text")
.addColumn("last_synced_database_version", "integer")
.addColumn("last_synced_at", "timestamptz")
.addColumn("version", "integer", (column) => column.notNull().defaultTo(1))
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addUniqueConstraint("catalog_columns_table_name_key", ["table_id", "name"])
.addCheckConstraint("catalog_columns_name_check", sql`char_length(name) between 1 and 128`)
.addCheckConstraint("catalog_columns_ordinal_check", sql`ordinal_position > 0`)
.addCheckConstraint("catalog_columns_pk_position_check", sql`primary_key_position is null or primary_key_position > 0`)
.addCheckConstraint("catalog_columns_version_check", sql`version > 0`)
.execute();
await db.schema.createIndex("catalog_columns_table_id_idx")
.on("catalog_columns").column("table_id").execute();
await db.schema.createTable("catalog_relationships")
.addColumn("id", "uuid", (column) => column.primaryKey())
.addColumn("database_id", "uuid", (column) => column.notNull()
.references("workspace_databases.id").onDelete("cascade"))
.addColumn("constraint_name", "text", (column) => column.notNull())
.addColumn("source_table_id", "uuid", (column) => column.notNull()
.references("catalog_tables.id").onDelete("cascade"))
.addColumn("target_table_id", "uuid", (column) => column.notNull()
.references("catalog_tables.id").onDelete("cascade"))
.addColumn("update_rule", "text", (column) => column.notNull())
.addColumn("delete_rule", "text", (column) => column.notNull())
.addColumn("deferrable", "boolean", (column) => column.notNull().defaultTo(false))
.addColumn("initially_deferred", "boolean", (column) => column.notNull().defaultTo(false))
.addColumn("last_synced_database_version", "integer")
.addColumn("last_synced_at", "timestamptz")
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addUniqueConstraint("catalog_relationships_source_constraint_key", ["source_table_id", "constraint_name"])
.execute();
await db.schema.createIndex("catalog_relationships_database_id_idx")
.on("catalog_relationships").column("database_id").execute();
await db.schema.createTable("catalog_relationship_columns")
.addColumn("relationship_id", "uuid", (column) => column.notNull()
.references("catalog_relationships.id").onDelete("cascade"))
.addColumn("position", "integer", (column) => column.notNull())
.addColumn("source_column_id", "uuid", (column) => column.notNull()
.references("catalog_columns.id").onDelete("cascade"))
.addColumn("target_column_id", "uuid", (column) => column.notNull()
.references("catalog_columns.id").onDelete("cascade"))
.addPrimaryKeyConstraint("catalog_relationship_columns_pkey", ["relationship_id", "position"])
.addCheckConstraint("catalog_relationship_columns_position_check", sql`position > 0`)
.execute();
await db.schema.createTable("catalog_sync_runs")
.addColumn("id", "uuid", (column) => column.primaryKey())
.addColumn("database_id", "uuid", (column) => column.notNull()
.references("workspace_databases.id").onDelete("cascade"))
.addColumn("scope", "text", (column) => column.notNull())
.addColumn("table_ids", "jsonb", (column) => column.notNull().defaultTo(sql`'[]'::jsonb`))
.addColumn("state", "text", (column) => column.notNull())
.addColumn("phase", "text", (column) => column.notNull())
.addColumn("requested_database_version", "integer", (column) => column.notNull())
.addColumn("observed_snapshot", "jsonb")
.addColumn("planned_diff", "jsonb")
.addColumn("confirmation_token", "text")
.addColumn("counts", "jsonb", (column) => column.notNull().defaultTo(sql`'{}'::jsonb`))
.addColumn("error_code", "text")
.addColumn("error_message", "text")
.addColumn("cancel_requested", "boolean", (column) => column.notNull().defaultTo(false))
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addColumn("started_at", "timestamptz")
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addColumn("finished_at", "timestamptz")
.addColumn("heartbeat_at", "timestamptz")
.addColumn("lease_owner", "text")
.addColumn("lease_expires_at", "timestamptz")
.execute();
await db.schema.createIndex("catalog_sync_runs_database_created_idx")
.on("catalog_sync_runs").columns(["database_id", "created_at"]).execute();
await sql`CREATE UNIQUE INDEX catalog_sync_runs_one_active_per_database
ON catalog_sync_runs (database_id)
WHERE state IN ('queued', 'running', 'awaiting_confirmation', 'applying')`.execute(db);
await db.schema.createTable("catalog_sync_events")
.addColumn("id", "bigserial", (column) => column.primaryKey())
.addColumn("run_id", "uuid", (column) => column.notNull()
.references("catalog_sync_runs.id").onDelete("cascade"))
.addColumn("sequence", "integer", (column) => column.notNull())
.addColumn("level", "text", (column) => column.notNull())
.addColumn("event_type", "text", (column) => column.notNull())
.addColumn("message", "text", (column) => column.notNull())
.addColumn("data", "jsonb", (column) => column.notNull().defaultTo(sql`'{}'::jsonb`))
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
.addUniqueConstraint("catalog_sync_events_run_sequence_key", ["run_id", "sequence"])
.execute();
await db.schema.createIndex("catalog_sync_events_run_id_idx")
.on("catalog_sync_events").columns(["run_id", "sequence"]).execute();
}
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
await db.schema.dropTable("catalog_sync_events").execute();
await db.schema.dropTable("catalog_sync_runs").execute();
await db.schema.dropTable("catalog_relationship_columns").execute();
await db.schema.dropTable("catalog_relationships").execute();
await db.schema.dropTable("catalog_columns").execute();
await db.schema.alterTable("catalog_tables")
.dropColumn("last_synced_database_version")
.dropColumn("last_synced_at")
.execute();
await db.schema.alterTable("workspace_databases")
.dropColumn("schema_synced_version")
.dropColumn("schema_synced_at")
.execute();
}
@@ -0,0 +1,29 @@
import { type Kysely, sql } from "kysely";
import type { CatalogDatabase } from "../repository.js";
/**
* `catalog_sync_events.id` is the first catalog-owned identity sequence.
* Table default privileges do not cover sequences, and without USAGE the
* runtime can create a run but cannot append its first event.
*/
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
await sql`DO $catalog_privileges$
BEGIN
IF EXISTS (SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = 'thothii_catalog_runtime') THEN
EXECUTE 'GRANT USAGE, SELECT ON SEQUENCE catalog_sync_events_id_seq TO thothii_catalog_runtime';
EXECUTE 'ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO thothii_catalog_runtime';
END IF;
END
$catalog_privileges$`.execute(db);
}
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
await sql`DO $catalog_privileges$
BEGIN
IF EXISTS (SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = 'thothii_catalog_runtime') THEN
EXECUTE 'ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE USAGE, SELECT ON SEQUENCES FROM thothii_catalog_runtime';
EXECUTE 'REVOKE USAGE, SELECT ON SEQUENCE catalog_sync_events_id_seq FROM thothii_catalog_runtime';
END IF;
END
$catalog_privileges$`.execute(db);
}
@@ -0,0 +1,28 @@
import { CatalogOperationInProgressError } from "./types.js";
/** Serializes connection tests and schema synchronization for each catalog database. */
export class CatalogOperationCoordinator {
private readonly active = new Set<string>();
reserve(databaseId: string): () => void {
if (this.active.has(databaseId)) {
throw new CatalogOperationInProgressError("A database operation is already in progress");
}
this.active.add(databaseId);
let released = false;
return () => {
if (released) return;
released = true;
this.active.delete(databaseId);
};
}
async run<T>(databaseId: string, operation: () => Promise<T>): Promise<T> {
const release = this.reserve(databaseId);
try {
return await operation();
} finally {
release();
}
}
}
+258
View File
@@ -0,0 +1,258 @@
import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process";
import { readFile } from "node:fs/promises";
import { fileURLToPath } from "node:url";
import { Duplex } from "node:stream";
import { setTimeout as delay } from "node:timers/promises";
import { Client, type ClientConfig } from "pg";
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
import { CATALOG_SECRET_IDS } from "./secrets.js";
import { CatalogConnectorError, type WorkspaceDatabase } from "./types.js";
export interface CatalogDatabaseClient {
query(sql: string, values: readonly unknown[]): Promise<{ rows: Array<Record<string, unknown>> }>;
end(): Promise<void>;
}
export interface CatalogPostgresAccess {
connect(database: WorkspaceDatabase, signal: AbortSignal): Promise<CatalogDatabaseClient>;
}
type SpawnSsh = (
command: string,
args: readonly string[],
options: { env: NodeJS.ProcessEnv },
) => ChildProcessWithoutNullStreams;
interface AccessDependencies {
createClient?: (config: ClientConfig) => Client;
spawnSsh?: SpawnSsh;
sshBinary?: string;
askpassPath?: string;
connectTimeoutMs?: number;
}
function required(value: string | number | undefined): string | number {
if (value === undefined || value === "") throw new CatalogConnectorError("Database binding is incomplete");
return value;
}
function connectionSsl(ca: string | undefined, servername: string | undefined): ClientConfig["ssl"] {
if (!ca && !servername) return false;
return {
...(ca ? { ca } : {}),
...(servername ? { servername } : {}),
rejectUnauthorized: true,
};
}
export function buildSshArguments(input: {
sshHost: string;
sshPort: number;
sshUsername: string;
targetHost: string;
targetPort: number;
privateKeyFile: string;
knownHostsFile: string;
passphraseFile?: string;
connectTimeoutMs: number;
}): string[] {
const batchMode = input.passphraseFile ? "no" : "yes";
return [
"-F", "/dev/null",
"-T",
"-o", `BatchMode=${batchMode}`,
"-o", "StrictHostKeyChecking=yes",
"-o", `UserKnownHostsFile=${input.knownHostsFile}`,
"-o", "GlobalKnownHostsFile=/dev/null",
"-o", "IdentitiesOnly=yes",
"-o", "IdentityAgent=none",
"-o", `IdentityFile=${input.privateKeyFile}`,
"-o", "PreferredAuthentications=publickey",
"-o", "PasswordAuthentication=no",
"-o", "KbdInteractiveAuthentication=no",
"-o", "ConnectionAttempts=1",
"-o", `ConnectTimeout=${Math.max(1, Math.ceil(input.connectTimeoutMs / 1_000))}`,
"-o", "ServerAliveInterval=5",
"-o", "ServerAliveCountMax=1",
"-o", "NumberOfPasswordPrompts=1",
"-o", "RequestTTY=no",
"-o", "LogLevel=ERROR",
"-p", String(input.sshPort),
"-W", `${input.targetHost}:${input.targetPort}`,
"--", `${input.sshUsername}@${input.sshHost}`,
];
}
async function stopChild(child: ChildProcessWithoutNullStreams): Promise<void> {
if (child.exitCode !== null || child.signalCode !== null) return;
child.kill("SIGTERM");
await Promise.race([
new Promise<void>((resolve) => child.once("exit", () => resolve())),
delay(500).then(() => undefined),
]);
if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL");
}
function sshDuplex(child: ChildProcessWithoutNullStreams): Duplex {
let ended = false;
let stream: Duplex;
const forward = () => {
let chunk: Buffer | string | null;
while ((chunk = child.stdout.read() as Buffer | string | null) !== null) {
if (!stream.push(chunk)) break;
}
};
const finish = () => {
if (ended) return;
ended = true;
stream.push(null);
};
const fail = (error: Error) => stream.destroy(error);
stream = new Duplex({
read: forward,
write: (chunk, encoding, callback) => child.stdin.write(chunk, encoding, callback),
final: (callback) => child.stdin.end(callback),
destroy: (error, callback) => {
child.stdout.off("readable", forward);
child.stdout.off("end", finish);
child.stdout.off("error", fail);
child.stdin.off("error", fail);
callback(error);
},
});
child.stdout.on("readable", forward);
child.stdout.once("end", finish);
child.stdout.once("error", fail);
child.stdin.once("error", fail);
return stream;
}
/**
* Deep connection module for direct and SSH-forwarded PostgreSQL access. It owns secret leases,
* TLS, OpenSSH lifecycle, abort propagation, and pg cleanup behind one connect interface.
*/
export class ConcreteCatalogPostgresAccess implements CatalogPostgresAccess {
private readonly createClient: (config: ClientConfig) => Client;
private readonly spawnSsh: SpawnSsh;
private readonly sshBinary: string;
private readonly askpassPath: string;
private readonly connectTimeoutMs: number;
constructor(
private readonly secretStore: WorkspaceSecretStore,
dependencies: AccessDependencies = {},
) {
this.createClient = dependencies.createClient ?? ((config) => new Client(config));
this.spawnSsh = dependencies.spawnSsh ?? ((command, args, options) => (
spawn(command, [...args], { ...options, stdio: ["pipe", "pipe", "pipe"] })
));
this.sshBinary = dependencies.sshBinary ?? process.env.THT_SSH_BIN ?? "ssh";
this.askpassPath = dependencies.askpassPath
?? process.env.THT_SSH_ASKPASS_BIN
?? fileURLToPath(new URL("../../scripts/ssh-askpass.mjs", import.meta.url));
this.connectTimeoutMs = dependencies.connectTimeoutMs ?? 5_000;
}
async connect(database: WorkspaceDatabase, signal: AbortSignal): Promise<CatalogDatabaseClient> {
if (database.binding.transport === "rest_api") {
throw new CatalogConnectorError("REST is not a PostgreSQL wire binding");
}
const ids: string[] = [CATALOG_SECRET_IDS.password, CATALOG_SECRET_IDS.tlsCa];
if (database.binding.transport === "ssh_tunnel") {
ids.push(
CATALOG_SECRET_IDS.sshPrivateKey,
CATALOG_SECRET_IDS.sshPrivateKeyPassphrase,
CATALOG_SECRET_IDS.sshKnownHosts,
);
}
const materialized = this.secretStore.materialize(database.workspaceId, ids);
let child: ChildProcessWithoutNullStreams | undefined;
let stream: Duplex | undefined;
let client: Client | undefined;
let ended = false;
const close = async () => {
if (ended) return;
ended = true;
signal.removeEventListener("abort", abort);
if (client) await client.end().catch(() => undefined);
stream?.destroy();
if (child) await stopChild(child);
materialized.release();
};
const abort = () => { void close(); };
try {
const passwordFile = materialized.files.get(CATALOG_SECRET_IDS.password);
if (!passwordFile) throw new CatalogConnectorError("Database password is not configured");
const password = await readFile(passwordFile, "utf8");
const tlsCaFile = materialized.files.get(CATALOG_SECRET_IDS.tlsCa);
const tlsCa = tlsCaFile ? await readFile(tlsCaFile, "utf8") : undefined;
let host: string;
let port: number;
if (database.binding.transport === "ssh_tunnel") {
const privateKeyFile = materialized.files.get(CATALOG_SECRET_IDS.sshPrivateKey);
const knownHostsFile = materialized.files.get(CATALOG_SECRET_IDS.sshKnownHosts);
if (!privateKeyFile || !knownHostsFile) {
throw new CatalogConnectorError("SSH private key and known hosts are required");
}
const passphraseFile = materialized.files.get(CATALOG_SECRET_IDS.sshPrivateKeyPassphrase);
host = String(required(database.binding.sshTargetHost));
port = Number(required(database.binding.sshTargetPort));
const args = buildSshArguments({
sshHost: String(required(database.binding.sshHost)),
sshPort: Number(required(database.binding.sshPort)),
sshUsername: String(required(database.binding.sshUsername)),
targetHost: host,
targetPort: port,
privateKeyFile,
knownHostsFile,
passphraseFile,
connectTimeoutMs: this.connectTimeoutMs,
});
child = this.spawnSsh(this.sshBinary, args, {
env: {
...process.env,
LC_ALL: "C",
...(passphraseFile ? {
DISPLAY: "thothii",
SSH_ASKPASS: this.askpassPath,
SSH_ASKPASS_REQUIRE: "force",
THT_SSH_PASSPHRASE_FILE: passphraseFile,
} : {}),
},
});
stream = sshDuplex(child);
child.once("error", () => stream?.destroy(new CatalogConnectorError("SSH process failed")));
child.once("exit", (code) => {
if (!ended && code !== 0) stream?.destroy(new CatalogConnectorError("SSH tunnel failed"));
});
child.stderr.on("data", () => undefined);
} else {
host = String(required(database.binding.host));
port = Number(required(database.binding.port));
}
client = this.createClient({
host,
port,
database: database.databaseName,
user: String(required(database.binding.username)),
password,
ssl: connectionSsl(tlsCa, database.binding.tlsServername),
connectionTimeoutMillis: this.connectTimeoutMs,
...(stream ? { stream: () => stream } : {}),
});
signal.addEventListener("abort", abort, { once: true });
await client.connect();
return {
query: async (sql, values) => await client!.query(sql, [...values]),
end: close,
};
} catch (error) {
await close();
if (error instanceof CatalogConnectorError) throw error;
throw new CatalogConnectorError("PostgreSQL connector failed");
}
}
}
File diff suppressed because it is too large Load Diff
+498
View File
@@ -0,0 +1,498 @@
import { readFile } from "node:fs/promises";
import { z } from "zod";
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
import type { CatalogPostgresAccess } from "./postgres-access.js";
import { CATALOG_SECRET_IDS } from "./secrets.js";
import {
CatalogConnectorError,
CatalogSchemaCapabilityUnavailableError,
type CatalogSyncPhase,
type ObservedCatalogColumn,
type ObservedCatalogRelationship,
type ObservedCatalogTable,
type ObservedSchemaSnapshot,
type WorkspaceDatabase,
} from "./types.js";
export type CatalogSchemaScanProgress = (
phase: Extract<CatalogSyncPhase, "connecting" | "scanning_tables" | "scanning_columns" | "scanning_relationships">,
counts?: { tables?: number; columns?: number; relationships?: number },
) => Promise<void> | void;
export interface CatalogSchemaIntrospector {
scan(
database: WorkspaceDatabase,
signal: AbortSignal,
progress?: CatalogSchemaScanProgress,
): Promise<ObservedSchemaSnapshot>;
}
const identifier = z.string().min(1).max(128);
const nullableText = z.string().nullable();
const capability = z.enum(["available", "unavailable"]);
const restSnapshotSchema = z.object({
schemaVersion: z.literal(1),
capabilities: z.object({
tables: capability,
columns: capability,
relationships: capability,
}).strict(),
tables: z.array(z.object({
name: identifier,
sourceComment: nullableText,
}).strict()),
columns: z.array(z.object({
tableName: identifier,
name: identifier,
ordinalPosition: z.number().int().positive(),
dataType: z.string().min(1).max(2_000),
isNullable: z.boolean(),
defaultExpression: nullableText,
primaryKeyPosition: z.number().int().positive().nullable(),
sourceComment: nullableText,
}).strict()),
relationships: z.array(z.object({
constraintName: identifier,
sourceTableName: identifier,
targetTableName: identifier,
updateRule: z.string().min(1).max(64),
deleteRule: z.string().min(1).max(64),
deferrable: z.boolean(),
initiallyDeferred: z.boolean(),
columns: z.array(z.object({
position: z.number().int().positive(),
sourceColumnName: identifier,
targetColumnName: identifier,
}).strict()).min(1),
}).strict()),
}).strict();
function sqlString(value: string): string {
return `'${value.replaceAll("'", "''")}'`;
}
function restSnapshotQuery(schemaName: string): string {
const schema = sqlString(schemaName);
return `WITH target_schema AS (
SELECT oid
FROM pg_catalog.pg_namespace
WHERE nspname = ${schema}
),
observed_tables AS (
SELECT c.oid,
c.relname AS name,
d.description AS source_comment
FROM pg_catalog.pg_class c
JOIN target_schema n ON n.oid = c.relnamespace
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = 0
WHERE c.relkind IN ('r', 'p')
),
primary_key_columns AS (
SELECT i.indrelid AS table_oid,
key.attnum,
key.ordinality::integer AS position
FROM pg_catalog.pg_index i
CROSS JOIN LATERAL unnest(i.indkey) WITH ORDINALITY AS key(attnum, ordinality)
WHERE i.indisprimary
),
observed_columns AS (
SELECT table_info.name AS table_name,
a.attname AS name,
a.attnum::integer AS ordinal_position,
pg_catalog.format_type(a.atttypid, a.atttypmod) AS data_type,
NOT a.attnotnull AS is_nullable,
pg_catalog.pg_get_expr(ad.adbin, ad.adrelid) AS default_expression,
pk.position AS primary_key_position,
d.description AS source_comment
FROM observed_tables table_info
JOIN pg_catalog.pg_attribute a ON a.attrelid = table_info.oid
LEFT JOIN pg_catalog.pg_attrdef ad ON ad.adrelid = table_info.oid AND ad.adnum = a.attnum
LEFT JOIN pg_catalog.pg_description d ON d.objoid = table_info.oid AND d.objsubid = a.attnum
LEFT JOIN primary_key_columns pk ON pk.table_oid = table_info.oid AND pk.attnum = a.attnum
WHERE a.attnum > 0
AND NOT a.attisdropped
),
relationship_pairs AS (
SELECT con.oid AS constraint_oid,
con.conname AS constraint_name,
source_table.relname AS source_table_name,
target_table.relname AS target_table_name,
CASE con.confupdtype
WHEN 'a' THEN 'NO ACTION'
WHEN 'r' THEN 'RESTRICT'
WHEN 'c' THEN 'CASCADE'
WHEN 'n' THEN 'SET NULL'
WHEN 'd' THEN 'SET DEFAULT'
END AS update_rule,
CASE con.confdeltype
WHEN 'a' THEN 'NO ACTION'
WHEN 'r' THEN 'RESTRICT'
WHEN 'c' THEN 'CASCADE'
WHEN 'n' THEN 'SET NULL'
WHEN 'd' THEN 'SET DEFAULT'
END AS delete_rule,
con.condeferrable AS is_deferrable,
con.condeferred AS initially_deferred,
source_key.ordinality::integer AS position,
source_column.attname AS source_column_name,
target_column.attname AS target_column_name
FROM pg_catalog.pg_constraint con
JOIN pg_catalog.pg_class source_table ON source_table.oid = con.conrelid
JOIN target_schema source_namespace ON source_namespace.oid = source_table.relnamespace
JOIN pg_catalog.pg_class target_table ON target_table.oid = con.confrelid
JOIN target_schema target_namespace ON target_namespace.oid = target_table.relnamespace
JOIN LATERAL unnest(con.conkey) WITH ORDINALITY AS source_key(attnum, ordinality) ON true
JOIN LATERAL unnest(con.confkey) WITH ORDINALITY AS target_key(attnum, ordinality)
ON target_key.ordinality = source_key.ordinality
JOIN pg_catalog.pg_attribute source_column
ON source_column.attrelid = source_table.oid AND source_column.attnum = source_key.attnum
JOIN pg_catalog.pg_attribute target_column
ON target_column.attrelid = target_table.oid AND target_column.attnum = target_key.attnum
WHERE con.contype = 'f'
),
observed_relationships AS (
SELECT constraint_oid,
constraint_name,
source_table_name,
target_table_name,
update_rule,
delete_rule,
is_deferrable,
initially_deferred,
pg_catalog.jsonb_agg(
pg_catalog.jsonb_build_object(
'position', position,
'sourceColumnName', source_column_name,
'targetColumnName', target_column_name
) ORDER BY position
) AS columns
FROM relationship_pairs
GROUP BY constraint_oid, constraint_name, source_table_name, target_table_name,
update_rule, delete_rule, is_deferrable, initially_deferred
)
SELECT 1 AS "schemaVersion",
pg_catalog.jsonb_build_object(
'tables', 'available',
'columns', 'available',
'relationships', 'available'
) AS capabilities,
COALESCE((
SELECT pg_catalog.jsonb_agg(
pg_catalog.jsonb_build_object('name', name, 'sourceComment', source_comment)
ORDER BY name
)
FROM observed_tables
), '[]'::jsonb) AS tables,
COALESCE((
SELECT pg_catalog.jsonb_agg(
pg_catalog.jsonb_build_object(
'tableName', table_name,
'name', name,
'ordinalPosition', ordinal_position,
'dataType', data_type,
'isNullable', is_nullable,
'defaultExpression', default_expression,
'primaryKeyPosition', primary_key_position,
'sourceComment', source_comment
) ORDER BY table_name, ordinal_position
)
FROM observed_columns
), '[]'::jsonb) AS columns,
COALESCE((
SELECT pg_catalog.jsonb_agg(
pg_catalog.jsonb_build_object(
'constraintName', constraint_name,
'sourceTableName', source_table_name,
'targetTableName', target_table_name,
'updateRule', update_rule,
'deleteRule', delete_rule,
'deferrable', is_deferrable,
'initiallyDeferred', initially_deferred,
'columns', columns
) ORDER BY source_table_name, constraint_name
)
FROM observed_relationships
), '[]'::jsonb) AS relationships
FROM target_schema`;
}
function required(value: string | undefined): string {
if (!value) throw new CatalogConnectorError("Database binding is incomplete");
return value;
}
function textOrNull(value: unknown): string | null {
return typeof value === "string" && value.length > 0 ? value : null;
}
function actionRule(value: unknown): string {
const rules: Record<string, string> = {
a: "NO ACTION",
r: "RESTRICT",
c: "CASCADE",
n: "SET NULL",
d: "SET DEFAULT",
};
const rule = rules[String(value)];
if (!rule) throw new CatalogConnectorError("Schema introspection returned an unknown relationship action");
return rule;
}
function normalized(snapshot: ObservedSchemaSnapshot): ObservedSchemaSnapshot {
const tables = new Map<string, ObservedCatalogTable>();
for (const table of snapshot.tables) {
if (tables.has(table.name)) throw new CatalogConnectorError("Schema introspection returned duplicate tables");
tables.set(table.name, table);
}
const columns = new Map<string, ObservedCatalogColumn>();
for (const column of snapshot.columns) {
const key = `${column.tableName}\u0000${column.name}`;
if (columns.has(key)) throw new CatalogConnectorError("Schema introspection returned duplicate columns");
columns.set(key, column);
}
const relationships = new Map<string, ObservedCatalogRelationship>();
for (const relationship of snapshot.relationships) {
const key = `${relationship.sourceTableName}\u0000${relationship.constraintName}`;
if (relationships.has(key)) throw new CatalogConnectorError("Schema introspection returned duplicate relationships");
relationships.set(key, {
...relationship,
columns: [...relationship.columns].sort((a, b) => a.position - b.position),
});
}
return {
schemaVersion: 1,
capabilities: snapshot.capabilities,
tables: [...tables.values()].sort((a, b) => a.name.localeCompare(b.name)),
columns: [...columns.values()].sort((a, b) => (
a.tableName.localeCompare(b.tableName) || a.ordinalPosition - b.ordinalPosition
)),
relationships: [...relationships.values()].sort((a, b) => (
a.sourceTableName.localeCompare(b.sourceTableName) || a.constraintName.localeCompare(b.constraintName)
)),
};
}
export class ConcreteCatalogSchemaIntrospector implements CatalogSchemaIntrospector {
constructor(
private readonly postgres: CatalogPostgresAccess,
private readonly secretStore: WorkspaceSecretStore,
) {}
async scan(
database: WorkspaceDatabase,
signal: AbortSignal,
progress?: CatalogSchemaScanProgress,
): Promise<ObservedSchemaSnapshot> {
return database.binding.transport === "rest_api"
? await this.scanRest(database, signal, progress)
: await this.scanPostgres(database, signal, progress);
}
private async scanPostgres(
database: WorkspaceDatabase,
signal: AbortSignal,
progress?: CatalogSchemaScanProgress,
): Promise<ObservedSchemaSnapshot> {
await progress?.("connecting");
const client = await this.postgres.connect(database, signal);
try {
const schema = await client.query(
"SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname = $1) AS present",
[database.schema],
);
if (schema.rows[0]?.present !== true) throw new CatalogConnectorError("Database schema is unavailable");
await progress?.("scanning_tables");
const tableResult = await client.query(
`SELECT c.relname AS name, d.description AS source_comment
FROM pg_catalog.pg_class c
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = 0
WHERE c.relkind IN ('r', 'p') AND n.nspname = $1
ORDER BY c.relname`,
[database.schema],
);
const tables: ObservedCatalogTable[] = tableResult.rows.map((row) => ({
name: String(row.name),
sourceComment: textOrNull(row.source_comment),
}));
await progress?.("scanning_tables", { tables: tables.length });
await progress?.("scanning_columns", { tables: tables.length });
const columnResult = await client.query(
`SELECT c.relname AS table_name,
a.attname AS name,
a.attnum::integer AS ordinal_position,
pg_catalog.format_type(a.atttypid, a.atttypmod) AS data_type,
NOT a.attnotnull AS is_nullable,
pg_catalog.pg_get_expr(ad.adbin, ad.adrelid) AS default_expression,
pk.position AS primary_key_position,
d.description AS source_comment
FROM pg_catalog.pg_class c
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid
LEFT JOIN pg_catalog.pg_attrdef ad ON ad.adrelid = c.oid AND ad.adnum = a.attnum
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = a.attnum
LEFT JOIN LATERAL (
SELECT key.ordinality::integer AS position
FROM pg_catalog.pg_index i
CROSS JOIN LATERAL unnest(i.indkey) WITH ORDINALITY AS key(attnum, ordinality)
WHERE i.indrelid = c.oid AND i.indisprimary AND key.attnum = a.attnum
LIMIT 1
) pk ON true
WHERE c.relkind IN ('r', 'p')
AND n.nspname = $1
AND a.attnum > 0
AND NOT a.attisdropped
ORDER BY c.relname, a.attnum`,
[database.schema],
);
const columns: ObservedCatalogColumn[] = columnResult.rows.map((row) => ({
tableName: String(row.table_name),
name: String(row.name),
ordinalPosition: Number(row.ordinal_position),
dataType: String(row.data_type),
isNullable: row.is_nullable === true,
defaultExpression: textOrNull(row.default_expression),
primaryKeyPosition: row.primary_key_position === null || row.primary_key_position === undefined
? null
: Number(row.primary_key_position),
sourceComment: textOrNull(row.source_comment),
}));
await progress?.("scanning_columns", { tables: tables.length, columns: columns.length });
await progress?.("scanning_relationships", { tables: tables.length, columns: columns.length });
const relationshipResult = await client.query(
`SELECT con.conname AS constraint_name,
source_table.relname AS source_table_name,
target_table.relname AS target_table_name,
con.confupdtype AS update_action,
con.confdeltype AS delete_action,
con.condeferrable AS deferrable,
con.condeferred AS initially_deferred,
source_key.ordinality::integer AS position,
source_column.attname AS source_column_name,
target_column.attname AS target_column_name
FROM pg_catalog.pg_constraint con
JOIN pg_catalog.pg_class source_table ON source_table.oid = con.conrelid
JOIN pg_catalog.pg_namespace source_namespace ON source_namespace.oid = source_table.relnamespace
JOIN pg_catalog.pg_class target_table ON target_table.oid = con.confrelid
JOIN pg_catalog.pg_namespace target_namespace ON target_namespace.oid = target_table.relnamespace
JOIN LATERAL unnest(con.conkey) WITH ORDINALITY AS source_key(attnum, ordinality) ON true
JOIN LATERAL unnest(con.confkey) WITH ORDINALITY AS target_key(attnum, ordinality)
ON target_key.ordinality = source_key.ordinality
JOIN pg_catalog.pg_attribute source_column
ON source_column.attrelid = source_table.oid AND source_column.attnum = source_key.attnum
JOIN pg_catalog.pg_attribute target_column
ON target_column.attrelid = target_table.oid AND target_column.attnum = target_key.attnum
WHERE con.contype = 'f'
AND source_namespace.nspname = $1
AND target_namespace.nspname = $1
ORDER BY source_table.relname, con.conname, source_key.ordinality`,
[database.schema],
);
const relationshipMap = new Map<string, ObservedCatalogRelationship>();
for (const row of relationshipResult.rows) {
const sourceTableName = String(row.source_table_name);
const constraintName = String(row.constraint_name);
const key = `${sourceTableName}\u0000${constraintName}`;
const current = relationshipMap.get(key) ?? {
constraintName,
sourceTableName,
targetTableName: String(row.target_table_name),
updateRule: actionRule(row.update_action),
deleteRule: actionRule(row.delete_action),
deferrable: row.deferrable === true,
initiallyDeferred: row.initially_deferred === true,
columns: [],
};
current.columns.push({
position: Number(row.position),
sourceColumnName: String(row.source_column_name),
targetColumnName: String(row.target_column_name),
});
relationshipMap.set(key, current);
}
const relationships = [...relationshipMap.values()];
await progress?.("scanning_relationships", {
tables: tables.length,
columns: columns.length,
relationships: relationships.length,
});
return normalized({
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables,
columns,
relationships,
});
} finally {
await client.end();
}
}
private async scanRest(
database: WorkspaceDatabase,
signal: AbortSignal,
progress?: CatalogSchemaScanProgress,
): Promise<ObservedSchemaSnapshot> {
await progress?.("connecting");
const auth = database.binding.restAuth ?? "bearer";
const materialized = this.secretStore.materialize(
database.workspaceId,
auth === "none" ? [] : [CATALOG_SECRET_IDS.apiKey],
);
try {
const headers: Record<string, string> = { "content-type": "application/json" };
if (auth !== "none") {
const credentialFile = materialized.files.get(CATALOG_SECRET_IDS.apiKey);
if (!credentialFile) throw new CatalogConnectorError("REST API key is not configured");
const credential = (await readFile(credentialFile, "utf8")).trim();
if (auth === "bearer") headers.authorization = `Bearer ${credential}`;
else headers["x-api-key"] = credential;
}
const baseUrl = required(database.binding.baseUrl).replace(/\/+$/, "");
const response = await fetch(`${baseUrl}/rpc/schema_snapshot`, {
method: "POST",
headers,
body: JSON.stringify({ schema_name: database.schema }),
signal,
});
let body: unknown;
if (response.ok) {
body = await response.json();
} else if (response.status === 404) {
const fallback = await fetch(`${baseUrl}/rpc/run_query`, {
method: "POST",
headers,
body: JSON.stringify({ query_text: restSnapshotQuery(database.schema) }),
signal,
});
if (!fallback.ok) throw new CatalogSchemaCapabilityUnavailableError("schema_snapshot");
const rows: unknown = await fallback.json();
if (!Array.isArray(rows) || rows.length !== 1) {
throw new CatalogConnectorError("REST schema snapshot fallback is invalid");
}
body = rows[0];
} else {
throw new CatalogSchemaCapabilityUnavailableError("schema_snapshot");
}
const parsed = restSnapshotSchema.safeParse(body);
if (!parsed.success) throw new CatalogConnectorError("REST schema snapshot is invalid");
const snapshot = normalized(parsed.data);
await progress?.("scanning_tables", { tables: snapshot.tables.length });
await progress?.("scanning_columns", { tables: snapshot.tables.length, columns: snapshot.columns.length });
await progress?.("scanning_relationships", {
tables: snapshot.tables.length,
columns: snapshot.columns.length,
relationships: snapshot.relationships.length,
});
return snapshot;
} catch (error) {
if (error instanceof CatalogConnectorError) throw error;
throw new CatalogConnectorError("REST schema introspection failed");
} finally {
materialized.release();
}
}
}
+10
View File
@@ -0,0 +1,10 @@
export const CATALOG_SECRET_IDS = {
password: "catalog.dwh.password",
apiKey: "catalog.dwh.api_key",
sshPrivateKey: "catalog.dwh.ssh_private_key",
sshPrivateKeyPassphrase: "catalog.dwh.ssh_private_key_passphrase",
sshKnownHosts: "catalog.dwh.ssh_known_hosts",
tlsCa: "catalog.dwh.tls_ca",
} as const;
export type CatalogSecretName = keyof typeof CATALOG_SECRET_IDS;
+228
View File
@@ -0,0 +1,228 @@
import { buildInstallationContract } from "../workspaces/contracts.js";
import { resolveBinding } from "../workspaces/bindings.js";
import type { WorkspaceRegistry } from "../workspaces/registry.js";
import type { WorkspaceDescriptor } from "../workspaces/schema.js";
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
import { createConcreteDiagnosticAdapters } from "../workspaces/diagnostics.js";
import { CatalogOperationCoordinator } from "./operation-coordinator.js";
import {
ConcreteCatalogPostgresAccess,
type CatalogPostgresAccess,
} from "./postgres-access.js";
import type {
CatalogRepository,
DatabaseBinding,
DatabaseConfigurationInput,
DatabaseTestResult,
WorkspaceDatabase,
} from "./types.js";
import { CATALOG_SECRET_IDS, type CatalogSecretName } from "./secrets.js";
export { CATALOG_SECRET_IDS, type CatalogSecretName } from "./secrets.js";
export interface CatalogListItem extends Omit<WorkspaceDatabase, "id"> {
id?: string;
workspaceName: string;
workspaceDescription?: string;
workspaceAvailable: boolean;
configured: boolean;
secrets: Record<CatalogSecretName, boolean>;
}
function bindingValue(workspace: WorkspaceDescriptor, values: Record<string, string>, suffix: string) {
const variable = buildInstallationContract(workspace).variables.find((entry) => (
entry.role === "DWH" && entry.suffix === suffix
));
return variable ? values[variable.name] : undefined;
}
function numeric(value: string | undefined): number | undefined {
if (!value) return undefined;
const parsed = Number(value);
return Number.isInteger(parsed) && parsed >= 1 && parsed <= 65_535 ? parsed : undefined;
}
function yamlBinding(workspace: WorkspaceDescriptor, secretRoots: readonly string[]): DatabaseBinding {
const effective = resolveBinding(workspace, "DWH", process.env, secretRoots);
const value = (suffix: string) => bindingValue(workspace, effective.values, suffix);
return {
transport: effective.transport,
host: value("HOST"),
port: numeric(value("PORT")) ?? workspace.dwh.port,
username: value("USER"),
baseUrl: value("BASE_URL"),
restPath: workspace.diagnostics?.dwh_rest?.path ?? "/health",
restAuth: workspace.diagnostics?.dwh_rest?.auth ?? "bearer",
tlsServername: value("TLS_SERVERNAME"),
sshHost: value("SSH_HOST"),
sshPort: numeric(value("SSH_PORT")),
sshUsername: value("SSH_USER"),
sshTargetHost: value("SSH_TARGET_HOST"),
sshTargetPort: numeric(value("SSH_TARGET_PORT")),
};
}
function secretState(store: WorkspaceSecretStore, workspaceId: string): Record<CatalogSecretName, boolean> {
return Object.fromEntries(Object.entries(CATALOG_SECRET_IDS).map(([name, id]) => (
[name, store.has(workspaceId, id)]
))) as Record<CatalogSecretName, boolean>;
}
export class CatalogService {
private readonly adapters = createConcreteDiagnosticAdapters();
constructor(
private readonly repository: CatalogRepository,
private readonly registry: WorkspaceRegistry,
private readonly secretStore: WorkspaceSecretStore,
private readonly secretRoots: readonly string[],
private readonly diagnosticTimeoutMs: number,
private readonly postgres: CatalogPostgresAccess = new ConcreteCatalogPostgresAccess(secretStore, {
connectTimeoutMs: diagnosticTimeoutMs,
}),
private readonly operations: CatalogOperationCoordinator = new CatalogOperationCoordinator(),
) {}
async list(): Promise<CatalogListItem[]> {
const [workspaces, configured] = await Promise.all([
this.registry.listCatalog(),
this.repository.list(),
]);
const byWorkspace = new Map(configured.map((database) => [database.workspaceId, database]));
const active = await Promise.all(workspaces.map(async (entry) => {
const database = byWorkspace.get(entry.id);
const { workspace } = await this.registry.read(entry.id);
const base = database ?? {
workspaceId: entry.id,
engine: "postgres" as const,
databaseName: workspace.dwh.database,
schema: workspace.dwh.schema,
version: 0,
createdAt: "",
updatedAt: "",
binding: yamlBinding(workspace, this.secretRoots),
connectionStatus: "untested" as const,
};
return {
...base,
workspaceName: entry.name,
workspaceDescription: entry.description,
workspaceAvailable: true,
configured: database !== undefined,
secrets: secretState(this.secretStore, entry.id),
};
}));
const known = new Set(workspaces.map((entry) => entry.id));
const orphaned: CatalogListItem[] = configured
.filter((database) => !known.has(database.workspaceId))
.map((database) => ({
...database,
workspaceName: database.workspaceId,
workspaceDescription: "Workspace is no longer present in the repository catalog.",
workspaceAvailable: false,
configured: true,
secrets: secretState(this.secretStore, database.workspaceId),
}));
return [...active, ...orphaned];
}
async ensureWorkspace(workspaceId: string): Promise<WorkspaceDescriptor> {
const { workspace } = await this.registry.read(workspaceId);
return workspace;
}
async normalizeInput(input: DatabaseConfigurationInput): Promise<DatabaseConfigurationInput> {
const workspace = await this.ensureWorkspace(input.workspaceId);
if (input.binding.transport !== "rest_api") return input;
return {
...input,
binding: {
...input.binding,
restPath: workspace.diagnostics?.dwh_rest?.path ?? "/health",
},
};
}
configuredSecrets(workspaceId: string): Record<CatalogSecretName, boolean> {
return secretState(this.secretStore, workspaceId);
}
replaceSecrets(workspaceId: string, values: Partial<Record<CatalogSecretName, string>>): void {
const encoded: Record<string, string> = {};
for (const [name, value] of Object.entries(values) as Array<[CatalogSecretName, string | undefined]>) {
if (value !== undefined && value.length > 0) encoded[CATALOG_SECRET_IDS[name]] = value;
}
if (Object.keys(encoded).length > 0) this.secretStore.putMany(workspaceId, encoded);
}
forgetSecrets(workspaceId: string): void {
for (const id of Object.values(CATALOG_SECRET_IDS)) this.secretStore.forget(workspaceId, id);
}
async test(database: WorkspaceDatabase): Promise<DatabaseTestResult> {
return await this.operations.run(database.id, async () => {
const testedAt = new Date().toISOString();
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), this.diagnosticTimeoutMs);
const required = database.binding.transport === "rest_api"
? database.binding.restAuth === "none" ? [] : [CATALOG_SECRET_IDS.apiKey]
: [];
const materialized = this.secretStore.materialize(database.workspaceId, required);
try {
if (database.binding.transport !== "rest_api") {
const client = await this.postgres.connect(database, controller.signal);
try {
const result = await client.query(
`SELECT current_database() AS database,
CASE WHEN pg_catalog.has_schema_privilege(
current_user,
(SELECT oid FROM pg_catalog.pg_namespace WHERE nspname = $1),
'USAGE'
) THEN $1 ELSE NULL END AS schema`,
[database.schema],
);
const row = result.rows[0];
if (row?.database !== database.databaseName || row.schema !== database.schema) {
throw new Error("Database identity mismatch");
}
} finally {
await client.end();
}
} else {
const credentialId = CATALOG_SECRET_IDS.apiKey;
await this.adapters.probeConnector({
role: "dwh",
transport: database.binding.transport,
baseUrl: database.binding.baseUrl,
credentialFile: materialized.files.get(credentialId),
resource: { database: database.databaseName, schema: database.schema },
timeoutMs: this.diagnosticTimeoutMs,
signal: controller.signal,
diagnostic: {
method: "GET" as const,
path: database.binding.restPath ?? "/health",
auth: database.binding.restAuth ?? "bearer",
},
});
}
return {
connectionStatus: "reachable",
testedVersion: database.version,
lastTestedAt: testedAt,
};
} catch {
return {
connectionStatus: "failed",
testedVersion: database.version,
lastTestedAt: testedAt,
errorCode: "connector_unavailable",
errorMessage: "The database connector could not be reached or authenticated.",
};
} finally {
clearTimeout(timer);
controller.abort();
materialized.release();
}
});
}
}
+311
View File
@@ -0,0 +1,311 @@
import { randomUUID } from "node:crypto";
import type { CatalogOperationCoordinator } from "./operation-coordinator.js";
import type { CatalogSchemaIntrospector, CatalogSchemaScanProgress } from "./schema-introspector.js";
import {
CatalogConflictError,
CatalogConnectorError,
CatalogSchemaCapabilityUnavailableError,
type CatalogRepository,
type CatalogSchemaDiff,
type CatalogSyncCounts,
type CatalogSyncRun,
type CatalogSyncScope,
type ObservedSchemaSnapshot,
type WorkspaceDatabase,
} from "./types.js";
class SyncCancelledError extends Error {}
const TERMINAL_STATES = new Set<CatalogSyncRun["state"]>([
"succeeded", "failed", "cancelled", "interrupted",
]);
function destructive(diff: CatalogSchemaDiff): boolean {
return diff.deletedTables.length > 0
|| diff.deletedColumns.length > 0
|| diff.deletedRelationships.length > 0;
}
function fingerprint(snapshot: ObservedSchemaSnapshot): string {
return JSON.stringify(snapshot);
}
function safeFailure(error: unknown): { code: string; message: string } {
if (error instanceof CatalogSchemaCapabilityUnavailableError) {
const label = error.capability === "schema_snapshot" ? "schema snapshot" : error.capability;
return {
code: "schema_capability_unavailable",
message: `This database binding does not provide the ${label} capability.`,
};
}
if (error instanceof CatalogConnectorError) {
return { code: "schema_introspection_failed", message: "The database schema could not be read safely." };
}
return { code: "schema_sync_failed", message: "Schema synchronization failed." };
}
export class CatalogSyncWorker {
private readonly workerId = randomUUID();
private readonly controllers = new Map<string, AbortController>();
private readonly reservations = new Map<string, () => void>();
private stopping = false;
constructor(
private readonly repository: CatalogRepository,
private readonly introspector: CatalogSchemaIntrospector,
private readonly operations: CatalogOperationCoordinator,
private readonly timeoutMs: number,
) {}
async initialize(): Promise<void> {
if (!(await this.repository.available())) return;
await this.repository.interruptActiveSyncRuns();
await this.repository.pruneSyncEvents(new Date(Date.now() - 30 * 24 * 60 * 60 * 1_000).toISOString());
}
async start(database: WorkspaceDatabase, scope: CatalogSyncScope, tableIds: readonly string[]): Promise<CatalogSyncRun> {
this.assertReady(database);
const uniqueTableIds = [...new Set(tableIds)];
if (scope === "columns") {
const tables = await Promise.all(uniqueTableIds.map((tableId) => this.repository.getTable(database.id, tableId)));
if (tables.some((table) => !table)) throw new CatalogConflictError("One or more selected tables no longer exist");
}
if (scope !== "columns" && uniqueTableIds.length > 0) {
throw new CatalogConflictError("Table selection is only valid for a column synchronization");
}
const release = this.operations.reserve(database.id);
try {
const run = await this.repository.createSyncRun(database.id, scope, uniqueTableIds, database.version);
this.reservations.set(run.id, release);
await this.repository.appendSyncEvent(run.id, "info", "queued", "Synchronization queued.", { scope });
this.launch(run.id);
return run;
} catch (error) {
release();
throw error;
}
}
async confirm(runId: string, confirmationToken: string): Promise<CatalogSyncRun | undefined> {
const run = await this.repository.getSyncRun(runId);
if (!run) return undefined;
if (run.state !== "awaiting_confirmation" || !run.observedSnapshot || run.confirmationToken !== confirmationToken) {
throw new CatalogConflictError("Synchronization confirmation is no longer valid");
}
await this.repository.appendSyncEvent(run.id, "info", "confirmation_received", "Destructive changes were confirmed.");
const queued = await this.repository.updateSyncRun(run.id, {
state: "queued",
phase: "queued",
confirmationToken: null,
leaseOwner: null,
leaseExpiresAt: null,
});
this.launch(run.id, fingerprint(run.observedSnapshot));
return queued;
}
async cancel(runId: string): Promise<CatalogSyncRun | undefined> {
const run = await this.repository.getSyncRun(runId);
if (!run) return undefined;
if (run.state === "applying" || TERMINAL_STATES.has(run.state)) return run;
await this.repository.requestSyncRunCancellation(runId);
this.controllers.get(runId)?.abort();
if (run.state === "queued" || run.state === "awaiting_confirmation") {
const cancelled = await this.repository.updateSyncRun(runId, {
state: "cancelled",
phase: "completed",
finishedAt: new Date().toISOString(),
observedSnapshot: null,
plannedDiff: null,
confirmationToken: null,
leaseOwner: null,
leaseExpiresAt: null,
});
await this.repository.appendSyncEvent(runId, "warning", "cancelled", "Synchronization cancelled.");
this.release(runId);
return cancelled;
}
return await this.repository.getSyncRun(runId);
}
async retry(runId: string): Promise<CatalogSyncRun | undefined> {
const previous = await this.repository.getSyncRun(runId);
if (!previous) return undefined;
if (!TERMINAL_STATES.has(previous.state)) {
throw new CatalogConflictError("Only a finished synchronization can be retried");
}
const database = await this.repository.get(previous.databaseId);
if (!database) return undefined;
return await this.start(database, previous.scope, previous.tableIds);
}
async stop(): Promise<void> {
this.stopping = true;
for (const controller of this.controllers.values()) controller.abort();
if (await this.repository.available()) await this.repository.interruptActiveSyncRuns();
for (const runId of [...this.reservations.keys()]) this.release(runId);
}
private launch(runId: string, confirmedFingerprint?: string): void {
queueMicrotask(() => {
void this.execute(runId, confirmedFingerprint).catch(() => undefined);
});
}
private async execute(runId: string, confirmedFingerprint?: string): Promise<void> {
if (this.stopping) return;
const leaseExpiresAt = new Date(Date.now() + 20_000).toISOString();
const claimed = await this.repository.claimSyncRun(runId, this.workerId, leaseExpiresAt);
if (!claimed) return;
const controller = new AbortController();
this.controllers.set(runId, controller);
let timedOut = false;
const timeout = setTimeout(() => {
timedOut = true;
controller.abort();
}, this.timeoutMs);
const heartbeat = setInterval(() => {
void this.repository.updateSyncRun(runId, {
heartbeatAt: new Date().toISOString(),
leaseExpiresAt: new Date(Date.now() + 20_000).toISOString(),
});
}, 5_000);
try {
await this.repository.appendSyncEvent(runId, "info", "started", "Synchronization started.");
const database = await this.repository.get(claimed.databaseId);
if (!database || database.version !== claimed.requestedDatabaseVersion) {
throw new CatalogConflictError("Database binding changed before synchronization started");
}
this.assertReady(database);
const progress: CatalogSchemaScanProgress = async (phase, counts) => {
await this.checkCancelled(runId);
await this.repository.updateSyncRun(runId, {
phase,
heartbeatAt: new Date().toISOString(),
...(counts ? { counts } : {}),
});
await this.repository.appendSyncEvent(runId, "info", phase, this.phaseMessage(phase), counts ?? {});
};
const snapshot = await this.introspector.scan(database, controller.signal, progress);
await this.checkCancelled(runId);
this.assertCapability(claimed.scope, snapshot);
const counts: CatalogSyncCounts = {
tables: snapshot.tables.length,
columns: snapshot.columns.length,
relationships: snapshot.relationships.length,
};
await this.repository.updateSyncRun(runId, { phase: "planning", counts, observedSnapshot: snapshot });
await this.repository.appendSyncEvent(runId, "info", "planning", "Schema changes are being planned.", { ...counts });
const diff = await this.repository.planSchemaSync(claimed.databaseId, claimed.scope, claimed.tableIds, snapshot);
await this.checkCancelled(runId);
if (destructive(diff) && fingerprint(snapshot) !== confirmedFingerprint) {
const token = randomUUID();
const waiting = await this.repository.updateSyncRun(runId, {
state: "awaiting_confirmation",
phase: "awaiting_confirmation",
observedSnapshot: snapshot,
plannedDiff: diff,
confirmationToken: token,
counts,
leaseOwner: null,
leaseExpiresAt: null,
});
await this.repository.appendSyncEvent(runId, "warning", "confirmation_required", "Confirmation is required before removing catalog objects.", {
deletedTables: diff.deletedTables.length,
deletedColumns: diff.deletedColumns.length,
deletedRelationships: diff.deletedRelationships.length,
});
if (!waiting) throw new Error("Synchronization run disappeared");
return;
}
await this.repository.updateSyncRun(runId, { state: "applying", phase: "applying", plannedDiff: diff });
await this.repository.appendSyncEvent(runId, "info", "applying", "Catalog changes are being applied atomically.");
this.controllers.delete(runId);
const applied = await this.repository.applySchemaSync(
claimed.databaseId,
claimed.requestedDatabaseVersion,
claimed.scope,
claimed.tableIds,
snapshot,
);
if (!applied) throw new CatalogConflictError("Database binding changed before schema changes were applied");
await this.repository.updateSyncRun(runId, {
state: "succeeded",
phase: "completed",
counts: applied,
finishedAt: new Date().toISOString(),
observedSnapshot: null,
plannedDiff: null,
confirmationToken: null,
heartbeatAt: new Date().toISOString(),
leaseOwner: null,
leaseExpiresAt: null,
});
await this.repository.appendSyncEvent(runId, "info", "succeeded", "Synchronization completed.", { ...applied });
this.release(runId);
} catch (error) {
const current = await this.repository.getSyncRun(runId);
const cancelled = !timedOut && (error instanceof SyncCancelledError || controller.signal.aborted || current?.cancelRequested);
const failure = timedOut
? { code: "schema_sync_timed_out", message: "Schema synchronization timed out." }
: safeFailure(error);
await this.repository.updateSyncRun(runId, {
state: cancelled ? "cancelled" : "failed",
phase: "completed",
errorCode: cancelled ? null : failure.code,
errorMessage: cancelled ? null : failure.message,
finishedAt: new Date().toISOString(),
observedSnapshot: null,
plannedDiff: null,
confirmationToken: null,
leaseOwner: null,
leaseExpiresAt: null,
});
await this.repository.appendSyncEvent(
runId,
cancelled ? "warning" : "error",
cancelled ? "cancelled" : "failed",
cancelled ? "Synchronization cancelled." : failure.message,
);
this.release(runId);
} finally {
clearTimeout(timeout);
clearInterval(heartbeat);
this.controllers.delete(runId);
}
}
private assertReady(database: WorkspaceDatabase): void {
if (database.connectionStatus !== "reachable" || database.testedVersion !== database.version) {
throw new CatalogConflictError("Test the current database binding before synchronizing its schema");
}
}
private assertCapability(scope: CatalogSyncScope, snapshot: ObservedSchemaSnapshot): void {
const required = scope === "all" ? ["tables", "columns", "relationships"] as const : [scope] as const;
for (const name of required) {
if (snapshot.capabilities[name] !== "available") {
throw new CatalogSchemaCapabilityUnavailableError(name);
}
}
}
private async checkCancelled(runId: string): Promise<void> {
const run = await this.repository.getSyncRun(runId);
if (run?.cancelRequested) throw new SyncCancelledError("Synchronization cancelled");
}
private release(runId: string): void {
this.reservations.get(runId)?.();
this.reservations.delete(runId);
}
private phaseMessage(phase: Parameters<CatalogSchemaScanProgress>[0]): string {
if (phase === "connecting") return "Connecting to the database.";
if (phase === "scanning_tables") return "Reading tables.";
if (phase === "scanning_columns") return "Reading columns and primary keys.";
return "Reading foreign-key relationships.";
}
}
+133
View File
@@ -0,0 +1,133 @@
import { readFile } from "node:fs/promises";
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
import type { CatalogPostgresAccess } from "./postgres-access.js";
import { CATALOG_SECRET_IDS } from "./secrets.js";
import {
CatalogConnectorError,
type ObservedCatalogTable,
type WorkspaceDatabase,
} from "./types.js";
export interface CatalogTableIntrospector {
scan(database: WorkspaceDatabase, signal: AbortSignal): Promise<ObservedCatalogTable[]>;
}
function normalize(rows: readonly ObservedCatalogTable[]): ObservedCatalogTable[] {
const byName = new Map<string, ObservedCatalogTable>();
for (const row of rows) {
if (typeof row.name !== "string" || row.name.length === 0 || row.name.length > 128) {
throw new CatalogConnectorError("Schema introspection response is invalid");
}
if (row.sourceComment !== null && typeof row.sourceComment !== "string") {
throw new CatalogConnectorError("Schema introspection response is invalid");
}
byName.set(row.name, row);
}
return [...byName.values()]
.sort((left, right) => left.name.localeCompare(right.name));
}
function requireText(value: string | undefined): string {
if (!value) throw new CatalogConnectorError("Database binding is incomplete");
return value;
}
export class ConcreteCatalogTableIntrospector implements CatalogTableIntrospector {
constructor(
private readonly postgres: CatalogPostgresAccess,
private readonly secretStore: WorkspaceSecretStore,
) {}
async scan(database: WorkspaceDatabase, signal: AbortSignal): Promise<ObservedCatalogTable[]> {
return database.binding.transport === "rest_api"
? await this.scanRest(database, signal)
: await this.scanPostgres(database, signal);
}
private async scanPostgres(
database: WorkspaceDatabase,
signal: AbortSignal,
): Promise<ObservedCatalogTable[]> {
const client = await this.postgres.connect(database, signal);
try {
const schema = await client.query(
"SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname = $1) AS present",
[database.schema],
);
if (schema.rows[0]?.present !== true) throw new CatalogConnectorError("Database schema is unavailable");
const result = await client.query(
`SELECT c.relname AS name, d.description AS source_comment
FROM pg_catalog.pg_class c
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = 0
WHERE c.relkind IN ('r', 'p') AND n.nspname = $1
ORDER BY c.relname`,
[database.schema],
);
return normalize(result.rows.map((row) => ({
name: String(row.name),
sourceComment: typeof row.source_comment === "string" && row.source_comment.length > 0
? row.source_comment
: null,
})));
} finally {
await client.end();
}
}
private async scanRest(
database: WorkspaceDatabase,
signal: AbortSignal,
): Promise<ObservedCatalogTable[]> {
const auth = database.binding.restAuth ?? "bearer";
const required = auth === "none" ? [] : [CATALOG_SECRET_IDS.apiKey];
const materialized = this.secretStore.materialize(database.workspaceId, required);
try {
const headers: Record<string, string> = { "content-type": "application/json" };
if (auth !== "none") {
const credentialFile = materialized.files.get(CATALOG_SECRET_IDS.apiKey);
if (!credentialFile) throw new CatalogConnectorError("REST API key is not configured");
const credential = (await readFile(credentialFile, "utf8")).trim();
if (auth === "bearer") headers.authorization = `Bearer ${credential}`;
else headers["x-api-key"] = credential;
}
const baseUrl = requireText(database.binding.baseUrl).replace(/\/+$/, "");
const response = await fetch(`${baseUrl}/rpc/list_tables`, {
method: "POST",
headers,
body: JSON.stringify({ schema_name: database.schema }),
signal,
});
if (!response.ok) throw new CatalogConnectorError("REST schema introspection failed");
const body: unknown = await response.json();
if (!Array.isArray(body)) throw new CatalogConnectorError("REST schema response is invalid");
const rows: ObservedCatalogTable[] = [];
for (const item of body) {
if (!item || typeof item !== "object") {
throw new CatalogConnectorError("REST schema response is invalid");
}
const row = item as Record<string, unknown>;
if (typeof row.type !== "string") {
throw new CatalogConnectorError("REST schema response is invalid");
}
if (row.type !== "TABLE") continue;
if (typeof row.table !== "string" || row.table.length === 0 || row.table.length > 128) {
throw new CatalogConnectorError("REST schema response is invalid");
}
if (row.comment !== undefined && row.comment !== null && typeof row.comment !== "string") {
throw new CatalogConnectorError("REST schema response is invalid");
}
rows.push({
name: row.table,
sourceComment: typeof row.comment === "string" && row.comment.length > 0 ? row.comment : null,
});
}
return normalize(rows);
} catch (error) {
if (error instanceof CatalogConnectorError) throw error;
throw new CatalogConnectorError("REST schema introspection failed");
} finally {
materialized.release();
}
}
}
+45
View File
@@ -0,0 +1,45 @@
import type {
CatalogRepository,
CatalogTable,
} from "./types.js";
export class CatalogTableService {
constructor(
private readonly repository: CatalogRepository,
) {}
async list(databaseId: string): Promise<CatalogTable[]> {
return await this.repository.listTables(databaseId);
}
async updateDescription(
databaseId: string,
tableId: string,
expectedVersion: number,
description: string | null,
): Promise<CatalogTable | undefined> {
const normalized = description?.trim() || null;
return await this.repository.updateTableDescription(
databaseId,
tableId,
expectedVersion,
normalized,
);
}
async updateMetadata(
databaseId: string,
tableId: string,
expectedVersion: number,
description: string | null,
generatedDescription: string | null,
): Promise<CatalogTable | undefined> {
return await this.repository.updateTableMetadata(
databaseId,
tableId,
expectedVersion,
description?.trim() || null,
generatedDescription?.trim() || null,
);
}
}
+339
View File
@@ -0,0 +1,339 @@
export const DATABASE_TRANSPORTS = ["postgres_direct", "rest_api", "ssh_tunnel"] as const;
export type DatabaseTransport = (typeof DATABASE_TRANSPORTS)[number];
export type ConnectionStatus = "untested" | "reachable" | "failed";
export interface DatabaseBinding {
transport: DatabaseTransport;
host?: string;
port?: number;
username?: string;
baseUrl?: string;
restPath?: string;
restAuth?: "none" | "bearer" | "x-api-key";
tlsServername?: string;
sshHost?: string;
sshPort?: number;
sshUsername?: string;
sshTargetHost?: string;
sshTargetPort?: number;
}
export interface WorkspaceDatabase {
id: string;
workspaceId: string;
engine: "postgres";
databaseName: string;
schema: string;
version: number;
createdAt: string;
updatedAt: string;
binding: DatabaseBinding;
connectionStatus: ConnectionStatus;
testedVersion?: number;
lastTestedAt?: string;
lastErrorCode?: string;
lastErrorMessage?: string;
schemaSyncedVersion?: number;
schemaSyncedAt?: string;
}
export interface DatabaseConfigurationInput {
workspaceId: string;
engine: "postgres";
databaseName: string;
schema: string;
binding: DatabaseBinding;
}
export interface DatabaseTestResult {
connectionStatus: Exclude<ConnectionStatus, "untested">;
testedVersion: number;
lastTestedAt: string;
errorCode?: string;
errorMessage?: string;
}
export interface CatalogTable {
id: string;
databaseId: string;
name: string;
sourceComment: string | null;
description: string | null;
generatedDescription: string | null;
lastSyncedDatabaseVersion: number | null;
lastSyncedAt: string | null;
version: number;
createdAt: string;
updatedAt: string;
}
export interface ObservedCatalogTable {
name: string;
sourceComment: string | null;
}
export interface CatalogColumn {
id: string;
tableId: string;
name: string;
ordinalPosition: number;
dataType: string;
isNullable: boolean;
defaultExpression: string | null;
primaryKeyPosition: number | null;
isPrimaryKey: boolean;
isForeignKey: boolean;
foreignKeyCount: number;
sourceComment: string | null;
description: string | null;
generatedDescription: string | null;
lastSyncedDatabaseVersion: number | null;
lastSyncedAt: string | null;
version: number;
createdAt: string;
updatedAt: string;
}
export interface ObservedCatalogColumn {
tableName: string;
name: string;
ordinalPosition: number;
dataType: string;
isNullable: boolean;
defaultExpression: string | null;
primaryKeyPosition: number | null;
sourceComment: string | null;
}
export interface CatalogRelationshipColumn {
position: number;
sourceColumnId: string;
sourceColumnName: string;
targetColumnId: string;
targetColumnName: string;
}
export interface CatalogRelationship {
id: string;
databaseId: string;
constraintName: string;
sourceTableId: string;
sourceTableName: string;
targetTableId: string;
targetTableName: string;
updateRule: string;
deleteRule: string;
deferrable: boolean;
initiallyDeferred: boolean;
columns: CatalogRelationshipColumn[];
lastSyncedDatabaseVersion: number | null;
lastSyncedAt: string | null;
createdAt: string;
updatedAt: string;
}
export interface ObservedRelationshipColumn {
position: number;
sourceColumnName: string;
targetColumnName: string;
}
export interface ObservedCatalogRelationship {
constraintName: string;
sourceTableName: string;
targetTableName: string;
updateRule: string;
deleteRule: string;
deferrable: boolean;
initiallyDeferred: boolean;
columns: ObservedRelationshipColumn[];
}
export type IntrospectionCapabilityState = "available" | "unavailable";
export interface ObservedSchemaSnapshot {
schemaVersion: 1;
capabilities: {
tables: IntrospectionCapabilityState;
columns: IntrospectionCapabilityState;
relationships: IntrospectionCapabilityState;
};
tables: ObservedCatalogTable[];
columns: ObservedCatalogColumn[];
relationships: ObservedCatalogRelationship[];
}
export type CatalogSyncScope = "tables" | "columns" | "relationships" | "all";
export type CatalogSyncState =
| "queued" | "running" | "awaiting_confirmation" | "applying"
| "succeeded" | "failed" | "cancelled" | "interrupted";
export type CatalogSyncPhase =
| "queued" | "connecting" | "scanning_tables" | "scanning_columns"
| "scanning_relationships" | "planning" | "awaiting_confirmation"
| "applying" | "completed";
export interface CatalogSchemaDiff {
deletedTables: string[];
deletedColumns: Array<{ tableName: string; columnName: string }>;
deletedRelationships: Array<{ sourceTableName: string; constraintName: string }>;
}
export interface CatalogSyncCounts {
tables?: number;
columns?: number;
relationships?: number;
created?: number;
updated?: number;
deleted?: number;
}
export interface CatalogSyncRun {
id: string;
databaseId: string;
scope: CatalogSyncScope;
tableIds: string[];
state: CatalogSyncState;
phase: CatalogSyncPhase;
requestedDatabaseVersion: number;
observedSnapshot: ObservedSchemaSnapshot | null;
plannedDiff: CatalogSchemaDiff | null;
confirmationToken: string | null;
counts: CatalogSyncCounts;
errorCode: string | null;
errorMessage: string | null;
cancelRequested: boolean;
createdAt: string;
startedAt: string | null;
updatedAt: string;
finishedAt: string | null;
heartbeatAt: string | null;
leaseOwner: string | null;
leaseExpiresAt: string | null;
}
export interface CatalogSyncEvent {
id: number;
runId: string;
sequence: number;
level: "info" | "warning" | "error";
eventType: string;
message: string;
data: Record<string, unknown>;
createdAt: string;
}
export interface CatalogSyncRunUpdate {
state?: CatalogSyncState;
phase?: CatalogSyncPhase;
observedSnapshot?: ObservedSchemaSnapshot | null;
plannedDiff?: CatalogSchemaDiff | null;
confirmationToken?: string | null;
counts?: CatalogSyncCounts;
errorCode?: string | null;
errorMessage?: string | null;
cancelRequested?: boolean;
startedAt?: string | null;
finishedAt?: string | null;
heartbeatAt?: string | null;
leaseOwner?: string | null;
leaseExpiresAt?: string | null;
}
export type TableSyncRepositoryResult =
| { kind: "confirmation_required"; deletedNames: string[] }
| {
kind: "applied";
createdCount: number;
updatedCount: number;
deletedCount: number;
tables: CatalogTable[];
};
export interface CatalogRepository {
list(): Promise<WorkspaceDatabase[]>;
get(id: string): Promise<WorkspaceDatabase | undefined>;
getByWorkspace(workspaceId: string): Promise<WorkspaceDatabase | undefined>;
create(input: DatabaseConfigurationInput): Promise<WorkspaceDatabase>;
update(id: string, expectedVersion: number, input: DatabaseConfigurationInput): Promise<WorkspaceDatabase | undefined>;
recordTest(id: string, expectedVersion: number, result: DatabaseTestResult): Promise<WorkspaceDatabase | undefined>;
touch(id: string, expectedVersion: number): Promise<WorkspaceDatabase | undefined>;
delete(id: string, expectedVersion: number): Promise<boolean>;
listTables(databaseId: string): Promise<CatalogTable[]>;
getTable(databaseId: string, tableId: string): Promise<CatalogTable | undefined>;
updateTableDescription(
databaseId: string,
tableId: string,
expectedVersion: number,
description: string | null,
): Promise<CatalogTable | undefined>;
updateTableMetadata(
databaseId: string,
tableId: string,
expectedVersion: number,
description: string | null,
generatedDescription: string | null,
): Promise<CatalogTable | undefined>;
listColumns(databaseId: string, tableId: string): Promise<CatalogColumn[]>;
getColumn(databaseId: string, tableId: string, columnId: string): Promise<CatalogColumn | undefined>;
updateColumnMetadata(
databaseId: string,
tableId: string,
columnId: string,
expectedVersion: number,
description: string | null,
generatedDescription: string | null,
): Promise<CatalogColumn | undefined>;
listRelationships(databaseId: string): Promise<CatalogRelationship[]>;
planSchemaSync(
databaseId: string,
scope: CatalogSyncScope,
tableIds: readonly string[],
snapshot: ObservedSchemaSnapshot,
): Promise<CatalogSchemaDiff>;
applySchemaSync(
databaseId: string,
expectedDatabaseVersion: number,
scope: CatalogSyncScope,
tableIds: readonly string[],
snapshot: ObservedSchemaSnapshot,
): Promise<CatalogSyncCounts | undefined>;
createSyncRun(
databaseId: string,
scope: CatalogSyncScope,
tableIds: readonly string[],
requestedDatabaseVersion: number,
): Promise<CatalogSyncRun>;
getSyncRun(runId: string): Promise<CatalogSyncRun | undefined>;
claimSyncRun(runId: string, workerId: string, leaseExpiresAt: string): Promise<CatalogSyncRun | undefined>;
listSyncRuns(databaseId: string, limit?: number): Promise<CatalogSyncRun[]>;
updateSyncRun(runId: string, update: CatalogSyncRunUpdate): Promise<CatalogSyncRun | undefined>;
requestSyncRunCancellation(runId: string): Promise<CatalogSyncRun | undefined>;
appendSyncEvent(
runId: string,
level: CatalogSyncEvent["level"],
eventType: string,
message: string,
data?: Record<string, unknown>,
): Promise<CatalogSyncEvent>;
listSyncEvents(runId: string, afterSequence?: number): Promise<CatalogSyncEvent[]>;
pruneSyncEvents(before: string): Promise<void>;
interruptActiveSyncRuns(): Promise<void>;
reconcileTables(
databaseId: string,
expectedDatabaseVersion: number,
observed: readonly ObservedCatalogTable[],
confirmedDeletedNames: readonly string[],
): Promise<TableSyncRepositoryResult | undefined>;
available(): Promise<boolean>;
close?(): Promise<void>;
}
export class CatalogConflictError extends Error {}
export class CatalogUnavailableError extends Error {}
export class CatalogOperationInProgressError extends Error {}
export class CatalogConnectorError extends Error {}
export class CatalogSchemaCapabilityUnavailableError extends CatalogConnectorError {
constructor(readonly capability: "schema_snapshot" | "tables" | "columns" | "relationships") {
super(`Schema introspection capability '${capability}' is unavailable`);
}
}
+41
View File
@@ -7,6 +7,7 @@ import {
} from "./auth/config.js";
import { createProjectedAuthenticationConfigProvider } from "./auth/runtime-projection.js";
import type { WorkspaceRegistryConfig } from "./workspaces/types.js";
import type { CatalogConnectionConfig } from "./catalog/repository.js";
export interface AppConfig {
host: string; port: number; harnessDir: string; thtBin: string; piBin: string;
@@ -20,6 +21,8 @@ export interface AppConfig {
host?: string; port?: number; database?: string; runtimeUser?: string;
runtimePasswordFile?: string; sslmode?: "verify-ca" | "verify-full"; sslrootcert?: string;
};
/** Installation-local metadata catalog. Omitted installations expose an unavailable admin surface. */
catalogDatabase?: CatalogConnectionConfig;
defaults: { provider?: string; model?: string; thinking?: string };
maxPiProcesses: number;
settingsFile: string;
@@ -40,6 +43,7 @@ export interface AppConfig {
/** Explicit compatibility mode for old loopback clients that send `workspace` in POST /sessions. */
legacyWorkspaceMode: boolean;
workspaceDiagnosticTimeoutMs: number;
catalogSyncTimeoutMs: number;
workspaceRegistry: WorkspaceRegistryConfig;
workspaceSecretStoreRoot: string;
workspaceSecretRuntimeRoot: string;
@@ -127,6 +131,14 @@ function diagnosticTimeout(value: string | undefined): number {
return timeout;
}
function catalogSyncTimeout(value: string | undefined): number {
const timeout = Number(value ?? 600_000);
if (!Number.isSafeInteger(timeout) || timeout < 1_000 || timeout > 3_600_000) {
throw new Error("catalog synchronization timeout configuration is invalid");
}
return timeout;
}
function piManagementTimeout(value: string | undefined): number {
const timeout = Number(value ?? 8_000);
if (!Number.isSafeInteger(timeout) || timeout < 1 || timeout > 30_000) {
@@ -176,6 +188,33 @@ function positiveDimension(value: string | undefined, fallback: number): number
return parsed;
}
function catalogDatabase(env: Record<string, string | undefined>): CatalogConnectionConfig | undefined {
const value = env.THT_CATALOG_DATABASE_URL;
if (value !== undefined) {
try {
const parsed = new URL(value);
if ((parsed.protocol !== "postgres:" && parsed.protocol !== "postgresql:")
|| !parsed.hostname || !parsed.pathname.slice(1) || parsed.hash || parsed.search) throw new Error();
return { connectionString: value };
} catch {
throw new Error("catalog database configuration is invalid");
}
}
const host = env.THT_CATALOG_DB_HOST;
if (host === undefined) return undefined;
const port = Number(env.THT_CATALOG_DB_PORT ?? 5432);
const database = env.THT_CATALOG_DB_NAME;
const user = env.THT_CATALOG_RUNTIME_USER;
const passwordFile = env.THT_CATALOG_RUNTIME_PASSWORD_FILE;
try {
if (!host.trim() || !database?.trim() || !user?.trim() || !passwordFile
|| !path.isAbsolute(passwordFile) || !Number.isInteger(port) || port < 1 || port > 65_535) throw new Error();
return { host, port, database, user, passwordFile };
} catch {
throw new Error("catalog database configuration is invalid");
}
}
export function loadConfig(
env: Record<string, string | undefined>,
options: { surface?: "application" | "workspace-maintenance" } = {},
@@ -356,6 +395,7 @@ export function loadConfig(
authentication,
publicExposure,
sessionStorage,
catalogDatabase: catalogDatabase(env),
defaults: { provider: env.PI_PROVIDER, model: env.PI_MODEL, thinking: env.PI_THINKING },
maxPiProcesses: Number(env.MAX_PI_PROCESSES ?? 4),
settingsFile,
@@ -370,6 +410,7 @@ export function loadConfig(
dwhPrecheck: env.THT_DWH_PRECHECK === "true" || env.THT_DWH_PRECHECK === "1",
legacyWorkspaceMode: legacyWorkspaceMode === "local",
workspaceDiagnosticTimeoutMs: diagnosticTimeout(env.THT_WORKSPACE_DIAGNOSTIC_TIMEOUT_MS),
catalogSyncTimeoutMs: catalogSyncTimeout(env.THT_CATALOG_SYNC_TIMEOUT_MS),
workspaceRegistry,
workspaceSecretStoreRoot,
workspaceSecretRuntimeRoot,
+218
View File
@@ -0,0 +1,218 @@
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { z } from "zod";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import { CatalogService, type CatalogSecretName } from "../catalog/service.js";
import { WorkspaceRegistryError } from "../workspaces/git-repository.js";
import {
CatalogConflictError,
CatalogOperationInProgressError,
CatalogUnavailableError,
DATABASE_TRANSPORTS,
type CatalogRepository,
type DatabaseConfigurationInput,
} from "../catalog/types.js";
import type { CatalogOperationCoordinator } from "../catalog/operation-coordinator.js";
const idSchema = z.uuid();
const workspaceIdSchema = z.string().regex(/^[a-z][a-z0-9-]{2,62}$/);
const identifier = z.string().trim().min(1).max(128).regex(/^[A-Za-z_][A-Za-z0-9_$-]*$/);
const nonEmpty = z.string().trim().min(1).max(512);
const port = z.number().int().min(1).max(65_535);
const optionalText = nonEmpty.optional();
const sshHost = z.string().trim().min(1).max(255).regex(/^[A-Za-z0-9_.:\[\]-]+$/).optional();
const sshUsername = z.string().trim().min(1).max(128).regex(/^[A-Za-z0-9._-]+$/).optional();
const bindingSchema = z.object({
transport: z.enum(DATABASE_TRANSPORTS),
host: optionalText,
port: port.optional(),
username: optionalText,
baseUrl: z.url().max(2048).optional(),
restPath: z.string().regex(/^\/(?!\/)[^?#\\\u0000-\u001f]*$/).max(512).optional(),
restAuth: z.enum(["none", "bearer", "x-api-key"]).optional(),
tlsServername: optionalText,
sshHost,
sshPort: port.optional(),
sshUsername,
sshTargetHost: sshHost,
sshTargetPort: port.optional(),
}).strict().superRefine((binding, context) => {
const required = binding.transport === "postgres_direct"
? ["host", "port", "username"] as const
: binding.transport === "rest_api"
? ["baseUrl", "restPath", "restAuth"] as const
: ["username", "sshHost", "sshPort", "sshUsername", "sshTargetHost", "sshTargetPort"] as const;
for (const field of required) {
if (binding[field] === undefined) context.addIssue({ code: "custom", path: [field], message: "Required" });
}
});
const configSchema = z.object({
workspaceId: workspaceIdSchema,
engine: z.literal("postgres"),
databaseName: identifier,
schema: identifier,
binding: bindingSchema,
}).strict();
const updateSchema = configSchema.extend({ version: z.number().int().positive() });
const secretNames = [
"password",
"apiKey",
"sshPrivateKey",
"sshPrivateKeyPassphrase",
"sshKnownHosts",
"tlsCa",
] as const;
const secretsSchema = z.object({
version: z.number().int().positive(),
values: z.partialRecord(z.enum(secretNames), z.string().min(1).max(65_536)).refine((values) => Object.keys(values).length > 0),
}).strict();
const versionQuery = z.object({ version: z.coerce.number().int().positive() });
function safeError(reply: FastifyReply, error: unknown) {
if (error instanceof CatalogUnavailableError) {
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
}
if (error instanceof CatalogConflictError) {
return reply.code(409).send({ code: "database_conflict", message: "This workspace already has a database configuration." });
}
if (error instanceof CatalogOperationInProgressError) {
return reply.code(409).send({ code: "database_operation_in_progress", message: "A database operation is already in progress." });
}
if (error instanceof z.ZodError) {
return reply.code(400).send({ code: "database_invalid", message: "Database configuration is invalid." });
}
if (error instanceof WorkspaceRegistryError) {
return reply.code(400).send({ code: "database_invalid", message: "Database configuration is invalid." });
}
return reply.code(500).send({ code: "database_operation_failed", message: "Database operation failed." });
}
function manage(request: FastifyRequest, reply: FastifyReply) {
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
}
export function catalogDatabaseRoutes(
app: FastifyInstance,
deps: { repository: CatalogRepository; service: CatalogService; operations?: CatalogOperationCoordinator },
): void {
const mutate = async <T>(databaseId: string, operation: () => Promise<T>): Promise<T> => (
deps.operations ? await deps.operations.run(databaseId, operation) : await operation()
);
const activeSyncRun = async (databaseId: string) => {
const run = (await deps.repository.listSyncRuns(databaseId, 5)).find((candidate) =>
["queued", "running", "awaiting_confirmation", "applying"].includes(candidate.state));
if (!run) return undefined;
const {
observedSnapshot: _snapshot,
leaseOwner: _leaseOwner,
leaseExpiresAt: _leaseExpiresAt,
...summary
} = run;
return summary;
};
app.get("/catalog/status", async (request, reply) => {
if (!manage(request, reply)) return reply;
return { available: await deps.repository.available() };
});
app.get("/catalog/databases", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const rows = await deps.service.list();
return await Promise.all(rows.map(async (row) => (
row.id ? { ...row, activeSyncRun: await activeSyncRun(row.id) } : row
)));
} catch (error) { return safeError(reply, error); }
});
app.get("/catalog/databases/:id", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const id = idSchema.parse((request.params as { id?: unknown }).id);
const database = await deps.repository.get(id);
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
return {
...database,
configured: true,
secrets: deps.service.configuredSecrets(database.workspaceId),
activeSyncRun: await activeSyncRun(database.id),
};
} catch (error) { return safeError(reply, error); }
});
app.post("/catalog/databases", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const input = configSchema.parse(request.body) as DatabaseConfigurationInput;
const created = await deps.repository.create(await deps.service.normalizeInput(input));
return reply.code(201).send({ ...created, configured: true, secrets: deps.service.configuredSecrets(created.workspaceId) });
} catch (error) { return safeError(reply, error); }
});
app.patch("/catalog/databases/:id", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const id = idSchema.parse((request.params as { id?: unknown }).id);
const { version, ...input } = updateSchema.parse(request.body);
const current = await deps.repository.get(id);
if (!current) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
if (current.workspaceId !== input.workspaceId) {
return reply.code(400).send({ code: "database_invalid", message: "Database configuration is invalid." });
}
const updated = await mutate(id, async () => await deps.repository.update(
id, version, await deps.service.normalizeInput(input as DatabaseConfigurationInput),
));
if (!updated) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
return { ...updated, configured: true, secrets: deps.service.configuredSecrets(updated.workspaceId) };
} catch (error) { return safeError(reply, error); }
});
app.put("/catalog/databases/:id/secrets", async (request, reply) => {
if (!manage(request, reply)) return reply;
if (!isPrincipalContext(requirePermission(request, reply, "workspace.secrets.manage"))) return reply;
try {
const id = idSchema.parse((request.params as { id?: unknown }).id);
const { version, values } = secretsSchema.parse(request.body);
const database = await deps.repository.get(id);
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
if (database.version !== version) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
const updated = await mutate(id, async () => {
const touched = await deps.repository.touch(id, version);
if (touched) deps.service.replaceSecrets(database.workspaceId, values as Partial<Record<CatalogSecretName, string>>);
return touched;
});
if (!updated) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
return { ...updated, configured: true, secrets: deps.service.configuredSecrets(updated.workspaceId) };
} catch (error) { return safeError(reply, error); }
});
app.post("/catalog/databases/:id/test", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const id = idSchema.parse((request.params as { id?: unknown }).id);
const { version } = z.object({ version: z.number().int().positive() }).strict().parse(request.body);
const database = await deps.repository.get(id);
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
if (database.version !== version) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
const tested = await deps.repository.recordTest(id, version, await deps.service.test(database));
if (!tested) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
return { ...tested, configured: true, secrets: deps.service.configuredSecrets(tested.workspaceId) };
} catch (error) { return safeError(reply, error); }
});
app.delete("/catalog/databases/:id", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const id = idSchema.parse((request.params as { id?: unknown }).id);
const { version } = versionQuery.parse(request.query);
const database = await deps.repository.get(id);
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
const deleted = await mutate(id, async () => {
const removed = await deps.repository.delete(id, version);
if (removed) deps.service.forgetSecrets(database.workspaceId);
return removed;
});
if (!deleted) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
return reply.code(204).send();
} catch (error) { return safeError(reply, error); }
});
}
+230
View File
@@ -0,0 +1,230 @@
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { setTimeout as delay } from "node:timers/promises";
import { z } from "zod";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import type { CatalogSyncWorker } from "../catalog/sync-worker.js";
import {
CatalogConflictError,
CatalogConnectorError,
CatalogOperationInProgressError,
CatalogUnavailableError,
type CatalogRepository,
type CatalogSyncRun,
} from "../catalog/types.js";
const idSchema = z.uuid();
const metadataSchema = z.object({
version: z.number().int().positive(),
description: z.string().max(20_000).nullable(),
generatedDescription: z.string().max(20_000).nullable(),
}).strict();
const createRunSchema = z.object({
version: z.number().int().positive(),
scope: z.enum(["tables", "columns", "relationships", "all"]),
tableIds: z.array(idSchema).max(10_000).default([]),
}).strict();
const confirmationSchema = z.object({ confirmationToken: z.string().uuid() }).strict();
const eventQuerySchema = z.object({ after: z.coerce.number().int().nonnegative().default(0) });
function manage(request: FastifyRequest, reply: FastifyReply) {
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
}
function safeError(reply: FastifyReply, error: unknown) {
if (error instanceof CatalogUnavailableError) {
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
}
if (error instanceof CatalogConflictError || error instanceof CatalogOperationInProgressError) {
return reply.code(409).send({ code: "schema_sync_conflict", message: error.message });
}
if (error instanceof CatalogConnectorError) {
return reply.code(502).send({ code: "schema_introspection_failed", message: "The database schema could not be read safely." });
}
if (error instanceof z.ZodError) {
return reply.code(400).send({ code: "schema_request_invalid", message: "Schema request is invalid." });
}
return reply.code(500).send({ code: "schema_operation_failed", message: "Schema operation failed." });
}
function normalized(value: string | null): string | null {
return value?.trim() || null;
}
function publicRun(run: CatalogSyncRun) {
const {
observedSnapshot: _snapshot,
leaseOwner: _leaseOwner,
leaseExpiresAt: _leaseExpiresAt,
...result
} = run;
return result;
}
export function catalogSchemaRoutes(
app: FastifyInstance,
deps: { repository: CatalogRepository; worker: CatalogSyncWorker },
): void {
app.get("/catalog/databases/:databaseId/tables/:tableId/columns", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const params = request.params as { databaseId?: unknown; tableId?: unknown };
const databaseId = idSchema.parse(params.databaseId);
const tableId = idSchema.parse(params.tableId);
if (!(await deps.repository.getTable(databaseId, tableId))) {
return reply.code(404).send({ code: "table_not_found", message: "Catalog table was not found." });
}
return await deps.repository.listColumns(databaseId, tableId);
} catch (error) { return safeError(reply, error); }
});
app.patch("/catalog/databases/:databaseId/tables/:tableId/columns/:columnId", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const params = request.params as { databaseId?: unknown; tableId?: unknown; columnId?: unknown };
const databaseId = idSchema.parse(params.databaseId);
const tableId = idSchema.parse(params.tableId);
const columnId = idSchema.parse(params.columnId);
const input = metadataSchema.parse(request.body);
const current = await deps.repository.getColumn(databaseId, tableId, columnId);
if (!current) return reply.code(404).send({ code: "column_not_found", message: "Catalog column was not found." });
if (current.version !== input.version) {
return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
}
const updated = await deps.repository.updateColumnMetadata(
databaseId,
tableId,
columnId,
input.version,
normalized(input.description),
normalized(input.generatedDescription),
);
if (!updated) return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
return updated;
} catch (error) { return safeError(reply, error); }
});
app.get("/catalog/databases/:databaseId/relationships", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
if (!(await deps.repository.get(databaseId))) {
return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
}
return await deps.repository.listRelationships(databaseId);
} catch (error) { return safeError(reply, error); }
});
app.post("/catalog/databases/:databaseId/sync-runs", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
const input = createRunSchema.parse(request.body);
const database = await deps.repository.get(databaseId);
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
if (database.version !== input.version) {
return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
}
return reply.code(202).send(publicRun(await deps.worker.start(database, input.scope, input.tableIds)));
} catch (error) { return safeError(reply, error); }
});
app.get("/catalog/databases/:databaseId/sync-runs", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
if (!(await deps.repository.get(databaseId))) {
return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
}
return (await deps.repository.listSyncRuns(databaseId)).map(publicRun);
} catch (error) { return safeError(reply, error); }
});
app.get("/catalog/sync-runs/:runId", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
const run = await deps.repository.getSyncRun(runId);
return run ? publicRun(run) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
} catch (error) { return safeError(reply, error); }
});
app.post("/catalog/sync-runs/:runId/confirm", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
const { confirmationToken } = confirmationSchema.parse(request.body);
const run = await deps.worker.confirm(runId, confirmationToken);
return run ? publicRun(run) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
} catch (error) { return safeError(reply, error); }
});
app.post("/catalog/sync-runs/:runId/cancel", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
const run = await deps.worker.cancel(runId);
return run ? publicRun(run) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
} catch (error) { return safeError(reply, error); }
});
app.post("/catalog/sync-runs/:runId/retry", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
const run = await deps.worker.retry(runId);
return run ? reply.code(202).send(publicRun(run)) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
} catch (error) { return safeError(reply, error); }
});
app.get("/catalog/sync-runs/:runId/events", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
let after = eventQuerySchema.parse(request.query).after;
const headerCursor = Number(request.headers["last-event-id"]);
if (Number.isInteger(headerCursor) && headerCursor >= 0) after = Math.max(after, headerCursor);
if (!(await deps.repository.getSyncRun(runId))) {
return reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
}
reply.hijack();
reply.raw.writeHead(200, {
"content-type": "text/event-stream; charset=utf-8",
"cache-control": "no-cache, no-transform",
connection: "keep-alive",
"x-accel-buffering": "no",
});
const controller = new AbortController();
request.raw.once("close", () => controller.abort());
let lastRunUpdate = "";
while (!controller.signal.aborted) {
const events = await deps.repository.listSyncEvents(runId, after);
for (const event of events) {
after = event.sequence;
reply.raw.write(`id: ${event.sequence}\nevent: log\ndata: ${JSON.stringify(event)}\n\n`);
}
const run = await deps.repository.getSyncRun(runId);
if (!run) break;
if (run.updatedAt !== lastRunUpdate) {
lastRunUpdate = run.updatedAt;
reply.raw.write(`event: run\ndata: ${JSON.stringify(publicRun(run))}\n\n`);
}
if (["succeeded", "failed", "cancelled", "interrupted"].includes(run.state)) break;
await delay(500, undefined, { signal: controller.signal }).catch(() => undefined);
}
reply.raw.end();
return reply;
} catch (error) { return safeError(reply, error); }
});
app.get("/catalog/sync-runs/:runId/events-list", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
const after = eventQuerySchema.parse(request.query).after;
if (!(await deps.repository.getSyncRun(runId))) {
return reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
}
return await deps.repository.listSyncEvents(runId, after);
} catch (error) { return safeError(reply, error); }
});
}
+79
View File
@@ -0,0 +1,79 @@
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
import { z } from "zod";
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
import type { CatalogTableService } from "../catalog/table-service.js";
import {
CatalogConnectorError,
CatalogOperationInProgressError,
CatalogUnavailableError,
type CatalogRepository,
} from "../catalog/types.js";
const idSchema = z.uuid();
const updateSchema = z.object({
version: z.number().int().positive(),
description: z.string().max(20_000).nullable(),
generatedDescription: z.string().max(20_000).nullable().optional(),
}).strict();
function manage(request: FastifyRequest, reply: FastifyReply) {
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
}
function safeError(reply: FastifyReply, error: unknown) {
if (error instanceof CatalogUnavailableError) {
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
}
if (error instanceof CatalogOperationInProgressError) {
return reply.code(409).send({ code: "database_operation_in_progress", message: "A database operation is already in progress." });
}
if (error instanceof CatalogConnectorError) {
return reply.code(502).send({ code: "table_introspection_failed", message: "Database tables could not be read." });
}
if (error instanceof z.ZodError) {
return reply.code(400).send({ code: "table_invalid", message: "Table request is invalid." });
}
return reply.code(500).send({ code: "table_operation_failed", message: "Table operation failed." });
}
export function catalogTableRoutes(
app: FastifyInstance,
deps: { repository: CatalogRepository; service: CatalogTableService },
): void {
app.get("/catalog/databases/:databaseId/tables", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
if (!(await deps.repository.get(databaseId))) {
return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
}
return await deps.service.list(databaseId);
} catch (error) { return safeError(reply, error); }
});
app.patch("/catalog/databases/:databaseId/tables/:tableId", async (request, reply) => {
if (!manage(request, reply)) return reply;
try {
const params = request.params as { databaseId?: unknown; tableId?: unknown };
const databaseId = idSchema.parse(params.databaseId);
const tableId = idSchema.parse(params.tableId);
const input = updateSchema.parse(request.body);
const { version, description } = input;
const current = await deps.repository.getTable(databaseId, tableId);
if (!current) return reply.code(404).send({ code: "table_not_found", message: "Catalog table was not found." });
if (current.version !== version) {
return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
}
const updated = await deps.service.updateMetadata(
databaseId,
tableId,
version,
description,
input.generatedDescription === undefined ? current.generatedDescription : input.generatedDescription,
);
if (!updated) return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
return updated;
} catch (error) { return safeError(reply, error); }
});
}
+1
View File
@@ -187,6 +187,7 @@ test("roles collapse duplicates and admin contains all administrative permission
"settings.manage",
"workspace.manage",
"workspace.secrets.manage",
"database.manage",
"pi.manage",
"auth.diagnostics.read",
]);
+1 -1
View File
@@ -114,7 +114,7 @@ test.each([
const created = await fixture.app.thothiiAuthSessionStore?.create({
principal: {
issuer: "local", subject: user.id, displayName: user.username, roles: ["admin"],
permissions: ["session.use", "session.read_all", "session.manage_all", "settings.manage", "workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read"],
permissions: ["session.use", "session.read_all", "session.manage_all", "settings.manage", "workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read"],
isAdmin: true,
},
method: "local",
+1 -1
View File
@@ -130,7 +130,7 @@ test("local login sets a non-persistent opaque session cookie and exposes only a
roles: ["admin"],
permissions: [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
],
isAdmin: true,
csrfToken: expect.stringMatching(/^[A-Za-z0-9_-]{43}$/),
+2 -2
View File
@@ -32,7 +32,7 @@ test("local mode resolves a stable local principal", async () => {
roles: ["admin"],
permissions: [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
],
isAdmin: true,
});
@@ -74,7 +74,7 @@ test("upstream mode accepts only normalized proxy principal headers", async () =
issuer: "portal", subject: "42", displayName: "Alice", roles: ["user", "admin"],
permissions: [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
],
isAdmin: true,
});
+2 -2
View File
@@ -15,14 +15,14 @@ const admin: PrincipalContext = {
issuer: "oidc", subject: "admin", roles: ["admin"],
permissions: [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
],
isAdmin: true,
};
const catalog: readonly Permission[] = [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
];
test("permission matrix gives role-less identities no access, users session use, and admins every catalog permission", () => {
@@ -0,0 +1,122 @@
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, expect, test, vi } from "vitest";
import { buildApp } from "../src/app.js";
import { loadConfig } from "../src/config.js";
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
import type { WorkspaceRegistry, WorkspaceRevision } from "../src/workspaces/registry.js";
import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
const roots: string[] = [];
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
const workspace: WorkspaceDescriptor = {
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
dwh: {
engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432,
supported_transports: ["postgres_direct", "rest_api"],
},
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } },
};
const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" };
function setup() {
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-secret-"));
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-secret-runtime-"));
roots.push(secretRoot, runtimeRoot);
const secretStore = new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" });
const repository = new MemoryCatalogRepository();
const registry = {
list: vi.fn(async () => [revision]),
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
read: vi.fn(async () => ({ workspace, revision })),
} as unknown as WorkspaceRegistry;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
thtRunner: {} as never,
workspaceRegistry: registry,
workspaceSecretStore: secretStore,
catalogRepository: repository,
workspaceDiagnoser: vi.fn(),
});
return { app, secretStore, repository };
}
const direct = {
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
};
test("lists every YAML workspace and creates its one database configuration", async () => {
const { app } = setup();
const initial = await app.inject({ method: "GET", url: "/catalog/databases" });
expect(initial.statusCode).toBe(200);
expect(initial.json()).toMatchObject([{ workspaceId: "psd-clinical", configured: false, databaseName: "warehouse" }]);
const created = await app.inject({ method: "POST", url: "/catalog/databases", payload: direct });
expect(created.statusCode).toBe(201);
expect(created.json()).toMatchObject({ configured: true, workspaceId: "psd-clinical", version: 1 });
expect((await app.inject({ method: "POST", url: "/catalog/databases", payload: direct })).statusCode).toBe(409);
const listed = await app.inject({ method: "GET", url: "/catalog/databases" });
expect(listed.json()).toMatchObject([{ configured: true, binding: { transport: "postgres_direct", host: "db.internal" } }]);
});
test("lists orphaned records and takes the REST diagnostic path from workspace YAML", async () => {
const { app, repository } = setup();
await repository.create({
workspaceId: "removed-workspace",
engine: "postgres",
databaseName: "legacy",
schema: "public",
binding: { transport: "postgres_direct", host: "legacy.internal", port: 5432, username: "reader" },
});
const created = await app.inject({
method: "POST",
url: "/catalog/databases",
payload: {
...direct,
binding: {
transport: "rest_api", baseUrl: "https://psd.example/api", restPath: "/client-controlled", restAuth: "bearer",
},
},
});
expect(created.statusCode).toBe(201);
expect(created.json()).toMatchObject({ binding: { restPath: "/health" } });
const rows = (await app.inject({ method: "GET", url: "/catalog/databases" })).json();
expect(rows).toEqual(expect.arrayContaining([
expect.objectContaining({ workspaceId: "removed-workspace", configured: true, workspaceAvailable: false }),
expect.objectContaining({ workspaceId: "psd-clinical", configured: true, workspaceAvailable: true }),
]));
});
test("uses optimistic versions, keeps secrets write-only, and hard-deletes only local configuration", async () => {
const { app, secretStore } = setup();
const created = (await app.inject({ method: "POST", url: "/catalog/databases", payload: direct })).json();
const stale = await app.inject({ method: "PATCH", url: `/catalog/databases/${created.id}`, payload: { ...direct, version: 99 } });
expect(stale.statusCode).toBe(409);
const secret = await app.inject({
method: "PUT", url: `/catalog/databases/${created.id}/secrets`,
payload: { version: 1, values: { password: "do-not-return-this" } },
});
expect(secret.statusCode).toBe(200);
expect(secret.body).not.toContain("do-not-return-this");
expect(secret.json()).toMatchObject({ version: 2, secrets: { password: true } });
expect(secretStore.has("psd-clinical", "catalog.dwh.password")).toBe(true);
const removed = await app.inject({ method: "DELETE", url: `/catalog/databases/${created.id}?version=2` });
expect(removed.statusCode).toBe(204);
expect(secretStore.has("psd-clinical", "catalog.dwh.password")).toBe(false);
expect((await app.inject({ method: "GET", url: "/catalog/databases" })).json()).toMatchObject([{ configured: false }]);
});
@@ -0,0 +1,171 @@
import { EventEmitter } from "node:events";
import { existsSync, mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { PassThrough } from "node:stream";
import type { ChildProcessWithoutNullStreams } from "node:child_process";
import type { Client, ClientConfig } from "pg";
import { afterEach, expect, test, vi } from "vitest";
import {
buildSshArguments,
ConcreteCatalogPostgresAccess,
} from "../src/catalog/postgres-access.js";
import { CATALOG_SECRET_IDS } from "../src/catalog/secrets.js";
import type { WorkspaceDatabase } from "../src/catalog/types.js";
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
const roots: string[] = [];
afterEach(() => {
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
});
function secretStore() {
const root = mkdtempSync(join(tmpdir(), "catalog-ssh-secrets-"));
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-ssh-runtime-"));
roots.push(root, runtimeRoot);
return new WorkspaceSecretStore({ root, runtimeRoot, installationId: "test" });
}
function sshDatabase(): WorkspaceDatabase {
return {
id: "11111111-1111-4111-8111-111111111111",
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
version: 4,
createdAt: "2026-08-27T08:00:00Z",
updatedAt: "2026-08-27T09:00:00Z",
connectionStatus: "reachable",
binding: {
transport: "ssh_tunnel",
username: "warehouse_reader",
sshHost: "bastion.internal",
sshPort: 2222,
sshUsername: "tunnel_user",
sshTargetHost: "postgres.internal",
sshTargetPort: 5432,
tlsServername: "postgres.internal",
},
};
}
function fakeChild(): ChildProcessWithoutNullStreams {
const child = new EventEmitter() as EventEmitter & {
stdin: PassThrough;
stdout: PassThrough;
stderr: PassThrough;
exitCode: number | null;
signalCode: NodeJS.Signals | null;
kill: (signal?: NodeJS.Signals | number) => boolean;
};
child.stdin = new PassThrough();
child.stdout = new PassThrough();
child.stderr = new PassThrough();
child.exitCode = null;
child.signalCode = null;
child.kill = vi.fn((signal: NodeJS.Signals | number = "SIGTERM") => {
child.signalCode = typeof signal === "string" ? signal : "SIGTERM";
child.emit("exit", null, child.signalCode);
return true;
});
return child as unknown as ChildProcessWithoutNullStreams;
}
test("builds a strict host-verified OpenSSH stdio tunnel", () => {
const args = buildSshArguments({
sshHost: "bastion.internal",
sshPort: 2222,
sshUsername: "tunnel_user",
targetHost: "postgres.internal",
targetPort: 5432,
privateKeyFile: "/runtime/id",
knownHostsFile: "/runtime/known_hosts",
passphraseFile: "/runtime/passphrase",
connectTimeoutMs: 5_001,
});
expect(args).toEqual(expect.arrayContaining([
"-F", "/dev/null",
"-o", "BatchMode=no",
"-o", "StrictHostKeyChecking=yes",
"-o", "UserKnownHostsFile=/runtime/known_hosts",
"-o", "GlobalKnownHostsFile=/dev/null",
"-o", "IdentitiesOnly=yes",
"-o", "IdentityAgent=none",
"-o", "PasswordAuthentication=no",
"-o", "KbdInteractiveAuthentication=no",
"-o", "ConnectTimeout=6",
"-W", "postgres.internal:5432",
"--", "tunnel_user@bastion.internal",
]));
});
test("connects pg through OpenSSH, supplies askpass, and releases all secret leases", async () => {
const store = secretStore();
store.putMany("psd-clinical", {
[CATALOG_SECRET_IDS.password]: "db-password ",
[CATALOG_SECRET_IDS.sshPrivateKey]: "PRIVATE KEY\n",
[CATALOG_SECRET_IDS.sshPrivateKeyPassphrase]: "key-passphrase",
[CATALOG_SECRET_IDS.sshKnownHosts]: "bastion.internal ssh-ed25519 AAAATEST\n",
[CATALOG_SECRET_IDS.tlsCa]: "CA CERTIFICATE\n",
});
const child = fakeChild();
let clientConfig: ClientConfig | undefined;
let spawnCall: { command: string; args: readonly string[]; env: NodeJS.ProcessEnv } | undefined;
const end = vi.fn(async () => undefined);
const query = vi.fn(async () => ({ rows: [{ ok: true }] }));
const connect = vi.fn(async () => undefined);
const access = new ConcreteCatalogPostgresAccess(store, {
sshBinary: "/usr/bin/ssh",
askpassPath: "/app/ssh-askpass.mjs",
connectTimeoutMs: 5_000,
spawnSsh: (command, args, options) => {
spawnCall = { command, args, env: options.env };
return child;
},
createClient: (config) => {
clientConfig = config;
return { connect, query, end } as unknown as Client;
},
});
const client = await access.connect(sshDatabase(), new AbortController().signal);
expect(connect).toHaveBeenCalledOnce();
expect(clientConfig).toMatchObject({
host: "postgres.internal",
port: 5432,
database: "warehouse",
user: "warehouse_reader",
password: "db-password ",
connectionTimeoutMillis: 5_000,
ssl: {
ca: "CA CERTIFICATE\n",
servername: "postgres.internal",
rejectUnauthorized: true,
},
});
expect(clientConfig?.stream).toBeTypeOf("function");
expect(spawnCall?.command).toBe("/usr/bin/ssh");
expect(spawnCall?.args.some((argument) => argument.startsWith("IdentityFile="))).toBe(true);
expect(spawnCall?.args.some((argument) => argument.startsWith("UserKnownHostsFile="))).toBe(true);
expect(spawnCall?.env).toMatchObject({
DISPLAY: "thothii",
SSH_ASKPASS: "/app/ssh-askpass.mjs",
SSH_ASKPASS_REQUIRE: "force",
});
const leasedPaths = spawnCall!.args
.filter((argument) => argument.startsWith("IdentityFile=") || argument.startsWith("UserKnownHostsFile="))
.map((argument) => argument.slice(argument.indexOf("=") + 1));
leasedPaths.push(spawnCall!.env.THT_SSH_PASSPHRASE_FILE!);
expect(leasedPaths.every(existsSync)).toBe(true);
await expect(client.query("SELECT 1", [])).resolves.toEqual({ rows: [{ ok: true }] });
await client.end();
expect(end).toHaveBeenCalledOnce();
expect(child.kill).toHaveBeenCalledWith("SIGTERM");
expect(leasedPaths.some(existsSync)).toBe(false);
});
@@ -0,0 +1,125 @@
import { spawnSync } from "node:child_process";
import { PostgreSqlContainer } from "@testcontainers/postgresql";
import { CamelCasePlugin, Kysely, PostgresDialect, sql } from "kysely";
import { Pool } from "pg";
import { expect, test } from "vitest";
import { KyselyCatalogRepository, type CatalogDatabase } from "../src/catalog/repository.js";
import type { ObservedSchemaSnapshot } from "../src/catalog/types.js";
import { up as upDatabases } from "../src/catalog/migrations/001_workspace_databases.js";
import { up as upTables } from "../src/catalog/migrations/002_catalog_tables.js";
import { up as upSchemaSync } from "../src/catalog/migrations/003_catalog_schema_sync.js";
import { up as upRuntimeSequencePrivileges } from "../src/catalog/migrations/004_catalog_runtime_sequence_privileges.js";
const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per workspace and optimistic updates", async () => {
const container = await new PostgreSqlContainer("postgres:17.6-bookworm").start();
const db = new Kysely<CatalogDatabase>({
dialect: new PostgresDialect({ pool: new Pool({ connectionString: container.getConnectionUri() }) }),
plugins: [new CamelCasePlugin()],
});
try {
await upDatabases(db);
await upTables(db);
await upSchemaSync(db);
await sql`CREATE ROLE thothii_catalog_runtime`.execute(db);
await upRuntimeSequencePrivileges(db);
const sequencePrivilege = await sql<{ allowed: boolean }>`
SELECT has_sequence_privilege(
'thothii_catalog_runtime',
'catalog_sync_events_id_seq',
'USAGE'
) AS allowed
`.execute(db);
expect(sequencePrivilege.rows[0]?.allowed).toBe(true);
const repository = new KyselyCatalogRepository(db);
const input = {
workspaceId: "psd-clinical",
engine: "postgres" as const,
databaseName: "warehouse",
schema: "datawarehouse",
binding: { transport: "rest_api" as const, baseUrl: "https://psd.example/api", restPath: "/health", restAuth: "bearer" as const },
};
const created = await repository.create(input);
expect(created).toMatchObject({ version: 1, connectionStatus: "untested", binding: { transport: "rest_api" } });
await expect(repository.create(input)).rejects.toThrow("Workspace database already exists");
expect(await repository.update(created.id, 99, input)).toBeUndefined();
const synchronized = await repository.reconcileTables(created.id, 1, [
{ name: "patients", sourceComment: "Clinical patients" },
{ name: "visits", sourceComment: null },
], []);
expect(synchronized).toMatchObject({ kind: "applied", createdCount: 2, deletedCount: 0 });
const patients = (await repository.listTables(created.id))[0];
expect(await repository.updateTableDescription(
created.id,
patients.id,
patients.version,
"Curated patients",
)).toMatchObject({ description: "Curated patients", sourceComment: "Clinical patients", version: 2 });
const visits = (await repository.listTables(created.id)).find((table) => table.name === "visits")!;
const fullColumnsSnapshot: ObservedSchemaSnapshot = {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: [
{ name: "patients", sourceComment: "Clinical patients" },
{ name: "visits", sourceComment: null },
],
columns: [
{ tableName: "patients", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
{ tableName: "patients", name: "name", ordinalPosition: 2, dataType: "text", isNullable: true, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
{ tableName: "visits", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
{ tableName: "visits", name: "patient_id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
],
relationships: [],
};
expect(await repository.applySchemaSync(created.id, 1, "columns", [], fullColumnsSnapshot))
.toMatchObject({ created: 4, deleted: 0 });
expect((await repository.listColumns(created.id, patients.id)).map((column) => column.name))
.toEqual(["id", "name"]);
expect((await repository.listColumns(created.id, visits.id)).map((column) => column.name))
.toEqual(["id", "patient_id"]);
const reducedColumnsSnapshot: ObservedSchemaSnapshot = {
...fullColumnsSnapshot,
columns: fullColumnsSnapshot.columns.filter((column) => column.name === "id"),
};
expect(await repository.planSchemaSync(created.id, "columns", [], reducedColumnsSnapshot)).toMatchObject({
deletedColumns: [
{ tableName: "patients", columnName: "name" },
{ tableName: "visits", columnName: "patient_id" },
],
});
expect(await repository.planSchemaSync(created.id, "columns", [patients.id], reducedColumnsSnapshot)).toMatchObject({
deletedColumns: [{ tableName: "patients", columnName: "name" }],
});
expect(await repository.applySchemaSync(created.id, 1, "columns", [patients.id], reducedColumnsSnapshot))
.toMatchObject({ deleted: 1 });
expect((await repository.listColumns(created.id, patients.id)).map((column) => column.name))
.toEqual(["id"]);
expect((await repository.listColumns(created.id, visits.id)).map((column) => column.name))
.toEqual(["id", "patient_id"]);
expect(await repository.reconcileTables(created.id, 1, [
{ name: "patients", sourceComment: "Updated physical comment" },
], [])).toEqual({ kind: "confirmation_required", deletedNames: ["visits"] });
expect(await repository.reconcileTables(created.id, 1, [
{ name: "patients", sourceComment: "Updated physical comment" },
], ["visits"])).toMatchObject({ kind: "applied", updatedCount: 1, deletedCount: 1 });
const syncRun = await repository.createSyncRun(created.id, "columns", [patients.id], 1);
expect(syncRun).toMatchObject({
databaseId: created.id,
scope: "columns",
tableIds: [patients.id],
state: "queued",
});
expect(await repository.listSyncRuns(created.id)).toEqual([
expect.objectContaining({ id: syncRun.id, tableIds: [patients.id] }),
]);
expect(await repository.update(created.id, 1, { ...input, schema: "public" })).toMatchObject({ version: 2, schema: "public" });
expect(await repository.delete(created.id, 2)).toBe(true);
expect(await repository.list()).toEqual([]);
expect(await repository.listTables(created.id)).toEqual([]);
} finally {
await db.destroy();
await container.stop();
}
}, 60_000);
@@ -0,0 +1,194 @@
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, expect, test, vi } from "vitest";
import type { CatalogDatabaseClient, CatalogPostgresAccess } from "../src/catalog/postgres-access.js";
import { ConcreteCatalogSchemaIntrospector } from "../src/catalog/schema-introspector.js";
import {
CatalogSchemaCapabilityUnavailableError,
type WorkspaceDatabase,
} from "../src/catalog/types.js";
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
const roots: string[] = [];
afterEach(() => {
vi.unstubAllGlobals();
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
});
function store() {
const root = mkdtempSync(join(tmpdir(), "catalog-schema-introspection-secrets-"));
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-schema-introspection-runtime-"));
roots.push(root, runtimeRoot);
return new WorkspaceSecretStore({ root, runtimeRoot, installationId: "test" });
}
function database(binding: WorkspaceDatabase["binding"]): WorkspaceDatabase {
return {
id: "11111111-1111-4111-8111-111111111111",
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
binding,
version: 4,
connectionStatus: "reachable",
testedVersion: 4,
createdAt: "2026-08-27T08:00:00Z",
updatedAt: "2026-08-27T09:00:00Z",
};
}
test("reads columns, ordered composite keys, and physical relationships from one PostgreSQL connection", async () => {
const query = vi.fn()
.mockResolvedValueOnce({ rows: [{ present: true }] })
.mockResolvedValueOnce({ rows: [{ name: "visits", source_comment: "Visits" }] })
.mockResolvedValueOnce({ rows: [
{ table_name: "visits", name: "tenant_id", ordinal_position: 1, data_type: "uuid", is_nullable: false, default_expression: null, primary_key_position: 1, source_comment: null },
{ table_name: "visits", name: "patient_id", ordinal_position: 2, data_type: "bigint", is_nullable: false, default_expression: null, primary_key_position: 2, source_comment: "Patient" },
] })
.mockResolvedValueOnce({ rows: [
{ constraint_name: "visits_patient_fkey", source_table_name: "visits", target_table_name: "patients", update_action: "a", delete_action: "c", deferrable: true, initially_deferred: false, position: 1, source_column_name: "tenant_id", target_column_name: "tenant_id" },
{ constraint_name: "visits_patient_fkey", source_table_name: "visits", target_table_name: "patients", update_action: "a", delete_action: "c", deferrable: true, initially_deferred: false, position: 2, source_column_name: "patient_id", target_column_name: "id" },
] });
const end = vi.fn(async () => undefined);
const client: CatalogDatabaseClient = { query, end };
const postgres: CatalogPostgresAccess = { connect: vi.fn(async () => client) };
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
const result = await introspector.scan(database({
transport: "ssh_tunnel",
username: "reader",
sshHost: "bastion.internal",
sshPort: 22,
sshUsername: "tunnel",
sshTargetHost: "db.internal",
sshTargetPort: 5432,
}), new AbortController().signal);
expect(result.capabilities).toEqual({ tables: "available", columns: "available", relationships: "available" });
expect(result.columns).toMatchObject([
{ name: "tenant_id", primaryKeyPosition: 1, isNullable: false },
{ name: "patient_id", primaryKeyPosition: 2, sourceComment: "Patient" },
]);
expect(result.relationships).toEqual([expect.objectContaining({
constraintName: "visits_patient_fkey",
updateRule: "NO ACTION",
deleteRule: "CASCADE",
deferrable: true,
columns: [
{ position: 1, sourceColumnName: "tenant_id", targetColumnName: "tenant_id" },
{ position: 2, sourceColumnName: "patient_id", targetColumnName: "id" },
],
})]);
expect(query.mock.calls[2][0]).toContain("format_type");
expect(query.mock.calls[3][0]).toContain("WITH ORDINALITY");
expect(query.mock.calls.slice(1).every((call) => call[1][0] === "datawarehouse")).toBe(true);
expect(end).toHaveBeenCalledOnce();
});
test("uses the typed full REST snapshot RPC and preserves explicit capability unavailability", async () => {
const response = {
schemaVersion: 1,
capabilities: { tables: "available", columns: "unavailable", relationships: "unavailable" },
tables: [{ name: "patients", sourceComment: null }],
columns: [],
relationships: [],
};
const fetchMock = vi.fn(async () => new Response(JSON.stringify(response), { status: 200, headers: { "content-type": "application/json" } }));
vi.stubGlobal("fetch", fetchMock);
const postgres: CatalogPostgresAccess = { connect: vi.fn(async () => { throw new Error("wire access must not be used"); }) };
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
const result = await introspector.scan(database({
transport: "rest_api", baseUrl: "https://connector.internal/api/", restPath: "/health", restAuth: "none",
}), new AbortController().signal);
expect(result).toEqual(response);
expect(fetchMock).toHaveBeenCalledWith(
"https://connector.internal/api/rpc/schema_snapshot",
expect.objectContaining({ method: "POST", body: JSON.stringify({ schema_name: "datawarehouse" }) }),
);
});
test("falls back to one read-only REST query when the snapshot RPC is absent", async () => {
const response = {
schemaVersion: 1 as const,
capabilities: { tables: "available" as const, columns: "available" as const, relationships: "available" as const },
tables: [
{ name: "patients", sourceComment: "Clinical patients" },
{ name: "visits", sourceComment: null },
],
columns: [
{ tableName: "patients", name: "tenant_id", ordinalPosition: 1, dataType: "uuid", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: "Tenant" },
{ tableName: "patients", name: "id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: "nextval('patients_id_seq'::regclass)", primaryKeyPosition: 2, sourceComment: null },
{ tableName: "visits", name: "tenant_id", ordinalPosition: 1, dataType: "uuid", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
{ tableName: "visits", name: "patient_id", ordinalPosition: 2, dataType: "bigint", isNullable: true, defaultExpression: null, primaryKeyPosition: null, sourceComment: "Owning patient" },
],
relationships: [{
constraintName: "visits_patient_fkey",
sourceTableName: "visits",
targetTableName: "patients",
updateRule: "CASCADE",
deleteRule: "RESTRICT",
deferrable: true,
initiallyDeferred: false,
columns: [
{ position: 1, sourceColumnName: "tenant_id", targetColumnName: "tenant_id" },
{ position: 2, sourceColumnName: "patient_id", targetColumnName: "id" },
],
}],
};
const fetchMock = vi.fn()
.mockResolvedValueOnce(new Response(null, { status: 404 }))
.mockResolvedValueOnce(new Response(JSON.stringify([response]), {
status: 200,
headers: { "content-type": "application/json" },
}));
vi.stubGlobal("fetch", fetchMock);
const postgres: CatalogPostgresAccess = {
connect: vi.fn(async () => { throw new Error("wire access must not be used"); }),
};
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
const result = await introspector.scan(database({
transport: "rest_api",
baseUrl: "https://connector.internal/api/",
restPath: "/health",
restAuth: "none",
}), new AbortController().signal);
expect(result).toEqual(response);
expect(fetchMock).toHaveBeenCalledTimes(2);
expect(fetchMock.mock.calls[0]).toEqual([
"https://connector.internal/api/rpc/schema_snapshot",
expect.objectContaining({ method: "POST", body: JSON.stringify({ schema_name: "datawarehouse" }) }),
]);
expect(fetchMock.mock.calls[1][0]).toBe("https://connector.internal/api/rpc/run_query");
const fallbackRequest = fetchMock.mock.calls[1][1] as RequestInit;
expect(fallbackRequest).toMatchObject({ method: "POST" });
const fallbackBody = JSON.parse(String(fallbackRequest.body)) as { query_text: string };
expect(Object.keys(fallbackBody)).toEqual(["query_text"]);
expect(fallbackBody.query_text).toMatch(/^\s*WITH\b/);
expect(fallbackBody.query_text).toContain("pg_catalog.pg_constraint");
expect(fallbackBody.query_text).not.toMatch(/\b(INSERT|UPDATE|DROP|ALTER|CREATE|TRUNCATE)\b/i);
});
test("classifies a missing REST snapshot RPC as an explicit binding capability", async () => {
vi.stubGlobal("fetch", vi.fn(async () => new Response(null, { status: 404 })));
const postgres: CatalogPostgresAccess = {
connect: vi.fn(async () => { throw new Error("wire access must not be used"); }),
};
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
const scan = introspector.scan(database({
transport: "rest_api",
baseUrl: "https://connector.internal/api/",
restPath: "/health",
restAuth: "none",
}), new AbortController().signal);
await expect(scan).rejects.toMatchObject<CatalogSchemaCapabilityUnavailableError>({
capability: "schema_snapshot",
});
});
+224
View File
@@ -0,0 +1,224 @@
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, expect, test, vi } from "vitest";
import { buildApp } from "../src/app.js";
import { loadConfig } from "../src/config.js";
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
import type { CatalogSchemaIntrospector } from "../src/catalog/schema-introspector.js";
import type { CatalogSyncRun, ObservedSchemaSnapshot } from "../src/catalog/types.js";
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
import type { WorkspaceRegistry, WorkspaceRevision } from "../src/workspaces/registry.js";
import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
const roots: string[] = [];
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
const workspace: WorkspaceDescriptor = {
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
dwh: { engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432, supported_transports: ["postgres_direct"] },
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
};
const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" };
function snapshot(): ObservedSchemaSnapshot {
return {
schemaVersion: 1,
capabilities: { tables: "available", columns: "available", relationships: "available" },
tables: [
{ name: "patients", sourceComment: "Clinical patients" },
{ name: "visits", sourceComment: "Patient visits" },
],
columns: [
{ tableName: "patients", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: "Patient key" },
{ tableName: "visits", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
{ tableName: "visits", name: "patient_id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: "Owning patient" },
],
relationships: [{
constraintName: "visits_patient_id_fkey",
sourceTableName: "visits",
targetTableName: "patients",
updateRule: "NO ACTION",
deleteRule: "CASCADE",
deferrable: false,
initiallyDeferred: false,
columns: [{ position: 1, sourceColumnName: "patient_id", targetColumnName: "id" }],
}],
};
}
async function waitFor(repository: MemoryCatalogRepository, runId: string, state: CatalogSyncRun["state"]): Promise<CatalogSyncRun> {
for (let attempt = 0; attempt < 100; attempt += 1) {
const run = await repository.getSyncRun(runId);
if (run?.state === state) return run;
await new Promise((resolve) => setTimeout(resolve, 5));
}
throw new Error(`Run ${runId} did not reach ${state}`);
}
async function setup() {
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-schema-secret-"));
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-schema-runtime-"));
roots.push(secretRoot, runtimeRoot);
const repository = new MemoryCatalogRepository();
const created = await repository.create({
workspaceId: "psd-clinical", engine: "postgres", databaseName: "warehouse", schema: "datawarehouse",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
});
await repository.recordTest(created.id, created.version, {
connectionStatus: "reachable", testedVersion: created.version, lastTestedAt: new Date().toISOString(),
});
let observed = snapshot();
const scan = vi.fn(async (_database, _signal, progress) => {
await progress?.("connecting");
await progress?.("scanning_tables", { tables: observed.tables.length });
await progress?.("scanning_columns", { tables: observed.tables.length, columns: observed.columns.length });
await progress?.("scanning_relationships", { relationships: observed.relationships.length });
return structuredClone(observed);
});
const introspector: CatalogSchemaIntrospector = { scan };
const registry = {
list: vi.fn(async () => [revision]),
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
read: vi.fn(async () => ({ workspace, revision })),
} as unknown as WorkspaceRegistry;
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
thtRunner: {} as never,
workspaceRegistry: registry,
workspaceSecretStore: new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" }),
catalogRepository: repository,
catalogSchemaIntrospector: introspector,
workspaceDiagnoser: vi.fn(),
});
return {
app, repository, database: (await repository.get(created.id))!, scan,
setObserved(next: ObservedSchemaSnapshot) { observed = next; },
};
}
test("synchronizes a full physical schema and derives primary and foreign key flags", async () => {
const { app, repository, database } = await setup();
const started = await app.inject({
method: "POST", url: `/catalog/databases/${database.id}/sync-runs`,
payload: { version: database.version, scope: "all", tableIds: [] },
});
expect(started.statusCode).toBe(202);
const completed = await waitFor(repository, started.json().id, "succeeded");
expect(completed.counts).toMatchObject({ tables: 2, columns: 3, relationships: 1 });
const tables = await repository.listTables(database.id);
const visits = tables.find((table) => table.name === "visits")!;
const columns = (await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables/${visits.id}/columns` })).json();
expect(columns).toMatchObject([
{ name: "id", isPrimaryKey: true, primaryKeyPosition: 1, isForeignKey: false },
{ name: "patient_id", isPrimaryKey: false, isForeignKey: true, foreignKeyCount: 1 },
]);
const relationships = (await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/relationships` })).json();
expect(relationships).toMatchObject([{ constraintName: "visits_patient_id_fkey", columns: [{ sourceColumnName: "patient_id", targetColumnName: "id" }] }]);
expect((await repository.get(database.id))?.schemaSyncedVersion).toBe(database.version);
});
test("synchronizes columns for every catalog table when no table selection is supplied", async () => {
const { app, repository, database, setObserved } = await setup();
const tablesRun = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/sync-runs`,
payload: { version: database.version, scope: "tables", tableIds: [] },
});
expect(tablesRun.statusCode).toBe(202);
await waitFor(repository, tablesRun.json().id, "succeeded");
const tables = await repository.listTables(database.id);
expect(tables.map((table) => table.name)).toEqual(["patients", "visits"]);
const columnsRun = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/sync-runs`,
payload: { version: database.version, scope: "columns", tableIds: [] },
});
expect(columnsRun.statusCode).toBe(202);
await waitFor(repository, columnsRun.json().id, "succeeded");
const patients = tables.find((table) => table.name === "patients")!;
const visits = tables.find((table) => table.name === "visits")!;
expect((await repository.listColumns(database.id, patients.id)).map((column) => column.name)).toEqual(["id"]);
expect((await repository.listColumns(database.id, visits.id)).map((column) => column.name)).toEqual(["id", "patient_id"]);
const next = snapshot();
next.columns = next.columns.filter((column) => column.name !== "id");
setObserved(next);
const selectedDestructiveRun = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/sync-runs`,
payload: { version: database.version, scope: "columns", tableIds: [patients.id] },
});
expect(selectedDestructiveRun.statusCode).toBe(202);
const selectedWaiting = await waitFor(repository, selectedDestructiveRun.json().id, "awaiting_confirmation");
expect(selectedWaiting.plannedDiff?.deletedColumns).toEqual([
{ tableName: "patients", columnName: "id" },
]);
expect((await app.inject({
method: "POST",
url: `/catalog/sync-runs/${selectedWaiting.id}/cancel`,
})).statusCode).toBe(200);
const destructiveRun = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/sync-runs`,
payload: { version: database.version, scope: "columns", tableIds: [] },
});
expect(destructiveRun.statusCode).toBe(202);
const waiting = await waitFor(repository, destructiveRun.json().id, "awaiting_confirmation");
expect(waiting.plannedDiff?.deletedColumns).toEqual([
{ tableName: "patients", columnName: "id" },
{ tableName: "visits", columnName: "id" },
]);
});
test("keeps generated descriptions editable and preserves them across synchronization", async () => {
const { app, repository, database } = await setup();
const first = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
await waitFor(repository, first.json().id, "succeeded");
const patients = (await repository.listTables(database.id)).find((table) => table.name === "patients")!;
const editedTable = await app.inject({
method: "PATCH", url: `/catalog/databases/${database.id}/tables/${patients.id}`,
payload: { version: patients.version, description: null, generatedDescription: "Generated table draft" },
});
expect(editedTable.json()).toMatchObject({ description: null, generatedDescription: "Generated table draft" });
const idColumn = (await repository.listColumns(database.id, patients.id))[0];
const editedColumn = await app.inject({
method: "PATCH", url: `/catalog/databases/${database.id}/tables/${patients.id}/columns/${idColumn.id}`,
payload: { version: idColumn.version, description: "Reviewed key", generatedDescription: "Generated key draft" },
});
expect(editedColumn.json()).toMatchObject({ description: "Reviewed key", generatedDescription: "Generated key draft" });
const second = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
await waitFor(repository, second.json().id, "succeeded");
expect(await repository.getTable(database.id, patients.id)).toMatchObject({ generatedDescription: "Generated table draft" });
expect(await repository.getColumn(database.id, patients.id, idColumn.id)).toMatchObject({ description: "Reviewed key", generatedDescription: "Generated key draft" });
});
test("waits for confirmation and rescans before applying destructive changes", async () => {
const { app, repository, database, scan, setObserved } = await setup();
const first = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
await waitFor(repository, first.json().id, "succeeded");
const next = snapshot();
next.tables = next.tables.filter((table) => table.name !== "visits");
next.columns = next.columns.filter((column) => column.tableName !== "visits");
next.relationships = [];
setObserved(next);
const destructive = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
const waiting = await waitFor(repository, destructive.json().id, "awaiting_confirmation");
expect(waiting.plannedDiff).toMatchObject({ deletedTables: ["visits"] });
expect(await repository.listTables(database.id)).toHaveLength(2);
const confirmed = await app.inject({
method: "POST", url: `/catalog/sync-runs/${waiting.id}/confirm`, payload: { confirmationToken: waiting.confirmationToken },
});
expect(confirmed.statusCode).toBe(200);
await waitFor(repository, waiting.id, "succeeded");
expect((await repository.listTables(database.id)).map((table) => table.name)).toEqual(["patients"]);
expect(scan).toHaveBeenCalledTimes(3);
});
@@ -0,0 +1,124 @@
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, expect, test, vi } from "vitest";
import type {
CatalogDatabaseClient,
CatalogPostgresAccess,
} from "../src/catalog/postgres-access.js";
import { CATALOG_SECRET_IDS } from "../src/catalog/secrets.js";
import { ConcreteCatalogTableIntrospector } from "../src/catalog/table-introspector.js";
import type { WorkspaceDatabase } from "../src/catalog/types.js";
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
const roots: string[] = [];
afterEach(() => {
vi.unstubAllGlobals();
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
});
function secretStore() {
const root = mkdtempSync(join(tmpdir(), "catalog-table-introspection-secrets-"));
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-table-introspection-runtime-"));
roots.push(root, runtimeRoot);
return new WorkspaceSecretStore({ root, runtimeRoot, installationId: "test" });
}
function database(binding: WorkspaceDatabase["binding"]): WorkspaceDatabase {
return {
id: "11111111-1111-4111-8111-111111111111",
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
version: 4,
createdAt: "2026-08-27T08:00:00Z",
updatedAt: "2026-08-27T09:00:00Z",
connectionStatus: "reachable",
binding,
};
}
test("reads only ordinary and partitioned PostgreSQL tables from the configured schema", async () => {
const query = vi.fn()
.mockResolvedValueOnce({ rows: [{ present: true }] })
.mockResolvedValueOnce({ rows: [
{ name: "visits", source_comment: null },
{ name: "patients", source_comment: "Clinical patients" },
] });
const end = vi.fn(async () => undefined);
const client: CatalogDatabaseClient = { query, end };
const postgres: CatalogPostgresAccess = { connect: vi.fn(async () => client) };
const introspector = new ConcreteCatalogTableIntrospector(postgres, secretStore());
const tables = await introspector.scan(database({
transport: "postgres_direct",
host: "db.internal",
port: 5432,
username: "reader",
}), new AbortController().signal);
expect(tables).toEqual([
{ name: "patients", sourceComment: "Clinical patients" },
{ name: "visits", sourceComment: null },
]);
expect(query.mock.calls[1][0]).toContain("c.relkind IN ('r', 'p')");
expect(query.mock.calls[1][0]).not.toContain("'v'");
expect(query.mock.calls[1][1]).toEqual(["datawarehouse"]);
expect(end).toHaveBeenCalledOnce();
});
test("uses the typed REST table RPC and ignores non-table objects", async () => {
const store = secretStore();
store.put("psd-clinical", CATALOG_SECRET_IDS.apiKey, "rest-secret");
const fetchMock = vi.fn(async () => new Response(JSON.stringify([
{ type: "VIEW", table: "patient_view", comment: "Not a table" },
{ type: "TABLE", table: "visits", comment: null },
{ type: "TABLE", table: "patients", comment: "Clinical patients" },
]), { status: 200, headers: { "content-type": "application/json" } }));
vi.stubGlobal("fetch", fetchMock);
const postgres: CatalogPostgresAccess = {
connect: vi.fn(async () => { throw new Error("PostgreSQL wire access must not be used"); }),
};
const introspector = new ConcreteCatalogTableIntrospector(postgres, store);
const tables = await introspector.scan(database({
transport: "rest_api",
baseUrl: "https://connector.internal/api/",
restPath: "/health",
restAuth: "x-api-key",
}), new AbortController().signal);
expect(tables).toEqual([
{ name: "patients", sourceComment: "Clinical patients" },
{ name: "visits", sourceComment: null },
]);
expect(fetchMock).toHaveBeenCalledWith(
"https://connector.internal/api/rpc/list_tables",
expect.objectContaining({
method: "POST",
headers: expect.objectContaining({ "x-api-key": "rest-secret" }),
body: JSON.stringify({ schema_name: "datawarehouse" }),
}),
);
});
test("fails closed when a REST table row violates the typed contract", async () => {
const store = secretStore();
const fetchMock = vi.fn(async () => new Response(JSON.stringify([
{ type: "TABLE", table_name: "patients", comment: "Wrong field name" },
]), { status: 200, headers: { "content-type": "application/json" } }));
vi.stubGlobal("fetch", fetchMock);
const postgres: CatalogPostgresAccess = {
connect: vi.fn(async () => { throw new Error("PostgreSQL wire access must not be used"); }),
};
const introspector = new ConcreteCatalogTableIntrospector(postgres, store);
await expect(introspector.scan(database({
transport: "rest_api",
baseUrl: "https://connector.internal/api",
restPath: "/health",
restAuth: "none",
}), new AbortController().signal)).rejects.toThrow("REST schema response is invalid");
});
+126
View File
@@ -0,0 +1,126 @@
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, expect, test, vi } from "vitest";
import { buildApp } from "../src/app.js";
import { loadConfig } from "../src/config.js";
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
import type { WorkspaceRegistry, WorkspaceRevision } from "../src/workspaces/registry.js";
import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
const roots: string[] = [];
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
const workspace: WorkspaceDescriptor = {
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
dwh: {
engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432,
supported_transports: ["postgres_direct", "rest_api"],
},
semantic_index: {
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
},
llm_policy: { allowed: ["zai/glm-5.2"] },
diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } },
};
const revision: WorkspaceRevision = {
id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml",
};
async function setup() {
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-table-secret-"));
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-table-runtime-"));
roots.push(secretRoot, runtimeRoot);
const repository = new MemoryCatalogRepository();
const database = await repository.create({
workspaceId: "psd-clinical",
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
});
await repository.recordTest(database.id, database.version, {
connectionStatus: "reachable",
testedVersion: database.version,
lastTestedAt: new Date().toISOString(),
});
const scan = vi.fn(async () => [
{ name: "patients", sourceComment: "Clinical patients" },
{ name: "visits", sourceComment: null },
]);
const registry = {
list: vi.fn(async () => [revision]),
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
read: vi.fn(async () => ({ workspace, revision })),
} as unknown as WorkspaceRegistry;
const secretStore = new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" });
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
thtRunner: {} as never,
workspaceRegistry: registry,
workspaceSecretStore: secretStore,
catalogRepository: repository,
workspaceDiagnoser: vi.fn(),
});
return { app, repository, database: (await repository.get(database.id))!, scan };
}
test("lists physical tables and updates only review metadata", async () => {
const { app, repository, database, scan } = await setup();
const synchronized = await repository.reconcileTables(database.id, database.version, await scan(), []);
expect(synchronized).toMatchObject({ kind: "applied", createdCount: 2, deletedCount: 0 });
expect(scan).toHaveBeenCalledOnce();
const tables = (await app.inject({
method: "GET", url: `/catalog/databases/${database.id}/tables`,
})).json();
expect(tables.map((table: { name: string }) => table.name)).toEqual(["patients", "visits"]);
const patients = tables[0];
const edited = await app.inject({
method: "PATCH",
url: `/catalog/databases/${database.id}/tables/${patients.id}`,
payload: { version: patients.version, description: "Curated patient registry" },
});
expect(edited.statusCode).toBe(200);
expect(edited.json()).toMatchObject({
name: "patients",
sourceComment: "Clinical patients",
description: "Curated patient registry",
version: 2,
});
});
test("requires an exact deletion confirmation before applying the atomic diff", async () => {
const { app, repository, database, scan } = await setup();
await repository.reconcileTables(database.id, database.version, await scan(), []);
scan.mockResolvedValue([{ name: "patients", sourceComment: "Clinical patients" }]);
const preview = await repository.reconcileTables(database.id, database.version, await scan(), []);
expect(preview).toEqual({ kind: "confirmation_required", deletedNames: ["visits"] });
expect((await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables` })).json()).toHaveLength(2);
const applied = await repository.reconcileTables(database.id, database.version, await scan(), ["visits"]);
expect(applied).toMatchObject({ kind: "applied", deletedCount: 1 });
expect((await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables` })).json()).toMatchObject([
{ name: "patients" },
]);
});
test("refuses synchronization until the current binding has passed its connection test", async () => {
const { app, repository, database } = await setup();
await repository.update(database.id, database.version, {
workspaceId: database.workspaceId,
engine: database.engine,
databaseName: database.databaseName,
schema: database.schema,
binding: database.binding,
});
const response = await app.inject({
method: "POST",
url: `/catalog/databases/${database.id}/sync-runs`,
payload: { version: database.version + 1, scope: "tables", tableIds: [] },
});
expect(response.statusCode).toBe(409);
expect(response.json()).toMatchObject({ code: "schema_sync_conflict" });
});
+19
View File
@@ -289,3 +289,22 @@ test("loadConfig accepts only an absolute generic model key file", () => {
expect(() => loadConfig({ THT_MODEL_API_KEY_FILE: " /run/secrets/key" }))
.toThrow(/model credential configuration is invalid/);
});
test("loadConfig accepts a file-backed catalog role and rejects partial catalog configuration", () => {
expect(loadConfig({
THT_CATALOG_DB_HOST: "catalog-db",
THT_CATALOG_DB_NAME: "thothii_catalog",
THT_CATALOG_RUNTIME_USER: "thothii_catalog_runtime",
THT_CATALOG_RUNTIME_PASSWORD_FILE: "/run/secrets/catalog_runtime_password",
}).catalogDatabase).toEqual({
host: "catalog-db",
port: 5432,
database: "thothii_catalog",
user: "thothii_catalog_runtime",
passwordFile: "/run/secrets/catalog_runtime_password",
});
expect(() => loadConfig({ THT_CATALOG_DB_HOST: "catalog-db" }))
.toThrow(/catalog database configuration is invalid/);
expect(() => loadConfig({ THT_CATALOG_DATABASE_URL: "https://catalog.invalid/db" }))
.toThrow(/catalog database configuration is invalid/);
});
+58
View File
@@ -25,6 +25,11 @@ services:
THT_PI_AUTH_FILE: /home/thoth/.pi/agent/auth.json
THT_AUTH_CONFIG_FILE: /run/thothii-auth/auth.yaml
THT_AUTH_STATE_ROOT: /data/auth
THT_CATALOG_DB_HOST: catalog-db
THT_CATALOG_DB_PORT: "5432"
THT_CATALOG_DB_NAME: thothii_catalog
THT_CATALOG_RUNTIME_USER: thothii_catalog_runtime
THT_CATALOG_RUNTIME_PASSWORD_FILE: /run/secrets/catalog_runtime_password
THT_DB_NAME: ${THT_DB_NAME:-}
THT_DWH_REST_URL: ${THT_DWH_REST_URL:-}
THT_LLM_URL: ${THT_LLM_URL:-}
@@ -47,6 +52,7 @@ services:
secrets:
- source: thothii_secrets
target: thothii.secrets
- catalog_runtime_password
healthcheck:
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"]
interval: 15s
@@ -54,6 +60,8 @@ services:
retries: 5
start_period: 30s
depends_on:
catalog-db:
condition: service_healthy
qdrant:
condition: service_healthy
embedding-model-init:
@@ -61,6 +69,51 @@ services:
networks:
- thothii
catalog-db:
image: postgres:17.6-bookworm@sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3
environment:
POSTGRES_DB: thothii_catalog
POSTGRES_USER: thothii_catalog_migrate
POSTGRES_PASSWORD_FILE: /run/secrets/catalog_migrator_password
volumes:
- catalog-data:/var/lib/postgresql/data
- ./docker/catalog-db-init.sql:/docker-entrypoint-initdb.d/010-runtime-role.sql:ro
secrets:
- catalog_runtime_password
- catalog_migrator_password
healthcheck:
test:
- CMD-SHELL
- >-
pg_isready -U thothii_catalog_migrate -d thothii_catalog
&& test "$(psql -U thothii_catalog_migrate -d thothii_catalog -Atqc
"select count(*) from pg_catalog.pg_roles where rolname = 'thothii_catalog_runtime'")" = "1"
interval: 5s
timeout: 3s
retries: 12
start_period: 10s
networks:
- thothii
catalog-migrate:
image: thothii-core:local
profiles: [catalog-maintenance]
pull_policy: never
command: ["node", "/app/backend/dist/catalog/migrate.js"]
environment:
THT_CATALOG_DB_HOST: catalog-db
THT_CATALOG_DB_PORT: "5432"
THT_CATALOG_DB_NAME: thothii_catalog
THT_CATALOG_MIGRATOR_USER: thothii_catalog_migrate
THT_CATALOG_MIGRATOR_PASSWORD_FILE: /run/secrets/catalog_migrator_password
secrets:
- catalog_migrator_password
depends_on:
catalog-db:
condition: service_healthy
networks:
- thothii
workspace-maintenance:
image: thothii-core:local
profiles: [workspace-maintenance]
@@ -209,7 +262,12 @@ volumes:
qdrant-data:
embedding-models:
auth-state:
catalog-data:
secrets:
thothii_secrets:
file: "${THT_SECRETS_FILE:?set THT_SECRETS_FILE}"
catalog_runtime_password:
file: "${THT_CATALOG_RUNTIME_PASSWORD_SOURCE:-./deploy/secrets/catalog-runtime-password}"
catalog_migrator_password:
file: "${THT_CATALOG_MIGRATOR_PASSWORD_SOURCE:-./deploy/secrets/catalog-migrator-password}"
+3
View File
@@ -7,6 +7,9 @@ MAX_PI_PROCESSES=4
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
THT_CATALOG_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/catalog-runtime-password
THT_CATALOG_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/catalog-migrator-password
THT_CATALOG_SYNC_TIMEOUT_MS=600000
THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git
THT_WORKSPACE_GIT_BRANCH=main
+3
View File
@@ -6,6 +6,9 @@ MAX_PI_PROCESSES=4
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
THT_CATALOG_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/catalog-runtime-password
THT_CATALOG_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/catalog-migrator-password
THT_CATALOG_SYNC_TIMEOUT_MS=600000
THT_DATA_ROOT=/srv/thothii/data
THT_PI_STATE_ROOT=/srv/thothii/pi-state
+25
View File
@@ -0,0 +1,25 @@
DO $bootstrap$
DECLARE
runtime_password text := trim(both E'\r\n' from pg_read_file('/run/secrets/catalog_runtime_password'));
BEGIN
IF runtime_password = '' THEN
RAISE EXCEPTION 'catalog runtime password is empty';
END IF;
IF NOT EXISTS (
SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = 'thothii_catalog_runtime'
) THEN
EXECUTE format(
'CREATE ROLE thothii_catalog_runtime LOGIN PASSWORD %L',
runtime_password
);
END IF;
END
$bootstrap$;
GRANT CONNECT ON DATABASE thothii_catalog TO thothii_catalog_runtime;
GRANT USAGE ON SCHEMA public TO thothii_catalog_runtime;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO thothii_catalog_runtime;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT USAGE, SELECT ON SEQUENCES TO thothii_catalog_runtime;
+2
View File
@@ -96,7 +96,9 @@ RUN ln -s /opt/venv /app/harness/.venv
# Backend: dist + node_modules (stesso Node major 24 + glibc bookworm → compatibili)
COPY --from=backend-build /src/backend/dist /app/backend/dist
COPY --from=backend-build /src/backend/node_modules /app/backend/node_modules
COPY backend/scripts/ssh-askpass.mjs /app/backend/scripts/ssh-askpass.mjs
COPY backend/package*.json /app/backend/
RUN chmod 0755 /app/backend/scripts/ssh-askpass.mjs
# Runtime Pi is installed only from the committed lockfile. The image exposes its immutable
# executable directly, so no host Pi installation or writable global npm directory is needed.
+10 -5
View File
@@ -2,10 +2,12 @@
ThothII userà un Metadata Catalog PostgreSQL interno per conservare, per ogni workspace, la
struttura fisica acquisita interrogando il relativo database e i metadati semantici generati con
l'AI. Ogni workspace avrà un solo Workspace Database; identità e lista dei workspace resteranno
autorevoli in `thoth-workspaces.yaml`, mentre il catalogo ne conserverà soltanto il riferimento
stabile. `schema/annotations.yaml` verrà sostituito come input del core in uno step successivo;
l'interfaccia e il lifecycle amministrativi resteranno separati dal workflow NL→SQL.
l'AI. Ogni Workspace Database conserverà un `workspace_id` obbligatorio e univoco: questo realizza
l'associazione uno-a-uno senza introdurre nel catalogo una tabella Workspace o una foreign key SQL.
Identità e lista dei workspace resteranno autorevoli in `thoth-workspaces.yaml`; il servizio
validerà il riferimento contro quel catalogo. `schema/annotations.yaml` verrà sostituito come input
del core in uno step successivo; l'interfaccia e il lifecycle amministrativi resteranno separati dal
workflow NL→SQL.
## Considered Options
@@ -18,4 +20,7 @@ l'interfaccia e il lifecycle amministrativi resteranno separati dal workflow NL
PSD importerà le annotations esistenti; gli altri workspace genereranno i metadati da zero. Il
cutover futuro dovrà sostituire consapevolmente i consumatori delle annotations e verificarne
l'equivalenza semantica. Le sessioni di test esistenti non sono un vincolo di migrazione.
l'equivalenza semantica. PostgreSQL può garantire che uno stesso `workspace_id` non sia assegnato a
due database, ma l'esistenza del workspace e la gestione di rename o rimozioni restano responsabilità
del confine applicativo con il catalogo YAML. Le sessioni di test esistenti non sono un vincolo di
migrazione.
@@ -0,0 +1,19 @@
# Riferimenti al secret store per i Workspace Database
Il Metadata Catalog non conserverà credenziali o chiavi dei Workspace Database. Riuserà il secret
store cifrato già posseduto da ThothII e conserverà soltanto riferimenti ai requisiti segreti del
workspace, evitando un secondo vault e impedendo che API, esportazioni o log espongano i valori.
## Considered Options
- Copiare i campi testuali del modello ThothAI avrebbe semplificato il CRUD, ma avrebbe conservato
password e passphrase in chiaro.
- Introdurre subito un secondo vault avrebbe separato il catalogo dal runtime workspace, ma avrebbe
duplicato cifratura, rotazione, autorizzazioni e procedure operative senza un'esigenza distinta.
## Consequences
Il catalogo e il runtime condividono l'identità dei requisiti segreti, mentre permessi e API del
catalogo restano separati. Sostituzione e cancellazione di un Workspace Database dovranno definire
esplicitamente il lifecycle dei relativi riferimenti senza includere i valori nelle transazioni del
catalogo PostgreSQL.
@@ -0,0 +1,26 @@
# Binding di database specifiche dell'installazione
Il Metadata Catalog separa il Workspace Database logico dalla Database Binding che lo rende
raggiungibile in una specifica installazione. Esiste un solo Workspace Database per `workspace_id`
e una sola binding attiva nel catalogo di ciascuna installazione; per esempio PSD usa REST in locale
e PostgreSQL diretto sul server senza diventare due database distinti.
Nel modello finale il catalogo è autorevole per engine, nome fisico, schema, capability e binding,
mentre `thoth-workspaces.yaml` conserva l'identità del workspace. Gli attuali campi DWH dei
descriptor sono una sorgente di bootstrap da importare e confrontare durante un cutover esplicito,
non una seconda fonte di verità permanente.
## Considered Options
- Conservare un record database per ogni trasporto avrebbe duplicato identità, struttura e
metadati dello stesso DWH fra locale e server.
- Conservare permanentemente i dati DWH sia nello YAML sia nel catalogo avrebbe introdotto
conflitti non risolvibili deterministicamente.
- Rendere globali le binding avrebbe mescolato endpoint e credenziali che appartengono a
installazioni con topologie e confini di sicurezza differenti.
## Consequences
La lista amministrativa unisce workspace YAML, database configurati e record orphaned. Il cutover
deve importare e confrontare la configurazione esistente prima di rimuoverla dai descriptor; il
runtime non deve osservare simultaneamente due autorità discordanti.
@@ -0,0 +1,22 @@
# Metadata Catalog nello stesso backend con Kysely
Il Metadata Catalog vive nello stesso processo Fastify come modulo isolato, invece di introdurre un
microservizio. Usa il driver `pg` già presente attraverso Kysely per query, transazioni e migrazioni
tipizzate; route, repository, service, readiness e diagnostica restano separati dal workflow e
l'indisponibilità del catalogo non rende indisponibili sessioni o SSE.
## Considered Options
- Un microservizio avrebbe conservato letteralmente il backend bridge senza database, ma avrebbe
aggiunto deployment, autenticazione e failure mode per un solo contesto amministrativo.
- Usare soltanto `pg` avrebbe evitato una dipendenza, ma avrebbe richiesto infrastruttura locale per
transazioni, tipi delle righe, ordinamento e locking delle migrazioni.
- Drizzle o Prisma avrebbero aggiunto schema DSL, generatori e toolchain non necessari a un servizio
che vuole mantenere SQL e constraint PostgreSQL espliciti.
## Consequences
Il backend possiede una connection pool del catalogo e la chiude con il lifecycle Fastify. Le
migrazioni timestampate sono compilate insieme al backend ma vengono eseguite soltanto da
`catalog:migrate`, con credenziali migrator separate dal ruolo DML usato a runtime. Il modulo resta
dietro un'interfaccia repository per mantenere unit test e route test indipendenti da PostgreSQL.
@@ -0,0 +1,27 @@
# Hard-delete catalog tables during synchronization
An explicit Table Synchronization makes the Catalog Table membership exactly match a successful
observation of the Workspace Database: new tables are created, source metadata is refreshed, and
absent tables plus their future column and relationship children are permanently deleted. Physical
membership cannot be edited manually.
The external scan runs without holding a catalog transaction. Its diff is applied atomically only
while the Workspace Database version still matches the scanned binding. Failed scans change
nothing, and a non-empty removal set must exactly match the names confirmed by the operator; a
changed second scan therefore requires a new confirmation.
## Considered Options
- Soft deletion would preserve descriptions across accidental removals, but would add hidden state,
restore rules, and ambiguity about whether the catalog still represents the physical schema.
- Rename detection based on similarity would preserve metadata in some cases, but could silently
attach curated semantics to the wrong physical table.
- Append-only introspection, as in the legacy importer, would leave stale tables in the catalog and
make downstream schema linking unreliable.
## Consequences
A physical rename is delete plus create and loses curated metadata. The UI previews permanent
deletions, and future Catalog Column and Relationship records must cascade with their table. The
catalog remains an exact projection of the last accepted successful scan without tombstones or
restore lifecycle.
@@ -0,0 +1,8 @@
# Separate physical and logical relationships
ThothII persists each database-declared foreign-key constraint as an immutable Catalog
Relationship with ordered column pairs, so composite keys retain their identity and the database
remains the authority for physical structure. Curated or AI-inferred Logical Relationships will
use a separate future model and lifecycle rather than being mixed with physical constraints or
denormalized into textual column fields; this keeps synchronization authoritative without
preventing later semantic enrichment.
@@ -0,0 +1,9 @@
# Use durable runs for authoritative schema synchronization
All table, column, relationship, and full-schema synchronizations run as durable background
Catalog Sync Runs rather than separate synchronous and asynchronous implementations. Each scope
is authoritative within its boundary, while Synchronize All observes one complete schema snapshot;
destructive diffs require confirmation and source revalidation before an atomic, fail-closed
catalog transaction. A persistent per-database lock, progress events, interruption handling, and
binding-version freshness make long operations observable and prevent two processes or stale
configuration from producing a partially trusted catalog.
+7
View File
@@ -0,0 +1,7 @@
# Domain Docs
This is a single-context repository.
Before exploring, read `CONTEXT.md` at the repository root and the relevant decisions in
`docs/adr/`. Use the project's terminology from `CONTEXT.md` in issue titles, proposals, and
tests. Surface conflicts with an ADR instead of silently overriding it.
+28
View File
@@ -0,0 +1,28 @@
# Issue tracker: Gitea
Issues and specs for this repository live in the self-hosted Gitea repository:
`https://git.tylconsulting.it/mptyl/ThothII`.
## Conventions
- **Create an issue**: use the repository's Gitea web UI, or the Gitea REST API when an
authenticated token with issue scope is available.
- **Read and list issues**: use the Gitea web UI or authenticated API; include labels and comments.
- **Apply or remove labels**: use the issue's label controls or the Gitea API.
- **Comment and close**: use the issue page or the Gitea API.
- Do not use `gh issue ...` for this repository: the `github` remote is a mirror, not the canonical
issue tracker.
## Repository identity
- Canonical Git remote: `origin` → `https://git.tylconsulting.it/mptyl/ThothII.git`
- GitHub mirror: `github` → `https://github.com/mptyl/ThothII.git`
- Canonical issue URL: `https://git.tylconsulting.it/mptyl/ThothII/issues`
## When a skill says “publish to the issue tracker”
Create an issue in the canonical Gitea repository.
## When a skill says “fetch the relevant ticket”
Read the referenced issue in the canonical Gitea repository.
+19
View File
@@ -0,0 +1,19 @@
# Triage Labels
The skills speak in terms of five canonical triage roles. This file maps those roles to the labels
used in the canonical Gitea issue tracker.
| Label in mattpocock/skills | Label in Gitea | Meaning |
| --- | --- | --- |
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
| `needs-info` | `needs-info` | Waiting on reporter for more information |
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for AFK agent work |
| `ready-for-human` | `ready-for-human` | Requires human implementation |
| `wontfix` | `wontfix` | Will not be actioned |
Issue type labels:
| Label in Gitea | Meaning |
| --- | --- |
| `bug` | Something is not working |
| `enhancement` | New feature or request |
+7 -2
View File
@@ -4,7 +4,10 @@ This page complements the [architecture overview](overview.md) with the module s
## Modules and dependencies
The frontend communicates with the backend through REST and SSE. The backend does not own session persistence: it starts Pi, invokes the `tht` CLI, and forwards events. The harness contains the workflow, the Python CLI, and adapters for the DWH and vector store.
The frontend communicates with the backend through REST and SSE. The backend does not own session
persistence: it starts Pi, invokes the `tht` CLI, and forwards events. It does own the separate
installation-local database catalog. The harness contains the workflow, the Python CLI, and
adapters for the DWH and vector store.
```mermaid
flowchart LR
@@ -18,6 +21,8 @@ flowchart LR
THT --> DWH["DWH\nread-only"]
THT --> VDB["Qdrant / vector store"]
BE --> CFG["settings.json\nworkspace registry"]
BE --> CAT["catalog-db\nPostgreSQL + Kysely"]
BE -->|catalog Test + Table Sync| DWH
FE -.->|renders widgets| EXT
```
@@ -26,7 +31,7 @@ Dipendenze principali:
| Module | Depends on | Responsibility |
| --- | --- | --- |
| `frontend/` | Backend REST and SSE APIs | UI, gate widgets, and in-memory transcript |
| `backend/src/` | Pi, `tht`, configuration, and workspace registry | Transport, session lifecycle, and APIs |
| `backend/src/` | Pi, `tht`, configuration, workspace registry, catalog PostgreSQL, and read-only DWH connectors | Transport, session lifecycle, catalog CRUD, connection tests, table introspection, and APIs |
| `harness/.pi/` | Pi and `tht phase` | Workflow orchestration and human-in-the-loop gates |
| `harness/tht/` | Filesystem, DWH, and vector store | Persistence, CLI, Evidence, schema, and preprocessing |
| workspace repository | `source/`, `curated/`, manifest, and artifacts | Versioned Evidence source and session output |
+13 -4
View File
@@ -11,6 +11,7 @@ For sessions, roles, groups, diagnostics, and recovery, see the [authentication
flowchart LR
USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
FE --> BE["Backend\nFastify"]
BE --> CATALOG["Metadata catalog\nPostgreSQL"]
BE --> PI["Pi\nRPC per sessione"]
PI --> THT["tht and harness\nworkflow and persistence"]
THT --> DWH["DWH\nread only"]
@@ -27,8 +28,8 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
| Layer | Stack | Ruolo |
|---|---|---|
| **harness/** | Python (`tht` CLI) + Pi gate extension (JS) | Owns the workflow and **all** persistence |
| **backend/** | Fastify + TypeScript | Thin bridge with no database of its own |
| **harness/** | Python (`tht` CLI) + Pi gate extension (JS) | Owns the workflow and all session persistence |
| **backend/** | Fastify + TypeScript + Kysely | Session bridge plus the isolated administrative metadata catalog |
| **frontend/** | React 18 + Vite | UI that renders gate widgets and rebuilds the live transcript from the SSE stream |
## The harness owns the workflow
@@ -39,14 +40,22 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
A session is a directory under `sessions/` (the workspace defines the path): `session_manifest.yaml`, phase artifacts (`question.md`, `schema_linking.json`, `sql_final.sql`, and others), and `review_decisions.jsonl`. The contract says: *"persisted state is the truth; what is not recorded did not happen"*. There is no verbatim transcript store. A resumed Pi process rebuilds context from `tht session show <id>` and the artifacts on disk.
## The backend is a thin bridge with no database
## The backend bridges sessions and owns the metadata catalog
- `ThtRunner` runs `tht` subcommands in a shell.
- `PiProcessManager` runs one Pi child process per session and bridges its RPC stream.
- `SessionBridge` maps Pi RPC events to client events (`ui_request` / `text_delta` / `info`).
- `SseHub` distributes these events to the browser over SSE.
- `CatalogService` joins authoritative YAML workspace identities with installation-local database
configurations stored in PostgreSQL through Kysely.
- `CatalogTableService` reconciles persisted Catalog Tables with a successful external schema scan;
`ConcreteCatalogTableIntrospector` isolates direct PostgreSQL, typed REST, and SSH-tunnel access.
Application settings live in a JSON file (`backend/data/settings.json`), not in a database.
Application settings remain in `backend/data/settings.json`; session state remains in harness phase
documents. PostgreSQL stores only the administrative database catalog, bindings, observed tables,
and curated descriptions. Connector secrets remain write-only in the encrypted workspace secret
store. Catalog SSH support is limited to connection tests and table synchronization; it does not
change the session runtime binding contract.
## Human-in-the-loop gate contract
+88
View File
@@ -0,0 +1,88 @@
# Catalog schema snapshot RPC
Il percorso preferito per un binding `rest_api` espone al catalogo un'unica fotografia tipizzata
dello schema:
```http
POST /rpc/schema_snapshot
Content-Type: application/json
{"schema_name":"datawarehouse"}
```
La risposta è un oggetto JSON con `schemaVersion: 1`, capability esplicite e tre collezioni. Una
capability non disponibile deve essere dichiarata `unavailable`: non deve essere simulata con una
lista vuota.
```json
{
"schemaVersion": 1,
"capabilities": {
"tables": "available",
"columns": "available",
"relationships": "available"
},
"tables": [
{ "name": "patients", "sourceComment": "Clinical patients" }
],
"columns": [
{
"tableName": "patients",
"name": "id",
"ordinalPosition": 1,
"dataType": "bigint",
"isNullable": false,
"defaultExpression": null,
"primaryKeyPosition": 1,
"sourceComment": "Patient identifier"
}
],
"relationships": [
{
"constraintName": "visits_patient_id_fkey",
"sourceTableName": "visits",
"targetTableName": "patients",
"updateRule": "NO ACTION",
"deleteRule": "CASCADE",
"deferrable": false,
"initiallyDeferred": false,
"columns": [
{ "position": 1, "sourceColumnName": "patient_id", "targetColumnName": "id" }
]
}
]
}
```
## Fallback compatibile tramite `run_query`
Se e soltanto se il server non espone `POST /rpc/schema_snapshot`, il catalogo può ottenere la
stessa fotografia mediante una singola istruzione read-only inviata all'RPC già esistente:
```http
POST /rpc/run_query
Content-Type: application/json
{"query_text":"WITH ... SELECT ..."}
```
La query è costruita dal catalogo, interroga soltanto il catalogo PostgreSQL dello schema
configurato e aggrega tabelle, colonne, primary key e foreign key nella stessa istruzione. Non sono
ammessi più round trip, query per tabella o assemblaggi client-side di osservazioni effettuate in
momenti diversi. Il nome schema deve essere validato come identificatore e quotato come valore SQL,
non interpolato come SQL libero.
`run_query` restituisce un array JSON di righe. Per questo fallback l'array deve contenere
esattamente una riga e quella riga deve essere esattamente l'oggetto snapshot v1 sopra descritto,
con `schemaVersion`, `capabilities`, `tables`, `columns` e `relationships`; campi mancanti,
aggiuntivi o di tipo diverso rendono invalida l'intera fotografia. La risposta non è un contratto
alternativo o più permissivo: cambia soltanto il trasporto della stessa snapshot stretta.
`position` e `primaryKeyPosition` sono uno-based. Le coppie ordinate permettono foreign key
composte. Il catalogo rifiuta l'intera fotografia se il JSON non rispetta il contratto o se la
capability richiesta dal tipo di sincronizzazione è `unavailable`; in entrambi i casi non applica
alcuna modifica. Il fallback viene tentato soltanto quando l'RPC preferito risulta assente, non per
nascondere una snapshot malformata o un errore operativo del server. Se anche `run_query` non è
disponibile, la query viene rifiutata, la risposta non contiene una singola snapshot v1 valida o una
capability richiesta è `unavailable`, il run fallisce senza aggiornamenti parziali e senza esporre
il corpo remoto.
+4 -2
View File
@@ -92,8 +92,10 @@ evidence:
Changes from older workspaces:
- the database is reached only through **REST** or **direct Postgres** (`rest_api` /
`postgres_direct`); the SSH tunnel remains disabled;
- NL→SQL sessions reach the database only through **REST** or **direct Postgres** (`rest_api` /
`postgres_direct`); `ssh_tunnel` remains disabled for session runtime. The separate Database
management surface supports SSH for **Test connection** and **Sync tables**, with a private key,
mandatory `known_hosts`, and an optional key passphrase;
- the semantic index is **internal** (Qdrant plus `qwen3-embedding:0.6b`, 1024 dimensions, cosine);
- **filesystem** Evidence lives in the repository (`<id>/evidence`) and is materialized from the
pinned Git commit (P6). HTTP Evidence is also supported.
@@ -1,8 +1,9 @@
# Metadata Catalog di ThothII: ricognizione ThothAI e percorso incrementale
Data: 2026-08-26
Stato: ricognizione completata; step 1 implementato; scelte tecnologiche degli step successivi
deliberatamente rinviate.
Data: 2026-08-26; aggiornato 2026-08-27
Stato: ricognizione e progettazione completate; navigazione, CRUD Workspace Database, Catalog
Table, Catalog Column, Catalog Relationship e sincronizzazione durevole dello schema implementati
il 2026-08-27. Generazione AI e integrazione con il workflow core restano negli step successivi.
## Obiettivo
@@ -10,9 +11,11 @@ ThothII deve introdurre un contesto amministrativo separato, il **Metadata Catal
il database associato a ciascun workspace, la sua struttura fisica introspezionata e i metadati
semantici oggi rappresentati da `schema/annotations.yaml`.
Il programma procede per step indipendenti. Il primo step aggiunge soltanto l'accesso dalla sidebar
destra a una superficie centrale vuota. Non introduce PostgreSQL, API CRUD, introspezione o
integrazioni con il workflow core.
Il programma procede per step indipendenti. Il primo step ha aggiunto l'accesso dalla sidebar; il
secondo ha sostituito la superficie vuota con il CRUD di configurazione, il PostgreSQL interno e i
test di connessione; gli step successivi hanno aggiunto navigazione gerarchica, colonne, relazioni
fisiche e sincronizzazione durevole dell'intero schema. Non introduce ancora generazione AI o
integrazione con il workflow core.
Questa analisi usa come riferimento il working tree legacy osservato in
`Thoth/ThothAI`. Non è stato verificato che quel contenuto corrisponda a una release o a un tag
@@ -20,8 +23,10 @@ canonico; i percorsi e i comportamenti descrivono il sorgente disponibile il 202
## Decisioni già confermate
1. Ogni workspace è associato a un solo Workspace Database e ogni Workspace Database appartiene a
un solo workspace.
1. Ogni Workspace Database appartiene a un solo workspace tramite un `workspace_id` obbligatorio e
univoco; un workspace può avere al massimo un Workspace Database. Poiché i workspace non sono
righe del catalogo PostgreSQL, l'associazione è un riferimento logico validato contro
`thoth-workspaces.yaml`, non una foreign key SQL.
2. Il CRUD non crea né rinomina workspace. Identità e lista ordinata dei workspace restano
autorevoli in `thoth-workspaces.yaml`; il catalogo conserva il loro identificatore stabile.
3. La struttura fisica viene acquisita interrogando il database esterno tramite i dati di
@@ -38,6 +43,146 @@ canonico; i percorsi e i comportamenti descrivono il sorgente disponibile il 202
introduce un router.
9. La pagina iniziale è vuota, segue il tema, nasconde l'intera colonna core e non interrompe una
sessione live. Le azioni di apertura, resume o creazione sessione riportano al core.
10. La compatibilità con il modello ThothAI è semantica, non una copia letterale: configurazione e
contenuti semantici sono campi relazionali mutabili, mentre identità e appartenenza della
struttura fisica derivano dall'introspezione; i segreti restano nel secret store e lo stato dei
job non viene mescolato ai dati amministrativi.
11. Il CRUD amministra il Metadata Catalog e non esegue DDL sul database esterno, che resta
read-only.
12. La prima versione supporta PostgreSQL; il confine di introspezione dovrà permettere di
aggiungere altri dialetti senza cambiare il modello del catalogo.
13. I segreti dei Workspace Database riusano il secret store cifrato di ThothII. Il catalogo
conserva riferimenti ai segreti e nessuna API, esportazione o log ne restituisce i valori.
14. La UI usa AG Grid Community per la lista master e un pannello React separato per il dettaglio;
non dipende dalle funzionalità master-detail di AG Grid Enterprise.
15. Un Workspace Database il cui `workspace_id` scompare dal catalogo YAML non viene cancellato
automaticamente: diventa orphaned e può soltanto essere recuperato, riassegnato o eliminato
esplicitamente da un amministratore.
16. La prima vertical slice gestisce configurazione del Workspace Database, riferimenti ai segreti,
test di connessione e stato. La seconda gestisce le Catalog Table: la collezione e i nomi sono
controllati dall'introspezione, mentre la descrizione curata è modificabile. Le slice successive
hanno aggiunto Catalog Column, Catalog Relationship e sincronizzazione durevole dello schema.
17. Il modello non conserva il `name` libero di ThothAI: nome e ID visualizzati appartengono al
workspace YAML, mentre `database_name` identifica il database PostgreSQL esterno.
18. Database management supporta i tre trasporti già riconosciuti da ThothII: `postgres_direct`,
`rest_api` e `ssh_tunnel`. PSD rimane un solo Workspace Database: usa la connessione diretta sul
server e l'endpoint REST in locale tramite una Database Binding specifica dell'installazione.
Questo supporto non abilita automaticamente `ssh_tunnel` nel runtime NL→SQL.
19. Una configurazione può essere salvata prima di una connessione riuscita. Il test separato
produce uno stato `untested`, `reachable` o `failed`; attivazione e introspezione richiedono uno
stato raggiungibile.
20. Il CRUD e il test di connessione richiedono `database.manage`; inserimento e sostituzione dei
segreti continuano a richiedere `workspace.secrets.manage`.
21. Il Workspace Database e il modo di raggiungerlo sono entità distinte. Ogni catalogo di
installazione conserva una sola Database Binding attiva per workspace: PSD usa `rest_api` in
locale e `postgres_direct` sul server senza duplicare il Workspace Database.
22. Nel modello finale il Metadata Catalog è autorevole per engine, `database_name`, schema,
capacità e binding. Lo YAML resta autorevole per identità e contenuti del workspace; i campi
DWH correnti saranno importati, confrontati e rimossi soltanto durante un cutover esplicito.
23. La lista master è l'unione fra workspace YAML e record del catalogo: mostra workspace
`unconfigured`, database configurati e record `orphaned`.
24. Ogni introspezione registra le capability disponibili. Una capability `unavailable` non viene
rappresentata come una collezione osservata ma vuota; REST può completare con successo anche
quando indici o enum non sono supportati.
25. Il Metadata Catalog non introduce snapshot, draft o pubblicazioni. Configurazione e contenuti
semantici, inclusi quelli futuri generati dall'AI, sono normali campi modificabili; la struttura
osservata cambia soltanto con una sincronizzazione esplicita.
26. Il normale Delete elimina realmente il Workspace Database, la Database Binding e i relativi
record catalogo e segreti. Non modifica il DWH esterno né il repository YAML; il workspace torna
visibile nella lista master come `unconfigured`.
27. La prima versione gestisce un solo schema obbligatorio per Workspace Database, identificato
dalla coppia `database_name + schema`; per PSD la coppia è `postgres + datawarehouse`.
28. I record mantengono soltanto `created_at`, `updated_at` e un contatore `version` per optimistic
concurrency. Non esistono storico delle revisioni, rollback o audit applicativo delle modifiche.
29. `workspace_databases` conserva soltanto UUID, `workspace_id` unique, engine, `database_name`,
schema, timestamp e version. Il nome visualizzato appartiene al workspace YAML.
30. Ogni Workspace Database ha al massimo una riga `database_bindings`. Una singola tabella usa
check constraint dipendenti da `transport` per i campi direct, REST e SSH; non esiste un flag
`active`, perché ciascuna installazione conserva una sola binding.
31. `rest_api` configura il Thoth REST Connector tipizzato: base URL, autenticazione e TLS sono dati
della binding, mentre path RPC e shape delle risposte appartengono al contratto applicativo e non
sono liberamente configurabili.
32. Il test connessione usa soltanto una configurazione già salvata ed è associato alla sua
`version`. Ogni modifica della binding o dei segreti invalida il risultato precedente e riporta
lo stato a `untested`.
33. Password, API key e chiavi sono write-only: l'API espone soltanto `configured`, un campo vuoto
conserva il valore esistente e la sostituzione è un'azione esplicita. Delete rimuove anche i
segreti associati.
34. La pagina usa AG Grid come master e un form React come detail, con sezioni Database, Connection
e TLS/SSH condizionali. La toolbar offre `Add database`; le righe `unconfigured` offrono
`Configure`. Entrambe selezionano esclusivamente workspace YAML senza un database e creano il
record soltanto al Save; `workspace_id` diventa immutabile dopo la creazione.
35. La grid mostra workspace, database, schema, transport, endpoint, stato connessione e ultimo
aggiornamento. Su schermi piccoli il dettaglio occupa il pannello completo. Il cambio riga con
modifiche non salvate e Delete richiedono conferma, senza conferma testuale tipizzata.
36. La Database Binding conserva `connection_status`, `tested_version`, `last_tested_at`, un codice
errore e un messaggio breve sanificato. Non conserva stack trace, DSN, credenziali o output grezzo
del driver.
37. Le API vivono sotto `/api/catalog`: list/create di `/databases`, get/patch/delete di
`/databases/:id`, sostituzione dei segreti sotto `/databases/:id/secrets`, test connessione sotto
`/databases/:id/test` e list/patch/sync delle tabelle sotto `/databases/:id/tables`.
38. `GET /api/catalog/databases` restituisce l'intera master list unificata; AG Grid Community applica
client-side ricerca, filtri e ordinamento. La prima versione non introduce paginazione server o
funzionalità AG Grid Enterprise.
39. Il Metadata Catalog vive nello stesso processo Fastify come modulo isolato con repository,
service, route, diagnostica e readiness proprie. L'indisponibilità del catalogo non modifica
sessioni, SSE o health del core e non giustifica ancora un microservizio separato.
40. Il backend mantiene `pg@8.22.0` e aggiunge `kysely@0.29.5` per query e transazioni tipizzate. Le
migrazioni Kysely sono timestampate, compilate con il backend ed eseguite da un comando
`catalog:migrate` separato; l'applicazione non migra automaticamente il database all'avvio.
41. Lo stack aggiunge un servizio interno `catalog-db` con volume persistente, ruolo runtime DML,
ruolo migrator DDL e job one-shot `catalog-migrate`. Un catalogo indisponibile produce 503 sulle
sole route catalogo.
42. La prima vertical slice è amministrativa: scrive il catalogo ma non cambia ancora il runtime di
sessioni e workflow, che continua a usare YAML e binding correnti fino al cutover esplicito.
43. `Configure` precompila senza salvare engine, database e schema dal descriptor e i dati non
sensibili dalla binding effettiva. L'amministratore verifica, inserisce i segreti e salva; non
esiste importazione silenziosa.
44. Unit e route test usano un repository fake; una suite PostgreSQL Testcontainers separata verifica
migrazioni, constraint, transazioni, optimistic concurrency e cascade. SQLite ed emulatori non
sono sostituti ammessi per questi test.
45. La navigazione delle entità catalogo è gerarchica e senza scorciatoie globali: `Databases →
Database → Overview | Tables → Table`. Non esistono una voce globale Tables, un filtro globale
Database o una preselezione implicita; Columns continuerà sotto Table e Relationships sotto
Database.
46. Una Catalog Table conserva nome fisico, `source_comment`, descrizione curata nullable,
`generated_description` nullable per lo step AI futuro, version e timestamp. La UI mostra come
tre campi indipendenti senza fallback visivo: source comment read-only, generated description
modificabile e description modificabile. I valori null restano celle e controlli vuoti.
47. Le Catalog Table non possono essere aggiunte, rinominate o cancellate manualmente. `Sync tables`
legge dal database esterno le tabelle PostgreSQL ordinarie e partizionate dello schema scelto;
viste e materialized view sono escluse.
48. La sincronizzazione è esplicita. La scansione avviene fuori dalla transazione del catalogo; il
diff viene applicato atomicamente soltanto se la version del Workspace Database è ancora quella
sottoposta a scansione. Una scansione fallita non modifica il catalogo.
49. Tabelle nuove vengono create, i commenti sorgente vengono aggiornati e quelle non più osservate
vengono eliminate definitivamente. La rimozione di tabelle, colonne o relazioni richiede la
conferma dell'esatto piano distruttivo; se il secondo scan produce una fotografia differente,
l'applicazione richiede una nuova conferma.
50. Un rename fisico è intenzionalmente delete più create e perde i metadati curati. Le colonne e
relazioni dipendenti vengono eliminate in cascade insieme alla Catalog Table.
51. L'introspezione vive nel modulo catalogo Fastify dietro un adapter. PostgreSQL diretto e tunnel
SSH usano il catalogo `pg_catalog`; REST preferisce il contratto tipizzato
`POST /rpc/schema_snapshot` e, quando quell'RPC non è esposto, usa come fallback compatibile una
singola query read-only tramite `POST /rpc/run_query`. Entrambi i percorsi devono produrre la
stessa fotografia v1 stretta descritta in `docs/contracts/catalog-schema-snapshot.md`.
52. Test connessione e sincronizzazione sono serializzati per Workspace Database, hanno timeout e
richiedono che la binding nella version corrente abbia un test `reachable` prima di qualsiasi
Catalog Sync Run. La scansione asincrona ha un timeout separato, di default dieci minuti.
53. Il tunnel SSH usa OpenSSH in modalità stdio `-W`, chiave privata e passphrase opzionale dal
secret store, `known_hosts` obbligatorio, `StrictHostKeyChecking=yes`, agent e configurazione
globale disabilitati. Non è ammesso TOFU. TLS PostgreSQL con CA e server name resta verificato
anche attraverso il tunnel.
54. In questo slice `ssh_tunnel` è una binding supportata da Database management per Test connection
e Schema Sync. Il renderer e il runtime delle sessioni NL→SQL restano fuori scope e continuano a
rifiutarla finché non verrà deciso il relativo cutover.
55. I menu di azione a livello Workspace Database espongono separatamente `Synchronize tables`,
`Synchronize all columns`, `Synchronize relationships` e `Synchronize all`. Su una selezione di
database lo scope scelto viene avviato per ogni database idoneo; non viene sostituito
implicitamente con una sincronizzazione completa.
56. Per lo scope Columns, `tableIds` vuoto significa tutte le Catalog Table correnti del Workspace
Database; `tableIds` valorizzato limita invece la riconciliazione alle tabelle indicate. La grid
Tables espone `Synchronize columns` sulle tabelle selezionate.
## Correzione del modello mentale corrente
@@ -205,8 +350,8 @@ template Django.
Il comportamento è principalmente additivo: usa `get_or_create` o controlli `exists`, aggiorna
alcuni commenti, ma non riconcilia in modo completo rename, rimozioni o drift. Non va copiato così
com'è. Il futuro processo ThothII dovrà almeno distinguere scansione, differenze osservate e
applicazione della nuova snapshot.
com'è. Il processo ThothII implementato distingue scansione, differenze osservate e applicazione
della nuova snapshot.
### Generazione AI legacy
@@ -318,7 +463,8 @@ rendering, retrieval, LSH o SQL generation.
### Vincoli minimi da progettare
- `workspace_id` unico sul Workspace Database;
- `workspace_id` obbligatorio e unico sul Workspace Database, con esistenza validata contro il
catalogo YAML dal servizio applicativo;
- nome tabella unico nel database e schema appropriato;
- nome colonna unico nella tabella;
- relationship unica secondo il modello, anche per chiavi composite;
@@ -348,8 +494,7 @@ L'export legacy della struttura include username e password in chiaro. Il modell
password, passphrase SSH e altri segreti in `CharField`; non è stata trovata cifratura applicativa,
nonostante un testo admin affermi il contrario.
Per ThothII resta da decidere nello step infrastrutturale quali dati di connessione siano normali
metadati e quali siano secret reference. In ogni caso:
ThothII distingue i metadati di connessione dai riferimenti al secret store cifrato. In ogni caso:
- nessun endpoint o export deve restituire segreti;
- log ed errori devono sanificare DSN e credenziali;
@@ -357,6 +502,11 @@ metadati e quali siano secret reference. In ogni caso:
- il catalogo non deve riusare credenziali del DWH, delle sessioni o di Qdrant;
- test connessione e introspezione devono usare timeout e privilegi read-only.
La binding REST corrente richiede una verifica prima del cutover: il renderer emette
`ssl_ca_file`, mentre il modello Python espone `ssl_ca`; il percorso della CA privata potrebbe quindi
non essere consumato. PSD richiede TLS con CA privata in locale, perciò questo disallineamento deve
essere corretto e coperto da un test end-to-end prima di affidare il profilo REST al catalogo.
## Percorso incrementale
### Step 1: accesso alla superficie vuota
@@ -384,15 +534,17 @@ npx tsc -b
### Step 2: contratto di dominio e schema relazionale
Da progettare con un nuovo round decisionale: campi, secret reference, dialetti supportati,
namespace/schema, snapshot fisiche, relazioni fisiche/logiche e lifecycle dell'output AI. Nessuna
tecnologia ORM o migration tool è stata scelta in questo documento.
Progettazione della vertical slice completata: Workspace Database, Database Binding, singolo schema,
riferimenti al secret store, optimistic concurrency e capability per trasporto hanno contratti
espliciti. Configurazione e contenuti semantici restano mutabili; la struttura fisica osservata è
sincronizzata e non modificabile manualmente.
### Step 3: PostgreSQL interno e migrazioni
Da progettare separatamente dal core: servizio, volume, ruoli runtime/migrator/backup, health e
readiness dedicati, backup/restore e diagnostica. La sua indisponibilità non dovrà cambiare
`core /health` o interrompere una sessione.
PostgreSQL interno con volume e ruoli runtime/migrator separati. Il modulo catalogo usa Kysely sopra
il driver `pg`; le migrazioni compilate vengono applicate soltanto dal comando `catalog:migrate` e
mai allo startup Fastify. Health, readiness e diagnostica restano dedicate; l'indisponibilità del
catalogo non cambia `core /health` e non interrompe una sessione.
### Step 4: API CRUD
@@ -401,20 +553,56 @@ Gli endpoint dovranno vivere sotto un namespace catalogo e non riutilizzare le r
### Step 5: UI CRUD
Liste e form per Workspace Database, tabelle, colonne e relazioni, costruiti con React/Vite e il
design system ThothII. La gerarchia e i filtri ThothAI sono il riferimento funzionale; Django Admin
non è il riferimento tecnologico o visuale.
Workspace Database, Catalog Table, Catalog Column e Catalog Relationship sono implementati con
React/Vite e il design system ThothII.
La navigazione è gerarchica e locale al database (`Overview | Tables`), senza menu o filtri globali
per tipo di entità. La grid delle tabelle non offre Add/Delete; il dettaglio full-width mantiene
immutabili i fatti fisici e consente di modificare separatamente Description e Generated
Description. Colonne e relazioni seguono la stessa gerarchia: Columns appartiene al dettaglio
della tabella, Relationships al database. I valori descrittivi null sono mostrati come celle e
campi vuoti, senza fallback visivi o placeholder `Not set` che nascondano quale sorgente è
effettivamente valorizzata.
Le griglie che dispongono di azioni massive usano checkbox e una toolbar contestuale con conteggio,
menu `Actions` e cancellazione della selezione. La selezione identifica ID espliciti, può essere
accumulata attraverso i filtri e viene azzerata dopo successo, nuova sincronizzazione o uscita
dalla pagina; un'azione è all-or-nothing se un elemento non è idoneo. I menu a livello database
espongono gli scope fisici come azioni distinte: `Synchronize tables`, `Synchronize all columns`,
`Synchronize relationships` e `Synchronize all`. La grid Tables espone invece `Synchronize
columns` per le tabelle selezionate. Test connection resta un'azione distinta; griglie senza azioni
non mostrano controlli di selezione inerti.
### Step 6: introspezione
Connessione read-only, preview delle differenze, acquisizione di una Physical Schema Snapshot,
policy per rename/rimozioni e stato del job. Nessuna chiamata lunga dovrà mantenere aperta una
transazione CRUD.
Catalog Table, Catalog Column e Catalog Relationship sono implementate per PostgreSQL diretto,
Thoth REST Connector e tunnel SSH. La scansione read-only è separata dalla transazione; una
riconciliazione atomica crea, aggiorna i commenti sorgente ed elimina, dopo conferma, i fatti fisici
assenti senza rendere modificabile manualmente la struttura osservata. Gli scope autorevoli sono
Tables per database e Physical Relationships per database. Per Columns, `tableIds` vuoto include
tutte le Catalog Table correnti, mentre una lista di ID limita lo scope al sottoinsieme esplicito;
`Synchronize all` osserva tutti e tre gli scope in un unico snapshot e li riconcilia insieme. Tutti
gli scope sono eseguiti come Catalog Sync Run durevoli in background, non attraverso implementazioni
sincrone e asincrone separate. Un run che prevede cancellazioni conserva il diff, attende una
conferma esplicita e verifica nuovamente lo snapshot prima dell'applicazione; se la sorgente è
cambiata, invalida la conferma. Ogni applicazione è atomica e fail-closed: errori, timeout o
capability non disponibili non producono aggiornamenti parziali.
PK e FK devono essere visibili sulle Catalog Column senza duplicare le stringhe denormalizzate di
ThothAI. La posizione nella primary key è un fatto osservato della colonna; membership e conteggio
FK sono proiezioni derivate dalle Catalog Relationship e dalle loro coppie ordinate, aggiornate
nella stessa transazione di riconciliazione.
Ogni scope registra la versione della Database Binding osservata e l'istante dell'ultima
sincronizzazione. Una modifica della binding conserva il catalogo precedente ma lo marca stale;
solo un `Synchronize all` riuscito rende nuovamente corrente l'intero schema.
### Step 7: generazione AI dei metadati
Generazione di descrizioni e altri campi equivalenti alle annotations, editing umano e gestione
esplicita di errori o output non validi. Approvazione/versioning saranno decisi in questo step.
Generated Description è una proposta distinta e modificabile: un revisore può correggerla prima
di consolidarla esplicitamente come Description. Lo slice AI dovrà decidere e implementare anche
alias semantici, descrizioni dei valori, sinonimi e concetti per tabelle e colonne, oltre alla
gestione esplicita di errori e output non validi. La generazione AI e l'azione di consolidamento non
appartengono allo slice di introspezione dello schema.
### Step 8: migrazione PSD
@@ -427,11 +615,24 @@ ricevono import legacy.
Rimuovere la dipendenza da `annotations.yaml` soltanto dopo avere un contratto equivalente,
test di rendering/search/Qdrant e una policy di disponibilità. Le sessioni di test esistenti
possono essere eliminate, ma le nuove sessioni non devono osservare aggiornamenti parziali.
Questo cutover è esplicitamente rinviato fino al completamento del database dei metadati. Il primo
gate successivo obbligatorio sarà valutare l'integrazione del Catalog Schema Snapshot con il
workflow core e lo schema-linking corrente; il rinvio non autorizza a dimenticare o assorbire
implicitamente il lavoro in altri slice.
### Step 10: operazioni e accettazione
Backup/restore reale, diagnostica, metriche, audit, permessi definitivi, hardening degli export e
test di failure isolation fra catalogo e workflow.
Backup/restore reale, diagnostica, metriche, permessi definitivi, hardening degli export e
test di failure isolation fra catalogo e workflow. I Catalog Sync Run hanno un solo job attivo per
Workspace Database, sono concorrenti fra database diversi e usano un lock persistente. Un pannello
operativo non modale rimane visibile durante la navigazione del database, mostra fasi, contatori,
tempo trascorso e log sanitizzato via SSE con polling di fallback, e offre Confirm, Cancel e Retry
quando consentiti. Un restart marca `interrupted` i run rimasti attivi; il retry crea un nuovo run.
Le modifiche ai metadati restano consentite durante la scansione e sono preservate dall'applicazione.
Il worker gira inizialmente nello stesso servizio Fastify ma dietro un'interfaccia estraibile, con
coda, lease e heartbeat persistiti nel catalog-db. I riepiloghi dei run non scadono; gli eventi
dettagliati sono conservati per 30 giorni, mentre snapshot e diff completi vengono eliminati dopo
la conclusione lasciando conteggi, decisioni e una sintesi sanitizzata dell'esito.
## Verifiche del core da conservare per il cutover
@@ -476,15 +677,11 @@ Definiscono gli effetti semantici e le guardie da mantenere o sostituire consape
Le seguenti scelte non appartengono allo step 1:
- framework del servizio catalogo e libreria di accesso PostgreSQL;
- collocazione e protezione delle credenziali dei Workspace Database;
- supporto iniziale di dialetti diversi da PostgreSQL;
- uno o più schema namespace per database;
- policy di reconciliation per rename e delete;
- modello delle foreign key composite;
- distinzione persistente fra relationship fisiche e logiche;
- lifecycle draft/review/approval dell'output AI;
- versionamento, audit e rollback;
- lifecycle dei riferimenti ai segreti durante sostituzione e cancellazione;
- criteri per aggiungere dialetti successivi a PostgreSQL;
- criteri per un'eventuale estensione futura a più schemi per database;
- lifecycle e gestione amministrativa delle future Logical Relationship;
- alias semantici, descrizioni dei valori, sinonimi e concetti prodotti o assistiti dall'AI;
- formato e momento del cutover dal file al database interno;
- permission definitiva separata da `workspace.manage`.
+265
View File
@@ -0,0 +1,265 @@
import { apiFetch, assertSameOriginRequestUrl, BASE } from "./client";
import { joinBackendPath } from "./runtime-config";
export type DatabaseTransport = "postgres_direct" | "rest_api" | "ssh_tunnel";
export type ConnectionStatus = "untested" | "reachable" | "failed";
export type CatalogSecretName =
| "password"
| "apiKey"
| "sshPrivateKey"
| "sshPrivateKeyPassphrase"
| "sshKnownHosts"
| "tlsCa";
export interface DatabaseBinding {
transport: DatabaseTransport;
host?: string;
port?: number;
username?: string;
baseUrl?: string;
restPath?: string;
restAuth?: "none" | "bearer" | "x-api-key";
tlsServername?: string;
sshHost?: string;
sshPort?: number;
sshUsername?: string;
sshTargetHost?: string;
sshTargetPort?: number;
}
export interface CatalogDatabase {
id?: string;
workspaceId: string;
workspaceName: string;
workspaceDescription?: string;
workspaceAvailable: boolean;
configured: boolean;
engine: "postgres";
databaseName: string;
schema: string;
version: number;
createdAt: string;
updatedAt: string;
binding: DatabaseBinding;
connectionStatus: ConnectionStatus;
testedVersion?: number;
lastTestedAt?: string;
lastErrorCode?: string;
lastErrorMessage?: string;
schemaSyncedVersion?: number;
schemaSyncedAt?: string;
activeSyncRun?: CatalogSyncRun;
secrets: Record<CatalogSecretName, boolean>;
}
export interface DatabaseConfiguration {
workspaceId: string;
engine: "postgres";
databaseName: string;
schema: string;
binding: DatabaseBinding;
}
export interface CatalogTable {
id: string;
databaseId: string;
name: string;
sourceComment: string | null;
description: string | null;
generatedDescription: string | null;
lastSyncedDatabaseVersion?: number | null;
lastSyncedAt?: string | null;
version: number;
createdAt: string;
updatedAt: string;
}
export interface CatalogColumn {
id: string;
tableId: string;
name: string;
ordinalPosition: number;
dataType: string;
isNullable: boolean;
defaultExpression: string | null;
primaryKeyPosition: number | null;
isPrimaryKey: boolean;
isForeignKey: boolean;
foreignKeyCount: number;
sourceComment: string | null;
description: string | null;
generatedDescription: string | null;
lastSyncedDatabaseVersion: number | null;
lastSyncedAt: string | null;
version: number;
createdAt: string;
updatedAt: string;
}
export interface CatalogRelationshipColumn {
position: number;
sourceColumnId: string;
sourceColumnName: string;
targetColumnId: string;
targetColumnName: string;
}
export interface CatalogRelationship {
id: string;
databaseId: string;
constraintName: string;
sourceTableId: string;
sourceTableName: string;
targetTableId: string;
targetTableName: string;
updateRule: string;
deleteRule: string;
deferrable: boolean;
initiallyDeferred: boolean;
columns: CatalogRelationshipColumn[];
lastSyncedDatabaseVersion: number | null;
lastSyncedAt: string | null;
createdAt: string;
updatedAt: string;
}
export type CatalogSyncScope = "tables" | "columns" | "relationships" | "all";
export type CatalogSyncState = "queued" | "running" | "awaiting_confirmation" | "applying"
| "succeeded" | "failed" | "cancelled" | "interrupted";
export type CatalogSyncPhase = "queued" | "connecting" | "scanning_tables" | "scanning_columns"
| "scanning_relationships" | "planning" | "awaiting_confirmation" | "applying" | "completed";
export interface CatalogSchemaDiff {
deletedTables: string[];
deletedColumns: Array<{ tableName: string; columnName: string }>;
deletedRelationships: Array<{ sourceTableName: string; constraintName: string }>;
}
export interface CatalogSyncRun {
id: string;
databaseId: string;
scope: CatalogSyncScope;
tableIds: string[];
state: CatalogSyncState;
phase: CatalogSyncPhase;
requestedDatabaseVersion: number;
plannedDiff: CatalogSchemaDiff | null;
confirmationToken: string | null;
counts: { tables?: number; columns?: number; relationships?: number; created?: number; updated?: number; deleted?: number };
errorCode: string | null;
errorMessage: string | null;
cancelRequested: boolean;
createdAt: string;
startedAt: string | null;
updatedAt: string;
finishedAt: string | null;
heartbeatAt: string | null;
}
export interface CatalogSyncEvent {
id: number;
runId: string;
sequence: number;
level: "info" | "warning" | "error";
eventType: string;
message: string;
data: Record<string, unknown>;
createdAt: string;
}
export const listCatalogDatabases = () => apiFetch<CatalogDatabase[]>("/catalog/databases");
export const createCatalogDatabase = (input: DatabaseConfiguration) =>
apiFetch<CatalogDatabase>("/catalog/databases", { method: "POST", body: JSON.stringify(input) });
export const updateCatalogDatabase = (id: string, version: number, input: DatabaseConfiguration) =>
apiFetch<CatalogDatabase>(`/catalog/databases/${encodeURIComponent(id)}`, {
method: "PATCH",
body: JSON.stringify({ ...input, version }),
});
export const replaceCatalogDatabaseSecrets = (
id: string,
version: number,
values: Partial<Record<CatalogSecretName, string>>,
) => apiFetch<CatalogDatabase>(`/catalog/databases/${encodeURIComponent(id)}/secrets`, {
method: "PUT",
body: JSON.stringify({ version, values }),
});
export const testCatalogDatabase = (id: string, version: number) =>
apiFetch<CatalogDatabase>(`/catalog/databases/${encodeURIComponent(id)}/test`, {
method: "POST",
body: JSON.stringify({ version }),
});
export const deleteCatalogDatabase = (id: string, version: number) =>
apiFetch<void>(`/catalog/databases/${encodeURIComponent(id)}?version=${version}`, { method: "DELETE" });
export const listCatalogTables = (databaseId: string) =>
apiFetch<CatalogTable[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/tables`);
export const updateCatalogTableMetadata = (
databaseId: string,
tableId: string,
version: number,
description: string | null,
generatedDescription: string | null,
) => apiFetch<CatalogTable>(
`/catalog/databases/${encodeURIComponent(databaseId)}/tables/${encodeURIComponent(tableId)}`,
{ method: "PATCH", body: JSON.stringify({ version, description, generatedDescription }) },
);
export const listCatalogColumns = (databaseId: string, tableId: string) =>
apiFetch<CatalogColumn[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/tables/${encodeURIComponent(tableId)}/columns`);
export const updateCatalogColumnMetadata = (
databaseId: string,
tableId: string,
columnId: string,
version: number,
description: string | null,
generatedDescription: string | null,
) => apiFetch<CatalogColumn>(
`/catalog/databases/${encodeURIComponent(databaseId)}/tables/${encodeURIComponent(tableId)}/columns/${encodeURIComponent(columnId)}`,
{ method: "PATCH", body: JSON.stringify({ version, description, generatedDescription }) },
);
export const listCatalogRelationships = (databaseId: string) =>
apiFetch<CatalogRelationship[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/relationships`);
export const startCatalogSync = (
databaseId: string,
version: number,
scope: CatalogSyncScope,
tableIds: string[] = [],
) => apiFetch<CatalogSyncRun>(`/catalog/databases/${encodeURIComponent(databaseId)}/sync-runs`, {
method: "POST",
body: JSON.stringify({ version, scope, tableIds }),
});
export const listCatalogSyncRuns = (databaseId: string) =>
apiFetch<CatalogSyncRun[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/sync-runs`);
export const getCatalogSyncRun = (runId: string) =>
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}`);
export const listCatalogSyncEvents = (runId: string, after = 0) =>
apiFetch<CatalogSyncEvent[]>(`/catalog/sync-runs/${encodeURIComponent(runId)}/events-list?after=${after}`);
export const confirmCatalogSync = (runId: string, confirmationToken: string) =>
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}/confirm`, {
method: "POST", body: JSON.stringify({ confirmationToken }),
});
export const cancelCatalogSync = (runId: string) =>
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}/cancel`, { method: "POST" });
export const retryCatalogSync = (runId: string) =>
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}/retry`, { method: "POST" });
export function catalogSyncEventsUrl(runId: string, after = 0): string {
const url = joinBackendPath(BASE, `/catalog/sync-runs/${encodeURIComponent(runId)}/events?after=${after}`);
assertSameOriginRequestUrl(url);
return url;
}
+17
View File
@@ -14,6 +14,10 @@ const safeErrorCodes = new Set([
"git_unavailable", "git_auth_failed", "git_non_fast_forward", "connector_unavailable",
"semantic_index_incompatible", "pi_management_forbidden", "pi_management_unavailable",
"pi_management_invalid_config", "pi_management_write_failed",
"catalog_unavailable", "database_conflict", "database_invalid", "database_not_found",
"database_stale", "database_operation_failed",
"schema_sync_conflict", "schema_introspection_failed", "schema_request_invalid",
"schema_operation_failed", "sync_run_not_found", "table_stale", "column_stale",
]);
type SafeErrorPayload = {
@@ -46,6 +50,19 @@ const localCodeMessages: Record<string, string> = {
pi_management_unavailable: "Pi management is unavailable.",
pi_management_invalid_config: "The Pi configuration is invalid.",
pi_management_write_failed: "The Pi configuration could not be saved.",
catalog_unavailable: "The database catalog is unavailable.",
database_conflict: "This workspace already has a database configuration.",
database_invalid: "The database configuration is invalid.",
database_not_found: "The database configuration was not found.",
database_stale: "The database configuration changed. Reload and try again.",
database_operation_failed: "The database operation failed.",
schema_sync_conflict: "A schema synchronization is already active or no longer current.",
schema_introspection_failed: "The database schema could not be read safely.",
schema_request_invalid: "The schema request is invalid.",
schema_operation_failed: "The schema operation failed.",
sync_run_not_found: "The synchronization run was not found.",
table_stale: "Table metadata changed. Reload and try again.",
column_stale: "Column metadata changed. Reload and try again.",
};
const localStatusMessages: Record<number, string> = {
+49
View File
@@ -296,3 +296,52 @@
0%, 100% { transform: scale(1); box-shadow: 0 0 0 0 oklch(var(--primary) / 0.4); }
50% { transform: scale(1.15); box-shadow: 0 0 0 5px oklch(var(--primary) / 0); }
}
@layer components {
.thot-database-grid {
--ag-font-family: var(--font-sans);
--ag-font-size: 0.8125rem;
--ag-background-color: oklch(var(--card));
--ag-foreground-color: oklch(var(--foreground));
--ag-header-background-color: oklch(var(--muted) / 0.72);
--ag-header-foreground-color: oklch(var(--foreground));
--ag-border-color: oklch(var(--border));
--ag-row-border-color: oklch(var(--border) / 0.72);
--ag-odd-row-background-color: oklch(var(--muted) / 0.18);
--ag-row-hover-color: oklch(var(--muted) / 0.55);
--ag-selected-row-background-color: oklch(var(--primary) / 0.07);
--ag-input-focus-border-color: oklch(var(--primary) / 0.6);
--ag-range-selection-border-color: oklch(var(--primary) / 0.6);
--ag-header-column-separator-color: oklch(var(--border));
--ag-header-column-separator-display: block;
--ag-wrapper-border-radius: 0;
--ag-cell-horizontal-padding: 12px;
}
.thot-database-grid .ag-root-wrapper {
border: 0;
}
.thot-database-grid .ag-cell {
display: flex;
align-items: center;
}
.thot-database-grid .ag-header-cell-label {
font-weight: 700;
}
.thot-database-grid .ag-cell-focus:not(.ag-cell-range-selected) {
outline: 2px solid oklch(var(--ring) / 0.45);
outline-offset: -2px;
}
.thot-database-grid .ag-cell.thot-database-actions-cell {
padding-inline: 2px;
}
.thot-database-grid .ag-cell.thot-database-actions-cell.ag-cell-focus:not(.ag-cell-range-selected) {
outline: none;
}
.thot-database-grid .ag-cell.thot-database-status-cell {
padding-inline: 4px;
}
.thot-database-grid .ag-pinned-right-header,
.thot-database-grid .ag-pinned-right-cols-container {
box-shadow: -1px 0 0 oklch(var(--border));
}
}
@@ -23,7 +23,7 @@ beforeEach(() => {
issuer: "test",
subject: "test",
roles: ["admin"],
permissions: ["session.use", "workspace.manage", "workspace.secrets.manage", "pi.manage"],
permissions: ["session.use", "workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage"],
isAdmin: true,
csrfToken: null,
session: null,
@@ -52,7 +52,7 @@ beforeEach(() => {
);
});
test("opens the blank database management surface and returns to the core", async () => {
test("opens the database management surface and returns to the core", async () => {
renderShell();
const composer = screen.getByRole("textbox", { name: /new question/i });
@@ -122,7 +122,7 @@ test("keeps a live core session connected and returns when that session is opene
expect(FakeEventSource.instances).toHaveLength(1);
});
test("hides all management entries from non-admin users", () => {
test("keeps read-safe workspace access but hides privileged management entries", () => {
clearAuthState();
setAuthState({
issuer: "test",
@@ -136,7 +136,52 @@ test("hides all management entries from non-admin users", () => {
renderShell();
expect(screen.queryByRole("button", { name: "Workspace management" })).not.toBeInTheDocument();
expect(screen.getByRole("button", { name: "Workspace management" })).toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Database management" })).not.toBeInTheDocument();
expect(screen.queryByRole("button", { name: "Pi management" })).not.toBeInTheDocument();
});
test("does not leave database management without confirming a dirty form", async () => {
server.use(http.get("/api/catalog/databases", () => HttpResponse.json([{
id: "11111111-1111-4111-8111-111111111111",
workspaceId: "psd-clinical",
workspaceName: "Policlinico San Donato",
workspaceAvailable: true,
configured: true,
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
version: 3,
createdAt: "2026-08-27T08:00:00Z",
updatedAt: "2026-08-27T09:00:00Z",
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
connectionStatus: "untested",
secrets: {
password: false,
apiKey: false,
sshPrivateKey: false,
sshPrivateKeyPassphrase: false,
sshKnownHosts: false,
tlsCa: false,
},
}])));
const confirm = vi.spyOn(window, "confirm").mockReturnValue(false);
renderShell();
await userEvent.click(screen.getByRole("button", { name: "Database management" }));
await userEvent.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
const schema = screen.getByLabelText("Schema");
await userEvent.clear(schema);
await userEvent.type(schema, "reporting");
await userEvent.click(screen.getByRole("button", { name: "New session" }));
expect(confirm).toHaveBeenCalledWith("Discard unsaved database changes and leave database management?");
expect(screen.getByRole("main", { name: "Database management" })).toBeVisible();
expect(schema).toHaveValue("reporting");
confirm.mockReturnValue(true);
await userEvent.click(screen.getByRole("button", { name: "New session" }));
await waitFor(() => expect(screen.queryByRole("main", { name: "Database management" })).not.toBeInTheDocument());
confirm.mockRestore();
});
+49 -17
View File
@@ -32,7 +32,7 @@ import {
import type { SessionScope, SessionSummary } from "../api/types";
import { useAuthGeneration, useAuthUser } from "../auth/authState";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { useEffect, useMemo, useRef, useState } from "react";
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import type { CSSProperties } from "react";
import { captureAuthOperation, isAuthOperationCurrent, StaleAuthOperationError, type AuthOperationGuard } from "../auth/authOperation";
@@ -104,8 +104,8 @@ export function AppShell({ canLogout }: AppShellProps) {
const canReadAllSessions = permissions.includes("session.read_all");
const canManageWorkspace = permissions.includes("workspace.manage");
const canManageWorkspaceSecrets = permissions.includes("workspace.secrets.manage");
const canManageDatabase = permissions.includes("database.manage");
const canManagePi = permissions.includes("pi.manage");
const isAdmin = authenticatedUser?.isAdmin === true;
const authGeneration = useAuthGeneration();
const { data: sessions = [] } = useQuery<SessionSummary[]>({
queryKey: ["sessions", sessionScope],
@@ -123,6 +123,10 @@ export function AppShell({ canLogout }: AppShellProps) {
const queryClient = useQueryClient();
const [showActivity, setShowActivity] = useState(false);
const [activeSurface, setActiveSurface] = useState<ActiveSurface>("core");
const databaseNavigationRef = useRef({ dirty: false, busy: false });
const updateDatabaseNavigationState = useCallback((state: { dirty: boolean; busy: boolean }) => {
databaseNavigationRef.current = state;
}, []);
const [workspaceManagerOpen, setWorkspaceManagerOpen] = useState(false);
const [piManagementOpen, setPiManagementOpen] = useState(false);
const [activeOpen, setActiveOpen] = useState(true);
@@ -197,7 +201,20 @@ export function AppShell({ canLogout }: AppShellProps) {
setSelectedSessionIds(selected ? new Set(sessions.map((session) => session.id)) : new Set());
}
function canLeaveDatabaseManagement(): boolean {
if (activeSurface !== "database-management") return true;
if (databaseNavigationRef.current.busy) {
toast.info("Wait for the database operation to finish before leaving this page");
return false;
}
if (databaseNavigationRef.current.dirty) {
return window.confirm("Discard unsaved database changes and leave database management?");
}
return true;
}
function openPanel(id: string) {
if (!canLeaveDatabaseManagement()) return;
const s = sessions.find((x) => x.id === id);
if (!s) return;
setActiveSurface("core");
@@ -207,7 +224,7 @@ export function AppShell({ canLogout }: AppShellProps) {
// completed sessions keep the read-only documents panel (with its explicit Resume),
// so a mere click never spawns a runtime.
if (s.active && s.status === "open" && !s.archived && !isForeignSession(s)) {
void doResume(id);
void doResume(id, true);
return;
}
setPanelSession(s); setShowActivity(false);
@@ -219,7 +236,8 @@ export function AppShell({ canLogout }: AppShellProps) {
return next;
});
}
async function doResume(id: string) {
async function doResume(id: string, databaseExitApproved = false) {
if (!databaseExitApproved && !canLeaveDatabaseManagement()) return;
setActiveSurface("core");
const guard = captureAuthOperation({ sessionId: id, disposalEpoch: operationEpochRef.current });
if (!guard) return;
@@ -502,6 +520,7 @@ export function AppShell({ canLogout }: AppShellProps) {
}, [lastSystemEvent]);
function startNewSession() {
if (!canLeaveDatabaseManagement()) return;
setActiveSurface("core");
invalidateResumeIntent();
newSessionOperationRef.current = null;
@@ -566,6 +585,7 @@ export function AppShell({ canLogout }: AppShellProps) {
}
async function signOut() {
if (!canLeaveDatabaseManagement()) return;
await logoutUser();
}
@@ -619,7 +639,13 @@ export function AppShell({ canLogout }: AppShellProps) {
{/* Conversation column */}
<div data-testid="conversation-column" className="flex min-w-0 flex-1 flex-col">
{activeSurface === "database-management" && <DatabaseManagementPage />}
{activeSurface === "database-management" && (
<DatabaseManagementPage
canManage={canManageDatabase}
canManageSecrets={canManageWorkspaceSecrets}
onNavigationStateChange={updateDatabaseNavigationState}
/>
)}
<div
aria-hidden={activeSurface !== "core"}
className={activeSurface === "core" ? "contents" : "hidden"}
@@ -738,16 +764,19 @@ export function AppShell({ canLogout }: AppShellProps) {
>
New session
</Button>
{isAdmin && (
<>
<Button
variant="outline"
size="sm"
<Button
variant="outline"
size="sm"
className="w-full"
onClick={() => setWorkspaceManagerOpen(true)}
>
Workspace management
</Button>
onClick={() => {
if (!canLeaveDatabaseManagement()) return;
setActiveSurface("core");
setWorkspaceManagerOpen(true);
}}
>
Workspace management
</Button>
{canManageDatabase && (
<Button
variant="outline"
size="sm"
@@ -757,14 +786,17 @@ export function AppShell({ canLogout }: AppShellProps) {
>
Database management
</Button>
</>
)}
{isAdmin && canManagePi && (
{canManagePi && (
<Button
variant="outline"
size="sm"
className="w-full"
onClick={() => setPiManagementOpen(true)}
onClick={() => {
if (!canLeaveDatabaseManagement()) return;
setActiveSurface("core");
setPiManagementOpen(true);
}}
>
Pi management
</Button>
@@ -0,0 +1,842 @@
import { act, render, screen, waitFor, within } from "@testing-library/react";
import userEvent from "@testing-library/user-event";
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { http, HttpResponse } from "msw";
import { server } from "../test/msw";
import type { CatalogColumn, CatalogDatabase, CatalogSyncRun, CatalogTable } from "../api/catalog-databases";
import { DatabaseManagementPage } from "./DatabaseManagementPage";
const noSecrets = {
password: false,
apiKey: false,
sshPrivateKey: false,
sshPrivateKeyPassphrase: false,
sshKnownHosts: false,
tlsCa: false,
};
function makeDatabase(overrides: Partial<CatalogDatabase> = {}): CatalogDatabase {
return {
id: "11111111-1111-4111-8111-111111111111",
workspaceId: "psd-clinical",
workspaceName: "Policlinico San Donato",
workspaceAvailable: true,
configured: true,
engine: "postgres",
databaseName: "warehouse",
schema: "datawarehouse",
version: 3,
createdAt: "2026-08-27T08:00:00Z",
updatedAt: "2026-08-27T09:00:00Z",
binding: {
transport: "postgres_direct",
host: "db.internal",
port: 5432,
username: "reader",
},
connectionStatus: "untested",
secrets: { ...noSecrets },
...overrides,
};
}
const unconfigured = makeDatabase({
id: undefined,
workspaceId: "lab",
workspaceName: "Research laboratory",
configured: false,
databaseName: "lab_warehouse",
schema: "analytics",
version: 0,
createdAt: "",
updatedAt: "",
binding: { transport: "postgres_direct", port: 5432 },
});
const orphan = makeDatabase({
id: "22222222-2222-4222-8222-222222222222",
workspaceId: "retired",
workspaceName: "Retired workspace",
workspaceAvailable: false,
});
function renderPage({
rows = [makeDatabase(), unconfigured, orphan],
canManage = true,
canManageSecrets = true,
onNavigationStateChange,
}: {
rows?: CatalogDatabase[] | (() => CatalogDatabase[]);
canManage?: boolean;
canManageSecrets?: boolean;
onNavigationStateChange?: (state: { dirty: boolean; busy: boolean }) => void;
} = {}) {
server.use(http.get("/api/catalog/databases", () => HttpResponse.json(
typeof rows === "function" ? rows() : rows,
)));
const client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
const view = render(
<QueryClientProvider client={client}>
<DatabaseManagementPage
canManage={canManage}
canManageSecrets={canManageSecrets}
onNavigationStateChange={onNavigationStateChange}
/>
</QueryClientProvider>,
);
return { ...view, client };
}
function makeSyncRun(
scope: CatalogSyncRun["scope"],
tableIds: string[] = [],
): CatalogSyncRun {
return {
id: `sync-${scope}`,
databaseId: "11111111-1111-4111-8111-111111111111",
scope,
tableIds,
state: "queued",
phase: "queued",
requestedDatabaseVersion: 3,
plannedDiff: null,
confirmationToken: null,
counts: {},
errorCode: null,
errorMessage: null,
cancelRequested: false,
createdAt: "2026-08-27T10:00:00Z",
startedAt: null,
updatedAt: "2026-08-27T10:00:00Z",
finishedAt: null,
heartbeatAt: null,
};
}
function registerCompletedSyncRun(run: CatalogSyncRun) {
const completed: CatalogSyncRun = {
...run,
state: "succeeded",
phase: "completed",
startedAt: "2026-08-27T10:00:00Z",
finishedAt: "2026-08-27T10:00:01Z",
};
server.use(
http.get("/api/catalog/sync-runs/:runId", () => HttpResponse.json(completed)),
http.get("/api/catalog/sync-runs/:runId/events-list", () => HttpResponse.json([])),
http.get("/api/catalog/databases/:databaseId/sync-runs", () => HttpResponse.json([completed])),
);
}
const synchronizationScopes = [
{ scope: "tables", label: "Synchronize tables" },
{ scope: "columns", label: "Synchronize all columns" },
{ scope: "relationships", label: "Synchronize relationships" },
{ scope: "all", label: "Synchronize all" },
] as const;
test("starts with a full-width list and applies the row action matrix", async () => {
renderPage();
expect(await screen.findByRole("button", { name: "View Policlinico San Donato" })).toBeEnabled();
expect(screen.getByRole("button", { name: "Edit Policlinico San Donato" })).toBeEnabled();
expect(screen.getByRole("button", { name: "Delete Policlinico San Donato" })).toBeEnabled();
expect(screen.getByRole("button", { name: "View Research laboratory" })).toBeEnabled();
expect(screen.getByRole("button", { name: "Edit Research laboratory" })).toBeEnabled();
expect(screen.getByRole("button", { name: "Delete Research laboratory" })).toBeDisabled();
expect(screen.getByRole("button", { name: "View Retired workspace" })).toBeEnabled();
expect(screen.getByRole("button", { name: "Edit Retired workspace" })).toBeDisabled();
expect(screen.getByRole("button", { name: "Delete Retired workspace" })).toBeEnabled();
expect(screen.queryByRole("heading", { name: "Database details" })).not.toBeInTheDocument();
});
test.each(synchronizationScopes)(
"selected database Actions offers and starts $scope synchronization",
async ({ scope, label }) => {
const user = userEvent.setup();
let startBody: unknown;
const run = makeSyncRun(scope);
registerCompletedSyncRun(run);
server.use(http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
startBody = await request.json();
return HttpResponse.json(run, { status: 202 });
}));
renderPage({
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
});
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox"));
await user.click(screen.getByRole("button", { name: "Actions" }));
expect(await screen.findByRole("menuitem", { name: synchronizationScopes[0].label })).toBeEnabled();
for (const option of synchronizationScopes.slice(1)) {
expect(screen.getByRole("menuitem", { name: option.label })).toBeEnabled();
}
await user.click(screen.getByRole("menuitem", { name: label }));
await waitFor(() => expect(startBody).toEqual({ version: 3, scope, tableIds: [] }));
},
);
test("presents completed synchronization steps as success and skips unneeded confirmation", async () => {
const user = userEvent.setup();
const run = {
...makeSyncRun("columns"),
counts: { tables: 163, columns: 2_275 },
};
registerCompletedSyncRun(run);
server.use(http.post("/api/catalog/databases/:databaseId/sync-runs", () => (
HttpResponse.json(run, { status: 202 })
)));
renderPage({
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
});
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
await user.click(within(databaseRow).getByRole("checkbox"));
await user.click(screen.getByRole("button", { name: "Actions" }));
await user.click(await screen.findByRole("menuitem", { name: "Synchronize all columns" }));
expect(await screen.findByRole("heading", { name: "Succeeded" })).toBeVisible();
const steps = screen.getByRole("list", { name: "Synchronization steps" });
expect(steps).toHaveClass("grid-cols-2", "sm:grid-cols-3");
const connected = within(steps).getByRole("listitem", { name: "Connected, completed" });
expect(connected.querySelector("svg")).toHaveClass("text-[oklch(var(--success))]");
expect(within(steps).getByText("Tables read")).toBeVisible();
const confirmation = within(steps).getByRole("listitem", {
name: "Confirmation not required, not required",
});
expect(confirmation.querySelector("svg")).toHaveClass("text-muted-foreground");
const completed = within(steps).getByRole("listitem", { name: "Completed, completed" });
expect(completed.querySelector("svg")).toHaveClass("text-[oklch(var(--success))]");
expect(steps.querySelectorAll(".text-primary")).toHaveLength(0);
});
test("database Overview exposes every synchronization scope", async () => {
const user = userEvent.setup();
let startBody: unknown;
const run = makeSyncRun("relationships");
registerCompletedSyncRun(run);
server.use(http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
startBody = await request.json();
return HttpResponse.json(run, { status: 202 });
}));
renderPage({
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
});
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("button", { name: "Synchronize database schema" }));
expect(await screen.findByRole("menuitem", { name: synchronizationScopes[0].label })).toBeEnabled();
for (const option of synchronizationScopes.slice(1)) {
expect(screen.getByRole("menuitem", { name: option.label })).toBeEnabled();
}
await user.click(screen.getByRole("menuitem", { name: "Synchronize relationships" }));
await waitFor(() => expect(startBody).toEqual({
version: 3,
scope: "relationships",
tableIds: [],
}));
});
test("replaces the list with View and returns focus to the originating action", async () => {
const user = userEvent.setup();
renderPage({ rows: [makeDatabase()] });
const view = await screen.findByRole("button", { name: "View Policlinico San Donato" });
await user.click(view);
expect(screen.getByRole("heading", { name: "Database details" })).toBeVisible();
expect(screen.getByDisplayValue("db.internal")).toHaveAttribute("readonly");
expect(screen.getByRole("button", { name: "Test connection" })).toBeEnabled();
expect(screen.queryByRole("button", { name: "Save changes" })).not.toBeInTheDocument();
await user.click(screen.getByRole("button", { name: "Back to list" }));
await waitFor(() => expect(view).toHaveFocus());
expect(screen.getByRole("button", { name: "View Policlinico San Donato" })).toBeVisible();
});
test("opens an unconfigured row as Edit while creating its first saved configuration", async () => {
const user = userEvent.setup();
let postBody: unknown;
const saved = makeDatabase({
...unconfigured,
id: "33333333-3333-4333-8333-333333333333",
configured: true,
version: 1,
binding: { transport: "rest_api", baseUrl: "https://psd.example/api", restPath: "/health", restAuth: "x-api-key" },
});
let rows = [unconfigured];
server.use(
http.post("/api/catalog/databases", async ({ request }) => {
postBody = await request.json();
rows = [saved];
return HttpResponse.json(saved, { status: 201 });
}),
);
renderPage({ rows: () => rows });
await user.click(await screen.findByRole("button", { name: "Edit Research laboratory" }));
expect(screen.getByRole("heading", { name: "Edit database" })).toBeVisible();
expect(screen.getByLabelText("Workspace")).toBeDisabled();
await user.selectOptions(screen.getByLabelText("Transport"), "rest_api");
await user.type(screen.getByLabelText("Base URL"), "https://psd.example/api");
expect(screen.getByLabelText("Diagnostic endpoint")).toHaveValue("/health");
expect(screen.getByLabelText("Diagnostic endpoint")).toHaveAttribute("readonly");
await user.click(screen.getByRole("button", { name: "Save database" }));
await waitFor(() => expect(postBody).toMatchObject({
workspaceId: "lab",
binding: expect.objectContaining({ transport: "rest_api", baseUrl: "https://psd.example/api" }),
}));
expect(await screen.findByRole("heading", { name: "Edit database" })).toBeVisible();
});
test("global Add offers only unconfigured workspaces and lets the operator choose one", async () => {
const user = userEvent.setup();
const second = makeDatabase({
...unconfigured,
workspaceId: "radiology",
workspaceName: "Radiology",
databaseName: "radiology_dwh",
});
renderPage({ rows: [makeDatabase(), unconfigured, second] });
await user.click(await screen.findByRole("button", { name: "Add database" }));
expect(screen.getByRole("heading", { name: "Add database" })).toBeVisible();
expect(screen.getByRole("button", { name: "Add database" })).toBeVisible();
const selector = screen.getByLabelText("Workspace");
expect(selector).toBeEnabled();
expect(screen.queryByRole("option", { name: "Policlinico San Donato" })).not.toBeInTheDocument();
await user.selectOptions(selector, "radiology");
expect(screen.getByDisplayValue("radiology_dwh")).toBeVisible();
});
test("guards a dirty Edit, disables testing, and reports navigation state", async () => {
const user = userEvent.setup();
const navigation = vi.fn();
const confirm = vi.spyOn(window, "confirm").mockReturnValue(false);
renderPage({ rows: [makeDatabase()], onNavigationStateChange: navigation });
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
const schema = screen.getByLabelText("Schema");
await user.clear(schema);
await user.type(schema, "reporting");
expect(screen.getByRole("button", { name: "Test connection" })).toBeDisabled();
await waitFor(() => expect(navigation).toHaveBeenLastCalledWith({ dirty: true, busy: false }));
await user.click(screen.getByRole("button", { name: "Back to list" }));
expect(confirm).toHaveBeenCalledWith("Discard unsaved database changes?");
expect(screen.getByRole("heading", { name: "Edit database" })).toBeVisible();
confirm.mockReturnValue(true);
await user.click(screen.getByRole("button", { name: "Back to list" }));
expect(await screen.findByRole("button", { name: "Edit Policlinico San Donato" })).toBeVisible();
});
test("uses an in-page destructive form and returns the YAML workspace to Not configured", async () => {
const user = userEvent.setup();
let row = makeDatabase();
let deleteCalled = false;
let deleteVersion: string | null = null;
server.use(
http.delete("/api/catalog/databases/:id", ({ request }) => {
deleteCalled = true;
deleteVersion = new URL(request.url).searchParams.get("version");
row = {
...row,
id: undefined,
configured: false,
version: 0,
updatedAt: "",
connectionStatus: "untested",
secrets: { ...noSecrets },
};
return new HttpResponse(null, { status: 204 });
}),
);
renderPage({ rows: () => [row] });
await user.click(await screen.findByRole("button", { name: "Delete Policlinico San Donato" }));
expect(screen.getByRole("heading", { name: "Delete database" })).toBeVisible();
expect(screen.getByText("This removes the local database configuration.")).toBeVisible();
expect(screen.queryByRole("dialog")).not.toBeInTheDocument();
await user.click(screen.getByRole("button", { name: "Delete database" }));
await waitFor(() => expect(deleteCalled).toBe(true));
expect(deleteVersion).toBe("3");
expect(await screen.findByRole("button", { name: "Delete Policlinico San Donato" })).toBeDisabled();
expect(screen.getByText("Not configured")).toBeVisible();
});
test("tests only the persisted version and updates the visible connection status", async () => {
const user = userEvent.setup();
let testBody: unknown;
let row = makeDatabase();
server.use(http.post("/api/catalog/databases/:id/test", async ({ request }) => {
testBody = await request.json();
row = makeDatabase({ version: 4, connectionStatus: "reachable", testedVersion: 4 });
return HttpResponse.json(row);
}));
renderPage({ rows: () => [row] });
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("button", { name: "Test connection" }));
await waitFor(() => expect(testBody).toEqual({ version: 3 }));
expect(within(screen.getByRole("region", { name: "Database details form" })).getByText("Reachable")).toBeVisible();
});
test("retains a stale draft and requires an explicit reload", async () => {
const user = userEvent.setup();
server.use(
http.patch("/api/catalog/databases/:id", () => HttpResponse.json({
code: "database_stale",
message: "The database changed",
}, { status: 409 })),
);
renderPage({ rows: [makeDatabase()] });
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
const schema = screen.getByLabelText("Schema");
await user.clear(schema);
await user.type(schema, "draft_schema");
await user.click(screen.getByRole("button", { name: "Save changes" }));
expect(await screen.findByText("A newer database configuration is available.")).toBeVisible();
expect(schema).toHaveValue("draft_schema");
expect(screen.getByRole("button", { name: "Save changes" })).toBeDisabled();
await user.click(screen.getByRole("button", { name: "Keep editing" }));
expect(screen.getByText("Reload the latest values before this configuration can be changed.")).toBeVisible();
expect(schema).toHaveValue("draft_schema");
});
test("marks an open form stale when a background refetch advances the catalog version", async () => {
const user = userEvent.setup();
const { client } = renderPage({ rows: [makeDatabase()] });
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
act(() => {
client.setQueryData(["catalog-databases"], [makeDatabase({ version: 4, schema: "server_schema" })]);
});
expect(await screen.findByText("A newer database configuration is available.")).toBeVisible();
expect(screen.getByLabelText("Schema")).toHaveValue("datawarehouse");
expect(screen.getByRole("button", { name: "Save changes" })).toBeDisabled();
});
test("does not submit credentials hidden by a transport or authentication change", async () => {
const user = userEvent.setup();
let putCalled = false;
let patchBody: Record<string, unknown> | undefined;
let row = makeDatabase();
server.use(
http.patch("/api/catalog/databases/:id", async ({ request }) => {
patchBody = await request.json() as Record<string, unknown>;
row = makeDatabase({
version: 4,
binding: { transport: "rest_api", baseUrl: "https://psd.example/api", restPath: "/health", restAuth: "none" },
});
return HttpResponse.json(row);
}),
http.put("/api/catalog/databases/:id/secrets", () => {
putCalled = true;
return HttpResponse.json(row);
}),
);
renderPage({ rows: () => [row] });
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
await user.type(screen.getByLabelText("Password"), "must-not-be-sent");
await user.selectOptions(screen.getByLabelText("Transport"), "rest_api");
await user.type(screen.getByLabelText("Base URL"), "https://psd.example/api");
await user.selectOptions(screen.getByLabelText("Authentication"), "none");
await user.click(screen.getByRole("button", { name: "Save changes" }));
await waitFor(() => expect(patchBody).toBeDefined());
expect(putCalled).toBe(false);
expect(patchBody).toMatchObject({
version: 3,
binding: {
transport: "rest_api",
baseUrl: "https://psd.example/api",
restPath: "/health",
restAuth: "none",
},
});
expect((patchBody?.binding as Record<string, unknown>).host).toBeUndefined();
});
test("preserves typed secrets and offers a retry when the configuration save is only partial", async () => {
const user = userEvent.setup();
let putCalls = 0;
let row = makeDatabase();
server.use(
http.put("/api/catalog/databases/:id/secrets", async ({ request }) => {
putCalls += 1;
const body = await request.json();
expect(body).toEqual({ version: 3, values: { password: "transient-secret" } });
if (putCalls === 1) {
return HttpResponse.json({
code: "catalog_unavailable",
message: "Secret store unavailable",
}, { status: 503 });
}
row = makeDatabase({ version: 4, secrets: { ...noSecrets, password: true } });
return HttpResponse.json(row);
}),
);
renderPage({ rows: () => [row] });
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
await user.type(screen.getByLabelText("Password"), "transient-secret");
await user.click(screen.getByRole("button", { name: "Save changes" }));
expect(await screen.findByText("Database configuration saved; secret update could not be confirmed.")).toBeVisible();
expect(screen.getByLabelText("Password")).toHaveValue("transient-secret");
expect(screen.getByRole("button", { name: "Retry secrets" })).toBeEnabled();
await user.clear(screen.getByLabelText("Schema"));
await user.type(screen.getByLabelText("Schema"), "reporting_after_retry");
await user.click(screen.getByRole("button", { name: "Retry secrets" }));
await waitFor(() => expect(putCalls).toBe(2));
expect(screen.queryByText("Database configuration saved; secret update could not be confirmed.")).not.toBeInTheDocument();
expect(screen.getByLabelText("Schema")).toHaveValue("reporting_after_retry");
expect(screen.getByRole("button", { name: "Save changes" })).toBeEnabled();
});
test("shows secret status without exposing or enabling values when permission is absent", async () => {
const user = userEvent.setup();
renderPage({
rows: [makeDatabase({ secrets: { ...noSecrets, password: true } })],
canManageSecrets: false,
});
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
expect(screen.getByLabelText("Password")).toHaveValue("Configured");
expect(screen.getByLabelText("Password")).toHaveAttribute("readonly");
await user.click(screen.getByRole("button", { name: "Back to list" }));
await user.click(screen.getByRole("button", { name: "Edit Policlinico San Donato" }));
expect(await screen.findByRole("heading", { name: "Edit database" })).toBeVisible();
expect(screen.getByLabelText("Password")).toBeDisabled();
expect(screen.getByText(/requires the workspace\.secrets\.manage permission/i)).toBeVisible();
});
const patientsTable: CatalogTable = {
id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
databaseId: "11111111-1111-4111-8111-111111111111",
name: "patients",
sourceComment: "Patients imported from the clinical source",
description: null,
generatedDescription: null,
version: 1,
createdAt: "2026-08-27T08:00:00Z",
updatedAt: "2026-08-27T09:00:00Z",
};
test("filters catalog tables as the operator types", async () => {
const user = userEvent.setup();
const mediciTable: CatalogTable = {
...patientsTable,
id: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
name: "bridge_medici",
sourceComment: "Doctors participating in clinical care",
};
server.use(
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([
patientsTable,
mediciTable,
])),
);
renderPage({
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
});
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
const tablesRegion = await screen.findByRole("region", { name: "Tables for Policlinico San Donato" });
expect(await screen.findByRole("button", { name: "Edit description for patients" })).toBeVisible();
expect(screen.getByRole("button", { name: "Edit description for bridge_medici" })).toBeVisible();
await user.type(screen.getByRole("textbox", { name: "Search tables" }), "medici");
expect(screen.getByRole("button", { name: "Edit description for bridge_medici" })).toBeVisible();
expect(await within(tablesRegion).findByText("1 of 2")).toBeVisible();
await waitFor(() => {
expect(screen.queryByRole("button", { name: "Edit description for patients" })).not.toBeInTheDocument();
});
});
test("selected table exposes a direct Synchronize columns action and sends its id", async () => {
const user = userEvent.setup();
let startBody: unknown;
const run = makeSyncRun("columns", [patientsTable.id]);
registerCompletedSyncRun(run);
server.use(
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
startBody = await request.json();
return HttpResponse.json(run, { status: 202 });
}),
);
renderPage({
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
});
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
const tableRow = await screen.findByRole("row", { name: /patients/ });
await user.click(within(tableRow).getByRole("checkbox"));
const synchronizeColumns = screen.getByRole("button", { name: "Synchronize columns" });
expect(synchronizeColumns).toBeVisible();
expect(synchronizeColumns).toBeEnabled();
expect(screen.queryByRole("button", { name: "Actions" })).not.toBeInTheDocument();
await user.click(synchronizeColumns);
await waitFor(() => expect(startBody).toEqual({
version: 3,
scope: "columns",
tableIds: [patientsTable.id],
}));
await waitFor(() => expect(screen.queryByText("1 selected")).not.toBeInTheDocument());
});
test("navigates purely from a database to its tables and edits review metadata", async () => {
const user = userEvent.setup();
let patchBody: unknown;
server.use(
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
http.patch("/api/catalog/databases/:databaseId/tables/:tableId", async ({ request }) => {
patchBody = await request.json();
return HttpResponse.json({
...patientsTable,
description: "Registry used for longitudinal patient analysis",
version: 2,
});
}),
);
renderPage({
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
});
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
expect(await screen.findByRole("region", { name: "Tables for Policlinico San Donato" })).toBeVisible();
expect(screen.getByRole("tab", { name: "Tables" })).toHaveAttribute("aria-selected", "true");
expect(screen.queryByLabelText("Database filter")).not.toBeInTheDocument();
await user.click(await screen.findByRole("button", { name: "Edit description for patients" }));
expect(screen.getByLabelText("Physical table name")).toHaveValue("patients");
expect(screen.getByLabelText("Physical table name")).toHaveAttribute("readonly");
expect(screen.getByLabelText("Source comment")).toHaveAttribute("readonly");
await user.type(
screen.getByLabelText(/^Description/),
"Registry used for longitudinal patient analysis",
);
await user.click(screen.getByRole("button", { name: "Save metadata" }));
await waitFor(() => expect(patchBody).toEqual({
version: 1,
description: "Registry used for longitudinal patient analysis",
generatedDescription: null,
}));
await user.click(screen.getByRole("button", { name: "Back to tables" }));
await user.click(screen.getByRole("tab", { name: "Overview" }));
expect(await screen.findByRole("heading", { name: "Database details" })).toBeVisible();
});
test("navigates from a table to columns and from the database to physical relationships", async () => {
const user = userEvent.setup();
let columnPatch: unknown;
const idColumn: CatalogColumn = {
id: "dddddddd-dddd-4ddd-8ddd-dddddddddddd",
tableId: patientsTable.id,
name: "id",
ordinalPosition: 1,
dataType: "bigint",
isNullable: false,
defaultExpression: null,
primaryKeyPosition: 1,
isPrimaryKey: true,
isForeignKey: true,
foreignKeyCount: 1,
sourceComment: "Patient identifier",
description: null,
generatedDescription: null,
lastSyncedDatabaseVersion: 3,
lastSyncedAt: "2026-08-27T10:00:00Z",
version: 1,
createdAt: "2026-08-27T10:00:00Z",
updatedAt: "2026-08-27T10:00:00Z",
};
server.use(
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
http.get("/api/catalog/databases/:databaseId/tables/:tableId/columns", () => HttpResponse.json([idColumn])),
http.patch("/api/catalog/databases/:databaseId/tables/:tableId/columns/:columnId", async ({ request }) => {
columnPatch = await request.json();
return HttpResponse.json({ ...idColumn, generatedDescription: "Generated identifier draft", version: 2 });
}),
http.get("/api/catalog/databases/:databaseId/relationships", () => HttpResponse.json([{
id: "eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee",
databaseId: patientsTable.databaseId,
constraintName: "visits_patient_id_fkey",
sourceTableId: "ffffffff-ffff-4fff-8fff-ffffffffffff",
sourceTableName: "visits",
targetTableId: patientsTable.id,
targetTableName: "patients",
updateRule: "NO ACTION",
deleteRule: "CASCADE",
deferrable: false,
initiallyDeferred: false,
columns: [{ position: 1, sourceColumnId: "1", sourceColumnName: "patient_id", targetColumnId: idColumn.id, targetColumnName: "id" }],
lastSyncedDatabaseVersion: 3,
lastSyncedAt: "2026-08-27T10:00:00Z",
createdAt: "2026-08-27T10:00:00Z",
updatedAt: "2026-08-27T10:00:00Z",
}])),
);
renderPage({ rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })] });
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
expect(screen.getByRole("tab", { name: "Columns" })).toHaveAttribute("aria-selected", "true");
expect(await screen.findByText("PK")).toBeVisible();
expect(screen.getByText("FK")).toBeVisible();
await user.click(screen.getByRole("button", { name: "Edit metadata for id" }));
await user.type(screen.getByLabelText("Generated description"), "Generated identifier draft");
await user.click(screen.getByRole("button", { name: "Save metadata" }));
await waitFor(() => expect(columnPatch).toEqual({
version: 1,
description: null,
generatedDescription: "Generated identifier draft",
}));
await user.click(screen.getByRole("tab", { name: "Relationships" }));
expect(await screen.findByText("visits_patient_id_fkey")).toBeVisible();
});
test("opens the durable job drawer and confirms its exact destructive plan", async () => {
const user = userEvent.setup();
const startBodies: unknown[] = [];
const confirmationBodies: unknown[] = [];
const queued: CatalogSyncRun = {
id: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
databaseId: "11111111-1111-4111-8111-111111111111",
scope: "tables",
tableIds: [],
state: "queued",
phase: "queued",
requestedDatabaseVersion: 3,
plannedDiff: null,
confirmationToken: null,
counts: {},
errorCode: null,
errorMessage: null,
cancelRequested: false,
createdAt: "2026-08-27T10:00:00Z",
startedAt: null,
updatedAt: "2026-08-27T10:00:00Z",
finishedAt: null,
heartbeatAt: null,
};
const waiting: CatalogSyncRun = {
...queued,
state: "awaiting_confirmation",
phase: "awaiting_confirmation",
confirmationToken: "cccccccc-cccc-4ccc-8ccc-cccccccccccc",
plannedDiff: { deletedTables: ["legacy_visits"], deletedColumns: [], deletedRelationships: [] },
counts: { tables: 1 },
startedAt: "2026-08-27T10:00:00Z",
updatedAt: "2026-08-27T10:00:01Z",
};
server.use(
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
startBodies.push(await request.json());
return HttpResponse.json(queued, { status: 202 });
}),
http.get("/api/catalog/sync-runs/:runId", () => HttpResponse.json(waiting)),
http.get("/api/catalog/sync-runs/:runId/events-list", () => HttpResponse.json([])),
http.get("/api/catalog/databases/:databaseId/sync-runs", () => HttpResponse.json([waiting])),
http.post("/api/catalog/sync-runs/:runId/confirm", async ({ request }) => {
confirmationBodies.push(await request.json());
return HttpResponse.json({ ...waiting, state: "queued", phase: "queued", confirmationToken: null });
}),
);
renderPage({
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
});
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "Sync tables" }));
const synchronizationDrawer = await screen.findByRole("complementary", {
name: "Schema synchronization",
});
expect(synchronizationDrawer).toHaveClass("inset-y-2", "sm:inset-y-4");
expect(within(synchronizationDrawer).getByRole("listitem", {
name: "Confirmation required, in progress",
})).toHaveAttribute("aria-current", "step");
expect(await screen.findByText("table · legacy_visits")).toBeVisible();
expect(startBodies).toEqual([{ version: 3, scope: "tables", tableIds: [] }]);
await user.click(screen.getByRole("button", { name: "Confirm removals" }));
await waitFor(() => expect(confirmationBodies).toEqual([
{ confirmationToken: "cccccccc-cccc-4ccc-8ccc-cccccccccccc" },
]));
});
test("keeps a stale table draft but requires an explicit reload before another save", async () => {
const user = userEvent.setup();
const current = {
...patientsTable,
description: "Description saved by another editor",
version: 2,
};
let latest = patientsTable;
server.use(
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([latest])),
http.patch("/api/catalog/databases/:databaseId/tables/:tableId", () => {
latest = current;
return HttpResponse.json({
code: "table_stale",
message: "Table description changed. Reload and try again.",
}, { status: 409 });
}),
);
renderPage({
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
});
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
await user.click(screen.getByRole("tab", { name: "Tables" }));
await user.click(await screen.findByRole("button", { name: "Edit description for patients" }));
const description = screen.getByLabelText(/^Description/);
await user.type(description, "My unsaved draft");
await user.click(screen.getByRole("button", { name: "Save metadata" }));
expect(await screen.findByText("A newer table description is available.")).toBeVisible();
expect(description).toHaveValue("My unsaved draft");
expect(screen.getByRole("button", { name: "Save metadata" })).toBeDisabled();
await user.click(screen.getByRole("button", { name: "Keep editing" }));
expect(screen.getByText("Reload the latest value before this description can be saved.")).toBeVisible();
await user.click(screen.getByRole("button", { name: "Reload latest" }));
await waitFor(() => expect(description).toHaveValue("Description saved by another editor"));
expect(screen.getByRole("button", { name: "Save metadata" })).toBeDisabled();
});
+753 -5
View File
@@ -1,8 +1,756 @@
export function DatabaseManagementPage() {
import {
useCallback,
useEffect,
useMemo,
useRef,
useState,
} from "react";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { Plus, RefreshCw } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../components/ui/button";
import { ApiError, apiErrorMessage } from "../api/client";
import {
createCatalogDatabase,
deleteCatalogDatabase,
listCatalogDatabases,
replaceCatalogDatabaseSecrets,
startCatalogSync,
testCatalogDatabase,
updateCatalogDatabase,
type CatalogDatabase,
type CatalogSecretName,
type CatalogSyncScope,
type CatalogSyncRun,
type DatabaseBinding,
type DatabaseTransport,
} from "../api/catalog-databases";
import { DatabaseGrid } from "./database-management/DatabaseGrid";
import { DatabaseForm } from "./database-management/DatabaseForm";
import { DatabaseTables } from "./database-management/DatabaseTables";
import { DatabaseRelationships } from "./database-management/DatabaseRelationships";
import { CatalogSyncDrawer } from "./database-management/CatalogSyncDrawer";
import {
configurationFingerprint,
configurationFromDraft,
draftFrom,
hasSecretChanges,
secretReplacements,
type DatabaseBusyAction,
type DatabaseFormDraft,
type DatabaseFormMode,
type DatabaseNavigationState,
type DatabaseScreen,
} from "./database-management/model";
const DATABASE_QUERY_KEY = ["catalog-databases"] as const;
const SYNC_STARTED_MESSAGES: Record<CatalogSyncScope, string> = {
tables: "Table synchronization started",
columns: "Column synchronization started",
relationships: "Relationship synchronization started",
all: "Full schema synchronization started",
};
interface Props {
canManage: boolean;
canManageSecrets: boolean;
onNavigationStateChange?: (state: DatabaseNavigationState) => void;
}
interface FormSource {
id?: string;
version: number;
}
function isStaleError(error: unknown): boolean {
return error instanceof ApiError && error.code === "database_stale";
}
export function DatabaseManagementPage({
canManage,
canManageSecrets,
onNavigationStateChange,
}: Props) {
const queryClient = useQueryClient();
const {
data,
isLoading,
isError,
isFetching,
refetch,
} = useQuery({
queryKey: DATABASE_QUERY_KEY,
queryFn: listCatalogDatabases,
retry: false,
});
const rows = data ?? [];
const [screen, setScreen] = useState<DatabaseScreen>({ kind: "list" });
const [draft, setDraft] = useState<DatabaseFormDraft | null>(null);
const [baseline, setBaseline] = useState("");
const [formSource, setFormSource] = useState<FormSource | null>(null);
const [search, setSearch] = useState("");
const [busyAction, setBusyAction] = useState<DatabaseBusyAction>(null);
const [stale, setStale] = useState(false);
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
const [partialSecretFailure, setPartialSecretFailure] = useState<string | null>(null);
const [tablesNavigationState, setTablesNavigationState] = useState<DatabaseNavigationState>({
dirty: false,
busy: false,
});
const [activeSyncRun, setActiveSyncRun] = useState<CatalogSyncRun | null>(null);
const [syncDrawerOpen, setSyncDrawerOpen] = useState(false);
const originRef = useRef<HTMLElement | null>(null);
const searchInputRef = useRef<HTMLInputElement>(null);
const formHeadingRef = useRef<HTMLHeadingElement>(null);
const activeRow = screen.kind === "list"
? undefined
: rows.find((row) => row.workspaceId === screen.workspaceId);
const availableWorkspaces = useMemo(
() => rows.filter((row) => row.workspaceAvailable && !row.configured),
[rows],
);
const editable = screen.kind === "add" || screen.kind === "edit";
const configurationDirty = Boolean(
editable
&& draft
&& configurationFingerprint(draft) !== baseline,
);
const dirty = Boolean(editable && draft && (configurationDirty || hasSecretChanges(draft)));
const busy = busyAction !== null;
const navigationDirty = screen.kind === "tables" ? tablesNavigationState.dirty : dirty;
const navigationBusy = screen.kind === "tables" ? tablesNavigationState.busy : busy;
useEffect(() => {
onNavigationStateChange?.({ dirty: navigationDirty, busy: navigationBusy });
}, [navigationBusy, navigationDirty, onNavigationStateChange]);
useEffect(() => () => {
onNavigationStateChange?.({ dirty: false, busy: false });
}, [onNavigationStateChange]);
useEffect(() => {
const warn = (event: BeforeUnloadEvent) => {
if (!navigationDirty && !navigationBusy) return;
event.preventDefault();
};
window.addEventListener("beforeunload", warn);
return () => window.removeEventListener("beforeunload", warn);
}, [navigationBusy, navigationDirty]);
useEffect(() => {
if (screen.kind === "list" || !activeRow || !formSource || busy) return;
if (activeRow.id === formSource.id && activeRow.version === formSource.version) return;
setStale(true);
setStaleBannerOpen(true);
}, [activeRow, busy, formSource, screen.kind]);
useEffect(() => {
if (screen.kind === "list") return;
const timer = window.setTimeout(() => formHeadingRef.current?.focus(), 0);
return () => window.clearTimeout(timer);
}, [screen.kind, screen.kind === "list" ? "" : screen.workspaceId]);
const cacheSavedRow = useCallback((saved: CatalogDatabase) => {
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => {
const existing = current.findIndex((row) => row.workspaceId === saved.workspaceId);
if (existing < 0) return [...current, saved];
return current.map((row, index) => index === existing ? saved : row);
});
}, [queryClient]);
const restoreListFocus = useCallback(() => {
window.setTimeout(() => {
if (originRef.current?.isConnected) {
originRef.current.focus();
} else {
searchInputRef.current?.focus();
}
}, 0);
}, []);
const showList = useCallback(() => {
setScreen({ kind: "list" });
setDraft(null);
setBaseline("");
setFormSource(null);
setStale(false);
setStaleBannerOpen(true);
setPartialSecretFailure(null);
setTablesNavigationState({ dirty: false, busy: false });
restoreListFocus();
}, [restoreListFocus]);
useEffect(() => {
if (screen.kind === "list" || isLoading || activeRow) return;
showList();
toast.info("This database configuration is no longer available");
}, [activeRow, isLoading, screen.kind, showList]);
useEffect(() => {
if (!["tables", "relationships"].includes(screen.kind) || !activeRow || (activeRow.configured && activeRow.id)) return;
showList();
toast.info("Save the database configuration before managing tables");
}, [activeRow, screen.kind, showList]);
const backToList = useCallback(() => {
if (busy) return;
if (dirty && !window.confirm("Discard unsaved database changes?")) return;
showList();
}, [busy, dirty, showList]);
const openForm = useCallback((
mode: DatabaseFormMode,
row: CatalogDatabase,
origin: HTMLElement,
workspaceLocked = false,
) => {
const nextDraft = draftFrom(row);
originRef.current = origin;
setDraft(nextDraft);
setBaseline(configurationFingerprint(nextDraft));
setFormSource({ id: row.id, version: row.version });
setStale(false);
setStaleBannerOpen(true);
setPartialSecretFailure(null);
setScreen({
kind: mode,
workspaceId: row.workspaceId,
...(mode === "add" ? { workspaceLocked } : {}),
});
}, []);
const viewRow = useCallback((row: CatalogDatabase, origin: HTMLButtonElement) => {
openForm("view", row, origin);
}, [openForm]);
const editRow = useCallback((row: CatalogDatabase, origin: HTMLButtonElement) => {
openForm(row.configured ? "edit" : "add", row, origin, !row.configured);
}, [openForm]);
const deleteRow = useCallback((row: CatalogDatabase, origin: HTMLButtonElement) => {
openForm("delete", row, origin);
}, [openForm]);
const openTables = useCallback((row: CatalogDatabase, origin?: HTMLElement) => {
if (!row.configured || !row.id) return;
if (origin) originRef.current = origin;
setDraft(null);
setBaseline("");
setFormSource(null);
setStale(false);
setPartialSecretFailure(null);
setTablesNavigationState({ dirty: false, busy: false });
setScreen({ kind: "tables", workspaceId: row.workspaceId });
}, []);
const openRelationships = useCallback((row: CatalogDatabase) => {
if (!row.configured || !row.id) return;
setDraft(null);
setBaseline("");
setFormSource(null);
setStale(false);
setPartialSecretFailure(null);
setTablesNavigationState({ dirty: false, busy: false });
setScreen({ kind: "relationships", workspaceId: row.workspaceId });
}, []);
const openOverview = useCallback((row: CatalogDatabase) => {
const nextDraft = draftFrom(row);
setDraft(nextDraft);
setBaseline(configurationFingerprint(nextDraft));
setFormSource({ id: row.id, version: row.version });
setStale(false);
setStaleBannerOpen(true);
setPartialSecretFailure(null);
setTablesNavigationState({ dirty: false, busy: false });
setScreen({ kind: "view", workspaceId: row.workspaceId });
}, []);
const addDatabase = useCallback((origin: HTMLElement) => {
const workspace = availableWorkspaces[0];
if (!workspace || !canManage) return;
openForm("add", workspace, origin, false);
}, [availableWorkspaces, canManage, openForm]);
const changeWorkspace = useCallback((workspaceId: string) => {
const workspace = availableWorkspaces.find((row) => row.workspaceId === workspaceId);
if (!workspace) return;
if (dirty && !window.confirm("Discard changes and choose another workspace?")) return;
const nextDraft = draftFrom(workspace);
setDraft(nextDraft);
setBaseline(configurationFingerprint(nextDraft));
setFormSource({ id: workspace.id, version: workspace.version });
setStale(false);
setPartialSecretFailure(null);
setScreen({ kind: "add", workspaceId, workspaceLocked: false });
}, [availableWorkspaces, dirty]);
const changeField = useCallback((field: "databaseName" | "schema", value: string) => {
setDraft((current) => current ? { ...current, [field]: value } : current);
}, []);
const changeTransport = useCallback((transport: DatabaseTransport) => {
setDraft((current) => current
? { ...current, binding: { ...current.binding, transport } }
: current);
}, []);
const changeBinding = useCallback(<K extends keyof DatabaseBinding>(
key: K,
value: DatabaseBinding[K],
) => {
setDraft((current) => current
? { ...current, binding: { ...current.binding, [key]: value } }
: current);
}, []);
const changeSecret = useCallback((name: CatalogSecretName, value: string) => {
setDraft((current) => current
? { ...current, secrets: { ...current.secrets, [name]: value } }
: current);
}, []);
const markStale = useCallback(() => {
setStale(true);
setStaleBannerOpen(true);
}, []);
const save = useCallback(async () => {
if (!activeRow || !formSource || !draft || !editable || !canManage || stale || busy) return;
setBusyAction("save");
setPartialSecretFailure(null);
try {
const input = configurationFromDraft(draft);
let saved = activeRow;
const shouldPersistConfiguration = !formSource.id || configurationDirty;
if (shouldPersistConfiguration) {
saved = formSource.id
? await updateCatalogDatabase(formSource.id, formSource.version, input)
: await createCatalogDatabase(input);
setFormSource({ id: saved.id, version: saved.version });
cacheSavedRow(saved);
}
const replacements = secretReplacements(draft);
if (Object.keys(replacements).length > 0) {
if (!canManageSecrets) throw new Error("Secret replacement is not permitted");
try {
saved = await replaceCatalogDatabaseSecrets(
saved.id ?? formSource.id!,
shouldPersistConfiguration ? saved.version : formSource.version,
replacements,
);
setFormSource({ id: saved.id, version: saved.version });
cacheSavedRow(saved);
} catch (error) {
setBaseline(configurationFingerprint(draft));
setScreen({ kind: "edit", workspaceId: saved.workspaceId });
setPartialSecretFailure(apiErrorMessage(error));
if (isStaleError(error)) {
markStale();
} else {
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
}
toast.warning("Database configuration saved, but secrets still need attention");
return;
}
}
const nextDraft = draftFrom(saved);
setDraft(nextDraft);
setBaseline(configurationFingerprint(nextDraft));
setFormSource({ id: saved.id, version: saved.version });
setScreen({ kind: "edit", workspaceId: saved.workspaceId });
setStale(false);
setPartialSecretFailure(null);
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
toast.success(activeRow.configured ? "Database configuration saved" : "Database configuration added");
} catch (error) {
if (isStaleError(error)) {
markStale();
} else {
toast.error(apiErrorMessage(error));
}
} finally {
setBusyAction(null);
}
}, [
activeRow,
busy,
cacheSavedRow,
canManage,
canManageSecrets,
configurationDirty,
draft,
editable,
formSource,
markStale,
queryClient,
stale,
]);
const retrySecrets = useCallback(async () => {
if (!activeRow?.configured || !formSource?.id || !draft || !canManageSecrets || busy || stale) return;
const replacements = secretReplacements(draft);
if (Object.keys(replacements).length === 0) {
setPartialSecretFailure(null);
return;
}
setBusyAction("retry-secrets");
try {
const saved = await replaceCatalogDatabaseSecrets(formSource.id, formSource.version, replacements);
setFormSource({ id: saved.id, version: saved.version });
cacheSavedRow(saved);
setDraft({ ...draft, secrets: {} });
setPartialSecretFailure(null);
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
toast.success("Database secrets saved");
} catch (error) {
setPartialSecretFailure(apiErrorMessage(error));
if (isStaleError(error)) markStale();
toast.error(apiErrorMessage(error));
} finally {
setBusyAction(null);
}
}, [activeRow, busy, cacheSavedRow, canManageSecrets, draft, formSource, markStale, queryClient, stale]);
const testConnection = useCallback(async () => {
if (
!activeRow?.configured
|| !formSource?.id
|| !draft
|| !canManage
|| dirty
|| stale
|| busy
) return;
setBusyAction("test");
try {
const tested = await testCatalogDatabase(formSource.id, formSource.version);
setFormSource({ id: tested.id, version: tested.version });
cacheSavedRow(tested);
const nextDraft = draftFrom(tested);
setDraft(nextDraft);
setBaseline(configurationFingerprint(nextDraft));
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
if (tested.connectionStatus === "reachable") {
toast.success("Database connection is reachable");
} else {
toast.error(tested.lastErrorMessage ?? "Database connection failed");
}
} catch (error) {
if (isStaleError(error)) markStale();
else toast.error(apiErrorMessage(error));
} finally {
setBusyAction(null);
}
}, [activeRow, busy, cacheSavedRow, canManage, dirty, draft, formSource, markStale, queryClient, stale]);
const remove = useCallback(async () => {
if (!activeRow?.configured || !formSource?.id || !canManage || busy || stale) return;
setBusyAction("delete");
try {
await deleteCatalogDatabase(formSource.id, formSource.version);
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => {
if (!activeRow.workspaceAvailable) {
return current.filter((row) => row.workspaceId !== activeRow.workspaceId);
}
return current.map((row) => row.workspaceId === activeRow.workspaceId
? {
...row,
id: undefined,
configured: false,
version: 0,
createdAt: "",
updatedAt: "",
connectionStatus: "untested",
testedVersion: undefined,
lastTestedAt: undefined,
lastErrorCode: undefined,
lastErrorMessage: undefined,
secrets: {
password: false,
apiKey: false,
sshPrivateKey: false,
sshPrivateKeyPassphrase: false,
sshKnownHosts: false,
tlsCa: false,
},
}
: row);
});
setBusyAction(null);
showList();
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
toast.success("Database configuration deleted");
} catch (error) {
if (isStaleError(error)) markStale();
else toast.error(apiErrorMessage(error));
} finally {
setBusyAction(null);
}
}, [activeRow, busy, canManage, formSource, markStale, queryClient, showList, stale]);
const reloadLatest = useCallback(async () => {
if (busy || screen.kind === "list") return;
setBusyAction("reload");
try {
const result = await refetch();
if (result.error) throw result.error;
const latest = result.data?.find((row) => row.workspaceId === screen.workspaceId);
if (!latest) {
showList();
toast.info("This database configuration is no longer available");
return;
}
const nextDraft = draftFrom(latest);
setDraft(nextDraft);
setBaseline(configurationFingerprint(nextDraft));
setFormSource({ id: latest.id, version: latest.version });
setStale(false);
setStaleBannerOpen(true);
setPartialSecretFailure(null);
if (!latest.configured && screen.kind === "delete") {
showList();
toast.info("This database configuration has already been deleted");
} else if (screen.kind === "edit" && !latest.configured) {
setScreen({ kind: "add", workspaceId: latest.workspaceId, workspaceLocked: true });
}
} catch (error) {
toast.error(apiErrorMessage(error));
} finally {
setBusyAction(null);
}
}, [busy, refetch, screen, showList]);
const refreshList = useCallback(async () => {
if (isFetching) return;
try {
const result = await refetch();
if (result.error) throw result.error;
} catch (error) {
toast.error(apiErrorMessage(error));
}
}, [isFetching, refetch]);
const updateTrackedSyncRun = useCallback((run: CatalogSyncRun) => {
setActiveSyncRun(run);
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => current.map((row) => (
row.id === run.databaseId
? { ...row, activeSyncRun: ["queued", "running", "awaiting_confirmation", "applying"].includes(run.state) ? run : undefined }
: row
)));
}, [queryClient]);
const rememberSyncRun = useCallback((run: CatalogSyncRun) => {
updateTrackedSyncRun(run);
setSyncDrawerOpen(true);
}, [updateTrackedSyncRun]);
const openSync = useCallback((row?: CatalogDatabase) => {
const run = row?.activeSyncRun ?? activeSyncRun;
if (!run) return;
setActiveSyncRun(run);
setSyncDrawerOpen(true);
}, [activeSyncRun]);
const testSelected = useCallback(async (selected: CatalogDatabase[]) => {
try {
const tested = await Promise.all(selected.map((row) => testCatalogDatabase(row.id!, row.version)));
for (const row of tested) cacheSavedRow(row);
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
toast.success(`${tested.length} connection${tested.length === 1 ? "" : "s"} tested`);
} catch (error) {
toast.error(apiErrorMessage(error));
throw error;
}
}, [cacheSavedRow, queryClient]);
const syncSelected = useCallback(async (selected: CatalogDatabase[], scope: CatalogSyncScope) => {
try {
const runs = await Promise.all(selected.map((row) => startCatalogSync(row.id!, row.version, scope)));
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => current.map((row) => {
const run = runs.find((candidate) => candidate.databaseId === row.id);
return run ? { ...row, activeSyncRun: run } : row;
}));
if (runs[0]) rememberSyncRun(runs[0]);
toast.success(`${runs.length} schema synchronization${runs.length === 1 ? "" : "s"} started`);
} catch (error) {
toast.error(apiErrorMessage(error));
throw error;
}
}, [queryClient, rememberSyncRun]);
const syncDatabase = useCallback(async (scope: CatalogSyncScope) => {
if (!activeRow?.id || !activeRow.configured) return;
try {
rememberSyncRun(await startCatalogSync(activeRow.id, activeRow.version, scope));
toast.success(SYNC_STARTED_MESSAGES[scope]);
} catch (error) { toast.error(apiErrorMessage(error)); }
}, [activeRow, rememberSyncRun]);
const catalogChanged = useCallback(async () => {
const databaseId = activeSyncRun?.databaseId;
if (databaseId) {
await Promise.all([
queryClient.invalidateQueries({ queryKey: ["catalog-tables", databaseId] }),
queryClient.invalidateQueries({ queryKey: ["catalog-relationships", databaseId] }),
queryClient.invalidateQueries({ queryKey: ["catalog-sync-runs", databaseId] }),
]);
}
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
}, [activeSyncRun, queryClient]);
const formVisible = screen.kind !== "list" && screen.kind !== "tables" && screen.kind !== "relationships" && activeRow && draft;
const tablesVisible = screen.kind === "tables" && activeRow?.configured && activeRow.id;
const relationshipsVisible = screen.kind === "relationships" && activeRow?.configured && activeRow.id;
const currentActiveRun = activeSyncRun && ["queued", "running", "awaiting_confirmation", "applying"].includes(activeSyncRun.state)
? activeSyncRun
: activeRow?.activeSyncRun && ["queued", "running", "awaiting_confirmation", "applying"].includes(activeRow.activeSyncRun.state)
? activeRow.activeSyncRun
: undefined;
return (
<main
aria-label="Database management"
className="flex-1 overflow-y-auto bg-background"
/>
<main aria-label="Database management" className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
<header className="flex flex-wrap items-center justify-between gap-4 border-b border-border bg-card px-4 py-4 sm:px-5">
<div>
<p className="thot-label mb-1">Administration</p>
<h1 className="font-heading text-xl font-semibold tracking-tight">Database management</h1>
<p className="mt-1 text-sm text-muted-foreground">
One database configuration for each repository workspace.
</p>
</div>
{screen.kind === "list" ? (
<div className="flex items-center gap-2">
<Button type="button" variant="outline" disabled={isFetching} onClick={() => void refreshList()}>
<RefreshCw className={isFetching ? "animate-spin" : ""} /> Refresh
</Button>
<Button
type="button"
disabled={!canManage || availableWorkspaces.length === 0}
title={availableWorkspaces.length === 0 ? "Every available workspace is already configured" : undefined}
onClick={(event) => addDatabase(event.currentTarget)}
>
<Plus /> Add database
</Button>
</div>
) : null}
</header>
<div className="relative flex min-h-0 flex-1">
<div
aria-hidden={screen.kind !== "list"}
className={screen.kind === "list"
? "flex min-h-0 flex-1"
: "pointer-events-none invisible absolute inset-0 flex min-h-0"
}
>
{isError && rows.length === 0 ? (
<div className="grid flex-1 place-items-center p-6">
<div className="max-w-lg rounded-md border border-destructive/30 bg-destructive/8 p-5 text-sm" role="alert">
<p className="font-semibold text-destructive">The database catalog is unavailable.</p>
<p className="mt-1 leading-5 text-muted-foreground">Verify the catalog service and database migrations, then try again.</p>
<Button type="button" variant="outline" className="mt-4" disabled={isFetching} onClick={() => void refreshList()}>
<RefreshCw /> Try again
</Button>
</div>
</div>
) : (
<DatabaseGrid
rows={rows}
isLoading={isLoading}
isFetching={isFetching}
search={search}
canManage={canManage}
searchInputRef={searchInputRef}
onSearchChange={setSearch}
onView={viewRow}
onEdit={editRow}
onDelete={deleteRow}
onOpenSync={openSync}
onTestSelected={testSelected}
onSyncSelected={syncSelected}
/>
)}
</div>
{formVisible ? (
<DatabaseForm
mode={screen.kind as DatabaseFormMode}
row={activeRow}
availableWorkspaces={availableWorkspaces}
workspaceLocked={screen.kind === "add" && Boolean(screen.workspaceLocked)}
draft={draft}
dirty={dirty}
canManage={canManage}
canManageSecrets={canManageSecrets}
busyAction={busyAction}
stale={stale}
staleBannerOpen={staleBannerOpen}
partialSecretFailure={partialSecretFailure}
headingRef={formHeadingRef}
onBack={backToList}
onWorkspaceChange={changeWorkspace}
onFieldChange={changeField}
onTransportChange={changeTransport}
onBindingChange={changeBinding}
onSecretChange={changeSecret}
onSave={() => void save()}
onTest={() => void testConnection()}
onDelete={() => void remove()}
onReloadLatest={() => void reloadLatest()}
onKeepEditing={() => setStaleBannerOpen(false)}
onRetrySecrets={() => void retrySecrets()}
onOpenTables={(origin) => openTables(activeRow, origin)}
onOpenRelationships={() => openRelationships(activeRow)}
onSync={(scope) => void syncDatabase(scope)}
onOpenSync={() => openSync(activeRow)}
activeSyncRun={currentActiveRun}
/>
) : null}
{tablesVisible ? (
<DatabaseTables
database={activeRow}
canManage={canManage}
activeRun={currentActiveRun}
onBackToDatabases={showList}
onOpenOverview={() => openOverview(activeRow)}
onOpenRelationships={() => openRelationships(activeRow)}
onNavigationStateChange={setTablesNavigationState}
onRunStarted={rememberSyncRun}
onOpenSync={() => openSync(activeRow)}
/>
) : null}
{relationshipsVisible ? (
<DatabaseRelationships
database={{ ...activeRow, activeSyncRun: currentActiveRun }}
canManage={canManage}
onBackToDatabases={showList}
onOpenOverview={() => openOverview(activeRow)}
onOpenTables={() => openTables(activeRow)}
onRunStarted={rememberSyncRun}
onOpenSync={() => openSync(activeRow)}
/>
) : null}
</div>
<CatalogSyncDrawer
databaseId={activeSyncRun?.databaseId ?? activeRow?.id ?? null}
runId={activeSyncRun?.id ?? null}
open={syncDrawerOpen && Boolean(activeSyncRun)}
onClose={() => setSyncDrawerOpen(false)}
onRunChange={rememberSyncRun}
onRunUpdate={updateTrackedSyncRun}
onCatalogChanged={() => void catalogChanged()}
/>
</main>
);
}
@@ -0,0 +1,402 @@
import { useEffect, useMemo, useRef, useState } from "react";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { AlertTriangle, Check, CheckCircle2, Circle, LoaderCircle, Minus, RotateCcw, X } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../../components/ui/button";
import { apiErrorMessage } from "../../api/client";
import {
cancelCatalogSync,
catalogSyncEventsUrl,
confirmCatalogSync,
getCatalogSyncRun,
listCatalogSyncEvents,
listCatalogSyncRuns,
retryCatalogSync,
type CatalogSyncEvent,
type CatalogSyncPhase,
type CatalogSyncRun,
} from "../../api/catalog-databases";
interface Props {
databaseId: string | null;
runId: string | null;
open: boolean;
onClose: () => void;
onRunChange: (run: CatalogSyncRun) => void;
onRunUpdate: (run: CatalogSyncRun) => void;
onCatalogChanged: () => void;
}
const phases: CatalogSyncPhase[] = [
"queued", "connecting", "scanning_tables", "scanning_columns",
"scanning_relationships", "planning", "awaiting_confirmation", "applying", "completed",
];
const phaseEnteredByEvent: Record<string, CatalogSyncPhase> = {
queued: "queued",
started: "connecting",
connecting: "connecting",
scanning_tables: "scanning_tables",
scanning_columns: "scanning_columns",
scanning_relationships: "scanning_relationships",
planning: "planning",
confirmation_required: "awaiting_confirmation",
confirmation_received: "awaiting_confirmation",
applying: "applying",
succeeded: "completed",
};
const phaseLabels: Record<CatalogSyncPhase, { pending: string; current: string; completed: string }> = {
queued: { pending: "Queue", current: "Queued", completed: "Queued" },
connecting: { pending: "Connect", current: "Connecting", completed: "Connected" },
scanning_tables: { pending: "Read tables", current: "Reading tables", completed: "Tables read" },
scanning_columns: { pending: "Read columns", current: "Reading columns", completed: "Columns read" },
scanning_relationships: {
pending: "Read relationships",
current: "Reading relationships",
completed: "Relationships read",
},
planning: { pending: "Plan changes", current: "Planning changes", completed: "Changes planned" },
awaiting_confirmation: {
pending: "Confirm changes if needed",
current: "Confirmation required",
completed: "Changes confirmed",
},
applying: { pending: "Apply changes", current: "Applying changes", completed: "Changes applied" },
completed: { pending: "Complete", current: "Completing", completed: "Completed" },
};
function terminal(run?: CatalogSyncRun): boolean {
return Boolean(run && ["succeeded", "failed", "cancelled", "interrupted"].includes(run.state));
}
function elapsed(run: CatalogSyncRun, now: number): string {
const start = new Date(run.startedAt ?? run.createdAt).getTime();
const end = run.finishedAt ? new Date(run.finishedAt).getTime() : now;
const seconds = Math.max(0, Math.floor((end - start) / 1000));
const minutes = Math.floor(seconds / 60);
return minutes ? `${minutes}m ${seconds % 60}s` : `${seconds}s`;
}
function stateLabel(run: CatalogSyncRun): string {
return run.state.replaceAll("_", " ");
}
export function CatalogSyncDrawer({
databaseId,
runId,
open,
onClose,
onRunChange,
onRunUpdate,
onCatalogChanged,
}: Props) {
const queryClient = useQueryClient();
const [events, setEvents] = useState<CatalogSyncEvent[]>([]);
const [now, setNow] = useState(Date.now());
const [action, setAction] = useState<"confirm" | "cancel" | "retry" | null>(null);
const lastSequence = useRef(0);
const notifiedRun = useRef<string | null>(null);
const runQuery = useQuery({
queryKey: ["catalog-sync-run", runId],
queryFn: () => getCatalogSyncRun(runId!),
enabled: Boolean(runId),
retry: false,
refetchInterval: (query) => terminal(query.state.data) ? false : 1_000,
});
const run = runQuery.data;
const historyQuery = useQuery({
queryKey: ["catalog-sync-runs", databaseId],
queryFn: () => listCatalogSyncRuns(databaseId!),
enabled: Boolean(databaseId && open),
retry: false,
refetchInterval: run && !terminal(run) ? 2_000 : false,
});
const eventQuery = useQuery({
queryKey: ["catalog-sync-events", runId],
queryFn: () => listCatalogSyncEvents(runId!, lastSequence.current),
enabled: Boolean(runId && open),
retry: false,
refetchInterval: run && terminal(run) ? false : 1_500,
});
const mergeEvents = (incoming: CatalogSyncEvent[]) => {
if (incoming.length === 0) return;
setEvents((current) => {
const bySequence = new Map(current.map((event) => [event.sequence, event]));
for (const event of incoming) bySequence.set(event.sequence, event);
const merged = [...bySequence.values()].sort((a, b) => a.sequence - b.sequence);
lastSequence.current = merged.at(-1)?.sequence ?? lastSequence.current;
return merged;
});
};
useEffect(() => {
setEvents([]);
lastSequence.current = 0;
}, [runId]);
useEffect(() => { mergeEvents(eventQuery.data ?? []); }, [eventQuery.data]);
useEffect(() => {
if (!runId || !open || typeof EventSource === "undefined") return;
const source = new EventSource(catalogSyncEventsUrl(runId));
const log = (event: MessageEvent<string>) => {
try { mergeEvents([JSON.parse(event.data) as CatalogSyncEvent]); } catch { /* polling remains authoritative */ }
};
const update = (event: MessageEvent<string>) => {
try {
const next = JSON.parse(event.data) as CatalogSyncRun;
queryClient.setQueryData(["catalog-sync-run", runId], next);
} catch { /* polling remains authoritative */ }
};
source.addEventListener("log", log as EventListener);
source.addEventListener("run", update as EventListener);
source.onerror = () => source.close();
return () => source.close();
}, [open, queryClient, runId]);
useEffect(() => {
if (!run || terminal(run)) return;
const timer = window.setInterval(() => setNow(Date.now()), 1_000);
return () => window.clearInterval(timer);
}, [run]);
useEffect(() => {
if (!run) return;
onRunUpdate(run);
queryClient.setQueryData<CatalogSyncRun[]>(["catalog-sync-runs", run.databaseId], (current) => {
if (!current) return [run];
const found = current.some((item) => item.id === run.id);
return found
? current.map((item) => item.id === run.id ? run : item)
: [run, ...current];
});
}, [onRunUpdate, queryClient, run]);
useEffect(() => {
if (!run || run.state !== "succeeded" || notifiedRun.current === run.id) return;
notifiedRun.current = run.id;
onCatalogChanged();
}, [onCatalogChanged, run]);
const currentPhase = run ? phases.indexOf(run.phase) : -1;
const knownEvents = [...events, ...(eventQuery.data ?? [])];
const confirmationWasRequired = run?.state === "awaiting_confirmation" || knownEvents.some((event) => (
event.eventType === "confirmation_required" || event.eventType === "confirmation_received"
));
const confirmationWasReceived = knownEvents.some((event) => event.eventType === "confirmation_received")
|| (run?.state === "succeeded" && confirmationWasRequired);
const furthestEventPhase = knownEvents.reduce((furthest, event) => {
const phase = phaseEnteredByEvent[event.eventType];
return phase ? Math.max(furthest, phases.indexOf(phase)) : furthest;
}, -1);
const confirmationPhase = phases.indexOf("awaiting_confirmation");
const passedConfirmation = run?.state === "succeeded"
|| (run && !terminal(run) ? currentPhase > confirmationPhase : furthestEventPhase > confirmationPhase);
const deletionCount = useMemo(() => run?.plannedDiff
? run.plannedDiff.deletedTables.length + run.plannedDiff.deletedColumns.length + run.plannedDiff.deletedRelationships.length
: 0, [run]);
const perform = async (kind: "confirm" | "cancel" | "retry") => {
if (!run) return;
setAction(kind);
try {
const next = kind === "confirm"
? await confirmCatalogSync(run.id, run.confirmationToken!)
: kind === "cancel"
? await cancelCatalogSync(run.id)
: await retryCatalogSync(run.id);
onRunChange(next);
queryClient.setQueryData(["catalog-sync-run", next.id], next);
await historyQuery.refetch();
} catch (error) {
toast.error(apiErrorMessage(error));
} finally {
setAction(null);
}
};
if (!open) return null;
return (
<aside
aria-label="Schema synchronization"
className="fixed inset-y-2 right-0 z-40 flex w-full max-w-[440px] flex-col border-l border-border bg-background shadow-2xl sm:inset-y-4"
>
<div className="flex items-start justify-between gap-4 border-b border-border px-5 py-4">
<div>
<p className="thot-label">Schema synchronization</p>
<h2 className="mt-1 font-heading text-xl font-semibold">{run ? `${stateLabel(run)[0].toUpperCase()}${stateLabel(run).slice(1)}` : "Loading"}</h2>
{run ? <p className="mt-1 text-sm text-muted-foreground">{run.scope} · {elapsed(run, now)}</p> : null}
</div>
<Button type="button" variant="ghost" size="icon-lg" aria-label="Close synchronization drawer" onClick={onClose}>
<X aria-hidden="true" />
</Button>
</div>
<div className="min-h-0 flex-1 overflow-y-auto px-5 py-5">
{runQuery.isLoading ? <p className="text-sm text-muted-foreground">Loading synchronization…</p> : null}
{run ? (
<>
<section aria-labelledby="synchronization-steps-heading">
<h3 id="synchronization-steps-heading" className="thot-label mb-3">Synchronization steps</h3>
<ol className="grid grid-cols-2 gap-x-4 gap-y-2.5 sm:grid-cols-3" aria-label="Synchronization steps">
{phases.map((phase, index) => {
const confirmationUnknown = phase === "awaiting_confirmation"
&& passedConfirmation
&& !eventQuery.isFetched
&& !confirmationWasRequired;
const confirmationSkipped = phase === "awaiting_confirmation"
&& passedConfirmation
&& eventQuery.isFetched
&& !confirmationWasRequired;
const terminalOutcome = phase === "completed" && terminal(run) && run.state !== "succeeded";
const failedOutcome = terminalOutcome && run.state === "failed";
const interruptedOutcome = terminalOutcome && run.state === "interrupted";
const priorPhaseCompleted = terminal(run)
? index < furthestEventPhase
: index < currentPhase;
const complete = !confirmationUnknown && !confirmationSkipped && !terminalOutcome
&& (phase === "awaiting_confirmation"
? confirmationWasReceived
: run.state === "succeeded" || priorPhaseCompleted);
const current = index === currentPhase && !terminal(run);
const label = phase === "awaiting_confirmation"
? confirmationSkipped
? "Confirmation not required"
: confirmationUnknown
? "Checking confirmation"
: confirmationWasReceived
? phaseLabels[phase].completed
: current
? phaseLabels[phase].current
: complete
? phaseLabels[phase].completed
: phaseLabels[phase].pending
: terminalOutcome
? `${stateLabel(run)[0].toUpperCase()}${stateLabel(run).slice(1)}`
: complete
? phaseLabels[phase].completed
: current
? phaseLabels[phase].current
: phaseLabels[phase].pending;
const status = terminalOutcome
? run.state
: confirmationSkipped
? "not required"
: complete
? "completed"
: current
? "in progress"
: "pending";
return (
<li
key={phase}
aria-current={current ? "step" : undefined}
aria-label={`${label}, ${status}`}
className={`flex items-center gap-2 text-xs ${failedOutcome ? "font-semibold text-destructive" : interruptedOutcome ? "font-semibold text-amber-600" : terminalOutcome || current ? "font-semibold text-foreground" : "text-muted-foreground"}`}
>
{confirmationSkipped ? (
<Minus aria-hidden="true" className="size-3.5 text-muted-foreground" />
) : failedOutcome ? (
<X aria-hidden="true" className="size-3.5 text-destructive" />
) : interruptedOutcome ? (
<AlertTriangle aria-hidden="true" className="size-3.5 text-amber-600" />
) : terminalOutcome ? (
<Minus aria-hidden="true" className="size-3.5 text-muted-foreground" />
) : complete ? (
<CheckCircle2 aria-hidden="true" className="size-3.5 text-[oklch(var(--success))]" />
) : current ? (
<LoaderCircle aria-hidden="true" className="size-3.5 animate-spin text-amber-600" />
) : (
<Circle aria-hidden="true" className="size-3.5" />
)}
{label}
</li>
);
})}
</ol>
</section>
<div className="mt-5 grid grid-cols-3 gap-2">
{(["tables", "columns", "relationships"] as const).map((name) => (
<div key={name} className="rounded-md border border-border bg-muted/25 px-3 py-2">
<p className="text-[11px] font-semibold uppercase tracking-wide text-muted-foreground">{name}</p>
<p className="mt-1 text-lg font-semibold tabular-nums">{run.counts[name] ?? ""}</p>
</div>
))}
</div>
{run.state === "awaiting_confirmation" && run.plannedDiff ? (
<section className="mt-5 rounded-md border border-amber-500/50 bg-amber-500/8 p-4" aria-label="Destructive changes">
<div className="flex gap-3">
<AlertTriangle className="mt-0.5 size-5 shrink-0 text-amber-600" />
<div>
<h3 className="font-semibold">Confirm {deletionCount} removals</h3>
<p className="mt-1 text-sm text-muted-foreground">The database will be scanned again before anything is removed.</p>
</div>
</div>
<ul className="mt-3 max-h-40 space-y-1 overflow-y-auto font-mono text-xs">
{run.plannedDiff.deletedTables.map((name) => <li key={`t-${name}`}>table · {name}</li>)}
{run.plannedDiff.deletedColumns.map((item) => <li key={`c-${item.tableName}-${item.columnName}`}>column · {item.tableName}.{item.columnName}</li>)}
{run.plannedDiff.deletedRelationships.map((item) => <li key={`r-${item.sourceTableName}-${item.constraintName}`}>relationship · {item.sourceTableName}.{item.constraintName}</li>)}
</ul>
</section>
) : null}
{run.errorMessage ? (
<div className="mt-5 rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
{run.errorMessage}
</div>
) : null}
<section className="mt-5" aria-label="Synchronization log">
<div className="mb-2 flex items-center justify-between">
<h3 className="thot-label">Live log</h3>
<span className="text-xs tabular-nums text-muted-foreground">{events.length}</span>
</div>
<div className="max-h-64 overflow-y-auto rounded-md bg-zinc-950 p-3 font-mono text-xs leading-5 text-zinc-200">
{events.length === 0 ? <p className="text-zinc-500">Waiting for events…</p> : events.map((event) => (
<p key={event.sequence} className={event.level === "error" ? "text-red-300" : event.level === "warning" ? "text-amber-300" : undefined}>
<span className="mr-2 text-zinc-500">{new Date(event.createdAt).toLocaleTimeString()}</span>{event.message}
</p>
))}
</div>
</section>
<section className="mt-5" aria-label="Synchronization history">
<h3 className="thot-label mb-2">Recent runs</h3>
<div className="divide-y divide-border rounded-md border border-border">
{(historyQuery.data ?? []).slice(0, 5).map((item) => (
<button key={item.id} type="button" className="flex w-full items-center justify-between gap-3 px-3 py-2 text-left text-sm hover:bg-muted/50" onClick={() => onRunChange(item)}>
<span>{item.scope}</span>
<span className="text-xs text-muted-foreground">{stateLabel(item)} · {new Date(item.createdAt).toLocaleString()}</span>
</button>
))}
</div>
</section>
</>
) : null}
</div>
{run ? (
<div className="flex justify-end gap-2 border-t border-border px-5 py-4">
{run.state === "awaiting_confirmation" ? (
<Button type="button" disabled={action !== null} onClick={() => void perform("confirm")}>
{action === "confirm" ? <LoaderCircle className="animate-spin" /> : <Check />} Confirm removals
</Button>
) : null}
{["queued", "running", "awaiting_confirmation"].includes(run.state) ? (
<Button type="button" variant="outline" disabled={action !== null} onClick={() => void perform("cancel")}>Cancel</Button>
) : null}
{["failed", "cancelled", "interrupted"].includes(run.state) ? (
<Button type="button" disabled={action !== null} onClick={() => void perform("retry")}>
<RotateCcw className={action === "retry" ? "animate-spin" : ""} /> Retry
</Button>
) : null}
</div>
) : null}
</aside>
);
}
@@ -0,0 +1,192 @@
import { useEffect, useMemo, useRef, useState } from "react";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { AgGridReact } from "ag-grid-react";
import type { ColDef, ICellRendererParams } from "ag-grid-community";
import { KeyRound, Link2, Pencil, RefreshCw, Save } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../../components/ui/button";
import { ApiError, apiErrorMessage } from "../../api/client";
import {
listCatalogColumns,
updateCatalogColumnMetadata,
type CatalogColumn,
type CatalogTable,
} from "../../api/catalog-databases";
import type { DatabaseNavigationState } from "./model";
interface Props {
databaseId: string;
table: CatalogTable;
canManage: boolean;
onNavigationStateChange: (state: DatabaseNavigationState) => void;
onSync: () => void;
}
interface GridContext {
canManage: boolean;
onEdit: (column: CatalogColumn, origin: HTMLButtonElement) => void;
}
function KeyCell({ data }: ICellRendererParams<CatalogColumn>) {
if (!data) return null;
return (
<div className="flex h-full items-center gap-1.5">
{data.isPrimaryKey ? <span className="inline-flex items-center gap-1 rounded bg-primary/10 px-1.5 py-0.5 text-xs font-semibold text-primary"><KeyRound className="size-3" />PK{data.primaryKeyPosition && data.primaryKeyPosition > 1 ? ` ${data.primaryKeyPosition}` : ""}</span> : null}
{data.isForeignKey ? <span className="inline-flex items-center gap-1 rounded bg-muted px-1.5 py-0.5 text-xs font-semibold text-muted-foreground"><Link2 className="size-3" />FK{data.foreignKeyCount > 1 ? ` ${data.foreignKeyCount}` : ""}</span> : null}
</div>
);
}
function ActionCell({ data, context }: ICellRendererParams<CatalogColumn, unknown, GridContext>) {
if (!data || !context) return null;
return (
<div className="flex h-full items-center justify-end" onClick={(event) => event.stopPropagation()}>
<Button type="button" variant="ghost" size="icon-lg" disabled={!context.canManage} aria-label={`Edit metadata for ${data.name}`} onClick={(event) => context.onEdit(data, event.currentTarget)}>
<Pencil aria-hidden="true" />
</Button>
</div>
);
}
export function DatabaseColumns({ databaseId, table, canManage, onNavigationStateChange, onSync }: Props) {
const queryClient = useQueryClient();
const queryKey = ["catalog-columns", databaseId, table.id] as const;
const { data = [], isLoading, isFetching, refetch } = useQuery({
queryKey,
queryFn: () => listCatalogColumns(databaseId, table.id),
retry: false,
});
const [search, setSearch] = useState("");
const [editingId, setEditingId] = useState<string | null>(null);
const [description, setDescription] = useState("");
const [generatedDescription, setGeneratedDescription] = useState("");
const [baseline, setBaseline] = useState("");
const [version, setVersion] = useState<number | null>(null);
const [stale, setStale] = useState(false);
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
const [busy, setBusy] = useState(false);
const originRef = useRef<HTMLButtonElement | null>(null);
const active = editingId ? data.find((column) => column.id === editingId) : undefined;
const fingerprint = JSON.stringify([description, generatedDescription]);
const dirty = Boolean(editingId && fingerprint !== baseline);
useEffect(() => { onNavigationStateChange({ dirty, busy }); }, [busy, dirty, onNavigationStateChange]);
useEffect(() => {
if (!editingId || !active || version === active.version || busy) return;
setStale(true);
setStaleBannerOpen(true);
}, [active, busy, editingId, version]);
const edit = (column: CatalogColumn, origin: HTMLButtonElement) => {
originRef.current = origin;
setEditingId(column.id);
setDescription(column.description ?? "");
setGeneratedDescription(column.generatedDescription ?? "");
setBaseline(JSON.stringify([column.description ?? "", column.generatedDescription ?? ""]));
setVersion(column.version);
setStale(false);
setStaleBannerOpen(true);
};
const closeEditor = () => {
if (busy) return;
if (dirty && !window.confirm("Discard unsaved column metadata?")) return;
setEditingId(null);
window.setTimeout(() => originRef.current?.focus(), 0);
};
const save = async () => {
if (!active || version === null) return;
setBusy(true);
try {
const updated = await updateCatalogColumnMetadata(
databaseId, table.id, active.id, version,
description.trim() || null, generatedDescription.trim() || null,
);
queryClient.setQueryData<CatalogColumn[]>(queryKey, (current = []) => current.map((column) => column.id === updated.id ? updated : column));
setDescription(updated.description ?? "");
setGeneratedDescription(updated.generatedDescription ?? "");
setBaseline(JSON.stringify([updated.description ?? "", updated.generatedDescription ?? ""]));
setVersion(updated.version);
setStale(false);
toast.success("Column metadata saved");
} catch (error) {
if (error instanceof ApiError && error.code === "column_stale") {
await refetch();
setStale(true);
setStaleBannerOpen(true);
} else toast.error(apiErrorMessage(error));
} finally { setBusy(false); }
};
const reloadColumn = async () => {
if (!editingId) return;
setBusy(true);
try {
const result = await refetch();
const latest = result.data?.find((column) => column.id === editingId);
if (!latest) { closeEditor(); return; }
setDescription(latest.description ?? "");
setGeneratedDescription(latest.generatedDescription ?? "");
setBaseline(JSON.stringify([latest.description ?? "", latest.generatedDescription ?? ""]));
setVersion(latest.version);
setStale(false);
setStaleBannerOpen(true);
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(false); }
};
const columns = useMemo<ColDef<CatalogColumn>[]>(() => [
{ field: "ordinalPosition", headerName: "#", width: 64, maxWidth: 64, filter: "agNumberColumnFilter" },
{ field: "name", headerName: "Name", minWidth: 190, flex: 1, cellClass: "font-mono text-xs" },
{ field: "dataType", headerName: "Type", minWidth: 150, flex: 0.8, cellClass: "font-mono text-xs" },
{ headerName: "Keys", minWidth: 125, width: 125, sortable: false, filter: false, cellRenderer: KeyCell },
{ headerName: "Nullable", minWidth: 100, width: 100, valueGetter: ({ data: row }) => row ? (row.isNullable ? "Yes" : "No") : "" },
{ field: "sourceComment", headerName: "Source comment", minWidth: 220, flex: 1.2, valueFormatter: ({ value }) => value ?? "" },
{ field: "generatedDescription", headerName: "Generated description", minWidth: 230, flex: 1.2, valueFormatter: ({ value }) => value ?? "" },
{ field: "description", headerName: "Description", minWidth: 230, flex: 1.2, valueFormatter: ({ value }) => value ?? "" },
{ colId: "actions", headerName: "", width: 64, maxWidth: 64, pinned: "right", sortable: false, filter: false, resizable: false, cellRenderer: ActionCell },
], []);
const context = useMemo<GridContext>(() => ({ canManage, onEdit: edit }), [canManage, data]);
if (editingId && active) {
return (
<div className="mx-auto w-full max-w-4xl px-4 py-6 sm:px-5">
<Button type="button" variant="ghost" className="-ml-2 mb-4" disabled={busy} onClick={closeEditor}>← Back to columns</Button>
<p className="thot-label">{table.name} · column</p>
<h3 className="mt-1 font-heading text-2xl font-semibold">{active.name}</h3>
<p className="mt-1 text-sm text-muted-foreground">Physical schema fields are read-only. Review fields can be edited.</p>
{stale ? (
staleBannerOpen ? <div className="mt-4 rounded-md border border-amber-500/50 bg-amber-500/8 p-4 text-sm"><p className="font-semibold">Newer column metadata is available.</p><p className="mt-1 text-muted-foreground">Your draft is preserved until you reload.</p><div className="mt-3 flex gap-2"><Button type="button" variant="outline" onClick={() => void reloadColumn()}>Reload latest</Button><Button type="button" variant="ghost" onClick={() => setStaleBannerOpen(false)}>Keep editing</Button></div></div>
: <div className="mt-4 flex items-center justify-between gap-3 rounded-md border border-amber-500/40 bg-amber-500/8 px-4 py-3 text-sm"><span>Reload the latest value before this metadata can be saved.</span><Button type="button" variant="outline" onClick={() => void reloadColumn()}>Reload latest</Button></div>
) : null}
<div className="mt-5 grid gap-4 border-t border-border pt-5 sm:grid-cols-2">
<label className="grid gap-1.5 text-sm font-semibold">Physical column name<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={active.name} readOnly /></label>
<label className="grid gap-1.5 text-sm font-semibold">Native type<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={active.dataType} readOnly /></label>
<label className="grid gap-1.5 text-sm font-semibold">Ordinal position<input className="h-9 rounded-md border border-input bg-muted/35 px-3" value={active.ordinalPosition} readOnly /></label>
<label className="grid gap-1.5 text-sm font-semibold">Default expression<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={active.defaultExpression ?? ""} readOnly /></label>
<label className="grid gap-1.5 text-sm font-semibold sm:col-span-2">Source comment<textarea className="min-h-24 rounded-md border border-input bg-muted/35 px-3 py-2 text-sm" value={active.sourceComment ?? ""} readOnly /></label>
<label className="grid gap-1.5 text-sm font-semibold sm:col-span-2">Generated description<textarea className="min-h-28 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={generatedDescription} disabled={!canManage || busy} onChange={(event) => setGeneratedDescription(event.target.value)} /></label>
<label className="grid gap-1.5 text-sm font-semibold sm:col-span-2">Description<textarea className="min-h-32 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={description} disabled={!canManage || busy} onChange={(event) => setDescription(event.target.value)} /></label>
</div>
<div className="mt-5 flex justify-end gap-2 border-t border-border pt-4">
<Button type="button" variant="outline" disabled={busy} onClick={closeEditor}>Cancel</Button>
<Button type="button" disabled={!canManage || !dirty || busy || stale} onClick={() => void save()}><Save />{busy ? "Saving…" : "Save metadata"}</Button>
</div>
</div>
);
}
return (
<div className="flex min-h-0 flex-1 flex-col">
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
<span className="thot-label whitespace-nowrap">Catalog columns</span>
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search columns" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
<Button type="button" disabled={!canManage || busy} onClick={onSync}><RefreshCw />Sync columns</Button>
</div>
<div className="relative min-h-[280px] flex-1">
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
<AgGridReact<CatalogColumn> rowData={data} columnDefs={columns} context={context} loading={isLoading} quickFilterText={search} defaultColDef={{ sortable: true, filter: true, resizable: true }} getRowId={({ data: row }) => row.id} rowHeight={44} headerHeight={38} animateRows={false} overlayNoRowsTemplate="No columns synchronized for this table." />
</div>
</div>
</div>
);
}
@@ -0,0 +1,510 @@
import type { RefObject, ReactNode } from "react";
import {
ArrowLeft,
Database,
RefreshCw,
Save,
TestTube2,
Trash2,
} from "lucide-react";
import { Button } from "../../components/ui/button";
import type {
CatalogDatabase,
CatalogSecretName,
CatalogSyncScope,
CatalogSyncRun,
DatabaseBinding,
DatabaseTransport,
} from "../../api/catalog-databases";
import type {
DatabaseBusyAction,
DatabaseFormDraft,
DatabaseFormMode,
} from "./model";
import { statusLabel } from "./model";
import { DatabaseSyncMenu } from "./DatabaseSyncMenu";
const inputClass = "h-9 w-full rounded-md border border-input bg-card px-3 text-sm outline-none transition focus:border-primary/60 focus:ring-3 focus:ring-ring/15 read-only:bg-muted/40 read-only:text-muted-foreground disabled:bg-muted disabled:text-muted-foreground";
const labelClass = "grid min-w-0 gap-1.5 text-xs font-medium text-foreground";
interface FieldProps {
label: string;
hint?: string;
children: ReactNode;
}
function Field({ label, hint, children }: FieldProps) {
return (
<label className={labelClass}>
<span>{label}</span>
{children}
{hint ? <span className="font-normal leading-4 text-muted-foreground">{hint}</span> : null}
</label>
);
}
interface SecretFieldProps {
label: string;
name: CatalogSecretName;
row: CatalogDatabase;
draft: DatabaseFormDraft;
readOnly: boolean;
canManageSecrets: boolean;
onChange: (name: CatalogSecretName, value: string) => void;
multiline?: boolean;
}
function SecretField({
label,
name,
row,
draft,
readOnly,
canManageSecrets,
onChange,
multiline = false,
}: SecretFieldProps) {
if (readOnly) {
return (
<Field label={label} hint="Secret values are never displayed.">
<input
className={inputClass}
value={row.secrets[name] ? "Configured" : "Not configured"}
readOnly
aria-label={label}
/>
</Field>
);
}
const permissionHint = canManageSecrets
? undefined
: "You do not have permission to replace secret values.";
return (
<Field label={label} hint={permissionHint}>
{multiline ? (
<textarea
className={`${inputClass} min-h-24 resize-y py-2 font-mono text-xs leading-5`}
aria-label={label}
autoComplete="off"
value={draft.secrets[name] ?? ""}
disabled={!canManageSecrets}
placeholder={row.secrets[name] ? "Configured; leave blank to keep" : "Paste value"}
onChange={(event) => onChange(name, event.target.value)}
/>
) : (
<input
className={inputClass}
type="password"
aria-label={label}
autoComplete="new-password"
value={draft.secrets[name] ?? ""}
disabled={!canManageSecrets}
placeholder={row.secrets[name] ? "Configured; leave blank to keep" : "Not configured"}
onChange={(event) => onChange(name, event.target.value)}
/>
)}
</Field>
);
}
interface DatabaseFormProps {
mode: DatabaseFormMode;
row: CatalogDatabase;
availableWorkspaces: CatalogDatabase[];
workspaceLocked: boolean;
draft: DatabaseFormDraft;
dirty: boolean;
canManage: boolean;
canManageSecrets: boolean;
busyAction: DatabaseBusyAction;
stale: boolean;
staleBannerOpen: boolean;
partialSecretFailure: string | null;
headingRef: RefObject<HTMLHeadingElement>;
onBack: () => void;
onWorkspaceChange: (workspaceId: string) => void;
onFieldChange: (field: "databaseName" | "schema", value: string) => void;
onTransportChange: (transport: DatabaseTransport) => void;
onBindingChange: <K extends keyof DatabaseBinding>(key: K, value: DatabaseBinding[K]) => void;
onSecretChange: (name: CatalogSecretName, value: string) => void;
onSave: () => void;
onTest: () => void;
onDelete: () => void;
onReloadLatest: () => void;
onKeepEditing: () => void;
onRetrySecrets: () => void;
onOpenTables: (origin: HTMLButtonElement) => void;
onOpenRelationships: () => void;
onSync: (scope: CatalogSyncScope) => void;
onOpenSync: () => void;
activeSyncRun?: CatalogSyncRun;
}
function formTitle(mode: DatabaseFormMode, workspaceLocked: boolean): string {
if (mode === "add") return workspaceLocked ? "Edit database" : "Add database";
if (mode === "edit") return "Edit database";
if (mode === "delete") return "Delete database";
return "Database details";
}
export function DatabaseForm({
mode,
row,
availableWorkspaces,
workspaceLocked,
draft,
dirty,
canManage,
canManageSecrets,
busyAction,
stale,
staleBannerOpen,
partialSecretFailure,
headingRef,
onBack,
onWorkspaceChange,
onFieldChange,
onTransportChange,
onBindingChange,
onSecretChange,
onSave,
onTest,
onDelete,
onReloadLatest,
onKeepEditing,
onRetrySecrets,
onOpenTables,
onOpenRelationships,
onSync,
onOpenSync,
activeSyncRun,
}: DatabaseFormProps) {
const title = formTitle(mode, workspaceLocked);
const readOnly = mode === "view" || mode === "delete";
const editable = mode === "add" || mode === "edit";
const busy = busyAction !== null;
const testDisabled = !canManage
|| !row.configured
|| !row.id
|| dirty
|| stale
|| busy;
const saveDisabled = !canManage
|| stale
|| busy
|| (mode === "edit" && !dirty);
const bindingReady = row.connectionStatus === "reachable" && row.testedVersion === row.version;
const form = (
<form
className="grid gap-5"
onSubmit={(event) => {
event.preventDefault();
if (editable) onSave();
}}
>
{mode === "delete" ? (
<div className="rounded-md border border-destructive/35 bg-destructive/8 p-4 text-sm text-foreground" role="alert">
<p className="font-semibold text-destructive">This removes the local database configuration.</p>
<p className="mt-1 leading-5 text-muted-foreground">
{row.workspaceAvailable
? "The repository workspace remains available and will return to the list as Not configured. "
: "This orphaned catalog row will disappear from the list. "}
Secret references associated with this configuration are removed too.
</p>
</div>
) : null}
{stale ? (
staleBannerOpen ? (
<div className="rounded-md border border-amber-300/70 bg-amber-50 p-4 text-sm text-amber-950 dark:border-amber-700 dark:bg-amber-950/30 dark:text-amber-100" role="alert">
<p className="font-semibold">A newer database configuration is available.</p>
<p className="mt-1 leading-5">Reload the latest values before saving, testing, or deleting. Your draft is still intact.</p>
<div className="mt-3 flex flex-wrap gap-2">
<Button type="button" size="sm" variant="outline" disabled={busy} onClick={onReloadLatest}>
<RefreshCw /> Reload latest
</Button>
<Button type="button" size="sm" variant="ghost" disabled={busy} onClick={onKeepEditing}>
Keep editing
</Button>
</div>
</div>
) : (
<div className="rounded-md border border-amber-300/70 bg-amber-50 px-4 py-3 text-sm text-amber-950 dark:border-amber-700 dark:bg-amber-950/30 dark:text-amber-100" role="status">
Reload the latest values before this configuration can be changed.
</div>
)
) : null}
{partialSecretFailure ? (
<div className="rounded-md border border-amber-300/70 bg-amber-50 p-4 text-sm text-amber-950 dark:border-amber-700 dark:bg-amber-950/30 dark:text-amber-100" role="alert">
<p className="font-semibold">Database configuration saved; secret update could not be confirmed.</p>
<p className="mt-1 leading-5">{partialSecretFailure} The values you entered are retained in this form.</p>
<Button type="button" size="sm" variant="outline" className="mt-3" disabled={busy || stale} onClick={onRetrySecrets}>
<RefreshCw /> {busyAction === "retry-secrets" ? "Retrying…" : "Retry secrets"}
</Button>
</div>
) : null}
{editable && !canManageSecrets ? (
<div className="rounded-md border border-border bg-muted/45 px-4 py-3 text-sm text-muted-foreground" role="note">
Secret status is visible, but replacing secret values requires the workspace.secrets.manage permission.
</div>
) : null}
{!row.workspaceAvailable ? (
<div className="rounded-md border border-destructive/30 bg-destructive/8 px-4 py-3 text-sm text-destructive" role="alert">
This configuration references a workspace that is no longer present in the repository YAML.
</div>
) : null}
<div className="grid gap-4 md:grid-cols-2">
{mode === "add" ? (
<Field label="Workspace">
<select
className={inputClass}
aria-label="Workspace"
value={draft.workspaceId}
disabled={workspaceLocked || busy}
onChange={(event) => onWorkspaceChange(event.target.value)}
>
{availableWorkspaces.map((workspace) => (
<option key={workspace.workspaceId} value={workspace.workspaceId}>
{workspace.workspaceName}
</option>
))}
</select>
</Field>
) : (
<Field label="Workspace">
<input className={inputClass} value={row.workspaceName} readOnly />
</Field>
)}
<Field label="Workspace ID">
<input className={`${inputClass} font-mono text-xs`} value={row.workspaceId} readOnly />
</Field>
<Field label="Database engine">
<input className={inputClass} value="PostgreSQL" readOnly />
</Field>
<Field label="Transport">
<select
className={inputClass}
aria-label="Transport"
value={draft.binding.transport}
disabled={readOnly || busy}
onChange={(event) => onTransportChange(event.target.value as DatabaseTransport)}
>
<option value="postgres_direct">Direct PostgreSQL</option>
<option value="rest_api">REST API</option>
<option value="ssh_tunnel">SSH tunnel</option>
</select>
</Field>
<Field label="Database name">
<input
className={inputClass}
required={editable}
value={draft.databaseName}
readOnly={readOnly}
disabled={busy}
onChange={(event) => onFieldChange("databaseName", event.target.value)}
/>
</Field>
<Field label="Schema">
<input
className={inputClass}
required={editable}
value={draft.schema}
readOnly={readOnly}
disabled={busy}
onChange={(event) => onFieldChange("schema", event.target.value)}
/>
</Field>
</div>
<div className="border-t border-border pt-5">
<p className="thot-label mb-4">Installation binding</p>
{draft.binding.transport === "postgres_direct" ? (
<div className="grid gap-4 md:grid-cols-2">
<Field label="Host">
<input className={inputClass} required={editable} value={draft.binding.host ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("host", event.target.value)} />
</Field>
<Field label="Port">
<input className={inputClass} type="number" min={1} max={65535} required={editable} value={draft.binding.port ?? 5432} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("port", Number(event.target.value))} />
</Field>
<Field label="Username">
<input className={inputClass} required={editable} value={draft.binding.username ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("username", event.target.value)} />
</Field>
<Field label="TLS server name">
<input className={inputClass} value={draft.binding.tlsServername ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("tlsServername", event.target.value || undefined)} />
</Field>
<SecretField label="Password" name="password" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
<SecretField label="TLS CA certificate" name="tlsCa" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
</div>
) : null}
{draft.binding.transport === "rest_api" ? (
<div className="grid gap-4 md:grid-cols-2">
<Field label="Base URL">
<input className={inputClass} type="url" required={editable} placeholder="https://dwh.example/api" value={draft.binding.baseUrl ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("baseUrl", event.target.value)} />
</Field>
<Field label="Diagnostic endpoint" hint="This relative path is fixed by the runtime contract.">
<input className={inputClass} value={draft.binding.restPath ?? "/health"} readOnly aria-label="Diagnostic endpoint" />
</Field>
<Field label="Authentication">
<select className={inputClass} aria-label="Authentication" value={draft.binding.restAuth ?? "x-api-key"} disabled={readOnly || busy} onChange={(event) => onBindingChange("restAuth", event.target.value as DatabaseBinding["restAuth"])}>
<option value="x-api-key">X-API-Key</option>
<option value="bearer">Bearer token</option>
<option value="none">None</option>
</select>
</Field>
{draft.binding.restAuth !== "none" ? (
<SecretField label="API key" name="apiKey" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
) : null}
</div>
) : null}
{draft.binding.transport === "ssh_tunnel" ? (
<div className="grid gap-4 md:grid-cols-2">
<div className="rounded-md border border-border bg-muted/45 px-4 py-3 text-sm text-muted-foreground md:col-span-2" role="note">
SSH host verification is strict. Provide the trusted known_hosts entry before testing this binding.
</div>
<Field label="Database username">
<input className={inputClass} required={editable} value={draft.binding.username ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("username", event.target.value)} />
</Field>
<SecretField label="Database password" name="password" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
<Field label="SSH host">
<input className={inputClass} required={editable} value={draft.binding.sshHost ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshHost", event.target.value)} />
</Field>
<Field label="SSH port">
<input className={inputClass} type="number" min={1} max={65535} required={editable} value={draft.binding.sshPort ?? 22} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshPort", Number(event.target.value))} />
</Field>
<Field label="SSH username">
<input className={inputClass} required={editable} value={draft.binding.sshUsername ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshUsername", event.target.value)} />
</Field>
<SecretField label="SSH private key" name="sshPrivateKey" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} multiline />
<SecretField label="Private key passphrase" name="sshPrivateKeyPassphrase" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
<SecretField label="SSH known hosts" name="sshKnownHosts" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} multiline />
<Field label="Target host">
<input className={inputClass} required={editable} value={draft.binding.sshTargetHost ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshTargetHost", event.target.value)} />
</Field>
<Field label="Target port">
<input className={inputClass} type="number" min={1} max={65535} required={editable} value={draft.binding.sshTargetPort ?? 5432} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshTargetPort", Number(event.target.value))} />
</Field>
<Field label="TLS server name">
<input className={inputClass} value={draft.binding.tlsServername ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("tlsServername", event.target.value || undefined)} />
</Field>
<SecretField label="TLS CA certificate" name="tlsCa" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} multiline />
</div>
) : null}
</div>
{row.lastErrorMessage ? (
<div className="rounded-md border border-destructive/25 bg-destructive/8 px-4 py-3 text-sm text-destructive" role="status">
Last connection error: {row.lastErrorMessage}
</div>
) : null}
<div className="flex flex-wrap items-center justify-end gap-2 border-t border-border pt-4">
{mode === "view" || mode === "edit" ? (
<Button type="button" variant="outline" disabled={testDisabled} onClick={onTest}>
<TestTube2 /> {busyAction === "test" ? "Testing…" : "Test connection"}
</Button>
) : null}
{mode === "add" || mode === "edit" ? (
<Button type="submit" disabled={saveDisabled}>
<Save />
{busyAction === "save"
? "Saving…"
: mode === "add"
? workspaceLocked ? "Save database" : "Add database"
: "Save changes"}
</Button>
) : null}
{mode === "delete" ? (
<Button type="button" variant="destructive" disabled={!canManage || !row.configured || !row.id || busy || stale} onClick={onDelete}>
<Trash2 /> {busyAction === "delete" ? "Deleting…" : "Delete database"}
</Button>
) : null}
</div>
</form>
);
return (
<section aria-label={`${title} form`} className="min-h-0 flex-1 overflow-y-auto bg-background">
<div className="mx-auto w-full max-w-[900px] px-4 py-5 sm:px-6 sm:py-6">
<Button type="button" variant="ghost" className="-ml-2 mb-5" disabled={busy} onClick={onBack}>
<ArrowLeft /> Back to list
</Button>
<div className="mb-6 flex flex-wrap items-start justify-between gap-4 border-b border-border pb-5">
<div className="min-w-0">
<p className="thot-label mb-1">Database management</p>
<h2 ref={headingRef} tabIndex={-1} className="font-heading text-2xl font-semibold tracking-tight outline-none">
{title}
</h2>
<p className="mt-1 text-sm text-muted-foreground">
{row.workspaceName} · {row.workspaceId}
</p>
{row.lastTestedAt ? (
<p className="mt-1 text-xs text-muted-foreground">Last tested {new Date(row.lastTestedAt).toLocaleString()}</p>
) : null}
</div>
<div className="flex flex-wrap items-center justify-end gap-2">
<Database className="size-4 text-muted-foreground" aria-hidden="true" />
<span className="rounded-full border border-border bg-muted px-2.5 py-1 text-xs font-semibold text-muted-foreground">
{statusLabel(row)}
</span>
{mode === "view" && activeSyncRun ? (
<Button type="button" variant="outline" onClick={onOpenSync}><RefreshCw className="animate-spin" />View sync</Button>
) : null}
{mode === "view" ? (
<DatabaseSyncMenu
disabled={!canManage || !bindingReady || busy || Boolean(activeSyncRun)}
disabledReason={!bindingReady ? "Test the current database binding before synchronizing the schema" : undefined}
onSelect={onSync}
/>
) : null}
</div>
</div>
{mode === "view" ? (
<nav aria-label="Database sections" className="mb-6 flex gap-1 border-b border-border" role="tablist">
<button
type="button"
role="tab"
aria-selected="true"
className="border-b-2 border-primary px-3 py-2 text-sm font-semibold text-foreground"
>
Overview
</button>
<button
type="button"
role="tab"
aria-selected="false"
disabled={!row.configured || !row.id}
className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground transition hover:text-foreground focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/20 disabled:cursor-not-allowed disabled:opacity-45"
onClick={(event) => onOpenTables(event.currentTarget)}
>
Tables
</button>
<button
type="button"
role="tab"
aria-selected="false"
disabled={!row.configured || !row.id}
className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground transition hover:text-foreground focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/20 disabled:cursor-not-allowed disabled:opacity-45"
onClick={onOpenRelationships}
>
Relationships
</button>
</nav>
) : null}
{form}
</div>
</section>
);
}
@@ -0,0 +1,329 @@
import { useEffect, useMemo, useRef, useState, type RefObject } from "react";
import { Menu } from "@base-ui/react/menu";
import { AgGridReact } from "ag-grid-react";
import {
AllCommunityModule,
ModuleRegistry,
type ColDef,
type ICellRendererParams,
} from "ag-grid-community";
import "ag-grid-community/styles/ag-grid.css";
import "ag-grid-community/styles/ag-theme-alpine.css";
import { ChevronDown, Eye, Pencil, RefreshCw, Trash2, X } from "lucide-react";
import { Button } from "../../components/ui/button";
import type { CatalogDatabase, CatalogSyncScope } from "../../api/catalog-databases";
import { statusLabel } from "./model";
import { databaseSyncItemClass, databaseSyncScopes } from "./DatabaseSyncMenu";
ModuleRegistry.registerModules([AllCommunityModule]);
interface DatabaseGridProps {
rows: CatalogDatabase[];
isLoading: boolean;
isFetching: boolean;
search: string;
canManage: boolean;
searchInputRef: RefObject<HTMLInputElement>;
onSearchChange: (value: string) => void;
onView: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
onEdit: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
onDelete: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
onOpenSync: (row: CatalogDatabase) => void;
onTestSelected: (rows: CatalogDatabase[]) => Promise<void>;
onSyncSelected: (rows: CatalogDatabase[], scope: CatalogSyncScope) => Promise<void>;
}
interface DatabaseGridContext {
canManage: boolean;
onView: DatabaseGridProps["onView"];
onEdit: DatabaseGridProps["onEdit"];
onDelete: DatabaseGridProps["onDelete"];
onOpenSync: DatabaseGridProps["onOpenSync"];
}
function useCompactViewport(): boolean {
const [compact, setCompact] = useState(false);
useEffect(() => {
if (typeof window.matchMedia !== "function") return;
const query = window.matchMedia("(max-width: 767px)");
const update = () => setCompact(query.matches);
update();
query.addEventListener("change", update);
return () => query.removeEventListener("change", update);
}, []);
return compact;
}
function StatusPill({ row }: { row: CatalogDatabase }) {
const tone = !row.workspaceAvailable
? "border-destructive/30 bg-destructive/10 text-destructive"
: !row.configured
? "border-border bg-muted text-muted-foreground"
: row.connectionStatus === "reachable"
? "border-emerald-300/70 bg-emerald-50 text-emerald-800 dark:bg-emerald-950/30 dark:text-emerald-300"
: row.connectionStatus === "failed"
? "border-destructive/30 bg-destructive/10 text-destructive"
: "border-amber-300/70 bg-amber-50 text-amber-900 dark:bg-amber-950/30 dark:text-amber-300";
return <div className="flex items-center gap-1.5"><span className={`inline-flex max-w-full truncate rounded-full border px-2 py-0.5 text-[11px] font-semibold ${tone}`}>{statusLabel(row)}</span>{row.activeSyncRun ? <span className="inline-flex rounded-full border border-primary/30 bg-primary/8 px-2 py-0.5 text-[11px] font-semibold text-primary">Syncing</span> : null}</div>;
}
function DatabaseActionsCell({
data,
context,
}: ICellRendererParams<CatalogDatabase, unknown, DatabaseGridContext>) {
if (!data || !context) return null;
const editDisabled = !context.canManage || !data.workspaceAvailable;
const deleteDisabled = !context.canManage || !data.configured;
const editLabel = `Edit ${data.workspaceName}`;
const editReasonId = `database-edit-reason-${data.workspaceId}`;
const deleteReasonId = `database-delete-reason-${data.workspaceId}`;
return (
<div className="flex h-full items-center justify-end gap-0.5" onClick={(event) => event.stopPropagation()}>
{data.activeSyncRun ? (
<Button type="button" variant="ghost" size="icon-lg" title={`View synchronization for ${data.workspaceName}`} aria-label={`View synchronization for ${data.workspaceName}`} onClick={() => context.onOpenSync(data)}>
<RefreshCw className="animate-spin" aria-hidden="true" />
</Button>
) : null}
<Button
type="button"
variant="ghost"
size="icon-lg"
title={`View ${data.workspaceName}`}
aria-label={`View ${data.workspaceName}`}
onClick={(event) => context.onView(data, event.currentTarget)}
>
<Eye aria-hidden="true" />
</Button>
<span title={editDisabled ? "This database configuration cannot be changed" : editLabel}>
<Button
type="button"
variant="ghost"
size="icon-lg"
aria-label={editLabel}
aria-describedby={editDisabled ? editReasonId : undefined}
disabled={editDisabled}
onClick={(event) => context.onEdit(data, event.currentTarget)}
>
<Pencil aria-hidden="true" />
</Button>
{editDisabled ? <span id={editReasonId} className="sr-only">This database configuration cannot be changed.</span> : null}
</span>
<span title={deleteDisabled ? "Save this database configuration before deleting it" : `Delete ${data.workspaceName}`}>
<Button
type="button"
variant="ghost"
size="icon-lg"
className="text-destructive hover:bg-destructive/10 hover:text-destructive"
aria-label={`Delete ${data.workspaceName}`}
aria-describedby={deleteDisabled ? deleteReasonId : undefined}
disabled={deleteDisabled}
onClick={(event) => context.onDelete(data, event.currentTarget)}
>
<Trash2 aria-hidden="true" />
</Button>
{deleteDisabled ? <span id={deleteReasonId} className="sr-only">Save this database configuration before deleting it.</span> : null}
</span>
</div>
);
}
export function DatabaseGrid({
rows,
isLoading,
isFetching,
search,
canManage,
searchInputRef,
onSearchChange,
onView,
onEdit,
onDelete,
onOpenSync,
onTestSelected,
onSyncSelected,
}: DatabaseGridProps) {
const compact = useCompactViewport();
const gridRef = useRef<AgGridReact<CatalogDatabase>>(null);
const [selectedRows, setSelectedRows] = useState<CatalogDatabase[]>([]);
const [action, setAction] = useState<"test" | "sync" | null>(null);
const context = useMemo<DatabaseGridContext>(
() => ({ canManage, onView, onEdit, onDelete, onOpenSync }),
[canManage, onView, onEdit, onDelete, onOpenSync],
);
const columnDefs = useMemo<ColDef<CatalogDatabase>[]>(() => {
const workspace: ColDef<CatalogDatabase> = {
field: "workspaceName",
headerName: "Workspace",
minWidth: compact ? 112 : 190,
flex: compact ? undefined : 1.35,
width: compact ? 112 : undefined,
};
const database: ColDef<CatalogDatabase> = {
field: "databaseName", headerName: "Database", minWidth: 145, flex: 1,
};
const schema: ColDef<CatalogDatabase> = {
field: "schema", headerName: "Schema", minWidth: 140, flex: 0.9,
};
const transport: ColDef<CatalogDatabase> = {
field: "binding.transport",
headerName: "Transport",
minWidth: 135,
flex: 0.9,
valueFormatter: ({ value }) => String(value ?? "").replaceAll("_", " "),
};
const endpoint: ColDef<CatalogDatabase> = {
headerName: "Endpoint",
minWidth: 190,
flex: 1.2,
valueGetter: ({ data }) => {
if (!data) return "";
if (data.binding.transport === "rest_api") return data.binding.baseUrl ?? "";
if (data.binding.transport === "ssh_tunnel") return data.binding.sshHost ?? "";
return data.binding.host ? `${data.binding.host}:${data.binding.port ?? 5432}` : "";
},
};
const status: ColDef<CatalogDatabase> = {
headerName: "Status",
minWidth: compact ? 96 : 130,
width: compact ? 96 : undefined,
flex: compact ? undefined : 0.8,
cellClass: compact ? "thot-database-status-cell" : undefined,
valueGetter: ({ data }) => (data ? statusLabel(data) : ""),
cellRenderer: ({ data }: ICellRendererParams<CatalogDatabase>) => (
data ? <StatusPill row={data} /> : null
),
};
const updated: ColDef<CatalogDatabase> = {
field: "updatedAt",
headerName: "Updated",
minWidth: 170,
flex: 0.9,
valueFormatter: ({ value }) => value ? new Date(String(value)).toLocaleString() : "",
};
const actionsWidth = compact ? 148 : 164;
const actions: ColDef<CatalogDatabase> = {
colId: "actions",
headerName: "Actions",
width: actionsWidth,
minWidth: actionsWidth,
maxWidth: actionsWidth,
pinned: "right",
lockPinned: true,
sortable: false,
filter: false,
resizable: false,
suppressMovable: true,
suppressHeaderMenuButton: true,
cellClass: "thot-database-actions-cell",
cellRenderer: DatabaseActionsCell,
};
return compact
? [workspace, status, database, schema, transport, endpoint, updated, actions]
: [workspace, database, schema, transport, endpoint, status, updated, actions];
}, [compact]);
const configuredCount = rows.filter((row) => row.configured).length;
const hiddenSelected = selectedRows.filter((row) => {
const term = search.trim().toLocaleLowerCase();
return term && ![row.workspaceName, row.databaseName, row.schema].some((value) => value.toLocaleLowerCase().includes(term));
}).length;
const canTestSelection = canManage && selectedRows.length > 0 && selectedRows.every((row) => row.configured && row.id && !row.activeSyncRun);
const canSyncSelection = canManage && selectedRows.length > 0 && selectedRows.every((row) => row.configured && row.id && row.connectionStatus === "reachable" && row.testedVersion === row.version && !row.activeSyncRun);
const perform = async (kind: "test" | "sync", operation: () => Promise<void>) => {
setAction(kind);
try {
await operation();
gridRef.current?.api.deselectAll();
setSelectedRows([]);
} finally { setAction(null); }
};
return (
<section aria-label="Workspace databases" className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
{selectedRows.length > 0 ? (
<>
<span className="text-sm font-semibold">{selectedRows.length} selected</span>
{hiddenSelected ? <span className="text-xs text-muted-foreground">{hiddenSelected} hidden by filter</span> : null}
<Menu.Root>
<Menu.Trigger className="inline-flex h-8 items-center justify-center gap-2 rounded-md border border-input bg-background px-3 text-sm font-medium hover:bg-muted disabled:pointer-events-none disabled:opacity-50" disabled={action !== null}>Actions <ChevronDown className="size-4" /></Menu.Trigger>
<Menu.Portal>
<Menu.Positioner side="bottom" align="start" sideOffset={4}>
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canTestSelection} onClick={() => void perform("test", () => onTestSelected(selectedRows))}>Test connections</Menu.Item>
<Menu.Separator className="my-1 h-px bg-border" />
{databaseSyncScopes.map(({ scope, label }) => (
<Menu.Item
key={scope}
className={databaseSyncItemClass}
disabled={!canSyncSelection}
onClick={() => void perform("sync", () => onSyncSelected(selectedRows, scope))}
>
{label}
</Menu.Item>
))}
</Menu.Popup>
</Menu.Positioner>
</Menu.Portal>
</Menu.Root>
<Button type="button" variant="ghost" disabled={action !== null} onClick={() => { gridRef.current?.api.deselectAll(); setSelectedRows([]); }}><X />Clear</Button>
</>
) : <><span className="thot-label whitespace-nowrap">Workspace databases</span><input
ref={searchInputRef}
className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none transition focus:border-primary/60 focus:ring-3 focus:ring-ring/15"
aria-label="Search databases"
placeholder="Search"
value={search}
onChange={(event) => onSearchChange(event.target.value)}
/></>}
<span className="whitespace-nowrap text-xs tabular-nums text-muted-foreground">
{configuredCount}/{rows.length}
</span>
{isFetching && !isLoading ? (
<span className="text-xs text-muted-foreground" role="status">Refreshing</span>
) : null}
</div>
<div className="relative min-h-[280px] flex-1">
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
<AgGridReact<CatalogDatabase>
ref={gridRef}
rowData={rows}
columnDefs={columnDefs}
context={context}
loading={isLoading}
quickFilterText={search}
defaultColDef={{
sortable: true,
filter: true,
resizable: true,
suppressHeaderMenuButton: false,
}}
getRowId={({ data }) => data.workspaceId}
rowSelection={{ mode: "multiRow", selectAll: "filtered", enableClickSelection: false }}
selectionColumnDef={{ width: 44, maxWidth: 44, pinned: "left" }}
onSelectionChanged={({ api }) => setSelectedRows(api.getSelectedRows())}
rowHeight={44}
headerHeight={38}
animateRows={false}
/>
</div>
{isLoading ? (
<div className="pointer-events-none absolute inset-0 grid place-items-center bg-card/80 text-sm text-muted-foreground" role="status">
Loading database configurations…
</div>
) : rows.length === 0 ? (
<div className="pointer-events-none absolute inset-0 grid place-items-center bg-card px-6 text-center text-sm text-muted-foreground" role="status">
No repository workspaces are available for database configuration.
</div>
) : null}
</div>
</section>
);
}
@@ -0,0 +1,103 @@
import { useMemo, useState } from "react";
import { useQuery } from "@tanstack/react-query";
import { AgGridReact } from "ag-grid-react";
import type { ColDef } from "ag-grid-community";
import { ArrowLeft, RefreshCw } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../../components/ui/button";
import { apiErrorMessage } from "../../api/client";
import {
listCatalogRelationships,
startCatalogSync,
type CatalogDatabase,
type CatalogRelationship,
type CatalogSyncRun,
} from "../../api/catalog-databases";
interface Props {
database: CatalogDatabase;
canManage: boolean;
onBackToDatabases: () => void;
onOpenOverview: () => void;
onOpenTables: () => void;
onRunStarted: (run: CatalogSyncRun) => void;
onOpenSync: () => void;
}
export function DatabaseRelationships({
database,
canManage,
onBackToDatabases,
onOpenOverview,
onOpenTables,
onRunStarted,
onOpenSync,
}: Props) {
const databaseId = database.id!;
const { data = [], isLoading, isFetching, refetch } = useQuery({
queryKey: ["catalog-relationships", databaseId],
queryFn: () => listCatalogRelationships(databaseId),
retry: false,
});
const [search, setSearch] = useState("");
const [busy, setBusy] = useState(false);
const bindingReady = database.connectionStatus === "reachable" && database.testedVersion === database.version;
const synchronize = async (scope: "relationships" | "all") => {
setBusy(true);
try {
const run = await startCatalogSync(databaseId, database.version, scope);
onRunStarted(run);
toast.success(scope === "all" ? "Full schema synchronization started" : "Relationship synchronization started");
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(false); }
};
const columns = useMemo<ColDef<CatalogRelationship>[]>(() => [
{ field: "constraintName", headerName: "Constraint", minWidth: 220, flex: 1, cellClass: "font-mono text-xs" },
{ field: "sourceTableName", headerName: "Source table", minWidth: 200, flex: 1 },
{ headerName: "Source columns", minWidth: 200, flex: 1, valueGetter: ({ data: row }) => row?.columns.map((pair) => pair.sourceColumnName).join(", ") ?? "" },
{ field: "targetTableName", headerName: "Target table", minWidth: 200, flex: 1 },
{ headerName: "Target columns", minWidth: 200, flex: 1, valueGetter: ({ data: row }) => row?.columns.map((pair) => pair.targetColumnName).join(", ") ?? "" },
{ field: "updateRule", headerName: "On update", minWidth: 125, width: 125 },
{ field: "deleteRule", headerName: "On delete", minWidth: 125, width: 125 },
], []);
return (
<section aria-label={`Relationships for ${database.workspaceName}`} className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
<div className="px-4 pt-5 sm:px-5">
<Button type="button" variant="ghost" className="-ml-2 mb-4" disabled={busy} onClick={onBackToDatabases}><ArrowLeft />Back to databases</Button>
<div className="flex flex-wrap items-start justify-between gap-4 border-b border-border pb-5">
<div>
<p className="thot-label mb-1">Database management</p>
<h2 className="font-heading text-2xl font-semibold tracking-tight">{database.workspaceName}</h2>
<p className="mt-1 text-sm text-muted-foreground">{database.databaseName} · {database.schema}</p>
</div>
<div className="flex gap-2">
{database.activeSyncRun ? <Button type="button" variant="outline" onClick={onOpenSync}><RefreshCw className="animate-spin" />View sync</Button> : null}
<Button type="button" disabled={!canManage || !bindingReady || busy || Boolean(database.activeSyncRun)} onClick={() => void synchronize("all")}><RefreshCw />Sync all</Button>
</div>
</div>
<nav aria-label="Database sections" className="flex gap-1 border-b border-border" role="tablist">
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={onOpenOverview}>Overview</button>
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={onOpenTables}>Tables</button>
<button type="button" role="tab" aria-selected="true" className="border-b-2 border-primary px-3 py-2 text-sm font-semibold text-foreground">Relationships</button>
</nav>
</div>
<div className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
<span className="thot-label whitespace-nowrap">Physical relationships</span>
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search relationships" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
<Button type="button" disabled={!canManage || !bindingReady || busy || Boolean(database.activeSyncRun)} title={!bindingReady ? "Test the current database binding before synchronizing relationships" : undefined} onClick={() => void synchronize("relationships")}><RefreshCw />Sync relationships</Button>
</div>
{!bindingReady ? <div className="border-b border-border bg-muted/35 px-4 py-3 text-sm text-muted-foreground">Test the current database binding from Overview before synchronizing relationships.</div> : null}
<div className="relative min-h-[280px] flex-1">
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
<AgGridReact<CatalogRelationship> rowData={data} columnDefs={columns} loading={isLoading} quickFilterText={search} defaultColDef={{ sortable: true, filter: true, resizable: true }} getRowId={({ data: row }) => row.id} rowHeight={44} headerHeight={38} animateRows={false} overlayNoRowsTemplate="No physical relationships synchronized for this database." />
</div>
</div>
</div>
</section>
);
}
@@ -0,0 +1,54 @@
import { Menu } from "@base-ui/react/menu";
import { ChevronDown, RefreshCw } from "lucide-react";
import { buttonVariants } from "../../components/ui/button";
import type { CatalogSyncScope } from "../../api/catalog-databases";
export const databaseSyncItemClass = [
"flex cursor-default select-none items-center rounded-md px-3 py-2 text-sm outline-none",
"data-[highlighted]:bg-muted data-[disabled]:opacity-45",
].join(" ");
export const databaseSyncScopes: ReadonlyArray<{ scope: CatalogSyncScope; label: string }> = [
{ scope: "tables", label: "Synchronize tables" },
{ scope: "columns", label: "Synchronize all columns" },
{ scope: "relationships", label: "Synchronize relationships" },
{ scope: "all", label: "Synchronize all" },
];
interface DatabaseSyncMenuProps {
disabled: boolean;
disabledReason?: string;
onSelect: (scope: CatalogSyncScope) => void;
}
export function DatabaseSyncMenu({ disabled, disabledReason, onSelect }: DatabaseSyncMenuProps) {
return (
<Menu.Root>
<Menu.Trigger
type="button"
className={buttonVariants()}
disabled={disabled}
title={disabledReason}
aria-label="Synchronize database schema"
>
<RefreshCw />Synchronize<ChevronDown />
</Menu.Trigger>
<Menu.Portal>
<Menu.Positioner side="bottom" align="end" sideOffset={4}>
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
{databaseSyncScopes.map(({ scope, label }) => (
<Menu.Item
key={scope}
className={databaseSyncItemClass}
disabled={disabled}
onClick={() => onSelect(scope)}
>
{label}
</Menu.Item>
))}
</Menu.Popup>
</Menu.Positioner>
</Menu.Portal>
</Menu.Root>
);
}
@@ -0,0 +1,315 @@
import { useEffect, useMemo, useRef, useState } from "react";
import { useQuery, useQueryClient } from "@tanstack/react-query";
import { AgGridReact } from "ag-grid-react";
import type { ColDef, ICellRendererParams } from "ag-grid-community";
import { ArrowLeft, Columns3, Pencil, RefreshCw, Save, X } from "lucide-react";
import { toast } from "sonner";
import { Button } from "../../components/ui/button";
import { ApiError, apiErrorMessage } from "../../api/client";
import {
listCatalogTables,
startCatalogSync,
updateCatalogTableMetadata,
type CatalogDatabase,
type CatalogSyncRun,
type CatalogTable,
} from "../../api/catalog-databases";
import type { DatabaseNavigationState } from "./model";
import { DatabaseColumns } from "./DatabaseColumns";
interface Props {
database: CatalogDatabase;
canManage: boolean;
activeRun?: CatalogSyncRun;
onBackToDatabases: () => void;
onOpenOverview: () => void;
onOpenRelationships: () => void;
onNavigationStateChange: (state: DatabaseNavigationState) => void;
onRunStarted: (run: CatalogSyncRun) => void;
onOpenSync: () => void;
}
interface TableGridContext {
canManage: boolean;
onOpen: (table: CatalogTable, section: "overview" | "columns", origin: HTMLButtonElement) => void;
}
function TableActionsCell({ data, context }: ICellRendererParams<CatalogTable, unknown, TableGridContext>) {
if (!data || !context) return null;
return (
<div className="flex h-full items-center justify-end" onClick={(event) => event.stopPropagation()}>
<Button type="button" variant="ghost" size="icon-lg" aria-label={`View columns for ${data.name}`} title={`View columns for ${data.name}`} onClick={(event) => context.onOpen(data, "columns", event.currentTarget)}>
<Columns3 aria-hidden="true" />
</Button>
<Button type="button" variant="ghost" size="icon-lg" aria-label={`Edit description for ${data.name}`} title={`Edit metadata for ${data.name}`} disabled={!context.canManage} onClick={(event) => context.onOpen(data, "overview", event.currentTarget)}>
<Pencil aria-hidden="true" />
</Button>
</div>
);
}
export function DatabaseTables({
database,
canManage,
activeRun,
onBackToDatabases,
onOpenOverview,
onOpenRelationships,
onNavigationStateChange,
onRunStarted,
onOpenSync,
}: Props) {
const databaseId = database.id!;
const queryClient = useQueryClient();
const queryKey = ["catalog-tables", databaseId] as const;
const { data = [], isLoading, isFetching, refetch } = useQuery({ queryKey, queryFn: () => listCatalogTables(databaseId), retry: false });
const [search, setSearch] = useState("");
const [displayedCount, setDisplayedCount] = useState(0);
const [selectedIds, setSelectedIds] = useState<string[]>([]);
const [activeTableId, setActiveTableId] = useState<string | null>(null);
const [tableSection, setTableSection] = useState<"overview" | "columns">("overview");
const [description, setDescription] = useState("");
const [generatedDescription, setGeneratedDescription] = useState("");
const [baseline, setBaseline] = useState("");
const [editorVersion, setEditorVersion] = useState<number | null>(null);
const [stale, setStale] = useState(false);
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
const [busy, setBusy] = useState<"sync" | "save" | null>(null);
const [columnNavigation, setColumnNavigation] = useState<DatabaseNavigationState>({ dirty: false, busy: false });
const gridRef = useRef<AgGridReact<CatalogTable>>(null);
const originRef = useRef<HTMLButtonElement | null>(null);
const headingRef = useRef<HTMLHeadingElement>(null);
const activeTable = activeTableId ? data.find((table) => table.id === activeTableId) : undefined;
const editorFingerprint = JSON.stringify([description, generatedDescription]);
const editorDirty = Boolean(activeTableId && editorFingerprint !== baseline);
const dirty = tableSection === "columns" ? columnNavigation.dirty : editorDirty;
const navigationBusy = busy !== null || (tableSection === "columns" && columnNavigation.busy);
const bindingReady = database.connectionStatus === "reachable" && database.testedVersion === database.version;
const currentRun = activeRun ?? database.activeSyncRun;
useEffect(() => { onNavigationStateChange({ dirty, busy: navigationBusy }); }, [dirty, navigationBusy, onNavigationStateChange]);
useEffect(() => {
if (!activeTableId || !activeTable || editorVersion === activeTable.version || busy) return;
setStale(true);
setStaleBannerOpen(true);
}, [activeTable, activeTableId, busy, editorVersion]);
const openTable = (table: CatalogTable, section: "overview" | "columns", origin: HTMLButtonElement) => {
originRef.current = origin;
setActiveTableId(table.id);
setTableSection(section);
setDescription(table.description ?? "");
setGeneratedDescription(table.generatedDescription ?? "");
setBaseline(JSON.stringify([table.description ?? "", table.generatedDescription ?? ""]));
setEditorVersion(table.version);
setStale(false);
setStaleBannerOpen(true);
setColumnNavigation({ dirty: false, busy: false });
};
const leaveTable = () => {
if (navigationBusy) return;
if (dirty && !window.confirm("Discard unsaved metadata?")) return;
setActiveTableId(null);
setTableSection("overview");
window.setTimeout(() => originRef.current?.focus(), 0);
};
const navigateDatabase = (target: "databases" | "overview" | "relationships") => {
if (navigationBusy) return;
if (dirty && !window.confirm("Discard unsaved metadata?")) return;
if (target === "databases") onBackToDatabases();
else if (target === "overview") onOpenOverview();
else onOpenRelationships();
};
const save = async () => {
if (!activeTable || editorVersion === null) return;
setBusy("save");
try {
const updated = await updateCatalogTableMetadata(
databaseId, activeTable.id, editorVersion,
description.trim() || null, generatedDescription.trim() || null,
);
queryClient.setQueryData<CatalogTable[]>(queryKey, (current = []) => current.map((table) => table.id === updated.id ? updated : table));
setDescription(updated.description ?? "");
setGeneratedDescription(updated.generatedDescription ?? "");
setBaseline(JSON.stringify([updated.description ?? "", updated.generatedDescription ?? ""]));
setEditorVersion(updated.version);
setStale(false);
toast.success("Table metadata saved");
} catch (error) {
if (error instanceof ApiError && error.code === "table_stale") {
await refetch();
setStale(true);
setStaleBannerOpen(true);
} else toast.error(apiErrorMessage(error));
} finally { setBusy(null); }
};
const reloadTable = async () => {
if (!activeTableId) return;
setBusy("sync");
try {
const result = await refetch();
const latest = result.data?.find((table) => table.id === activeTableId);
if (!latest) { leaveTable(); return; }
setDescription(latest.description ?? "");
setGeneratedDescription(latest.generatedDescription ?? "");
setBaseline(JSON.stringify([latest.description ?? "", latest.generatedDescription ?? ""]));
setEditorVersion(latest.version);
setStale(false);
setStaleBannerOpen(true);
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(null); }
};
const synchronize = async (scope: "tables" | "columns" | "all", tableIds: string[] = []) => {
setBusy("sync");
try {
const run = await startCatalogSync(databaseId, database.version, scope, tableIds);
onRunStarted(run);
gridRef.current?.api.deselectAll();
setSelectedIds([]);
toast.success(scope === "all" ? "Full schema synchronization started" : scope === "columns" ? "Column synchronization started" : "Table synchronization started");
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(null); }
};
const columns = useMemo<ColDef<CatalogTable>[]>(() => [
{ field: "name", headerName: "Name", minWidth: 250, flex: 1, cellClass: "font-mono text-xs" },
{ field: "sourceComment", headerName: "Source comment", minWidth: 260, flex: 1.25, valueFormatter: ({ value }) => value ?? "" },
{ field: "generatedDescription", headerName: "Generated description", minWidth: 280, flex: 1.25, valueFormatter: ({ value }) => value ?? "" },
{ field: "description", headerName: "Description", minWidth: 280, flex: 1.25, valueFormatter: ({ value }) => value ?? "" },
{ colId: "actions", headerName: "", width: 104, minWidth: 104, maxWidth: 104, pinned: "right", sortable: false, filter: false, resizable: false, cellRenderer: TableActionsCell },
], []);
const context = useMemo<TableGridContext>(() => ({ canManage, onOpen: openTable }), [canManage, data]);
const visibleSelected = useMemo(() => {
const term = search.trim().toLocaleLowerCase();
if (!term) return selectedIds.length;
const selected = new Set(selectedIds);
return data.filter((table) => selected.has(table.id) && [table.name, table.sourceComment, table.generatedDescription, table.description]
.some((value) => value?.toLocaleLowerCase().includes(term))).length;
}, [data, search, selectedIds]);
const databaseHeader = (
<div className="px-4 pt-5 sm:px-5">
<Button type="button" variant="ghost" className="-ml-2 mb-4" disabled={navigationBusy} onClick={() => navigateDatabase("databases")}><ArrowLeft />Back to databases</Button>
<div className="flex flex-wrap items-start justify-between gap-4 border-b border-border pb-5">
<div>
<p className="thot-label mb-1">Database management</p>
<h2 ref={headingRef} tabIndex={-1} className="font-heading text-2xl font-semibold tracking-tight outline-none">{database.workspaceName}</h2>
<p className="mt-1 text-sm text-muted-foreground">{database.databaseName} · {database.schema}</p>
</div>
<div className="flex gap-2">
{currentRun ? <Button type="button" variant="outline" onClick={onOpenSync}><RefreshCw className="animate-spin" />View sync</Button> : null}
<Button type="button" disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)} onClick={() => void synchronize("all")}><RefreshCw />Sync all</Button>
</div>
</div>
<nav aria-label="Database sections" className="flex gap-1 border-b border-border" role="tablist">
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={() => navigateDatabase("overview")}>Overview</button>
<button type="button" role="tab" aria-selected="true" className="border-b-2 border-primary px-3 py-2 text-sm font-semibold text-foreground">Tables</button>
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={() => navigateDatabase("relationships")}>Relationships</button>
</nav>
</div>
);
if (activeTable) {
return (
<section aria-label={`Table ${activeTable.name}`} className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
{databaseHeader}
<div className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
<div className="border-b border-border px-4 pt-4">
<Button type="button" variant="ghost" className="-ml-2 mb-2" onClick={leaveTable}><ArrowLeft />Back to tables</Button>
<p className="thot-label">Physical table</p>
<h3 className="mt-1 font-heading text-xl font-semibold">{activeTable.name}</h3>
<nav aria-label="Table sections" className="mt-3 flex gap-1" role="tablist">
<button type="button" role="tab" aria-selected={tableSection === "overview"} className={`border-b-2 px-3 py-2 text-sm font-semibold ${tableSection === "overview" ? "border-primary text-foreground" : "border-transparent text-muted-foreground"}`} onClick={() => setTableSection("overview")}>Overview</button>
<button type="button" role="tab" aria-selected={tableSection === "columns"} className={`border-b-2 px-3 py-2 text-sm font-semibold ${tableSection === "columns" ? "border-primary text-foreground" : "border-transparent text-muted-foreground"}`} onClick={() => setTableSection("columns")}>Columns</button>
</nav>
</div>
{tableSection === "columns" ? (
<DatabaseColumns databaseId={databaseId} table={activeTable} canManage={canManage} onNavigationStateChange={setColumnNavigation} onSync={() => void synchronize("columns", [activeTable.id])} />
) : (
<div className="min-h-0 overflow-y-auto px-4 py-5">
<p className="text-sm text-muted-foreground">Only review metadata can be changed.</p>
{stale ? (
staleBannerOpen ? (
<div className="mt-4 rounded-md border border-amber-500/50 bg-amber-500/8 p-4 text-sm">
<p className="font-semibold">A newer table description is available.</p>
<p className="mt-1 text-muted-foreground">Your draft is preserved until you reload the current metadata.</p>
<div className="mt-3 flex gap-2"><Button type="button" variant="outline" onClick={() => void reloadTable()}>Reload latest</Button><Button type="button" variant="ghost" onClick={() => setStaleBannerOpen(false)}>Keep editing</Button></div>
</div>
) : <div className="mt-4 flex items-center justify-between gap-3 rounded-md border border-amber-500/40 bg-amber-500/8 px-4 py-3 text-sm"><span>Reload the latest value before this description can be saved.</span><Button type="button" variant="outline" onClick={() => void reloadTable()}>Reload latest</Button></div>
) : null}
<div className="mt-5 grid gap-4">
<label className="grid gap-1.5 text-sm font-semibold">Physical table name<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={activeTable.name} readOnly /></label>
<label className="grid gap-1.5 text-sm font-semibold">Source comment<textarea className="min-h-24 rounded-md border border-input bg-muted/35 px-3 py-2 text-sm" value={activeTable.sourceComment ?? ""} readOnly /></label>
<label className="grid gap-1.5 text-sm font-semibold">Generated description<textarea className="min-h-32 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={generatedDescription} disabled={!canManage || busy !== null} onChange={(event) => setGeneratedDescription(event.target.value)} /></label>
<label className="grid gap-1.5 text-sm font-semibold">Description<textarea className="min-h-36 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={description} disabled={!canManage || busy !== null} onChange={(event) => setDescription(event.target.value)} /><span className="font-normal text-muted-foreground">Leave blank to keep the curated description empty.</span></label>
</div>
<div className="mt-5 flex justify-end gap-2 border-t border-border pt-4">
<Button type="button" variant="outline" disabled={busy !== null} onClick={leaveTable}>Cancel</Button>
<Button type="button" disabled={!canManage || !editorDirty || busy !== null || stale} onClick={() => void save()}><Save />{busy === "save" ? "Saving…" : "Save metadata"}</Button>
</div>
</div>
)}
</div>
</section>
);
}
return (
<section aria-label={`Tables for ${database.workspaceName}`} className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
{databaseHeader}
<div className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
{selectedIds.length > 0 ? (
<>
<span className="text-sm font-semibold">{selectedIds.length} selected</span>
{visibleSelected !== selectedIds.length ? <span className="text-xs text-muted-foreground">{selectedIds.length - visibleSelected} hidden by filter</span> : null}
<Button
type="button"
variant="outline"
disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)}
title={!bindingReady ? "Test the current database binding before synchronizing columns" : undefined}
onClick={() => void synchronize("columns", selectedIds)}
>
<RefreshCw className={busy === "sync" ? "animate-spin" : ""} />Synchronize columns
</Button>
<Button type="button" variant="ghost" onClick={() => { gridRef.current?.api.deselectAll(); setSelectedIds([]); }}><X />Clear</Button>
</>
) : (
<>
<span className="thot-label whitespace-nowrap">Catalog tables</span>
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search tables" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
<span
aria-live="polite"
className="whitespace-nowrap text-xs tabular-nums text-muted-foreground"
>
{search.trim() ? `${displayedCount} of ${data.length}` : displayedCount}
</span>
<Button type="button" variant="outline" disabled={isFetching || busy !== null} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
<Button type="button" disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)} title={!bindingReady ? "Test the current database binding before synchronizing tables" : undefined} onClick={() => void synchronize("tables")}><RefreshCw className={busy === "sync" ? "animate-spin" : ""} />Sync tables</Button>
</>
)}
</div>
{!bindingReady ? <div className="border-b border-border bg-muted/35 px-4 py-3 text-sm text-muted-foreground">Test the current database binding from Overview before synchronizing tables.</div> : null}
<div className="relative min-h-[280px] flex-1">
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
<AgGridReact<CatalogTable>
ref={gridRef}
rowData={data}
columnDefs={columns}
context={context}
loading={isLoading}
quickFilterText={search}
defaultColDef={{ sortable: true, filter: true, resizable: true }}
getRowId={({ data: table }) => table.id}
rowSelection={{ mode: "multiRow", selectAll: "filtered", enableClickSelection: false }}
onSelectionChanged={({ api }) => setSelectedIds(api.getSelectedRows().map((table) => table.id))}
onModelUpdated={({ api }) => setDisplayedCount(api.getDisplayedRowCount())}
rowHeight={44}
headerHeight={38}
animateRows={false}
overlayNoRowsTemplate="No catalog tables. Test the binding, then sync tables."
/>
</div>
</div>
</div>
</section>
);
}
@@ -0,0 +1,120 @@
import type {
CatalogDatabase,
CatalogSecretName,
DatabaseConfiguration,
} from "../../api/catalog-databases";
export type DatabaseFormMode = "add" | "view" | "edit" | "delete";
export type DatabaseScreen =
| { kind: "list" }
| { kind: "tables"; workspaceId: string }
| { kind: "relationships"; workspaceId: string }
| {
kind: DatabaseFormMode;
workspaceId: string;
workspaceLocked?: boolean;
};
export type DatabaseBusyAction = "save" | "test" | "delete" | "retry-secrets" | "reload" | null;
export type SecretDraft = Partial<Record<CatalogSecretName, string>>;
export interface DatabaseFormDraft extends DatabaseConfiguration {
secrets: SecretDraft;
}
export interface DatabaseNavigationState {
dirty: boolean;
busy: boolean;
}
export function draftFrom(row: CatalogDatabase): DatabaseFormDraft {
return {
workspaceId: row.workspaceId,
engine: "postgres",
databaseName: row.databaseName,
schema: row.schema,
binding: {
port: 5432,
restPath: "/health",
restAuth: "x-api-key",
sshPort: 22,
sshTargetPort: 5432,
...row.binding,
},
secrets: {},
};
}
export function configurationFromDraft(draft: DatabaseFormDraft): DatabaseConfiguration {
const binding = draft.binding.transport === "postgres_direct"
? {
transport: draft.binding.transport,
host: draft.binding.host,
port: draft.binding.port,
username: draft.binding.username,
tlsServername: draft.binding.tlsServername,
}
: draft.binding.transport === "rest_api"
? {
transport: draft.binding.transport,
baseUrl: draft.binding.baseUrl,
restPath: draft.binding.restPath,
restAuth: draft.binding.restAuth,
}
: {
transport: draft.binding.transport,
username: draft.binding.username,
tlsServername: draft.binding.tlsServername,
sshHost: draft.binding.sshHost,
sshPort: draft.binding.sshPort,
sshUsername: draft.binding.sshUsername,
sshTargetHost: draft.binding.sshTargetHost,
sshTargetPort: draft.binding.sshTargetPort,
};
return {
workspaceId: draft.workspaceId,
engine: draft.engine,
databaseName: draft.databaseName,
schema: draft.schema,
binding,
};
}
export function configurationFingerprint(draft: DatabaseFormDraft): string {
return JSON.stringify(configurationFromDraft(draft));
}
export function hasSecretChanges(draft: DatabaseFormDraft | null): boolean {
return Boolean(draft && Object.keys(secretReplacements(draft)).length);
}
export function secretReplacements(draft: DatabaseFormDraft): SecretDraft {
const allowed: CatalogSecretName[] = draft.binding.transport === "postgres_direct"
? ["password", "tlsCa"]
: draft.binding.transport === "rest_api"
? draft.binding.restAuth === "none" ? [] : ["apiKey"]
: [
"password",
"sshPrivateKey",
"sshPrivateKeyPassphrase",
"sshKnownHosts",
"tlsCa",
];
return Object.fromEntries(
Object.entries(draft.secrets).filter(([name, value]) => (
allowed.includes(name as CatalogSecretName) && Boolean(value)
)),
) as SecretDraft;
}
export function statusLabel(row: CatalogDatabase): string {
if (!row.workspaceAvailable) return "Workspace missing";
if (!row.configured) return "Not configured";
if (row.connectionStatus === "reachable") return "Reachable";
if (row.connectionStatus === "failed") return "Failed";
return "Untested";
}
+2 -1
View File
@@ -6,8 +6,9 @@ export const server = setupServer(
issuer: "mock", subject: "test-user", displayName: "Test user", roles: ["admin"],
permissions: [
"session.use", "session.read_all", "session.manage_all", "settings.manage",
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
], isAdmin: true, csrfToken: "c".repeat(43), session: null,
})),
http.get("/api/health/dwh", () => HttpResponse.json({ ok: true, detail: "ok" })),
http.get("/api/catalog/databases", () => HttpResponse.json([])),
);
+10 -2
View File
@@ -21,5 +21,13 @@ if [[ "${THOTH_ENABLE_EMBEDDING_GPU:-0}" == "1" ]]; then
compose_files+=(-f "$ROOT/deploy/compose.embedding-gpu.yaml")
fi
exec docker compose --env-file "$LOCAL_ENV_FILE" \
"${compose_files[@]}" up --build "$@"
compose=(docker compose --env-file "$LOCAL_ENV_FILE" "${compose_files[@]}")
# Migrations are an explicit one-shot operation, never hidden in backend startup. The local
# launcher runs that operation before bringing the foreground stack up so a fresh checkout is
# immediately usable while production operators can invoke the same service during rollout.
"${compose[@]}" build core
"${compose[@]}" up -d catalog-db
"${compose[@]}" run --rm catalog-migrate
exec "${compose[@]}" up --build "$@"
+1 -1
View File
@@ -64,7 +64,7 @@ for profile in local server; do
const fs = require("fs");
const [path, profile] = process.argv.slice(2);
const config = JSON.parse(fs.readFileSync(path, "utf8"));
const expected = "core,embedding,embedding-model-init,frontend,qdrant";
const expected = "catalog-db,core,embedding,embedding-model-init,frontend,qdrant";
if (Object.keys(config.services).sort().join(",") !== expected) {
throw new Error(profile + ": install stack must be exactly " + expected);
}
+22 -1
View File
@@ -6,6 +6,9 @@ cd "$(dirname "$0")/.."
test -f .env.example
test -f deploy/env/local.env.example
test -f deploy/env/server.env.example
test -f docker/catalog-db-init.sql
grep -q "pg_read_file('/run/secrets/catalog_runtime_password')" docker/catalog-db-init.sql
grep -q "CREATE ROLE thothii_catalog_runtime" docker/catalog-db-init.sql
rendered=$(mktemp)
trap 'rm -f "$rendered"' EXIT HUP INT TERM
@@ -18,7 +21,7 @@ const fs = require("fs");
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
const services = Object.keys(config.services).sort();
if (services.join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
if (services.join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
throw new Error(`unexpected service set: ${services.join(",")}`);
}
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
@@ -26,6 +29,7 @@ if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify
}
const expectedVolumes = [
"auth-state",
"catalog-data",
"embedding-models",
"pi-state",
"qdrant-data",
@@ -43,12 +47,19 @@ const frontend = config.services.frontend;
const qdrant = config.services.qdrant;
const embedding = config.services.embedding;
const modelInit = config.services["embedding-model-init"];
const catalog = config.services["catalog-db"];
if (!frontend.ports?.some((port) => port.host_ip === "127.0.0.1")) {
throw new Error("local frontend must publish a loopback port");
}
for (const service of [embedding, modelInit]) {
if ((service.ports || []).length !== 0) throw new Error("private semantic services must not publish host ports");
}
if ((catalog.ports || []).length !== 0) throw new Error("catalog database must not publish host ports");
if (!catalog.healthcheck) throw new Error("catalog database must define a healthcheck");
const catalogHealthcheck = JSON.stringify(catalog.healthcheck.test || []);
if (!catalogHealthcheck.includes("pg_isready") || !catalogHealthcheck.includes("thothii_catalog_runtime")) {
throw new Error("catalog database healthcheck must verify readiness and the runtime role");
}
if (!qdrant.ports?.some((port) => port.host_ip === "127.0.0.1" && Number(port.target) === 6333)) {
throw new Error("local Qdrant dashboard must publish only its loopback port");
}
@@ -65,6 +76,9 @@ if (embedding.image !== "ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279
if (modelInit.image !== "ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a") {
throw new Error("embedding-model-init image must be pinned by version and digest");
}
if (catalog.image !== "postgres:17.6-bookworm@sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3") {
throw new Error("catalog PostgreSQL image must be pinned by version and digest");
}
const env = core.environment || {};
for (const [key, value] of Object.entries({
THT_WORKSPACE_INSTALLATION_ID: "local",
@@ -74,6 +88,10 @@ for (const [key, value] of Object.entries({
THT_INTERNAL_EMBEDDING_URL: "http://embedding:11434",
THT_INTERNAL_EMBEDDING_MODEL: "qwen3-embedding:0.6b",
THT_INTERNAL_EMBEDDING_DIMENSIONS: "1024",
THT_CATALOG_DB_HOST: "catalog-db",
THT_CATALOG_DB_NAME: "thothii_catalog",
THT_CATALOG_RUNTIME_USER: "thothii_catalog_runtime",
THT_CATALOG_RUNTIME_PASSWORD_FILE: "/run/secrets/catalog_runtime_password",
})) {
if (env[key] !== value) throw new Error(`unexpected core ${key}: ${env[key]}`);
}
@@ -98,6 +116,9 @@ const depends = core.depends_on || {};
if (depends.qdrant?.condition !== "service_healthy") {
throw new Error("core must wait for qdrant health");
}
if (depends["catalog-db"]?.condition !== "service_healthy") {
throw new Error("core must wait for catalog database health");
}
if (depends["embedding-model-init"]?.condition !== "service_completed_successfully") {
throw new Error("core must wait for embedding-model-init success");
}
+13 -12
View File
@@ -25,14 +25,14 @@ const fs = require("fs");
const [configPath, profile] = process.argv.slice(2);
const config = JSON.parse(fs.readFileSync(configPath, "utf8"));
const services = Object.keys(config.services).sort();
if (services.join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
throw new Error("mandatory stack must include core, frontend, qdrant, embedding, and embedding-model-init");
if (services.join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
throw new Error("mandatory stack must include catalog-db, core, frontend, qdrant, embedding, and embedding-model-init");
}
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
throw new Error("forbidden application coupling");
}
if (!config.networks || !config.networks.thothii) throw new Error("base stack must define the thothii network");
for (const volume of ["qdrant-data", "embedding-models"]) {
for (const volume of ["catalog-data", "qdrant-data", "embedding-models"]) {
if (!config.volumes || !config.volumes[volume]) throw new Error(`missing required volume: ${volume}`);
}
if (profile === "local") {
@@ -114,8 +114,8 @@ const bundleSecrets = runtimeSecrets.filter(
if (bundleSecrets.length !== 1) {
throw new Error("core must receive exactly one canonical runtime secret bundle");
}
if (profile === "local" && runtimeSecrets.length !== 1) {
throw new Error("local core must receive only the canonical runtime secret bundle");
if (profile === "local" && runtimeSecrets.length !== 2) {
throw new Error("local core must receive only its runtime bundle and catalog password");
}
if (profile === "server") {
const targets = new Set(runtimeSecrets.map((secret) => secret.target));
@@ -171,14 +171,14 @@ const fs = require("fs");
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
const services = Object.keys(config.services).sort();
if (services.join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
throw new Error("mandatory stack must include core, frontend, qdrant, embedding, and embedding-model-init");
if (services.join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
throw new Error("mandatory stack must include catalog-db, core, frontend, qdrant, embedding, and embedding-model-init");
}
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
throw new Error("forbidden application coupling");
}
if (!config.networks || !config.networks.thothii) throw new Error("base stack must define the thothii network");
for (const volume of ["settings", "pi-state", "workspace-registry", "sessions", "qdrant-data", "embedding-models"]) {
for (const volume of ["settings", "pi-state", "workspace-registry", "sessions", "catalog-data", "qdrant-data", "embedding-models"]) {
if (!config.volumes || !config.volumes[volume]) throw new Error(`missing required volume: ${volume}`);
}
if (!Object.hasOwn(config.services.core.environment || {}, "THT_LLM_URL")) {
@@ -222,10 +222,11 @@ if (config.services.core.environment?.THT_SECRETS_FILE !== "/run/secrets/thothii
throw new Error("core must read the canonical runtime secret bundle from /run/secrets");
}
const runtimeSecrets = config.services.core.secrets || [];
if (runtimeSecrets.length !== 1
|| runtimeSecrets[0].source !== "thothii_secrets"
|| runtimeSecrets[0].target !== "thothii.secrets") {
throw new Error("core must receive exactly the canonical runtime secret bundle");
const hasTarget = (name) => runtimeSecrets.some((secret) => secret.target === name || secret.target?.endsWith(`/${name}`));
if (runtimeSecrets.length !== 2
|| !hasTarget("thothii.secrets")
|| !hasTarget("catalog_runtime_password")) {
throw new Error("core must receive only the canonical runtime bundle and catalog password");
}
if ((config.services.frontend.secrets || []).length !== 0) {
throw new Error("frontend must not receive runtime secrets");
+4 -4
View File
@@ -275,8 +275,8 @@ overrides:
)
$render = Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("config", "--services")) -Label "render Windows Compose from spaced path"
$services = @($render.StdOut -split "`r?`n" | Where-Object { $_ } | Sort-Object)
if (($services -join ",") -ne "core,embedding,embedding-model-init,frontend,qdrant") {
throw "rendered Windows stack must contain the canonical five services"
if (($services -join ",") -ne "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
throw "rendered Windows stack must contain the canonical six services"
}
Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("config", "--quiet")) -Label "validate Windows Compose from spaced path" | Out-Null
@@ -287,8 +287,8 @@ overrides:
Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("up", "--detach", "--wait", "--wait-timeout", "180")) -Label "start canonical Windows stack" -TimeoutSeconds 300 | Out-Null
$running = Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("ps", "--status", "running", "--services")) -Label "inspect running Windows services"
$runningServices = @($running.StdOut -split "`r?`n" | Where-Object { $_ } | Sort-Object)
if (($runningServices -join ",") -ne "core,embedding,frontend,qdrant") {
throw "bounded Windows startup did not leave the four long-running services ready"
if (($runningServices -join ",") -ne "catalog-db,core,embedding,frontend,qdrant") {
throw "bounded Windows startup did not leave the five long-running services ready"
}
Invoke-BoundedNative -FilePath $tht -Arguments @("--installation", $installation, "status") -Label "invoke installation-aware Windows tht in spaced path" | Out-Null
}
+3 -3
View File
@@ -1945,7 +1945,7 @@ verify_local_installation_example() {
const fs = require("fs");
const [path, authConfigRoot] = process.argv.slice(2);
const config = JSON.parse(fs.readFileSync(path, "utf8"));
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
if (Object.keys(config.services).sort().join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
throw new Error("local installation example must render the internal semantic stack");
}
const authMount = (config.services.core.volumes || []).find(
@@ -2065,7 +2065,7 @@ verify_server_installation_example() {
const fs = require("fs");
const [path, authConfigRoot] = process.argv.slice(2);
const config = JSON.parse(fs.readFileSync(path, "utf8"));
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
if (Object.keys(config.services).sort().join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
throw new Error("server installation example must render the internal semantic stack");
}
const core = config.services.core;
@@ -2216,7 +2216,7 @@ verify_compose_fixtures() {
const fs = require("fs");
const [path, profile, authConfigRoot] = process.argv.slice(2);
const config = JSON.parse(fs.readFileSync(path, "utf8"));
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
if (Object.keys(config.services).sort().join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
throw new Error(profile + ": mandatory stack must include the internal semantic services");
}
const core = config.services.core;