feat: implement metadata catalog database management
This commit is contained in:
@@ -1,5 +1,19 @@
|
||||
# AGENTS.md
|
||||
|
||||
## Agent skills
|
||||
|
||||
### Issue tracker
|
||||
|
||||
Issues for this repository live in the self-hosted Gitea repository at `https://git.tylconsulting.it/mptyl/ThothII`; use its web UI or authenticated Gitea API. See `docs/agents/issue-tracker.md`.
|
||||
|
||||
### Triage labels
|
||||
|
||||
Use the canonical labels `needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, and `wontfix`. See `docs/agents/triage-labels.md`.
|
||||
|
||||
### Domain docs
|
||||
|
||||
This is a single-context repository with root `CONTEXT.md` and `docs/adr/`. See `docs/agents/domain.md`.
|
||||
|
||||
This file provides guidance to Codex (Codex.ai/code) when working with code in this repository.
|
||||
|
||||
## Start here
|
||||
|
||||
+73
-11
@@ -246,10 +246,23 @@ precedente o di un altro workspace.
|
||||
|
||||
## Catalogo dei metadati
|
||||
|
||||
**Workspace Database** — Il database associato in modo uno-a-uno a un workspace,
|
||||
considerato nella sua interezza fisica: tutte le tabelle, le colonne e le relazioni
|
||||
disponibili. La sua struttura fisica viene acquisita interrogando il database; il
|
||||
Metadata Catalog non crea né possiede l'identità del workspace.
|
||||
**Workspace Database** — Il database che appartiene a un solo workspace e non può essere
|
||||
condiviso con altri workspace; un workspace può averne al massimo uno. È considerato nella
|
||||
coppia composta dal database PostgreSQL e da un solo schema: tutte le tabelle, le colonne e
|
||||
le relazioni catalogate appartengono a quello schema. Il Metadata Catalog conserva
|
||||
l'associazione, ma non crea né possiede l'identità del workspace.
|
||||
|
||||
**Database Binding** — La configurazione specifica di un'installazione che seleziona un
|
||||
trasporto e fornisce i riferimenti necessari a raggiungere un Workspace Database. Non è una
|
||||
seconda identità del database e non viene condivisa automaticamente fra installazioni.
|
||||
|
||||
**Thoth REST Connector** — Il trasporto REST tipizzato con cui ThothII interroga ed
|
||||
introspeziona un Workspace Database attraverso il contratto RPC DWH supportato. Non è un
|
||||
client configurabile per API REST arbitrarie.
|
||||
|
||||
**Orphaned Workspace Database** — Un Workspace Database il cui workspace non è più presente
|
||||
nel catalogo autorevole. Rimane conservato per il recupero amministrativo, ma non può essere
|
||||
usato dal workflow finché non viene riassegnato a un workspace esistente.
|
||||
|
||||
**Metadata Catalog** — Il contesto amministrativo che raccoglie e cura i metadati di un
|
||||
Workspace Database. Non definisce quali elementi partecipano al workflow SQL.
|
||||
@@ -257,12 +270,61 @@ Workspace Database. Non definisce quali elementi partecipano al workflow SQL.
|
||||
**Database Profile** — L'insieme curato di scope, descrizioni e metadati semantici
|
||||
associato a un Workspace Database.
|
||||
|
||||
**Physical Schema Snapshot** — L'inventario della struttura fisica osservata in un
|
||||
Workspace Database durante una specifica introspezione. Non è un progetto dello schema
|
||||
né un'autorizzazione a modificarne la struttura.
|
||||
**Physical Table** — Una tabella osservata nello schema esterno di un Workspace Database.
|
||||
La sua identità e il suo nome appartengono al database esterno, non al Metadata Catalog.
|
||||
|
||||
**AI Proposal** — Un contenuto generato con l'ausilio dell'AI che non è ancora stato
|
||||
approvato come contenuto canonico.
|
||||
**Catalog Table** — La rappresentazione persistita di una Physical Table nel Metadata Catalog.
|
||||
La sua appartenenza e identità fisica derivano esclusivamente dall'introspezione; soltanto i suoi
|
||||
Catalog Metadata possono essere curati amministrativamente.
|
||||
_Avoid_: SqlTable, managed table
|
||||
|
||||
**Publication** — Una versione approvata e immutabile dei contenuti del Metadata
|
||||
Catalog resa disponibile ai suoi consumatori.
|
||||
**Physical Column** — Una colonna osservata in una Physical Table, inclusi nome, posizione,
|
||||
tipo e appartenenza a chiavi dichiarate. La sua identità e i suoi fatti strutturali appartengono
|
||||
al database esterno.
|
||||
|
||||
**Catalog Column** — La rappresentazione persistita di una Physical Column nel Metadata Catalog.
|
||||
I fatti osservati sono governati dalla sincronizzazione; Description e Generated Description
|
||||
sono metadati amministrativi modificabili.
|
||||
_Avoid_: SqlColumn, managed column
|
||||
|
||||
**Physical Relationship** — Un vincolo foreign key dichiarato nel database esterno. La sua
|
||||
identità comprende il vincolo e la sequenza ordinata delle coppie di colonne che lo compongono.
|
||||
|
||||
**Catalog Relationship** — La rappresentazione persistita di una Physical Relationship nel
|
||||
Metadata Catalog. È governata esclusivamente dall'introspezione e non è creata o modificata
|
||||
manualmente.
|
||||
_Avoid_: denormalized FK, relationship string
|
||||
|
||||
**Logical Relationship** — Una relazione semantica curata o inferita che non corrisponde
|
||||
necessariamente a un vincolo fisico. Ha ownership e lifecycle distinti da Catalog Relationship.
|
||||
|
||||
**Description** — Il testo curato e consolidato che descrive una Catalog Table o Catalog Column
|
||||
per gli usi downstream.
|
||||
|
||||
**Generated Description** — Una proposta modificabile sottoposta a revisione umana prima di
|
||||
essere consolidata come Description. Rimane distinta dal commento osservato nel database.
|
||||
_Avoid_: generated comment, source comment
|
||||
|
||||
**Table Synchronization** — La riconciliazione esplicita che rende le Catalog Table di un
|
||||
Workspace Database uguali alle Physical Table osservate: crea quelle nuove, aggiorna i metadati
|
||||
di origine ed elimina definitivamente quelle assenti. Non modifica mai il database esterno.
|
||||
_Avoid_: table import
|
||||
|
||||
**Schema Synchronization** — La riconciliazione esplicita e autorevole di tabelle, colonne e
|
||||
Catalog Relationship di un Workspace Database. Può operare su uno scope specifico oppure su
|
||||
un unico snapshot completo tramite Synchronize All.
|
||||
|
||||
**Catalog Sync Run** — L'esecuzione durevole in background di una Schema Synchronization, con
|
||||
scope, stato, avanzamento e log propri. Al massimo un run per Workspace Database può essere attivo.
|
||||
|
||||
**Catalog Freshness** — La corrispondenza fra uno scope sincronizzato e la versione corrente
|
||||
della Database Binding. Uno scope rimane consultabile ma è stale finché non viene sincronizzato
|
||||
con la binding corrente.
|
||||
|
||||
**Catalog Metadata** — I campi mutabili che descrivono database, tabelle, colonne e relazioni,
|
||||
distinti dai fatti strutturali governati dalla sincronizzazione. Possono essere popolati dall'AI,
|
||||
da un'importazione o da una modifica amministrativa senza cambiare il database esterno.
|
||||
|
||||
**Introspection Capability** — Una categoria di struttura fisica che una Database Binding
|
||||
può osservare, come tabelle, colonne, relazioni, indici o enum. Una capability non disponibile
|
||||
è distinta da una capability osservata che non ha restituito elementi.
|
||||
|
||||
+49
-3
@@ -1,6 +1,6 @@
|
||||
# ThothII — Project State
|
||||
|
||||
Last updated: 2026-08-26.
|
||||
Last updated: 2026-08-27.
|
||||
|
||||
This file is the short operational snapshot. Stable commands and the architecture mental model
|
||||
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
|
||||
@@ -16,8 +16,9 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
|
||||
```
|
||||
|
||||
The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence.
|
||||
The backend is a process/RPC/SSE bridge without a database of its own. The frontend renders the
|
||||
review gates and keeps the live transcript in memory. See
|
||||
The backend remains a process/RPC/SSE bridge for sessions and now also owns an isolated PostgreSQL
|
||||
metadata catalog for administrative database configuration. The frontend renders the review gates
|
||||
and keeps the live transcript in memory. See
|
||||
`docs/architecture/components.md` for the detailed component and data-flow map.
|
||||
|
||||
## Evidence restructuring — accepted
|
||||
@@ -62,6 +63,51 @@ Workspace descriptors use schema v3. For PSD, workspace content and runtime root
|
||||
separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside
|
||||
Git and are supplied only through installation-local protected files.
|
||||
|
||||
## Database management
|
||||
|
||||
The database, table, and authoritative physical-schema catalog slices are implemented. Database
|
||||
management opens a responsive AG Grid master-detail surface, lists every YAML workspace, creates
|
||||
at most one PostgreSQL database configuration per workspace, edits direct PostgreSQL, REST API, or
|
||||
SSH-tunnel installation bindings, replaces write-only encrypted secrets, and tests supported
|
||||
connector bindings.
|
||||
|
||||
Configured databases use pure hierarchical navigation through `Overview`, `Tables`, and
|
||||
`Relationships`; a selected table has `Overview` and `Columns`. Physical membership, source
|
||||
comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are
|
||||
immutable projections of the external schema. Curated and generated descriptions are editable;
|
||||
generated descriptions start null and AI generation/consolidation is deferred.
|
||||
|
||||
Schema refresh is one durable asynchronous engine with database-table, database-column,
|
||||
selected-table-column, relationship, and full-database actions. Database-level menus expose the
|
||||
table, all-column, relationship, and full scopes separately; selecting tables exposes column
|
||||
synchronization for that subset. Runs have one-active-job-per-database exclusion, leases and
|
||||
restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before
|
||||
apply, retained history, and a live SSE log with polling fallback. Null metadata renders blank
|
||||
rather than as a placeholder.
|
||||
|
||||
Direct PostgreSQL and strict known-host-verified OpenSSH use `pg_catalog`. REST bindings use the
|
||||
typed full-snapshot `POST /rpc/schema_snapshot` contract when available. Servers such as the
|
||||
current PSD endpoint that exposes only `POST /rpc/run_query` use one catalog-owned read-only query
|
||||
to return the exact same strict v1 snapshot in a single round trip. Both paths remain fail-closed:
|
||||
an absent capability, query error, partial result, or invalid snapshot applies no catalog changes.
|
||||
SSH is not yet enabled for NL→SQL session runtime.
|
||||
|
||||
The catalog runs in the internal `catalog-db` PostgreSQL service. Kysely migrations are an explicit
|
||||
one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before local startup. Runtime
|
||||
sessions still consume the existing workspace configuration in this slice: database-management
|
||||
records do not yet change the NL→SQL handoff. The accepted design is recorded in
|
||||
`docs/plans/2026-08-26-metadata-catalog-from-thothai.md`, the snapshot contract under
|
||||
`docs/contracts/`, and ADRs 0001–0007.
|
||||
|
||||
Semantic aliases, value descriptions, synonyms, concepts, AI metadata generation/consolidation,
|
||||
and logical relationships remain deferred to their dedicated slices.
|
||||
|
||||
Integration of the completed metadata catalog with core schema-linking is explicitly deferred
|
||||
until the database, table, column, relationship, and synchronization slices are complete. At that
|
||||
point the next required design gate is to compare the catalog snapshot with the current DWH
|
||||
preprocessing/schema-linking contracts and plan the cutover; this follow-up must not be treated as
|
||||
optional cleanup or silently omitted.
|
||||
|
||||
## Active deployment work and manual gates
|
||||
|
||||
### PSD server deployment program
|
||||
|
||||
@@ -1,15 +1,16 @@
|
||||
# ThothII
|
||||
|
||||
ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/`tht`
|
||||
core. The portable deployment runs exactly two application services; data services remain
|
||||
external in this profile, except for the mandatory internal semantic services bundled in Compose.
|
||||
core. The portable deployment runs two application services plus the installation-local metadata
|
||||
catalog; DWH and LLM services remain external. Semantic services are bundled in Compose.
|
||||
|
||||
Authentication is configured through the single host CLI tht: see the [local authentication guide](docs/install/authentication-local.md),
|
||||
[generic OIDC guide](docs/install/authentication-oidc.md), and [manual acceptance matrix](docs/testing/authentication-manual-acceptance.md).
|
||||
|
||||
## Docker Compose: local startup
|
||||
|
||||
Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external,
|
||||
Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`,
|
||||
`catalog-db`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external,
|
||||
configurable endpoints—even when they are co-located with ThothII.
|
||||
|
||||
From a fresh clone, run these commands from the repository root:
|
||||
@@ -17,17 +18,28 @@ From a fresh clone, run these commands from the repository root:
|
||||
```sh
|
||||
cp deploy/env/local.env.example deploy/env/local.env
|
||||
# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
|
||||
docker compose --env-file deploy/env/local.env \
|
||||
-f compose.yaml -f deploy/compose.local.yaml up --build -d
|
||||
./scripts/run-stack.sh
|
||||
```
|
||||
|
||||
`./scripts/run-stack.sh` runs this same base+local command in the foreground. The core image
|
||||
contains its Pi runtime; no host `pi` executable is used. For a server installation:
|
||||
The launcher builds the core, starts `catalog-db`, runs the explicit one-shot Kysely migrations,
|
||||
then runs the base+local stack in the foreground. Migrations never run implicitly in backend
|
||||
startup. The core image contains its Pi runtime; no host `pi` executable is used. For a server
|
||||
installation, build the image, start the catalog, and run the same migration service before the
|
||||
application rollout:
|
||||
|
||||
```sh
|
||||
cp deploy/env/server.env.example deploy/env/server.env
|
||||
# Edit all absolute storage, Pi/secret/session files, and endpoint paths.
|
||||
sudo scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001
|
||||
docker compose --env-file deploy/env/server.env \
|
||||
-f compose.yaml -f deploy/compose.server.yaml \
|
||||
-f deploy/compose.session-server.yaml.example build core
|
||||
docker compose --env-file deploy/env/server.env \
|
||||
-f compose.yaml -f deploy/compose.server.yaml \
|
||||
-f deploy/compose.session-server.yaml.example up -d catalog-db
|
||||
docker compose --env-file deploy/env/server.env \
|
||||
-f compose.yaml -f deploy/compose.server.yaml \
|
||||
-f deploy/compose.session-server.yaml.example run --rm catalog-migrate
|
||||
docker compose --env-file deploy/env/server.env \
|
||||
-f compose.yaml -f deploy/compose.server.yaml \
|
||||
-f deploy/compose.session-server.yaml.example up --build -d
|
||||
@@ -101,10 +113,12 @@ schema, Evidence, and Memory records share that collection and stay separated by
|
||||
`kind`.
|
||||
<!-- workspace-descriptor-contract:end -->
|
||||
|
||||
Connector `ssh_tunnel` bindings are diagnostic-only in this release: their bounded probe always
|
||||
cleans up the loopback forward and returns `workspace_not_activatable`; session creation is rejected
|
||||
before persistence. Git registry access over SSH is unaffected. Use direct or REST connector
|
||||
transport for runtime sessions.
|
||||
For NL→SQL runtime sessions, connector `ssh_tunnel` bindings remain diagnostic-only: their bounded
|
||||
probe cleans up the loopback forward and returns `workspace_not_activatable`; session creation is
|
||||
rejected before persistence. Database management is a separate boundary and supports a strict
|
||||
OpenSSH tunnel for **Test connection** and **Sync tables**, using a private key, optional passphrase,
|
||||
mandatory `known_hosts`, and optional PostgreSQL TLS CA/server name. Git registry access over SSH is
|
||||
unaffected. Use direct or REST connector transport for runtime sessions.
|
||||
|
||||
`docker-compose.dev.yml` is deliberately local: both published ports bind to `127.0.0.1`,
|
||||
`THT_SESSION_STORAGE=local`, and `THT_HOME=/data/local-home`. Do not set
|
||||
|
||||
Generated
+2049
File diff suppressed because it is too large
Load Diff
@@ -6,6 +6,7 @@
|
||||
"dev": "tsx watch src/server.ts",
|
||||
"prebuild": "node scripts/clean-dist.mjs",
|
||||
"build": "tsc -p tsconfig.json",
|
||||
"catalog:migrate": "node dist/catalog/migrate.js",
|
||||
"test": "vitest run",
|
||||
"start": "node dist/server.js",
|
||||
"test:schema-v3-verifier": "python3 -I -B scripts/test_revision_state_policy.py && node --test scripts/verify-workspace-descriptor-files.test.mjs scripts/revision-state-policy.test.mjs"
|
||||
@@ -16,12 +17,14 @@
|
||||
"@fastify/rate-limit": "11.2.0",
|
||||
"@types/pg": "^8.20.3",
|
||||
"fastify": "^5.0.0",
|
||||
"kysely": "^0.29.5",
|
||||
"openid-client": "6.8.5",
|
||||
"pg": "^8.22.0",
|
||||
"yaml": "^2.9.0",
|
||||
"zod": "^4.4.3"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@testcontainers/postgresql": "^12.1.0",
|
||||
"@types/node": "24.13.3",
|
||||
"tsx": "^4.19.0",
|
||||
"typescript": "^5.6.0",
|
||||
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/usr/bin/env node
|
||||
import { readFileSync } from "node:fs";
|
||||
|
||||
const path = process.env.THT_SSH_PASSPHRASE_FILE;
|
||||
if (!path) process.exit(1);
|
||||
process.stdout.write(readFileSync(path));
|
||||
@@ -37,6 +37,17 @@ import { supportsSessionRuntime } from "./workspaces/bindings.js";
|
||||
import { resolveRuntimeBindingsWithWorkspaceSecrets } from "./workspaces/secret-requirements.js";
|
||||
import type { WorkspaceDescriptor } from "./workspaces/schema.js";
|
||||
import { WorkspaceSecretStore } from "./workspaces/secret-store.js";
|
||||
import { createCatalogRepository } from "./catalog/repository.js";
|
||||
import type { CatalogRepository } from "./catalog/types.js";
|
||||
import { CatalogService } from "./catalog/service.js";
|
||||
import { catalogDatabaseRoutes } from "./routes/catalog-databases.js";
|
||||
import { CatalogOperationCoordinator } from "./catalog/operation-coordinator.js";
|
||||
import { ConcreteCatalogPostgresAccess, type CatalogPostgresAccess } from "./catalog/postgres-access.js";
|
||||
import { CatalogTableService } from "./catalog/table-service.js";
|
||||
import { catalogTableRoutes } from "./routes/catalog-tables.js";
|
||||
import { ConcreteCatalogSchemaIntrospector, type CatalogSchemaIntrospector } from "./catalog/schema-introspector.js";
|
||||
import { CatalogSyncWorker } from "./catalog/sync-worker.js";
|
||||
import { catalogSchemaRoutes } from "./routes/catalog-schema.js";
|
||||
|
||||
export interface BuildAppDeps {
|
||||
thtRunner?: ThtRunner;
|
||||
@@ -49,6 +60,13 @@ export interface BuildAppDeps {
|
||||
workspaceRegistry?: WorkspaceRegistry;
|
||||
workspaceDiagnoser?: WorkspaceDiagnoser;
|
||||
workspaceSecretStore?: WorkspaceSecretStore;
|
||||
catalogRepository?: CatalogRepository;
|
||||
catalogService?: CatalogService;
|
||||
catalogPostgresAccess?: CatalogPostgresAccess;
|
||||
catalogTableService?: CatalogTableService;
|
||||
catalogSchemaIntrospector?: CatalogSchemaIntrospector;
|
||||
catalogSyncWorker?: CatalogSyncWorker;
|
||||
catalogOperationCoordinator?: CatalogOperationCoordinator;
|
||||
workspaceRuntimeSupport?: (workspace: WorkspaceDescriptor) => boolean;
|
||||
maintenanceBarrier?: MaintenanceBarrier;
|
||||
piManagement?: PiManagementService;
|
||||
@@ -121,6 +139,37 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
|
||||
const mgr = deps?.mgr ?? new PiProcessManager(config, deps?.spawnFn ? { spawnFn: deps.spawnFn } : undefined);
|
||||
const hub = deps?.hub ?? new SseHub();
|
||||
const workspaceRegistry = deps?.workspaceRegistry ?? new WorkspaceRegistry(config.workspaceRegistry);
|
||||
const catalogRepository = deps?.catalogRepository ?? createCatalogRepository(config.catalogDatabase);
|
||||
const catalogOperationCoordinator = deps?.catalogOperationCoordinator ?? new CatalogOperationCoordinator();
|
||||
const catalogPostgresAccess = deps?.catalogPostgresAccess ?? new ConcreteCatalogPostgresAccess(
|
||||
workspaceSecretStore,
|
||||
{ connectTimeoutMs: config.workspaceDiagnosticTimeoutMs },
|
||||
);
|
||||
const catalogService = deps?.catalogService ?? new CatalogService(
|
||||
catalogRepository,
|
||||
workspaceRegistry,
|
||||
workspaceSecretStore,
|
||||
config.workspaceRegistry.secretRoots,
|
||||
config.workspaceDiagnosticTimeoutMs,
|
||||
catalogPostgresAccess,
|
||||
catalogOperationCoordinator,
|
||||
);
|
||||
const catalogTableService = deps?.catalogTableService ?? new CatalogTableService(catalogRepository);
|
||||
const catalogSchemaIntrospector = deps?.catalogSchemaIntrospector ?? new ConcreteCatalogSchemaIntrospector(
|
||||
catalogPostgresAccess,
|
||||
workspaceSecretStore,
|
||||
);
|
||||
const catalogSyncWorker = deps?.catalogSyncWorker ?? new CatalogSyncWorker(
|
||||
catalogRepository,
|
||||
catalogSchemaIntrospector,
|
||||
catalogOperationCoordinator,
|
||||
config.catalogSyncTimeoutMs,
|
||||
);
|
||||
app.addHook("onReady", async () => { await catalogSyncWorker.initialize(); });
|
||||
if (!deps?.catalogRepository && catalogRepository.close) {
|
||||
app.addHook("onClose", async () => { await catalogRepository.close?.(); });
|
||||
}
|
||||
app.addHook("onClose", async () => { await catalogSyncWorker.stop(); });
|
||||
const workspaceDiagnoser = deps?.workspaceDiagnoser
|
||||
?? createProductionWorkspaceDiagnoser(config.workspaceDiagnosticTimeoutMs, undefined, {
|
||||
internalQdrantUrl: config.internalQdrantUrl,
|
||||
@@ -334,6 +383,9 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
|
||||
authDiagnoser,
|
||||
secretStore: workspaceSecretStore,
|
||||
});
|
||||
catalogDatabaseRoutes(app, { repository: catalogRepository, service: catalogService, operations: catalogOperationCoordinator });
|
||||
catalogTableRoutes(app, { repository: catalogRepository, service: catalogTableService });
|
||||
catalogSchemaRoutes(app, { repository: catalogRepository, worker: catalogSyncWorker });
|
||||
settingsRoutes(app, { cfg: config, listModels, getSettings });
|
||||
piManagementRoutes(app, { service: piManagement });
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ const MAX_MAPPED_GROUPS = 128;
|
||||
const ROLES = ["user", "admin"] as const;
|
||||
export const PERMISSION_CATALOG: readonly Permission[] = [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
];
|
||||
|
||||
const invalid = (): Error => new Error("authentication configuration is invalid");
|
||||
|
||||
@@ -33,7 +33,7 @@ const EMPTY_HKDF_SALT = Buffer.alloc(0);
|
||||
const ROLES = ["user", "admin"] as const;
|
||||
const PERMISSIONS = [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
] as const satisfies readonly Permission[];
|
||||
|
||||
const invalid = (): Error => new Error("auth_session_store_invalid");
|
||||
|
||||
@@ -5,7 +5,7 @@ export type Role = "user" | "admin";
|
||||
export type Permission =
|
||||
| "session.use" | "session.read_all" | "session.manage_all"
|
||||
| "settings.manage" | "workspace.manage" | "workspace.secrets.manage"
|
||||
| "pi.manage" | "auth.diagnostics.read";
|
||||
| "database.manage" | "pi.manage" | "auth.diagnostics.read";
|
||||
|
||||
export interface AuthenticationSessionConfig {
|
||||
regularTtlSeconds: number;
|
||||
|
||||
@@ -0,0 +1,692 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import {
|
||||
CatalogConflictError,
|
||||
CatalogConnectorError,
|
||||
type CatalogColumn,
|
||||
type CatalogRelationship,
|
||||
type CatalogSchemaDiff,
|
||||
type CatalogSyncCounts,
|
||||
type CatalogSyncEvent,
|
||||
type CatalogSyncRun,
|
||||
type CatalogSyncRunUpdate,
|
||||
type CatalogSyncScope,
|
||||
type CatalogTable,
|
||||
type CatalogRepository,
|
||||
type DatabaseConfigurationInput,
|
||||
type DatabaseTestResult,
|
||||
type ObservedCatalogTable,
|
||||
type ObservedSchemaSnapshot,
|
||||
type TableSyncRepositoryResult,
|
||||
type WorkspaceDatabase,
|
||||
} from "./types.js";
|
||||
|
||||
function clone(value: WorkspaceDatabase): WorkspaceDatabase {
|
||||
return structuredClone(value);
|
||||
}
|
||||
|
||||
/** Deterministic repository used by route tests and local contract consumers. */
|
||||
export class MemoryCatalogRepository implements CatalogRepository {
|
||||
private readonly records = new Map<string, WorkspaceDatabase>();
|
||||
private readonly tables = new Map<string, CatalogTable>();
|
||||
private readonly columns = new Map<string, CatalogColumn>();
|
||||
private readonly relationships = new Map<string, CatalogRelationship>();
|
||||
private readonly syncRuns = new Map<string, CatalogSyncRun>();
|
||||
private readonly syncEvents = new Map<string, CatalogSyncEvent[]>();
|
||||
|
||||
async list(): Promise<WorkspaceDatabase[]> {
|
||||
return [...this.records.values()].sort((a, b) => a.workspaceId.localeCompare(b.workspaceId)).map(clone);
|
||||
}
|
||||
async get(id: string): Promise<WorkspaceDatabase | undefined> {
|
||||
const value = this.records.get(id);
|
||||
return value ? clone(value) : undefined;
|
||||
}
|
||||
async getByWorkspace(workspaceId: string): Promise<WorkspaceDatabase | undefined> {
|
||||
const value = [...this.records.values()].find((record) => record.workspaceId === workspaceId);
|
||||
return value ? clone(value) : undefined;
|
||||
}
|
||||
async create(input: DatabaseConfigurationInput): Promise<WorkspaceDatabase> {
|
||||
if ([...this.records.values()].some((record) => record.workspaceId === input.workspaceId)) {
|
||||
throw new CatalogConflictError("Workspace database already exists");
|
||||
}
|
||||
const now = new Date().toISOString();
|
||||
const record: WorkspaceDatabase = {
|
||||
id: randomUUID(),
|
||||
...structuredClone(input),
|
||||
version: 1,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
connectionStatus: "untested",
|
||||
};
|
||||
this.records.set(record.id, record);
|
||||
return clone(record);
|
||||
}
|
||||
async update(id: string, expectedVersion: number, input: DatabaseConfigurationInput): Promise<WorkspaceDatabase | undefined> {
|
||||
const current = this.records.get(id);
|
||||
if (!current || current.version !== expectedVersion) return undefined;
|
||||
if ([...this.records.values()].some((record) => record.id !== id && record.workspaceId === input.workspaceId)) {
|
||||
throw new CatalogConflictError("Workspace database already exists");
|
||||
}
|
||||
const updated: WorkspaceDatabase = {
|
||||
...current,
|
||||
...structuredClone(input),
|
||||
version: current.version + 1,
|
||||
updatedAt: new Date().toISOString(),
|
||||
connectionStatus: "untested",
|
||||
testedVersion: undefined,
|
||||
lastTestedAt: undefined,
|
||||
lastErrorCode: undefined,
|
||||
lastErrorMessage: undefined,
|
||||
schemaSyncedVersion: undefined,
|
||||
schemaSyncedAt: undefined,
|
||||
};
|
||||
this.records.set(id, updated);
|
||||
return clone(updated);
|
||||
}
|
||||
async recordTest(id: string, expectedVersion: number, result: DatabaseTestResult): Promise<WorkspaceDatabase | undefined> {
|
||||
const current = this.records.get(id);
|
||||
if (!current || current.version !== expectedVersion) return undefined;
|
||||
const updated = {
|
||||
...current,
|
||||
connectionStatus: result.connectionStatus,
|
||||
testedVersion: result.testedVersion,
|
||||
lastTestedAt: result.lastTestedAt,
|
||||
lastErrorCode: result.errorCode,
|
||||
lastErrorMessage: result.errorMessage,
|
||||
};
|
||||
this.records.set(id, updated);
|
||||
return clone(updated);
|
||||
}
|
||||
async touch(id: string, expectedVersion: number): Promise<WorkspaceDatabase | undefined> {
|
||||
const current = this.records.get(id);
|
||||
if (!current || current.version !== expectedVersion) return undefined;
|
||||
return await this.update(id, expectedVersion, {
|
||||
workspaceId: current.workspaceId,
|
||||
engine: current.engine,
|
||||
databaseName: current.databaseName,
|
||||
schema: current.schema,
|
||||
binding: current.binding,
|
||||
});
|
||||
}
|
||||
async delete(id: string, expectedVersion: number): Promise<boolean> {
|
||||
const current = this.records.get(id);
|
||||
if (!current || current.version !== expectedVersion) return false;
|
||||
for (const [tableId, table] of this.tables) {
|
||||
if (table.databaseId === id) {
|
||||
this.tables.delete(tableId);
|
||||
for (const [columnId, column] of this.columns) if (column.tableId === tableId) this.columns.delete(columnId);
|
||||
}
|
||||
}
|
||||
for (const [relationshipId, relationship] of this.relationships) {
|
||||
if (relationship.databaseId === id) this.relationships.delete(relationshipId);
|
||||
}
|
||||
return this.records.delete(id);
|
||||
}
|
||||
async listTables(databaseId: string): Promise<CatalogTable[]> {
|
||||
return [...this.tables.values()]
|
||||
.filter((table) => table.databaseId === databaseId)
|
||||
.sort((a, b) => a.name.localeCompare(b.name))
|
||||
.map((table) => structuredClone(table));
|
||||
}
|
||||
async getTable(databaseId: string, tableId: string): Promise<CatalogTable | undefined> {
|
||||
const table = this.tables.get(tableId);
|
||||
return table?.databaseId === databaseId ? structuredClone(table) : undefined;
|
||||
}
|
||||
async updateTableDescription(
|
||||
databaseId: string,
|
||||
tableId: string,
|
||||
expectedVersion: number,
|
||||
description: string | null,
|
||||
): Promise<CatalogTable | undefined> {
|
||||
const current = this.tables.get(tableId);
|
||||
if (!current || current.databaseId !== databaseId || current.version !== expectedVersion) return undefined;
|
||||
const updated = {
|
||||
...current,
|
||||
description,
|
||||
version: current.version + 1,
|
||||
updatedAt: new Date().toISOString(),
|
||||
};
|
||||
this.tables.set(tableId, updated);
|
||||
return structuredClone(updated);
|
||||
}
|
||||
async updateTableMetadata(
|
||||
databaseId: string,
|
||||
tableId: string,
|
||||
expectedVersion: number,
|
||||
description: string | null,
|
||||
generatedDescription: string | null,
|
||||
): Promise<CatalogTable | undefined> {
|
||||
const current = this.tables.get(tableId);
|
||||
if (!current || current.databaseId !== databaseId || current.version !== expectedVersion) return undefined;
|
||||
const updated = {
|
||||
...current, description, generatedDescription, version: current.version + 1,
|
||||
updatedAt: new Date().toISOString(),
|
||||
};
|
||||
this.tables.set(tableId, updated);
|
||||
return structuredClone(updated);
|
||||
}
|
||||
|
||||
async listColumns(databaseId: string, tableId: string): Promise<CatalogColumn[]> {
|
||||
const table = this.tables.get(tableId);
|
||||
if (!table || table.databaseId !== databaseId) return [];
|
||||
return [...this.columns.values()].filter((column) => column.tableId === tableId)
|
||||
.sort((a, b) => a.ordinalPosition - b.ordinalPosition).map((column) => structuredClone(column));
|
||||
}
|
||||
|
||||
async getColumn(databaseId: string, tableId: string, columnId: string): Promise<CatalogColumn | undefined> {
|
||||
const table = this.tables.get(tableId);
|
||||
const column = this.columns.get(columnId);
|
||||
return table?.databaseId === databaseId && column?.tableId === tableId ? structuredClone(column) : undefined;
|
||||
}
|
||||
|
||||
async updateColumnMetadata(
|
||||
databaseId: string,
|
||||
tableId: string,
|
||||
columnId: string,
|
||||
expectedVersion: number,
|
||||
description: string | null,
|
||||
generatedDescription: string | null,
|
||||
): Promise<CatalogColumn | undefined> {
|
||||
const current = await this.getColumn(databaseId, tableId, columnId);
|
||||
if (!current || current.version !== expectedVersion) return undefined;
|
||||
const updated = { ...current, description, generatedDescription, version: current.version + 1, updatedAt: new Date().toISOString() };
|
||||
this.columns.set(columnId, updated);
|
||||
return structuredClone(updated);
|
||||
}
|
||||
|
||||
async listRelationships(databaseId: string): Promise<CatalogRelationship[]> {
|
||||
return [...this.relationships.values()].filter((relationship) => relationship.databaseId === databaseId)
|
||||
.sort((a, b) => `${a.sourceTableName}.${a.constraintName}`.localeCompare(`${b.sourceTableName}.${b.constraintName}`))
|
||||
.map((relationship) => structuredClone(relationship));
|
||||
}
|
||||
|
||||
async planSchemaSync(
|
||||
databaseId: string,
|
||||
scope: CatalogSyncScope,
|
||||
tableIds: readonly string[],
|
||||
snapshot: ObservedSchemaSnapshot,
|
||||
): Promise<CatalogSchemaDiff> {
|
||||
this.assertSnapshotCapability(scope, snapshot);
|
||||
const tables = await this.listTables(databaseId);
|
||||
const selectedTableIds = new Set(tableIds);
|
||||
const selectedTables = scope === "columns" && selectedTableIds.size > 0
|
||||
? tables.filter((table) => selectedTableIds.has(table.id))
|
||||
: tables;
|
||||
const observedTables = new Set(snapshot.tables.map((table) => table.name));
|
||||
const observedColumns = new Set(snapshot.columns.map((column) => `${column.tableName}\u0000${column.name}`));
|
||||
const observedRelationships = new Set(
|
||||
snapshot.relationships.map((relationship) => `${relationship.sourceTableName}\u0000${relationship.constraintName}`),
|
||||
);
|
||||
const deletedTableIds = new Set(tables.filter((table) => !observedTables.has(table.name)).map((table) => table.id));
|
||||
|
||||
return {
|
||||
deletedTables: scope === "tables" || scope === "all"
|
||||
? tables.filter((table) => !observedTables.has(table.name)).map((table) => table.name).sort()
|
||||
: [],
|
||||
deletedColumns: scope === "tables" || scope === "columns" || scope === "all"
|
||||
? [...this.columns.values()]
|
||||
.filter((column) => scope === "tables" ? deletedTableIds.has(column.tableId) : selectedTables.some((table) => table.id === column.tableId))
|
||||
.filter((column) => {
|
||||
const table = tables.find((candidate) => candidate.id === column.tableId);
|
||||
return table && (scope === "tables" || !observedColumns.has(`${table.name}\u0000${column.name}`));
|
||||
})
|
||||
.map((column) => ({
|
||||
tableName: tables.find((table) => table.id === column.tableId)?.name ?? "",
|
||||
columnName: column.name,
|
||||
}))
|
||||
.sort((a, b) => `${a.tableName}.${a.columnName}`.localeCompare(`${b.tableName}.${b.columnName}`))
|
||||
: [],
|
||||
deletedRelationships: scope === "tables" || scope === "relationships" || scope === "all"
|
||||
? [...this.relationships.values()]
|
||||
.filter((relationship) => relationship.databaseId === databaseId)
|
||||
.filter((relationship) => scope === "tables"
|
||||
? deletedTableIds.has(relationship.sourceTableId) || deletedTableIds.has(relationship.targetTableId)
|
||||
: !observedRelationships.has(`${relationship.sourceTableName}\u0000${relationship.constraintName}`))
|
||||
.map((relationship) => ({
|
||||
sourceTableName: relationship.sourceTableName,
|
||||
constraintName: relationship.constraintName,
|
||||
}))
|
||||
.sort((a, b) => `${a.sourceTableName}.${a.constraintName}`.localeCompare(`${b.sourceTableName}.${b.constraintName}`))
|
||||
: [],
|
||||
};
|
||||
}
|
||||
|
||||
async applySchemaSync(
|
||||
databaseId: string,
|
||||
expectedDatabaseVersion: number,
|
||||
scope: CatalogSyncScope,
|
||||
tableIds: readonly string[],
|
||||
snapshot: ObservedSchemaSnapshot,
|
||||
): Promise<CatalogSyncCounts | undefined> {
|
||||
const database = this.records.get(databaseId);
|
||||
if (!database || database.version !== expectedDatabaseVersion) return undefined;
|
||||
this.assertSnapshotCapability(scope, snapshot);
|
||||
|
||||
const tableBackup = structuredClone([...this.tables.entries()]);
|
||||
const columnBackup = structuredClone([...this.columns.entries()]);
|
||||
const relationshipBackup = structuredClone([...this.relationships.entries()]);
|
||||
const now = new Date().toISOString();
|
||||
let created = 0;
|
||||
let updated = 0;
|
||||
let deleted = 0;
|
||||
|
||||
try {
|
||||
if (scope === "tables" || scope === "all") {
|
||||
const observedByName = new Map(snapshot.tables.map((table) => [table.name, table]));
|
||||
const existing = await this.listTables(databaseId);
|
||||
for (const table of existing) {
|
||||
const observed = observedByName.get(table.name);
|
||||
if (!observed) {
|
||||
this.deleteTable(table.id);
|
||||
deleted += 1;
|
||||
} else {
|
||||
const changed = table.sourceComment !== observed.sourceComment;
|
||||
this.tables.set(table.id, {
|
||||
...table,
|
||||
sourceComment: observed.sourceComment,
|
||||
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||
lastSyncedAt: now,
|
||||
version: changed ? table.version + 1 : table.version,
|
||||
updatedAt: changed ? now : table.updatedAt,
|
||||
});
|
||||
if (changed) updated += 1;
|
||||
observedByName.delete(table.name);
|
||||
}
|
||||
}
|
||||
for (const observed of observedByName.values()) {
|
||||
const id = randomUUID();
|
||||
this.tables.set(id, {
|
||||
id,
|
||||
databaseId,
|
||||
name: observed.name,
|
||||
sourceComment: observed.sourceComment,
|
||||
description: null,
|
||||
generatedDescription: null,
|
||||
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||
lastSyncedAt: now,
|
||||
version: 1,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
});
|
||||
created += 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (scope === "columns" || scope === "all") {
|
||||
const currentTables = await this.listTables(databaseId);
|
||||
const selectedIds = scope === "all" || tableIds.length === 0
|
||||
? new Set(currentTables.map((table) => table.id))
|
||||
: new Set(tableIds);
|
||||
const selectedTables = currentTables.filter((table) => selectedIds.has(table.id));
|
||||
if (scope === "columns" && selectedTables.length !== selectedIds.size) {
|
||||
throw new CatalogConnectorError("One or more selected tables no longer exist");
|
||||
}
|
||||
const selectedNames = new Set(selectedTables.map((table) => table.name));
|
||||
const tableByName = new Map(selectedTables.map((table) => [table.name, table]));
|
||||
const observedByKey = new Map(
|
||||
snapshot.columns
|
||||
.filter((column) => selectedNames.has(column.tableName))
|
||||
.map((column) => [`${column.tableName}\u0000${column.name}`, column]),
|
||||
);
|
||||
for (const column of [...this.columns.values()].filter((candidate) => selectedIds.has(candidate.tableId))) {
|
||||
const table = selectedTables.find((candidate) => candidate.id === column.tableId);
|
||||
if (!table) continue;
|
||||
const key = `${table.name}\u0000${column.name}`;
|
||||
const observed = observedByKey.get(key);
|
||||
if (!observed) {
|
||||
this.deleteColumn(column.id);
|
||||
deleted += 1;
|
||||
} else {
|
||||
const changed = column.ordinalPosition !== observed.ordinalPosition
|
||||
|| column.dataType !== observed.dataType
|
||||
|| column.isNullable !== observed.isNullable
|
||||
|| column.defaultExpression !== observed.defaultExpression
|
||||
|| column.primaryKeyPosition !== observed.primaryKeyPosition
|
||||
|| column.sourceComment !== observed.sourceComment;
|
||||
this.columns.set(column.id, {
|
||||
...column,
|
||||
ordinalPosition: observed.ordinalPosition,
|
||||
dataType: observed.dataType,
|
||||
isNullable: observed.isNullable,
|
||||
defaultExpression: observed.defaultExpression,
|
||||
primaryKeyPosition: observed.primaryKeyPosition,
|
||||
isPrimaryKey: observed.primaryKeyPosition !== null,
|
||||
sourceComment: observed.sourceComment,
|
||||
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||
lastSyncedAt: now,
|
||||
version: changed ? column.version + 1 : column.version,
|
||||
updatedAt: changed ? now : column.updatedAt,
|
||||
});
|
||||
if (changed) updated += 1;
|
||||
observedByKey.delete(key);
|
||||
}
|
||||
}
|
||||
for (const observed of observedByKey.values()) {
|
||||
const table = tableByName.get(observed.tableName);
|
||||
if (!table) continue;
|
||||
const id = randomUUID();
|
||||
this.columns.set(id, {
|
||||
id,
|
||||
tableId: table.id,
|
||||
name: observed.name,
|
||||
ordinalPosition: observed.ordinalPosition,
|
||||
dataType: observed.dataType,
|
||||
isNullable: observed.isNullable,
|
||||
defaultExpression: observed.defaultExpression,
|
||||
primaryKeyPosition: observed.primaryKeyPosition,
|
||||
isPrimaryKey: observed.primaryKeyPosition !== null,
|
||||
isForeignKey: false,
|
||||
foreignKeyCount: 0,
|
||||
sourceComment: observed.sourceComment,
|
||||
description: null,
|
||||
generatedDescription: null,
|
||||
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||
lastSyncedAt: now,
|
||||
version: 1,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
});
|
||||
created += 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (scope === "relationships" || scope === "all") {
|
||||
const tables = await this.listTables(databaseId);
|
||||
const tableByName = new Map(tables.map((table) => [table.name, table]));
|
||||
const existing = [...this.relationships.values()].filter((relationship) => relationship.databaseId === databaseId);
|
||||
const existingByKey = new Map(existing.map((relationship) => [`${relationship.sourceTableName}\u0000${relationship.constraintName}`, relationship]));
|
||||
const observedKeys = new Set(snapshot.relationships.map((relationship) => `${relationship.sourceTableName}\u0000${relationship.constraintName}`));
|
||||
for (const relationship of existing) {
|
||||
if (!observedKeys.has(`${relationship.sourceTableName}\u0000${relationship.constraintName}`)) {
|
||||
this.relationships.delete(relationship.id);
|
||||
deleted += 1;
|
||||
}
|
||||
}
|
||||
for (const observed of snapshot.relationships) {
|
||||
const sourceTable = tableByName.get(observed.sourceTableName);
|
||||
const targetTable = tableByName.get(observed.targetTableName);
|
||||
if (!sourceTable || !targetTable) {
|
||||
throw new CatalogConnectorError(`Relationship ${observed.constraintName} refers to an unknown table`);
|
||||
}
|
||||
const pairs = observed.columns.map((pair) => {
|
||||
const source = [...this.columns.values()].find((column) => column.tableId === sourceTable.id && column.name === pair.sourceColumnName);
|
||||
const target = [...this.columns.values()].find((column) => column.tableId === targetTable.id && column.name === pair.targetColumnName);
|
||||
if (!source || !target) {
|
||||
throw new CatalogConnectorError(`Relationship ${observed.constraintName} refers to an unknown column`);
|
||||
}
|
||||
return {
|
||||
position: pair.position,
|
||||
sourceColumnId: source.id,
|
||||
sourceColumnName: source.name,
|
||||
targetColumnId: target.id,
|
||||
targetColumnName: target.name,
|
||||
};
|
||||
}).sort((a, b) => a.position - b.position);
|
||||
const key = `${observed.sourceTableName}\u0000${observed.constraintName}`;
|
||||
const current = existingByKey.get(key);
|
||||
const comparable = current && JSON.stringify({
|
||||
target: current.targetTableName,
|
||||
update: current.updateRule,
|
||||
delete: current.deleteRule,
|
||||
deferrable: current.deferrable,
|
||||
deferred: current.initiallyDeferred,
|
||||
columns: current.columns.map((pair) => [pair.position, pair.sourceColumnName, pair.targetColumnName]),
|
||||
});
|
||||
const nextComparable = JSON.stringify({
|
||||
target: observed.targetTableName,
|
||||
update: observed.updateRule,
|
||||
delete: observed.deleteRule,
|
||||
deferrable: observed.deferrable,
|
||||
deferred: observed.initiallyDeferred,
|
||||
columns: pairs.map((pair) => [pair.position, pair.sourceColumnName, pair.targetColumnName]),
|
||||
});
|
||||
const id = current?.id ?? randomUUID();
|
||||
this.relationships.set(id, {
|
||||
id,
|
||||
databaseId,
|
||||
constraintName: observed.constraintName,
|
||||
sourceTableId: sourceTable.id,
|
||||
sourceTableName: sourceTable.name,
|
||||
targetTableId: targetTable.id,
|
||||
targetTableName: targetTable.name,
|
||||
updateRule: observed.updateRule,
|
||||
deleteRule: observed.deleteRule,
|
||||
deferrable: observed.deferrable,
|
||||
initiallyDeferred: observed.initiallyDeferred,
|
||||
columns: pairs,
|
||||
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||
lastSyncedAt: now,
|
||||
createdAt: current?.createdAt ?? now,
|
||||
updatedAt: comparable === nextComparable ? (current?.updatedAt ?? now) : now,
|
||||
});
|
||||
if (!current) created += 1;
|
||||
else if (comparable !== nextComparable) updated += 1;
|
||||
}
|
||||
this.refreshForeignKeyFlags(databaseId);
|
||||
}
|
||||
|
||||
if (scope === "all") {
|
||||
this.records.set(databaseId, { ...database, schemaSyncedVersion: expectedDatabaseVersion, schemaSyncedAt: now });
|
||||
}
|
||||
return {
|
||||
tables: (await this.listTables(databaseId)).length,
|
||||
columns: [...this.columns.values()].filter((column) => this.tables.get(column.tableId)?.databaseId === databaseId).length,
|
||||
relationships: (await this.listRelationships(databaseId)).length,
|
||||
created,
|
||||
updated,
|
||||
deleted,
|
||||
};
|
||||
} catch (error) {
|
||||
this.tables.clear();
|
||||
this.columns.clear();
|
||||
this.relationships.clear();
|
||||
for (const [id, table] of tableBackup) this.tables.set(id, table);
|
||||
for (const [id, column] of columnBackup) this.columns.set(id, column);
|
||||
for (const [id, relationship] of relationshipBackup) this.relationships.set(id, relationship);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async createSyncRun(
|
||||
databaseId: string,
|
||||
scope: CatalogSyncScope,
|
||||
tableIds: readonly string[],
|
||||
requestedDatabaseVersion: number,
|
||||
): Promise<CatalogSyncRun> {
|
||||
const activeStates = new Set<CatalogSyncRun["state"]>(["queued", "running", "awaiting_confirmation", "applying"]);
|
||||
if ([...this.syncRuns.values()].some((run) => run.databaseId === databaseId && activeStates.has(run.state))) {
|
||||
throw new CatalogConflictError("A schema synchronization is already active for this database");
|
||||
}
|
||||
const now = new Date().toISOString();
|
||||
const run: CatalogSyncRun = {
|
||||
id: randomUUID(), databaseId, scope, tableIds: [...tableIds], state: "queued", phase: "queued",
|
||||
requestedDatabaseVersion, observedSnapshot: null, plannedDiff: null, confirmationToken: null,
|
||||
counts: {}, errorCode: null, errorMessage: null, cancelRequested: false,
|
||||
createdAt: now, startedAt: null, updatedAt: now, finishedAt: null, heartbeatAt: null,
|
||||
leaseOwner: null, leaseExpiresAt: null,
|
||||
};
|
||||
this.syncRuns.set(run.id, run);
|
||||
return structuredClone(run);
|
||||
}
|
||||
|
||||
async getSyncRun(runId: string): Promise<CatalogSyncRun | undefined> {
|
||||
const run = this.syncRuns.get(runId);
|
||||
return run ? structuredClone(run) : undefined;
|
||||
}
|
||||
|
||||
async claimSyncRun(runId: string, workerId: string, leaseExpiresAt: string): Promise<CatalogSyncRun | undefined> {
|
||||
const run = this.syncRuns.get(runId);
|
||||
if (!run || run.state !== "queued") return undefined;
|
||||
if (run.leaseOwner && run.leaseOwner !== workerId && run.leaseExpiresAt && run.leaseExpiresAt > new Date().toISOString()) {
|
||||
return undefined;
|
||||
}
|
||||
return await this.updateSyncRun(runId, {
|
||||
state: "running",
|
||||
startedAt: run.startedAt ?? new Date().toISOString(),
|
||||
heartbeatAt: new Date().toISOString(),
|
||||
leaseOwner: workerId,
|
||||
leaseExpiresAt,
|
||||
});
|
||||
}
|
||||
|
||||
async listSyncRuns(databaseId: string, limit = 20): Promise<CatalogSyncRun[]> {
|
||||
return [...this.syncRuns.values()].filter((run) => run.databaseId === databaseId)
|
||||
.sort((a, b) => b.createdAt.localeCompare(a.createdAt)).slice(0, limit).map((run) => structuredClone(run));
|
||||
}
|
||||
|
||||
async updateSyncRun(runId: string, update: CatalogSyncRunUpdate): Promise<CatalogSyncRun | undefined> {
|
||||
const current = this.syncRuns.get(runId);
|
||||
if (!current) return undefined;
|
||||
const updated = { ...current, ...structuredClone(update), updatedAt: new Date().toISOString() };
|
||||
this.syncRuns.set(runId, updated);
|
||||
return structuredClone(updated);
|
||||
}
|
||||
|
||||
async requestSyncRunCancellation(runId: string): Promise<CatalogSyncRun | undefined> {
|
||||
const run = this.syncRuns.get(runId);
|
||||
if (!run) return undefined;
|
||||
if (!["queued", "running", "awaiting_confirmation"].includes(run.state)) return structuredClone(run);
|
||||
return await this.updateSyncRun(runId, { cancelRequested: true });
|
||||
}
|
||||
|
||||
async appendSyncEvent(
|
||||
runId: string,
|
||||
level: CatalogSyncEvent["level"],
|
||||
eventType: string,
|
||||
message: string,
|
||||
data: Record<string, unknown> = {},
|
||||
): Promise<CatalogSyncEvent> {
|
||||
const events = this.syncEvents.get(runId) ?? [];
|
||||
const event: CatalogSyncEvent = {
|
||||
id: [...this.syncEvents.values()].reduce((count, values) => count + values.length, 0) + 1,
|
||||
runId, sequence: events.length + 1, level, eventType, message, data: structuredClone(data),
|
||||
createdAt: new Date().toISOString(),
|
||||
};
|
||||
events.push(event);
|
||||
this.syncEvents.set(runId, events);
|
||||
return structuredClone(event);
|
||||
}
|
||||
|
||||
async listSyncEvents(runId: string, afterSequence = 0): Promise<CatalogSyncEvent[]> {
|
||||
return (this.syncEvents.get(runId) ?? []).filter((event) => event.sequence > afterSequence).map((event) => structuredClone(event));
|
||||
}
|
||||
|
||||
async pruneSyncEvents(before: string): Promise<void> {
|
||||
for (const [runId, events] of this.syncEvents) {
|
||||
this.syncEvents.set(runId, events.filter((event) => event.createdAt >= before));
|
||||
}
|
||||
}
|
||||
|
||||
async interruptActiveSyncRuns(): Promise<void> {
|
||||
for (const run of this.syncRuns.values()) {
|
||||
if (["queued", "running", "awaiting_confirmation", "applying"].includes(run.state)) {
|
||||
await this.updateSyncRun(run.id, {
|
||||
state: "interrupted", phase: "completed", finishedAt: new Date().toISOString(),
|
||||
errorCode: "SYNC_INTERRUPTED", errorMessage: "Synchronization was interrupted by a service restart",
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async reconcileTables(
|
||||
databaseId: string,
|
||||
expectedDatabaseVersion: number,
|
||||
observed: readonly ObservedCatalogTable[],
|
||||
confirmedDeletedNames: readonly string[],
|
||||
): Promise<TableSyncRepositoryResult | undefined> {
|
||||
const database = this.records.get(databaseId);
|
||||
if (!database || database.version !== expectedDatabaseVersion) return undefined;
|
||||
const existing = await this.listTables(databaseId);
|
||||
const observedNames = new Set(observed.map((table) => table.name));
|
||||
const deletedNames = existing.filter((table) => !observedNames.has(table.name)).map((table) => table.name).sort();
|
||||
const confirmation = [...new Set(confirmedDeletedNames)].sort();
|
||||
if (deletedNames.length > 0 && JSON.stringify(deletedNames) !== JSON.stringify(confirmation)) {
|
||||
return { kind: "confirmation_required", deletedNames };
|
||||
}
|
||||
|
||||
const byName = new Map(existing.map((table) => [table.name, table]));
|
||||
let createdCount = 0;
|
||||
let updatedCount = 0;
|
||||
for (const observedTable of observed) {
|
||||
const current = byName.get(observedTable.name);
|
||||
const now = new Date().toISOString();
|
||||
if (!current) {
|
||||
const created: CatalogTable = {
|
||||
id: randomUUID(),
|
||||
databaseId,
|
||||
name: observedTable.name,
|
||||
sourceComment: observedTable.sourceComment,
|
||||
description: null,
|
||||
generatedDescription: null,
|
||||
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||
lastSyncedAt: now,
|
||||
version: 1,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
};
|
||||
this.tables.set(created.id, created);
|
||||
createdCount += 1;
|
||||
} else if (current.sourceComment !== observedTable.sourceComment) {
|
||||
this.tables.set(current.id, {
|
||||
...current,
|
||||
sourceComment: observedTable.sourceComment,
|
||||
version: current.version + 1,
|
||||
updatedAt: now,
|
||||
});
|
||||
updatedCount += 1;
|
||||
}
|
||||
}
|
||||
for (const table of existing) {
|
||||
if (deletedNames.includes(table.name)) this.deleteTable(table.id);
|
||||
}
|
||||
return {
|
||||
kind: "applied",
|
||||
createdCount,
|
||||
updatedCount,
|
||||
deletedCount: deletedNames.length,
|
||||
tables: await this.listTables(databaseId),
|
||||
};
|
||||
}
|
||||
|
||||
private assertSnapshotCapability(scope: CatalogSyncScope, snapshot: ObservedSchemaSnapshot): void {
|
||||
const required = scope === "all" ? ["tables", "columns", "relationships"] as const : [scope] as const;
|
||||
for (const capability of required) {
|
||||
if (snapshot.capabilities[capability] !== "available") {
|
||||
throw new CatalogConnectorError(`Schema introspection capability '${capability}' is unavailable`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private deleteTable(tableId: string): void {
|
||||
this.tables.delete(tableId);
|
||||
for (const column of [...this.columns.values()]) if (column.tableId === tableId) this.deleteColumn(column.id);
|
||||
for (const relationship of [...this.relationships.values()]) {
|
||||
if (relationship.sourceTableId === tableId || relationship.targetTableId === tableId) {
|
||||
this.relationships.delete(relationship.id);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private deleteColumn(columnId: string): void {
|
||||
this.columns.delete(columnId);
|
||||
for (const relationship of [...this.relationships.values()]) {
|
||||
if (relationship.columns.some((pair) => pair.sourceColumnId === columnId || pair.targetColumnId === columnId)) {
|
||||
this.relationships.delete(relationship.id);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private refreshForeignKeyFlags(databaseId: string): void {
|
||||
const counts = new Map<string, number>();
|
||||
for (const relationship of this.relationships.values()) {
|
||||
if (relationship.databaseId !== databaseId) continue;
|
||||
for (const pair of relationship.columns) counts.set(pair.sourceColumnId, (counts.get(pair.sourceColumnId) ?? 0) + 1);
|
||||
}
|
||||
for (const column of this.columns.values()) {
|
||||
if (this.tables.get(column.tableId)?.databaseId !== databaseId) continue;
|
||||
const foreignKeyCount = counts.get(column.id) ?? 0;
|
||||
this.columns.set(column.id, { ...column, isForeignKey: foreignKeyCount > 0, foreignKeyCount });
|
||||
}
|
||||
}
|
||||
|
||||
async available(): Promise<boolean> { return true; }
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
import { CamelCasePlugin, Kysely, PostgresDialect } from "kysely";
|
||||
import { Migrator, type MigrationProvider } from "kysely/migration";
|
||||
import { Pool } from "pg";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import type { CatalogDatabase } from "./repository.js";
|
||||
import * as initialMigration from "./migrations/001_workspace_databases.js";
|
||||
import * as catalogTablesMigration from "./migrations/002_catalog_tables.js";
|
||||
import * as catalogSchemaSyncMigration from "./migrations/003_catalog_schema_sync.js";
|
||||
import * as catalogRuntimeSequencePrivilegesMigration from "./migrations/004_catalog_runtime_sequence_privileges.js";
|
||||
|
||||
const connectionString = process.env.THT_CATALOG_MIGRATOR_DATABASE_URL;
|
||||
const host = process.env.THT_CATALOG_DB_HOST;
|
||||
const database = process.env.THT_CATALOG_DB_NAME;
|
||||
const user = process.env.THT_CATALOG_MIGRATOR_USER;
|
||||
const passwordFile = process.env.THT_CATALOG_MIGRATOR_PASSWORD_FILE;
|
||||
if (!connectionString && (!host || !database || !user || !passwordFile)) {
|
||||
throw new Error("catalog migrator database configuration is required");
|
||||
}
|
||||
const pool = new Pool(connectionString ? { connectionString, max: 1 } : {
|
||||
host,
|
||||
port: Number(process.env.THT_CATALOG_DB_PORT ?? 5432),
|
||||
database,
|
||||
user,
|
||||
password: async () => (await readFile(passwordFile!, "utf8")).trim(),
|
||||
max: 1,
|
||||
});
|
||||
|
||||
const db = new Kysely<CatalogDatabase>({
|
||||
dialect: new PostgresDialect({ pool }),
|
||||
plugins: [new CamelCasePlugin()],
|
||||
});
|
||||
const provider: MigrationProvider = {
|
||||
async getMigrations() {
|
||||
return {
|
||||
"001_workspace_databases": initialMigration,
|
||||
"002_catalog_tables": catalogTablesMigration,
|
||||
"003_catalog_schema_sync": catalogSchemaSyncMigration,
|
||||
"004_catalog_runtime_sequence_privileges": catalogRuntimeSequencePrivilegesMigration,
|
||||
};
|
||||
},
|
||||
};
|
||||
|
||||
try {
|
||||
const result = await new Migrator({ db, provider }).migrateToLatest();
|
||||
for (const item of result.results ?? []) {
|
||||
process.stdout.write(`${item.migrationName}: ${item.status}\n`);
|
||||
}
|
||||
if (result.error) throw result.error;
|
||||
} finally {
|
||||
await db.destroy();
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
import { sql, type Kysely } from "kysely";
|
||||
import type { CatalogDatabase } from "../repository.js";
|
||||
|
||||
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||
await db.schema.createTable("workspace_databases")
|
||||
.addColumn("id", "uuid", (column) => column.primaryKey())
|
||||
.addColumn("workspace_id", "text", (column) => column.notNull().unique())
|
||||
.addColumn("engine", "text", (column) => column.notNull())
|
||||
.addColumn("database_name", "text", (column) => column.notNull())
|
||||
.addColumn("schema_name", "text", (column) => column.notNull())
|
||||
.addColumn("version", "integer", (column) => column.notNull().defaultTo(1))
|
||||
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addCheckConstraint("workspace_databases_engine_check", sql`engine = 'postgres'`)
|
||||
.addCheckConstraint("workspace_databases_version_check", sql`version > 0`)
|
||||
.execute();
|
||||
|
||||
await db.schema.createTable("database_bindings")
|
||||
.addColumn("database_id", "uuid", (column) => column.primaryKey()
|
||||
.references("workspace_databases.id").onDelete("cascade"))
|
||||
.addColumn("transport", "text", (column) => column.notNull())
|
||||
.addColumn("host", "text")
|
||||
.addColumn("port", "integer")
|
||||
.addColumn("username", "text")
|
||||
.addColumn("base_url", "text")
|
||||
.addColumn("rest_path", "text")
|
||||
.addColumn("rest_auth", "text")
|
||||
.addColumn("tls_servername", "text")
|
||||
.addColumn("ssh_host", "text")
|
||||
.addColumn("ssh_port", "integer")
|
||||
.addColumn("ssh_username", "text")
|
||||
.addColumn("ssh_target_host", "text")
|
||||
.addColumn("ssh_target_port", "integer")
|
||||
.addColumn("connection_status", "text", (column) => column.notNull().defaultTo("untested"))
|
||||
.addColumn("tested_version", "integer")
|
||||
.addColumn("last_tested_at", "timestamptz")
|
||||
.addColumn("last_error_code", "text")
|
||||
.addColumn("last_error_message", "text")
|
||||
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addCheckConstraint("database_bindings_transport_check", sql`transport in ('postgres_direct', 'rest_api', 'ssh_tunnel')`)
|
||||
.addCheckConstraint("database_bindings_status_check", sql`connection_status in ('untested', 'reachable', 'failed')`)
|
||||
.addCheckConstraint("database_bindings_port_check", sql`port is null or port between 1 and 65535`)
|
||||
.addCheckConstraint("database_bindings_ssh_port_check", sql`ssh_port is null or ssh_port between 1 and 65535`)
|
||||
.addCheckConstraint("database_bindings_ssh_target_port_check", sql`ssh_target_port is null or ssh_target_port between 1 and 65535`)
|
||||
.addCheckConstraint("database_bindings_transport_fields_check", sql`
|
||||
(transport = 'postgres_direct' and host is not null and port is not null and username is not null)
|
||||
or (transport = 'rest_api' and base_url is not null and rest_path is not null and rest_auth is not null)
|
||||
or (transport = 'ssh_tunnel' and username is not null and ssh_host is not null
|
||||
and ssh_port is not null and ssh_username is not null and ssh_target_host is not null
|
||||
and ssh_target_port is not null)
|
||||
`)
|
||||
.execute();
|
||||
}
|
||||
|
||||
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||
await db.schema.dropTable("database_bindings").execute();
|
||||
await db.schema.dropTable("workspace_databases").execute();
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
import { sql, type Kysely } from "kysely";
|
||||
import type { CatalogDatabase } from "../repository.js";
|
||||
|
||||
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||
await db.schema.createTable("catalog_tables")
|
||||
.addColumn("id", "uuid", (column) => column.primaryKey())
|
||||
.addColumn("database_id", "uuid", (column) => column.notNull()
|
||||
.references("workspace_databases.id").onDelete("cascade"))
|
||||
.addColumn("name", "text", (column) => column.notNull())
|
||||
.addColumn("source_comment", "text")
|
||||
.addColumn("description", "text")
|
||||
.addColumn("generated_description", "text")
|
||||
.addColumn("version", "integer", (column) => column.notNull().defaultTo(1))
|
||||
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addUniqueConstraint("catalog_tables_database_name_key", ["database_id", "name"])
|
||||
.addCheckConstraint("catalog_tables_name_check", sql`char_length(name) between 1 and 128`)
|
||||
.addCheckConstraint("catalog_tables_version_check", sql`version > 0`)
|
||||
.execute();
|
||||
|
||||
await db.schema.createIndex("catalog_tables_database_id_idx")
|
||||
.on("catalog_tables").column("database_id").execute();
|
||||
}
|
||||
|
||||
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||
await db.schema.dropTable("catalog_tables").execute();
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
import { sql, type Kysely } from "kysely";
|
||||
import type { CatalogDatabase } from "../repository.js";
|
||||
|
||||
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||
await db.schema.alterTable("workspace_databases")
|
||||
.addColumn("schema_synced_version", "integer")
|
||||
.addColumn("schema_synced_at", "timestamptz")
|
||||
.execute();
|
||||
|
||||
await db.schema.alterTable("catalog_tables")
|
||||
.addColumn("last_synced_database_version", "integer")
|
||||
.addColumn("last_synced_at", "timestamptz")
|
||||
.execute();
|
||||
|
||||
await db.schema.createTable("catalog_columns")
|
||||
.addColumn("id", "uuid", (column) => column.primaryKey())
|
||||
.addColumn("table_id", "uuid", (column) => column.notNull()
|
||||
.references("catalog_tables.id").onDelete("cascade"))
|
||||
.addColumn("name", "text", (column) => column.notNull())
|
||||
.addColumn("ordinal_position", "integer", (column) => column.notNull())
|
||||
.addColumn("data_type", "text", (column) => column.notNull())
|
||||
.addColumn("is_nullable", "boolean", (column) => column.notNull())
|
||||
.addColumn("default_expression", "text")
|
||||
.addColumn("primary_key_position", "integer")
|
||||
.addColumn("source_comment", "text")
|
||||
.addColumn("description", "text")
|
||||
.addColumn("generated_description", "text")
|
||||
.addColumn("last_synced_database_version", "integer")
|
||||
.addColumn("last_synced_at", "timestamptz")
|
||||
.addColumn("version", "integer", (column) => column.notNull().defaultTo(1))
|
||||
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addUniqueConstraint("catalog_columns_table_name_key", ["table_id", "name"])
|
||||
.addCheckConstraint("catalog_columns_name_check", sql`char_length(name) between 1 and 128`)
|
||||
.addCheckConstraint("catalog_columns_ordinal_check", sql`ordinal_position > 0`)
|
||||
.addCheckConstraint("catalog_columns_pk_position_check", sql`primary_key_position is null or primary_key_position > 0`)
|
||||
.addCheckConstraint("catalog_columns_version_check", sql`version > 0`)
|
||||
.execute();
|
||||
await db.schema.createIndex("catalog_columns_table_id_idx")
|
||||
.on("catalog_columns").column("table_id").execute();
|
||||
|
||||
await db.schema.createTable("catalog_relationships")
|
||||
.addColumn("id", "uuid", (column) => column.primaryKey())
|
||||
.addColumn("database_id", "uuid", (column) => column.notNull()
|
||||
.references("workspace_databases.id").onDelete("cascade"))
|
||||
.addColumn("constraint_name", "text", (column) => column.notNull())
|
||||
.addColumn("source_table_id", "uuid", (column) => column.notNull()
|
||||
.references("catalog_tables.id").onDelete("cascade"))
|
||||
.addColumn("target_table_id", "uuid", (column) => column.notNull()
|
||||
.references("catalog_tables.id").onDelete("cascade"))
|
||||
.addColumn("update_rule", "text", (column) => column.notNull())
|
||||
.addColumn("delete_rule", "text", (column) => column.notNull())
|
||||
.addColumn("deferrable", "boolean", (column) => column.notNull().defaultTo(false))
|
||||
.addColumn("initially_deferred", "boolean", (column) => column.notNull().defaultTo(false))
|
||||
.addColumn("last_synced_database_version", "integer")
|
||||
.addColumn("last_synced_at", "timestamptz")
|
||||
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addUniqueConstraint("catalog_relationships_source_constraint_key", ["source_table_id", "constraint_name"])
|
||||
.execute();
|
||||
await db.schema.createIndex("catalog_relationships_database_id_idx")
|
||||
.on("catalog_relationships").column("database_id").execute();
|
||||
|
||||
await db.schema.createTable("catalog_relationship_columns")
|
||||
.addColumn("relationship_id", "uuid", (column) => column.notNull()
|
||||
.references("catalog_relationships.id").onDelete("cascade"))
|
||||
.addColumn("position", "integer", (column) => column.notNull())
|
||||
.addColumn("source_column_id", "uuid", (column) => column.notNull()
|
||||
.references("catalog_columns.id").onDelete("cascade"))
|
||||
.addColumn("target_column_id", "uuid", (column) => column.notNull()
|
||||
.references("catalog_columns.id").onDelete("cascade"))
|
||||
.addPrimaryKeyConstraint("catalog_relationship_columns_pkey", ["relationship_id", "position"])
|
||||
.addCheckConstraint("catalog_relationship_columns_position_check", sql`position > 0`)
|
||||
.execute();
|
||||
|
||||
await db.schema.createTable("catalog_sync_runs")
|
||||
.addColumn("id", "uuid", (column) => column.primaryKey())
|
||||
.addColumn("database_id", "uuid", (column) => column.notNull()
|
||||
.references("workspace_databases.id").onDelete("cascade"))
|
||||
.addColumn("scope", "text", (column) => column.notNull())
|
||||
.addColumn("table_ids", "jsonb", (column) => column.notNull().defaultTo(sql`'[]'::jsonb`))
|
||||
.addColumn("state", "text", (column) => column.notNull())
|
||||
.addColumn("phase", "text", (column) => column.notNull())
|
||||
.addColumn("requested_database_version", "integer", (column) => column.notNull())
|
||||
.addColumn("observed_snapshot", "jsonb")
|
||||
.addColumn("planned_diff", "jsonb")
|
||||
.addColumn("confirmation_token", "text")
|
||||
.addColumn("counts", "jsonb", (column) => column.notNull().defaultTo(sql`'{}'::jsonb`))
|
||||
.addColumn("error_code", "text")
|
||||
.addColumn("error_message", "text")
|
||||
.addColumn("cancel_requested", "boolean", (column) => column.notNull().defaultTo(false))
|
||||
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addColumn("started_at", "timestamptz")
|
||||
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addColumn("finished_at", "timestamptz")
|
||||
.addColumn("heartbeat_at", "timestamptz")
|
||||
.addColumn("lease_owner", "text")
|
||||
.addColumn("lease_expires_at", "timestamptz")
|
||||
.execute();
|
||||
await db.schema.createIndex("catalog_sync_runs_database_created_idx")
|
||||
.on("catalog_sync_runs").columns(["database_id", "created_at"]).execute();
|
||||
await sql`CREATE UNIQUE INDEX catalog_sync_runs_one_active_per_database
|
||||
ON catalog_sync_runs (database_id)
|
||||
WHERE state IN ('queued', 'running', 'awaiting_confirmation', 'applying')`.execute(db);
|
||||
|
||||
await db.schema.createTable("catalog_sync_events")
|
||||
.addColumn("id", "bigserial", (column) => column.primaryKey())
|
||||
.addColumn("run_id", "uuid", (column) => column.notNull()
|
||||
.references("catalog_sync_runs.id").onDelete("cascade"))
|
||||
.addColumn("sequence", "integer", (column) => column.notNull())
|
||||
.addColumn("level", "text", (column) => column.notNull())
|
||||
.addColumn("event_type", "text", (column) => column.notNull())
|
||||
.addColumn("message", "text", (column) => column.notNull())
|
||||
.addColumn("data", "jsonb", (column) => column.notNull().defaultTo(sql`'{}'::jsonb`))
|
||||
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||
.addUniqueConstraint("catalog_sync_events_run_sequence_key", ["run_id", "sequence"])
|
||||
.execute();
|
||||
await db.schema.createIndex("catalog_sync_events_run_id_idx")
|
||||
.on("catalog_sync_events").columns(["run_id", "sequence"]).execute();
|
||||
}
|
||||
|
||||
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||
await db.schema.dropTable("catalog_sync_events").execute();
|
||||
await db.schema.dropTable("catalog_sync_runs").execute();
|
||||
await db.schema.dropTable("catalog_relationship_columns").execute();
|
||||
await db.schema.dropTable("catalog_relationships").execute();
|
||||
await db.schema.dropTable("catalog_columns").execute();
|
||||
await db.schema.alterTable("catalog_tables")
|
||||
.dropColumn("last_synced_database_version")
|
||||
.dropColumn("last_synced_at")
|
||||
.execute();
|
||||
await db.schema.alterTable("workspace_databases")
|
||||
.dropColumn("schema_synced_version")
|
||||
.dropColumn("schema_synced_at")
|
||||
.execute();
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
import { type Kysely, sql } from "kysely";
|
||||
import type { CatalogDatabase } from "../repository.js";
|
||||
|
||||
/**
|
||||
* `catalog_sync_events.id` is the first catalog-owned identity sequence.
|
||||
* Table default privileges do not cover sequences, and without USAGE the
|
||||
* runtime can create a run but cannot append its first event.
|
||||
*/
|
||||
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||
await sql`DO $catalog_privileges$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = 'thothii_catalog_runtime') THEN
|
||||
EXECUTE 'GRANT USAGE, SELECT ON SEQUENCE catalog_sync_events_id_seq TO thothii_catalog_runtime';
|
||||
EXECUTE 'ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO thothii_catalog_runtime';
|
||||
END IF;
|
||||
END
|
||||
$catalog_privileges$`.execute(db);
|
||||
}
|
||||
|
||||
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||
await sql`DO $catalog_privileges$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = 'thothii_catalog_runtime') THEN
|
||||
EXECUTE 'ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE USAGE, SELECT ON SEQUENCES FROM thothii_catalog_runtime';
|
||||
EXECUTE 'REVOKE USAGE, SELECT ON SEQUENCE catalog_sync_events_id_seq FROM thothii_catalog_runtime';
|
||||
END IF;
|
||||
END
|
||||
$catalog_privileges$`.execute(db);
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
import { CatalogOperationInProgressError } from "./types.js";
|
||||
|
||||
/** Serializes connection tests and schema synchronization for each catalog database. */
|
||||
export class CatalogOperationCoordinator {
|
||||
private readonly active = new Set<string>();
|
||||
|
||||
reserve(databaseId: string): () => void {
|
||||
if (this.active.has(databaseId)) {
|
||||
throw new CatalogOperationInProgressError("A database operation is already in progress");
|
||||
}
|
||||
this.active.add(databaseId);
|
||||
let released = false;
|
||||
return () => {
|
||||
if (released) return;
|
||||
released = true;
|
||||
this.active.delete(databaseId);
|
||||
};
|
||||
}
|
||||
|
||||
async run<T>(databaseId: string, operation: () => Promise<T>): Promise<T> {
|
||||
const release = this.reserve(databaseId);
|
||||
try {
|
||||
return await operation();
|
||||
} finally {
|
||||
release();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,258 @@
|
||||
import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { Duplex } from "node:stream";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { Client, type ClientConfig } from "pg";
|
||||
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
|
||||
import { CATALOG_SECRET_IDS } from "./secrets.js";
|
||||
import { CatalogConnectorError, type WorkspaceDatabase } from "./types.js";
|
||||
|
||||
export interface CatalogDatabaseClient {
|
||||
query(sql: string, values: readonly unknown[]): Promise<{ rows: Array<Record<string, unknown>> }>;
|
||||
end(): Promise<void>;
|
||||
}
|
||||
|
||||
export interface CatalogPostgresAccess {
|
||||
connect(database: WorkspaceDatabase, signal: AbortSignal): Promise<CatalogDatabaseClient>;
|
||||
}
|
||||
|
||||
type SpawnSsh = (
|
||||
command: string,
|
||||
args: readonly string[],
|
||||
options: { env: NodeJS.ProcessEnv },
|
||||
) => ChildProcessWithoutNullStreams;
|
||||
|
||||
interface AccessDependencies {
|
||||
createClient?: (config: ClientConfig) => Client;
|
||||
spawnSsh?: SpawnSsh;
|
||||
sshBinary?: string;
|
||||
askpassPath?: string;
|
||||
connectTimeoutMs?: number;
|
||||
}
|
||||
|
||||
function required(value: string | number | undefined): string | number {
|
||||
if (value === undefined || value === "") throw new CatalogConnectorError("Database binding is incomplete");
|
||||
return value;
|
||||
}
|
||||
|
||||
function connectionSsl(ca: string | undefined, servername: string | undefined): ClientConfig["ssl"] {
|
||||
if (!ca && !servername) return false;
|
||||
return {
|
||||
...(ca ? { ca } : {}),
|
||||
...(servername ? { servername } : {}),
|
||||
rejectUnauthorized: true,
|
||||
};
|
||||
}
|
||||
|
||||
export function buildSshArguments(input: {
|
||||
sshHost: string;
|
||||
sshPort: number;
|
||||
sshUsername: string;
|
||||
targetHost: string;
|
||||
targetPort: number;
|
||||
privateKeyFile: string;
|
||||
knownHostsFile: string;
|
||||
passphraseFile?: string;
|
||||
connectTimeoutMs: number;
|
||||
}): string[] {
|
||||
const batchMode = input.passphraseFile ? "no" : "yes";
|
||||
return [
|
||||
"-F", "/dev/null",
|
||||
"-T",
|
||||
"-o", `BatchMode=${batchMode}`,
|
||||
"-o", "StrictHostKeyChecking=yes",
|
||||
"-o", `UserKnownHostsFile=${input.knownHostsFile}`,
|
||||
"-o", "GlobalKnownHostsFile=/dev/null",
|
||||
"-o", "IdentitiesOnly=yes",
|
||||
"-o", "IdentityAgent=none",
|
||||
"-o", `IdentityFile=${input.privateKeyFile}`,
|
||||
"-o", "PreferredAuthentications=publickey",
|
||||
"-o", "PasswordAuthentication=no",
|
||||
"-o", "KbdInteractiveAuthentication=no",
|
||||
"-o", "ConnectionAttempts=1",
|
||||
"-o", `ConnectTimeout=${Math.max(1, Math.ceil(input.connectTimeoutMs / 1_000))}`,
|
||||
"-o", "ServerAliveInterval=5",
|
||||
"-o", "ServerAliveCountMax=1",
|
||||
"-o", "NumberOfPasswordPrompts=1",
|
||||
"-o", "RequestTTY=no",
|
||||
"-o", "LogLevel=ERROR",
|
||||
"-p", String(input.sshPort),
|
||||
"-W", `${input.targetHost}:${input.targetPort}`,
|
||||
"--", `${input.sshUsername}@${input.sshHost}`,
|
||||
];
|
||||
}
|
||||
|
||||
async function stopChild(child: ChildProcessWithoutNullStreams): Promise<void> {
|
||||
if (child.exitCode !== null || child.signalCode !== null) return;
|
||||
child.kill("SIGTERM");
|
||||
await Promise.race([
|
||||
new Promise<void>((resolve) => child.once("exit", () => resolve())),
|
||||
delay(500).then(() => undefined),
|
||||
]);
|
||||
if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL");
|
||||
}
|
||||
|
||||
function sshDuplex(child: ChildProcessWithoutNullStreams): Duplex {
|
||||
let ended = false;
|
||||
let stream: Duplex;
|
||||
const forward = () => {
|
||||
let chunk: Buffer | string | null;
|
||||
while ((chunk = child.stdout.read() as Buffer | string | null) !== null) {
|
||||
if (!stream.push(chunk)) break;
|
||||
}
|
||||
};
|
||||
const finish = () => {
|
||||
if (ended) return;
|
||||
ended = true;
|
||||
stream.push(null);
|
||||
};
|
||||
const fail = (error: Error) => stream.destroy(error);
|
||||
stream = new Duplex({
|
||||
read: forward,
|
||||
write: (chunk, encoding, callback) => child.stdin.write(chunk, encoding, callback),
|
||||
final: (callback) => child.stdin.end(callback),
|
||||
destroy: (error, callback) => {
|
||||
child.stdout.off("readable", forward);
|
||||
child.stdout.off("end", finish);
|
||||
child.stdout.off("error", fail);
|
||||
child.stdin.off("error", fail);
|
||||
callback(error);
|
||||
},
|
||||
});
|
||||
child.stdout.on("readable", forward);
|
||||
child.stdout.once("end", finish);
|
||||
child.stdout.once("error", fail);
|
||||
child.stdin.once("error", fail);
|
||||
return stream;
|
||||
}
|
||||
|
||||
/**
|
||||
* Deep connection module for direct and SSH-forwarded PostgreSQL access. It owns secret leases,
|
||||
* TLS, OpenSSH lifecycle, abort propagation, and pg cleanup behind one connect interface.
|
||||
*/
|
||||
export class ConcreteCatalogPostgresAccess implements CatalogPostgresAccess {
|
||||
private readonly createClient: (config: ClientConfig) => Client;
|
||||
private readonly spawnSsh: SpawnSsh;
|
||||
private readonly sshBinary: string;
|
||||
private readonly askpassPath: string;
|
||||
private readonly connectTimeoutMs: number;
|
||||
|
||||
constructor(
|
||||
private readonly secretStore: WorkspaceSecretStore,
|
||||
dependencies: AccessDependencies = {},
|
||||
) {
|
||||
this.createClient = dependencies.createClient ?? ((config) => new Client(config));
|
||||
this.spawnSsh = dependencies.spawnSsh ?? ((command, args, options) => (
|
||||
spawn(command, [...args], { ...options, stdio: ["pipe", "pipe", "pipe"] })
|
||||
));
|
||||
this.sshBinary = dependencies.sshBinary ?? process.env.THT_SSH_BIN ?? "ssh";
|
||||
this.askpassPath = dependencies.askpassPath
|
||||
?? process.env.THT_SSH_ASKPASS_BIN
|
||||
?? fileURLToPath(new URL("../../scripts/ssh-askpass.mjs", import.meta.url));
|
||||
this.connectTimeoutMs = dependencies.connectTimeoutMs ?? 5_000;
|
||||
}
|
||||
|
||||
async connect(database: WorkspaceDatabase, signal: AbortSignal): Promise<CatalogDatabaseClient> {
|
||||
if (database.binding.transport === "rest_api") {
|
||||
throw new CatalogConnectorError("REST is not a PostgreSQL wire binding");
|
||||
}
|
||||
const ids: string[] = [CATALOG_SECRET_IDS.password, CATALOG_SECRET_IDS.tlsCa];
|
||||
if (database.binding.transport === "ssh_tunnel") {
|
||||
ids.push(
|
||||
CATALOG_SECRET_IDS.sshPrivateKey,
|
||||
CATALOG_SECRET_IDS.sshPrivateKeyPassphrase,
|
||||
CATALOG_SECRET_IDS.sshKnownHosts,
|
||||
);
|
||||
}
|
||||
const materialized = this.secretStore.materialize(database.workspaceId, ids);
|
||||
let child: ChildProcessWithoutNullStreams | undefined;
|
||||
let stream: Duplex | undefined;
|
||||
let client: Client | undefined;
|
||||
let ended = false;
|
||||
const close = async () => {
|
||||
if (ended) return;
|
||||
ended = true;
|
||||
signal.removeEventListener("abort", abort);
|
||||
if (client) await client.end().catch(() => undefined);
|
||||
stream?.destroy();
|
||||
if (child) await stopChild(child);
|
||||
materialized.release();
|
||||
};
|
||||
const abort = () => { void close(); };
|
||||
|
||||
try {
|
||||
const passwordFile = materialized.files.get(CATALOG_SECRET_IDS.password);
|
||||
if (!passwordFile) throw new CatalogConnectorError("Database password is not configured");
|
||||
const password = await readFile(passwordFile, "utf8");
|
||||
const tlsCaFile = materialized.files.get(CATALOG_SECRET_IDS.tlsCa);
|
||||
const tlsCa = tlsCaFile ? await readFile(tlsCaFile, "utf8") : undefined;
|
||||
let host: string;
|
||||
let port: number;
|
||||
|
||||
if (database.binding.transport === "ssh_tunnel") {
|
||||
const privateKeyFile = materialized.files.get(CATALOG_SECRET_IDS.sshPrivateKey);
|
||||
const knownHostsFile = materialized.files.get(CATALOG_SECRET_IDS.sshKnownHosts);
|
||||
if (!privateKeyFile || !knownHostsFile) {
|
||||
throw new CatalogConnectorError("SSH private key and known hosts are required");
|
||||
}
|
||||
const passphraseFile = materialized.files.get(CATALOG_SECRET_IDS.sshPrivateKeyPassphrase);
|
||||
host = String(required(database.binding.sshTargetHost));
|
||||
port = Number(required(database.binding.sshTargetPort));
|
||||
const args = buildSshArguments({
|
||||
sshHost: String(required(database.binding.sshHost)),
|
||||
sshPort: Number(required(database.binding.sshPort)),
|
||||
sshUsername: String(required(database.binding.sshUsername)),
|
||||
targetHost: host,
|
||||
targetPort: port,
|
||||
privateKeyFile,
|
||||
knownHostsFile,
|
||||
passphraseFile,
|
||||
connectTimeoutMs: this.connectTimeoutMs,
|
||||
});
|
||||
child = this.spawnSsh(this.sshBinary, args, {
|
||||
env: {
|
||||
...process.env,
|
||||
LC_ALL: "C",
|
||||
...(passphraseFile ? {
|
||||
DISPLAY: "thothii",
|
||||
SSH_ASKPASS: this.askpassPath,
|
||||
SSH_ASKPASS_REQUIRE: "force",
|
||||
THT_SSH_PASSPHRASE_FILE: passphraseFile,
|
||||
} : {}),
|
||||
},
|
||||
});
|
||||
stream = sshDuplex(child);
|
||||
child.once("error", () => stream?.destroy(new CatalogConnectorError("SSH process failed")));
|
||||
child.once("exit", (code) => {
|
||||
if (!ended && code !== 0) stream?.destroy(new CatalogConnectorError("SSH tunnel failed"));
|
||||
});
|
||||
child.stderr.on("data", () => undefined);
|
||||
} else {
|
||||
host = String(required(database.binding.host));
|
||||
port = Number(required(database.binding.port));
|
||||
}
|
||||
|
||||
client = this.createClient({
|
||||
host,
|
||||
port,
|
||||
database: database.databaseName,
|
||||
user: String(required(database.binding.username)),
|
||||
password,
|
||||
ssl: connectionSsl(tlsCa, database.binding.tlsServername),
|
||||
connectionTimeoutMillis: this.connectTimeoutMs,
|
||||
...(stream ? { stream: () => stream } : {}),
|
||||
});
|
||||
signal.addEventListener("abort", abort, { once: true });
|
||||
await client.connect();
|
||||
return {
|
||||
query: async (sql, values) => await client!.query(sql, [...values]),
|
||||
end: close,
|
||||
};
|
||||
} catch (error) {
|
||||
await close();
|
||||
if (error instanceof CatalogConnectorError) throw error;
|
||||
throw new CatalogConnectorError("PostgreSQL connector failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,498 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { z } from "zod";
|
||||
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
|
||||
import type { CatalogPostgresAccess } from "./postgres-access.js";
|
||||
import { CATALOG_SECRET_IDS } from "./secrets.js";
|
||||
import {
|
||||
CatalogConnectorError,
|
||||
CatalogSchemaCapabilityUnavailableError,
|
||||
type CatalogSyncPhase,
|
||||
type ObservedCatalogColumn,
|
||||
type ObservedCatalogRelationship,
|
||||
type ObservedCatalogTable,
|
||||
type ObservedSchemaSnapshot,
|
||||
type WorkspaceDatabase,
|
||||
} from "./types.js";
|
||||
|
||||
export type CatalogSchemaScanProgress = (
|
||||
phase: Extract<CatalogSyncPhase, "connecting" | "scanning_tables" | "scanning_columns" | "scanning_relationships">,
|
||||
counts?: { tables?: number; columns?: number; relationships?: number },
|
||||
) => Promise<void> | void;
|
||||
|
||||
export interface CatalogSchemaIntrospector {
|
||||
scan(
|
||||
database: WorkspaceDatabase,
|
||||
signal: AbortSignal,
|
||||
progress?: CatalogSchemaScanProgress,
|
||||
): Promise<ObservedSchemaSnapshot>;
|
||||
}
|
||||
|
||||
const identifier = z.string().min(1).max(128);
|
||||
const nullableText = z.string().nullable();
|
||||
const capability = z.enum(["available", "unavailable"]);
|
||||
const restSnapshotSchema = z.object({
|
||||
schemaVersion: z.literal(1),
|
||||
capabilities: z.object({
|
||||
tables: capability,
|
||||
columns: capability,
|
||||
relationships: capability,
|
||||
}).strict(),
|
||||
tables: z.array(z.object({
|
||||
name: identifier,
|
||||
sourceComment: nullableText,
|
||||
}).strict()),
|
||||
columns: z.array(z.object({
|
||||
tableName: identifier,
|
||||
name: identifier,
|
||||
ordinalPosition: z.number().int().positive(),
|
||||
dataType: z.string().min(1).max(2_000),
|
||||
isNullable: z.boolean(),
|
||||
defaultExpression: nullableText,
|
||||
primaryKeyPosition: z.number().int().positive().nullable(),
|
||||
sourceComment: nullableText,
|
||||
}).strict()),
|
||||
relationships: z.array(z.object({
|
||||
constraintName: identifier,
|
||||
sourceTableName: identifier,
|
||||
targetTableName: identifier,
|
||||
updateRule: z.string().min(1).max(64),
|
||||
deleteRule: z.string().min(1).max(64),
|
||||
deferrable: z.boolean(),
|
||||
initiallyDeferred: z.boolean(),
|
||||
columns: z.array(z.object({
|
||||
position: z.number().int().positive(),
|
||||
sourceColumnName: identifier,
|
||||
targetColumnName: identifier,
|
||||
}).strict()).min(1),
|
||||
}).strict()),
|
||||
}).strict();
|
||||
|
||||
function sqlString(value: string): string {
|
||||
return `'${value.replaceAll("'", "''")}'`;
|
||||
}
|
||||
|
||||
function restSnapshotQuery(schemaName: string): string {
|
||||
const schema = sqlString(schemaName);
|
||||
return `WITH target_schema AS (
|
||||
SELECT oid
|
||||
FROM pg_catalog.pg_namespace
|
||||
WHERE nspname = ${schema}
|
||||
),
|
||||
observed_tables AS (
|
||||
SELECT c.oid,
|
||||
c.relname AS name,
|
||||
d.description AS source_comment
|
||||
FROM pg_catalog.pg_class c
|
||||
JOIN target_schema n ON n.oid = c.relnamespace
|
||||
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = 0
|
||||
WHERE c.relkind IN ('r', 'p')
|
||||
),
|
||||
primary_key_columns AS (
|
||||
SELECT i.indrelid AS table_oid,
|
||||
key.attnum,
|
||||
key.ordinality::integer AS position
|
||||
FROM pg_catalog.pg_index i
|
||||
CROSS JOIN LATERAL unnest(i.indkey) WITH ORDINALITY AS key(attnum, ordinality)
|
||||
WHERE i.indisprimary
|
||||
),
|
||||
observed_columns AS (
|
||||
SELECT table_info.name AS table_name,
|
||||
a.attname AS name,
|
||||
a.attnum::integer AS ordinal_position,
|
||||
pg_catalog.format_type(a.atttypid, a.atttypmod) AS data_type,
|
||||
NOT a.attnotnull AS is_nullable,
|
||||
pg_catalog.pg_get_expr(ad.adbin, ad.adrelid) AS default_expression,
|
||||
pk.position AS primary_key_position,
|
||||
d.description AS source_comment
|
||||
FROM observed_tables table_info
|
||||
JOIN pg_catalog.pg_attribute a ON a.attrelid = table_info.oid
|
||||
LEFT JOIN pg_catalog.pg_attrdef ad ON ad.adrelid = table_info.oid AND ad.adnum = a.attnum
|
||||
LEFT JOIN pg_catalog.pg_description d ON d.objoid = table_info.oid AND d.objsubid = a.attnum
|
||||
LEFT JOIN primary_key_columns pk ON pk.table_oid = table_info.oid AND pk.attnum = a.attnum
|
||||
WHERE a.attnum > 0
|
||||
AND NOT a.attisdropped
|
||||
),
|
||||
relationship_pairs AS (
|
||||
SELECT con.oid AS constraint_oid,
|
||||
con.conname AS constraint_name,
|
||||
source_table.relname AS source_table_name,
|
||||
target_table.relname AS target_table_name,
|
||||
CASE con.confupdtype
|
||||
WHEN 'a' THEN 'NO ACTION'
|
||||
WHEN 'r' THEN 'RESTRICT'
|
||||
WHEN 'c' THEN 'CASCADE'
|
||||
WHEN 'n' THEN 'SET NULL'
|
||||
WHEN 'd' THEN 'SET DEFAULT'
|
||||
END AS update_rule,
|
||||
CASE con.confdeltype
|
||||
WHEN 'a' THEN 'NO ACTION'
|
||||
WHEN 'r' THEN 'RESTRICT'
|
||||
WHEN 'c' THEN 'CASCADE'
|
||||
WHEN 'n' THEN 'SET NULL'
|
||||
WHEN 'd' THEN 'SET DEFAULT'
|
||||
END AS delete_rule,
|
||||
con.condeferrable AS is_deferrable,
|
||||
con.condeferred AS initially_deferred,
|
||||
source_key.ordinality::integer AS position,
|
||||
source_column.attname AS source_column_name,
|
||||
target_column.attname AS target_column_name
|
||||
FROM pg_catalog.pg_constraint con
|
||||
JOIN pg_catalog.pg_class source_table ON source_table.oid = con.conrelid
|
||||
JOIN target_schema source_namespace ON source_namespace.oid = source_table.relnamespace
|
||||
JOIN pg_catalog.pg_class target_table ON target_table.oid = con.confrelid
|
||||
JOIN target_schema target_namespace ON target_namespace.oid = target_table.relnamespace
|
||||
JOIN LATERAL unnest(con.conkey) WITH ORDINALITY AS source_key(attnum, ordinality) ON true
|
||||
JOIN LATERAL unnest(con.confkey) WITH ORDINALITY AS target_key(attnum, ordinality)
|
||||
ON target_key.ordinality = source_key.ordinality
|
||||
JOIN pg_catalog.pg_attribute source_column
|
||||
ON source_column.attrelid = source_table.oid AND source_column.attnum = source_key.attnum
|
||||
JOIN pg_catalog.pg_attribute target_column
|
||||
ON target_column.attrelid = target_table.oid AND target_column.attnum = target_key.attnum
|
||||
WHERE con.contype = 'f'
|
||||
),
|
||||
observed_relationships AS (
|
||||
SELECT constraint_oid,
|
||||
constraint_name,
|
||||
source_table_name,
|
||||
target_table_name,
|
||||
update_rule,
|
||||
delete_rule,
|
||||
is_deferrable,
|
||||
initially_deferred,
|
||||
pg_catalog.jsonb_agg(
|
||||
pg_catalog.jsonb_build_object(
|
||||
'position', position,
|
||||
'sourceColumnName', source_column_name,
|
||||
'targetColumnName', target_column_name
|
||||
) ORDER BY position
|
||||
) AS columns
|
||||
FROM relationship_pairs
|
||||
GROUP BY constraint_oid, constraint_name, source_table_name, target_table_name,
|
||||
update_rule, delete_rule, is_deferrable, initially_deferred
|
||||
)
|
||||
SELECT 1 AS "schemaVersion",
|
||||
pg_catalog.jsonb_build_object(
|
||||
'tables', 'available',
|
||||
'columns', 'available',
|
||||
'relationships', 'available'
|
||||
) AS capabilities,
|
||||
COALESCE((
|
||||
SELECT pg_catalog.jsonb_agg(
|
||||
pg_catalog.jsonb_build_object('name', name, 'sourceComment', source_comment)
|
||||
ORDER BY name
|
||||
)
|
||||
FROM observed_tables
|
||||
), '[]'::jsonb) AS tables,
|
||||
COALESCE((
|
||||
SELECT pg_catalog.jsonb_agg(
|
||||
pg_catalog.jsonb_build_object(
|
||||
'tableName', table_name,
|
||||
'name', name,
|
||||
'ordinalPosition', ordinal_position,
|
||||
'dataType', data_type,
|
||||
'isNullable', is_nullable,
|
||||
'defaultExpression', default_expression,
|
||||
'primaryKeyPosition', primary_key_position,
|
||||
'sourceComment', source_comment
|
||||
) ORDER BY table_name, ordinal_position
|
||||
)
|
||||
FROM observed_columns
|
||||
), '[]'::jsonb) AS columns,
|
||||
COALESCE((
|
||||
SELECT pg_catalog.jsonb_agg(
|
||||
pg_catalog.jsonb_build_object(
|
||||
'constraintName', constraint_name,
|
||||
'sourceTableName', source_table_name,
|
||||
'targetTableName', target_table_name,
|
||||
'updateRule', update_rule,
|
||||
'deleteRule', delete_rule,
|
||||
'deferrable', is_deferrable,
|
||||
'initiallyDeferred', initially_deferred,
|
||||
'columns', columns
|
||||
) ORDER BY source_table_name, constraint_name
|
||||
)
|
||||
FROM observed_relationships
|
||||
), '[]'::jsonb) AS relationships
|
||||
FROM target_schema`;
|
||||
}
|
||||
|
||||
function required(value: string | undefined): string {
|
||||
if (!value) throw new CatalogConnectorError("Database binding is incomplete");
|
||||
return value;
|
||||
}
|
||||
|
||||
function textOrNull(value: unknown): string | null {
|
||||
return typeof value === "string" && value.length > 0 ? value : null;
|
||||
}
|
||||
|
||||
function actionRule(value: unknown): string {
|
||||
const rules: Record<string, string> = {
|
||||
a: "NO ACTION",
|
||||
r: "RESTRICT",
|
||||
c: "CASCADE",
|
||||
n: "SET NULL",
|
||||
d: "SET DEFAULT",
|
||||
};
|
||||
const rule = rules[String(value)];
|
||||
if (!rule) throw new CatalogConnectorError("Schema introspection returned an unknown relationship action");
|
||||
return rule;
|
||||
}
|
||||
|
||||
function normalized(snapshot: ObservedSchemaSnapshot): ObservedSchemaSnapshot {
|
||||
const tables = new Map<string, ObservedCatalogTable>();
|
||||
for (const table of snapshot.tables) {
|
||||
if (tables.has(table.name)) throw new CatalogConnectorError("Schema introspection returned duplicate tables");
|
||||
tables.set(table.name, table);
|
||||
}
|
||||
const columns = new Map<string, ObservedCatalogColumn>();
|
||||
for (const column of snapshot.columns) {
|
||||
const key = `${column.tableName}\u0000${column.name}`;
|
||||
if (columns.has(key)) throw new CatalogConnectorError("Schema introspection returned duplicate columns");
|
||||
columns.set(key, column);
|
||||
}
|
||||
const relationships = new Map<string, ObservedCatalogRelationship>();
|
||||
for (const relationship of snapshot.relationships) {
|
||||
const key = `${relationship.sourceTableName}\u0000${relationship.constraintName}`;
|
||||
if (relationships.has(key)) throw new CatalogConnectorError("Schema introspection returned duplicate relationships");
|
||||
relationships.set(key, {
|
||||
...relationship,
|
||||
columns: [...relationship.columns].sort((a, b) => a.position - b.position),
|
||||
});
|
||||
}
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
capabilities: snapshot.capabilities,
|
||||
tables: [...tables.values()].sort((a, b) => a.name.localeCompare(b.name)),
|
||||
columns: [...columns.values()].sort((a, b) => (
|
||||
a.tableName.localeCompare(b.tableName) || a.ordinalPosition - b.ordinalPosition
|
||||
)),
|
||||
relationships: [...relationships.values()].sort((a, b) => (
|
||||
a.sourceTableName.localeCompare(b.sourceTableName) || a.constraintName.localeCompare(b.constraintName)
|
||||
)),
|
||||
};
|
||||
}
|
||||
|
||||
export class ConcreteCatalogSchemaIntrospector implements CatalogSchemaIntrospector {
|
||||
constructor(
|
||||
private readonly postgres: CatalogPostgresAccess,
|
||||
private readonly secretStore: WorkspaceSecretStore,
|
||||
) {}
|
||||
|
||||
async scan(
|
||||
database: WorkspaceDatabase,
|
||||
signal: AbortSignal,
|
||||
progress?: CatalogSchemaScanProgress,
|
||||
): Promise<ObservedSchemaSnapshot> {
|
||||
return database.binding.transport === "rest_api"
|
||||
? await this.scanRest(database, signal, progress)
|
||||
: await this.scanPostgres(database, signal, progress);
|
||||
}
|
||||
|
||||
private async scanPostgres(
|
||||
database: WorkspaceDatabase,
|
||||
signal: AbortSignal,
|
||||
progress?: CatalogSchemaScanProgress,
|
||||
): Promise<ObservedSchemaSnapshot> {
|
||||
await progress?.("connecting");
|
||||
const client = await this.postgres.connect(database, signal);
|
||||
try {
|
||||
const schema = await client.query(
|
||||
"SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname = $1) AS present",
|
||||
[database.schema],
|
||||
);
|
||||
if (schema.rows[0]?.present !== true) throw new CatalogConnectorError("Database schema is unavailable");
|
||||
|
||||
await progress?.("scanning_tables");
|
||||
const tableResult = await client.query(
|
||||
`SELECT c.relname AS name, d.description AS source_comment
|
||||
FROM pg_catalog.pg_class c
|
||||
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
|
||||
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = 0
|
||||
WHERE c.relkind IN ('r', 'p') AND n.nspname = $1
|
||||
ORDER BY c.relname`,
|
||||
[database.schema],
|
||||
);
|
||||
const tables: ObservedCatalogTable[] = tableResult.rows.map((row) => ({
|
||||
name: String(row.name),
|
||||
sourceComment: textOrNull(row.source_comment),
|
||||
}));
|
||||
await progress?.("scanning_tables", { tables: tables.length });
|
||||
|
||||
await progress?.("scanning_columns", { tables: tables.length });
|
||||
const columnResult = await client.query(
|
||||
`SELECT c.relname AS table_name,
|
||||
a.attname AS name,
|
||||
a.attnum::integer AS ordinal_position,
|
||||
pg_catalog.format_type(a.atttypid, a.atttypmod) AS data_type,
|
||||
NOT a.attnotnull AS is_nullable,
|
||||
pg_catalog.pg_get_expr(ad.adbin, ad.adrelid) AS default_expression,
|
||||
pk.position AS primary_key_position,
|
||||
d.description AS source_comment
|
||||
FROM pg_catalog.pg_class c
|
||||
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
|
||||
JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid
|
||||
LEFT JOIN pg_catalog.pg_attrdef ad ON ad.adrelid = c.oid AND ad.adnum = a.attnum
|
||||
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = a.attnum
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT key.ordinality::integer AS position
|
||||
FROM pg_catalog.pg_index i
|
||||
CROSS JOIN LATERAL unnest(i.indkey) WITH ORDINALITY AS key(attnum, ordinality)
|
||||
WHERE i.indrelid = c.oid AND i.indisprimary AND key.attnum = a.attnum
|
||||
LIMIT 1
|
||||
) pk ON true
|
||||
WHERE c.relkind IN ('r', 'p')
|
||||
AND n.nspname = $1
|
||||
AND a.attnum > 0
|
||||
AND NOT a.attisdropped
|
||||
ORDER BY c.relname, a.attnum`,
|
||||
[database.schema],
|
||||
);
|
||||
const columns: ObservedCatalogColumn[] = columnResult.rows.map((row) => ({
|
||||
tableName: String(row.table_name),
|
||||
name: String(row.name),
|
||||
ordinalPosition: Number(row.ordinal_position),
|
||||
dataType: String(row.data_type),
|
||||
isNullable: row.is_nullable === true,
|
||||
defaultExpression: textOrNull(row.default_expression),
|
||||
primaryKeyPosition: row.primary_key_position === null || row.primary_key_position === undefined
|
||||
? null
|
||||
: Number(row.primary_key_position),
|
||||
sourceComment: textOrNull(row.source_comment),
|
||||
}));
|
||||
await progress?.("scanning_columns", { tables: tables.length, columns: columns.length });
|
||||
|
||||
await progress?.("scanning_relationships", { tables: tables.length, columns: columns.length });
|
||||
const relationshipResult = await client.query(
|
||||
`SELECT con.conname AS constraint_name,
|
||||
source_table.relname AS source_table_name,
|
||||
target_table.relname AS target_table_name,
|
||||
con.confupdtype AS update_action,
|
||||
con.confdeltype AS delete_action,
|
||||
con.condeferrable AS deferrable,
|
||||
con.condeferred AS initially_deferred,
|
||||
source_key.ordinality::integer AS position,
|
||||
source_column.attname AS source_column_name,
|
||||
target_column.attname AS target_column_name
|
||||
FROM pg_catalog.pg_constraint con
|
||||
JOIN pg_catalog.pg_class source_table ON source_table.oid = con.conrelid
|
||||
JOIN pg_catalog.pg_namespace source_namespace ON source_namespace.oid = source_table.relnamespace
|
||||
JOIN pg_catalog.pg_class target_table ON target_table.oid = con.confrelid
|
||||
JOIN pg_catalog.pg_namespace target_namespace ON target_namespace.oid = target_table.relnamespace
|
||||
JOIN LATERAL unnest(con.conkey) WITH ORDINALITY AS source_key(attnum, ordinality) ON true
|
||||
JOIN LATERAL unnest(con.confkey) WITH ORDINALITY AS target_key(attnum, ordinality)
|
||||
ON target_key.ordinality = source_key.ordinality
|
||||
JOIN pg_catalog.pg_attribute source_column
|
||||
ON source_column.attrelid = source_table.oid AND source_column.attnum = source_key.attnum
|
||||
JOIN pg_catalog.pg_attribute target_column
|
||||
ON target_column.attrelid = target_table.oid AND target_column.attnum = target_key.attnum
|
||||
WHERE con.contype = 'f'
|
||||
AND source_namespace.nspname = $1
|
||||
AND target_namespace.nspname = $1
|
||||
ORDER BY source_table.relname, con.conname, source_key.ordinality`,
|
||||
[database.schema],
|
||||
);
|
||||
const relationshipMap = new Map<string, ObservedCatalogRelationship>();
|
||||
for (const row of relationshipResult.rows) {
|
||||
const sourceTableName = String(row.source_table_name);
|
||||
const constraintName = String(row.constraint_name);
|
||||
const key = `${sourceTableName}\u0000${constraintName}`;
|
||||
const current = relationshipMap.get(key) ?? {
|
||||
constraintName,
|
||||
sourceTableName,
|
||||
targetTableName: String(row.target_table_name),
|
||||
updateRule: actionRule(row.update_action),
|
||||
deleteRule: actionRule(row.delete_action),
|
||||
deferrable: row.deferrable === true,
|
||||
initiallyDeferred: row.initially_deferred === true,
|
||||
columns: [],
|
||||
};
|
||||
current.columns.push({
|
||||
position: Number(row.position),
|
||||
sourceColumnName: String(row.source_column_name),
|
||||
targetColumnName: String(row.target_column_name),
|
||||
});
|
||||
relationshipMap.set(key, current);
|
||||
}
|
||||
const relationships = [...relationshipMap.values()];
|
||||
await progress?.("scanning_relationships", {
|
||||
tables: tables.length,
|
||||
columns: columns.length,
|
||||
relationships: relationships.length,
|
||||
});
|
||||
return normalized({
|
||||
schemaVersion: 1,
|
||||
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||
tables,
|
||||
columns,
|
||||
relationships,
|
||||
});
|
||||
} finally {
|
||||
await client.end();
|
||||
}
|
||||
}
|
||||
|
||||
private async scanRest(
|
||||
database: WorkspaceDatabase,
|
||||
signal: AbortSignal,
|
||||
progress?: CatalogSchemaScanProgress,
|
||||
): Promise<ObservedSchemaSnapshot> {
|
||||
await progress?.("connecting");
|
||||
const auth = database.binding.restAuth ?? "bearer";
|
||||
const materialized = this.secretStore.materialize(
|
||||
database.workspaceId,
|
||||
auth === "none" ? [] : [CATALOG_SECRET_IDS.apiKey],
|
||||
);
|
||||
try {
|
||||
const headers: Record<string, string> = { "content-type": "application/json" };
|
||||
if (auth !== "none") {
|
||||
const credentialFile = materialized.files.get(CATALOG_SECRET_IDS.apiKey);
|
||||
if (!credentialFile) throw new CatalogConnectorError("REST API key is not configured");
|
||||
const credential = (await readFile(credentialFile, "utf8")).trim();
|
||||
if (auth === "bearer") headers.authorization = `Bearer ${credential}`;
|
||||
else headers["x-api-key"] = credential;
|
||||
}
|
||||
const baseUrl = required(database.binding.baseUrl).replace(/\/+$/, "");
|
||||
const response = await fetch(`${baseUrl}/rpc/schema_snapshot`, {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: JSON.stringify({ schema_name: database.schema }),
|
||||
signal,
|
||||
});
|
||||
let body: unknown;
|
||||
if (response.ok) {
|
||||
body = await response.json();
|
||||
} else if (response.status === 404) {
|
||||
const fallback = await fetch(`${baseUrl}/rpc/run_query`, {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: JSON.stringify({ query_text: restSnapshotQuery(database.schema) }),
|
||||
signal,
|
||||
});
|
||||
if (!fallback.ok) throw new CatalogSchemaCapabilityUnavailableError("schema_snapshot");
|
||||
const rows: unknown = await fallback.json();
|
||||
if (!Array.isArray(rows) || rows.length !== 1) {
|
||||
throw new CatalogConnectorError("REST schema snapshot fallback is invalid");
|
||||
}
|
||||
body = rows[0];
|
||||
} else {
|
||||
throw new CatalogSchemaCapabilityUnavailableError("schema_snapshot");
|
||||
}
|
||||
const parsed = restSnapshotSchema.safeParse(body);
|
||||
if (!parsed.success) throw new CatalogConnectorError("REST schema snapshot is invalid");
|
||||
const snapshot = normalized(parsed.data);
|
||||
await progress?.("scanning_tables", { tables: snapshot.tables.length });
|
||||
await progress?.("scanning_columns", { tables: snapshot.tables.length, columns: snapshot.columns.length });
|
||||
await progress?.("scanning_relationships", {
|
||||
tables: snapshot.tables.length,
|
||||
columns: snapshot.columns.length,
|
||||
relationships: snapshot.relationships.length,
|
||||
});
|
||||
return snapshot;
|
||||
} catch (error) {
|
||||
if (error instanceof CatalogConnectorError) throw error;
|
||||
throw new CatalogConnectorError("REST schema introspection failed");
|
||||
} finally {
|
||||
materialized.release();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
export const CATALOG_SECRET_IDS = {
|
||||
password: "catalog.dwh.password",
|
||||
apiKey: "catalog.dwh.api_key",
|
||||
sshPrivateKey: "catalog.dwh.ssh_private_key",
|
||||
sshPrivateKeyPassphrase: "catalog.dwh.ssh_private_key_passphrase",
|
||||
sshKnownHosts: "catalog.dwh.ssh_known_hosts",
|
||||
tlsCa: "catalog.dwh.tls_ca",
|
||||
} as const;
|
||||
|
||||
export type CatalogSecretName = keyof typeof CATALOG_SECRET_IDS;
|
||||
@@ -0,0 +1,228 @@
|
||||
import { buildInstallationContract } from "../workspaces/contracts.js";
|
||||
import { resolveBinding } from "../workspaces/bindings.js";
|
||||
import type { WorkspaceRegistry } from "../workspaces/registry.js";
|
||||
import type { WorkspaceDescriptor } from "../workspaces/schema.js";
|
||||
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
|
||||
import { createConcreteDiagnosticAdapters } from "../workspaces/diagnostics.js";
|
||||
import { CatalogOperationCoordinator } from "./operation-coordinator.js";
|
||||
import {
|
||||
ConcreteCatalogPostgresAccess,
|
||||
type CatalogPostgresAccess,
|
||||
} from "./postgres-access.js";
|
||||
import type {
|
||||
CatalogRepository,
|
||||
DatabaseBinding,
|
||||
DatabaseConfigurationInput,
|
||||
DatabaseTestResult,
|
||||
WorkspaceDatabase,
|
||||
} from "./types.js";
|
||||
import { CATALOG_SECRET_IDS, type CatalogSecretName } from "./secrets.js";
|
||||
|
||||
export { CATALOG_SECRET_IDS, type CatalogSecretName } from "./secrets.js";
|
||||
|
||||
export interface CatalogListItem extends Omit<WorkspaceDatabase, "id"> {
|
||||
id?: string;
|
||||
workspaceName: string;
|
||||
workspaceDescription?: string;
|
||||
workspaceAvailable: boolean;
|
||||
configured: boolean;
|
||||
secrets: Record<CatalogSecretName, boolean>;
|
||||
}
|
||||
|
||||
function bindingValue(workspace: WorkspaceDescriptor, values: Record<string, string>, suffix: string) {
|
||||
const variable = buildInstallationContract(workspace).variables.find((entry) => (
|
||||
entry.role === "DWH" && entry.suffix === suffix
|
||||
));
|
||||
return variable ? values[variable.name] : undefined;
|
||||
}
|
||||
|
||||
function numeric(value: string | undefined): number | undefined {
|
||||
if (!value) return undefined;
|
||||
const parsed = Number(value);
|
||||
return Number.isInteger(parsed) && parsed >= 1 && parsed <= 65_535 ? parsed : undefined;
|
||||
}
|
||||
|
||||
function yamlBinding(workspace: WorkspaceDescriptor, secretRoots: readonly string[]): DatabaseBinding {
|
||||
const effective = resolveBinding(workspace, "DWH", process.env, secretRoots);
|
||||
const value = (suffix: string) => bindingValue(workspace, effective.values, suffix);
|
||||
return {
|
||||
transport: effective.transport,
|
||||
host: value("HOST"),
|
||||
port: numeric(value("PORT")) ?? workspace.dwh.port,
|
||||
username: value("USER"),
|
||||
baseUrl: value("BASE_URL"),
|
||||
restPath: workspace.diagnostics?.dwh_rest?.path ?? "/health",
|
||||
restAuth: workspace.diagnostics?.dwh_rest?.auth ?? "bearer",
|
||||
tlsServername: value("TLS_SERVERNAME"),
|
||||
sshHost: value("SSH_HOST"),
|
||||
sshPort: numeric(value("SSH_PORT")),
|
||||
sshUsername: value("SSH_USER"),
|
||||
sshTargetHost: value("SSH_TARGET_HOST"),
|
||||
sshTargetPort: numeric(value("SSH_TARGET_PORT")),
|
||||
};
|
||||
}
|
||||
|
||||
function secretState(store: WorkspaceSecretStore, workspaceId: string): Record<CatalogSecretName, boolean> {
|
||||
return Object.fromEntries(Object.entries(CATALOG_SECRET_IDS).map(([name, id]) => (
|
||||
[name, store.has(workspaceId, id)]
|
||||
))) as Record<CatalogSecretName, boolean>;
|
||||
}
|
||||
|
||||
export class CatalogService {
|
||||
private readonly adapters = createConcreteDiagnosticAdapters();
|
||||
|
||||
constructor(
|
||||
private readonly repository: CatalogRepository,
|
||||
private readonly registry: WorkspaceRegistry,
|
||||
private readonly secretStore: WorkspaceSecretStore,
|
||||
private readonly secretRoots: readonly string[],
|
||||
private readonly diagnosticTimeoutMs: number,
|
||||
private readonly postgres: CatalogPostgresAccess = new ConcreteCatalogPostgresAccess(secretStore, {
|
||||
connectTimeoutMs: diagnosticTimeoutMs,
|
||||
}),
|
||||
private readonly operations: CatalogOperationCoordinator = new CatalogOperationCoordinator(),
|
||||
) {}
|
||||
|
||||
async list(): Promise<CatalogListItem[]> {
|
||||
const [workspaces, configured] = await Promise.all([
|
||||
this.registry.listCatalog(),
|
||||
this.repository.list(),
|
||||
]);
|
||||
const byWorkspace = new Map(configured.map((database) => [database.workspaceId, database]));
|
||||
const active = await Promise.all(workspaces.map(async (entry) => {
|
||||
const database = byWorkspace.get(entry.id);
|
||||
const { workspace } = await this.registry.read(entry.id);
|
||||
const base = database ?? {
|
||||
workspaceId: entry.id,
|
||||
engine: "postgres" as const,
|
||||
databaseName: workspace.dwh.database,
|
||||
schema: workspace.dwh.schema,
|
||||
version: 0,
|
||||
createdAt: "",
|
||||
updatedAt: "",
|
||||
binding: yamlBinding(workspace, this.secretRoots),
|
||||
connectionStatus: "untested" as const,
|
||||
};
|
||||
return {
|
||||
...base,
|
||||
workspaceName: entry.name,
|
||||
workspaceDescription: entry.description,
|
||||
workspaceAvailable: true,
|
||||
configured: database !== undefined,
|
||||
secrets: secretState(this.secretStore, entry.id),
|
||||
};
|
||||
}));
|
||||
const known = new Set(workspaces.map((entry) => entry.id));
|
||||
const orphaned: CatalogListItem[] = configured
|
||||
.filter((database) => !known.has(database.workspaceId))
|
||||
.map((database) => ({
|
||||
...database,
|
||||
workspaceName: database.workspaceId,
|
||||
workspaceDescription: "Workspace is no longer present in the repository catalog.",
|
||||
workspaceAvailable: false,
|
||||
configured: true,
|
||||
secrets: secretState(this.secretStore, database.workspaceId),
|
||||
}));
|
||||
return [...active, ...orphaned];
|
||||
}
|
||||
|
||||
async ensureWorkspace(workspaceId: string): Promise<WorkspaceDescriptor> {
|
||||
const { workspace } = await this.registry.read(workspaceId);
|
||||
return workspace;
|
||||
}
|
||||
|
||||
async normalizeInput(input: DatabaseConfigurationInput): Promise<DatabaseConfigurationInput> {
|
||||
const workspace = await this.ensureWorkspace(input.workspaceId);
|
||||
if (input.binding.transport !== "rest_api") return input;
|
||||
return {
|
||||
...input,
|
||||
binding: {
|
||||
...input.binding,
|
||||
restPath: workspace.diagnostics?.dwh_rest?.path ?? "/health",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
configuredSecrets(workspaceId: string): Record<CatalogSecretName, boolean> {
|
||||
return secretState(this.secretStore, workspaceId);
|
||||
}
|
||||
|
||||
replaceSecrets(workspaceId: string, values: Partial<Record<CatalogSecretName, string>>): void {
|
||||
const encoded: Record<string, string> = {};
|
||||
for (const [name, value] of Object.entries(values) as Array<[CatalogSecretName, string | undefined]>) {
|
||||
if (value !== undefined && value.length > 0) encoded[CATALOG_SECRET_IDS[name]] = value;
|
||||
}
|
||||
if (Object.keys(encoded).length > 0) this.secretStore.putMany(workspaceId, encoded);
|
||||
}
|
||||
|
||||
forgetSecrets(workspaceId: string): void {
|
||||
for (const id of Object.values(CATALOG_SECRET_IDS)) this.secretStore.forget(workspaceId, id);
|
||||
}
|
||||
|
||||
async test(database: WorkspaceDatabase): Promise<DatabaseTestResult> {
|
||||
return await this.operations.run(database.id, async () => {
|
||||
const testedAt = new Date().toISOString();
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), this.diagnosticTimeoutMs);
|
||||
const required = database.binding.transport === "rest_api"
|
||||
? database.binding.restAuth === "none" ? [] : [CATALOG_SECRET_IDS.apiKey]
|
||||
: [];
|
||||
const materialized = this.secretStore.materialize(database.workspaceId, required);
|
||||
try {
|
||||
if (database.binding.transport !== "rest_api") {
|
||||
const client = await this.postgres.connect(database, controller.signal);
|
||||
try {
|
||||
const result = await client.query(
|
||||
`SELECT current_database() AS database,
|
||||
CASE WHEN pg_catalog.has_schema_privilege(
|
||||
current_user,
|
||||
(SELECT oid FROM pg_catalog.pg_namespace WHERE nspname = $1),
|
||||
'USAGE'
|
||||
) THEN $1 ELSE NULL END AS schema`,
|
||||
[database.schema],
|
||||
);
|
||||
const row = result.rows[0];
|
||||
if (row?.database !== database.databaseName || row.schema !== database.schema) {
|
||||
throw new Error("Database identity mismatch");
|
||||
}
|
||||
} finally {
|
||||
await client.end();
|
||||
}
|
||||
} else {
|
||||
const credentialId = CATALOG_SECRET_IDS.apiKey;
|
||||
await this.adapters.probeConnector({
|
||||
role: "dwh",
|
||||
transport: database.binding.transport,
|
||||
baseUrl: database.binding.baseUrl,
|
||||
credentialFile: materialized.files.get(credentialId),
|
||||
resource: { database: database.databaseName, schema: database.schema },
|
||||
timeoutMs: this.diagnosticTimeoutMs,
|
||||
signal: controller.signal,
|
||||
diagnostic: {
|
||||
method: "GET" as const,
|
||||
path: database.binding.restPath ?? "/health",
|
||||
auth: database.binding.restAuth ?? "bearer",
|
||||
},
|
||||
});
|
||||
}
|
||||
return {
|
||||
connectionStatus: "reachable",
|
||||
testedVersion: database.version,
|
||||
lastTestedAt: testedAt,
|
||||
};
|
||||
} catch {
|
||||
return {
|
||||
connectionStatus: "failed",
|
||||
testedVersion: database.version,
|
||||
lastTestedAt: testedAt,
|
||||
errorCode: "connector_unavailable",
|
||||
errorMessage: "The database connector could not be reached or authenticated.",
|
||||
};
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
controller.abort();
|
||||
materialized.release();
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,311 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import type { CatalogOperationCoordinator } from "./operation-coordinator.js";
|
||||
import type { CatalogSchemaIntrospector, CatalogSchemaScanProgress } from "./schema-introspector.js";
|
||||
import {
|
||||
CatalogConflictError,
|
||||
CatalogConnectorError,
|
||||
CatalogSchemaCapabilityUnavailableError,
|
||||
type CatalogRepository,
|
||||
type CatalogSchemaDiff,
|
||||
type CatalogSyncCounts,
|
||||
type CatalogSyncRun,
|
||||
type CatalogSyncScope,
|
||||
type ObservedSchemaSnapshot,
|
||||
type WorkspaceDatabase,
|
||||
} from "./types.js";
|
||||
|
||||
class SyncCancelledError extends Error {}
|
||||
|
||||
const TERMINAL_STATES = new Set<CatalogSyncRun["state"]>([
|
||||
"succeeded", "failed", "cancelled", "interrupted",
|
||||
]);
|
||||
|
||||
function destructive(diff: CatalogSchemaDiff): boolean {
|
||||
return diff.deletedTables.length > 0
|
||||
|| diff.deletedColumns.length > 0
|
||||
|| diff.deletedRelationships.length > 0;
|
||||
}
|
||||
|
||||
function fingerprint(snapshot: ObservedSchemaSnapshot): string {
|
||||
return JSON.stringify(snapshot);
|
||||
}
|
||||
|
||||
function safeFailure(error: unknown): { code: string; message: string } {
|
||||
if (error instanceof CatalogSchemaCapabilityUnavailableError) {
|
||||
const label = error.capability === "schema_snapshot" ? "schema snapshot" : error.capability;
|
||||
return {
|
||||
code: "schema_capability_unavailable",
|
||||
message: `This database binding does not provide the ${label} capability.`,
|
||||
};
|
||||
}
|
||||
if (error instanceof CatalogConnectorError) {
|
||||
return { code: "schema_introspection_failed", message: "The database schema could not be read safely." };
|
||||
}
|
||||
return { code: "schema_sync_failed", message: "Schema synchronization failed." };
|
||||
}
|
||||
|
||||
export class CatalogSyncWorker {
|
||||
private readonly workerId = randomUUID();
|
||||
private readonly controllers = new Map<string, AbortController>();
|
||||
private readonly reservations = new Map<string, () => void>();
|
||||
private stopping = false;
|
||||
|
||||
constructor(
|
||||
private readonly repository: CatalogRepository,
|
||||
private readonly introspector: CatalogSchemaIntrospector,
|
||||
private readonly operations: CatalogOperationCoordinator,
|
||||
private readonly timeoutMs: number,
|
||||
) {}
|
||||
|
||||
async initialize(): Promise<void> {
|
||||
if (!(await this.repository.available())) return;
|
||||
await this.repository.interruptActiveSyncRuns();
|
||||
await this.repository.pruneSyncEvents(new Date(Date.now() - 30 * 24 * 60 * 60 * 1_000).toISOString());
|
||||
}
|
||||
|
||||
async start(database: WorkspaceDatabase, scope: CatalogSyncScope, tableIds: readonly string[]): Promise<CatalogSyncRun> {
|
||||
this.assertReady(database);
|
||||
const uniqueTableIds = [...new Set(tableIds)];
|
||||
if (scope === "columns") {
|
||||
const tables = await Promise.all(uniqueTableIds.map((tableId) => this.repository.getTable(database.id, tableId)));
|
||||
if (tables.some((table) => !table)) throw new CatalogConflictError("One or more selected tables no longer exist");
|
||||
}
|
||||
if (scope !== "columns" && uniqueTableIds.length > 0) {
|
||||
throw new CatalogConflictError("Table selection is only valid for a column synchronization");
|
||||
}
|
||||
const release = this.operations.reserve(database.id);
|
||||
try {
|
||||
const run = await this.repository.createSyncRun(database.id, scope, uniqueTableIds, database.version);
|
||||
this.reservations.set(run.id, release);
|
||||
await this.repository.appendSyncEvent(run.id, "info", "queued", "Synchronization queued.", { scope });
|
||||
this.launch(run.id);
|
||||
return run;
|
||||
} catch (error) {
|
||||
release();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async confirm(runId: string, confirmationToken: string): Promise<CatalogSyncRun | undefined> {
|
||||
const run = await this.repository.getSyncRun(runId);
|
||||
if (!run) return undefined;
|
||||
if (run.state !== "awaiting_confirmation" || !run.observedSnapshot || run.confirmationToken !== confirmationToken) {
|
||||
throw new CatalogConflictError("Synchronization confirmation is no longer valid");
|
||||
}
|
||||
await this.repository.appendSyncEvent(run.id, "info", "confirmation_received", "Destructive changes were confirmed.");
|
||||
const queued = await this.repository.updateSyncRun(run.id, {
|
||||
state: "queued",
|
||||
phase: "queued",
|
||||
confirmationToken: null,
|
||||
leaseOwner: null,
|
||||
leaseExpiresAt: null,
|
||||
});
|
||||
this.launch(run.id, fingerprint(run.observedSnapshot));
|
||||
return queued;
|
||||
}
|
||||
|
||||
async cancel(runId: string): Promise<CatalogSyncRun | undefined> {
|
||||
const run = await this.repository.getSyncRun(runId);
|
||||
if (!run) return undefined;
|
||||
if (run.state === "applying" || TERMINAL_STATES.has(run.state)) return run;
|
||||
await this.repository.requestSyncRunCancellation(runId);
|
||||
this.controllers.get(runId)?.abort();
|
||||
if (run.state === "queued" || run.state === "awaiting_confirmation") {
|
||||
const cancelled = await this.repository.updateSyncRun(runId, {
|
||||
state: "cancelled",
|
||||
phase: "completed",
|
||||
finishedAt: new Date().toISOString(),
|
||||
observedSnapshot: null,
|
||||
plannedDiff: null,
|
||||
confirmationToken: null,
|
||||
leaseOwner: null,
|
||||
leaseExpiresAt: null,
|
||||
});
|
||||
await this.repository.appendSyncEvent(runId, "warning", "cancelled", "Synchronization cancelled.");
|
||||
this.release(runId);
|
||||
return cancelled;
|
||||
}
|
||||
return await this.repository.getSyncRun(runId);
|
||||
}
|
||||
|
||||
async retry(runId: string): Promise<CatalogSyncRun | undefined> {
|
||||
const previous = await this.repository.getSyncRun(runId);
|
||||
if (!previous) return undefined;
|
||||
if (!TERMINAL_STATES.has(previous.state)) {
|
||||
throw new CatalogConflictError("Only a finished synchronization can be retried");
|
||||
}
|
||||
const database = await this.repository.get(previous.databaseId);
|
||||
if (!database) return undefined;
|
||||
return await this.start(database, previous.scope, previous.tableIds);
|
||||
}
|
||||
|
||||
async stop(): Promise<void> {
|
||||
this.stopping = true;
|
||||
for (const controller of this.controllers.values()) controller.abort();
|
||||
if (await this.repository.available()) await this.repository.interruptActiveSyncRuns();
|
||||
for (const runId of [...this.reservations.keys()]) this.release(runId);
|
||||
}
|
||||
|
||||
private launch(runId: string, confirmedFingerprint?: string): void {
|
||||
queueMicrotask(() => {
|
||||
void this.execute(runId, confirmedFingerprint).catch(() => undefined);
|
||||
});
|
||||
}
|
||||
|
||||
private async execute(runId: string, confirmedFingerprint?: string): Promise<void> {
|
||||
if (this.stopping) return;
|
||||
const leaseExpiresAt = new Date(Date.now() + 20_000).toISOString();
|
||||
const claimed = await this.repository.claimSyncRun(runId, this.workerId, leaseExpiresAt);
|
||||
if (!claimed) return;
|
||||
const controller = new AbortController();
|
||||
this.controllers.set(runId, controller);
|
||||
let timedOut = false;
|
||||
const timeout = setTimeout(() => {
|
||||
timedOut = true;
|
||||
controller.abort();
|
||||
}, this.timeoutMs);
|
||||
const heartbeat = setInterval(() => {
|
||||
void this.repository.updateSyncRun(runId, {
|
||||
heartbeatAt: new Date().toISOString(),
|
||||
leaseExpiresAt: new Date(Date.now() + 20_000).toISOString(),
|
||||
});
|
||||
}, 5_000);
|
||||
|
||||
try {
|
||||
await this.repository.appendSyncEvent(runId, "info", "started", "Synchronization started.");
|
||||
const database = await this.repository.get(claimed.databaseId);
|
||||
if (!database || database.version !== claimed.requestedDatabaseVersion) {
|
||||
throw new CatalogConflictError("Database binding changed before synchronization started");
|
||||
}
|
||||
this.assertReady(database);
|
||||
const progress: CatalogSchemaScanProgress = async (phase, counts) => {
|
||||
await this.checkCancelled(runId);
|
||||
await this.repository.updateSyncRun(runId, {
|
||||
phase,
|
||||
heartbeatAt: new Date().toISOString(),
|
||||
...(counts ? { counts } : {}),
|
||||
});
|
||||
await this.repository.appendSyncEvent(runId, "info", phase, this.phaseMessage(phase), counts ?? {});
|
||||
};
|
||||
const snapshot = await this.introspector.scan(database, controller.signal, progress);
|
||||
await this.checkCancelled(runId);
|
||||
this.assertCapability(claimed.scope, snapshot);
|
||||
const counts: CatalogSyncCounts = {
|
||||
tables: snapshot.tables.length,
|
||||
columns: snapshot.columns.length,
|
||||
relationships: snapshot.relationships.length,
|
||||
};
|
||||
await this.repository.updateSyncRun(runId, { phase: "planning", counts, observedSnapshot: snapshot });
|
||||
await this.repository.appendSyncEvent(runId, "info", "planning", "Schema changes are being planned.", { ...counts });
|
||||
const diff = await this.repository.planSchemaSync(claimed.databaseId, claimed.scope, claimed.tableIds, snapshot);
|
||||
await this.checkCancelled(runId);
|
||||
if (destructive(diff) && fingerprint(snapshot) !== confirmedFingerprint) {
|
||||
const token = randomUUID();
|
||||
const waiting = await this.repository.updateSyncRun(runId, {
|
||||
state: "awaiting_confirmation",
|
||||
phase: "awaiting_confirmation",
|
||||
observedSnapshot: snapshot,
|
||||
plannedDiff: diff,
|
||||
confirmationToken: token,
|
||||
counts,
|
||||
leaseOwner: null,
|
||||
leaseExpiresAt: null,
|
||||
});
|
||||
await this.repository.appendSyncEvent(runId, "warning", "confirmation_required", "Confirmation is required before removing catalog objects.", {
|
||||
deletedTables: diff.deletedTables.length,
|
||||
deletedColumns: diff.deletedColumns.length,
|
||||
deletedRelationships: diff.deletedRelationships.length,
|
||||
});
|
||||
if (!waiting) throw new Error("Synchronization run disappeared");
|
||||
return;
|
||||
}
|
||||
|
||||
await this.repository.updateSyncRun(runId, { state: "applying", phase: "applying", plannedDiff: diff });
|
||||
await this.repository.appendSyncEvent(runId, "info", "applying", "Catalog changes are being applied atomically.");
|
||||
this.controllers.delete(runId);
|
||||
const applied = await this.repository.applySchemaSync(
|
||||
claimed.databaseId,
|
||||
claimed.requestedDatabaseVersion,
|
||||
claimed.scope,
|
||||
claimed.tableIds,
|
||||
snapshot,
|
||||
);
|
||||
if (!applied) throw new CatalogConflictError("Database binding changed before schema changes were applied");
|
||||
await this.repository.updateSyncRun(runId, {
|
||||
state: "succeeded",
|
||||
phase: "completed",
|
||||
counts: applied,
|
||||
finishedAt: new Date().toISOString(),
|
||||
observedSnapshot: null,
|
||||
plannedDiff: null,
|
||||
confirmationToken: null,
|
||||
heartbeatAt: new Date().toISOString(),
|
||||
leaseOwner: null,
|
||||
leaseExpiresAt: null,
|
||||
});
|
||||
await this.repository.appendSyncEvent(runId, "info", "succeeded", "Synchronization completed.", { ...applied });
|
||||
this.release(runId);
|
||||
} catch (error) {
|
||||
const current = await this.repository.getSyncRun(runId);
|
||||
const cancelled = !timedOut && (error instanceof SyncCancelledError || controller.signal.aborted || current?.cancelRequested);
|
||||
const failure = timedOut
|
||||
? { code: "schema_sync_timed_out", message: "Schema synchronization timed out." }
|
||||
: safeFailure(error);
|
||||
await this.repository.updateSyncRun(runId, {
|
||||
state: cancelled ? "cancelled" : "failed",
|
||||
phase: "completed",
|
||||
errorCode: cancelled ? null : failure.code,
|
||||
errorMessage: cancelled ? null : failure.message,
|
||||
finishedAt: new Date().toISOString(),
|
||||
observedSnapshot: null,
|
||||
plannedDiff: null,
|
||||
confirmationToken: null,
|
||||
leaseOwner: null,
|
||||
leaseExpiresAt: null,
|
||||
});
|
||||
await this.repository.appendSyncEvent(
|
||||
runId,
|
||||
cancelled ? "warning" : "error",
|
||||
cancelled ? "cancelled" : "failed",
|
||||
cancelled ? "Synchronization cancelled." : failure.message,
|
||||
);
|
||||
this.release(runId);
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
clearInterval(heartbeat);
|
||||
this.controllers.delete(runId);
|
||||
}
|
||||
}
|
||||
|
||||
private assertReady(database: WorkspaceDatabase): void {
|
||||
if (database.connectionStatus !== "reachable" || database.testedVersion !== database.version) {
|
||||
throw new CatalogConflictError("Test the current database binding before synchronizing its schema");
|
||||
}
|
||||
}
|
||||
|
||||
private assertCapability(scope: CatalogSyncScope, snapshot: ObservedSchemaSnapshot): void {
|
||||
const required = scope === "all" ? ["tables", "columns", "relationships"] as const : [scope] as const;
|
||||
for (const name of required) {
|
||||
if (snapshot.capabilities[name] !== "available") {
|
||||
throw new CatalogSchemaCapabilityUnavailableError(name);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async checkCancelled(runId: string): Promise<void> {
|
||||
const run = await this.repository.getSyncRun(runId);
|
||||
if (run?.cancelRequested) throw new SyncCancelledError("Synchronization cancelled");
|
||||
}
|
||||
|
||||
private release(runId: string): void {
|
||||
this.reservations.get(runId)?.();
|
||||
this.reservations.delete(runId);
|
||||
}
|
||||
|
||||
private phaseMessage(phase: Parameters<CatalogSchemaScanProgress>[0]): string {
|
||||
if (phase === "connecting") return "Connecting to the database.";
|
||||
if (phase === "scanning_tables") return "Reading tables.";
|
||||
if (phase === "scanning_columns") return "Reading columns and primary keys.";
|
||||
return "Reading foreign-key relationships.";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,133 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
|
||||
import type { CatalogPostgresAccess } from "./postgres-access.js";
|
||||
import { CATALOG_SECRET_IDS } from "./secrets.js";
|
||||
import {
|
||||
CatalogConnectorError,
|
||||
type ObservedCatalogTable,
|
||||
type WorkspaceDatabase,
|
||||
} from "./types.js";
|
||||
|
||||
export interface CatalogTableIntrospector {
|
||||
scan(database: WorkspaceDatabase, signal: AbortSignal): Promise<ObservedCatalogTable[]>;
|
||||
}
|
||||
|
||||
function normalize(rows: readonly ObservedCatalogTable[]): ObservedCatalogTable[] {
|
||||
const byName = new Map<string, ObservedCatalogTable>();
|
||||
for (const row of rows) {
|
||||
if (typeof row.name !== "string" || row.name.length === 0 || row.name.length > 128) {
|
||||
throw new CatalogConnectorError("Schema introspection response is invalid");
|
||||
}
|
||||
if (row.sourceComment !== null && typeof row.sourceComment !== "string") {
|
||||
throw new CatalogConnectorError("Schema introspection response is invalid");
|
||||
}
|
||||
byName.set(row.name, row);
|
||||
}
|
||||
return [...byName.values()]
|
||||
.sort((left, right) => left.name.localeCompare(right.name));
|
||||
}
|
||||
|
||||
function requireText(value: string | undefined): string {
|
||||
if (!value) throw new CatalogConnectorError("Database binding is incomplete");
|
||||
return value;
|
||||
}
|
||||
|
||||
export class ConcreteCatalogTableIntrospector implements CatalogTableIntrospector {
|
||||
constructor(
|
||||
private readonly postgres: CatalogPostgresAccess,
|
||||
private readonly secretStore: WorkspaceSecretStore,
|
||||
) {}
|
||||
|
||||
async scan(database: WorkspaceDatabase, signal: AbortSignal): Promise<ObservedCatalogTable[]> {
|
||||
return database.binding.transport === "rest_api"
|
||||
? await this.scanRest(database, signal)
|
||||
: await this.scanPostgres(database, signal);
|
||||
}
|
||||
|
||||
private async scanPostgres(
|
||||
database: WorkspaceDatabase,
|
||||
signal: AbortSignal,
|
||||
): Promise<ObservedCatalogTable[]> {
|
||||
const client = await this.postgres.connect(database, signal);
|
||||
try {
|
||||
const schema = await client.query(
|
||||
"SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname = $1) AS present",
|
||||
[database.schema],
|
||||
);
|
||||
if (schema.rows[0]?.present !== true) throw new CatalogConnectorError("Database schema is unavailable");
|
||||
const result = await client.query(
|
||||
`SELECT c.relname AS name, d.description AS source_comment
|
||||
FROM pg_catalog.pg_class c
|
||||
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
|
||||
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = 0
|
||||
WHERE c.relkind IN ('r', 'p') AND n.nspname = $1
|
||||
ORDER BY c.relname`,
|
||||
[database.schema],
|
||||
);
|
||||
return normalize(result.rows.map((row) => ({
|
||||
name: String(row.name),
|
||||
sourceComment: typeof row.source_comment === "string" && row.source_comment.length > 0
|
||||
? row.source_comment
|
||||
: null,
|
||||
})));
|
||||
} finally {
|
||||
await client.end();
|
||||
}
|
||||
}
|
||||
|
||||
private async scanRest(
|
||||
database: WorkspaceDatabase,
|
||||
signal: AbortSignal,
|
||||
): Promise<ObservedCatalogTable[]> {
|
||||
const auth = database.binding.restAuth ?? "bearer";
|
||||
const required = auth === "none" ? [] : [CATALOG_SECRET_IDS.apiKey];
|
||||
const materialized = this.secretStore.materialize(database.workspaceId, required);
|
||||
try {
|
||||
const headers: Record<string, string> = { "content-type": "application/json" };
|
||||
if (auth !== "none") {
|
||||
const credentialFile = materialized.files.get(CATALOG_SECRET_IDS.apiKey);
|
||||
if (!credentialFile) throw new CatalogConnectorError("REST API key is not configured");
|
||||
const credential = (await readFile(credentialFile, "utf8")).trim();
|
||||
if (auth === "bearer") headers.authorization = `Bearer ${credential}`;
|
||||
else headers["x-api-key"] = credential;
|
||||
}
|
||||
const baseUrl = requireText(database.binding.baseUrl).replace(/\/+$/, "");
|
||||
const response = await fetch(`${baseUrl}/rpc/list_tables`, {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: JSON.stringify({ schema_name: database.schema }),
|
||||
signal,
|
||||
});
|
||||
if (!response.ok) throw new CatalogConnectorError("REST schema introspection failed");
|
||||
const body: unknown = await response.json();
|
||||
if (!Array.isArray(body)) throw new CatalogConnectorError("REST schema response is invalid");
|
||||
const rows: ObservedCatalogTable[] = [];
|
||||
for (const item of body) {
|
||||
if (!item || typeof item !== "object") {
|
||||
throw new CatalogConnectorError("REST schema response is invalid");
|
||||
}
|
||||
const row = item as Record<string, unknown>;
|
||||
if (typeof row.type !== "string") {
|
||||
throw new CatalogConnectorError("REST schema response is invalid");
|
||||
}
|
||||
if (row.type !== "TABLE") continue;
|
||||
if (typeof row.table !== "string" || row.table.length === 0 || row.table.length > 128) {
|
||||
throw new CatalogConnectorError("REST schema response is invalid");
|
||||
}
|
||||
if (row.comment !== undefined && row.comment !== null && typeof row.comment !== "string") {
|
||||
throw new CatalogConnectorError("REST schema response is invalid");
|
||||
}
|
||||
rows.push({
|
||||
name: row.table,
|
||||
sourceComment: typeof row.comment === "string" && row.comment.length > 0 ? row.comment : null,
|
||||
});
|
||||
}
|
||||
return normalize(rows);
|
||||
} catch (error) {
|
||||
if (error instanceof CatalogConnectorError) throw error;
|
||||
throw new CatalogConnectorError("REST schema introspection failed");
|
||||
} finally {
|
||||
materialized.release();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
import type {
|
||||
CatalogRepository,
|
||||
CatalogTable,
|
||||
} from "./types.js";
|
||||
|
||||
export class CatalogTableService {
|
||||
constructor(
|
||||
private readonly repository: CatalogRepository,
|
||||
) {}
|
||||
|
||||
async list(databaseId: string): Promise<CatalogTable[]> {
|
||||
return await this.repository.listTables(databaseId);
|
||||
}
|
||||
|
||||
async updateDescription(
|
||||
databaseId: string,
|
||||
tableId: string,
|
||||
expectedVersion: number,
|
||||
description: string | null,
|
||||
): Promise<CatalogTable | undefined> {
|
||||
const normalized = description?.trim() || null;
|
||||
return await this.repository.updateTableDescription(
|
||||
databaseId,
|
||||
tableId,
|
||||
expectedVersion,
|
||||
normalized,
|
||||
);
|
||||
}
|
||||
|
||||
async updateMetadata(
|
||||
databaseId: string,
|
||||
tableId: string,
|
||||
expectedVersion: number,
|
||||
description: string | null,
|
||||
generatedDescription: string | null,
|
||||
): Promise<CatalogTable | undefined> {
|
||||
return await this.repository.updateTableMetadata(
|
||||
databaseId,
|
||||
tableId,
|
||||
expectedVersion,
|
||||
description?.trim() || null,
|
||||
generatedDescription?.trim() || null,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,339 @@
|
||||
export const DATABASE_TRANSPORTS = ["postgres_direct", "rest_api", "ssh_tunnel"] as const;
|
||||
export type DatabaseTransport = (typeof DATABASE_TRANSPORTS)[number];
|
||||
|
||||
export type ConnectionStatus = "untested" | "reachable" | "failed";
|
||||
|
||||
export interface DatabaseBinding {
|
||||
transport: DatabaseTransport;
|
||||
host?: string;
|
||||
port?: number;
|
||||
username?: string;
|
||||
baseUrl?: string;
|
||||
restPath?: string;
|
||||
restAuth?: "none" | "bearer" | "x-api-key";
|
||||
tlsServername?: string;
|
||||
sshHost?: string;
|
||||
sshPort?: number;
|
||||
sshUsername?: string;
|
||||
sshTargetHost?: string;
|
||||
sshTargetPort?: number;
|
||||
}
|
||||
|
||||
export interface WorkspaceDatabase {
|
||||
id: string;
|
||||
workspaceId: string;
|
||||
engine: "postgres";
|
||||
databaseName: string;
|
||||
schema: string;
|
||||
version: number;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
binding: DatabaseBinding;
|
||||
connectionStatus: ConnectionStatus;
|
||||
testedVersion?: number;
|
||||
lastTestedAt?: string;
|
||||
lastErrorCode?: string;
|
||||
lastErrorMessage?: string;
|
||||
schemaSyncedVersion?: number;
|
||||
schemaSyncedAt?: string;
|
||||
}
|
||||
|
||||
export interface DatabaseConfigurationInput {
|
||||
workspaceId: string;
|
||||
engine: "postgres";
|
||||
databaseName: string;
|
||||
schema: string;
|
||||
binding: DatabaseBinding;
|
||||
}
|
||||
|
||||
export interface DatabaseTestResult {
|
||||
connectionStatus: Exclude<ConnectionStatus, "untested">;
|
||||
testedVersion: number;
|
||||
lastTestedAt: string;
|
||||
errorCode?: string;
|
||||
errorMessage?: string;
|
||||
}
|
||||
|
||||
export interface CatalogTable {
|
||||
id: string;
|
||||
databaseId: string;
|
||||
name: string;
|
||||
sourceComment: string | null;
|
||||
description: string | null;
|
||||
generatedDescription: string | null;
|
||||
lastSyncedDatabaseVersion: number | null;
|
||||
lastSyncedAt: string | null;
|
||||
version: number;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export interface ObservedCatalogTable {
|
||||
name: string;
|
||||
sourceComment: string | null;
|
||||
}
|
||||
|
||||
export interface CatalogColumn {
|
||||
id: string;
|
||||
tableId: string;
|
||||
name: string;
|
||||
ordinalPosition: number;
|
||||
dataType: string;
|
||||
isNullable: boolean;
|
||||
defaultExpression: string | null;
|
||||
primaryKeyPosition: number | null;
|
||||
isPrimaryKey: boolean;
|
||||
isForeignKey: boolean;
|
||||
foreignKeyCount: number;
|
||||
sourceComment: string | null;
|
||||
description: string | null;
|
||||
generatedDescription: string | null;
|
||||
lastSyncedDatabaseVersion: number | null;
|
||||
lastSyncedAt: string | null;
|
||||
version: number;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export interface ObservedCatalogColumn {
|
||||
tableName: string;
|
||||
name: string;
|
||||
ordinalPosition: number;
|
||||
dataType: string;
|
||||
isNullable: boolean;
|
||||
defaultExpression: string | null;
|
||||
primaryKeyPosition: number | null;
|
||||
sourceComment: string | null;
|
||||
}
|
||||
|
||||
export interface CatalogRelationshipColumn {
|
||||
position: number;
|
||||
sourceColumnId: string;
|
||||
sourceColumnName: string;
|
||||
targetColumnId: string;
|
||||
targetColumnName: string;
|
||||
}
|
||||
|
||||
export interface CatalogRelationship {
|
||||
id: string;
|
||||
databaseId: string;
|
||||
constraintName: string;
|
||||
sourceTableId: string;
|
||||
sourceTableName: string;
|
||||
targetTableId: string;
|
||||
targetTableName: string;
|
||||
updateRule: string;
|
||||
deleteRule: string;
|
||||
deferrable: boolean;
|
||||
initiallyDeferred: boolean;
|
||||
columns: CatalogRelationshipColumn[];
|
||||
lastSyncedDatabaseVersion: number | null;
|
||||
lastSyncedAt: string | null;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export interface ObservedRelationshipColumn {
|
||||
position: number;
|
||||
sourceColumnName: string;
|
||||
targetColumnName: string;
|
||||
}
|
||||
|
||||
export interface ObservedCatalogRelationship {
|
||||
constraintName: string;
|
||||
sourceTableName: string;
|
||||
targetTableName: string;
|
||||
updateRule: string;
|
||||
deleteRule: string;
|
||||
deferrable: boolean;
|
||||
initiallyDeferred: boolean;
|
||||
columns: ObservedRelationshipColumn[];
|
||||
}
|
||||
|
||||
export type IntrospectionCapabilityState = "available" | "unavailable";
|
||||
export interface ObservedSchemaSnapshot {
|
||||
schemaVersion: 1;
|
||||
capabilities: {
|
||||
tables: IntrospectionCapabilityState;
|
||||
columns: IntrospectionCapabilityState;
|
||||
relationships: IntrospectionCapabilityState;
|
||||
};
|
||||
tables: ObservedCatalogTable[];
|
||||
columns: ObservedCatalogColumn[];
|
||||
relationships: ObservedCatalogRelationship[];
|
||||
}
|
||||
|
||||
export type CatalogSyncScope = "tables" | "columns" | "relationships" | "all";
|
||||
export type CatalogSyncState =
|
||||
| "queued" | "running" | "awaiting_confirmation" | "applying"
|
||||
| "succeeded" | "failed" | "cancelled" | "interrupted";
|
||||
export type CatalogSyncPhase =
|
||||
| "queued" | "connecting" | "scanning_tables" | "scanning_columns"
|
||||
| "scanning_relationships" | "planning" | "awaiting_confirmation"
|
||||
| "applying" | "completed";
|
||||
|
||||
export interface CatalogSchemaDiff {
|
||||
deletedTables: string[];
|
||||
deletedColumns: Array<{ tableName: string; columnName: string }>;
|
||||
deletedRelationships: Array<{ sourceTableName: string; constraintName: string }>;
|
||||
}
|
||||
|
||||
export interface CatalogSyncCounts {
|
||||
tables?: number;
|
||||
columns?: number;
|
||||
relationships?: number;
|
||||
created?: number;
|
||||
updated?: number;
|
||||
deleted?: number;
|
||||
}
|
||||
|
||||
export interface CatalogSyncRun {
|
||||
id: string;
|
||||
databaseId: string;
|
||||
scope: CatalogSyncScope;
|
||||
tableIds: string[];
|
||||
state: CatalogSyncState;
|
||||
phase: CatalogSyncPhase;
|
||||
requestedDatabaseVersion: number;
|
||||
observedSnapshot: ObservedSchemaSnapshot | null;
|
||||
plannedDiff: CatalogSchemaDiff | null;
|
||||
confirmationToken: string | null;
|
||||
counts: CatalogSyncCounts;
|
||||
errorCode: string | null;
|
||||
errorMessage: string | null;
|
||||
cancelRequested: boolean;
|
||||
createdAt: string;
|
||||
startedAt: string | null;
|
||||
updatedAt: string;
|
||||
finishedAt: string | null;
|
||||
heartbeatAt: string | null;
|
||||
leaseOwner: string | null;
|
||||
leaseExpiresAt: string | null;
|
||||
}
|
||||
|
||||
export interface CatalogSyncEvent {
|
||||
id: number;
|
||||
runId: string;
|
||||
sequence: number;
|
||||
level: "info" | "warning" | "error";
|
||||
eventType: string;
|
||||
message: string;
|
||||
data: Record<string, unknown>;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export interface CatalogSyncRunUpdate {
|
||||
state?: CatalogSyncState;
|
||||
phase?: CatalogSyncPhase;
|
||||
observedSnapshot?: ObservedSchemaSnapshot | null;
|
||||
plannedDiff?: CatalogSchemaDiff | null;
|
||||
confirmationToken?: string | null;
|
||||
counts?: CatalogSyncCounts;
|
||||
errorCode?: string | null;
|
||||
errorMessage?: string | null;
|
||||
cancelRequested?: boolean;
|
||||
startedAt?: string | null;
|
||||
finishedAt?: string | null;
|
||||
heartbeatAt?: string | null;
|
||||
leaseOwner?: string | null;
|
||||
leaseExpiresAt?: string | null;
|
||||
}
|
||||
|
||||
export type TableSyncRepositoryResult =
|
||||
| { kind: "confirmation_required"; deletedNames: string[] }
|
||||
| {
|
||||
kind: "applied";
|
||||
createdCount: number;
|
||||
updatedCount: number;
|
||||
deletedCount: number;
|
||||
tables: CatalogTable[];
|
||||
};
|
||||
|
||||
export interface CatalogRepository {
|
||||
list(): Promise<WorkspaceDatabase[]>;
|
||||
get(id: string): Promise<WorkspaceDatabase | undefined>;
|
||||
getByWorkspace(workspaceId: string): Promise<WorkspaceDatabase | undefined>;
|
||||
create(input: DatabaseConfigurationInput): Promise<WorkspaceDatabase>;
|
||||
update(id: string, expectedVersion: number, input: DatabaseConfigurationInput): Promise<WorkspaceDatabase | undefined>;
|
||||
recordTest(id: string, expectedVersion: number, result: DatabaseTestResult): Promise<WorkspaceDatabase | undefined>;
|
||||
touch(id: string, expectedVersion: number): Promise<WorkspaceDatabase | undefined>;
|
||||
delete(id: string, expectedVersion: number): Promise<boolean>;
|
||||
listTables(databaseId: string): Promise<CatalogTable[]>;
|
||||
getTable(databaseId: string, tableId: string): Promise<CatalogTable | undefined>;
|
||||
updateTableDescription(
|
||||
databaseId: string,
|
||||
tableId: string,
|
||||
expectedVersion: number,
|
||||
description: string | null,
|
||||
): Promise<CatalogTable | undefined>;
|
||||
updateTableMetadata(
|
||||
databaseId: string,
|
||||
tableId: string,
|
||||
expectedVersion: number,
|
||||
description: string | null,
|
||||
generatedDescription: string | null,
|
||||
): Promise<CatalogTable | undefined>;
|
||||
listColumns(databaseId: string, tableId: string): Promise<CatalogColumn[]>;
|
||||
getColumn(databaseId: string, tableId: string, columnId: string): Promise<CatalogColumn | undefined>;
|
||||
updateColumnMetadata(
|
||||
databaseId: string,
|
||||
tableId: string,
|
||||
columnId: string,
|
||||
expectedVersion: number,
|
||||
description: string | null,
|
||||
generatedDescription: string | null,
|
||||
): Promise<CatalogColumn | undefined>;
|
||||
listRelationships(databaseId: string): Promise<CatalogRelationship[]>;
|
||||
planSchemaSync(
|
||||
databaseId: string,
|
||||
scope: CatalogSyncScope,
|
||||
tableIds: readonly string[],
|
||||
snapshot: ObservedSchemaSnapshot,
|
||||
): Promise<CatalogSchemaDiff>;
|
||||
applySchemaSync(
|
||||
databaseId: string,
|
||||
expectedDatabaseVersion: number,
|
||||
scope: CatalogSyncScope,
|
||||
tableIds: readonly string[],
|
||||
snapshot: ObservedSchemaSnapshot,
|
||||
): Promise<CatalogSyncCounts | undefined>;
|
||||
createSyncRun(
|
||||
databaseId: string,
|
||||
scope: CatalogSyncScope,
|
||||
tableIds: readonly string[],
|
||||
requestedDatabaseVersion: number,
|
||||
): Promise<CatalogSyncRun>;
|
||||
getSyncRun(runId: string): Promise<CatalogSyncRun | undefined>;
|
||||
claimSyncRun(runId: string, workerId: string, leaseExpiresAt: string): Promise<CatalogSyncRun | undefined>;
|
||||
listSyncRuns(databaseId: string, limit?: number): Promise<CatalogSyncRun[]>;
|
||||
updateSyncRun(runId: string, update: CatalogSyncRunUpdate): Promise<CatalogSyncRun | undefined>;
|
||||
requestSyncRunCancellation(runId: string): Promise<CatalogSyncRun | undefined>;
|
||||
appendSyncEvent(
|
||||
runId: string,
|
||||
level: CatalogSyncEvent["level"],
|
||||
eventType: string,
|
||||
message: string,
|
||||
data?: Record<string, unknown>,
|
||||
): Promise<CatalogSyncEvent>;
|
||||
listSyncEvents(runId: string, afterSequence?: number): Promise<CatalogSyncEvent[]>;
|
||||
pruneSyncEvents(before: string): Promise<void>;
|
||||
interruptActiveSyncRuns(): Promise<void>;
|
||||
reconcileTables(
|
||||
databaseId: string,
|
||||
expectedDatabaseVersion: number,
|
||||
observed: readonly ObservedCatalogTable[],
|
||||
confirmedDeletedNames: readonly string[],
|
||||
): Promise<TableSyncRepositoryResult | undefined>;
|
||||
available(): Promise<boolean>;
|
||||
close?(): Promise<void>;
|
||||
}
|
||||
|
||||
export class CatalogConflictError extends Error {}
|
||||
export class CatalogUnavailableError extends Error {}
|
||||
export class CatalogOperationInProgressError extends Error {}
|
||||
export class CatalogConnectorError extends Error {}
|
||||
export class CatalogSchemaCapabilityUnavailableError extends CatalogConnectorError {
|
||||
constructor(readonly capability: "schema_snapshot" | "tables" | "columns" | "relationships") {
|
||||
super(`Schema introspection capability '${capability}' is unavailable`);
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import {
|
||||
} from "./auth/config.js";
|
||||
import { createProjectedAuthenticationConfigProvider } from "./auth/runtime-projection.js";
|
||||
import type { WorkspaceRegistryConfig } from "./workspaces/types.js";
|
||||
import type { CatalogConnectionConfig } from "./catalog/repository.js";
|
||||
|
||||
export interface AppConfig {
|
||||
host: string; port: number; harnessDir: string; thtBin: string; piBin: string;
|
||||
@@ -20,6 +21,8 @@ export interface AppConfig {
|
||||
host?: string; port?: number; database?: string; runtimeUser?: string;
|
||||
runtimePasswordFile?: string; sslmode?: "verify-ca" | "verify-full"; sslrootcert?: string;
|
||||
};
|
||||
/** Installation-local metadata catalog. Omitted installations expose an unavailable admin surface. */
|
||||
catalogDatabase?: CatalogConnectionConfig;
|
||||
defaults: { provider?: string; model?: string; thinking?: string };
|
||||
maxPiProcesses: number;
|
||||
settingsFile: string;
|
||||
@@ -40,6 +43,7 @@ export interface AppConfig {
|
||||
/** Explicit compatibility mode for old loopback clients that send `workspace` in POST /sessions. */
|
||||
legacyWorkspaceMode: boolean;
|
||||
workspaceDiagnosticTimeoutMs: number;
|
||||
catalogSyncTimeoutMs: number;
|
||||
workspaceRegistry: WorkspaceRegistryConfig;
|
||||
workspaceSecretStoreRoot: string;
|
||||
workspaceSecretRuntimeRoot: string;
|
||||
@@ -127,6 +131,14 @@ function diagnosticTimeout(value: string | undefined): number {
|
||||
return timeout;
|
||||
}
|
||||
|
||||
function catalogSyncTimeout(value: string | undefined): number {
|
||||
const timeout = Number(value ?? 600_000);
|
||||
if (!Number.isSafeInteger(timeout) || timeout < 1_000 || timeout > 3_600_000) {
|
||||
throw new Error("catalog synchronization timeout configuration is invalid");
|
||||
}
|
||||
return timeout;
|
||||
}
|
||||
|
||||
function piManagementTimeout(value: string | undefined): number {
|
||||
const timeout = Number(value ?? 8_000);
|
||||
if (!Number.isSafeInteger(timeout) || timeout < 1 || timeout > 30_000) {
|
||||
@@ -176,6 +188,33 @@ function positiveDimension(value: string | undefined, fallback: number): number
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function catalogDatabase(env: Record<string, string | undefined>): CatalogConnectionConfig | undefined {
|
||||
const value = env.THT_CATALOG_DATABASE_URL;
|
||||
if (value !== undefined) {
|
||||
try {
|
||||
const parsed = new URL(value);
|
||||
if ((parsed.protocol !== "postgres:" && parsed.protocol !== "postgresql:")
|
||||
|| !parsed.hostname || !parsed.pathname.slice(1) || parsed.hash || parsed.search) throw new Error();
|
||||
return { connectionString: value };
|
||||
} catch {
|
||||
throw new Error("catalog database configuration is invalid");
|
||||
}
|
||||
}
|
||||
const host = env.THT_CATALOG_DB_HOST;
|
||||
if (host === undefined) return undefined;
|
||||
const port = Number(env.THT_CATALOG_DB_PORT ?? 5432);
|
||||
const database = env.THT_CATALOG_DB_NAME;
|
||||
const user = env.THT_CATALOG_RUNTIME_USER;
|
||||
const passwordFile = env.THT_CATALOG_RUNTIME_PASSWORD_FILE;
|
||||
try {
|
||||
if (!host.trim() || !database?.trim() || !user?.trim() || !passwordFile
|
||||
|| !path.isAbsolute(passwordFile) || !Number.isInteger(port) || port < 1 || port > 65_535) throw new Error();
|
||||
return { host, port, database, user, passwordFile };
|
||||
} catch {
|
||||
throw new Error("catalog database configuration is invalid");
|
||||
}
|
||||
}
|
||||
|
||||
export function loadConfig(
|
||||
env: Record<string, string | undefined>,
|
||||
options: { surface?: "application" | "workspace-maintenance" } = {},
|
||||
@@ -356,6 +395,7 @@ export function loadConfig(
|
||||
authentication,
|
||||
publicExposure,
|
||||
sessionStorage,
|
||||
catalogDatabase: catalogDatabase(env),
|
||||
defaults: { provider: env.PI_PROVIDER, model: env.PI_MODEL, thinking: env.PI_THINKING },
|
||||
maxPiProcesses: Number(env.MAX_PI_PROCESSES ?? 4),
|
||||
settingsFile,
|
||||
@@ -370,6 +410,7 @@ export function loadConfig(
|
||||
dwhPrecheck: env.THT_DWH_PRECHECK === "true" || env.THT_DWH_PRECHECK === "1",
|
||||
legacyWorkspaceMode: legacyWorkspaceMode === "local",
|
||||
workspaceDiagnosticTimeoutMs: diagnosticTimeout(env.THT_WORKSPACE_DIAGNOSTIC_TIMEOUT_MS),
|
||||
catalogSyncTimeoutMs: catalogSyncTimeout(env.THT_CATALOG_SYNC_TIMEOUT_MS),
|
||||
workspaceRegistry,
|
||||
workspaceSecretStoreRoot,
|
||||
workspaceSecretRuntimeRoot,
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
|
||||
import { z } from "zod";
|
||||
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||
import { CatalogService, type CatalogSecretName } from "../catalog/service.js";
|
||||
import { WorkspaceRegistryError } from "../workspaces/git-repository.js";
|
||||
import {
|
||||
CatalogConflictError,
|
||||
CatalogOperationInProgressError,
|
||||
CatalogUnavailableError,
|
||||
DATABASE_TRANSPORTS,
|
||||
type CatalogRepository,
|
||||
type DatabaseConfigurationInput,
|
||||
} from "../catalog/types.js";
|
||||
import type { CatalogOperationCoordinator } from "../catalog/operation-coordinator.js";
|
||||
|
||||
const idSchema = z.uuid();
|
||||
const workspaceIdSchema = z.string().regex(/^[a-z][a-z0-9-]{2,62}$/);
|
||||
const identifier = z.string().trim().min(1).max(128).regex(/^[A-Za-z_][A-Za-z0-9_$-]*$/);
|
||||
const nonEmpty = z.string().trim().min(1).max(512);
|
||||
const port = z.number().int().min(1).max(65_535);
|
||||
const optionalText = nonEmpty.optional();
|
||||
const sshHost = z.string().trim().min(1).max(255).regex(/^[A-Za-z0-9_.:\[\]-]+$/).optional();
|
||||
const sshUsername = z.string().trim().min(1).max(128).regex(/^[A-Za-z0-9._-]+$/).optional();
|
||||
const bindingSchema = z.object({
|
||||
transport: z.enum(DATABASE_TRANSPORTS),
|
||||
host: optionalText,
|
||||
port: port.optional(),
|
||||
username: optionalText,
|
||||
baseUrl: z.url().max(2048).optional(),
|
||||
restPath: z.string().regex(/^\/(?!\/)[^?#\\\u0000-\u001f]*$/).max(512).optional(),
|
||||
restAuth: z.enum(["none", "bearer", "x-api-key"]).optional(),
|
||||
tlsServername: optionalText,
|
||||
sshHost,
|
||||
sshPort: port.optional(),
|
||||
sshUsername,
|
||||
sshTargetHost: sshHost,
|
||||
sshTargetPort: port.optional(),
|
||||
}).strict().superRefine((binding, context) => {
|
||||
const required = binding.transport === "postgres_direct"
|
||||
? ["host", "port", "username"] as const
|
||||
: binding.transport === "rest_api"
|
||||
? ["baseUrl", "restPath", "restAuth"] as const
|
||||
: ["username", "sshHost", "sshPort", "sshUsername", "sshTargetHost", "sshTargetPort"] as const;
|
||||
for (const field of required) {
|
||||
if (binding[field] === undefined) context.addIssue({ code: "custom", path: [field], message: "Required" });
|
||||
}
|
||||
});
|
||||
const configSchema = z.object({
|
||||
workspaceId: workspaceIdSchema,
|
||||
engine: z.literal("postgres"),
|
||||
databaseName: identifier,
|
||||
schema: identifier,
|
||||
binding: bindingSchema,
|
||||
}).strict();
|
||||
const updateSchema = configSchema.extend({ version: z.number().int().positive() });
|
||||
const secretNames = [
|
||||
"password",
|
||||
"apiKey",
|
||||
"sshPrivateKey",
|
||||
"sshPrivateKeyPassphrase",
|
||||
"sshKnownHosts",
|
||||
"tlsCa",
|
||||
] as const;
|
||||
const secretsSchema = z.object({
|
||||
version: z.number().int().positive(),
|
||||
values: z.partialRecord(z.enum(secretNames), z.string().min(1).max(65_536)).refine((values) => Object.keys(values).length > 0),
|
||||
}).strict();
|
||||
const versionQuery = z.object({ version: z.coerce.number().int().positive() });
|
||||
|
||||
function safeError(reply: FastifyReply, error: unknown) {
|
||||
if (error instanceof CatalogUnavailableError) {
|
||||
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
|
||||
}
|
||||
if (error instanceof CatalogConflictError) {
|
||||
return reply.code(409).send({ code: "database_conflict", message: "This workspace already has a database configuration." });
|
||||
}
|
||||
if (error instanceof CatalogOperationInProgressError) {
|
||||
return reply.code(409).send({ code: "database_operation_in_progress", message: "A database operation is already in progress." });
|
||||
}
|
||||
if (error instanceof z.ZodError) {
|
||||
return reply.code(400).send({ code: "database_invalid", message: "Database configuration is invalid." });
|
||||
}
|
||||
if (error instanceof WorkspaceRegistryError) {
|
||||
return reply.code(400).send({ code: "database_invalid", message: "Database configuration is invalid." });
|
||||
}
|
||||
return reply.code(500).send({ code: "database_operation_failed", message: "Database operation failed." });
|
||||
}
|
||||
|
||||
function manage(request: FastifyRequest, reply: FastifyReply) {
|
||||
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
|
||||
}
|
||||
|
||||
export function catalogDatabaseRoutes(
|
||||
app: FastifyInstance,
|
||||
deps: { repository: CatalogRepository; service: CatalogService; operations?: CatalogOperationCoordinator },
|
||||
): void {
|
||||
const mutate = async <T>(databaseId: string, operation: () => Promise<T>): Promise<T> => (
|
||||
deps.operations ? await deps.operations.run(databaseId, operation) : await operation()
|
||||
);
|
||||
const activeSyncRun = async (databaseId: string) => {
|
||||
const run = (await deps.repository.listSyncRuns(databaseId, 5)).find((candidate) =>
|
||||
["queued", "running", "awaiting_confirmation", "applying"].includes(candidate.state));
|
||||
if (!run) return undefined;
|
||||
const {
|
||||
observedSnapshot: _snapshot,
|
||||
leaseOwner: _leaseOwner,
|
||||
leaseExpiresAt: _leaseExpiresAt,
|
||||
...summary
|
||||
} = run;
|
||||
return summary;
|
||||
};
|
||||
app.get("/catalog/status", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
return { available: await deps.repository.available() };
|
||||
});
|
||||
|
||||
app.get("/catalog/databases", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const rows = await deps.service.list();
|
||||
return await Promise.all(rows.map(async (row) => (
|
||||
row.id ? { ...row, activeSyncRun: await activeSyncRun(row.id) } : row
|
||||
)));
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.get("/catalog/databases/:id", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const id = idSchema.parse((request.params as { id?: unknown }).id);
|
||||
const database = await deps.repository.get(id);
|
||||
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||
return {
|
||||
...database,
|
||||
configured: true,
|
||||
secrets: deps.service.configuredSecrets(database.workspaceId),
|
||||
activeSyncRun: await activeSyncRun(database.id),
|
||||
};
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.post("/catalog/databases", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const input = configSchema.parse(request.body) as DatabaseConfigurationInput;
|
||||
const created = await deps.repository.create(await deps.service.normalizeInput(input));
|
||||
return reply.code(201).send({ ...created, configured: true, secrets: deps.service.configuredSecrets(created.workspaceId) });
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.patch("/catalog/databases/:id", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const id = idSchema.parse((request.params as { id?: unknown }).id);
|
||||
const { version, ...input } = updateSchema.parse(request.body);
|
||||
const current = await deps.repository.get(id);
|
||||
if (!current) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||
if (current.workspaceId !== input.workspaceId) {
|
||||
return reply.code(400).send({ code: "database_invalid", message: "Database configuration is invalid." });
|
||||
}
|
||||
const updated = await mutate(id, async () => await deps.repository.update(
|
||||
id, version, await deps.service.normalizeInput(input as DatabaseConfigurationInput),
|
||||
));
|
||||
if (!updated) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||
return { ...updated, configured: true, secrets: deps.service.configuredSecrets(updated.workspaceId) };
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.put("/catalog/databases/:id/secrets", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
if (!isPrincipalContext(requirePermission(request, reply, "workspace.secrets.manage"))) return reply;
|
||||
try {
|
||||
const id = idSchema.parse((request.params as { id?: unknown }).id);
|
||||
const { version, values } = secretsSchema.parse(request.body);
|
||||
const database = await deps.repository.get(id);
|
||||
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||
if (database.version !== version) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||
const updated = await mutate(id, async () => {
|
||||
const touched = await deps.repository.touch(id, version);
|
||||
if (touched) deps.service.replaceSecrets(database.workspaceId, values as Partial<Record<CatalogSecretName, string>>);
|
||||
return touched;
|
||||
});
|
||||
if (!updated) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||
return { ...updated, configured: true, secrets: deps.service.configuredSecrets(updated.workspaceId) };
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.post("/catalog/databases/:id/test", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const id = idSchema.parse((request.params as { id?: unknown }).id);
|
||||
const { version } = z.object({ version: z.number().int().positive() }).strict().parse(request.body);
|
||||
const database = await deps.repository.get(id);
|
||||
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||
if (database.version !== version) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||
const tested = await deps.repository.recordTest(id, version, await deps.service.test(database));
|
||||
if (!tested) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||
return { ...tested, configured: true, secrets: deps.service.configuredSecrets(tested.workspaceId) };
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.delete("/catalog/databases/:id", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const id = idSchema.parse((request.params as { id?: unknown }).id);
|
||||
const { version } = versionQuery.parse(request.query);
|
||||
const database = await deps.repository.get(id);
|
||||
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||
const deleted = await mutate(id, async () => {
|
||||
const removed = await deps.repository.delete(id, version);
|
||||
if (removed) deps.service.forgetSecrets(database.workspaceId);
|
||||
return removed;
|
||||
});
|
||||
if (!deleted) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||
return reply.code(204).send();
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,230 @@
|
||||
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { z } from "zod";
|
||||
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||
import type { CatalogSyncWorker } from "../catalog/sync-worker.js";
|
||||
import {
|
||||
CatalogConflictError,
|
||||
CatalogConnectorError,
|
||||
CatalogOperationInProgressError,
|
||||
CatalogUnavailableError,
|
||||
type CatalogRepository,
|
||||
type CatalogSyncRun,
|
||||
} from "../catalog/types.js";
|
||||
|
||||
const idSchema = z.uuid();
|
||||
const metadataSchema = z.object({
|
||||
version: z.number().int().positive(),
|
||||
description: z.string().max(20_000).nullable(),
|
||||
generatedDescription: z.string().max(20_000).nullable(),
|
||||
}).strict();
|
||||
const createRunSchema = z.object({
|
||||
version: z.number().int().positive(),
|
||||
scope: z.enum(["tables", "columns", "relationships", "all"]),
|
||||
tableIds: z.array(idSchema).max(10_000).default([]),
|
||||
}).strict();
|
||||
const confirmationSchema = z.object({ confirmationToken: z.string().uuid() }).strict();
|
||||
const eventQuerySchema = z.object({ after: z.coerce.number().int().nonnegative().default(0) });
|
||||
|
||||
function manage(request: FastifyRequest, reply: FastifyReply) {
|
||||
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
|
||||
}
|
||||
|
||||
function safeError(reply: FastifyReply, error: unknown) {
|
||||
if (error instanceof CatalogUnavailableError) {
|
||||
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
|
||||
}
|
||||
if (error instanceof CatalogConflictError || error instanceof CatalogOperationInProgressError) {
|
||||
return reply.code(409).send({ code: "schema_sync_conflict", message: error.message });
|
||||
}
|
||||
if (error instanceof CatalogConnectorError) {
|
||||
return reply.code(502).send({ code: "schema_introspection_failed", message: "The database schema could not be read safely." });
|
||||
}
|
||||
if (error instanceof z.ZodError) {
|
||||
return reply.code(400).send({ code: "schema_request_invalid", message: "Schema request is invalid." });
|
||||
}
|
||||
return reply.code(500).send({ code: "schema_operation_failed", message: "Schema operation failed." });
|
||||
}
|
||||
|
||||
function normalized(value: string | null): string | null {
|
||||
return value?.trim() || null;
|
||||
}
|
||||
|
||||
function publicRun(run: CatalogSyncRun) {
|
||||
const {
|
||||
observedSnapshot: _snapshot,
|
||||
leaseOwner: _leaseOwner,
|
||||
leaseExpiresAt: _leaseExpiresAt,
|
||||
...result
|
||||
} = run;
|
||||
return result;
|
||||
}
|
||||
|
||||
export function catalogSchemaRoutes(
|
||||
app: FastifyInstance,
|
||||
deps: { repository: CatalogRepository; worker: CatalogSyncWorker },
|
||||
): void {
|
||||
app.get("/catalog/databases/:databaseId/tables/:tableId/columns", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const params = request.params as { databaseId?: unknown; tableId?: unknown };
|
||||
const databaseId = idSchema.parse(params.databaseId);
|
||||
const tableId = idSchema.parse(params.tableId);
|
||||
if (!(await deps.repository.getTable(databaseId, tableId))) {
|
||||
return reply.code(404).send({ code: "table_not_found", message: "Catalog table was not found." });
|
||||
}
|
||||
return await deps.repository.listColumns(databaseId, tableId);
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.patch("/catalog/databases/:databaseId/tables/:tableId/columns/:columnId", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const params = request.params as { databaseId?: unknown; tableId?: unknown; columnId?: unknown };
|
||||
const databaseId = idSchema.parse(params.databaseId);
|
||||
const tableId = idSchema.parse(params.tableId);
|
||||
const columnId = idSchema.parse(params.columnId);
|
||||
const input = metadataSchema.parse(request.body);
|
||||
const current = await deps.repository.getColumn(databaseId, tableId, columnId);
|
||||
if (!current) return reply.code(404).send({ code: "column_not_found", message: "Catalog column was not found." });
|
||||
if (current.version !== input.version) {
|
||||
return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
|
||||
}
|
||||
const updated = await deps.repository.updateColumnMetadata(
|
||||
databaseId,
|
||||
tableId,
|
||||
columnId,
|
||||
input.version,
|
||||
normalized(input.description),
|
||||
normalized(input.generatedDescription),
|
||||
);
|
||||
if (!updated) return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
|
||||
return updated;
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.get("/catalog/databases/:databaseId/relationships", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
|
||||
if (!(await deps.repository.get(databaseId))) {
|
||||
return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||
}
|
||||
return await deps.repository.listRelationships(databaseId);
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.post("/catalog/databases/:databaseId/sync-runs", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
|
||||
const input = createRunSchema.parse(request.body);
|
||||
const database = await deps.repository.get(databaseId);
|
||||
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||
if (database.version !== input.version) {
|
||||
return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||
}
|
||||
return reply.code(202).send(publicRun(await deps.worker.start(database, input.scope, input.tableIds)));
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.get("/catalog/databases/:databaseId/sync-runs", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
|
||||
if (!(await deps.repository.get(databaseId))) {
|
||||
return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||
}
|
||||
return (await deps.repository.listSyncRuns(databaseId)).map(publicRun);
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.get("/catalog/sync-runs/:runId", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||
const run = await deps.repository.getSyncRun(runId);
|
||||
return run ? publicRun(run) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.post("/catalog/sync-runs/:runId/confirm", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||
const { confirmationToken } = confirmationSchema.parse(request.body);
|
||||
const run = await deps.worker.confirm(runId, confirmationToken);
|
||||
return run ? publicRun(run) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.post("/catalog/sync-runs/:runId/cancel", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||
const run = await deps.worker.cancel(runId);
|
||||
return run ? publicRun(run) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.post("/catalog/sync-runs/:runId/retry", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||
const run = await deps.worker.retry(runId);
|
||||
return run ? reply.code(202).send(publicRun(run)) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.get("/catalog/sync-runs/:runId/events", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||
let after = eventQuerySchema.parse(request.query).after;
|
||||
const headerCursor = Number(request.headers["last-event-id"]);
|
||||
if (Number.isInteger(headerCursor) && headerCursor >= 0) after = Math.max(after, headerCursor);
|
||||
if (!(await deps.repository.getSyncRun(runId))) {
|
||||
return reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||
}
|
||||
reply.hijack();
|
||||
reply.raw.writeHead(200, {
|
||||
"content-type": "text/event-stream; charset=utf-8",
|
||||
"cache-control": "no-cache, no-transform",
|
||||
connection: "keep-alive",
|
||||
"x-accel-buffering": "no",
|
||||
});
|
||||
const controller = new AbortController();
|
||||
request.raw.once("close", () => controller.abort());
|
||||
let lastRunUpdate = "";
|
||||
while (!controller.signal.aborted) {
|
||||
const events = await deps.repository.listSyncEvents(runId, after);
|
||||
for (const event of events) {
|
||||
after = event.sequence;
|
||||
reply.raw.write(`id: ${event.sequence}\nevent: log\ndata: ${JSON.stringify(event)}\n\n`);
|
||||
}
|
||||
const run = await deps.repository.getSyncRun(runId);
|
||||
if (!run) break;
|
||||
if (run.updatedAt !== lastRunUpdate) {
|
||||
lastRunUpdate = run.updatedAt;
|
||||
reply.raw.write(`event: run\ndata: ${JSON.stringify(publicRun(run))}\n\n`);
|
||||
}
|
||||
if (["succeeded", "failed", "cancelled", "interrupted"].includes(run.state)) break;
|
||||
await delay(500, undefined, { signal: controller.signal }).catch(() => undefined);
|
||||
}
|
||||
reply.raw.end();
|
||||
return reply;
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.get("/catalog/sync-runs/:runId/events-list", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||
const after = eventQuerySchema.parse(request.query).after;
|
||||
if (!(await deps.repository.getSyncRun(runId))) {
|
||||
return reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||
}
|
||||
return await deps.repository.listSyncEvents(runId, after);
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
|
||||
import { z } from "zod";
|
||||
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||
import type { CatalogTableService } from "../catalog/table-service.js";
|
||||
import {
|
||||
CatalogConnectorError,
|
||||
CatalogOperationInProgressError,
|
||||
CatalogUnavailableError,
|
||||
type CatalogRepository,
|
||||
} from "../catalog/types.js";
|
||||
|
||||
const idSchema = z.uuid();
|
||||
const updateSchema = z.object({
|
||||
version: z.number().int().positive(),
|
||||
description: z.string().max(20_000).nullable(),
|
||||
generatedDescription: z.string().max(20_000).nullable().optional(),
|
||||
}).strict();
|
||||
|
||||
function manage(request: FastifyRequest, reply: FastifyReply) {
|
||||
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
|
||||
}
|
||||
|
||||
function safeError(reply: FastifyReply, error: unknown) {
|
||||
if (error instanceof CatalogUnavailableError) {
|
||||
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
|
||||
}
|
||||
if (error instanceof CatalogOperationInProgressError) {
|
||||
return reply.code(409).send({ code: "database_operation_in_progress", message: "A database operation is already in progress." });
|
||||
}
|
||||
if (error instanceof CatalogConnectorError) {
|
||||
return reply.code(502).send({ code: "table_introspection_failed", message: "Database tables could not be read." });
|
||||
}
|
||||
if (error instanceof z.ZodError) {
|
||||
return reply.code(400).send({ code: "table_invalid", message: "Table request is invalid." });
|
||||
}
|
||||
return reply.code(500).send({ code: "table_operation_failed", message: "Table operation failed." });
|
||||
}
|
||||
|
||||
export function catalogTableRoutes(
|
||||
app: FastifyInstance,
|
||||
deps: { repository: CatalogRepository; service: CatalogTableService },
|
||||
): void {
|
||||
app.get("/catalog/databases/:databaseId/tables", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
|
||||
if (!(await deps.repository.get(databaseId))) {
|
||||
return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||
}
|
||||
return await deps.service.list(databaseId);
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
app.patch("/catalog/databases/:databaseId/tables/:tableId", async (request, reply) => {
|
||||
if (!manage(request, reply)) return reply;
|
||||
try {
|
||||
const params = request.params as { databaseId?: unknown; tableId?: unknown };
|
||||
const databaseId = idSchema.parse(params.databaseId);
|
||||
const tableId = idSchema.parse(params.tableId);
|
||||
const input = updateSchema.parse(request.body);
|
||||
const { version, description } = input;
|
||||
const current = await deps.repository.getTable(databaseId, tableId);
|
||||
if (!current) return reply.code(404).send({ code: "table_not_found", message: "Catalog table was not found." });
|
||||
if (current.version !== version) {
|
||||
return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
|
||||
}
|
||||
const updated = await deps.service.updateMetadata(
|
||||
databaseId,
|
||||
tableId,
|
||||
version,
|
||||
description,
|
||||
input.generatedDescription === undefined ? current.generatedDescription : input.generatedDescription,
|
||||
);
|
||||
if (!updated) return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
|
||||
return updated;
|
||||
} catch (error) { return safeError(reply, error); }
|
||||
});
|
||||
|
||||
}
|
||||
@@ -187,6 +187,7 @@ test("roles collapse duplicates and admin contains all administrative permission
|
||||
"settings.manage",
|
||||
"workspace.manage",
|
||||
"workspace.secrets.manage",
|
||||
"database.manage",
|
||||
"pi.manage",
|
||||
"auth.diagnostics.read",
|
||||
]);
|
||||
|
||||
@@ -114,7 +114,7 @@ test.each([
|
||||
const created = await fixture.app.thothiiAuthSessionStore?.create({
|
||||
principal: {
|
||||
issuer: "local", subject: user.id, displayName: user.username, roles: ["admin"],
|
||||
permissions: ["session.use", "session.read_all", "session.manage_all", "settings.manage", "workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read"],
|
||||
permissions: ["session.use", "session.read_all", "session.manage_all", "settings.manage", "workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read"],
|
||||
isAdmin: true,
|
||||
},
|
||||
method: "local",
|
||||
|
||||
@@ -130,7 +130,7 @@ test("local login sets a non-persistent opaque session cookie and exposes only a
|
||||
roles: ["admin"],
|
||||
permissions: [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
],
|
||||
isAdmin: true,
|
||||
csrfToken: expect.stringMatching(/^[A-Za-z0-9_-]{43}$/),
|
||||
|
||||
@@ -32,7 +32,7 @@ test("local mode resolves a stable local principal", async () => {
|
||||
roles: ["admin"],
|
||||
permissions: [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
],
|
||||
isAdmin: true,
|
||||
});
|
||||
@@ -74,7 +74,7 @@ test("upstream mode accepts only normalized proxy principal headers", async () =
|
||||
issuer: "portal", subject: "42", displayName: "Alice", roles: ["user", "admin"],
|
||||
permissions: [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
],
|
||||
isAdmin: true,
|
||||
});
|
||||
|
||||
@@ -15,14 +15,14 @@ const admin: PrincipalContext = {
|
||||
issuer: "oidc", subject: "admin", roles: ["admin"],
|
||||
permissions: [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
],
|
||||
isAdmin: true,
|
||||
};
|
||||
|
||||
const catalog: readonly Permission[] = [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
];
|
||||
|
||||
test("permission matrix gives role-less identities no access, users session use, and admins every catalog permission", () => {
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, expect, test, vi } from "vitest";
|
||||
import { buildApp } from "../src/app.js";
|
||||
import { loadConfig } from "../src/config.js";
|
||||
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
|
||||
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||
import type { WorkspaceRegistry, WorkspaceRevision } from "../src/workspaces/registry.js";
|
||||
import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
|
||||
|
||||
const workspace: WorkspaceDescriptor = {
|
||||
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
|
||||
dwh: {
|
||||
engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432,
|
||||
supported_transports: ["postgres_direct", "rest_api"],
|
||||
},
|
||||
semantic_index: {
|
||||
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
|
||||
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
|
||||
},
|
||||
llm_policy: { allowed: ["zai/glm-5.2"] },
|
||||
diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } },
|
||||
};
|
||||
const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" };
|
||||
|
||||
function setup() {
|
||||
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-secret-"));
|
||||
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-secret-runtime-"));
|
||||
roots.push(secretRoot, runtimeRoot);
|
||||
const secretStore = new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" });
|
||||
const repository = new MemoryCatalogRepository();
|
||||
const registry = {
|
||||
list: vi.fn(async () => [revision]),
|
||||
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
|
||||
read: vi.fn(async () => ({ workspace, revision })),
|
||||
} as unknown as WorkspaceRegistry;
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
|
||||
thtRunner: {} as never,
|
||||
workspaceRegistry: registry,
|
||||
workspaceSecretStore: secretStore,
|
||||
catalogRepository: repository,
|
||||
workspaceDiagnoser: vi.fn(),
|
||||
});
|
||||
return { app, secretStore, repository };
|
||||
}
|
||||
|
||||
const direct = {
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||
};
|
||||
|
||||
test("lists every YAML workspace and creates its one database configuration", async () => {
|
||||
const { app } = setup();
|
||||
const initial = await app.inject({ method: "GET", url: "/catalog/databases" });
|
||||
expect(initial.statusCode).toBe(200);
|
||||
expect(initial.json()).toMatchObject([{ workspaceId: "psd-clinical", configured: false, databaseName: "warehouse" }]);
|
||||
|
||||
const created = await app.inject({ method: "POST", url: "/catalog/databases", payload: direct });
|
||||
expect(created.statusCode).toBe(201);
|
||||
expect(created.json()).toMatchObject({ configured: true, workspaceId: "psd-clinical", version: 1 });
|
||||
expect((await app.inject({ method: "POST", url: "/catalog/databases", payload: direct })).statusCode).toBe(409);
|
||||
|
||||
const listed = await app.inject({ method: "GET", url: "/catalog/databases" });
|
||||
expect(listed.json()).toMatchObject([{ configured: true, binding: { transport: "postgres_direct", host: "db.internal" } }]);
|
||||
});
|
||||
|
||||
test("lists orphaned records and takes the REST diagnostic path from workspace YAML", async () => {
|
||||
const { app, repository } = setup();
|
||||
await repository.create({
|
||||
workspaceId: "removed-workspace",
|
||||
engine: "postgres",
|
||||
databaseName: "legacy",
|
||||
schema: "public",
|
||||
binding: { transport: "postgres_direct", host: "legacy.internal", port: 5432, username: "reader" },
|
||||
});
|
||||
const created = await app.inject({
|
||||
method: "POST",
|
||||
url: "/catalog/databases",
|
||||
payload: {
|
||||
...direct,
|
||||
binding: {
|
||||
transport: "rest_api", baseUrl: "https://psd.example/api", restPath: "/client-controlled", restAuth: "bearer",
|
||||
},
|
||||
},
|
||||
});
|
||||
expect(created.statusCode).toBe(201);
|
||||
expect(created.json()).toMatchObject({ binding: { restPath: "/health" } });
|
||||
|
||||
const rows = (await app.inject({ method: "GET", url: "/catalog/databases" })).json();
|
||||
expect(rows).toEqual(expect.arrayContaining([
|
||||
expect.objectContaining({ workspaceId: "removed-workspace", configured: true, workspaceAvailable: false }),
|
||||
expect.objectContaining({ workspaceId: "psd-clinical", configured: true, workspaceAvailable: true }),
|
||||
]));
|
||||
});
|
||||
|
||||
test("uses optimistic versions, keeps secrets write-only, and hard-deletes only local configuration", async () => {
|
||||
const { app, secretStore } = setup();
|
||||
const created = (await app.inject({ method: "POST", url: "/catalog/databases", payload: direct })).json();
|
||||
const stale = await app.inject({ method: "PATCH", url: `/catalog/databases/${created.id}`, payload: { ...direct, version: 99 } });
|
||||
expect(stale.statusCode).toBe(409);
|
||||
|
||||
const secret = await app.inject({
|
||||
method: "PUT", url: `/catalog/databases/${created.id}/secrets`,
|
||||
payload: { version: 1, values: { password: "do-not-return-this" } },
|
||||
});
|
||||
expect(secret.statusCode).toBe(200);
|
||||
expect(secret.body).not.toContain("do-not-return-this");
|
||||
expect(secret.json()).toMatchObject({ version: 2, secrets: { password: true } });
|
||||
expect(secretStore.has("psd-clinical", "catalog.dwh.password")).toBe(true);
|
||||
|
||||
const removed = await app.inject({ method: "DELETE", url: `/catalog/databases/${created.id}?version=2` });
|
||||
expect(removed.statusCode).toBe(204);
|
||||
expect(secretStore.has("psd-clinical", "catalog.dwh.password")).toBe(false);
|
||||
expect((await app.inject({ method: "GET", url: "/catalog/databases" })).json()).toMatchObject([{ configured: false }]);
|
||||
});
|
||||
@@ -0,0 +1,171 @@
|
||||
import { EventEmitter } from "node:events";
|
||||
import { existsSync, mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { PassThrough } from "node:stream";
|
||||
import type { ChildProcessWithoutNullStreams } from "node:child_process";
|
||||
import type { Client, ClientConfig } from "pg";
|
||||
import { afterEach, expect, test, vi } from "vitest";
|
||||
import {
|
||||
buildSshArguments,
|
||||
ConcreteCatalogPostgresAccess,
|
||||
} from "../src/catalog/postgres-access.js";
|
||||
import { CATALOG_SECRET_IDS } from "../src/catalog/secrets.js";
|
||||
import type { WorkspaceDatabase } from "../src/catalog/types.js";
|
||||
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||
|
||||
const roots: string[] = [];
|
||||
|
||||
afterEach(() => {
|
||||
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function secretStore() {
|
||||
const root = mkdtempSync(join(tmpdir(), "catalog-ssh-secrets-"));
|
||||
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-ssh-runtime-"));
|
||||
roots.push(root, runtimeRoot);
|
||||
return new WorkspaceSecretStore({ root, runtimeRoot, installationId: "test" });
|
||||
}
|
||||
|
||||
function sshDatabase(): WorkspaceDatabase {
|
||||
return {
|
||||
id: "11111111-1111-4111-8111-111111111111",
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
version: 4,
|
||||
createdAt: "2026-08-27T08:00:00Z",
|
||||
updatedAt: "2026-08-27T09:00:00Z",
|
||||
connectionStatus: "reachable",
|
||||
binding: {
|
||||
transport: "ssh_tunnel",
|
||||
username: "warehouse_reader",
|
||||
sshHost: "bastion.internal",
|
||||
sshPort: 2222,
|
||||
sshUsername: "tunnel_user",
|
||||
sshTargetHost: "postgres.internal",
|
||||
sshTargetPort: 5432,
|
||||
tlsServername: "postgres.internal",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function fakeChild(): ChildProcessWithoutNullStreams {
|
||||
const child = new EventEmitter() as EventEmitter & {
|
||||
stdin: PassThrough;
|
||||
stdout: PassThrough;
|
||||
stderr: PassThrough;
|
||||
exitCode: number | null;
|
||||
signalCode: NodeJS.Signals | null;
|
||||
kill: (signal?: NodeJS.Signals | number) => boolean;
|
||||
};
|
||||
child.stdin = new PassThrough();
|
||||
child.stdout = new PassThrough();
|
||||
child.stderr = new PassThrough();
|
||||
child.exitCode = null;
|
||||
child.signalCode = null;
|
||||
child.kill = vi.fn((signal: NodeJS.Signals | number = "SIGTERM") => {
|
||||
child.signalCode = typeof signal === "string" ? signal : "SIGTERM";
|
||||
child.emit("exit", null, child.signalCode);
|
||||
return true;
|
||||
});
|
||||
return child as unknown as ChildProcessWithoutNullStreams;
|
||||
}
|
||||
|
||||
test("builds a strict host-verified OpenSSH stdio tunnel", () => {
|
||||
const args = buildSshArguments({
|
||||
sshHost: "bastion.internal",
|
||||
sshPort: 2222,
|
||||
sshUsername: "tunnel_user",
|
||||
targetHost: "postgres.internal",
|
||||
targetPort: 5432,
|
||||
privateKeyFile: "/runtime/id",
|
||||
knownHostsFile: "/runtime/known_hosts",
|
||||
passphraseFile: "/runtime/passphrase",
|
||||
connectTimeoutMs: 5_001,
|
||||
});
|
||||
|
||||
expect(args).toEqual(expect.arrayContaining([
|
||||
"-F", "/dev/null",
|
||||
"-o", "BatchMode=no",
|
||||
"-o", "StrictHostKeyChecking=yes",
|
||||
"-o", "UserKnownHostsFile=/runtime/known_hosts",
|
||||
"-o", "GlobalKnownHostsFile=/dev/null",
|
||||
"-o", "IdentitiesOnly=yes",
|
||||
"-o", "IdentityAgent=none",
|
||||
"-o", "PasswordAuthentication=no",
|
||||
"-o", "KbdInteractiveAuthentication=no",
|
||||
"-o", "ConnectTimeout=6",
|
||||
"-W", "postgres.internal:5432",
|
||||
"--", "tunnel_user@bastion.internal",
|
||||
]));
|
||||
});
|
||||
|
||||
test("connects pg through OpenSSH, supplies askpass, and releases all secret leases", async () => {
|
||||
const store = secretStore();
|
||||
store.putMany("psd-clinical", {
|
||||
[CATALOG_SECRET_IDS.password]: "db-password ",
|
||||
[CATALOG_SECRET_IDS.sshPrivateKey]: "PRIVATE KEY\n",
|
||||
[CATALOG_SECRET_IDS.sshPrivateKeyPassphrase]: "key-passphrase",
|
||||
[CATALOG_SECRET_IDS.sshKnownHosts]: "bastion.internal ssh-ed25519 AAAATEST\n",
|
||||
[CATALOG_SECRET_IDS.tlsCa]: "CA CERTIFICATE\n",
|
||||
});
|
||||
const child = fakeChild();
|
||||
let clientConfig: ClientConfig | undefined;
|
||||
let spawnCall: { command: string; args: readonly string[]; env: NodeJS.ProcessEnv } | undefined;
|
||||
const end = vi.fn(async () => undefined);
|
||||
const query = vi.fn(async () => ({ rows: [{ ok: true }] }));
|
||||
const connect = vi.fn(async () => undefined);
|
||||
|
||||
const access = new ConcreteCatalogPostgresAccess(store, {
|
||||
sshBinary: "/usr/bin/ssh",
|
||||
askpassPath: "/app/ssh-askpass.mjs",
|
||||
connectTimeoutMs: 5_000,
|
||||
spawnSsh: (command, args, options) => {
|
||||
spawnCall = { command, args, env: options.env };
|
||||
return child;
|
||||
},
|
||||
createClient: (config) => {
|
||||
clientConfig = config;
|
||||
return { connect, query, end } as unknown as Client;
|
||||
},
|
||||
});
|
||||
|
||||
const client = await access.connect(sshDatabase(), new AbortController().signal);
|
||||
expect(connect).toHaveBeenCalledOnce();
|
||||
expect(clientConfig).toMatchObject({
|
||||
host: "postgres.internal",
|
||||
port: 5432,
|
||||
database: "warehouse",
|
||||
user: "warehouse_reader",
|
||||
password: "db-password ",
|
||||
connectionTimeoutMillis: 5_000,
|
||||
ssl: {
|
||||
ca: "CA CERTIFICATE\n",
|
||||
servername: "postgres.internal",
|
||||
rejectUnauthorized: true,
|
||||
},
|
||||
});
|
||||
expect(clientConfig?.stream).toBeTypeOf("function");
|
||||
expect(spawnCall?.command).toBe("/usr/bin/ssh");
|
||||
expect(spawnCall?.args.some((argument) => argument.startsWith("IdentityFile="))).toBe(true);
|
||||
expect(spawnCall?.args.some((argument) => argument.startsWith("UserKnownHostsFile="))).toBe(true);
|
||||
expect(spawnCall?.env).toMatchObject({
|
||||
DISPLAY: "thothii",
|
||||
SSH_ASKPASS: "/app/ssh-askpass.mjs",
|
||||
SSH_ASKPASS_REQUIRE: "force",
|
||||
});
|
||||
const leasedPaths = spawnCall!.args
|
||||
.filter((argument) => argument.startsWith("IdentityFile=") || argument.startsWith("UserKnownHostsFile="))
|
||||
.map((argument) => argument.slice(argument.indexOf("=") + 1));
|
||||
leasedPaths.push(spawnCall!.env.THT_SSH_PASSPHRASE_FILE!);
|
||||
expect(leasedPaths.every(existsSync)).toBe(true);
|
||||
|
||||
await expect(client.query("SELECT 1", [])).resolves.toEqual({ rows: [{ ok: true }] });
|
||||
await client.end();
|
||||
|
||||
expect(end).toHaveBeenCalledOnce();
|
||||
expect(child.kill).toHaveBeenCalledWith("SIGTERM");
|
||||
expect(leasedPaths.some(existsSync)).toBe(false);
|
||||
});
|
||||
@@ -0,0 +1,125 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { PostgreSqlContainer } from "@testcontainers/postgresql";
|
||||
import { CamelCasePlugin, Kysely, PostgresDialect, sql } from "kysely";
|
||||
import { Pool } from "pg";
|
||||
import { expect, test } from "vitest";
|
||||
import { KyselyCatalogRepository, type CatalogDatabase } from "../src/catalog/repository.js";
|
||||
import type { ObservedSchemaSnapshot } from "../src/catalog/types.js";
|
||||
import { up as upDatabases } from "../src/catalog/migrations/001_workspace_databases.js";
|
||||
import { up as upTables } from "../src/catalog/migrations/002_catalog_tables.js";
|
||||
import { up as upSchemaSync } from "../src/catalog/migrations/003_catalog_schema_sync.js";
|
||||
import { up as upRuntimeSequencePrivileges } from "../src/catalog/migrations/004_catalog_runtime_sequence_privileges.js";
|
||||
|
||||
const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
|
||||
|
||||
test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per workspace and optimistic updates", async () => {
|
||||
const container = await new PostgreSqlContainer("postgres:17.6-bookworm").start();
|
||||
const db = new Kysely<CatalogDatabase>({
|
||||
dialect: new PostgresDialect({ pool: new Pool({ connectionString: container.getConnectionUri() }) }),
|
||||
plugins: [new CamelCasePlugin()],
|
||||
});
|
||||
try {
|
||||
await upDatabases(db);
|
||||
await upTables(db);
|
||||
await upSchemaSync(db);
|
||||
await sql`CREATE ROLE thothii_catalog_runtime`.execute(db);
|
||||
await upRuntimeSequencePrivileges(db);
|
||||
const sequencePrivilege = await sql<{ allowed: boolean }>`
|
||||
SELECT has_sequence_privilege(
|
||||
'thothii_catalog_runtime',
|
||||
'catalog_sync_events_id_seq',
|
||||
'USAGE'
|
||||
) AS allowed
|
||||
`.execute(db);
|
||||
expect(sequencePrivilege.rows[0]?.allowed).toBe(true);
|
||||
const repository = new KyselyCatalogRepository(db);
|
||||
const input = {
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres" as const,
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
binding: { transport: "rest_api" as const, baseUrl: "https://psd.example/api", restPath: "/health", restAuth: "bearer" as const },
|
||||
};
|
||||
const created = await repository.create(input);
|
||||
expect(created).toMatchObject({ version: 1, connectionStatus: "untested", binding: { transport: "rest_api" } });
|
||||
await expect(repository.create(input)).rejects.toThrow("Workspace database already exists");
|
||||
expect(await repository.update(created.id, 99, input)).toBeUndefined();
|
||||
const synchronized = await repository.reconcileTables(created.id, 1, [
|
||||
{ name: "patients", sourceComment: "Clinical patients" },
|
||||
{ name: "visits", sourceComment: null },
|
||||
], []);
|
||||
expect(synchronized).toMatchObject({ kind: "applied", createdCount: 2, deletedCount: 0 });
|
||||
const patients = (await repository.listTables(created.id))[0];
|
||||
expect(await repository.updateTableDescription(
|
||||
created.id,
|
||||
patients.id,
|
||||
patients.version,
|
||||
"Curated patients",
|
||||
)).toMatchObject({ description: "Curated patients", sourceComment: "Clinical patients", version: 2 });
|
||||
const visits = (await repository.listTables(created.id)).find((table) => table.name === "visits")!;
|
||||
const fullColumnsSnapshot: ObservedSchemaSnapshot = {
|
||||
schemaVersion: 1,
|
||||
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||
tables: [
|
||||
{ name: "patients", sourceComment: "Clinical patients" },
|
||||
{ name: "visits", sourceComment: null },
|
||||
],
|
||||
columns: [
|
||||
{ tableName: "patients", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
|
||||
{ tableName: "patients", name: "name", ordinalPosition: 2, dataType: "text", isNullable: true, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
|
||||
{ tableName: "visits", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
|
||||
{ tableName: "visits", name: "patient_id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
|
||||
],
|
||||
relationships: [],
|
||||
};
|
||||
expect(await repository.applySchemaSync(created.id, 1, "columns", [], fullColumnsSnapshot))
|
||||
.toMatchObject({ created: 4, deleted: 0 });
|
||||
expect((await repository.listColumns(created.id, patients.id)).map((column) => column.name))
|
||||
.toEqual(["id", "name"]);
|
||||
expect((await repository.listColumns(created.id, visits.id)).map((column) => column.name))
|
||||
.toEqual(["id", "patient_id"]);
|
||||
|
||||
const reducedColumnsSnapshot: ObservedSchemaSnapshot = {
|
||||
...fullColumnsSnapshot,
|
||||
columns: fullColumnsSnapshot.columns.filter((column) => column.name === "id"),
|
||||
};
|
||||
expect(await repository.planSchemaSync(created.id, "columns", [], reducedColumnsSnapshot)).toMatchObject({
|
||||
deletedColumns: [
|
||||
{ tableName: "patients", columnName: "name" },
|
||||
{ tableName: "visits", columnName: "patient_id" },
|
||||
],
|
||||
});
|
||||
expect(await repository.planSchemaSync(created.id, "columns", [patients.id], reducedColumnsSnapshot)).toMatchObject({
|
||||
deletedColumns: [{ tableName: "patients", columnName: "name" }],
|
||||
});
|
||||
expect(await repository.applySchemaSync(created.id, 1, "columns", [patients.id], reducedColumnsSnapshot))
|
||||
.toMatchObject({ deleted: 1 });
|
||||
expect((await repository.listColumns(created.id, patients.id)).map((column) => column.name))
|
||||
.toEqual(["id"]);
|
||||
expect((await repository.listColumns(created.id, visits.id)).map((column) => column.name))
|
||||
.toEqual(["id", "patient_id"]);
|
||||
expect(await repository.reconcileTables(created.id, 1, [
|
||||
{ name: "patients", sourceComment: "Updated physical comment" },
|
||||
], [])).toEqual({ kind: "confirmation_required", deletedNames: ["visits"] });
|
||||
expect(await repository.reconcileTables(created.id, 1, [
|
||||
{ name: "patients", sourceComment: "Updated physical comment" },
|
||||
], ["visits"])).toMatchObject({ kind: "applied", updatedCount: 1, deletedCount: 1 });
|
||||
const syncRun = await repository.createSyncRun(created.id, "columns", [patients.id], 1);
|
||||
expect(syncRun).toMatchObject({
|
||||
databaseId: created.id,
|
||||
scope: "columns",
|
||||
tableIds: [patients.id],
|
||||
state: "queued",
|
||||
});
|
||||
expect(await repository.listSyncRuns(created.id)).toEqual([
|
||||
expect.objectContaining({ id: syncRun.id, tableIds: [patients.id] }),
|
||||
]);
|
||||
expect(await repository.update(created.id, 1, { ...input, schema: "public" })).toMatchObject({ version: 2, schema: "public" });
|
||||
expect(await repository.delete(created.id, 2)).toBe(true);
|
||||
expect(await repository.list()).toEqual([]);
|
||||
expect(await repository.listTables(created.id)).toEqual([]);
|
||||
} finally {
|
||||
await db.destroy();
|
||||
await container.stop();
|
||||
}
|
||||
}, 60_000);
|
||||
@@ -0,0 +1,194 @@
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, expect, test, vi } from "vitest";
|
||||
import type { CatalogDatabaseClient, CatalogPostgresAccess } from "../src/catalog/postgres-access.js";
|
||||
import { ConcreteCatalogSchemaIntrospector } from "../src/catalog/schema-introspector.js";
|
||||
import {
|
||||
CatalogSchemaCapabilityUnavailableError,
|
||||
type WorkspaceDatabase,
|
||||
} from "../src/catalog/types.js";
|
||||
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function store() {
|
||||
const root = mkdtempSync(join(tmpdir(), "catalog-schema-introspection-secrets-"));
|
||||
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-schema-introspection-runtime-"));
|
||||
roots.push(root, runtimeRoot);
|
||||
return new WorkspaceSecretStore({ root, runtimeRoot, installationId: "test" });
|
||||
}
|
||||
|
||||
function database(binding: WorkspaceDatabase["binding"]): WorkspaceDatabase {
|
||||
return {
|
||||
id: "11111111-1111-4111-8111-111111111111",
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
binding,
|
||||
version: 4,
|
||||
connectionStatus: "reachable",
|
||||
testedVersion: 4,
|
||||
createdAt: "2026-08-27T08:00:00Z",
|
||||
updatedAt: "2026-08-27T09:00:00Z",
|
||||
};
|
||||
}
|
||||
|
||||
test("reads columns, ordered composite keys, and physical relationships from one PostgreSQL connection", async () => {
|
||||
const query = vi.fn()
|
||||
.mockResolvedValueOnce({ rows: [{ present: true }] })
|
||||
.mockResolvedValueOnce({ rows: [{ name: "visits", source_comment: "Visits" }] })
|
||||
.mockResolvedValueOnce({ rows: [
|
||||
{ table_name: "visits", name: "tenant_id", ordinal_position: 1, data_type: "uuid", is_nullable: false, default_expression: null, primary_key_position: 1, source_comment: null },
|
||||
{ table_name: "visits", name: "patient_id", ordinal_position: 2, data_type: "bigint", is_nullable: false, default_expression: null, primary_key_position: 2, source_comment: "Patient" },
|
||||
] })
|
||||
.mockResolvedValueOnce({ rows: [
|
||||
{ constraint_name: "visits_patient_fkey", source_table_name: "visits", target_table_name: "patients", update_action: "a", delete_action: "c", deferrable: true, initially_deferred: false, position: 1, source_column_name: "tenant_id", target_column_name: "tenant_id" },
|
||||
{ constraint_name: "visits_patient_fkey", source_table_name: "visits", target_table_name: "patients", update_action: "a", delete_action: "c", deferrable: true, initially_deferred: false, position: 2, source_column_name: "patient_id", target_column_name: "id" },
|
||||
] });
|
||||
const end = vi.fn(async () => undefined);
|
||||
const client: CatalogDatabaseClient = { query, end };
|
||||
const postgres: CatalogPostgresAccess = { connect: vi.fn(async () => client) };
|
||||
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
|
||||
|
||||
const result = await introspector.scan(database({
|
||||
transport: "ssh_tunnel",
|
||||
username: "reader",
|
||||
sshHost: "bastion.internal",
|
||||
sshPort: 22,
|
||||
sshUsername: "tunnel",
|
||||
sshTargetHost: "db.internal",
|
||||
sshTargetPort: 5432,
|
||||
}), new AbortController().signal);
|
||||
|
||||
expect(result.capabilities).toEqual({ tables: "available", columns: "available", relationships: "available" });
|
||||
expect(result.columns).toMatchObject([
|
||||
{ name: "tenant_id", primaryKeyPosition: 1, isNullable: false },
|
||||
{ name: "patient_id", primaryKeyPosition: 2, sourceComment: "Patient" },
|
||||
]);
|
||||
expect(result.relationships).toEqual([expect.objectContaining({
|
||||
constraintName: "visits_patient_fkey",
|
||||
updateRule: "NO ACTION",
|
||||
deleteRule: "CASCADE",
|
||||
deferrable: true,
|
||||
columns: [
|
||||
{ position: 1, sourceColumnName: "tenant_id", targetColumnName: "tenant_id" },
|
||||
{ position: 2, sourceColumnName: "patient_id", targetColumnName: "id" },
|
||||
],
|
||||
})]);
|
||||
expect(query.mock.calls[2][0]).toContain("format_type");
|
||||
expect(query.mock.calls[3][0]).toContain("WITH ORDINALITY");
|
||||
expect(query.mock.calls.slice(1).every((call) => call[1][0] === "datawarehouse")).toBe(true);
|
||||
expect(end).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
test("uses the typed full REST snapshot RPC and preserves explicit capability unavailability", async () => {
|
||||
const response = {
|
||||
schemaVersion: 1,
|
||||
capabilities: { tables: "available", columns: "unavailable", relationships: "unavailable" },
|
||||
tables: [{ name: "patients", sourceComment: null }],
|
||||
columns: [],
|
||||
relationships: [],
|
||||
};
|
||||
const fetchMock = vi.fn(async () => new Response(JSON.stringify(response), { status: 200, headers: { "content-type": "application/json" } }));
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
const postgres: CatalogPostgresAccess = { connect: vi.fn(async () => { throw new Error("wire access must not be used"); }) };
|
||||
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
|
||||
|
||||
const result = await introspector.scan(database({
|
||||
transport: "rest_api", baseUrl: "https://connector.internal/api/", restPath: "/health", restAuth: "none",
|
||||
}), new AbortController().signal);
|
||||
|
||||
expect(result).toEqual(response);
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
"https://connector.internal/api/rpc/schema_snapshot",
|
||||
expect.objectContaining({ method: "POST", body: JSON.stringify({ schema_name: "datawarehouse" }) }),
|
||||
);
|
||||
});
|
||||
|
||||
test("falls back to one read-only REST query when the snapshot RPC is absent", async () => {
|
||||
const response = {
|
||||
schemaVersion: 1 as const,
|
||||
capabilities: { tables: "available" as const, columns: "available" as const, relationships: "available" as const },
|
||||
tables: [
|
||||
{ name: "patients", sourceComment: "Clinical patients" },
|
||||
{ name: "visits", sourceComment: null },
|
||||
],
|
||||
columns: [
|
||||
{ tableName: "patients", name: "tenant_id", ordinalPosition: 1, dataType: "uuid", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: "Tenant" },
|
||||
{ tableName: "patients", name: "id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: "nextval('patients_id_seq'::regclass)", primaryKeyPosition: 2, sourceComment: null },
|
||||
{ tableName: "visits", name: "tenant_id", ordinalPosition: 1, dataType: "uuid", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
|
||||
{ tableName: "visits", name: "patient_id", ordinalPosition: 2, dataType: "bigint", isNullable: true, defaultExpression: null, primaryKeyPosition: null, sourceComment: "Owning patient" },
|
||||
],
|
||||
relationships: [{
|
||||
constraintName: "visits_patient_fkey",
|
||||
sourceTableName: "visits",
|
||||
targetTableName: "patients",
|
||||
updateRule: "CASCADE",
|
||||
deleteRule: "RESTRICT",
|
||||
deferrable: true,
|
||||
initiallyDeferred: false,
|
||||
columns: [
|
||||
{ position: 1, sourceColumnName: "tenant_id", targetColumnName: "tenant_id" },
|
||||
{ position: 2, sourceColumnName: "patient_id", targetColumnName: "id" },
|
||||
],
|
||||
}],
|
||||
};
|
||||
const fetchMock = vi.fn()
|
||||
.mockResolvedValueOnce(new Response(null, { status: 404 }))
|
||||
.mockResolvedValueOnce(new Response(JSON.stringify([response]), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
}));
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
const postgres: CatalogPostgresAccess = {
|
||||
connect: vi.fn(async () => { throw new Error("wire access must not be used"); }),
|
||||
};
|
||||
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
|
||||
|
||||
const result = await introspector.scan(database({
|
||||
transport: "rest_api",
|
||||
baseUrl: "https://connector.internal/api/",
|
||||
restPath: "/health",
|
||||
restAuth: "none",
|
||||
}), new AbortController().signal);
|
||||
|
||||
expect(result).toEqual(response);
|
||||
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||
expect(fetchMock.mock.calls[0]).toEqual([
|
||||
"https://connector.internal/api/rpc/schema_snapshot",
|
||||
expect.objectContaining({ method: "POST", body: JSON.stringify({ schema_name: "datawarehouse" }) }),
|
||||
]);
|
||||
expect(fetchMock.mock.calls[1][0]).toBe("https://connector.internal/api/rpc/run_query");
|
||||
const fallbackRequest = fetchMock.mock.calls[1][1] as RequestInit;
|
||||
expect(fallbackRequest).toMatchObject({ method: "POST" });
|
||||
const fallbackBody = JSON.parse(String(fallbackRequest.body)) as { query_text: string };
|
||||
expect(Object.keys(fallbackBody)).toEqual(["query_text"]);
|
||||
expect(fallbackBody.query_text).toMatch(/^\s*WITH\b/);
|
||||
expect(fallbackBody.query_text).toContain("pg_catalog.pg_constraint");
|
||||
expect(fallbackBody.query_text).not.toMatch(/\b(INSERT|UPDATE|DROP|ALTER|CREATE|TRUNCATE)\b/i);
|
||||
});
|
||||
|
||||
test("classifies a missing REST snapshot RPC as an explicit binding capability", async () => {
|
||||
vi.stubGlobal("fetch", vi.fn(async () => new Response(null, { status: 404 })));
|
||||
const postgres: CatalogPostgresAccess = {
|
||||
connect: vi.fn(async () => { throw new Error("wire access must not be used"); }),
|
||||
};
|
||||
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
|
||||
|
||||
const scan = introspector.scan(database({
|
||||
transport: "rest_api",
|
||||
baseUrl: "https://connector.internal/api/",
|
||||
restPath: "/health",
|
||||
restAuth: "none",
|
||||
}), new AbortController().signal);
|
||||
|
||||
await expect(scan).rejects.toMatchObject<CatalogSchemaCapabilityUnavailableError>({
|
||||
capability: "schema_snapshot",
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,224 @@
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, expect, test, vi } from "vitest";
|
||||
import { buildApp } from "../src/app.js";
|
||||
import { loadConfig } from "../src/config.js";
|
||||
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
|
||||
import type { CatalogSchemaIntrospector } from "../src/catalog/schema-introspector.js";
|
||||
import type { CatalogSyncRun, ObservedSchemaSnapshot } from "../src/catalog/types.js";
|
||||
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||
import type { WorkspaceRegistry, WorkspaceRevision } from "../src/workspaces/registry.js";
|
||||
import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
|
||||
|
||||
const workspace: WorkspaceDescriptor = {
|
||||
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
|
||||
dwh: { engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432, supported_transports: ["postgres_direct"] },
|
||||
semantic_index: {
|
||||
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
|
||||
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
|
||||
},
|
||||
llm_policy: { allowed: ["zai/glm-5.2"] },
|
||||
};
|
||||
const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" };
|
||||
|
||||
function snapshot(): ObservedSchemaSnapshot {
|
||||
return {
|
||||
schemaVersion: 1,
|
||||
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||
tables: [
|
||||
{ name: "patients", sourceComment: "Clinical patients" },
|
||||
{ name: "visits", sourceComment: "Patient visits" },
|
||||
],
|
||||
columns: [
|
||||
{ tableName: "patients", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: "Patient key" },
|
||||
{ tableName: "visits", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
|
||||
{ tableName: "visits", name: "patient_id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: "Owning patient" },
|
||||
],
|
||||
relationships: [{
|
||||
constraintName: "visits_patient_id_fkey",
|
||||
sourceTableName: "visits",
|
||||
targetTableName: "patients",
|
||||
updateRule: "NO ACTION",
|
||||
deleteRule: "CASCADE",
|
||||
deferrable: false,
|
||||
initiallyDeferred: false,
|
||||
columns: [{ position: 1, sourceColumnName: "patient_id", targetColumnName: "id" }],
|
||||
}],
|
||||
};
|
||||
}
|
||||
|
||||
async function waitFor(repository: MemoryCatalogRepository, runId: string, state: CatalogSyncRun["state"]): Promise<CatalogSyncRun> {
|
||||
for (let attempt = 0; attempt < 100; attempt += 1) {
|
||||
const run = await repository.getSyncRun(runId);
|
||||
if (run?.state === state) return run;
|
||||
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||
}
|
||||
throw new Error(`Run ${runId} did not reach ${state}`);
|
||||
}
|
||||
|
||||
async function setup() {
|
||||
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-schema-secret-"));
|
||||
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-schema-runtime-"));
|
||||
roots.push(secretRoot, runtimeRoot);
|
||||
const repository = new MemoryCatalogRepository();
|
||||
const created = await repository.create({
|
||||
workspaceId: "psd-clinical", engine: "postgres", databaseName: "warehouse", schema: "datawarehouse",
|
||||
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||
});
|
||||
await repository.recordTest(created.id, created.version, {
|
||||
connectionStatus: "reachable", testedVersion: created.version, lastTestedAt: new Date().toISOString(),
|
||||
});
|
||||
let observed = snapshot();
|
||||
const scan = vi.fn(async (_database, _signal, progress) => {
|
||||
await progress?.("connecting");
|
||||
await progress?.("scanning_tables", { tables: observed.tables.length });
|
||||
await progress?.("scanning_columns", { tables: observed.tables.length, columns: observed.columns.length });
|
||||
await progress?.("scanning_relationships", { relationships: observed.relationships.length });
|
||||
return structuredClone(observed);
|
||||
});
|
||||
const introspector: CatalogSchemaIntrospector = { scan };
|
||||
const registry = {
|
||||
list: vi.fn(async () => [revision]),
|
||||
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
|
||||
read: vi.fn(async () => ({ workspace, revision })),
|
||||
} as unknown as WorkspaceRegistry;
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
|
||||
thtRunner: {} as never,
|
||||
workspaceRegistry: registry,
|
||||
workspaceSecretStore: new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" }),
|
||||
catalogRepository: repository,
|
||||
catalogSchemaIntrospector: introspector,
|
||||
workspaceDiagnoser: vi.fn(),
|
||||
});
|
||||
return {
|
||||
app, repository, database: (await repository.get(created.id))!, scan,
|
||||
setObserved(next: ObservedSchemaSnapshot) { observed = next; },
|
||||
};
|
||||
}
|
||||
|
||||
test("synchronizes a full physical schema and derives primary and foreign key flags", async () => {
|
||||
const { app, repository, database } = await setup();
|
||||
const started = await app.inject({
|
||||
method: "POST", url: `/catalog/databases/${database.id}/sync-runs`,
|
||||
payload: { version: database.version, scope: "all", tableIds: [] },
|
||||
});
|
||||
expect(started.statusCode).toBe(202);
|
||||
const completed = await waitFor(repository, started.json().id, "succeeded");
|
||||
expect(completed.counts).toMatchObject({ tables: 2, columns: 3, relationships: 1 });
|
||||
|
||||
const tables = await repository.listTables(database.id);
|
||||
const visits = tables.find((table) => table.name === "visits")!;
|
||||
const columns = (await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables/${visits.id}/columns` })).json();
|
||||
expect(columns).toMatchObject([
|
||||
{ name: "id", isPrimaryKey: true, primaryKeyPosition: 1, isForeignKey: false },
|
||||
{ name: "patient_id", isPrimaryKey: false, isForeignKey: true, foreignKeyCount: 1 },
|
||||
]);
|
||||
const relationships = (await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/relationships` })).json();
|
||||
expect(relationships).toMatchObject([{ constraintName: "visits_patient_id_fkey", columns: [{ sourceColumnName: "patient_id", targetColumnName: "id" }] }]);
|
||||
expect((await repository.get(database.id))?.schemaSyncedVersion).toBe(database.version);
|
||||
});
|
||||
|
||||
test("synchronizes columns for every catalog table when no table selection is supplied", async () => {
|
||||
const { app, repository, database, setObserved } = await setup();
|
||||
const tablesRun = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/sync-runs`,
|
||||
payload: { version: database.version, scope: "tables", tableIds: [] },
|
||||
});
|
||||
expect(tablesRun.statusCode).toBe(202);
|
||||
await waitFor(repository, tablesRun.json().id, "succeeded");
|
||||
const tables = await repository.listTables(database.id);
|
||||
expect(tables.map((table) => table.name)).toEqual(["patients", "visits"]);
|
||||
|
||||
const columnsRun = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/sync-runs`,
|
||||
payload: { version: database.version, scope: "columns", tableIds: [] },
|
||||
});
|
||||
expect(columnsRun.statusCode).toBe(202);
|
||||
await waitFor(repository, columnsRun.json().id, "succeeded");
|
||||
const patients = tables.find((table) => table.name === "patients")!;
|
||||
const visits = tables.find((table) => table.name === "visits")!;
|
||||
expect((await repository.listColumns(database.id, patients.id)).map((column) => column.name)).toEqual(["id"]);
|
||||
expect((await repository.listColumns(database.id, visits.id)).map((column) => column.name)).toEqual(["id", "patient_id"]);
|
||||
|
||||
const next = snapshot();
|
||||
next.columns = next.columns.filter((column) => column.name !== "id");
|
||||
setObserved(next);
|
||||
const selectedDestructiveRun = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/sync-runs`,
|
||||
payload: { version: database.version, scope: "columns", tableIds: [patients.id] },
|
||||
});
|
||||
expect(selectedDestructiveRun.statusCode).toBe(202);
|
||||
const selectedWaiting = await waitFor(repository, selectedDestructiveRun.json().id, "awaiting_confirmation");
|
||||
expect(selectedWaiting.plannedDiff?.deletedColumns).toEqual([
|
||||
{ tableName: "patients", columnName: "id" },
|
||||
]);
|
||||
expect((await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/sync-runs/${selectedWaiting.id}/cancel`,
|
||||
})).statusCode).toBe(200);
|
||||
|
||||
const destructiveRun = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/sync-runs`,
|
||||
payload: { version: database.version, scope: "columns", tableIds: [] },
|
||||
});
|
||||
expect(destructiveRun.statusCode).toBe(202);
|
||||
const waiting = await waitFor(repository, destructiveRun.json().id, "awaiting_confirmation");
|
||||
expect(waiting.plannedDiff?.deletedColumns).toEqual([
|
||||
{ tableName: "patients", columnName: "id" },
|
||||
{ tableName: "visits", columnName: "id" },
|
||||
]);
|
||||
});
|
||||
|
||||
test("keeps generated descriptions editable and preserves them across synchronization", async () => {
|
||||
const { app, repository, database } = await setup();
|
||||
const first = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
|
||||
await waitFor(repository, first.json().id, "succeeded");
|
||||
const patients = (await repository.listTables(database.id)).find((table) => table.name === "patients")!;
|
||||
const editedTable = await app.inject({
|
||||
method: "PATCH", url: `/catalog/databases/${database.id}/tables/${patients.id}`,
|
||||
payload: { version: patients.version, description: null, generatedDescription: "Generated table draft" },
|
||||
});
|
||||
expect(editedTable.json()).toMatchObject({ description: null, generatedDescription: "Generated table draft" });
|
||||
const idColumn = (await repository.listColumns(database.id, patients.id))[0];
|
||||
const editedColumn = await app.inject({
|
||||
method: "PATCH", url: `/catalog/databases/${database.id}/tables/${patients.id}/columns/${idColumn.id}`,
|
||||
payload: { version: idColumn.version, description: "Reviewed key", generatedDescription: "Generated key draft" },
|
||||
});
|
||||
expect(editedColumn.json()).toMatchObject({ description: "Reviewed key", generatedDescription: "Generated key draft" });
|
||||
|
||||
const second = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
|
||||
await waitFor(repository, second.json().id, "succeeded");
|
||||
expect(await repository.getTable(database.id, patients.id)).toMatchObject({ generatedDescription: "Generated table draft" });
|
||||
expect(await repository.getColumn(database.id, patients.id, idColumn.id)).toMatchObject({ description: "Reviewed key", generatedDescription: "Generated key draft" });
|
||||
});
|
||||
|
||||
test("waits for confirmation and rescans before applying destructive changes", async () => {
|
||||
const { app, repository, database, scan, setObserved } = await setup();
|
||||
const first = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
|
||||
await waitFor(repository, first.json().id, "succeeded");
|
||||
const next = snapshot();
|
||||
next.tables = next.tables.filter((table) => table.name !== "visits");
|
||||
next.columns = next.columns.filter((column) => column.tableName !== "visits");
|
||||
next.relationships = [];
|
||||
setObserved(next);
|
||||
|
||||
const destructive = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
|
||||
const waiting = await waitFor(repository, destructive.json().id, "awaiting_confirmation");
|
||||
expect(waiting.plannedDiff).toMatchObject({ deletedTables: ["visits"] });
|
||||
expect(await repository.listTables(database.id)).toHaveLength(2);
|
||||
const confirmed = await app.inject({
|
||||
method: "POST", url: `/catalog/sync-runs/${waiting.id}/confirm`, payload: { confirmationToken: waiting.confirmationToken },
|
||||
});
|
||||
expect(confirmed.statusCode).toBe(200);
|
||||
await waitFor(repository, waiting.id, "succeeded");
|
||||
expect((await repository.listTables(database.id)).map((table) => table.name)).toEqual(["patients"]);
|
||||
expect(scan).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
@@ -0,0 +1,124 @@
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, expect, test, vi } from "vitest";
|
||||
import type {
|
||||
CatalogDatabaseClient,
|
||||
CatalogPostgresAccess,
|
||||
} from "../src/catalog/postgres-access.js";
|
||||
import { CATALOG_SECRET_IDS } from "../src/catalog/secrets.js";
|
||||
import { ConcreteCatalogTableIntrospector } from "../src/catalog/table-introspector.js";
|
||||
import type { WorkspaceDatabase } from "../src/catalog/types.js";
|
||||
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||
|
||||
const roots: string[] = [];
|
||||
|
||||
afterEach(() => {
|
||||
vi.unstubAllGlobals();
|
||||
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function secretStore() {
|
||||
const root = mkdtempSync(join(tmpdir(), "catalog-table-introspection-secrets-"));
|
||||
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-table-introspection-runtime-"));
|
||||
roots.push(root, runtimeRoot);
|
||||
return new WorkspaceSecretStore({ root, runtimeRoot, installationId: "test" });
|
||||
}
|
||||
|
||||
function database(binding: WorkspaceDatabase["binding"]): WorkspaceDatabase {
|
||||
return {
|
||||
id: "11111111-1111-4111-8111-111111111111",
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
version: 4,
|
||||
createdAt: "2026-08-27T08:00:00Z",
|
||||
updatedAt: "2026-08-27T09:00:00Z",
|
||||
connectionStatus: "reachable",
|
||||
binding,
|
||||
};
|
||||
}
|
||||
|
||||
test("reads only ordinary and partitioned PostgreSQL tables from the configured schema", async () => {
|
||||
const query = vi.fn()
|
||||
.mockResolvedValueOnce({ rows: [{ present: true }] })
|
||||
.mockResolvedValueOnce({ rows: [
|
||||
{ name: "visits", source_comment: null },
|
||||
{ name: "patients", source_comment: "Clinical patients" },
|
||||
] });
|
||||
const end = vi.fn(async () => undefined);
|
||||
const client: CatalogDatabaseClient = { query, end };
|
||||
const postgres: CatalogPostgresAccess = { connect: vi.fn(async () => client) };
|
||||
const introspector = new ConcreteCatalogTableIntrospector(postgres, secretStore());
|
||||
|
||||
const tables = await introspector.scan(database({
|
||||
transport: "postgres_direct",
|
||||
host: "db.internal",
|
||||
port: 5432,
|
||||
username: "reader",
|
||||
}), new AbortController().signal);
|
||||
|
||||
expect(tables).toEqual([
|
||||
{ name: "patients", sourceComment: "Clinical patients" },
|
||||
{ name: "visits", sourceComment: null },
|
||||
]);
|
||||
expect(query.mock.calls[1][0]).toContain("c.relkind IN ('r', 'p')");
|
||||
expect(query.mock.calls[1][0]).not.toContain("'v'");
|
||||
expect(query.mock.calls[1][1]).toEqual(["datawarehouse"]);
|
||||
expect(end).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
test("uses the typed REST table RPC and ignores non-table objects", async () => {
|
||||
const store = secretStore();
|
||||
store.put("psd-clinical", CATALOG_SECRET_IDS.apiKey, "rest-secret");
|
||||
const fetchMock = vi.fn(async () => new Response(JSON.stringify([
|
||||
{ type: "VIEW", table: "patient_view", comment: "Not a table" },
|
||||
{ type: "TABLE", table: "visits", comment: null },
|
||||
{ type: "TABLE", table: "patients", comment: "Clinical patients" },
|
||||
]), { status: 200, headers: { "content-type": "application/json" } }));
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
const postgres: CatalogPostgresAccess = {
|
||||
connect: vi.fn(async () => { throw new Error("PostgreSQL wire access must not be used"); }),
|
||||
};
|
||||
const introspector = new ConcreteCatalogTableIntrospector(postgres, store);
|
||||
|
||||
const tables = await introspector.scan(database({
|
||||
transport: "rest_api",
|
||||
baseUrl: "https://connector.internal/api/",
|
||||
restPath: "/health",
|
||||
restAuth: "x-api-key",
|
||||
}), new AbortController().signal);
|
||||
|
||||
expect(tables).toEqual([
|
||||
{ name: "patients", sourceComment: "Clinical patients" },
|
||||
{ name: "visits", sourceComment: null },
|
||||
]);
|
||||
expect(fetchMock).toHaveBeenCalledWith(
|
||||
"https://connector.internal/api/rpc/list_tables",
|
||||
expect.objectContaining({
|
||||
method: "POST",
|
||||
headers: expect.objectContaining({ "x-api-key": "rest-secret" }),
|
||||
body: JSON.stringify({ schema_name: "datawarehouse" }),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
test("fails closed when a REST table row violates the typed contract", async () => {
|
||||
const store = secretStore();
|
||||
const fetchMock = vi.fn(async () => new Response(JSON.stringify([
|
||||
{ type: "TABLE", table_name: "patients", comment: "Wrong field name" },
|
||||
]), { status: 200, headers: { "content-type": "application/json" } }));
|
||||
vi.stubGlobal("fetch", fetchMock);
|
||||
const postgres: CatalogPostgresAccess = {
|
||||
connect: vi.fn(async () => { throw new Error("PostgreSQL wire access must not be used"); }),
|
||||
};
|
||||
const introspector = new ConcreteCatalogTableIntrospector(postgres, store);
|
||||
|
||||
await expect(introspector.scan(database({
|
||||
transport: "rest_api",
|
||||
baseUrl: "https://connector.internal/api",
|
||||
restPath: "/health",
|
||||
restAuth: "none",
|
||||
}), new AbortController().signal)).rejects.toThrow("REST schema response is invalid");
|
||||
});
|
||||
@@ -0,0 +1,126 @@
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, expect, test, vi } from "vitest";
|
||||
import { buildApp } from "../src/app.js";
|
||||
import { loadConfig } from "../src/config.js";
|
||||
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
|
||||
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||
import type { WorkspaceRegistry, WorkspaceRevision } from "../src/workspaces/registry.js";
|
||||
import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
|
||||
|
||||
const workspace: WorkspaceDescriptor = {
|
||||
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
|
||||
dwh: {
|
||||
engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432,
|
||||
supported_transports: ["postgres_direct", "rest_api"],
|
||||
},
|
||||
semantic_index: {
|
||||
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
|
||||
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
|
||||
},
|
||||
llm_policy: { allowed: ["zai/glm-5.2"] },
|
||||
diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } },
|
||||
};
|
||||
const revision: WorkspaceRevision = {
|
||||
id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml",
|
||||
};
|
||||
|
||||
async function setup() {
|
||||
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-table-secret-"));
|
||||
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-table-runtime-"));
|
||||
roots.push(secretRoot, runtimeRoot);
|
||||
const repository = new MemoryCatalogRepository();
|
||||
const database = await repository.create({
|
||||
workspaceId: "psd-clinical",
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||
});
|
||||
await repository.recordTest(database.id, database.version, {
|
||||
connectionStatus: "reachable",
|
||||
testedVersion: database.version,
|
||||
lastTestedAt: new Date().toISOString(),
|
||||
});
|
||||
const scan = vi.fn(async () => [
|
||||
{ name: "patients", sourceComment: "Clinical patients" },
|
||||
{ name: "visits", sourceComment: null },
|
||||
]);
|
||||
const registry = {
|
||||
list: vi.fn(async () => [revision]),
|
||||
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
|
||||
read: vi.fn(async () => ({ workspace, revision })),
|
||||
} as unknown as WorkspaceRegistry;
|
||||
const secretStore = new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" });
|
||||
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
|
||||
thtRunner: {} as never,
|
||||
workspaceRegistry: registry,
|
||||
workspaceSecretStore: secretStore,
|
||||
catalogRepository: repository,
|
||||
workspaceDiagnoser: vi.fn(),
|
||||
});
|
||||
return { app, repository, database: (await repository.get(database.id))!, scan };
|
||||
}
|
||||
|
||||
test("lists physical tables and updates only review metadata", async () => {
|
||||
const { app, repository, database, scan } = await setup();
|
||||
const synchronized = await repository.reconcileTables(database.id, database.version, await scan(), []);
|
||||
expect(synchronized).toMatchObject({ kind: "applied", createdCount: 2, deletedCount: 0 });
|
||||
expect(scan).toHaveBeenCalledOnce();
|
||||
|
||||
const tables = (await app.inject({
|
||||
method: "GET", url: `/catalog/databases/${database.id}/tables`,
|
||||
})).json();
|
||||
expect(tables.map((table: { name: string }) => table.name)).toEqual(["patients", "visits"]);
|
||||
const patients = tables[0];
|
||||
const edited = await app.inject({
|
||||
method: "PATCH",
|
||||
url: `/catalog/databases/${database.id}/tables/${patients.id}`,
|
||||
payload: { version: patients.version, description: "Curated patient registry" },
|
||||
});
|
||||
expect(edited.statusCode).toBe(200);
|
||||
expect(edited.json()).toMatchObject({
|
||||
name: "patients",
|
||||
sourceComment: "Clinical patients",
|
||||
description: "Curated patient registry",
|
||||
version: 2,
|
||||
});
|
||||
});
|
||||
|
||||
test("requires an exact deletion confirmation before applying the atomic diff", async () => {
|
||||
const { app, repository, database, scan } = await setup();
|
||||
await repository.reconcileTables(database.id, database.version, await scan(), []);
|
||||
scan.mockResolvedValue([{ name: "patients", sourceComment: "Clinical patients" }]);
|
||||
|
||||
const preview = await repository.reconcileTables(database.id, database.version, await scan(), []);
|
||||
expect(preview).toEqual({ kind: "confirmation_required", deletedNames: ["visits"] });
|
||||
expect((await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables` })).json()).toHaveLength(2);
|
||||
|
||||
const applied = await repository.reconcileTables(database.id, database.version, await scan(), ["visits"]);
|
||||
expect(applied).toMatchObject({ kind: "applied", deletedCount: 1 });
|
||||
expect((await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables` })).json()).toMatchObject([
|
||||
{ name: "patients" },
|
||||
]);
|
||||
});
|
||||
|
||||
test("refuses synchronization until the current binding has passed its connection test", async () => {
|
||||
const { app, repository, database } = await setup();
|
||||
await repository.update(database.id, database.version, {
|
||||
workspaceId: database.workspaceId,
|
||||
engine: database.engine,
|
||||
databaseName: database.databaseName,
|
||||
schema: database.schema,
|
||||
binding: database.binding,
|
||||
});
|
||||
const response = await app.inject({
|
||||
method: "POST",
|
||||
url: `/catalog/databases/${database.id}/sync-runs`,
|
||||
payload: { version: database.version + 1, scope: "tables", tableIds: [] },
|
||||
});
|
||||
expect(response.statusCode).toBe(409);
|
||||
expect(response.json()).toMatchObject({ code: "schema_sync_conflict" });
|
||||
});
|
||||
@@ -289,3 +289,22 @@ test("loadConfig accepts only an absolute generic model key file", () => {
|
||||
expect(() => loadConfig({ THT_MODEL_API_KEY_FILE: " /run/secrets/key" }))
|
||||
.toThrow(/model credential configuration is invalid/);
|
||||
});
|
||||
|
||||
test("loadConfig accepts a file-backed catalog role and rejects partial catalog configuration", () => {
|
||||
expect(loadConfig({
|
||||
THT_CATALOG_DB_HOST: "catalog-db",
|
||||
THT_CATALOG_DB_NAME: "thothii_catalog",
|
||||
THT_CATALOG_RUNTIME_USER: "thothii_catalog_runtime",
|
||||
THT_CATALOG_RUNTIME_PASSWORD_FILE: "/run/secrets/catalog_runtime_password",
|
||||
}).catalogDatabase).toEqual({
|
||||
host: "catalog-db",
|
||||
port: 5432,
|
||||
database: "thothii_catalog",
|
||||
user: "thothii_catalog_runtime",
|
||||
passwordFile: "/run/secrets/catalog_runtime_password",
|
||||
});
|
||||
expect(() => loadConfig({ THT_CATALOG_DB_HOST: "catalog-db" }))
|
||||
.toThrow(/catalog database configuration is invalid/);
|
||||
expect(() => loadConfig({ THT_CATALOG_DATABASE_URL: "https://catalog.invalid/db" }))
|
||||
.toThrow(/catalog database configuration is invalid/);
|
||||
});
|
||||
|
||||
@@ -25,6 +25,11 @@ services:
|
||||
THT_PI_AUTH_FILE: /home/thoth/.pi/agent/auth.json
|
||||
THT_AUTH_CONFIG_FILE: /run/thothii-auth/auth.yaml
|
||||
THT_AUTH_STATE_ROOT: /data/auth
|
||||
THT_CATALOG_DB_HOST: catalog-db
|
||||
THT_CATALOG_DB_PORT: "5432"
|
||||
THT_CATALOG_DB_NAME: thothii_catalog
|
||||
THT_CATALOG_RUNTIME_USER: thothii_catalog_runtime
|
||||
THT_CATALOG_RUNTIME_PASSWORD_FILE: /run/secrets/catalog_runtime_password
|
||||
THT_DB_NAME: ${THT_DB_NAME:-}
|
||||
THT_DWH_REST_URL: ${THT_DWH_REST_URL:-}
|
||||
THT_LLM_URL: ${THT_LLM_URL:-}
|
||||
@@ -47,6 +52,7 @@ services:
|
||||
secrets:
|
||||
- source: thothii_secrets
|
||||
target: thothii.secrets
|
||||
- catalog_runtime_password
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"]
|
||||
interval: 15s
|
||||
@@ -54,6 +60,8 @@ services:
|
||||
retries: 5
|
||||
start_period: 30s
|
||||
depends_on:
|
||||
catalog-db:
|
||||
condition: service_healthy
|
||||
qdrant:
|
||||
condition: service_healthy
|
||||
embedding-model-init:
|
||||
@@ -61,6 +69,51 @@ services:
|
||||
networks:
|
||||
- thothii
|
||||
|
||||
catalog-db:
|
||||
image: postgres:17.6-bookworm@sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3
|
||||
environment:
|
||||
POSTGRES_DB: thothii_catalog
|
||||
POSTGRES_USER: thothii_catalog_migrate
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/catalog_migrator_password
|
||||
volumes:
|
||||
- catalog-data:/var/lib/postgresql/data
|
||||
- ./docker/catalog-db-init.sql:/docker-entrypoint-initdb.d/010-runtime-role.sql:ro
|
||||
secrets:
|
||||
- catalog_runtime_password
|
||||
- catalog_migrator_password
|
||||
healthcheck:
|
||||
test:
|
||||
- CMD-SHELL
|
||||
- >-
|
||||
pg_isready -U thothii_catalog_migrate -d thothii_catalog
|
||||
&& test "$(psql -U thothii_catalog_migrate -d thothii_catalog -Atqc
|
||||
"select count(*) from pg_catalog.pg_roles where rolname = 'thothii_catalog_runtime'")" = "1"
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 12
|
||||
start_period: 10s
|
||||
networks:
|
||||
- thothii
|
||||
|
||||
catalog-migrate:
|
||||
image: thothii-core:local
|
||||
profiles: [catalog-maintenance]
|
||||
pull_policy: never
|
||||
command: ["node", "/app/backend/dist/catalog/migrate.js"]
|
||||
environment:
|
||||
THT_CATALOG_DB_HOST: catalog-db
|
||||
THT_CATALOG_DB_PORT: "5432"
|
||||
THT_CATALOG_DB_NAME: thothii_catalog
|
||||
THT_CATALOG_MIGRATOR_USER: thothii_catalog_migrate
|
||||
THT_CATALOG_MIGRATOR_PASSWORD_FILE: /run/secrets/catalog_migrator_password
|
||||
secrets:
|
||||
- catalog_migrator_password
|
||||
depends_on:
|
||||
catalog-db:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- thothii
|
||||
|
||||
workspace-maintenance:
|
||||
image: thothii-core:local
|
||||
profiles: [workspace-maintenance]
|
||||
@@ -209,7 +262,12 @@ volumes:
|
||||
qdrant-data:
|
||||
embedding-models:
|
||||
auth-state:
|
||||
catalog-data:
|
||||
|
||||
secrets:
|
||||
thothii_secrets:
|
||||
file: "${THT_SECRETS_FILE:?set THT_SECRETS_FILE}"
|
||||
catalog_runtime_password:
|
||||
file: "${THT_CATALOG_RUNTIME_PASSWORD_SOURCE:-./deploy/secrets/catalog-runtime-password}"
|
||||
catalog_migrator_password:
|
||||
file: "${THT_CATALOG_MIGRATOR_PASSWORD_SOURCE:-./deploy/secrets/catalog-migrator-password}"
|
||||
|
||||
Vendored
+3
@@ -7,6 +7,9 @@ MAX_PI_PROCESSES=4
|
||||
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
|
||||
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
|
||||
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
|
||||
THT_CATALOG_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/catalog-runtime-password
|
||||
THT_CATALOG_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/catalog-migrator-password
|
||||
THT_CATALOG_SYNC_TIMEOUT_MS=600000
|
||||
|
||||
THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git
|
||||
THT_WORKSPACE_GIT_BRANCH=main
|
||||
|
||||
Vendored
+3
@@ -6,6 +6,9 @@ MAX_PI_PROCESSES=4
|
||||
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
|
||||
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
|
||||
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
|
||||
THT_CATALOG_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/catalog-runtime-password
|
||||
THT_CATALOG_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/catalog-migrator-password
|
||||
THT_CATALOG_SYNC_TIMEOUT_MS=600000
|
||||
|
||||
THT_DATA_ROOT=/srv/thothii/data
|
||||
THT_PI_STATE_ROOT=/srv/thothii/pi-state
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
DO $bootstrap$
|
||||
DECLARE
|
||||
runtime_password text := trim(both E'\r\n' from pg_read_file('/run/secrets/catalog_runtime_password'));
|
||||
BEGIN
|
||||
IF runtime_password = '' THEN
|
||||
RAISE EXCEPTION 'catalog runtime password is empty';
|
||||
END IF;
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = 'thothii_catalog_runtime'
|
||||
) THEN
|
||||
EXECUTE format(
|
||||
'CREATE ROLE thothii_catalog_runtime LOGIN PASSWORD %L',
|
||||
runtime_password
|
||||
);
|
||||
END IF;
|
||||
END
|
||||
$bootstrap$;
|
||||
|
||||
GRANT CONNECT ON DATABASE thothii_catalog TO thothii_catalog_runtime;
|
||||
GRANT USAGE ON SCHEMA public TO thothii_catalog_runtime;
|
||||
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||||
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO thothii_catalog_runtime;
|
||||
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||||
GRANT USAGE, SELECT ON SEQUENCES TO thothii_catalog_runtime;
|
||||
@@ -96,7 +96,9 @@ RUN ln -s /opt/venv /app/harness/.venv
|
||||
# Backend: dist + node_modules (stesso Node major 24 + glibc bookworm → compatibili)
|
||||
COPY --from=backend-build /src/backend/dist /app/backend/dist
|
||||
COPY --from=backend-build /src/backend/node_modules /app/backend/node_modules
|
||||
COPY backend/scripts/ssh-askpass.mjs /app/backend/scripts/ssh-askpass.mjs
|
||||
COPY backend/package*.json /app/backend/
|
||||
RUN chmod 0755 /app/backend/scripts/ssh-askpass.mjs
|
||||
|
||||
# Runtime Pi is installed only from the committed lockfile. The image exposes its immutable
|
||||
# executable directly, so no host Pi installation or writable global npm directory is needed.
|
||||
|
||||
@@ -2,10 +2,12 @@
|
||||
|
||||
ThothII userà un Metadata Catalog PostgreSQL interno per conservare, per ogni workspace, la
|
||||
struttura fisica acquisita interrogando il relativo database e i metadati semantici generati con
|
||||
l'AI. Ogni workspace avrà un solo Workspace Database; identità e lista dei workspace resteranno
|
||||
autorevoli in `thoth-workspaces.yaml`, mentre il catalogo ne conserverà soltanto il riferimento
|
||||
stabile. `schema/annotations.yaml` verrà sostituito come input del core in uno step successivo;
|
||||
l'interfaccia e il lifecycle amministrativi resteranno separati dal workflow NL→SQL.
|
||||
l'AI. Ogni Workspace Database conserverà un `workspace_id` obbligatorio e univoco: questo realizza
|
||||
l'associazione uno-a-uno senza introdurre nel catalogo una tabella Workspace o una foreign key SQL.
|
||||
Identità e lista dei workspace resteranno autorevoli in `thoth-workspaces.yaml`; il servizio
|
||||
validerà il riferimento contro quel catalogo. `schema/annotations.yaml` verrà sostituito come input
|
||||
del core in uno step successivo; l'interfaccia e il lifecycle amministrativi resteranno separati dal
|
||||
workflow NL→SQL.
|
||||
|
||||
## Considered Options
|
||||
|
||||
@@ -18,4 +20,7 @@ l'interfaccia e il lifecycle amministrativi resteranno separati dal workflow NL
|
||||
|
||||
PSD importerà le annotations esistenti; gli altri workspace genereranno i metadati da zero. Il
|
||||
cutover futuro dovrà sostituire consapevolmente i consumatori delle annotations e verificarne
|
||||
l'equivalenza semantica. Le sessioni di test esistenti non sono un vincolo di migrazione.
|
||||
l'equivalenza semantica. PostgreSQL può garantire che uno stesso `workspace_id` non sia assegnato a
|
||||
due database, ma l'esistenza del workspace e la gestione di rename o rimozioni restano responsabilità
|
||||
del confine applicativo con il catalogo YAML. Le sessioni di test esistenti non sono un vincolo di
|
||||
migrazione.
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
# Riferimenti al secret store per i Workspace Database
|
||||
|
||||
Il Metadata Catalog non conserverà credenziali o chiavi dei Workspace Database. Riuserà il secret
|
||||
store cifrato già posseduto da ThothII e conserverà soltanto riferimenti ai requisiti segreti del
|
||||
workspace, evitando un secondo vault e impedendo che API, esportazioni o log espongano i valori.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- Copiare i campi testuali del modello ThothAI avrebbe semplificato il CRUD, ma avrebbe conservato
|
||||
password e passphrase in chiaro.
|
||||
- Introdurre subito un secondo vault avrebbe separato il catalogo dal runtime workspace, ma avrebbe
|
||||
duplicato cifratura, rotazione, autorizzazioni e procedure operative senza un'esigenza distinta.
|
||||
|
||||
## Consequences
|
||||
|
||||
Il catalogo e il runtime condividono l'identità dei requisiti segreti, mentre permessi e API del
|
||||
catalogo restano separati. Sostituzione e cancellazione di un Workspace Database dovranno definire
|
||||
esplicitamente il lifecycle dei relativi riferimenti senza includere i valori nelle transazioni del
|
||||
catalogo PostgreSQL.
|
||||
@@ -0,0 +1,26 @@
|
||||
# Binding di database specifiche dell'installazione
|
||||
|
||||
Il Metadata Catalog separa il Workspace Database logico dalla Database Binding che lo rende
|
||||
raggiungibile in una specifica installazione. Esiste un solo Workspace Database per `workspace_id`
|
||||
e una sola binding attiva nel catalogo di ciascuna installazione; per esempio PSD usa REST in locale
|
||||
e PostgreSQL diretto sul server senza diventare due database distinti.
|
||||
|
||||
Nel modello finale il catalogo è autorevole per engine, nome fisico, schema, capability e binding,
|
||||
mentre `thoth-workspaces.yaml` conserva l'identità del workspace. Gli attuali campi DWH dei
|
||||
descriptor sono una sorgente di bootstrap da importare e confrontare durante un cutover esplicito,
|
||||
non una seconda fonte di verità permanente.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- Conservare un record database per ogni trasporto avrebbe duplicato identità, struttura e
|
||||
metadati dello stesso DWH fra locale e server.
|
||||
- Conservare permanentemente i dati DWH sia nello YAML sia nel catalogo avrebbe introdotto
|
||||
conflitti non risolvibili deterministicamente.
|
||||
- Rendere globali le binding avrebbe mescolato endpoint e credenziali che appartengono a
|
||||
installazioni con topologie e confini di sicurezza differenti.
|
||||
|
||||
## Consequences
|
||||
|
||||
La lista amministrativa unisce workspace YAML, database configurati e record orphaned. Il cutover
|
||||
deve importare e confrontare la configurazione esistente prima di rimuoverla dai descriptor; il
|
||||
runtime non deve osservare simultaneamente due autorità discordanti.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Metadata Catalog nello stesso backend con Kysely
|
||||
|
||||
Il Metadata Catalog vive nello stesso processo Fastify come modulo isolato, invece di introdurre un
|
||||
microservizio. Usa il driver `pg` già presente attraverso Kysely per query, transazioni e migrazioni
|
||||
tipizzate; route, repository, service, readiness e diagnostica restano separati dal workflow e
|
||||
l'indisponibilità del catalogo non rende indisponibili sessioni o SSE.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- Un microservizio avrebbe conservato letteralmente il backend bridge senza database, ma avrebbe
|
||||
aggiunto deployment, autenticazione e failure mode per un solo contesto amministrativo.
|
||||
- Usare soltanto `pg` avrebbe evitato una dipendenza, ma avrebbe richiesto infrastruttura locale per
|
||||
transazioni, tipi delle righe, ordinamento e locking delle migrazioni.
|
||||
- Drizzle o Prisma avrebbero aggiunto schema DSL, generatori e toolchain non necessari a un servizio
|
||||
che vuole mantenere SQL e constraint PostgreSQL espliciti.
|
||||
|
||||
## Consequences
|
||||
|
||||
Il backend possiede una connection pool del catalogo e la chiude con il lifecycle Fastify. Le
|
||||
migrazioni timestampate sono compilate insieme al backend ma vengono eseguite soltanto da
|
||||
`catalog:migrate`, con credenziali migrator separate dal ruolo DML usato a runtime. Il modulo resta
|
||||
dietro un'interfaccia repository per mantenere unit test e route test indipendenti da PostgreSQL.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Hard-delete catalog tables during synchronization
|
||||
|
||||
An explicit Table Synchronization makes the Catalog Table membership exactly match a successful
|
||||
observation of the Workspace Database: new tables are created, source metadata is refreshed, and
|
||||
absent tables plus their future column and relationship children are permanently deleted. Physical
|
||||
membership cannot be edited manually.
|
||||
|
||||
The external scan runs without holding a catalog transaction. Its diff is applied atomically only
|
||||
while the Workspace Database version still matches the scanned binding. Failed scans change
|
||||
nothing, and a non-empty removal set must exactly match the names confirmed by the operator; a
|
||||
changed second scan therefore requires a new confirmation.
|
||||
|
||||
## Considered Options
|
||||
|
||||
- Soft deletion would preserve descriptions across accidental removals, but would add hidden state,
|
||||
restore rules, and ambiguity about whether the catalog still represents the physical schema.
|
||||
- Rename detection based on similarity would preserve metadata in some cases, but could silently
|
||||
attach curated semantics to the wrong physical table.
|
||||
- Append-only introspection, as in the legacy importer, would leave stale tables in the catalog and
|
||||
make downstream schema linking unreliable.
|
||||
|
||||
## Consequences
|
||||
|
||||
A physical rename is delete plus create and loses curated metadata. The UI previews permanent
|
||||
deletions, and future Catalog Column and Relationship records must cascade with their table. The
|
||||
catalog remains an exact projection of the last accepted successful scan without tombstones or
|
||||
restore lifecycle.
|
||||
@@ -0,0 +1,8 @@
|
||||
# Separate physical and logical relationships
|
||||
|
||||
ThothII persists each database-declared foreign-key constraint as an immutable Catalog
|
||||
Relationship with ordered column pairs, so composite keys retain their identity and the database
|
||||
remains the authority for physical structure. Curated or AI-inferred Logical Relationships will
|
||||
use a separate future model and lifecycle rather than being mixed with physical constraints or
|
||||
denormalized into textual column fields; this keeps synchronization authoritative without
|
||||
preventing later semantic enrichment.
|
||||
@@ -0,0 +1,9 @@
|
||||
# Use durable runs for authoritative schema synchronization
|
||||
|
||||
All table, column, relationship, and full-schema synchronizations run as durable background
|
||||
Catalog Sync Runs rather than separate synchronous and asynchronous implementations. Each scope
|
||||
is authoritative within its boundary, while Synchronize All observes one complete schema snapshot;
|
||||
destructive diffs require confirmation and source revalidation before an atomic, fail-closed
|
||||
catalog transaction. A persistent per-database lock, progress events, interruption handling, and
|
||||
binding-version freshness make long operations observable and prevent two processes or stale
|
||||
configuration from producing a partially trusted catalog.
|
||||
@@ -0,0 +1,7 @@
|
||||
# Domain Docs
|
||||
|
||||
This is a single-context repository.
|
||||
|
||||
Before exploring, read `CONTEXT.md` at the repository root and the relevant decisions in
|
||||
`docs/adr/`. Use the project's terminology from `CONTEXT.md` in issue titles, proposals, and
|
||||
tests. Surface conflicts with an ADR instead of silently overriding it.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Issue tracker: Gitea
|
||||
|
||||
Issues and specs for this repository live in the self-hosted Gitea repository:
|
||||
`https://git.tylconsulting.it/mptyl/ThothII`.
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Create an issue**: use the repository's Gitea web UI, or the Gitea REST API when an
|
||||
authenticated token with issue scope is available.
|
||||
- **Read and list issues**: use the Gitea web UI or authenticated API; include labels and comments.
|
||||
- **Apply or remove labels**: use the issue's label controls or the Gitea API.
|
||||
- **Comment and close**: use the issue page or the Gitea API.
|
||||
- Do not use `gh issue ...` for this repository: the `github` remote is a mirror, not the canonical
|
||||
issue tracker.
|
||||
|
||||
## Repository identity
|
||||
|
||||
- Canonical Git remote: `origin` → `https://git.tylconsulting.it/mptyl/ThothII.git`
|
||||
- GitHub mirror: `github` → `https://github.com/mptyl/ThothII.git`
|
||||
- Canonical issue URL: `https://git.tylconsulting.it/mptyl/ThothII/issues`
|
||||
|
||||
## When a skill says “publish to the issue tracker”
|
||||
|
||||
Create an issue in the canonical Gitea repository.
|
||||
|
||||
## When a skill says “fetch the relevant ticket”
|
||||
|
||||
Read the referenced issue in the canonical Gitea repository.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Triage Labels
|
||||
|
||||
The skills speak in terms of five canonical triage roles. This file maps those roles to the labels
|
||||
used in the canonical Gitea issue tracker.
|
||||
|
||||
| Label in mattpocock/skills | Label in Gitea | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
|
||||
| `needs-info` | `needs-info` | Waiting on reporter for more information |
|
||||
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for AFK agent work |
|
||||
| `ready-for-human` | `ready-for-human` | Requires human implementation |
|
||||
| `wontfix` | `wontfix` | Will not be actioned |
|
||||
|
||||
Issue type labels:
|
||||
|
||||
| Label in Gitea | Meaning |
|
||||
| --- | --- |
|
||||
| `bug` | Something is not working |
|
||||
| `enhancement` | New feature or request |
|
||||
@@ -4,7 +4,10 @@ This page complements the [architecture overview](overview.md) with the module s
|
||||
|
||||
## Modules and dependencies
|
||||
|
||||
The frontend communicates with the backend through REST and SSE. The backend does not own session persistence: it starts Pi, invokes the `tht` CLI, and forwards events. The harness contains the workflow, the Python CLI, and adapters for the DWH and vector store.
|
||||
The frontend communicates with the backend through REST and SSE. The backend does not own session
|
||||
persistence: it starts Pi, invokes the `tht` CLI, and forwards events. It does own the separate
|
||||
installation-local database catalog. The harness contains the workflow, the Python CLI, and
|
||||
adapters for the DWH and vector store.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
@@ -18,6 +21,8 @@ flowchart LR
|
||||
THT --> DWH["DWH\nread-only"]
|
||||
THT --> VDB["Qdrant / vector store"]
|
||||
BE --> CFG["settings.json\nworkspace registry"]
|
||||
BE --> CAT["catalog-db\nPostgreSQL + Kysely"]
|
||||
BE -->|catalog Test + Table Sync| DWH
|
||||
FE -.->|renders widgets| EXT
|
||||
```
|
||||
|
||||
@@ -26,7 +31,7 @@ Dipendenze principali:
|
||||
| Module | Depends on | Responsibility |
|
||||
| --- | --- | --- |
|
||||
| `frontend/` | Backend REST and SSE APIs | UI, gate widgets, and in-memory transcript |
|
||||
| `backend/src/` | Pi, `tht`, configuration, and workspace registry | Transport, session lifecycle, and APIs |
|
||||
| `backend/src/` | Pi, `tht`, configuration, workspace registry, catalog PostgreSQL, and read-only DWH connectors | Transport, session lifecycle, catalog CRUD, connection tests, table introspection, and APIs |
|
||||
| `harness/.pi/` | Pi and `tht phase` | Workflow orchestration and human-in-the-loop gates |
|
||||
| `harness/tht/` | Filesystem, DWH, and vector store | Persistence, CLI, Evidence, schema, and preprocessing |
|
||||
| workspace repository | `source/`, `curated/`, manifest, and artifacts | Versioned Evidence source and session output |
|
||||
|
||||
@@ -11,6 +11,7 @@ For sessions, roles, groups, diagnostics, and recovery, see the [authentication
|
||||
flowchart LR
|
||||
USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
|
||||
FE --> BE["Backend\nFastify"]
|
||||
BE --> CATALOG["Metadata catalog\nPostgreSQL"]
|
||||
BE --> PI["Pi\nRPC per sessione"]
|
||||
PI --> THT["tht and harness\nworkflow and persistence"]
|
||||
THT --> DWH["DWH\nread only"]
|
||||
@@ -27,8 +28,8 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
|
||||
|
||||
| Layer | Stack | Ruolo |
|
||||
|---|---|---|
|
||||
| **harness/** | Python (`tht` CLI) + Pi gate extension (JS) | Owns the workflow and **all** persistence |
|
||||
| **backend/** | Fastify + TypeScript | Thin bridge with no database of its own |
|
||||
| **harness/** | Python (`tht` CLI) + Pi gate extension (JS) | Owns the workflow and all session persistence |
|
||||
| **backend/** | Fastify + TypeScript + Kysely | Session bridge plus the isolated administrative metadata catalog |
|
||||
| **frontend/** | React 18 + Vite | UI that renders gate widgets and rebuilds the live transcript from the SSE stream |
|
||||
|
||||
## The harness owns the workflow
|
||||
@@ -39,14 +40,22 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
|
||||
|
||||
A session is a directory under `sessions/` (the workspace defines the path): `session_manifest.yaml`, phase artifacts (`question.md`, `schema_linking.json`, `sql_final.sql`, and others), and `review_decisions.jsonl`. The contract says: *"persisted state is the truth; what is not recorded did not happen"*. There is no verbatim transcript store. A resumed Pi process rebuilds context from `tht session show <id>` and the artifacts on disk.
|
||||
|
||||
## The backend is a thin bridge with no database
|
||||
## The backend bridges sessions and owns the metadata catalog
|
||||
|
||||
- `ThtRunner` runs `tht` subcommands in a shell.
|
||||
- `PiProcessManager` runs one Pi child process per session and bridges its RPC stream.
|
||||
- `SessionBridge` maps Pi RPC events to client events (`ui_request` / `text_delta` / `info`).
|
||||
- `SseHub` distributes these events to the browser over SSE.
|
||||
- `CatalogService` joins authoritative YAML workspace identities with installation-local database
|
||||
configurations stored in PostgreSQL through Kysely.
|
||||
- `CatalogTableService` reconciles persisted Catalog Tables with a successful external schema scan;
|
||||
`ConcreteCatalogTableIntrospector` isolates direct PostgreSQL, typed REST, and SSH-tunnel access.
|
||||
|
||||
Application settings live in a JSON file (`backend/data/settings.json`), not in a database.
|
||||
Application settings remain in `backend/data/settings.json`; session state remains in harness phase
|
||||
documents. PostgreSQL stores only the administrative database catalog, bindings, observed tables,
|
||||
and curated descriptions. Connector secrets remain write-only in the encrypted workspace secret
|
||||
store. Catalog SSH support is limited to connection tests and table synchronization; it does not
|
||||
change the session runtime binding contract.
|
||||
|
||||
## Human-in-the-loop gate contract
|
||||
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# Catalog schema snapshot RPC
|
||||
|
||||
Il percorso preferito per un binding `rest_api` espone al catalogo un'unica fotografia tipizzata
|
||||
dello schema:
|
||||
|
||||
```http
|
||||
POST /rpc/schema_snapshot
|
||||
Content-Type: application/json
|
||||
|
||||
{"schema_name":"datawarehouse"}
|
||||
```
|
||||
|
||||
La risposta è un oggetto JSON con `schemaVersion: 1`, capability esplicite e tre collezioni. Una
|
||||
capability non disponibile deve essere dichiarata `unavailable`: non deve essere simulata con una
|
||||
lista vuota.
|
||||
|
||||
```json
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"capabilities": {
|
||||
"tables": "available",
|
||||
"columns": "available",
|
||||
"relationships": "available"
|
||||
},
|
||||
"tables": [
|
||||
{ "name": "patients", "sourceComment": "Clinical patients" }
|
||||
],
|
||||
"columns": [
|
||||
{
|
||||
"tableName": "patients",
|
||||
"name": "id",
|
||||
"ordinalPosition": 1,
|
||||
"dataType": "bigint",
|
||||
"isNullable": false,
|
||||
"defaultExpression": null,
|
||||
"primaryKeyPosition": 1,
|
||||
"sourceComment": "Patient identifier"
|
||||
}
|
||||
],
|
||||
"relationships": [
|
||||
{
|
||||
"constraintName": "visits_patient_id_fkey",
|
||||
"sourceTableName": "visits",
|
||||
"targetTableName": "patients",
|
||||
"updateRule": "NO ACTION",
|
||||
"deleteRule": "CASCADE",
|
||||
"deferrable": false,
|
||||
"initiallyDeferred": false,
|
||||
"columns": [
|
||||
{ "position": 1, "sourceColumnName": "patient_id", "targetColumnName": "id" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Fallback compatibile tramite `run_query`
|
||||
|
||||
Se e soltanto se il server non espone `POST /rpc/schema_snapshot`, il catalogo può ottenere la
|
||||
stessa fotografia mediante una singola istruzione read-only inviata all'RPC già esistente:
|
||||
|
||||
```http
|
||||
POST /rpc/run_query
|
||||
Content-Type: application/json
|
||||
|
||||
{"query_text":"WITH ... SELECT ..."}
|
||||
```
|
||||
|
||||
La query è costruita dal catalogo, interroga soltanto il catalogo PostgreSQL dello schema
|
||||
configurato e aggrega tabelle, colonne, primary key e foreign key nella stessa istruzione. Non sono
|
||||
ammessi più round trip, query per tabella o assemblaggi client-side di osservazioni effettuate in
|
||||
momenti diversi. Il nome schema deve essere validato come identificatore e quotato come valore SQL,
|
||||
non interpolato come SQL libero.
|
||||
|
||||
`run_query` restituisce un array JSON di righe. Per questo fallback l'array deve contenere
|
||||
esattamente una riga e quella riga deve essere esattamente l'oggetto snapshot v1 sopra descritto,
|
||||
con `schemaVersion`, `capabilities`, `tables`, `columns` e `relationships`; campi mancanti,
|
||||
aggiuntivi o di tipo diverso rendono invalida l'intera fotografia. La risposta non è un contratto
|
||||
alternativo o più permissivo: cambia soltanto il trasporto della stessa snapshot stretta.
|
||||
|
||||
`position` e `primaryKeyPosition` sono uno-based. Le coppie ordinate permettono foreign key
|
||||
composte. Il catalogo rifiuta l'intera fotografia se il JSON non rispetta il contratto o se la
|
||||
capability richiesta dal tipo di sincronizzazione è `unavailable`; in entrambi i casi non applica
|
||||
alcuna modifica. Il fallback viene tentato soltanto quando l'RPC preferito risulta assente, non per
|
||||
nascondere una snapshot malformata o un errore operativo del server. Se anche `run_query` non è
|
||||
disponibile, la query viene rifiutata, la risposta non contiene una singola snapshot v1 valida o una
|
||||
capability richiesta è `unavailable`, il run fallisce senza aggiornamenti parziali e senza esporre
|
||||
il corpo remoto.
|
||||
@@ -92,8 +92,10 @@ evidence:
|
||||
|
||||
Changes from older workspaces:
|
||||
|
||||
- the database is reached only through **REST** or **direct Postgres** (`rest_api` /
|
||||
`postgres_direct`); the SSH tunnel remains disabled;
|
||||
- NL→SQL sessions reach the database only through **REST** or **direct Postgres** (`rest_api` /
|
||||
`postgres_direct`); `ssh_tunnel` remains disabled for session runtime. The separate Database
|
||||
management surface supports SSH for **Test connection** and **Sync tables**, with a private key,
|
||||
mandatory `known_hosts`, and an optional key passphrase;
|
||||
- the semantic index is **internal** (Qdrant plus `qwen3-embedding:0.6b`, 1024 dimensions, cosine);
|
||||
- **filesystem** Evidence lives in the repository (`<id>/evidence`) and is materialized from the
|
||||
pinned Git commit (P6). HTTP Evidence is also supported.
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
# Metadata Catalog di ThothII: ricognizione ThothAI e percorso incrementale
|
||||
|
||||
Data: 2026-08-26
|
||||
Stato: ricognizione completata; step 1 implementato; scelte tecnologiche degli step successivi
|
||||
deliberatamente rinviate.
|
||||
Data: 2026-08-26; aggiornato 2026-08-27
|
||||
Stato: ricognizione e progettazione completate; navigazione, CRUD Workspace Database, Catalog
|
||||
Table, Catalog Column, Catalog Relationship e sincronizzazione durevole dello schema implementati
|
||||
il 2026-08-27. Generazione AI e integrazione con il workflow core restano negli step successivi.
|
||||
|
||||
## Obiettivo
|
||||
|
||||
@@ -10,9 +11,11 @@ ThothII deve introdurre un contesto amministrativo separato, il **Metadata Catal
|
||||
il database associato a ciascun workspace, la sua struttura fisica introspezionata e i metadati
|
||||
semantici oggi rappresentati da `schema/annotations.yaml`.
|
||||
|
||||
Il programma procede per step indipendenti. Il primo step aggiunge soltanto l'accesso dalla sidebar
|
||||
destra a una superficie centrale vuota. Non introduce PostgreSQL, API CRUD, introspezione o
|
||||
integrazioni con il workflow core.
|
||||
Il programma procede per step indipendenti. Il primo step ha aggiunto l'accesso dalla sidebar; il
|
||||
secondo ha sostituito la superficie vuota con il CRUD di configurazione, il PostgreSQL interno e i
|
||||
test di connessione; gli step successivi hanno aggiunto navigazione gerarchica, colonne, relazioni
|
||||
fisiche e sincronizzazione durevole dell'intero schema. Non introduce ancora generazione AI o
|
||||
integrazione con il workflow core.
|
||||
|
||||
Questa analisi usa come riferimento il working tree legacy osservato in
|
||||
`Thoth/ThothAI`. Non è stato verificato che quel contenuto corrisponda a una release o a un tag
|
||||
@@ -20,8 +23,10 @@ canonico; i percorsi e i comportamenti descrivono il sorgente disponibile il 202
|
||||
|
||||
## Decisioni già confermate
|
||||
|
||||
1. Ogni workspace è associato a un solo Workspace Database e ogni Workspace Database appartiene a
|
||||
un solo workspace.
|
||||
1. Ogni Workspace Database appartiene a un solo workspace tramite un `workspace_id` obbligatorio e
|
||||
univoco; un workspace può avere al massimo un Workspace Database. Poiché i workspace non sono
|
||||
righe del catalogo PostgreSQL, l'associazione è un riferimento logico validato contro
|
||||
`thoth-workspaces.yaml`, non una foreign key SQL.
|
||||
2. Il CRUD non crea né rinomina workspace. Identità e lista ordinata dei workspace restano
|
||||
autorevoli in `thoth-workspaces.yaml`; il catalogo conserva il loro identificatore stabile.
|
||||
3. La struttura fisica viene acquisita interrogando il database esterno tramite i dati di
|
||||
@@ -38,6 +43,146 @@ canonico; i percorsi e i comportamenti descrivono il sorgente disponibile il 202
|
||||
introduce un router.
|
||||
9. La pagina iniziale è vuota, segue il tema, nasconde l'intera colonna core e non interrompe una
|
||||
sessione live. Le azioni di apertura, resume o creazione sessione riportano al core.
|
||||
10. La compatibilità con il modello ThothAI è semantica, non una copia letterale: configurazione e
|
||||
contenuti semantici sono campi relazionali mutabili, mentre identità e appartenenza della
|
||||
struttura fisica derivano dall'introspezione; i segreti restano nel secret store e lo stato dei
|
||||
job non viene mescolato ai dati amministrativi.
|
||||
11. Il CRUD amministra il Metadata Catalog e non esegue DDL sul database esterno, che resta
|
||||
read-only.
|
||||
12. La prima versione supporta PostgreSQL; il confine di introspezione dovrà permettere di
|
||||
aggiungere altri dialetti senza cambiare il modello del catalogo.
|
||||
13. I segreti dei Workspace Database riusano il secret store cifrato di ThothII. Il catalogo
|
||||
conserva riferimenti ai segreti e nessuna API, esportazione o log ne restituisce i valori.
|
||||
14. La UI usa AG Grid Community per la lista master e un pannello React separato per il dettaglio;
|
||||
non dipende dalle funzionalità master-detail di AG Grid Enterprise.
|
||||
15. Un Workspace Database il cui `workspace_id` scompare dal catalogo YAML non viene cancellato
|
||||
automaticamente: diventa orphaned e può soltanto essere recuperato, riassegnato o eliminato
|
||||
esplicitamente da un amministratore.
|
||||
16. La prima vertical slice gestisce configurazione del Workspace Database, riferimenti ai segreti,
|
||||
test di connessione e stato. La seconda gestisce le Catalog Table: la collezione e i nomi sono
|
||||
controllati dall'introspezione, mentre la descrizione curata è modificabile. Le slice successive
|
||||
hanno aggiunto Catalog Column, Catalog Relationship e sincronizzazione durevole dello schema.
|
||||
17. Il modello non conserva il `name` libero di ThothAI: nome e ID visualizzati appartengono al
|
||||
workspace YAML, mentre `database_name` identifica il database PostgreSQL esterno.
|
||||
18. Database management supporta i tre trasporti già riconosciuti da ThothII: `postgres_direct`,
|
||||
`rest_api` e `ssh_tunnel`. PSD rimane un solo Workspace Database: usa la connessione diretta sul
|
||||
server e l'endpoint REST in locale tramite una Database Binding specifica dell'installazione.
|
||||
Questo supporto non abilita automaticamente `ssh_tunnel` nel runtime NL→SQL.
|
||||
19. Una configurazione può essere salvata prima di una connessione riuscita. Il test separato
|
||||
produce uno stato `untested`, `reachable` o `failed`; attivazione e introspezione richiedono uno
|
||||
stato raggiungibile.
|
||||
20. Il CRUD e il test di connessione richiedono `database.manage`; inserimento e sostituzione dei
|
||||
segreti continuano a richiedere `workspace.secrets.manage`.
|
||||
21. Il Workspace Database e il modo di raggiungerlo sono entità distinte. Ogni catalogo di
|
||||
installazione conserva una sola Database Binding attiva per workspace: PSD usa `rest_api` in
|
||||
locale e `postgres_direct` sul server senza duplicare il Workspace Database.
|
||||
22. Nel modello finale il Metadata Catalog è autorevole per engine, `database_name`, schema,
|
||||
capacità e binding. Lo YAML resta autorevole per identità e contenuti del workspace; i campi
|
||||
DWH correnti saranno importati, confrontati e rimossi soltanto durante un cutover esplicito.
|
||||
23. La lista master è l'unione fra workspace YAML e record del catalogo: mostra workspace
|
||||
`unconfigured`, database configurati e record `orphaned`.
|
||||
24. Ogni introspezione registra le capability disponibili. Una capability `unavailable` non viene
|
||||
rappresentata come una collezione osservata ma vuota; REST può completare con successo anche
|
||||
quando indici o enum non sono supportati.
|
||||
25. Il Metadata Catalog non introduce snapshot, draft o pubblicazioni. Configurazione e contenuti
|
||||
semantici, inclusi quelli futuri generati dall'AI, sono normali campi modificabili; la struttura
|
||||
osservata cambia soltanto con una sincronizzazione esplicita.
|
||||
26. Il normale Delete elimina realmente il Workspace Database, la Database Binding e i relativi
|
||||
record catalogo e segreti. Non modifica il DWH esterno né il repository YAML; il workspace torna
|
||||
visibile nella lista master come `unconfigured`.
|
||||
27. La prima versione gestisce un solo schema obbligatorio per Workspace Database, identificato
|
||||
dalla coppia `database_name + schema`; per PSD la coppia è `postgres + datawarehouse`.
|
||||
28. I record mantengono soltanto `created_at`, `updated_at` e un contatore `version` per optimistic
|
||||
concurrency. Non esistono storico delle revisioni, rollback o audit applicativo delle modifiche.
|
||||
29. `workspace_databases` conserva soltanto UUID, `workspace_id` unique, engine, `database_name`,
|
||||
schema, timestamp e version. Il nome visualizzato appartiene al workspace YAML.
|
||||
30. Ogni Workspace Database ha al massimo una riga `database_bindings`. Una singola tabella usa
|
||||
check constraint dipendenti da `transport` per i campi direct, REST e SSH; non esiste un flag
|
||||
`active`, perché ciascuna installazione conserva una sola binding.
|
||||
31. `rest_api` configura il Thoth REST Connector tipizzato: base URL, autenticazione e TLS sono dati
|
||||
della binding, mentre path RPC e shape delle risposte appartengono al contratto applicativo e non
|
||||
sono liberamente configurabili.
|
||||
32. Il test connessione usa soltanto una configurazione già salvata ed è associato alla sua
|
||||
`version`. Ogni modifica della binding o dei segreti invalida il risultato precedente e riporta
|
||||
lo stato a `untested`.
|
||||
33. Password, API key e chiavi sono write-only: l'API espone soltanto `configured`, un campo vuoto
|
||||
conserva il valore esistente e la sostituzione è un'azione esplicita. Delete rimuove anche i
|
||||
segreti associati.
|
||||
34. La pagina usa AG Grid come master e un form React come detail, con sezioni Database, Connection
|
||||
e TLS/SSH condizionali. La toolbar offre `Add database`; le righe `unconfigured` offrono
|
||||
`Configure`. Entrambe selezionano esclusivamente workspace YAML senza un database e creano il
|
||||
record soltanto al Save; `workspace_id` diventa immutabile dopo la creazione.
|
||||
35. La grid mostra workspace, database, schema, transport, endpoint, stato connessione e ultimo
|
||||
aggiornamento. Su schermi piccoli il dettaglio occupa il pannello completo. Il cambio riga con
|
||||
modifiche non salvate e Delete richiedono conferma, senza conferma testuale tipizzata.
|
||||
36. La Database Binding conserva `connection_status`, `tested_version`, `last_tested_at`, un codice
|
||||
errore e un messaggio breve sanificato. Non conserva stack trace, DSN, credenziali o output grezzo
|
||||
del driver.
|
||||
37. Le API vivono sotto `/api/catalog`: list/create di `/databases`, get/patch/delete di
|
||||
`/databases/:id`, sostituzione dei segreti sotto `/databases/:id/secrets`, test connessione sotto
|
||||
`/databases/:id/test` e list/patch/sync delle tabelle sotto `/databases/:id/tables`.
|
||||
38. `GET /api/catalog/databases` restituisce l'intera master list unificata; AG Grid Community applica
|
||||
client-side ricerca, filtri e ordinamento. La prima versione non introduce paginazione server o
|
||||
funzionalità AG Grid Enterprise.
|
||||
39. Il Metadata Catalog vive nello stesso processo Fastify come modulo isolato con repository,
|
||||
service, route, diagnostica e readiness proprie. L'indisponibilità del catalogo non modifica
|
||||
sessioni, SSE o health del core e non giustifica ancora un microservizio separato.
|
||||
40. Il backend mantiene `pg@8.22.0` e aggiunge `kysely@0.29.5` per query e transazioni tipizzate. Le
|
||||
migrazioni Kysely sono timestampate, compilate con il backend ed eseguite da un comando
|
||||
`catalog:migrate` separato; l'applicazione non migra automaticamente il database all'avvio.
|
||||
41. Lo stack aggiunge un servizio interno `catalog-db` con volume persistente, ruolo runtime DML,
|
||||
ruolo migrator DDL e job one-shot `catalog-migrate`. Un catalogo indisponibile produce 503 sulle
|
||||
sole route catalogo.
|
||||
42. La prima vertical slice è amministrativa: scrive il catalogo ma non cambia ancora il runtime di
|
||||
sessioni e workflow, che continua a usare YAML e binding correnti fino al cutover esplicito.
|
||||
43. `Configure` precompila senza salvare engine, database e schema dal descriptor e i dati non
|
||||
sensibili dalla binding effettiva. L'amministratore verifica, inserisce i segreti e salva; non
|
||||
esiste importazione silenziosa.
|
||||
44. Unit e route test usano un repository fake; una suite PostgreSQL Testcontainers separata verifica
|
||||
migrazioni, constraint, transazioni, optimistic concurrency e cascade. SQLite ed emulatori non
|
||||
sono sostituti ammessi per questi test.
|
||||
45. La navigazione delle entità catalogo è gerarchica e senza scorciatoie globali: `Databases →
|
||||
Database → Overview | Tables → Table`. Non esistono una voce globale Tables, un filtro globale
|
||||
Database o una preselezione implicita; Columns continuerà sotto Table e Relationships sotto
|
||||
Database.
|
||||
46. Una Catalog Table conserva nome fisico, `source_comment`, descrizione curata nullable,
|
||||
`generated_description` nullable per lo step AI futuro, version e timestamp. La UI mostra come
|
||||
tre campi indipendenti senza fallback visivo: source comment read-only, generated description
|
||||
modificabile e description modificabile. I valori null restano celle e controlli vuoti.
|
||||
47. Le Catalog Table non possono essere aggiunte, rinominate o cancellate manualmente. `Sync tables`
|
||||
legge dal database esterno le tabelle PostgreSQL ordinarie e partizionate dello schema scelto;
|
||||
viste e materialized view sono escluse.
|
||||
48. La sincronizzazione è esplicita. La scansione avviene fuori dalla transazione del catalogo; il
|
||||
diff viene applicato atomicamente soltanto se la version del Workspace Database è ancora quella
|
||||
sottoposta a scansione. Una scansione fallita non modifica il catalogo.
|
||||
49. Tabelle nuove vengono create, i commenti sorgente vengono aggiornati e quelle non più osservate
|
||||
vengono eliminate definitivamente. La rimozione di tabelle, colonne o relazioni richiede la
|
||||
conferma dell'esatto piano distruttivo; se il secondo scan produce una fotografia differente,
|
||||
l'applicazione richiede una nuova conferma.
|
||||
50. Un rename fisico è intenzionalmente delete più create e perde i metadati curati. Le colonne e
|
||||
relazioni dipendenti vengono eliminate in cascade insieme alla Catalog Table.
|
||||
51. L'introspezione vive nel modulo catalogo Fastify dietro un adapter. PostgreSQL diretto e tunnel
|
||||
SSH usano il catalogo `pg_catalog`; REST preferisce il contratto tipizzato
|
||||
`POST /rpc/schema_snapshot` e, quando quell'RPC non è esposto, usa come fallback compatibile una
|
||||
singola query read-only tramite `POST /rpc/run_query`. Entrambi i percorsi devono produrre la
|
||||
stessa fotografia v1 stretta descritta in `docs/contracts/catalog-schema-snapshot.md`.
|
||||
52. Test connessione e sincronizzazione sono serializzati per Workspace Database, hanno timeout e
|
||||
richiedono che la binding nella version corrente abbia un test `reachable` prima di qualsiasi
|
||||
Catalog Sync Run. La scansione asincrona ha un timeout separato, di default dieci minuti.
|
||||
53. Il tunnel SSH usa OpenSSH in modalità stdio `-W`, chiave privata e passphrase opzionale dal
|
||||
secret store, `known_hosts` obbligatorio, `StrictHostKeyChecking=yes`, agent e configurazione
|
||||
globale disabilitati. Non è ammesso TOFU. TLS PostgreSQL con CA e server name resta verificato
|
||||
anche attraverso il tunnel.
|
||||
54. In questo slice `ssh_tunnel` è una binding supportata da Database management per Test connection
|
||||
e Schema Sync. Il renderer e il runtime delle sessioni NL→SQL restano fuori scope e continuano a
|
||||
rifiutarla finché non verrà deciso il relativo cutover.
|
||||
55. I menu di azione a livello Workspace Database espongono separatamente `Synchronize tables`,
|
||||
`Synchronize all columns`, `Synchronize relationships` e `Synchronize all`. Su una selezione di
|
||||
database lo scope scelto viene avviato per ogni database idoneo; non viene sostituito
|
||||
implicitamente con una sincronizzazione completa.
|
||||
56. Per lo scope Columns, `tableIds` vuoto significa tutte le Catalog Table correnti del Workspace
|
||||
Database; `tableIds` valorizzato limita invece la riconciliazione alle tabelle indicate. La grid
|
||||
Tables espone `Synchronize columns` sulle tabelle selezionate.
|
||||
|
||||
## Correzione del modello mentale corrente
|
||||
|
||||
@@ -205,8 +350,8 @@ template Django.
|
||||
|
||||
Il comportamento è principalmente additivo: usa `get_or_create` o controlli `exists`, aggiorna
|
||||
alcuni commenti, ma non riconcilia in modo completo rename, rimozioni o drift. Non va copiato così
|
||||
com'è. Il futuro processo ThothII dovrà almeno distinguere scansione, differenze osservate e
|
||||
applicazione della nuova snapshot.
|
||||
com'è. Il processo ThothII implementato distingue scansione, differenze osservate e applicazione
|
||||
della nuova snapshot.
|
||||
|
||||
### Generazione AI legacy
|
||||
|
||||
@@ -318,7 +463,8 @@ rendering, retrieval, LSH o SQL generation.
|
||||
|
||||
### Vincoli minimi da progettare
|
||||
|
||||
- `workspace_id` unico sul Workspace Database;
|
||||
- `workspace_id` obbligatorio e unico sul Workspace Database, con esistenza validata contro il
|
||||
catalogo YAML dal servizio applicativo;
|
||||
- nome tabella unico nel database e schema appropriato;
|
||||
- nome colonna unico nella tabella;
|
||||
- relationship unica secondo il modello, anche per chiavi composite;
|
||||
@@ -348,8 +494,7 @@ L'export legacy della struttura include username e password in chiaro. Il modell
|
||||
password, passphrase SSH e altri segreti in `CharField`; non è stata trovata cifratura applicativa,
|
||||
nonostante un testo admin affermi il contrario.
|
||||
|
||||
Per ThothII resta da decidere nello step infrastrutturale quali dati di connessione siano normali
|
||||
metadati e quali siano secret reference. In ogni caso:
|
||||
ThothII distingue i metadati di connessione dai riferimenti al secret store cifrato. In ogni caso:
|
||||
|
||||
- nessun endpoint o export deve restituire segreti;
|
||||
- log ed errori devono sanificare DSN e credenziali;
|
||||
@@ -357,6 +502,11 @@ metadati e quali siano secret reference. In ogni caso:
|
||||
- il catalogo non deve riusare credenziali del DWH, delle sessioni o di Qdrant;
|
||||
- test connessione e introspezione devono usare timeout e privilegi read-only.
|
||||
|
||||
La binding REST corrente richiede una verifica prima del cutover: il renderer emette
|
||||
`ssl_ca_file`, mentre il modello Python espone `ssl_ca`; il percorso della CA privata potrebbe quindi
|
||||
non essere consumato. PSD richiede TLS con CA privata in locale, perciò questo disallineamento deve
|
||||
essere corretto e coperto da un test end-to-end prima di affidare il profilo REST al catalogo.
|
||||
|
||||
## Percorso incrementale
|
||||
|
||||
### Step 1: accesso alla superficie vuota
|
||||
@@ -384,15 +534,17 @@ npx tsc -b
|
||||
|
||||
### Step 2: contratto di dominio e schema relazionale
|
||||
|
||||
Da progettare con un nuovo round decisionale: campi, secret reference, dialetti supportati,
|
||||
namespace/schema, snapshot fisiche, relazioni fisiche/logiche e lifecycle dell'output AI. Nessuna
|
||||
tecnologia ORM o migration tool è stata scelta in questo documento.
|
||||
Progettazione della vertical slice completata: Workspace Database, Database Binding, singolo schema,
|
||||
riferimenti al secret store, optimistic concurrency e capability per trasporto hanno contratti
|
||||
espliciti. Configurazione e contenuti semantici restano mutabili; la struttura fisica osservata è
|
||||
sincronizzata e non modificabile manualmente.
|
||||
|
||||
### Step 3: PostgreSQL interno e migrazioni
|
||||
|
||||
Da progettare separatamente dal core: servizio, volume, ruoli runtime/migrator/backup, health e
|
||||
readiness dedicati, backup/restore e diagnostica. La sua indisponibilità non dovrà cambiare
|
||||
`core /health` o interrompere una sessione.
|
||||
PostgreSQL interno con volume e ruoli runtime/migrator separati. Il modulo catalogo usa Kysely sopra
|
||||
il driver `pg`; le migrazioni compilate vengono applicate soltanto dal comando `catalog:migrate` e
|
||||
mai allo startup Fastify. Health, readiness e diagnostica restano dedicate; l'indisponibilità del
|
||||
catalogo non cambia `core /health` e non interrompe una sessione.
|
||||
|
||||
### Step 4: API CRUD
|
||||
|
||||
@@ -401,20 +553,56 @@ Gli endpoint dovranno vivere sotto un namespace catalogo e non riutilizzare le r
|
||||
|
||||
### Step 5: UI CRUD
|
||||
|
||||
Liste e form per Workspace Database, tabelle, colonne e relazioni, costruiti con React/Vite e il
|
||||
design system ThothII. La gerarchia e i filtri ThothAI sono il riferimento funzionale; Django Admin
|
||||
non è il riferimento tecnologico o visuale.
|
||||
Workspace Database, Catalog Table, Catalog Column e Catalog Relationship sono implementati con
|
||||
React/Vite e il design system ThothII.
|
||||
La navigazione è gerarchica e locale al database (`Overview | Tables`), senza menu o filtri globali
|
||||
per tipo di entità. La grid delle tabelle non offre Add/Delete; il dettaglio full-width mantiene
|
||||
immutabili i fatti fisici e consente di modificare separatamente Description e Generated
|
||||
Description. Colonne e relazioni seguono la stessa gerarchia: Columns appartiene al dettaglio
|
||||
della tabella, Relationships al database. I valori descrittivi null sono mostrati come celle e
|
||||
campi vuoti, senza fallback visivi o placeholder `Not set` che nascondano quale sorgente è
|
||||
effettivamente valorizzata.
|
||||
|
||||
Le griglie che dispongono di azioni massive usano checkbox e una toolbar contestuale con conteggio,
|
||||
menu `Actions` e cancellazione della selezione. La selezione identifica ID espliciti, può essere
|
||||
accumulata attraverso i filtri e viene azzerata dopo successo, nuova sincronizzazione o uscita
|
||||
dalla pagina; un'azione è all-or-nothing se un elemento non è idoneo. I menu a livello database
|
||||
espongono gli scope fisici come azioni distinte: `Synchronize tables`, `Synchronize all columns`,
|
||||
`Synchronize relationships` e `Synchronize all`. La grid Tables espone invece `Synchronize
|
||||
columns` per le tabelle selezionate. Test connection resta un'azione distinta; griglie senza azioni
|
||||
non mostrano controlli di selezione inerti.
|
||||
|
||||
### Step 6: introspezione
|
||||
|
||||
Connessione read-only, preview delle differenze, acquisizione di una Physical Schema Snapshot,
|
||||
policy per rename/rimozioni e stato del job. Nessuna chiamata lunga dovrà mantenere aperta una
|
||||
transazione CRUD.
|
||||
Catalog Table, Catalog Column e Catalog Relationship sono implementate per PostgreSQL diretto,
|
||||
Thoth REST Connector e tunnel SSH. La scansione read-only è separata dalla transazione; una
|
||||
riconciliazione atomica crea, aggiorna i commenti sorgente ed elimina, dopo conferma, i fatti fisici
|
||||
assenti senza rendere modificabile manualmente la struttura osservata. Gli scope autorevoli sono
|
||||
Tables per database e Physical Relationships per database. Per Columns, `tableIds` vuoto include
|
||||
tutte le Catalog Table correnti, mentre una lista di ID limita lo scope al sottoinsieme esplicito;
|
||||
`Synchronize all` osserva tutti e tre gli scope in un unico snapshot e li riconcilia insieme. Tutti
|
||||
gli scope sono eseguiti come Catalog Sync Run durevoli in background, non attraverso implementazioni
|
||||
sincrone e asincrone separate. Un run che prevede cancellazioni conserva il diff, attende una
|
||||
conferma esplicita e verifica nuovamente lo snapshot prima dell'applicazione; se la sorgente è
|
||||
cambiata, invalida la conferma. Ogni applicazione è atomica e fail-closed: errori, timeout o
|
||||
capability non disponibili non producono aggiornamenti parziali.
|
||||
|
||||
PK e FK devono essere visibili sulle Catalog Column senza duplicare le stringhe denormalizzate di
|
||||
ThothAI. La posizione nella primary key è un fatto osservato della colonna; membership e conteggio
|
||||
FK sono proiezioni derivate dalle Catalog Relationship e dalle loro coppie ordinate, aggiornate
|
||||
nella stessa transazione di riconciliazione.
|
||||
|
||||
Ogni scope registra la versione della Database Binding osservata e l'istante dell'ultima
|
||||
sincronizzazione. Una modifica della binding conserva il catalogo precedente ma lo marca stale;
|
||||
solo un `Synchronize all` riuscito rende nuovamente corrente l'intero schema.
|
||||
|
||||
### Step 7: generazione AI dei metadati
|
||||
|
||||
Generazione di descrizioni e altri campi equivalenti alle annotations, editing umano e gestione
|
||||
esplicita di errori o output non validi. Approvazione/versioning saranno decisi in questo step.
|
||||
Generated Description è una proposta distinta e modificabile: un revisore può correggerla prima
|
||||
di consolidarla esplicitamente come Description. Lo slice AI dovrà decidere e implementare anche
|
||||
alias semantici, descrizioni dei valori, sinonimi e concetti per tabelle e colonne, oltre alla
|
||||
gestione esplicita di errori e output non validi. La generazione AI e l'azione di consolidamento non
|
||||
appartengono allo slice di introspezione dello schema.
|
||||
|
||||
### Step 8: migrazione PSD
|
||||
|
||||
@@ -427,11 +615,24 @@ ricevono import legacy.
|
||||
Rimuovere la dipendenza da `annotations.yaml` soltanto dopo avere un contratto equivalente,
|
||||
test di rendering/search/Qdrant e una policy di disponibilità. Le sessioni di test esistenti
|
||||
possono essere eliminate, ma le nuove sessioni non devono osservare aggiornamenti parziali.
|
||||
Questo cutover è esplicitamente rinviato fino al completamento del database dei metadati. Il primo
|
||||
gate successivo obbligatorio sarà valutare l'integrazione del Catalog Schema Snapshot con il
|
||||
workflow core e lo schema-linking corrente; il rinvio non autorizza a dimenticare o assorbire
|
||||
implicitamente il lavoro in altri slice.
|
||||
|
||||
### Step 10: operazioni e accettazione
|
||||
|
||||
Backup/restore reale, diagnostica, metriche, audit, permessi definitivi, hardening degli export e
|
||||
test di failure isolation fra catalogo e workflow.
|
||||
Backup/restore reale, diagnostica, metriche, permessi definitivi, hardening degli export e
|
||||
test di failure isolation fra catalogo e workflow. I Catalog Sync Run hanno un solo job attivo per
|
||||
Workspace Database, sono concorrenti fra database diversi e usano un lock persistente. Un pannello
|
||||
operativo non modale rimane visibile durante la navigazione del database, mostra fasi, contatori,
|
||||
tempo trascorso e log sanitizzato via SSE con polling di fallback, e offre Confirm, Cancel e Retry
|
||||
quando consentiti. Un restart marca `interrupted` i run rimasti attivi; il retry crea un nuovo run.
|
||||
Le modifiche ai metadati restano consentite durante la scansione e sono preservate dall'applicazione.
|
||||
Il worker gira inizialmente nello stesso servizio Fastify ma dietro un'interfaccia estraibile, con
|
||||
coda, lease e heartbeat persistiti nel catalog-db. I riepiloghi dei run non scadono; gli eventi
|
||||
dettagliati sono conservati per 30 giorni, mentre snapshot e diff completi vengono eliminati dopo
|
||||
la conclusione lasciando conteggi, decisioni e una sintesi sanitizzata dell'esito.
|
||||
|
||||
## Verifiche del core da conservare per il cutover
|
||||
|
||||
@@ -476,15 +677,11 @@ Definiscono gli effetti semantici e le guardie da mantenere o sostituire consape
|
||||
|
||||
Le seguenti scelte non appartengono allo step 1:
|
||||
|
||||
- framework del servizio catalogo e libreria di accesso PostgreSQL;
|
||||
- collocazione e protezione delle credenziali dei Workspace Database;
|
||||
- supporto iniziale di dialetti diversi da PostgreSQL;
|
||||
- uno o più schema namespace per database;
|
||||
- policy di reconciliation per rename e delete;
|
||||
- modello delle foreign key composite;
|
||||
- distinzione persistente fra relationship fisiche e logiche;
|
||||
- lifecycle draft/review/approval dell'output AI;
|
||||
- versionamento, audit e rollback;
|
||||
- lifecycle dei riferimenti ai segreti durante sostituzione e cancellazione;
|
||||
- criteri per aggiungere dialetti successivi a PostgreSQL;
|
||||
- criteri per un'eventuale estensione futura a più schemi per database;
|
||||
- lifecycle e gestione amministrativa delle future Logical Relationship;
|
||||
- alias semantici, descrizioni dei valori, sinonimi e concetti prodotti o assistiti dall'AI;
|
||||
- formato e momento del cutover dal file al database interno;
|
||||
- permission definitiva separata da `workspace.manage`.
|
||||
|
||||
|
||||
@@ -0,0 +1,265 @@
|
||||
import { apiFetch, assertSameOriginRequestUrl, BASE } from "./client";
|
||||
import { joinBackendPath } from "./runtime-config";
|
||||
|
||||
export type DatabaseTransport = "postgres_direct" | "rest_api" | "ssh_tunnel";
|
||||
export type ConnectionStatus = "untested" | "reachable" | "failed";
|
||||
export type CatalogSecretName =
|
||||
| "password"
|
||||
| "apiKey"
|
||||
| "sshPrivateKey"
|
||||
| "sshPrivateKeyPassphrase"
|
||||
| "sshKnownHosts"
|
||||
| "tlsCa";
|
||||
|
||||
export interface DatabaseBinding {
|
||||
transport: DatabaseTransport;
|
||||
host?: string;
|
||||
port?: number;
|
||||
username?: string;
|
||||
baseUrl?: string;
|
||||
restPath?: string;
|
||||
restAuth?: "none" | "bearer" | "x-api-key";
|
||||
tlsServername?: string;
|
||||
sshHost?: string;
|
||||
sshPort?: number;
|
||||
sshUsername?: string;
|
||||
sshTargetHost?: string;
|
||||
sshTargetPort?: number;
|
||||
}
|
||||
|
||||
export interface CatalogDatabase {
|
||||
id?: string;
|
||||
workspaceId: string;
|
||||
workspaceName: string;
|
||||
workspaceDescription?: string;
|
||||
workspaceAvailable: boolean;
|
||||
configured: boolean;
|
||||
engine: "postgres";
|
||||
databaseName: string;
|
||||
schema: string;
|
||||
version: number;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
binding: DatabaseBinding;
|
||||
connectionStatus: ConnectionStatus;
|
||||
testedVersion?: number;
|
||||
lastTestedAt?: string;
|
||||
lastErrorCode?: string;
|
||||
lastErrorMessage?: string;
|
||||
schemaSyncedVersion?: number;
|
||||
schemaSyncedAt?: string;
|
||||
activeSyncRun?: CatalogSyncRun;
|
||||
secrets: Record<CatalogSecretName, boolean>;
|
||||
}
|
||||
|
||||
export interface DatabaseConfiguration {
|
||||
workspaceId: string;
|
||||
engine: "postgres";
|
||||
databaseName: string;
|
||||
schema: string;
|
||||
binding: DatabaseBinding;
|
||||
}
|
||||
|
||||
export interface CatalogTable {
|
||||
id: string;
|
||||
databaseId: string;
|
||||
name: string;
|
||||
sourceComment: string | null;
|
||||
description: string | null;
|
||||
generatedDescription: string | null;
|
||||
lastSyncedDatabaseVersion?: number | null;
|
||||
lastSyncedAt?: string | null;
|
||||
version: number;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export interface CatalogColumn {
|
||||
id: string;
|
||||
tableId: string;
|
||||
name: string;
|
||||
ordinalPosition: number;
|
||||
dataType: string;
|
||||
isNullable: boolean;
|
||||
defaultExpression: string | null;
|
||||
primaryKeyPosition: number | null;
|
||||
isPrimaryKey: boolean;
|
||||
isForeignKey: boolean;
|
||||
foreignKeyCount: number;
|
||||
sourceComment: string | null;
|
||||
description: string | null;
|
||||
generatedDescription: string | null;
|
||||
lastSyncedDatabaseVersion: number | null;
|
||||
lastSyncedAt: string | null;
|
||||
version: number;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export interface CatalogRelationshipColumn {
|
||||
position: number;
|
||||
sourceColumnId: string;
|
||||
sourceColumnName: string;
|
||||
targetColumnId: string;
|
||||
targetColumnName: string;
|
||||
}
|
||||
|
||||
export interface CatalogRelationship {
|
||||
id: string;
|
||||
databaseId: string;
|
||||
constraintName: string;
|
||||
sourceTableId: string;
|
||||
sourceTableName: string;
|
||||
targetTableId: string;
|
||||
targetTableName: string;
|
||||
updateRule: string;
|
||||
deleteRule: string;
|
||||
deferrable: boolean;
|
||||
initiallyDeferred: boolean;
|
||||
columns: CatalogRelationshipColumn[];
|
||||
lastSyncedDatabaseVersion: number | null;
|
||||
lastSyncedAt: string | null;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
export type CatalogSyncScope = "tables" | "columns" | "relationships" | "all";
|
||||
export type CatalogSyncState = "queued" | "running" | "awaiting_confirmation" | "applying"
|
||||
| "succeeded" | "failed" | "cancelled" | "interrupted";
|
||||
export type CatalogSyncPhase = "queued" | "connecting" | "scanning_tables" | "scanning_columns"
|
||||
| "scanning_relationships" | "planning" | "awaiting_confirmation" | "applying" | "completed";
|
||||
|
||||
export interface CatalogSchemaDiff {
|
||||
deletedTables: string[];
|
||||
deletedColumns: Array<{ tableName: string; columnName: string }>;
|
||||
deletedRelationships: Array<{ sourceTableName: string; constraintName: string }>;
|
||||
}
|
||||
|
||||
export interface CatalogSyncRun {
|
||||
id: string;
|
||||
databaseId: string;
|
||||
scope: CatalogSyncScope;
|
||||
tableIds: string[];
|
||||
state: CatalogSyncState;
|
||||
phase: CatalogSyncPhase;
|
||||
requestedDatabaseVersion: number;
|
||||
plannedDiff: CatalogSchemaDiff | null;
|
||||
confirmationToken: string | null;
|
||||
counts: { tables?: number; columns?: number; relationships?: number; created?: number; updated?: number; deleted?: number };
|
||||
errorCode: string | null;
|
||||
errorMessage: string | null;
|
||||
cancelRequested: boolean;
|
||||
createdAt: string;
|
||||
startedAt: string | null;
|
||||
updatedAt: string;
|
||||
finishedAt: string | null;
|
||||
heartbeatAt: string | null;
|
||||
}
|
||||
|
||||
export interface CatalogSyncEvent {
|
||||
id: number;
|
||||
runId: string;
|
||||
sequence: number;
|
||||
level: "info" | "warning" | "error";
|
||||
eventType: string;
|
||||
message: string;
|
||||
data: Record<string, unknown>;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
export const listCatalogDatabases = () => apiFetch<CatalogDatabase[]>("/catalog/databases");
|
||||
|
||||
export const createCatalogDatabase = (input: DatabaseConfiguration) =>
|
||||
apiFetch<CatalogDatabase>("/catalog/databases", { method: "POST", body: JSON.stringify(input) });
|
||||
|
||||
export const updateCatalogDatabase = (id: string, version: number, input: DatabaseConfiguration) =>
|
||||
apiFetch<CatalogDatabase>(`/catalog/databases/${encodeURIComponent(id)}`, {
|
||||
method: "PATCH",
|
||||
body: JSON.stringify({ ...input, version }),
|
||||
});
|
||||
|
||||
export const replaceCatalogDatabaseSecrets = (
|
||||
id: string,
|
||||
version: number,
|
||||
values: Partial<Record<CatalogSecretName, string>>,
|
||||
) => apiFetch<CatalogDatabase>(`/catalog/databases/${encodeURIComponent(id)}/secrets`, {
|
||||
method: "PUT",
|
||||
body: JSON.stringify({ version, values }),
|
||||
});
|
||||
|
||||
export const testCatalogDatabase = (id: string, version: number) =>
|
||||
apiFetch<CatalogDatabase>(`/catalog/databases/${encodeURIComponent(id)}/test`, {
|
||||
method: "POST",
|
||||
body: JSON.stringify({ version }),
|
||||
});
|
||||
|
||||
export const deleteCatalogDatabase = (id: string, version: number) =>
|
||||
apiFetch<void>(`/catalog/databases/${encodeURIComponent(id)}?version=${version}`, { method: "DELETE" });
|
||||
|
||||
export const listCatalogTables = (databaseId: string) =>
|
||||
apiFetch<CatalogTable[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/tables`);
|
||||
|
||||
export const updateCatalogTableMetadata = (
|
||||
databaseId: string,
|
||||
tableId: string,
|
||||
version: number,
|
||||
description: string | null,
|
||||
generatedDescription: string | null,
|
||||
) => apiFetch<CatalogTable>(
|
||||
`/catalog/databases/${encodeURIComponent(databaseId)}/tables/${encodeURIComponent(tableId)}`,
|
||||
{ method: "PATCH", body: JSON.stringify({ version, description, generatedDescription }) },
|
||||
);
|
||||
|
||||
export const listCatalogColumns = (databaseId: string, tableId: string) =>
|
||||
apiFetch<CatalogColumn[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/tables/${encodeURIComponent(tableId)}/columns`);
|
||||
|
||||
export const updateCatalogColumnMetadata = (
|
||||
databaseId: string,
|
||||
tableId: string,
|
||||
columnId: string,
|
||||
version: number,
|
||||
description: string | null,
|
||||
generatedDescription: string | null,
|
||||
) => apiFetch<CatalogColumn>(
|
||||
`/catalog/databases/${encodeURIComponent(databaseId)}/tables/${encodeURIComponent(tableId)}/columns/${encodeURIComponent(columnId)}`,
|
||||
{ method: "PATCH", body: JSON.stringify({ version, description, generatedDescription }) },
|
||||
);
|
||||
|
||||
export const listCatalogRelationships = (databaseId: string) =>
|
||||
apiFetch<CatalogRelationship[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/relationships`);
|
||||
|
||||
export const startCatalogSync = (
|
||||
databaseId: string,
|
||||
version: number,
|
||||
scope: CatalogSyncScope,
|
||||
tableIds: string[] = [],
|
||||
) => apiFetch<CatalogSyncRun>(`/catalog/databases/${encodeURIComponent(databaseId)}/sync-runs`, {
|
||||
method: "POST",
|
||||
body: JSON.stringify({ version, scope, tableIds }),
|
||||
});
|
||||
|
||||
export const listCatalogSyncRuns = (databaseId: string) =>
|
||||
apiFetch<CatalogSyncRun[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/sync-runs`);
|
||||
|
||||
export const getCatalogSyncRun = (runId: string) =>
|
||||
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}`);
|
||||
|
||||
export const listCatalogSyncEvents = (runId: string, after = 0) =>
|
||||
apiFetch<CatalogSyncEvent[]>(`/catalog/sync-runs/${encodeURIComponent(runId)}/events-list?after=${after}`);
|
||||
|
||||
export const confirmCatalogSync = (runId: string, confirmationToken: string) =>
|
||||
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}/confirm`, {
|
||||
method: "POST", body: JSON.stringify({ confirmationToken }),
|
||||
});
|
||||
|
||||
export const cancelCatalogSync = (runId: string) =>
|
||||
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}/cancel`, { method: "POST" });
|
||||
|
||||
export const retryCatalogSync = (runId: string) =>
|
||||
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}/retry`, { method: "POST" });
|
||||
|
||||
export function catalogSyncEventsUrl(runId: string, after = 0): string {
|
||||
const url = joinBackendPath(BASE, `/catalog/sync-runs/${encodeURIComponent(runId)}/events?after=${after}`);
|
||||
assertSameOriginRequestUrl(url);
|
||||
return url;
|
||||
}
|
||||
@@ -14,6 +14,10 @@ const safeErrorCodes = new Set([
|
||||
"git_unavailable", "git_auth_failed", "git_non_fast_forward", "connector_unavailable",
|
||||
"semantic_index_incompatible", "pi_management_forbidden", "pi_management_unavailable",
|
||||
"pi_management_invalid_config", "pi_management_write_failed",
|
||||
"catalog_unavailable", "database_conflict", "database_invalid", "database_not_found",
|
||||
"database_stale", "database_operation_failed",
|
||||
"schema_sync_conflict", "schema_introspection_failed", "schema_request_invalid",
|
||||
"schema_operation_failed", "sync_run_not_found", "table_stale", "column_stale",
|
||||
]);
|
||||
|
||||
type SafeErrorPayload = {
|
||||
@@ -46,6 +50,19 @@ const localCodeMessages: Record<string, string> = {
|
||||
pi_management_unavailable: "Pi management is unavailable.",
|
||||
pi_management_invalid_config: "The Pi configuration is invalid.",
|
||||
pi_management_write_failed: "The Pi configuration could not be saved.",
|
||||
catalog_unavailable: "The database catalog is unavailable.",
|
||||
database_conflict: "This workspace already has a database configuration.",
|
||||
database_invalid: "The database configuration is invalid.",
|
||||
database_not_found: "The database configuration was not found.",
|
||||
database_stale: "The database configuration changed. Reload and try again.",
|
||||
database_operation_failed: "The database operation failed.",
|
||||
schema_sync_conflict: "A schema synchronization is already active or no longer current.",
|
||||
schema_introspection_failed: "The database schema could not be read safely.",
|
||||
schema_request_invalid: "The schema request is invalid.",
|
||||
schema_operation_failed: "The schema operation failed.",
|
||||
sync_run_not_found: "The synchronization run was not found.",
|
||||
table_stale: "Table metadata changed. Reload and try again.",
|
||||
column_stale: "Column metadata changed. Reload and try again.",
|
||||
};
|
||||
|
||||
const localStatusMessages: Record<number, string> = {
|
||||
|
||||
@@ -296,3 +296,52 @@
|
||||
0%, 100% { transform: scale(1); box-shadow: 0 0 0 0 oklch(var(--primary) / 0.4); }
|
||||
50% { transform: scale(1.15); box-shadow: 0 0 0 5px oklch(var(--primary) / 0); }
|
||||
}
|
||||
|
||||
@layer components {
|
||||
.thot-database-grid {
|
||||
--ag-font-family: var(--font-sans);
|
||||
--ag-font-size: 0.8125rem;
|
||||
--ag-background-color: oklch(var(--card));
|
||||
--ag-foreground-color: oklch(var(--foreground));
|
||||
--ag-header-background-color: oklch(var(--muted) / 0.72);
|
||||
--ag-header-foreground-color: oklch(var(--foreground));
|
||||
--ag-border-color: oklch(var(--border));
|
||||
--ag-row-border-color: oklch(var(--border) / 0.72);
|
||||
--ag-odd-row-background-color: oklch(var(--muted) / 0.18);
|
||||
--ag-row-hover-color: oklch(var(--muted) / 0.55);
|
||||
--ag-selected-row-background-color: oklch(var(--primary) / 0.07);
|
||||
--ag-input-focus-border-color: oklch(var(--primary) / 0.6);
|
||||
--ag-range-selection-border-color: oklch(var(--primary) / 0.6);
|
||||
--ag-header-column-separator-color: oklch(var(--border));
|
||||
--ag-header-column-separator-display: block;
|
||||
--ag-wrapper-border-radius: 0;
|
||||
--ag-cell-horizontal-padding: 12px;
|
||||
}
|
||||
.thot-database-grid .ag-root-wrapper {
|
||||
border: 0;
|
||||
}
|
||||
.thot-database-grid .ag-cell {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
}
|
||||
.thot-database-grid .ag-header-cell-label {
|
||||
font-weight: 700;
|
||||
}
|
||||
.thot-database-grid .ag-cell-focus:not(.ag-cell-range-selected) {
|
||||
outline: 2px solid oklch(var(--ring) / 0.45);
|
||||
outline-offset: -2px;
|
||||
}
|
||||
.thot-database-grid .ag-cell.thot-database-actions-cell {
|
||||
padding-inline: 2px;
|
||||
}
|
||||
.thot-database-grid .ag-cell.thot-database-actions-cell.ag-cell-focus:not(.ag-cell-range-selected) {
|
||||
outline: none;
|
||||
}
|
||||
.thot-database-grid .ag-cell.thot-database-status-cell {
|
||||
padding-inline: 4px;
|
||||
}
|
||||
.thot-database-grid .ag-pinned-right-header,
|
||||
.thot-database-grid .ag-pinned-right-cols-container {
|
||||
box-shadow: -1px 0 0 oklch(var(--border));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -23,7 +23,7 @@ beforeEach(() => {
|
||||
issuer: "test",
|
||||
subject: "test",
|
||||
roles: ["admin"],
|
||||
permissions: ["session.use", "workspace.manage", "workspace.secrets.manage", "pi.manage"],
|
||||
permissions: ["session.use", "workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage"],
|
||||
isAdmin: true,
|
||||
csrfToken: null,
|
||||
session: null,
|
||||
@@ -52,7 +52,7 @@ beforeEach(() => {
|
||||
);
|
||||
});
|
||||
|
||||
test("opens the blank database management surface and returns to the core", async () => {
|
||||
test("opens the database management surface and returns to the core", async () => {
|
||||
renderShell();
|
||||
|
||||
const composer = screen.getByRole("textbox", { name: /new question/i });
|
||||
@@ -122,7 +122,7 @@ test("keeps a live core session connected and returns when that session is opene
|
||||
expect(FakeEventSource.instances).toHaveLength(1);
|
||||
});
|
||||
|
||||
test("hides all management entries from non-admin users", () => {
|
||||
test("keeps read-safe workspace access but hides privileged management entries", () => {
|
||||
clearAuthState();
|
||||
setAuthState({
|
||||
issuer: "test",
|
||||
@@ -136,7 +136,52 @@ test("hides all management entries from non-admin users", () => {
|
||||
|
||||
renderShell();
|
||||
|
||||
expect(screen.queryByRole("button", { name: "Workspace management" })).not.toBeInTheDocument();
|
||||
expect(screen.getByRole("button", { name: "Workspace management" })).toBeInTheDocument();
|
||||
expect(screen.queryByRole("button", { name: "Database management" })).not.toBeInTheDocument();
|
||||
expect(screen.queryByRole("button", { name: "Pi management" })).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
test("does not leave database management without confirming a dirty form", async () => {
|
||||
server.use(http.get("/api/catalog/databases", () => HttpResponse.json([{
|
||||
id: "11111111-1111-4111-8111-111111111111",
|
||||
workspaceId: "psd-clinical",
|
||||
workspaceName: "Policlinico San Donato",
|
||||
workspaceAvailable: true,
|
||||
configured: true,
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
version: 3,
|
||||
createdAt: "2026-08-27T08:00:00Z",
|
||||
updatedAt: "2026-08-27T09:00:00Z",
|
||||
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||
connectionStatus: "untested",
|
||||
secrets: {
|
||||
password: false,
|
||||
apiKey: false,
|
||||
sshPrivateKey: false,
|
||||
sshPrivateKeyPassphrase: false,
|
||||
sshKnownHosts: false,
|
||||
tlsCa: false,
|
||||
},
|
||||
}])));
|
||||
const confirm = vi.spyOn(window, "confirm").mockReturnValue(false);
|
||||
renderShell();
|
||||
|
||||
await userEvent.click(screen.getByRole("button", { name: "Database management" }));
|
||||
await userEvent.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||
const schema = screen.getByLabelText("Schema");
|
||||
await userEvent.clear(schema);
|
||||
await userEvent.type(schema, "reporting");
|
||||
|
||||
await userEvent.click(screen.getByRole("button", { name: "New session" }));
|
||||
|
||||
expect(confirm).toHaveBeenCalledWith("Discard unsaved database changes and leave database management?");
|
||||
expect(screen.getByRole("main", { name: "Database management" })).toBeVisible();
|
||||
expect(schema).toHaveValue("reporting");
|
||||
|
||||
confirm.mockReturnValue(true);
|
||||
await userEvent.click(screen.getByRole("button", { name: "New session" }));
|
||||
await waitFor(() => expect(screen.queryByRole("main", { name: "Database management" })).not.toBeInTheDocument());
|
||||
confirm.mockRestore();
|
||||
});
|
||||
|
||||
@@ -32,7 +32,7 @@ import {
|
||||
import type { SessionScope, SessionSummary } from "../api/types";
|
||||
import { useAuthGeneration, useAuthUser } from "../auth/authState";
|
||||
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { useEffect, useMemo, useRef, useState } from "react";
|
||||
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
|
||||
import type { CSSProperties } from "react";
|
||||
import { captureAuthOperation, isAuthOperationCurrent, StaleAuthOperationError, type AuthOperationGuard } from "../auth/authOperation";
|
||||
|
||||
@@ -104,8 +104,8 @@ export function AppShell({ canLogout }: AppShellProps) {
|
||||
const canReadAllSessions = permissions.includes("session.read_all");
|
||||
const canManageWorkspace = permissions.includes("workspace.manage");
|
||||
const canManageWorkspaceSecrets = permissions.includes("workspace.secrets.manage");
|
||||
const canManageDatabase = permissions.includes("database.manage");
|
||||
const canManagePi = permissions.includes("pi.manage");
|
||||
const isAdmin = authenticatedUser?.isAdmin === true;
|
||||
const authGeneration = useAuthGeneration();
|
||||
const { data: sessions = [] } = useQuery<SessionSummary[]>({
|
||||
queryKey: ["sessions", sessionScope],
|
||||
@@ -123,6 +123,10 @@ export function AppShell({ canLogout }: AppShellProps) {
|
||||
const queryClient = useQueryClient();
|
||||
const [showActivity, setShowActivity] = useState(false);
|
||||
const [activeSurface, setActiveSurface] = useState<ActiveSurface>("core");
|
||||
const databaseNavigationRef = useRef({ dirty: false, busy: false });
|
||||
const updateDatabaseNavigationState = useCallback((state: { dirty: boolean; busy: boolean }) => {
|
||||
databaseNavigationRef.current = state;
|
||||
}, []);
|
||||
const [workspaceManagerOpen, setWorkspaceManagerOpen] = useState(false);
|
||||
const [piManagementOpen, setPiManagementOpen] = useState(false);
|
||||
const [activeOpen, setActiveOpen] = useState(true);
|
||||
@@ -197,7 +201,20 @@ export function AppShell({ canLogout }: AppShellProps) {
|
||||
setSelectedSessionIds(selected ? new Set(sessions.map((session) => session.id)) : new Set());
|
||||
}
|
||||
|
||||
function canLeaveDatabaseManagement(): boolean {
|
||||
if (activeSurface !== "database-management") return true;
|
||||
if (databaseNavigationRef.current.busy) {
|
||||
toast.info("Wait for the database operation to finish before leaving this page");
|
||||
return false;
|
||||
}
|
||||
if (databaseNavigationRef.current.dirty) {
|
||||
return window.confirm("Discard unsaved database changes and leave database management?");
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
function openPanel(id: string) {
|
||||
if (!canLeaveDatabaseManagement()) return;
|
||||
const s = sessions.find((x) => x.id === id);
|
||||
if (!s) return;
|
||||
setActiveSurface("core");
|
||||
@@ -207,7 +224,7 @@ export function AppShell({ canLogout }: AppShellProps) {
|
||||
// completed sessions keep the read-only documents panel (with its explicit Resume),
|
||||
// so a mere click never spawns a runtime.
|
||||
if (s.active && s.status === "open" && !s.archived && !isForeignSession(s)) {
|
||||
void doResume(id);
|
||||
void doResume(id, true);
|
||||
return;
|
||||
}
|
||||
setPanelSession(s); setShowActivity(false);
|
||||
@@ -219,7 +236,8 @@ export function AppShell({ canLogout }: AppShellProps) {
|
||||
return next;
|
||||
});
|
||||
}
|
||||
async function doResume(id: string) {
|
||||
async function doResume(id: string, databaseExitApproved = false) {
|
||||
if (!databaseExitApproved && !canLeaveDatabaseManagement()) return;
|
||||
setActiveSurface("core");
|
||||
const guard = captureAuthOperation({ sessionId: id, disposalEpoch: operationEpochRef.current });
|
||||
if (!guard) return;
|
||||
@@ -502,6 +520,7 @@ export function AppShell({ canLogout }: AppShellProps) {
|
||||
}, [lastSystemEvent]);
|
||||
|
||||
function startNewSession() {
|
||||
if (!canLeaveDatabaseManagement()) return;
|
||||
setActiveSurface("core");
|
||||
invalidateResumeIntent();
|
||||
newSessionOperationRef.current = null;
|
||||
@@ -566,6 +585,7 @@ export function AppShell({ canLogout }: AppShellProps) {
|
||||
}
|
||||
|
||||
async function signOut() {
|
||||
if (!canLeaveDatabaseManagement()) return;
|
||||
await logoutUser();
|
||||
}
|
||||
|
||||
@@ -619,7 +639,13 @@ export function AppShell({ canLogout }: AppShellProps) {
|
||||
|
||||
{/* Conversation column */}
|
||||
<div data-testid="conversation-column" className="flex min-w-0 flex-1 flex-col">
|
||||
{activeSurface === "database-management" && <DatabaseManagementPage />}
|
||||
{activeSurface === "database-management" && (
|
||||
<DatabaseManagementPage
|
||||
canManage={canManageDatabase}
|
||||
canManageSecrets={canManageWorkspaceSecrets}
|
||||
onNavigationStateChange={updateDatabaseNavigationState}
|
||||
/>
|
||||
)}
|
||||
<div
|
||||
aria-hidden={activeSurface !== "core"}
|
||||
className={activeSurface === "core" ? "contents" : "hidden"}
|
||||
@@ -738,16 +764,19 @@ export function AppShell({ canLogout }: AppShellProps) {
|
||||
>
|
||||
New session
|
||||
</Button>
|
||||
{isAdmin && (
|
||||
<>
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
className="w-full"
|
||||
onClick={() => setWorkspaceManagerOpen(true)}
|
||||
>
|
||||
Workspace management
|
||||
</Button>
|
||||
onClick={() => {
|
||||
if (!canLeaveDatabaseManagement()) return;
|
||||
setActiveSurface("core");
|
||||
setWorkspaceManagerOpen(true);
|
||||
}}
|
||||
>
|
||||
Workspace management
|
||||
</Button>
|
||||
{canManageDatabase && (
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
@@ -757,14 +786,17 @@ export function AppShell({ canLogout }: AppShellProps) {
|
||||
>
|
||||
Database management
|
||||
</Button>
|
||||
</>
|
||||
)}
|
||||
{isAdmin && canManagePi && (
|
||||
{canManagePi && (
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
className="w-full"
|
||||
onClick={() => setPiManagementOpen(true)}
|
||||
onClick={() => {
|
||||
if (!canLeaveDatabaseManagement()) return;
|
||||
setActiveSurface("core");
|
||||
setPiManagementOpen(true);
|
||||
}}
|
||||
>
|
||||
Pi management
|
||||
</Button>
|
||||
|
||||
@@ -0,0 +1,842 @@
|
||||
import { act, render, screen, waitFor, within } from "@testing-library/react";
|
||||
import userEvent from "@testing-library/user-event";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import { http, HttpResponse } from "msw";
|
||||
import { server } from "../test/msw";
|
||||
import type { CatalogColumn, CatalogDatabase, CatalogSyncRun, CatalogTable } from "../api/catalog-databases";
|
||||
import { DatabaseManagementPage } from "./DatabaseManagementPage";
|
||||
|
||||
const noSecrets = {
|
||||
password: false,
|
||||
apiKey: false,
|
||||
sshPrivateKey: false,
|
||||
sshPrivateKeyPassphrase: false,
|
||||
sshKnownHosts: false,
|
||||
tlsCa: false,
|
||||
};
|
||||
|
||||
function makeDatabase(overrides: Partial<CatalogDatabase> = {}): CatalogDatabase {
|
||||
return {
|
||||
id: "11111111-1111-4111-8111-111111111111",
|
||||
workspaceId: "psd-clinical",
|
||||
workspaceName: "Policlinico San Donato",
|
||||
workspaceAvailable: true,
|
||||
configured: true,
|
||||
engine: "postgres",
|
||||
databaseName: "warehouse",
|
||||
schema: "datawarehouse",
|
||||
version: 3,
|
||||
createdAt: "2026-08-27T08:00:00Z",
|
||||
updatedAt: "2026-08-27T09:00:00Z",
|
||||
binding: {
|
||||
transport: "postgres_direct",
|
||||
host: "db.internal",
|
||||
port: 5432,
|
||||
username: "reader",
|
||||
},
|
||||
connectionStatus: "untested",
|
||||
secrets: { ...noSecrets },
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
const unconfigured = makeDatabase({
|
||||
id: undefined,
|
||||
workspaceId: "lab",
|
||||
workspaceName: "Research laboratory",
|
||||
configured: false,
|
||||
databaseName: "lab_warehouse",
|
||||
schema: "analytics",
|
||||
version: 0,
|
||||
createdAt: "",
|
||||
updatedAt: "",
|
||||
binding: { transport: "postgres_direct", port: 5432 },
|
||||
});
|
||||
|
||||
const orphan = makeDatabase({
|
||||
id: "22222222-2222-4222-8222-222222222222",
|
||||
workspaceId: "retired",
|
||||
workspaceName: "Retired workspace",
|
||||
workspaceAvailable: false,
|
||||
});
|
||||
|
||||
function renderPage({
|
||||
rows = [makeDatabase(), unconfigured, orphan],
|
||||
canManage = true,
|
||||
canManageSecrets = true,
|
||||
onNavigationStateChange,
|
||||
}: {
|
||||
rows?: CatalogDatabase[] | (() => CatalogDatabase[]);
|
||||
canManage?: boolean;
|
||||
canManageSecrets?: boolean;
|
||||
onNavigationStateChange?: (state: { dirty: boolean; busy: boolean }) => void;
|
||||
} = {}) {
|
||||
server.use(http.get("/api/catalog/databases", () => HttpResponse.json(
|
||||
typeof rows === "function" ? rows() : rows,
|
||||
)));
|
||||
const client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
|
||||
const view = render(
|
||||
<QueryClientProvider client={client}>
|
||||
<DatabaseManagementPage
|
||||
canManage={canManage}
|
||||
canManageSecrets={canManageSecrets}
|
||||
onNavigationStateChange={onNavigationStateChange}
|
||||
/>
|
||||
</QueryClientProvider>,
|
||||
);
|
||||
return { ...view, client };
|
||||
}
|
||||
|
||||
function makeSyncRun(
|
||||
scope: CatalogSyncRun["scope"],
|
||||
tableIds: string[] = [],
|
||||
): CatalogSyncRun {
|
||||
return {
|
||||
id: `sync-${scope}`,
|
||||
databaseId: "11111111-1111-4111-8111-111111111111",
|
||||
scope,
|
||||
tableIds,
|
||||
state: "queued",
|
||||
phase: "queued",
|
||||
requestedDatabaseVersion: 3,
|
||||
plannedDiff: null,
|
||||
confirmationToken: null,
|
||||
counts: {},
|
||||
errorCode: null,
|
||||
errorMessage: null,
|
||||
cancelRequested: false,
|
||||
createdAt: "2026-08-27T10:00:00Z",
|
||||
startedAt: null,
|
||||
updatedAt: "2026-08-27T10:00:00Z",
|
||||
finishedAt: null,
|
||||
heartbeatAt: null,
|
||||
};
|
||||
}
|
||||
|
||||
function registerCompletedSyncRun(run: CatalogSyncRun) {
|
||||
const completed: CatalogSyncRun = {
|
||||
...run,
|
||||
state: "succeeded",
|
||||
phase: "completed",
|
||||
startedAt: "2026-08-27T10:00:00Z",
|
||||
finishedAt: "2026-08-27T10:00:01Z",
|
||||
};
|
||||
server.use(
|
||||
http.get("/api/catalog/sync-runs/:runId", () => HttpResponse.json(completed)),
|
||||
http.get("/api/catalog/sync-runs/:runId/events-list", () => HttpResponse.json([])),
|
||||
http.get("/api/catalog/databases/:databaseId/sync-runs", () => HttpResponse.json([completed])),
|
||||
);
|
||||
}
|
||||
|
||||
const synchronizationScopes = [
|
||||
{ scope: "tables", label: "Synchronize tables" },
|
||||
{ scope: "columns", label: "Synchronize all columns" },
|
||||
{ scope: "relationships", label: "Synchronize relationships" },
|
||||
{ scope: "all", label: "Synchronize all" },
|
||||
] as const;
|
||||
|
||||
test("starts with a full-width list and applies the row action matrix", async () => {
|
||||
renderPage();
|
||||
|
||||
expect(await screen.findByRole("button", { name: "View Policlinico San Donato" })).toBeEnabled();
|
||||
expect(screen.getByRole("button", { name: "Edit Policlinico San Donato" })).toBeEnabled();
|
||||
expect(screen.getByRole("button", { name: "Delete Policlinico San Donato" })).toBeEnabled();
|
||||
|
||||
expect(screen.getByRole("button", { name: "View Research laboratory" })).toBeEnabled();
|
||||
expect(screen.getByRole("button", { name: "Edit Research laboratory" })).toBeEnabled();
|
||||
expect(screen.getByRole("button", { name: "Delete Research laboratory" })).toBeDisabled();
|
||||
|
||||
expect(screen.getByRole("button", { name: "View Retired workspace" })).toBeEnabled();
|
||||
expect(screen.getByRole("button", { name: "Edit Retired workspace" })).toBeDisabled();
|
||||
expect(screen.getByRole("button", { name: "Delete Retired workspace" })).toBeEnabled();
|
||||
expect(screen.queryByRole("heading", { name: "Database details" })).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
test.each(synchronizationScopes)(
|
||||
"selected database Actions offers and starts $scope synchronization",
|
||||
async ({ scope, label }) => {
|
||||
const user = userEvent.setup();
|
||||
let startBody: unknown;
|
||||
const run = makeSyncRun(scope);
|
||||
registerCompletedSyncRun(run);
|
||||
server.use(http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
|
||||
startBody = await request.json();
|
||||
return HttpResponse.json(run, { status: 202 });
|
||||
}));
|
||||
renderPage({
|
||||
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||
});
|
||||
|
||||
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
|
||||
await user.click(within(databaseRow).getByRole("checkbox"));
|
||||
await user.click(screen.getByRole("button", { name: "Actions" }));
|
||||
|
||||
expect(await screen.findByRole("menuitem", { name: synchronizationScopes[0].label })).toBeEnabled();
|
||||
for (const option of synchronizationScopes.slice(1)) {
|
||||
expect(screen.getByRole("menuitem", { name: option.label })).toBeEnabled();
|
||||
}
|
||||
await user.click(screen.getByRole("menuitem", { name: label }));
|
||||
|
||||
await waitFor(() => expect(startBody).toEqual({ version: 3, scope, tableIds: [] }));
|
||||
},
|
||||
);
|
||||
|
||||
test("presents completed synchronization steps as success and skips unneeded confirmation", async () => {
|
||||
const user = userEvent.setup();
|
||||
const run = {
|
||||
...makeSyncRun("columns"),
|
||||
counts: { tables: 163, columns: 2_275 },
|
||||
};
|
||||
registerCompletedSyncRun(run);
|
||||
server.use(http.post("/api/catalog/databases/:databaseId/sync-runs", () => (
|
||||
HttpResponse.json(run, { status: 202 })
|
||||
)));
|
||||
renderPage({
|
||||
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||
});
|
||||
|
||||
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
|
||||
await user.click(within(databaseRow).getByRole("checkbox"));
|
||||
await user.click(screen.getByRole("button", { name: "Actions" }));
|
||||
await user.click(await screen.findByRole("menuitem", { name: "Synchronize all columns" }));
|
||||
|
||||
expect(await screen.findByRole("heading", { name: "Succeeded" })).toBeVisible();
|
||||
const steps = screen.getByRole("list", { name: "Synchronization steps" });
|
||||
expect(steps).toHaveClass("grid-cols-2", "sm:grid-cols-3");
|
||||
const connected = within(steps).getByRole("listitem", { name: "Connected, completed" });
|
||||
expect(connected.querySelector("svg")).toHaveClass("text-[oklch(var(--success))]");
|
||||
expect(within(steps).getByText("Tables read")).toBeVisible();
|
||||
const confirmation = within(steps).getByRole("listitem", {
|
||||
name: "Confirmation not required, not required",
|
||||
});
|
||||
expect(confirmation.querySelector("svg")).toHaveClass("text-muted-foreground");
|
||||
const completed = within(steps).getByRole("listitem", { name: "Completed, completed" });
|
||||
expect(completed.querySelector("svg")).toHaveClass("text-[oklch(var(--success))]");
|
||||
expect(steps.querySelectorAll(".text-primary")).toHaveLength(0);
|
||||
});
|
||||
|
||||
test("database Overview exposes every synchronization scope", async () => {
|
||||
const user = userEvent.setup();
|
||||
let startBody: unknown;
|
||||
const run = makeSyncRun("relationships");
|
||||
registerCompletedSyncRun(run);
|
||||
server.use(http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
|
||||
startBody = await request.json();
|
||||
return HttpResponse.json(run, { status: 202 });
|
||||
}));
|
||||
renderPage({
|
||||
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||
});
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||
await user.click(screen.getByRole("button", { name: "Synchronize database schema" }));
|
||||
|
||||
expect(await screen.findByRole("menuitem", { name: synchronizationScopes[0].label })).toBeEnabled();
|
||||
for (const option of synchronizationScopes.slice(1)) {
|
||||
expect(screen.getByRole("menuitem", { name: option.label })).toBeEnabled();
|
||||
}
|
||||
await user.click(screen.getByRole("menuitem", { name: "Synchronize relationships" }));
|
||||
|
||||
await waitFor(() => expect(startBody).toEqual({
|
||||
version: 3,
|
||||
scope: "relationships",
|
||||
tableIds: [],
|
||||
}));
|
||||
});
|
||||
|
||||
test("replaces the list with View and returns focus to the originating action", async () => {
|
||||
const user = userEvent.setup();
|
||||
renderPage({ rows: [makeDatabase()] });
|
||||
const view = await screen.findByRole("button", { name: "View Policlinico San Donato" });
|
||||
|
||||
await user.click(view);
|
||||
|
||||
expect(screen.getByRole("heading", { name: "Database details" })).toBeVisible();
|
||||
expect(screen.getByDisplayValue("db.internal")).toHaveAttribute("readonly");
|
||||
expect(screen.getByRole("button", { name: "Test connection" })).toBeEnabled();
|
||||
expect(screen.queryByRole("button", { name: "Save changes" })).not.toBeInTheDocument();
|
||||
|
||||
await user.click(screen.getByRole("button", { name: "Back to list" }));
|
||||
|
||||
await waitFor(() => expect(view).toHaveFocus());
|
||||
expect(screen.getByRole("button", { name: "View Policlinico San Donato" })).toBeVisible();
|
||||
});
|
||||
|
||||
test("opens an unconfigured row as Edit while creating its first saved configuration", async () => {
|
||||
const user = userEvent.setup();
|
||||
let postBody: unknown;
|
||||
const saved = makeDatabase({
|
||||
...unconfigured,
|
||||
id: "33333333-3333-4333-8333-333333333333",
|
||||
configured: true,
|
||||
version: 1,
|
||||
binding: { transport: "rest_api", baseUrl: "https://psd.example/api", restPath: "/health", restAuth: "x-api-key" },
|
||||
});
|
||||
let rows = [unconfigured];
|
||||
server.use(
|
||||
http.post("/api/catalog/databases", async ({ request }) => {
|
||||
postBody = await request.json();
|
||||
rows = [saved];
|
||||
return HttpResponse.json(saved, { status: 201 });
|
||||
}),
|
||||
);
|
||||
renderPage({ rows: () => rows });
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "Edit Research laboratory" }));
|
||||
|
||||
expect(screen.getByRole("heading", { name: "Edit database" })).toBeVisible();
|
||||
expect(screen.getByLabelText("Workspace")).toBeDisabled();
|
||||
await user.selectOptions(screen.getByLabelText("Transport"), "rest_api");
|
||||
await user.type(screen.getByLabelText("Base URL"), "https://psd.example/api");
|
||||
expect(screen.getByLabelText("Diagnostic endpoint")).toHaveValue("/health");
|
||||
expect(screen.getByLabelText("Diagnostic endpoint")).toHaveAttribute("readonly");
|
||||
|
||||
await user.click(screen.getByRole("button", { name: "Save database" }));
|
||||
|
||||
await waitFor(() => expect(postBody).toMatchObject({
|
||||
workspaceId: "lab",
|
||||
binding: expect.objectContaining({ transport: "rest_api", baseUrl: "https://psd.example/api" }),
|
||||
}));
|
||||
expect(await screen.findByRole("heading", { name: "Edit database" })).toBeVisible();
|
||||
});
|
||||
|
||||
test("global Add offers only unconfigured workspaces and lets the operator choose one", async () => {
|
||||
const user = userEvent.setup();
|
||||
const second = makeDatabase({
|
||||
...unconfigured,
|
||||
workspaceId: "radiology",
|
||||
workspaceName: "Radiology",
|
||||
databaseName: "radiology_dwh",
|
||||
});
|
||||
renderPage({ rows: [makeDatabase(), unconfigured, second] });
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "Add database" }));
|
||||
|
||||
expect(screen.getByRole("heading", { name: "Add database" })).toBeVisible();
|
||||
expect(screen.getByRole("button", { name: "Add database" })).toBeVisible();
|
||||
const selector = screen.getByLabelText("Workspace");
|
||||
expect(selector).toBeEnabled();
|
||||
expect(screen.queryByRole("option", { name: "Policlinico San Donato" })).not.toBeInTheDocument();
|
||||
await user.selectOptions(selector, "radiology");
|
||||
expect(screen.getByDisplayValue("radiology_dwh")).toBeVisible();
|
||||
});
|
||||
|
||||
test("guards a dirty Edit, disables testing, and reports navigation state", async () => {
|
||||
const user = userEvent.setup();
|
||||
const navigation = vi.fn();
|
||||
const confirm = vi.spyOn(window, "confirm").mockReturnValue(false);
|
||||
renderPage({ rows: [makeDatabase()], onNavigationStateChange: navigation });
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||
const schema = screen.getByLabelText("Schema");
|
||||
await user.clear(schema);
|
||||
await user.type(schema, "reporting");
|
||||
|
||||
expect(screen.getByRole("button", { name: "Test connection" })).toBeDisabled();
|
||||
await waitFor(() => expect(navigation).toHaveBeenLastCalledWith({ dirty: true, busy: false }));
|
||||
await user.click(screen.getByRole("button", { name: "Back to list" }));
|
||||
expect(confirm).toHaveBeenCalledWith("Discard unsaved database changes?");
|
||||
expect(screen.getByRole("heading", { name: "Edit database" })).toBeVisible();
|
||||
|
||||
confirm.mockReturnValue(true);
|
||||
await user.click(screen.getByRole("button", { name: "Back to list" }));
|
||||
expect(await screen.findByRole("button", { name: "Edit Policlinico San Donato" })).toBeVisible();
|
||||
});
|
||||
|
||||
test("uses an in-page destructive form and returns the YAML workspace to Not configured", async () => {
|
||||
const user = userEvent.setup();
|
||||
let row = makeDatabase();
|
||||
let deleteCalled = false;
|
||||
let deleteVersion: string | null = null;
|
||||
server.use(
|
||||
http.delete("/api/catalog/databases/:id", ({ request }) => {
|
||||
deleteCalled = true;
|
||||
deleteVersion = new URL(request.url).searchParams.get("version");
|
||||
row = {
|
||||
...row,
|
||||
id: undefined,
|
||||
configured: false,
|
||||
version: 0,
|
||||
updatedAt: "",
|
||||
connectionStatus: "untested",
|
||||
secrets: { ...noSecrets },
|
||||
};
|
||||
return new HttpResponse(null, { status: 204 });
|
||||
}),
|
||||
);
|
||||
renderPage({ rows: () => [row] });
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "Delete Policlinico San Donato" }));
|
||||
expect(screen.getByRole("heading", { name: "Delete database" })).toBeVisible();
|
||||
expect(screen.getByText("This removes the local database configuration.")).toBeVisible();
|
||||
expect(screen.queryByRole("dialog")).not.toBeInTheDocument();
|
||||
|
||||
await user.click(screen.getByRole("button", { name: "Delete database" }));
|
||||
|
||||
await waitFor(() => expect(deleteCalled).toBe(true));
|
||||
expect(deleteVersion).toBe("3");
|
||||
expect(await screen.findByRole("button", { name: "Delete Policlinico San Donato" })).toBeDisabled();
|
||||
expect(screen.getByText("Not configured")).toBeVisible();
|
||||
});
|
||||
|
||||
test("tests only the persisted version and updates the visible connection status", async () => {
|
||||
const user = userEvent.setup();
|
||||
let testBody: unknown;
|
||||
let row = makeDatabase();
|
||||
server.use(http.post("/api/catalog/databases/:id/test", async ({ request }) => {
|
||||
testBody = await request.json();
|
||||
row = makeDatabase({ version: 4, connectionStatus: "reachable", testedVersion: 4 });
|
||||
return HttpResponse.json(row);
|
||||
}));
|
||||
renderPage({ rows: () => [row] });
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||
await user.click(screen.getByRole("button", { name: "Test connection" }));
|
||||
|
||||
await waitFor(() => expect(testBody).toEqual({ version: 3 }));
|
||||
expect(within(screen.getByRole("region", { name: "Database details form" })).getByText("Reachable")).toBeVisible();
|
||||
});
|
||||
|
||||
test("retains a stale draft and requires an explicit reload", async () => {
|
||||
const user = userEvent.setup();
|
||||
server.use(
|
||||
http.patch("/api/catalog/databases/:id", () => HttpResponse.json({
|
||||
code: "database_stale",
|
||||
message: "The database changed",
|
||||
}, { status: 409 })),
|
||||
);
|
||||
renderPage({ rows: [makeDatabase()] });
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||
const schema = screen.getByLabelText("Schema");
|
||||
await user.clear(schema);
|
||||
await user.type(schema, "draft_schema");
|
||||
await user.click(screen.getByRole("button", { name: "Save changes" }));
|
||||
|
||||
expect(await screen.findByText("A newer database configuration is available.")).toBeVisible();
|
||||
expect(schema).toHaveValue("draft_schema");
|
||||
expect(screen.getByRole("button", { name: "Save changes" })).toBeDisabled();
|
||||
|
||||
await user.click(screen.getByRole("button", { name: "Keep editing" }));
|
||||
expect(screen.getByText("Reload the latest values before this configuration can be changed.")).toBeVisible();
|
||||
expect(schema).toHaveValue("draft_schema");
|
||||
});
|
||||
|
||||
test("marks an open form stale when a background refetch advances the catalog version", async () => {
|
||||
const user = userEvent.setup();
|
||||
const { client } = renderPage({ rows: [makeDatabase()] });
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||
act(() => {
|
||||
client.setQueryData(["catalog-databases"], [makeDatabase({ version: 4, schema: "server_schema" })]);
|
||||
});
|
||||
|
||||
expect(await screen.findByText("A newer database configuration is available.")).toBeVisible();
|
||||
expect(screen.getByLabelText("Schema")).toHaveValue("datawarehouse");
|
||||
expect(screen.getByRole("button", { name: "Save changes" })).toBeDisabled();
|
||||
});
|
||||
|
||||
test("does not submit credentials hidden by a transport or authentication change", async () => {
|
||||
const user = userEvent.setup();
|
||||
let putCalled = false;
|
||||
let patchBody: Record<string, unknown> | undefined;
|
||||
let row = makeDatabase();
|
||||
server.use(
|
||||
http.patch("/api/catalog/databases/:id", async ({ request }) => {
|
||||
patchBody = await request.json() as Record<string, unknown>;
|
||||
row = makeDatabase({
|
||||
version: 4,
|
||||
binding: { transport: "rest_api", baseUrl: "https://psd.example/api", restPath: "/health", restAuth: "none" },
|
||||
});
|
||||
return HttpResponse.json(row);
|
||||
}),
|
||||
http.put("/api/catalog/databases/:id/secrets", () => {
|
||||
putCalled = true;
|
||||
return HttpResponse.json(row);
|
||||
}),
|
||||
);
|
||||
renderPage({ rows: () => [row] });
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||
await user.type(screen.getByLabelText("Password"), "must-not-be-sent");
|
||||
await user.selectOptions(screen.getByLabelText("Transport"), "rest_api");
|
||||
await user.type(screen.getByLabelText("Base URL"), "https://psd.example/api");
|
||||
await user.selectOptions(screen.getByLabelText("Authentication"), "none");
|
||||
await user.click(screen.getByRole("button", { name: "Save changes" }));
|
||||
|
||||
await waitFor(() => expect(patchBody).toBeDefined());
|
||||
expect(putCalled).toBe(false);
|
||||
expect(patchBody).toMatchObject({
|
||||
version: 3,
|
||||
binding: {
|
||||
transport: "rest_api",
|
||||
baseUrl: "https://psd.example/api",
|
||||
restPath: "/health",
|
||||
restAuth: "none",
|
||||
},
|
||||
});
|
||||
expect((patchBody?.binding as Record<string, unknown>).host).toBeUndefined();
|
||||
});
|
||||
|
||||
test("preserves typed secrets and offers a retry when the configuration save is only partial", async () => {
|
||||
const user = userEvent.setup();
|
||||
let putCalls = 0;
|
||||
let row = makeDatabase();
|
||||
server.use(
|
||||
http.put("/api/catalog/databases/:id/secrets", async ({ request }) => {
|
||||
putCalls += 1;
|
||||
const body = await request.json();
|
||||
expect(body).toEqual({ version: 3, values: { password: "transient-secret" } });
|
||||
if (putCalls === 1) {
|
||||
return HttpResponse.json({
|
||||
code: "catalog_unavailable",
|
||||
message: "Secret store unavailable",
|
||||
}, { status: 503 });
|
||||
}
|
||||
row = makeDatabase({ version: 4, secrets: { ...noSecrets, password: true } });
|
||||
return HttpResponse.json(row);
|
||||
}),
|
||||
);
|
||||
renderPage({ rows: () => [row] });
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||
await user.type(screen.getByLabelText("Password"), "transient-secret");
|
||||
await user.click(screen.getByRole("button", { name: "Save changes" }));
|
||||
|
||||
expect(await screen.findByText("Database configuration saved; secret update could not be confirmed.")).toBeVisible();
|
||||
expect(screen.getByLabelText("Password")).toHaveValue("transient-secret");
|
||||
expect(screen.getByRole("button", { name: "Retry secrets" })).toBeEnabled();
|
||||
|
||||
await user.clear(screen.getByLabelText("Schema"));
|
||||
await user.type(screen.getByLabelText("Schema"), "reporting_after_retry");
|
||||
await user.click(screen.getByRole("button", { name: "Retry secrets" }));
|
||||
|
||||
await waitFor(() => expect(putCalls).toBe(2));
|
||||
expect(screen.queryByText("Database configuration saved; secret update could not be confirmed.")).not.toBeInTheDocument();
|
||||
expect(screen.getByLabelText("Schema")).toHaveValue("reporting_after_retry");
|
||||
expect(screen.getByRole("button", { name: "Save changes" })).toBeEnabled();
|
||||
});
|
||||
|
||||
test("shows secret status without exposing or enabling values when permission is absent", async () => {
|
||||
const user = userEvent.setup();
|
||||
renderPage({
|
||||
rows: [makeDatabase({ secrets: { ...noSecrets, password: true } })],
|
||||
canManageSecrets: false,
|
||||
});
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||
expect(screen.getByLabelText("Password")).toHaveValue("Configured");
|
||||
expect(screen.getByLabelText("Password")).toHaveAttribute("readonly");
|
||||
|
||||
await user.click(screen.getByRole("button", { name: "Back to list" }));
|
||||
await user.click(screen.getByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||
expect(await screen.findByRole("heading", { name: "Edit database" })).toBeVisible();
|
||||
expect(screen.getByLabelText("Password")).toBeDisabled();
|
||||
expect(screen.getByText(/requires the workspace\.secrets\.manage permission/i)).toBeVisible();
|
||||
});
|
||||
|
||||
const patientsTable: CatalogTable = {
|
||||
id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
|
||||
databaseId: "11111111-1111-4111-8111-111111111111",
|
||||
name: "patients",
|
||||
sourceComment: "Patients imported from the clinical source",
|
||||
description: null,
|
||||
generatedDescription: null,
|
||||
version: 1,
|
||||
createdAt: "2026-08-27T08:00:00Z",
|
||||
updatedAt: "2026-08-27T09:00:00Z",
|
||||
};
|
||||
|
||||
test("filters catalog tables as the operator types", async () => {
|
||||
const user = userEvent.setup();
|
||||
const mediciTable: CatalogTable = {
|
||||
...patientsTable,
|
||||
id: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
|
||||
name: "bridge_medici",
|
||||
sourceComment: "Doctors participating in clinical care",
|
||||
};
|
||||
server.use(
|
||||
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([
|
||||
patientsTable,
|
||||
mediciTable,
|
||||
])),
|
||||
);
|
||||
renderPage({
|
||||
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||
});
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||
const tablesRegion = await screen.findByRole("region", { name: "Tables for Policlinico San Donato" });
|
||||
expect(await screen.findByRole("button", { name: "Edit description for patients" })).toBeVisible();
|
||||
expect(screen.getByRole("button", { name: "Edit description for bridge_medici" })).toBeVisible();
|
||||
|
||||
await user.type(screen.getByRole("textbox", { name: "Search tables" }), "medici");
|
||||
|
||||
expect(screen.getByRole("button", { name: "Edit description for bridge_medici" })).toBeVisible();
|
||||
expect(await within(tablesRegion).findByText("1 of 2")).toBeVisible();
|
||||
await waitFor(() => {
|
||||
expect(screen.queryByRole("button", { name: "Edit description for patients" })).not.toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
|
||||
test("selected table exposes a direct Synchronize columns action and sends its id", async () => {
|
||||
const user = userEvent.setup();
|
||||
let startBody: unknown;
|
||||
const run = makeSyncRun("columns", [patientsTable.id]);
|
||||
registerCompletedSyncRun(run);
|
||||
server.use(
|
||||
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
|
||||
http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
|
||||
startBody = await request.json();
|
||||
return HttpResponse.json(run, { status: 202 });
|
||||
}),
|
||||
);
|
||||
renderPage({
|
||||
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||
});
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||
const tableRow = await screen.findByRole("row", { name: /patients/ });
|
||||
await user.click(within(tableRow).getByRole("checkbox"));
|
||||
|
||||
const synchronizeColumns = screen.getByRole("button", { name: "Synchronize columns" });
|
||||
expect(synchronizeColumns).toBeVisible();
|
||||
expect(synchronizeColumns).toBeEnabled();
|
||||
expect(screen.queryByRole("button", { name: "Actions" })).not.toBeInTheDocument();
|
||||
await user.click(synchronizeColumns);
|
||||
|
||||
await waitFor(() => expect(startBody).toEqual({
|
||||
version: 3,
|
||||
scope: "columns",
|
||||
tableIds: [patientsTable.id],
|
||||
}));
|
||||
await waitFor(() => expect(screen.queryByText("1 selected")).not.toBeInTheDocument());
|
||||
});
|
||||
|
||||
test("navigates purely from a database to its tables and edits review metadata", async () => {
|
||||
const user = userEvent.setup();
|
||||
let patchBody: unknown;
|
||||
server.use(
|
||||
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
|
||||
http.patch("/api/catalog/databases/:databaseId/tables/:tableId", async ({ request }) => {
|
||||
patchBody = await request.json();
|
||||
return HttpResponse.json({
|
||||
...patientsTable,
|
||||
description: "Registry used for longitudinal patient analysis",
|
||||
version: 2,
|
||||
});
|
||||
}),
|
||||
);
|
||||
renderPage({
|
||||
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||
});
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||
|
||||
expect(await screen.findByRole("region", { name: "Tables for Policlinico San Donato" })).toBeVisible();
|
||||
expect(screen.getByRole("tab", { name: "Tables" })).toHaveAttribute("aria-selected", "true");
|
||||
expect(screen.queryByLabelText("Database filter")).not.toBeInTheDocument();
|
||||
await user.click(await screen.findByRole("button", { name: "Edit description for patients" }));
|
||||
|
||||
expect(screen.getByLabelText("Physical table name")).toHaveValue("patients");
|
||||
expect(screen.getByLabelText("Physical table name")).toHaveAttribute("readonly");
|
||||
expect(screen.getByLabelText("Source comment")).toHaveAttribute("readonly");
|
||||
await user.type(
|
||||
screen.getByLabelText(/^Description/),
|
||||
"Registry used for longitudinal patient analysis",
|
||||
);
|
||||
await user.click(screen.getByRole("button", { name: "Save metadata" }));
|
||||
|
||||
await waitFor(() => expect(patchBody).toEqual({
|
||||
version: 1,
|
||||
description: "Registry used for longitudinal patient analysis",
|
||||
generatedDescription: null,
|
||||
}));
|
||||
await user.click(screen.getByRole("button", { name: "Back to tables" }));
|
||||
await user.click(screen.getByRole("tab", { name: "Overview" }));
|
||||
expect(await screen.findByRole("heading", { name: "Database details" })).toBeVisible();
|
||||
});
|
||||
|
||||
test("navigates from a table to columns and from the database to physical relationships", async () => {
|
||||
const user = userEvent.setup();
|
||||
let columnPatch: unknown;
|
||||
const idColumn: CatalogColumn = {
|
||||
id: "dddddddd-dddd-4ddd-8ddd-dddddddddddd",
|
||||
tableId: patientsTable.id,
|
||||
name: "id",
|
||||
ordinalPosition: 1,
|
||||
dataType: "bigint",
|
||||
isNullable: false,
|
||||
defaultExpression: null,
|
||||
primaryKeyPosition: 1,
|
||||
isPrimaryKey: true,
|
||||
isForeignKey: true,
|
||||
foreignKeyCount: 1,
|
||||
sourceComment: "Patient identifier",
|
||||
description: null,
|
||||
generatedDescription: null,
|
||||
lastSyncedDatabaseVersion: 3,
|
||||
lastSyncedAt: "2026-08-27T10:00:00Z",
|
||||
version: 1,
|
||||
createdAt: "2026-08-27T10:00:00Z",
|
||||
updatedAt: "2026-08-27T10:00:00Z",
|
||||
};
|
||||
server.use(
|
||||
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
|
||||
http.get("/api/catalog/databases/:databaseId/tables/:tableId/columns", () => HttpResponse.json([idColumn])),
|
||||
http.patch("/api/catalog/databases/:databaseId/tables/:tableId/columns/:columnId", async ({ request }) => {
|
||||
columnPatch = await request.json();
|
||||
return HttpResponse.json({ ...idColumn, generatedDescription: "Generated identifier draft", version: 2 });
|
||||
}),
|
||||
http.get("/api/catalog/databases/:databaseId/relationships", () => HttpResponse.json([{
|
||||
id: "eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee",
|
||||
databaseId: patientsTable.databaseId,
|
||||
constraintName: "visits_patient_id_fkey",
|
||||
sourceTableId: "ffffffff-ffff-4fff-8fff-ffffffffffff",
|
||||
sourceTableName: "visits",
|
||||
targetTableId: patientsTable.id,
|
||||
targetTableName: "patients",
|
||||
updateRule: "NO ACTION",
|
||||
deleteRule: "CASCADE",
|
||||
deferrable: false,
|
||||
initiallyDeferred: false,
|
||||
columns: [{ position: 1, sourceColumnId: "1", sourceColumnName: "patient_id", targetColumnId: idColumn.id, targetColumnName: "id" }],
|
||||
lastSyncedDatabaseVersion: 3,
|
||||
lastSyncedAt: "2026-08-27T10:00:00Z",
|
||||
createdAt: "2026-08-27T10:00:00Z",
|
||||
updatedAt: "2026-08-27T10:00:00Z",
|
||||
}])),
|
||||
);
|
||||
renderPage({ rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })] });
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
|
||||
expect(screen.getByRole("tab", { name: "Columns" })).toHaveAttribute("aria-selected", "true");
|
||||
expect(await screen.findByText("PK")).toBeVisible();
|
||||
expect(screen.getByText("FK")).toBeVisible();
|
||||
await user.click(screen.getByRole("button", { name: "Edit metadata for id" }));
|
||||
await user.type(screen.getByLabelText("Generated description"), "Generated identifier draft");
|
||||
await user.click(screen.getByRole("button", { name: "Save metadata" }));
|
||||
await waitFor(() => expect(columnPatch).toEqual({
|
||||
version: 1,
|
||||
description: null,
|
||||
generatedDescription: "Generated identifier draft",
|
||||
}));
|
||||
await user.click(screen.getByRole("tab", { name: "Relationships" }));
|
||||
expect(await screen.findByText("visits_patient_id_fkey")).toBeVisible();
|
||||
});
|
||||
|
||||
test("opens the durable job drawer and confirms its exact destructive plan", async () => {
|
||||
const user = userEvent.setup();
|
||||
const startBodies: unknown[] = [];
|
||||
const confirmationBodies: unknown[] = [];
|
||||
const queued: CatalogSyncRun = {
|
||||
id: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
|
||||
databaseId: "11111111-1111-4111-8111-111111111111",
|
||||
scope: "tables",
|
||||
tableIds: [],
|
||||
state: "queued",
|
||||
phase: "queued",
|
||||
requestedDatabaseVersion: 3,
|
||||
plannedDiff: null,
|
||||
confirmationToken: null,
|
||||
counts: {},
|
||||
errorCode: null,
|
||||
errorMessage: null,
|
||||
cancelRequested: false,
|
||||
createdAt: "2026-08-27T10:00:00Z",
|
||||
startedAt: null,
|
||||
updatedAt: "2026-08-27T10:00:00Z",
|
||||
finishedAt: null,
|
||||
heartbeatAt: null,
|
||||
};
|
||||
const waiting: CatalogSyncRun = {
|
||||
...queued,
|
||||
state: "awaiting_confirmation",
|
||||
phase: "awaiting_confirmation",
|
||||
confirmationToken: "cccccccc-cccc-4ccc-8ccc-cccccccccccc",
|
||||
plannedDiff: { deletedTables: ["legacy_visits"], deletedColumns: [], deletedRelationships: [] },
|
||||
counts: { tables: 1 },
|
||||
startedAt: "2026-08-27T10:00:00Z",
|
||||
updatedAt: "2026-08-27T10:00:01Z",
|
||||
};
|
||||
server.use(
|
||||
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
|
||||
http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
|
||||
startBodies.push(await request.json());
|
||||
return HttpResponse.json(queued, { status: 202 });
|
||||
}),
|
||||
http.get("/api/catalog/sync-runs/:runId", () => HttpResponse.json(waiting)),
|
||||
http.get("/api/catalog/sync-runs/:runId/events-list", () => HttpResponse.json([])),
|
||||
http.get("/api/catalog/databases/:databaseId/sync-runs", () => HttpResponse.json([waiting])),
|
||||
http.post("/api/catalog/sync-runs/:runId/confirm", async ({ request }) => {
|
||||
confirmationBodies.push(await request.json());
|
||||
return HttpResponse.json({ ...waiting, state: "queued", phase: "queued", confirmationToken: null });
|
||||
}),
|
||||
);
|
||||
renderPage({
|
||||
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||
});
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||
await user.click(await screen.findByRole("button", { name: "Sync tables" }));
|
||||
|
||||
const synchronizationDrawer = await screen.findByRole("complementary", {
|
||||
name: "Schema synchronization",
|
||||
});
|
||||
expect(synchronizationDrawer).toHaveClass("inset-y-2", "sm:inset-y-4");
|
||||
expect(within(synchronizationDrawer).getByRole("listitem", {
|
||||
name: "Confirmation required, in progress",
|
||||
})).toHaveAttribute("aria-current", "step");
|
||||
expect(await screen.findByText("table · legacy_visits")).toBeVisible();
|
||||
expect(startBodies).toEqual([{ version: 3, scope: "tables", tableIds: [] }]);
|
||||
await user.click(screen.getByRole("button", { name: "Confirm removals" }));
|
||||
await waitFor(() => expect(confirmationBodies).toEqual([
|
||||
{ confirmationToken: "cccccccc-cccc-4ccc-8ccc-cccccccccccc" },
|
||||
]));
|
||||
});
|
||||
|
||||
test("keeps a stale table draft but requires an explicit reload before another save", async () => {
|
||||
const user = userEvent.setup();
|
||||
const current = {
|
||||
...patientsTable,
|
||||
description: "Description saved by another editor",
|
||||
version: 2,
|
||||
};
|
||||
let latest = patientsTable;
|
||||
server.use(
|
||||
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([latest])),
|
||||
http.patch("/api/catalog/databases/:databaseId/tables/:tableId", () => {
|
||||
latest = current;
|
||||
return HttpResponse.json({
|
||||
code: "table_stale",
|
||||
message: "Table description changed. Reload and try again.",
|
||||
}, { status: 409 });
|
||||
}),
|
||||
);
|
||||
renderPage({
|
||||
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||
});
|
||||
|
||||
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||
await user.click(await screen.findByRole("button", { name: "Edit description for patients" }));
|
||||
const description = screen.getByLabelText(/^Description/);
|
||||
await user.type(description, "My unsaved draft");
|
||||
await user.click(screen.getByRole("button", { name: "Save metadata" }));
|
||||
|
||||
expect(await screen.findByText("A newer table description is available.")).toBeVisible();
|
||||
expect(description).toHaveValue("My unsaved draft");
|
||||
expect(screen.getByRole("button", { name: "Save metadata" })).toBeDisabled();
|
||||
await user.click(screen.getByRole("button", { name: "Keep editing" }));
|
||||
expect(screen.getByText("Reload the latest value before this description can be saved.")).toBeVisible();
|
||||
await user.click(screen.getByRole("button", { name: "Reload latest" }));
|
||||
|
||||
await waitFor(() => expect(description).toHaveValue("Description saved by another editor"));
|
||||
expect(screen.getByRole("button", { name: "Save metadata" })).toBeDisabled();
|
||||
});
|
||||
@@ -1,8 +1,756 @@
|
||||
export function DatabaseManagementPage() {
|
||||
import {
|
||||
useCallback,
|
||||
useEffect,
|
||||
useMemo,
|
||||
useRef,
|
||||
useState,
|
||||
} from "react";
|
||||
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { Plus, RefreshCw } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
import { Button } from "../components/ui/button";
|
||||
import { ApiError, apiErrorMessage } from "../api/client";
|
||||
import {
|
||||
createCatalogDatabase,
|
||||
deleteCatalogDatabase,
|
||||
listCatalogDatabases,
|
||||
replaceCatalogDatabaseSecrets,
|
||||
startCatalogSync,
|
||||
testCatalogDatabase,
|
||||
updateCatalogDatabase,
|
||||
type CatalogDatabase,
|
||||
type CatalogSecretName,
|
||||
type CatalogSyncScope,
|
||||
type CatalogSyncRun,
|
||||
type DatabaseBinding,
|
||||
type DatabaseTransport,
|
||||
} from "../api/catalog-databases";
|
||||
import { DatabaseGrid } from "./database-management/DatabaseGrid";
|
||||
import { DatabaseForm } from "./database-management/DatabaseForm";
|
||||
import { DatabaseTables } from "./database-management/DatabaseTables";
|
||||
import { DatabaseRelationships } from "./database-management/DatabaseRelationships";
|
||||
import { CatalogSyncDrawer } from "./database-management/CatalogSyncDrawer";
|
||||
import {
|
||||
configurationFingerprint,
|
||||
configurationFromDraft,
|
||||
draftFrom,
|
||||
hasSecretChanges,
|
||||
secretReplacements,
|
||||
type DatabaseBusyAction,
|
||||
type DatabaseFormDraft,
|
||||
type DatabaseFormMode,
|
||||
type DatabaseNavigationState,
|
||||
type DatabaseScreen,
|
||||
} from "./database-management/model";
|
||||
|
||||
const DATABASE_QUERY_KEY = ["catalog-databases"] as const;
|
||||
const SYNC_STARTED_MESSAGES: Record<CatalogSyncScope, string> = {
|
||||
tables: "Table synchronization started",
|
||||
columns: "Column synchronization started",
|
||||
relationships: "Relationship synchronization started",
|
||||
all: "Full schema synchronization started",
|
||||
};
|
||||
|
||||
interface Props {
|
||||
canManage: boolean;
|
||||
canManageSecrets: boolean;
|
||||
onNavigationStateChange?: (state: DatabaseNavigationState) => void;
|
||||
}
|
||||
|
||||
interface FormSource {
|
||||
id?: string;
|
||||
version: number;
|
||||
}
|
||||
|
||||
function isStaleError(error: unknown): boolean {
|
||||
return error instanceof ApiError && error.code === "database_stale";
|
||||
}
|
||||
|
||||
export function DatabaseManagementPage({
|
||||
canManage,
|
||||
canManageSecrets,
|
||||
onNavigationStateChange,
|
||||
}: Props) {
|
||||
const queryClient = useQueryClient();
|
||||
const {
|
||||
data,
|
||||
isLoading,
|
||||
isError,
|
||||
isFetching,
|
||||
refetch,
|
||||
} = useQuery({
|
||||
queryKey: DATABASE_QUERY_KEY,
|
||||
queryFn: listCatalogDatabases,
|
||||
retry: false,
|
||||
});
|
||||
const rows = data ?? [];
|
||||
|
||||
const [screen, setScreen] = useState<DatabaseScreen>({ kind: "list" });
|
||||
const [draft, setDraft] = useState<DatabaseFormDraft | null>(null);
|
||||
const [baseline, setBaseline] = useState("");
|
||||
const [formSource, setFormSource] = useState<FormSource | null>(null);
|
||||
const [search, setSearch] = useState("");
|
||||
const [busyAction, setBusyAction] = useState<DatabaseBusyAction>(null);
|
||||
const [stale, setStale] = useState(false);
|
||||
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
|
||||
const [partialSecretFailure, setPartialSecretFailure] = useState<string | null>(null);
|
||||
const [tablesNavigationState, setTablesNavigationState] = useState<DatabaseNavigationState>({
|
||||
dirty: false,
|
||||
busy: false,
|
||||
});
|
||||
const [activeSyncRun, setActiveSyncRun] = useState<CatalogSyncRun | null>(null);
|
||||
const [syncDrawerOpen, setSyncDrawerOpen] = useState(false);
|
||||
|
||||
const originRef = useRef<HTMLElement | null>(null);
|
||||
const searchInputRef = useRef<HTMLInputElement>(null);
|
||||
const formHeadingRef = useRef<HTMLHeadingElement>(null);
|
||||
|
||||
const activeRow = screen.kind === "list"
|
||||
? undefined
|
||||
: rows.find((row) => row.workspaceId === screen.workspaceId);
|
||||
const availableWorkspaces = useMemo(
|
||||
() => rows.filter((row) => row.workspaceAvailable && !row.configured),
|
||||
[rows],
|
||||
);
|
||||
const editable = screen.kind === "add" || screen.kind === "edit";
|
||||
const configurationDirty = Boolean(
|
||||
editable
|
||||
&& draft
|
||||
&& configurationFingerprint(draft) !== baseline,
|
||||
);
|
||||
const dirty = Boolean(editable && draft && (configurationDirty || hasSecretChanges(draft)));
|
||||
const busy = busyAction !== null;
|
||||
const navigationDirty = screen.kind === "tables" ? tablesNavigationState.dirty : dirty;
|
||||
const navigationBusy = screen.kind === "tables" ? tablesNavigationState.busy : busy;
|
||||
|
||||
useEffect(() => {
|
||||
onNavigationStateChange?.({ dirty: navigationDirty, busy: navigationBusy });
|
||||
}, [navigationBusy, navigationDirty, onNavigationStateChange]);
|
||||
|
||||
useEffect(() => () => {
|
||||
onNavigationStateChange?.({ dirty: false, busy: false });
|
||||
}, [onNavigationStateChange]);
|
||||
|
||||
useEffect(() => {
|
||||
const warn = (event: BeforeUnloadEvent) => {
|
||||
if (!navigationDirty && !navigationBusy) return;
|
||||
event.preventDefault();
|
||||
};
|
||||
window.addEventListener("beforeunload", warn);
|
||||
return () => window.removeEventListener("beforeunload", warn);
|
||||
}, [navigationBusy, navigationDirty]);
|
||||
|
||||
useEffect(() => {
|
||||
if (screen.kind === "list" || !activeRow || !formSource || busy) return;
|
||||
if (activeRow.id === formSource.id && activeRow.version === formSource.version) return;
|
||||
setStale(true);
|
||||
setStaleBannerOpen(true);
|
||||
}, [activeRow, busy, formSource, screen.kind]);
|
||||
|
||||
useEffect(() => {
|
||||
if (screen.kind === "list") return;
|
||||
const timer = window.setTimeout(() => formHeadingRef.current?.focus(), 0);
|
||||
return () => window.clearTimeout(timer);
|
||||
}, [screen.kind, screen.kind === "list" ? "" : screen.workspaceId]);
|
||||
|
||||
const cacheSavedRow = useCallback((saved: CatalogDatabase) => {
|
||||
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => {
|
||||
const existing = current.findIndex((row) => row.workspaceId === saved.workspaceId);
|
||||
if (existing < 0) return [...current, saved];
|
||||
return current.map((row, index) => index === existing ? saved : row);
|
||||
});
|
||||
}, [queryClient]);
|
||||
|
||||
const restoreListFocus = useCallback(() => {
|
||||
window.setTimeout(() => {
|
||||
if (originRef.current?.isConnected) {
|
||||
originRef.current.focus();
|
||||
} else {
|
||||
searchInputRef.current?.focus();
|
||||
}
|
||||
}, 0);
|
||||
}, []);
|
||||
|
||||
const showList = useCallback(() => {
|
||||
setScreen({ kind: "list" });
|
||||
setDraft(null);
|
||||
setBaseline("");
|
||||
setFormSource(null);
|
||||
setStale(false);
|
||||
setStaleBannerOpen(true);
|
||||
setPartialSecretFailure(null);
|
||||
setTablesNavigationState({ dirty: false, busy: false });
|
||||
restoreListFocus();
|
||||
}, [restoreListFocus]);
|
||||
|
||||
useEffect(() => {
|
||||
if (screen.kind === "list" || isLoading || activeRow) return;
|
||||
showList();
|
||||
toast.info("This database configuration is no longer available");
|
||||
}, [activeRow, isLoading, screen.kind, showList]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!["tables", "relationships"].includes(screen.kind) || !activeRow || (activeRow.configured && activeRow.id)) return;
|
||||
showList();
|
||||
toast.info("Save the database configuration before managing tables");
|
||||
}, [activeRow, screen.kind, showList]);
|
||||
|
||||
const backToList = useCallback(() => {
|
||||
if (busy) return;
|
||||
if (dirty && !window.confirm("Discard unsaved database changes?")) return;
|
||||
showList();
|
||||
}, [busy, dirty, showList]);
|
||||
|
||||
const openForm = useCallback((
|
||||
mode: DatabaseFormMode,
|
||||
row: CatalogDatabase,
|
||||
origin: HTMLElement,
|
||||
workspaceLocked = false,
|
||||
) => {
|
||||
const nextDraft = draftFrom(row);
|
||||
originRef.current = origin;
|
||||
setDraft(nextDraft);
|
||||
setBaseline(configurationFingerprint(nextDraft));
|
||||
setFormSource({ id: row.id, version: row.version });
|
||||
setStale(false);
|
||||
setStaleBannerOpen(true);
|
||||
setPartialSecretFailure(null);
|
||||
setScreen({
|
||||
kind: mode,
|
||||
workspaceId: row.workspaceId,
|
||||
...(mode === "add" ? { workspaceLocked } : {}),
|
||||
});
|
||||
}, []);
|
||||
|
||||
const viewRow = useCallback((row: CatalogDatabase, origin: HTMLButtonElement) => {
|
||||
openForm("view", row, origin);
|
||||
}, [openForm]);
|
||||
|
||||
const editRow = useCallback((row: CatalogDatabase, origin: HTMLButtonElement) => {
|
||||
openForm(row.configured ? "edit" : "add", row, origin, !row.configured);
|
||||
}, [openForm]);
|
||||
|
||||
const deleteRow = useCallback((row: CatalogDatabase, origin: HTMLButtonElement) => {
|
||||
openForm("delete", row, origin);
|
||||
}, [openForm]);
|
||||
|
||||
const openTables = useCallback((row: CatalogDatabase, origin?: HTMLElement) => {
|
||||
if (!row.configured || !row.id) return;
|
||||
if (origin) originRef.current = origin;
|
||||
setDraft(null);
|
||||
setBaseline("");
|
||||
setFormSource(null);
|
||||
setStale(false);
|
||||
setPartialSecretFailure(null);
|
||||
setTablesNavigationState({ dirty: false, busy: false });
|
||||
setScreen({ kind: "tables", workspaceId: row.workspaceId });
|
||||
}, []);
|
||||
|
||||
const openRelationships = useCallback((row: CatalogDatabase) => {
|
||||
if (!row.configured || !row.id) return;
|
||||
setDraft(null);
|
||||
setBaseline("");
|
||||
setFormSource(null);
|
||||
setStale(false);
|
||||
setPartialSecretFailure(null);
|
||||
setTablesNavigationState({ dirty: false, busy: false });
|
||||
setScreen({ kind: "relationships", workspaceId: row.workspaceId });
|
||||
}, []);
|
||||
|
||||
const openOverview = useCallback((row: CatalogDatabase) => {
|
||||
const nextDraft = draftFrom(row);
|
||||
setDraft(nextDraft);
|
||||
setBaseline(configurationFingerprint(nextDraft));
|
||||
setFormSource({ id: row.id, version: row.version });
|
||||
setStale(false);
|
||||
setStaleBannerOpen(true);
|
||||
setPartialSecretFailure(null);
|
||||
setTablesNavigationState({ dirty: false, busy: false });
|
||||
setScreen({ kind: "view", workspaceId: row.workspaceId });
|
||||
}, []);
|
||||
|
||||
const addDatabase = useCallback((origin: HTMLElement) => {
|
||||
const workspace = availableWorkspaces[0];
|
||||
if (!workspace || !canManage) return;
|
||||
openForm("add", workspace, origin, false);
|
||||
}, [availableWorkspaces, canManage, openForm]);
|
||||
|
||||
const changeWorkspace = useCallback((workspaceId: string) => {
|
||||
const workspace = availableWorkspaces.find((row) => row.workspaceId === workspaceId);
|
||||
if (!workspace) return;
|
||||
if (dirty && !window.confirm("Discard changes and choose another workspace?")) return;
|
||||
const nextDraft = draftFrom(workspace);
|
||||
setDraft(nextDraft);
|
||||
setBaseline(configurationFingerprint(nextDraft));
|
||||
setFormSource({ id: workspace.id, version: workspace.version });
|
||||
setStale(false);
|
||||
setPartialSecretFailure(null);
|
||||
setScreen({ kind: "add", workspaceId, workspaceLocked: false });
|
||||
}, [availableWorkspaces, dirty]);
|
||||
|
||||
const changeField = useCallback((field: "databaseName" | "schema", value: string) => {
|
||||
setDraft((current) => current ? { ...current, [field]: value } : current);
|
||||
}, []);
|
||||
|
||||
const changeTransport = useCallback((transport: DatabaseTransport) => {
|
||||
setDraft((current) => current
|
||||
? { ...current, binding: { ...current.binding, transport } }
|
||||
: current);
|
||||
}, []);
|
||||
|
||||
const changeBinding = useCallback(<K extends keyof DatabaseBinding>(
|
||||
key: K,
|
||||
value: DatabaseBinding[K],
|
||||
) => {
|
||||
setDraft((current) => current
|
||||
? { ...current, binding: { ...current.binding, [key]: value } }
|
||||
: current);
|
||||
}, []);
|
||||
|
||||
const changeSecret = useCallback((name: CatalogSecretName, value: string) => {
|
||||
setDraft((current) => current
|
||||
? { ...current, secrets: { ...current.secrets, [name]: value } }
|
||||
: current);
|
||||
}, []);
|
||||
|
||||
const markStale = useCallback(() => {
|
||||
setStale(true);
|
||||
setStaleBannerOpen(true);
|
||||
}, []);
|
||||
|
||||
const save = useCallback(async () => {
|
||||
if (!activeRow || !formSource || !draft || !editable || !canManage || stale || busy) return;
|
||||
setBusyAction("save");
|
||||
setPartialSecretFailure(null);
|
||||
|
||||
try {
|
||||
const input = configurationFromDraft(draft);
|
||||
let saved = activeRow;
|
||||
const shouldPersistConfiguration = !formSource.id || configurationDirty;
|
||||
|
||||
if (shouldPersistConfiguration) {
|
||||
saved = formSource.id
|
||||
? await updateCatalogDatabase(formSource.id, formSource.version, input)
|
||||
: await createCatalogDatabase(input);
|
||||
setFormSource({ id: saved.id, version: saved.version });
|
||||
cacheSavedRow(saved);
|
||||
}
|
||||
|
||||
const replacements = secretReplacements(draft);
|
||||
if (Object.keys(replacements).length > 0) {
|
||||
if (!canManageSecrets) throw new Error("Secret replacement is not permitted");
|
||||
try {
|
||||
saved = await replaceCatalogDatabaseSecrets(
|
||||
saved.id ?? formSource.id!,
|
||||
shouldPersistConfiguration ? saved.version : formSource.version,
|
||||
replacements,
|
||||
);
|
||||
setFormSource({ id: saved.id, version: saved.version });
|
||||
cacheSavedRow(saved);
|
||||
} catch (error) {
|
||||
setBaseline(configurationFingerprint(draft));
|
||||
setScreen({ kind: "edit", workspaceId: saved.workspaceId });
|
||||
setPartialSecretFailure(apiErrorMessage(error));
|
||||
if (isStaleError(error)) {
|
||||
markStale();
|
||||
} else {
|
||||
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||
}
|
||||
toast.warning("Database configuration saved, but secrets still need attention");
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const nextDraft = draftFrom(saved);
|
||||
setDraft(nextDraft);
|
||||
setBaseline(configurationFingerprint(nextDraft));
|
||||
setFormSource({ id: saved.id, version: saved.version });
|
||||
setScreen({ kind: "edit", workspaceId: saved.workspaceId });
|
||||
setStale(false);
|
||||
setPartialSecretFailure(null);
|
||||
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||
toast.success(activeRow.configured ? "Database configuration saved" : "Database configuration added");
|
||||
} catch (error) {
|
||||
if (isStaleError(error)) {
|
||||
markStale();
|
||||
} else {
|
||||
toast.error(apiErrorMessage(error));
|
||||
}
|
||||
} finally {
|
||||
setBusyAction(null);
|
||||
}
|
||||
}, [
|
||||
activeRow,
|
||||
busy,
|
||||
cacheSavedRow,
|
||||
canManage,
|
||||
canManageSecrets,
|
||||
configurationDirty,
|
||||
draft,
|
||||
editable,
|
||||
formSource,
|
||||
markStale,
|
||||
queryClient,
|
||||
stale,
|
||||
]);
|
||||
|
||||
const retrySecrets = useCallback(async () => {
|
||||
if (!activeRow?.configured || !formSource?.id || !draft || !canManageSecrets || busy || stale) return;
|
||||
const replacements = secretReplacements(draft);
|
||||
if (Object.keys(replacements).length === 0) {
|
||||
setPartialSecretFailure(null);
|
||||
return;
|
||||
}
|
||||
|
||||
setBusyAction("retry-secrets");
|
||||
try {
|
||||
const saved = await replaceCatalogDatabaseSecrets(formSource.id, formSource.version, replacements);
|
||||
setFormSource({ id: saved.id, version: saved.version });
|
||||
cacheSavedRow(saved);
|
||||
setDraft({ ...draft, secrets: {} });
|
||||
setPartialSecretFailure(null);
|
||||
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||
toast.success("Database secrets saved");
|
||||
} catch (error) {
|
||||
setPartialSecretFailure(apiErrorMessage(error));
|
||||
if (isStaleError(error)) markStale();
|
||||
toast.error(apiErrorMessage(error));
|
||||
} finally {
|
||||
setBusyAction(null);
|
||||
}
|
||||
}, [activeRow, busy, cacheSavedRow, canManageSecrets, draft, formSource, markStale, queryClient, stale]);
|
||||
|
||||
const testConnection = useCallback(async () => {
|
||||
if (
|
||||
!activeRow?.configured
|
||||
|| !formSource?.id
|
||||
|| !draft
|
||||
|| !canManage
|
||||
|| dirty
|
||||
|| stale
|
||||
|| busy
|
||||
) return;
|
||||
|
||||
setBusyAction("test");
|
||||
try {
|
||||
const tested = await testCatalogDatabase(formSource.id, formSource.version);
|
||||
setFormSource({ id: tested.id, version: tested.version });
|
||||
cacheSavedRow(tested);
|
||||
const nextDraft = draftFrom(tested);
|
||||
setDraft(nextDraft);
|
||||
setBaseline(configurationFingerprint(nextDraft));
|
||||
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||
if (tested.connectionStatus === "reachable") {
|
||||
toast.success("Database connection is reachable");
|
||||
} else {
|
||||
toast.error(tested.lastErrorMessage ?? "Database connection failed");
|
||||
}
|
||||
} catch (error) {
|
||||
if (isStaleError(error)) markStale();
|
||||
else toast.error(apiErrorMessage(error));
|
||||
} finally {
|
||||
setBusyAction(null);
|
||||
}
|
||||
}, [activeRow, busy, cacheSavedRow, canManage, dirty, draft, formSource, markStale, queryClient, stale]);
|
||||
|
||||
const remove = useCallback(async () => {
|
||||
if (!activeRow?.configured || !formSource?.id || !canManage || busy || stale) return;
|
||||
setBusyAction("delete");
|
||||
try {
|
||||
await deleteCatalogDatabase(formSource.id, formSource.version);
|
||||
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => {
|
||||
if (!activeRow.workspaceAvailable) {
|
||||
return current.filter((row) => row.workspaceId !== activeRow.workspaceId);
|
||||
}
|
||||
return current.map((row) => row.workspaceId === activeRow.workspaceId
|
||||
? {
|
||||
...row,
|
||||
id: undefined,
|
||||
configured: false,
|
||||
version: 0,
|
||||
createdAt: "",
|
||||
updatedAt: "",
|
||||
connectionStatus: "untested",
|
||||
testedVersion: undefined,
|
||||
lastTestedAt: undefined,
|
||||
lastErrorCode: undefined,
|
||||
lastErrorMessage: undefined,
|
||||
secrets: {
|
||||
password: false,
|
||||
apiKey: false,
|
||||
sshPrivateKey: false,
|
||||
sshPrivateKeyPassphrase: false,
|
||||
sshKnownHosts: false,
|
||||
tlsCa: false,
|
||||
},
|
||||
}
|
||||
: row);
|
||||
});
|
||||
setBusyAction(null);
|
||||
showList();
|
||||
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||
toast.success("Database configuration deleted");
|
||||
} catch (error) {
|
||||
if (isStaleError(error)) markStale();
|
||||
else toast.error(apiErrorMessage(error));
|
||||
} finally {
|
||||
setBusyAction(null);
|
||||
}
|
||||
}, [activeRow, busy, canManage, formSource, markStale, queryClient, showList, stale]);
|
||||
|
||||
const reloadLatest = useCallback(async () => {
|
||||
if (busy || screen.kind === "list") return;
|
||||
setBusyAction("reload");
|
||||
try {
|
||||
const result = await refetch();
|
||||
if (result.error) throw result.error;
|
||||
const latest = result.data?.find((row) => row.workspaceId === screen.workspaceId);
|
||||
if (!latest) {
|
||||
showList();
|
||||
toast.info("This database configuration is no longer available");
|
||||
return;
|
||||
}
|
||||
const nextDraft = draftFrom(latest);
|
||||
setDraft(nextDraft);
|
||||
setBaseline(configurationFingerprint(nextDraft));
|
||||
setFormSource({ id: latest.id, version: latest.version });
|
||||
setStale(false);
|
||||
setStaleBannerOpen(true);
|
||||
setPartialSecretFailure(null);
|
||||
if (!latest.configured && screen.kind === "delete") {
|
||||
showList();
|
||||
toast.info("This database configuration has already been deleted");
|
||||
} else if (screen.kind === "edit" && !latest.configured) {
|
||||
setScreen({ kind: "add", workspaceId: latest.workspaceId, workspaceLocked: true });
|
||||
}
|
||||
} catch (error) {
|
||||
toast.error(apiErrorMessage(error));
|
||||
} finally {
|
||||
setBusyAction(null);
|
||||
}
|
||||
}, [busy, refetch, screen, showList]);
|
||||
|
||||
const refreshList = useCallback(async () => {
|
||||
if (isFetching) return;
|
||||
try {
|
||||
const result = await refetch();
|
||||
if (result.error) throw result.error;
|
||||
} catch (error) {
|
||||
toast.error(apiErrorMessage(error));
|
||||
}
|
||||
}, [isFetching, refetch]);
|
||||
|
||||
const updateTrackedSyncRun = useCallback((run: CatalogSyncRun) => {
|
||||
setActiveSyncRun(run);
|
||||
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => current.map((row) => (
|
||||
row.id === run.databaseId
|
||||
? { ...row, activeSyncRun: ["queued", "running", "awaiting_confirmation", "applying"].includes(run.state) ? run : undefined }
|
||||
: row
|
||||
)));
|
||||
}, [queryClient]);
|
||||
|
||||
const rememberSyncRun = useCallback((run: CatalogSyncRun) => {
|
||||
updateTrackedSyncRun(run);
|
||||
setSyncDrawerOpen(true);
|
||||
}, [updateTrackedSyncRun]);
|
||||
|
||||
const openSync = useCallback((row?: CatalogDatabase) => {
|
||||
const run = row?.activeSyncRun ?? activeSyncRun;
|
||||
if (!run) return;
|
||||
setActiveSyncRun(run);
|
||||
setSyncDrawerOpen(true);
|
||||
}, [activeSyncRun]);
|
||||
|
||||
const testSelected = useCallback(async (selected: CatalogDatabase[]) => {
|
||||
try {
|
||||
const tested = await Promise.all(selected.map((row) => testCatalogDatabase(row.id!, row.version)));
|
||||
for (const row of tested) cacheSavedRow(row);
|
||||
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||
toast.success(`${tested.length} connection${tested.length === 1 ? "" : "s"} tested`);
|
||||
} catch (error) {
|
||||
toast.error(apiErrorMessage(error));
|
||||
throw error;
|
||||
}
|
||||
}, [cacheSavedRow, queryClient]);
|
||||
|
||||
const syncSelected = useCallback(async (selected: CatalogDatabase[], scope: CatalogSyncScope) => {
|
||||
try {
|
||||
const runs = await Promise.all(selected.map((row) => startCatalogSync(row.id!, row.version, scope)));
|
||||
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => current.map((row) => {
|
||||
const run = runs.find((candidate) => candidate.databaseId === row.id);
|
||||
return run ? { ...row, activeSyncRun: run } : row;
|
||||
}));
|
||||
if (runs[0]) rememberSyncRun(runs[0]);
|
||||
toast.success(`${runs.length} schema synchronization${runs.length === 1 ? "" : "s"} started`);
|
||||
} catch (error) {
|
||||
toast.error(apiErrorMessage(error));
|
||||
throw error;
|
||||
}
|
||||
}, [queryClient, rememberSyncRun]);
|
||||
|
||||
const syncDatabase = useCallback(async (scope: CatalogSyncScope) => {
|
||||
if (!activeRow?.id || !activeRow.configured) return;
|
||||
try {
|
||||
rememberSyncRun(await startCatalogSync(activeRow.id, activeRow.version, scope));
|
||||
toast.success(SYNC_STARTED_MESSAGES[scope]);
|
||||
} catch (error) { toast.error(apiErrorMessage(error)); }
|
||||
}, [activeRow, rememberSyncRun]);
|
||||
|
||||
const catalogChanged = useCallback(async () => {
|
||||
const databaseId = activeSyncRun?.databaseId;
|
||||
if (databaseId) {
|
||||
await Promise.all([
|
||||
queryClient.invalidateQueries({ queryKey: ["catalog-tables", databaseId] }),
|
||||
queryClient.invalidateQueries({ queryKey: ["catalog-relationships", databaseId] }),
|
||||
queryClient.invalidateQueries({ queryKey: ["catalog-sync-runs", databaseId] }),
|
||||
]);
|
||||
}
|
||||
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||
}, [activeSyncRun, queryClient]);
|
||||
|
||||
const formVisible = screen.kind !== "list" && screen.kind !== "tables" && screen.kind !== "relationships" && activeRow && draft;
|
||||
const tablesVisible = screen.kind === "tables" && activeRow?.configured && activeRow.id;
|
||||
const relationshipsVisible = screen.kind === "relationships" && activeRow?.configured && activeRow.id;
|
||||
const currentActiveRun = activeSyncRun && ["queued", "running", "awaiting_confirmation", "applying"].includes(activeSyncRun.state)
|
||||
? activeSyncRun
|
||||
: activeRow?.activeSyncRun && ["queued", "running", "awaiting_confirmation", "applying"].includes(activeRow.activeSyncRun.state)
|
||||
? activeRow.activeSyncRun
|
||||
: undefined;
|
||||
|
||||
return (
|
||||
<main
|
||||
aria-label="Database management"
|
||||
className="flex-1 overflow-y-auto bg-background"
|
||||
/>
|
||||
<main aria-label="Database management" className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
|
||||
<header className="flex flex-wrap items-center justify-between gap-4 border-b border-border bg-card px-4 py-4 sm:px-5">
|
||||
<div>
|
||||
<p className="thot-label mb-1">Administration</p>
|
||||
<h1 className="font-heading text-xl font-semibold tracking-tight">Database management</h1>
|
||||
<p className="mt-1 text-sm text-muted-foreground">
|
||||
One database configuration for each repository workspace.
|
||||
</p>
|
||||
</div>
|
||||
{screen.kind === "list" ? (
|
||||
<div className="flex items-center gap-2">
|
||||
<Button type="button" variant="outline" disabled={isFetching} onClick={() => void refreshList()}>
|
||||
<RefreshCw className={isFetching ? "animate-spin" : ""} /> Refresh
|
||||
</Button>
|
||||
<Button
|
||||
type="button"
|
||||
disabled={!canManage || availableWorkspaces.length === 0}
|
||||
title={availableWorkspaces.length === 0 ? "Every available workspace is already configured" : undefined}
|
||||
onClick={(event) => addDatabase(event.currentTarget)}
|
||||
>
|
||||
<Plus /> Add database
|
||||
</Button>
|
||||
</div>
|
||||
) : null}
|
||||
</header>
|
||||
|
||||
<div className="relative flex min-h-0 flex-1">
|
||||
<div
|
||||
aria-hidden={screen.kind !== "list"}
|
||||
className={screen.kind === "list"
|
||||
? "flex min-h-0 flex-1"
|
||||
: "pointer-events-none invisible absolute inset-0 flex min-h-0"
|
||||
}
|
||||
>
|
||||
{isError && rows.length === 0 ? (
|
||||
<div className="grid flex-1 place-items-center p-6">
|
||||
<div className="max-w-lg rounded-md border border-destructive/30 bg-destructive/8 p-5 text-sm" role="alert">
|
||||
<p className="font-semibold text-destructive">The database catalog is unavailable.</p>
|
||||
<p className="mt-1 leading-5 text-muted-foreground">Verify the catalog service and database migrations, then try again.</p>
|
||||
<Button type="button" variant="outline" className="mt-4" disabled={isFetching} onClick={() => void refreshList()}>
|
||||
<RefreshCw /> Try again
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<DatabaseGrid
|
||||
rows={rows}
|
||||
isLoading={isLoading}
|
||||
isFetching={isFetching}
|
||||
search={search}
|
||||
canManage={canManage}
|
||||
searchInputRef={searchInputRef}
|
||||
onSearchChange={setSearch}
|
||||
onView={viewRow}
|
||||
onEdit={editRow}
|
||||
onDelete={deleteRow}
|
||||
onOpenSync={openSync}
|
||||
onTestSelected={testSelected}
|
||||
onSyncSelected={syncSelected}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
|
||||
{formVisible ? (
|
||||
<DatabaseForm
|
||||
mode={screen.kind as DatabaseFormMode}
|
||||
row={activeRow}
|
||||
availableWorkspaces={availableWorkspaces}
|
||||
workspaceLocked={screen.kind === "add" && Boolean(screen.workspaceLocked)}
|
||||
draft={draft}
|
||||
dirty={dirty}
|
||||
canManage={canManage}
|
||||
canManageSecrets={canManageSecrets}
|
||||
busyAction={busyAction}
|
||||
stale={stale}
|
||||
staleBannerOpen={staleBannerOpen}
|
||||
partialSecretFailure={partialSecretFailure}
|
||||
headingRef={formHeadingRef}
|
||||
onBack={backToList}
|
||||
onWorkspaceChange={changeWorkspace}
|
||||
onFieldChange={changeField}
|
||||
onTransportChange={changeTransport}
|
||||
onBindingChange={changeBinding}
|
||||
onSecretChange={changeSecret}
|
||||
onSave={() => void save()}
|
||||
onTest={() => void testConnection()}
|
||||
onDelete={() => void remove()}
|
||||
onReloadLatest={() => void reloadLatest()}
|
||||
onKeepEditing={() => setStaleBannerOpen(false)}
|
||||
onRetrySecrets={() => void retrySecrets()}
|
||||
onOpenTables={(origin) => openTables(activeRow, origin)}
|
||||
onOpenRelationships={() => openRelationships(activeRow)}
|
||||
onSync={(scope) => void syncDatabase(scope)}
|
||||
onOpenSync={() => openSync(activeRow)}
|
||||
activeSyncRun={currentActiveRun}
|
||||
/>
|
||||
) : null}
|
||||
|
||||
{tablesVisible ? (
|
||||
<DatabaseTables
|
||||
database={activeRow}
|
||||
canManage={canManage}
|
||||
activeRun={currentActiveRun}
|
||||
onBackToDatabases={showList}
|
||||
onOpenOverview={() => openOverview(activeRow)}
|
||||
onOpenRelationships={() => openRelationships(activeRow)}
|
||||
onNavigationStateChange={setTablesNavigationState}
|
||||
onRunStarted={rememberSyncRun}
|
||||
onOpenSync={() => openSync(activeRow)}
|
||||
/>
|
||||
) : null}
|
||||
|
||||
{relationshipsVisible ? (
|
||||
<DatabaseRelationships
|
||||
database={{ ...activeRow, activeSyncRun: currentActiveRun }}
|
||||
canManage={canManage}
|
||||
onBackToDatabases={showList}
|
||||
onOpenOverview={() => openOverview(activeRow)}
|
||||
onOpenTables={() => openTables(activeRow)}
|
||||
onRunStarted={rememberSyncRun}
|
||||
onOpenSync={() => openSync(activeRow)}
|
||||
/>
|
||||
) : null}
|
||||
</div>
|
||||
<CatalogSyncDrawer
|
||||
databaseId={activeSyncRun?.databaseId ?? activeRow?.id ?? null}
|
||||
runId={activeSyncRun?.id ?? null}
|
||||
open={syncDrawerOpen && Boolean(activeSyncRun)}
|
||||
onClose={() => setSyncDrawerOpen(false)}
|
||||
onRunChange={rememberSyncRun}
|
||||
onRunUpdate={updateTrackedSyncRun}
|
||||
onCatalogChanged={() => void catalogChanged()}
|
||||
/>
|
||||
</main>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,402 @@
|
||||
import { useEffect, useMemo, useRef, useState } from "react";
|
||||
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { AlertTriangle, Check, CheckCircle2, Circle, LoaderCircle, Minus, RotateCcw, X } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
import { Button } from "../../components/ui/button";
|
||||
import { apiErrorMessage } from "../../api/client";
|
||||
import {
|
||||
cancelCatalogSync,
|
||||
catalogSyncEventsUrl,
|
||||
confirmCatalogSync,
|
||||
getCatalogSyncRun,
|
||||
listCatalogSyncEvents,
|
||||
listCatalogSyncRuns,
|
||||
retryCatalogSync,
|
||||
type CatalogSyncEvent,
|
||||
type CatalogSyncPhase,
|
||||
type CatalogSyncRun,
|
||||
} from "../../api/catalog-databases";
|
||||
|
||||
interface Props {
|
||||
databaseId: string | null;
|
||||
runId: string | null;
|
||||
open: boolean;
|
||||
onClose: () => void;
|
||||
onRunChange: (run: CatalogSyncRun) => void;
|
||||
onRunUpdate: (run: CatalogSyncRun) => void;
|
||||
onCatalogChanged: () => void;
|
||||
}
|
||||
|
||||
const phases: CatalogSyncPhase[] = [
|
||||
"queued", "connecting", "scanning_tables", "scanning_columns",
|
||||
"scanning_relationships", "planning", "awaiting_confirmation", "applying", "completed",
|
||||
];
|
||||
|
||||
const phaseEnteredByEvent: Record<string, CatalogSyncPhase> = {
|
||||
queued: "queued",
|
||||
started: "connecting",
|
||||
connecting: "connecting",
|
||||
scanning_tables: "scanning_tables",
|
||||
scanning_columns: "scanning_columns",
|
||||
scanning_relationships: "scanning_relationships",
|
||||
planning: "planning",
|
||||
confirmation_required: "awaiting_confirmation",
|
||||
confirmation_received: "awaiting_confirmation",
|
||||
applying: "applying",
|
||||
succeeded: "completed",
|
||||
};
|
||||
|
||||
const phaseLabels: Record<CatalogSyncPhase, { pending: string; current: string; completed: string }> = {
|
||||
queued: { pending: "Queue", current: "Queued", completed: "Queued" },
|
||||
connecting: { pending: "Connect", current: "Connecting", completed: "Connected" },
|
||||
scanning_tables: { pending: "Read tables", current: "Reading tables", completed: "Tables read" },
|
||||
scanning_columns: { pending: "Read columns", current: "Reading columns", completed: "Columns read" },
|
||||
scanning_relationships: {
|
||||
pending: "Read relationships",
|
||||
current: "Reading relationships",
|
||||
completed: "Relationships read",
|
||||
},
|
||||
planning: { pending: "Plan changes", current: "Planning changes", completed: "Changes planned" },
|
||||
awaiting_confirmation: {
|
||||
pending: "Confirm changes if needed",
|
||||
current: "Confirmation required",
|
||||
completed: "Changes confirmed",
|
||||
},
|
||||
applying: { pending: "Apply changes", current: "Applying changes", completed: "Changes applied" },
|
||||
completed: { pending: "Complete", current: "Completing", completed: "Completed" },
|
||||
};
|
||||
|
||||
function terminal(run?: CatalogSyncRun): boolean {
|
||||
return Boolean(run && ["succeeded", "failed", "cancelled", "interrupted"].includes(run.state));
|
||||
}
|
||||
|
||||
function elapsed(run: CatalogSyncRun, now: number): string {
|
||||
const start = new Date(run.startedAt ?? run.createdAt).getTime();
|
||||
const end = run.finishedAt ? new Date(run.finishedAt).getTime() : now;
|
||||
const seconds = Math.max(0, Math.floor((end - start) / 1000));
|
||||
const minutes = Math.floor(seconds / 60);
|
||||
return minutes ? `${minutes}m ${seconds % 60}s` : `${seconds}s`;
|
||||
}
|
||||
|
||||
function stateLabel(run: CatalogSyncRun): string {
|
||||
return run.state.replaceAll("_", " ");
|
||||
}
|
||||
|
||||
export function CatalogSyncDrawer({
|
||||
databaseId,
|
||||
runId,
|
||||
open,
|
||||
onClose,
|
||||
onRunChange,
|
||||
onRunUpdate,
|
||||
onCatalogChanged,
|
||||
}: Props) {
|
||||
const queryClient = useQueryClient();
|
||||
const [events, setEvents] = useState<CatalogSyncEvent[]>([]);
|
||||
const [now, setNow] = useState(Date.now());
|
||||
const [action, setAction] = useState<"confirm" | "cancel" | "retry" | null>(null);
|
||||
const lastSequence = useRef(0);
|
||||
const notifiedRun = useRef<string | null>(null);
|
||||
|
||||
const runQuery = useQuery({
|
||||
queryKey: ["catalog-sync-run", runId],
|
||||
queryFn: () => getCatalogSyncRun(runId!),
|
||||
enabled: Boolean(runId),
|
||||
retry: false,
|
||||
refetchInterval: (query) => terminal(query.state.data) ? false : 1_000,
|
||||
});
|
||||
const run = runQuery.data;
|
||||
const historyQuery = useQuery({
|
||||
queryKey: ["catalog-sync-runs", databaseId],
|
||||
queryFn: () => listCatalogSyncRuns(databaseId!),
|
||||
enabled: Boolean(databaseId && open),
|
||||
retry: false,
|
||||
refetchInterval: run && !terminal(run) ? 2_000 : false,
|
||||
});
|
||||
const eventQuery = useQuery({
|
||||
queryKey: ["catalog-sync-events", runId],
|
||||
queryFn: () => listCatalogSyncEvents(runId!, lastSequence.current),
|
||||
enabled: Boolean(runId && open),
|
||||
retry: false,
|
||||
refetchInterval: run && terminal(run) ? false : 1_500,
|
||||
});
|
||||
|
||||
const mergeEvents = (incoming: CatalogSyncEvent[]) => {
|
||||
if (incoming.length === 0) return;
|
||||
setEvents((current) => {
|
||||
const bySequence = new Map(current.map((event) => [event.sequence, event]));
|
||||
for (const event of incoming) bySequence.set(event.sequence, event);
|
||||
const merged = [...bySequence.values()].sort((a, b) => a.sequence - b.sequence);
|
||||
lastSequence.current = merged.at(-1)?.sequence ?? lastSequence.current;
|
||||
return merged;
|
||||
});
|
||||
};
|
||||
|
||||
useEffect(() => {
|
||||
setEvents([]);
|
||||
lastSequence.current = 0;
|
||||
}, [runId]);
|
||||
|
||||
useEffect(() => { mergeEvents(eventQuery.data ?? []); }, [eventQuery.data]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!runId || !open || typeof EventSource === "undefined") return;
|
||||
const source = new EventSource(catalogSyncEventsUrl(runId));
|
||||
const log = (event: MessageEvent<string>) => {
|
||||
try { mergeEvents([JSON.parse(event.data) as CatalogSyncEvent]); } catch { /* polling remains authoritative */ }
|
||||
};
|
||||
const update = (event: MessageEvent<string>) => {
|
||||
try {
|
||||
const next = JSON.parse(event.data) as CatalogSyncRun;
|
||||
queryClient.setQueryData(["catalog-sync-run", runId], next);
|
||||
} catch { /* polling remains authoritative */ }
|
||||
};
|
||||
source.addEventListener("log", log as EventListener);
|
||||
source.addEventListener("run", update as EventListener);
|
||||
source.onerror = () => source.close();
|
||||
return () => source.close();
|
||||
}, [open, queryClient, runId]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!run || terminal(run)) return;
|
||||
const timer = window.setInterval(() => setNow(Date.now()), 1_000);
|
||||
return () => window.clearInterval(timer);
|
||||
}, [run]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!run) return;
|
||||
onRunUpdate(run);
|
||||
queryClient.setQueryData<CatalogSyncRun[]>(["catalog-sync-runs", run.databaseId], (current) => {
|
||||
if (!current) return [run];
|
||||
const found = current.some((item) => item.id === run.id);
|
||||
return found
|
||||
? current.map((item) => item.id === run.id ? run : item)
|
||||
: [run, ...current];
|
||||
});
|
||||
}, [onRunUpdate, queryClient, run]);
|
||||
|
||||
useEffect(() => {
|
||||
if (!run || run.state !== "succeeded" || notifiedRun.current === run.id) return;
|
||||
notifiedRun.current = run.id;
|
||||
onCatalogChanged();
|
||||
}, [onCatalogChanged, run]);
|
||||
|
||||
const currentPhase = run ? phases.indexOf(run.phase) : -1;
|
||||
const knownEvents = [...events, ...(eventQuery.data ?? [])];
|
||||
const confirmationWasRequired = run?.state === "awaiting_confirmation" || knownEvents.some((event) => (
|
||||
event.eventType === "confirmation_required" || event.eventType === "confirmation_received"
|
||||
));
|
||||
const confirmationWasReceived = knownEvents.some((event) => event.eventType === "confirmation_received")
|
||||
|| (run?.state === "succeeded" && confirmationWasRequired);
|
||||
const furthestEventPhase = knownEvents.reduce((furthest, event) => {
|
||||
const phase = phaseEnteredByEvent[event.eventType];
|
||||
return phase ? Math.max(furthest, phases.indexOf(phase)) : furthest;
|
||||
}, -1);
|
||||
const confirmationPhase = phases.indexOf("awaiting_confirmation");
|
||||
const passedConfirmation = run?.state === "succeeded"
|
||||
|| (run && !terminal(run) ? currentPhase > confirmationPhase : furthestEventPhase > confirmationPhase);
|
||||
const deletionCount = useMemo(() => run?.plannedDiff
|
||||
? run.plannedDiff.deletedTables.length + run.plannedDiff.deletedColumns.length + run.plannedDiff.deletedRelationships.length
|
||||
: 0, [run]);
|
||||
|
||||
const perform = async (kind: "confirm" | "cancel" | "retry") => {
|
||||
if (!run) return;
|
||||
setAction(kind);
|
||||
try {
|
||||
const next = kind === "confirm"
|
||||
? await confirmCatalogSync(run.id, run.confirmationToken!)
|
||||
: kind === "cancel"
|
||||
? await cancelCatalogSync(run.id)
|
||||
: await retryCatalogSync(run.id);
|
||||
onRunChange(next);
|
||||
queryClient.setQueryData(["catalog-sync-run", next.id], next);
|
||||
await historyQuery.refetch();
|
||||
} catch (error) {
|
||||
toast.error(apiErrorMessage(error));
|
||||
} finally {
|
||||
setAction(null);
|
||||
}
|
||||
};
|
||||
|
||||
if (!open) return null;
|
||||
return (
|
||||
<aside
|
||||
aria-label="Schema synchronization"
|
||||
className="fixed inset-y-2 right-0 z-40 flex w-full max-w-[440px] flex-col border-l border-border bg-background shadow-2xl sm:inset-y-4"
|
||||
>
|
||||
<div className="flex items-start justify-between gap-4 border-b border-border px-5 py-4">
|
||||
<div>
|
||||
<p className="thot-label">Schema synchronization</p>
|
||||
<h2 className="mt-1 font-heading text-xl font-semibold">{run ? `${stateLabel(run)[0].toUpperCase()}${stateLabel(run).slice(1)}` : "Loading"}</h2>
|
||||
{run ? <p className="mt-1 text-sm text-muted-foreground">{run.scope} · {elapsed(run, now)}</p> : null}
|
||||
</div>
|
||||
<Button type="button" variant="ghost" size="icon-lg" aria-label="Close synchronization drawer" onClick={onClose}>
|
||||
<X aria-hidden="true" />
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
<div className="min-h-0 flex-1 overflow-y-auto px-5 py-5">
|
||||
{runQuery.isLoading ? <p className="text-sm text-muted-foreground">Loading synchronization…</p> : null}
|
||||
{run ? (
|
||||
<>
|
||||
<section aria-labelledby="synchronization-steps-heading">
|
||||
<h3 id="synchronization-steps-heading" className="thot-label mb-3">Synchronization steps</h3>
|
||||
<ol className="grid grid-cols-2 gap-x-4 gap-y-2.5 sm:grid-cols-3" aria-label="Synchronization steps">
|
||||
{phases.map((phase, index) => {
|
||||
const confirmationUnknown = phase === "awaiting_confirmation"
|
||||
&& passedConfirmation
|
||||
&& !eventQuery.isFetched
|
||||
&& !confirmationWasRequired;
|
||||
const confirmationSkipped = phase === "awaiting_confirmation"
|
||||
&& passedConfirmation
|
||||
&& eventQuery.isFetched
|
||||
&& !confirmationWasRequired;
|
||||
const terminalOutcome = phase === "completed" && terminal(run) && run.state !== "succeeded";
|
||||
const failedOutcome = terminalOutcome && run.state === "failed";
|
||||
const interruptedOutcome = terminalOutcome && run.state === "interrupted";
|
||||
const priorPhaseCompleted = terminal(run)
|
||||
? index < furthestEventPhase
|
||||
: index < currentPhase;
|
||||
const complete = !confirmationUnknown && !confirmationSkipped && !terminalOutcome
|
||||
&& (phase === "awaiting_confirmation"
|
||||
? confirmationWasReceived
|
||||
: run.state === "succeeded" || priorPhaseCompleted);
|
||||
const current = index === currentPhase && !terminal(run);
|
||||
const label = phase === "awaiting_confirmation"
|
||||
? confirmationSkipped
|
||||
? "Confirmation not required"
|
||||
: confirmationUnknown
|
||||
? "Checking confirmation"
|
||||
: confirmationWasReceived
|
||||
? phaseLabels[phase].completed
|
||||
: current
|
||||
? phaseLabels[phase].current
|
||||
: complete
|
||||
? phaseLabels[phase].completed
|
||||
: phaseLabels[phase].pending
|
||||
: terminalOutcome
|
||||
? `${stateLabel(run)[0].toUpperCase()}${stateLabel(run).slice(1)}`
|
||||
: complete
|
||||
? phaseLabels[phase].completed
|
||||
: current
|
||||
? phaseLabels[phase].current
|
||||
: phaseLabels[phase].pending;
|
||||
const status = terminalOutcome
|
||||
? run.state
|
||||
: confirmationSkipped
|
||||
? "not required"
|
||||
: complete
|
||||
? "completed"
|
||||
: current
|
||||
? "in progress"
|
||||
: "pending";
|
||||
return (
|
||||
<li
|
||||
key={phase}
|
||||
aria-current={current ? "step" : undefined}
|
||||
aria-label={`${label}, ${status}`}
|
||||
className={`flex items-center gap-2 text-xs ${failedOutcome ? "font-semibold text-destructive" : interruptedOutcome ? "font-semibold text-amber-600" : terminalOutcome || current ? "font-semibold text-foreground" : "text-muted-foreground"}`}
|
||||
>
|
||||
{confirmationSkipped ? (
|
||||
<Minus aria-hidden="true" className="size-3.5 text-muted-foreground" />
|
||||
) : failedOutcome ? (
|
||||
<X aria-hidden="true" className="size-3.5 text-destructive" />
|
||||
) : interruptedOutcome ? (
|
||||
<AlertTriangle aria-hidden="true" className="size-3.5 text-amber-600" />
|
||||
) : terminalOutcome ? (
|
||||
<Minus aria-hidden="true" className="size-3.5 text-muted-foreground" />
|
||||
) : complete ? (
|
||||
<CheckCircle2 aria-hidden="true" className="size-3.5 text-[oklch(var(--success))]" />
|
||||
) : current ? (
|
||||
<LoaderCircle aria-hidden="true" className="size-3.5 animate-spin text-amber-600" />
|
||||
) : (
|
||||
<Circle aria-hidden="true" className="size-3.5" />
|
||||
)}
|
||||
{label}
|
||||
</li>
|
||||
);
|
||||
})}
|
||||
</ol>
|
||||
</section>
|
||||
|
||||
<div className="mt-5 grid grid-cols-3 gap-2">
|
||||
{(["tables", "columns", "relationships"] as const).map((name) => (
|
||||
<div key={name} className="rounded-md border border-border bg-muted/25 px-3 py-2">
|
||||
<p className="text-[11px] font-semibold uppercase tracking-wide text-muted-foreground">{name}</p>
|
||||
<p className="mt-1 text-lg font-semibold tabular-nums">{run.counts[name] ?? ""}</p>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
|
||||
{run.state === "awaiting_confirmation" && run.plannedDiff ? (
|
||||
<section className="mt-5 rounded-md border border-amber-500/50 bg-amber-500/8 p-4" aria-label="Destructive changes">
|
||||
<div className="flex gap-3">
|
||||
<AlertTriangle className="mt-0.5 size-5 shrink-0 text-amber-600" />
|
||||
<div>
|
||||
<h3 className="font-semibold">Confirm {deletionCount} removals</h3>
|
||||
<p className="mt-1 text-sm text-muted-foreground">The database will be scanned again before anything is removed.</p>
|
||||
</div>
|
||||
</div>
|
||||
<ul className="mt-3 max-h-40 space-y-1 overflow-y-auto font-mono text-xs">
|
||||
{run.plannedDiff.deletedTables.map((name) => <li key={`t-${name}`}>table · {name}</li>)}
|
||||
{run.plannedDiff.deletedColumns.map((item) => <li key={`c-${item.tableName}-${item.columnName}`}>column · {item.tableName}.{item.columnName}</li>)}
|
||||
{run.plannedDiff.deletedRelationships.map((item) => <li key={`r-${item.sourceTableName}-${item.constraintName}`}>relationship · {item.sourceTableName}.{item.constraintName}</li>)}
|
||||
</ul>
|
||||
</section>
|
||||
) : null}
|
||||
|
||||
{run.errorMessage ? (
|
||||
<div className="mt-5 rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
|
||||
{run.errorMessage}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
<section className="mt-5" aria-label="Synchronization log">
|
||||
<div className="mb-2 flex items-center justify-between">
|
||||
<h3 className="thot-label">Live log</h3>
|
||||
<span className="text-xs tabular-nums text-muted-foreground">{events.length}</span>
|
||||
</div>
|
||||
<div className="max-h-64 overflow-y-auto rounded-md bg-zinc-950 p-3 font-mono text-xs leading-5 text-zinc-200">
|
||||
{events.length === 0 ? <p className="text-zinc-500">Waiting for events…</p> : events.map((event) => (
|
||||
<p key={event.sequence} className={event.level === "error" ? "text-red-300" : event.level === "warning" ? "text-amber-300" : undefined}>
|
||||
<span className="mr-2 text-zinc-500">{new Date(event.createdAt).toLocaleTimeString()}</span>{event.message}
|
||||
</p>
|
||||
))}
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section className="mt-5" aria-label="Synchronization history">
|
||||
<h3 className="thot-label mb-2">Recent runs</h3>
|
||||
<div className="divide-y divide-border rounded-md border border-border">
|
||||
{(historyQuery.data ?? []).slice(0, 5).map((item) => (
|
||||
<button key={item.id} type="button" className="flex w-full items-center justify-between gap-3 px-3 py-2 text-left text-sm hover:bg-muted/50" onClick={() => onRunChange(item)}>
|
||||
<span>{item.scope}</span>
|
||||
<span className="text-xs text-muted-foreground">{stateLabel(item)} · {new Date(item.createdAt).toLocaleString()}</span>
|
||||
</button>
|
||||
))}
|
||||
</div>
|
||||
</section>
|
||||
</>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
{run ? (
|
||||
<div className="flex justify-end gap-2 border-t border-border px-5 py-4">
|
||||
{run.state === "awaiting_confirmation" ? (
|
||||
<Button type="button" disabled={action !== null} onClick={() => void perform("confirm")}>
|
||||
{action === "confirm" ? <LoaderCircle className="animate-spin" /> : <Check />} Confirm removals
|
||||
</Button>
|
||||
) : null}
|
||||
{["queued", "running", "awaiting_confirmation"].includes(run.state) ? (
|
||||
<Button type="button" variant="outline" disabled={action !== null} onClick={() => void perform("cancel")}>Cancel</Button>
|
||||
) : null}
|
||||
{["failed", "cancelled", "interrupted"].includes(run.state) ? (
|
||||
<Button type="button" disabled={action !== null} onClick={() => void perform("retry")}>
|
||||
<RotateCcw className={action === "retry" ? "animate-spin" : ""} /> Retry
|
||||
</Button>
|
||||
) : null}
|
||||
</div>
|
||||
) : null}
|
||||
</aside>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
import { useEffect, useMemo, useRef, useState } from "react";
|
||||
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { AgGridReact } from "ag-grid-react";
|
||||
import type { ColDef, ICellRendererParams } from "ag-grid-community";
|
||||
import { KeyRound, Link2, Pencil, RefreshCw, Save } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
import { Button } from "../../components/ui/button";
|
||||
import { ApiError, apiErrorMessage } from "../../api/client";
|
||||
import {
|
||||
listCatalogColumns,
|
||||
updateCatalogColumnMetadata,
|
||||
type CatalogColumn,
|
||||
type CatalogTable,
|
||||
} from "../../api/catalog-databases";
|
||||
import type { DatabaseNavigationState } from "./model";
|
||||
|
||||
interface Props {
|
||||
databaseId: string;
|
||||
table: CatalogTable;
|
||||
canManage: boolean;
|
||||
onNavigationStateChange: (state: DatabaseNavigationState) => void;
|
||||
onSync: () => void;
|
||||
}
|
||||
|
||||
interface GridContext {
|
||||
canManage: boolean;
|
||||
onEdit: (column: CatalogColumn, origin: HTMLButtonElement) => void;
|
||||
}
|
||||
|
||||
function KeyCell({ data }: ICellRendererParams<CatalogColumn>) {
|
||||
if (!data) return null;
|
||||
return (
|
||||
<div className="flex h-full items-center gap-1.5">
|
||||
{data.isPrimaryKey ? <span className="inline-flex items-center gap-1 rounded bg-primary/10 px-1.5 py-0.5 text-xs font-semibold text-primary"><KeyRound className="size-3" />PK{data.primaryKeyPosition && data.primaryKeyPosition > 1 ? ` ${data.primaryKeyPosition}` : ""}</span> : null}
|
||||
{data.isForeignKey ? <span className="inline-flex items-center gap-1 rounded bg-muted px-1.5 py-0.5 text-xs font-semibold text-muted-foreground"><Link2 className="size-3" />FK{data.foreignKeyCount > 1 ? ` ${data.foreignKeyCount}` : ""}</span> : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function ActionCell({ data, context }: ICellRendererParams<CatalogColumn, unknown, GridContext>) {
|
||||
if (!data || !context) return null;
|
||||
return (
|
||||
<div className="flex h-full items-center justify-end" onClick={(event) => event.stopPropagation()}>
|
||||
<Button type="button" variant="ghost" size="icon-lg" disabled={!context.canManage} aria-label={`Edit metadata for ${data.name}`} onClick={(event) => context.onEdit(data, event.currentTarget)}>
|
||||
<Pencil aria-hidden="true" />
|
||||
</Button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export function DatabaseColumns({ databaseId, table, canManage, onNavigationStateChange, onSync }: Props) {
|
||||
const queryClient = useQueryClient();
|
||||
const queryKey = ["catalog-columns", databaseId, table.id] as const;
|
||||
const { data = [], isLoading, isFetching, refetch } = useQuery({
|
||||
queryKey,
|
||||
queryFn: () => listCatalogColumns(databaseId, table.id),
|
||||
retry: false,
|
||||
});
|
||||
const [search, setSearch] = useState("");
|
||||
const [editingId, setEditingId] = useState<string | null>(null);
|
||||
const [description, setDescription] = useState("");
|
||||
const [generatedDescription, setGeneratedDescription] = useState("");
|
||||
const [baseline, setBaseline] = useState("");
|
||||
const [version, setVersion] = useState<number | null>(null);
|
||||
const [stale, setStale] = useState(false);
|
||||
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const originRef = useRef<HTMLButtonElement | null>(null);
|
||||
const active = editingId ? data.find((column) => column.id === editingId) : undefined;
|
||||
const fingerprint = JSON.stringify([description, generatedDescription]);
|
||||
const dirty = Boolean(editingId && fingerprint !== baseline);
|
||||
|
||||
useEffect(() => { onNavigationStateChange({ dirty, busy }); }, [busy, dirty, onNavigationStateChange]);
|
||||
useEffect(() => {
|
||||
if (!editingId || !active || version === active.version || busy) return;
|
||||
setStale(true);
|
||||
setStaleBannerOpen(true);
|
||||
}, [active, busy, editingId, version]);
|
||||
|
||||
const edit = (column: CatalogColumn, origin: HTMLButtonElement) => {
|
||||
originRef.current = origin;
|
||||
setEditingId(column.id);
|
||||
setDescription(column.description ?? "");
|
||||
setGeneratedDescription(column.generatedDescription ?? "");
|
||||
setBaseline(JSON.stringify([column.description ?? "", column.generatedDescription ?? ""]));
|
||||
setVersion(column.version);
|
||||
setStale(false);
|
||||
setStaleBannerOpen(true);
|
||||
};
|
||||
const closeEditor = () => {
|
||||
if (busy) return;
|
||||
if (dirty && !window.confirm("Discard unsaved column metadata?")) return;
|
||||
setEditingId(null);
|
||||
window.setTimeout(() => originRef.current?.focus(), 0);
|
||||
};
|
||||
const save = async () => {
|
||||
if (!active || version === null) return;
|
||||
setBusy(true);
|
||||
try {
|
||||
const updated = await updateCatalogColumnMetadata(
|
||||
databaseId, table.id, active.id, version,
|
||||
description.trim() || null, generatedDescription.trim() || null,
|
||||
);
|
||||
queryClient.setQueryData<CatalogColumn[]>(queryKey, (current = []) => current.map((column) => column.id === updated.id ? updated : column));
|
||||
setDescription(updated.description ?? "");
|
||||
setGeneratedDescription(updated.generatedDescription ?? "");
|
||||
setBaseline(JSON.stringify([updated.description ?? "", updated.generatedDescription ?? ""]));
|
||||
setVersion(updated.version);
|
||||
setStale(false);
|
||||
toast.success("Column metadata saved");
|
||||
} catch (error) {
|
||||
if (error instanceof ApiError && error.code === "column_stale") {
|
||||
await refetch();
|
||||
setStale(true);
|
||||
setStaleBannerOpen(true);
|
||||
} else toast.error(apiErrorMessage(error));
|
||||
} finally { setBusy(false); }
|
||||
};
|
||||
const reloadColumn = async () => {
|
||||
if (!editingId) return;
|
||||
setBusy(true);
|
||||
try {
|
||||
const result = await refetch();
|
||||
const latest = result.data?.find((column) => column.id === editingId);
|
||||
if (!latest) { closeEditor(); return; }
|
||||
setDescription(latest.description ?? "");
|
||||
setGeneratedDescription(latest.generatedDescription ?? "");
|
||||
setBaseline(JSON.stringify([latest.description ?? "", latest.generatedDescription ?? ""]));
|
||||
setVersion(latest.version);
|
||||
setStale(false);
|
||||
setStaleBannerOpen(true);
|
||||
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(false); }
|
||||
};
|
||||
|
||||
const columns = useMemo<ColDef<CatalogColumn>[]>(() => [
|
||||
{ field: "ordinalPosition", headerName: "#", width: 64, maxWidth: 64, filter: "agNumberColumnFilter" },
|
||||
{ field: "name", headerName: "Name", minWidth: 190, flex: 1, cellClass: "font-mono text-xs" },
|
||||
{ field: "dataType", headerName: "Type", minWidth: 150, flex: 0.8, cellClass: "font-mono text-xs" },
|
||||
{ headerName: "Keys", minWidth: 125, width: 125, sortable: false, filter: false, cellRenderer: KeyCell },
|
||||
{ headerName: "Nullable", minWidth: 100, width: 100, valueGetter: ({ data: row }) => row ? (row.isNullable ? "Yes" : "No") : "" },
|
||||
{ field: "sourceComment", headerName: "Source comment", minWidth: 220, flex: 1.2, valueFormatter: ({ value }) => value ?? "" },
|
||||
{ field: "generatedDescription", headerName: "Generated description", minWidth: 230, flex: 1.2, valueFormatter: ({ value }) => value ?? "" },
|
||||
{ field: "description", headerName: "Description", minWidth: 230, flex: 1.2, valueFormatter: ({ value }) => value ?? "" },
|
||||
{ colId: "actions", headerName: "", width: 64, maxWidth: 64, pinned: "right", sortable: false, filter: false, resizable: false, cellRenderer: ActionCell },
|
||||
], []);
|
||||
const context = useMemo<GridContext>(() => ({ canManage, onEdit: edit }), [canManage, data]);
|
||||
|
||||
if (editingId && active) {
|
||||
return (
|
||||
<div className="mx-auto w-full max-w-4xl px-4 py-6 sm:px-5">
|
||||
<Button type="button" variant="ghost" className="-ml-2 mb-4" disabled={busy} onClick={closeEditor}>← Back to columns</Button>
|
||||
<p className="thot-label">{table.name} · column</p>
|
||||
<h3 className="mt-1 font-heading text-2xl font-semibold">{active.name}</h3>
|
||||
<p className="mt-1 text-sm text-muted-foreground">Physical schema fields are read-only. Review fields can be edited.</p>
|
||||
{stale ? (
|
||||
staleBannerOpen ? <div className="mt-4 rounded-md border border-amber-500/50 bg-amber-500/8 p-4 text-sm"><p className="font-semibold">Newer column metadata is available.</p><p className="mt-1 text-muted-foreground">Your draft is preserved until you reload.</p><div className="mt-3 flex gap-2"><Button type="button" variant="outline" onClick={() => void reloadColumn()}>Reload latest</Button><Button type="button" variant="ghost" onClick={() => setStaleBannerOpen(false)}>Keep editing</Button></div></div>
|
||||
: <div className="mt-4 flex items-center justify-between gap-3 rounded-md border border-amber-500/40 bg-amber-500/8 px-4 py-3 text-sm"><span>Reload the latest value before this metadata can be saved.</span><Button type="button" variant="outline" onClick={() => void reloadColumn()}>Reload latest</Button></div>
|
||||
) : null}
|
||||
<div className="mt-5 grid gap-4 border-t border-border pt-5 sm:grid-cols-2">
|
||||
<label className="grid gap-1.5 text-sm font-semibold">Physical column name<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={active.name} readOnly /></label>
|
||||
<label className="grid gap-1.5 text-sm font-semibold">Native type<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={active.dataType} readOnly /></label>
|
||||
<label className="grid gap-1.5 text-sm font-semibold">Ordinal position<input className="h-9 rounded-md border border-input bg-muted/35 px-3" value={active.ordinalPosition} readOnly /></label>
|
||||
<label className="grid gap-1.5 text-sm font-semibold">Default expression<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={active.defaultExpression ?? ""} readOnly /></label>
|
||||
<label className="grid gap-1.5 text-sm font-semibold sm:col-span-2">Source comment<textarea className="min-h-24 rounded-md border border-input bg-muted/35 px-3 py-2 text-sm" value={active.sourceComment ?? ""} readOnly /></label>
|
||||
<label className="grid gap-1.5 text-sm font-semibold sm:col-span-2">Generated description<textarea className="min-h-28 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={generatedDescription} disabled={!canManage || busy} onChange={(event) => setGeneratedDescription(event.target.value)} /></label>
|
||||
<label className="grid gap-1.5 text-sm font-semibold sm:col-span-2">Description<textarea className="min-h-32 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={description} disabled={!canManage || busy} onChange={(event) => setDescription(event.target.value)} /></label>
|
||||
</div>
|
||||
<div className="mt-5 flex justify-end gap-2 border-t border-border pt-4">
|
||||
<Button type="button" variant="outline" disabled={busy} onClick={closeEditor}>Cancel</Button>
|
||||
<Button type="button" disabled={!canManage || !dirty || busy || stale} onClick={() => void save()}><Save />{busy ? "Saving…" : "Save metadata"}</Button>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div className="flex min-h-0 flex-1 flex-col">
|
||||
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
|
||||
<span className="thot-label whitespace-nowrap">Catalog columns</span>
|
||||
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search columns" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
|
||||
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
|
||||
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
|
||||
<Button type="button" disabled={!canManage || busy} onClick={onSync}><RefreshCw />Sync columns</Button>
|
||||
</div>
|
||||
<div className="relative min-h-[280px] flex-1">
|
||||
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
|
||||
<AgGridReact<CatalogColumn> rowData={data} columnDefs={columns} context={context} loading={isLoading} quickFilterText={search} defaultColDef={{ sortable: true, filter: true, resizable: true }} getRowId={({ data: row }) => row.id} rowHeight={44} headerHeight={38} animateRows={false} overlayNoRowsTemplate="No columns synchronized for this table." />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,510 @@
|
||||
import type { RefObject, ReactNode } from "react";
|
||||
import {
|
||||
ArrowLeft,
|
||||
Database,
|
||||
RefreshCw,
|
||||
Save,
|
||||
TestTube2,
|
||||
Trash2,
|
||||
} from "lucide-react";
|
||||
import { Button } from "../../components/ui/button";
|
||||
import type {
|
||||
CatalogDatabase,
|
||||
CatalogSecretName,
|
||||
CatalogSyncScope,
|
||||
CatalogSyncRun,
|
||||
DatabaseBinding,
|
||||
DatabaseTransport,
|
||||
} from "../../api/catalog-databases";
|
||||
import type {
|
||||
DatabaseBusyAction,
|
||||
DatabaseFormDraft,
|
||||
DatabaseFormMode,
|
||||
} from "./model";
|
||||
import { statusLabel } from "./model";
|
||||
import { DatabaseSyncMenu } from "./DatabaseSyncMenu";
|
||||
|
||||
const inputClass = "h-9 w-full rounded-md border border-input bg-card px-3 text-sm outline-none transition focus:border-primary/60 focus:ring-3 focus:ring-ring/15 read-only:bg-muted/40 read-only:text-muted-foreground disabled:bg-muted disabled:text-muted-foreground";
|
||||
const labelClass = "grid min-w-0 gap-1.5 text-xs font-medium text-foreground";
|
||||
|
||||
interface FieldProps {
|
||||
label: string;
|
||||
hint?: string;
|
||||
children: ReactNode;
|
||||
}
|
||||
|
||||
function Field({ label, hint, children }: FieldProps) {
|
||||
return (
|
||||
<label className={labelClass}>
|
||||
<span>{label}</span>
|
||||
{children}
|
||||
{hint ? <span className="font-normal leading-4 text-muted-foreground">{hint}</span> : null}
|
||||
</label>
|
||||
);
|
||||
}
|
||||
|
||||
interface SecretFieldProps {
|
||||
label: string;
|
||||
name: CatalogSecretName;
|
||||
row: CatalogDatabase;
|
||||
draft: DatabaseFormDraft;
|
||||
readOnly: boolean;
|
||||
canManageSecrets: boolean;
|
||||
onChange: (name: CatalogSecretName, value: string) => void;
|
||||
multiline?: boolean;
|
||||
}
|
||||
|
||||
function SecretField({
|
||||
label,
|
||||
name,
|
||||
row,
|
||||
draft,
|
||||
readOnly,
|
||||
canManageSecrets,
|
||||
onChange,
|
||||
multiline = false,
|
||||
}: SecretFieldProps) {
|
||||
if (readOnly) {
|
||||
return (
|
||||
<Field label={label} hint="Secret values are never displayed.">
|
||||
<input
|
||||
className={inputClass}
|
||||
value={row.secrets[name] ? "Configured" : "Not configured"}
|
||||
readOnly
|
||||
aria-label={label}
|
||||
/>
|
||||
</Field>
|
||||
);
|
||||
}
|
||||
|
||||
const permissionHint = canManageSecrets
|
||||
? undefined
|
||||
: "You do not have permission to replace secret values.";
|
||||
|
||||
return (
|
||||
<Field label={label} hint={permissionHint}>
|
||||
{multiline ? (
|
||||
<textarea
|
||||
className={`${inputClass} min-h-24 resize-y py-2 font-mono text-xs leading-5`}
|
||||
aria-label={label}
|
||||
autoComplete="off"
|
||||
value={draft.secrets[name] ?? ""}
|
||||
disabled={!canManageSecrets}
|
||||
placeholder={row.secrets[name] ? "Configured; leave blank to keep" : "Paste value"}
|
||||
onChange={(event) => onChange(name, event.target.value)}
|
||||
/>
|
||||
) : (
|
||||
<input
|
||||
className={inputClass}
|
||||
type="password"
|
||||
aria-label={label}
|
||||
autoComplete="new-password"
|
||||
value={draft.secrets[name] ?? ""}
|
||||
disabled={!canManageSecrets}
|
||||
placeholder={row.secrets[name] ? "Configured; leave blank to keep" : "Not configured"}
|
||||
onChange={(event) => onChange(name, event.target.value)}
|
||||
/>
|
||||
)}
|
||||
</Field>
|
||||
);
|
||||
}
|
||||
|
||||
interface DatabaseFormProps {
|
||||
mode: DatabaseFormMode;
|
||||
row: CatalogDatabase;
|
||||
availableWorkspaces: CatalogDatabase[];
|
||||
workspaceLocked: boolean;
|
||||
draft: DatabaseFormDraft;
|
||||
dirty: boolean;
|
||||
canManage: boolean;
|
||||
canManageSecrets: boolean;
|
||||
busyAction: DatabaseBusyAction;
|
||||
stale: boolean;
|
||||
staleBannerOpen: boolean;
|
||||
partialSecretFailure: string | null;
|
||||
headingRef: RefObject<HTMLHeadingElement>;
|
||||
onBack: () => void;
|
||||
onWorkspaceChange: (workspaceId: string) => void;
|
||||
onFieldChange: (field: "databaseName" | "schema", value: string) => void;
|
||||
onTransportChange: (transport: DatabaseTransport) => void;
|
||||
onBindingChange: <K extends keyof DatabaseBinding>(key: K, value: DatabaseBinding[K]) => void;
|
||||
onSecretChange: (name: CatalogSecretName, value: string) => void;
|
||||
onSave: () => void;
|
||||
onTest: () => void;
|
||||
onDelete: () => void;
|
||||
onReloadLatest: () => void;
|
||||
onKeepEditing: () => void;
|
||||
onRetrySecrets: () => void;
|
||||
onOpenTables: (origin: HTMLButtonElement) => void;
|
||||
onOpenRelationships: () => void;
|
||||
onSync: (scope: CatalogSyncScope) => void;
|
||||
onOpenSync: () => void;
|
||||
activeSyncRun?: CatalogSyncRun;
|
||||
}
|
||||
|
||||
function formTitle(mode: DatabaseFormMode, workspaceLocked: boolean): string {
|
||||
if (mode === "add") return workspaceLocked ? "Edit database" : "Add database";
|
||||
if (mode === "edit") return "Edit database";
|
||||
if (mode === "delete") return "Delete database";
|
||||
return "Database details";
|
||||
}
|
||||
|
||||
export function DatabaseForm({
|
||||
mode,
|
||||
row,
|
||||
availableWorkspaces,
|
||||
workspaceLocked,
|
||||
draft,
|
||||
dirty,
|
||||
canManage,
|
||||
canManageSecrets,
|
||||
busyAction,
|
||||
stale,
|
||||
staleBannerOpen,
|
||||
partialSecretFailure,
|
||||
headingRef,
|
||||
onBack,
|
||||
onWorkspaceChange,
|
||||
onFieldChange,
|
||||
onTransportChange,
|
||||
onBindingChange,
|
||||
onSecretChange,
|
||||
onSave,
|
||||
onTest,
|
||||
onDelete,
|
||||
onReloadLatest,
|
||||
onKeepEditing,
|
||||
onRetrySecrets,
|
||||
onOpenTables,
|
||||
onOpenRelationships,
|
||||
onSync,
|
||||
onOpenSync,
|
||||
activeSyncRun,
|
||||
}: DatabaseFormProps) {
|
||||
const title = formTitle(mode, workspaceLocked);
|
||||
const readOnly = mode === "view" || mode === "delete";
|
||||
const editable = mode === "add" || mode === "edit";
|
||||
const busy = busyAction !== null;
|
||||
const testDisabled = !canManage
|
||||
|| !row.configured
|
||||
|| !row.id
|
||||
|| dirty
|
||||
|| stale
|
||||
|| busy;
|
||||
const saveDisabled = !canManage
|
||||
|| stale
|
||||
|| busy
|
||||
|| (mode === "edit" && !dirty);
|
||||
const bindingReady = row.connectionStatus === "reachable" && row.testedVersion === row.version;
|
||||
|
||||
const form = (
|
||||
<form
|
||||
className="grid gap-5"
|
||||
onSubmit={(event) => {
|
||||
event.preventDefault();
|
||||
if (editable) onSave();
|
||||
}}
|
||||
>
|
||||
{mode === "delete" ? (
|
||||
<div className="rounded-md border border-destructive/35 bg-destructive/8 p-4 text-sm text-foreground" role="alert">
|
||||
<p className="font-semibold text-destructive">This removes the local database configuration.</p>
|
||||
<p className="mt-1 leading-5 text-muted-foreground">
|
||||
{row.workspaceAvailable
|
||||
? "The repository workspace remains available and will return to the list as Not configured. "
|
||||
: "This orphaned catalog row will disappear from the list. "}
|
||||
Secret references associated with this configuration are removed too.
|
||||
</p>
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{stale ? (
|
||||
staleBannerOpen ? (
|
||||
<div className="rounded-md border border-amber-300/70 bg-amber-50 p-4 text-sm text-amber-950 dark:border-amber-700 dark:bg-amber-950/30 dark:text-amber-100" role="alert">
|
||||
<p className="font-semibold">A newer database configuration is available.</p>
|
||||
<p className="mt-1 leading-5">Reload the latest values before saving, testing, or deleting. Your draft is still intact.</p>
|
||||
<div className="mt-3 flex flex-wrap gap-2">
|
||||
<Button type="button" size="sm" variant="outline" disabled={busy} onClick={onReloadLatest}>
|
||||
<RefreshCw /> Reload latest
|
||||
</Button>
|
||||
<Button type="button" size="sm" variant="ghost" disabled={busy} onClick={onKeepEditing}>
|
||||
Keep editing
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
) : (
|
||||
<div className="rounded-md border border-amber-300/70 bg-amber-50 px-4 py-3 text-sm text-amber-950 dark:border-amber-700 dark:bg-amber-950/30 dark:text-amber-100" role="status">
|
||||
Reload the latest values before this configuration can be changed.
|
||||
</div>
|
||||
)
|
||||
) : null}
|
||||
|
||||
{partialSecretFailure ? (
|
||||
<div className="rounded-md border border-amber-300/70 bg-amber-50 p-4 text-sm text-amber-950 dark:border-amber-700 dark:bg-amber-950/30 dark:text-amber-100" role="alert">
|
||||
<p className="font-semibold">Database configuration saved; secret update could not be confirmed.</p>
|
||||
<p className="mt-1 leading-5">{partialSecretFailure} The values you entered are retained in this form.</p>
|
||||
<Button type="button" size="sm" variant="outline" className="mt-3" disabled={busy || stale} onClick={onRetrySecrets}>
|
||||
<RefreshCw /> {busyAction === "retry-secrets" ? "Retrying…" : "Retry secrets"}
|
||||
</Button>
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{editable && !canManageSecrets ? (
|
||||
<div className="rounded-md border border-border bg-muted/45 px-4 py-3 text-sm text-muted-foreground" role="note">
|
||||
Secret status is visible, but replacing secret values requires the workspace.secrets.manage permission.
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{!row.workspaceAvailable ? (
|
||||
<div className="rounded-md border border-destructive/30 bg-destructive/8 px-4 py-3 text-sm text-destructive" role="alert">
|
||||
This configuration references a workspace that is no longer present in the repository YAML.
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
<div className="grid gap-4 md:grid-cols-2">
|
||||
{mode === "add" ? (
|
||||
<Field label="Workspace">
|
||||
<select
|
||||
className={inputClass}
|
||||
aria-label="Workspace"
|
||||
value={draft.workspaceId}
|
||||
disabled={workspaceLocked || busy}
|
||||
onChange={(event) => onWorkspaceChange(event.target.value)}
|
||||
>
|
||||
{availableWorkspaces.map((workspace) => (
|
||||
<option key={workspace.workspaceId} value={workspace.workspaceId}>
|
||||
{workspace.workspaceName}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
</Field>
|
||||
) : (
|
||||
<Field label="Workspace">
|
||||
<input className={inputClass} value={row.workspaceName} readOnly />
|
||||
</Field>
|
||||
)}
|
||||
<Field label="Workspace ID">
|
||||
<input className={`${inputClass} font-mono text-xs`} value={row.workspaceId} readOnly />
|
||||
</Field>
|
||||
<Field label="Database engine">
|
||||
<input className={inputClass} value="PostgreSQL" readOnly />
|
||||
</Field>
|
||||
<Field label="Transport">
|
||||
<select
|
||||
className={inputClass}
|
||||
aria-label="Transport"
|
||||
value={draft.binding.transport}
|
||||
disabled={readOnly || busy}
|
||||
onChange={(event) => onTransportChange(event.target.value as DatabaseTransport)}
|
||||
>
|
||||
<option value="postgres_direct">Direct PostgreSQL</option>
|
||||
<option value="rest_api">REST API</option>
|
||||
<option value="ssh_tunnel">SSH tunnel</option>
|
||||
</select>
|
||||
</Field>
|
||||
<Field label="Database name">
|
||||
<input
|
||||
className={inputClass}
|
||||
required={editable}
|
||||
value={draft.databaseName}
|
||||
readOnly={readOnly}
|
||||
disabled={busy}
|
||||
onChange={(event) => onFieldChange("databaseName", event.target.value)}
|
||||
/>
|
||||
</Field>
|
||||
<Field label="Schema">
|
||||
<input
|
||||
className={inputClass}
|
||||
required={editable}
|
||||
value={draft.schema}
|
||||
readOnly={readOnly}
|
||||
disabled={busy}
|
||||
onChange={(event) => onFieldChange("schema", event.target.value)}
|
||||
/>
|
||||
</Field>
|
||||
</div>
|
||||
|
||||
<div className="border-t border-border pt-5">
|
||||
<p className="thot-label mb-4">Installation binding</p>
|
||||
|
||||
{draft.binding.transport === "postgres_direct" ? (
|
||||
<div className="grid gap-4 md:grid-cols-2">
|
||||
<Field label="Host">
|
||||
<input className={inputClass} required={editable} value={draft.binding.host ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("host", event.target.value)} />
|
||||
</Field>
|
||||
<Field label="Port">
|
||||
<input className={inputClass} type="number" min={1} max={65535} required={editable} value={draft.binding.port ?? 5432} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("port", Number(event.target.value))} />
|
||||
</Field>
|
||||
<Field label="Username">
|
||||
<input className={inputClass} required={editable} value={draft.binding.username ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("username", event.target.value)} />
|
||||
</Field>
|
||||
<Field label="TLS server name">
|
||||
<input className={inputClass} value={draft.binding.tlsServername ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("tlsServername", event.target.value || undefined)} />
|
||||
</Field>
|
||||
<SecretField label="Password" name="password" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
|
||||
<SecretField label="TLS CA certificate" name="tlsCa" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{draft.binding.transport === "rest_api" ? (
|
||||
<div className="grid gap-4 md:grid-cols-2">
|
||||
<Field label="Base URL">
|
||||
<input className={inputClass} type="url" required={editable} placeholder="https://dwh.example/api" value={draft.binding.baseUrl ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("baseUrl", event.target.value)} />
|
||||
</Field>
|
||||
<Field label="Diagnostic endpoint" hint="This relative path is fixed by the runtime contract.">
|
||||
<input className={inputClass} value={draft.binding.restPath ?? "/health"} readOnly aria-label="Diagnostic endpoint" />
|
||||
</Field>
|
||||
<Field label="Authentication">
|
||||
<select className={inputClass} aria-label="Authentication" value={draft.binding.restAuth ?? "x-api-key"} disabled={readOnly || busy} onChange={(event) => onBindingChange("restAuth", event.target.value as DatabaseBinding["restAuth"])}>
|
||||
<option value="x-api-key">X-API-Key</option>
|
||||
<option value="bearer">Bearer token</option>
|
||||
<option value="none">None</option>
|
||||
</select>
|
||||
</Field>
|
||||
{draft.binding.restAuth !== "none" ? (
|
||||
<SecretField label="API key" name="apiKey" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
|
||||
) : null}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
{draft.binding.transport === "ssh_tunnel" ? (
|
||||
<div className="grid gap-4 md:grid-cols-2">
|
||||
<div className="rounded-md border border-border bg-muted/45 px-4 py-3 text-sm text-muted-foreground md:col-span-2" role="note">
|
||||
SSH host verification is strict. Provide the trusted known_hosts entry before testing this binding.
|
||||
</div>
|
||||
<Field label="Database username">
|
||||
<input className={inputClass} required={editable} value={draft.binding.username ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("username", event.target.value)} />
|
||||
</Field>
|
||||
<SecretField label="Database password" name="password" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
|
||||
<Field label="SSH host">
|
||||
<input className={inputClass} required={editable} value={draft.binding.sshHost ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshHost", event.target.value)} />
|
||||
</Field>
|
||||
<Field label="SSH port">
|
||||
<input className={inputClass} type="number" min={1} max={65535} required={editable} value={draft.binding.sshPort ?? 22} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshPort", Number(event.target.value))} />
|
||||
</Field>
|
||||
<Field label="SSH username">
|
||||
<input className={inputClass} required={editable} value={draft.binding.sshUsername ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshUsername", event.target.value)} />
|
||||
</Field>
|
||||
<SecretField label="SSH private key" name="sshPrivateKey" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} multiline />
|
||||
<SecretField label="Private key passphrase" name="sshPrivateKeyPassphrase" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
|
||||
<SecretField label="SSH known hosts" name="sshKnownHosts" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} multiline />
|
||||
<Field label="Target host">
|
||||
<input className={inputClass} required={editable} value={draft.binding.sshTargetHost ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshTargetHost", event.target.value)} />
|
||||
</Field>
|
||||
<Field label="Target port">
|
||||
<input className={inputClass} type="number" min={1} max={65535} required={editable} value={draft.binding.sshTargetPort ?? 5432} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshTargetPort", Number(event.target.value))} />
|
||||
</Field>
|
||||
<Field label="TLS server name">
|
||||
<input className={inputClass} value={draft.binding.tlsServername ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("tlsServername", event.target.value || undefined)} />
|
||||
</Field>
|
||||
<SecretField label="TLS CA certificate" name="tlsCa" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} multiline />
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
|
||||
{row.lastErrorMessage ? (
|
||||
<div className="rounded-md border border-destructive/25 bg-destructive/8 px-4 py-3 text-sm text-destructive" role="status">
|
||||
Last connection error: {row.lastErrorMessage}
|
||||
</div>
|
||||
) : null}
|
||||
|
||||
<div className="flex flex-wrap items-center justify-end gap-2 border-t border-border pt-4">
|
||||
{mode === "view" || mode === "edit" ? (
|
||||
<Button type="button" variant="outline" disabled={testDisabled} onClick={onTest}>
|
||||
<TestTube2 /> {busyAction === "test" ? "Testing…" : "Test connection"}
|
||||
</Button>
|
||||
) : null}
|
||||
{mode === "add" || mode === "edit" ? (
|
||||
<Button type="submit" disabled={saveDisabled}>
|
||||
<Save />
|
||||
{busyAction === "save"
|
||||
? "Saving…"
|
||||
: mode === "add"
|
||||
? workspaceLocked ? "Save database" : "Add database"
|
||||
: "Save changes"}
|
||||
</Button>
|
||||
) : null}
|
||||
{mode === "delete" ? (
|
||||
<Button type="button" variant="destructive" disabled={!canManage || !row.configured || !row.id || busy || stale} onClick={onDelete}>
|
||||
<Trash2 /> {busyAction === "delete" ? "Deleting…" : "Delete database"}
|
||||
</Button>
|
||||
) : null}
|
||||
</div>
|
||||
</form>
|
||||
);
|
||||
|
||||
return (
|
||||
<section aria-label={`${title} form`} className="min-h-0 flex-1 overflow-y-auto bg-background">
|
||||
<div className="mx-auto w-full max-w-[900px] px-4 py-5 sm:px-6 sm:py-6">
|
||||
<Button type="button" variant="ghost" className="-ml-2 mb-5" disabled={busy} onClick={onBack}>
|
||||
<ArrowLeft /> Back to list
|
||||
</Button>
|
||||
|
||||
<div className="mb-6 flex flex-wrap items-start justify-between gap-4 border-b border-border pb-5">
|
||||
<div className="min-w-0">
|
||||
<p className="thot-label mb-1">Database management</p>
|
||||
<h2 ref={headingRef} tabIndex={-1} className="font-heading text-2xl font-semibold tracking-tight outline-none">
|
||||
{title}
|
||||
</h2>
|
||||
<p className="mt-1 text-sm text-muted-foreground">
|
||||
{row.workspaceName} · {row.workspaceId}
|
||||
</p>
|
||||
{row.lastTestedAt ? (
|
||||
<p className="mt-1 text-xs text-muted-foreground">Last tested {new Date(row.lastTestedAt).toLocaleString()}</p>
|
||||
) : null}
|
||||
</div>
|
||||
<div className="flex flex-wrap items-center justify-end gap-2">
|
||||
<Database className="size-4 text-muted-foreground" aria-hidden="true" />
|
||||
<span className="rounded-full border border-border bg-muted px-2.5 py-1 text-xs font-semibold text-muted-foreground">
|
||||
{statusLabel(row)}
|
||||
</span>
|
||||
{mode === "view" && activeSyncRun ? (
|
||||
<Button type="button" variant="outline" onClick={onOpenSync}><RefreshCw className="animate-spin" />View sync</Button>
|
||||
) : null}
|
||||
{mode === "view" ? (
|
||||
<DatabaseSyncMenu
|
||||
disabled={!canManage || !bindingReady || busy || Boolean(activeSyncRun)}
|
||||
disabledReason={!bindingReady ? "Test the current database binding before synchronizing the schema" : undefined}
|
||||
onSelect={onSync}
|
||||
/>
|
||||
) : null}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{mode === "view" ? (
|
||||
<nav aria-label="Database sections" className="mb-6 flex gap-1 border-b border-border" role="tablist">
|
||||
<button
|
||||
type="button"
|
||||
role="tab"
|
||||
aria-selected="true"
|
||||
className="border-b-2 border-primary px-3 py-2 text-sm font-semibold text-foreground"
|
||||
>
|
||||
Overview
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
role="tab"
|
||||
aria-selected="false"
|
||||
disabled={!row.configured || !row.id}
|
||||
className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground transition hover:text-foreground focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/20 disabled:cursor-not-allowed disabled:opacity-45"
|
||||
onClick={(event) => onOpenTables(event.currentTarget)}
|
||||
>
|
||||
Tables
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
role="tab"
|
||||
aria-selected="false"
|
||||
disabled={!row.configured || !row.id}
|
||||
className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground transition hover:text-foreground focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/20 disabled:cursor-not-allowed disabled:opacity-45"
|
||||
onClick={onOpenRelationships}
|
||||
>
|
||||
Relationships
|
||||
</button>
|
||||
</nav>
|
||||
) : null}
|
||||
|
||||
{form}
|
||||
</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,329 @@
|
||||
import { useEffect, useMemo, useRef, useState, type RefObject } from "react";
|
||||
import { Menu } from "@base-ui/react/menu";
|
||||
import { AgGridReact } from "ag-grid-react";
|
||||
import {
|
||||
AllCommunityModule,
|
||||
ModuleRegistry,
|
||||
type ColDef,
|
||||
type ICellRendererParams,
|
||||
} from "ag-grid-community";
|
||||
import "ag-grid-community/styles/ag-grid.css";
|
||||
import "ag-grid-community/styles/ag-theme-alpine.css";
|
||||
import { ChevronDown, Eye, Pencil, RefreshCw, Trash2, X } from "lucide-react";
|
||||
import { Button } from "../../components/ui/button";
|
||||
import type { CatalogDatabase, CatalogSyncScope } from "../../api/catalog-databases";
|
||||
import { statusLabel } from "./model";
|
||||
import { databaseSyncItemClass, databaseSyncScopes } from "./DatabaseSyncMenu";
|
||||
|
||||
ModuleRegistry.registerModules([AllCommunityModule]);
|
||||
|
||||
interface DatabaseGridProps {
|
||||
rows: CatalogDatabase[];
|
||||
isLoading: boolean;
|
||||
isFetching: boolean;
|
||||
search: string;
|
||||
canManage: boolean;
|
||||
searchInputRef: RefObject<HTMLInputElement>;
|
||||
onSearchChange: (value: string) => void;
|
||||
onView: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
|
||||
onEdit: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
|
||||
onDelete: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
|
||||
onOpenSync: (row: CatalogDatabase) => void;
|
||||
onTestSelected: (rows: CatalogDatabase[]) => Promise<void>;
|
||||
onSyncSelected: (rows: CatalogDatabase[], scope: CatalogSyncScope) => Promise<void>;
|
||||
}
|
||||
|
||||
interface DatabaseGridContext {
|
||||
canManage: boolean;
|
||||
onView: DatabaseGridProps["onView"];
|
||||
onEdit: DatabaseGridProps["onEdit"];
|
||||
onDelete: DatabaseGridProps["onDelete"];
|
||||
onOpenSync: DatabaseGridProps["onOpenSync"];
|
||||
}
|
||||
|
||||
function useCompactViewport(): boolean {
|
||||
const [compact, setCompact] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
if (typeof window.matchMedia !== "function") return;
|
||||
const query = window.matchMedia("(max-width: 767px)");
|
||||
const update = () => setCompact(query.matches);
|
||||
update();
|
||||
query.addEventListener("change", update);
|
||||
return () => query.removeEventListener("change", update);
|
||||
}, []);
|
||||
|
||||
return compact;
|
||||
}
|
||||
|
||||
function StatusPill({ row }: { row: CatalogDatabase }) {
|
||||
const tone = !row.workspaceAvailable
|
||||
? "border-destructive/30 bg-destructive/10 text-destructive"
|
||||
: !row.configured
|
||||
? "border-border bg-muted text-muted-foreground"
|
||||
: row.connectionStatus === "reachable"
|
||||
? "border-emerald-300/70 bg-emerald-50 text-emerald-800 dark:bg-emerald-950/30 dark:text-emerald-300"
|
||||
: row.connectionStatus === "failed"
|
||||
? "border-destructive/30 bg-destructive/10 text-destructive"
|
||||
: "border-amber-300/70 bg-amber-50 text-amber-900 dark:bg-amber-950/30 dark:text-amber-300";
|
||||
|
||||
return <div className="flex items-center gap-1.5"><span className={`inline-flex max-w-full truncate rounded-full border px-2 py-0.5 text-[11px] font-semibold ${tone}`}>{statusLabel(row)}</span>{row.activeSyncRun ? <span className="inline-flex rounded-full border border-primary/30 bg-primary/8 px-2 py-0.5 text-[11px] font-semibold text-primary">Syncing</span> : null}</div>;
|
||||
}
|
||||
|
||||
function DatabaseActionsCell({
|
||||
data,
|
||||
context,
|
||||
}: ICellRendererParams<CatalogDatabase, unknown, DatabaseGridContext>) {
|
||||
if (!data || !context) return null;
|
||||
|
||||
const editDisabled = !context.canManage || !data.workspaceAvailable;
|
||||
const deleteDisabled = !context.canManage || !data.configured;
|
||||
const editLabel = `Edit ${data.workspaceName}`;
|
||||
const editReasonId = `database-edit-reason-${data.workspaceId}`;
|
||||
const deleteReasonId = `database-delete-reason-${data.workspaceId}`;
|
||||
|
||||
return (
|
||||
<div className="flex h-full items-center justify-end gap-0.5" onClick={(event) => event.stopPropagation()}>
|
||||
{data.activeSyncRun ? (
|
||||
<Button type="button" variant="ghost" size="icon-lg" title={`View synchronization for ${data.workspaceName}`} aria-label={`View synchronization for ${data.workspaceName}`} onClick={() => context.onOpenSync(data)}>
|
||||
<RefreshCw className="animate-spin" aria-hidden="true" />
|
||||
</Button>
|
||||
) : null}
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
size="icon-lg"
|
||||
title={`View ${data.workspaceName}`}
|
||||
aria-label={`View ${data.workspaceName}`}
|
||||
onClick={(event) => context.onView(data, event.currentTarget)}
|
||||
>
|
||||
<Eye aria-hidden="true" />
|
||||
</Button>
|
||||
<span title={editDisabled ? "This database configuration cannot be changed" : editLabel}>
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
size="icon-lg"
|
||||
aria-label={editLabel}
|
||||
aria-describedby={editDisabled ? editReasonId : undefined}
|
||||
disabled={editDisabled}
|
||||
onClick={(event) => context.onEdit(data, event.currentTarget)}
|
||||
>
|
||||
<Pencil aria-hidden="true" />
|
||||
</Button>
|
||||
{editDisabled ? <span id={editReasonId} className="sr-only">This database configuration cannot be changed.</span> : null}
|
||||
</span>
|
||||
<span title={deleteDisabled ? "Save this database configuration before deleting it" : `Delete ${data.workspaceName}`}>
|
||||
<Button
|
||||
type="button"
|
||||
variant="ghost"
|
||||
size="icon-lg"
|
||||
className="text-destructive hover:bg-destructive/10 hover:text-destructive"
|
||||
aria-label={`Delete ${data.workspaceName}`}
|
||||
aria-describedby={deleteDisabled ? deleteReasonId : undefined}
|
||||
disabled={deleteDisabled}
|
||||
onClick={(event) => context.onDelete(data, event.currentTarget)}
|
||||
>
|
||||
<Trash2 aria-hidden="true" />
|
||||
</Button>
|
||||
{deleteDisabled ? <span id={deleteReasonId} className="sr-only">Save this database configuration before deleting it.</span> : null}
|
||||
</span>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export function DatabaseGrid({
|
||||
rows,
|
||||
isLoading,
|
||||
isFetching,
|
||||
search,
|
||||
canManage,
|
||||
searchInputRef,
|
||||
onSearchChange,
|
||||
onView,
|
||||
onEdit,
|
||||
onDelete,
|
||||
onOpenSync,
|
||||
onTestSelected,
|
||||
onSyncSelected,
|
||||
}: DatabaseGridProps) {
|
||||
const compact = useCompactViewport();
|
||||
const gridRef = useRef<AgGridReact<CatalogDatabase>>(null);
|
||||
const [selectedRows, setSelectedRows] = useState<CatalogDatabase[]>([]);
|
||||
const [action, setAction] = useState<"test" | "sync" | null>(null);
|
||||
const context = useMemo<DatabaseGridContext>(
|
||||
() => ({ canManage, onView, onEdit, onDelete, onOpenSync }),
|
||||
[canManage, onView, onEdit, onDelete, onOpenSync],
|
||||
);
|
||||
|
||||
const columnDefs = useMemo<ColDef<CatalogDatabase>[]>(() => {
|
||||
const workspace: ColDef<CatalogDatabase> = {
|
||||
field: "workspaceName",
|
||||
headerName: "Workspace",
|
||||
minWidth: compact ? 112 : 190,
|
||||
flex: compact ? undefined : 1.35,
|
||||
width: compact ? 112 : undefined,
|
||||
};
|
||||
const database: ColDef<CatalogDatabase> = {
|
||||
field: "databaseName", headerName: "Database", minWidth: 145, flex: 1,
|
||||
};
|
||||
const schema: ColDef<CatalogDatabase> = {
|
||||
field: "schema", headerName: "Schema", minWidth: 140, flex: 0.9,
|
||||
};
|
||||
const transport: ColDef<CatalogDatabase> = {
|
||||
field: "binding.transport",
|
||||
headerName: "Transport",
|
||||
minWidth: 135,
|
||||
flex: 0.9,
|
||||
valueFormatter: ({ value }) => String(value ?? "").replaceAll("_", " "),
|
||||
};
|
||||
const endpoint: ColDef<CatalogDatabase> = {
|
||||
headerName: "Endpoint",
|
||||
minWidth: 190,
|
||||
flex: 1.2,
|
||||
valueGetter: ({ data }) => {
|
||||
if (!data) return "";
|
||||
if (data.binding.transport === "rest_api") return data.binding.baseUrl ?? "";
|
||||
if (data.binding.transport === "ssh_tunnel") return data.binding.sshHost ?? "";
|
||||
return data.binding.host ? `${data.binding.host}:${data.binding.port ?? 5432}` : "";
|
||||
},
|
||||
};
|
||||
const status: ColDef<CatalogDatabase> = {
|
||||
headerName: "Status",
|
||||
minWidth: compact ? 96 : 130,
|
||||
width: compact ? 96 : undefined,
|
||||
flex: compact ? undefined : 0.8,
|
||||
cellClass: compact ? "thot-database-status-cell" : undefined,
|
||||
valueGetter: ({ data }) => (data ? statusLabel(data) : ""),
|
||||
cellRenderer: ({ data }: ICellRendererParams<CatalogDatabase>) => (
|
||||
data ? <StatusPill row={data} /> : null
|
||||
),
|
||||
};
|
||||
const updated: ColDef<CatalogDatabase> = {
|
||||
field: "updatedAt",
|
||||
headerName: "Updated",
|
||||
minWidth: 170,
|
||||
flex: 0.9,
|
||||
valueFormatter: ({ value }) => value ? new Date(String(value)).toLocaleString() : "",
|
||||
};
|
||||
const actionsWidth = compact ? 148 : 164;
|
||||
const actions: ColDef<CatalogDatabase> = {
|
||||
colId: "actions",
|
||||
headerName: "Actions",
|
||||
width: actionsWidth,
|
||||
minWidth: actionsWidth,
|
||||
maxWidth: actionsWidth,
|
||||
pinned: "right",
|
||||
lockPinned: true,
|
||||
sortable: false,
|
||||
filter: false,
|
||||
resizable: false,
|
||||
suppressMovable: true,
|
||||
suppressHeaderMenuButton: true,
|
||||
cellClass: "thot-database-actions-cell",
|
||||
cellRenderer: DatabaseActionsCell,
|
||||
};
|
||||
|
||||
return compact
|
||||
? [workspace, status, database, schema, transport, endpoint, updated, actions]
|
||||
: [workspace, database, schema, transport, endpoint, status, updated, actions];
|
||||
}, [compact]);
|
||||
|
||||
const configuredCount = rows.filter((row) => row.configured).length;
|
||||
const hiddenSelected = selectedRows.filter((row) => {
|
||||
const term = search.trim().toLocaleLowerCase();
|
||||
return term && ![row.workspaceName, row.databaseName, row.schema].some((value) => value.toLocaleLowerCase().includes(term));
|
||||
}).length;
|
||||
const canTestSelection = canManage && selectedRows.length > 0 && selectedRows.every((row) => row.configured && row.id && !row.activeSyncRun);
|
||||
const canSyncSelection = canManage && selectedRows.length > 0 && selectedRows.every((row) => row.configured && row.id && row.connectionStatus === "reachable" && row.testedVersion === row.version && !row.activeSyncRun);
|
||||
const perform = async (kind: "test" | "sync", operation: () => Promise<void>) => {
|
||||
setAction(kind);
|
||||
try {
|
||||
await operation();
|
||||
gridRef.current?.api.deselectAll();
|
||||
setSelectedRows([]);
|
||||
} finally { setAction(null); }
|
||||
};
|
||||
|
||||
return (
|
||||
<section aria-label="Workspace databases" className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
|
||||
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
|
||||
{selectedRows.length > 0 ? (
|
||||
<>
|
||||
<span className="text-sm font-semibold">{selectedRows.length} selected</span>
|
||||
{hiddenSelected ? <span className="text-xs text-muted-foreground">{hiddenSelected} hidden by filter</span> : null}
|
||||
<Menu.Root>
|
||||
<Menu.Trigger className="inline-flex h-8 items-center justify-center gap-2 rounded-md border border-input bg-background px-3 text-sm font-medium hover:bg-muted disabled:pointer-events-none disabled:opacity-50" disabled={action !== null}>Actions <ChevronDown className="size-4" /></Menu.Trigger>
|
||||
<Menu.Portal>
|
||||
<Menu.Positioner side="bottom" align="start" sideOffset={4}>
|
||||
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
|
||||
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canTestSelection} onClick={() => void perform("test", () => onTestSelected(selectedRows))}>Test connections</Menu.Item>
|
||||
<Menu.Separator className="my-1 h-px bg-border" />
|
||||
{databaseSyncScopes.map(({ scope, label }) => (
|
||||
<Menu.Item
|
||||
key={scope}
|
||||
className={databaseSyncItemClass}
|
||||
disabled={!canSyncSelection}
|
||||
onClick={() => void perform("sync", () => onSyncSelected(selectedRows, scope))}
|
||||
>
|
||||
{label}
|
||||
</Menu.Item>
|
||||
))}
|
||||
</Menu.Popup>
|
||||
</Menu.Positioner>
|
||||
</Menu.Portal>
|
||||
</Menu.Root>
|
||||
<Button type="button" variant="ghost" disabled={action !== null} onClick={() => { gridRef.current?.api.deselectAll(); setSelectedRows([]); }}><X />Clear</Button>
|
||||
</>
|
||||
) : <><span className="thot-label whitespace-nowrap">Workspace databases</span><input
|
||||
ref={searchInputRef}
|
||||
className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none transition focus:border-primary/60 focus:ring-3 focus:ring-ring/15"
|
||||
aria-label="Search databases"
|
||||
placeholder="Search"
|
||||
value={search}
|
||||
onChange={(event) => onSearchChange(event.target.value)}
|
||||
/></>}
|
||||
<span className="whitespace-nowrap text-xs tabular-nums text-muted-foreground">
|
||||
{configuredCount}/{rows.length}
|
||||
</span>
|
||||
{isFetching && !isLoading ? (
|
||||
<span className="text-xs text-muted-foreground" role="status">Refreshing</span>
|
||||
) : null}
|
||||
</div>
|
||||
<div className="relative min-h-[280px] flex-1">
|
||||
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
|
||||
<AgGridReact<CatalogDatabase>
|
||||
ref={gridRef}
|
||||
rowData={rows}
|
||||
columnDefs={columnDefs}
|
||||
context={context}
|
||||
loading={isLoading}
|
||||
quickFilterText={search}
|
||||
defaultColDef={{
|
||||
sortable: true,
|
||||
filter: true,
|
||||
resizable: true,
|
||||
suppressHeaderMenuButton: false,
|
||||
}}
|
||||
getRowId={({ data }) => data.workspaceId}
|
||||
rowSelection={{ mode: "multiRow", selectAll: "filtered", enableClickSelection: false }}
|
||||
selectionColumnDef={{ width: 44, maxWidth: 44, pinned: "left" }}
|
||||
onSelectionChanged={({ api }) => setSelectedRows(api.getSelectedRows())}
|
||||
rowHeight={44}
|
||||
headerHeight={38}
|
||||
animateRows={false}
|
||||
/>
|
||||
</div>
|
||||
{isLoading ? (
|
||||
<div className="pointer-events-none absolute inset-0 grid place-items-center bg-card/80 text-sm text-muted-foreground" role="status">
|
||||
Loading database configurations…
|
||||
</div>
|
||||
) : rows.length === 0 ? (
|
||||
<div className="pointer-events-none absolute inset-0 grid place-items-center bg-card px-6 text-center text-sm text-muted-foreground" role="status">
|
||||
No repository workspaces are available for database configuration.
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
import { useMemo, useState } from "react";
|
||||
import { useQuery } from "@tanstack/react-query";
|
||||
import { AgGridReact } from "ag-grid-react";
|
||||
import type { ColDef } from "ag-grid-community";
|
||||
import { ArrowLeft, RefreshCw } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
import { Button } from "../../components/ui/button";
|
||||
import { apiErrorMessage } from "../../api/client";
|
||||
import {
|
||||
listCatalogRelationships,
|
||||
startCatalogSync,
|
||||
type CatalogDatabase,
|
||||
type CatalogRelationship,
|
||||
type CatalogSyncRun,
|
||||
} from "../../api/catalog-databases";
|
||||
|
||||
interface Props {
|
||||
database: CatalogDatabase;
|
||||
canManage: boolean;
|
||||
onBackToDatabases: () => void;
|
||||
onOpenOverview: () => void;
|
||||
onOpenTables: () => void;
|
||||
onRunStarted: (run: CatalogSyncRun) => void;
|
||||
onOpenSync: () => void;
|
||||
}
|
||||
|
||||
export function DatabaseRelationships({
|
||||
database,
|
||||
canManage,
|
||||
onBackToDatabases,
|
||||
onOpenOverview,
|
||||
onOpenTables,
|
||||
onRunStarted,
|
||||
onOpenSync,
|
||||
}: Props) {
|
||||
const databaseId = database.id!;
|
||||
const { data = [], isLoading, isFetching, refetch } = useQuery({
|
||||
queryKey: ["catalog-relationships", databaseId],
|
||||
queryFn: () => listCatalogRelationships(databaseId),
|
||||
retry: false,
|
||||
});
|
||||
const [search, setSearch] = useState("");
|
||||
const [busy, setBusy] = useState(false);
|
||||
const bindingReady = database.connectionStatus === "reachable" && database.testedVersion === database.version;
|
||||
|
||||
const synchronize = async (scope: "relationships" | "all") => {
|
||||
setBusy(true);
|
||||
try {
|
||||
const run = await startCatalogSync(databaseId, database.version, scope);
|
||||
onRunStarted(run);
|
||||
toast.success(scope === "all" ? "Full schema synchronization started" : "Relationship synchronization started");
|
||||
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(false); }
|
||||
};
|
||||
|
||||
const columns = useMemo<ColDef<CatalogRelationship>[]>(() => [
|
||||
{ field: "constraintName", headerName: "Constraint", minWidth: 220, flex: 1, cellClass: "font-mono text-xs" },
|
||||
{ field: "sourceTableName", headerName: "Source table", minWidth: 200, flex: 1 },
|
||||
{ headerName: "Source columns", minWidth: 200, flex: 1, valueGetter: ({ data: row }) => row?.columns.map((pair) => pair.sourceColumnName).join(", ") ?? "" },
|
||||
{ field: "targetTableName", headerName: "Target table", minWidth: 200, flex: 1 },
|
||||
{ headerName: "Target columns", minWidth: 200, flex: 1, valueGetter: ({ data: row }) => row?.columns.map((pair) => pair.targetColumnName).join(", ") ?? "" },
|
||||
{ field: "updateRule", headerName: "On update", minWidth: 125, width: 125 },
|
||||
{ field: "deleteRule", headerName: "On delete", minWidth: 125, width: 125 },
|
||||
], []);
|
||||
|
||||
return (
|
||||
<section aria-label={`Relationships for ${database.workspaceName}`} className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
|
||||
<div className="px-4 pt-5 sm:px-5">
|
||||
<Button type="button" variant="ghost" className="-ml-2 mb-4" disabled={busy} onClick={onBackToDatabases}><ArrowLeft />Back to databases</Button>
|
||||
<div className="flex flex-wrap items-start justify-between gap-4 border-b border-border pb-5">
|
||||
<div>
|
||||
<p className="thot-label mb-1">Database management</p>
|
||||
<h2 className="font-heading text-2xl font-semibold tracking-tight">{database.workspaceName}</h2>
|
||||
<p className="mt-1 text-sm text-muted-foreground">{database.databaseName} · {database.schema}</p>
|
||||
</div>
|
||||
<div className="flex gap-2">
|
||||
{database.activeSyncRun ? <Button type="button" variant="outline" onClick={onOpenSync}><RefreshCw className="animate-spin" />View sync</Button> : null}
|
||||
<Button type="button" disabled={!canManage || !bindingReady || busy || Boolean(database.activeSyncRun)} onClick={() => void synchronize("all")}><RefreshCw />Sync all</Button>
|
||||
</div>
|
||||
</div>
|
||||
<nav aria-label="Database sections" className="flex gap-1 border-b border-border" role="tablist">
|
||||
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={onOpenOverview}>Overview</button>
|
||||
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={onOpenTables}>Tables</button>
|
||||
<button type="button" role="tab" aria-selected="true" className="border-b-2 border-primary px-3 py-2 text-sm font-semibold text-foreground">Relationships</button>
|
||||
</nav>
|
||||
</div>
|
||||
<div className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
|
||||
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
|
||||
<span className="thot-label whitespace-nowrap">Physical relationships</span>
|
||||
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search relationships" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
|
||||
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
|
||||
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
|
||||
<Button type="button" disabled={!canManage || !bindingReady || busy || Boolean(database.activeSyncRun)} title={!bindingReady ? "Test the current database binding before synchronizing relationships" : undefined} onClick={() => void synchronize("relationships")}><RefreshCw />Sync relationships</Button>
|
||||
</div>
|
||||
{!bindingReady ? <div className="border-b border-border bg-muted/35 px-4 py-3 text-sm text-muted-foreground">Test the current database binding from Overview before synchronizing relationships.</div> : null}
|
||||
<div className="relative min-h-[280px] flex-1">
|
||||
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
|
||||
<AgGridReact<CatalogRelationship> rowData={data} columnDefs={columns} loading={isLoading} quickFilterText={search} defaultColDef={{ sortable: true, filter: true, resizable: true }} getRowId={({ data: row }) => row.id} rowHeight={44} headerHeight={38} animateRows={false} overlayNoRowsTemplate="No physical relationships synchronized for this database." />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import { Menu } from "@base-ui/react/menu";
|
||||
import { ChevronDown, RefreshCw } from "lucide-react";
|
||||
import { buttonVariants } from "../../components/ui/button";
|
||||
import type { CatalogSyncScope } from "../../api/catalog-databases";
|
||||
|
||||
export const databaseSyncItemClass = [
|
||||
"flex cursor-default select-none items-center rounded-md px-3 py-2 text-sm outline-none",
|
||||
"data-[highlighted]:bg-muted data-[disabled]:opacity-45",
|
||||
].join(" ");
|
||||
|
||||
export const databaseSyncScopes: ReadonlyArray<{ scope: CatalogSyncScope; label: string }> = [
|
||||
{ scope: "tables", label: "Synchronize tables" },
|
||||
{ scope: "columns", label: "Synchronize all columns" },
|
||||
{ scope: "relationships", label: "Synchronize relationships" },
|
||||
{ scope: "all", label: "Synchronize all" },
|
||||
];
|
||||
|
||||
interface DatabaseSyncMenuProps {
|
||||
disabled: boolean;
|
||||
disabledReason?: string;
|
||||
onSelect: (scope: CatalogSyncScope) => void;
|
||||
}
|
||||
|
||||
export function DatabaseSyncMenu({ disabled, disabledReason, onSelect }: DatabaseSyncMenuProps) {
|
||||
return (
|
||||
<Menu.Root>
|
||||
<Menu.Trigger
|
||||
type="button"
|
||||
className={buttonVariants()}
|
||||
disabled={disabled}
|
||||
title={disabledReason}
|
||||
aria-label="Synchronize database schema"
|
||||
>
|
||||
<RefreshCw />Synchronize<ChevronDown />
|
||||
</Menu.Trigger>
|
||||
<Menu.Portal>
|
||||
<Menu.Positioner side="bottom" align="end" sideOffset={4}>
|
||||
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
|
||||
{databaseSyncScopes.map(({ scope, label }) => (
|
||||
<Menu.Item
|
||||
key={scope}
|
||||
className={databaseSyncItemClass}
|
||||
disabled={disabled}
|
||||
onClick={() => onSelect(scope)}
|
||||
>
|
||||
{label}
|
||||
</Menu.Item>
|
||||
))}
|
||||
</Menu.Popup>
|
||||
</Menu.Positioner>
|
||||
</Menu.Portal>
|
||||
</Menu.Root>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,315 @@
|
||||
import { useEffect, useMemo, useRef, useState } from "react";
|
||||
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { AgGridReact } from "ag-grid-react";
|
||||
import type { ColDef, ICellRendererParams } from "ag-grid-community";
|
||||
import { ArrowLeft, Columns3, Pencil, RefreshCw, Save, X } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
import { Button } from "../../components/ui/button";
|
||||
import { ApiError, apiErrorMessage } from "../../api/client";
|
||||
import {
|
||||
listCatalogTables,
|
||||
startCatalogSync,
|
||||
updateCatalogTableMetadata,
|
||||
type CatalogDatabase,
|
||||
type CatalogSyncRun,
|
||||
type CatalogTable,
|
||||
} from "../../api/catalog-databases";
|
||||
import type { DatabaseNavigationState } from "./model";
|
||||
import { DatabaseColumns } from "./DatabaseColumns";
|
||||
|
||||
interface Props {
|
||||
database: CatalogDatabase;
|
||||
canManage: boolean;
|
||||
activeRun?: CatalogSyncRun;
|
||||
onBackToDatabases: () => void;
|
||||
onOpenOverview: () => void;
|
||||
onOpenRelationships: () => void;
|
||||
onNavigationStateChange: (state: DatabaseNavigationState) => void;
|
||||
onRunStarted: (run: CatalogSyncRun) => void;
|
||||
onOpenSync: () => void;
|
||||
}
|
||||
|
||||
interface TableGridContext {
|
||||
canManage: boolean;
|
||||
onOpen: (table: CatalogTable, section: "overview" | "columns", origin: HTMLButtonElement) => void;
|
||||
}
|
||||
|
||||
function TableActionsCell({ data, context }: ICellRendererParams<CatalogTable, unknown, TableGridContext>) {
|
||||
if (!data || !context) return null;
|
||||
return (
|
||||
<div className="flex h-full items-center justify-end" onClick={(event) => event.stopPropagation()}>
|
||||
<Button type="button" variant="ghost" size="icon-lg" aria-label={`View columns for ${data.name}`} title={`View columns for ${data.name}`} onClick={(event) => context.onOpen(data, "columns", event.currentTarget)}>
|
||||
<Columns3 aria-hidden="true" />
|
||||
</Button>
|
||||
<Button type="button" variant="ghost" size="icon-lg" aria-label={`Edit description for ${data.name}`} title={`Edit metadata for ${data.name}`} disabled={!context.canManage} onClick={(event) => context.onOpen(data, "overview", event.currentTarget)}>
|
||||
<Pencil aria-hidden="true" />
|
||||
</Button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
export function DatabaseTables({
|
||||
database,
|
||||
canManage,
|
||||
activeRun,
|
||||
onBackToDatabases,
|
||||
onOpenOverview,
|
||||
onOpenRelationships,
|
||||
onNavigationStateChange,
|
||||
onRunStarted,
|
||||
onOpenSync,
|
||||
}: Props) {
|
||||
const databaseId = database.id!;
|
||||
const queryClient = useQueryClient();
|
||||
const queryKey = ["catalog-tables", databaseId] as const;
|
||||
const { data = [], isLoading, isFetching, refetch } = useQuery({ queryKey, queryFn: () => listCatalogTables(databaseId), retry: false });
|
||||
const [search, setSearch] = useState("");
|
||||
const [displayedCount, setDisplayedCount] = useState(0);
|
||||
const [selectedIds, setSelectedIds] = useState<string[]>([]);
|
||||
const [activeTableId, setActiveTableId] = useState<string | null>(null);
|
||||
const [tableSection, setTableSection] = useState<"overview" | "columns">("overview");
|
||||
const [description, setDescription] = useState("");
|
||||
const [generatedDescription, setGeneratedDescription] = useState("");
|
||||
const [baseline, setBaseline] = useState("");
|
||||
const [editorVersion, setEditorVersion] = useState<number | null>(null);
|
||||
const [stale, setStale] = useState(false);
|
||||
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
|
||||
const [busy, setBusy] = useState<"sync" | "save" | null>(null);
|
||||
const [columnNavigation, setColumnNavigation] = useState<DatabaseNavigationState>({ dirty: false, busy: false });
|
||||
const gridRef = useRef<AgGridReact<CatalogTable>>(null);
|
||||
const originRef = useRef<HTMLButtonElement | null>(null);
|
||||
const headingRef = useRef<HTMLHeadingElement>(null);
|
||||
const activeTable = activeTableId ? data.find((table) => table.id === activeTableId) : undefined;
|
||||
const editorFingerprint = JSON.stringify([description, generatedDescription]);
|
||||
const editorDirty = Boolean(activeTableId && editorFingerprint !== baseline);
|
||||
const dirty = tableSection === "columns" ? columnNavigation.dirty : editorDirty;
|
||||
const navigationBusy = busy !== null || (tableSection === "columns" && columnNavigation.busy);
|
||||
const bindingReady = database.connectionStatus === "reachable" && database.testedVersion === database.version;
|
||||
const currentRun = activeRun ?? database.activeSyncRun;
|
||||
|
||||
useEffect(() => { onNavigationStateChange({ dirty, busy: navigationBusy }); }, [dirty, navigationBusy, onNavigationStateChange]);
|
||||
useEffect(() => {
|
||||
if (!activeTableId || !activeTable || editorVersion === activeTable.version || busy) return;
|
||||
setStale(true);
|
||||
setStaleBannerOpen(true);
|
||||
}, [activeTable, activeTableId, busy, editorVersion]);
|
||||
|
||||
const openTable = (table: CatalogTable, section: "overview" | "columns", origin: HTMLButtonElement) => {
|
||||
originRef.current = origin;
|
||||
setActiveTableId(table.id);
|
||||
setTableSection(section);
|
||||
setDescription(table.description ?? "");
|
||||
setGeneratedDescription(table.generatedDescription ?? "");
|
||||
setBaseline(JSON.stringify([table.description ?? "", table.generatedDescription ?? ""]));
|
||||
setEditorVersion(table.version);
|
||||
setStale(false);
|
||||
setStaleBannerOpen(true);
|
||||
setColumnNavigation({ dirty: false, busy: false });
|
||||
};
|
||||
const leaveTable = () => {
|
||||
if (navigationBusy) return;
|
||||
if (dirty && !window.confirm("Discard unsaved metadata?")) return;
|
||||
setActiveTableId(null);
|
||||
setTableSection("overview");
|
||||
window.setTimeout(() => originRef.current?.focus(), 0);
|
||||
};
|
||||
const navigateDatabase = (target: "databases" | "overview" | "relationships") => {
|
||||
if (navigationBusy) return;
|
||||
if (dirty && !window.confirm("Discard unsaved metadata?")) return;
|
||||
if (target === "databases") onBackToDatabases();
|
||||
else if (target === "overview") onOpenOverview();
|
||||
else onOpenRelationships();
|
||||
};
|
||||
const save = async () => {
|
||||
if (!activeTable || editorVersion === null) return;
|
||||
setBusy("save");
|
||||
try {
|
||||
const updated = await updateCatalogTableMetadata(
|
||||
databaseId, activeTable.id, editorVersion,
|
||||
description.trim() || null, generatedDescription.trim() || null,
|
||||
);
|
||||
queryClient.setQueryData<CatalogTable[]>(queryKey, (current = []) => current.map((table) => table.id === updated.id ? updated : table));
|
||||
setDescription(updated.description ?? "");
|
||||
setGeneratedDescription(updated.generatedDescription ?? "");
|
||||
setBaseline(JSON.stringify([updated.description ?? "", updated.generatedDescription ?? ""]));
|
||||
setEditorVersion(updated.version);
|
||||
setStale(false);
|
||||
toast.success("Table metadata saved");
|
||||
} catch (error) {
|
||||
if (error instanceof ApiError && error.code === "table_stale") {
|
||||
await refetch();
|
||||
setStale(true);
|
||||
setStaleBannerOpen(true);
|
||||
} else toast.error(apiErrorMessage(error));
|
||||
} finally { setBusy(null); }
|
||||
};
|
||||
const reloadTable = async () => {
|
||||
if (!activeTableId) return;
|
||||
setBusy("sync");
|
||||
try {
|
||||
const result = await refetch();
|
||||
const latest = result.data?.find((table) => table.id === activeTableId);
|
||||
if (!latest) { leaveTable(); return; }
|
||||
setDescription(latest.description ?? "");
|
||||
setGeneratedDescription(latest.generatedDescription ?? "");
|
||||
setBaseline(JSON.stringify([latest.description ?? "", latest.generatedDescription ?? ""]));
|
||||
setEditorVersion(latest.version);
|
||||
setStale(false);
|
||||
setStaleBannerOpen(true);
|
||||
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(null); }
|
||||
};
|
||||
const synchronize = async (scope: "tables" | "columns" | "all", tableIds: string[] = []) => {
|
||||
setBusy("sync");
|
||||
try {
|
||||
const run = await startCatalogSync(databaseId, database.version, scope, tableIds);
|
||||
onRunStarted(run);
|
||||
gridRef.current?.api.deselectAll();
|
||||
setSelectedIds([]);
|
||||
toast.success(scope === "all" ? "Full schema synchronization started" : scope === "columns" ? "Column synchronization started" : "Table synchronization started");
|
||||
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(null); }
|
||||
};
|
||||
|
||||
const columns = useMemo<ColDef<CatalogTable>[]>(() => [
|
||||
{ field: "name", headerName: "Name", minWidth: 250, flex: 1, cellClass: "font-mono text-xs" },
|
||||
{ field: "sourceComment", headerName: "Source comment", minWidth: 260, flex: 1.25, valueFormatter: ({ value }) => value ?? "" },
|
||||
{ field: "generatedDescription", headerName: "Generated description", minWidth: 280, flex: 1.25, valueFormatter: ({ value }) => value ?? "" },
|
||||
{ field: "description", headerName: "Description", minWidth: 280, flex: 1.25, valueFormatter: ({ value }) => value ?? "" },
|
||||
{ colId: "actions", headerName: "", width: 104, minWidth: 104, maxWidth: 104, pinned: "right", sortable: false, filter: false, resizable: false, cellRenderer: TableActionsCell },
|
||||
], []);
|
||||
const context = useMemo<TableGridContext>(() => ({ canManage, onOpen: openTable }), [canManage, data]);
|
||||
const visibleSelected = useMemo(() => {
|
||||
const term = search.trim().toLocaleLowerCase();
|
||||
if (!term) return selectedIds.length;
|
||||
const selected = new Set(selectedIds);
|
||||
return data.filter((table) => selected.has(table.id) && [table.name, table.sourceComment, table.generatedDescription, table.description]
|
||||
.some((value) => value?.toLocaleLowerCase().includes(term))).length;
|
||||
}, [data, search, selectedIds]);
|
||||
|
||||
const databaseHeader = (
|
||||
<div className="px-4 pt-5 sm:px-5">
|
||||
<Button type="button" variant="ghost" className="-ml-2 mb-4" disabled={navigationBusy} onClick={() => navigateDatabase("databases")}><ArrowLeft />Back to databases</Button>
|
||||
<div className="flex flex-wrap items-start justify-between gap-4 border-b border-border pb-5">
|
||||
<div>
|
||||
<p className="thot-label mb-1">Database management</p>
|
||||
<h2 ref={headingRef} tabIndex={-1} className="font-heading text-2xl font-semibold tracking-tight outline-none">{database.workspaceName}</h2>
|
||||
<p className="mt-1 text-sm text-muted-foreground">{database.databaseName} · {database.schema}</p>
|
||||
</div>
|
||||
<div className="flex gap-2">
|
||||
{currentRun ? <Button type="button" variant="outline" onClick={onOpenSync}><RefreshCw className="animate-spin" />View sync</Button> : null}
|
||||
<Button type="button" disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)} onClick={() => void synchronize("all")}><RefreshCw />Sync all</Button>
|
||||
</div>
|
||||
</div>
|
||||
<nav aria-label="Database sections" className="flex gap-1 border-b border-border" role="tablist">
|
||||
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={() => navigateDatabase("overview")}>Overview</button>
|
||||
<button type="button" role="tab" aria-selected="true" className="border-b-2 border-primary px-3 py-2 text-sm font-semibold text-foreground">Tables</button>
|
||||
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={() => navigateDatabase("relationships")}>Relationships</button>
|
||||
</nav>
|
||||
</div>
|
||||
);
|
||||
|
||||
if (activeTable) {
|
||||
return (
|
||||
<section aria-label={`Table ${activeTable.name}`} className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
|
||||
{databaseHeader}
|
||||
<div className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
|
||||
<div className="border-b border-border px-4 pt-4">
|
||||
<Button type="button" variant="ghost" className="-ml-2 mb-2" onClick={leaveTable}><ArrowLeft />Back to tables</Button>
|
||||
<p className="thot-label">Physical table</p>
|
||||
<h3 className="mt-1 font-heading text-xl font-semibold">{activeTable.name}</h3>
|
||||
<nav aria-label="Table sections" className="mt-3 flex gap-1" role="tablist">
|
||||
<button type="button" role="tab" aria-selected={tableSection === "overview"} className={`border-b-2 px-3 py-2 text-sm font-semibold ${tableSection === "overview" ? "border-primary text-foreground" : "border-transparent text-muted-foreground"}`} onClick={() => setTableSection("overview")}>Overview</button>
|
||||
<button type="button" role="tab" aria-selected={tableSection === "columns"} className={`border-b-2 px-3 py-2 text-sm font-semibold ${tableSection === "columns" ? "border-primary text-foreground" : "border-transparent text-muted-foreground"}`} onClick={() => setTableSection("columns")}>Columns</button>
|
||||
</nav>
|
||||
</div>
|
||||
{tableSection === "columns" ? (
|
||||
<DatabaseColumns databaseId={databaseId} table={activeTable} canManage={canManage} onNavigationStateChange={setColumnNavigation} onSync={() => void synchronize("columns", [activeTable.id])} />
|
||||
) : (
|
||||
<div className="min-h-0 overflow-y-auto px-4 py-5">
|
||||
<p className="text-sm text-muted-foreground">Only review metadata can be changed.</p>
|
||||
{stale ? (
|
||||
staleBannerOpen ? (
|
||||
<div className="mt-4 rounded-md border border-amber-500/50 bg-amber-500/8 p-4 text-sm">
|
||||
<p className="font-semibold">A newer table description is available.</p>
|
||||
<p className="mt-1 text-muted-foreground">Your draft is preserved until you reload the current metadata.</p>
|
||||
<div className="mt-3 flex gap-2"><Button type="button" variant="outline" onClick={() => void reloadTable()}>Reload latest</Button><Button type="button" variant="ghost" onClick={() => setStaleBannerOpen(false)}>Keep editing</Button></div>
|
||||
</div>
|
||||
) : <div className="mt-4 flex items-center justify-between gap-3 rounded-md border border-amber-500/40 bg-amber-500/8 px-4 py-3 text-sm"><span>Reload the latest value before this description can be saved.</span><Button type="button" variant="outline" onClick={() => void reloadTable()}>Reload latest</Button></div>
|
||||
) : null}
|
||||
<div className="mt-5 grid gap-4">
|
||||
<label className="grid gap-1.5 text-sm font-semibold">Physical table name<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={activeTable.name} readOnly /></label>
|
||||
<label className="grid gap-1.5 text-sm font-semibold">Source comment<textarea className="min-h-24 rounded-md border border-input bg-muted/35 px-3 py-2 text-sm" value={activeTable.sourceComment ?? ""} readOnly /></label>
|
||||
<label className="grid gap-1.5 text-sm font-semibold">Generated description<textarea className="min-h-32 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={generatedDescription} disabled={!canManage || busy !== null} onChange={(event) => setGeneratedDescription(event.target.value)} /></label>
|
||||
<label className="grid gap-1.5 text-sm font-semibold">Description<textarea className="min-h-36 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={description} disabled={!canManage || busy !== null} onChange={(event) => setDescription(event.target.value)} /><span className="font-normal text-muted-foreground">Leave blank to keep the curated description empty.</span></label>
|
||||
</div>
|
||||
<div className="mt-5 flex justify-end gap-2 border-t border-border pt-4">
|
||||
<Button type="button" variant="outline" disabled={busy !== null} onClick={leaveTable}>Cancel</Button>
|
||||
<Button type="button" disabled={!canManage || !editorDirty || busy !== null || stale} onClick={() => void save()}><Save />{busy === "save" ? "Saving…" : "Save metadata"}</Button>
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<section aria-label={`Tables for ${database.workspaceName}`} className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
|
||||
{databaseHeader}
|
||||
<div className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
|
||||
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
|
||||
{selectedIds.length > 0 ? (
|
||||
<>
|
||||
<span className="text-sm font-semibold">{selectedIds.length} selected</span>
|
||||
{visibleSelected !== selectedIds.length ? <span className="text-xs text-muted-foreground">{selectedIds.length - visibleSelected} hidden by filter</span> : null}
|
||||
<Button
|
||||
type="button"
|
||||
variant="outline"
|
||||
disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)}
|
||||
title={!bindingReady ? "Test the current database binding before synchronizing columns" : undefined}
|
||||
onClick={() => void synchronize("columns", selectedIds)}
|
||||
>
|
||||
<RefreshCw className={busy === "sync" ? "animate-spin" : ""} />Synchronize columns
|
||||
</Button>
|
||||
<Button type="button" variant="ghost" onClick={() => { gridRef.current?.api.deselectAll(); setSelectedIds([]); }}><X />Clear</Button>
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<span className="thot-label whitespace-nowrap">Catalog tables</span>
|
||||
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search tables" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
|
||||
<span
|
||||
aria-live="polite"
|
||||
className="whitespace-nowrap text-xs tabular-nums text-muted-foreground"
|
||||
>
|
||||
{search.trim() ? `${displayedCount} of ${data.length}` : displayedCount}
|
||||
</span>
|
||||
<Button type="button" variant="outline" disabled={isFetching || busy !== null} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
|
||||
<Button type="button" disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)} title={!bindingReady ? "Test the current database binding before synchronizing tables" : undefined} onClick={() => void synchronize("tables")}><RefreshCw className={busy === "sync" ? "animate-spin" : ""} />Sync tables</Button>
|
||||
</>
|
||||
)}
|
||||
</div>
|
||||
{!bindingReady ? <div className="border-b border-border bg-muted/35 px-4 py-3 text-sm text-muted-foreground">Test the current database binding from Overview before synchronizing tables.</div> : null}
|
||||
<div className="relative min-h-[280px] flex-1">
|
||||
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
|
||||
<AgGridReact<CatalogTable>
|
||||
ref={gridRef}
|
||||
rowData={data}
|
||||
columnDefs={columns}
|
||||
context={context}
|
||||
loading={isLoading}
|
||||
quickFilterText={search}
|
||||
defaultColDef={{ sortable: true, filter: true, resizable: true }}
|
||||
getRowId={({ data: table }) => table.id}
|
||||
rowSelection={{ mode: "multiRow", selectAll: "filtered", enableClickSelection: false }}
|
||||
onSelectionChanged={({ api }) => setSelectedIds(api.getSelectedRows().map((table) => table.id))}
|
||||
onModelUpdated={({ api }) => setDisplayedCount(api.getDisplayedRowCount())}
|
||||
rowHeight={44}
|
||||
headerHeight={38}
|
||||
animateRows={false}
|
||||
overlayNoRowsTemplate="No catalog tables. Test the binding, then sync tables."
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,120 @@
|
||||
import type {
|
||||
CatalogDatabase,
|
||||
CatalogSecretName,
|
||||
DatabaseConfiguration,
|
||||
} from "../../api/catalog-databases";
|
||||
|
||||
export type DatabaseFormMode = "add" | "view" | "edit" | "delete";
|
||||
|
||||
export type DatabaseScreen =
|
||||
| { kind: "list" }
|
||||
| { kind: "tables"; workspaceId: string }
|
||||
| { kind: "relationships"; workspaceId: string }
|
||||
| {
|
||||
kind: DatabaseFormMode;
|
||||
workspaceId: string;
|
||||
workspaceLocked?: boolean;
|
||||
};
|
||||
|
||||
export type DatabaseBusyAction = "save" | "test" | "delete" | "retry-secrets" | "reload" | null;
|
||||
|
||||
export type SecretDraft = Partial<Record<CatalogSecretName, string>>;
|
||||
|
||||
export interface DatabaseFormDraft extends DatabaseConfiguration {
|
||||
secrets: SecretDraft;
|
||||
}
|
||||
|
||||
export interface DatabaseNavigationState {
|
||||
dirty: boolean;
|
||||
busy: boolean;
|
||||
}
|
||||
|
||||
export function draftFrom(row: CatalogDatabase): DatabaseFormDraft {
|
||||
return {
|
||||
workspaceId: row.workspaceId,
|
||||
engine: "postgres",
|
||||
databaseName: row.databaseName,
|
||||
schema: row.schema,
|
||||
binding: {
|
||||
port: 5432,
|
||||
restPath: "/health",
|
||||
restAuth: "x-api-key",
|
||||
sshPort: 22,
|
||||
sshTargetPort: 5432,
|
||||
...row.binding,
|
||||
},
|
||||
secrets: {},
|
||||
};
|
||||
}
|
||||
|
||||
export function configurationFromDraft(draft: DatabaseFormDraft): DatabaseConfiguration {
|
||||
const binding = draft.binding.transport === "postgres_direct"
|
||||
? {
|
||||
transport: draft.binding.transport,
|
||||
host: draft.binding.host,
|
||||
port: draft.binding.port,
|
||||
username: draft.binding.username,
|
||||
tlsServername: draft.binding.tlsServername,
|
||||
}
|
||||
: draft.binding.transport === "rest_api"
|
||||
? {
|
||||
transport: draft.binding.transport,
|
||||
baseUrl: draft.binding.baseUrl,
|
||||
restPath: draft.binding.restPath,
|
||||
restAuth: draft.binding.restAuth,
|
||||
}
|
||||
: {
|
||||
transport: draft.binding.transport,
|
||||
username: draft.binding.username,
|
||||
tlsServername: draft.binding.tlsServername,
|
||||
sshHost: draft.binding.sshHost,
|
||||
sshPort: draft.binding.sshPort,
|
||||
sshUsername: draft.binding.sshUsername,
|
||||
sshTargetHost: draft.binding.sshTargetHost,
|
||||
sshTargetPort: draft.binding.sshTargetPort,
|
||||
};
|
||||
|
||||
return {
|
||||
workspaceId: draft.workspaceId,
|
||||
engine: draft.engine,
|
||||
databaseName: draft.databaseName,
|
||||
schema: draft.schema,
|
||||
binding,
|
||||
};
|
||||
}
|
||||
|
||||
export function configurationFingerprint(draft: DatabaseFormDraft): string {
|
||||
return JSON.stringify(configurationFromDraft(draft));
|
||||
}
|
||||
|
||||
export function hasSecretChanges(draft: DatabaseFormDraft | null): boolean {
|
||||
return Boolean(draft && Object.keys(secretReplacements(draft)).length);
|
||||
}
|
||||
|
||||
export function secretReplacements(draft: DatabaseFormDraft): SecretDraft {
|
||||
const allowed: CatalogSecretName[] = draft.binding.transport === "postgres_direct"
|
||||
? ["password", "tlsCa"]
|
||||
: draft.binding.transport === "rest_api"
|
||||
? draft.binding.restAuth === "none" ? [] : ["apiKey"]
|
||||
: [
|
||||
"password",
|
||||
"sshPrivateKey",
|
||||
"sshPrivateKeyPassphrase",
|
||||
"sshKnownHosts",
|
||||
"tlsCa",
|
||||
];
|
||||
|
||||
return Object.fromEntries(
|
||||
Object.entries(draft.secrets).filter(([name, value]) => (
|
||||
allowed.includes(name as CatalogSecretName) && Boolean(value)
|
||||
)),
|
||||
) as SecretDraft;
|
||||
}
|
||||
|
||||
export function statusLabel(row: CatalogDatabase): string {
|
||||
if (!row.workspaceAvailable) return "Workspace missing";
|
||||
if (!row.configured) return "Not configured";
|
||||
if (row.connectionStatus === "reachable") return "Reachable";
|
||||
if (row.connectionStatus === "failed") return "Failed";
|
||||
return "Untested";
|
||||
}
|
||||
@@ -6,8 +6,9 @@ export const server = setupServer(
|
||||
issuer: "mock", subject: "test-user", displayName: "Test user", roles: ["admin"],
|
||||
permissions: [
|
||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
||||
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||
], isAdmin: true, csrfToken: "c".repeat(43), session: null,
|
||||
})),
|
||||
http.get("/api/health/dwh", () => HttpResponse.json({ ok: true, detail: "ok" })),
|
||||
http.get("/api/catalog/databases", () => HttpResponse.json([])),
|
||||
);
|
||||
|
||||
+10
-2
@@ -21,5 +21,13 @@ if [[ "${THOTH_ENABLE_EMBEDDING_GPU:-0}" == "1" ]]; then
|
||||
compose_files+=(-f "$ROOT/deploy/compose.embedding-gpu.yaml")
|
||||
fi
|
||||
|
||||
exec docker compose --env-file "$LOCAL_ENV_FILE" \
|
||||
"${compose_files[@]}" up --build "$@"
|
||||
compose=(docker compose --env-file "$LOCAL_ENV_FILE" "${compose_files[@]}")
|
||||
|
||||
# Migrations are an explicit one-shot operation, never hidden in backend startup. The local
|
||||
# launcher runs that operation before bringing the foreground stack up so a fresh checkout is
|
||||
# immediately usable while production operators can invoke the same service during rollout.
|
||||
"${compose[@]}" build core
|
||||
"${compose[@]}" up -d catalog-db
|
||||
"${compose[@]}" run --rm catalog-migrate
|
||||
|
||||
exec "${compose[@]}" up --build "$@"
|
||||
|
||||
@@ -64,7 +64,7 @@ for profile in local server; do
|
||||
const fs = require("fs");
|
||||
const [path, profile] = process.argv.slice(2);
|
||||
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||
const expected = "core,embedding,embedding-model-init,frontend,qdrant";
|
||||
const expected = "catalog-db,core,embedding,embedding-model-init,frontend,qdrant";
|
||||
if (Object.keys(config.services).sort().join(",") !== expected) {
|
||||
throw new Error(profile + ": install stack must be exactly " + expected);
|
||||
}
|
||||
|
||||
@@ -6,6 +6,9 @@ cd "$(dirname "$0")/.."
|
||||
test -f .env.example
|
||||
test -f deploy/env/local.env.example
|
||||
test -f deploy/env/server.env.example
|
||||
test -f docker/catalog-db-init.sql
|
||||
grep -q "pg_read_file('/run/secrets/catalog_runtime_password')" docker/catalog-db-init.sql
|
||||
grep -q "CREATE ROLE thothii_catalog_runtime" docker/catalog-db-init.sql
|
||||
|
||||
rendered=$(mktemp)
|
||||
trap 'rm -f "$rendered"' EXIT HUP INT TERM
|
||||
@@ -18,7 +21,7 @@ const fs = require("fs");
|
||||
|
||||
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
|
||||
const services = Object.keys(config.services).sort();
|
||||
if (services.join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
if (services.join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
throw new Error(`unexpected service set: ${services.join(",")}`);
|
||||
}
|
||||
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
|
||||
@@ -26,6 +29,7 @@ if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify
|
||||
}
|
||||
const expectedVolumes = [
|
||||
"auth-state",
|
||||
"catalog-data",
|
||||
"embedding-models",
|
||||
"pi-state",
|
||||
"qdrant-data",
|
||||
@@ -43,12 +47,19 @@ const frontend = config.services.frontend;
|
||||
const qdrant = config.services.qdrant;
|
||||
const embedding = config.services.embedding;
|
||||
const modelInit = config.services["embedding-model-init"];
|
||||
const catalog = config.services["catalog-db"];
|
||||
if (!frontend.ports?.some((port) => port.host_ip === "127.0.0.1")) {
|
||||
throw new Error("local frontend must publish a loopback port");
|
||||
}
|
||||
for (const service of [embedding, modelInit]) {
|
||||
if ((service.ports || []).length !== 0) throw new Error("private semantic services must not publish host ports");
|
||||
}
|
||||
if ((catalog.ports || []).length !== 0) throw new Error("catalog database must not publish host ports");
|
||||
if (!catalog.healthcheck) throw new Error("catalog database must define a healthcheck");
|
||||
const catalogHealthcheck = JSON.stringify(catalog.healthcheck.test || []);
|
||||
if (!catalogHealthcheck.includes("pg_isready") || !catalogHealthcheck.includes("thothii_catalog_runtime")) {
|
||||
throw new Error("catalog database healthcheck must verify readiness and the runtime role");
|
||||
}
|
||||
if (!qdrant.ports?.some((port) => port.host_ip === "127.0.0.1" && Number(port.target) === 6333)) {
|
||||
throw new Error("local Qdrant dashboard must publish only its loopback port");
|
||||
}
|
||||
@@ -65,6 +76,9 @@ if (embedding.image !== "ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279
|
||||
if (modelInit.image !== "ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a") {
|
||||
throw new Error("embedding-model-init image must be pinned by version and digest");
|
||||
}
|
||||
if (catalog.image !== "postgres:17.6-bookworm@sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3") {
|
||||
throw new Error("catalog PostgreSQL image must be pinned by version and digest");
|
||||
}
|
||||
const env = core.environment || {};
|
||||
for (const [key, value] of Object.entries({
|
||||
THT_WORKSPACE_INSTALLATION_ID: "local",
|
||||
@@ -74,6 +88,10 @@ for (const [key, value] of Object.entries({
|
||||
THT_INTERNAL_EMBEDDING_URL: "http://embedding:11434",
|
||||
THT_INTERNAL_EMBEDDING_MODEL: "qwen3-embedding:0.6b",
|
||||
THT_INTERNAL_EMBEDDING_DIMENSIONS: "1024",
|
||||
THT_CATALOG_DB_HOST: "catalog-db",
|
||||
THT_CATALOG_DB_NAME: "thothii_catalog",
|
||||
THT_CATALOG_RUNTIME_USER: "thothii_catalog_runtime",
|
||||
THT_CATALOG_RUNTIME_PASSWORD_FILE: "/run/secrets/catalog_runtime_password",
|
||||
})) {
|
||||
if (env[key] !== value) throw new Error(`unexpected core ${key}: ${env[key]}`);
|
||||
}
|
||||
@@ -98,6 +116,9 @@ const depends = core.depends_on || {};
|
||||
if (depends.qdrant?.condition !== "service_healthy") {
|
||||
throw new Error("core must wait for qdrant health");
|
||||
}
|
||||
if (depends["catalog-db"]?.condition !== "service_healthy") {
|
||||
throw new Error("core must wait for catalog database health");
|
||||
}
|
||||
if (depends["embedding-model-init"]?.condition !== "service_completed_successfully") {
|
||||
throw new Error("core must wait for embedding-model-init success");
|
||||
}
|
||||
|
||||
@@ -25,14 +25,14 @@ const fs = require("fs");
|
||||
const [configPath, profile] = process.argv.slice(2);
|
||||
const config = JSON.parse(fs.readFileSync(configPath, "utf8"));
|
||||
const services = Object.keys(config.services).sort();
|
||||
if (services.join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
throw new Error("mandatory stack must include core, frontend, qdrant, embedding, and embedding-model-init");
|
||||
if (services.join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
throw new Error("mandatory stack must include catalog-db, core, frontend, qdrant, embedding, and embedding-model-init");
|
||||
}
|
||||
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
|
||||
throw new Error("forbidden application coupling");
|
||||
}
|
||||
if (!config.networks || !config.networks.thothii) throw new Error("base stack must define the thothii network");
|
||||
for (const volume of ["qdrant-data", "embedding-models"]) {
|
||||
for (const volume of ["catalog-data", "qdrant-data", "embedding-models"]) {
|
||||
if (!config.volumes || !config.volumes[volume]) throw new Error(`missing required volume: ${volume}`);
|
||||
}
|
||||
if (profile === "local") {
|
||||
@@ -114,8 +114,8 @@ const bundleSecrets = runtimeSecrets.filter(
|
||||
if (bundleSecrets.length !== 1) {
|
||||
throw new Error("core must receive exactly one canonical runtime secret bundle");
|
||||
}
|
||||
if (profile === "local" && runtimeSecrets.length !== 1) {
|
||||
throw new Error("local core must receive only the canonical runtime secret bundle");
|
||||
if (profile === "local" && runtimeSecrets.length !== 2) {
|
||||
throw new Error("local core must receive only its runtime bundle and catalog password");
|
||||
}
|
||||
if (profile === "server") {
|
||||
const targets = new Set(runtimeSecrets.map((secret) => secret.target));
|
||||
@@ -171,14 +171,14 @@ const fs = require("fs");
|
||||
|
||||
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
|
||||
const services = Object.keys(config.services).sort();
|
||||
if (services.join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
throw new Error("mandatory stack must include core, frontend, qdrant, embedding, and embedding-model-init");
|
||||
if (services.join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
throw new Error("mandatory stack must include catalog-db, core, frontend, qdrant, embedding, and embedding-model-init");
|
||||
}
|
||||
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
|
||||
throw new Error("forbidden application coupling");
|
||||
}
|
||||
if (!config.networks || !config.networks.thothii) throw new Error("base stack must define the thothii network");
|
||||
for (const volume of ["settings", "pi-state", "workspace-registry", "sessions", "qdrant-data", "embedding-models"]) {
|
||||
for (const volume of ["settings", "pi-state", "workspace-registry", "sessions", "catalog-data", "qdrant-data", "embedding-models"]) {
|
||||
if (!config.volumes || !config.volumes[volume]) throw new Error(`missing required volume: ${volume}`);
|
||||
}
|
||||
if (!Object.hasOwn(config.services.core.environment || {}, "THT_LLM_URL")) {
|
||||
@@ -222,10 +222,11 @@ if (config.services.core.environment?.THT_SECRETS_FILE !== "/run/secrets/thothii
|
||||
throw new Error("core must read the canonical runtime secret bundle from /run/secrets");
|
||||
}
|
||||
const runtimeSecrets = config.services.core.secrets || [];
|
||||
if (runtimeSecrets.length !== 1
|
||||
|| runtimeSecrets[0].source !== "thothii_secrets"
|
||||
|| runtimeSecrets[0].target !== "thothii.secrets") {
|
||||
throw new Error("core must receive exactly the canonical runtime secret bundle");
|
||||
const hasTarget = (name) => runtimeSecrets.some((secret) => secret.target === name || secret.target?.endsWith(`/${name}`));
|
||||
if (runtimeSecrets.length !== 2
|
||||
|| !hasTarget("thothii.secrets")
|
||||
|| !hasTarget("catalog_runtime_password")) {
|
||||
throw new Error("core must receive only the canonical runtime bundle and catalog password");
|
||||
}
|
||||
if ((config.services.frontend.secrets || []).length !== 0) {
|
||||
throw new Error("frontend must not receive runtime secrets");
|
||||
|
||||
@@ -275,8 +275,8 @@ overrides:
|
||||
)
|
||||
$render = Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("config", "--services")) -Label "render Windows Compose from spaced path"
|
||||
$services = @($render.StdOut -split "`r?`n" | Where-Object { $_ } | Sort-Object)
|
||||
if (($services -join ",") -ne "core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
throw "rendered Windows stack must contain the canonical five services"
|
||||
if (($services -join ",") -ne "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
throw "rendered Windows stack must contain the canonical six services"
|
||||
}
|
||||
Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("config", "--quiet")) -Label "validate Windows Compose from spaced path" | Out-Null
|
||||
|
||||
@@ -287,8 +287,8 @@ overrides:
|
||||
Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("up", "--detach", "--wait", "--wait-timeout", "180")) -Label "start canonical Windows stack" -TimeoutSeconds 300 | Out-Null
|
||||
$running = Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("ps", "--status", "running", "--services")) -Label "inspect running Windows services"
|
||||
$runningServices = @($running.StdOut -split "`r?`n" | Where-Object { $_ } | Sort-Object)
|
||||
if (($runningServices -join ",") -ne "core,embedding,frontend,qdrant") {
|
||||
throw "bounded Windows startup did not leave the four long-running services ready"
|
||||
if (($runningServices -join ",") -ne "catalog-db,core,embedding,frontend,qdrant") {
|
||||
throw "bounded Windows startup did not leave the five long-running services ready"
|
||||
}
|
||||
Invoke-BoundedNative -FilePath $tht -Arguments @("--installation", $installation, "status") -Label "invoke installation-aware Windows tht in spaced path" | Out-Null
|
||||
}
|
||||
|
||||
@@ -1945,7 +1945,7 @@ verify_local_installation_example() {
|
||||
const fs = require("fs");
|
||||
const [path, authConfigRoot] = process.argv.slice(2);
|
||||
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
if (Object.keys(config.services).sort().join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
throw new Error("local installation example must render the internal semantic stack");
|
||||
}
|
||||
const authMount = (config.services.core.volumes || []).find(
|
||||
@@ -2065,7 +2065,7 @@ verify_server_installation_example() {
|
||||
const fs = require("fs");
|
||||
const [path, authConfigRoot] = process.argv.slice(2);
|
||||
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
if (Object.keys(config.services).sort().join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
throw new Error("server installation example must render the internal semantic stack");
|
||||
}
|
||||
const core = config.services.core;
|
||||
@@ -2216,7 +2216,7 @@ verify_compose_fixtures() {
|
||||
const fs = require("fs");
|
||||
const [path, profile, authConfigRoot] = process.argv.slice(2);
|
||||
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
if (Object.keys(config.services).sort().join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||
throw new Error(profile + ": mandatory stack must include the internal semantic services");
|
||||
}
|
||||
const core = config.services.core;
|
||||
|
||||
Reference in New Issue
Block a user