feat: implement metadata catalog database management
This commit is contained in:
@@ -1,5 +1,19 @@
|
|||||||
# AGENTS.md
|
# AGENTS.md
|
||||||
|
|
||||||
|
## Agent skills
|
||||||
|
|
||||||
|
### Issue tracker
|
||||||
|
|
||||||
|
Issues for this repository live in the self-hosted Gitea repository at `https://git.tylconsulting.it/mptyl/ThothII`; use its web UI or authenticated Gitea API. See `docs/agents/issue-tracker.md`.
|
||||||
|
|
||||||
|
### Triage labels
|
||||||
|
|
||||||
|
Use the canonical labels `needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, and `wontfix`. See `docs/agents/triage-labels.md`.
|
||||||
|
|
||||||
|
### Domain docs
|
||||||
|
|
||||||
|
This is a single-context repository with root `CONTEXT.md` and `docs/adr/`. See `docs/agents/domain.md`.
|
||||||
|
|
||||||
This file provides guidance to Codex (Codex.ai/code) when working with code in this repository.
|
This file provides guidance to Codex (Codex.ai/code) when working with code in this repository.
|
||||||
|
|
||||||
## Start here
|
## Start here
|
||||||
|
|||||||
+73
-11
@@ -246,10 +246,23 @@ precedente o di un altro workspace.
|
|||||||
|
|
||||||
## Catalogo dei metadati
|
## Catalogo dei metadati
|
||||||
|
|
||||||
**Workspace Database** — Il database associato in modo uno-a-uno a un workspace,
|
**Workspace Database** — Il database che appartiene a un solo workspace e non può essere
|
||||||
considerato nella sua interezza fisica: tutte le tabelle, le colonne e le relazioni
|
condiviso con altri workspace; un workspace può averne al massimo uno. È considerato nella
|
||||||
disponibili. La sua struttura fisica viene acquisita interrogando il database; il
|
coppia composta dal database PostgreSQL e da un solo schema: tutte le tabelle, le colonne e
|
||||||
Metadata Catalog non crea né possiede l'identità del workspace.
|
le relazioni catalogate appartengono a quello schema. Il Metadata Catalog conserva
|
||||||
|
l'associazione, ma non crea né possiede l'identità del workspace.
|
||||||
|
|
||||||
|
**Database Binding** — La configurazione specifica di un'installazione che seleziona un
|
||||||
|
trasporto e fornisce i riferimenti necessari a raggiungere un Workspace Database. Non è una
|
||||||
|
seconda identità del database e non viene condivisa automaticamente fra installazioni.
|
||||||
|
|
||||||
|
**Thoth REST Connector** — Il trasporto REST tipizzato con cui ThothII interroga ed
|
||||||
|
introspeziona un Workspace Database attraverso il contratto RPC DWH supportato. Non è un
|
||||||
|
client configurabile per API REST arbitrarie.
|
||||||
|
|
||||||
|
**Orphaned Workspace Database** — Un Workspace Database il cui workspace non è più presente
|
||||||
|
nel catalogo autorevole. Rimane conservato per il recupero amministrativo, ma non può essere
|
||||||
|
usato dal workflow finché non viene riassegnato a un workspace esistente.
|
||||||
|
|
||||||
**Metadata Catalog** — Il contesto amministrativo che raccoglie e cura i metadati di un
|
**Metadata Catalog** — Il contesto amministrativo che raccoglie e cura i metadati di un
|
||||||
Workspace Database. Non definisce quali elementi partecipano al workflow SQL.
|
Workspace Database. Non definisce quali elementi partecipano al workflow SQL.
|
||||||
@@ -257,12 +270,61 @@ Workspace Database. Non definisce quali elementi partecipano al workflow SQL.
|
|||||||
**Database Profile** — L'insieme curato di scope, descrizioni e metadati semantici
|
**Database Profile** — L'insieme curato di scope, descrizioni e metadati semantici
|
||||||
associato a un Workspace Database.
|
associato a un Workspace Database.
|
||||||
|
|
||||||
**Physical Schema Snapshot** — L'inventario della struttura fisica osservata in un
|
**Physical Table** — Una tabella osservata nello schema esterno di un Workspace Database.
|
||||||
Workspace Database durante una specifica introspezione. Non è un progetto dello schema
|
La sua identità e il suo nome appartengono al database esterno, non al Metadata Catalog.
|
||||||
né un'autorizzazione a modificarne la struttura.
|
|
||||||
|
|
||||||
**AI Proposal** — Un contenuto generato con l'ausilio dell'AI che non è ancora stato
|
**Catalog Table** — La rappresentazione persistita di una Physical Table nel Metadata Catalog.
|
||||||
approvato come contenuto canonico.
|
La sua appartenenza e identità fisica derivano esclusivamente dall'introspezione; soltanto i suoi
|
||||||
|
Catalog Metadata possono essere curati amministrativamente.
|
||||||
|
_Avoid_: SqlTable, managed table
|
||||||
|
|
||||||
**Publication** — Una versione approvata e immutabile dei contenuti del Metadata
|
**Physical Column** — Una colonna osservata in una Physical Table, inclusi nome, posizione,
|
||||||
Catalog resa disponibile ai suoi consumatori.
|
tipo e appartenenza a chiavi dichiarate. La sua identità e i suoi fatti strutturali appartengono
|
||||||
|
al database esterno.
|
||||||
|
|
||||||
|
**Catalog Column** — La rappresentazione persistita di una Physical Column nel Metadata Catalog.
|
||||||
|
I fatti osservati sono governati dalla sincronizzazione; Description e Generated Description
|
||||||
|
sono metadati amministrativi modificabili.
|
||||||
|
_Avoid_: SqlColumn, managed column
|
||||||
|
|
||||||
|
**Physical Relationship** — Un vincolo foreign key dichiarato nel database esterno. La sua
|
||||||
|
identità comprende il vincolo e la sequenza ordinata delle coppie di colonne che lo compongono.
|
||||||
|
|
||||||
|
**Catalog Relationship** — La rappresentazione persistita di una Physical Relationship nel
|
||||||
|
Metadata Catalog. È governata esclusivamente dall'introspezione e non è creata o modificata
|
||||||
|
manualmente.
|
||||||
|
_Avoid_: denormalized FK, relationship string
|
||||||
|
|
||||||
|
**Logical Relationship** — Una relazione semantica curata o inferita che non corrisponde
|
||||||
|
necessariamente a un vincolo fisico. Ha ownership e lifecycle distinti da Catalog Relationship.
|
||||||
|
|
||||||
|
**Description** — Il testo curato e consolidato che descrive una Catalog Table o Catalog Column
|
||||||
|
per gli usi downstream.
|
||||||
|
|
||||||
|
**Generated Description** — Una proposta modificabile sottoposta a revisione umana prima di
|
||||||
|
essere consolidata come Description. Rimane distinta dal commento osservato nel database.
|
||||||
|
_Avoid_: generated comment, source comment
|
||||||
|
|
||||||
|
**Table Synchronization** — La riconciliazione esplicita che rende le Catalog Table di un
|
||||||
|
Workspace Database uguali alle Physical Table osservate: crea quelle nuove, aggiorna i metadati
|
||||||
|
di origine ed elimina definitivamente quelle assenti. Non modifica mai il database esterno.
|
||||||
|
_Avoid_: table import
|
||||||
|
|
||||||
|
**Schema Synchronization** — La riconciliazione esplicita e autorevole di tabelle, colonne e
|
||||||
|
Catalog Relationship di un Workspace Database. Può operare su uno scope specifico oppure su
|
||||||
|
un unico snapshot completo tramite Synchronize All.
|
||||||
|
|
||||||
|
**Catalog Sync Run** — L'esecuzione durevole in background di una Schema Synchronization, con
|
||||||
|
scope, stato, avanzamento e log propri. Al massimo un run per Workspace Database può essere attivo.
|
||||||
|
|
||||||
|
**Catalog Freshness** — La corrispondenza fra uno scope sincronizzato e la versione corrente
|
||||||
|
della Database Binding. Uno scope rimane consultabile ma è stale finché non viene sincronizzato
|
||||||
|
con la binding corrente.
|
||||||
|
|
||||||
|
**Catalog Metadata** — I campi mutabili che descrivono database, tabelle, colonne e relazioni,
|
||||||
|
distinti dai fatti strutturali governati dalla sincronizzazione. Possono essere popolati dall'AI,
|
||||||
|
da un'importazione o da una modifica amministrativa senza cambiare il database esterno.
|
||||||
|
|
||||||
|
**Introspection Capability** — Una categoria di struttura fisica che una Database Binding
|
||||||
|
può osservare, come tabelle, colonne, relazioni, indici o enum. Una capability non disponibile
|
||||||
|
è distinta da una capability osservata che non ha restituito elementi.
|
||||||
|
|||||||
+49
-3
@@ -1,6 +1,6 @@
|
|||||||
# ThothII — Project State
|
# ThothII — Project State
|
||||||
|
|
||||||
Last updated: 2026-08-26.
|
Last updated: 2026-08-27.
|
||||||
|
|
||||||
This file is the short operational snapshot. Stable commands and the architecture mental model
|
This file is the short operational snapshot. Stable commands and the architecture mental model
|
||||||
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
|
live in `AGENTS.md`; current design and runtime contracts live under `docs/architecture/`,
|
||||||
@@ -16,8 +16,9 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
|
|||||||
```
|
```
|
||||||
|
|
||||||
The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence.
|
The harness owns the deterministic eight-phase NL→SQL workflow and all session persistence.
|
||||||
The backend is a process/RPC/SSE bridge without a database of its own. The frontend renders the
|
The backend remains a process/RPC/SSE bridge for sessions and now also owns an isolated PostgreSQL
|
||||||
review gates and keeps the live transcript in memory. See
|
metadata catalog for administrative database configuration. The frontend renders the review gates
|
||||||
|
and keeps the live transcript in memory. See
|
||||||
`docs/architecture/components.md` for the detailed component and data-flow map.
|
`docs/architecture/components.md` for the detailed component and data-flow map.
|
||||||
|
|
||||||
## Evidence restructuring — accepted
|
## Evidence restructuring — accepted
|
||||||
@@ -62,6 +63,51 @@ Workspace descriptors use schema v3. For PSD, workspace content and runtime root
|
|||||||
separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside
|
separate uncommitted repository `/Users/mp/projects/tht-workspace-psd`. Secrets remain outside
|
||||||
Git and are supplied only through installation-local protected files.
|
Git and are supplied only through installation-local protected files.
|
||||||
|
|
||||||
|
## Database management
|
||||||
|
|
||||||
|
The database, table, and authoritative physical-schema catalog slices are implemented. Database
|
||||||
|
management opens a responsive AG Grid master-detail surface, lists every YAML workspace, creates
|
||||||
|
at most one PostgreSQL database configuration per workspace, edits direct PostgreSQL, REST API, or
|
||||||
|
SSH-tunnel installation bindings, replaces write-only encrypted secrets, and tests supported
|
||||||
|
connector bindings.
|
||||||
|
|
||||||
|
Configured databases use pure hierarchical navigation through `Overview`, `Tables`, and
|
||||||
|
`Relationships`; a selected table has `Overview` and `Columns`. Physical membership, source
|
||||||
|
comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are
|
||||||
|
immutable projections of the external schema. Curated and generated descriptions are editable;
|
||||||
|
generated descriptions start null and AI generation/consolidation is deferred.
|
||||||
|
|
||||||
|
Schema refresh is one durable asynchronous engine with database-table, database-column,
|
||||||
|
selected-table-column, relationship, and full-database actions. Database-level menus expose the
|
||||||
|
table, all-column, relationship, and full scopes separately; selecting tables exposes column
|
||||||
|
synchronization for that subset. Runs have one-active-job-per-database exclusion, leases and
|
||||||
|
restart recovery, atomic apply, destructive-diff confirmation with re-scan, cancellation before
|
||||||
|
apply, retained history, and a live SSE log with polling fallback. Null metadata renders blank
|
||||||
|
rather than as a placeholder.
|
||||||
|
|
||||||
|
Direct PostgreSQL and strict known-host-verified OpenSSH use `pg_catalog`. REST bindings use the
|
||||||
|
typed full-snapshot `POST /rpc/schema_snapshot` contract when available. Servers such as the
|
||||||
|
current PSD endpoint that exposes only `POST /rpc/run_query` use one catalog-owned read-only query
|
||||||
|
to return the exact same strict v1 snapshot in a single round trip. Both paths remain fail-closed:
|
||||||
|
an absent capability, query error, partial result, or invalid snapshot applies no catalog changes.
|
||||||
|
SSH is not yet enabled for NL→SQL session runtime.
|
||||||
|
|
||||||
|
The catalog runs in the internal `catalog-db` PostgreSQL service. Kysely migrations are an explicit
|
||||||
|
one-shot `catalog-migrate` operation; `scripts/run-stack.sh` runs it before local startup. Runtime
|
||||||
|
sessions still consume the existing workspace configuration in this slice: database-management
|
||||||
|
records do not yet change the NL→SQL handoff. The accepted design is recorded in
|
||||||
|
`docs/plans/2026-08-26-metadata-catalog-from-thothai.md`, the snapshot contract under
|
||||||
|
`docs/contracts/`, and ADRs 0001–0007.
|
||||||
|
|
||||||
|
Semantic aliases, value descriptions, synonyms, concepts, AI metadata generation/consolidation,
|
||||||
|
and logical relationships remain deferred to their dedicated slices.
|
||||||
|
|
||||||
|
Integration of the completed metadata catalog with core schema-linking is explicitly deferred
|
||||||
|
until the database, table, column, relationship, and synchronization slices are complete. At that
|
||||||
|
point the next required design gate is to compare the catalog snapshot with the current DWH
|
||||||
|
preprocessing/schema-linking contracts and plan the cutover; this follow-up must not be treated as
|
||||||
|
optional cleanup or silently omitted.
|
||||||
|
|
||||||
## Active deployment work and manual gates
|
## Active deployment work and manual gates
|
||||||
|
|
||||||
### PSD server deployment program
|
### PSD server deployment program
|
||||||
|
|||||||
@@ -1,15 +1,16 @@
|
|||||||
# ThothII
|
# ThothII
|
||||||
|
|
||||||
ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/`tht`
|
ThothII is a human-reviewed NL-to-SQL workflow with a React frontend and a Fastify/Pi/`tht`
|
||||||
core. The portable deployment runs exactly two application services; data services remain
|
core. The portable deployment runs two application services plus the installation-local metadata
|
||||||
external in this profile, except for the mandatory internal semantic services bundled in Compose.
|
catalog; DWH and LLM services remain external. Semantic services are bundled in Compose.
|
||||||
|
|
||||||
Authentication is configured through the single host CLI tht: see the [local authentication guide](docs/install/authentication-local.md),
|
Authentication is configured through the single host CLI tht: see the [local authentication guide](docs/install/authentication-local.md),
|
||||||
[generic OIDC guide](docs/install/authentication-oidc.md), and [manual acceptance matrix](docs/testing/authentication-manual-acceptance.md).
|
[generic OIDC guide](docs/install/authentication-oidc.md), and [manual acceptance matrix](docs/testing/authentication-manual-acceptance.md).
|
||||||
|
|
||||||
## Docker Compose: local startup
|
## Docker Compose: local startup
|
||||||
|
|
||||||
Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external,
|
Requirements: Docker Engine with Compose v2. The mandatory stack is `frontend`, `core`,
|
||||||
|
`catalog-db`, `qdrant`, `embedding`, and the one-shot `embedding-model-init`. DWH and LLM remain external,
|
||||||
configurable endpoints—even when they are co-located with ThothII.
|
configurable endpoints—even when they are co-located with ThothII.
|
||||||
|
|
||||||
From a fresh clone, run these commands from the repository root:
|
From a fresh clone, run these commands from the repository root:
|
||||||
@@ -17,17 +18,28 @@ From a fresh clone, run these commands from the repository root:
|
|||||||
```sh
|
```sh
|
||||||
cp deploy/env/local.env.example deploy/env/local.env
|
cp deploy/env/local.env.example deploy/env/local.env
|
||||||
# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
|
# Edit deploy/env/local.env, including PI_AUTH_FILE, THT_SECRETS_FILE, and external endpoints.
|
||||||
docker compose --env-file deploy/env/local.env \
|
./scripts/run-stack.sh
|
||||||
-f compose.yaml -f deploy/compose.local.yaml up --build -d
|
|
||||||
```
|
```
|
||||||
|
|
||||||
`./scripts/run-stack.sh` runs this same base+local command in the foreground. The core image
|
The launcher builds the core, starts `catalog-db`, runs the explicit one-shot Kysely migrations,
|
||||||
contains its Pi runtime; no host `pi` executable is used. For a server installation:
|
then runs the base+local stack in the foreground. Migrations never run implicitly in backend
|
||||||
|
startup. The core image contains its Pi runtime; no host `pi` executable is used. For a server
|
||||||
|
installation, build the image, start the catalog, and run the same migration service before the
|
||||||
|
application rollout:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
cp deploy/env/server.env.example deploy/env/server.env
|
cp deploy/env/server.env.example deploy/env/server.env
|
||||||
# Edit all absolute storage, Pi/secret/session files, and endpoint paths.
|
# Edit all absolute storage, Pi/secret/session files, and endpoint paths.
|
||||||
sudo scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001
|
sudo scripts/prepare-server-pi-state.sh /srv/thothii/pi-state 10001 10001
|
||||||
|
docker compose --env-file deploy/env/server.env \
|
||||||
|
-f compose.yaml -f deploy/compose.server.yaml \
|
||||||
|
-f deploy/compose.session-server.yaml.example build core
|
||||||
|
docker compose --env-file deploy/env/server.env \
|
||||||
|
-f compose.yaml -f deploy/compose.server.yaml \
|
||||||
|
-f deploy/compose.session-server.yaml.example up -d catalog-db
|
||||||
|
docker compose --env-file deploy/env/server.env \
|
||||||
|
-f compose.yaml -f deploy/compose.server.yaml \
|
||||||
|
-f deploy/compose.session-server.yaml.example run --rm catalog-migrate
|
||||||
docker compose --env-file deploy/env/server.env \
|
docker compose --env-file deploy/env/server.env \
|
||||||
-f compose.yaml -f deploy/compose.server.yaml \
|
-f compose.yaml -f deploy/compose.server.yaml \
|
||||||
-f deploy/compose.session-server.yaml.example up --build -d
|
-f deploy/compose.session-server.yaml.example up --build -d
|
||||||
@@ -101,10 +113,12 @@ schema, Evidence, and Memory records share that collection and stay separated by
|
|||||||
`kind`.
|
`kind`.
|
||||||
<!-- workspace-descriptor-contract:end -->
|
<!-- workspace-descriptor-contract:end -->
|
||||||
|
|
||||||
Connector `ssh_tunnel` bindings are diagnostic-only in this release: their bounded probe always
|
For NL→SQL runtime sessions, connector `ssh_tunnel` bindings remain diagnostic-only: their bounded
|
||||||
cleans up the loopback forward and returns `workspace_not_activatable`; session creation is rejected
|
probe cleans up the loopback forward and returns `workspace_not_activatable`; session creation is
|
||||||
before persistence. Git registry access over SSH is unaffected. Use direct or REST connector
|
rejected before persistence. Database management is a separate boundary and supports a strict
|
||||||
transport for runtime sessions.
|
OpenSSH tunnel for **Test connection** and **Sync tables**, using a private key, optional passphrase,
|
||||||
|
mandatory `known_hosts`, and optional PostgreSQL TLS CA/server name. Git registry access over SSH is
|
||||||
|
unaffected. Use direct or REST connector transport for runtime sessions.
|
||||||
|
|
||||||
`docker-compose.dev.yml` is deliberately local: both published ports bind to `127.0.0.1`,
|
`docker-compose.dev.yml` is deliberately local: both published ports bind to `127.0.0.1`,
|
||||||
`THT_SESSION_STORAGE=local`, and `THT_HOME=/data/local-home`. Do not set
|
`THT_SESSION_STORAGE=local`, and `THT_HOME=/data/local-home`. Do not set
|
||||||
|
|||||||
Generated
+2049
File diff suppressed because it is too large
Load Diff
@@ -6,6 +6,7 @@
|
|||||||
"dev": "tsx watch src/server.ts",
|
"dev": "tsx watch src/server.ts",
|
||||||
"prebuild": "node scripts/clean-dist.mjs",
|
"prebuild": "node scripts/clean-dist.mjs",
|
||||||
"build": "tsc -p tsconfig.json",
|
"build": "tsc -p tsconfig.json",
|
||||||
|
"catalog:migrate": "node dist/catalog/migrate.js",
|
||||||
"test": "vitest run",
|
"test": "vitest run",
|
||||||
"start": "node dist/server.js",
|
"start": "node dist/server.js",
|
||||||
"test:schema-v3-verifier": "python3 -I -B scripts/test_revision_state_policy.py && node --test scripts/verify-workspace-descriptor-files.test.mjs scripts/revision-state-policy.test.mjs"
|
"test:schema-v3-verifier": "python3 -I -B scripts/test_revision_state_policy.py && node --test scripts/verify-workspace-descriptor-files.test.mjs scripts/revision-state-policy.test.mjs"
|
||||||
@@ -16,12 +17,14 @@
|
|||||||
"@fastify/rate-limit": "11.2.0",
|
"@fastify/rate-limit": "11.2.0",
|
||||||
"@types/pg": "^8.20.3",
|
"@types/pg": "^8.20.3",
|
||||||
"fastify": "^5.0.0",
|
"fastify": "^5.0.0",
|
||||||
|
"kysely": "^0.29.5",
|
||||||
"openid-client": "6.8.5",
|
"openid-client": "6.8.5",
|
||||||
"pg": "^8.22.0",
|
"pg": "^8.22.0",
|
||||||
"yaml": "^2.9.0",
|
"yaml": "^2.9.0",
|
||||||
"zod": "^4.4.3"
|
"zod": "^4.4.3"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
|
"@testcontainers/postgresql": "^12.1.0",
|
||||||
"@types/node": "24.13.3",
|
"@types/node": "24.13.3",
|
||||||
"tsx": "^4.19.0",
|
"tsx": "^4.19.0",
|
||||||
"typescript": "^5.6.0",
|
"typescript": "^5.6.0",
|
||||||
|
|||||||
Executable
+6
@@ -0,0 +1,6 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
import { readFileSync } from "node:fs";
|
||||||
|
|
||||||
|
const path = process.env.THT_SSH_PASSPHRASE_FILE;
|
||||||
|
if (!path) process.exit(1);
|
||||||
|
process.stdout.write(readFileSync(path));
|
||||||
@@ -37,6 +37,17 @@ import { supportsSessionRuntime } from "./workspaces/bindings.js";
|
|||||||
import { resolveRuntimeBindingsWithWorkspaceSecrets } from "./workspaces/secret-requirements.js";
|
import { resolveRuntimeBindingsWithWorkspaceSecrets } from "./workspaces/secret-requirements.js";
|
||||||
import type { WorkspaceDescriptor } from "./workspaces/schema.js";
|
import type { WorkspaceDescriptor } from "./workspaces/schema.js";
|
||||||
import { WorkspaceSecretStore } from "./workspaces/secret-store.js";
|
import { WorkspaceSecretStore } from "./workspaces/secret-store.js";
|
||||||
|
import { createCatalogRepository } from "./catalog/repository.js";
|
||||||
|
import type { CatalogRepository } from "./catalog/types.js";
|
||||||
|
import { CatalogService } from "./catalog/service.js";
|
||||||
|
import { catalogDatabaseRoutes } from "./routes/catalog-databases.js";
|
||||||
|
import { CatalogOperationCoordinator } from "./catalog/operation-coordinator.js";
|
||||||
|
import { ConcreteCatalogPostgresAccess, type CatalogPostgresAccess } from "./catalog/postgres-access.js";
|
||||||
|
import { CatalogTableService } from "./catalog/table-service.js";
|
||||||
|
import { catalogTableRoutes } from "./routes/catalog-tables.js";
|
||||||
|
import { ConcreteCatalogSchemaIntrospector, type CatalogSchemaIntrospector } from "./catalog/schema-introspector.js";
|
||||||
|
import { CatalogSyncWorker } from "./catalog/sync-worker.js";
|
||||||
|
import { catalogSchemaRoutes } from "./routes/catalog-schema.js";
|
||||||
|
|
||||||
export interface BuildAppDeps {
|
export interface BuildAppDeps {
|
||||||
thtRunner?: ThtRunner;
|
thtRunner?: ThtRunner;
|
||||||
@@ -49,6 +60,13 @@ export interface BuildAppDeps {
|
|||||||
workspaceRegistry?: WorkspaceRegistry;
|
workspaceRegistry?: WorkspaceRegistry;
|
||||||
workspaceDiagnoser?: WorkspaceDiagnoser;
|
workspaceDiagnoser?: WorkspaceDiagnoser;
|
||||||
workspaceSecretStore?: WorkspaceSecretStore;
|
workspaceSecretStore?: WorkspaceSecretStore;
|
||||||
|
catalogRepository?: CatalogRepository;
|
||||||
|
catalogService?: CatalogService;
|
||||||
|
catalogPostgresAccess?: CatalogPostgresAccess;
|
||||||
|
catalogTableService?: CatalogTableService;
|
||||||
|
catalogSchemaIntrospector?: CatalogSchemaIntrospector;
|
||||||
|
catalogSyncWorker?: CatalogSyncWorker;
|
||||||
|
catalogOperationCoordinator?: CatalogOperationCoordinator;
|
||||||
workspaceRuntimeSupport?: (workspace: WorkspaceDescriptor) => boolean;
|
workspaceRuntimeSupport?: (workspace: WorkspaceDescriptor) => boolean;
|
||||||
maintenanceBarrier?: MaintenanceBarrier;
|
maintenanceBarrier?: MaintenanceBarrier;
|
||||||
piManagement?: PiManagementService;
|
piManagement?: PiManagementService;
|
||||||
@@ -121,6 +139,37 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
|
|||||||
const mgr = deps?.mgr ?? new PiProcessManager(config, deps?.spawnFn ? { spawnFn: deps.spawnFn } : undefined);
|
const mgr = deps?.mgr ?? new PiProcessManager(config, deps?.spawnFn ? { spawnFn: deps.spawnFn } : undefined);
|
||||||
const hub = deps?.hub ?? new SseHub();
|
const hub = deps?.hub ?? new SseHub();
|
||||||
const workspaceRegistry = deps?.workspaceRegistry ?? new WorkspaceRegistry(config.workspaceRegistry);
|
const workspaceRegistry = deps?.workspaceRegistry ?? new WorkspaceRegistry(config.workspaceRegistry);
|
||||||
|
const catalogRepository = deps?.catalogRepository ?? createCatalogRepository(config.catalogDatabase);
|
||||||
|
const catalogOperationCoordinator = deps?.catalogOperationCoordinator ?? new CatalogOperationCoordinator();
|
||||||
|
const catalogPostgresAccess = deps?.catalogPostgresAccess ?? new ConcreteCatalogPostgresAccess(
|
||||||
|
workspaceSecretStore,
|
||||||
|
{ connectTimeoutMs: config.workspaceDiagnosticTimeoutMs },
|
||||||
|
);
|
||||||
|
const catalogService = deps?.catalogService ?? new CatalogService(
|
||||||
|
catalogRepository,
|
||||||
|
workspaceRegistry,
|
||||||
|
workspaceSecretStore,
|
||||||
|
config.workspaceRegistry.secretRoots,
|
||||||
|
config.workspaceDiagnosticTimeoutMs,
|
||||||
|
catalogPostgresAccess,
|
||||||
|
catalogOperationCoordinator,
|
||||||
|
);
|
||||||
|
const catalogTableService = deps?.catalogTableService ?? new CatalogTableService(catalogRepository);
|
||||||
|
const catalogSchemaIntrospector = deps?.catalogSchemaIntrospector ?? new ConcreteCatalogSchemaIntrospector(
|
||||||
|
catalogPostgresAccess,
|
||||||
|
workspaceSecretStore,
|
||||||
|
);
|
||||||
|
const catalogSyncWorker = deps?.catalogSyncWorker ?? new CatalogSyncWorker(
|
||||||
|
catalogRepository,
|
||||||
|
catalogSchemaIntrospector,
|
||||||
|
catalogOperationCoordinator,
|
||||||
|
config.catalogSyncTimeoutMs,
|
||||||
|
);
|
||||||
|
app.addHook("onReady", async () => { await catalogSyncWorker.initialize(); });
|
||||||
|
if (!deps?.catalogRepository && catalogRepository.close) {
|
||||||
|
app.addHook("onClose", async () => { await catalogRepository.close?.(); });
|
||||||
|
}
|
||||||
|
app.addHook("onClose", async () => { await catalogSyncWorker.stop(); });
|
||||||
const workspaceDiagnoser = deps?.workspaceDiagnoser
|
const workspaceDiagnoser = deps?.workspaceDiagnoser
|
||||||
?? createProductionWorkspaceDiagnoser(config.workspaceDiagnosticTimeoutMs, undefined, {
|
?? createProductionWorkspaceDiagnoser(config.workspaceDiagnosticTimeoutMs, undefined, {
|
||||||
internalQdrantUrl: config.internalQdrantUrl,
|
internalQdrantUrl: config.internalQdrantUrl,
|
||||||
@@ -334,6 +383,9 @@ export function buildApp(config: AppConfig, deps?: BuildAppDeps): FastifyInstanc
|
|||||||
authDiagnoser,
|
authDiagnoser,
|
||||||
secretStore: workspaceSecretStore,
|
secretStore: workspaceSecretStore,
|
||||||
});
|
});
|
||||||
|
catalogDatabaseRoutes(app, { repository: catalogRepository, service: catalogService, operations: catalogOperationCoordinator });
|
||||||
|
catalogTableRoutes(app, { repository: catalogRepository, service: catalogTableService });
|
||||||
|
catalogSchemaRoutes(app, { repository: catalogRepository, worker: catalogSyncWorker });
|
||||||
settingsRoutes(app, { cfg: config, listModels, getSettings });
|
settingsRoutes(app, { cfg: config, listModels, getSettings });
|
||||||
piManagementRoutes(app, { service: piManagement });
|
piManagementRoutes(app, { service: piManagement });
|
||||||
|
|
||||||
|
|||||||
@@ -38,7 +38,7 @@ const MAX_MAPPED_GROUPS = 128;
|
|||||||
const ROLES = ["user", "admin"] as const;
|
const ROLES = ["user", "admin"] as const;
|
||||||
export const PERMISSION_CATALOG: readonly Permission[] = [
|
export const PERMISSION_CATALOG: readonly Permission[] = [
|
||||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||||
];
|
];
|
||||||
|
|
||||||
const invalid = (): Error => new Error("authentication configuration is invalid");
|
const invalid = (): Error => new Error("authentication configuration is invalid");
|
||||||
|
|||||||
@@ -33,7 +33,7 @@ const EMPTY_HKDF_SALT = Buffer.alloc(0);
|
|||||||
const ROLES = ["user", "admin"] as const;
|
const ROLES = ["user", "admin"] as const;
|
||||||
const PERMISSIONS = [
|
const PERMISSIONS = [
|
||||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||||
] as const satisfies readonly Permission[];
|
] as const satisfies readonly Permission[];
|
||||||
|
|
||||||
const invalid = (): Error => new Error("auth_session_store_invalid");
|
const invalid = (): Error => new Error("auth_session_store_invalid");
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ export type Role = "user" | "admin";
|
|||||||
export type Permission =
|
export type Permission =
|
||||||
| "session.use" | "session.read_all" | "session.manage_all"
|
| "session.use" | "session.read_all" | "session.manage_all"
|
||||||
| "settings.manage" | "workspace.manage" | "workspace.secrets.manage"
|
| "settings.manage" | "workspace.manage" | "workspace.secrets.manage"
|
||||||
| "pi.manage" | "auth.diagnostics.read";
|
| "database.manage" | "pi.manage" | "auth.diagnostics.read";
|
||||||
|
|
||||||
export interface AuthenticationSessionConfig {
|
export interface AuthenticationSessionConfig {
|
||||||
regularTtlSeconds: number;
|
regularTtlSeconds: number;
|
||||||
|
|||||||
@@ -0,0 +1,692 @@
|
|||||||
|
import { randomUUID } from "node:crypto";
|
||||||
|
import {
|
||||||
|
CatalogConflictError,
|
||||||
|
CatalogConnectorError,
|
||||||
|
type CatalogColumn,
|
||||||
|
type CatalogRelationship,
|
||||||
|
type CatalogSchemaDiff,
|
||||||
|
type CatalogSyncCounts,
|
||||||
|
type CatalogSyncEvent,
|
||||||
|
type CatalogSyncRun,
|
||||||
|
type CatalogSyncRunUpdate,
|
||||||
|
type CatalogSyncScope,
|
||||||
|
type CatalogTable,
|
||||||
|
type CatalogRepository,
|
||||||
|
type DatabaseConfigurationInput,
|
||||||
|
type DatabaseTestResult,
|
||||||
|
type ObservedCatalogTable,
|
||||||
|
type ObservedSchemaSnapshot,
|
||||||
|
type TableSyncRepositoryResult,
|
||||||
|
type WorkspaceDatabase,
|
||||||
|
} from "./types.js";
|
||||||
|
|
||||||
|
function clone(value: WorkspaceDatabase): WorkspaceDatabase {
|
||||||
|
return structuredClone(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Deterministic repository used by route tests and local contract consumers. */
|
||||||
|
export class MemoryCatalogRepository implements CatalogRepository {
|
||||||
|
private readonly records = new Map<string, WorkspaceDatabase>();
|
||||||
|
private readonly tables = new Map<string, CatalogTable>();
|
||||||
|
private readonly columns = new Map<string, CatalogColumn>();
|
||||||
|
private readonly relationships = new Map<string, CatalogRelationship>();
|
||||||
|
private readonly syncRuns = new Map<string, CatalogSyncRun>();
|
||||||
|
private readonly syncEvents = new Map<string, CatalogSyncEvent[]>();
|
||||||
|
|
||||||
|
async list(): Promise<WorkspaceDatabase[]> {
|
||||||
|
return [...this.records.values()].sort((a, b) => a.workspaceId.localeCompare(b.workspaceId)).map(clone);
|
||||||
|
}
|
||||||
|
async get(id: string): Promise<WorkspaceDatabase | undefined> {
|
||||||
|
const value = this.records.get(id);
|
||||||
|
return value ? clone(value) : undefined;
|
||||||
|
}
|
||||||
|
async getByWorkspace(workspaceId: string): Promise<WorkspaceDatabase | undefined> {
|
||||||
|
const value = [...this.records.values()].find((record) => record.workspaceId === workspaceId);
|
||||||
|
return value ? clone(value) : undefined;
|
||||||
|
}
|
||||||
|
async create(input: DatabaseConfigurationInput): Promise<WorkspaceDatabase> {
|
||||||
|
if ([...this.records.values()].some((record) => record.workspaceId === input.workspaceId)) {
|
||||||
|
throw new CatalogConflictError("Workspace database already exists");
|
||||||
|
}
|
||||||
|
const now = new Date().toISOString();
|
||||||
|
const record: WorkspaceDatabase = {
|
||||||
|
id: randomUUID(),
|
||||||
|
...structuredClone(input),
|
||||||
|
version: 1,
|
||||||
|
createdAt: now,
|
||||||
|
updatedAt: now,
|
||||||
|
connectionStatus: "untested",
|
||||||
|
};
|
||||||
|
this.records.set(record.id, record);
|
||||||
|
return clone(record);
|
||||||
|
}
|
||||||
|
async update(id: string, expectedVersion: number, input: DatabaseConfigurationInput): Promise<WorkspaceDatabase | undefined> {
|
||||||
|
const current = this.records.get(id);
|
||||||
|
if (!current || current.version !== expectedVersion) return undefined;
|
||||||
|
if ([...this.records.values()].some((record) => record.id !== id && record.workspaceId === input.workspaceId)) {
|
||||||
|
throw new CatalogConflictError("Workspace database already exists");
|
||||||
|
}
|
||||||
|
const updated: WorkspaceDatabase = {
|
||||||
|
...current,
|
||||||
|
...structuredClone(input),
|
||||||
|
version: current.version + 1,
|
||||||
|
updatedAt: new Date().toISOString(),
|
||||||
|
connectionStatus: "untested",
|
||||||
|
testedVersion: undefined,
|
||||||
|
lastTestedAt: undefined,
|
||||||
|
lastErrorCode: undefined,
|
||||||
|
lastErrorMessage: undefined,
|
||||||
|
schemaSyncedVersion: undefined,
|
||||||
|
schemaSyncedAt: undefined,
|
||||||
|
};
|
||||||
|
this.records.set(id, updated);
|
||||||
|
return clone(updated);
|
||||||
|
}
|
||||||
|
async recordTest(id: string, expectedVersion: number, result: DatabaseTestResult): Promise<WorkspaceDatabase | undefined> {
|
||||||
|
const current = this.records.get(id);
|
||||||
|
if (!current || current.version !== expectedVersion) return undefined;
|
||||||
|
const updated = {
|
||||||
|
...current,
|
||||||
|
connectionStatus: result.connectionStatus,
|
||||||
|
testedVersion: result.testedVersion,
|
||||||
|
lastTestedAt: result.lastTestedAt,
|
||||||
|
lastErrorCode: result.errorCode,
|
||||||
|
lastErrorMessage: result.errorMessage,
|
||||||
|
};
|
||||||
|
this.records.set(id, updated);
|
||||||
|
return clone(updated);
|
||||||
|
}
|
||||||
|
async touch(id: string, expectedVersion: number): Promise<WorkspaceDatabase | undefined> {
|
||||||
|
const current = this.records.get(id);
|
||||||
|
if (!current || current.version !== expectedVersion) return undefined;
|
||||||
|
return await this.update(id, expectedVersion, {
|
||||||
|
workspaceId: current.workspaceId,
|
||||||
|
engine: current.engine,
|
||||||
|
databaseName: current.databaseName,
|
||||||
|
schema: current.schema,
|
||||||
|
binding: current.binding,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
async delete(id: string, expectedVersion: number): Promise<boolean> {
|
||||||
|
const current = this.records.get(id);
|
||||||
|
if (!current || current.version !== expectedVersion) return false;
|
||||||
|
for (const [tableId, table] of this.tables) {
|
||||||
|
if (table.databaseId === id) {
|
||||||
|
this.tables.delete(tableId);
|
||||||
|
for (const [columnId, column] of this.columns) if (column.tableId === tableId) this.columns.delete(columnId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const [relationshipId, relationship] of this.relationships) {
|
||||||
|
if (relationship.databaseId === id) this.relationships.delete(relationshipId);
|
||||||
|
}
|
||||||
|
return this.records.delete(id);
|
||||||
|
}
|
||||||
|
async listTables(databaseId: string): Promise<CatalogTable[]> {
|
||||||
|
return [...this.tables.values()]
|
||||||
|
.filter((table) => table.databaseId === databaseId)
|
||||||
|
.sort((a, b) => a.name.localeCompare(b.name))
|
||||||
|
.map((table) => structuredClone(table));
|
||||||
|
}
|
||||||
|
async getTable(databaseId: string, tableId: string): Promise<CatalogTable | undefined> {
|
||||||
|
const table = this.tables.get(tableId);
|
||||||
|
return table?.databaseId === databaseId ? structuredClone(table) : undefined;
|
||||||
|
}
|
||||||
|
async updateTableDescription(
|
||||||
|
databaseId: string,
|
||||||
|
tableId: string,
|
||||||
|
expectedVersion: number,
|
||||||
|
description: string | null,
|
||||||
|
): Promise<CatalogTable | undefined> {
|
||||||
|
const current = this.tables.get(tableId);
|
||||||
|
if (!current || current.databaseId !== databaseId || current.version !== expectedVersion) return undefined;
|
||||||
|
const updated = {
|
||||||
|
...current,
|
||||||
|
description,
|
||||||
|
version: current.version + 1,
|
||||||
|
updatedAt: new Date().toISOString(),
|
||||||
|
};
|
||||||
|
this.tables.set(tableId, updated);
|
||||||
|
return structuredClone(updated);
|
||||||
|
}
|
||||||
|
async updateTableMetadata(
|
||||||
|
databaseId: string,
|
||||||
|
tableId: string,
|
||||||
|
expectedVersion: number,
|
||||||
|
description: string | null,
|
||||||
|
generatedDescription: string | null,
|
||||||
|
): Promise<CatalogTable | undefined> {
|
||||||
|
const current = this.tables.get(tableId);
|
||||||
|
if (!current || current.databaseId !== databaseId || current.version !== expectedVersion) return undefined;
|
||||||
|
const updated = {
|
||||||
|
...current, description, generatedDescription, version: current.version + 1,
|
||||||
|
updatedAt: new Date().toISOString(),
|
||||||
|
};
|
||||||
|
this.tables.set(tableId, updated);
|
||||||
|
return structuredClone(updated);
|
||||||
|
}
|
||||||
|
|
||||||
|
async listColumns(databaseId: string, tableId: string): Promise<CatalogColumn[]> {
|
||||||
|
const table = this.tables.get(tableId);
|
||||||
|
if (!table || table.databaseId !== databaseId) return [];
|
||||||
|
return [...this.columns.values()].filter((column) => column.tableId === tableId)
|
||||||
|
.sort((a, b) => a.ordinalPosition - b.ordinalPosition).map((column) => structuredClone(column));
|
||||||
|
}
|
||||||
|
|
||||||
|
async getColumn(databaseId: string, tableId: string, columnId: string): Promise<CatalogColumn | undefined> {
|
||||||
|
const table = this.tables.get(tableId);
|
||||||
|
const column = this.columns.get(columnId);
|
||||||
|
return table?.databaseId === databaseId && column?.tableId === tableId ? structuredClone(column) : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
async updateColumnMetadata(
|
||||||
|
databaseId: string,
|
||||||
|
tableId: string,
|
||||||
|
columnId: string,
|
||||||
|
expectedVersion: number,
|
||||||
|
description: string | null,
|
||||||
|
generatedDescription: string | null,
|
||||||
|
): Promise<CatalogColumn | undefined> {
|
||||||
|
const current = await this.getColumn(databaseId, tableId, columnId);
|
||||||
|
if (!current || current.version !== expectedVersion) return undefined;
|
||||||
|
const updated = { ...current, description, generatedDescription, version: current.version + 1, updatedAt: new Date().toISOString() };
|
||||||
|
this.columns.set(columnId, updated);
|
||||||
|
return structuredClone(updated);
|
||||||
|
}
|
||||||
|
|
||||||
|
async listRelationships(databaseId: string): Promise<CatalogRelationship[]> {
|
||||||
|
return [...this.relationships.values()].filter((relationship) => relationship.databaseId === databaseId)
|
||||||
|
.sort((a, b) => `${a.sourceTableName}.${a.constraintName}`.localeCompare(`${b.sourceTableName}.${b.constraintName}`))
|
||||||
|
.map((relationship) => structuredClone(relationship));
|
||||||
|
}
|
||||||
|
|
||||||
|
async planSchemaSync(
|
||||||
|
databaseId: string,
|
||||||
|
scope: CatalogSyncScope,
|
||||||
|
tableIds: readonly string[],
|
||||||
|
snapshot: ObservedSchemaSnapshot,
|
||||||
|
): Promise<CatalogSchemaDiff> {
|
||||||
|
this.assertSnapshotCapability(scope, snapshot);
|
||||||
|
const tables = await this.listTables(databaseId);
|
||||||
|
const selectedTableIds = new Set(tableIds);
|
||||||
|
const selectedTables = scope === "columns" && selectedTableIds.size > 0
|
||||||
|
? tables.filter((table) => selectedTableIds.has(table.id))
|
||||||
|
: tables;
|
||||||
|
const observedTables = new Set(snapshot.tables.map((table) => table.name));
|
||||||
|
const observedColumns = new Set(snapshot.columns.map((column) => `${column.tableName}\u0000${column.name}`));
|
||||||
|
const observedRelationships = new Set(
|
||||||
|
snapshot.relationships.map((relationship) => `${relationship.sourceTableName}\u0000${relationship.constraintName}`),
|
||||||
|
);
|
||||||
|
const deletedTableIds = new Set(tables.filter((table) => !observedTables.has(table.name)).map((table) => table.id));
|
||||||
|
|
||||||
|
return {
|
||||||
|
deletedTables: scope === "tables" || scope === "all"
|
||||||
|
? tables.filter((table) => !observedTables.has(table.name)).map((table) => table.name).sort()
|
||||||
|
: [],
|
||||||
|
deletedColumns: scope === "tables" || scope === "columns" || scope === "all"
|
||||||
|
? [...this.columns.values()]
|
||||||
|
.filter((column) => scope === "tables" ? deletedTableIds.has(column.tableId) : selectedTables.some((table) => table.id === column.tableId))
|
||||||
|
.filter((column) => {
|
||||||
|
const table = tables.find((candidate) => candidate.id === column.tableId);
|
||||||
|
return table && (scope === "tables" || !observedColumns.has(`${table.name}\u0000${column.name}`));
|
||||||
|
})
|
||||||
|
.map((column) => ({
|
||||||
|
tableName: tables.find((table) => table.id === column.tableId)?.name ?? "",
|
||||||
|
columnName: column.name,
|
||||||
|
}))
|
||||||
|
.sort((a, b) => `${a.tableName}.${a.columnName}`.localeCompare(`${b.tableName}.${b.columnName}`))
|
||||||
|
: [],
|
||||||
|
deletedRelationships: scope === "tables" || scope === "relationships" || scope === "all"
|
||||||
|
? [...this.relationships.values()]
|
||||||
|
.filter((relationship) => relationship.databaseId === databaseId)
|
||||||
|
.filter((relationship) => scope === "tables"
|
||||||
|
? deletedTableIds.has(relationship.sourceTableId) || deletedTableIds.has(relationship.targetTableId)
|
||||||
|
: !observedRelationships.has(`${relationship.sourceTableName}\u0000${relationship.constraintName}`))
|
||||||
|
.map((relationship) => ({
|
||||||
|
sourceTableName: relationship.sourceTableName,
|
||||||
|
constraintName: relationship.constraintName,
|
||||||
|
}))
|
||||||
|
.sort((a, b) => `${a.sourceTableName}.${a.constraintName}`.localeCompare(`${b.sourceTableName}.${b.constraintName}`))
|
||||||
|
: [],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async applySchemaSync(
|
||||||
|
databaseId: string,
|
||||||
|
expectedDatabaseVersion: number,
|
||||||
|
scope: CatalogSyncScope,
|
||||||
|
tableIds: readonly string[],
|
||||||
|
snapshot: ObservedSchemaSnapshot,
|
||||||
|
): Promise<CatalogSyncCounts | undefined> {
|
||||||
|
const database = this.records.get(databaseId);
|
||||||
|
if (!database || database.version !== expectedDatabaseVersion) return undefined;
|
||||||
|
this.assertSnapshotCapability(scope, snapshot);
|
||||||
|
|
||||||
|
const tableBackup = structuredClone([...this.tables.entries()]);
|
||||||
|
const columnBackup = structuredClone([...this.columns.entries()]);
|
||||||
|
const relationshipBackup = structuredClone([...this.relationships.entries()]);
|
||||||
|
const now = new Date().toISOString();
|
||||||
|
let created = 0;
|
||||||
|
let updated = 0;
|
||||||
|
let deleted = 0;
|
||||||
|
|
||||||
|
try {
|
||||||
|
if (scope === "tables" || scope === "all") {
|
||||||
|
const observedByName = new Map(snapshot.tables.map((table) => [table.name, table]));
|
||||||
|
const existing = await this.listTables(databaseId);
|
||||||
|
for (const table of existing) {
|
||||||
|
const observed = observedByName.get(table.name);
|
||||||
|
if (!observed) {
|
||||||
|
this.deleteTable(table.id);
|
||||||
|
deleted += 1;
|
||||||
|
} else {
|
||||||
|
const changed = table.sourceComment !== observed.sourceComment;
|
||||||
|
this.tables.set(table.id, {
|
||||||
|
...table,
|
||||||
|
sourceComment: observed.sourceComment,
|
||||||
|
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||||
|
lastSyncedAt: now,
|
||||||
|
version: changed ? table.version + 1 : table.version,
|
||||||
|
updatedAt: changed ? now : table.updatedAt,
|
||||||
|
});
|
||||||
|
if (changed) updated += 1;
|
||||||
|
observedByName.delete(table.name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const observed of observedByName.values()) {
|
||||||
|
const id = randomUUID();
|
||||||
|
this.tables.set(id, {
|
||||||
|
id,
|
||||||
|
databaseId,
|
||||||
|
name: observed.name,
|
||||||
|
sourceComment: observed.sourceComment,
|
||||||
|
description: null,
|
||||||
|
generatedDescription: null,
|
||||||
|
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||||
|
lastSyncedAt: now,
|
||||||
|
version: 1,
|
||||||
|
createdAt: now,
|
||||||
|
updatedAt: now,
|
||||||
|
});
|
||||||
|
created += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (scope === "columns" || scope === "all") {
|
||||||
|
const currentTables = await this.listTables(databaseId);
|
||||||
|
const selectedIds = scope === "all" || tableIds.length === 0
|
||||||
|
? new Set(currentTables.map((table) => table.id))
|
||||||
|
: new Set(tableIds);
|
||||||
|
const selectedTables = currentTables.filter((table) => selectedIds.has(table.id));
|
||||||
|
if (scope === "columns" && selectedTables.length !== selectedIds.size) {
|
||||||
|
throw new CatalogConnectorError("One or more selected tables no longer exist");
|
||||||
|
}
|
||||||
|
const selectedNames = new Set(selectedTables.map((table) => table.name));
|
||||||
|
const tableByName = new Map(selectedTables.map((table) => [table.name, table]));
|
||||||
|
const observedByKey = new Map(
|
||||||
|
snapshot.columns
|
||||||
|
.filter((column) => selectedNames.has(column.tableName))
|
||||||
|
.map((column) => [`${column.tableName}\u0000${column.name}`, column]),
|
||||||
|
);
|
||||||
|
for (const column of [...this.columns.values()].filter((candidate) => selectedIds.has(candidate.tableId))) {
|
||||||
|
const table = selectedTables.find((candidate) => candidate.id === column.tableId);
|
||||||
|
if (!table) continue;
|
||||||
|
const key = `${table.name}\u0000${column.name}`;
|
||||||
|
const observed = observedByKey.get(key);
|
||||||
|
if (!observed) {
|
||||||
|
this.deleteColumn(column.id);
|
||||||
|
deleted += 1;
|
||||||
|
} else {
|
||||||
|
const changed = column.ordinalPosition !== observed.ordinalPosition
|
||||||
|
|| column.dataType !== observed.dataType
|
||||||
|
|| column.isNullable !== observed.isNullable
|
||||||
|
|| column.defaultExpression !== observed.defaultExpression
|
||||||
|
|| column.primaryKeyPosition !== observed.primaryKeyPosition
|
||||||
|
|| column.sourceComment !== observed.sourceComment;
|
||||||
|
this.columns.set(column.id, {
|
||||||
|
...column,
|
||||||
|
ordinalPosition: observed.ordinalPosition,
|
||||||
|
dataType: observed.dataType,
|
||||||
|
isNullable: observed.isNullable,
|
||||||
|
defaultExpression: observed.defaultExpression,
|
||||||
|
primaryKeyPosition: observed.primaryKeyPosition,
|
||||||
|
isPrimaryKey: observed.primaryKeyPosition !== null,
|
||||||
|
sourceComment: observed.sourceComment,
|
||||||
|
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||||
|
lastSyncedAt: now,
|
||||||
|
version: changed ? column.version + 1 : column.version,
|
||||||
|
updatedAt: changed ? now : column.updatedAt,
|
||||||
|
});
|
||||||
|
if (changed) updated += 1;
|
||||||
|
observedByKey.delete(key);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const observed of observedByKey.values()) {
|
||||||
|
const table = tableByName.get(observed.tableName);
|
||||||
|
if (!table) continue;
|
||||||
|
const id = randomUUID();
|
||||||
|
this.columns.set(id, {
|
||||||
|
id,
|
||||||
|
tableId: table.id,
|
||||||
|
name: observed.name,
|
||||||
|
ordinalPosition: observed.ordinalPosition,
|
||||||
|
dataType: observed.dataType,
|
||||||
|
isNullable: observed.isNullable,
|
||||||
|
defaultExpression: observed.defaultExpression,
|
||||||
|
primaryKeyPosition: observed.primaryKeyPosition,
|
||||||
|
isPrimaryKey: observed.primaryKeyPosition !== null,
|
||||||
|
isForeignKey: false,
|
||||||
|
foreignKeyCount: 0,
|
||||||
|
sourceComment: observed.sourceComment,
|
||||||
|
description: null,
|
||||||
|
generatedDescription: null,
|
||||||
|
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||||
|
lastSyncedAt: now,
|
||||||
|
version: 1,
|
||||||
|
createdAt: now,
|
||||||
|
updatedAt: now,
|
||||||
|
});
|
||||||
|
created += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (scope === "relationships" || scope === "all") {
|
||||||
|
const tables = await this.listTables(databaseId);
|
||||||
|
const tableByName = new Map(tables.map((table) => [table.name, table]));
|
||||||
|
const existing = [...this.relationships.values()].filter((relationship) => relationship.databaseId === databaseId);
|
||||||
|
const existingByKey = new Map(existing.map((relationship) => [`${relationship.sourceTableName}\u0000${relationship.constraintName}`, relationship]));
|
||||||
|
const observedKeys = new Set(snapshot.relationships.map((relationship) => `${relationship.sourceTableName}\u0000${relationship.constraintName}`));
|
||||||
|
for (const relationship of existing) {
|
||||||
|
if (!observedKeys.has(`${relationship.sourceTableName}\u0000${relationship.constraintName}`)) {
|
||||||
|
this.relationships.delete(relationship.id);
|
||||||
|
deleted += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const observed of snapshot.relationships) {
|
||||||
|
const sourceTable = tableByName.get(observed.sourceTableName);
|
||||||
|
const targetTable = tableByName.get(observed.targetTableName);
|
||||||
|
if (!sourceTable || !targetTable) {
|
||||||
|
throw new CatalogConnectorError(`Relationship ${observed.constraintName} refers to an unknown table`);
|
||||||
|
}
|
||||||
|
const pairs = observed.columns.map((pair) => {
|
||||||
|
const source = [...this.columns.values()].find((column) => column.tableId === sourceTable.id && column.name === pair.sourceColumnName);
|
||||||
|
const target = [...this.columns.values()].find((column) => column.tableId === targetTable.id && column.name === pair.targetColumnName);
|
||||||
|
if (!source || !target) {
|
||||||
|
throw new CatalogConnectorError(`Relationship ${observed.constraintName} refers to an unknown column`);
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
position: pair.position,
|
||||||
|
sourceColumnId: source.id,
|
||||||
|
sourceColumnName: source.name,
|
||||||
|
targetColumnId: target.id,
|
||||||
|
targetColumnName: target.name,
|
||||||
|
};
|
||||||
|
}).sort((a, b) => a.position - b.position);
|
||||||
|
const key = `${observed.sourceTableName}\u0000${observed.constraintName}`;
|
||||||
|
const current = existingByKey.get(key);
|
||||||
|
const comparable = current && JSON.stringify({
|
||||||
|
target: current.targetTableName,
|
||||||
|
update: current.updateRule,
|
||||||
|
delete: current.deleteRule,
|
||||||
|
deferrable: current.deferrable,
|
||||||
|
deferred: current.initiallyDeferred,
|
||||||
|
columns: current.columns.map((pair) => [pair.position, pair.sourceColumnName, pair.targetColumnName]),
|
||||||
|
});
|
||||||
|
const nextComparable = JSON.stringify({
|
||||||
|
target: observed.targetTableName,
|
||||||
|
update: observed.updateRule,
|
||||||
|
delete: observed.deleteRule,
|
||||||
|
deferrable: observed.deferrable,
|
||||||
|
deferred: observed.initiallyDeferred,
|
||||||
|
columns: pairs.map((pair) => [pair.position, pair.sourceColumnName, pair.targetColumnName]),
|
||||||
|
});
|
||||||
|
const id = current?.id ?? randomUUID();
|
||||||
|
this.relationships.set(id, {
|
||||||
|
id,
|
||||||
|
databaseId,
|
||||||
|
constraintName: observed.constraintName,
|
||||||
|
sourceTableId: sourceTable.id,
|
||||||
|
sourceTableName: sourceTable.name,
|
||||||
|
targetTableId: targetTable.id,
|
||||||
|
targetTableName: targetTable.name,
|
||||||
|
updateRule: observed.updateRule,
|
||||||
|
deleteRule: observed.deleteRule,
|
||||||
|
deferrable: observed.deferrable,
|
||||||
|
initiallyDeferred: observed.initiallyDeferred,
|
||||||
|
columns: pairs,
|
||||||
|
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||||
|
lastSyncedAt: now,
|
||||||
|
createdAt: current?.createdAt ?? now,
|
||||||
|
updatedAt: comparable === nextComparable ? (current?.updatedAt ?? now) : now,
|
||||||
|
});
|
||||||
|
if (!current) created += 1;
|
||||||
|
else if (comparable !== nextComparable) updated += 1;
|
||||||
|
}
|
||||||
|
this.refreshForeignKeyFlags(databaseId);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (scope === "all") {
|
||||||
|
this.records.set(databaseId, { ...database, schemaSyncedVersion: expectedDatabaseVersion, schemaSyncedAt: now });
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
tables: (await this.listTables(databaseId)).length,
|
||||||
|
columns: [...this.columns.values()].filter((column) => this.tables.get(column.tableId)?.databaseId === databaseId).length,
|
||||||
|
relationships: (await this.listRelationships(databaseId)).length,
|
||||||
|
created,
|
||||||
|
updated,
|
||||||
|
deleted,
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
this.tables.clear();
|
||||||
|
this.columns.clear();
|
||||||
|
this.relationships.clear();
|
||||||
|
for (const [id, table] of tableBackup) this.tables.set(id, table);
|
||||||
|
for (const [id, column] of columnBackup) this.columns.set(id, column);
|
||||||
|
for (const [id, relationship] of relationshipBackup) this.relationships.set(id, relationship);
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async createSyncRun(
|
||||||
|
databaseId: string,
|
||||||
|
scope: CatalogSyncScope,
|
||||||
|
tableIds: readonly string[],
|
||||||
|
requestedDatabaseVersion: number,
|
||||||
|
): Promise<CatalogSyncRun> {
|
||||||
|
const activeStates = new Set<CatalogSyncRun["state"]>(["queued", "running", "awaiting_confirmation", "applying"]);
|
||||||
|
if ([...this.syncRuns.values()].some((run) => run.databaseId === databaseId && activeStates.has(run.state))) {
|
||||||
|
throw new CatalogConflictError("A schema synchronization is already active for this database");
|
||||||
|
}
|
||||||
|
const now = new Date().toISOString();
|
||||||
|
const run: CatalogSyncRun = {
|
||||||
|
id: randomUUID(), databaseId, scope, tableIds: [...tableIds], state: "queued", phase: "queued",
|
||||||
|
requestedDatabaseVersion, observedSnapshot: null, plannedDiff: null, confirmationToken: null,
|
||||||
|
counts: {}, errorCode: null, errorMessage: null, cancelRequested: false,
|
||||||
|
createdAt: now, startedAt: null, updatedAt: now, finishedAt: null, heartbeatAt: null,
|
||||||
|
leaseOwner: null, leaseExpiresAt: null,
|
||||||
|
};
|
||||||
|
this.syncRuns.set(run.id, run);
|
||||||
|
return structuredClone(run);
|
||||||
|
}
|
||||||
|
|
||||||
|
async getSyncRun(runId: string): Promise<CatalogSyncRun | undefined> {
|
||||||
|
const run = this.syncRuns.get(runId);
|
||||||
|
return run ? structuredClone(run) : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
async claimSyncRun(runId: string, workerId: string, leaseExpiresAt: string): Promise<CatalogSyncRun | undefined> {
|
||||||
|
const run = this.syncRuns.get(runId);
|
||||||
|
if (!run || run.state !== "queued") return undefined;
|
||||||
|
if (run.leaseOwner && run.leaseOwner !== workerId && run.leaseExpiresAt && run.leaseExpiresAt > new Date().toISOString()) {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
return await this.updateSyncRun(runId, {
|
||||||
|
state: "running",
|
||||||
|
startedAt: run.startedAt ?? new Date().toISOString(),
|
||||||
|
heartbeatAt: new Date().toISOString(),
|
||||||
|
leaseOwner: workerId,
|
||||||
|
leaseExpiresAt,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async listSyncRuns(databaseId: string, limit = 20): Promise<CatalogSyncRun[]> {
|
||||||
|
return [...this.syncRuns.values()].filter((run) => run.databaseId === databaseId)
|
||||||
|
.sort((a, b) => b.createdAt.localeCompare(a.createdAt)).slice(0, limit).map((run) => structuredClone(run));
|
||||||
|
}
|
||||||
|
|
||||||
|
async updateSyncRun(runId: string, update: CatalogSyncRunUpdate): Promise<CatalogSyncRun | undefined> {
|
||||||
|
const current = this.syncRuns.get(runId);
|
||||||
|
if (!current) return undefined;
|
||||||
|
const updated = { ...current, ...structuredClone(update), updatedAt: new Date().toISOString() };
|
||||||
|
this.syncRuns.set(runId, updated);
|
||||||
|
return structuredClone(updated);
|
||||||
|
}
|
||||||
|
|
||||||
|
async requestSyncRunCancellation(runId: string): Promise<CatalogSyncRun | undefined> {
|
||||||
|
const run = this.syncRuns.get(runId);
|
||||||
|
if (!run) return undefined;
|
||||||
|
if (!["queued", "running", "awaiting_confirmation"].includes(run.state)) return structuredClone(run);
|
||||||
|
return await this.updateSyncRun(runId, { cancelRequested: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
async appendSyncEvent(
|
||||||
|
runId: string,
|
||||||
|
level: CatalogSyncEvent["level"],
|
||||||
|
eventType: string,
|
||||||
|
message: string,
|
||||||
|
data: Record<string, unknown> = {},
|
||||||
|
): Promise<CatalogSyncEvent> {
|
||||||
|
const events = this.syncEvents.get(runId) ?? [];
|
||||||
|
const event: CatalogSyncEvent = {
|
||||||
|
id: [...this.syncEvents.values()].reduce((count, values) => count + values.length, 0) + 1,
|
||||||
|
runId, sequence: events.length + 1, level, eventType, message, data: structuredClone(data),
|
||||||
|
createdAt: new Date().toISOString(),
|
||||||
|
};
|
||||||
|
events.push(event);
|
||||||
|
this.syncEvents.set(runId, events);
|
||||||
|
return structuredClone(event);
|
||||||
|
}
|
||||||
|
|
||||||
|
async listSyncEvents(runId: string, afterSequence = 0): Promise<CatalogSyncEvent[]> {
|
||||||
|
return (this.syncEvents.get(runId) ?? []).filter((event) => event.sequence > afterSequence).map((event) => structuredClone(event));
|
||||||
|
}
|
||||||
|
|
||||||
|
async pruneSyncEvents(before: string): Promise<void> {
|
||||||
|
for (const [runId, events] of this.syncEvents) {
|
||||||
|
this.syncEvents.set(runId, events.filter((event) => event.createdAt >= before));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async interruptActiveSyncRuns(): Promise<void> {
|
||||||
|
for (const run of this.syncRuns.values()) {
|
||||||
|
if (["queued", "running", "awaiting_confirmation", "applying"].includes(run.state)) {
|
||||||
|
await this.updateSyncRun(run.id, {
|
||||||
|
state: "interrupted", phase: "completed", finishedAt: new Date().toISOString(),
|
||||||
|
errorCode: "SYNC_INTERRUPTED", errorMessage: "Synchronization was interrupted by a service restart",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async reconcileTables(
|
||||||
|
databaseId: string,
|
||||||
|
expectedDatabaseVersion: number,
|
||||||
|
observed: readonly ObservedCatalogTable[],
|
||||||
|
confirmedDeletedNames: readonly string[],
|
||||||
|
): Promise<TableSyncRepositoryResult | undefined> {
|
||||||
|
const database = this.records.get(databaseId);
|
||||||
|
if (!database || database.version !== expectedDatabaseVersion) return undefined;
|
||||||
|
const existing = await this.listTables(databaseId);
|
||||||
|
const observedNames = new Set(observed.map((table) => table.name));
|
||||||
|
const deletedNames = existing.filter((table) => !observedNames.has(table.name)).map((table) => table.name).sort();
|
||||||
|
const confirmation = [...new Set(confirmedDeletedNames)].sort();
|
||||||
|
if (deletedNames.length > 0 && JSON.stringify(deletedNames) !== JSON.stringify(confirmation)) {
|
||||||
|
return { kind: "confirmation_required", deletedNames };
|
||||||
|
}
|
||||||
|
|
||||||
|
const byName = new Map(existing.map((table) => [table.name, table]));
|
||||||
|
let createdCount = 0;
|
||||||
|
let updatedCount = 0;
|
||||||
|
for (const observedTable of observed) {
|
||||||
|
const current = byName.get(observedTable.name);
|
||||||
|
const now = new Date().toISOString();
|
||||||
|
if (!current) {
|
||||||
|
const created: CatalogTable = {
|
||||||
|
id: randomUUID(),
|
||||||
|
databaseId,
|
||||||
|
name: observedTable.name,
|
||||||
|
sourceComment: observedTable.sourceComment,
|
||||||
|
description: null,
|
||||||
|
generatedDescription: null,
|
||||||
|
lastSyncedDatabaseVersion: expectedDatabaseVersion,
|
||||||
|
lastSyncedAt: now,
|
||||||
|
version: 1,
|
||||||
|
createdAt: now,
|
||||||
|
updatedAt: now,
|
||||||
|
};
|
||||||
|
this.tables.set(created.id, created);
|
||||||
|
createdCount += 1;
|
||||||
|
} else if (current.sourceComment !== observedTable.sourceComment) {
|
||||||
|
this.tables.set(current.id, {
|
||||||
|
...current,
|
||||||
|
sourceComment: observedTable.sourceComment,
|
||||||
|
version: current.version + 1,
|
||||||
|
updatedAt: now,
|
||||||
|
});
|
||||||
|
updatedCount += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const table of existing) {
|
||||||
|
if (deletedNames.includes(table.name)) this.deleteTable(table.id);
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
kind: "applied",
|
||||||
|
createdCount,
|
||||||
|
updatedCount,
|
||||||
|
deletedCount: deletedNames.length,
|
||||||
|
tables: await this.listTables(databaseId),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private assertSnapshotCapability(scope: CatalogSyncScope, snapshot: ObservedSchemaSnapshot): void {
|
||||||
|
const required = scope === "all" ? ["tables", "columns", "relationships"] as const : [scope] as const;
|
||||||
|
for (const capability of required) {
|
||||||
|
if (snapshot.capabilities[capability] !== "available") {
|
||||||
|
throw new CatalogConnectorError(`Schema introspection capability '${capability}' is unavailable`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private deleteTable(tableId: string): void {
|
||||||
|
this.tables.delete(tableId);
|
||||||
|
for (const column of [...this.columns.values()]) if (column.tableId === tableId) this.deleteColumn(column.id);
|
||||||
|
for (const relationship of [...this.relationships.values()]) {
|
||||||
|
if (relationship.sourceTableId === tableId || relationship.targetTableId === tableId) {
|
||||||
|
this.relationships.delete(relationship.id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private deleteColumn(columnId: string): void {
|
||||||
|
this.columns.delete(columnId);
|
||||||
|
for (const relationship of [...this.relationships.values()]) {
|
||||||
|
if (relationship.columns.some((pair) => pair.sourceColumnId === columnId || pair.targetColumnId === columnId)) {
|
||||||
|
this.relationships.delete(relationship.id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private refreshForeignKeyFlags(databaseId: string): void {
|
||||||
|
const counts = new Map<string, number>();
|
||||||
|
for (const relationship of this.relationships.values()) {
|
||||||
|
if (relationship.databaseId !== databaseId) continue;
|
||||||
|
for (const pair of relationship.columns) counts.set(pair.sourceColumnId, (counts.get(pair.sourceColumnId) ?? 0) + 1);
|
||||||
|
}
|
||||||
|
for (const column of this.columns.values()) {
|
||||||
|
if (this.tables.get(column.tableId)?.databaseId !== databaseId) continue;
|
||||||
|
const foreignKeyCount = counts.get(column.id) ?? 0;
|
||||||
|
this.columns.set(column.id, { ...column, isForeignKey: foreignKeyCount > 0, foreignKeyCount });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async available(): Promise<boolean> { return true; }
|
||||||
|
}
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
import { CamelCasePlugin, Kysely, PostgresDialect } from "kysely";
|
||||||
|
import { Migrator, type MigrationProvider } from "kysely/migration";
|
||||||
|
import { Pool } from "pg";
|
||||||
|
import { readFile } from "node:fs/promises";
|
||||||
|
import type { CatalogDatabase } from "./repository.js";
|
||||||
|
import * as initialMigration from "./migrations/001_workspace_databases.js";
|
||||||
|
import * as catalogTablesMigration from "./migrations/002_catalog_tables.js";
|
||||||
|
import * as catalogSchemaSyncMigration from "./migrations/003_catalog_schema_sync.js";
|
||||||
|
import * as catalogRuntimeSequencePrivilegesMigration from "./migrations/004_catalog_runtime_sequence_privileges.js";
|
||||||
|
|
||||||
|
const connectionString = process.env.THT_CATALOG_MIGRATOR_DATABASE_URL;
|
||||||
|
const host = process.env.THT_CATALOG_DB_HOST;
|
||||||
|
const database = process.env.THT_CATALOG_DB_NAME;
|
||||||
|
const user = process.env.THT_CATALOG_MIGRATOR_USER;
|
||||||
|
const passwordFile = process.env.THT_CATALOG_MIGRATOR_PASSWORD_FILE;
|
||||||
|
if (!connectionString && (!host || !database || !user || !passwordFile)) {
|
||||||
|
throw new Error("catalog migrator database configuration is required");
|
||||||
|
}
|
||||||
|
const pool = new Pool(connectionString ? { connectionString, max: 1 } : {
|
||||||
|
host,
|
||||||
|
port: Number(process.env.THT_CATALOG_DB_PORT ?? 5432),
|
||||||
|
database,
|
||||||
|
user,
|
||||||
|
password: async () => (await readFile(passwordFile!, "utf8")).trim(),
|
||||||
|
max: 1,
|
||||||
|
});
|
||||||
|
|
||||||
|
const db = new Kysely<CatalogDatabase>({
|
||||||
|
dialect: new PostgresDialect({ pool }),
|
||||||
|
plugins: [new CamelCasePlugin()],
|
||||||
|
});
|
||||||
|
const provider: MigrationProvider = {
|
||||||
|
async getMigrations() {
|
||||||
|
return {
|
||||||
|
"001_workspace_databases": initialMigration,
|
||||||
|
"002_catalog_tables": catalogTablesMigration,
|
||||||
|
"003_catalog_schema_sync": catalogSchemaSyncMigration,
|
||||||
|
"004_catalog_runtime_sequence_privileges": catalogRuntimeSequencePrivilegesMigration,
|
||||||
|
};
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await new Migrator({ db, provider }).migrateToLatest();
|
||||||
|
for (const item of result.results ?? []) {
|
||||||
|
process.stdout.write(`${item.migrationName}: ${item.status}\n`);
|
||||||
|
}
|
||||||
|
if (result.error) throw result.error;
|
||||||
|
} finally {
|
||||||
|
await db.destroy();
|
||||||
|
}
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
import { sql, type Kysely } from "kysely";
|
||||||
|
import type { CatalogDatabase } from "../repository.js";
|
||||||
|
|
||||||
|
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||||
|
await db.schema.createTable("workspace_databases")
|
||||||
|
.addColumn("id", "uuid", (column) => column.primaryKey())
|
||||||
|
.addColumn("workspace_id", "text", (column) => column.notNull().unique())
|
||||||
|
.addColumn("engine", "text", (column) => column.notNull())
|
||||||
|
.addColumn("database_name", "text", (column) => column.notNull())
|
||||||
|
.addColumn("schema_name", "text", (column) => column.notNull())
|
||||||
|
.addColumn("version", "integer", (column) => column.notNull().defaultTo(1))
|
||||||
|
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addCheckConstraint("workspace_databases_engine_check", sql`engine = 'postgres'`)
|
||||||
|
.addCheckConstraint("workspace_databases_version_check", sql`version > 0`)
|
||||||
|
.execute();
|
||||||
|
|
||||||
|
await db.schema.createTable("database_bindings")
|
||||||
|
.addColumn("database_id", "uuid", (column) => column.primaryKey()
|
||||||
|
.references("workspace_databases.id").onDelete("cascade"))
|
||||||
|
.addColumn("transport", "text", (column) => column.notNull())
|
||||||
|
.addColumn("host", "text")
|
||||||
|
.addColumn("port", "integer")
|
||||||
|
.addColumn("username", "text")
|
||||||
|
.addColumn("base_url", "text")
|
||||||
|
.addColumn("rest_path", "text")
|
||||||
|
.addColumn("rest_auth", "text")
|
||||||
|
.addColumn("tls_servername", "text")
|
||||||
|
.addColumn("ssh_host", "text")
|
||||||
|
.addColumn("ssh_port", "integer")
|
||||||
|
.addColumn("ssh_username", "text")
|
||||||
|
.addColumn("ssh_target_host", "text")
|
||||||
|
.addColumn("ssh_target_port", "integer")
|
||||||
|
.addColumn("connection_status", "text", (column) => column.notNull().defaultTo("untested"))
|
||||||
|
.addColumn("tested_version", "integer")
|
||||||
|
.addColumn("last_tested_at", "timestamptz")
|
||||||
|
.addColumn("last_error_code", "text")
|
||||||
|
.addColumn("last_error_message", "text")
|
||||||
|
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addCheckConstraint("database_bindings_transport_check", sql`transport in ('postgres_direct', 'rest_api', 'ssh_tunnel')`)
|
||||||
|
.addCheckConstraint("database_bindings_status_check", sql`connection_status in ('untested', 'reachable', 'failed')`)
|
||||||
|
.addCheckConstraint("database_bindings_port_check", sql`port is null or port between 1 and 65535`)
|
||||||
|
.addCheckConstraint("database_bindings_ssh_port_check", sql`ssh_port is null or ssh_port between 1 and 65535`)
|
||||||
|
.addCheckConstraint("database_bindings_ssh_target_port_check", sql`ssh_target_port is null or ssh_target_port between 1 and 65535`)
|
||||||
|
.addCheckConstraint("database_bindings_transport_fields_check", sql`
|
||||||
|
(transport = 'postgres_direct' and host is not null and port is not null and username is not null)
|
||||||
|
or (transport = 'rest_api' and base_url is not null and rest_path is not null and rest_auth is not null)
|
||||||
|
or (transport = 'ssh_tunnel' and username is not null and ssh_host is not null
|
||||||
|
and ssh_port is not null and ssh_username is not null and ssh_target_host is not null
|
||||||
|
and ssh_target_port is not null)
|
||||||
|
`)
|
||||||
|
.execute();
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||||
|
await db.schema.dropTable("database_bindings").execute();
|
||||||
|
await db.schema.dropTable("workspace_databases").execute();
|
||||||
|
}
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
import { sql, type Kysely } from "kysely";
|
||||||
|
import type { CatalogDatabase } from "../repository.js";
|
||||||
|
|
||||||
|
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||||
|
await db.schema.createTable("catalog_tables")
|
||||||
|
.addColumn("id", "uuid", (column) => column.primaryKey())
|
||||||
|
.addColumn("database_id", "uuid", (column) => column.notNull()
|
||||||
|
.references("workspace_databases.id").onDelete("cascade"))
|
||||||
|
.addColumn("name", "text", (column) => column.notNull())
|
||||||
|
.addColumn("source_comment", "text")
|
||||||
|
.addColumn("description", "text")
|
||||||
|
.addColumn("generated_description", "text")
|
||||||
|
.addColumn("version", "integer", (column) => column.notNull().defaultTo(1))
|
||||||
|
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addUniqueConstraint("catalog_tables_database_name_key", ["database_id", "name"])
|
||||||
|
.addCheckConstraint("catalog_tables_name_check", sql`char_length(name) between 1 and 128`)
|
||||||
|
.addCheckConstraint("catalog_tables_version_check", sql`version > 0`)
|
||||||
|
.execute();
|
||||||
|
|
||||||
|
await db.schema.createIndex("catalog_tables_database_id_idx")
|
||||||
|
.on("catalog_tables").column("database_id").execute();
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||||
|
await db.schema.dropTable("catalog_tables").execute();
|
||||||
|
}
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
import { sql, type Kysely } from "kysely";
|
||||||
|
import type { CatalogDatabase } from "../repository.js";
|
||||||
|
|
||||||
|
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||||
|
await db.schema.alterTable("workspace_databases")
|
||||||
|
.addColumn("schema_synced_version", "integer")
|
||||||
|
.addColumn("schema_synced_at", "timestamptz")
|
||||||
|
.execute();
|
||||||
|
|
||||||
|
await db.schema.alterTable("catalog_tables")
|
||||||
|
.addColumn("last_synced_database_version", "integer")
|
||||||
|
.addColumn("last_synced_at", "timestamptz")
|
||||||
|
.execute();
|
||||||
|
|
||||||
|
await db.schema.createTable("catalog_columns")
|
||||||
|
.addColumn("id", "uuid", (column) => column.primaryKey())
|
||||||
|
.addColumn("table_id", "uuid", (column) => column.notNull()
|
||||||
|
.references("catalog_tables.id").onDelete("cascade"))
|
||||||
|
.addColumn("name", "text", (column) => column.notNull())
|
||||||
|
.addColumn("ordinal_position", "integer", (column) => column.notNull())
|
||||||
|
.addColumn("data_type", "text", (column) => column.notNull())
|
||||||
|
.addColumn("is_nullable", "boolean", (column) => column.notNull())
|
||||||
|
.addColumn("default_expression", "text")
|
||||||
|
.addColumn("primary_key_position", "integer")
|
||||||
|
.addColumn("source_comment", "text")
|
||||||
|
.addColumn("description", "text")
|
||||||
|
.addColumn("generated_description", "text")
|
||||||
|
.addColumn("last_synced_database_version", "integer")
|
||||||
|
.addColumn("last_synced_at", "timestamptz")
|
||||||
|
.addColumn("version", "integer", (column) => column.notNull().defaultTo(1))
|
||||||
|
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addUniqueConstraint("catalog_columns_table_name_key", ["table_id", "name"])
|
||||||
|
.addCheckConstraint("catalog_columns_name_check", sql`char_length(name) between 1 and 128`)
|
||||||
|
.addCheckConstraint("catalog_columns_ordinal_check", sql`ordinal_position > 0`)
|
||||||
|
.addCheckConstraint("catalog_columns_pk_position_check", sql`primary_key_position is null or primary_key_position > 0`)
|
||||||
|
.addCheckConstraint("catalog_columns_version_check", sql`version > 0`)
|
||||||
|
.execute();
|
||||||
|
await db.schema.createIndex("catalog_columns_table_id_idx")
|
||||||
|
.on("catalog_columns").column("table_id").execute();
|
||||||
|
|
||||||
|
await db.schema.createTable("catalog_relationships")
|
||||||
|
.addColumn("id", "uuid", (column) => column.primaryKey())
|
||||||
|
.addColumn("database_id", "uuid", (column) => column.notNull()
|
||||||
|
.references("workspace_databases.id").onDelete("cascade"))
|
||||||
|
.addColumn("constraint_name", "text", (column) => column.notNull())
|
||||||
|
.addColumn("source_table_id", "uuid", (column) => column.notNull()
|
||||||
|
.references("catalog_tables.id").onDelete("cascade"))
|
||||||
|
.addColumn("target_table_id", "uuid", (column) => column.notNull()
|
||||||
|
.references("catalog_tables.id").onDelete("cascade"))
|
||||||
|
.addColumn("update_rule", "text", (column) => column.notNull())
|
||||||
|
.addColumn("delete_rule", "text", (column) => column.notNull())
|
||||||
|
.addColumn("deferrable", "boolean", (column) => column.notNull().defaultTo(false))
|
||||||
|
.addColumn("initially_deferred", "boolean", (column) => column.notNull().defaultTo(false))
|
||||||
|
.addColumn("last_synced_database_version", "integer")
|
||||||
|
.addColumn("last_synced_at", "timestamptz")
|
||||||
|
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addUniqueConstraint("catalog_relationships_source_constraint_key", ["source_table_id", "constraint_name"])
|
||||||
|
.execute();
|
||||||
|
await db.schema.createIndex("catalog_relationships_database_id_idx")
|
||||||
|
.on("catalog_relationships").column("database_id").execute();
|
||||||
|
|
||||||
|
await db.schema.createTable("catalog_relationship_columns")
|
||||||
|
.addColumn("relationship_id", "uuid", (column) => column.notNull()
|
||||||
|
.references("catalog_relationships.id").onDelete("cascade"))
|
||||||
|
.addColumn("position", "integer", (column) => column.notNull())
|
||||||
|
.addColumn("source_column_id", "uuid", (column) => column.notNull()
|
||||||
|
.references("catalog_columns.id").onDelete("cascade"))
|
||||||
|
.addColumn("target_column_id", "uuid", (column) => column.notNull()
|
||||||
|
.references("catalog_columns.id").onDelete("cascade"))
|
||||||
|
.addPrimaryKeyConstraint("catalog_relationship_columns_pkey", ["relationship_id", "position"])
|
||||||
|
.addCheckConstraint("catalog_relationship_columns_position_check", sql`position > 0`)
|
||||||
|
.execute();
|
||||||
|
|
||||||
|
await db.schema.createTable("catalog_sync_runs")
|
||||||
|
.addColumn("id", "uuid", (column) => column.primaryKey())
|
||||||
|
.addColumn("database_id", "uuid", (column) => column.notNull()
|
||||||
|
.references("workspace_databases.id").onDelete("cascade"))
|
||||||
|
.addColumn("scope", "text", (column) => column.notNull())
|
||||||
|
.addColumn("table_ids", "jsonb", (column) => column.notNull().defaultTo(sql`'[]'::jsonb`))
|
||||||
|
.addColumn("state", "text", (column) => column.notNull())
|
||||||
|
.addColumn("phase", "text", (column) => column.notNull())
|
||||||
|
.addColumn("requested_database_version", "integer", (column) => column.notNull())
|
||||||
|
.addColumn("observed_snapshot", "jsonb")
|
||||||
|
.addColumn("planned_diff", "jsonb")
|
||||||
|
.addColumn("confirmation_token", "text")
|
||||||
|
.addColumn("counts", "jsonb", (column) => column.notNull().defaultTo(sql`'{}'::jsonb`))
|
||||||
|
.addColumn("error_code", "text")
|
||||||
|
.addColumn("error_message", "text")
|
||||||
|
.addColumn("cancel_requested", "boolean", (column) => column.notNull().defaultTo(false))
|
||||||
|
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addColumn("started_at", "timestamptz")
|
||||||
|
.addColumn("updated_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addColumn("finished_at", "timestamptz")
|
||||||
|
.addColumn("heartbeat_at", "timestamptz")
|
||||||
|
.addColumn("lease_owner", "text")
|
||||||
|
.addColumn("lease_expires_at", "timestamptz")
|
||||||
|
.execute();
|
||||||
|
await db.schema.createIndex("catalog_sync_runs_database_created_idx")
|
||||||
|
.on("catalog_sync_runs").columns(["database_id", "created_at"]).execute();
|
||||||
|
await sql`CREATE UNIQUE INDEX catalog_sync_runs_one_active_per_database
|
||||||
|
ON catalog_sync_runs (database_id)
|
||||||
|
WHERE state IN ('queued', 'running', 'awaiting_confirmation', 'applying')`.execute(db);
|
||||||
|
|
||||||
|
await db.schema.createTable("catalog_sync_events")
|
||||||
|
.addColumn("id", "bigserial", (column) => column.primaryKey())
|
||||||
|
.addColumn("run_id", "uuid", (column) => column.notNull()
|
||||||
|
.references("catalog_sync_runs.id").onDelete("cascade"))
|
||||||
|
.addColumn("sequence", "integer", (column) => column.notNull())
|
||||||
|
.addColumn("level", "text", (column) => column.notNull())
|
||||||
|
.addColumn("event_type", "text", (column) => column.notNull())
|
||||||
|
.addColumn("message", "text", (column) => column.notNull())
|
||||||
|
.addColumn("data", "jsonb", (column) => column.notNull().defaultTo(sql`'{}'::jsonb`))
|
||||||
|
.addColumn("created_at", "timestamptz", (column) => column.notNull().defaultTo(sql`now()`))
|
||||||
|
.addUniqueConstraint("catalog_sync_events_run_sequence_key", ["run_id", "sequence"])
|
||||||
|
.execute();
|
||||||
|
await db.schema.createIndex("catalog_sync_events_run_id_idx")
|
||||||
|
.on("catalog_sync_events").columns(["run_id", "sequence"]).execute();
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||||
|
await db.schema.dropTable("catalog_sync_events").execute();
|
||||||
|
await db.schema.dropTable("catalog_sync_runs").execute();
|
||||||
|
await db.schema.dropTable("catalog_relationship_columns").execute();
|
||||||
|
await db.schema.dropTable("catalog_relationships").execute();
|
||||||
|
await db.schema.dropTable("catalog_columns").execute();
|
||||||
|
await db.schema.alterTable("catalog_tables")
|
||||||
|
.dropColumn("last_synced_database_version")
|
||||||
|
.dropColumn("last_synced_at")
|
||||||
|
.execute();
|
||||||
|
await db.schema.alterTable("workspace_databases")
|
||||||
|
.dropColumn("schema_synced_version")
|
||||||
|
.dropColumn("schema_synced_at")
|
||||||
|
.execute();
|
||||||
|
}
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
import { type Kysely, sql } from "kysely";
|
||||||
|
import type { CatalogDatabase } from "../repository.js";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `catalog_sync_events.id` is the first catalog-owned identity sequence.
|
||||||
|
* Table default privileges do not cover sequences, and without USAGE the
|
||||||
|
* runtime can create a run but cannot append its first event.
|
||||||
|
*/
|
||||||
|
export async function up(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||||
|
await sql`DO $catalog_privileges$
|
||||||
|
BEGIN
|
||||||
|
IF EXISTS (SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = 'thothii_catalog_runtime') THEN
|
||||||
|
EXECUTE 'GRANT USAGE, SELECT ON SEQUENCE catalog_sync_events_id_seq TO thothii_catalog_runtime';
|
||||||
|
EXECUTE 'ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO thothii_catalog_runtime';
|
||||||
|
END IF;
|
||||||
|
END
|
||||||
|
$catalog_privileges$`.execute(db);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function down(db: Kysely<CatalogDatabase>): Promise<void> {
|
||||||
|
await sql`DO $catalog_privileges$
|
||||||
|
BEGIN
|
||||||
|
IF EXISTS (SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = 'thothii_catalog_runtime') THEN
|
||||||
|
EXECUTE 'ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE USAGE, SELECT ON SEQUENCES FROM thothii_catalog_runtime';
|
||||||
|
EXECUTE 'REVOKE USAGE, SELECT ON SEQUENCE catalog_sync_events_id_seq FROM thothii_catalog_runtime';
|
||||||
|
END IF;
|
||||||
|
END
|
||||||
|
$catalog_privileges$`.execute(db);
|
||||||
|
}
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
import { CatalogOperationInProgressError } from "./types.js";
|
||||||
|
|
||||||
|
/** Serializes connection tests and schema synchronization for each catalog database. */
|
||||||
|
export class CatalogOperationCoordinator {
|
||||||
|
private readonly active = new Set<string>();
|
||||||
|
|
||||||
|
reserve(databaseId: string): () => void {
|
||||||
|
if (this.active.has(databaseId)) {
|
||||||
|
throw new CatalogOperationInProgressError("A database operation is already in progress");
|
||||||
|
}
|
||||||
|
this.active.add(databaseId);
|
||||||
|
let released = false;
|
||||||
|
return () => {
|
||||||
|
if (released) return;
|
||||||
|
released = true;
|
||||||
|
this.active.delete(databaseId);
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async run<T>(databaseId: string, operation: () => Promise<T>): Promise<T> {
|
||||||
|
const release = this.reserve(databaseId);
|
||||||
|
try {
|
||||||
|
return await operation();
|
||||||
|
} finally {
|
||||||
|
release();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,258 @@
|
|||||||
|
import { spawn, type ChildProcessWithoutNullStreams } from "node:child_process";
|
||||||
|
import { readFile } from "node:fs/promises";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
import { Duplex } from "node:stream";
|
||||||
|
import { setTimeout as delay } from "node:timers/promises";
|
||||||
|
import { Client, type ClientConfig } from "pg";
|
||||||
|
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
|
||||||
|
import { CATALOG_SECRET_IDS } from "./secrets.js";
|
||||||
|
import { CatalogConnectorError, type WorkspaceDatabase } from "./types.js";
|
||||||
|
|
||||||
|
export interface CatalogDatabaseClient {
|
||||||
|
query(sql: string, values: readonly unknown[]): Promise<{ rows: Array<Record<string, unknown>> }>;
|
||||||
|
end(): Promise<void>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogPostgresAccess {
|
||||||
|
connect(database: WorkspaceDatabase, signal: AbortSignal): Promise<CatalogDatabaseClient>;
|
||||||
|
}
|
||||||
|
|
||||||
|
type SpawnSsh = (
|
||||||
|
command: string,
|
||||||
|
args: readonly string[],
|
||||||
|
options: { env: NodeJS.ProcessEnv },
|
||||||
|
) => ChildProcessWithoutNullStreams;
|
||||||
|
|
||||||
|
interface AccessDependencies {
|
||||||
|
createClient?: (config: ClientConfig) => Client;
|
||||||
|
spawnSsh?: SpawnSsh;
|
||||||
|
sshBinary?: string;
|
||||||
|
askpassPath?: string;
|
||||||
|
connectTimeoutMs?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
function required(value: string | number | undefined): string | number {
|
||||||
|
if (value === undefined || value === "") throw new CatalogConnectorError("Database binding is incomplete");
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function connectionSsl(ca: string | undefined, servername: string | undefined): ClientConfig["ssl"] {
|
||||||
|
if (!ca && !servername) return false;
|
||||||
|
return {
|
||||||
|
...(ca ? { ca } : {}),
|
||||||
|
...(servername ? { servername } : {}),
|
||||||
|
rejectUnauthorized: true,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function buildSshArguments(input: {
|
||||||
|
sshHost: string;
|
||||||
|
sshPort: number;
|
||||||
|
sshUsername: string;
|
||||||
|
targetHost: string;
|
||||||
|
targetPort: number;
|
||||||
|
privateKeyFile: string;
|
||||||
|
knownHostsFile: string;
|
||||||
|
passphraseFile?: string;
|
||||||
|
connectTimeoutMs: number;
|
||||||
|
}): string[] {
|
||||||
|
const batchMode = input.passphraseFile ? "no" : "yes";
|
||||||
|
return [
|
||||||
|
"-F", "/dev/null",
|
||||||
|
"-T",
|
||||||
|
"-o", `BatchMode=${batchMode}`,
|
||||||
|
"-o", "StrictHostKeyChecking=yes",
|
||||||
|
"-o", `UserKnownHostsFile=${input.knownHostsFile}`,
|
||||||
|
"-o", "GlobalKnownHostsFile=/dev/null",
|
||||||
|
"-o", "IdentitiesOnly=yes",
|
||||||
|
"-o", "IdentityAgent=none",
|
||||||
|
"-o", `IdentityFile=${input.privateKeyFile}`,
|
||||||
|
"-o", "PreferredAuthentications=publickey",
|
||||||
|
"-o", "PasswordAuthentication=no",
|
||||||
|
"-o", "KbdInteractiveAuthentication=no",
|
||||||
|
"-o", "ConnectionAttempts=1",
|
||||||
|
"-o", `ConnectTimeout=${Math.max(1, Math.ceil(input.connectTimeoutMs / 1_000))}`,
|
||||||
|
"-o", "ServerAliveInterval=5",
|
||||||
|
"-o", "ServerAliveCountMax=1",
|
||||||
|
"-o", "NumberOfPasswordPrompts=1",
|
||||||
|
"-o", "RequestTTY=no",
|
||||||
|
"-o", "LogLevel=ERROR",
|
||||||
|
"-p", String(input.sshPort),
|
||||||
|
"-W", `${input.targetHost}:${input.targetPort}`,
|
||||||
|
"--", `${input.sshUsername}@${input.sshHost}`,
|
||||||
|
];
|
||||||
|
}
|
||||||
|
|
||||||
|
async function stopChild(child: ChildProcessWithoutNullStreams): Promise<void> {
|
||||||
|
if (child.exitCode !== null || child.signalCode !== null) return;
|
||||||
|
child.kill("SIGTERM");
|
||||||
|
await Promise.race([
|
||||||
|
new Promise<void>((resolve) => child.once("exit", () => resolve())),
|
||||||
|
delay(500).then(() => undefined),
|
||||||
|
]);
|
||||||
|
if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL");
|
||||||
|
}
|
||||||
|
|
||||||
|
function sshDuplex(child: ChildProcessWithoutNullStreams): Duplex {
|
||||||
|
let ended = false;
|
||||||
|
let stream: Duplex;
|
||||||
|
const forward = () => {
|
||||||
|
let chunk: Buffer | string | null;
|
||||||
|
while ((chunk = child.stdout.read() as Buffer | string | null) !== null) {
|
||||||
|
if (!stream.push(chunk)) break;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
const finish = () => {
|
||||||
|
if (ended) return;
|
||||||
|
ended = true;
|
||||||
|
stream.push(null);
|
||||||
|
};
|
||||||
|
const fail = (error: Error) => stream.destroy(error);
|
||||||
|
stream = new Duplex({
|
||||||
|
read: forward,
|
||||||
|
write: (chunk, encoding, callback) => child.stdin.write(chunk, encoding, callback),
|
||||||
|
final: (callback) => child.stdin.end(callback),
|
||||||
|
destroy: (error, callback) => {
|
||||||
|
child.stdout.off("readable", forward);
|
||||||
|
child.stdout.off("end", finish);
|
||||||
|
child.stdout.off("error", fail);
|
||||||
|
child.stdin.off("error", fail);
|
||||||
|
callback(error);
|
||||||
|
},
|
||||||
|
});
|
||||||
|
child.stdout.on("readable", forward);
|
||||||
|
child.stdout.once("end", finish);
|
||||||
|
child.stdout.once("error", fail);
|
||||||
|
child.stdin.once("error", fail);
|
||||||
|
return stream;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Deep connection module for direct and SSH-forwarded PostgreSQL access. It owns secret leases,
|
||||||
|
* TLS, OpenSSH lifecycle, abort propagation, and pg cleanup behind one connect interface.
|
||||||
|
*/
|
||||||
|
export class ConcreteCatalogPostgresAccess implements CatalogPostgresAccess {
|
||||||
|
private readonly createClient: (config: ClientConfig) => Client;
|
||||||
|
private readonly spawnSsh: SpawnSsh;
|
||||||
|
private readonly sshBinary: string;
|
||||||
|
private readonly askpassPath: string;
|
||||||
|
private readonly connectTimeoutMs: number;
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private readonly secretStore: WorkspaceSecretStore,
|
||||||
|
dependencies: AccessDependencies = {},
|
||||||
|
) {
|
||||||
|
this.createClient = dependencies.createClient ?? ((config) => new Client(config));
|
||||||
|
this.spawnSsh = dependencies.spawnSsh ?? ((command, args, options) => (
|
||||||
|
spawn(command, [...args], { ...options, stdio: ["pipe", "pipe", "pipe"] })
|
||||||
|
));
|
||||||
|
this.sshBinary = dependencies.sshBinary ?? process.env.THT_SSH_BIN ?? "ssh";
|
||||||
|
this.askpassPath = dependencies.askpassPath
|
||||||
|
?? process.env.THT_SSH_ASKPASS_BIN
|
||||||
|
?? fileURLToPath(new URL("../../scripts/ssh-askpass.mjs", import.meta.url));
|
||||||
|
this.connectTimeoutMs = dependencies.connectTimeoutMs ?? 5_000;
|
||||||
|
}
|
||||||
|
|
||||||
|
async connect(database: WorkspaceDatabase, signal: AbortSignal): Promise<CatalogDatabaseClient> {
|
||||||
|
if (database.binding.transport === "rest_api") {
|
||||||
|
throw new CatalogConnectorError("REST is not a PostgreSQL wire binding");
|
||||||
|
}
|
||||||
|
const ids: string[] = [CATALOG_SECRET_IDS.password, CATALOG_SECRET_IDS.tlsCa];
|
||||||
|
if (database.binding.transport === "ssh_tunnel") {
|
||||||
|
ids.push(
|
||||||
|
CATALOG_SECRET_IDS.sshPrivateKey,
|
||||||
|
CATALOG_SECRET_IDS.sshPrivateKeyPassphrase,
|
||||||
|
CATALOG_SECRET_IDS.sshKnownHosts,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const materialized = this.secretStore.materialize(database.workspaceId, ids);
|
||||||
|
let child: ChildProcessWithoutNullStreams | undefined;
|
||||||
|
let stream: Duplex | undefined;
|
||||||
|
let client: Client | undefined;
|
||||||
|
let ended = false;
|
||||||
|
const close = async () => {
|
||||||
|
if (ended) return;
|
||||||
|
ended = true;
|
||||||
|
signal.removeEventListener("abort", abort);
|
||||||
|
if (client) await client.end().catch(() => undefined);
|
||||||
|
stream?.destroy();
|
||||||
|
if (child) await stopChild(child);
|
||||||
|
materialized.release();
|
||||||
|
};
|
||||||
|
const abort = () => { void close(); };
|
||||||
|
|
||||||
|
try {
|
||||||
|
const passwordFile = materialized.files.get(CATALOG_SECRET_IDS.password);
|
||||||
|
if (!passwordFile) throw new CatalogConnectorError("Database password is not configured");
|
||||||
|
const password = await readFile(passwordFile, "utf8");
|
||||||
|
const tlsCaFile = materialized.files.get(CATALOG_SECRET_IDS.tlsCa);
|
||||||
|
const tlsCa = tlsCaFile ? await readFile(tlsCaFile, "utf8") : undefined;
|
||||||
|
let host: string;
|
||||||
|
let port: number;
|
||||||
|
|
||||||
|
if (database.binding.transport === "ssh_tunnel") {
|
||||||
|
const privateKeyFile = materialized.files.get(CATALOG_SECRET_IDS.sshPrivateKey);
|
||||||
|
const knownHostsFile = materialized.files.get(CATALOG_SECRET_IDS.sshKnownHosts);
|
||||||
|
if (!privateKeyFile || !knownHostsFile) {
|
||||||
|
throw new CatalogConnectorError("SSH private key and known hosts are required");
|
||||||
|
}
|
||||||
|
const passphraseFile = materialized.files.get(CATALOG_SECRET_IDS.sshPrivateKeyPassphrase);
|
||||||
|
host = String(required(database.binding.sshTargetHost));
|
||||||
|
port = Number(required(database.binding.sshTargetPort));
|
||||||
|
const args = buildSshArguments({
|
||||||
|
sshHost: String(required(database.binding.sshHost)),
|
||||||
|
sshPort: Number(required(database.binding.sshPort)),
|
||||||
|
sshUsername: String(required(database.binding.sshUsername)),
|
||||||
|
targetHost: host,
|
||||||
|
targetPort: port,
|
||||||
|
privateKeyFile,
|
||||||
|
knownHostsFile,
|
||||||
|
passphraseFile,
|
||||||
|
connectTimeoutMs: this.connectTimeoutMs,
|
||||||
|
});
|
||||||
|
child = this.spawnSsh(this.sshBinary, args, {
|
||||||
|
env: {
|
||||||
|
...process.env,
|
||||||
|
LC_ALL: "C",
|
||||||
|
...(passphraseFile ? {
|
||||||
|
DISPLAY: "thothii",
|
||||||
|
SSH_ASKPASS: this.askpassPath,
|
||||||
|
SSH_ASKPASS_REQUIRE: "force",
|
||||||
|
THT_SSH_PASSPHRASE_FILE: passphraseFile,
|
||||||
|
} : {}),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
stream = sshDuplex(child);
|
||||||
|
child.once("error", () => stream?.destroy(new CatalogConnectorError("SSH process failed")));
|
||||||
|
child.once("exit", (code) => {
|
||||||
|
if (!ended && code !== 0) stream?.destroy(new CatalogConnectorError("SSH tunnel failed"));
|
||||||
|
});
|
||||||
|
child.stderr.on("data", () => undefined);
|
||||||
|
} else {
|
||||||
|
host = String(required(database.binding.host));
|
||||||
|
port = Number(required(database.binding.port));
|
||||||
|
}
|
||||||
|
|
||||||
|
client = this.createClient({
|
||||||
|
host,
|
||||||
|
port,
|
||||||
|
database: database.databaseName,
|
||||||
|
user: String(required(database.binding.username)),
|
||||||
|
password,
|
||||||
|
ssl: connectionSsl(tlsCa, database.binding.tlsServername),
|
||||||
|
connectionTimeoutMillis: this.connectTimeoutMs,
|
||||||
|
...(stream ? { stream: () => stream } : {}),
|
||||||
|
});
|
||||||
|
signal.addEventListener("abort", abort, { once: true });
|
||||||
|
await client.connect();
|
||||||
|
return {
|
||||||
|
query: async (sql, values) => await client!.query(sql, [...values]),
|
||||||
|
end: close,
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
await close();
|
||||||
|
if (error instanceof CatalogConnectorError) throw error;
|
||||||
|
throw new CatalogConnectorError("PostgreSQL connector failed");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,498 @@
|
|||||||
|
import { readFile } from "node:fs/promises";
|
||||||
|
import { z } from "zod";
|
||||||
|
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
|
||||||
|
import type { CatalogPostgresAccess } from "./postgres-access.js";
|
||||||
|
import { CATALOG_SECRET_IDS } from "./secrets.js";
|
||||||
|
import {
|
||||||
|
CatalogConnectorError,
|
||||||
|
CatalogSchemaCapabilityUnavailableError,
|
||||||
|
type CatalogSyncPhase,
|
||||||
|
type ObservedCatalogColumn,
|
||||||
|
type ObservedCatalogRelationship,
|
||||||
|
type ObservedCatalogTable,
|
||||||
|
type ObservedSchemaSnapshot,
|
||||||
|
type WorkspaceDatabase,
|
||||||
|
} from "./types.js";
|
||||||
|
|
||||||
|
export type CatalogSchemaScanProgress = (
|
||||||
|
phase: Extract<CatalogSyncPhase, "connecting" | "scanning_tables" | "scanning_columns" | "scanning_relationships">,
|
||||||
|
counts?: { tables?: number; columns?: number; relationships?: number },
|
||||||
|
) => Promise<void> | void;
|
||||||
|
|
||||||
|
export interface CatalogSchemaIntrospector {
|
||||||
|
scan(
|
||||||
|
database: WorkspaceDatabase,
|
||||||
|
signal: AbortSignal,
|
||||||
|
progress?: CatalogSchemaScanProgress,
|
||||||
|
): Promise<ObservedSchemaSnapshot>;
|
||||||
|
}
|
||||||
|
|
||||||
|
const identifier = z.string().min(1).max(128);
|
||||||
|
const nullableText = z.string().nullable();
|
||||||
|
const capability = z.enum(["available", "unavailable"]);
|
||||||
|
const restSnapshotSchema = z.object({
|
||||||
|
schemaVersion: z.literal(1),
|
||||||
|
capabilities: z.object({
|
||||||
|
tables: capability,
|
||||||
|
columns: capability,
|
||||||
|
relationships: capability,
|
||||||
|
}).strict(),
|
||||||
|
tables: z.array(z.object({
|
||||||
|
name: identifier,
|
||||||
|
sourceComment: nullableText,
|
||||||
|
}).strict()),
|
||||||
|
columns: z.array(z.object({
|
||||||
|
tableName: identifier,
|
||||||
|
name: identifier,
|
||||||
|
ordinalPosition: z.number().int().positive(),
|
||||||
|
dataType: z.string().min(1).max(2_000),
|
||||||
|
isNullable: z.boolean(),
|
||||||
|
defaultExpression: nullableText,
|
||||||
|
primaryKeyPosition: z.number().int().positive().nullable(),
|
||||||
|
sourceComment: nullableText,
|
||||||
|
}).strict()),
|
||||||
|
relationships: z.array(z.object({
|
||||||
|
constraintName: identifier,
|
||||||
|
sourceTableName: identifier,
|
||||||
|
targetTableName: identifier,
|
||||||
|
updateRule: z.string().min(1).max(64),
|
||||||
|
deleteRule: z.string().min(1).max(64),
|
||||||
|
deferrable: z.boolean(),
|
||||||
|
initiallyDeferred: z.boolean(),
|
||||||
|
columns: z.array(z.object({
|
||||||
|
position: z.number().int().positive(),
|
||||||
|
sourceColumnName: identifier,
|
||||||
|
targetColumnName: identifier,
|
||||||
|
}).strict()).min(1),
|
||||||
|
}).strict()),
|
||||||
|
}).strict();
|
||||||
|
|
||||||
|
function sqlString(value: string): string {
|
||||||
|
return `'${value.replaceAll("'", "''")}'`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function restSnapshotQuery(schemaName: string): string {
|
||||||
|
const schema = sqlString(schemaName);
|
||||||
|
return `WITH target_schema AS (
|
||||||
|
SELECT oid
|
||||||
|
FROM pg_catalog.pg_namespace
|
||||||
|
WHERE nspname = ${schema}
|
||||||
|
),
|
||||||
|
observed_tables AS (
|
||||||
|
SELECT c.oid,
|
||||||
|
c.relname AS name,
|
||||||
|
d.description AS source_comment
|
||||||
|
FROM pg_catalog.pg_class c
|
||||||
|
JOIN target_schema n ON n.oid = c.relnamespace
|
||||||
|
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = 0
|
||||||
|
WHERE c.relkind IN ('r', 'p')
|
||||||
|
),
|
||||||
|
primary_key_columns AS (
|
||||||
|
SELECT i.indrelid AS table_oid,
|
||||||
|
key.attnum,
|
||||||
|
key.ordinality::integer AS position
|
||||||
|
FROM pg_catalog.pg_index i
|
||||||
|
CROSS JOIN LATERAL unnest(i.indkey) WITH ORDINALITY AS key(attnum, ordinality)
|
||||||
|
WHERE i.indisprimary
|
||||||
|
),
|
||||||
|
observed_columns AS (
|
||||||
|
SELECT table_info.name AS table_name,
|
||||||
|
a.attname AS name,
|
||||||
|
a.attnum::integer AS ordinal_position,
|
||||||
|
pg_catalog.format_type(a.atttypid, a.atttypmod) AS data_type,
|
||||||
|
NOT a.attnotnull AS is_nullable,
|
||||||
|
pg_catalog.pg_get_expr(ad.adbin, ad.adrelid) AS default_expression,
|
||||||
|
pk.position AS primary_key_position,
|
||||||
|
d.description AS source_comment
|
||||||
|
FROM observed_tables table_info
|
||||||
|
JOIN pg_catalog.pg_attribute a ON a.attrelid = table_info.oid
|
||||||
|
LEFT JOIN pg_catalog.pg_attrdef ad ON ad.adrelid = table_info.oid AND ad.adnum = a.attnum
|
||||||
|
LEFT JOIN pg_catalog.pg_description d ON d.objoid = table_info.oid AND d.objsubid = a.attnum
|
||||||
|
LEFT JOIN primary_key_columns pk ON pk.table_oid = table_info.oid AND pk.attnum = a.attnum
|
||||||
|
WHERE a.attnum > 0
|
||||||
|
AND NOT a.attisdropped
|
||||||
|
),
|
||||||
|
relationship_pairs AS (
|
||||||
|
SELECT con.oid AS constraint_oid,
|
||||||
|
con.conname AS constraint_name,
|
||||||
|
source_table.relname AS source_table_name,
|
||||||
|
target_table.relname AS target_table_name,
|
||||||
|
CASE con.confupdtype
|
||||||
|
WHEN 'a' THEN 'NO ACTION'
|
||||||
|
WHEN 'r' THEN 'RESTRICT'
|
||||||
|
WHEN 'c' THEN 'CASCADE'
|
||||||
|
WHEN 'n' THEN 'SET NULL'
|
||||||
|
WHEN 'd' THEN 'SET DEFAULT'
|
||||||
|
END AS update_rule,
|
||||||
|
CASE con.confdeltype
|
||||||
|
WHEN 'a' THEN 'NO ACTION'
|
||||||
|
WHEN 'r' THEN 'RESTRICT'
|
||||||
|
WHEN 'c' THEN 'CASCADE'
|
||||||
|
WHEN 'n' THEN 'SET NULL'
|
||||||
|
WHEN 'd' THEN 'SET DEFAULT'
|
||||||
|
END AS delete_rule,
|
||||||
|
con.condeferrable AS is_deferrable,
|
||||||
|
con.condeferred AS initially_deferred,
|
||||||
|
source_key.ordinality::integer AS position,
|
||||||
|
source_column.attname AS source_column_name,
|
||||||
|
target_column.attname AS target_column_name
|
||||||
|
FROM pg_catalog.pg_constraint con
|
||||||
|
JOIN pg_catalog.pg_class source_table ON source_table.oid = con.conrelid
|
||||||
|
JOIN target_schema source_namespace ON source_namespace.oid = source_table.relnamespace
|
||||||
|
JOIN pg_catalog.pg_class target_table ON target_table.oid = con.confrelid
|
||||||
|
JOIN target_schema target_namespace ON target_namespace.oid = target_table.relnamespace
|
||||||
|
JOIN LATERAL unnest(con.conkey) WITH ORDINALITY AS source_key(attnum, ordinality) ON true
|
||||||
|
JOIN LATERAL unnest(con.confkey) WITH ORDINALITY AS target_key(attnum, ordinality)
|
||||||
|
ON target_key.ordinality = source_key.ordinality
|
||||||
|
JOIN pg_catalog.pg_attribute source_column
|
||||||
|
ON source_column.attrelid = source_table.oid AND source_column.attnum = source_key.attnum
|
||||||
|
JOIN pg_catalog.pg_attribute target_column
|
||||||
|
ON target_column.attrelid = target_table.oid AND target_column.attnum = target_key.attnum
|
||||||
|
WHERE con.contype = 'f'
|
||||||
|
),
|
||||||
|
observed_relationships AS (
|
||||||
|
SELECT constraint_oid,
|
||||||
|
constraint_name,
|
||||||
|
source_table_name,
|
||||||
|
target_table_name,
|
||||||
|
update_rule,
|
||||||
|
delete_rule,
|
||||||
|
is_deferrable,
|
||||||
|
initially_deferred,
|
||||||
|
pg_catalog.jsonb_agg(
|
||||||
|
pg_catalog.jsonb_build_object(
|
||||||
|
'position', position,
|
||||||
|
'sourceColumnName', source_column_name,
|
||||||
|
'targetColumnName', target_column_name
|
||||||
|
) ORDER BY position
|
||||||
|
) AS columns
|
||||||
|
FROM relationship_pairs
|
||||||
|
GROUP BY constraint_oid, constraint_name, source_table_name, target_table_name,
|
||||||
|
update_rule, delete_rule, is_deferrable, initially_deferred
|
||||||
|
)
|
||||||
|
SELECT 1 AS "schemaVersion",
|
||||||
|
pg_catalog.jsonb_build_object(
|
||||||
|
'tables', 'available',
|
||||||
|
'columns', 'available',
|
||||||
|
'relationships', 'available'
|
||||||
|
) AS capabilities,
|
||||||
|
COALESCE((
|
||||||
|
SELECT pg_catalog.jsonb_agg(
|
||||||
|
pg_catalog.jsonb_build_object('name', name, 'sourceComment', source_comment)
|
||||||
|
ORDER BY name
|
||||||
|
)
|
||||||
|
FROM observed_tables
|
||||||
|
), '[]'::jsonb) AS tables,
|
||||||
|
COALESCE((
|
||||||
|
SELECT pg_catalog.jsonb_agg(
|
||||||
|
pg_catalog.jsonb_build_object(
|
||||||
|
'tableName', table_name,
|
||||||
|
'name', name,
|
||||||
|
'ordinalPosition', ordinal_position,
|
||||||
|
'dataType', data_type,
|
||||||
|
'isNullable', is_nullable,
|
||||||
|
'defaultExpression', default_expression,
|
||||||
|
'primaryKeyPosition', primary_key_position,
|
||||||
|
'sourceComment', source_comment
|
||||||
|
) ORDER BY table_name, ordinal_position
|
||||||
|
)
|
||||||
|
FROM observed_columns
|
||||||
|
), '[]'::jsonb) AS columns,
|
||||||
|
COALESCE((
|
||||||
|
SELECT pg_catalog.jsonb_agg(
|
||||||
|
pg_catalog.jsonb_build_object(
|
||||||
|
'constraintName', constraint_name,
|
||||||
|
'sourceTableName', source_table_name,
|
||||||
|
'targetTableName', target_table_name,
|
||||||
|
'updateRule', update_rule,
|
||||||
|
'deleteRule', delete_rule,
|
||||||
|
'deferrable', is_deferrable,
|
||||||
|
'initiallyDeferred', initially_deferred,
|
||||||
|
'columns', columns
|
||||||
|
) ORDER BY source_table_name, constraint_name
|
||||||
|
)
|
||||||
|
FROM observed_relationships
|
||||||
|
), '[]'::jsonb) AS relationships
|
||||||
|
FROM target_schema`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function required(value: string | undefined): string {
|
||||||
|
if (!value) throw new CatalogConnectorError("Database binding is incomplete");
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function textOrNull(value: unknown): string | null {
|
||||||
|
return typeof value === "string" && value.length > 0 ? value : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function actionRule(value: unknown): string {
|
||||||
|
const rules: Record<string, string> = {
|
||||||
|
a: "NO ACTION",
|
||||||
|
r: "RESTRICT",
|
||||||
|
c: "CASCADE",
|
||||||
|
n: "SET NULL",
|
||||||
|
d: "SET DEFAULT",
|
||||||
|
};
|
||||||
|
const rule = rules[String(value)];
|
||||||
|
if (!rule) throw new CatalogConnectorError("Schema introspection returned an unknown relationship action");
|
||||||
|
return rule;
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalized(snapshot: ObservedSchemaSnapshot): ObservedSchemaSnapshot {
|
||||||
|
const tables = new Map<string, ObservedCatalogTable>();
|
||||||
|
for (const table of snapshot.tables) {
|
||||||
|
if (tables.has(table.name)) throw new CatalogConnectorError("Schema introspection returned duplicate tables");
|
||||||
|
tables.set(table.name, table);
|
||||||
|
}
|
||||||
|
const columns = new Map<string, ObservedCatalogColumn>();
|
||||||
|
for (const column of snapshot.columns) {
|
||||||
|
const key = `${column.tableName}\u0000${column.name}`;
|
||||||
|
if (columns.has(key)) throw new CatalogConnectorError("Schema introspection returned duplicate columns");
|
||||||
|
columns.set(key, column);
|
||||||
|
}
|
||||||
|
const relationships = new Map<string, ObservedCatalogRelationship>();
|
||||||
|
for (const relationship of snapshot.relationships) {
|
||||||
|
const key = `${relationship.sourceTableName}\u0000${relationship.constraintName}`;
|
||||||
|
if (relationships.has(key)) throw new CatalogConnectorError("Schema introspection returned duplicate relationships");
|
||||||
|
relationships.set(key, {
|
||||||
|
...relationship,
|
||||||
|
columns: [...relationship.columns].sort((a, b) => a.position - b.position),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
schemaVersion: 1,
|
||||||
|
capabilities: snapshot.capabilities,
|
||||||
|
tables: [...tables.values()].sort((a, b) => a.name.localeCompare(b.name)),
|
||||||
|
columns: [...columns.values()].sort((a, b) => (
|
||||||
|
a.tableName.localeCompare(b.tableName) || a.ordinalPosition - b.ordinalPosition
|
||||||
|
)),
|
||||||
|
relationships: [...relationships.values()].sort((a, b) => (
|
||||||
|
a.sourceTableName.localeCompare(b.sourceTableName) || a.constraintName.localeCompare(b.constraintName)
|
||||||
|
)),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export class ConcreteCatalogSchemaIntrospector implements CatalogSchemaIntrospector {
|
||||||
|
constructor(
|
||||||
|
private readonly postgres: CatalogPostgresAccess,
|
||||||
|
private readonly secretStore: WorkspaceSecretStore,
|
||||||
|
) {}
|
||||||
|
|
||||||
|
async scan(
|
||||||
|
database: WorkspaceDatabase,
|
||||||
|
signal: AbortSignal,
|
||||||
|
progress?: CatalogSchemaScanProgress,
|
||||||
|
): Promise<ObservedSchemaSnapshot> {
|
||||||
|
return database.binding.transport === "rest_api"
|
||||||
|
? await this.scanRest(database, signal, progress)
|
||||||
|
: await this.scanPostgres(database, signal, progress);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async scanPostgres(
|
||||||
|
database: WorkspaceDatabase,
|
||||||
|
signal: AbortSignal,
|
||||||
|
progress?: CatalogSchemaScanProgress,
|
||||||
|
): Promise<ObservedSchemaSnapshot> {
|
||||||
|
await progress?.("connecting");
|
||||||
|
const client = await this.postgres.connect(database, signal);
|
||||||
|
try {
|
||||||
|
const schema = await client.query(
|
||||||
|
"SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname = $1) AS present",
|
||||||
|
[database.schema],
|
||||||
|
);
|
||||||
|
if (schema.rows[0]?.present !== true) throw new CatalogConnectorError("Database schema is unavailable");
|
||||||
|
|
||||||
|
await progress?.("scanning_tables");
|
||||||
|
const tableResult = await client.query(
|
||||||
|
`SELECT c.relname AS name, d.description AS source_comment
|
||||||
|
FROM pg_catalog.pg_class c
|
||||||
|
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
|
||||||
|
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = 0
|
||||||
|
WHERE c.relkind IN ('r', 'p') AND n.nspname = $1
|
||||||
|
ORDER BY c.relname`,
|
||||||
|
[database.schema],
|
||||||
|
);
|
||||||
|
const tables: ObservedCatalogTable[] = tableResult.rows.map((row) => ({
|
||||||
|
name: String(row.name),
|
||||||
|
sourceComment: textOrNull(row.source_comment),
|
||||||
|
}));
|
||||||
|
await progress?.("scanning_tables", { tables: tables.length });
|
||||||
|
|
||||||
|
await progress?.("scanning_columns", { tables: tables.length });
|
||||||
|
const columnResult = await client.query(
|
||||||
|
`SELECT c.relname AS table_name,
|
||||||
|
a.attname AS name,
|
||||||
|
a.attnum::integer AS ordinal_position,
|
||||||
|
pg_catalog.format_type(a.atttypid, a.atttypmod) AS data_type,
|
||||||
|
NOT a.attnotnull AS is_nullable,
|
||||||
|
pg_catalog.pg_get_expr(ad.adbin, ad.adrelid) AS default_expression,
|
||||||
|
pk.position AS primary_key_position,
|
||||||
|
d.description AS source_comment
|
||||||
|
FROM pg_catalog.pg_class c
|
||||||
|
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
|
||||||
|
JOIN pg_catalog.pg_attribute a ON a.attrelid = c.oid
|
||||||
|
LEFT JOIN pg_catalog.pg_attrdef ad ON ad.adrelid = c.oid AND ad.adnum = a.attnum
|
||||||
|
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = a.attnum
|
||||||
|
LEFT JOIN LATERAL (
|
||||||
|
SELECT key.ordinality::integer AS position
|
||||||
|
FROM pg_catalog.pg_index i
|
||||||
|
CROSS JOIN LATERAL unnest(i.indkey) WITH ORDINALITY AS key(attnum, ordinality)
|
||||||
|
WHERE i.indrelid = c.oid AND i.indisprimary AND key.attnum = a.attnum
|
||||||
|
LIMIT 1
|
||||||
|
) pk ON true
|
||||||
|
WHERE c.relkind IN ('r', 'p')
|
||||||
|
AND n.nspname = $1
|
||||||
|
AND a.attnum > 0
|
||||||
|
AND NOT a.attisdropped
|
||||||
|
ORDER BY c.relname, a.attnum`,
|
||||||
|
[database.schema],
|
||||||
|
);
|
||||||
|
const columns: ObservedCatalogColumn[] = columnResult.rows.map((row) => ({
|
||||||
|
tableName: String(row.table_name),
|
||||||
|
name: String(row.name),
|
||||||
|
ordinalPosition: Number(row.ordinal_position),
|
||||||
|
dataType: String(row.data_type),
|
||||||
|
isNullable: row.is_nullable === true,
|
||||||
|
defaultExpression: textOrNull(row.default_expression),
|
||||||
|
primaryKeyPosition: row.primary_key_position === null || row.primary_key_position === undefined
|
||||||
|
? null
|
||||||
|
: Number(row.primary_key_position),
|
||||||
|
sourceComment: textOrNull(row.source_comment),
|
||||||
|
}));
|
||||||
|
await progress?.("scanning_columns", { tables: tables.length, columns: columns.length });
|
||||||
|
|
||||||
|
await progress?.("scanning_relationships", { tables: tables.length, columns: columns.length });
|
||||||
|
const relationshipResult = await client.query(
|
||||||
|
`SELECT con.conname AS constraint_name,
|
||||||
|
source_table.relname AS source_table_name,
|
||||||
|
target_table.relname AS target_table_name,
|
||||||
|
con.confupdtype AS update_action,
|
||||||
|
con.confdeltype AS delete_action,
|
||||||
|
con.condeferrable AS deferrable,
|
||||||
|
con.condeferred AS initially_deferred,
|
||||||
|
source_key.ordinality::integer AS position,
|
||||||
|
source_column.attname AS source_column_name,
|
||||||
|
target_column.attname AS target_column_name
|
||||||
|
FROM pg_catalog.pg_constraint con
|
||||||
|
JOIN pg_catalog.pg_class source_table ON source_table.oid = con.conrelid
|
||||||
|
JOIN pg_catalog.pg_namespace source_namespace ON source_namespace.oid = source_table.relnamespace
|
||||||
|
JOIN pg_catalog.pg_class target_table ON target_table.oid = con.confrelid
|
||||||
|
JOIN pg_catalog.pg_namespace target_namespace ON target_namespace.oid = target_table.relnamespace
|
||||||
|
JOIN LATERAL unnest(con.conkey) WITH ORDINALITY AS source_key(attnum, ordinality) ON true
|
||||||
|
JOIN LATERAL unnest(con.confkey) WITH ORDINALITY AS target_key(attnum, ordinality)
|
||||||
|
ON target_key.ordinality = source_key.ordinality
|
||||||
|
JOIN pg_catalog.pg_attribute source_column
|
||||||
|
ON source_column.attrelid = source_table.oid AND source_column.attnum = source_key.attnum
|
||||||
|
JOIN pg_catalog.pg_attribute target_column
|
||||||
|
ON target_column.attrelid = target_table.oid AND target_column.attnum = target_key.attnum
|
||||||
|
WHERE con.contype = 'f'
|
||||||
|
AND source_namespace.nspname = $1
|
||||||
|
AND target_namespace.nspname = $1
|
||||||
|
ORDER BY source_table.relname, con.conname, source_key.ordinality`,
|
||||||
|
[database.schema],
|
||||||
|
);
|
||||||
|
const relationshipMap = new Map<string, ObservedCatalogRelationship>();
|
||||||
|
for (const row of relationshipResult.rows) {
|
||||||
|
const sourceTableName = String(row.source_table_name);
|
||||||
|
const constraintName = String(row.constraint_name);
|
||||||
|
const key = `${sourceTableName}\u0000${constraintName}`;
|
||||||
|
const current = relationshipMap.get(key) ?? {
|
||||||
|
constraintName,
|
||||||
|
sourceTableName,
|
||||||
|
targetTableName: String(row.target_table_name),
|
||||||
|
updateRule: actionRule(row.update_action),
|
||||||
|
deleteRule: actionRule(row.delete_action),
|
||||||
|
deferrable: row.deferrable === true,
|
||||||
|
initiallyDeferred: row.initially_deferred === true,
|
||||||
|
columns: [],
|
||||||
|
};
|
||||||
|
current.columns.push({
|
||||||
|
position: Number(row.position),
|
||||||
|
sourceColumnName: String(row.source_column_name),
|
||||||
|
targetColumnName: String(row.target_column_name),
|
||||||
|
});
|
||||||
|
relationshipMap.set(key, current);
|
||||||
|
}
|
||||||
|
const relationships = [...relationshipMap.values()];
|
||||||
|
await progress?.("scanning_relationships", {
|
||||||
|
tables: tables.length,
|
||||||
|
columns: columns.length,
|
||||||
|
relationships: relationships.length,
|
||||||
|
});
|
||||||
|
return normalized({
|
||||||
|
schemaVersion: 1,
|
||||||
|
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||||
|
tables,
|
||||||
|
columns,
|
||||||
|
relationships,
|
||||||
|
});
|
||||||
|
} finally {
|
||||||
|
await client.end();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async scanRest(
|
||||||
|
database: WorkspaceDatabase,
|
||||||
|
signal: AbortSignal,
|
||||||
|
progress?: CatalogSchemaScanProgress,
|
||||||
|
): Promise<ObservedSchemaSnapshot> {
|
||||||
|
await progress?.("connecting");
|
||||||
|
const auth = database.binding.restAuth ?? "bearer";
|
||||||
|
const materialized = this.secretStore.materialize(
|
||||||
|
database.workspaceId,
|
||||||
|
auth === "none" ? [] : [CATALOG_SECRET_IDS.apiKey],
|
||||||
|
);
|
||||||
|
try {
|
||||||
|
const headers: Record<string, string> = { "content-type": "application/json" };
|
||||||
|
if (auth !== "none") {
|
||||||
|
const credentialFile = materialized.files.get(CATALOG_SECRET_IDS.apiKey);
|
||||||
|
if (!credentialFile) throw new CatalogConnectorError("REST API key is not configured");
|
||||||
|
const credential = (await readFile(credentialFile, "utf8")).trim();
|
||||||
|
if (auth === "bearer") headers.authorization = `Bearer ${credential}`;
|
||||||
|
else headers["x-api-key"] = credential;
|
||||||
|
}
|
||||||
|
const baseUrl = required(database.binding.baseUrl).replace(/\/+$/, "");
|
||||||
|
const response = await fetch(`${baseUrl}/rpc/schema_snapshot`, {
|
||||||
|
method: "POST",
|
||||||
|
headers,
|
||||||
|
body: JSON.stringify({ schema_name: database.schema }),
|
||||||
|
signal,
|
||||||
|
});
|
||||||
|
let body: unknown;
|
||||||
|
if (response.ok) {
|
||||||
|
body = await response.json();
|
||||||
|
} else if (response.status === 404) {
|
||||||
|
const fallback = await fetch(`${baseUrl}/rpc/run_query`, {
|
||||||
|
method: "POST",
|
||||||
|
headers,
|
||||||
|
body: JSON.stringify({ query_text: restSnapshotQuery(database.schema) }),
|
||||||
|
signal,
|
||||||
|
});
|
||||||
|
if (!fallback.ok) throw new CatalogSchemaCapabilityUnavailableError("schema_snapshot");
|
||||||
|
const rows: unknown = await fallback.json();
|
||||||
|
if (!Array.isArray(rows) || rows.length !== 1) {
|
||||||
|
throw new CatalogConnectorError("REST schema snapshot fallback is invalid");
|
||||||
|
}
|
||||||
|
body = rows[0];
|
||||||
|
} else {
|
||||||
|
throw new CatalogSchemaCapabilityUnavailableError("schema_snapshot");
|
||||||
|
}
|
||||||
|
const parsed = restSnapshotSchema.safeParse(body);
|
||||||
|
if (!parsed.success) throw new CatalogConnectorError("REST schema snapshot is invalid");
|
||||||
|
const snapshot = normalized(parsed.data);
|
||||||
|
await progress?.("scanning_tables", { tables: snapshot.tables.length });
|
||||||
|
await progress?.("scanning_columns", { tables: snapshot.tables.length, columns: snapshot.columns.length });
|
||||||
|
await progress?.("scanning_relationships", {
|
||||||
|
tables: snapshot.tables.length,
|
||||||
|
columns: snapshot.columns.length,
|
||||||
|
relationships: snapshot.relationships.length,
|
||||||
|
});
|
||||||
|
return snapshot;
|
||||||
|
} catch (error) {
|
||||||
|
if (error instanceof CatalogConnectorError) throw error;
|
||||||
|
throw new CatalogConnectorError("REST schema introspection failed");
|
||||||
|
} finally {
|
||||||
|
materialized.release();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
export const CATALOG_SECRET_IDS = {
|
||||||
|
password: "catalog.dwh.password",
|
||||||
|
apiKey: "catalog.dwh.api_key",
|
||||||
|
sshPrivateKey: "catalog.dwh.ssh_private_key",
|
||||||
|
sshPrivateKeyPassphrase: "catalog.dwh.ssh_private_key_passphrase",
|
||||||
|
sshKnownHosts: "catalog.dwh.ssh_known_hosts",
|
||||||
|
tlsCa: "catalog.dwh.tls_ca",
|
||||||
|
} as const;
|
||||||
|
|
||||||
|
export type CatalogSecretName = keyof typeof CATALOG_SECRET_IDS;
|
||||||
@@ -0,0 +1,228 @@
|
|||||||
|
import { buildInstallationContract } from "../workspaces/contracts.js";
|
||||||
|
import { resolveBinding } from "../workspaces/bindings.js";
|
||||||
|
import type { WorkspaceRegistry } from "../workspaces/registry.js";
|
||||||
|
import type { WorkspaceDescriptor } from "../workspaces/schema.js";
|
||||||
|
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
|
||||||
|
import { createConcreteDiagnosticAdapters } from "../workspaces/diagnostics.js";
|
||||||
|
import { CatalogOperationCoordinator } from "./operation-coordinator.js";
|
||||||
|
import {
|
||||||
|
ConcreteCatalogPostgresAccess,
|
||||||
|
type CatalogPostgresAccess,
|
||||||
|
} from "./postgres-access.js";
|
||||||
|
import type {
|
||||||
|
CatalogRepository,
|
||||||
|
DatabaseBinding,
|
||||||
|
DatabaseConfigurationInput,
|
||||||
|
DatabaseTestResult,
|
||||||
|
WorkspaceDatabase,
|
||||||
|
} from "./types.js";
|
||||||
|
import { CATALOG_SECRET_IDS, type CatalogSecretName } from "./secrets.js";
|
||||||
|
|
||||||
|
export { CATALOG_SECRET_IDS, type CatalogSecretName } from "./secrets.js";
|
||||||
|
|
||||||
|
export interface CatalogListItem extends Omit<WorkspaceDatabase, "id"> {
|
||||||
|
id?: string;
|
||||||
|
workspaceName: string;
|
||||||
|
workspaceDescription?: string;
|
||||||
|
workspaceAvailable: boolean;
|
||||||
|
configured: boolean;
|
||||||
|
secrets: Record<CatalogSecretName, boolean>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function bindingValue(workspace: WorkspaceDescriptor, values: Record<string, string>, suffix: string) {
|
||||||
|
const variable = buildInstallationContract(workspace).variables.find((entry) => (
|
||||||
|
entry.role === "DWH" && entry.suffix === suffix
|
||||||
|
));
|
||||||
|
return variable ? values[variable.name] : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
function numeric(value: string | undefined): number | undefined {
|
||||||
|
if (!value) return undefined;
|
||||||
|
const parsed = Number(value);
|
||||||
|
return Number.isInteger(parsed) && parsed >= 1 && parsed <= 65_535 ? parsed : undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
function yamlBinding(workspace: WorkspaceDescriptor, secretRoots: readonly string[]): DatabaseBinding {
|
||||||
|
const effective = resolveBinding(workspace, "DWH", process.env, secretRoots);
|
||||||
|
const value = (suffix: string) => bindingValue(workspace, effective.values, suffix);
|
||||||
|
return {
|
||||||
|
transport: effective.transport,
|
||||||
|
host: value("HOST"),
|
||||||
|
port: numeric(value("PORT")) ?? workspace.dwh.port,
|
||||||
|
username: value("USER"),
|
||||||
|
baseUrl: value("BASE_URL"),
|
||||||
|
restPath: workspace.diagnostics?.dwh_rest?.path ?? "/health",
|
||||||
|
restAuth: workspace.diagnostics?.dwh_rest?.auth ?? "bearer",
|
||||||
|
tlsServername: value("TLS_SERVERNAME"),
|
||||||
|
sshHost: value("SSH_HOST"),
|
||||||
|
sshPort: numeric(value("SSH_PORT")),
|
||||||
|
sshUsername: value("SSH_USER"),
|
||||||
|
sshTargetHost: value("SSH_TARGET_HOST"),
|
||||||
|
sshTargetPort: numeric(value("SSH_TARGET_PORT")),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function secretState(store: WorkspaceSecretStore, workspaceId: string): Record<CatalogSecretName, boolean> {
|
||||||
|
return Object.fromEntries(Object.entries(CATALOG_SECRET_IDS).map(([name, id]) => (
|
||||||
|
[name, store.has(workspaceId, id)]
|
||||||
|
))) as Record<CatalogSecretName, boolean>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export class CatalogService {
|
||||||
|
private readonly adapters = createConcreteDiagnosticAdapters();
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private readonly repository: CatalogRepository,
|
||||||
|
private readonly registry: WorkspaceRegistry,
|
||||||
|
private readonly secretStore: WorkspaceSecretStore,
|
||||||
|
private readonly secretRoots: readonly string[],
|
||||||
|
private readonly diagnosticTimeoutMs: number,
|
||||||
|
private readonly postgres: CatalogPostgresAccess = new ConcreteCatalogPostgresAccess(secretStore, {
|
||||||
|
connectTimeoutMs: diagnosticTimeoutMs,
|
||||||
|
}),
|
||||||
|
private readonly operations: CatalogOperationCoordinator = new CatalogOperationCoordinator(),
|
||||||
|
) {}
|
||||||
|
|
||||||
|
async list(): Promise<CatalogListItem[]> {
|
||||||
|
const [workspaces, configured] = await Promise.all([
|
||||||
|
this.registry.listCatalog(),
|
||||||
|
this.repository.list(),
|
||||||
|
]);
|
||||||
|
const byWorkspace = new Map(configured.map((database) => [database.workspaceId, database]));
|
||||||
|
const active = await Promise.all(workspaces.map(async (entry) => {
|
||||||
|
const database = byWorkspace.get(entry.id);
|
||||||
|
const { workspace } = await this.registry.read(entry.id);
|
||||||
|
const base = database ?? {
|
||||||
|
workspaceId: entry.id,
|
||||||
|
engine: "postgres" as const,
|
||||||
|
databaseName: workspace.dwh.database,
|
||||||
|
schema: workspace.dwh.schema,
|
||||||
|
version: 0,
|
||||||
|
createdAt: "",
|
||||||
|
updatedAt: "",
|
||||||
|
binding: yamlBinding(workspace, this.secretRoots),
|
||||||
|
connectionStatus: "untested" as const,
|
||||||
|
};
|
||||||
|
return {
|
||||||
|
...base,
|
||||||
|
workspaceName: entry.name,
|
||||||
|
workspaceDescription: entry.description,
|
||||||
|
workspaceAvailable: true,
|
||||||
|
configured: database !== undefined,
|
||||||
|
secrets: secretState(this.secretStore, entry.id),
|
||||||
|
};
|
||||||
|
}));
|
||||||
|
const known = new Set(workspaces.map((entry) => entry.id));
|
||||||
|
const orphaned: CatalogListItem[] = configured
|
||||||
|
.filter((database) => !known.has(database.workspaceId))
|
||||||
|
.map((database) => ({
|
||||||
|
...database,
|
||||||
|
workspaceName: database.workspaceId,
|
||||||
|
workspaceDescription: "Workspace is no longer present in the repository catalog.",
|
||||||
|
workspaceAvailable: false,
|
||||||
|
configured: true,
|
||||||
|
secrets: secretState(this.secretStore, database.workspaceId),
|
||||||
|
}));
|
||||||
|
return [...active, ...orphaned];
|
||||||
|
}
|
||||||
|
|
||||||
|
async ensureWorkspace(workspaceId: string): Promise<WorkspaceDescriptor> {
|
||||||
|
const { workspace } = await this.registry.read(workspaceId);
|
||||||
|
return workspace;
|
||||||
|
}
|
||||||
|
|
||||||
|
async normalizeInput(input: DatabaseConfigurationInput): Promise<DatabaseConfigurationInput> {
|
||||||
|
const workspace = await this.ensureWorkspace(input.workspaceId);
|
||||||
|
if (input.binding.transport !== "rest_api") return input;
|
||||||
|
return {
|
||||||
|
...input,
|
||||||
|
binding: {
|
||||||
|
...input.binding,
|
||||||
|
restPath: workspace.diagnostics?.dwh_rest?.path ?? "/health",
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
configuredSecrets(workspaceId: string): Record<CatalogSecretName, boolean> {
|
||||||
|
return secretState(this.secretStore, workspaceId);
|
||||||
|
}
|
||||||
|
|
||||||
|
replaceSecrets(workspaceId: string, values: Partial<Record<CatalogSecretName, string>>): void {
|
||||||
|
const encoded: Record<string, string> = {};
|
||||||
|
for (const [name, value] of Object.entries(values) as Array<[CatalogSecretName, string | undefined]>) {
|
||||||
|
if (value !== undefined && value.length > 0) encoded[CATALOG_SECRET_IDS[name]] = value;
|
||||||
|
}
|
||||||
|
if (Object.keys(encoded).length > 0) this.secretStore.putMany(workspaceId, encoded);
|
||||||
|
}
|
||||||
|
|
||||||
|
forgetSecrets(workspaceId: string): void {
|
||||||
|
for (const id of Object.values(CATALOG_SECRET_IDS)) this.secretStore.forget(workspaceId, id);
|
||||||
|
}
|
||||||
|
|
||||||
|
async test(database: WorkspaceDatabase): Promise<DatabaseTestResult> {
|
||||||
|
return await this.operations.run(database.id, async () => {
|
||||||
|
const testedAt = new Date().toISOString();
|
||||||
|
const controller = new AbortController();
|
||||||
|
const timer = setTimeout(() => controller.abort(), this.diagnosticTimeoutMs);
|
||||||
|
const required = database.binding.transport === "rest_api"
|
||||||
|
? database.binding.restAuth === "none" ? [] : [CATALOG_SECRET_IDS.apiKey]
|
||||||
|
: [];
|
||||||
|
const materialized = this.secretStore.materialize(database.workspaceId, required);
|
||||||
|
try {
|
||||||
|
if (database.binding.transport !== "rest_api") {
|
||||||
|
const client = await this.postgres.connect(database, controller.signal);
|
||||||
|
try {
|
||||||
|
const result = await client.query(
|
||||||
|
`SELECT current_database() AS database,
|
||||||
|
CASE WHEN pg_catalog.has_schema_privilege(
|
||||||
|
current_user,
|
||||||
|
(SELECT oid FROM pg_catalog.pg_namespace WHERE nspname = $1),
|
||||||
|
'USAGE'
|
||||||
|
) THEN $1 ELSE NULL END AS schema`,
|
||||||
|
[database.schema],
|
||||||
|
);
|
||||||
|
const row = result.rows[0];
|
||||||
|
if (row?.database !== database.databaseName || row.schema !== database.schema) {
|
||||||
|
throw new Error("Database identity mismatch");
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
await client.end();
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
const credentialId = CATALOG_SECRET_IDS.apiKey;
|
||||||
|
await this.adapters.probeConnector({
|
||||||
|
role: "dwh",
|
||||||
|
transport: database.binding.transport,
|
||||||
|
baseUrl: database.binding.baseUrl,
|
||||||
|
credentialFile: materialized.files.get(credentialId),
|
||||||
|
resource: { database: database.databaseName, schema: database.schema },
|
||||||
|
timeoutMs: this.diagnosticTimeoutMs,
|
||||||
|
signal: controller.signal,
|
||||||
|
diagnostic: {
|
||||||
|
method: "GET" as const,
|
||||||
|
path: database.binding.restPath ?? "/health",
|
||||||
|
auth: database.binding.restAuth ?? "bearer",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
connectionStatus: "reachable",
|
||||||
|
testedVersion: database.version,
|
||||||
|
lastTestedAt: testedAt,
|
||||||
|
};
|
||||||
|
} catch {
|
||||||
|
return {
|
||||||
|
connectionStatus: "failed",
|
||||||
|
testedVersion: database.version,
|
||||||
|
lastTestedAt: testedAt,
|
||||||
|
errorCode: "connector_unavailable",
|
||||||
|
errorMessage: "The database connector could not be reached or authenticated.",
|
||||||
|
};
|
||||||
|
} finally {
|
||||||
|
clearTimeout(timer);
|
||||||
|
controller.abort();
|
||||||
|
materialized.release();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,311 @@
|
|||||||
|
import { randomUUID } from "node:crypto";
|
||||||
|
import type { CatalogOperationCoordinator } from "./operation-coordinator.js";
|
||||||
|
import type { CatalogSchemaIntrospector, CatalogSchemaScanProgress } from "./schema-introspector.js";
|
||||||
|
import {
|
||||||
|
CatalogConflictError,
|
||||||
|
CatalogConnectorError,
|
||||||
|
CatalogSchemaCapabilityUnavailableError,
|
||||||
|
type CatalogRepository,
|
||||||
|
type CatalogSchemaDiff,
|
||||||
|
type CatalogSyncCounts,
|
||||||
|
type CatalogSyncRun,
|
||||||
|
type CatalogSyncScope,
|
||||||
|
type ObservedSchemaSnapshot,
|
||||||
|
type WorkspaceDatabase,
|
||||||
|
} from "./types.js";
|
||||||
|
|
||||||
|
class SyncCancelledError extends Error {}
|
||||||
|
|
||||||
|
const TERMINAL_STATES = new Set<CatalogSyncRun["state"]>([
|
||||||
|
"succeeded", "failed", "cancelled", "interrupted",
|
||||||
|
]);
|
||||||
|
|
||||||
|
function destructive(diff: CatalogSchemaDiff): boolean {
|
||||||
|
return diff.deletedTables.length > 0
|
||||||
|
|| diff.deletedColumns.length > 0
|
||||||
|
|| diff.deletedRelationships.length > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
function fingerprint(snapshot: ObservedSchemaSnapshot): string {
|
||||||
|
return JSON.stringify(snapshot);
|
||||||
|
}
|
||||||
|
|
||||||
|
function safeFailure(error: unknown): { code: string; message: string } {
|
||||||
|
if (error instanceof CatalogSchemaCapabilityUnavailableError) {
|
||||||
|
const label = error.capability === "schema_snapshot" ? "schema snapshot" : error.capability;
|
||||||
|
return {
|
||||||
|
code: "schema_capability_unavailable",
|
||||||
|
message: `This database binding does not provide the ${label} capability.`,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
if (error instanceof CatalogConnectorError) {
|
||||||
|
return { code: "schema_introspection_failed", message: "The database schema could not be read safely." };
|
||||||
|
}
|
||||||
|
return { code: "schema_sync_failed", message: "Schema synchronization failed." };
|
||||||
|
}
|
||||||
|
|
||||||
|
export class CatalogSyncWorker {
|
||||||
|
private readonly workerId = randomUUID();
|
||||||
|
private readonly controllers = new Map<string, AbortController>();
|
||||||
|
private readonly reservations = new Map<string, () => void>();
|
||||||
|
private stopping = false;
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private readonly repository: CatalogRepository,
|
||||||
|
private readonly introspector: CatalogSchemaIntrospector,
|
||||||
|
private readonly operations: CatalogOperationCoordinator,
|
||||||
|
private readonly timeoutMs: number,
|
||||||
|
) {}
|
||||||
|
|
||||||
|
async initialize(): Promise<void> {
|
||||||
|
if (!(await this.repository.available())) return;
|
||||||
|
await this.repository.interruptActiveSyncRuns();
|
||||||
|
await this.repository.pruneSyncEvents(new Date(Date.now() - 30 * 24 * 60 * 60 * 1_000).toISOString());
|
||||||
|
}
|
||||||
|
|
||||||
|
async start(database: WorkspaceDatabase, scope: CatalogSyncScope, tableIds: readonly string[]): Promise<CatalogSyncRun> {
|
||||||
|
this.assertReady(database);
|
||||||
|
const uniqueTableIds = [...new Set(tableIds)];
|
||||||
|
if (scope === "columns") {
|
||||||
|
const tables = await Promise.all(uniqueTableIds.map((tableId) => this.repository.getTable(database.id, tableId)));
|
||||||
|
if (tables.some((table) => !table)) throw new CatalogConflictError("One or more selected tables no longer exist");
|
||||||
|
}
|
||||||
|
if (scope !== "columns" && uniqueTableIds.length > 0) {
|
||||||
|
throw new CatalogConflictError("Table selection is only valid for a column synchronization");
|
||||||
|
}
|
||||||
|
const release = this.operations.reserve(database.id);
|
||||||
|
try {
|
||||||
|
const run = await this.repository.createSyncRun(database.id, scope, uniqueTableIds, database.version);
|
||||||
|
this.reservations.set(run.id, release);
|
||||||
|
await this.repository.appendSyncEvent(run.id, "info", "queued", "Synchronization queued.", { scope });
|
||||||
|
this.launch(run.id);
|
||||||
|
return run;
|
||||||
|
} catch (error) {
|
||||||
|
release();
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async confirm(runId: string, confirmationToken: string): Promise<CatalogSyncRun | undefined> {
|
||||||
|
const run = await this.repository.getSyncRun(runId);
|
||||||
|
if (!run) return undefined;
|
||||||
|
if (run.state !== "awaiting_confirmation" || !run.observedSnapshot || run.confirmationToken !== confirmationToken) {
|
||||||
|
throw new CatalogConflictError("Synchronization confirmation is no longer valid");
|
||||||
|
}
|
||||||
|
await this.repository.appendSyncEvent(run.id, "info", "confirmation_received", "Destructive changes were confirmed.");
|
||||||
|
const queued = await this.repository.updateSyncRun(run.id, {
|
||||||
|
state: "queued",
|
||||||
|
phase: "queued",
|
||||||
|
confirmationToken: null,
|
||||||
|
leaseOwner: null,
|
||||||
|
leaseExpiresAt: null,
|
||||||
|
});
|
||||||
|
this.launch(run.id, fingerprint(run.observedSnapshot));
|
||||||
|
return queued;
|
||||||
|
}
|
||||||
|
|
||||||
|
async cancel(runId: string): Promise<CatalogSyncRun | undefined> {
|
||||||
|
const run = await this.repository.getSyncRun(runId);
|
||||||
|
if (!run) return undefined;
|
||||||
|
if (run.state === "applying" || TERMINAL_STATES.has(run.state)) return run;
|
||||||
|
await this.repository.requestSyncRunCancellation(runId);
|
||||||
|
this.controllers.get(runId)?.abort();
|
||||||
|
if (run.state === "queued" || run.state === "awaiting_confirmation") {
|
||||||
|
const cancelled = await this.repository.updateSyncRun(runId, {
|
||||||
|
state: "cancelled",
|
||||||
|
phase: "completed",
|
||||||
|
finishedAt: new Date().toISOString(),
|
||||||
|
observedSnapshot: null,
|
||||||
|
plannedDiff: null,
|
||||||
|
confirmationToken: null,
|
||||||
|
leaseOwner: null,
|
||||||
|
leaseExpiresAt: null,
|
||||||
|
});
|
||||||
|
await this.repository.appendSyncEvent(runId, "warning", "cancelled", "Synchronization cancelled.");
|
||||||
|
this.release(runId);
|
||||||
|
return cancelled;
|
||||||
|
}
|
||||||
|
return await this.repository.getSyncRun(runId);
|
||||||
|
}
|
||||||
|
|
||||||
|
async retry(runId: string): Promise<CatalogSyncRun | undefined> {
|
||||||
|
const previous = await this.repository.getSyncRun(runId);
|
||||||
|
if (!previous) return undefined;
|
||||||
|
if (!TERMINAL_STATES.has(previous.state)) {
|
||||||
|
throw new CatalogConflictError("Only a finished synchronization can be retried");
|
||||||
|
}
|
||||||
|
const database = await this.repository.get(previous.databaseId);
|
||||||
|
if (!database) return undefined;
|
||||||
|
return await this.start(database, previous.scope, previous.tableIds);
|
||||||
|
}
|
||||||
|
|
||||||
|
async stop(): Promise<void> {
|
||||||
|
this.stopping = true;
|
||||||
|
for (const controller of this.controllers.values()) controller.abort();
|
||||||
|
if (await this.repository.available()) await this.repository.interruptActiveSyncRuns();
|
||||||
|
for (const runId of [...this.reservations.keys()]) this.release(runId);
|
||||||
|
}
|
||||||
|
|
||||||
|
private launch(runId: string, confirmedFingerprint?: string): void {
|
||||||
|
queueMicrotask(() => {
|
||||||
|
void this.execute(runId, confirmedFingerprint).catch(() => undefined);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private async execute(runId: string, confirmedFingerprint?: string): Promise<void> {
|
||||||
|
if (this.stopping) return;
|
||||||
|
const leaseExpiresAt = new Date(Date.now() + 20_000).toISOString();
|
||||||
|
const claimed = await this.repository.claimSyncRun(runId, this.workerId, leaseExpiresAt);
|
||||||
|
if (!claimed) return;
|
||||||
|
const controller = new AbortController();
|
||||||
|
this.controllers.set(runId, controller);
|
||||||
|
let timedOut = false;
|
||||||
|
const timeout = setTimeout(() => {
|
||||||
|
timedOut = true;
|
||||||
|
controller.abort();
|
||||||
|
}, this.timeoutMs);
|
||||||
|
const heartbeat = setInterval(() => {
|
||||||
|
void this.repository.updateSyncRun(runId, {
|
||||||
|
heartbeatAt: new Date().toISOString(),
|
||||||
|
leaseExpiresAt: new Date(Date.now() + 20_000).toISOString(),
|
||||||
|
});
|
||||||
|
}, 5_000);
|
||||||
|
|
||||||
|
try {
|
||||||
|
await this.repository.appendSyncEvent(runId, "info", "started", "Synchronization started.");
|
||||||
|
const database = await this.repository.get(claimed.databaseId);
|
||||||
|
if (!database || database.version !== claimed.requestedDatabaseVersion) {
|
||||||
|
throw new CatalogConflictError("Database binding changed before synchronization started");
|
||||||
|
}
|
||||||
|
this.assertReady(database);
|
||||||
|
const progress: CatalogSchemaScanProgress = async (phase, counts) => {
|
||||||
|
await this.checkCancelled(runId);
|
||||||
|
await this.repository.updateSyncRun(runId, {
|
||||||
|
phase,
|
||||||
|
heartbeatAt: new Date().toISOString(),
|
||||||
|
...(counts ? { counts } : {}),
|
||||||
|
});
|
||||||
|
await this.repository.appendSyncEvent(runId, "info", phase, this.phaseMessage(phase), counts ?? {});
|
||||||
|
};
|
||||||
|
const snapshot = await this.introspector.scan(database, controller.signal, progress);
|
||||||
|
await this.checkCancelled(runId);
|
||||||
|
this.assertCapability(claimed.scope, snapshot);
|
||||||
|
const counts: CatalogSyncCounts = {
|
||||||
|
tables: snapshot.tables.length,
|
||||||
|
columns: snapshot.columns.length,
|
||||||
|
relationships: snapshot.relationships.length,
|
||||||
|
};
|
||||||
|
await this.repository.updateSyncRun(runId, { phase: "planning", counts, observedSnapshot: snapshot });
|
||||||
|
await this.repository.appendSyncEvent(runId, "info", "planning", "Schema changes are being planned.", { ...counts });
|
||||||
|
const diff = await this.repository.planSchemaSync(claimed.databaseId, claimed.scope, claimed.tableIds, snapshot);
|
||||||
|
await this.checkCancelled(runId);
|
||||||
|
if (destructive(diff) && fingerprint(snapshot) !== confirmedFingerprint) {
|
||||||
|
const token = randomUUID();
|
||||||
|
const waiting = await this.repository.updateSyncRun(runId, {
|
||||||
|
state: "awaiting_confirmation",
|
||||||
|
phase: "awaiting_confirmation",
|
||||||
|
observedSnapshot: snapshot,
|
||||||
|
plannedDiff: diff,
|
||||||
|
confirmationToken: token,
|
||||||
|
counts,
|
||||||
|
leaseOwner: null,
|
||||||
|
leaseExpiresAt: null,
|
||||||
|
});
|
||||||
|
await this.repository.appendSyncEvent(runId, "warning", "confirmation_required", "Confirmation is required before removing catalog objects.", {
|
||||||
|
deletedTables: diff.deletedTables.length,
|
||||||
|
deletedColumns: diff.deletedColumns.length,
|
||||||
|
deletedRelationships: diff.deletedRelationships.length,
|
||||||
|
});
|
||||||
|
if (!waiting) throw new Error("Synchronization run disappeared");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
await this.repository.updateSyncRun(runId, { state: "applying", phase: "applying", plannedDiff: diff });
|
||||||
|
await this.repository.appendSyncEvent(runId, "info", "applying", "Catalog changes are being applied atomically.");
|
||||||
|
this.controllers.delete(runId);
|
||||||
|
const applied = await this.repository.applySchemaSync(
|
||||||
|
claimed.databaseId,
|
||||||
|
claimed.requestedDatabaseVersion,
|
||||||
|
claimed.scope,
|
||||||
|
claimed.tableIds,
|
||||||
|
snapshot,
|
||||||
|
);
|
||||||
|
if (!applied) throw new CatalogConflictError("Database binding changed before schema changes were applied");
|
||||||
|
await this.repository.updateSyncRun(runId, {
|
||||||
|
state: "succeeded",
|
||||||
|
phase: "completed",
|
||||||
|
counts: applied,
|
||||||
|
finishedAt: new Date().toISOString(),
|
||||||
|
observedSnapshot: null,
|
||||||
|
plannedDiff: null,
|
||||||
|
confirmationToken: null,
|
||||||
|
heartbeatAt: new Date().toISOString(),
|
||||||
|
leaseOwner: null,
|
||||||
|
leaseExpiresAt: null,
|
||||||
|
});
|
||||||
|
await this.repository.appendSyncEvent(runId, "info", "succeeded", "Synchronization completed.", { ...applied });
|
||||||
|
this.release(runId);
|
||||||
|
} catch (error) {
|
||||||
|
const current = await this.repository.getSyncRun(runId);
|
||||||
|
const cancelled = !timedOut && (error instanceof SyncCancelledError || controller.signal.aborted || current?.cancelRequested);
|
||||||
|
const failure = timedOut
|
||||||
|
? { code: "schema_sync_timed_out", message: "Schema synchronization timed out." }
|
||||||
|
: safeFailure(error);
|
||||||
|
await this.repository.updateSyncRun(runId, {
|
||||||
|
state: cancelled ? "cancelled" : "failed",
|
||||||
|
phase: "completed",
|
||||||
|
errorCode: cancelled ? null : failure.code,
|
||||||
|
errorMessage: cancelled ? null : failure.message,
|
||||||
|
finishedAt: new Date().toISOString(),
|
||||||
|
observedSnapshot: null,
|
||||||
|
plannedDiff: null,
|
||||||
|
confirmationToken: null,
|
||||||
|
leaseOwner: null,
|
||||||
|
leaseExpiresAt: null,
|
||||||
|
});
|
||||||
|
await this.repository.appendSyncEvent(
|
||||||
|
runId,
|
||||||
|
cancelled ? "warning" : "error",
|
||||||
|
cancelled ? "cancelled" : "failed",
|
||||||
|
cancelled ? "Synchronization cancelled." : failure.message,
|
||||||
|
);
|
||||||
|
this.release(runId);
|
||||||
|
} finally {
|
||||||
|
clearTimeout(timeout);
|
||||||
|
clearInterval(heartbeat);
|
||||||
|
this.controllers.delete(runId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private assertReady(database: WorkspaceDatabase): void {
|
||||||
|
if (database.connectionStatus !== "reachable" || database.testedVersion !== database.version) {
|
||||||
|
throw new CatalogConflictError("Test the current database binding before synchronizing its schema");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private assertCapability(scope: CatalogSyncScope, snapshot: ObservedSchemaSnapshot): void {
|
||||||
|
const required = scope === "all" ? ["tables", "columns", "relationships"] as const : [scope] as const;
|
||||||
|
for (const name of required) {
|
||||||
|
if (snapshot.capabilities[name] !== "available") {
|
||||||
|
throw new CatalogSchemaCapabilityUnavailableError(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async checkCancelled(runId: string): Promise<void> {
|
||||||
|
const run = await this.repository.getSyncRun(runId);
|
||||||
|
if (run?.cancelRequested) throw new SyncCancelledError("Synchronization cancelled");
|
||||||
|
}
|
||||||
|
|
||||||
|
private release(runId: string): void {
|
||||||
|
this.reservations.get(runId)?.();
|
||||||
|
this.reservations.delete(runId);
|
||||||
|
}
|
||||||
|
|
||||||
|
private phaseMessage(phase: Parameters<CatalogSchemaScanProgress>[0]): string {
|
||||||
|
if (phase === "connecting") return "Connecting to the database.";
|
||||||
|
if (phase === "scanning_tables") return "Reading tables.";
|
||||||
|
if (phase === "scanning_columns") return "Reading columns and primary keys.";
|
||||||
|
return "Reading foreign-key relationships.";
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
import { readFile } from "node:fs/promises";
|
||||||
|
import type { WorkspaceSecretStore } from "../workspaces/secret-store.js";
|
||||||
|
import type { CatalogPostgresAccess } from "./postgres-access.js";
|
||||||
|
import { CATALOG_SECRET_IDS } from "./secrets.js";
|
||||||
|
import {
|
||||||
|
CatalogConnectorError,
|
||||||
|
type ObservedCatalogTable,
|
||||||
|
type WorkspaceDatabase,
|
||||||
|
} from "./types.js";
|
||||||
|
|
||||||
|
export interface CatalogTableIntrospector {
|
||||||
|
scan(database: WorkspaceDatabase, signal: AbortSignal): Promise<ObservedCatalogTable[]>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalize(rows: readonly ObservedCatalogTable[]): ObservedCatalogTable[] {
|
||||||
|
const byName = new Map<string, ObservedCatalogTable>();
|
||||||
|
for (const row of rows) {
|
||||||
|
if (typeof row.name !== "string" || row.name.length === 0 || row.name.length > 128) {
|
||||||
|
throw new CatalogConnectorError("Schema introspection response is invalid");
|
||||||
|
}
|
||||||
|
if (row.sourceComment !== null && typeof row.sourceComment !== "string") {
|
||||||
|
throw new CatalogConnectorError("Schema introspection response is invalid");
|
||||||
|
}
|
||||||
|
byName.set(row.name, row);
|
||||||
|
}
|
||||||
|
return [...byName.values()]
|
||||||
|
.sort((left, right) => left.name.localeCompare(right.name));
|
||||||
|
}
|
||||||
|
|
||||||
|
function requireText(value: string | undefined): string {
|
||||||
|
if (!value) throw new CatalogConnectorError("Database binding is incomplete");
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
export class ConcreteCatalogTableIntrospector implements CatalogTableIntrospector {
|
||||||
|
constructor(
|
||||||
|
private readonly postgres: CatalogPostgresAccess,
|
||||||
|
private readonly secretStore: WorkspaceSecretStore,
|
||||||
|
) {}
|
||||||
|
|
||||||
|
async scan(database: WorkspaceDatabase, signal: AbortSignal): Promise<ObservedCatalogTable[]> {
|
||||||
|
return database.binding.transport === "rest_api"
|
||||||
|
? await this.scanRest(database, signal)
|
||||||
|
: await this.scanPostgres(database, signal);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async scanPostgres(
|
||||||
|
database: WorkspaceDatabase,
|
||||||
|
signal: AbortSignal,
|
||||||
|
): Promise<ObservedCatalogTable[]> {
|
||||||
|
const client = await this.postgres.connect(database, signal);
|
||||||
|
try {
|
||||||
|
const schema = await client.query(
|
||||||
|
"SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname = $1) AS present",
|
||||||
|
[database.schema],
|
||||||
|
);
|
||||||
|
if (schema.rows[0]?.present !== true) throw new CatalogConnectorError("Database schema is unavailable");
|
||||||
|
const result = await client.query(
|
||||||
|
`SELECT c.relname AS name, d.description AS source_comment
|
||||||
|
FROM pg_catalog.pg_class c
|
||||||
|
JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
|
||||||
|
LEFT JOIN pg_catalog.pg_description d ON d.objoid = c.oid AND d.objsubid = 0
|
||||||
|
WHERE c.relkind IN ('r', 'p') AND n.nspname = $1
|
||||||
|
ORDER BY c.relname`,
|
||||||
|
[database.schema],
|
||||||
|
);
|
||||||
|
return normalize(result.rows.map((row) => ({
|
||||||
|
name: String(row.name),
|
||||||
|
sourceComment: typeof row.source_comment === "string" && row.source_comment.length > 0
|
||||||
|
? row.source_comment
|
||||||
|
: null,
|
||||||
|
})));
|
||||||
|
} finally {
|
||||||
|
await client.end();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async scanRest(
|
||||||
|
database: WorkspaceDatabase,
|
||||||
|
signal: AbortSignal,
|
||||||
|
): Promise<ObservedCatalogTable[]> {
|
||||||
|
const auth = database.binding.restAuth ?? "bearer";
|
||||||
|
const required = auth === "none" ? [] : [CATALOG_SECRET_IDS.apiKey];
|
||||||
|
const materialized = this.secretStore.materialize(database.workspaceId, required);
|
||||||
|
try {
|
||||||
|
const headers: Record<string, string> = { "content-type": "application/json" };
|
||||||
|
if (auth !== "none") {
|
||||||
|
const credentialFile = materialized.files.get(CATALOG_SECRET_IDS.apiKey);
|
||||||
|
if (!credentialFile) throw new CatalogConnectorError("REST API key is not configured");
|
||||||
|
const credential = (await readFile(credentialFile, "utf8")).trim();
|
||||||
|
if (auth === "bearer") headers.authorization = `Bearer ${credential}`;
|
||||||
|
else headers["x-api-key"] = credential;
|
||||||
|
}
|
||||||
|
const baseUrl = requireText(database.binding.baseUrl).replace(/\/+$/, "");
|
||||||
|
const response = await fetch(`${baseUrl}/rpc/list_tables`, {
|
||||||
|
method: "POST",
|
||||||
|
headers,
|
||||||
|
body: JSON.stringify({ schema_name: database.schema }),
|
||||||
|
signal,
|
||||||
|
});
|
||||||
|
if (!response.ok) throw new CatalogConnectorError("REST schema introspection failed");
|
||||||
|
const body: unknown = await response.json();
|
||||||
|
if (!Array.isArray(body)) throw new CatalogConnectorError("REST schema response is invalid");
|
||||||
|
const rows: ObservedCatalogTable[] = [];
|
||||||
|
for (const item of body) {
|
||||||
|
if (!item || typeof item !== "object") {
|
||||||
|
throw new CatalogConnectorError("REST schema response is invalid");
|
||||||
|
}
|
||||||
|
const row = item as Record<string, unknown>;
|
||||||
|
if (typeof row.type !== "string") {
|
||||||
|
throw new CatalogConnectorError("REST schema response is invalid");
|
||||||
|
}
|
||||||
|
if (row.type !== "TABLE") continue;
|
||||||
|
if (typeof row.table !== "string" || row.table.length === 0 || row.table.length > 128) {
|
||||||
|
throw new CatalogConnectorError("REST schema response is invalid");
|
||||||
|
}
|
||||||
|
if (row.comment !== undefined && row.comment !== null && typeof row.comment !== "string") {
|
||||||
|
throw new CatalogConnectorError("REST schema response is invalid");
|
||||||
|
}
|
||||||
|
rows.push({
|
||||||
|
name: row.table,
|
||||||
|
sourceComment: typeof row.comment === "string" && row.comment.length > 0 ? row.comment : null,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return normalize(rows);
|
||||||
|
} catch (error) {
|
||||||
|
if (error instanceof CatalogConnectorError) throw error;
|
||||||
|
throw new CatalogConnectorError("REST schema introspection failed");
|
||||||
|
} finally {
|
||||||
|
materialized.release();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
import type {
|
||||||
|
CatalogRepository,
|
||||||
|
CatalogTable,
|
||||||
|
} from "./types.js";
|
||||||
|
|
||||||
|
export class CatalogTableService {
|
||||||
|
constructor(
|
||||||
|
private readonly repository: CatalogRepository,
|
||||||
|
) {}
|
||||||
|
|
||||||
|
async list(databaseId: string): Promise<CatalogTable[]> {
|
||||||
|
return await this.repository.listTables(databaseId);
|
||||||
|
}
|
||||||
|
|
||||||
|
async updateDescription(
|
||||||
|
databaseId: string,
|
||||||
|
tableId: string,
|
||||||
|
expectedVersion: number,
|
||||||
|
description: string | null,
|
||||||
|
): Promise<CatalogTable | undefined> {
|
||||||
|
const normalized = description?.trim() || null;
|
||||||
|
return await this.repository.updateTableDescription(
|
||||||
|
databaseId,
|
||||||
|
tableId,
|
||||||
|
expectedVersion,
|
||||||
|
normalized,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
async updateMetadata(
|
||||||
|
databaseId: string,
|
||||||
|
tableId: string,
|
||||||
|
expectedVersion: number,
|
||||||
|
description: string | null,
|
||||||
|
generatedDescription: string | null,
|
||||||
|
): Promise<CatalogTable | undefined> {
|
||||||
|
return await this.repository.updateTableMetadata(
|
||||||
|
databaseId,
|
||||||
|
tableId,
|
||||||
|
expectedVersion,
|
||||||
|
description?.trim() || null,
|
||||||
|
generatedDescription?.trim() || null,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,339 @@
|
|||||||
|
export const DATABASE_TRANSPORTS = ["postgres_direct", "rest_api", "ssh_tunnel"] as const;
|
||||||
|
export type DatabaseTransport = (typeof DATABASE_TRANSPORTS)[number];
|
||||||
|
|
||||||
|
export type ConnectionStatus = "untested" | "reachable" | "failed";
|
||||||
|
|
||||||
|
export interface DatabaseBinding {
|
||||||
|
transport: DatabaseTransport;
|
||||||
|
host?: string;
|
||||||
|
port?: number;
|
||||||
|
username?: string;
|
||||||
|
baseUrl?: string;
|
||||||
|
restPath?: string;
|
||||||
|
restAuth?: "none" | "bearer" | "x-api-key";
|
||||||
|
tlsServername?: string;
|
||||||
|
sshHost?: string;
|
||||||
|
sshPort?: number;
|
||||||
|
sshUsername?: string;
|
||||||
|
sshTargetHost?: string;
|
||||||
|
sshTargetPort?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface WorkspaceDatabase {
|
||||||
|
id: string;
|
||||||
|
workspaceId: string;
|
||||||
|
engine: "postgres";
|
||||||
|
databaseName: string;
|
||||||
|
schema: string;
|
||||||
|
version: number;
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
binding: DatabaseBinding;
|
||||||
|
connectionStatus: ConnectionStatus;
|
||||||
|
testedVersion?: number;
|
||||||
|
lastTestedAt?: string;
|
||||||
|
lastErrorCode?: string;
|
||||||
|
lastErrorMessage?: string;
|
||||||
|
schemaSyncedVersion?: number;
|
||||||
|
schemaSyncedAt?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DatabaseConfigurationInput {
|
||||||
|
workspaceId: string;
|
||||||
|
engine: "postgres";
|
||||||
|
databaseName: string;
|
||||||
|
schema: string;
|
||||||
|
binding: DatabaseBinding;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DatabaseTestResult {
|
||||||
|
connectionStatus: Exclude<ConnectionStatus, "untested">;
|
||||||
|
testedVersion: number;
|
||||||
|
lastTestedAt: string;
|
||||||
|
errorCode?: string;
|
||||||
|
errorMessage?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogTable {
|
||||||
|
id: string;
|
||||||
|
databaseId: string;
|
||||||
|
name: string;
|
||||||
|
sourceComment: string | null;
|
||||||
|
description: string | null;
|
||||||
|
generatedDescription: string | null;
|
||||||
|
lastSyncedDatabaseVersion: number | null;
|
||||||
|
lastSyncedAt: string | null;
|
||||||
|
version: number;
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ObservedCatalogTable {
|
||||||
|
name: string;
|
||||||
|
sourceComment: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogColumn {
|
||||||
|
id: string;
|
||||||
|
tableId: string;
|
||||||
|
name: string;
|
||||||
|
ordinalPosition: number;
|
||||||
|
dataType: string;
|
||||||
|
isNullable: boolean;
|
||||||
|
defaultExpression: string | null;
|
||||||
|
primaryKeyPosition: number | null;
|
||||||
|
isPrimaryKey: boolean;
|
||||||
|
isForeignKey: boolean;
|
||||||
|
foreignKeyCount: number;
|
||||||
|
sourceComment: string | null;
|
||||||
|
description: string | null;
|
||||||
|
generatedDescription: string | null;
|
||||||
|
lastSyncedDatabaseVersion: number | null;
|
||||||
|
lastSyncedAt: string | null;
|
||||||
|
version: number;
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ObservedCatalogColumn {
|
||||||
|
tableName: string;
|
||||||
|
name: string;
|
||||||
|
ordinalPosition: number;
|
||||||
|
dataType: string;
|
||||||
|
isNullable: boolean;
|
||||||
|
defaultExpression: string | null;
|
||||||
|
primaryKeyPosition: number | null;
|
||||||
|
sourceComment: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogRelationshipColumn {
|
||||||
|
position: number;
|
||||||
|
sourceColumnId: string;
|
||||||
|
sourceColumnName: string;
|
||||||
|
targetColumnId: string;
|
||||||
|
targetColumnName: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogRelationship {
|
||||||
|
id: string;
|
||||||
|
databaseId: string;
|
||||||
|
constraintName: string;
|
||||||
|
sourceTableId: string;
|
||||||
|
sourceTableName: string;
|
||||||
|
targetTableId: string;
|
||||||
|
targetTableName: string;
|
||||||
|
updateRule: string;
|
||||||
|
deleteRule: string;
|
||||||
|
deferrable: boolean;
|
||||||
|
initiallyDeferred: boolean;
|
||||||
|
columns: CatalogRelationshipColumn[];
|
||||||
|
lastSyncedDatabaseVersion: number | null;
|
||||||
|
lastSyncedAt: string | null;
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ObservedRelationshipColumn {
|
||||||
|
position: number;
|
||||||
|
sourceColumnName: string;
|
||||||
|
targetColumnName: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ObservedCatalogRelationship {
|
||||||
|
constraintName: string;
|
||||||
|
sourceTableName: string;
|
||||||
|
targetTableName: string;
|
||||||
|
updateRule: string;
|
||||||
|
deleteRule: string;
|
||||||
|
deferrable: boolean;
|
||||||
|
initiallyDeferred: boolean;
|
||||||
|
columns: ObservedRelationshipColumn[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export type IntrospectionCapabilityState = "available" | "unavailable";
|
||||||
|
export interface ObservedSchemaSnapshot {
|
||||||
|
schemaVersion: 1;
|
||||||
|
capabilities: {
|
||||||
|
tables: IntrospectionCapabilityState;
|
||||||
|
columns: IntrospectionCapabilityState;
|
||||||
|
relationships: IntrospectionCapabilityState;
|
||||||
|
};
|
||||||
|
tables: ObservedCatalogTable[];
|
||||||
|
columns: ObservedCatalogColumn[];
|
||||||
|
relationships: ObservedCatalogRelationship[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export type CatalogSyncScope = "tables" | "columns" | "relationships" | "all";
|
||||||
|
export type CatalogSyncState =
|
||||||
|
| "queued" | "running" | "awaiting_confirmation" | "applying"
|
||||||
|
| "succeeded" | "failed" | "cancelled" | "interrupted";
|
||||||
|
export type CatalogSyncPhase =
|
||||||
|
| "queued" | "connecting" | "scanning_tables" | "scanning_columns"
|
||||||
|
| "scanning_relationships" | "planning" | "awaiting_confirmation"
|
||||||
|
| "applying" | "completed";
|
||||||
|
|
||||||
|
export interface CatalogSchemaDiff {
|
||||||
|
deletedTables: string[];
|
||||||
|
deletedColumns: Array<{ tableName: string; columnName: string }>;
|
||||||
|
deletedRelationships: Array<{ sourceTableName: string; constraintName: string }>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogSyncCounts {
|
||||||
|
tables?: number;
|
||||||
|
columns?: number;
|
||||||
|
relationships?: number;
|
||||||
|
created?: number;
|
||||||
|
updated?: number;
|
||||||
|
deleted?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogSyncRun {
|
||||||
|
id: string;
|
||||||
|
databaseId: string;
|
||||||
|
scope: CatalogSyncScope;
|
||||||
|
tableIds: string[];
|
||||||
|
state: CatalogSyncState;
|
||||||
|
phase: CatalogSyncPhase;
|
||||||
|
requestedDatabaseVersion: number;
|
||||||
|
observedSnapshot: ObservedSchemaSnapshot | null;
|
||||||
|
plannedDiff: CatalogSchemaDiff | null;
|
||||||
|
confirmationToken: string | null;
|
||||||
|
counts: CatalogSyncCounts;
|
||||||
|
errorCode: string | null;
|
||||||
|
errorMessage: string | null;
|
||||||
|
cancelRequested: boolean;
|
||||||
|
createdAt: string;
|
||||||
|
startedAt: string | null;
|
||||||
|
updatedAt: string;
|
||||||
|
finishedAt: string | null;
|
||||||
|
heartbeatAt: string | null;
|
||||||
|
leaseOwner: string | null;
|
||||||
|
leaseExpiresAt: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogSyncEvent {
|
||||||
|
id: number;
|
||||||
|
runId: string;
|
||||||
|
sequence: number;
|
||||||
|
level: "info" | "warning" | "error";
|
||||||
|
eventType: string;
|
||||||
|
message: string;
|
||||||
|
data: Record<string, unknown>;
|
||||||
|
createdAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogSyncRunUpdate {
|
||||||
|
state?: CatalogSyncState;
|
||||||
|
phase?: CatalogSyncPhase;
|
||||||
|
observedSnapshot?: ObservedSchemaSnapshot | null;
|
||||||
|
plannedDiff?: CatalogSchemaDiff | null;
|
||||||
|
confirmationToken?: string | null;
|
||||||
|
counts?: CatalogSyncCounts;
|
||||||
|
errorCode?: string | null;
|
||||||
|
errorMessage?: string | null;
|
||||||
|
cancelRequested?: boolean;
|
||||||
|
startedAt?: string | null;
|
||||||
|
finishedAt?: string | null;
|
||||||
|
heartbeatAt?: string | null;
|
||||||
|
leaseOwner?: string | null;
|
||||||
|
leaseExpiresAt?: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type TableSyncRepositoryResult =
|
||||||
|
| { kind: "confirmation_required"; deletedNames: string[] }
|
||||||
|
| {
|
||||||
|
kind: "applied";
|
||||||
|
createdCount: number;
|
||||||
|
updatedCount: number;
|
||||||
|
deletedCount: number;
|
||||||
|
tables: CatalogTable[];
|
||||||
|
};
|
||||||
|
|
||||||
|
export interface CatalogRepository {
|
||||||
|
list(): Promise<WorkspaceDatabase[]>;
|
||||||
|
get(id: string): Promise<WorkspaceDatabase | undefined>;
|
||||||
|
getByWorkspace(workspaceId: string): Promise<WorkspaceDatabase | undefined>;
|
||||||
|
create(input: DatabaseConfigurationInput): Promise<WorkspaceDatabase>;
|
||||||
|
update(id: string, expectedVersion: number, input: DatabaseConfigurationInput): Promise<WorkspaceDatabase | undefined>;
|
||||||
|
recordTest(id: string, expectedVersion: number, result: DatabaseTestResult): Promise<WorkspaceDatabase | undefined>;
|
||||||
|
touch(id: string, expectedVersion: number): Promise<WorkspaceDatabase | undefined>;
|
||||||
|
delete(id: string, expectedVersion: number): Promise<boolean>;
|
||||||
|
listTables(databaseId: string): Promise<CatalogTable[]>;
|
||||||
|
getTable(databaseId: string, tableId: string): Promise<CatalogTable | undefined>;
|
||||||
|
updateTableDescription(
|
||||||
|
databaseId: string,
|
||||||
|
tableId: string,
|
||||||
|
expectedVersion: number,
|
||||||
|
description: string | null,
|
||||||
|
): Promise<CatalogTable | undefined>;
|
||||||
|
updateTableMetadata(
|
||||||
|
databaseId: string,
|
||||||
|
tableId: string,
|
||||||
|
expectedVersion: number,
|
||||||
|
description: string | null,
|
||||||
|
generatedDescription: string | null,
|
||||||
|
): Promise<CatalogTable | undefined>;
|
||||||
|
listColumns(databaseId: string, tableId: string): Promise<CatalogColumn[]>;
|
||||||
|
getColumn(databaseId: string, tableId: string, columnId: string): Promise<CatalogColumn | undefined>;
|
||||||
|
updateColumnMetadata(
|
||||||
|
databaseId: string,
|
||||||
|
tableId: string,
|
||||||
|
columnId: string,
|
||||||
|
expectedVersion: number,
|
||||||
|
description: string | null,
|
||||||
|
generatedDescription: string | null,
|
||||||
|
): Promise<CatalogColumn | undefined>;
|
||||||
|
listRelationships(databaseId: string): Promise<CatalogRelationship[]>;
|
||||||
|
planSchemaSync(
|
||||||
|
databaseId: string,
|
||||||
|
scope: CatalogSyncScope,
|
||||||
|
tableIds: readonly string[],
|
||||||
|
snapshot: ObservedSchemaSnapshot,
|
||||||
|
): Promise<CatalogSchemaDiff>;
|
||||||
|
applySchemaSync(
|
||||||
|
databaseId: string,
|
||||||
|
expectedDatabaseVersion: number,
|
||||||
|
scope: CatalogSyncScope,
|
||||||
|
tableIds: readonly string[],
|
||||||
|
snapshot: ObservedSchemaSnapshot,
|
||||||
|
): Promise<CatalogSyncCounts | undefined>;
|
||||||
|
createSyncRun(
|
||||||
|
databaseId: string,
|
||||||
|
scope: CatalogSyncScope,
|
||||||
|
tableIds: readonly string[],
|
||||||
|
requestedDatabaseVersion: number,
|
||||||
|
): Promise<CatalogSyncRun>;
|
||||||
|
getSyncRun(runId: string): Promise<CatalogSyncRun | undefined>;
|
||||||
|
claimSyncRun(runId: string, workerId: string, leaseExpiresAt: string): Promise<CatalogSyncRun | undefined>;
|
||||||
|
listSyncRuns(databaseId: string, limit?: number): Promise<CatalogSyncRun[]>;
|
||||||
|
updateSyncRun(runId: string, update: CatalogSyncRunUpdate): Promise<CatalogSyncRun | undefined>;
|
||||||
|
requestSyncRunCancellation(runId: string): Promise<CatalogSyncRun | undefined>;
|
||||||
|
appendSyncEvent(
|
||||||
|
runId: string,
|
||||||
|
level: CatalogSyncEvent["level"],
|
||||||
|
eventType: string,
|
||||||
|
message: string,
|
||||||
|
data?: Record<string, unknown>,
|
||||||
|
): Promise<CatalogSyncEvent>;
|
||||||
|
listSyncEvents(runId: string, afterSequence?: number): Promise<CatalogSyncEvent[]>;
|
||||||
|
pruneSyncEvents(before: string): Promise<void>;
|
||||||
|
interruptActiveSyncRuns(): Promise<void>;
|
||||||
|
reconcileTables(
|
||||||
|
databaseId: string,
|
||||||
|
expectedDatabaseVersion: number,
|
||||||
|
observed: readonly ObservedCatalogTable[],
|
||||||
|
confirmedDeletedNames: readonly string[],
|
||||||
|
): Promise<TableSyncRepositoryResult | undefined>;
|
||||||
|
available(): Promise<boolean>;
|
||||||
|
close?(): Promise<void>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export class CatalogConflictError extends Error {}
|
||||||
|
export class CatalogUnavailableError extends Error {}
|
||||||
|
export class CatalogOperationInProgressError extends Error {}
|
||||||
|
export class CatalogConnectorError extends Error {}
|
||||||
|
export class CatalogSchemaCapabilityUnavailableError extends CatalogConnectorError {
|
||||||
|
constructor(readonly capability: "schema_snapshot" | "tables" | "columns" | "relationships") {
|
||||||
|
super(`Schema introspection capability '${capability}' is unavailable`);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,6 +7,7 @@ import {
|
|||||||
} from "./auth/config.js";
|
} from "./auth/config.js";
|
||||||
import { createProjectedAuthenticationConfigProvider } from "./auth/runtime-projection.js";
|
import { createProjectedAuthenticationConfigProvider } from "./auth/runtime-projection.js";
|
||||||
import type { WorkspaceRegistryConfig } from "./workspaces/types.js";
|
import type { WorkspaceRegistryConfig } from "./workspaces/types.js";
|
||||||
|
import type { CatalogConnectionConfig } from "./catalog/repository.js";
|
||||||
|
|
||||||
export interface AppConfig {
|
export interface AppConfig {
|
||||||
host: string; port: number; harnessDir: string; thtBin: string; piBin: string;
|
host: string; port: number; harnessDir: string; thtBin: string; piBin: string;
|
||||||
@@ -20,6 +21,8 @@ export interface AppConfig {
|
|||||||
host?: string; port?: number; database?: string; runtimeUser?: string;
|
host?: string; port?: number; database?: string; runtimeUser?: string;
|
||||||
runtimePasswordFile?: string; sslmode?: "verify-ca" | "verify-full"; sslrootcert?: string;
|
runtimePasswordFile?: string; sslmode?: "verify-ca" | "verify-full"; sslrootcert?: string;
|
||||||
};
|
};
|
||||||
|
/** Installation-local metadata catalog. Omitted installations expose an unavailable admin surface. */
|
||||||
|
catalogDatabase?: CatalogConnectionConfig;
|
||||||
defaults: { provider?: string; model?: string; thinking?: string };
|
defaults: { provider?: string; model?: string; thinking?: string };
|
||||||
maxPiProcesses: number;
|
maxPiProcesses: number;
|
||||||
settingsFile: string;
|
settingsFile: string;
|
||||||
@@ -40,6 +43,7 @@ export interface AppConfig {
|
|||||||
/** Explicit compatibility mode for old loopback clients that send `workspace` in POST /sessions. */
|
/** Explicit compatibility mode for old loopback clients that send `workspace` in POST /sessions. */
|
||||||
legacyWorkspaceMode: boolean;
|
legacyWorkspaceMode: boolean;
|
||||||
workspaceDiagnosticTimeoutMs: number;
|
workspaceDiagnosticTimeoutMs: number;
|
||||||
|
catalogSyncTimeoutMs: number;
|
||||||
workspaceRegistry: WorkspaceRegistryConfig;
|
workspaceRegistry: WorkspaceRegistryConfig;
|
||||||
workspaceSecretStoreRoot: string;
|
workspaceSecretStoreRoot: string;
|
||||||
workspaceSecretRuntimeRoot: string;
|
workspaceSecretRuntimeRoot: string;
|
||||||
@@ -127,6 +131,14 @@ function diagnosticTimeout(value: string | undefined): number {
|
|||||||
return timeout;
|
return timeout;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function catalogSyncTimeout(value: string | undefined): number {
|
||||||
|
const timeout = Number(value ?? 600_000);
|
||||||
|
if (!Number.isSafeInteger(timeout) || timeout < 1_000 || timeout > 3_600_000) {
|
||||||
|
throw new Error("catalog synchronization timeout configuration is invalid");
|
||||||
|
}
|
||||||
|
return timeout;
|
||||||
|
}
|
||||||
|
|
||||||
function piManagementTimeout(value: string | undefined): number {
|
function piManagementTimeout(value: string | undefined): number {
|
||||||
const timeout = Number(value ?? 8_000);
|
const timeout = Number(value ?? 8_000);
|
||||||
if (!Number.isSafeInteger(timeout) || timeout < 1 || timeout > 30_000) {
|
if (!Number.isSafeInteger(timeout) || timeout < 1 || timeout > 30_000) {
|
||||||
@@ -176,6 +188,33 @@ function positiveDimension(value: string | undefined, fallback: number): number
|
|||||||
return parsed;
|
return parsed;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function catalogDatabase(env: Record<string, string | undefined>): CatalogConnectionConfig | undefined {
|
||||||
|
const value = env.THT_CATALOG_DATABASE_URL;
|
||||||
|
if (value !== undefined) {
|
||||||
|
try {
|
||||||
|
const parsed = new URL(value);
|
||||||
|
if ((parsed.protocol !== "postgres:" && parsed.protocol !== "postgresql:")
|
||||||
|
|| !parsed.hostname || !parsed.pathname.slice(1) || parsed.hash || parsed.search) throw new Error();
|
||||||
|
return { connectionString: value };
|
||||||
|
} catch {
|
||||||
|
throw new Error("catalog database configuration is invalid");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const host = env.THT_CATALOG_DB_HOST;
|
||||||
|
if (host === undefined) return undefined;
|
||||||
|
const port = Number(env.THT_CATALOG_DB_PORT ?? 5432);
|
||||||
|
const database = env.THT_CATALOG_DB_NAME;
|
||||||
|
const user = env.THT_CATALOG_RUNTIME_USER;
|
||||||
|
const passwordFile = env.THT_CATALOG_RUNTIME_PASSWORD_FILE;
|
||||||
|
try {
|
||||||
|
if (!host.trim() || !database?.trim() || !user?.trim() || !passwordFile
|
||||||
|
|| !path.isAbsolute(passwordFile) || !Number.isInteger(port) || port < 1 || port > 65_535) throw new Error();
|
||||||
|
return { host, port, database, user, passwordFile };
|
||||||
|
} catch {
|
||||||
|
throw new Error("catalog database configuration is invalid");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
export function loadConfig(
|
export function loadConfig(
|
||||||
env: Record<string, string | undefined>,
|
env: Record<string, string | undefined>,
|
||||||
options: { surface?: "application" | "workspace-maintenance" } = {},
|
options: { surface?: "application" | "workspace-maintenance" } = {},
|
||||||
@@ -356,6 +395,7 @@ export function loadConfig(
|
|||||||
authentication,
|
authentication,
|
||||||
publicExposure,
|
publicExposure,
|
||||||
sessionStorage,
|
sessionStorage,
|
||||||
|
catalogDatabase: catalogDatabase(env),
|
||||||
defaults: { provider: env.PI_PROVIDER, model: env.PI_MODEL, thinking: env.PI_THINKING },
|
defaults: { provider: env.PI_PROVIDER, model: env.PI_MODEL, thinking: env.PI_THINKING },
|
||||||
maxPiProcesses: Number(env.MAX_PI_PROCESSES ?? 4),
|
maxPiProcesses: Number(env.MAX_PI_PROCESSES ?? 4),
|
||||||
settingsFile,
|
settingsFile,
|
||||||
@@ -370,6 +410,7 @@ export function loadConfig(
|
|||||||
dwhPrecheck: env.THT_DWH_PRECHECK === "true" || env.THT_DWH_PRECHECK === "1",
|
dwhPrecheck: env.THT_DWH_PRECHECK === "true" || env.THT_DWH_PRECHECK === "1",
|
||||||
legacyWorkspaceMode: legacyWorkspaceMode === "local",
|
legacyWorkspaceMode: legacyWorkspaceMode === "local",
|
||||||
workspaceDiagnosticTimeoutMs: diagnosticTimeout(env.THT_WORKSPACE_DIAGNOSTIC_TIMEOUT_MS),
|
workspaceDiagnosticTimeoutMs: diagnosticTimeout(env.THT_WORKSPACE_DIAGNOSTIC_TIMEOUT_MS),
|
||||||
|
catalogSyncTimeoutMs: catalogSyncTimeout(env.THT_CATALOG_SYNC_TIMEOUT_MS),
|
||||||
workspaceRegistry,
|
workspaceRegistry,
|
||||||
workspaceSecretStoreRoot,
|
workspaceSecretStoreRoot,
|
||||||
workspaceSecretRuntimeRoot,
|
workspaceSecretRuntimeRoot,
|
||||||
|
|||||||
@@ -0,0 +1,218 @@
|
|||||||
|
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
|
||||||
|
import { z } from "zod";
|
||||||
|
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||||
|
import { CatalogService, type CatalogSecretName } from "../catalog/service.js";
|
||||||
|
import { WorkspaceRegistryError } from "../workspaces/git-repository.js";
|
||||||
|
import {
|
||||||
|
CatalogConflictError,
|
||||||
|
CatalogOperationInProgressError,
|
||||||
|
CatalogUnavailableError,
|
||||||
|
DATABASE_TRANSPORTS,
|
||||||
|
type CatalogRepository,
|
||||||
|
type DatabaseConfigurationInput,
|
||||||
|
} from "../catalog/types.js";
|
||||||
|
import type { CatalogOperationCoordinator } from "../catalog/operation-coordinator.js";
|
||||||
|
|
||||||
|
const idSchema = z.uuid();
|
||||||
|
const workspaceIdSchema = z.string().regex(/^[a-z][a-z0-9-]{2,62}$/);
|
||||||
|
const identifier = z.string().trim().min(1).max(128).regex(/^[A-Za-z_][A-Za-z0-9_$-]*$/);
|
||||||
|
const nonEmpty = z.string().trim().min(1).max(512);
|
||||||
|
const port = z.number().int().min(1).max(65_535);
|
||||||
|
const optionalText = nonEmpty.optional();
|
||||||
|
const sshHost = z.string().trim().min(1).max(255).regex(/^[A-Za-z0-9_.:\[\]-]+$/).optional();
|
||||||
|
const sshUsername = z.string().trim().min(1).max(128).regex(/^[A-Za-z0-9._-]+$/).optional();
|
||||||
|
const bindingSchema = z.object({
|
||||||
|
transport: z.enum(DATABASE_TRANSPORTS),
|
||||||
|
host: optionalText,
|
||||||
|
port: port.optional(),
|
||||||
|
username: optionalText,
|
||||||
|
baseUrl: z.url().max(2048).optional(),
|
||||||
|
restPath: z.string().regex(/^\/(?!\/)[^?#\\\u0000-\u001f]*$/).max(512).optional(),
|
||||||
|
restAuth: z.enum(["none", "bearer", "x-api-key"]).optional(),
|
||||||
|
tlsServername: optionalText,
|
||||||
|
sshHost,
|
||||||
|
sshPort: port.optional(),
|
||||||
|
sshUsername,
|
||||||
|
sshTargetHost: sshHost,
|
||||||
|
sshTargetPort: port.optional(),
|
||||||
|
}).strict().superRefine((binding, context) => {
|
||||||
|
const required = binding.transport === "postgres_direct"
|
||||||
|
? ["host", "port", "username"] as const
|
||||||
|
: binding.transport === "rest_api"
|
||||||
|
? ["baseUrl", "restPath", "restAuth"] as const
|
||||||
|
: ["username", "sshHost", "sshPort", "sshUsername", "sshTargetHost", "sshTargetPort"] as const;
|
||||||
|
for (const field of required) {
|
||||||
|
if (binding[field] === undefined) context.addIssue({ code: "custom", path: [field], message: "Required" });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
const configSchema = z.object({
|
||||||
|
workspaceId: workspaceIdSchema,
|
||||||
|
engine: z.literal("postgres"),
|
||||||
|
databaseName: identifier,
|
||||||
|
schema: identifier,
|
||||||
|
binding: bindingSchema,
|
||||||
|
}).strict();
|
||||||
|
const updateSchema = configSchema.extend({ version: z.number().int().positive() });
|
||||||
|
const secretNames = [
|
||||||
|
"password",
|
||||||
|
"apiKey",
|
||||||
|
"sshPrivateKey",
|
||||||
|
"sshPrivateKeyPassphrase",
|
||||||
|
"sshKnownHosts",
|
||||||
|
"tlsCa",
|
||||||
|
] as const;
|
||||||
|
const secretsSchema = z.object({
|
||||||
|
version: z.number().int().positive(),
|
||||||
|
values: z.partialRecord(z.enum(secretNames), z.string().min(1).max(65_536)).refine((values) => Object.keys(values).length > 0),
|
||||||
|
}).strict();
|
||||||
|
const versionQuery = z.object({ version: z.coerce.number().int().positive() });
|
||||||
|
|
||||||
|
function safeError(reply: FastifyReply, error: unknown) {
|
||||||
|
if (error instanceof CatalogUnavailableError) {
|
||||||
|
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
|
||||||
|
}
|
||||||
|
if (error instanceof CatalogConflictError) {
|
||||||
|
return reply.code(409).send({ code: "database_conflict", message: "This workspace already has a database configuration." });
|
||||||
|
}
|
||||||
|
if (error instanceof CatalogOperationInProgressError) {
|
||||||
|
return reply.code(409).send({ code: "database_operation_in_progress", message: "A database operation is already in progress." });
|
||||||
|
}
|
||||||
|
if (error instanceof z.ZodError) {
|
||||||
|
return reply.code(400).send({ code: "database_invalid", message: "Database configuration is invalid." });
|
||||||
|
}
|
||||||
|
if (error instanceof WorkspaceRegistryError) {
|
||||||
|
return reply.code(400).send({ code: "database_invalid", message: "Database configuration is invalid." });
|
||||||
|
}
|
||||||
|
return reply.code(500).send({ code: "database_operation_failed", message: "Database operation failed." });
|
||||||
|
}
|
||||||
|
|
||||||
|
function manage(request: FastifyRequest, reply: FastifyReply) {
|
||||||
|
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function catalogDatabaseRoutes(
|
||||||
|
app: FastifyInstance,
|
||||||
|
deps: { repository: CatalogRepository; service: CatalogService; operations?: CatalogOperationCoordinator },
|
||||||
|
): void {
|
||||||
|
const mutate = async <T>(databaseId: string, operation: () => Promise<T>): Promise<T> => (
|
||||||
|
deps.operations ? await deps.operations.run(databaseId, operation) : await operation()
|
||||||
|
);
|
||||||
|
const activeSyncRun = async (databaseId: string) => {
|
||||||
|
const run = (await deps.repository.listSyncRuns(databaseId, 5)).find((candidate) =>
|
||||||
|
["queued", "running", "awaiting_confirmation", "applying"].includes(candidate.state));
|
||||||
|
if (!run) return undefined;
|
||||||
|
const {
|
||||||
|
observedSnapshot: _snapshot,
|
||||||
|
leaseOwner: _leaseOwner,
|
||||||
|
leaseExpiresAt: _leaseExpiresAt,
|
||||||
|
...summary
|
||||||
|
} = run;
|
||||||
|
return summary;
|
||||||
|
};
|
||||||
|
app.get("/catalog/status", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
return { available: await deps.repository.available() };
|
||||||
|
});
|
||||||
|
|
||||||
|
app.get("/catalog/databases", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const rows = await deps.service.list();
|
||||||
|
return await Promise.all(rows.map(async (row) => (
|
||||||
|
row.id ? { ...row, activeSyncRun: await activeSyncRun(row.id) } : row
|
||||||
|
)));
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.get("/catalog/databases/:id", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const id = idSchema.parse((request.params as { id?: unknown }).id);
|
||||||
|
const database = await deps.repository.get(id);
|
||||||
|
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||||
|
return {
|
||||||
|
...database,
|
||||||
|
configured: true,
|
||||||
|
secrets: deps.service.configuredSecrets(database.workspaceId),
|
||||||
|
activeSyncRun: await activeSyncRun(database.id),
|
||||||
|
};
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.post("/catalog/databases", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const input = configSchema.parse(request.body) as DatabaseConfigurationInput;
|
||||||
|
const created = await deps.repository.create(await deps.service.normalizeInput(input));
|
||||||
|
return reply.code(201).send({ ...created, configured: true, secrets: deps.service.configuredSecrets(created.workspaceId) });
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.patch("/catalog/databases/:id", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const id = idSchema.parse((request.params as { id?: unknown }).id);
|
||||||
|
const { version, ...input } = updateSchema.parse(request.body);
|
||||||
|
const current = await deps.repository.get(id);
|
||||||
|
if (!current) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||||
|
if (current.workspaceId !== input.workspaceId) {
|
||||||
|
return reply.code(400).send({ code: "database_invalid", message: "Database configuration is invalid." });
|
||||||
|
}
|
||||||
|
const updated = await mutate(id, async () => await deps.repository.update(
|
||||||
|
id, version, await deps.service.normalizeInput(input as DatabaseConfigurationInput),
|
||||||
|
));
|
||||||
|
if (!updated) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||||
|
return { ...updated, configured: true, secrets: deps.service.configuredSecrets(updated.workspaceId) };
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.put("/catalog/databases/:id/secrets", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
if (!isPrincipalContext(requirePermission(request, reply, "workspace.secrets.manage"))) return reply;
|
||||||
|
try {
|
||||||
|
const id = idSchema.parse((request.params as { id?: unknown }).id);
|
||||||
|
const { version, values } = secretsSchema.parse(request.body);
|
||||||
|
const database = await deps.repository.get(id);
|
||||||
|
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||||
|
if (database.version !== version) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||||
|
const updated = await mutate(id, async () => {
|
||||||
|
const touched = await deps.repository.touch(id, version);
|
||||||
|
if (touched) deps.service.replaceSecrets(database.workspaceId, values as Partial<Record<CatalogSecretName, string>>);
|
||||||
|
return touched;
|
||||||
|
});
|
||||||
|
if (!updated) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||||
|
return { ...updated, configured: true, secrets: deps.service.configuredSecrets(updated.workspaceId) };
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.post("/catalog/databases/:id/test", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const id = idSchema.parse((request.params as { id?: unknown }).id);
|
||||||
|
const { version } = z.object({ version: z.number().int().positive() }).strict().parse(request.body);
|
||||||
|
const database = await deps.repository.get(id);
|
||||||
|
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||||
|
if (database.version !== version) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||||
|
const tested = await deps.repository.recordTest(id, version, await deps.service.test(database));
|
||||||
|
if (!tested) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||||
|
return { ...tested, configured: true, secrets: deps.service.configuredSecrets(tested.workspaceId) };
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.delete("/catalog/databases/:id", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const id = idSchema.parse((request.params as { id?: unknown }).id);
|
||||||
|
const { version } = versionQuery.parse(request.query);
|
||||||
|
const database = await deps.repository.get(id);
|
||||||
|
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||||
|
const deleted = await mutate(id, async () => {
|
||||||
|
const removed = await deps.repository.delete(id, version);
|
||||||
|
if (removed) deps.service.forgetSecrets(database.workspaceId);
|
||||||
|
return removed;
|
||||||
|
});
|
||||||
|
if (!deleted) return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||||
|
return reply.code(204).send();
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -0,0 +1,230 @@
|
|||||||
|
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
|
||||||
|
import { setTimeout as delay } from "node:timers/promises";
|
||||||
|
import { z } from "zod";
|
||||||
|
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||||
|
import type { CatalogSyncWorker } from "../catalog/sync-worker.js";
|
||||||
|
import {
|
||||||
|
CatalogConflictError,
|
||||||
|
CatalogConnectorError,
|
||||||
|
CatalogOperationInProgressError,
|
||||||
|
CatalogUnavailableError,
|
||||||
|
type CatalogRepository,
|
||||||
|
type CatalogSyncRun,
|
||||||
|
} from "../catalog/types.js";
|
||||||
|
|
||||||
|
const idSchema = z.uuid();
|
||||||
|
const metadataSchema = z.object({
|
||||||
|
version: z.number().int().positive(),
|
||||||
|
description: z.string().max(20_000).nullable(),
|
||||||
|
generatedDescription: z.string().max(20_000).nullable(),
|
||||||
|
}).strict();
|
||||||
|
const createRunSchema = z.object({
|
||||||
|
version: z.number().int().positive(),
|
||||||
|
scope: z.enum(["tables", "columns", "relationships", "all"]),
|
||||||
|
tableIds: z.array(idSchema).max(10_000).default([]),
|
||||||
|
}).strict();
|
||||||
|
const confirmationSchema = z.object({ confirmationToken: z.string().uuid() }).strict();
|
||||||
|
const eventQuerySchema = z.object({ after: z.coerce.number().int().nonnegative().default(0) });
|
||||||
|
|
||||||
|
function manage(request: FastifyRequest, reply: FastifyReply) {
|
||||||
|
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
|
||||||
|
}
|
||||||
|
|
||||||
|
function safeError(reply: FastifyReply, error: unknown) {
|
||||||
|
if (error instanceof CatalogUnavailableError) {
|
||||||
|
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
|
||||||
|
}
|
||||||
|
if (error instanceof CatalogConflictError || error instanceof CatalogOperationInProgressError) {
|
||||||
|
return reply.code(409).send({ code: "schema_sync_conflict", message: error.message });
|
||||||
|
}
|
||||||
|
if (error instanceof CatalogConnectorError) {
|
||||||
|
return reply.code(502).send({ code: "schema_introspection_failed", message: "The database schema could not be read safely." });
|
||||||
|
}
|
||||||
|
if (error instanceof z.ZodError) {
|
||||||
|
return reply.code(400).send({ code: "schema_request_invalid", message: "Schema request is invalid." });
|
||||||
|
}
|
||||||
|
return reply.code(500).send({ code: "schema_operation_failed", message: "Schema operation failed." });
|
||||||
|
}
|
||||||
|
|
||||||
|
function normalized(value: string | null): string | null {
|
||||||
|
return value?.trim() || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function publicRun(run: CatalogSyncRun) {
|
||||||
|
const {
|
||||||
|
observedSnapshot: _snapshot,
|
||||||
|
leaseOwner: _leaseOwner,
|
||||||
|
leaseExpiresAt: _leaseExpiresAt,
|
||||||
|
...result
|
||||||
|
} = run;
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function catalogSchemaRoutes(
|
||||||
|
app: FastifyInstance,
|
||||||
|
deps: { repository: CatalogRepository; worker: CatalogSyncWorker },
|
||||||
|
): void {
|
||||||
|
app.get("/catalog/databases/:databaseId/tables/:tableId/columns", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const params = request.params as { databaseId?: unknown; tableId?: unknown };
|
||||||
|
const databaseId = idSchema.parse(params.databaseId);
|
||||||
|
const tableId = idSchema.parse(params.tableId);
|
||||||
|
if (!(await deps.repository.getTable(databaseId, tableId))) {
|
||||||
|
return reply.code(404).send({ code: "table_not_found", message: "Catalog table was not found." });
|
||||||
|
}
|
||||||
|
return await deps.repository.listColumns(databaseId, tableId);
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.patch("/catalog/databases/:databaseId/tables/:tableId/columns/:columnId", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const params = request.params as { databaseId?: unknown; tableId?: unknown; columnId?: unknown };
|
||||||
|
const databaseId = idSchema.parse(params.databaseId);
|
||||||
|
const tableId = idSchema.parse(params.tableId);
|
||||||
|
const columnId = idSchema.parse(params.columnId);
|
||||||
|
const input = metadataSchema.parse(request.body);
|
||||||
|
const current = await deps.repository.getColumn(databaseId, tableId, columnId);
|
||||||
|
if (!current) return reply.code(404).send({ code: "column_not_found", message: "Catalog column was not found." });
|
||||||
|
if (current.version !== input.version) {
|
||||||
|
return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
|
||||||
|
}
|
||||||
|
const updated = await deps.repository.updateColumnMetadata(
|
||||||
|
databaseId,
|
||||||
|
tableId,
|
||||||
|
columnId,
|
||||||
|
input.version,
|
||||||
|
normalized(input.description),
|
||||||
|
normalized(input.generatedDescription),
|
||||||
|
);
|
||||||
|
if (!updated) return reply.code(409).send({ code: "column_stale", message: "Column metadata changed. Reload and try again." });
|
||||||
|
return updated;
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.get("/catalog/databases/:databaseId/relationships", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
|
||||||
|
if (!(await deps.repository.get(databaseId))) {
|
||||||
|
return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||||
|
}
|
||||||
|
return await deps.repository.listRelationships(databaseId);
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.post("/catalog/databases/:databaseId/sync-runs", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
|
||||||
|
const input = createRunSchema.parse(request.body);
|
||||||
|
const database = await deps.repository.get(databaseId);
|
||||||
|
if (!database) return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||||
|
if (database.version !== input.version) {
|
||||||
|
return reply.code(409).send({ code: "database_stale", message: "Database configuration changed. Reload and try again." });
|
||||||
|
}
|
||||||
|
return reply.code(202).send(publicRun(await deps.worker.start(database, input.scope, input.tableIds)));
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.get("/catalog/databases/:databaseId/sync-runs", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
|
||||||
|
if (!(await deps.repository.get(databaseId))) {
|
||||||
|
return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||||
|
}
|
||||||
|
return (await deps.repository.listSyncRuns(databaseId)).map(publicRun);
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.get("/catalog/sync-runs/:runId", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||||
|
const run = await deps.repository.getSyncRun(runId);
|
||||||
|
return run ? publicRun(run) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.post("/catalog/sync-runs/:runId/confirm", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||||
|
const { confirmationToken } = confirmationSchema.parse(request.body);
|
||||||
|
const run = await deps.worker.confirm(runId, confirmationToken);
|
||||||
|
return run ? publicRun(run) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.post("/catalog/sync-runs/:runId/cancel", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||||
|
const run = await deps.worker.cancel(runId);
|
||||||
|
return run ? publicRun(run) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.post("/catalog/sync-runs/:runId/retry", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||||
|
const run = await deps.worker.retry(runId);
|
||||||
|
return run ? reply.code(202).send(publicRun(run)) : reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.get("/catalog/sync-runs/:runId/events", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||||
|
let after = eventQuerySchema.parse(request.query).after;
|
||||||
|
const headerCursor = Number(request.headers["last-event-id"]);
|
||||||
|
if (Number.isInteger(headerCursor) && headerCursor >= 0) after = Math.max(after, headerCursor);
|
||||||
|
if (!(await deps.repository.getSyncRun(runId))) {
|
||||||
|
return reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||||
|
}
|
||||||
|
reply.hijack();
|
||||||
|
reply.raw.writeHead(200, {
|
||||||
|
"content-type": "text/event-stream; charset=utf-8",
|
||||||
|
"cache-control": "no-cache, no-transform",
|
||||||
|
connection: "keep-alive",
|
||||||
|
"x-accel-buffering": "no",
|
||||||
|
});
|
||||||
|
const controller = new AbortController();
|
||||||
|
request.raw.once("close", () => controller.abort());
|
||||||
|
let lastRunUpdate = "";
|
||||||
|
while (!controller.signal.aborted) {
|
||||||
|
const events = await deps.repository.listSyncEvents(runId, after);
|
||||||
|
for (const event of events) {
|
||||||
|
after = event.sequence;
|
||||||
|
reply.raw.write(`id: ${event.sequence}\nevent: log\ndata: ${JSON.stringify(event)}\n\n`);
|
||||||
|
}
|
||||||
|
const run = await deps.repository.getSyncRun(runId);
|
||||||
|
if (!run) break;
|
||||||
|
if (run.updatedAt !== lastRunUpdate) {
|
||||||
|
lastRunUpdate = run.updatedAt;
|
||||||
|
reply.raw.write(`event: run\ndata: ${JSON.stringify(publicRun(run))}\n\n`);
|
||||||
|
}
|
||||||
|
if (["succeeded", "failed", "cancelled", "interrupted"].includes(run.state)) break;
|
||||||
|
await delay(500, undefined, { signal: controller.signal }).catch(() => undefined);
|
||||||
|
}
|
||||||
|
reply.raw.end();
|
||||||
|
return reply;
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.get("/catalog/sync-runs/:runId/events-list", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const runId = idSchema.parse((request.params as { runId?: unknown }).runId);
|
||||||
|
const after = eventQuerySchema.parse(request.query).after;
|
||||||
|
if (!(await deps.repository.getSyncRun(runId))) {
|
||||||
|
return reply.code(404).send({ code: "sync_run_not_found", message: "Synchronization run was not found." });
|
||||||
|
}
|
||||||
|
return await deps.repository.listSyncEvents(runId, after);
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -0,0 +1,79 @@
|
|||||||
|
import type { FastifyInstance, FastifyReply, FastifyRequest } from "fastify";
|
||||||
|
import { z } from "zod";
|
||||||
|
import { isPrincipalContext, requirePermission } from "../auth/authorization.js";
|
||||||
|
import type { CatalogTableService } from "../catalog/table-service.js";
|
||||||
|
import {
|
||||||
|
CatalogConnectorError,
|
||||||
|
CatalogOperationInProgressError,
|
||||||
|
CatalogUnavailableError,
|
||||||
|
type CatalogRepository,
|
||||||
|
} from "../catalog/types.js";
|
||||||
|
|
||||||
|
const idSchema = z.uuid();
|
||||||
|
const updateSchema = z.object({
|
||||||
|
version: z.number().int().positive(),
|
||||||
|
description: z.string().max(20_000).nullable(),
|
||||||
|
generatedDescription: z.string().max(20_000).nullable().optional(),
|
||||||
|
}).strict();
|
||||||
|
|
||||||
|
function manage(request: FastifyRequest, reply: FastifyReply) {
|
||||||
|
return isPrincipalContext(requirePermission(request, reply, "database.manage"));
|
||||||
|
}
|
||||||
|
|
||||||
|
function safeError(reply: FastifyReply, error: unknown) {
|
||||||
|
if (error instanceof CatalogUnavailableError) {
|
||||||
|
return reply.code(503).send({ code: "catalog_unavailable", message: "Database catalog is unavailable." });
|
||||||
|
}
|
||||||
|
if (error instanceof CatalogOperationInProgressError) {
|
||||||
|
return reply.code(409).send({ code: "database_operation_in_progress", message: "A database operation is already in progress." });
|
||||||
|
}
|
||||||
|
if (error instanceof CatalogConnectorError) {
|
||||||
|
return reply.code(502).send({ code: "table_introspection_failed", message: "Database tables could not be read." });
|
||||||
|
}
|
||||||
|
if (error instanceof z.ZodError) {
|
||||||
|
return reply.code(400).send({ code: "table_invalid", message: "Table request is invalid." });
|
||||||
|
}
|
||||||
|
return reply.code(500).send({ code: "table_operation_failed", message: "Table operation failed." });
|
||||||
|
}
|
||||||
|
|
||||||
|
export function catalogTableRoutes(
|
||||||
|
app: FastifyInstance,
|
||||||
|
deps: { repository: CatalogRepository; service: CatalogTableService },
|
||||||
|
): void {
|
||||||
|
app.get("/catalog/databases/:databaseId/tables", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const databaseId = idSchema.parse((request.params as { databaseId?: unknown }).databaseId);
|
||||||
|
if (!(await deps.repository.get(databaseId))) {
|
||||||
|
return reply.code(404).send({ code: "database_not_found", message: "Database configuration was not found." });
|
||||||
|
}
|
||||||
|
return await deps.service.list(databaseId);
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
app.patch("/catalog/databases/:databaseId/tables/:tableId", async (request, reply) => {
|
||||||
|
if (!manage(request, reply)) return reply;
|
||||||
|
try {
|
||||||
|
const params = request.params as { databaseId?: unknown; tableId?: unknown };
|
||||||
|
const databaseId = idSchema.parse(params.databaseId);
|
||||||
|
const tableId = idSchema.parse(params.tableId);
|
||||||
|
const input = updateSchema.parse(request.body);
|
||||||
|
const { version, description } = input;
|
||||||
|
const current = await deps.repository.getTable(databaseId, tableId);
|
||||||
|
if (!current) return reply.code(404).send({ code: "table_not_found", message: "Catalog table was not found." });
|
||||||
|
if (current.version !== version) {
|
||||||
|
return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
|
||||||
|
}
|
||||||
|
const updated = await deps.service.updateMetadata(
|
||||||
|
databaseId,
|
||||||
|
tableId,
|
||||||
|
version,
|
||||||
|
description,
|
||||||
|
input.generatedDescription === undefined ? current.generatedDescription : input.generatedDescription,
|
||||||
|
);
|
||||||
|
if (!updated) return reply.code(409).send({ code: "table_stale", message: "Table description changed. Reload and try again." });
|
||||||
|
return updated;
|
||||||
|
} catch (error) { return safeError(reply, error); }
|
||||||
|
});
|
||||||
|
|
||||||
|
}
|
||||||
@@ -187,6 +187,7 @@ test("roles collapse duplicates and admin contains all administrative permission
|
|||||||
"settings.manage",
|
"settings.manage",
|
||||||
"workspace.manage",
|
"workspace.manage",
|
||||||
"workspace.secrets.manage",
|
"workspace.secrets.manage",
|
||||||
|
"database.manage",
|
||||||
"pi.manage",
|
"pi.manage",
|
||||||
"auth.diagnostics.read",
|
"auth.diagnostics.read",
|
||||||
]);
|
]);
|
||||||
|
|||||||
@@ -114,7 +114,7 @@ test.each([
|
|||||||
const created = await fixture.app.thothiiAuthSessionStore?.create({
|
const created = await fixture.app.thothiiAuthSessionStore?.create({
|
||||||
principal: {
|
principal: {
|
||||||
issuer: "local", subject: user.id, displayName: user.username, roles: ["admin"],
|
issuer: "local", subject: user.id, displayName: user.username, roles: ["admin"],
|
||||||
permissions: ["session.use", "session.read_all", "session.manage_all", "settings.manage", "workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read"],
|
permissions: ["session.use", "session.read_all", "session.manage_all", "settings.manage", "workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read"],
|
||||||
isAdmin: true,
|
isAdmin: true,
|
||||||
},
|
},
|
||||||
method: "local",
|
method: "local",
|
||||||
|
|||||||
@@ -130,7 +130,7 @@ test("local login sets a non-persistent opaque session cookie and exposes only a
|
|||||||
roles: ["admin"],
|
roles: ["admin"],
|
||||||
permissions: [
|
permissions: [
|
||||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||||
],
|
],
|
||||||
isAdmin: true,
|
isAdmin: true,
|
||||||
csrfToken: expect.stringMatching(/^[A-Za-z0-9_-]{43}$/),
|
csrfToken: expect.stringMatching(/^[A-Za-z0-9_-]{43}$/),
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ test("local mode resolves a stable local principal", async () => {
|
|||||||
roles: ["admin"],
|
roles: ["admin"],
|
||||||
permissions: [
|
permissions: [
|
||||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||||
],
|
],
|
||||||
isAdmin: true,
|
isAdmin: true,
|
||||||
});
|
});
|
||||||
@@ -74,7 +74,7 @@ test("upstream mode accepts only normalized proxy principal headers", async () =
|
|||||||
issuer: "portal", subject: "42", displayName: "Alice", roles: ["user", "admin"],
|
issuer: "portal", subject: "42", displayName: "Alice", roles: ["user", "admin"],
|
||||||
permissions: [
|
permissions: [
|
||||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||||
],
|
],
|
||||||
isAdmin: true,
|
isAdmin: true,
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -15,14 +15,14 @@ const admin: PrincipalContext = {
|
|||||||
issuer: "oidc", subject: "admin", roles: ["admin"],
|
issuer: "oidc", subject: "admin", roles: ["admin"],
|
||||||
permissions: [
|
permissions: [
|
||||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||||
],
|
],
|
||||||
isAdmin: true,
|
isAdmin: true,
|
||||||
};
|
};
|
||||||
|
|
||||||
const catalog: readonly Permission[] = [
|
const catalog: readonly Permission[] = [
|
||||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||||
];
|
];
|
||||||
|
|
||||||
test("permission matrix gives role-less identities no access, users session use, and admins every catalog permission", () => {
|
test("permission matrix gives role-less identities no access, users session use, and admins every catalog permission", () => {
|
||||||
|
|||||||
@@ -0,0 +1,122 @@
|
|||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { afterEach, expect, test, vi } from "vitest";
|
||||||
|
import { buildApp } from "../src/app.js";
|
||||||
|
import { loadConfig } from "../src/config.js";
|
||||||
|
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
|
||||||
|
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||||
|
import type { WorkspaceRegistry, WorkspaceRevision } from "../src/workspaces/registry.js";
|
||||||
|
import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
|
||||||
|
|
||||||
|
const roots: string[] = [];
|
||||||
|
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
|
||||||
|
|
||||||
|
const workspace: WorkspaceDescriptor = {
|
||||||
|
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
|
||||||
|
dwh: {
|
||||||
|
engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432,
|
||||||
|
supported_transports: ["postgres_direct", "rest_api"],
|
||||||
|
},
|
||||||
|
semantic_index: {
|
||||||
|
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
|
||||||
|
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
|
||||||
|
},
|
||||||
|
llm_policy: { allowed: ["zai/glm-5.2"] },
|
||||||
|
diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } },
|
||||||
|
};
|
||||||
|
const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" };
|
||||||
|
|
||||||
|
function setup() {
|
||||||
|
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-secret-"));
|
||||||
|
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-secret-runtime-"));
|
||||||
|
roots.push(secretRoot, runtimeRoot);
|
||||||
|
const secretStore = new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" });
|
||||||
|
const repository = new MemoryCatalogRepository();
|
||||||
|
const registry = {
|
||||||
|
list: vi.fn(async () => [revision]),
|
||||||
|
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
|
||||||
|
read: vi.fn(async () => ({ workspace, revision })),
|
||||||
|
} as unknown as WorkspaceRegistry;
|
||||||
|
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
|
||||||
|
thtRunner: {} as never,
|
||||||
|
workspaceRegistry: registry,
|
||||||
|
workspaceSecretStore: secretStore,
|
||||||
|
catalogRepository: repository,
|
||||||
|
workspaceDiagnoser: vi.fn(),
|
||||||
|
});
|
||||||
|
return { app, secretStore, repository };
|
||||||
|
}
|
||||||
|
|
||||||
|
const direct = {
|
||||||
|
workspaceId: "psd-clinical",
|
||||||
|
engine: "postgres",
|
||||||
|
databaseName: "warehouse",
|
||||||
|
schema: "datawarehouse",
|
||||||
|
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||||
|
};
|
||||||
|
|
||||||
|
test("lists every YAML workspace and creates its one database configuration", async () => {
|
||||||
|
const { app } = setup();
|
||||||
|
const initial = await app.inject({ method: "GET", url: "/catalog/databases" });
|
||||||
|
expect(initial.statusCode).toBe(200);
|
||||||
|
expect(initial.json()).toMatchObject([{ workspaceId: "psd-clinical", configured: false, databaseName: "warehouse" }]);
|
||||||
|
|
||||||
|
const created = await app.inject({ method: "POST", url: "/catalog/databases", payload: direct });
|
||||||
|
expect(created.statusCode).toBe(201);
|
||||||
|
expect(created.json()).toMatchObject({ configured: true, workspaceId: "psd-clinical", version: 1 });
|
||||||
|
expect((await app.inject({ method: "POST", url: "/catalog/databases", payload: direct })).statusCode).toBe(409);
|
||||||
|
|
||||||
|
const listed = await app.inject({ method: "GET", url: "/catalog/databases" });
|
||||||
|
expect(listed.json()).toMatchObject([{ configured: true, binding: { transport: "postgres_direct", host: "db.internal" } }]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("lists orphaned records and takes the REST diagnostic path from workspace YAML", async () => {
|
||||||
|
const { app, repository } = setup();
|
||||||
|
await repository.create({
|
||||||
|
workspaceId: "removed-workspace",
|
||||||
|
engine: "postgres",
|
||||||
|
databaseName: "legacy",
|
||||||
|
schema: "public",
|
||||||
|
binding: { transport: "postgres_direct", host: "legacy.internal", port: 5432, username: "reader" },
|
||||||
|
});
|
||||||
|
const created = await app.inject({
|
||||||
|
method: "POST",
|
||||||
|
url: "/catalog/databases",
|
||||||
|
payload: {
|
||||||
|
...direct,
|
||||||
|
binding: {
|
||||||
|
transport: "rest_api", baseUrl: "https://psd.example/api", restPath: "/client-controlled", restAuth: "bearer",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(created.statusCode).toBe(201);
|
||||||
|
expect(created.json()).toMatchObject({ binding: { restPath: "/health" } });
|
||||||
|
|
||||||
|
const rows = (await app.inject({ method: "GET", url: "/catalog/databases" })).json();
|
||||||
|
expect(rows).toEqual(expect.arrayContaining([
|
||||||
|
expect.objectContaining({ workspaceId: "removed-workspace", configured: true, workspaceAvailable: false }),
|
||||||
|
expect.objectContaining({ workspaceId: "psd-clinical", configured: true, workspaceAvailable: true }),
|
||||||
|
]));
|
||||||
|
});
|
||||||
|
|
||||||
|
test("uses optimistic versions, keeps secrets write-only, and hard-deletes only local configuration", async () => {
|
||||||
|
const { app, secretStore } = setup();
|
||||||
|
const created = (await app.inject({ method: "POST", url: "/catalog/databases", payload: direct })).json();
|
||||||
|
const stale = await app.inject({ method: "PATCH", url: `/catalog/databases/${created.id}`, payload: { ...direct, version: 99 } });
|
||||||
|
expect(stale.statusCode).toBe(409);
|
||||||
|
|
||||||
|
const secret = await app.inject({
|
||||||
|
method: "PUT", url: `/catalog/databases/${created.id}/secrets`,
|
||||||
|
payload: { version: 1, values: { password: "do-not-return-this" } },
|
||||||
|
});
|
||||||
|
expect(secret.statusCode).toBe(200);
|
||||||
|
expect(secret.body).not.toContain("do-not-return-this");
|
||||||
|
expect(secret.json()).toMatchObject({ version: 2, secrets: { password: true } });
|
||||||
|
expect(secretStore.has("psd-clinical", "catalog.dwh.password")).toBe(true);
|
||||||
|
|
||||||
|
const removed = await app.inject({ method: "DELETE", url: `/catalog/databases/${created.id}?version=2` });
|
||||||
|
expect(removed.statusCode).toBe(204);
|
||||||
|
expect(secretStore.has("psd-clinical", "catalog.dwh.password")).toBe(false);
|
||||||
|
expect((await app.inject({ method: "GET", url: "/catalog/databases" })).json()).toMatchObject([{ configured: false }]);
|
||||||
|
});
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
import { EventEmitter } from "node:events";
|
||||||
|
import { existsSync, mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { PassThrough } from "node:stream";
|
||||||
|
import type { ChildProcessWithoutNullStreams } from "node:child_process";
|
||||||
|
import type { Client, ClientConfig } from "pg";
|
||||||
|
import { afterEach, expect, test, vi } from "vitest";
|
||||||
|
import {
|
||||||
|
buildSshArguments,
|
||||||
|
ConcreteCatalogPostgresAccess,
|
||||||
|
} from "../src/catalog/postgres-access.js";
|
||||||
|
import { CATALOG_SECRET_IDS } from "../src/catalog/secrets.js";
|
||||||
|
import type { WorkspaceDatabase } from "../src/catalog/types.js";
|
||||||
|
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||||
|
|
||||||
|
const roots: string[] = [];
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
function secretStore() {
|
||||||
|
const root = mkdtempSync(join(tmpdir(), "catalog-ssh-secrets-"));
|
||||||
|
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-ssh-runtime-"));
|
||||||
|
roots.push(root, runtimeRoot);
|
||||||
|
return new WorkspaceSecretStore({ root, runtimeRoot, installationId: "test" });
|
||||||
|
}
|
||||||
|
|
||||||
|
function sshDatabase(): WorkspaceDatabase {
|
||||||
|
return {
|
||||||
|
id: "11111111-1111-4111-8111-111111111111",
|
||||||
|
workspaceId: "psd-clinical",
|
||||||
|
engine: "postgres",
|
||||||
|
databaseName: "warehouse",
|
||||||
|
schema: "datawarehouse",
|
||||||
|
version: 4,
|
||||||
|
createdAt: "2026-08-27T08:00:00Z",
|
||||||
|
updatedAt: "2026-08-27T09:00:00Z",
|
||||||
|
connectionStatus: "reachable",
|
||||||
|
binding: {
|
||||||
|
transport: "ssh_tunnel",
|
||||||
|
username: "warehouse_reader",
|
||||||
|
sshHost: "bastion.internal",
|
||||||
|
sshPort: 2222,
|
||||||
|
sshUsername: "tunnel_user",
|
||||||
|
sshTargetHost: "postgres.internal",
|
||||||
|
sshTargetPort: 5432,
|
||||||
|
tlsServername: "postgres.internal",
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function fakeChild(): ChildProcessWithoutNullStreams {
|
||||||
|
const child = new EventEmitter() as EventEmitter & {
|
||||||
|
stdin: PassThrough;
|
||||||
|
stdout: PassThrough;
|
||||||
|
stderr: PassThrough;
|
||||||
|
exitCode: number | null;
|
||||||
|
signalCode: NodeJS.Signals | null;
|
||||||
|
kill: (signal?: NodeJS.Signals | number) => boolean;
|
||||||
|
};
|
||||||
|
child.stdin = new PassThrough();
|
||||||
|
child.stdout = new PassThrough();
|
||||||
|
child.stderr = new PassThrough();
|
||||||
|
child.exitCode = null;
|
||||||
|
child.signalCode = null;
|
||||||
|
child.kill = vi.fn((signal: NodeJS.Signals | number = "SIGTERM") => {
|
||||||
|
child.signalCode = typeof signal === "string" ? signal : "SIGTERM";
|
||||||
|
child.emit("exit", null, child.signalCode);
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
return child as unknown as ChildProcessWithoutNullStreams;
|
||||||
|
}
|
||||||
|
|
||||||
|
test("builds a strict host-verified OpenSSH stdio tunnel", () => {
|
||||||
|
const args = buildSshArguments({
|
||||||
|
sshHost: "bastion.internal",
|
||||||
|
sshPort: 2222,
|
||||||
|
sshUsername: "tunnel_user",
|
||||||
|
targetHost: "postgres.internal",
|
||||||
|
targetPort: 5432,
|
||||||
|
privateKeyFile: "/runtime/id",
|
||||||
|
knownHostsFile: "/runtime/known_hosts",
|
||||||
|
passphraseFile: "/runtime/passphrase",
|
||||||
|
connectTimeoutMs: 5_001,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(args).toEqual(expect.arrayContaining([
|
||||||
|
"-F", "/dev/null",
|
||||||
|
"-o", "BatchMode=no",
|
||||||
|
"-o", "StrictHostKeyChecking=yes",
|
||||||
|
"-o", "UserKnownHostsFile=/runtime/known_hosts",
|
||||||
|
"-o", "GlobalKnownHostsFile=/dev/null",
|
||||||
|
"-o", "IdentitiesOnly=yes",
|
||||||
|
"-o", "IdentityAgent=none",
|
||||||
|
"-o", "PasswordAuthentication=no",
|
||||||
|
"-o", "KbdInteractiveAuthentication=no",
|
||||||
|
"-o", "ConnectTimeout=6",
|
||||||
|
"-W", "postgres.internal:5432",
|
||||||
|
"--", "tunnel_user@bastion.internal",
|
||||||
|
]));
|
||||||
|
});
|
||||||
|
|
||||||
|
test("connects pg through OpenSSH, supplies askpass, and releases all secret leases", async () => {
|
||||||
|
const store = secretStore();
|
||||||
|
store.putMany("psd-clinical", {
|
||||||
|
[CATALOG_SECRET_IDS.password]: "db-password ",
|
||||||
|
[CATALOG_SECRET_IDS.sshPrivateKey]: "PRIVATE KEY\n",
|
||||||
|
[CATALOG_SECRET_IDS.sshPrivateKeyPassphrase]: "key-passphrase",
|
||||||
|
[CATALOG_SECRET_IDS.sshKnownHosts]: "bastion.internal ssh-ed25519 AAAATEST\n",
|
||||||
|
[CATALOG_SECRET_IDS.tlsCa]: "CA CERTIFICATE\n",
|
||||||
|
});
|
||||||
|
const child = fakeChild();
|
||||||
|
let clientConfig: ClientConfig | undefined;
|
||||||
|
let spawnCall: { command: string; args: readonly string[]; env: NodeJS.ProcessEnv } | undefined;
|
||||||
|
const end = vi.fn(async () => undefined);
|
||||||
|
const query = vi.fn(async () => ({ rows: [{ ok: true }] }));
|
||||||
|
const connect = vi.fn(async () => undefined);
|
||||||
|
|
||||||
|
const access = new ConcreteCatalogPostgresAccess(store, {
|
||||||
|
sshBinary: "/usr/bin/ssh",
|
||||||
|
askpassPath: "/app/ssh-askpass.mjs",
|
||||||
|
connectTimeoutMs: 5_000,
|
||||||
|
spawnSsh: (command, args, options) => {
|
||||||
|
spawnCall = { command, args, env: options.env };
|
||||||
|
return child;
|
||||||
|
},
|
||||||
|
createClient: (config) => {
|
||||||
|
clientConfig = config;
|
||||||
|
return { connect, query, end } as unknown as Client;
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
const client = await access.connect(sshDatabase(), new AbortController().signal);
|
||||||
|
expect(connect).toHaveBeenCalledOnce();
|
||||||
|
expect(clientConfig).toMatchObject({
|
||||||
|
host: "postgres.internal",
|
||||||
|
port: 5432,
|
||||||
|
database: "warehouse",
|
||||||
|
user: "warehouse_reader",
|
||||||
|
password: "db-password ",
|
||||||
|
connectionTimeoutMillis: 5_000,
|
||||||
|
ssl: {
|
||||||
|
ca: "CA CERTIFICATE\n",
|
||||||
|
servername: "postgres.internal",
|
||||||
|
rejectUnauthorized: true,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(clientConfig?.stream).toBeTypeOf("function");
|
||||||
|
expect(spawnCall?.command).toBe("/usr/bin/ssh");
|
||||||
|
expect(spawnCall?.args.some((argument) => argument.startsWith("IdentityFile="))).toBe(true);
|
||||||
|
expect(spawnCall?.args.some((argument) => argument.startsWith("UserKnownHostsFile="))).toBe(true);
|
||||||
|
expect(spawnCall?.env).toMatchObject({
|
||||||
|
DISPLAY: "thothii",
|
||||||
|
SSH_ASKPASS: "/app/ssh-askpass.mjs",
|
||||||
|
SSH_ASKPASS_REQUIRE: "force",
|
||||||
|
});
|
||||||
|
const leasedPaths = spawnCall!.args
|
||||||
|
.filter((argument) => argument.startsWith("IdentityFile=") || argument.startsWith("UserKnownHostsFile="))
|
||||||
|
.map((argument) => argument.slice(argument.indexOf("=") + 1));
|
||||||
|
leasedPaths.push(spawnCall!.env.THT_SSH_PASSPHRASE_FILE!);
|
||||||
|
expect(leasedPaths.every(existsSync)).toBe(true);
|
||||||
|
|
||||||
|
await expect(client.query("SELECT 1", [])).resolves.toEqual({ rows: [{ ok: true }] });
|
||||||
|
await client.end();
|
||||||
|
|
||||||
|
expect(end).toHaveBeenCalledOnce();
|
||||||
|
expect(child.kill).toHaveBeenCalledWith("SIGTERM");
|
||||||
|
expect(leasedPaths.some(existsSync)).toBe(false);
|
||||||
|
});
|
||||||
@@ -0,0 +1,125 @@
|
|||||||
|
import { spawnSync } from "node:child_process";
|
||||||
|
import { PostgreSqlContainer } from "@testcontainers/postgresql";
|
||||||
|
import { CamelCasePlugin, Kysely, PostgresDialect, sql } from "kysely";
|
||||||
|
import { Pool } from "pg";
|
||||||
|
import { expect, test } from "vitest";
|
||||||
|
import { KyselyCatalogRepository, type CatalogDatabase } from "../src/catalog/repository.js";
|
||||||
|
import type { ObservedSchemaSnapshot } from "../src/catalog/types.js";
|
||||||
|
import { up as upDatabases } from "../src/catalog/migrations/001_workspace_databases.js";
|
||||||
|
import { up as upTables } from "../src/catalog/migrations/002_catalog_tables.js";
|
||||||
|
import { up as upSchemaSync } from "../src/catalog/migrations/003_catalog_schema_sync.js";
|
||||||
|
import { up as upRuntimeSequencePrivileges } from "../src/catalog/migrations/004_catalog_runtime_sequence_privileges.js";
|
||||||
|
|
||||||
|
const dockerAvailable = spawnSync("docker", ["info"], { stdio: "ignore" }).status === 0;
|
||||||
|
|
||||||
|
test.skipIf(!dockerAvailable)("PostgreSQL migration enforces one database per workspace and optimistic updates", async () => {
|
||||||
|
const container = await new PostgreSqlContainer("postgres:17.6-bookworm").start();
|
||||||
|
const db = new Kysely<CatalogDatabase>({
|
||||||
|
dialect: new PostgresDialect({ pool: new Pool({ connectionString: container.getConnectionUri() }) }),
|
||||||
|
plugins: [new CamelCasePlugin()],
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
await upDatabases(db);
|
||||||
|
await upTables(db);
|
||||||
|
await upSchemaSync(db);
|
||||||
|
await sql`CREATE ROLE thothii_catalog_runtime`.execute(db);
|
||||||
|
await upRuntimeSequencePrivileges(db);
|
||||||
|
const sequencePrivilege = await sql<{ allowed: boolean }>`
|
||||||
|
SELECT has_sequence_privilege(
|
||||||
|
'thothii_catalog_runtime',
|
||||||
|
'catalog_sync_events_id_seq',
|
||||||
|
'USAGE'
|
||||||
|
) AS allowed
|
||||||
|
`.execute(db);
|
||||||
|
expect(sequencePrivilege.rows[0]?.allowed).toBe(true);
|
||||||
|
const repository = new KyselyCatalogRepository(db);
|
||||||
|
const input = {
|
||||||
|
workspaceId: "psd-clinical",
|
||||||
|
engine: "postgres" as const,
|
||||||
|
databaseName: "warehouse",
|
||||||
|
schema: "datawarehouse",
|
||||||
|
binding: { transport: "rest_api" as const, baseUrl: "https://psd.example/api", restPath: "/health", restAuth: "bearer" as const },
|
||||||
|
};
|
||||||
|
const created = await repository.create(input);
|
||||||
|
expect(created).toMatchObject({ version: 1, connectionStatus: "untested", binding: { transport: "rest_api" } });
|
||||||
|
await expect(repository.create(input)).rejects.toThrow("Workspace database already exists");
|
||||||
|
expect(await repository.update(created.id, 99, input)).toBeUndefined();
|
||||||
|
const synchronized = await repository.reconcileTables(created.id, 1, [
|
||||||
|
{ name: "patients", sourceComment: "Clinical patients" },
|
||||||
|
{ name: "visits", sourceComment: null },
|
||||||
|
], []);
|
||||||
|
expect(synchronized).toMatchObject({ kind: "applied", createdCount: 2, deletedCount: 0 });
|
||||||
|
const patients = (await repository.listTables(created.id))[0];
|
||||||
|
expect(await repository.updateTableDescription(
|
||||||
|
created.id,
|
||||||
|
patients.id,
|
||||||
|
patients.version,
|
||||||
|
"Curated patients",
|
||||||
|
)).toMatchObject({ description: "Curated patients", sourceComment: "Clinical patients", version: 2 });
|
||||||
|
const visits = (await repository.listTables(created.id)).find((table) => table.name === "visits")!;
|
||||||
|
const fullColumnsSnapshot: ObservedSchemaSnapshot = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||||
|
tables: [
|
||||||
|
{ name: "patients", sourceComment: "Clinical patients" },
|
||||||
|
{ name: "visits", sourceComment: null },
|
||||||
|
],
|
||||||
|
columns: [
|
||||||
|
{ tableName: "patients", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
|
||||||
|
{ tableName: "patients", name: "name", ordinalPosition: 2, dataType: "text", isNullable: true, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
|
||||||
|
{ tableName: "visits", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
|
||||||
|
{ tableName: "visits", name: "patient_id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
|
||||||
|
],
|
||||||
|
relationships: [],
|
||||||
|
};
|
||||||
|
expect(await repository.applySchemaSync(created.id, 1, "columns", [], fullColumnsSnapshot))
|
||||||
|
.toMatchObject({ created: 4, deleted: 0 });
|
||||||
|
expect((await repository.listColumns(created.id, patients.id)).map((column) => column.name))
|
||||||
|
.toEqual(["id", "name"]);
|
||||||
|
expect((await repository.listColumns(created.id, visits.id)).map((column) => column.name))
|
||||||
|
.toEqual(["id", "patient_id"]);
|
||||||
|
|
||||||
|
const reducedColumnsSnapshot: ObservedSchemaSnapshot = {
|
||||||
|
...fullColumnsSnapshot,
|
||||||
|
columns: fullColumnsSnapshot.columns.filter((column) => column.name === "id"),
|
||||||
|
};
|
||||||
|
expect(await repository.planSchemaSync(created.id, "columns", [], reducedColumnsSnapshot)).toMatchObject({
|
||||||
|
deletedColumns: [
|
||||||
|
{ tableName: "patients", columnName: "name" },
|
||||||
|
{ tableName: "visits", columnName: "patient_id" },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
expect(await repository.planSchemaSync(created.id, "columns", [patients.id], reducedColumnsSnapshot)).toMatchObject({
|
||||||
|
deletedColumns: [{ tableName: "patients", columnName: "name" }],
|
||||||
|
});
|
||||||
|
expect(await repository.applySchemaSync(created.id, 1, "columns", [patients.id], reducedColumnsSnapshot))
|
||||||
|
.toMatchObject({ deleted: 1 });
|
||||||
|
expect((await repository.listColumns(created.id, patients.id)).map((column) => column.name))
|
||||||
|
.toEqual(["id"]);
|
||||||
|
expect((await repository.listColumns(created.id, visits.id)).map((column) => column.name))
|
||||||
|
.toEqual(["id", "patient_id"]);
|
||||||
|
expect(await repository.reconcileTables(created.id, 1, [
|
||||||
|
{ name: "patients", sourceComment: "Updated physical comment" },
|
||||||
|
], [])).toEqual({ kind: "confirmation_required", deletedNames: ["visits"] });
|
||||||
|
expect(await repository.reconcileTables(created.id, 1, [
|
||||||
|
{ name: "patients", sourceComment: "Updated physical comment" },
|
||||||
|
], ["visits"])).toMatchObject({ kind: "applied", updatedCount: 1, deletedCount: 1 });
|
||||||
|
const syncRun = await repository.createSyncRun(created.id, "columns", [patients.id], 1);
|
||||||
|
expect(syncRun).toMatchObject({
|
||||||
|
databaseId: created.id,
|
||||||
|
scope: "columns",
|
||||||
|
tableIds: [patients.id],
|
||||||
|
state: "queued",
|
||||||
|
});
|
||||||
|
expect(await repository.listSyncRuns(created.id)).toEqual([
|
||||||
|
expect.objectContaining({ id: syncRun.id, tableIds: [patients.id] }),
|
||||||
|
]);
|
||||||
|
expect(await repository.update(created.id, 1, { ...input, schema: "public" })).toMatchObject({ version: 2, schema: "public" });
|
||||||
|
expect(await repository.delete(created.id, 2)).toBe(true);
|
||||||
|
expect(await repository.list()).toEqual([]);
|
||||||
|
expect(await repository.listTables(created.id)).toEqual([]);
|
||||||
|
} finally {
|
||||||
|
await db.destroy();
|
||||||
|
await container.stop();
|
||||||
|
}
|
||||||
|
}, 60_000);
|
||||||
@@ -0,0 +1,194 @@
|
|||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { afterEach, expect, test, vi } from "vitest";
|
||||||
|
import type { CatalogDatabaseClient, CatalogPostgresAccess } from "../src/catalog/postgres-access.js";
|
||||||
|
import { ConcreteCatalogSchemaIntrospector } from "../src/catalog/schema-introspector.js";
|
||||||
|
import {
|
||||||
|
CatalogSchemaCapabilityUnavailableError,
|
||||||
|
type WorkspaceDatabase,
|
||||||
|
} from "../src/catalog/types.js";
|
||||||
|
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||||
|
|
||||||
|
const roots: string[] = [];
|
||||||
|
afterEach(() => {
|
||||||
|
vi.unstubAllGlobals();
|
||||||
|
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
function store() {
|
||||||
|
const root = mkdtempSync(join(tmpdir(), "catalog-schema-introspection-secrets-"));
|
||||||
|
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-schema-introspection-runtime-"));
|
||||||
|
roots.push(root, runtimeRoot);
|
||||||
|
return new WorkspaceSecretStore({ root, runtimeRoot, installationId: "test" });
|
||||||
|
}
|
||||||
|
|
||||||
|
function database(binding: WorkspaceDatabase["binding"]): WorkspaceDatabase {
|
||||||
|
return {
|
||||||
|
id: "11111111-1111-4111-8111-111111111111",
|
||||||
|
workspaceId: "psd-clinical",
|
||||||
|
engine: "postgres",
|
||||||
|
databaseName: "warehouse",
|
||||||
|
schema: "datawarehouse",
|
||||||
|
binding,
|
||||||
|
version: 4,
|
||||||
|
connectionStatus: "reachable",
|
||||||
|
testedVersion: 4,
|
||||||
|
createdAt: "2026-08-27T08:00:00Z",
|
||||||
|
updatedAt: "2026-08-27T09:00:00Z",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test("reads columns, ordered composite keys, and physical relationships from one PostgreSQL connection", async () => {
|
||||||
|
const query = vi.fn()
|
||||||
|
.mockResolvedValueOnce({ rows: [{ present: true }] })
|
||||||
|
.mockResolvedValueOnce({ rows: [{ name: "visits", source_comment: "Visits" }] })
|
||||||
|
.mockResolvedValueOnce({ rows: [
|
||||||
|
{ table_name: "visits", name: "tenant_id", ordinal_position: 1, data_type: "uuid", is_nullable: false, default_expression: null, primary_key_position: 1, source_comment: null },
|
||||||
|
{ table_name: "visits", name: "patient_id", ordinal_position: 2, data_type: "bigint", is_nullable: false, default_expression: null, primary_key_position: 2, source_comment: "Patient" },
|
||||||
|
] })
|
||||||
|
.mockResolvedValueOnce({ rows: [
|
||||||
|
{ constraint_name: "visits_patient_fkey", source_table_name: "visits", target_table_name: "patients", update_action: "a", delete_action: "c", deferrable: true, initially_deferred: false, position: 1, source_column_name: "tenant_id", target_column_name: "tenant_id" },
|
||||||
|
{ constraint_name: "visits_patient_fkey", source_table_name: "visits", target_table_name: "patients", update_action: "a", delete_action: "c", deferrable: true, initially_deferred: false, position: 2, source_column_name: "patient_id", target_column_name: "id" },
|
||||||
|
] });
|
||||||
|
const end = vi.fn(async () => undefined);
|
||||||
|
const client: CatalogDatabaseClient = { query, end };
|
||||||
|
const postgres: CatalogPostgresAccess = { connect: vi.fn(async () => client) };
|
||||||
|
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
|
||||||
|
|
||||||
|
const result = await introspector.scan(database({
|
||||||
|
transport: "ssh_tunnel",
|
||||||
|
username: "reader",
|
||||||
|
sshHost: "bastion.internal",
|
||||||
|
sshPort: 22,
|
||||||
|
sshUsername: "tunnel",
|
||||||
|
sshTargetHost: "db.internal",
|
||||||
|
sshTargetPort: 5432,
|
||||||
|
}), new AbortController().signal);
|
||||||
|
|
||||||
|
expect(result.capabilities).toEqual({ tables: "available", columns: "available", relationships: "available" });
|
||||||
|
expect(result.columns).toMatchObject([
|
||||||
|
{ name: "tenant_id", primaryKeyPosition: 1, isNullable: false },
|
||||||
|
{ name: "patient_id", primaryKeyPosition: 2, sourceComment: "Patient" },
|
||||||
|
]);
|
||||||
|
expect(result.relationships).toEqual([expect.objectContaining({
|
||||||
|
constraintName: "visits_patient_fkey",
|
||||||
|
updateRule: "NO ACTION",
|
||||||
|
deleteRule: "CASCADE",
|
||||||
|
deferrable: true,
|
||||||
|
columns: [
|
||||||
|
{ position: 1, sourceColumnName: "tenant_id", targetColumnName: "tenant_id" },
|
||||||
|
{ position: 2, sourceColumnName: "patient_id", targetColumnName: "id" },
|
||||||
|
],
|
||||||
|
})]);
|
||||||
|
expect(query.mock.calls[2][0]).toContain("format_type");
|
||||||
|
expect(query.mock.calls[3][0]).toContain("WITH ORDINALITY");
|
||||||
|
expect(query.mock.calls.slice(1).every((call) => call[1][0] === "datawarehouse")).toBe(true);
|
||||||
|
expect(end).toHaveBeenCalledOnce();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("uses the typed full REST snapshot RPC and preserves explicit capability unavailability", async () => {
|
||||||
|
const response = {
|
||||||
|
schemaVersion: 1,
|
||||||
|
capabilities: { tables: "available", columns: "unavailable", relationships: "unavailable" },
|
||||||
|
tables: [{ name: "patients", sourceComment: null }],
|
||||||
|
columns: [],
|
||||||
|
relationships: [],
|
||||||
|
};
|
||||||
|
const fetchMock = vi.fn(async () => new Response(JSON.stringify(response), { status: 200, headers: { "content-type": "application/json" } }));
|
||||||
|
vi.stubGlobal("fetch", fetchMock);
|
||||||
|
const postgres: CatalogPostgresAccess = { connect: vi.fn(async () => { throw new Error("wire access must not be used"); }) };
|
||||||
|
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
|
||||||
|
|
||||||
|
const result = await introspector.scan(database({
|
||||||
|
transport: "rest_api", baseUrl: "https://connector.internal/api/", restPath: "/health", restAuth: "none",
|
||||||
|
}), new AbortController().signal);
|
||||||
|
|
||||||
|
expect(result).toEqual(response);
|
||||||
|
expect(fetchMock).toHaveBeenCalledWith(
|
||||||
|
"https://connector.internal/api/rpc/schema_snapshot",
|
||||||
|
expect.objectContaining({ method: "POST", body: JSON.stringify({ schema_name: "datawarehouse" }) }),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("falls back to one read-only REST query when the snapshot RPC is absent", async () => {
|
||||||
|
const response = {
|
||||||
|
schemaVersion: 1 as const,
|
||||||
|
capabilities: { tables: "available" as const, columns: "available" as const, relationships: "available" as const },
|
||||||
|
tables: [
|
||||||
|
{ name: "patients", sourceComment: "Clinical patients" },
|
||||||
|
{ name: "visits", sourceComment: null },
|
||||||
|
],
|
||||||
|
columns: [
|
||||||
|
{ tableName: "patients", name: "tenant_id", ordinalPosition: 1, dataType: "uuid", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: "Tenant" },
|
||||||
|
{ tableName: "patients", name: "id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: "nextval('patients_id_seq'::regclass)", primaryKeyPosition: 2, sourceComment: null },
|
||||||
|
{ tableName: "visits", name: "tenant_id", ordinalPosition: 1, dataType: "uuid", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: null },
|
||||||
|
{ tableName: "visits", name: "patient_id", ordinalPosition: 2, dataType: "bigint", isNullable: true, defaultExpression: null, primaryKeyPosition: null, sourceComment: "Owning patient" },
|
||||||
|
],
|
||||||
|
relationships: [{
|
||||||
|
constraintName: "visits_patient_fkey",
|
||||||
|
sourceTableName: "visits",
|
||||||
|
targetTableName: "patients",
|
||||||
|
updateRule: "CASCADE",
|
||||||
|
deleteRule: "RESTRICT",
|
||||||
|
deferrable: true,
|
||||||
|
initiallyDeferred: false,
|
||||||
|
columns: [
|
||||||
|
{ position: 1, sourceColumnName: "tenant_id", targetColumnName: "tenant_id" },
|
||||||
|
{ position: 2, sourceColumnName: "patient_id", targetColumnName: "id" },
|
||||||
|
],
|
||||||
|
}],
|
||||||
|
};
|
||||||
|
const fetchMock = vi.fn()
|
||||||
|
.mockResolvedValueOnce(new Response(null, { status: 404 }))
|
||||||
|
.mockResolvedValueOnce(new Response(JSON.stringify([response]), {
|
||||||
|
status: 200,
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
}));
|
||||||
|
vi.stubGlobal("fetch", fetchMock);
|
||||||
|
const postgres: CatalogPostgresAccess = {
|
||||||
|
connect: vi.fn(async () => { throw new Error("wire access must not be used"); }),
|
||||||
|
};
|
||||||
|
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
|
||||||
|
|
||||||
|
const result = await introspector.scan(database({
|
||||||
|
transport: "rest_api",
|
||||||
|
baseUrl: "https://connector.internal/api/",
|
||||||
|
restPath: "/health",
|
||||||
|
restAuth: "none",
|
||||||
|
}), new AbortController().signal);
|
||||||
|
|
||||||
|
expect(result).toEqual(response);
|
||||||
|
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||||
|
expect(fetchMock.mock.calls[0]).toEqual([
|
||||||
|
"https://connector.internal/api/rpc/schema_snapshot",
|
||||||
|
expect.objectContaining({ method: "POST", body: JSON.stringify({ schema_name: "datawarehouse" }) }),
|
||||||
|
]);
|
||||||
|
expect(fetchMock.mock.calls[1][0]).toBe("https://connector.internal/api/rpc/run_query");
|
||||||
|
const fallbackRequest = fetchMock.mock.calls[1][1] as RequestInit;
|
||||||
|
expect(fallbackRequest).toMatchObject({ method: "POST" });
|
||||||
|
const fallbackBody = JSON.parse(String(fallbackRequest.body)) as { query_text: string };
|
||||||
|
expect(Object.keys(fallbackBody)).toEqual(["query_text"]);
|
||||||
|
expect(fallbackBody.query_text).toMatch(/^\s*WITH\b/);
|
||||||
|
expect(fallbackBody.query_text).toContain("pg_catalog.pg_constraint");
|
||||||
|
expect(fallbackBody.query_text).not.toMatch(/\b(INSERT|UPDATE|DROP|ALTER|CREATE|TRUNCATE)\b/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("classifies a missing REST snapshot RPC as an explicit binding capability", async () => {
|
||||||
|
vi.stubGlobal("fetch", vi.fn(async () => new Response(null, { status: 404 })));
|
||||||
|
const postgres: CatalogPostgresAccess = {
|
||||||
|
connect: vi.fn(async () => { throw new Error("wire access must not be used"); }),
|
||||||
|
};
|
||||||
|
const introspector = new ConcreteCatalogSchemaIntrospector(postgres, store());
|
||||||
|
|
||||||
|
const scan = introspector.scan(database({
|
||||||
|
transport: "rest_api",
|
||||||
|
baseUrl: "https://connector.internal/api/",
|
||||||
|
restPath: "/health",
|
||||||
|
restAuth: "none",
|
||||||
|
}), new AbortController().signal);
|
||||||
|
|
||||||
|
await expect(scan).rejects.toMatchObject<CatalogSchemaCapabilityUnavailableError>({
|
||||||
|
capability: "schema_snapshot",
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,224 @@
|
|||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { afterEach, expect, test, vi } from "vitest";
|
||||||
|
import { buildApp } from "../src/app.js";
|
||||||
|
import { loadConfig } from "../src/config.js";
|
||||||
|
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
|
||||||
|
import type { CatalogSchemaIntrospector } from "../src/catalog/schema-introspector.js";
|
||||||
|
import type { CatalogSyncRun, ObservedSchemaSnapshot } from "../src/catalog/types.js";
|
||||||
|
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||||
|
import type { WorkspaceRegistry, WorkspaceRevision } from "../src/workspaces/registry.js";
|
||||||
|
import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
|
||||||
|
|
||||||
|
const roots: string[] = [];
|
||||||
|
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
|
||||||
|
|
||||||
|
const workspace: WorkspaceDescriptor = {
|
||||||
|
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
|
||||||
|
dwh: { engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432, supported_transports: ["postgres_direct"] },
|
||||||
|
semantic_index: {
|
||||||
|
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
|
||||||
|
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
|
||||||
|
},
|
||||||
|
llm_policy: { allowed: ["zai/glm-5.2"] },
|
||||||
|
};
|
||||||
|
const revision: WorkspaceRevision = { id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml" };
|
||||||
|
|
||||||
|
function snapshot(): ObservedSchemaSnapshot {
|
||||||
|
return {
|
||||||
|
schemaVersion: 1,
|
||||||
|
capabilities: { tables: "available", columns: "available", relationships: "available" },
|
||||||
|
tables: [
|
||||||
|
{ name: "patients", sourceComment: "Clinical patients" },
|
||||||
|
{ name: "visits", sourceComment: "Patient visits" },
|
||||||
|
],
|
||||||
|
columns: [
|
||||||
|
{ tableName: "patients", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: "Patient key" },
|
||||||
|
{ tableName: "visits", name: "id", ordinalPosition: 1, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: 1, sourceComment: null },
|
||||||
|
{ tableName: "visits", name: "patient_id", ordinalPosition: 2, dataType: "bigint", isNullable: false, defaultExpression: null, primaryKeyPosition: null, sourceComment: "Owning patient" },
|
||||||
|
],
|
||||||
|
relationships: [{
|
||||||
|
constraintName: "visits_patient_id_fkey",
|
||||||
|
sourceTableName: "visits",
|
||||||
|
targetTableName: "patients",
|
||||||
|
updateRule: "NO ACTION",
|
||||||
|
deleteRule: "CASCADE",
|
||||||
|
deferrable: false,
|
||||||
|
initiallyDeferred: false,
|
||||||
|
columns: [{ position: 1, sourceColumnName: "patient_id", targetColumnName: "id" }],
|
||||||
|
}],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function waitFor(repository: MemoryCatalogRepository, runId: string, state: CatalogSyncRun["state"]): Promise<CatalogSyncRun> {
|
||||||
|
for (let attempt = 0; attempt < 100; attempt += 1) {
|
||||||
|
const run = await repository.getSyncRun(runId);
|
||||||
|
if (run?.state === state) return run;
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 5));
|
||||||
|
}
|
||||||
|
throw new Error(`Run ${runId} did not reach ${state}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function setup() {
|
||||||
|
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-schema-secret-"));
|
||||||
|
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-schema-runtime-"));
|
||||||
|
roots.push(secretRoot, runtimeRoot);
|
||||||
|
const repository = new MemoryCatalogRepository();
|
||||||
|
const created = await repository.create({
|
||||||
|
workspaceId: "psd-clinical", engine: "postgres", databaseName: "warehouse", schema: "datawarehouse",
|
||||||
|
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||||
|
});
|
||||||
|
await repository.recordTest(created.id, created.version, {
|
||||||
|
connectionStatus: "reachable", testedVersion: created.version, lastTestedAt: new Date().toISOString(),
|
||||||
|
});
|
||||||
|
let observed = snapshot();
|
||||||
|
const scan = vi.fn(async (_database, _signal, progress) => {
|
||||||
|
await progress?.("connecting");
|
||||||
|
await progress?.("scanning_tables", { tables: observed.tables.length });
|
||||||
|
await progress?.("scanning_columns", { tables: observed.tables.length, columns: observed.columns.length });
|
||||||
|
await progress?.("scanning_relationships", { relationships: observed.relationships.length });
|
||||||
|
return structuredClone(observed);
|
||||||
|
});
|
||||||
|
const introspector: CatalogSchemaIntrospector = { scan };
|
||||||
|
const registry = {
|
||||||
|
list: vi.fn(async () => [revision]),
|
||||||
|
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
|
||||||
|
read: vi.fn(async () => ({ workspace, revision })),
|
||||||
|
} as unknown as WorkspaceRegistry;
|
||||||
|
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
|
||||||
|
thtRunner: {} as never,
|
||||||
|
workspaceRegistry: registry,
|
||||||
|
workspaceSecretStore: new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" }),
|
||||||
|
catalogRepository: repository,
|
||||||
|
catalogSchemaIntrospector: introspector,
|
||||||
|
workspaceDiagnoser: vi.fn(),
|
||||||
|
});
|
||||||
|
return {
|
||||||
|
app, repository, database: (await repository.get(created.id))!, scan,
|
||||||
|
setObserved(next: ObservedSchemaSnapshot) { observed = next; },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test("synchronizes a full physical schema and derives primary and foreign key flags", async () => {
|
||||||
|
const { app, repository, database } = await setup();
|
||||||
|
const started = await app.inject({
|
||||||
|
method: "POST", url: `/catalog/databases/${database.id}/sync-runs`,
|
||||||
|
payload: { version: database.version, scope: "all", tableIds: [] },
|
||||||
|
});
|
||||||
|
expect(started.statusCode).toBe(202);
|
||||||
|
const completed = await waitFor(repository, started.json().id, "succeeded");
|
||||||
|
expect(completed.counts).toMatchObject({ tables: 2, columns: 3, relationships: 1 });
|
||||||
|
|
||||||
|
const tables = await repository.listTables(database.id);
|
||||||
|
const visits = tables.find((table) => table.name === "visits")!;
|
||||||
|
const columns = (await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables/${visits.id}/columns` })).json();
|
||||||
|
expect(columns).toMatchObject([
|
||||||
|
{ name: "id", isPrimaryKey: true, primaryKeyPosition: 1, isForeignKey: false },
|
||||||
|
{ name: "patient_id", isPrimaryKey: false, isForeignKey: true, foreignKeyCount: 1 },
|
||||||
|
]);
|
||||||
|
const relationships = (await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/relationships` })).json();
|
||||||
|
expect(relationships).toMatchObject([{ constraintName: "visits_patient_id_fkey", columns: [{ sourceColumnName: "patient_id", targetColumnName: "id" }] }]);
|
||||||
|
expect((await repository.get(database.id))?.schemaSyncedVersion).toBe(database.version);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("synchronizes columns for every catalog table when no table selection is supplied", async () => {
|
||||||
|
const { app, repository, database, setObserved } = await setup();
|
||||||
|
const tablesRun = await app.inject({
|
||||||
|
method: "POST",
|
||||||
|
url: `/catalog/databases/${database.id}/sync-runs`,
|
||||||
|
payload: { version: database.version, scope: "tables", tableIds: [] },
|
||||||
|
});
|
||||||
|
expect(tablesRun.statusCode).toBe(202);
|
||||||
|
await waitFor(repository, tablesRun.json().id, "succeeded");
|
||||||
|
const tables = await repository.listTables(database.id);
|
||||||
|
expect(tables.map((table) => table.name)).toEqual(["patients", "visits"]);
|
||||||
|
|
||||||
|
const columnsRun = await app.inject({
|
||||||
|
method: "POST",
|
||||||
|
url: `/catalog/databases/${database.id}/sync-runs`,
|
||||||
|
payload: { version: database.version, scope: "columns", tableIds: [] },
|
||||||
|
});
|
||||||
|
expect(columnsRun.statusCode).toBe(202);
|
||||||
|
await waitFor(repository, columnsRun.json().id, "succeeded");
|
||||||
|
const patients = tables.find((table) => table.name === "patients")!;
|
||||||
|
const visits = tables.find((table) => table.name === "visits")!;
|
||||||
|
expect((await repository.listColumns(database.id, patients.id)).map((column) => column.name)).toEqual(["id"]);
|
||||||
|
expect((await repository.listColumns(database.id, visits.id)).map((column) => column.name)).toEqual(["id", "patient_id"]);
|
||||||
|
|
||||||
|
const next = snapshot();
|
||||||
|
next.columns = next.columns.filter((column) => column.name !== "id");
|
||||||
|
setObserved(next);
|
||||||
|
const selectedDestructiveRun = await app.inject({
|
||||||
|
method: "POST",
|
||||||
|
url: `/catalog/databases/${database.id}/sync-runs`,
|
||||||
|
payload: { version: database.version, scope: "columns", tableIds: [patients.id] },
|
||||||
|
});
|
||||||
|
expect(selectedDestructiveRun.statusCode).toBe(202);
|
||||||
|
const selectedWaiting = await waitFor(repository, selectedDestructiveRun.json().id, "awaiting_confirmation");
|
||||||
|
expect(selectedWaiting.plannedDiff?.deletedColumns).toEqual([
|
||||||
|
{ tableName: "patients", columnName: "id" },
|
||||||
|
]);
|
||||||
|
expect((await app.inject({
|
||||||
|
method: "POST",
|
||||||
|
url: `/catalog/sync-runs/${selectedWaiting.id}/cancel`,
|
||||||
|
})).statusCode).toBe(200);
|
||||||
|
|
||||||
|
const destructiveRun = await app.inject({
|
||||||
|
method: "POST",
|
||||||
|
url: `/catalog/databases/${database.id}/sync-runs`,
|
||||||
|
payload: { version: database.version, scope: "columns", tableIds: [] },
|
||||||
|
});
|
||||||
|
expect(destructiveRun.statusCode).toBe(202);
|
||||||
|
const waiting = await waitFor(repository, destructiveRun.json().id, "awaiting_confirmation");
|
||||||
|
expect(waiting.plannedDiff?.deletedColumns).toEqual([
|
||||||
|
{ tableName: "patients", columnName: "id" },
|
||||||
|
{ tableName: "visits", columnName: "id" },
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("keeps generated descriptions editable and preserves them across synchronization", async () => {
|
||||||
|
const { app, repository, database } = await setup();
|
||||||
|
const first = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
|
||||||
|
await waitFor(repository, first.json().id, "succeeded");
|
||||||
|
const patients = (await repository.listTables(database.id)).find((table) => table.name === "patients")!;
|
||||||
|
const editedTable = await app.inject({
|
||||||
|
method: "PATCH", url: `/catalog/databases/${database.id}/tables/${patients.id}`,
|
||||||
|
payload: { version: patients.version, description: null, generatedDescription: "Generated table draft" },
|
||||||
|
});
|
||||||
|
expect(editedTable.json()).toMatchObject({ description: null, generatedDescription: "Generated table draft" });
|
||||||
|
const idColumn = (await repository.listColumns(database.id, patients.id))[0];
|
||||||
|
const editedColumn = await app.inject({
|
||||||
|
method: "PATCH", url: `/catalog/databases/${database.id}/tables/${patients.id}/columns/${idColumn.id}`,
|
||||||
|
payload: { version: idColumn.version, description: "Reviewed key", generatedDescription: "Generated key draft" },
|
||||||
|
});
|
||||||
|
expect(editedColumn.json()).toMatchObject({ description: "Reviewed key", generatedDescription: "Generated key draft" });
|
||||||
|
|
||||||
|
const second = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
|
||||||
|
await waitFor(repository, second.json().id, "succeeded");
|
||||||
|
expect(await repository.getTable(database.id, patients.id)).toMatchObject({ generatedDescription: "Generated table draft" });
|
||||||
|
expect(await repository.getColumn(database.id, patients.id, idColumn.id)).toMatchObject({ description: "Reviewed key", generatedDescription: "Generated key draft" });
|
||||||
|
});
|
||||||
|
|
||||||
|
test("waits for confirmation and rescans before applying destructive changes", async () => {
|
||||||
|
const { app, repository, database, scan, setObserved } = await setup();
|
||||||
|
const first = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
|
||||||
|
await waitFor(repository, first.json().id, "succeeded");
|
||||||
|
const next = snapshot();
|
||||||
|
next.tables = next.tables.filter((table) => table.name !== "visits");
|
||||||
|
next.columns = next.columns.filter((column) => column.tableName !== "visits");
|
||||||
|
next.relationships = [];
|
||||||
|
setObserved(next);
|
||||||
|
|
||||||
|
const destructive = await app.inject({ method: "POST", url: `/catalog/databases/${database.id}/sync-runs`, payload: { version: database.version, scope: "all", tableIds: [] } });
|
||||||
|
const waiting = await waitFor(repository, destructive.json().id, "awaiting_confirmation");
|
||||||
|
expect(waiting.plannedDiff).toMatchObject({ deletedTables: ["visits"] });
|
||||||
|
expect(await repository.listTables(database.id)).toHaveLength(2);
|
||||||
|
const confirmed = await app.inject({
|
||||||
|
method: "POST", url: `/catalog/sync-runs/${waiting.id}/confirm`, payload: { confirmationToken: waiting.confirmationToken },
|
||||||
|
});
|
||||||
|
expect(confirmed.statusCode).toBe(200);
|
||||||
|
await waitFor(repository, waiting.id, "succeeded");
|
||||||
|
expect((await repository.listTables(database.id)).map((table) => table.name)).toEqual(["patients"]);
|
||||||
|
expect(scan).toHaveBeenCalledTimes(3);
|
||||||
|
});
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { afterEach, expect, test, vi } from "vitest";
|
||||||
|
import type {
|
||||||
|
CatalogDatabaseClient,
|
||||||
|
CatalogPostgresAccess,
|
||||||
|
} from "../src/catalog/postgres-access.js";
|
||||||
|
import { CATALOG_SECRET_IDS } from "../src/catalog/secrets.js";
|
||||||
|
import { ConcreteCatalogTableIntrospector } from "../src/catalog/table-introspector.js";
|
||||||
|
import type { WorkspaceDatabase } from "../src/catalog/types.js";
|
||||||
|
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||||
|
|
||||||
|
const roots: string[] = [];
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
vi.unstubAllGlobals();
|
||||||
|
for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
function secretStore() {
|
||||||
|
const root = mkdtempSync(join(tmpdir(), "catalog-table-introspection-secrets-"));
|
||||||
|
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-table-introspection-runtime-"));
|
||||||
|
roots.push(root, runtimeRoot);
|
||||||
|
return new WorkspaceSecretStore({ root, runtimeRoot, installationId: "test" });
|
||||||
|
}
|
||||||
|
|
||||||
|
function database(binding: WorkspaceDatabase["binding"]): WorkspaceDatabase {
|
||||||
|
return {
|
||||||
|
id: "11111111-1111-4111-8111-111111111111",
|
||||||
|
workspaceId: "psd-clinical",
|
||||||
|
engine: "postgres",
|
||||||
|
databaseName: "warehouse",
|
||||||
|
schema: "datawarehouse",
|
||||||
|
version: 4,
|
||||||
|
createdAt: "2026-08-27T08:00:00Z",
|
||||||
|
updatedAt: "2026-08-27T09:00:00Z",
|
||||||
|
connectionStatus: "reachable",
|
||||||
|
binding,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test("reads only ordinary and partitioned PostgreSQL tables from the configured schema", async () => {
|
||||||
|
const query = vi.fn()
|
||||||
|
.mockResolvedValueOnce({ rows: [{ present: true }] })
|
||||||
|
.mockResolvedValueOnce({ rows: [
|
||||||
|
{ name: "visits", source_comment: null },
|
||||||
|
{ name: "patients", source_comment: "Clinical patients" },
|
||||||
|
] });
|
||||||
|
const end = vi.fn(async () => undefined);
|
||||||
|
const client: CatalogDatabaseClient = { query, end };
|
||||||
|
const postgres: CatalogPostgresAccess = { connect: vi.fn(async () => client) };
|
||||||
|
const introspector = new ConcreteCatalogTableIntrospector(postgres, secretStore());
|
||||||
|
|
||||||
|
const tables = await introspector.scan(database({
|
||||||
|
transport: "postgres_direct",
|
||||||
|
host: "db.internal",
|
||||||
|
port: 5432,
|
||||||
|
username: "reader",
|
||||||
|
}), new AbortController().signal);
|
||||||
|
|
||||||
|
expect(tables).toEqual([
|
||||||
|
{ name: "patients", sourceComment: "Clinical patients" },
|
||||||
|
{ name: "visits", sourceComment: null },
|
||||||
|
]);
|
||||||
|
expect(query.mock.calls[1][0]).toContain("c.relkind IN ('r', 'p')");
|
||||||
|
expect(query.mock.calls[1][0]).not.toContain("'v'");
|
||||||
|
expect(query.mock.calls[1][1]).toEqual(["datawarehouse"]);
|
||||||
|
expect(end).toHaveBeenCalledOnce();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("uses the typed REST table RPC and ignores non-table objects", async () => {
|
||||||
|
const store = secretStore();
|
||||||
|
store.put("psd-clinical", CATALOG_SECRET_IDS.apiKey, "rest-secret");
|
||||||
|
const fetchMock = vi.fn(async () => new Response(JSON.stringify([
|
||||||
|
{ type: "VIEW", table: "patient_view", comment: "Not a table" },
|
||||||
|
{ type: "TABLE", table: "visits", comment: null },
|
||||||
|
{ type: "TABLE", table: "patients", comment: "Clinical patients" },
|
||||||
|
]), { status: 200, headers: { "content-type": "application/json" } }));
|
||||||
|
vi.stubGlobal("fetch", fetchMock);
|
||||||
|
const postgres: CatalogPostgresAccess = {
|
||||||
|
connect: vi.fn(async () => { throw new Error("PostgreSQL wire access must not be used"); }),
|
||||||
|
};
|
||||||
|
const introspector = new ConcreteCatalogTableIntrospector(postgres, store);
|
||||||
|
|
||||||
|
const tables = await introspector.scan(database({
|
||||||
|
transport: "rest_api",
|
||||||
|
baseUrl: "https://connector.internal/api/",
|
||||||
|
restPath: "/health",
|
||||||
|
restAuth: "x-api-key",
|
||||||
|
}), new AbortController().signal);
|
||||||
|
|
||||||
|
expect(tables).toEqual([
|
||||||
|
{ name: "patients", sourceComment: "Clinical patients" },
|
||||||
|
{ name: "visits", sourceComment: null },
|
||||||
|
]);
|
||||||
|
expect(fetchMock).toHaveBeenCalledWith(
|
||||||
|
"https://connector.internal/api/rpc/list_tables",
|
||||||
|
expect.objectContaining({
|
||||||
|
method: "POST",
|
||||||
|
headers: expect.objectContaining({ "x-api-key": "rest-secret" }),
|
||||||
|
body: JSON.stringify({ schema_name: "datawarehouse" }),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("fails closed when a REST table row violates the typed contract", async () => {
|
||||||
|
const store = secretStore();
|
||||||
|
const fetchMock = vi.fn(async () => new Response(JSON.stringify([
|
||||||
|
{ type: "TABLE", table_name: "patients", comment: "Wrong field name" },
|
||||||
|
]), { status: 200, headers: { "content-type": "application/json" } }));
|
||||||
|
vi.stubGlobal("fetch", fetchMock);
|
||||||
|
const postgres: CatalogPostgresAccess = {
|
||||||
|
connect: vi.fn(async () => { throw new Error("PostgreSQL wire access must not be used"); }),
|
||||||
|
};
|
||||||
|
const introspector = new ConcreteCatalogTableIntrospector(postgres, store);
|
||||||
|
|
||||||
|
await expect(introspector.scan(database({
|
||||||
|
transport: "rest_api",
|
||||||
|
baseUrl: "https://connector.internal/api",
|
||||||
|
restPath: "/health",
|
||||||
|
restAuth: "none",
|
||||||
|
}), new AbortController().signal)).rejects.toThrow("REST schema response is invalid");
|
||||||
|
});
|
||||||
@@ -0,0 +1,126 @@
|
|||||||
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { afterEach, expect, test, vi } from "vitest";
|
||||||
|
import { buildApp } from "../src/app.js";
|
||||||
|
import { loadConfig } from "../src/config.js";
|
||||||
|
import { MemoryCatalogRepository } from "../src/catalog/memory-repository.js";
|
||||||
|
import { WorkspaceSecretStore } from "../src/workspaces/secret-store.js";
|
||||||
|
import type { WorkspaceRegistry, WorkspaceRevision } from "../src/workspaces/registry.js";
|
||||||
|
import type { WorkspaceDescriptor } from "../src/workspaces/schema.js";
|
||||||
|
|
||||||
|
const roots: string[] = [];
|
||||||
|
afterEach(() => { for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); });
|
||||||
|
|
||||||
|
const workspace: WorkspaceDescriptor = {
|
||||||
|
workspace: { schema_version: 3, id: "psd-clinical", name: "Policlinico San Donato", language: "it" },
|
||||||
|
dwh: {
|
||||||
|
engine: "postgres", database: "warehouse", schema: "datawarehouse", port: 5432,
|
||||||
|
supported_transports: ["postgres_direct", "rest_api"],
|
||||||
|
},
|
||||||
|
semantic_index: {
|
||||||
|
vector_store: { engine: "qdrant", collection: "psd", dimensions: 1024, distance: "cosine" },
|
||||||
|
embedding: { provider: "ollama_internal", model: "qwen3-embedding:0.6b", dimensions: 1024 },
|
||||||
|
},
|
||||||
|
llm_policy: { allowed: ["zai/glm-5.2"] },
|
||||||
|
diagnostics: { dwh_rest: { method: "GET", path: "/health", auth: "bearer", response: { database: "database", schema: "schema" } } },
|
||||||
|
};
|
||||||
|
const revision: WorkspaceRevision = {
|
||||||
|
id: "psd-clinical", commit: "a".repeat(40), blob: "b".repeat(40), snapshotPath: "/tmp/psd.yaml",
|
||||||
|
};
|
||||||
|
|
||||||
|
async function setup() {
|
||||||
|
const secretRoot = mkdtempSync(join(tmpdir(), "catalog-table-secret-"));
|
||||||
|
const runtimeRoot = mkdtempSync(join(tmpdir(), "catalog-table-runtime-"));
|
||||||
|
roots.push(secretRoot, runtimeRoot);
|
||||||
|
const repository = new MemoryCatalogRepository();
|
||||||
|
const database = await repository.create({
|
||||||
|
workspaceId: "psd-clinical",
|
||||||
|
engine: "postgres",
|
||||||
|
databaseName: "warehouse",
|
||||||
|
schema: "datawarehouse",
|
||||||
|
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||||
|
});
|
||||||
|
await repository.recordTest(database.id, database.version, {
|
||||||
|
connectionStatus: "reachable",
|
||||||
|
testedVersion: database.version,
|
||||||
|
lastTestedAt: new Date().toISOString(),
|
||||||
|
});
|
||||||
|
const scan = vi.fn(async () => [
|
||||||
|
{ name: "patients", sourceComment: "Clinical patients" },
|
||||||
|
{ name: "visits", sourceComment: null },
|
||||||
|
]);
|
||||||
|
const registry = {
|
||||||
|
list: vi.fn(async () => [revision]),
|
||||||
|
listCatalog: vi.fn(async () => [{ id: "psd-clinical", name: "Policlinico San Donato", configurationState: "ready", revision }]),
|
||||||
|
read: vi.fn(async () => ({ workspace, revision })),
|
||||||
|
} as unknown as WorkspaceRegistry;
|
||||||
|
const secretStore = new WorkspaceSecretStore({ root: secretRoot, runtimeRoot, installationId: "test" });
|
||||||
|
const app = buildApp(loadConfig({ THT_HARNESS_DIR: "/missing", NODE_ENV: "test" }), {
|
||||||
|
thtRunner: {} as never,
|
||||||
|
workspaceRegistry: registry,
|
||||||
|
workspaceSecretStore: secretStore,
|
||||||
|
catalogRepository: repository,
|
||||||
|
workspaceDiagnoser: vi.fn(),
|
||||||
|
});
|
||||||
|
return { app, repository, database: (await repository.get(database.id))!, scan };
|
||||||
|
}
|
||||||
|
|
||||||
|
test("lists physical tables and updates only review metadata", async () => {
|
||||||
|
const { app, repository, database, scan } = await setup();
|
||||||
|
const synchronized = await repository.reconcileTables(database.id, database.version, await scan(), []);
|
||||||
|
expect(synchronized).toMatchObject({ kind: "applied", createdCount: 2, deletedCount: 0 });
|
||||||
|
expect(scan).toHaveBeenCalledOnce();
|
||||||
|
|
||||||
|
const tables = (await app.inject({
|
||||||
|
method: "GET", url: `/catalog/databases/${database.id}/tables`,
|
||||||
|
})).json();
|
||||||
|
expect(tables.map((table: { name: string }) => table.name)).toEqual(["patients", "visits"]);
|
||||||
|
const patients = tables[0];
|
||||||
|
const edited = await app.inject({
|
||||||
|
method: "PATCH",
|
||||||
|
url: `/catalog/databases/${database.id}/tables/${patients.id}`,
|
||||||
|
payload: { version: patients.version, description: "Curated patient registry" },
|
||||||
|
});
|
||||||
|
expect(edited.statusCode).toBe(200);
|
||||||
|
expect(edited.json()).toMatchObject({
|
||||||
|
name: "patients",
|
||||||
|
sourceComment: "Clinical patients",
|
||||||
|
description: "Curated patient registry",
|
||||||
|
version: 2,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("requires an exact deletion confirmation before applying the atomic diff", async () => {
|
||||||
|
const { app, repository, database, scan } = await setup();
|
||||||
|
await repository.reconcileTables(database.id, database.version, await scan(), []);
|
||||||
|
scan.mockResolvedValue([{ name: "patients", sourceComment: "Clinical patients" }]);
|
||||||
|
|
||||||
|
const preview = await repository.reconcileTables(database.id, database.version, await scan(), []);
|
||||||
|
expect(preview).toEqual({ kind: "confirmation_required", deletedNames: ["visits"] });
|
||||||
|
expect((await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables` })).json()).toHaveLength(2);
|
||||||
|
|
||||||
|
const applied = await repository.reconcileTables(database.id, database.version, await scan(), ["visits"]);
|
||||||
|
expect(applied).toMatchObject({ kind: "applied", deletedCount: 1 });
|
||||||
|
expect((await app.inject({ method: "GET", url: `/catalog/databases/${database.id}/tables` })).json()).toMatchObject([
|
||||||
|
{ name: "patients" },
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("refuses synchronization until the current binding has passed its connection test", async () => {
|
||||||
|
const { app, repository, database } = await setup();
|
||||||
|
await repository.update(database.id, database.version, {
|
||||||
|
workspaceId: database.workspaceId,
|
||||||
|
engine: database.engine,
|
||||||
|
databaseName: database.databaseName,
|
||||||
|
schema: database.schema,
|
||||||
|
binding: database.binding,
|
||||||
|
});
|
||||||
|
const response = await app.inject({
|
||||||
|
method: "POST",
|
||||||
|
url: `/catalog/databases/${database.id}/sync-runs`,
|
||||||
|
payload: { version: database.version + 1, scope: "tables", tableIds: [] },
|
||||||
|
});
|
||||||
|
expect(response.statusCode).toBe(409);
|
||||||
|
expect(response.json()).toMatchObject({ code: "schema_sync_conflict" });
|
||||||
|
});
|
||||||
@@ -289,3 +289,22 @@ test("loadConfig accepts only an absolute generic model key file", () => {
|
|||||||
expect(() => loadConfig({ THT_MODEL_API_KEY_FILE: " /run/secrets/key" }))
|
expect(() => loadConfig({ THT_MODEL_API_KEY_FILE: " /run/secrets/key" }))
|
||||||
.toThrow(/model credential configuration is invalid/);
|
.toThrow(/model credential configuration is invalid/);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test("loadConfig accepts a file-backed catalog role and rejects partial catalog configuration", () => {
|
||||||
|
expect(loadConfig({
|
||||||
|
THT_CATALOG_DB_HOST: "catalog-db",
|
||||||
|
THT_CATALOG_DB_NAME: "thothii_catalog",
|
||||||
|
THT_CATALOG_RUNTIME_USER: "thothii_catalog_runtime",
|
||||||
|
THT_CATALOG_RUNTIME_PASSWORD_FILE: "/run/secrets/catalog_runtime_password",
|
||||||
|
}).catalogDatabase).toEqual({
|
||||||
|
host: "catalog-db",
|
||||||
|
port: 5432,
|
||||||
|
database: "thothii_catalog",
|
||||||
|
user: "thothii_catalog_runtime",
|
||||||
|
passwordFile: "/run/secrets/catalog_runtime_password",
|
||||||
|
});
|
||||||
|
expect(() => loadConfig({ THT_CATALOG_DB_HOST: "catalog-db" }))
|
||||||
|
.toThrow(/catalog database configuration is invalid/);
|
||||||
|
expect(() => loadConfig({ THT_CATALOG_DATABASE_URL: "https://catalog.invalid/db" }))
|
||||||
|
.toThrow(/catalog database configuration is invalid/);
|
||||||
|
});
|
||||||
|
|||||||
@@ -25,6 +25,11 @@ services:
|
|||||||
THT_PI_AUTH_FILE: /home/thoth/.pi/agent/auth.json
|
THT_PI_AUTH_FILE: /home/thoth/.pi/agent/auth.json
|
||||||
THT_AUTH_CONFIG_FILE: /run/thothii-auth/auth.yaml
|
THT_AUTH_CONFIG_FILE: /run/thothii-auth/auth.yaml
|
||||||
THT_AUTH_STATE_ROOT: /data/auth
|
THT_AUTH_STATE_ROOT: /data/auth
|
||||||
|
THT_CATALOG_DB_HOST: catalog-db
|
||||||
|
THT_CATALOG_DB_PORT: "5432"
|
||||||
|
THT_CATALOG_DB_NAME: thothii_catalog
|
||||||
|
THT_CATALOG_RUNTIME_USER: thothii_catalog_runtime
|
||||||
|
THT_CATALOG_RUNTIME_PASSWORD_FILE: /run/secrets/catalog_runtime_password
|
||||||
THT_DB_NAME: ${THT_DB_NAME:-}
|
THT_DB_NAME: ${THT_DB_NAME:-}
|
||||||
THT_DWH_REST_URL: ${THT_DWH_REST_URL:-}
|
THT_DWH_REST_URL: ${THT_DWH_REST_URL:-}
|
||||||
THT_LLM_URL: ${THT_LLM_URL:-}
|
THT_LLM_URL: ${THT_LLM_URL:-}
|
||||||
@@ -47,6 +52,7 @@ services:
|
|||||||
secrets:
|
secrets:
|
||||||
- source: thothii_secrets
|
- source: thothii_secrets
|
||||||
target: thothii.secrets
|
target: thothii.secrets
|
||||||
|
- catalog_runtime_password
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"]
|
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8787/health"]
|
||||||
interval: 15s
|
interval: 15s
|
||||||
@@ -54,6 +60,8 @@ services:
|
|||||||
retries: 5
|
retries: 5
|
||||||
start_period: 30s
|
start_period: 30s
|
||||||
depends_on:
|
depends_on:
|
||||||
|
catalog-db:
|
||||||
|
condition: service_healthy
|
||||||
qdrant:
|
qdrant:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
embedding-model-init:
|
embedding-model-init:
|
||||||
@@ -61,6 +69,51 @@ services:
|
|||||||
networks:
|
networks:
|
||||||
- thothii
|
- thothii
|
||||||
|
|
||||||
|
catalog-db:
|
||||||
|
image: postgres:17.6-bookworm@sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3
|
||||||
|
environment:
|
||||||
|
POSTGRES_DB: thothii_catalog
|
||||||
|
POSTGRES_USER: thothii_catalog_migrate
|
||||||
|
POSTGRES_PASSWORD_FILE: /run/secrets/catalog_migrator_password
|
||||||
|
volumes:
|
||||||
|
- catalog-data:/var/lib/postgresql/data
|
||||||
|
- ./docker/catalog-db-init.sql:/docker-entrypoint-initdb.d/010-runtime-role.sql:ro
|
||||||
|
secrets:
|
||||||
|
- catalog_runtime_password
|
||||||
|
- catalog_migrator_password
|
||||||
|
healthcheck:
|
||||||
|
test:
|
||||||
|
- CMD-SHELL
|
||||||
|
- >-
|
||||||
|
pg_isready -U thothii_catalog_migrate -d thothii_catalog
|
||||||
|
&& test "$(psql -U thothii_catalog_migrate -d thothii_catalog -Atqc
|
||||||
|
"select count(*) from pg_catalog.pg_roles where rolname = 'thothii_catalog_runtime'")" = "1"
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 12
|
||||||
|
start_period: 10s
|
||||||
|
networks:
|
||||||
|
- thothii
|
||||||
|
|
||||||
|
catalog-migrate:
|
||||||
|
image: thothii-core:local
|
||||||
|
profiles: [catalog-maintenance]
|
||||||
|
pull_policy: never
|
||||||
|
command: ["node", "/app/backend/dist/catalog/migrate.js"]
|
||||||
|
environment:
|
||||||
|
THT_CATALOG_DB_HOST: catalog-db
|
||||||
|
THT_CATALOG_DB_PORT: "5432"
|
||||||
|
THT_CATALOG_DB_NAME: thothii_catalog
|
||||||
|
THT_CATALOG_MIGRATOR_USER: thothii_catalog_migrate
|
||||||
|
THT_CATALOG_MIGRATOR_PASSWORD_FILE: /run/secrets/catalog_migrator_password
|
||||||
|
secrets:
|
||||||
|
- catalog_migrator_password
|
||||||
|
depends_on:
|
||||||
|
catalog-db:
|
||||||
|
condition: service_healthy
|
||||||
|
networks:
|
||||||
|
- thothii
|
||||||
|
|
||||||
workspace-maintenance:
|
workspace-maintenance:
|
||||||
image: thothii-core:local
|
image: thothii-core:local
|
||||||
profiles: [workspace-maintenance]
|
profiles: [workspace-maintenance]
|
||||||
@@ -209,7 +262,12 @@ volumes:
|
|||||||
qdrant-data:
|
qdrant-data:
|
||||||
embedding-models:
|
embedding-models:
|
||||||
auth-state:
|
auth-state:
|
||||||
|
catalog-data:
|
||||||
|
|
||||||
secrets:
|
secrets:
|
||||||
thothii_secrets:
|
thothii_secrets:
|
||||||
file: "${THT_SECRETS_FILE:?set THT_SECRETS_FILE}"
|
file: "${THT_SECRETS_FILE:?set THT_SECRETS_FILE}"
|
||||||
|
catalog_runtime_password:
|
||||||
|
file: "${THT_CATALOG_RUNTIME_PASSWORD_SOURCE:-./deploy/secrets/catalog-runtime-password}"
|
||||||
|
catalog_migrator_password:
|
||||||
|
file: "${THT_CATALOG_MIGRATOR_PASSWORD_SOURCE:-./deploy/secrets/catalog-migrator-password}"
|
||||||
|
|||||||
Vendored
+3
@@ -7,6 +7,9 @@ MAX_PI_PROCESSES=4
|
|||||||
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
|
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
|
||||||
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
|
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
|
||||||
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
|
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
|
||||||
|
THT_CATALOG_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/catalog-runtime-password
|
||||||
|
THT_CATALOG_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/catalog-migrator-password
|
||||||
|
THT_CATALOG_SYNC_TIMEOUT_MS=600000
|
||||||
|
|
||||||
THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git
|
THT_WORKSPACE_GIT_REMOTE=https://git.example.invalid/platform/thoth-workspaces.git
|
||||||
THT_WORKSPACE_GIT_BRANCH=main
|
THT_WORKSPACE_GIT_BRANCH=main
|
||||||
|
|||||||
Vendored
+3
@@ -6,6 +6,9 @@ MAX_PI_PROCESSES=4
|
|||||||
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
|
PI_AUTH_FILE=/absolute/path/to/pi-auth.json
|
||||||
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
|
THT_SECRETS_FILE=/absolute/path/to/thothii.secrets
|
||||||
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
|
THT_AUTH_CONFIG_ROOT=/absolute/path/to/thothii-auth
|
||||||
|
THT_CATALOG_RUNTIME_PASSWORD_SOURCE=/absolute/path/to/catalog-runtime-password
|
||||||
|
THT_CATALOG_MIGRATOR_PASSWORD_SOURCE=/absolute/path/to/catalog-migrator-password
|
||||||
|
THT_CATALOG_SYNC_TIMEOUT_MS=600000
|
||||||
|
|
||||||
THT_DATA_ROOT=/srv/thothii/data
|
THT_DATA_ROOT=/srv/thothii/data
|
||||||
THT_PI_STATE_ROOT=/srv/thothii/pi-state
|
THT_PI_STATE_ROOT=/srv/thothii/pi-state
|
||||||
|
|||||||
@@ -0,0 +1,25 @@
|
|||||||
|
DO $bootstrap$
|
||||||
|
DECLARE
|
||||||
|
runtime_password text := trim(both E'\r\n' from pg_read_file('/run/secrets/catalog_runtime_password'));
|
||||||
|
BEGIN
|
||||||
|
IF runtime_password = '' THEN
|
||||||
|
RAISE EXCEPTION 'catalog runtime password is empty';
|
||||||
|
END IF;
|
||||||
|
|
||||||
|
IF NOT EXISTS (
|
||||||
|
SELECT 1 FROM pg_catalog.pg_roles WHERE rolname = 'thothii_catalog_runtime'
|
||||||
|
) THEN
|
||||||
|
EXECUTE format(
|
||||||
|
'CREATE ROLE thothii_catalog_runtime LOGIN PASSWORD %L',
|
||||||
|
runtime_password
|
||||||
|
);
|
||||||
|
END IF;
|
||||||
|
END
|
||||||
|
$bootstrap$;
|
||||||
|
|
||||||
|
GRANT CONNECT ON DATABASE thothii_catalog TO thothii_catalog_runtime;
|
||||||
|
GRANT USAGE ON SCHEMA public TO thothii_catalog_runtime;
|
||||||
|
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||||||
|
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO thothii_catalog_runtime;
|
||||||
|
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||||||
|
GRANT USAGE, SELECT ON SEQUENCES TO thothii_catalog_runtime;
|
||||||
@@ -96,7 +96,9 @@ RUN ln -s /opt/venv /app/harness/.venv
|
|||||||
# Backend: dist + node_modules (stesso Node major 24 + glibc bookworm → compatibili)
|
# Backend: dist + node_modules (stesso Node major 24 + glibc bookworm → compatibili)
|
||||||
COPY --from=backend-build /src/backend/dist /app/backend/dist
|
COPY --from=backend-build /src/backend/dist /app/backend/dist
|
||||||
COPY --from=backend-build /src/backend/node_modules /app/backend/node_modules
|
COPY --from=backend-build /src/backend/node_modules /app/backend/node_modules
|
||||||
|
COPY backend/scripts/ssh-askpass.mjs /app/backend/scripts/ssh-askpass.mjs
|
||||||
COPY backend/package*.json /app/backend/
|
COPY backend/package*.json /app/backend/
|
||||||
|
RUN chmod 0755 /app/backend/scripts/ssh-askpass.mjs
|
||||||
|
|
||||||
# Runtime Pi is installed only from the committed lockfile. The image exposes its immutable
|
# Runtime Pi is installed only from the committed lockfile. The image exposes its immutable
|
||||||
# executable directly, so no host Pi installation or writable global npm directory is needed.
|
# executable directly, so no host Pi installation or writable global npm directory is needed.
|
||||||
|
|||||||
@@ -2,10 +2,12 @@
|
|||||||
|
|
||||||
ThothII userà un Metadata Catalog PostgreSQL interno per conservare, per ogni workspace, la
|
ThothII userà un Metadata Catalog PostgreSQL interno per conservare, per ogni workspace, la
|
||||||
struttura fisica acquisita interrogando il relativo database e i metadati semantici generati con
|
struttura fisica acquisita interrogando il relativo database e i metadati semantici generati con
|
||||||
l'AI. Ogni workspace avrà un solo Workspace Database; identità e lista dei workspace resteranno
|
l'AI. Ogni Workspace Database conserverà un `workspace_id` obbligatorio e univoco: questo realizza
|
||||||
autorevoli in `thoth-workspaces.yaml`, mentre il catalogo ne conserverà soltanto il riferimento
|
l'associazione uno-a-uno senza introdurre nel catalogo una tabella Workspace o una foreign key SQL.
|
||||||
stabile. `schema/annotations.yaml` verrà sostituito come input del core in uno step successivo;
|
Identità e lista dei workspace resteranno autorevoli in `thoth-workspaces.yaml`; il servizio
|
||||||
l'interfaccia e il lifecycle amministrativi resteranno separati dal workflow NL→SQL.
|
validerà il riferimento contro quel catalogo. `schema/annotations.yaml` verrà sostituito come input
|
||||||
|
del core in uno step successivo; l'interfaccia e il lifecycle amministrativi resteranno separati dal
|
||||||
|
workflow NL→SQL.
|
||||||
|
|
||||||
## Considered Options
|
## Considered Options
|
||||||
|
|
||||||
@@ -18,4 +20,7 @@ l'interfaccia e il lifecycle amministrativi resteranno separati dal workflow NL
|
|||||||
|
|
||||||
PSD importerà le annotations esistenti; gli altri workspace genereranno i metadati da zero. Il
|
PSD importerà le annotations esistenti; gli altri workspace genereranno i metadati da zero. Il
|
||||||
cutover futuro dovrà sostituire consapevolmente i consumatori delle annotations e verificarne
|
cutover futuro dovrà sostituire consapevolmente i consumatori delle annotations e verificarne
|
||||||
l'equivalenza semantica. Le sessioni di test esistenti non sono un vincolo di migrazione.
|
l'equivalenza semantica. PostgreSQL può garantire che uno stesso `workspace_id` non sia assegnato a
|
||||||
|
due database, ma l'esistenza del workspace e la gestione di rename o rimozioni restano responsabilità
|
||||||
|
del confine applicativo con il catalogo YAML. Le sessioni di test esistenti non sono un vincolo di
|
||||||
|
migrazione.
|
||||||
|
|||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# Riferimenti al secret store per i Workspace Database
|
||||||
|
|
||||||
|
Il Metadata Catalog non conserverà credenziali o chiavi dei Workspace Database. Riuserà il secret
|
||||||
|
store cifrato già posseduto da ThothII e conserverà soltanto riferimenti ai requisiti segreti del
|
||||||
|
workspace, evitando un secondo vault e impedendo che API, esportazioni o log espongano i valori.
|
||||||
|
|
||||||
|
## Considered Options
|
||||||
|
|
||||||
|
- Copiare i campi testuali del modello ThothAI avrebbe semplificato il CRUD, ma avrebbe conservato
|
||||||
|
password e passphrase in chiaro.
|
||||||
|
- Introdurre subito un secondo vault avrebbe separato il catalogo dal runtime workspace, ma avrebbe
|
||||||
|
duplicato cifratura, rotazione, autorizzazioni e procedure operative senza un'esigenza distinta.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Il catalogo e il runtime condividono l'identità dei requisiti segreti, mentre permessi e API del
|
||||||
|
catalogo restano separati. Sostituzione e cancellazione di un Workspace Database dovranno definire
|
||||||
|
esplicitamente il lifecycle dei relativi riferimenti senza includere i valori nelle transazioni del
|
||||||
|
catalogo PostgreSQL.
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# Binding di database specifiche dell'installazione
|
||||||
|
|
||||||
|
Il Metadata Catalog separa il Workspace Database logico dalla Database Binding che lo rende
|
||||||
|
raggiungibile in una specifica installazione. Esiste un solo Workspace Database per `workspace_id`
|
||||||
|
e una sola binding attiva nel catalogo di ciascuna installazione; per esempio PSD usa REST in locale
|
||||||
|
e PostgreSQL diretto sul server senza diventare due database distinti.
|
||||||
|
|
||||||
|
Nel modello finale il catalogo è autorevole per engine, nome fisico, schema, capability e binding,
|
||||||
|
mentre `thoth-workspaces.yaml` conserva l'identità del workspace. Gli attuali campi DWH dei
|
||||||
|
descriptor sono una sorgente di bootstrap da importare e confrontare durante un cutover esplicito,
|
||||||
|
non una seconda fonte di verità permanente.
|
||||||
|
|
||||||
|
## Considered Options
|
||||||
|
|
||||||
|
- Conservare un record database per ogni trasporto avrebbe duplicato identità, struttura e
|
||||||
|
metadati dello stesso DWH fra locale e server.
|
||||||
|
- Conservare permanentemente i dati DWH sia nello YAML sia nel catalogo avrebbe introdotto
|
||||||
|
conflitti non risolvibili deterministicamente.
|
||||||
|
- Rendere globali le binding avrebbe mescolato endpoint e credenziali che appartengono a
|
||||||
|
installazioni con topologie e confini di sicurezza differenti.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
La lista amministrativa unisce workspace YAML, database configurati e record orphaned. Il cutover
|
||||||
|
deve importare e confrontare la configurazione esistente prima di rimuoverla dai descriptor; il
|
||||||
|
runtime non deve osservare simultaneamente due autorità discordanti.
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# Metadata Catalog nello stesso backend con Kysely
|
||||||
|
|
||||||
|
Il Metadata Catalog vive nello stesso processo Fastify come modulo isolato, invece di introdurre un
|
||||||
|
microservizio. Usa il driver `pg` già presente attraverso Kysely per query, transazioni e migrazioni
|
||||||
|
tipizzate; route, repository, service, readiness e diagnostica restano separati dal workflow e
|
||||||
|
l'indisponibilità del catalogo non rende indisponibili sessioni o SSE.
|
||||||
|
|
||||||
|
## Considered Options
|
||||||
|
|
||||||
|
- Un microservizio avrebbe conservato letteralmente il backend bridge senza database, ma avrebbe
|
||||||
|
aggiunto deployment, autenticazione e failure mode per un solo contesto amministrativo.
|
||||||
|
- Usare soltanto `pg` avrebbe evitato una dipendenza, ma avrebbe richiesto infrastruttura locale per
|
||||||
|
transazioni, tipi delle righe, ordinamento e locking delle migrazioni.
|
||||||
|
- Drizzle o Prisma avrebbero aggiunto schema DSL, generatori e toolchain non necessari a un servizio
|
||||||
|
che vuole mantenere SQL e constraint PostgreSQL espliciti.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
Il backend possiede una connection pool del catalogo e la chiude con il lifecycle Fastify. Le
|
||||||
|
migrazioni timestampate sono compilate insieme al backend ma vengono eseguite soltanto da
|
||||||
|
`catalog:migrate`, con credenziali migrator separate dal ruolo DML usato a runtime. Il modulo resta
|
||||||
|
dietro un'interfaccia repository per mantenere unit test e route test indipendenti da PostgreSQL.
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Hard-delete catalog tables during synchronization
|
||||||
|
|
||||||
|
An explicit Table Synchronization makes the Catalog Table membership exactly match a successful
|
||||||
|
observation of the Workspace Database: new tables are created, source metadata is refreshed, and
|
||||||
|
absent tables plus their future column and relationship children are permanently deleted. Physical
|
||||||
|
membership cannot be edited manually.
|
||||||
|
|
||||||
|
The external scan runs without holding a catalog transaction. Its diff is applied atomically only
|
||||||
|
while the Workspace Database version still matches the scanned binding. Failed scans change
|
||||||
|
nothing, and a non-empty removal set must exactly match the names confirmed by the operator; a
|
||||||
|
changed second scan therefore requires a new confirmation.
|
||||||
|
|
||||||
|
## Considered Options
|
||||||
|
|
||||||
|
- Soft deletion would preserve descriptions across accidental removals, but would add hidden state,
|
||||||
|
restore rules, and ambiguity about whether the catalog still represents the physical schema.
|
||||||
|
- Rename detection based on similarity would preserve metadata in some cases, but could silently
|
||||||
|
attach curated semantics to the wrong physical table.
|
||||||
|
- Append-only introspection, as in the legacy importer, would leave stale tables in the catalog and
|
||||||
|
make downstream schema linking unreliable.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
A physical rename is delete plus create and loses curated metadata. The UI previews permanent
|
||||||
|
deletions, and future Catalog Column and Relationship records must cascade with their table. The
|
||||||
|
catalog remains an exact projection of the last accepted successful scan without tombstones or
|
||||||
|
restore lifecycle.
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# Separate physical and logical relationships
|
||||||
|
|
||||||
|
ThothII persists each database-declared foreign-key constraint as an immutable Catalog
|
||||||
|
Relationship with ordered column pairs, so composite keys retain their identity and the database
|
||||||
|
remains the authority for physical structure. Curated or AI-inferred Logical Relationships will
|
||||||
|
use a separate future model and lifecycle rather than being mixed with physical constraints or
|
||||||
|
denormalized into textual column fields; this keeps synchronization authoritative without
|
||||||
|
preventing later semantic enrichment.
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
# Use durable runs for authoritative schema synchronization
|
||||||
|
|
||||||
|
All table, column, relationship, and full-schema synchronizations run as durable background
|
||||||
|
Catalog Sync Runs rather than separate synchronous and asynchronous implementations. Each scope
|
||||||
|
is authoritative within its boundary, while Synchronize All observes one complete schema snapshot;
|
||||||
|
destructive diffs require confirmation and source revalidation before an atomic, fail-closed
|
||||||
|
catalog transaction. A persistent per-database lock, progress events, interruption handling, and
|
||||||
|
binding-version freshness make long operations observable and prevent two processes or stale
|
||||||
|
configuration from producing a partially trusted catalog.
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
# Domain Docs
|
||||||
|
|
||||||
|
This is a single-context repository.
|
||||||
|
|
||||||
|
Before exploring, read `CONTEXT.md` at the repository root and the relevant decisions in
|
||||||
|
`docs/adr/`. Use the project's terminology from `CONTEXT.md` in issue titles, proposals, and
|
||||||
|
tests. Surface conflicts with an ADR instead of silently overriding it.
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Issue tracker: Gitea
|
||||||
|
|
||||||
|
Issues and specs for this repository live in the self-hosted Gitea repository:
|
||||||
|
`https://git.tylconsulting.it/mptyl/ThothII`.
|
||||||
|
|
||||||
|
## Conventions
|
||||||
|
|
||||||
|
- **Create an issue**: use the repository's Gitea web UI, or the Gitea REST API when an
|
||||||
|
authenticated token with issue scope is available.
|
||||||
|
- **Read and list issues**: use the Gitea web UI or authenticated API; include labels and comments.
|
||||||
|
- **Apply or remove labels**: use the issue's label controls or the Gitea API.
|
||||||
|
- **Comment and close**: use the issue page or the Gitea API.
|
||||||
|
- Do not use `gh issue ...` for this repository: the `github` remote is a mirror, not the canonical
|
||||||
|
issue tracker.
|
||||||
|
|
||||||
|
## Repository identity
|
||||||
|
|
||||||
|
- Canonical Git remote: `origin` → `https://git.tylconsulting.it/mptyl/ThothII.git`
|
||||||
|
- GitHub mirror: `github` → `https://github.com/mptyl/ThothII.git`
|
||||||
|
- Canonical issue URL: `https://git.tylconsulting.it/mptyl/ThothII/issues`
|
||||||
|
|
||||||
|
## When a skill says “publish to the issue tracker”
|
||||||
|
|
||||||
|
Create an issue in the canonical Gitea repository.
|
||||||
|
|
||||||
|
## When a skill says “fetch the relevant ticket”
|
||||||
|
|
||||||
|
Read the referenced issue in the canonical Gitea repository.
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# Triage Labels
|
||||||
|
|
||||||
|
The skills speak in terms of five canonical triage roles. This file maps those roles to the labels
|
||||||
|
used in the canonical Gitea issue tracker.
|
||||||
|
|
||||||
|
| Label in mattpocock/skills | Label in Gitea | Meaning |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
|
||||||
|
| `needs-info` | `needs-info` | Waiting on reporter for more information |
|
||||||
|
| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for AFK agent work |
|
||||||
|
| `ready-for-human` | `ready-for-human` | Requires human implementation |
|
||||||
|
| `wontfix` | `wontfix` | Will not be actioned |
|
||||||
|
|
||||||
|
Issue type labels:
|
||||||
|
|
||||||
|
| Label in Gitea | Meaning |
|
||||||
|
| --- | --- |
|
||||||
|
| `bug` | Something is not working |
|
||||||
|
| `enhancement` | New feature or request |
|
||||||
@@ -4,7 +4,10 @@ This page complements the [architecture overview](overview.md) with the module s
|
|||||||
|
|
||||||
## Modules and dependencies
|
## Modules and dependencies
|
||||||
|
|
||||||
The frontend communicates with the backend through REST and SSE. The backend does not own session persistence: it starts Pi, invokes the `tht` CLI, and forwards events. The harness contains the workflow, the Python CLI, and adapters for the DWH and vector store.
|
The frontend communicates with the backend through REST and SSE. The backend does not own session
|
||||||
|
persistence: it starts Pi, invokes the `tht` CLI, and forwards events. It does own the separate
|
||||||
|
installation-local database catalog. The harness contains the workflow, the Python CLI, and
|
||||||
|
adapters for the DWH and vector store.
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart LR
|
flowchart LR
|
||||||
@@ -18,6 +21,8 @@ flowchart LR
|
|||||||
THT --> DWH["DWH\nread-only"]
|
THT --> DWH["DWH\nread-only"]
|
||||||
THT --> VDB["Qdrant / vector store"]
|
THT --> VDB["Qdrant / vector store"]
|
||||||
BE --> CFG["settings.json\nworkspace registry"]
|
BE --> CFG["settings.json\nworkspace registry"]
|
||||||
|
BE --> CAT["catalog-db\nPostgreSQL + Kysely"]
|
||||||
|
BE -->|catalog Test + Table Sync| DWH
|
||||||
FE -.->|renders widgets| EXT
|
FE -.->|renders widgets| EXT
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -26,7 +31,7 @@ Dipendenze principali:
|
|||||||
| Module | Depends on | Responsibility |
|
| Module | Depends on | Responsibility |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `frontend/` | Backend REST and SSE APIs | UI, gate widgets, and in-memory transcript |
|
| `frontend/` | Backend REST and SSE APIs | UI, gate widgets, and in-memory transcript |
|
||||||
| `backend/src/` | Pi, `tht`, configuration, and workspace registry | Transport, session lifecycle, and APIs |
|
| `backend/src/` | Pi, `tht`, configuration, workspace registry, catalog PostgreSQL, and read-only DWH connectors | Transport, session lifecycle, catalog CRUD, connection tests, table introspection, and APIs |
|
||||||
| `harness/.pi/` | Pi and `tht phase` | Workflow orchestration and human-in-the-loop gates |
|
| `harness/.pi/` | Pi and `tht phase` | Workflow orchestration and human-in-the-loop gates |
|
||||||
| `harness/tht/` | Filesystem, DWH, and vector store | Persistence, CLI, Evidence, schema, and preprocessing |
|
| `harness/tht/` | Filesystem, DWH, and vector store | Persistence, CLI, Evidence, schema, and preprocessing |
|
||||||
| workspace repository | `source/`, `curated/`, manifest, and artifacts | Versioned Evidence source and session output |
|
| workspace repository | `source/`, `curated/`, manifest, and artifacts | Versioned Evidence source and session output |
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ For sessions, roles, groups, diagnostics, and recovery, see the [authentication
|
|||||||
flowchart LR
|
flowchart LR
|
||||||
USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
|
USER["Reviewer"] --> FE["Frontend\nReact and SSE"]
|
||||||
FE --> BE["Backend\nFastify"]
|
FE --> BE["Backend\nFastify"]
|
||||||
|
BE --> CATALOG["Metadata catalog\nPostgreSQL"]
|
||||||
BE --> PI["Pi\nRPC per sessione"]
|
BE --> PI["Pi\nRPC per sessione"]
|
||||||
PI --> THT["tht and harness\nworkflow and persistence"]
|
PI --> THT["tht and harness\nworkflow and persistence"]
|
||||||
THT --> DWH["DWH\nread only"]
|
THT --> DWH["DWH\nread only"]
|
||||||
@@ -27,8 +28,8 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
|
|||||||
|
|
||||||
| Layer | Stack | Ruolo |
|
| Layer | Stack | Ruolo |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **harness/** | Python (`tht` CLI) + Pi gate extension (JS) | Owns the workflow and **all** persistence |
|
| **harness/** | Python (`tht` CLI) + Pi gate extension (JS) | Owns the workflow and all session persistence |
|
||||||
| **backend/** | Fastify + TypeScript | Thin bridge with no database of its own |
|
| **backend/** | Fastify + TypeScript + Kysely | Session bridge plus the isolated administrative metadata catalog |
|
||||||
| **frontend/** | React 18 + Vite | UI that renders gate widgets and rebuilds the live transcript from the SSE stream |
|
| **frontend/** | React 18 + Vite | UI that renders gate widgets and rebuilds the live transcript from the SSE stream |
|
||||||
|
|
||||||
## The harness owns the workflow
|
## The harness owns the workflow
|
||||||
@@ -39,14 +40,22 @@ frontend (React/SSE) → backend (Fastify) → pi --mode rpc → tht/harness →
|
|||||||
|
|
||||||
A session is a directory under `sessions/` (the workspace defines the path): `session_manifest.yaml`, phase artifacts (`question.md`, `schema_linking.json`, `sql_final.sql`, and others), and `review_decisions.jsonl`. The contract says: *"persisted state is the truth; what is not recorded did not happen"*. There is no verbatim transcript store. A resumed Pi process rebuilds context from `tht session show <id>` and the artifacts on disk.
|
A session is a directory under `sessions/` (the workspace defines the path): `session_manifest.yaml`, phase artifacts (`question.md`, `schema_linking.json`, `sql_final.sql`, and others), and `review_decisions.jsonl`. The contract says: *"persisted state is the truth; what is not recorded did not happen"*. There is no verbatim transcript store. A resumed Pi process rebuilds context from `tht session show <id>` and the artifacts on disk.
|
||||||
|
|
||||||
## The backend is a thin bridge with no database
|
## The backend bridges sessions and owns the metadata catalog
|
||||||
|
|
||||||
- `ThtRunner` runs `tht` subcommands in a shell.
|
- `ThtRunner` runs `tht` subcommands in a shell.
|
||||||
- `PiProcessManager` runs one Pi child process per session and bridges its RPC stream.
|
- `PiProcessManager` runs one Pi child process per session and bridges its RPC stream.
|
||||||
- `SessionBridge` maps Pi RPC events to client events (`ui_request` / `text_delta` / `info`).
|
- `SessionBridge` maps Pi RPC events to client events (`ui_request` / `text_delta` / `info`).
|
||||||
- `SseHub` distributes these events to the browser over SSE.
|
- `SseHub` distributes these events to the browser over SSE.
|
||||||
|
- `CatalogService` joins authoritative YAML workspace identities with installation-local database
|
||||||
|
configurations stored in PostgreSQL through Kysely.
|
||||||
|
- `CatalogTableService` reconciles persisted Catalog Tables with a successful external schema scan;
|
||||||
|
`ConcreteCatalogTableIntrospector` isolates direct PostgreSQL, typed REST, and SSH-tunnel access.
|
||||||
|
|
||||||
Application settings live in a JSON file (`backend/data/settings.json`), not in a database.
|
Application settings remain in `backend/data/settings.json`; session state remains in harness phase
|
||||||
|
documents. PostgreSQL stores only the administrative database catalog, bindings, observed tables,
|
||||||
|
and curated descriptions. Connector secrets remain write-only in the encrypted workspace secret
|
||||||
|
store. Catalog SSH support is limited to connection tests and table synchronization; it does not
|
||||||
|
change the session runtime binding contract.
|
||||||
|
|
||||||
## Human-in-the-loop gate contract
|
## Human-in-the-loop gate contract
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,88 @@
|
|||||||
|
# Catalog schema snapshot RPC
|
||||||
|
|
||||||
|
Il percorso preferito per un binding `rest_api` espone al catalogo un'unica fotografia tipizzata
|
||||||
|
dello schema:
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /rpc/schema_snapshot
|
||||||
|
Content-Type: application/json
|
||||||
|
|
||||||
|
{"schema_name":"datawarehouse"}
|
||||||
|
```
|
||||||
|
|
||||||
|
La risposta è un oggetto JSON con `schemaVersion: 1`, capability esplicite e tre collezioni. Una
|
||||||
|
capability non disponibile deve essere dichiarata `unavailable`: non deve essere simulata con una
|
||||||
|
lista vuota.
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"schemaVersion": 1,
|
||||||
|
"capabilities": {
|
||||||
|
"tables": "available",
|
||||||
|
"columns": "available",
|
||||||
|
"relationships": "available"
|
||||||
|
},
|
||||||
|
"tables": [
|
||||||
|
{ "name": "patients", "sourceComment": "Clinical patients" }
|
||||||
|
],
|
||||||
|
"columns": [
|
||||||
|
{
|
||||||
|
"tableName": "patients",
|
||||||
|
"name": "id",
|
||||||
|
"ordinalPosition": 1,
|
||||||
|
"dataType": "bigint",
|
||||||
|
"isNullable": false,
|
||||||
|
"defaultExpression": null,
|
||||||
|
"primaryKeyPosition": 1,
|
||||||
|
"sourceComment": "Patient identifier"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"relationships": [
|
||||||
|
{
|
||||||
|
"constraintName": "visits_patient_id_fkey",
|
||||||
|
"sourceTableName": "visits",
|
||||||
|
"targetTableName": "patients",
|
||||||
|
"updateRule": "NO ACTION",
|
||||||
|
"deleteRule": "CASCADE",
|
||||||
|
"deferrable": false,
|
||||||
|
"initiallyDeferred": false,
|
||||||
|
"columns": [
|
||||||
|
{ "position": 1, "sourceColumnName": "patient_id", "targetColumnName": "id" }
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fallback compatibile tramite `run_query`
|
||||||
|
|
||||||
|
Se e soltanto se il server non espone `POST /rpc/schema_snapshot`, il catalogo può ottenere la
|
||||||
|
stessa fotografia mediante una singola istruzione read-only inviata all'RPC già esistente:
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /rpc/run_query
|
||||||
|
Content-Type: application/json
|
||||||
|
|
||||||
|
{"query_text":"WITH ... SELECT ..."}
|
||||||
|
```
|
||||||
|
|
||||||
|
La query è costruita dal catalogo, interroga soltanto il catalogo PostgreSQL dello schema
|
||||||
|
configurato e aggrega tabelle, colonne, primary key e foreign key nella stessa istruzione. Non sono
|
||||||
|
ammessi più round trip, query per tabella o assemblaggi client-side di osservazioni effettuate in
|
||||||
|
momenti diversi. Il nome schema deve essere validato come identificatore e quotato come valore SQL,
|
||||||
|
non interpolato come SQL libero.
|
||||||
|
|
||||||
|
`run_query` restituisce un array JSON di righe. Per questo fallback l'array deve contenere
|
||||||
|
esattamente una riga e quella riga deve essere esattamente l'oggetto snapshot v1 sopra descritto,
|
||||||
|
con `schemaVersion`, `capabilities`, `tables`, `columns` e `relationships`; campi mancanti,
|
||||||
|
aggiuntivi o di tipo diverso rendono invalida l'intera fotografia. La risposta non è un contratto
|
||||||
|
alternativo o più permissivo: cambia soltanto il trasporto della stessa snapshot stretta.
|
||||||
|
|
||||||
|
`position` e `primaryKeyPosition` sono uno-based. Le coppie ordinate permettono foreign key
|
||||||
|
composte. Il catalogo rifiuta l'intera fotografia se il JSON non rispetta il contratto o se la
|
||||||
|
capability richiesta dal tipo di sincronizzazione è `unavailable`; in entrambi i casi non applica
|
||||||
|
alcuna modifica. Il fallback viene tentato soltanto quando l'RPC preferito risulta assente, non per
|
||||||
|
nascondere una snapshot malformata o un errore operativo del server. Se anche `run_query` non è
|
||||||
|
disponibile, la query viene rifiutata, la risposta non contiene una singola snapshot v1 valida o una
|
||||||
|
capability richiesta è `unavailable`, il run fallisce senza aggiornamenti parziali e senza esporre
|
||||||
|
il corpo remoto.
|
||||||
@@ -92,8 +92,10 @@ evidence:
|
|||||||
|
|
||||||
Changes from older workspaces:
|
Changes from older workspaces:
|
||||||
|
|
||||||
- the database is reached only through **REST** or **direct Postgres** (`rest_api` /
|
- NL→SQL sessions reach the database only through **REST** or **direct Postgres** (`rest_api` /
|
||||||
`postgres_direct`); the SSH tunnel remains disabled;
|
`postgres_direct`); `ssh_tunnel` remains disabled for session runtime. The separate Database
|
||||||
|
management surface supports SSH for **Test connection** and **Sync tables**, with a private key,
|
||||||
|
mandatory `known_hosts`, and an optional key passphrase;
|
||||||
- the semantic index is **internal** (Qdrant plus `qwen3-embedding:0.6b`, 1024 dimensions, cosine);
|
- the semantic index is **internal** (Qdrant plus `qwen3-embedding:0.6b`, 1024 dimensions, cosine);
|
||||||
- **filesystem** Evidence lives in the repository (`<id>/evidence`) and is materialized from the
|
- **filesystem** Evidence lives in the repository (`<id>/evidence`) and is materialized from the
|
||||||
pinned Git commit (P6). HTTP Evidence is also supported.
|
pinned Git commit (P6). HTTP Evidence is also supported.
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
# Metadata Catalog di ThothII: ricognizione ThothAI e percorso incrementale
|
# Metadata Catalog di ThothII: ricognizione ThothAI e percorso incrementale
|
||||||
|
|
||||||
Data: 2026-08-26
|
Data: 2026-08-26; aggiornato 2026-08-27
|
||||||
Stato: ricognizione completata; step 1 implementato; scelte tecnologiche degli step successivi
|
Stato: ricognizione e progettazione completate; navigazione, CRUD Workspace Database, Catalog
|
||||||
deliberatamente rinviate.
|
Table, Catalog Column, Catalog Relationship e sincronizzazione durevole dello schema implementati
|
||||||
|
il 2026-08-27. Generazione AI e integrazione con il workflow core restano negli step successivi.
|
||||||
|
|
||||||
## Obiettivo
|
## Obiettivo
|
||||||
|
|
||||||
@@ -10,9 +11,11 @@ ThothII deve introdurre un contesto amministrativo separato, il **Metadata Catal
|
|||||||
il database associato a ciascun workspace, la sua struttura fisica introspezionata e i metadati
|
il database associato a ciascun workspace, la sua struttura fisica introspezionata e i metadati
|
||||||
semantici oggi rappresentati da `schema/annotations.yaml`.
|
semantici oggi rappresentati da `schema/annotations.yaml`.
|
||||||
|
|
||||||
Il programma procede per step indipendenti. Il primo step aggiunge soltanto l'accesso dalla sidebar
|
Il programma procede per step indipendenti. Il primo step ha aggiunto l'accesso dalla sidebar; il
|
||||||
destra a una superficie centrale vuota. Non introduce PostgreSQL, API CRUD, introspezione o
|
secondo ha sostituito la superficie vuota con il CRUD di configurazione, il PostgreSQL interno e i
|
||||||
integrazioni con il workflow core.
|
test di connessione; gli step successivi hanno aggiunto navigazione gerarchica, colonne, relazioni
|
||||||
|
fisiche e sincronizzazione durevole dell'intero schema. Non introduce ancora generazione AI o
|
||||||
|
integrazione con il workflow core.
|
||||||
|
|
||||||
Questa analisi usa come riferimento il working tree legacy osservato in
|
Questa analisi usa come riferimento il working tree legacy osservato in
|
||||||
`Thoth/ThothAI`. Non è stato verificato che quel contenuto corrisponda a una release o a un tag
|
`Thoth/ThothAI`. Non è stato verificato che quel contenuto corrisponda a una release o a un tag
|
||||||
@@ -20,8 +23,10 @@ canonico; i percorsi e i comportamenti descrivono il sorgente disponibile il 202
|
|||||||
|
|
||||||
## Decisioni già confermate
|
## Decisioni già confermate
|
||||||
|
|
||||||
1. Ogni workspace è associato a un solo Workspace Database e ogni Workspace Database appartiene a
|
1. Ogni Workspace Database appartiene a un solo workspace tramite un `workspace_id` obbligatorio e
|
||||||
un solo workspace.
|
univoco; un workspace può avere al massimo un Workspace Database. Poiché i workspace non sono
|
||||||
|
righe del catalogo PostgreSQL, l'associazione è un riferimento logico validato contro
|
||||||
|
`thoth-workspaces.yaml`, non una foreign key SQL.
|
||||||
2. Il CRUD non crea né rinomina workspace. Identità e lista ordinata dei workspace restano
|
2. Il CRUD non crea né rinomina workspace. Identità e lista ordinata dei workspace restano
|
||||||
autorevoli in `thoth-workspaces.yaml`; il catalogo conserva il loro identificatore stabile.
|
autorevoli in `thoth-workspaces.yaml`; il catalogo conserva il loro identificatore stabile.
|
||||||
3. La struttura fisica viene acquisita interrogando il database esterno tramite i dati di
|
3. La struttura fisica viene acquisita interrogando il database esterno tramite i dati di
|
||||||
@@ -38,6 +43,146 @@ canonico; i percorsi e i comportamenti descrivono il sorgente disponibile il 202
|
|||||||
introduce un router.
|
introduce un router.
|
||||||
9. La pagina iniziale è vuota, segue il tema, nasconde l'intera colonna core e non interrompe una
|
9. La pagina iniziale è vuota, segue il tema, nasconde l'intera colonna core e non interrompe una
|
||||||
sessione live. Le azioni di apertura, resume o creazione sessione riportano al core.
|
sessione live. Le azioni di apertura, resume o creazione sessione riportano al core.
|
||||||
|
10. La compatibilità con il modello ThothAI è semantica, non una copia letterale: configurazione e
|
||||||
|
contenuti semantici sono campi relazionali mutabili, mentre identità e appartenenza della
|
||||||
|
struttura fisica derivano dall'introspezione; i segreti restano nel secret store e lo stato dei
|
||||||
|
job non viene mescolato ai dati amministrativi.
|
||||||
|
11. Il CRUD amministra il Metadata Catalog e non esegue DDL sul database esterno, che resta
|
||||||
|
read-only.
|
||||||
|
12. La prima versione supporta PostgreSQL; il confine di introspezione dovrà permettere di
|
||||||
|
aggiungere altri dialetti senza cambiare il modello del catalogo.
|
||||||
|
13. I segreti dei Workspace Database riusano il secret store cifrato di ThothII. Il catalogo
|
||||||
|
conserva riferimenti ai segreti e nessuna API, esportazione o log ne restituisce i valori.
|
||||||
|
14. La UI usa AG Grid Community per la lista master e un pannello React separato per il dettaglio;
|
||||||
|
non dipende dalle funzionalità master-detail di AG Grid Enterprise.
|
||||||
|
15. Un Workspace Database il cui `workspace_id` scompare dal catalogo YAML non viene cancellato
|
||||||
|
automaticamente: diventa orphaned e può soltanto essere recuperato, riassegnato o eliminato
|
||||||
|
esplicitamente da un amministratore.
|
||||||
|
16. La prima vertical slice gestisce configurazione del Workspace Database, riferimenti ai segreti,
|
||||||
|
test di connessione e stato. La seconda gestisce le Catalog Table: la collezione e i nomi sono
|
||||||
|
controllati dall'introspezione, mentre la descrizione curata è modificabile. Le slice successive
|
||||||
|
hanno aggiunto Catalog Column, Catalog Relationship e sincronizzazione durevole dello schema.
|
||||||
|
17. Il modello non conserva il `name` libero di ThothAI: nome e ID visualizzati appartengono al
|
||||||
|
workspace YAML, mentre `database_name` identifica il database PostgreSQL esterno.
|
||||||
|
18. Database management supporta i tre trasporti già riconosciuti da ThothII: `postgres_direct`,
|
||||||
|
`rest_api` e `ssh_tunnel`. PSD rimane un solo Workspace Database: usa la connessione diretta sul
|
||||||
|
server e l'endpoint REST in locale tramite una Database Binding specifica dell'installazione.
|
||||||
|
Questo supporto non abilita automaticamente `ssh_tunnel` nel runtime NL→SQL.
|
||||||
|
19. Una configurazione può essere salvata prima di una connessione riuscita. Il test separato
|
||||||
|
produce uno stato `untested`, `reachable` o `failed`; attivazione e introspezione richiedono uno
|
||||||
|
stato raggiungibile.
|
||||||
|
20. Il CRUD e il test di connessione richiedono `database.manage`; inserimento e sostituzione dei
|
||||||
|
segreti continuano a richiedere `workspace.secrets.manage`.
|
||||||
|
21. Il Workspace Database e il modo di raggiungerlo sono entità distinte. Ogni catalogo di
|
||||||
|
installazione conserva una sola Database Binding attiva per workspace: PSD usa `rest_api` in
|
||||||
|
locale e `postgres_direct` sul server senza duplicare il Workspace Database.
|
||||||
|
22. Nel modello finale il Metadata Catalog è autorevole per engine, `database_name`, schema,
|
||||||
|
capacità e binding. Lo YAML resta autorevole per identità e contenuti del workspace; i campi
|
||||||
|
DWH correnti saranno importati, confrontati e rimossi soltanto durante un cutover esplicito.
|
||||||
|
23. La lista master è l'unione fra workspace YAML e record del catalogo: mostra workspace
|
||||||
|
`unconfigured`, database configurati e record `orphaned`.
|
||||||
|
24. Ogni introspezione registra le capability disponibili. Una capability `unavailable` non viene
|
||||||
|
rappresentata come una collezione osservata ma vuota; REST può completare con successo anche
|
||||||
|
quando indici o enum non sono supportati.
|
||||||
|
25. Il Metadata Catalog non introduce snapshot, draft o pubblicazioni. Configurazione e contenuti
|
||||||
|
semantici, inclusi quelli futuri generati dall'AI, sono normali campi modificabili; la struttura
|
||||||
|
osservata cambia soltanto con una sincronizzazione esplicita.
|
||||||
|
26. Il normale Delete elimina realmente il Workspace Database, la Database Binding e i relativi
|
||||||
|
record catalogo e segreti. Non modifica il DWH esterno né il repository YAML; il workspace torna
|
||||||
|
visibile nella lista master come `unconfigured`.
|
||||||
|
27. La prima versione gestisce un solo schema obbligatorio per Workspace Database, identificato
|
||||||
|
dalla coppia `database_name + schema`; per PSD la coppia è `postgres + datawarehouse`.
|
||||||
|
28. I record mantengono soltanto `created_at`, `updated_at` e un contatore `version` per optimistic
|
||||||
|
concurrency. Non esistono storico delle revisioni, rollback o audit applicativo delle modifiche.
|
||||||
|
29. `workspace_databases` conserva soltanto UUID, `workspace_id` unique, engine, `database_name`,
|
||||||
|
schema, timestamp e version. Il nome visualizzato appartiene al workspace YAML.
|
||||||
|
30. Ogni Workspace Database ha al massimo una riga `database_bindings`. Una singola tabella usa
|
||||||
|
check constraint dipendenti da `transport` per i campi direct, REST e SSH; non esiste un flag
|
||||||
|
`active`, perché ciascuna installazione conserva una sola binding.
|
||||||
|
31. `rest_api` configura il Thoth REST Connector tipizzato: base URL, autenticazione e TLS sono dati
|
||||||
|
della binding, mentre path RPC e shape delle risposte appartengono al contratto applicativo e non
|
||||||
|
sono liberamente configurabili.
|
||||||
|
32. Il test connessione usa soltanto una configurazione già salvata ed è associato alla sua
|
||||||
|
`version`. Ogni modifica della binding o dei segreti invalida il risultato precedente e riporta
|
||||||
|
lo stato a `untested`.
|
||||||
|
33. Password, API key e chiavi sono write-only: l'API espone soltanto `configured`, un campo vuoto
|
||||||
|
conserva il valore esistente e la sostituzione è un'azione esplicita. Delete rimuove anche i
|
||||||
|
segreti associati.
|
||||||
|
34. La pagina usa AG Grid come master e un form React come detail, con sezioni Database, Connection
|
||||||
|
e TLS/SSH condizionali. La toolbar offre `Add database`; le righe `unconfigured` offrono
|
||||||
|
`Configure`. Entrambe selezionano esclusivamente workspace YAML senza un database e creano il
|
||||||
|
record soltanto al Save; `workspace_id` diventa immutabile dopo la creazione.
|
||||||
|
35. La grid mostra workspace, database, schema, transport, endpoint, stato connessione e ultimo
|
||||||
|
aggiornamento. Su schermi piccoli il dettaglio occupa il pannello completo. Il cambio riga con
|
||||||
|
modifiche non salvate e Delete richiedono conferma, senza conferma testuale tipizzata.
|
||||||
|
36. La Database Binding conserva `connection_status`, `tested_version`, `last_tested_at`, un codice
|
||||||
|
errore e un messaggio breve sanificato. Non conserva stack trace, DSN, credenziali o output grezzo
|
||||||
|
del driver.
|
||||||
|
37. Le API vivono sotto `/api/catalog`: list/create di `/databases`, get/patch/delete di
|
||||||
|
`/databases/:id`, sostituzione dei segreti sotto `/databases/:id/secrets`, test connessione sotto
|
||||||
|
`/databases/:id/test` e list/patch/sync delle tabelle sotto `/databases/:id/tables`.
|
||||||
|
38. `GET /api/catalog/databases` restituisce l'intera master list unificata; AG Grid Community applica
|
||||||
|
client-side ricerca, filtri e ordinamento. La prima versione non introduce paginazione server o
|
||||||
|
funzionalità AG Grid Enterprise.
|
||||||
|
39. Il Metadata Catalog vive nello stesso processo Fastify come modulo isolato con repository,
|
||||||
|
service, route, diagnostica e readiness proprie. L'indisponibilità del catalogo non modifica
|
||||||
|
sessioni, SSE o health del core e non giustifica ancora un microservizio separato.
|
||||||
|
40. Il backend mantiene `pg@8.22.0` e aggiunge `kysely@0.29.5` per query e transazioni tipizzate. Le
|
||||||
|
migrazioni Kysely sono timestampate, compilate con il backend ed eseguite da un comando
|
||||||
|
`catalog:migrate` separato; l'applicazione non migra automaticamente il database all'avvio.
|
||||||
|
41. Lo stack aggiunge un servizio interno `catalog-db` con volume persistente, ruolo runtime DML,
|
||||||
|
ruolo migrator DDL e job one-shot `catalog-migrate`. Un catalogo indisponibile produce 503 sulle
|
||||||
|
sole route catalogo.
|
||||||
|
42. La prima vertical slice è amministrativa: scrive il catalogo ma non cambia ancora il runtime di
|
||||||
|
sessioni e workflow, che continua a usare YAML e binding correnti fino al cutover esplicito.
|
||||||
|
43. `Configure` precompila senza salvare engine, database e schema dal descriptor e i dati non
|
||||||
|
sensibili dalla binding effettiva. L'amministratore verifica, inserisce i segreti e salva; non
|
||||||
|
esiste importazione silenziosa.
|
||||||
|
44. Unit e route test usano un repository fake; una suite PostgreSQL Testcontainers separata verifica
|
||||||
|
migrazioni, constraint, transazioni, optimistic concurrency e cascade. SQLite ed emulatori non
|
||||||
|
sono sostituti ammessi per questi test.
|
||||||
|
45. La navigazione delle entità catalogo è gerarchica e senza scorciatoie globali: `Databases →
|
||||||
|
Database → Overview | Tables → Table`. Non esistono una voce globale Tables, un filtro globale
|
||||||
|
Database o una preselezione implicita; Columns continuerà sotto Table e Relationships sotto
|
||||||
|
Database.
|
||||||
|
46. Una Catalog Table conserva nome fisico, `source_comment`, descrizione curata nullable,
|
||||||
|
`generated_description` nullable per lo step AI futuro, version e timestamp. La UI mostra come
|
||||||
|
tre campi indipendenti senza fallback visivo: source comment read-only, generated description
|
||||||
|
modificabile e description modificabile. I valori null restano celle e controlli vuoti.
|
||||||
|
47. Le Catalog Table non possono essere aggiunte, rinominate o cancellate manualmente. `Sync tables`
|
||||||
|
legge dal database esterno le tabelle PostgreSQL ordinarie e partizionate dello schema scelto;
|
||||||
|
viste e materialized view sono escluse.
|
||||||
|
48. La sincronizzazione è esplicita. La scansione avviene fuori dalla transazione del catalogo; il
|
||||||
|
diff viene applicato atomicamente soltanto se la version del Workspace Database è ancora quella
|
||||||
|
sottoposta a scansione. Una scansione fallita non modifica il catalogo.
|
||||||
|
49. Tabelle nuove vengono create, i commenti sorgente vengono aggiornati e quelle non più osservate
|
||||||
|
vengono eliminate definitivamente. La rimozione di tabelle, colonne o relazioni richiede la
|
||||||
|
conferma dell'esatto piano distruttivo; se il secondo scan produce una fotografia differente,
|
||||||
|
l'applicazione richiede una nuova conferma.
|
||||||
|
50. Un rename fisico è intenzionalmente delete più create e perde i metadati curati. Le colonne e
|
||||||
|
relazioni dipendenti vengono eliminate in cascade insieme alla Catalog Table.
|
||||||
|
51. L'introspezione vive nel modulo catalogo Fastify dietro un adapter. PostgreSQL diretto e tunnel
|
||||||
|
SSH usano il catalogo `pg_catalog`; REST preferisce il contratto tipizzato
|
||||||
|
`POST /rpc/schema_snapshot` e, quando quell'RPC non è esposto, usa come fallback compatibile una
|
||||||
|
singola query read-only tramite `POST /rpc/run_query`. Entrambi i percorsi devono produrre la
|
||||||
|
stessa fotografia v1 stretta descritta in `docs/contracts/catalog-schema-snapshot.md`.
|
||||||
|
52. Test connessione e sincronizzazione sono serializzati per Workspace Database, hanno timeout e
|
||||||
|
richiedono che la binding nella version corrente abbia un test `reachable` prima di qualsiasi
|
||||||
|
Catalog Sync Run. La scansione asincrona ha un timeout separato, di default dieci minuti.
|
||||||
|
53. Il tunnel SSH usa OpenSSH in modalità stdio `-W`, chiave privata e passphrase opzionale dal
|
||||||
|
secret store, `known_hosts` obbligatorio, `StrictHostKeyChecking=yes`, agent e configurazione
|
||||||
|
globale disabilitati. Non è ammesso TOFU. TLS PostgreSQL con CA e server name resta verificato
|
||||||
|
anche attraverso il tunnel.
|
||||||
|
54. In questo slice `ssh_tunnel` è una binding supportata da Database management per Test connection
|
||||||
|
e Schema Sync. Il renderer e il runtime delle sessioni NL→SQL restano fuori scope e continuano a
|
||||||
|
rifiutarla finché non verrà deciso il relativo cutover.
|
||||||
|
55. I menu di azione a livello Workspace Database espongono separatamente `Synchronize tables`,
|
||||||
|
`Synchronize all columns`, `Synchronize relationships` e `Synchronize all`. Su una selezione di
|
||||||
|
database lo scope scelto viene avviato per ogni database idoneo; non viene sostituito
|
||||||
|
implicitamente con una sincronizzazione completa.
|
||||||
|
56. Per lo scope Columns, `tableIds` vuoto significa tutte le Catalog Table correnti del Workspace
|
||||||
|
Database; `tableIds` valorizzato limita invece la riconciliazione alle tabelle indicate. La grid
|
||||||
|
Tables espone `Synchronize columns` sulle tabelle selezionate.
|
||||||
|
|
||||||
## Correzione del modello mentale corrente
|
## Correzione del modello mentale corrente
|
||||||
|
|
||||||
@@ -205,8 +350,8 @@ template Django.
|
|||||||
|
|
||||||
Il comportamento è principalmente additivo: usa `get_or_create` o controlli `exists`, aggiorna
|
Il comportamento è principalmente additivo: usa `get_or_create` o controlli `exists`, aggiorna
|
||||||
alcuni commenti, ma non riconcilia in modo completo rename, rimozioni o drift. Non va copiato così
|
alcuni commenti, ma non riconcilia in modo completo rename, rimozioni o drift. Non va copiato così
|
||||||
com'è. Il futuro processo ThothII dovrà almeno distinguere scansione, differenze osservate e
|
com'è. Il processo ThothII implementato distingue scansione, differenze osservate e applicazione
|
||||||
applicazione della nuova snapshot.
|
della nuova snapshot.
|
||||||
|
|
||||||
### Generazione AI legacy
|
### Generazione AI legacy
|
||||||
|
|
||||||
@@ -318,7 +463,8 @@ rendering, retrieval, LSH o SQL generation.
|
|||||||
|
|
||||||
### Vincoli minimi da progettare
|
### Vincoli minimi da progettare
|
||||||
|
|
||||||
- `workspace_id` unico sul Workspace Database;
|
- `workspace_id` obbligatorio e unico sul Workspace Database, con esistenza validata contro il
|
||||||
|
catalogo YAML dal servizio applicativo;
|
||||||
- nome tabella unico nel database e schema appropriato;
|
- nome tabella unico nel database e schema appropriato;
|
||||||
- nome colonna unico nella tabella;
|
- nome colonna unico nella tabella;
|
||||||
- relationship unica secondo il modello, anche per chiavi composite;
|
- relationship unica secondo il modello, anche per chiavi composite;
|
||||||
@@ -348,8 +494,7 @@ L'export legacy della struttura include username e password in chiaro. Il modell
|
|||||||
password, passphrase SSH e altri segreti in `CharField`; non è stata trovata cifratura applicativa,
|
password, passphrase SSH e altri segreti in `CharField`; non è stata trovata cifratura applicativa,
|
||||||
nonostante un testo admin affermi il contrario.
|
nonostante un testo admin affermi il contrario.
|
||||||
|
|
||||||
Per ThothII resta da decidere nello step infrastrutturale quali dati di connessione siano normali
|
ThothII distingue i metadati di connessione dai riferimenti al secret store cifrato. In ogni caso:
|
||||||
metadati e quali siano secret reference. In ogni caso:
|
|
||||||
|
|
||||||
- nessun endpoint o export deve restituire segreti;
|
- nessun endpoint o export deve restituire segreti;
|
||||||
- log ed errori devono sanificare DSN e credenziali;
|
- log ed errori devono sanificare DSN e credenziali;
|
||||||
@@ -357,6 +502,11 @@ metadati e quali siano secret reference. In ogni caso:
|
|||||||
- il catalogo non deve riusare credenziali del DWH, delle sessioni o di Qdrant;
|
- il catalogo non deve riusare credenziali del DWH, delle sessioni o di Qdrant;
|
||||||
- test connessione e introspezione devono usare timeout e privilegi read-only.
|
- test connessione e introspezione devono usare timeout e privilegi read-only.
|
||||||
|
|
||||||
|
La binding REST corrente richiede una verifica prima del cutover: il renderer emette
|
||||||
|
`ssl_ca_file`, mentre il modello Python espone `ssl_ca`; il percorso della CA privata potrebbe quindi
|
||||||
|
non essere consumato. PSD richiede TLS con CA privata in locale, perciò questo disallineamento deve
|
||||||
|
essere corretto e coperto da un test end-to-end prima di affidare il profilo REST al catalogo.
|
||||||
|
|
||||||
## Percorso incrementale
|
## Percorso incrementale
|
||||||
|
|
||||||
### Step 1: accesso alla superficie vuota
|
### Step 1: accesso alla superficie vuota
|
||||||
@@ -384,15 +534,17 @@ npx tsc -b
|
|||||||
|
|
||||||
### Step 2: contratto di dominio e schema relazionale
|
### Step 2: contratto di dominio e schema relazionale
|
||||||
|
|
||||||
Da progettare con un nuovo round decisionale: campi, secret reference, dialetti supportati,
|
Progettazione della vertical slice completata: Workspace Database, Database Binding, singolo schema,
|
||||||
namespace/schema, snapshot fisiche, relazioni fisiche/logiche e lifecycle dell'output AI. Nessuna
|
riferimenti al secret store, optimistic concurrency e capability per trasporto hanno contratti
|
||||||
tecnologia ORM o migration tool è stata scelta in questo documento.
|
espliciti. Configurazione e contenuti semantici restano mutabili; la struttura fisica osservata è
|
||||||
|
sincronizzata e non modificabile manualmente.
|
||||||
|
|
||||||
### Step 3: PostgreSQL interno e migrazioni
|
### Step 3: PostgreSQL interno e migrazioni
|
||||||
|
|
||||||
Da progettare separatamente dal core: servizio, volume, ruoli runtime/migrator/backup, health e
|
PostgreSQL interno con volume e ruoli runtime/migrator separati. Il modulo catalogo usa Kysely sopra
|
||||||
readiness dedicati, backup/restore e diagnostica. La sua indisponibilità non dovrà cambiare
|
il driver `pg`; le migrazioni compilate vengono applicate soltanto dal comando `catalog:migrate` e
|
||||||
`core /health` o interrompere una sessione.
|
mai allo startup Fastify. Health, readiness e diagnostica restano dedicate; l'indisponibilità del
|
||||||
|
catalogo non cambia `core /health` e non interrompe una sessione.
|
||||||
|
|
||||||
### Step 4: API CRUD
|
### Step 4: API CRUD
|
||||||
|
|
||||||
@@ -401,20 +553,56 @@ Gli endpoint dovranno vivere sotto un namespace catalogo e non riutilizzare le r
|
|||||||
|
|
||||||
### Step 5: UI CRUD
|
### Step 5: UI CRUD
|
||||||
|
|
||||||
Liste e form per Workspace Database, tabelle, colonne e relazioni, costruiti con React/Vite e il
|
Workspace Database, Catalog Table, Catalog Column e Catalog Relationship sono implementati con
|
||||||
design system ThothII. La gerarchia e i filtri ThothAI sono il riferimento funzionale; Django Admin
|
React/Vite e il design system ThothII.
|
||||||
non è il riferimento tecnologico o visuale.
|
La navigazione è gerarchica e locale al database (`Overview | Tables`), senza menu o filtri globali
|
||||||
|
per tipo di entità. La grid delle tabelle non offre Add/Delete; il dettaglio full-width mantiene
|
||||||
|
immutabili i fatti fisici e consente di modificare separatamente Description e Generated
|
||||||
|
Description. Colonne e relazioni seguono la stessa gerarchia: Columns appartiene al dettaglio
|
||||||
|
della tabella, Relationships al database. I valori descrittivi null sono mostrati come celle e
|
||||||
|
campi vuoti, senza fallback visivi o placeholder `Not set` che nascondano quale sorgente è
|
||||||
|
effettivamente valorizzata.
|
||||||
|
|
||||||
|
Le griglie che dispongono di azioni massive usano checkbox e una toolbar contestuale con conteggio,
|
||||||
|
menu `Actions` e cancellazione della selezione. La selezione identifica ID espliciti, può essere
|
||||||
|
accumulata attraverso i filtri e viene azzerata dopo successo, nuova sincronizzazione o uscita
|
||||||
|
dalla pagina; un'azione è all-or-nothing se un elemento non è idoneo. I menu a livello database
|
||||||
|
espongono gli scope fisici come azioni distinte: `Synchronize tables`, `Synchronize all columns`,
|
||||||
|
`Synchronize relationships` e `Synchronize all`. La grid Tables espone invece `Synchronize
|
||||||
|
columns` per le tabelle selezionate. Test connection resta un'azione distinta; griglie senza azioni
|
||||||
|
non mostrano controlli di selezione inerti.
|
||||||
|
|
||||||
### Step 6: introspezione
|
### Step 6: introspezione
|
||||||
|
|
||||||
Connessione read-only, preview delle differenze, acquisizione di una Physical Schema Snapshot,
|
Catalog Table, Catalog Column e Catalog Relationship sono implementate per PostgreSQL diretto,
|
||||||
policy per rename/rimozioni e stato del job. Nessuna chiamata lunga dovrà mantenere aperta una
|
Thoth REST Connector e tunnel SSH. La scansione read-only è separata dalla transazione; una
|
||||||
transazione CRUD.
|
riconciliazione atomica crea, aggiorna i commenti sorgente ed elimina, dopo conferma, i fatti fisici
|
||||||
|
assenti senza rendere modificabile manualmente la struttura osservata. Gli scope autorevoli sono
|
||||||
|
Tables per database e Physical Relationships per database. Per Columns, `tableIds` vuoto include
|
||||||
|
tutte le Catalog Table correnti, mentre una lista di ID limita lo scope al sottoinsieme esplicito;
|
||||||
|
`Synchronize all` osserva tutti e tre gli scope in un unico snapshot e li riconcilia insieme. Tutti
|
||||||
|
gli scope sono eseguiti come Catalog Sync Run durevoli in background, non attraverso implementazioni
|
||||||
|
sincrone e asincrone separate. Un run che prevede cancellazioni conserva il diff, attende una
|
||||||
|
conferma esplicita e verifica nuovamente lo snapshot prima dell'applicazione; se la sorgente è
|
||||||
|
cambiata, invalida la conferma. Ogni applicazione è atomica e fail-closed: errori, timeout o
|
||||||
|
capability non disponibili non producono aggiornamenti parziali.
|
||||||
|
|
||||||
|
PK e FK devono essere visibili sulle Catalog Column senza duplicare le stringhe denormalizzate di
|
||||||
|
ThothAI. La posizione nella primary key è un fatto osservato della colonna; membership e conteggio
|
||||||
|
FK sono proiezioni derivate dalle Catalog Relationship e dalle loro coppie ordinate, aggiornate
|
||||||
|
nella stessa transazione di riconciliazione.
|
||||||
|
|
||||||
|
Ogni scope registra la versione della Database Binding osservata e l'istante dell'ultima
|
||||||
|
sincronizzazione. Una modifica della binding conserva il catalogo precedente ma lo marca stale;
|
||||||
|
solo un `Synchronize all` riuscito rende nuovamente corrente l'intero schema.
|
||||||
|
|
||||||
### Step 7: generazione AI dei metadati
|
### Step 7: generazione AI dei metadati
|
||||||
|
|
||||||
Generazione di descrizioni e altri campi equivalenti alle annotations, editing umano e gestione
|
Generated Description è una proposta distinta e modificabile: un revisore può correggerla prima
|
||||||
esplicita di errori o output non validi. Approvazione/versioning saranno decisi in questo step.
|
di consolidarla esplicitamente come Description. Lo slice AI dovrà decidere e implementare anche
|
||||||
|
alias semantici, descrizioni dei valori, sinonimi e concetti per tabelle e colonne, oltre alla
|
||||||
|
gestione esplicita di errori e output non validi. La generazione AI e l'azione di consolidamento non
|
||||||
|
appartengono allo slice di introspezione dello schema.
|
||||||
|
|
||||||
### Step 8: migrazione PSD
|
### Step 8: migrazione PSD
|
||||||
|
|
||||||
@@ -427,11 +615,24 @@ ricevono import legacy.
|
|||||||
Rimuovere la dipendenza da `annotations.yaml` soltanto dopo avere un contratto equivalente,
|
Rimuovere la dipendenza da `annotations.yaml` soltanto dopo avere un contratto equivalente,
|
||||||
test di rendering/search/Qdrant e una policy di disponibilità. Le sessioni di test esistenti
|
test di rendering/search/Qdrant e una policy di disponibilità. Le sessioni di test esistenti
|
||||||
possono essere eliminate, ma le nuove sessioni non devono osservare aggiornamenti parziali.
|
possono essere eliminate, ma le nuove sessioni non devono osservare aggiornamenti parziali.
|
||||||
|
Questo cutover è esplicitamente rinviato fino al completamento del database dei metadati. Il primo
|
||||||
|
gate successivo obbligatorio sarà valutare l'integrazione del Catalog Schema Snapshot con il
|
||||||
|
workflow core e lo schema-linking corrente; il rinvio non autorizza a dimenticare o assorbire
|
||||||
|
implicitamente il lavoro in altri slice.
|
||||||
|
|
||||||
### Step 10: operazioni e accettazione
|
### Step 10: operazioni e accettazione
|
||||||
|
|
||||||
Backup/restore reale, diagnostica, metriche, audit, permessi definitivi, hardening degli export e
|
Backup/restore reale, diagnostica, metriche, permessi definitivi, hardening degli export e
|
||||||
test di failure isolation fra catalogo e workflow.
|
test di failure isolation fra catalogo e workflow. I Catalog Sync Run hanno un solo job attivo per
|
||||||
|
Workspace Database, sono concorrenti fra database diversi e usano un lock persistente. Un pannello
|
||||||
|
operativo non modale rimane visibile durante la navigazione del database, mostra fasi, contatori,
|
||||||
|
tempo trascorso e log sanitizzato via SSE con polling di fallback, e offre Confirm, Cancel e Retry
|
||||||
|
quando consentiti. Un restart marca `interrupted` i run rimasti attivi; il retry crea un nuovo run.
|
||||||
|
Le modifiche ai metadati restano consentite durante la scansione e sono preservate dall'applicazione.
|
||||||
|
Il worker gira inizialmente nello stesso servizio Fastify ma dietro un'interfaccia estraibile, con
|
||||||
|
coda, lease e heartbeat persistiti nel catalog-db. I riepiloghi dei run non scadono; gli eventi
|
||||||
|
dettagliati sono conservati per 30 giorni, mentre snapshot e diff completi vengono eliminati dopo
|
||||||
|
la conclusione lasciando conteggi, decisioni e una sintesi sanitizzata dell'esito.
|
||||||
|
|
||||||
## Verifiche del core da conservare per il cutover
|
## Verifiche del core da conservare per il cutover
|
||||||
|
|
||||||
@@ -476,15 +677,11 @@ Definiscono gli effetti semantici e le guardie da mantenere o sostituire consape
|
|||||||
|
|
||||||
Le seguenti scelte non appartengono allo step 1:
|
Le seguenti scelte non appartengono allo step 1:
|
||||||
|
|
||||||
- framework del servizio catalogo e libreria di accesso PostgreSQL;
|
- lifecycle dei riferimenti ai segreti durante sostituzione e cancellazione;
|
||||||
- collocazione e protezione delle credenziali dei Workspace Database;
|
- criteri per aggiungere dialetti successivi a PostgreSQL;
|
||||||
- supporto iniziale di dialetti diversi da PostgreSQL;
|
- criteri per un'eventuale estensione futura a più schemi per database;
|
||||||
- uno o più schema namespace per database;
|
- lifecycle e gestione amministrativa delle future Logical Relationship;
|
||||||
- policy di reconciliation per rename e delete;
|
- alias semantici, descrizioni dei valori, sinonimi e concetti prodotti o assistiti dall'AI;
|
||||||
- modello delle foreign key composite;
|
|
||||||
- distinzione persistente fra relationship fisiche e logiche;
|
|
||||||
- lifecycle draft/review/approval dell'output AI;
|
|
||||||
- versionamento, audit e rollback;
|
|
||||||
- formato e momento del cutover dal file al database interno;
|
- formato e momento del cutover dal file al database interno;
|
||||||
- permission definitiva separata da `workspace.manage`.
|
- permission definitiva separata da `workspace.manage`.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,265 @@
|
|||||||
|
import { apiFetch, assertSameOriginRequestUrl, BASE } from "./client";
|
||||||
|
import { joinBackendPath } from "./runtime-config";
|
||||||
|
|
||||||
|
export type DatabaseTransport = "postgres_direct" | "rest_api" | "ssh_tunnel";
|
||||||
|
export type ConnectionStatus = "untested" | "reachable" | "failed";
|
||||||
|
export type CatalogSecretName =
|
||||||
|
| "password"
|
||||||
|
| "apiKey"
|
||||||
|
| "sshPrivateKey"
|
||||||
|
| "sshPrivateKeyPassphrase"
|
||||||
|
| "sshKnownHosts"
|
||||||
|
| "tlsCa";
|
||||||
|
|
||||||
|
export interface DatabaseBinding {
|
||||||
|
transport: DatabaseTransport;
|
||||||
|
host?: string;
|
||||||
|
port?: number;
|
||||||
|
username?: string;
|
||||||
|
baseUrl?: string;
|
||||||
|
restPath?: string;
|
||||||
|
restAuth?: "none" | "bearer" | "x-api-key";
|
||||||
|
tlsServername?: string;
|
||||||
|
sshHost?: string;
|
||||||
|
sshPort?: number;
|
||||||
|
sshUsername?: string;
|
||||||
|
sshTargetHost?: string;
|
||||||
|
sshTargetPort?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogDatabase {
|
||||||
|
id?: string;
|
||||||
|
workspaceId: string;
|
||||||
|
workspaceName: string;
|
||||||
|
workspaceDescription?: string;
|
||||||
|
workspaceAvailable: boolean;
|
||||||
|
configured: boolean;
|
||||||
|
engine: "postgres";
|
||||||
|
databaseName: string;
|
||||||
|
schema: string;
|
||||||
|
version: number;
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
binding: DatabaseBinding;
|
||||||
|
connectionStatus: ConnectionStatus;
|
||||||
|
testedVersion?: number;
|
||||||
|
lastTestedAt?: string;
|
||||||
|
lastErrorCode?: string;
|
||||||
|
lastErrorMessage?: string;
|
||||||
|
schemaSyncedVersion?: number;
|
||||||
|
schemaSyncedAt?: string;
|
||||||
|
activeSyncRun?: CatalogSyncRun;
|
||||||
|
secrets: Record<CatalogSecretName, boolean>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DatabaseConfiguration {
|
||||||
|
workspaceId: string;
|
||||||
|
engine: "postgres";
|
||||||
|
databaseName: string;
|
||||||
|
schema: string;
|
||||||
|
binding: DatabaseBinding;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogTable {
|
||||||
|
id: string;
|
||||||
|
databaseId: string;
|
||||||
|
name: string;
|
||||||
|
sourceComment: string | null;
|
||||||
|
description: string | null;
|
||||||
|
generatedDescription: string | null;
|
||||||
|
lastSyncedDatabaseVersion?: number | null;
|
||||||
|
lastSyncedAt?: string | null;
|
||||||
|
version: number;
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogColumn {
|
||||||
|
id: string;
|
||||||
|
tableId: string;
|
||||||
|
name: string;
|
||||||
|
ordinalPosition: number;
|
||||||
|
dataType: string;
|
||||||
|
isNullable: boolean;
|
||||||
|
defaultExpression: string | null;
|
||||||
|
primaryKeyPosition: number | null;
|
||||||
|
isPrimaryKey: boolean;
|
||||||
|
isForeignKey: boolean;
|
||||||
|
foreignKeyCount: number;
|
||||||
|
sourceComment: string | null;
|
||||||
|
description: string | null;
|
||||||
|
generatedDescription: string | null;
|
||||||
|
lastSyncedDatabaseVersion: number | null;
|
||||||
|
lastSyncedAt: string | null;
|
||||||
|
version: number;
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogRelationshipColumn {
|
||||||
|
position: number;
|
||||||
|
sourceColumnId: string;
|
||||||
|
sourceColumnName: string;
|
||||||
|
targetColumnId: string;
|
||||||
|
targetColumnName: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogRelationship {
|
||||||
|
id: string;
|
||||||
|
databaseId: string;
|
||||||
|
constraintName: string;
|
||||||
|
sourceTableId: string;
|
||||||
|
sourceTableName: string;
|
||||||
|
targetTableId: string;
|
||||||
|
targetTableName: string;
|
||||||
|
updateRule: string;
|
||||||
|
deleteRule: string;
|
||||||
|
deferrable: boolean;
|
||||||
|
initiallyDeferred: boolean;
|
||||||
|
columns: CatalogRelationshipColumn[];
|
||||||
|
lastSyncedDatabaseVersion: number | null;
|
||||||
|
lastSyncedAt: string | null;
|
||||||
|
createdAt: string;
|
||||||
|
updatedAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type CatalogSyncScope = "tables" | "columns" | "relationships" | "all";
|
||||||
|
export type CatalogSyncState = "queued" | "running" | "awaiting_confirmation" | "applying"
|
||||||
|
| "succeeded" | "failed" | "cancelled" | "interrupted";
|
||||||
|
export type CatalogSyncPhase = "queued" | "connecting" | "scanning_tables" | "scanning_columns"
|
||||||
|
| "scanning_relationships" | "planning" | "awaiting_confirmation" | "applying" | "completed";
|
||||||
|
|
||||||
|
export interface CatalogSchemaDiff {
|
||||||
|
deletedTables: string[];
|
||||||
|
deletedColumns: Array<{ tableName: string; columnName: string }>;
|
||||||
|
deletedRelationships: Array<{ sourceTableName: string; constraintName: string }>;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogSyncRun {
|
||||||
|
id: string;
|
||||||
|
databaseId: string;
|
||||||
|
scope: CatalogSyncScope;
|
||||||
|
tableIds: string[];
|
||||||
|
state: CatalogSyncState;
|
||||||
|
phase: CatalogSyncPhase;
|
||||||
|
requestedDatabaseVersion: number;
|
||||||
|
plannedDiff: CatalogSchemaDiff | null;
|
||||||
|
confirmationToken: string | null;
|
||||||
|
counts: { tables?: number; columns?: number; relationships?: number; created?: number; updated?: number; deleted?: number };
|
||||||
|
errorCode: string | null;
|
||||||
|
errorMessage: string | null;
|
||||||
|
cancelRequested: boolean;
|
||||||
|
createdAt: string;
|
||||||
|
startedAt: string | null;
|
||||||
|
updatedAt: string;
|
||||||
|
finishedAt: string | null;
|
||||||
|
heartbeatAt: string | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface CatalogSyncEvent {
|
||||||
|
id: number;
|
||||||
|
runId: string;
|
||||||
|
sequence: number;
|
||||||
|
level: "info" | "warning" | "error";
|
||||||
|
eventType: string;
|
||||||
|
message: string;
|
||||||
|
data: Record<string, unknown>;
|
||||||
|
createdAt: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export const listCatalogDatabases = () => apiFetch<CatalogDatabase[]>("/catalog/databases");
|
||||||
|
|
||||||
|
export const createCatalogDatabase = (input: DatabaseConfiguration) =>
|
||||||
|
apiFetch<CatalogDatabase>("/catalog/databases", { method: "POST", body: JSON.stringify(input) });
|
||||||
|
|
||||||
|
export const updateCatalogDatabase = (id: string, version: number, input: DatabaseConfiguration) =>
|
||||||
|
apiFetch<CatalogDatabase>(`/catalog/databases/${encodeURIComponent(id)}`, {
|
||||||
|
method: "PATCH",
|
||||||
|
body: JSON.stringify({ ...input, version }),
|
||||||
|
});
|
||||||
|
|
||||||
|
export const replaceCatalogDatabaseSecrets = (
|
||||||
|
id: string,
|
||||||
|
version: number,
|
||||||
|
values: Partial<Record<CatalogSecretName, string>>,
|
||||||
|
) => apiFetch<CatalogDatabase>(`/catalog/databases/${encodeURIComponent(id)}/secrets`, {
|
||||||
|
method: "PUT",
|
||||||
|
body: JSON.stringify({ version, values }),
|
||||||
|
});
|
||||||
|
|
||||||
|
export const testCatalogDatabase = (id: string, version: number) =>
|
||||||
|
apiFetch<CatalogDatabase>(`/catalog/databases/${encodeURIComponent(id)}/test`, {
|
||||||
|
method: "POST",
|
||||||
|
body: JSON.stringify({ version }),
|
||||||
|
});
|
||||||
|
|
||||||
|
export const deleteCatalogDatabase = (id: string, version: number) =>
|
||||||
|
apiFetch<void>(`/catalog/databases/${encodeURIComponent(id)}?version=${version}`, { method: "DELETE" });
|
||||||
|
|
||||||
|
export const listCatalogTables = (databaseId: string) =>
|
||||||
|
apiFetch<CatalogTable[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/tables`);
|
||||||
|
|
||||||
|
export const updateCatalogTableMetadata = (
|
||||||
|
databaseId: string,
|
||||||
|
tableId: string,
|
||||||
|
version: number,
|
||||||
|
description: string | null,
|
||||||
|
generatedDescription: string | null,
|
||||||
|
) => apiFetch<CatalogTable>(
|
||||||
|
`/catalog/databases/${encodeURIComponent(databaseId)}/tables/${encodeURIComponent(tableId)}`,
|
||||||
|
{ method: "PATCH", body: JSON.stringify({ version, description, generatedDescription }) },
|
||||||
|
);
|
||||||
|
|
||||||
|
export const listCatalogColumns = (databaseId: string, tableId: string) =>
|
||||||
|
apiFetch<CatalogColumn[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/tables/${encodeURIComponent(tableId)}/columns`);
|
||||||
|
|
||||||
|
export const updateCatalogColumnMetadata = (
|
||||||
|
databaseId: string,
|
||||||
|
tableId: string,
|
||||||
|
columnId: string,
|
||||||
|
version: number,
|
||||||
|
description: string | null,
|
||||||
|
generatedDescription: string | null,
|
||||||
|
) => apiFetch<CatalogColumn>(
|
||||||
|
`/catalog/databases/${encodeURIComponent(databaseId)}/tables/${encodeURIComponent(tableId)}/columns/${encodeURIComponent(columnId)}`,
|
||||||
|
{ method: "PATCH", body: JSON.stringify({ version, description, generatedDescription }) },
|
||||||
|
);
|
||||||
|
|
||||||
|
export const listCatalogRelationships = (databaseId: string) =>
|
||||||
|
apiFetch<CatalogRelationship[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/relationships`);
|
||||||
|
|
||||||
|
export const startCatalogSync = (
|
||||||
|
databaseId: string,
|
||||||
|
version: number,
|
||||||
|
scope: CatalogSyncScope,
|
||||||
|
tableIds: string[] = [],
|
||||||
|
) => apiFetch<CatalogSyncRun>(`/catalog/databases/${encodeURIComponent(databaseId)}/sync-runs`, {
|
||||||
|
method: "POST",
|
||||||
|
body: JSON.stringify({ version, scope, tableIds }),
|
||||||
|
});
|
||||||
|
|
||||||
|
export const listCatalogSyncRuns = (databaseId: string) =>
|
||||||
|
apiFetch<CatalogSyncRun[]>(`/catalog/databases/${encodeURIComponent(databaseId)}/sync-runs`);
|
||||||
|
|
||||||
|
export const getCatalogSyncRun = (runId: string) =>
|
||||||
|
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}`);
|
||||||
|
|
||||||
|
export const listCatalogSyncEvents = (runId: string, after = 0) =>
|
||||||
|
apiFetch<CatalogSyncEvent[]>(`/catalog/sync-runs/${encodeURIComponent(runId)}/events-list?after=${after}`);
|
||||||
|
|
||||||
|
export const confirmCatalogSync = (runId: string, confirmationToken: string) =>
|
||||||
|
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}/confirm`, {
|
||||||
|
method: "POST", body: JSON.stringify({ confirmationToken }),
|
||||||
|
});
|
||||||
|
|
||||||
|
export const cancelCatalogSync = (runId: string) =>
|
||||||
|
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}/cancel`, { method: "POST" });
|
||||||
|
|
||||||
|
export const retryCatalogSync = (runId: string) =>
|
||||||
|
apiFetch<CatalogSyncRun>(`/catalog/sync-runs/${encodeURIComponent(runId)}/retry`, { method: "POST" });
|
||||||
|
|
||||||
|
export function catalogSyncEventsUrl(runId: string, after = 0): string {
|
||||||
|
const url = joinBackendPath(BASE, `/catalog/sync-runs/${encodeURIComponent(runId)}/events?after=${after}`);
|
||||||
|
assertSameOriginRequestUrl(url);
|
||||||
|
return url;
|
||||||
|
}
|
||||||
@@ -14,6 +14,10 @@ const safeErrorCodes = new Set([
|
|||||||
"git_unavailable", "git_auth_failed", "git_non_fast_forward", "connector_unavailable",
|
"git_unavailable", "git_auth_failed", "git_non_fast_forward", "connector_unavailable",
|
||||||
"semantic_index_incompatible", "pi_management_forbidden", "pi_management_unavailable",
|
"semantic_index_incompatible", "pi_management_forbidden", "pi_management_unavailable",
|
||||||
"pi_management_invalid_config", "pi_management_write_failed",
|
"pi_management_invalid_config", "pi_management_write_failed",
|
||||||
|
"catalog_unavailable", "database_conflict", "database_invalid", "database_not_found",
|
||||||
|
"database_stale", "database_operation_failed",
|
||||||
|
"schema_sync_conflict", "schema_introspection_failed", "schema_request_invalid",
|
||||||
|
"schema_operation_failed", "sync_run_not_found", "table_stale", "column_stale",
|
||||||
]);
|
]);
|
||||||
|
|
||||||
type SafeErrorPayload = {
|
type SafeErrorPayload = {
|
||||||
@@ -46,6 +50,19 @@ const localCodeMessages: Record<string, string> = {
|
|||||||
pi_management_unavailable: "Pi management is unavailable.",
|
pi_management_unavailable: "Pi management is unavailable.",
|
||||||
pi_management_invalid_config: "The Pi configuration is invalid.",
|
pi_management_invalid_config: "The Pi configuration is invalid.",
|
||||||
pi_management_write_failed: "The Pi configuration could not be saved.",
|
pi_management_write_failed: "The Pi configuration could not be saved.",
|
||||||
|
catalog_unavailable: "The database catalog is unavailable.",
|
||||||
|
database_conflict: "This workspace already has a database configuration.",
|
||||||
|
database_invalid: "The database configuration is invalid.",
|
||||||
|
database_not_found: "The database configuration was not found.",
|
||||||
|
database_stale: "The database configuration changed. Reload and try again.",
|
||||||
|
database_operation_failed: "The database operation failed.",
|
||||||
|
schema_sync_conflict: "A schema synchronization is already active or no longer current.",
|
||||||
|
schema_introspection_failed: "The database schema could not be read safely.",
|
||||||
|
schema_request_invalid: "The schema request is invalid.",
|
||||||
|
schema_operation_failed: "The schema operation failed.",
|
||||||
|
sync_run_not_found: "The synchronization run was not found.",
|
||||||
|
table_stale: "Table metadata changed. Reload and try again.",
|
||||||
|
column_stale: "Column metadata changed. Reload and try again.",
|
||||||
};
|
};
|
||||||
|
|
||||||
const localStatusMessages: Record<number, string> = {
|
const localStatusMessages: Record<number, string> = {
|
||||||
|
|||||||
@@ -296,3 +296,52 @@
|
|||||||
0%, 100% { transform: scale(1); box-shadow: 0 0 0 0 oklch(var(--primary) / 0.4); }
|
0%, 100% { transform: scale(1); box-shadow: 0 0 0 0 oklch(var(--primary) / 0.4); }
|
||||||
50% { transform: scale(1.15); box-shadow: 0 0 0 5px oklch(var(--primary) / 0); }
|
50% { transform: scale(1.15); box-shadow: 0 0 0 5px oklch(var(--primary) / 0); }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@layer components {
|
||||||
|
.thot-database-grid {
|
||||||
|
--ag-font-family: var(--font-sans);
|
||||||
|
--ag-font-size: 0.8125rem;
|
||||||
|
--ag-background-color: oklch(var(--card));
|
||||||
|
--ag-foreground-color: oklch(var(--foreground));
|
||||||
|
--ag-header-background-color: oklch(var(--muted) / 0.72);
|
||||||
|
--ag-header-foreground-color: oklch(var(--foreground));
|
||||||
|
--ag-border-color: oklch(var(--border));
|
||||||
|
--ag-row-border-color: oklch(var(--border) / 0.72);
|
||||||
|
--ag-odd-row-background-color: oklch(var(--muted) / 0.18);
|
||||||
|
--ag-row-hover-color: oklch(var(--muted) / 0.55);
|
||||||
|
--ag-selected-row-background-color: oklch(var(--primary) / 0.07);
|
||||||
|
--ag-input-focus-border-color: oklch(var(--primary) / 0.6);
|
||||||
|
--ag-range-selection-border-color: oklch(var(--primary) / 0.6);
|
||||||
|
--ag-header-column-separator-color: oklch(var(--border));
|
||||||
|
--ag-header-column-separator-display: block;
|
||||||
|
--ag-wrapper-border-radius: 0;
|
||||||
|
--ag-cell-horizontal-padding: 12px;
|
||||||
|
}
|
||||||
|
.thot-database-grid .ag-root-wrapper {
|
||||||
|
border: 0;
|
||||||
|
}
|
||||||
|
.thot-database-grid .ag-cell {
|
||||||
|
display: flex;
|
||||||
|
align-items: center;
|
||||||
|
}
|
||||||
|
.thot-database-grid .ag-header-cell-label {
|
||||||
|
font-weight: 700;
|
||||||
|
}
|
||||||
|
.thot-database-grid .ag-cell-focus:not(.ag-cell-range-selected) {
|
||||||
|
outline: 2px solid oklch(var(--ring) / 0.45);
|
||||||
|
outline-offset: -2px;
|
||||||
|
}
|
||||||
|
.thot-database-grid .ag-cell.thot-database-actions-cell {
|
||||||
|
padding-inline: 2px;
|
||||||
|
}
|
||||||
|
.thot-database-grid .ag-cell.thot-database-actions-cell.ag-cell-focus:not(.ag-cell-range-selected) {
|
||||||
|
outline: none;
|
||||||
|
}
|
||||||
|
.thot-database-grid .ag-cell.thot-database-status-cell {
|
||||||
|
padding-inline: 4px;
|
||||||
|
}
|
||||||
|
.thot-database-grid .ag-pinned-right-header,
|
||||||
|
.thot-database-grid .ag-pinned-right-cols-container {
|
||||||
|
box-shadow: -1px 0 0 oklch(var(--border));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ beforeEach(() => {
|
|||||||
issuer: "test",
|
issuer: "test",
|
||||||
subject: "test",
|
subject: "test",
|
||||||
roles: ["admin"],
|
roles: ["admin"],
|
||||||
permissions: ["session.use", "workspace.manage", "workspace.secrets.manage", "pi.manage"],
|
permissions: ["session.use", "workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage"],
|
||||||
isAdmin: true,
|
isAdmin: true,
|
||||||
csrfToken: null,
|
csrfToken: null,
|
||||||
session: null,
|
session: null,
|
||||||
@@ -52,7 +52,7 @@ beforeEach(() => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
test("opens the blank database management surface and returns to the core", async () => {
|
test("opens the database management surface and returns to the core", async () => {
|
||||||
renderShell();
|
renderShell();
|
||||||
|
|
||||||
const composer = screen.getByRole("textbox", { name: /new question/i });
|
const composer = screen.getByRole("textbox", { name: /new question/i });
|
||||||
@@ -122,7 +122,7 @@ test("keeps a live core session connected and returns when that session is opene
|
|||||||
expect(FakeEventSource.instances).toHaveLength(1);
|
expect(FakeEventSource.instances).toHaveLength(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
test("hides all management entries from non-admin users", () => {
|
test("keeps read-safe workspace access but hides privileged management entries", () => {
|
||||||
clearAuthState();
|
clearAuthState();
|
||||||
setAuthState({
|
setAuthState({
|
||||||
issuer: "test",
|
issuer: "test",
|
||||||
@@ -136,7 +136,52 @@ test("hides all management entries from non-admin users", () => {
|
|||||||
|
|
||||||
renderShell();
|
renderShell();
|
||||||
|
|
||||||
expect(screen.queryByRole("button", { name: "Workspace management" })).not.toBeInTheDocument();
|
expect(screen.getByRole("button", { name: "Workspace management" })).toBeInTheDocument();
|
||||||
expect(screen.queryByRole("button", { name: "Database management" })).not.toBeInTheDocument();
|
expect(screen.queryByRole("button", { name: "Database management" })).not.toBeInTheDocument();
|
||||||
expect(screen.queryByRole("button", { name: "Pi management" })).not.toBeInTheDocument();
|
expect(screen.queryByRole("button", { name: "Pi management" })).not.toBeInTheDocument();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test("does not leave database management without confirming a dirty form", async () => {
|
||||||
|
server.use(http.get("/api/catalog/databases", () => HttpResponse.json([{
|
||||||
|
id: "11111111-1111-4111-8111-111111111111",
|
||||||
|
workspaceId: "psd-clinical",
|
||||||
|
workspaceName: "Policlinico San Donato",
|
||||||
|
workspaceAvailable: true,
|
||||||
|
configured: true,
|
||||||
|
engine: "postgres",
|
||||||
|
databaseName: "warehouse",
|
||||||
|
schema: "datawarehouse",
|
||||||
|
version: 3,
|
||||||
|
createdAt: "2026-08-27T08:00:00Z",
|
||||||
|
updatedAt: "2026-08-27T09:00:00Z",
|
||||||
|
binding: { transport: "postgres_direct", host: "db.internal", port: 5432, username: "reader" },
|
||||||
|
connectionStatus: "untested",
|
||||||
|
secrets: {
|
||||||
|
password: false,
|
||||||
|
apiKey: false,
|
||||||
|
sshPrivateKey: false,
|
||||||
|
sshPrivateKeyPassphrase: false,
|
||||||
|
sshKnownHosts: false,
|
||||||
|
tlsCa: false,
|
||||||
|
},
|
||||||
|
}])));
|
||||||
|
const confirm = vi.spyOn(window, "confirm").mockReturnValue(false);
|
||||||
|
renderShell();
|
||||||
|
|
||||||
|
await userEvent.click(screen.getByRole("button", { name: "Database management" }));
|
||||||
|
await userEvent.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||||
|
const schema = screen.getByLabelText("Schema");
|
||||||
|
await userEvent.clear(schema);
|
||||||
|
await userEvent.type(schema, "reporting");
|
||||||
|
|
||||||
|
await userEvent.click(screen.getByRole("button", { name: "New session" }));
|
||||||
|
|
||||||
|
expect(confirm).toHaveBeenCalledWith("Discard unsaved database changes and leave database management?");
|
||||||
|
expect(screen.getByRole("main", { name: "Database management" })).toBeVisible();
|
||||||
|
expect(schema).toHaveValue("reporting");
|
||||||
|
|
||||||
|
confirm.mockReturnValue(true);
|
||||||
|
await userEvent.click(screen.getByRole("button", { name: "New session" }));
|
||||||
|
await waitFor(() => expect(screen.queryByRole("main", { name: "Database management" })).not.toBeInTheDocument());
|
||||||
|
confirm.mockRestore();
|
||||||
|
});
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ import {
|
|||||||
import type { SessionScope, SessionSummary } from "../api/types";
|
import type { SessionScope, SessionSummary } from "../api/types";
|
||||||
import { useAuthGeneration, useAuthUser } from "../auth/authState";
|
import { useAuthGeneration, useAuthUser } from "../auth/authState";
|
||||||
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||||
import { useEffect, useMemo, useRef, useState } from "react";
|
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
|
||||||
import type { CSSProperties } from "react";
|
import type { CSSProperties } from "react";
|
||||||
import { captureAuthOperation, isAuthOperationCurrent, StaleAuthOperationError, type AuthOperationGuard } from "../auth/authOperation";
|
import { captureAuthOperation, isAuthOperationCurrent, StaleAuthOperationError, type AuthOperationGuard } from "../auth/authOperation";
|
||||||
|
|
||||||
@@ -104,8 +104,8 @@ export function AppShell({ canLogout }: AppShellProps) {
|
|||||||
const canReadAllSessions = permissions.includes("session.read_all");
|
const canReadAllSessions = permissions.includes("session.read_all");
|
||||||
const canManageWorkspace = permissions.includes("workspace.manage");
|
const canManageWorkspace = permissions.includes("workspace.manage");
|
||||||
const canManageWorkspaceSecrets = permissions.includes("workspace.secrets.manage");
|
const canManageWorkspaceSecrets = permissions.includes("workspace.secrets.manage");
|
||||||
|
const canManageDatabase = permissions.includes("database.manage");
|
||||||
const canManagePi = permissions.includes("pi.manage");
|
const canManagePi = permissions.includes("pi.manage");
|
||||||
const isAdmin = authenticatedUser?.isAdmin === true;
|
|
||||||
const authGeneration = useAuthGeneration();
|
const authGeneration = useAuthGeneration();
|
||||||
const { data: sessions = [] } = useQuery<SessionSummary[]>({
|
const { data: sessions = [] } = useQuery<SessionSummary[]>({
|
||||||
queryKey: ["sessions", sessionScope],
|
queryKey: ["sessions", sessionScope],
|
||||||
@@ -123,6 +123,10 @@ export function AppShell({ canLogout }: AppShellProps) {
|
|||||||
const queryClient = useQueryClient();
|
const queryClient = useQueryClient();
|
||||||
const [showActivity, setShowActivity] = useState(false);
|
const [showActivity, setShowActivity] = useState(false);
|
||||||
const [activeSurface, setActiveSurface] = useState<ActiveSurface>("core");
|
const [activeSurface, setActiveSurface] = useState<ActiveSurface>("core");
|
||||||
|
const databaseNavigationRef = useRef({ dirty: false, busy: false });
|
||||||
|
const updateDatabaseNavigationState = useCallback((state: { dirty: boolean; busy: boolean }) => {
|
||||||
|
databaseNavigationRef.current = state;
|
||||||
|
}, []);
|
||||||
const [workspaceManagerOpen, setWorkspaceManagerOpen] = useState(false);
|
const [workspaceManagerOpen, setWorkspaceManagerOpen] = useState(false);
|
||||||
const [piManagementOpen, setPiManagementOpen] = useState(false);
|
const [piManagementOpen, setPiManagementOpen] = useState(false);
|
||||||
const [activeOpen, setActiveOpen] = useState(true);
|
const [activeOpen, setActiveOpen] = useState(true);
|
||||||
@@ -197,7 +201,20 @@ export function AppShell({ canLogout }: AppShellProps) {
|
|||||||
setSelectedSessionIds(selected ? new Set(sessions.map((session) => session.id)) : new Set());
|
setSelectedSessionIds(selected ? new Set(sessions.map((session) => session.id)) : new Set());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function canLeaveDatabaseManagement(): boolean {
|
||||||
|
if (activeSurface !== "database-management") return true;
|
||||||
|
if (databaseNavigationRef.current.busy) {
|
||||||
|
toast.info("Wait for the database operation to finish before leaving this page");
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (databaseNavigationRef.current.dirty) {
|
||||||
|
return window.confirm("Discard unsaved database changes and leave database management?");
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
function openPanel(id: string) {
|
function openPanel(id: string) {
|
||||||
|
if (!canLeaveDatabaseManagement()) return;
|
||||||
const s = sessions.find((x) => x.id === id);
|
const s = sessions.find((x) => x.id === id);
|
||||||
if (!s) return;
|
if (!s) return;
|
||||||
setActiveSurface("core");
|
setActiveSurface("core");
|
||||||
@@ -207,7 +224,7 @@ export function AppShell({ canLogout }: AppShellProps) {
|
|||||||
// completed sessions keep the read-only documents panel (with its explicit Resume),
|
// completed sessions keep the read-only documents panel (with its explicit Resume),
|
||||||
// so a mere click never spawns a runtime.
|
// so a mere click never spawns a runtime.
|
||||||
if (s.active && s.status === "open" && !s.archived && !isForeignSession(s)) {
|
if (s.active && s.status === "open" && !s.archived && !isForeignSession(s)) {
|
||||||
void doResume(id);
|
void doResume(id, true);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
setPanelSession(s); setShowActivity(false);
|
setPanelSession(s); setShowActivity(false);
|
||||||
@@ -219,7 +236,8 @@ export function AppShell({ canLogout }: AppShellProps) {
|
|||||||
return next;
|
return next;
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
async function doResume(id: string) {
|
async function doResume(id: string, databaseExitApproved = false) {
|
||||||
|
if (!databaseExitApproved && !canLeaveDatabaseManagement()) return;
|
||||||
setActiveSurface("core");
|
setActiveSurface("core");
|
||||||
const guard = captureAuthOperation({ sessionId: id, disposalEpoch: operationEpochRef.current });
|
const guard = captureAuthOperation({ sessionId: id, disposalEpoch: operationEpochRef.current });
|
||||||
if (!guard) return;
|
if (!guard) return;
|
||||||
@@ -502,6 +520,7 @@ export function AppShell({ canLogout }: AppShellProps) {
|
|||||||
}, [lastSystemEvent]);
|
}, [lastSystemEvent]);
|
||||||
|
|
||||||
function startNewSession() {
|
function startNewSession() {
|
||||||
|
if (!canLeaveDatabaseManagement()) return;
|
||||||
setActiveSurface("core");
|
setActiveSurface("core");
|
||||||
invalidateResumeIntent();
|
invalidateResumeIntent();
|
||||||
newSessionOperationRef.current = null;
|
newSessionOperationRef.current = null;
|
||||||
@@ -566,6 +585,7 @@ export function AppShell({ canLogout }: AppShellProps) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async function signOut() {
|
async function signOut() {
|
||||||
|
if (!canLeaveDatabaseManagement()) return;
|
||||||
await logoutUser();
|
await logoutUser();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -619,7 +639,13 @@ export function AppShell({ canLogout }: AppShellProps) {
|
|||||||
|
|
||||||
{/* Conversation column */}
|
{/* Conversation column */}
|
||||||
<div data-testid="conversation-column" className="flex min-w-0 flex-1 flex-col">
|
<div data-testid="conversation-column" className="flex min-w-0 flex-1 flex-col">
|
||||||
{activeSurface === "database-management" && <DatabaseManagementPage />}
|
{activeSurface === "database-management" && (
|
||||||
|
<DatabaseManagementPage
|
||||||
|
canManage={canManageDatabase}
|
||||||
|
canManageSecrets={canManageWorkspaceSecrets}
|
||||||
|
onNavigationStateChange={updateDatabaseNavigationState}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
<div
|
<div
|
||||||
aria-hidden={activeSurface !== "core"}
|
aria-hidden={activeSurface !== "core"}
|
||||||
className={activeSurface === "core" ? "contents" : "hidden"}
|
className={activeSurface === "core" ? "contents" : "hidden"}
|
||||||
@@ -738,16 +764,19 @@ export function AppShell({ canLogout }: AppShellProps) {
|
|||||||
>
|
>
|
||||||
New session
|
New session
|
||||||
</Button>
|
</Button>
|
||||||
{isAdmin && (
|
|
||||||
<>
|
|
||||||
<Button
|
<Button
|
||||||
variant="outline"
|
variant="outline"
|
||||||
size="sm"
|
size="sm"
|
||||||
className="w-full"
|
className="w-full"
|
||||||
onClick={() => setWorkspaceManagerOpen(true)}
|
onClick={() => {
|
||||||
|
if (!canLeaveDatabaseManagement()) return;
|
||||||
|
setActiveSurface("core");
|
||||||
|
setWorkspaceManagerOpen(true);
|
||||||
|
}}
|
||||||
>
|
>
|
||||||
Workspace management
|
Workspace management
|
||||||
</Button>
|
</Button>
|
||||||
|
{canManageDatabase && (
|
||||||
<Button
|
<Button
|
||||||
variant="outline"
|
variant="outline"
|
||||||
size="sm"
|
size="sm"
|
||||||
@@ -757,14 +786,17 @@ export function AppShell({ canLogout }: AppShellProps) {
|
|||||||
>
|
>
|
||||||
Database management
|
Database management
|
||||||
</Button>
|
</Button>
|
||||||
</>
|
|
||||||
)}
|
)}
|
||||||
{isAdmin && canManagePi && (
|
{canManagePi && (
|
||||||
<Button
|
<Button
|
||||||
variant="outline"
|
variant="outline"
|
||||||
size="sm"
|
size="sm"
|
||||||
className="w-full"
|
className="w-full"
|
||||||
onClick={() => setPiManagementOpen(true)}
|
onClick={() => {
|
||||||
|
if (!canLeaveDatabaseManagement()) return;
|
||||||
|
setActiveSurface("core");
|
||||||
|
setPiManagementOpen(true);
|
||||||
|
}}
|
||||||
>
|
>
|
||||||
Pi management
|
Pi management
|
||||||
</Button>
|
</Button>
|
||||||
|
|||||||
@@ -0,0 +1,842 @@
|
|||||||
|
import { act, render, screen, waitFor, within } from "@testing-library/react";
|
||||||
|
import userEvent from "@testing-library/user-event";
|
||||||
|
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||||
|
import { http, HttpResponse } from "msw";
|
||||||
|
import { server } from "../test/msw";
|
||||||
|
import type { CatalogColumn, CatalogDatabase, CatalogSyncRun, CatalogTable } from "../api/catalog-databases";
|
||||||
|
import { DatabaseManagementPage } from "./DatabaseManagementPage";
|
||||||
|
|
||||||
|
const noSecrets = {
|
||||||
|
password: false,
|
||||||
|
apiKey: false,
|
||||||
|
sshPrivateKey: false,
|
||||||
|
sshPrivateKeyPassphrase: false,
|
||||||
|
sshKnownHosts: false,
|
||||||
|
tlsCa: false,
|
||||||
|
};
|
||||||
|
|
||||||
|
function makeDatabase(overrides: Partial<CatalogDatabase> = {}): CatalogDatabase {
|
||||||
|
return {
|
||||||
|
id: "11111111-1111-4111-8111-111111111111",
|
||||||
|
workspaceId: "psd-clinical",
|
||||||
|
workspaceName: "Policlinico San Donato",
|
||||||
|
workspaceAvailable: true,
|
||||||
|
configured: true,
|
||||||
|
engine: "postgres",
|
||||||
|
databaseName: "warehouse",
|
||||||
|
schema: "datawarehouse",
|
||||||
|
version: 3,
|
||||||
|
createdAt: "2026-08-27T08:00:00Z",
|
||||||
|
updatedAt: "2026-08-27T09:00:00Z",
|
||||||
|
binding: {
|
||||||
|
transport: "postgres_direct",
|
||||||
|
host: "db.internal",
|
||||||
|
port: 5432,
|
||||||
|
username: "reader",
|
||||||
|
},
|
||||||
|
connectionStatus: "untested",
|
||||||
|
secrets: { ...noSecrets },
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const unconfigured = makeDatabase({
|
||||||
|
id: undefined,
|
||||||
|
workspaceId: "lab",
|
||||||
|
workspaceName: "Research laboratory",
|
||||||
|
configured: false,
|
||||||
|
databaseName: "lab_warehouse",
|
||||||
|
schema: "analytics",
|
||||||
|
version: 0,
|
||||||
|
createdAt: "",
|
||||||
|
updatedAt: "",
|
||||||
|
binding: { transport: "postgres_direct", port: 5432 },
|
||||||
|
});
|
||||||
|
|
||||||
|
const orphan = makeDatabase({
|
||||||
|
id: "22222222-2222-4222-8222-222222222222",
|
||||||
|
workspaceId: "retired",
|
||||||
|
workspaceName: "Retired workspace",
|
||||||
|
workspaceAvailable: false,
|
||||||
|
});
|
||||||
|
|
||||||
|
function renderPage({
|
||||||
|
rows = [makeDatabase(), unconfigured, orphan],
|
||||||
|
canManage = true,
|
||||||
|
canManageSecrets = true,
|
||||||
|
onNavigationStateChange,
|
||||||
|
}: {
|
||||||
|
rows?: CatalogDatabase[] | (() => CatalogDatabase[]);
|
||||||
|
canManage?: boolean;
|
||||||
|
canManageSecrets?: boolean;
|
||||||
|
onNavigationStateChange?: (state: { dirty: boolean; busy: boolean }) => void;
|
||||||
|
} = {}) {
|
||||||
|
server.use(http.get("/api/catalog/databases", () => HttpResponse.json(
|
||||||
|
typeof rows === "function" ? rows() : rows,
|
||||||
|
)));
|
||||||
|
const client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
|
||||||
|
const view = render(
|
||||||
|
<QueryClientProvider client={client}>
|
||||||
|
<DatabaseManagementPage
|
||||||
|
canManage={canManage}
|
||||||
|
canManageSecrets={canManageSecrets}
|
||||||
|
onNavigationStateChange={onNavigationStateChange}
|
||||||
|
/>
|
||||||
|
</QueryClientProvider>,
|
||||||
|
);
|
||||||
|
return { ...view, client };
|
||||||
|
}
|
||||||
|
|
||||||
|
function makeSyncRun(
|
||||||
|
scope: CatalogSyncRun["scope"],
|
||||||
|
tableIds: string[] = [],
|
||||||
|
): CatalogSyncRun {
|
||||||
|
return {
|
||||||
|
id: `sync-${scope}`,
|
||||||
|
databaseId: "11111111-1111-4111-8111-111111111111",
|
||||||
|
scope,
|
||||||
|
tableIds,
|
||||||
|
state: "queued",
|
||||||
|
phase: "queued",
|
||||||
|
requestedDatabaseVersion: 3,
|
||||||
|
plannedDiff: null,
|
||||||
|
confirmationToken: null,
|
||||||
|
counts: {},
|
||||||
|
errorCode: null,
|
||||||
|
errorMessage: null,
|
||||||
|
cancelRequested: false,
|
||||||
|
createdAt: "2026-08-27T10:00:00Z",
|
||||||
|
startedAt: null,
|
||||||
|
updatedAt: "2026-08-27T10:00:00Z",
|
||||||
|
finishedAt: null,
|
||||||
|
heartbeatAt: null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function registerCompletedSyncRun(run: CatalogSyncRun) {
|
||||||
|
const completed: CatalogSyncRun = {
|
||||||
|
...run,
|
||||||
|
state: "succeeded",
|
||||||
|
phase: "completed",
|
||||||
|
startedAt: "2026-08-27T10:00:00Z",
|
||||||
|
finishedAt: "2026-08-27T10:00:01Z",
|
||||||
|
};
|
||||||
|
server.use(
|
||||||
|
http.get("/api/catalog/sync-runs/:runId", () => HttpResponse.json(completed)),
|
||||||
|
http.get("/api/catalog/sync-runs/:runId/events-list", () => HttpResponse.json([])),
|
||||||
|
http.get("/api/catalog/databases/:databaseId/sync-runs", () => HttpResponse.json([completed])),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const synchronizationScopes = [
|
||||||
|
{ scope: "tables", label: "Synchronize tables" },
|
||||||
|
{ scope: "columns", label: "Synchronize all columns" },
|
||||||
|
{ scope: "relationships", label: "Synchronize relationships" },
|
||||||
|
{ scope: "all", label: "Synchronize all" },
|
||||||
|
] as const;
|
||||||
|
|
||||||
|
test("starts with a full-width list and applies the row action matrix", async () => {
|
||||||
|
renderPage();
|
||||||
|
|
||||||
|
expect(await screen.findByRole("button", { name: "View Policlinico San Donato" })).toBeEnabled();
|
||||||
|
expect(screen.getByRole("button", { name: "Edit Policlinico San Donato" })).toBeEnabled();
|
||||||
|
expect(screen.getByRole("button", { name: "Delete Policlinico San Donato" })).toBeEnabled();
|
||||||
|
|
||||||
|
expect(screen.getByRole("button", { name: "View Research laboratory" })).toBeEnabled();
|
||||||
|
expect(screen.getByRole("button", { name: "Edit Research laboratory" })).toBeEnabled();
|
||||||
|
expect(screen.getByRole("button", { name: "Delete Research laboratory" })).toBeDisabled();
|
||||||
|
|
||||||
|
expect(screen.getByRole("button", { name: "View Retired workspace" })).toBeEnabled();
|
||||||
|
expect(screen.getByRole("button", { name: "Edit Retired workspace" })).toBeDisabled();
|
||||||
|
expect(screen.getByRole("button", { name: "Delete Retired workspace" })).toBeEnabled();
|
||||||
|
expect(screen.queryByRole("heading", { name: "Database details" })).not.toBeInTheDocument();
|
||||||
|
});
|
||||||
|
|
||||||
|
test.each(synchronizationScopes)(
|
||||||
|
"selected database Actions offers and starts $scope synchronization",
|
||||||
|
async ({ scope, label }) => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
let startBody: unknown;
|
||||||
|
const run = makeSyncRun(scope);
|
||||||
|
registerCompletedSyncRun(run);
|
||||||
|
server.use(http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
|
||||||
|
startBody = await request.json();
|
||||||
|
return HttpResponse.json(run, { status: 202 });
|
||||||
|
}));
|
||||||
|
renderPage({
|
||||||
|
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||||
|
});
|
||||||
|
|
||||||
|
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
|
||||||
|
await user.click(within(databaseRow).getByRole("checkbox"));
|
||||||
|
await user.click(screen.getByRole("button", { name: "Actions" }));
|
||||||
|
|
||||||
|
expect(await screen.findByRole("menuitem", { name: synchronizationScopes[0].label })).toBeEnabled();
|
||||||
|
for (const option of synchronizationScopes.slice(1)) {
|
||||||
|
expect(screen.getByRole("menuitem", { name: option.label })).toBeEnabled();
|
||||||
|
}
|
||||||
|
await user.click(screen.getByRole("menuitem", { name: label }));
|
||||||
|
|
||||||
|
await waitFor(() => expect(startBody).toEqual({ version: 3, scope, tableIds: [] }));
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
test("presents completed synchronization steps as success and skips unneeded confirmation", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
const run = {
|
||||||
|
...makeSyncRun("columns"),
|
||||||
|
counts: { tables: 163, columns: 2_275 },
|
||||||
|
};
|
||||||
|
registerCompletedSyncRun(run);
|
||||||
|
server.use(http.post("/api/catalog/databases/:databaseId/sync-runs", () => (
|
||||||
|
HttpResponse.json(run, { status: 202 })
|
||||||
|
)));
|
||||||
|
renderPage({
|
||||||
|
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||||
|
});
|
||||||
|
|
||||||
|
const databaseRow = await screen.findByRole("row", { name: /Policlinico San Donato/ });
|
||||||
|
await user.click(within(databaseRow).getByRole("checkbox"));
|
||||||
|
await user.click(screen.getByRole("button", { name: "Actions" }));
|
||||||
|
await user.click(await screen.findByRole("menuitem", { name: "Synchronize all columns" }));
|
||||||
|
|
||||||
|
expect(await screen.findByRole("heading", { name: "Succeeded" })).toBeVisible();
|
||||||
|
const steps = screen.getByRole("list", { name: "Synchronization steps" });
|
||||||
|
expect(steps).toHaveClass("grid-cols-2", "sm:grid-cols-3");
|
||||||
|
const connected = within(steps).getByRole("listitem", { name: "Connected, completed" });
|
||||||
|
expect(connected.querySelector("svg")).toHaveClass("text-[oklch(var(--success))]");
|
||||||
|
expect(within(steps).getByText("Tables read")).toBeVisible();
|
||||||
|
const confirmation = within(steps).getByRole("listitem", {
|
||||||
|
name: "Confirmation not required, not required",
|
||||||
|
});
|
||||||
|
expect(confirmation.querySelector("svg")).toHaveClass("text-muted-foreground");
|
||||||
|
const completed = within(steps).getByRole("listitem", { name: "Completed, completed" });
|
||||||
|
expect(completed.querySelector("svg")).toHaveClass("text-[oklch(var(--success))]");
|
||||||
|
expect(steps.querySelectorAll(".text-primary")).toHaveLength(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("database Overview exposes every synchronization scope", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
let startBody: unknown;
|
||||||
|
const run = makeSyncRun("relationships");
|
||||||
|
registerCompletedSyncRun(run);
|
||||||
|
server.use(http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
|
||||||
|
startBody = await request.json();
|
||||||
|
return HttpResponse.json(run, { status: 202 });
|
||||||
|
}));
|
||||||
|
renderPage({
|
||||||
|
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||||
|
});
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||||
|
await user.click(screen.getByRole("button", { name: "Synchronize database schema" }));
|
||||||
|
|
||||||
|
expect(await screen.findByRole("menuitem", { name: synchronizationScopes[0].label })).toBeEnabled();
|
||||||
|
for (const option of synchronizationScopes.slice(1)) {
|
||||||
|
expect(screen.getByRole("menuitem", { name: option.label })).toBeEnabled();
|
||||||
|
}
|
||||||
|
await user.click(screen.getByRole("menuitem", { name: "Synchronize relationships" }));
|
||||||
|
|
||||||
|
await waitFor(() => expect(startBody).toEqual({
|
||||||
|
version: 3,
|
||||||
|
scope: "relationships",
|
||||||
|
tableIds: [],
|
||||||
|
}));
|
||||||
|
});
|
||||||
|
|
||||||
|
test("replaces the list with View and returns focus to the originating action", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
renderPage({ rows: [makeDatabase()] });
|
||||||
|
const view = await screen.findByRole("button", { name: "View Policlinico San Donato" });
|
||||||
|
|
||||||
|
await user.click(view);
|
||||||
|
|
||||||
|
expect(screen.getByRole("heading", { name: "Database details" })).toBeVisible();
|
||||||
|
expect(screen.getByDisplayValue("db.internal")).toHaveAttribute("readonly");
|
||||||
|
expect(screen.getByRole("button", { name: "Test connection" })).toBeEnabled();
|
||||||
|
expect(screen.queryByRole("button", { name: "Save changes" })).not.toBeInTheDocument();
|
||||||
|
|
||||||
|
await user.click(screen.getByRole("button", { name: "Back to list" }));
|
||||||
|
|
||||||
|
await waitFor(() => expect(view).toHaveFocus());
|
||||||
|
expect(screen.getByRole("button", { name: "View Policlinico San Donato" })).toBeVisible();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("opens an unconfigured row as Edit while creating its first saved configuration", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
let postBody: unknown;
|
||||||
|
const saved = makeDatabase({
|
||||||
|
...unconfigured,
|
||||||
|
id: "33333333-3333-4333-8333-333333333333",
|
||||||
|
configured: true,
|
||||||
|
version: 1,
|
||||||
|
binding: { transport: "rest_api", baseUrl: "https://psd.example/api", restPath: "/health", restAuth: "x-api-key" },
|
||||||
|
});
|
||||||
|
let rows = [unconfigured];
|
||||||
|
server.use(
|
||||||
|
http.post("/api/catalog/databases", async ({ request }) => {
|
||||||
|
postBody = await request.json();
|
||||||
|
rows = [saved];
|
||||||
|
return HttpResponse.json(saved, { status: 201 });
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
renderPage({ rows: () => rows });
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "Edit Research laboratory" }));
|
||||||
|
|
||||||
|
expect(screen.getByRole("heading", { name: "Edit database" })).toBeVisible();
|
||||||
|
expect(screen.getByLabelText("Workspace")).toBeDisabled();
|
||||||
|
await user.selectOptions(screen.getByLabelText("Transport"), "rest_api");
|
||||||
|
await user.type(screen.getByLabelText("Base URL"), "https://psd.example/api");
|
||||||
|
expect(screen.getByLabelText("Diagnostic endpoint")).toHaveValue("/health");
|
||||||
|
expect(screen.getByLabelText("Diagnostic endpoint")).toHaveAttribute("readonly");
|
||||||
|
|
||||||
|
await user.click(screen.getByRole("button", { name: "Save database" }));
|
||||||
|
|
||||||
|
await waitFor(() => expect(postBody).toMatchObject({
|
||||||
|
workspaceId: "lab",
|
||||||
|
binding: expect.objectContaining({ transport: "rest_api", baseUrl: "https://psd.example/api" }),
|
||||||
|
}));
|
||||||
|
expect(await screen.findByRole("heading", { name: "Edit database" })).toBeVisible();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("global Add offers only unconfigured workspaces and lets the operator choose one", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
const second = makeDatabase({
|
||||||
|
...unconfigured,
|
||||||
|
workspaceId: "radiology",
|
||||||
|
workspaceName: "Radiology",
|
||||||
|
databaseName: "radiology_dwh",
|
||||||
|
});
|
||||||
|
renderPage({ rows: [makeDatabase(), unconfigured, second] });
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "Add database" }));
|
||||||
|
|
||||||
|
expect(screen.getByRole("heading", { name: "Add database" })).toBeVisible();
|
||||||
|
expect(screen.getByRole("button", { name: "Add database" })).toBeVisible();
|
||||||
|
const selector = screen.getByLabelText("Workspace");
|
||||||
|
expect(selector).toBeEnabled();
|
||||||
|
expect(screen.queryByRole("option", { name: "Policlinico San Donato" })).not.toBeInTheDocument();
|
||||||
|
await user.selectOptions(selector, "radiology");
|
||||||
|
expect(screen.getByDisplayValue("radiology_dwh")).toBeVisible();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("guards a dirty Edit, disables testing, and reports navigation state", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
const navigation = vi.fn();
|
||||||
|
const confirm = vi.spyOn(window, "confirm").mockReturnValue(false);
|
||||||
|
renderPage({ rows: [makeDatabase()], onNavigationStateChange: navigation });
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||||
|
const schema = screen.getByLabelText("Schema");
|
||||||
|
await user.clear(schema);
|
||||||
|
await user.type(schema, "reporting");
|
||||||
|
|
||||||
|
expect(screen.getByRole("button", { name: "Test connection" })).toBeDisabled();
|
||||||
|
await waitFor(() => expect(navigation).toHaveBeenLastCalledWith({ dirty: true, busy: false }));
|
||||||
|
await user.click(screen.getByRole("button", { name: "Back to list" }));
|
||||||
|
expect(confirm).toHaveBeenCalledWith("Discard unsaved database changes?");
|
||||||
|
expect(screen.getByRole("heading", { name: "Edit database" })).toBeVisible();
|
||||||
|
|
||||||
|
confirm.mockReturnValue(true);
|
||||||
|
await user.click(screen.getByRole("button", { name: "Back to list" }));
|
||||||
|
expect(await screen.findByRole("button", { name: "Edit Policlinico San Donato" })).toBeVisible();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("uses an in-page destructive form and returns the YAML workspace to Not configured", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
let row = makeDatabase();
|
||||||
|
let deleteCalled = false;
|
||||||
|
let deleteVersion: string | null = null;
|
||||||
|
server.use(
|
||||||
|
http.delete("/api/catalog/databases/:id", ({ request }) => {
|
||||||
|
deleteCalled = true;
|
||||||
|
deleteVersion = new URL(request.url).searchParams.get("version");
|
||||||
|
row = {
|
||||||
|
...row,
|
||||||
|
id: undefined,
|
||||||
|
configured: false,
|
||||||
|
version: 0,
|
||||||
|
updatedAt: "",
|
||||||
|
connectionStatus: "untested",
|
||||||
|
secrets: { ...noSecrets },
|
||||||
|
};
|
||||||
|
return new HttpResponse(null, { status: 204 });
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
renderPage({ rows: () => [row] });
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "Delete Policlinico San Donato" }));
|
||||||
|
expect(screen.getByRole("heading", { name: "Delete database" })).toBeVisible();
|
||||||
|
expect(screen.getByText("This removes the local database configuration.")).toBeVisible();
|
||||||
|
expect(screen.queryByRole("dialog")).not.toBeInTheDocument();
|
||||||
|
|
||||||
|
await user.click(screen.getByRole("button", { name: "Delete database" }));
|
||||||
|
|
||||||
|
await waitFor(() => expect(deleteCalled).toBe(true));
|
||||||
|
expect(deleteVersion).toBe("3");
|
||||||
|
expect(await screen.findByRole("button", { name: "Delete Policlinico San Donato" })).toBeDisabled();
|
||||||
|
expect(screen.getByText("Not configured")).toBeVisible();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("tests only the persisted version and updates the visible connection status", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
let testBody: unknown;
|
||||||
|
let row = makeDatabase();
|
||||||
|
server.use(http.post("/api/catalog/databases/:id/test", async ({ request }) => {
|
||||||
|
testBody = await request.json();
|
||||||
|
row = makeDatabase({ version: 4, connectionStatus: "reachable", testedVersion: 4 });
|
||||||
|
return HttpResponse.json(row);
|
||||||
|
}));
|
||||||
|
renderPage({ rows: () => [row] });
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||||
|
await user.click(screen.getByRole("button", { name: "Test connection" }));
|
||||||
|
|
||||||
|
await waitFor(() => expect(testBody).toEqual({ version: 3 }));
|
||||||
|
expect(within(screen.getByRole("region", { name: "Database details form" })).getByText("Reachable")).toBeVisible();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("retains a stale draft and requires an explicit reload", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
server.use(
|
||||||
|
http.patch("/api/catalog/databases/:id", () => HttpResponse.json({
|
||||||
|
code: "database_stale",
|
||||||
|
message: "The database changed",
|
||||||
|
}, { status: 409 })),
|
||||||
|
);
|
||||||
|
renderPage({ rows: [makeDatabase()] });
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||||
|
const schema = screen.getByLabelText("Schema");
|
||||||
|
await user.clear(schema);
|
||||||
|
await user.type(schema, "draft_schema");
|
||||||
|
await user.click(screen.getByRole("button", { name: "Save changes" }));
|
||||||
|
|
||||||
|
expect(await screen.findByText("A newer database configuration is available.")).toBeVisible();
|
||||||
|
expect(schema).toHaveValue("draft_schema");
|
||||||
|
expect(screen.getByRole("button", { name: "Save changes" })).toBeDisabled();
|
||||||
|
|
||||||
|
await user.click(screen.getByRole("button", { name: "Keep editing" }));
|
||||||
|
expect(screen.getByText("Reload the latest values before this configuration can be changed.")).toBeVisible();
|
||||||
|
expect(schema).toHaveValue("draft_schema");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("marks an open form stale when a background refetch advances the catalog version", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
const { client } = renderPage({ rows: [makeDatabase()] });
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||||
|
act(() => {
|
||||||
|
client.setQueryData(["catalog-databases"], [makeDatabase({ version: 4, schema: "server_schema" })]);
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(await screen.findByText("A newer database configuration is available.")).toBeVisible();
|
||||||
|
expect(screen.getByLabelText("Schema")).toHaveValue("datawarehouse");
|
||||||
|
expect(screen.getByRole("button", { name: "Save changes" })).toBeDisabled();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("does not submit credentials hidden by a transport or authentication change", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
let putCalled = false;
|
||||||
|
let patchBody: Record<string, unknown> | undefined;
|
||||||
|
let row = makeDatabase();
|
||||||
|
server.use(
|
||||||
|
http.patch("/api/catalog/databases/:id", async ({ request }) => {
|
||||||
|
patchBody = await request.json() as Record<string, unknown>;
|
||||||
|
row = makeDatabase({
|
||||||
|
version: 4,
|
||||||
|
binding: { transport: "rest_api", baseUrl: "https://psd.example/api", restPath: "/health", restAuth: "none" },
|
||||||
|
});
|
||||||
|
return HttpResponse.json(row);
|
||||||
|
}),
|
||||||
|
http.put("/api/catalog/databases/:id/secrets", () => {
|
||||||
|
putCalled = true;
|
||||||
|
return HttpResponse.json(row);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
renderPage({ rows: () => [row] });
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||||
|
await user.type(screen.getByLabelText("Password"), "must-not-be-sent");
|
||||||
|
await user.selectOptions(screen.getByLabelText("Transport"), "rest_api");
|
||||||
|
await user.type(screen.getByLabelText("Base URL"), "https://psd.example/api");
|
||||||
|
await user.selectOptions(screen.getByLabelText("Authentication"), "none");
|
||||||
|
await user.click(screen.getByRole("button", { name: "Save changes" }));
|
||||||
|
|
||||||
|
await waitFor(() => expect(patchBody).toBeDefined());
|
||||||
|
expect(putCalled).toBe(false);
|
||||||
|
expect(patchBody).toMatchObject({
|
||||||
|
version: 3,
|
||||||
|
binding: {
|
||||||
|
transport: "rest_api",
|
||||||
|
baseUrl: "https://psd.example/api",
|
||||||
|
restPath: "/health",
|
||||||
|
restAuth: "none",
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect((patchBody?.binding as Record<string, unknown>).host).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("preserves typed secrets and offers a retry when the configuration save is only partial", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
let putCalls = 0;
|
||||||
|
let row = makeDatabase();
|
||||||
|
server.use(
|
||||||
|
http.put("/api/catalog/databases/:id/secrets", async ({ request }) => {
|
||||||
|
putCalls += 1;
|
||||||
|
const body = await request.json();
|
||||||
|
expect(body).toEqual({ version: 3, values: { password: "transient-secret" } });
|
||||||
|
if (putCalls === 1) {
|
||||||
|
return HttpResponse.json({
|
||||||
|
code: "catalog_unavailable",
|
||||||
|
message: "Secret store unavailable",
|
||||||
|
}, { status: 503 });
|
||||||
|
}
|
||||||
|
row = makeDatabase({ version: 4, secrets: { ...noSecrets, password: true } });
|
||||||
|
return HttpResponse.json(row);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
renderPage({ rows: () => [row] });
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||||
|
await user.type(screen.getByLabelText("Password"), "transient-secret");
|
||||||
|
await user.click(screen.getByRole("button", { name: "Save changes" }));
|
||||||
|
|
||||||
|
expect(await screen.findByText("Database configuration saved; secret update could not be confirmed.")).toBeVisible();
|
||||||
|
expect(screen.getByLabelText("Password")).toHaveValue("transient-secret");
|
||||||
|
expect(screen.getByRole("button", { name: "Retry secrets" })).toBeEnabled();
|
||||||
|
|
||||||
|
await user.clear(screen.getByLabelText("Schema"));
|
||||||
|
await user.type(screen.getByLabelText("Schema"), "reporting_after_retry");
|
||||||
|
await user.click(screen.getByRole("button", { name: "Retry secrets" }));
|
||||||
|
|
||||||
|
await waitFor(() => expect(putCalls).toBe(2));
|
||||||
|
expect(screen.queryByText("Database configuration saved; secret update could not be confirmed.")).not.toBeInTheDocument();
|
||||||
|
expect(screen.getByLabelText("Schema")).toHaveValue("reporting_after_retry");
|
||||||
|
expect(screen.getByRole("button", { name: "Save changes" })).toBeEnabled();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("shows secret status without exposing or enabling values when permission is absent", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
renderPage({
|
||||||
|
rows: [makeDatabase({ secrets: { ...noSecrets, password: true } })],
|
||||||
|
canManageSecrets: false,
|
||||||
|
});
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||||
|
expect(screen.getByLabelText("Password")).toHaveValue("Configured");
|
||||||
|
expect(screen.getByLabelText("Password")).toHaveAttribute("readonly");
|
||||||
|
|
||||||
|
await user.click(screen.getByRole("button", { name: "Back to list" }));
|
||||||
|
await user.click(screen.getByRole("button", { name: "Edit Policlinico San Donato" }));
|
||||||
|
expect(await screen.findByRole("heading", { name: "Edit database" })).toBeVisible();
|
||||||
|
expect(screen.getByLabelText("Password")).toBeDisabled();
|
||||||
|
expect(screen.getByText(/requires the workspace\.secrets\.manage permission/i)).toBeVisible();
|
||||||
|
});
|
||||||
|
|
||||||
|
const patientsTable: CatalogTable = {
|
||||||
|
id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
|
||||||
|
databaseId: "11111111-1111-4111-8111-111111111111",
|
||||||
|
name: "patients",
|
||||||
|
sourceComment: "Patients imported from the clinical source",
|
||||||
|
description: null,
|
||||||
|
generatedDescription: null,
|
||||||
|
version: 1,
|
||||||
|
createdAt: "2026-08-27T08:00:00Z",
|
||||||
|
updatedAt: "2026-08-27T09:00:00Z",
|
||||||
|
};
|
||||||
|
|
||||||
|
test("filters catalog tables as the operator types", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
const mediciTable: CatalogTable = {
|
||||||
|
...patientsTable,
|
||||||
|
id: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
|
||||||
|
name: "bridge_medici",
|
||||||
|
sourceComment: "Doctors participating in clinical care",
|
||||||
|
};
|
||||||
|
server.use(
|
||||||
|
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([
|
||||||
|
patientsTable,
|
||||||
|
mediciTable,
|
||||||
|
])),
|
||||||
|
);
|
||||||
|
renderPage({
|
||||||
|
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||||
|
});
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||||
|
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||||
|
const tablesRegion = await screen.findByRole("region", { name: "Tables for Policlinico San Donato" });
|
||||||
|
expect(await screen.findByRole("button", { name: "Edit description for patients" })).toBeVisible();
|
||||||
|
expect(screen.getByRole("button", { name: "Edit description for bridge_medici" })).toBeVisible();
|
||||||
|
|
||||||
|
await user.type(screen.getByRole("textbox", { name: "Search tables" }), "medici");
|
||||||
|
|
||||||
|
expect(screen.getByRole("button", { name: "Edit description for bridge_medici" })).toBeVisible();
|
||||||
|
expect(await within(tablesRegion).findByText("1 of 2")).toBeVisible();
|
||||||
|
await waitFor(() => {
|
||||||
|
expect(screen.queryByRole("button", { name: "Edit description for patients" })).not.toBeInTheDocument();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("selected table exposes a direct Synchronize columns action and sends its id", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
let startBody: unknown;
|
||||||
|
const run = makeSyncRun("columns", [patientsTable.id]);
|
||||||
|
registerCompletedSyncRun(run);
|
||||||
|
server.use(
|
||||||
|
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
|
||||||
|
http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
|
||||||
|
startBody = await request.json();
|
||||||
|
return HttpResponse.json(run, { status: 202 });
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
renderPage({
|
||||||
|
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||||
|
});
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||||
|
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||||
|
const tableRow = await screen.findByRole("row", { name: /patients/ });
|
||||||
|
await user.click(within(tableRow).getByRole("checkbox"));
|
||||||
|
|
||||||
|
const synchronizeColumns = screen.getByRole("button", { name: "Synchronize columns" });
|
||||||
|
expect(synchronizeColumns).toBeVisible();
|
||||||
|
expect(synchronizeColumns).toBeEnabled();
|
||||||
|
expect(screen.queryByRole("button", { name: "Actions" })).not.toBeInTheDocument();
|
||||||
|
await user.click(synchronizeColumns);
|
||||||
|
|
||||||
|
await waitFor(() => expect(startBody).toEqual({
|
||||||
|
version: 3,
|
||||||
|
scope: "columns",
|
||||||
|
tableIds: [patientsTable.id],
|
||||||
|
}));
|
||||||
|
await waitFor(() => expect(screen.queryByText("1 selected")).not.toBeInTheDocument());
|
||||||
|
});
|
||||||
|
|
||||||
|
test("navigates purely from a database to its tables and edits review metadata", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
let patchBody: unknown;
|
||||||
|
server.use(
|
||||||
|
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
|
||||||
|
http.patch("/api/catalog/databases/:databaseId/tables/:tableId", async ({ request }) => {
|
||||||
|
patchBody = await request.json();
|
||||||
|
return HttpResponse.json({
|
||||||
|
...patientsTable,
|
||||||
|
description: "Registry used for longitudinal patient analysis",
|
||||||
|
version: 2,
|
||||||
|
});
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
renderPage({
|
||||||
|
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||||
|
});
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||||
|
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||||
|
|
||||||
|
expect(await screen.findByRole("region", { name: "Tables for Policlinico San Donato" })).toBeVisible();
|
||||||
|
expect(screen.getByRole("tab", { name: "Tables" })).toHaveAttribute("aria-selected", "true");
|
||||||
|
expect(screen.queryByLabelText("Database filter")).not.toBeInTheDocument();
|
||||||
|
await user.click(await screen.findByRole("button", { name: "Edit description for patients" }));
|
||||||
|
|
||||||
|
expect(screen.getByLabelText("Physical table name")).toHaveValue("patients");
|
||||||
|
expect(screen.getByLabelText("Physical table name")).toHaveAttribute("readonly");
|
||||||
|
expect(screen.getByLabelText("Source comment")).toHaveAttribute("readonly");
|
||||||
|
await user.type(
|
||||||
|
screen.getByLabelText(/^Description/),
|
||||||
|
"Registry used for longitudinal patient analysis",
|
||||||
|
);
|
||||||
|
await user.click(screen.getByRole("button", { name: "Save metadata" }));
|
||||||
|
|
||||||
|
await waitFor(() => expect(patchBody).toEqual({
|
||||||
|
version: 1,
|
||||||
|
description: "Registry used for longitudinal patient analysis",
|
||||||
|
generatedDescription: null,
|
||||||
|
}));
|
||||||
|
await user.click(screen.getByRole("button", { name: "Back to tables" }));
|
||||||
|
await user.click(screen.getByRole("tab", { name: "Overview" }));
|
||||||
|
expect(await screen.findByRole("heading", { name: "Database details" })).toBeVisible();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("navigates from a table to columns and from the database to physical relationships", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
let columnPatch: unknown;
|
||||||
|
const idColumn: CatalogColumn = {
|
||||||
|
id: "dddddddd-dddd-4ddd-8ddd-dddddddddddd",
|
||||||
|
tableId: patientsTable.id,
|
||||||
|
name: "id",
|
||||||
|
ordinalPosition: 1,
|
||||||
|
dataType: "bigint",
|
||||||
|
isNullable: false,
|
||||||
|
defaultExpression: null,
|
||||||
|
primaryKeyPosition: 1,
|
||||||
|
isPrimaryKey: true,
|
||||||
|
isForeignKey: true,
|
||||||
|
foreignKeyCount: 1,
|
||||||
|
sourceComment: "Patient identifier",
|
||||||
|
description: null,
|
||||||
|
generatedDescription: null,
|
||||||
|
lastSyncedDatabaseVersion: 3,
|
||||||
|
lastSyncedAt: "2026-08-27T10:00:00Z",
|
||||||
|
version: 1,
|
||||||
|
createdAt: "2026-08-27T10:00:00Z",
|
||||||
|
updatedAt: "2026-08-27T10:00:00Z",
|
||||||
|
};
|
||||||
|
server.use(
|
||||||
|
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
|
||||||
|
http.get("/api/catalog/databases/:databaseId/tables/:tableId/columns", () => HttpResponse.json([idColumn])),
|
||||||
|
http.patch("/api/catalog/databases/:databaseId/tables/:tableId/columns/:columnId", async ({ request }) => {
|
||||||
|
columnPatch = await request.json();
|
||||||
|
return HttpResponse.json({ ...idColumn, generatedDescription: "Generated identifier draft", version: 2 });
|
||||||
|
}),
|
||||||
|
http.get("/api/catalog/databases/:databaseId/relationships", () => HttpResponse.json([{
|
||||||
|
id: "eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeee",
|
||||||
|
databaseId: patientsTable.databaseId,
|
||||||
|
constraintName: "visits_patient_id_fkey",
|
||||||
|
sourceTableId: "ffffffff-ffff-4fff-8fff-ffffffffffff",
|
||||||
|
sourceTableName: "visits",
|
||||||
|
targetTableId: patientsTable.id,
|
||||||
|
targetTableName: "patients",
|
||||||
|
updateRule: "NO ACTION",
|
||||||
|
deleteRule: "CASCADE",
|
||||||
|
deferrable: false,
|
||||||
|
initiallyDeferred: false,
|
||||||
|
columns: [{ position: 1, sourceColumnId: "1", sourceColumnName: "patient_id", targetColumnId: idColumn.id, targetColumnName: "id" }],
|
||||||
|
lastSyncedDatabaseVersion: 3,
|
||||||
|
lastSyncedAt: "2026-08-27T10:00:00Z",
|
||||||
|
createdAt: "2026-08-27T10:00:00Z",
|
||||||
|
updatedAt: "2026-08-27T10:00:00Z",
|
||||||
|
}])),
|
||||||
|
);
|
||||||
|
renderPage({ rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })] });
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||||
|
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||||
|
await user.click(await screen.findByRole("button", { name: "View columns for patients" }));
|
||||||
|
expect(screen.getByRole("tab", { name: "Columns" })).toHaveAttribute("aria-selected", "true");
|
||||||
|
expect(await screen.findByText("PK")).toBeVisible();
|
||||||
|
expect(screen.getByText("FK")).toBeVisible();
|
||||||
|
await user.click(screen.getByRole("button", { name: "Edit metadata for id" }));
|
||||||
|
await user.type(screen.getByLabelText("Generated description"), "Generated identifier draft");
|
||||||
|
await user.click(screen.getByRole("button", { name: "Save metadata" }));
|
||||||
|
await waitFor(() => expect(columnPatch).toEqual({
|
||||||
|
version: 1,
|
||||||
|
description: null,
|
||||||
|
generatedDescription: "Generated identifier draft",
|
||||||
|
}));
|
||||||
|
await user.click(screen.getByRole("tab", { name: "Relationships" }));
|
||||||
|
expect(await screen.findByText("visits_patient_id_fkey")).toBeVisible();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("opens the durable job drawer and confirms its exact destructive plan", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
const startBodies: unknown[] = [];
|
||||||
|
const confirmationBodies: unknown[] = [];
|
||||||
|
const queued: CatalogSyncRun = {
|
||||||
|
id: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
|
||||||
|
databaseId: "11111111-1111-4111-8111-111111111111",
|
||||||
|
scope: "tables",
|
||||||
|
tableIds: [],
|
||||||
|
state: "queued",
|
||||||
|
phase: "queued",
|
||||||
|
requestedDatabaseVersion: 3,
|
||||||
|
plannedDiff: null,
|
||||||
|
confirmationToken: null,
|
||||||
|
counts: {},
|
||||||
|
errorCode: null,
|
||||||
|
errorMessage: null,
|
||||||
|
cancelRequested: false,
|
||||||
|
createdAt: "2026-08-27T10:00:00Z",
|
||||||
|
startedAt: null,
|
||||||
|
updatedAt: "2026-08-27T10:00:00Z",
|
||||||
|
finishedAt: null,
|
||||||
|
heartbeatAt: null,
|
||||||
|
};
|
||||||
|
const waiting: CatalogSyncRun = {
|
||||||
|
...queued,
|
||||||
|
state: "awaiting_confirmation",
|
||||||
|
phase: "awaiting_confirmation",
|
||||||
|
confirmationToken: "cccccccc-cccc-4ccc-8ccc-cccccccccccc",
|
||||||
|
plannedDiff: { deletedTables: ["legacy_visits"], deletedColumns: [], deletedRelationships: [] },
|
||||||
|
counts: { tables: 1 },
|
||||||
|
startedAt: "2026-08-27T10:00:00Z",
|
||||||
|
updatedAt: "2026-08-27T10:00:01Z",
|
||||||
|
};
|
||||||
|
server.use(
|
||||||
|
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([patientsTable])),
|
||||||
|
http.post("/api/catalog/databases/:databaseId/sync-runs", async ({ request }) => {
|
||||||
|
startBodies.push(await request.json());
|
||||||
|
return HttpResponse.json(queued, { status: 202 });
|
||||||
|
}),
|
||||||
|
http.get("/api/catalog/sync-runs/:runId", () => HttpResponse.json(waiting)),
|
||||||
|
http.get("/api/catalog/sync-runs/:runId/events-list", () => HttpResponse.json([])),
|
||||||
|
http.get("/api/catalog/databases/:databaseId/sync-runs", () => HttpResponse.json([waiting])),
|
||||||
|
http.post("/api/catalog/sync-runs/:runId/confirm", async ({ request }) => {
|
||||||
|
confirmationBodies.push(await request.json());
|
||||||
|
return HttpResponse.json({ ...waiting, state: "queued", phase: "queued", confirmationToken: null });
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
renderPage({
|
||||||
|
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||||
|
});
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||||
|
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||||
|
await user.click(await screen.findByRole("button", { name: "Sync tables" }));
|
||||||
|
|
||||||
|
const synchronizationDrawer = await screen.findByRole("complementary", {
|
||||||
|
name: "Schema synchronization",
|
||||||
|
});
|
||||||
|
expect(synchronizationDrawer).toHaveClass("inset-y-2", "sm:inset-y-4");
|
||||||
|
expect(within(synchronizationDrawer).getByRole("listitem", {
|
||||||
|
name: "Confirmation required, in progress",
|
||||||
|
})).toHaveAttribute("aria-current", "step");
|
||||||
|
expect(await screen.findByText("table · legacy_visits")).toBeVisible();
|
||||||
|
expect(startBodies).toEqual([{ version: 3, scope: "tables", tableIds: [] }]);
|
||||||
|
await user.click(screen.getByRole("button", { name: "Confirm removals" }));
|
||||||
|
await waitFor(() => expect(confirmationBodies).toEqual([
|
||||||
|
{ confirmationToken: "cccccccc-cccc-4ccc-8ccc-cccccccccccc" },
|
||||||
|
]));
|
||||||
|
});
|
||||||
|
|
||||||
|
test("keeps a stale table draft but requires an explicit reload before another save", async () => {
|
||||||
|
const user = userEvent.setup();
|
||||||
|
const current = {
|
||||||
|
...patientsTable,
|
||||||
|
description: "Description saved by another editor",
|
||||||
|
version: 2,
|
||||||
|
};
|
||||||
|
let latest = patientsTable;
|
||||||
|
server.use(
|
||||||
|
http.get("/api/catalog/databases/:databaseId/tables", () => HttpResponse.json([latest])),
|
||||||
|
http.patch("/api/catalog/databases/:databaseId/tables/:tableId", () => {
|
||||||
|
latest = current;
|
||||||
|
return HttpResponse.json({
|
||||||
|
code: "table_stale",
|
||||||
|
message: "Table description changed. Reload and try again.",
|
||||||
|
}, { status: 409 });
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
renderPage({
|
||||||
|
rows: [makeDatabase({ connectionStatus: "reachable", testedVersion: 3 })],
|
||||||
|
});
|
||||||
|
|
||||||
|
await user.click(await screen.findByRole("button", { name: "View Policlinico San Donato" }));
|
||||||
|
await user.click(screen.getByRole("tab", { name: "Tables" }));
|
||||||
|
await user.click(await screen.findByRole("button", { name: "Edit description for patients" }));
|
||||||
|
const description = screen.getByLabelText(/^Description/);
|
||||||
|
await user.type(description, "My unsaved draft");
|
||||||
|
await user.click(screen.getByRole("button", { name: "Save metadata" }));
|
||||||
|
|
||||||
|
expect(await screen.findByText("A newer table description is available.")).toBeVisible();
|
||||||
|
expect(description).toHaveValue("My unsaved draft");
|
||||||
|
expect(screen.getByRole("button", { name: "Save metadata" })).toBeDisabled();
|
||||||
|
await user.click(screen.getByRole("button", { name: "Keep editing" }));
|
||||||
|
expect(screen.getByText("Reload the latest value before this description can be saved.")).toBeVisible();
|
||||||
|
await user.click(screen.getByRole("button", { name: "Reload latest" }));
|
||||||
|
|
||||||
|
await waitFor(() => expect(description).toHaveValue("Description saved by another editor"));
|
||||||
|
expect(screen.getByRole("button", { name: "Save metadata" })).toBeDisabled();
|
||||||
|
});
|
||||||
@@ -1,8 +1,756 @@
|
|||||||
export function DatabaseManagementPage() {
|
import {
|
||||||
|
useCallback,
|
||||||
|
useEffect,
|
||||||
|
useMemo,
|
||||||
|
useRef,
|
||||||
|
useState,
|
||||||
|
} from "react";
|
||||||
|
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||||
|
import { Plus, RefreshCw } from "lucide-react";
|
||||||
|
import { toast } from "sonner";
|
||||||
|
import { Button } from "../components/ui/button";
|
||||||
|
import { ApiError, apiErrorMessage } from "../api/client";
|
||||||
|
import {
|
||||||
|
createCatalogDatabase,
|
||||||
|
deleteCatalogDatabase,
|
||||||
|
listCatalogDatabases,
|
||||||
|
replaceCatalogDatabaseSecrets,
|
||||||
|
startCatalogSync,
|
||||||
|
testCatalogDatabase,
|
||||||
|
updateCatalogDatabase,
|
||||||
|
type CatalogDatabase,
|
||||||
|
type CatalogSecretName,
|
||||||
|
type CatalogSyncScope,
|
||||||
|
type CatalogSyncRun,
|
||||||
|
type DatabaseBinding,
|
||||||
|
type DatabaseTransport,
|
||||||
|
} from "../api/catalog-databases";
|
||||||
|
import { DatabaseGrid } from "./database-management/DatabaseGrid";
|
||||||
|
import { DatabaseForm } from "./database-management/DatabaseForm";
|
||||||
|
import { DatabaseTables } from "./database-management/DatabaseTables";
|
||||||
|
import { DatabaseRelationships } from "./database-management/DatabaseRelationships";
|
||||||
|
import { CatalogSyncDrawer } from "./database-management/CatalogSyncDrawer";
|
||||||
|
import {
|
||||||
|
configurationFingerprint,
|
||||||
|
configurationFromDraft,
|
||||||
|
draftFrom,
|
||||||
|
hasSecretChanges,
|
||||||
|
secretReplacements,
|
||||||
|
type DatabaseBusyAction,
|
||||||
|
type DatabaseFormDraft,
|
||||||
|
type DatabaseFormMode,
|
||||||
|
type DatabaseNavigationState,
|
||||||
|
type DatabaseScreen,
|
||||||
|
} from "./database-management/model";
|
||||||
|
|
||||||
|
const DATABASE_QUERY_KEY = ["catalog-databases"] as const;
|
||||||
|
const SYNC_STARTED_MESSAGES: Record<CatalogSyncScope, string> = {
|
||||||
|
tables: "Table synchronization started",
|
||||||
|
columns: "Column synchronization started",
|
||||||
|
relationships: "Relationship synchronization started",
|
||||||
|
all: "Full schema synchronization started",
|
||||||
|
};
|
||||||
|
|
||||||
|
interface Props {
|
||||||
|
canManage: boolean;
|
||||||
|
canManageSecrets: boolean;
|
||||||
|
onNavigationStateChange?: (state: DatabaseNavigationState) => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface FormSource {
|
||||||
|
id?: string;
|
||||||
|
version: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isStaleError(error: unknown): boolean {
|
||||||
|
return error instanceof ApiError && error.code === "database_stale";
|
||||||
|
}
|
||||||
|
|
||||||
|
export function DatabaseManagementPage({
|
||||||
|
canManage,
|
||||||
|
canManageSecrets,
|
||||||
|
onNavigationStateChange,
|
||||||
|
}: Props) {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
const {
|
||||||
|
data,
|
||||||
|
isLoading,
|
||||||
|
isError,
|
||||||
|
isFetching,
|
||||||
|
refetch,
|
||||||
|
} = useQuery({
|
||||||
|
queryKey: DATABASE_QUERY_KEY,
|
||||||
|
queryFn: listCatalogDatabases,
|
||||||
|
retry: false,
|
||||||
|
});
|
||||||
|
const rows = data ?? [];
|
||||||
|
|
||||||
|
const [screen, setScreen] = useState<DatabaseScreen>({ kind: "list" });
|
||||||
|
const [draft, setDraft] = useState<DatabaseFormDraft | null>(null);
|
||||||
|
const [baseline, setBaseline] = useState("");
|
||||||
|
const [formSource, setFormSource] = useState<FormSource | null>(null);
|
||||||
|
const [search, setSearch] = useState("");
|
||||||
|
const [busyAction, setBusyAction] = useState<DatabaseBusyAction>(null);
|
||||||
|
const [stale, setStale] = useState(false);
|
||||||
|
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
|
||||||
|
const [partialSecretFailure, setPartialSecretFailure] = useState<string | null>(null);
|
||||||
|
const [tablesNavigationState, setTablesNavigationState] = useState<DatabaseNavigationState>({
|
||||||
|
dirty: false,
|
||||||
|
busy: false,
|
||||||
|
});
|
||||||
|
const [activeSyncRun, setActiveSyncRun] = useState<CatalogSyncRun | null>(null);
|
||||||
|
const [syncDrawerOpen, setSyncDrawerOpen] = useState(false);
|
||||||
|
|
||||||
|
const originRef = useRef<HTMLElement | null>(null);
|
||||||
|
const searchInputRef = useRef<HTMLInputElement>(null);
|
||||||
|
const formHeadingRef = useRef<HTMLHeadingElement>(null);
|
||||||
|
|
||||||
|
const activeRow = screen.kind === "list"
|
||||||
|
? undefined
|
||||||
|
: rows.find((row) => row.workspaceId === screen.workspaceId);
|
||||||
|
const availableWorkspaces = useMemo(
|
||||||
|
() => rows.filter((row) => row.workspaceAvailable && !row.configured),
|
||||||
|
[rows],
|
||||||
|
);
|
||||||
|
const editable = screen.kind === "add" || screen.kind === "edit";
|
||||||
|
const configurationDirty = Boolean(
|
||||||
|
editable
|
||||||
|
&& draft
|
||||||
|
&& configurationFingerprint(draft) !== baseline,
|
||||||
|
);
|
||||||
|
const dirty = Boolean(editable && draft && (configurationDirty || hasSecretChanges(draft)));
|
||||||
|
const busy = busyAction !== null;
|
||||||
|
const navigationDirty = screen.kind === "tables" ? tablesNavigationState.dirty : dirty;
|
||||||
|
const navigationBusy = screen.kind === "tables" ? tablesNavigationState.busy : busy;
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
onNavigationStateChange?.({ dirty: navigationDirty, busy: navigationBusy });
|
||||||
|
}, [navigationBusy, navigationDirty, onNavigationStateChange]);
|
||||||
|
|
||||||
|
useEffect(() => () => {
|
||||||
|
onNavigationStateChange?.({ dirty: false, busy: false });
|
||||||
|
}, [onNavigationStateChange]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
const warn = (event: BeforeUnloadEvent) => {
|
||||||
|
if (!navigationDirty && !navigationBusy) return;
|
||||||
|
event.preventDefault();
|
||||||
|
};
|
||||||
|
window.addEventListener("beforeunload", warn);
|
||||||
|
return () => window.removeEventListener("beforeunload", warn);
|
||||||
|
}, [navigationBusy, navigationDirty]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (screen.kind === "list" || !activeRow || !formSource || busy) return;
|
||||||
|
if (activeRow.id === formSource.id && activeRow.version === formSource.version) return;
|
||||||
|
setStale(true);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
}, [activeRow, busy, formSource, screen.kind]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (screen.kind === "list") return;
|
||||||
|
const timer = window.setTimeout(() => formHeadingRef.current?.focus(), 0);
|
||||||
|
return () => window.clearTimeout(timer);
|
||||||
|
}, [screen.kind, screen.kind === "list" ? "" : screen.workspaceId]);
|
||||||
|
|
||||||
|
const cacheSavedRow = useCallback((saved: CatalogDatabase) => {
|
||||||
|
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => {
|
||||||
|
const existing = current.findIndex((row) => row.workspaceId === saved.workspaceId);
|
||||||
|
if (existing < 0) return [...current, saved];
|
||||||
|
return current.map((row, index) => index === existing ? saved : row);
|
||||||
|
});
|
||||||
|
}, [queryClient]);
|
||||||
|
|
||||||
|
const restoreListFocus = useCallback(() => {
|
||||||
|
window.setTimeout(() => {
|
||||||
|
if (originRef.current?.isConnected) {
|
||||||
|
originRef.current.focus();
|
||||||
|
} else {
|
||||||
|
searchInputRef.current?.focus();
|
||||||
|
}
|
||||||
|
}, 0);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const showList = useCallback(() => {
|
||||||
|
setScreen({ kind: "list" });
|
||||||
|
setDraft(null);
|
||||||
|
setBaseline("");
|
||||||
|
setFormSource(null);
|
||||||
|
setStale(false);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
setPartialSecretFailure(null);
|
||||||
|
setTablesNavigationState({ dirty: false, busy: false });
|
||||||
|
restoreListFocus();
|
||||||
|
}, [restoreListFocus]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (screen.kind === "list" || isLoading || activeRow) return;
|
||||||
|
showList();
|
||||||
|
toast.info("This database configuration is no longer available");
|
||||||
|
}, [activeRow, isLoading, screen.kind, showList]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!["tables", "relationships"].includes(screen.kind) || !activeRow || (activeRow.configured && activeRow.id)) return;
|
||||||
|
showList();
|
||||||
|
toast.info("Save the database configuration before managing tables");
|
||||||
|
}, [activeRow, screen.kind, showList]);
|
||||||
|
|
||||||
|
const backToList = useCallback(() => {
|
||||||
|
if (busy) return;
|
||||||
|
if (dirty && !window.confirm("Discard unsaved database changes?")) return;
|
||||||
|
showList();
|
||||||
|
}, [busy, dirty, showList]);
|
||||||
|
|
||||||
|
const openForm = useCallback((
|
||||||
|
mode: DatabaseFormMode,
|
||||||
|
row: CatalogDatabase,
|
||||||
|
origin: HTMLElement,
|
||||||
|
workspaceLocked = false,
|
||||||
|
) => {
|
||||||
|
const nextDraft = draftFrom(row);
|
||||||
|
originRef.current = origin;
|
||||||
|
setDraft(nextDraft);
|
||||||
|
setBaseline(configurationFingerprint(nextDraft));
|
||||||
|
setFormSource({ id: row.id, version: row.version });
|
||||||
|
setStale(false);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
setPartialSecretFailure(null);
|
||||||
|
setScreen({
|
||||||
|
kind: mode,
|
||||||
|
workspaceId: row.workspaceId,
|
||||||
|
...(mode === "add" ? { workspaceLocked } : {}),
|
||||||
|
});
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const viewRow = useCallback((row: CatalogDatabase, origin: HTMLButtonElement) => {
|
||||||
|
openForm("view", row, origin);
|
||||||
|
}, [openForm]);
|
||||||
|
|
||||||
|
const editRow = useCallback((row: CatalogDatabase, origin: HTMLButtonElement) => {
|
||||||
|
openForm(row.configured ? "edit" : "add", row, origin, !row.configured);
|
||||||
|
}, [openForm]);
|
||||||
|
|
||||||
|
const deleteRow = useCallback((row: CatalogDatabase, origin: HTMLButtonElement) => {
|
||||||
|
openForm("delete", row, origin);
|
||||||
|
}, [openForm]);
|
||||||
|
|
||||||
|
const openTables = useCallback((row: CatalogDatabase, origin?: HTMLElement) => {
|
||||||
|
if (!row.configured || !row.id) return;
|
||||||
|
if (origin) originRef.current = origin;
|
||||||
|
setDraft(null);
|
||||||
|
setBaseline("");
|
||||||
|
setFormSource(null);
|
||||||
|
setStale(false);
|
||||||
|
setPartialSecretFailure(null);
|
||||||
|
setTablesNavigationState({ dirty: false, busy: false });
|
||||||
|
setScreen({ kind: "tables", workspaceId: row.workspaceId });
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const openRelationships = useCallback((row: CatalogDatabase) => {
|
||||||
|
if (!row.configured || !row.id) return;
|
||||||
|
setDraft(null);
|
||||||
|
setBaseline("");
|
||||||
|
setFormSource(null);
|
||||||
|
setStale(false);
|
||||||
|
setPartialSecretFailure(null);
|
||||||
|
setTablesNavigationState({ dirty: false, busy: false });
|
||||||
|
setScreen({ kind: "relationships", workspaceId: row.workspaceId });
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const openOverview = useCallback((row: CatalogDatabase) => {
|
||||||
|
const nextDraft = draftFrom(row);
|
||||||
|
setDraft(nextDraft);
|
||||||
|
setBaseline(configurationFingerprint(nextDraft));
|
||||||
|
setFormSource({ id: row.id, version: row.version });
|
||||||
|
setStale(false);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
setPartialSecretFailure(null);
|
||||||
|
setTablesNavigationState({ dirty: false, busy: false });
|
||||||
|
setScreen({ kind: "view", workspaceId: row.workspaceId });
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const addDatabase = useCallback((origin: HTMLElement) => {
|
||||||
|
const workspace = availableWorkspaces[0];
|
||||||
|
if (!workspace || !canManage) return;
|
||||||
|
openForm("add", workspace, origin, false);
|
||||||
|
}, [availableWorkspaces, canManage, openForm]);
|
||||||
|
|
||||||
|
const changeWorkspace = useCallback((workspaceId: string) => {
|
||||||
|
const workspace = availableWorkspaces.find((row) => row.workspaceId === workspaceId);
|
||||||
|
if (!workspace) return;
|
||||||
|
if (dirty && !window.confirm("Discard changes and choose another workspace?")) return;
|
||||||
|
const nextDraft = draftFrom(workspace);
|
||||||
|
setDraft(nextDraft);
|
||||||
|
setBaseline(configurationFingerprint(nextDraft));
|
||||||
|
setFormSource({ id: workspace.id, version: workspace.version });
|
||||||
|
setStale(false);
|
||||||
|
setPartialSecretFailure(null);
|
||||||
|
setScreen({ kind: "add", workspaceId, workspaceLocked: false });
|
||||||
|
}, [availableWorkspaces, dirty]);
|
||||||
|
|
||||||
|
const changeField = useCallback((field: "databaseName" | "schema", value: string) => {
|
||||||
|
setDraft((current) => current ? { ...current, [field]: value } : current);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const changeTransport = useCallback((transport: DatabaseTransport) => {
|
||||||
|
setDraft((current) => current
|
||||||
|
? { ...current, binding: { ...current.binding, transport } }
|
||||||
|
: current);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const changeBinding = useCallback(<K extends keyof DatabaseBinding>(
|
||||||
|
key: K,
|
||||||
|
value: DatabaseBinding[K],
|
||||||
|
) => {
|
||||||
|
setDraft((current) => current
|
||||||
|
? { ...current, binding: { ...current.binding, [key]: value } }
|
||||||
|
: current);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const changeSecret = useCallback((name: CatalogSecretName, value: string) => {
|
||||||
|
setDraft((current) => current
|
||||||
|
? { ...current, secrets: { ...current.secrets, [name]: value } }
|
||||||
|
: current);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const markStale = useCallback(() => {
|
||||||
|
setStale(true);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const save = useCallback(async () => {
|
||||||
|
if (!activeRow || !formSource || !draft || !editable || !canManage || stale || busy) return;
|
||||||
|
setBusyAction("save");
|
||||||
|
setPartialSecretFailure(null);
|
||||||
|
|
||||||
|
try {
|
||||||
|
const input = configurationFromDraft(draft);
|
||||||
|
let saved = activeRow;
|
||||||
|
const shouldPersistConfiguration = !formSource.id || configurationDirty;
|
||||||
|
|
||||||
|
if (shouldPersistConfiguration) {
|
||||||
|
saved = formSource.id
|
||||||
|
? await updateCatalogDatabase(formSource.id, formSource.version, input)
|
||||||
|
: await createCatalogDatabase(input);
|
||||||
|
setFormSource({ id: saved.id, version: saved.version });
|
||||||
|
cacheSavedRow(saved);
|
||||||
|
}
|
||||||
|
|
||||||
|
const replacements = secretReplacements(draft);
|
||||||
|
if (Object.keys(replacements).length > 0) {
|
||||||
|
if (!canManageSecrets) throw new Error("Secret replacement is not permitted");
|
||||||
|
try {
|
||||||
|
saved = await replaceCatalogDatabaseSecrets(
|
||||||
|
saved.id ?? formSource.id!,
|
||||||
|
shouldPersistConfiguration ? saved.version : formSource.version,
|
||||||
|
replacements,
|
||||||
|
);
|
||||||
|
setFormSource({ id: saved.id, version: saved.version });
|
||||||
|
cacheSavedRow(saved);
|
||||||
|
} catch (error) {
|
||||||
|
setBaseline(configurationFingerprint(draft));
|
||||||
|
setScreen({ kind: "edit", workspaceId: saved.workspaceId });
|
||||||
|
setPartialSecretFailure(apiErrorMessage(error));
|
||||||
|
if (isStaleError(error)) {
|
||||||
|
markStale();
|
||||||
|
} else {
|
||||||
|
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||||
|
}
|
||||||
|
toast.warning("Database configuration saved, but secrets still need attention");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const nextDraft = draftFrom(saved);
|
||||||
|
setDraft(nextDraft);
|
||||||
|
setBaseline(configurationFingerprint(nextDraft));
|
||||||
|
setFormSource({ id: saved.id, version: saved.version });
|
||||||
|
setScreen({ kind: "edit", workspaceId: saved.workspaceId });
|
||||||
|
setStale(false);
|
||||||
|
setPartialSecretFailure(null);
|
||||||
|
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||||
|
toast.success(activeRow.configured ? "Database configuration saved" : "Database configuration added");
|
||||||
|
} catch (error) {
|
||||||
|
if (isStaleError(error)) {
|
||||||
|
markStale();
|
||||||
|
} else {
|
||||||
|
toast.error(apiErrorMessage(error));
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
setBusyAction(null);
|
||||||
|
}
|
||||||
|
}, [
|
||||||
|
activeRow,
|
||||||
|
busy,
|
||||||
|
cacheSavedRow,
|
||||||
|
canManage,
|
||||||
|
canManageSecrets,
|
||||||
|
configurationDirty,
|
||||||
|
draft,
|
||||||
|
editable,
|
||||||
|
formSource,
|
||||||
|
markStale,
|
||||||
|
queryClient,
|
||||||
|
stale,
|
||||||
|
]);
|
||||||
|
|
||||||
|
const retrySecrets = useCallback(async () => {
|
||||||
|
if (!activeRow?.configured || !formSource?.id || !draft || !canManageSecrets || busy || stale) return;
|
||||||
|
const replacements = secretReplacements(draft);
|
||||||
|
if (Object.keys(replacements).length === 0) {
|
||||||
|
setPartialSecretFailure(null);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
setBusyAction("retry-secrets");
|
||||||
|
try {
|
||||||
|
const saved = await replaceCatalogDatabaseSecrets(formSource.id, formSource.version, replacements);
|
||||||
|
setFormSource({ id: saved.id, version: saved.version });
|
||||||
|
cacheSavedRow(saved);
|
||||||
|
setDraft({ ...draft, secrets: {} });
|
||||||
|
setPartialSecretFailure(null);
|
||||||
|
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||||
|
toast.success("Database secrets saved");
|
||||||
|
} catch (error) {
|
||||||
|
setPartialSecretFailure(apiErrorMessage(error));
|
||||||
|
if (isStaleError(error)) markStale();
|
||||||
|
toast.error(apiErrorMessage(error));
|
||||||
|
} finally {
|
||||||
|
setBusyAction(null);
|
||||||
|
}
|
||||||
|
}, [activeRow, busy, cacheSavedRow, canManageSecrets, draft, formSource, markStale, queryClient, stale]);
|
||||||
|
|
||||||
|
const testConnection = useCallback(async () => {
|
||||||
|
if (
|
||||||
|
!activeRow?.configured
|
||||||
|
|| !formSource?.id
|
||||||
|
|| !draft
|
||||||
|
|| !canManage
|
||||||
|
|| dirty
|
||||||
|
|| stale
|
||||||
|
|| busy
|
||||||
|
) return;
|
||||||
|
|
||||||
|
setBusyAction("test");
|
||||||
|
try {
|
||||||
|
const tested = await testCatalogDatabase(formSource.id, formSource.version);
|
||||||
|
setFormSource({ id: tested.id, version: tested.version });
|
||||||
|
cacheSavedRow(tested);
|
||||||
|
const nextDraft = draftFrom(tested);
|
||||||
|
setDraft(nextDraft);
|
||||||
|
setBaseline(configurationFingerprint(nextDraft));
|
||||||
|
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||||
|
if (tested.connectionStatus === "reachable") {
|
||||||
|
toast.success("Database connection is reachable");
|
||||||
|
} else {
|
||||||
|
toast.error(tested.lastErrorMessage ?? "Database connection failed");
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
if (isStaleError(error)) markStale();
|
||||||
|
else toast.error(apiErrorMessage(error));
|
||||||
|
} finally {
|
||||||
|
setBusyAction(null);
|
||||||
|
}
|
||||||
|
}, [activeRow, busy, cacheSavedRow, canManage, dirty, draft, formSource, markStale, queryClient, stale]);
|
||||||
|
|
||||||
|
const remove = useCallback(async () => {
|
||||||
|
if (!activeRow?.configured || !formSource?.id || !canManage || busy || stale) return;
|
||||||
|
setBusyAction("delete");
|
||||||
|
try {
|
||||||
|
await deleteCatalogDatabase(formSource.id, formSource.version);
|
||||||
|
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => {
|
||||||
|
if (!activeRow.workspaceAvailable) {
|
||||||
|
return current.filter((row) => row.workspaceId !== activeRow.workspaceId);
|
||||||
|
}
|
||||||
|
return current.map((row) => row.workspaceId === activeRow.workspaceId
|
||||||
|
? {
|
||||||
|
...row,
|
||||||
|
id: undefined,
|
||||||
|
configured: false,
|
||||||
|
version: 0,
|
||||||
|
createdAt: "",
|
||||||
|
updatedAt: "",
|
||||||
|
connectionStatus: "untested",
|
||||||
|
testedVersion: undefined,
|
||||||
|
lastTestedAt: undefined,
|
||||||
|
lastErrorCode: undefined,
|
||||||
|
lastErrorMessage: undefined,
|
||||||
|
secrets: {
|
||||||
|
password: false,
|
||||||
|
apiKey: false,
|
||||||
|
sshPrivateKey: false,
|
||||||
|
sshPrivateKeyPassphrase: false,
|
||||||
|
sshKnownHosts: false,
|
||||||
|
tlsCa: false,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
: row);
|
||||||
|
});
|
||||||
|
setBusyAction(null);
|
||||||
|
showList();
|
||||||
|
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||||
|
toast.success("Database configuration deleted");
|
||||||
|
} catch (error) {
|
||||||
|
if (isStaleError(error)) markStale();
|
||||||
|
else toast.error(apiErrorMessage(error));
|
||||||
|
} finally {
|
||||||
|
setBusyAction(null);
|
||||||
|
}
|
||||||
|
}, [activeRow, busy, canManage, formSource, markStale, queryClient, showList, stale]);
|
||||||
|
|
||||||
|
const reloadLatest = useCallback(async () => {
|
||||||
|
if (busy || screen.kind === "list") return;
|
||||||
|
setBusyAction("reload");
|
||||||
|
try {
|
||||||
|
const result = await refetch();
|
||||||
|
if (result.error) throw result.error;
|
||||||
|
const latest = result.data?.find((row) => row.workspaceId === screen.workspaceId);
|
||||||
|
if (!latest) {
|
||||||
|
showList();
|
||||||
|
toast.info("This database configuration is no longer available");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const nextDraft = draftFrom(latest);
|
||||||
|
setDraft(nextDraft);
|
||||||
|
setBaseline(configurationFingerprint(nextDraft));
|
||||||
|
setFormSource({ id: latest.id, version: latest.version });
|
||||||
|
setStale(false);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
setPartialSecretFailure(null);
|
||||||
|
if (!latest.configured && screen.kind === "delete") {
|
||||||
|
showList();
|
||||||
|
toast.info("This database configuration has already been deleted");
|
||||||
|
} else if (screen.kind === "edit" && !latest.configured) {
|
||||||
|
setScreen({ kind: "add", workspaceId: latest.workspaceId, workspaceLocked: true });
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
toast.error(apiErrorMessage(error));
|
||||||
|
} finally {
|
||||||
|
setBusyAction(null);
|
||||||
|
}
|
||||||
|
}, [busy, refetch, screen, showList]);
|
||||||
|
|
||||||
|
const refreshList = useCallback(async () => {
|
||||||
|
if (isFetching) return;
|
||||||
|
try {
|
||||||
|
const result = await refetch();
|
||||||
|
if (result.error) throw result.error;
|
||||||
|
} catch (error) {
|
||||||
|
toast.error(apiErrorMessage(error));
|
||||||
|
}
|
||||||
|
}, [isFetching, refetch]);
|
||||||
|
|
||||||
|
const updateTrackedSyncRun = useCallback((run: CatalogSyncRun) => {
|
||||||
|
setActiveSyncRun(run);
|
||||||
|
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => current.map((row) => (
|
||||||
|
row.id === run.databaseId
|
||||||
|
? { ...row, activeSyncRun: ["queued", "running", "awaiting_confirmation", "applying"].includes(run.state) ? run : undefined }
|
||||||
|
: row
|
||||||
|
)));
|
||||||
|
}, [queryClient]);
|
||||||
|
|
||||||
|
const rememberSyncRun = useCallback((run: CatalogSyncRun) => {
|
||||||
|
updateTrackedSyncRun(run);
|
||||||
|
setSyncDrawerOpen(true);
|
||||||
|
}, [updateTrackedSyncRun]);
|
||||||
|
|
||||||
|
const openSync = useCallback((row?: CatalogDatabase) => {
|
||||||
|
const run = row?.activeSyncRun ?? activeSyncRun;
|
||||||
|
if (!run) return;
|
||||||
|
setActiveSyncRun(run);
|
||||||
|
setSyncDrawerOpen(true);
|
||||||
|
}, [activeSyncRun]);
|
||||||
|
|
||||||
|
const testSelected = useCallback(async (selected: CatalogDatabase[]) => {
|
||||||
|
try {
|
||||||
|
const tested = await Promise.all(selected.map((row) => testCatalogDatabase(row.id!, row.version)));
|
||||||
|
for (const row of tested) cacheSavedRow(row);
|
||||||
|
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||||
|
toast.success(`${tested.length} connection${tested.length === 1 ? "" : "s"} tested`);
|
||||||
|
} catch (error) {
|
||||||
|
toast.error(apiErrorMessage(error));
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}, [cacheSavedRow, queryClient]);
|
||||||
|
|
||||||
|
const syncSelected = useCallback(async (selected: CatalogDatabase[], scope: CatalogSyncScope) => {
|
||||||
|
try {
|
||||||
|
const runs = await Promise.all(selected.map((row) => startCatalogSync(row.id!, row.version, scope)));
|
||||||
|
queryClient.setQueryData<CatalogDatabase[]>(DATABASE_QUERY_KEY, (current = []) => current.map((row) => {
|
||||||
|
const run = runs.find((candidate) => candidate.databaseId === row.id);
|
||||||
|
return run ? { ...row, activeSyncRun: run } : row;
|
||||||
|
}));
|
||||||
|
if (runs[0]) rememberSyncRun(runs[0]);
|
||||||
|
toast.success(`${runs.length} schema synchronization${runs.length === 1 ? "" : "s"} started`);
|
||||||
|
} catch (error) {
|
||||||
|
toast.error(apiErrorMessage(error));
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}, [queryClient, rememberSyncRun]);
|
||||||
|
|
||||||
|
const syncDatabase = useCallback(async (scope: CatalogSyncScope) => {
|
||||||
|
if (!activeRow?.id || !activeRow.configured) return;
|
||||||
|
try {
|
||||||
|
rememberSyncRun(await startCatalogSync(activeRow.id, activeRow.version, scope));
|
||||||
|
toast.success(SYNC_STARTED_MESSAGES[scope]);
|
||||||
|
} catch (error) { toast.error(apiErrorMessage(error)); }
|
||||||
|
}, [activeRow, rememberSyncRun]);
|
||||||
|
|
||||||
|
const catalogChanged = useCallback(async () => {
|
||||||
|
const databaseId = activeSyncRun?.databaseId;
|
||||||
|
if (databaseId) {
|
||||||
|
await Promise.all([
|
||||||
|
queryClient.invalidateQueries({ queryKey: ["catalog-tables", databaseId] }),
|
||||||
|
queryClient.invalidateQueries({ queryKey: ["catalog-relationships", databaseId] }),
|
||||||
|
queryClient.invalidateQueries({ queryKey: ["catalog-sync-runs", databaseId] }),
|
||||||
|
]);
|
||||||
|
}
|
||||||
|
await queryClient.invalidateQueries({ queryKey: DATABASE_QUERY_KEY });
|
||||||
|
}, [activeSyncRun, queryClient]);
|
||||||
|
|
||||||
|
const formVisible = screen.kind !== "list" && screen.kind !== "tables" && screen.kind !== "relationships" && activeRow && draft;
|
||||||
|
const tablesVisible = screen.kind === "tables" && activeRow?.configured && activeRow.id;
|
||||||
|
const relationshipsVisible = screen.kind === "relationships" && activeRow?.configured && activeRow.id;
|
||||||
|
const currentActiveRun = activeSyncRun && ["queued", "running", "awaiting_confirmation", "applying"].includes(activeSyncRun.state)
|
||||||
|
? activeSyncRun
|
||||||
|
: activeRow?.activeSyncRun && ["queued", "running", "awaiting_confirmation", "applying"].includes(activeRow.activeSyncRun.state)
|
||||||
|
? activeRow.activeSyncRun
|
||||||
|
: undefined;
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<main
|
<main aria-label="Database management" className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
|
||||||
aria-label="Database management"
|
<header className="flex flex-wrap items-center justify-between gap-4 border-b border-border bg-card px-4 py-4 sm:px-5">
|
||||||
className="flex-1 overflow-y-auto bg-background"
|
<div>
|
||||||
|
<p className="thot-label mb-1">Administration</p>
|
||||||
|
<h1 className="font-heading text-xl font-semibold tracking-tight">Database management</h1>
|
||||||
|
<p className="mt-1 text-sm text-muted-foreground">
|
||||||
|
One database configuration for each repository workspace.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
{screen.kind === "list" ? (
|
||||||
|
<div className="flex items-center gap-2">
|
||||||
|
<Button type="button" variant="outline" disabled={isFetching} onClick={() => void refreshList()}>
|
||||||
|
<RefreshCw className={isFetching ? "animate-spin" : ""} /> Refresh
|
||||||
|
</Button>
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
disabled={!canManage || availableWorkspaces.length === 0}
|
||||||
|
title={availableWorkspaces.length === 0 ? "Every available workspace is already configured" : undefined}
|
||||||
|
onClick={(event) => addDatabase(event.currentTarget)}
|
||||||
|
>
|
||||||
|
<Plus /> Add database
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<div className="relative flex min-h-0 flex-1">
|
||||||
|
<div
|
||||||
|
aria-hidden={screen.kind !== "list"}
|
||||||
|
className={screen.kind === "list"
|
||||||
|
? "flex min-h-0 flex-1"
|
||||||
|
: "pointer-events-none invisible absolute inset-0 flex min-h-0"
|
||||||
|
}
|
||||||
|
>
|
||||||
|
{isError && rows.length === 0 ? (
|
||||||
|
<div className="grid flex-1 place-items-center p-6">
|
||||||
|
<div className="max-w-lg rounded-md border border-destructive/30 bg-destructive/8 p-5 text-sm" role="alert">
|
||||||
|
<p className="font-semibold text-destructive">The database catalog is unavailable.</p>
|
||||||
|
<p className="mt-1 leading-5 text-muted-foreground">Verify the catalog service and database migrations, then try again.</p>
|
||||||
|
<Button type="button" variant="outline" className="mt-4" disabled={isFetching} onClick={() => void refreshList()}>
|
||||||
|
<RefreshCw /> Try again
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<DatabaseGrid
|
||||||
|
rows={rows}
|
||||||
|
isLoading={isLoading}
|
||||||
|
isFetching={isFetching}
|
||||||
|
search={search}
|
||||||
|
canManage={canManage}
|
||||||
|
searchInputRef={searchInputRef}
|
||||||
|
onSearchChange={setSearch}
|
||||||
|
onView={viewRow}
|
||||||
|
onEdit={editRow}
|
||||||
|
onDelete={deleteRow}
|
||||||
|
onOpenSync={openSync}
|
||||||
|
onTestSelected={testSelected}
|
||||||
|
onSyncSelected={syncSelected}
|
||||||
/>
|
/>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{formVisible ? (
|
||||||
|
<DatabaseForm
|
||||||
|
mode={screen.kind as DatabaseFormMode}
|
||||||
|
row={activeRow}
|
||||||
|
availableWorkspaces={availableWorkspaces}
|
||||||
|
workspaceLocked={screen.kind === "add" && Boolean(screen.workspaceLocked)}
|
||||||
|
draft={draft}
|
||||||
|
dirty={dirty}
|
||||||
|
canManage={canManage}
|
||||||
|
canManageSecrets={canManageSecrets}
|
||||||
|
busyAction={busyAction}
|
||||||
|
stale={stale}
|
||||||
|
staleBannerOpen={staleBannerOpen}
|
||||||
|
partialSecretFailure={partialSecretFailure}
|
||||||
|
headingRef={formHeadingRef}
|
||||||
|
onBack={backToList}
|
||||||
|
onWorkspaceChange={changeWorkspace}
|
||||||
|
onFieldChange={changeField}
|
||||||
|
onTransportChange={changeTransport}
|
||||||
|
onBindingChange={changeBinding}
|
||||||
|
onSecretChange={changeSecret}
|
||||||
|
onSave={() => void save()}
|
||||||
|
onTest={() => void testConnection()}
|
||||||
|
onDelete={() => void remove()}
|
||||||
|
onReloadLatest={() => void reloadLatest()}
|
||||||
|
onKeepEditing={() => setStaleBannerOpen(false)}
|
||||||
|
onRetrySecrets={() => void retrySecrets()}
|
||||||
|
onOpenTables={(origin) => openTables(activeRow, origin)}
|
||||||
|
onOpenRelationships={() => openRelationships(activeRow)}
|
||||||
|
onSync={(scope) => void syncDatabase(scope)}
|
||||||
|
onOpenSync={() => openSync(activeRow)}
|
||||||
|
activeSyncRun={currentActiveRun}
|
||||||
|
/>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{tablesVisible ? (
|
||||||
|
<DatabaseTables
|
||||||
|
database={activeRow}
|
||||||
|
canManage={canManage}
|
||||||
|
activeRun={currentActiveRun}
|
||||||
|
onBackToDatabases={showList}
|
||||||
|
onOpenOverview={() => openOverview(activeRow)}
|
||||||
|
onOpenRelationships={() => openRelationships(activeRow)}
|
||||||
|
onNavigationStateChange={setTablesNavigationState}
|
||||||
|
onRunStarted={rememberSyncRun}
|
||||||
|
onOpenSync={() => openSync(activeRow)}
|
||||||
|
/>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{relationshipsVisible ? (
|
||||||
|
<DatabaseRelationships
|
||||||
|
database={{ ...activeRow, activeSyncRun: currentActiveRun }}
|
||||||
|
canManage={canManage}
|
||||||
|
onBackToDatabases={showList}
|
||||||
|
onOpenOverview={() => openOverview(activeRow)}
|
||||||
|
onOpenTables={() => openTables(activeRow)}
|
||||||
|
onRunStarted={rememberSyncRun}
|
||||||
|
onOpenSync={() => openSync(activeRow)}
|
||||||
|
/>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
<CatalogSyncDrawer
|
||||||
|
databaseId={activeSyncRun?.databaseId ?? activeRow?.id ?? null}
|
||||||
|
runId={activeSyncRun?.id ?? null}
|
||||||
|
open={syncDrawerOpen && Boolean(activeSyncRun)}
|
||||||
|
onClose={() => setSyncDrawerOpen(false)}
|
||||||
|
onRunChange={rememberSyncRun}
|
||||||
|
onRunUpdate={updateTrackedSyncRun}
|
||||||
|
onCatalogChanged={() => void catalogChanged()}
|
||||||
|
/>
|
||||||
|
</main>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,402 @@
|
|||||||
|
import { useEffect, useMemo, useRef, useState } from "react";
|
||||||
|
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||||
|
import { AlertTriangle, Check, CheckCircle2, Circle, LoaderCircle, Minus, RotateCcw, X } from "lucide-react";
|
||||||
|
import { toast } from "sonner";
|
||||||
|
import { Button } from "../../components/ui/button";
|
||||||
|
import { apiErrorMessage } from "../../api/client";
|
||||||
|
import {
|
||||||
|
cancelCatalogSync,
|
||||||
|
catalogSyncEventsUrl,
|
||||||
|
confirmCatalogSync,
|
||||||
|
getCatalogSyncRun,
|
||||||
|
listCatalogSyncEvents,
|
||||||
|
listCatalogSyncRuns,
|
||||||
|
retryCatalogSync,
|
||||||
|
type CatalogSyncEvent,
|
||||||
|
type CatalogSyncPhase,
|
||||||
|
type CatalogSyncRun,
|
||||||
|
} from "../../api/catalog-databases";
|
||||||
|
|
||||||
|
interface Props {
|
||||||
|
databaseId: string | null;
|
||||||
|
runId: string | null;
|
||||||
|
open: boolean;
|
||||||
|
onClose: () => void;
|
||||||
|
onRunChange: (run: CatalogSyncRun) => void;
|
||||||
|
onRunUpdate: (run: CatalogSyncRun) => void;
|
||||||
|
onCatalogChanged: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
const phases: CatalogSyncPhase[] = [
|
||||||
|
"queued", "connecting", "scanning_tables", "scanning_columns",
|
||||||
|
"scanning_relationships", "planning", "awaiting_confirmation", "applying", "completed",
|
||||||
|
];
|
||||||
|
|
||||||
|
const phaseEnteredByEvent: Record<string, CatalogSyncPhase> = {
|
||||||
|
queued: "queued",
|
||||||
|
started: "connecting",
|
||||||
|
connecting: "connecting",
|
||||||
|
scanning_tables: "scanning_tables",
|
||||||
|
scanning_columns: "scanning_columns",
|
||||||
|
scanning_relationships: "scanning_relationships",
|
||||||
|
planning: "planning",
|
||||||
|
confirmation_required: "awaiting_confirmation",
|
||||||
|
confirmation_received: "awaiting_confirmation",
|
||||||
|
applying: "applying",
|
||||||
|
succeeded: "completed",
|
||||||
|
};
|
||||||
|
|
||||||
|
const phaseLabels: Record<CatalogSyncPhase, { pending: string; current: string; completed: string }> = {
|
||||||
|
queued: { pending: "Queue", current: "Queued", completed: "Queued" },
|
||||||
|
connecting: { pending: "Connect", current: "Connecting", completed: "Connected" },
|
||||||
|
scanning_tables: { pending: "Read tables", current: "Reading tables", completed: "Tables read" },
|
||||||
|
scanning_columns: { pending: "Read columns", current: "Reading columns", completed: "Columns read" },
|
||||||
|
scanning_relationships: {
|
||||||
|
pending: "Read relationships",
|
||||||
|
current: "Reading relationships",
|
||||||
|
completed: "Relationships read",
|
||||||
|
},
|
||||||
|
planning: { pending: "Plan changes", current: "Planning changes", completed: "Changes planned" },
|
||||||
|
awaiting_confirmation: {
|
||||||
|
pending: "Confirm changes if needed",
|
||||||
|
current: "Confirmation required",
|
||||||
|
completed: "Changes confirmed",
|
||||||
|
},
|
||||||
|
applying: { pending: "Apply changes", current: "Applying changes", completed: "Changes applied" },
|
||||||
|
completed: { pending: "Complete", current: "Completing", completed: "Completed" },
|
||||||
|
};
|
||||||
|
|
||||||
|
function terminal(run?: CatalogSyncRun): boolean {
|
||||||
|
return Boolean(run && ["succeeded", "failed", "cancelled", "interrupted"].includes(run.state));
|
||||||
|
}
|
||||||
|
|
||||||
|
function elapsed(run: CatalogSyncRun, now: number): string {
|
||||||
|
const start = new Date(run.startedAt ?? run.createdAt).getTime();
|
||||||
|
const end = run.finishedAt ? new Date(run.finishedAt).getTime() : now;
|
||||||
|
const seconds = Math.max(0, Math.floor((end - start) / 1000));
|
||||||
|
const minutes = Math.floor(seconds / 60);
|
||||||
|
return minutes ? `${minutes}m ${seconds % 60}s` : `${seconds}s`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function stateLabel(run: CatalogSyncRun): string {
|
||||||
|
return run.state.replaceAll("_", " ");
|
||||||
|
}
|
||||||
|
|
||||||
|
export function CatalogSyncDrawer({
|
||||||
|
databaseId,
|
||||||
|
runId,
|
||||||
|
open,
|
||||||
|
onClose,
|
||||||
|
onRunChange,
|
||||||
|
onRunUpdate,
|
||||||
|
onCatalogChanged,
|
||||||
|
}: Props) {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
const [events, setEvents] = useState<CatalogSyncEvent[]>([]);
|
||||||
|
const [now, setNow] = useState(Date.now());
|
||||||
|
const [action, setAction] = useState<"confirm" | "cancel" | "retry" | null>(null);
|
||||||
|
const lastSequence = useRef(0);
|
||||||
|
const notifiedRun = useRef<string | null>(null);
|
||||||
|
|
||||||
|
const runQuery = useQuery({
|
||||||
|
queryKey: ["catalog-sync-run", runId],
|
||||||
|
queryFn: () => getCatalogSyncRun(runId!),
|
||||||
|
enabled: Boolean(runId),
|
||||||
|
retry: false,
|
||||||
|
refetchInterval: (query) => terminal(query.state.data) ? false : 1_000,
|
||||||
|
});
|
||||||
|
const run = runQuery.data;
|
||||||
|
const historyQuery = useQuery({
|
||||||
|
queryKey: ["catalog-sync-runs", databaseId],
|
||||||
|
queryFn: () => listCatalogSyncRuns(databaseId!),
|
||||||
|
enabled: Boolean(databaseId && open),
|
||||||
|
retry: false,
|
||||||
|
refetchInterval: run && !terminal(run) ? 2_000 : false,
|
||||||
|
});
|
||||||
|
const eventQuery = useQuery({
|
||||||
|
queryKey: ["catalog-sync-events", runId],
|
||||||
|
queryFn: () => listCatalogSyncEvents(runId!, lastSequence.current),
|
||||||
|
enabled: Boolean(runId && open),
|
||||||
|
retry: false,
|
||||||
|
refetchInterval: run && terminal(run) ? false : 1_500,
|
||||||
|
});
|
||||||
|
|
||||||
|
const mergeEvents = (incoming: CatalogSyncEvent[]) => {
|
||||||
|
if (incoming.length === 0) return;
|
||||||
|
setEvents((current) => {
|
||||||
|
const bySequence = new Map(current.map((event) => [event.sequence, event]));
|
||||||
|
for (const event of incoming) bySequence.set(event.sequence, event);
|
||||||
|
const merged = [...bySequence.values()].sort((a, b) => a.sequence - b.sequence);
|
||||||
|
lastSequence.current = merged.at(-1)?.sequence ?? lastSequence.current;
|
||||||
|
return merged;
|
||||||
|
});
|
||||||
|
};
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
setEvents([]);
|
||||||
|
lastSequence.current = 0;
|
||||||
|
}, [runId]);
|
||||||
|
|
||||||
|
useEffect(() => { mergeEvents(eventQuery.data ?? []); }, [eventQuery.data]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!runId || !open || typeof EventSource === "undefined") return;
|
||||||
|
const source = new EventSource(catalogSyncEventsUrl(runId));
|
||||||
|
const log = (event: MessageEvent<string>) => {
|
||||||
|
try { mergeEvents([JSON.parse(event.data) as CatalogSyncEvent]); } catch { /* polling remains authoritative */ }
|
||||||
|
};
|
||||||
|
const update = (event: MessageEvent<string>) => {
|
||||||
|
try {
|
||||||
|
const next = JSON.parse(event.data) as CatalogSyncRun;
|
||||||
|
queryClient.setQueryData(["catalog-sync-run", runId], next);
|
||||||
|
} catch { /* polling remains authoritative */ }
|
||||||
|
};
|
||||||
|
source.addEventListener("log", log as EventListener);
|
||||||
|
source.addEventListener("run", update as EventListener);
|
||||||
|
source.onerror = () => source.close();
|
||||||
|
return () => source.close();
|
||||||
|
}, [open, queryClient, runId]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!run || terminal(run)) return;
|
||||||
|
const timer = window.setInterval(() => setNow(Date.now()), 1_000);
|
||||||
|
return () => window.clearInterval(timer);
|
||||||
|
}, [run]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!run) return;
|
||||||
|
onRunUpdate(run);
|
||||||
|
queryClient.setQueryData<CatalogSyncRun[]>(["catalog-sync-runs", run.databaseId], (current) => {
|
||||||
|
if (!current) return [run];
|
||||||
|
const found = current.some((item) => item.id === run.id);
|
||||||
|
return found
|
||||||
|
? current.map((item) => item.id === run.id ? run : item)
|
||||||
|
: [run, ...current];
|
||||||
|
});
|
||||||
|
}, [onRunUpdate, queryClient, run]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (!run || run.state !== "succeeded" || notifiedRun.current === run.id) return;
|
||||||
|
notifiedRun.current = run.id;
|
||||||
|
onCatalogChanged();
|
||||||
|
}, [onCatalogChanged, run]);
|
||||||
|
|
||||||
|
const currentPhase = run ? phases.indexOf(run.phase) : -1;
|
||||||
|
const knownEvents = [...events, ...(eventQuery.data ?? [])];
|
||||||
|
const confirmationWasRequired = run?.state === "awaiting_confirmation" || knownEvents.some((event) => (
|
||||||
|
event.eventType === "confirmation_required" || event.eventType === "confirmation_received"
|
||||||
|
));
|
||||||
|
const confirmationWasReceived = knownEvents.some((event) => event.eventType === "confirmation_received")
|
||||||
|
|| (run?.state === "succeeded" && confirmationWasRequired);
|
||||||
|
const furthestEventPhase = knownEvents.reduce((furthest, event) => {
|
||||||
|
const phase = phaseEnteredByEvent[event.eventType];
|
||||||
|
return phase ? Math.max(furthest, phases.indexOf(phase)) : furthest;
|
||||||
|
}, -1);
|
||||||
|
const confirmationPhase = phases.indexOf("awaiting_confirmation");
|
||||||
|
const passedConfirmation = run?.state === "succeeded"
|
||||||
|
|| (run && !terminal(run) ? currentPhase > confirmationPhase : furthestEventPhase > confirmationPhase);
|
||||||
|
const deletionCount = useMemo(() => run?.plannedDiff
|
||||||
|
? run.plannedDiff.deletedTables.length + run.plannedDiff.deletedColumns.length + run.plannedDiff.deletedRelationships.length
|
||||||
|
: 0, [run]);
|
||||||
|
|
||||||
|
const perform = async (kind: "confirm" | "cancel" | "retry") => {
|
||||||
|
if (!run) return;
|
||||||
|
setAction(kind);
|
||||||
|
try {
|
||||||
|
const next = kind === "confirm"
|
||||||
|
? await confirmCatalogSync(run.id, run.confirmationToken!)
|
||||||
|
: kind === "cancel"
|
||||||
|
? await cancelCatalogSync(run.id)
|
||||||
|
: await retryCatalogSync(run.id);
|
||||||
|
onRunChange(next);
|
||||||
|
queryClient.setQueryData(["catalog-sync-run", next.id], next);
|
||||||
|
await historyQuery.refetch();
|
||||||
|
} catch (error) {
|
||||||
|
toast.error(apiErrorMessage(error));
|
||||||
|
} finally {
|
||||||
|
setAction(null);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
if (!open) return null;
|
||||||
|
return (
|
||||||
|
<aside
|
||||||
|
aria-label="Schema synchronization"
|
||||||
|
className="fixed inset-y-2 right-0 z-40 flex w-full max-w-[440px] flex-col border-l border-border bg-background shadow-2xl sm:inset-y-4"
|
||||||
|
>
|
||||||
|
<div className="flex items-start justify-between gap-4 border-b border-border px-5 py-4">
|
||||||
|
<div>
|
||||||
|
<p className="thot-label">Schema synchronization</p>
|
||||||
|
<h2 className="mt-1 font-heading text-xl font-semibold">{run ? `${stateLabel(run)[0].toUpperCase()}${stateLabel(run).slice(1)}` : "Loading"}</h2>
|
||||||
|
{run ? <p className="mt-1 text-sm text-muted-foreground">{run.scope} · {elapsed(run, now)}</p> : null}
|
||||||
|
</div>
|
||||||
|
<Button type="button" variant="ghost" size="icon-lg" aria-label="Close synchronization drawer" onClick={onClose}>
|
||||||
|
<X aria-hidden="true" />
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="min-h-0 flex-1 overflow-y-auto px-5 py-5">
|
||||||
|
{runQuery.isLoading ? <p className="text-sm text-muted-foreground">Loading synchronization…</p> : null}
|
||||||
|
{run ? (
|
||||||
|
<>
|
||||||
|
<section aria-labelledby="synchronization-steps-heading">
|
||||||
|
<h3 id="synchronization-steps-heading" className="thot-label mb-3">Synchronization steps</h3>
|
||||||
|
<ol className="grid grid-cols-2 gap-x-4 gap-y-2.5 sm:grid-cols-3" aria-label="Synchronization steps">
|
||||||
|
{phases.map((phase, index) => {
|
||||||
|
const confirmationUnknown = phase === "awaiting_confirmation"
|
||||||
|
&& passedConfirmation
|
||||||
|
&& !eventQuery.isFetched
|
||||||
|
&& !confirmationWasRequired;
|
||||||
|
const confirmationSkipped = phase === "awaiting_confirmation"
|
||||||
|
&& passedConfirmation
|
||||||
|
&& eventQuery.isFetched
|
||||||
|
&& !confirmationWasRequired;
|
||||||
|
const terminalOutcome = phase === "completed" && terminal(run) && run.state !== "succeeded";
|
||||||
|
const failedOutcome = terminalOutcome && run.state === "failed";
|
||||||
|
const interruptedOutcome = terminalOutcome && run.state === "interrupted";
|
||||||
|
const priorPhaseCompleted = terminal(run)
|
||||||
|
? index < furthestEventPhase
|
||||||
|
: index < currentPhase;
|
||||||
|
const complete = !confirmationUnknown && !confirmationSkipped && !terminalOutcome
|
||||||
|
&& (phase === "awaiting_confirmation"
|
||||||
|
? confirmationWasReceived
|
||||||
|
: run.state === "succeeded" || priorPhaseCompleted);
|
||||||
|
const current = index === currentPhase && !terminal(run);
|
||||||
|
const label = phase === "awaiting_confirmation"
|
||||||
|
? confirmationSkipped
|
||||||
|
? "Confirmation not required"
|
||||||
|
: confirmationUnknown
|
||||||
|
? "Checking confirmation"
|
||||||
|
: confirmationWasReceived
|
||||||
|
? phaseLabels[phase].completed
|
||||||
|
: current
|
||||||
|
? phaseLabels[phase].current
|
||||||
|
: complete
|
||||||
|
? phaseLabels[phase].completed
|
||||||
|
: phaseLabels[phase].pending
|
||||||
|
: terminalOutcome
|
||||||
|
? `${stateLabel(run)[0].toUpperCase()}${stateLabel(run).slice(1)}`
|
||||||
|
: complete
|
||||||
|
? phaseLabels[phase].completed
|
||||||
|
: current
|
||||||
|
? phaseLabels[phase].current
|
||||||
|
: phaseLabels[phase].pending;
|
||||||
|
const status = terminalOutcome
|
||||||
|
? run.state
|
||||||
|
: confirmationSkipped
|
||||||
|
? "not required"
|
||||||
|
: complete
|
||||||
|
? "completed"
|
||||||
|
: current
|
||||||
|
? "in progress"
|
||||||
|
: "pending";
|
||||||
|
return (
|
||||||
|
<li
|
||||||
|
key={phase}
|
||||||
|
aria-current={current ? "step" : undefined}
|
||||||
|
aria-label={`${label}, ${status}`}
|
||||||
|
className={`flex items-center gap-2 text-xs ${failedOutcome ? "font-semibold text-destructive" : interruptedOutcome ? "font-semibold text-amber-600" : terminalOutcome || current ? "font-semibold text-foreground" : "text-muted-foreground"}`}
|
||||||
|
>
|
||||||
|
{confirmationSkipped ? (
|
||||||
|
<Minus aria-hidden="true" className="size-3.5 text-muted-foreground" />
|
||||||
|
) : failedOutcome ? (
|
||||||
|
<X aria-hidden="true" className="size-3.5 text-destructive" />
|
||||||
|
) : interruptedOutcome ? (
|
||||||
|
<AlertTriangle aria-hidden="true" className="size-3.5 text-amber-600" />
|
||||||
|
) : terminalOutcome ? (
|
||||||
|
<Minus aria-hidden="true" className="size-3.5 text-muted-foreground" />
|
||||||
|
) : complete ? (
|
||||||
|
<CheckCircle2 aria-hidden="true" className="size-3.5 text-[oklch(var(--success))]" />
|
||||||
|
) : current ? (
|
||||||
|
<LoaderCircle aria-hidden="true" className="size-3.5 animate-spin text-amber-600" />
|
||||||
|
) : (
|
||||||
|
<Circle aria-hidden="true" className="size-3.5" />
|
||||||
|
)}
|
||||||
|
{label}
|
||||||
|
</li>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
</ol>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<div className="mt-5 grid grid-cols-3 gap-2">
|
||||||
|
{(["tables", "columns", "relationships"] as const).map((name) => (
|
||||||
|
<div key={name} className="rounded-md border border-border bg-muted/25 px-3 py-2">
|
||||||
|
<p className="text-[11px] font-semibold uppercase tracking-wide text-muted-foreground">{name}</p>
|
||||||
|
<p className="mt-1 text-lg font-semibold tabular-nums">{run.counts[name] ?? ""}</p>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{run.state === "awaiting_confirmation" && run.plannedDiff ? (
|
||||||
|
<section className="mt-5 rounded-md border border-amber-500/50 bg-amber-500/8 p-4" aria-label="Destructive changes">
|
||||||
|
<div className="flex gap-3">
|
||||||
|
<AlertTriangle className="mt-0.5 size-5 shrink-0 text-amber-600" />
|
||||||
|
<div>
|
||||||
|
<h3 className="font-semibold">Confirm {deletionCount} removals</h3>
|
||||||
|
<p className="mt-1 text-sm text-muted-foreground">The database will be scanned again before anything is removed.</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<ul className="mt-3 max-h-40 space-y-1 overflow-y-auto font-mono text-xs">
|
||||||
|
{run.plannedDiff.deletedTables.map((name) => <li key={`t-${name}`}>table · {name}</li>)}
|
||||||
|
{run.plannedDiff.deletedColumns.map((item) => <li key={`c-${item.tableName}-${item.columnName}`}>column · {item.tableName}.{item.columnName}</li>)}
|
||||||
|
{run.plannedDiff.deletedRelationships.map((item) => <li key={`r-${item.sourceTableName}-${item.constraintName}`}>relationship · {item.sourceTableName}.{item.constraintName}</li>)}
|
||||||
|
</ul>
|
||||||
|
</section>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{run.errorMessage ? (
|
||||||
|
<div className="mt-5 rounded-md border border-destructive/35 bg-destructive/5 px-4 py-3 text-sm text-destructive">
|
||||||
|
{run.errorMessage}
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
<section className="mt-5" aria-label="Synchronization log">
|
||||||
|
<div className="mb-2 flex items-center justify-between">
|
||||||
|
<h3 className="thot-label">Live log</h3>
|
||||||
|
<span className="text-xs tabular-nums text-muted-foreground">{events.length}</span>
|
||||||
|
</div>
|
||||||
|
<div className="max-h-64 overflow-y-auto rounded-md bg-zinc-950 p-3 font-mono text-xs leading-5 text-zinc-200">
|
||||||
|
{events.length === 0 ? <p className="text-zinc-500">Waiting for events…</p> : events.map((event) => (
|
||||||
|
<p key={event.sequence} className={event.level === "error" ? "text-red-300" : event.level === "warning" ? "text-amber-300" : undefined}>
|
||||||
|
<span className="mr-2 text-zinc-500">{new Date(event.createdAt).toLocaleTimeString()}</span>{event.message}
|
||||||
|
</p>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<section className="mt-5" aria-label="Synchronization history">
|
||||||
|
<h3 className="thot-label mb-2">Recent runs</h3>
|
||||||
|
<div className="divide-y divide-border rounded-md border border-border">
|
||||||
|
{(historyQuery.data ?? []).slice(0, 5).map((item) => (
|
||||||
|
<button key={item.id} type="button" className="flex w-full items-center justify-between gap-3 px-3 py-2 text-left text-sm hover:bg-muted/50" onClick={() => onRunChange(item)}>
|
||||||
|
<span>{item.scope}</span>
|
||||||
|
<span className="text-xs text-muted-foreground">{stateLabel(item)} · {new Date(item.createdAt).toLocaleString()}</span>
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
</>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{run ? (
|
||||||
|
<div className="flex justify-end gap-2 border-t border-border px-5 py-4">
|
||||||
|
{run.state === "awaiting_confirmation" ? (
|
||||||
|
<Button type="button" disabled={action !== null} onClick={() => void perform("confirm")}>
|
||||||
|
{action === "confirm" ? <LoaderCircle className="animate-spin" /> : <Check />} Confirm removals
|
||||||
|
</Button>
|
||||||
|
) : null}
|
||||||
|
{["queued", "running", "awaiting_confirmation"].includes(run.state) ? (
|
||||||
|
<Button type="button" variant="outline" disabled={action !== null} onClick={() => void perform("cancel")}>Cancel</Button>
|
||||||
|
) : null}
|
||||||
|
{["failed", "cancelled", "interrupted"].includes(run.state) ? (
|
||||||
|
<Button type="button" disabled={action !== null} onClick={() => void perform("retry")}>
|
||||||
|
<RotateCcw className={action === "retry" ? "animate-spin" : ""} /> Retry
|
||||||
|
</Button>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
</aside>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,192 @@
|
|||||||
|
import { useEffect, useMemo, useRef, useState } from "react";
|
||||||
|
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||||
|
import { AgGridReact } from "ag-grid-react";
|
||||||
|
import type { ColDef, ICellRendererParams } from "ag-grid-community";
|
||||||
|
import { KeyRound, Link2, Pencil, RefreshCw, Save } from "lucide-react";
|
||||||
|
import { toast } from "sonner";
|
||||||
|
import { Button } from "../../components/ui/button";
|
||||||
|
import { ApiError, apiErrorMessage } from "../../api/client";
|
||||||
|
import {
|
||||||
|
listCatalogColumns,
|
||||||
|
updateCatalogColumnMetadata,
|
||||||
|
type CatalogColumn,
|
||||||
|
type CatalogTable,
|
||||||
|
} from "../../api/catalog-databases";
|
||||||
|
import type { DatabaseNavigationState } from "./model";
|
||||||
|
|
||||||
|
interface Props {
|
||||||
|
databaseId: string;
|
||||||
|
table: CatalogTable;
|
||||||
|
canManage: boolean;
|
||||||
|
onNavigationStateChange: (state: DatabaseNavigationState) => void;
|
||||||
|
onSync: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface GridContext {
|
||||||
|
canManage: boolean;
|
||||||
|
onEdit: (column: CatalogColumn, origin: HTMLButtonElement) => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
function KeyCell({ data }: ICellRendererParams<CatalogColumn>) {
|
||||||
|
if (!data) return null;
|
||||||
|
return (
|
||||||
|
<div className="flex h-full items-center gap-1.5">
|
||||||
|
{data.isPrimaryKey ? <span className="inline-flex items-center gap-1 rounded bg-primary/10 px-1.5 py-0.5 text-xs font-semibold text-primary"><KeyRound className="size-3" />PK{data.primaryKeyPosition && data.primaryKeyPosition > 1 ? ` ${data.primaryKeyPosition}` : ""}</span> : null}
|
||||||
|
{data.isForeignKey ? <span className="inline-flex items-center gap-1 rounded bg-muted px-1.5 py-0.5 text-xs font-semibold text-muted-foreground"><Link2 className="size-3" />FK{data.foreignKeyCount > 1 ? ` ${data.foreignKeyCount}` : ""}</span> : null}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function ActionCell({ data, context }: ICellRendererParams<CatalogColumn, unknown, GridContext>) {
|
||||||
|
if (!data || !context) return null;
|
||||||
|
return (
|
||||||
|
<div className="flex h-full items-center justify-end" onClick={(event) => event.stopPropagation()}>
|
||||||
|
<Button type="button" variant="ghost" size="icon-lg" disabled={!context.canManage} aria-label={`Edit metadata for ${data.name}`} onClick={(event) => context.onEdit(data, event.currentTarget)}>
|
||||||
|
<Pencil aria-hidden="true" />
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function DatabaseColumns({ databaseId, table, canManage, onNavigationStateChange, onSync }: Props) {
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
const queryKey = ["catalog-columns", databaseId, table.id] as const;
|
||||||
|
const { data = [], isLoading, isFetching, refetch } = useQuery({
|
||||||
|
queryKey,
|
||||||
|
queryFn: () => listCatalogColumns(databaseId, table.id),
|
||||||
|
retry: false,
|
||||||
|
});
|
||||||
|
const [search, setSearch] = useState("");
|
||||||
|
const [editingId, setEditingId] = useState<string | null>(null);
|
||||||
|
const [description, setDescription] = useState("");
|
||||||
|
const [generatedDescription, setGeneratedDescription] = useState("");
|
||||||
|
const [baseline, setBaseline] = useState("");
|
||||||
|
const [version, setVersion] = useState<number | null>(null);
|
||||||
|
const [stale, setStale] = useState(false);
|
||||||
|
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const originRef = useRef<HTMLButtonElement | null>(null);
|
||||||
|
const active = editingId ? data.find((column) => column.id === editingId) : undefined;
|
||||||
|
const fingerprint = JSON.stringify([description, generatedDescription]);
|
||||||
|
const dirty = Boolean(editingId && fingerprint !== baseline);
|
||||||
|
|
||||||
|
useEffect(() => { onNavigationStateChange({ dirty, busy }); }, [busy, dirty, onNavigationStateChange]);
|
||||||
|
useEffect(() => {
|
||||||
|
if (!editingId || !active || version === active.version || busy) return;
|
||||||
|
setStale(true);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
}, [active, busy, editingId, version]);
|
||||||
|
|
||||||
|
const edit = (column: CatalogColumn, origin: HTMLButtonElement) => {
|
||||||
|
originRef.current = origin;
|
||||||
|
setEditingId(column.id);
|
||||||
|
setDescription(column.description ?? "");
|
||||||
|
setGeneratedDescription(column.generatedDescription ?? "");
|
||||||
|
setBaseline(JSON.stringify([column.description ?? "", column.generatedDescription ?? ""]));
|
||||||
|
setVersion(column.version);
|
||||||
|
setStale(false);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
};
|
||||||
|
const closeEditor = () => {
|
||||||
|
if (busy) return;
|
||||||
|
if (dirty && !window.confirm("Discard unsaved column metadata?")) return;
|
||||||
|
setEditingId(null);
|
||||||
|
window.setTimeout(() => originRef.current?.focus(), 0);
|
||||||
|
};
|
||||||
|
const save = async () => {
|
||||||
|
if (!active || version === null) return;
|
||||||
|
setBusy(true);
|
||||||
|
try {
|
||||||
|
const updated = await updateCatalogColumnMetadata(
|
||||||
|
databaseId, table.id, active.id, version,
|
||||||
|
description.trim() || null, generatedDescription.trim() || null,
|
||||||
|
);
|
||||||
|
queryClient.setQueryData<CatalogColumn[]>(queryKey, (current = []) => current.map((column) => column.id === updated.id ? updated : column));
|
||||||
|
setDescription(updated.description ?? "");
|
||||||
|
setGeneratedDescription(updated.generatedDescription ?? "");
|
||||||
|
setBaseline(JSON.stringify([updated.description ?? "", updated.generatedDescription ?? ""]));
|
||||||
|
setVersion(updated.version);
|
||||||
|
setStale(false);
|
||||||
|
toast.success("Column metadata saved");
|
||||||
|
} catch (error) {
|
||||||
|
if (error instanceof ApiError && error.code === "column_stale") {
|
||||||
|
await refetch();
|
||||||
|
setStale(true);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
} else toast.error(apiErrorMessage(error));
|
||||||
|
} finally { setBusy(false); }
|
||||||
|
};
|
||||||
|
const reloadColumn = async () => {
|
||||||
|
if (!editingId) return;
|
||||||
|
setBusy(true);
|
||||||
|
try {
|
||||||
|
const result = await refetch();
|
||||||
|
const latest = result.data?.find((column) => column.id === editingId);
|
||||||
|
if (!latest) { closeEditor(); return; }
|
||||||
|
setDescription(latest.description ?? "");
|
||||||
|
setGeneratedDescription(latest.generatedDescription ?? "");
|
||||||
|
setBaseline(JSON.stringify([latest.description ?? "", latest.generatedDescription ?? ""]));
|
||||||
|
setVersion(latest.version);
|
||||||
|
setStale(false);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(false); }
|
||||||
|
};
|
||||||
|
|
||||||
|
const columns = useMemo<ColDef<CatalogColumn>[]>(() => [
|
||||||
|
{ field: "ordinalPosition", headerName: "#", width: 64, maxWidth: 64, filter: "agNumberColumnFilter" },
|
||||||
|
{ field: "name", headerName: "Name", minWidth: 190, flex: 1, cellClass: "font-mono text-xs" },
|
||||||
|
{ field: "dataType", headerName: "Type", minWidth: 150, flex: 0.8, cellClass: "font-mono text-xs" },
|
||||||
|
{ headerName: "Keys", minWidth: 125, width: 125, sortable: false, filter: false, cellRenderer: KeyCell },
|
||||||
|
{ headerName: "Nullable", minWidth: 100, width: 100, valueGetter: ({ data: row }) => row ? (row.isNullable ? "Yes" : "No") : "" },
|
||||||
|
{ field: "sourceComment", headerName: "Source comment", minWidth: 220, flex: 1.2, valueFormatter: ({ value }) => value ?? "" },
|
||||||
|
{ field: "generatedDescription", headerName: "Generated description", minWidth: 230, flex: 1.2, valueFormatter: ({ value }) => value ?? "" },
|
||||||
|
{ field: "description", headerName: "Description", minWidth: 230, flex: 1.2, valueFormatter: ({ value }) => value ?? "" },
|
||||||
|
{ colId: "actions", headerName: "", width: 64, maxWidth: 64, pinned: "right", sortable: false, filter: false, resizable: false, cellRenderer: ActionCell },
|
||||||
|
], []);
|
||||||
|
const context = useMemo<GridContext>(() => ({ canManage, onEdit: edit }), [canManage, data]);
|
||||||
|
|
||||||
|
if (editingId && active) {
|
||||||
|
return (
|
||||||
|
<div className="mx-auto w-full max-w-4xl px-4 py-6 sm:px-5">
|
||||||
|
<Button type="button" variant="ghost" className="-ml-2 mb-4" disabled={busy} onClick={closeEditor}>← Back to columns</Button>
|
||||||
|
<p className="thot-label">{table.name} · column</p>
|
||||||
|
<h3 className="mt-1 font-heading text-2xl font-semibold">{active.name}</h3>
|
||||||
|
<p className="mt-1 text-sm text-muted-foreground">Physical schema fields are read-only. Review fields can be edited.</p>
|
||||||
|
{stale ? (
|
||||||
|
staleBannerOpen ? <div className="mt-4 rounded-md border border-amber-500/50 bg-amber-500/8 p-4 text-sm"><p className="font-semibold">Newer column metadata is available.</p><p className="mt-1 text-muted-foreground">Your draft is preserved until you reload.</p><div className="mt-3 flex gap-2"><Button type="button" variant="outline" onClick={() => void reloadColumn()}>Reload latest</Button><Button type="button" variant="ghost" onClick={() => setStaleBannerOpen(false)}>Keep editing</Button></div></div>
|
||||||
|
: <div className="mt-4 flex items-center justify-between gap-3 rounded-md border border-amber-500/40 bg-amber-500/8 px-4 py-3 text-sm"><span>Reload the latest value before this metadata can be saved.</span><Button type="button" variant="outline" onClick={() => void reloadColumn()}>Reload latest</Button></div>
|
||||||
|
) : null}
|
||||||
|
<div className="mt-5 grid gap-4 border-t border-border pt-5 sm:grid-cols-2">
|
||||||
|
<label className="grid gap-1.5 text-sm font-semibold">Physical column name<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={active.name} readOnly /></label>
|
||||||
|
<label className="grid gap-1.5 text-sm font-semibold">Native type<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={active.dataType} readOnly /></label>
|
||||||
|
<label className="grid gap-1.5 text-sm font-semibold">Ordinal position<input className="h-9 rounded-md border border-input bg-muted/35 px-3" value={active.ordinalPosition} readOnly /></label>
|
||||||
|
<label className="grid gap-1.5 text-sm font-semibold">Default expression<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={active.defaultExpression ?? ""} readOnly /></label>
|
||||||
|
<label className="grid gap-1.5 text-sm font-semibold sm:col-span-2">Source comment<textarea className="min-h-24 rounded-md border border-input bg-muted/35 px-3 py-2 text-sm" value={active.sourceComment ?? ""} readOnly /></label>
|
||||||
|
<label className="grid gap-1.5 text-sm font-semibold sm:col-span-2">Generated description<textarea className="min-h-28 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={generatedDescription} disabled={!canManage || busy} onChange={(event) => setGeneratedDescription(event.target.value)} /></label>
|
||||||
|
<label className="grid gap-1.5 text-sm font-semibold sm:col-span-2">Description<textarea className="min-h-32 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={description} disabled={!canManage || busy} onChange={(event) => setDescription(event.target.value)} /></label>
|
||||||
|
</div>
|
||||||
|
<div className="mt-5 flex justify-end gap-2 border-t border-border pt-4">
|
||||||
|
<Button type="button" variant="outline" disabled={busy} onClick={closeEditor}>Cancel</Button>
|
||||||
|
<Button type="button" disabled={!canManage || !dirty || busy || stale} onClick={() => void save()}><Save />{busy ? "Saving…" : "Save metadata"}</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="flex min-h-0 flex-1 flex-col">
|
||||||
|
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
|
||||||
|
<span className="thot-label whitespace-nowrap">Catalog columns</span>
|
||||||
|
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search columns" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
|
||||||
|
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
|
||||||
|
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
|
||||||
|
<Button type="button" disabled={!canManage || busy} onClick={onSync}><RefreshCw />Sync columns</Button>
|
||||||
|
</div>
|
||||||
|
<div className="relative min-h-[280px] flex-1">
|
||||||
|
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
|
||||||
|
<AgGridReact<CatalogColumn> rowData={data} columnDefs={columns} context={context} loading={isLoading} quickFilterText={search} defaultColDef={{ sortable: true, filter: true, resizable: true }} getRowId={({ data: row }) => row.id} rowHeight={44} headerHeight={38} animateRows={false} overlayNoRowsTemplate="No columns synchronized for this table." />
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,510 @@
|
|||||||
|
import type { RefObject, ReactNode } from "react";
|
||||||
|
import {
|
||||||
|
ArrowLeft,
|
||||||
|
Database,
|
||||||
|
RefreshCw,
|
||||||
|
Save,
|
||||||
|
TestTube2,
|
||||||
|
Trash2,
|
||||||
|
} from "lucide-react";
|
||||||
|
import { Button } from "../../components/ui/button";
|
||||||
|
import type {
|
||||||
|
CatalogDatabase,
|
||||||
|
CatalogSecretName,
|
||||||
|
CatalogSyncScope,
|
||||||
|
CatalogSyncRun,
|
||||||
|
DatabaseBinding,
|
||||||
|
DatabaseTransport,
|
||||||
|
} from "../../api/catalog-databases";
|
||||||
|
import type {
|
||||||
|
DatabaseBusyAction,
|
||||||
|
DatabaseFormDraft,
|
||||||
|
DatabaseFormMode,
|
||||||
|
} from "./model";
|
||||||
|
import { statusLabel } from "./model";
|
||||||
|
import { DatabaseSyncMenu } from "./DatabaseSyncMenu";
|
||||||
|
|
||||||
|
const inputClass = "h-9 w-full rounded-md border border-input bg-card px-3 text-sm outline-none transition focus:border-primary/60 focus:ring-3 focus:ring-ring/15 read-only:bg-muted/40 read-only:text-muted-foreground disabled:bg-muted disabled:text-muted-foreground";
|
||||||
|
const labelClass = "grid min-w-0 gap-1.5 text-xs font-medium text-foreground";
|
||||||
|
|
||||||
|
interface FieldProps {
|
||||||
|
label: string;
|
||||||
|
hint?: string;
|
||||||
|
children: ReactNode;
|
||||||
|
}
|
||||||
|
|
||||||
|
function Field({ label, hint, children }: FieldProps) {
|
||||||
|
return (
|
||||||
|
<label className={labelClass}>
|
||||||
|
<span>{label}</span>
|
||||||
|
{children}
|
||||||
|
{hint ? <span className="font-normal leading-4 text-muted-foreground">{hint}</span> : null}
|
||||||
|
</label>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
interface SecretFieldProps {
|
||||||
|
label: string;
|
||||||
|
name: CatalogSecretName;
|
||||||
|
row: CatalogDatabase;
|
||||||
|
draft: DatabaseFormDraft;
|
||||||
|
readOnly: boolean;
|
||||||
|
canManageSecrets: boolean;
|
||||||
|
onChange: (name: CatalogSecretName, value: string) => void;
|
||||||
|
multiline?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
function SecretField({
|
||||||
|
label,
|
||||||
|
name,
|
||||||
|
row,
|
||||||
|
draft,
|
||||||
|
readOnly,
|
||||||
|
canManageSecrets,
|
||||||
|
onChange,
|
||||||
|
multiline = false,
|
||||||
|
}: SecretFieldProps) {
|
||||||
|
if (readOnly) {
|
||||||
|
return (
|
||||||
|
<Field label={label} hint="Secret values are never displayed.">
|
||||||
|
<input
|
||||||
|
className={inputClass}
|
||||||
|
value={row.secrets[name] ? "Configured" : "Not configured"}
|
||||||
|
readOnly
|
||||||
|
aria-label={label}
|
||||||
|
/>
|
||||||
|
</Field>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const permissionHint = canManageSecrets
|
||||||
|
? undefined
|
||||||
|
: "You do not have permission to replace secret values.";
|
||||||
|
|
||||||
|
return (
|
||||||
|
<Field label={label} hint={permissionHint}>
|
||||||
|
{multiline ? (
|
||||||
|
<textarea
|
||||||
|
className={`${inputClass} min-h-24 resize-y py-2 font-mono text-xs leading-5`}
|
||||||
|
aria-label={label}
|
||||||
|
autoComplete="off"
|
||||||
|
value={draft.secrets[name] ?? ""}
|
||||||
|
disabled={!canManageSecrets}
|
||||||
|
placeholder={row.secrets[name] ? "Configured; leave blank to keep" : "Paste value"}
|
||||||
|
onChange={(event) => onChange(name, event.target.value)}
|
||||||
|
/>
|
||||||
|
) : (
|
||||||
|
<input
|
||||||
|
className={inputClass}
|
||||||
|
type="password"
|
||||||
|
aria-label={label}
|
||||||
|
autoComplete="new-password"
|
||||||
|
value={draft.secrets[name] ?? ""}
|
||||||
|
disabled={!canManageSecrets}
|
||||||
|
placeholder={row.secrets[name] ? "Configured; leave blank to keep" : "Not configured"}
|
||||||
|
onChange={(event) => onChange(name, event.target.value)}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
</Field>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
interface DatabaseFormProps {
|
||||||
|
mode: DatabaseFormMode;
|
||||||
|
row: CatalogDatabase;
|
||||||
|
availableWorkspaces: CatalogDatabase[];
|
||||||
|
workspaceLocked: boolean;
|
||||||
|
draft: DatabaseFormDraft;
|
||||||
|
dirty: boolean;
|
||||||
|
canManage: boolean;
|
||||||
|
canManageSecrets: boolean;
|
||||||
|
busyAction: DatabaseBusyAction;
|
||||||
|
stale: boolean;
|
||||||
|
staleBannerOpen: boolean;
|
||||||
|
partialSecretFailure: string | null;
|
||||||
|
headingRef: RefObject<HTMLHeadingElement>;
|
||||||
|
onBack: () => void;
|
||||||
|
onWorkspaceChange: (workspaceId: string) => void;
|
||||||
|
onFieldChange: (field: "databaseName" | "schema", value: string) => void;
|
||||||
|
onTransportChange: (transport: DatabaseTransport) => void;
|
||||||
|
onBindingChange: <K extends keyof DatabaseBinding>(key: K, value: DatabaseBinding[K]) => void;
|
||||||
|
onSecretChange: (name: CatalogSecretName, value: string) => void;
|
||||||
|
onSave: () => void;
|
||||||
|
onTest: () => void;
|
||||||
|
onDelete: () => void;
|
||||||
|
onReloadLatest: () => void;
|
||||||
|
onKeepEditing: () => void;
|
||||||
|
onRetrySecrets: () => void;
|
||||||
|
onOpenTables: (origin: HTMLButtonElement) => void;
|
||||||
|
onOpenRelationships: () => void;
|
||||||
|
onSync: (scope: CatalogSyncScope) => void;
|
||||||
|
onOpenSync: () => void;
|
||||||
|
activeSyncRun?: CatalogSyncRun;
|
||||||
|
}
|
||||||
|
|
||||||
|
function formTitle(mode: DatabaseFormMode, workspaceLocked: boolean): string {
|
||||||
|
if (mode === "add") return workspaceLocked ? "Edit database" : "Add database";
|
||||||
|
if (mode === "edit") return "Edit database";
|
||||||
|
if (mode === "delete") return "Delete database";
|
||||||
|
return "Database details";
|
||||||
|
}
|
||||||
|
|
||||||
|
export function DatabaseForm({
|
||||||
|
mode,
|
||||||
|
row,
|
||||||
|
availableWorkspaces,
|
||||||
|
workspaceLocked,
|
||||||
|
draft,
|
||||||
|
dirty,
|
||||||
|
canManage,
|
||||||
|
canManageSecrets,
|
||||||
|
busyAction,
|
||||||
|
stale,
|
||||||
|
staleBannerOpen,
|
||||||
|
partialSecretFailure,
|
||||||
|
headingRef,
|
||||||
|
onBack,
|
||||||
|
onWorkspaceChange,
|
||||||
|
onFieldChange,
|
||||||
|
onTransportChange,
|
||||||
|
onBindingChange,
|
||||||
|
onSecretChange,
|
||||||
|
onSave,
|
||||||
|
onTest,
|
||||||
|
onDelete,
|
||||||
|
onReloadLatest,
|
||||||
|
onKeepEditing,
|
||||||
|
onRetrySecrets,
|
||||||
|
onOpenTables,
|
||||||
|
onOpenRelationships,
|
||||||
|
onSync,
|
||||||
|
onOpenSync,
|
||||||
|
activeSyncRun,
|
||||||
|
}: DatabaseFormProps) {
|
||||||
|
const title = formTitle(mode, workspaceLocked);
|
||||||
|
const readOnly = mode === "view" || mode === "delete";
|
||||||
|
const editable = mode === "add" || mode === "edit";
|
||||||
|
const busy = busyAction !== null;
|
||||||
|
const testDisabled = !canManage
|
||||||
|
|| !row.configured
|
||||||
|
|| !row.id
|
||||||
|
|| dirty
|
||||||
|
|| stale
|
||||||
|
|| busy;
|
||||||
|
const saveDisabled = !canManage
|
||||||
|
|| stale
|
||||||
|
|| busy
|
||||||
|
|| (mode === "edit" && !dirty);
|
||||||
|
const bindingReady = row.connectionStatus === "reachable" && row.testedVersion === row.version;
|
||||||
|
|
||||||
|
const form = (
|
||||||
|
<form
|
||||||
|
className="grid gap-5"
|
||||||
|
onSubmit={(event) => {
|
||||||
|
event.preventDefault();
|
||||||
|
if (editable) onSave();
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
{mode === "delete" ? (
|
||||||
|
<div className="rounded-md border border-destructive/35 bg-destructive/8 p-4 text-sm text-foreground" role="alert">
|
||||||
|
<p className="font-semibold text-destructive">This removes the local database configuration.</p>
|
||||||
|
<p className="mt-1 leading-5 text-muted-foreground">
|
||||||
|
{row.workspaceAvailable
|
||||||
|
? "The repository workspace remains available and will return to the list as Not configured. "
|
||||||
|
: "This orphaned catalog row will disappear from the list. "}
|
||||||
|
Secret references associated with this configuration are removed too.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{stale ? (
|
||||||
|
staleBannerOpen ? (
|
||||||
|
<div className="rounded-md border border-amber-300/70 bg-amber-50 p-4 text-sm text-amber-950 dark:border-amber-700 dark:bg-amber-950/30 dark:text-amber-100" role="alert">
|
||||||
|
<p className="font-semibold">A newer database configuration is available.</p>
|
||||||
|
<p className="mt-1 leading-5">Reload the latest values before saving, testing, or deleting. Your draft is still intact.</p>
|
||||||
|
<div className="mt-3 flex flex-wrap gap-2">
|
||||||
|
<Button type="button" size="sm" variant="outline" disabled={busy} onClick={onReloadLatest}>
|
||||||
|
<RefreshCw /> Reload latest
|
||||||
|
</Button>
|
||||||
|
<Button type="button" size="sm" variant="ghost" disabled={busy} onClick={onKeepEditing}>
|
||||||
|
Keep editing
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<div className="rounded-md border border-amber-300/70 bg-amber-50 px-4 py-3 text-sm text-amber-950 dark:border-amber-700 dark:bg-amber-950/30 dark:text-amber-100" role="status">
|
||||||
|
Reload the latest values before this configuration can be changed.
|
||||||
|
</div>
|
||||||
|
)
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{partialSecretFailure ? (
|
||||||
|
<div className="rounded-md border border-amber-300/70 bg-amber-50 p-4 text-sm text-amber-950 dark:border-amber-700 dark:bg-amber-950/30 dark:text-amber-100" role="alert">
|
||||||
|
<p className="font-semibold">Database configuration saved; secret update could not be confirmed.</p>
|
||||||
|
<p className="mt-1 leading-5">{partialSecretFailure} The values you entered are retained in this form.</p>
|
||||||
|
<Button type="button" size="sm" variant="outline" className="mt-3" disabled={busy || stale} onClick={onRetrySecrets}>
|
||||||
|
<RefreshCw /> {busyAction === "retry-secrets" ? "Retrying…" : "Retry secrets"}
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{editable && !canManageSecrets ? (
|
||||||
|
<div className="rounded-md border border-border bg-muted/45 px-4 py-3 text-sm text-muted-foreground" role="note">
|
||||||
|
Secret status is visible, but replacing secret values requires the workspace.secrets.manage permission.
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{!row.workspaceAvailable ? (
|
||||||
|
<div className="rounded-md border border-destructive/30 bg-destructive/8 px-4 py-3 text-sm text-destructive" role="alert">
|
||||||
|
This configuration references a workspace that is no longer present in the repository YAML.
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
<div className="grid gap-4 md:grid-cols-2">
|
||||||
|
{mode === "add" ? (
|
||||||
|
<Field label="Workspace">
|
||||||
|
<select
|
||||||
|
className={inputClass}
|
||||||
|
aria-label="Workspace"
|
||||||
|
value={draft.workspaceId}
|
||||||
|
disabled={workspaceLocked || busy}
|
||||||
|
onChange={(event) => onWorkspaceChange(event.target.value)}
|
||||||
|
>
|
||||||
|
{availableWorkspaces.map((workspace) => (
|
||||||
|
<option key={workspace.workspaceId} value={workspace.workspaceId}>
|
||||||
|
{workspace.workspaceName}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
</Field>
|
||||||
|
) : (
|
||||||
|
<Field label="Workspace">
|
||||||
|
<input className={inputClass} value={row.workspaceName} readOnly />
|
||||||
|
</Field>
|
||||||
|
)}
|
||||||
|
<Field label="Workspace ID">
|
||||||
|
<input className={`${inputClass} font-mono text-xs`} value={row.workspaceId} readOnly />
|
||||||
|
</Field>
|
||||||
|
<Field label="Database engine">
|
||||||
|
<input className={inputClass} value="PostgreSQL" readOnly />
|
||||||
|
</Field>
|
||||||
|
<Field label="Transport">
|
||||||
|
<select
|
||||||
|
className={inputClass}
|
||||||
|
aria-label="Transport"
|
||||||
|
value={draft.binding.transport}
|
||||||
|
disabled={readOnly || busy}
|
||||||
|
onChange={(event) => onTransportChange(event.target.value as DatabaseTransport)}
|
||||||
|
>
|
||||||
|
<option value="postgres_direct">Direct PostgreSQL</option>
|
||||||
|
<option value="rest_api">REST API</option>
|
||||||
|
<option value="ssh_tunnel">SSH tunnel</option>
|
||||||
|
</select>
|
||||||
|
</Field>
|
||||||
|
<Field label="Database name">
|
||||||
|
<input
|
||||||
|
className={inputClass}
|
||||||
|
required={editable}
|
||||||
|
value={draft.databaseName}
|
||||||
|
readOnly={readOnly}
|
||||||
|
disabled={busy}
|
||||||
|
onChange={(event) => onFieldChange("databaseName", event.target.value)}
|
||||||
|
/>
|
||||||
|
</Field>
|
||||||
|
<Field label="Schema">
|
||||||
|
<input
|
||||||
|
className={inputClass}
|
||||||
|
required={editable}
|
||||||
|
value={draft.schema}
|
||||||
|
readOnly={readOnly}
|
||||||
|
disabled={busy}
|
||||||
|
onChange={(event) => onFieldChange("schema", event.target.value)}
|
||||||
|
/>
|
||||||
|
</Field>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="border-t border-border pt-5">
|
||||||
|
<p className="thot-label mb-4">Installation binding</p>
|
||||||
|
|
||||||
|
{draft.binding.transport === "postgres_direct" ? (
|
||||||
|
<div className="grid gap-4 md:grid-cols-2">
|
||||||
|
<Field label="Host">
|
||||||
|
<input className={inputClass} required={editable} value={draft.binding.host ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("host", event.target.value)} />
|
||||||
|
</Field>
|
||||||
|
<Field label="Port">
|
||||||
|
<input className={inputClass} type="number" min={1} max={65535} required={editable} value={draft.binding.port ?? 5432} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("port", Number(event.target.value))} />
|
||||||
|
</Field>
|
||||||
|
<Field label="Username">
|
||||||
|
<input className={inputClass} required={editable} value={draft.binding.username ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("username", event.target.value)} />
|
||||||
|
</Field>
|
||||||
|
<Field label="TLS server name">
|
||||||
|
<input className={inputClass} value={draft.binding.tlsServername ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("tlsServername", event.target.value || undefined)} />
|
||||||
|
</Field>
|
||||||
|
<SecretField label="Password" name="password" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
|
||||||
|
<SecretField label="TLS CA certificate" name="tlsCa" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{draft.binding.transport === "rest_api" ? (
|
||||||
|
<div className="grid gap-4 md:grid-cols-2">
|
||||||
|
<Field label="Base URL">
|
||||||
|
<input className={inputClass} type="url" required={editable} placeholder="https://dwh.example/api" value={draft.binding.baseUrl ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("baseUrl", event.target.value)} />
|
||||||
|
</Field>
|
||||||
|
<Field label="Diagnostic endpoint" hint="This relative path is fixed by the runtime contract.">
|
||||||
|
<input className={inputClass} value={draft.binding.restPath ?? "/health"} readOnly aria-label="Diagnostic endpoint" />
|
||||||
|
</Field>
|
||||||
|
<Field label="Authentication">
|
||||||
|
<select className={inputClass} aria-label="Authentication" value={draft.binding.restAuth ?? "x-api-key"} disabled={readOnly || busy} onChange={(event) => onBindingChange("restAuth", event.target.value as DatabaseBinding["restAuth"])}>
|
||||||
|
<option value="x-api-key">X-API-Key</option>
|
||||||
|
<option value="bearer">Bearer token</option>
|
||||||
|
<option value="none">None</option>
|
||||||
|
</select>
|
||||||
|
</Field>
|
||||||
|
{draft.binding.restAuth !== "none" ? (
|
||||||
|
<SecretField label="API key" name="apiKey" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{draft.binding.transport === "ssh_tunnel" ? (
|
||||||
|
<div className="grid gap-4 md:grid-cols-2">
|
||||||
|
<div className="rounded-md border border-border bg-muted/45 px-4 py-3 text-sm text-muted-foreground md:col-span-2" role="note">
|
||||||
|
SSH host verification is strict. Provide the trusted known_hosts entry before testing this binding.
|
||||||
|
</div>
|
||||||
|
<Field label="Database username">
|
||||||
|
<input className={inputClass} required={editable} value={draft.binding.username ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("username", event.target.value)} />
|
||||||
|
</Field>
|
||||||
|
<SecretField label="Database password" name="password" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
|
||||||
|
<Field label="SSH host">
|
||||||
|
<input className={inputClass} required={editable} value={draft.binding.sshHost ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshHost", event.target.value)} />
|
||||||
|
</Field>
|
||||||
|
<Field label="SSH port">
|
||||||
|
<input className={inputClass} type="number" min={1} max={65535} required={editable} value={draft.binding.sshPort ?? 22} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshPort", Number(event.target.value))} />
|
||||||
|
</Field>
|
||||||
|
<Field label="SSH username">
|
||||||
|
<input className={inputClass} required={editable} value={draft.binding.sshUsername ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshUsername", event.target.value)} />
|
||||||
|
</Field>
|
||||||
|
<SecretField label="SSH private key" name="sshPrivateKey" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} multiline />
|
||||||
|
<SecretField label="Private key passphrase" name="sshPrivateKeyPassphrase" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} />
|
||||||
|
<SecretField label="SSH known hosts" name="sshKnownHosts" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} multiline />
|
||||||
|
<Field label="Target host">
|
||||||
|
<input className={inputClass} required={editable} value={draft.binding.sshTargetHost ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshTargetHost", event.target.value)} />
|
||||||
|
</Field>
|
||||||
|
<Field label="Target port">
|
||||||
|
<input className={inputClass} type="number" min={1} max={65535} required={editable} value={draft.binding.sshTargetPort ?? 5432} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("sshTargetPort", Number(event.target.value))} />
|
||||||
|
</Field>
|
||||||
|
<Field label="TLS server name">
|
||||||
|
<input className={inputClass} value={draft.binding.tlsServername ?? ""} readOnly={readOnly} disabled={busy} onChange={(event) => onBindingChange("tlsServername", event.target.value || undefined)} />
|
||||||
|
</Field>
|
||||||
|
<SecretField label="TLS CA certificate" name="tlsCa" row={row} draft={draft} readOnly={readOnly} canManageSecrets={canManageSecrets} onChange={onSecretChange} multiline />
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{row.lastErrorMessage ? (
|
||||||
|
<div className="rounded-md border border-destructive/25 bg-destructive/8 px-4 py-3 text-sm text-destructive" role="status">
|
||||||
|
Last connection error: {row.lastErrorMessage}
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
<div className="flex flex-wrap items-center justify-end gap-2 border-t border-border pt-4">
|
||||||
|
{mode === "view" || mode === "edit" ? (
|
||||||
|
<Button type="button" variant="outline" disabled={testDisabled} onClick={onTest}>
|
||||||
|
<TestTube2 /> {busyAction === "test" ? "Testing…" : "Test connection"}
|
||||||
|
</Button>
|
||||||
|
) : null}
|
||||||
|
{mode === "add" || mode === "edit" ? (
|
||||||
|
<Button type="submit" disabled={saveDisabled}>
|
||||||
|
<Save />
|
||||||
|
{busyAction === "save"
|
||||||
|
? "Saving…"
|
||||||
|
: mode === "add"
|
||||||
|
? workspaceLocked ? "Save database" : "Add database"
|
||||||
|
: "Save changes"}
|
||||||
|
</Button>
|
||||||
|
) : null}
|
||||||
|
{mode === "delete" ? (
|
||||||
|
<Button type="button" variant="destructive" disabled={!canManage || !row.configured || !row.id || busy || stale} onClick={onDelete}>
|
||||||
|
<Trash2 /> {busyAction === "delete" ? "Deleting…" : "Delete database"}
|
||||||
|
</Button>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
</form>
|
||||||
|
);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<section aria-label={`${title} form`} className="min-h-0 flex-1 overflow-y-auto bg-background">
|
||||||
|
<div className="mx-auto w-full max-w-[900px] px-4 py-5 sm:px-6 sm:py-6">
|
||||||
|
<Button type="button" variant="ghost" className="-ml-2 mb-5" disabled={busy} onClick={onBack}>
|
||||||
|
<ArrowLeft /> Back to list
|
||||||
|
</Button>
|
||||||
|
|
||||||
|
<div className="mb-6 flex flex-wrap items-start justify-between gap-4 border-b border-border pb-5">
|
||||||
|
<div className="min-w-0">
|
||||||
|
<p className="thot-label mb-1">Database management</p>
|
||||||
|
<h2 ref={headingRef} tabIndex={-1} className="font-heading text-2xl font-semibold tracking-tight outline-none">
|
||||||
|
{title}
|
||||||
|
</h2>
|
||||||
|
<p className="mt-1 text-sm text-muted-foreground">
|
||||||
|
{row.workspaceName} · {row.workspaceId}
|
||||||
|
</p>
|
||||||
|
{row.lastTestedAt ? (
|
||||||
|
<p className="mt-1 text-xs text-muted-foreground">Last tested {new Date(row.lastTestedAt).toLocaleString()}</p>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
<div className="flex flex-wrap items-center justify-end gap-2">
|
||||||
|
<Database className="size-4 text-muted-foreground" aria-hidden="true" />
|
||||||
|
<span className="rounded-full border border-border bg-muted px-2.5 py-1 text-xs font-semibold text-muted-foreground">
|
||||||
|
{statusLabel(row)}
|
||||||
|
</span>
|
||||||
|
{mode === "view" && activeSyncRun ? (
|
||||||
|
<Button type="button" variant="outline" onClick={onOpenSync}><RefreshCw className="animate-spin" />View sync</Button>
|
||||||
|
) : null}
|
||||||
|
{mode === "view" ? (
|
||||||
|
<DatabaseSyncMenu
|
||||||
|
disabled={!canManage || !bindingReady || busy || Boolean(activeSyncRun)}
|
||||||
|
disabledReason={!bindingReady ? "Test the current database binding before synchronizing the schema" : undefined}
|
||||||
|
onSelect={onSync}
|
||||||
|
/>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{mode === "view" ? (
|
||||||
|
<nav aria-label="Database sections" className="mb-6 flex gap-1 border-b border-border" role="tablist">
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
role="tab"
|
||||||
|
aria-selected="true"
|
||||||
|
className="border-b-2 border-primary px-3 py-2 text-sm font-semibold text-foreground"
|
||||||
|
>
|
||||||
|
Overview
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
role="tab"
|
||||||
|
aria-selected="false"
|
||||||
|
disabled={!row.configured || !row.id}
|
||||||
|
className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground transition hover:text-foreground focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/20 disabled:cursor-not-allowed disabled:opacity-45"
|
||||||
|
onClick={(event) => onOpenTables(event.currentTarget)}
|
||||||
|
>
|
||||||
|
Tables
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
role="tab"
|
||||||
|
aria-selected="false"
|
||||||
|
disabled={!row.configured || !row.id}
|
||||||
|
className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground transition hover:text-foreground focus-visible:outline-none focus-visible:ring-3 focus-visible:ring-ring/20 disabled:cursor-not-allowed disabled:opacity-45"
|
||||||
|
onClick={onOpenRelationships}
|
||||||
|
>
|
||||||
|
Relationships
|
||||||
|
</button>
|
||||||
|
</nav>
|
||||||
|
) : null}
|
||||||
|
|
||||||
|
{form}
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,329 @@
|
|||||||
|
import { useEffect, useMemo, useRef, useState, type RefObject } from "react";
|
||||||
|
import { Menu } from "@base-ui/react/menu";
|
||||||
|
import { AgGridReact } from "ag-grid-react";
|
||||||
|
import {
|
||||||
|
AllCommunityModule,
|
||||||
|
ModuleRegistry,
|
||||||
|
type ColDef,
|
||||||
|
type ICellRendererParams,
|
||||||
|
} from "ag-grid-community";
|
||||||
|
import "ag-grid-community/styles/ag-grid.css";
|
||||||
|
import "ag-grid-community/styles/ag-theme-alpine.css";
|
||||||
|
import { ChevronDown, Eye, Pencil, RefreshCw, Trash2, X } from "lucide-react";
|
||||||
|
import { Button } from "../../components/ui/button";
|
||||||
|
import type { CatalogDatabase, CatalogSyncScope } from "../../api/catalog-databases";
|
||||||
|
import { statusLabel } from "./model";
|
||||||
|
import { databaseSyncItemClass, databaseSyncScopes } from "./DatabaseSyncMenu";
|
||||||
|
|
||||||
|
ModuleRegistry.registerModules([AllCommunityModule]);
|
||||||
|
|
||||||
|
interface DatabaseGridProps {
|
||||||
|
rows: CatalogDatabase[];
|
||||||
|
isLoading: boolean;
|
||||||
|
isFetching: boolean;
|
||||||
|
search: string;
|
||||||
|
canManage: boolean;
|
||||||
|
searchInputRef: RefObject<HTMLInputElement>;
|
||||||
|
onSearchChange: (value: string) => void;
|
||||||
|
onView: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
|
||||||
|
onEdit: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
|
||||||
|
onDelete: (row: CatalogDatabase, origin: HTMLButtonElement) => void;
|
||||||
|
onOpenSync: (row: CatalogDatabase) => void;
|
||||||
|
onTestSelected: (rows: CatalogDatabase[]) => Promise<void>;
|
||||||
|
onSyncSelected: (rows: CatalogDatabase[], scope: CatalogSyncScope) => Promise<void>;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface DatabaseGridContext {
|
||||||
|
canManage: boolean;
|
||||||
|
onView: DatabaseGridProps["onView"];
|
||||||
|
onEdit: DatabaseGridProps["onEdit"];
|
||||||
|
onDelete: DatabaseGridProps["onDelete"];
|
||||||
|
onOpenSync: DatabaseGridProps["onOpenSync"];
|
||||||
|
}
|
||||||
|
|
||||||
|
function useCompactViewport(): boolean {
|
||||||
|
const [compact, setCompact] = useState(false);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
if (typeof window.matchMedia !== "function") return;
|
||||||
|
const query = window.matchMedia("(max-width: 767px)");
|
||||||
|
const update = () => setCompact(query.matches);
|
||||||
|
update();
|
||||||
|
query.addEventListener("change", update);
|
||||||
|
return () => query.removeEventListener("change", update);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
return compact;
|
||||||
|
}
|
||||||
|
|
||||||
|
function StatusPill({ row }: { row: CatalogDatabase }) {
|
||||||
|
const tone = !row.workspaceAvailable
|
||||||
|
? "border-destructive/30 bg-destructive/10 text-destructive"
|
||||||
|
: !row.configured
|
||||||
|
? "border-border bg-muted text-muted-foreground"
|
||||||
|
: row.connectionStatus === "reachable"
|
||||||
|
? "border-emerald-300/70 bg-emerald-50 text-emerald-800 dark:bg-emerald-950/30 dark:text-emerald-300"
|
||||||
|
: row.connectionStatus === "failed"
|
||||||
|
? "border-destructive/30 bg-destructive/10 text-destructive"
|
||||||
|
: "border-amber-300/70 bg-amber-50 text-amber-900 dark:bg-amber-950/30 dark:text-amber-300";
|
||||||
|
|
||||||
|
return <div className="flex items-center gap-1.5"><span className={`inline-flex max-w-full truncate rounded-full border px-2 py-0.5 text-[11px] font-semibold ${tone}`}>{statusLabel(row)}</span>{row.activeSyncRun ? <span className="inline-flex rounded-full border border-primary/30 bg-primary/8 px-2 py-0.5 text-[11px] font-semibold text-primary">Syncing</span> : null}</div>;
|
||||||
|
}
|
||||||
|
|
||||||
|
function DatabaseActionsCell({
|
||||||
|
data,
|
||||||
|
context,
|
||||||
|
}: ICellRendererParams<CatalogDatabase, unknown, DatabaseGridContext>) {
|
||||||
|
if (!data || !context) return null;
|
||||||
|
|
||||||
|
const editDisabled = !context.canManage || !data.workspaceAvailable;
|
||||||
|
const deleteDisabled = !context.canManage || !data.configured;
|
||||||
|
const editLabel = `Edit ${data.workspaceName}`;
|
||||||
|
const editReasonId = `database-edit-reason-${data.workspaceId}`;
|
||||||
|
const deleteReasonId = `database-delete-reason-${data.workspaceId}`;
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="flex h-full items-center justify-end gap-0.5" onClick={(event) => event.stopPropagation()}>
|
||||||
|
{data.activeSyncRun ? (
|
||||||
|
<Button type="button" variant="ghost" size="icon-lg" title={`View synchronization for ${data.workspaceName}`} aria-label={`View synchronization for ${data.workspaceName}`} onClick={() => context.onOpenSync(data)}>
|
||||||
|
<RefreshCw className="animate-spin" aria-hidden="true" />
|
||||||
|
</Button>
|
||||||
|
) : null}
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
variant="ghost"
|
||||||
|
size="icon-lg"
|
||||||
|
title={`View ${data.workspaceName}`}
|
||||||
|
aria-label={`View ${data.workspaceName}`}
|
||||||
|
onClick={(event) => context.onView(data, event.currentTarget)}
|
||||||
|
>
|
||||||
|
<Eye aria-hidden="true" />
|
||||||
|
</Button>
|
||||||
|
<span title={editDisabled ? "This database configuration cannot be changed" : editLabel}>
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
variant="ghost"
|
||||||
|
size="icon-lg"
|
||||||
|
aria-label={editLabel}
|
||||||
|
aria-describedby={editDisabled ? editReasonId : undefined}
|
||||||
|
disabled={editDisabled}
|
||||||
|
onClick={(event) => context.onEdit(data, event.currentTarget)}
|
||||||
|
>
|
||||||
|
<Pencil aria-hidden="true" />
|
||||||
|
</Button>
|
||||||
|
{editDisabled ? <span id={editReasonId} className="sr-only">This database configuration cannot be changed.</span> : null}
|
||||||
|
</span>
|
||||||
|
<span title={deleteDisabled ? "Save this database configuration before deleting it" : `Delete ${data.workspaceName}`}>
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
variant="ghost"
|
||||||
|
size="icon-lg"
|
||||||
|
className="text-destructive hover:bg-destructive/10 hover:text-destructive"
|
||||||
|
aria-label={`Delete ${data.workspaceName}`}
|
||||||
|
aria-describedby={deleteDisabled ? deleteReasonId : undefined}
|
||||||
|
disabled={deleteDisabled}
|
||||||
|
onClick={(event) => context.onDelete(data, event.currentTarget)}
|
||||||
|
>
|
||||||
|
<Trash2 aria-hidden="true" />
|
||||||
|
</Button>
|
||||||
|
{deleteDisabled ? <span id={deleteReasonId} className="sr-only">Save this database configuration before deleting it.</span> : null}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function DatabaseGrid({
|
||||||
|
rows,
|
||||||
|
isLoading,
|
||||||
|
isFetching,
|
||||||
|
search,
|
||||||
|
canManage,
|
||||||
|
searchInputRef,
|
||||||
|
onSearchChange,
|
||||||
|
onView,
|
||||||
|
onEdit,
|
||||||
|
onDelete,
|
||||||
|
onOpenSync,
|
||||||
|
onTestSelected,
|
||||||
|
onSyncSelected,
|
||||||
|
}: DatabaseGridProps) {
|
||||||
|
const compact = useCompactViewport();
|
||||||
|
const gridRef = useRef<AgGridReact<CatalogDatabase>>(null);
|
||||||
|
const [selectedRows, setSelectedRows] = useState<CatalogDatabase[]>([]);
|
||||||
|
const [action, setAction] = useState<"test" | "sync" | null>(null);
|
||||||
|
const context = useMemo<DatabaseGridContext>(
|
||||||
|
() => ({ canManage, onView, onEdit, onDelete, onOpenSync }),
|
||||||
|
[canManage, onView, onEdit, onDelete, onOpenSync],
|
||||||
|
);
|
||||||
|
|
||||||
|
const columnDefs = useMemo<ColDef<CatalogDatabase>[]>(() => {
|
||||||
|
const workspace: ColDef<CatalogDatabase> = {
|
||||||
|
field: "workspaceName",
|
||||||
|
headerName: "Workspace",
|
||||||
|
minWidth: compact ? 112 : 190,
|
||||||
|
flex: compact ? undefined : 1.35,
|
||||||
|
width: compact ? 112 : undefined,
|
||||||
|
};
|
||||||
|
const database: ColDef<CatalogDatabase> = {
|
||||||
|
field: "databaseName", headerName: "Database", minWidth: 145, flex: 1,
|
||||||
|
};
|
||||||
|
const schema: ColDef<CatalogDatabase> = {
|
||||||
|
field: "schema", headerName: "Schema", minWidth: 140, flex: 0.9,
|
||||||
|
};
|
||||||
|
const transport: ColDef<CatalogDatabase> = {
|
||||||
|
field: "binding.transport",
|
||||||
|
headerName: "Transport",
|
||||||
|
minWidth: 135,
|
||||||
|
flex: 0.9,
|
||||||
|
valueFormatter: ({ value }) => String(value ?? "").replaceAll("_", " "),
|
||||||
|
};
|
||||||
|
const endpoint: ColDef<CatalogDatabase> = {
|
||||||
|
headerName: "Endpoint",
|
||||||
|
minWidth: 190,
|
||||||
|
flex: 1.2,
|
||||||
|
valueGetter: ({ data }) => {
|
||||||
|
if (!data) return "";
|
||||||
|
if (data.binding.transport === "rest_api") return data.binding.baseUrl ?? "";
|
||||||
|
if (data.binding.transport === "ssh_tunnel") return data.binding.sshHost ?? "";
|
||||||
|
return data.binding.host ? `${data.binding.host}:${data.binding.port ?? 5432}` : "";
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const status: ColDef<CatalogDatabase> = {
|
||||||
|
headerName: "Status",
|
||||||
|
minWidth: compact ? 96 : 130,
|
||||||
|
width: compact ? 96 : undefined,
|
||||||
|
flex: compact ? undefined : 0.8,
|
||||||
|
cellClass: compact ? "thot-database-status-cell" : undefined,
|
||||||
|
valueGetter: ({ data }) => (data ? statusLabel(data) : ""),
|
||||||
|
cellRenderer: ({ data }: ICellRendererParams<CatalogDatabase>) => (
|
||||||
|
data ? <StatusPill row={data} /> : null
|
||||||
|
),
|
||||||
|
};
|
||||||
|
const updated: ColDef<CatalogDatabase> = {
|
||||||
|
field: "updatedAt",
|
||||||
|
headerName: "Updated",
|
||||||
|
minWidth: 170,
|
||||||
|
flex: 0.9,
|
||||||
|
valueFormatter: ({ value }) => value ? new Date(String(value)).toLocaleString() : "",
|
||||||
|
};
|
||||||
|
const actionsWidth = compact ? 148 : 164;
|
||||||
|
const actions: ColDef<CatalogDatabase> = {
|
||||||
|
colId: "actions",
|
||||||
|
headerName: "Actions",
|
||||||
|
width: actionsWidth,
|
||||||
|
minWidth: actionsWidth,
|
||||||
|
maxWidth: actionsWidth,
|
||||||
|
pinned: "right",
|
||||||
|
lockPinned: true,
|
||||||
|
sortable: false,
|
||||||
|
filter: false,
|
||||||
|
resizable: false,
|
||||||
|
suppressMovable: true,
|
||||||
|
suppressHeaderMenuButton: true,
|
||||||
|
cellClass: "thot-database-actions-cell",
|
||||||
|
cellRenderer: DatabaseActionsCell,
|
||||||
|
};
|
||||||
|
|
||||||
|
return compact
|
||||||
|
? [workspace, status, database, schema, transport, endpoint, updated, actions]
|
||||||
|
: [workspace, database, schema, transport, endpoint, status, updated, actions];
|
||||||
|
}, [compact]);
|
||||||
|
|
||||||
|
const configuredCount = rows.filter((row) => row.configured).length;
|
||||||
|
const hiddenSelected = selectedRows.filter((row) => {
|
||||||
|
const term = search.trim().toLocaleLowerCase();
|
||||||
|
return term && ![row.workspaceName, row.databaseName, row.schema].some((value) => value.toLocaleLowerCase().includes(term));
|
||||||
|
}).length;
|
||||||
|
const canTestSelection = canManage && selectedRows.length > 0 && selectedRows.every((row) => row.configured && row.id && !row.activeSyncRun);
|
||||||
|
const canSyncSelection = canManage && selectedRows.length > 0 && selectedRows.every((row) => row.configured && row.id && row.connectionStatus === "reachable" && row.testedVersion === row.version && !row.activeSyncRun);
|
||||||
|
const perform = async (kind: "test" | "sync", operation: () => Promise<void>) => {
|
||||||
|
setAction(kind);
|
||||||
|
try {
|
||||||
|
await operation();
|
||||||
|
gridRef.current?.api.deselectAll();
|
||||||
|
setSelectedRows([]);
|
||||||
|
} finally { setAction(null); }
|
||||||
|
};
|
||||||
|
|
||||||
|
return (
|
||||||
|
<section aria-label="Workspace databases" className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
|
||||||
|
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
|
||||||
|
{selectedRows.length > 0 ? (
|
||||||
|
<>
|
||||||
|
<span className="text-sm font-semibold">{selectedRows.length} selected</span>
|
||||||
|
{hiddenSelected ? <span className="text-xs text-muted-foreground">{hiddenSelected} hidden by filter</span> : null}
|
||||||
|
<Menu.Root>
|
||||||
|
<Menu.Trigger className="inline-flex h-8 items-center justify-center gap-2 rounded-md border border-input bg-background px-3 text-sm font-medium hover:bg-muted disabled:pointer-events-none disabled:opacity-50" disabled={action !== null}>Actions <ChevronDown className="size-4" /></Menu.Trigger>
|
||||||
|
<Menu.Portal>
|
||||||
|
<Menu.Positioner side="bottom" align="start" sideOffset={4}>
|
||||||
|
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
|
||||||
|
<Menu.Item className="rounded-md px-3 py-2 text-sm outline-none data-[highlighted]:bg-muted data-[disabled]:opacity-45" disabled={!canTestSelection} onClick={() => void perform("test", () => onTestSelected(selectedRows))}>Test connections</Menu.Item>
|
||||||
|
<Menu.Separator className="my-1 h-px bg-border" />
|
||||||
|
{databaseSyncScopes.map(({ scope, label }) => (
|
||||||
|
<Menu.Item
|
||||||
|
key={scope}
|
||||||
|
className={databaseSyncItemClass}
|
||||||
|
disabled={!canSyncSelection}
|
||||||
|
onClick={() => void perform("sync", () => onSyncSelected(selectedRows, scope))}
|
||||||
|
>
|
||||||
|
{label}
|
||||||
|
</Menu.Item>
|
||||||
|
))}
|
||||||
|
</Menu.Popup>
|
||||||
|
</Menu.Positioner>
|
||||||
|
</Menu.Portal>
|
||||||
|
</Menu.Root>
|
||||||
|
<Button type="button" variant="ghost" disabled={action !== null} onClick={() => { gridRef.current?.api.deselectAll(); setSelectedRows([]); }}><X />Clear</Button>
|
||||||
|
</>
|
||||||
|
) : <><span className="thot-label whitespace-nowrap">Workspace databases</span><input
|
||||||
|
ref={searchInputRef}
|
||||||
|
className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none transition focus:border-primary/60 focus:ring-3 focus:ring-ring/15"
|
||||||
|
aria-label="Search databases"
|
||||||
|
placeholder="Search"
|
||||||
|
value={search}
|
||||||
|
onChange={(event) => onSearchChange(event.target.value)}
|
||||||
|
/></>}
|
||||||
|
<span className="whitespace-nowrap text-xs tabular-nums text-muted-foreground">
|
||||||
|
{configuredCount}/{rows.length}
|
||||||
|
</span>
|
||||||
|
{isFetching && !isLoading ? (
|
||||||
|
<span className="text-xs text-muted-foreground" role="status">Refreshing</span>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
<div className="relative min-h-[280px] flex-1">
|
||||||
|
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
|
||||||
|
<AgGridReact<CatalogDatabase>
|
||||||
|
ref={gridRef}
|
||||||
|
rowData={rows}
|
||||||
|
columnDefs={columnDefs}
|
||||||
|
context={context}
|
||||||
|
loading={isLoading}
|
||||||
|
quickFilterText={search}
|
||||||
|
defaultColDef={{
|
||||||
|
sortable: true,
|
||||||
|
filter: true,
|
||||||
|
resizable: true,
|
||||||
|
suppressHeaderMenuButton: false,
|
||||||
|
}}
|
||||||
|
getRowId={({ data }) => data.workspaceId}
|
||||||
|
rowSelection={{ mode: "multiRow", selectAll: "filtered", enableClickSelection: false }}
|
||||||
|
selectionColumnDef={{ width: 44, maxWidth: 44, pinned: "left" }}
|
||||||
|
onSelectionChanged={({ api }) => setSelectedRows(api.getSelectedRows())}
|
||||||
|
rowHeight={44}
|
||||||
|
headerHeight={38}
|
||||||
|
animateRows={false}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
{isLoading ? (
|
||||||
|
<div className="pointer-events-none absolute inset-0 grid place-items-center bg-card/80 text-sm text-muted-foreground" role="status">
|
||||||
|
Loading database configurations…
|
||||||
|
</div>
|
||||||
|
) : rows.length === 0 ? (
|
||||||
|
<div className="pointer-events-none absolute inset-0 grid place-items-center bg-card px-6 text-center text-sm text-muted-foreground" role="status">
|
||||||
|
No repository workspaces are available for database configuration.
|
||||||
|
</div>
|
||||||
|
) : null}
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
import { useMemo, useState } from "react";
|
||||||
|
import { useQuery } from "@tanstack/react-query";
|
||||||
|
import { AgGridReact } from "ag-grid-react";
|
||||||
|
import type { ColDef } from "ag-grid-community";
|
||||||
|
import { ArrowLeft, RefreshCw } from "lucide-react";
|
||||||
|
import { toast } from "sonner";
|
||||||
|
import { Button } from "../../components/ui/button";
|
||||||
|
import { apiErrorMessage } from "../../api/client";
|
||||||
|
import {
|
||||||
|
listCatalogRelationships,
|
||||||
|
startCatalogSync,
|
||||||
|
type CatalogDatabase,
|
||||||
|
type CatalogRelationship,
|
||||||
|
type CatalogSyncRun,
|
||||||
|
} from "../../api/catalog-databases";
|
||||||
|
|
||||||
|
interface Props {
|
||||||
|
database: CatalogDatabase;
|
||||||
|
canManage: boolean;
|
||||||
|
onBackToDatabases: () => void;
|
||||||
|
onOpenOverview: () => void;
|
||||||
|
onOpenTables: () => void;
|
||||||
|
onRunStarted: (run: CatalogSyncRun) => void;
|
||||||
|
onOpenSync: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function DatabaseRelationships({
|
||||||
|
database,
|
||||||
|
canManage,
|
||||||
|
onBackToDatabases,
|
||||||
|
onOpenOverview,
|
||||||
|
onOpenTables,
|
||||||
|
onRunStarted,
|
||||||
|
onOpenSync,
|
||||||
|
}: Props) {
|
||||||
|
const databaseId = database.id!;
|
||||||
|
const { data = [], isLoading, isFetching, refetch } = useQuery({
|
||||||
|
queryKey: ["catalog-relationships", databaseId],
|
||||||
|
queryFn: () => listCatalogRelationships(databaseId),
|
||||||
|
retry: false,
|
||||||
|
});
|
||||||
|
const [search, setSearch] = useState("");
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const bindingReady = database.connectionStatus === "reachable" && database.testedVersion === database.version;
|
||||||
|
|
||||||
|
const synchronize = async (scope: "relationships" | "all") => {
|
||||||
|
setBusy(true);
|
||||||
|
try {
|
||||||
|
const run = await startCatalogSync(databaseId, database.version, scope);
|
||||||
|
onRunStarted(run);
|
||||||
|
toast.success(scope === "all" ? "Full schema synchronization started" : "Relationship synchronization started");
|
||||||
|
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(false); }
|
||||||
|
};
|
||||||
|
|
||||||
|
const columns = useMemo<ColDef<CatalogRelationship>[]>(() => [
|
||||||
|
{ field: "constraintName", headerName: "Constraint", minWidth: 220, flex: 1, cellClass: "font-mono text-xs" },
|
||||||
|
{ field: "sourceTableName", headerName: "Source table", minWidth: 200, flex: 1 },
|
||||||
|
{ headerName: "Source columns", minWidth: 200, flex: 1, valueGetter: ({ data: row }) => row?.columns.map((pair) => pair.sourceColumnName).join(", ") ?? "" },
|
||||||
|
{ field: "targetTableName", headerName: "Target table", minWidth: 200, flex: 1 },
|
||||||
|
{ headerName: "Target columns", minWidth: 200, flex: 1, valueGetter: ({ data: row }) => row?.columns.map((pair) => pair.targetColumnName).join(", ") ?? "" },
|
||||||
|
{ field: "updateRule", headerName: "On update", minWidth: 125, width: 125 },
|
||||||
|
{ field: "deleteRule", headerName: "On delete", minWidth: 125, width: 125 },
|
||||||
|
], []);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<section aria-label={`Relationships for ${database.workspaceName}`} className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
|
||||||
|
<div className="px-4 pt-5 sm:px-5">
|
||||||
|
<Button type="button" variant="ghost" className="-ml-2 mb-4" disabled={busy} onClick={onBackToDatabases}><ArrowLeft />Back to databases</Button>
|
||||||
|
<div className="flex flex-wrap items-start justify-between gap-4 border-b border-border pb-5">
|
||||||
|
<div>
|
||||||
|
<p className="thot-label mb-1">Database management</p>
|
||||||
|
<h2 className="font-heading text-2xl font-semibold tracking-tight">{database.workspaceName}</h2>
|
||||||
|
<p className="mt-1 text-sm text-muted-foreground">{database.databaseName} · {database.schema}</p>
|
||||||
|
</div>
|
||||||
|
<div className="flex gap-2">
|
||||||
|
{database.activeSyncRun ? <Button type="button" variant="outline" onClick={onOpenSync}><RefreshCw className="animate-spin" />View sync</Button> : null}
|
||||||
|
<Button type="button" disabled={!canManage || !bindingReady || busy || Boolean(database.activeSyncRun)} onClick={() => void synchronize("all")}><RefreshCw />Sync all</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<nav aria-label="Database sections" className="flex gap-1 border-b border-border" role="tablist">
|
||||||
|
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={onOpenOverview}>Overview</button>
|
||||||
|
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={onOpenTables}>Tables</button>
|
||||||
|
<button type="button" role="tab" aria-selected="true" className="border-b-2 border-primary px-3 py-2 text-sm font-semibold text-foreground">Relationships</button>
|
||||||
|
</nav>
|
||||||
|
</div>
|
||||||
|
<div className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
|
||||||
|
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
|
||||||
|
<span className="thot-label whitespace-nowrap">Physical relationships</span>
|
||||||
|
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search relationships" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
|
||||||
|
<span className="text-xs tabular-nums text-muted-foreground">{data.length}</span>
|
||||||
|
<Button type="button" variant="outline" disabled={isFetching || busy} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
|
||||||
|
<Button type="button" disabled={!canManage || !bindingReady || busy || Boolean(database.activeSyncRun)} title={!bindingReady ? "Test the current database binding before synchronizing relationships" : undefined} onClick={() => void synchronize("relationships")}><RefreshCw />Sync relationships</Button>
|
||||||
|
</div>
|
||||||
|
{!bindingReady ? <div className="border-b border-border bg-muted/35 px-4 py-3 text-sm text-muted-foreground">Test the current database binding from Overview before synchronizing relationships.</div> : null}
|
||||||
|
<div className="relative min-h-[280px] flex-1">
|
||||||
|
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
|
||||||
|
<AgGridReact<CatalogRelationship> rowData={data} columnDefs={columns} loading={isLoading} quickFilterText={search} defaultColDef={{ sortable: true, filter: true, resizable: true }} getRowId={({ data: row }) => row.id} rowHeight={44} headerHeight={38} animateRows={false} overlayNoRowsTemplate="No physical relationships synchronized for this database." />
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
import { Menu } from "@base-ui/react/menu";
|
||||||
|
import { ChevronDown, RefreshCw } from "lucide-react";
|
||||||
|
import { buttonVariants } from "../../components/ui/button";
|
||||||
|
import type { CatalogSyncScope } from "../../api/catalog-databases";
|
||||||
|
|
||||||
|
export const databaseSyncItemClass = [
|
||||||
|
"flex cursor-default select-none items-center rounded-md px-3 py-2 text-sm outline-none",
|
||||||
|
"data-[highlighted]:bg-muted data-[disabled]:opacity-45",
|
||||||
|
].join(" ");
|
||||||
|
|
||||||
|
export const databaseSyncScopes: ReadonlyArray<{ scope: CatalogSyncScope; label: string }> = [
|
||||||
|
{ scope: "tables", label: "Synchronize tables" },
|
||||||
|
{ scope: "columns", label: "Synchronize all columns" },
|
||||||
|
{ scope: "relationships", label: "Synchronize relationships" },
|
||||||
|
{ scope: "all", label: "Synchronize all" },
|
||||||
|
];
|
||||||
|
|
||||||
|
interface DatabaseSyncMenuProps {
|
||||||
|
disabled: boolean;
|
||||||
|
disabledReason?: string;
|
||||||
|
onSelect: (scope: CatalogSyncScope) => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function DatabaseSyncMenu({ disabled, disabledReason, onSelect }: DatabaseSyncMenuProps) {
|
||||||
|
return (
|
||||||
|
<Menu.Root>
|
||||||
|
<Menu.Trigger
|
||||||
|
type="button"
|
||||||
|
className={buttonVariants()}
|
||||||
|
disabled={disabled}
|
||||||
|
title={disabledReason}
|
||||||
|
aria-label="Synchronize database schema"
|
||||||
|
>
|
||||||
|
<RefreshCw />Synchronize<ChevronDown />
|
||||||
|
</Menu.Trigger>
|
||||||
|
<Menu.Portal>
|
||||||
|
<Menu.Positioner side="bottom" align="end" sideOffset={4}>
|
||||||
|
<Menu.Popup className="z-50 min-w-64 rounded-lg bg-popover p-1 text-popover-foreground shadow-md ring-1 ring-foreground/10 outline-none">
|
||||||
|
{databaseSyncScopes.map(({ scope, label }) => (
|
||||||
|
<Menu.Item
|
||||||
|
key={scope}
|
||||||
|
className={databaseSyncItemClass}
|
||||||
|
disabled={disabled}
|
||||||
|
onClick={() => onSelect(scope)}
|
||||||
|
>
|
||||||
|
{label}
|
||||||
|
</Menu.Item>
|
||||||
|
))}
|
||||||
|
</Menu.Popup>
|
||||||
|
</Menu.Positioner>
|
||||||
|
</Menu.Portal>
|
||||||
|
</Menu.Root>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,315 @@
|
|||||||
|
import { useEffect, useMemo, useRef, useState } from "react";
|
||||||
|
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||||
|
import { AgGridReact } from "ag-grid-react";
|
||||||
|
import type { ColDef, ICellRendererParams } from "ag-grid-community";
|
||||||
|
import { ArrowLeft, Columns3, Pencil, RefreshCw, Save, X } from "lucide-react";
|
||||||
|
import { toast } from "sonner";
|
||||||
|
import { Button } from "../../components/ui/button";
|
||||||
|
import { ApiError, apiErrorMessage } from "../../api/client";
|
||||||
|
import {
|
||||||
|
listCatalogTables,
|
||||||
|
startCatalogSync,
|
||||||
|
updateCatalogTableMetadata,
|
||||||
|
type CatalogDatabase,
|
||||||
|
type CatalogSyncRun,
|
||||||
|
type CatalogTable,
|
||||||
|
} from "../../api/catalog-databases";
|
||||||
|
import type { DatabaseNavigationState } from "./model";
|
||||||
|
import { DatabaseColumns } from "./DatabaseColumns";
|
||||||
|
|
||||||
|
interface Props {
|
||||||
|
database: CatalogDatabase;
|
||||||
|
canManage: boolean;
|
||||||
|
activeRun?: CatalogSyncRun;
|
||||||
|
onBackToDatabases: () => void;
|
||||||
|
onOpenOverview: () => void;
|
||||||
|
onOpenRelationships: () => void;
|
||||||
|
onNavigationStateChange: (state: DatabaseNavigationState) => void;
|
||||||
|
onRunStarted: (run: CatalogSyncRun) => void;
|
||||||
|
onOpenSync: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface TableGridContext {
|
||||||
|
canManage: boolean;
|
||||||
|
onOpen: (table: CatalogTable, section: "overview" | "columns", origin: HTMLButtonElement) => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
function TableActionsCell({ data, context }: ICellRendererParams<CatalogTable, unknown, TableGridContext>) {
|
||||||
|
if (!data || !context) return null;
|
||||||
|
return (
|
||||||
|
<div className="flex h-full items-center justify-end" onClick={(event) => event.stopPropagation()}>
|
||||||
|
<Button type="button" variant="ghost" size="icon-lg" aria-label={`View columns for ${data.name}`} title={`View columns for ${data.name}`} onClick={(event) => context.onOpen(data, "columns", event.currentTarget)}>
|
||||||
|
<Columns3 aria-hidden="true" />
|
||||||
|
</Button>
|
||||||
|
<Button type="button" variant="ghost" size="icon-lg" aria-label={`Edit description for ${data.name}`} title={`Edit metadata for ${data.name}`} disabled={!context.canManage} onClick={(event) => context.onOpen(data, "overview", event.currentTarget)}>
|
||||||
|
<Pencil aria-hidden="true" />
|
||||||
|
</Button>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function DatabaseTables({
|
||||||
|
database,
|
||||||
|
canManage,
|
||||||
|
activeRun,
|
||||||
|
onBackToDatabases,
|
||||||
|
onOpenOverview,
|
||||||
|
onOpenRelationships,
|
||||||
|
onNavigationStateChange,
|
||||||
|
onRunStarted,
|
||||||
|
onOpenSync,
|
||||||
|
}: Props) {
|
||||||
|
const databaseId = database.id!;
|
||||||
|
const queryClient = useQueryClient();
|
||||||
|
const queryKey = ["catalog-tables", databaseId] as const;
|
||||||
|
const { data = [], isLoading, isFetching, refetch } = useQuery({ queryKey, queryFn: () => listCatalogTables(databaseId), retry: false });
|
||||||
|
const [search, setSearch] = useState("");
|
||||||
|
const [displayedCount, setDisplayedCount] = useState(0);
|
||||||
|
const [selectedIds, setSelectedIds] = useState<string[]>([]);
|
||||||
|
const [activeTableId, setActiveTableId] = useState<string | null>(null);
|
||||||
|
const [tableSection, setTableSection] = useState<"overview" | "columns">("overview");
|
||||||
|
const [description, setDescription] = useState("");
|
||||||
|
const [generatedDescription, setGeneratedDescription] = useState("");
|
||||||
|
const [baseline, setBaseline] = useState("");
|
||||||
|
const [editorVersion, setEditorVersion] = useState<number | null>(null);
|
||||||
|
const [stale, setStale] = useState(false);
|
||||||
|
const [staleBannerOpen, setStaleBannerOpen] = useState(true);
|
||||||
|
const [busy, setBusy] = useState<"sync" | "save" | null>(null);
|
||||||
|
const [columnNavigation, setColumnNavigation] = useState<DatabaseNavigationState>({ dirty: false, busy: false });
|
||||||
|
const gridRef = useRef<AgGridReact<CatalogTable>>(null);
|
||||||
|
const originRef = useRef<HTMLButtonElement | null>(null);
|
||||||
|
const headingRef = useRef<HTMLHeadingElement>(null);
|
||||||
|
const activeTable = activeTableId ? data.find((table) => table.id === activeTableId) : undefined;
|
||||||
|
const editorFingerprint = JSON.stringify([description, generatedDescription]);
|
||||||
|
const editorDirty = Boolean(activeTableId && editorFingerprint !== baseline);
|
||||||
|
const dirty = tableSection === "columns" ? columnNavigation.dirty : editorDirty;
|
||||||
|
const navigationBusy = busy !== null || (tableSection === "columns" && columnNavigation.busy);
|
||||||
|
const bindingReady = database.connectionStatus === "reachable" && database.testedVersion === database.version;
|
||||||
|
const currentRun = activeRun ?? database.activeSyncRun;
|
||||||
|
|
||||||
|
useEffect(() => { onNavigationStateChange({ dirty, busy: navigationBusy }); }, [dirty, navigationBusy, onNavigationStateChange]);
|
||||||
|
useEffect(() => {
|
||||||
|
if (!activeTableId || !activeTable || editorVersion === activeTable.version || busy) return;
|
||||||
|
setStale(true);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
}, [activeTable, activeTableId, busy, editorVersion]);
|
||||||
|
|
||||||
|
const openTable = (table: CatalogTable, section: "overview" | "columns", origin: HTMLButtonElement) => {
|
||||||
|
originRef.current = origin;
|
||||||
|
setActiveTableId(table.id);
|
||||||
|
setTableSection(section);
|
||||||
|
setDescription(table.description ?? "");
|
||||||
|
setGeneratedDescription(table.generatedDescription ?? "");
|
||||||
|
setBaseline(JSON.stringify([table.description ?? "", table.generatedDescription ?? ""]));
|
||||||
|
setEditorVersion(table.version);
|
||||||
|
setStale(false);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
setColumnNavigation({ dirty: false, busy: false });
|
||||||
|
};
|
||||||
|
const leaveTable = () => {
|
||||||
|
if (navigationBusy) return;
|
||||||
|
if (dirty && !window.confirm("Discard unsaved metadata?")) return;
|
||||||
|
setActiveTableId(null);
|
||||||
|
setTableSection("overview");
|
||||||
|
window.setTimeout(() => originRef.current?.focus(), 0);
|
||||||
|
};
|
||||||
|
const navigateDatabase = (target: "databases" | "overview" | "relationships") => {
|
||||||
|
if (navigationBusy) return;
|
||||||
|
if (dirty && !window.confirm("Discard unsaved metadata?")) return;
|
||||||
|
if (target === "databases") onBackToDatabases();
|
||||||
|
else if (target === "overview") onOpenOverview();
|
||||||
|
else onOpenRelationships();
|
||||||
|
};
|
||||||
|
const save = async () => {
|
||||||
|
if (!activeTable || editorVersion === null) return;
|
||||||
|
setBusy("save");
|
||||||
|
try {
|
||||||
|
const updated = await updateCatalogTableMetadata(
|
||||||
|
databaseId, activeTable.id, editorVersion,
|
||||||
|
description.trim() || null, generatedDescription.trim() || null,
|
||||||
|
);
|
||||||
|
queryClient.setQueryData<CatalogTable[]>(queryKey, (current = []) => current.map((table) => table.id === updated.id ? updated : table));
|
||||||
|
setDescription(updated.description ?? "");
|
||||||
|
setGeneratedDescription(updated.generatedDescription ?? "");
|
||||||
|
setBaseline(JSON.stringify([updated.description ?? "", updated.generatedDescription ?? ""]));
|
||||||
|
setEditorVersion(updated.version);
|
||||||
|
setStale(false);
|
||||||
|
toast.success("Table metadata saved");
|
||||||
|
} catch (error) {
|
||||||
|
if (error instanceof ApiError && error.code === "table_stale") {
|
||||||
|
await refetch();
|
||||||
|
setStale(true);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
} else toast.error(apiErrorMessage(error));
|
||||||
|
} finally { setBusy(null); }
|
||||||
|
};
|
||||||
|
const reloadTable = async () => {
|
||||||
|
if (!activeTableId) return;
|
||||||
|
setBusy("sync");
|
||||||
|
try {
|
||||||
|
const result = await refetch();
|
||||||
|
const latest = result.data?.find((table) => table.id === activeTableId);
|
||||||
|
if (!latest) { leaveTable(); return; }
|
||||||
|
setDescription(latest.description ?? "");
|
||||||
|
setGeneratedDescription(latest.generatedDescription ?? "");
|
||||||
|
setBaseline(JSON.stringify([latest.description ?? "", latest.generatedDescription ?? ""]));
|
||||||
|
setEditorVersion(latest.version);
|
||||||
|
setStale(false);
|
||||||
|
setStaleBannerOpen(true);
|
||||||
|
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(null); }
|
||||||
|
};
|
||||||
|
const synchronize = async (scope: "tables" | "columns" | "all", tableIds: string[] = []) => {
|
||||||
|
setBusy("sync");
|
||||||
|
try {
|
||||||
|
const run = await startCatalogSync(databaseId, database.version, scope, tableIds);
|
||||||
|
onRunStarted(run);
|
||||||
|
gridRef.current?.api.deselectAll();
|
||||||
|
setSelectedIds([]);
|
||||||
|
toast.success(scope === "all" ? "Full schema synchronization started" : scope === "columns" ? "Column synchronization started" : "Table synchronization started");
|
||||||
|
} catch (error) { toast.error(apiErrorMessage(error)); } finally { setBusy(null); }
|
||||||
|
};
|
||||||
|
|
||||||
|
const columns = useMemo<ColDef<CatalogTable>[]>(() => [
|
||||||
|
{ field: "name", headerName: "Name", minWidth: 250, flex: 1, cellClass: "font-mono text-xs" },
|
||||||
|
{ field: "sourceComment", headerName: "Source comment", minWidth: 260, flex: 1.25, valueFormatter: ({ value }) => value ?? "" },
|
||||||
|
{ field: "generatedDescription", headerName: "Generated description", minWidth: 280, flex: 1.25, valueFormatter: ({ value }) => value ?? "" },
|
||||||
|
{ field: "description", headerName: "Description", minWidth: 280, flex: 1.25, valueFormatter: ({ value }) => value ?? "" },
|
||||||
|
{ colId: "actions", headerName: "", width: 104, minWidth: 104, maxWidth: 104, pinned: "right", sortable: false, filter: false, resizable: false, cellRenderer: TableActionsCell },
|
||||||
|
], []);
|
||||||
|
const context = useMemo<TableGridContext>(() => ({ canManage, onOpen: openTable }), [canManage, data]);
|
||||||
|
const visibleSelected = useMemo(() => {
|
||||||
|
const term = search.trim().toLocaleLowerCase();
|
||||||
|
if (!term) return selectedIds.length;
|
||||||
|
const selected = new Set(selectedIds);
|
||||||
|
return data.filter((table) => selected.has(table.id) && [table.name, table.sourceComment, table.generatedDescription, table.description]
|
||||||
|
.some((value) => value?.toLocaleLowerCase().includes(term))).length;
|
||||||
|
}, [data, search, selectedIds]);
|
||||||
|
|
||||||
|
const databaseHeader = (
|
||||||
|
<div className="px-4 pt-5 sm:px-5">
|
||||||
|
<Button type="button" variant="ghost" className="-ml-2 mb-4" disabled={navigationBusy} onClick={() => navigateDatabase("databases")}><ArrowLeft />Back to databases</Button>
|
||||||
|
<div className="flex flex-wrap items-start justify-between gap-4 border-b border-border pb-5">
|
||||||
|
<div>
|
||||||
|
<p className="thot-label mb-1">Database management</p>
|
||||||
|
<h2 ref={headingRef} tabIndex={-1} className="font-heading text-2xl font-semibold tracking-tight outline-none">{database.workspaceName}</h2>
|
||||||
|
<p className="mt-1 text-sm text-muted-foreground">{database.databaseName} · {database.schema}</p>
|
||||||
|
</div>
|
||||||
|
<div className="flex gap-2">
|
||||||
|
{currentRun ? <Button type="button" variant="outline" onClick={onOpenSync}><RefreshCw className="animate-spin" />View sync</Button> : null}
|
||||||
|
<Button type="button" disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)} onClick={() => void synchronize("all")}><RefreshCw />Sync all</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<nav aria-label="Database sections" className="flex gap-1 border-b border-border" role="tablist">
|
||||||
|
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={() => navigateDatabase("overview")}>Overview</button>
|
||||||
|
<button type="button" role="tab" aria-selected="true" className="border-b-2 border-primary px-3 py-2 text-sm font-semibold text-foreground">Tables</button>
|
||||||
|
<button type="button" role="tab" aria-selected="false" className="border-b-2 border-transparent px-3 py-2 text-sm font-semibold text-muted-foreground hover:text-foreground" onClick={() => navigateDatabase("relationships")}>Relationships</button>
|
||||||
|
</nav>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
|
||||||
|
if (activeTable) {
|
||||||
|
return (
|
||||||
|
<section aria-label={`Table ${activeTable.name}`} className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
|
||||||
|
{databaseHeader}
|
||||||
|
<div className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
|
||||||
|
<div className="border-b border-border px-4 pt-4">
|
||||||
|
<Button type="button" variant="ghost" className="-ml-2 mb-2" onClick={leaveTable}><ArrowLeft />Back to tables</Button>
|
||||||
|
<p className="thot-label">Physical table</p>
|
||||||
|
<h3 className="mt-1 font-heading text-xl font-semibold">{activeTable.name}</h3>
|
||||||
|
<nav aria-label="Table sections" className="mt-3 flex gap-1" role="tablist">
|
||||||
|
<button type="button" role="tab" aria-selected={tableSection === "overview"} className={`border-b-2 px-3 py-2 text-sm font-semibold ${tableSection === "overview" ? "border-primary text-foreground" : "border-transparent text-muted-foreground"}`} onClick={() => setTableSection("overview")}>Overview</button>
|
||||||
|
<button type="button" role="tab" aria-selected={tableSection === "columns"} className={`border-b-2 px-3 py-2 text-sm font-semibold ${tableSection === "columns" ? "border-primary text-foreground" : "border-transparent text-muted-foreground"}`} onClick={() => setTableSection("columns")}>Columns</button>
|
||||||
|
</nav>
|
||||||
|
</div>
|
||||||
|
{tableSection === "columns" ? (
|
||||||
|
<DatabaseColumns databaseId={databaseId} table={activeTable} canManage={canManage} onNavigationStateChange={setColumnNavigation} onSync={() => void synchronize("columns", [activeTable.id])} />
|
||||||
|
) : (
|
||||||
|
<div className="min-h-0 overflow-y-auto px-4 py-5">
|
||||||
|
<p className="text-sm text-muted-foreground">Only review metadata can be changed.</p>
|
||||||
|
{stale ? (
|
||||||
|
staleBannerOpen ? (
|
||||||
|
<div className="mt-4 rounded-md border border-amber-500/50 bg-amber-500/8 p-4 text-sm">
|
||||||
|
<p className="font-semibold">A newer table description is available.</p>
|
||||||
|
<p className="mt-1 text-muted-foreground">Your draft is preserved until you reload the current metadata.</p>
|
||||||
|
<div className="mt-3 flex gap-2"><Button type="button" variant="outline" onClick={() => void reloadTable()}>Reload latest</Button><Button type="button" variant="ghost" onClick={() => setStaleBannerOpen(false)}>Keep editing</Button></div>
|
||||||
|
</div>
|
||||||
|
) : <div className="mt-4 flex items-center justify-between gap-3 rounded-md border border-amber-500/40 bg-amber-500/8 px-4 py-3 text-sm"><span>Reload the latest value before this description can be saved.</span><Button type="button" variant="outline" onClick={() => void reloadTable()}>Reload latest</Button></div>
|
||||||
|
) : null}
|
||||||
|
<div className="mt-5 grid gap-4">
|
||||||
|
<label className="grid gap-1.5 text-sm font-semibold">Physical table name<input className="h-9 rounded-md border border-input bg-muted/35 px-3 font-mono text-xs" value={activeTable.name} readOnly /></label>
|
||||||
|
<label className="grid gap-1.5 text-sm font-semibold">Source comment<textarea className="min-h-24 rounded-md border border-input bg-muted/35 px-3 py-2 text-sm" value={activeTable.sourceComment ?? ""} readOnly /></label>
|
||||||
|
<label className="grid gap-1.5 text-sm font-semibold">Generated description<textarea className="min-h-32 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={generatedDescription} disabled={!canManage || busy !== null} onChange={(event) => setGeneratedDescription(event.target.value)} /></label>
|
||||||
|
<label className="grid gap-1.5 text-sm font-semibold">Description<textarea className="min-h-36 rounded-md border border-input bg-card px-3 py-2 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" value={description} disabled={!canManage || busy !== null} onChange={(event) => setDescription(event.target.value)} /><span className="font-normal text-muted-foreground">Leave blank to keep the curated description empty.</span></label>
|
||||||
|
</div>
|
||||||
|
<div className="mt-5 flex justify-end gap-2 border-t border-border pt-4">
|
||||||
|
<Button type="button" variant="outline" disabled={busy !== null} onClick={leaveTable}>Cancel</Button>
|
||||||
|
<Button type="button" disabled={!canManage || !editorDirty || busy !== null || stale} onClick={() => void save()}><Save />{busy === "save" ? "Saving…" : "Save metadata"}</Button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<section aria-label={`Tables for ${database.workspaceName}`} className="flex min-h-0 flex-1 flex-col overflow-hidden bg-background">
|
||||||
|
{databaseHeader}
|
||||||
|
<div className="mx-3 mb-4 mt-4 flex min-h-0 flex-1 flex-col overflow-hidden rounded-md border border-border bg-card sm:mx-5">
|
||||||
|
<div className="flex min-h-12 flex-wrap items-center gap-3 border-b border-border px-3 py-2">
|
||||||
|
{selectedIds.length > 0 ? (
|
||||||
|
<>
|
||||||
|
<span className="text-sm font-semibold">{selectedIds.length} selected</span>
|
||||||
|
{visibleSelected !== selectedIds.length ? <span className="text-xs text-muted-foreground">{selectedIds.length - visibleSelected} hidden by filter</span> : null}
|
||||||
|
<Button
|
||||||
|
type="button"
|
||||||
|
variant="outline"
|
||||||
|
disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)}
|
||||||
|
title={!bindingReady ? "Test the current database binding before synchronizing columns" : undefined}
|
||||||
|
onClick={() => void synchronize("columns", selectedIds)}
|
||||||
|
>
|
||||||
|
<RefreshCw className={busy === "sync" ? "animate-spin" : ""} />Synchronize columns
|
||||||
|
</Button>
|
||||||
|
<Button type="button" variant="ghost" onClick={() => { gridRef.current?.api.deselectAll(); setSelectedIds([]); }}><X />Clear</Button>
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<span className="thot-label whitespace-nowrap">Catalog tables</span>
|
||||||
|
<input className="h-8 min-w-40 flex-1 rounded-md border border-input bg-background px-2.5 text-sm outline-none focus:border-primary/60 focus:ring-3 focus:ring-ring/15" aria-label="Search tables" placeholder="Search" value={search} onChange={(event) => setSearch(event.target.value)} />
|
||||||
|
<span
|
||||||
|
aria-live="polite"
|
||||||
|
className="whitespace-nowrap text-xs tabular-nums text-muted-foreground"
|
||||||
|
>
|
||||||
|
{search.trim() ? `${displayedCount} of ${data.length}` : displayedCount}
|
||||||
|
</span>
|
||||||
|
<Button type="button" variant="outline" disabled={isFetching || busy !== null} onClick={() => void refetch()}><RefreshCw className={isFetching ? "animate-spin" : ""} />Refresh</Button>
|
||||||
|
<Button type="button" disabled={!canManage || !bindingReady || busy !== null || Boolean(currentRun)} title={!bindingReady ? "Test the current database binding before synchronizing tables" : undefined} onClick={() => void synchronize("tables")}><RefreshCw className={busy === "sync" ? "animate-spin" : ""} />Sync tables</Button>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
{!bindingReady ? <div className="border-b border-border bg-muted/35 px-4 py-3 text-sm text-muted-foreground">Test the current database binding from Overview before synchronizing tables.</div> : null}
|
||||||
|
<div className="relative min-h-[280px] flex-1">
|
||||||
|
<div className="thot-database-grid ag-theme-alpine absolute inset-0 h-full w-full">
|
||||||
|
<AgGridReact<CatalogTable>
|
||||||
|
ref={gridRef}
|
||||||
|
rowData={data}
|
||||||
|
columnDefs={columns}
|
||||||
|
context={context}
|
||||||
|
loading={isLoading}
|
||||||
|
quickFilterText={search}
|
||||||
|
defaultColDef={{ sortable: true, filter: true, resizable: true }}
|
||||||
|
getRowId={({ data: table }) => table.id}
|
||||||
|
rowSelection={{ mode: "multiRow", selectAll: "filtered", enableClickSelection: false }}
|
||||||
|
onSelectionChanged={({ api }) => setSelectedIds(api.getSelectedRows().map((table) => table.id))}
|
||||||
|
onModelUpdated={({ api }) => setDisplayedCount(api.getDisplayedRowCount())}
|
||||||
|
rowHeight={44}
|
||||||
|
headerHeight={38}
|
||||||
|
animateRows={false}
|
||||||
|
overlayNoRowsTemplate="No catalog tables. Test the binding, then sync tables."
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
import type {
|
||||||
|
CatalogDatabase,
|
||||||
|
CatalogSecretName,
|
||||||
|
DatabaseConfiguration,
|
||||||
|
} from "../../api/catalog-databases";
|
||||||
|
|
||||||
|
export type DatabaseFormMode = "add" | "view" | "edit" | "delete";
|
||||||
|
|
||||||
|
export type DatabaseScreen =
|
||||||
|
| { kind: "list" }
|
||||||
|
| { kind: "tables"; workspaceId: string }
|
||||||
|
| { kind: "relationships"; workspaceId: string }
|
||||||
|
| {
|
||||||
|
kind: DatabaseFormMode;
|
||||||
|
workspaceId: string;
|
||||||
|
workspaceLocked?: boolean;
|
||||||
|
};
|
||||||
|
|
||||||
|
export type DatabaseBusyAction = "save" | "test" | "delete" | "retry-secrets" | "reload" | null;
|
||||||
|
|
||||||
|
export type SecretDraft = Partial<Record<CatalogSecretName, string>>;
|
||||||
|
|
||||||
|
export interface DatabaseFormDraft extends DatabaseConfiguration {
|
||||||
|
secrets: SecretDraft;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface DatabaseNavigationState {
|
||||||
|
dirty: boolean;
|
||||||
|
busy: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function draftFrom(row: CatalogDatabase): DatabaseFormDraft {
|
||||||
|
return {
|
||||||
|
workspaceId: row.workspaceId,
|
||||||
|
engine: "postgres",
|
||||||
|
databaseName: row.databaseName,
|
||||||
|
schema: row.schema,
|
||||||
|
binding: {
|
||||||
|
port: 5432,
|
||||||
|
restPath: "/health",
|
||||||
|
restAuth: "x-api-key",
|
||||||
|
sshPort: 22,
|
||||||
|
sshTargetPort: 5432,
|
||||||
|
...row.binding,
|
||||||
|
},
|
||||||
|
secrets: {},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function configurationFromDraft(draft: DatabaseFormDraft): DatabaseConfiguration {
|
||||||
|
const binding = draft.binding.transport === "postgres_direct"
|
||||||
|
? {
|
||||||
|
transport: draft.binding.transport,
|
||||||
|
host: draft.binding.host,
|
||||||
|
port: draft.binding.port,
|
||||||
|
username: draft.binding.username,
|
||||||
|
tlsServername: draft.binding.tlsServername,
|
||||||
|
}
|
||||||
|
: draft.binding.transport === "rest_api"
|
||||||
|
? {
|
||||||
|
transport: draft.binding.transport,
|
||||||
|
baseUrl: draft.binding.baseUrl,
|
||||||
|
restPath: draft.binding.restPath,
|
||||||
|
restAuth: draft.binding.restAuth,
|
||||||
|
}
|
||||||
|
: {
|
||||||
|
transport: draft.binding.transport,
|
||||||
|
username: draft.binding.username,
|
||||||
|
tlsServername: draft.binding.tlsServername,
|
||||||
|
sshHost: draft.binding.sshHost,
|
||||||
|
sshPort: draft.binding.sshPort,
|
||||||
|
sshUsername: draft.binding.sshUsername,
|
||||||
|
sshTargetHost: draft.binding.sshTargetHost,
|
||||||
|
sshTargetPort: draft.binding.sshTargetPort,
|
||||||
|
};
|
||||||
|
|
||||||
|
return {
|
||||||
|
workspaceId: draft.workspaceId,
|
||||||
|
engine: draft.engine,
|
||||||
|
databaseName: draft.databaseName,
|
||||||
|
schema: draft.schema,
|
||||||
|
binding,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export function configurationFingerprint(draft: DatabaseFormDraft): string {
|
||||||
|
return JSON.stringify(configurationFromDraft(draft));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function hasSecretChanges(draft: DatabaseFormDraft | null): boolean {
|
||||||
|
return Boolean(draft && Object.keys(secretReplacements(draft)).length);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function secretReplacements(draft: DatabaseFormDraft): SecretDraft {
|
||||||
|
const allowed: CatalogSecretName[] = draft.binding.transport === "postgres_direct"
|
||||||
|
? ["password", "tlsCa"]
|
||||||
|
: draft.binding.transport === "rest_api"
|
||||||
|
? draft.binding.restAuth === "none" ? [] : ["apiKey"]
|
||||||
|
: [
|
||||||
|
"password",
|
||||||
|
"sshPrivateKey",
|
||||||
|
"sshPrivateKeyPassphrase",
|
||||||
|
"sshKnownHosts",
|
||||||
|
"tlsCa",
|
||||||
|
];
|
||||||
|
|
||||||
|
return Object.fromEntries(
|
||||||
|
Object.entries(draft.secrets).filter(([name, value]) => (
|
||||||
|
allowed.includes(name as CatalogSecretName) && Boolean(value)
|
||||||
|
)),
|
||||||
|
) as SecretDraft;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function statusLabel(row: CatalogDatabase): string {
|
||||||
|
if (!row.workspaceAvailable) return "Workspace missing";
|
||||||
|
if (!row.configured) return "Not configured";
|
||||||
|
if (row.connectionStatus === "reachable") return "Reachable";
|
||||||
|
if (row.connectionStatus === "failed") return "Failed";
|
||||||
|
return "Untested";
|
||||||
|
}
|
||||||
@@ -6,8 +6,9 @@ export const server = setupServer(
|
|||||||
issuer: "mock", subject: "test-user", displayName: "Test user", roles: ["admin"],
|
issuer: "mock", subject: "test-user", displayName: "Test user", roles: ["admin"],
|
||||||
permissions: [
|
permissions: [
|
||||||
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
"session.use", "session.read_all", "session.manage_all", "settings.manage",
|
||||||
"workspace.manage", "workspace.secrets.manage", "pi.manage", "auth.diagnostics.read",
|
"workspace.manage", "workspace.secrets.manage", "database.manage", "pi.manage", "auth.diagnostics.read",
|
||||||
], isAdmin: true, csrfToken: "c".repeat(43), session: null,
|
], isAdmin: true, csrfToken: "c".repeat(43), session: null,
|
||||||
})),
|
})),
|
||||||
http.get("/api/health/dwh", () => HttpResponse.json({ ok: true, detail: "ok" })),
|
http.get("/api/health/dwh", () => HttpResponse.json({ ok: true, detail: "ok" })),
|
||||||
|
http.get("/api/catalog/databases", () => HttpResponse.json([])),
|
||||||
);
|
);
|
||||||
|
|||||||
+10
-2
@@ -21,5 +21,13 @@ if [[ "${THOTH_ENABLE_EMBEDDING_GPU:-0}" == "1" ]]; then
|
|||||||
compose_files+=(-f "$ROOT/deploy/compose.embedding-gpu.yaml")
|
compose_files+=(-f "$ROOT/deploy/compose.embedding-gpu.yaml")
|
||||||
fi
|
fi
|
||||||
|
|
||||||
exec docker compose --env-file "$LOCAL_ENV_FILE" \
|
compose=(docker compose --env-file "$LOCAL_ENV_FILE" "${compose_files[@]}")
|
||||||
"${compose_files[@]}" up --build "$@"
|
|
||||||
|
# Migrations are an explicit one-shot operation, never hidden in backend startup. The local
|
||||||
|
# launcher runs that operation before bringing the foreground stack up so a fresh checkout is
|
||||||
|
# immediately usable while production operators can invoke the same service during rollout.
|
||||||
|
"${compose[@]}" build core
|
||||||
|
"${compose[@]}" up -d catalog-db
|
||||||
|
"${compose[@]}" run --rm catalog-migrate
|
||||||
|
|
||||||
|
exec "${compose[@]}" up --build "$@"
|
||||||
|
|||||||
@@ -64,7 +64,7 @@ for profile in local server; do
|
|||||||
const fs = require("fs");
|
const fs = require("fs");
|
||||||
const [path, profile] = process.argv.slice(2);
|
const [path, profile] = process.argv.slice(2);
|
||||||
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||||
const expected = "core,embedding,embedding-model-init,frontend,qdrant";
|
const expected = "catalog-db,core,embedding,embedding-model-init,frontend,qdrant";
|
||||||
if (Object.keys(config.services).sort().join(",") !== expected) {
|
if (Object.keys(config.services).sort().join(",") !== expected) {
|
||||||
throw new Error(profile + ": install stack must be exactly " + expected);
|
throw new Error(profile + ": install stack must be exactly " + expected);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -6,6 +6,9 @@ cd "$(dirname "$0")/.."
|
|||||||
test -f .env.example
|
test -f .env.example
|
||||||
test -f deploy/env/local.env.example
|
test -f deploy/env/local.env.example
|
||||||
test -f deploy/env/server.env.example
|
test -f deploy/env/server.env.example
|
||||||
|
test -f docker/catalog-db-init.sql
|
||||||
|
grep -q "pg_read_file('/run/secrets/catalog_runtime_password')" docker/catalog-db-init.sql
|
||||||
|
grep -q "CREATE ROLE thothii_catalog_runtime" docker/catalog-db-init.sql
|
||||||
|
|
||||||
rendered=$(mktemp)
|
rendered=$(mktemp)
|
||||||
trap 'rm -f "$rendered"' EXIT HUP INT TERM
|
trap 'rm -f "$rendered"' EXIT HUP INT TERM
|
||||||
@@ -18,7 +21,7 @@ const fs = require("fs");
|
|||||||
|
|
||||||
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
|
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
|
||||||
const services = Object.keys(config.services).sort();
|
const services = Object.keys(config.services).sort();
|
||||||
if (services.join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
if (services.join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||||
throw new Error(`unexpected service set: ${services.join(",")}`);
|
throw new Error(`unexpected service set: ${services.join(",")}`);
|
||||||
}
|
}
|
||||||
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
|
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
|
||||||
@@ -26,6 +29,7 @@ if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify
|
|||||||
}
|
}
|
||||||
const expectedVolumes = [
|
const expectedVolumes = [
|
||||||
"auth-state",
|
"auth-state",
|
||||||
|
"catalog-data",
|
||||||
"embedding-models",
|
"embedding-models",
|
||||||
"pi-state",
|
"pi-state",
|
||||||
"qdrant-data",
|
"qdrant-data",
|
||||||
@@ -43,12 +47,19 @@ const frontend = config.services.frontend;
|
|||||||
const qdrant = config.services.qdrant;
|
const qdrant = config.services.qdrant;
|
||||||
const embedding = config.services.embedding;
|
const embedding = config.services.embedding;
|
||||||
const modelInit = config.services["embedding-model-init"];
|
const modelInit = config.services["embedding-model-init"];
|
||||||
|
const catalog = config.services["catalog-db"];
|
||||||
if (!frontend.ports?.some((port) => port.host_ip === "127.0.0.1")) {
|
if (!frontend.ports?.some((port) => port.host_ip === "127.0.0.1")) {
|
||||||
throw new Error("local frontend must publish a loopback port");
|
throw new Error("local frontend must publish a loopback port");
|
||||||
}
|
}
|
||||||
for (const service of [embedding, modelInit]) {
|
for (const service of [embedding, modelInit]) {
|
||||||
if ((service.ports || []).length !== 0) throw new Error("private semantic services must not publish host ports");
|
if ((service.ports || []).length !== 0) throw new Error("private semantic services must not publish host ports");
|
||||||
}
|
}
|
||||||
|
if ((catalog.ports || []).length !== 0) throw new Error("catalog database must not publish host ports");
|
||||||
|
if (!catalog.healthcheck) throw new Error("catalog database must define a healthcheck");
|
||||||
|
const catalogHealthcheck = JSON.stringify(catalog.healthcheck.test || []);
|
||||||
|
if (!catalogHealthcheck.includes("pg_isready") || !catalogHealthcheck.includes("thothii_catalog_runtime")) {
|
||||||
|
throw new Error("catalog database healthcheck must verify readiness and the runtime role");
|
||||||
|
}
|
||||||
if (!qdrant.ports?.some((port) => port.host_ip === "127.0.0.1" && Number(port.target) === 6333)) {
|
if (!qdrant.ports?.some((port) => port.host_ip === "127.0.0.1" && Number(port.target) === 6333)) {
|
||||||
throw new Error("local Qdrant dashboard must publish only its loopback port");
|
throw new Error("local Qdrant dashboard must publish only its loopback port");
|
||||||
}
|
}
|
||||||
@@ -65,6 +76,9 @@ if (embedding.image !== "ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279
|
|||||||
if (modelInit.image !== "ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a") {
|
if (modelInit.image !== "ollama/ollama:0.32.0@sha256:57f573b47f1f71ebb445789f279fe3e596a8beab182f7cf486db9205bad87c5a") {
|
||||||
throw new Error("embedding-model-init image must be pinned by version and digest");
|
throw new Error("embedding-model-init image must be pinned by version and digest");
|
||||||
}
|
}
|
||||||
|
if (catalog.image !== "postgres:17.6-bookworm@sha256:f3bd19c606e442c3d7bdfa8002e03fe260a1023351e0ea4598032022b68dd6e3") {
|
||||||
|
throw new Error("catalog PostgreSQL image must be pinned by version and digest");
|
||||||
|
}
|
||||||
const env = core.environment || {};
|
const env = core.environment || {};
|
||||||
for (const [key, value] of Object.entries({
|
for (const [key, value] of Object.entries({
|
||||||
THT_WORKSPACE_INSTALLATION_ID: "local",
|
THT_WORKSPACE_INSTALLATION_ID: "local",
|
||||||
@@ -74,6 +88,10 @@ for (const [key, value] of Object.entries({
|
|||||||
THT_INTERNAL_EMBEDDING_URL: "http://embedding:11434",
|
THT_INTERNAL_EMBEDDING_URL: "http://embedding:11434",
|
||||||
THT_INTERNAL_EMBEDDING_MODEL: "qwen3-embedding:0.6b",
|
THT_INTERNAL_EMBEDDING_MODEL: "qwen3-embedding:0.6b",
|
||||||
THT_INTERNAL_EMBEDDING_DIMENSIONS: "1024",
|
THT_INTERNAL_EMBEDDING_DIMENSIONS: "1024",
|
||||||
|
THT_CATALOG_DB_HOST: "catalog-db",
|
||||||
|
THT_CATALOG_DB_NAME: "thothii_catalog",
|
||||||
|
THT_CATALOG_RUNTIME_USER: "thothii_catalog_runtime",
|
||||||
|
THT_CATALOG_RUNTIME_PASSWORD_FILE: "/run/secrets/catalog_runtime_password",
|
||||||
})) {
|
})) {
|
||||||
if (env[key] !== value) throw new Error(`unexpected core ${key}: ${env[key]}`);
|
if (env[key] !== value) throw new Error(`unexpected core ${key}: ${env[key]}`);
|
||||||
}
|
}
|
||||||
@@ -98,6 +116,9 @@ const depends = core.depends_on || {};
|
|||||||
if (depends.qdrant?.condition !== "service_healthy") {
|
if (depends.qdrant?.condition !== "service_healthy") {
|
||||||
throw new Error("core must wait for qdrant health");
|
throw new Error("core must wait for qdrant health");
|
||||||
}
|
}
|
||||||
|
if (depends["catalog-db"]?.condition !== "service_healthy") {
|
||||||
|
throw new Error("core must wait for catalog database health");
|
||||||
|
}
|
||||||
if (depends["embedding-model-init"]?.condition !== "service_completed_successfully") {
|
if (depends["embedding-model-init"]?.condition !== "service_completed_successfully") {
|
||||||
throw new Error("core must wait for embedding-model-init success");
|
throw new Error("core must wait for embedding-model-init success");
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -25,14 +25,14 @@ const fs = require("fs");
|
|||||||
const [configPath, profile] = process.argv.slice(2);
|
const [configPath, profile] = process.argv.slice(2);
|
||||||
const config = JSON.parse(fs.readFileSync(configPath, "utf8"));
|
const config = JSON.parse(fs.readFileSync(configPath, "utf8"));
|
||||||
const services = Object.keys(config.services).sort();
|
const services = Object.keys(config.services).sort();
|
||||||
if (services.join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
if (services.join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||||
throw new Error("mandatory stack must include core, frontend, qdrant, embedding, and embedding-model-init");
|
throw new Error("mandatory stack must include catalog-db, core, frontend, qdrant, embedding, and embedding-model-init");
|
||||||
}
|
}
|
||||||
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
|
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
|
||||||
throw new Error("forbidden application coupling");
|
throw new Error("forbidden application coupling");
|
||||||
}
|
}
|
||||||
if (!config.networks || !config.networks.thothii) throw new Error("base stack must define the thothii network");
|
if (!config.networks || !config.networks.thothii) throw new Error("base stack must define the thothii network");
|
||||||
for (const volume of ["qdrant-data", "embedding-models"]) {
|
for (const volume of ["catalog-data", "qdrant-data", "embedding-models"]) {
|
||||||
if (!config.volumes || !config.volumes[volume]) throw new Error(`missing required volume: ${volume}`);
|
if (!config.volumes || !config.volumes[volume]) throw new Error(`missing required volume: ${volume}`);
|
||||||
}
|
}
|
||||||
if (profile === "local") {
|
if (profile === "local") {
|
||||||
@@ -114,8 +114,8 @@ const bundleSecrets = runtimeSecrets.filter(
|
|||||||
if (bundleSecrets.length !== 1) {
|
if (bundleSecrets.length !== 1) {
|
||||||
throw new Error("core must receive exactly one canonical runtime secret bundle");
|
throw new Error("core must receive exactly one canonical runtime secret bundle");
|
||||||
}
|
}
|
||||||
if (profile === "local" && runtimeSecrets.length !== 1) {
|
if (profile === "local" && runtimeSecrets.length !== 2) {
|
||||||
throw new Error("local core must receive only the canonical runtime secret bundle");
|
throw new Error("local core must receive only its runtime bundle and catalog password");
|
||||||
}
|
}
|
||||||
if (profile === "server") {
|
if (profile === "server") {
|
||||||
const targets = new Set(runtimeSecrets.map((secret) => secret.target));
|
const targets = new Set(runtimeSecrets.map((secret) => secret.target));
|
||||||
@@ -171,14 +171,14 @@ const fs = require("fs");
|
|||||||
|
|
||||||
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
|
const config = JSON.parse(fs.readFileSync(process.argv[2], "utf8"));
|
||||||
const services = Object.keys(config.services).sort();
|
const services = Object.keys(config.services).sort();
|
||||||
if (services.join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
if (services.join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||||
throw new Error("mandatory stack must include core, frontend, qdrant, embedding, and embedding-model-init");
|
throw new Error("mandatory stack must include catalog-db, core, frontend, qdrant, embedding, and embedding-model-init");
|
||||||
}
|
}
|
||||||
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
|
if (/omics_portal|chirone|localllm_default|\/home\/chirone/i.test(JSON.stringify(config))) {
|
||||||
throw new Error("forbidden application coupling");
|
throw new Error("forbidden application coupling");
|
||||||
}
|
}
|
||||||
if (!config.networks || !config.networks.thothii) throw new Error("base stack must define the thothii network");
|
if (!config.networks || !config.networks.thothii) throw new Error("base stack must define the thothii network");
|
||||||
for (const volume of ["settings", "pi-state", "workspace-registry", "sessions", "qdrant-data", "embedding-models"]) {
|
for (const volume of ["settings", "pi-state", "workspace-registry", "sessions", "catalog-data", "qdrant-data", "embedding-models"]) {
|
||||||
if (!config.volumes || !config.volumes[volume]) throw new Error(`missing required volume: ${volume}`);
|
if (!config.volumes || !config.volumes[volume]) throw new Error(`missing required volume: ${volume}`);
|
||||||
}
|
}
|
||||||
if (!Object.hasOwn(config.services.core.environment || {}, "THT_LLM_URL")) {
|
if (!Object.hasOwn(config.services.core.environment || {}, "THT_LLM_URL")) {
|
||||||
@@ -222,10 +222,11 @@ if (config.services.core.environment?.THT_SECRETS_FILE !== "/run/secrets/thothii
|
|||||||
throw new Error("core must read the canonical runtime secret bundle from /run/secrets");
|
throw new Error("core must read the canonical runtime secret bundle from /run/secrets");
|
||||||
}
|
}
|
||||||
const runtimeSecrets = config.services.core.secrets || [];
|
const runtimeSecrets = config.services.core.secrets || [];
|
||||||
if (runtimeSecrets.length !== 1
|
const hasTarget = (name) => runtimeSecrets.some((secret) => secret.target === name || secret.target?.endsWith(`/${name}`));
|
||||||
|| runtimeSecrets[0].source !== "thothii_secrets"
|
if (runtimeSecrets.length !== 2
|
||||||
|| runtimeSecrets[0].target !== "thothii.secrets") {
|
|| !hasTarget("thothii.secrets")
|
||||||
throw new Error("core must receive exactly the canonical runtime secret bundle");
|
|| !hasTarget("catalog_runtime_password")) {
|
||||||
|
throw new Error("core must receive only the canonical runtime bundle and catalog password");
|
||||||
}
|
}
|
||||||
if ((config.services.frontend.secrets || []).length !== 0) {
|
if ((config.services.frontend.secrets || []).length !== 0) {
|
||||||
throw new Error("frontend must not receive runtime secrets");
|
throw new Error("frontend must not receive runtime secrets");
|
||||||
|
|||||||
@@ -275,8 +275,8 @@ overrides:
|
|||||||
)
|
)
|
||||||
$render = Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("config", "--services")) -Label "render Windows Compose from spaced path"
|
$render = Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("config", "--services")) -Label "render Windows Compose from spaced path"
|
||||||
$services = @($render.StdOut -split "`r?`n" | Where-Object { $_ } | Sort-Object)
|
$services = @($render.StdOut -split "`r?`n" | Where-Object { $_ } | Sort-Object)
|
||||||
if (($services -join ",") -ne "core,embedding,embedding-model-init,frontend,qdrant") {
|
if (($services -join ",") -ne "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||||
throw "rendered Windows stack must contain the canonical five services"
|
throw "rendered Windows stack must contain the canonical six services"
|
||||||
}
|
}
|
||||||
Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("config", "--quiet")) -Label "validate Windows Compose from spaced path" | Out-Null
|
Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("config", "--quiet")) -Label "validate Windows Compose from spaced path" | Out-Null
|
||||||
|
|
||||||
@@ -287,8 +287,8 @@ overrides:
|
|||||||
Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("up", "--detach", "--wait", "--wait-timeout", "180")) -Label "start canonical Windows stack" -TimeoutSeconds 300 | Out-Null
|
Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("up", "--detach", "--wait", "--wait-timeout", "180")) -Label "start canonical Windows stack" -TimeoutSeconds 300 | Out-Null
|
||||||
$running = Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("ps", "--status", "running", "--services")) -Label "inspect running Windows services"
|
$running = Invoke-BoundedNative -FilePath "docker" -Arguments ($composeArguments + @("ps", "--status", "running", "--services")) -Label "inspect running Windows services"
|
||||||
$runningServices = @($running.StdOut -split "`r?`n" | Where-Object { $_ } | Sort-Object)
|
$runningServices = @($running.StdOut -split "`r?`n" | Where-Object { $_ } | Sort-Object)
|
||||||
if (($runningServices -join ",") -ne "core,embedding,frontend,qdrant") {
|
if (($runningServices -join ",") -ne "catalog-db,core,embedding,frontend,qdrant") {
|
||||||
throw "bounded Windows startup did not leave the four long-running services ready"
|
throw "bounded Windows startup did not leave the five long-running services ready"
|
||||||
}
|
}
|
||||||
Invoke-BoundedNative -FilePath $tht -Arguments @("--installation", $installation, "status") -Label "invoke installation-aware Windows tht in spaced path" | Out-Null
|
Invoke-BoundedNative -FilePath $tht -Arguments @("--installation", $installation, "status") -Label "invoke installation-aware Windows tht in spaced path" | Out-Null
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1945,7 +1945,7 @@ verify_local_installation_example() {
|
|||||||
const fs = require("fs");
|
const fs = require("fs");
|
||||||
const [path, authConfigRoot] = process.argv.slice(2);
|
const [path, authConfigRoot] = process.argv.slice(2);
|
||||||
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||||
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
if (Object.keys(config.services).sort().join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||||
throw new Error("local installation example must render the internal semantic stack");
|
throw new Error("local installation example must render the internal semantic stack");
|
||||||
}
|
}
|
||||||
const authMount = (config.services.core.volumes || []).find(
|
const authMount = (config.services.core.volumes || []).find(
|
||||||
@@ -2065,7 +2065,7 @@ verify_server_installation_example() {
|
|||||||
const fs = require("fs");
|
const fs = require("fs");
|
||||||
const [path, authConfigRoot] = process.argv.slice(2);
|
const [path, authConfigRoot] = process.argv.slice(2);
|
||||||
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||||
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
if (Object.keys(config.services).sort().join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||||
throw new Error("server installation example must render the internal semantic stack");
|
throw new Error("server installation example must render the internal semantic stack");
|
||||||
}
|
}
|
||||||
const core = config.services.core;
|
const core = config.services.core;
|
||||||
@@ -2216,7 +2216,7 @@ verify_compose_fixtures() {
|
|||||||
const fs = require("fs");
|
const fs = require("fs");
|
||||||
const [path, profile, authConfigRoot] = process.argv.slice(2);
|
const [path, profile, authConfigRoot] = process.argv.slice(2);
|
||||||
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
const config = JSON.parse(fs.readFileSync(path, "utf8"));
|
||||||
if (Object.keys(config.services).sort().join(",") !== "core,embedding,embedding-model-init,frontend,qdrant") {
|
if (Object.keys(config.services).sort().join(",") !== "catalog-db,core,embedding,embedding-model-init,frontend,qdrant") {
|
||||||
throw new Error(profile + ": mandatory stack must include the internal semantic services");
|
throw new Error(profile + ": mandatory stack must include the internal semantic services");
|
||||||
}
|
}
|
||||||
const core = config.services.core;
|
const core = config.services.core;
|
||||||
|
|||||||
Reference in New Issue
Block a user