ThothII/docs/operations/database-management.md

# Database management

Database Management is an administrative catalog for an external PostgreSQL schema. It is separate
from workspace preprocessing and, today, does not change the DWH binding used by the NL→SQL
session workflow.

## What the catalog owns

For each YAML workspace, an administrator may configure at most one Metadata Catalog binding. It holds
the database name, schema, connection binding, write-only encrypted secrets, observed physical
schema, optional curated descriptions, generated descriptions, and durable operation history.

It does **not** become the external source of truth. Tables, columns, types, defaults,
nullability, primary-key positions, and ordered foreign-key pairs are observations of the source
schema and cannot be manually created, renamed, or structurally edited. Descriptions are the
editable metadata.

## Navigate the Fleet Ledger surface

Database Management opens Fleet Ledger inside the normal application shell. Only one data grid is
shown at a time: choose a database to see its tables, choose a table to see its columns, or open the
database's relationships view. Use the emphasized back control or breadcrumb to return to the parent
grid.

The KPI strip reports tables, columns, sensitive columns, relationships, and description coverage.
It uses `GET /catalog/metrics` without `databaseId` for installation totals and with `databaseId` for
the current database. Choose a selection-scoped operation from the action selector and then press
**Run**; unavailable operations remain listed with an explanation. Row-specific actions are the icon
controls in the final column, and each navigation or action icon has an immediate conceptual tooltip.
An unconfigured workspace exposes **Configure catalog** directly on its row; there is no global
database-creation action and the selected workspace cannot be changed in the configuration form.

The master grid keeps three independent states visible:

- **Revision / Evidence** comes from the active immutable workspace revision. Filesystem Evidence is
  materialized with that revision; remote Evidence is reported as configured-but-unverified or as
  requiring credentials.
- **NL→SQL runtime** is calculated from the workspace DWH/Evidence requirements and runtime secret
  store. It also reports transports, such as SSH, that are diagnostic-only and unsupported by sessions.
- **Metadata Catalog** reports whether the installation-local catalog configuration exists, then shows
  its separately versioned connection-test or synchronization state.

Configuration, object details, metadata editors, synchronization history, description history,
sensitive-field review, and suggestion-run history open in right-side drawers backed by the
production catalog APIs.
Closing a history drawer does not cancel a durable background run. Existing permission checks,
dirty/busy navigation guards, stale-state handling, and write-only secret behavior continue to
apply.

For temporary comparison in development or staging, add `?db-ui=legacy`; the parameter is honored
only by Vite development or an environment explicitly configured with
`VITE_DB_MANAGEMENT_LEGACY=true`. The separate prototype on port `5173` is not the application and
remains available only until the integrated Fleet Ledger surface passes owner acceptance.

## Configure and test a database

1. Open **Database Management** and find the repository workspace marked **Not configured**.
2. Choose **Configure catalog** on that row. Configure its PostgreSQL catalog binding with
   `postgres_direct`, `rest_api`, or `ssh_tunnel` and
   complete the binding fields that the chosen transport requires.
3. Enter secrets only when replacing them. They remain write-only and are never returned by the
   application.
4. Use **Test connection** whenever you want an informational connectivity check. Its result does
   not enable or disable catalog operations.

SSH uses a private key, optional key passphrase, mandatory `known_hosts`, and optional PostgreSQL
TLS CA/server name. REST prefers `POST /rpc/schema_snapshot`; when it is absent, the catalog may
use the same strict v1 snapshot through one read-only `POST /rpc/run_query`. An unavailable
capability, malformed snapshot, or connector error applies no catalog changes. See the
[schema snapshot contract](../contracts/catalog-schema-snapshot.md).

## Synchronize authoritative schema metadata

Schema synchronization reads the external database and reconciles the installation-local catalog.
It never changes the source database. Every synchronization attempts a fresh connection when it
runs; an unreachable server or rejected credential fails that run without changing catalog data.
The available synchronization scopes are **tables**,
**columns**, **relationships**, and **all**, but the UI exposes them at different levels:

| Location | Action | Effective scope |
| --- | --- | --- |
| Database view | **Synchronize tables** | All tables in the selected database |
| Database view | **Synchronize relationships** | All physical foreign-key relationships in the selected database |
| Database view | **Synchronize all** | Tables, columns, and physical relationships in the selected database |
| Tables view | **Synchronize database tables** | All tables in the selected database. Selecting a table enables the action, but does not narrow its scope. |
| Tables view | **Synchronize columns for selected tables** | Columns belonging to the selected tables |
| Columns view | **Synchronize columns for this table** | All columns belonging to the table currently open. Selecting at least one column enables the action, but does not narrow its scope to that column. |

There is no database-level column action and no synchronization action for an individual column.
The selection requirement in the tables and columns views controls whether the action selector can
be used; it is not always the same as the synchronization target. The **Sync all** button in the
tables view is the direct shortcut for the full-database scope.

Connection tests are informational and are never a synchronization prerequisite. Each
synchronization tests its own access while reading the schema; an unavailable connection fails
that operation with a connector error. Only one catalog operation can be active for a database at
a time; explicit cleanup shares this exclusion.

### What a synchronization does

Every run first creates a durable queued operation and reads a schema snapshot. The scan reports
these phases:

1. Connect to the database.
2. Read tables.
3. Read columns and primary-key positions.
4. Read foreign-key relationships.
5. Calculate the planned catalog difference.
6. Apply only the requested scope.

The scan currently reads the complete physical snapshot, including foreign keys, even when the
requested scope is only tables or columns. This is required by the introspection contract and is
why the log can mention foreign-key reading during a table synchronization. Reading those keys
does not by itself create or update catalog relationships:

- **Synchronize tables** writes table membership and source comments. If a table disappears, its
  catalog columns and physical relationships are removed through the table cascade.
- **Synchronize columns** writes column membership and structural attributes for all tables or for
  the selected table subset. It does not write physical relationships.
- **Synchronize relationships** writes the physical relationships derived from the source foreign
  keys. It does not create generated or manual logical relationships.
- **Synchronize all** applies all three scopes and marks the database schema version as fully
  synchronized.

The run scans first and publishes a durable operation. If it detects a destructive difference, it
requires confirmation and re-scans before applying. You can cancel before apply; completed and
failed runs remain in history. The live log is delivered over SSE with a polling fallback.

The synchronization history records the requested scope, progress phases, planned changes,
confirmation, result counts, and errors. Closing the history drawer does not cancel a running
operation; it can be reopened from the database synchronization history control.

Explicit cleanup is different from source synchronization: administrators can clear selected
table/relationship or column/relationship catalog metadata without changing the external source,
the connection binding, or secrets. Deleting a table cascades to its columns and relationships.

## Generate and consolidate descriptions

Generated descriptions can be requested for selected tables, selected columns, every eligible
target, or targets with a missing generated description. The backend accepts one installation-wide
run and processes targets sequentially. Every catalog column has a **Sensitive** flag, which defaults
to `false`, including after a newly discovered column is synchronized. Before generation, an
administrator can run local sensitivity analysis over the selected database, tables, or columns.
The analysis combines structural metadata with bounded read-only inspection of source values. It
uses no generative AI and no installation-catalog model. Assessments remain an unsaved draft until
a human reviews and saves them; the reviewer may reverse any proposal.

The page exposes separate histories for description generation and sensitivity analysis. Analysis
history stores the local policy version, scope, status, aggregate `sensitive`, `non_sensitive`, and
`unknown` counts, timestamps, and sanitized events. It does not store source values, per-column
proposals, NER spans, or worker diagnostics; closing an unsaved review discards that draft. The
rules, time bounds, and optional CPU-only NER profile are documented in
[Local sensitivity analysis](sensitivity-analysis.md).

For a column with `sensitive=false`, the worker may read at most five source rows and five
representative non-null values through a read-only connector. For `sensitive=true`, the source query
does not request that column's values; deterministic plausible values derived only from its name and
type take their place in the model prompt. The prompt does not identify those values as synthetic, so
the model can still describe the field as if it had received representative data.

Each successful result is persisted immediately. Stop terminates the active helper but retains
earlier results. A helper has at most one provider retry; three consecutively exhausted technical
batches fail the run. Stale queued/running work is marked interrupted at startup and can be
unlocked only when no local worker/helper is live. There is no automatic resume and no public
description-generation CLI.

Review generated text before copying it into the curated **Description** field. Because the flag
defaults to `false`, an administrator must review the classification and mark protected fields before
starting generation. Changing a flag affects future generations only; existing generated or curated
descriptions are not regenerated. Real and substituted samples remain transient and are not persisted
or returned to the browser.

The database-level **Copy generated descriptions to all columns** action applies every non-empty
AI-generated column description to the corresponding curated **Description** field in one atomic
operation. It skips empty generated descriptions, reports copied and skipped counts, and retains the
generated text. Because this can replace reviewed descriptions, the interface requires explicit
confirmation before applying it.

The decisions behind this surface are [ADRs 0001–0011](../adr/0001-postgres-metadata-catalog.md)
and the detailed acceptance record is
[AI catalog description generation acceptance](../testing/2026-08-29-ai-catalog-description-generation-acceptance.md).