ThothII/docs/operations/database-management.md

Database management

Database Management is an administrative catalog for an external PostgreSQL schema. It is separate from workspace preprocessing and, today, does not change the DWH binding used by the NL→SQL session workflow.

What the catalog owns

For each YAML workspace, an administrator may create at most one database configuration. It holds the database name, schema, connection binding, write-only encrypted secrets, observed physical schema, optional curated descriptions, generated descriptions, and durable operation history.

It does not become the external source of truth. Tables, columns, types, defaults, nullability, primary-key positions, and ordered foreign-key pairs are observations of the source schema and cannot be manually created, renamed, or structurally edited. Descriptions are the editable metadata.

Configure and test a database

1. Open Database Management and choose a workspace.
2. Create its PostgreSQL configuration. Choose postgres_direct, rest_api, or ssh_tunnel and complete the binding fields that the chosen transport requires.
3. Enter secrets only when replacing them. They remain write-only and are never returned by the application.
4. Run Test connection before any synchronization.

SSH uses a private key, optional key passphrase, mandatory known_hosts, and optional PostgreSQL TLS CA/server name. REST prefers POST /rpc/schema_snapshot; when it is absent, the catalog may use the same strict v1 snapshot through one read-only POST /rpc/run_query. An unavailable capability, malformed snapshot, or connector error applies no catalog changes. See the schema snapshot contract.

Synchronize authoritative schema metadata

Start a synchronization from a database or a selected table set. The available scopes are tables, columns, relationships, and all. One database can have only one active catalog operation at a time; cleanup shares this exclusion.

The run scans first and publishes a durable operation. If it detects a destructive difference, it requires confirmation and re-scans before applying. You can cancel before apply; completed and failed runs remain in history. The live log is delivered over SSE with a polling fallback.

Explicit cleanup is different from source synchronization: administrators can clear selected table/relationship or column/relationship catalog metadata without changing the external source, the connection binding, or secrets. Deleting a table cascades to its columns and relationships.

Generate and consolidate descriptions

Generated descriptions can be requested for selected tables, selected columns, every eligible target, or targets with a missing generated description. The backend accepts one installation-wide run and processes targets sequentially. Every catalog column has a Sensitive flag, which defaults to false, including after a newly discovered column is synchronized. Before generation, an administrator can ask the configured model to suggest flags from structural metadata only (database, schema, table and column names, data types, nullability, primary keys, and foreign keys). Suggestions remain an unsaved draft until a human reviews and saves them.

For a column with sensitive=false, the worker may read at most five source rows and five representative non-null values through a read-only connector. For sensitive=true, the source query does not request that column's values; deterministic plausible values derived only from its name and type take their place in the model prompt. The prompt does not identify those values as synthetic, so the model can still describe the field as if it had received representative data.

Each successful result is persisted immediately. Stop terminates the active helper but retains earlier results. A helper has at most one provider retry; three consecutively exhausted technical batches fail the run. Stale queued/running work is marked interrupted at startup and can be unlocked only when no local worker/helper is live. There is no automatic resume and no public description-generation CLI.

Review generated text before copying it into the curated Description field. Because the flag defaults to false, an administrator must review the classification and mark protected fields before starting generation. Changing a flag affects future generations only; existing generated or curated descriptions are not regenerated. Real and substituted samples remain transient and are not persisted or returned to the browser.

The decisions behind this surface are ADRs 0001–0011 and the detailed acceptance record is AI catalog description generation acceptance.