diff --git a/CONTEXT.md b/CONTEXT.md index 70908c4a..5cb94c78 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -378,6 +378,14 @@ esplicitamente selezionate; le richieste ampie vengono divise in batch bounded, l'utente imposta i flag dopo aver rivisto la proposta completa. _Avoid_: PII filter, sample filter +**Sensitive Data Suggestion Run** — Il tentativo amministrativo tracciato con cui il modello +propone Sensitive Data Flag dai soli metadati strutturali. Conserva stato e conteggi aggregati, +ma non i suggerimenti per colonna, che restano una proposta transitoria fino al salvataggio umano. + +**Sensitive Data Suggestion Event** — Una riga testuale ordinata e sanitizzata che registra +l'avvio, l'esito o l'errore di una Sensitive Data Suggestion Run senza conservare prompt, +risposte grezze del provider o proposte per colonna. + **Introspection Capability** — Una categoria di struttura fisica che una Database Binding può osservare, come tabelle, colonne, relazioni, indici o enum. Una capability non disponibile è distinta da una capability osservata che non ha restituito elementi. diff --git a/PROJECT_STATE.md b/PROJECT_STATE.md index 9dbdfbde..5007fddb 100644 --- a/PROJECT_STATE.md +++ b/PROJECT_STATE.md @@ -66,13 +66,21 @@ Git and are supplied only through installation-local protected files. ## Database management The database, table, and authoritative physical-schema catalog slices are implemented. Database -management opens a responsive AG Grid master-detail surface, lists every YAML workspace, creates -at most one PostgreSQL database configuration per workspace, edits direct PostgreSQL, REST API, or -SSH-tunnel installation bindings, replaces write-only encrypted secrets, and tests supported -connector bindings. +Management now opens the Fleet Ledger presentation by default inside `AppShell`, lists every YAML +workspace, creates at most one PostgreSQL database configuration per workspace, edits direct +PostgreSQL, REST API, or SSH-tunnel installation bindings, replaces write-only encrypted secrets, +and tests supported connector bindings. The surface keeps one responsive AG Grid visible at a time: +databases lead to tables, tables lead to columns, and relationships are a sibling database view. +Parent navigation remains explicit through the breadcrumb and emphasized back control. -Configured databases use pure hierarchical navigation through `Overview`, `Tables`, and -`Relationships`; a selected table has `Overview` and `Columns`. Physical membership, source +Selection-scoped operations use one action selector plus an explicit **Run** control; ineligible +actions remain visible with their disabled reason, while row-scoped actions stay in the pinned final +column. The KPI strip reads installation-wide or selected-database aggregates from +`GET /catalog/metrics`. Database configuration, metadata editors, synchronization history, +description history, and sensitive-field review/history use the production APIs in right-side +drawers rather than prototype fixtures; closing a history drawer does not stop its background run. + +Physical membership, source comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are projections of the external schema. They cannot be created, renamed, or structurally edited by hand, but administrators can explicitly clear catalog tables, columns, or relationships without @@ -82,6 +90,12 @@ relationships. Curated and generated descriptions are editable; generated descri and Database Management can generate or consolidate them for selected tables, selected columns, all targets, or only targets whose Generated Description is missing. +The previous Database Management renderer remains a temporary comparison fallback for development +and staging only: `?db-ui=legacy` is honored in Vite development or when +`VITE_DB_MANAGEMENT_LEGACY=true`; it is not a production presentation. The standalone Fleet Ledger +prototype on port `5173` also remains temporary until owner acceptance of the integrated surface, +after which both migration aids can be removed. + Schema refresh is one durable asynchronous engine with database-table, database-column, selected-table-column, relationship, and full-database actions. Database-level menus expose the table, all-column, relationship, and full scopes separately; selecting tables exposes column @@ -115,6 +129,10 @@ only on structural metadata for one selected database, selected tables, or selec backend divides large scopes into deterministic model requests of at most ten columns, also bounded by helper message size, and combines their results, but the proposal remains an unsaved draft until the human reviews and saves it. +Each started suggestion attempt records a separate Sensitive Data Suggestion Run with aggregate +counters and safe ordered events. This operational history never stores per-column proposals, +prompts, raw model output, or provider diagnostics; reloading still discards an unsaved review +draft. For unprotected columns, up to five source rows and five representative non-null values may be sent transiently to the configured model provider. Protected columns are omitted from source reads and replaced in the prompt by deterministic plausible values derived only from their metadata. Existing diff --git a/docs/adr/0011-gate-source-samples-with-a-sensitive-data-flag.md b/docs/adr/0011-gate-source-samples-with-a-sensitive-data-flag.md index 9a8563cb..20d5670f 100644 --- a/docs/adr/0011-gate-source-samples-with-a-sensitive-data-flag.md +++ b/docs/adr/0011-gate-source-samples-with-a-sensitive-data-flag.md @@ -5,9 +5,10 @@ status: accepted # Gate source samples with a Sensitive Data Flag Each Catalog Column has one human-set `sensitive` boolean, defaulting to `false`. An AI may prefill -draft suggestions from structural metadata only, but the user decides and only the boolean is -persisted; there is no rationale, history, audit ledger, fingerprint, review state, or retroactive -regeneration of existing descriptions. +draft suggestions from structural metadata only, but the user decides and the Catalog Column +persists only the current boolean. It stores no rationale, prior flag values, proposal fingerprint, +review state, or audit ledger, and changing the flag does not retroactively regenerate existing +descriptions. The user starts a suggestion from an explicit selection at database, table, or column level. A database request accepts exactly one selected database; a table or column request contains only the @@ -16,6 +17,13 @@ metadata into deterministic model requests of at most ten columns, also bounded size, and returns one combined draft for human review. No flag changes until the user saves the reviewed draft. +Each started suggestion attempt persists a separate Sensitive Data Suggestion Run containing its +database, scope, selected model, lifecycle status, aggregate suggestion counts, timestamps, +sanitized error summary, and ordered sanitized events. It does not persist selected target IDs, +per-column proposals, prompts, raw provider output, or provider diagnostics. This is operational +history of the AI attempt, not an audit trail of human flag decisions; reloading discards an unsaved +proposal. + Description generation extends ADR-0010 by sending bounded real values when `sensitive` is false and deterministic plausible synthetic values when it is true, without identifying the synthetic values to the model. The false default deliberately favors the expected stable schemas and the diff --git a/docs/agents/domain.md b/docs/agents/domain.md index dead047c..2e67b213 100644 --- a/docs/agents/domain.md +++ b/docs/agents/domain.md @@ -5,3 +5,6 @@ This is a single-context repository. Before exploring, read `CONTEXT.md` at the repository root and the relevant decisions in `docs/adr/`. Use the project's terminology from `CONTEXT.md` in issue titles, proposals, and tests. Surface conflicts with an ADR instead of silently overriding it. + +When the vocabulary is missing or ambiguous, clarify the term with the `domain-modeling` skill and +record the result in `CONTEXT.md` or an ADR before relying on it in implementation work. diff --git a/docs/agents/issue-tracker.md b/docs/agents/issue-tracker.md index 207d49ce..9664f780 100644 --- a/docs/agents/issue-tracker.md +++ b/docs/agents/issue-tracker.md @@ -9,7 +9,10 @@ Issues and specs for this repository live in the self-hosted Gitea repository: authenticated token with issue scope is available. - **Read and list issues**: use the Gitea web UI or authenticated API; include labels and comments. - **Apply or remove labels**: use the issue's label controls or the Gitea API. -- **Comment and close**: use the issue page or the Gitea API. +- **Comment and close**: record the resolution on the issue before closing it, using the issue page + or the Gitea API. +- In durable repository documents, cite an issue with its title and canonical link rather than only + its number. - Do not use `gh issue ...` for this repository: the `github` remote is a mirror, not the canonical issue tracker. diff --git a/docs/architecture/components.md b/docs/architecture/components.md index 084a9946..ae25112a 100644 --- a/docs/architecture/components.md +++ b/docs/architecture/components.md @@ -67,6 +67,26 @@ sequenceDiagram The backend uses `ThtRunner` for CLI subprocesses, `PiProcessManager` for one Pi process per session, `SessionBridge` to adapt RPC events, and `SseHub` to distribute them to clients. +## Database Management frontend + +`AppShell` mounts Fleet Ledger as the default Database Management presentation. The controller keeps +the existing React Query, AG Grid, permission, dirty-state, synchronization, SSE, and polling +contracts; Fleet Ledger changes the information architecture without introducing a second catalog +client. It shows exactly one grid at a time along the database → table → column hierarchy, with +relationships as a sibling database view. An emphasized back control and breadcrumb move to the +parent view. + +Selected-row operations are exposed through a single action selector and explicit **Run** control. +Row-specific actions remain icon controls in the pinned final column. Configuration, metadata +editing, synchronization, description generation, and sensitive-field review/history use the real +catalog state and open in right-side drawers. A drawer can close independently of a durable run. + +The KPI strip calls `GET /catalog/metrics`: omitting `databaseId` returns installation-wide catalog +aggregates, while supplying it scopes the same aggregate contract to the selected database. The +previous renderer is reachable only as a temporary development/staging comparison with +`?db-ui=legacy` when Vite development or `VITE_DB_MANAGEMENT_LEGACY=true` enables it. The standalone +prototype on port `5173` remains outside `AppShell` only until the integrated surface is accepted. + ## Catalog description generation Catalog description generation is a backend-owned administrative operation, separate from the @@ -83,6 +103,12 @@ or second orchestration subsystem. A target receives at most one provider retry; exhausted technical batches fail the run. Stale work is marked interrupted at startup and must be explicitly unlocked; it never resumes automatically. +Sensitive-field suggestion generation remains a synchronous administrative request, but each +attempt has its own durable run and ordered sanitized events. This history is separate from +Description Generation because its lifecycle and counters differ. Only execution metadata and +aggregate counts are stored; proposed flags, prompts, raw model output, and provider diagnostics +remain transient. + ## Main backend classes The diagram shows the classes that form the bridge between the browser, Pi, and `tht`. Fastify routes receive requests and delegate to these services. diff --git a/docs/operations/database-management.md b/docs/operations/database-management.md index df7695b3..af2a86cc 100644 --- a/docs/operations/database-management.md +++ b/docs/operations/database-management.md @@ -15,6 +15,31 @@ nullability, primary-key positions, and ordered foreign-key pairs are observatio schema and cannot be manually created, renamed, or structurally edited. Descriptions are the editable metadata. +## Navigate the Fleet Ledger surface + +Database Management opens Fleet Ledger inside the normal application shell. Only one data grid is +shown at a time: choose a database to see its tables, choose a table to see its columns, or open the +database's relationships view. Use the emphasized back control or breadcrumb to return to the parent +grid. + +The KPI strip reports tables, columns, sensitive columns, relationships, and description coverage. +It uses `GET /catalog/metrics` without `databaseId` for installation totals and with `databaseId` for +the current database. Choose a selection-scoped operation from the action selector and then press +**Run**; unavailable operations remain listed with an explanation. Row-specific actions are the icon +controls in the final column, and each navigation or action icon has an immediate conceptual tooltip. + +Configuration, object details, metadata editors, synchronization history, description history, +sensitive-field review, and suggestion-run history open in right-side drawers backed by the +production catalog APIs. +Closing a history drawer does not cancel a durable background run. Existing permission checks, +dirty/busy navigation guards, stale-state handling, and write-only secret behavior continue to +apply. + +For temporary comparison in development or staging, add `?db-ui=legacy`; the parameter is honored +only by Vite development or an environment explicitly configured with +`VITE_DB_MANAGEMENT_LEGACY=true`. The separate prototype on port `5173` is not the application and +remains available only until the integrated Fleet Ledger surface passes owner acceptance. + ## Configure and test a database 1. Open **Database Management** and choose a workspace. @@ -54,6 +79,11 @@ administrator can ask the configured model to suggest flags from structural meta schema, table and column names, data types, nullability, primary keys, and foreign keys). Suggestions remain an unsaved draft until a human reviews and saves them. +The page exposes separate histories for description generation and sensitive-field suggestions. +Sensitive-suggestion history stores the selected model, scope, status, aggregate counts, timestamps, +and sanitized events. It does not store the proposed per-column flags, prompts, raw model output, or +provider diagnostics; closing an unsaved review still discards that draft. + For a column with `sensitive=false`, the worker may read at most five source rows and five representative non-null values through a read-only connector. For `sensitive=true`, the source query does not request that column's values; deterministic plausible values derived only from its name and