docs: reconcile catalog run history and fleet state
This commit is contained in:
@@ -378,6 +378,14 @@ esplicitamente selezionate; le richieste ampie vengono divise in batch bounded,
|
|||||||
l'utente imposta i flag dopo aver rivisto la proposta completa.
|
l'utente imposta i flag dopo aver rivisto la proposta completa.
|
||||||
_Avoid_: PII filter, sample filter
|
_Avoid_: PII filter, sample filter
|
||||||
|
|
||||||
|
**Sensitive Data Suggestion Run** — Il tentativo amministrativo tracciato con cui il modello
|
||||||
|
propone Sensitive Data Flag dai soli metadati strutturali. Conserva stato e conteggi aggregati,
|
||||||
|
ma non i suggerimenti per colonna, che restano una proposta transitoria fino al salvataggio umano.
|
||||||
|
|
||||||
|
**Sensitive Data Suggestion Event** — Una riga testuale ordinata e sanitizzata che registra
|
||||||
|
l'avvio, l'esito o l'errore di una Sensitive Data Suggestion Run senza conservare prompt,
|
||||||
|
risposte grezze del provider o proposte per colonna.
|
||||||
|
|
||||||
**Introspection Capability** — Una categoria di struttura fisica che una Database Binding
|
**Introspection Capability** — Una categoria di struttura fisica che una Database Binding
|
||||||
può osservare, come tabelle, colonne, relazioni, indici o enum. Una capability non disponibile
|
può osservare, come tabelle, colonne, relazioni, indici o enum. Una capability non disponibile
|
||||||
è distinta da una capability osservata che non ha restituito elementi.
|
è distinta da una capability osservata che non ha restituito elementi.
|
||||||
|
|||||||
+24
-6
@@ -66,13 +66,21 @@ Git and are supplied only through installation-local protected files.
|
|||||||
## Database management
|
## Database management
|
||||||
|
|
||||||
The database, table, and authoritative physical-schema catalog slices are implemented. Database
|
The database, table, and authoritative physical-schema catalog slices are implemented. Database
|
||||||
management opens a responsive AG Grid master-detail surface, lists every YAML workspace, creates
|
Management now opens the Fleet Ledger presentation by default inside `AppShell`, lists every YAML
|
||||||
at most one PostgreSQL database configuration per workspace, edits direct PostgreSQL, REST API, or
|
workspace, creates at most one PostgreSQL database configuration per workspace, edits direct
|
||||||
SSH-tunnel installation bindings, replaces write-only encrypted secrets, and tests supported
|
PostgreSQL, REST API, or SSH-tunnel installation bindings, replaces write-only encrypted secrets,
|
||||||
connector bindings.
|
and tests supported connector bindings. The surface keeps one responsive AG Grid visible at a time:
|
||||||
|
databases lead to tables, tables lead to columns, and relationships are a sibling database view.
|
||||||
|
Parent navigation remains explicit through the breadcrumb and emphasized back control.
|
||||||
|
|
||||||
Configured databases use pure hierarchical navigation through `Overview`, `Tables`, and
|
Selection-scoped operations use one action selector plus an explicit **Run** control; ineligible
|
||||||
`Relationships`; a selected table has `Overview` and `Columns`. Physical membership, source
|
actions remain visible with their disabled reason, while row-scoped actions stay in the pinned final
|
||||||
|
column. The KPI strip reads installation-wide or selected-database aggregates from
|
||||||
|
`GET /catalog/metrics`. Database configuration, metadata editors, synchronization history,
|
||||||
|
description history, and sensitive-field review/history use the production APIs in right-side
|
||||||
|
drawers rather than prototype fixtures; closing a history drawer does not stop its background run.
|
||||||
|
|
||||||
|
Physical membership, source
|
||||||
comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are
|
comments, column types/default/nullability/PK positions, and constraint-level ordered FK pairs are
|
||||||
projections of the external schema. They cannot be created, renamed, or structurally edited by
|
projections of the external schema. They cannot be created, renamed, or structurally edited by
|
||||||
hand, but administrators can explicitly clear catalog tables, columns, or relationships without
|
hand, but administrators can explicitly clear catalog tables, columns, or relationships without
|
||||||
@@ -82,6 +90,12 @@ relationships. Curated and generated descriptions are editable; generated descri
|
|||||||
and Database Management can generate or consolidate them for selected tables, selected columns,
|
and Database Management can generate or consolidate them for selected tables, selected columns,
|
||||||
all targets, or only targets whose Generated Description is missing.
|
all targets, or only targets whose Generated Description is missing.
|
||||||
|
|
||||||
|
The previous Database Management renderer remains a temporary comparison fallback for development
|
||||||
|
and staging only: `?db-ui=legacy` is honored in Vite development or when
|
||||||
|
`VITE_DB_MANAGEMENT_LEGACY=true`; it is not a production presentation. The standalone Fleet Ledger
|
||||||
|
prototype on port `5173` also remains temporary until owner acceptance of the integrated surface,
|
||||||
|
after which both migration aids can be removed.
|
||||||
|
|
||||||
Schema refresh is one durable asynchronous engine with database-table, database-column,
|
Schema refresh is one durable asynchronous engine with database-table, database-column,
|
||||||
selected-table-column, relationship, and full-database actions. Database-level menus expose the
|
selected-table-column, relationship, and full-database actions. Database-level menus expose the
|
||||||
table, all-column, relationship, and full scopes separately; selecting tables exposes column
|
table, all-column, relationship, and full scopes separately; selecting tables exposes column
|
||||||
@@ -115,6 +129,10 @@ only on structural metadata for one selected database, selected tables, or selec
|
|||||||
backend divides large scopes into deterministic model requests of at most ten columns, also bounded
|
backend divides large scopes into deterministic model requests of at most ten columns, also bounded
|
||||||
by helper message size, and combines their results, but the proposal remains an unsaved draft until
|
by helper message size, and combines their results, but the proposal remains an unsaved draft until
|
||||||
the human reviews and saves it.
|
the human reviews and saves it.
|
||||||
|
Each started suggestion attempt records a separate Sensitive Data Suggestion Run with aggregate
|
||||||
|
counters and safe ordered events. This operational history never stores per-column proposals,
|
||||||
|
prompts, raw model output, or provider diagnostics; reloading still discards an unsaved review
|
||||||
|
draft.
|
||||||
For unprotected columns, up to five source rows and five representative non-null values may be sent
|
For unprotected columns, up to five source rows and five representative non-null values may be sent
|
||||||
transiently to the configured model provider. Protected columns are omitted from source reads and
|
transiently to the configured model provider. Protected columns are omitted from source reads and
|
||||||
replaced in the prompt by deterministic plausible values derived only from their metadata. Existing
|
replaced in the prompt by deterministic plausible values derived only from their metadata. Existing
|
||||||
|
|||||||
@@ -5,9 +5,10 @@ status: accepted
|
|||||||
# Gate source samples with a Sensitive Data Flag
|
# Gate source samples with a Sensitive Data Flag
|
||||||
|
|
||||||
Each Catalog Column has one human-set `sensitive` boolean, defaulting to `false`. An AI may prefill
|
Each Catalog Column has one human-set `sensitive` boolean, defaulting to `false`. An AI may prefill
|
||||||
draft suggestions from structural metadata only, but the user decides and only the boolean is
|
draft suggestions from structural metadata only, but the user decides and the Catalog Column
|
||||||
persisted; there is no rationale, history, audit ledger, fingerprint, review state, or retroactive
|
persists only the current boolean. It stores no rationale, prior flag values, proposal fingerprint,
|
||||||
regeneration of existing descriptions.
|
review state, or audit ledger, and changing the flag does not retroactively regenerate existing
|
||||||
|
descriptions.
|
||||||
|
|
||||||
The user starts a suggestion from an explicit selection at database, table, or column level. A
|
The user starts a suggestion from an explicit selection at database, table, or column level. A
|
||||||
database request accepts exactly one selected database; a table or column request contains only the
|
database request accepts exactly one selected database; a table or column request contains only the
|
||||||
@@ -16,6 +17,13 @@ metadata into deterministic model requests of at most ten columns, also bounded
|
|||||||
size, and returns one combined draft for human review. No flag changes until the user saves the
|
size, and returns one combined draft for human review. No flag changes until the user saves the
|
||||||
reviewed draft.
|
reviewed draft.
|
||||||
|
|
||||||
|
Each started suggestion attempt persists a separate Sensitive Data Suggestion Run containing its
|
||||||
|
database, scope, selected model, lifecycle status, aggregate suggestion counts, timestamps,
|
||||||
|
sanitized error summary, and ordered sanitized events. It does not persist selected target IDs,
|
||||||
|
per-column proposals, prompts, raw provider output, or provider diagnostics. This is operational
|
||||||
|
history of the AI attempt, not an audit trail of human flag decisions; reloading discards an unsaved
|
||||||
|
proposal.
|
||||||
|
|
||||||
Description generation extends ADR-0010 by sending bounded real values when `sensitive` is false
|
Description generation extends ADR-0010 by sending bounded real values when `sensitive` is false
|
||||||
and deterministic plausible synthetic values when it is true, without identifying the synthetic
|
and deterministic plausible synthetic values when it is true, without identifying the synthetic
|
||||||
values to the model. The false default deliberately favors the expected stable schemas and the
|
values to the model. The false default deliberately favors the expected stable schemas and the
|
||||||
|
|||||||
@@ -5,3 +5,6 @@ This is a single-context repository.
|
|||||||
Before exploring, read `CONTEXT.md` at the repository root and the relevant decisions in
|
Before exploring, read `CONTEXT.md` at the repository root and the relevant decisions in
|
||||||
`docs/adr/`. Use the project's terminology from `CONTEXT.md` in issue titles, proposals, and
|
`docs/adr/`. Use the project's terminology from `CONTEXT.md` in issue titles, proposals, and
|
||||||
tests. Surface conflicts with an ADR instead of silently overriding it.
|
tests. Surface conflicts with an ADR instead of silently overriding it.
|
||||||
|
|
||||||
|
When the vocabulary is missing or ambiguous, clarify the term with the `domain-modeling` skill and
|
||||||
|
record the result in `CONTEXT.md` or an ADR before relying on it in implementation work.
|
||||||
|
|||||||
@@ -9,7 +9,10 @@ Issues and specs for this repository live in the self-hosted Gitea repository:
|
|||||||
authenticated token with issue scope is available.
|
authenticated token with issue scope is available.
|
||||||
- **Read and list issues**: use the Gitea web UI or authenticated API; include labels and comments.
|
- **Read and list issues**: use the Gitea web UI or authenticated API; include labels and comments.
|
||||||
- **Apply or remove labels**: use the issue's label controls or the Gitea API.
|
- **Apply or remove labels**: use the issue's label controls or the Gitea API.
|
||||||
- **Comment and close**: use the issue page or the Gitea API.
|
- **Comment and close**: record the resolution on the issue before closing it, using the issue page
|
||||||
|
or the Gitea API.
|
||||||
|
- In durable repository documents, cite an issue with its title and canonical link rather than only
|
||||||
|
its number.
|
||||||
- Do not use `gh issue ...` for this repository: the `github` remote is a mirror, not the canonical
|
- Do not use `gh issue ...` for this repository: the `github` remote is a mirror, not the canonical
|
||||||
issue tracker.
|
issue tracker.
|
||||||
|
|
||||||
|
|||||||
@@ -67,6 +67,26 @@ sequenceDiagram
|
|||||||
|
|
||||||
The backend uses `ThtRunner` for CLI subprocesses, `PiProcessManager` for one Pi process per session, `SessionBridge` to adapt RPC events, and `SseHub` to distribute them to clients.
|
The backend uses `ThtRunner` for CLI subprocesses, `PiProcessManager` for one Pi process per session, `SessionBridge` to adapt RPC events, and `SseHub` to distribute them to clients.
|
||||||
|
|
||||||
|
## Database Management frontend
|
||||||
|
|
||||||
|
`AppShell` mounts Fleet Ledger as the default Database Management presentation. The controller keeps
|
||||||
|
the existing React Query, AG Grid, permission, dirty-state, synchronization, SSE, and polling
|
||||||
|
contracts; Fleet Ledger changes the information architecture without introducing a second catalog
|
||||||
|
client. It shows exactly one grid at a time along the database → table → column hierarchy, with
|
||||||
|
relationships as a sibling database view. An emphasized back control and breadcrumb move to the
|
||||||
|
parent view.
|
||||||
|
|
||||||
|
Selected-row operations are exposed through a single action selector and explicit **Run** control.
|
||||||
|
Row-specific actions remain icon controls in the pinned final column. Configuration, metadata
|
||||||
|
editing, synchronization, description generation, and sensitive-field review/history use the real
|
||||||
|
catalog state and open in right-side drawers. A drawer can close independently of a durable run.
|
||||||
|
|
||||||
|
The KPI strip calls `GET /catalog/metrics`: omitting `databaseId` returns installation-wide catalog
|
||||||
|
aggregates, while supplying it scopes the same aggregate contract to the selected database. The
|
||||||
|
previous renderer is reachable only as a temporary development/staging comparison with
|
||||||
|
`?db-ui=legacy` when Vite development or `VITE_DB_MANAGEMENT_LEGACY=true` enables it. The standalone
|
||||||
|
prototype on port `5173` remains outside `AppShell` only until the integrated surface is accepted.
|
||||||
|
|
||||||
## Catalog description generation
|
## Catalog description generation
|
||||||
|
|
||||||
Catalog description generation is a backend-owned administrative operation, separate from the
|
Catalog description generation is a backend-owned administrative operation, separate from the
|
||||||
@@ -83,6 +103,12 @@ or second orchestration subsystem. A target receives at most one provider retry;
|
|||||||
exhausted technical batches fail the run. Stale work is marked interrupted at startup and must be
|
exhausted technical batches fail the run. Stale work is marked interrupted at startup and must be
|
||||||
explicitly unlocked; it never resumes automatically.
|
explicitly unlocked; it never resumes automatically.
|
||||||
|
|
||||||
|
Sensitive-field suggestion generation remains a synchronous administrative request, but each
|
||||||
|
attempt has its own durable run and ordered sanitized events. This history is separate from
|
||||||
|
Description Generation because its lifecycle and counters differ. Only execution metadata and
|
||||||
|
aggregate counts are stored; proposed flags, prompts, raw model output, and provider diagnostics
|
||||||
|
remain transient.
|
||||||
|
|
||||||
## Main backend classes
|
## Main backend classes
|
||||||
|
|
||||||
The diagram shows the classes that form the bridge between the browser, Pi, and `tht`. Fastify routes receive requests and delegate to these services.
|
The diagram shows the classes that form the bridge between the browser, Pi, and `tht`. Fastify routes receive requests and delegate to these services.
|
||||||
|
|||||||
@@ -15,6 +15,31 @@ nullability, primary-key positions, and ordered foreign-key pairs are observatio
|
|||||||
schema and cannot be manually created, renamed, or structurally edited. Descriptions are the
|
schema and cannot be manually created, renamed, or structurally edited. Descriptions are the
|
||||||
editable metadata.
|
editable metadata.
|
||||||
|
|
||||||
|
## Navigate the Fleet Ledger surface
|
||||||
|
|
||||||
|
Database Management opens Fleet Ledger inside the normal application shell. Only one data grid is
|
||||||
|
shown at a time: choose a database to see its tables, choose a table to see its columns, or open the
|
||||||
|
database's relationships view. Use the emphasized back control or breadcrumb to return to the parent
|
||||||
|
grid.
|
||||||
|
|
||||||
|
The KPI strip reports tables, columns, sensitive columns, relationships, and description coverage.
|
||||||
|
It uses `GET /catalog/metrics` without `databaseId` for installation totals and with `databaseId` for
|
||||||
|
the current database. Choose a selection-scoped operation from the action selector and then press
|
||||||
|
**Run**; unavailable operations remain listed with an explanation. Row-specific actions are the icon
|
||||||
|
controls in the final column, and each navigation or action icon has an immediate conceptual tooltip.
|
||||||
|
|
||||||
|
Configuration, object details, metadata editors, synchronization history, description history,
|
||||||
|
sensitive-field review, and suggestion-run history open in right-side drawers backed by the
|
||||||
|
production catalog APIs.
|
||||||
|
Closing a history drawer does not cancel a durable background run. Existing permission checks,
|
||||||
|
dirty/busy navigation guards, stale-state handling, and write-only secret behavior continue to
|
||||||
|
apply.
|
||||||
|
|
||||||
|
For temporary comparison in development or staging, add `?db-ui=legacy`; the parameter is honored
|
||||||
|
only by Vite development or an environment explicitly configured with
|
||||||
|
`VITE_DB_MANAGEMENT_LEGACY=true`. The separate prototype on port `5173` is not the application and
|
||||||
|
remains available only until the integrated Fleet Ledger surface passes owner acceptance.
|
||||||
|
|
||||||
## Configure and test a database
|
## Configure and test a database
|
||||||
|
|
||||||
1. Open **Database Management** and choose a workspace.
|
1. Open **Database Management** and choose a workspace.
|
||||||
@@ -54,6 +79,11 @@ administrator can ask the configured model to suggest flags from structural meta
|
|||||||
schema, table and column names, data types, nullability, primary keys, and foreign keys). Suggestions
|
schema, table and column names, data types, nullability, primary keys, and foreign keys). Suggestions
|
||||||
remain an unsaved draft until a human reviews and saves them.
|
remain an unsaved draft until a human reviews and saves them.
|
||||||
|
|
||||||
|
The page exposes separate histories for description generation and sensitive-field suggestions.
|
||||||
|
Sensitive-suggestion history stores the selected model, scope, status, aggregate counts, timestamps,
|
||||||
|
and sanitized events. It does not store the proposed per-column flags, prompts, raw model output, or
|
||||||
|
provider diagnostics; closing an unsaved review still discards that draft.
|
||||||
|
|
||||||
For a column with `sensitive=false`, the worker may read at most five source rows and five
|
For a column with `sensitive=false`, the worker may read at most five source rows and five
|
||||||
representative non-null values through a read-only connector. For `sensitive=true`, the source query
|
representative non-null values through a read-only connector. For `sensitive=true`, the source query
|
||||||
does not request that column's values; deterministic plausible values derived only from its name and
|
does not request that column's values; deterministic plausible values derived only from its name and
|
||||||
|
|||||||
Reference in New Issue
Block a user