Publish documentation for 2f53512e4d
This commit is contained in:
@@ -0,0 +1,316 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
|
||||
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
|
||||
|
||||
<link rel="canonical" href="https://git.tylconsulting.it/thothii-docs/operations/database-management/">
|
||||
<link rel="shortcut icon" href="../../img/favicon.ico">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0" />
|
||||
<title>Databases and descriptions - ThothII Docs</title>
|
||||
<link href="../../css/bootstrap-3.3.7.min.css" rel="stylesheet">
|
||||
<link href="../../css/font-awesome-4.7.0.css" rel="stylesheet">
|
||||
<link href="../../css/base.css" rel="stylesheet">
|
||||
<link rel="stylesheet" href="../../css/highlight.css">
|
||||
<link href="../../stylesheets/extra.css" rel="stylesheet">
|
||||
<!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
|
||||
<!--[if lt IE 9]>
|
||||
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
|
||||
<script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
|
||||
<![endif]-->
|
||||
|
||||
<script src="../../js/jquery-3.2.1.min.js"></script>
|
||||
<script src="../../js/bootstrap-3.3.7.min.js"></script>
|
||||
<script src="../../js/highlight.pack.js"></script>
|
||||
|
||||
<base target="_top">
|
||||
<script>
|
||||
var base_url = '../..';
|
||||
var is_top_frame = false;
|
||||
|
||||
var pageToc = [
|
||||
{title: "Database management", url: "#_top", children: [
|
||||
{title: "What the catalog owns", url: "#what-the-catalog-owns" },
|
||||
{title: "Navigate the Fleet Ledger surface", url: "#navigate-the-fleet-ledger-surface" },
|
||||
{title: "Configure and test a database", url: "#configure-and-test-a-database" },
|
||||
{title: "Synchronize authoritative schema metadata", url: "#synchronize-authoritative-schema-metadata" },
|
||||
{title: "Generate and consolidate descriptions", url: "#generate-and-consolidate-descriptions" },
|
||||
]},
|
||||
];
|
||||
|
||||
</script>
|
||||
<script src="../../js/base.js"></script>
|
||||
<script src="../../javascripts/layout-init.js"></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<script>
|
||||
if (is_top_frame) { $('body').addClass('wm-top-page'); }
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
<div class="container-fluid wm-page-content">
|
||||
<a name="_top"></a>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
|
||||
|
||||
<div class="wm-article-nav pull-right">
|
||||
<a href="../sensitivity-analysis/" class="btn btn-xs btn-default pull-right">
|
||||
Next
|
||||
<i class="fa fa-chevron-right" aria-hidden="true"></i>
|
||||
</a>
|
||||
<a href="../sensitivity-analysis/" class="btn btn-xs btn-link">
|
||||
Sensitive columns
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<div class="wm-article-nav">
|
||||
<a href="../workspaces/" class="btn btn-xs btn-default pull-left">
|
||||
<i class="fa fa-chevron-left" aria-hidden="true"></i>
|
||||
Previous</a><a href="../workspaces/" class="btn btn-xs btn-link">
|
||||
Workspaces
|
||||
</a>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<h1 id="database-management">Database management<a class="headerlink" href="#database-management" title="Permanent link">¶</a></h1>
|
||||
<p>Database Management is the administrative PostgreSQL Metadata Catalog for an external PostgreSQL
|
||||
schema. Its binding and metadata are the sole database source used by workspace preprocessing and
|
||||
the NL→SQL session workflow.</p>
|
||||
<h2 id="what-the-catalog-owns">What the catalog owns<a class="headerlink" href="#what-the-catalog-owns" title="Permanent link">¶</a></h2>
|
||||
<p>For each workspace identity, an administrator may configure at most one Metadata Catalog binding. It holds
|
||||
the database name, schema, connection binding, write-only encrypted secrets, observed physical
|
||||
schema, optional curated descriptions, generated descriptions, and durable operation history.</p>
|
||||
<p>It does <strong>not</strong> become the external source of truth. Tables, columns, types, defaults,
|
||||
nullability, primary-key positions, and ordered foreign-key pairs are observations of the source
|
||||
schema and cannot be manually created, renamed, or structurally edited. Descriptions are the
|
||||
editable metadata.</p>
|
||||
<h2 id="navigate-the-fleet-ledger-surface">Navigate the Fleet Ledger surface<a class="headerlink" href="#navigate-the-fleet-ledger-surface" title="Permanent link">¶</a></h2>
|
||||
<p>Database Management opens Fleet Ledger inside the normal application shell. Only one data grid is
|
||||
shown at a time: choose a database to see its tables, choose a table to see its columns, or open the
|
||||
database's relationships view. Use the emphasized back control or breadcrumb to return to the parent
|
||||
grid.</p>
|
||||
<p>The KPI strip reports tables, columns, sensitive columns, relationships, and description coverage.
|
||||
It uses <code>GET /catalog/metrics</code> without <code>databaseId</code> for installation totals and with <code>databaseId</code> for
|
||||
the current database. Choose a selection-scoped operation from the action selector and then press
|
||||
<strong>Run</strong>; unavailable operations remain listed with an explanation. Row-specific actions are the icon
|
||||
controls in the final column, and each navigation or action icon has an immediate conceptual tooltip.
|
||||
An unconfigured workspace exposes <strong>Configure catalog</strong> directly on its row; there is no global
|
||||
database-creation action and the selected workspace cannot be changed in the configuration form.</p>
|
||||
<p>The master grid keeps three independent states visible:</p>
|
||||
<ul>
|
||||
<li><strong>Revision / Evidence</strong> comes from the active immutable workspace revision. Filesystem Evidence is
|
||||
materialized with that revision; remote Evidence is reported as configured-but-unverified or as
|
||||
requiring credentials.</li>
|
||||
<li><strong>NL→SQL runtime</strong> is calculated from the workspace DWH/Evidence requirements and runtime secret
|
||||
store. It also reports transports, such as SSH, that are diagnostic-only and unsupported by sessions.</li>
|
||||
<li><strong>Metadata Catalog</strong> reports whether the installation-local catalog configuration exists, then shows
|
||||
its separately versioned connection-test or synchronization state.</li>
|
||||
</ul>
|
||||
<p>Configuration, object details, metadata editors, synchronization history, description history,
|
||||
sensitive-field review, and suggestion-run history open in right-side drawers backed by the
|
||||
production catalog APIs.
|
||||
Closing a history drawer does not cancel a durable background run. Existing permission checks,
|
||||
dirty/busy navigation guards, stale-state handling, and write-only secret behavior continue to
|
||||
apply.</p>
|
||||
<p>For temporary comparison in development or staging, add <code>?db-ui=legacy</code>; the parameter is honored
|
||||
only by Vite development or an environment explicitly configured with
|
||||
<code>VITE_DB_MANAGEMENT_LEGACY=true</code>. The separate prototype on port <code>5173</code> is not the application and
|
||||
remains available only until the integrated Fleet Ledger surface passes owner acceptance.</p>
|
||||
<h2 id="configure-and-test-a-database">Configure and test a database<a class="headerlink" href="#configure-and-test-a-database" title="Permanent link">¶</a></h2>
|
||||
<ol>
|
||||
<li>Open <strong>Database Management</strong> and find the repository workspace marked <strong>Not configured</strong>.</li>
|
||||
<li>Choose <strong>Configure catalog</strong> on that row. Configure its PostgreSQL catalog binding with
|
||||
<code>postgres_direct</code>, <code>rest_api</code>, or <code>ssh_tunnel</code> and
|
||||
complete the binding fields that the chosen transport requires.</li>
|
||||
<li>Enter secrets only when replacing them. They remain write-only and are never returned by the
|
||||
application.</li>
|
||||
<li>Use <strong>Test connection</strong> whenever you want an informational connectivity check. Its result does
|
||||
not enable or disable catalog operations.</li>
|
||||
</ol>
|
||||
<p>SSH uses a private key, optional key passphrase, mandatory <code>known_hosts</code>, and optional PostgreSQL
|
||||
TLS CA/server name. REST prefers <code>POST /rpc/schema_snapshot</code>; when it is absent, the catalog may
|
||||
use the same strict v1 snapshot through one read-only <code>POST /rpc/run_query</code>. An unavailable
|
||||
capability, malformed snapshot, or connector error applies no catalog changes. See the
|
||||
<a href="https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/catalog-schema-snapshot.md">schema snapshot contract</a>.</p>
|
||||
<h2 id="synchronize-authoritative-schema-metadata">Synchronize authoritative schema metadata<a class="headerlink" href="#synchronize-authoritative-schema-metadata" title="Permanent link">¶</a></h2>
|
||||
<p>Schema synchronization reads the external database and reconciles the installation-local catalog.
|
||||
It never changes the source database. Every synchronization attempts a fresh connection when it
|
||||
runs; an unreachable server or rejected credential fails that run without changing catalog data.
|
||||
The available synchronization scopes are <strong>tables</strong>,
|
||||
<strong>columns</strong>, <strong>relationships</strong>, and <strong>all</strong>, but the UI exposes them at different levels:</p>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Location</th>
|
||||
<th>Action</th>
|
||||
<th>Effective scope</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>Database view</td>
|
||||
<td><strong>Synchronize tables</strong></td>
|
||||
<td>All tables in the selected database</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Database view</td>
|
||||
<td><strong>Synchronize relationships</strong></td>
|
||||
<td>All physical foreign-key relationships in the selected database</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Database view</td>
|
||||
<td><strong>Synchronize all</strong></td>
|
||||
<td>Tables, columns, and physical relationships in the selected database</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Tables view</td>
|
||||
<td><strong>Synchronize database tables</strong></td>
|
||||
<td>All tables in the selected database. Selecting a table enables the action, but does not narrow its scope.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Tables view</td>
|
||||
<td><strong>Synchronize columns for selected tables</strong></td>
|
||||
<td>Columns belonging to the selected tables</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Columns view</td>
|
||||
<td><strong>Synchronize columns for this table</strong></td>
|
||||
<td>All columns belonging to the table currently open. Selecting at least one column enables the action, but does not narrow its scope to that column.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p>There is no database-level column action and no synchronization action for an individual column.
|
||||
The selection requirement in the tables and columns views controls whether the action selector can
|
||||
be used; it is not always the same as the synchronization target. The <strong>Sync all</strong> button in the
|
||||
tables view is the direct shortcut for the full-database scope.</p>
|
||||
<p>Connection tests are informational and are never a synchronization prerequisite. Each
|
||||
synchronization tests its own access while reading the schema; an unavailable connection fails
|
||||
that operation with a connector error. Only one catalog operation can be active for a database at
|
||||
a time; explicit cleanup shares this exclusion.</p>
|
||||
<h3 id="what-a-synchronization-does">What a synchronization does<a class="headerlink" href="#what-a-synchronization-does" title="Permanent link">¶</a></h3>
|
||||
<p>Every run first creates a durable queued operation and reads a schema snapshot. The scan reports
|
||||
these phases:</p>
|
||||
<ol>
|
||||
<li>Connect to the database.</li>
|
||||
<li>Read tables.</li>
|
||||
<li>Read columns and primary-key positions.</li>
|
||||
<li>Read foreign-key relationships.</li>
|
||||
<li>Calculate the planned catalog difference.</li>
|
||||
<li>Apply only the requested scope.</li>
|
||||
</ol>
|
||||
<p>The scan currently reads the complete physical snapshot, including foreign keys, even when the
|
||||
requested scope is only tables or columns. This is required by the introspection contract and is
|
||||
why the log can mention foreign-key reading during a table synchronization. Reading those keys
|
||||
does not by itself create or update catalog relationships:</p>
|
||||
<ul>
|
||||
<li><strong>Synchronize tables</strong> writes table membership and source comments. If a table disappears, its
|
||||
catalog columns and physical relationships are removed through the table cascade.</li>
|
||||
<li><strong>Synchronize columns</strong> writes column membership and structural attributes for all tables or for
|
||||
the selected table subset. It does not write physical relationships.</li>
|
||||
<li><strong>Synchronize relationships</strong> writes the physical relationships derived from the source foreign
|
||||
keys. It does not create generated or manual logical relationships.</li>
|
||||
<li><strong>Synchronize all</strong> applies all three scopes and marks the database schema version as fully
|
||||
synchronized.</li>
|
||||
</ul>
|
||||
<p>The run scans first and publishes a durable operation. If it detects a destructive difference, it
|
||||
requires confirmation and re-scans before applying. You can cancel before apply; completed and
|
||||
failed runs remain in history. The live log is delivered over SSE with a polling fallback.</p>
|
||||
<p>The synchronization history records the requested scope, progress phases, planned changes,
|
||||
confirmation, result counts, and errors. Closing the history drawer does not cancel a running
|
||||
operation; it can be reopened from the database synchronization history control.</p>
|
||||
<p>Explicit cleanup is different from source synchronization: administrators can clear selected
|
||||
table/relationship or column/relationship catalog metadata without changing the external source,
|
||||
the connection binding, or secrets. Deleting a table cascades to its columns and relationships.</p>
|
||||
<h2 id="generate-and-consolidate-descriptions">Generate and consolidate descriptions<a class="headerlink" href="#generate-and-consolidate-descriptions" title="Permanent link">¶</a></h2>
|
||||
<p>Generated descriptions can be requested for selected tables, selected columns, every eligible
|
||||
target, or targets with a missing generated description. The backend accepts one installation-wide
|
||||
run and processes targets sequentially. Every catalog column has a <strong>Sensitive</strong> flag, which defaults
|
||||
to <code>false</code>, including after a newly discovered column is synchronized. Before generation, an
|
||||
administrator can run local sensitivity analysis over the selected database, tables, or columns.
|
||||
The analysis combines structural metadata with bounded read-only inspection of source values. It
|
||||
uses no generative AI and no installation-catalog model. Assessments remain an unsaved draft until
|
||||
a human reviews and saves them; the reviewer may reverse any proposal.</p>
|
||||
<p>The page exposes separate histories for description generation and sensitivity analysis. Analysis
|
||||
history stores the local policy version, scope, status, aggregate <code>sensitive</code>, <code>non_sensitive</code>, and
|
||||
<code>unknown</code> counts, timestamps, and sanitized events. It does not store source values, per-column
|
||||
proposals, NER spans, or worker diagnostics; closing an unsaved review discards that draft. The
|
||||
rules, time bounds, and optional CPU-only NER profile are documented in
|
||||
<a href="../sensitivity-analysis/">Local sensitivity analysis</a>.</p>
|
||||
<p>For a column with <code>sensitive=false</code>, the worker may read at most five source rows and five
|
||||
representative non-null values through a read-only connector. For <code>sensitive=true</code>, the source query
|
||||
does not request that column's values; deterministic plausible values derived only from its name and
|
||||
type take their place in the model prompt. The prompt does not identify those values as synthetic, so
|
||||
the model can still describe the field as if it had received representative data.</p>
|
||||
<p>Each successful result is persisted immediately. Stop terminates the active helper but retains
|
||||
earlier results. A helper has at most one provider retry; three consecutively exhausted technical
|
||||
batches fail the run. Stale queued/running work is marked interrupted at startup and can be
|
||||
unlocked only when no local worker/helper is live. There is no automatic resume and no public
|
||||
description-generation CLI.</p>
|
||||
<p>Review generated text before copying it into the curated <strong>Description</strong> field. Because the flag
|
||||
defaults to <code>false</code>, an administrator must review the classification and mark protected fields before
|
||||
starting generation. Changing a flag affects future generations only; existing generated or curated
|
||||
descriptions are not regenerated. Real and substituted samples remain transient and are not persisted
|
||||
or returned to the browser.</p>
|
||||
<p>The database-level <strong>Copy generated description to descriptions</strong> action applies every non-empty
|
||||
AI-generated table and column description to the corresponding curated <strong>Description</strong> field in one
|
||||
atomic operation. It skips empty generated descriptions, reports aggregate copied and skipped
|
||||
counts, and retains the generated text. Because this can replace reviewed descriptions, the
|
||||
interface requires explicit confirmation before applying it.</p>
|
||||
<p>The decisions behind this surface are <a href="https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/adr/0001-postgres-metadata-catalog.md">ADRs 0001–0011</a>
|
||||
and the detailed acceptance record is
|
||||
<a href="https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/testing/2026-08-29-ai-catalog-description-generation-acceptance.md">AI catalog description generation acceptance</a>.</p>
|
||||
|
||||
<br>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
|
||||
|
||||
<div class="wm-article-nav pull-right">
|
||||
<a href="../sensitivity-analysis/" class="btn btn-xs btn-default pull-right">
|
||||
Next
|
||||
<i class="fa fa-chevron-right" aria-hidden="true"></i>
|
||||
</a>
|
||||
<a href="../sensitivity-analysis/" class="btn btn-xs btn-link">
|
||||
Sensitive columns
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<div class="wm-article-nav">
|
||||
<a href="../workspaces/" class="btn btn-xs btn-default pull-left">
|
||||
<i class="fa fa-chevron-left" aria-hidden="true"></i>
|
||||
Previous</a><a href="../workspaces/" class="btn btn-xs btn-link">
|
||||
Workspaces
|
||||
</a>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<br>
|
||||
</div>
|
||||
|
||||
<footer class="container-fluid wm-page-content">
|
||||
<p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a> using <a href="https://github.com/gristlabs/mkdocs-windmill">Windmill</a> theme by Grist Labs.</p>
|
||||
</footer>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,245 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
|
||||
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
|
||||
|
||||
<link rel="canonical" href="https://git.tylconsulting.it/thothii-docs/operations/sensitivity-analysis/">
|
||||
<link rel="shortcut icon" href="../../img/favicon.ico">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0" />
|
||||
<title>Sensitive columns - ThothII Docs</title>
|
||||
<link href="../../css/bootstrap-3.3.7.min.css" rel="stylesheet">
|
||||
<link href="../../css/font-awesome-4.7.0.css" rel="stylesheet">
|
||||
<link href="../../css/base.css" rel="stylesheet">
|
||||
<link rel="stylesheet" href="../../css/highlight.css">
|
||||
<link href="../../stylesheets/extra.css" rel="stylesheet">
|
||||
<!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
|
||||
<!--[if lt IE 9]>
|
||||
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
|
||||
<script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
|
||||
<![endif]-->
|
||||
|
||||
<script src="../../js/jquery-3.2.1.min.js"></script>
|
||||
<script src="../../js/bootstrap-3.3.7.min.js"></script>
|
||||
<script src="../../js/highlight.pack.js"></script>
|
||||
|
||||
<base target="_top">
|
||||
<script>
|
||||
var base_url = '../..';
|
||||
var is_top_frame = false;
|
||||
|
||||
var pageToc = [
|
||||
{title: "Local sensitivity analysis", url: "#_top", children: [
|
||||
{title: "Default policy", url: "#default-policy" },
|
||||
{title: "Optional CPU-only GLiNER2 evidence", url: "#optional-cpu-only-gliner2-evidence" },
|
||||
{title: "Prepare and enable the optional profile", url: "#prepare-and-enable-the-optional-profile" },
|
||||
{title: "Acceptance on a real database", url: "#acceptance-on-a-real-database" },
|
||||
]},
|
||||
];
|
||||
|
||||
</script>
|
||||
<script src="../../js/base.js"></script>
|
||||
<script src="../../javascripts/layout-init.js"></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<script>
|
||||
if (is_top_frame) { $('body').addClass('wm-top-page'); }
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
<div class="container-fluid wm-page-content">
|
||||
<a name="_top"></a>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
|
||||
|
||||
<div class="wm-article-nav pull-right">
|
||||
<a href="../../evidence/" class="btn btn-xs btn-default pull-right">
|
||||
Next
|
||||
<i class="fa fa-chevron-right" aria-hidden="true"></i>
|
||||
</a>
|
||||
<a href="../../evidence/" class="btn btn-xs btn-link">
|
||||
Evidence
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<div class="wm-article-nav">
|
||||
<a href="../database-management/" class="btn btn-xs btn-default pull-left">
|
||||
<i class="fa fa-chevron-left" aria-hidden="true"></i>
|
||||
Previous</a><a href="../database-management/" class="btn btn-xs btn-link">
|
||||
Databases and descriptions
|
||||
</a>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<h1 id="local-sensitivity-analysis">Local sensitivity analysis<a class="headerlink" href="#local-sensitivity-analysis" title="Permanent link">¶</a></h1>
|
||||
<p>Database Management can assess selected columns without sending their metadata or contents to a
|
||||
generative model. The feature is advisory: it creates a transient review draft, while the catalog's
|
||||
Sensitive Data Flag changes only when an administrator explicitly saves a choice. The administrator
|
||||
may set either value, including overriding a <code>sensitive</code> proposal.</p>
|
||||
<h2 id="default-policy">Default policy<a class="headerlink" href="#default-policy" title="Permanent link">¶</a></h2>
|
||||
<p><code>SensitivityClassifier</code> is the only column-level decision point. The versioned <code>sensitivity-v4</code>
|
||||
policy combines:</p>
|
||||
<ul>
|
||||
<li>a structural exclusion for declared <code>bigint</code> primary-key columns and undeclared <code>bigint</code>
|
||||
columns following the exact <code>pk</code> naming convention; their values are non-informative identifiers
|
||||
and are therefore not inspected as possible sensitive content. The evidence distinguishes
|
||||
declared constraints from convention-based inference;</li>
|
||||
<li>normalized column-name rules for direct identifiers, credentials, and health data;</li>
|
||||
<li>validated content rules for email, Italian fiscal code and VAT, passport, identity-card and
|
||||
driving-licence identifiers, phone numbers, IBAN/BIC, payment-card checksums, IP/MAC addresses,
|
||||
URLs, UUIDs, access keys, private-key markers, sensitive keys inside bounded recursive JSON, and
|
||||
a reviewed Italian clinical-term dictionary;</li>
|
||||
<li>a conservative length rule: any observed textual value longer than 500 characters makes the
|
||||
entire column sensitive.</li>
|
||||
</ul>
|
||||
<p>One decisive value is enough to classify the column as <code>sensitive</code> and removes it from subsequent
|
||||
passes. Binary or otherwise uninspectable column types are also proposed as <code>sensitive</code>, because
|
||||
their contents cannot be cleared by the textual rules. A completed analysis has only two draft
|
||||
outcomes: <code>sensitive</code> and <code>non_sensitive</code>. Empty or all-null columns are <code>non_sensitive</code> with
|
||||
<code>no_values</code> coverage; a sampled column with no match is <code>non_sensitive</code> with explicit sampled
|
||||
coverage. The administrator remains free to reverse either proposal before saving it.</p>
|
||||
<p>Source reads are database-specific, but decisions are database-independent. PostgreSQL direct and
|
||||
REST <code>run_query</code> adapters project at most 501 characters per value, use only <code>SELECT</code>, and never
|
||||
persist source values. Tables proven to contain at most 1,000 rows are fully scanned. Larger tables
|
||||
are processed breadth-first so every table gets the cheapest pass before any table gets a deeper
|
||||
one:</p>
|
||||
<ol>
|
||||
<li>inspect up to 300 non-null values per unresolved column;</li>
|
||||
<li>inspect up to 700 additional values, reaching a 1,000-value target;</li>
|
||||
<li>for unresolved text, JSON, and XML columns only, inspect up to 2,000 additional values, reaching
|
||||
a 3,000-value target.</li>
|
||||
</ol>
|
||||
<p>At most two tables are scanned concurrently, and the database adapter groups at most 25 columns in
|
||||
one source query. Each probe or value query has a five-second statement timeout; PostgreSQL-wire
|
||||
reads run in a read-only transaction and always end with rollback. Sampling is bounded and
|
||||
repeatable for a policy version. If a randomized sample is empty or reaches its query timeout, the
|
||||
adapter tries one sequential bounded sample; if that also times out, the source error fails the run
|
||||
and returns no review instead of manufacturing <code>unknown</code> decisions.</p>
|
||||
<p>There is no global sixty-second analysis deadline. Work is bounded by sample counts, per-query
|
||||
timeouts, and early column exits. The operation is interrupted only when its request connection is
|
||||
aborted or the backend restarts. Historical or interrupted run counters named <code>unknown</code> represent
|
||||
columns that were not processed; <code>unknown</code> is not a <code>sensitivity-v4</code> column assessment.</p>
|
||||
<p>History stores only the policy version, aggregate outcomes, timestamps, and fixed operational
|
||||
events. Sanitized rule IDs are returned in the transient review and shadow report, not persisted.
|
||||
Neither path stores values, matched spans, prompts, or free-form model output.</p>
|
||||
<h2 id="optional-cpu-only-gliner2-evidence">Optional CPU-only GLiNER2 evidence<a class="headerlink" href="#optional-cpu-only-gliner2-evidence" title="Permanent link">¶</a></h2>
|
||||
<p>The deterministic engine works without Python NER. An installation may opt into
|
||||
<code>fastino/gliner2-privacy-filter-PII-multi</code> for unresolved short text. It runs in a persistent local
|
||||
Python worker, adds sanitized evidence, and never becomes a second decision point. The worker:</p>
|
||||
<ul>
|
||||
<li>loads a local model directory only and forces Hugging Face/Transformers offline mode;</li>
|
||||
<li>starts warming in the background when the backend starts; an analysis never waits for warm-up
|
||||
and skips NER until the worker is ready, so loading cannot consume the run's NER allowance;</li>
|
||||
<li>hides CUDA and HIP devices and loads weights with <code>map_location="cpu"</code>;</li>
|
||||
<li>starts with a scrubbed environment, then installs a fail-closed seccomp filter that denies
|
||||
network syscalls before accepting source text (the Python socket API is disabled as defense in depth);</li>
|
||||
<li>receives at most two 500-character candidates per table by default, selected breadth-first
|
||||
across unresolved columns, and shares a ten-second NER allowance across the whole run;</li>
|
||||
<li>returns only column ID, normalized label, and confidence; source text and entity spans are not
|
||||
returned or stored;</li>
|
||||
<li>is skipped on timeout, startup failure, invalid output, or absent configuration. No LLM fallback
|
||||
is selected.</li>
|
||||
</ul>
|
||||
<p>The pinned model revision is <code>c153999da5f4c509df4322b0c6a1baf3d2c284d7</code>. GLiNER2 and the model
|
||||
are Apache-2.0; the published mDeBERTa base is MIT. The optional runtime pins
|
||||
<code>gliner2[local]==2.0.0</code>, <code>transformers==4.57.6</code>, and the CPU-only PyTorch wheel
|
||||
<code>torch==2.14.0+cpu</code>. It lives in <code>/opt/sensitivity-ner</code>, is not installed in the default core image,
|
||||
and does not install CUDA packages.</p>
|
||||
<p>The current upstream checkpoint was saved by Transformers 5.8 even though GLiNER2 2.0.0 officially
|
||||
requires Transformers <code><5</code>; the resulting tokenizer error is independently reported in
|
||||
<a href="https://github.com/fastino-ai/GLiNER2/issues/145">GLiNER2 issue 145</a>. At startup ThothII leaves the
|
||||
pinned model directory unchanged and creates a temporary symlink view that maps the checkpoint's
|
||||
<code>extra_special_tokens</code> list to the Transformers 4 name <code>additional_special_tokens</code>. Any other or
|
||||
ambiguous shape fails closed and leaves the optional NER unavailable. The offline CPU smoke test
|
||||
must remain part of every dependency or model revision update.</p>
|
||||
<h2 id="prepare-and-enable-the-optional-profile">Prepare and enable the optional profile<a class="headerlink" href="#prepare-and-enable-the-optional-profile" title="Permanent link">¶</a></h2>
|
||||
<p>Download happens during explicit installation, never during inference:</p>
|
||||
<pre class="highlight"><code class="language-bash">./scripts/fetch-sensitivity-ner-model.sh /absolute/path/to/gliner2-pii</code></pre>
|
||||
<p>The script builds the separate <code>thothii-core:sensitivity-ner</code> image, downloads the exact revision, and writes
|
||||
<code>MODEL_SHA256SUMS</code>. Keep the model directory outside the repository. Then set:</p>
|
||||
<pre class="highlight"><code class="language-bash">export THOTH_ENABLE_SENSITIVITY_NER=1
|
||||
export THT_SENSITIVITY_NER_MODEL_DIR=/absolute/path/to/gliner2-pii
|
||||
export THT_SENSITIVITY_NER_THREADS=2
|
||||
./scripts/run-stack.sh</code></pre>
|
||||
<p>For an operator-managed Compose invocation, include <code>deploy/compose.sensitivity-ner.yaml</code> after the
|
||||
base and installation overlays. The core build argument <code>INSTALL_SENSITIVITY_NER=true</code> installs the
|
||||
optional Python dependencies into their isolated virtualenv. The model mount is read-only. Values
|
||||
above eight threads are rejected; start with two so classification cannot contend heavily with
|
||||
other CPU workloads.</p>
|
||||
<h2 id="acceptance-on-a-real-database">Acceptance on a real database<a class="headerlink" href="#acceptance-on-a-real-database" title="Permanent link">¶</a></h2>
|
||||
<p>Run the first evaluation in shadow mode: read the source with its existing read-only role, do not
|
||||
save proposed flags, and report only aggregate counts, rule IDs, coverage, and timings. Never copy
|
||||
matched values into test output. Use a separately approved, labeled Italian corpus to calculate
|
||||
precision and recall; source values must remain inside the authorized environment.</p>
|
||||
<p>Inside the configured core runtime, the non-mutating command is:</p>
|
||||
<pre class="highlight"><code class="language-bash">npm run sensitivity:shadow -- <workspace-id></code></pre>
|
||||
<p>It reads catalog metadata and source values but emits one aggregate JSON object with no database,
|
||||
table, column, or source-value detail. It neither creates an analysis run nor updates a flag.</p>
|
||||
<p>Enabling NER by default requires all of these gates:</p>
|
||||
<ol>
|
||||
<li>the pinned artifact and <code>MODEL_SHA256SUMS</code> are archived with the installation inventory;</li>
|
||||
<li>the Python dependency/license inventory contains only redistribution-compatible licenses;</li>
|
||||
<li>the CPU benchmark stays within the configured NER allowance and does not use a GPU;</li>
|
||||
<li>the labeled Italian evaluation meets thresholds approved by the product owner.</li>
|
||||
</ol>
|
||||
<p>If a gate fails, leave NER disabled. The deterministic policy remains available and produces the
|
||||
binary draft from its scan coverage; no content is sent to an internal or external LLM.</p>
|
||||
<p>Benchmarks from a particular installation are not a guarantee for another database or machine.
|
||||
NER remains opt-in until an approved evaluation establishes that additional findings justify
|
||||
their false-positive rate and operational cost. Keep benchmark and release records with the
|
||||
installation's technical evidence, separate from this operator procedure.</p>
|
||||
|
||||
<br>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
|
||||
|
||||
<div class="wm-article-nav pull-right">
|
||||
<a href="../../evidence/" class="btn btn-xs btn-default pull-right">
|
||||
Next
|
||||
<i class="fa fa-chevron-right" aria-hidden="true"></i>
|
||||
</a>
|
||||
<a href="../../evidence/" class="btn btn-xs btn-link">
|
||||
Evidence
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<div class="wm-article-nav">
|
||||
<a href="../database-management/" class="btn btn-xs btn-default pull-left">
|
||||
<i class="fa fa-chevron-left" aria-hidden="true"></i>
|
||||
Previous</a><a href="../database-management/" class="btn btn-xs btn-link">
|
||||
Databases and descriptions
|
||||
</a>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<br>
|
||||
</div>
|
||||
|
||||
<footer class="container-fluid wm-page-content">
|
||||
<p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a> using <a href="https://github.com/gristlabs/mkdocs-windmill">Windmill</a> theme by Grist Labs.</p>
|
||||
</footer>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,224 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
|
||||
|
||||
<meta charset="utf-8">
|
||||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
|
||||
|
||||
<link rel="canonical" href="https://git.tylconsulting.it/thothii-docs/operations/workspaces/">
|
||||
<link rel="shortcut icon" href="../../img/favicon.ico">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0" />
|
||||
<title>Workspaces - ThothII Docs</title>
|
||||
<link href="../../css/bootstrap-3.3.7.min.css" rel="stylesheet">
|
||||
<link href="../../css/font-awesome-4.7.0.css" rel="stylesheet">
|
||||
<link href="../../css/base.css" rel="stylesheet">
|
||||
<link rel="stylesheet" href="../../css/highlight.css">
|
||||
<link href="../../stylesheets/extra.css" rel="stylesheet">
|
||||
<!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
|
||||
<!--[if lt IE 9]>
|
||||
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
|
||||
<script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
|
||||
<![endif]-->
|
||||
|
||||
<script src="../../js/jquery-3.2.1.min.js"></script>
|
||||
<script src="../../js/bootstrap-3.3.7.min.js"></script>
|
||||
<script src="../../js/highlight.pack.js"></script>
|
||||
|
||||
<base target="_top">
|
||||
<script>
|
||||
var base_url = '../..';
|
||||
var is_top_frame = false;
|
||||
|
||||
var pageToc = [
|
||||
{title: "Workspace operations", url: "#_top", children: [
|
||||
{title: "Roles and ownership", url: "#roles-and-ownership" },
|
||||
{title: "Operator sequence", url: "#operator-sequence" },
|
||||
{title: "Transport and revision rules", url: "#transport-and-revision-rules" },
|
||||
]},
|
||||
];
|
||||
|
||||
</script>
|
||||
<script src="../../js/base.js"></script>
|
||||
<script src="../../javascripts/layout-init.js"></script>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<script>
|
||||
if (is_top_frame) { $('body').addClass('wm-top-page'); }
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
<div class="container-fluid wm-page-content">
|
||||
<a name="_top"></a>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
|
||||
|
||||
<div class="wm-article-nav pull-right">
|
||||
<a href="../database-management/" class="btn btn-xs btn-default pull-right">
|
||||
Next
|
||||
<i class="fa fa-chevron-right" aria-hidden="true"></i>
|
||||
</a>
|
||||
<a href="../database-management/" class="btn btn-xs btn-link">
|
||||
Databases and descriptions
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<div class="wm-article-nav">
|
||||
<a href="../../general/pi-configuration/" class="btn btn-xs btn-default pull-left">
|
||||
<i class="fa fa-chevron-left" aria-hidden="true"></i>
|
||||
Previous</a><a href="../../general/pi-configuration/" class="btn btn-xs btn-link">
|
||||
Model configuration
|
||||
</a>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
|
||||
|
||||
<h1 id="workspace-operations">Workspace operations<a class="headerlink" href="#workspace-operations" title="Permanent link">¶</a></h1>
|
||||
<p>A workspace is curator-owned Git content plus installation-local runtime bindings. It is the
|
||||
boundary between what can be published and what can be used by an installation.</p>
|
||||
<h2 id="roles-and-ownership">Roles and ownership<a class="headerlink" href="#roles-and-ownership" title="Permanent link">¶</a></h2>
|
||||
<table>
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Role</th>
|
||||
<th>Owns</th>
|
||||
<th>Does not own</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>Curator</td>
|
||||
<td><code>thoth-workspaces.yaml</code>, <code><id>/workspace.yaml</code>, and Evidence</td>
|
||||
<td>database metadata, installation secrets, or active runtime bindings</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Installation operator</td>
|
||||
<td>Git source, selected workspace, write-only runtime secrets, validation, connectivity, and preprocessing</td>
|
||||
<td>commits or pushes to the workspace repository</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Reviewer</td>
|
||||
<td>NL→SQL decisions in a pinned session</td>
|
||||
<td>workspace publication or preprocessing</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<!-- non-workspace-migration:start -->
|
||||
<p>The root catalog has <code>schema_version: 1</code> and an ordered list of workspace identities.</p>
|
||||
<!-- non-workspace-migration:end -->
|
||||
|
||||
<!-- workspace-descriptor-contract:start -->
|
||||
<p>Schema v4 is the only accepted workspace descriptor. Schema v1, v2, and v3 workspace descriptors
|
||||
are rejected before activation. Each catalog entry must have a matching descriptor at
|
||||
<code><id>/workspace.yaml</code> in the same Git commit. The application validates a complete candidate
|
||||
revision and activates it atomically; invalid content leaves the preceding active revision in
|
||||
place. A v4 descriptor contains only workspace identity and optional Evidence configuration; it
|
||||
does not contain a database or database metadata.</p>
|
||||
<!-- workspace-descriptor-contract:end -->
|
||||
|
||||
<!-- non-workspace-migration:start -->
|
||||
<p>Create a clean v4 descriptor with <code>workspace</code> and optional <code>evidence</code>. Do not import the old DWH,
|
||||
diagnostics, annotation, <code>llm_policy</code>, or <code>semantic_index</code> blocks. Configure the database in
|
||||
Database Management. ThothII never rewrites the curator-owned repository during pull.</p>
|
||||
<!-- non-workspace-migration:end -->
|
||||
|
||||
<h2 id="operator-sequence">Operator sequence<a class="headerlink" href="#operator-sequence" title="Permanent link">¶</a></h2>
|
||||
<ol>
|
||||
<li>Curate and push a complete repository revision. Do not put DWH passwords, API keys, private
|
||||
keys, or signed URLs in this repository.</li>
|
||||
<li>In the application, update the workspace repository. This fetches and validates the candidate;
|
||||
it never edits the remote repository.</li>
|
||||
<li>Select the workspace. Supply or replace its write-only Evidence runtime secrets, configure its
|
||||
database in <strong>Database Management</strong>, then run <strong>Validate workspace source</strong> and <strong>Test workspace
|
||||
connections</strong>. The workspace connection test uses that same current database configuration for
|
||||
DWH connectivity and also checks the workspace Evidence and installation semantic services.</li>
|
||||
<li>Select it as the installation workspace before creating sessions.</li>
|
||||
<li>Expand <strong>Administration</strong> in the right sidebar and run <strong>Preprocessing</strong>. The button is available
|
||||
when all prerequisites are satisfied: it shows <strong>Run</strong> when preprocessing is required,
|
||||
<strong>Run again</strong> when the workspace is already current, and <strong>Retry</strong> after a failure. The same
|
||||
operation is available from the host CLI for unattended administration. <strong>Clear</strong>, immediately
|
||||
to the left, removes only replaceable Schema/Evidence vectors, LSH, corpus, and checkpoints after
|
||||
an inline confirmation; it preserves Memory and solved questions. <code>--json</code> keeps CLI stdout
|
||||
machine-readable.</li>
|
||||
</ol>
|
||||
<pre class="highlight"><code class="language-sh">INSTALLATION=/absolute/path/thothii-installation.yaml
|
||||
WORKSPACE=example-workspace
|
||||
|
||||
tht --installation "$INSTALLATION" workspace inspect --workspace "$WORKSPACE" --json
|
||||
tht --installation "$INSTALLATION" workspace preprocess run --workspace "$WORKSPACE" --json
|
||||
tht --installation "$INSTALLATION" workspace preprocess clear --workspace "$WORKSPACE" --json</code></pre>
|
||||
<p>The command snapshots tables, columns, descriptions, sensitivity, and active relationships from
|
||||
PostgreSQL, samples eligible DWH values for LSH, and replaces the schema/Evidence vector slices.
|
||||
It is rerunnable but not resumable and has no rollback. Catalog sync and description generation
|
||||
remain separate operations and must already be complete.</p>
|
||||
<p>After clear, the sidebar reports <strong>Required</strong> and the core rejects new sessions until a complete run
|
||||
succeeds. Clear can be repeated safely: an already absent reference collection or derived path is a
|
||||
no-op, and the separate Memory collection is never a cleanup target.</p>
|
||||
<p>The sidebar retains no run history. If the current prerequisite blocks a start, it explains what
|
||||
must be completed and correctly reports that there is no run log. If the last run failed, it shows
|
||||
only that run's safe stage, error code, and finish time; use <code>docker compose logs core</code> for the
|
||||
corresponding service log.</p>
|
||||
<p>The contract gives exact validation, exit code, and JSON rules in
|
||||
<a href="https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-preprocessing-cli.md">Workspace preprocessing CLI</a>. For Evidence source
|
||||
forms and the schema-v4 descriptor contract, see
|
||||
<a href="https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-evidence-v3.md">Workspace Evidence v3</a>.</p>
|
||||
<h2 id="transport-and-revision-rules">Transport and revision rules<a class="headerlink" href="#transport-and-revision-rules" title="Permanent link">¶</a></h2>
|
||||
<p>Runtime sessions support direct PostgreSQL and REST bindings. SSH tunnel bindings are diagnostic
|
||||
only for this path, so they cannot admit an NL→SQL session. Database Management and <strong>Test
|
||||
workspace connections</strong> share the current database binding, including its strict known-host SSH
|
||||
path; the remaining workspace diagnostics cover Evidence and installation semantic services.</p>
|
||||
<p>Every new session pins the active Git revision. Snapshot cleanup retains revisions still
|
||||
referenced by unarchived sessions. A later pull can prepare a future session but cannot alter a
|
||||
resume.</p>
|
||||
|
||||
<br>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
|
||||
|
||||
<div class="wm-article-nav pull-right">
|
||||
<a href="../database-management/" class="btn btn-xs btn-default pull-right">
|
||||
Next
|
||||
<i class="fa fa-chevron-right" aria-hidden="true"></i>
|
||||
</a>
|
||||
<a href="../database-management/" class="btn btn-xs btn-link">
|
||||
Databases and descriptions
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<div class="wm-article-nav">
|
||||
<a href="../../general/pi-configuration/" class="btn btn-xs btn-default pull-left">
|
||||
<i class="fa fa-chevron-left" aria-hidden="true"></i>
|
||||
Previous</a><a href="../../general/pi-configuration/" class="btn btn-xs btn-link">
|
||||
Model configuration
|
||||
</a>
|
||||
</div>
|
||||
|
||||
</div>
|
||||
|
||||
<br>
|
||||
</div>
|
||||
|
||||
<footer class="container-fluid wm-page-content">
|
||||
<p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a> using <a href="https://github.com/gristlabs/mkdocs-windmill">Windmill</a> theme by Grist Labs.</p>
|
||||
</footer>
|
||||
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user