Publish documentation for 2f53512e4d

This commit is contained in:
Gitea Actions
2026-09-26 22:42:08 +00:00
commit cb50f5a6a4
55 changed files with 33829 additions and 0 deletions
+316
View File
@@ -0,0 +1,316 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="canonical" href="https://git.tylconsulting.it/thothii-docs/operations/database-management/">
<link rel="shortcut icon" href="../../img/favicon.ico">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0" />
<title>Databases and descriptions - ThothII Docs</title>
<link href="../../css/bootstrap-3.3.7.min.css" rel="stylesheet">
<link href="../../css/font-awesome-4.7.0.css" rel="stylesheet">
<link href="../../css/base.css" rel="stylesheet">
<link rel="stylesheet" href="../../css/highlight.css">
<link href="../../stylesheets/extra.css" rel="stylesheet">
<!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
<![endif]-->
<script src="../../js/jquery-3.2.1.min.js"></script>
<script src="../../js/bootstrap-3.3.7.min.js"></script>
<script src="../../js/highlight.pack.js"></script>
<base target="_top">
<script>
var base_url = '../..';
var is_top_frame = false;
var pageToc = [
{title: "Database management", url: "#_top", children: [
{title: "What the catalog owns", url: "#what-the-catalog-owns" },
{title: "Navigate the Fleet Ledger surface", url: "#navigate-the-fleet-ledger-surface" },
{title: "Configure and test a database", url: "#configure-and-test-a-database" },
{title: "Synchronize authoritative schema metadata", url: "#synchronize-authoritative-schema-metadata" },
{title: "Generate and consolidate descriptions", url: "#generate-and-consolidate-descriptions" },
]},
];
</script>
<script src="../../js/base.js"></script>
<script src="../../javascripts/layout-init.js"></script>
</head>
<body>
<script>
if (is_top_frame) { $('body').addClass('wm-top-page'); }
</script>
<div class="container-fluid wm-page-content">
<a name="_top"></a>
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
<div class="wm-article-nav pull-right">
<a href="../sensitivity-analysis/" class="btn btn-xs btn-default pull-right">
Next
<i class="fa fa-chevron-right" aria-hidden="true"></i>
</a>
<a href="../sensitivity-analysis/" class="btn btn-xs btn-link">
Sensitive columns
</a>
</div>
<div class="wm-article-nav">
<a href="../workspaces/" class="btn btn-xs btn-default pull-left">
<i class="fa fa-chevron-left" aria-hidden="true"></i>
Previous</a><a href="../workspaces/" class="btn btn-xs btn-link">
Workspaces
</a>
</div>
</div>
<h1 id="database-management">Database management<a class="headerlink" href="#database-management" title="Permanent link">&para;</a></h1>
<p>Database Management is the administrative PostgreSQL Metadata Catalog for an external PostgreSQL
schema. Its binding and metadata are the sole database source used by workspace preprocessing and
the NL→SQL session workflow.</p>
<h2 id="what-the-catalog-owns">What the catalog owns<a class="headerlink" href="#what-the-catalog-owns" title="Permanent link">&para;</a></h2>
<p>For each workspace identity, an administrator may configure at most one Metadata Catalog binding. It holds
the database name, schema, connection binding, write-only encrypted secrets, observed physical
schema, optional curated descriptions, generated descriptions, and durable operation history.</p>
<p>It does <strong>not</strong> become the external source of truth. Tables, columns, types, defaults,
nullability, primary-key positions, and ordered foreign-key pairs are observations of the source
schema and cannot be manually created, renamed, or structurally edited. Descriptions are the
editable metadata.</p>
<h2 id="navigate-the-fleet-ledger-surface">Navigate the Fleet Ledger surface<a class="headerlink" href="#navigate-the-fleet-ledger-surface" title="Permanent link">&para;</a></h2>
<p>Database Management opens Fleet Ledger inside the normal application shell. Only one data grid is
shown at a time: choose a database to see its tables, choose a table to see its columns, or open the
database's relationships view. Use the emphasized back control or breadcrumb to return to the parent
grid.</p>
<p>The KPI strip reports tables, columns, sensitive columns, relationships, and description coverage.
It uses <code>GET /catalog/metrics</code> without <code>databaseId</code> for installation totals and with <code>databaseId</code> for
the current database. Choose a selection-scoped operation from the action selector and then press
<strong>Run</strong>; unavailable operations remain listed with an explanation. Row-specific actions are the icon
controls in the final column, and each navigation or action icon has an immediate conceptual tooltip.
An unconfigured workspace exposes <strong>Configure catalog</strong> directly on its row; there is no global
database-creation action and the selected workspace cannot be changed in the configuration form.</p>
<p>The master grid keeps three independent states visible:</p>
<ul>
<li><strong>Revision / Evidence</strong> comes from the active immutable workspace revision. Filesystem Evidence is
materialized with that revision; remote Evidence is reported as configured-but-unverified or as
requiring credentials.</li>
<li><strong>NL→SQL runtime</strong> is calculated from the workspace DWH/Evidence requirements and runtime secret
store. It also reports transports, such as SSH, that are diagnostic-only and unsupported by sessions.</li>
<li><strong>Metadata Catalog</strong> reports whether the installation-local catalog configuration exists, then shows
its separately versioned connection-test or synchronization state.</li>
</ul>
<p>Configuration, object details, metadata editors, synchronization history, description history,
sensitive-field review, and suggestion-run history open in right-side drawers backed by the
production catalog APIs.
Closing a history drawer does not cancel a durable background run. Existing permission checks,
dirty/busy navigation guards, stale-state handling, and write-only secret behavior continue to
apply.</p>
<p>For temporary comparison in development or staging, add <code>?db-ui=legacy</code>; the parameter is honored
only by Vite development or an environment explicitly configured with
<code>VITE_DB_MANAGEMENT_LEGACY=true</code>. The separate prototype on port <code>5173</code> is not the application and
remains available only until the integrated Fleet Ledger surface passes owner acceptance.</p>
<h2 id="configure-and-test-a-database">Configure and test a database<a class="headerlink" href="#configure-and-test-a-database" title="Permanent link">&para;</a></h2>
<ol>
<li>Open <strong>Database Management</strong> and find the repository workspace marked <strong>Not configured</strong>.</li>
<li>Choose <strong>Configure catalog</strong> on that row. Configure its PostgreSQL catalog binding with
<code>postgres_direct</code>, <code>rest_api</code>, or <code>ssh_tunnel</code> and
complete the binding fields that the chosen transport requires.</li>
<li>Enter secrets only when replacing them. They remain write-only and are never returned by the
application.</li>
<li>Use <strong>Test connection</strong> whenever you want an informational connectivity check. Its result does
not enable or disable catalog operations.</li>
</ol>
<p>SSH uses a private key, optional key passphrase, mandatory <code>known_hosts</code>, and optional PostgreSQL
TLS CA/server name. REST prefers <code>POST /rpc/schema_snapshot</code>; when it is absent, the catalog may
use the same strict v1 snapshot through one read-only <code>POST /rpc/run_query</code>. An unavailable
capability, malformed snapshot, or connector error applies no catalog changes. See the
<a href="https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/catalog-schema-snapshot.md">schema snapshot contract</a>.</p>
<h2 id="synchronize-authoritative-schema-metadata">Synchronize authoritative schema metadata<a class="headerlink" href="#synchronize-authoritative-schema-metadata" title="Permanent link">&para;</a></h2>
<p>Schema synchronization reads the external database and reconciles the installation-local catalog.
It never changes the source database. Every synchronization attempts a fresh connection when it
runs; an unreachable server or rejected credential fails that run without changing catalog data.
The available synchronization scopes are <strong>tables</strong>,
<strong>columns</strong>, <strong>relationships</strong>, and <strong>all</strong>, but the UI exposes them at different levels:</p>
<table>
<thead>
<tr>
<th>Location</th>
<th>Action</th>
<th>Effective scope</th>
</tr>
</thead>
<tbody>
<tr>
<td>Database view</td>
<td><strong>Synchronize tables</strong></td>
<td>All tables in the selected database</td>
</tr>
<tr>
<td>Database view</td>
<td><strong>Synchronize relationships</strong></td>
<td>All physical foreign-key relationships in the selected database</td>
</tr>
<tr>
<td>Database view</td>
<td><strong>Synchronize all</strong></td>
<td>Tables, columns, and physical relationships in the selected database</td>
</tr>
<tr>
<td>Tables view</td>
<td><strong>Synchronize database tables</strong></td>
<td>All tables in the selected database. Selecting a table enables the action, but does not narrow its scope.</td>
</tr>
<tr>
<td>Tables view</td>
<td><strong>Synchronize columns for selected tables</strong></td>
<td>Columns belonging to the selected tables</td>
</tr>
<tr>
<td>Columns view</td>
<td><strong>Synchronize columns for this table</strong></td>
<td>All columns belonging to the table currently open. Selecting at least one column enables the action, but does not narrow its scope to that column.</td>
</tr>
</tbody>
</table>
<p>There is no database-level column action and no synchronization action for an individual column.
The selection requirement in the tables and columns views controls whether the action selector can
be used; it is not always the same as the synchronization target. The <strong>Sync all</strong> button in the
tables view is the direct shortcut for the full-database scope.</p>
<p>Connection tests are informational and are never a synchronization prerequisite. Each
synchronization tests its own access while reading the schema; an unavailable connection fails
that operation with a connector error. Only one catalog operation can be active for a database at
a time; explicit cleanup shares this exclusion.</p>
<h3 id="what-a-synchronization-does">What a synchronization does<a class="headerlink" href="#what-a-synchronization-does" title="Permanent link">&para;</a></h3>
<p>Every run first creates a durable queued operation and reads a schema snapshot. The scan reports
these phases:</p>
<ol>
<li>Connect to the database.</li>
<li>Read tables.</li>
<li>Read columns and primary-key positions.</li>
<li>Read foreign-key relationships.</li>
<li>Calculate the planned catalog difference.</li>
<li>Apply only the requested scope.</li>
</ol>
<p>The scan currently reads the complete physical snapshot, including foreign keys, even when the
requested scope is only tables or columns. This is required by the introspection contract and is
why the log can mention foreign-key reading during a table synchronization. Reading those keys
does not by itself create or update catalog relationships:</p>
<ul>
<li><strong>Synchronize tables</strong> writes table membership and source comments. If a table disappears, its
catalog columns and physical relationships are removed through the table cascade.</li>
<li><strong>Synchronize columns</strong> writes column membership and structural attributes for all tables or for
the selected table subset. It does not write physical relationships.</li>
<li><strong>Synchronize relationships</strong> writes the physical relationships derived from the source foreign
keys. It does not create generated or manual logical relationships.</li>
<li><strong>Synchronize all</strong> applies all three scopes and marks the database schema version as fully
synchronized.</li>
</ul>
<p>The run scans first and publishes a durable operation. If it detects a destructive difference, it
requires confirmation and re-scans before applying. You can cancel before apply; completed and
failed runs remain in history. The live log is delivered over SSE with a polling fallback.</p>
<p>The synchronization history records the requested scope, progress phases, planned changes,
confirmation, result counts, and errors. Closing the history drawer does not cancel a running
operation; it can be reopened from the database synchronization history control.</p>
<p>Explicit cleanup is different from source synchronization: administrators can clear selected
table/relationship or column/relationship catalog metadata without changing the external source,
the connection binding, or secrets. Deleting a table cascades to its columns and relationships.</p>
<h2 id="generate-and-consolidate-descriptions">Generate and consolidate descriptions<a class="headerlink" href="#generate-and-consolidate-descriptions" title="Permanent link">&para;</a></h2>
<p>Generated descriptions can be requested for selected tables, selected columns, every eligible
target, or targets with a missing generated description. The backend accepts one installation-wide
run and processes targets sequentially. Every catalog column has a <strong>Sensitive</strong> flag, which defaults
to <code>false</code>, including after a newly discovered column is synchronized. Before generation, an
administrator can run local sensitivity analysis over the selected database, tables, or columns.
The analysis combines structural metadata with bounded read-only inspection of source values. It
uses no generative AI and no installation-catalog model. Assessments remain an unsaved draft until
a human reviews and saves them; the reviewer may reverse any proposal.</p>
<p>The page exposes separate histories for description generation and sensitivity analysis. Analysis
history stores the local policy version, scope, status, aggregate <code>sensitive</code>, <code>non_sensitive</code>, and
<code>unknown</code> counts, timestamps, and sanitized events. It does not store source values, per-column
proposals, NER spans, or worker diagnostics; closing an unsaved review discards that draft. The
rules, time bounds, and optional CPU-only NER profile are documented in
<a href="../sensitivity-analysis/">Local sensitivity analysis</a>.</p>
<p>For a column with <code>sensitive=false</code>, the worker may read at most five source rows and five
representative non-null values through a read-only connector. For <code>sensitive=true</code>, the source query
does not request that column's values; deterministic plausible values derived only from its name and
type take their place in the model prompt. The prompt does not identify those values as synthetic, so
the model can still describe the field as if it had received representative data.</p>
<p>Each successful result is persisted immediately. Stop terminates the active helper but retains
earlier results. A helper has at most one provider retry; three consecutively exhausted technical
batches fail the run. Stale queued/running work is marked interrupted at startup and can be
unlocked only when no local worker/helper is live. There is no automatic resume and no public
description-generation CLI.</p>
<p>Review generated text before copying it into the curated <strong>Description</strong> field. Because the flag
defaults to <code>false</code>, an administrator must review the classification and mark protected fields before
starting generation. Changing a flag affects future generations only; existing generated or curated
descriptions are not regenerated. Real and substituted samples remain transient and are not persisted
or returned to the browser.</p>
<p>The database-level <strong>Copy generated description to descriptions</strong> action applies every non-empty
AI-generated table and column description to the corresponding curated <strong>Description</strong> field in one
atomic operation. It skips empty generated descriptions, reports aggregate copied and skipped
counts, and retains the generated text. Because this can replace reviewed descriptions, the
interface requires explicit confirmation before applying it.</p>
<p>The decisions behind this surface are <a href="https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/adr/0001-postgres-metadata-catalog.md">ADRs 0001–0011</a>
and the detailed acceptance record is
<a href="https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/testing/2026-08-29-ai-catalog-description-generation-acceptance.md">AI catalog description generation acceptance</a>.</p>
<br>
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
<div class="wm-article-nav pull-right">
<a href="../sensitivity-analysis/" class="btn btn-xs btn-default pull-right">
Next
<i class="fa fa-chevron-right" aria-hidden="true"></i>
</a>
<a href="../sensitivity-analysis/" class="btn btn-xs btn-link">
Sensitive columns
</a>
</div>
<div class="wm-article-nav">
<a href="../workspaces/" class="btn btn-xs btn-default pull-left">
<i class="fa fa-chevron-left" aria-hidden="true"></i>
Previous</a><a href="../workspaces/" class="btn btn-xs btn-link">
Workspaces
</a>
</div>
</div>
<br>
</div>
<footer class="container-fluid wm-page-content">
<p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a> using <a href="https://github.com/gristlabs/mkdocs-windmill">Windmill</a> theme by Grist Labs.</p>
</footer>
</body>
</html>
+245
View File
@@ -0,0 +1,245 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="canonical" href="https://git.tylconsulting.it/thothii-docs/operations/sensitivity-analysis/">
<link rel="shortcut icon" href="../../img/favicon.ico">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0" />
<title>Sensitive columns - ThothII Docs</title>
<link href="../../css/bootstrap-3.3.7.min.css" rel="stylesheet">
<link href="../../css/font-awesome-4.7.0.css" rel="stylesheet">
<link href="../../css/base.css" rel="stylesheet">
<link rel="stylesheet" href="../../css/highlight.css">
<link href="../../stylesheets/extra.css" rel="stylesheet">
<!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
<![endif]-->
<script src="../../js/jquery-3.2.1.min.js"></script>
<script src="../../js/bootstrap-3.3.7.min.js"></script>
<script src="../../js/highlight.pack.js"></script>
<base target="_top">
<script>
var base_url = '../..';
var is_top_frame = false;
var pageToc = [
{title: "Local sensitivity analysis", url: "#_top", children: [
{title: "Default policy", url: "#default-policy" },
{title: "Optional CPU-only GLiNER2 evidence", url: "#optional-cpu-only-gliner2-evidence" },
{title: "Prepare and enable the optional profile", url: "#prepare-and-enable-the-optional-profile" },
{title: "Acceptance on a real database", url: "#acceptance-on-a-real-database" },
]},
];
</script>
<script src="../../js/base.js"></script>
<script src="../../javascripts/layout-init.js"></script>
</head>
<body>
<script>
if (is_top_frame) { $('body').addClass('wm-top-page'); }
</script>
<div class="container-fluid wm-page-content">
<a name="_top"></a>
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
<div class="wm-article-nav pull-right">
<a href="../../evidence/" class="btn btn-xs btn-default pull-right">
Next
<i class="fa fa-chevron-right" aria-hidden="true"></i>
</a>
<a href="../../evidence/" class="btn btn-xs btn-link">
Evidence
</a>
</div>
<div class="wm-article-nav">
<a href="../database-management/" class="btn btn-xs btn-default pull-left">
<i class="fa fa-chevron-left" aria-hidden="true"></i>
Previous</a><a href="../database-management/" class="btn btn-xs btn-link">
Databases and descriptions
</a>
</div>
</div>
<h1 id="local-sensitivity-analysis">Local sensitivity analysis<a class="headerlink" href="#local-sensitivity-analysis" title="Permanent link">&para;</a></h1>
<p>Database Management can assess selected columns without sending their metadata or contents to a
generative model. The feature is advisory: it creates a transient review draft, while the catalog's
Sensitive Data Flag changes only when an administrator explicitly saves a choice. The administrator
may set either value, including overriding a <code>sensitive</code> proposal.</p>
<h2 id="default-policy">Default policy<a class="headerlink" href="#default-policy" title="Permanent link">&para;</a></h2>
<p><code>SensitivityClassifier</code> is the only column-level decision point. The versioned <code>sensitivity-v4</code>
policy combines:</p>
<ul>
<li>a structural exclusion for declared <code>bigint</code> primary-key columns and undeclared <code>bigint</code>
columns following the exact <code>pk</code> naming convention; their values are non-informative identifiers
and are therefore not inspected as possible sensitive content. The evidence distinguishes
declared constraints from convention-based inference;</li>
<li>normalized column-name rules for direct identifiers, credentials, and health data;</li>
<li>validated content rules for email, Italian fiscal code and VAT, passport, identity-card and
driving-licence identifiers, phone numbers, IBAN/BIC, payment-card checksums, IP/MAC addresses,
URLs, UUIDs, access keys, private-key markers, sensitive keys inside bounded recursive JSON, and
a reviewed Italian clinical-term dictionary;</li>
<li>a conservative length rule: any observed textual value longer than 500 characters makes the
entire column sensitive.</li>
</ul>
<p>One decisive value is enough to classify the column as <code>sensitive</code> and removes it from subsequent
passes. Binary or otherwise uninspectable column types are also proposed as <code>sensitive</code>, because
their contents cannot be cleared by the textual rules. A completed analysis has only two draft
outcomes: <code>sensitive</code> and <code>non_sensitive</code>. Empty or all-null columns are <code>non_sensitive</code> with
<code>no_values</code> coverage; a sampled column with no match is <code>non_sensitive</code> with explicit sampled
coverage. The administrator remains free to reverse either proposal before saving it.</p>
<p>Source reads are database-specific, but decisions are database-independent. PostgreSQL direct and
REST <code>run_query</code> adapters project at most 501 characters per value, use only <code>SELECT</code>, and never
persist source values. Tables proven to contain at most 1,000 rows are fully scanned. Larger tables
are processed breadth-first so every table gets the cheapest pass before any table gets a deeper
one:</p>
<ol>
<li>inspect up to 300 non-null values per unresolved column;</li>
<li>inspect up to 700 additional values, reaching a 1,000-value target;</li>
<li>for unresolved text, JSON, and XML columns only, inspect up to 2,000 additional values, reaching
a 3,000-value target.</li>
</ol>
<p>At most two tables are scanned concurrently, and the database adapter groups at most 25 columns in
one source query. Each probe or value query has a five-second statement timeout; PostgreSQL-wire
reads run in a read-only transaction and always end with rollback. Sampling is bounded and
repeatable for a policy version. If a randomized sample is empty or reaches its query timeout, the
adapter tries one sequential bounded sample; if that also times out, the source error fails the run
and returns no review instead of manufacturing <code>unknown</code> decisions.</p>
<p>There is no global sixty-second analysis deadline. Work is bounded by sample counts, per-query
timeouts, and early column exits. The operation is interrupted only when its request connection is
aborted or the backend restarts. Historical or interrupted run counters named <code>unknown</code> represent
columns that were not processed; <code>unknown</code> is not a <code>sensitivity-v4</code> column assessment.</p>
<p>History stores only the policy version, aggregate outcomes, timestamps, and fixed operational
events. Sanitized rule IDs are returned in the transient review and shadow report, not persisted.
Neither path stores values, matched spans, prompts, or free-form model output.</p>
<h2 id="optional-cpu-only-gliner2-evidence">Optional CPU-only GLiNER2 evidence<a class="headerlink" href="#optional-cpu-only-gliner2-evidence" title="Permanent link">&para;</a></h2>
<p>The deterministic engine works without Python NER. An installation may opt into
<code>fastino/gliner2-privacy-filter-PII-multi</code> for unresolved short text. It runs in a persistent local
Python worker, adds sanitized evidence, and never becomes a second decision point. The worker:</p>
<ul>
<li>loads a local model directory only and forces Hugging Face/Transformers offline mode;</li>
<li>starts warming in the background when the backend starts; an analysis never waits for warm-up
and skips NER until the worker is ready, so loading cannot consume the run's NER allowance;</li>
<li>hides CUDA and HIP devices and loads weights with <code>map_location="cpu"</code>;</li>
<li>starts with a scrubbed environment, then installs a fail-closed seccomp filter that denies
network syscalls before accepting source text (the Python socket API is disabled as defense in depth);</li>
<li>receives at most two 500-character candidates per table by default, selected breadth-first
across unresolved columns, and shares a ten-second NER allowance across the whole run;</li>
<li>returns only column ID, normalized label, and confidence; source text and entity spans are not
returned or stored;</li>
<li>is skipped on timeout, startup failure, invalid output, or absent configuration. No LLM fallback
is selected.</li>
</ul>
<p>The pinned model revision is <code>c153999da5f4c509df4322b0c6a1baf3d2c284d7</code>. GLiNER2 and the model
are Apache-2.0; the published mDeBERTa base is MIT. The optional runtime pins
<code>gliner2[local]==2.0.0</code>, <code>transformers==4.57.6</code>, and the CPU-only PyTorch wheel
<code>torch==2.14.0+cpu</code>. It lives in <code>/opt/sensitivity-ner</code>, is not installed in the default core image,
and does not install CUDA packages.</p>
<p>The current upstream checkpoint was saved by Transformers 5.8 even though GLiNER2 2.0.0 officially
requires Transformers <code>&lt;5</code>; the resulting tokenizer error is independently reported in
<a href="https://github.com/fastino-ai/GLiNER2/issues/145">GLiNER2 issue 145</a>. At startup ThothII leaves the
pinned model directory unchanged and creates a temporary symlink view that maps the checkpoint's
<code>extra_special_tokens</code> list to the Transformers 4 name <code>additional_special_tokens</code>. Any other or
ambiguous shape fails closed and leaves the optional NER unavailable. The offline CPU smoke test
must remain part of every dependency or model revision update.</p>
<h2 id="prepare-and-enable-the-optional-profile">Prepare and enable the optional profile<a class="headerlink" href="#prepare-and-enable-the-optional-profile" title="Permanent link">&para;</a></h2>
<p>Download happens during explicit installation, never during inference:</p>
<pre class="highlight"><code class="language-bash">./scripts/fetch-sensitivity-ner-model.sh /absolute/path/to/gliner2-pii</code></pre>
<p>The script builds the separate <code>thothii-core:sensitivity-ner</code> image, downloads the exact revision, and writes
<code>MODEL_SHA256SUMS</code>. Keep the model directory outside the repository. Then set:</p>
<pre class="highlight"><code class="language-bash">export THOTH_ENABLE_SENSITIVITY_NER=1
export THT_SENSITIVITY_NER_MODEL_DIR=/absolute/path/to/gliner2-pii
export THT_SENSITIVITY_NER_THREADS=2
./scripts/run-stack.sh</code></pre>
<p>For an operator-managed Compose invocation, include <code>deploy/compose.sensitivity-ner.yaml</code> after the
base and installation overlays. The core build argument <code>INSTALL_SENSITIVITY_NER=true</code> installs the
optional Python dependencies into their isolated virtualenv. The model mount is read-only. Values
above eight threads are rejected; start with two so classification cannot contend heavily with
other CPU workloads.</p>
<h2 id="acceptance-on-a-real-database">Acceptance on a real database<a class="headerlink" href="#acceptance-on-a-real-database" title="Permanent link">&para;</a></h2>
<p>Run the first evaluation in shadow mode: read the source with its existing read-only role, do not
save proposed flags, and report only aggregate counts, rule IDs, coverage, and timings. Never copy
matched values into test output. Use a separately approved, labeled Italian corpus to calculate
precision and recall; source values must remain inside the authorized environment.</p>
<p>Inside the configured core runtime, the non-mutating command is:</p>
<pre class="highlight"><code class="language-bash">npm run sensitivity:shadow -- &lt;workspace-id&gt;</code></pre>
<p>It reads catalog metadata and source values but emits one aggregate JSON object with no database,
table, column, or source-value detail. It neither creates an analysis run nor updates a flag.</p>
<p>Enabling NER by default requires all of these gates:</p>
<ol>
<li>the pinned artifact and <code>MODEL_SHA256SUMS</code> are archived with the installation inventory;</li>
<li>the Python dependency/license inventory contains only redistribution-compatible licenses;</li>
<li>the CPU benchmark stays within the configured NER allowance and does not use a GPU;</li>
<li>the labeled Italian evaluation meets thresholds approved by the product owner.</li>
</ol>
<p>If a gate fails, leave NER disabled. The deterministic policy remains available and produces the
binary draft from its scan coverage; no content is sent to an internal or external LLM.</p>
<p>Benchmarks from a particular installation are not a guarantee for another database or machine.
NER remains opt-in until an approved evaluation establishes that additional findings justify
their false-positive rate and operational cost. Keep benchmark and release records with the
installation's technical evidence, separate from this operator procedure.</p>
<br>
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
<div class="wm-article-nav pull-right">
<a href="../../evidence/" class="btn btn-xs btn-default pull-right">
Next
<i class="fa fa-chevron-right" aria-hidden="true"></i>
</a>
<a href="../../evidence/" class="btn btn-xs btn-link">
Evidence
</a>
</div>
<div class="wm-article-nav">
<a href="../database-management/" class="btn btn-xs btn-default pull-left">
<i class="fa fa-chevron-left" aria-hidden="true"></i>
Previous</a><a href="../database-management/" class="btn btn-xs btn-link">
Databases and descriptions
</a>
</div>
</div>
<br>
</div>
<footer class="container-fluid wm-page-content">
<p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a> using <a href="https://github.com/gristlabs/mkdocs-windmill">Windmill</a> theme by Grist Labs.</p>
</footer>
</body>
</html>
+224
View File
@@ -0,0 +1,224 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<link rel="canonical" href="https://git.tylconsulting.it/thothii-docs/operations/workspaces/">
<link rel="shortcut icon" href="../../img/favicon.ico">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0" />
<title>Workspaces - ThothII Docs</title>
<link href="../../css/bootstrap-3.3.7.min.css" rel="stylesheet">
<link href="../../css/font-awesome-4.7.0.css" rel="stylesheet">
<link href="../../css/base.css" rel="stylesheet">
<link rel="stylesheet" href="../../css/highlight.css">
<link href="../../stylesheets/extra.css" rel="stylesheet">
<!-- HTML5 shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond.js/1.3.0/respond.min.js"></script>
<![endif]-->
<script src="../../js/jquery-3.2.1.min.js"></script>
<script src="../../js/bootstrap-3.3.7.min.js"></script>
<script src="../../js/highlight.pack.js"></script>
<base target="_top">
<script>
var base_url = '../..';
var is_top_frame = false;
var pageToc = [
{title: "Workspace operations", url: "#_top", children: [
{title: "Roles and ownership", url: "#roles-and-ownership" },
{title: "Operator sequence", url: "#operator-sequence" },
{title: "Transport and revision rules", url: "#transport-and-revision-rules" },
]},
];
</script>
<script src="../../js/base.js"></script>
<script src="../../javascripts/layout-init.js"></script>
</head>
<body>
<script>
if (is_top_frame) { $('body').addClass('wm-top-page'); }
</script>
<div class="container-fluid wm-page-content">
<a name="_top"></a>
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
<div class="wm-article-nav pull-right">
<a href="../database-management/" class="btn btn-xs btn-default pull-right">
Next
<i class="fa fa-chevron-right" aria-hidden="true"></i>
</a>
<a href="../database-management/" class="btn btn-xs btn-link">
Databases and descriptions
</a>
</div>
<div class="wm-article-nav">
<a href="../../general/pi-configuration/" class="btn btn-xs btn-default pull-left">
<i class="fa fa-chevron-left" aria-hidden="true"></i>
Previous</a><a href="../../general/pi-configuration/" class="btn btn-xs btn-link">
Model configuration
</a>
</div>
</div>
<h1 id="workspace-operations">Workspace operations<a class="headerlink" href="#workspace-operations" title="Permanent link">&para;</a></h1>
<p>A workspace is curator-owned Git content plus installation-local runtime bindings. It is the
boundary between what can be published and what can be used by an installation.</p>
<h2 id="roles-and-ownership">Roles and ownership<a class="headerlink" href="#roles-and-ownership" title="Permanent link">&para;</a></h2>
<table>
<thead>
<tr>
<th>Role</th>
<th>Owns</th>
<th>Does not own</th>
</tr>
</thead>
<tbody>
<tr>
<td>Curator</td>
<td><code>thoth-workspaces.yaml</code>, <code>&lt;id&gt;/workspace.yaml</code>, and Evidence</td>
<td>database metadata, installation secrets, or active runtime bindings</td>
</tr>
<tr>
<td>Installation operator</td>
<td>Git source, selected workspace, write-only runtime secrets, validation, connectivity, and preprocessing</td>
<td>commits or pushes to the workspace repository</td>
</tr>
<tr>
<td>Reviewer</td>
<td>NL→SQL decisions in a pinned session</td>
<td>workspace publication or preprocessing</td>
</tr>
</tbody>
</table>
<!-- non-workspace-migration:start -->
<p>The root catalog has <code>schema_version: 1</code> and an ordered list of workspace identities.</p>
<!-- non-workspace-migration:end -->
<!-- workspace-descriptor-contract:start -->
<p>Schema v4 is the only accepted workspace descriptor. Schema v1, v2, and v3 workspace descriptors
are rejected before activation. Each catalog entry must have a matching descriptor at
<code>&lt;id&gt;/workspace.yaml</code> in the same Git commit. The application validates a complete candidate
revision and activates it atomically; invalid content leaves the preceding active revision in
place. A v4 descriptor contains only workspace identity and optional Evidence configuration; it
does not contain a database or database metadata.</p>
<!-- workspace-descriptor-contract:end -->
<!-- non-workspace-migration:start -->
<p>Create a clean v4 descriptor with <code>workspace</code> and optional <code>evidence</code>. Do not import the old DWH,
diagnostics, annotation, <code>llm_policy</code>, or <code>semantic_index</code> blocks. Configure the database in
Database Management. ThothII never rewrites the curator-owned repository during pull.</p>
<!-- non-workspace-migration:end -->
<h2 id="operator-sequence">Operator sequence<a class="headerlink" href="#operator-sequence" title="Permanent link">&para;</a></h2>
<ol>
<li>Curate and push a complete repository revision. Do not put DWH passwords, API keys, private
keys, or signed URLs in this repository.</li>
<li>In the application, update the workspace repository. This fetches and validates the candidate;
it never edits the remote repository.</li>
<li>Select the workspace. Supply or replace its write-only Evidence runtime secrets, configure its
database in <strong>Database Management</strong>, then run <strong>Validate workspace source</strong> and <strong>Test workspace
connections</strong>. The workspace connection test uses that same current database configuration for
DWH connectivity and also checks the workspace Evidence and installation semantic services.</li>
<li>Select it as the installation workspace before creating sessions.</li>
<li>Expand <strong>Administration</strong> in the right sidebar and run <strong>Preprocessing</strong>. The button is available
when all prerequisites are satisfied: it shows <strong>Run</strong> when preprocessing is required,
<strong>Run again</strong> when the workspace is already current, and <strong>Retry</strong> after a failure. The same
operation is available from the host CLI for unattended administration. <strong>Clear</strong>, immediately
to the left, removes only replaceable Schema/Evidence vectors, LSH, corpus, and checkpoints after
an inline confirmation; it preserves Memory and solved questions. <code>--json</code> keeps CLI stdout
machine-readable.</li>
</ol>
<pre class="highlight"><code class="language-sh">INSTALLATION=/absolute/path/thothii-installation.yaml
WORKSPACE=example-workspace
tht --installation "$INSTALLATION" workspace inspect --workspace "$WORKSPACE" --json
tht --installation "$INSTALLATION" workspace preprocess run --workspace "$WORKSPACE" --json
tht --installation "$INSTALLATION" workspace preprocess clear --workspace "$WORKSPACE" --json</code></pre>
<p>The command snapshots tables, columns, descriptions, sensitivity, and active relationships from
PostgreSQL, samples eligible DWH values for LSH, and replaces the schema/Evidence vector slices.
It is rerunnable but not resumable and has no rollback. Catalog sync and description generation
remain separate operations and must already be complete.</p>
<p>After clear, the sidebar reports <strong>Required</strong> and the core rejects new sessions until a complete run
succeeds. Clear can be repeated safely: an already absent reference collection or derived path is a
no-op, and the separate Memory collection is never a cleanup target.</p>
<p>The sidebar retains no run history. If the current prerequisite blocks a start, it explains what
must be completed and correctly reports that there is no run log. If the last run failed, it shows
only that run's safe stage, error code, and finish time; use <code>docker compose logs core</code> for the
corresponding service log.</p>
<p>The contract gives exact validation, exit code, and JSON rules in
<a href="https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-preprocessing-cli.md">Workspace preprocessing CLI</a>. For Evidence source
forms and the schema-v4 descriptor contract, see
<a href="https://git.tylconsulting.it/mptyl/ThothII/src/branch/main/docs/contracts/workspace-evidence-v3.md">Workspace Evidence v3</a>.</p>
<h2 id="transport-and-revision-rules">Transport and revision rules<a class="headerlink" href="#transport-and-revision-rules" title="Permanent link">&para;</a></h2>
<p>Runtime sessions support direct PostgreSQL and REST bindings. SSH tunnel bindings are diagnostic
only for this path, so they cannot admit an NL→SQL session. Database Management and <strong>Test
workspace connections</strong> share the current database binding, including its strict known-host SSH
path; the remaining workspace diagnostics cover Evidence and installation semantic services.</p>
<p>Every new session pins the active Git revision. Snapshot cleanup retains revisions still
referenced by unarchived sessions. A later pull can prepare a future session but cannot alter a
resume.</p>
<br>
<div class="row wm-article-nav-buttons" role="navigation" aria-label="navigation">
<div class="wm-article-nav pull-right">
<a href="../database-management/" class="btn btn-xs btn-default pull-right">
Next
<i class="fa fa-chevron-right" aria-hidden="true"></i>
</a>
<a href="../database-management/" class="btn btn-xs btn-link">
Databases and descriptions
</a>
</div>
<div class="wm-article-nav">
<a href="../../general/pi-configuration/" class="btn btn-xs btn-default pull-left">
<i class="fa fa-chevron-left" aria-hidden="true"></i>
Previous</a><a href="../../general/pi-configuration/" class="btn btn-xs btn-link">
Model configuration
</a>
</div>
</div>
<br>
</div>
<footer class="container-fluid wm-page-content">
<p>Documentation built with <a href="https://www.mkdocs.org/">MkDocs</a> using <a href="https://github.com/gristlabs/mkdocs-windmill">Windmill</a> theme by Grist Labs.</p>
</footer>
</body>
</html>