feat: classify sensitive columns locally

This commit is contained in:
Codex
2026-09-03 02:11:13 +02:00
parent 7b87e95427
commit f114d0065a
57 changed files with 4038 additions and 1149 deletions
@@ -21,13 +21,21 @@ completed scan with no finding may produce `non_sensitive`; an incomplete scan w
produces `unknown`.
Ambiguous text may additionally be sent to an optional local NER detector only while time remains.
The detector runs on CPU, receives no tools or network access, does not persist source values, and
returns evidence rather than the column decision. The initial supported detector is
By default it receives at most two candidates per table and shares a ten-second allowance across
the entire analysis run. The detector runs on CPU, receives no tools or network access (enforced
inside the worker with a fail-closed seccomp network-syscall filter), does not
persist source values, and returns evidence rather than the column decision. The initial supported detector is
[`fastino/gliner2-privacy-filter-PII-multi`](https://huggingface.co/fastino/gliner2-privacy-filter-PII-multi),
used through the Apache-2.0 GLiNER2 Python library with a pinned model revision. Its model weights
and GLiNER2 code are Apache-2.0, and its mDeBERTa base model is MIT. It is trained for seven
languages including Italian and can run on CPU without using the installation's GPUs.
The selected checkpoint currently carries Transformers 5 tokenizer metadata while the released
GLiNER2 2.0.0 runtime requires Transformers 4. ThothII may bridge only that known key rename in a
temporary view of the immutable, checksummed artifact; unexpected or ambiguous metadata fails
closed. Removing the compatibility bridge requires an offline smoke test against a corrected,
pinned upstream release.
The NER detector is an optional installation asset because its weights and runtime are materially
larger than the deterministic TypeScript engine. It is invoked only for otherwise unresolved text,
never for values already classified by a decisive rule. If it is disabled, unavailable, times out,