feat: classify sensitive columns locally

2026-09-03 02:11:13 +02:00
parent 7b87e95427
commit f114d0065a
57 changed files with 4038 additions and 1149 deletions
@@ -0,0 +1,37 @@
+# PSD sensitivity shadow evaluation
+
+Date: 2026-09-02
+
+This report records an aggregate, non-mutating evaluation of `sensitivity-v1` against the PSD
+workspace. The source data warehouse was accessed through the configured read-only connector. The
+shadow command did not create an analysis run, update catalog metadata, or save Sensitive Data
+Flags. No database, table, column, source value, matched span, or free-form diagnostic was emitted.
+
+The runner was the local Docker `arm64` CPU environment connected to the PSD source; this was not a
+benchmark of the PSD production server. Both runs used the same 2,275 catalog columns and a
+60-second analysis deadline.
+
+| Profile | Sensitive | Non-sensitive | Unknown | NER findings | Analysis time |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Deterministic policy | 57 | 8 | 2,210 | 0 | 60,017 ms |
+| CPU NER, pre-warmed, two candidates/table, 10 s shared allowance | 68 | 0 | 2,207 | 11 | 60,022 ms |
+
+The deterministic run produced findings from metadata, phone-number, and Italian clinical-term
+rules. The optional NER run identified eleven additional unresolved text candidates, but its
+inference time reduced the source coverage reached before the global deadline. The number of
+definitive non-sensitive assessments consequently fell from eight to zero, so this broad shadow
+run does not justify enabling NER by default.
+
+The separate offline synthetic Italian smoke test succeeded with a `full_name` finding at high
+confidence. The image dependency check reported no broken requirements, and PyTorch reported
+`cuda=False`, no CUDA runtime, and zero GPU devices. A defense-in-depth test retained a raw socket
+constructor before Python-level blocking and confirmed that the worker's seccomp filter still
+rejected the socket syscall with `EPERM`.
+
+## Acceptance outcome
+
+- Keep the deterministic TypeScript policy enabled by default.
+- Keep GLiNER2 available only through the explicit CPU-only installation profile.
+- Do not enable NER by default for PSD on the basis of this shadow run.
+- Reconsider the PSD setting only after a benchmark on the actual target CPU and a labeled Italian
+  corpus demonstrate a useful precision/recall gain without unacceptable coverage loss.
@@ -0,0 +1,36 @@
+# Optional sensitivity NER license inventory
+
+This inventory covers the isolated `/opt/sensitivity-ner` Python environment built from
+`backend/python/sensitivity-ner-requirements.txt` on 2 September 2026. It is a technical release
+gate, not legal advice. Every dependency is version-locked; changing any version requires
+regenerating this inventory and rerunning the offline CPU smoke test.
+
+The optional runtime also dynamically links Debian's `libseccomp2` (LGPL-2.1-only) solely to
+install its kernel-enforced network syscall filter; no libseccomp source is incorporated into ThothII.
+
+No dependency or selected model uses a non-commercial, research-only, source-available, GPL, or
+AGPL license. MPL-2.0, PSF-2.0, and the permissive composite licenses below allow free-of-charge and
+commercial use, but distributors must still preserve their applicable notices and license texts.
+
+| License family | Locked packages |
+| --- | --- |
+| Apache-2.0 | `accelerate==1.14.0`, `gliner2==2.0.0`, `hf-xet==1.6.0`, `huggingface-hub==0.36.2`, `peft==0.20.0`, `requests==2.34.2`, `safetensors==0.8.0`, `tokenizers==0.22.2`, `transformers==4.57.6` |
+| MIT | `annotated-types==0.8.0`, `charset-normalizer==3.5.1`, `filelock==3.32.5`, `pydantic==2.13.5`, `pydantic-core==2.46.5`, `PyYAML==6.0.3`, `typing-inspection==0.4.4`, `urllib3==2.7.0` |
+| BSD-2/3-Clause | `fsspec==2026.7.0`, `idna==3.19`, `Jinja2==3.1.6`, `MarkupSafe==3.0.3`, `mpmath==1.3.0`, `networkx==3.6.1`, `psutil==7.2.2`, `sympy==1.14.0` |
+| MPL-2.0 or mixed MPL/MIT | `certifi==2026.7.22`, `tqdm==4.70.0` |
+| PSF-2.0 | `typing-extensions==4.16.0` |
+| Composite permissive | `numpy==2.5.2` (BSD-3-Clause, 0BSD, MIT, Zlib, CC0), `packaging==26.3` (Apache-2.0 or BSD-2-Clause), `regex==2026.9.3` (Apache-2.0 and CNRI-Python), `torch==2.14.0+cpu` (Apache-2.0, LLVM exception, BSD, BSL-1.0, MIT) |
+
+The selected `fastino/gliner2-privacy-filter-PII-multi` weights at revision
+`c153999da5f4c509df4322b0c6a1baf3d2c284d7` are marked Apache-2.0 in the
+[model card](https://huggingface.co/fastino/gliner2-privacy-filter-PII-multi). Its published
+`microsoft/mdeberta-v3-base` base model is MIT. The Fastino training corpus is described as
+synthetic but is not published, so the training process is not independently reproducible.
+
+Before distributing the optional image or model pack:
+
+1. retain the upstream license and notice files for all packaged wheels, system libraries, and weights;
+2. archive `THOTHII_MODEL_REVISION` and the verified `MODEL_SHA256SUMS` beside the model;
+3. verify that `pip check` succeeds in the isolated environment;
+4. compare the installed distribution/version set with this inventory;
+5. repeat the licensing review if an upstream artifact or dependency changes.