feat: classify sensitive columns locally

2026-09-03 02:11:13 +02:00
parent 7b87e95427
commit f114d0065a
57 changed files with 4038 additions and 1149 deletions
@@ -179,6 +179,16 @@ Urchade model. This does not prove Italian clinical accuracy: the published SPY
 English and the training corpus is synthetic. Those are quality and reproducibility limitations,
 not a reason to reintroduce a generative LLM into the classifier.

+An implementation smoke test found a packaging incompatibility in the selected upstream versions:
+the checkpoint was written by Transformers 5.8, while `gliner2[local]==2.0.0` requires
+`transformers>=4.38,<5`. Every published checkpoint revision has the same tokenizer metadata, and
+[upstream issue 145](https://github.com/fastino-ai/GLiNER2/issues/145) reports the identical failure.
+ThothII therefore pins Transformers 4.57.6 and performs the minimal documented-key conversion in a
+temporary local view, without changing the downloaded model or its checksums. This compatibility
+shim was accepted only after an offline CPU smoke test detected Italian names, dates, locations,
+and usernames; any unexpected metadata shape fails closed. A future upstream fix must replace,
+not silently stack on, this shim.
+
 ## Fit with current ThothII design

 The current backend already owns source sampling and the human-owned `Sensitive Data Flag`; source