70 lines
4.7 KiB
Markdown
70 lines
4.7 KiB
Markdown
# Slide 08: validation sources
|
|
|
|
Verified on 2026-09-20 against `aritmolab/chirone-etl` on the hospital Gitea,
|
|
main snapshot `2ab00106188f298c1a0c1e2c48b3a4dbc66c45b5`. These are source-code
|
|
and documentation checks, not a new audit of patient data or a production test run.
|
|
|
|
## Quality gates
|
|
|
|
[Extraction specification, SC-003 and SC-004](https://git-aritmolab.policlinicosandonato.it/aritmolab/chirone-etl/src/commit/2ab00106188f298c1a0c1e2c48b3a4dbc66c45b5/include/textanalysis/specs/001-cr-conclusioni-extraction/spec.md)
|
|
defines procedure classification accuracy of at least 95% against a random sample
|
|
of 100 manually reviewed records. The separate 85% target is coverage: at least one
|
|
recognized arrhythmia in that proportion of records, assuming most procedures
|
|
concern known arrhythmias. It is not diagnostic accuracy, sensitivity or precision.
|
|
The repository search also confirms these definitions in the TextAnalysis README.
|
|
The inspected sources do not establish measured results or a completed 100-record
|
|
review for every release. The slide therefore calls these figures targets.
|
|
|
|
## Clinical criteria and cohort selection
|
|
|
|
[Confirmed-diagnosis mart documentation](https://git-aritmolab.policlinicosandonato.it/aritmolab/chirone-etl/src/commit/2ab00106188f298c1a0c1e2c48b3a4dbc66c45b5/docs/marts/procedure-event-marts/mart_diagnosi_lettera_confermata.md)
|
|
identifies the Superset “Diagnosi Lettere” charts (570/571), in the “Test Provocativi
|
|
& Diagnosi Lettere” dashboard, as consumers of this dataset.
|
|
|
|
[The SQL implementation](https://git-aritmolab.policlinicosandonato.it/aritmolab/chirone-etl/src/commit/2ab00106188f298c1a0c1e2c48b3a4dbc66c45b5/dbt/models/marts/mart_diagnosi_lettera_confermata.sql)
|
|
first requires patient attribution and a non-negated finding. For Brugada and
|
|
long QT syndrome it additionally requires a positive provocative test for that
|
|
condition OR an ablation recorded for that pathology, at patient level. It does
|
|
not impose that extra condition on other pathologies. These are operational
|
|
cohort-selection rules, not a universal clinical diagnostic standard.
|
|
|
|
No formal sign-off or individual clinical validation of the entire Brugada cohort
|
|
was established by the consulted sources. The point is titled “Clinical criteria”:
|
|
the queries apply the selection rules and Superset displays their result. The
|
|
speaker script explains that a disease mention alone is insufficient for cohort
|
|
inclusion; the popup retains the Brugada/long QT rule as a project-specific example.
|
|
|
|
## Feedback loop: the approximately 893 records
|
|
|
|
[Known limits and version changelog](https://git-aritmolab.policlinicosandonato.it/aritmolab/chirone-etl/src/commit/2ab00106188f298c1a0c1e2c48b3a4dbc66c45b5/docs/architecture/textanalysis-known-limits.md)
|
|
records an audit on 2026-06-11 and the v1.3.1 correction on 2026-06-12. It reports
|
|
approximately 893 Brugada false positives in the newer letter format (8.4% of
|
|
non-negated Brugada records). This is a record count, not a distinct-patient count.
|
|
|
|
[Regression tests, class TestNegationExtended](https://git-aritmolab.policlinicosandonato.it/aritmolab/chirone-etl/src/commit/2ab00106188f298c1a0c1e2c48b3a4dbc66c45b5/include/textanalysis/tests/unit/test_letter_cascade_analyzer.py)
|
|
explain the defect: the negation patterns recognized “negato” and “negata”, but
|
|
omitted “negativo”. The fix adds `negativ[oai]` to both pre- and post-context
|
|
negation detection. The test sentence “Test alla flecainide negativo per sindrome
|
|
di Brugada” must still yield a Brugada finding, now with `negato=True`; a separate
|
|
test preserves the affirmed classification of a positive Brugada diagnosis.
|
|
Thus “reclassified” is more accurate than “removed”. These tests were inspected,
|
|
not executed in this presentation workspace.
|
|
|
|
## Retained context and limits
|
|
|
|
The mart documentation describes propagation of `attribuzione` and `negato`
|
|
through integration, warehouse bridge and the mentions mart. The confirmed view
|
|
filters them without deleting the underlying mentions dataset.
|
|
|
|
The known-limits document describes a context window of up to 60 characters on
|
|
either side of a match, bounded by a full stop, semicolon or newline. Uncertainty
|
|
phrases such as “sospetto di” and “per escludere” are not automatically definitive
|
|
negations. The same-pathology deduplication tests show that an affirmed patient
|
|
finding takes priority over a negated or family finding. Therefore the slide does
|
|
not claim verbatim preservation of every repeated match or complete extraction
|
|
of all clinical information.
|
|
|
|
The mart page's historical “first matching field wins” caveat predates the v1.3.0
|
|
multi-field union. For current field handling, the later known-limits document
|
|
and unit tests take precedence.
|