Clinical Fidelity Belt — Evaluation gate for AI-humanized radiology reports
A four-layer evaluation gate that decides whether an AI-humanized radiology report still preserves clinical severity well enough for a patient to read — plus a documented negative result: a transformer clinical NER backend that collapsed from 56% to 8.6% entity recall.
Clinical Fidelity Belt started at the IABiomed 2026 hackathon as an evaluation gate for AI-humanized radiology reports: a four-layer check (readability, clinical coverage, entity recall, factual fidelity) that decides whether a patient-friendly rewrite still preserves the original's clinical severity, formalizing the line between legitimate simplification and dangerous attenuation. It compares three independent fidelity backends — lexical entailment, NLI-based contradiction detection (mDeBERTa-v3), and an LLM-as-judge (Gemini) — against an adversarial test set with deliberately seeded severity attenuations. The most interesting finding is negative: the transformer-based clinical NER backend dropped entity recall from 56% to 8.6% against a hand-curated technical-to-lay dictionary, a failure diagnosed in detail to separate fixable pattern bugs from limitations inherent to lexical matching. It ships as a Python package with 30 offline tests, CI across Python 3.10/3.12/3.14, a pinned environment, a data card, and documented limitations — scoped honestly as a proof of concept validated on a single synthetic clinical case.
The problem
Radiology reports written for clinicians are unreadable to most patients, and LLMs can rewrite them in plain language — but simplification and information loss look identical from the outside. A rewritten report can quietly downplay a finding's severity without the drop in technical density that would normally signal a problem, and there was no established, empirically validated way to tell an appropriate rewrite from a dangerous one before it reaches a patient.
The solution
I built a four-layer gate — readability, clinical coverage, entity recall, and factual fidelity — that a humanized report must pass before reaching a patient, and benchmarked three candidate fidelity backends against each other on an adversarial set with deliberately seeded attenuations: lexical entailment, NLI-based contradiction detection via mDeBERTa-v3, and an LLM-as-judge using Gemini. When the clinical NER backend underperformed sharply (56% → 8.6% entity recall against a hand-curated dictionary), instead of hiding the result I diagnosed it, separating bugs I could fix from limitations inherent to lexical matching against clinical language. The whole thing is packaged as a tested Python library — 30 offline tests, CI on three Python versions, a pinned environment, and a data card — scoped honestly as a single-case proof of concept.