Validating an Automated Fracture-Detection Model Against Consultant Read: Study Protocol

Status

Ongoing — protocol registered, retrospective image collection underway. This page describes the study design; no results are available yet.

Background

Automated fracture-detection tools are increasingly available, but their real-world accuracy varies by anatomical region, fracture type and the population they were trained on. Before relying on any such tool in our own practice, we believe it needs to be validated against our own imaging and patient population rather than taken on the vendor's published accuracy figures alone.

Design summary

• Population: A retrospective, de-identified set of wrist and hand X-rays from our imaging archive, spanning confirmed fractures (scaphoid, distal radius, metacarpal and phalangeal patterns) and a matched set of negative studies.
• Comparator: Each image is independently reviewed by the automated tool and by two fellowship-trained reviewers blinded to the tool's output, with a third senior reviewer adjudicating disagreements.
• Primary outcome: Sensitivity and specificity of the automated tool against the adjudicated reference standard, reported separately by fracture location, since we expect accuracy to vary meaningfully between, for example, a displaced metacarpal fracture and a subtle scaphoid waist fracture.
• Secondary outcome: Time-to-flag compared with standard reporting turnaround, as a practical measure of triage value independent of accuracy.

Why fracture-location subgroup reporting matters

A tool's overall accuracy figure can mask meaningful weak points — subtle, easily-missed fracture types (scaphoid fractures being the classic example in this anatomical region) are exactly the cases where a decision-support tool would add the most value, and exactly the cases most likely to be under-represented in an aggregate accuracy number.

What we'll do with the results

If validation supports it, the tool would be introduced as a second-reader aid in our own reporting workflow — never as a stand-alone diagnostic step — with performance re-audited periodically. If it doesn't meet an acceptable accuracy threshold for a specific fracture type, that limitation will be reported openly rather than adopting the tool uncritically.

Collaboration

We're open to sharing protocol details or comparing methodology with other centres running similar validation work. Reach out via the Contact page.