Extensive benchmarking of a method that estimates external model performance from limited statistical characteristics

Tal El-Hay, Jenna M. Reps, Chen Yanover

Research output: Contribution to journalArticlepeer-review

Abstract

Predictive model performance may deteriorate when applied to data sources that were not used for training, thus, external validation is a key step in successful model deployment. As access to patient-level external data sources is typically limited, we recently proposed a method that estimates external model performance using only external summary statistics. Here, we benchmark the proposed method on multiple tasks using five large heterogeneous US data sources, where each, in turn, plays the role of an internal source and the remaining—external. Results showed accurate estimations for all metrics: 95th error percentiles for the area under the receiver operating characteristics (discrimination), calibration-in-the-large (calibration), Brier and scaled Brier scores (overall accuracy) of 0.03, 0.08, 0.0002, and 0.07, respectively. These results demonstrate the feasibility of estimating the transportability of prediction models using an internal cohort and external statistics. It may become an important accelerator of model deployment.

Original languageEnglish
Article number59
Journalnpj Digital Medicine
Volume8
Issue number1
DOIs
StatePublished - 1 Dec 2025
Externally publishedYes

ASJC Scopus subject areas

  • Medicine (miscellaneous)
  • Health Informatics
  • Computer Science Applications
  • Health Information Management

Fingerprint

Dive into the research topics of 'Extensive benchmarking of a method that estimates external model performance from limited statistical characteristics'. Together they form a unique fingerprint.

Cite this