Comparison of machine learning and conventional statistical modeling for predicting readmission following acute heart failure hospitalization

Introduction — Developing accurate models for predicting the risk of 30-day readmission is a major healthcare interest. Evidence suggests that models developed using machine learning (ML) may have better discrimination than conventional statistical models (CSM), but the calibration of such models is unclear.

Objectives — To compare models developed using ML with those developed using CSM to predict 30-day readmission for cardiovascular and non-cardiovascular causes in HF patients.

Methods — We retrospectively enrolled 10,919 patients with HF (> 18 years) discharged alive from a hospital or emergency department (2004-2007) in Ontario, Canada. The study sample was randomly divided into training and validation sets in a 2:1 ratio. CSMs to predict 30-day readmission were developed using Fine-Gray subdistribution hazards regression (treating death as a competing risk), and the ML algorithm employed random survival forests for competing risks (RSF-CR). Models were evaluated in the validation set using both discrimination and calibration metrics.

Results — In the validation sample of 3602 patients, RSF-CR (c-statistic=0.620) showed similar discrimination to the Fine-Gray competing risk model (c-statistic=0.621) for 30-day cardiovascular readmission. In contrast, for 30-day non-cardiovascular readmission, the FineGray model (c-tatistic=0.641) slightly outperformed the RSF-CR model (c-statistic=0.632). For both outcomes, The Fine-Gray model displayed better calibration than RSF-CR using calibration plots of observed vs. predicted risks across the deciles of predicted risk.

Conclusions — Fine-Gray models had similar discrimination but superior calibration to the RSFCR model, highlighting the importance of reporting calibration metrics for ML-based prediction models. The discrimination was modest in all readmission prediction models regardless of the methods used.