Predictive accuracy study: comparing a statistical model to clinicians’ estimates of outcomes after coronary bypass surgery

Background — The purpose of this study was to compare clinicians’ prior probability estimates of operative mortality (OM) and prolonged intensive care unit stay (ICU) length of stay greater than 48 hours after coronary artery bypass graft surgery (CABG) with estimates derived from statistical models alone.

Methods — Nine clinicians estimated the predicted probability of OM and ICU stay greater than 48 hours from an abstract of information for each of 100 patients selected from the 1996 to 1997 database of 1,904 patients who underwent isolated CABG. Logistic regression models were used to calculate the predicted probability of OM and ICU stay greater than 48 hours for each patient. The study sample was split into two parts; clinicians were randomly given access to a predictive rule to guide their judgements for one part of the study.

Results — Clinicians’ estimates were similar with or without access to the rule, and both parts of the study were therefore pooled. Clinicians significantly overestimated the probability of OM (model 6.3% ± 1%, clinicians 7.6% ± 3%, p = 0.0001) and ICU stay greater than 48 hours (model 25% ± 2%, clinicians 28% ± 1%, p = 0.0012). Clinicians’ estimates of OM were not significantly higher than the model’s for nonsurvivors (0.8% ± 0.7%, p = 0.2), but were significantly higher for survivors (1.4% ± 0.3%, p = 0.039).

Conclusions — Clinicians trusted their own empiric estimates rather than a predictive rule and overestimated the probability of OM and ICU stay greater than 48 hours.