Early Warning Scores and Machine Learning: Building Deterioration Models That Work
Early Warning Scores and Machine Learning: Building Deterioration Models That Work
Early Warning Scores and Machine Learning: Building Deterioration Models That Work
The National Early Warning Score 2 (NEWS2) and the Modified Early Warning Score (MEWS) are simple. They aggregate vital signs, heart rate, respiratory rate, blood pressure, temperature, oxygen saturation, and level of consciousness, into a single number. When that number crosses a threshold, someone is supposed to respond. The concept is sound. The implementation, in most hospitals I have worked in, is uneven.
ML-augmented deterioration prediction was supposed to fix this. In many cases, it has made things worse.
The Epic Sepsis Model problem
The most widely deployed ML deterioration model in North America is Epic's Sepsis Model, embedded in the Epic EHR and active at hundreds of hospitals. In 2021, Wong et al. published an external validation study at Michigan Medicine that found the model had an area under the ROC curve of 0.63, far below the 0.76 to 0.83 that Epic reported internally. The positive predictive value was 12%. That means roughly 7 out of 8 alerts were false positives.
Sendak et al., in a 2020 study at Duke University Health System, found similar results. Their analysis showed that the model's performance degraded significantly in patient subgroups that differed from the training population. The model had been trained on retrospective data from a specific set of hospitals and generalized poorly.
This is a pattern. Prospective external validation of clinical ML models almost always shows lower performance than retrospective internal validation. The reasons are well-understood: distribution shift, differences in documentation practices, and temporal drift as treatment protocols change. These are solvable engineering problems, but the solutions require ongoing investment in model monitoring and retraining that most health systems cannot resource.
Alert fatigue is the real enemy
Even a model with excellent discrimination (AUC above 0.85) will cause problems if it fires too often. A floor nurse on a medical unit might receive 15 to 20 electronic alerts per shift. When the deterioration model adds another 5, most of them false positives, the rational response is to ignore them. This is alert fatigue.
The Sendak group proposed a tiered alerting framework: low-risk alerts go to a dashboard that charge nurses review hourly, medium-risk alerts trigger a bedside nursing assessment, and high-risk alerts page the rapid response team directly. This is a workflow design, not a model improvement. I think it is more important than any algorithmic advance in the deterioration prediction space.
What actually works
The hospitals where I have seen early warning systems produce real improvements share a common feature. They pair the scoring system, whether traditional NEWS2 or ML-augmented, with a structured escalation protocol that is owned by nursing leadership. The score triggers a specific action. That action is documented. There is a feedback loop: when a rapid response is called, the team reviews whether the early warning system flagged the patient, and if it did not, they investigate why.
At Southlake Regional Health Centre in Newmarket, Ontario, a nurse-led implementation of a modified NEWS protocol reduced cardiac arrest rates on medical wards by 30% over two years. The scoring system they used was simple. The escalation protocol was specific. The nursing leadership had buy-in from hospital administration to enforce compliance. No ML was involved.
I am not arguing against machine learning in deterioration prediction. I am arguing that the algorithm is the least important component. The workflow, the escalation protocol, and the human willingness to act on an alert determine whether patients survive. A mediocre model inside a well-designed clinical workflow will outperform a sophisticated model that generates alerts nobody responds to.