Skip to main content
Download PDF

Measuring case severity: a novel tool for benchmarking and clinical documentation improvement

BMC Health Services Research volume 22, Article number: 513 (2022) Cite this article

Abstract

Background

Severity of illness (SOI) is an All Patients Refined Diagnosis Related Groups (APR DRG) modifier based on comorbidity capture. Tracking SOI helps hospitals improve performance and resource distribution. Furthermore, benchmarking SOI plays a key role in Quality Improvement (QI) efforts such as Clinical Documentation Improvement (CDI) programs. The current SOI system highly relies on the 3 M APR DRG grouper that is updated annually, making it difficult to track severity longitudinally and benchmark against hospitals with different patient populations. Here, we describe an alternative SOI scoring system that is grouper-independent and that can be tracked longitudinally.

Methods

Admission data for 2019–2020 U.S. News and World Report Honor Roll facilities were downloaded from the Vizient Clinical Database and split into training and testing datasets. Elixhauser comorbidities, body systems developed from the Healthcare Cost and Utilization Project (HCUP), and ICD-10-CM complication and comorbidity (CC/MCC) indicators were selected as the predictors for orthogonal polynomial regression models to predict patients’ admission and discharge SOI. Receiver operating characteristic (ROC) and Precision-Recall (PR) analysis, and prediction accuracy were used to evaluate model performance.

Results

In the training dataset, the full model including both Elixhauser comorbidities and body system CC/MCC indicators had the highest ROC AUC, PR AUC and predication accuracy for both admission (ROC AUC: 92.9%; PR AUC: 91.0%; prediction accuracy: 85.4%) and discharge SOI (ROC AUC: 93.6%; PR AUC: 92.8%; prediction accuracy: 86.2%). The model including only body system CC/MCC indicators had similar performance for admission (ROC AUC: 92.4%; PR AUC: 90.4%; prediction accuracy: 84.8%) and discharge SOI (ROC AUC: 93.1%; PR AUC: 92.2%; prediction accuracy: 85.6%) as the full model. The model including only Elixhauser comorbidities exhibited the lowest performance. Similarly, in the validation dataset, the prediction accuracy was 86.2% for the full model, 85.6% for the body system model, and 79.3% for the comorbidity model. With fewer variables and less model complexity, the body system model was more efficient and was determined to be the optimal model. The probabilities generated from this model, named J_Score and J_Score_POA, successfully measured SOI and had practical applications in assessment of CDI performance.

Conclusions

The J_Scores generated from the body system model have significant value in evaluating admission and discharge severity of illness. We believe that this new scoring system will provide a useful tool for healthcare institutions to benchmark patients’ illness severity and augment Quality Improvement (QI) efforts.

Peer Review reports

Background

Healthcare institutions increasingly emphasize improved outcomes and performance and focus efforts to improve quality of care and patient safety while lowering costs [1, 2]. They cannot determine whether their efforts are satisfactory without tracking outcomes and comparing with peers, so benchmarking is widely applied within healthcare organizations to improve their clinical performance and management of operations [3,4,5]. Hospitals have also started Clinical Documentation Improvement (CDI) programs to improve documentation quality; these programs ensure better patient outcomes, optimized data quality and accurate reimbursement [6, 7].

Vizient is the largest member-driven health care performance improvement company in the U.S. and it provides services to about 95% of the nation’s academic medical centers and more than 50% of the nation’s acute care health systems [8]. Using data collected by Vizient, members can benchmark many key performance indicators such as Case Mix Index (CMI), Length of Stay (LOS), Expected Risk of Mortality (EROM), and Severity of Illness (SOI) [9,10,11,12]. SOI describes the disease severity in hospitalized patients and measures the physical effects of disease on a patient. It is a powerful tool to track resource consumption and to track patient outcomes. In addition, SOI is also closely related to cost, revenue, and profit [13]. The admission and discharge SOI are created by the 3 M coding algorithm. The admission SOI is important for hospitals to measure the health status and severity of illness of a patient when he/she is admitted. Hospitals can use admission SOI to estimate resource distribution. Discharge SOI can be used for prospective payment and risk adjustment in quality reporting. Benchmarking SOI helps hospitals better evaluate their clinical performance and distribution of resources by comparing them to peers. The SOI levels presented in Vizient data come from the All Patients Refined Diagnosis Related Groups (APR DRG) classification system developed by 3M [14, 15] Each APR DRG has four categorical severity levels: minor, moderate, major and extreme. The SOI subclasses are related to the APR DRG grouper that is updated annually by 3M. However, there are several limitations using the current SOI system. Firstly, cross-category comparison of disease severity is less meaningful. The same SOI level from different APR DRG does not mean the same level of disease burden. Secondly, it is hard to compare severity among institutions because of each institution’s unique patient mix. Lastly, it is difficult to track yearly clinical performance using longitudinal data, given that the grouper is updated annually. Therefore, we sought to develop a novel measure of SOI that is grouper independent.

To find the appropriate predictors, we compared several models targeting Elixhauser comorbidities, body systems for chronic condition indicators, and complication or comorbidity (CC) or major complication or comorbidity indicators (MCC) [16,17,18,19]. The Elixhauser comorbidities are a comprehensive set of measures to identify different pre-existing conditions based on secondary diagnoses listed in hospital administrative data. The system was developed by Anne Elixhauser using all adult, nonmaternal inpatients from acute care hospitals in California in 1992 [20]. It includes 30 comorbidity measures that are associated with considerable increases in LOS, hospital charges, and mortality. The comorbidities are usually not directly related to the primary reason for the inpatient stay, but they have a possible effect on outcomes used to assess the quality of care. The Agency for Healthcare Research and Quality (AHRQ) has created a powerful Healthcare Cost and Utilization Project (HCUP) tool called Elixhauser Comorbidity Software Refined for ICD-10-CM, which can be applied to ICD-10 diagnosis codes to create a comorbidity profile [16, 17]. AHRQ also created another tool to categorize ICD-10-CM codes into 18 body systems [19]. Body systems allow us to correct for regional differences in patient mix and comorbidity-driven DRG modifiers, so that we can compare the intensity of severity that is independent on types of diseases. These tools provide the potential indicators for predicting the severity of illness. Another valuable resource is the list of all of the ICD-10 codes that are defined as either a CC or MCC diagnoses, released by Center for Medicare & Medicaid Services (CMS) [21]. We combined CC/MCC levels with body systems and created a 3-level indicator for each body system, indicating whether the body system has any CC or MCC diagnosis code. In this study, the Elixhauser comorbidities and 18 body systems with CC/MCC indicators were used as predictors to evaluate case severity. Instead of predicting four categorized SOI levels, we aimed to develop a model to better predict high and low illness severity that is independent of APR DRG assignment. Additionally, the probabilities generated from the model would be used as a quantitative measurement of SOI.

Methods

Data Sources

This study included 923,266 inpatient cases discharged between July 1st, 2018 and June 30th, 2019 from the 2019–2020 U.S. News and World Report Best Hospitals Honor Roll [22]. All cases used the same version of APR DRG. This study included adult inpatients with age >  = 18 (including maternity). Hospital encounters classified as inpatient status were included. Observation, Emergency Department, and Outpatient encounters were not included. Patients may be admitted multiple times; each hospital admission is a separate encounter in the study. Patients’ clinical data were downloaded from Vizient Clinical Data Base (CDB). Body system categories and Elixhauser Comorbidity indicators came from HCUP [16, 19]. The list of CC/MCC diagnosis codes was downloaded from the FY2019 CMS final rule and correction notice data files [21, 23]. Information related to case review and query by clinical documentation specialists came from an internal database within our institution. All the data used in this study were deidentified before analysis in accordance with HIPPA guidelines.

Independent Variables and Outcome

In this paper, we predicted both admission and discharge SOI. All the diagnosis codes from a patient were used to predict the SOI at discharge, and only the present-on-admission (POA) and exempt diagnosis codes were used to predict the SOI on admission [24].

Body system CC/MCC level indicators

ICD-10 diagnosis codes were grouped into 18 body systems using HCUP software [19]. We combined the body system indicators and CC/MCC levels to create 18 variables. Each ordinal variable had three severity levels: 0, 1, and 2. Level 0 represented no diagnosis code in the specific body system. Level 1 represented there were one or more diagnosis codes in the body system, but none of them were CC or MCC codes. Level 2 indicated there was at least one CC or MCC diagnosis code in the body system.

Elixhauser comorbidity indicators

Elixhauser comorbidities were generated from Elixhauser comorbidity software v2019.2 (beta version) and SAS analysis program (COMOANALY_ICD10CM_2019_1.sas) that were downloaded from HCUP [17]. Each variable indicated whether the patient had a specific type of Elixhauser comorbidity based on the secondary diagnosis codes.

Outcome

APR DRG SOI minor and moderate levels (SOI = 1 or 2) were grouped into a low severity category, and major and extreme levels (SOI = 3 or 4) were considered as high severity. Our models predicted the probability of being a high SOI case, so, the outcome is a binary response variable. The probabilities calculated from the model were named J_Score_POA for admission SOI and J_Score for discharge SOI, which can be used to quantitatively measure the severity.

Variable Selection and Model Validation

Data were randomly split into training and testing datasets in a ratio of three to one. Orthogonal polynomial regression was applied to the training dataset [25,26,27,28]. Three models were constructed and compared: the full model, the comorbidity model, and the body system model. The full model included all the Elixhauser comorbidities and 18 body systems with 3 levels of CC/MCC indicators (48 variables), the comorbidity model only included Elixhauser comorbidities (30 variables), and the body system model only involved body system CC/MCC indicators (18 variables). Receiver Operating Characteristic (ROC) and Precision-Recall (PR) analysis, together with prediction accuracy were used to measure the performance of the classification models [29, 30]. Prediction accuracy was identified as the percentage of correctly classified cases using the optimal cutoff points from ROC curves. Finally, models were applied to the independent testing dataset to predict the category of case severity using optimal cut-off points obtained from the training dataset. For admission SOI, the optimal cutoff points were 0.422 for the full model, 0.413 for the comorbidity model, and 0.411 for the body system model. For discharge SOI, the optimal cutoff points were 0.401 for the full model, 0.407 for the comorbidity model, and 0.412 for the body system model.

Statistical analyses

Descriptive statistics (counts, mean, and proportions) were used to examine the similarity between training and testing datasets. The optimal cutoff point was defined as the value that minimizes the Euclidean distance between the ROC curve and the upper left corner of the graph using the training dataset. All the statistical analyses were performed in RStudio software (version 1.3.959). Statistical significance was defined as p < 0.05. No imputations were performed in the analysis.

Results

Characteristics of discharged cases

Nine hundred twenty-three thousand two hundred sixty-six discharged cases extracted from the Vizient Clinical Data Base were randomly split into training data (692,450 cases) and testing data (230,816 cases). We first compared the patients’ demographic and clinical information (Table 1). The mean age of patients from two datasets was 49 years old. Around 53% of patients were female, and the distribution of race categories was also very close (63% White, 15% Black, 5% Asian, and 11% other race). Patients in both data sets also had comparable admit and discharge APR DRG SOI levels.

Table 1 Distribution of Patients’ demographic characteristics and severity of illness for training dataset and testing dataset
Full size table

We then further looked at patients’ Elixhauser comorbidities and the MS-DRG complication and comorbidity (CC or MCC) diagnosis indicators in Clinical Classifications Software Refined (CCSR) body systems (Supplementary table 1). Patients in the two datasets had similar proportions of the 30 Elixhauser comorbidities and three CC/MCC levels in 18 body systems. Hypertension (HTN) and fluid, electrolyte disorders (LYTES), deficiency anemias (ANEMDEF) and chronic pulmonary disease (CHRNLUNG) were the top 4 comorbidities observed in patients: about 26% of patients with HTN, 24% with LYTES, 18% with ANEMDEF, and 17% with CHRNLUNG. Body systems 3 (Endocrine; nutritional; and metabolic diseases and immunity disorders), 7 (Diseases of the circulatory system) and 10 (Diseases of the genitourinary system) had the greatest number of patients with CC or MCC diagnosis codes. Overall, the training data and testing data samples were similar in both demographic and clinical characteristics.

J_Scores generated from the body system model were reliable for predicting case severity

We applied three orthogonal polynomial regression models to the training data to predict the severity levels of each case. The prediction of admission and discharge SOI used the same algorithm; they differ based on diagnoses present on admission and discharge, respectively. To evaluate the performance, we examined ROC curves and PR curves. As shown in Fig. 1, when using all the diagnosis codes to map the Elixhauser comorbidities and body system CC/MCC indicators in discharge SOI models, the full model and body system model demonstrated similar AUC scores in both ROC curves (93.6% and 93.1%) and PR curves (92.8% and 92.2%). The comorbidity model had the lowest AUC (ROC: 86.1%, PR: 85.0%). In the admission SOI models, including only the POA diagnosis codes, the full model and body system model also outperformed the comorbidity model.

Fig. 1
figure 1

ROC curves and Precision-Recall curves for regression models predicting admission and discharge SOI

Full size image

To validate the performance of the models, we checked the prediction accuracy in the testing dataset using optimal cutoff points obtained from the training dataset as described above (Table 2). For discharge SOI, the full model had a prediction accuracy of 86.2%. The body system accuracy was 85.6%. The comorbidity model had the lowest prediction accuracy (79.3%). Similar results were observed in the admission SOI models using POA diagnosis codes.

Table 2 Prediction accuracy of models for predicting admission and discharge SOI
Full size table

We also analyzed the proportion of cases with high admission severity for our institution and other 2019-2020 U.S. News and World Report Honor Roll hospitals, by comparing 3M APR DRG SOI obtained from Vizient and the predicted results from the three models (Supplementary Fig. 1). The trends from the full model and the body system model approximated the measured APR DRG SOI. Consistent with previous results, the comorbidity model displayed the largest deviation. Although the full model and the body system model showed similar performance, the body system model exhibited greater efficiency by only including 18 predictors while the full model had 48. In summary, based on the ROC analysis, precision recall analysis, prediction accuracy and counts of predictors, the body system model was determined to be the optimal model.

Besides prediction of categorical severity levels, the probability generated from the body system model also indicated the chance a case would be high severity. We referred to these probabilities as “J_Scores.” J_Scores have two versions, J_Score and J_Score_POA, depending on what diagnosis codes are used. J_Score is based on all the diagnoses coded on a case. J_Score_POA is based on only the present-on-admission (POA) and exempt diagnoses. A higher J_Score indicates more severity. To further demonstrate the J_Scores, we calculated J_Score and J_Score_POA for all the 923,266 inpatient cases, and then plotted the histograms to check their distribution (Fig. 2). As shown in Fig. 2, the histograms were bimodal with one peak at each end, indicating that the body system model performed well in separating the high and low severity cases. Notably, approximately 6.3% of the cases had a J_Score_POA of 1, and 9.1% of the cases had an overall J_Score of 1.

Fig. 2
figure 2

Distribution of J_Score_POA and J_Score

Full size image

Utilization of J_Scores in benchmarking and clinical documentation improvement

To further illustrate the value of J_Scores in benchmarking and CDI, we analyzed the trend of average J_Score and J_Score_POA for our institution and the other 2019–2020 US News and World Report Honor Roll hospitals (Fig. 3). Not surprisingly, the average J_Score was higher than the J_Score_POA, reflecting that more symptoms were identified, and more diagnosis codes were added after admission. The slightly increasing trend of J_Scores shows that the cases discharged in 2019 had higher severity than discharged cases in 2018. The gap between J_Score_POA and J_Score indicated non-POA problems that were found and treated during patients’ hospital stay.

Fig. 3
figure 3

Trend of average J_Score and J_Score_POA

Full size image

Another application of J_Scores is to assess the performance of our Clinical Documentation Improvement program. We looked at the J_Scores for inpatient cases that had been reviewed or queried by clinical documentation specialists (CDS) (Fig. 4). Notably, the average J_Score_POA of CDS reviewed cases were much higher than non-reviewed cases, indicating that CDS were selecting cases with higher severity for review. The average J_Score was notably higher than the average of J_Score_POA for CDS reviewed or queried cases while the averages of J_Score_POA and J_Score were similar for non-reviewed or non-queried cases, suggesting that CDS activities are helpful in capturing illness severity and improving the quality of documentation.

Fig. 4
figure 4

Trend of J_Score and J_Score_POA for cases reviewed or queried by clinical documentation specialists

Full size image

Discussion

In this study, we sought to develop a consistent measure of severity that is independent from the APR DRG grouper. To achieve this goal, we mapped diagnosis codes to Elixhauser comorbidities, CCI body systems, and CC/MCC indicators to create predictors and then applied orthogonal polynomial regression models to predict case severity of illness. This method can be used as an alternative way to estimate patients’ severity. We compared three models by evaluating their performance through ROC analysis, prediction accuracy, and the counts of predictors. Eventually, the body system model with CC/MCC indicators was considered the best because of higher AUC, higher accuracy rate and fewer variables. The probabilities calculated from this model were named “J_Scores,'' which serve as a measure of severity. Then, we compared the severity obtained from our model with APR DRG SOI levels acquired from the Vizient database and found that the proportions of high severity cases were similar, indicating that our model had great value in benchmarking (Supplemental Fig. 1). Plus, the large sample size of patients from 21 facilities ensured reliable results.

Although J_Scores exhibit great values in evaluating severity, the methodology differs from APR DRG SOI. The SOI subclasses developed from 3 M are determined from 3 phases with 18 steps in total after the APRDRG is assigned to a patient, incorporating the secondary diagnoses, the impact of principal diagnosis, age, OR procedure, non-OR procedures, and multiple OR procedures [31]. Each APR DRG has four subclasses of SOI. However, the severity scores generated from our model are based on affected body systems and their complication and comorbidity (CC) levels, which are not specified for APR DRG grouper. One caveat of our model is that the optimistic and promising predictive ability for severity is based on adult inpatient cases, and it needs to be validated in pediatric inpatients because pediatric cases do not have the same patterns of comorbidities as adults. Additionally, the model needs to be refreshed or re-evaluated annually to make sure it incorporates the latest version of the diagnosis codes from CMS and the updates on the body system assignment from AHRQ.

In our institution, we applied J_Scores to Vizient data to benchmark severity of illness and integrated J_Scores in CDI analysis. It was found that the average J_Score_POA of cases selected for review by CDS were much higher than the J_Score_POA of cases not selected, suggesting that our CDS were targeting cases with higher severity for review. We also envision the utility of J_Scores in audit processes, given that effective post-coding audits include a review of high-risk MS-DRGs, SOI, and risk of mortality.

Conclusions

In this study, we developed a novel method to measure SOI using diagnoses with body system and CC/MCC indicators. It is independent from APR DRG and can be used to better evaluate or compare severity in patients from different disease categories. The results demonstrated that J_Scores generated from the body system model offer reliable predictability of patients’ illness severity on admission and at discharge. Overall, this new scoring system provides a useful tool for hospitals to benchmark SOI, assess CDI programs and direct case review to improve clinical performance and quality.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from Vizient Clinical Data Base and the corresponding author on reasonable request.

Abbreviations

AHRQ:

Agency for Healthcare Research and Quality

APR DRG:

All Patients Refined Diagnosis Related Groups

CC:

Complication and Comorbidity

CDB:

Clinical Data Base

CDI:

Clinical Documentation Improvement

CMI:

Case Mix Index

CMS:

Center for Medicare & Medicaid Services

EROM:

Expected Risk of Mortality

HCUP:

Healthcare Cost and Utilization Project

LOS:

Length of Stay

MCC:

Major Complication and Comorbidity

MS-DRG:

Medicare Severity-Diagnosis Related Groups

POA:

Present-On-Admission

PR:

Precision-Recall

QI:

Quality Improvement

ROC:

Receiver Operating Characteristic

SOI:

Severity of Illness

References

  1. Evaluation of performance in health care. In: International Series in Operations Research and Management Science. New York LLC: Springer; 2014. p. 3–14.

    Google Scholar 

  2. Ozcan YA. Health care benchmarking and performance evaluation. Boston: Springer US; 2014. ISBN: 978-0-387-75448-2. https://doi.org/10.1007/978-1-4899-7472-3.

  3. Van Lent WA, De Beer RD, Van Harten WH. International benchmarking of specialty hospitals. A series of case studies on comprehensive cancer centres. BMC Health Serv Res. 2010;10:253. https://doi.org/10.1186/1472-6963-10-253.

  4. Mosel D, Gift B. Collaborative benchmarking in health care. Jt Comm J Qual Improv. 1994;20:239.

    CAS  PubMed  Google Scholar 

  5. Wind A, van Harten WH. Benchmarking specialty hospitals, a scoping review on theory and practice. BMC Health Serv Res. 2017;17:245. https://doi.org/10.1186/s12913-017-2154-y.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Rodenberg H, Shay L, Sheffield K, Dange Y. The expanding role of clinical documentation improvement programs in research and analytics. Perspect Heal Inf Manag. 2019;16(Winter):1d. eCollection Winter 2019. Accessed 28 June 2021.

  7. Hospitals & Physician Groups Benefit from a Strong CDI Program | Conifer Health Solutions. https://www.coniferhealth.com/knowledge-center/how-hospitals-and-physician-groups-benefit-from-a-strong-cdi-program/. Accessed 28 June 2021.

  8. What we do | Driving performance improvement in health care. https://www.vizientinc.com/what-we-do. Accessed 28 Oct 2020.

  9. Latimer K. The Art of Care: A Report on the 2019 Vizient Connections Education Summit. Am J Med Qual. 2020;35(1_suppl):5S-111S. https://doi.org/10.1177/1062860620904721.

    Article  Google Scholar 

  10. Mendez CM, Harrington DW, Christenson P, Spellberg B. Impact of hospital variables on case mix index as a marker of disease severity. Popul Health Manag. 2014;17:28–34. https://doi.org/10.1089/pop.2013.0002.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Zalatimo O, Ranasinghe M, Harbaugh RE, Iantosca M. Impact of improved documentation on an academic neurosurgical practice: Clinical article. J Neurosurg. 2014;120:756–63. https://doi.org/10.3171/2013.11.JNS13852.

    Article  PubMed  Google Scholar 

  12. Improving and measuring inpatient documentation of medical care within the MS-DRG system: education, monitoring, and normalized case mix index. 2014;11(Summer):1c. eCollection 2014. http://library.ahima.org/doc?oid=301192#.YNnXjflKiUk. Accessed 28 June 2021.

  13. Carpenter CE, Rosko MD, Louis DZ, Yuen EJ. Severity of illness and profitability: A patient level analysis. Heal Serv Manag Res. 1999;12:217–26. https://doi.org/10.1177/095148489901200403.

    Article  CAS  Google Scholar 

  14. All Patient Refined DRGs (APR-DRGs) An overview. https://www.3m.com/3M/en_US/health-information-systems-us/drive-value-based-care/patient-classification-methodologies/apr-drgs/. Accessed 28 June 2021.

  15. Averill Norbert Goldfield RF, Hughes JS, Bonazelli Elizabeth McCullough Barbara A Steinbeck Robert Mullin JC, Tang AM, Muldoon Lisa Turner J, Gay J. ALL PATIENT REFINED DIAGNOSIS RELATED GROUPS (APR-DRGs) methodology overview 3M health information systems. https://www.hcup-us.ahrq.gov/db/nation/nis/APR-DRGsV20MethodologyOverviewandBibliography.pdf. Accessed 20 Jan 2021.

  16. Elixhauser Comorbidity Software, Version 3.7. https://www.hcup-us.ahrq.gov/toolssoftware/comorbidity/comorbidity.jsp. Accessed 28 Oct 2020.

  17. Tools Archive for Elixhauser Comorbidity Software Refined for ICD-10-CM. https://www.hcup-us.ahrq.gov/toolssoftware/comorbidityicd10/comorbidity_icd10_archive.jsp. Accessed 18 Jan 2021.

  18. HCUP-US Home Page. https://www.hcup-us.ahrq.gov/. Accessed 18 Jan 2021.

  19. Tools Archive for the Chronic Condition Indicators for ICD-10-CM. https://www.hcup-us.ahrq.gov/toolssoftware/chronic_icd10/chronic_icd10_archive.jsp. Accessed 18 Jan 2021.

  20. Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care. 1998;36(1):8–27. https://doi.org/10.1097/00005650-199801000-00004 PMID: 9431328.

    Article  CAS  PubMed  Google Scholar 

  21. ICD-10-CM/PCS MS-DRG v37.0 Definitions Manual. https://www.cms.gov/icd10m/version37-fullcode-cms/fullcode_cms/P0031.html. Accessed 18 Jan 2021.

  22. US News’ Best Hospitals 2019–20 Honor Roll. https://www.beckershospitalreview.com/rankings-and-ratings/us-news-best-hospitals-2019-20-honor-roll.html. Accessed 20 Jan 2021.

  23. FY 2019 IPPS Proposed Rule Home Page | CMS. https://www.cms.gov/Medicare/Medicare-Fee-for-Service-Payment/AcuteInpatientPPS/FY2019-IPPS-Proposed-Rule-Home-Page. Accessed 28 Jun 2021.

  24. Hospital-acquired conditions and present on admission indicator reporting provision. https://www.cms.gov/Outreach-and-Education/Medicare-Learning-Network-MLN/MLNProducts/Downloads/wPOA-Fact-Sheet.pdf. Accessed 28 Jan 2021.

  25. Guo-LiangTian. The comparison between polynomial regression and orthogonal polynomial regression. Stat Probability Lett. 1998;38(4):289–94.

  26. Narula SC. Orthogonal polynomial regression. Int Stat Rev. 1979;47(1):31–6. https://doi.org/10.2307/1403204.

    Article  Google Scholar 

  27. Smyth GK. Polynomial approximation. Encyclopedia Biostat. 1998;13.

  28. Clark VP. Orthogonal Polynomial Regression for the Detection of Response Variability in Event-Related fMRI. Elsevier. 2002. https://doi.org/10.1006/nimg.2002.1100.

    Article  Google Scholar 

  29. Fawcett T. An introduction to ROC analysis. Pattern Recognit Lett. 2006;27:861–74.

    Article  Google Scholar 

  30. Powers, D.M.W. Evaluation: from precision, recall and f-factor to ROC, informedness, markedness & correlation. J Mach Learn Technol. 2011;2(1):37–63. ISSN: 2229-3981 & ISSN: 2229-399X.

  31. 3M All Patient Refined Diagnosis Related Groups (APR DRGs). https://www.forwardhealth.wi.gov/kw/pdf/handouts/3M_APR_DRG_Presentation.pdf. Accessed 18 Jan 2021.

Download references

Acknowledgements

The authors gratefully acknowledge the CDI team at Michigan Medicine, Beverly Merian and Ruth Lutes, for insightful comments.

Funding

The publication of this study was supported by the internal funding from the University of Michigan Health Sidney Licht fund.

Author information

Authors and Affiliations

  1. Revenue Cycle Department, Michigan Medicine, University of Michigan Health, Ann Arbor, MI, 48105, USA

    Jie Xiang

  2. Domino’s Pizza, Inc, Ann Arbor, MI, USA

    Jie Xiang

  3. Financial Services Department, Michigan Medicine, University of Michigan Health, Ann Arbor, MI, USA

    Paul W. Durance

  4. Quality Analytics, Quality Department, Michigan Medicine, University of Michigan Health, Ann Arbor, MI, USA

    Louisa C. Griffes

  5. Department of Public Health Sciences, Henry Ford Health System, Detroit, MI, USA

    Yalei Chen

  6. Departments of Physical Medicine & Rehabilitation and Revenue Cycle, Michigan Medicine, University of Michigan Health, Ann Arbor, MI, USA

    Rishi R. Bakshi

Authors
  1. Jie Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul W. Durance
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Louisa C. Griffes
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yalei Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rishi R. Bakshi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Study conception and design: JX, PWD, RRB; data acquisition: JX, PWD; data analysis: JX, YC; interpretation of results: JX, PWD, LCG, YC, RRB; Each author contributed significant intellectual content during manuscript drafting and revision, and all authors read and approved the final version for submission.

Corresponding author

Correspondence to Jie Xiang.

Ethics declarations

Ethics approval and consent to participate

Not applicable. This study only included deidentified data, which was determined to be not regulated by Institutional Review Boards (IRBMED) from University of Michigan (submission ID: HUM00202359).

Consent for publication

Not applicable.

Competing interests

Authors declare that they have no conflicts of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:

 Supplementary table 1. Proportion of Elixhauser comorbidities and body system levels for training dataset and testing dataset. Supplementary figure 1. Proportion of cases with high severity of illness on admission.   

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiang, J., Durance, P.W., Griffes, L.C. et al. Measuring case severity: a novel tool for benchmarking and clinical documentation improvement. BMC Health Serv Res 22, 513 (2022). https://doi.org/10.1186/s12913-022-07935-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12913-022-07935-1

Keywords

BMC Health Services Research

ISSN: 1472-6963

Contact us