Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients

Standard

Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients : A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records. / Vagliano, Iacopo ; Schut, Martijn C.; Abu-Hanna, Ameen et al.

In: International journal of medical informatics, Vol. 167, 104863, 01.11.2022, p. 104863.

Research output: Contribution to journal › Article › Academic › peer-review

Harvard

Vagliano, I , Schut, MC , Abu-Hanna, A , Dongelmans, DA, de Lange, DW, Gommers, D, Cremer, OL, Bosman, RJ, Rigter, S, Wils, E-J, Frenzel, T, de Jong, R, Peters, MAA, Kamps, MJA, Ramnarain, D, Nowitzky, R, Nooteboom, FGCA, de Ruijter, W, Urlings-Strop, LC, Smit, EGM, Mehagnoul-Schipper, DJ, Dormans, T, de Jager, CPC, Hendriks, SHA, Achterberg, S, Oostdijk, E, Reidinga, AC, Festen-Spanjer, B, Brunnekreef, GB, Cornet, AD, van den Tempel, W, Boelens, AD, Koetsier, P, Lens, J, Faber, HJ, Karakus, A, Entjes, R, de Jong, P, Rettig, TCD, Reuland, MC, Arbous, S, Fleuren, LM, Dam, TA, Thoral, PJ, Lalisang, RCA, Tonutti, M, de Bruin, DP, Elbers, PWG & de Keizer, NF 2022, 'Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients: A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records', International journal of medical informatics, vol. 167, 104863, pp. 104863. https://doi.org/10.1016/j.ijmedinf.2022.104863

BibTeX

@article{6e46d7b1b94141ee99bde02f92bbe9b8,

title = "Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients: A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records",

abstract = "Purpose: To assess, validate and compare the predictive performance of models for in-hospital mortality of COVID-19 patients admitted to the intensive care unit (ICU) over two different waves of infections. Our models were built with high-granular Electronic Health Records (EHR) data versus less-granular registry data. Methods: Observational study of all COVID-19 patients admitted to 19 Dutch ICUs participating in both the national quality registry National Intensive Care Evaluation (NICE) and the EHR-based Dutch Data Warehouse (hereafter EHR). Multiple models were developed on data from the first 24 h of ICU admissions from February to June 2020 (first COVID-19 wave) and validated on prospective patients admitted to the same ICUs between July and December 2020 (second COVID-19 wave). We assessed model discrimination, calibration, and the degree of relatedness between development and validation population. Coefficients were used to identify relevant risk factors. Results: A total of 1533 patients from the EHR and 1563 from the registry were included. With high granular EHR data, the average AUROC was 0.69 (standard deviation of 0.05) for the internal validation, and the AUROC was 0.75 for the temporal validation. The registry model achieved an average AUROC of 0.76 (standard deviation of 0.05) in the internal validation and 0.77 in the temporal validation. In the EHR data, age, and respiratory-system related variables were the most important risk factors identified. In the NICE registry data, age and chronic respiratory insufficiency were the most important risk factors. Conclusion: In our study, prognostic models built on less-granular but readily-available registry data had similar performance to models built on high-granular EHR data and showed similar transportability to a prospective COVID-19 population. Future research is needed to verify whether this finding can be confirmed for upcoming waves.",

keywords = "Covid-19 [C01.748.610.763.500], Critical care [E02.760.190], Electronic Health Record [E05.318.308.940.968.625.500], In-hospital mortality [E05.318.308.985.550.400], Machine learning [G17.035.250.500], Prognosis [E01.789]",

author = "Iacopo Vagliano and Schut, {Martijn C.} and Ameen Abu-Hanna and Dongelmans, {Dave A.} and {de Lange}, {Dylan W.} and Diederik Gommers and Cremer, {Olaf L.} and Bosman, {Rob J.} and Sander Rigter and Evert-Jan Wils and Tim Frenzel and {de Jong}, Remko and Peters, {Marco A. A.} and Kamps, {Marlijn J. A.} and Dharmanand Ramnarain and Ralph Nowitzky and Nooteboom, {Fleur G. C. A.} and {de Ruijter}, Wouter and Urlings-Strop, {Louise C.} and Smit, {Ellen G. M.} and Mehagnoul-Schipper, {D. Jannet} and Tom Dormans and {de Jager}, {Cornelis P. C.} and Hendriks, {Stefaan H. A.} and Sefanja Achterberg and Evelien Oostdijk and Reidinga, {Auke C.} and Barbara Festen-Spanjer and Brunnekreef, {Gert B.} and Cornet, {Alexander D.} and {van den Tempel}, Walter and Boelens, {Age D.} and Peter Koetsier and Judith Lens and Faber, {Harald J.} and A. Karakus and Robert Entjes and {de Jong}, Paul and Rettig, {Thijs C. D.} and Reuland, {M. C.} and Sesmu Arbous and Fleuren, {Lucas M.} and Dam, {Tariq A.} and Thoral, {Patrick J.} and Lalisang, {Robbert C. A.} and Michele Tonutti and {de Bruin}, {Daan P.} and Elbers, {Paul W. G.} and {de Keizer}, {Nicolette F.}",

note = "Funding Information: We thank Sylvia Brinkman for her support with the extraction and understanding of the NICE data. The study protocol was reviewed by the Medical Ethics Committee of the Amsterdam Medical Center, the Netherlands. This committee provided a waiver from formal approval (W20_273 # 20.308) and informed consent since this trial does not fall within the scope of the Dutch Medical Research (Human Subjects) Act. This research was funded by The Netherlands Organisation for Health Research and Development (ZonMw) COVID-19 Programme in the bottom-up focus area 1 “Predictive diagnostics and treatment” for theme 3 “Risk analysis and prognostics” (project number 10430 01 201 0011: IRIS). The funder had no role in the design of the study or writing the manuscript. All participating hospitals have access to the Dutch ICU Data Warehouse and NICE data. The NICE registry data are available under conditions as described on the NICE website at stichting-nice.nl/extractieverzoek_procedure.jsp (in Dutch). External researchers can get access to the Dutch ICU Data Warehouse in collaboration with any of the participating hospitals. The list of collaborators is available in the co-author list and in the collaborators section, through the corresponding author, and through the contact details on amsterdammedicaldatascience.nl. Research questions have to be in line with the DSA; to investigate the course of COVID-19 in the ICU and to research potential treatments. Researchers have sign a code of conduct before accessing the data. The code used for our analyses is publicly available at bitbucket.org/aumc-kik/automl4covid. Funding Information: This research was funded by The Netherlands Organisation for Health Research and Development (ZonMw) COVID-19 Programme in the bottom-up focus area 1 “Predictive diagnostics and treatment” for theme 3 “Risk analysis and prognostics” (project number 10430 01 201 0011: IRIS). The funder had no role in the design of the study or writing the manuscript. Publisher Copyright: {\textcopyright} 2022 The Author(s)",

year = "2022",

month = nov,

day = "1",

doi = "10.1016/j.ijmedinf.2022.104863",

language = "English",

volume = "167",

pages = "104863",

journal = "International journal of medical informatics",

issn = "1386-5056",

publisher = "Elsevier Ireland Ltd",

}

RIS

TY - JOUR

T1 - Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients

T2 - A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records

AU - Vagliano, Iacopo

AU - Schut, Martijn C.

AU - Abu-Hanna, Ameen

AU - Dongelmans, Dave A.

AU - de Lange, Dylan W.

AU - Gommers, Diederik

AU - Cremer, Olaf L.

AU - Bosman, Rob J.

AU - Rigter, Sander

AU - Wils, Evert-Jan

AU - Frenzel, Tim

AU - de Jong, Remko

AU - Peters, Marco A. A.

AU - Kamps, Marlijn J. A.

AU - Ramnarain, Dharmanand

AU - Nowitzky, Ralph

AU - Nooteboom, Fleur G. C. A.

AU - de Ruijter, Wouter

AU - Urlings-Strop, Louise C.

AU - Smit, Ellen G. M.

AU - Mehagnoul-Schipper, D. Jannet

AU - Dormans, Tom

AU - de Jager, Cornelis P. C.

AU - Hendriks, Stefaan H. A.

AU - Achterberg, Sefanja

AU - Oostdijk, Evelien

AU - Reidinga, Auke C.

AU - Festen-Spanjer, Barbara

AU - Brunnekreef, Gert B.

AU - Cornet, Alexander D.

AU - van den Tempel, Walter

AU - Boelens, Age D.

AU - Koetsier, Peter

AU - Lens, Judith

AU - Faber, Harald J.

AU - Karakus, A.

AU - Entjes, Robert

AU - de Jong, Paul

AU - Rettig, Thijs C. D.

AU - Reuland, M. C.

AU - Arbous, Sesmu

AU - Fleuren, Lucas M.

AU - Dam, Tariq A.

AU - Thoral, Patrick J.

AU - Lalisang, Robbert C. A.

AU - Tonutti, Michele

AU - de Bruin, Daan P.

AU - Elbers, Paul W. G.

AU - de Keizer, Nicolette F.

N1 - Funding Information: We thank Sylvia Brinkman for her support with the extraction and understanding of the NICE data. The study protocol was reviewed by the Medical Ethics Committee of the Amsterdam Medical Center, the Netherlands. This committee provided a waiver from formal approval (W20_273 # 20.308) and informed consent since this trial does not fall within the scope of the Dutch Medical Research (Human Subjects) Act. This research was funded by The Netherlands Organisation for Health Research and Development (ZonMw) COVID-19 Programme in the bottom-up focus area 1 “Predictive diagnostics and treatment” for theme 3 “Risk analysis and prognostics” (project number 10430 01 201 0011: IRIS). The funder had no role in the design of the study or writing the manuscript. All participating hospitals have access to the Dutch ICU Data Warehouse and NICE data. The NICE registry data are available under conditions as described on the NICE website at stichting-nice.nl/extractieverzoek_procedure.jsp (in Dutch). External researchers can get access to the Dutch ICU Data Warehouse in collaboration with any of the participating hospitals. The list of collaborators is available in the co-author list and in the collaborators section, through the corresponding author, and through the contact details on amsterdammedicaldatascience.nl. Research questions have to be in line with the DSA; to investigate the course of COVID-19 in the ICU and to research potential treatments. Researchers have sign a code of conduct before accessing the data. The code used for our analyses is publicly available at bitbucket.org/aumc-kik/automl4covid. Funding Information: This research was funded by The Netherlands Organisation for Health Research and Development (ZonMw) COVID-19 Programme in the bottom-up focus area 1 “Predictive diagnostics and treatment” for theme 3 “Risk analysis and prognostics” (project number 10430 01 201 0011: IRIS). The funder had no role in the design of the study or writing the manuscript. Publisher Copyright: © 2022 The Author(s)

PY - 2022/11/1

Y1 - 2022/11/1

N2 - Purpose: To assess, validate and compare the predictive performance of models for in-hospital mortality of COVID-19 patients admitted to the intensive care unit (ICU) over two different waves of infections. Our models were built with high-granular Electronic Health Records (EHR) data versus less-granular registry data. Methods: Observational study of all COVID-19 patients admitted to 19 Dutch ICUs participating in both the national quality registry National Intensive Care Evaluation (NICE) and the EHR-based Dutch Data Warehouse (hereafter EHR). Multiple models were developed on data from the first 24 h of ICU admissions from February to June 2020 (first COVID-19 wave) and validated on prospective patients admitted to the same ICUs between July and December 2020 (second COVID-19 wave). We assessed model discrimination, calibration, and the degree of relatedness between development and validation population. Coefficients were used to identify relevant risk factors. Results: A total of 1533 patients from the EHR and 1563 from the registry were included. With high granular EHR data, the average AUROC was 0.69 (standard deviation of 0.05) for the internal validation, and the AUROC was 0.75 for the temporal validation. The registry model achieved an average AUROC of 0.76 (standard deviation of 0.05) in the internal validation and 0.77 in the temporal validation. In the EHR data, age, and respiratory-system related variables were the most important risk factors identified. In the NICE registry data, age and chronic respiratory insufficiency were the most important risk factors. Conclusion: In our study, prognostic models built on less-granular but readily-available registry data had similar performance to models built on high-granular EHR data and showed similar transportability to a prospective COVID-19 population. Future research is needed to verify whether this finding can be confirmed for upcoming waves.

AB - Purpose: To assess, validate and compare the predictive performance of models for in-hospital mortality of COVID-19 patients admitted to the intensive care unit (ICU) over two different waves of infections. Our models were built with high-granular Electronic Health Records (EHR) data versus less-granular registry data. Methods: Observational study of all COVID-19 patients admitted to 19 Dutch ICUs participating in both the national quality registry National Intensive Care Evaluation (NICE) and the EHR-based Dutch Data Warehouse (hereafter EHR). Multiple models were developed on data from the first 24 h of ICU admissions from February to June 2020 (first COVID-19 wave) and validated on prospective patients admitted to the same ICUs between July and December 2020 (second COVID-19 wave). We assessed model discrimination, calibration, and the degree of relatedness between development and validation population. Coefficients were used to identify relevant risk factors. Results: A total of 1533 patients from the EHR and 1563 from the registry were included. With high granular EHR data, the average AUROC was 0.69 (standard deviation of 0.05) for the internal validation, and the AUROC was 0.75 for the temporal validation. The registry model achieved an average AUROC of 0.76 (standard deviation of 0.05) in the internal validation and 0.77 in the temporal validation. In the EHR data, age, and respiratory-system related variables were the most important risk factors identified. In the NICE registry data, age and chronic respiratory insufficiency were the most important risk factors. Conclusion: In our study, prognostic models built on less-granular but readily-available registry data had similar performance to models built on high-granular EHR data and showed similar transportability to a prospective COVID-19 population. Future research is needed to verify whether this finding can be confirmed for upcoming waves.

KW - Covid-19 [C01.748.610.763.500]

KW - Critical care [E02.760.190]

KW - Electronic Health Record [E05.318.308.940.968.625.500]

KW - In-hospital mortality [E05.318.308.985.550.400]

KW - Machine learning [G17.035.250.500]

KW - Prognosis [E01.789]

UR - http://www.scopus.com/inward/record.url?scp=85138464263&partnerID=8YFLogxK

U2 - 10.1016/j.ijmedinf.2022.104863

DO - 10.1016/j.ijmedinf.2022.104863

M3 - Article

C2 - 36162166

VL - 167

SP - 104863

JO - International journal of medical informatics

JF - International journal of medical informatics

SN - 1386-5056

M1 - 104863

ER -

ID: 26147681

Generic Portal