Standard

Impact of a Machine Learning-Based Decision Support System for Urinary Tract Infections : Prospective Observational Study in 36 Primary Care Practices. / Herter, Willem Ernst; Khuc, Janine; Cinà, Giovanni et al.

In: JMIR medical informatics, Vol. 10, No. 5, e27795, 01.05.2022, p. e27795.

Research output: Contribution to journalArticleAcademicpeer-review

Harvard

Herter, WE, Khuc, J, Cinà, G, Knottnerus, BJ, Numans, ME, Wiewel, MA, Bonten, TN, de Bruin, DP, van Esch, T, Chavannes, NH & Verheij, RA 2022, 'Impact of a Machine Learning-Based Decision Support System for Urinary Tract Infections: Prospective Observational Study in 36 Primary Care Practices', JMIR medical informatics, vol. 10, no. 5, e27795, pp. e27795. https://doi.org/10.2196/27795, https://doi.org/10.2196/27795

APA

Herter, W. E., Khuc, J., Cinà, G., Knottnerus, B. J., Numans, M. E., Wiewel, M. A., Bonten, T. N., de Bruin, D. P., van Esch, T., Chavannes, N. H., & Verheij, R. A. (2022). Impact of a Machine Learning-Based Decision Support System for Urinary Tract Infections: Prospective Observational Study in 36 Primary Care Practices. JMIR medical informatics, 10(5), e27795. [e27795]. https://doi.org/10.2196/27795, https://doi.org/10.2196/27795

Vancouver

Herter WE, Khuc J, Cinà G, Knottnerus BJ, Numans ME, Wiewel MA et al. Impact of a Machine Learning-Based Decision Support System for Urinary Tract Infections: Prospective Observational Study in 36 Primary Care Practices. JMIR medical informatics. 2022 May 1;10(5):e27795. e27795. doi: 10.2196/27795, 10.2196/27795

Author

BibTeX

@article{8e18fae0b7e845e08b3c3a44e8ecdadb,
title = "Impact of a Machine Learning-Based Decision Support System for Urinary Tract Infections: Prospective Observational Study in 36 Primary Care Practices",
abstract = "Background: There is increasing attention on machine learning (ML)-based clinical decision support systems (CDSS), but their added value and pitfalls are very rarely evaluated in clinical practice. We implemented a CDSS to aid general practitioners (GPs) in treating patients with urinary tract infections (UTIs), which are a significant health burden worldwide. Objective: This study aims to prospectively assess the impact of this CDSS on treatment success and change in antibiotic prescription behavior of the physician. In doing so, we hope to identify drivers and obstacles that positively impact the quality of health care practice with ML. Methods: The CDSS was developed by Pacmed, Nivel, and Leiden University Medical Center (LUMC). The CDSS presents the expected outcomes of treatments, using interpretable decision trees as ML classifiers. Treatment success was defined as a subsequent period of 28 days during which no new antibiotic treatment for UTI was needed. In this prospective observational study, 36 primary care practices used the software for 4 months. Furthermore, 29 control practices were identified using propensity score-matching. All analyses were performed using electronic health records from the Nivel Primary Care Database. Patients for whom the software was used were identified in the Nivel database by sequential matching using CDSS use data. We compared the proportion of successful treatments before and during the study within the treatment arm. The same analysis was performed for the control practices and the patient subgroup the software was definitely used for. All analyses, including that of physicians' prescription behavior, were statistically tested using 2-sided z tests with an α level of.05. Results: In the treatment practices, 4998 observations were included before and 3422 observations (of 2423 unique patients) were included during the implementation period. In the control practices, 5044 observations were included before and 3360 observations were included during the implementation period. The proportion of successful treatments increased significantly from 75% to 80% in treatment practices (z=5.47, P < .001). No significant difference was detected in control practices (76% before and 76% during the pilot, z=0.02; P = .98). Of the 2423 patients, we identified 734 (30.29%) in the CDSS use database in the Nivel database. For these patients, the proportion of successful treatments during the study was 83%-a statistically significant difference, with 75% of successful treatments before the study in the treatment practices (z=4.95; P < .001). Conclusions: The introduction of the CDSS as an intervention in the 36 treatment practices was associated with a statistically significant improvement in treatment success. We excluded temporal effects and validated the results with the subgroup analysis in patients for whom we were certain that the software was used. This study shows important strengths and points of attention for the development and implementation of an ML-based CDSS in clinical practice.",
keywords = "ML, artificial intelligence, clinical decision support system, implementation study, information technology, machine learning, urinary tract infections",
author = "Herter, {Willem Ernst} and Janine Khuc and Giovanni Cin{\`a} and Knottnerus, {Bart J} and Numans, {Mattijs E} and Wiewel, {Maryse A} and Bonten, {Tobias N} and {de Bruin}, {Daan P} and {van Esch}, Thamar and Chavannes, {Niels H} and Verheij, {Robert A}",
note = "Funding Information: Zilveren Kruis, CZ Groep, and Menzis, 3 large health insurers in the Netherlands, partly funded this study. The funding sources had no role in the study design, analysis, interpretation of data, writing of the report, or submission of the article for publication. Publisher Copyright: {\textcopyright} Willem Ernst Herter, Janine Khuc, Giovanni Cin{\`a}, Bart J Knottnerus, Mattijs E Numans, Maryse A Wiewel, Tobias N Bonten, Daan P de Bruin, Thamar van Esch, Niels H Chavannes, Robert A Verheij.",
year = "2022",
month = may,
day = "1",
doi = "10.2196/27795",
language = "English",
volume = "10",
pages = "e27795",
journal = "JMIR medical informatics",
issn = "2291-9694",
publisher = "JMIR Publications Inc.",
number = "5",

}

RIS

TY - JOUR

T1 - Impact of a Machine Learning-Based Decision Support System for Urinary Tract Infections

T2 - Prospective Observational Study in 36 Primary Care Practices

AU - Herter, Willem Ernst

AU - Khuc, Janine

AU - Cinà, Giovanni

AU - Knottnerus, Bart J

AU - Numans, Mattijs E

AU - Wiewel, Maryse A

AU - Bonten, Tobias N

AU - de Bruin, Daan P

AU - van Esch, Thamar

AU - Chavannes, Niels H

AU - Verheij, Robert A

N1 - Funding Information: Zilveren Kruis, CZ Groep, and Menzis, 3 large health insurers in the Netherlands, partly funded this study. The funding sources had no role in the study design, analysis, interpretation of data, writing of the report, or submission of the article for publication. Publisher Copyright: © Willem Ernst Herter, Janine Khuc, Giovanni Cinà, Bart J Knottnerus, Mattijs E Numans, Maryse A Wiewel, Tobias N Bonten, Daan P de Bruin, Thamar van Esch, Niels H Chavannes, Robert A Verheij.

PY - 2022/5/1

Y1 - 2022/5/1

N2 - Background: There is increasing attention on machine learning (ML)-based clinical decision support systems (CDSS), but their added value and pitfalls are very rarely evaluated in clinical practice. We implemented a CDSS to aid general practitioners (GPs) in treating patients with urinary tract infections (UTIs), which are a significant health burden worldwide. Objective: This study aims to prospectively assess the impact of this CDSS on treatment success and change in antibiotic prescription behavior of the physician. In doing so, we hope to identify drivers and obstacles that positively impact the quality of health care practice with ML. Methods: The CDSS was developed by Pacmed, Nivel, and Leiden University Medical Center (LUMC). The CDSS presents the expected outcomes of treatments, using interpretable decision trees as ML classifiers. Treatment success was defined as a subsequent period of 28 days during which no new antibiotic treatment for UTI was needed. In this prospective observational study, 36 primary care practices used the software for 4 months. Furthermore, 29 control practices were identified using propensity score-matching. All analyses were performed using electronic health records from the Nivel Primary Care Database. Patients for whom the software was used were identified in the Nivel database by sequential matching using CDSS use data. We compared the proportion of successful treatments before and during the study within the treatment arm. The same analysis was performed for the control practices and the patient subgroup the software was definitely used for. All analyses, including that of physicians' prescription behavior, were statistically tested using 2-sided z tests with an α level of.05. Results: In the treatment practices, 4998 observations were included before and 3422 observations (of 2423 unique patients) were included during the implementation period. In the control practices, 5044 observations were included before and 3360 observations were included during the implementation period. The proportion of successful treatments increased significantly from 75% to 80% in treatment practices (z=5.47, P < .001). No significant difference was detected in control practices (76% before and 76% during the pilot, z=0.02; P = .98). Of the 2423 patients, we identified 734 (30.29%) in the CDSS use database in the Nivel database. For these patients, the proportion of successful treatments during the study was 83%-a statistically significant difference, with 75% of successful treatments before the study in the treatment practices (z=4.95; P < .001). Conclusions: The introduction of the CDSS as an intervention in the 36 treatment practices was associated with a statistically significant improvement in treatment success. We excluded temporal effects and validated the results with the subgroup analysis in patients for whom we were certain that the software was used. This study shows important strengths and points of attention for the development and implementation of an ML-based CDSS in clinical practice.

AB - Background: There is increasing attention on machine learning (ML)-based clinical decision support systems (CDSS), but their added value and pitfalls are very rarely evaluated in clinical practice. We implemented a CDSS to aid general practitioners (GPs) in treating patients with urinary tract infections (UTIs), which are a significant health burden worldwide. Objective: This study aims to prospectively assess the impact of this CDSS on treatment success and change in antibiotic prescription behavior of the physician. In doing so, we hope to identify drivers and obstacles that positively impact the quality of health care practice with ML. Methods: The CDSS was developed by Pacmed, Nivel, and Leiden University Medical Center (LUMC). The CDSS presents the expected outcomes of treatments, using interpretable decision trees as ML classifiers. Treatment success was defined as a subsequent period of 28 days during which no new antibiotic treatment for UTI was needed. In this prospective observational study, 36 primary care practices used the software for 4 months. Furthermore, 29 control practices were identified using propensity score-matching. All analyses were performed using electronic health records from the Nivel Primary Care Database. Patients for whom the software was used were identified in the Nivel database by sequential matching using CDSS use data. We compared the proportion of successful treatments before and during the study within the treatment arm. The same analysis was performed for the control practices and the patient subgroup the software was definitely used for. All analyses, including that of physicians' prescription behavior, were statistically tested using 2-sided z tests with an α level of.05. Results: In the treatment practices, 4998 observations were included before and 3422 observations (of 2423 unique patients) were included during the implementation period. In the control practices, 5044 observations were included before and 3360 observations were included during the implementation period. The proportion of successful treatments increased significantly from 75% to 80% in treatment practices (z=5.47, P < .001). No significant difference was detected in control practices (76% before and 76% during the pilot, z=0.02; P = .98). Of the 2423 patients, we identified 734 (30.29%) in the CDSS use database in the Nivel database. For these patients, the proportion of successful treatments during the study was 83%-a statistically significant difference, with 75% of successful treatments before the study in the treatment practices (z=4.95; P < .001). Conclusions: The introduction of the CDSS as an intervention in the 36 treatment practices was associated with a statistically significant improvement in treatment success. We excluded temporal effects and validated the results with the subgroup analysis in patients for whom we were certain that the software was used. This study shows important strengths and points of attention for the development and implementation of an ML-based CDSS in clinical practice.

KW - ML

KW - artificial intelligence

KW - clinical decision support system

KW - implementation study

KW - information technology

KW - machine learning

KW - urinary tract infections

UR - http://www.scopus.com/inward/record.url?scp=85129639467&partnerID=8YFLogxK

U2 - 10.2196/27795

DO - 10.2196/27795

M3 - Article

C2 - 35507396

VL - 10

SP - e27795

JO - JMIR medical informatics

JF - JMIR medical informatics

SN - 2291-9694

IS - 5

M1 - e27795

ER -

ID: 23559188