Effect of knowledgebase transition of a clinical decision support system on medication order and alert patterns in an emergency department
This was a retrospective study that used data from an EMR. This study was approved by the Institutional Review Board (IRB) of Samsung Medical Center (IRB no. 2021-01-169). The requirement for informed consent was waived by the Institutional Review Board of Samsung Medical Center because de-identified data was used for analysis, and the study is retrospective and observational. All methods were performed and reported in accordance with “Strengthening the Reporting of Observational Studies in Epidemiology” (STROBE) guidelines18, and in accordance with the relevant guidelines and regulations. It was not appropriate or possible to involve patients or the public in the design, or conduct, or reporting, or dissemination plans of our research public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
The study was conducted in the emergency department (ED) of a tertiary urban academic medical center in Seoul, Korea. It is an acute care teaching hospital that receives approximately two million outpatients annually and has 1975 beds. Approximately 1000 doctors and 6000 nurses work in the institute. The ED has 69 beds with about 35 doctors, and the average annual patient volume ranges from 75,000 to 80,000. In Korea, only medical doctors can legally prescribe medication orders with very restricted exceptions.
Electronic medical record system
The EMR used in this study was an internally developed system, which was rolled out in July 2016, replacing the previous internally developed EMR. The new EMR is a part of a hospital information system called Data Analytics and Research Window for Integrated kNowledge (DARWIN). DARWIN is a comprehensive system that contains computerized order entry from physicians as well as nurses, pharmacists, and billing and research support departments and even includes patient portal and web services.
Computerized physician order entry and passive alert with inline text
Within DARWIN’s computerized physician order entry (CPOE) process, the prescription process is carried out as a sequence of actions. Once the process is initiated, a patient is selected, the diagnosis is confirmed, and orders for tests and medications are followed. When ordering a specific medicine, the medicine is searched for, and specifics such as doses, routes, and duration are entered. A schematic of this process is presented in Fig. 1. Alerts are categorized into two groups. The first group comprises non-adjustable alerts where users must change the drug in response to the alert because factors such as age and allergy cannot be changed. The other refers to adjustable alerts where users can change the specifics, such as dosage and route, to make the order consistent with the alert algorithm.
Clinical decision support system design: passive inline text
A passive clinical decision support (CDS) system is integrated into the DARWIN’s computerized physician order entry (CPOE) for prescriptions. A passive CDS system is not likely to interfere in the work process of physicians. Although this type of alert may reduce alert fatigue, it may also result in decreased effectiveness of the CDS system19,20. As shown in Fig. 1, the alert appears before confirming the order.
Clinical decision support system knowledgebase transition
In addition to the user interface, a knowledgebase (KB) for the CDS system was purchased commercially. Initially, the KB was supplied by Medi-Span (Wolters Kluwer Health, Philadelphia, PA, USA), with monthly updates. The types of alerts (domain) were related to age, allergy, disease, duplication, sex, lactation, pregnancy, dose, drug-drug interaction (DDI), and route. The KB was then changed to KIMS POC (KIMS, Seoul, Korea). The new KB covers a smaller range of medications and does not provide disease-drug and duplication alerts. The update interval was a week. In addition, the new KB was lower in cost than the previous KB (Table 1).
Patients who visited the emergency department from January 2018 to December 2020 and were prescribed medication during their visit were eligible for inclusion in this study. Patients’ basic characteristics and clinical information regarding their visits to the emergency department were collected. The wash-out period was set from May 2019 to July 2019 (three months) to reflect the adjustment period for technical changes. All medication-related orders and alerts for these patients, and the basic information of the physicians were included in the analysis.
Data extraction and preparation
Data on patients, physicians, and medication-related orders and alerts were extracted from the clinical data warehouse (CDW) of the study site. Patient data included age, sex, triage score, and visit date. The specifics of the physicians were also collected, including specialty department and position (trainee versus faculty [board-certified physicians]). Order data included patient identifier (ID), prescribing physician’s information, and prescribed medication information (order log ID, order date, drug name, dose, duration, and route). Alert data included patient ID, physician’s information, prescribed medication information, alert log ID, type of alert, and alert messages.
Definition of orders
In this study, medication-related orders for all ED-based orders were included with a few exclusion criteria. Pro re nata orders were excluded because the final confirmation was made by nurses who did not receive medication-related alerts. Administrative order record data and fluid-type medications were excluded. The excluded orders did not generate any alerts.
The data comprised confirmed orders and intended (but withdrawn) orders. Multiple sequential alerts for a patient on the same medication provided by the same doctor were counted as a single order to reflect the intention of the physician. Order data captured dosage alteration within the same medication but failed to reflect drug changes after an alert has been generated. Given that a physician’s intention to change medication was unclear on hindsight, the new order was counted independently (Fig. 1).
Definition of alerts and alert overrides
A set of definitions was required because an alert is generated before the order is confirmed. For non-adjustable alerts—age, sex, duplication, or DDI alerts—users have no other way to resolve the alert than to replace the selected drug. In this case, only the alert data remained without an order (Fig. 2). For adjustable alerts—dose or route alerts—users can either replace the drug to another or change the dose/route specifics to resolve the alert. Renal alerts were classified as non-adjustable or adjustable based on the alert messages.
A single event in the order data was a single confirmed medication-related order. Alert data were utilized to add intended (but withdrawn) order cases. To clarify a physician’s intention regarding the alert override, multiple alerts generated for a given drug were grouped appropriately based on a set of rules. If multiple adjustable alerts were given during the adjustment, only the final attempt was recorded. If a single drug led to multiple types of alerts, only one of each alert type was recorded.
A general rule was applied to define whether the alert was overridden. If a physician decided to delete the drug from the order after a non-adjustable or adjustable alert was given, processes (1) and (4) in Fig. 2, the physician was considered to not override the alert. If the physician decided to continue with the initial order after a non-adjustable or adjustable alert was given, processes (2) and (5), the physician was considered to have overridden the alert. Process (3) describes the case where a physician made adjustments to the drug dose or route after an adjustable alert was fired. If a physician followed the alert accordingly, the alert was not considered overridden.
Data analysis and visualization
The study period was divided into two periods, A and B, based on the timing of the KB transition. The basic characteristics of the patients and alerts were described using simple statistics. Patient characteristics were compared for the two periods, and p-values were computed using a chi-squared test at a 0.001 significance level.
Changes in the order and alert patterns of commonly used drugs before and after the transition were examined. The alert rate (alert count divided by order count) and change in alert rate (period B alert rate minus period A alert rate) were computed. Drugs were then sub-grouped according to whether their alert rate increased or decreased after the transition compared to that before. The top 20 most commonly prescribed drugs during the study period were selected, and their alert patterns were examined. Changes in alert rate were tested using a chi-squared test and Fisher’s test at a 0.001 significance level. All analyses were performed using the statistical software, R (v4.0.3).
A direct comparison of alert types between the two vendors was not possible. Six out of 10 types of alerts—sex, pregnancy, lactation, disease, duplication, and route alerts—were not included by the period B vendor. Instead, the period B vendor provided a renal-type alert. Age-type alerts were initially not provided separately by the vendor used during period B. Instead, they were included as a subgroup under dose-type alerts. These subgroups were grouped prospectively as age-type alerts based on alert messages. The alert type composition of the alerts generated was broken down by month and visualized in conjunction with monthly override rates to observe the change in alert types over time.