Improving the assessment of adverse drug reactions using the Naranjo Algorithm in daily practice: The Japan Adverse Drug Events Study

Abstract It is difficult to determine adverse drug reactions (ADRs) in daily complicated clinical practice in which many kinds of drugs are prescribed. We evaluated how well the Naranjo Algorithm (NA) categorized ADRs among suspected ADRs. The Japan Adverse Drug Events (JADE) study was a prospective cohort study of 3459 inpatients. After all suspected ADRs were reported from research assistants, a single physician reviewer independently assigned an NA score to each. After all NA score of suspected ADRs were scored, two physician reviewers discussed and determined ADRs based on the literature. We investigated the sensitivity and specificity of NA and each component to categorize ADRs among suspected ADRs. A total of 1579 suspected ADRs were reported in 962 patients. Physician reviewers determined 997 ADRs. The percentage of ADRs was 94% if the total NA score reached 5. The modified NA consisted of 5 components that showed high classification abilities; its area under the curve (AUC) was 0.92 for categorizing ADRs, the same as the original. When we set the total NA score cut‐off value to 5, specificity was 0.95 and sensitivity was 0.59. When we reclassified NA components as binary variables, the specificity increased to 0.98 with a cut‐off value of 4 and yielded an AUC of 0.93. In conclusion, we showed that both NA and modified NA could categorize ADRs among suspected ADRs with a high likelihood in daily clinical practice.

among complicated suspected symptoms could be useful for healthcare professionals to take action proactively as well as to confirm the probability of ADRs retrospectively.
Naranjo et al proposed a tool to evaluate the probability of true ADRs from suspected ADRs, 1,2 and it has been widely used as the Naranjo Algorithm (NA). [3][4][5][6] In addition to the NA, several assessment tools have been developed, such as the Liverpool adverse drug reaction causality assessment tool 7

and the French Causality Assessment
Method. 8 These tools are used to evaluate the probability of an ADR rather than to screen ADRs from suspected ADRs prospectively to take action. While the NA is a traditional tool, it consists of 10 components, and it is complicated to calculate the total score and would require time to utilize it in a daily clinical setting. To save time and resources, a convenient tool to categorize ADRs with high specificity is needed.
We recently conducted the Japan Adverse Drug Events (JADE) study, which evaluated the incidence of ADRs and medication errors among Japanese hospitalized inpatients. [9][10][11][12][13][14] In the present study, we evaluated the usefulness of the NA to categorize ADRs among suspected ADRs using the JADE database and tried to modify it into a convenient tool to use in daily clinical practice.

| Study design and patient population
The JADE study was a multicenter prospective cohort study that included 3459 inpatients aged ≥15 years. The study site was three urban tertiary care hospitals in Japan, patients admitted at 15 randomly selected medical and surgical wards as well as three intensive care units from January through June 2004 were eligible for this study. 9 The institutional review boards of the three participating hospitals approved the study. Informed consent was waived because all data were collected in daily practice.

| Naranjo Algorithm
The NA consists of 10 components assessing the likelihood of ADRs. 1,2 Each component is scored from À1 to +2 based on the findings of each event, including (1) previous conclusive reports, (2) time course, (3) improvement after withdrawal or treatment, (4) re-emergence after rechallenge, (5) other causative conditions of symptoms, (6) response to placebo if used, (7) evidence in blood of toxicity, (8) dose response, (9) similar reactions before, and (10) other objective evidence.

| Data collection and review process
Research assistants, who were trained nurses or nursing students, reviewed all medical charts, along with laboratory results, incident reports, and prescription queries by pharmacists with the standardized form daily. They reported any suspected ADRs that might be potential ADRs in a standard manner. 15 After all suspected ADRs were reported from research assistants, a single physician reviewer independently assigned an NA score to each suspected ADR. After all NA score of suspected ADRs were scored, two independent physician reviewers evaluated all suspected ADRs and classified them as confirmed ADRs or not. If discordance happened, such discordance was resolved through discussion to reach consensus.

| Statistical analyses
A continuous variable is presented as the mean AE standard deviation (SD) and categorical variables are shown as numbers and percentages. We expressed the distribution of NA scores in each component as the percentage of confirmed ADRs among suspected ADRs for each score in each component. We evaluated the percentage of confirmed ADRs among suspected ADRs for each total NA score.
ADRs which are confirmed by physician reviewers are considered as true positive. All suspected ADRs were categorized as positive or negative based on the NA score; then sensitivity and specificity were calculated by these figures. We constructed a receiver operating characteristic (ROC) curve for the summed score of all and selected NA components to compare the categorization abilities of original and modified NA scores. To simplify the NA for convenient use, we reclassified NA components as binary variables. For example, an NA component that had three possible scores, such as +2, 0, and À1 or +1, 0, and 1, were converted to +1 and 0 in which the positive score was converted to +1 and the 0 and negative scores were summarized as 0. We carried out all analyses using JMP 11.2 (SAS Institute Inc., Cary, NC, USA) software.  and 37% (n = 577) of suspected ADRs were classified with a 0 score (no or do not know) for component 1 (Table 3). 3.2 | Relationship between total NA score and

ADRs percentage of suspected ADRs
The total NA score calculated for each suspected ADR ranged from À2 to 11. The most frequent total NA score was 0 (n=403) followed by 5 (n=280). The percentage of ADRs was 56% if the total NA score was 1, and it gradually increased to 94% if the total NA score reached 5 ( Figure 3). We did not show the total NA scores of À2 and À1 since only 2 and 0 suspected ADRs, respectively, were assigned these scores.  (17) Antidiabetics 139 (14) Antipsychotics 119 (12) Dyslipidemic agents 73 (7) Analgesics 42 (4) NSAIDs, nonsteroidal anti-inflammatory drugs.
MURAYAMA ET AL.  Figure 4A). Since more than 97% of suspected ADRs were assigned a score of 0 for components 6 through 10, we considered that these components were not useful in the real-world setting. We generated a modified NA that consisted of components 1

| DISCUSSION
We showed that the NA was able to categorize ADRs among suspected ADRs efficiently in daily clinical practice using the large-scale JADE database, 9 which was independent with a consensus panel by physicians' reviewers. While each NA component showed relatively high sensitivity or specificity, we evaluated the sensitivity or specificity for the total NA score, since healthcare professionals usually make a decision from multiple factors in the actual clinical setting.
We also showed that the modified NA, consisting of components 1 through 5, also effectively categorized ADRs with a high likelihood.
We further modified the NA to include all binary scores for compo- In previous studies, the NA was utilized retrospectively to evaluate the probabilities of ADRs in a specific case or cohort. [3][4][5][6] In this study, however, we showed that the NA had high predictive accuracy for determining true ADRs among suspected ADRs, which could contribute to safety monitoring activities by healthcare professionals or pharmaceutical manufacturers. If the modified NA score is simultaneously reported with a suspected ADR, a health authority or pharmaceutical manufacturers could evaluate the suspected ADR more easily and quickly and could allocate time and resources more effectively. For example, pharmaceutical manufacturers could start an intensive survey giving priority to a suspected ADR with a high modified NA score. Additionally, healthcare professionals could start preclinical studies to clarify the mechanism of ADRs focusing on a high modified NA score. Thus, the modified NA score could help healthcare professionals or pharmaceutical manufacturers take their own action in preventing ADRs as early as possible before health authorities issue a warning or guidance.
NA was reported to show poor performance for causality assessment of hepatic adverse reactions. 16,17 On the other hand, NA and modified NA were able to categorize ADRs among suspected ADRs including hepatic adverse reactions in the current study. However, the number of hepatic adverse reactions was limited in the current study, the reliability to assess such hepatic adverse reactions was uncertain. Further studies which address the accuracy of NA and modified NA against hepatic adverse reaction should be considered.
Other than the NA, Gallagher et al reported the usefulness of the Liverpool adverse drug reaction causality assessment tool. 7 Although this tool also tried to simplify the NA and increase its credibility, their study had different objectives. It takes time to evaluate one case and provide an outcome (possible, probable, or definite) using the probability tree in the Liverpool tool. Additionally, this tool does not provide any score to be evaluated for sensitivity and specificity, similar to the NA. Also WHO-UMC causality assessment could be another simple tool to categorize ADR. 18  occurred in this study. Therefore, we could not exclude the possibility of synergistic effects of multiple drugs and drug-drug interaction.
In conclusion, we assessed the categorization abilities of the original and modified NAs in daily practice and found that the modified NA could be easily used to categorize actual ADRs among suspected ADRs with high predictive accuracy. Therefore, use of the modified NA could help to save time and resources and categorize ADRs more effectively and promptly in daily clinical practice. Additionally, utilizing this tool for a pharmacovigilance system could be useful to enable professionals take prompt action in developing a strategy to prevent and mitigate the adverseness of ADRs.

ACKNOWLEDGEMENT
The JADE study for adult inpatients was conducted by the following