This bibliogaphy of evaluation studies of decision support systems is provided as a service: a listing here does not imply that OpenClinical endorses the methods used or the conclusions reached. Members and visitors are invited to submit further references for publication.

Reference

System and medical domain

Sintchenko V, Coiera E, Iredell JR, Gilbert GL. Comparative impact of guidelines, clinical data, and decision support on prescribing decisions: an interactive web experiment with simulated cases. J Am Med Inform Assoc. 2004 Jan-Feb;11(1):71-7. [PubMed]   []

" OBJECTIVE: The aim of this study was to compare the clinical impact of computerized decision support with and without electronic access to clinical guidelines and laboratory data on antibiotic prescribing decisions. DESIGN: A crossover trial was conducted of four levels of computerized decision support-no support, antibiotic guidelines, laboratory reports, and laboratory reports plus a decision support system (DSS), randomly allocated to eight simulated clinical cases accessed by the Web. MEASUREMENTS: Rate of intervention adoption was measured by frequency of accessing information support, cost of use was measured by time taken to complete each case, and effectiveness of decision was measured by correctness of and self-reported confidence in individual prescribing decisions. Clinical impact score was measured by adoption rate and decision effectiveness. RESULTS: Thirty-one intensive care and infectious disease specialist physicians (ICPs and IDPs) participated in the study. Ventilator-associated pneumonia treatment guidelines were used in 24 (39%) of the 62 case scenarios for which they were available, microbiology reports in 36 (58%), and the DSS in 37 (60%). The use of all forms of information support did not affect clinicians' confidence in their decisions. Their use of the DSS plus microbiology report improved the agreement of decisions with those of an expert panel from 65% to 97% (p=0.0002), or to 67% (p=0.002) when antibiotic guidelines only were accessed. Significantly fewer IDPs than ICPs accessed information support in making treatment decisions. On average, it took 245 seconds to make a decision using the DSS compared with 113 seconds for unaided prescribing... The DSS plus microbiology reports had the highest clinical impact score (0.58), greater than that of electronic guidelines (0.26) and electronic laboratory reports (0.45). CONCLUSION: When used, computer-based decision support significantly improved decision quality. In measuring the impact of decision support systems, both their effetiveness in improving decisions and their likely rate of adoption in the clinical environment need to be considered. Clinicians chose to use antibiotic guidelines for one third and microbiology reports or the DSS for about two thirds of cases when they were available to assist their prescribing decisions. "

Tierney WM, Overhage JM, Murray MD et al. Effects of computerized guidelines for managing heart disease in primary care. J Gen Intern Med. 2003 Dec;18(12):967-76. [PubMed]   [blackwell-synergy.com]

" BACKGROUND: Electronic information systems have been proposed as one means to reduce medical errors of commission (doing the wrong thing) and omission (not providing indicated care). OBJECTIVE: To assess the effects of computer-based cardiac care suggestions. DESIGN: A randomized, controlled trial targeting primary care physicians and pharmacists. SUBJECTS: A total of 706 outpatients with heart failure and/or ischemic heart disease. INTERVENTIONS: Evidence-based cardiac care suggestions, approved by a panel of local cardiologists and general internists, were displayed to physicians and pharmacists as they cared for enrolled patients. MEASUREMENTS: Adherence with the care suggestions, generic and condition-specific quality of life, acute exacerbations of their cardiac disease, medication compliance, health care costs, satisfaction with care, and physicians' attitudes toward guidelines. RESULTS: Subjects were followed for 1 year during which they made 3,419 primary care visits and were eligible for 2,609 separate cardiac care suggestions. The intervention had no effect on physicians' adherence to the care suggestions (23% for intervention patients vs 22% for controls). There were no intervention-control differences in quality of life, medication compliance, health care utilization, costs, or satisfaction with care. Physicians viewed guidelines as providing helpful information but constraining their practice and not helpful in making decisions for individual patients. CONCLUSIONS: Care suggestions generated by a sophisticated electronic medical record system failed to improve adherence to accepted practice guidelines or outcomes for patients with heart disease. Future studies must weigh the benefits and costs of different (and perhaps more Draconian) methods of affecting clinician behavior. "

Van Wyk JT, Van Wijk MA, Moorman PW, Mosseveld M, Van Der Lei J. Cholgate - a randomized controlled trial comparing the effect of automated and on-demand decision support on the management of cardiovascular disease factors in primary care. Proc AMIA Symp. 2003;:1040. [PubMed]

" Automated and on-demand decision support systems integrated into an electronic medical record have proven to be an effective implementation strategy for guidelines. Cholgate is a randomized controlled trial comparing the effect of automated and on-demand decision support on the management of cardiovascular disease factors in primary care. "

van Wijk MA, van der Lei J, Mosseveld M, Bohnen AM, van Bemmel JH. Compliance of general practitioners with a guideline-based decision support system for ordering blood tests. Clin Chem. 2002 Jan;48(1):55-60. [PubMed]   [clinchem.org]

" BACKGROUND: Guidelines are viewed as a mechanism for disseminating a rapidly increasing body of knowledge. We determined the compliance of Dutch general practitioners with the recommendations for blood test ordering as defined in the guidelines of the Dutch College of General Practitioners. METHODS: We performed an audit of guideline compliance over a 12-month period (March 1996 through February 1997). In an observational study, a guideline-based decision support system for blood test ordering, BloodLink, was integrated with the electronic patient records of 31 general practitioners practicing in 23 practices (16 solo). BloodLink followed the guidelines of the Dutch College of General Practitioners. We determined compliance by comparing the recommendations for test ordering with the test(s) actually ordered. Compliance was expressed as the percentage of order forms that followed the recommendations for test ordering. RESULTS: Of 12 668 orders generated, 9091 (71%) used the decision-support software rather than the paper order forms. Twelve indications accounted for>80% of the 7346 order forms that selected a testing indication in BloodLink. The most frequently used indication for test ordering was "vague complaints" (2209 order forms; 30.1%). Of the 7346 order forms, 39% were compliant. The most frequent type of noncompliance was the addition of tests. Six of the 12 tests most frequently added to the order forms were supported by revisions of guidelines that occurred within 3 years after the intervention period. CONCLUSIONS: In general practice, noncompliance with guidelines is predominantly caused by adding tests. We conclude that noncompliance with a guideline seems to be partly caused by practitioners applying new medical insight before it is incorporated in a revision of that guideline. "

Rousseau N, McColl E, Newton J, Grimshaw J, Eccles M. Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. BMJ. 2003 Feb 8;326(7384):314. [PubMed]   [PubMed Central]

" OBJECTIVE: To understand the factors influencing the adoption of a computerised clinical decision support system for two chronic diseases in general practice. DESIGN: Practice based, longitudinal, qualitative interview study. SETTING: Five general practices in north east England. PARTICIPANTS: 13 respondents (two practice managers, three nurses, and eight general practitioners) gave a total of 19 semistructured interviews. 40 people in practices included in the randomised controlled trial (34 doctors, three nurses) and interview study (three doctors, one previously interviewed) gave feedback. RESULTS: Negative comments about the decision support system significantly outweighed the positive or neutral comments. Three main areas of concern among clinicians emerged: timing of the guideline trigger, ease of use of the system, and helpfulness of the content. Respondents did not feel that the system fitted well within the general practice context. Experience of "on-demand" information sources, which were generally more positively viewed, informed the comments about the system. Some general practitioners suggested that nurses might find the guideline content more clinically useful and might be more prepared to use a computerised decision support system, but lack of feedback from nurses who had experienced the system limited the ability to assess this. CONCLUSIONS: Significant barriers exist to the use of complex clinical decision support systems for chronic disease by general practitioners. Key issues include the relevance and accuracy of messages and the flexibility to respond to other factors influencing decision making in primary care. "

Price GJ, McCluggage WG, Morrison M ML, et al. Computerized diagnostic decision support system for the classification of preinvasive cervical squamous lesions. Hum Pathol. 2003 Nov;34(11):1193-203. [PubMed]    [Science Direct]

" Previous studies have revealed considerable interobserver and intraobserver variation in the histological classification of preinvasive cervical squamous lesions. The aim of the present study was to develop a decision support system (DSS) for the histological interpretation of these lesions. Knowledge and uncertainty were represented in the form of a Bayesian belief network that permitted the storage of diagnostic knowledge and, for a given case, the collection of evidence in a cumulative manner that provided a final probability for the possible diagnostic outcomes. The network comprised 8 diagnostic histological features (evidence nodes) that were each independently linked to the diagnosis (decision node) by a conditional probability matrix... DSSs such as this ... not only have a role to play in enhancing decision making but also in the study of diagnostic protocol, education, self-assessment, and quality control. "

Tural C, Ruiz L, Holtzer C et al; Havana Study Group. Clinical utility of HIV-1 genotyping and expert advice: the Havana trial. AIDS. 2002 Jan 25;16(2):209-18. [PubMed]    []

OBJECTIVE: To determine whether HIV-1 genotyping and expert advice add additional short-term virologic benefit in guiding antiretroviral changes in HIV+ drug-experienced patients.

Quaglini S, Grandi M, Baiardi P et al. A computerized guideline for pressure ulcer prevention. Int J Med Inf. 2000 Sep;58-59:207-17. [PubMed]

" This paper illustrates the implementation of a computerized guideline for pressure ulcer prevention. In particular, it describes the aspects related to the site-specification of a guideline delivered by the Agency for Health Care Policy Research (AHCPR), to its integration with the electronic patient record, and to its implementation within the clinical routine. The primary goal of the system is both to facilitate nurses assessing the risk of ulcer development, and to manage patients at risk by producing daily prevention work-plans. Concerning this functionality, particular attention has been paid to manage nurse's non-compliance with the guideline suggestions and to collect data for evaluating the guideline impact. Moreover, since it is well known that nurses are often over-loaded, the human computer interaction has been studied in such a way to optimise the time spent for data input. An additional functionality of the system is the novice nurses' education - they can browse a graphical representation of the guideline, asking details about the different tasks, and they can simulate patients to obtain real-time advice. The educational tool is written in Java and it is based on a representation of the guideline as a relational database. A preliminary evaluation of the system has been performed and the results are presented on the management of about 40 patients. "

Berner ES, Webster GD, Shugerman AA et al. Performance of four computer-based diagnostic systems N Engl J Med 1994;330:1792-6. [PubMed]    [NEJM]

Assessment of the diagnostic capabilities of four internal medicine diagnostic systems: Dxplain, Iliad, Meditel, and QMR. "Background Computer-based diagnostic systems are available commercially, but there has been limited evaluation of their performance. We assessed the diagnostic capabilities of four internal medicine diagnostic systems: Dxplain, Iliad, Meditel, and QMR. ... Results No single computer program scored better than the others on all performance measures. Among all cases and all programs, the proportion of correct diagnoses ranged from 0.52 to 0.71, and the mean proportion of relevant diagnoses ranged from 0.19 to 0.37. On average, less than half the diagnoses on the experts' original list of reasonable diagnoses were suggested by any of the programs. However, each program suggested an average of approximately two additional diagnoses per case that the experts found relevant but had not originally considered. Conclusions The results provide a profile of the strengths and limitations of these computer programs. The programs should be used by physicians who can identify and use the relevant information and ignore the irrelevant information that can be produced. "

Collste G, Shahsavar N, Gill H. A decision support system for diabetes care: ethical aspects. Methods Inf Med. 1999 Dec;38(4-5):313-6. [PubMed]    [Schattauer] [Sweden]

" The design and implementation of a decision support system for diabetes care is examined from an ethical perspective".

Hunt DL, Haynes RB, Hayward RS, Pim MA, Horsman J. Patient-specific evidence-based care recommendations for diabetes mellitus: development and initial clinic experience with a computerized decision support system. [PubMed]

QMR Evaluation of self-administered computer-based questionnaire (containing evidence-based care recommendations) for diabetes patients

Berner ES, Maisiak RS, Cobbs CG, Taunton OD. Effects of a decision support system on physicians' diagnostic performance. J Am Med Inform Assoc. 1999 Sep-Oct;6(5):420-7. [PubMed]    [PubMedCentral]

Study on how the information provided by a diagnostic decision support system (here, QMR) for clinical cases of varying diagnostic difficulty affects physicians' diagnostic performance.

Emery J, Walton R, Coulson A et al. Computer support for recording and interpreting family histories of breast and ovarian cancer in primary care (RAGs): qualitative evaluation with simulated patients. British Medical Journal - BMJ 1999;319:32-36 [PubMed]   [BMJ] Emery J, Walton R, Murphy M et al, Computer support for interpreting family histories of breast and ovarian cancer in primary care: comparative study with simulated cases. BMJ. 2000 Jul 1;321(7252):28-32. [PubMed]   [PubMedCentral]

Qualitative evaluations of RAGs, a DSS for risk assessment in breast cancer.

Walton RT, Gierl C, Yudkin P et al. Evaluation of computer support for prescribing (CAPSULE) using simulated cases. BMJ. 1997 Sep 27;315(7111):791-5. [PubMed]    [BMJ]

Evaluation of CAPSULE, primary care prescribing DSS.

MAM van Wijk, J van der Lei, M Mosseveld et al. Assessment of decision support for blood test ordering in primary care. Annals of Internal Medicine, Volume 134 Number 4; 274-281; 2001. [Summary on Bandolier] [Ann Intern Med]

" "

Thomas KW, Dayton CS, Peterson MW. Evaluation of internet-based clinical decision support systems. J Med Internet Res. 1999 Oct-Dec;1(2):E6. [PubMed]    [JMIR] [Asthma Education: Interactive Guidelines (Adapted in part from the National Asthma Education Program 2 Guidelines)].

Study of the effect of two web-enabled clinical DSSs for asthma and tuberculosis preventive therapy on the quality of decision making (i.e. compliance with national guidelines).

Persson M, Mjorndal T, Carlberg B et al. Evaluation of a computer-based decision support system for treatment of hypertension with drugs: retrospective, nonintervention testing of cost and guideline adherence. J Intern Med. 2000 Jan;247(1):87-93. [PubMed]    [J Intern Med]

Study on the effect of a computerized decision support guideline for drug treatment of hypertension on the quality, safety, and cost of treatment.

Friedman CP, Elstein AS, Wolf FM, Murphy GC, Franz TM, Heckerling PS, Fine PL, Miller TM, Abraham V. Enhancement of clinicians' diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. JAMA. 1999 Nov 17;282(19):1851-6. [PubMed]    []

Evaluation of 2 diagnostic DSS: ILIAD (version 4.2) and Quick Medical Reference (QMR; version 3.7.1).

Kristensen B, Andreassen S, Leibovici L, Riekehr C, Kjaer AG, Schonheyder HC. Empirical treatment of bacteraemic urinary tract infection. Evaluation of a decision support system. Dan Med Bull. 1999 Sep;46(4):349-53. [PubMed]    []

Evaluation of DSS for guidance of empirical antibiotic therapy in patients with bacteraemia originating from the urinary tract. The DSS was based upon a causal probabilistic network.

NHS decision-support system (Prodigy), which offers contextual clinical guidance in general practice consultations.

Wilson RG, Purves IN, Smith D. Utilisation of computerised clinical guidance in general practice consultations. Stud Health Technol Inform. 2000;77:229-33. [PubMed]    [] Rogers J, Jain NL, Hayes GM. Evaluation of an implementation of PRODIGY phase two. Proc AMIA Symp. 1999;:604-8. [PubMed]    [Proc. AMIA'99]

Goals: "To test the null hypothesis that oral anticoagulation care can be provided at least as well in primary care through a nurse-led clinic, involving near-patient testing and computerized decision support software, compared with routine hospital management based on a variety of clinical outcome measures."

" Fitzmaurice DA, Hobbs FD, Murray ET et al. Oral anticoagulation management in primary care with the use of computerized decision support and near-patient testing: a randomized, controlled trial. Arch Intern Med 2000 Aug 14-28;160(15):2343-8 [PubMed]    [Arch Intern Med]

Study of the effect of using a DSS for oral anticoagulation monitoring in primary care. Measured: clinical outcomes, adverse events and patient acceptability.

" Fitzmaurice DA, Hobbs FD, Murray ET et al. Evaluation of computerized decision support for oral anticoagulation management based in primary care. Br J Gen Pract. 1996 Sep;46(410):533-5. [PubMed]    []

Questionnaire-based study of GPs satisfaction with diagnostic DSS for primary care.

" Dupuits FM, Hasman A. User satisfaction of general practitioners with HIOS+, a medical decision support system. Comput Methods Programs Biomed 1995 Jul;47(2):183-8 [PubMed]    [Comput Methods Programs Biomed]

Study of performance of a DSS for the diagnosis of pneumonia combining a Bayesian network and a natural language understanding system. Comparison of results from the Bayesian network alone with results from the network combined with NL system.

" Aronsky D, Fiszman M, Chapman WW, Haug PJ. Combining decision support methodologies to diagnose pneumonia. Proc AMIA Symp. 2001;:12-6. [PubMed]    []

Development and evaluation of a decision support system prototype to help with the prevention of adverse drug events by detecting drug-drug interactions in drug orders.

" Del Fiol G, Rocha BH, Nohama P. Design, implementation and evaluation of a clinical decision support system to prevent adverse drug events. Stud Health Technol Inform. 2000;77:740-4. [PubMed]    []

Study of knowledge base architecture of a drug ordering decision support system that detects drug-drug interactions in drug orders. Focuses on issues related to knowledge base maintenance and integration with external systems.

" Del Fiol G, Rocha BH, Kuperman GJ, Bates DW, Nohama P. Comparison of two knowledge bases on the detection of drug-drug interactions. Proc AMIA Symp. 2000;:171-5. [PubMed]    []

DIABNET: Causal Probabilistic Network that analyses monitoring data and proposes quantitative changes in insulin therapy and qualitative diet modifications. Evaluation of system performance based on questionnaires and a comparison between the system's and experts' proposals.

" Hernando ME, Gomez EJ, Corcoy R, del Pozo F. Evaluation of DIABNET, a decision support system for therapy planning in gestational diabetes. Comput Methods Programs Biomed. 2000 Jul;62(3):235-48. [PubMed]    [Comput Methods Programs Biomed]

Cluster randomised controlled trial in UK. Conclusion: "The computer based clinical decision support system did not confer any benefit in absolute risk reduction or blood pressure control and requires further development and evaluation before use in clinical care can be recommended."

" Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ. Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ. 2000 Mar 11;320(7236):686-90. [PubMed]    [PubMedCentral]    [BMJ]

Web-based therapeutic dss for Type I Diabetes. Combines Case-Based Reasoning with Rule-Based Reasoning. Initial evaluation based on simulated patients.

" Montani S, Bellazzi R. Integrating case based and rule based reasoning in a decision support system: evaluation with simulated patients. Proc AMIA Symp. 1999;:887-91. [PubMed]    [AMIA'99]

"After a four-month real-life experimentation of the [hypertext] system, a survey was conducted among the users. The observed compliance, significantly higher than the best figures found in the literature, and the clinicians objective and subjective evaluation of the system reinforced the implementation choices adopted in OncoDoc."

" Seroussi B, Bouaud J, Antoine EC. Users' evaluation of OncoDoc, a breast cancer therapeutic guideline delivered at the point of care. Proc AMIA Symp 1999;:384-9 [PubMed]    [AMIA]

Aim: "To assess medical emergency radiology referral practice compared with a set of French guidelines and to measure the efficiency of computer-based guidelines on unnecessary medical imaging." Conclusion: "While the computer provided advice that was tailored to the needs of individual patients, concurrent with care, the effect of our intervention was weak. However, our study identified the few situations that were responsible for the majority of unnecessary radiological requests; we expect that this result could help clinicians and radiologists to develop more specific actions for these situations. "

Carton M, Auvert B, Guerini H et al. Assessment of Radiological Referral Practice and Effect of Computer-based Guidelines on Radiological Requests in Two Emergency Departments. Clin Radiol 2002 Feb;57(2):123-8 [PubMed]    [Clin Radiol]

" The CaDet system, as well as some preliminary results of the clinical experience accumulated in its use, are described. These preliminary results suggest that the approach may be useful in improving cancer risk assessment and screening in primary care setups. "

Fuchs J, Heller I, Topilsky M, Inbar M. CaDet, a computer-based clinical decision support system for early cancer detection. Cancer Detect Prev. 1999;23(1):78-87. [PubMed]