TY - JOUR
T1 - Evaluation of an architecture for intelligent query and exploration of time-oriented clinical data
AU - Martins, Susana B.
AU - Shahar, Yuval
AU - Goren-Bar, Dina
AU - Galperin, Maya
AU - Kaizer, Herbert
AU - Basso, Lawrence V.
AU - McNaughton, Deborah
AU - Goldstein, Mary K.
N1 - Funding Information:
This research was supported in part by NIH award no. LM-06806. We thank Drs. Aneel Advani, Eitan Lunenfeld, and Laszlo Vaszar for their participation in the studies and for their comments on the KNAVE-II interface, Dr. Meirav Taieb-Mimon for her comments regarding the analysis, and Mr. Denis Klimov for his comments on information visualization. The views expressed are those of the authors and not necessarily those of the Dept. Veterans Affairs or other affiliated organizations.
PY - 2008/5/1
Y1 - 2008/5/1
N2 - Objective: Evaluate KNAVE-II, a knowledge-based framework for visualization, interpretation, and exploration of longitudinal clinical data, clinical concepts and patterns. KNAVE-II mediates queries to a distributed temporal-abstraction architecture (IDAN), which uses a knowledge-based problem-solving method specializing in on-the-fly computation of clinical queries. Methods: A two-phase, balanced cross-over study to compare efficiency and satisfaction of a group of clinicians when answering queries of variable complexity about time-oriented clinical data, typical for oncology protocols, using KNAVE-II, versus standard methods: both paper charts and a popular electronic spreadsheet (ESS) in Phase I; an ESS in Phase II. The measurements included the time required to answer and the correctness of answer for each query and each complexity category, and for all queries, assessed versus a predetermined gold standard set by a domain expert. User satisfaction was assessed by the Standard Usability Score (SUS) tool-specific questionnaire and by a "Usability of Tool Comparison" comparative questionnaire developed for this study. Results: In both evaluations, subjects answered higher-complexity queries significantly faster using KNAVE-II than when using paper charts or an ESS up to a mean of 255 s difference per query versus the ESS for hard queries (p = 0.0003) in the second evaluation. Average correctness scores when using KNAVE-II versus paper charts, in the first phase, and the ESS, in the second phase, were significantly higher over all queries. In the second evaluation, 91.6% (110/120) of all of the questions asked within queries of all levels produced correct answers using KNAVE-II, opposed to only 57.5% (69/120) using the ESS (p < 0.0001). User satisfaction with KNAVE-II was significantly superior compared to using either a paper chart or the ESS (p = 0.006). Clinicians ranked KNAVE-II superior to both paper and the ESS. Conclusions: An evaluation of the functionality and usability of KNAVE-II and its supporting knowledge-based temporal-mediation architecture has produced highly encouraging results regarding saving of physician time, enhancement of accuracy of clinical assessment, and user satisfaction.
AB - Objective: Evaluate KNAVE-II, a knowledge-based framework for visualization, interpretation, and exploration of longitudinal clinical data, clinical concepts and patterns. KNAVE-II mediates queries to a distributed temporal-abstraction architecture (IDAN), which uses a knowledge-based problem-solving method specializing in on-the-fly computation of clinical queries. Methods: A two-phase, balanced cross-over study to compare efficiency and satisfaction of a group of clinicians when answering queries of variable complexity about time-oriented clinical data, typical for oncology protocols, using KNAVE-II, versus standard methods: both paper charts and a popular electronic spreadsheet (ESS) in Phase I; an ESS in Phase II. The measurements included the time required to answer and the correctness of answer for each query and each complexity category, and for all queries, assessed versus a predetermined gold standard set by a domain expert. User satisfaction was assessed by the Standard Usability Score (SUS) tool-specific questionnaire and by a "Usability of Tool Comparison" comparative questionnaire developed for this study. Results: In both evaluations, subjects answered higher-complexity queries significantly faster using KNAVE-II than when using paper charts or an ESS up to a mean of 255 s difference per query versus the ESS for hard queries (p = 0.0003) in the second evaluation. Average correctness scores when using KNAVE-II versus paper charts, in the first phase, and the ESS, in the second phase, were significantly higher over all queries. In the second evaluation, 91.6% (110/120) of all of the questions asked within queries of all levels produced correct answers using KNAVE-II, opposed to only 57.5% (69/120) using the ESS (p < 0.0001). User satisfaction with KNAVE-II was significantly superior compared to using either a paper chart or the ESS (p = 0.006). Clinicians ranked KNAVE-II superior to both paper and the ESS. Conclusions: An evaluation of the functionality and usability of KNAVE-II and its supporting knowledge-based temporal-mediation architecture has produced highly encouraging results regarding saving of physician time, enhancement of accuracy of clinical assessment, and user satisfaction.
KW - Clinical decision-support systems
KW - Human-computer interface
KW - Information-visualization
KW - Intelligent user interfaces
KW - Knowledge-based systems
KW - Medical informatics
KW - Temporal reasoning
UR - http://www.scopus.com/inward/record.url?scp=42749088985&partnerID=8YFLogxK
U2 - 10.1016/j.artmed.2008.03.006
DO - 10.1016/j.artmed.2008.03.006
M3 - Article
C2 - 18442899
AN - SCOPUS:42749088985
SN - 0933-3657
VL - 43
SP - 17
EP - 34
JO - Artificial Intelligence in Medicine
JF - Artificial Intelligence in Medicine
IS - 1
ER -
