The Center for Knowledge Management (CKM) team’s expertise in evidence provision, search strategy development, and artificial intelligence (AI) tool assessment serves as a resource to the medical center for evaluating emerging tools and their potential role in clinical knowledge workflows. Informed by our in-depth investigations into large language models for clinical question answering, CKM has taken on a key role in educating the medical center on appropriate use of large language models (LLMs) for clinical evidence queries. Our team is well-positioned to aid the organization as new AI tools for clinical evidence emerge, offering our professional judgment on the utility of LLMs for consulting the medical literature and communicating important Vanderbilt Health policies and safeguards that should be applied when incorporating these tools into our workflows.
For an example of CKM’s AI communication contributions, please see the video:
Artificial Intelligence Research
CKM’s competencies in this area have been developed via a series of investigations, initiated in 2023, into the use of large language models (LLMs) in knowledge management and medical librarianship. Our work examines where generative AI can effectively augment knowledge management workflows and where the expertise of information scientists continues to be essential.
Our studies have evaluated AI tools’ performance in answering clinical questions in comparison with information scientists and other LLMs, as well as their ability to provide verifiable supporting references and develop effective biomedical literature search strategies. LLMs assessed include Vanderbilt Health’s internally managed chatbot, frontier models, and retrieval-augmented generation (RAG) models. Throughout these investigations, CKM has engaged in iterative prompt engineering to generate high-quality evidence summaries and implement LLM-assisted assessment methods. Another focus of our work is the characterization of publisher and funding agency policies governing the use of AI in scholarly research and writing.
Overall, CKM’s findings suggest that generative AI can serve as a valuable partner in knowledge management but not a replacement for librarian and information scientist expertise. The highest-quality answers emerged when human expertise was combined with insights from large language models. Our research to date has also found that no single model independently and comprehensively answered all clinical questions, reinforcing the value of consulting multiple tools and maintaining information scientists’ expert oversight.