Files and links (2)
Published (Version of record)CC BY V4.0, Open Access
Published (Version of record)CC BY V4.0, Open
Journal article
Evaluation framework for conversational AI agents in pharmacy education: A scoping review of key characteristics and outcome measures
Research in social and administrative pharmacy, Vol.21(10), pp.729-742
10/2025
PMID: 40393870
Appears in Recent Faculty of Health Publications
Metrics
1 Record Views
Abstract
Background: Innovative and scalable training solutions, such as conversational artificial intelligence agents (CAIAs), are essential to meet the complex demands of pharmacy education and practice.
Objectives: To explore and synthesize the key characteristics and outcome measures of CAIAs in pharmacy education and propose an evaluation framework for their use in pharmacy education.
Methods: A scoping review identified studies published between 2020 and 2025. An evaluation framework was developed to capture key CAIA characteristics and outcome measures using the World Health Organization's digital health framework.
Results: Of 961 studies screened, six met inclusion criteria. Five studies originated from English-speaking countries. CAIAs supported training in communication skills (n = 3), human resource management (n = 2), and HIV care (n = 1). Learner interaction with CAIAs was variable and often poorly described. Common features included scenario-based learning, immediate, real-time, automated feedback, interactive-learning, and modalities. Frequently evaluated outcomes were functionality (n = 5), user experience (n = 4), cost-benefit (n = 3), and user characteristics (n = 3). Educational outcome measures of confidence, knowledge and skills were included in three studies. All studies used text-based interaction; two included audiovisual elements, one combined text and voice, and two relied solely on text. Most (n = 5) involved single-user formats. Three studies were in evaluation stage one (feasibility/usability), two were in stage two (effectiveness) and one in stage three (efficacy). Outcomes demonstrated a low uptake for CAIAs, but indicate increased learner confidence, knowledge and communication skills. Eleven educational features were added to our evaluation framework and three educational outcome categories.
Conclusion: This review contributed to the development of a framework to guide the design and evaluation of CAIAs in pharmacy education. CAIAs have been introduced in pharmacy education but remain in the early stages of adoption. Further research is needed to validate their effectiveness and expand their use in pharmacy as well test the use of the framework in other healthcare disciplines.
Details
- Title
- Evaluation framework for conversational AI agents in pharmacy education: A scoping review of key characteristics and outcome measures
- Creators
- Sabrina Winona Pit - The University of SydneyMohammad Hamiduzzaman - The University of SydneyCarl R. Schneider - The University of SydneyFrances Barraclough - The University of Sydney
- Publication Details
- Research in social and administrative pharmacy, Vol.21(10), pp.729-742
- Publisher
- Elsevier Inc
- Number of pages
- 14
- Grant note
- This work was supported by the University of Sydney DVCR Strategic Research Impact Proof of Concept Fund, NSW, Australia [grant number POC 36-2023].
- Identifiers
- 991013343970502368
- Copyright
- © 2025 The Authors.
- Academic Unit
- Faculty of Health
- Language
- English
- Resource Type
- Journal article