AI versus human-generated multiple-choice questions for medical education: a cohort study in a high-stakes examination

BMC Medical Education, Journal Year: 2025, Volume and Issue: 25(1)

Published: Feb. 8, 2025

Language: Английский

---

Large Language Models as Tools to Generate Radiology Board-Style Multiple-Choice Questions

Academic Radiology, Journal Year: 2024, Volume and Issue: 31(9), P. 3872 - 3878

Published: July 15, 2024

Rationale and ObjectivesTo determine the potential of large language models (LLMs) to be used as tools by radiology educators create board-style multiple choice questions (MCQs), answers, rationales.MethodsTwo LLMs (Llama 2 GPT-4) were develop 104 MCQs based on American Board Radiology exam blueprint. Two board-certified radiologists assessed each MCQ using a 10-point Likert scale across five criteria—clarity, relevance, suitability for board level difficulty, quality distractors, adequacy rationale. For comparison, from prior College (ACR) Diagnostic In-Training (DXIT) exams also these criteria, with blinded question source.ResultsMean scores (±standard deviation) clarity, suitability, rationale 8.7 (±1.4), 9.2 (±1.3), 9.0 (±1.2), 8.4 (±1.9), 7.2 (±2.2), respectively, Llama 2; 9.9 (±0.4), (±0.5), 9.8 (±0.3), GPT-4; (±0.2), (±0.6), ACR DXIT items (p < 0.001 vs. all criteria; no statistically significant difference GPT-4 DXIT). The accuracy model-generated answers was 69% 100% GPT-4.ConclusionA state-of-the art LLM such may rationales enhance preparation materials expand banks, allow further use teaching learning tools. To rationales. source. Mean GPT-4. A

Language: Английский

---

Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis (Preprint)

Journal of Medical Internet Research, Journal Year: 2024, Volume and Issue: 26, P. e66114 - e66114

Published: Dec. 10, 2024

Background Large language models (LLMs) are increasingly integrated into medical education, with transformative potential for learning and assessment. However, their performance across diverse exams globally has remained underexplored. Objective This study aims to introduce MedExamLLM, a comprehensive platform designed systematically evaluate the of LLMs on worldwide. Specifically, seeks (1) compile curate data worldwide exams; (2) analyze trends disparities in LLM capabilities geographic regions, languages, contexts; (3) provide resource researchers, educators, developers explore advance integration artificial intelligence education. Methods A systematic search was conducted April 25, 2024, PubMed database identify relevant publications. Inclusion criteria encompassed peer-reviewed, English-language, original research articles that evaluated at least one exams. Exclusion included review articles, non-English publications, preprints, studies without performance. The screening process candidate publications independently by 2 researchers ensure accuracy reliability. Data, including exam information, model performance, availability, references, were manually curated, standardized, organized. These curated MedExamLLM platform, enabling its functionality visualize geographic, linguistic, characteristics. web developed focus accessibility, interactivity, scalability support continuous updates user engagement. Results total 193 final analysis. comprised information 16 198 28 countries 15 languages from year 2009 2023. United States accounted highest number related English being dominant used these Generative Pretrained Transformer (GPT) series models, especially GPT-4, demonstrated superior achieving pass rates significantly higher than other LLMs. analysis revealed significant variability different linguistic contexts. Conclusions is an open-source, freely accessible, publicly available online providing evaluation evidence knowledge about around world. serves as valuable fields clinical medicine intelligence. By synthesizing capabilities, provides insights Limitations include biases source exclusion literature. Future should address gaps methods enhance

Language: Английский

---

Insights Gained from Using AI to Produce Cases for Problem-Based Learning

Published: Feb. 27, 2025

Language: Английский

---

ChatGPT usage and attitudes are driven by perceptions of usefulness, ease of use, risks, and psycho-social impact: a study among university students in the UAE

Frontiers in Education, Journal Year: 2024, Volume and Issue: 9

Published: Aug. 7, 2024

Background The use of ChatGPT among university students has gained a recent popularity. current study aimed to assess the factors driving attitude and usage as an example generative artificial intelligence (genAI) in United Arab Emirates (UAE). Methods This cross-sectional was based on previously validated Technology Acceptance Model (TAM)-based survey instrument termed TAME-ChatGPT. self-administered e-survey distributed by emails for enrolled UAE universities during September–December 2023 using convenience-based approach. Assessment demographic academic variables, TAME-ChatGPT constructs’ roles conducted univariate followed multivariate analyses. Results final sample comprised 608 participants, 91.0% whom heard while 85.4% used before study. Univariate analysis indicated that positive associated with three constructs namely, lower perceived risks, anxiety, higher scores technology/social influence. For usage, being male, nationality, point grade average (GPA) well four usefulness, risks use, behavior/cognitive construct ease-of-use construct. In analysis, only explained variance towards (80.8%) its (76.9%). Conclusion findings is commonplace UAE. determinants included cognitive behavioral factors, ease determined These should be considered understanding motivators successful adoption genAI including education.

Language: Английский

---

Beginner-Level Tips for Medical Educators: Guidance on Selection, Prompt Engineering, and the Use of Artificial Intelligence Chatbots

Medical Science Educator, Journal Year: 2024, Volume and Issue: 34(6), P. 1571 - 1576

Published: Aug. 17, 2024

Language: Английский

---

Automatic Generation of Medical Case-Based Multiple-Choice Questions (MCQs): A Review of Methodologies, Applications, Evaluation, and Future Directions

Big Data and Cognitive Computing, Journal Year: 2024, Volume and Issue: 8(10), P. 139 - 139

Published: Oct. 17, 2024

This paper offers an in-depth review of the latest advancements in automatic generation medical case-based multiple-choice questions (MCQs). The creation educational materials, particularly MCQs, is pivotal enhancing teaching effectiveness and student engagement education. In this review, we explore various algorithms techniques that have been developed for generating MCQs from case studies. Recent innovations natural language processing (NLP) machine learning (ML) garnered considerable attention. Our analysis evaluates categorizes leading approaches, highlighting their capabilities practical applications. Additionally, synthesizes existing evidence, detailing strengths, limitations, gaps current practices. By contributing to broader conversation on how technology can support education, not only assesses present state but also suggests future directions improvement. We advocate development more advanced adaptable mechanisms enhance thereby supporting effective experiences

Language: Английский

---

Challenging the curve: can ChatGPT-generated MCQs reduce grade inflation in pharmacy education

Frontiers in Pharmacology, Journal Year: 2025, Volume and Issue: 16

Published: Jan. 29, 2025

Introduction Grade inflation in higher education poses challenges to maintaining academic standards, particularly pharmacy education, where assessing student competency is crucial. This study investigates the impact of AI-generated multiple-choice questions (MCQs) on exam difficulty and reliability a management course at Saudi university. Methods A quasi-experimental design compared 2024 midterm exam, featuring ChatGPT-generated MCQs, with 2023 that utilized human-generated questions. Both exams covered identical topics. Exam was assessed using Kuder-Richardson Formula 20 (KR-20), while discrimination indices were analyzed. Statistical tests, including t-tests chi-square conducted compare performance metrics. Results The demonstrated (KR-20 = 0.83) 0.78). included greater proportion moderate (30%) one difficult question (3.3%), whereas had 93.3% easy mean score significantly lower (17.75 vs. 21.53, p &lt; 0.001), index improved (0.35 0.25, 0.007), indicating enhanced differentiation between students. Discussion findings suggest MCQs contribute rigor potential reduction grade inflation. However, careful review content remains essential ensure alignment objectives accuracy. Conclusion AI tools like ChatGPT offer promising opportunities enhance assessment integrity support fairer evaluations education.

Language: Английский

---

Chat GPT 4o VS Residents: French Language Evaluation in Ophthalmology

Deleted Journal, Journal Year: 2025, Volume and Issue: 2(1), P. 100104 - 100104

Published: Jan. 31, 2025

Language: Английский

---

A more precise interpretation of the potential value of artificial intelligence tools in medical education is needed

Postgraduate Medical Journal, Journal Year: 2025, Volume and Issue: unknown

Published: Jan. 29, 2025

Journal Article A more precise interpretation of the potential value artificial intelligence tools in medical education is needed Get access Hongnan Ye Department Medical Education and Research, Beijing Alumni Association China University, No. 9 Wenhuiyuan North Road, Haidian District, 100000, Corresponding author. Room 106, Building 3, Jindian Garden, China. E-mail: [email protected] https://orcid.org/0009-0004-9013-2919 Search for other works by this author on: Oxford Academic Google Scholar Postgraduate Journal, qgaf024, https://doi.org/10.1093/postmj/qgaf024 Published: 10 February 2025 history Received: 14 January Accepted: 28

Language: Английский

---