Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines

Journal of the American Medical Informatics Association, Journal Year: 2025, Volume and Issue: unknown

Published: Jan. 15, 2025

The objectives of this study are to synthesize findings from recent research retrieval-augmented generation (RAG) and large language models (LLMs) in biomedicine provide clinical development guidelines improve effectiveness. We conducted a systematic literature review meta-analysis. report was created adherence the Preferred Reporting Items for Systematic Reviews Meta-Analyses 2020 analysis. Searches were performed 3 databases (PubMed, Embase, PsycINFO) using terms related "retrieval augmented generation" "large model," articles published 2023 2024. selected studies that compared baseline LLM performance with RAG performance. developed random-effect meta-analysis model, odds ratio as effect size. Among 335 studies, 20 included review. pooled size 1.35, 95% confidence interval 1.19-1.53, indicating statistically significant (P = .001). reported tasks, LLMs, retrieval sources strategies, well evaluation methods. Building on our review, we Guidelines Unified Implementation Development Enhanced Applications Clinical Settings inform applications RAG. Overall, implementation showed 1.35 increase LLMs. Future should focus (1) system-level enhancement: combination agent, (2) knowledge-level deep integration knowledge into LLM, (3) integration-level integrating systems within electronic health records.

Language: Английский

---

Generative Large Language Models in Electronic Health Records for Patient Care Since 2023: A Systematic Review

medRxiv (Cold Spring Harbor Laboratory), Journal Year: 2024, Volume and Issue: unknown

Published: Aug. 12, 2024

Background: Generative Large language models (LLMs) represent a significant advancement in natural processing, achieving state-of-the-art performance across various tasks. However, their application clinical settings using real electronic health records (EHRs) is still rare and presents numerous challenges. Objective: This study aims to systematically review the use of generative LLMs, effectiveness relevant techniques patient care-related topics involving EHRs, summarize challenges faced, suggest future directions. Methods: A Boolean search for peer-reviewed articles was conducted on May 19th, 2024 PubMed Web Science include research published since 2023, which one month after release ChatGPT. The results were deduplicated. Multiple reviewers, including biomedical informaticians, computer scientists, physician, screened publications eligibility data extraction. Only studies utilizing LLMs analyze EHR included. We summarized prompt engineering, fine-tuning, multimodal data, evaluation matrices. Additionally, we identified current applying as reported by included proposed Results: initial 6,328 unique studies, with 76 screening. Of these, 67 (88.2%) employed zero-shot prompting, five them 100% accuracy specific Nine used advanced prompting strategies; four tested these strategies experimentally, finding that engineering improved performance, noting non-linear relationship between number examples improvement. Eight explored fine-tuning all improvements tasks, but three noted potential degradation certain two utilized LLM-based decision-making enabled accurate disease diagnosis prognosis. 55 different metrics 22 purposes, such correctness, completeness, conciseness. Two investigated LLM bias, detecting no bias other male patients received more appropriate suggestions. Six hallucinations, fabricating names structured thyroid ultrasound reports. Additional not limited impersonal tone consultations, made uncomfortable, difficulty had understanding responses. Conclusion: Our indicates few have computational enhance performance. diverse highlight need standardization. currently cannot replace physicians due

Language: Английский

---

Natural language processing of electronic health records for early detection of cognitive decline: a systematic review

npj Digital Medicine, Journal Year: 2025, Volume and Issue: 8(1)

Published: March 1, 2025

Abstract This systematic review evaluated natural language processing (NLP) approaches for detecting cognitive impairment in electronic health record clinical notes. Following PRISMA guidelines, we analyzed 18 studies ( n = 1,064,530) that employed rule-based algorithms (67%), traditional machine learning (28%), and deep (17%). NLP models demonstrated robust performance identifying decline, with median sensitivity 0.88 (IQR 0.74–0.91) specificity 0.96 0.81–0.99). Deep architectures achieved superior results, area under the receiver operating characteristic curves up to 0.997. Major implementation challenges included incomplete data capture, inconsistent documentation practices, limited external validation. While demonstrates promise, successful translation requires establishing standardized approaches, improving access annotated datasets, developing equitable deployment frameworks.

Language: Английский

---

Harnessing large language model agents for healthy aging

Medicine Plus, Journal Year: 2025, Volume and Issue: unknown, P. 100084 - 100084

Published: April 1, 2025

Language: Английский

---

CD-Tron: Leveraging large clinical language model for early detection of cognitive decline from electronic health records

Journal of Biomedical Informatics, Journal Year: 2025, Volume and Issue: 166, P. 104830 - 104830

Published: May 2, 2025

Language: Английский

---