Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o

Clinical Rheumatology, Journal Year: 2024, Volume and Issue: 43(11), P. 3507 - 3513

Published: Sept. 28, 2024

Language: Английский

---

Vignette-based comparative analysis of ChatGPT and specialist treatment decisions for rheumatic patients: results of the Rheum2Guide study

Rheumatology International, Journal Year: 2024, Volume and Issue: 44(10), P. 2043 - 2053

Published: Aug. 10, 2024

Abstract Background The complex nature of rheumatic diseases poses considerable challenges for clinicians when developing individualized treatment plans. Large language models (LLMs) such as ChatGPT could enable decision support. Objective To compare plans generated by ChatGPT-3.5 and GPT-4 to those a clinical rheumatology board (RB). Design/methods Fictional patient vignettes were created GPT-3.5, GPT-4, the RB queried provide respective first- second-line with underlying justifications. Four rheumatologists from different centers, blinded origin plans, selected overall preferred concept assessed plans’ safety, EULAR guideline adherence, medical adequacy, quality, justification their completeness well vignette difficulty using 5-point Likert scale. Results 20 fictional covering various varying levels assembled total 160 ratings assessed. In 68.8% (110/160) cases, raters RB’s over (16.3%; 26/160) GPT-3.5 (15.0%; 24/160). GPT-4’s chosen more frequently first-line treatments compared GPT-3.5. No significant safety differences observed between Rheumatologists’ received significantly higher in appropriateness, quality. Ratings did not correlate difficulty. LLM-generated notably longer detailed. Conclusion safe, high-quality diseases, demonstrating promise Future research should investigate detailed standardized prompts impact LLM usage on decisions.

Language: Английский

---

Advancing rheumatology with natural language processing: insights and prospects from a systematic review

Rheumatology Advances in Practice, Journal Year: 2024, Volume and Issue: 8(4)

Published: Jan. 1, 2024

Abstract Objectives Natural language processing (NLP) and large models (LLMs) have emerged as powerful tools in healthcare, offering advanced methods for analysing unstructured clinical texts. This systematic review aims to evaluate the current applications of NLP LLMs rheumatology, focusing on their potential improve disease detection, diagnosis patient management. Methods We screened seven databases. included original research articles that evaluated performance rheumatology. Data extraction risk bias assessment were performed independently by two reviewers, following Preferred Reporting Items Systematic Reviews Meta-Analyses guidelines. The Quality Assessment Tool Observational Cohort Cross-Sectional Studies was used bias. Results Of 1491 initially identified, 35 studies met inclusion criteria. These utilized various data types, including electronic medical records notes, employed like Bidirectional Encoder Representations from Transformers Generative Pre-trained Transformers. High accuracy observed detecting conditions such RA, SpAs gout. use also showed promise managing diseases predicting flares. Conclusion significant enhancing rheumatology improving diagnostic personalizing care. While RA gout are well developed, further is needed extend these technologies rarer more complex conditions. Overcoming limitations through targeted essential fully realizing NLP’s practice.

Language: Английский

---

Common Biases, Difficulties, and Errors in Clinical Reasoning in Veterinary Medical Encounters with a Case Example

Encyclopedia, Journal Year: 2025, Volume and Issue: 5(1), P. 14 - 14

Published: Jan. 20, 2025

Clinical reasoning is an essential competence of veterinary graduands. Unfortunately, clinical and, therefore, the quality provided medical services are prone to bias, difficulties, and errors. The literature on biases, errors in education scarce or focused theoretical rather than practical application. In this review, we address practicality learning teaching learners utilizing a example cow with prolapsed uterus complicated by hypocalcemia hypomagnesemia. Learners should be guided through all stages as much possible under direct supervision. common encounters may differ between development learner, more difficulties occurring earlier (Observer, Reporter, ±Interpreter) but heuristic biases at later (Manager, Educator, ±Interpreter). However, occur any learner stage. Therefore, remediation use strategies that tailored level also specific encounter (e.g., client, patient, context).

Language: Английский

---

Harnessing Large Language Models for Rheumatic Disease Diagnosis: Advancing Hybrid Care and Task Shifting

International Journal of Rheumatic Diseases, Journal Year: 2025, Volume and Issue: 28(2)

Published: Feb. 1, 2025

Rheumatology is facing an expanding care gap, as the number of newly referred patients continues to outpace availability rheumatologists [1], resulting in longer diagnostic delays—often weeks months—that lead irreversible damage, poorer treatment outcomes, and higher societal costs [2]. Patients physicians alike struggle with fluctuating, often nonspecific symptoms (e.g., joint pain), this challenge compounded by limited awareness rheumatic diseases among both general population practitioners. The poor specificity referrals inability traditional triage approaches improve situation widen gap further. Although patient education integral rheumatology care, it remains underutilized due inadequate reimbursement workforce shortages, leaving many feeling poorly informed about their disease. Clinicians also face a significant time burden clinical documentation [3], especially for patients. In response these multifaceted challenges, digital health technologies (DHT) have emerged promising cornerstone enhance diagnosis, information provision, education, alleviating shortages. With rapid proliferation smartphones advanced DHT, delivery models should be reevaluated leverage innovations [4]. Task-shifting increasingly being implemented mitigate wherein tasks are delegated from nurses, medical students, or other healthcare professionals. However, task-shifting scale cost-efficiency DHT could significantly widespread implementation [5]. Currently increasing numbers turn online platforms initial symptom assessment [6], decision support systems (DDSS), that can empower receive preliminary diagnoses within minutes. computer-aided diagnosis has existed decades [7], adoption been hindered usability [8], including time-intensive data entry [9] restricted querying options. These limitations affect static, printed leaves scrolling through lengthy materials rather than engaging open-ended, personalized exploration. To bridge recently made advancements large-language-model-technology (LLM) used unprecedented scalability multimodal processing. Therefore usability, performance, patient-provider relationship improved integrating LLM-driven collaborative triad By continuously processing patient- provider-generated data, LLMs deliver more personalized, accessible, dynamic transform aiming close gap. demonstrated remarkable proficiency reasoning ability process large datasets across various fields rare [10]. passively evaluating vast amount available facilitate accelerated identification at-risk individuals, enabling proactive approach without imposing additional burdens on LLM capabilities highlighted out-performing human experts standardized exams such United States Medical Licensing Examination (USMLE) [11]. Importantly, direct comparison study, ChatGPT's accuracy was found not inferior experienced [12]. Both were given same anamnestic real presenting service. Notably, model exhibited exceptional sensitivity identifying inflammatory (IRDs), correctly listing accurate top three options 86% IRD cases—surpassing 74% success rate rheumatologists. Building this, another publication Venerito Iannone utilized locally fine-tuned LLM, optimized prompt engineering, diagnose fibromyalgia analyzing subtle expressions pain emotion communications [13]. This innovative achieved 87% AUROC 0.86, underscoring potential tackle challenges associated subjective linguistically intricate conditions broadening scope considerations highlighting less obvious conditions. Additionally multiple studies capable extracting dialogues, even when descriptions expressed simple colloquial language [14]. linguistic adaptability allows effectively comprehend narratives identify cues might overlooked assessments. Combined structured nature multi-turn capability shown applications One gaining traction introduction summarizing conversations, generating notes, critical keywords. Research area introduced note formats like K-SOAP domain-specific CliniKnote, which combine simulated doctor-patient dialogues meticulously curated notes. Through fine-tuning, prompting strategies, sophisticated NLP methods, efficiency quality documentation, ultimately reducing clinician workload effective [15]. Further potentials educational applications, exemplified LLMs' address queries accuracy, empathy, comprehensiveness. For instance, ChatGPT-4 tested questions commonly posed systemic lupus erythematosus, its responses only rated empathic but qualitatively better those expert [16]. stem transformer-based architectures underlying [17]. large, diverse knowledge sources—from guidelines authoritative research publications [18]—these maintain extensive contextual understanding dynamically incorporate new information. As result, hold streamline broaden range differential considered. doing so, they may help alleviate workload, patient-centered elevate overall delivery. deployment AI-driven tools faces regulatory hurdles. Determining intended purpose central classification either non-medical devices, distinction directly influences compliance requirements. Under EU AI Act, general-purpose supporting decisions stringent obligations, regarding transparency, risk classification, post-market monitoring. Simultaneously, requirements necessitate robust evaluation, posing validating AI's predictive capabilities. Ensuring alignment frameworks advancing while safeguarding safety regulations. While must addressed, pose inherent risks hallucinations—plausible yet incorrect unverifiable Med-HALT framework, where GPT 3.5 severely hallucinating different complex tasks. field precision paramount, inaccuracies misguide decisions, jeopardizing [19]. transparency explainability become challenging, making grounding crucial research. A technique attention Retrieval-Augmented Generation (RAG). RAG addresses issue first database containing known related user's question input. It retrieves semantically similar text blocks likely answer generate appropriate content. then produces output based solely retrieved information, allowing accurately cite source enables users verify model's against literature explore subject further reviewing referenced documents, [20]. illustrated Figure 1, enhances verifiability outputs relevant, validated base. use academic search engines, effectiveness contexts—particularly diagnosis—remains largely unexplored. Collaborative efforts developers, clinicians, researchers essential optimize utility mitigating risks. exploration into methods developing specialized tailored effectiveness. Integrating presents transformative opportunity reduce delays, education. Despite existing synergistic advancement innovation gaps, professional experience providers, fostering efficient care. Fabian Lechner Johannes Knitza drafted manuscript. Sebastian Kuhn provided suggestions, reviewed edited manuscript several times. We thank Björn Hirte graphical support. declares honoraria Lilly, Novo Nordisk, Siemens Healthineers, Diabetes.de, German Diabetes Association (DDG). founder shareholder MED.digital GmbH. Abbvie, GSK, Vila Health, consulting fees AstraZeneca, BMS, Boehringer Ingelheim, Chugai, GAIA, Galapagos, Janssen, Medac, Novartis, Pfizer, Sobi, Rheumaakademie, UCB, Health Werfen. authors nothing report.

Language: Английский

---

Análisis del rendimiento de ChatGPT-4 en las preguntas de oftalmología del examen MIR

Archivos de la Sociedad Española de Oftalmología, Journal Year: 2025, Volume and Issue: unknown

Published: March 1, 2025

---

RAGing ahead in rheumatology: new language model architectures to tame artificial intelligence

Therapeutic Advances in Musculoskeletal Disease, Journal Year: 2025, Volume and Issue: 17

Published: April 1, 2025

Artificial intelligence (AI) is increasingly transforming rheumatology with research on disease detection, monitoring, and outcome prediction through the analysis of large datasets. The advent generative models language (LLMs) has expanded AI’s capabilities, particularly in natural processing (NLP) tasks such as question-answering medical literature synthesis. While NLP shown promise identifying rheumatic diseases from electronic health records high accuracy, LLMs face significant challenges, including hallucinations a lack domain-specific knowledge, which limit their reliability specialized fields like rheumatology. Retrieval-augmented generation (RAG) emerges solution to these limitations by integrating real-time access external, databases. RAG enhances accuracy relevance AI-generated responses retrieving pertinent information during process, reducing hallucinations, improving trustworthiness AI applications. This architecture allows for precise, context-aware outputs can handle unstructured data effectively. Despite its success other industries, application medicine, specifically rheumatology, remains underexplored. Potential applications include up-to-date clinical guidelines, summarizing complex patient histories data, aiding identification trials, enhancing pharmacovigilance efforts, supporting personalized education. also offers advantages privacy enabling local handling reliance large, general-purpose models. Future directions involve fine-tuned, smaller exploring multimodal that process diverse types. Challenges infrastructure costs, concerns, need evaluation metrics must be addressed. Nevertheless, presents promising opportunity improve offering more accountable, sustainable approach advanced into practice research.

Language: Английский

---

From Web to RheumaLpack: Creating a Linguistic Corpus for Exploitation and Knowledge Discovery in Rheumatology

medRxiv (Cold Spring Harbor Laboratory), Journal Year: 2024, Volume and Issue: unknown

Published: April 27, 2024

A bstract This study introduces RheumaLinguisticpack ( RheumaLpack ), the first specialised linguistic web corpus designed for field of musculoskeletal disorders. By combining mining (i.e., scraping) and natural language processing (NLP) techniques, as well clinical expertise, systematically captures curates structured unstructured data across a spectrum sources including trials registers ClinicalTrials.gov bibliographic databases PubMed), medical agencies (i.e. EMA), social media Reddit), accredited health websites MedlinePlus, Harvard Health Publishing, Cleveland Clinic). Given complexity rheumatic diseases (RMDs) their significant impact on quality life, this resource can be proposed useful tool to train algorithms that could mitigate diseases’ effects. Therefore, aims improve training artificial intelligence (AI) facilitate knowledge discovery in RMDs. The development involved systematic six-step methodology covering identification, characterisation, selection, collection, processing, description. result is non-annotated, monolingual, dynamic corpus, featuring almost 3 million records spanning from 2000 2023. represents pioneering contribution rheumatology research, providing advanced AI NLP applications. highlights value address challenges posed by diseases, illustrating corpus’s potential research treatment paradigms rheumatology. Finally, shown replicated obtain other specialities. code details how build RheumaL inguistic ) pack are also provided dissemination such resource.

Language: Английский

---

Qualitative metrics from the biomedical literature for evaluating large language models in clinical decision-making: a narrative review

BMC Medical Informatics and Decision Making, Journal Year: 2024, Volume and Issue: 24(1)

Published: Nov. 26, 2024

The large language models (LLMs), most notably ChatGPT, released since November 30, 2022, have prompted shifting attention to their use in medicine, particularly for supporting clinical decision-making. However, there is little consensus the medical community on how LLM performance contexts should be evaluated. We performed a literature review of PubMed identify publications between December 1, and April 2024, that discussed assessments LLM-generated diagnoses or treatment plans. selected 108 relevant articles from analysis. frequently used LLMs were GPT-3.5, GPT-4, Bard, LLaMa/Alpaca-based models, Bing Chat. five criteria scoring outputs "accuracy", "completeness", "appropriateness", "insight", "consistency". defining high-quality been consistently by researchers over past 1.5 years. identified high degree variation studies reported findings assessed performance. Standardized reporting qualitative evaluation metrics assess quality can developed facilitate research healthcare.

Language: Английский

---

Exploring the use of ChatGPT/GPT-4 for patient follow-up after oral surgeries

International Journal of Oral and Maxillofacial Surgery, Journal Year: 2024, Volume and Issue: 53(10), P. 867 - 872

Published: April 24, 2024

Language: Английский

---