Structured Prompt Interrogation and Recursive Extraction of Semantics (SPIRES): a method for populating knowledge bases using zero-shot learning

Bioinformatics, Journal Year: 2024, Volume and Issue: 40(3)

Published: Feb. 21, 2024

Abstract Motivation Creating knowledge bases and ontologies is a time consuming task that relies on manual curation. AI/NLP approaches can assist expert curators in populating these bases, but current rely extensive training data, are not able to populate arbitrarily complex nested schemas. Results Here we present Structured Prompt Interrogation Recursive Extraction of Semantics (SPIRES), Knowledge approach the ability Large Language Models (LLMs) perform zero-shot learning general-purpose query answering from flexible prompts return information conforming specified schema. Given detailed, user-defined schema an input text, SPIRES recursively performs prompt interrogation against LLM obtain set responses matching provided uses existing vocabularies provide identifiers for matched elements. We examples applying different domains, including extraction food recipes, multi-species cellular signaling pathways, disease treatments, multi-step drug mechanisms, chemical relationships. Current accuracy comparable mid-range Relation methods, greatly surpasses LLM’s native capability grounding entities with unique identifiers. has advantage easy customization, flexibility, and, crucially, new tasks absence any data. This method supports general strategy leveraging language interpreting capabilities LLMs assemble assisting curation acquisition while supporting validation publicly-available databases external LLM. Availability implementation available as part open source OntoGPT package: https://github.com/monarch-initiative/ontogpt.

Language: Английский

---

Testing and Evaluation of Health Care Applications of Large Language Models

JAMA, Journal Year: 2024, Volume and Issue: unknown

Published: Oct. 15, 2024

Importance Large language models (LLMs) can assist in various health care activities, but current evaluation approaches may not adequately identify the most useful application areas. Objective To summarize existing evaluations of LLMs terms 5 components: (1) data type, (2) task, (3) natural processing (NLP) and understanding (NLU) tasks, (4) dimension evaluation, (5) medical specialty. Data Sources A systematic search PubMed Web Science was performed for studies published between January 1, 2022, February 19, 2024. Study Selection Studies evaluating 1 or more care. Extraction Synthesis Three independent reviewers categorized via keyword searches based on used, NLP NLU dimensions Results Of 519 reviewed, 2024, only 5% used real patient LLM evaluation. The common tasks were assessing knowledge such as answering licensing examination questions (44.5%) making diagnoses (19.5%). Administrative assigning billing codes (0.2%) writing prescriptions less studied. For focused question (84.2%), while summarization (8.9%) conversational dialogue (3.3%) infrequent. Almost all (95.4%) accuracy primary evaluation; fairness, bias, toxicity (15.8%), deployment considerations (4.6%), calibration uncertainty (1.2%) infrequently measured. Finally, specialty area, generic applications (25.6%), internal medicine (16.4%), surgery (11.4%), ophthalmology (6.9%), with nuclear (0.6%), physical (0.4%), genetics being least represented. Conclusions Relevance Existing mostly focus examinations, without consideration data. Dimensions received limited attention. Future should adopt standardized metrics, use clinical data, broaden to include a wider range specialties.

Language: Английский

---

FDA Perspective on the Regulation of Artificial Intelligence in Health Care and Biomedicine

JAMA, Journal Year: 2024, Volume and Issue: unknown

Published: Oct. 15, 2024

Importance Advances in artificial intelligence (AI) must be matched by efforts to better understand and evaluate how AI performs across health care biomedicine as well develop appropriate regulatory frameworks. This Special Communication reviews the history of US Food Drug Administration’s (FDA) regulation AI; presents potential uses medical product development, clinical research, care; concepts that merit consideration system adapts AI’s unique challenges. Observations The FDA has authorized almost 1000 AI-enabled devices received hundreds submissions for drugs used their discovery development. Health needs coordinated all regulated industries, government, with international organizations. Regulators will need advance flexible mechanisms keep up pace change care. Sponsors transparent about regulators proficiency evaluating use premarket A life cycle management approach incorporating recurrent local postmarket performance monitoring should central large language models are needed. Approaches necessary balance entire spectrum ecosystem interests, from firms start-ups. evaluation focus on patient outcomes financial optimization developers, payers, systems. Conclusions Relevance Strong oversight protects long-term success industries focusing technologies improve health. continue play a role ensuring safe, effective, trustworthy tools lives patients clinicians alike. However, involved entities attend rigor this transformative technology merits.

Language: Английский

---

Ensuring useful adoption of generative artificial intelligence in healthcare

Journal of the American Medical Informatics Association, Journal Year: 2024, Volume and Issue: 31(6), P. 1441 - 1444

Published: March 7, 2024

Abstract Objectives This article aims to examine how generative artificial intelligence (AI) can be adopted with the most value in health systems, response Executive Order on AI. Materials and Methods We reviewed technology has historically been deployed healthcare, evaluated recent examples of deployments both traditional AI (GenAI) a lens value. Results Traditional GenAI are different technologies terms their capability modes current deployment, which have implications systems. Discussion when applied framework top-down realize healthcare. short term unclear value, but encouraging more bottom-up adoption potential provide benefit systems patients. Conclusion healthcare for patients adapt culturally grow this new its patterns.

Language: Английский

---

Empowering personalized pharmacogenomics with generative AI solutions

Journal of the American Medical Informatics Association, Journal Year: 2024, Volume and Issue: 31(6), P. 1356 - 1366

Published: March 6, 2024

Abstract Objective This study evaluates an AI assistant developed using OpenAI’s GPT-4 for interpreting pharmacogenomic (PGx) testing results, aiming to improve decision-making and knowledge sharing in clinical genetics enhance patient care with equitable access. Materials Methods The employs retrieval-augmented generation (RAG), which combines retrieval generative techniques, by harnessing a base (KB) that comprises data from the Clinical Pharmacogenetics Implementation Consortium (CPIC). It uses context-aware generate tailored responses user queries this KB, further refined through prompt engineering guardrails. Results Evaluated against specialized PGx question catalog, showed high efficacy addressing queries. Compared ChatGPT 3.5, it demonstrated better performance, especially provider-specific requiring citations. Key areas improvement include enhancing accuracy, relevancy, representative language responses. Discussion integration of RAG significantly enhanced assistant’s utility. RAG’s ability incorporate domain-specific CPIC data, including recent literature, proved beneficial. Challenges persist, such as need genetic/PGx models accuracy relevancy ethical, regulatory, safety concerns. Conclusion underscores AI’s potential transforming healthcare provider support accessibility complex information. While careful implementation large like is necessary, clear they can substantially understanding data. With development, these tools could augment expertise, productivity, delivery equitable, patient-centered services.

Language: Английский

---

Addressing 6 challenges in generative AI for digital health: A scoping review

PLOS Digital Health, Journal Year: 2024, Volume and Issue: 3(5), P. e0000503 - e0000503

Published: May 23, 2024

Generative artificial intelligence (AI) can exhibit biases, compromise data privacy, misinterpret prompts that are adversarial attacks, and produce hallucinations. Despite the potential of generative AI for many applications in digital health, practitioners must understand these tools their limitations. This scoping review pays particular attention to challenges with technologies medical settings surveys solutions. Using PubMed, we identified a total 120 articles published by March 2024, which reference evaluate medicine, from synthesized themes suggestions future work. After first discussing general background on AI, focus collecting presenting 6 key health specific measures be taken mitigate challenges. Overall, bias, hallucination, regulatory compliance were frequently considered, while other concerns around such as overreliance text models, misprompting, jailbreaking, not commonly evaluated current literature.

Language: Английский

---

Optimizing large language models in digestive disease: strategies and challenges to improve clinical outcomes

Liver International, Journal Year: 2024, Volume and Issue: 44(9), P. 2114 - 2124

Published: May 31, 2024

Abstract Large Language Models (LLMs) are transformer‐based neural networks with billions of parameters trained on very large text corpora from diverse sources. LLMs have the potential to improve healthcare due their capability parse complex concepts and generate context‐based responses. The interest in has not spared digestive disease academics, who mainly investigated foundational LLM accuracy, which ranges 25% 90% is influenced by lack standardized rules report methodologies results for LLM‐oriented research. In addition, a critical issue absence universally accepted definition varying binary scalar interpretations, often tied grader expertise without reference clinical guidelines. We address strategies challenges increase accuracy. particular, can be infused domain knowledge using Retrieval Augmented Generation (RAG) or Supervised Fine‐Tuning (SFT) reinforcement learning human feedback (RLHF). RAG faces in‐context window limits accurate information retrieval provided context. SFT, deeper adaptation method, computationally demanding requires specialized knowledge. may patient quality care across field diseases, where physicians engaged screening, treatment surveillance broad range pathologies SFT RLHF could decision‐making outcomes. However, despite potential, safe deployment still needs overcome hurdles suggesting need that integrate advanced model training.

Language: Английский

---

Regulate Artificial Intelligence in Health Care by Prioritizing Patient Outcomes

JAMA, Journal Year: 2024, Volume and Issue: 331(8), P. 639 - 639

Published: Jan. 29, 2024

This Viewpoint argues for a shift in focus by the White House executive order on artificial intelligence from regulatory targets to patient outcomes.

Language: Английский

---

Systematic review: The use of large language models as medical chatbots in digestive diseases

Alimentary Pharmacology & Therapeutics, Journal Year: 2024, Volume and Issue: 60(2), P. 144 - 166

Published: May 27, 2024

Interest in large language models (LLMs), such as OpenAI's ChatGPT, across multiple specialties has grown a source of patient-facing medical advice and provider-facing clinical decision support. The accuracy LLM responses for gastroenterology hepatology-related questions is unknown.

Language: Английский

---

Ética e inteligencia artificial

Revista Clínica Española, Journal Year: 2024, Volume and Issue: 224(3), P. 178 - 186

Published: Feb. 23, 2024

---