Large Language Models’ Accuracy in Emulating Human Experts’ Evaluation of Public Sentiments about Heated Tobacco Products on Social Media: Evaluation Study

Journal of Medical Internet Research, Год журнала: 2025, Номер 27, С. e63631 - e63631

Опубликована: Март 4, 2025

Sentiment analysis of alternative tobacco products discussed on social media is crucial in control research. Large language models (LLMs) are artificial intelligence that were trained extensive text data to emulate the linguistic patterns humans. LLMs may hold potential streamline time-consuming and labor-intensive process human sentiment analysis. This study aimed examine accuracy replicating evaluation messages relevant heated (HTPs). GPT-3.5 GPT-4 Turbo (OpenAI) used classify 500 Facebook (Meta Platforms) Twitter (subsequently rebranded X) messages. Each set consisted 200 human-labeled anti-HTPs, pro-HTPs, 100 neutral The evaluated each message up 20 times generate multiple response instances reporting its classification decisions. majority labels from these responses assigned as a model's decision for message. models' decisions then compared with those evaluators. accurately replicated 61.2% 57% demonstrated higher accuracies overall, 81.7% 77% Turbo's 3 reached 99% achieved instances. was anti- pro-HTP Most misclassifications occurred when or incorrectly classified irrelevant by model, whereas showed improvements across all categories reduced misclassifications, especially categorized irrelevant. can be analyze about HTPs. Results suggest reach approximately 80% results experts, even small number labeling generated model. A risk using misrepresentation overall due differences categories. Although this issue could newer future efforts should explore mechanisms underlying discrepancies how address them systematically.

Язык: Английский

---

Unlocking language barriers: Assessing pre-trained large language models across multilingual tasks and unveiling the black box with Explainable Artificial Intelligence

Engineering Applications of Artificial Intelligence, Год журнала: 2025, Номер 149, С. 110136 - 110136

Опубликована: Март 12, 2025

Язык: Английский

---

Identifying healthcare needs with patient experience reviews using ChatGPT

PLoS ONE, Год журнала: 2025, Номер 20(3), С. e0313442 - e0313442

Опубликована: Март 18, 2025

Background Valuable findings can be obtained through data mining in patients’ online reviews. Also identifying healthcare needs from the patient’s perspective more accurately improve quality of care and experience visit. Thereby avoiding unnecessary waste health resources. The large language model (LLM) a promising tool due to research that demonstrates its outstanding performance potential directions such as mining, management, more. Objective We aim propose methodology address this problem, specifically, recent breakthrough LLM leveraged for effectively understanding patient Methods used 504,198 reviews collected medical platform, haodf.com. create Aspect Based Sentiment Analysis (ABSA) templates, which categorized into three categories, reflecting areas concern patients. With introduction thought chains, we embedded ABSA templates prompts ChatGPT, was then identify needs. Results Our method has weighted total precision 0.944, compared direct narrative tasks ChatGPT-4o, have 0.890. Weighted recall F1 scores also reached 0.884 0.912 respectively, surpassing 0.802 0.843 “direct narratives ChatGPT.” Finally, accuracy sampling methods 91.8%, 91.7%, 91.2%, with an average over 91.5%. Conclusions Combining ChatGPT achieve satisfactory results analyzing As our work applies other LLMs, shed light on demands patients consumers novel models, contribute agenda enhancing better resource allocations effectively.

Язык: Английский

---

Communicating Scientific Norms in the Hybrid Media Environment: A Mixed-Method Analysis of Social Media Engagement With Watchdog Science Journalism

Journalism & Mass Communication Quarterly, Год журнала: 2025, Номер unknown

Опубликована: Апрель 26, 2025

Hybrid media systems have reconfigured online journalism and mass communication such that people can engage more easily in multi-directional discourse about the norms of science. We investigate this reconfiguration with a mixed-methods study X page “Retraction Watch,” which produces hybrid “watchdog science journalism” on violations scientific norms. Results show Retraction Watch’s is not necessarily an inclusive forum for open debate also find Watch prioritizes aspects may resonate its audience. This has implications how communicators journalists approach (hybrid)

Язык: Английский

---

Using Large Language Models for sentiment analysis of health-related social media data: empirical evaluation and practical tips

medRxiv (Cold Spring Harbor Laboratory), Год журнала: 2024, Номер unknown

Опубликована: Март 20, 2024

Abstract Health-related social media data generated by patients and the public provide valuable insights into patient experiences opinions toward health issues such as vaccination medical treatments. Using Natural Language Processing (NLP) methods to analyze data, however, often requires high-quality annotations that are difficult obtain. The recent emergence of Large Models (LLMs) Generative Pre-trained Transformers (GPTs) has shown promising performance on a variety NLP tasks in domain with little no annotated data. However, their potential analyzing health-related remains underexplored. In this paper, we report empirical evaluations LLMs (GPT-3.5-Turbo, FLAN-T5, BERT-based models) common task data: sentiment analysis for identifying issues. We explored how different prompting fine-tuning strategies affect datasets across diverse topics, including Healthcare Reform, vaccination, mask wearing, healthcare service quality. found outperformed VADER, widely used off-the-shelf tool, but far from being able produce accurate labels. can be improved data-specific prompts information about context, task, targets. highest performing models were fine-tuned aggregated provided practical tips researchers use optimal outcomes. also discuss future work needed continue improve minimal annotations.

Язык: Английский

---

Pediatric Cancer Communication on Twitter: Natural Language Processing and Qualitative Content Analysis

JMIR Cancer, Год журнала: 2024, Номер 10, С. e52061 - e52061

Опубликована: Май 7, 2024

Background During the COVID-19 pandemic, Twitter (recently rebranded as “X”) was most widely used social media platform with over 2 million cancer-related tweets. The increasing use of among patients and family members, providers, organizations has allowed for novel methods studying cancer communication. Objective This study aimed to examine pediatric cancer–related tweets capture experiences survivors cancer, their caregivers, medical other stakeholders. We assessed public sentiment content related a time period representative pandemic. Methods All English-language posted from December 11, 2019, May 7, 2022, globally, were obtained using application programming interface. Sentiment analyses computed based on Bing, AFINN, NRC lexicons. conducted supplemental nonlexicon-based analysis ChatGPT (version 3.0) validate our findings random subset 150 qualitative manually code 800 Results A total 161,135 unique identified. showed that there more positive words than negative words. Via Bing lexicon, common support, love, amazing, heaven, happy, grief, risk, hard, abuse, miss. categorized under types positive, trust, joy. Overall consistent across lexicons confirmed analysis. Percent agreement between raters coding 91%, top 10 codes awareness, personal experiences, research, caregiver patient policy law, treatment, end life, pharmaceuticals drugs, survivorship. Qualitative users commonly promote awareness share perspective or caregivers. frequently health knowledge dissemination research federal policies support treatment affordable care. Conclusions may serve an effective means researchers communication around globe. Despite mental crisis during overall sentiments positive. Content focused information, raising cancer.

Язык: Английский

---

Comparing ChatGPT's correction and feedback comments with that of educators in the context of primary students' short essays written in English and Greek

Education and Information Technologies, Год журнала: 2024, Номер unknown

Опубликована: Июль 27, 2024

Язык: Английский

---

Large language models can enable inductive thematic analysis of a social media corpus in a single prompt: Human validation study (Preprint)

JMIR Infodemiology, Год журнала: 2024, Номер 4, С. e59641 - e59641

Опубликована: Июль 1, 2024

Manually analyzing public health-related content from social media provides valuable insights into the beliefs, attitudes, and behaviors of individuals, shedding light on trends patterns that can inform understanding, policy decisions, targeted interventions, communication strategies. Unfortunately, time effort needed well-trained human subject matter experts makes extensive manual listening unfeasible. Generative large language models (LLMs) potentially summarize interpret amounts text, but it is unclear to what extent LLMs glean subtle meanings in sets posts reasonably report themes.

Язык: Английский

---

An Application of Advanced Sentiment Analysis on X (Twitter) Utilising Large Language Models for the Precise Prediction of Election Outcomes

International Research Journal of Modernization in Engineering Technology and Science, Год журнала: 2024, Номер unknown

Опубликована: Апрель 23, 2024

In this research project, we harness the capability of Large Language Models especially GPT to perform sentiment analysis on Twitter, aiming forecast election results.The widespread adoption digital technologies has precipitated a notable escalation in creation user-generated content, thereby catalysing transformative shift communication dynamics across multiple platforms.Social media platforms, particular, have emerged as reservoirs invaluable behavioural data, offering profound insights spectrum disciplines including politics, e-commerce, education and medical.Engaging political tweet mining for predictive analytics presents formidable hurdles, notably encompassing precise determination accuracy identification propagandistic narratives.We propose LLMs, particularly GPT, solution due their adeptness natural language processing (NLP) tasks.LLMs' extensive training enables them understand intricate linguistic nuances, context.Their generative capabilities ensure coherent text production, crucial analysis.Leveraging these advantages, aim predict outcomes Indian Lok Sabha Elections 2024 through using models.This addresses pressing need robust methodologies predicting results by tapping into power LLMs NLP techniques.

Язык: Английский

---

Large Language Models Can Enable Inductive Thematic Analysis of a Social Media Corpus in a Single Prompt: Human Validation Study (Preprint)

Опубликована: Апрель 24, 2024

Manually analyzing public health–related content from social media provides valuable insights into the beliefs, attitudes, and behaviors of individuals, shedding light on trends patterns that can inform understanding, policy decisions, targeted interventions, communication strategies. Unfortunately, time effort needed well-trained human subject matter experts makes extensive manual listening unfeasible. Generative large language models (LLMs) potentially summarize interpret amounts text, but it is unclear to what extent LLMs glean subtle health-related meanings in sets posts reasonably report themes. We aimed assess feasibility using for topic model selection or inductive thematic analysis contents by attempting answer following question: Can conduct as effectively humans did a prior study, at least reasonably, judged experts? asked same research question used set both LLM relevant topics themes was conducted manually published study about vaccine rhetoric. results background this experiment comparing analyses with 3 LLMs: GPT4-32K, Claude-instant-100K, Claude-2-100K. also assessed if multiple had equivalent ability consistency repeated each LLM. The generally gave high rankings chosen previously most relevant. reject null hypothesis (<i>P</i>&lt;.001, overall comparison) conclude these are more likely include human-rated top 5 areas their than would occur chance. Regarding theme identification, identified several similar those humans, very low hallucination rates. Variability occurred between test runs an individual Despite not consistently matching human-generated themes, found generated were still reasonable efficiently process media–based data sets. extract such deem reasonable. However, we unable show tested replicate depth extracting data. There vast potential, once better validated, automated LLM-based real-time common rare health conditions, informing understanding public’s interests concerns determining ideas address them.

Язык: Английский

---