A Systematic Review on Multimodal Emotion Recognition: Building Blocks, Current State, Applications, and Challenges

IEEE Access, Journal Year: 2024, Volume and Issue: 12, P. 103976 - 104019

Published: Jan. 1, 2024

Emotion recognition involves accurately interpreting human emotions from various sources and modalities, including questionnaires, verbal, physiological signals. With its broad applications in affective computing, computational creativity, human-robot interactions, market research, the field has seen a surge interest recent years. This paper presents systematic review of multimodal emotion (MER) techniques developed 2014 to 2024, encompassing signals, facial, body gesture, speech as well emerging methods like sketches recognition. The explores models, distinguishing between emotions, feelings, sentiments, moods, along with emotional expression, categorized both artistic non-verbal ways. It also discusses background automated systems introduces seven criteria for evaluating modalities alongside current state analysis MER, drawn human-centric perspective this field. By selecting PRISMA guidelines carefully analyzing 45 selected articles, provides comprehensive perspectives into existing studies, datasets, technical approaches, identified gaps, future directions MER. highlights challenges

Language: Английский

---

Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review

Applied Sciences, Journal Year: 2024, Volume and Issue: 14(17), P. 8071 - 8071

Published: Sept. 9, 2024

The dynamic expressions of emotion convey both the emotional and functional states an individual’s interactions. Recognizing helps us understand human feelings thoughts. Systems frameworks designed to recognize automatically can use various affective signals as inputs, such visual, vocal physiological signals. However, recognition via a single modality be affected by sources noise that are specific fact different may indistinguishable. This review examines current state multimodal methods integrate or modalities for practical computing. Recent empirical evidence on deep learning used fine-grained is reviewed, with discussions robustness issues methods. elaborates profound challenges solutions required high-quality system, emphasizing benefits expression analysis, which aids in detecting subtle micro-expressions, importance fusion improving accuracy. literature was comprehensively searched databases records covering topic computing, followed rigorous screening selection relevant studies. results show effectiveness limited availability training data, insufficient context awareness, posed real-world cases noisy missing modalities. findings suggest requires better representation input refined feature extraction, optimized aggregation within framework, along incorporating state-of-the-art recognizing expressions.

Language: Английский

---

The Analysis of Emotion-Aware Personalized Recommendations via Multimodal Data Fusion in the Field of Art

Journal of Organizational and End User Computing, Journal Year: 2025, Volume and Issue: 37(1), P. 1 - 29

Published: Jan. 23, 2025

This paper proposes an emotion-aware personalized recommendation system (EPR-IoT) based on IoT data and multimodal emotion fusion, aiming to address the limitations of traditional systems in capturing users' emotional states artistic product consumption real time. With proliferation smart devices, physiological signals such as heart rate skin conductance—which are strongly correlated with states—provide new opportunities for recognition. For example, increase is typically associated emotions like anxiety, anger, or fear, while a decrease linked relaxation joy. Similarly, conductance rises arousal, particularly during stress fear. These signals, combined text, speech, video art products, fused construct emotion-driven model capable dynamically adjusting recommended content.

Language: Английский

---

Balancing Concerns—AI and Moral Agency in Medicine

JAMA Cardiology, Journal Year: 2025, Volume and Issue: unknown

Published: Feb. 26, 2025

Our website uses cookies to enhance your experience. By continuing use our site, or clicking "Continue," you are agreeing Cookie Policy | Continue JAMA Cardiology HomeNew OnlineCurrent IssueFor Authors Podcast JAMA+ AI Journals Network Open Dermatology Health Forum Internal Medicine Neurology Oncology Ophthalmology Otolaryngology–Head & Neck Surgery Pediatrics Psychiatry Archives of (1919-1959) JN Learning / CMESubscribeJobsInstitutions LibrariansReprints Permissions Terms Use Privacy Accessibility Statement 2025 American Medical Association. All Rights Reserved Search Archive Input Term Sign In Individual inCreate an Account Access through institution Purchase Options: Buy this article Rent Subscribe the journal

Language: Английский

---

Trust and Acceptance Challenges in the Adoption of AI Applications in Health Care: Quantitative Survey Analysis

Journal of Medical Internet Research, Journal Year: 2025, Volume and Issue: 27, P. e65567 - e65567

Published: March 21, 2025

Background Artificial intelligence (AI) has potential to transform health care, but its successful implementation depends on the trust and acceptance of consumers patients. Understanding factors that influence attitudes toward AI is crucial for effective adoption. Despite AI’s growing integration into consumer patient remains a critical challenge. Research largely focused applications or attitudes, lacking comprehensive analysis how factors, such as demographics, personality traits, technology knowledge, affect interact across different care contexts. Objective We aimed investigate people’s in use cases determine context perceived risk individuals’ propensity accept specific scenarios. Methods collected analyzed web-based survey data from 1100 Finnish participants, presenting them with 8 care: 5 (62%) noninvasive (eg, activity monitoring mental support) 3 (38%) physical interventions AI-controlled robotic surgery). Respondents evaluated intention use, trust, willingness trade off personal these cases. Gradient boosted tree regression models were trained predict responses based 33 demographic-, personality-, technology-related variables. To interpret results our predictive models, we used Shapley additive explanations method, game theory–based approach explaining output machine learning models. It quantifies contribution each feature individual predictions, allowing us relative importance various their interactions shaping participants’ care. Results Consumer technology, traits primary drivers Use ranked by acceptance, monitors being most preferred. However, case had less impact general than expected. Nonlinear dependencies observed, including an inverted U-shaped pattern positivity self-reported knowledge. Certain more disorganized careless, associated positive Women seemed cautious about men. Conclusions The findings highlight complex interplay influencing are driven rather service providers should consider demographic when designing implementing systems study demonstrates using decision-making tools interacting clients applications.

Language: Английский

---

Quantum neural networks for multimodal sentiment, emotion, and sarcasm analysis

Alexandria Engineering Journal, Journal Year: 2025, Volume and Issue: 124, P. 170 - 187

Published: April 2, 2025

Language: Английский

---

Multimodal learning-based speech enhancement and separation, recent innovations, new horizons, challenges and real-world applications

Computers in Biology and Medicine, Journal Year: 2025, Volume and Issue: 190, P. 110082 - 110082

Published: April 3, 2025

Language: Английский

---

Ethical Considerations in Emotion Recognition Research

Deleted Journal, Journal Year: 2025, Volume and Issue: 7(2), P. 43 - 43

Published: May 29, 2025

The deployment of emotion-recognition technologies expands across healthcare education and gaming sectors to improve human–computer interaction. These systems examine facial expressions together with vocal tone physiological signals, which include pupil size electroencephalogram (EEG), detect emotional states deliver customized responses. technology provides benefits through accessibility, responsiveness, adaptability but generates multiple complex ethical issues. combination profiling biased algorithmic interpretations culturally diverse affective data collection without meaningful consent presents major concerns. increased presence these in classrooms, therapy sessions, personal devices makes the potential for misuse or misinterpretation more critical. paper integrates findings from literature review initial studies create a conceptual framework that prioritizes dignity, accountability, user agency addresses risks includes safeguards participants’ well-being. introduces structural minimization, adaptive mechanisms, transparent model logic as complete solution than privacy fairness approaches. authors present functional recommendations guide developers ethically robust match principles regulatory requirements. development real-time feedback loops awareness should be combined clear disclosures about use participatory design practices. successful oversight requires interdisciplinary work between researchers, policymakers, designers, ethicists. practical developing computing advance field while maintaining responsible governance academic research industry settings. hold particular importance high-stakes applications including healthcare, education, workplace monitoring technology.

Language: Английский

---

Pain Level Classification Using Eye-Tracking Metrics and Machine Learning Models

Computers, Journal Year: 2025, Volume and Issue: 14(6), P. 212 - 212

Published: May 30, 2025

Pain estimation is a critical aspect of healthcare, particularly for patients who are unable to communicate discomfort effectively. The traditional methods, such as self-reporting or observational scales, subjective and prone bias. This study proposes novel system non-invasive pain using eye-tracking technology advanced machine learning models. methodology begins with preprocessing steps, including resizing, normalization, data augmentation, prepare high-quality input face images. DeepLabV3+ employed the precise segmentation eye regions, achieving 95% accuracy. Feature extraction performed VGG16, capturing key metrics pupil size, blink rate, saccade velocity. Multiple models, Random Forest, SVM, MLP, XGBoost, NGBoost, trained on extracted features. XGBoost achieves highest classification accuracy 99.5%, demonstrating its robustness level scale from 0 5. feature analysis SHAP values reveals that size rate contribute most predictions, contribution scores 0.42 0.35, respectively. loss curves confirm rapid convergence during training, ensuring reliable segmentation. work highlights transformative potential combining estimation, significant applications in human–computer interaction, assistive technologies.

Language: Английский

---

Benchmarking deep Facial Expression Recognition: An extensive protocol with balanced dataset in the wild

Engineering Applications of Artificial Intelligence, Journal Year: 2024, Volume and Issue: 136, P. 108983 - 108983

Published: July 18, 2024

Facial expression recognition (FER) is crucial in enhancing human-computer interaction. While current FER methods, leveraging various open-source deep learning models and training techniques, have shown promising accuracy generalizability, their efficacy often diminishes real-world scenarios that are not extensively studied. Addressing this gap, we introduce a novel in-the-wild balanced testing facial dataset designed for cross-domain validation, called BTFER. We rigorously evaluated widely utilized networks self-designed architectures, adhering to standardized protocol. Additionally, explored different configurations, including input resolutions, class balance management, pre-trained strategies, ascertain impact on performance. Through comprehensive across three major datasets our in-depth cross-validation, ranked these network architectures formulated series of practical guidelines implementing learning-based solutions real-life applications. This paper also delves into the ethical considerations, privacy concerns, regulatory aspects relevant deployment technologies sectors such as marketing, education, entertainment, healthcare, aiming foster responsible effective use. The BTFER implementation code available Kaggle Github, respectively.

Language: Английский

---