Expert Systems· 2026Q2
Evaluating Multimodal Emotion Recognition in Proactive Conversational Agents: A User Study
- 0citations
- Q2SCImago
- 2026year
Short summary
A study of 20 users interacting with a generative AI agent found that facial expressions (visual cues) were unreliable for detecting emotions, with users often maintaining a neutral 'poker face' even when feeling positive. Linguistic analysis of verbal expressions proved significantly more accurate.
AI-generated from the title and abstract; the full text is not read.
Key points
- Facial expressions were unreliable for detecting user emotions in AI interactions, with users often displaying a neutral 'poker face'.
- Linguistic analysis of users' verbal expressions was significantly more accurate than visual cues for emotion recognition.
- Conversational agents can elicit specific emotions by adapting themes and using structured language (e.g., empathy, humor).
- Uncalibrated AI proactivity can cause user disengagement and a perception of artificiality.
AI-generated from the title and abstract; the full text is not read.
Abstract
ABSTRACT This article presents a multimodal emotion recognition module integrated into a proactive Socially Interactive Agent (SIA) powered by generative artificial intelligence. The system evaluates real‐time affective states through two distinct channels: a computer vision‐based facial recognition module and a semantic linguistic analysis engine. To validate the framework, an empirical study was conducted with 20 users who engaged in dynamic, unscripted dialogues with the conversational agent. The findings reveal a significant discrepancy between automated visual cues and actual internal emotional states. When interacting with the AI, users consistently exhibited a ‘poker face’ effect, displaying serious, concentrated facial expressions even when experiencing positive emotions. Consequently, the generative AI linguistic analysis proved significantly more reliable, by contextualizing the users' verbal expressions. Furthermore, an analysis of the interaction dynamics demonstrated that SIAs can effectively elicit specific emotions by adapting conversational themes and employing structured linguistic patterns, such as empathetic or humorous language. However, the study also noted that instances of uncalibrated proactivity occasionally led to user disengagement and a perception of artificiality. Ultimately, this research highlights the necessity of refining SIAs to dynamically adapt to users' emotional evolution, relying on deep linguistic context to foster more natural, human‐like interactions.
The authors' abstract, as published at the source. Expert Systems, 2026 · DOI ↗
Continue with a free account
Ask the paper: 3 free questions a day about this paper; save it, get its citation, new summaries every day for your field. Takeaways are Premium.
Continue free on the webSign in with Google or Apple; no card needed. You come back to this paper.
On your phone:
Field: Experimental and Cognitive Psychology
Experimental and Cognitive PsychologyPsychology