The Impact of Musical Stimuli on the Interpretation of Facial Expressions: An Eye-Tracking Study

Abstract

Facial recognition are a cornerstone of human social interaction and emotional bonding. People rely on facial recognition to analyze others’ emotions, intentions, and social cues. Facial emotions are particularly effective in conveying information in social contexts. It has the benefits of speed, directness, and universality across cultures. Music, as an emerging area of emotional study, has the power to evoke and alter individual emotions which has the potential to affect emotion recognition. This study examines the impact of different types of music on the judgement of facial emotions. The present research adopts a mixed design, with two types of music as the between-subject variable and three types of facial emotions as the within-subject variable. Participants were asked to evaluate facial expressions while listening to different musical pieces, and an eye-tracker recorded their viewing behaviors. The eye-tracking data, serving as the dependent variable, offers quantifiable details of the observation process. Analysis indicates that under sad music, participants are more likely to rate neutral faces positively. While under happy music, they tend to rate neutral faces more negatively. Eye-tracking results reveal a tendency for participants to focus more on the eye region when viewing faces based on increased total fixation duration, fixation counts, and quicker time to first fixation. Sad music prompts faster gaze towards the eyes, with a shorter time to first fixation compared to happy music. Moreover, when viewing negative emotional faces, participants’ gaze towards the nose is quicker than when viewing positive faces, regardless of the music types. This study contributes a novel perspective to cross-modal emotion recognition research which highlights the interaction between music and facial expression interpretation process. The results suggest new insights for emotion recognition training in music therapy, clinical psychology, and education.