Multimodal Superiority; Vibrotactile and Sound in Robot-Human Emotion communication

Emotional expression through touch and hearing is a nuanced form of communication that plays a vital role in human-robot interaction. Yet, a key question remains: how can emotions and touch gestures be effectively conveyed between individuals using both haptic and auditory cues? This study investigates a multimodal approach that combines haptic and auditory signals to enhance emotion and gesture communication in human-robot interaction. By employing a 5x5 grid of 25 vibration motors synchronized with audio playback, we address existing research gaps that largely focus on single-sensory channels. Our main objectives are to determine (1) whether combined haptic-audio feedback improves decoding accuracy compared to single modalities, (2) the potential of gestures alone to convey specific emotions, and (3) the unique benefits each sensory channel provides in interpreting emotional cues. In this experiment, 32 participants interpreted 10 emotions and 6 gestures through vibrational, auditory, and combined cues, rating each stimulus on arousal and valence scales. Findings indicate that the combined haptic-audio modality significantly improves decoding accuracy compared to single channels, while each individual channel (vibration or sound) contributes effectively to emotion recognition with specific advantages in different contexts. However, gestures alone generally lacked the ability to communicate distinct emotions. These results highlight the benefits of multimodal feedback for conveying subtle emotional content and underscore the complementary roles of vibration and sound in enhancing human-robot interaction.

This paper is still under submission, and we will update it as soon as possible after it is online.