We are developing a method for extraction of sequential pattern of nonverbal behaviors in the context of multiparty conversation for extracting machine-readable interaction protocols that we unconsciously use in our daily conversations. We have build interaction corpora containing multi-viewpoint videos, voice sound, motion data, and eye tracking data of each participant to the conversation as well as interaction primitives articulated from the above data, i.e., turn taking of speech, back-channel feedback, head nodding, viewpoint movement, pointing gesture, etc. In our interaction mining method, the sequential patterns of these nonverbal behaviors are represented in N-gram (Fig.1). In the N-gram representation, we could find characteristic patterns depending on individual conversational situations such as turn-taking mechanisms among the participants of poster presentation.
Fig.1: An example of sequential pattern of nonverbal behaviors shown as N-gram representation.
Language:
Meta