Beyond MAGIC: Matching Collaborative Gestures using an optimization-based Approach

Gestures are a key aspect of communication during collaboration: through gestures we can express ideas, inquires and formalize instructions as we collaborate. Nevertheless, gesture analysis is not currently used to assess quality of task collaboration. One possible reason for this is that there is no consensus on how to represent and compare gestures from the semantic standpoint. To address this, this paper introduces three novel approaches to compare gestures performed by individuals as they collaborate to complete a physical task. Our approach relies on solving three variations of an integer optimization assignment problem, i.e. based on gesture similarity, based on temporal synchrony, and based on a combination of both. We collected the gestures of 40 participants (divided into 20 pairs) as they performed two collaborative tasks, and generated a human baseline that compared and matched their gestures. Afterwards, our gesture comparison approach was evaluated against other gestures comparison approaches based on how well they replicated the human baseline. Our approach outperformed the other approaches, agreeing with the human baseline over 85% of the times. Thus, the obtained results support the proposed technique for gesture comparison. This in turn can lead to the development of better methods to evaluate collaborative physical tasks.