Optimal modality selection for multimodal human-machine systems using RIMAG

Interpersonal communication in human teams is multimodal by nature and hybrid robot-human teams should be capable of utilizing diverse verbal and non-verbal communication channels (e.g. gestures, speech, and gaze). Additionally, this interaction must fulfill requirements such as speed, accuracy and resilience. While multimodal communication has been researched and human-robot mixed team communication frameworks have been developed, the computation of an effective combination of communication modalities (multimodal lexicon) to maximize effectiveness is an untapped area of research. The proposed framework objectively determines the set of optimal lexicons through multiobjective optimization of performance metrics over all feasible lexicons. The methodology is applied to the surgical setting, where a robotic nurse can collaborate with a surgical team by delivering surgical instruments as required. In this time-sensitive, high-risk context, performance metrics are obtained through a mixture of real-world experiments and simulation. Experimental results validate the predictability of the method since predicted optimal lexicons significantly (p < 0.01) outperform predicted suboptimal lexicons in time, error rate and false positive rates.