Collaboration with a robotic scrub nurse

The main use of robotics in surgery is not to replace the surgeon or surgical nurses but to work with them during surgery. A significant advantage of gesture-based communication is it requires no special training by the surgeon. Gesturing comes naturally to surgeons since their hands are already their main tools. Moreover, hand signs are the standard method for requesting surgical instruments, and gestures are not affected by ambient noise in the OR. A multimodal solution combining voice and gesture provides redundancy needed to assure proper instrument delivery. Gestures for robotic control have been the focus of much research since the early 1980s. Early work was done with Richard A. Bolt's Put-That-There interface followed by others using magnetic sensors or gloves to encode hand signs. The streaming depth maps captured through the Kinect sensor are processed by the gesture recognition module while a microphone concurrently captures voice commands interpreted by the speech recognition module. Following recognition, a command is transmitted to the robot through an application that controls a Fanuc LR Mate 200iC robotic arm across the network through a Telnet interface.