Generalization of Deep Learning Gesture Classification in Robotic-Assisted Surgical Data: From Dry Lab to Clinical-Like Data

Danit Itzkovich, Yarden Sharon, Anthony Jarc, Yael Refaely, Ilana Nisky

Research output: Contribution to journalArticlepeer-review

5 Scopus citations


Objective: Robotic-assisted minimally invasive surgery (RAMIS) became a common practice in modern medicine and is widely studied. Surgical procedures require prolonged and complex movements; therefore, classifying surgical gestures could be helpful to characterize surgeon performance. The public release of the JIGSAWS dataset facilitates the development of classification algorithms; however, it is not known how algorithms trained on dry-lab data generalize to real surgical situations. Methods: We trained a Long Short-Term Memory (LSTM) network for the classification of dry lab and clinical-like data into gestures. Results: We show that a network that was trained on the JIGSAWS data does not generalize well to other dry-lab data and to clinical-like data. Using rotation augmentation improves performance on dry-lab tasks, but fails to improve the performance on clinical-like data. However, using the same network architecture, adding the six joint angles of the patient-side manipulators (PSMs) features, and training the network on the clinical-like data together lead to notable improvement in the classification of the clinical-like data. Discussion: Using the JIGSAWS dataset alone is insufficient for training a gesture classification network for clinical data. However, it can be very informative for determining the architecture of the network, and with training on a small sample of clinical data, can lead to acceptable classification performance. Significance: Developing efficient algorithms for gesture classification in clinical surgical data is expected to advance understanding of surgeon sensorimotor control in RAMIS, the automation of surgical skill evaluation, and the automation of surgery.

Original languageEnglish
Pages (from-to)1329-1340
Number of pages12
JournalIEEE Journal of Biomedical and Health Informatics
Issue number3
StatePublished - 1 Mar 2022


  • Augmentation
  • Clinical data
  • Machine learning
  • Medical robotics
  • Supervised learning
  • Surgical robotics

ASJC Scopus subject areas

  • Health Information Management
  • Electrical and Electronic Engineering
  • Biotechnology
  • Computer Science Applications


Dive into the research topics of 'Generalization of Deep Learning Gesture Classification in Robotic-Assisted Surgical Data: From Dry Lab to Clinical-Like Data'. Together they form a unique fingerprint.

Cite this