Restoring Eye Contact to the Virtual Classroom with Machine Learning

Ross Greer, Shlomo Dubnov

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


Nonverbal communication, in particular eye contact, is a critical element of the music classroom, shown to keep students on task, coordinate musical flow, and communicate improvisational ideas. Unfortunately, this nonverbal aspect to performance and pedagogy is lost in the virtual classroom. In this paper, we propose a machine learning system which uses single instance, single camera image frames as input to estimate the gaze target of a user seated in front of their computer, augmenting the user’s video feed with a display of the estimated gaze target and thereby restoring nonverbal communication of directed gaze. The proposed estimation system consists of modular machine learning blocks, leading to a target-oriented (rather than coordinate-oriented) gaze prediction. We instantiate one such example of the complete system to run a pilot study in a virtual music classroom over Zoom software. Inference time and accuracy meet benchmarks for videoconferencing applications, and quantitative and qualitative results of pilot experiments include improved success of cue interpretation and student-reported formation of collaborative, communicative relationships between conductor and musician.

Original languageEnglish
Title of host publicationCSEDU 2021 - Proceedings of the 13th International Conference on Computer Supported Education
EditorsBeno Csapo, James Uhomoibhi
PublisherScience and Technology Publications, Lda
Number of pages11
ISBN (Electronic)9789897585029
StatePublished - 1 Jan 2021
Externally publishedYes
Event13th International Conference on Computer Supported Education, CSEDU 2021 - Virtual, Online
Duration: 23 Apr 202125 Apr 2021

Publication series

NameInternational Conference on Computer Supported Education, CSEDU - Proceedings
ISSN (Electronic)2184-5026


Conference13th International Conference on Computer Supported Education, CSEDU 2021
CityVirtual, Online


  • Convolutional Neural Networks
  • Distance Learning
  • Gaze Estimation
  • Human-Computer Interaction
  • Music Education
  • Telematic Performance

ASJC Scopus subject areas

  • Computer Science Applications
  • Information Systems


Dive into the research topics of 'Restoring Eye Contact to the Virtual Classroom with Machine Learning'. Together they form a unique fingerprint.

Cite this