Using EfficientNet-B7 (CNN), Variational Auto Encoder (VAE) and Siamese Twins’ Networks to Evaluate Human Exercises as Super Objects in a TSSCI Images

Yoram Segal, Ofer Hadar, Lenka Lhotska

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


In this article, we introduce a new approach to human movement by defining the movement as a static super object represented by a single two-dimensional image. The described method is applicable in remote healthcare applications, such as physiotherapeutic exercises. It allows researchers to label and describe the entire exercise as a standalone object, isolated from the reference video. This approach allows us to perform various tasks, including detecting similar movements in a video, measuring and comparing movements, generating new similar movements, and defining choreography by controlling specific parameters in the human body skeleton. As a result of the presented approach, we can eliminate the need to label images manually, disregard the problem of finding the start and the end of an exercise, overcome synchronization issues between movements, and perform any deep learning network-based operation that processes super objects in images in general. As part of this article, we will demonstrate two application use cases: one illustrates how to verify and score a fitness exercise. In contrast, the other illustrates how to generate similar movements in the human skeleton space by addressing the challenge of supplying sufficient training data for deep learning applications (DL). A variational auto encoder (VAE) simulator and an EfficientNet-B7 classifier architecture embedded within a Siamese twin neural network are presented in this paper in order to demonstrate the two use cases. These use cases demonstrate the versatility of our innovative concept in measuring, categorizing, inferring human behavior, and generating gestures for other researchers.

Original languageEnglish
Article number874
JournalJournal of Personalized Medicine
Issue number5
StatePublished - 1 May 2023


  • MediaPipe (MP)
  • OpenPose (OP)
  • Siamese twin neural network
  • computational creativity
  • computational imagination
  • human body movements
  • human pose estimation (HPE)
  • rehabilitation
  • simulator
  • tree structure skeleton color image (TSSCI)
  • tree structure skeleton image (TSSI)
  • variational auto encoder (VAE)

ASJC Scopus subject areas

  • Medicine (miscellaneous)


Dive into the research topics of 'Using EfficientNet-B7 (CNN), Variational Auto Encoder (VAE) and Siamese Twins’ Networks to Evaluate Human Exercises as Super Objects in a TSSCI Images'. Together they form a unique fingerprint.

Cite this