Semantic Interaction with Human Motion Using Query-Based Recombinant Video Synthesis

Vignesh Gokul, Ganesh Prasanna Balakrishnan, Tammuz Dubnov, Shlomo Dubnov

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

The ability of a machine to understand the motion and behaviour of a particular actor is a very important task in machine vision. This problem has so many possible applications in domains such as motion retargeting, robot navigation, healthcare, psychology, augmented reality applications such as games etc. In this paper we demonstrate a human-robot interaction system based on a gestural query, where the computer response is a computer generated video of another human movement. This work differs from other recent video retargeting systems since it is not meant to modify the target video as such, but rather query a video database for the most responsive segment through gestural interpretation process. For this purpose we developed a generative video system capable of extracting the latent representation of free movements such as dance and expressive gesture, and querying and re-editing multiple found video segments in response to an input movement query. One of the main challenges in this approach is finding the 'units' of continuous movement input so that both the style of the target video and the relevant aspect of the query video would be related in a meaningful way. In this paper we describe a gestural motif extraction system that combines deep feature learning with structural similarity analysis to allow such query based human-computer motion interaction.

Original languageEnglish
Title of host publicationProceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages379-382
Number of pages4
ISBN (Electronic)9781728111988
DOIs
StatePublished - 22 Apr 2019
Externally publishedYes
Event2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019 - San Jose, United States
Duration: 28 Mar 201930 Mar 2019

Publication series

NameProceedings - 2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019

Conference

Conference2nd International Conference on Multimedia Information Processing and Retrieval, MIPR 2019
Country/TerritoryUnited States
CitySan Jose
Period28/03/1930/03/19

Keywords

  • Motif Detection
  • Variable Markov Oracle
  • Video Querying
  • Video Synthesis

Fingerprint

Dive into the research topics of 'Semantic Interaction with Human Motion Using Query-Based Recombinant Video Synthesis'. Together they form a unique fingerprint.

Cite this