Meta-X: A Technique for Reducing Communication in Geographically Distributed Computations

Foto Afrati, Shlomi Dolev, Shantanu Sharma, Jeffrey D. Ullman

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Computations, such as syndromic surveillance and e-commerce, are executed over the datasets collected from different geographical locations. Modern data processing systems, such as MapReduce/Hadoop or Spark, also, require to collect the data from different geographical locations to a single global location, before executing an application, and thus, result in a significant communication cost. While MapReduce/Hadoop and Spark have proven to be the most useful paradigms in the revolution of distributed computing, the federation of cloud and big-data activities is the challenge, wherein data processing should be modified to avoid (big) data migration across remote (cloud) sites. This is exactly our scope of work, where only the very essential data for obtaining the final result is transmitted, for reducing communication and processing, and for preserving data privacy as much as possible. In this work, we propose an algorithmic technique for geographically distributed computations, called Meta-X, that decreases the communication cost by allowing us to process and moves metadata to among different locations, instead of the entire datasets. We illustrate the usefulness of Meta-X in terms of MapReduce computations for different operations, such as equijoin, k-nearest-neighbors finding, and shortest path finding.

Original languageEnglish
Title of host publicationCyber Security Cryptography and Machine Learning - 5th International Symposium, CSCML 2021, Proceedings
EditorsShlomi Dolev, Oded Margalit, Benny Pinkas, Alexander Schwarzmann
PublisherSpringer Science and Business Media Deutschland GmbH
Pages467-486
Number of pages20
ISBN (Print)9783030780852
DOIs
StatePublished - 1 Jan 2021
Event5th International Symposium on Cyber Security Cryptography and Machine Learning, CSCML 2021 - Be'er Sheva, Israel
Duration: 8 Jul 20219 Jul 2021

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12716 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference5th International Symposium on Cyber Security Cryptography and Machine Learning, CSCML 2021
Country/TerritoryIsrael
CityBe'er Sheva
Period8/07/219/07/21

Keywords

  • Hadoop
  • MapReduce
  • Spark

Fingerprint

Dive into the research topics of 'Meta-X: A Technique for Reducing Communication in Geographically Distributed Computations'. Together they form a unique fingerprint.

Cite this