Optimizing Cloud Data Lake Queries with a Balanced Coverage Plan

Grisha Weintraub, Ehud Gudes, Shlomi Dolev, Jeffrey D. Ullman

Research output: Contribution to journalArticlepeer-review

Abstract

Cloud data lakes emerge as an inexpensive solution for storing very large amounts of data. The main idea is the separation of compute and storage layers. Thus, cheap cloud storage is used for storing the data, while compute engines are used for running analytics on this data in 'on-demand' mode. However, to perform any computation on the data in this architecture, the data should be moved from the storage layer to the compute layer over the network for each calculation. Obviously, that hurts calculation performance and requires huge network bandwidth. In this paper, we study different approaches to improve query performance in a data lake architecture. We define an optimization problem that can provably speed up data lake queries. We prove that the problem is NP-hard and suggest heuristic approaches. Then, we demonstrate through the experiments that our approach is feasible and efficient (up to ×30 query execution time improvement based on the TPC-H benchmark).

Original languageEnglish
Pages (from-to)84-99
Number of pages16
JournalIEEE Transactions on Cloud Computing
Volume12
Issue number1
DOIs
StatePublished - 1 Jan 2024

Keywords

  • Cloud storage
  • data lakes
  • query optimization

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Hardware and Architecture
  • Computer Networks and Communications
  • Computer Science Applications

Fingerprint

Dive into the research topics of 'Optimizing Cloud Data Lake Queries with a Balanced Coverage Plan'. Together they form a unique fingerprint.

Cite this