Needle in a haystack queries in cloud data lakes

Grisha Weintraub; Ehud Gudes; Shlomi Dolev

Needle in a haystack queries in cloud data lakes

Grisha Weintraub
, Ehud Gudes
, Shlomi Dolev

Research output: Contribution to journal › Conference article › peer-review

8 Scopus citations

Abstract

Cloud data lakes are a modern approach for storing large amounts of data in a convenient and inexpensive way. Query engines (e.g. Hive, Presto, SparkSQL) are used to run SQL queries on data lakes. Their main focus is on analytical queries while random reads are overlooked. In this paper, we present our approach for optimizing needle in a haystack queries in cloud data lakes. The main idea is to maintain an index structure that maps indexed column values to their files. According to our analysis and experimental evaluation, our solution imposes a reasonable storage overhead while providing an order of magnitude performance improvement.

Original language	English
Journal	CEUR Workshop Proceedings
Volume	2841
State	Published - 1 Jan 2021
Event	2021 Workshops of the EDBT/ICDT Joint Conference, EDBT/ICDT-WS 2021 - Virtual, Online, Cyprus Duration: 23 Mar 2021 → …

ASJC Scopus subject areas

General Computer Science

Cite this

@article{8c6177f72bba45b7bc9a7602053ccefc,

title = "Needle in a haystack queries in cloud data lakes",

abstract = "Cloud data lakes are a modern approach for storing large amounts of data in a convenient and inexpensive way. Query engines (e.g. Hive, Presto, SparkSQL) are used to run SQL queries on data lakes. Their main focus is on analytical queries while random reads are overlooked. In this paper, we present our approach for optimizing needle in a haystack queries in cloud data lakes. The main idea is to maintain an index structure that maps indexed column values to their files. According to our analysis and experimental evaluation, our solution imposes a reasonable storage overhead while providing an order of magnitude performance improvement.",

author = "Grisha Weintraub and Ehud Gudes and Shlomi Dolev",

note = "Publisher Copyright: {\textcopyright} 2021 Copyright for this paper by its author(s).; 2021 Workshops of the EDBT/ICDT Joint Conference, EDBT/ICDT-WS 2021 ; Conference date: 23-03-2021",

year = "2021",

month = jan,

day = "1",

language = "English",

volume = "2841",

journal = "CEUR Workshop Proceedings",

issn = "1613-0073",

publisher = "CEUR-WS",

}

TY - JOUR

T1 - Needle in a haystack queries in cloud data lakes

AU - Weintraub, Grisha

AU - Gudes, Ehud

AU - Dolev, Shlomi

PY - 2021/1/1

Y1 - 2021/1/1

N2 - Cloud data lakes are a modern approach for storing large amounts of data in a convenient and inexpensive way. Query engines (e.g. Hive, Presto, SparkSQL) are used to run SQL queries on data lakes. Their main focus is on analytical queries while random reads are overlooked. In this paper, we present our approach for optimizing needle in a haystack queries in cloud data lakes. The main idea is to maintain an index structure that maps indexed column values to their files. According to our analysis and experimental evaluation, our solution imposes a reasonable storage overhead while providing an order of magnitude performance improvement.

AB - Cloud data lakes are a modern approach for storing large amounts of data in a convenient and inexpensive way. Query engines (e.g. Hive, Presto, SparkSQL) are used to run SQL queries on data lakes. Their main focus is on analytical queries while random reads are overlooked. In this paper, we present our approach for optimizing needle in a haystack queries in cloud data lakes. The main idea is to maintain an index structure that maps indexed column values to their files. According to our analysis and experimental evaluation, our solution imposes a reasonable storage overhead while providing an order of magnitude performance improvement.

UR - https://www.scopus.com/pages/publications/85103510338

M3 - Conference article

AN - SCOPUS:85103510338

SN - 1613-0073

VL - 2841

JO - CEUR Workshop Proceedings

JF - CEUR Workshop Proceedings

T2 - 2021 Workshops of the EDBT/ICDT Joint Conference, EDBT/ICDT-WS 2021

Y2 - 23 March 2021

ER -

Needle in a haystack queries in cloud data lakes

Abstract

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this