DRIVE: One-bit Distributed Mean Estimation

Shay Vargaftik; Ran Ben Basat; Amit Portnoy; Gal Mendelson; Yaniv Ben-Itzhak; Michael Mitzenmacher

DRIVE: One-bit Distributed Mean Estimation

Shay Vargaftik
, Ran Ben Basat
, Amit Portnoy
, Gal Mendelson
, Yaniv Ben-Itzhak
, Michael Mitzenmacher

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

32 Scopus citations

Abstract

We consider the problem where n clients transmit d-dimensional real-valued vectors using d(1 + o(1)) bits each, in a manner that allows the receiver to approximately reconstruct their mean. Such compression problems naturally arise in distributed and federated learning. We provide novel mathematical results and derive computationally efficient algorithms that are more accurate than previous compression techniques. We evaluate our methods on a collection of distributed and federated learning tasks, using a variety of datasets, and show a consistent improvement over the state of the art.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021
Editors	Marc'Aurelio Ranzato, Alina Beygelzimer, Yann Dauphin, Percy S. Liang, Jenn Wortman Vaughan
Publisher	Neural information processing systems foundation
Pages	362-377
Number of pages	16
ISBN (Electronic)	9781713845393
State	Published - 1 Jan 2021
Event	35th Conference on Neural Information Processing Systems, NeurIPS 2021 - Virtual, Online Duration: 6 Dec 2021 → 14 Dec 2021

Publication series

Name	Advances in Neural Information Processing Systems
Volume	1
ISSN (Print)	1049-5258

Conference

Conference	35th Conference on Neural Information Processing Systems, NeurIPS 2021
City	Virtual, Online
Period	6/12/21 → 14/12/21

ASJC Scopus subject areas

Signal Processing
Information Systems
Computer Networks and Communications

Cite this

Vargaftik, S., Basat, R. B., Portnoy, A., Mendelson, G., Ben-Itzhak, Y., & Mitzenmacher, M. (2021). DRIVE: One-bit Distributed Mean Estimation. In MA. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, & J. Wortman Vaughan (Eds.), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021 (pp. 362-377). (Advances in Neural Information Processing Systems; Vol. 1). Neural information processing systems foundation.

Vargaftik, Shay ; Basat, Ran Ben ; Portnoy, Amit et al. / DRIVE : One-bit Distributed Mean Estimation. Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. editor / Marc'Aurelio Ranzato ; Alina Beygelzimer ; Yann Dauphin ; Percy S. Liang ; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. pp. 362-377 (Advances in Neural Information Processing Systems).

@inproceedings{177af0adce314c87b236f87e4bfc9144,

title = "DRIVE: One-bit Distributed Mean Estimation",

abstract = "We consider the problem where n clients transmit d-dimensional real-valued vectors using d(1 + o(1)) bits each, in a manner that allows the receiver to approximately reconstruct their mean. Such compression problems naturally arise in distributed and federated learning. We provide novel mathematical results and derive computationally efficient algorithms that are more accurate than previous compression techniques. We evaluate our methods on a collection of distributed and federated learning tasks, using a variety of datasets, and show a consistent improvement over the state of the art.",

author = "Shay Vargaftik and Basat, \{Ran Ben\} and Amit Portnoy and Gal Mendelson and Yaniv Ben-Itzhak and Michael Mitzenmacher",

note = "Publisher Copyright: {\textcopyright} 2021 Neural information processing systems foundation. All rights reserved.; 35th Conference on Neural Information Processing Systems, NeurIPS 2021 ; Conference date: 06-12-2021 Through 14-12-2021",

year = "2021",

month = jan,

day = "1",

language = "English",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

pages = "362--377",

editor = "Marc'Aurelio Ranzato and Alina Beygelzimer and Yann Dauphin and Liang, \{Percy S.\} and \{Wortman Vaughan\}, Jenn",

booktitle = "Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021",

}

Vargaftik, S, Basat, RB, Portnoy, A, Mendelson, G, Ben-Itzhak, Y & Mitzenmacher, M 2021, DRIVE: One-bit Distributed Mean Estimation. in MA Ranzato, A Beygelzimer, Y Dauphin, PS Liang & J Wortman Vaughan (eds), Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Advances in Neural Information Processing Systems, vol. 1, Neural information processing systems foundation, pp. 362-377, 35th Conference on Neural Information Processing Systems, NeurIPS 2021, Virtual, Online, 6/12/21.

DRIVE: One-bit Distributed Mean Estimation. / Vargaftik, Shay; Basat, Ran Ben; Portnoy, Amit et al.
Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. ed. / Marc'Aurelio Ranzato; Alina Beygelzimer; Yann Dauphin; Percy S. Liang; Jenn Wortman Vaughan. Neural information processing systems foundation, 2021. p. 362-377 (Advances in Neural Information Processing Systems; Vol. 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - DRIVE

T2 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

AU - Vargaftik, Shay

AU - Basat, Ran Ben

AU - Portnoy, Amit

AU - Mendelson, Gal

AU - Ben-Itzhak, Yaniv

AU - Mitzenmacher, Michael

PY - 2021/1/1

Y1 - 2021/1/1

N2 - We consider the problem where n clients transmit d-dimensional real-valued vectors using d(1 + o(1)) bits each, in a manner that allows the receiver to approximately reconstruct their mean. Such compression problems naturally arise in distributed and federated learning. We provide novel mathematical results and derive computationally efficient algorithms that are more accurate than previous compression techniques. We evaluate our methods on a collection of distributed and federated learning tasks, using a variety of datasets, and show a consistent improvement over the state of the art.

AB - We consider the problem where n clients transmit d-dimensional real-valued vectors using d(1 + o(1)) bits each, in a manner that allows the receiver to approximately reconstruct their mean. Such compression problems naturally arise in distributed and federated learning. We provide novel mathematical results and derive computationally efficient algorithms that are more accurate than previous compression techniques. We evaluate our methods on a collection of distributed and federated learning tasks, using a variety of datasets, and show a consistent improvement over the state of the art.

UR - https://www.scopus.com/pages/publications/85131600857

M3 - Conference contribution

AN - SCOPUS:85131600857

T3 - Advances in Neural Information Processing Systems

SP - 362

EP - 377

BT - Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021

A2 - Ranzato, Marc'Aurelio

A2 - Beygelzimer, Alina

A2 - Dauphin, Yann

A2 - Liang, Percy S.

A2 - Wortman Vaughan, Jenn

PB - Neural information processing systems foundation

Y2 - 6 December 2021 through 14 December 2021

ER -

Vargaftik S, Basat RB, Portnoy A, Mendelson G, Ben-Itzhak Y, Mitzenmacher M. DRIVE: One-bit Distributed Mean Estimation. In Ranzato MA, Beygelzimer A, Dauphin Y, Liang PS, Wortman Vaughan J, editors, Advances in Neural Information Processing Systems 34 - 35th Conference on Neural Information Processing Systems, NeurIPS 2021. Neural information processing systems foundation. 2021. p. 362-377. (Advances in Neural Information Processing Systems).

DRIVE: One-bit Distributed Mean Estimation

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this