Pretrained Models for Multilingual Federated Learning

Orion Weller; Marc Marone; Vladimir Braverman; Dawn Lawrie; Benjamin Van Durme

doi:10.18653/v1/2022.naacl-main.101

Pretrained Models for Multilingual Federated Learning

Orion Weller
, Marc Marone
, Vladimir Braverman
, Dawn Lawrie
, Benjamin Van Durme

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

21 Scopus citations

Abstract

Since the advent of Federated Learning (FL), research has applied these methods to natural language processing (NLP) tasks. Despite a plethora of papers in FL for NLP, no previous works have studied how multilingual text impacts FL algorithms. Furthermore, multilingual text provides an interesting avenue to examine the impact of non-IID text (e.g. different languages) on FL in naturally occurring data. We explore three multilingual language tasks, language modeling, machine translation, and text classification using differing federated and non-federated learning algorithms. Our results show that using pretrained models reduces the negative effects of FL, helping them to perform near or better than centralized (no privacy) learning, even when using non-IID partitioning.

Original language	English
Title of host publication	NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics
Subtitle of host publication	Human Language Technologies, Proceedings of the Conference
Publisher	Association for Computational Linguistics (ACL)
Pages	1413-1421
Number of pages	9
ISBN (Electronic)	9781955917711
DOIs	https://doi.org/10.18653/v1/2022.naacl-main.101
State	Published - 1 Jan 2022
Externally published	Yes
Event	2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022 - Hybrid, Seattle, United States Duration: 10 Jul 2022 → 15 Jul 2022

Publication series

Name	NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

Conference

Conference	2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022
Country/Territory	United States
City	Hybrid, Seattle
Period	10/07/22 → 15/07/22

ASJC Scopus subject areas

Computer Networks and Communications
Hardware and Architecture
Information Systems
Software

Access to Document

10.18653/v1/2022.naacl-main.101

Cite this

Weller, O., Marone, M., Braverman, V., Lawrie, D., & Van Durme, B. (2022). Pretrained Models for Multilingual Federated Learning. In NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference (pp. 1413-1421). (NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.naacl-main.101

Weller, Orion ; Marone, Marc ; Braverman, Vladimir et al. / Pretrained Models for Multilingual Federated Learning. NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2022. pp. 1413-1421 (NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference).

@inproceedings{376843f2e43f49b3b885120a2c435d0d,

title = "Pretrained Models for Multilingual Federated Learning",

abstract = "Since the advent of Federated Learning (FL), research has applied these methods to natural language processing (NLP) tasks. Despite a plethora of papers in FL for NLP, no previous works have studied how multilingual text impacts FL algorithms. Furthermore, multilingual text provides an interesting avenue to examine the impact of non-IID text (e.g. different languages) on FL in naturally occurring data. We explore three multilingual language tasks, language modeling, machine translation, and text classification using differing federated and non-federated learning algorithms. Our results show that using pretrained models reduces the negative effects of FL, helping them to perform near or better than centralized (no privacy) learning, even when using non-IID partitioning.",

author = "Orion Weller and Marc Marone and Vladimir Braverman and Dawn Lawrie and \{Van Durme\}, Benjamin",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022 ; Conference date: 10-07-2022 Through 15-07-2022",

year = "2022",

month = jan,

day = "1",

doi = "10.18653/v1/2022.naacl-main.101",

language = "English",

series = "NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "1413--1421",

booktitle = "NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics",

address = "United States",

}

Weller, O, Marone, M, Braverman, V, Lawrie, D & Van Durme, B 2022, Pretrained Models for Multilingual Federated Learning. in NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, Association for Computational Linguistics (ACL), pp. 1413-1421, 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Hybrid, Seattle, United States, 10/07/22. https://doi.org/10.18653/v1/2022.naacl-main.101

Pretrained Models for Multilingual Federated Learning. / Weller, Orion; Marone, Marc; Braverman, Vladimir et al.
NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2022. p. 1413-1421 (NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Pretrained Models for Multilingual Federated Learning

AU - Weller, Orion

AU - Marone, Marc

AU - Braverman, Vladimir

AU - Lawrie, Dawn

AU - Van Durme, Benjamin

PY - 2022/1/1

Y1 - 2022/1/1

N2 - Since the advent of Federated Learning (FL), research has applied these methods to natural language processing (NLP) tasks. Despite a plethora of papers in FL for NLP, no previous works have studied how multilingual text impacts FL algorithms. Furthermore, multilingual text provides an interesting avenue to examine the impact of non-IID text (e.g. different languages) on FL in naturally occurring data. We explore three multilingual language tasks, language modeling, machine translation, and text classification using differing federated and non-federated learning algorithms. Our results show that using pretrained models reduces the negative effects of FL, helping them to perform near or better than centralized (no privacy) learning, even when using non-IID partitioning.

AB - Since the advent of Federated Learning (FL), research has applied these methods to natural language processing (NLP) tasks. Despite a plethora of papers in FL for NLP, no previous works have studied how multilingual text impacts FL algorithms. Furthermore, multilingual text provides an interesting avenue to examine the impact of non-IID text (e.g. different languages) on FL in naturally occurring data. We explore three multilingual language tasks, language modeling, machine translation, and text classification using differing federated and non-federated learning algorithms. Our results show that using pretrained models reduces the negative effects of FL, helping them to perform near or better than centralized (no privacy) learning, even when using non-IID partitioning.

UR - https://www.scopus.com/pages/publications/85138330868

U2 - 10.18653/v1/2022.naacl-main.101

DO - 10.18653/v1/2022.naacl-main.101

M3 - Conference contribution

AN - SCOPUS:85138330868

T3 - NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference

SP - 1413

EP - 1421

BT - NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

T2 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022

Y2 - 10 July 2022 through 15 July 2022

ER -

Weller O, Marone M, Braverman V, Lawrie D, Van Durme B. Pretrained Models for Multilingual Federated Learning. In NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2022. p. 1413-1421. (NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference). doi: 10.18653/v1/2022.naacl-main.101

Pretrained Models for Multilingual Federated Learning

Abstract

Publication series

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this