Shrinking Bigfoot: Reducing wav2vec 2.0 footprint

Zilun Peng; Akshay Budhkar; Ilana Tuil; Jason Levy; Parinaz Sobhani; Raphael Cohen; Jumana Nassour

Shrinking Bigfoot: Reducing wav2vec 2.0 footprint

Zilun Peng
, Akshay Budhkar
, Ilana Tuil
, Jason Levy
, Parinaz Sobhani
, Raphael Cohen
, Jumana Nassour

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

17 Scopus citations

Abstract

Wav2vec 2.0 is a state-of-the-art speech recognition model which maps speech audio waveforms into latent representations. The largest version of wav2vec 2.0 contains 317 million parameters. Hence, the inference latency of wav2vec 2.0 will be a bottleneck in production, leading to high costs and a significant environmental footprint. To improve wav2vec’s applicability to a production setting, we explore multiple model compression methods borrowed from the domain of large language models. Using a teacher-student approach, we distilled the knowledge from the original wav2vec 2.0 model into a student model, which is 2 times faster, 4.8 times smaller than the original model. More importantly, the student model is 2 times more energy efficient than the original model in terms of CO₂ emission. This increase in performance is accomplished with only a 7% degradation in word error rate (WER). Our quantized model is 3.6 times smaller than the original model, with only a 0.1% degradation in WER. To the best of our knowledge, this is the first work that compresses wav2vec 2.0.

Original language	English
Title of host publication	SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP
Editors	Nafise Sadat Moosavi, Iryna Gurevych, Angela Fan, Thomas Wolf, Yufang Hou, Ana Marasovic, Sujith Ravi
Publisher	Association for Computational Linguistics (ACL)
Pages	134-141
Number of pages	8
ISBN (Electronic)	9781955917018
State	Published - 1 Jan 2021
Externally published	Yes
Event	2nd Workshop on Simple and Efficient Natural Language Processing, SustaiNLP 2021 - Virtual, Online Duration: 10 Nov 2021 → …

Publication series

Name	SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP

Conference

Conference	2nd Workshop on Simple and Efficient Natural Language Processing, SustaiNLP 2021
City	Virtual, Online
Period	10/11/21 → …

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 7 Affordable and Clean Energy
SDG 13 Climate Action

ASJC Scopus subject areas

Language and Linguistics
Computational Theory and Mathematics
Software
Linguistics and Language

Cite this

Peng, Z., Budhkar, A., Tuil, I., Levy, J., Sobhani, P., Cohen, R., & Nassour, J. (2021). Shrinking Bigfoot: Reducing wav2vec 2.0 footprint. In N. S. Moosavi, I. Gurevych, A. Fan, T. Wolf, Y. Hou, A. Marasovic, & S. Ravi (Eds.), SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP (pp. 134-141). (SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP). Association for Computational Linguistics (ACL).

Peng, Zilun ; Budhkar, Akshay ; Tuil, Ilana et al. / Shrinking Bigfoot : Reducing wav2vec 2.0 footprint. SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP. editor / Nafise Sadat Moosavi ; Iryna Gurevych ; Angela Fan ; Thomas Wolf ; Yufang Hou ; Ana Marasovic ; Sujith Ravi. Association for Computational Linguistics (ACL), 2021. pp. 134-141 (SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP).

@inproceedings{d00d84dc35c646f8a1fa2bdd376d8744,

title = "Shrinking Bigfoot: Reducing wav2vec 2.0 footprint",

abstract = "Wav2vec 2.0 is a state-of-the-art speech recognition model which maps speech audio waveforms into latent representations. The largest version of wav2vec 2.0 contains 317 million parameters. Hence, the inference latency of wav2vec 2.0 will be a bottleneck in production, leading to high costs and a significant environmental footprint. To improve wav2vec{\textquoteright}s applicability to a production setting, we explore multiple model compression methods borrowed from the domain of large language models. Using a teacher-student approach, we distilled the knowledge from the original wav2vec 2.0 model into a student model, which is 2 times faster, 4.8 times smaller than the original model. More importantly, the student model is 2 times more energy efficient than the original model in terms of CO2 emission. This increase in performance is accomplished with only a 7\% degradation in word error rate (WER). Our quantized model is 3.6 times smaller than the original model, with only a 0.1\% degradation in WER. To the best of our knowledge, this is the first work that compresses wav2vec 2.0.",

author = "Zilun Peng and Akshay Budhkar and Ilana Tuil and Jason Levy and Parinaz Sobhani and Raphael Cohen and Jumana Nassour",

note = "Publisher Copyright: {\textcopyright} 201 Association for Computational Linguistics.; 2nd Workshop on Simple and Efficient Natural Language Processing, SustaiNLP 2021 ; Conference date: 10-11-2021",

year = "2021",

month = jan,

day = "1",

language = "English",

series = "SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP",

publisher = "Association for Computational Linguistics (ACL)",

pages = "134--141",

editor = "Moosavi, \{Nafise Sadat\} and Iryna Gurevych and Angela Fan and Thomas Wolf and Yufang Hou and Ana Marasovic and Sujith Ravi",

booktitle = "SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP",

address = "United States",

}

Peng, Z, Budhkar, A, Tuil, I, Levy, J, Sobhani, P, Cohen, R & Nassour, J 2021, Shrinking Bigfoot: Reducing wav2vec 2.0 footprint. in NS Moosavi, I Gurevych, A Fan, T Wolf, Y Hou, A Marasovic & S Ravi (eds), SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP. SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP, Association for Computational Linguistics (ACL), pp. 134-141, 2nd Workshop on Simple and Efficient Natural Language Processing, SustaiNLP 2021, Virtual, Online, 10/11/21.

Shrinking Bigfoot: Reducing wav2vec 2.0 footprint. / Peng, Zilun; Budhkar, Akshay; Tuil, Ilana et al.
SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP. ed. / Nafise Sadat Moosavi; Iryna Gurevych; Angela Fan; Thomas Wolf; Yufang Hou; Ana Marasovic; Sujith Ravi. Association for Computational Linguistics (ACL), 2021. p. 134-141 (SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Shrinking Bigfoot

T2 - 2nd Workshop on Simple and Efficient Natural Language Processing, SustaiNLP 2021

AU - Peng, Zilun

AU - Budhkar, Akshay

AU - Tuil, Ilana

AU - Levy, Jason

AU - Sobhani, Parinaz

AU - Cohen, Raphael

AU - Nassour, Jumana

PY - 2021/1/1

Y1 - 2021/1/1

N2 - Wav2vec 2.0 is a state-of-the-art speech recognition model which maps speech audio waveforms into latent representations. The largest version of wav2vec 2.0 contains 317 million parameters. Hence, the inference latency of wav2vec 2.0 will be a bottleneck in production, leading to high costs and a significant environmental footprint. To improve wav2vec’s applicability to a production setting, we explore multiple model compression methods borrowed from the domain of large language models. Using a teacher-student approach, we distilled the knowledge from the original wav2vec 2.0 model into a student model, which is 2 times faster, 4.8 times smaller than the original model. More importantly, the student model is 2 times more energy efficient than the original model in terms of CO2 emission. This increase in performance is accomplished with only a 7% degradation in word error rate (WER). Our quantized model is 3.6 times smaller than the original model, with only a 0.1% degradation in WER. To the best of our knowledge, this is the first work that compresses wav2vec 2.0.

AB - Wav2vec 2.0 is a state-of-the-art speech recognition model which maps speech audio waveforms into latent representations. The largest version of wav2vec 2.0 contains 317 million parameters. Hence, the inference latency of wav2vec 2.0 will be a bottleneck in production, leading to high costs and a significant environmental footprint. To improve wav2vec’s applicability to a production setting, we explore multiple model compression methods borrowed from the domain of large language models. Using a teacher-student approach, we distilled the knowledge from the original wav2vec 2.0 model into a student model, which is 2 times faster, 4.8 times smaller than the original model. More importantly, the student model is 2 times more energy efficient than the original model in terms of CO2 emission. This increase in performance is accomplished with only a 7% degradation in word error rate (WER). Our quantized model is 3.6 times smaller than the original model, with only a 0.1% degradation in WER. To the best of our knowledge, this is the first work that compresses wav2vec 2.0.

UR - https://www.scopus.com/pages/publications/85134641134

M3 - Conference contribution

AN - SCOPUS:85134641134

T3 - SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP

SP - 134

EP - 141

BT - SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP

A2 - Moosavi, Nafise Sadat

A2 - Gurevych, Iryna

A2 - Fan, Angela

A2 - Wolf, Thomas

A2 - Hou, Yufang

A2 - Marasovic, Ana

A2 - Ravi, Sujith

PB - Association for Computational Linguistics (ACL)

Y2 - 10 November 2021

ER -

Peng Z, Budhkar A, Tuil I, Levy J, Sobhani P, Cohen R et al. Shrinking Bigfoot: Reducing wav2vec 2.0 footprint. In Moosavi NS, Gurevych I, Fan A, Wolf T, Hou Y, Marasovic A, Ravi S, editors, SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP. Association for Computational Linguistics (ACL). 2021. p. 134-141. (SustaiNLP 2021 - 2nd Workshop on Simple and Efficient Natural Language Processing, Proceedings of SustaiNLP).

Shrinking Bigfoot: Reducing wav2vec 2.0 footprint

Abstract

Publication series

Conference

UN SDGs

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this