Lexical Generalization Improves with Larger Models and Longer Training

Elron Bandel; Yoav Goldberg; Yanai Elazar

doi:10.18653/v1/2022.findings-emnlp.426

Lexical Generalization Improves with Larger Models and Longer Training

Elron Bandel
, Yoav Goldberg
, Yanai Elazar

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

While fine-tuned language models perform well on many tasks, they were also shown to rely on superficial surface features such as lexical overlap. Excessive utilization of such heuristics can lead to failure on challenging inputs. We analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension (using a novel contrastive dataset), and find that larger models are much less susceptible to adopting lexical overlap heuristics. We also find that longer training leads models to abandon lexical overlap heuristics. Finally, we provide evidence that the disparity between models size has its source in the pre-trained model.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics
Subtitle of host publication	EMNLP 2022
Editors	Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Publisher	Association for Computational Linguistics (ACL)
Pages	4427-4439
Number of pages	13
ISBN (Electronic)	9781959429432
DOIs	https://doi.org/10.18653/v1/2022.findings-emnlp.426
State	Published - 1 Jan 2022
Externally published	Yes
Event	2022 Findings of the Association for Computational Linguistics: EMNLP 2022 - Hybrid, Abu Dhabi, United Arab Emirates Duration: 7 Dec 2022 → 11 Dec 2022

Publication series

Name	Findings of the Association for Computational Linguistics: EMNLP 2022

Conference

Conference	2022 Findings of the Association for Computational Linguistics: EMNLP 2022
Country/Territory	United Arab Emirates
City	Hybrid, Abu Dhabi
Period	7/12/22 → 11/12/22

ASJC Scopus subject areas

Computational Theory and Mathematics
Computer Science Applications
Information Systems

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.18653/v1/2022.findings-emnlp.426

Cite this

Bandel, E., Goldberg, Y., & Elazar, Y. (2022). Lexical Generalization Improves with Larger Models and Longer Training. In Y. Goldberg, Z. Kozareva, & Y. Zhang (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2022 (pp. 4427-4439). (Findings of the Association for Computational Linguistics: EMNLP 2022). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.findings-emnlp.426

Bandel, Elron ; Goldberg, Yoav ; Elazar, Yanai. / Lexical Generalization Improves with Larger Models and Longer Training. Findings of the Association for Computational Linguistics: EMNLP 2022. editor / Yoav Goldberg ; Zornitsa Kozareva ; Yue Zhang. Association for Computational Linguistics (ACL), 2022. pp. 4427-4439 (Findings of the Association for Computational Linguistics: EMNLP 2022).

@inproceedings{3bf264b4ca164646bf1bef3eaaed2b81,

title = "Lexical Generalization Improves with Larger Models and Longer Training",

abstract = "While fine-tuned language models perform well on many tasks, they were also shown to rely on superficial surface features such as lexical overlap. Excessive utilization of such heuristics can lead to failure on challenging inputs. We analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension (using a novel contrastive dataset), and find that larger models are much less susceptible to adopting lexical overlap heuristics. We also find that longer training leads models to abandon lexical overlap heuristics. Finally, we provide evidence that the disparity between models size has its source in the pre-trained model.",

author = "Elron Bandel and Yoav Goldberg and Yanai Elazar",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 2022 Findings of the Association for Computational Linguistics: EMNLP 2022 ; Conference date: 07-12-2022 Through 11-12-2022",

year = "2022",

month = jan,

day = "1",

doi = "10.18653/v1/2022.findings-emnlp.426",

language = "English",

series = "Findings of the Association for Computational Linguistics: EMNLP 2022",

publisher = "Association for Computational Linguistics (ACL)",

pages = "4427--4439",

editor = "Yoav Goldberg and Zornitsa Kozareva and Yue Zhang",

booktitle = "Findings of the Association for Computational Linguistics",

address = "United States",

}

Bandel, E, Goldberg, Y & Elazar, Y 2022, Lexical Generalization Improves with Larger Models and Longer Training. in Y Goldberg, Z Kozareva & Y Zhang (eds), Findings of the Association for Computational Linguistics: EMNLP 2022. Findings of the Association for Computational Linguistics: EMNLP 2022, Association for Computational Linguistics (ACL), pp. 4427-4439, 2022 Findings of the Association for Computational Linguistics: EMNLP 2022, Hybrid, Abu Dhabi, United Arab Emirates, 7/12/22. https://doi.org/10.18653/v1/2022.findings-emnlp.426

Lexical Generalization Improves with Larger Models and Longer Training. / Bandel, Elron; Goldberg, Yoav; Elazar, Yanai.
Findings of the Association for Computational Linguistics: EMNLP 2022. ed. / Yoav Goldberg; Zornitsa Kozareva; Yue Zhang. Association for Computational Linguistics (ACL), 2022. p. 4427-4439 (Findings of the Association for Computational Linguistics: EMNLP 2022).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Lexical Generalization Improves with Larger Models and Longer Training

AU - Bandel, Elron

AU - Goldberg, Yoav

AU - Elazar, Yanai

PY - 2022/1/1

Y1 - 2022/1/1

N2 - While fine-tuned language models perform well on many tasks, they were also shown to rely on superficial surface features such as lexical overlap. Excessive utilization of such heuristics can lead to failure on challenging inputs. We analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension (using a novel contrastive dataset), and find that larger models are much less susceptible to adopting lexical overlap heuristics. We also find that longer training leads models to abandon lexical overlap heuristics. Finally, we provide evidence that the disparity between models size has its source in the pre-trained model.

AB - While fine-tuned language models perform well on many tasks, they were also shown to rely on superficial surface features such as lexical overlap. Excessive utilization of such heuristics can lead to failure on challenging inputs. We analyze the use of lexical overlap heuristics in natural language inference, paraphrase detection, and reading comprehension (using a novel contrastive dataset), and find that larger models are much less susceptible to adopting lexical overlap heuristics. We also find that longer training leads models to abandon lexical overlap heuristics. Finally, we provide evidence that the disparity between models size has its source in the pre-trained model.

UR - https://www.scopus.com/pages/publications/85149869602

U2 - 10.18653/v1/2022.findings-emnlp.426

DO - 10.18653/v1/2022.findings-emnlp.426

M3 - Conference contribution

AN - SCOPUS:85149869602

T3 - Findings of the Association for Computational Linguistics: EMNLP 2022

SP - 4427

EP - 4439

BT - Findings of the Association for Computational Linguistics

A2 - Goldberg, Yoav

A2 - Kozareva, Zornitsa

A2 - Zhang, Yue

PB - Association for Computational Linguistics (ACL)

T2 - 2022 Findings of the Association for Computational Linguistics: EMNLP 2022

Y2 - 7 December 2022 through 11 December 2022

ER -

Bandel E, Goldberg Y, Elazar Y. Lexical Generalization Improves with Larger Models and Longer Training. In Goldberg Y, Kozareva Z, Zhang Y, editors, Findings of the Association for Computational Linguistics: EMNLP 2022. Association for Computational Linguistics (ACL). 2022. p. 4427-4439. (Findings of the Association for Computational Linguistics: EMNLP 2022). doi: 10.18653/v1/2022.findings-emnlp.426

Lexical Generalization Improves with Larger Models and Longer Training

Abstract

Publication series

Conference

ASJC Scopus subject areas

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this