TY - GEN
T1 - Breaking NLI systems with sentences that require simple lexical inferences
AU - Glockner, Max
AU - Shwartz, Vered
AU - Goldberg, Yoav
N1 - Publisher Copyright:
© 2018 Association for Computational Linguistics
PY - 2018/1/1
Y1 - 2018/1/1
N2 - We create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. The new examples are simpler than the SNLI test set, containing sentences that differ by at most one word from sentences in the training set. Yet, the performance on the new test set is substantially worse across systems trained on SNLI, demonstrating that these systems are limited in their generalization ability, failing to capture many simple inferences.
AB - We create a new NLI test set that shows the deficiency of state-of-the-art models in inferences that require lexical and world knowledge. The new examples are simpler than the SNLI test set, containing sentences that differ by at most one word from sentences in the training set. Yet, the performance on the new test set is substantially worse across systems trained on SNLI, demonstrating that these systems are limited in their generalization ability, failing to capture many simple inferences.
UR - https://www.scopus.com/pages/publications/85061356150
U2 - 10.18653/v1/p18-2103
DO - 10.18653/v1/p18-2103
M3 - Conference contribution
AN - SCOPUS:85061356150
T3 - ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)
SP - 650
EP - 655
BT - ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Short Papers)
PB - Association for Computational Linguistics (ACL)
T2 - 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018
Y2 - 15 July 2018 through 20 July 2018
ER -