FlowchartQA: The First Large-Scale Benchmark for Reasoning over Flowcharts

Simon Tannert; Marcelo Feighelstein; Jasmina Bogojeska; Joseph Shtok; Assaf Arbelle; Peter W.J. Staar; Anika Schumann; Jonas Kuhn; Leonid Karlinsky

FlowchartQA: The First Large-Scale Benchmark for Reasoning over Flowcharts

Simon Tannert
, Marcelo Feighelstein
, Jasmina Bogojeska
, Joseph Shtok
, Assaf Arbelle
, Peter W.J. Staar
, Anika Schumann
, Jonas Kuhn
, Leonid Karlinsky

Research output: Contribution to conference › Paper › peer-review

5 Scopus citations

Abstract

In this paper, we present FlowchartQA, a new and unique large-scale benchmark for visual question answering (VQA) over flowcharts. FlowchartQA comprises close to 1M flowchart images and 6M question-answer pairs, covering various aspects of geometric and topological information contained in the charts. The questions have been carefully balanced to minimize biases. To accompany the proposed benchmark, we present a baseline model and perform comprehensive ablation studies and qualitative analyses to provide a solid foundation for future work. Our experimental results reveal interesting findings and demonstrate the potential of FlowchartQA as a testbed for flowchart understanding, which has been previously absent in the community.

Original language	English
Pages	34-46
Number of pages	13
State	Published - 1 Jan 2023
Externally published	Yes
Event	1st Workshop on Linguistic Insights from and for Multimodal Language Processing, LIMO 2023 - Ingolstadt, Germany Duration: 22 Sep 2023 → …

Conference

Conference	1st Workshop on Linguistic Insights from and for Multimodal Language Processing, LIMO 2023
Country/Territory	Germany
City	Ingolstadt
Period	22/09/23 → …

ASJC Scopus subject areas

Linguistics and Language
Language and Linguistics

Cite this

@conference{d2fa43299d414f08a809cb387f7e4ee8,

title = "FlowchartQA: The First Large-Scale Benchmark for Reasoning over Flowcharts",

abstract = "In this paper, we present FlowchartQA, a new and unique large-scale benchmark for visual question answering (VQA) over flowcharts. FlowchartQA comprises close to 1M flowchart images and 6M question-answer pairs, covering various aspects of geometric and topological information contained in the charts. The questions have been carefully balanced to minimize biases. To accompany the proposed benchmark, we present a baseline model and perform comprehensive ablation studies and qualitative analyses to provide a solid foundation for future work. Our experimental results reveal interesting findings and demonstrate the potential of FlowchartQA as a testbed for flowchart understanding, which has been previously absent in the community.",

author = "Simon Tannert and Marcelo Feighelstein and Jasmina Bogojeska and Joseph Shtok and Assaf Arbelle and Staar, \{Peter W.J.\} and Anika Schumann and Jonas Kuhn and Leonid Karlinsky",

note = "Publisher Copyright: {\textcopyright}2023 Association for Computational Linguistics.; 1st Workshop on Linguistic Insights from and for Multimodal Language Processing, LIMO 2023 ; Conference date: 22-09-2023",

year = "2023",

month = jan,

day = "1",

language = "English",

pages = "34--46",

}

TY - CONF

T1 - FlowchartQA

T2 - 1st Workshop on Linguistic Insights from and for Multimodal Language Processing, LIMO 2023

AU - Tannert, Simon

AU - Feighelstein, Marcelo

AU - Bogojeska, Jasmina

AU - Shtok, Joseph

AU - Arbelle, Assaf

AU - Staar, Peter W.J.

AU - Schumann, Anika

AU - Kuhn, Jonas

AU - Karlinsky, Leonid

PY - 2023/1/1

Y1 - 2023/1/1

N2 - In this paper, we present FlowchartQA, a new and unique large-scale benchmark for visual question answering (VQA) over flowcharts. FlowchartQA comprises close to 1M flowchart images and 6M question-answer pairs, covering various aspects of geometric and topological information contained in the charts. The questions have been carefully balanced to minimize biases. To accompany the proposed benchmark, we present a baseline model and perform comprehensive ablation studies and qualitative analyses to provide a solid foundation for future work. Our experimental results reveal interesting findings and demonstrate the potential of FlowchartQA as a testbed for flowchart understanding, which has been previously absent in the community.

AB - In this paper, we present FlowchartQA, a new and unique large-scale benchmark for visual question answering (VQA) over flowcharts. FlowchartQA comprises close to 1M flowchart images and 6M question-answer pairs, covering various aspects of geometric and topological information contained in the charts. The questions have been carefully balanced to minimize biases. To accompany the proposed benchmark, we present a baseline model and perform comprehensive ablation studies and qualitative analyses to provide a solid foundation for future work. Our experimental results reveal interesting findings and demonstrate the potential of FlowchartQA as a testbed for flowchart understanding, which has been previously absent in the community.

UR - https://www.scopus.com/pages/publications/85198217961

M3 - Paper

AN - SCOPUS:85198217961

SP - 34

EP - 46

Y2 - 22 September 2023

ER -

FlowchartQA: The First Large-Scale Benchmark for Reasoning over Flowcharts

Abstract

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this