TY - GEN
T1 - Advising OpenMP Parallelization via A Graph-Based Approach with Transformers
AU - Kadosh, Tal
AU - Schneider, Nadav
AU - Hasabnis, Niranjan
AU - Mattson, Timothy
AU - Pinter, Yuval
AU - Oren, Gal
N1 - Publisher Copyright:
© 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.
PY - 2023/1/1
Y1 - 2023/1/1
N2 - There is an ever-present need for shared memory parallelization schemes to exploit the full potential of multi-core architectures. The most common parallelization API addressing this need today is OpenMP. Nevertheless, writing parallel code manually is complex and effort-intensive. Thus, many deterministic source-to-source (S2S) compilers have emerged, intending to automate the process of translating serial to parallel code. However, recent studies have shown that these compilers are impractical in many scenarios. In this work, we combine the latest advancements in the field of AI and natural language processing (NLP) with the vast amount of open-source code to address the problem of automatic parallelization. Specifically, we propose a novel approach, called OMPify, to detect and predict the OpenMP pragmas and shared-memory attributes in parallel code, given its serial version. OMPify is based on a Transformer-based model that leverages a graph-based representation of source code that exploits the inherent structure of code. We evaluated our tool by predicting the parallelization pragmas and attributes of a large corpus of (over 54,000) snippets of serial code written in C and C++ languages (Open-OMP-Plus). Our results demonstrate that OMPify outperforms existing approaches — the general-purposed and popular ChatGPT and targeted PragFormer models — in terms of F1 score and accuracy. Specifically, OMPify achieves up to 90% accuracy on commonly-used OpenMP benchmark tests such as NAS, SPEC, and PolyBench. Additionally, we performed an ablation study to assess the impact of different model components and present interesting insights derived from the study. Lastly, we also explored the potential of using data augmentation and curriculum learning techniques to improve the model’s robustness and generalization capabilities. The dataset and source code necessary for reproducing our results are available at https://github.com/Scientific-Computing-Lab-NRCN/OMPify.
AB - There is an ever-present need for shared memory parallelization schemes to exploit the full potential of multi-core architectures. The most common parallelization API addressing this need today is OpenMP. Nevertheless, writing parallel code manually is complex and effort-intensive. Thus, many deterministic source-to-source (S2S) compilers have emerged, intending to automate the process of translating serial to parallel code. However, recent studies have shown that these compilers are impractical in many scenarios. In this work, we combine the latest advancements in the field of AI and natural language processing (NLP) with the vast amount of open-source code to address the problem of automatic parallelization. Specifically, we propose a novel approach, called OMPify, to detect and predict the OpenMP pragmas and shared-memory attributes in parallel code, given its serial version. OMPify is based on a Transformer-based model that leverages a graph-based representation of source code that exploits the inherent structure of code. We evaluated our tool by predicting the parallelization pragmas and attributes of a large corpus of (over 54,000) snippets of serial code written in C and C++ languages (Open-OMP-Plus). Our results demonstrate that OMPify outperforms existing approaches — the general-purposed and popular ChatGPT and targeted PragFormer models — in terms of F1 score and accuracy. Specifically, OMPify achieves up to 90% accuracy on commonly-used OpenMP benchmark tests such as NAS, SPEC, and PolyBench. Additionally, we performed an ablation study to assess the impact of different model components and present interesting insights derived from the study. Lastly, we also explored the potential of using data augmentation and curriculum learning techniques to improve the model’s robustness and generalization capabilities. The dataset and source code necessary for reproducing our results are available at https://github.com/Scientific-Computing-Lab-NRCN/OMPify.
KW - Code Completion
KW - Code Representations
KW - NLP
KW - OpenMP
KW - S2S Compilers
KW - Shared Memory Parallelism
KW - Transformers
UR - http://www.scopus.com/inward/record.url?scp=85172125841&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-40744-4_1
DO - 10.1007/978-3-031-40744-4_1
M3 - Conference contribution
AN - SCOPUS:85172125841
SN - 9783031407437
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 3
EP - 17
BT - OpenMP
A2 - McIntosh-Smith, Simon
A2 - Deakin, Tom
A2 - Klemm, Michael
A2 - de Supinski, Bronis R.
A2 - Klinkenberg, Jannis
PB - Springer Science and Business Media Deutschland GmbH
T2 - Proceedings of the 19th International Workshop on OpenMP, IWOMP 2023
Y2 - 13 September 2023 through 15 September 2023
ER -