The tao of parallelism in algorithms

Keshav Pingali, Donald Nguyen, Milind Kulkarni, Martin Burtscher, M. Amber Hassaan, Rashid Kaleem, Tsung Hsien Lee, Andrew Lenharth, Roman Manevich, Mario Méndez-Lojo, Dimitrios Prountzos, Xin Sui

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

296 Scopus citations


For more than thirty years, the parallel programming community has used the dependence graph as the main abstraction for reasoning about and exploiting parallelism in "regular" algorithms that use dense arrays, such as finite-differences and FFTs. In this paper, we argue that the dependence graph is not a suitable abstraction for algorithms in new application areas like machine learning and network analysis in which the key data structures are "irregular" data structures like graphs, trees, and sets. To address the need for better abstractions, we introduce a data-centric formulation of algorithms called the operator formulation in which an algorithm is expressed in terms of its action on data structures. This formulation is the basis for a structural analysis of algorithms that we call tao-analysis. Tao-analysis can be viewed as an abstraction of algorithms that distills out algorithmic properties important for parallelization. It reveals that a generalized form of data-parallelism called amorphous data-parallelism is ubiquitous in algorithms, and that, depending on the tao-structure of the algorithm, this parallelism may be exploited by compile-time, inspector-executor or optimistic parallelization, thereby unifying these seemingly unrelated parallelization techniques. Regular algorithms emerge as a special case of irregular algorithms, and many application-specific optimization techniques can be generalized to a broader context. These results suggest that the operator formulation and tao-analysis of algorithms can be the foundation of a systematic approach to parallel programming.

Original languageEnglish
Title of host publicationPLDI'11 - Proceedings of the 2011 ACM Conference on Programming Language Design and Implementation
Number of pages14
StatePublished - 7 Jul 2011
Externally publishedYes
Event32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI'11 - San Jose, CA, United States
Duration: 4 Jun 20118 Jun 2011

Publication series

NameProceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI)


Conference32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI'11
Country/TerritoryUnited States
CitySan Jose, CA


  • amorphous data-parallelism
  • galois system
  • irregular programs
  • operator formulation
  • tao-analysis

ASJC Scopus subject areas

  • Software


Dive into the research topics of 'The tao of parallelism in algorithms'. Together they form a unique fingerprint.

Cite this