What is the difference between parallel programming and concurrent programming? There is a lot of definitions in the literature.
"Executing simultaneously" vs. "in progress at the same time"
For instance, The Art of Concurrency defines the difference as follows:
A system is said to be concurrent if it can support two or more actions in progress at the same time. A system is said to be parallel if it can support two or more actions executing simultaneously. The key concept and difference between these definitions is the phrase "in progress."
This definition says that, in concurrent systems, multiple actions can be in progress (may not be executed) at the same time. Meanwhile, multiple actions are simultaneously executed in parallel systems. In fact, concurrency and parallelism are conceptually overlapped to some degree, but "in progress" clearly makes them different.
Even though such definition is concrete and precise, it is not intuitive enough; we cannot easily imagine what "in progress" indicates.
Recently, I am taking an online course Parallel Programming on Coursera, and the course gave me more intuitive and easy-to-understand definition of parallel and concurrent programming as follows.
Parallel: Efficiency is its main concern
When we consider parallel programming, programs use parallel hardwares to execute computation more quickly.
Here, "parallel hardwares" could be:
- multi-core processors
- symmetric multiprocessors
- graphics processing unit (GPU)
- field-programmable gate arrays (FPGAs)
- computer clusters
in wide-ranging scale.
More concretely, parallel programming requires us to think about:
- How does code divide original huge problem into smaller sub-problems?
- Which is the optimal use of parallel hardware?
Since parallel programming strongly focuses on speeding-up computational time, applications such as matrix multiplication, data analysis, 3D rendering and particle simulation can be discussed in the paradigm.
Concurrent: Modularity, responsiveness and maintainability are important
In parallel programming, multiple actions are strictly executed at the same time to improve efficiency. By contrast, multiple actions are not necessarily to be executed simultaneously in concurrent programming because of user-side manageability.
Concurrency cares about beyond efficiency, and our main concern is:
- When can an execution start?
- How can information exchange occur?
- How does code manage access to shared resources?
Well-organized practical applications such as web server, user interface and database should be implement in the concurrent paradigm. Even if parallelism is lost to some degree, convenience behind systems is more important in concurrent programming.
Distinguishing parallelism from concurrency is important to seek an appropriate way to solve large scale problems, but they are considered interchangeably in reality. The definitions provided by the online course are tremendously valuable to figure out the very similar but different two paradigms.
- Google Cloud Dataflow and Its Positive Impact on Developer Productivity
- Completed Functional Programming in Scala Specialization on Coursera, Finally
- Comparison of Running Time of Cached/Uncached Spark RDD
Last updated: 2022-09-02
Author: Takuya Kitazawa
Takuya Kitazawa is a freelance software developer, minimalistic traveler, ultralight hiker & runner, and craft beer enthusiast. With a decade of experience at start-up companies and Big Tech ranging from full-stack/machine-learning engineering to data science to product management, I am currently working at the intersection of technological and social aspects of data-driven applications.
- Opinions are my own and do not represent the views of organizations I am/was belonging to.
- I am doing my best to ensure the accuracy and fair use of the information. However, there might be some errors, outdated information, or biased subjective statements because the main purpose of this blog is to jot down my personal thoughts as soon as possible before conducting an extensive investigation. Visitors understand the limitations and rely on any information at their own risk.
- That said, if there is any issue with the content, please contact me so I can take the necessary action.