Parallel Programming vs. Concurrent Programming

Home > Blog > Parallel Programming vs. Concurrent Programming

2017-02-25

Parallel Programming vs. Concurrent Programming

What is the difference between parallel programming and concurrent programming? There is a lot of definitions in the literature.

"Executing simultaneously" vs. "in progress at the same time"

For instance, The Art of Concurrency defines the difference as follows:

A system is said to be concurrent if it can support two or more actions in progress at the same time. A system is said to be parallel if it can support two or more actions executing simultaneously. The key concept and difference between these definitions is the phrase "in progress."

This definition says that, in concurrent systems, multiple actions can be in progress (may not be executed) at the same time. Meanwhile, multiple actions are simultaneously executed in parallel systems. In fact, concurrency and parallelism are conceptually overlapped to some degree, but "in progress" clearly makes them different.

Even though such definition is concrete and precise, it is not intuitive enough; we cannot easily imagine what "in progress" indicates.

Recently, I am taking an online course Parallel Programming on Coursera, and the course gave me more intuitive and easy-to-understand definition of parallel and concurrent programming as follows.

Parallel: Efficiency is its main concern

When we consider parallel programming, programs use parallel hardwares to execute computation more quickly.

Here, "parallel hardwares" could be:

multi-core processors
symmetric multiprocessors
graphics processing unit (GPU)
field-programmable gate arrays (FPGAs)
computer clusters

in wide-ranging scale.

More concretely, parallel programming requires us to think about:

How does code divide original huge problem into smaller sub-problems?
Which is the optimal use of parallel hardware?

Since parallel programming strongly focuses on speeding-up computational time, applications such as matrix multiplication, data analysis, 3D rendering and particle simulation can be discussed in the paradigm.

Concurrent: Modularity, responsiveness and maintainability are important

In parallel programming, multiple actions are strictly executed at the same time to improve efficiency. By contrast, multiple actions are not necessarily to be executed simultaneously in concurrent programming because of user-side manageability.

Concurrency cares about beyond efficiency, and our main concern is:

When can an execution start?
How can information exchange occur?
How does code manage access to shared resources?

Well-organized practical applications such as web server, user interface and database should be implement in the concurrent paradigm. Even if parallelism is lost to some degree, convenience behind systems is more important in concurrent programming.

Distinguishing parallelism from concurrency is important to seek an appropriate way to solve large scale problems, but they are considered interchangeably in reality. The definitions provided by the online course are tremendously valuable to figure out the very similar but different two paradigms.

Author: Takuya Kitazawa

Takuya Kitazawa is a freelance software developer based in British Columbia, Canada. As a technologist specializing in AI and data-driven solutions, he has worked globally at Big Tech and start-up companies for a decade. At the intersection of tech and society, he is passionate about promoting the ethical use of information technologies through his mentoring, business consultation, and public engagement activities. See CV for more information, or contact at [email protected].

Now Gift a cup of coffee

Disclaimer

Opinions are my own and do not represent the views of organizations I am/was belonging to.
I am doing my best to ensure the accuracy and fair use of the information. However, there might be some errors, outdated information, or biased subjective statements because the main purpose of this blog is to jot down my personal thoughts as soon as possible before conducting an extensive investigation. Visitors understand the limitations and rely on any information at their own risk.
That said, if there is any issue with the content, please contact me so I can take the necessary action.