Code along video
Learning objectives
In this chapter, our goal is to simulate chemical reactions. We will build a reaction-diffusion simulator, in which a huge number of particles wander through space and react with each other when they meet. Simulations like this one become slow very quickly as the number of particles grows, and so we will learn a new tool to speed them up: parallel programming. Before we write any parallel code, this lesson builds our intuition. We will define parallelism, meet its often-confused cousin concurrency, see why computing becomes more parallel every year, and think about which problems are natural to split across many computers.
Code along summary
Setup
No code is needed for this lesson. Grab a pen and paper (or just your imagination) for the questions we ask along the way; we will start writing code in the next lesson.
Why parallel programming belongs in an introductory course
Parallel programming often waits until an advanced computer science course, and many students who take only one or two programming courses never see it at all. We think this is a missed opportunity, and the case for learning it early gets stronger every year. Many of the algorithms that are changing the world through AI are inherently parallel, and they run on computers that are inherently parallel. If you only ever take one programming course, it should give you at least a taste of what it means to make many processors work on a problem together.
What is parallelism?
STOP: Before reading on, how would you define parallel programming in a sentence?
Answers like “running code on multiple processors at the same time” are on the right track. We will take an even broader view, since parallelism is a concept that applies well beyond computers. Parallelism means dividing a task, or a collection of tasks, over multiple workers that work simultaneously. That adverb, simultaneously, is the heart of the idea.
For example, say that we have a big pile of mulch to spread over a garden. If we have friends to help, then we can divide the garden into sections and give one section to each person. The task is the same, but because the workers are spreading mulch at the same time, the job finishes sooner.
The same idea shows up all over computing. If you have used an AI assistant like Claude that reports it has launched several agents to work on your request, then you have watched parallelism in action: the assistant splits your task into pieces and hands each piece to a separate worker agent. Supercomputers are parallel too. What makes a supercomputer “super” is often less about any single processor being extraordinarily powerful and more about having many cores or machines that each work on part of a problem while coordinating with each other.
Why computing becomes more parallel every day
For decades, the main way to speed up programs was to wait for faster processors. That effort has run into diminishing returns: each individual processor is not getting faster at anything like the rate it once did. What has continued, however, is our ability to make processors smaller, more compact, and cheaper to manufacture. Your laptop very likely contains several processing cores, each of which can run instructions independently.
Put these two trends together. If each individual machine is not getting much faster, but we can afford more and more of them, then the only way to keep speeding up our programs is to split their work over many machines (or cores) at once. This is why parallelism is more relevant now than it has ever been.
Concurrency is a different idea
A second term that often appears in the same breath as parallelism is concurrency. If you have heard of threads, then you already have a good mental model for where we are headed, although we will not formally discuss threads in this course.
Many programmers treat concurrency as a synonym for parallelism, but the two concepts are distinct. They overlap a great deal in practice, yet we can have concurrency without parallelism, and we can have parallelism without concurrency. We will hold off on a precise definition of concurrency until we can see it in action in Python, and we will focus mostly on parallelism in this chapter. When we do return to concurrency, we will use analogies to make the difference between the two ideas feel straightforward.
Which problems are natural to parallelize?
To parallelize a problem means to convert a solution from running on one computer into one that takes advantage of many. Imagine that we hand you 100 computers instead of just one.
STOP: Which kinds of problems would be easy to solve faster with 100 computers instead of one? Which would be hard?
Protein folding is a tempting answer, but it depends on how we frame the problem. If we want to fold a single protein, it is not obvious how 100 computers would help; it is like having 100 hands reaching in to fold the same protein at once. Splitting a single fold across many machines is possible (projects like Folding@home do clever work along these lines), but it takes real ingenuity. If instead we want to fold 100 different proteins, then the answer is easy: run the same algorithm on each of the 100 computers, and give each one its own protein.
Many bioinformatics tasks have this same shape. If we want to scan 1,000 genomes for some motif, then each of our 100 computers can take 10 genomes. If we want to align many short fragments of DNA against a database, we can divide the fragments into groups and hand one group to each machine. When a large job consists of many independent smaller jobs, parallelizing it is natural.
Here are two more examples that will matter to us.
- Updating an array. Think of rendering the image on your screen, which is an array of pixels. With 16 computers, we could divide the screen into 16 rectangles of roughly equal size and make each computer responsible for one rectangle. This is the foundational idea behind graphics processing units (GPUs), which were designed for graphics long before they became the workhorses of AI.
- Monte Carlo simulation. If we want to run a million random trials and have 10 machines, we can give each machine 100,000 trials and combine their results at the end.
Exercise: Choose a simulation from an earlier chapter, such as the Game of Life or the election simulator. Describe in a few sentences how you would divide its work across four workers. Which pieces of work are independent of each other, and at what point would the workers need to combine or exchange their results?
Our plan for this chapter
Our big demo will be a particle simulator for chemical reactions, and the array-updating idea above will be central to parallelizing it. Before we get there, we will return to the foundations of the course and parallelize a Monte Carlo simulation, giving each core its share of the trials and then gathering the results.
But we will start with an even simpler problem: summing the values in a list. This task might seem too easy to study. We could write a for loop that adds each element to a running total in a few lines of code, or we could just call Python’s built-in sum(). Yet understanding how to split this simple task over many processors will teach us plenty, and everything we learn will carry over to the Monte Carlo and scientific simulations that follow.
Looking ahead
Now that we have a feel for what parallelism means and which problems split naturally across many workers, we are ready to write parallel code. In the next lesson, we will meet Python’s multiprocessing module and use it to sum a list by dividing the work across multiple processes.
