What is parallel computing

1. What is parallel computing

Hi again! Now that you've learned everything about databases let's talk about parallel computing. In data engineering, you often have to pull in data from several sources and join them together, clean them, or aggregate them. In this video, we'll see how this is possible for massive amounts of data.

2. Idea behind parallel computing

Before we go into the different kinds of tools that exist in the data engineering ecosystem, it's crucial to understand the concept of parallel computing. Parallel computing forms the basis of almost all modern data processing tools. However, why has it become so important in the world of big data? The main reason is memory and processing power, but mostly memory. When big data processing tools perform a processing task, they split it up into several smaller subtasks and distribute those over several computers. These are usually commodity computers: widely available and relatively inexpensive. Any one of them would take a long time on the whole task, but working in parallel on smaller pieces, they finish it faster.

3. The tailor shop

Let's look at an analogy. Let's say you're running a tailor shop and need to get a batch of 100 shirts finished. Your very best tailor finishes a shirt in 20 minutes. Other tailors typically take 1 hour per shirt. If just one tailor can work at a time, it's obvious you'd have to choose the quickest tailor to finish the job. However, if you can split the batch in 25 shirts each, having 4 mediocre tailors working in parallel is faster. A similar thing happens for big data processing tasks.

4. Benefits of parallel computing

As you'd expect, the obvious benefit of having multiple processing units is the extra processing power itself. However, there is another, and potentially more impactful benefit of parallel computing for big data. Instead of needing to load all of the data in one computer's memory, you can partition the data and load the subsets into memory of different computers. That means the memory footprint per computer is relatively small, and the data can fit in the memory closest to the processor, the RAM.

5. Risks of parallel computing

Before you start rewriting all your code to use parallel computing, keep in mind that this also comes at its cost. Splitting a task into subtask and merging the results of the subtasks back into one final result requires some communication between processes. That overhead becomes a bottleneck if the task is small, or if you have few processing units. With 2 units, a task of a few hundred milliseconds isn't worth splitting up. And because of the overhead, speed does not increase linearly. This effect is called parallel slowdown.

6. An example

Let's look into a more practical example. We're starting with a dataset of all Olympic events from 1896 until 2016. From this dataset, you want to get an average age of participants for each year. Say you have four processing units at your disposal. To spread the load, you split the task into subtasks: one average age calculation per year, which a groupby gives you. Then you distribute those subtasks over the four units. This is roughly how the first distributed algorithms like Hadoop MapReduce work, except there the units sit on separate machines.

7. multiprocessing.Pool

In code, there are several ways of implementing this. At a low level, we could use the `multiprocessing.Pool` API to spread work over several cores on the same machine. Here, `take_mean_age` takes one year and its group of records, and hands back the mean age for that year. We map that function over the groups a groupby gives us, using the `.map()` method of `Pool`. The 4 we pass to `Pool` means the work runs in 4 separate processes, on 4 cores. Concatenating the results gives us the final DataFrame.

8. dask

Several packages spare us that low-level code. The `dask` framework offers a DataFrame that does the groupby and apply with multiprocessing out of the box. You define the number of partitions, 4 here, and `dask` takes the mean within each part separately. One catch: `dask` is lazy, so nothing happens until you add `.compute()` at the end of the chain.

9. Let's practice!

That was the final example of this video. In the exercises, you'll use the packages yourself. Good luck!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.