1. What sort of problems benefit from parallel computing
You have multiple cores, congratulations. But how do you actually use them? Not everything in life benefits from having more resources. Just think about cooking a meal.
2. Cooking
An extra person or two to help out is great. But the helping hand can quickly turn into a hinderance.
Computing is no different. The vast majority of statistical methods
3. Running in parallel
just haven't been designed with parallel computing in mind. This is starting to change, but many standard methods just don't fit in the parallel paradigm. So what type of problems can exploit multi core CPUs? Suppose you want to
4. Monte-Carlo simulations
perform eight Monte-Carlo simulations. A standard way is to construct a for loop, and at each iteration you simulate a realization of the process. So my desktop machine has eight cores, so each core would simulate a single realization. Once all the cores are finished. We would combine the results.
This is the easiest type of parallel computing and is known as embarrassingly parallel, since little effort is needed to separate the problem into separate tasks. Let's consider a different loop.
5. Not everything runs in parallel
This loop sets the value of x[i] to the previous value. After running the loop, all values of the vector x are equal to 1. If we had eight cores, could we run this loop in parallel? The short, sad answer is no. In parallel computing we can't guarantee the order of operations. So if core 4 was executed first, it would need the value of x[3] which is currently 3 and not 1.
6. Rule of thumb
A general rule of thumb for determining if a for loop can be run in parallel is to think about running the loop backwards. In our Monte Carlo loop we can go forward or backwards and still get the same result.
7. Rule of thumb
But in this example, if we reverse the order of the loop, the calculation is wrong.
So remember, if you can write your loop in reverse, there's a good chance you can use multi-core computing.
8. Let's practice!
Now let's practice