Research diaries #6: Parallel in python

I haven't posted in ages. Today a code snippet for running stuff parallel

Python is one of those languages where you can really directly start working on solving your application instead of worrying about technical problems like how the code is excecuted. But sometimes it can be useful to think about how to let your code run quicker.

In most basic course the concept of running stuff in parallel is omitted because it not crucial to getting code to work. Parallel means that your run tasks at the same time. Your hardware can usually do this without any issues if your code is written in such a way that it know how to handle running it in parallel.

I think it is one the easiest way to finish the task quicker. And the method for this, because it is python is pretty simple. Let's go and code it up for a simple test problem.

We are going to need some modules:

from joblib import Parallel, delayed
import time, math

Let's define a simple function to run parallel

def my_fun(i):
    time.sleep(1)
    return i

Give it a quick test

num = 10
start = time.time()
for i in range(num):
    my_fun(i)
end = time.time()

If you print the end-start you will see this runs in about 10 seconds. But we if we now run it parallel:

start = time.time()
# n_jobs is the number of parallel jobs which has been set to 2
Parallel(n_jobs=2)(delayed(my_fun)(i) for i in range(num))
end = time.time()

we get an end-start time of about half of that: 5 seconds. Yay!

So what does the delayed exactly do? If you exclude it it will just run in 10 seconds. It seems like everything was run in the main thread! That's is exactly what happened. The delay makes sure that python halts the excution of my_fun and first orders all the tasks and then executes them parallel.

So unfortunately there are instances where parallel doesn't work because of how stuff works behind the scenes. This is the case for the neural network packages like keras. You will have to do extra work to get those to run parallel.

That's all for now.^^