Research diaries #6: Parallel in python

avatar

I haven't posted in ages. Today a code snippet for running stuff parallel

Python is one of those languages where you can really directly start working on solving your application instead of worrying about technical problems like how the code is excecuted. But sometimes it can be useful to think about how to let your code run quicker.

In most basic course the concept of running stuff in parallel is omitted because it not crucial to getting code to work. Parallel means that your run tasks at the same time. Your hardware can usually do this without any issues if your code is written in such a way that it know how to handle running it in parallel.

image.png

I think it is one the easiest way to finish the task quicker. And the method for this, because it is python is pretty simple. Let's go and code it up for a simple test problem.

We are going to need some modules:

from joblib import Parallel, delayed
import time, math

Let's define a simple function to run parallel

def my_fun(i):
    time.sleep(1)
    return i

Give it a quick test

num = 10
start = time.time()
for i in range(num):
    my_fun(i)
end = time.time()

If you print the end-start you will see this runs in about 10 seconds. But we if we now run it parallel:

start = time.time()
# n_jobs is the number of parallel jobs which has been set to 2
Parallel(n_jobs=2)(delayed(my_fun)(i) for i in range(num))
end = time.time()

we get an end-start time of about half of that: 5 seconds. Yay!

So what does the delayed exactly do? If you exclude it it will just run in 10 seconds. It seems like everything was run in the main thread! That's is exactly what happened. The delay makes sure that python halts the excution of my_fun and first orders all the tasks and then executes them parallel.

So unfortunately there are instances where parallel doesn't work because of how stuff works behind the scenes. This is the case for the neural network packages like keras. You will have to do extra work to get those to run parallel.

That's all for now.^^


Cat tax

cf8c66172dac6e3826efa41051318e4.jpg



0
0
0.000
8 comments
avatar

Hey! It's nice to see you post again. Hope you are well.

!discovery 31

0
0
0.000
avatar

I read the first half of this post while simultaneously reading the second half of the post and would have finished reading it in half the normal time but I didn't account for the cat tax so when I got there the system froze and I had to reboot my brain.

Welcome back!

0
0
0.000
avatar

Cats are disctraction professionals ^^

0
0
0.000
avatar

Thanks for your contribution to the STEMsocial community. Feel free to join us on discord to get to know the rest of us!

Please consider delegating to the @stemsocial account (85% of the curation rewards are returned).

You may also include @stemsocial as a beneficiary of the rewards of this post to get a stronger support. 
 

0
0
0.000