Research diaries #6: Parallel in python

in StemSocial5 months ago

I haven't posted in ages. Today a code snippet for running stuff parallel

Python is one of those languages where you can really directly start working on solving your application instead of worrying about technical problems like how the code is excecuted. But sometimes it can be useful to think about how to let your code run quicker.

In most basic course the concept of running stuff in parallel is omitted because it not crucial to getting code to work. Parallel means that your run tasks at the same time. Your hardware can usually do this without any issues if your code is written in such a way that it know how to handle running it in parallel.


I think it is one the easiest way to finish the task quicker. And the method for this, because it is python is pretty simple. Let's go and code it up for a simple test problem.

We are going to need some modules:

from joblib import Parallel, delayed
import time, math

Let's define a simple function to run parallel

def my_fun(i):
    return i

Give it a quick test

num = 10
start = time.time()
for i in range(num):
end = time.time()

If you print the end-start you will see this runs in about 10 seconds. But we if we now run it parallel:

start = time.time()
# n_jobs is the number of parallel jobs which has been set to 2
Parallel(n_jobs=2)(delayed(my_fun)(i) for i in range(num))
end = time.time()

we get an end-start time of about half of that: 5 seconds. Yay!

So what does the delayed exactly do? If you exclude it it will just run in 10 seconds. It seems like everything was run in the main thread! That's is exactly what happened. The delay makes sure that python halts the excution of my_fun and first orders all the tasks and then executes them parallel.

So unfortunately there are instances where parallel doesn't work because of how stuff works behind the scenes. This is the case for the neural network packages like keras. You will have to do extra work to get those to run parallel.

That's all for now.^^

Cat tax



I read the first half of this post while simultaneously reading the second half of the post and would have finished reading it in half the normal time but I didn't account for the cat tax so when I got there the system froze and I had to reboot my brain.

Welcome back!

Cats are disctraction professionals ^^

 5 months ago 

Hey! It's nice to see you post again. Hope you are well.

!discovery 31

Owl good here :D

This is a great tool for a big work


You have received a 1UP from @gwajnberg!

The @oneup-cartel will soon upvote you with:
@stem-curator, @vyb-curator, @pob-curator, @neoxag-curator
And they will bring !PIZZA 🍕.

Learn more about our delegation service to earn daily rewards. Join the Cartel on Discord.

This post was shared and voted inside the discord by the curators team of discovery-it
Join our community! hive-193212
Discovery-it is also a Witness, vote for us here
Delegate to us for passive income. Check our 80% fee-back Program

Thanks for your contribution to the STEMsocial community. Feel free to join us on discord to get to know the rest of us!

Please consider delegating to the @stemsocial account (85% of the curation rewards are returned).

You may also include @stemsocial as a beneficiary of the rewards of this post to get a stronger support.