Python: parallelism using multiprocessing and multithreading

Have been working with Python since last few months. Was building some small modules. Quite excited about my new learnings. Recently I’d developed a distributed parallel data import process by following the generic Pub-Sub pattern. Earlier I’d published a 3 tier design of the same, developed using PHP. Comparative to that, it’s a simpler 2 tier design and this time I developed the process using Python. Have used multiprocessing and multithreading to achieve parallelism.

Selection_001

To elaborate on the post, it’s better to get the concepts clear-

  • Multitasking: Ability to execute more than one task at the same time is known as multitasking.
  • Multithreading: It is a process of executing multiple threads simultaneously. Multithreading is also known as Thread-based Multitasking.
  • Multiprocessing: It is same as multitasking, however in multiprocessing more than one CPUs are involved. On the other hand, one CPU is involved in multitasking.
  • Parallel Processing: It refers to the utilization of multiple CPUs in a single computer system.

Here, a basic example to explain how to achieve multiprocessing and multithreading using Python. I’ve kept 10 images in a directory and would resize them in 10 different sizes. The processing would be done using 4 processes and each process will use 2 threads in parallel.

$ ls -lth /var/tmp/source/
total 61M
-rwxrwxrwx 1 kuntal kuntal 6.1M Jun 26  2015 DSCN0922.JPG
-rwxrwxrwx 1 kuntal kuntal 5.9M Jun 26  2015 DSCN0911.JPG
-rwxrwxrwx 1 kuntal kuntal 5.5M Jun 26  2015 DSCN0900.JPG
-rwxrwxrwx 1 kuntal kuntal 6.2M Jun 26  2015 DSCN0867.JPG
-rwxrwxrwx 1 kuntal kuntal 6.0M Jun 26  2015 DSCN0848.JPG
-rwxrwxrwx 1 kuntal kuntal 5.6M Jun 26  2015 DSCN0829.JPG
-rwxrwxrwx 1 kuntal kuntal 6.9M Jun 26  2015 DSCN0825.JPG
-rwxrwxrwx 1 kuntal kuntal 6.6M Jun 26  2015 DSCN0819.JPG
-rwxrwxrwx 1 kuntal kuntal 5.9M Jun 26  2015 DSCN0798.JPG
-rwxrwxrwx 1 kuntal kuntal 6.8M Jun 26  2015 DSCN0791.JPG
$ time python image_resizer.py
Running task from process 11215
Running task from process 11216
Running task from process 11218
Running task from process 11217
Running task from process 11215
Resized images: /var/tmp/target/100-100DSCN0911.JPG, /var/tmp/target/200-200DSCN0911.JPG, /var/tmp/target/300-300DSCN0911.JPG, /var/tmp/target/400-400DSCN0911.JPG, /var/tmp/target/500-500DSCN0911.JPG, /var/tmp/target/600-600DSCN0911.JPG, /var/tmp/target/700-700DSCN0911.JPG, /var/tmp/target/800-800DSCN0911.JPG, /var/tmp/target/900-900DSCN0911.JPG, /var/tmp/target/1000-1000DSCN0911.JPG
Running task from process 11216
Resized images: /var/tmp/target/100-100DSCN0922.JPG, /var/tmp/target/200-200DSCN0922.JPG, /var/tmp/target/300-300DSCN0922.JPG, /var/tmp/target/400-400DSCN0922.JPG, /var/tmp/target/500-500DSCN0922.JPG, /var/tmp/target/600-600DSCN0922.JPG, /var/tmp/target/700-700DSCN0922.JPG, /var/tmp/target/800-800DSCN0922.JPG, /var/tmp/target/900-900DSCN0922.JPG, /var/tmp/target/1000-1000DSCN0922.JPG
Running task from process 11217
Running task from process 11218
Resized images: /var/tmp/target/100-100DSCN0791.JPG, /var/tmp/target/200-200DSCN0791.JPG, /var/tmp/target/300-300DSCN0791.JPG, /var/tmp/target/400-400DSCN0791.JPG, /var/tmp/target/500-500DSCN0791.JPG, /var/tmp/target/600-600DSCN0791.JPG, /var/tmp/target/700-700DSCN0791.JPG, /var/tmp/target/800-800DSCN0791.JPG, /var/tmp/target/900-900DSCN0791.JPG, /var/tmp/target/1000-1000DSCN0791.JPG
Resized images: /var/tmp/target/100-100DSCN0798.JPG, /var/tmp/target/200-200DSCN0798.JPG, /var/tmp/target/300-300DSCN0798.JPG, /var/tmp/target/400-400DSCN0798.JPG, /var/tmp/target/500-500DSCN0798.JPG, /var/tmp/target/600-600DSCN0798.JPG, /var/tmp/target/700-700DSCN0798.JPG, /var/tmp/target/800-800DSCN0798.JPG, /var/tmp/target/900-900DSCN0798.JPG, /var/tmp/target/1000-1000DSCN0798.JPG
Running task from process 11216
Running task from process 11215
Resized images: /var/tmp/target/100-100DSCN0825.JPG, /var/tmp/target/200-200DSCN0825.JPG, /var/tmp/target/300-300DSCN0825.JPG, /var/tmp/target/400-400DSCN0825.JPG, /var/tmp/target/500-500DSCN0825.JPG, /var/tmp/target/600-600DSCN0825.JPG, /var/tmp/target/700-700DSCN0825.JPG, /var/tmp/target/800-800DSCN0825.JPG, /var/tmp/target/900-900DSCN0825.JPG, /var/tmp/target/1000-1000DSCN0825.JPG
Resized images: /var/tmp/target/100-100DSCN0829.JPG, /var/tmp/target/200-200DSCN0829.JPG, /var/tmp/target/300-300DSCN0829.JPG, /var/tmp/target/400-400DSCN0829.JPG, /var/tmp/target/500-500DSCN0829.JPG, /var/tmp/target/600-600DSCN0829.JPG, /var/tmp/target/700-700DSCN0829.JPG, /var/tmp/target/800-800DSCN0829.JPG, /var/tmp/target/900-900DSCN0829.JPG, /var/tmp/target/1000-1000DSCN0829.JPG
Resized images: /var/tmp/target/100-100DSCN0819.JPG, /var/tmp/target/200-200DSCN0819.JPG, /var/tmp/target/300-300DSCN0819.JPG, /var/tmp/target/400-400DSCN0819.JPG, /var/tmp/target/500-500DSCN0819.JPG, /var/tmp/target/600-600DSCN0819.JPG, /var/tmp/target/700-700DSCN0819.JPG, /var/tmp/target/800-800DSCN0819.JPG, /var/tmp/target/900-900DSCN0819.JPG, /var/tmp/target/1000-1000DSCN0819.JPG
Resized images: /var/tmp/target/100-100DSCN0900.JPG, /var/tmp/target/200-200DSCN0900.JPG, /var/tmp/target/300-300DSCN0900.JPG, /var/tmp/target/400-400DSCN0900.JPG, /var/tmp/target/500-500DSCN0900.JPG, /var/tmp/target/600-600DSCN0900.JPG, /var/tmp/target/700-700DSCN0900.JPG, /var/tmp/target/800-800DSCN0900.JPG, /var/tmp/target/900-900DSCN0900.JPG, /var/tmp/target/1000-1000DSCN0900.JPG
Resized images: /var/tmp/target/100-100DSCN0848.JPG, /var/tmp/target/200-200DSCN0848.JPG, /var/tmp/target/300-300DSCN0848.JPG, /var/tmp/target/400-400DSCN0848.JPG, /var/tmp/target/500-500DSCN0848.JPG, /var/tmp/target/600-600DSCN0848.JPG, /var/tmp/target/700-700DSCN0848.JPG, /var/tmp/target/800-800DSCN0848.JPG, /var/tmp/target/900-900DSCN0848.JPG, /var/tmp/target/1000-1000DSCN0848.JPG
Resized images: /var/tmp/target/100-100DSCN0867.JPG, /var/tmp/target/200-200DSCN0867.JPG, /var/tmp/target/300-300DSCN0867.JPG, /var/tmp/target/400-400DSCN0867.JPG, /var/tmp/target/500-500DSCN0867.JPG, /var/tmp/target/600-600DSCN0867.JPG, /var/tmp/target/700-700DSCN0867.JPG, /var/tmp/target/800-800DSCN0867.JPG, /var/tmp/target/900-900DSCN0867.JPG, /var/tmp/target/1000-1000DSCN0867.JPG

real	0m23.358s
user	1m25.792s
sys	0m4.044s

Executed in a regular machine which has Memory: 7.7 GiB and Processor: Core i5-7200U CPU @ 2.50GHz × 4

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s