Meisha, just like me, is a fan of Tom and Jerry. I have a good collection of Tom and Jerry videos. The only problem was all those videos are in mkv format and she had to open her laptop to watch those videos every time. I realised the need to convert all of those videos to avi format so that they can be played from the TV. Wrote a bash script to do the conversion for me. Used xargs for parallel processing; ultimately there is a good number of videos and time is precious. That’s all. Check out here, if you have any requirement of such conversion.
Have been working with Python since last few months. Was building some small modules. Quite excited about my new learnings. Recently I’d developed a distributed parallel data import process by following the generic Pub-Sub pattern. Earlier I’d published a 3 tier design of the same, developed using PHP. Comparative to that, it’s a simpler 2 tier design and this time I developed the process using Python. Have used multiprocessing and multithreading to achieve parallelism.
I have been working for a distributed process. In generic term this is a process which needs to run continuously in a distributed manner by following push-pull communication. RabbitMQ is a great tool to achieve this solution. I’ve extended worker model to integrate publisher along with the same. Continue reading
Supervisord is a simple and popular choice for process monitoring. Supervisord‘s primary purpose is to create and manage processes based on data in its configuration file. It does this by creating subprocesses. Each subprocess spawned by supervisor is managed for the entirety of its lifetime by supervisord. Continue reading
Last week I had a requirement to fetch details of some records. For simplicity let’s assume records are id of some entity; I need to get details from an API for those id. Let’s also assume that API accepts 100 id at maximum in a single request. I’d almost 1 million id to fetch details.
I love concurrency. Writing a concurrent program is always complex. There are different technologies which are most suitable to do this job. In an earlier post I briefed how to achieve parallelization using PHP. Parallel processing can be done using shell script also; by spawning multiple child processes. It’s fairly simple to handle basic concurrent tasks in that manner.
A basic example to achieve the same-
There are situations when we need to make our one-process task into a multi-process task so that we can take advantage of the operating system’s multitasking capabilities.
The basic conceptual difference between multiprocessing and multithreading– A process, is a unique instance of a program with its own memory space, own process ID number etc. Whereas a thread can be thought of as a virtual process, it does not have its own process ID, does not have its own memory space, but is still able to take advantage of multitasking.