Supervisord: Parallel child process spawning and monitoring

Supervisord is a simple and popular choice for process monitoring. Supervisord‘s primary purpose is to create and manage processes based on data in its configuration file. It does this by creating subprocesses. Each subprocess spawned by supervisor is managed for the entirety of its lifetime by supervisord.

Installation

kuntal@server1:~$ sudo apt-get install supervisor

Once installed, check status
kuntal@server1:~$ sudo service supervisor status
is running

Set up config

Program configs should go here
kuntal@server1:~$ ls -lt /etc/supervisor/conf.d/
total 0

Config
[program:test-config]
#/etc/supervisor/conf.d/test.conf
[program:test-config]
command=/usr/bin/php doSomething.php --forever
directory=/var/www/test/scripts/
process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
stderr_logfile=/var/log/supervisor/test.err.log
stdout_logfile=/var/log/supervisor/test.out.log
numprocs=10
stopasgroup=true
stopsignal=KILL

The conf should be placed in /etc/supervisor/conf.d/

kuntal@server1:~$ sudo cp -r /var/www/test/scripts/*.conf /etc/supervisor/conf.d/.

Common error when supervisor is asked to spawn multiple processes in parallel

kuntal@server1:~$ sudo supervisorctl reread
ERROR: CANT_REREAD: %(process_num) must be present within process_name when numprocs > 1

Add process_name=%(program_name)s_%(process_num)02d in the above configuration to fix that.
Once configuration file is created and saved, inform Supervisor about new program through the supervisorctl command. At first Supervisor needs to look for any new or changed program configurations in the /etc/supervisor/conf.d directory

kuntal@server1:~$ sudo supervisorctl reread
test-config: available

Followed by telling it to enact any changes
kuntal@server1:~$ sudo supervisorctl update
test-config: added process group

A reload is required for any subsequent changes
kuntal@server1:~$ sudo supervisorctl reload
Restarted supervisord

Managing Programs

Enter the interactive mode. start, stop, status, tail all the basic required commands can be done from supervisorctl

kuntal@server1:~$ sudo supervisorctl
test-config:test-config_00 FATAL Exited too quickly (process log may have details)
test-config:test-config_01 FATAL Exited too quickly (process log may have details)
test-config:test-config_02 FATAL Exited too quickly (process log may have details)
test-config:test-config_03 FATAL Exited too quickly (process log may have details)

Checked the log /var/log/supervisor/test.err.log. There was typo in executable command. Fixed and restarted.

kuntal@server1:~$ sudo supervisorctl update
test-config: stopped
test-config: updated process group
kuntal@server1:~$
kuntal@server1:~$ sudo supervisorctl
supervisor> status
test-config:test-config_00 RUNNING pid 14067, uptime 0:12:48
test-config:test-config_01 RUNNING pid 14066, uptime 0:12:48
test-config:test-config_02 RUNNING pid 14069, uptime 0:12:48
test-config:test-config_03 RUNNING pid 14068, uptime 0:12:48
test-config:test-config_04 RUNNING pid 14063, uptime 0:12:48
test-config:test-config_05 RUNNING pid 14062, uptime 0:12:48
test-config:test-config_06 RUNNING pid 14065, uptime 0:12:48
test-config:test-config_07 RUNNING pid 14064, uptime 0:12:48
test-config:test-config_08 RUNNING pid 14061, uptime 0:12:48
test-config:test-config_09 RUNNING pid 14060, uptime 0:12:48
supervisor> tail -f test-config:test-config_00
==> Press Ctrl-C to exit <==
doing something...
^C
supervisor>
supervisor> stop test-config:test-config_00
test-config:test-config_00: stopped
supervisor> status test-config:test-config_00
test-config:test-config_00 STOPPED Mar 22 05:51 AM
supervisor> start test-config:test-config_00
test-config:test-config_00: started
supervisor> restart test-config:test-config_00
test-config:test-config_00: stopped
test-config:test-config_00: started

Set up alert

Most importantly supervisor should update if any of the processes die or get stopped. Supervisord does this using its event listeners and event notifications. It has specific protocol. The event notification protocol is based on communication via a subprocess stdin and stdout. Supervisor sends specially-formatted input to an event listener process stdin and expects specially-formatted output from an event listener’s stdout, forming a request-response cycle. A protocol agreed upon between supervisor and the listener’s implementer allows listeners to process event notifications.
Conf
;/etc/supervisor/supervisord.conf
[eventlistener:test-config-supervisor]
command=/usr/bin/php ./console test:notification-command
directory=/var/www/test/scripts/
events=PROCESS_STATE_STOPPED,PROCESS_STATE_EXITED,PROCESS_STATE_UNKNOWN,PROCESS_STATE_FATAL,TICK_60
numprocs=1
stopasgroup=true
stopsignal=KILL
autostart=true
autorestart=true
startretries=1
stderr_logfile=/var/log/supervisor/test.supervisor.err.log
stdout_logfile=/var/log/supervisor/test.supervisor.out.log

Notification program
Could be anything e.g. send a mail or trigger notification alerts. Just need to be sure it follows Event Notification Protocol. Common mistake is to print verbose output. This discussion is helpful on the same. The basic rule is write data to the stdout as per the protocol e.g. OK, READY, RESULT and so on.

[References: Supervisor: A Process Control System, How To Install and Manage Supervisor on Ubuntu and Debian VPS, Monitoring Processes with Supervisord, Event Listeners and Event Notifications]

Advertisements

One thought on “Supervisord: Parallel child process spawning and monitoring

  1. Pingback: Distributed Job Processing: RabbitMQ | Implementing Brute force

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s