RabbitMQ: Cluster setup and mirrored queue

This post answers 5 basic questions on RabbitMQ.

  • What is a RabbitMQ broker?
  • How to set up a cluster?
  • How to detach a node from a cluster?
  • Why queue replication is important?
  • How to configure mirroring?

RabbitMQ broker

A RabbitMQ broker is a logical grouping of one or several Erlang nodes, each running the RabbitMQ application and sharing users, virtual hosts, queues, exchanges, bindings, and runtime parameters. The collection of nodes, grouped together to build a single logical unit is referred as a cluster.

Set up a cluster

  1. Ensure that all nodes have the same Erlang cookie file.
  2. Hostname Resolution
  3. Create the cluster

Starting independent nodes
kuntal@server1:~$ rabbitmq-server -detached
kuntal@server2:~$ rabbitmq-server -detached

Copy cookie file
kuntal@server1:~$ scp /var/lib/rabbitmq/.erlang.cookie kuntal@server2:/var/lib/rabbitmq/.erlang.cookie

Set up proper permission
Each of the nodes must have correct owner, group and permissions of the file erlang.cookie
kuntal@server2:~$ sudo chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
kuntal@server2:~$ sudo chmod 400 /var/lib/rabbitmq/.erlang.cookie

Hostname Resolution
Update local host files (e.g. /etc/hosts)

Verify individual status
kuntal@server1:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@server1' ...
[{nodes,[{disc,['rabbit@server1']}]},
{running_nodes,['rabbit@server1']},
{partitions,[]}]
...done.
kuntal@server2:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@server2' ...
[{nodes,[{disc,['rabbit@server2']}]},
{running_nodes,['rabbit@server2']},
{partitions,[]}]
...done.

Creating the cluster
kuntal@server2:~$ sudo rabbitmqctl stop_app
Stopping node rabbit@server2 ...done.
kuntal@server2:~$ sudo rabbitmqctl join_cluster rabbit@server1
Clustering node rabbit@server2 with [rabbit@server1] ...done.
kuntal@server2:~$ sudo rabbitmqctl start_app
Starting node rabbit@server2 ...done.

Note: joining a cluster implicitly resets the node, thus removing all resources and data that were previously present on that node.
Use –ram parameter with join_cluster if the joining node should be a RAM node.

kuntal@server1:~$ sudo rabbitmqctl cluster_status
Cluster status of node rabbit@server1 ...
[{nodes,[{disc,[rabbit@server1,rabbit@server2]}]},
{running_nodes,[rabbit@server1]},
{partitions,[]}]
...done.

Check status of queues and messages
kuntal@server1:~$ sudo rabbitmqctl list_queues name consumers messages;
Listing queues ...
testMsgQueue 2 0
anotherQueue 3 4525
testQueue 0 0
anotherTest 0 0

Delete queue from cluster
kuntal@server1:~$ sudo /usr/local/bin/rabbitmqadmin delete queue name=queuename;
queue deleted

Detach a node from a cluster

kuntal@server2:~$ sudo rabbitmqctl stop_app
Stopping node rabbit@server2 ...done.
kuntal@server2:~$ sudo rabbitmqctl reset
Resetting node rabbit@server2 ...done.
kuntal@server2:~$ sudo rabbitmqctl start_app
Starting node rabbit@server2 ...done.

Check status
kuntal@server2:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@server2' ...
[{nodes,[{disc,['rabbit@server2']}]},
{running_nodes,['rabbit@server2']},
{partitions,[]}]
...done.

Queue replication

All data/state required for the operation of a RabbitMQ broker is replicated across all nodes. An exception to this are queues, which by default reside on one node, though they are visible and reachable from all nodes. Queue’s visibility and accessibility from all the nodes in a cluster doesn’t mean queues are replicated by default in cluster. [Related: Duplicate Queues Across Multiple Nodes Without Mirroring]

While exchanges and bindings survive the loss of individual nodes but queues and their messages do not because a queue and its contents are located on one node. If that node goes down, queues along with the unacknowledged messages will be lost.

Mirrored queue setup helps to achieve highly available queues. Each mirrored queue consists of one master and one or more slaves, with the oldest slave being promoted to the new master if the old master disappears for any reason.

Mirrored queues in RabbitMQ improve the availability of service since it is resilient to failures. A working cluster is mandatory before configuring mirror.

Configure mirroring

Set the ha-mode policy key as required by running the following command on one of the nodes.

kuntal@server1:~$ sudo rabbitmqctl set_policy ha-test "^test" \ '{"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"}'
Setting policy "ha-test" for pattern "^test" to " {\"ha-mode\":\"exactly\",\"ha-params\":2,\"ha-sync-mode\":\"automatic\"}" with priority "0" ...
...done.

Policy where queues whose names begin with “test” are mirrored to any two nodes in the cluster, with automatic synchronisation.
In exactly ha-mode Queue is mirrored to count nodes in the cluster. If there are less than count nodes in the cluster, the queue is mirrored to all nodes. If there are more than count nodes in the cluster, and a node containing a mirror goes down, then a new mirror will be created on another node.

Verify policy
The created policy should get listed in all of the nodes.
kuntal@server2:~$ sudo rabbitmqctl list_policies;
Listing policies ...
/ ha-test all ^test {"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"} 0
...done.

rabbit vhost error
If a node shows Virtual host error, then most probably that node has not been restarted after joining in the cluster.
kuntal@server2:~$ sudo rabbitmqctl list_policies;
Listing policies ...
Error: {aborted,{no_exists,[rabbit_vhost,<>]}}

kuntal@server2:~$ sudo rabbitmqctl status
kuntal@server2:~$ sudo rabbitmqctl stop_app
Stopping node rabbit@server2 ...
...done.
kuntal@server2:~$ sudo rabbitmqctl start_app
Starting node rabbit@server2 ...
...done.
kuntal@server2:~$ sudo rabbitmqctl list_policies;
Listing policies ...
/ ha-test all ^test {"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"} 0
...done.

Remove policy
kuntal@server1:~$ sudo rabbitmqctl clear_policy ha-test;
Clearing policy "ha-test" ...
...done.

Note: Redefining policy with same name, will overwrite the existing one.

I’ll come up with another post on how I’m using this setup.

[References: RabbitMQ ClusteringHighly Available QueuesMirrored queues behind a load balancerHighly Available Message Queues using RabbitMQLoad Balancing a RabbitMQ Cluster]

Advertisements

One thought on “RabbitMQ: Cluster setup and mirrored queue

  1. Pingback: Distributed Job Processing: RabbitMQ | Implementing Brute force

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s