RabbitMQ: Cluster setup and mirrored queue

This post answers 5 basic questions on RabbitMQ.

  • What is a RabbitMQ broker?
  • How to set up a cluster?
  • How to detach a node from a cluster?
  • Why queue replication is important?
  • How to configure mirroring?

RabbitMQ broker

A RabbitMQ broker is a logical grouping of one or several Erlang nodes, each running the RabbitMQ application and sharing users, virtual hosts, queues, exchanges, bindings, and runtime parameters. The collection of nodes, grouped together to build a single logical unit is referred as a cluster.

Set up a cluster

  1. Ensure that all nodes have the same Erlang cookie file.
  2. Hostname Resolution
  3. Create the cluster

Starting independent nodes
kuntal@server1:~$ rabbitmq-server -detached
kuntal@server2:~$ rabbitmq-server -detached

Copy cookie file
kuntal@server1:~$ scp /var/lib/rabbitmq/.erlang.cookie kuntal@server2:/var/lib/rabbitmq/.erlang.cookie

Set up proper permission
Each of the nodes must have correct owner, group and permissions of the file erlang.cookie
kuntal@server2:~$ sudo chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
kuntal@server2:~$ sudo chmod 400 /var/lib/rabbitmq/.erlang.cookie

Hostname Resolution
Update local host files (e.g. /etc/hosts)

Verify individual status
kuntal@server1:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@server1' ...
[{nodes,[{disc,['rabbit@server1']}]},
{running_nodes,['rabbit@server1']},
{partitions,[]}]
...done.
kuntal@server2:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@server2' ...
[{nodes,[{disc,['rabbit@server2']}]},
{running_nodes,['rabbit@server2']},
{partitions,[]}]
...done.

Creating the cluster
kuntal@server2:~$ sudo rabbitmqctl stop_app
Stopping node rabbit@server2 ...done.
kuntal@server2:~$ sudo rabbitmqctl join_cluster rabbit@server1
Clustering node rabbit@server2 with [rabbit@server1] ...done.
kuntal@server2:~$ sudo rabbitmqctl start_app
Starting node rabbit@server2 ...done.

Note: joining a cluster implicitly resets the node, thus removing all resources and data that were previously present on that node.
Use –ram parameter with join_cluster if the joining node should be a RAM node.

kuntal@server1:~$ sudo rabbitmqctl cluster_status
Cluster status of node rabbit@server1 ...
[{nodes,[{disc,[rabbit@server1,rabbit@server2]}]},
{running_nodes,[rabbit@server1]},
{partitions,[]}]
...done.

Check status of queues and messages
kuntal@server1:~$ sudo rabbitmqctl list_queues name consumers messages messages_ready messages_unacknowledged
Listing queues ...
amq.gen-QpH_mAGLAfcK9RDesq-7Xw 0 0 0 0
amq.gen-cJHYJpqVVpgdtwSHTzBS2g 0 0 0 0
amq.gen-rDbxpsjGtK73EMcwBZ3mRA 0 0 0 0
test_mailer 0 6 6 0
message_send 0 0 0 0
test_msg_queue 0 0 0 0
wont_reveal_queue_name 0 0 0 0
test_queue_123 0 3515 3515 0
test_queue_321 2 29 27 2
...done.
kuntal@server1:~$

Delete queue from cluster
kuntal@server1:~$ sudo /usr/local/bin/rabbitmqadmin delete queue name=queuename;
queue deleted

Detach a node from a cluster

kuntal@server2:~$ sudo rabbitmqctl stop_app
Stopping node rabbit@server2 ...done.
kuntal@server2:~$ sudo rabbitmqctl reset
Resetting node rabbit@server2 ...done.
kuntal@server2:~$ sudo rabbitmqctl start_app
Starting node rabbit@server2 ...done.

Check status
kuntal@server2:~$ sudo rabbitmqctl cluster_status
Cluster status of node 'rabbit@server2' ...
[{nodes,[{disc,['rabbit@server2']}]},
{running_nodes,['rabbit@server2']},
{partitions,[]}]
...done.

Queue replication

All data/state required for the operation of a RabbitMQ broker is replicated across all nodes. An exception to this are queues, which by default reside on one node, though they are visible and reachable from all nodes. Queue’s visibility and accessibility from all the nodes in a cluster doesn’t mean queues are replicated by default in cluster. [Related: Duplicate Queues Across Multiple Nodes Without Mirroring]

While exchanges and bindings survive the loss of individual nodes but queues and their messages do not because a queue and its contents are located on one node. If that node goes down, queues along with the unacknowledged messages will be lost.

Mirrored queue setup helps to achieve highly available queues. Each mirrored queue consists of one master and one or more slaves, with the oldest slave being promoted to the new master if the old master disappears for any reason.

Mirrored queues in RabbitMQ improve the availability of service since it is resilient to failures. A working cluster is mandatory before configuring mirror.

Configure mirroring

Set the ha-mode policy key as required by running the following command on one of the nodes.

kuntal@server1:~$ sudo rabbitmqctl set_policy ha-test "^test" \ '{"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"}'
Setting policy "ha-test" for pattern "^test" to " {\"ha-mode\":\"exactly\",\"ha-params\":2,\"ha-sync-mode\":\"automatic\"}" with priority "0" ...
...done.

Policy where queues whose names begin with “test” are mirrored to any two nodes in the cluster, with automatic synchronisation.
In exactly ha-mode Queue is mirrored to count nodes in the cluster. If there are less than count nodes in the cluster, the queue is mirrored to all nodes. If there are more than count nodes in the cluster, and a node containing a mirror goes down, then a new mirror will be created on another node.

Verify policy
The created policy should get listed in all of the nodes.
kuntal@server2:~$ sudo rabbitmqctl list_policies;
Listing policies ...
/ ha-test all ^test {"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"} 0
...done.

rabbit vhost error
If a node shows Virtual host error, then most probably that node has not been restarted after joining in the cluster.
kuntal@server2:~$ sudo rabbitmqctl list_policies;
Listing policies ...
Error: {aborted,{no_exists,[rabbit_vhost,]}}

kuntal@server2:~$ sudo rabbitmqctl status
kuntal@server2:~$ sudo rabbitmqctl stop_app
Stopping node rabbit@server2 ...
...done.
kuntal@server2:~$ sudo rabbitmqctl start_app
Starting node rabbit@server2 ...
...done.
kuntal@server2:~$ sudo rabbitmqctl list_policies;
Listing policies ...
/ ha-test all ^test {"ha-mode":"exactly","ha-params":2,"ha-sync-mode":"automatic"} 0
...done.

Remove policy
kuntal@server1:~$ sudo rabbitmqctl clear_policy ha-test;
Clearing policy "ha-test" ...
...done.

Note: Redefining policy with same name, will overwrite the existing one.

I’ll come up with another post on how I’m using this setup.

[References: RabbitMQ ClusteringHighly Available QueuesMirrored queues behind a load balancerHighly Available Message Queues using RabbitMQLoad Balancing a RabbitMQ Cluster]

Advertisements

3 thoughts on “RabbitMQ: Cluster setup and mirrored queue

  1. we are exploring the queue-master-locator ‘min-masters’ policy, and it looks like it works OK when we create new Queues, but when for some reason we need to stop one node of the cluster, every queues existing on that node get promoted (master) to the same node, for example:

    Node A has 30 queues Node B has 0 queues Node C has 2 queues Node D has 3 queues

    When we stop Node A, all 30 queues get promoted to Node B, it is that the expected result?. We was hoping that the 30 queues would be distribuited across Node B, C, and D…

    Please we are starting to get crazy arround this. Has everyone experimented this scenario? and It is posible to achieve what we are expecting to happen when shutdown Node A in some automatic way?

    Our policy is defined as follow:

    Listing policies … prod ha queues ^ {“ha-mode”:”exactly”,”ha-params”:3,”ha-sync-mode”:”automatic”,”queue-master-locator”:”min-masters”}

    We have 4 nodes in the cluster, 2 RAM nodes and 2 Disk Nodes.

    The policy works just fine when we create new queues but it does nothing when we stop one node.

    Thanks

    Like

    • Hi Pablo,

      As per the policy you explained, it’s Mirroring policy. You should check out Highly Available (Mirrored) Queues https://www.rabbitmq.com/ha.html

      1. When we stop Node A, all 30 queues get promoted to Node B, it is that the expected result?. We was hoping that the 30 queues would be distribuited across Node B, C, and D…
      — Each mirrored queue consists of one master and one or more mirrors, with the oldest mirror being promoted to the new master if the old master disappears for any reason.

      2. Has everyone experimented this scenario?
      — That should be the expected scenario

      3. prod ha queues ^ {“ha-mode”:”exactly”,”ha-params”:3,”ha-sync-mode”:”automatic”,”queue-master-locator”:”min-masters”}
      — A count value of 2 means 1 queue master and 1 queue mirror. If the node running the queue master becomes unavailable, the queue mirror will be automatically promoted to master. In conclusion: NumberOfQueueMirrors = NumberOfNodes – 1.

      Like

  2. Pingback: Distributed Job Processing: RabbitMQ | Implementing Brute force

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s