The High Availability (HA) module provides active-passive, hot-standby messaging clusters to provide fault tolerant message delivery.

In an active-passive cluster only one broker, known as the primary, is active and serving clients at a time. The other brokers are standing by as backups. Changes on the primary are replicated to all the backups so they are always up-to-date or "hot". Backup brokers reject client connection attempts, to enforce the requirement that clients only connect to the primary.

If the primary fails, one of the backups is promoted to take over as the new primary. Clients fail-over to the new primary automatically. If there are multiple backups, the other backups also fail-over to become backups of the new primary.

This approach relies on an external cluster resource manager to detect failures, choose the new primary and handle network partitions. rgmanager is supported initially, but others may be supported in the future.

Some resource managers (including rgmanager) support virtual IP addresses. A virtual IP address is an IP address that can be relocated to any of the nodes in a cluster. The resource manager associates this address with the primary node in the cluster, and relocates it to the new primary when there is a failure. This simplifies configuration as you can publish a single IP address rather than a list.

A virtual IP address can be used by clients to connect to the primary. The following sections will explain how to configure virtual IP addresses for clients or brokers.

The broker must load the ha module, it is loaded by default. The following broker options are available for the HA module.

url = ["amqp:"][ user ["/" password] "@" ] addr ("," addr)*
addr = tcp_addr / rmda_addr / ssl_addr / ...
tcp_addr = ["tcp:"] host [":" port]
rdma_addr = "rdma:" host [":" port]
ssl_addr = "ssl:" host [":" port]'
		  

To configure a HA cluster you must set at least ha-cluster and ha-brokers-url.

Broker fail-over is managed by a cluster resource manager. An integration with rgmanager is provided, but it is possible to integrate with other resource managers.

The resource manager is responsible for starting the qpidd broker on each node in the cluster. The resource manager then promotes one of the brokers to be the primary. The other brokers connect to the primary as backups, using the URL provided in the ha-brokers-url configuration option.

Once connected, the backup brokers synchronize their state with the primary. When a backup is synchronized, or "hot", it is ready to take over if the primary fails. Backup brokers continually receive updates from the primary in order to stay synchronized.

If the primary fails, backup brokers go into fail-over mode. The resource manager must detect the failure and promote one of the backups to be the new primary. The other backups connect to the new primary and synchronize their state with it.

The resource manager is also responsible for protecting the cluster from split-brain conditions resulting from a network partition. A network partition divide a cluster into two sub-groups which cannot see each other. Usually a quorum voting algorithm is used that disables nodes in the inquorate sub-group.

This section assumes that you are already familiar with setting up and configuring clustered services using cman and rgmanager. It will show you how to configure an active-passive, hot-standby qpidd HA cluster with rgmanager.

chkconfig rgmanager on
chkconfig cman on
chkconfig qpidd off
	

critical Unexpected error: Daemon startup failed: Cannot lock /var/lib/qpidd/lock: Resource temporarily unavailable
	

You must provide a cluster.conf file to configure cman and rgmanager. Here is an example cluster.conf file for a cluster of 3 nodes named node1, node2 and node3. We will go through the configuration step-by-step.

      
<?xml version="1.0"?>
<!--
This is an example of a cluster.conf file to run qpidd HA under rgmanager.
This example assumes a 3 node cluster, with nodes named node1, node2 and node3.

NOTE: fencing is not shown, you must configure fencing appropriately for your cluster.
-->

<cluster name="qpid-test" config_version="18">
  <!-- The cluster has 3 nodes. Each has a unique nodeid and one vote
       for quorum. -->
  <clusternodes>
    <clusternode name="node1.example.com" nodeid="1"/>
    <clusternode name="node2.example.com" nodeid="2"/>
    <clusternode name="node3.example.com" nodeid="3"/>
  </clusternodes>

  <!-- Resouce Manager configuration. -->

   status_poll_interval is the interval in seconds that the resource manager checks the status
   of managed services. This affects how quickly the manager will detect failed services.
   -->
  <rm status_poll_interval="1">
    <!--
	There is a failoverdomain for each node containing just that node.
	This lets us stipulate that the qpidd service should always run on each node.
    -->
    <failoverdomains>
      <failoverdomain name="node1-domain" restricted="1">
	<failoverdomainnode name="node1.example.com"/>
      </failoverdomain>
      <failoverdomain name="node2-domain" restricted="1">
	<failoverdomainnode name="node2.example.com"/>
      </failoverdomain>
      <failoverdomain name="node3-domain" restricted="1">
	<failoverdomainnode name="node3.example.com"/>
      </failoverdomain>
    </failoverdomains>

    <resources>
      <!-- This script starts a qpidd broker acting as a backup. -->
      <script file="/etc/init.d/qpidd" name="qpidd"/>

      <!-- This script promotes the qpidd broker on this node to primary. -->
      <script file="/etc/init.d/qpidd-primary" name="qpidd-primary"/>

      <!--
          This is a virtual IP address for client traffic.
	  monitor_link="yes" means monitor the health of the NIC used for the VIP.
	  sleeptime="0" means don't delay when failing over the VIP to a new address.
      -->
      <ip address="20.0.20.200" monitor_link="yes" sleeptime="0"/>
    </resources>

    <!-- There is a qpidd service on each node, it should be restarted if it fails. -->
    <service name="node1-qpidd-service" domain="node1-domain" recovery="restart">
      <script ref="qpidd"/>
    </service>
    <service name="node2-qpidd-service" domain="node2-domain" recovery="restart">
      <script ref="qpidd"/>
    </service>
    <service name="node3-qpidd-service" domain="node3-domain"  recovery="restart">
      <script ref="qpidd"/>
    </service>

    <!-- There should always be a single qpidd-primary service, it can run on any node. -->
    <service name="qpidd-primary-service" autostart="1" exclusive="0" recovery="relocate">
      <script ref="qpidd-primary"/>
      <!-- The primary has the IP addresses for brokers and clients to connect. -->
      <ip ref="20.0.20.200"/>
    </service>
  </rm>
</cluster>
      
    

There is a failoverdomain for each node containing just that one node. This lets us stipulate that the qpidd service should always run on all nodes.

The resources section defines the qpidd script used to start the qpidd service. It also defines the qpid-primary script which does not actually start a new service, rather it promotes the existing qpidd broker to primary status.

The resources section also defines a virtual IP address for clients: 20.0.20.200.

qpidd.conf should contain these lines:

ha-cluster=yes
ha-brokers-url=20.0.20.1,20.0.20.2,20.0.20.3
    

The brokers connect to each other directly via the addresses listed in ha-brokers-url. Note the client and broker addresses are on separate sub-nets, this is recommended but not required.

The service section defines 3 qpidd services, one for each node. Each service is in a restricted fail-over domain containing just that node, and has the restart recovery policy. The effect of this is that rgmanager will run qpidd on each node, restarting if it fails.

There is a single qpidd-primary-service using the qpidd-primary script which is not restricted to a domain and has the relocate recovery policy. This means rgmanager will start qpidd-primary on one of the nodes when the cluster starts and will relocate it to another node if the original node fails. Running the qpidd-primary script does not start a new broker process, it promotes the existing broker to become the primary.

clusvcadm -d somenode-qpidd-service
clusvcadm -r qpidd-primary-service
        

Normally, clients are not allowed to connect to a backup broker. However management tools are allowed to connect to a backup brokers. If you use these tools you must not add or remove messages from replicated queues, nor create or delete replicated queues or exchanges as this will disrupt the replication process and may cause message loss.

qpid-ha allows you to view and change HA configuration settings.

The tools qpid-config, qpid-route and qpid-stat will connect to a backup if you pass the flag ha-admin on the command line.

By default, queues and exchanges are not replicated automatically. You can change the default behaviour by setting the ha-replicate configuration option. It has one of the following values:

You can over-ride the default for a particular queue or exchange by passing the argument qpid.replicate when creating the queue or exchange. It takes the same values as ha-replicate

Bindings are automatically replicated if the queue and exchange being bound both have replication all or configuration, they are not replicated otherwise.

You can create replicated queues and exchanges with the qpid-config management tool like this:

qpid-config add queue myqueue --replicate all
    

To create replicated queues and exchanges via the client API, add a node entry to the address like this:

"myqueue;{create:always,node:{x-declare:{arguments:{'qpid.replicate':all}}}}"
    

There are some built-in exchanges created automatically by the broker, these exchanges are never replicated. The built-in exchanges are the default (nameless) exchange, the AMQP standard exchanges (amq.direct, amq.topic, amq.fanout and amq.match) and the management exchanges (qpid.management, qmf.default.direct and qmf.default.topic)

Note that if you bind a replicated queue to one of these exchanges, the binding will not be replicated, so the queue will not have the binding after a fail-over.

Clients can only connect to the primary broker. Backup brokers reject any connection attempt by a client. Clients rejected by a backup broker will automatically fail-over until they connect to the primary.

Clients are configured with the URL for the cluster (details below for each type of client). There are two possibilities

In the first case, clients will repeatedly re-try each address in the URL until they successfully connect to the primary. In the second case the resource manager will assign the virtual IP address to the primary broker, so clients only need to re-try on a single address.

When the primary broker fails, clients re-try all known cluster addresses until they connect to the new primary. The client re-sends any messages that were previously sent but not acknowledged by the broker at the time of the failure. Similarly messages that have been sent by the broker, but not acknowledged by the client, are re-queued.

TCP can be slow to detect connection failures. A client can configure a connection to use a heartbeat to detect connection failure, and can specify a time interval for the heartbeat. If heartbeats are in use, failures will be detected no later than twice the heartbeat interval. The following sections explain how to enable heartbeat in each client.

Note: the following sections explain how to configure clients with multiple dresses, but if you are using a virtual IP address you only need to configure that one address for clients, you don't need to list all the addresses.

Suppose your cluster has 3 nodes: node1, node2 and node3 all using the default AMQP port, and you are not using a virtual IP address. To connect a client you need to specify the address(es) and set the reconnect property to true. The following sub-sections show how to connect each type of client.

qpid::messaging::Connection c("node1,node2,node3","{reconnect:true}");
      

qpid::messaging::Connection c("node1,node2,node3","{reconnect:true,heartbeat:10}");
      

connection = qpid.messaging.Connection.establish("node1", reconnect=True, reconnect_urls=["node1", "node2", "node3"])
      

connection = qpid.messaging.Connection.establish("node1", reconnect=True, reconnect_urls=["node1", "node2", "node3"], heartbeat=10)
      

	connectionfactory.qpidConnectionfactory = amqp://guest:guest@clientid/test?brokerlist='tcp://localhost:5672'&failover='failover_exchange'
      

	connectionfactory.qpidConnectionfactory = amqp://guest:guest@clientid/test?brokerlist='tcp://localhost:5672'&heartbeat='3'
      

This section outlines the HA specific aspects of security configuration. Please see Section 1.5, “Security” for more details on enabling authentication and setting up Access Control Lists.

When authentication is enabled you must set the credentials used by HA brokers with following options:

This identity is used to authorize federation links from backup to primary. It is also used to authorize actions on the backup to replicate primary state, for example creating queues and exchanges.

When authorization is enabled you must have an Access Control List with the following rule to allow HA replication to function. Suppose ha-username=USER

acl allow USER@QPID all all
    

To integrate with a different resource manager you must configure it to:

The qpid-ha command allows you to check if a broker is primary, and to promote a backup to primary.

To test if a broker is the primary:

qpid-ha -b broker-address status --expect=primary

This will return 0 if the broker at broker-address is the primary, non-0 otherwise.

To promote a broker to primary:

qpid-ha --cluster-manager -b broker-address promote

Note that promote is considered a "cluster manager only" command. Incorrect use of promote outside of the cluster manager could create a cluster with multiple primaries. Such a cluster will malfunction and lose data. "Cluster manager only" commands are not accessible in qpid-ha without the --cluster-manager option.

To list the full set of commands use:

qpid-ha --cluster-manager --help
    

If you use a persistent store for your messages then each broker in a cluster will have its own store. If the entire cluster fails and is restarted, the *first* broker that becomes primary will recover from its store. All the other brokers will clear their stores and get an update from the primary to ensure consistency.

This section applies to clusters that are using rgmanager as the cluster manager.

info SASL: Authentication failed: SASL(-13): user not found: Password verification failed
	

warning Client closed connection with 320: User anonymous@QPID federation connection denied. Systems with authentication enabled must specify ACL create link rules.
	

warning Client closed connection with 320: ACL denied anonymous@QPID creating a federation link.
	

qpid-ha status --all

<rm status_poll_interval=1>
	

<ip address="20.0.20.200" monitor_link="yes" sleeptime="0"/>
	

link-maintenance-interval=0.1
	

link-heartbeat-interval=5
	

Service Name                   Owner (Last)                   State
------- ----                   ----- ------                   -----
service:mrg33-qpidd-service    20.0.10.33                     started
service:mrg34-qpidd-service    20.0.10.34                     started
service:mrg35-qpidd-service    20.0.10.35                     started
service:qpidd-primary-service  (20.0.10.33)                   stopped