Menu Search

12.3. Two Node Cluster

12.3.1. Overview

In this HA solution, a cluster is formed with two nodes. one node serves as master and the other is a replica.

All data and state required for the operation of the virtual host is automatically sent from the master to the replica. This is called the replication stream. The master virtual host confirms each message is on the replica before the client transaction completes. The exact way the client awaits for the master and replica is gorverned by the durability configuration, which is discussed later. In this way, the replica remains ready to take over the role of the master if the master becomes unavailable.

It is important to note that there is an inherent limitation of two node clusters is that the replica node cannot make itself master automatically in the event of master failure. This is because the replica has no way to distinguish between a network partition (with potentially the master still alive on the other side of the partition) and the case of genuine master failure. (If the replica were to elect itself as master, the cluster would run the risk of a split-brain scenario). In the event of a master failure, a third party must designate the replica as primary. This process is described in more detail later.

Clients connect to the cluster using a failover url. This allows the client to maintain a connection to the master in a way that is transparent to the client application.

12.3.2. Depictions of cluster operation

In this section, the operation of the cluster is depicted through a series of figures supported by explanatory text.

Figure 12.1. Key for figures

Key to figures

12.3.2.1. Normal Operation

The figure below illustrates normal operation. Clients connecting to the cluster by way of the failover URL achieve a connection to the master. As clients perform work (message production, consumption, queue creation etc), the master additionally sends this data to the replica over the network.

Figure 12.2. Normal operation of a two-node cluster

Normal operation

12.3.2.2. Master Failure and Recovery

The figure below illustrates a sequence of events whereby the master suffers a failure and the replica is made the master to allow the clients to continue to work. Later the old master is repaired and comes back on-line in replica role.

The item numbers in this list apply to the numbered boxes in the figure below.

  1. System operating normally

  2. Master suffers a failure and disconnects all clients. Replica realises that it is no longer in contact with master. Clients begin to try to reconnect to the cluster, although these connection attempts will fail at this point.

  3. A third-party (an operator, a script or a combination of the two) verifies that the master has truely failed and is no longer running. If it has truely failed, the decision is made to designate the replica as primary, allowing it to assume the role of master despite the other node being down. This primary designation is performed using JMX.

  4. Client connections to the new master succeed and the service is restored , albeit without a replica.

  5. The old master is repaired and brought back on-line. It automatically rejoins the cluster in the replica role.

Figure 12.3. Failure of master and recovery sequence

Failure of master and subsequent recovery sequence

12.3.2.3. Replica Failure and Recovery

The figure that follows illustrates a sequence of events whereby the replica suffers a failure leaving the master to continue processing alone. Later the replica is repaired and is restarted. It rejoins the cluster so that it is once again ready to take over in the event of master failure.

The behavior of the replica failure case is governed by the designatedPrimary configuration item. If set true on the master, the master will continue to operate solo without outside intervention when the replica fails. If false, a third-party must designate the master as primary in order for it to continue solo.

The item numbers in this list apply to the numbered boxes in the figure below. This example assumes that designatedPrimary is true on the original master node.

  1. System operating normally

  2. Replica suffers a failure. Master realises that replica longer in contact but as designatedPrimary is true, master continues processing solo and thus client connections are uninterrupted by the loss of the replica. System continues operating normally, albeit with a single node.

  3. Replica is repaired.

  4. After catching up with missed work, replica is once again ready to take over in the event of master failure.

Figure 12.4. Failure of replica and subsequent recovery sequence

Failure of replica and subsequent recovery sequence

12.3.2.4. Network Partition and Recovery

The figure below illustrates the sequence of events that would occur if the network between master and replica were to suffer a partition, and the nodes were out of contact with one and other.

As with Replica Failure and Recovery, the behaviour is governed by the designatedPrimary. Only if designatedPrimary is true on the master, will the master continue solo.

The item numbers in this list apply to the numbered boxes in the figure below. This example assumes that designatedPrimary is true on the original master node.

  1. System operating normally

  2. Network suffers a failure. Master realises that replica longer in contact but as designatedPrimary is true, master continues processing solo and thus client connections are uninterrupted by the network partition between master and replica.

  3. Network is repaired.

  4. After catching up with missed work, replica is once again ready to take over in the event of master failure. System operating normally again.

Figure 12.5. Partition of the network separating master and replica

Network Partition and Recovery

12.3.2.5. Split Brain

A split-brain is a situation where the two node cluster has two masters. BDB normally strives to prevent this situation arising by preventing two nodes in a cluster being master at the same time. However, if the network suffers a partition, and the third-party intervenes incorrectly and makes the replica a second master a split-brain will be formed and both masters will proceed to perform work independently of one and other.

There is no automatic recovery from a split-brain.

Manual intervention will be required to choose which store will be retained as master and which will be discarded. Manual intervention will be required to identify and repeat the lost business transactions.

The item numbers in this list apply to the numbered boxes in the figure below.

  1. System operating normally

  2. Network suffers a failure. Master realises that replica longer in contact but as designatedPrimary is true, master continues processing solo. Client connections are uninterrupted by the network partition.

    A third-party erroneously designates the replica as primary while the original master continues running (now solo).

  3. As the nodes cannot see one and other, both behave as masters. Clients may perform work against both master nodes.

Figure 12.6. Split Brain

Split Brain