Overview

Cluster service is a distributed service and it allows finding its nodes within network automatically and controlling presence of no more than one leader node.

Cluster implementation is scalable and customizable and provides several endpoints for integration.

The ClusterManager interface manages interaction between application and cluster. In addition, it allows subscription for the current node state and for the cluster state.

With ClusterManager it is possible to subscribe for 2 groups of events:  

• LocalNodeLeaderListener – notification about leader events 
• LocalNodeBackupListener – notification about backup event

An application can use both listeners at the same time or the only one that most fit the current goal. For example, if an active node should have an active server, you have to implement only LocalNodeLeaderListener and put start and stop server’s calls into its methods.

To make it easier, we also provide LocalNodeLeaderListenerAdaptor and LocalNodeBackupListenerAdaptor. With their help you should only override necessary methods.

The default implementation of the Cluster Manager is based on Hazelcast. We use Hazelcast as a distributed cache with a cluster state. Hazelcast also helps to resolve nodes within a network.

Hazelcast supports several different transports including multicast and TCP. The default configuration uses multicast so you must have multicast enabled on your network for this to work or update the cluster.xml configuration file.

Cluster implementation can also configure the quorum for automatic leader election.

Sample of Cluster Manager API use

The Cluster Manager is provided with built-in default settings. This allows you to start the Cluster service with the minimum effort.

public class ClusterSample {
    public static void main(String[] args) {
        ClusterManager clusterManager = new HazelcastClusterManager();                                       //(1)
        clusterManager.addLocalNodeLeaderListener(new LocalNodeLeaderListenerAdaptor() {

            @Override
            public void onGranted() {                                                                        //(2)
                System.out.println("This node was elected as a leader");
            }

            @Override
            public void onRevoked() {                                                                        //(3)
                System.out.println("This node isn't leader any more");
            }
        });

        clusterManager.join();                                                                               //(4)
        clusterManager.electLeader(clusterManager.localNode().id());                                         //(5)

        //pause 1 sec
        try {
            Thread.sleep(1000);
        } catch (InterruptedException e) {
        }

        //print nodes
        clusterManager.nodes().stream().forEach(System.out::println);                                        //(6)   
	
        clusterManager.leave();                                                                              //(7)
    }
}

1. Create an instance of the Cluster Manager with default configuration.
2. This method will be called when the node becomes a leader.
3. This method will be called when the leader node services should be stopped.
4. Join a node to the cluster.
5. Assign this node as a cluster leader.
6. Print all nodes in this cluster.
7. Leave the cluster.

In this case, the cluster will use the 224.2.2.3:54327 UDP multicast address for communication. The size of the quorum for the default cluster is equal to 2 (the leader will be selected automatically if there are two or more nodes in the cluster).

Cluster service lifecycle

The cluster service uses exchange of events between nodes to manage the cluster and notify nodes about state changes.

Joining new node to cluster

After calling the join() method, the node subscribes to cluster events. The rest of the cluster nodes also will receive notification about new node in the cluster. On the node, which is a leader at this moment, LocalNodeLeaderListener.backupAdded() method will be called.

If a leader is present in the cluster, then the current node will be started in backup mode. If the leader is absent, then it may be initiated by the leader election procedure.

Automatic election of new leader

The procedure of automatic leader selection is started at the cluster init.

Each node at start receives information about available nodes in the cluster and decides whether or not currently be elected a leader. This decision is based on the presence or absence of a quorum (see Hazelcast configuration). It is considered that the cluster has a quorum (and the leader can be chosen) if

 S/2+1 >= Q

where S is the number of nodes in the cluster at this moment and Q is the configured quorum size.

If the quorum size is not specified, it is considered that even one node already composes a quorum.

The first time leader selection algorithm is:

1. If the new node is decided that it is necessary to choose a leader, it sends the LEADER_EVENT event to the cluster, and offers a new leader.

2. Once the proposed new leader gets the LEADER_EVENT event with its ID, it notifies the application about its new status by calling LocalNodeLeaderListener.onGranted() and sends the LEADER_STARTED_EVENT event with their ID back to the cluster.

3. When the rest of the cluster nodes receive the LEADER_STARTED_EVENT event , they set their state to BACKUP (mark itself as backup), and notify their applications about new status by calling LocalNodeBackupListener.onBackup().

Appointment of new leader

New leader can be assigned by calling HazelcastClusterManager.electLeader() method and passing the node ID. New leader can be assigned if cluster has a leader or if doesn’t have. In the first case, the previous leader will be deactivated.
The algorithm of assigning new leader is:

1. Node, which calls HazelcastClusterManager.electLeader() method, sends to the cluster LEADER_EVENT event with new leader ID.

2. When backup node receives LEADER EVENT event, it calls LocalNodeBackupListener.offBackup() method.

3. A node, which was elected as a new leader, launches a timer and waits for finishing the old leader. This mechanism was introduced to minimize the possibility of simultaneous work of the two leaders in the cluster. Old leader needs some time to complete its processes. The timer is used to protect cluster from endless waiting in case if the old leader becomes inaccessible or crashes during switching the leader.

4. Old leader receives the LEADER_EVENT event and notifies its application about the status change by calling LocalNodeLeaderListener.onRevoked(). After the successful completion of this call, it sends a LEADER_STOPPED_EVENT event to the cluster.

5. When the new leader node receives an LEADER_STOPPED_EVENT event (or it doesn’t receive this event during timeoutLeaderShutdown period), it notifies the application about new status by calling LocalNodeLeaderListener.onGranted() and sends to the cluster LEADER_STARTED_EVENT event with its ID.

6. When the rest of the cluster nodes receive the LEADER_STARTED_EVENT event, they change their state to BACKUP (mark themself as backup) and notify their applications about new status by calling LocalNodeBackupListener.onBackup().

Recall the leader

There is a way to recall the leader. After recalling, all nodes will be in the backup state.
The algorithm of recalling the leader is the following:

1. The node, which calls the HazelcastClusterManager.recallLeader() method, sends the LEADER_RECALL_EVENT event with the leader ID to the cluster.

2. The current leader receives the LEADER_RECALL_EVENT event and notifies its application about the status change by calling LocalNodeLeaderListener.onRevoked(). After the successful completion of this call, it sends the LEADER_STOPPED_EVENT event to the cluster.

Automatic leader re-election

Since version 1.4.0, by default the new leader can be automatically elected when the existing leader is gone. Configuration options are described here.