...

Cluster service is a distributed service and it that allows finding its nodes within a network automatically, and controlling the presence of no more than one leader node.

...

The ClusterManager interface manages interaction between an application and a cluster. In addition, it allows subscription for the current node state and for the cluster state.

With ClusterManager, it is possible to subscribe for 2 two groups of events:  

• LocalNodeLeaderListener – notification notifications about leader events 
• LocalNodeBackupListener – notification notifications about backup eventevents

An application can either use both listeners at the same time or the only one that most fit the current best fits a certain goal. For example, if an active node should have an active server, you only have to implement only implement LocalNodeLeaderListener and put start and stop server’s server calls into its methods.

To make it easier, we also provide LocalNodeLeaderListenerAdaptor and LocalNodeBackupListenerAdaptor. With their help you should only override necessary methods.

The default implementation of the Cluster Manager is based on Hazelcast. We use Hazelcast as a distributed cache with a cluster state. Hazelcast also helps to resolve nodes within a network.

...

Sample of Cluster Manager API use

The Cluster Manager is provided with built-in default settings. This allows you to start the Cluster service with the minimum effort.

public class ClusterSample {
    public static void main(String[] args) {
        ClusterManager clusterManager = new HazelcastClusterManager();                                       //(1)
        clusterManager.addLocalNodeLeaderListener(new LocalNodeLeaderListenerAdaptor() {

            @Override
            public void onGranted() {                                                                        //(2)
                System.out.println("This node was elected as a leader");
            }

            @Override
            public void onRevoked() {                                                                        //(3)
                System.out.println("This node isn't leader any more");
            }
        });

        clusterManager.join();                                                                               //(4)
        clusterManager.electLeader(clusterManager.localNode().id());                                         //(5)

        //pause 1 sec
        try {
            Thread.sleep(1000);
        } catch (InterruptedException e) {
        }

        //print nodes
        clusterManager.nodes().stream().forEach(System.out::println);                                        //(6)   
	
        clusterManager.leave();                                                                              //(7)
    }
}

...

Cluster service lifecycle

The cluster service uses the exchange of events between nodes to manage the cluster and notify nodes about state changes.

Joining a new node to a cluster

After calling the join() method, the node subscribes to cluster events. The rest of the cluster nodes will also will receive a notification about the new node in the cluster. On the node , which that is a leader at this moment, the LocalNodeLeaderListener.backupAdded() method will be called.

If a leader is present in the cluster, then the current node will be started in backup mode. If the leader is absent, then it may be initiated by the leader election procedure.

Automatic election of a new leader

The procedure of automatic leader selection is started at the cluster init.

Each node at start receives information about available nodes in the cluster and decides whether or not currently it should be elected a leader in that moment. This decision is based on the presence or absence of a quorum (see Hazelcast configuration). It is considered that the cluster has a quorum (and the leader can be chosen) if

...

where S is the number of nodes in the cluster at this the given moment and Q is the configured quorum size.

If the quorum size is not specified, it is considered that even one node already composes a quorum.

The first time leader selection algorithm is:

1. If the new node is has decided that it is necessary to choose a leader, it sends the LEADER_EVENT event to the cluster , and offers a new leader.

2. Once the proposed new leader gets the LEADER_EVENT event with its ID, it notifies the application about its new status by calling LocalNodeLeaderListener.onGranted() and sends the LEADER_STARTED_EVENT event with their its ID back to the cluster.

3. When the rest of the cluster nodes receive the LEADER_STARTED_EVENT event , they set their state to BACKUP (mark itself themselves as backup), and notify their applications about the new status by calling LocalNodeBackupListener.onBackup().

Appointment of a new leader

New A new leader can be assigned by calling the HazelcastClusterManager.electLeader() method and passing the node ID. New A new leader can be assigned if whether or not a cluster has a leader or if doesn’t have. In the first case, the previous leader . If a cluster already has a leader, it will be deactivated.
The algorithm of assigning a new leader is:

1. Node, which calls The node that calls the HazelcastClusterManager.electLeader() method , sends to the sendsthe cluster LEADER_EVENT event with a new leader ID.

2. When the backup node receives the LEADER EVENT event, it calls the LocalNodeBackupListener.offBackup() method.

3. A The node , which that was elected as a the new leader , launches a timer and waits for finishing the old leader to be finished. This mechanism was introduced to minimize the possibility of simultaneous work of the two leaders in the cluster working simultaneously. Old The old leader needs some time to complete its processes. The A timer is used to protect the cluster from endless waiting in case if when the old leader becomes inaccessible or crashes during switching the leaderwhile leaders are being switched.

4. Old The old leader receives the LEADER_EVENT event and notifies its application about the status change by calling LocalNodeLeaderListener.onRevoked(). After the successful completion of this call is completed successfully, it sends a LEADER_STOPPED_EVENT event to the cluster.

5. When the new leader node receives an a LEADER_STOPPED_EVENT event, (or if it doesn’t receive this event during timeoutLeaderShutdown period), it notifies the application about the new status by calling LocalNodeLeaderListener.onGranted() and sends to the cluster LEADER_STARTED_EVENT event with its ID.

6. When the rest of the cluster nodes receive the LEADER_STARTED_EVENT event, they change their state to BACKUP (mark themself themselves as backup) and notify their applications about the new status by calling LocalNodeBackupListener.onBackup().

...

There is a way to recall the leader. After recalling, all nodes will be in the backup state.
The algorithm of recalling the leader is the following:

1. The node , which that calls the HazelcastClusterManager.recallLeader() method , sends the LEADER_RECALL_EVENT event with the leader ID to the cluster.

2. The current leader receives the LEADER_RECALL_EVENT event and notifies its application about the status change by calling LocalNodeLeaderListener.onRevoked(). After the successful completion of this call, it sends the LEADER_STOPPED_EVENT event to the cluster.

...

Since version 1.4.0, by default, the new leader can be automatically elected when the existing leader is gone. Configuration options are described here.