Linux cluster

To service multiple FIX sessions reliably, FIXEdge can be deployed in a Linux HA cluster (Red Hat High-Availability Add-on Overview) with 2 or 3 nodes and shared storage for keeping

Linux cluster

To service multiple FIX sessions reliably, FIXEdge can be deployed in a Linux HA cluster (Red Hat High-Availability Add-on Overview) with 2 or 3 nodes and shared storage for keeping a session's state.

The possible configurable options of shared state are:

...

This approach gives a working, highly-available solution and works well for active-passive clusters with few nodes and static FIX session configuration.

Health checks

Health checks are used to figure out if a node or an application that runs on the node is operating properly.

The simplest way to do a node health check is to monitor the FIXEdge PID file.

More precise checks:

• Check the system status via FIXEdge Admin REST API by sending the special GET Request
• Establish FIX Admin monitoring session.

Shared physical device

Shared storage might be a SAN-attached device, Fibre channel attached device or TCP/IP attached device. The device might be attached to all nodes simultaneously, or the cluster resource manager can attach it to the active node only. The device might, in turn, be a resilient one, presenting the distributed file system with software or hardware replication between filesystem nodes. In the case of a geographically distributed cluster, the shared storage also can be distributed geographically in the same way as cluster nodes, and the cluster resource manager can attach the storage instance to the node in the same geo-location.

...

1. Network:
   1. The channel between FE and data provider - max wide
   2. dedicated network for cluster synchronization (heartbeats)
2. Open ports:
   1. high-availability service
   2. 8005/tcp
   3. 8901/tcp
   4. 8905/tcp
   5. 1234/udp
3. Application: 
   1. 1 Core per 1 Data Provider (recommended for latency)
   2. 1G for binaries
   3. Collocated nodes in one DC for max performance
   4. Health-check interval sufficient for FIXEdge to start
4. Storage:
   1. Mandatory Fibre Channel SAN storage for session logs - 1.5 TB, directly attached to both nodes. Encrypted via standard LVM features.
   2. Mandatory STONITH to ensure that FC storage is mounted on a single node
   3. Mandatory Archive for session logs - https://kbb2bits.b2bitsatlassian.comnet/wiki/display/B2BITS/FIXEdge+Capacity 
   4. Shared storage (NFS/SMB) for configuration - can be slow.
5. Experimental features:
   1. Start/stop from FIXICC - concept conflicts with cluster control.
6. Operating system: RHEL 7 or newer.

...

1. Install packages from a repository:

   Code Block
   language bash
   yum install corosync pcs pacemaker

2. Set the password for the hacluster  user:

   Code Block
   language bash
   passwd hacluster

3. Open ports on the firewall:

   Code Block
   language bash
   firewall-cmd --add-service=high-availability  firewall-cmd --runtime-to-permanent

4. Enable cluster services to run at system start-up:
   

   Code Block
   language bash
   systemctl enable pcsd corosync pacemaker

...

Set up a FIXEdge Cluster

Now we need to set up cluster resources for FIXEdge. Superuser privileges are required for all the steps.

...

to set up cluster resources for FIXEdge. Superuser privileges are required for all the steps.

1. Authorize nodes for hacluster  user:

   Code Block
   language bash
   pcs cluster auth NODE_1_NAME NODE_2_NAME -u hacluster

   Where NODE_1_NAME  and NODE_2_NAME are the hostnames of the servers that run FIXEdge.

   Info
   Note that these names must not resolve to 127.0.0.1 locally or use IP addresses instead of the hostnames.

2. Create the cluster and add nodes:

   Code Block
   language bash
   pcs cluster auth NODE setup --force --name fixedge_ha NODE_1_NAME NODE_2_NAME -u hacluster NAME

   Where NODE_1_NAME

    

   and NODE_2_NAME

    are

    are the hostnames of the servers that run FIXEdge.

   Info
   Note that these names must not resolve to 127.0.0.1 locally or use IP addresses instead of the hostnames.

   Create the cluster and add nodes

   of the hostnames.

3. Start cluster

   Code Block
   language bash
   pcs cluster start --all

   Now the cluster is starting up, you can check its status with commands:
   

   Code Block
   language bash
   pcs status cluster setuppcs --force --name fixedge_ha NODE_1_NAME NODE_2_NAME

   Where NODE_1_NAME and NODE_2_NAME are the hostnames of the servers that run FIXEdge.

   Info
   Note that these names must not resolve to 127.0.0.1 locally or use IP addresses instead of the hostnames.

   Start cluster

   status nodes

4. Disable resource migration on the first failure, since restarting on the same node takes less time than migration to another node:

   Code Block
   language bash
   pcs property set start-failure-is-fatal=false

5. For a two-node cluster we must disable the quorum, but do not do this for a three-node cluster. Quorum avoids the situation when the cluster cannot decide which node is active.

   Code Block
   language bash
   pcs clusterproperty startset no-quorum-all

   Now the cluster is starting up, you can check its status with commands

   policy=ignore

6. Add a virtual IP as a resource to the cluster:

   Code Block
   language bash
   pcs status cluster pcs status nodes

   Disable resource migration on the first failure, since restarting on the same node takes less time than migration to another node

   resource create virtual_ip ocf:heartbeat:IPaddr2 ip=<VIRTUAL_IP> cidr_netmask=32 op monitor interval=30s

   where <VIRTUAL_IP> is the IP that will be used by the FIX clients to connect to the cluster.

Use a Dual-Port SAN Device

1. Add an LVM group for the sessions' logs as a resource to the cluster:

   Code Block
   language bash
   pcs property set start-failure-is-fatal=false

   For a two-node cluster we must disable the quorum, but do not do this for a three-node cluster. Quorum avoids the situation when the cluster cannot decide which node is active.

   resource create logs_vg ocf:heartbeat:LVM volgrpname=logs

2. Add a filesystem for the sessions' logs as a resource to the cluster:

   Code Block
   language bash
   pcs property set no-quorum-policy=ignore

   Add a virtual IP

   resource create logs_fs ocf:heartbeat:Filesystem device=/dev/shared_logs_group/shared_logs directory=/opt/fixedge/FIXEdge1/log fstype=xfs

3. Add FIXEdge as a resource to the cluster:

   Code Block
   language bash
   pcs resource create virtual_ip ocf:heartbeat:IPaddr2 ip=<VIRTUAL_IP> cidr_netmask=32fixedge systemd:fixedge op start timeout=300s op stop timeout=60s op monitor interval=30s

   where <VIRTUAL_IP> is the IP that will be used by the FIX clients to connect to the cluster.

Use a Dual-Port SAN Device

1. Add an LVM group for the sessions' logs as a resource to the cluster

   10 timeout=60s meta migration-threshold=3

2. Make sure that all resources are started on the same node:

   Code Block
   language bash
   pcs resource create logs_vg ocf:heartbeat:LVM volgrpname=logs

   Add a filesystem for the sessions' logs as a resource to the cluster

   constraint colocation set virtual_ip logs_vg logs_fs fixedge sequential=true setoptions score=INFINITY

3. Make sure that resources are started in the proper order:

   Code Block
   language bash
   pcs resource create logs_fs ocf:heartbeat:Filesystem device=/dev/shared_logs_group/shared_logs directory=/opt/fixedge/FIXEdge1/log fstype=xfs

   Add FIXEdge as a resource to the cluster:

   constraint order set virtual_ip logs_vg logs_fs fixedge

Starting and stopping FIXEdge cluster resource from FIXICC

To avoid issues related to unexpected behavior by cluster software the user should prepare scripts that use cluster management commands.

Create scripts:

1. bin/FixEdge1.run.cluster.sh
   

   Code Block
   language bash title bin/FixEdge1.run.cluster.sh
   pcs resource create fixedge systemd:fixedge op start timeout=300s op stop timeout=60s op monitor interval=10 timeout=60s meta migration-threshold=3

   Make sure that all resources are started on the same node:

   enable fixedge

   This script is used for starting FIXEdge service as a cluster resource

2. bin/FixEdge1.stop.cluster.sh
   

   Code Block
   language bash title bin/FixEdge1.stop.cluster.sh
   pcs constraint colocation set virtual_ip logs_vg logs_fs fixedge sequential=true setoptions score=INFINITY

   Make sure that resources are started in the proper order:

   Code Block
   language bash
   pcs constraint order set virtual_ip logs_vg logs_fs fixedge resource disable fixedge

   This script is used for stopping FIXEdge service as a cluster resource

Update the paths to start and stop scripts in the fixicc-agent/conf/agent.properties.

StartFile = bin/FixEdge1.run.sh

StopFile = bin/FixEdge1.stop.sh

The user who runs fixicc-agent should have permission to operate the cluster. In some cases, it is required to add sudo for running pcs commands

How to Validate the Installation

...

Do these steps on both servers: NODE_1_NAME and NODE_2_NAME 

1. Download and install:

   Code Block
   language bash
   $ sudo wget -P /etc/yum.repos.d http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo $ sudo yum install glusterfs $ sudo yum install glusterfs-fuse $ sudo yum install glusterfs-server

2. Check installed version:
   

   Code Block
   language bash
   $ glusterfsd --version glusterfs 3.6.2 built on Jan 22 2015 12:58:10 Repository revision: git://git.gluster.com/glusterfs.git Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/> GlusterFS comes with ABSOLUTELY NO WARRANTY. It is licensed to you under your choice of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation.

3. Start glusterfs services on all servers and enable them to start automatically on startup:
   

   Code Block
   language bash
   $ sudo /etc/init.d/glusterd start $ sudo chkconfig glusterfsd on

...

1. On both nodes, install the needed software:

   Code Block
   language bash
   $ sudo yum install corosync pcs pacemaker

2. On both nodes, set the password for hacluster user ('epmc-cmcc' was used):

   Code Block
   language bash
   $ sudo passwd hacluster

3. Configure Firewall on both nodes to allow cluster traffic:

   Code Block
   language bash
   $ sudo iptables -I INPUT -m state --state NEW -p udp -m multiport --dports 5404,5405 -j ACCEPT $ sudo iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 2224 -j ACCEPT $ sudo iptables -I INPUT -p igmp -j ACCEPT $ sudo iptables -I INPUT -m addrtype --dst-type MULTICAST -j ACCEPT $ sudo service iptables save

4. Start the pcsd service on both nodes:

   Code Block
   language bash
   $ sudo systemctl start pcsd

5. From now on, all commands need to be executed on one node only. We can control the cluster by using PCS from one of the nodes.
   Since we will configure all nodes from one point, we need to authenticate on all nodes before we are allowed to change the configuration. Use the previously configured hacluster user and password to do this:

   Code Block
   language bash
   $ sudo pcs cluster auth NODE_1_NAME NODE_2_NAME  Username: hacluster Password: NODE_1_NAME: Authorized NODE_2_NAME: Authorized

6. Create the cluster and add nodes. This command creates the cluster node configuration in /etc/corosync.conf.

   Code Block
   language bash
   $ sudo pcs cluster setup --name fixedge_cluster NODE_1_NAME NODE_2_NAME  Shutting down pacemaker/corosync services... Redirecting to /bin/systemctl stop pacemaker.service Redirecting to /bin/systemctl stop corosync.service Killing any remaining services... Removing all cluster configuration files... NODE_1_NAME: Succeeded NODE_2_NAME: Succeeded

7. We can start cluster now:

   Code Block
   language bash
   $ sudo pcs cluster start --all NODE_1_NAME: Starting Cluster... NODE_2_NAME: Starting Cluster...

8. We can check cluster status:
   

   Code Block
   language bash
   $ sudo pcs status cluster Cluster Status: Last updated: Tue Jan 27 22:11:15 2015 Last change: Tue Jan 27 22:10:48 2015 via crmd on NODE_1_NAME Stack: corosync Current DC: NODE_1_NAME (1) - partition with quorum Version: 1.1.10-32.el7_0.1-368c726 2 Nodes configured 0 Resources configured $ sudo pcs status nodes Pacemaker Nodes: Online: NODE_1_NAME NODE_2_NAME Standby: Offline:   $ sudo corosync-cmapctl | grep members runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0 runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(10.17.131.127) runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1 runtime.totem.pg.mrp.srp.members.1.status (str) = joined runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0 runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(10.17.131.128) runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1 runtime.totem.pg.mrp.srp.members.2.status (str) = joined $ sudo pcs status corosync Membership information ---------------------- Nodeid Votes Name 1 1 NODE_1_NAME (local) 2 1 NODE_2_NAME

9. Disable the STONITH option as we don't have STONITH devices in our demo virtual environment:
   

   Code Block
   language bash
   $ sudo pcs property set stonith-enabled=false

10. For a two-node cluster we must disable the quorum:

    Code Block
    language bash
    $ sudo pcs property set no-quorum-policy=ignore $ sudo pcs property Cluster Properties: cluster-infrastructure: corosync dc-version: 1.1.10-32.el7_0.1-368c726 no-quorum-policy: ignore stonith-enabled: false

11. Add Virtual IP as a resource to the cluster:

    Code Block
    language bash
    $ sudo pcs resource create virtual_ip ocf:heartbeat:IPaddr2 ip=10.17.135.17 cidr_netmask=32 op monitor interval=30s $ sudo pcs status resources virtual_ip (ocf::heartbeat:IPaddr2): Started

12. Add FIXEdge as a resource to cluster:

    Code Block
    language bash
    $ sudo pcs resource create FIXEdge ocf:heartbeat:anything params binfile="/home/user/FixEdge/bin/FIXEdge" cmdline_options="/data/FixEdge1/conf/FIXEdge.properties" user="user" logfile="/home/user/FIXEdge_resource.log" errlogfile="/home/user/FIXEdge_resource_error.log"

    Note

    For some reason in the /usr/lib/ocf/resource.d/ of the installed cluster there are many missing agents, including ocf:heartbeat:anything. You need to modify the original version (which you can download here: https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/anything) to make it working. The working version of the agent is attached. This file should be copied to /usr/lib/ocf/resource.d/ and make executable: Code Block language bash $ sudo cp anything /usr/lib/ocf/resource.d/heartbeat/ $ sudo chmod a+rwx /usr/lib/ocf/resource.d/heartbeat/anything Also, to make this agent works the following lines shall be added to sudoers file: Code Block language bash $ sudo visudo Defaults !requiretty user ALL=(user) NOPASSWD: ALL root ALL=(user) NOPASSWD: ALL

13. In order to make sure that the Virtual IP and FIXEdge always stay together, we can add a constraint:

    Code Block
    language bash
    $ sudo pcs constraint colocation add FIXEdge virtual_ip INFINITY

14. To avoid the situation where the FIXEdge would start before the virtual IP is started or owned by a certain node, we need to add another constraint that determines the order of availability of both resources:

    Code Block
    language bash
    $ sudo pcs constraint order virtual_ip then FIXEdge Adding virtual_ip FIXEdge (kind: Mandatory) (Options: first-action=start then-action=start)

15. After configuring the cluster with the correct constraints, restart it and check the status:

    Code Block
    language bash
    $ sudo pcs cluster stop --all && sudo pcs cluster start --all NODE_1_NAME: Stopping Cluster... NODE_2_NAME: Stopping Cluster... NODE_2_NAME: Starting Cluster... NODE_1_NAME: Starting Cluster...

16. The cluster configuration is now completed.

...

The current article describes the Recovery Time Objective (RTO) and Recovery Point Objective(RPO) for Disaster recovery in case of active-passive Cluster configuration (see FIXEdge Failover Cluster installation).

Recovery Time Objective (RTO)

...

The session recovery procedure happens automatically. The missing messages should be recovered with a resend request procedure automatically.

Achievable RTO Values: "Greater than 0 seconds, up to and including 2 minutes"

...