FIXEdge node management

Add node

User can add node to cluster by adding configuration information in configuration file and running playbook (see deployment manual).

Please be careful with deployment of database part in playbook. During initial deployment scripts in playbook create clean database structure. If you create playbook for adding node you must exclude database part, otherwise all data will be lost. It is strictly recommended to backup database before any actions with playbooks.

Remove node

To remove FIXEdge node user must do the following

stop fixedge node ($sudo systemctl stop fixedge)
remove healthcheck from Consul ($sudo rm /srv/consul-agent/etc/fixedge-respond-check.json).
Stop consul-agent service if you want to remove node from list of nodes in Consul GUI ($sudo systemctl stop consul-agent)

Node maintenance

Configuration

Sessions and schedules properties are managed via Configuration Service with the help of Configuration Tool. All other configuration parameters are managed via local property files similar to standalone FIXEdge (see)

Logs

FIXEdge node logs are collected in centralized storage - Splunk system. You can concern How to integrate FIXEdge with Splunk document.

User must create data source input on tcp port=1514, sourcetype=log4j

Simple search example:

Cluster services management

Running several instances of services

Deployment of additional instances of services is implemented via playbook similar to adding of FIXEdge nodes. User can run several instances of Configuration Service. All instances are equal (there is no leader), users can use any of them. Consul Coordinator allows several instances of service running.

Removing of services can be done in the same way as for FIXEdge nodes.

Configuration service

Configuration Files

directory path for configuration files

/srv/configuration-service

Logs

directory path for logs:

/srv/configuration-service/logs

logs can be viewed by command

$journalctl -u configuration-service

Consul agent

Consul agent is deployed to each hardware unit to enable connection of cluster services with Consul server.

Configuration files

/srv/consul/etc

Logs

logs can be viewed by command

$journalctl -u consul-agent

Consul server

Configuration files

/srv/consul/etc

Logs

depends on customer configuration

Load-Balancer (HA-Proxy)

Configuration files

/etc/haproxy

Logs

/var/log/messages

Session management

User can add, modify and remove sessions with Configuration Tool.

Schedule management

User can add, modify and remove schedules with Configuration Tool.

Troubleshooting

Check health state of services and nodes

Health of nodes and services can be seen in Consul WebUI. Healthy services and nodes are displayed as below.

You can also check services state on hardware units where service or node is deployed to via Linux CLI commands.

$sudo systemctl status <service name>.

For example:

$sudo systemctl status fixedge

$sudo systemctl status consul-agent

Example of healthy service:

[user@ecsc00a02a94 etc]$ sudo systemctl status consul-agent
? consul-agent.service - Consul Agent
   Loaded: loaded (/usr/lib/systemd/system/consul-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-03-14 11:08:45 UTC; 21h ago
 Main PID: 18977 (consul)
    Tasks: 9
   Memory: 24.6M
   CGroup: /system.slice/consul-agent.service

Unhealthy node looks like

[user@ecsc00a03c92 ~]$ sudo systemctl status fixedge
? fixedge.service - FIXEdge Server
   Loaded: loaded (/usr/lib/systemd/system/fixedge.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Fri 2019-03-15 08:56:32 UTC; 2s ago
  Process: 12401 ExecStop=/srv/fixedge/bin/FixEdge1.stop.sh (code=exited, status=0/SUCCESS)
  Process: 10168 ExecStart=/srv/fixedge/bin/FixEdge1.run.sh (code=killed, signal=TERM)
 Main PID: 10168 (code=killed, signal=TERM)

adding node unsuccessful - host unreachable

Problem: when user tries to add new node running playbook he gets error message similar to:

PLAY [Deploy the FIXEdge daemon] **********************************************************************************************************************************************************************************
TASK [Gathering Facts] ********************************************************************************************************************************************************************************************
task path: /home/egor/work/fixedge-cluster-deployment-playbook/addfe.yml:22
fatal: [10.6.217.42]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.6.217.42 port 22: Connection timed out\r\n", "unreachable": true}
to retry, use: --limit @/home/egor/work/fixedge-cluster-deployment-playbook/addfe.retry
PLAY RECAP ********************************************************************************************************************************************************************************************************
10.6.217.42                : ok=0    changed=0    unreachable=1    failed=0
consul_server_1            : ok=2    changed=0    unreachable=0    failed=0
cs_1                       : ok=2    changed=0    unreachable=0    failed=0
fixedge_1                  : ok=2    changed=0    unreachable=0    failed=0
fixedge_2                  : ok=2    changed=0    unreachable=0    failed=0
haproxy                    : ok=2    changed=0    unreachable=0    failed=0
localhost                  : ok=2    changed=1    unreachable=0    failed=0

Description: node adding unsuccessful

Solution: Possible reason - node ip address is unreachable or user defined wrong ip address in configuration file. Check network connectivity and configuration file.

can not establish session - problems with database

Problem: when user tries to establish session he gets error message similar to:

[3/12/2019 6:22:33 PM] Session <FIXCLIENT, FIXEDGE> : session could join infinite messages in bunch.
[3/12/2019 6:22:33 PM] Session <FIXCLIENT, FIXEDGE> : FIX44 session was created.
[3/12/2019 6:22:33 PM] Session <FIXCLIENT, FIXEDGE> : incoming sequence number was forcedly set to 1.
[3/12/2019 6:22:33 PM] Cannot find the property "StorageCreationTime" in the storage "D:\Distr\test\FIX_Antenna_NET40_2.26.0_288\samples\SimpleClient\x64-Release\.\logs\FIXCLIENT-FIXEDGE_1903121422333791", used current date.
[3/12/2019 6:22:33 PM] Session <FIXCLIENT, FIXEDGE> : Message storage was reset Locally.
[3/12/2019 6:22:33 PM] Session <FIXCLIENT, FIXEDGE> : outgoing sequence number was forcedly set to 1.
[3/12/2019 6:22:33 PM] Session <FIXCLIENT, FIXEDGE> : Connection parameters were switched to primary set.
[3/12/2019 6:22:33 PM] Session <FIXCLIENT, FIXEDGE> : Connecting to 10.6.223.32:8901
[3/12/2019 6:22:33 PM] Session <FIXCLIENT, FIXEDGE> : Change state: old state=Initial new state=WaitForConfirmLogon
[3/12/2019 6:22:33 PM] Session <FIXCLIENT, FIXEDGE> : the telecommunication link error was detected (Connection::receive(), EOF from 10.6.223.32:8901).
[3/12/2019 6:22:33 PM] Start forced session reconnect for the session 'FIXCLIENTFIXEDGE'.
[3/12/2019 6:22:33 PM] Session <FIXCLIENT, FIXEDGE> : Change state: old state=WaitForConfirmLogon new state=Reconnect
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : Session 'FIXCLIENTFIXEDGE' tries to reconnect.
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : Asynchronously connecting to 10.6.223.32:8901
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : Asynchronous connect completed, error code: 0
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : the telecommunication link was restored.
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : Change state: old state=Reconnect new state=WaitForConfirmLogon
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : the telecommunication link error was detected (Connection::receive(), EOF from 10.6.223.32:8901).
[3/12/2019 6:22:34 PM] Start forced session reconnect for the session 'FIXCLIENTFIXEDGE'.
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : Change state: old state=WaitForConfirmLogon new state=Reconnect
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : Session 'FIXCLIENTFIXEDGE' tries to reconnect.
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : Asynchronously connecting to 10.6.223.32:8901
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : Asynchronous connect completed, error code: 0
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : the telecommunication link was restored.
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : Change state: old state=Reconnect new state=WaitForConfirmLogon
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : the telecommunication link error was detected (Connection::receive(), EOF from 10.6.223.32:8901).
[3/12/2019 6:22:34 PM] Start forced session reconnect for the session 'FIXCLIENTFIXEDGE'.
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : Change state: old state=WaitForConfirmLogon new state=Reconnect
[3/12/2019 6:22:34 PM] Forced session reconnect for the session 'FIXCLIENTFIXEDGE' was not started because its reconnect tries has expired.
[3/12/2019 6:22:34 PM] Received Logout message:
[3/12/2019 6:22:34 PM] Session <FIXCLIENT, FIXEDGE> : Change state: old state=Reconnect new state=NonGracefullyTerminated

Description: session is not established (Non-Gracefully Terminated)

Solution: Possible reason - some problems with storing FIX messages. Check network connectivity with corresponding database. Check database health.

can not establish session - load-balancer is unreachable or down

Problem: when trying to establish session user gets error message similar to:

[3/12/2019 3:59:05 PM] Session <FIXCLIENT, FIXEDGE1> : Incoming sequence number was restored: 2.
[3/12/2019 3:59:05 PM] Session <FIXCLIENT, FIXEDGE1> : Outgoing sequence number was restored: 3.
[3/12/2019 3:59:05 PM] Session <FIXCLIENT, FIXEDGE1> : data about the previous run were found in the message storage.
[3/12/2019 3:59:05 PM] Session <FIXCLIENT, FIXEDGE1> : session could join infinite messages in bunch.
[3/12/2019 3:59:05 PM] Session <FIXCLIENT, FIXEDGE1> : FIX44 session was created.
[3/12/2019 3:59:05 PM] Session <FIXCLIENT, FIXEDGE1> : incoming sequence number was forcedly set to 1.
[3/12/2019 3:59:05 PM] Cannot find the property "StorageCreationTime" in the storage "D:\Distr\test\FIX_Antenna_NET40_2.26.0_288\samples\SimpleClient\x64-Release\.\logs\FIXCLIENT-FIXEDGE1_1903121159059131", used current date.
[3/12/2019 3:59:05 PM] Session <FIXCLIENT, FIXEDGE1> : Message storage was reset Locally.
[3/12/2019 3:59:05 PM] Session <FIXCLIENT, FIXEDGE1> : outgoing sequence number was forcedly set to 1.
[3/12/2019 3:59:05 PM] Session <FIXCLIENT, FIXEDGE1> : Connection parameters were switched to primary set.
[3/12/2019 3:59:05 PM] Session <FIXCLIENT, FIXEDGE1> : Connecting to 10.6.223.32:8901
[3/12/2019 3:59:07 PM] Session <FIXCLIENT, FIXEDGE1> : Unable to establish connection: connect() to (10.6.223.32:8901) failed. No connection could be made because the target machine actively refused it. (Error code = 10061)
[3/12/2019 3:59:07 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=Initial new state=NonGracefullyTerminated
[3/12/2019 3:59:07 PM] connect() to (10.6.223.32:8901) failed. No connection could be made because the target machine actively refused it. (Error code = 10061)

Description: session is not established (Non-Gracefully Terminated)

Solution: check Load-Balancer (HA-Proxy) health, if service is down then start it. Check network connectivity from client workstation to load-balancer

can not establish session - service discovery does not work

Problem: when trying to establish session user gets error message similar to:

[3/12/2019 3:34:24 PM] Session <FIXCLIENT, FIXEDGE1> : Incoming sequence number was restored: 72.
[3/12/2019 3:34:24 PM] Session <FIXCLIENT, FIXEDGE1> : Outgoing sequence number was restored: 73.
[3/12/2019 3:34:24 PM] Session <FIXCLIENT, FIXEDGE1> : data about the previous run were found in the message storage.
[3/12/2019 3:34:24 PM] Session <FIXCLIENT, FIXEDGE1> : session could join infinite messages in bunch.
[3/12/2019 3:34:24 PM] Session <FIXCLIENT, FIXEDGE1> : FIX44 session was created.
[3/12/2019 3:34:24 PM] Session <FIXCLIENT, FIXEDGE1> : incoming sequence number was forcedly set to 1.
[3/12/2019 3:34:24 PM] Cannot find the property "StorageCreationTime" in the storage "D:\Distr\test\FIX_Antenna_NET40_2.26.0_288\samples\SimpleClient\x64-Release\.\logs\FIXCLIENT-FIXEDGE1_1903121134245091", used current date.
[3/12/2019 3:34:24 PM] Session <FIXCLIENT, FIXEDGE1> : Message storage was reset Locally.
[3/12/2019 3:34:24 PM] Session <FIXCLIENT, FIXEDGE1> : outgoing sequence number was forcedly set to 1.
[3/12/2019 3:34:24 PM] Session <FIXCLIENT, FIXEDGE1> : Connection parameters were switched to primary set.
[3/12/2019 3:34:24 PM] Session <FIXCLIENT, FIXEDGE1> : Connecting to 10.6.223.32:8901
[3/12/2019 3:34:24 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=Initial new state=WaitForConfirmLogon
[3/12/2019 3:34:27 PM] Session <FIXCLIENT, FIXEDGE1> : The session is non-gracefully terminated because the Logon acceptor did not respond in the given time frame (3 sec).
[3/12/2019 3:34:27 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=WaitForConfirmLogon new state=NonGracefullyTerminated
[3/12/2019 3:34:27 PM] Session <FIXCLIENT, FIXEDGE1> : active session was closed non-gracefully (The session is non-gracefully terminated because the Logon acceptor did not respond in the given time frame (3 sec).).
[3/12/2019 3:34:27 PM] Received Logout message:

Description: session is not established (Non-Gracefully Terminated)

Solution: check Service Discovery (Consul) health, if service is down then start it

can not establish session - no active nodes

Problem: when trying to establish session user gets error message similar to:

[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : Incoming sequence number was restored: 1.
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : Outgoing sequence number was restored: 1.
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : data about the previous run were found in the message storage.
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : session could join infinite messages in bunch.
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : FIX44 session was created.
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : incoming sequence number was forcedly set to 1.
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : Connection parameters were switched to primary set.
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : Connecting to 10.6.223.32:8901
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=Initial new state=WaitForConfirmLogon
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : the telecommunication link error was detected (Connection::receive(), EOF from 10.6.223.32:8901).
[3/12/2019 3:46:47 PM] Start forced session reconnect for the session 'FIXCLIENTFIXEDGE1'.
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=WaitForConfirmLogon new state=Reconnect
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : Session 'FIXCLIENTFIXEDGE1' tries to reconnect.
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : Asynchronously connecting to 10.6.223.32:8901
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : Asynchronous connect completed, error code: 0
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : the telecommunication link was restored.
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=Reconnect new state=WaitForConfirmLogon
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : the telecommunication link error was detected (Connection::receive(), EOF from 10.6.223.32:8901).
[3/12/2019 3:46:47 PM] Start forced session reconnect for the session 'FIXCLIENTFIXEDGE1'.
[3/12/2019 3:46:47 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=WaitForConfirmLogon new state=Reconnect
[3/12/2019 3:46:48 PM] Session <FIXCLIENT, FIXEDGE1> : Session 'FIXCLIENTFIXEDGE1' tries to reconnect.
[3/12/2019 3:46:48 PM] Session <FIXCLIENT, FIXEDGE1> : Asynchronously connecting to 10.6.223.32:8901
[3/12/2019 3:46:48 PM] Session <FIXCLIENT, FIXEDGE1> : Asynchronous connect completed, error code: 0
[3/12/2019 3:46:48 PM] Session <FIXCLIENT, FIXEDGE1> : the telecommunication link was restored.
[3/12/2019 3:46:48 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=Reconnect new state=WaitForConfirmLogon
[3/12/2019 3:46:48 PM] Session <FIXCLIENT, FIXEDGE1> : the telecommunication link error was detected (Connection::receive(), EOF from 10.6.223.32:8901).
[3/12/2019 3:46:48 PM] Start forced session reconnect for the session 'FIXCLIENTFIXEDGE1'.
[3/12/2019 3:46:48 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=WaitForConfirmLogon new state=Reconnect
[3/12/2019 3:46:48 PM] Forced session reconnect for the session 'FIXCLIENTFIXEDGE1' was not started because its reconnect tries has expired.
[3/12/2019 3:46:48 PM] Received Logout message:
[3/12/2019 3:46:48 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=Reconnect new state=NonGracefullyTerminated

Description: session is not established (Non-Gracefully Terminated)

Solution: check FIXEdge nodes health - possible reason is all nodes are down. If so start at least one node and try to establish session again.

can not establish session - invalid login

Problem: when trying to establish session user gets error message similar to:

[3/12/2019 3:19:26 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=WaitForConfirmLogon new state=Reconnect
[3/12/2019 3:19:26 PM] Session <FIXCLIENT, FIXEDGE1> : Session 'FIXCLIENTFIXEDGE1' tries to reconnect.
[3/12/2019 3:19:26 PM] Session <FIXCLIENT, FIXEDGE1> : Asynchronously connecting to 10.6.223.32:8901
[3/12/2019 3:19:26 PM] Session <FIXCLIENT, FIXEDGE1> : Asynchronous connect completed, error code: 0
[3/12/2019 3:19:26 PM] Session <FIXCLIENT, FIXEDGE1> : the telecommunication link was restored.
[3/12/2019 3:19:26 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=Reconnect new state=WaitForConfirmLogon
[3/12/2019 3:19:26 PM] Session <FIXCLIENT, FIXEDGE1> : the telecommunication link error was detected (Connection::receive(), EOF from 10.6.223.32:8901).
[3/12/2019 3:19:26 PM] Start forced session reconnect for the session 'FIXCLIENTFIXEDGE1'.
[3/12/2019 3:19:26 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=WaitForConfirmLogon new state=Reconnect
[3/12/2019 3:19:26 PM] Forced session reconnect for the session 'FIXCLIENTFIXEDGE1' was not started because its reconnect tries has expired.
[3/12/2019 3:19:26 PM] Received Logout message:
[3/12/2019 3:19:26 PM] Session <FIXCLIENT, FIXEDGE1> : Change state: old state=Reconnect new state=NonGracefullyTerminated

Description: session is not established (Non-Gracefully Terminated)

Solution: check login and password parameters of session

EPAM-B2BITS products for Capital Markets

FIXEdge Cluster Solution Operation and Maintenance Guide

FIXEdge node management

Add node

Remove node

Node maintenance

Configuration

Logs

Cluster services management

Running several instances of services

Configuration service

Configuration Files

Logs

Consul agent

Configuration files

Logs

Consul server

Configuration files

Logs

Load-Balancer (HA-Proxy)

Configuration files

Logs

Session management

Schedule management

Troubleshooting

Check health state of services and nodes

adding node unsuccessful - host unreachable

can not establish session - problems with database

can not establish session - load-balancer is unreachable or down

can not establish session - service discovery does not work

can not establish session - no active nodes

can not establish session - invalid login