The replication tool works on the following platforms: Windows (x64): Win7, Win8, Windows Server 2008, Windows Server 2012; Linux (x64): CentOS7, Ubuntu 16.04.
The distribution package of the Replication Tool is ZIP-archive for Windows and ZIP-archive for Linux containing the following files:
Script | Purpose |
---|---|
bin/admintool.bat | runs the admin tool mode |
bin/b2b_replication.exe | application file |
bin/client.bat | runs replication client |
bin/client.service.install.bat | installs the replication tool as a Windows service in client mode |
bin/client.service.run.bat | runs the replication tool as a Windows service in client mode |
bin/client.service.stop.bat | stops the replication tool as a Windows service in client mode |
bin/client.service.uninstall.bat | uninstalls the replication tool as a Windows service in client mode |
bin/server.bat | runs replication server |
bin/server.service.install.bat | installs the replication tool as a Windows service in server mode |
bin/server.service.run.bat | runs the replication tool as a Windows service in server mode |
bin/server.service.stop.bat | stops the replication tool as a Windows service in server mode |
bin/server.service.uninstall.bat | uninstalls the replication tool as a Windows service in server mode |
bin/standalone.bat | runs the standalone mode |
bin/replication.properties | properties file |
doc/Replication tool.docx | instructions file |
Script | Purpose |
---|---|
bin/admintool.sh | runs the admin tool mode |
bin/b2b_replication | application file |
bin/server.sh | runs replication server |
bin/client.sh | runs replication client |
bin/standalone.sh | runs the standalone mode |
bin/linux_replication.properties | properties file |
doc/Replication tool.docx | instructions file |
After the first launch of the replication tool, the following files will appear in the setup folder:
For correct usage of the Replication tool, the user must set the several properties of the current FIX Antenna-based application.
Property Name | Prohibited Value(s) | Allowed Value(s) |
---|---|---|
StorageType | transient splitPersistent null | Persistent PersistentMM |
MessageStorage.TimestampUnit | microsecond nanosecond | millisecond |
LogIncomingMessages | false | true |
Property Name | Prohibited Value(s) | Allowed Value(s) |
---|---|---|
storageFactory | SlicedFileStorageFactory InMemoryStorageFactory | FilesystemStorageFactory MMFStorageFactory |
incomingStorageIndexed | false | true |
outgoingStorageIndexed | false | true |
timestampsInLogs | false | true |
Resetting sequence numbers in the FIX Antenna Java's session is not supportable. A new session must be created instead. |
RT supports two main working modes:
A detailed description of the setup process for each mode is given below.
A client-server mode is used in the case when there is a need to replicate data to a different host(s).
The corresponding installation process is described below.
List aliases of the logs folders to be replicated in the replication.server.folders property.
If you replicate one directory you can leave this property empty.
Set the default path to the source logs directory in the replication.server.folder.default.path property.
If you replicate one directory, use it here. If you replicate several directories, use one of them here.
Set path to each directory being replicated in the corresponding replication.server.folder.AAA.path property, where AAA is the directory alias from the replication.server.folders property.
If you replicate one directory, you can skip this step. The default value set in the replication.server.folder.default.path property will be used instead.
If you replicate several directories set path to each of them under corresponding alias.
Start the replication manually. You can use the server.bat script from the bin folder of the distribution package. As well, the command "b2b_replication.exe" is used to start the replication tool server.
Start the replication as a Windows service. For the installation of the Replication Tool as a Windows service and schedules its automatic launch, the command "b2b_replication.exe --installService -n <ServiceServerName>" should be used or the file "server.service.install.bat". Then the user can start the installed service by using the Windows application service or type the command "%SystemRoot%\system32\net.exe start <ServiceServerName>" or use the file "server.service.run.bat".
[NOTE] 20141016-10:18:11.460 [7804] [ReplicationServer] - Waiting for incoming connection. |
C:\B2BITS\replication_tool>netstat -na | find "1090" TCP 0.0.0.0:1090 0.0.0.0:0 LISTENING |
List aliases of the logs folders to be replicated in the replication.client.folders property.
Folders' aliases should match the ones from the replication.server.folders list from the server configuration file. If the property is empty or not specified, all folders represented by the server are accepted and replicated into the default destination folder.
Set the default path to the source logs directory in the replication.client.folder.default.path property.
If you replicate one directory, use it here. If you replicate several directories, use one of them here.
Set path to each directory being replicated in the corresponding replication.client.folder.AAA.path property, where AAA is the directory alias from replication.client.folder.default.path property.
If you skip this step, the default destination directory will be used for all logs being replicated.
Start the replication manually. You can use the client.bat script from the bin folder of the distribution package. As well, the command "b2b_replication.exe --client" is used to start the replication tool client.
Start the replication as a Windows service. For the installation of the Replication Tool as a Windows service and schedules its automatic launch, the command "b2b_replication.exe --installService -n <ServiceClientName> --serviceStartArgs --client" should be used or the file "client.service.install.bat". Then the user can start the installed service by using the Windows application service or type the command "%SystemRoot%\system32\net.exe start <ServiceClientName>" or use the file "client.service.run.bat".
[NOTE] 20141016-10:35:31.623 [5508] [ReplicationClient] - Connected to server. |
Standalone mode is a mode when the client (subscriber) and server (publisher) are running within the same process. There is no TCP/IP connection in this mode. Standalone mode is preferable in a case when logs being replicated and copy should be located on the same machine. In order to set up the replication tool you need:
List aliases of the logs folders to be replicated in the replication.server.folders property.
If you replicate one directory you can leave the property empty.
Set the default path to the source logs directory in the replication.server.folder.default.path property.
If you replicate one directory, use it here. If you replicate several directories, use one of them here.
Set path to each directory being replicated in the corresponding replication.server.folder.AAA.path property, where AAA is the directory alias from the replication.server.folders property.
If you replicate one directory, you can skip this step. The default value set in the replication.server.folder.default.path property will be used instead.
If you replicate several directories set path to each of them under corresponding alias.
Set default path to the destination directory in the replication.client.folder.default.path property.
Set destination directory for each source directory in the corresponding replication.client.folder.AAA.path property, where AAA is the directory alias from the replication.server.folders property. If you skip this step, the default destination directory will be used for all logs being replicated.
Start the replication manually. You can use the standalone.bat script from the bin folder of the distribution package. As well, the command "b2b_replication.exe --standalone" is used to start the replication tool in standalone mode.
Start the replication as a Windows service. For the installation of the Replication Tool as a Windows service and schedules its automatic launch, the command "b2b_replication.exe --installService -n <ServiceStandaloneName> --serviceStartArgs --standalone" should be used. Then the user can start the installed service by using the Windows application service or type the command "%SystemRoot%\system32\net.exe start <ServiceStandaloneName>".
A client-server mode is used in the case when there is a need to replicate data to a different host(s).
The corresponding installation process is described below.
Configure server instance of the replication tool on the host where the primary files are created:
List aliases of the logs folders to be replicated in the replication.server.folders property.
If you replicate one directory you can leave this property empty.
Set the default path to the source logs directory in the replication.server.folder.default.path property.
If you replicate one directory, use it here. If you replicate several directories, use one of them here.
Set path to each directory being replicated in the corresponding replication.server.folder.AAA.path property, where AAA is the directory alias from the replication.server.folders property.
If you replicate one directory, you can skip this step. The default value set in the replication.server.folder.default.path property will be used instead.
If you replicate several directories set path to each of them under corresponding alias.
Run server.sh to start the replication process or use the command "./b2b_replication --propfile=./linux_replication.properties".
[NOTE] 20141016-10:18:11.460 [7804] [ReplicationServer] - Waiting for incoming connection. |
C:\B2BITS\replication_tool>netstat -na | find "1090" TCP 0.0.0.0:1090 0.0.0.0:0 LISTENING |
List aliases of the logs folders to be replicated in the replication.client.folders property.
Folders' aliases should match the ones from the replication.server.folders list from the server configuration file. If the property is empty or not specified, all folders represented by the server are accepted and replicated into default destination folder.
Set the default path to the source logs directory in the replication.client.folder.default.path property.
If you replicate one directory, use it here. If you replicate several directories, use one of them here.
Set path to each directory being replicated in the corresponding replication.client.folder.AAA.path property, where AAA is the directory alias from replication.client.folder.default.path property.
If you skip this step, the default destination directory will be used for all logs being replicated.
Run client.sh to start the replication process or use the command "./b2b_replication --client --propfile=./linux_replication.properties".
[NOTE] 20141016-10:35:31.623 [5508] [ReplicationClient] - Connected to server. |
Standalone mode is a mode when the client (subscriber) and server (publisher) are running within the same process. There is no TCP/IP connection in this mode. Standalone mode is preferable in a case when logs being replicated and copy should be located on the same machine. In order to set up the replication tool you need:
List aliases of the logs folders to be replicated in the replication.server.folders property.
If you replicate one directory you can leave the property empty.
Set the default path to the source logs directory in the replication.server.folder.default.path property.
If you replicate one directory, use it here. If you replicate several directories, use one of them here.
Set path to each directory being replicated in the corresponding replication.server.folder.AAA.path property, where AAA is the directory alias from the replication.server.folders property.
If you replicate one directory, you can skip this step. The default value set in the replication.server.folder.default.path property will be used instead.
If you replicate several directories set path to each of them under corresponding alias.
Set default path to the destination directory in the replication.client.folder.default.path property.
Set destination directory for each source directory in the corresponding replication.client.folder.AAA.path property, where AAA is the directory alias from the replication.server.folders property. If you skip this step, the default destination directory will be used for all logs being replicated.
Start the standalone.sh or use the command "b2b_replication --standalone --propfile=./linux_replication.properties" to start replication process.
Property | Is Required | Default Value | Description | |
---|---|---|---|---|
Server configurationThe section is used in Standalone mode and Client-Server mode on the server side. | ||||
replication.server.port | Y | 1090 | Server listen port. | |
replication.server.password | Y | - | A password that must be used by the client to be able to log in to the server. | |
replication.server.login | Y | - | A client's login that must be used by the client to be able to log in to the server. | |
replication.server.folders | N | - | A comma-separated list of aliases of folders to be monitored. If not specified or empty, default settings are used. If not empty "replication.server.folder.AAA.path" property for each folder from the list should be specified, where "AAA" is the folder alias. | |
replication.server.folder.default.recursive | Y | true | If set to 'true' then sub-folders are recursively added to the monitor list for replication. If set to 'false' then only files from the folder itself are replicated. Valid values: true | false | |
replication.server.folder.default.path | N | - | The default path to a folder to be monitored. | |
replication.server.folder.default.maxItemsPerUpdate | Y | 100 | A maximum number of log records that can be transferred in one message for this folder. Increasing this value may increase throughput but decrease response time and vice-versa. | |
replication.server.folder.default.filter | N | .* | Used only if the replication.server.folder.default.recursive property is set to 'true'. FIX session logs are added recursively to the monitoring list if the relative path matches this regexp. If not specified all FIX session logs are added. Please refer to the page for more details. | |
replication.server.folder.default.checkMinimalIndexFileSizeJAVA | Y | - |
This property is applicable for Java logs only. If set to 'true' the size of index files will be checked. Empty index files won't be replicated. If set to 'false' the size of index files won't be checked. Valid values: true | false | |
replication.server.folder.default.regexpConfFileBindingJAVA | Y | - |
Regexp postfix rule for matching config java files with log sessions. Valid values: regexp expressions Recommended value: -[0-9]{4}-[0-9]{2}-[0-9]{2}_[0-9]{2}-[0-9]{2}-[0-9]{2} | |
replication.server.folder.default.removeBackslashesInConfJAVA | N | true |
If set to 'true', all '\' backslashes will be removed from JAVA configuration files during replication. Valid values: true | false | |
replication.server.folder.AAA.recursive | N | replication.server.folder.default.recursive | If set to 'true' then all sub-folders of the folder AAA are recursively added to the monitor list for replication. If set to 'false' then only files from the folder itself are replicated. If not specified, replication.server.folder.default.recursive value is used for the folder. Valid values: true | false | |
replication.server.folder.AAA.path | Required if alias AAA is specified in the replication.server.folders property | - | Path to the monitored folder with alias AAA. | |
replication.server.folder.AAA.maxItemsPerUpdate | N | replication.server.folder.default.maxItemsPerUpdate | A maximum number of log records that can be transferred in one message for this folder. If not specified, the replication.server.folder.default.maxItemsPerUpdate value is used for the folder. | |
replication.server.folder.AAA.filter | N | - | Used only if replication.server.folder.AAA.recursive = true or if replication.server.folder.AAA.recursive is not specified and replication.server.folder.default.recursive = true. Sub-folders of the folder AAA are added recursively to the monitoring list if the relative path matches this regexp. If not specified all sub-folders are added. Please find further information. | |
replication.server.timeoutMs | N | 10000 | The time interval of milliseconds from the connection to the receiving invitation message from the client. | |
replication.server.filePollingIntervalM | N | 1 | Files poll interval. Server checks for new records in logs on this interval, milliseconds. | |
replication.server.foldersPollIntervalS | N | 10 | Folders poll interval. Server checks for folders' changes in this interval. It is actual for *nix systems if the filesystem doesn't support Inotify functionality, Otherwise and on the Windows platform server uses a notification mechanism to work on demand. The value is in seconds. | |
replication.server.confsPollIntervalS | N | 10 | Conf files poll interval. Server checks for modification of the .conf file on this interval, seconds. | |
replication.server.terminateOnOpenSessionError | N | false | If the parameter is true, then in case of a log session reader/writer creation error, the application logs a fatal error and stops immediately. Valid values: true | false | |
Client configurationThe section is used in Standalone mode and Client-Server mode on the client side. | ||||
replication.client.host | Y | - | IP address or hostname to connect. | |
replication.client.port | Y | - | Port to connect. It should be the same as the replication.server.port value from the server configuration file. | |
replication.client.login | Y | - | Client login. It should be the same as the replication.server.login value from the server configuration file. | |
replication.client.password | Y | - | Password. It should be the same as the replication.server.password value from the server configuration file. | |
replication.client.folders | N | - | A comma-separated list of aliases of folders to be accepted. Folders' aliases should match the ones from the replication.server.folders list from the server configuration file. The client ignores any folder represented by the server if their alias is not listed here, except the case when the property is empty or not specified at all. In this case, all folders represented by the server are accepted and replicated into the default destination folder. | |
replication.client.folder.default.path | N | - | Default destination folder path. | |
replication.client.folder.default.filter | N | .* | Replicated FIX session logs filter. If specified, only FIX session logs from this folder and all sub-folders that correspond to the regexp are accepted. If not specified, all FIX session logs represented by the server are accepted by the client. Please find further information. | |
replication.client.folder.default.recursive | N | true | If set to 'true', the client will accept sub-folders of the folder recursively. Valid values: true | false | |
replication.client.folder.AAA.path | Required if alias AAA is specified in the replication.client.folders property | - | Path to store folder with alias AAA. | |
replication.client.timeoutMs | N | 10000 | Timeout for reconnecting try. | |
replication.client.foldersPollIntervalS | N | 10 | This value overrides the same value in the server configuration. | |
replication.client.confsPollIntervalS | N | 10 | This value overrides the same value in the server configuration. | |
replication.client.heartBeatIntervalS | N | 5 | Heartbeat interval. This is the heartbeat interval used in TCP/IP communication (non-standalone mode) to detect connection breaks. The value is in seconds. | |
replication.client.postStatsIntervalM | N | 5 | Statistics post interval. The client posts statistics information into the log on this interval. Setting this value to '0' disables statistics posting. This value is in minutes. For more information on logging, check the Logging section. | |
replication.client.folder.default.maxItemsPerUpdate | N | 100 | This value overrides the same value in the server configuration. | |
replication.client.terminateOnOpenSessionError | N | false | If the parameter is true the application stops working with a FATAL error if there are issues during files creation. Valid values: true | false | |
Admintool configurationThis section is used in Admintool and Client mode. | ||||
replication.admintool.host | Y | - | IP address or hostname to connect. Note: Admintool connects to the client instance, not the server one. | |
replication.admintool.port | Y | - | For Client mode: port to open listener for administrative commands from Admintool. For Admintool mode: port to connect. | |
replication.admintool.login | Y | - | For Client mode: admintool login which must be used by admintool to be able to login to the client. For Admintool mode: admintool login. | |
replication.admintool.password | Y | - | For Client mode: admintool password which must be used by admintool to be able to login to the client. For Admintool mode: admintool password. | |
replication.admintool.heartBeatIntervalS | N | 50 | This value has the same meaning as for client | |
Logger configurationThis section is used for all modes. | ||||
Log.File.RootDir | Y | ./ | ||
Log.Device | Y | - | Target device. Supported values are:
| |
Log.DebugIsOn | Y | false | Turns on/off logging on a certain level. | |
Log.NoteIsOn | Y | false | Turns on/off logging on a certain level. | |
Log.WarnIsOn | Y | false | Turns on/off logging on a certain level. | |
Log.ErrorIsOn | Y | false | Turns on/off logging on a certain level. | |
Log.FatalIsOn | Y | false | Turns on/off logging on a certain level. | |
Log.Cycling | Y | - | Turns cycling on/off. | |
Log.File.Name | Y | - | Log file name. | |
Log.File.Recreate | Y | - | If true, then the file will be recreated on each start. | |
Log.File.AutoFlush | Y | true | If set to true, then the buffer will be flushed after each logging call. |
Command | Description |
---|---|
| Displays the help message. |
-p [ --pidfile ] arg | Specifies pid file to use. The default value is b2b_replication.pid. |
-c [–propfile] arg | Specifies configuration file to use. The default value is replication.properties. |
-s [--client] | Starts tool as a client (subscriber). |
-a [--standalone] | Starts tool in standalone mode when the client (subscriber) and server (publisher) are running within the same process. |
-t [ --admintool ] | Starts tool as admin tool. |
--persistentfile arg | Specifies a persistent properties file. This file is used to store folders’ states and is applicable to client and standalone mode only. The default value is persistent.properties. |
-l [ --nologo ] | Tells the tool to avoid showing startup banner. It is useful, for example, in admintool mode if further output processing is used. |
--admintoolCommand arg | Specifies admintool command to execute in admintool mode. Commands supported: 'list', 'stop', 'start', 'isstopped'. The default is 'list'. |
--admintollCommandArg arg | Specifies argument(s) for admintoolCommand. This option can be used more than once if several arguments need to be passed. |
--admintoolCommand start --admintoolCommandArg arg1 | Starts previously stopped folder replication. Arguments:
|
--admintoolCommand stop --admintoolCommandArg arg1 --admintollCommandArg arg2 | Stops folder replication. Arguments:
|
--admintoolCommand isstopped --admintoolCommandArg arg1 | Displays “true” if the folder is stopped or “false” otherwise. Arguments:
|
--admintoolCommand list | Lists all folders with their state and message count. |
Notes:
Specific commands to configure Replication tool as Windows service | |
---|---|
-i [--installService] | Installs the tool as a Windows service. |
| Uninstalls Windows service installed for the tool. |
-n [--serviceName] arg | Specifies service name to use for --installService and --uninstallService commands. The default value is “ReplicationService”. |
-d [--dependService] arg | Used with “–installService”, specifies service the service being installed depends on. Can be used more than once if more than one service must be added to the dependency list. |
--serviceStartArgs arg | Used with “–installService”, specifies command line options to be passed when service being installed is started by Windows Service Manage. |
The user must run service commands as a system administrator |
Every few minutes, the log records stats messages.
Configuring the parameter replication.client.postStatsIntervalM, allows the client to modify the interval with which stats messages are recorded. The value of this parameter is in minutes.
An example of a stats message:
[INFO] 20210428-13:42:28.975 [1796] [ReplicationClient] - ===================================== Tags stats ===================================== EchoServer: Status Active. Message count 0. Last data received at 2021-Apr-28 13:39:40. EchoServer2: Status Active. Message count 0. Last data received at 2021-Apr-28 13:39:43. Total active 2 of 2 ====================================================================================== |
If no folders are being replicated, the stats message will also contain the following:
[INFO] 20210428-13:45:16.870 [18056] [ReplicationClient] - ======================================= Tags stats ======================================= No tags are being replicated! ========================================================================================== |
Admin tool is the utility to easily monitor and control the replication process. It allows to change the replication folder's configuration by using several commands instead of configuring properties files.
For Window operating system
Admin tool is used to connect to the client and control the replication process remotely using simple command-line instructions. In order to use replication tool in the administrative mode you need:
Note: you can launch admintool in the client's folder with the same configuration file. In this case, they will share the same set of properties in the Admintool configuration section file.
For Linux operating system
Admin tool is used to connect to the client and control the replication process remotely using simple command-line instructions. In order to use replication tool in the administrative mode you need:
Note: you can launch admintool in the client's folder with the same configuration file. In this case, they will share the same set of properties in the Admintool configuration section file.
There are examples of admin tool commands.
Note: for Linux add the argument "--propfile=./linux_replication.properties".
List all monitored folders:
b2b_replication.exe --admintool --admintoolCommand list Alternative: admintool.bat list |
Stop replication of the folder FE1 gracefully:
b2b_replication.exe --admintool --admintoolCommand stop --admintoolCommandArg FE1 Alternative: admintool.bat stop FE1 |
Start over replication of the folder FE1:
b2b_replication.exe --admintool --admintoolCommand start --admintoolCommandArg FE1 Alternative: admintool.bat start FE1 |
Check if replication of the folder is stopped:
b2b_replication.exe --admintool --admintoolCommand isstopped --admintoolCommandArg FE1 Alternative: admintool.bat isstopped FE1 |
Stop replication of the folder FE1 immediately:
b2b_replication.exe --admintool --admintoolCommand stop --admintoolCommandArg FE1 --admintoolCommandArg regular Alternative: admintool.bat stop FE1 regular |
Stop replication of the folder FE1 immediately even though a graceful stop is in progress:
press the "Ctrl" and "C" keys at the same time, then enter the command into the command line: b2b_replication.exe --admintool --admintoolCommand stop --admintoolCommandArg FE1 --admintoolCommandArg force Alternative: admintool.bat stop FE1 force |
Note: for Linux add the argument "--propfile=./linux_replication.properties".
Start the tool in server mode with default properties file:
b2b_replication Alternative: server.bat |
Start the tool in server mode with custom properties file myreplication.properties:
b2b_replication --propfile=myreplication.properties |
Start the tool in client mode with custom persistent properties file mypersistent.properties:
b2b_replication --client --persistentfile=mypersistent.properties Alternative: client.bat |
Start the tool in standalone mode with the default configuration:
b2b_replication --standalone Alternative: standalone.bat |
Install the Windows service with a custom name and two dependencies in standalone mode:
b2b_replication --installService --serviceName=HFTReplication --dependService=Service1 --dependService=Service2 --serviceStartArgs="--standalone" |
Uninstall the Windows service from the example above:
b2b_replication --uninstallService -serviceName=HFTReplication Alternative: server.service.uninstall.bat/ client.service.uninstall.bat |
# Folders list to monitor. It is comma-separated list of folders' tags that server represents to clients. # If it is empty or not defined then it is threated in the same way if it value was set to 'default'. replication.server.folders = hftdma, hftdma2 # Default folder configuration.It gets in power if no folders are configured.# This configuration values except 'path' are taken for other configurations if particular values are not defined. # Folder path. It is path to the working folder for the tag. replication.server.folder.default.path = D:/replication # if the value is true then server looks for all sub-folders recursively and adds them to the monitor list. # It uses configuration values for the tag in this case for all sub-folders. Default is 'false'. replication.server.folder.default.recursive = false # folder walk regexp filter. If 'recursive' is true then server walks through # sub-folders recursively and adds them if relative path matches this regexp only. # If this parameter is not defined then all sub-folders are added. #replication.server.folder.default.filter = .* |