FIXGrep User Guide
Overview
FIXGrep is a console command-line utility that provides extended mechanisms for FIX log analyses and message tracking.
This tool can process 20000 messages per second. FIXGrep can do context searches by interpreting tag names and value descriptions.
In addition, FIXGrep can extract the sequence of changes for orders or other FIX business objects and identify their final states.
Installation on Linux
Follow the instructions below to install FIXGrep on Linux servers or upgrade FIXGrep to a newer version.
- Download the FIXGrep version for Linux from the Client Space or request access via b2bits.com website.
Extract the archive to the required directory.
tar -xzvf FIXGrep-1.2.12.0-linux-x64.tar.gz -C /FIXEdge/
- Request the license from the sales@btobits.com or use the one you already own.
Copy the license into the directory where the files are located.
cp /tmp/download/fixgrep.license /FIXEdge/fixgrep-1.2.12.0/
Run the query.
./fixgrep 35="0" /log/FIXEdge/*.in -o result.log
Invoking FIXGrep
The general synopsis of the FIXGrep command line is:
fixgrep EXPRESSION [FILE...] [keys...] [command] fixgrep --config FILE
./fixgrep EXPRESSION [FILE...] [keys...] [command] ./fixgrep --config FILE
EXPRESSION is a parameter – which is used without a specific key in front and it is specified right after the command name.
The following format is supported: “<tag>=<regexp> <tag>=<regexp>”, conditions are separated by space (" "), tag and regular expression are separated by equality sign ("="), number of conditions is unlimited, the whole pattern is quoted.
fixgrep "35=D 55=WMI|WJ.*" ./messages.in --list
FILE is a file path. It can be specified by mask. If multiple files are specified they must be separated by space (" "). If the file name contains space it must be enclosed in double quotes.
fixgrep "35=A" ./test.txt ./me* --list
Keys
Output: -o (--output)
The key is used to direct output data to a file. When -o is specified output data is saved to the specified file without being printed in stdout.
fixgrep "35=D|F" ./te* -o result.log
Output: -ecsv (--export-csv)
The key is used to direct output data to the file in CSV format.
The header file is used to put messages. Each message is a line. Only fields specified in the header will be extracted from messages.
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=D" ./message* -ecsv ./report.csv -o ./report.csv [fixmgr@localhost FIXGrep]$ ./fixgrep "35=D" ./test1.in -ecsv report.csv -o report_out.csv
report.csv before execution:
11,38,55
report.csv after execution:
,11,38,55
,00002-0511-2412-0-01092008,20,OTI
Sort: -sa (--sort-ascending), -sd (--sort-descending)
The key is used to show messages in a certain order by tag values. Messages can be sorted by more than one criterion however only one criterion can be specified for each tag. Sorting is applied in order of appearance. Tags involved in sorting criteria are separated by space. If the tag is not found message is moved to the end of the list sorted by this criterion.
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=D 55=K.*" ./me* -sd 55 -sa 11 [fixmgr@localhost FIXGrep]$ ./fixgrep "35=D 55=K.*" ./te* -sd 55 11 -sa 200
Filter: -m (--match), -nm (--not-match)
The key is used to get messages, which correspond to certain criteria. The match criterion checks the value of the specified tag against the specified regular expression. The --match criterion is satisfied when the field value matches the expression, while the --not-match criterion is satisfied when the field value does not match the expression.
Criteria are checked in the order of appearance which provides users with the ability to write more efficient rules. It is possible to specify more than one criterion for the same tag, and all expressions will be applied. Tag and regular expression are separated by space. The message is shown only if all conditions are satisfied (true) i.e. if at least one match condition fails message is not shown. If the tag, for which the expression is defined, is not present in the message then the condition is considered as failed.
fixgrep "35=.*" ./message.in -m "35" "D|F" "200" "200801" "55" "X.*" -nm "55" "XL|XEQ"
Filter: -p (--present), -np (--not-present)
The -p key is used to filter messages by tag presence. All messages, that do not contain specified tags, are not present in the result list.
The -np key is used to filter messages by tag presence. Only those messages, that do not contain specified tags, are present in the result list.
fixgrep "35=.*" ./messages.in -p 11 -np 41 -o report.txt
Filter: -t (--time), -nt (--not-time)
The key is used to get messages by time. Only messages, which are in the specified time interval - earlier than the first value and later than the second value - are shown in the result (or not in the specified interval if -nt is used).
The condition can be applied only to the FIX fields of the following data types: month-year, UTCTimestamp, UTCTimeOnly, UTCDateOnly, LocalMktDate, TZTimeOnly, TZTimestamp, UTCDate. Values must be in the format according to the data type.
The key can appear more than once. In this case, it is applied in the order of appearance.
fixgrep "35=D" ./messages.in -t 52 "20230109-13:33:00.000" "20230109-13:33:05.000"
Details: -D=<N> (--details=<N>)
The key is used to change the friendliness of the messages. Numbers are replaced by human-readable text depending on the level of the specified detail. The key is optional.
- -D=0 (--details=0) – simple raw data view i.e. tags and values are shown as is (default, the same as not specified).
[fixmgr@localhost FIXGrep]$ ./fixgrep "35=A" ./test1.in -nt 52 20080109-21:27:00.00098 20080109-21:27:00.00098 -sw "|" -D=0 FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "test1.in"... 8=FIX.4.2|9=73|35=A|49=SNQX|56=DBSUSH|34=1|50=SNPY|52=20080109-13:32:04.000|98=0|108=60|10=242| Completed.
- -D=1 (--details=1) – tags are shown by name.
[fixmgr@localhost FIXGrep]$ ./fixgrep "35=A" ./test1.in -nt 52 20080109-21:27:00.00098 20080109-21:27:00.00098 -sw "|" -D=1 FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "test1.in"... BeginString=FIX.4.2|BodyLength=73|MsgType=A|SenderCompID=SNQX|TargetCompID=DBSUSH|MsgSeqNum=1|SenderSubID=SNPY|SendingTime=20080109-13:32:04.000|EncryptMethod=0|HeartBtInt=60|CheckSum=242| Completed.
- -D=2 (--details=2) – tags are shown by names, and values are shown by description (the message can become big and multi-string).
[fixmgr@localhost FIXGrep]$ ./fixgrep "35=A" ./test1.in -nt 52 20080109-21:27:00.00098 20080109-21:27:00.00098 -sw "|" -D=2 FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "test1.in"... BeginString=FIX42|BodyLength=73|MsgType=LOGON|SenderCompID=SNQX|TargetCompID=DBSUSH|MsgSeqNum=1|SenderSubID=SNPY|SendingTime=20080109-13:32:04.000|EncryptMethod=None / other|HeartBtInt=60|CheckSum=242| Completed.
Separator: -sw (--soh-write)
It is possible to set more human-friendly symbols as field separators. A single value can be specified for -sw (soh-write) to override the separator used in the source by the one specified by the user.
Keys are optional.
[fixmgr@localhost FIXGrep]$ ./fixgrep "35=A" ./messages.in -sw " " FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "messages.in"... 8=FIX.4.2 9=73 35=A 49=SNQX 56=DBSUSH 34=1 50=SNPY 52=20080109-13:32:04.000 98=0 108=60 10=242 8=FIX.4.2 9=76 35=A 49=SNQX 56=DBSUSH 34=3782 50=SNPY 52=20080109-21:27:00.000 98=0 108=60 10=151 Completed.
Additional keys: -C, -L
The -L key instructs to output file list where messages that satisfy conditions are found, messages themselves are not written.
The -C key instructs to output the number/count of found messages, messages themselves are not written.
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=.*" ./me* -m 55 "WJJ" -L -C FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "message1.in"... Processing file "message2.in"... Processing file "message3.in"... Found 5 message(s) Found in file(s): ./message3.in Completed.
Command: --list (-l)
Writes all found FIX messages to the stdout taking into account all modifications implied by keys.
This is the default command i.e. if no command is specified the --list command is executed.
Command: --validate (-v=)
Applies validation to all found FIX messages according to the specified validation level taking into account all modifications implied by keys. Writes to the stdout all found errors.
- -v=0 (--validate=0) – no validation, default, the same as not specified.
- -v=1 (--validate=1) – check if the message is well-formed.
- -v=2 (--validate=2) – check if the message is well-formed, and all required fields are present.
- -v=3 (--validate=3) – check if the message is well-formed, all required fields are present, and all field values are correct.
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=.*" ./incorrect_messages.in -v=3 FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "incorrect_messages.in"... Field value 'DSTOK' does not meet ValBlock dictionary conditions in tag ExDestination (100) in message New Order - Single (D) with sequence number 2. 1 errors found.
Command: --order-status-list (-osl)
The command is used to print the status of the orders for given matching conditions.
fixgrep "35=.*" ./message.in -osl
Command: --history-order (--history, -ho)
The command is used to build a life cycle for the order.
For the given ClOrdID(11) extract back and forward all orders, replaces, cancels, executions, and rejects participated in the order life-cycle.
ClOrdID(11) of any messages involved in the life-cycle can be specified, not only for order.
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=.*" ./EP* -ho "E06" FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "EPAR7AFIX.in"... Order E06: EXC=>FIX 8=FIX.4.4☺9=184☺35=8☺49=EXC☺56=FIX☺34=36☺52=20230306-11:30:16.750☺369=36☺37=80☺11=E06☺17=403389☺150=0☺39=0☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺151=50☺14=0☺6=0☺60=20230306-11:30:16.738☺10=067☺ Status: New 0/50 @0 EXC=>FIX 8=FIX.4.4☺9=201☺35=8☺49=EXC☺56=FIX☺34=37☺52=20230306-11:30:16.750☺369=36☺37=80☺11=E06☺17=38A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=7☺31=8922.5☺151=43☺14=7☺6=8922.5☺60=20230306-11:30:16.738☺10=243☺ Status: Partially filled 7/50 @8922 EXC=>FIX 8=FIX.4.4☺9=205☺35=8☺49=EXC☺56=FIX☺34=38☺52=20230306-11:30:16.750☺369=36☺37=80☺11=E06☺17=39A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=8☺31=8922☺151=35☺14=15☺6=8922.233333☺60=20230306-11:30:16.738☺10=187☺ Status: Partially filled 15/50 @8922 EXC=>FIX 8=FIX.4.4☺9=207☺35=8☺49=EXC☺56=FIX☺34=39☺52=20230306-11:30:16.751☺369=36☺37=80☺11=E06☺17=40A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=3☺31=8921.5☺151=32☺14=18☺6=8922.111111☺60=20230306-11:30:16.738☺10=001☺ Status: Partially filled 18/50 @8922 EXC=>FIX 8=FIX.4.4☺9=205☺35=8☺49=EXC☺56=FIX☺34=40☺52=20230306-11:30:16.751☺369=36☺37=80☺11=E06☺17=41A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=8☺31=8921☺151=24☺14=26☺6=8921.769231☺60=20230306-11:30:16.738☺10=176☺ Status: Partially filled 26/50 @8921 EXC=>FIX 8=FIX.4.4☺9=207☺35=8☺49=EXC☺56=FIX☺34=41☺52=20230306-11:30:16.751☺369=36☺37=80☺11=E06☺17=42A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=1☺31=8920.5☺151=23☺14=27☺6=8921.722222☺60=20230306-11:30:16.738☺10=005☺ Status: Partially filled 27/50 @8921 EXC=>FIX 8=FIX.4.4☺9=200☺35=8☺49=EXC☺56=FIX☺34=42☺52=20230306-11:30:16.751☺369=36☺37=80☺11=E06☺17=43A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=4☺31=8920☺151=19☺14=31☺6=8921.5☺60=20230306-11:30:16.738☺10=174☺ Status: Partially filled 31/50 @8921 EXC=>FIX 8=FIX.4.4☺9=205☺35=8☺49=EXC☺56=FIX☺34=43☺52=20230306-11:30:16.751☺369=36☺37=80☺11=E06☺17=44A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=1☺31=8919.5☺151=18☺14=32☺6=8921.4375☺60=20230306-11:30:16.738☺10=179☺ Status: Partially filled 32/50 @8921 EXC=>FIX 8=FIX.4.4☺9=205☺35=8☺49=EXC☺56=FIX☺34=44☺52=20230306-11:30:16.752☺369=36☺37=80☺11=E06☺17=45A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=4☺31=8919☺151=14☺14=36☺6=8921.166667☺60=20230306-11:30:16.738☺10=196☺ Status: Partially filled 36/50 @8921 EXC=>FIX 8=FIX.4.4☺9=206☺35=8☺49=EXC☺56=FIX☺34=45☺52=20230306-11:30:16.752☺369=36☺37=80☺11=E06☺17=46A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=6☺31=8918.5☺151=8☺14=42☺6=8920.785714☺60=20230306-11:30:16.738☺10=251☺ Status: Partially filled 42/50 @8920 EXC=>FIX 8=FIX.4.4☺9=200☺35=8☺49=EXC☺56=FIX☺34=46☺52=20230306-11:30:16.752☺369=36☺37=80☺11=E06☺17=47A☺150=F☺39=2☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=8☺31=8900☺151=0☺14=50☺6=8917.46☺60=20230306-11:30:16.738☺10=191☺ Status: Filled 50/50 @8917
Command: --history-session (-hs)
For the given session ID – pair of SenderCompID(49) and TargetCompID(56) – build the history for the FIX session.
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=.*" ./messages.in -hs "SNQX" "DBSUSH" FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "messages.in"... 8=FIX.4.2☺9=73☺35=A☺49=SNQX☺56=DBSUSH☺34=1☺50=SNPY☺52=20080109-13:32:04.000☺98=0☺108=60☺10=242☺ State "SNQXDBSUSH" I: "Wait for confirm logon" 20080109-13:32:04.000 8=FIX.4.2☺9=87☺35=1☺49=SNQX☺56=DBSUSH☺34=2☺50=SNPY☺52=20080109-13:32:09.000☺112=Gap Fill Completed...☺10=169☺ State "SNQXDBSUSH" I: "Wait for confirm logon" 20080109-13:32:09.000 8=FIX.4.2☺9=56☺35=5☺49=SNQX☺56=DBSUSH☺34=3769☺52=20080109-21:25:59.000☺10=145☺ State "SNQXDBSUSH" I: "Wait for confirm logout" 20080109-21:25:59.000 8=FIX.4.2☺9=76☺35=A☺49=SNQX☺56=DBSUSH☺34=3782☺50=SNPY☺52=20080109-21:27:00.000☺98=0☺108=60☺10=151☺ State "SNQXDBSUSH" I: "Wait for confirm logout" 20080109-21:27:00.000 8=FIX.4.2☺9=68☺35=2☺49=SNQX☺56=DBSUSH☺34=3783☺52=20080109-21:27:00.000☺7=6248☺16=0☺10=159☺ State "SNQXDBSUSH" I: "Wait for confirm logout" 20080109-21:27:00.000 8=FIX.4.2☺9=97☺35=4☺49=SNQX☺56=DBSUSH☺34=3770☺43=Y☺52=20080109-21:27:00.000☺122=20080109-21:27:00☺123=Y☺36=3784☺10=093☺ State "SNQXDBSUSH" I: "Wait for confirm logout" 20080109-21:27:00.000 8=FIX.4.2☺9=90☺35=1☺49=SNQX☺56=DBSUSH☺34=3784☺50=SNPY☺52=20080109-21:27:05.000☺112=Gap Fill Completed...☺10=070☺ State "SNQXDBSUSH" I: "Wait for confirm logout" 20080109-21:27:05.000 Completed.
Command: --search (-s)
FIXGrep performs a context search in the loaded data by some plain text. Human readable text can be specified for search e.g. “39=0” or “OrdStatus=0” or “OrdStatus=New”.
Each message is converted into 3 forms:
- The raw FIX message.
- The detailed FIX message where tags are replaced by field names.
- The wide form where tags are replaced by field names, and values when possible are replaced by descriptions.
Search is performed in all these 3 lists.
fixgrep "35=.*" ./message.in -s "Symbol=QHN"
Configuration file: -c (–config)
All keys can be specified in the configuration file. All extra spaces and new lines are removed while the file is parsed. The key cannot be combined with other keys.
Configuration file content is interpreted as it was passed in the command line.
fixgrep --config cmd.txt
"35=.*" ./me* -m 55 "WJJ" -L -C
Command: --send-tcp
This command is used to establish a TCP connection and send a sequence of bytes from a file. E.g. replay messages from the session log.
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=.*" ./messages.in --send-tcp localhost 9106 FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "messages.in"... Sent 908487 of 908487 bytes.