Overview
FIX Grep is a console command-line utility that provides extended mechanisms for FIX log analyses and message tracking.
Tool can process 20000 messages per second. FIXGrep can do context search by interpreting tags names and values descriptions.
In addition FIXGrep can extract the sequence of changes for orders or other FIX business objects and identify their final states.
Invoking FIXGrep
The general synopsis of the FIXGrep command line is:
fixgrep EXPRESSION [FILE...] [keys...] [command] fixgrep --config FILE
./fixgrep EXPRESSION [FILE...] [keys...] [command] ./fixgrep --config FILE
EXPRESSION is a parameter – which is used without specific key in front and it is specified right after the command name.
The following format is supported: “<tag>=<regexp> <tag>=<regexp>”, conditions are separated by “ ” (space), tag and regular expression are separated by “=” (equality sign), number if conditions is unlimited, the whole pattern is quoted.
fixgrep "35=D 55=WMI|WJ.*" ./messages.in --list
FILE is a file path. It can be specified by mask. If multiple files are specified they must be separated by space (" "). If file name contains space it must be enclosed in double quotes.
fixgrep "35=A" ./test.txt ./me* --list
Keys
Output: -o (--output)
Key is used to direct output data to a file. When -o is specified out put data is saved to the specified file without being printed in stdout.
fixgrep "35=D|F" ./te* -o result.log
Output: -ecsv (--export-csv)
Key is used to direct output data to file in CSV format.
Header file is used to put messages. Each message is a line. Only fields, specified in header, are to be extracted from messages.
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=D" ./message* -ecsv ./report.csv -o ./report.csv [fixmgr@localhost FIXGrep]$ ./fixgrep "35=D" ./test1.in -ecsv report.csv -o report_out.csv
report.csv before execution:
11,38,55
report.csv after execution:
,11,38,55
,00002-0511-2412-0-01092008,20,OTI
Sort: -sa (--sort-ascending), -sd (--sort-descending)
Key is used to show messages in certain order by tags values. Messages can be sorted by more than one criterion however only one criterion can be specified for each tag. Sorting is applied in order of appearance. Tags involved in sorting criterion are separated by space. If tag is not found message is moved to the end of the list sorted by this criterion.
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=D 55=K.*" ./me* -sd 55 -sa 11 [fixmgr@localhost FIXGrep]$ ./fixgrep "35=D 55=K.*" ./te* -sd 55 11 -sa 200
Filter: -m (--match), -nm (--not-match)
Key is used to get messages, which correspond to the certain criteria. Match criterion check the value of specified tag against specified regular expression. In case of --match criterion is satisfied if field value matches expression, in case of --not-match criterion is satisfied if field value does not match expression. Criteria are checked in the order of appearance (which gives user ability to write more efficient rule). It is possible to specify more than one criterion for the same tag, all expressions will be applied. Tag and regular expression are separated by space. Message is shown only if all conditions are satisfied (true) i.e. if at least one match condition fails message is not shown. If tag, for which expression is defined, is not present in message then condition is considered as failed.
fixgrep "35=.*" ./message.in -m "35" "D|F" "200" "200801" "55" "X.*" -nm "55" "XL|XEQ"
Filter: -p (--present), -np (--not-present)
Key -p is used to filter message by tag presence. All messages, which do not contain specified tags, are not present in the result list.
Key -np is used to filter message by tag presence. Only those messages, which do not contain specified tags, are present in the result list.
fixgrep "35=.*" ./messages.in -p 11 -np 41 -o report.txt
Filter: -t (--time), -nt (--not-time)
Key is used to get messages by time. Only messages, which are in the specified time interval (earlier than first value and later than second value) are shown in the result (or not in the specified interval if -nt is used). Condition can be applied only to the FIX fields, which have types: month-year, UTCTimestamp, UTCTimeOnly, UTCDateOnly, LocalMktDate, TZTimeOnly, TZTimestamp, UTCDate. Values must be in the format according to the data type. Key can appear more than once; in this case it is applied in the order of appearance.
fixgrep "35=D" ./messages.in -t 52 "20230109-13:33:00.000" "20230109-13:33:05.000"
Details: -D=<N> (--details=<N>)
Key is used to change messages friendliness. Numbers are replaced by human readable text depending on the specified details level. Key is optional.
- -D=0 (--details=0) – simple raw data view i.e. tags and values are shown as is (default, the same as not specified)
[fixmgr@localhost FIXGrep]$ ./fixgrep "35=A" ./test1.in -nt 52 20080109-21:27:00.00098 20080109-21:27:00.00098 -sw "|" -D=0 FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "test1.in"... 8=FIX.4.2|9=73|35=A|49=SNQX|56=DBSUSH|34=1|50=SNPY|52=20080109-13:32:04.000|98=0|108=60|10=242| Completed.
- -D=1 (--details=1) – tags are shown by name
[fixmgr@localhost FIXGrep]$ ./fixgrep "35=A" ./test1.in -nt 52 20080109-21:27:00.00098 20080109-21:27:00.00098 -sw "|" -D=1 FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "test1.in"... BeginString=FIX.4.2|BodyLength=73|MsgType=A|SenderCompID=SNQX|TargetCompID=DBSUSH|MsgSeqNum=1|SenderSubID=SNPY|SendingTime=20080109-13:32:04.000|EncryptMethod=0|HeartBtInt=60|CheckSum=242| Completed.
- -D=2 (--details=2) – tags are shown by names, values are shown by description (message can become big and multi-string)
[fixmgr@localhost FIXGrep]$ ./fixgrep "35=A" ./test1.in -nt 52 20080109-21:27:00.00098 20080109-21:27:00.00098 -sw "|" -D=2 FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "test1.in"... BeginString=FIX42|BodyLength=73|MsgType=LOGON|SenderCompID=SNQX|TargetCompID=DBSUSH|MsgSeqNum=1|SenderSubID=SNPY|SendingTime=20080109-13:32:04.000|EncryptMethod=None / other|HeartBtInt=60|CheckSum=242| Completed.
Separator: -sw (--soh-write)
It is possible to set more human friendly symbol as field separator. Single value can be specified for -sw (soh-write) to override separator used in source by the one specified by user.
Keys are optional.
[fixmgr@localhost FIXGrep]$ ./fixgrep "35=A" ./messages.in -sw " " FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "messages.in"... 8=FIX.4.2 9=73 35=A 49=SNQX 56=DBSUSH 34=1 50=SNPY 52=20080109-13:32:04.000 98=0 108=60 10=242 8=FIX.4.2 9=76 35=A 49=SNQX 56=DBSUSH 34=3782 50=SNPY 52=20080109-21:27:00.000 98=0 108=60 10=151 Completed.
Additional keys: -C, -L
“-L” key instructs to output file list where message that satisfy conditions are found, messages themselves are not written;
“-C” key instructs to output the number (count) of found messages, messages themselves are not written;
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=.*" ./me* -m 55 "WJJ" -L -C FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "message1.in"... Processing file "message2.in"... Processing file "message3.in"... Found 5 message(s) Found in file(s): ./message3.in Completed.
Command: --list (-l)
Writes all found FIX messages (taking into account all modifications implied by keys) to the stdout.
This is default command i.e. if no command is specified the --list command is executed.
Command: --validate (-v=)
Applies validation to all found FIX messages (taking into account all modifications implied by keys) according to the specified validation level. Writes to the stdout all found errors.
- -v=0 (--validate=0) – no validation (default, the same as not specified)
- -v=1 (--validate=1) – check if message well formed
- -v=2 (--validate=2) – check if message is well formed, all required fields presents
- -v=3 (--validate=3) – check if message is well formed, all required fields presents, and all fields values are correct
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=.*" ./incorrect_messages.in -v=3 FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "incorrect_messages.in"... Field value 'DSTOK' does not meet ValBlock dictionary conditions in tag ExDestination (100) in message New Order - Single (D) with sequence number 2. 1 errors found.
Command: --order-status-list (-osl)
The command is used to print status of the orders for given matching conditions.
fixgrep "35=.*" ./message.in -osl
Command: --history-order (--history, -ho)
The command is used to build life-cycle for the order.
For the given client order ID (FIX tag 11) extract back and forward all orders, replaces, cancels, executions and rejects participated in order life-cycle (CLOrdID of any messages involved in life-cycle can be specified, not only for order).
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=.*" ./EP* -ho "E06" FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "EPAR7AFIX.in"... Order E06: EXC=>FIX 8=FIX.4.4☺9=184☺35=8☺49=EXC☺56=FIX☺34=36☺52=20230306-11:30:16.750☺369=36☺37=80☺11=E06☺17=403389☺150=0☺39=0☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺151=50☺14=0☺6=0☺60=20230306-11:30:16.738☺10=067☺ Status: New 0/50 @0 EXC=>FIX 8=FIX.4.4☺9=201☺35=8☺49=EXC☺56=FIX☺34=37☺52=20230306-11:30:16.750☺369=36☺37=80☺11=E06☺17=38A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=7☺31=8922.5☺151=43☺14=7☺6=8922.5☺60=20230306-11:30:16.738☺10=243☺ Status: Partially filled 7/50 @8922 EXC=>FIX 8=FIX.4.4☺9=205☺35=8☺49=EXC☺56=FIX☺34=38☺52=20230306-11:30:16.750☺369=36☺37=80☺11=E06☺17=39A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=8☺31=8922☺151=35☺14=15☺6=8922.233333☺60=20230306-11:30:16.738☺10=187☺ Status: Partially filled 15/50 @8922 EXC=>FIX 8=FIX.4.4☺9=207☺35=8☺49=EXC☺56=FIX☺34=39☺52=20230306-11:30:16.751☺369=36☺37=80☺11=E06☺17=40A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=3☺31=8921.5☺151=32☺14=18☺6=8922.111111☺60=20230306-11:30:16.738☺10=001☺ Status: Partially filled 18/50 @8922 EXC=>FIX 8=FIX.4.4☺9=205☺35=8☺49=EXC☺56=FIX☺34=40☺52=20230306-11:30:16.751☺369=36☺37=80☺11=E06☺17=41A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=8☺31=8921☺151=24☺14=26☺6=8921.769231☺60=20230306-11:30:16.738☺10=176☺ Status: Partially filled 26/50 @8921 EXC=>FIX 8=FIX.4.4☺9=207☺35=8☺49=EXC☺56=FIX☺34=41☺52=20230306-11:30:16.751☺369=36☺37=80☺11=E06☺17=42A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=1☺31=8920.5☺151=23☺14=27☺6=8921.722222☺60=20230306-11:30:16.738☺10=005☺ Status: Partially filled 27/50 @8921 EXC=>FIX 8=FIX.4.4☺9=200☺35=8☺49=EXC☺56=FIX☺34=42☺52=20230306-11:30:16.751☺369=36☺37=80☺11=E06☺17=43A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=4☺31=8920☺151=19☺14=31☺6=8921.5☺60=20230306-11:30:16.738☺10=174☺ Status: Partially filled 31/50 @8921 EXC=>FIX 8=FIX.4.4☺9=205☺35=8☺49=EXC☺56=FIX☺34=43☺52=20230306-11:30:16.751☺369=36☺37=80☺11=E06☺17=44A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=1☺31=8919.5☺151=18☺14=32☺6=8921.4375☺60=20230306-11:30:16.738☺10=179☺ Status: Partially filled 32/50 @8921 EXC=>FIX 8=FIX.4.4☺9=205☺35=8☺49=EXC☺56=FIX☺34=44☺52=20230306-11:30:16.752☺369=36☺37=80☺11=E06☺17=45A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=4☺31=8919☺151=14☺14=36☺6=8921.166667☺60=20230306-11:30:16.738☺10=196☺ Status: Partially filled 36/50 @8921 EXC=>FIX 8=FIX.4.4☺9=206☺35=8☺49=EXC☺56=FIX☺34=45☺52=20230306-11:30:16.752☺369=36☺37=80☺11=E06☺17=46A☺150=F☺39=1☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=6☺31=8918.5☺151=8☺14=42☺6=8920.785714☺60=20230306-11:30:16.738☺10=251☺ Status: Partially filled 42/50 @8920 EXC=>FIX 8=FIX.4.4☺9=200☺35=8☺49=EXC☺56=FIX☺34=46☺52=20230306-11:30:16.752☺369=36☺37=80☺11=E06☺17=47A☺150=F☺39=2☺581=2☺55=CA☺54=2☺38=50☺40=2☺44=8900☺59=0☺18=2☺529=D☺32=8☺31=8900☺151=0☺14=50☺6=8917.46☺60=20230306-11:30:16.738☺10=191☺ Status: Filled 50/50 @8917
Command: --history-session (-hs)
For the given session ID (pair SenderCompID and TargetCompID) build the history for FIX session.
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=.*" ./messages.in -hs "SNQX" "DBSUSH" FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "messages.in"... 8=FIX.4.2☺9=73☺35=A☺49=SNQX☺56=DBSUSH☺34=1☺50=SNPY☺52=20080109-13:32:04.000☺98=0☺108=60☺10=242☺ State "SNQXDBSUSH" I: "Wait for confirm logon" 20080109-13:32:04.000 8=FIX.4.2☺9=87☺35=1☺49=SNQX☺56=DBSUSH☺34=2☺50=SNPY☺52=20080109-13:32:09.000☺112=Gap Fill Completed...☺10=169☺ State "SNQXDBSUSH" I: "Wait for confirm logon" 20080109-13:32:09.000 8=FIX.4.2☺9=56☺35=5☺49=SNQX☺56=DBSUSH☺34=3769☺52=20080109-21:25:59.000☺10=145☺ State "SNQXDBSUSH" I: "Wait for confirm logout" 20080109-21:25:59.000 8=FIX.4.2☺9=76☺35=A☺49=SNQX☺56=DBSUSH☺34=3782☺50=SNPY☺52=20080109-21:27:00.000☺98=0☺108=60☺10=151☺ State "SNQXDBSUSH" I: "Wait for confirm logout" 20080109-21:27:00.000 8=FIX.4.2☺9=68☺35=2☺49=SNQX☺56=DBSUSH☺34=3783☺52=20080109-21:27:00.000☺7=6248☺16=0☺10=159☺ State "SNQXDBSUSH" I: "Wait for confirm logout" 20080109-21:27:00.000 8=FIX.4.2☺9=97☺35=4☺49=SNQX☺56=DBSUSH☺34=3770☺43=Y☺52=20080109-21:27:00.000☺122=20080109-21:27:00☺123=Y☺36=3784☺10=093☺ State "SNQXDBSUSH" I: "Wait for confirm logout" 20080109-21:27:00.000 8=FIX.4.2☺9=90☺35=1☺49=SNQX☺56=DBSUSH☺34=3784☺50=SNPY☺52=20080109-21:27:05.000☺112=Gap Fill Completed...☺10=070☺ State "SNQXDBSUSH" I: "Wait for confirm logout" 20080109-21:27:05.000 Completed.
Command: --search (-s)
FIX Grep performs a context search in the loaded data by some plain text. Human readable text can be specified for search e.g. “39=0” or “OrdStatus=0” or “OrdStatus=New”.
Each message is converted to 3 forms: raw FIX message, detailed FIX message (tags are replaced by fields names) and wide form (tags are replaced by field names, values when possible are replaced by descriptions). Search is performed in all these 3 lists.
fixgrep "35=.*" ./message.in -s "Symbol=QHN"
Configuration file: -c (–config)
All keys can be specified in the configuration file. All extra spaces and new lines are removed while file is parsed. Key cannot be combined with other keys. Configuration file content is interpreted as it was passed in command line.
fixgrep --config cmd.txt
"35=.*" ./me* -m 55 "WJJ" -L -C
Command: --send-tcp
This command is used to establish TCP connection and send sequence of bytes from a file. E.g. replay messages from session log.
c:\Program Files (x86)\B2BITS\FIXGrep>fixgrep "35=.*" ./messages.in --send-tcp localhost 9106 FIXGrep 1.5.0.135 Copyright (C) B2BITS EPAM Systems Company. All Rights Reserved. Processing file "messages.in"... Sent 908487 of 908487 bytes.