How to use FIXEye-Splunk integration capabilities
General Notes
Splunk is a system for searching, monitoring, and analyzing machine-generated big data. It is often used for financial or trading information storage as well. This document is intended to describe how to configure FIXEye-Splunk connection and search for fix data in Splunk storage using FIXEye application.
Managing connections
Create new connection
- Open 'Add connection...' dialog by clicking '>' button or choose 'Add Splunk...' menu item with right-click context menu in file window.
- Fill-in all input boxes in the dialog and presses 'Add' button. For connection to work user must be previously created in Splunk system. URL is web link to Splunk system.
- The newly created connection to Splunk appears in file window.
Edit connection
- Right-click an existing Splunk connection and choose 'Edit connection'.
- 'Edit connection...' dialog appears. Input boxes of the dialog contain corresponding values of the connection.
- Change values. Presses 'Save' button if you want changes to be saved or use 'Cancel' button for changes to be discarded.
Remove connection
- Right-clicks an existing Splunk connection and choose 'Remove connection'.
- Confirmation dialog appears. Press "Yes" if you want to remove connection and "No" if you want to keep it.
Types of search
it is important to be aware that to search using FIXEye-Splunk connection means that all searching job is taking place on Splunk side. FIXEye sends search string to Splunk, gets the result and displays it. So FIXEye has to use splunk syntax and it is really different from FIXEye native syntax. For user convenience some simple patterns such as tag=value may be used but for more complicated searching syntax will be different and it is recommended to study splunk documentation.
Search in Raw FIX messages
It is the case when FIX messages are imported in Splunk "as is". On the picture below it is seen that each event is a FIX message and nothing more.
This is the main scenario of using Splunk as a back-end for FIXEye.
Example1: simple search query (49=ICE)
Example2: filter + search query (49=ICE in filter and 34=2 in search box)
Regular expressions work in filter but doesn't work in search queries.
Search in Log record
FIX messages can be a part of some log record having several fields related to a tool produced this record with one field containing raw FIX message. This can be a result when an application forwards its logs into Splunk and includes raw FIX messages in those logs.
For example let us have such records in Splunk:
FIXEye "show all" for this pattern will be:
It can be seen that only record with fix message is displayed. It is possible to use search criteria concerning other records if you know the name of the fields or possible values.
Examples:
Example3: search query = AME
Example4: search query = IC* (note! to search for a part of the string * symbol must be used.)
For more complicated search cases user must use splunk syntax and the search string must begin with "search". FIXEye passes this search string without any processing to Splunk and awaits raw FIX messages Splunk will return in the first field.
Example5: string "search "35=*" AND "49=oil" | rex "(?<Message>8=FIX.+\x0110=\d{3}\x01)" | search Message != '' | table Message"
Explanation:
| part of expression | description |
|---|---|
| search | In splunk it is a command for search. In FIXEye it is a mandatory word meaning usage of splunk syntax in search string. |
| "35=*" AND "49=oil" | An expression defining what we want to find. Quotes mean that we want to search text pattern not a splunk field. * means any symbol. AND is logical conjunction. Expression means find patterns containing both 35=* and 49=oil |
| | | a pipeline symbol - results from the output of previous processing step are put to the input of next step |
| rex | field extraction command. It takes as an input a regular expression, searches in the result of previous step and puts the search result in a field |
| "(?<Message>8=FIX.+\x0110=\d{3}\x01)" | Input for rex command. Regular expression providing extraction of FIX message (which start with 8=FIX and ends with SOH10=any3digitsSOH sequence). <Message> is the name of a field containing extracted FIX message put in brackets. |
| search Message !='' | search for FIX messages that are not empty ('' is two apostrophes). Message is the name from the previous pipeline step |
| table Message | Pick only Message group from all groups that is in the output of previous pipeline step |
The search result is shown below.
Note, that FIX message must contain SOH symbols as delimiters. The space delimiters will not work!
There is a complicated way to deal with space delimiters. FIX message should be converted on the fly by adding a step into Splunk search pipeline to replace space symbol used as delimiter by SOH symbol which is supported by FIXEye. To do so please check Splunk documentation.
Troubleshooting
- "Connection is not added" error. You can see exclamation marks on the right side of the fields. When pointer is over the mark pop-up explanations can be seen.
Incorrect URL - change URL, it must have correct form, e.g. https://localhost:8089 for local machine.
User name can't be empty - put in valid user name
Password can't be empty - put in password for user
- "Error occurred while parsing messages (in Details: 401 Unauthorized Warning: login failed )" error - Check user name and password in connection properties