How to use SO_BUSY_POLL socket option


There is a new low latency socket option available in RHEL 7.1 called SO_BUSY_POLL that can be used with FIX Antenna C++ to reduce the latency of message receiving. The socket option can be applied to a small number of FIX session connections created in AGGRESSIVE_RECEIVE or AGGRESSIVE_SEND_AND_RECEIVE modes. The number of connections covered should be kept small because each socket in this mode keeps one CPU core completely busy.

SO_BUSY_POLL option should be supported by the appropriate network device driver, RHEL 7.1 documentation lists the drivers that currently support this option, see “ Configuring busy polling” for actual driver list:

You can find more details about this option here: Intel Whitepaper - Open Source Kernel Enhancements for Low-Latency Sockets using Busy Poll

Linux kernel support:

Kernel should be built with CONFIG_NET_RX_BUSY_POLL compilation option which is enabled by default in RHEL 7.x, no changes are needed for this OS.

To check if this option is enabled, the following command can be used:

# cat /boot/config-3.10.0-229.el7.x86_64 | grep CONFIG_NET_RX_BUSY_POLL

Expected result is: “CONFIG_NET_RX_BUSY_POLL=y”

To check if busy poll feature is available for a specific network device, run:

# ethtool -k device | grep "busy-poll"

Expected result is: “busy-poll: on [fixed]”

Configuration Steps

In order to use this option in Fix Antenna C++ the following steps should be made:

  1. Configure the system to use this specific busy poll wait time value:

    # sysctl net.core.busy_poll=600000

    Engine::SessionExtraParameters params;
    params.socketPriority_ = Engine::AGGRESSIVE_ RECEIVE_SOCKET_OP_PRIORITY;
  3. Enable busy poll option via Fix Antenna API for this session:

    params.socketBusyPollTime_ = 600000         //   set 600 milliseconds

    set aggressive receive delay value to 500ms.

    params. aggressiveReceiveDelay_ = 500   // set 500 milliseconds
  4. Alternatively you can set the above properties in file rather than via the API: = 600000 = 500

Note that SO_BUSY_POLL socket option is supported only by FIX Antenna 2.15.0 C++ and higher.

Test Results

A modified Benchmark/Latency test was used in these tests. There were inserted 10ms delays between posts to make the test more realistic and the message rate similar to what occurs in practice. Number of messages = 1000.

The test was conducted on RHEL7.1i7 3.6 GHz CPUMyricom 10G NICs; packages were passed through 10G cross-over cableeverything was installed on a single machine.

  1. SO_BUSY_POLL is enabled using SocketBusyPollTime. This is a new mode. = 600000 = 500
    iterations = 1,000

    Total latency (nanoseconds):   <- time from sending till receiving the message
    MIN: 13549
    MAX: 130952
    AVG: 16396

  2. SO_BUSY_POLL is disabled.
    Total latency (nanoseconds):
    MIN: 39530
    MAX: 166082
    AVG: 47499

  3. SO_BUSY_POLL is disabled, but “busy poll” is enabled inside of Antenna using installation. = 0

    Total latency (nanoseconds):
    MIN: 15854
    MAX: 123549
    AVG: 18831

As we can see the enabled SO_BUSY_POLL option is the best choice.

The results of one more test are provided below. It was the test in which messages were sent without pauses. The latency is less when the SO_BUSY_POLL option is enabled though this test is not from the real world.

  1. SO_BUSY_POLL is enabled using SocketBusyPollTime = 600000 = 500
    iterations = 1,000,000

    Total latency (nanoseconds):
    MIN: 12153
    MAX: 119149
    AVG: 15602

  2. SO_BUSY_POLL is disabled = 0

    Total latency (nanoseconds):
    MIN: 13968
    MAX: 409688
    AVG: 17255