How to Use RX/TX Network Timestamping in FIXAntenna C++

Overview

FIXAntenna C++ provides network timestamping capabilities leveraging the Linux kernel's SO_TIMESTAMPING feature. This functionality was initially introduced primarily for UDP timestamping (Linux 2.6.30) and has evolved, notably up to Linux kernel 6.2 which improved TCP stream tracing reliability with the SOF_TIMESTAMPING_OPT_ID_TCP option.

OpenOnload also adopted SO_TIMESTAMPING (since version 8.2), offering users a unified API compared to previous proprietary methods.

This feature allows capturing several nanosecond-precise time points as TCP stream data travels through the system. It provides detailed timing information from the moment an application initiates a send until the counterparty confirms receipt of the bytes.

Timestamping Data Provided

The SO_TIMESTAMPING feature provides the following time points:

• For RX (receive) timestamping:

  • The time point when the final byte of the data read from the socket was received by the NIC.

  • The time point when the final byte of the data read from the socket was received by the Linux kernel.

• For TX (transmit) timestamping:

  • The time point when the final byte of the data sent to the socket was sent to the kernel's TCP packet scheduler.

  • The time point when the final byte of the data sent to the socket was sent to the NIC driver from the kernel.

  • The time point when the final byte of the data sent to the socket was sent to the wire.

  • The time point when the counterparty has acknowledged receiving the TCP byte stream up to the final byte of the data sent to the socket.

Configuration

Timestamping needs to be configured at both the engine and session levels.

Engine Configuration

• TimestampingHandlerCpuAffinity: Specifies the CPU affinity for the dedicated thread that processes TX timestamps.

  • Recommendation: Assign a dedicated CPU core to this thread to ensure timestamps are processed with minimal delay.

  • Format: Use the standard format for CPU affinity settings within FIXAntenna C++.

Session Configuration

Timestamping modes are enabled within the session's SessionExtraParameters.

• SessionRXTXTimestamping: Defines the specific timestamping points to be captured for the session.

  • Format: Specify desired modes as a comma-separated list.

  • Possible Values:

    • RX_HARDWARE: Capture time when the last byte is received from the wire by the NIC.

    • RX_SOFTWARE: Capture time when the last byte is received by the kernel/driver from the NIC.

    • RX_USERSPACE: Capture time when the last byte is received by FIXAntenna from the kernel/driver.

    • TX_HARDWARE: Capture time when the last byte is sent to the wire.

    • TX_SOFTWARE: Capture time when the last byte is sent from the kernel/driver.

    • TX_USERSPACE: Capture time when the last byte is sent from FIXAntenna to the kernel/driver.

    • TX_SCHEDULER: Capture time when the kernel's TCP packet scheduler receives the last byte.

    • TX_ACK: Capture time when the counterparty acknowledges receipt of the TCP stream up to the last byte sent.

Receiving Timestamp Data

Receiving RX Timestamps

Receiving RX timestamps is relatively straightforward. After configuring the socket options via session settings, the timestamps become available within a special control message (cmsg) attached to the data received from the socket.

The process involves:

1. Calling recvmsg with a properly configured msghdr structure, including a buffer for control messages.

2. Iterating through the control messages (cmsg) received using CMSG_FIRSTHDR and CMSG_NXTHDR.

3. Identifying the timestamp message by checking if cmsg->cmsg_level == SOL_SOCKET and cmsg->cmsg_type == SO_TIMESTAMPING_NEW.

4. Extracting the timestamp data using CMSG_DATA(cmsg) and casting it to the appropriate structure (e.g., scm_timestamping64*).

Receiving TX Timestamps

Receiving TX timestamps is more complex and involves several nuances:

1. Retrieval Method: TX timestamps must be received from the socket's error queue using recvmsg with the MSG_ERRQUEUE flag.

2. Asynchronous Nature: Timestamp messages arrive asynchronously. Their order relative to application sends or even other timestamp messages is not guaranteed due to TCP's nature (reordering, retransmissions).

3. Availability Delay: Timestamps are often not available instantly and, in some network conditions or error scenarios, may not become available (and thus cannot be received) at all.

4. OPT_ID Matching: The identifier (OPT_ID) associated with a TX timestamp corresponds to the sequence number of the last byte of the sent buffer within the entire TCP stream. This ID is generated by the kernel/NIC, not set by the application. The application must calculate the expected last byte position for each sent buffer to match it with the received timestamp's OPT_ID.

5. Variable Latency: The time it takes for TX timestamps to arrive in the MSG_ERRQUEUE can range significantly, from nanoseconds to potentially dozens of seconds, depending on TCP stream state and network conditions (e.g., waiting for acknowledgements).

6. Buffer Consumption: Messages in the MSG_ERRQUEUE consume the socket's buffer space.

7. Circular Buffer: The MSG_ERRQUEUE buffer is circular. If timestamps are not consumed quickly enough, new incoming error messages (including newer timestamps) can overwrite older ones, leading to data loss.

User API for Accessing Timestamps

FIXAntenna C++ provides an API to manage the complexities of timestamp retrieval, particularly for TX timestamps. The design makes the following assumptions:

• It is the user's responsibility to map sent/received messages to their corresponding timestamps. The library provides the data and the means (TimepointHolder) for this mapping.

• It is the user's responsibility to decide how to react if timestamps indicate network or processing issues. The library provides the means to check the status.

• It is the user's responsibility to handle potential delays in receiving TX timestamps. The library provides the data as soon as it's available but does not inherently alter FIX message exchange logic based on TX timestamp arrival times.

• Receiving TX timestamps immediately upon arrival is considered less critical than providing a reliable way for the user to retrieve them when needed (or wait for them).

API Components

• FIXMessage::getTimepointHolder() const:

  • Returns a Utils::Timestamping::TimepointHolder::Ptr (which is a std::shared_ptr<TimepointHolder>).

  • This holder instance is available immediately after a message is submitted to the engine for sending, or immediately for received messages.

  • Each instance is unique per send/receive operation. Users can store this pointer along with any relevant application context.

• class TimepointHolder:

  • TimepointStatus getTimepointStatus(Timepoint point) [[nodiscard]]:

    • Checks the current status of a specific timestamp point (e.g., TX_HARDWARE, RX_SOFTWARE). See statuses below.

  • timespec getTimepoint(Timepoint point):

    • Retrieves the actual timestamp value (timespec).

    • Important: Only call this if getTimepointStatus returns READY.

    • Throws ValueUnavailableException if the status is UNAVAILABLE.

    • Throws ValueNotReadyException if the status is NOT_REACHED or WAITING.

Timepoint Statuses

The getTimepointStatus method can return one of the following TimepointStatus values:

• NOT_REACHED: The message flow has not yet reached the point where the timestamp is generated or its promise is set. (Note: Currently not used but potentially applicable for future features).

• WAITING: The message has passed the point where the timestamp should be generated (the internal promise is set), and FIXAntenna is expecting the timestamp value to arrive soon (e.g., waiting for kernel/NIC response or ACK).

• READY: The timestamp value has been received by the library and is available to be read using getTimepoint().

• UNAVAILABLE: This timestamp type is either turned off in the session configuration or could not be retrieved (and will not be received).

Internal Mechanics

1. When a FIX message enters the library for sending, it's associated with a TimepointHandler structure containing futures for all configured time points.

2. The corresponding promise set is stored internally, ready for the next buffer to be sent.

3. Once the message is serialized to a byte array, the promise set is linked to the calculated position (sequence number) of the last byte of that array within the TCP stream.

4. A dedicated timestamp processing thread (whose affinity can be set via TimestampingHandlerCpuAffinity) receives timestamp data (from recvmsg on the main socket for RX, or recvmsg with MSG_ERRQUEUE for TX).

5. As timestamps arrive, this thread matches them to the correct promise (using the last byte position/OPT_ID for TX) and fulfills the promise, making the data available via the TimepointHolder. This separation minimizes interference with the main FIX message processing flow.

Applications and Use Cases

Network timestamping enables various latency-sensitive and diagnostic applications:

• Network diagnostic and troubleshooting.

• Latency-sensitive order execution (High-Frequency Trading - HFT).

• Post-trade latency analysis (identifying delays in order routing and processing).

• Market data alignment (correlating FIX messages with market data snapshots based on precise timing).

• Smart order routing (real-time latency monitoring to adjust routing decisions).

• Nanosecond-precise trade replay and debugging.

• Tuning wire-to-market latency (estimating execution priority or slippage under load).

• Precise monitoring and alerting (real-time counterparty latency measurements to detect issues).

Important Considerations

• Byte Stream Based: Timestamping is tied to byte positions within the TCP stream, not TCP packet boundaries or logical FIX messages directly. A single send or receive call's data might cross packet boundaries.

• Matching Challenges: If your application separates networking and message processing logic, matching parsed FIX messages back to the exact byte positions reported by timestamps can be challenging.

• TX Timestamp Delays: There is no 'reasonable' fixed timeout to wait for TX timestamps. Waiting is necessary for diagnostics, but delays are inherent. Dropping timestamp data because it's "late" undermines network diagnostic capabilities precisely when they might be needed most.

• Mapping to FIX Messages: There is no foolproof, built-in way to automatically associate a received timestamp directly with a specific logical FIX message. While message sequence numbers seem like an obvious solution, the FIX transport layer might send other messages (e.g., Test Requests, Heartbeats) independently, complicating sequence-based mapping for the user. Users must implement their own mapping logic, typically using the unique TimepointHolder instance associated with each send/receive.