EVL networking in a nutshell

EVL features a simple network stack which currently implements the UDP protocol from the PF_INET domain (IPv4), and raw ethernet datagrams from the PF_PACKET domain, both over an ethernet medium. This provides datagram-oriented networking support to applications with real-time communication requirements, through the common networking device infrastructure. Out-of-band communication happens via out-of-band network ports as provided by Dovetail.

Out-of-band network ports

An out-of-band network port as defined by the Dovetail interface is a communication channel hosted by a regular Linux network device (netdev), which provides a way for real-time applications to send and receive packets from the out-of-band execution stage. Both real (physical) devices and virtual (VLAN) interfaces sitting on top of the latter can become out-of-band network ports independently from each other. However, when a VLAN device is turned into an out-of-band port, input diversion is automatically enabled on the underlying real device.

Enabling a network device as an out-of-band port can be done in three ways:

  • Using the `evl net -ei <interface>’ management command.

  • Calling the evl_net_enable_port() service from your application.

  • writing a non-zero value to /sys/class/net/<netdev>/oob_port. Conversely, writing zero disables the port.

Enabling a network device as an out-of-band port does NOT magically make its driver oob-capable. Doing so always requires code changes to the stock kernel driver for the relevant network interface controller. Enabling the port merely allows out-of-band traffic to be submitted to the network device by applications running on the out-of-band stage. If the driver is not oob-capable, ingress packets are (usually) received from the NAPI poll routine running on the in-band stage then forwarded to out-of-band consumers, egress packets are relayed from out-of-band senders to the driver transmit routine running on the in-band stage. In other words, when the driver is not oob-capable, no real-time requirement can be met by EVL, the worst-case latency depends on the real-time capabilities of the in-band kernel.

Redirecting the ingress traffic to EVL

EVL relies on Dovetail which can redirect ingress traffic between NIC drivers and the EVL core directly, either from the in-band or out-of-band stages, depending on whether such drivers have been adapted in order to handle traffic from the out-of-band stage. EVL can recognize the ethernet traffic it should handle based on two types of input filters:

  • VLAN tagging. Packets flowing into an out-of-band VLAN are redirected to the EVL network stack, others are left for handling by the regular / in-band network stack. Any valid VLAN number may be assigned to EVL for accepting traffic from it.

Since the 802.1Q standard has been around for quite some time by now, most ethernet switches should pass on frames with the ethertype information set to the 802.1Q TPID “as is” to some hardware port, and they should also be able to cope with the four additional octets involved in VLAN tagging without having to lower the MTU everywhere (most equipments even support jumbo frames these days).

  • eBPF filtering. An eBPF program can be installed on any network device hosting an out-of-band port to decide whether an ingress packet should be processed by the EVL network stack, the regular (in-band) stack, or dropped.

By returning the appropriate status code, the filter program can decide to either:

  • Accept the packet for handling by the EVL network stack (EVL_RX_ACCEPT).
  • Hand over the packet to the regular / in-band stack instead (EVL_RX_SKIP).
  • Postpone the decision to applying the VLAN matching rule (EVL_RX_VLAN).
  • Drop the packet, which won’t enter any network stack as a consequence (EVL_RX_DROP).

When present, the eBPF program takes precedence over the VLAN matching rule. In absence of eBPF filter, the VLAN matching rule applies. Packets which EVL did not accept are transparently forwarded to the regular (in-band) network stack.

The following diagram illustrates the flow of an incoming packet from the network interface controller to the service access point in the application, i.e. the socket.

Alt text

Egress traffic handling

EVL is in charge of building the outgoing packets for conveying the data sent by applications which run on the out-of-band execution stage. Analogously to receiving the ingress traffic, EVL can either send the outgoing packets directly from the out-of-band stage, or hand them over to the regular network stack on the in-band stage, depending on whether the NIC driver has been adapted in order to handle traffic from the out-of-band stage directly.

It is fundamental to understand that enabling an out-of-band port on a network device does not per se ensure end-to-end real-time communication through it. For this guarantee to be provided, the NIC driver must have been made oob-capable, so that packet transmit and receive operations to/from the hardware do happen directly from the out-of-band stage. However, in any case, EVL still guarantees that threads calling out-of-band receive/transmit services won’t demote to the in-band execution stage when doing so.

Typical use cases

Dedicating a network interface to out-of-band traffic

When multiple network interfaces are available from the hardware platform, dedicating one or more of them to sending and/or receiving out-of-band traffic is best to decrease network latency for real-time applications.

Sharing a network interface between in-band and out-of-band traffic

The hardware platform might have a single network interface available, in which case VLAN tagging may come in handy to direct incoming packets either to the EVL network stack or the regular (in-band) one. Obviously, this comes at a cost with respect to latency, since the in-band traffic might slow down the out-of-band packets at the hardware level. Likewise, the hardware would be shared for transmitting both in-band and out-of-band originated packets. This said, depending on the real-time requirements, that cost may still be within budget for many applications.

Dealing with complex out-of-band detection rules

An eBPF program allows deep inspection of the packet data before issuing any decision about which network stack should handle the traffic. One may rely on this feature to implement complex out-of-band traffic detection rules.

Out-of-band support in NIC drivers

Unlike Xenomai 3 with the RTnet stack, EVL provides a network stack which does not require EVL-specific drivers. Instead, the capabilities of the stock NIC drivers can be extended to supporting out-of-band I/O operations for dealing with ingress and egress traffic, based on facilities provided by Dovetail in that area. If a driver does not advertise out-of-band I/O capabilities, EVL automatically offloads the final I/O operations to the in-band network stack for talking to the network device, allowing the application code to keep running on the out-of-band stage in parallel.

Although EVL does not require the NIC driver code to be oob-capable, i.e. conveying ingress and egress traffic directly from the out-of-band execution stage, having such support in place is the only way to have a complete, end-to-end real-time networking path, from the Ethernet wire to the application code. In other words, one may still use stock ethernet controller drivers along with the EVL network stack, at the expense of the real-time performance which would depend on the low-latency capabilities of the host kernel.

Out-of-band packet routing

Input routing

In short, EVL does no input routing. At all. All ingress packets accepted by EVL should either be delivered to a receiver on the local host or dropped, there is no forwarding of any sort of such data to remote hosts.

Output routing

EVL does a basic form of output routing for IPv4 over Ethernet, enough to satisfy the requirements of the UDP protocol for determining the proper output device to submit outgoing packets to, along with the MAC address to put into the Ethernet destination field of those packets. However, an out-of-band sender cannot perform direct lookups into the regular (in-band) kernel tables to find the routing information it needs, this would contradict the stage separation rule. In order to address this issue, EVL maintains two front caches containing the related information:

  • a table of routes indexed on the IPv4 address of the receiving peer. Each entry refers to the routing information determined by the in-band kernel for those outgoing routes (i.e. a struct rt_table item).

  • a table of ARP entries indexed on a composite key combining the output device found by the in-band routing process for reaching the peer, and the IPv4 address where it may be reached from that device. Each entry contains the MAC Ethernet address to be written to the destination field of a packet for sending it to the given peer. This cache tracks the updates to its counterpart maintained by the regular (in-band) kernel.

In other words, EVL does not calculate any route, does not determine any destination MAC address by itself: it simply collects and learns this information from the in-band kernel when the latter estabishes a new route via some network device. When an out-of-band port is enabled for that device, EVL receives a notification about such route, recording this information into its own caches, which in turn can be accessed directly from the out-of-band stage by applications.