EVL features a simple network stack which currently implements the UDP
protocol from the PF_INET
domain (IPv4), and raw ethernet datagrams
from the PF_PACKET
domain, both over an ethernet medium. This
provides datagram-oriented networking support to applications with
real-time communication requirements, through the common networking
device infrastructure. Out-of-band communication happens via
out-of-band network ports as
provided by Dovetail.
An out-of-band network port as defined by the Dovetail interface is a communication channel hosted by a regular Linux network device (netdev), which provides a way for real-time applications to send and receive packets from the out-of-band execution stage. Both real (physical) devices and virtual (VLAN) interfaces sitting on top of the latter can become out-of-band network ports independently from each other. However, when a VLAN device is turned into an out-of-band port, input diversion is automatically enabled on the underlying real device.
Enabling a network device as an out-of-band port can be done in three ways:
Using the `evl net -ei <interface>’ management command.
Calling the evl_net_enable_port() service from your application.
writing a non-zero value to
/sys/class/net/<netdev>/oob_port
. Conversely, writing zero
disables the port.
Enabling a network device as an out-of-band port does NOT magically make its driver oob-capable. Doing so always requires code changes to the stock kernel driver for the relevant network interface controller. Enabling the port merely allows out-of-band traffic to be submitted to the network device by applications running on the out-of-band stage. If the driver is not oob-capable, ingress packets are (usually) received from the NAPI poll routine running on the in-band stage then forwarded to out-of-band consumers, egress packets are relayed from out-of-band senders to the driver transmit routine running on the in-band stage. In other words, when the driver is not oob-capable, no real-time requirement can be met by EVL, the worst-case latency depends on the real-time capabilities of the in-band kernel.
EVL relies on Dovetail which can redirect ingress traffic between NIC drivers and the EVL core directly, either from the in-band or out-of-band stages, depending on whether such drivers have been adapted in order to handle traffic from the out-of-band stage. EVL can recognize the ethernet traffic it should handle based on two types of input filters:
Since the 802.1Q standard
has been around for quite some time by now, most ethernet switches
should pass on frames with the ethertype
information set to the
802.1Q TPID “as is” to some hardware port, and they should also be
able to cope with the four additional octets involved in VLAN tagging
without having to lower the MTU everywhere (most equipments even
support jumbo frames these days).
By returning the appropriate status code, the filter program can decide to either:
EVL_RX_ACCEPT
).EVL_RX_SKIP
).EVL_RX_VLAN
).EVL_RX_DROP
).When present, the eBPF program takes precedence over the VLAN matching rule. In absence of eBPF filter, the VLAN matching rule applies. Packets which EVL did not accept are transparently forwarded to the regular (in-band) network stack.
The following diagram illustrates the flow of an incoming packet from the network interface controller to the service access point in the application, i.e. the socket.
EVL is in charge of building the outgoing packets for conveying the data sent by applications which run on the out-of-band execution stage. Analogously to receiving the ingress traffic, EVL can either send the outgoing packets directly from the out-of-band stage, or hand them over to the regular network stack on the in-band stage, depending on whether the NIC driver has been adapted in order to handle traffic from the out-of-band stage directly.
It is fundamental to understand that enabling an out-of-band port on a network device does not per se ensure end-to-end real-time communication through it. For this guarantee to be provided, the NIC driver must have been made oob-capable, so that packet transmit and receive operations to/from the hardware do happen directly from the out-of-band stage. However, in any case, EVL still guarantees that threads calling out-of-band receive/transmit services won’t demote to the in-band execution stage when doing so.
When multiple network interfaces are available from the hardware platform, dedicating one or more of them to sending and/or receiving out-of-band traffic is best to decrease network latency for real-time applications.
The hardware platform might have a single network interface available, in which case VLAN tagging may come in handy to direct incoming packets either to the EVL network stack or the regular (in-band) one. Obviously, this comes at a cost with respect to latency, since the in-band traffic might slow down the out-of-band packets at the hardware level. Likewise, the hardware would be shared for transmitting both in-band and out-of-band originated packets. This said, depending on the real-time requirements, that cost may still be within budget for many applications.
An eBPF program allows deep inspection of the packet data before issuing any decision about which network stack should handle the traffic. One may rely on this feature to implement complex out-of-band traffic detection rules.
Unlike Xenomai 3 with the RTnet stack, EVL provides a network stack which does not require EVL-specific drivers. Instead, the capabilities of the stock NIC drivers can be extended to supporting out-of-band I/O operations for dealing with ingress and egress traffic, based on facilities provided by Dovetail in that area. If a driver does not advertise out-of-band I/O capabilities, EVL automatically offloads the final I/O operations to the in-band network stack for talking to the network device, allowing the application code to keep running on the out-of-band stage in parallel.
Although EVL does not require the NIC driver code to be oob-capable, i.e. conveying ingress and egress traffic directly from the out-of-band execution stage, having such support in place is the only way to have a complete, end-to-end real-time networking path, from the Ethernet wire to the application code. In other words, one may still use stock ethernet controller drivers along with the EVL network stack, at the expense of the real-time performance which would depend on the low-latency capabilities of the host kernel.
In short, EVL does no input routing. At all. All ingress packets accepted by EVL should either be delivered to a receiver on the local host or dropped, there is no forwarding of any sort of such data to remote hosts.
EVL does a basic form of output routing for IPv4 over Ethernet, enough to satisfy the requirements of the UDP protocol for determining the proper output device to submit outgoing packets to, along with the MAC address to put into the Ethernet destination field of those packets. However, an out-of-band sender cannot perform direct lookups into the regular (in-band) kernel tables to find the routing information it needs, this would contradict the stage separation rule. In order to address this issue, EVL maintains two front caches containing the related information:
a table of routes indexed on the IPv4 address of the receiving peer.
Each entry refers to the routing information determined by the
in-band kernel for those outgoing routes (i.e. a struct rt_table
item).
a table of ARP entries indexed on a composite key combining the output device found by the in-band routing process for reaching the peer, and the IPv4 address where it may be reached from that device. Each entry contains the MAC Ethernet address to be written to the destination field of a packet for sending it to the given peer. This cache tracks the updates to its counterpart maintained by the regular (in-band) kernel.
In other words, EVL does not calculate any route, does not determine any destination MAC address by itself: it simply collects and learns this information from the in-band kernel when the latter estabishes a new route via some network device. When an out-of-band port is enabled for that device, EVL receives a notification about such route, recording this information into its own caches, which in turn can be accessed directly from the out-of-band stage by applications.