Out-of-band I/O services

Communicating with real-time capable device drivers

Using the EVL kernel API, you can extend an existing driver for supporting out-of-band I/O operations, or even write one from scratch. Both character-based I/O and socket protocol drivers are supported.

On the user side, application can exchange data with, send requests to these real-time capable drivers from the out-of-band execution stage with the a couple of additional services libevl provides.

You may notice that several POSIX file I/O services such as open(2), socket(2), close(2), fcntl(2), mmap(2) and so on have no out-of-band counterpart in the following list. The reason is that we don’t need them: opening, closing or mapping a file are inherently non-deterministic operations, which may block for an unspecified amount of time for a number of reasons, depending on the underlying file and current runtime conditions. Besides, those are hardly useful in a time-critical loop.

However, issuing data transfers and control requests to the driver is definitely something we may want to happen within a bounded time, hence directly from the out-of-band execution stage.

Since the EVL core exports every public element as a character device which can be accessed from /dev/evl, libevl can interface with elements from other processes through the out-of-band I/O requests documented here, which are sent to the corresponding devices.

Opening an out-of-band capable I/O channel

Since the EVL core does not redefine the open(2) and socket(2) calls, there has to be a way to tell the kernel code managing the device and/or protocol that we want to enable out-of-band operations.

In most cases, we don’t have to do so though, because the purpose of the corresponding device driver is all about providing out-of-band services, so enabling them for any connecting file is implicit. For instance, most of the drivers accessed through the /dev/evl file hierarchy turn on out-of-band services automatically.

However, some drivers might distinguish between out-of-band capable files and others, providing a different set of services. Typically, a regular in-band driver which is extended in order to handle out-of-band requests too should be told when to do so for any given file.

To meet this requirement, Dovetail introduces the additional open flag O_OOB, which can be passed to open(2) ORed into the flags argument. Similarly, it defines the SOCK_OOB flag which can be passed to socket(2) ORed into the type argument for the same purpose. If the receiving driver implements opt-in out-of-band services, passing this flag when opening a file/socket should enable them.

Not all devices drivers may support out-of-band operations (the overwhelming majority does not). Whether passing either O_OOB or SOCK_OOB to them when opening a file/socket would cause an error, or the flag would just be ignored depends on the driver code.

Out-of-band I/O services

ssize_t oob_read(int efd, void *buf, size_t count)

This is the strict equivalent to the standard read(2) system call, for sending the request from the out-of-band stage to an EVL driver. In other words, oob_read() attempts to read up to count bytes from file descriptor fd into the buffer starting at buf, from the out-of-band execution stage.

The caller must be an EVL thread, which may be switched automatically by the EVL core to the out-of-band execution stage as a result of this call.

  • efd

    A file descriptor obtained by opening a real-time capable driver which we want to read from.

  • buf

    A buffer to receive the data.

  • count

    The number of bytes to read at most, which should fit into buf.

    • oob_read() returns the actual number of bytes read, copied to buf on success. Otherwise, -1 is returned, and errno is set to the error code:

    EBADF if fd does not refer to a real-time capable driver, or fd was not opened for reading.

    EINVAL if fd does not support the .oob_read operation.

    EFAULT if buf points to invalid memory.

    EAGAIN fd is marked as non-blocking (O_NONBLOCK), and the read operation would block.

    Other driver-specific error codes may be returned, such as:

    ENOBUFS fd is a cross-buffer file descriptor, and there is no ring buffer space associated with the outbound traffic (i.e. o_bufsz parameter was zero when creating the cross-buffer).

    EINVAL fd is a cross-buffer file descriptor, and count is greater than the size of the ring buffer associated with the inbound traffic. (i.e. the i_bufsz parameter given when creating the cross-buffer).

    EFBIG fd is a proxy file descriptor, and count is larger than the size of the input buffer as specified in the call to [evl_create_proxy()] (/core/user-api/proxy/#evl_create_proxy).

    EINVAL fd is a proxy file descriptor, and count is not a multiple of the input granularity as specified in the call to [evl_create_proxy()] (/core/user-api/proxy/#evl_create_proxy).

    ENXIO fd is a proxy file descriptor which is not available for input. See EVL_CLONE_INPUT.

    ssize_t oob_write(int efd, const void *buf, size_t count)

    This is the strict equivalent to the standard write(2) system call, for sending the request from the out-of-band stage to an EVL driver. In other words, oob_write() attempts to write up to count bytes to file descriptor fd from the buffer starting at buf, from the out-of-band execution stage.

    The caller must be an EVL thread, which may be switched automatically by the EVL core to the out-of-band execution stage as a result of this call.

  • efd

    A file descriptor obtained by opening a real-time capable driver which we want to read from.

  • buf

    A buffer containing the data to be written.

  • count

    The number of bytes to write starting from buf.

    • oob_write() returns the actual number of bytes written from buf on success. Otherwise, -1 is returned, and errno is set to the error code:

    EBADF if fd does not refer to a real-time capable driver, or fd was not opened for writing.

    EINVAL if fd does not support the .oob_write operation.

    EFAULT if buf points to invalid memory.

    EAGAIN fd is marked as non-blocking (O_NONBLOCK), and the write operation would block.

    Other driver-specific error codes may be returned, such as:

    ENOBUFS fd is a cross-buffer file descriptor, and there is no ring buffer space associated with the outbound traffic (i.e. o_bufsz parameter was zero when creating the cross-buffer).

    EINVAL fd is a cross-buffer file descriptor, and count is greater than the size of the ring buffer associated with the outbound traffic. (i.e. the o_bufsz parameter given when creating the cross-buffer).

    EFBIG fd is a proxy file descriptor, and count is larger than the size of the output buffer as specified in the call to [evl_create_proxy()] (/core/user-api/proxy/#evl_create_proxy).

    EINVAL fd is a proxy file descriptor, and count is not a multiple of the output granularity as specified in the call to [evl_create_proxy()] (/core/user-api/proxy/#evl_create_proxy).

    ENXIO fd is a proxy file descriptor which is not available for output. See EVL_CLONE_OUTPUT.

    int oob_ioctl(int efd, unsigned long request, ...)

    This is the strict equivalent to the standard ioctl(2) system call, for sending the I/O control request from the out-of-band stage to an EVL driver. In other words, oob_ioctl() issues request to file descriptor fd from the out-of-band execution stage.

    The caller must be an EVL thread, which may be switched automatically by the EVL core to the out-of-band execution stage as a result of this call.

  • efd

    A file descriptor obtained by opening a real-time capable driver which we want to send a request to.

  • request

    The I/O control request code.

  • ...

    An optional variable argument list which applies to request.

    • oob_ioctl() returns zero on success. Otherwise, -1 is returned, and errno is set to the error code:

    EBADF if fd does not refer to a real-time capable driver.

    ENOTTY if fd does not support the .oob_ioctl operation, or the driver does not implement request.

    EFAULT if buf points to invalid memory.

    EAGAIN fd is marked as non-blocking (O_NONBLOCK), and the control request would block.

    Other driver-specific error codes may be returned.

    ssize_t oob_recvmsg(int s, struct oob_msghdr *msghdr, const struct timespec *timeout, int flags)

    This is an equivalent to the standard recvmsg(2) system call, for sending the request from the out-of-band stage to an EVL driver with a socket-based interface. In other words, oob_recvmsg() is used to receive messages from an out-of-band capable EVL socket from the out-of-band execution stage.

    The caller must be an EVL thread, which may be switched automatically by the EVL core to the out-of-band execution stage as a result of this call.

  • s

    A socket descriptor obtained from a regular socket(2) call, with the SOCK_OOB flag set in the type argument, denoting that out-of-band services are enabled for the socket.

  • msghdr

    A pointer to a structure containing the multiple arguments to this call, which is described below.

  • timeout

    A time limit to wait for a message before the call returns on error. The built-in clock EVL_CLOCK_MONOTONIC is used for tracking the elapsed time. If NULL is passed, the call is allowed to wait indefinitely for a message.

  • flags

    A set of flags further qualifying the operation. Only the following flags should be recognized for out-of-band requests:

    • MSG_DONTWAIT causes the call to fail with the error EAGAIN if no message is immediately available at the time of the call. MSG_DONTWAIT is implied if O_NONBLOCK was set for the socket descriptor via the fcntl(2) F_SETFL operation.

    • MSG_PEEK causes the receive operation to return data from the beginning of the receive queue without removing that data from the queue. Thus, a subsequent receive call will return the same data.

    • oob_recvmsg() returns the actual number of bytes received on success. Otherwise, -1 is returned, and errno is set to the error code:

    EBADF if s does not refer to a valid socket opened with the SOCK_OOB type flag set, or s was not opened for reading.

    EINVAL if s does not support the .oob_ioctl operation.

    EFAULT if msghdr, or any buffer it refers to indirectly points to invalid memory.

    EAGAIN s is marked as non-blocking (O_NONBLOCK), or MSG_DONTWAIT is set in flags, and the receive operation would block.

    ETIMEDOUT the timeout fired before the operation could complete successfully.

    ssize_t oob_sendmsg(int s, const struct oob_msghdr *msghdr, const struct timespec *timeout, int flags)

    This call is equivalent to the standard sendmsg(2) system call, for sending the request from the out-of-band stage to an EVL driver with a socket-based interface. In other words, oob_sendmsg() is used to send messages to an out-of-band capable EVL socket from the out-of-band execution stage.

    The caller must be an EVL thread, which may be switched automatically by the EVL core to the out-of-band execution stage as a result of this call.

  • s

    A socket descriptor obtained from a regular socket(2) call, with the SOCK_OOB flag set in the type argument, denoting that out-of-band services are enabled for the socket.

  • msghdr

    A pointer to a structure containing the multiple arguments to this call, which is described below.

  • timeout

    A time limit to wait for an internal buffer to be available for sending the message before the call returns on error. The built-in clock EVL_CLOCK_MONOTONIC is used for tracking the elapsed time. If NULL is passed, the call is allowed to wait indefinitely for a buffer.

  • flags

    A set of flags further qualifying the operation. Only the following flag should be recognized for out-of-band requests:

    • MSG_DONTWAIT causes the call to fail with the error EAGAIN if no buffer is immediately available at the time of the call for sending the message. MSG_DONTWAIT is implied if O_NONBLOCK was set for the socket descriptor via the fcntl(2) F_SETFL operation.

    • oob_sendmsg() returns the actual number of bytes sent on success. Otherwise, -1 is returned, and errno is set to the error code:

    EBADF if s does not refer to a valid socket opened with the SOCK_OOB type flag set, or s was not opened for writing.

    EINVAL if s does not support the .oob_ioctl operation.

    EFAULT if msghdr, or any buffer it refers to indirectly points to invalid memory.

    EAGAIN s is marked as non-blocking (O_NONBLOCK), or MSG_DONTWAIT is set in flags, and the send operation would block.

    ETIMEDOUT the timeout fired before the operation could complete successfully.

    –

    The out-of-band message header

    The structure oob_msghdr which is passed to the oob_recvmsg() and oob_sendmsg() calls is defined as follows:

           struct oob_msghdr {
           void           *msg_name;       /* Optional address */
           socklen_t       msg_namelen;    /* Size of address */
           struct iovec   *msg_iov;        /* Scatter/gather array */
           size_t          msg_iovlen;     /* # elements in msg_iov */
           void           *msg_control;    /* Ancillary data, see below */
           size_t          msg_controllen; /* Ancillary data buffer len */
           int             msg_flags;      /* Flags on received message */
           struct timespec msg_time;       /* Optional time, see below */
       };

       struct iovec {                    /* Scatter/gather array items */
           void  *iov_base;              /* Starting address */
           size_t iov_len;               /* Number of bytes to transfer */
       };

    The msg_name field points to a caller-allocated buffer that is used to return the source address if the socket is unconnected. The caller should set msg_namelen to the size of this buffer before this call. On success, oob_recvmsg() updates msg_namelen to contain the length of the returned address. If the application does not need to know the source address, msg_name can be specified as NULL.

    The fields msg_iov and msg_iovlen describe scatter-gather locations pointing at the message data being sent or received, as discussed in readv(2).

    The field msg_control points to a buffer for other protocol control-related messages or miscellaneous ancillary data. When either oob_recvmsg() or oob_sendmsg() is called, msg_controllen should contain the length of the available buffer in msg_control. On success, oob_recvmsg() updates msg_controllen to contain the actual length of the control message sequence returned by the call.

    The msg_flags field is only set on return of oob_recvmsg(). It can contain any of the flags which may be returned by recvmsg(2).

    msg_time may be used to send or receive timestamping information to/from the protocol driver implementing out-of-band operations.

    Protocol drivers should no attach any meaning to MSG_OOB when operating in out-of-band mode, so that no additional confusion arises with the common usage of this flag with recvmsg(2).

    Last modified: Fri, 19 Jul 2024 17:48:09 +0200