Does FreeBSD have sendmmsg or recvmmsg system calls?

[Mark Delany]

On 03Jan16, Adrian Chadd allegedly wrote:
> It doesn't help at low connection rates. It helps at high connection /
> concurrency rates as the time going in/out of the kernel and getting
> back to steady state execution changes.

This is what I found when I did a comparison at $dayjob. You obviously
need more than one message per *mmsg() call to benefit from the
syscall amortization which can only occur at very high message rates.

It also depends on whether the *mmsg() calls are implemented as mere
wrappers around recvmsg() and sendmsg() or whether the implementation
goes deeper into the driver to get better batching benefits. (I
vaguely thought that one or two drivers in Linux had real recvmmsg()
support.)

But there are a number of other factors that come into play beyond
message rates.

The first is that a typical UDP application does very little work per
message thus the syscall costs are proportionally more significant
than for TCP applications. Eg. DNS or NTP vs HTTP. The second is that
a typical syscall sequence of a high performance program likely
involves 4 syscalls per packet: select/kqueue/epoll, recvmsg,
gettimeofday and sendmsg. The third is that a high performance program
probably is multi-threaded so add in a couple of locking syscalls
around select/kqueue and you're up to 6 syscalls per request packet.

That's a lot of syscalls for an application that might only need a
small number of memory references to process a request.

The final thing is that on both FBSD and Linux I found a surprising
amount of serialization within the UDP socket stack so having multiple
threads sit on a blocking recvmsg() at high packet rates is not as
productive as you might expect. This level of serialization surprized
me given that UDP processing within the kernel should be pretty
minimal.

Combining all these factors I found it impossible to construct any
sort of UDP application that would run any where near line rates on
modern beefy machines.

It it's not apparent, these are all arguments for implementing *mmsg()
calls and for looking at UDP stack serialization.

Fortunately for FBSD there is netmap which is moderately well suited
to UDP applications... though you still have to do a fair amount of
packet decode and encode and it's only applicable in dedicated or
specialized deployments.

A final observation is that the semantics of *mmsg() may benefit from
a bit of scrutiny as they cannot accurately represent some conditions
possible with multiple *msg() calls. Eg, if a signal arrives after
more than zero packets have been processed by recvmmsg() what is the
correct return value? -1 or the count of messages returned?


Mark.