AF_UNIX socketpair dgram queue sizes

From: Jan Schaumann via freebsd-net <freebsd-net_at_freebsd.org>
Date: Wed, 10 Nov 2021 01:57:20 UTC
Hello,

I'm trying to wrap my head around the buffer sizes
relevant to AF_UNIX/PF_LOCAL dgram socketpairs.

On a FreeBSD/amd64 13.0 system, creating a socketpair
and simply writing a single byte in a loop to the
non-blocking write end without reading the data, I can
perform 64 writes before causing EAGAIN, yielding 1088
bytes in FIONREAD on the read end (indicating 16 bytes
per datagram overhead).

This is well below the total net.local.dgram.recvspace
= 4096 bytes.  I would have expected to be able to
perform 240 1 byte writes (240 + 240*16 = 4080).

Now if I try to write SO_SNDBUF = 2048 bytes on each
iteration (or subsequently as many as I can until
EAGAIN), then I can send one datagram with 2048 bytes
and one datagram with 2016 bytes, filling recvspace as
(2 * 16) + (2048 + 2016) = 4096.

But at smaller sizes, it looks like the recvspace is
not filled completely: writes in chunks of > 803 bytes
will fill recvspace up to 4096 bytes, but below 803
bytes, recvspace is not maxed out.

Does anybody know why smaller datagrams can't fill
recvspace?  Or what I'm missing / misunderstanding
about the recvspace here?

-Jan