bhyve: cannot send jumbo frames from linux or freebsd guest

[Peter Grehan]

Hi,

> * From FreeBSD guest:
> # tcpdump -vvn -i vtnet1 &
> tcpdump: listening on vtnet1, link-type EN10MB (Ethernet), capture
> size 262144 bytes
> # ping -c 1 -D -s 1995 192.168.253.1
> PING 192.168.253.1 (192.168.253.1): 1995 data bytes
> 23:08:46.808187 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
> ICMP (1), length 2023)
>     192.168.253.4 > 192.168.253.1: ICMP echo request, id 58116, seq 0,
> length 2003
> 23:08:46.808430 IP truncated-ip - 1 bytes missing! (tos 0x0, ttl 64,
> id 0, offset 0, flags [DF], proto ICMP (1), length 2023)
>     192.168.253.1 > 192.168.253.4: ICMP echo reply, id 58116, seq 0, length 2003

  This looks like a bug in the FreeBSD virtio-net driver bpf code, since 
the host dump shows those sized packets going out correctly.

> Could anyone explain why I cannot send jumbo frames from Linux and FreeBSD
> guests, but I *can* send them from Windows guest? Am I missing o
> missunderstanding something? Thanks in advanced, any comment o suggestion will
> be wellcome.

  There is currently a bug in bhyve with handling of received frames. 
The virtio implementation advertises 'rx-merge' capability, which allows 
frames to span buffers that have been queued by the guest. However, 
bhyve currently doesn't copy the buffer across multiple buffers.

  I suspect that the Windows virtio guest driver doesn't use rx-merge 
but instead uses 'indirect descriptors', which is why it works.

  A short-term workaround for FreeBSD and Linux guests is to modify 
bhyve to not advertise the rx-merge capability:

usr.sbin/bhyve/pci_virtio_net.c
#define VTNET_S_HOSTCAPS      \
-  ( VIRTIO_NET_F_MAC | VIRTIO_NET_F_MRG_RXBUF | VIRTIO_NET_F_STATUS | \
+  ( VIRTIO_NET_F_MAC | VIRTIO_NET_F_STATUS | \


  (Note that this doesn't work with Net/OpenBSD, where jumbo frames are 
only supported via rx-merge.)

  I hope to have a fix for the rx-merge issue in the near future - it's 
a more efficient way of handling jumbo frames than indirect descriptors.

later,

Peter.