bhyve virtio-net MTU

Sat Mar 7 12:50:00 UTC 2015

Hi,

I'm running FreeBSD 10.1 on a dedicated server, with a Linux VM in bhyve. The handbook tells you to bridge the tap interface to a real network interface, but that's not an option for me because I only have one IPv4 address. So instead, I assigned an internal IP address to bridge0 and used pf(4) to set up NAT routing.
All of this works without any issues, but I would like to increase the MTU of 1500. It's a virtual interface after all, why should it be so low? FreeBSD's loopback interface's MTU is 16384 and on Linux, it's even 65536.

So I used ifconfig(8) to increase the MTU of tap0 and just like the manpage says, bridge0 had the same MTU after I added tap0. On the Linux side, I did the same with eth0 and then I sent 2000 byte pings to the host machine.

It seems to work, but let's use tcpdump(8) to make sure:

02:02:46.244678 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1686, seq 3, length 2008
02:02:46.244983 IP 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1686, seq 3, length 1480
02:02:46.245061 IP 10.42.42.1 > 10.42.42.100: ip-proto-1
02:02:47.244953 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1686, seq 4, length 2008
02:02:47.245347 IP 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1686, seq 4, length 1480
02:02:47.245422 IP 10.42.42.1 > 10.42.42.100: ip-proto-1

The entire request goes through the virtio NIC, tap0, bridge0 and finally to the host's kernel without any issues. Which then sends a _fragmented_ reply because apparently it still thinks the MTU is 1500.

A quick check with route(8) confirms this:

# route show 10.42.42.100
   route to: 10.42.42.100
destination: 10.42.42.0
       mask: 255.255.255.0
        fib: 0
  interface: bridge0
      flags: <UP,DONE,STATIC,PINNED>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0 

So I manually forced a bigger MTU:

# route change -net 10.42.42.0 -mtu 15000
change net 10.42.42.0

But now the reply packets get truncated instead of fragmented:

02:07:36.921165 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 3, length 2008
02:07:36.921624 IP truncated-ip - 6 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 3, length 2008
02:07:37.921042 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 4, length 2008
02:07:37.921499 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 4, length 2008
02:07:38.921522 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 5, length 2008
02:07:38.922253 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 5, length 2008
02:07:39.921432 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 6, length 2008
02:07:39.922165 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 6, length 2008
02:07:40.921513 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 7, length 2008
02:07:40.922245 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 7, length 2008
02:07:41.921393 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 8, length 2008
02:07:41.922160 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 8, length 2008
02:07:42.921504 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 9, length 2008
02:07:42.922348 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 9, length 2008
02:07:43.923031 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 10, length 2008
02:07:43.924904 IP truncated-ip - 6 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 10, length 2008
02:07:44.926832 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 11, length 2008
02:07:44.928511 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 11, length 2008
02:07:45.936968 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 12, length 2008
02:07:45.937722 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 12, length 2008
02:07:46.937453 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 13, length 2008
02:07:46.938161 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 13, length 2008
^C
22 packets captured
24 packets received by filter
0 packets dropped by kernel

This last dump was on the Linux side. Dumping at tap0 shows that the reply packets are still okay when they reach bhyve. Apparently they get truncated by bhyve's virtio-net as increasing the MTU works just fine with VirtualBox's virtio-net (at least on my Linux machine).

Any ideas on how I can fix this? I had a quick look at the code and while a comment indicates that Ethernet-sized packets are assumed (https://svnweb.freebsd.org/base/release/10.1.0/usr.sbin/bhyve/pci_virtio_net.c?revision=274417&view=markup#l257), I was unable to find code that confirms this.

Noah