bhyve virtio-net MTU

Sat Mar 7 16:21:57 UTC 2015

On 2015-03-07 07:49, Noah Bergbauer wrote:
> Hi,
> 
> I'm running FreeBSD 10.1 on a dedicated server, with a Linux VM in bhyve. The handbook tells you to bridge the tap interface to a real network interface, but that's not an option for me because I only have one IPv4 address. So instead, I assigned an internal IP address to bridge0 and used pf(4) to set up NAT routing.
> All of this works without any issues, but I would like to increase the MTU of 1500. It's a virtual interface after all, why should it be so low? FreeBSD's loopback interface's MTU is 16384 and on Linux, it's even 65536.
> 
> So I used ifconfig(8) to increase the MTU of tap0 and just like the manpage says, bridge0 had the same MTU after I added tap0. On the Linux side, I did the same with eth0 and then I sent 2000 byte pings to the host machine.
> 
> It seems to work, but let's use tcpdump(8) to make sure:
> 
> 02:02:46.244678 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1686, seq 3, length 2008
> 02:02:46.244983 IP 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1686, seq 3, length 1480
> 02:02:46.245061 IP 10.42.42.1 > 10.42.42.100: ip-proto-1
> 02:02:47.244953 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1686, seq 4, length 2008
> 02:02:47.245347 IP 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1686, seq 4, length 1480
> 02:02:47.245422 IP 10.42.42.1 > 10.42.42.100: ip-proto-1
> 
> The entire request goes through the virtio NIC, tap0, bridge0 and finally to the host's kernel without any issues. Which then sends a _fragmented_ reply because apparently it still thinks the MTU is 1500.
> 
> A quick check with route(8) confirms this:
> 
> # route show 10.42.42.100
>    route to: 10.42.42.100
> destination: 10.42.42.0
>        mask: 255.255.255.0
>         fib: 0
>   interface: bridge0
>       flags: <UP,DONE,STATIC,PINNED>
>  recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
>        0         0         0         0      1500         1         0 
> 
> So I manually forced a bigger MTU:
> 
> # route change -net 10.42.42.0 -mtu 15000
> change net 10.42.42.0
> 
> But now the reply packets get truncated instead of fragmented:
> 
> 02:07:36.921165 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 3, length 2008
> 02:07:36.921624 IP truncated-ip - 6 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 3, length 2008
> 02:07:37.921042 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 4, length 2008
> 02:07:37.921499 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 4, length 2008
> 02:07:38.921522 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 5, length 2008
> 02:07:38.922253 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 5, length 2008
> 02:07:39.921432 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 6, length 2008
> 02:07:39.922165 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 6, length 2008
> 02:07:40.921513 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 7, length 2008
> 02:07:40.922245 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 7, length 2008
> 02:07:41.921393 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 8, length 2008
> 02:07:41.922160 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 8, length 2008
> 02:07:42.921504 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 9, length 2008
> 02:07:42.922348 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 9, length 2008
> 02:07:43.923031 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 10, length 2008
> 02:07:43.924904 IP truncated-ip - 6 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 10, length 2008
> 02:07:44.926832 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 11, length 2008
> 02:07:44.928511 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 11, length 2008
> 02:07:45.936968 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 12, length 2008
> 02:07:45.937722 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 12, length 2008
> 02:07:46.937453 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 13, length 2008
> 02:07:46.938161 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 13, length 2008
> ^C
> 22 packets captured
> 24 packets received by filter
> 0 packets dropped by kernel
> 
> This last dump was on the Linux side. Dumping at tap0 shows that the reply packets are still okay when they reach bhyve. Apparently they get truncated by bhyve's virtio-net as increasing the MTU works just fine with VirtualBox's virtio-net (at least on my Linux machine).
> 
> Any ideas on how I can fix this? I had a quick look at the code and while a comment indicates that Ethernet-sized packets are assumed (https://svnweb.freebsd.org/base/release/10.1.0/usr.sbin/bhyve/pci_virtio_net.c?revision=274417&view=markup#l257), I was unable to find code that confirms this.
> 
> Noah
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe at freebsd.org"
> 

There is a 'do not fragment' flag you can sent with ping to help debug this.

-- 
Allan Jude

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 834 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-virtualization/attachments/20150307/3369102f/attachment.sig>