bhyve: cannot send jumbo frames from linux or freebsd guest

Sun Dec 4 22:35:29 UTC 2016

Hi everybody,

On my FreeBSD 11.0-RELEASE I am running three bhyve virtual machines:

1- Windows Server 2012
2- Centos 6
3- FreeBSD 11.0-RELEASE

An iSCSI target is configured in the host, and the three above guests runs a
iSCSI initiator. I use iSCSI to present zvols to guests, and so be
able to increase
the guests storage without rebooting. This setup works as expected: initiators
connect to target, and everything run smoothly.

In order to boost performance, I want to enable jumbo frames in the private,
non routing, iSCSI subnet. The subnet is 192.168.253/24, and is configured
as follows:

# ifconfig bridge6
bridge6: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: vm-iscsi0
        ether 02:10:ef:9e:c0:06
        inet 192.168.253.1 netmask 0xffffff00 broadcast 192.168.253.255
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap9 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 22 priority 128 path cost 2000000
        member: tap7 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 20 priority 128 path cost 2000000
        member: tap6 flags=943<LEARNING,DISCOVER,PRIVATE,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 19 priority 128 path cost 2000000

192.168.253.1 is the ip address of the FreeBSD bhyve host (the iSCSI target) in
this private subnet.  Notice that I have set mtu=9000 in the bridge interface.
Automatically, the taps interfaces inherit that setting when the guests starts
up.

Inside each guest, I have configured the mtu setting to 9000. In the FreeBSD and
Linux, I get, respectively:

# ifconfig vtnet1
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
metric 0 mtu 9000
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 58:9c:fc:05:55:66
        inet 192.168.253.4 netmask 0xffffff00 broadcast 192.168.253.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active

# ifconfig eth2
eth2      Link encap:Ethernet  HWaddr 58:9C:FC:0E:9D:01
          inet addr:192.168.253.3  Bcast:192.168.253.255  Mask:255.255.255.0
          inet6 addr: fe80::5a9c:fcff:fe0e:9d01/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:19 errors:0 dropped:0 overruns:0 frame:0
          TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:65470 (63.9 KiB)  TX bytes:108972 (106.4 KiB)

In order to test if jumbo frames are set properly, I run the following from the
Windows guest:

C:\Users\Administrator>ping -n 1 -f -l 8972 192.168.253.1

Pinging 192.168.253.1 with 8972 bytes of data:
Reply from 192.168.253.1: bytes=8972 time<1ms TTL=64

Ping statistics for 192.168.253.1:
    Packets: Sent = 1, Received = 1, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

As you can see, jumbo frames seems to work properly from Windows guest, as
I can trasmit a 8972 bytes frame with no fragmentation. A tcpdump in the host
side shows the following:

23:22:42.111043 IP (tos 0x0, ttl 128, id 20704, offset 0, flags [DF],
proto ICMP (1), length 9000)
    192.168.253.2 > 192.168.253.1: ICMP echo request, id 1, seq 301, length 8980
23:22:42.111106 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
ICMP (1), length 9000)
    192.168.253.1 > 192.168.253.2: ICMP echo reply, id 1, seq 301, length 8980

But, If I repeat the analogous test from FreeBSD or Linux guest, I get wrong
results, as I will explain.

>From  Linux guest I run:

# ping -c 4 -s 8972 -M do 192.168.253.1
PING 192.168.253.1 (192.168.253.1) 8972(9000) bytes of data.

--- 192.168.253.1 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 13000ms

If I decrease the packet size to 4042, I get a succesful answer from the iSCSI
target:

# ping -c 4 -s 4042 -M do 192.168.253.1
PING 192.168.253.1 (192.168.253.1) 4042(4070) bytes of data.
4050 bytes from 192.168.253.1: icmp_seq=1 ttl=64 time=0.191 ms
4050 bytes from 192.168.253.1: icmp_seq=2 ttl=64 time=0.203 ms
4050 bytes from 192.168.253.1: icmp_seq=3 ttl=64 time=0.159 ms
4050 bytes from 192.168.253.1: icmp_seq=4 ttl=64 time=0.191 ms

--- 192.168.253.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.159/0.186/0.203/0.016 ms

With a 4043 packet size, the ping fails again.

>From FreeBSD guest, the test would be:

# ping -c 4 -D -s 8972 192.168.253.1
PING 192.168.253.1 (192.168.253.1): 8972 data bytes

--- 192.168.253.1 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

After some trial and error, I get 1994 as the upper bound:

# ping -c 4 -D -s 1994 192.168.253.1
PING 192.168.253.1 (192.168.253.1): 1994 data bytes
2002 bytes from 192.168.253.1: icmp_seq=0 ttl=64 time=0.203 ms
2002 bytes from 192.168.253.1: icmp_seq=1 ttl=64 time=0.239 ms
2002 bytes from 192.168.253.1: icmp_seq=2 ttl=64 time=0.224 ms
2002 bytes from 192.168.253.1: icmp_seq=3 ttl=64 time=0.214 ms

--- 192.168.253.1 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.203/0.220/0.239/0.013 ms

I have run tcpdump on both FreeBSD guest and host to debug the issue and I get
the following:

* From FreeBSD guest:
# tcpdump -vvn -i vtnet1 &
tcpdump: listening on vtnet1, link-type EN10MB (Ethernet), capture
size 262144 bytes
# ping -c 1 -D -s 1995 192.168.253.1
PING 192.168.253.1 (192.168.253.1): 1995 data bytes
23:08:46.808187 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
ICMP (1), length 2023)
    192.168.253.4 > 192.168.253.1: ICMP echo request, id 58116, seq 0,
length 2003
23:08:46.808430 IP truncated-ip - 1 bytes missing! (tos 0x0, ttl 64,
id 0, offset 0, flags [DF], proto ICMP (1), length 2023)
    192.168.253.1 > 192.168.253.4: ICMP echo reply, id 58116, seq 0, length 2003

--- 192.168.253.1 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss

* From FreeBSD host:
# tcpdump -vvn -i bridge6
tcpdump: listening on bridge6, link-type EN10MB (Ethernet), capture
size 262144 bytes
23:09:58.315600 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
ICMP (1), length 2023)
    192.168.253.4 > 192.168.253.1: ICMP echo request, id 59140, seq 0,
length 2003
23:09:58.315663 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto
ICMP (1), length 2023)
    192.168.253.1 > 192.168.253.4: ICMP echo reply, id 59140, seq 0, length 2003

Notice the "truncated-ip - 1 bytes missing!" from the guest. Indeed, is clear
that the issue is related to the Linux or FreeBSD guest, as from Windows guest
I cant transmit jumbo frames.

Could anyone explain why I cannot send jumbo frames from Linux and FreeBSD
guests, but I *can* send them from Windows guest? Am I missing o
missunderstanding something? Thanks in advanced, any comment o suggestion will
be wellcome. Best regards.