[Bug 243392] vmx driver input buffer corruption

Thu Jan 16 14:32:54 UTC 2020

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243392

            Bug ID: 243392
           Summary: vmx driver input buffer corruption
           Product: Base System
           Version: 12.1-RELEASE
          Hardware: Any
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: kern
          Assignee: bugs at FreeBSD.org
          Reporter: alexandr.oleynikov at gmail.com

Tested on 12.1-Release, 12.1-Release-p1 and with patch regarding tso from
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236999

All VM running on the same host with VMware ESXi, 6.7.0, 15160138 
 1 VM - FreeBSD 12.1
 2 VM - Any OS (tested with different, second one is irellevant)

After some network load packets in the interface queue being to get stuck

Reproduce in differents enviroment with different netowrk load
Easiest way to reproduce - copy large file from second VM to FreeBSD using
samba or ftp.
Some output examples:
172.31.255.2 - FreeBSD
172.31.255.3 - second VM

Ping:
8008 bytes from 172.31.255.3: icmp_seq=321 ttl=128 time=0.375 ms
8008 bytes from 172.31.255.3: icmp_seq=322 ttl=128 time=0.229 ms
8008 bytes from 172.31.255.3: icmp_seq=323 ttl=128 time=0.179 ms
8008 bytes from 172.31.255.3: icmp_seq=324 ttl=128 time=0.203 ms
8008 bytes from 172.31.255.3: icmp_seq=325 ttl=128 time=0.278 ms
8008 bytes from 172.31.255.3: icmp_seq=326 ttl=128 time=0.178 ms
8008 bytes from 172.31.255.3: icmp_seq=327 ttl=128 time=2.204 ms
8008 bytes from 172.31.255.3: icmp_seq=328 ttl=128 time=0.226 ms
8008 bytes from 172.31.255.3: icmp_seq=329 ttl=128 time=0.221 ms
8008 bytes from 172.31.255.3: icmp_seq=330 ttl=128 time=0.466 ms
8008 bytes from 172.31.255.3: icmp_seq=331 ttl=128 time=0.253 ms   
8008 bytes from 172.31.255.3: icmp_seq=350 ttl=128 time=1020.494 ms  <-- start
file copy
8008 bytes from 172.31.255.3: icmp_seq=362 ttl=128 time=1037.839 ms
8008 bytes from 172.31.255.3: icmp_seq=381 ttl=128 time=4162.085 ms
8008 bytes from 172.31.255.3: icmp_seq=382 ttl=128 time=5202.940 ms
8008 bytes from 172.31.255.3: icmp_seq=383 ttl=128 time=6200.121 ms
8008 bytes from 172.31.255.3: icmp_seq=385 ttl=128 time=7261.939 ms
8008 bytes from 172.31.255.3: icmp_seq=386 ttl=128 time=7200.745 ms
8008 bytes from 172.31.255.3: icmp_seq=390 ttl=128 time=4100.259 ms

tcpdump after propblem appear:

# tcpdump -i vmx1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:09:41.700833 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149,
seq 0, length 8008
16:09:42.727307 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149,
seq 1, length 8008
16:09:43.761866 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149,
seq 2, length 8008
16:09:43.762117 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 23149, seq
0, length 8008
16:09:44.771901 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149,
seq 3, length 8008
16:09:44.978178 IP truncated-ip - 7810 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 23149, seq 1, length 8008
16:09:45.788996 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149,
seq 4, length 8008
16:09:46.039983 IP truncated-ip - 7928 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 23149, seq 2, length 8008
16:09:46.290583 IP truncated-ip - 7928 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 23149, seq 3, length 8008
16:09:46.801911 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149,
seq 5, length 8008
16:09:47.861870 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149,
seq 6, length 8008
16:09:48.071886 IP truncated-ip - 7822 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 23149, seq 4, length 8008
16:09:48.871905 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149,
seq 7, length 8008
16:09:49.802033 IP truncated-ip - 7928 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 23149, seq 5, length 8008
16:09:49.891036 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149,
seq 8, length 8008
16:09:50.802001 IP truncated-ip - 7928 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 23149, seq 6, length 8008
16:09:50.901923 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149,
seq 9, length 8008
16:09:51.961848 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 23149,
seq 10, length 8008

Seems like new arrived packets pushed out old packets from the queue

After some time even more than 40 seconds gap can exist between sending and
receiving packet.

tcpdump running on the same time on second VM shows sending a response
immediately after receiving request.

Disabling/enabling any driver features like tso, csum, etc, changing mtu
irrelevant to the problem.

-- 
You are receiving this mail because:
You are the assignee for the bug.