[Bug 243392] vmx driver input buffer corruption
bugzilla-noreply at freebsd.org
bugzilla-noreply at freebsd.org
Fri Jan 17 10:37:58 UTC 2020
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=243392
--- Comment #4 from alexandr.oleynikov at gmail.com ---
I did some more tests. Hope this will provide some more information.
First one with recompiled kernel with TSO patch. As a network load was a file
coping to server using samba
ifconfig vmx1
vmx1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu
9000
options=e403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:50:56:be:f0:13
inet 172.31.255.2 netmask 0xffffff00 broadcast 172.31.255.255
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
# tcpdump -i vmx1 icmp &
# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmx1, link-type EN10MB (Ethernet), capture size 262144 bytes
# ping -s 8000 172.31.255.3
PING 172.31.255.3 (172.31.255.3): 8000 data bytes
11:59:07.108253 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 0, length 8008
11:59:07.108425 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq
0, length 8008
8008 bytes from 172.31.255.3: icmp_seq=0 ttl=128 time=0.226 ms
11:59:08.126583 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 1, length 8008
11:59:08.126754 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq
1, length 8008
8008 bytes from 172.31.255.3: icmp_seq=1 ttl=128 time=0.213 ms
--- skipped some lines ---
12:00:20.401492 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 71, length 8008
8008 bytes from 172.31.255.3: icmp_seq=71 ttl=128 time=0.550 ms
12:00:20.402010 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq
71, length 8008
12:00:21.408758 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 72, length 8008
8008 bytes from 172.31.255.3: icmp_seq=72 ttl=128 time=2.303 ms
12:00:21.410995 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq
72, length 8008
12:00:24.527165 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 73, length 8008
8008 bytes from 172.31.255.3: icmp_seq=73 ttl=128 time=133.291 ms
12:00:24.592341 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq
73, length 8008
12:00:25.569300 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 74, length 8008
12:00:25.662953 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq
74, length 8008
--- after seqnum 73 packets received by kernel and seen with tcpdump but not
returned to ping process
--- skipped some lines ---
12:01:27.114142 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 134, length 8008
12:01:27.160943 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq
134, length 8008
12:01:28.125972 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 135, length 8008
12:01:28.126346 IP truncated-ip - 7982 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 30548, seq 135, length 8008
--- received malformed L2 frame from seqnum >= 135
12:01:29.198552 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 136, length 8008
12:01:29.223302 IP truncated-ip - 7810 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 30548, seq 136, length 8008
12:01:30.214849 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 137, length 8008
12:01:30.221687 IP truncated-ip - 7822 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 30548, seq 137, length 8008
12:01:31.246460 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 138, length 8008
--- skip some lines
12:01:37.514942 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 144, length 8008
12:01:37.517865 IP truncated-ip - 7808 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 30548, seq 144, length 8008
12:01:38.579626 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 145, length 8008
12:01:38.615120 IP truncated-ip - 7928 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 30548, seq 145, length 8008
12:01:39.603253 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 146, length 8008
12:01:40.614996 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 147, length 8008
12:01:40.615183 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq
146, length 8008
--- difference in 1 second between sending and receiveng reply from seqnum 146
12:01:40.615201 IP truncated-ip - 7928 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 30548, seq 147, length 8008
8008 bytes from 172.31.255.3: icmp_seq=146 ttl=128 time=1011.985 ms
12:01:41.657600 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 148, length 8008
12:01:42.701072 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 149, length 8008
12:01:42.701321 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq
148, length 8008
8008 bytes from 172.31.255.3: icmp_seq=148 ttl=128 time=1043.763 ms
12:01:43.615120 IP truncated-ip - 7928 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 30548, seq 149, length 8008
12:01:43.714982 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 150, length 8008
12:01:43.988367 IP truncated-ip - 7808 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 30548, seq 150, length 8008
12:01:44.787457 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 151, length 8008
12:01:44.788966 IP truncated-ip - 7782 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 30548, seq 151, length 8008
12:01:45.815011 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 152, length 8008
12:01:45.970727 IP truncated-ip - 7976 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 30548, seq 152, length 8008
12:01:46.834089 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 153, length 8008
12:01:47.615212 IP truncated-ip - 7928 bytes missing! 172.31.255.3 >
172.31.255.2: ICMP echo reply, id 30548, seq 153, length 8008
12:01:47.897600 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 154, length 8008
12:01:48.914981 IP 172.31.255.2 > 172.31.255.3: ICMP echo request, id 30548,
seq 155, length 8008
12:01:48.915192 IP 172.31.255.3 > 172.31.255.2: ICMP echo reply, id 30548, seq
154, length 8008
8008 bytes from 172.31.255.3: icmp_seq=154 ttl=128 time=1017.638 ms
--- some packet reveived undamaged but with delay in 1 second
When i try using iperf as network load source in most cases was kernel panic as
result:
Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address = 0x0
fault code = supervisor write data, page not present
instruction pointer = 0x20:0xffffffff80cef252
stack pointer = 0x28:0xfffffe00753547c0
frame pointer = 0x28:0xfffffe00753548a0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (if_io_tqg_1)
trap number = 12
panic: page fault
cpuid = 1
time = 1579255990
KDB: stack backtrace:
#0 0xffffffff80c1d297 at kdb_backtrace+0x67
#1 0xffffffff80bd05cd at vpanic+0x19d
#2 0xffffffff80bd0423 at panic+0x43
#3 0xffffffff810a7d2c at trap_fatal+0x39c
#4 0xffffffff810a7d79 at trap_pfault+0x49
#5 0xffffffff810a736f at trap+0x29f
#6 0xffffffff81081a0c at calltrap+0x8
#7 0xffffffff80ce9be5 at _task_fn_rx+0x75
#8 0xffffffff80c1bb54 at gtaskqueue_run_locked+0x144
#9 0xffffffff80c1b7b8 at gtaskqueue_thread_loop+0x98
#10 0xffffffff80b90c23 at fork_exit+0x83
#11 0xffffffff81082a4e at fork_trampoline+0xe
Uptime: 16m42s
Then reverting to default kernel, disabling tso and reboot:
# uname -a
FreeBSD ******************* 12.1-RELEASE-p1 FreeBSD 12.1-RELEASE-p1 GENERIC
amd64
# ifconfig vmx1
vmx1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=e400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:50:56:be:f0:13
inet 172.31.255.2 netmask 0xffffff00 broadcast 172.31.255.255
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
#
iperf3 -c 172.31.255.2 -p 1234
Connecting to host 172.31.255.2, port 1234
[ 5] local 172.31.255.5 port 32466 connected to 172.31.255.2 port 1234
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.03 sec 497 MBytes 4.05 Gbits/sec 11 8.74 KBytes
[ 5] 1.03-2.07 sec 0.00 Bytes 0.00 bits/sec 3 8.74 KBytes
[ 5] 2.07-3.06 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 3.06-4.02 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 4.02-5.01 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 5.01-6.03 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
[ 5] 6.03-7.06 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 7.06-8.04 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 8.04-9.07 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes
[ 5] 9.07-10.01 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.01 sec 497 MBytes 416 Mbits/sec 18 sender
[ 5] 0.00-10.60 sec 496 MBytes 393 Mbits/sec receiver
# ping -s 8000 172.31.255.5
PING 172.31.255.5 (172.31.255.5): 8000 data bytes
8008 bytes from 172.31.255.5: icmp_seq=0 ttl=64 time=0.322 ms
12:22:09.903151 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 0, length 8008
12:22:09.903253 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
0, length 8008
12:22:10.922205 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 1, length 8008
12:22:10.922300 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
1, length 8008
8008 bytes from 172.31.255.5: icmp_seq=1 ttl=64 time=0.147 ms
12:22:11.969930 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 2, length 8008
12:22:11.970035 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
2, length 8008
8008 bytes from 172.31.255.5: icmp_seq=2 ttl=64 time=0.159 ms
12:22:12.997254 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 3, length 8008
12:22:12.997386 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
3, length 8008
8008 bytes from 172.31.255.5: icmp_seq=3 ttl=64 time=0.175 ms
12:22:14.029823 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 4, length 8008
12:22:14.030017 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
4, length 8008
8008 bytes from 172.31.255.5: icmp_seq=4 ttl=64 time=0.237 ms
12:22:15.058570 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 5, length 8008
12:22:15.058769 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
5, length 8008
8008 bytes from 172.31.255.5: icmp_seq=5 ttl=64 time=0.241 ms
12:22:16.096803 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 6, length 8008
12:22:16.096896 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
6, length 8008
8008 bytes from 172.31.255.5: icmp_seq=6 ttl=64 time=0.139 ms
12:22:17.136966 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 7, length 8008
12:22:17.137224 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
7, length 8008
12:22:18.164014 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 8, length 8008
12:22:18.164194 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
8, length 8008
--- packets stops sending to ping process
--- skip some lines ---
-- but after some time packets againg sending to ping process
12:26:15.636917 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 238, length 8008
12:26:15.637147 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
238, length 8008
12:26:16.696907 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 239, length 8008
12:26:16.697100 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
239, length 8008
8008 bytes from 172.31.255.5: icmp_seq=239 ttl=64 time=0.256 ms
12:26:17.756044 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 240, length 8008
12:26:17.756178 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
240, length 8008
8008 bytes from 172.31.255.5: icmp_seq=240 ttl=64 time=0.190 ms
12:26:18.796861 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 241, length 8008
12:26:18.796982 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
241, length 8008
8008 bytes from 172.31.255.5: icmp_seq=241 ttl=64 time=0.176 ms
12:26:19.836847 IP 172.31.255.2 > 172.31.255.5: ICMP echo request, id 10122,
seq 242, length 8008
12:26:19.836981 IP 172.31.255.5 > 172.31.255.2: ICMP echo reply, id 10122, seq
242, length 8008
8008 bytes from 172.31.255.5: icmp_seq=242 ttl=64 time=0.192 ms
--
You are receiving this mail because:
You are the assignee for the bug.
More information about the freebsd-net
mailing list