From nobody Wed Jul 20 17:51:22 2022 X-Original-To: net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Lp3Cv0SMtz4WXvR for ; Wed, 20 Jul 2022 17:51:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Lp3Ct6rFdz497W for ; Wed, 20 Jul 2022 17:51:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Lp3Ct5x3Yz161h for ; Wed, 20 Jul 2022 17:51:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 26KHpMdD016076 for ; Wed, 20 Jul 2022 17:51:22 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 26KHpMCS016075 for net@FreeBSD.org; Wed, 20 Jul 2022 17:51:22 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: net@FreeBSD.org Subject: [Bug 264257] [tcp] Panic: Fatal trap 12: page fault while in kernel mode (if_io_tqg_4) - m_copydata ... at /usr/src/sys/kern/uipc_mbuf.c:659 Date: Wed, 20 Jul 2022 17:51:22 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: crash, needs-qa X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: clearscreen@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: net@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? maintainer-feedback? maintainer-feedback- maintainer-feedback? mfc-stable13? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1658339482; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=S/NoR6wrYSu/hVTCim3w7KUowJpbEcJTV/2sdbhbr6U=; b=YcFkpkS6/7aEmkTW8r44Spq6x1OL6Wh44LqDA4qRaL8hyPflY3dWmbOqlLLSNo6Naimq98 8JopfAexPCP0k/BNNzG97GzSec5jzi10I/sGJMPQGzLlwrOcru7bUeZHjBSRzp5KP101TG 3TLxmKkPkXraYpac93es/CsedKIyOkXLtoQKrArBObVFENdY2tyXyGDrkuA+UEMjBAliAR 4JXKbcDNu1YW5Jo+dZZxKAgchSUH71nhTSDVRRj+1K3BPWRmfI0YbRy8p75GLOkH2M2YD6 8oEmt12W59bJFXo/NelcN0x7qW35XTmp+fy8kbuQfIMONjKo6PRDRs/k9l6X8w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1658339482; a=rsa-sha256; cv=none; b=k3W69x7/w4Mc4k/HuwyoWHCmTZsT1OEpqBp1s5ERrj4ric9vYNIToZhM6q79Bc1rAF+C2I bVniXmulsAvhto3CV1zCGIBOjf4V5lvSdncELB7UM8ZxtLjq5Z5j6XlES0uDxtn9tz4z0W iKVKh/sJhIqkN8Ns0cg2GMQitV0L8PRptkf6JY5QqJhYqX14552wz2atd3TMuyYyo3VMcT J8m/wV1vQkbjcvIV/ppnmMHb166n/LwzxIvf3qd+ox4YtAyT1Hh/Wh0xwDgpdfmNpJzXLs tQsTGvgNcT+AHkmuXMLLT9gA/FCZ8eNiH7UD5EWT7+D1gZy2NNxc9Pp5VYCAPg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264257 --- Comment #84 from Chad Smith --- (In reply to Richard Scheffenegger from comment #80) Grabbed the netstat -s output from a recent crash file where the system had been up for an hour before it crashed again, also posting some additional i= nfo.=20 We have a test bench set up and we are trying to reproduce this behavior on LAN. It seems in perfect network conditions a massively parallel iperf run across a bridged interface for 24 hours does not trigger this. Looking for ideas on how to simulate internet-like network conditions. Short of hooking= the other end of our test bench up to an internet connection in another city I = am out of ideas. Open to suggestions. # uname -a FreeBSD 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64 Fatal trap 12: page fault while in kernel mode cpuid =3D 4; apic id =3D 04 fault virtual address =3D 0x10 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff8234a783 stack pointer =3D 0x0:0xfffffe00c45a5a50 frame pointer =3D 0x0:0xfffffe00c45a5a60 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 0 (if_io_tqg_4) trap number =3D 12 panic: page fault cpuid =3D 4 time =3D 1657212869 KDB: stack backtrace: #0 0xffffffff80c69465 at kdb_backtrace+0x65 #1 0xffffffff80c1bb1f at vpanic+0x17f #2 0xffffffff80c1b993 at panic+0x43 #3 0xffffffff810afdf5 at trap_fatal+0x385 #4 0xffffffff810afe4f at trap_pfault+0x4f #5 0xffffffff81087528 at calltrap+0x8 #6 0xffffffff80d55ec7 at pfil_run_hooks+0x97 #7 0xffffffff8239af37 at bridge_pfil+0x497 #8 0xffffffff8239d5a3 at bridge_forward+0x323 #9 0xffffffff8239cef1 at bridge_input+0x4c1 #10 0xffffffff80d380fd at ether_nh_input+0x21d #11 0xffffffff80d53089 at netisr_dispatch_src+0xb9 #12 0xffffffff80d372d9 at ether_input+0x69 #13 0xffffffff80d4f4d7 at iflib_rxeof+0xc27 #14 0xffffffff80d49b22 at _task_fn_rx+0x72 #15 0xffffffff80c67e9d at gtaskqueue_run_locked+0x15d #16 0xffffffff80c67b12 at gtaskqueue_thread_loop+0xc2 #17 0xffffffff80bd8a5e at fork_exit+0x7e ------------------------------------------------------------------------ netstat -s tcp: 144916 packets sent 10016 data packets (765551 bytes) 0 data packets (0 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 133061 ack-only packets (306 delayed) 0 URG only packets 0 window probe packets 7 window update packets 1832 control packets 275476 packets received 13046 acks (for 767206 bytes) 17 duplicate acks 0 UDP tunneled pkts 0 UDP tunneled pkt cnt with errors 0 acks for unsent data 269667 packets (337208234 bytes) received in-sequence 17 completely duplicate packets (1304 bytes) 0 old duplicate packets 0 packets with some dup. data (0 bytes duped) 13 out-of-order packets (16952 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 0 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to full reassembly queue 177 connection requests 1561 connection accepts 0 bad connection attempts 0 listen queue overflows 0 ignored RSTs in the windows 1661 connections established (including accepts) 6 times used RTT from hostcache 6 times used RTT variance from hostcache 0 times used slow-start threshold from hostcache 1680 connections closed (including 0 drops) 6 connections updated cached RTT on close 6 connections updated cached RTT variance on close 0 connections updated cached ssthresh on close 3 embryonic connections dropped 13046 segments updated rtt (of 9912 attempts) 74 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 254 correct ACK header predictions 260118 correct data packet header predictions 1561 syncache entries added 0 retransmitted 0 dupsyn 0 dropped 1561 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 1561 cookies sent 0 cookies received 1 hostcache entry added 0 bucket overflow 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK scoreboard overflow 0 packets with ECN CE bit set 0 packets with ECN ECT(0) bit set 0 packets with ECN ECT(1) bit set 0 successful ECN handshakes 0 times ECN reduced the congestion window 0 packets with matching signature received 0 packets with bad signature received 0 times failed to make signature due to no SA 0 times unexpected signature received 0 times no signature provided by segment 0 Path MTU discovery black hole detection activations 0 Path MTU discovery black hole detection min MSS activations 0 Path MTU discovery black hole detection failures 154 times connection in TIME-WAIT responded with ACK 0 times connection in TIME-WAIT was actively recycled 0 times connection in TIME-WAIT responded with RST TCP connection count by state: 0 connections in CLOSED state 8 connections in LISTEN state 0 connections in SYN_SENT state 0 connections in SYN_RCVD state 6 connections in ESTABLISHED state 0 connections in CLOSE_WAIT state 0 connections in FIN_WAIT_1 state 0 connections in CLOSING state 0 connections in LAST_ACK state 0 connections in FIN_WAIT_2 state 64 connections in TIME_WAIT state udp: 288632 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 0 with no checksum 13740 dropped due to no socket 2422 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 272470 delivered 179590 datagrams output 0 times multicast source filter matched ip: 648231 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 564854 packets for this host 75426 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 7951 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 324597 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 2 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header icmp: 75428 calls to icmp_error 0 errors not generated in response to an icmp message Output histogram: echo reply: 80 destination unreachable: 2 0 messages with bad code fields 0 messages less than the minimum length 0 messages with bad checksum 0 messages with bad length 0 multicast echo requests ignored 0 multicast timestamp requests ignored Input histogram: destination unreachable: 65 echo: 80 80 message responses generated 0 invalid return addresses 0 no return routes arp: 3 ARP requests sent 1 ARP request failed to sent 72 ARP replies sent 20625 ARP requests received 773 ARP replies received 74733 ARP packets received 0 total packets dropped due to no ARP entry 0 ARP entrys timed out 0 Duplicate IPs seen ip6: 0 total packets received 0 with size smaller than minimum 0 with data size < data length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 fragments that exceeded limit 0 atomic fragments 0 packets reassembled ok 0 packets for this host 0 packets forwarded 0 packets not forwardable 0 redirects sent 0 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 packets that violated scope rules 0 multicast packets which we don't join Mbuf statistics: 0 one mbuf 0 one ext mbuf 0 two or more ext mbuf 0 packets whose headers are not contiguous 0 tunneling packets that can't find gif 0 packets discarded because of too many headers 0 failures of source address selection icmp6: 0 calls to icmp6_error 0 errors not generated in response to an icmp6 message 0 errors not generated because of rate limitation 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Histogram of error messages to be generated: 0 no route 0 administratively prohibited 0 beyond scope 0 address unreachable 0 port unreachable 0 packet too big 0 time exceed transit 0 time exceed reassembly 0 erroneous header field 0 unrecognized next header 0 unrecognized option 0 redirect 0 unknown 0 message responses generated 0 messages with too many ND options 0 messages with bad ND options 0 bad neighbor solicitation messages 0 bad neighbor advertisement messages 0 bad router solicitation messages 0 bad router advertisement messages 0 bad redirect messages 0 default routers overflows 0 prefix overflows 0 neighbour entries overflows 0 redirect overflows 0 messages with invalid hop limit 0 path MTU changes rip6: 0 messages received 0 checksum calculations on inbound 0 messages with bad checksum 0 messages dropped due to no socket 0 multicast messages dropped due to no socket 0 messages dropped due to full socket buffers 0 delivered 0 datagrams output pfkey: 0 requests sent from userland 0 bytes sent from userland 0 messages with invalid length field 0 messages with invalid version field 0 messages with invalid message type field 0 messages too short 0 messages with memory allocation failure 0 messages with duplicate extension 0 messages with invalid extension type 0 messages with invalid sa type 0 messages with invalid address extension 0 requests sent to userland 0 bytes sent to userland 0 messages toward single socket 0 messages toward all sockets 0 messages toward registered sockets 0 messages with memory allocation failure --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.=