[Bug 264257] [tcp] Panic: Fatal trap 12: page fault while in kernel mode (if_io_tqg_4) - m_copydata ... at /usr/src/sys/kern/uipc_mbuf.c:659

From: <bugzilla-noreply_at_freebsd.org>
Date: Wed, 20 Jul 2022 17:51:22 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=264257

--- Comment #84 from Chad Smith <clearscreen@gmail.com> ---
(In reply to Richard Scheffenegger from comment #80)

Grabbed the netstat -s output from a recent crash file where the system had
been up for an hour before it crashed again, also posting some additional info. 

We have a test bench set up and we are trying to reproduce this behavior on
LAN. It seems in perfect network conditions a massively parallel iperf run
across a bridged interface for 24 hours does not trigger this. Looking for
ideas on how to simulate internet-like network conditions. Short of hooking the
other end of our test bench up to an internet connection in another city I am
out of ideas. Open to suggestions.

# uname -a
FreeBSD 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212
GENERIC amd64

Fatal trap 12: page fault while in kernel mode
cpuid = 4; apic id = 04
fault virtual address   = 0x10
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff8234a783
stack pointer           = 0x0:0xfffffe00c45a5a50
frame pointer           = 0x0:0xfffffe00c45a5a60
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 0 (if_io_tqg_4)
trap number             = 12
panic: page fault
cpuid = 4
time = 1657212869
KDB: stack backtrace:
#0 0xffffffff80c69465 at kdb_backtrace+0x65
#1 0xffffffff80c1bb1f at vpanic+0x17f
#2 0xffffffff80c1b993 at panic+0x43
#3 0xffffffff810afdf5 at trap_fatal+0x385
#4 0xffffffff810afe4f at trap_pfault+0x4f
#5 0xffffffff81087528 at calltrap+0x8
#6 0xffffffff80d55ec7 at pfil_run_hooks+0x97
#7 0xffffffff8239af37 at bridge_pfil+0x497
#8 0xffffffff8239d5a3 at bridge_forward+0x323
#9 0xffffffff8239cef1 at bridge_input+0x4c1
#10 0xffffffff80d380fd at ether_nh_input+0x21d
#11 0xffffffff80d53089 at netisr_dispatch_src+0xb9
#12 0xffffffff80d372d9 at ether_input+0x69
#13 0xffffffff80d4f4d7 at iflib_rxeof+0xc27
#14 0xffffffff80d49b22 at _task_fn_rx+0x72
#15 0xffffffff80c67e9d at gtaskqueue_run_locked+0x15d
#16 0xffffffff80c67b12 at gtaskqueue_thread_loop+0xc2
#17 0xffffffff80bd8a5e at fork_exit+0x7e

------------------------------------------------------------------------
netstat -s

tcp:
        144916 packets sent
                10016 data packets (765551 bytes)
                0 data packets (0 bytes) retransmitted
                0 data packets unnecessarily retransmitted
                0 resends initiated by MTU discovery
                133061 ack-only packets (306 delayed)
                0 URG only packets
                0 window probe packets
                7 window update packets
                1832 control packets
        275476 packets received
                13046 acks (for 767206 bytes)
                17 duplicate acks
                0 UDP tunneled pkts
                0 UDP tunneled pkt cnt with errors
                0 acks for unsent data
                269667 packets (337208234 bytes) received in-sequence
                17 completely duplicate packets (1304 bytes)
                0 old duplicate packets
                0 packets with some dup. data (0 bytes duped)
                13 out-of-order packets (16952 bytes)
                0 packets (0 bytes) of data after window
                0 window probes
                0 window update packets
                0 packets received after close
                0 discarded for bad checksums
                0 discarded for bad header offset fields
                0 discarded because packet too short
                0 discarded due to full reassembly queue
        177 connection requests
        1561 connection accepts
        0 bad connection attempts
        0 listen queue overflows
        0 ignored RSTs in the windows
        1661 connections established (including accepts)
                6 times used RTT from hostcache
                6 times used RTT variance from hostcache
                0 times used slow-start threshold from hostcache
        1680 connections closed (including 0 drops)
                6 connections updated cached RTT on close
                6 connections updated cached RTT variance on close
                0 connections updated cached ssthresh on close
        3 embryonic connections dropped
        13046 segments updated rtt (of 9912 attempts)
        74 retransmit timeouts
                0 connections dropped by rexmit timeout
        0 persist timeouts
                0 connections dropped by persist timeout
        0 Connections (fin_wait_2) dropped because of timeout
        0 keepalive timeouts
                0 keepalive probes sent
                0 connections dropped by keepalive
        254 correct ACK header predictions
        260118 correct data packet header predictions
        1561 syncache entries added
                0 retransmitted
                0 dupsyn
                0 dropped
                1561 completed
                0 bucket overflow
                0 cache overflow
                0 reset
                0 stale
                0 aborted
                0 badack
                0 unreach
                0 zone failures
        1561 cookies sent
        0 cookies received
        1 hostcache entry added
                0 bucket overflow
        0 SACK recovery episodes
        0 segment rexmits in SACK recovery episodes
        0 byte rexmits in SACK recovery episodes
        0 SACK options (SACK blocks) received
        0 SACK options (SACK blocks) sent
        0 SACK scoreboard overflow
        0 packets with ECN CE bit set
        0 packets with ECN ECT(0) bit set
        0 packets with ECN ECT(1) bit set
        0 successful ECN handshakes
        0 times ECN reduced the congestion window
        0 packets with matching signature received
        0 packets with bad signature received
        0 times failed to make signature due to no SA
        0 times unexpected signature received
        0 times no signature provided by segment
        0 Path MTU discovery black hole detection activations
        0 Path MTU discovery black hole detection min MSS activations
        0 Path MTU discovery black hole detection failures
        154 times connection in TIME-WAIT responded with ACK
        0 times connection in TIME-WAIT was actively recycled
        0 times connection in TIME-WAIT responded with RST
TCP connection count by state:
        0 connections in CLOSED state
        8 connections in LISTEN state
        0 connections in SYN_SENT state
        0 connections in SYN_RCVD state
        6 connections in ESTABLISHED state
        0 connections in CLOSE_WAIT state
        0 connections in FIN_WAIT_1 state
        0 connections in CLOSING state
        0 connections in LAST_ACK state
        0 connections in FIN_WAIT_2 state
        64 connections in TIME_WAIT state
udp:
        288632 datagrams received
        0 with incomplete header
        0 with bad data length field
        0 with bad checksum
        0 with no checksum
        13740 dropped due to no socket
        2422 broadcast/multicast datagrams undelivered
        0 dropped due to full socket buffers
        0 not for hashed pcb
        272470 delivered
        179590 datagrams output
        0 times multicast source filter matched
ip:
        648231 total packets received
        0 bad header checksums
        0 with size smaller than minimum
        0 with data size < data length
        0 with ip length > max ip packet size
        0 with header length < data size
        0 with data length < header length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 packets reassembled ok
        564854 packets for this host
        75426 packets for unknown/unsupported protocol
        0 packets forwarded (0 packets fast forwarded)
        7951 packets not forwardable
        0 packets received for unknown multicast group
        0 redirects sent
        324597 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        2 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 tunneling packets that can't find gif
        0 datagrams with bad address in header
icmp:
        75428 calls to icmp_error
        0 errors not generated in response to an icmp message
        Output histogram:
                echo reply: 80
                destination unreachable: 2
        0 messages with bad code fields
        0 messages less than the minimum length
        0 messages with bad checksum
        0 messages with bad length
        0 multicast echo requests ignored
        0 multicast timestamp requests ignored
        Input histogram:
                destination unreachable: 65
                echo: 80
        80 message responses generated
        0 invalid return addresses
        0 no return routes
arp:
        3 ARP requests sent
        1 ARP request failed to sent
        72 ARP replies sent
        20625 ARP requests received
        773 ARP replies received
        74733 ARP packets received
        0 total packets dropped due to no ARP entry
        0 ARP entrys timed out
        0 Duplicate IPs seen
ip6:
        0 total packets received
        0 with size smaller than minimum
        0 with data size < data length
        0 with bad options
        0 with incorrect version number
        0 fragments received
        0 fragments dropped (dup or out of space)
        0 fragments dropped after timeout
        0 fragments that exceeded limit
        0 atomic fragments
        0 packets reassembled ok
        0 packets for this host
        0 packets forwarded
        0 packets not forwardable
        0 redirects sent
        0 packets sent from this host
        0 packets sent with fabricated ip header
        0 output packets dropped due to no bufs, etc.
        0 output packets discarded due to no route
        0 output datagrams fragmented
        0 fragments created
        0 datagrams that can't be fragmented
        0 packets that violated scope rules
        0 multicast packets which we don't join
        Mbuf statistics:
                0 one mbuf
                0 one ext mbuf
                0 two or more ext mbuf
        0 packets whose headers are not contiguous
        0 tunneling packets that can't find gif
        0 packets discarded because of too many headers
        0 failures of source address selection
icmp6:
        0 calls to icmp6_error
        0 errors not generated in response to an icmp6 message
        0 errors not generated because of rate limitation
        0 messages with bad code fields
        0 messages < minimum length
        0 bad checksums
        0 messages with bad length
        Histogram of error messages to be generated:
                0 no route
                0 administratively prohibited
                0 beyond scope
                0 address unreachable
                0 port unreachable
                0 packet too big
                0 time exceed transit
                0 time exceed reassembly
                0 erroneous header field
                0 unrecognized next header
                0 unrecognized option
                0 redirect
                0 unknown
        0 message responses generated
        0 messages with too many ND options
        0 messages with bad ND options
        0 bad neighbor solicitation messages
        0 bad neighbor advertisement messages
        0 bad router solicitation messages
        0 bad router advertisement messages
        0 bad redirect messages
        0 default routers overflows
        0 prefix overflows
        0 neighbour entries overflows
        0 redirect overflows
        0 messages with invalid hop limit
        0 path MTU changes
rip6:
        0 messages received
        0 checksum calculations on inbound
        0 messages with bad checksum
        0 messages dropped due to no socket
        0 multicast messages dropped due to no socket
        0 messages dropped due to full socket buffers
        0 delivered
        0 datagrams output
pfkey:
        0 requests sent from userland
        0 bytes sent from userland
        0 messages with invalid length field
        0 messages with invalid version field
        0 messages with invalid message type field
        0 messages too short
        0 messages with memory allocation failure
        0 messages with duplicate extension
        0 messages with invalid extension type
        0 messages with invalid sa type
        0 messages with invalid address extension
        0 requests sent to userland
        0 bytes sent to userland
        0 messages toward single socket
        0 messages toward all sockets
        0 messages toward registered sockets
        0 messages with memory allocation failure

-- 
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.