From nobody Mon Jan 02 00:52:50 2023 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Nlcm25Kt9z1LrGJ for ; Mon, 2 Jan 2023 00:52:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Nlcm23jDpz4MbZ for ; Mon, 2 Jan 2023 00:52:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1672620770; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Dt3hFnswChXj4CnnEzRwoD8fI0dZtoM+r0kakJfSFrQ=; b=MhkLMNS77N5y6jHwK+ikihVTVM7OOWxii8LBwTVixRm0LkiBjfjU6/25DsVKuTfgcKmCSF KpHXdv6cYMMEk6QifrsCs3UtXez4jU3pFStOeWYd0vu360A5jk7tCEo7CXud46iwHJjY6+ Q3XKdZTK9lmTA4uveBhUP5ZPUp2PTRJPTKNFA2jo7WLOz80gCIUE4Vi0qxf1v9ZdM13OIB wZTu0Z3gEbZTn0YeaEMa4QMwcmglBR3atvpDH3kb6joGJ3PUyHQI9DFu/axFXGC6kJNC6i iC/JQR3EkSxrf5vSEE2DSm1iSsxIEb9HMTKE5Qpcx3l4dKq0tYfwrC1Yt1Vxog== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1672620770; a=rsa-sha256; cv=none; b=xiHgo/9LLcJoVH2dGgAMNRdhtxQnZbSFUEbCEWI5wvh+3dWltY1VqOjFqmJu0OmL2P0ty8 kdumlaiJ2hbAgdDJtaqekUIm5SoP062uQS+AT1+b1X3ITh05g1/K5x2x4MpZWkT5KSMBtw +e1v4D/5S6uaGPkUC9hNse0DzGwb7AfsxFQZywtjpSS2jhQ0v9I/Qcu60CU/j7n1f7qELf FTk/kVs8j6kjOzzm/0kMfDTIjBFE+YwEmGjmmPynz2WX/UE7mHmERyWFmg3uJKhekkS9oO JB5np734GlcN2rgDlHgkkjJQi7s78/HCChiB/SwQtA6lCMUd806zh26VAMk9ag== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Nlcm22pLrzntF for ; Mon, 2 Jan 2023 00:52:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 3020qoXD023355 for ; Mon, 2 Jan 2023 00:52:50 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 3020qobF023354 for bugs@FreeBSD.org; Mon, 2 Jan 2023 00:52:50 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 268699] vtnet: page faults under high bandwidth load Date: Mon, 02 Jan 2023 00:52:50 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: nc@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D268699 Bug ID: 268699 Summary: vtnet: page faults under high bandwidth load Product: Base System Version: 13.1-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: nc@FreeBSD.org I run three high-bandwidth Tor exit relay on FreeBSD 13.1-RELEASE, in KVM V= PSes at BuyVM.net and AlienData.com (and more at other non-KVM hosts), namely 4GB RAM, dedicated CPU core AMD Ryzen VPS services (sort of virtual dedicated servers). Using the vtnet Ethernet adapter, I frequently get page faults like this, especially if I consistently push ~250-300 Mbps of traffic for a few days. Full page fault attached. Fatal trap 12: page fault while in kernel mode cpuid =3D 0; apic id =3D 00 fault virtual address =3D 0x8 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80cb8088 stack pointer =3D 0x0:0xfffffe00634be830 frame pointer =3D 0x0:0xfffffe00634be870 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 12 (irq25: virtio_pci0) trap number =3D 12 panic: page fault cpuid =3D 0 time =3D 1672613133 KDB: stack backtrace: #0 0xffffffff80c694a5 at kdb_backtrace+0x65 #1 0xffffffff80c1bb5f at vpanic+0x17f #2 0xffffffff80c1b9d3 at panic+0x43 #3 0xffffffff810afdf5 at trap_fatal+0x385 #4 0xffffffff810afe4f at trap_pfault+0x4f #5 0xffffffff810875b8 at calltrap+0x8 #6 0xffffffff80cb8297 at sbdrop+0x37 #7 0xffffffff80dd8021 at tcp_do_segment+0x2df1 #8 0xffffffff80dd44f1 at tcp_input_with_port+0xb61 #9 0xffffffff80dd3929 at tcp6_input_with_port+0x69 #10 0xffffffff80dd4ceb at tcp6_input+0xb #11 0xffffffff80e16eae at ip6_input+0x95e #12 0xffffffff80d530c9 at netisr_dispatch_src+0xb9 #13 0xffffffff80d36ee8 at ether_demux+0x138 #14 0xffffffff80d38275 at ether_nh_input+0x355 #15 0xffffffff80d530c9 at netisr_dispatch_src+0xb9 #16 0xffffffff80d37319 at ether_input+0x69 #17 0xffffffff80a51fde at vtnet_rxq_eof+0x73e Uptime: 16h0m55s Dumping 660 out of 4062 MB:..3%..13%..22%..32%..42%..51%..61%..71%..83%..93% __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 55 __asm("movq %%gs:%P1,%0" : "=3Dr" (td) : "n" (offsetof(stru= ct pcpu, (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 #1 doadump (textdump=3D) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xffffffff80c1b75c in kern_reboot (howto=3D260) at /usr/src/sys/kern/kern_shutdown.c:487 #3 0xffffffff80c1bbce in vpanic (fmt=3D0xffffffff811b4fb9 "%s",=20 ap=3D) at /usr/src/sys/kern/kern_shutdown.c:920 #4 0xffffffff80c1b9d3 in panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:844 #5 0xffffffff810afdf5 in trap_fatal (frame=3D0xfffffe00634be770, eva=3D8) at /usr/src/sys/amd64/amd64/trap.c:944 #6 0xffffffff810afe4f in trap_pfault (frame=3D0xfffffe00634be770,=20 usermode=3Dfalse, signo=3D, ucode=3D) at /usr/src/sys/amd64/amd64/trap.c:763 #7 #8 sbcut_internal (sb=3Dsb@entry=3D0xfffff800acf4b9c0, len=3D2306, len@ent= ry=3D2856) at /usr/src/sys/kern/uipc_sockbuf.c:1488 #9 0xffffffff80cb8297 in sbdrop (sb=3Dsb@entry=3D0xfffff800acf4b9c0, len= =3D2856) at /usr/src/sys/kern/uipc_sockbuf.c:1597 #10 0xffffffff80dd8021 in tcp_do_segment (m=3D,=20 th=3D, so=3D0x1, tp=3D0xfffffe00a90d4950, drop_hdrlen=3D= 72,=20 tlen=3D, iptos=3D0 '\000') at /usr/src/sys/netinet/tcp_input.c:1869 #11 0xffffffff80dd44f1 in tcp_input_with_port (mp=3D,=20 mp@entry=3D0xfffffe00634beb38, offp=3D,=20 offp@entry=3D0xfffffe00634beb30, proto=3D, port=3D0) at /usr/src/sys/netinet/tcp_input.c:1414 #12 0xffffffff80dd3929 in tcp6_input_with_port (mp=3D0xfffffe00634beb38,=20 offp=3D0xfffffe00634beb30, proto=3D, port=3Dport@entry= =3D0) at /usr/src/sys/netinet/tcp_input.c:627 #13 0xffffffff80dd4ceb in tcp6_input (mp=3D0xfffff800acf4b9c0,=20 offp=3D0xfffff800908a7800, proto=3D1665919160) at /usr/src/sys/netinet/tcp_input.c:634 #14 0xffffffff80e16eae in ip6_input (m=3D0x0) at /usr/src/sys/netinet6/ip6_input.c:929 #15 0xffffffff80d530c9 in netisr_dispatch_src (proto=3D6,=20 source=3Dsource@entry=3D0, m=3D0xfffff8004001c100) at /usr/src/sys/net/netisr.c:1143 #16 0xffffffff80d5349f in netisr_dispatch (proto=3D2901719488,=20 m=3D0xfffffe00634be8b8) at /usr/src/sys/net/netisr.c:1234 #17 0xffffffff80d36ee8 in ether_demux (ifp=3Difp@entry=3D0xfffff8000360e000= ,=20 m=3D0xfffff800908a7800) at /usr/src/sys/net/if_ethersubr.c:921 #18 0xffffffff80d38275 in ether_input_internal (ifp=3D0xfffff8000360e000,=20 m=3D0xfffff800908a7800) at /usr/src/sys/net/if_ethersubr.c:707 #19 ether_nh_input (m=3D) at /usr/src/sys/net/if_ethersubr.c= :737 #20 0xffffffff80d530c9 in netisr_dispatch_src (proto=3Dproto@entry=3D5,=20 source=3Dsource@entry=3D0, m=3Dm@entry=3D0xfffff8004001c100) at /usr/src/sys/net/netisr.c:1143 #21 0xffffffff80d5349f in netisr_dispatch (proto=3D2901719488, proto@entry= =3D5,=20 m=3D0xfffffe00634be8b8, m@entry=3D0xfffff8004001c100) at /usr/src/sys/net/netisr.c:1234 #22 0xffffffff80d37319 in ether_input (ifp=3D,=20 m=3D0xfffff8004001c100) at /usr/src/sys/net/if_ethersubr.c:828 #23 0xffffffff80a51fde in vtnet_rxq_input (rxq=3D0xfffff800037e1480,=20 m=3D0xfffff8004001c100, hdr=3D) at /usr/src/sys/dev/virtio/network/if_vtnet.c:2043 #24 vtnet_rxq_eof (rxq=3D, rxq@entry=3D0xfffff800037e1480) at /usr/src/sys/dev/virtio/network/if_vtnet.c:2147 #25 0xffffffff80a51797 in vtnet_rx_vq_process (rxq=3D0xfffff800037e1480,=20 tries=3D) at /usr/src/sys/dev/virtio/network/if_vtnet.c:= 2211 #26 0xffffffff80bdbcfa in intr_event_execute_handlers (ie=3D0xfffff800037c3= e00,=20 p=3D) at /usr/src/sys/kern/kern_intr.c:1168 #27 ithread_execute_handlers (ie=3D, p=3D) at /usr/src/sys/kern/kern_intr.c:1181 #28 ithread_loop (arg=3Darg@entry=3D0xfffff80003625840) at /usr/src/sys/kern/kern_intr.c:1269 #29 0xffffffff80bd8a9e in fork_exit ( callout=3D0xffffffff80bdbaa0 , arg=3D0xfffff80003625840,= =20 frame=3D0xfffffe00634bef40) at /usr/src/sys/kern/kern_fork.c:1093 #30 #31 mi_startup () at /usr/src/sys/kern/init_main.c:322 Backtrace stopped: Cannot access memory at address 0x8 (kgdb)=20 /etc/sysctl.conf is: kern.ipc.soacceptqueue=3D2048 net.inet.ip.portrange.first=3D1024 net.inet.tcp.cc.algorithm=3Dhtcp net.inet.tcp.cc.htcp.adaptive_backoff=3D1 net.inet.tcp.cc.htcp.rtt_scaling=3D1 net.inet.tcp.recvbuf_max=3D4194304 net.inet.tcp.recvspace=3D65536 net.inet.tcp.sendbuf_inc=3D65536 net.inet.tcp.sendbuf_max=3D4194304 net.inet.tcp.sendspace=3D65536 net.inet.tcp.abc_l_var=3D44 net.inet.tcp.initcwnd_segments=3D44 net.inet.tcp.rfc6675_pipe=3D1 /boot/loader.conf is: cc_htcp_load=3D"YES" Side notes: To sysctl.conf, if I add "net.inet.tcp.tso=3D0", I can get about a day or t= wo of uptime (from ~6-14 hours), but eventually page fault, as has happened on my BuyVM VPS. Right now, I'm trying "-rxcsum -txcsum -tso -lro -txcsum6 -vlanhwtag -vlanhwtso" in the "ifconfig_vtnet0" line/config, to see if that improves anything. Core dump attached --=20 You are receiving this mail because: You are the assignee for the bug.=