From nobody Fri May 27 05:26:36 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id F36AB1B407B4 for ; Fri, 27 May 2022 05:26:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4L8YFS025Tz3LT9 for ; Fri, 27 May 2022 05:26:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D20AD1BF6A for ; Fri, 27 May 2022 05:26:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 24R5QZmF057701 for ; Fri, 27 May 2022 05:26:35 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 24R5QZS9057700 for bugs@FreeBSD.org; Fri, 27 May 2022 05:26:35 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 264273] FreeBSD 13 nginx kernel panic probably sendfile Date: Fri, 27 May 2022 05:26:36 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: bblister@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1653629196; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=IEJ950vdAZLuJ3I9LeoxrP0NIeBdmjE7q+2cp54+OzQ=; b=HyjswzwpIf/oJzPWrR0illmlN3RoAyDUXhArr+x2sh/IlRUNjBK4HdZySGQSdrfHTk4Sbn rRSqyoG1oBVORmPxGmRo1B30oseOyfMacLGZ6AKLXAUCxzVUXZqOaiu4dCTAif45juqtx5 E1z9tR0r+41m8rZgRA/f6mwi6aCU+eQwupid4/gZ33IYyFTJcJfNGSeB/ozKosORWhE9nD izNuvitmk7j7dwpZj3+CyEiWTUKUXnzpEv2eMuTLv/6CceNFCyv06XVkoRu25MH302sk2G ZY2pqhunFyMJ/sdt0mucR0TAKnqPi7lmE5rykMV7v/rZsT/rHrrbvrUvGxn11g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1653629196; a=rsa-sha256; cv=none; b=oFPCjXuqyUJ7DvMLF12cGTj08fhaUdCN8+tJbFXcQ1rn1hi6rpzlD0CtmtzJuuP7tNt6Gz Hy63TOM0XxBelUHOvu1dpo8iNWIC5P6QTbreD/h6GXpTo0Q2yI16i4/lMdf97ZHaONQwZX 8uRcD8xbhNur/ug//+bontf0OY36+SxuRE3DVfq6Sd7rTJw1d2Y8xpzGT0r2YoPOTL0ZIR IDA9wWsIrmZid9gfjQHVdoy/FVDbyN+eS01WchYiO+vJRhYyoHipNTTOJdPJo5DQ4Dvu2a P9fHYYT24PryfbNDE5Y3hBbiMSnhlzo0m6hM2dRTMBIpDnee1IaRwdQ/iSg8PQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264273 Bug ID: 264273 Summary: FreeBSD 13 nginx kernel panic probably sendfile Product: Base System Version: 13.0-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: bblister@gmail.com I performed a binary upgrade from 12 to 13 Before 8 hours. I went to sleep,= and in the morning I saw 2 kernel panics have occurred.=20 Both traces indicate that all started from nginx and especially vn_sendfile. nginx is installed from binary packages. This is a production server running a couple of websites, inside a cloud VP= S.=20 It was running 12.2-RELEASE-p14 rock solid without any reboot, without any panic. The configuration is not changed. I issued now a sysctl net.inet.tcp.tso=3D0=20 to see if it will make the server more reliable. If kernel still panics I will have to disable sendfile in nginx. I cannot switch to stable and perform kernel patches, because this is in production. I can issue commands or change configuration or display specific config files. In >uname -a FreeBSD arch.ece.uowm.gr 13.0-RELEASE-p11 FreeBSD 13.0-RELEASE-p11 #0: Tue = Apr=20 5 18:54:35 UTC 2022=20=20=20=20 root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC= =20 amd64 Kernel panics: May 27 04:13:49 arch kernel: Fatal trap 12: page fault while in kernel mode May 27 04:13:49 arch kernel: cpuid =3D 2; apic id =3D 02 May 27 04:13:49 arch kernel: fault virtual address =3D 0x58 May 27 04:13:49 arch kernel: fault code =3D supervisor read data, p= age not present May 27 04:13:49 arch kernel: instruction pointer =3D 0x20:0xffffffff81086c80 May 27 04:13:49 arch kernel: stack pointer =3D 0x28:0xfffffe0073dedf50 May 27 04:13:49 arch kernel: frame pointer =3D 0x28:0xfffffe0073dedf50 May 27 04:13:49 arch kernel: code segment =3D base 0x0, limit 0xfffff, type 0x1b May 27 04:13:49 arch kernel: =3D DPL 0, pres 1, long 1, = def32 0, gran 1 May 27 04:13:49 arch kernel: processor eflags =3D interrupt enabled, resu= me, IOPL =3D 0 May 27 04:13:49 arch kernel: current process =3D 65527 (nginx) May 27 04:13:49 arch kernel: trap number =3D 12 May 27 04:13:49 arch kernel: panic: page fault May 27 04:13:49 arch kernel: cpuid =3D 2 May 27 04:13:49 arch kernel: time =3D 1653612365 May 27 04:13:49 arch kernel: KDB: stack backtrace: May 27 04:13:49 arch kernel: #0 0xffffffff80c57535 at kdb_backtrace+0x65 May 27 04:13:49 arch kernel: #1 0xffffffff80c09f11 at vpanic+0x181 May 27 04:13:49 arch kernel: #2 0xffffffff80c09d83 at panic+0x43 May 27 04:13:49 arch kernel: #3 0xffffffff8108b1a7 at trap_fatal+0x387 May 27 04:13:49 arch kernel: #4 0xffffffff8108b1ff at trap_pfault+0x4f May 27 04:13:49 arch kernel: #5 0xffffffff8108a85d at trap+0x27d May 27 04:13:49 arch kernel: #6 0xffffffff81061f08 at calltrap+0x8 May 27 04:13:49 arch kernel: #7 0xffffffff80c9c38f at m_pullup+0x1af May 27 04:13:49 arch kernel: #8 0xffffffff821162bf at ipfw_chk+0x3fcf May 27 04:13:49 arch kernel: #9 0xffffffff8211805c at ipfw_check_frame+0x13c May 27 04:13:49 arch kernel: #10 0xffffffff80d422c7 at pfil_run_hooks+0x97 May 27 04:13:49 arch kernel: #11 0xffffffff80d23bf4 at ether_output_frame+0= x94 May 27 04:13:49 arch kernel: #12 0xffffffff80d23b08 at ether_output+0x6b8 May 27 04:13:49 arch kernel: #13 0xffffffff80db3ca5 at ip_output_send+0x75 May 27 04:13:49 arch kernel: #14 0xffffffff80db3ac2 at ip_output+0x12b2 May 27 04:13:49 arch kernel: #15 0xffffffff80dc9ab4 at tcp_output+0x1b04 May 27 04:13:49 arch kernel: #16 0xffffffff80ddb189 at tcp_usr_send+0x229 May 27 04:13:49 arch kernel: #17 0xffffffff80c07e2a at vn_sendfile+0x197a May 27 04:13:49 arch kernel: Uptime: 1h42m58s --------------------Another Panic-------------------- May 27 05:09:25 arch kernel: Fatal trap 12: page fault while in kernel mode May 27 05:09:25 arch kernel: cpuid =3D 0; apic id =3D 00 May 27 05:09:25 arch kernel: fault virtual address =3D 0x580 May 27 05:09:25 arch kernel: fault code =3D supervisor read data, p= age not present May 27 05:09:25 arch kernel: instruction pointer =3D 0x20:0xffffffff81086c80 May 27 05:09:25 arch kernel: stack pointer =3D 0x28:0xfffffe00bb153f50 May 27 05:09:25 arch kernel: frame pointer =3D 0x28:0xfffffe00bb153f50 May 27 05:09:25 arch kernel: code segment =3D base 0x0, limit 0xfffff, type 0x1b May 27 05:09:25 arch kernel: =3D DPL 0, pres 1, long 1, = def32 0, gran 1 May 27 05:09:25 arch kernel: processor eflags =3D interrupt enabled, resu= me, IOPL =3D 0 May 27 05:09:25 arch kernel: current process =3D 97681 (nginx) May 27 05:09:25 arch kernel: trap number =3D 12 May 27 05:09:25 arch kernel: panic: page fault May 27 05:09:25 arch kernel: cpuid =3D 0 May 27 05:09:25 arch kernel: time =3D 1653615832 May 27 05:09:25 arch kernel: KDB: stack backtrace: May 27 05:09:25 arch kernel: #0 0xffffffff80c57535 at kdb_backtrace+0x65 May 27 05:09:25 arch kernel: #1 0xffffffff80c09f11 at vpanic+0x181 May 27 05:09:25 arch kernel: #2 0xffffffff80c09d83 at panic+0x43 May 27 05:09:25 arch kernel: #3 0xffffffff8108b1a7 at trap_fatal+0x387 May 27 05:09:25 arch kernel: #4 0xffffffff8108b1ff at trap_pfault+0x4f May 27 05:09:25 arch kernel: #5 0xffffffff8108a85d at trap+0x27d May 27 05:09:25 arch kernel: #6 0xffffffff81061f08 at calltrap+0x8 May 27 05:09:25 arch kernel: #7 0xffffffff80c9c38f at m_pullup+0x1af May 27 05:09:25 arch kernel: #8 0xffffffff8211f2bf at ipfw_chk+0x3fcf May 27 05:09:25 arch kernel: #9 0xffffffff8212105c at ipfw_check_frame+0x13c May 27 05:09:25 arch kernel: #10 0xffffffff80d422c7 at pfil_run_hooks+0x97 May 27 05:09:25 arch kernel: #11 0xffffffff80d23bf4 at ether_output_frame+0= x94 May 27 05:09:25 arch kernel: #12 0xffffffff80d23b08 at ether_output+0x6b8 May 27 05:09:25 arch kernel: #13 0xffffffff80db3ca5 at ip_output_send+0x75 May 27 05:09:25 arch kernel: #14 0xffffffff80db3ac2 at ip_output+0x12b2 May 27 05:09:25 arch kernel: #15 0xffffffff80dc9ab4 at tcp_output+0x1b04 May 27 05:09:25 arch kernel: #16 0xffffffff80ddb189 at tcp_usr_send+0x229 May 27 05:09:25 arch kernel: #17 0xffffffff80c07e2a at vn_sendfile+0x197a May 27 05:09:25 arch kernel: Uptime: 56m29s --=20 You are receiving this mail because: You are the assignee for the bug.=