From nobody Fri Dec 17 03:16:21 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DB16518F92E9 for ; Fri, 17 Dec 2021 03:16:22 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:7ae3:b5ff:fe1b:23b4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.lerctr.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JFYzV1RLyz4Z7s; Fri, 17 Dec 2021 03:16:22 +0000 (UTC) (envelope-from ler@lerctr.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=ler2019; h=Content-Transfer-Encoding:Content-Type:Message-ID:References: In-Reply-To:Subject:Cc:To:From:Date:MIME-Version:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=ZdVr4c33ku13UDD7gLE/xSnFhkN9pr8nK3cqOpncz+Q=; b=Nju3+cTmrfVFjvgvdO7JTyG+ig FdLcs/QXuqQG31rCVhs6lAR7SByuEtYLaZa1ugr67+lwd+yCrANIZsllcsLSIBeUzOAUEpBonUvBu yhK/xxhos/7+c0/vqyxzdCPFy6P9xajR4gSRdiWjXl60/CZ8uCcGHgENW9Imtqp00yboBzVdt0QRS uw/Pu+oKPRobZDWvp6Hc3q+QQu9q2DmfIfym6evk8LpBSiQkWovPiTwUMDcpxlT+Ra3ETCxZRN6kT XNMmEZP/XId+lOenGLxmKKt3aRiFbrHnJJljD0bqbedRpYsStwkOix2rMZ8NQBPFB+CSL/Kkty38H EJLcw+ew==; Received-SPF: pass (thebighonker.lerctr.org: domain of lerctr.org designates 2001:470:1f0f:3ad:bb:dcff:fe50:d900 as permitted sender) client-ip=2001:470:1f0f:3ad:bb:dcff:fe50:d900; envelope-from=ler@lerctr.org; helo=webmail.lerctr.org; Received: from thebighonker.lerctr.org ([2001:470:1f0f:3ad:bb:dcff:fe50:d900]:22628 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2 (FreeBSD)) (envelope-from ) id 1my3jJ-000GFA-Fd; Thu, 16 Dec 2021 21:16:21 -0600 Received: from 76-250-255-117.lightspeed.austtx.sbcglobal.net ([76.250.255.117]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Thu, 16 Dec 2021 21:16:21 -0600 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Date: Thu, 16 Dec 2021 21:16:21 -0600 From: Larry Rosenman To: Alexander Motin Cc: Freebsd current Subject: Re: Panic: Page Fault in Kernel: Yesterday's CURRENT In-Reply-To: References: <3d1b5249a2c51670de496fad9e8b054c@lerctr.org> <9852ae04-6dd0-1cd4-13fe-e97c68e71b37@FreeBSD.org> Message-ID: <35ce3d9ab427375f09b44c2ced1e1704@lerctr.org> X-Sender: ler@lerctr.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4JFYzV1RLyz4Z7s X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=lerctr.org header.s=ler2019 header.b=Nju3+cTm; dmarc=pass (policy=none) header.from=lerctr.org; spf=pass (mx1.freebsd.org: domain of ler@lerctr.org designates 2001:470:1f0f:3ad:7ae3:b5ff:fe1b:23b4 as permitted sender) smtp.mailfrom=ler@lerctr.org X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[lerctr.org:s=ler2019]; FREEFALL_USER(0.00)[ler]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[lerctr.org:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[lerctr.org,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 12/16/2021 9:03 pm, Larry Rosenman wrote: > On 12/10/2021 10:43 am, Larry Rosenman wrote: >> 14-2021_12_07-1217 - - 1.87G 2021-12-07 >> 12:17 >> 14-2021_12_09-1957 NR / 121G 2021-12-09 >> 19:57 >> >> If that's any help >> >> On 12/10/2021 10:36 am, Alexander Motin wrote: >>> Hi Larry, >>> >>> This looks like some use-after-free or otherwise corrupted callout >>> structure. Unfortunately the backtrace does not tell what was the >>> callout. When was the previous update to look what could change? >>> >>> On 10.12.2021 11:24, Larry Rosenman wrote: >>>> FreeBSD borg.lerctr.org 14.0-CURRENT FreeBSD 14.0-CURRENT #15 >>>> main-n251537-ab639f2398b: Thu Dec  9 19:45:37 CST 2021     >>>> root@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL  >>>> amd64 >>>> >>>> VMCORE *IS* available. >>>> >>>> >>>> >>>> >>>> Unread portion of the kernel message buffer: >>>> kernel trap 12 with interrupts disabled >>>> >>>> >>>> Fatal trap 12: page fault while in kernel mode >>>> cpuid = 0; apic id = 20 >>>> fault virtual address   = 0x0 >>>> fault code              = supervisor write data, page not present >>>> instruction pointer     = 0x20:0xffffffff804e0db4 >>>> stack pointer           = 0x0:0xfffffe0434de4e10 >>>> frame pointer           = 0x0:0xfffffe0434de4e70 >>>> code segment            = base 0x0, limit 0xfffff, type 0x1b >>>>                         = DPL 0, pres 1, long 1, def32 0, gran 1 >>>> processor eflags        = resume, IOPL = 0 >>>> current process         = 82990 (c++) >>>> trap number             = 12 >>>> panic: page fault >>>> cpuid = 0 >>>> time = 1639111198 >>>> KDB: stack backtrace: >>>> #0 0xffffffff8050fc95 at kdb_backtrace+0x65 >>>> #1 0xffffffff804c468f at vpanic+0x17f >>>> #2 0xffffffff804c4503 at panic+0x43 >>>> #3 0xffffffff807a2195 at trap_fatal+0x385 >>>> #4 0xffffffff807a21ef at trap_pfault+0x4f >>>> #5 0xffffffff80779c78 at calltrap+0x8 >>>> #6 0xffffffff8045ddb8 at handleevents+0x188 >>>> #7 0xffffffff8045ea3e at timercb+0x24e >>>> #8 0xffffffff807ca9eb at lapic_handle_timer+0x9b >>>> #9 0xffffffff8077b9b1 at Xtimerint+0xb1 >>>> Uptime: 2h28m57s >>>> Dumping 12829 out of 131023 >>>> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% >>>> >>>> __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 >>>> 55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" >>>> (offsetof(struct pcpu, >>>> (kgdb) #0  __curthread () at >>>> /usr/src/sys/amd64/include/pcpu_aux.h:55 >>>> #1  doadump (textdump=) >>>>     at /usr/src/sys/kern/kern_shutdown.c:399 >>>> #2  0xffffffff804c428c in kern_reboot (howto=260) >>>>     at /usr/src/sys/kern/kern_shutdown.c:487 >>>> #3  0xffffffff804c46fe in vpanic (fmt=0xffffffff807e1276 "%s", >>>>     ap=) at /usr/src/sys/kern/kern_shutdown.c:920 >>>> #4  0xffffffff804c4503 in panic (fmt=) >>>>     at /usr/src/sys/kern/kern_shutdown.c:844 >>>> #5  0xffffffff807a2195 in trap_fatal (frame=0xfffffe0434de4d50, >>>> eva=0) >>>>     at /usr/src/sys/amd64/amd64/trap.c:946 >>>> #6  0xffffffff807a21ef in trap_pfault (frame=0xfffffe0434de4d50, >>>>     usermode=false, signo=, ucode=) >>>>     at /usr/src/sys/amd64/amd64/trap.c:765 >>>> #7  >>>> #8  0xffffffff804e0db4 in callout_process >>>> (now=now@entry=38385536922300) >>>>     at /usr/src/sys/kern/kern_timeout.c:488 >>>> #9  0xffffffff8045ddb8 in handleevents >>>> (now=now@entry=38385536922300, >>>>     fake=fake@entry=0) at /usr/src/sys/kern/kern_clocksource.c:213 >>>> #10 0xffffffff8045ea3e in timercb (et=0xffffffff80d475e0 , >>>>     arg=) at /usr/src/sys/kern/kern_clocksource.c:357 >>>> #11 0xffffffff807ca9eb in lapic_handle_timer >>>> (frame=0xfffffe0434de4f40) >>>>     at /usr/src/sys/x86/x86/local_apic.c:1364 >>>> #12 >>>> #13 0x000000080df42bb6 in ?? () >>>> Backtrace stopped: Cannot access memory at address 0x7ffffdef2c90 >>>> (kgdb) >>>> >>>> ------------------------------------------------------------------------ >>>> > ' > > I got a new crash on a today's current: > ❯ more core.txt.1 > borg.lerctr.org dumped core - see /var/crash/vmcore.1 > > Thu Dec 16 17:01:37 CST 2021 > > FreeBSD borg.lerctr.org 14.0-CURRENT FreeBSD 14.0-CURRENT #22 > main-n251748-c610426c4de: Thu Dec 16 09:22:52 CST 2021 > root@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL > amd64 > > panic: page fault > > GNU gdb (GDB) 11.1 [GDB v11.1 for FreeBSD] > Copyright (C) 2021 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > Type "show copying" and "show warranty" for details. > This GDB was configured as "x86_64-portbld-freebsd14.0". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > . > Find the GDB manual and other documentation resources online at: > . > > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from /boot/kernel/kernel... > Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug... > > Unread portion of the kernel message buffer: > kernel trap 12 with interrupts disabled > > > Fatal trap 12: page fault while in kernel mode > cpuid = 0; apic id = 20 > fault virtual address = 0x0 > fault code = supervisor write data, page not present > instruction pointer = 0x20:0xffffffff804e0a34 > stack pointer = 0x0:0xfffffe03441a0e10 > frame pointer = 0x0:0xfffffe03441a0e70 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = resume, IOPL = 0 > current process = 86479 (c++) > trap number = 12 > panic: page fault > cpuid = 0 > time = 1639694532 > KDB: stack backtrace: > #0 0xffffffff8050f9e5 at kdb_backtrace+0x65 > #1 0xffffffff804c430f at vpanic+0x17f > #2 0xffffffff804c4183 at panic+0x43 > #3 0xffffffff807a2195 at trap_fatal+0x385 > #4 0xffffffff807a21ef at trap_pfault+0x4f > #5 0xffffffff80779728 at calltrap+0x8 > #6 0xffffffff8045da08 at handleevents+0x188 > #7 0xffffffff8045e68e at timercb+0x24e > #8 0xffffffff807ca9eb at lapic_handle_timer+0x9b > #9 0xffffffff8077b461 at Xtimerint+0xb1 > Uptime: 7h7m44s > Dumping 13614 out of 131023 > MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% > > __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > 55 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" > (offsetof(struct pcpu, > (kgdb) #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > #1 doadump (textdump=) > at /usr/src/sys/kern/kern_shutdown.c:399 > #2 0xffffffff804c3f0c in kern_reboot (howto=260) > at /usr/src/sys/kern/kern_shutdown.c:487 > #3 0xffffffff804c437e in vpanic (fmt=0xffffffff807e1291 "%s", > ap=) at /usr/src/sys/kern/kern_shutdown.c:920 > #4 0xffffffff804c4183 in panic (fmt=) > at /usr/src/sys/kern/kern_shutdown.c:844 > #5 0xffffffff807a2195 in trap_fatal (frame=0xfffffe03441a0d50, eva=0) > at /usr/src/sys/amd64/amd64/trap.c:946 > #6 0xffffffff807a21ef in trap_pfault (frame=0xfffffe03441a0d50, > usermode=false, signo=, ucode=) > at /usr/src/sys/amd64/amd64/trap.c:765 > #7 > #8 0xffffffff804e0a34 in callout_process > (now=now@entry=110228055503582) > at /usr/src/sys/kern/kern_timeout.c:488 > #9 0xffffffff8045da08 in handleevents (now=now@entry=110228055503582, > fake=fake@entry=0) at /usr/src/sys/kern/kern_clocksource.c:213 > #10 0xffffffff8045e68e in timercb (et=0xffffffff80d47660 , > arg=) at /usr/src/sys/kern/kern_clocksource.c:357 > #11 0xffffffff807ca9eb in lapic_handle_timer (frame=0xfffffe03441a0f40) > at /usr/src/sys/x86/x86/local_apic.c:1364 > #12 > #13 0x0000000003883679 in ?? () > Backtrace stopped: Cannot access memory at address 0x7fffffff6f20 > (kgdb) > > > Core is ALSO available > > Boot Env: > ❯ bectl list > BE Active Mountpoint Space Created > 14-2021-10-26_1554 - - 1.82G 2021-10-26 15:54 > 14-2021-11-03-1800 - - 162M 2021-11-03 18:00 > 14-2021_10_19-1900 - - 1.80G 2021-10-19 18:57 > 14-2021_10_20-0800 - - 1.94G 2021-10-20 08:01 > 14-2021_11_18-1241 - - 1.86G 2021-11-18 11:41 > 14-2021_11_20-1417 - - 1.85G 2021-11-20 13:17 > 14-2021_11_23-1111 - - 1.87G 2021-11-23 11:11 > 14-2021_11_25-1312 - - 1.87G 2021-11-25 12:12 > 14-2021_12_04-2220 - - 13.7M 2021-12-04 22:20 > 14-2021_12_07-1217 - - 1.87G 2021-12-07 12:17 > 14-2021_12_09-1957 - - 1.89G 2021-12-09 19:57 > 14-2021_12_14-0224 - - 1.82G 2021-12-14 02:24 > 14-2021_12_15-0923 - - 18.6M 2021-12-15 09:23 > 14-2021_12_15-2133 - - 1.83G 2021-12-15 21:33 > 14-2021_12_15-2257 - - 1.82G 2021-12-15 22:57 > 14-2021_12_16-0924 NR / 129G 2021-12-16 09:24 > 14-main-first - - 2.69G 2021-10-02 20:11 > 14.0-CURRENT-2021-10-04_1051 - - 16.6M 2021-10-04 10:51 > 14.0-CURRENT_2021-10-06_184540 - - 12.9M 2021-10-06 18:46 > 14.0-CURRENT_2021-11-04_091349 - - 17.7M 2021-11-04 09:13 > 14.0-CURRENT_2021-12-05_204803 - - 10.6M 2021-12-05 20:48 > r363086 - - 4.19G 2020-07-10 15:37 > > what else can I supply? FTR Both crashes were during LONG poudriere runs rebuilding all 800+ ports I use after a FreeBSD_Version bump. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106