From nobody Fri Dec 17 20:07:25 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id DB5821907507 for ; Fri, 17 Dec 2021 20:07:26 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:7ae3:b5ff:fe1b:23b4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.lerctr.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JG0Q653Zjz3MVD; Fri, 17 Dec 2021 20:07:26 +0000 (UTC) (envelope-from ler@lerctr.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=ler2019; h=Content-Transfer-Encoding:Content-Type:Message-ID:References: In-Reply-To:Subject:Cc:To:From:Date:MIME-Version:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=3eWsZdqS1ERFywv0Uw5HQB9uvrU1r7+VrT0nNic8EHM=; b=HMpgRCTPYkeqt1KSxqnCO8MPfF 0qtuLDopGb3jF4ynjVEJ4AU67pu65xiiP/6RYBvME/xBW1sjFhfYdaksvKNp4bPeVUgxJz6ze9qEm Grl60g2i3QFl7QhlOPWcDIijw23b/h6dzbcKFJJ0XoHKhcHq4gqwC3goeOPuWiCMOSzvkkzY4vwjz ulZsp+aO1zhTnLtj78joETqTJZcZCk6bIQxbLpprwh57AV2OnzwNzfGhf3t0F3JEOArF5fJd8LRde /k6YffiyWjNQASDJV9PRMxQnyqLiIyaKP0yW8P2vWEtiXPu+FjbZVRPn3Lj0aGrFvhTmqNeuQkrkK DjidBB8Q==; Received-SPF: pass (thebighonker.lerctr.org: domain of lerctr.org designates 2001:470:1f0f:3ad:bb:dcff:fe50:d900 as permitted sender) client-ip=2001:470:1f0f:3ad:bb:dcff:fe50:d900; envelope-from=ler@lerctr.org; helo=webmail.lerctr.org; Received: from thebighonker.lerctr.org ([2001:470:1f0f:3ad:bb:dcff:fe50:d900]:36733 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2 (FreeBSD)) (envelope-from ) id 1myJVl-0005eW-R4; Fri, 17 Dec 2021 14:07:25 -0600 Received: from 76-250-255-117.lightspeed.austtx.sbcglobal.net ([76.250.255.117]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Fri, 17 Dec 2021 14:07:25 -0600 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Date: Fri, 17 Dec 2021 14:07:25 -0600 From: Larry Rosenman To: Mark Johnston Cc: Alexander Motin , Freebsd current Subject: Re: Panic: Page Fault in Kernel: Yesterday's CURRENT In-Reply-To: References: <3d1b5249a2c51670de496fad9e8b054c@lerctr.org> <9852ae04-6dd0-1cd4-13fe-e97c68e71b37@FreeBSD.org> Message-ID: X-Sender: ler@lerctr.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4JG0Q653Zjz3MVD X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On 12/17/2021 1:36 pm, Mark Johnston wrote: > On Fri, Dec 10, 2021 at 10:43:19AM -0600, Larry Rosenman wrote: >> 14-2021_12_07-1217 - - 1.87G 2021-12-07 >> 12:17 >> 14-2021_12_09-1957 NR / 121G 2021-12-09 >> 19:57 >> >> If that's any help > > I can't tell what this is saying. A kernel built on the 7th does not > crash, or...? Which revision did you update from before you started > seeing crashes? > > From a kgdb session it'd be useful to see output from > > (kgdb) frame 8 > (kgdb) p/x *tmp > > to start. > Correct, the 7th didn't panic, but the 9th did, and yesterday's too. Grrr ler in borg in /mnt🔒 on ☁️ (us-east-1) ❯ kgdb -c /var/crash/vmcore.0 /mnt/boot/kernel/kernel GNU gdb (GDB) 11.1 [GDB v11.1 for FreeBSD] Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd14.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /mnt/boot/kernel/kernel... (No debugging symbols found in /mnt/boot/kernel/kernel) Failed to open vmcore: /var/crash/vmcore.0: Permission denied (kgdb) bt No stack. quitb) ler in borg in /mnt🔒 on ☁️ (us-east-1) took 6s ❯ sudo chmod +r /var/crash/* ler in borg in /mnt🔒 on ☁️ (us-east-1) ❯ kgdb -c /var/crash/vmcore.0 /mnt/boot/kernel/kernel GNU gdb (GDB) 11.1 [GDB v11.1 for FreeBSD] Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd14.0". Type "show configuration" for configuration details. For bug reporting instructions, please see: . Find the GDB manual and other documentation resources online at: . For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /mnt/boot/kernel/kernel... (No debugging symbols found in /mnt/boot/kernel/kernel) /wrkdirs/usr/ports/devel/gdb/work-py37/gdb-11.1/gdb/thread.c:1345: internal-error: void switch_to_thread(thread_info *): Assertion `thr != NULL' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. Quit this debugging session? (y or n) n This is a bug, please report it. For instructions, see: . /wrkdirs/usr/ports/devel/gdb/work-py37/gdb-11.1/gdb/thread.c:1345: internal-error: void switch_to_thread(thread_info *): Assertion `thr != NULL' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. Create a core file of GDB? (y or n) n Command aborted. (kgdb) bt No thread selected. (kgdb) fr 8 No thread selected. (kgdb) >> On 12/10/2021 10:36 am, Alexander Motin wrote: >> > Hi Larry, >> > >> > This looks like some use-after-free or otherwise corrupted callout >> > structure. Unfortunately the backtrace does not tell what was the >> > callout. When was the previous update to look what could change? >> > >> > On 10.12.2021 11:24, Larry Rosenman wrote: >> >> FreeBSD borg.lerctr.org 14.0-CURRENT FreeBSD 14.0-CURRENT #15 >> >> main-n251537-ab639f2398b: Thu Dec  9 19:45:37 CST 2021     >> >> root@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/LER-MINIMAL  >> >> amd64 >> >> >> >> VMCORE *IS* available. >> >> >> >> >> >> >> >> >> >> Unread portion of the kernel message buffer: >> >> kernel trap 12 with interrupts disabled >> >> >> >> >> >> Fatal trap 12: page fault while in kernel mode >> >> cpuid = 0; apic id = 20 >> >> fault virtual address   = 0x0 >> >> fault code              = supervisor write data, page not present >> >> instruction pointer     = 0x20:0xffffffff804e0db4 >> >> stack pointer           = 0x0:0xfffffe0434de4e10 >> >> frame pointer           = 0x0:0xfffffe0434de4e70 >> >> code segment            = base 0x0, limit 0xfffff, type 0x1b >> >>                         = DPL 0, pres 1, long 1, def32 0, gran 1 >> >> processor eflags        = resume, IOPL = 0 >> >> current process         = 82990 (c++) >> >> trap number             = 12 >> >> panic: page fault >> >> cpuid = 0 >> >> time = 1639111198 >> >> KDB: stack backtrace: >> >> #0 0xffffffff8050fc95 at kdb_backtrace+0x65 >> >> #1 0xffffffff804c468f at vpanic+0x17f >> >> #2 0xffffffff804c4503 at panic+0x43 >> >> #3 0xffffffff807a2195 at trap_fatal+0x385 >> >> #4 0xffffffff807a21ef at trap_pfault+0x4f >> >> #5 0xffffffff80779c78 at calltrap+0x8 >> >> #6 0xffffffff8045ddb8 at handleevents+0x188 >> >> #7 0xffffffff8045ea3e at timercb+0x24e >> >> #8 0xffffffff807ca9eb at lapic_handle_timer+0x9b >> >> #9 0xffffffff8077b9b1 at Xtimerint+0xb1 >> >> Uptime: 2h28m57s >> >> Dumping 12829 out of 131023 >> >> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% >> >> >> >> __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 >> >> 55              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" >> >> (offsetof(struct pcpu, >> >> (kgdb) #0  __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 >> >> #1  doadump (textdump=) >> >>     at /usr/src/sys/kern/kern_shutdown.c:399 >> >> #2  0xffffffff804c428c in kern_reboot (howto=260) >> >>     at /usr/src/sys/kern/kern_shutdown.c:487 >> >> #3  0xffffffff804c46fe in vpanic (fmt=0xffffffff807e1276 "%s", >> >>     ap=) at /usr/src/sys/kern/kern_shutdown.c:920 >> >> #4  0xffffffff804c4503 in panic (fmt=) >> >>     at /usr/src/sys/kern/kern_shutdown.c:844 >> >> #5  0xffffffff807a2195 in trap_fatal (frame=0xfffffe0434de4d50, eva=0) >> >>     at /usr/src/sys/amd64/amd64/trap.c:946 >> >> #6  0xffffffff807a21ef in trap_pfault (frame=0xfffffe0434de4d50, >> >>     usermode=false, signo=, ucode=) >> >>     at /usr/src/sys/amd64/amd64/trap.c:765 >> >> #7  >> >> #8  0xffffffff804e0db4 in callout_process >> >> (now=now@entry=38385536922300) >> >>     at /usr/src/sys/kern/kern_timeout.c:488 >> >> #9  0xffffffff8045ddb8 in handleevents (now=now@entry=38385536922300, >> >>     fake=fake@entry=0) at /usr/src/sys/kern/kern_clocksource.c:213 >> >> #10 0xffffffff8045ea3e in timercb (et=0xffffffff80d475e0 , >> >>     arg=) at /usr/src/sys/kern/kern_clocksource.c:357 >> >> #11 0xffffffff807ca9eb in lapic_handle_timer >> >> (frame=0xfffffe0434de4f40) >> >>     at /usr/src/sys/x86/x86/local_apic.c:1364 >> >> #12 >> >> #13 0x000000080df42bb6 in ?? () >> >> Backtrace stopped: Cannot access memory at address 0x7ffffdef2c90 >> >> (kgdb) -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106