From nobody Wed Sep 01 03:07:53 2021 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7FC2F179DF24 for ; Wed, 1 Sep 2021 03:08:10 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ua1-x92a.google.com (mail-ua1-x92a.google.com [IPv6:2607:f8b0:4864:20::92a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GzpsQ31Ddz4tkT for ; Wed, 1 Sep 2021 03:08:10 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-ua1-x92a.google.com with SMTP id r20so592038uat.2 for ; Tue, 31 Aug 2021 20:08:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=XZobRO5AVdmQaeRiD300sVTU4hViUkuJRQbiuOazAjs=; b=xPgxL+yacIRICiriA0nhYPg+tUf6cle8bhkk8MtIEGE1g3lk/LUWX5y8wnmyinV0tQ eDoGcRcNuI7Vf6h6q1y/ZjrTbdYwFe1dSSjnAzQPgA8cBPvjZzHxbgr3M1DDjBCl4gYS 58UN6W+kz2ykOYRHKPYuYRgTS1yR40YdgvWEEjOUPpSmtsKTC7M5bT+Hl2A+CPahVZE2 6gwDh49SXQ3e6X67wuT21Qq+hn8VFNgnAaSZ9MSJRnwQS7Y2oAHDgJo7wo7AjPC/U45o ezXMeP+nIpLCqethKrEkNPbKvXxMIJQ0mLop+ZrrcczrfWfrxEwss+v2OE6XWyjvqEjH Dhrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=XZobRO5AVdmQaeRiD300sVTU4hViUkuJRQbiuOazAjs=; b=iAZTTeOlSIPq6KPMTlqtAfww7yRrSeafJUpQmc+1UolAfATvWgu9OQigoKvkdtYoBg VeLKemn6HdvzdsERGoIqrxIqg10rrLf2NBnYyzXZsMRZaah0JuzsnmDGl4QpTXeaCT/L DIyEr8/9iChz/6q/hHizbVX8e3HwOESENRS2hoA0Izs5cASndnH6PjUSeU+XQXKH8fqU YZuXtGGNIbUzzkwDgEWkGUjNNZ+Dt9Wo3bVvUKVHxSy5Hf8vDBOrUBEE5KASFHVuOD3+ v7hkiZrzpXnpdwVYxgCiH8cms/pqPMTBN1z2CyzHOjlCKuHZT0O3Al6OO81BWrsdkO72 8UUA== X-Gm-Message-State: AOAM532SUVxa95qnOZIiklY0YBfXn2m1UnCkEb8TW5iyDj1T3m7lDFMU WO1prcz0g8oeWtlqJMdLN33jyfRc1U4vZehKtzeAmw== X-Google-Smtp-Source: ABdhPJybgl5yw6SrrDnXyr3Q3o1Hp7UvflbxRxyg2zIf2wagZHYNv0KIZd7Dq5vkoadW19iJ0Akpf/4FDUfbq8X2oNE= X-Received: by 2002:ab0:3413:: with SMTP id z19mr21360660uap.39.1630465684230; Tue, 31 Aug 2021 20:08:04 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <5476ea21-9e8a-32f5-08ff-add46c02d910@freebsd.org> In-Reply-To: <5476ea21-9e8a-32f5-08ff-add46c02d910@freebsd.org> From: Warner Losh Date: Tue, 31 Aug 2021 21:07:53 -0600 Message-ID: Subject: Re: Patched gpsd and /dev/pps0 results in "sleeping thread" kernel panic To: Craig Leres Cc: FreeBSD Hackers , Ian Lepore Content-Type: multipart/alternative; boundary="000000000000ed626c05cae65f94" X-Rspamd-Queue-Id: 4GzpsQ31Ddz4tkT X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: Y --000000000000ed626c05cae65f94 Content-Type: text/plain; charset="UTF-8" On Tue, Aug 31, 2021 at 8:11 PM Craig Leres wrote: > > On 8/31/21 6:54 PM, Warner Losh wrote: > > Can you post a traceback? We recently tightened up the sleeping > conditions > > and many of the panics are straight forward to fix > > Is this sufficient? > Maybe. I'm guessing that the pps driver isn't setting driver_mtx for the pps_state that it keeps, so when we're sleeping in pps_ioctl we're not dropping the the ppbus lock, which leads to this diagnostic. Warner > Craig > > toc2 4 # kgdb /boot/kernel/kernel /var/crash/vmcore.1 > GNU gdb (GDB) 10.2 [GDB v10.2 for FreeBSD] > Copyright (C) 2021 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. > Type "show copying" and "show warranty" for details. > This GDB was configured as "x86_64-portbld-freebsd12.2". > Type "show configuration" for configuration details. > For bug reporting instructions, please see: > . > Find the GDB manual and other documentation resources online at: > . > > For help, type "help". > Type "apropos word" to search for commands related to "word"... > Reading symbols from /boot/kernel/kernel... > Reading symbols from /boot/kernel.LBLNET/kernel.debug... > > Unread portion of the kernel message buffer: > Sleeping thread (tid 101408, pid 2046) owns a non-sleepable lock > KDB: stack backtrace of thread 101408: > sched_switch() at sched_switch+0x630/frame 0xfffffe0070e4f760 > mi_switch() at mi_switch+0xd4/frame 0xfffffe0070e4f790 > sleepq_catch_signals() at sleepq_catch_signals+0x403/frame > 0xfffffe0070e4f7e0 > sleepq_timedwait_sig() at sleepq_timedwait_sig+0x14/frame > 0xfffffe0070e4f820 > _sleep() at _sleep+0x1b3/frame 0xfffffe0070e4f8a0 > pps_ioctl() at pps_ioctl+0x298/frame 0xfffffe0070e4f8f0 > ppsioctl() at ppsioctl+0x48/frame 0xfffffe0070e4f920 > devfs_ioctl() at devfs_ioctl+0xb0/frame 0xfffffe0070e4f970 > VOP_IOCTL_APV() at VOP_IOCTL_APV+0x7b/frame 0xfffffe0070e4f9a0 > vn_ioctl() at vn_ioctl+0x16a/frame 0xfffffe0070e4fab0 > devfs_ioctl_f() at devfs_ioctl_f+0x1e/frame 0xfffffe0070e4fad0 > kern_ioctl() at kern_ioctl+0x2b7/frame 0xfffffe0070e4fb30 > sys_ioctl() at sys_ioctl+0xfa/frame 0xfffffe0070e4fc00 > amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe0070e4fd30 > fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0070e4fd30 > --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8004c899a, rsp = > 0x7fffdfdfc6a8, rbp = 0x7fffdfdfc730 --- > panic: sleeping thread > cpuid = 8 > time = 1630456142 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame > 0xfffffe005ab73ab0 > vpanic() at vpanic+0x17b/frame 0xfffffe005ab73b00 > panic() at panic+0x43/frame 0xfffffe005ab73b60 > propagate_priority() at propagate_priority+0x282/frame 0xfffffe005ab73b90 > turnstile_wait() at turnstile_wait+0x30c/frame 0xfffffe005ab73be0 > __mtx_lock_sleep() at __mtx_lock_sleep+0x199/frame 0xfffffe005ab73c70 > ppcintr() at ppcintr+0x2a0/frame 0xfffffe005ab73c90 > ithread_loop() at ithread_loop+0x23c/frame 0xfffffe005ab73cf0 > fork_exit() at fork_exit+0x7e/frame 0xfffffe005ab73d30 > fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe005ab73d30 > --- trap 0, rip = 0, rsp = 0, rbp = 0 --- > Uptime: 17m20s > Dumping 607 out of 12240 > MB:..3%..11%..22%..32%..43%..51%..61%..72%..82%..93% > > __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > 55 /usr/src/sys/amd64/include/pcpu_aux.h: No such file or directory. > (kgdb) bt > #0 __curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:55 > #1 doadump (textdump=1) at ../../../kern/kern_shutdown.c:371 > #2 0xffffffff80b83ada in kern_reboot (howto=260) > at ../../../kern/kern_shutdown.c:451 > #3 0xffffffff80b83f33 in vpanic (fmt=, ap=) > at ../../../kern/kern_shutdown.c:880 > #4 0xffffffff80b83d53 in panic (fmt=) > at ../../../kern/kern_shutdown.c:807 > #5 0xffffffff80be7152 in propagate_priority (td=0xfffff80196959740) > at ../../../kern/subr_turnstile.c:228 > #6 0xffffffff80be7d1c in turnstile_wait (ts=0xfffff8000357a780, > owner=, queue=0) at ../../../kern/subr_turnstile.c:785 > #7 0xffffffff80b62ca9 in __mtx_lock_sleep (c=0xfffff80003969bd0, > v=) at ../../../kern/kern_mutex.c:654 > #8 0xffffffff8086fcc0 in ppcintr (arg=0xfffff80003969b00) > at ../../../dev/ppc/ppc.c:1546 > #9 0xffffffff80b4637c in intr_event_execute_handlers (p=, > ie=0xfffff800030d9d00) at ../../../kern/kern_intr.c:1143 > #10 ithread_execute_handlers (p=, ie=0xfffff800030d9d00) > at ../../../kern/kern_intr.c:1156 > #11 ithread_loop (arg=0xfffff800039aa7e0) at ../../../kern/kern_intr.c:1236 > #12 0xffffffff80b42e1e in fork_exit ( > callout=0xffffffff80b46140 , arg=0xfffff800039aa7e0, > frame=0xfffffe005ab73d40) at ../../../kern/kern_fork.c:1080 > #13 > (kgdb) > --000000000000ed626c05cae65f94--