From nobody Mon Dec 27 19:17:33 2021 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 24FF51903011 for ; Mon, 27 Dec 2021 19:17:36 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:7ae3:b5ff:fe1b:23b4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.lerctr.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JN6r00HPZz4VlW; Mon, 27 Dec 2021 19:17:36 +0000 (UTC) (envelope-from ler@lerctr.org) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=ler2019; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=aUCRk54zcwCbB4pmoRZ+mQfpMruJjof491qHqTh26eg=; b=FFZcUHXCngNizQ/O03m9IQ/37i uKIctYJSoGQHkhmuffNzwEP/HpfCjRVlBahiqxxUvukJH8WXqPOD60FgZMGgGe5LHq0Y2SWVl6RW+ qjkOyamsnbDFqG9qb+DKE+8x7cmDK8deja9VupWrI7P0NNpd3et0NZHkVOzf3K/t1QWlf8HpAF1jV sw3USgxcL204IwUhqUYJsZey5JRLEaDqJk3lNQ2NiEXOCZ+ShXsnKt18v6K2KjFRRPdYaoVScFOk5 Jm84r4YSgOMxrMUUS1AfuhvOSfhozIgb6LU476word7A3QSftzJ4Yd4jfgTkFspF8LiR3ygFMDFTN unHohB9A==; Received-SPF: softfail (thebighonker.lerctr.org: transitioning domain of lerctr.org does not designate 76.250.255.117 as permitted sender) client-ip=76.250.255.117; envelope-from=ler@lerctr.org; helo=borg.lerctr.org; Received: from 76-250-255-117.lightspeed.austtx.sbcglobal.net ([76.250.255.117]:26282 helo=borg.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.94.2 (FreeBSD)) (envelope-from ) id 1n1vV0-0009d2-J6; Mon, 27 Dec 2021 13:17:34 -0600 Date: Mon, 27 Dec 2021 13:17:33 -0600 From: Larry Rosenman To: Konstantin Belousov Cc: Gleb Smirnoff , Alexander Motin , current@freebsd.org Subject: Re: My -CURRENT crashes.... Message-ID: <20211227191733.7mo64dvqejfsx4ck@borg.lerctr.org> References: <286c830efc0e12e3e7a7e9b2ede31c28@lerctr.org> <45ee5689-b24c-51b5-d7b7-33fd73ee7dce@FreeBSD.org> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4JN6r00HPZz4VlW X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On Mon, Dec 27, 2021 at 09:15:53PM +0200, Konstantin Belousov wrote: > On Mon, Dec 27, 2021 at 10:58:02AM -0800, Gleb Smirnoff wrote: > > On Mon, Dec 27, 2021 at 01:43:01PM -0500, Alexander Motin wrote: > > A> > This allows us to deduct that the callout belongs to proc subsystem and > > A> > we can retrieve the proc it points to: c_lock - 0x128 = 0xfffff8030521e548 > > A> > It is ccache in PRS_NORMAL state. And the "tmp" in our stack frame is its > > A> > p_itcallout. > > A> > > > A> > So there is something that would zero out most of the p_itcallout while > > A> > it is scheduled? > > A> > > A> So carefully zero it, but keep the lock pointer... The only way that > > A> comes to mind is callout_init_mtx() in do_fork() if we assume the > > A> process has completed and the struct proc was reused. I guess if we > > A> could somehow leak scheduled callout in exit1(). May be we could add > > A> some more assertions to try catch callout still being active there. > > > > Note that _callout_stop_safe(p_itcallout) is the only place in kernel where > > CS_EXECUTING is used. > > I would start asking are there any third-party modules loaded. Nope. Id Refs Address Size Name 1 239 0xffffffff80200000 d94b58 kernel 2 1 0xffffffff81441000 f990 ehci.ko 3 12 0xffffffff81451000 3da98 usb.ko 4 1 0xffffffff8148f000 70ae00 zfs.ko 5 5 0xffffffff81b9a000 5338 xdr.ko 6 1 0xffffffff81ba0000 ccf0 ukbd.ko 7 7 0xffffffff81bad000 5248 hid.ko 8 1 0xffffffff81bb3000 b2c0 uhci.ko 9 1 0xffffffff8203d000 cec8 tmpfs.ko 10 1 0xffffffff8204a000 3538 fdescfs.ko 11 2 0xffffffff8204e000 3240 procfs.ko 12 3 0xffffffff82052000 5778 pseudofs.ko 13 1 0xffffffff82058000 9290 aesni.ko 14 1 0xffffffff82062000 20f0 coretemp.ko 15 1 0xffffffff82065000 3238 filemon.ko 16 1 0xffffffff82069000 2dd58 linux.ko 17 4 0xffffffff82097000 aea8 linux_common.ko 18 1 0xffffffff820a2000 4250 ichsmb.ko 19 2 0xffffffff820a7000 2180 smbus.ko 20 1 0xffffffff820aa000 4c10 ichwd.ko 21 1 0xffffffff820af000 2220 cpuctl.ko 22 1 0xffffffff820b2000 4338 cryptodev.ko 23 1 0xffffffff820b7000 2238 dtraceall.ko 24 8 0xffffffff820ba000 8a60 opensolaris.ko 25 8 0xffffffff82200000 84a300 dtrace.ko 26 1 0xffffffff820c3000 2274 dtmalloc.ko 27 1 0xffffffff820c6000 3331 fbt.ko 28 1 0xffffffff820ca000 56570 fasttrap.ko 29 1 0xffffffff82121000 2258 sdt.ko 30 1 0xffffffff82124000 91b4 systrace.ko 31 1 0xffffffff8212e000 91b4 systrace_freebsd32.ko 32 1 0xffffffff82138000 234c profile.ko 33 1 0xffffffff8213b000 8b38 ipmi.ko 34 3 0xffffffff82144000 45b0 efirt.ko 35 1 0xffffffff82149000 75b0 if_bridge.ko 36 1 0xffffffff82151000 50d8 bridgestp.ko 37 1 0xffffffff82157000 1662c hwpmc.ko 38 1 0xffffffff8216e000 28bb8 tcp_rack.ko 39 1 0xffffffff82197000 21b8 mfip.ko 40 2 0xffffffff82a4b000 84470 cam.ko 41 1 0xffffffff8219a000 7d38 ioat.ko 42 1 0xffffffff821a2000 48888 if_bce.ko 43 1 0xffffffff82ad0000 17a50 miibus.ko 44 1 0xffffffff821eb000 44b0 usb_quirk.ko 45 1 0xffffffff821f0000 b3a8 usb_template.ko 46 1 0xffffffff821fc000 3268 ums.ko 47 1 0xffffffff82ae8000 92d0 xhci.ko 48 1 0xffffffff82af2000 6120 ohci.ko 49 1 0xffffffff82af9000 43ef8 nfscl.ko 50 3 0xffffffff82b3d000 18cf0 nfscommon.ko 51 3 0xffffffff82b56000 2168 nfssvc.ko 52 4 0xffffffff82b59000 138a0 krpc.ko 53 1 0xffffffff82b6d000 4e638 nfsd.ko 54 1 0xffffffff82bbc000 bdc0 nfslockd.ko 55 1 0xffffffff82bc8000 4168 ataintel.ko 56 2 0xffffffff82bcd000 8358 ata.ko 57 1 0xffffffff82bd6000 5388 atapci.ko 58 1 0xffffffff82bdc000 4d40 geom_label.ko 59 1 0xffffffff82be1000 29f58 linux64.ko 60 1 0xffffffff82c0b000 2260 pty.ko 61 1 0xffffffff82c0e000 639c linprocfs.ko 62 1 0xffffffff82c15000 3284 linsysfs.ko 63 1 0xffffffff82c19000 3378 acpi_wmi.ko 64 1 0xffffffff82c1d000 2280 uhid.ko 65 1 0xffffffff82c20000 3320 usbhid.ko 66 1 0xffffffff82c24000 31f8 hidbus.ko 67 1 0xffffffff82c28000 32c0 wmt.ko 68 1 0xffffffff82c2c000 41a38 pf.ko 69 1 0xffffffff82c6e000 2a08 mac_ntpd.ko 70 5 0xffffffff82c71000 fb28 netgraph.ko 71 1 0xffffffff82c81000 63f8 ng_netflow.ko 72 1 0xffffffff82c88000 41e8 ng_ksocket.ko 73 1 0xffffffff82c8d000 3180 ng_ether.ko 74 1 0xffffffff82c91000 3918 ng_socket.ko 75 1 0xffffffff82c95000 4708 nullfs.ko -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106