Re: My -CURRENT crashes....

From: Larry Rosenman <ler_at_lerctr.org>
Date: Mon, 27 Dec 2021 19:17:33 UTC
On Mon, Dec 27, 2021 at 09:15:53PM +0200, Konstantin Belousov wrote:
> On Mon, Dec 27, 2021 at 10:58:02AM -0800, Gleb Smirnoff wrote:
> > On Mon, Dec 27, 2021 at 01:43:01PM -0500, Alexander Motin wrote:
> > A> > This allows us to deduct that the callout belongs to proc subsystem and
> > A> > we can retrieve the proc it points to: c_lock - 0x128 = 0xfffff8030521e548
> > A> > It is ccache in PRS_NORMAL state. And the "tmp" in our stack frame is its
> > A> > p_itcallout.
> > A> > 
> > A> > So there is something that would zero out most of the p_itcallout while
> > A> > it is scheduled?
> > A> 
> > A> So carefully zero it, but keep the lock pointer...  The only way that
> > A> comes to mind is callout_init_mtx() in do_fork() if we assume the
> > A> process has completed and the struct proc was reused.  I guess if we
> > A> could somehow leak scheduled callout in exit1().  May be we could add
> > A> some more assertions to try catch callout still being active there.
> > 
> > Note that _callout_stop_safe(p_itcallout) is the only place in kernel where
> > CS_EXECUTING is used.
> 
> I would start asking are there any third-party modules loaded.

Nope.

Id Refs Address                Size Name
 1  239 0xffffffff80200000   d94b58 kernel
 2    1 0xffffffff81441000     f990 ehci.ko
 3   12 0xffffffff81451000    3da98 usb.ko
 4    1 0xffffffff8148f000   70ae00 zfs.ko
 5    5 0xffffffff81b9a000     5338 xdr.ko
 6    1 0xffffffff81ba0000     ccf0 ukbd.ko
 7    7 0xffffffff81bad000     5248 hid.ko
 8    1 0xffffffff81bb3000     b2c0 uhci.ko
 9    1 0xffffffff8203d000     cec8 tmpfs.ko
10    1 0xffffffff8204a000     3538 fdescfs.ko
11    2 0xffffffff8204e000     3240 procfs.ko
12    3 0xffffffff82052000     5778 pseudofs.ko
13    1 0xffffffff82058000     9290 aesni.ko
14    1 0xffffffff82062000     20f0 coretemp.ko
15    1 0xffffffff82065000     3238 filemon.ko
16    1 0xffffffff82069000    2dd58 linux.ko
17    4 0xffffffff82097000     aea8 linux_common.ko
18    1 0xffffffff820a2000     4250 ichsmb.ko
19    2 0xffffffff820a7000     2180 smbus.ko
20    1 0xffffffff820aa000     4c10 ichwd.ko
21    1 0xffffffff820af000     2220 cpuctl.ko
22    1 0xffffffff820b2000     4338 cryptodev.ko
23    1 0xffffffff820b7000     2238 dtraceall.ko
24    8 0xffffffff820ba000     8a60 opensolaris.ko
25    8 0xffffffff82200000   84a300 dtrace.ko
26    1 0xffffffff820c3000     2274 dtmalloc.ko
27    1 0xffffffff820c6000     3331 fbt.ko
28    1 0xffffffff820ca000    56570 fasttrap.ko
29    1 0xffffffff82121000     2258 sdt.ko
30    1 0xffffffff82124000     91b4 systrace.ko
31    1 0xffffffff8212e000     91b4 systrace_freebsd32.ko
32    1 0xffffffff82138000     234c profile.ko
33    1 0xffffffff8213b000     8b38 ipmi.ko
34    3 0xffffffff82144000     45b0 efirt.ko
35    1 0xffffffff82149000     75b0 if_bridge.ko
36    1 0xffffffff82151000     50d8 bridgestp.ko
37    1 0xffffffff82157000    1662c hwpmc.ko
38    1 0xffffffff8216e000    28bb8 tcp_rack.ko
39    1 0xffffffff82197000     21b8 mfip.ko
40    2 0xffffffff82a4b000    84470 cam.ko
41    1 0xffffffff8219a000     7d38 ioat.ko
42    1 0xffffffff821a2000    48888 if_bce.ko
43    1 0xffffffff82ad0000    17a50 miibus.ko
44    1 0xffffffff821eb000     44b0 usb_quirk.ko
45    1 0xffffffff821f0000     b3a8 usb_template.ko
46    1 0xffffffff821fc000     3268 ums.ko
47    1 0xffffffff82ae8000     92d0 xhci.ko
48    1 0xffffffff82af2000     6120 ohci.ko
49    1 0xffffffff82af9000    43ef8 nfscl.ko
50    3 0xffffffff82b3d000    18cf0 nfscommon.ko
51    3 0xffffffff82b56000     2168 nfssvc.ko
52    4 0xffffffff82b59000    138a0 krpc.ko
53    1 0xffffffff82b6d000    4e638 nfsd.ko
54    1 0xffffffff82bbc000     bdc0 nfslockd.ko
55    1 0xffffffff82bc8000     4168 ataintel.ko
56    2 0xffffffff82bcd000     8358 ata.ko
57    1 0xffffffff82bd6000     5388 atapci.ko
58    1 0xffffffff82bdc000     4d40 geom_label.ko
59    1 0xffffffff82be1000    29f58 linux64.ko
60    1 0xffffffff82c0b000     2260 pty.ko
61    1 0xffffffff82c0e000     639c linprocfs.ko
62    1 0xffffffff82c15000     3284 linsysfs.ko
63    1 0xffffffff82c19000     3378 acpi_wmi.ko
64    1 0xffffffff82c1d000     2280 uhid.ko
65    1 0xffffffff82c20000     3320 usbhid.ko
66    1 0xffffffff82c24000     31f8 hidbus.ko
67    1 0xffffffff82c28000     32c0 wmt.ko
68    1 0xffffffff82c2c000    41a38 pf.ko
69    1 0xffffffff82c6e000     2a08 mac_ntpd.ko
70    5 0xffffffff82c71000     fb28 netgraph.ko
71    1 0xffffffff82c81000     63f8 ng_netflow.ko
72    1 0xffffffff82c88000     41e8 ng_ksocket.ko
73    1 0xffffffff82c8d000     3180 ng_ether.ko
74    1 0xffffffff82c91000     3918 ng_socket.ko
75    1 0xffffffff82c95000     4708 nullfs.ko
-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640                 E-Mail: ler@lerctr.org
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106