Re: My -CURRENT crashes....

On Fri, Dec 17, 2021 at 01:27:11PM -0600, Larry Rosenman wrote:
L> Can someone look at the messages I posted to -CURRENT, most recent 
L> today, with random
L> Callout(?) crashes after long (>6 hour) poudriere runs?
L> 
L> I have core's available.

I asked Larry to obtain a core with INVARIANTS and now we have one.

Sharing what I've found to brainstorm. Trap happens in LIST_REMOVE()
kern_timeout.c:488 because the entry doesn't have a prev pointer, e.g.
doesn't belong to any list.

#6  0xffffffff807be075 in trap_pfault (frame=0xfffffe02d3393d50, usermode=false, signo=<optimized out>, ucode=<optimized out>)
    at /usr/src/sys/amd64/amd64/trap.c:765
#7  <signal handler called>
#8  0xffffffff804e5609 in callout_process (now=now@entry=100465191785818) at /usr/src/sys/kern/kern_timeout.c:488
#9  0xffffffff80460fc5 in handleevents (now=now@entry=100465191785818, fake=fake@entry=0) at /usr/src/sys/kern/kern_clocksource.c:213
#10 0xffffffff80461a66 in timercb (et=0xffffffff80d47980 <lapic_et>, arg=<optimized out>) at /usr/src/sys/kern/kern_clocksource.c:357
#11 0xffffffff807e6beb in lapic_handle_timer (frame=0xfffffe02d3393f40) at /usr/src/sys/x86/x86/local_apic.c:1364

(kgdb) p *tmp
$13 = {c_links = {le = {le_next = 0x0, le_prev = 0x0}, sle = {sle_next = 0x0}, tqe = {tqe_next = 0x0, tqe_prev = 0x0}}, c_time = 0, 
  c_precision = 0, c_arg = 0x0, c_func = 0x0, c_lock = 0xfffff8030521e670, c_flags = 0, c_iflags = 0, c_cpu = 0}

Useful here is the c_lock, which points into "process lock" lockobject.

This allows us to deduct that the callout belongs to proc subsystem and
we can retrieve the proc it points to: c_lock - 0x128 = 0xfffff8030521e548
It is ccache in PRS_NORMAL state. And the "tmp" in our stack frame is its
p_itcallout.

So there is something that would zero out most of the p_itcallout while
it is scheduled?

-- 
Gleb Smirnoff