Panic in 8.1 in softclock(): mutex Giant not owned at ../../../../sys/kern/kern_mutex.c:185

[Vijay Singh]

Hi John,

>> (kgdb-amd64-6.8-71) f 5
>> #5  0xffffffff803b73e6 in softclock (arg=0xffffffff80c90f40) at
>> ../../../../sys/kern/kern_timeout.c:461
>> 461                                           class->lc_unlock(c_lock);
>> (kgdb-amd64-6.8-71) l
>> 456                                           lastfunc = c_func;
>> 457                                   }
>> 458   #endif
>> 459                                   CTR1(KTR_CALLOUT, "callout %p finished", c);
>> 460                                   if ((c_flags & CALLOUT_RETURNUNLOCKED) == 0)
>> 461                                           class->lc_unlock(c_lock);
>> 462                           skip:
>> 463                                   CC_LOCK(cc);
>> 464                                   /*
>> 465                                    * If the current callout is locally
>> (kgdb-amd64-6.8-71) i local
>> c_func = (void (*)(void *)) 0xffffffff8060d550 <atkbd_timeout>
>
> You'll need to see if there is a way that this routine can drop the lock.  If
> you can reproduce this panic easily, then perhaps try building with KTR and
> KTR_LOCK traces enabled and use 'show ktr' in ddb to see when this thread
> dropped Giant.

An update. I haven't been able to enable KTR in my environment.
However I found some interesting.

In the code below:
(kgdb-amd64-6.8-48) l
402					if (c_lock != NULL) {
403						if (c_lock == &Giant.lock_object) {
404							mtx_lock(&Giant);
405							mtx_assert(&Giant, MA_OWNED);
406						} else {
407							class->lc_lock(c_lock, sharedlock);
408						}

I hit the assert in line 405. So right after locking Giant, it is
found unlocked!!!

Everytime this hits, I see that kldload is running on the other CPU:

(kgdb-amd64-6.8-48) show_ps_running
CPU     ticks pri thread                  tid             pid
state       VSZ       RSS cmd
  0         2  52 0xffffff000424c800   100006    11      TDS_RUNNING
     0K        0K [intr]
  1         3  76 0xffffff0004618400   100065    1481 TDS_RUNNING
3808K      760K kldload

Reporting it here to see if someone has seen this before.

-vijay