How best to debug locking/scheduler problems

[John Baldwin]

On Monday 15 June 2009 5:53:05 pm Mel Flynn wrote:
> Hi,
> 
> I'm trying to get to the bottom of a bug with getpeername() and certain kde4 
> applications which is probably as low-level as the libthr and the scheduler.
> 
> From browsing various related files in sys/kern it seems KTR is a good bet 
to 
> get the information needed, yet it isn't really well supported in userland. 
> For one, I've got no clue other then logging console output(?) how to 
retrieve 
> the lock info or filter it in userland from reading ktr(9) and alq(9). Gdb 
is 
> useless as the process doesn't give the information gdb wants and gdb just 
> hangs in wait. ktrace also does not provide anything as there are no more 
> syscalls being made, so I'll have to get to the bottom of this by tracing 
and 
> filtering.
> 
> Short description of the problem:
> a process never gets out of mi_switch and remains locked even init tries to 
> shut it down.
> 
> % procstat -t 4283
> 
>   PID    TID COMM             TDNAME           CPU  PRI STATE   WCHAN    
>  4283 100215 kdeinit4         -                  0  128 lock    *unp_mtx  
> % procstat -k 4283
> 
>   PID    TID COMM             TDNAME           KSTACK                       
>  4283 100215 kdeinit4         -                mi_switch turnstile_wait 
> _mtx_lock_sleep uipc_peeraddr kern_getpeername getpeername syscall 
> Xint0x80_syscall 
> % ps -ww 4283
>   PID  TT  STAT      TIME COMMAND
>  4283  ??  T      0:00.38 kdeinit4: kdeinit4: kio_http http 
> local:/tmp/ksocket-mel/klauncherxJ1635.slave-socket local:/tmp/ksocket-
> mel/plasmayC1653.slave-socket (kdeinit4)
> 
> %ls -l /tmp/ksocket-mel/
> 
> total 2
> -rw-rw-r--  1 mel  wheel  62 Jun 14 22:55 KSMserver__0
> srw-------  1 mel  wheel   0 Jun 14 22:55 kdeinit4__0
> srwxrwxr-x  1 mel  wheel   0 Jun 14 22:55 klauncherxJ1635.slave-socket

You can use kgdb and the scripts at www.freebsd.org/~jhb/gdb.  Simply 
run 'kgdb' as root and do 'lcd /folder/with/scripts' and 'source gdb6'.  You 
can then do 'lockchain 4283' to find who holds the lock this thread is 
blocked on and what state they are in.  You can use 'show lockchain 4283' in 
DDB for similar info as well.

-- 
John Baldwin