6.2 sporadically locks up

[pluknet]

2009/6/16 John Baldwin <jhb at freebsd.org>:
> On Tuesday 16 June 2009 6:23:47 am pluknet wrote:
>> Hi all.
>>
>> This is one of livelocks we have on a weekly basis.
>> Yes, we do still use ULE scheduler on 6.2 and not moved to 7 yet.
>> Any thought?
>>
>> db> ps
>>  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
>> 70304 69700 69670     0  R                           sh
>> 70303 70292 93818  3572  RL      CPU 2               chrsh
>> 70302 70294 93818  3572  R                           crond
>> 70299 93818 93818     0  R       CPU 1               crond
>> 70298 93818 93818     0  R                           crond
>> 70294 93818 93818  3572  S       piperd   0xd1d8d330 crond
>> 70292 93818 93818  3572  R                           crond
>> 70284 70279 70040 10229  S       biord    0xdbe2e4e8 perl5.8.8
>> 70283 70278 93818 10229  SL      biord    0xdbd70710 exim-4.63-0
>> 70279 70040 70040 10229  S       wait     0xc9005860 sh
>> 70278 69996 93818 10229  S       wait     0xcaf4ac90 sh
>> 70191  4680  4680  9738  S       select   0xc0a12944 httpd
>> 70190  4796  4796 10008  R                           httpd
>> 70188  5043  5043 30532  RL                          httpd
>> 70043 69999 70043  3572  Ss      select   0xc0a12944 wget
>> 70042 70000 70042  3572  Ss      select   0xc0a12944 wget
>> 70041 70001 70041  3572  Ss      select   0xc0a12944 wget
>> 70040 69996 70040 10229  Ss      piperd   0xca35e990 perl5.8.8
>> 70039 70002 70039  3572  Ss      select   0xc0a12944 wget
>
> This is not a full listing so one cannot assume it is a deadlock.

Ok, usually that listing doesn't show anything interesting in this
sort of lockup.
I'll share a full ps output next time (sure, rather soon).

>
>> db> show lockchain Giant
>> thread -3420549 (pid 434, ) ??? (0xc099cb0c)
>
> You would use 'show lock' or perhaps 'show turnstile' with specific lock
> variables.  'show lockchain' needs a TID or PID.

Ok.
As for turnstile, it showed nothing at all, hence omitted.

>
>> db> show allpcpu
>> cpuid        = 0
>> curthread    = 0xc7cfec80: pid 18 "swi4: clock sio"
>>
>> cpuid        = 1
>> curthread    = 0xc99f9960: pid 70299 "crond"
>>
>> cpuid        = 2
>> curthread    = 0xc99f9af0: pid 70303 "chrsh"
>>
>> cpuid        = 3
>> curthread    = 0xd087d320: pid 69700 "sh"
>>
>> cpuid        = 4
>> curthread    = 0xc98f84b0: pid 69604 "httpd"
>>
>> cpuid        = 5
>> curthread    = 0xcaebe190: pid 69598 "httpd"
>>
>> cpuid        = 6
>> curthread    = 0xc7cfe960: pid 27 "irq17: bce1 aacu0"
>>
>> cpuid        = 7
>> curthread    = 0xc837fe10: pid 69711 "arcconf"
>
> This is far more useful output than the truncated 'ps'.  From this, all of the
> CPUs are busy (in at least some deadlocks, all the CPUs would be idle
> instead).  There are several deadlocks fixed since 6.2 that I am aware of,
> but this doesn't look like any of those.  I'm not sure why you aren't getting
> useful stack traces of running threads.

I'll do next time. I thought it would be similar to bt PID output and
simply didn't include.

As for allpcpu, I often see the picture, when one CPU runs the "irq17:
bce1 aacu0" thread
and another one runs arcconf. I wonder if that might be a source of
bad locking or races, or..
The arcconf utility uses ioctl that goes into aac/aacu(4) internals.

> Perhaps DDB in 6.2 doesn't know to
> look in stoppcbs[].  Hmm, looks like 6.2 only does that if you are using
> KDB_STOP_NMI.  Are you using that kernel option?  If not, you probably want
> to.

No, I'm not. Will that add an additional visible overhead on a running system?

>
> --
> John Baldwin
>

Thank you.

-- 
wbr,
pluknet