Proposed 6.2 em RELEASE patch
Mike Tancsa
mike at sentex.net
Mon Nov 13 00:40:59 UTC 2006
At 11:41 AM 11/12/2006, Scott Long wrote:
>Mike Tancsa wrote:
>>At 01:42 AM 11/11/2006, Scott Long wrote:
>>driver. What will help me is if you can hook up a serial console to
>>>your machine and see if it can be made to drop to the debugger while it
>>>is under load and otherwise unresponsive. If you can, getting a process
>>>dump might help confirm where each CPU is spending its time.
>>
>>./netblast 192.168.88.218 500 110 1000
>>I compiled in the various debugging options and on the serial
>>console I get a few
>>Expensive timeout(9) function: 0xc0601e48(0) 0.024135749 s
>>and the serial console
>
>One CPU seems to be stuck in softclock. My guess here is that there
>is significant lock contention. Please try the following:
>
>1. Use addr2line or gdb on the kernel to find out what function is at
>0xc0601e48 (the address that was printed above). This address will
>change every time you recompile the kernel, so you'll need to reacquire
>it from the console each time.
# addr2line 0xc0601e48 -e kernel.debug -f
nfsrv_timer
/usr/src/sys/nfsserver/nfs_srvsock.c:809
#
and
# addr2line 0xc0561444 -e kernel.debug -f
sleepq_timeout
/usr/src/sys/kern/subr_sleepqueue.c:724
I dont have any nfs activity on the box
>2. Try compiling in WITNESS and running the test as before, then break
>into the debugger as before. Run 'show locks'. I'm not sure how
>fruitful this will be, WITNESS might make it unbearably slow.
It was in that kernel already
All it would generally show is
exclusive spin mutex sio r = 0 (0xc07c81c0) locked @
/usr/src/sys/dev/sio/sio.c:1390
But I did notice in /var/log/kernel
Nov 12 00:04:09 R2 kernel: exclusive spin mutex sio r = 0
(0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390
Nov 12 00:04:09 R2 kernel: exclusive sleep mutex em1 (network driver)
r = 0 (0xc64bc9d4) locked @ /usr/src/sys/dev/ em/if_em.c:3569
Nov 12 00:04:09 R2 kernel: exclusive spin mutex sio r = 0
(0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390
Nov 12 19:27:39 R2 kernel: KDB: enter: Line break on console
Nov 12 19:27:39 R2 kernel: exclusive spin mutex sio r = 0
(0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390
Nov 12 19:28:13 R2 kernel: KDB: enter: Line break on console
Nov 12 19:28:13 R2 kernel: exclusive spin mutex sio r = 0
(0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390
Nov 12 19:31:37 R2 kernel: KDB: enter: Line break on console
Nov 12 19:31:37 R2 kernel: exclusive spin mutex sio r = 0
(0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390
Nov 12 19:34:22 R2 kernel: KDB: enter: Line break on console
Nov 12 19:34:22 R2 kernel: exclusive spin mutex sio r = 0
(0xc07c81c0) locked @ /usr/src/sys/dev/sio/sio.c:1390
>3. Turn on MUTEX_PROFILING according to the text in /sys/conf/NOTES and
>the man page by the same name.
>4. Remove ADAPTIVE_GIANT from your config and retest. If that doesn't
>show a difference, then try setting NO_ADAPTIVE_MUTEXES.
OK, did this recompiling without WITNESS and INVARIANTS and took out
ADAPTIVE_GIANT as instructed in the man pages
# sysctl -a | grep mutex
debug.mutex.prof.enable: 1
debug.mutex.prof.acquisitions: 584672946
debug.mutex.prof.records: 450
debug.mutex.prof.maxrecords: 1000
debug.mutex.prof.rejected: 0
debug.mutex.prof.hashsize: 1009
debug.mutex.prof.collisions: 75
debug.mutex.prof.stats:
10 1263 226 5 0 0
/usr/src/sys/kern/sys_pipe.c:1345 (pipe mutex)
3 682 369 1 13 6908
/usr/src/sys/vm/vm_fault.c:851 (vm page queue mutex)
10 2256 623 3 23 15
/usr/src/sys/i386/i386/pmap.c:1891 (vm page queue mutex)
120 883 401 2 15 26
/usr/src/sys/vm/vm_fault.c:909 (vm page queue mutex)
25 25 1 25 0 0
/usr/src/sys/i386/i386/pmap.c:2572 (vm page queue mutex)
19 169 33 5 1 1
/usr/src/sys/vm/vm_object.c:1801 (vm page queue mutex)
4 24 9 2 0 0
/usr/src/sys/vm/vm_object.c:646 (vm page queue mutex)
3 23 8 2 1 1
/usr/src/sys/kern/vfs_bio.c:3384 (vm page queue mutex)
2 12 6 2 0 0
/usr/src/sys/vm/vm_pageout.c:1511 (vm page queue mutex)
4 120 44 2 0 0
/usr/src/sys/kern/vfs_bio.c:3315 (vm page queue mutex)
3 100 44 2 0 0
/usr/src/sys/kern/vfs_bio.c:3120 (vm page queue mutex)
3 9 4 2 0 0
/usr/src/sys/kern/vfs_bio.c:3577 (vm page queue mutex)
1 1 1 1 0 4
/usr/src/sys/kern/sys_pipe.c:1270 (pipe mutex)
2 16 8 2 0 0
/usr/src/sys/vm/vm_page.c:1481 (vm page queue mutex)
2 8 4 2 0 0
/usr/src/sys/kern/kern_exec.c:871 (vm page queue mutex)
145 713 14 50 5 4
/usr/src/sys/vm/vm_map.c:1504 (vm page queue mutex)
2 7 4 1 0 0
/usr/src/sys/vm/vm_glue.c:273 (vm page queue mutex)
2 8 4 2 0 0
/usr/src/sys/vm/vm_glue.c:309 (vm page queue mutex)
2 8 4 2 0 0
/usr/src/sys/kern/kern_exec.c:893 (vm page queue mutex)
36 357 42 8 2 1
/usr/src/sys/i386/i386/pmap.c:1624 (vm page queue mutex)
5 152 64 2 2 2
/usr/src/sys/vm/vm_fault.c:346 (vm page queue mutex)
2 61 24 2 2 1
/usr/src/sys/vm/vm_fault.c:136 (vm page queue mutex)
5 48 10 4 0 2
/usr/src/sys/i386/i386/pmap.c:1782 (vm page queue mutex)
11 1451 193 7 24 13
/usr/src/sys/vm/vm_fault.c:1009 (vm page queue mutex)
3 3868 1692 2 8 4
/usr/src/sys/kern/sys_pipe.c:993 (pipe mutex)
6 4681 1692 2 46 6
/usr/src/sys/kern/sys_pipe.c:1145 (pipe mutex)
2 643 372 1 0 0
/usr/src/sys/vm/vm_fault.c:1092 (vm page queue mutex)
3 3235 1648 1 5 44
/usr/src/sys/kern/sys_pipe.c:591 (pipe mutex)
5 4235 1647 2 2 2
/usr/src/sys/kern/sys_pipe.c:628 (pipe mutex)
1 331 222 1 0 0
/usr/src/sys/vm/vm_kern.c:367 (vm page queue mutex)
2 337 222 1 0 0
/usr/src/sys/vm/vm_kern.c:404 (vm page queue mutex)
debug.mutex.prof.reset: 0
>Thanks,
>
>Scott
More information about the freebsd-stable
mailing list