2 x quad-core system is slower that 2 x dual core on FreeBSD
Alexey Popov
lol at chistydom.ru
Tue Nov 20 00:36:42 PST 2007
Hi.
Kris Kennaway wrote:
> OK, but that is on 6.x and only (I guess) on the 8 core machine. What
> is needed is at least a comparison of the two machines under identical
> conditions, and preferably on 7.0.
Right, that was 8-core on 6-STABLE.
> It looks like the most serious issue might be due to lockmgr contention,
> but on 6.x lockmgr usage is not profiled directly (it only shows up via
> secondary mutexes). Comparing two 7.0 traces should help to shed more
> light on the subject.
At this moment I have no 4-core servers running FreeBSD 7. Take a look
at lock profiling of 8-core server on 7-STABLE in attach. If there's no
evidence of the problem I'll try 7-STABLE on the production 4-core server.
Also could you explain what to look for in the lock profiling results?
Does large "wait_total" values indicate problem or other columns???
With best regards,
Alexey Popov
-------------- next part --------------
debug.lock.prof.stats:
max total wait_total count avg wait_avg cnt_hold cnt_lock name
10 56 0 7 8 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:512 (rw:unp_global_rwlock)
18 165 0 17 9 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:1099 (sleep mutex:udp)
73 579 0 16 36 0 0 0 /usr/src/sys/geom/geom_disk.c:198 (sleep mutex:g_disk_done)
79 133 0 2 66 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:530 (rw:unp_global_rwlock)
29 100 0 5 20 0 0 0 /usr/src/sys/vm/vm_object.c:1298 (sleep mutex:vm object)
25 72 0 5 14 0 0 0 /usr/src/sys/vm/vm_object.c:1299 (sleep mutex:vm object)
27 12013 0 4251 2 0 3 0 /usr/src/sys/net/if_ethersubr.c:405 (sleep mutex:em0)
25 253 0 16 15 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:1113 (sleep mutex:udp)
3215 250635 0 933 268 0 0 0 /usr/src/sys/vm/vm_map.c:2293 (sleep mutex:vm object)
2 5 0 3 1 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:2048)
8 43 0 7 6 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:557 (rw:unp_global_rwlock)
117 415 0 10 41 0 0 0 /usr/src/sys/kern/kern_exit.c:428 (sx:proctree)
813 21953 0 1551 14 0 0 0 /usr/src/sys/kern/subr_sleepqueue.c:390 (sleep mutex:process lock)
12 21 0 2 10 0 0 0 /usr/src/sys/kern/vfs_bio.c:1738 (sleep mutex:buf queue lock)
5 13 0 3 4 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:MAP ENTRY)
140 140 0 1 140 0 0 0 /usr/src/sys/vm/vm_kern.c:296 (sleep mutex:system map)
4 14 0 5 2 0 0 0 /usr/src/sys/vm/vm_object.c:1378 (sleep mutex:vm object)
44 112 0 3 37 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:622 (rw:unp_global_rwlock)
560 730 0 21 34 0 0 0 /usr/src/sys/kern/kern_descrip.c:665 (sx:filedesc structure)
4 34 0 14 2 0 0 0 /usr/src/sys/kern/kern_exec.c:854 (sleep mutex:vm page queue mutex)
18 3477 0 1242 2 0 0 0 /usr/src/sys/kern/uipc_socket.c:1136 (sleep mutex:so_snd)
11 419 0 60 6 0 0 0 /usr/src/sys/kern/vfs_bio.c:2107 (sleep mutex:buffer daemon lock)
5 39 0 14 2 0 0 0 /usr/src/sys/kern/kern_exec.c:876 (sleep mutex:vm page queue mutex)
7 108 0 23 4 0 0 0 /usr/src/sys/vm/vm_fault.c:649 (sleep mutex:vm object)
5 105 0 36 2 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:827 (sleep mutex:so_rcv)
1 1 0 1 1 0 0 0 /usr/src/sys/vm/uma_core.c:889 (sleep mutex:128 Bucket)
118 8011 0 423 18 0 0 0 /usr/src/sys/vm/vm_fault.c:667 (sleep mutex:vm object)
3 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf_jumbo_pagesize)
3 7 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UPCALL)
56 56 0 1 56 0 0 0 /usr/src/sys/fs/fifofs/fifo_vnops.c:716 (sleep mutex:Giant)
55 343 0 8 42 0 0 0 /usr/src/sys/kern/kern_exec.c:494 (lockmgr:ufs)
5 72 0 27 2 0 0 0 /usr/src/sys/netinet/ip_input.c:1218 (sleep mutex:rtentry)
127 1467 0 16 91 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:843 (sleep mutex:inp)
81 81 0 1 81 0 0 0 /usr/src/sys/fs/fifofs/fifo_vnops.c:738 (sleep mutex:Giant)
5 22 0 7 3 0 0 0 /usr/src/sys/kern/uipc_socket.c:2230 (sleep mutex:so_rcv)
11 1283 0 485 2 0 0 0 /usr/src/sys/vm/vm_mmap.c:1318 (sleep mutex:process lock)
15 123 0 12 10 0 0 0 /usr/src/sys/kern/vfs_bio.c:3426 (sleep mutex:vm object)
3 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ACL UMA zone)
17 1838 0 260 7 0 0 0 /usr/src/sys/vm/vm_map.c:2499 (sleep mutex:vm object)
5 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:L VFS Cache)
3 4 0 2 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:260 (sleep mutex:process lock)
11 1068 0 379 2 0 0 0 /usr/src/sys/vm/vm_fault.c:770 (sleep mutex:vm object)
26 49269 52123 14029 3 3 5404 6646 /usr/src/sys/kern/kern_intr.c:706 (spin mutex:sched lock)
25 417 0 38 10 0 0 0 /usr/src/sys/kern/vfs_bio.c:3498 (sleep mutex:vm object)
5 58 0 18 3 0 0 0 /usr/src/sys/kern/kern_umtx.c:2703 (spin mutex:umtx lock)
3 6 0 2 3 0 0 0 /usr/src/sys/kern/vfs_cluster.c:910 (sleep mutex:vm object)
5 50 0 17 2 0 0 0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:tcpcb)
5 48 0 16 3 0 0 0 /usr/src/sys/kern/kern_event.c:758 (sleep mutex:protect sysfilt_ops)
5 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UMA Kegs)
20 124 0 11 11 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:1421 (sleep mutex:tcp)
389 16941 128 80 211 1 1 1 /usr/src/sys/kern/kern_exit.c:684 (sx:proctree)
10 43 0 16 2 0 0 0 /usr/src/sys/kern/kern_event.c:774 (sleep mutex:protect sysfilt_ops)
5 470 0 193 2 0 0 0 /usr/src/sys/kern/vfs_vnops.c:508 (sleep mutex:sleep mtxpool)
3 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:PROC)
3 8 0 4 2 0 0 0 /usr/src/sys/kern/tty.c:850 (sleep mutex:process lock)
1024 1484197 92173 279122 5 0 3689 5478 /usr/src/sys/kern/vfs_cache.c:327 (sleep mutex:Name Cache)
6 513 0 193 2 0 0 0 /usr/src/sys/kern/vfs_vnops.c:529 (sleep mutex:sleep mtxpool)
3072 8093 0 137 59 0 0 0 /usr/src/sys/vm/vm_object.c:1627 (sleep mutex:vm object)
12 8938 162 3618 2 0 1 2 /usr/src/sys/kern/vfs_subr.c:2946 (sleep mutex:vnode interlock)
3 20 0 8 2 0 0 0 /usr/src/sys/kern/imgact_elf.c:763 (sleep mutex:process lock)
5 110 0 47 2 0 0 0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:so_rcv)
5 5 0 1 5 0 0 0 /usr/src/sys/kern/kern_descrip.c:385 (sleep mutex:process lock)
1537 421522 80 47192 8 0 2 3 /usr/src/sys/vm/vm_fault.c:886 (sleep mutex:vm object)
2 4 0 2 2 0 0 0 /usr/src/sys/vm/vm_map.c:2631 (sleep mutex:vm object)
4 13 0 4 3 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:902 (sleep mutex:pmap)
3 7 0 3 2 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:6023 (sleep mutex:Softdep Lock)
31 207 0 10 20 0 0 0 /usr/src/sys/kern/kern_exit.c:519 (spin mutex:process slock)
1125 1304 0 6 217 0 0 0 /usr/src/sys/ufs/ffs/ffs_vnops.c:214 (sleep mutex:vnode interlock)
41 267 0 10 26 0 0 0 /usr/src/sys/kern/kern_exit.c:522 (spin mutex:process slock)
5 46 0 17 2 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:1053 (sleep mutex:inp)
2 7 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:TURNSTILE)
5 11 0 3 3 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:722 (sleep mutex:so_rcv)
2 2 0 1 2 0 0 0 /usr/src/sys/vm/vm_pager.c:336 (sleep mutex:pbuf mutex)
2 5 0 3 1 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:pipe)
82 985 0 16 61 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:1073 (sleep mutex:inp)
2 7 0 4 1 0 0 0 /usr/src/sys/vm/vm_object.c:1695 (sleep mutex:vm object)
7 7 0 1 7 0 0 0 /usr/src/sys/ufs/ffs/ffs_vnops.c:240 (sleep mutex:vnode interlock)
31 53 0 2 26 0 0 0 /usr/src/sys/kern/vfs_bio.c:3641 (sleep mutex:vm object)
31 142 0 8 17 0 0 0 /usr/src/sys/kern/kern_exec.c:749 (lockmgr:ufs)
3 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:umtx pi)
890 16000 257 2778 5 0 16 19 /usr/src/sys/kern/uipc_socket.c:2469 (sleep mutex:so_rcv)
3 36 0 16 2 0 0 0 /usr/src/sys/kern/kern_event.c:1495 (sx:filedesc structure)
5 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mt_zone)
12 8840 281 3618 2 0 1 2 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1226 (sleep mutex:vnode interlock)
18 18 0 1 18 0 0 0 /usr/src/sys/kern/kern_resource.c:186 (sleep mutex:process lock)
14 71 0 17 4 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:1100 (sleep mutex:inp)
7337 46187 293 125 369 2 15 11 /usr/src/sys/kern/kern_synch.c:235 (sleep mutex:Giant)
5 91 0 30 3 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:96 (sleep mutex:so_snd)
788 157277 374 48292 3 0 15 17 /usr/src/sys/vm/vm_fault.c:959 (sleep mutex:vm object)
20 165 0 16 10 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:1114 (sleep mutex:inp)
17 1073 0 172 6 0 0 0 /usr/src/sys/vm/vm_fault.c:966 (sleep mutex:vm object)
199 382 0 2 191 0 0 0 /usr/src/sys/kern/vfs_bio.c:3680 (sleep mutex:vm object)
11 44 0 10 4 0 0 0 /usr/src/sys/kern/kern_sig.c:3289 (sleep mutex:sigacts)
16 32 0 7 4 0 0 0 /usr/src/sys/ufs/ffs/ffs_vnops.c:289 (sleep mutex:vnode interlock)
36 2034 0 545 3 0 0 0 /usr/src/sys/kern/kern_timeout.c:241 (sleep mutex:tcp_sc_head)
3 18 0 8 2 0 0 0 /usr/src/sys/kern/kern_sig.c:3299 (sleep mutex:sigacts)
5 5 0 1 5 0 0 0 /usr/src/sys/vm/vm_pager.c:413 (sleep mutex:pbuf mutex)
1770 67722 0 25037 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:1025 (sx:filedesc structure)
4 35 0 12 2 0 0 0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1269 (sleep mutex:vnode interlock)
18 143 0 16 8 0 0 0 /usr/src/sys/kern/uipc_socket.c:2528 (sleep mutex:so_rcv)
31 31 0 1 31 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:810 (sleep mutex:so_rcv)
11 76 59 10 7 5 5 8 /usr/src/sys/kern/kern_thread.c:467 (spin mutex:sched lock)
5 59 0 19 3 0 0 0 /usr/src/sys/kern/kern_prot.c:1874 (sleep mutex:process lock)
17 698 40 215 3 0 2 5 /usr/src/sys/kern/vfs_cache.c:500 (sleep mutex:Name Cache)
1933 898106 9292 157103 5 0 767 273 /usr/src/sys/kern/vfs_vnops.c:807 (sleep mutex:vnode interlock)
1 1 0 1 1 0 0 0 /usr/src/sys/kern/kern_prot.c:1933 (sleep mutex:session)
78 4796 0 1585 3 0 0 0 /usr/src/sys/kern/sys_generic.c:893 (sleep mutex:process lock)
3 21 35 10 2 3 4 5 /usr/src/sys/kern/kern_thread.c:501 (spin mutex:sched lock)
22 17432 0 6393 2 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2282 (rw:ipf cache rwlock)
5 17 0 6 2 0 0 0 /usr/src/sys/netinet/in_pcb.c:789 (sleep mutex:inp)
138 10295 2002 1718 5 1 43 87 /usr/src/sys/netinet/tcp_input.c:1191 (sleep mutex:so_rcv)
3 7 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:socket)
3 8 0 3 2 0 0 0 /usr/src/sys/kern/vfs_syscalls.c:3951 (sleep mutex:sleep mtxpool)
37 44 0 3 14 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:873 (sleep mutex:so_rcv)
4 15 0 4 3 0 0 0 /usr/src/sys/netinet/in_pcb.c:808 (sleep mutex:inp)
6 6 0 1 6 0 0 0 /usr/src/sys/kern/kern_prot.c:1932 (sleep mutex:process lock)
5 13 0 4 3 0 0 0 /usr/src/sys/vm/uma_core.c:889 (sleep mutex:MAP ENTRY)
13 102 0 12 8 0 0 0 /usr/src/sys/kern/vfs_subr.c:3162 (sleep mutex:vnode interlock)
11 357 0 120 2 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/ip_frag.c:799 (rw:ipf IP NAT-Frag rwlock)
23 3800 0 1448 2 0 0 0 /usr/src/sys/kern/kern_exit.c:712 (spin mutex:process slock)
106 15879 0 530 29 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:231 (sleep mutex:so_snd)
9 136 1190 24 5 49 0 2 /usr/src/sys/sys/buf.h:276 (sleep mutex:lockbuilder mtxpool)
3 17 0 7 2 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:3742 (sleep mutex:process lock)
10 211 0 38 5 0 0 0 /usr/src/sys/kern/subr_eventhandler.c:212 (sleep mutex:eventhandler)
5 34 0 12 2 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:6244 (sleep mutex:Softdep Lock)
36 2220 0 699 3 0 0 0 /usr/src/sys/vm/vm_object.c:1884 (sleep mutex:vm object)
3275 294404 0 414 711 0 0 0 /usr/src/sys/vm/vm_mmap.c:557 (sx:user map)
38 3642 18 449 8 0 0 1 /usr/src/sys/sys/buf.h:296 (sleep mutex:lockbuilder mtxpool)
11 370 0 134 2 0 0 0 /usr/src/sys/kern/sys_socket.c:121 (sleep mutex:so_rcv)
11 365 0 135 2 0 0 0 /usr/src/sys/kern/sys_socket.c:147 (sleep mutex:so_rcv)
12 2447 0 721 3 0 0 0 /usr/src/sys/netinet/tcp_timewait.c:643 (sleep mutex:inp)
11 362 0 135 2 0 0 0 /usr/src/sys/kern/sys_socket.c:150 (sleep mutex:so_rcv)
10 106 0 16 6 0 0 0 /usr/src/sys/kern/uipc_socket.c:2680 (sleep mutex:so_rcv)
5 149 0 49 3 0 0 0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:mbuf)
3 45 0 21 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:659 (sleep mutex:process lock)
5 11 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:16 Bucket)
4 20 0 7 2 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:3815 (sleep mutex:process lock)
166 66254 0 972 68 0 0 0 /usr/src/sys/netinet/tcp_timer.c:169 (sleep mutex:inp)
41 41 0 1 41 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:1178 (rw:unp_global_rwlock)
11 6503 29 2472 2 0 0 2 /usr/src/sys/kern/kern_resource.c:1240 (sleep mutex:sleep mtxpool)
22 155 0 11 14 0 0 0 /usr/src/sys/netinet/tcp_subr.c:1379 (sleep mutex:isn_mtx)
4 11 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:64)
4 26 0 10 2 0 0 0 /usr/src/sys/kern/kern_exit.c:215 (sleep mutex:p_peers)
26 54225 87944 13349 4 6 7262 8564 /usr/src/sys/kern/kern_intr.c:1131 (spin mutex:sched lock)
12 6364 19 2522 2 0 1 1 /usr/src/sys/kern/kern_resource.c:1266 (sleep mutex:sleep mtxpool)
1054 13021 0 24 542 0 0 0 /usr/src/sys/sys/buf.h:280 (lockmgr:bufwait)
6 6 0 1 6 0 0 0 /usr/src/sys/kern/kern_resource.c:1277 (sleep mutex:sleep mtxpool)
12 159 0 62 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:1544 (sleep mutex:sleep mtxpool)
865 18644 0 449 41 0 0 0 /usr/src/sys/sys/buf.h:301 (lockmgr:getblk)
40 50477 0 6001 8 0 0 0 /usr/src/sys/netinet/tcp_subr.c:1426 (sleep mutex:isn_mtx)
4 72 0 24 3 0 0 0 /usr/src/sys/kern/kern_exec.c:903 (sleep mutex:process_exec)
1 2 0 2 1 0 0 0 /usr/src/sys/kern/vfs_syscalls.c:3864 (sx:filedesc structure)
4 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:KNOTE)
4 53 0 22 2 0 0 0 /usr/src/sys/kern/kern_resource.c:1314 (sleep mutex:sleep mtxpool)
9 36 0 10 3 0 0 0 /usr/src/sys/kern/sysv_sem.c:1288 (sleep mutex:sem)
5 30 0 10 3 0 0 0 /usr/src/sys/kern/kern_exit.c:284 (sleep mutex:p_peers)
12 63 0 7 9 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:1268 (rw:unp_global_rwlock)
4 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:512)
33 4035 16 1516 2 0 3 1 /usr/src/sys/netinet/tcp_subr.c:1606 (sleep mutex:rtentry)
78 7551 0 2794 2 0 0 0 /usr/src/sys/kern/kern_resource.c:1339 (sleep mutex:sleep mtxpool)
5 5 0 1 5 0 0 0 /usr/src/sys/vm/uma_core.c:2195 (sleep mutex:UMA Slabs)
18 25 0 3 8 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:NAMEI)
4 30 0 11 2 0 0 0 /usr/src/sys/kern/uipc_socket.c:2820 (sleep mutex:so_rcv)
452 39938 9430 14524 2 0 124 424 /usr/src/sys/kern/vfs_cache.c:774 (sleep mutex:Name Cache)
20 4204 0 1021 4 0 0 0 /usr/src/sys/kern/uipc_socket.c:2832 (sleep mutex:so_rcv)
1339 50216 129 8527 5 0 6 6 /usr/src/sys/net/route.c:147 (sleep mutex:radix node head)
61 1403 0 494 2 0 0 0 /usr/src/sys/kern/uipc_socket.c:2846 (sleep mutex:so_rcv)
4 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:KMAP ENTRY)
5 323 0 120 2 0 0 0 /usr/src/sys/netinet/igmp.c:446 (sleep mutex:igmp_mtx)
30 194 0 33 5 0 1 0 /usr/src/sys/kern/uipc_socket.c:2863 (sleep mutex:so_rcv)
416 55229737 13245866 1108825 49 11 957433 991447 /usr/src/sys/kern/subr_sleepqueue.c:232 (spin mutex:sleepq chain)
5 108 0 36 3 0 0 0 /usr/src/sys/kern/uipc_socket.c:2875 (sleep mutex:so_rcv)
77 294 0 5 58 0 0 0 /usr/src/sys/kern/vfs_lookup.c:411 (lockmgr:devfs)
5 103 0 38 2 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:972 (sleep mutex:so_rcv)
910 168337 0 15207 11 0 0 0 /usr/src/sys/kern/kern_descrip.c:1394 (sx:filedesc structure)
12 176 5 59 2 0 1 2 /usr/src/sys/kern/uipc_socket.c:2894 (sleep mutex:so_rcv)
5 5 0 1 5 0 0 0 /usr/src/sys/kern/vfs_cluster.c:753 (sleep mutex:vnode interlock)
3 20 0 10 2 0 0 0 /usr/src/sys/kern/kern_exit.c:470 (sleep mutex:sigacts)
4 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UMA Zones)
1 3 0 2 1 0 0 0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1656 (sleep mutex:vnode interlock)
157 33577 0 13815 2 0 0 0 /usr/src/sys/vm/vnode_pager.c:110 (sleep mutex:vm object)
4 19 0 7 2 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:513 (sleep mutex:unp_mtx)
17 157 0 18 8 0 0 0 /usr/src/sys/kern/kern_resource.c:1206 (sleep mutex:uidinfo hash)
29 41876 29826 11242 3 2 3898 3653 /usr/src/sys/kern/kern_switch.c:182 (spin mutex:sched lock)
7 7 0 1 7 0 0 0 /usr/src/sys/kern/kern_resource.c:1211 (sleep mutex:uidinfo hash)
38 23181 49 6027 3 0 0 2 /usr/src/sys/dev/em/if_em.c:4305 (sleep mutex:em0)
19 132 0 10 13 0 0 0 /usr/src/sys/kern/kern_descrip.c:1429 (sx:filedesc structure)
22 9489 14390 3404 2 4 11 305 /usr/src/sys/kern/sched_4bsd.c:385 (spin mutex:sleepq chain)
576 4177 0 1236 3 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:531 (sleep mutex:so_snd)
3 5 0 2 2 0 0 0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1678 (sleep mutex:vnode interlock)
74 122 0 2 61 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:533 (sleep mutex:unp_mtx)
68 111 0 2 55 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:536 (sleep mutex:unp_mtx)
4 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:SWAPMETA)
8 208 6 64 3 0 0 2 /usr/src/sys/kern/vfs_bio.c:3002 (sleep mutex:bdone lock)
5 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:32 Bucket)
6 422 0 120 3 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/ip_auth.c:563 (rw:ipf IP User-Auth rwlock)
4 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:DIRHASH)
118 265 0 5 53 0 0 0 /usr/src/sys/netinet/tcp_timer.c:438 (sleep mutex:inp)
11 33 0 11 3 0 0 0 /usr/src/sys/kern/kern_resource.c:677 (sleep mutex:process lock)
4 23 0 9 2 0 0 0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1729 (sleep mutex:vnode interlock)
8 8 0 1 8 0 0 0 /usr/src/sys/kern/vfs_cluster.c:841 (sleep mutex:vnode interlock)
4 4 0 1 4 0 0 0 /usr/src/sys/kern/kern_resource.c:1276 (sleep mutex:uidinfo hash)
2 9 0 4 2 0 0 0 /usr/src/sys/kern/vfs_bio.c:1320 (sleep mutex:vm page queue mutex)
3 20 0 8 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:1502 (sx:filedesc structure)
22 381 0 66 5 0 0 0 /usr/src/sys/kern/kern_sysctl.c:1396 (sx:sysctl lock)
20 87 0 8 10 0 0 0 /usr/src/sys/kern/kern_time.c:611 (sleep mutex:process lock)
10 399 0 158 2 0 0 0 /usr/src/sys/vm/vnode_pager.c:208 (sleep mutex:vm object)
22 19817 8844 7311 2 1 1444 783 /usr/src/sys/kern/kern_switch.c:272 (spin mutex:sched lock)
3533 34520 0 24 1438 0 10 0 /usr/src/sys/kern/vfs_mount.c:2029 (sleep mutex:struct mount mtx)
13 5235 8321 1872 2 4 425 901 /usr/src/sys/kern/subr_turnstile.c:203 (spin mutex:sched lock)
21 1435 0 530 2 0 0 0 /usr/src/sys/kern/uipc_socket.c:280 (sleep mutex:so_glabel)
40 95 0 3 31 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:623 (sleep mutex:unp_mtx)
35 79 0 3 26 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:626 (sleep mutex:unp_mtx)
5 7 0 2 3 0 0 0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1773 (sleep mutex:vnode interlock)
5 1008 0 352 2 0 0 0 /usr/src/sys/kern/uipc_socket.c:299 (sleep mutex:so_glabel)
4 27 0 10 2 0 0 0 /usr/src/sys/kern/kern_exit.c:409 (sx:allproc)
108 38884 714 1551 25 0 59 26 /usr/src/sys/kern/subr_sleepqueue.c:409 (spin mutex:sleepq chain)
6 31 0 10 3 0 0 0 /usr/src/sys/kern/kern_descrip.c:1552 (sx:filedesc structure)
17 80 48 24 3 2 1 4 /usr/src/sys/kern/vfs_mount.c:2067 (sleep mutex:struct mount mtx)
74 3029 0 63 48 0 0 0 /usr/src/sys/kern/kern_time.c:657 (sleep mutex:process lock)
1 1 0 1 1 0 0 0 /usr/src/sys/vm/uma_core.c:2436 (sleep mutex:UMA Slabs)
3 3 0 1 3 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:1332 (sleep mutex:so_rcv)
4 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UMA RCntSlabs)
17 1892 0 739 2 0 0 0 /usr/src/sys/kern/vfs_subr.c:2985 (sleep mutex:vnode_free_list)
4 12 0 3 4 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ata_composite)
4 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:Files)
12 1911 0 739 2 0 0 0 /usr/src/sys/kern/vfs_subr.c:3013 (sleep mutex:vnode_free_list)
5 26 0 12 2 0 0 0 /usr/src/sys/kern/kern_resource.c:796 (sleep mutex:process lock)
5 22 0 7 3 0 0 0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:inpcb)
2 7 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:VMSPACE)
22 2859 0 1071 2 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:291 (sleep mutex:process lock)
16 20 0 3 6 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:726 (sleep mutex:unp_mtx)
67 967 0 35 27 0 0 0 /usr/src/sys/kern/imgact_elf.c:322 (sx:user map)
5 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:16)
7 145 0 45 3 0 0 0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:mbuf)
4 29 0 10 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:1656 (sx:filedesc structure)
6 108 0 35 3 0 0 0 /usr/src/sys/vm/vnode_pager.c:361 (sleep mutex:vm object)
14 93 0 10 9 0 0 0 /usr/src/sys/kern/kern_descrip.c:1670 (sx:filedesc structure)
5 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UMA Hash)
25 25 0 1 25 0 0 0 /usr/src/sys/kern/vfs_bio.c:1531 (sleep mutex:vm page queue mutex)
36 36 0 1 36 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:804 (sleep mutex:unp_mtx)
5 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:tcptw)
1210 40478 24 15322 2 0 0 2 /usr/src/sys/kern/kern_descrip.c:2007 (sleep mutex:sleep mtxpool)
4 39 0 15 2 0 0 0 /usr/src/sys/kern/vfs_bio.c:294 (sleep mutex:needsbuffer lock)
11 75 0 12 6 0 1 0 /usr/src/sys/vm/vm_pageout.c:1446 (sleep mutex:vm page queue free mutex)
3 5 0 2 2 0 0 0 /usr/src/sys/dev/mfi/mfi.c:2497 (sleep mutex:MFI I/O lock)
10 1124 0 430 2 0 0 0 /usr/src/sys/kern/vfs_bio.c:321 (sleep mutex:needsbuffer lock)
43 546 0 16 34 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:241 (sleep mutex:so_rcv)
48 837 0 70 11 0 0 0 /usr/src/sys/kern/kern_resource.c:958 (sleep mutex:process lock)
57 96 0 3 32 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:872 (sleep mutex:unp_mtx)
22 640 0 67 9 0 0 0 /usr/src/sys/kern/kern_sig.c:2289 (spin mutex:sleepq chain)
22 3451 0 1295 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:1785 (sx:filedesc structure)
7 993 0 394 2 0 0 0 /usr/src/sys/kern/vfs_bio.c:363 (sleep mutex:needsbuffer lock)
2143 16776 113 120 139 0 1 3 /usr/src/sys/netinet/tcp_timer.c:128 (sleep mutex:tcp)
3 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:itimer)
5 23 0 8 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:1809 (sx:filedesc structure)
2698 2728989 26170 276752 9 0 395 945 /usr/src/sys/kern/vfs_cache.c:440 (sleep mutex:vnode interlock)
174 7682 3499 605 12 5 64 61 /usr/src/sys/kern/sys_generic.c:750 (sleep mutex:sellck)
632 37764 0 15216 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:1287 (sleep mutex:process lock)
29 5347 198 972 5 0 1 3 /usr/src/sys/netinet/tcp_timer.c:155 (sleep mutex:tcp)
123 19492 3735 609 32 6 63 62 /usr/src/sys/kern/sys_generic.c:759 (sleep mutex:sellck)
5 36 0 14 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:1834 (sx:filedesc structure)
5 5 0 1 5 0 0 0 /usr/src/sys/vm/vm_pager.c:475 (sleep mutex:vnode interlock)
790 120453 5 49016 2 0 1 1 /usr/src/sys/kern/kern_descrip.c:2139 (sleep mutex:sleep mtxpool)
17 117 0 10 11 0 0 0 /usr/src/sys/kern/kern_exit.c:296 (sleep mutex:Giant)
5 33 0 12 2 0 0 0 /usr/src/sys/vm/vm_pageout.c:1478 (sleep mutex:vm page queue mutex)
7297 41165 0 12 3430 0 0 0 /usr/src/sys/kern/vfs_subr.c:1638 (lockmgr:syncer)
10 30 0 10 3 0 0 0 /usr/src/sys/kern/kern_exit.c:747 (sx:allproc)
15 120 0 14 8 0 0 0 /usr/src/sys/kern/kern_exec.c:820 (sleep mutex:vm object)
170 27362 0 495 55 0 0 0 /usr/src/sys/netinet/tcp_syncache.c:648 (sleep mutex:inp)
30 30 0 1 30 0 0 0 /usr/src/sys/kern/subr_sleepqueue.c:754 (spin mutex:sleepq chain)
1367 85932 7908 15207 5 0 13 11 /usr/src/sys/kern/kern_descrip.c:1366 (sx:filelist lock)
5 43 0 18 2 0 0 0 /usr/src/sys/kern/vfs_bio.c:3841 (sleep mutex:vnode interlock)
5 61 0 20 3 0 0 0 /usr/src/sys/kern/vfs_bio.c:3851 (sleep mutex:vnode interlock)
12 22 0 2 11 0 0 0 /usr/src/sys/kern/kern_proc.c:304 (sleep mutex:Giant)
12 82 0 10 8 0 0 0 /usr/src/sys/kern/kern_exit.c:138 (sleep mutex:process lock)
79 34309 55 3216 10 0 90 2 /usr/src/sys/kern/subr_sleepqueue.c:781 (spin mutex:sleepq chain)
12 436 0 120 3 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/ip_nat.c:4446 (rw:ipf IP NAT rwlock)
17 28 0 2 14 0 0 0 /usr/src/sys/kern/kern_proc.c:320 (sleep mutex:Giant)
5 26 0 10 2 0 0 0 /usr/src/sys/kern/kern_exit.c:799 (sx:allproc)
5 98 0 35 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:2225 (sleep mutex:sleep mtxpool)
5 93 0 35 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:2239 (sleep mutex:sleep mtxpool)
5 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:unpcb)
4 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:32)
13 57 0 15 3 0 0 0 /usr/src/sys/vm/vm_kern.c:440 (sx:user map)
5 215 0 60 3 0 0 0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:buffer daemon lock)
45 77 0 2 38 0 0 0 /usr/src/sys/kern/kern_prot.c:325 (sx:proctree)
133 16420 5 482 34 0 0 1 /usr/src/sys/amd64/amd64/pmap.c:1959 (sleep mutex:pmap)
9 9 0 1 9 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:733 (sleep mutex:so_snd)
1123 24802 431 1585 15 0 3 9 /usr/src/sys/kern/sys_generic.c:924 (sleep mutex:sellck)
161 1784 0 15 118 0 0 0 /usr/src/sys/vm/vm_kern.c:469 (sx:user map)
12 229 0 70 3 0 0 0 /usr/src/sys/kern/kern_lockf.c:192 (sleep mutex:vnode interlock)
1228 32787 88 2778 11 0 24 11 /usr/src/sys/kern/uipc_socket.c:2468 (sleep mutex:so_snd)
31 85 0 4 21 0 0 0 /usr/src/sys/kern/kern_proc.c:398 (sleep mutex:Giant)
1075 52892 372 2446 21 0 34 11 /usr/src/sys/kern/sys_generic.c:933 (sleep mutex:sellck)
4 6 0 2 3 0 0 0 /usr/src/sys/vm/uma_core.c:2195 (sleep mutex:128 Bucket)
11 33 0 10 3 0 0 0 /usr/src/sys/kern/kern_exit.c:239 (sleep mutex:process lock)
18 132 0 10 13 0 0 0 /usr/src/sys/kern/kern_proc.c:424 (sleep mutex:Giant)
2 13 0 7 1 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:1135 (sleep mutex:unp_mtx)
21 4820 9516 1914 2 4 562 978 /usr/src/sys/kern/subr_turnstile.c:728 (spin mutex:sched lock)
66 120 0 2 60 0 0 0 /usr/src/sys/kern/kern_prot.c:384 (sx:proctree)
210 20049818 258237284 1017559 19 253 82293 1011233 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:lockbuilder mtxpool)
19 473 0 127 3 0 0 0 /usr/src/sys/kern/kern_sig.c:761 (spin mutex:process slock)
14 50 0 4 12 0 1 0 /usr/src/sys/kern/kern_proc.c:456 (sleep mutex:Giant)
4 12 0 4 3 0 0 0 /usr/src/sys/kern/kern_proc.c:242 (sleep mutex:process lock)
24 112788 0 43460 2 0 0 0 /usr/src/sys/kern/kern_clock.c:219 (spin mutex:process slock)
5 49 0 13 3 0 0 0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:inpcb)
29 121765 163587 48015 2 3 15449 21288 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock)
23 15715 36 6001 2 0 0 10 /usr/src/sys/kern/kern_timeout.c:191 (spin mutex:callout)
5 48 0 16 3 0 0 0 /usr/src/sys/kern/kern_event.c:841 (sleep mutex:kqueue)
3 5 0 2 2 0 0 0 /usr/src/sys/kern/kern_exit.c:349 (sleep mutex:session)
10 41 0 5 8 0 0 0 /usr/src/sys/netinet/tcp_timer.c:423 (sleep mutex:tcp)
3 6 0 2 3 0 0 0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:MAP ENTRY)
4 6 0 2 3 0 0 0 /usr/src/sys/kern/kern_proc.c:322 (sleep mutex:session)
4 15 0 5 3 0 0 0 /usr/src/sys/kern/tty.c:2081 (sleep mutex:process lock)
72 12092 168 4251 2 0 0 10 /usr/src/sys/dev/em/if_em.c:907 (sleep mutex:em0)
560 1168 0 3 389 0 0 0 /usr/src/sys/kern/vfs_subr.c:1638 (lockmgr:ufs)
1 3 0 2 1 0 0 0 /usr/src/sys/kern/kern_proc.c:305 (sleep mutex:process lock)
165 29014 0 1616 17 0 0 0 /usr/src/sys/netinet/tcp_input.c:2248 (sleep mutex:so_rcv)
10 27 0 3 9 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:895 (sleep mutex:so_snd)
19 3268 160 1201 2 0 4 6 /usr/src/sys/kern/sys_generic.c:1079 (sleep mutex:sellck)
33 33 0 1 33 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:1250 (sleep mutex:unp_mtx)
25 25 0 1 25 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:1251 (sleep mutex:unp_mtx)
79 40082 778 4251 9 0 41 38 /usr/src/sys/dev/em/if_em.c:938 (sleep mutex:em0)
224 17452 90 1414 12 0 4 6 /usr/src/sys/geom/geom_io.c:68 (sleep mutex:bio queue)
9 186 0 46 4 0 0 0 /usr/src/sys/kern/sys_pipe.c:572 (sleep mutex:pipe mutex)
84 1755 0 178 9 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2140 (sleep mutex:pmap)
4 20 0 7 2 0 0 0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:mbuf_jumbo_pagesize)
5 21 0 7 3 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:1269 (sleep mutex:unp_mtx)
4 27 0 10 2 0 0 0 /usr/src/sys/kern/kern_exit.c:399 (sleep mutex:process lock)
3 13 0 6 2 0 0 0 /usr/src/sys/ufs/ufs/ufs_vnops.c:175 (sleep mutex:vnode interlock)
82 49803 650 3600 13 0 14 21 /usr/src/sys/kern/kern_timeout.c:241 (sleep mutex:Giant)
24 78334 5370 28112 2 0 475 1134 /usr/src/sys/kern/kern_timeout.c:296 (spin mutex:callout)
10 90 0 16 5 0 0 0 /usr/src/sys/kern/kern_event.c:947 (sleep mutex:kqueue)
7 24 0 5 4 0 0 0 /usr/src/sys/kern/sys_pipe.c:609 (sleep mutex:pipe mutex)
108 23828 849 1430 16 0 8 23 /usr/src/sys/kern/sys_generic.c:1127 (sleep mutex:sellck)
13 4261 100 1568 2 0 19 23 /usr/src/sys/kern/subr_turnstile.c:710 (spin mutex:td_contested)
42 13330 5483 1568 8 3 433 710 /usr/src/sys/kern/subr_turnstile.c:894 (spin mutex:sched lock)
6 877 0 346 2 0 0 0 /usr/src/sys/kern/subr_turnstile.c:718 (spin mutex:td_contested)
9 9 0 1 9 0 0 0 /usr/src/sys/kern/kern_exit.c:434 (sleep mutex:process lock)
13 386 0 135 2 0 0 0 /usr/src/sys/kern/sys_socket.c:153 (sleep mutex:so_snd)
3 7 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:Mountpoints)
3 11 0 4 2 0 0 0 /usr/src/sys/kern/kern_proc.c:401 (sleep mutex:process lock)
144 877 0 10 87 0 1 0 /usr/src/sys/kern/kern_exit.c:449 (sleep mutex:process lock)
43 6987 0 1021 6 0 0 0 /usr/src/sys/kern/uipc_socket.c:2831 (sleep mutex:accept)
3 13 0 6 2 0 0 0 /usr/src/sys/kern/kern_switch.c:272 (spin mutex:turnstile lock)
2 2 0 1 2 0 0 0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1149 (sleep mutex:FFS)
23 6277 36 2280 2 0 4 8 /usr/src/sys/kern/kern_timeout.c:346 (spin mutex:callout)
36 354 397 67 5 5 10 44 /usr/src/sys/kern/subr_turnstile.c:203 (spin mutex:turnstile lock)
11 2274 196 847 2 0 0 8 /usr/src/sys/kern/sys_generic.c:1140 (spin mutex:sleepq chain)
128 736 0 10 73 0 1 0 /usr/src/sys/kern/kern_exit.c:469 (sleep mutex:process lock)
10 101 0 36 2 0 0 0 /usr/src/sys/kern/kern_event.c:1001 (sleep mutex:kqueue)
5 25 0 10 2 0 0 0 /usr/src/sys/kern/kern_proc.c:426 (sleep mutex:process lock)
4 12 0 5 2 0 0 0 /usr/src/sys/kern/kern_sig.c:959 (spin mutex:process slock)
5 50 0 20 2 0 0 0 /usr/src/sys/kern/kern_event.c:1021 (sleep mutex:kqueue)
23 16935 15065 6242 2 2 1672 1782 /usr/src/sys/kern/kern_clock.c:412 (spin mutex:sched lock)
11 303 0 103 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:782 (sleep mutex:sigio lock)
4 21 0 10 2 0 0 0 /usr/src/sys/kern/kern_kse.c:91 (spin mutex:kse lock)
157 2965 0 44 67 0 0 0 /usr/src/sys/vm/vm_map.c:1173 (sx:user map)
1129 265766 0 47196 5 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2261 (sleep mutex:pmap)
1451 34722 0 13799 2 0 0 0 /usr/src/sys/ufs/ufs/ufs_vnops.c:293 (sleep mutex:vnode interlock)
83 34697 1 13809 2 0 1 1 /usr/src/sys/kern/vfs_syscalls.c:1070 (sleep mutex:sleep mtxpool)
4 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:sackhole)
9 128 0 22 5 0 0 0 /usr/src/sys/ufs/ufs/ufs_dirhash.c:350 (sleep mutex:dirhash)
977 7961 0 449 17 0 0 0 /usr/src/sys/vm/vm_map.c:1202 (sx:user map)
24 95073 8174 34284 2 0 1212 1644 /usr/src/sys/kern/kern_timeout.c:419 (spin mutex:callout)
5 53 0 16 3 0 0 0 /usr/src/sys/kern/kern_event.c:1074 (sleep mutex:kqueue)
3 3 0 1 3 0 0 0 /usr/src/sys/kern/kern_event.c:1664 (sleep mutex:tty)
12 3470 0 1568 2 0 0 0 /usr/src/sys/kern/subr_turnstile.c:834 (spin mutex:td_contested)
22 15717 0 6001 2 0 0 0 /usr/src/sys/nfsserver/nfs_srvsock.c:796 (sleep mutex:nfsd_mtx)
5 54 0 21 2 0 0 0 /usr/src/sys/ufs/ufs/ufs_dirhash.c:368 (sleep mutex:dirhash)
5 255 0 107 2 0 0 0 /usr/src/sys/kern/vfs_syscalls.c:813 (sx:filedesc structure)
11 117 0 35 3 0 0 0 /usr/src/sys/ufs/ffs/ffs_vnops.c:683 (sleep mutex:process lock)
267 267 0 1 267 0 0 0 /usr/src/sys/vm/vm_map.c:2412 (sleep mutex:system map)
9 51 0 8 6 0 0 0 /usr/src/sys/kern/subr_eventhandler.c:215 (sleep mutex:process_exec)
2 8 0 4 2 0 0 0 /usr/src/sys/kern/kern_proc.c:574 (sleep mutex:session)
12 1266 0 480 2 0 0 0 /usr/src/sys/kern/kern_resource.c:633 (spin mutex:sleepq chain)
1335 51166 154 8527 6 0 4 9 /usr/src/sys/net/route.c:197 (sleep mutex:rtentry)
5 320 0 120 2 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/ip_frag.c:786 (rw:ipf fragment rwlock)
11 2839 3455 1117 2 3 278 378 /usr/src/sys/kern/subr_trap.c:182 (spin mutex:sched lock)
10 77 0 12 6 0 0 0 /usr/src/sys/kern/vfs_subr.c:3170 (sleep mutex:mountlist)
1255 129532 27 47159 2 0 0 1 /usr/src/sys/amd64/amd64/trap.c:586 (sleep mutex:process lock)
3 20 29 11 1 2 0 1 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:process lock)
1 3 0 2 1 0 0 0 /usr/src/sys/kern/kern_proc.c:554 (sleep mutex:process lock)
3306 139256 0 1241 112 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:145 (sx:so_snd_sx)
2014 126223 0 47159 2 0 0 0 /usr/src/sys/amd64/amd64/trap.c:595 (sleep mutex:process lock)
11 3674 13 1568 2 0 0 2 /usr/src/sys/kern/subr_turnstile.c:895 (spin mutex:td_contested)
1332 162499 426032 62509 2 6 31 2414 /usr/src/sys/ufs/ufs/ufs_vnops.c:388 (sleep mutex:vnode interlock)
21 11580 318 4211 2 0 45 62 /usr/src/sys/kern/kern_timeout.c:500 (spin mutex:callout)
37 250 0 33 7 0 0 0 /usr/src/sys/kern/uipc_socket.c:2864 (sleep mutex:so_snd)
40 9171 0 480 19 0 0 0 /usr/src/sys/kern/kern_timeout.c:241 (sleep mutex:process lock)
11 136 0 36 3 0 0 0 /usr/src/sys/kern/uipc_socket.c:2880 (sleep mutex:so_snd)
5 86 25 30 2 0 0 1 /usr/src/sys/netinet/tcp_timewait.c:253 (sleep mutex:so_rcv)
2 2 0 1 2 0 0 0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:1024)
3 43 0 20 2 0 0 0 /usr/src/sys/ufs/ufs/ufs_dirhash.c:457 (sleep mutex:dirhash)
26 5248 1749 1050 4 1 335 182 /usr/src/sys/kern/subr_trap.c:236 (spin mutex:sched lock)
609 3014 0 8 376 0 0 0 /usr/src/sys/kern/imgact_elf.c:672 (lockmgr:ufs)
10 105 0 20 5 0 0 0 /usr/src/sys/kern/kern_event.c:1182 (sleep mutex:kqueue)
41 255 25 59 4 0 0 1 /usr/src/sys/kern/uipc_socket.c:2899 (sleep mutex:so_snd)
4 27 0 10 2 0 0 0 /usr/src/sys/netinet/tcp_timewait.c:272 (sleep mutex:so_rcv)
37 12139 38 1448 8 0 1 2 /usr/src/sys/kern/kern_exit.c:686 (sleep mutex:process lock)
38 1101 0 175 6 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2441 (sleep mutex:pmap)
7 7 0 1 7 0 0 0 /usr/src/sys/kern/vfs_syscalls.c:951 (sx:filedesc structure)
5 86 0 36 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:985 (sleep mutex:sigio lock)
32 2530 0 567 4 0 0 0 /usr/src/sys/kern/kern_sig.c:663 (sleep mutex:sigacts)
13 6815 0 2590 2 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2463 (sleep mutex:pmap)
23 2266 0 481 4 0 0 0 /usr/src/sys/vm/vnode_pager.c:1192 (sleep mutex:vm object)
3 22 0 10 2 0 0 0 /usr/src/sys/kern/kern_exit.c:722 (sleep mutex:process lock)
5 38 0 16 2 0 0 0 /usr/src/sys/kern/kern_event.c:1257 (sleep mutex:kqueue)
86 1997 0 46 43 0 0 0 /usr/src/sys/dev/random/yarrow.c:191 (sleep mutex:random reseed)
10 28 0 3 9 0 0 0 /usr/src/sys/kern/kern_event.c:1266 (sleep mutex:kqueue)
4 11 0 3 3 0 0 0 /usr/src/sys/kern/kern_event.c:1275 (sleep mutex:kqueue)
60450466 1060829007 1316714222 340355 3116 3868 62072 61264 /usr/src/sys/kern/vfs_subr.c:2035 (lockmgr:ufs)
22 135 0 10 13 0 0 0 /usr/src/sys/kern/kern_fork.c:546 (sleep mutex:process group)
55 4580 0 467 9 0 0 0 /usr/src/sys/vm/vnode_pager.c:1222 (sleep mutex:vm object)
6 13 0 3 4 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:hostcache)
5 22 0 10 2 0 0 0 /usr/src/sys/kern/kern_exit.c:759 (sleep mutex:process lock)
8 468 0 120 3 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/ip_state.c:3052 (rw:ipf IP state rwlock)
11 203 0 67 3 0 0 0 /usr/src/sys/amd64/amd64/machdep.c:339 (sleep mutex:sigacts)
4 25 0 10 2 0 0 0 /usr/src/sys/kern/kern_exit.c:762 (sleep mutex:process lock)
6 30 0 9 3 0 0 0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:socket)
15 448 0 60 7 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:750 (sleep mutex:mountlist)
2326 16457165 20663004 1123871 14 18 274201 75173 /usr/src/sys/kern/kern_lock.c:200 (sleep mutex:lockbuilder mtxpool)
207 724 0 4 181 0 0 0 /usr/src/sys/kern/imgact_elf.c:805 (lockmgr:ufs)
3 17 0 8 2 0 0 0 /usr/src/sys/netinet/tcp_subr.c:782 (sleep mutex:so_rcv)
5 16 0 5 3 0 0 0 /usr/src/sys/kern/sys_pipe.c:974 (sleep mutex:pipe mutex)
114 5650 0 300 18 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:763 (sleep mutex:mountlist)
113 17694 11995 3024 5 3 213 1036 /usr/src/sys/kern/subr_turnstile.c:549 (spin mutex:turnstile lock)
202 9591282 0 4966397 1 0 932 0 /usr/src/sys/kern/subr_turnstile.c:555 (spin mutex:turnstile lock)
13 36 0 5 7 0 0 0 /usr/src/sys/dev/random/yarrow.c:278 (sleep mutex:random reseed)
74 1479 0 47 31 0 0 0 /usr/src/sys/amd64/amd64/mp_machdep.c:845 (spin mutex:smp rendezvous)
10 2319 0 900 2 0 0 0 /usr/src/sys/libkern/arc4random.c:137 (sleep mutex:arc4_mtx)
13 13 0 1 13 0 0 0 /usr/src/sys/kern/sys_socket.c:105 (sleep mutex:process lock)
119 34926 989 8006 4 0 34 56 /usr/src/sys/dev/em/if_em.c:1396 (sleep mutex:em0)
31 659 0 41 16 0 0 0 /usr/src/sys/kern/vfs_subr.c:2130 (lockmgr:ufs)
3 7 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:divcb)
34 77 0 6 12 0 0 0 /usr/src/sys/kern/kern_exit.c:848 (sleep mutex:process lock)
143 52143 1858 1568 33 1 8 87 /usr/src/sys/kern/subr_turnstile.c:593 (spin mutex:turnstile lock)
3 22 0 8 2 0 0 0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:mbuf_jumbo_pagesize)
5 46 0 16 2 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:1123 (sleep mutex:so_rcv)
60 2276 0 290 7 0 0 0 /usr/src/sys/amd64/amd64/mp_machdep.c:882 (spin mutex:smp rendezvous)
7 1367 0 490 2 0 0 0 /usr/src/sys/vm/vm_unix.c:81 (sleep mutex:process lock)
6 855 0 301 2 0 0 0 /usr/src/sys/netinet/tcp_timewait.c:491 (sleep mutex:so_rcv)
2 2 0 1 2 0 0 0 /usr/src/sys/kern/kern_exit.c:874 (sleep mutex:process lock)
4 17 0 8 2 0 0 0 /usr/src/sys/vm/vm_map.c:2709 (sleep mutex:process lock)
12 43 0 4 10 0 0 0 /usr/src/sys/vm/vm_map.c:1524 (sleep mutex:vm page queue mutex)
80 2562 0 74 34 0 0 0 /usr/src/sys/vm/vm_map.c:1560 (sx:user map)
9 110 0 22 5 0 0 0 /usr/src/sys/ufs/ufs/ufs_dirhash.c:349 (sleep mutex:dirhash list)
47 2034 978 171 11 5 4 3 /usr/src/sys/vm/vm_map.c:1534 (sleep mutex:vm page queue mutex)
2 20 0 10 2 0 0 0 /usr/src/sys/kern/kern_fork.c:767 (sleep mutex:schedtail)
14 59 0 10 5 0 0 0 /usr/src/sys/kern/subr_eventhandler.c:215 (sleep mutex:process_exit)
15 72 0 16 4 0 0 0 /usr/src/sys/kern/kern_event.c:1442 (sleep mutex:kqueue)
6 103 0 32 3 0 0 0 /usr/src/sys/kern/vfs_subr.c:1573 (sleep mutex:Syncer mtx)
23 139 0 35 3 0 0 0 /usr/src/sys/kern/vfs_bio.c:1280 (sleep mutex:vm object)
68 38631 12588 15205 2 0 0 15 /usr/src/sys/kern/kern_descrip.c:2179 (sx:filelist lock)
19 80 0 5 16 0 0 0 /usr/src/sys/kern/sys_pipe.c:1127 (sleep mutex:pipe mutex)
35 400 0 12 33 0 0 0 /usr/src/sys/netinet/ip_fw2.c:4877 (sleep mutex:IPFW dynamic rules)
3 11 0 4 2 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2695 (sleep mutex:pmap)
105 3835 0 60 63 0 2 0 /usr/src/sys/kern/kern_timeout.c:241 (sleep mutex:em0)
46 13416 0 601 22 0 0 0 /usr/src/sys/geom/geom_event.c:185 (sx:GEOM topology)
5 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:FFS1 dinode)
23 4066 0 1500 2 0 0 0 /usr/src/sys/dev/syscons/syscons.c:1800 (spin mutex:syscons video lock)
13 31 0 7 4 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:3744 (sleep mutex:Softdep Lock)
4 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:g_bio)
18 84 0 10 8 0 0 0 /usr/src/sys/kern/kern_conf.c:348 (sleep mutex:Giant)
2 6 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ksiginfo)
41 272 0 10 27 0 0 0 /usr/src/sys/kern/kern_fork.c:545 (sx:proctree)
4 35 0 12 2 0 0 0 /usr/src/sys/kern/vfs_subr.c:2944 (sleep mutex:struct mount mtx)
2361 26024 284 204 127 1 24 12 /usr/src/sys/vm/vm_object.c:641 (sleep mutex:vm page queue mutex)
5 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ripcb)
124 2659 0 151 17 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2734 (sleep mutex:pmap)
120 1858 0 151 12 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2735 (sleep mutex:pmap)
20 20 0 1 20 0 0 0 /usr/src/sys/kern/kern_conf.c:360 (sleep mutex:Giant)
36 4233 0 506 8 0 0 0 /usr/src/sys/netinet/tcp_input.c:2867 (sleep mutex:so_rcv)
59 1534 0 123 12 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2737 (sleep mutex:pmap)
56 1551 0 123 12 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2738 (sleep mutex:pmap)
12 46 0 8 5 0 0 0 /usr/src/sys/kern/kern_sig.c:952 (sleep mutex:sigacts)
5 37 0 10 3 0 0 0 /usr/src/sys/kern/subr_eventhandler.c:215 (sleep mutex:schedtail)
20 201 0 19 10 0 0 0 /usr/src/sys/kern/vfs_subr.c:1653 (sleep mutex:Syncer mtx)
46 131 0 5 26 0 0 0 /usr/src/sys/kern/kern_conf.c:372 (sleep mutex:Giant)
1728 128342 13097 14524 8 0 32 46 /usr/src/sys/kern/vfs_cache.c:692 (sleep mutex:Giant)
5 36 0 10 3 0 0 0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:NAMEI)
13 59 19 7 8 2 0 1 /usr/src/sys/kern/kern_conf.c:384 (sleep mutex:Giant)
22 2020 0 601 3 0 0 0 /usr/src/sys/geom/geom_event.c:233 (sx:GEOM topology)
5 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:VNODEPOLL)
4 12 0 3 4 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:VNODE)
2 2 0 1 2 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:3785 (sleep mutex:Softdep Lock)
25 25 0 1 25 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:3794 (sleep mutex:Softdep Lock)
11 225 0 72 3 0 0 0 /usr/src/sys/vm/vm_glue.c:183 (sx:user map)
26 386 0 21 18 0 1 0 /usr/src/sys/kern/kern_event.c:1568 (sleep mutex:kqueue)
4 32 54 12 2 4 0 1 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1208 (sleep mutex:struct mount mtx)
5 34 0 12 2 0 0 0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1217 (sleep mutex:struct mount mtx)
4 41 0 16 2 0 0 0 /usr/src/sys/kern/kern_event.c:1598 (sleep mutex:kqueue)
7 166 0 60 2 0 0 0 /usr/src/sys/net/if.c:1473 (sleep mutex:ifnet)
340 295281 10543 4618 63 2 207 179 /usr/src/sys/netinet/tcp_input.c:479 (sleep mutex:inp)
5 25 19 10 2 1 5 5 /usr/src/sys/kern/kern_fork.c:383 (spin mutex:sched lock)
10 90 0 19 4 0 0 0 /usr/src/sys/fs/devfs/devfs_vnops.c:194 (sleep mutex:devfs interlock)
2390 2633 0 3 877 0 0 0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1250 (sleep mutex:struct mount mtx)
3 34 0 16 2 0 0 0 /usr/src/sys/kern/kern_event.c:1620 (sleep mutex:kqueue)
12 15 0 3 5 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ipq)
5 250 0 96 2 0 0 0 /usr/src/sys/vm/vm_map.c:1800 (sx:user map)
10 34 0 5 6 0 0 0 /usr/src/sys/kern/sys_pipe.c:1327 (sleep mutex:pipe mutex)
5 31 0 12 2 0 0 0 /usr/src/sys/vm/vm_glue.c:274 (sleep mutex:vm page queue mutex)
1594 1535506 35138225 558021 2 62 1181 117583 /usr/src/sys/kern/kern_lock.c:583 (sleep mutex:lockbuilder mtxpool)
16 479 0 67 7 0 0 0 /usr/src/sys/kern/subr_trap.c:248 (sleep mutex:sigacts)
24 5352 132 1864 2 0 0 5 /usr/src/sys/netinet/tcp_input.c:2025 (sleep mutex:so_snd)
31 31 0 1 31 0 0 0 /usr/src/sys/kern/vfs_bio.c:1530 (sleep mutex:vm object)
6 1292 335 496 2 0 1 4 /usr/src/sys/kern/kern_lock.c:607 (sleep mutex:lockbuilder mtxpool)
107 186602 0 12002 15 0 0 0 /usr/src/sys/kern/subr_taskqueue.c:71 (spin mutex:fast_taskqueue)
5 31 0 12 2 0 0 0 /usr/src/sys/vm/vm_glue.c:310 (sleep mutex:vm page queue mutex)
419 3234 0 15 215 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2950 (sleep mutex:pmap)
4 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:SLEEPQUEUE)
5 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:DP fakepg)
4 32 0 12 2 0 0 0 /usr/src/sys/kern/vfs_subr.c:3179 (sleep mutex:struct mount mtx)
12 597 0 235 2 0 0 0 /usr/src/sys/vm/vm_object.c:225 (sleep mutex:vm object_list)
5 41 0 12 3 0 0 0 /usr/src/sys/kern/vfs_subr.c:3185 (sleep mutex:struct mount mtx)
97 20060 0 495 40 0 0 0 /usr/src/sys/netinet/tcp_input.c:636 (sleep mutex:inp)
5 29 0 9 3 0 0 0 /usr/src/sys/kern/vfs_subr.c:1888 (sleep mutex:Syncer mtx)
3 21 0 7 3 0 0 0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:socket)
1022 35624 180 13861 2 0 109 7 /usr/src/sys/kern/vfs_vnops.c:903 (sleep mutex:struct mount mtx)
4 83 0 33 2 0 0 0 /usr/src/sys/vm/vm_map.c:3092 (sleep mutex:system map)
12 1057 33 379 2 0 2 3 /usr/src/sys/vm/vm_fault.c:137 (sleep mutex:vm page queue mutex)
22 2098 0 600 3 0 0 0 /usr/src/sys/dev/random/randomdev_soft.c:249 (spin mutex:entropy harvest mutex)
3515419 3755160 0 2109 1780 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:145 (sx:so_rcv_sx)
3 8 0 4 2 0 0 0 /usr/src/sys/kern/kern_proc.c:1206 (sleep mutex:process lock)
3 36 0 15 2 0 0 0 /usr/src/sys/kern/kern_exec.c:318 (sleep mutex:process lock)
3 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:FFS2 dinode)
5 18 0 7 2 0 0 0 /usr/src/sys/fs/devfs/devfs_vnops.c:325 (sx:proctree)
2 3 0 2 1 0 0 0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:Files)
13 1991 2 594 3 0 0 1 /usr/src/sys/dev/random/randomdev_soft.c:269 (spin mutex:entropy harvest mutex)
5 88 12 34 2 0 0 1 /usr/src/sys/kern/kern_event.c:1802 (sleep mutex:kqueue)
39 188 0 16 11 0 0 0 /usr/src/sys/kern/sys_pipe.c:1461 (sleep mutex:pipe mutex)
4 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ata_request)
469 1141 0 4 285 0 0 0 /usr/src/sys/kern/vfs_subr.c:1638 (lockmgr:devfs)
1475 357918 829 13709 26 0 0 2 /usr/src/sys/kern/vfs_vnops.c:288 (lockmgr:ufs)
23 23 0 1 23 0 0 0 /usr/src/sys/kern/uipc_socket.c:481 (sleep mutex:so_rcv)
23 16712 0 6027 2 0 0 0 /usr/src/sys/dev/random/randomdev_soft.c:308 (spin mutex:entropy harvest mutex)
5 47 0 16 2 0 0 0 /usr/src/sys/kern/sys_pipe.c:1501 (sleep mutex:pipe mutex)
6 144 0 46 3 0 0 0 /usr/src/sys/kern/vfs_bio.c:429 (sleep mutex:buffer daemon lock)
960 265391 0 101087 2 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:3074 (sleep mutex:pmap)
5 41 0 16 2 0 0 0 /usr/src/sys/kern/kern_event.c:1868 (sleep mutex:kqueue)
6 29 0 10 2 0 0 0 /usr/src/sys/kern/kern_fork.c:658 (spin mutex:process slock)
26 139 57 16 8 3 0 1 /usr/src/sys/dev/mfi/mfi_disk.c:258 (sleep mutex:MFI I/O lock)
43 144 0 5 28 0 0 0 /usr/src/sys/fs/devfs/devfs_vnops.c:375 (lockmgr:devfs)
23 3037 3056 600 5 5 322 377 /usr/src/sys/kern/sched_4bsd.c:271 (spin mutex:sched lock)
5 31 43 10 3 4 3 5 /usr/src/sys/kern/kern_fork.c:667 (spin mutex:sched lock)
38 38 0 1 38 0 0 0 /usr/src/sys/vm/vm_kern.c:334 (sleep mutex:vm object)
57 1556 0 328 4 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:255 (sleep mutex:inp)
20 33680 4195 13861 2 0 50 116 /usr/src/sys/kern/vfs_vnops.c:1046 (sleep mutex:struct mount mtx)
4 11 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:VM OBJECT)
3 17 0 8 2 0 0 0 /usr/src/sys/kern/sys_pipe.c:366 (sleep mutex:sleep mtxpool)
6 49 0 19 2 0 0 0 /usr/src/sys/fs/devfs/devfs_vnops.c:201 (sx:devfsmount)
4 19 0 8 2 0 0 0 /usr/src/sys/kern/sys_pipe.c:381 (sleep mutex:sleep mtxpool)
12 13957 50 6001 2 0 2 11 /usr/src/sys/kern/kern_clock.c:264 (spin mutex:callout)
63 63 0 1 63 0 0 0 /usr/src/sys/vm/vm_kern.c:403 (sleep mutex:vm object)
33 171 0 8 21 0 0 0 /usr/src/sys/kern/kern_exec.c:505 (sleep mutex:process lock)
23 3053 278 1112 2 0 3 8 /usr/src/sys/vm/vm_fault.c:340 (sleep mutex:vm page queue mutex)
471 84026 0 6206 13 0 0 0 /usr/src/sys/kern/sched_4bsd.c:382 (spin mutex:process slock)
30 4559 0 467 9 0 0 0 /usr/src/sys/vm/vnode_pager.c:1204 (sleep mutex:vnode interlock)
5 174 0 55 3 0 0 0 /usr/src/sys/netinet/ip_fw2.c:1303 (sleep mutex:IPFW dynamic rules)
428 24067 602 7226 3 0 3 20 /usr/src/sys/netinet/tcp_output.c:270 (sleep mutex:so_snd)
10 8210 17484 2800 2 6 1039 1446 /usr/src/sys/kern/sched_4bsd.c:385 (spin mutex:sched lock)
296 361 0 6 60 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:4239 (sleep mutex:Softdep Lock)
5 28 0 10 2 0 0 0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:NAMEI)
84 4311 0 499 8 0 0 0 /usr/src/sys/netinet/tcp_syncache.c:316 (sleep mutex:tcp_sc_head)
11 216 0 73 2 0 0 0 /usr/src/sys/kern/kern_sig.c:1961 (spin mutex:process slock)
132 16865 5178 1461 11 3 92 100 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:sellck)
5 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:syncache)
25 2723309 236100 1027898 2 0 18742 33987 /usr/src/sys/kern/subr_sleepqueue.c:331 (spin mutex:sched lock)
5 32 0 10 3 0 0 0 /usr/src/sys/kern/kern_descrip.c:1472 (sleep mutex:fdesc)
6 209 0 71 2 0 0 0 /usr/src/sys/kern/uipc_socket.c:686 (sleep mutex:so_rcv)
47 11875 135 4251 2 0 1 6 /usr/src/sys/netinet/ip_output.c:594 (sleep mutex:rtentry)
6953267 7713157 0 92 83838 0 0 0 /usr/src/sys/kern/vfs_vnops.c:515 (lockmgr:ufs)
39 5574 0 567 9 0 0 0 /usr/src/sys/kern/kern_sig.c:661 (sleep mutex:process lock)
5 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:rtentry)
13 89 0 10 8 0 0 0 /usr/src/sys/netinet/tcp_timewait.c:271 (sleep mutex:accept)
6 97 0 36 2 0 0 0 /usr/src/sys/kern/uipc_socket.c:735 (sleep mutex:so_rcv)
3 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:THREAD)
77 89 0 2 44 0 0 0 /usr/src/sys/kern/vfs_syscalls.c:3784 (lockmgr:ufs)
2 7 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf_ext_refcnt)
20 34058 0 13936 2 0 0 0 /usr/src/sys/kern/kern_prot.c:91 (sleep mutex:process lock)
261 1916 0 11 174 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:470 (sleep mutex:inp)
330 5506 0 35 157 0 0 0 /usr/src/sys/kern/vfs_vnops.c:578 (lockmgr:ufs)
30 4236 9198 1551 2 5 724 1055 /usr/src/sys/kern/subr_sleepqueue.c:411 (spin mutex:sched lock)
11 466 0 182 2 0 0 0 /usr/src/sys/vm/vm_map.c:980 (sleep mutex:vm object)
3 5 0 2 2 0 0 0 /usr/src/sys/kern/kern_prot.c:109 (sleep mutex:process lock)
18 607 0 67 9 0 0 0 /usr/src/sys/amd64/amd64/machdep.c:338 (sleep mutex:process lock)
2 5 0 2 2 0 0 0 /usr/src/sys/kern/kern_prot.c:128 (sleep mutex:process lock)
1342 189258 0 13884 13 0 1 0 /usr/src/sys/kern/vfs_vnops.c:613 (lockmgr:ufs)
371 203548 60 4618 44 0 10 3 /usr/src/sys/netinet/tcp_input.c:400 (sleep mutex:tcp)
125 19394 0 1006 19 0 0 0 /usr/src/sys/netinet/tcp_syncache.c:464 (sleep mutex:tcp_sc_head)
41 981 0 60 16 0 0 0 /usr/src/sys/kern/vfs_subr.c:730 (sleep mutex:vnode_free_list)
4 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:NFSMOUNT)
5 60 0 16 3 0 0 0 /usr/src/sys/netinet/ip_fw2.c:1466 (sleep mutex:IPFW dynamic rules)
6 603 0 204 2 0 0 0 /usr/src/sys/vm/vm_object.c:667 (sleep mutex:vm object_list)
38 209 0 10 20 0 0 0 /usr/src/sys/kern/kern_fork.c:267 (sx:allproc)
3 7 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:IPFW dynamic rule)
14 39 0 5 7 0 0 0 /usr/src/sys/vm/vm_object.c:1331 (sleep mutex:vm page queue mutex)
126 1249 0 19 65 0 0 0 /usr/src/sys/kern/vfs_subr.c:2035 (lockmgr:devfs)
3199 27657 139 4251 6 0 10 9 /usr/src/sys/net/route.c:1287 (sleep mutex:rtentry)
5 182 0 67 2 0 0 0 /usr/src/sys/amd64/amd64/machdep.c:414 (sleep mutex:process lock)
10 58 0 8 7 0 0 0 /usr/src/sys/netinet/tcp_subr.c:781 (sleep mutex:accept)
136 1219 0 18 67 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:571 (sleep mutex:inp)
46 25509 0 4251 6 0 1 0 /usr/src/sys/net/route.c:1303 (sleep mutex:rtentry)
5 40 0 12 3 0 0 0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:vm page queue free mutex)
3 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:tcpcb)
36 1207 0 67 18 0 0 0 /usr/src/sys/kern/kern_sig.c:2177 (spin mutex:process slock)
18 33378 37 13926 2 0 2 3 /usr/src/sys/kern/kern_conf.c:69 (sleep mutex:cdev)
5 30 0 10 3 0 0 0 /usr/src/sys/kern/kern_descrip.c:1673 (sleep mutex:fdesc)
5 93 13 36 2 0 0 1 /usr/src/sys/kern/uipc_syscalls.c:404 (sleep mutex:so_rcv)
1 1 0 1 1 0 0 0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:Files)
26 2199187 45543 1023647 2 0 16907 5743 /usr/src/sys/kern/subr_sleepqueue.c:541 (spin mutex:sched lock)
2 13 0 7 1 0 0 0 /usr/src/sys/kern/kern_exec.c:784 (sleep mutex:process lock)
11 105 0 36 2 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:606 (sleep mutex:inp)
14 64 0 7 9 0 0 0 /usr/src/sys/fs/devfs/devfs_vfsops.c:157 (sx:devfsmount)
5 30 60 10 3 6 4 6 /usr/src/sys/kern/sched_4bsd.c:650 (spin mutex:sched lock)
5 24 0 10 2 0 1 0 /usr/src/sys/kern/sched_4bsd.c:653 (spin mutex:sched lock)
136 15949 969 482 33 2 60 14 /usr/src/sys/amd64/amd64/pmap.c:1958 (sleep mutex:vm page queue mutex)
4305 28301 0 23 1230 0 0 0 /usr/src/sys/vm/vm_map.c:2412 (sx:user map)
22 8680 20146 2700 3 7 1358 2146 /usr/src/sys/kern/subr_sleepqueue.c:576 (spin mutex:sched lock)
3 3 1 1 3 1 0 1 /usr/src/sys/kern/sched_4bsd.c:682 (spin mutex:sched lock)
31 160 0 7 22 0 0 0 /usr/src/sys/fs/devfs/devfs_vnops.c:789 (lockmgr:devfs)
19 2655 0 301 8 0 0 0 /usr/src/sys/netinet/tcp_timewait.c:490 (sleep mutex:accept)
1769 134142 6505 46891 2 0 121 236 /usr/src/sys/vm/vm_page.c:1030 (sleep mutex:vm page queue free mutex)
124 1598 0 37 43 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:697 (sleep mutex:inp)
29 265 0 19 13 0 0 0 /usr/src/sys/fs/devfs/devfs_vnops.c:197 (sleep mutex:vnode interlock)
3 10 0 3 3 0 0 0 /usr/src/sys/kern/vfs_subr.c:1545 (sleep mutex:vnode interlock)
5 58 0 19 3 0 0 0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:kqueue)
6 578 0 180 3 0 0 0 /usr/src/sys/kern/kern_kthread.c:194 (sleep mutex:process lock)
5 44 0 12 3 0 0 0 /usr/src/sys/kern/vfs_bio.c:3437 (sleep mutex:vm page queue mutex)
145 36802 9668 2017 18 4 178 208 /usr/src/sys/netinet/tcp_usrreq.c:729 (sleep mutex:inp)
5 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UMA Slabs)
3 21 0 8 2 0 0 0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:tcptw)
12 491 31 179 2 0 0 2 /usr/src/sys/kern/vfs_bio.c:1487 (sleep mutex:vnode interlock)
23 52319 0 19274 2 0 1 0 /usr/src/sys/kern/kern_sig.c:996 (sleep mutex:process lock)
419 467 0 18 25 0 0 0 /usr/src/sys/kern/uipc_syscalls.c:565 (sleep mutex:so_rcv)
18 123 0 12 10 0 0 0 /usr/src/sys/fs/devfs/devfs_vnops.c:691 (sx:devfsmount)
3293 121661 0 1236 98 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:779 (sleep mutex:inp)
18 373 36 42 8 0 3 2 /usr/src/sys/vm/vm_object.c:1551 (sleep mutex:vm page queue mutex)
12 164 0 38 4 0 0 0 /usr/src/sys/kern/vfs_bio.c:3499 (sleep mutex:vm page queue mutex)
11 1618 2939 609 2 4 274 322 /usr/src/sys/kern/sys_generic.c:753 (spin mutex:sched lock)
8 89 0 19 4 0 0 0 /usr/src/sys/kern/vfs_subr.c:1622 (sleep mutex:vnode interlock)
48 147 0 5 29 0 0 0 /usr/src/sys/netinet/tcp_reass.c:267 (sleep mutex:so_rcv)
11 2933 983 1071 2 0 23 25 /usr/src/sys/vm/vm_object.c:1570 (sleep mutex:vm page queue mutex)
3310 11238 0 7 1605 0 0 0 /usr/src/sys/vm/vm_map.c:2584 (sx:user map)
1 3 0 2 1 0 0 0 /usr/src/sys/vm/uma_core.c:2394 (sleep mutex:tcptw)
13 140 0 19 7 0 0 0 /usr/src/sys/kern/vfs_subr.c:1642 (sleep mutex:vnode interlock)
85 1742 0 178 9 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2139 (sleep mutex:vm page queue mutex)
11 1593 3229 602 2 5 253 379 /usr/src/sys/kern/sys_generic.c:778 (spin mutex:sched lock)
33 6922534 286114 1029804 6 0 35897 37195 /usr/src/sys/kern/sched_4bsd.c:847 (spin mutex:sched lock)
5 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:tcpreass)
1192 19128319 6157083 63412 301 97 42132 23511 /usr/src/sys/kern/vfs_hash.c:79 (sleep mutex:vnode interlock)
4 6 10 2 3 5 1 2 /usr/src/sys/kern/subr_sleepqueue.c:754 (spin mutex:sched lock)
12 1617 2958 604 2 4 191 243 /usr/src/sys/kern/sys_generic.c:795 (spin mutex:sched lock)
31 1302 0 455 2 0 0 0 /usr/src/sys/vm/vm_object.c:352 (sleep mutex:vm object)
12 63 0 7 9 0 0 0 /usr/src/sys/fs/devfs/devfs_vnops.c:351 (sleep mutex:vnode interlock)
5 108 0 38 2 0 0 0 /usr/src/sys/kern/vfs_bio.c:2326 (sleep mutex:vm object)
23 39 0 3 13 0 0 0 /usr/src/sys/kern/kern_prot.c:481 (sleep mutex:process lock)
7 22 0 4 5 0 0 0 /usr/src/sys/kern/kern_event.c:1664 (sleep mutex:pipe mutex)
68 443 0 9 49 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:254 (sleep mutex:tcp)
1436 130021 40386 47192 2 0 101 428 /usr/src/sys/vm/vm_fault.c:887 (sleep mutex:vm page queue mutex)
29 4125 0 506 8 0 0 0 /usr/src/sys/netinet/tcp_input.c:2850 (sleep mutex:so_snd)
1 1 0 1 1 0 0 0 /usr/src/sys/fs/fifofs/fifo_vnops.c:297 (sleep mutex:sleep mtxpool)
19 829 0 67 12 0 0 0 /usr/src/sys/kern/subr_trap.c:247 (sleep mutex:process lock)
43 110 0 5 22 0 0 0 /usr/src/sys/kern/vfs_bio.c:1649 (sleep mutex:vnode interlock)
13 13 0 1 13 0 0 0 /usr/src/sys/kern/kern_resource.c:266 (spin mutex:process slock)
2368 158777 22 48298 3 0 1 1 /usr/src/sys/vm/vm_object.c:441 (sleep mutex:vm object)
4 21 11 6 3 1 2 1 /usr/src/sys/kern/subr_sleepqueue.c:852 (spin mutex:sched lock)
19 3678 0 490 7 0 0 0 /usr/src/sys/vm/vm_unix.c:88 (sx:user map)
1 1 0 1 1 0 0 0 /usr/src/sys/fs/fifofs/fifo_vnops.c:224 (sleep mutex:fifo mutex)
10 1147 0 441 2 0 0 0 /usr/src/sys/vm/vm_mmap.c:284 (sleep mutex:process lock)
1129 258864 49465 47196 5 1 250 443 /usr/src/sys/amd64/amd64/pmap.c:2260 (sleep mutex:vm page queue mutex)
5 21 0 8 2 0 0 0 /usr/src/sys/vm/vm_map.c:2713 (sx:user map)
5 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf_cluster)
5 50 0 16 3 0 0 0 /usr/src/sys/kern/kern_event.c:534 (sleep mutex:sleep mtxpool)
3 16 0 6 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf)
6 197 0 60 3 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:733 (sleep mutex:Softdep Lock)
62 955 0 47 20 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:956 (sleep mutex:inp)
1008 130585 890 48074 2 0 45 47 /usr/src/sys/vm/vm_page.c:1315 (sleep mutex:vm page queue free mutex)
22 6465 12743 2446 2 5 909 1166 /usr/src/sys/kern/sys_generic.c:927 (spin mutex:sched lock)
135 4483 0 1551 2 0 0 0 /usr/src/sys/kern/subr_sleepqueue.c:392 (sleep mutex:sigacts)
961 4160 0 181 22 0 0 0 /usr/src/sys/vm/vm_map.c:1467 (sleep mutex:vm object)
37 92 0 5 18 0 0 0 /usr/src/sys/kern/kern_prot.c:593 (sleep mutex:process lock)
3089 8221 0 28 293 0 0 0 /usr/src/sys/vm/vm_object.c:497 (sleep mutex:vm object)
22 8647 0 3098 2 0 0 0 /usr/src/sys/kern/uipc_syscalls.c:135 (sleep mutex:sleep mtxpool)
13 79 0 8 9 0 0 0 /usr/src/sys/kern/vfs_bio.c:3692 (sleep mutex:vm page queue mutex)
31 20026 126 2590 7 0 8 10 /usr/src/sys/vm/vm_fault.c:981 (sleep mutex:vm page queue mutex)
36 354 0 60 5 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:770 (sleep mutex:Softdep Lock)
21 2330 6130 862 2 7 386 512 /usr/src/sys/kern/sys_generic.c:950 (spin mutex:sched lock)
3 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:256)
2 6 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf_jumbo_16k)
11 4343 7674 1584 2 4 693 845 /usr/src/sys/kern/sys_generic.c:967 (spin mutex:sched lock)
4 8 0 3 2 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:4889 (sleep mutex:Softdep Lock)
23 1412 0 503 2 0 0 0 /usr/src/sys/kern/uipc_socket.c:409 (sleep mutex:accept)
18 189 0 25 7 0 0 0 /usr/src/sys/kern/vfs_subr.c:1860 (sleep mutex:vnode interlock)
5 82 0 31 2 0 0 0 /usr/src/sys/kern/uipc_syscalls.c:183 (sleep mutex:sleep mtxpool)
15 32 0 3 10 0 0 0 /usr/src/sys/kern/kern_prot.c:646 (sleep mutex:process lock)
34 251 0 74 3 0 0 0 /usr/src/sys/kern/kern_sig.c:2102 (sleep mutex:sigacts)
27 127819 3909 62089 2 0 1192 414 /usr/src/sys/kern/kern_synch.c:216 (spin mutex:sched lock)
6 378 0 129 2 0 0 0 /usr/src/sys/vm/vm_object.c:567 (sleep mutex:vm object)
3208 246599 218 346 712 0 63 13 /usr/src/sys/vm/vm_object.c:1816 (sleep mutex:vm page queue mutex)
13 981 0 300 3 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:843 (sleep mutex:Softdep Lock)
62 770 0 28 27 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:385 (sleep mutex:udp)
41 1224 0 66 18 0 2 0 /usr/src/sys/kern/kern_sysctl.c:1334 (sleep mutex:Giant)
4 45 0 21 2 0 0 0 /usr/src/sys/vm/uma_core.c:889 (sleep mutex:tcpcb)
5 15 0 4 3 0 0 0 /usr/src/sys/vm/vm_fault.c:1061 (sleep mutex:vm page queue mutex)
22 1330 0 495 2 0 0 0 /usr/src/sys/kern/uipc_socket.c:451 (sleep mutex:accept)
6 1759 0 601 2 0 0 0 /usr/src/sys/geom/geom_event.c:187 (sleep mutex:GEOM orphanage)
42 2034552 3065332 1029804 1 2 40557 274671 /usr/src/sys/kern/sched_4bsd.c:1102 (spin mutex:sched lock)
3 5 38 2 2 19 0 1 /usr/src/sys/kern/sched_4bsd.c:385 (spin mutex:turnstile lock)
19 5672 0 601 9 0 0 0 /usr/src/sys/geom/geom_event.c:201 (sleep mutex:GEOM orphanage)
22 4135 0 1530 2 0 0 0 /usr/src/sys/netinet/tcp_hostcache.c:291 (sleep mutex:tcp_hc_entry)
604 1524 0 3 508 0 0 0 /usr/src/sys/vm/uma_core.c:1492 (sleep mutex:UMA lock)
5 207 0 60 3 0 0 0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:Softdep Lock)
273 1991 0 11 181 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:467 (sleep mutex:tcp)
57 260 0 7 37 0 0 0 /usr/src/sys/kern/vfs_vnops.c:288 (lockmgr:devfs)
20 49 0 4 12 0 0 0 /usr/src/sys/kern/kern_prot.c:745 (sleep mutex:process lock)
19 33978 5496 13850 2 0 57 152 /usr/src/sys/kern/vfs_mount.c:432 (sleep mutex:struct mount mtx)
94 213487 28811 63735 3 0 298 852 /usr/src/sys/kern/vfs_mount.c:441 (sleep mutex:struct mount mtx)
72 6654 391 2152 3 0 17 25 /usr/src/sys/kern/uipc_socket.c:1424 (sleep mutex:so_rcv)
11 11 0 1 11 0 0 0 /usr/src/sys/kern/vfs_syscalls.c:367 (lockmgr:ufs)
3 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:MAP)
3 3 0 1 3 0 0 0 /usr/src/sys/ufs/ufs/ufs_vnops.c:2038 (sleep mutex:vnode interlock)
5 62 176 21 2 8 12 16 /usr/src/sys/kern/sys_generic.c:1140 (spin mutex:sched lock)
10 1128 0 297 3 0 0 0 /usr/src/sys/kern/vfs_subr.c:2019 (sleep mutex:vnode interlock)
22 86 0 6 14 0 0 0 /usr/src/sys/kern/kern_prot.c:807 (sleep mutex:process lock)
24 17047 0 6001 2 0 0 0 /usr/src/sys/netinet/ip_dummynet.c:739 (sleep mutex:dummynet)
7 1158 0 424 2 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:116 (sleep mutex:so_rcv)
145 1332 0 18 74 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:568 (sleep mutex:tcp)
2378 902001 805071 340953 2 2 213 5895 /usr/src/sys/kern/vfs_subr.c:2039 (sleep mutex:vnode interlock)
5 5 0 1 5 0 0 0 /usr/src/sys/ufs/ffs/ffs_softdep.c:999 (sleep mutex:Softdep Lock)
12 55 0 16 3 0 0 0 /usr/src/sys/kern/kern_event.c:530 (sx:filedesc structure)
5 117 0 40 2 0 0 0 /usr/src/sys/kern/kern_exit.c:233 (sleep mutex:process_exit)
2449 493209 107985 193425 2 0 1229 3624 /usr/src/sys/kern/vfs_subr.c:2062 (sleep mutex:vnode interlock)
22 16740 10 6001 2 0 0 3 /usr/src/sys/kern/kern_synch.c:320 (spin mutex:fast_taskqueue)
5 9 0 2 4 0 0 0 /usr/src/sys/kern/kern_sig.c:1468 (sleep mutex:process lock)
23 155940 124384 62154 2 2 12262 18314 /usr/src/sys/kern/sched_4bsd.c:1281 (spin mutex:sched lock)
5 10 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:32)
4 41 0 16 2 0 0 0 /usr/src/sys/netinet/tcp_usrreq.c:1253 (sleep mutex:inp)
11 38 0 4 9 0 0 0 /usr/src/sys/kern/vfs_default.c:410 (sleep mutex:vnode interlock)
4 11 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:64 Bucket)
771 165545 345 64596 2 0 1198 17 /usr/src/sys/kern/vfs_subr.c:2101 (sleep mutex:vnode interlock)
5 91 0 36 2 0 0 0 /usr/src/sys/kern/uipc_syscalls.c:426 (sleep mutex:sleep mtxpool)
15 98 0 12 8 0 0 0 /usr/src/sys/vm/vm_glue.c:257 (sleep mutex:vm object)
25 3959 0 480 8 0 0 0 /usr/src/sys/kern/kern_resource.c:631 (spin mutex:process slock)
5 182 0 67 2 0 0 0 /usr/src/sys/kern/kern_sig.c:591 (spin mutex:sleepq chain)
11 43 0 7 6 0 0 0 /usr/src/sys/kern/vfs_default.c:442 (sleep mutex:vnode interlock)
6 150 0 52 2 0 0 0 /usr/src/sys/kern/vfs_subr.c:2131 (sleep mutex:vnode interlock)
18 673 0 71 9 0 0 0 /usr/src/sys/kern/uipc_socket.c:685 (sleep mutex:accept)
102 12942 0 530 24 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:232 (sleep mutex:so_rcv)
3215 3001405 0 47192 63 0 0 0 /usr/src/sys/vm/vm_map.c:3092 (sx:user map)
4 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:NFSNODE)
17 102 0 10 10 0 0 0 /usr/src/sys/kern/kern_fork.c:390 (sleep mutex:process lock)
9 50 0 10 5 0 0 0 /usr/src/sys/kern/kern_fork.c:391 (sleep mutex:process lock)
10 336 0 120 2 0 0 0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:6587 (rw:ipf token rwlock)
3054 16077774 3516313 469783 34 7 14825 16542 /usr/src/sys/kern/vfs_subr.c:2159 (sleep mutex:vnode interlock)
135 1820 117 37 49 3 2 1 /usr/src/sys/netinet/tcp_usrreq.c:694 (sleep mutex:tcp)
4 4 0 1 4 0 0 0 /usr/src/sys/kern/kern_resource.c:698 (spin mutex:process slock)
6 41 0 17 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:428 (sleep mutex:sleep mtxpool)
34 1370323 8870716 411996 3 21 481393 410555 /usr/src/sys/kern/sched_4bsd.c:1381 (spin mutex:sched lock)
1182 330039 18436 63730 5 0 711 478 /usr/src/sys/kern/vfs_subr.c:337 (sleep mutex:struct mount mtx)
90 825 0 16 51 0 1 0 /usr/src/sys/dev/mfi/mfi.c:912 (sleep mutex:MFI I/O lock)
4 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:inpcb)
9 272 0 99 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:441 (sleep mutex:sleep mtxpool)
5 54 0 16 3 0 0 0 /usr/src/sys/amd64/amd64/io_apic.c:212 (spin mutex:icu)
5 11 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:S VFS Cache)
4 6 0 2 3 0 0 0 /usr/src/sys/kern/vfs_subr.c:2210 (sleep mutex:vnode interlock)
5 181 0 70 2 0 0 0 /usr/src/sys/kern/kern_time.c:630 (spin mutex:process slock)
39 10389 383 2916 3 0 33 25 /usr/src/sys/kern/uipc_socket.c:1670 (sleep mutex:so_rcv)
68 68 0 1 68 0 0 0 /usr/src/sys/kern/kern_prot.c:1008 (sleep mutex:process lock)
25 165 0 10 16 0 0 0 /usr/src/sys/kern/kern_fork.c:457 (sleep mutex:process lock)
19 112 0 10 11 0 0 0 /usr/src/sys/kern/kern_fork.c:458 (sleep mutex:process lock)
5 67 0 16 4 0 0 0 /usr/src/sys/amd64/amd64/io_apic.c:229 (spin mutex:icu)
18 329 0 36 9 0 0 0 /usr/src/sys/kern/kern_event.c:993 (sleep mutex:sleep mtxpool)
12 28 0 8 3 0 0 0 /usr/src/sys/kern/vfs_subr.c:2232 (sleep mutex:vnode interlock)
122 3770 0 274 13 0 0 0 /usr/src/sys/amd64/amd64/pmap.c:2732 (sleep mutex:vm page queue mutex)
3 15 0 7 2 0 0 0 /usr/src/sys/fs/devfs/devfs_vnops.c:803 (sleep mutex:sleep mtxpool)
18 41 0 3 13 0 0 0 /usr/src/sys/kern/uipc_sockbuf.c:343 (sleep mutex:so_rcv)
12 205 0 54 3 0 0 0 /usr/src/sys/kern/kern_event.c:1664 (sleep mutex:so_rcv)
5 12 0 3 4 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:128 Bucket)
12 12 0 1 12 0 0 0 /usr/src/sys/kern/vfs_vnops.c:613 (lockmgr:devfs)
1567 12936770 2661709 215390 60 12 4383 14140 /usr/src/sys/kern/vfs_subr.c:2277 (sleep mutex:vnode interlock)
195 1961598 20879159 1146748 1 18 63631 988242 /usr/src/sys/kern/subr_turnstile.c:536 (spin mutex:turnstile chain)
3 25 0 10 2 0 0 0 /usr/src/sys/kern/kern_fork.c:556 (sleep mutex:session)
125 30888316 143499369 4969421 6 28 4824196 4557442 /usr/src/sys/kern/subr_turnstile.c:546 (spin mutex:turnstile chain)
12 688 0 248 2 0 0 0 /usr/src/sys/kern/vfs_bio.c:1353 (sleep mutex:buf queue lock)
3 9 0 4 2 0 0 0 /usr/src/sys/kern/kern_proc.c:263 (sleep mutex:process group)
21 364 0 34 10 0 0 0 /usr/src/sys/kern/vfs_cache.c:444 (lockmgr:ufs)
21 21 0 1 21 0 0 0 /usr/src/sys/kern/kern_prot.c:1085 (sleep mutex:process lock)
22 323 0 36 8 0 0 0 /usr/src/sys/kern/uipc_syscalls.c:371 (sleep mutex:accept)
36 253 0 10 25 0 0 0 /usr/src/sys/kern/kern_fork.c:547 (sleep mutex:process lock)
22 162 0 10 16 0 0 0 /usr/src/sys/kern/kern_fork.c:548 (sleep mutex:process lock)
2 5 0 2 2 0 0 0 /usr/src/sys/kern/uipc_syscalls.c:643 (sleep mutex:sleep mtxpool)
2 4 0 2 2 0 0 0 /usr/src/sys/kern/uipc_syscalls.c:648 (sleep mutex:sleep mtxpool)
1 3 0 2 1 0 0 0 /usr/src/sys/kern/kern_proc.c:308 (sleep mutex:process group)
4 24 0 10 2 0 0 0 /usr/src/sys/kern/kern_fork.c:574 (sleep mutex:ktrace)
3 8 0 3 2 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:FFS inode)
8 46 0 16 2 0 0 0 /usr/src/sys/kern/vfs_bio.c:340 (sleep mutex:runningbufspace lock)
3 6 0 2 3 0 0 0 /usr/src/sys/kern/kern_proc.c:325 (sleep mutex:process group)
2 5 0 3 1 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf_jumbo_9k)
15 5638 0 2020 2 0 0 0 /usr/src/sys/kern/uipc_socket.c:1826 (sleep mutex:so_rcv)
1837 498920 58 47192 10 0 3 2 /usr/src/sys/vm/vm_fault.c:293 (sleep mutex:vm object)
5 31 0 9 3 0 0 0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:pipe mutex)
3805 6501672 354014 63411 102 5 1129 1147 /usr/src/sys/kern/vfs_hash.c:71 (sleep mutex:vfs hash)
31 1642 0 219 7 0 0 0 /usr/src/sys/kern/vfs_bio.c:1468 (sleep mutex:buf queue lock)
4 9 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:udpcb)
5 43 0 18 2 0 0 0 /usr/src/sys/kern/kern_prot.c:1203 (sleep mutex:process lock)
70 1242 138 47 26 2 0 5 /usr/src/sys/netinet/tcp_usrreq.c:955 (sleep mutex:tcp)
22 62 0 4 15 0 0 0 /usr/src/sys/kern/kern_proc.c:399 (sleep mutex:process group)
12 37 0 4 9 0 0 0 /usr/src/sys/kern/kern_proc.c:400 (sleep mutex:process group)
3 42 0 21 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:676 (sleep mutex:sleep mtxpool)
4 9 0 3 3 0 0 0 /usr/src/sys/kern/uipc_socket.c:1888 (sleep mutex:so_rcv)
5 69 0 28 2 0 0 0 /usr/src/sys/vm/vm_object.c:1720 (sleep mutex:vm object_list)
4 25 0 10 2 0 0 0 /usr/src/sys/kern/kern_fork.c:676 (sleep mutex:process lock)
5 10 0 3 3 0 0 0 /usr/src/sys/kern/uipc_socket.c:1895 (sleep mutex:so_rcv)
5 20 0 7 2 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:365 (rw:unp_global_rwlock)
5 77 0 33 2 0 0 0 /usr/src/sys/kern/kern_descrip.c:409 (sx:filedesc structure)
11 81 0 10 8 0 0 0 /usr/src/sys/kern/kern_proc.c:425 (sleep mutex:process group)
418 3199 63 15 213 4 5 2 /usr/src/sys/amd64/amd64/pmap.c:2949 (sleep mutex:vm page queue mutex)
9 32 0 10 3 0 0 0 /usr/src/sys/kern/kern_fork.c:691 (sleep mutex:process lock)
51 644 0 16 40 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:549 (sleep mutex:inp)
9 29 0 4 7 0 0 0 /usr/src/sys/kern/kern_proc.c:457 (sleep mutex:process group)
2380 4079839 3754994 65658 62 57 1997 2784 /usr/src/sys/kern/vfs_lookup.c:411 (lockmgr:ufs)
5 16 0 4 4 0 0 0 /usr/src/sys/kern/kern_proc.c:476 (sleep mutex:process group)
9 58 0 10 5 0 0 0 /usr/src/sys/kern/kern_exit.c:297 (sx:proctree)
13 112 0 17 6 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:1014 (sleep mutex:udp)
11 481 0 179 2 0 0 0 /usr/src/sys/vm/vm_object.c:1214 (sleep mutex:vm object)
2 5 0 3 1 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:4096)
6 95 156 35 2 4 15 17 /usr/src/sys/kern/kern_resource.c:1041 (spin mutex:sched lock)
5 51 0 16 3 0 0 0 /usr/src/sys/kern/uipc_socket.c:961 (sleep mutex:so_snd)
4 11 0 3 3 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:1024)
38 500 0 35 14 0 0 0 /usr/src/sys/kern/kern_resource.c:1059 (spin mutex:process slock)
6 358 0 120 2 0 0 0 /usr/src/sys/netinet/ip_input.c:1086 (sleep mutex:ipqlock)
229 2126 0 523 4 0 0 0 /usr/src/sys/netinet/in_pcb.c:217 (sleep mutex:inp)
25 2555 0 449 5 0 0 0 /usr/src/sys/kern/vfs_bio.c:2460 (sleep mutex:vnode interlock)
15 150 0 17 8 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:1052 (sleep mutex:udp)
8 426 0 157 2 0 0 0 /usr/src/sys/vm/vm_object.c:1247 (sleep mutex:vm object)
30 540 0 60 9 0 0 0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:Syncer mtx)
5 12 0 3 4 0 0 0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:128)
5 10 0 4 2 0 0 0 /usr/src/sys/kern/uipc_socket.c:2016 (sleep mutex:so_rcv)
2 15 0 7 2 0 0 0 /usr/src/sys/kern/uipc_usrreq.c:498 (rw:unp_global_rwlock)
88 1068 0 16 66 0 0 0 /usr/src/sys/netinet/udp_usrreq.c:1072 (sleep mutex:udp)
6 45 0 13 3 0 0 0 /usr/src/sys/kern/vfs_bio.c:3213 (sleep mutex:vm object)
More information about the freebsd-stable
mailing list