2 x quad-core system is slower that 2 x dual core on FreeBSD

Alexey Popov lol at chistydom.ru
Tue Nov 20 00:36:42 PST 2007


Hi.

Kris Kennaway wrote:
> OK, but that is on 6.x and only (I guess) on the 8 core machine.  What 
> is needed is at least a comparison of the two machines under identical 
> conditions, and preferably on 7.0.
Right, that was 8-core on 6-STABLE.

> It looks like the most serious issue might be due to lockmgr contention, 
> but on 6.x lockmgr usage is not profiled directly (it only shows up via 
> secondary mutexes).  Comparing two 7.0 traces should help to shed more 
> light on the subject.
At this moment I have no 4-core servers running FreeBSD 7. Take a look 
at lock profiling of 8-core server on 7-STABLE in attach. If there's no 
evidence of the problem I'll try 7-STABLE on the production 4-core server.

Also could you explain what to look for in the lock profiling results? 
Does large "wait_total" values indicate problem or other columns???

With best regards,
Alexey Popov
-------------- next part --------------
debug.lock.prof.stats: 
   max        total   wait_total       count   avg wait_avg     cnt_hold     cnt_lock name
    10           56            0           7     8     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:512 (rw:unp_global_rwlock)
    18          165            0          17     9     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:1099 (sleep mutex:udp)
    73          579            0          16    36     0            0            0 /usr/src/sys/geom/geom_disk.c:198 (sleep mutex:g_disk_done)
    79          133            0           2    66     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:530 (rw:unp_global_rwlock)
    29          100            0           5    20     0            0            0 /usr/src/sys/vm/vm_object.c:1298 (sleep mutex:vm object)
    25           72            0           5    14     0            0            0 /usr/src/sys/vm/vm_object.c:1299 (sleep mutex:vm object)
    27        12013            0        4251     2     0            3            0 /usr/src/sys/net/if_ethersubr.c:405 (sleep mutex:em0)
    25          253            0          16    15     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:1113 (sleep mutex:udp)
  3215       250635            0         933   268     0            0            0 /usr/src/sys/vm/vm_map.c:2293 (sleep mutex:vm object)
     2            5            0           3     1     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:2048)
     8           43            0           7     6     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:557 (rw:unp_global_rwlock)
   117          415            0          10    41     0            0            0 /usr/src/sys/kern/kern_exit.c:428 (sx:proctree)
   813        21953            0        1551    14     0            0            0 /usr/src/sys/kern/subr_sleepqueue.c:390 (sleep mutex:process lock)
    12           21            0           2    10     0            0            0 /usr/src/sys/kern/vfs_bio.c:1738 (sleep mutex:buf queue lock)
     5           13            0           3     4     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:MAP ENTRY)
   140          140            0           1   140     0            0            0 /usr/src/sys/vm/vm_kern.c:296 (sleep mutex:system map)
     4           14            0           5     2     0            0            0 /usr/src/sys/vm/vm_object.c:1378 (sleep mutex:vm object)
    44          112            0           3    37     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:622 (rw:unp_global_rwlock)
   560          730            0          21    34     0            0            0 /usr/src/sys/kern/kern_descrip.c:665 (sx:filedesc structure)
     4           34            0          14     2     0            0            0 /usr/src/sys/kern/kern_exec.c:854 (sleep mutex:vm page queue mutex)
    18         3477            0        1242     2     0            0            0 /usr/src/sys/kern/uipc_socket.c:1136 (sleep mutex:so_snd)
    11          419            0          60     6     0            0            0 /usr/src/sys/kern/vfs_bio.c:2107 (sleep mutex:buffer daemon lock)
     5           39            0          14     2     0            0            0 /usr/src/sys/kern/kern_exec.c:876 (sleep mutex:vm page queue mutex)
     7          108            0          23     4     0            0            0 /usr/src/sys/vm/vm_fault.c:649 (sleep mutex:vm object)
     5          105            0          36     2     0            0            0 /usr/src/sys/kern/uipc_sockbuf.c:827 (sleep mutex:so_rcv)
     1            1            0           1     1     0            0            0 /usr/src/sys/vm/uma_core.c:889 (sleep mutex:128 Bucket)
   118         8011            0         423    18     0            0            0 /usr/src/sys/vm/vm_fault.c:667 (sleep mutex:vm object)
     3            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf_jumbo_pagesize)
     3            7            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UPCALL)
    56           56            0           1    56     0            0            0 /usr/src/sys/fs/fifofs/fifo_vnops.c:716 (sleep mutex:Giant)
    55          343            0           8    42     0            0            0 /usr/src/sys/kern/kern_exec.c:494 (lockmgr:ufs)
     5           72            0          27     2     0            0            0 /usr/src/sys/netinet/ip_input.c:1218 (sleep mutex:rtentry)
   127         1467            0          16    91     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:843 (sleep mutex:inp)
    81           81            0           1    81     0            0            0 /usr/src/sys/fs/fifofs/fifo_vnops.c:738 (sleep mutex:Giant)
     5           22            0           7     3     0            0            0 /usr/src/sys/kern/uipc_socket.c:2230 (sleep mutex:so_rcv)
    11         1283            0         485     2     0            0            0 /usr/src/sys/vm/vm_mmap.c:1318 (sleep mutex:process lock)
    15          123            0          12    10     0            0            0 /usr/src/sys/kern/vfs_bio.c:3426 (sleep mutex:vm object)
     3           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ACL UMA zone)
    17         1838            0         260     7     0            0            0 /usr/src/sys/vm/vm_map.c:2499 (sleep mutex:vm object)
     5           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:L VFS Cache)
     3            4            0           2     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:260 (sleep mutex:process lock)
    11         1068            0         379     2     0            0            0 /usr/src/sys/vm/vm_fault.c:770 (sleep mutex:vm object)
    26        49269        52123       14029     3     3         5404         6646 /usr/src/sys/kern/kern_intr.c:706 (spin mutex:sched lock)
    25          417            0          38    10     0            0            0 /usr/src/sys/kern/vfs_bio.c:3498 (sleep mutex:vm object)
     5           58            0          18     3     0            0            0 /usr/src/sys/kern/kern_umtx.c:2703 (spin mutex:umtx lock)
     3            6            0           2     3     0            0            0 /usr/src/sys/kern/vfs_cluster.c:910 (sleep mutex:vm object)
     5           50            0          17     2     0            0            0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:tcpcb)
     5           48            0          16     3     0            0            0 /usr/src/sys/kern/kern_event.c:758 (sleep mutex:protect sysfilt_ops)
     5           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UMA Kegs)
    20          124            0          11    11     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:1421 (sleep mutex:tcp)
   389        16941          128          80   211     1            1            1 /usr/src/sys/kern/kern_exit.c:684 (sx:proctree)
    10           43            0          16     2     0            0            0 /usr/src/sys/kern/kern_event.c:774 (sleep mutex:protect sysfilt_ops)
     5          470            0         193     2     0            0            0 /usr/src/sys/kern/vfs_vnops.c:508 (sleep mutex:sleep mtxpool)
     3            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:PROC)
     3            8            0           4     2     0            0            0 /usr/src/sys/kern/tty.c:850 (sleep mutex:process lock)
  1024      1484197        92173      279122     5     0         3689         5478 /usr/src/sys/kern/vfs_cache.c:327 (sleep mutex:Name Cache)
     6          513            0         193     2     0            0            0 /usr/src/sys/kern/vfs_vnops.c:529 (sleep mutex:sleep mtxpool)
  3072         8093            0         137    59     0            0            0 /usr/src/sys/vm/vm_object.c:1627 (sleep mutex:vm object)
    12         8938          162        3618     2     0            1            2 /usr/src/sys/kern/vfs_subr.c:2946 (sleep mutex:vnode interlock)
     3           20            0           8     2     0            0            0 /usr/src/sys/kern/imgact_elf.c:763 (sleep mutex:process lock)
     5          110            0          47     2     0            0            0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:so_rcv)
     5            5            0           1     5     0            0            0 /usr/src/sys/kern/kern_descrip.c:385 (sleep mutex:process lock)
  1537       421522           80       47192     8     0            2            3 /usr/src/sys/vm/vm_fault.c:886 (sleep mutex:vm object)
     2            4            0           2     2     0            0            0 /usr/src/sys/vm/vm_map.c:2631 (sleep mutex:vm object)
     4           13            0           4     3     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:902 (sleep mutex:pmap)
     3            7            0           3     2     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:6023 (sleep mutex:Softdep Lock)
    31          207            0          10    20     0            0            0 /usr/src/sys/kern/kern_exit.c:519 (spin mutex:process slock)
  1125         1304            0           6   217     0            0            0 /usr/src/sys/ufs/ffs/ffs_vnops.c:214 (sleep mutex:vnode interlock)
    41          267            0          10    26     0            0            0 /usr/src/sys/kern/kern_exit.c:522 (spin mutex:process slock)
     5           46            0          17     2     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:1053 (sleep mutex:inp)
     2            7            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:TURNSTILE)
     5           11            0           3     3     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:722 (sleep mutex:so_rcv)
     2            2            0           1     2     0            0            0 /usr/src/sys/vm/vm_pager.c:336 (sleep mutex:pbuf mutex)
     2            5            0           3     1     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:pipe)
    82          985            0          16    61     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:1073 (sleep mutex:inp)
     2            7            0           4     1     0            0            0 /usr/src/sys/vm/vm_object.c:1695 (sleep mutex:vm object)
     7            7            0           1     7     0            0            0 /usr/src/sys/ufs/ffs/ffs_vnops.c:240 (sleep mutex:vnode interlock)
    31           53            0           2    26     0            0            0 /usr/src/sys/kern/vfs_bio.c:3641 (sleep mutex:vm object)
    31          142            0           8    17     0            0            0 /usr/src/sys/kern/kern_exec.c:749 (lockmgr:ufs)
     3            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:umtx pi)
   890        16000          257        2778     5     0           16           19 /usr/src/sys/kern/uipc_socket.c:2469 (sleep mutex:so_rcv)
     3           36            0          16     2     0            0            0 /usr/src/sys/kern/kern_event.c:1495 (sx:filedesc structure)
     5            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mt_zone)
    12         8840          281        3618     2     0            1            2 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1226 (sleep mutex:vnode interlock)
    18           18            0           1    18     0            0            0 /usr/src/sys/kern/kern_resource.c:186 (sleep mutex:process lock)
    14           71            0          17     4     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:1100 (sleep mutex:inp)
  7337        46187          293         125   369     2           15           11 /usr/src/sys/kern/kern_synch.c:235 (sleep mutex:Giant)
     5           91            0          30     3     0            0            0 /usr/src/sys/kern/uipc_sockbuf.c:96 (sleep mutex:so_snd)
   788       157277          374       48292     3     0           15           17 /usr/src/sys/vm/vm_fault.c:959 (sleep mutex:vm object)
    20          165            0          16    10     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:1114 (sleep mutex:inp)
    17         1073            0         172     6     0            0            0 /usr/src/sys/vm/vm_fault.c:966 (sleep mutex:vm object)
   199          382            0           2   191     0            0            0 /usr/src/sys/kern/vfs_bio.c:3680 (sleep mutex:vm object)
    11           44            0          10     4     0            0            0 /usr/src/sys/kern/kern_sig.c:3289 (sleep mutex:sigacts)
    16           32            0           7     4     0            0            0 /usr/src/sys/ufs/ffs/ffs_vnops.c:289 (sleep mutex:vnode interlock)
    36         2034            0         545     3     0            0            0 /usr/src/sys/kern/kern_timeout.c:241 (sleep mutex:tcp_sc_head)
     3           18            0           8     2     0            0            0 /usr/src/sys/kern/kern_sig.c:3299 (sleep mutex:sigacts)
     5            5            0           1     5     0            0            0 /usr/src/sys/vm/vm_pager.c:413 (sleep mutex:pbuf mutex)
  1770        67722            0       25037     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:1025 (sx:filedesc structure)
     4           35            0          12     2     0            0            0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1269 (sleep mutex:vnode interlock)
    18          143            0          16     8     0            0            0 /usr/src/sys/kern/uipc_socket.c:2528 (sleep mutex:so_rcv)
    31           31            0           1    31     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:810 (sleep mutex:so_rcv)
    11           76           59          10     7     5            5            8 /usr/src/sys/kern/kern_thread.c:467 (spin mutex:sched lock)
     5           59            0          19     3     0            0            0 /usr/src/sys/kern/kern_prot.c:1874 (sleep mutex:process lock)
    17          698           40         215     3     0            2            5 /usr/src/sys/kern/vfs_cache.c:500 (sleep mutex:Name Cache)
  1933       898106         9292      157103     5     0          767          273 /usr/src/sys/kern/vfs_vnops.c:807 (sleep mutex:vnode interlock)
     1            1            0           1     1     0            0            0 /usr/src/sys/kern/kern_prot.c:1933 (sleep mutex:session)
    78         4796            0        1585     3     0            0            0 /usr/src/sys/kern/sys_generic.c:893 (sleep mutex:process lock)
     3           21           35          10     2     3            4            5 /usr/src/sys/kern/kern_thread.c:501 (spin mutex:sched lock)
    22        17432            0        6393     2     0            0            0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:2282 (rw:ipf cache rwlock)
     5           17            0           6     2     0            0            0 /usr/src/sys/netinet/in_pcb.c:789 (sleep mutex:inp)
   138        10295         2002        1718     5     1           43           87 /usr/src/sys/netinet/tcp_input.c:1191 (sleep mutex:so_rcv)
     3            7            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:socket)
     3            8            0           3     2     0            0            0 /usr/src/sys/kern/vfs_syscalls.c:3951 (sleep mutex:sleep mtxpool)
    37           44            0           3    14     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:873 (sleep mutex:so_rcv)
     4           15            0           4     3     0            0            0 /usr/src/sys/netinet/in_pcb.c:808 (sleep mutex:inp)
     6            6            0           1     6     0            0            0 /usr/src/sys/kern/kern_prot.c:1932 (sleep mutex:process lock)
     5           13            0           4     3     0            0            0 /usr/src/sys/vm/uma_core.c:889 (sleep mutex:MAP ENTRY)
    13          102            0          12     8     0            0            0 /usr/src/sys/kern/vfs_subr.c:3162 (sleep mutex:vnode interlock)
    11          357            0         120     2     0            0            0 /usr/src/sys/contrib/ipfilter/netinet/ip_frag.c:799 (rw:ipf IP NAT-Frag rwlock)
    23         3800            0        1448     2     0            0            0 /usr/src/sys/kern/kern_exit.c:712 (spin mutex:process slock)
   106        15879            0         530    29     0            0            0 /usr/src/sys/kern/uipc_sockbuf.c:231 (sleep mutex:so_snd)
     9          136         1190          24     5    49            0            2 /usr/src/sys/sys/buf.h:276 (sleep mutex:lockbuilder mtxpool)
     3           17            0           7     2     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:3742 (sleep mutex:process lock)
    10          211            0          38     5     0            0            0 /usr/src/sys/kern/subr_eventhandler.c:212 (sleep mutex:eventhandler)
     5           34            0          12     2     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:6244 (sleep mutex:Softdep Lock)
    36         2220            0         699     3     0            0            0 /usr/src/sys/vm/vm_object.c:1884 (sleep mutex:vm object)
  3275       294404            0         414   711     0            0            0 /usr/src/sys/vm/vm_mmap.c:557 (sx:user map)
    38         3642           18         449     8     0            0            1 /usr/src/sys/sys/buf.h:296 (sleep mutex:lockbuilder mtxpool)
    11          370            0         134     2     0            0            0 /usr/src/sys/kern/sys_socket.c:121 (sleep mutex:so_rcv)
    11          365            0         135     2     0            0            0 /usr/src/sys/kern/sys_socket.c:147 (sleep mutex:so_rcv)
    12         2447            0         721     3     0            0            0 /usr/src/sys/netinet/tcp_timewait.c:643 (sleep mutex:inp)
    11          362            0         135     2     0            0            0 /usr/src/sys/kern/sys_socket.c:150 (sleep mutex:so_rcv)
    10          106            0          16     6     0            0            0 /usr/src/sys/kern/uipc_socket.c:2680 (sleep mutex:so_rcv)
     5          149            0          49     3     0            0            0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:mbuf)
     3           45            0          21     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:659 (sleep mutex:process lock)
     5           11            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:16 Bucket)
     4           20            0           7     2     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:3815 (sleep mutex:process lock)
   166        66254            0         972    68     0            0            0 /usr/src/sys/netinet/tcp_timer.c:169 (sleep mutex:inp)
    41           41            0           1    41     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:1178 (rw:unp_global_rwlock)
    11         6503           29        2472     2     0            0            2 /usr/src/sys/kern/kern_resource.c:1240 (sleep mutex:sleep mtxpool)
    22          155            0          11    14     0            0            0 /usr/src/sys/netinet/tcp_subr.c:1379 (sleep mutex:isn_mtx)
     4           11            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:64)
     4           26            0          10     2     0            0            0 /usr/src/sys/kern/kern_exit.c:215 (sleep mutex:p_peers)
    26        54225        87944       13349     4     6         7262         8564 /usr/src/sys/kern/kern_intr.c:1131 (spin mutex:sched lock)
    12         6364           19        2522     2     0            1            1 /usr/src/sys/kern/kern_resource.c:1266 (sleep mutex:sleep mtxpool)
  1054        13021            0          24   542     0            0            0 /usr/src/sys/sys/buf.h:280 (lockmgr:bufwait)
     6            6            0           1     6     0            0            0 /usr/src/sys/kern/kern_resource.c:1277 (sleep mutex:sleep mtxpool)
    12          159            0          62     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:1544 (sleep mutex:sleep mtxpool)
   865        18644            0         449    41     0            0            0 /usr/src/sys/sys/buf.h:301 (lockmgr:getblk)
    40        50477            0        6001     8     0            0            0 /usr/src/sys/netinet/tcp_subr.c:1426 (sleep mutex:isn_mtx)
     4           72            0          24     3     0            0            0 /usr/src/sys/kern/kern_exec.c:903 (sleep mutex:process_exec)
     1            2            0           2     1     0            0            0 /usr/src/sys/kern/vfs_syscalls.c:3864 (sx:filedesc structure)
     4            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:KNOTE)
     4           53            0          22     2     0            0            0 /usr/src/sys/kern/kern_resource.c:1314 (sleep mutex:sleep mtxpool)
     9           36            0          10     3     0            0            0 /usr/src/sys/kern/sysv_sem.c:1288 (sleep mutex:sem)
     5           30            0          10     3     0            0            0 /usr/src/sys/kern/kern_exit.c:284 (sleep mutex:p_peers)
    12           63            0           7     9     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:1268 (rw:unp_global_rwlock)
     4           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:512)
    33         4035           16        1516     2     0            3            1 /usr/src/sys/netinet/tcp_subr.c:1606 (sleep mutex:rtentry)
    78         7551            0        2794     2     0            0            0 /usr/src/sys/kern/kern_resource.c:1339 (sleep mutex:sleep mtxpool)
     5            5            0           1     5     0            0            0 /usr/src/sys/vm/uma_core.c:2195 (sleep mutex:UMA Slabs)
    18           25            0           3     8     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:NAMEI)
     4           30            0          11     2     0            0            0 /usr/src/sys/kern/uipc_socket.c:2820 (sleep mutex:so_rcv)
   452        39938         9430       14524     2     0          124          424 /usr/src/sys/kern/vfs_cache.c:774 (sleep mutex:Name Cache)
    20         4204            0        1021     4     0            0            0 /usr/src/sys/kern/uipc_socket.c:2832 (sleep mutex:so_rcv)
  1339        50216          129        8527     5     0            6            6 /usr/src/sys/net/route.c:147 (sleep mutex:radix node head)
    61         1403            0         494     2     0            0            0 /usr/src/sys/kern/uipc_socket.c:2846 (sleep mutex:so_rcv)
     4            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:KMAP ENTRY)
     5          323            0         120     2     0            0            0 /usr/src/sys/netinet/igmp.c:446 (sleep mutex:igmp_mtx)
    30          194            0          33     5     0            1            0 /usr/src/sys/kern/uipc_socket.c:2863 (sleep mutex:so_rcv)
   416     55229737     13245866     1108825    49    11       957433       991447 /usr/src/sys/kern/subr_sleepqueue.c:232 (spin mutex:sleepq chain)
     5          108            0          36     3     0            0            0 /usr/src/sys/kern/uipc_socket.c:2875 (sleep mutex:so_rcv)
    77          294            0           5    58     0            0            0 /usr/src/sys/kern/vfs_lookup.c:411 (lockmgr:devfs)
     5          103            0          38     2     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:972 (sleep mutex:so_rcv)
   910       168337            0       15207    11     0            0            0 /usr/src/sys/kern/kern_descrip.c:1394 (sx:filedesc structure)
    12          176            5          59     2     0            1            2 /usr/src/sys/kern/uipc_socket.c:2894 (sleep mutex:so_rcv)
     5            5            0           1     5     0            0            0 /usr/src/sys/kern/vfs_cluster.c:753 (sleep mutex:vnode interlock)
     3           20            0          10     2     0            0            0 /usr/src/sys/kern/kern_exit.c:470 (sleep mutex:sigacts)
     4            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UMA Zones)
     1            3            0           2     1     0            0            0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1656 (sleep mutex:vnode interlock)
   157        33577            0       13815     2     0            0            0 /usr/src/sys/vm/vnode_pager.c:110 (sleep mutex:vm object)
     4           19            0           7     2     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:513 (sleep mutex:unp_mtx)
    17          157            0          18     8     0            0            0 /usr/src/sys/kern/kern_resource.c:1206 (sleep mutex:uidinfo hash)
    29        41876        29826       11242     3     2         3898         3653 /usr/src/sys/kern/kern_switch.c:182 (spin mutex:sched lock)
     7            7            0           1     7     0            0            0 /usr/src/sys/kern/kern_resource.c:1211 (sleep mutex:uidinfo hash)
    38        23181           49        6027     3     0            0            2 /usr/src/sys/dev/em/if_em.c:4305 (sleep mutex:em0)
    19          132            0          10    13     0            0            0 /usr/src/sys/kern/kern_descrip.c:1429 (sx:filedesc structure)
    22         9489        14390        3404     2     4           11          305 /usr/src/sys/kern/sched_4bsd.c:385 (spin mutex:sleepq chain)
   576         4177            0        1236     3     0            0            0 /usr/src/sys/kern/uipc_sockbuf.c:531 (sleep mutex:so_snd)
     3            5            0           2     2     0            0            0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1678 (sleep mutex:vnode interlock)
    74          122            0           2    61     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:533 (sleep mutex:unp_mtx)
    68          111            0           2    55     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:536 (sleep mutex:unp_mtx)
     4           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:SWAPMETA)
     8          208            6          64     3     0            0            2 /usr/src/sys/kern/vfs_bio.c:3002 (sleep mutex:bdone lock)
     5           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:32 Bucket)
     6          422            0         120     3     0            0            0 /usr/src/sys/contrib/ipfilter/netinet/ip_auth.c:563 (rw:ipf IP User-Auth rwlock)
     4            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:DIRHASH)
   118          265            0           5    53     0            0            0 /usr/src/sys/netinet/tcp_timer.c:438 (sleep mutex:inp)
    11           33            0          11     3     0            0            0 /usr/src/sys/kern/kern_resource.c:677 (sleep mutex:process lock)
     4           23            0           9     2     0            0            0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1729 (sleep mutex:vnode interlock)
     8            8            0           1     8     0            0            0 /usr/src/sys/kern/vfs_cluster.c:841 (sleep mutex:vnode interlock)
     4            4            0           1     4     0            0            0 /usr/src/sys/kern/kern_resource.c:1276 (sleep mutex:uidinfo hash)
     2            9            0           4     2     0            0            0 /usr/src/sys/kern/vfs_bio.c:1320 (sleep mutex:vm page queue mutex)
     3           20            0           8     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:1502 (sx:filedesc structure)
    22          381            0          66     5     0            0            0 /usr/src/sys/kern/kern_sysctl.c:1396 (sx:sysctl lock)
    20           87            0           8    10     0            0            0 /usr/src/sys/kern/kern_time.c:611 (sleep mutex:process lock)
    10          399            0         158     2     0            0            0 /usr/src/sys/vm/vnode_pager.c:208 (sleep mutex:vm object)
    22        19817         8844        7311     2     1         1444          783 /usr/src/sys/kern/kern_switch.c:272 (spin mutex:sched lock)
  3533        34520            0          24  1438     0           10            0 /usr/src/sys/kern/vfs_mount.c:2029 (sleep mutex:struct mount mtx)
    13         5235         8321        1872     2     4          425          901 /usr/src/sys/kern/subr_turnstile.c:203 (spin mutex:sched lock)
    21         1435            0         530     2     0            0            0 /usr/src/sys/kern/uipc_socket.c:280 (sleep mutex:so_glabel)
    40           95            0           3    31     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:623 (sleep mutex:unp_mtx)
    35           79            0           3    26     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:626 (sleep mutex:unp_mtx)
     5            7            0           2     3     0            0            0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1773 (sleep mutex:vnode interlock)
     5         1008            0         352     2     0            0            0 /usr/src/sys/kern/uipc_socket.c:299 (sleep mutex:so_glabel)
     4           27            0          10     2     0            0            0 /usr/src/sys/kern/kern_exit.c:409 (sx:allproc)
   108        38884          714        1551    25     0           59           26 /usr/src/sys/kern/subr_sleepqueue.c:409 (spin mutex:sleepq chain)
     6           31            0          10     3     0            0            0 /usr/src/sys/kern/kern_descrip.c:1552 (sx:filedesc structure)
    17           80           48          24     3     2            1            4 /usr/src/sys/kern/vfs_mount.c:2067 (sleep mutex:struct mount mtx)
    74         3029            0          63    48     0            0            0 /usr/src/sys/kern/kern_time.c:657 (sleep mutex:process lock)
     1            1            0           1     1     0            0            0 /usr/src/sys/vm/uma_core.c:2436 (sleep mutex:UMA Slabs)
     3            3            0           1     3     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:1332 (sleep mutex:so_rcv)
     4            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UMA RCntSlabs)
    17         1892            0         739     2     0            0            0 /usr/src/sys/kern/vfs_subr.c:2985 (sleep mutex:vnode_free_list)
     4           12            0           3     4     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ata_composite)
     4            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:Files)
    12         1911            0         739     2     0            0            0 /usr/src/sys/kern/vfs_subr.c:3013 (sleep mutex:vnode_free_list)
     5           26            0          12     2     0            0            0 /usr/src/sys/kern/kern_resource.c:796 (sleep mutex:process lock)
     5           22            0           7     3     0            0            0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:inpcb)
     2            7            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:VMSPACE)
    22         2859            0        1071     2     0            0            0 /usr/src/sys/kern/uipc_sockbuf.c:291 (sleep mutex:process lock)
    16           20            0           3     6     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:726 (sleep mutex:unp_mtx)
    67          967            0          35    27     0            0            0 /usr/src/sys/kern/imgact_elf.c:322 (sx:user map)
     5            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:16)
     7          145            0          45     3     0            0            0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:mbuf)
     4           29            0          10     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:1656 (sx:filedesc structure)
     6          108            0          35     3     0            0            0 /usr/src/sys/vm/vnode_pager.c:361 (sleep mutex:vm object)
    14           93            0          10     9     0            0            0 /usr/src/sys/kern/kern_descrip.c:1670 (sx:filedesc structure)
     5           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UMA Hash)
    25           25            0           1    25     0            0            0 /usr/src/sys/kern/vfs_bio.c:1531 (sleep mutex:vm page queue mutex)
    36           36            0           1    36     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:804 (sleep mutex:unp_mtx)
     5           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:tcptw)
  1210        40478           24       15322     2     0            0            2 /usr/src/sys/kern/kern_descrip.c:2007 (sleep mutex:sleep mtxpool)
     4           39            0          15     2     0            0            0 /usr/src/sys/kern/vfs_bio.c:294 (sleep mutex:needsbuffer lock)
    11           75            0          12     6     0            1            0 /usr/src/sys/vm/vm_pageout.c:1446 (sleep mutex:vm page queue free mutex)
     3            5            0           2     2     0            0            0 /usr/src/sys/dev/mfi/mfi.c:2497 (sleep mutex:MFI I/O lock)
    10         1124            0         430     2     0            0            0 /usr/src/sys/kern/vfs_bio.c:321 (sleep mutex:needsbuffer lock)
    43          546            0          16    34     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:241 (sleep mutex:so_rcv)
    48          837            0          70    11     0            0            0 /usr/src/sys/kern/kern_resource.c:958 (sleep mutex:process lock)
    57           96            0           3    32     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:872 (sleep mutex:unp_mtx)
    22          640            0          67     9     0            0            0 /usr/src/sys/kern/kern_sig.c:2289 (spin mutex:sleepq chain)
    22         3451            0        1295     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:1785 (sx:filedesc structure)
     7          993            0         394     2     0            0            0 /usr/src/sys/kern/vfs_bio.c:363 (sleep mutex:needsbuffer lock)
  2143        16776          113         120   139     0            1            3 /usr/src/sys/netinet/tcp_timer.c:128 (sleep mutex:tcp)
     3            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:itimer)
     5           23            0           8     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:1809 (sx:filedesc structure)
  2698      2728989        26170      276752     9     0          395          945 /usr/src/sys/kern/vfs_cache.c:440 (sleep mutex:vnode interlock)
   174         7682         3499         605    12     5           64           61 /usr/src/sys/kern/sys_generic.c:750 (sleep mutex:sellck)
   632        37764            0       15216     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:1287 (sleep mutex:process lock)
    29         5347          198         972     5     0            1            3 /usr/src/sys/netinet/tcp_timer.c:155 (sleep mutex:tcp)
   123        19492         3735         609    32     6           63           62 /usr/src/sys/kern/sys_generic.c:759 (sleep mutex:sellck)
     5           36            0          14     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:1834 (sx:filedesc structure)
     5            5            0           1     5     0            0            0 /usr/src/sys/vm/vm_pager.c:475 (sleep mutex:vnode interlock)
   790       120453            5       49016     2     0            1            1 /usr/src/sys/kern/kern_descrip.c:2139 (sleep mutex:sleep mtxpool)
    17          117            0          10    11     0            0            0 /usr/src/sys/kern/kern_exit.c:296 (sleep mutex:Giant)
     5           33            0          12     2     0            0            0 /usr/src/sys/vm/vm_pageout.c:1478 (sleep mutex:vm page queue mutex)
  7297        41165            0          12  3430     0            0            0 /usr/src/sys/kern/vfs_subr.c:1638 (lockmgr:syncer)
    10           30            0          10     3     0            0            0 /usr/src/sys/kern/kern_exit.c:747 (sx:allproc)
    15          120            0          14     8     0            0            0 /usr/src/sys/kern/kern_exec.c:820 (sleep mutex:vm object)
   170        27362            0         495    55     0            0            0 /usr/src/sys/netinet/tcp_syncache.c:648 (sleep mutex:inp)
    30           30            0           1    30     0            0            0 /usr/src/sys/kern/subr_sleepqueue.c:754 (spin mutex:sleepq chain)
  1367        85932         7908       15207     5     0           13           11 /usr/src/sys/kern/kern_descrip.c:1366 (sx:filelist lock)
     5           43            0          18     2     0            0            0 /usr/src/sys/kern/vfs_bio.c:3841 (sleep mutex:vnode interlock)
     5           61            0          20     3     0            0            0 /usr/src/sys/kern/vfs_bio.c:3851 (sleep mutex:vnode interlock)
    12           22            0           2    11     0            0            0 /usr/src/sys/kern/kern_proc.c:304 (sleep mutex:Giant)
    12           82            0          10     8     0            0            0 /usr/src/sys/kern/kern_exit.c:138 (sleep mutex:process lock)
    79        34309           55        3216    10     0           90            2 /usr/src/sys/kern/subr_sleepqueue.c:781 (spin mutex:sleepq chain)
    12          436            0         120     3     0            0            0 /usr/src/sys/contrib/ipfilter/netinet/ip_nat.c:4446 (rw:ipf IP NAT rwlock)
    17           28            0           2    14     0            0            0 /usr/src/sys/kern/kern_proc.c:320 (sleep mutex:Giant)
     5           26            0          10     2     0            0            0 /usr/src/sys/kern/kern_exit.c:799 (sx:allproc)
     5           98            0          35     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:2225 (sleep mutex:sleep mtxpool)
     5           93            0          35     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:2239 (sleep mutex:sleep mtxpool)
     5           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:unpcb)
     4           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:32)
    13           57            0          15     3     0            0            0 /usr/src/sys/vm/vm_kern.c:440 (sx:user map)
     5          215            0          60     3     0            0            0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:buffer daemon lock)
    45           77            0           2    38     0            0            0 /usr/src/sys/kern/kern_prot.c:325 (sx:proctree)
   133        16420            5         482    34     0            0            1 /usr/src/sys/amd64/amd64/pmap.c:1959 (sleep mutex:pmap)
     9            9            0           1     9     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:733 (sleep mutex:so_snd)
  1123        24802          431        1585    15     0            3            9 /usr/src/sys/kern/sys_generic.c:924 (sleep mutex:sellck)
   161         1784            0          15   118     0            0            0 /usr/src/sys/vm/vm_kern.c:469 (sx:user map)
    12          229            0          70     3     0            0            0 /usr/src/sys/kern/kern_lockf.c:192 (sleep mutex:vnode interlock)
  1228        32787           88        2778    11     0           24           11 /usr/src/sys/kern/uipc_socket.c:2468 (sleep mutex:so_snd)
    31           85            0           4    21     0            0            0 /usr/src/sys/kern/kern_proc.c:398 (sleep mutex:Giant)
  1075        52892          372        2446    21     0           34           11 /usr/src/sys/kern/sys_generic.c:933 (sleep mutex:sellck)
     4            6            0           2     3     0            0            0 /usr/src/sys/vm/uma_core.c:2195 (sleep mutex:128 Bucket)
    11           33            0          10     3     0            0            0 /usr/src/sys/kern/kern_exit.c:239 (sleep mutex:process lock)
    18          132            0          10    13     0            0            0 /usr/src/sys/kern/kern_proc.c:424 (sleep mutex:Giant)
     2           13            0           7     1     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:1135 (sleep mutex:unp_mtx)
    21         4820         9516        1914     2     4          562          978 /usr/src/sys/kern/subr_turnstile.c:728 (spin mutex:sched lock)
    66          120            0           2    60     0            0            0 /usr/src/sys/kern/kern_prot.c:384 (sx:proctree)
   210     20049818    258237284     1017559    19   253        82293      1011233 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:lockbuilder mtxpool)
    19          473            0         127     3     0            0            0 /usr/src/sys/kern/kern_sig.c:761 (spin mutex:process slock)
    14           50            0           4    12     0            1            0 /usr/src/sys/kern/kern_proc.c:456 (sleep mutex:Giant)
     4           12            0           4     3     0            0            0 /usr/src/sys/kern/kern_proc.c:242 (sleep mutex:process lock)
    24       112788            0       43460     2     0            0            0 /usr/src/sys/kern/kern_clock.c:219 (spin mutex:process slock)
     5           49            0          13     3     0            0            0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:inpcb)
    29       121765       163587       48015     2     3        15449        21288 /usr/src/sys/kern/kern_clock.c:224 (spin mutex:sched lock)
    23        15715           36        6001     2     0            0           10 /usr/src/sys/kern/kern_timeout.c:191 (spin mutex:callout)
     5           48            0          16     3     0            0            0 /usr/src/sys/kern/kern_event.c:841 (sleep mutex:kqueue)
     3            5            0           2     2     0            0            0 /usr/src/sys/kern/kern_exit.c:349 (sleep mutex:session)
    10           41            0           5     8     0            0            0 /usr/src/sys/netinet/tcp_timer.c:423 (sleep mutex:tcp)
     3            6            0           2     3     0            0            0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:MAP ENTRY)
     4            6            0           2     3     0            0            0 /usr/src/sys/kern/kern_proc.c:322 (sleep mutex:session)
     4           15            0           5     3     0            0            0 /usr/src/sys/kern/tty.c:2081 (sleep mutex:process lock)
    72        12092          168        4251     2     0            0           10 /usr/src/sys/dev/em/if_em.c:907 (sleep mutex:em0)
   560         1168            0           3   389     0            0            0 /usr/src/sys/kern/vfs_subr.c:1638 (lockmgr:ufs)
     1            3            0           2     1     0            0            0 /usr/src/sys/kern/kern_proc.c:305 (sleep mutex:process lock)
   165        29014            0        1616    17     0            0            0 /usr/src/sys/netinet/tcp_input.c:2248 (sleep mutex:so_rcv)
    10           27            0           3     9     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:895 (sleep mutex:so_snd)
    19         3268          160        1201     2     0            4            6 /usr/src/sys/kern/sys_generic.c:1079 (sleep mutex:sellck)
    33           33            0           1    33     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:1250 (sleep mutex:unp_mtx)
    25           25            0           1    25     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:1251 (sleep mutex:unp_mtx)
    79        40082          778        4251     9     0           41           38 /usr/src/sys/dev/em/if_em.c:938 (sleep mutex:em0)
   224        17452           90        1414    12     0            4            6 /usr/src/sys/geom/geom_io.c:68 (sleep mutex:bio queue)
     9          186            0          46     4     0            0            0 /usr/src/sys/kern/sys_pipe.c:572 (sleep mutex:pipe mutex)
    84         1755            0         178     9     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2140 (sleep mutex:pmap)
     4           20            0           7     2     0            0            0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:mbuf_jumbo_pagesize)
     5           21            0           7     3     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:1269 (sleep mutex:unp_mtx)
     4           27            0          10     2     0            0            0 /usr/src/sys/kern/kern_exit.c:399 (sleep mutex:process lock)
     3           13            0           6     2     0            0            0 /usr/src/sys/ufs/ufs/ufs_vnops.c:175 (sleep mutex:vnode interlock)
    82        49803          650        3600    13     0           14           21 /usr/src/sys/kern/kern_timeout.c:241 (sleep mutex:Giant)
    24        78334         5370       28112     2     0          475         1134 /usr/src/sys/kern/kern_timeout.c:296 (spin mutex:callout)
    10           90            0          16     5     0            0            0 /usr/src/sys/kern/kern_event.c:947 (sleep mutex:kqueue)
     7           24            0           5     4     0            0            0 /usr/src/sys/kern/sys_pipe.c:609 (sleep mutex:pipe mutex)
   108        23828          849        1430    16     0            8           23 /usr/src/sys/kern/sys_generic.c:1127 (sleep mutex:sellck)
    13         4261          100        1568     2     0           19           23 /usr/src/sys/kern/subr_turnstile.c:710 (spin mutex:td_contested)
    42        13330         5483        1568     8     3          433          710 /usr/src/sys/kern/subr_turnstile.c:894 (spin mutex:sched lock)
     6          877            0         346     2     0            0            0 /usr/src/sys/kern/subr_turnstile.c:718 (spin mutex:td_contested)
     9            9            0           1     9     0            0            0 /usr/src/sys/kern/kern_exit.c:434 (sleep mutex:process lock)
    13          386            0         135     2     0            0            0 /usr/src/sys/kern/sys_socket.c:153 (sleep mutex:so_snd)
     3            7            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:Mountpoints)
     3           11            0           4     2     0            0            0 /usr/src/sys/kern/kern_proc.c:401 (sleep mutex:process lock)
   144          877            0          10    87     0            1            0 /usr/src/sys/kern/kern_exit.c:449 (sleep mutex:process lock)
    43         6987            0        1021     6     0            0            0 /usr/src/sys/kern/uipc_socket.c:2831 (sleep mutex:accept)
     3           13            0           6     2     0            0            0 /usr/src/sys/kern/kern_switch.c:272 (spin mutex:turnstile lock)
     2            2            0           1     2     0            0            0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1149 (sleep mutex:FFS)
    23         6277           36        2280     2     0            4            8 /usr/src/sys/kern/kern_timeout.c:346 (spin mutex:callout)
    36          354          397          67     5     5           10           44 /usr/src/sys/kern/subr_turnstile.c:203 (spin mutex:turnstile lock)
    11         2274          196         847     2     0            0            8 /usr/src/sys/kern/sys_generic.c:1140 (spin mutex:sleepq chain)
   128          736            0          10    73     0            1            0 /usr/src/sys/kern/kern_exit.c:469 (sleep mutex:process lock)
    10          101            0          36     2     0            0            0 /usr/src/sys/kern/kern_event.c:1001 (sleep mutex:kqueue)
     5           25            0          10     2     0            0            0 /usr/src/sys/kern/kern_proc.c:426 (sleep mutex:process lock)
     4           12            0           5     2     0            0            0 /usr/src/sys/kern/kern_sig.c:959 (spin mutex:process slock)
     5           50            0          20     2     0            0            0 /usr/src/sys/kern/kern_event.c:1021 (sleep mutex:kqueue)
    23        16935        15065        6242     2     2         1672         1782 /usr/src/sys/kern/kern_clock.c:412 (spin mutex:sched lock)
    11          303            0         103     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:782 (sleep mutex:sigio lock)
     4           21            0          10     2     0            0            0 /usr/src/sys/kern/kern_kse.c:91 (spin mutex:kse lock)
   157         2965            0          44    67     0            0            0 /usr/src/sys/vm/vm_map.c:1173 (sx:user map)
  1129       265766            0       47196     5     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2261 (sleep mutex:pmap)
  1451        34722            0       13799     2     0            0            0 /usr/src/sys/ufs/ufs/ufs_vnops.c:293 (sleep mutex:vnode interlock)
    83        34697            1       13809     2     0            1            1 /usr/src/sys/kern/vfs_syscalls.c:1070 (sleep mutex:sleep mtxpool)
     4            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:sackhole)
     9          128            0          22     5     0            0            0 /usr/src/sys/ufs/ufs/ufs_dirhash.c:350 (sleep mutex:dirhash)
   977         7961            0         449    17     0            0            0 /usr/src/sys/vm/vm_map.c:1202 (sx:user map)
    24        95073         8174       34284     2     0         1212         1644 /usr/src/sys/kern/kern_timeout.c:419 (spin mutex:callout)
     5           53            0          16     3     0            0            0 /usr/src/sys/kern/kern_event.c:1074 (sleep mutex:kqueue)
     3            3            0           1     3     0            0            0 /usr/src/sys/kern/kern_event.c:1664 (sleep mutex:tty)
    12         3470            0        1568     2     0            0            0 /usr/src/sys/kern/subr_turnstile.c:834 (spin mutex:td_contested)
    22        15717            0        6001     2     0            0            0 /usr/src/sys/nfsserver/nfs_srvsock.c:796 (sleep mutex:nfsd_mtx)
     5           54            0          21     2     0            0            0 /usr/src/sys/ufs/ufs/ufs_dirhash.c:368 (sleep mutex:dirhash)
     5          255            0         107     2     0            0            0 /usr/src/sys/kern/vfs_syscalls.c:813 (sx:filedesc structure)
    11          117            0          35     3     0            0            0 /usr/src/sys/ufs/ffs/ffs_vnops.c:683 (sleep mutex:process lock)
   267          267            0           1   267     0            0            0 /usr/src/sys/vm/vm_map.c:2412 (sleep mutex:system map)
     9           51            0           8     6     0            0            0 /usr/src/sys/kern/subr_eventhandler.c:215 (sleep mutex:process_exec)
     2            8            0           4     2     0            0            0 /usr/src/sys/kern/kern_proc.c:574 (sleep mutex:session)
    12         1266            0         480     2     0            0            0 /usr/src/sys/kern/kern_resource.c:633 (spin mutex:sleepq chain)
  1335        51166          154        8527     6     0            4            9 /usr/src/sys/net/route.c:197 (sleep mutex:rtentry)
     5          320            0         120     2     0            0            0 /usr/src/sys/contrib/ipfilter/netinet/ip_frag.c:786 (rw:ipf fragment rwlock)
    11         2839         3455        1117     2     3          278          378 /usr/src/sys/kern/subr_trap.c:182 (spin mutex:sched lock)
    10           77            0          12     6     0            0            0 /usr/src/sys/kern/vfs_subr.c:3170 (sleep mutex:mountlist)
  1255       129532           27       47159     2     0            0            1 /usr/src/sys/amd64/amd64/trap.c:586 (sleep mutex:process lock)
     3           20           29          11     1     2            0            1 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:process lock)
     1            3            0           2     1     0            0            0 /usr/src/sys/kern/kern_proc.c:554 (sleep mutex:process lock)
  3306       139256            0        1241   112     0            0            0 /usr/src/sys/kern/uipc_sockbuf.c:145 (sx:so_snd_sx)
  2014       126223            0       47159     2     0            0            0 /usr/src/sys/amd64/amd64/trap.c:595 (sleep mutex:process lock)
    11         3674           13        1568     2     0            0            2 /usr/src/sys/kern/subr_turnstile.c:895 (spin mutex:td_contested)
  1332       162499       426032       62509     2     6           31         2414 /usr/src/sys/ufs/ufs/ufs_vnops.c:388 (sleep mutex:vnode interlock)
    21        11580          318        4211     2     0           45           62 /usr/src/sys/kern/kern_timeout.c:500 (spin mutex:callout)
    37          250            0          33     7     0            0            0 /usr/src/sys/kern/uipc_socket.c:2864 (sleep mutex:so_snd)
    40         9171            0         480    19     0            0            0 /usr/src/sys/kern/kern_timeout.c:241 (sleep mutex:process lock)
    11          136            0          36     3     0            0            0 /usr/src/sys/kern/uipc_socket.c:2880 (sleep mutex:so_snd)
     5           86           25          30     2     0            0            1 /usr/src/sys/netinet/tcp_timewait.c:253 (sleep mutex:so_rcv)
     2            2            0           1     2     0            0            0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:1024)
     3           43            0          20     2     0            0            0 /usr/src/sys/ufs/ufs/ufs_dirhash.c:457 (sleep mutex:dirhash)
    26         5248         1749        1050     4     1          335          182 /usr/src/sys/kern/subr_trap.c:236 (spin mutex:sched lock)
   609         3014            0           8   376     0            0            0 /usr/src/sys/kern/imgact_elf.c:672 (lockmgr:ufs)
    10          105            0          20     5     0            0            0 /usr/src/sys/kern/kern_event.c:1182 (sleep mutex:kqueue)
    41          255           25          59     4     0            0            1 /usr/src/sys/kern/uipc_socket.c:2899 (sleep mutex:so_snd)
     4           27            0          10     2     0            0            0 /usr/src/sys/netinet/tcp_timewait.c:272 (sleep mutex:so_rcv)
    37        12139           38        1448     8     0            1            2 /usr/src/sys/kern/kern_exit.c:686 (sleep mutex:process lock)
    38         1101            0         175     6     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2441 (sleep mutex:pmap)
     7            7            0           1     7     0            0            0 /usr/src/sys/kern/vfs_syscalls.c:951 (sx:filedesc structure)
     5           86            0          36     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:985 (sleep mutex:sigio lock)
    32         2530            0         567     4     0            0            0 /usr/src/sys/kern/kern_sig.c:663 (sleep mutex:sigacts)
    13         6815            0        2590     2     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2463 (sleep mutex:pmap)
    23         2266            0         481     4     0            0            0 /usr/src/sys/vm/vnode_pager.c:1192 (sleep mutex:vm object)
     3           22            0          10     2     0            0            0 /usr/src/sys/kern/kern_exit.c:722 (sleep mutex:process lock)
     5           38            0          16     2     0            0            0 /usr/src/sys/kern/kern_event.c:1257 (sleep mutex:kqueue)
    86         1997            0          46    43     0            0            0 /usr/src/sys/dev/random/yarrow.c:191 (sleep mutex:random reseed)
    10           28            0           3     9     0            0            0 /usr/src/sys/kern/kern_event.c:1266 (sleep mutex:kqueue)
     4           11            0           3     3     0            0            0 /usr/src/sys/kern/kern_event.c:1275 (sleep mutex:kqueue)
60450466   1060829007   1316714222      340355  3116  3868        62072        61264 /usr/src/sys/kern/vfs_subr.c:2035 (lockmgr:ufs)
    22          135            0          10    13     0            0            0 /usr/src/sys/kern/kern_fork.c:546 (sleep mutex:process group)
    55         4580            0         467     9     0            0            0 /usr/src/sys/vm/vnode_pager.c:1222 (sleep mutex:vm object)
     6           13            0           3     4     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:hostcache)
     5           22            0          10     2     0            0            0 /usr/src/sys/kern/kern_exit.c:759 (sleep mutex:process lock)
     8          468            0         120     3     0            0            0 /usr/src/sys/contrib/ipfilter/netinet/ip_state.c:3052 (rw:ipf IP state rwlock)
    11          203            0          67     3     0            0            0 /usr/src/sys/amd64/amd64/machdep.c:339 (sleep mutex:sigacts)
     4           25            0          10     2     0            0            0 /usr/src/sys/kern/kern_exit.c:762 (sleep mutex:process lock)
     6           30            0           9     3     0            0            0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:socket)
    15          448            0          60     7     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:750 (sleep mutex:mountlist)
  2326     16457165     20663004     1123871    14    18       274201        75173 /usr/src/sys/kern/kern_lock.c:200 (sleep mutex:lockbuilder mtxpool)
   207          724            0           4   181     0            0            0 /usr/src/sys/kern/imgact_elf.c:805 (lockmgr:ufs)
     3           17            0           8     2     0            0            0 /usr/src/sys/netinet/tcp_subr.c:782 (sleep mutex:so_rcv)
     5           16            0           5     3     0            0            0 /usr/src/sys/kern/sys_pipe.c:974 (sleep mutex:pipe mutex)
   114         5650            0         300    18     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:763 (sleep mutex:mountlist)
   113        17694        11995        3024     5     3          213         1036 /usr/src/sys/kern/subr_turnstile.c:549 (spin mutex:turnstile lock)
   202      9591282            0     4966397     1     0          932            0 /usr/src/sys/kern/subr_turnstile.c:555 (spin mutex:turnstile lock)
    13           36            0           5     7     0            0            0 /usr/src/sys/dev/random/yarrow.c:278 (sleep mutex:random reseed)
    74         1479            0          47    31     0            0            0 /usr/src/sys/amd64/amd64/mp_machdep.c:845 (spin mutex:smp rendezvous)
    10         2319            0         900     2     0            0            0 /usr/src/sys/libkern/arc4random.c:137 (sleep mutex:arc4_mtx)
    13           13            0           1    13     0            0            0 /usr/src/sys/kern/sys_socket.c:105 (sleep mutex:process lock)
   119        34926          989        8006     4     0           34           56 /usr/src/sys/dev/em/if_em.c:1396 (sleep mutex:em0)
    31          659            0          41    16     0            0            0 /usr/src/sys/kern/vfs_subr.c:2130 (lockmgr:ufs)
     3            7            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:divcb)
    34           77            0           6    12     0            0            0 /usr/src/sys/kern/kern_exit.c:848 (sleep mutex:process lock)
   143        52143         1858        1568    33     1            8           87 /usr/src/sys/kern/subr_turnstile.c:593 (spin mutex:turnstile lock)
     3           22            0           8     2     0            0            0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:mbuf_jumbo_pagesize)
     5           46            0          16     2     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:1123 (sleep mutex:so_rcv)
    60         2276            0         290     7     0            0            0 /usr/src/sys/amd64/amd64/mp_machdep.c:882 (spin mutex:smp rendezvous)
     7         1367            0         490     2     0            0            0 /usr/src/sys/vm/vm_unix.c:81 (sleep mutex:process lock)
     6          855            0         301     2     0            0            0 /usr/src/sys/netinet/tcp_timewait.c:491 (sleep mutex:so_rcv)
     2            2            0           1     2     0            0            0 /usr/src/sys/kern/kern_exit.c:874 (sleep mutex:process lock)
     4           17            0           8     2     0            0            0 /usr/src/sys/vm/vm_map.c:2709 (sleep mutex:process lock)
    12           43            0           4    10     0            0            0 /usr/src/sys/vm/vm_map.c:1524 (sleep mutex:vm page queue mutex)
    80         2562            0          74    34     0            0            0 /usr/src/sys/vm/vm_map.c:1560 (sx:user map)
     9          110            0          22     5     0            0            0 /usr/src/sys/ufs/ufs/ufs_dirhash.c:349 (sleep mutex:dirhash list)
    47         2034          978         171    11     5            4            3 /usr/src/sys/vm/vm_map.c:1534 (sleep mutex:vm page queue mutex)
     2           20            0          10     2     0            0            0 /usr/src/sys/kern/kern_fork.c:767 (sleep mutex:schedtail)
    14           59            0          10     5     0            0            0 /usr/src/sys/kern/subr_eventhandler.c:215 (sleep mutex:process_exit)
    15           72            0          16     4     0            0            0 /usr/src/sys/kern/kern_event.c:1442 (sleep mutex:kqueue)
     6          103            0          32     3     0            0            0 /usr/src/sys/kern/vfs_subr.c:1573 (sleep mutex:Syncer mtx)
    23          139            0          35     3     0            0            0 /usr/src/sys/kern/vfs_bio.c:1280 (sleep mutex:vm object)
    68        38631        12588       15205     2     0            0           15 /usr/src/sys/kern/kern_descrip.c:2179 (sx:filelist lock)
    19           80            0           5    16     0            0            0 /usr/src/sys/kern/sys_pipe.c:1127 (sleep mutex:pipe mutex)
    35          400            0          12    33     0            0            0 /usr/src/sys/netinet/ip_fw2.c:4877 (sleep mutex:IPFW dynamic rules)
     3           11            0           4     2     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2695 (sleep mutex:pmap)
   105         3835            0          60    63     0            2            0 /usr/src/sys/kern/kern_timeout.c:241 (sleep mutex:em0)
    46        13416            0         601    22     0            0            0 /usr/src/sys/geom/geom_event.c:185 (sx:GEOM topology)
     5           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:FFS1 dinode)
    23         4066            0        1500     2     0            0            0 /usr/src/sys/dev/syscons/syscons.c:1800 (spin mutex:syscons video lock)
    13           31            0           7     4     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:3744 (sleep mutex:Softdep Lock)
     4           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:g_bio)
    18           84            0          10     8     0            0            0 /usr/src/sys/kern/kern_conf.c:348 (sleep mutex:Giant)
     2            6            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ksiginfo)
    41          272            0          10    27     0            0            0 /usr/src/sys/kern/kern_fork.c:545 (sx:proctree)
     4           35            0          12     2     0            0            0 /usr/src/sys/kern/vfs_subr.c:2944 (sleep mutex:struct mount mtx)
  2361        26024          284         204   127     1           24           12 /usr/src/sys/vm/vm_object.c:641 (sleep mutex:vm page queue mutex)
     5            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ripcb)
   124         2659            0         151    17     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2734 (sleep mutex:pmap)
   120         1858            0         151    12     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2735 (sleep mutex:pmap)
    20           20            0           1    20     0            0            0 /usr/src/sys/kern/kern_conf.c:360 (sleep mutex:Giant)
    36         4233            0         506     8     0            0            0 /usr/src/sys/netinet/tcp_input.c:2867 (sleep mutex:so_rcv)
    59         1534            0         123    12     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2737 (sleep mutex:pmap)
    56         1551            0         123    12     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2738 (sleep mutex:pmap)
    12           46            0           8     5     0            0            0 /usr/src/sys/kern/kern_sig.c:952 (sleep mutex:sigacts)
     5           37            0          10     3     0            0            0 /usr/src/sys/kern/subr_eventhandler.c:215 (sleep mutex:schedtail)
    20          201            0          19    10     0            0            0 /usr/src/sys/kern/vfs_subr.c:1653 (sleep mutex:Syncer mtx)
    46          131            0           5    26     0            0            0 /usr/src/sys/kern/kern_conf.c:372 (sleep mutex:Giant)
  1728       128342        13097       14524     8     0           32           46 /usr/src/sys/kern/vfs_cache.c:692 (sleep mutex:Giant)
     5           36            0          10     3     0            0            0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:NAMEI)
    13           59           19           7     8     2            0            1 /usr/src/sys/kern/kern_conf.c:384 (sleep mutex:Giant)
    22         2020            0         601     3     0            0            0 /usr/src/sys/geom/geom_event.c:233 (sx:GEOM topology)
     5           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:VNODEPOLL)
     4           12            0           3     4     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:VNODE)
     2            2            0           1     2     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:3785 (sleep mutex:Softdep Lock)
    25           25            0           1    25     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:3794 (sleep mutex:Softdep Lock)
    11          225            0          72     3     0            0            0 /usr/src/sys/vm/vm_glue.c:183 (sx:user map)
    26          386            0          21    18     0            1            0 /usr/src/sys/kern/kern_event.c:1568 (sleep mutex:kqueue)
     4           32           54          12     2     4            0            1 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1208 (sleep mutex:struct mount mtx)
     5           34            0          12     2     0            0            0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1217 (sleep mutex:struct mount mtx)
     4           41            0          16     2     0            0            0 /usr/src/sys/kern/kern_event.c:1598 (sleep mutex:kqueue)
     7          166            0          60     2     0            0            0 /usr/src/sys/net/if.c:1473 (sleep mutex:ifnet)
   340       295281        10543        4618    63     2          207          179 /usr/src/sys/netinet/tcp_input.c:479 (sleep mutex:inp)
     5           25           19          10     2     1            5            5 /usr/src/sys/kern/kern_fork.c:383 (spin mutex:sched lock)
    10           90            0          19     4     0            0            0 /usr/src/sys/fs/devfs/devfs_vnops.c:194 (sleep mutex:devfs interlock)
  2390         2633            0           3   877     0            0            0 /usr/src/sys/ufs/ffs/ffs_vfsops.c:1250 (sleep mutex:struct mount mtx)
     3           34            0          16     2     0            0            0 /usr/src/sys/kern/kern_event.c:1620 (sleep mutex:kqueue)
    12           15            0           3     5     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ipq)
     5          250            0          96     2     0            0            0 /usr/src/sys/vm/vm_map.c:1800 (sx:user map)
    10           34            0           5     6     0            0            0 /usr/src/sys/kern/sys_pipe.c:1327 (sleep mutex:pipe mutex)
     5           31            0          12     2     0            0            0 /usr/src/sys/vm/vm_glue.c:274 (sleep mutex:vm page queue mutex)
  1594      1535506     35138225      558021     2    62         1181       117583 /usr/src/sys/kern/kern_lock.c:583 (sleep mutex:lockbuilder mtxpool)
    16          479            0          67     7     0            0            0 /usr/src/sys/kern/subr_trap.c:248 (sleep mutex:sigacts)
    24         5352          132        1864     2     0            0            5 /usr/src/sys/netinet/tcp_input.c:2025 (sleep mutex:so_snd)
    31           31            0           1    31     0            0            0 /usr/src/sys/kern/vfs_bio.c:1530 (sleep mutex:vm object)
     6         1292          335         496     2     0            1            4 /usr/src/sys/kern/kern_lock.c:607 (sleep mutex:lockbuilder mtxpool)
   107       186602            0       12002    15     0            0            0 /usr/src/sys/kern/subr_taskqueue.c:71 (spin mutex:fast_taskqueue)
     5           31            0          12     2     0            0            0 /usr/src/sys/vm/vm_glue.c:310 (sleep mutex:vm page queue mutex)
   419         3234            0          15   215     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2950 (sleep mutex:pmap)
     4           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:SLEEPQUEUE)
     5            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:DP fakepg)
     4           32            0          12     2     0            0            0 /usr/src/sys/kern/vfs_subr.c:3179 (sleep mutex:struct mount mtx)
    12          597            0         235     2     0            0            0 /usr/src/sys/vm/vm_object.c:225 (sleep mutex:vm object_list)
     5           41            0          12     3     0            0            0 /usr/src/sys/kern/vfs_subr.c:3185 (sleep mutex:struct mount mtx)
    97        20060            0         495    40     0            0            0 /usr/src/sys/netinet/tcp_input.c:636 (sleep mutex:inp)
     5           29            0           9     3     0            0            0 /usr/src/sys/kern/vfs_subr.c:1888 (sleep mutex:Syncer mtx)
     3           21            0           7     3     0            0            0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:socket)
  1022        35624          180       13861     2     0          109            7 /usr/src/sys/kern/vfs_vnops.c:903 (sleep mutex:struct mount mtx)
     4           83            0          33     2     0            0            0 /usr/src/sys/vm/vm_map.c:3092 (sleep mutex:system map)
    12         1057           33         379     2     0            2            3 /usr/src/sys/vm/vm_fault.c:137 (sleep mutex:vm page queue mutex)
    22         2098            0         600     3     0            0            0 /usr/src/sys/dev/random/randomdev_soft.c:249 (spin mutex:entropy harvest mutex)
3515419      3755160            0        2109  1780     0            0            0 /usr/src/sys/kern/uipc_sockbuf.c:145 (sx:so_rcv_sx)
     3            8            0           4     2     0            0            0 /usr/src/sys/kern/kern_proc.c:1206 (sleep mutex:process lock)
     3           36            0          15     2     0            0            0 /usr/src/sys/kern/kern_exec.c:318 (sleep mutex:process lock)
     3            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:FFS2 dinode)
     5           18            0           7     2     0            0            0 /usr/src/sys/fs/devfs/devfs_vnops.c:325 (sx:proctree)
     2            3            0           2     1     0            0            0 /usr/src/sys/vm/uma_core.c:1874 (sleep mutex:Files)
    13         1991            2         594     3     0            0            1 /usr/src/sys/dev/random/randomdev_soft.c:269 (spin mutex:entropy harvest mutex)
     5           88           12          34     2     0            0            1 /usr/src/sys/kern/kern_event.c:1802 (sleep mutex:kqueue)
    39          188            0          16    11     0            0            0 /usr/src/sys/kern/sys_pipe.c:1461 (sleep mutex:pipe mutex)
     4           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:ata_request)
   469         1141            0           4   285     0            0            0 /usr/src/sys/kern/vfs_subr.c:1638 (lockmgr:devfs)
  1475       357918          829       13709    26     0            0            2 /usr/src/sys/kern/vfs_vnops.c:288 (lockmgr:ufs)
    23           23            0           1    23     0            0            0 /usr/src/sys/kern/uipc_socket.c:481 (sleep mutex:so_rcv)
    23        16712            0        6027     2     0            0            0 /usr/src/sys/dev/random/randomdev_soft.c:308 (spin mutex:entropy harvest mutex)
     5           47            0          16     2     0            0            0 /usr/src/sys/kern/sys_pipe.c:1501 (sleep mutex:pipe mutex)
     6          144            0          46     3     0            0            0 /usr/src/sys/kern/vfs_bio.c:429 (sleep mutex:buffer daemon lock)
   960       265391            0      101087     2     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:3074 (sleep mutex:pmap)
     5           41            0          16     2     0            0            0 /usr/src/sys/kern/kern_event.c:1868 (sleep mutex:kqueue)
     6           29            0          10     2     0            0            0 /usr/src/sys/kern/kern_fork.c:658 (spin mutex:process slock)
    26          139           57          16     8     3            0            1 /usr/src/sys/dev/mfi/mfi_disk.c:258 (sleep mutex:MFI I/O lock)
    43          144            0           5    28     0            0            0 /usr/src/sys/fs/devfs/devfs_vnops.c:375 (lockmgr:devfs)
    23         3037         3056         600     5     5          322          377 /usr/src/sys/kern/sched_4bsd.c:271 (spin mutex:sched lock)
     5           31           43          10     3     4            3            5 /usr/src/sys/kern/kern_fork.c:667 (spin mutex:sched lock)
    38           38            0           1    38     0            0            0 /usr/src/sys/vm/vm_kern.c:334 (sleep mutex:vm object)
    57         1556            0         328     4     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:255 (sleep mutex:inp)
    20        33680         4195       13861     2     0           50          116 /usr/src/sys/kern/vfs_vnops.c:1046 (sleep mutex:struct mount mtx)
     4           11            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:VM OBJECT)
     3           17            0           8     2     0            0            0 /usr/src/sys/kern/sys_pipe.c:366 (sleep mutex:sleep mtxpool)
     6           49            0          19     2     0            0            0 /usr/src/sys/fs/devfs/devfs_vnops.c:201 (sx:devfsmount)
     4           19            0           8     2     0            0            0 /usr/src/sys/kern/sys_pipe.c:381 (sleep mutex:sleep mtxpool)
    12        13957           50        6001     2     0            2           11 /usr/src/sys/kern/kern_clock.c:264 (spin mutex:callout)
    63           63            0           1    63     0            0            0 /usr/src/sys/vm/vm_kern.c:403 (sleep mutex:vm object)
    33          171            0           8    21     0            0            0 /usr/src/sys/kern/kern_exec.c:505 (sleep mutex:process lock)
    23         3053          278        1112     2     0            3            8 /usr/src/sys/vm/vm_fault.c:340 (sleep mutex:vm page queue mutex)
   471        84026            0        6206    13     0            0            0 /usr/src/sys/kern/sched_4bsd.c:382 (spin mutex:process slock)
    30         4559            0         467     9     0            0            0 /usr/src/sys/vm/vnode_pager.c:1204 (sleep mutex:vnode interlock)
     5          174            0          55     3     0            0            0 /usr/src/sys/netinet/ip_fw2.c:1303 (sleep mutex:IPFW dynamic rules)
   428        24067          602        7226     3     0            3           20 /usr/src/sys/netinet/tcp_output.c:270 (sleep mutex:so_snd)
    10         8210        17484        2800     2     6         1039         1446 /usr/src/sys/kern/sched_4bsd.c:385 (spin mutex:sched lock)
   296          361            0           6    60     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:4239 (sleep mutex:Softdep Lock)
     5           28            0          10     2     0            0            0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:NAMEI)
    84         4311            0         499     8     0            0            0 /usr/src/sys/netinet/tcp_syncache.c:316 (sleep mutex:tcp_sc_head)
    11          216            0          73     2     0            0            0 /usr/src/sys/kern/kern_sig.c:1961 (spin mutex:process slock)
   132        16865         5178        1461    11     3           92          100 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:sellck)
     5           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:syncache)
    25      2723309       236100     1027898     2     0        18742        33987 /usr/src/sys/kern/subr_sleepqueue.c:331 (spin mutex:sched lock)
     5           32            0          10     3     0            0            0 /usr/src/sys/kern/kern_descrip.c:1472 (sleep mutex:fdesc)
     6          209            0          71     2     0            0            0 /usr/src/sys/kern/uipc_socket.c:686 (sleep mutex:so_rcv)
    47        11875          135        4251     2     0            1            6 /usr/src/sys/netinet/ip_output.c:594 (sleep mutex:rtentry)
6953267      7713157            0          92 83838     0            0            0 /usr/src/sys/kern/vfs_vnops.c:515 (lockmgr:ufs)
    39         5574            0         567     9     0            0            0 /usr/src/sys/kern/kern_sig.c:661 (sleep mutex:process lock)
     5            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:rtentry)
    13           89            0          10     8     0            0            0 /usr/src/sys/netinet/tcp_timewait.c:271 (sleep mutex:accept)
     6           97            0          36     2     0            0            0 /usr/src/sys/kern/uipc_socket.c:735 (sleep mutex:so_rcv)
     3            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:THREAD)
    77           89            0           2    44     0            0            0 /usr/src/sys/kern/vfs_syscalls.c:3784 (lockmgr:ufs)
     2            7            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf_ext_refcnt)
    20        34058            0       13936     2     0            0            0 /usr/src/sys/kern/kern_prot.c:91 (sleep mutex:process lock)
   261         1916            0          11   174     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:470 (sleep mutex:inp)
   330         5506            0          35   157     0            0            0 /usr/src/sys/kern/vfs_vnops.c:578 (lockmgr:ufs)
    30         4236         9198        1551     2     5          724         1055 /usr/src/sys/kern/subr_sleepqueue.c:411 (spin mutex:sched lock)
    11          466            0         182     2     0            0            0 /usr/src/sys/vm/vm_map.c:980 (sleep mutex:vm object)
     3            5            0           2     2     0            0            0 /usr/src/sys/kern/kern_prot.c:109 (sleep mutex:process lock)
    18          607            0          67     9     0            0            0 /usr/src/sys/amd64/amd64/machdep.c:338 (sleep mutex:process lock)
     2            5            0           2     2     0            0            0 /usr/src/sys/kern/kern_prot.c:128 (sleep mutex:process lock)
  1342       189258            0       13884    13     0            1            0 /usr/src/sys/kern/vfs_vnops.c:613 (lockmgr:ufs)
   371       203548           60        4618    44     0           10            3 /usr/src/sys/netinet/tcp_input.c:400 (sleep mutex:tcp)
   125        19394            0        1006    19     0            0            0 /usr/src/sys/netinet/tcp_syncache.c:464 (sleep mutex:tcp_sc_head)
    41          981            0          60    16     0            0            0 /usr/src/sys/kern/vfs_subr.c:730 (sleep mutex:vnode_free_list)
     4            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:NFSMOUNT)
     5           60            0          16     3     0            0            0 /usr/src/sys/netinet/ip_fw2.c:1466 (sleep mutex:IPFW dynamic rules)
     6          603            0         204     2     0            0            0 /usr/src/sys/vm/vm_object.c:667 (sleep mutex:vm object_list)
    38          209            0          10    20     0            0            0 /usr/src/sys/kern/kern_fork.c:267 (sx:allproc)
     3            7            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:IPFW dynamic rule)
    14           39            0           5     7     0            0            0 /usr/src/sys/vm/vm_object.c:1331 (sleep mutex:vm page queue mutex)
   126         1249            0          19    65     0            0            0 /usr/src/sys/kern/vfs_subr.c:2035 (lockmgr:devfs)
  3199        27657          139        4251     6     0           10            9 /usr/src/sys/net/route.c:1287 (sleep mutex:rtentry)
     5          182            0          67     2     0            0            0 /usr/src/sys/amd64/amd64/machdep.c:414 (sleep mutex:process lock)
    10           58            0           8     7     0            0            0 /usr/src/sys/netinet/tcp_subr.c:781 (sleep mutex:accept)
   136         1219            0          18    67     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:571 (sleep mutex:inp)
    46        25509            0        4251     6     0            1            0 /usr/src/sys/net/route.c:1303 (sleep mutex:rtentry)
     5           40            0          12     3     0            0            0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:vm page queue free mutex)
     3            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:tcpcb)
    36         1207            0          67    18     0            0            0 /usr/src/sys/kern/kern_sig.c:2177 (spin mutex:process slock)
    18        33378           37       13926     2     0            2            3 /usr/src/sys/kern/kern_conf.c:69 (sleep mutex:cdev)
     5           30            0          10     3     0            0            0 /usr/src/sys/kern/kern_descrip.c:1673 (sleep mutex:fdesc)
     5           93           13          36     2     0            0            1 /usr/src/sys/kern/uipc_syscalls.c:404 (sleep mutex:so_rcv)
     1            1            0           1     1     0            0            0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:Files)
    26      2199187        45543     1023647     2     0        16907         5743 /usr/src/sys/kern/subr_sleepqueue.c:541 (spin mutex:sched lock)
     2           13            0           7     1     0            0            0 /usr/src/sys/kern/kern_exec.c:784 (sleep mutex:process lock)
    11          105            0          36     2     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:606 (sleep mutex:inp)
    14           64            0           7     9     0            0            0 /usr/src/sys/fs/devfs/devfs_vfsops.c:157 (sx:devfsmount)
     5           30           60          10     3     6            4            6 /usr/src/sys/kern/sched_4bsd.c:650 (spin mutex:sched lock)
     5           24            0          10     2     0            1            0 /usr/src/sys/kern/sched_4bsd.c:653 (spin mutex:sched lock)
   136        15949          969         482    33     2           60           14 /usr/src/sys/amd64/amd64/pmap.c:1958 (sleep mutex:vm page queue mutex)
  4305        28301            0          23  1230     0            0            0 /usr/src/sys/vm/vm_map.c:2412 (sx:user map)
    22         8680        20146        2700     3     7         1358         2146 /usr/src/sys/kern/subr_sleepqueue.c:576 (spin mutex:sched lock)
     3            3            1           1     3     1            0            1 /usr/src/sys/kern/sched_4bsd.c:682 (spin mutex:sched lock)
    31          160            0           7    22     0            0            0 /usr/src/sys/fs/devfs/devfs_vnops.c:789 (lockmgr:devfs)
    19         2655            0         301     8     0            0            0 /usr/src/sys/netinet/tcp_timewait.c:490 (sleep mutex:accept)
  1769       134142         6505       46891     2     0          121          236 /usr/src/sys/vm/vm_page.c:1030 (sleep mutex:vm page queue free mutex)
   124         1598            0          37    43     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:697 (sleep mutex:inp)
    29          265            0          19    13     0            0            0 /usr/src/sys/fs/devfs/devfs_vnops.c:197 (sleep mutex:vnode interlock)
     3           10            0           3     3     0            0            0 /usr/src/sys/kern/vfs_subr.c:1545 (sleep mutex:vnode interlock)
     5           58            0          19     3     0            0            0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:kqueue)
     6          578            0         180     3     0            0            0 /usr/src/sys/kern/kern_kthread.c:194 (sleep mutex:process lock)
     5           44            0          12     3     0            0            0 /usr/src/sys/kern/vfs_bio.c:3437 (sleep mutex:vm page queue mutex)
   145        36802         9668        2017    18     4          178          208 /usr/src/sys/netinet/tcp_usrreq.c:729 (sleep mutex:inp)
     5            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:UMA Slabs)
     3           21            0           8     2     0            0            0 /usr/src/sys/vm/uma_core.c:2335 (sleep mutex:tcptw)
    12          491           31         179     2     0            0            2 /usr/src/sys/kern/vfs_bio.c:1487 (sleep mutex:vnode interlock)
    23        52319            0       19274     2     0            1            0 /usr/src/sys/kern/kern_sig.c:996 (sleep mutex:process lock)
   419          467            0          18    25     0            0            0 /usr/src/sys/kern/uipc_syscalls.c:565 (sleep mutex:so_rcv)
    18          123            0          12    10     0            0            0 /usr/src/sys/fs/devfs/devfs_vnops.c:691 (sx:devfsmount)
  3293       121661            0        1236    98     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:779 (sleep mutex:inp)
    18          373           36          42     8     0            3            2 /usr/src/sys/vm/vm_object.c:1551 (sleep mutex:vm page queue mutex)
    12          164            0          38     4     0            0            0 /usr/src/sys/kern/vfs_bio.c:3499 (sleep mutex:vm page queue mutex)
    11         1618         2939         609     2     4          274          322 /usr/src/sys/kern/sys_generic.c:753 (spin mutex:sched lock)
     8           89            0          19     4     0            0            0 /usr/src/sys/kern/vfs_subr.c:1622 (sleep mutex:vnode interlock)
    48          147            0           5    29     0            0            0 /usr/src/sys/netinet/tcp_reass.c:267 (sleep mutex:so_rcv)
    11         2933          983        1071     2     0           23           25 /usr/src/sys/vm/vm_object.c:1570 (sleep mutex:vm page queue mutex)
  3310        11238            0           7  1605     0            0            0 /usr/src/sys/vm/vm_map.c:2584 (sx:user map)
     1            3            0           2     1     0            0            0 /usr/src/sys/vm/uma_core.c:2394 (sleep mutex:tcptw)
    13          140            0          19     7     0            0            0 /usr/src/sys/kern/vfs_subr.c:1642 (sleep mutex:vnode interlock)
    85         1742            0         178     9     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2139 (sleep mutex:vm page queue mutex)
    11         1593         3229         602     2     5          253          379 /usr/src/sys/kern/sys_generic.c:778 (spin mutex:sched lock)
    33      6922534       286114     1029804     6     0        35897        37195 /usr/src/sys/kern/sched_4bsd.c:847 (spin mutex:sched lock)
     5            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:tcpreass)
  1192     19128319      6157083       63412   301    97        42132        23511 /usr/src/sys/kern/vfs_hash.c:79 (sleep mutex:vnode interlock)
     4            6           10           2     3     5            1            2 /usr/src/sys/kern/subr_sleepqueue.c:754 (spin mutex:sched lock)
    12         1617         2958         604     2     4          191          243 /usr/src/sys/kern/sys_generic.c:795 (spin mutex:sched lock)
    31         1302            0         455     2     0            0            0 /usr/src/sys/vm/vm_object.c:352 (sleep mutex:vm object)
    12           63            0           7     9     0            0            0 /usr/src/sys/fs/devfs/devfs_vnops.c:351 (sleep mutex:vnode interlock)
     5          108            0          38     2     0            0            0 /usr/src/sys/kern/vfs_bio.c:2326 (sleep mutex:vm object)
    23           39            0           3    13     0            0            0 /usr/src/sys/kern/kern_prot.c:481 (sleep mutex:process lock)
     7           22            0           4     5     0            0            0 /usr/src/sys/kern/kern_event.c:1664 (sleep mutex:pipe mutex)
    68          443            0           9    49     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:254 (sleep mutex:tcp)
  1436       130021        40386       47192     2     0          101          428 /usr/src/sys/vm/vm_fault.c:887 (sleep mutex:vm page queue mutex)
    29         4125            0         506     8     0            0            0 /usr/src/sys/netinet/tcp_input.c:2850 (sleep mutex:so_snd)
     1            1            0           1     1     0            0            0 /usr/src/sys/fs/fifofs/fifo_vnops.c:297 (sleep mutex:sleep mtxpool)
    19          829            0          67    12     0            0            0 /usr/src/sys/kern/subr_trap.c:247 (sleep mutex:process lock)
    43          110            0           5    22     0            0            0 /usr/src/sys/kern/vfs_bio.c:1649 (sleep mutex:vnode interlock)
    13           13            0           1    13     0            0            0 /usr/src/sys/kern/kern_resource.c:266 (spin mutex:process slock)
  2368       158777           22       48298     3     0            1            1 /usr/src/sys/vm/vm_object.c:441 (sleep mutex:vm object)
     4           21           11           6     3     1            2            1 /usr/src/sys/kern/subr_sleepqueue.c:852 (spin mutex:sched lock)
    19         3678            0         490     7     0            0            0 /usr/src/sys/vm/vm_unix.c:88 (sx:user map)
     1            1            0           1     1     0            0            0 /usr/src/sys/fs/fifofs/fifo_vnops.c:224 (sleep mutex:fifo mutex)
    10         1147            0         441     2     0            0            0 /usr/src/sys/vm/vm_mmap.c:284 (sleep mutex:process lock)
  1129       258864        49465       47196     5     1          250          443 /usr/src/sys/amd64/amd64/pmap.c:2260 (sleep mutex:vm page queue mutex)
     5           21            0           8     2     0            0            0 /usr/src/sys/vm/vm_map.c:2713 (sx:user map)
     5           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf_cluster)
     5           50            0          16     3     0            0            0 /usr/src/sys/kern/kern_event.c:534 (sleep mutex:sleep mtxpool)
     3           16            0           6     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf)
     6          197            0          60     3     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:733 (sleep mutex:Softdep Lock)
    62          955            0          47    20     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:956 (sleep mutex:inp)
  1008       130585          890       48074     2     0           45           47 /usr/src/sys/vm/vm_page.c:1315 (sleep mutex:vm page queue free mutex)
    22         6465        12743        2446     2     5          909         1166 /usr/src/sys/kern/sys_generic.c:927 (spin mutex:sched lock)
   135         4483            0        1551     2     0            0            0 /usr/src/sys/kern/subr_sleepqueue.c:392 (sleep mutex:sigacts)
   961         4160            0         181    22     0            0            0 /usr/src/sys/vm/vm_map.c:1467 (sleep mutex:vm object)
    37           92            0           5    18     0            0            0 /usr/src/sys/kern/kern_prot.c:593 (sleep mutex:process lock)
  3089         8221            0          28   293     0            0            0 /usr/src/sys/vm/vm_object.c:497 (sleep mutex:vm object)
    22         8647            0        3098     2     0            0            0 /usr/src/sys/kern/uipc_syscalls.c:135 (sleep mutex:sleep mtxpool)
    13           79            0           8     9     0            0            0 /usr/src/sys/kern/vfs_bio.c:3692 (sleep mutex:vm page queue mutex)
    31        20026          126        2590     7     0            8           10 /usr/src/sys/vm/vm_fault.c:981 (sleep mutex:vm page queue mutex)
    36          354            0          60     5     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:770 (sleep mutex:Softdep Lock)
    21         2330         6130         862     2     7          386          512 /usr/src/sys/kern/sys_generic.c:950 (spin mutex:sched lock)
     3           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:256)
     2            6            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf_jumbo_16k)
    11         4343         7674        1584     2     4          693          845 /usr/src/sys/kern/sys_generic.c:967 (spin mutex:sched lock)
     4            8            0           3     2     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:4889 (sleep mutex:Softdep Lock)
    23         1412            0         503     2     0            0            0 /usr/src/sys/kern/uipc_socket.c:409 (sleep mutex:accept)
    18          189            0          25     7     0            0            0 /usr/src/sys/kern/vfs_subr.c:1860 (sleep mutex:vnode interlock)
     5           82            0          31     2     0            0            0 /usr/src/sys/kern/uipc_syscalls.c:183 (sleep mutex:sleep mtxpool)
    15           32            0           3    10     0            0            0 /usr/src/sys/kern/kern_prot.c:646 (sleep mutex:process lock)
    34          251            0          74     3     0            0            0 /usr/src/sys/kern/kern_sig.c:2102 (sleep mutex:sigacts)
    27       127819         3909       62089     2     0         1192          414 /usr/src/sys/kern/kern_synch.c:216 (spin mutex:sched lock)
     6          378            0         129     2     0            0            0 /usr/src/sys/vm/vm_object.c:567 (sleep mutex:vm object)
  3208       246599          218         346   712     0           63           13 /usr/src/sys/vm/vm_object.c:1816 (sleep mutex:vm page queue mutex)
    13          981            0         300     3     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:843 (sleep mutex:Softdep Lock)
    62          770            0          28    27     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:385 (sleep mutex:udp)
    41         1224            0          66    18     0            2            0 /usr/src/sys/kern/kern_sysctl.c:1334 (sleep mutex:Giant)
     4           45            0          21     2     0            0            0 /usr/src/sys/vm/uma_core.c:889 (sleep mutex:tcpcb)
     5           15            0           4     3     0            0            0 /usr/src/sys/vm/vm_fault.c:1061 (sleep mutex:vm page queue mutex)
    22         1330            0         495     2     0            0            0 /usr/src/sys/kern/uipc_socket.c:451 (sleep mutex:accept)
     6         1759            0         601     2     0            0            0 /usr/src/sys/geom/geom_event.c:187 (sleep mutex:GEOM orphanage)
    42      2034552      3065332     1029804     1     2        40557       274671 /usr/src/sys/kern/sched_4bsd.c:1102 (spin mutex:sched lock)
     3            5           38           2     2    19            0            1 /usr/src/sys/kern/sched_4bsd.c:385 (spin mutex:turnstile lock)
    19         5672            0         601     9     0            0            0 /usr/src/sys/geom/geom_event.c:201 (sleep mutex:GEOM orphanage)
    22         4135            0        1530     2     0            0            0 /usr/src/sys/netinet/tcp_hostcache.c:291 (sleep mutex:tcp_hc_entry)
   604         1524            0           3   508     0            0            0 /usr/src/sys/vm/uma_core.c:1492 (sleep mutex:UMA lock)
     5          207            0          60     3     0            0            0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:Softdep Lock)
   273         1991            0          11   181     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:467 (sleep mutex:tcp)
    57          260            0           7    37     0            0            0 /usr/src/sys/kern/vfs_vnops.c:288 (lockmgr:devfs)
    20           49            0           4    12     0            0            0 /usr/src/sys/kern/kern_prot.c:745 (sleep mutex:process lock)
    19        33978         5496       13850     2     0           57          152 /usr/src/sys/kern/vfs_mount.c:432 (sleep mutex:struct mount mtx)
    94       213487        28811       63735     3     0          298          852 /usr/src/sys/kern/vfs_mount.c:441 (sleep mutex:struct mount mtx)
    72         6654          391        2152     3     0           17           25 /usr/src/sys/kern/uipc_socket.c:1424 (sleep mutex:so_rcv)
    11           11            0           1    11     0            0            0 /usr/src/sys/kern/vfs_syscalls.c:367 (lockmgr:ufs)
     3            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:MAP)
     3            3            0           1     3     0            0            0 /usr/src/sys/ufs/ufs/ufs_vnops.c:2038 (sleep mutex:vnode interlock)
     5           62          176          21     2     8           12           16 /usr/src/sys/kern/sys_generic.c:1140 (spin mutex:sched lock)
    10         1128            0         297     3     0            0            0 /usr/src/sys/kern/vfs_subr.c:2019 (sleep mutex:vnode interlock)
    22           86            0           6    14     0            0            0 /usr/src/sys/kern/kern_prot.c:807 (sleep mutex:process lock)
    24        17047            0        6001     2     0            0            0 /usr/src/sys/netinet/ip_dummynet.c:739 (sleep mutex:dummynet)
     7         1158            0         424     2     0            0            0 /usr/src/sys/kern/uipc_sockbuf.c:116 (sleep mutex:so_rcv)
   145         1332            0          18    74     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:568 (sleep mutex:tcp)
  2378       902001       805071      340953     2     2          213         5895 /usr/src/sys/kern/vfs_subr.c:2039 (sleep mutex:vnode interlock)
     5            5            0           1     5     0            0            0 /usr/src/sys/ufs/ffs/ffs_softdep.c:999 (sleep mutex:Softdep Lock)
    12           55            0          16     3     0            0            0 /usr/src/sys/kern/kern_event.c:530 (sx:filedesc structure)
     5          117            0          40     2     0            0            0 /usr/src/sys/kern/kern_exit.c:233 (sleep mutex:process_exit)
  2449       493209       107985      193425     2     0         1229         3624 /usr/src/sys/kern/vfs_subr.c:2062 (sleep mutex:vnode interlock)
    22        16740           10        6001     2     0            0            3 /usr/src/sys/kern/kern_synch.c:320 (spin mutex:fast_taskqueue)
     5            9            0           2     4     0            0            0 /usr/src/sys/kern/kern_sig.c:1468 (sleep mutex:process lock)
    23       155940       124384       62154     2     2        12262        18314 /usr/src/sys/kern/sched_4bsd.c:1281 (spin mutex:sched lock)
     5           10            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:32)
     4           41            0          16     2     0            0            0 /usr/src/sys/netinet/tcp_usrreq.c:1253 (sleep mutex:inp)
    11           38            0           4     9     0            0            0 /usr/src/sys/kern/vfs_default.c:410 (sleep mutex:vnode interlock)
     4           11            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:64 Bucket)
   771       165545          345       64596     2     0         1198           17 /usr/src/sys/kern/vfs_subr.c:2101 (sleep mutex:vnode interlock)
     5           91            0          36     2     0            0            0 /usr/src/sys/kern/uipc_syscalls.c:426 (sleep mutex:sleep mtxpool)
    15           98            0          12     8     0            0            0 /usr/src/sys/vm/vm_glue.c:257 (sleep mutex:vm object)
    25         3959            0         480     8     0            0            0 /usr/src/sys/kern/kern_resource.c:631 (spin mutex:process slock)
     5          182            0          67     2     0            0            0 /usr/src/sys/kern/kern_sig.c:591 (spin mutex:sleepq chain)
    11           43            0           7     6     0            0            0 /usr/src/sys/kern/vfs_default.c:442 (sleep mutex:vnode interlock)
     6          150            0          52     2     0            0            0 /usr/src/sys/kern/vfs_subr.c:2131 (sleep mutex:vnode interlock)
    18          673            0          71     9     0            0            0 /usr/src/sys/kern/uipc_socket.c:685 (sleep mutex:accept)
   102        12942            0         530    24     0            0            0 /usr/src/sys/kern/uipc_sockbuf.c:232 (sleep mutex:so_rcv)
  3215      3001405            0       47192    63     0            0            0 /usr/src/sys/vm/vm_map.c:3092 (sx:user map)
     4            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:NFSNODE)
    17          102            0          10    10     0            0            0 /usr/src/sys/kern/kern_fork.c:390 (sleep mutex:process lock)
     9           50            0          10     5     0            0            0 /usr/src/sys/kern/kern_fork.c:391 (sleep mutex:process lock)
    10          336            0         120     2     0            0            0 /usr/src/sys/contrib/ipfilter/netinet/fil.c:6587 (rw:ipf token rwlock)
  3054     16077774      3516313      469783    34     7        14825        16542 /usr/src/sys/kern/vfs_subr.c:2159 (sleep mutex:vnode interlock)
   135         1820          117          37    49     3            2            1 /usr/src/sys/netinet/tcp_usrreq.c:694 (sleep mutex:tcp)
     4            4            0           1     4     0            0            0 /usr/src/sys/kern/kern_resource.c:698 (spin mutex:process slock)
     6           41            0          17     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:428 (sleep mutex:sleep mtxpool)
    34      1370323      8870716      411996     3    21       481393       410555 /usr/src/sys/kern/sched_4bsd.c:1381 (spin mutex:sched lock)
  1182       330039        18436       63730     5     0          711          478 /usr/src/sys/kern/vfs_subr.c:337 (sleep mutex:struct mount mtx)
    90          825            0          16    51     0            1            0 /usr/src/sys/dev/mfi/mfi.c:912 (sleep mutex:MFI I/O lock)
     4            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:inpcb)
     9          272            0          99     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:441 (sleep mutex:sleep mtxpool)
     5           54            0          16     3     0            0            0 /usr/src/sys/amd64/amd64/io_apic.c:212 (spin mutex:icu)
     5           11            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:S VFS Cache)
     4            6            0           2     3     0            0            0 /usr/src/sys/kern/vfs_subr.c:2210 (sleep mutex:vnode interlock)
     5          181            0          70     2     0            0            0 /usr/src/sys/kern/kern_time.c:630 (spin mutex:process slock)
    39        10389          383        2916     3     0           33           25 /usr/src/sys/kern/uipc_socket.c:1670 (sleep mutex:so_rcv)
    68           68            0           1    68     0            0            0 /usr/src/sys/kern/kern_prot.c:1008 (sleep mutex:process lock)
    25          165            0          10    16     0            0            0 /usr/src/sys/kern/kern_fork.c:457 (sleep mutex:process lock)
    19          112            0          10    11     0            0            0 /usr/src/sys/kern/kern_fork.c:458 (sleep mutex:process lock)
     5           67            0          16     4     0            0            0 /usr/src/sys/amd64/amd64/io_apic.c:229 (spin mutex:icu)
    18          329            0          36     9     0            0            0 /usr/src/sys/kern/kern_event.c:993 (sleep mutex:sleep mtxpool)
    12           28            0           8     3     0            0            0 /usr/src/sys/kern/vfs_subr.c:2232 (sleep mutex:vnode interlock)
   122         3770            0         274    13     0            0            0 /usr/src/sys/amd64/amd64/pmap.c:2732 (sleep mutex:vm page queue mutex)
     3           15            0           7     2     0            0            0 /usr/src/sys/fs/devfs/devfs_vnops.c:803 (sleep mutex:sleep mtxpool)
    18           41            0           3    13     0            0            0 /usr/src/sys/kern/uipc_sockbuf.c:343 (sleep mutex:so_rcv)
    12          205            0          54     3     0            0            0 /usr/src/sys/kern/kern_event.c:1664 (sleep mutex:so_rcv)
     5           12            0           3     4     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:128 Bucket)
    12           12            0           1    12     0            0            0 /usr/src/sys/kern/vfs_vnops.c:613 (lockmgr:devfs)
  1567     12936770      2661709      215390    60    12         4383        14140 /usr/src/sys/kern/vfs_subr.c:2277 (sleep mutex:vnode interlock)
   195      1961598     20879159     1146748     1    18        63631       988242 /usr/src/sys/kern/subr_turnstile.c:536 (spin mutex:turnstile chain)
     3           25            0          10     2     0            0            0 /usr/src/sys/kern/kern_fork.c:556 (sleep mutex:session)
   125     30888316    143499369     4969421     6    28      4824196      4557442 /usr/src/sys/kern/subr_turnstile.c:546 (spin mutex:turnstile chain)
    12          688            0         248     2     0            0            0 /usr/src/sys/kern/vfs_bio.c:1353 (sleep mutex:buf queue lock)
     3            9            0           4     2     0            0            0 /usr/src/sys/kern/kern_proc.c:263 (sleep mutex:process group)
    21          364            0          34    10     0            0            0 /usr/src/sys/kern/vfs_cache.c:444 (lockmgr:ufs)
    21           21            0           1    21     0            0            0 /usr/src/sys/kern/kern_prot.c:1085 (sleep mutex:process lock)
    22          323            0          36     8     0            0            0 /usr/src/sys/kern/uipc_syscalls.c:371 (sleep mutex:accept)
    36          253            0          10    25     0            0            0 /usr/src/sys/kern/kern_fork.c:547 (sleep mutex:process lock)
    22          162            0          10    16     0            0            0 /usr/src/sys/kern/kern_fork.c:548 (sleep mutex:process lock)
     2            5            0           2     2     0            0            0 /usr/src/sys/kern/uipc_syscalls.c:643 (sleep mutex:sleep mtxpool)
     2            4            0           2     2     0            0            0 /usr/src/sys/kern/uipc_syscalls.c:648 (sleep mutex:sleep mtxpool)
     1            3            0           2     1     0            0            0 /usr/src/sys/kern/kern_proc.c:308 (sleep mutex:process group)
     4           24            0          10     2     0            0            0 /usr/src/sys/kern/kern_fork.c:574 (sleep mutex:ktrace)
     3            8            0           3     2     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:FFS inode)
     8           46            0          16     2     0            0            0 /usr/src/sys/kern/vfs_bio.c:340 (sleep mutex:runningbufspace lock)
     3            6            0           2     3     0            0            0 /usr/src/sys/kern/kern_proc.c:325 (sleep mutex:process group)
     2            5            0           3     1     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:mbuf_jumbo_9k)
    15         5638            0        2020     2     0            0            0 /usr/src/sys/kern/uipc_socket.c:1826 (sleep mutex:so_rcv)
  1837       498920           58       47192    10     0            3            2 /usr/src/sys/vm/vm_fault.c:293 (sleep mutex:vm object)
     5           31            0           9     3     0            0            0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:pipe mutex)
  3805      6501672       354014       63411   102     5         1129         1147 /usr/src/sys/kern/vfs_hash.c:71 (sleep mutex:vfs hash)
    31         1642            0         219     7     0            0            0 /usr/src/sys/kern/vfs_bio.c:1468 (sleep mutex:buf queue lock)
     4            9            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:udpcb)
     5           43            0          18     2     0            0            0 /usr/src/sys/kern/kern_prot.c:1203 (sleep mutex:process lock)
    70         1242          138          47    26     2            0            5 /usr/src/sys/netinet/tcp_usrreq.c:955 (sleep mutex:tcp)
    22           62            0           4    15     0            0            0 /usr/src/sys/kern/kern_proc.c:399 (sleep mutex:process group)
    12           37            0           4     9     0            0            0 /usr/src/sys/kern/kern_proc.c:400 (sleep mutex:process group)
     3           42            0          21     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:676 (sleep mutex:sleep mtxpool)
     4            9            0           3     3     0            0            0 /usr/src/sys/kern/uipc_socket.c:1888 (sleep mutex:so_rcv)
     5           69            0          28     2     0            0            0 /usr/src/sys/vm/vm_object.c:1720 (sleep mutex:vm object_list)
     4           25            0          10     2     0            0            0 /usr/src/sys/kern/kern_fork.c:676 (sleep mutex:process lock)
     5           10            0           3     3     0            0            0 /usr/src/sys/kern/uipc_socket.c:1895 (sleep mutex:so_rcv)
     5           20            0           7     2     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:365 (rw:unp_global_rwlock)
     5           77            0          33     2     0            0            0 /usr/src/sys/kern/kern_descrip.c:409 (sx:filedesc structure)
    11           81            0          10     8     0            0            0 /usr/src/sys/kern/kern_proc.c:425 (sleep mutex:process group)
   418         3199           63          15   213     4            5            2 /usr/src/sys/amd64/amd64/pmap.c:2949 (sleep mutex:vm page queue mutex)
     9           32            0          10     3     0            0            0 /usr/src/sys/kern/kern_fork.c:691 (sleep mutex:process lock)
    51          644            0          16    40     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:549 (sleep mutex:inp)
     9           29            0           4     7     0            0            0 /usr/src/sys/kern/kern_proc.c:457 (sleep mutex:process group)
  2380      4079839      3754994       65658    62    57         1997         2784 /usr/src/sys/kern/vfs_lookup.c:411 (lockmgr:ufs)
     5           16            0           4     4     0            0            0 /usr/src/sys/kern/kern_proc.c:476 (sleep mutex:process group)
     9           58            0          10     5     0            0            0 /usr/src/sys/kern/kern_exit.c:297 (sx:proctree)
    13          112            0          17     6     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:1014 (sleep mutex:udp)
    11          481            0         179     2     0            0            0 /usr/src/sys/vm/vm_object.c:1214 (sleep mutex:vm object)
     2            5            0           3     1     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:4096)
     6           95          156          35     2     4           15           17 /usr/src/sys/kern/kern_resource.c:1041 (spin mutex:sched lock)
     5           51            0          16     3     0            0            0 /usr/src/sys/kern/uipc_socket.c:961 (sleep mutex:so_snd)
     4           11            0           3     3     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:1024)
    38          500            0          35    14     0            0            0 /usr/src/sys/kern/kern_resource.c:1059 (spin mutex:process slock)
     6          358            0         120     2     0            0            0 /usr/src/sys/netinet/ip_input.c:1086 (sleep mutex:ipqlock)
   229         2126            0         523     4     0            0            0 /usr/src/sys/netinet/in_pcb.c:217 (sleep mutex:inp)
    25         2555            0         449     5     0            0            0 /usr/src/sys/kern/vfs_bio.c:2460 (sleep mutex:vnode interlock)
    15          150            0          17     8     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:1052 (sleep mutex:udp)
     8          426            0         157     2     0            0            0 /usr/src/sys/vm/vm_object.c:1247 (sleep mutex:vm object)
    30          540            0          60     9     0            0            0 /usr/src/sys/kern/kern_mutex.c:141 (sleep mutex:Syncer mtx)
     5           12            0           3     4     0            0            0 /usr/src/sys/vm/uma_core.c:414 (sleep mutex:128)
     5           10            0           4     2     0            0            0 /usr/src/sys/kern/uipc_socket.c:2016 (sleep mutex:so_rcv)
     2           15            0           7     2     0            0            0 /usr/src/sys/kern/uipc_usrreq.c:498 (rw:unp_global_rwlock)
    88         1068            0          16    66     0            0            0 /usr/src/sys/netinet/udp_usrreq.c:1072 (sleep mutex:udp)
     6           45            0          13     3     0            0            0 /usr/src/sys/kern/vfs_bio.c:3213 (sleep mutex:vm object)


More information about the freebsd-stable mailing list