FreeBSD 6.3 deadlock (vm_map?) with DDB output

Stef Walter stef-list at memberwebs.com
Sun Jun 15 11:23:20 UTC 2008


I've been trying to track down a deadlock on some newish production
servers running FreeBSD 6.3-RELEASE-p2. The deadlock occurs on a
specific (although mundane) hardware configuration, and each of several
servers running this hardware deadlock about once per week.

Although I suspect that this is not hardware related, from a (naive)
perusal of the attached stack traces.

Forgive me if my interpretation of this is all wrong, but I'm pretty
desperate for help. So here's my basic understanding of the deadlock:

These processes seem to be waiting on the page queue mutex:
 sendmail (in vm_mmap > vm_map_find > vm_map_insert > vm_map_pmap_enter)
 bsnmpd (in malloc, uma_large_malloc > page_alloc > kmem_malloc)
 httpd (in trap > trap_pfault > vm_fault)
 [g_up] (in g_vfs_done > bufdone)

The page queue mutex is held by rsync process:
 rsync (in trap > trap_pfault > vm_fault > pmap_enter)

Rsync kernel process (in pmap_enter) was interrupted while holding the
page queue lock?


Giant is enabled in loader.conf due to the needs of the pf firewall when
dealing with user credentials lookups. I do not believe that Giant plays
into this deadlock. Kernel config attached.

Any and all help or info is welcome. Thanks in advance.

Stef Walter
-------------- next part --------------


# --------------------------------------------------------------------
# FREEBSD BUILD 

FreeBSD m2.npubs.com 6.3-RELEASE-p2 FreeBSD 6.3-RELEASE-p2 #0: Tue Jun 10 02:06:41 UTC 2008     nate at m2.npubs.com:/usr/obj/usr/src/sys/RACK2DBG  i386

# --------------------------------------------------------------------
# 

# --------------------------------------------------------------------
# PROCESS LIST

db> ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
35638 35637  4448    80  LJ     *vm page  0xc8d5f780 sendmail
35637 40546  4448    80  SJ      wait     0xc9e79218 sh
35620  1801  1801    80  SJ      lockf    0xcceb4d00 httpd
35593  1801  1801    80  SJ      lockf    0xcce46900 httpd
35566  1801  1801    80  SJ      kqread   0xc8f8e000 httpd
35438 15070 15063    70  SJ      sbwait   0xd017220c postgres
35401  4527  4527 10219  SJ      select   0xc080b7e4 cleanup
35337  4527  4527 10219  SJ      select   0xc080b7e4 pickup
35144  4413  4413    80  SJ      select   0xc080b7e4 httpd
34797  8611  8611     0  SJ      select   0xc080b7e4 virtual
34784  8611  8611   125  SJ      select   0xc080b7e4 smtpd
34781  8611  8611   125  SJ      select   0xc080b7e4 smtp
34761  8611  8611   125  SJ      select   0xc080b7e4 proxymap
34760  8611  8611   125  SJ      select   0xc080b7e4 smtpd
34757  8611  8611   125  SJ      select   0xc080b7e4 smtp
34750  8611  8611   125  SJ      select   0xc080b7e4 cleanup
34087  6435  6435   125  SJ      kqread   0xcae9f900 anvil
34075  6435  6435   125  SJ      kqread   0xcfb5b600 smtpd
33801 15070 15063    70  SJ      sbwait   0xd01a8bc8 postgres
33796  4527  4527 10219  SJ      select   0xc080b7e4 bounce
33423  1801  1801    80  SJ      select   0xc080b7e4 httpd
33417 15070 15063    70  SJ      sbwait   0xd087f638 postgres
33392  1801  1801    80  SJ      select   0xc080b7e4 httpd
33207  4527  4527 10219  SJ      select   0xc080b7e4 flush
33200  4527  4527 10219  SJ      select   0xc080b7e4 bounce
33125  4527  4527 10219  SJ      kqread   0xcb2ff900 smtp
33118  4527  4527 10219  SJ      select   0xc080b7e4 smtp
33113  4527  4527 10219  SJ      select   0xc080b7e4 smtp
33108  4527  4527 10219  SJ      select   0xc080b7e4 smtp
33101  4527  4527 10219  SJ      select   0xc080b7e4 smtp
33094  4527  4527 10219  SJ      select   0xc080b7e4 smtp
33090  4527  4527 10219  SJ      select   0xc080b7e4 smtp
33070 33067 33067     0  LJ     *Giant    0xcaaf97c0 lynx
33067 33066 33067     0  SsJ     wait     0xc9fd9648 sh
33066  4564  4564     0  SJ      piperd   0xd0ab7000 cron
33056  4527  4527 10219  SJ      select   0xc080b7e4 smtp
33055  4527  4527 10219  SJ      lockf    0xc992f480 bounce
32994 15070 15063    70  SJ      sbwait   0xc9cdc370 postgres
32976 15070 15063    70  SJ      sbwait   0xd0f8e4d4 postgres
32971  4527  4527 10219  SJ      select   0xc080b7e4 proxymap
32970  4527  4527 10219  SJ      select   0xc080b7e4 smtpd
32869 15070 15063    70  SJ      sbwait   0xc8fcc370 postgres
32852  9298  9298 10102  SJ      select   0xc080b7e4 cleanup
32837 32836 32836     0  S       select   0xc080b7e4 rsync
32836  7802 32836     0  RLs                         rsync
32774  4527  4527     0  SJ      select   0xc080b7e4 local
32754  1801  1801    80  LLJ    *vm page  0xc8d5f780 httpd
32753 15070 15063    70  SJ      sbwait   0xc9c8ce90 postgres
32747 15070 15063    70  SJ      sbwait   0xd01a8370 postgres
32739 15070 15063    70  SJ      sbwait   0xccdd3d2c postgres
32710     1 32710     0  Ss      biord    0xdcb814d4 jail-measure
32633  1801  1801    80  SJ      select   0xc080b7e4 httpd
32623  1801  1801    80  SJ      select   0xc080b7e4 httpd
32599  1801  1801    80  SJ      select   0xc080b7e4 httpd
32598  1801  1801    80  SJ      select   0xc080b7e4 httpd
32597  1801  1801    80  SJ      select   0xc080b7e4 httpd
32596  1801  1801    80  SJ      lockf    0xcceb43c0 httpd
32595  1801  1801    80  SJ      select   0xc080b7e4 httpd
32301  4413  4413    80  SJ      lockf    0xcce44680 httpd
32255  4413  4413    80  SJ      lockf    0xcce44340 httpd
32206 15070 15063    70  SJ      sbwait   0xc9cdf638 postgres
32163  1801  1801    80  SJ      select   0xc080b7e4 httpd
32147  4527  4527 10219  SJ      select   0xc080b7e4 trivial-rewrite
31910  8611  8611   125  SJ      select   0xc080b7e4 smtpd
31740  8611  8611   125  SJ      select   0xc080b7e4 trivial-rewrite
30378  4413  4413    80  SJ      lockf    0xcd551b40 httpd
30355  4413  4413    80  SJ      lockf    0xcceb41c0 httpd
29801  4413  4413    80  SJ      lockf    0xcb331340 httpd
29375  9298  9298 10102  SJ      select   0xc080b7e4 pickup
28283  4413  4413    80  SJ      lockf    0xcb3317c0 httpd
28247  4413  4413    80  SJ      kqread   0xcb092200 httpd
28113  9298  9298 10102  SJ      select   0xc080b7e4 smtpd
28067  4413  4413    80  SJ      lockf    0xd0ff87c0 httpd
26846  4413  4413    80  SJ      select   0xc080b7e4 httpd
26581  9298  9298 10102  SJ      select   0xc080b7e4 smtpd
26529 26528  8216   125  SJ      select   0xc080b7e4 imapd
26528  8216  8216     0  SJ      select   0xc080b7e4 couriertls
26424  4413  4413    80  SJ      lockf    0xc92fe900 httpd
23902 15070 15063    70  SJ      sbwait   0xccdcb0a8 postgres
23891  1801  1801    80  SJ      select   0xc080b7e4 httpd
23721  8611  8611   125  SJ      select   0xc080b7e4 pickup
22472  6435  6435   125  SJ      kqread   0xcfb0cb00 pickup
21951  8958  8958   125  SJ      kqread   0xcfb5b200 pickup
19724  3711  3711   125  SJ      kqread   0xcfbee800 pickup
17335  9298  9298 10102  SJ      select   0xc080b7e4 proxymap
17334  9298  9298 10102  SJ      select   0xc080b7e4 trivial-rewrite
11733 40103 40103    80  SJ      accept   0xd087d03a httpd
11732 40103 40103    80  SJ      accept   0xd087d03a httpd
11731 40103 40103    80  SJ      accept   0xd087d03a httpd
11730 40103 40103    80  SJ      accept   0xd087d03a httpd
11728 40103 40103    80  SJ      accept   0xd087d03a httpd
11727 40103 40103    80  SJ      accept   0xd087d03a httpd
11726 40103 40103    80  SJ      accept   0xd087d03a httpd
 2181     1  2181   181  SsJ     (threaded)          nagios
100858                   S       select   0xc080b7e4 nagios
100434                   S       nanslp   0xc07be46c nagios
87785  7530  7530    80  SJ      accept   0xca004b5a httpd
61004  8697  8697    80  SJ      accept   0xca96519e httpd
56340  1917  1917    80  SJ      lockf    0xcb3311c0 httpd
53973  8611  8611   125  SJ      select   0xc080b7e4 anvil
52531 17166 52531     0  R+J     CPU 3               ee
45059  7530  7530    80  SJ      accept   0xca004b5a httpd
41862  4448  4448    80  SJ      accept   0xc9c7d5ca httpd
41181  4448  4448    80  SJ      accept   0xc9c7d5ca httpd
41180  4448  4448    80  SJ      accept   0xc9c7d5ca httpd
41179  4448  4448    80  SJ      accept   0xc9c7d5ca httpd
41178  7530  7530    80  SJ      accept   0xca004b5a httpd
41175  4448  4448    80  SJ      accept   0xc9c7d5ca httpd
40748  9036  9036    80  SJ      accept   0xca965466 httpd
40742  9036  9036    80  SJ      accept   0xca965466 httpd
40739  9036  9036    80  SJ      accept   0xca965466 httpd
40726  9036  9036    80  SJ      accept   0xca965466 httpd
40546  4448  4448    80  SJ      wait     0xd024c000 httpd
40444  1917  1917    80  SJ      lockf    0xcaa00580 httpd
40443  1917  1917    80  SJ      kqread   0xcf271100 httpd
40442  1917  1917    80  SJ      lockf    0xcce44500 httpd
40441  1917  1917    80  SJ      lockf    0xcd551480 httpd
40440  1917  1917    80  SJ      lockf    0xc8dba6c0 httpd
40428  8697  8697    80  SJ      accept   0xca96519e httpd
40427  8697  8697    80  SJ      accept   0xca96519e httpd
40426  8697  8697    80  SJ      accept   0xca96519e httpd
40425  8697  8697    80  SJ      accept   0xca96519e httpd
40424  8697  8697    80  SJ      accept   0xca96519e httpd
40384  9036  9036    80  SJ      accept   0xca965466 httpd
40378  7530  7530    80  SJ      accept   0xca004b5a httpd
40377  7530  7530    80  SJ      accept   0xca004b5a httpd
40376  7530  7530    80  SJ      accept   0xca004b5a httpd
40375  7530  7530    80  SJ      accept   0xca004b5a httpd
40374  7530  7530    80  SJ      accept   0xca004b5a httpd
40372  9036  9036    80  SJ      accept   0xca965466 httpd
40371  9036  9036    80  SJ      accept   0xca965466 httpd
40370  9036  9036    80  SJ      accept   0xca965466 httpd
40369  9036  9036    80  SJ      accept   0xca965466 httpd
40368  9036  9036    80  SJ      accept   0xca965466 httpd
40334 40332  4448     0  SJ      piperd   0xcca2e198 cronolog
40333  4448  4448    80  SJ      accept   0xc9c7d5ca httpd
40332  4448  4448     0  SJ      wait     0xcb2cf218 sh
40310 40309 40309     0  SJ      piperd   0xc9257cc0 sockin
40309  8742 40309     0  SsJ     wait     0xd0171430 sh
40294 40291 40291     0  SJ      piperd   0xc8fe7cc0 sockin
40291  6919 40291     0  SsJ     wait     0xcdccba78 sh
36263     1 36263     0  SsJ     (threaded)          rrdbotd
100452                   S       kserel   0xce155d54 rrdbotd
100188                   S       select   0xc080b7e4 rrdbotd
100984                   S       kserel   0xce155d54 rrdbotd
100311                   S       kserel   0xce155d54 rrdbotd
100271                   S       select   0xc080b7e4 rrdbotd
100268                   S       kserel   0xce155d54 rrdbotd
100624                   S       ksesigwa 0xcaa27780 rrdbotd
17166 17165 17166     0  S+J     wait     0xc994d000 bash
17165 17155 17152     0  S+J     wait     0xd0d11218 su
17155 17152 17152     0  S+      wait     0xc9ec0a78 sh
17152  1980 17152     0  S+      wait     0xc8a42648 sh
15079 15078 15063    70  S+J     select   0xc080b7e4 postgres
15078 15070 15063    70  S+J     select   0xc080b7e4 postgres
15077 15070 15063    70  S+J     select   0xc080b7e4 postgres
15074 15070 15063    70  S+J     select   0xc080b7e4 postgres
15070     1 15063    70  S+J     select   0xc080b7e4 postgres
 1980     1  1980     0  S+      wait     0xcaa27430 bash
31591  2354  2354    80  SJ      lockf    0xc8c8ee80 httpd
31578  2354  2354    80  SJ      lockf    0xc8c8e640 httpd
40103     1 40103     0  SsJ     nanslp   0xc07be46c httpd
38293     1 38293     0  SsJ     nanslp   0xc07be46c cron
38286     1 38286     0  SsJ     select   0xc080b7e4 sshd
38224     1 38224     0  SsJ     select   0xc080b7e4 syslogd
91887     1 91886     0  SJ      select   0xc080b7e4 ruby18
91876     1 91875     0  SJ      select   0xc080b7e4 ruby18
61971 61968  8645 65534  SJ      piperd   0xcd2d3000 cat
61970 61967  8645 65534  SJ      piperd   0xcca51cc0 cat
61968     1  8645 65534  SJ      wait     0xce938c90 sh
61967     1  8645 65534  SJ      wait     0xc9d2c860 sh
64190  2354  2354    80  SJ      kqread   0xcbe8e500 httpd
19433 19430  8645 65534  SJ      piperd   0xcffe74c8 cat
19432 19431  8645 65534  SJ      piperd   0xcad1d000 cat
19431     1  8645 65534  SJ      wait     0xca5b1860 sh
19430     1  8645 65534  SJ      wait     0xd0195430 sh
72427  3606  3606    80  SJ      select   0xc080b7e4 httpd
95713  3606  3606    80  SJ      lockf    0xcd551800 httpd
79740  3606  3606    80  SJ      lockf    0xc8c8e7c0 httpd
36833     1 36833     0  SsJ     select   0xc080b7e4 ruby
15574  3606  3606    80  SJ      lockf    0xcaa00680 httpd
13381  6677  6677    80  SJ      accept   0xc9df4466 httpd
11234  9367  9367    80  SJ      lockf    0xc8d16a80 httpd
10994  9367  9367    80  SJ      lockf    0xd00c2900 httpd
 7909  6752  6677    80  SJ      select   0xc080b7e4 ruby
73129  6677  6677    80  SJ      accept   0xc9df4466 httpd
58056  6752  6677    80  SJ      select   0xc080b7e4 ruby
41698  3606  3606    80  SJ      lockf    0xc8a53740 httpd
41694  8569  8569    80  SJ      accept   0xca965b5a httpd
37070  3266  3266    80  SJ      kqread   0xcadb3700 httpd
37069  3266  3266    80  SJ      lockf    0xcb7f4280 httpd
37068  3266  3266    80  SJ      lockf    0xcec83700 httpd
30486  9367  9367    80  SJ      lockf    0xd0ff8800 httpd
30262  9367  9367    80  SJ      lockf    0xca6e3d40 httpd
11692  8743  8743    80  SJ      lockf    0xcb7f4b80 httpd
11690  8743  8743    80  SJ      select   0xc080b7e4 httpd
11599  8743  8743    80  SJ      lockf    0xcb331bc0 httpd
11214  8743  8743    80  SJ      lockf    0xcb0df340 httpd
11169  8743  8743    80  SJ      lockf    0xd0043c00 httpd
 9660  6677  6677    80  SJ      accept   0xc9df4466 httpd
 9659  6752  6677    80  SJ      select   0xc080b7e4 ruby
 9445  3266  3266    80  SJ      lockf    0xcb0df900 httpd
 9407  8611  8611   125  SJ      select   0xc080b7e4 tlsmgr
 9404  2354  2354    80  SJ      lockf    0xc95798c0 httpd
 9401  9367  9367    80  SJ      lockf    0xcb7f4ac0 httpd
 9400  9367  9367    80  SJ      lockf    0xc95628c0 httpd
 9399  9367  9367    80  SJ      lockf    0xd05bd080 httpd
 9398  9367  9367    80  SJ      kqread   0xca619400 httpd
 9397  9367  9367    80  SJ      lockf    0xc9b5b440 httpd
 9382     1  9382     0  SsJ     nanslp   0xc07be46c cron
 9375     1  9375     0  SsJ     select   0xc080b7e4 sshd
 9367     1  9367     0  SsJ     nanslp   0xc07be46c httpd
 9359  3266  3266    80  SJ      lockf    0xc957a0c0 httpd
 9334  9333  8760    70  SJ      select   0xc080b7e4 postgres
 9333  8760  8760    70  SJ      select   0xc080b7e4 postgres
 9332  8760  8760    70  SJ      select   0xc080b7e4 postgres
 9328     1  9328     0  SsJ     nanslp   0xc07be46c cron
 9320     1  9320     0  SsJ     select   0xc080b7e4 inetd
 9315     1  9315     0  SsJ     accept   0xcaec119e saslauthd1
 9304  9298  9298 10102  SJ      select   0xc080b7e4 qmgr
 9298     1  9298     0  SsJ     select   0xc080b7e4 master
 9268  2354  2354    80  SJ      lockf    0xcec83ac0 httpd
 9241     1  9241     0  SsJ     select   0xc080b7e4 bsnmpd
 9214     1  9214     0  SsJ     nanslp   0xc07be46c cron
 9202     1  9202     0  SsJ     select   0xc080b7e4 sshd
 9139  9107  9090    88  SJ      (threaded)          mysqld
100427                   S       kserel   0xc9a0b4b4 mysqld
100308                   S       kserel   0xc9a0b4b4 mysqld
100276                   S       kserel   0xc9a0b4b4 mysqld
100582                   S       kserel   0xc9a0b4b4 mysqld
100572                   S       select   0xc080b7e4 mysqld
100568                   S       kserel   0xcadf8394 mysqld
100571                   S       sigwait  0xeca04c14 mysqld
100564                   S       ksesigwa 0xc9b7f780 mysqld
 9107     1  9090    88  SJ      wait     0xcac42a78 sh
 9093     1  9093   225  SsJ     select   0xc080b7e4 perl5.8.8
 9068     1  9068     0  SsJ     nanslp   0xc07be46c cron
 9058     1  9058     0  SsJ     select   0xc080b7e4 sshd
 9042     1    41     0  SJ      piperd   0xcad1dcc0 logger
 9041     1    41     0  SJ      fifoor   0xca9e6c58 tail
 9036     1  9036     0  SsJ     select   0xc080b7e4 httpd
 9035  8996  8980    88  SJ      (threaded)          mysqld
100855                   S       kserel   0xc8cee394 mysqld
101020                   S       sbwait   0xc950fd2c mysqld
100994                   S       select   0xc080b7e4 mysqld
101012                   S       sbwait   0xcce6320c mysqld
101021                   S       sbwait   0xd08750a8 mysqld
100393                   S       kserel   0xc8cee394 mysqld
100999                   S       sbwait   0xd0eab370 mysqld
101189                   S       sbwait   0xd0aa1bc8 mysqld
100578                   S       kserel   0xc8cee394 mysqld
101017                   S       kserel   0xc8cee394 mysqld
100573                   S       kserel   0xcadf82d4 mysqld
100558                   S       sigwait  0xec83bc14 mysqld
100555                   S       ksesigwa 0xc97f4780 mysqld
 8996     1  8980    88  SJ      wait     0xca0c4860 sh
 8963  8958  8958   125  SJ      kqread   0xcabf6400 qmgr
 8958     1  8958     0  SsJ     kqread   0xc8d71500 master
 8893     1  8893     0  SsJ     nanslp   0xc07be46c cron
 8874     1  8874     0  SsJ     select   0xc080b7e4 sshd
 8844     1  8844     0  SsJ     select   0xc080b7e4 sshd
 8811     1  8811     0  SsJ     nanslp   0xc07be46c cron
 8804  8569  8569    80  SJ      accept   0xca965b5a httpd
 8802  8569  8569    80  SJ      accept   0xca965b5a httpd
 8801  8569  8569    80  SJ      accept   0xca965b5a httpd
 8799  8569  8569    80  SJ      accept   0xca965b5a httpd
 8798     1  8798     0  SsJ     select   0xc080b7e4 sshd
 8797  8569  8569    80  SJ      accept   0xca965b5a httpd
 8791     1  8791     0  SsJ     select   0xc080b7e4 proftpd
 8769  8743  8743    80  SJ      lockf    0xcd6aeb00 httpd
 8768  8743  8743    80  SJ      lockf    0xcb7f4880 httpd
 8767  8743  8743    80  SJ      lockf    0xcb06a580 httpd
 8766  8743  8743    80  SJ      lockf    0xcb0df140 httpd
 8765  8743  8743    80  SJ      lockf    0xc8fb1700 httpd
 8760     1  8760    70  SsJ     select   0xc080b7e4 postgres
 8743     1  8743     0  SsJ     select   0xc080b7e4 httpd
 8742     1  8742     0  SsJ     select   0xc080b7e4 syslogd
 8715     1  8715     0  SsJ     nanslp   0xc07be46c cron
 8706     1  8706     0  SsJ     select   0xc080b7e4 sshd
 8697     1  8697     0  SsJ     nanslp   0xc07be46c httpd
 8645     1  8645 65534  SsJ     (threaded)          proxsmtpd
100309                   S       kserel   0xca663154 proxsmtpd
100975                   S       kserel   0xca663154 proxsmtpd
101018                   S       kserel   0xca663154 proxsmtpd
100649                   S       kserel   0xca663154 proxsmtpd
101013                   S       accept   0xca9655ca proxsmtpd
100581                   S       ksesigwa 0xcaaf7138 proxsmtpd
 8617  8611  8611   125  SJ      select   0xc080b7e4 qmgr
 8611     1  8611     0  SsJ     select   0xc080b7e4 master
 8569     1  8569     0  SsJ     nanslp   0xc07be46c httpd
 8496     1  8496     0  SsJ     select   0xc080b7e4 syslogd
 8437     1  8437     0  SsJ     select   0xc080b7e4 syslogd
 8300  8170  8169     0  SJ      select   0xc080b7e4 authdaemond
 8299  8170  8169     0  SJ      select   0xc080b7e4 authdaemond
 8298  8170  8169     0  SJ      select   0xc080b7e4 authdaemond
 8295     1  8295     0  SsJ     select   0xc080b7e4 syslogd
 8272  8271  8272     0  SJ      select   0xc080b7e4 couriertcpd
 8271     1  8271     0  SJ      piperd   0xca366b28 courierlogger
 8253  8252  8253     0  SJ      select   0xc080b7e4 couriertcpd
 8252     1  8252     0  SJ      piperd   0xc904f990 courierlogger
 8229  8228  8229     0  SJ      select   0xc080b7e4 couriertcpd
 8228     1  8228     0  SJ      piperd   0xc9e1cb28 courierlogger
 8216  8215  8216     0  SJ      select   0xc080b7e4 couriertcpd
 8215     1  8215     0  SJ      piperd   0xca610330 courierlogger
 8170  8169  8169     0  SJ      select   0xc080b7e4 authdaemond
 8169     1  8169     0  SJ      piperd   0xc9dc1cc0 courierlogger
 8121     1  8121     0  SsJ     nanslp   0xc07be46c cron
 8105     1  8105     0  SsJ     select   0xc080b7e4 sshd
 8082     1  8082     0  SsJ     select   0xc080b7e4 syslogd
 8081     1  8081     0  SsJ     select   0xc080b7e4 syslogd
 7979     1  7979     0  SsJ     nanslp   0xc07be46c cron
 7972     1  7972     0  SsJ     select   0xc080b7e4 sshd
 7848     1  7848     0  Ss+     ttyin    0xc8a6e410 getty
 7847     1  7847     0  Ss+     ttyin    0xc8a74810 getty
 7846     1  7846     0  Ss+     ttyin    0xc8a73010 getty
 7845     1  7845     0  Ss+     ttyin    0xc8a73410 getty
 7844     1  7844     0  Ss+     ttyin    0xc8a70c10 getty
 7843     1  7843     0  Ss+     ttyin    0xc8a73c10 getty
 7842     1  7842     0  Ss+     ttyin    0xc8a76010 getty
 7841     1  7841     0  Ss+     ttyin    0xc8a75010 getty
 7840     1  7840     0  Ss+     ttyin    0xc8a74410 getty
 7829     1  7829     0  Ls     *vm page  0xc8d5f780 bsnmpd
 7802     1  7802     0  Ss      select   0xc080b7e4 inetd
 7695     1  7695     0  SsJ     select   0xc080b7e4 syslogd
 7530     1  7530     0  SsJ     nanslp   0xc07be46c httpd
 7302     1  7302     0  SsJ     select   0xc080b7e4 bsnmpd
 6919     1  6919     0  SsJ     select   0xc080b7e4 syslogd
 6754  6677  6677    80  SJ      accept   0xc9df4466 httpd
 6753  6677  6677    80  SJ      accept   0xc9df4466 httpd
 6752  6677  6677    80  SJ      select   0xc080b7e4 httpd
 6694     1  6694     0  SsJ     nanslp   0xc07be46c cron
 6687     1  6687     0  SsJ     select   0xc080b7e4 sshd
 6677     1  6677     0  SsJ     nanslp   0xc07be46c httpd
 6639  6578  6577    88  S+J     piperd   0xca3517f8 logger
 6638  6578  6577    88  S+J     (threaded)          mysqld
100566                   S       kserel   0xc8e765d4 mysqld
100383                   S       kserel   0xc8e765d4 mysqld
100988                   S       kserel   0xc8e765d4 mysqld
101056                   S       kserel   0xc8e765d4 mysqld
101005                   S       select   0xc080b7e4 mysqld
100392                   S       sbwait   0xcceda370 mysqld
100387                   S       sigwait  0xec6adc14 mysqld
100381                   S       ksesigwa 0xc9a09568 mysqld
 6578     1  6577    88  S+J     wait     0xca0c6430 sh
 6568  6567  6568     0  S+J     select   0xc080b7e4 couriertcpd
 6567     1  6567     0  S+J     piperd   0xca365660 courierlogger
 6509  6508  6509     0  S+J     select   0xc080b7e4 couriertcpd
 6508     1  6508     0  S+J     piperd   0xc99fe990 courierlogger
 6505     1  6505     0  SsJ     select   0xc080b7e4 syslogd
 6454  6435  6435   125  SJ      kqread   0xca2a1400 qmgr
 6435     1  6435     0  SsJ     kqread   0xca122500 master
 5653  5529  5521     0  S+J     select   0xc080b7e4 authdaemond
 5652  5529  5521     0  S+J     select   0xc080b7e4 authdaemond
 5651  5529  5521     0  S+J     select   0xc080b7e4 authdaemond
 5650  5529  5521     0  S+J     select   0xc080b7e4 authdaemond
 5649  5529  5521     0  S+J     select   0xc080b7e4 authdaemond
 5529  5521  5521     0  S+J     select   0xc080b7e4 authdaemond
 5521     1  5521     0  S+J     piperd   0xc8ee7cc0 courierlogger
 5337     1  5337     0  SsJ     select   0xc080b7e4 syslogd
 5166     1  5166     0  SsJ     nanslp   0xc07be46c cron
 5153     1  5153     0  SsJ     select   0xc080b7e4 sshd
 5117     1  5117     0  SsJ     (threaded)          httpauthd
100563                   S       kserel   0xc8cee2d4 httpauthd
100652                   S       kserel   0xc8cee2d4 httpauthd
100468                   S       kserel   0xc8cee2d4 httpauthd
100444                   S       kserel   0xc8cee2d4 httpauthd
100183                   S       sbwait   0xc9df4bc8 httpauthd
100277                   S       sbwait   0xcb305370 httpauthd
100382                   S       sbwait   0xcafa0638 httpauthd
100314                   S       sbwait   0xcb1054d4 httpauthd
100499                   S       sbwait   0xcae5aa64 httpauthd
100223                   S       sbwait   0xcae4d4d4 httpauthd
100310                   S       sbwait   0xcb0bfd2c httpauthd
100973                   S       sbwait   0xcebd34d4 httpauthd
100388                   S       sbwait   0xcae4d20c httpauthd
100646                   S       sbwait   0xd0aa0900 httpauthd
100658                   S       sbwait   0xc8d3779c httpauthd
100471                   S       sbwait   0xcae4ca64 httpauthd
100985                   S       sbwait   0xc9ce420c httpauthd
100175                   S       sbwait   0xd0cdae90 httpauthd
100317                   S       sbwait   0xcebce4d4 httpauthd
100580                   S       sbwait   0xcb105638 httpauthd
100587                   S       accept   0xc9e0319e httpauthd
100272                   S       sbwait   0xcb0aa0a8 httpauthd
100556                   S       sbwait   0xcae59e90 httpauthd
100562                   S       sbwait   0xcb154e90 httpauthd
100306                   S       sbwait   0xd0173e90 httpauthd
100221                   S       sbwait   0xcebc4370 httpauthd
100448                   S       sbwait   0xd01744d4 httpauthd
100380                   S       sbwait   0xca004638 httpauthd
100318                   S       sbwait   0xc9c7da64 httpauthd
100560                   S       sbwait   0xcaec0370 httpauthd
100205                   S       sbwait   0xc9dd9e90 httpauthd
100394                   S       sbwait   0xcb0aaa64 httpauthd
100251                   S       sbwait   0xcabc120c httpauthd
101072                   S       sbwait   0xd07560a8 httpauthd
100575                   S       sbwait   0xccdd7bc8 httpauthd
101207                   S       sbwait   0xcfd28370 httpauthd
101211                   S       sbwait   0xd087c370 httpauthd
100469                   S       sbwait   0xd0177d2c httpauthd
101004                   S       sbwait   0xc8fa0638 httpauthd
100253                   S       sbwait   0xc9e04d2c httpauthd
100585                   S       sbwait   0xca265a64 httpauthd
100270                   S       sbwait   0xcaf8f20c httpauthd
100313                   S       sbwait   0xd01780a8 httpauthd
100673                   S       sbwait   0xcaf8ce90 httpauthd
100651                   S       sbwait   0xcae6f638 httpauthd
100324                   S       sbwait   0xcfd27370 httpauthd
100259                   S       sbwait   0xc9ce0638 httpauthd
100391                   S       sbwait   0xc9dd9370 httpauthd
100321                   S       sbwait   0xd01a8900 httpauthd
100476                   S       sbwait   0xcafa0a64 httpauthd
101190                   S       sbwait   0xcaf8fbc8 httpauthd
100316                   S       sbwait   0xc9ce379c httpauthd
100657                   S       sbwait   0xcb090638 httpauthd
100594                   S       sbwait   0xd01720a8 httpauthd
100559                   S       sbwait   0xd01a9638 httpauthd
100583                   S       sbwait   0xcaf8b900 httpauthd
100574                   S       sbwait   0xcb154a64 httpauthd
100364                   S       sbwait   0xc9cd8bc8 httpauthd
100567                   S       sbwait   0xcf485e90 httpauthd
100971                   S       sbwait   0xca254900 httpauthd
101016                   S       sbwait   0xccdf2d2c httpauthd
100977                   S       sbwait   0xcb0aa638 httpauthd
100447                   S       sbwait   0xc9cd9d2c httpauthd
100856                   S       sbwait   0xd0176d2c httpauthd
100586                   S       sbwait   0xcebcfe90 httpauthd
100249                   S       sbwait   0xc8d3b370 httpauthd
100972                   S       sbwait   0xc8d3ba64 httpauthd
100450                   S       sbwait   0xcaa86370 httpauthd
100384                   S       sbwait   0xcebcebc8 httpauthd
100981                   S       sbwait   0xccdcba64 httpauthd
100591                   S       sbwait   0xd01760a8 httpauthd
100998                   S       sbwait   0xccdf279c httpauthd
100320                   S       sbwait   0xcab88d2c httpauthd
101031                   S       sbwait   0xcdb35a64 httpauthd
100349                   S       sbwait   0xcf4a7bc8 httpauthd
100589                   S       sbwait   0xcaf8b79c httpauthd
100386                   S       sbwait   0xd0cd60a8 httpauthd
100979                   S       sbwait   0xcae4de90 httpauthd
100590                   S       sbwait   0xccdcbd2c httpauthd
100569                   S       sbwait   0xcacd2bc8 httpauthd
101008                   S       sbwait   0xcebd3638 httpauthd
100672                   S       sbwait   0xcb153900 httpauthd
100557                   S       sbwait   0xcb153a64 httpauthd
100541                   S       sbwait   0xcae59bc8 httpauthd
100389                   S       sbwait   0xca265900 httpauthd
100415                   S       sbwait   0xcaf8e0a8 httpauthd
100577                   S       sbwait   0xce18cbc8 httpauthd
101025                   S       sbwait   0xcf4a7d2c httpauthd
100561                   S       sbwait   0xcebd0d2c httpauthd
100385                   S       sbwait   0xd01774d4 httpauthd
100256                   S       sbwait   0xcceb74d4 httpauthd
100284                   S       sbwait   0xcaf8cd2c httpauthd
101040                   S       sbwait   0xc9cdcbc8 httpauthd
100319                   S       sbwait   0xd0172d2c httpauthd
100274                   S       sbwait   0xcb0bfe90 httpauthd
100269                   S       sbwait   0xd0cd84d4 httpauthd
101011                   S       sbwait   0xcb3294d4 httpauthd
101069                   S       sbwait   0xd0cdb79c httpauthd
100449                   S       sbwait   0xcce6379c httpauthd
100970                   S       sbwait   0xc8f75a64 httpauthd
101002                   S       sbwait   0xcb2de0a8 httpauthd
100983                   S       sbwait   0xc8d3b638 httpauthd
100312                   S       ksesigwa 0xc97f4bb0 httpauthd
 5000     1  5000   389  SsJ     (threaded)          slapd
100993                   S       kserel   0xc97f6c34 slapd
100445                   S       kserel   0xc97f6c34 slapd
100694                   S       kserel   0xc97f6c34 slapd
100247                   S       kserel   0xc97f6c34 slapd
100857                   S       select   0xc080b7e4 slapd
100305                   S       ksesigwa 0xc9e24138 slapd
 4867     1  4867     0  SsJ     select   0xc080b7e4 inetd
 4847     1  4847     0  SsJ     accept   0xc9c7de22 svnserve
 4835     1  4834     0  SJ      select   0xc080b7e4 ruby18
 4796     1  4796     0  SsJ     select   0xc080b7e4 syslogd
 4712     1  4711     0  SJ      select   0xc080b7e4 ruby18
 4707     1  4706     0  SJ      select   0xc080b7e4 ruby18
 4701     1  4700     0  SJ      select   0xc080b7e4 ruby18
 4648     1  4647     0  SJ      select   0xc080b7e4 ruby18
 4564     1  4564     0  SsJ     nanslp   0xc07be46c cron
 4555     1  4555     0  SsJ     select   0xc080b7e4 inetd
 4550     1  4550     0  SsJ     accept   0xc9c7dcbe saslauthd1
 4542     1  4541     0  SJ      select   0xc080b7e4 ruby18
 4535  4527  4527 10219  SJ      select   0xc080b7e4 qmgr
 4527     1  4527     0  SsJ     select   0xc080b7e4 master
 4454     1    41     0  S+J     piperd   0xc8ee07f8 logger
 4453     1    41     0  S+J     fifoor   0xc8d04a38 tail
 4448     1  4448     0  SsJ     select   0xc080b7e4 httpd
 4428     1  4428     0  SsJ     nanslp   0xc07be46c cron
 4421     1  4421     0  SsJ     select   0xc080b7e4 sshd
 4413     1  4413     0  SsJ     nanslp   0xc07be46c httpd
 4396     1  4396     0  SsJ     select   0xc080b7e4 sshd
 4335     1  4335     0  SsJ     select   0xc080b7e4 syslogd
 4174  4106  4105    88  S+J     (threaded)          mysqld
100565                   S       kserel   0xc9a0b394 mysqld
100519                   S       kserel   0xc9a0b394 mysqld
100976                   S       kserel   0xc9a0b394 mysqld
100980                   S       kserel   0xc9a0b394 mysqld
100859                   S       select   0xc080b7e4 mysqld
101014                   S       sbwait   0xc950fe90 mysqld
100991                   S       sbwait   0xd0cd7638 mysqld
101003                   S       sbwait   0xcdb354d4 mysqld
101001                   S       sbwait   0xd074820c mysqld
100588                   S       sbwait   0xccdcb370 mysqld
100454                   S       sbwait   0xcaf8c79c mysqld
100273                   S       sigwait  0xec480c14 mysqld
100267                   S       ksesigwa 0xc9b7fdc8 mysqld
 4106     1  4105    88  S+J     wait     0xc97f4218 sh
 4033     1  4033     0  SsJ     select   0xc080b7e4 syslogd
 3879     1  3879     0  SsJ     select   0xc080b7e4 inetd
 3865     1  3865     0  SsJ     nanslp   0xc07be46c cron
 3858     1  3858     0  SsJ     select   0xc080b7e4 sshd
 3857  3820  3817    88  S+J     (threaded)          mysqld
100252                   S       kserel   0xc8cee4b4 mysqld
100365                   S       kserel   0xc8cee4b4 mysqld
100245                   S       select   0xc080b7e4 mysqld
100323                   S       kserel   0xc8cee4b4 mysqld
100650                   S       kserel   0xc8cee4b4 mysqld
100250                   S       kserel   0xc9a0b634 mysqld
100244                   S       sigwait  0xe9ef5c14 mysqld
100242                   S       ksesigwa 0xc97f4138 mysqld
 3820     1  3817    88  S+J     wait     0xc994d430 sh
 3818  3606  3606    80  SJ      lockf    0xc97e7180 httpd
 3816  3606  3606    80  SJ      lockf    0xccecb780 httpd
 3815  3606  3606    80  SJ      lockf    0xc9579c80 httpd
 3814  3606  3606    80  SJ      lockf    0xcb06a880 httpd
 3813  3606  3606    80  SJ      lockf    0xc9ec7900 httpd
 3729  3711  3711   125  SJ      kqread   0xc9920300 qmgr
 3711     1  3711     0  SsJ     kqread   0xc9914200 master
 3613     1  3613     0  SsJ     select   0xc080b7e4 proftpd
 3606     1  3606     0  SsJ     select   0xc080b7e4 httpd
 3472     1  3472     0  SsJ     select   0xc080b7e4 syslogd
 3436  3266  3266    80  SJ      lockf    0xd00437c0 httpd
 3435  3266  3266    80  SJ      lockf    0xcec83d80 httpd
 3434  3266  3266    80  SJ      lockf    0xc8dba200 httpd
 3433  3266  3266    80  SJ      lockf    0xd0210540 httpd
 3432  3266  3266    80  SJ      lockf    0xc8fd8b80 httpd
 3398     1  3398     0  SsJ     select   0xc080b7e4 inetd
 3379     1  3379     0  SsJ     nanslp   0xc07be46c cron
 3344     1  3344     0  SsJ     select   0xc080b7e4 sshd
 3288  3279  3266     0  SJ      piperd   0xc927ab28 cronolog
 3287  3276  3266     0  SJ      piperd   0xc8d54330 cronolog
 3284  3275  3266     0  SJ      piperd   0xc927a990 cronolog
 3283  3272  3266     0  SJ      piperd   0xc8d55330 cronolog
 3279  3266  3266     0  SJ      wait     0xc90c7430 sh
 3276  3266  3266     0  SJ      wait     0xc95a5c90 sh
 3275  3266  3266     0  SJ      wait     0xc97ef000 sh
 3272  3266  3266     0  SJ      wait     0xc97ef218 sh
 3266     1  3266     0  SsJ     nanslp   0xc07be46c httpd
 3128  3073  3072    88  S+J     (threaded)          mysqld
100198                   S       kserel   0xc8950e74 mysqld
100204                   S       kserel   0xc8950e74 mysqld
100206                   S       kserel   0xc8950e74 mysqld
100202                   S       select   0xc080b7e4 mysqld
100203                   S       kserel   0xc8950e74 mysqld
100200                   S       sigwait  0xec3dac14 mysqld
100197                   S       ksesigwa 0xc8cc0780 mysqld
 3073     1  3072    88  S+J     wait     0xc90c6a78 sh
 3022     1  3022     0  SsJ     select   0xc080b7e4 syslogd
 2545  2354  2354    80  SJ      lockf    0xc8c8e700 httpd
 2542  2354  2354    80  SJ      lockf    0xc8c8e340 httpd
 2541  2354  2354    80  SJ      lockf    0xccebf4c0 httpd
 2539  2354  2354    80  SJ      lockf    0xc8c8e740 httpd
 2538  2354  2354    80  SJ      lockf    0xc8c8eb40 httpd
 2369     1  2369     0  SsJ     nanslp   0xc07be46c cron
 2362     1  2362     0  SsJ     select   0xc080b7e4 sshd
 2354     1  2354     0  SsJ     nanslp   0xc07be46c httpd
 2259  2258  2255    70  SJ      select   0xc080b7e4 postgres
 2258  2255  2255    70  SJ      select   0xc080b7e4 postgres
 2257  2255  2255    70  SJ      select   0xc080b7e4 postgres
 2255     1  2255    70  SsJ     select   0xc080b7e4 postgres
 2199     1  2199     0  SsJ     select   0xc080b7e4 syslogd
 1917     1  1917     0  SsJ     nanslp   0xc07be46c httpd
 1892     1  1892     0  SsJ     nanslp   0xc07be46c cron
 1885     1  1885     0  SsJ     select   0xc080b7e4 sshd
 1842     1  1842     0  SsJ     nanslp   0xc07be46c cron
 1833     1  1833     0  SsJ     select   0xc080b7e4 sshd
 1801     1  1801     0  SsJ     nanslp   0xc07be46c httpd
 1800     1  1800     0  SsJ     select   0xc080b7e4 syslogd
 1476     1  1476     0  SsJ     select   0xc080b7e4 syslogd
 1161     1    41     0  S+J     piperd   0xc9257660 logger
 1160     1    41     0  S+J     fifoor   0xc8c9ca58 tail
 1152     1  1152     0  SsJ     nanslp   0xc07be46c cron
 1090     1  1090     0  SsJ     select   0xc080b7e4 syslogd
  680     1   680     0  Ss      nanslp   0xc07be46c cron
  673     1   673     0  Ss      select   0xc080b7e4 sshd
  644     1   643     0  S       nanslp   0xc07be46c python
  613     1   613     0  Ss      select   0xc080b7e4 ntpd
  524     1   524     0  Ss      select   0xc080b7e4 syslogd
  469     1   469     0  Ss      select   0xc080b7e4 devd
   40     0     0     0  SL      -        0xec126d04 [schedcpu]
   39     0     0     0  SL      sdflush  0xc081caf4 [softdepflush]
   38     0     0     0  SL      syncer   0xc07be1dc [syncer]
   37     0     0     0  SL      vlruwt   0xc8a58000 [vnlru]
   36     0     0     0  SL      psleep   0xc080bd60 [bufdaemon]
   35     0     0     0  SL      pgzero   0xc081dac4 [pagezero]
   34     0     0     0  SL      psleep   0xc081d5cc [vmdaemon]
   33     0     0     0  SL      psleep   0xc081d588 [pagedaemon]
   32     0     0     0  WL                          [swi0: sio]
   31     0     0     0  WL                          [irq1: atkbd0]
   30     0     0     0  WL                          [irq19: atapci1]
   29     0     0     0  WL                          [irq15: ata1]
   28     0     0     0  WL                          [irq14: ata0]
   27     0     0     0  LL     *Giant    0xcaaf97c0 [irq17: em1]
   26     0     0     0  LL     *Giant    0xcaaf97c0 [irq16: em0]
   25     0     0     0  WL                          [irq9: acpi0]
   24     0     0     0  SL      -        0xc89a3c80 [acpi_task_2]
   23     0     0     0  SL      -        0xc89a3c80 [acpi_task_1]
   22     0     0     0  SL      -        0xc89a3c80 [acpi_task_0]
   21     0     0     0  WL                          [swi2: cambio]
    9     0     0     0  SL      ccb_scan 0xc07b8b84 [xpt_thrd]
    8     0     0     0  SL      -        0xc89a3e00 [kqueue taskq]
   20     0     0     0  WL                          [swi5: +]
    7     0     0     0  SL      -        0xc89d7080 [thread taskq]
   19     0     0     0  WL                          [swi6: Giant taskq]
   18     0     0     0  WL                          [swi6: task queue]
   17     0     0     0  SL      -        0xc07bae00 [yarrow]
    6     0     0     0  SL      crypto_r 0xc081c7c4 [crypto returns]
    5     0     0     0  SL      crypto_w 0xc081c784 [crypto]
    4     0     0     0  SL      -        0xc07bb928 [g_down]
    3     0     0     0  LL     *vm page  0xc8d5f780 [g_up]
    2     0     0     0  SL      -        0xc07bb91c [g_event]
   16     0     0     0  WL                          [swi1: net]
   15     0     0     0  WL                          [swi3: vm]
   14     0     0     0  LL     *Giant    0xcaaf97c0 [swi4: clock sio]
   13     0     0     0  RL      CPU 0               [idle: cpu0]
   12     0     0     0  RL      CPU 1               [idle: cpu1]
   11     0     0     0  RL      CPU 2               [idle: cpu2]
   10     0     0     0  RL                          [idle: cpu3]
    1     0     1     0  SLs     wait     0xc8951000 [init]
    0     0     0     0  WLs                         [swapper]


# --------------------------------------------------------------------
# CPU LISTING

db> show pcpu
cpuid        = 1
curthread    = 0xc894c900: pid 12 "idle: cpu1"
curpcb       = 0xe81f2d90
fpcurthread  = none
idlethread   = 0xc894c900: pid 12 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
spin locks held:

db> show allpcpu
Current CPU: 1

cpuid        = 0
curthread    = 0xc894ca80: pid 13 "idle: cpu0"
curpcb       = 0xe81f5d90
fpcurthread  = none
idlethread   = 0xc894ca80: pid 13 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
spin locks held:

cpuid        = 1
curthread    = 0xc894c900: pid 12 "idle: cpu1"
curpcb       = 0xe81f2d90
fpcurthread  = none
idlethread   = 0xc894c900: pid 12 "idle: cpu1"
APIC ID      = 1
currentldt   = 0x50
spin locks held:

cpuid        = 2
curthread    = 0xc894c780: pid 11 "idle: cpu2"
curpcb       = 0xe81efd90
fpcurthread  = none
idlethread   = 0xc894c780: pid 11 "idle: cpu2"
APIC ID      = 2
currentldt   = 0x50
spin locks held:

cpuid        = 3
curthread    = 0xca268900: pid 52531 "ee"
curpcb       = 0xec710d90
fpcurthread  = none
idlethread   = 0xc894c600: pid 10 "idle: cpu3"
APIC ID      = 3
currentldt   = 0x50
spin locks held:

# --------------------------------------------------------------------
# LOCK LISTING

db> show locks


db> show alllocks
Process 35638 (sendmail) thread 0xd0530600 (101044)
exclusive sleep mutex vm object (standard object) r = 0 (0xc9c01318) locked @ /usr/src/sys/vm/vm_map.c:1472
exclusive sx user map r = 0 (0xc8fb7734) locked @ /usr/src/sys/vm/vm_map.c:1208
Process 32836 (rsync) thread 0xc9ec1900 (100360)
exclusive sleep mutex pmap r = 0 (0xca89f560) locked @ /usr/src/sys/i386/i386/pmap.c:2041
exclusive sleep mutex vm page queue mutex r = 0 (0xc081d540) locked @ /usr/src/sys/i386/i386/pmap.c:2040
exclusive sx user map r = 0 (0xca89f4e4) locked @ /usr/src/sys/vm/vm_map.c:3105
Process 32754 (httpd) thread 0xcfcc5600 (101170)
exclusive sleep mutex vm object (standard object) r = 0 (0xcc8378c4) locked @ /usr/src/sys/vm/vm_fault.c:297
exclusive sx user map r = 0 (0xc984a4e4) locked @ /usr/src/sys/vm/vm_map.c:3105
Process 7829 (bsnmpd) thread 0xca196180 (100409)
exclusive sleep mutex vm object (kmem object) r = 0 (0xc081d440) locked @ /usr/src/sys/vm/vm_kern.c:325
exclusive sleep mutex system map r = 0 (0xc1068144) locked @ /usr/src/sys/vm/vm_kern.c:295
exclusive sx sysctl lock r = 0 (0xc07be240) locked @ /usr/src/sys/kern/kern_sysctl.c:1375
exclusive sleep mutex Giant r = 0 (0xc07bdb80) locked @ /usr/src/sys/kern/kern_sysctl.c:1313
Process 3 (g_up) thread 0xc894ec00 (100015)
exclusive sleep mutex vm object (standard object) r = 0 (0xcaf6e8c4) locked @ /usr/src/sys/kern/vfs_bio.c:3120

db> show lockedvnods
Locked vnodes

0xcdcce6cc: tag ufs, type VDIR
    usecount 2, writecount 0, refcount 4 mountedhere 0
    flags ()
    v_object 0xcaf6e8c4 ref 0 pages 1
     lock type ufs: EXCL (count 1) by thread 0xc97f1000 (pid 32710)#0 0xc054d7f1 at lockmgr+0x4ed
#1 0xc06a0f96 at ffs_lock+0x76
#2 0xc07170ab at VOP_LOCK_APV+0x87
#3 0xc05bd370 at vn_lock+0xac
#4 0xc05b9f0b at getdirentries+0xff
#5 0xc07046b3 at syscall+0x25b
#6 0xc06efccf at Xint0x80_syscall+0x1f

	ino 20398955, on dev ad8s1e

# --------------------------------------------------------------------
# PROCESS STACK TRACES

# SENDMAIL 
db> trace 35638
Tracing pid 35638 tid 101044 td 0xd0530600
sched_switch(d0530600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,d0530600,0,c0768214,5f8) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c0768214,5f8) at _mtx_lock_flags+0xa2
vm_map_pmap_enter(c8fb76f0,28095000,5,c9c01318,0,0,a000,10) at vm_map_pmap_enter+0x1df
vm_map_insert(c8fb76f0,c9c01318,0,0,28095000,2809f000,5,7,112) at vm_map_insert+0x28c
vm_map_find(c8fb76f0,c9c01318,0,0,ed810cc8,a000,1,5,7,112) at vm_map_find+0x86
vm_mmap(c8fb76f0,ed810cc8,a000,5,7,20002,2,c9cd0ae0,0,0) at vm_mmap+0x266
mmap(d0530600,ed810d04) at mmap+0x31f
syscall(2807003b,2806003b,bfbf003b,28082060,0,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (198, FreeBSD ELF32, nosys), eip = 0x2806cd63, esp = 0xbfbfea7c, ebp = 0xbfbfeab8 ---

# RSYNC 
db> trace 32836
Tracing pid 32836 tid 100360 td 0xc9ec1900
sched_switch(c9ec1900,0,2) at sched_switch+0x177
mi_switch(2,0,c07bdb40,0,c074c42d,...) at mi_switch+0x270
critical_exit(1,c9ec1900,0,9d6f000,bfc275bc,...) at critical_exit+0x8b
intr_execute_handlers(c893f6e4,ec552b6c,13,ec552bd4,c06f0033,...) at intr_execute_handlers+0x129
lapic_handle_intr(35) at lapic_handle_intr+0x2e
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc0700897, esp = 0xec552bb0, ebp = 0xec552bd4 ---
pmap_enter(ca89f560,9d6f000,c332e380,7,0,ced476b4,0,c0767c8b,383) at pmap_enter+0x1bf
vm_fault(ca89f4a0,9d6f000,2,8,c9ec1900,...) at vm_fault+0x10b8
trap_pfault(ec552d38,1,9d6ffb8,9d6ffb8,0,...) at trap_pfault+0xce
trap(bfbf003b,3b,bfbf003b,9d6ffb8,0,...) at trap+0x1eb
calltrap() at calltrap+0x5
--- trap 0xc, eip = 0x2818527f, esp = 0xbfbf9670, ebp = 0xbfbf9b08 ---

# HTTPD
db> trace 32754
Tracing pid 32754 tid 101170 td 0xcfcc5600
sched_switch(cfcc5600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,cfcc5600,0,c0767c8b,356) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c0767c8b,356,c07bdb40,...) at _mtx_lock_flags+0xa2
vm_fault(c984a4a0,88d1000,2,8,cfcc5600,...) at vm_fault+0xfa7
trap_pfault(ed840d38,1,88d1d90,88d1d90,0,...) at trap_pfault+0xce
trap(80a003b,875003b,bfbf003b,2360,8382000,...) at trap+0x1eb
calltrap() at calltrap+0x5
--- trap 0xc, eip = 0x286a34c1, esp = 0xbfbfb860, ebp = 0xbfbfb898 ---

# BSNMPD
db> trace 7829
Tracing pid 7829 tid 100409 td 0xca196180
sched_switch(ca196180,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,ca196180,0,c076806c,16f) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c076806c,16f,10928000,...) at _mtx_lock_flags+0xa2
kmem_malloc(c10680c0,7000,102,ec649b64,c06b330f,...) at kmem_malloc+0x282
page_alloc(0,7000,ec649b57,102) at page_alloc+0x1a
uma_large_malloc(7000,102,64f0,c05469d6,ec649bf4,...) at uma_large_malloc+0x3b
malloc(64f0,c078fb40,102,c057de5a,ec649bf4,...) at malloc+0xf3
sysctl_jail_list(c078f220,0,0,ec649bf4,c078f220,...) at sysctl_jail_list+0x8e
sysctl_root(0,ec649c74,3,ec649bf4) at sysctl_root+0x11b
userland_sysctl(ca196180,ec649c74,3,0,bfbfc46c,0,0,0,ec649c70,0,c07bdb80,0,c074b5cd,521) at userland_sysctl+0xf4
__sysctl(ca196180,ec649d04) at __sysctl+0x77
syscall(bfbe003b,2818003b,bfbe003b,3,bfbfc46c,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (202, FreeBSD ELF32, __sysctl), eip = 0x2814e77b, esp = 0xbfbfc34c, ebp = 0xbfbfc388 ---

# G_UP
db> trace 3
Tracing pid 3 tid 100015 td 0xc894ec00
sched_switch(c894ec00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c081d540,c9ec1900,0,c081d540,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c081d540,c894ec00,0,c0752afd,c43) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c081d540,0,c0752afd,c43,c06b29a0,...) at _mtx_lock_flags+0xa2
bufdone(dcb814d4) at bufdone+0x176
g_vfs_done(cc0806b4) at g_vfs_done+0x8a
biodone(cc0806b4) at biodone+0x58
g_io_schedule_up(c894ec00) at g_io_schedule_up+0xcb
g_up_procbody(0,e8216d38,0,c0523700,0,...) at g_up_procbody+0x5a
fork_exit(c0523700,0,e8216d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8216d6c, ebp = 0 ---

# EE
db> trace 52531
Tracing pid 52531 tid 100453 td 0xca268900
sched_switch(3dc,c074f76f,ec710b90,c0550d8e,c080b7c0,...) at sched_switch+0x177
sellock(c055154c,c07497db,a,c055154c,0,...) at sellock
(null)(786574,50414d4b,544e4520,75005952,20726573,...) at 0x9


# SWI4: CLOCK SIO
db> trace 14
Tracing pid 14 tid 100002 td 0xc894cc00
sched_switch(c894cc00,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,c894cc00,0,c074bcf0,102) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c074bcf0,102) at _mtx_lock_flags+0xa2
softclock(0) at softclock+0x18f
ithread_execute_handlers(c894b430,c89a2b80) at ithread_execute_handlers+0xe6
ithread_loop(c89288e0,e81f8d38,c89288e0,c05456e0,0,...) at ithread_loop+0x67
fork_exit(c05456e0,c89288e0,e81f8d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe81f8d6c, ebp = 0 ---


# EM0
db> trace 27
Tracing pid 27 tid 100017 td 0xc894e900
sched_switch(c894e900,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,c894e900,0,c0747942,2a9) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c0747942,2a9) at _mtx_lock_flags+0xa2
ithread_execute_handlers(c89a4648,c89a1480) at ithread_execute_handlers+0xde
ithread_loop(c8a438d0,e8210d38,c8a438d0,c05456e0,0,...) at ithread_loop+0x67
fork_exit(c05456e0,c8a438d0,e8210d38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe8210d6c, ebp = 0 ---


# em1
db> trace 26
Tracing pid 26 tid 100018 td 0xc894e780
sched_switch(c894e780,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,c894e780,0,c0747942,2a9) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c0747942,2a9) at _mtx_lock_flags+0xa2
ithread_execute_handlers(c89a4860,c89a1500) at ithread_execute_handlers+0xde
ithread_loop(c8a3f6a0,e820dd38,c8a3f6a0,c05456e0,0,...) at ithread_loop+0x67
fork_exit(c05456e0,c8a3f6a0,e820dd38) at fork_exit+0xa0
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xe820dd6c, ebp = 0 ---


# LYNX
db> trace 33070
Tracing pid 33070 tid 100668 td 0xca02f600
sched_switch(ca02f600,0,1) at sched_switch+0x177
mi_switch(1,0) at mi_switch+0x270
turnstile_wait(c07bdb80,ca196180,0,c07bdb80,2,c07497e6,233) at turnstile_wait+0x35a
_mtx_lock_sleep(c07bdb80,ca02f600,0,c074fb3c,ec) at _mtx_lock_sleep+0x14c
_mtx_lock_flags(c07bdb80,0,c074fb3c,ec) at _mtx_lock_flags+0xa2
soo_poll(d015b000,40,d06e1b80,ca02f600) at soo_poll+0x2a
selscan(ca02f600,ec5d5b94,ec5d5b84,14,c080b7c0,0,c074f76f,300) at selscan+0x1a5
kern_select(ca02f600,14,bfbfe304,0,0,...) at kern_select+0x33b
select(ca02f600,ec5d5d04) at select+0x44
syscall(3b,bfbf003b,bfbf003b,0,bfbfe304,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x281e6be4, esp = 0xbfbfe2a8, ebp = 0xbfbfe384 ---


-------------- next part --------------
#
# RACK2 -- M2 config file
#

machine		i386
cpu		I686_CPU
ident		RACK2

# To statically compile in device wiring instead of /boot/device.hints
#hints		"GENERIC.hints"		# Default places to look for devices.

makeoptions	DEBUG=-g		# Build kernel with gdb(1) debug symbols

options 	SCHED_4BSD		# 4BSD scheduler
options 	PREEMPTION		# Enable kernel thread preemption
options 	INET			# InterNETworking
options 	INET6			# IPv6 communications protocols
options 	FFS			# Berkeley Fast Filesystem
options 	SOFTUPDATES		# Enable FFS soft updates support
options 	UFS_ACL			# Support for access control lists
options 	UFS_DIRHASH		# Improve performance on big directories
options 	MD_ROOT			# MD is a potential root device
options 	NFSCLIENT		# Network Filesystem Client
options 	NFSSERVER		# Network Filesystem Server
options 	NFS_ROOT		# NFS usable as /, requires NFSCLIENT
options 	MSDOSFS			# MSDOS Filesystem
options 	CD9660			# ISO 9660 Filesystem
options 	PROCFS			# Process filesystem (requires PSEUDOFS)
options 	PSEUDOFS		# Pseudo-filesystem framework
options 	GEOM_GPT		# GUID Partition Tables.
options 	COMPAT_43		# Compatible with BSD 4.3 [KEEP THIS!]
options 	COMPAT_FREEBSD4		# Compatible with FreeBSD4
options 	COMPAT_FREEBSD5		# Compatible with FreeBSD5
options 	SCSI_DELAY=5000		# Delay (in ms) before probing SCSI
options 	KTRACE			# ktrace(1) support
options 	SYSVSHM			# SYSV-style shared memory
options 	SYSVMSG			# SYSV-style message queues
options 	SYSVSEM			# SYSV-style semaphores
options 	_KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev
options 	ADAPTIVE_GIANT		# Giant mutex is adaptive.

options		SMP
options         FAST_IPSEC
options         IPFIREWALL
options         IPFIREWALL_DEFAULT_TO_ACCEPT

options         ALTQ
options         ALTQ_CBQ
options         ALTQ_RED
options         ALTQ_RIO
options         ALTQ_HFSC
options         ALTQ_CDNR
options         ALTQ_PRIQ

options KDB
options KDB_TRACE
options KDB_UNATTENDED
options DDB
options INVARIANTS
options INVARIANT_SUPPORT
options WITNESS
options WITNESS_SKIPSPIN
options DIAGNOSTIC
options BREAK_TO_DEBUGGER
options DEBUG_LOCKS
options DEBUG_VFS_LOCKS


device          crypto
device		apic			# I/O APIC

# Bus support.
device		eisa
device		pci

# Floppy drives
device		fdc

# ATA and ATAPI devices
device		ata
device		atadisk		# ATA disk drives
device		ataraid		# ATA RAID drives
device		atapicd		# ATAPI CDROM drives
device		atapifd		# ATAPI floppy drives
device		atapist		# ATAPI tape drives
options 	ATA_STATIC_ID	# Static device numbering

# SCSI Controllers
#device		ahb		# EISA AHA1742 family
device		ahc		# AHA2940 and onboard AIC7xxx devices
#options 	AHC_REG_PRETTY_PRINT	# Print register bitfields in debug
					# output.  Adds ~128k to driver.
device		ahd		# AHA39320/29320 and onboard AIC79xx devices
#options 	AHD_REG_PRETTY_PRINT	# Print register bitfields in debug
 					# output.  Adds ~215k to driver.
#device		amd		# AMD 53C974 (Tekram DC-390(T))
#device		isp		# Qlogic family
#device 	ispfw		# Firmware for QLogic HBAs- normally a module
#device		mpt		# LSI-Logic MPT-Fusion
#device		ncr		# NCR/Symbios Logic
#device		sym		# NCR/Symbios Logic (newer chipsets + those of `ncr')
#device		trm		# Tekram DC395U/UW/F DC315U adapters

#device		adv		# Advansys SCSI adapters
#device		adw		# Advansys wide SCSI adapters
device		aha		# Adaptec 154x SCSI adapters
device		aic		# Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
#device		bt		# Buslogic/Mylex MultiMaster SCSI adapters

#device		ncv		# NCR 53C500
#device		nsp		# Workbit Ninja SCSI-3
#device		stg		# TMC 18C30/18C50

# SCSI peripherals
device		scbus		# SCSI bus (required for SCSI)
#device		ch		# SCSI media changers
device		da		# Direct Access (disks)
#device		sa		# Sequential Access (tape etc)
device		cd		# CD
device		pass		# Passthrough device (direct SCSI access)
#device		ses		# SCSI Environmental Services (and SAF-TE)

# RAID controllers interfaced to the SCSI subsystem
#device		amr		# AMI MegaRAID
#device		arcmsr		# Areca SATA II RAID
#device		asr		# DPT SmartRAID V, VI and Adaptec SCSI RAID
#device		ciss		# Compaq Smart RAID 5*
#device		dpt		# DPT Smartcache III, IV - See NOTES for options
#device		hptmv		# Highpoint RocketRAID 182x
#device		hptrr		# Highpoint RocketRAID 17xx, 22xx, 23xx, 25xx
#device		rr232x		# Highpoint RocketRAID 232x
#device		iir		# Intel Integrated RAID
#device		ips		# IBM (Adaptec) ServeRAID
#device		mly		# Mylex AcceleRAID/eXtremeRAID
#device		twa		# 3ware 9000 series PATA/SATA RAID

# RAID controllers
device		aac		# Adaptec FSA RAID
device		aacp		# SCSI passthrough for aac (requires CAM)
#device		ida		# Compaq Smart RAID
#device		mfi		# LSI MegaRAID SAS
#device		mlx		# Mylex DAC960 family
#device		pst		# Promise Supertrak SX6000
#device		twe		# 3ware ATA RAID

# atkbdc0 controls both the keyboard and the PS/2 mouse
device		atkbdc		# AT keyboard controller
device		atkbd		# AT keyboard
device		psm		# PS/2 mouse

device		kbdmux		# keyboard multiplexer

device		vga		# VGA video card driver

device		splash		# Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device		sc

# Enable this for the pcvt (VT220 compatible) console driver
#device		vt
#options 	XSERVER		# support for X server on a vt console
#options 	FAT_CURSOR	# start with block cursor

device		agp		# support several AGP chipsets

# Power management support (see NOTES for more options)
#device		apm
# Add suspend/resume support for the i8254.
device		pmtimer

# PCCARD (PCMCIA) support
# PCMCIA and cardbus bridge support
#device		cbb		# cardbus (yenta) bridge
#device		pccard		# PC Card (16-bit) bus
#device		cardbus		# CardBus (32-bit) bus

# Serial (COM) ports
device		sio		# 8250, 16[45]50 based serial ports

# Parallel port
#device		ppc
#device		ppbus		# Parallel port bus (required)
#device		lpt		# Printer
#device		plip		# TCP/IP over parallel
#device		ppi		# Parallel port interface device
#device		vpo		# Requires scbus and da

# If you've got a "dumb" serial or parallel PCI card that is
# supported by the puc(4) glue driver, uncomment the following
# line to enable it (connects to the sio and/or ppc drivers):
#device		puc

# PCI Ethernet NICs.
#device		de		# DEC/Intel DC21x4x (``Tulip'')
device		em		# Intel PRO/1000 adapter Gigabit Ethernet Card
device		ixgb		# Intel PRO/10GbE Ethernet Card
#device		txp		# 3Com 3cR990 (``Typhoon'')
#device		vx		# 3Com 3c590, 3c595 (``Vortex'')

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device		miibus		# MII bus support
#device		bce		# Broadcom BCM5706/BCM5708 Gigabit Ethernet
#device		bfe		# Broadcom BCM440x 10/100 Ethernet
device		bge		# Broadcom BCM570xx Gigabit Ethernet
device		dc		# DEC/Intel 21143 and various workalikes
device		fxp		# Intel EtherExpress PRO/100B (82557, 82558)
#device		lge		# Level 1 LXT1001 gigabit Ethernet
#device		msk		# Marvell/SysKonnect Yukon II Gigabit Ethernet
#device		nge		# NatSemi DP83820 gigabit Ethernet
#device		nve		# nVidia nForce MCP on-board Ethernet Networking
#device		pcn		# AMD Am79C97x PCI 10/100(precedence over 'lnc')
device		re		# RealTek 8139C+/8169/8169S/8110S
device		rl		# RealTek 8129/8139
#device		sf		# Adaptec AIC-6915 (``Starfire'')
device		sis		# Silicon Integrated Systems SiS 900/SiS 7016
#device		sk		# SysKonnect SK-984x & SK-982x gigabit Ethernet
#device		ste		# Sundance ST201 (D-Link DFE-550TX)
#device		stge		# Sundance/Tamarack TC9021 gigabit Ethernet
#device		ti		# Alteon Networks Tigon I/II gigabit Ethernet
#device		tl		# Texas Instruments ThunderLAN
#device		tx		# SMC EtherPower II (83c170 ``EPIC'')
#device		vge		# VIA VT612x gigabit Ethernet
device		vr		# VIA Rhine, Rhine II
#device		wb		# Winbond W89C840F
device		xl		# 3Com 3c90x (``Boomerang'', ``Cyclone'')

# ISA Ethernet NICs.  pccard NICs included.
#device		cs		# Crystal Semiconductor CS89x0 NIC
# 'device ed' requires 'device miibus'
#device		ed		# NE[12]000, SMC Ultra, 3c503, DS8390 cards
#device		ex		# Intel EtherExpress Pro/10 and Pro/10+
#device		ep		# Etherlink III based cards
#device		fe		# Fujitsu MB8696x based cards
#device		ie		# EtherExpress 8/16, 3C507, StarLAN 10 etc.
device		lnc		# NE2100, NE32-VL Lance Ethernet cards
#device		sn		# SMC's 9000 series of Ethernet chips
#device		xe		# Xircom pccard Ethernet

# Wireless NIC cards
#device		wlan		# 802.11 support
#device		wlan_wep	# 802.11 WEP support
#device		wlan_ccmp	# 802.11 CCMP support
#device		wlan_tkip	# 802.11 TKIP support
#device		an		# Aironet 4500/4800 802.11 wireless NICs.
#device		ath		# Atheros pci/cardbus NIC's
#device		ath_hal		# Atheros HAL (Hardware Access Layer)
#device		ath_rate_sample	# SampleRate tx rate control for ath
#device		awi		# BayStack 660 and others
#device		ral		# Ralink Technology RT2500 wireless NICs.
#device		wi		# WaveLAN/Intersil/Symbol 802.11 wireless NICs.
#device		wl		# Older non 802.11 Wavelan wireless NIC.

# Pseudo devices.
device		loop		# Network loopback
device		random		# Entropy device
device		ether		# Ethernet support
device		sl		# Kernel SLIP
device		ppp		# Kernel PPP
device		tun		# Packet tunnel.
device		pty		# Pseudo-ttys (telnet etc)
device		md		# Memory "disks"
device		gif		# IPv6 and IPv4 tunneling
device		faith		# IPv6-to-IPv4 relaying (translation)

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device		bpf		# Berkeley packet filter

# USB support
#device		uhci		# UHCI PCI->USB interface
#device		ohci		# OHCI PCI->USB interface
#device		ehci		# EHCI PCI->USB interface (USB 2.0)
#device		usb		# USB Bus (required)
#device		udbp		# USB Double Bulk Pipe devices
#device		ugen		# Generic
#device		uhid		# "Human Interface Devices"
#device		ukbd		# Keyboard
#device		ulpt		# Printer
#device		umass		# Disks/Mass storage - Requires scbus and da
#device		ums		# Mouse
#device		ural		# Ralink Technology RT2500USB wireless NICs
#device		urio		# Diamond Rio 500 MP3 player
#device		uscanner	# Scanners
# USB Ethernet, requires miibus
#device		aue		# ADMtek USB Ethernet
#device		axe		# ASIX Electronics USB Ethernet
#device		cdce		# Generic USB over Ethernet
#device		cue		# CATC USB Ethernet
#device		kue		# Kawasaki LSI USB Ethernet
#device		rue		# RealTek RTL8150 USB Ethernet

# FireWire support
#device		firewire	# FireWire bus code
#device		sbp		# SCSI over FireWire (Requires scbus and da)
#device		fwe		# Ethernet over FireWire (non-standard!)


More information about the freebsd-hackers mailing list