VFS deadlock using dump -L on HEAD -- VOP_INACTIVE acquires
snapshot file system suspend lock after vnode lock on
delete-after-last-close?
Robert Watson
rwatson at FreeBSD.org
Fri Feb 18 14:51:26 GMT 2005
I upgraded a box to a recent 6.x this morning, and ran into the following
apparently deadlock while backing up a partitionusing dump -L. It appears
to have gotten stuck during mksnap_ffs. Some interpretation below, but
the high level summary is that it looks to me like we have a lock order
issue between the snapshot write lock for the file system and vnode locks,
which occurs because vrele() calls VOP_INACTIVE with the vnode lock held,
and ufs_inactive() then proceeds to get a write reference to the file
system. It could be a fix is to grab the write reference to the file
system in vrele() before the vnode lock, to aovid grabbing the write
reference in ufs_inactive().
Robert N M Watson
www:~> more backup.csh
#!/bin/csh
set DATE=`date +"%Y%m%d"`
dump -L -f - -a -u -0 /home/pnn | gzip -9 > ${DATE}-backup.dump.gz
www:~> df
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/twed0s1a 507630 103502 363518 22% /
devfs 1 1 0 100% /dev
/dev/twed0s1f 89359640 370128 81840742 0% /home/pnn
/dev/twed0s1e 16244334 3211132 11733656 21% /usr
/dev/twed0s1d 8122126 1541978 5930378 21% /var
www:~> ./backup.csh
load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k
On the serial console:
telnet> send break
KDB: enter: Line break on console
[thread pid 11 tid 100004 ]
Stopped at kdb_enter+0x2b: nop
db> show lockedvnods
Locked vnodes
0xc13c49b4: tag ufs, type VDIR
usecount 153, writecount 0, refcount 1 mountedhere 0
flags (VV_ROOT)
v_object 0
lock type ufs: EXCL (count 1) by thread 0xc1807a10 (pid 3057) with
152 pend
ing
ino 2, on dev twed0s1f (229, 9)
0xc15268a0: tag ufs, type VDIR
usecount 2, writecount 0, refcount 1 mountedhere 0
flags ()
v_object 0
lock type ufs: EXCL (count 1) by thread 0xc137e000 (pid 3062) with 1
pendin
g
ino 3085322, on dev twed0s1f (229, 9)
0xc163bbdc: tag ufs, type VDIR
usecount 2, writecount 0, refcount 1 mountedhere 0
flags ()
v_object 0
lock type ufs: EXCL (count 1) by thread 0xc162dcf0 (pid 3056) with 1
pendin
g
ino 3133686, on dev twed0s1f (229, 9)
0xc17e933c: tag ufs, type VREG
usecount 1, writecount 0, refcount 0 mountedhere 0
flags ()
v_object 0xc17e7210
lock type ufs: EXCL (count 1) by thread 0xc1996450 (pid 3045) with 1
pendin
g
ino 3134802, on dev twed0s1f (229, 9)
0xc18419b4: tag ufs, type VREG
usecount 0, writecount 0, refcount 0 mountedhere 0
flags ()
v_object 0xc1826000
lock type ufs: EXCL (count 1) by thread 0xc19b2450 (pid 3047)
ino 3132762, on dev twed0s1f (229, 9)
0xc1ac933c: tag ufs, type VREG
usecount 1, writecount 0, refcount 865 mountedhere 0
flags ()
v_object 0
lock type ufs: EXCL (count 1) by thread 0xc162d8a0 (pid 2993)
ino 9891841, on dev twed0s1f (229, 9)
0xc19fd9b4: tag ufs, type VREG
usecount 0, writecount 0, refcount 0 mountedhere 0
flags ()
v_object 0xc1a93948
lock type ufs: EXCL (count 1) by thread 0xc1807b80 (pid 2609)
ino 3197600, on dev twed0s1f (229, 9)
db> ps
pid proc uid ppid pgrp flag stat wmesg wchan cmd
3296 c1cf93f8 1062 3292 3292 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
3295 c1cf91fc 1062 3293 3293 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
3293 c1cf9be8 1062 3290 3293 0004000 [SLPQ wait 0xc1cf9be8][SLP] sh
3292 c1d66000 1062 3289 3292 0004000 [SLPQ wait 0xc1d66000][SLP] sh
3290 c19b0000 0 390 390 0000000 [SLPQ piperd 0xc1655480][SLP] cron
3289 c1d661fc 0 390 390 0000000 [SLPQ piperd 0xc19a3d80][SLP] cron
3271 c1d66be8 1062 3268 3268 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
3268 c1d69000 1062 3267 3268 0004000 [SLPQ wait 0xc1d69000][SLP] sh
3267 c1d669ec 0 390 390 0000000 [SLPQ piperd 0xc13aac00][SLP] cron
3247 c1cf95f4 1062 3245 3245 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
3245 c1cf99ec 1062 3243 3245 0004000 [SLPQ wait 0xc1cf99ec][SLP] sh
3243 c1cf9de4 0 390 390 0000000 [SLPQ piperd 0xc13aa480][SLP] cron
3227 c1d66de4 1062 3222 3222 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
3226 c1c8f1fc 1062 3223 3223 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
3223 c18063f8 1062 3218 3223 0004000 [SLPQ wait 0xc18063f8][SLP] sh
3222 c1805000 1062 3217 3222 0004000 [SLPQ wait 0xc1805000][SLP] sh
3218 c19ae000 0 390 390 0000000 [SLPQ piperd 0xc13a9a80][SLP] cron
3217 c18067f0 0 390 390 0000000 [SLPQ piperd 0xc13a9300][SLP] cron
3197 c1c8f3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3196 c1c8f5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3195 c1c8f7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3194 c1c8f9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3193 c1c8fbe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3192 c1c8fde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3191 c1cf6000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3190 c1cf61fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3189 c1cf63f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3188 c1cf65f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3187 c1cf67f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3186 c1cf69ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3185 c1cf6be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3184 c1cf6de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3183 c1cf9000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3182 c1c8e1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3181 c1c8e5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3180 c1c8e9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3179 c1c1e3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3178 c1c1e5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3177 c1c1e7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3176 c1c1e9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3175 c1c1ebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3174 c1c1ede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3173 c1c88000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3172 c1c881fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3171 c1c883f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3170 c1c885f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3169 c1c887f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3168 c1c889ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3167 c1c88be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3166 c1c88de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3165 c1c8e000 1062 3163 3163 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh
3163 c1c8e3f8 1062 3161 3163 0004000 [SLPQ wait 0xc1c8e3f8][SLP] sh
3161 c1c8e7f0 0 390 390 0000000 [SLPQ piperd 0xc19a3000][SLP] cron
3159 c1c8ebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3158 c1c8ede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3157 c1c8f000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3156 c1bc37f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3155 c1bc39ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3154 c1bc3be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3153 c1bc3de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3152 c1c19000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3151 c1c191fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3150 c1c193f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3149 c1c195f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3148 c1c197f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3147 c1c199ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3146 c1c19be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3145 c1c19de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3144 c1c1a000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3143 c1c1a1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3142 c1c1a3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3141 c1c1a5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3140 c1c1a7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3139 c1c1a9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3138 c1c1abe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3137 c1c1ade4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3136 c1c1e000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3135 c1c1e1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3134 c1bc1de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3133 c1bc1be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3132 c1b837f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3131 c1b839ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3130 c1b83be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3129 c1b83de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3128 c1bc1000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3127 c1bc11fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3126 c1bc13f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3125 c1bc15f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3124 c1bc17f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3123 c1bc19ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3120 c1bc3000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3119 c1bc31fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3118 c1bc33f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3117 c1bc35f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3116 c1abb7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3115 c1abb9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3114 c1abbbe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3113 c1abbde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3112 c1b82000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3111 c1b821fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3110 c1b823f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3109 c1b825f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3108 c1b827f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3107 c1b829ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3106 c1b82be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3105 c1b82de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3104 c1b83000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3103 c1b831fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3102 c1b833f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3101 c1b835f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3100 c19b07f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3099 c19b09ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3098 c19b0be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3097 c19b0de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3096 c1aba000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3095 c1aba1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3094 c1aba3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3093 c1aba5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3092 c1aba7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3091 c1aba9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3090 c1ababe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3089 c1abade4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3088 c1abb000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3087 c1abb1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3086 c1abb3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3085 c16291fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3084 c1abb5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3083 c1510be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3082 c162c3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3081 c18061fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3080 c19957f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3079 c18055f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3078 c19aede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3077 c19959ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3076 c19aebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3075 c16297f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3074 c16293f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3073 c19ae5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3072 c19ae1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3071 c19951fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3070 c19ae7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3069 c12ab7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3068 c1510de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3067 c1805be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3066 c1994000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3065 c19ae9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3064 c137d3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3063 c150f000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3062 c1510000 80 2910 403 0004000 [SLPQ ufs 0xc163bca0][SLP]
download.cgi
3060 c19953f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3059 c19947f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3058 c16299ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3057 c1806be8 80 403 403 0000100 [SLPQ ufs 0xc1526964][SLP] httpd
3056 c1994be8 80 403 403 0000100 [SLPQ ufs 0xc17e9400][SLP] httpd
3055 c1806000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3053 c137dde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3051 c19b05f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3048 c19941fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
3047 c19b03f8 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP]
httpd
3045 c1995be8 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP]
httpd
3036 c19945f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
2993 c162c1fc 1000 2992 2987 0004102 [SLPQ ufs 0xc13c4a78][SLP]
mksnap_ffs
2992 c162cbe8 1000 2990 2987 0004002 [SLPQ wait 0xc162cbe8][SLP] sh
2991 c1995de4 1000 2987 2987 0004002 [SLPQ piperd 0xc13a9780][SLP] gzip
2990 c150f5f4 1000 2987 2987 0004002 [SLPQ wait 0xc150f5f4][SLP] dump
2987 c1806de4 1000 1762 2987 0004002 [SLPQ pause 0xc1806e18][SLP] csh
2910 c1994de4 80 403 403 0000100 [SLPQ wait 0xc1994de4][SLP] httpd
2909 c18053f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
2908 c19ae3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
2843 c1995000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
2839 c19955f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
2669 c1629000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
2609 c18069ec 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP]
httpd
2598 c19943f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
1981 c18051fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
1899 c16295f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
1762 c162cde4 1000 1761 1762 0004002 [SLPQ pause 0xc162ce18][SLP] tcsh
1761 c162c000 1000 1759 1759 0000100 [SLPQ select 0xc078a744][SLP] sshd
1759 c1629be8 0 368 1759 0004100 [SLPQ sbwait 0xc1429480][SLP] sshd
1754 c18065f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd
526 c162c9ec 1062 516 526 0004100 [SLPQ piperd 0xc13a9900][SLP]
sendmail
525 c162c7f0 1062 1 525 0004000 [SLPQ accept 0xc1429186][SLP] perl
516 c18057f0 0 390 390 0000000 [SLPQ piperd 0xc13a9c00][SLP] cron
447 c12ab5f4 0 1 447 0004002 [SLPQ ttyin 0xc1318410][SLP] getty
446 c15107f0 0 1 446 0004002 [SLPQ ttyin 0xc12f5810][SLP] getty
445 c15105f4 0 1 445 0004002 [SLPQ ttyin 0xc12f6810][SLP] getty
444 c15101fc 0 1 444 0004002 [SLPQ ttyin 0xc12f6410][SLP] getty
443 c150f9ec 0 1 443 0004002 [SLPQ ttyin 0xc12ef410][SLP] getty
442 c15103f8 0 1 442 0004002 [SLPQ ttyin 0xc12ef810][SLP] getty
441 c150f3f8 0 1 441 0004002 [SLPQ ttyin 0xc12efc10][SLP] getty
440 c150f1fc 0 1 440 0004002 [SLPQ ttyin 0xc12f5010][SLP] getty
439 c150fde4 0 1 439 0004002 [SLPQ ttyin 0xc12f5410][SLP] getty
427 c150fbe8 0 1 427 0000000 [SLPQ select 0xc078a744][SLP]
inetd
403 c150f7f0 0 1 403 0000000 [SLPQ select 0xc078a744][SLP]
httpd
390 c12ab9ec 0 1 390 0000000 [SLPQ nanslp 0xc0785dac][SLP] cron
378 c137d000 25 1 378 0000100 [SLPQ pause 0xc137d034][SLP]
sendmail
374 c12abbe8 0 1 374 0000100 [SLPQ select 0xc078a744][SLP]
sendmail
368 c137d1fc 0 1 368 0000100 [SLPQ select 0xc078a744][SLP] sshd
252 c12ab3f8 0 1 252 0000000 [SLPQ select 0xc078a744][SLP]
syslogd
227 c12abde4 0 1 227 0000000 [SLPQ select 0xc078a744][SLP] devd
46 c137d5f4 0 0 0 0000204 [SLPQ - 0xcb30cd14][SLP] schedcpu
45 c137d7f0 0 0 0 0000204 [SLPQ - 0xc07923ec][SLP] nfsiod 3
44 c137d9ec 0 0 0 0000204 [SLPQ - 0xc07923e8][SLP] nfsiod 2
43 c137dbe8 0 0 0 0000204 [SLPQ - 0xc07923e4][SLP] nfsiod 1
42 c1243be8 0 0 0 0000204 [SLPQ - 0xc07923e0][SLP] nfsiod 0
41 c1243de4 0 0 0 0000204 [SLPQ vlruwt 0xc1243de4][SLP]
vnlru
40 c12a8000 0 0 0 0000204 [SLPQ syncer 0xc0785b0c][SLP]
syncer
39 c12a81fc 0 0 0 0000204 [SLPQ psleep 0xc078ac90][SLP]
bufdaemon
38 c12a83f8 0 0 0 000020c [SLPQ pgzero 0xc07989e4][SLP]
pagezero
9 c12a85f4 0 0 0 0000204 [SLPQ psleep 0xc0798534][SLP]
vmdaemon
8 c12a87f0 0 0 0 0000204 [SLPQ psleep 0xc07984f0][SLP]
pagedaemon
37 c12a89ec 0 0 0 0000204 [IWAIT] swi0: sio
7 c12a8be8 0 0 0 0000204 [SLPQ - 0xc120983c][SLP] fdc0
36 c12a8de4 0 0 0 0000204 [SLPQ usbtsk 0xc0780814][SLP]
usbtask
35 c12ab000 0 0 0 0000204 [SLPQ usbevt 0xc12bb210][SLP] usb0
34 c12ab1fc 0 0 0 0000204 [IWAIT] swi5:+
6 c12335f4 0 0 0 0000204 [SLPQ - 0xc1263dc0][SLP] thread
taskq
33 c12337f0 0 0 0 0000204 [IWAIT] swi6:+
32 c12339ec 0 0 0 0000204 [IWAIT] swi6: task queue
5 c1233be8 0 0 0 0000204 [SLPQ - 0xc128e040][SLP] kqueue
taskq
31 c1233de4 0 0 0 0000204 [IWAIT] swi2: cambio
30 c1243000 0 0 0 0000204 [SLPQ - 0xc077e7a0][SLP] yarrow
4 c12431fc 0 0 0 0000204 [SLPQ - 0xc0783148][SLP] g_down
3 c12433f8 0 0 0 0000204 [SLPQ - 0xc0783144][SLP] g_up
2 c12435f4 0 0 0 0000204 [SLPQ - 0xc078313c][SLP] g_event
29 c12437f0 0 0 0 0000204 [IWAIT] swi3: vm
28 c12439ec 0 0 0 000020c [IWAIT] swi4: clock sio
27 c12191fc 0 0 0 0000204 [IWAIT] swi1: net
26 c12193f8 0 0 0 0000204 [IWAIT] irq15: ata1
25 c12195f4 0 0 0 0000204 [IWAIT] irq14: ata0
24 c12197f0 0 0 0 0000204 [IWAIT] irq13:
23 c12199ec 0 0 0 0000204 [IWAIT] irq12:
22 c1219be8 0 0 0 0000204 [IWAIT] irq11:
21 c1219de4 0 0 0 0000204 [IWAIT] irq10:
20 c1233000 0 0 0 0000204 [IWAIT] irq9: twe0
19 c12331fc 0 0 0 0000204 [IWAIT] irq8: rtc
18 c12333f8 0 0 0 0000204 [IWAIT] irq7: ppc0
17 c1211000 0 0 0 0000204 [IWAIT] irq6: fdc0
16 c12111fc 0 0 0 0000204 [IWAIT] irq5: bge0
15 c12113f8 0 0 0 0000204 [IWAIT] irq4: sio0
14 c12115f4 0 0 0 0000204 [IWAIT] irq3: uhci0
13 c12117f0 0 0 0 0000204 [IWAIT] irq1: atkbd0
12 c12119ec 0 0 0 0000204 [IWAIT] irq0: clk
11 c1211be8 0 0 0 000020c [CPU 0] idle
1 c1211de4 0 0 1 0004200 [SLPQ wait 0xc1211de4][SLP] init
10 c1219000 0 0 0 0000204 [SLPQ ktrace 0xc0783d38][SLP]
ktrace
0 c07832a0 0 0 0 0000200 [SLPQ sched 0xc07832a0][SLP]
swapper
517 c162c5f4 1062 516 517 0006000 zomb[INACTIVE] sh
db> show pcpu
cpuid = 0
curthread = 0xc12125c0: pid 11 "idle"
curpcb = 0xc69b7da0
fpcurthread = none
idlethread = 0xc12125c0: pid 11 "idle"
APIC ID = 0
currentldt = 0x28
Joe 'sh' process in ufs:
db> trace 3296
Tracing pid 3296 tid 100255 td 0xc1cfa730
sched_switch(c1cfa730,0,1) at sched_switch+0x17f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c13c4a78,cbad385c,c0526901,c13c4a78,0) at sleepq_switch+0xe0
sleepq_wait(c13c4a78,0,0,c13c4a78,1010040) at sleepq_wait+0x30
msleep(c13c4a78,c0784458,50,c06f08fe,0) at msleep+0x2d1
acquire(cbad38b4,1010040,600,c1cfa730,0) at acquire+0x9a
lockmgr(c13c4a78,1010002,c13c49b4,c1cfa730,cbad38d4) at lockmgr+0x382
vop_stdlock(cbad3904) at vop_stdlock+0x1b
VOP_LOCK_APV(c076f340,cbad3904,cbad38f4,c06c279e,cbad3904) at
VOP_LOCK_APV+0x7e
ffs_lock(cbad3904) at ffs_lock+0x3b
VOP_LOCK_APV(c076ed60,cbad3904) at VOP_LOCK_APV+0x7e
vn_lock(c13c49b4,10002,c1cfa730,2,c136a800) at vn_lock+0xc2
vget(c13c49b4,10002,c1cfa730,c1cfa730,c13c4bdc) at vget+0xc9
ufs_ihashget(c136a800,2,2,cbad3a00,c06e98f4) at ufs_ihashget+0x98
ffs_vget(c1370800,2,2,cbad3a00) at ffs_vget+0x37
ufs_root(c1370800,cbad3a3c,c1cfa730,1,1) at ufs_root+0x18
lookup(cbad3c34,c05197b6,1,c1cfa730,c06ea9fd) at lookup+0x593
namei(cbad3c34,c05197b6,c1cf9460,1,0) at namei+0x34e
do_execve(c1cfa730,cbad3cc4,0,0,cbad3cc4) at do_execve+0x15d
kern_execve(c1cfa730,cbad3cc4,0,c6295000,c6295000) at kern_execve+0x74
execve(c1cfa730,cbad3d14,3,0,282) at execve+0x32
syscall(2f,2f,2f,8067058,806708c) at syscall+0x213
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (59, FreeBSD ELF32, execve), eip = 0x28121d5b, esp =
0xbfbfec1c, ebp = 0xbfbfec48 ---
httpd holding a VV_ROOT vnode lock, blocked in ufs:
db> trace 3057
Tracing pid 3057 tid 100107 td 0xc1807a10
sched_switch(c1807a10,0,1) at sched_switch+0x17f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c1526964,cb707994,c0526901,c1526964,0) at sleepq_switch+0xe0
sleepq_wait(c1526964,0,0,c1526964,1010040) at sleepq_wait+0x30
msleep(c1526964,c0785244,50,c06f08fe,0) at msleep+0x2d1
acquire(cb7079ec,1010040,600,c1807a10,0) at acquire+0x9a
lockmgr(c1526964,1010002,c15268a0,c1807a10,cb707a0c) at lockmgr+0x382
vop_stdlock(cb707a3c) at vop_stdlock+0x1b
VOP_LOCK_APV(c076f340,cb707a3c,cb707a2c,c06c279e,cb707a3c) at
VOP_LOCK_APV+0x7e
ffs_lock(cb707a3c) at ffs_lock+0x3b
VOP_LOCK_APV(c076ed60,cb707a3c) at VOP_LOCK_APV+0x7e
vn_lock(c15268a0,10002,c1807a10,cb707b24,cb707c38) at vn_lock+0xc2
vget(c15268a0,2,c1807a10,1b8,c1807a10) at vget+0xc9
vfs_cache_lookup(cb707b24) at vfs_cache_lookup+0x1bd
VOP_LOOKUP_APV(c076ed60,cb707b24) at VOP_LOOKUP_APV+0x7e
lookup(cb707c10,cb707b70,1,c1807a10,1) at lookup+0x359
namei(cb707c10,cb707bc4,c0672ebb,c19cbad0,c0703dc4) at namei+0x34e
kern_stat(c1807a10,81541f8,0,cb707c84) at kern_stat+0x35
stat(c1807a10,cb707d14,2,0,296) at stat+0x1b
syscall(280b002f,2f,bfbf002f,81541f8,8170) at syscall+0x213
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0x2839359b, esp =
0xbfbfe9dc, ebp = 0xbfbfea78 ---
download.cgi blocked in ufs holding a directory vnode lock:
db> trace 3062
Tracing pid 3062 tid 100050 td 0xc137e000
sched_switch(c137e000,0,1) at sched_switch+0x17f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c163bca0,cb2e5844,c0526901,c163bca0,0) at sleepq_switch+0xe0
sleepq_wait(c163bca0,0,0,c163bca0,1010040) at sleepq_wait+0x30
msleep(c163bca0,c0784578,50,c06f08fe,0) at msleep+0x2d1
acquire(cb2e589c,1010040,600,c137e000,0) at acquire+0x9a
lockmgr(c163bca0,1010002,c163bbdc,c137e000,cb2e58bc) at lockmgr+0x382
vop_stdlock(cb2e58ec) at vop_stdlock+0x1b
VOP_LOCK_APV(c076f340,cb2e58ec,cb2e58dc,c06c279e,cb2e58ec) at
VOP_LOCK_APV+0x7e
ffs_lock(cb2e58ec) at ffs_lock+0x3b
VOP_LOCK_APV(c076ed60,cb2e58ec) at VOP_LOCK_APV+0x7e
vn_lock(c163bbdc,10002,c137e000,cb2e59d4,cb2e5c04) at vn_lock+0xc2
vget(c163bbdc,2,c137e000,1e0,c137e000) at vget+0xc9
vfs_cache_lookup(cb2e59d4) at vfs_cache_lookup+0x1bd
VOP_LOOKUP_APV(c076ed60,cb2e59d4) at VOP_LOOKUP_APV+0x7e
lookup(cb2e5bdc,cb2e5a76,1,c137e000,cb2e5a80) at lookup+0x359
namei(cb2e5bdc,0,d7ec0005,c137e000,c050f370) at namei+0x34e
vn_open_cred(cb2e5bdc,cb2e5cdc,1a4,c19ace00,3) at vn_open_cred+0x27e
vn_open(cb2e5bdc,cb2e5cdc,1a4,3,c07849f8) at vn_open+0x1e
kern_open(c137e000,804a794,0,1,1b6) at kern_open+0xb6
open(c137e000,cb2e5d14,3,1,292) at open+0x1a
syscall(2f,2f,2f,4,2814e280) at syscall+0x213
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x280ba23b, esp = 0xbfbfe10c,
ebp = 0xbfbfe138 ---
httpd holding a directory lock blocked in ufs:
db> trace 3056
Tracing pid 3056 tid 100099 td 0xc162dcf0
sched_switch(c162dcf0,0,1) at sched_switch+0x17f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c17e9400,cb6be994,c0526901,c17e9400,0) at sleepq_switch+0xe0
sleepq_wait(c17e9400,0,0,c17e9400,1010040) at sleepq_wait+0x30
msleep(c17e9400,c0784c38,50,c06f08fe,0) at msleep+0x2d1
acquire(cb6be9ec,1010040,600,c162dcf0,0) at acquire+0x9a
lockmgr(c17e9400,1010002,c17e933c,c162dcf0,cb6bea0c) at lockmgr+0x382
vop_stdlock(cb6bea3c) at vop_stdlock+0x1b
VOP_LOCK_APV(c076f340,cb6bea3c,cb6bea2c,c06c279e,cb6bea3c) at
VOP_LOCK_APV+0x7e
ffs_lock(cb6bea3c) at ffs_lock+0x3b
VOP_LOCK_APV(c076ed60,cb6bea3c) at VOP_LOCK_APV+0x7e
vn_lock(c17e933c,10002,c162dcf0,cb6beb24,cb6bec38) at vn_lock+0xc2
vget(c17e933c,2,c162dcf0,210,c162dcf0) at vget+0xc9
vfs_cache_lookup(cb6beb24) at vfs_cache_lookup+0x1bd
VOP_LOOKUP_APV(c076ed60,cb6beb24) at VOP_LOOKUP_APV+0x7e
lookup(cb6bec10,c17e9450,1,c162dcf0,75c) at lookup+0x359
namei(cb6bec10,c05197b6,c0798060,1,c06ea9fd) at namei+0x34e
kern_stat(c162dcf0,8161ad8,0,cb6bec84) at kern_stat+0x35
stat(c162dcf0,cb6bed14,2,0,292) at stat+0x1b
syscall(280b002f,2f,bfbf002f,8161ad8,8170) at syscall+0x213
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0x2839359b, esp =
0xbfbfe75c, ebp = 0xbfbfe7f8 ---
httpd holding a file lock, blocked in suspfs from ufs_inactive:
db> trace 3045
Tracing pid 3045 tid 100123 td 0xc1996450
sched_switch(c1996450,0,1) at sched_switch+0x17f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c137086c,cb6ecb8c,c0526901,c137086c,0) at sleepq_switch+0xe0
sleepq_wait(c137086c,0,0,1,c153be38) at sleepq_wait+0x30
msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1
vn_write_suspend_wait(c17e933c,c1370800,1,0,c06f2634) at
vn_write_suspend_wait+0xe4
ufs_inactive(cb6ecc14) at ufs_inactive+0x186
VOP_INACTIVE_APV(c076ed60,cb6ecc14) at VOP_INACTIVE_APV+0x7e
vrele(c17e933c,c17e7210,0,c07046ce,19d) at vrele+0x10e
vm_object_vndeallocate(c17e7210) at vm_object_vndeallocate+0xbd
vm_object_deallocate(c17e7210,c17e7210,0,c0703dc4,89e) at
vm_object_deallocate+0x85
vm_map_entry_delete(c19cbbb8,c16258c4,284c8000,c19cbbb8,c16258c4) at
vm_map_entry_delete+0x130
vm_map_delete(c19cbbb8,284c8000,284c9000,0,c1996450) at
vm_map_delete+0x18f
munmap(c1996450,cb6ecd14,2,3,292) at munmap+0xa3
syscall(280b002f,2f,bfbf002f,2831e058,815f2d8) at syscall+0x213
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp =
0xbfbfc6fc, ebp = 0xbfbfc718 ---
Another httpd holding a file lock, blocked in suspfs from ufs_inactive:
db> trace 3047
Tracing pid 3047 tid 100143 td 0xc19b2450
sched_switch(c19b2450,0,1) at sched_switch+0x17f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c137086c,cb781b8c,c0526901,c137086c,0) at sleepq_switch+0xe0
sleepq_wait(c137086c,0,0,1,c141ec08) at sleepq_wait+0x30
msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1
vn_write_suspend_wait(c18419b4,c1370800,1,0,c06f2634) at
vn_write_suspend_wait+0xe4
ufs_inactive(cb781c14) at ufs_inactive+0x186
VOP_INACTIVE_APV(c076ed60,cb781c14) at VOP_INACTIVE_APV+0x7e
vrele(c18419b4,c1826000,0,c07046ce,19d) at vrele+0x10e
vm_object_vndeallocate(c1826000) at vm_object_vndeallocate+0xbd
vm_object_deallocate(c1826000,c1826000,0,c0703dc4,89e) at
vm_object_deallocate+0x85
vm_map_entry_delete(c19cbe10,c18bfd8c,284cd000,c19cbe10,c18bfd8c) at
vm_map_entry_delete+0x130
vm_map_delete(c19cbe10,284cd000,284ce000,0,c19b2450) at
vm_map_delete+0x18f
munmap(c19b2450,cb781d14,2,3,292) at munmap+0xa3
syscall(280b002f,2f,bfbf002f,2831e058,815d228) at syscall+0x213
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp =
0xbfbfc6fc, ebp = 0xbfbfc718 ---
mksnap_ffs, creating the snapshot, but blocked on a vnode lock:
db> trace 2993
Tracing pid 2993 tid 100096 td 0xc162d8a0
sched_switch(c162d8a0,0,1) at sched_switch+0x17f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c13c4a78,cb6b560c,c0526901,c13c4a78,0) at sleepq_switch+0xe0
sleepq_wait(c13c4a78,0,0,c13c4a78,1010040) at sleepq_wait+0x30
msleep(c13c4a78,c0784458,50,c06f08fe,0) at msleep+0x2d1
acquire(cb6b5664,1010040,600,c162d8a0,0) at acquire+0x9a
lockmgr(c13c4a78,1010002,c13c49b4,c162d8a0,cb6b5684) at lockmgr+0x382
vop_stdlock(cb6b56b4) at vop_stdlock+0x1b
VOP_LOCK_APV(c076f340,cb6b56b4,cb6b56a4,c06c279e,cb6b56b4) at
VOP_LOCK_APV+0x7e
ffs_lock(cb6b56b4) at ffs_lock+0x3b
VOP_LOCK_APV(c076ed60,cb6b56b4) at VOP_LOCK_APV+0x7e
vn_lock(c13c49b4,10002,c162d8a0) at vn_lock+0xc2
ffs_snapshot(c1370800,c14ccbc0,c14ccbc0,c1369b00,0) at ffs_snapshot+0x1242
ffs_mount(c1370800,c162d8a0,20000000,201000,c13c49b4) at ffs_mount+0x7a5
vfs_domount(c162d8a0,c13207b0,c141c030,1211000,c1320670) at
vfs_domount+0x576
vfs_donmount(c162d8a0,1211000,cb6b5c14,c1422880,e) at vfs_donmount+0xce
kernel_mount(c1320690,1211000,bfbfed33,0,0) at kernel_mount+0x6d
ffs_cmount(c1320690,bfbfe578,1211000,c162d8a0,c076ea80) at ffs_cmount+0x5d
mount(c162d8a0,cb6b5d14,4,2,206) at mount+0x156
syscall(2f,2f,2f,bfbfe7c8,bfbfed29) at syscall+0x213
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (21, FreeBSD ELF32, mount), eip = 0x280b80db, esp =
0xbfbfe4ec, ebp = 0xbfbfebe0 ---
Anothr httpd blocked in suspfs while holding a file vnode lock, trying to
unmap:
db> trace 2609
Tracing pid 2609 tid 100108 td 0xc1807b80
sched_switch(c1807b80,0,1) at sched_switch+0x17f
mi_switch(1,0) at mi_switch+0x264
sleepq_switch(c137086c,cb70ab8c,c0526901,c137086c,0) at sleepq_switch+0xe0
sleepq_wait(c137086c,0,0,1,c14cae38) at sleepq_wait+0x30
msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1
vn_write_suspend_wait(c19fd9b4,c1370800,1,0,c06f2634) at
vn_write_suspend_wait+0xe4
ufs_inactive(cb70ac14) at ufs_inactive+0x186
VOP_INACTIVE_APV(c076ed60,cb70ac14) at VOP_INACTIVE_APV+0x7e
vrele(c19fd9b4,c1a93948,0,c07046ce,19d) at vrele+0x10e
vm_object_vndeallocate(c1a93948) at vm_object_vndeallocate+0xbd
vm_object_deallocate(c1a93948,c1a93948,0,c0703dc4,89e) at
vm_object_deallocate+0x85
vm_map_entry_delete(c18bd834,c18cb3b8,284c7000,c18bd834,c18cb3b8) at
vm_map_entry_delete+0x130
vm_map_delete(c18bd834,284c7000,284c8000,0,c1807b80) at
vm_map_delete+0x18f
munmap(c1807b80,cb70ad14,2,37,296) at munmap+0xa3
syscall(2f,2f,bfbf002f,2831e058,815d0c8) at syscall+0x213
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp =
0xbfbfc5cc, ebp = 0xbfbfc5e8 ---
No WITNESS in this kernel.
My first glance reading is that there's a deadlock between the snapshot
lock and vnode locks, which ocurs because a vnode lock has been acquired
as part of the mmap tear-down, which leads to vn_write_suspend() being
called in ufs_inactive as the inode has a zero reference count so has to
be GC'd, which involves writing to the file system. VOP_INACTIVE is
called by vrele() with the vnode lock held. My understanding is that,
generally, we acquire the file system snapshot lock before vnode locks as
part of the VFS lock order.
This could be a misinterpretation. I'll try to get a dump, but this is a
twe box so I may not be able to.
Robert N M Watson
More information about the freebsd-current
mailing list