NFSD hang

Wed Sep 28 00:06:51 UTC 2011

Jeremy Chadwick wrote:
> On Tue, Sep 27, 2011 at 05:41:57AM -0700, Kirill Yelizarov wrote:
> > From: Jeremy Chadwick <freebsd at jdc.parodius.com>
> > To: Kirill Yelizarov <ykirill at yahoo.com>
> > Cc: rmacklem at uoguelph.ca; freebsd-stable at freebsd.org
> > Sent: Tuesday, September 27, 2011 3:59 PM
> > Subject: Re: NFSD hang
> >
> > On Tue, Sep 27, 2011 at 04:04:10AM -0700, Kirill Yelizarov wrote:
> > > I found a had sync enabled on my server so I set?
> > > zfs?set?sync=disabled data
> > > and will look for failures. Are there any other setting for nfs
> > > over zfs i can check or set?
> > >
> > > ________________________________
> > >
> > > # uname -a
> > > FreeBSD brat.faberlic.com 8.2-STABLE FreeBSD 8.2-STABLE #0: Thu
> > > Jun? 9 11:22:38 MSD 2011???? root@**:/usr/obj/usr/src/sys/BRAT?
> > > amd64 Sources were taken at that time
> > >
> > > There are a lot of this. Should i paste them all here or part is
> > > enough?
> > >
> > > brat# procstat -k -k 1666
> > > ? PID??? TID COMM???????????? TDNAME??????????
> > > KSTACK??????????????????????
> > > ?1666 100323 nfsd???????????? nfsd: master???? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939 svc_run+0x8b
> > > nfssvc_nfsd+0x97 nfssvc_nfsserver+0x53 nfssvc+0x44
> > > syscallenter+0x186 syscall+0x40 Xfast_syscall+0xe2
> > > ?1666 100391 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100392 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100393 nfsd???????????? nfsd: service???
> > > <running>???????????????????
> > > ?1666 100394 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100395 nfsd???????????? nfsd: service???
> > > <running>???????????????????
> > > ?1666 100396 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100397 nfsd???????????? nfsd: service???
> > > <running>???????????????????
> > > ?1666 100398 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100399 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100400 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100401 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100402 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100403 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100404 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100405 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100406 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100407 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100408 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100409 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100410 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100411 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100412 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100413 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100414 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100415 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100416 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100417 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100418 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100419 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100420 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100421 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100422 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100423 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100424 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100425 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100426 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100427 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100428 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100429 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100430 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100431 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100432 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100433 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100434 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100435 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100436 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100437 nfsd???????????? nfsd: service???
> > > <running>???????????????????
> > > ?1666 100438 nfsd???????????? nfsd: service???
> > > <running>???????????????????
> > > ?1666 100439 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100440 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100441 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100442 nfsd???????????? nfsd: service???
> > > <running>???????????????????
> > > ?1666 100443 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100444 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100445 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100446 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > > ?1666 100447 nfsd???????????? nfsd: service??? mi_switch+0x176
> > > sleepq_catch_signals+0x309 sleepq_timedwait_sig+0x12
> > > _cv_timedwait_sig+0x11d svc_run_internal+0x939
> > > svc_thread_start+0xb fork_exit+0x114 fork_trampoline+0xe
> > >
> > >
> > >
> > > ________________________________
> > > From: Jeremy Chadwick <freebsd at jdc.parodius.com>
> > > To: Kirill Yelizarov <ykirill at yahoo.com>
> > > Cc: "freebsd-stable at freebsd.org" <freebsd-stable at freebsd.org>
> > > Sent: Monday, September 26, 2011 10:32 AM
> > > Subject: Re: NFSD hang
> > >
> > > On Sun, Sep 25, 2011 at 11:14:30PM -0700, Kirill Yelizarov wrote:
> > > > Good Day!
> > > > I'v got a problem with nfs share on zfs volume. Everything
> > > > worked fine for a few month and now it hang. This share stores
> > > > logs from 9 servers at night, about 1-2Gb from each server. ZFS
> > > > is filled to 26% and it is v28
> > > >
> > > > last pid: 46573;? load averages: 195.82, 199.86,
> > > > 200.12??????????????????????????????????????????????????????????????????????????????
> > > > up 108+21:56:50 10:05:06
> > > > 432 processes: 208 running, 224 sleeping
> > > > CPU:? 0.0% user,? 0.0% nice,? 100% system,? 0.0% interrupt,?
> > > > 0.0% idle
> > > > Mem: 280M Active, 1469M Inact, 9584M Wired, 161M Cache, 1232M
> > > > Buf, 311M Free
> > > > Swap: 16G Total, 16G Free
> > > >
> > > > ? PID USERNAME????? THR PRI NICE?? SIZE??? RES STATE?? C??
> > > > TIME?? WCPU COMMAND
> > > > ?1666 root????????? 256? 76??? 0? 5788K? 5120K RUN??? 14 476.8H
> > > > 1508.64% nfsd
> > > >
> > > > # zpool list
> > > > NAME?? SIZE? ALLOC?? FREE??? CAP? DEDUP? HEALTH? ALTROOT
> > > > data? 3.62T?? 954G? 2.69T??? 25%? 1.00x? ONLINE? -
> > > >
> > > > # zfs list
> > > > NAME?? USED? AVAIL? REFER? MOUNTPOINT
> > > > data?? 954G? 2.64T?? 954G? /data
> > > >
> > > > # zfs mount
> > > > data??????????????????????????? /data
> > > >
> > > > What should i look for to resolve it?
> > >
> > > What version of FreeBSD exactly, and what build date?
> > >
> > > Please provide output from "procstat -k -k 1666" (yes, two -k's).
> >
> > Can you explain the correlation between the "sync" parameter (which
> > I
> > have to assume was set to "standard" -- the default -- on all of
> > your
> > filesystems) and your nfsd issue?? I do not see the correlation.
> >
> > My intention of asking for procstat -k -k output (which you did
> > provide;
> > thank you) was for Rick Macklem (who's currently working on NFS on
> > FreeBSD) to chime in with some insights.? He may be busy, but I've
> > CC'd
> > him here.
> >
> > I found it in the wiki http://wiki.freebsd.org/ZFSTuningGuide. So i
> > gave it a try. I thought it is somehow related with zfs because i
> > couldn't even run ls on zfs volume. I had to reset this server
> > because it didn't respond to init commands.
> 
> I still don't see any indication in the procstat output that your
> problem is ZFS-related. To me looks like nfsd is spinning hard; on
> what
> I do not know, but I don't see any ZFS functions in the stack list.
> 
> I would strongly recommend you reconsider tinkering with the "sync"
> parameter, and instead wait for Rick to chime in with some information
> or requests for further details.
> 
Well, I didn't chime in because I don't really have anything useful to say.
I don't think several nfsd threads should be in "run state", but I don't have
any insight as to why they would be. (The other nfsd threads are just waiting
for RPC requests from clients, which is normal.)

I suspect something is making those threads loop, but I don't know how to
figure out where? (In the bad old days, I would have exscaped to a debugger and
looked where the program counter was, then repeated after a "cont" a few times,
to see where they were executing. However, I have no idea how to do that on a
multicore system, even if you still had the system sitting there?
If someone does know how to do this, please feel free to chime in;-)

Beyond the above, if it happens again, trying to look for some resource
exhaustion might help. "vmstat -m" and "vmstat -z" gives you the dynamically
allocated stuff.

I know nothing about zfs, but if "sync=disable" makes it ignore VOP_SYNC()
ops, it will be "risky" for NFS exported volumes. (NFS assumes everything
related to a file is committed to stable storage such that it won't be lost
upon a crash/reboot, once VOP_SYNC() has been called for the vnode. If that
isn't the case and your server crashes, you could lose recent file modifications.
Some care, some don't, but you need to be aware of this.)

rick
ps: You could try the new/experimental server by adding the "-e" option to
    both mountd and nfsd. It might make a difference, since it does certain
    things like the duplicate request cache, differently.

> Furthermore, your reply removed Rick from the thread. I've put him
> back
> in the CC list. Please follow mailing list etiquette. Thank you.
> 
> --
> | Jeremy Chadwick jdc at parodius.com |
> | Parodius Networking http://www.parodius.com/ |
> | UNIX Systems Administrator Mountain View, CA, US |
> | Making life hard for others since 1977. PGP 4BD6C0CB |
> 
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
> "freebsd-stable-unsubscribe at freebsd.org"