ZFS freeze/livelock

[Ronald Klop]

On Sun, 10 Oct 2010 21:34:15 +0200, Jeremy Chadwick  
<freebsd at jdc.parodius.com> wrote:

> On Sun, Oct 10, 2010 at 05:34:39PM +0200, Willem Jan Withagen wrote:
>> Just had my FreeBSD freeze on me with what I would think is sort of
>> an livelock....
>>
>> While I was receiving zfs snapshots on my data pool.
>>
>> Top and systat just kept running,
>> but anything getting near a shell (and perhaps disk-io) ended up in:
>>
>> root at zfs.digiware.nl# gpart create -s gpt da6
>> load: 0.00  cmd: csh 12393 [zfsvfs->z_teardown_inactive_lock] 26.12r
>> 0.00u 0.00s 0% 2480k
>> load: 0.10  cmd: csh 12393 [zfsvfs->z_teardown_inactive_lock] 96.01r
>> 0.00u 0.00s 0% 2480k
>>
>> Trying to execute to execute shutdown -r now had no effect what so ever.
>> Neither did the three-finger salute.
>> (Well at least not in 60 sec I was willing to wait.)
>>
>> Only way out of this situation was hard-reset. And I do have to
>> admit I like ZFS for the speed it recovers after unexpected reboot.
>>
>> To bad there was no alt-ctrl-backspace escape to debugger compiled
>> in. I'll do that with the next kernel, just in case.
>>
>> So the only data point I can give is the ^T output above.
>
> We don't know what FreeBSD version you're using (specifically uname -a
> output, since build date matters), but if it's RELENG_8 with ZFS v15,
> you might check out this thread (be sure to read Kai and I's diagnoses):
>
> http://lists.freebsd.org/pipermail/freebsd-fs/2010-October/009687.html
>
> I'm in the process of moving all of my machines, including my home
> server, over to gmirror.  (Home machine started showing signs of serious
> ZFS performance degredation; mutt doing a stat() on 24 files and
> directories total taking literally 0.4 seconds on a dual-core machine.
> Makes no sense, doesn't happen with UFS2, I'm done.)
>

Sorry to hear it didn't work out for you this time. But if you are running  
very important things on very fresh code you should make some testing  
stage or fail over to older versions available or be able to go back from  
backup or ... . At my company we roll out new minor version updates of  
mysql now and than, but always make sure we have an old version available.  
Our customers are more important than running the latest versions.

Home machines are different with regards to having plenty of backup  
machines, but is it possible to give a developer an temporary account to  
debug this? That would help the project going forward.

Ronald.