immense delayed write to file system (ZFS and UFS2), performance issues

[O. Hartmann]

On 01/19/10 10:09, krad wrote:
> 2010/1/18 Morgan Wesstr�m<freebsd-questions at pp.dyndns.biz>
>
>> O. Hartmann wrote:
>>> I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes.
>>> All boxes have the most recent STABLE. One box is a UP system, two
>>> others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores
>>> (Dell Poweredge III).
>>>
>>> Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or
>>> so, sometimes the I/O performance drops massively when doing 'svn
>>> update', 'make world' or even 'make kernel'. It doesn't matter what
>>> memory and how many cpu the box has, it get stuck for several seconds
>>> and freezing. On the UP box, this is sometimes for 10 - 20 seconds.
>>> A very interesting phenomenon is the massively delayed file writing on
>>> ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm
>>> and having in another Xterminal my shell for compiling this file, it
>>> takes sometimes up to 20 seconds to get the file updated after it has
>>> been written. It's like having an old, slow NFS connection with long
>>> cache delays.
>>> These massively delayed file transactions are not necessarely under
>>> heavy load, sometimes they occur in a relaxed situation. They seem to
>>> occur much more often on the UP box than on the SMP boxes, but this
>>> strange phenomenon also occur on the Dell Poweredge II, which has 16GB
>>> RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and
>>> UFS2 filesystems as well. It is hardly reproducable.
>>>
>>> Is there any known issue?
>>>
>>> Ragrds,
>>> Oliver
>>
>>
>> The disks involved don't happen to be Western Digital Green Power disks,
>> do they? The Intelli-Park function in these disks are wrecking havoc
>> with I/O in Linux-land at least, causing massive stalls and iowait
>> through the roof during the 25-30 seconds it takes for the heads to
>> unload after parking. I have two of these disks sitting on my desk now
>> collecting dust...
>> /Morgan
>> _______________________________________________
>> freebsd-questions at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
>> To unsubscribe, send any mail to "
>> freebsd-questions-unsubscribe at freebsd.org"
>>
>
>
> ZFS is copy on write, therefore to optimize the write performance it delays
> writes for a long as possible, upto a set maximum time. It will then flush
> to the disks. How long this time is depends on how much free ram you have
> available. Assuming processes are eating up all your ram I would imagine you
> are hitting the max limit. I'm not sure exactly what its set to on bsd but I
> know the default on opensolaris is 30s. I think this explains your delayed
> writes.
>
> Not sure what will cause the lock ups though.
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"

This could end in a bad situation, where one process writes a files, say 
with some arbitrary stuff and another successing process is intended to 
read this file. even if the processes are run serial, those 'delays' 
could break the chain! The delay situation in a development environment 
is harsh, but in other circumstances it could develop very bad.

I see this strange behaviour now for several weeks, something essential 
has changed in the code, I guess.
On UP boxes the situation is worse sometimes, on SMp boxes with lots of 
RAM ( 8 and 16 GB and 4 or 8 CPU cores) it is still bad. I have a server 
that acts as a 'rsync' backup system gathering data from satellite 
servers from time to time. Since this problem of slowness occured, this 
4-core 8 gig RAM box crawls for minutes. Even when X11 is disabled 
working on console is 'bumpy': terminal out slows down, mouse pointer 
jumps etc.As I wrote, the same on a 8 core/16 gig box, but not that harsh.