Write cache, is write cache, is write cache?

[Bernd Walter]

On Sat, Jan 22, 2011 at 10:39:13AM +0000, Karl Pielorz wrote:
> 
> Hi,
> 
> I've a small HP server I've been using recently (an NL36). I've got ZFS 
> setup on it, and it runs quite nicely.
> 
> I was using the server for zeroing some drives the other day - and noticed 
> that a:
> 
>  dd if=/dev/zero of=/dev/ada0 bs=2m
> 
> Gives around 12Mbyte/sec throughput when that's all that's running on the 
> machine.
> 
> Looking in the BIOS is a "Enabled drive write cache" option - which was set 
> to 'No'. Changing it to 'Yes' - I now get around 90-120Mbyte/sec doing the 
> same thing.
> 
> Knowing all the issues with IDE drives and write caches - is there any way 
> of telling if this would be safe to enable with ZFS? (i.e. if the option is 
> likely to be making the drive completely ignore flush requests?) - or if 
> it's still honouring the various 'write through' options if set on data to 
> be written?
> 
> I'm presuming DD won't by default be writing the data with the 'flush' bit 
> set - as it probably doesn't know about it.
> 
> Is there anyway of testing this? (say using some tool to write the data 
> using either lots of 'cache flush' or 'write through' stuff) - and seeing 
> if the performance drops back to nearer the 12Mbyte/sec?
> 
> I've not enabled the option with the ZFS drives in the machine - I suppose 
> I could test it.
> 
> Write performance on the unit isn't that bad [it's not stunning] - though 
> with 4 drives in a mirrored set - it probably helps hide some of the impact 
> this option might have.

You are not testing real world performance with dd.
Let me explain.
You write in 2MB chunks, but those are splitted into smaller chunks.
I don't know how big, but lets say 64k - maybe it's bigger with ada driver.
I also don't know if those chunks are written at the same time, but also
let asumme they are not and your values also say they are not.
So for the drive it sees requests to write 64k at once.
If the drive has write cache enabled it directly returns, if the cache
is not enabled then it waits for the physical write to take place.
Now the drive gets the next 64k - if the cache is enabled it directly
returns - if the cache is not enabled it has to wait a complete disk
rotation to write the data, since the blocks to be written just passed
the head during the last write.
The situation is that with cache you write the whole track in one roation,
while writing without cache it takes multiple rotations.
This limits the speed to one write per rotation, which is for a 10krpm
drive with 64k writes 640Mbyte per minute, or 11Mbyte/s.
Now the good side is that with ada you have command queueing, which means
that the OS doesn't need to wait for a write to take place before issuing
the next one - it can issue multiple write requests.
So in the end you can fill up a whole track with outstanding data, the
drive can write a whole track at once and every write call just waits
in parallel.
But with dd you write to the raw device, so there is no caching done by
any filesystem and requests are serialized to the drive - therefor you
won't see any benefit from NCQ as there are no multiple outstanding write
requests known by the OS.
I think msdosfs can't really use that feature, but UFS can, so with
command queueing you can almost get the write cache performance without
the risks.

-- 
B.Walter <bernd at bwct.de> http://www.bwct.de
Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.