ZFS vdev I/O questions

[Mark Felder]

On Tue, Jul 16, 2013 at 02:41:31PM +0300, Daniel Kalchev wrote:
> I am observing some "strange" behaviour with I/O spread on ZFS vdevs and 
> thought I might ask if someone has observed it too.
> 

--SNIP--

> Drives da0-da5 were Hitachi Deskstar 7K3000 (Hitachi HDS723030ALA640, 
> firmware MKAOA3B0) -- these are 512 byte sector drives, but da0 has been 
> replaced by Seagate Barracuda 7200.14 (AF) (ST3000DM001-1CH166, firmware 
> CC24) -- this is an 4k sector drive of a new generation (notice the 
> relatively 'old' firmware, that can't be upgraded).

--SNIP--

> The other observation I have is with the first vdev: the 512b drives do 
> a lot of I/O fast, complete first and then sit idle, while da0 continues 
> to write for many more seconds. They consistently show many more IOPS 
> than the other drives for this type of activity -- on streaming writes 
> all drives behave more or less the same. It is only on this un-dedup 
> scenario where the difference is so much pronounced.
> 
> All the vdevs in the pool are with ashift=12 so the theory that ZFS 
> actually issues 512b writes to these drives can't be true, can it?
> 
> Another worry is this Seagate Barracuda 7200.14 (AF) 
> (ST3000DM001-1CH166, firmware CC24) drive. It seems constantly 
> under-performing. Does anyone know if it is so different from the 
> ST3000DM001-9YN166 drives? Might be, I should just replace it?
>

A lot of information here.

Those Hitachis are great drives. The addition of the Barracuda with
different performance characteristics could be part of the problem. I'm
glad you pointed out that the pool ashift=12 so we can try to rule that
out. I'd be quite interested in knowing if some or perhaps even all of
your issues go away simply by replacing that drive with another Hitachi.