ZFS makes SSDs faster than memory!

[Attila Nagy]

On 07/23/10 15:21, Ivan Voras wrote:
> On 07/23/10 15:15, Attila Nagy wrote:
>
>    
>> When pulling this amount of data out of the machine, the disks aren't
>> saturated, they are at around 10-20% of utilization according to gstat.
>> BTW, remember that two RAIDZ2 in stripe isn't RAID60. In RAIDZ2 every
>> read involves a full stripe (er, block) read for checksum validation,
>> which means at a 128 kiB blocksize and with 12 disks in a RAIDZ2 pool,
>> all disks provide their part of that 128k read.
>> That's why a RAIDZ2 pool's IO performance equals of one disk's.
>>      
> Yes, in case of random IOPS you are correct - and in your case it would
> mean that the files are horribly fragmented (torrent downloads? :)). For
> sequential IO, even RAIDZ/1/2 will give N-1/2/3 times the performance of
> a single drive because prefetching will kick in.
>    
Especially when prefetching is disabled. :)
I've had problems with it previously and it was left in this state. 
Turning it off makes that 11 MiBps 30 MiBps, and the disk utilization 
increase, but of course it doesn't affect the fact that SSDs are faster 
than RAM with FreeBSD/zfs.
In fact, it made things worse. SSDs AND HDDs are now faster than RAM. :-O
Damn. ;)
>    
>> The disks in a normal 20-30 MiBps network load do about 30-40 read IOPS,
>> you are right that they are capable of more (around 100-120).
>>      
> Except for the possible fragmentation issue, I think you should get much
> better throughput even with 30-40 IOPS per drive.
>    
This is an FTP/HTTP server, like ftp.freebsd.org (well, exactly like 
that). If there is 20-30 MiBps network load that may be because nobody 
wants to fetch more data. :)