ZFS makes SSDs faster than memory!

[Attila Nagy]

On 07/23/10 14:15, Ivan Voras wrote:
> On 07/23/10 12:28, Attila Nagy wrote:
>    
>> Hi,
>>
>> I've came across a strange issue. On a file server (ftp/http/rsync)
>> there is a dual SSD based L2ARC configured for a pool of 24 disks:
>>      
>    
>> fetch -o /dev/null -4
>> http://ftp.fsn.hu/pub/CDROM-Images/opensolaris/osol-0906-106a-ai-sparc.iso
>> /dev/null                                     100% of  493 MB   11 MBps
>>      
> If I understand your setup and your benchmark correctly, you are saying
> you have achieved 11 megabytes / s performance out of a volume of 24
> RAIDZ2 drives split into two parts (so it's like RAID 60). Doesn't this
> number seem extremely low to you, considering that (if recent models)
> each of your drives can probably pull at least 70 MB/s?
>    
First of all, it's not an isolated system, there are active users on it. 
But you are right, 11 MiBps is far from being the max out of this 
hardware, even considering that the CPUs are somewhat old (2xAMD Opteron 
246, 2GHz).
When pulling this amount of data out of the machine, the disks aren't 
saturated, they are at around 10-20% of utilization according to gstat.
BTW, remember that two RAIDZ2 in stripe isn't RAID60. In RAIDZ2 every 
read involves a full stripe (er, block) read for checksum validation, 
which means at a 128 kiB blocksize and with 12 disks in a RAIDZ2 pool, 
all disks provide their part of that 128k read.
That's why a RAIDZ2 pool's IO performance equals of one disk's.
The disks in a normal 20-30 MiBps network load do about 30-40 read IOPS, 
you are right that they are capable of more (around 100-120).