ZFS makes SSDs faster than memory!

[Attila Nagy]

On 07/23/10 14:50, Bernd Walter wrote:
> On Fri, Jul 23, 2010 at 02:15:44PM +0200, Ivan Voras wrote:
>    
>> On 07/23/10 12:28, Attila Nagy wrote:
>>      
>>> Hi,
>>>
>>> I've came across a strange issue. On a file server (ftp/http/rsync)
>>> there is a dual SSD based L2ARC configured for a pool of 24 disks:
>>>        
>>      
>>> fetch -o /dev/null -4
>>> http://ftp.fsn.hu/pub/CDROM-Images/opensolaris/osol-0906-106a-ai-sparc.iso
>>> /dev/null                                     100% of  493 MB   11 MBps
>>>        
>> If I understand your setup and your benchmark correctly, you are saying
>> you have achieved 11 megabytes / s performance out of a volume of 24
>> RAIDZ2 drives split into two parts (so it's like RAID 60). Doesn't this
>> number seem extremely low to you, considering that (if recent models)
>> each of your drives can probably pull at least 70 MB/s?
>>      
> It is also quite strange that a linear read file gets stored in L2ARC,
> which usually holds random accessed data.
> Maybe it is very fragmented on disks.
> L2ARC with MLC drives usually is much slower than modern disks when
> it comes to linear reads.
>    
There is no linear reads here from the PoV of the disks. Exactly one 
stream of linear read is linear read, but two streams are not. :)
Maybe I should have written this first, but I'm not the only one reading 
from the machine.
For random reads even the cheapest MLC outperforms a 7k2 SATA disk (only 
reads), and this is an Intel stuff, which can do 3000 RIOPS easily.
> Are there any facts backup your assumption that data is really
> read from memory, SSD, disk in the named cases?
> E.g. by ARC/L2ARC and IO statistics.
>    
Yes. When downloading from L2ARC:
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
     0    174    174  21505    0.8      0      0    0.0   13.3| ad4
     0    169    169  21479    0.9      0      0    0.0   15.0| ad6
when downloading from ARC:
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
     0     26     19   1129    0.6      7     78    0.4    1.3| ad4
     0     19     12   1436    1.1      7     78    0.3    1.4| ad6