ZFS makes SSDs faster than memory!

[Attila Nagy]

Hi,

I've came across a strange issue. On a file server (ftp/http/rsync) 
there is a dual SSD based L2ARC configured for a pool of 24 disks:
         NAME                 STATE     READ WRITE CKSUM
         data                 ONLINE       0     0     0
           raidz2             ONLINE       0     0     0
             label/disk20-01  ONLINE       0     0     0
             label/disk20-02  ONLINE       0     0     0
             label/disk20-03  ONLINE       0     0     0
             label/disk20-04  ONLINE       0     0     0
             label/disk20-05  ONLINE       0     0     0
             label/disk20-06  ONLINE       0     0     0
             label/disk20-07  ONLINE       0     0     0
             label/disk20-08  ONLINE       0     0     0
             label/disk20-09  ONLINE       0     0     0
             label/disk20-10  ONLINE       0     0     0
             label/disk20-11  ONLINE       0     0     0
             label/disk20-12  ONLINE       0     0     0
           raidz2             ONLINE       0     0     0
             label/disk21-01  ONLINE       0     0     0
             label/disk21-02  ONLINE       0     0     0
             label/disk21-03  ONLINE       0     0     0
             label/disk21-04  ONLINE       0     0     0
             label/disk21-05  ONLINE       0     0     0
             label/disk21-06  ONLINE       0     0     0
             label/disk21-07  ONLINE       0     0     0
             label/disk21-08  ONLINE       0     0     0
             label/disk21-09  ONLINE       0     0     0
             label/disk21-10  ONLINE       0     0     0
             label/disk21-11  ONLINE       0     0     0
             label/disk21-12  ONLINE       0     0     0
         cache
           ad4s2              ONLINE       0     0     0
           ad6s2              ONLINE       0     0     0

and there is about 6GB of ARC (in memory).

The strange thing is that when I fetch a file with HTTP, it comes from 
disks:
fetch -o /dev/null -4 
http://ftp.fsn.hu/pub/CDROM-Images/opensolaris/osol-0906-106a-ai-sparc.iso
/dev/null                                     100% of  493 MB   11 MBps 
00m00s
a second fetch right after the above comes from in-memory ARC:
/dev/null                                     100% of  493 MB   28 MBps 
00m00s

If I wait some minutes, the blocks are evicted from memory to SSD, so 
the next fetch comes from the L2ARC on the two SSDs:
/dev/null                                     100% of  493 MB   43 MBps 
00m00s
and if I re-fetch it right after the above (blocks will be served from 
memory again), I get:
/dev/null                                     100% of  493 MB   27 MBps 
00m00s

And this is very consistent. I watch L2ARC activity with gstat/zpool 
iostat and when the bits are served from that, the throughput is 
constantly higher with at least 10MBps, and the next download will 
always be slower, no matter that it comes from memory.

The effect can't be seen on localhost, with dd:
# dd if=osol-0906-106a-ai-sparc.iso of=/dev/null bs=1M
493+1 records in
493+1 records out
516968448 bytes transferred in 3.310415 secs (156164240 bytes/sec) 
(comes from L2ARC)
# dd if=osol-0906-106a-ai-sparc.iso of=/dev/null bs=1M
493+1 records in
493+1 records out
516968448 bytes transferred in 0.861610 secs (600002886 bytes/sec) 
(comes from memory)

The daemons run in jails and they see the data through a read only 
nullfs mount. However dd-ing from that nullfs mount also gives fast 
throughput:
# dd if=osol-0906-106a-ai-sparc.iso of=/dev/null bs=1M
493+1 records in
493+1 records out
516968448 bytes transferred in 0.718028 secs (719983859 bytes/sec)

I'm lost. Anyone else can reproduce this?

BTW, I see another strange issue with ZFS. On the site there are sparse 
files for testing clients' compatibility with big files.
Fetching these files are slower than the ones, which contains real data 
and therefore moves the disks!
What should cause this? It seems very unnatural to fetch a zero byte 
file with 10 MBps, which doesn't involve even a bit of disk IO, while 
fetching real files with real disk IO faster...

Thanks,