ZFS: is getting zfs slower with more data on the fs?

[Oliver Lehmann]

Hi,

I have for example 2 filesystems:

  NAME            zroot/usr              zroot/usr/testfs
  TYPE            filesystem             filesystem
  CREATION        Wed Sep  7 21:45 2011  Tue Jan 22 21:47 2013
  USED            15.6G                  31K
  AVAIL           7.21G                  20.0G
  REFER           12.8G                  31K
  RATIO           1.00x                  1.00x
  MOUNTED         yes                    yes
  ORIGIN          -                      -
  QUOTA           none                   none
  RESERV          none                   none
  VOLSIZE         -                      -
  VOLBLOCK        -                      -
  RECSIZE         128K                   128K
  MOUNTPOINT      legacy                 legacy
  SHARENFS        off                    off
  CHECKSUM        fletcher4              fletcher4
  COMPRESS        off                    off
  ATIME           off                    off
  DEVICES         on                     on
  EXEC            on                     on
  SETUID          on                     on
  RDONLY          off                    off
  JAILED          off                    off
  SNAPDIR         hidden                 hidden
  ACLMODE         discard                discard
  ACLINHERIT      restricted             restricted
  CANMOUNT        on                     on
  XATTR           off                    off
  COPIES          1                      1
  VERSION         5                      5
  UTF8ONLY        off                    off
  NORMALIZATION   none                   none
  CASE            sensitive              sensitive
  VSCAN           off                    off
  NBMAND          off                    off
  SHARESMB        off                    off
  REFQUOTA        20G                    20G
  REFRESERV       none                   none
  PRIMARYCACHE    all                    all
  SECONDARYCACHE  all                    all
  USEDSNAP        0                      0
  USEDDS          12.8G                  31K
  USEDCHILD       2.82G                  0
  USEDREFRESERV   0                      0
  DEFER_DESTROY   -                      -
  USERREFS        -                      -
  LOGBIAS         latency                latency
  DEDUP           off                    off
  MLSLABEL        standard               standard
  SYNC            1.00x                  1.00x
  REFRATIO        12.8G                  31K
  WRITTEN         -                      -

And I wonder why I have different write performance on
each of those filesystems:


zroot/usr:

Version  1.97       ------Sequential Output------ --Sequential Input-  
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--  
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP   
/sec %CP
nudel.salatsc 6544M    18  99 48439  50 37901  47    46  99 188508  87  
214.8  14
Latency               585ms    1550ms     780ms     223ms     388ms     214ms


zroot/usr/testfs:

Version  1.97       ------Sequential Output------ --Sequential Input-  
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--  
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP   
/sec %CP
nudel.salatsc 6544M    18  99 80818  91 55605  74    46  99 184481  85  
225.7  15
Latency               631ms    1159ms    2156ms     199ms     361ms     239ms



root at nudel testfs> zpool status
   pool: zroot
  state: ONLINE
   scan: scrub repaired 0 in 0h9m with 0 errors on Fri Jan  4 15:55:10 2013
config:

         NAME        STATE     READ WRITE CKSUM
         zroot       ONLINE       0     0     0
           mirror-0  ONLINE       0     0     0
             ada0p3  ONLINE       0     0     0
             ada1p3  ONLINE       0     0     0

errors: No known data errors
root at nudel testfs>


/usr is filled with 12GB of data, /usr/testfs is empty.

Why is it like this? I'm trying to figure out how to configure a ZFS fs as
fast as possible for things like /usr/obj where I do not care about data
security that much. My "testfs" is a filesystem where I wanted to test
different things - but now it is faster without any changes?!

PS: please keep me CCed