ZFS Performance FreeBSD 9.0 vs. Openindiana

Franz Schober franz.schober at firmos.at
Thu Mar 29 09:09:04 UTC 2012


Hi,

I am testing a new storage hardware (Xeon E5606, 24 GB Ram, LSI1068 
controller in non-raid mode, 8 7200 SATA Disks, 2 OCZ SSD ZIL Mirror), 
for a NFS/iSCSI storage system.

Currently I'm doing basic performance tests of the ZFS filesystem. We 
are using FreeBSD for a couple of years for various services in our 
software development company, so FreeBSD is our first choice.

For performance tests, I use the iozone benchmark with a multi-streaming 
concurrency test,
iozone -o -c -t 8 -r 128k -s 4G (Sync Mode, 8 concurrent workers, 128 k 
Recordsize, 4G working file for every worker to run not only in cache).

ZPool ist a stripe of the 8 SATA Disks and a mirror of the two SSD Log 
disks (here the FreeBSD zpool status):
   pool: pools
  state: ONLINE
  scan: none requested
config:

         NAME        STATE     READ WRITE CKSUM
         pools       ONLINE       0     0     0
           da0       ONLINE       0     0     0
           da1       ONLINE       0     0     0
           da2       ONLINE       0     0     0
           da3       ONLINE       0     0     0
           da4       ONLINE       0     0     0
           da5       ONLINE       0     0     0
           da6       ONLINE       0     0     0
           da7       ONLINE       0     0     0
         logs
           mirror-8  ONLINE       0     0     0
             ada0    ONLINE       0     0     0
             ada1    ONLINE       0     0     0

Comparing FreeBSD 9.0-RELEASE (amd64) to OpenIndiana 151a, I get a 
performance difference, where FreeBSD is slower in most tests,
the pools is freshly created before each test:

FreeBSD:
     Iozone: Performance Test of File I/O
             Version $Revision: 3.397 $
         Compiled for 64 bit mode.
         Build: freebsd

     Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                  Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                  Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                  Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                  Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave 
Boone,
                  Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                  Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                  Ben England.

     Run began: Thu Mar 29 10:39:47 2012

     SYNC Mode.
     Include close in write timing
     Record Size 128 KB
     File size set to 4194304 KB
     Command line used: iozone -o -c -t 8 -r 128k -s 4G
     Output is in Kbytes/sec
     Time Resolution = 0.000001 seconds.
     Processor cache size set to 1024 Kbytes.
     Processor cache line size set to 32 bytes.
     File stride size set to 17 * record size.
     Throughput test with 8 processes
     Each process writes a 4194304 Kbyte file in 128 Kbyte records

     Children see throughput for  8 initial writers     =  150780.28 KB/sec
     Parent sees throughput for  8 initial writers     =  150767.08 KB/sec
     Min throughput per process             =   18846.89 KB/sec
     Max throughput per process             =   18849.44 KB/sec
     Avg throughput per process             =   18847.53 KB/sec
     Min xfer                     = 4193792.00 KB

     Children see throughput for  8 rewriters     =  150051.28 KB/sec
     Parent sees throughput for  8 rewriters     =  150048.26 KB/sec
     Min throughput per process             =   18615.49 KB/sec
     Max throughput per process             =   18837.79 KB/sec
     Avg throughput per process             =   18756.41 KB/sec
     Min xfer                     = 4144896.00 KB

     Children see throughput for  8 readers         = 1001472.78 KB/sec
     Parent sees throughput for  8 readers         = 1001417.37 KB/sec
     Min throughput per process             =  118845.12 KB/sec
     Max throughput per process             =  138027.58 KB/sec
     Avg throughput per process             =  125184.10 KB/sec
     Min xfer                     = 3611520.00 KB

     Children see throughput for 8 re-readers     =  876152.78 KB/sec
     Parent sees throughput for 8 re-readers     =  870610.71 KB/sec
     Min throughput per process             =  106323.10 KB/sec
     Max throughput per process             =  114136.91 KB/sec
     Avg throughput per process             =  109519.10 KB/sec
     Min xfer                     = 3909120.00 KB

     Children see throughput for 8 reverse readers     = 1148130.64 KB/sec
     Parent sees throughput for 8 reverse readers     = 1142916.22 KB/sec
     Min throughput per process             =  126814.09 KB/sec
     Max throughput per process             =  172099.73 KB/sec
     Avg throughput per process             =  143516.33 KB/sec
     Min xfer                     = 3117440.00 KB

     Children see throughput for 8 stride readers     =  299141.80 KB/sec
     Parent sees throughput for 8 stride readers     =  298399.38 KB/sec
     Min throughput per process             =   29238.70 KB/sec
     Max throughput per process             =   45995.27 KB/sec
     Avg throughput per process             =   37392.73 KB/sec
     Min xfer                     = 2667264.00 KB

     Children see throughput for 8 random readers     =  115979.46 KB/sec
     Parent sees throughput for 8 random readers     =  115974.77 KB/sec
     Min throughput per process             =   14312.92 KB/sec
     Max throughput per process             =   14753.71 KB/sec
     Avg throughput per process             =   14497.43 KB/sec
     Min xfer                     = 4068992.00 KB

     Children see throughput for 8 mixed workload     =  188689.19 KB/sec
     Parent sees throughput for 8 mixed workload     =  188673.96 KB/sec
     Min throughput per process             =   12175.37 KB/sec
     Max throughput per process             =   34954.12 KB/sec
     Avg throughput per process             =   23586.15 KB/sec
     Min xfer                     = 1460992.00 KB

     Children see throughput for 8 random writers     =  147947.42 KB/sec
     Parent sees throughput for 8 random writers     =  147939.94 KB/sec
     Min throughput per process             =   18492.69 KB/sec
     Max throughput per process             =   18494.70 KB/sec
     Avg throughput per process             =   18493.43 KB/sec
     Min xfer                     = 4193920.00 KB

     Children see throughput for 8 pwrite writers     =  149780.00 KB/sec
     Parent sees throughput for 8 pwrite writers     =  149772.91 KB/sec
     Min throughput per process             =   18721.60 KB/sec
     Max throughput per process             =   18723.81 KB/sec
     Avg throughput per process             =   18722.50 KB/sec
     Min xfer                     = 4193792.00 KB

     Children see throughput for 8 pread readers     = 1014824.33 KB/sec
     Parent sees throughput for 8 pread readers     = 1014743.46 KB/sec
     Min throughput per process             =  114311.04 KB/sec
     Max throughput per process             =  150336.52 KB/sec
     Avg throughput per process             =  126853.04 KB/sec
     Min xfer                     = 3189376.00 KB



iozone test complete.



OpenIndiana:

     Iozone: Performance Test of File I/O
             Version $Revision: 3.323 $
         Compiled for 32 bit mode.
         Build: Solaris10cc-64

     Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                  Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                  Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                  Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                  Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
                  Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root.

     Run began: Fri Mar 16 14:11:48 2012

     SYNC Mode.
     Include close in write timing
     Record Size 128 KB
     File size set to 4194304 KB
     Command line used: /usr/benchmarks/iozone/iozone -o -c -t 8 -r 128k 
-s 4G
     Output is in Kbytes/sec
     Time Resolution = 0.000001 seconds.
     Processor cache size set to 1024 Kbytes.
     Processor cache line size set to 32 bytes.
     File stride size set to 17 * record size.
     Throughput test with 8 processes
     Each process writes a 4194304 Kbyte file in 128 Kbyte records

     Children see throughput for  8 initial writers     =  163315.97 KB/sec
     Parent sees throughput for  8 initial writers     =  163238.81 KB/sec
     Min throughput per process             =   20399.37 KB/sec
     Max throughput per process             =   20423.74 KB/sec
     Avg throughput per process             =   20414.50 KB/sec
     Min xfer                     = 4189440.00 KB

     Children see throughput for  8 rewriters     =  163415.44 KB/sec
     Parent sees throughput for  8 rewriters     =  163298.96 KB/sec
     Min throughput per process             =   20266.25 KB/sec
     Max throughput per process             =   20611.95 KB/sec
     Avg throughput per process             =   20426.93 KB/sec
     Min xfer                     = 4123904.00 KB

     Children see throughput for  8 readers         = 1180533.94 KB/sec
     Parent sees throughput for  8 readers         = 1180090.09 KB/sec
     Min throughput per process             =  121104.02 KB/sec
     Max throughput per process             =  198012.62 KB/sec
     Avg throughput per process             =  147566.74 KB/sec
     Min xfer                     = 2565376.00 KB

     Children see throughput for 8 re-readers     =  951965.89 KB/sec
     Parent sees throughput for 8 re-readers     =  948751.85 KB/sec
     Min throughput per process             =  118049.84 KB/sec
     Max throughput per process             =  120618.28 KB/sec
     Avg throughput per process             =  118995.74 KB/sec
     Min xfer                     = 4113536.00 KB

     Children see throughput for 8 reverse readers     = 3024070.31 KB/sec
     Parent sees throughput for 8 reverse readers     = 2867010.77 KB/sec
     Min throughput per process             =  318376.59 KB/sec
     Max throughput per process             =  557625.75 KB/sec
     Avg throughput per process             =  378008.79 KB/sec
     Min xfer                     = 2563968.00 KB

     Children see throughput for 8 stride readers     =  335398.05 KB/sec
     Parent sees throughput for 8 stride readers     =  331540.81 KB/sec
     Min throughput per process             =   32520.95 KB/sec
     Max throughput per process             =   62529.34 KB/sec
     Avg throughput per process             =   41924.76 KB/sec
     Min xfer                     = 2210048.00 KB

     Children see throughput for 8 random readers     =  120422.72 KB/sec
     Parent sees throughput for 8 random readers     =  120418.39 KB/sec
     Min throughput per process             =   14456.03 KB/sec
     Max throughput per process             =   15568.85 KB/sec
     Avg throughput per process             =   15052.84 KB/sec
     Min xfer                     = 3894528.00 KB

     Children see throughput for 8 mixed workload     =  191391.66 KB/sec
     Parent sees throughput for 8 mixed workload     =  191249.48 KB/sec
     Min throughput per process             =   12364.50 KB/sec
     Max throughput per process             =   35442.05 KB/sec
     Avg throughput per process             =   23923.96 KB/sec
     Min xfer                     = 1463296.00 KB

     Children see throughput for 8 random writers     =  163556.06 KB/sec
     Parent sees throughput for 8 random writers     =  163218.36 KB/sec
     Min throughput per process             =   20411.16 KB/sec
     Max throughput per process             =   20484.72 KB/sec
     Avg throughput per process             =   20444.51 KB/sec
     Min xfer                     = 4179328.00 KB



iozone test complete.


Now my question: Which parameters in the ZFS Subsystem of FreeBSD are 
tuneable to reach the same performance in FreeBSD,
especially pushing the synchronous write performance ?

Thx a lot,
Franz Schober






More information about the freebsd-fs mailing list