FreeBSD 6.3/7.1 and Linux disk performance test

Sat Feb 7 04:26:17 PST 2009

On Sat, 7 Feb 2009, Omer Faruk Sen wrote:

> I have installed a new server to test performance results. BIOS and RAID 
> BIOS is the latest in this server ( Raid controller is a Intel SRCSASBB8I )

Hi Omer--

Comparing I/O and file system performance is a bit fraught with peril, 
especially given that the performance of disks varies a great deal based on 
where on the disk you're writing, etc.  Ellard and Seltzer have a nice NFS 
benchmarking paper that includes a "Benchmarking traps" section of note in 
this regard:

   http://www.eecs.harvard.edu/~margo/papers/freenix03/

The source of the performance difference you're seeing could come from a 
number of places:

- I/O to different parts of the disk due to partition layouts can perform
   quite differently -- try out diskinfo -t to get an idea of the kind of
   variance you can see.  For example, I get this on my local 3ware array:

   Transfer rates:
 	outside:       102400 kbytes in   4.056540 sec =    25243 kbytes/sec
 	middle:        102400 kbytes in   2.531003 sec =    40458 kbytes/sec
 	inside:        102400 kbytes in   3.947725 sec =    25939 kbytes/sec

   That's a massive performance difference and presumably entirely a result of
   disk layout.  The best way to address this is to make sure you're using the
   same part of the disk for all tests -- so if you're installing different
   OS's on different partitions, use a single shared partition for the test.

- File system can behave quite differently, especially if they vary in size
   and layout strategies.  Do the dd(1) directly to a disk partition, making
   sure to use the right device on Linux (I believe you want the character
   device in order to bypass the buffer cache and compare apples to apples).

So if you go ahead and do dd(1) directly to the same partition using 
unbuffered I/O and no file system, it should become more obvious as to whether 
we're looking at performance loss due to a device driver difference, a file 
system difference, or perhaps just a disk layout difference.

Robert N M Watson
Computer Laboratory
University of Cambridge

>
>
>
> # dmesg |grep -i mfi
> mfi0: <LSI MegaSAS 1078> port 0x2000-0x20ff mem
> 0xb8b00000-0xb8b3ffff,0xb8b40000-0xb8b7ffff irq 16 at device 0.0 on
> pci10
> mfi0: Megaraid SAS driver Ver 3.00
> mfi0: 1870 (287314652s/0x0020/info) - Shutdown command received from host
> mfi0: 1871 (boot + 3s/0x0020/info) - Firmware initialization started
> (PCI ID 0060/1000/1013/8086)
> mfi0: 1872 (boot + 3s/0x0020/info) - Firmware version 1.20.72-0562
> mfi0: 1873 (boot + 3s/0x0020/info) - Board Revision
> mfi0: 1874 (boot + 15s/0x0002/info) - Inserted: PD 08(e0xff/s8)
> mfi0: 1875 (boot + 15s/0x0002/info) - Inserted: PD 08(e0xff/s8) Info:
> enclPd=ffff, scsiType=0, portMap=00,
> sasAddr=5000c5000bcb90b1,0000000000000000
> mfi0: 1876 (boot + 15s/0x0002/info) - Inserted: PD 09(e0xff/s9)
> mfi0: 1877 (boot + 15s/0x0002/info) - Inserted: PD 09(e0xff/s9) Info:
> enclPd=ffff, scsiType=0, portMap=01,
> sasAddr=5000c5000bcb962d,0000000000000000
> mfi0: 1878 (boot + 15s/0x0002/info) - Inserted: PD 0a(e0xff/s10)
> mfi0: 1879 (boot + 15s/0x0002/info) - Inserted: PD 0a(e0xff/s10) Info:
> enclPd=ffff, scsiType=0, portMap=02,
> sasAddr=5000c5000bcb8f8d,0000000000000000
> mfi0: [ITHREAD]
> mfi0: 1880 (287314711s/0x0020/info) - Time established as 02/07/09
> 9:38:31; (52 seconds since power on)
> mfid0: <MFI Logical Disk> on mfi0
> mfid0: 556928MB (1140588544 sectors) RAID volume ''
>
>
> I have installed RHEL 5.3 x64 on this server and here is simple dd
> performance test:
>
> Last login: Fri Feb  6 12:15:32 2009 from 10.0.0.51
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 1.8473 seconds, 443 MB/s
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 2.05412 seconds, 399 MB/s
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 1.94096 seconds, 422 MB/s
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 2.1092 seconds, 388 MB/s
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 1.84414 seconds, 444 MB/s
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 2.09829 seconds, 390 MB/s
>
> And after that I have tested with FreeBSD 6.3 and 7.1:
>
> 6.3:
>
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.534685 secs (231760388 bytes/sec)
> # cd /var/tmp/
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.051668 secs (268443342 bytes/sec)
> # cd /
> # cd /var/tmp/
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.001266 secs (272951481 bytes/sec)
> # cd /home/
> cd: can't cd to /home/
> # cd /usr/
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.678156 secs (222720290 bytes/sec)
> # cd /boot
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> cd /usr/100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 2.985471 secs (274395563 bytes/sec)
> # local
> # pwd
> /usr/local
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.501986 secs (233924406 bytes/sec)
>
> 7.1:
>
> Filesystem       Size    Used   Avail Capacity  Mounted on
> /dev/mfid0s1a    9.7G    2.6G    6.3G    29%    /
> devfs            1.0K    1.0K      0B   100%    /dev
> /dev/mfid0s1d    478G    4.0K    440G     0%    /opt
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.441476 secs (238037393 bytes/sec)
> # cd /
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.132512 secs (261515371 bytes/sec)
> # cd /usr/local/
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.296514 secs (248504951 bytes/sec)
>
> # cd /usr
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.069655 secs (266870386 bytes/sec)
>
> as you can see there is a big difference in just simple dd test. Is
> there additional steps that I can follow to increase performance? By
> the way If I use write-thru cache on this raid card FreeBSD 6.3 and
> FreeBSD 7.1 only gives 13MB/s~ which is very very bad. Linux gives a
> slight decrease on dd test with write-thru cache but freebsd goes from
> 230-270 MB/s to only 13 MB/s
>
>
> Regards.
> _______________________________________________
> freebsd-hackers at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe at freebsd.org"
>