FreeBSD 6.3/7.1 and Linux disk performance test

Sat Feb 7 00:00:40 PST 2009

Hi,

I have installed a new server to test performance results. BIOS and
RAID BIOS is the latest in this server ( Raid controller is a Intel
SRCSASBB8I )

# dmesg |grep -i mfi
mfi0: <LSI MegaSAS 1078> port 0x2000-0x20ff mem
0xb8b00000-0xb8b3ffff,0xb8b40000-0xb8b7ffff irq 16 at device 0.0 on
pci10
mfi0: Megaraid SAS driver Ver 3.00
mfi0: 1870 (287314652s/0x0020/info) - Shutdown command received from host
mfi0: 1871 (boot + 3s/0x0020/info) - Firmware initialization started
(PCI ID 0060/1000/1013/8086)
mfi0: 1872 (boot + 3s/0x0020/info) - Firmware version 1.20.72-0562
mfi0: 1873 (boot + 3s/0x0020/info) - Board Revision
mfi0: 1874 (boot + 15s/0x0002/info) - Inserted: PD 08(e0xff/s8)
mfi0: 1875 (boot + 15s/0x0002/info) - Inserted: PD 08(e0xff/s8) Info:
enclPd=ffff, scsiType=0, portMap=00,
sasAddr=5000c5000bcb90b1,0000000000000000
mfi0: 1876 (boot + 15s/0x0002/info) - Inserted: PD 09(e0xff/s9)
mfi0: 1877 (boot + 15s/0x0002/info) - Inserted: PD 09(e0xff/s9) Info:
enclPd=ffff, scsiType=0, portMap=01,
sasAddr=5000c5000bcb962d,0000000000000000
mfi0: 1878 (boot + 15s/0x0002/info) - Inserted: PD 0a(e0xff/s10)
mfi0: 1879 (boot + 15s/0x0002/info) - Inserted: PD 0a(e0xff/s10) Info:
enclPd=ffff, scsiType=0, portMap=02,
sasAddr=5000c5000bcb8f8d,0000000000000000
mfi0: [ITHREAD]
mfi0: 1880 (287314711s/0x0020/info) - Time established as 02/07/09
9:38:31; (52 seconds since power on)
mfid0: <MFI Logical Disk> on mfi0
mfid0: 556928MB (1140588544 sectors) RAID volume ''

I have installed RHEL 5.3 x64 on this server and here is simple dd
performance test:

Last login: Fri Feb  6 12:15:32 2009 from 10.0.0.51
[root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes (819 MB) copied, 1.8473 seconds, 443 MB/s
[root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes (819 MB) copied, 2.05412 seconds, 399 MB/s
[root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes (819 MB) copied, 1.94096 seconds, 422 MB/s
[root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes (819 MB) copied, 2.1092 seconds, 388 MB/s
[root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes (819 MB) copied, 1.84414 seconds, 444 MB/s
[root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes (819 MB) copied, 2.09829 seconds, 390 MB/s

And after that I have tested with FreeBSD 6.3 and 7.1:

6.3:

# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes transferred in 3.534685 secs (231760388 bytes/sec)
# cd /var/tmp/
# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes transferred in 3.051668 secs (268443342 bytes/sec)
# cd /
# cd /var/tmp/
# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes transferred in 3.001266 secs (272951481 bytes/sec)
# cd /home/
cd: can't cd to /home/
# cd /usr/
# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes transferred in 3.678156 secs (222720290 bytes/sec)
# cd /boot
# dd if=/dev/zero of=bigfile bs=8192 count=100000
cd /usr/100000+0 records in
100000+0 records out
819200000 bytes transferred in 2.985471 secs (274395563 bytes/sec)
# local
# pwd
/usr/local
# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes transferred in 3.501986 secs (233924406 bytes/sec)

7.1:

Filesystem       Size    Used   Avail Capacity  Mounted on
/dev/mfid0s1a    9.7G    2.6G    6.3G    29%    /
devfs            1.0K    1.0K      0B   100%    /dev
/dev/mfid0s1d    478G    4.0K    440G     0%    /opt
# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes transferred in 3.441476 secs (238037393 bytes/sec)
# cd /
# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes transferred in 3.132512 secs (261515371 bytes/sec)
# cd /usr/local/
# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes transferred in 3.296514 secs (248504951 bytes/sec)

# cd /usr
# dd if=/dev/zero of=bigfile bs=8192 count=100000
100000+0 records in
100000+0 records out
819200000 bytes transferred in 3.069655 secs (266870386 bytes/sec)

as you can see there is a big difference in just simple dd test. Is
there additional steps that I can follow to increase performance? By
the way If I use write-thru cache on this raid card FreeBSD 6.3 and
FreeBSD 7.1 only gives 13MB/s~ which is very very bad. Linux gives a
slight decrease on dd test with write-thru cache but freebsd goes from
230-270 MB/s to only 13 MB/s

Regards.