FreeBSD 6.3/7.1 and Linux disk performance test

Pieter de Goeje pieter at degoeje.nl
Sat Feb 7 05:37:30 PST 2009


On Saturday 07 February 2009 09:00:37 Omer Faruk Sen wrote:
> Hi,
>
> I have installed a new server to test performance results. BIOS and
> RAID BIOS is the latest in this server ( Raid controller is a Intel
> SRCSASBB8I )
>
>
>
> # dmesg |grep -i mfi
> mfi0: <LSI MegaSAS 1078> port 0x2000-0x20ff mem
> 0xb8b00000-0xb8b3ffff,0xb8b40000-0xb8b7ffff irq 16 at device 0.0 on
> pci10
> mfi0: Megaraid SAS driver Ver 3.00
> mfi0: 1870 (287314652s/0x0020/info) - Shutdown command received from host
> mfi0: 1871 (boot + 3s/0x0020/info) - Firmware initialization started
> (PCI ID 0060/1000/1013/8086)
> mfi0: 1872 (boot + 3s/0x0020/info) - Firmware version 1.20.72-0562
> mfi0: 1873 (boot + 3s/0x0020/info) - Board Revision
> mfi0: 1874 (boot + 15s/0x0002/info) - Inserted: PD 08(e0xff/s8)
> mfi0: 1875 (boot + 15s/0x0002/info) - Inserted: PD 08(e0xff/s8) Info:
> enclPd=ffff, scsiType=0, portMap=00,
> sasAddr=5000c5000bcb90b1,0000000000000000
> mfi0: 1876 (boot + 15s/0x0002/info) - Inserted: PD 09(e0xff/s9)
> mfi0: 1877 (boot + 15s/0x0002/info) - Inserted: PD 09(e0xff/s9) Info:
> enclPd=ffff, scsiType=0, portMap=01,
> sasAddr=5000c5000bcb962d,0000000000000000
> mfi0: 1878 (boot + 15s/0x0002/info) - Inserted: PD 0a(e0xff/s10)
> mfi0: 1879 (boot + 15s/0x0002/info) - Inserted: PD 0a(e0xff/s10) Info:
> enclPd=ffff, scsiType=0, portMap=02,
> sasAddr=5000c5000bcb8f8d,0000000000000000
> mfi0: [ITHREAD]
> mfi0: 1880 (287314711s/0x0020/info) - Time established as 02/07/09
> 9:38:31; (52 seconds since power on)
> mfid0: <MFI Logical Disk> on mfi0
> mfid0: 556928MB (1140588544 sectors) RAID volume ''
>
>
> I have installed RHEL 5.3 x64 on this server and here is simple dd
> performance test:
>
> Last login: Fri Feb  6 12:15:32 2009 from 10.0.0.51
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 1.8473 seconds, 443 MB/s
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 2.05412 seconds, 399 MB/s
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 1.94096 seconds, 422 MB/s
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 2.1092 seconds, 388 MB/s
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 1.84414 seconds, 444 MB/s
> [root at localhost ~]# dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes (819 MB) copied, 2.09829 seconds, 390 MB/s
>
> And after that I have tested with FreeBSD 6.3 and 7.1:
>
> 6.3:
>
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.534685 secs (231760388 bytes/sec)
> # cd /var/tmp/
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.051668 secs (268443342 bytes/sec)
> # cd /
> # cd /var/tmp/
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.001266 secs (272951481 bytes/sec)
> # cd /home/
> cd: can't cd to /home/
> # cd /usr/
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.678156 secs (222720290 bytes/sec)
> # cd /boot
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> cd /usr/100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 2.985471 secs (274395563 bytes/sec)
> # local
> # pwd
> /usr/local
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.501986 secs (233924406 bytes/sec)
>
> 7.1:
>
> Filesystem       Size    Used   Avail Capacity  Mounted on
> /dev/mfid0s1a    9.7G    2.6G    6.3G    29%    /
> devfs            1.0K    1.0K      0B   100%    /dev
> /dev/mfid0s1d    478G    4.0K    440G     0%    /opt
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.441476 secs (238037393 bytes/sec)
> # cd /
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.132512 secs (261515371 bytes/sec)
> # cd /usr/local/
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.296514 secs (248504951 bytes/sec)
>
> # cd /usr
> # dd if=/dev/zero of=bigfile bs=8192 count=100000
> 100000+0 records in
> 100000+0 records out
> 819200000 bytes transferred in 3.069655 secs (266870386 bytes/sec)
>
> as you can see there is a big difference in just simple dd test. Is
> there additional steps that I can follow to increase performance? By
> the way If I use write-thru cache on this raid card FreeBSD 6.3 and
> FreeBSD 7.1 only gives 13MB/s~ which is very very bad. Linux gives a
> slight decrease on dd test with write-thru cache but freebsd goes from
> 230-270 MB/s to only 13 MB/s

I noticed lately that the blocksize and fragment size can make a huge 
difference on filesystem performance when dealing with big files. Performance 
went up from 130MB/s to 200MB/s (2 disk raid0), after I added the following 
parameters to newfs(8): -b 65536 -f 8192. This increases the block and 
fragment size by a factor of 4 over the the defaults.

Also, you use dd with a small blocksize. Try 64k or 128k instead.

- Pieter de Goeje


More information about the freebsd-hackers mailing list