FreeBSD 9.1 vs CentOS 6.3
Paul Pathiakis
pathiaki2 at yahoo.com
Sat Mar 23 00:37:25 UTC 2013
Hi,
There are several things about this that are highly suspect.
First, wipe out the hardware RAID. The processor doing RAID computation is, probably, MUCH slower than a core on the CPU. Even if it's RAID-1 (Simple Mirror) this RAID card is performing tasks that is does not need to do including replicating writes to two targets from the controller or checking it's cache, battery, etc. If it's possible to disable the onboard cache, do it.
(This controller seems to use the mfi driver so that's probably a re-badged LSI with an LSI200x chipset. (Lately, I have not been too thrilled with the mfi driver performance. The mpt seems much more stable and robust - my opinion not bashing to whoever wrote it.)
Take the card to full JBOD mode before testing your "throughput".
Also, make sure that all on-disk caches are "off". You do not want to get into a double-buffer flush corruption issue.
I see you are using two SSDs as ARC2 caches.... Quite frankly, junk them as well, for now. You may want to deal with increasing throughput once you find out what the issue is with the base throughput.
Create the appropriate pools/mirrors as necessary, all on simple JBOD.
From there, run a simple test to see what your throughput is on single spindle. (I'm talking SIMPLE. dd if=/dev/zero of=/<path>/zeroes.out bs=128k count=<some ridiculous number like 100000 or more> - This should be over 100 MB/sec throughput. (with those drives it may be 150 MB/sec or more)
After that, check the throughput of the card/channel. Write a very simple script script that performs the same throughput test on half the available spindles (something as ludicrous as the same dd command to half your spindles throwing it in the background as each kicks off - Then, do it for all of your drives)
This is an imperfect test but it's down and dirty. You can use the performance/benchmark tools later.
Let's just remember that the throughput of 6 Gb/sec is gigaBITS not gigaBYTES. There's a factor of 8 difference.... or 750 MB/sec... theoretical.
You should exceed channel speed with just 5 of these drives/spindles... use 6 to be sure (I would think you won't see aggregate throughput beyond 725 MB/sec.... If any tool is showing you that on any OS with this configuration, they aren't doing proper calculations and/or they are doing MB vs MiB.
If you can't get up to that speed, you may have a driver issue or channel issue.
(I was getting as high as 700 MB/sec with 6 SATA drives doing it this way.... No SSD, ASUS MB 990FX with 1100T 6 core proc, 16 GB RAM... )
P.
________________________________
From: Davide D'Amico <davide.damico at contactlab.com>
To: Евгений Хоркин <khorkin.eugeniy at gmail.com>
Cc: freebsd-performance at freebsd.org
Sent: Friday, March 22, 2013 12:15 PM
Subject: Re: FreeBSD 9.1 vs CentOS 6.3
Well, the I/O isn't the bottleneck (if you follow the link to freebsd-fs, you'll see iostats values) but it seems something related to cpu/scheduler or something else.
Now I am trying vmware 5 on the same server and a vm with centos6: the vm outperforms freebsd with every concurrency from 1 to 48 threads in sysbench oltp tests.
Thanks,
d.
"Евгений Хоркин" <khorkin.eugeniy at gmail.com> ha scritto:
>Hi Davide!
>
>Are you sure that disk is the bottleneck in your test?
>Does systat -vm 1 show 100% busy for disk ?
>
>Evgeny.
>
>
>2013/3/22 Davide D'Amico <davide.damico at contactlab.com>
>
>> Hi, I'm doing performance tests on a DELL R720, follows dmesg:
>>
>> Copyright (c) 1992-2012 The FreeBSD Project.
>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
>1994
>> The Regents of the University of California. All rights
>reserved.
>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>> FreeBSD 9.1-RELEASE #0: Tue Mar 19 10:24:21 CET 2013
>> root:/usr/obj/usr/src/sys/R720 amd64
>> CPU: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2500.05-MHz K8-class
>CPU)
>> Origin = "GenuineIntel" Id = 0x206d7 Family = 6 Model = 2d
>Stepping =
>> 7
>> Features=0xbfebfbff<FPU,VME,**DE,PSE,TSC,MSR,PAE,MCE,CX8,**
>> APIC,SEP,MTRR,PGE,MCA,CMOV,**PAT,PSE36,CLFLUSH,DTS,ACPI,**
>> MMX,FXSR,SSE,SSE2,SS,HTT,TM,**PBE>
>> Features2=0x1fbee3ff<SSE3,**PCLMULQDQ,DTES64,MON,DS_CPL,**
>> VMX,SMX,EST,TM2,SSSE3,CX16,**xTPR,PDCM,PCID,DCA,SSE4.1,**
>> SSE4.2,x2APIC,POPCNT,TSCDLT,**AESNI,XSAVE,OSXSAVE,AVX>
>> AMD Features=0x2c100800<SYSCALL,**NX,Page1GB,RDTSCP,LM>
>> AMD Features2=0x1<LAHF>
>> TSC: P-state invariant, performance statistics
>> real memory = 34359738368 (32768 MB)
>> avail memory = 33027436544 (31497 MB)
>> Event timer "LAPIC" quality 600
>> ACPI APIC Table: <DELL PE_SC3 >
>> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
>> FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 SMT threads
>> cpu0 (BSP): APIC ID: 0
>> cpu1 (AP): APIC ID: 1
>> cpu2 (AP): APIC ID: 10
>> cpu3 (AP): APIC ID: 11
>> cpu4 (AP): APIC ID: 32
>> cpu5 (AP): APIC ID: 33
>> cpu6 (AP): APIC ID: 42
>> cpu7 (AP): APIC ID: 43
>> ioapic1: Changing APIC ID to 1
>> ioapic2: Changing APIC ID to 2
>> ioapic0 <Version 2.0> irqs 0-23 on motherboard
>> ioapic1 <Version 2.0> irqs 32-55 on motherboard
>> ioapic2 <Version 2.0> irqs 64-87 on motherboard
>> kbd1 at kbdmux0
>> ctl: CAM Target Layer loaded
>> acpi0: <DELL PE_SC3> on motherboard
>> acpi0: Power Button (fixed)
>> acpi0: reservation of 0, 2000 (3) failed
>> cpu0: <ACPI CPU> on acpi0
>> cpu1: <ACPI CPU> on acpi0
>> cpu2: <ACPI CPU> on acpi0
>> cpu3: <ACPI CPU> on acpi0
>> cpu4: <ACPI CPU> on acpi0
>> cpu5: <ACPI CPU> on acpi0
>> cpu6: <ACPI CPU> on acpi0
>> cpu7: <ACPI CPU> on acpi0
>> atrtc0: <AT realtime clock> port 0x70-0x7f irq 8 on acpi0
>> Event timer "RTC" frequency 32768 Hz quality 0
>> attimer0: <AT timer> port 0x40-0x5f irq 0 on acpi0
>> Timecounter "i8254" frequency 1193182 Hz quality 0
>> Event timer "i8254" frequency 1193182 Hz quality 100
>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
>> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
>> hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
>acpi0
>> Timecounter "HPET" frequency 14318180 Hz quality 950
>> Event timer "HPET" frequency 14318180 Hz quality 550
>> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
>> pci0: <ACPI PCI bus> on pcib0
>> pcib1: <ACPI PCI-PCI bridge> irq 53 at device 1.0 on pci0
>> pci1: <ACPI PCI bus> on pcib1
>> igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
>> 0xfc80-0xfc9f mem 0xdcb00000-0xdcbfffff,**0xdcff0000-0xdcff3fff irq
>38 at
>> device 0.0 on pci1
>> igb0: Using MSIX interrupts with 9 vectors
>> igb0: Ethernet address: bc:30:5b:ef:27:b8
>> [...]
>> igb1: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
>> 0xfca0-0xfcbf mem 0xdcc00000-0xdccfffff,**0xdcff4000-0xdcff7fff irq
>37 at
>> device 0.1 on pci1
>> igb1: Using MSIX interrupts with 9 vectors
>> igb1: Ethernet address: bc:30:5b:ef:27:b9
>> [...]
>> igb2: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
>> 0xfcc0-0xfcdf mem 0xdcd00000-0xdcdfffff,**0xdcff8000-0xdcffbfff irq
>36 at
>> device 0.2 on pci1
>> igb2: Using MSIX interrupts with 9 vectors
>> igb2: Ethernet address: bc:30:5b:ef:27:ba
>> [...]
>> igb3: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
>> 0xfce0-0xfcff mem 0xdce00000-0xdcefffff,**0xdcffc000-0xdcffffff irq
>34 at
>> device 0.3 on pci1
>> igb3: Using MSIX interrupts with 9 vectors
>> igb3: Ethernet address: bc:30:5b:ef:27:bb
>> [...]
>> pcib2: <ACPI PCI-PCI bridge> irq 53 at device 2.0 on pci0
>> pci3: <ACPI PCI bus> on pcib2
>> pcib3: <ACPI PCI-PCI bridge> irq 53 at device 2.2 on pci0
>> pci2: <ACPI PCI bus> on pcib3
>> mfi0: <Dell PERC H710P Mini (monolithics)> port 0xec00-0xecff mem
>> 0xddffc000-0xddffffff,**0xddf80000-0xddfbffff irq 42 at device 0.0 on
>pci2
>> mfi0: Using MSI
>> mfi0: Megaraid SAS driver Ver 4.23
>> mfi0: MaxCmd = 3f0 MaxSgl = 46 state = b73c03f0
>> pcib4: <ACPI PCI-PCI bridge> irq 53 at device 3.0 on pci0
>> pci4: <ACPI PCI bus> on pcib4
>> pcib5: <ACPI PCI-PCI bridge> irq 53 at device 3.2 on pci0
>> pci5: <ACPI PCI bus> on pcib5
>> pci0: <base peripheral> at device 5.0 (no driver attached)
>> pci0: <base peripheral> at device 5.2 (no driver attached)
>> pcib6: <PCI-PCI bridge> irq 16 at device 17.0 on pci0
>> pci6: <PCI bus> on pcib6
>> pci0: <simple comms> at device 22.0 (no driver attached)
>> pci0: <simple comms> at device 22.1 (no driver attached)
>> ehci0: <EHCI (generic) USB 2.0 controller> mem 0xdf8fe000-0xdf8fe3ff
>irq
>> 23 at device 26.0 on pci0
>> usbus0: EHCI version 1.0
>> usbus0 on ehci0
>> pcib7: <ACPI PCI-PCI bridge> at device 28.0 on pci0
>> device_attach: pcib7 attach returned 6
>> pcib7: <ACPI PCI-PCI bridge> irq 19 at device 28.7 on pci0
>> pci8: <ACPI PCI bus> on pcib7
>> pcib8: <PCI-PCI bridge> at device 0.0 on pci8
>> pci9: <PCI bus> on pcib8
>> pcib9: <PCI-PCI bridge> at device 0.0 on pci9
>> pci10: <PCI bus> on pcib9
>> pcib10: <PCI-PCI bridge> at device 0.0 on pci10
>> pci11: <PCI bus> on pcib10
>> vgapci0: <VGA-compatible display> mem 0xd8000000-0xd8ffffff,**
>> 0xdeffc000-0xdeffffff,**0xde000000-0xde7fffff irq 19 at device 0.0 on
>> pci11
>> pcib11: <PCI-PCI bridge> at device 1.0 on pci9
>> pci12: <PCI bus> on pcib11
>> ehci1: <EHCI (generic) USB 2.0 controller> mem 0xdf8ff000-0xdf8ff3ff
>irq
>> 22 at device 29.0 on pci0
>> usbus1: EHCI version 1.0
>> usbus1 on ehci1
>> pcib12: <PCI-PCI bridge> at device 30.0 on pci0
>> pci13: <PCI bus> on pcib12
>> isab0: <PCI-ISA bridge> at device 31.0 on pci0
>> isa0: <ISA bus> on isab0
>> pcib13: <ACPI Host-PCI bridge> on acpi0
>> pci64: <ACPI PCI bus> on pcib13
>> pcib14: <ACPI PCI-PCI bridge> irq 85 at device 1.0 on pci64
>> pci65: <ACPI PCI bus> on pcib14
>> pcib15: <ACPI PCI-PCI bridge> irq 85 at device 2.0 on pci64
>> pci66: <ACPI PCI bus> on pcib15
>> pcib16: <ACPI PCI-PCI bridge> irq 85 at device 3.0 on pci64
>> pci67: <ACPI PCI bus> on pcib16
>> pcib17: <ACPI PCI-PCI bridge> irq 85 at device 3.2 on pci64
>> pci68: <ACPI PCI bus> on pcib17
>> pci64: <base peripheral> at device 5.0 (no driver attached)
>> pci64: <base peripheral> at device 5.2 (no driver attached)
>> pcib18: <ACPI Host-PCI bridge> on acpi0
>> pci63: <ACPI PCI bus> on pcib18
>> pci63: <base peripheral> at device 8.0 (no driver attached)
>> [...]
>> pci63: <dasp> at device 14.1 (no driver attached)
>> [...]
>> pci63: <dasp> at device 19.1 (no driver attached)
>> pci63: <dasp> at device 19.4 (no driver attached)
>> pci63: <dasp> at device 19.5 (no driver attached)
>> pci63: <base peripheral> at device 19.6 (no driver attached)
>> pcib19: <ACPI Host-PCI bridge> on acpi0
>> pci127: <ACPI PCI bus> on pcib19
>> [...]
>> pci127: <dasp> at device 14.1 (no driver attached)
>> [...]
>> pci127: <dasp> at device 19.1 (no driver attached)
>> pci127: <dasp> at device 19.4 (no driver attached)
>> pci127: <dasp> at device 19.5 (no driver attached)
>> pci127: <base peripheral> at device 19.6 (no driver attached)
>> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
>> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on
>acpi0
>> orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xec000-**0xeffff on
>isa0
>> sc0: <System console> at flags 0x100 on isa0
>> sc0: VGA <16 virtual consoles, flags=0x300>
>> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
>isa0
>> est0: <Enhanced SpeedStep Frequency Control> on cpu0
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 226d00001e00
>> device_attach: est0 attach returned 6
>> p4tcc0: <CPU Frequency Thermal Control> on cpu0
>> est1: <Enhanced SpeedStep Frequency Control> on cpu1
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 226d00001e00
>> device_attach: est1 attach returned 6
>> p4tcc1: <CPU Frequency Thermal Control> on cpu1
>> est2: <Enhanced SpeedStep Frequency Control> on cpu2
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 226d00001e00
>> device_attach: est2 attach returned 6
>> p4tcc2: <CPU Frequency Thermal Control> on cpu2
>> est3: <Enhanced SpeedStep Frequency Control> on cpu3
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 224400001e00
>> device_attach: est3 attach returned 6
>> p4tcc3: <CPU Frequency Thermal Control> on cpu3
>> est4: <Enhanced SpeedStep Frequency Control> on cpu4
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 226d00001e00
>> device_attach: est4 attach returned 6
>> p4tcc4: <CPU Frequency Thermal Control> on cpu4
>> est5: <Enhanced SpeedStep Frequency Control> on cpu5
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 224400001e00
>> device_attach: est5 attach returned 6
>> p4tcc5: <CPU Frequency Thermal Control> on cpu5
>> est6: <Enhanced SpeedStep Frequency Control> on cpu6
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 224400001e00
>> device_attach: est6 attach returned 6
>> p4tcc6: <CPU Frequency Thermal Control> on cpu6
>> est7: <Enhanced SpeedStep Frequency Control> on cpu7
>> est: CPU supports Enhanced Speedstep, but is not recognized.
>> est: cpu_vendor GenuineIntel, msr 224400001e00
>> device_attach: est7 attach returned 6
>> p4tcc7: <CPU Frequency Thermal Control> on cpu7
>> mfi0: 6040 (417120269s/0x0020/info) - Shutdown command received from
>host
>> mfi0: 6041 (boot + 4s/0x0020/info) - Firmware initialization started
>(PCI
>> ID 005b/1000/1f34/1028)
>> mfi0: 6042 (boot + 4s/0x0020/info) - Firmware version 3.130.05-1796
>> mfi0: 6043 (boot + 5s/0x0008/info) - Battery Present
>> mfi0: 6044 (boot + 5s/0x0020/info) - Package version 21.1.0-0007
>> mfi0: 6045 (boot + 5s/0x0020/info) - Board Revision A01
>> mfi0: 6046 (boot + 6s/0x0008/info) - Battery temperature is normal
>> mfi0: 6047 (boot + 6s/0x0008/info) - Current capacity of the battery
>is
>> above threshold
>> mfi0: 6048 (boot + 19s/0x0004/info) - Enclosure PD 20(c None/p1)
>> communication restored
>> mfi0: 6049 (boot + 19s/0x0002/info) - Inserted: Encl PD 20
>> mfi0: 6050 (boot + 19s/0x0002/info) - Inserted: PD 20(c None/p1)
>Info:
>> enclPd=20, scsiType=d, portMap=00, sasAddr=5e4ae020a26cb500,**
>> 0000000000000000
>> mfi0: 6051 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0)
>> mfi0: 6052 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info:
>> enclPd=20, scsiType=0, portMap=04, sasAddr=5000c50054982339,**
>> 0000000000000000
>> mfi0: 6053 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1)
>> mfi0: 6054 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info:
>> enclPd=20, scsiType=0, portMap=05, sasAddr=5000c50054986c91,**
>> 0000000000000000
>> mfi0: 6055 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2)
>> mfi0: 6056 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info:
>> enclPd=20, scsiType=0, portMap=06, sasAddr=5000c50053da8f75,**
>> 0000000000000000
>> mfi0: 6057 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3)
>> mfi0: 6058 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) Info:
>> enclPd=20, scsiType=0, portMap=07, sasAddr=5000c50053c57eb5,**
>> 0000000000000000
>> mfi0: 6059 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4)
>> mfi0: 6060 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4) Info:
>> enclPd=20, scsiType=0, portMap=00, sasAddr=5000c50053d364c5,**
>> 0000000000000000
>> mfi0: 6061 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5)
>> mfi0: 6062 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5) Info:
>> enclPd=20, scsiType=0, portMap=03, sasAddr=5000c50053da2b41,**
>> 0000000000000000
>> mfi0: 6063 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6)
>> mfi0: 6064 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) Info:
>> enclPd=20, scsiType=0, portMap=02, sasAddr=4433221102000000,**
>> 0000000000000000
>> mfi0: 6065 (boot + 19s/0x0002/WARN) - PD 06(e0x20/s6) is not a
>certified
>> drive
>> mfi0: 6066 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7)
>> mfi0: 6067 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7) Info:
>> enclPd=20, scsiType=0, portMap=01, sasAddr=4433221103000000,**
>> 0000000000000000
>> mfi0: 6068 (boot + 19s/0x0002/WARN) - PD 07(e0x20/s7) is not a
>certified
>> drive
>> mfi0: 6069 (417120312s/0x0020/info) - Time established as 03/20/13
>> 18:45:12; (32 seconds since power on)
>> mfi0: 6070 (417120445s/0x0020/info) - Host driver is loaded and
>operational
>> Timecounters tick every 1.000 msec
>> usbus0: 480Mbps High Speed USB v2.0
>> usbus1: 480Mbps High Speed USB v2.0
>> mfid0 on mfi0
>> mfid0: 285568MB (584843264 sectors) RAID volume 'BASE' is optimal
>> mfid1 on mfi0
>> mfid1: 121472MB (248774656 sectors) RAID volume 'L2ARC0' is optimal
>> mfid2 on mfi0
>> mfid2: 121472MB (248774656 sectors) RAID volume 'ZIL0' is optimal
>> mfid3 on mfi0
>> mfid3: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE1' is
>optimal
>> mfid4 on mfi0
>> mfid4: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE2' is
>optimal
>> ugen0.1: <Intel> at usbus0
>> uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on
>usbus0
>> ugen1.1: <Intel> at usbus1
>> uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on
>usbus1
>> uhub0: 2 ports with 2 removable, self powered
>> uhub1: 2 ports with 2 removable, self powered
>> ugen0.2: <vendor 0x8087> at usbus0
>> uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr
>2> on
>> usbus0
>> ugen1.2: <vendor 0x8087> at usbus1
>> uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr
>2> on
>> usbus1
>> uhub2: 6 ports with 6 removable, self powered
>> uhub3: 8 ports with 8 removable, self powered
>> ugen0.3: <no manufacturer> at usbus0
>> uhub4: <no manufacturer Gadget USB HUB, class 9/0, rev 2.00/0.00,
>addr 3>
>> on usbus0
>> SMP: AP CPU #1 Launched!
>> SMP: AP CPU #7 Launched!
>> SMP: AP CPU #3 Launched!
>> SMP: AP CPU #6 Launched!
>> SMP: AP CPU #2 Launched!
>> SMP: AP CPU #5 Launched!
>> SMP: AP CPU #4 Launched!
>> Timecounter "TSC-low" frequency 9765827 Hz quality 1000
>> Root mount waiting for: usbus0
>> uhub4: 6 ports with 6 removable, self powered
>> ugen0.4: <Avocent> at usbus0
>> ukbd0: <Keyboard> on usbus0
>> kbd0 at ukbd0
>> Root mount waiting for: usbus0
>> Root mount waiting for: usbus0
>> ugen0.5: <Avocent> at usbus0
>> umass0: <SCSI Transparent Interface 0> on usbus0
>> Trying to mount root from ufs:/dev/mfid0p3 [rw]...
>> cd0 at umass-sim0 bus 0 scbus1 target 0 lun 0
>> cd0: <iDRAC Virtual CD 0329> Removable CD-ROM SCSI-0 device
>> cd0: 40.000MB/s transfers
>> cd0: Attempt to query device size failed: NOT READY, Medium not
>present
>> da0 at umass-sim0 bus 0 scbus1 target 0 lun 1
>> da0: <iDRAC Virtual Floppy 0329> Removable Direct Access SCSI-0
>device
>> da0: 40.000MB/s transfers
>> da0: Attempt to query device size failed: NOT READY, Medium not
>present
>> ZFS filesystem version 5
>> ZFS storage pool version 28
>> ums0: <Mouse> on usbus0
>> ums0: 3 buttons and [Z] coordinates ID=0
>> ums1: <Mouse REL> on usbus0
>> ums1: 3 buttons and [XYZ] coordinates ID=0
>> igb0: link state changed to UP
>>
>> With this loader.conf:
>> vm.kmem_size_max="64G"
>> vm.kmem_size="48G"
>> vfs.zfs.arc_min="4096M"
>> vfs.zfs.arc_max="15872M"
>> vfs.zfs.txg.synctime="1"
>> vfs.zfs.txg.timeout="5"
>> vfs.zfs.write_limit_override="**0"
>> vfs.zfs.txg.write_limit_**override="0"
>> vfs.zfs.zil_disable="0"
>>
>> And with this sysctl.conf:
>> security.bsd.see_other_uids=0
>> net.inet.tcp.blackhole=2
>> net.inet.udp.blackhole=1
>> net.inet.tcp.sendspace=65536
>> net.inet.tcp.recvspace=65536
>> net.link.ether.inet.log_arp_**wrong_iface=0
>> kern.ipc.somaxconn=32768
>> kern.threads.max_threads_per_**proc=16384
>> kern.maxfiles=262144
>> kern.maxfilesperproc=131072
>> kern.ipc.nmbclusters=65536
>> kern.corefile="/var/coredumps/**%U.%N.%P.core"
>>
>> vfs.zfs.txg.timeout=5
>> vfs.zfs.l2arc_headroom=16
>> vfs.zfs.l2arc_write_max=**26843546
>> vfs.zfs.l2arc_write_boost=**26843546
>> vfs.zfs.vdev.min_pending=4
>> vfs.zfs.vdev.max_pending=64
>> vfs.zfs.l2arc_noprefetch=1
>> vfs.zfs.dedup.prefetch=0
>> vfs.zfs.prefetch_disable=1
>>
>> I will use this server as a mysql-5.6 dbserver so I have a root
>partition
>> using a hw raid1 and a /DATAZFS partition, follows configuration:
>>
>> root# mfiutil show drives
>> mfi0 Physical Drives:
>> 0 ( 279G) ONLINE <SEAGATE ST3300657SS ES65> SAS E1:S0
>> 1 ( 279G) ONLINE <SEAGATE ST3300657SS ES65> SAS E1:S1
>> 2 ( 558G) ONLINE <SEAGATE ST3600057SS ES65> SAS E1:S2
>> 3 ( 558G) ONLINE <SEAGATE ST3600057SS ES65> SAS E1:S3
>> 4 ( 558G) ONLINE <SEAGATE ST3600057SS ES65> SAS E1:S4
>> 5 ( 558G) ONLINE <SEAGATE ST3600057SS ES65> SAS E1:S5
>> 6 ( 119G) ONLINE <OCZ-VERTEX4 1.4> SATA E1:S6
>> 7 ( 119G) ONLINE <OCZ-VERTEX4 1.4> SATA E1:S7
>> root# mfiutil show volumes
>> mfi0 Volumes:
>> Id Size Level Stripe State Cache Name
>> mfid0 ( 278G) RAID-1 64k OPTIMAL Disabled <BASE>
>> mfid1 ( 118G) RAID-0 64k OPTIMAL Disabled <L2ARC0>
>> mfid2 ( 118G) RAID-0 64k OPTIMAL Disabled <ZIL0>
>> mfid3 ( 1116G) RAID-1 64k OPTIMAL Disabled <DATA>
>>
>> I have a /DATAZFS partition, too:
>> root# zfs get all DATAZFS
>> NAME PROPERTY VALUE SOURCE
>> DATAZFS type filesystem -
>> DATAZFS creation Thu Mar 21 10:45 2013 -
>> DATAZFS used 54.2G -
>> DATAZFS available 1.02T -
>> DATAZFS referenced 54.2G -
>> DATAZFS compressratio 1.00x -
>> DATAZFS mounted yes -
>> DATAZFS quota none default
>> DATAZFS reservation none default
>> DATAZFS recordsize 16K local
>> DATAZFS mountpoint /DATAZFS default
>> DATAZFS sharenfs off default
>> DATAZFS checksum on default
>> DATAZFS compression off default
>> DATAZFS atime off local
>> DATAZFS devices on default
>> DATAZFS exec on default
>> DATAZFS setuid on default
>> DATAZFS readonly off default
>> DATAZFS jailed off default
>> DATAZFS snapdir hidden default
>> DATAZFS aclmode discard default
>> DATAZFS aclinherit restricted default
>> DATAZFS canmount on default
>> DATAZFS xattr off temporary
>> DATAZFS copies 1 default
>> DATAZFS version 5 -
>> DATAZFS utf8only off -
>> DATAZFS normalization none -
>> DATAZFS casesensitivity sensitive -
>> DATAZFS vscan off default
>> DATAZFS nbmand off default
>> DATAZFS sharesmb off default
>> DATAZFS refquota none default
>> DATAZFS refreservation none default
>> DATAZFS primarycache all default
>> DATAZFS secondarycache all default
>> DATAZFS usedbysnapshots 0 -
>> DATAZFS usedbydataset 54.2G -
>> DATAZFS usedbychildren 1.41M -
>> DATAZFS usedbyrefreservation 0 -
>> DATAZFS logbias latency default
>> DATAZFS dedup off local
>> DATAZFS mlslabel -
>> DATAZFS sync standard default
>> DATAZFS refcompressratio 1.00x -
>> DATAZFS written 54.2G -
>>
>> root# zpool status -v
>> pool: DATAZFS
>> state: ONLINE
>> scan: none requested
>> config:
>>
>> NAME STATE READ WRITE CKSUM
>> DATAZFS ONLINE 0 0 0
>> mfid3 ONLINE 0 0 0
>> logs
>> mfid2 ONLINE 0 0 0
>> cache
>> mfid1 ONLINE 0 0 0
>>
>> errors: No known data errors
>>
>> root# mfiutil show firmware
>> mfi0 Firmware Package Version: 21.1.0-0007
>> mfi0 Firmware Images:
>> Name Version Date Time Status
>> BIOS 5.31.01_4.12.05.00_0x05110000 6/20/2012
>> 6/20/2012
>> active
>> CTLR 4.00-0014 Aug 04 2011 12:49:17 active
>> PCLI 05.00-03:#%00008 Feb 17 2011 14:03:12 active
>> APP 3.130.05-1796 Jul 27 2012 15:58:31 active
>> NVDT 2.1108.03-0094 Jun 22 2012 12:40:55 active
>> BTBL 2.03.00.00-0004 Jul 27 2012 16:39:29 active
>> BOOT 06.253.57.219 9/9/2010 15:32:25 active
>> root#
>>
>> Then, I installed CentOS on the same hardware, same version of mysql
>> (5.6.10-enterprise and 5.6.10-GA), and I tried an OLTP test (included
>in
>> sysbench-0.5) executing 2000000 queries on 10000000 rows, with these
>> results (transactions per seconds):
>>
>> BSD 9.1 amd64 ZFS (without tweaks):
>> 1 thread: 2780
>> 2 threads: 4371
>> 4 threads: 6876
>> 8 threads: 8202
>> 12 threads: 8077
>> 16 threads: 7780
>> 24 threads: 7563
>> 32 threads: 7632
>> 48 threads: 7960
>> 64 threads: 8062
>>
>> BSD 9.1 amd64 ZFS (with tweaks):
>> 1 thread: 2589
>> 2 threads: 4679
>> 4 threads: 6438
>> 8 threads: 7073
>> 12 threads: 7121
>> 16 threads: 7227
>> 24 threads: 7132
>> 32 threads: 7273
>> 48 threads: 7623
>> 64 threads: 7672
>>
>> CentOS 6.3 (ext4, without tweaks):
>> 1 thread: 4163
>> 2 threads: 7653
>> 4 threads: 10905
>> 8 threads: 12511
>> 12 threads: 13556
>> 16 threads: 14832
>> 24 threads: 16270
>> 32 threads: 16733
>> 48 threads: 16925
>> 64 threads: 16895
>>
>>
>> Well, any of you could ask "why zfs, please try ufs that is better":
>you
>> are right, but changing ZFS with UFS doesn't change so much:
>>
>http://lists.freebsd.org/**pipermail/freebsd-fs/2013-**March/016747.html<http://lists.freebsd.org/pipermail/freebsd-fs/2013-March/016747.html>
>>
>> So I hope I'm missing something because the performances of a
>standard
>> CentOS installation on the same hardware performs much better than a
>9.1
>> amd64 installation (I'm using the standard ULE scheduler included in
>the
>> 9.1 release).
>>
>> Could anyone help me?
>>
>> Thanks,
>> d.
>> ______________________________**_________________
>> freebsd-performance at freebsd.**org
><freebsd-performance at freebsd.org>mailing list
>>
>http://lists.freebsd.org/**mailman/listinfo/freebsd-**performance<http://lists.freebsd.org/mailman/listinfo/freebsd-performance>
>> To unsubscribe, send any mail to "freebsd-performance-**
>> unsubscribe at freebsd.org
><freebsd-performance-unsubscribe at freebsd.org>"
>>
--
d.
_______________________________________________
freebsd-performance at freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe at freebsd.org"
More information about the freebsd-performance
mailing list