Low nfs write throughput
Bane Ivosev
bane.ivosev at pmf.uns.ac.rs
Fri Nov 18 08:17:45 UTC 2011
and if you use zfs also try this
zfs set sync=disabled
On 11/18/11 04:10, Daryl Sayers wrote:
> Can anyone suggest why I am getting poor write performance from my nfs setup.
> I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards,
> 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with
> onboard Gb network cards connected to an idle network. The results below show
> that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It
> improves if I use async but a smbfs mount still beats it. I am using the same
> file, source and destinations for all tests. I have tried alternate Network
> cards with no resulting benefit.
>
> oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2
> 1950511+1 records in
> 1950511+1 records out
> 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec)
> 1950477+74 records in
> 1950511+1 records out
> 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s)
>
>
> oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt
> oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
> 7619+1 records in
> 7619+1 records out
> 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s)
>
>
> oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt
> oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
> 7619+1 records in
> 7619+1 records out
> 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s)
>
>
> oguido# mount -t smbfs //gemini/ufs /mnt
> oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
> 7619+1 records in
> 7619+1 records out
> 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s)
>
> Looking at a systat -v on the destination I see that the nfs test does not
> exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t.
> For the record I get reads of 22Mb/s without and 77Mb/s with async turned on
> for the nfs mount.
>
>
> A copy of dmesg:
> ================
>
> Copyright (c) 1992-2011 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011
> root at fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz 686-class CPU)
> Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11
> Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> Features2=0xe3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
> AMD Features=0x20100000<NX,LM>
> AMD Features2=0x1<LAHF>
> TSC: P-state invariant
> real memory = 4294967296 (4096 MB)
> avail memory = 3141234688 (2995 MB)
> ACPI APIC Table: <MSTEST OEMAPIC >
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> FreeBSD/SMP: 1 package(s) x 2 core(s)
> cpu0 (BSP): APIC ID: 0
> cpu1 (AP): APIC ID: 1
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> kbd1 at kbdmux0
> cryptosoft0: <software crypto> on motherboard
> acpi0: <MSTEST TESTONLY> on motherboard
> acpi0: [ITHREAD]
> acpi0: Power Button (fixed)
> acpi0: reservation of 0, a0000 (3) failed
> acpi0: reservation of 100000, bff00000 (3) failed
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
> cpu0: <ACPI CPU> on acpi0
> ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be 0xB1 (20101013/tbutils-354)
> cpu1: <ACPI CPU> on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> mpt0: <LSILogic SAS/SATA Adapter> port 0x7800-0x78ff mem 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on pci1
> mpt0: [ITHREAD]
> mpt0: MPI Version=1.5.18.0
> mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
> mpt0: 0 Active Volumes (2 Max)
> mpt0: 0 Hidden Drive Members (14 Max)
> uhci0: <Intel 82801H (ICH8) USB controller USB-D> port 0xdc00-0xdc1f irq 16 at device 26.0 on pci0
> uhci0: [ITHREAD]
> uhci0: LegSup = 0x2f00
> usbus0: <Intel 82801H (ICH8) USB controller USB-D> on uhci0
> uhci1: <Intel 82801H (ICH8) USB controller USB-E> port 0xe000-0xe01f irq 17 at device 26.1 on pci0
> uhci1: [ITHREAD]
> uhci1: LegSup = 0x2f00
> usbus1: <Intel 82801H (ICH8) USB controller USB-E> on uhci1
> ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> mem 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0
> ehci0: [ITHREAD]
> usbus2: EHCI version 1.0
> usbus2: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> on ehci0
> pci0: <multimedia, HDA> at device 27.0 (no driver attached)
> pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
> pci5: <ACPI PCI bus> on pcib2
> atapci0: <SiI 3132 SATA300 controller> port 0xac00-0xac7f mem 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on pci5
> atapci0: [ITHREAD]
> ata2: <ATA channel 0> on atapci0
> ata2: [ITHREAD]
> ata3: <ATA channel 1> on atapci0
> ata3: [ITHREAD]
> pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
> pci4: <ACPI PCI bus> on pcib3
> em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x9c00-0x9c1f mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 on pci4
> em0: Using an MSI interrupt
> em0: [FILTER]
> em0: Ethernet address: 00:1b:21:04:ac:11
> pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
> pci3: <ACPI PCI bus> on pcib4
> age0: <Attansic Technology Corp, L1 Gigabit Ethernet> mem 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3
> age0: 1280 Tx FIFO, 2364 Rx FIFO
> age0: Using 1 MSI messages.
> miibus0: <MII bus> on age0
> atphy0: <Atheros F1 10/100/1000 PHY> PHY 0 on miibus0
> atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
> age0: Ethernet address: 00:1a:92:d2:de:cc
> age0: [FILTER]
> pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
> pci2: <ACPI PCI bus> on pcib5
> atapci1: <JMicron JMB363 UDMA133 controller> port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2
> atapci1: [ITHREAD]
> atapci2: <AHCI SATA controller> on atapci1
> atapci2: [ITHREAD]
> atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported
> ata4: <ATA channel 0> on atapci2
> ata4: [ITHREAD]
> ata5: <ATA channel 1> on atapci2
> ata5: [ITHREAD]
> ata6: <ATA channel 0> on atapci1
> ata6: [ITHREAD]
> uhci2: <Intel 82801H (ICH8) USB controller USB-A> port 0xd480-0xd49f irq 23 at device 29.0 on pci0
> uhci2: [ITHREAD]
> uhci2: LegSup = 0x2f00
> usbus3: <Intel 82801H (ICH8) USB controller USB-A> on uhci2
> uhci3: <Intel 82801H (ICH8) USB controller USB-B> port 0xd800-0xd81f irq 19 at device 29.1 on pci0
> uhci3: [ITHREAD]
> uhci3: LegSup = 0x2f00
> usbus4: <Intel 82801H (ICH8) USB controller USB-B> on uhci3
> uhci4: <Intel 82801H (ICH8) USB controller USB-C> port 0xd880-0xd89f irq 18 at device 29.2 on pci0
> uhci4: [ITHREAD]
> uhci4: LegSup = 0x2f00
> usbus5: <Intel 82801H (ICH8) USB controller USB-C> on uhci4
> ehci1: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0
> ehci1: [ITHREAD]
> usbus6: EHCI version 1.0
> usbus6: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> on ehci1
> pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> pci6: <ACPI PCI bus> on pcib6
> vgapci0: <VGA-compatible display> mem 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff irq 22 at device 1.0 on pci6
> fwohci0: <VIA Fire II (VT6306)> port 0xbc00-0xbc7f mem 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6
> fwohci0: [ITHREAD]
> fwohci0: OHCI version 1.10 (ROM=1)
> fwohci0: No. of Isochronous channels is 4.
> fwohci0: EUI64 00:11:d8:00:01:39:52:d3
> fwohci0: Phy 1394a available S400, 2 ports.
> fwohci0: Link S400, max_rec 2048 bytes.
> firewire0: <IEEE1394(FireWire) bus> on fwohci0
> dcons_crom0: <dcons configuration ROM> on firewire0
> dcons_crom0: bus_addr 0x1494000
> fwe0: <Ethernet over FireWire> on firewire0
> if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3
> fwe0: Ethernet address: 02:11:d8:39:52:d3
> fwip0: <IP over FireWire> on firewire0
> fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, S400, maxrec 2048
> fwohci0: Initiate bus reset
> fwohci0: fwohci_intr_core: BUS reset
> fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> atapci3: <Intel ICH8 SATA300 controller> port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f irq 19 at device 31.2 on pci0
> atapci3: [ITHREAD]
> ata7: <ATA channel 0> on atapci3
> ata7: [ITHREAD]
> ata8: <ATA channel 1> on atapci3
> ata8: [ITHREAD]
> pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
> atapci4: <Intel ICH8 SATA300 controller> port 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f irq 19 at device 31.5 on pci0
> atapci4: [ITHREAD]
> ata9: <ATA channel 0> on atapci4
> ata9: [ITHREAD]
> ata10: <ATA channel 1> on atapci4
> ata10: [ITHREAD]
> acpi_button0: <Power Button> on acpi0
> atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> uart0: [FILTER]
> fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
> fdc0: [FILTER]
> acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
> Timecounter "HPET" frequency 14318180 Hz quality 900
> ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
> ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
> ppc0: FIFO with 16/16/9 bytes threshold
> ppc0: [ITHREAD]
> ppbus0: <Parallel port bus> on ppc0
> plip0: <PLIP network interface> on ppbus0
> plip0: [ITHREAD]
> lpt0: <Printer> on ppbus0
> lpt0: [ITHREAD]
> lpt0: Interrupt-driven port
> ppi0: <Parallel I/O> on ppbus0
> pmtimer0 on isa0
> orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
> ata0: [ITHREAD]
> ata1 at port 0x170-0x177,0x376 irq 15 on isa0
> ata1: [ITHREAD]
> atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> atkbd0: [ITHREAD]
> est0: <Enhanced SpeedStep Frequency Control> on cpu0
> p4tcc0: <CPU Frequency Thermal Control> on cpu0
> est1: <Enhanced SpeedStep Frequency Control> on cpu1
> p4tcc1: <CPU Frequency Thermal Control> on cpu1
> Timecounters tick every 1.000 msec
> IPsec: Initialized Security Association Processing.
> firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
> firewire0: bus manager 0
> usbus0: 12Mbps Full Speed USB v1.0
> usbus1: 12Mbps Full Speed USB v1.0
> usbus2: 480Mbps High Speed USB v2.0
> usbus3: 12Mbps Full Speed USB v1.0
> usbus4: 12Mbps Full Speed USB v1.0
> usbus5: 12Mbps Full Speed USB v1.0
> usbus6: 480Mbps High Speed USB v2.0
> ugen0.1: <Intel> at usbus0
> uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
> ugen1.1: <Intel> at usbus1
> uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
> ugen2.1: <Intel> at usbus2
> uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
> ugen3.1: <Intel> at usbus3
> uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
> ugen4.1: <Intel> at usbus4
> uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
> ugen5.1: <Intel> at usbus5
> uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
> ugen6.1: <Intel> at usbus6
> uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6
> acd0: DVDR <HL-DT-ST DVDRAM GH22NS50/TN02> at ata7-slave UDMA100 SATA 1.5Gb/s
> uhub0: 2 ports with 2 removable, self powered
> uhub1: 2 ports with 2 removable, self powered
> uhub3: 2 ports with 2 removable, self powered
> uhub4: 2 ports with 2 removable, self powered
> uhub5: 2 ports with 2 removable, self powered
> uhub2: 4 ports with 4 removable, self powered
> uhub6: 6 ports with 6 removable, self powered
> da1 at mpt0 bus 0 scbus0 target 1 lun 0
> da1: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device
> da1: 300.000MB/s transfers
> da1: Command Queueing enabled
> da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)
> da2 at mpt0 bus 0 scbus0 target 2 lun 0
> da2: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device
> da2: 300.000MB/s transfers
> da2: Command Queueing enabled
> da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)
> da0 at mpt0 bus 0 scbus0 target 0 lun 0
> da0: <FUJITSU MBC2073RC 0104> Fixed Direct Access SCSI-5 device
> da0: 300.000MB/s transfers
> da0: Command Queueing enabled
> da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C)
> SMP: AP CPU #1 Launched!
> ugen1.2: <NOVATEK> at usbus1
> ugen3.2: <Microsoft> at usbus3
> ukbd1: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1
> ums0: <Microsoft Microsoft Basic Optical Mouse v2.0, class 0/0, rev 1.10/1.99, addr 2> on usbus3
> ums0: 3 buttons and [XYZ] coordinates ID=0
> kbd2 at ukbd1
> uhid0: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on usbus1
> Trying to mount root from ufs:/dev/da0s1a
>
>
More information about the freebsd-stable
mailing list