Low nfs write throughput

Fri Nov 18 17:23:49 UTC 2011

Bane Ivosev wrote:
> and if you use zfs also try this
> 
> zfs set sync=disabled
> 
I know diddly about zfs, but I believe some others have improved
zfs performance for NFS writing by moving the ZIL log to a dedicated
device, sometimes an SSD. Apparently (again I'm not knowledgible) you
do have to be careful what SSD you use and how full you make it, if you
want good write performance on the SSD.

I should also note that use of these options (vfs.nfsrv.async=1 and the
above for zfs) is risky in the sense that recently written data can be
lost when a server crashes/reboots because the NFS clients don't know
to hold onto the data and re-write it after a server crash/reboot.

rick
ps: NFS write performance has been an issue since SUN released their
    first implementation of it in 1985. The "big" server vendors typically
    solve the problem with lots of non-volatile RAM in the server boxes.
    (This solution requires server code that specifically knows how to
     use this non-volatile RAM. Such code is not in the FreeBSD servers.)

> On 11/18/11 04:10, Daryl Sayers wrote:
> > Can anyone suggest why I am getting poor write performance from my
> > nfs setup.
> > I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus
> > mother boards,
> > 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives
> > with
> > onboard Gb network cards connected to an idle network. The results
> > below show
> > that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using
> > nfs. It
> > improves if I use async but a smbfs mount still beats it. I am using
> > the same
> > file, source and destinations for all tests. I have tried alternate
> > Network
> > cards with no resulting benefit.
> >
> > oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2
> > 1950511+1 records in
> > 1950511+1 records out
> > 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec)
> > 1950477+74 records in
> > 1950511+1 records out
> > 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec)
> > (98Mb/s)
> >
> >
> > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs
> > /mnt
> > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
> > 7619+1 records in
> > 7619+1 records out
> > 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec)
> > (15Mb/s)
> >
> >
> > oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async
> > gemini:/dsk/ufs /mnt
> > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
> > 7619+1 records in
> > 7619+1 records out
> > 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec)
> > (19Mb/s)
> >
> >
> > oguido# mount -t smbfs //gemini/ufs /mnt
> > oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k
> > 7619+1 records in
> > 7619+1 records out
> > 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec)
> > (33Mb/s)
> >
> > Looking at a systat -v on the destination I see that the nfs test
> > does not
> > exceed 16KB/t with 100% busy where the other tests reach up to
> > 128KB/t.
> > For the record I get reads of 22Mb/s without and 77Mb/s with async
> > turned on
> > for the nfs mount.
> >
> >
> > A copy of dmesg:
> > ================
> >
> > Copyright (c) 1992-2011 The FreeBSD Project.
> > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993,
> > 1994
> >         The Regents of the University of California. All rights
> >         reserved.
> > FreeBSD is a registered trademark of The FreeBSD Foundation.
> > FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011
> >     root at fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386
> > Timecounter "i8254" frequency 1193182 Hz quality 0
> > CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz
> > 686-class CPU)
> >   Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping =
> >   11
> >   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> >   Features2=0xe3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
> >   AMD Features=0x20100000<NX,LM>
> >   AMD Features2=0x1<LAHF>
> >   TSC: P-state invariant
> > real memory = 4294967296 (4096 MB)
> > avail memory = 3141234688 (2995 MB)
> > ACPI APIC Table: <MSTEST OEMAPIC >
> > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> > FreeBSD/SMP: 1 package(s) x 2 core(s)
> >  cpu0 (BSP): APIC ID: 0
> >  cpu1 (AP): APIC ID: 1
> > ioapic0 <Version 2.0> irqs 0-23 on motherboard
> > kbd1 at kbdmux0
> > cryptosoft0: <software crypto> on motherboard
> > acpi0: <MSTEST TESTONLY> on motherboard
> > acpi0: [ITHREAD]
> > acpi0: Power Button (fixed)
> > acpi0: reservation of 0, a0000 (3) failed
> > acpi0: reservation of 100000, bff00000 (3) failed
> > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
> > cpu0: <ACPI CPU> on acpi0
> > ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be
> > 0xB1 (20101013/tbutils-354)
> > cpu1: <ACPI CPU> on acpi0
> > pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> > pci0: <ACPI PCI bus> on pcib0
> > pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
> > pci1: <ACPI PCI bus> on pcib1
> > mpt0: <LSILogic SAS/SATA Adapter> port 0x7800-0x78ff mem
> > 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on
> > pci1
> > mpt0: [ITHREAD]
> > mpt0: MPI Version=1.5.18.0
> > mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 )
> > mpt0: 0 Active Volumes (2 Max)
> > mpt0: 0 Hidden Drive Members (14 Max)
> > uhci0: <Intel 82801H (ICH8) USB controller USB-D> port 0xdc00-0xdc1f
> > irq 16 at device 26.0 on pci0
> > uhci0: [ITHREAD]
> > uhci0: LegSup = 0x2f00
> > usbus0: <Intel 82801H (ICH8) USB controller USB-D> on uhci0
> > uhci1: <Intel 82801H (ICH8) USB controller USB-E> port 0xe000-0xe01f
> > irq 17 at device 26.1 on pci0
> > uhci1: [ITHREAD]
> > uhci1: LegSup = 0x2f00
> > usbus1: <Intel 82801H (ICH8) USB controller USB-E> on uhci1
> > ehci0: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> mem
> > 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0
> > ehci0: [ITHREAD]
> > usbus2: EHCI version 1.0
> > usbus2: <Intel 82801H (ICH8) USB 2.0 controller USB2-B> on ehci0
> > pci0: <multimedia, HDA> at device 27.0 (no driver attached)
> > pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
> > pci5: <ACPI PCI bus> on pcib2
> > atapci0: <SiI 3132 SATA300 controller> port 0xac00-0xac7f mem
> > 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on
> > pci5
> > atapci0: [ITHREAD]
> > ata2: <ATA channel 0> on atapci0
> > ata2: [ITHREAD]
> > ata3: <ATA channel 1> on atapci0
> > ata3: [ITHREAD]
> > pcib3: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
> > pci4: <ACPI PCI bus> on pcib3
> > em0: <Intel(R) PRO/1000 Network Connection 7.2.3> port 0x9c00-0x9c1f
> > mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0
> > on pci4
> > em0: Using an MSI interrupt
> > em0: [FILTER]
> > em0: Ethernet address: 00:1b:21:04:ac:11
> > pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
> > pci3: <ACPI PCI bus> on pcib4
> > age0: <Attansic Technology Corp, L1 Gigabit Ethernet> mem
> > 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3
> > age0: 1280 Tx FIFO, 2364 Rx FIFO
> > age0: Using 1 MSI messages.
> > miibus0: <MII bus> on age0
> > atphy0: <Atheros F1 10/100/1000 PHY> PHY 0 on miibus0
> > atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX,
> > 1000baseT-FDX, 1000baseT-FDX-master, auto
> > age0: Ethernet address: 00:1a:92:d2:de:cc
> > age0: [FILTER]
> > pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
> > pci2: <ACPI PCI bus> on pcib5
> > atapci1: <JMicron JMB363 UDMA133 controller> port
> > 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f
> > mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2
> > atapci1: [ITHREAD]
> > atapci2: <AHCI SATA controller> on atapci1
> > atapci2: [ITHREAD]
> > atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported
> > ata4: <ATA channel 0> on atapci2
> > ata4: [ITHREAD]
> > ata5: <ATA channel 1> on atapci2
> > ata5: [ITHREAD]
> > ata6: <ATA channel 0> on atapci1
> > ata6: [ITHREAD]
> > uhci2: <Intel 82801H (ICH8) USB controller USB-A> port 0xd480-0xd49f
> > irq 23 at device 29.0 on pci0
> > uhci2: [ITHREAD]
> > uhci2: LegSup = 0x2f00
> > usbus3: <Intel 82801H (ICH8) USB controller USB-A> on uhci2
> > uhci3: <Intel 82801H (ICH8) USB controller USB-B> port 0xd800-0xd81f
> > irq 19 at device 29.1 on pci0
> > uhci3: [ITHREAD]
> > uhci3: LegSup = 0x2f00
> > usbus4: <Intel 82801H (ICH8) USB controller USB-B> on uhci3
> > uhci4: <Intel 82801H (ICH8) USB controller USB-C> port 0xd880-0xd89f
> > irq 18 at device 29.2 on pci0
> > uhci4: [ITHREAD]
> > uhci4: LegSup = 0x2f00
> > usbus5: <Intel 82801H (ICH8) USB controller USB-C> on uhci4
> > ehci1: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> mem
> > 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0
> > ehci1: [ITHREAD]
> > usbus6: EHCI version 1.0
> > usbus6: <Intel 82801H (ICH8) USB 2.0 controller USB2-A> on ehci1
> > pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> > pci6: <ACPI PCI bus> on pcib6
> > vgapci0: <VGA-compatible display> mem
> > 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff
> > irq 22 at device 1.0 on pci6
> > fwohci0: <VIA Fire II (VT6306)> port 0xbc00-0xbc7f mem
> > 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6
> > fwohci0: [ITHREAD]
> > fwohci0: OHCI version 1.10 (ROM=1)
> > fwohci0: No. of Isochronous channels is 4.
> > fwohci0: EUI64 00:11:d8:00:01:39:52:d3
> > fwohci0: Phy 1394a available S400, 2 ports.
> > fwohci0: Link S400, max_rec 2048 bytes.
> > firewire0: <IEEE1394(FireWire) bus> on fwohci0
> > dcons_crom0: <dcons configuration ROM> on firewire0
> > dcons_crom0: bus_addr 0x1494000
> > fwe0: <Ethernet over FireWire> on firewire0
> > if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3
> > fwe0: Ethernet address: 02:11:d8:39:52:d3
> > fwip0: <IP over FireWire> on firewire0
> > fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000,
> > S400, maxrec 2048
> > fwohci0: Initiate bus reset
> > fwohci0: fwohci_intr_core: BUS reset
> > fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1,
> > CYCLEMASTER mode
> > isab0: <PCI-ISA bridge> at device 31.0 on pci0
> > isa0: <ISA bus> on isab0
> > atapci3: <Intel ICH8 SATA300 controller> port
> > 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f
> > irq 19 at device 31.2 on pci0
> > atapci3: [ITHREAD]
> > ata7: <ATA channel 0> on atapci3
> > ata7: [ITHREAD]
> > ata8: <ATA channel 1> on atapci3
> > ata8: [ITHREAD]
> > pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
> > atapci4: <Intel ICH8 SATA300 controller> port
> > 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f
> > irq 19 at device 31.5 on pci0
> > atapci4: [ITHREAD]
> > ata9: <ATA channel 0> on atapci4
> > ata9: [ITHREAD]
> > ata10: <ATA channel 1> on atapci4
> > ata10: [ITHREAD]
> > acpi_button0: <Power Button> on acpi0
> > atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
> > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on
> > acpi0
> > uart0: [FILTER]
> > fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6
> > drq 2 on acpi0
> > fdc0: [FILTER]
> > acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff
> > on acpi0
> > Timecounter "HPET" frequency 14318180 Hz quality 900
> > ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on
> > acpi0
> > ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
> > ppc0: FIFO with 16/16/9 bytes threshold
> > ppc0: [ITHREAD]
> > ppbus0: <Parallel port bus> on ppc0
> > plip0: <PLIP network interface> on ppbus0
> > plip0: [ITHREAD]
> > lpt0: <Printer> on ppbus0
> > lpt0: [ITHREAD]
> > lpt0: Interrupt-driven port
> > ppi0: <Parallel I/O> on ppbus0
> > pmtimer0 on isa0
> > orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff pnpid ORM0000 on
> > isa0
> > sc0: <System console> at flags 0x100 on isa0
> > sc0: VGA <16 virtual consoles, flags=0x300>
> > vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
> > isa0
> > ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0
> > ata0: [ITHREAD]
> > ata1 at port 0x170-0x177,0x376 irq 15 on isa0
> > ata1: [ITHREAD]
> > atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
> > atkbd0: <AT Keyboard> irq 1 on atkbdc0
> > kbd0 at atkbd0
> > atkbd0: [GIANT-LOCKED]
> > atkbd0: [ITHREAD]
> > est0: <Enhanced SpeedStep Frequency Control> on cpu0
> > p4tcc0: <CPU Frequency Thermal Control> on cpu0
> > est1: <Enhanced SpeedStep Frequency Control> on cpu1
> > p4tcc1: <CPU Frequency Thermal Control> on cpu1
> > Timecounters tick every 1.000 msec
> > IPsec: Initialized Security Association Processing.
> > firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me)
> > firewire0: bus manager 0
> > usbus0: 12Mbps Full Speed USB v1.0
> > usbus1: 12Mbps Full Speed USB v1.0
> > usbus2: 480Mbps High Speed USB v2.0
> > usbus3: 12Mbps Full Speed USB v1.0
> > usbus4: 12Mbps Full Speed USB v1.0
> > usbus5: 12Mbps Full Speed USB v1.0
> > usbus6: 480Mbps High Speed USB v2.0
> > ugen0.1: <Intel> at usbus0
> > uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
> > usbus0
> > ugen1.1: <Intel> at usbus1
> > uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
> > usbus1
> > ugen2.1: <Intel> at usbus2
> > uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on
> > usbus2
> > ugen3.1: <Intel> at usbus3
> > uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
> > usbus3
> > ugen4.1: <Intel> at usbus4
> > uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
> > usbus4
> > ugen5.1: <Intel> at usbus5
> > uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on
> > usbus5
> > ugen6.1: <Intel> at usbus6
> > uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on
> > usbus6
> > acd0: DVDR <HL-DT-ST DVDRAM GH22NS50/TN02> at ata7-slave UDMA100
> > SATA 1.5Gb/s
> > uhub0: 2 ports with 2 removable, self powered
> > uhub1: 2 ports with 2 removable, self powered
> > uhub3: 2 ports with 2 removable, self powered
> > uhub4: 2 ports with 2 removable, self powered
> > uhub5: 2 ports with 2 removable, self powered
> > uhub2: 4 ports with 4 removable, self powered
> > uhub6: 6 ports with 6 removable, self powered
> > da1 at mpt0 bus 0 scbus0 target 1 lun 0
> > da1: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device
> > da1: 300.000MB/s transfers
> > da1: Command Queueing enabled
> > da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)
> > da2 at mpt0 bus 0 scbus0 target 2 lun 0
> > da2: <SEAGATE ST9146852SS 0005> Fixed Direct Access SCSI-5 device
> > da2: 300.000MB/s transfers
> > da2: Command Queueing enabled
> > da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C)
> > da0 at mpt0 bus 0 scbus0 target 0 lun 0
> > da0: <FUJITSU MBC2073RC 0104> Fixed Direct Access SCSI-5 device
> > da0: 300.000MB/s transfers
> > da0: Command Queueing enabled
> > da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C)
> > SMP: AP CPU #1 Launched!
> > ugen1.2: <NOVATEK> at usbus1
> > ugen3.2: <Microsoft> at usbus3
> > ukbd1: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on
> > usbus1
> > ums0: <Microsoft Microsoft Basic Optical Mouse v2.0, class 0/0, rev
> > 1.10/1.99, addr 2> on usbus3
> > ums0: 3 buttons and [XYZ] coordinates ID=0
> > kbd2 at ukbd1
> > uhid0: <NOVATEK USB Keyboard, class 0/0, rev 1.10/1.12, addr 2> on
> > usbus1
> > Trying to mount root from ufs:/dev/da0s1a
> >
> >
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
> "freebsd-stable-unsubscribe at freebsd.org"