NFS over LAGG / lacp poor performance
Rick Macklem
rmacklem at uoguelph.ca
Fri Apr 25 11:48:59 UTC 2014
Marek Salwerowicz wrote:
> Hi list,
>
> I have two FreeBSD boxes (both based on SuperMicro X9DRD-7LN4F-JBOD
> motherboard, with 32GB RAM, 1 CPU :Intel(R) Xeon(R) CPU E5-2640 v2)
>
> storage1% uname -a
> FreeBSD storage1 9.1-RELEASE-p10 FreeBSD 9.1-RELEASE-p10 #0: Sun Jan
> 12
> 20:11:23 UTC 2014
> root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
> amd64
>
>
> storage2% uname
> -a
> FreeBSD storage2 10.0-RELEASE-p1 FreeBSD 10.0-RELEASE-p1 #0: Tue Apr
> 8
> 06:45:06 UTC 2014
> root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
> amd64
>
>
>
> They work as NFS storages for VMware ESXi.
>
> On both there are installed 4 igb 1Gbit NICs, with LACP and 2 VLANs
> (vlan900 is for NFS, vlan14 is for management/general purpose):
>
> on storage1:
> igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
> 0xd060-0xd07f mem 0xfbb60000-0xfbb7ffff,0xfbb8c000-0xfbb8ffff irq 42
> at
> device 0.0 on pci5
> igb1: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
> 0xd040-0xd05f mem 0xfbb40000-0xfbb5ffff,0xfbb88000-0xfbb8bfff irq 45
> at
> device 0.1 on pci5
> igb2: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
> 0xd020-0xd03f mem 0xfbb20000-0xfbb3ffff,0xfbb84000-0xfbb87fff irq 44
> at
> device 0.2 on pci5
> igb3: <Intel(R) PRO/1000 Network Connection version - 2.3.4> port
> 0xd000-0xd01f mem 0xfbb00000-0xfbb1ffff,0xfbb80000-0xfbb83fff irq 46
> at
> device 0.3 on pci5
>
> on storage2:
> igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
> 0xd060-0xd07f mem 0xfbb60000-0xfbb7ffff,0xfbb8c000-0xfbb8ffff irq 42
> at
> device 0.0 on pci5
> igb1: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
> 0xd040-0xd05f mem 0xfbb40000-0xfbb5ffff,0xfbb88000-0xfbb8bfff irq 45
> at
> device 0.1 on pci5
> igb2: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
> 0xd020-0xd03f mem 0xfbb20000-0xfbb3ffff,0xfbb84000-0xfbb87fff irq 44
> at
> device 0.2 on pci5
> igb3: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port
> 0xd000-0xd01f mem 0xfbb00000-0xfbb1ffff,0xfbb80000-0xfbb83fff irq 46
> at
> device 0.3 on pci5
>
>
> storage1% ifconfig -a
> igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 9000
>
> options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
> ether 00:25:90:c1:1d:18
> inet6 fe80::225:90ff:fec1:1d18%igb0 prefixlen 64 scopeid 0x1
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 9000
>
> options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
> ether 00:25:90:c1:1d:18
> inet6 fe80::225:90ff:fec1:1d19%igb1 prefixlen 64 scopeid 0x2
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 9000
>
> options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
> ether 00:25:90:c1:1d:18
> inet6 fe80::225:90ff:fec1:1d1a%igb2 prefixlen 64 scopeid 0x3
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 9000
>
> options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
> ether 00:25:90:c1:1d:18
> inet6 fe80::225:90ff:fec1:1d1b%igb3 prefixlen 64 scopeid 0x4
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
> inet 127.0.0.1 netmask 0xff000000
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu 9000
>
> options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
> ether 00:25:90:c1:1d:18
> inet6 fe80::225:90ff:fec1:1d18%lagg0 prefixlen 64 scopeid 0x8
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> laggproto lacp lagghash l2,l3,l4
> laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> vlan14: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu 9000
> options=103<RXCSUM,TXCSUM,TSO4>
> ether 00:25:90:c1:1d:18
> inet 192.168.1.65 netmask 0xffffff00 broadcast 192.168.1.255
> inet6 fe80::225:90ff:fec1:1d18%vlan14 prefixlen 64 scopeid
> 0x9
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> vlan: 14 parent interface: lagg0
> vlan900: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu
> 9000
> options=103<RXCSUM,TXCSUM,TSO4>
> ether 00:25:90:c1:1d:18
> inet 172.25.25.65 netmask 0xffffff00 broadcast 172.25.25.255
> inet6 fe80::225:90ff:fec1:1d18%vlan900 prefixlen 64 scopeid
> 0xa
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> vlan: 900 parent interface: lagg0
>
>
> storage2% ifconfig
> -a
> igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 9000
>
> options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
> ether 00:25:90:ca:3b:e0
> inet6 fe80::225:90ff:feca:3be0%igb0 prefixlen 64 scopeid 0x1
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 9000
>
> options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
> ether 00:25:90:ca:3b:e0
> inet6 fe80::225:90ff:feca:3be1%igb1 prefixlen 64 scopeid 0x2
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> igb2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 9000
>
> options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
> ether 00:25:90:ca:3b:e0
> inet6 fe80::225:90ff:feca:3be2%igb2 prefixlen 64 scopeid 0x3
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> igb3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 9000
>
> options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
> ether 00:25:90:ca:3b:e0
> inet6 fe80::225:90ff:feca:3be3%igb3 prefixlen 64 scopeid 0x4
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
> inet 127.0.0.1 netmask 0xff000000
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu 9000
>
> options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
> ether 00:25:90:ca:3b:e0
> inet6 fe80::225:90ff:feca:3be0%lagg0 prefixlen 64 scopeid 0x6
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> laggproto lacp lagghash l2,l3,l4
> laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
> vlan14: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu 9000
> options=303<RXCSUM,TXCSUM,TSO4,TSO6>
> ether 00:25:90:ca:3b:e0
> inet 192.168.1.72 netmask 0xffffff00 broadcast 192.168.1.255
> inet6 fe80::225:90ff:feca:3be0%vlan14 prefixlen 64 scopeid
> 0x7
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> vlan: 14 parent interface: lagg0
> vlan900: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> mtu
> 9000
> options=303<RXCSUM,TXCSUM,TSO4,TSO6>
> ether 00:25:90:ca:3b:e0
> inet 172.25.25.72 netmask 0xffffff00 broadcast 172.25.25.255
> inet6 fe80::225:90ff:feca:3be0%vlan900 prefixlen 64 scopeid
> 0x8
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: active
> vlan: 900 parent interface: lagg0
>
>
> Both boxes are connected to the same switch (HP 1910-48G)
>
> I need to transfer around 10 TB of data from storage1 to storage2
> I obeserve that during copying, only one NIC (instead of all 4) is
> used.
>
Well, you don't mention what command(s) you are using to transfer the
data, but I would guess you have one serial data transfer for each command.
(Put another way, if you are only running one command to transfer the data,
there will only be one RPC happening at a time and that will only use one
network interface.) I don't know anything about lagg, so I can't comment
related to it, but if there is only one NFS RPC at a time, you'll only
be transferring one message at a time on the wire.)
Adding the mount option "readahead=8" to the machine receiving the data
might help, if the data transfer command is being done there. (ie. The machine
the data is being copied to has the other one NFS mounted and it is where
you are running the data transfer command(s).)
If you can run commands concurrently to transfer parts of the data, that
might help, too.
Good luck with it, rick
> Boxes are not stressed during copying
>
> What's more, apart from having 1 NIC saturated (transfer around 120
> MB/s), I observe transfer rate on level of 70-80 MB/s
>
> I haven't changed any kernel configuration, nor sysctl's - am I
> missing
> sth ?
>
> Regards,
> Marek
>
> --
> Marek Salwerowicz
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to
> "freebsd-net-unsubscribe at freebsd.org"
>
More information about the freebsd-net
mailing list