NFS Performance: Weirder And Weirder

[Mehmet Erol Sanliturk]

On Sat, Mar 16, 2013 at 11:49 AM, Tim Daneliuk <tundra at tundraware.com>wrote:

> This is really weird.  A FreeBSD 9.1 system mounts the following:
>
> /dev/ad4s1a    989M    625M    285M    69%    /
> devfs          1.0k    1.0k      0B   100%    /dev
> /dev/ad4s1d    7.8G      1G    6.1G    14%    /var
> /dev/ad4s1e     48G    9.4G     35G    21%    /usr
> /dev/ad4s1f    390G    127G    231G    35%    /usr1
> /dev/ad6s1d    902G    710G    120G    86%    /usr1/BKU
>
> /usr1/something (under ad4s1f) and /usr1/BKU (all of ad6s1d) are
> exported for NFS mounting on the LAN.  I have tested the
> speeds of these two drives locally doing a 'dd if=/dev/zero ....'.
> Their speeds are quite comparable - around 55-60 MB/s so the
> problem below is not an artifact of a slow drive.
>
> The two mounts are imported like this on a Linux Mint 12 machine:
>
>
>   machine:/usr1/BKU     /BKU     nfs   rw,soft,intr          0  0
>   machine:/usr1/shared  /shared  nfs   rw,soft,intr          0  0
>
> Problem:
>
> When I write files from the LM12 machines to /BKU  the writes are
> 1/10 the speed of when writing to /shared.  Reads are fine in both
> cases, at near native disk speeds being reported.
>
> Someone here suggested I get rid of any symlinks in the mount and I did
> that to no avail.
>
>
> Incidentally, the only reason I just noticed this is that I upgraded the
> NIC on the FreeBSD machine and the switch into which it connects to
> 1000Base
> because the LM12 machine had a built in 1000Base NIC.  I also changed
> the cables on both machines to ensure they were not the problem.   Prior
> to this, I was bandwidth constrained by the 100Base so I never saw NFS
> performance as an issue.  When I upgraded, I expected faster transfers
> and when I didn't get them, I started this whole investigation.
>
> So ... I'm stumped:
>
> - It's not the drive or SATA ports because both drives show comparable
> performance.
> - It's not the cables because I can get great throughput on one of the NFS
> mountpoints.
> - It's neither NIC for the same reason.
>
> Does anyone:
>
> A) Have a clue what might be doing this
> B) Have a suggestion how to track down the problem
>
> Thanks,
>
> --
> ------------------------------**------------------------------**
> ----------------
> Tim Daneliuk     tundra at tundraware.com
> PGP Key:         http://www.tundraware.com/PGP/
>
>

With respect to your mount points : /usr1 is spanning TWO different
partitions :

/dev/ad4s1f    390G    127G    231G    35%    /usr1
/dev/ad6s1d    902G    710G    120G    86%    /usr1/BKU


because /usr1/BKU is a sub-directory of  /usr1 .


If you create a new directory , for example /usr2 , and /usr2/BKU , and
using this new separate directory for sharing , such as :

/dev/ad6s1d    902G    710G    120G    86%    /usr2/BKU

and

  machine:/usr2/BKU     /BKU     nfs   rw,soft,intr          0  0


 will it make difference ?


Mehmet Erol Sanliturk