NFS Performance issue against NetApp

Marc G. Fournier scrappy at hub.org
Sun May 12 02:27:22 UTC 2013


With

vfs.nfs.noconsist=3 ... 385595ms

nfsstat -z before startup, nfsstat -c after:

Client Info:
Rpc Counts:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
    332594         5     17238         0    224426    231137      3743         1
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
         0         0         0       307         0        71         0      8447
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0       509         0         0         0
Rpc Info:
  TimedOut   Invalid X Replies   Retries  Requests
         0         0         0         0    818479
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
    608296    332596    526200     17245    -95425    224426     13178    231137
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
         0         0      1050        55       502         7    543340      8448


============

vfs.nfs.noconsist=2 ... 392201ms

Client Info:
Rpc Counts:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
    332557         5     17228         0    224421    231131      3743         1
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
         0         0         0       307         0        72         0      8430
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0       502         0         0         0
Rpc Info:
  TimedOut   Invalid X Replies   Retries  Requests
         0         0         0         0    818395
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
    607834    332557    525801     17231    -95401    224421     13178    231131
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
         0         0      1028        56       502         0    542925      8431


============
vfs.nfs.noconsist=0 ... 391622ms


Client Info:
Rpc Counts:
   Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
    236122         5     17221         0    230575    230823      3743         1
    Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
         0         0         0       307         0        71         0      8425
     Mknod    Fsstat    Fsinfo  PathConf    Commit
         0       516         0         0         0
Rpc Info:
  TimedOut   Invalid X Replies   Retries  Requests
         0         0         0         0    727799
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
    711860    236124    526549     17225   -101525    230490     13178    230823
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
         0         0      1057        55       516         0    543709      8425


I checked a second time with nonconsist=0, and the nfsstat -c values seem to come out pretty much the same ...

I'm going to head down to the office and try again with Solaris (I'd have to re-install, since I used that system for the Solaris), and see what nfsstat -c results I get out of that ... will post a followup on this when completed ...



On 2013-05-10 5:32 PM, Rick Macklem wrote:
> Marc G. Fournier wrote:
>> FYI … I just installed Solaris 11 onto the same hardware and ran the
>> same test … so far, I'm seeing:
>>
>> Linux @ ~30s
>> Solaris @ ~44s
>>
>> OpenBSD @ ~200s
>> FreeBSD @ ~240s
>>
>> I've even tried FreeBSD 8.3 just to see if maybe its as 'newish' issue
>> … same as 9.x … I could see Linux 'cutting corners', but
>> Oracle/Solaris too … ?
>>
> The three client implementations (BSD, Linux, Solaris) were developed
> independently and, as such, will all implement somewaht different
> caching algorithms (the RFCs specify what goes on the wire, but say
> little w.r.t. client side caching).
>
> I have a attached a patch that might be useful for determining if
> the client side buffer cache consistency algorithm in FreeBSD is
> causing the slow startup of jboss. Do not run this patch on a
> production system, since it pretty well disables all buffer cache
> coherency (ie. if another client modifies a file, the patched client
> won't notice and will continue to cache stale file data).
>
> If the patch does speed up startup of jboss significantly, you can
> use the sysctl:
>   vfs.nfs.noconsist
> to check for which coherency check is involved by decreasing the
> value for the sysctl by 1 and then trying a startup again. (When
> vfs.nfs.noconsist=0, normal cache coherency will be applied.)
>
> I have no idea if buffer cache coherency is a factor, but trying
> the attached patch might determine if it is.
>
> Note that you have never posted updated "nfsstat -c" values.
> (Remember that what you posted indicated 88 RPCs, which seemed
>   bogus.) Finding out if FreeBSD does a lot more of certain RPCs
> that Linux/Solaris might help isolate what is going on.
>
> rick
>
>> On 2013-05-03, at 04:50 , Mark Felder <feld at feld.me> wrote:
>>
>>> On Thu, 02 May 2013 18:43:17 -0500, Marc G. Fournier
>>> <scrappy at hub.org> wrote:
>>>
>>>> Hadn't thought to do so with Linux, but …
>>>> Linux ……. 20732ms, 20117ms, 20935ms, 20130ms, 20560ms
>>>> FreeBSD .. 28996ms, 24794ms, 24702ms, 23311ms, 24153ms
>>> Please make sure both platforms are using similar atime settings. I
>>> think most distros use ext4 with diratime by default. I'd just do
>>> noatime on both platforms to be safe.
>>> _______________________________________________
>>> freebsd-fs at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>>> To unsubscribe, send any mail to
>>> "freebsd-fs-unsubscribe at freebsd.org"
>> _______________________________________________
>> freebsd-fs at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"



More information about the freebsd-fs mailing list