Initial NFS Test: Linux vs FreeBSD (769% slower)

[Marc G. Fournier]

On 2013-04-30, at 05:29 , Rick Macklem <rmacklem at uoguelph.ca> wrote:

> Marc G. Fournier wrote:
>> If you want, you can email me startup.pcap as an attachment and I'll
>> take
>> a look, but wireshark is pretty good at spotting TCP retransmits, etc.
>> 
>> 
>> 'k, at 4.x Gig in size, doubt your mail server will handle me sending
>> this to you :) Even compressed, it was going over 400M … I can if you
>> want it though …
>> 
>> 
>> Baring that, if you want to give me pointers as to what I should be
>> looking for? I have Wireshark installed / and the startup.pcap data
>> loaded … the only thing that is jumping out at me is a bunch of lines
>> where 'Length' is 32982 while most are 218 … highlighted in 'black
>> background, red font' … for example:
>> 
> The big one is a write RPC and it will be a little more than 32768, if
> you've set wsize=32768.
> 
> This can't be a capture for the "nfsstat" numbers you emailed the last time.
> (For one thing, that one didn't have any write RPCs counted.) Try and get
> a capture for the case where there are few NFS RPCs. (Did you capture for
> the first time doing the startup after doing a mount vs do an "nfsstat"
> for a subsequent startup?) Or, is this client doing something else on the
> network while the startup is happening?
> 
> I may take a look at it, to see if I can spot anything weird, but a capture
> when it only does 88 RPCs is going to be much easier to look at.

'k, here is what I just ran now … igb1 is the private IP network where the NFS mount is running from … host 192.168.1.5 is the host IP of the server that I'm using for testing … and the results again have WRITE calls in it … 

note that when I run the tcpdump command, I'm in /tmp on the local file system, so I'm not writing that log to the NFS server … and there is nothing else running on the NFS mount, since all tha tis on it is /usr/local/jboss… … the rest of /usr/local is on the local drives also … the WRITEs are to /usr/local/jboss/standalone/logs … so not sure why RPC Counts for Writes is showing 0 change …

I'm building a new bz2 file right now, but it looks pretty similar to the one I already sent you a URL for ...

root at server04:/tmp # nfsstat -c; tcpdump -i igb1 -s 0 -w startup.pcap host 192.168.1.5
Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
  2746536    821481    974263        18   2230948   2098303    160726      4954
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
     1862         0         0     14724       950     16272         0    330261
    Mknod    Fsstat    Fsinfo  PathConf    Commit
       12     30926         6         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0   9432366
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
 26331524   2746535  20540473    974231   2380075   2225802   2618800   2097243
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
     1262        18     46863     15678     29941         0  22516391    330264
tcpdump: listening on igb1, link-type EN10MB (Ethernet), capture size 65535 bytes
^C5454815 packets captured
5458144 packets received by filter
0 packets dropped by kernel


root at server04:/tmp # nfsstat -c
Client Info:
Rpc Counts:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
  2746603    821481    974276        18   2230953   2098303    160726      4954
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
     1862         0         0     14724       950     16272         0    330275
    Mknod    Fsstat    Fsinfo  PathConf    Commit
       12     30926         6         0         0
Rpc Info:
 TimedOut   Invalid X Replies   Retries  Requests
        0         0         0         0   9432465
Cache Info:
Attr Hits    Misses Lkup Hits    Misses BioR Hits    Misses BioW Hits    Misses
 26332545   2746602  20540712    974244   2380801   2225807   2618800   2097243
BioRLHits    Misses BioD Hits    Misses DirE Hits    Misses Accs Hits    Misses
     1262        18     46863     15678     29941         0  22516700    330278





> 
> rick
> 
>> 
>> 
>> 
>> 
>> i am doing a bzip2 compressed file right now, and will make it
>> available via HTTP, if you are interested …
>> 
>> 
>> 
>> 
>> 
>> 
>> The only other thing I can suggest is taking the "soft,intr" options
>> off
>> your mount and see if that has any effect. Maybe some syscall is
>> returning
>> EINTR and confusing jboss?
>> 
>> 
>> Tried removing soft,intr … no change, still around 240s …