Initial NFS Test: Linux vs FreeBSD (769% slower)

[Rick Macklem]

Marc G. Fournier wrote:
> On 2013-04-30, at 05:29 , Rick Macklem < rmacklem at uoguelph.ca > wrote:
> 
> 
> Marc G. Fournier wrote:
> 
> 
> If you want, you can email me startup.pcap as an attachment and I'll
> take
> a look, but wireshark is pretty good at spotting TCP retransmits, etc.
> 
> 
> 'k, at 4.x Gig in size, doubt your mail server will handle me sending
> this to you :) Even compressed, it was going over 400M … I can if you
> want it though …
> 
> 
> Baring that, if you want to give me pointers as to what I should be
> looking for? I have Wireshark installed / and the startup.pcap data
> loaded … the only thing that is jumping out at me is a bunch of lines
> where 'Length' is 32982 while most are 218 … highlighted in 'black
> background, red font' … for example:
> 
> The big one is a write RPC and it will be a little more than 32768, if
> you've set wsize=32768.
> 
> This can't be a capture for the "nfsstat" numbers you emailed the last
> time.
> (For one thing, that one didn't have any write RPCs counted.) Try and
> get
> a capture for the case where there are few NFS RPCs. (Did you capture
> for
> the first time doing the startup after doing a mount vs do an
> "nfsstat"
> for a subsequent startup?) Or, is this client doing something else on
> the
> network while the startup is happening?
> 
> I may take a look at it, to see if I can spot anything weird, but a
> capture
> when it only does 88 RPCs is going to be much easier to look at.
> 
> 
> 
> 'k, here is what I just ran now … igb1 is the private IP network where
> the NFS mount is running from … host 192.168.1.5 is the host IP of the
> server that I'm using for testing … and the results again have WRITE
> calls in it …
> 
> 
> note that when I run the tcpdump command, I'm in /tmp on the local
> file system, so I'm not writing that log to the NFS server … and there
> is nothing else running on the NFS mount, since all tha tis on it is
> /usr/local/jboss… … the rest of /usr/local is on the local drives also
> … the WRITEs are to /usr/local/jboss/standalone/logs … so not sure why
> RPC Counts for Writes is showing 0 change …
> 
> 
> I'm building a new bz2 file right now, but it looks pretty similar to
> the one I already sent you a URL for ...
> 
> 
> 
> root at server04:/tmp # nfsstat -c; tcpdump -i igb1 -s 0 -w startup.pcap
> host 192.168.1.5
> Client Info:
> Rpc Counts:
> Getattr Setattr Lookup Readlink Read Write Create Remove
> 2746536 821481 974263 18 2230948 2098303 160726 4954
> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
> 1862 0 0 14724 950 16272 0 330261
> Mknod Fsstat Fsinfo PathConf Commit
> 12 30926 6 0 0
> Rpc Info:
> TimedOut Invalid X Replies Retries Requests
> 0 0 0 0 9432366
> Cache Info:
> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses
> 26331524 2746535 20540473 974231 2380075 2225802 2618800 2097243
> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses
> 1262 18 46863 15678 29941 0 22516391 330264
> tcpdump: listening on igb1, link-type EN10MB (Ethernet), capture size
> 65535 bytes
> ^C5454815 packets captured
> 5458144 packets received by filter
> 0 packets dropped by kernel
> 
> 
> 
> 
> root at server04:/tmp # nfsstat -c
> Client Info:
> Rpc Counts:
> Getattr Setattr Lookup Readlink Read Write Create Remove
> 2746603 821481 974276 18 2230953 2098303 160726 4954
> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
> 1862 0 0 14724 950 16272 0 330275
> Mknod Fsstat Fsinfo PathConf Commit
> 12 30926 6 0 0
> Rpc Info:
> TimedOut Invalid X Replies Retries Requests
> 0 0 0 0 9432465
> Cache Info:
> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses
> 26332545 2746602 20540712 974244 2380801 2225807 2618800 2097243
> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses
> 1262 18 46863 15678 29941 0 22516700 330278
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> rick
> 
> 
> 
> 
> 
> 
> 
> i am doing a bzip2 compressed file right now, and will make it
> available via HTTP, if you are interested …
> 
> 
> 
> 
> 
> 
> The only other thing I can suggest is taking the "soft,intr" options
> off
> your mount and see if that has any effect. Maybe some syscall is
> returning
> EINTR and confusing jboss?
> 
> 
> Tried removing soft,intr … no change, still around 240s …
Well, I looked at the packet capture and, for some reason, it
repeatedly does a write of 1 byte to a file, followed by a read of
that file, over and over and ... again. I have no idea why the
app. does that.

I also don't know why the "nfsstat -c" is bogus. Are you using "-t oldnfs"
for the mount for these runs by any chance? If so, you need to use
"nfsstat -o -c".

rick
ps: To be honest, I doubt that anything can be done to speed this up,
    except to move it to a local disk.