Initial NFS Test: Linux vs FreeBSD (769% slower)

[Rick Macklem]

Marc G. Fournier wrote:
> On 2013-04-26, at 16:56 , Rick Macklem < rmacklem at uoguelph.ca > wrote:
> 
> 
> If you didn't unmount/remount between writing jboss to the server and
> timing the startup of it, please try it again after doing a
> dismount/mount.
> (Doing the dismount/mount on the Linux client resulted in the same #
> of
> reads as FreeBSD for a quick test I did, instead of none without the
> dismount/remount.)
> 
> 
> 'k, this one was tried on Friday, and even a full server reboot didn't
> make any difference in performance, whether the first run or
> subsequent ones … its just plain fast …
> 
> 
> 
> 
> 
> 
> A few other things to do:
> - Time multiple startups after doing a mount, to see if it only the
> first one that is slow.
> 
> 
> Tried … all are equally slow … best time so far has been ~230s … yup,
> after several start ups, its pretty consistently around the 240s mark
> …
> 
> 
> 
> 
> - Capture the RPC counts for both clients by doing "nfsstat -c" before
> and after the startup.
> 
> 
> FreeBSD:
> 
> 
> Before:
> 
> 
> 
> 
> 
> 
> Client Info:
> Rpc Counts:
> Getattr Setattr Lookup Readlink Read Write Create Remove
> 2745853 821481 973901 18 2230947 2098303 160726 4954
> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
> 1862 0 0 14724 950 16272 0 329756
> Mknod Fsstat Fsinfo PathConf Commit
> 12 30873 5 0 0
> Rpc Info:
> TimedOut Invalid X Replies Retries Requests
> 0 0 0 0 9430761
> Cache Info:
> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses
> 26322016 2745853 20537972 973869 2373488 2225801 2618800 2097243
> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses
> 1262 18 46863 15678 29941 0 22513185 329759
> 
> After:
> 
> 
> 
> 
> 
> 
> Client Info:
> Rpc Counts:
> Getattr Setattr Lookup Readlink Read Write Create Remove
> 2745919 821481 973912 18 2230947 2098303 160726 4954
> Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access
> 1862 0 0 14724 950 16272 0 329767
> Mknod Fsstat Fsinfo PathConf Commit
> 12 30873 5 0 0
> Rpc Info:
> TimedOut Invalid X Replies Retries Requests
> 0 0 0 0 9430849
> Cache Info:
> Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses
> 26323022 2745919 20538207 973880 2374208 2225801 2618800 2097243
> BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses
> 1262 18 46863 15678 29941 0 22513489 329770
> 
> 
> 
> 
> Okay, if I'm reading the above right … there doesn't look to be *alot*
> of difference between the Before n After … it doesn't look like its
> downing a whole lot of NFS ops … am I reading wrong?
> 
Yep. Taking the difference between before and after I see:
Getattr 66, Lookup 11, Access 11 for a total of 88 RPCs

That is "no load" on an NFS server.

> 
> 
> 
> If the above doesn't give you any good hints w.r.t. why it is slow,
> you can capture packets during the startup for both clients and look
> at them in wireshark, to try and figure out what the difference
> between
> the Linux and FreeBSD clients are for this case.
> 
> 
> If the above nfsstat output indicates this is warranted, then please
> provide more information on what I should run …
Well, you can capture packets for the above. Run on the client:
# tcpdump -s 0 -w startup.pcap host <server-hostname>
- start it before doing the startup and kill it after the startup
  has been completed.
You can then look at startup.pcap in wireshark. What would I be looking
for?
Strange stuff like TCP retries/reconnects or large time delays between
requests and replies. If all you see are 88 RPCs with replies that
arrive shortly after they are sent, then I have no idea why it is slow,
but it isn't the NFS protocol. (Maybe "jboss" doesn't like some attribute
returned when stat'ng a file and then does "who knows what that takes a
longgg time?".)

If you want, you can email me startup.pcap as an attachment and I'll take
a look, but wireshark is pretty good at spotting TCP retransmits, etc.

The only other thing I can suggest is taking the "soft,intr" options off
your mount and see if that has any effect. Maybe some syscall is returning
EINTR and confusing jboss?

Good luck with it, rick