Some 9.3 NFS testing

[Garrett Wollman]

Just thought I'd share this...

I've been doing some acceptance testing on 9.3 prior to upgrading my
production NFS servers.  My most recent test is running bonnie++ on
192 Ubuntu VMs in parallel, to independent directories in the same
server filesystem.  It hasn't fallen over yet (will probably take
another day or so to complete), and peaked at about 220k ops/s (but
this was NFSv4 so there's no FHA and it takes at least two ops for
every v3 RPC[1]).  bonnie++ is running with -D (O_DIRECT), but I'm
actually just using it as a load generator -- I don't care about the
output.

I have this system configured for a maximum of 64 nfsd threads, and
the test load has had it pegged for the past eight hours.  Right now
all of the load generators are doing the "small file" part of
bonnie++, so there's not a lot of activity but there are a lot of
synchronous operations; it's been doing 60k ops/s for the past five
hours.  Load average maxed out at about 24 early on in the test, and
has settled around 16-20 for this part of the test.

Here's what nfsstat -se has to say (note: not reset for this round of
testing):

Server Info:
  Getattr   Setattr    Lookup  Readlink      Read     Write    Create    Remove
1566655064 230074779 162549702         0 471311053 1466525587 149235773 115496945
   Rename      Link   Symlink     Mkdir     Rmdir   Readdir  RdirPlus    Access
      125         0         0       245       116   2032193  27485368 223929240
    Mknod    Fsstat    Fsinfo  PathConf    Commit   LookupP   SetClId SetClIdCf
        0        53       268       131  15999631         0       386       386
     Open  OpenAttr OpenDwnGr  OpenCfrm DelePurge   DeleRet     GetFH      Lock
 80924092         0         0       194         0         0  81110394         0
    LockT     LockU     Close    Verify   NVerify     PutFH  PutPubFH PutRootFH
        0         0  80578106         0         0 1203868156         0       193
    Renew RestoreFH    SaveFH   Secinfo RelLckOwn  V4Create
     1271         0        14       384         0       570
Server:
Retfailed    Faults   Clients
        0         0       191
OpenOwner     Opens LockOwner     Locks    Delegs 
      192       154         0         0         0 
Server Cache Stats:
   Inprog      Idem  Non-idem    Misses CacheSize   TCPPeak
        0         0         0 -167156883      1651    115531

I'd love to mix in some FreeBSD-generated loads but as discussed a
week or so ago, our NFS client can't handle reading directories from
which files are being deleted.

FWIW, I just ran a quick "pmcstat -T" and noted the following:

PMC: [unhalted-core-cycles] Samples: 775371 (100.0%) , 3264 unresolved
Key: q => exiting...
%SAMP IMAGE      FUNCTION             CALLERS
 24.0 kernel     _mtx_lock_sleep      _vm_map_lock:22.4 ...
  4.7 kernel     Xinvlrng
  4.7 kernel     _mtx_lock_spin       pmclog_reserve
  4.2 kernel     _sx_xlock_hard       _sx_xlock
  3.8 pmcstat    _init
  2.5 kernel     bcopy                vdev_queue_io_done
  1.7 kernel     _sx_xlock
  1.6 zfs.ko     lzjb_compress        zio_compress_data
  1.4 zfs.ko     lzjb_decompress      zio_decompress
  1.2 kernel     _sx_xunlock
  1.2 kernel     ipfw_chk             ipfw_check_hook
  1.1 libc.so.7  bsearch
  1.0 zfs.ko     fletcher_4_native    zio_checksum_compute
  1.0 kernel     vm_page_splay        vm_page_find_least
  1.0 kernel     cpu_idle_mwait       sched_idletd
  1.0 kernel     free
  0.9 kernel     bzero
  0.9 kernel     cpu_search_lowest    cpu_search_lowest
  0.8 kernel     vm_map_entry_splay   vm_map_lookup_entry
  0.8 kernel     cpu_search_highest   cpu_search_highest

I doubt that this is news to anybody.  Once I get the production
servers upgraded to 9.3, I'll be ready to start testing 10.1 on this
same setup.

-GAWollman

[1] I did previous testing, with smaller numbers of clients, using v3
as that is what we currently require our clients to use.  I switched
to v4 to try out the worst case -- after finding an OpenStack bug that
was preventing me from starting more than 16 load generators at a
time.