High CPU usage with newnfs(d) - seems to be a cache issue

[Rick Macklem]

Attila Nagy wrote:
> Hi,
> 
> I've observed some insane CPU usage on stable/9 at r255367.
> About the machine:
> CPU: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz (2400.14-MHz
> K8-class CPU)
> real memory  = 34359738368 (32768 MB)
> FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
> FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
> 
> It does some NFS serving like this (now running oldnfs) -not quite
> peak
> times actually:
> # nfsstat -w 1 -os
>   GtAttr Lookup Rdlink   Read  Write Rename Access  Rddir
>      763   7206      1    175     92      0    915   3589
>      748   7665     10    131     60      0    905   2923
>      787   9657     23    204     50      0    974   2387
>      517   9881      9    150     41      0    572   2321
>      709   8708     71    235     70      0   1220   3271
>      621   9157      9    254    208      0    928   2563
>      699   5336     29    271    103      0   1242   3448
>      656   4291     11    201    209      0   1119   3908
>      506   3722      0    215    183      0    970   2516
>      698   1476      1    151     66      0    903   2094
>      501   2865     11    268    117      0    995   1392
>      638   6284     46    233     47      0   1096   4847
>      893   7909     47    175     73      0    870   4070
>      651   3936     48    255     51      0    955   2514
>      424   4211     17    223     29      0    745   1458
>      589   8197     26    199     39      0    918   2983
> 
> It's being hammered by about 40 machines on multiple connections (it
> has
> 35 UFS file systems exported).
> 
> When running newnfs (admittedly in some stupid way, with -n 32, the
> profiling was made with this, maybe this causes some lock
> contention),
> it occasionally eats 1600% CPU (means: 0 idle).
> Lowering the thread number doesn't really solves the problem, I've
> seen
> -n X*100 CPU usage peaks lately on machines with lower (4-8) -n
> counts...
> 
> Doing a profiling with pmc shows that most of the time is spent in
> nfsrvd_updatecache and nfsrvd_getcache:
> http://pastebin.com/knyppv4d
> 
> Switching back to oldnfsd (even with -n 32) gives a stable 50-60% CPU
> usage (out of the "possible" 1600%) when loaded.
> 
> I know that there are some changes regarding this cache in the
> CURRENT
> code (along with the possibility to set some values with sysctls),
> but I
> can't run CURRENT.
> 
> Any ideas on how to improve newnfsd, so we can continue serving NFS
> in
> the future days, where I won't be able to switch back to the old one?
> :)
> 
Well, I put a 1 month MFC on r254337 (which I believe fixes this), so
it should be in stable/9 in about a week. Alternately, an uglier (but
semantically equivalent) patch can be found at:
  http://people.freebsd.org/~rmacklem/drc4-stable9.patch

rick

> Thanks,
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
>