High CPU usage with newnfs(d) - seems to be a cache issue

[Attila Nagy]

Hi,

On 09/10/13 00:57, Rick Macklem wrote:
> Attila Nagy wrote:
>> Hi,
>>
>> I've observed some insane CPU usage on stable/9 at r255367.
>> About the machine:
>> CPU: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz (2400.14-MHz
>> K8-class CPU)
>> real memory  = 34359738368 (32768 MB)
>> FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
>> FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
>>
>> It does some NFS serving like this (now running oldnfs) -not quite
>> peak
>> times actually:
>> # nfsstat -w 1 -os
>>    GtAttr Lookup Rdlink   Read  Write Rename Access  Rddir
>>       763   7206      1    175     92      0    915   3589
>>       748   7665     10    131     60      0    905   2923
>>       787   9657     23    204     50      0    974   2387
>>       517   9881      9    150     41      0    572   2321
>>       709   8708     71    235     70      0   1220   3271
>>       621   9157      9    254    208      0    928   2563
>>       699   5336     29    271    103      0   1242   3448
>>       656   4291     11    201    209      0   1119   3908
>>       506   3722      0    215    183      0    970   2516
>>       698   1476      1    151     66      0    903   2094
>>       501   2865     11    268    117      0    995   1392
>>       638   6284     46    233     47      0   1096   4847
>>       893   7909     47    175     73      0    870   4070
>>       651   3936     48    255     51      0    955   2514
>>       424   4211     17    223     29      0    745   1458
>>       589   8197     26    199     39      0    918   2983
>>
>> It's being hammered by about 40 machines on multiple connections (it
>> has
>> 35 UFS file systems exported).
>>
>> When running newnfs (admittedly in some stupid way, with -n 32, the
>> profiling was made with this, maybe this causes some lock
>> contention),
>> it occasionally eats 1600% CPU (means: 0 idle).
>> Lowering the thread number doesn't really solves the problem, I've
>> seen
>> -n X*100 CPU usage peaks lately on machines with lower (4-8) -n
>> counts...
>>
>> Doing a profiling with pmc shows that most of the time is spent in
>> nfsrvd_updatecache and nfsrvd_getcache:
>> http://pastebin.com/knyppv4d
>>
>> Switching back to oldnfsd (even with -n 32) gives a stable 50-60% CPU
>> usage (out of the "possible" 1600%) when loaded.
>>
>> I know that there are some changes regarding this cache in the
>> CURRENT
>> code (along with the possibility to set some values with sysctls),
>> but I
>> can't run CURRENT.
>>
>> Any ideas on how to improve newnfsd, so we can continue serving NFS
>> in
>> the future days, where I won't be able to switch back to the old one?
>> :)
>>
> Well, I put a 1 month MFC on r254337 (which I believe fixes this), so
> it should be in stable/9 in about a week. Alternately, an uglier (but
> semantically equivalent) patch can be found at:
>    http://people.freebsd.org/~rmacklem/drc4-stable9.patch
>
Great, I'm eagerly waiting for this to happen then. :)