High CPU usage with newnfs(d) - seems to be a cache issue

[Attila Nagy]

Hi,

I've observed some insane CPU usage on stable/9 at r255367.
About the machine:
CPU: Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz (2400.14-MHz 
K8-class CPU)
real memory  = 34359738368 (32768 MB)
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads

It does some NFS serving like this (now running oldnfs) -not quite peak 
times actually:
# nfsstat -w 1 -os
  GtAttr Lookup Rdlink   Read  Write Rename Access  Rddir
     763   7206      1    175     92      0    915   3589
     748   7665     10    131     60      0    905   2923
     787   9657     23    204     50      0    974   2387
     517   9881      9    150     41      0    572   2321
     709   8708     71    235     70      0   1220   3271
     621   9157      9    254    208      0    928   2563
     699   5336     29    271    103      0   1242   3448
     656   4291     11    201    209      0   1119   3908
     506   3722      0    215    183      0    970   2516
     698   1476      1    151     66      0    903   2094
     501   2865     11    268    117      0    995   1392
     638   6284     46    233     47      0   1096   4847
     893   7909     47    175     73      0    870   4070
     651   3936     48    255     51      0    955   2514
     424   4211     17    223     29      0    745   1458
     589   8197     26    199     39      0    918   2983

It's being hammered by about 40 machines on multiple connections (it has 
35 UFS file systems exported).

When running newnfs (admittedly in some stupid way, with -n 32, the 
profiling was made with this, maybe this causes some lock contention), 
it occasionally eats 1600% CPU (means: 0 idle).
Lowering the thread number doesn't really solves the problem, I've seen 
-n X*100 CPU usage peaks lately on machines with lower (4-8) -n counts...

Doing a profiling with pmc shows that most of the time is spent in 
nfsrvd_updatecache and nfsrvd_getcache:
http://pastebin.com/knyppv4d

Switching back to oldnfsd (even with -n 32) gives a stable 50-60% CPU 
usage (out of the "possible" 1600%) when loaded.

I know that there are some changes regarding this cache in the CURRENT 
code (along with the possibility to set some values with sysctls), but I 
can't run CURRENT.

Any ideas on how to improve newnfsd, so we can continue serving NFS in 
the future days, where I won't be able to switch back to the old one? :)

Thanks,