Max NFSD processes

[Eric Anderson]

Dan Nelson wrote:

>In the last episode (May 19), Eric Anderson said:
>  
>
>>I have several heavily used NFS servers, currently running FreeBSD
>>4.9-RELEASE.  I'm getting jammed up with all my nfsd processes being
>>busy, so clients see slow connections to the server.  I have the nfsd
>>starting with a count of 20, which is the max set in the nfsd.c file.
>>
>>Are there any risks I should be aware of before bumping up the max to 
>>say 40, or even 50? 
>>
>>What would it take to make this a sysctl adjustable value?
>>
>>Should the max be bumped higher by default nowdays?
>>    
>>
>
>What's the output of "ps ax | grep nfsd"?  How much CPU does the last
>nfsd process have?
>
>If your backend storage is a RAID with lots of disks, and your last
>nfsd is actually getting some use, then bumping up the nfsds will
>probably help.  Although if you're hitting a kernel bottleneck (locking
>for example), more nfsds won't do any good.
>  
>
Here's the output:
   97  ??  Is     0:00.01 nfsd: master (nfsd)
   99  ??  S      4:52.61 nfsd: server (nfsd)
  100  ??  S      1:15.74 nfsd: server (nfsd)
  101  ??  S      0:44.05 nfsd: server (nfsd)
  102  ??  S      0:31.79 nfsd: server (nfsd)
  103  ??  S      0:26.15 nfsd: server (nfsd)
  104  ??  S      0:20.36 nfsd: server (nfsd)
  105  ??  S      0:18.47 nfsd: server (nfsd)
  106  ??  S      0:16.86 nfsd: server (nfsd)
  107  ??  S      0:19.11 nfsd: server (nfsd)
  108  ??  S      0:16.68 nfsd: server (nfsd)
  109  ??  S      0:13.59 nfsd: server (nfsd)
  110  ??  S      0:13.60 nfsd: server (nfsd)
  111  ??  S      0:12.30 nfsd: server (nfsd)
  112  ??  S      0:12.44 nfsd: server (nfsd)
  113  ??  S      0:13.84 nfsd: server (nfsd)
  114  ??  S      0:12.65 nfsd: server (nfsd)
  115  ??  S      0:13.57 nfsd: server (nfsd)
  116  ??  S      0:11.31 nfsd: server (nfsd)
  117  ??  S      0:11.21 nfsd: server (nfsd)
  118  ??  I      0:11.99 nfsd: server (nfsd)

The machine has been up now less than 5 hours, and this is a 'quiet' time. 

During the 'slow' time, top showed the nfsd processes in "biorw" and 
"inode" states.  All were consumed in those states. 

The machine has two raid 5 arrays, with a hardware raid controller.  
iostat showed xfer speeds to the first array about 2MB/s, and nothing 
really abnormal about it.  Clients had difficulty with simple things 
like 'ls' on the partition.  mountd was responding quickly with mount 
requests (I believe), but once the mount was made, accessing the nfs 
disk was horribly slow.

Any more ideas?

I'm not subscribed on -questions or -net, so please keep me on the cc's.

Eric


-- 
------------------------------------------------------------------
Eric Anderson     Sr. Systems Administrator    Centaur Technology
Today is the tomorrow you worried about yesterday.
------------------------------------------------------------------