Namecache lock contention?

[Dan Nelson]

In the last episode (Jan 28), Ivan Voras said:
> I have this situation on a PHP server:
> 
> 36623 www         1  76    0   237M 30600K *Name   6   0:14 47.27% php-cgi
> 36638 www         1  76    0   237M 30600K *Name   3   0:14 46.97% php-cgi
> 36628 www         1 105    0   237M 30600K *Name   2   0:14 46.88% php-cgi
> 36627 www         1 105    0   237M 30600K *Name   0   0:14 46.78% php-cgi
> 36639 www         1 105    0   237M 30600K *Name   5   0:14 46.58% php-cgi
> 36643 www         1 105    0   237M 30600K *Name   7   0:14 46.39% php-cgi
> 36629 www         1  76    0   237M 30600K *Name   1   0:14 46.39% php-cgi
> 36642 www         1 105    0   237M 30600K *Name   2   0:14 46.39% php-cgi
> 36626 www         1 105    0   237M 30600K *Name   5   0:14 46.19% php-cgi
> 36654 www         1 105    0   237M 30600K *Name   7   0:13 46.19% php-cgi
> 36645 www         1 105    0   237M 30600K *Name   1   0:14 45.75% php-cgi
> 36625 www         1 105    0   237M 30600K *Name   0   0:14 45.56% php-cgi
> 36624 www         1 105    0   237M 30600K *Name   6   0:14 45.56% php-cgi
> 36630 www         1  76    0   237M 30600K *Name   7   0:14 45.17% php-cgi
> 36631 www         1 105    0   237M 30600K RUN     4   0:14 45.17% php-cgi
> 36636 www         1 105    0   237M 30600K *Name   3   0:14 44.87% php-cgi
> 
> It looks like periodically most or all of the php-cgi processes are
> blocked in "*Name" for long enough that "top" notices, then continue,
> probably in a "thundering herd" way.  From grepping inside /sys the most
> likely suspect seems to be something in the namecache, but I can't find
> exactly a symbol named "Name" or string beginning with "Name" that would
> be connected to a lock.

My guess would be:

kern/vfs_cache.c:151 static struct rwlock cache_lock;
kern/vfs_cache.c:152 RW_SYSINIT(vfscache, &cache_lock, "Name Cache");

The CACHE_*LOCK() macros.c in vfs_cache use cache_lock, so you've got lots
of possible contention points.  procstat -ka and/or dtrace might help you
determine exactly where.

-- 
	Dan Nelson
	dnelson at allantgroup.com