Namecache lock contention?

[Ivan Voras]

On 28 January 2011 16:15, John Baldwin <jhb at freebsd.org> wrote:
> On Friday, January 28, 2011 8:46:07 am Ivan Voras wrote:
>> I have this situation on a PHP server:
>>
>> 36623 www         1  76    0   237M 30600K *Name   6   0:14 47.27% php-cgi
>> 36638 www         1  76    0   237M 30600K *Name   3   0:14 46.97% php-cgi
>> 36628 www         1 105    0   237M 30600K *Name   2   0:14 46.88% php-cgi
>> 36627 www         1 105    0   237M 30600K *Name   0   0:14 46.78% php-cgi
>> 36639 www         1 105    0   237M 30600K *Name   5   0:14 46.58% php-cgi
>> 36643 www         1 105    0   237M 30600K *Name   7   0:14 46.39% php-cgi
>> 36629 www         1  76    0   237M 30600K *Name   1   0:14 46.39% php-cgi
>> 36642 www         1 105    0   237M 30600K *Name   2   0:14 46.39% php-cgi
>> 36626 www         1 105    0   237M 30600K *Name   5   0:14 46.19% php-cgi
>> 36654 www         1 105    0   237M 30600K *Name   7   0:13 46.19% php-cgi
>> 36645 www         1 105    0   237M 30600K *Name   1   0:14 45.75% php-cgi
>> 36625 www         1 105    0   237M 30600K *Name   0   0:14 45.56% php-cgi
>> 36624 www         1 105    0   237M 30600K *Name   6   0:14 45.56% php-cgi
>> 36630 www         1  76    0   237M 30600K *Name   7   0:14 45.17% php-cgi
>> 36631 www         1 105    0   237M 30600K RUN     4   0:14 45.17% php-cgi
>> 36636 www         1 105    0   237M 30600K *Name   3   0:14 44.87% php-cgi
>>
>> It looks like periodically most or all of the php-cgi processes are
>> blocked in "*Name" for long enough that "top" notices, then continue,
>> probably in a "thundering herd" way. From grepping inside /sys the most
>> likely suspect seems to be something in the namecache, but I can't find
>> exactly a symbol named "Name" or string beginning with "Name" that would
>> be connected to a lock.
>
> In vfs_cache.c:
>
> static struct rwlock cache_lock;
> RW_SYSINIT(vfscache, &cache_lock, "Name Cache");

You're right, I misread it as SYSCTL at a glance.

> What are the php scripts doing?  Do they all try to create and delete files at
> the same time (or do renames)?

Right again - they do simultaneously create session files and in rare
occasions (1%) delete them. These are "sharded" into a two-level
directory structure by single letter (/storage/a/b/file, i.e. 32^2
directories); dirhash is large enough.

During all this, the web server did around 60 PHP pages per second so
it doesn't look to me like there should be such noticable contention
(i.e. at most, there are 60 files/s created and on average 60/100
deletes). The file system is on softupdates, there's only light IO.

Typical vmstat is:

 procs      memory      page                    disks     faults         cpu
 r b w     avm    fre   flt  re  pi  po    fr  sr da0 da1   in   sy
cs us sy id

17 0 0   8730M  1240M     3   0   0   0   206   0   1   0 1948 266928
15079 65 34  1
19 0 0   8730M  1240M     0   0   0   0   290   0   1  24 1835 260618
15132 63 35  2
 7 0 0   8730M  1239M     0   0   0   0   200   0   0   0 1822 260783
14851 63 35  2
16 0 0   8730M  1239M     0   0   0   0   199   0 788   0 2744 259902
20465 61 37  2
16 0 0   8730M  1239M     0   0   0   0   210   0   0   0 1755 265081
17564 61 37  2

(8 cores; around 35% sys load across them - I'm trying to find out why).