SLOW rsync from bit0.us-west

[Konstantin Belousov]

On Sat, Jan 12, 2013 at 07:58:18PM +0000, Simon L. B. Nielsen wrote:
> On 12 January 2013 15:37, Simon L. B. Nielsen <simon at freebsd.org> wrote:
> > On 12 January 2013 05:21, John Marshall
> > <john.marshall at riverwillow.com.au> wrote:
> >> I have been updating the gnats collection on our mirror via rsync since
> >> the CVSup updates stopped but noticed this morning that the updates
> >> appeared to be stalled. I killed off jobs that appeared to be stuck
> >> downloading incremental file list (after about 20 minutes) but then
> >> thought to run rsync manually with -vvv. What I saw then was short
> >> bursts of messages like 'recv_file_name(bin/nnnn)' interspersed with
> >> 15-20 seconds of nothing. tcpdump showed just a handful of packets every
> >> 15 seconds or so. The update eventually finished after more than 2 hours.
> >
> > The server which runs bit0.us-west (which is also svn0.us-west), is
> > totally bogged down.
> >
> > CPU:  3.1% user,  0.0% nice, 96.0% system,  0.0% interrupt,  1.0% idle
> >
> > As it has just been updated to the latest stable/9 I'm a bit
> > suspicious of that...
> >
> > Thanks for letting us know. I will start poking people.
> 
> Please have another look and see if things aren't better now.
> 
> It looks like we are stressing nullfs more than people had before - at
> least with way more vnode / files so the server was spending all of
> its time doing vnode lookups.

What are your sysctl kerne.numvnodes and kern.maxvnodes ? How did you
determined that lookups were the cause ?

The deal with nullfs is that I MFCed a set of changes recently which
allow to keep nullfs vnodes cached. As a consequence, the nullfs vnodes
could be (usefully) locked shared not, which provides up to 100% increase
in the speed of the metadata-intensive operations on nullfs, as measured
by Peter Holm.

The drawback of the change is that nullfs vnodes are cached, which
effectively doubles the amount of vnodes the system operates on.

I added the nocache nullfs mount option to return to the pre-cached
behaviour, but this is not propagated to the stable/9 yet. Supposedly,
you could use the option if increasing kern.maxvnodes is not possible.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 834 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-hubs/attachments/20130112/2fbef32a/attachment.sig>