NFS Performance

[george+freebsd at m5p.com]

Among four machines on my network, I'm observing startling differences
in NFS performance.  All machines are AMD64, and rpc_statd, rpc_lockd,
and amd are enabled on all four machines.

wonderland:
hw.model: AMD Athlon(tm) II Dual-Core M32
hw.physmem: 293510758
ethernet: 100Mb/s
partition 1: FreeBSD 8.1-STABLE
partition 2: FreeBSD 7.3-STABLE

scollay:
hw.model: AMD Sempron(tm) 140 Processor
hw.physmem: 186312294
ethernet: 1000Mb/s
FreeBSD 8.1-PRERELEASE

sullivan:
hw.model: AMD Athlon(tm) 64 X2 Dual Core Processor 4800+
hw.physmem: 4279980032
ethernet 1000Mb/s
FreeBSD 7.2-RELEASE

mattapan:
hw.model: AMD Sempron(tm) Processor 2600+
hw.physmem: 456380416
ethernet: 1000Mb/s
FreeBSD 7.1-RELEASE

Observed bytes per second (dd if=filename of=/dev/null bs=65536):
Source machine:  mattapan         scollay          sullivan
Destination machine:
wonderland/7.3       870K            5.2M              1.8M
wonderland/8.1       496K            690K              420K
mattapan                              38M               28M
scollay               33M                               33M
sullivan              38M              5M

There is one 10/100/1000Mb/s ethernet switch between the various pairs
of machines.

I'm startled by the numbers for wonderland, first because of how much the
100Mb/s interface slows things down, but even more because of how much
difference there is on the identical hardware between FreeBSD 7 and
FreeBSD 8.

Even more annoying when running 8.1 on wonderland, NFS simply locks up
at random for roughly a minute and a half under high load (such as when
firefox does a gazillion locked references to my places.sqlite file),
leading to entertaining log message clusters such as:

Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:17:47 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:17:47 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:18:01 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:18:01 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:18:08 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:18:08 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:20:21 wonderland last message repeated 2 times
Dec 29 08:20:22 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:20:22 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:20:36 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:20:36 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:21:05 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:21:10 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:22:22 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:22:22 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:22:24 wonderland last message repeated 2 times
Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:22:24 wonderland last message repeated 2 times
Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: is alive again
Dec 29 08:22:27 wonderland kernel: nfs server home:/usr: not responding
Dec 29 08:22:27 wonderland kernel: nfs server home:/usr: is alive again

The time stamps on the "not responding" messages are lies; the lack of
response in each case was roughly a minute and a half earlier.  "home"
equals mattapan in the earlier table.  During the periods of NFS lack
of response, mattapan is still completely functional, and a concurrent
ssh session proceeds without difficulty.  I can also make this happen
(though not as easily) on NFS from sullivan to wonderland, but I haven't
seen it yet from scollay to wonderland.  This problem never occurs when
wonderland is running 7.3.

Any suggestions?                                     -- George Mitchell