Why is NFSv4 so slow?

[Rick C. Petty]

Hi.  I'm still having problems with NFSv4 being very laggy on one client.
When the NFSv4 server is at 50% idle CPU and the disks are < 1% busy, I am
getting horrible throughput on an idle client.  Using dd(1) with 1 MB block
size, when I try to read a > 100 MB file from the client, I'm getting
around 300-500 KiB/s.  On another client, I see upwards of 20 MiB/s with
the same test (on a different file).  On the broken client:

# uname -mv
FreeBSD 8.1-STABLE #5 r211534M: Sat Aug 28 15:53:10 CDT 2010 user at example.com:/usr/obj/usr/src/sys/GENERIC  i386

# ifconfig re0
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
	ether 00:e0:4c:xx:yy:zz
	inet xx.yy.zz.3 netmask 0xffffff00 broadcast xx.yy.zz.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active

# netstat -m
267/768/1035 mbufs in use (current/cache/total)
263/389/652/25600 mbuf clusters in use (current/cache/total/max)
263/377 mbuf+clusters out of packet secondary zone in use (current/cache)
0/20/20/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
592K/1050K/1642K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/5/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

# netstat -idn
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
re0    1500 <Link#1>      00:e0:4c:xx:yy:zz   232135     0     0    68984     0     0    0 
re0    1500 xx.yy.zz.0/2 xx.yy.zz.3         232127     -     -    68979     -     -    - 
nfe0*  1500 <Link#2>      00:22:15:xx:yy:zz        0     0     0        0     0     0    0 
plip0  1500 <Link#3>                               0     0     0        0     0     0    0 
lo0   16384 <Link#4>                              42     0     0       42     0     0    0 
lo0   16384 fe80:4::1/64  fe80:4::1                0     -     -        0     -     -    - 

lo0   16384 ::1/128       ::1                      0     -     -        0     -     -    - 
lo0   16384 127.0.0.0/8   127.0.0.1               42     -     -       42     -     -    - 

# sysctl kern.ipc.maxsockbuf
kern.ipc.maxsockbuf: 1048576
# sysctl net.inet.tcp.sendbuf_max
net.inet.tcp.sendbuf_max: 16777216
# sysctl net.inet.tcp.recvbuf_max
net.inet.tcp.recvbuf_max: 16777216
# sysctl net.inet.tcp.sendspace
net.inet.tcp.sendspace: 65536
# sysctl net.inet.tcp.recvspace
net.inet.tcp.recvspace: 131072

# sysctl hw.pci | grep msi
hw.pci.honor_msi_blacklist: 1
hw.pci.enable_msix: 1
hw.pci.enable_msi: 1

# vmstat -i
interrupt                          total       rate
irq14: ata0                           47          0
irq16: re0                        219278        191
irq21: ohci0+                       5939          5
irq22: vgapci0+                    77990         67
cpu0: timer                      2294451       1998
irq256: hdac0                      44069         38
cpu1: timer                      2293983       1998
Total                            4935757       4299

Any ideas?

-- Rick C. Petty