NFSv3 and v4 slow with newnfs code

[Nikolai Schupbach]

We are doing some performance testing on a new system. We have a OpenSolaris NFS server sharing a folder on a ZFS filesystem and a FreeBSD 8.1 NFS client. The machines are directly connected using 10GbE (no switch in-between).

Below are the performance figures we attained when doing simple 10GB dd write (dd if=/dev/zero of=/mnt/file.tmp bs=1M count=10240) and read (dd of=/dev/null if=/mnt/file.tmp) tests over NFS from the FreeBSD client using various mount options.

We performed these tests numerous times and all results are roughly the same for each test. We have tuned kern.ipc.maxsockbuf, net.inet.tcp.recvspace and net.inet.tcp.sendspace. This didn't result in any significant differences in the test results. Both NFS client and server NICs have MTU set to 9000; this improves performance noticeable.

Currently it appears sticking with the stable NFSv3 code yields the best results. Both NFSv3 and NFSv4 with the newnfs code has disappointing performance. We installed Linux on the client machine as a test and unfortunately Linux has the best performance by far. 

Are there any other options we can use to improve the performance of NFSv4 for large sequential writes and reads? 

FreeBSD NFSv4 \w newnfs
---
mount -t newnfs -o nfsv4 10.0.0.2:/volumes/datpool/test /mnt

Writing: 10737418240 bytes transferred in 236.627936 secs, 43 MB/s
Reading: 10737418240 bytes transferred in 131.385064 secs, 95 MB/s

mount -t newnfs -o nfsv4,wsize=32768,rsize=32768 10.0.0.2:/volumes/datpool/test /mnt

Writing: 10737418240 bytes transferred in 258.567077 secs, 39 MB/s
Reading: 10737418240 bytes transferred in 120.646962 secs, 85 MB/s


FreeBSD NFSv3 \w newnfs
---
mount -t newnfs -o nfsv4 10.0.0.2:/volumes/datpool/test /mnt

Writing 10737418240 bytes transferred in 215.963140 secs, 47 MB/s
Reading 10737418240 bytes transferred in 101.110872 secs, 101 MB/s


FreeBSD NFSv3 
---
mount -t nfs -o nfsv3 10.0.0.2:/volumes/datpool/test /mnt

Writing: 10737418240 bytes transferred in 161.739156 secs 63 MB/s
Reading: 10737418240 bytes transferred in 72.327859 secs, 129 MB/s


Linux NFSv4 (CentOS 5.5)
---
Writing: 10737418240 bytes (11 GB) copied, 143.776 seconds, 75 MB/s
Reading: 10737418240 bytes (11 GB) copied, 63.7012 seconds, 170 MB/s

--Nik