NFS Performance: Weirder And Weirder

[Tim Daneliuk]

This is really weird.  A FreeBSD 9.1 system mounts the following:

/dev/ad4s1a    989M    625M    285M    69%    /
devfs          1.0k    1.0k      0B   100%    /dev
/dev/ad4s1d    7.8G      1G    6.1G    14%    /var
/dev/ad4s1e     48G    9.4G     35G    21%    /usr
/dev/ad4s1f    390G    127G    231G    35%    /usr1
/dev/ad6s1d    902G    710G    120G    86%    /usr1/BKU

/usr1/something (under ad4s1f) and /usr1/BKU (all of ad6s1d) are
exported for NFS mounting on the LAN.  I have tested the
speeds of these two drives locally doing a 'dd if=/dev/zero ....'.
Their speeds are quite comparable - around 55-60 MB/s so the
problem below is not an artifact of a slow drive.

The two mounts are imported like this on a Linux Mint 12 machine:


   machine:/usr1/BKU     /BKU     nfs   rw,soft,intr          0  0
   machine:/usr1/shared  /shared  nfs   rw,soft,intr          0  0

Problem:

When I write files from the LM12 machines to /BKU  the writes are
1/10 the speed of when writing to /shared.  Reads are fine in both
cases, at near native disk speeds being reported.

Someone here suggested I get rid of any symlinks in the mount and I did
that to no avail.


Incidentally, the only reason I just noticed this is that I upgraded the
NIC on the FreeBSD machine and the switch into which it connects to 1000Base
because the LM12 machine had a built in 1000Base NIC.  I also changed
the cables on both machines to ensure they were not the problem.   Prior
to this, I was bandwidth constrained by the 100Base so I never saw NFS
performance as an issue.  When I upgraded, I expected faster transfers
and when I didn't get them, I started this whole investigation.

So ... I'm stumped:

- It's not the drive or SATA ports because both drives show comparable performance.
- It's not the cables because I can get great throughput on one of the NFS mountpoints.
- It's neither NIC for the same reason.

Does anyone:

A) Have a clue what might be doing this
B) Have a suggestion how to track down the problem

Thanks,

-- 
----------------------------------------------------------------------------
Tim Daneliuk     tundra at tundraware.com
PGP Key:         http://www.tundraware.com/PGP/