Weird NFS behaviour

[Ulrich Spoerlein]

Hi,

we have performance problems with our FreeBSD 6.2 based NFS server.
Picture the following setup:

FreeBSD Client ---> Samba-Server  ---> NFS-Server

all three machines are running FreeBSD 6.2 (the same image). The NFS
server is configured with 16 nfsd. sysctl.conf has
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536

Now, what's the problem: The Samba-Server mounts shares via NFS. All
servers are on Gigabit Ethernet and I get read transfer rates
exceeding 50MB/s from the NFS server.

This is all good and well, but if I copy a file via scp(1) (sic!) to
the samba server into the NFS mounted directory, not only do I
seldomly exceed 12MB/s but I also get a very strange traffic pattern
on the em0 interface of the samba server. I get _twice_ as much
incoming traffic on the em0 interface as outgoing traffic.

systat -if on samba:
          em0  in     24.726 MB/s         25.905 MB/s            3.046 GB
                 out    12.941 MB/s         13.558 MB/s            1.994 GB

systat -if on nfs-server
          em0  in     11.497 MB/s         12.999 MB/s            3.727 GB
                 out    11.878 MB/s         13.423 MB/s          995.485 MB

To stress, this is running:
gigabit-client:# scp large-file user at samba:/mnt/nfs-share/

The wicked part is this: If I copy a file from the samba server
directly to the NFS share (not as a passthrough), I get these traffic
patterns:
systat -if on samba:
           em0  in    432.724 KB/s        432.724 KB/s            3.772 GB
                 out    12.399 MB/s         12.399 MB/s            2.481 GB

systat -if on nfs:
           em0  in     12.091 MB/s         15.791 MB/s          184.766 MB
                 out   440.939 KB/s        562.521 KB/s            1.339 GB

This is running:
samba:# cp large-file /mnt/nfs-share/

What on earth is causing each received NFS packet to be _bounced_ to
the samba server when using ssh, scp, smbd, etc. And not when
generating the traffic locally?

nfsstat -s is showing an increase in READ calls similar to WRITE calls
when using the samba machine as pass-through. It is showing _no_
increase in READ calls when copying the files directly.

NB: All these test were run _without_ smbd running, it's just that
this server is designated to become our samba server.

Setting vfs.nfsrv.async=1 doubled write performance, but the weird
traffic pattern remains. (Am I asking for too much trouble by setting
async NFS?)

Thanks for any pointers!
Uli