NFS client slow on amd64 6.2-PRERELEASE #2

[Heinrich Rebehn]

Heinrich Rebehn wrote:
> Hi list,
> 
> despite recent improvements with the nfs code, client performance still
> seems to be a problem. I am getting < 2 MB/sec where i would expect at
> least 10 MB/sec.
> 
> My Setup:
> 
> Machine  | ANTSRV1            | ANTSRV2
> ==================================================
> Board    | Tyan 2882-D        | ASUS A7V8X-X
> CPU      | Opteron 246        | Athlon XP1800+
> Mem     | 2G                 | 1G
> NIC      | Broadcom BCM5704C  | Intel PRO/1000MT
> 
> OS:6.2-PRERELEASE #2 as of 01-OCT-2006
> 64Bit OS on ANTSRV1, 32Bit on ANTSRV2
> 
> The 2 machines are connected via a 1GB switch
> 
> ============================
> Trying UDP first
> ============================
> root at antsrv1 [/tmp] # mount_nfs -U -L antsrv2:/export/disk2 /mnt
> root at antsrv1 [/tmp] # rm /mnt/x/*
> root at antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/10MB.dat bs=1M 
> count=10
> 10+0 records in
> 10+0 records out
> 10485760 bytes transferred in 4.967248 secs (2110980 bytes/sec)
> 
> real    0m5.253s
> user    0m0.005s
> sys     0m0.022s
> root at antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/100MB.dat bs=1M
> count=100
> 100+0 records in
> 100+0 records out
> 104857600 bytes transferred in 69.020366 secs (1519227 bytes/sec)
> 
> real    1m9.276s
> user    0m0.028s
> sys     0m0.211s
> 
> ==============================
> Now TCP
> ==============================
> root at antsrv1 [/tmp] # umount /mnt
> root at antsrv1 [/tmp] # mount_nfs -T -L antsrv2:/export/disk2 /mnt
> root at antsrv1 [/tmp] # rm /mnt/x/*
> root at antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/10MB.dat bs=1M 
> count=10
> 10+0 records in
> 10+0 records out
> 10485760 bytes transferred in 5.289492 secs (1982376 bytes/sec)
> 
> real    0m5.312s
> user    0m0.001s
> sys     0m0.023s
> root at antsrv1 [/tmp] # time dd if=/dev/zero of=/mnt/x/100MB.dat bs=1M
> count=100
> 100+0 records in
> 100+0 records out
> 104857600 bytes transferred in 58.715595 secs (1785856 bytes/sec)
> 
> real    0m58.740s
> user    0m0.000s
> sys     0m0.204s
> 
> ===============================
> SCP
> ===============================
> root at antsrv1 [/tmp] # scp 100MB.dat antsrv2:/tmp/x
> Password:
> 100MB.dat                                            100%  100MB
> 11.1MB/s   00:09
> 
> ===============================
> Iperf
> ===============================
> root at antsrv1 [~] # iperf -c antsrv2
> ------------------------------------------------------------
> Client connecting to antsrv2, TCP port 5001
> TCP window size: 32.5 KByte (default)
> ------------------------------------------------------------
> [  3] local 134.102.176.16 port 59123 connected with 134.102.176.17 port
> 5001
> [  3]  0.0-10.0 sec    743 MBytes    623 Mbits/sec
> 
> NFS from a Linux machine (Kernel 2.6.17.8) (connected via 10Mb LAN)
> yields a constant value of ~10MB/sec (TCP and UDP), so the FreeBSD NFS
> server seems to be ok.
> 
> =================================
> Reversing the setup
> =================================
> root at antsrv2 [~] # mount_nfs -L -U antsrv1:/export/huge/x /mnt
> root at antsrv2 [~] # time dd if=/dev/zero of=/mnt/10MB.dat bs=1M count=10
> 10+0 records in
> 10+0 records out
> 10485760 bytes transferred in 0.533470 secs (19655758 bytes/sec)
> 
> real    0m0.547s
> user    0m0.001s
> sys     0m0.085s
> root at antsrv2 [~] # time dd if=/dev/zero of=/mnt/100MB.dat bs=1M count=100
> 100+0 records in
> 100+0 records out
> 104857600 bytes transferred in 7.647475 secs (13711401 bytes/sec)
> 
> real    0m7.673s
> user    0m0.000s
> sys     0m0.920s
> root at antsrv2 [~] # umount /mnt
> root at antsrv2 [~] # mount_nfs -L -T antsrv1:/export/huge/x /mnt
> root at antsrv2 [~] # rm /mnt/*
> root at antsrv2 [~] # time dd if=/dev/zero of=/mnt/10MB.dat bs=1M count=10
> 10+0 records in
> 10+0 records out
> 10485760 bytes transferred in 0.524917 secs (19976030 bytes/sec)
> 
> real    0m0.557s
> user    0m0.000s
> sys     0m0.085s
> root at antsrv2 [~] # time dd if=/dev/zero of=/mnt/100MB.dat bs=1M count=100
> 100+0 records in
> 100+0 records out
> 104857600 bytes transferred in 5.723110 secs (18321787 bytes/sec)
> 
> real    0m5.756s
> user    0m0.000s
> sys     0m0.704s
> 
> 
> This yields much better results. Could this be a 64bit-specific problem?
> 
> Both machines have statd and lockd running.
> 
> Any ideas where i could tune?
> 

Hmm, no one? Is 2MB/s on a Gbit link all i can expect on FreeBSD?

--Heinrich