1 Gb network NFS issues

Nicholas Kulikov lostar at rambler.ru
Tue Feb 5 22:29:28 PST 2008 

Hello all,

I'm trying setup NFS in 1 Gb home network and found some strange issues:

Current NFS performance is about 30-35 Mb/s but few days ago i had  
40-45 Mb/s. I've made some changes in server configuration (moved to  
FreeBSD 6.3 and changed network driver to nfe) and I can't catch what  
is wrong...

Network structure:


NFS server (FreeBSD 6.3)
            |
            | 1 Gb link
            |
NFS client (Mac OS X 10.5.1)

Server configuration
-----------------------------
nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 8000
	options=48<VLAN_MTU,POLLING>

Client configuration
------------------------
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 8000
	media: autoselect (100baseTX <full-duplex>) status: active

client mount share by using the following parameters: tcp vers=3  
rsize=40960 wsize=40960 readahead=16 rdirplus nolocks intr noatime


I've tried copy big file (about 3 Gb) and tcdump-ed client and server  
packets:

07:11:08.350744 IP (tos 0x0, ttl 64, id 51461, offset 0, flags [DF],  
proto TCP (6), length 176) 192.168.254.249.nfs >  
192.168.254.254.2468561782: reply ok 124 access attr: REG 660 ids  
1001/1006 sz 3958224896 nlink 1 rdev 133/134414584 fsid 5e fileid  
2013005 a/m/ctime 1199479211.000000 1199479280.000000  
1201346449.000000 c 001f
07:11:08.350793 IP (tos 0x0, ttl 64, id 46719, offset 0, flags [DF],  
proto TCP (6), length 52) 192.168.254.254.49158 >  
192.168.254.249.nfsd: ., cksum 0x7f70 (incorrect (-> 0xd9e5),  
22844:22844(0) ack 27581 win 65535 <nop,nop,timestamp 67511239 30749984>
07:11:08.350988 IP (tos 0x0, ttl 64, id 9526, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561783 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 0
07:11:08.351069 IP (tos 0x0, ttl 64, id 8470, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561784 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
32768
07:11:08.351145 IP (tos 0x0, ttl 64, id 65343, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561785 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
65536
07:11:08.351242 IP (tos 0x0, ttl 64, id 11022, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561786 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
98304
07:11:08.351314 IP (tos 0x0, ttl 64, id 59209, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561787 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
131072
07:11:08.351383 IP (tos 0x0, ttl 64, id 39733, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561788 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
163840
07:11:08.351450 IP (tos 0x0, ttl 64, id 33325, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561789 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
196608
07:11:08.351516 IP (tos 0x0, ttl 64, id 50481, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561790 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
229376
07:11:08.351582 IP (tos 0x0, ttl 64, id 50464, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561791 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
262144
07:11:08.351658 IP (tos 0x0, ttl 64, id 37215, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561792 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
294912
07:11:08.351739 IP (tos 0x0, ttl 64, id 6496, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561793 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
327680
07:11:08.351750 IP (tos 0x0, ttl 64, id 49495, offset 0, flags [DF],  
proto TCP (6), length 52) 192.168.254.249.nfsd >  
192.168.254.254.49158: ., cksum 0x15e5 (correct), 27581:27581(0) ack  
23076 win 49943 <nop,nop,timestamp 30749985 67511239>
07:11:08.351754 IP (tos 0x0, ttl 64, id 36632, offset 0, flags [DF],  
proto TCP (6), length 52) 192.168.254.249.nfsd >  
192.168.254.254.49158: ., cksum 0x1501 (correct), 27581:27581(0) ack  
23308 win 49939 <nop,nop,timestamp 30749985 67511239>
07:11:08.351758 IP (tos 0x0, ttl 64, id 21313, offset 0, flags [DF],  
proto TCP (6), length 52) 192.168.254.249.nfsd >  
192.168.254.254.49158: ., cksum 0x141c (correct), 27581:27581(0) ack  
23540 win 49936 <nop,nop,timestamp 30749985 67511239>
07:11:08.351783 IP (tos 0x0, ttl 64, id 50759, offset 0, flags [DF],  
proto TCP (6), length 52) 192.168.254.249.nfsd >  
192.168.254.254.49158: ., cksum 0x1338 (correct), 27581:27581(0) ack  
23772 win 49932 <nop,nop,timestamp 30749985 67511239>
07:11:08.351801 IP (tos 0x0, ttl 64, id 56942, offset 0, flags [DF],  
proto TCP (6), length 52) 192.168.254.249.nfsd >  
192.168.254.254.49158: ., cksum 0x1254 (correct), 27581:27581(0) ack  
24004 win 49928 <nop,nop,timestamp 30749985 67511239>
07:11:08.351818 IP (tos 0x0, ttl 64, id 13319, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561794 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
360448
07:11:08.351904 IP (tos 0x0, ttl 64, id 17022, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561795 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
393216
07:11:08.351981 IP (tos 0x0, ttl 64, id 45600, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561796 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
425984
07:11:08.351999 IP (tos 0x0, ttl 64, id 31024, offset 0, flags [DF],  
proto TCP (6), length 6196) 192.168.254.249.nfs >  
192.168.254.254.2468561783: reply ok 6144 read REG 660 ids 1001/1006  
sz 3958224896 nlink 1 rdev 133/134414584 fsid 5e fileid 2013005 a/m/ 
ctime 1199479211.000000 1199479280.000000 1201346449.000000 32768 bytes
07:11:08.352014 IP (tos 0x0, ttl 64, id 13078, offset 0, flags [DF],  
proto TCP (6), length 6196) 192.168.254.249.nfs > 192.168.254.254.0:  
reply ok 6144
07:11:08.352074 IP (tos 0x0, ttl 64, id 9792, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561797 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
458752
07:11:08.352085 IP (tos 0x0, ttl 64, id 61207, offset 0, flags [DF],  
proto TCP (6), length 6196) 192.168.254.249.nfs > 192.168.254.254.0:  
reply ok 6144
07:11:08.352143 IP (tos 0x0, ttl 64, id 63816, offset 0, flags [DF],  
proto TCP (6), length 52) 192.168.254.254.49158 >  
192.168.254.249.nfsd: ., cksum 0x7f70 (incorrect (-> 0x8b18),  
24584:24584(0) ack 46013 win 65535 <nop,nop,timestamp 67511239 30749985>
07:11:08.352162 IP (tos 0x0, ttl 64, id 60945, offset 0, flags [DF],  
proto TCP (6), length 52) 192.168.254.254.49158 >  
192.168.254.249.nfsd: ., cksum 0x7f70 (incorrect (-> 0x8b18),  
24584:24584(0) ack 46013 win 65535 <nop,nop,timestamp 67511239 30749985>
07:11:08.352177 IP (tos 0x0, ttl 64, id 3701, offset 0, flags [DF],  
proto TCP (6), length 168) 192.168.254.254.2468561798 >  
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @  
491520
07:11:08.352198 IP (tos 0x0, ttl 64, id 3073, offset 0, flags [DF],  
proto TCP (6), length 6196) 192.168.254.249.nfs >  
192.168.254.254.3681401976: reply Unknown rpc response code=3722304989  
6144
07:11:08.352203 IP (tos 0x0, ttl 64, id 2122, offset 0, flags [DF],  
proto TCP (6), length 6196) 192.168.254.249.nfs >  
192.168.254.254.3067833788: reply Unknown rpc response code=3722304989  
6144
07:11:08.352207 IP (tos 0x0, ttl 64, id 61294, offset 0, flags [DF],  
proto TCP (6), length 2232) 192.168.254.249.nfs >  
192.168.254.254.3067833788: reply Unknown rpc response code=3722304989  
2180

There are strange errors:
192.168.254.254.3681401976: reply Unknown rpc response

Looks like server successfully sent first piece of data  
"07:11:08.351999...length 6196) 192.168.254.249.nfs >  
192.168.254.254.2468561783: reply ok 6144...32768 bytes" and then  
report about errors in the following packets:
length 6196) 192.168.254.249.nfs > 192.168.254.254.3681401976: reply  
Unknown rpc response code=3722304989 6144
length 6196) 192.168.254.249.nfs > 192.168.254.254.3067833788: reply  
Unknown rpc response code=3722304989 6144
length 2232) 192.168.254.249.nfs > 192.168.254.254.3067833788: reply  
Unknown rpc response code=3722304989 2180

There is a lot of such errors in tcpdump logs.

NFS3 specification do not contains this error code description...

p.s. And why server is limited with read/write data size 32760? Is  
there a possibility to change it?

More information about the freebsd-net mailing list