Bad performance of 7.0 nfs client with Solaris nfs server
Eric Anderson
anderson at freebsd.org
Wed Feb 20 13:06:18 UTC 2008
Valerio Daelli wrote:
> On Feb 19, 2008 8:53 PM, Kris Kennaway <kris at freebsd.org> wrote:
>> Valerio Daelli wrote:
>>> Hi list
>>>
>>> we have a FreeBSD 7.0 NFS client (csup today, built world and kernel).
>>> It mounts a Solaris 10 NFS share.
>>> We have bad performance with 7.0 (3MB/s).
>>> We have tried both UDP and TCP mounts, both sync and async.
>>> This is our mount:
>>>
>>> nest.xx.xx:/data/export/hosts/bsd7.xx.xx/ /mnt/nest.xx.xx nfs
>>> noatime,async,-i,rw,-T,-3
>>>
>>> Both our server (7.0 and Solaris 10) are Gigabit Ethernet, both are HP
>>> Proliant DL360 i386 (NIC bge0):
>>>
>>> ---
>>> FreeBSD 7.0:
>>>
>>> [eldon at bsd7 ~]$ uname -a
>>> FreeBSD bsd7.xx.xx 7.0-RC2 FreeBSD 7.0-RC2 #1: Mon Feb 18 17:46:46 CET
>>> 2008 root at bsd7.xx.xx:/usr/obj/usr/src/sys/BSD7 i386
>>>
>>> ---
>>> This is our performance with iozone:
>>> command line:
>>>
>>> iozone -+q 1 -i 0 -i 1 -n 2048 -g 2G -Raceb iozone.xls -f
>>> /mnt/nest.xx.xx/iozone.bsd7/iozone.tmp
>>>
>>> FreeBSD 7:
>>>
>>> File stride size set to 17 * record size.
>>> random
>>> random bkwd record stride
>>> KB reclen write rewrite read reread read
>>> write read rewrite read fwrite frewrite fread freread
>>> 2048 1024 109883 101289 769058 779880
>>> 2048 2048 3812 3674 760479 767066
>>> 4096 1024 111156 106788 724692 728040
>>> 4096 2048 3336 2241 157132 733417
>>> 4096 4096 2829 3364 699351 699807
>>>
>>>
>>> As you can see, while with record length less than 1024KB the speed is
>>> 'fast', with record of 2048 or more (I've tried with record much
>>> bigger) we get only
>>> 3 MB/s.
>>> Is this a known issue? If you need more details please contact me, I
>>> am willing to do more tests to risolve this problem.
>> Can you characterize what is different about the NFS traffic when
>> FreeBSD and Solaris are the clients? Does other network traffic to the
>> server perform well? 2048 bytes records are > the 1500 byte MTU, so you
>> will be invoking packet fragmentation and reassembly. Maybe you are
>> getting packet loss for some reason. What does netstat say about
>> interface errors on the bge, and for the protocol layers?
>>
>> Follow-ups set to performance@
>>
>> Kris
>>
>
> Hi
> thanks for responding. I did a couple of test on 7.0, and I've not
> seen error on interfaces.
> Attached are netstat -s ad netstat -i.
>
>
> TCP mount
> ---
> root at bsd7:~ netstat -i
> Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
> bge0 1500 <Link#1> 00:0b:cd:37:45:e8 45444 0 110188 0 0
> bge0 1500 xx.xx.xx.xx bsd7 45427 - 110183 - -
> ---tcp:
> 1536 packets sent
> 396 data packets (68516 bytes)
> 0 data packets (0 bytes) retransmitted
> 0 data packets unnecessarily retransmitted
> 0 resends initiated by MTU discovery
> 1021 ack-only packets (6 delayed)
> 0 URG only packets
> 0 window probe packets
> 101 window update packets
> 18 control packets
> 2555 packets received
> 413 acks (for 68487 bytes)
> 1 duplicate ack
> 0 acks for unsent data
> 2173 packets (763594 bytes) received in-sequence
> 0 completely duplicate packets (0 bytes)
> 0 old duplicate packets
> 0 packets with some dup. data (0 bytes duped)
> 0 out-of-order packets (0 bytes)
> 0 packets (0 bytes) of data after window
> 0 window probes
> 0 window update packets
> 7 packets received after close
> 0 discarded for bad checksums
> 0 discarded for bad header offset fields
> 0 discarded because packet too short
> 0 discarded due to memory problems
> 10 connection requests
> 1 connection accept
> 0 bad connection attempts
> 0 listen queue overflows
> 0 ignored RSTs in the windows
> 11 connections established (including accepts)
> 10 connections closed (including 1 drop)
> 7 connections updated cached RTT on close
> 7 connections updated cached RTT variance on close
> 0 connections updated cached ssthresh on close
> 0 embryonic connections dropped
> 413 segments updated rtt (of 358 attempts)
> 0 retransmit timeouts
> 0 connections dropped by rexmit timeout
> 0 persist timeouts
> 0 connections dropped by persist timeout
> 0 Connections (fin_wait_2) dropped because of timeout
> 0 keepalive timeouts
> 0 keepalive probes sent
> 0 connections dropped by keepalive
> 0 correct ACK header predictions
> 2122 correct data packet header predictions
> 1 syncache entrie added
> 0 retransmitted
> 0 dupsyn
> 0 dropped
> 1 completed
> 0 bucket overflow
> 0 cache overflow
> 0 reset
> 0 stale
> 0 aborted
> 0 badack
> 0 unreach
> 0 zone failures
> 1 cookie sent
> 0 cookies received
> 0 SACK recovery episodes
> 0 segment rexmits in SACK recovery episodes
> 0 byte rexmits in SACK recovery episodes
> 0 SACK options (SACK blocks) received
> 0 SACK options (SACK blocks) sent
> 0 SACK scoreboard overflow
> [...]
> ip:
> 99904 total packets received
> 0 bad header checksums
> 0 with size smaller than minimum
> 0 with data size < data length
> 0 with ip length > max ip packet size
> 0 with header length < data size
> 0 with data length < header length
> 0 with bad options
> 0 with incorrect version number
> 61441 fragments received
> 0 fragments dropped (dup or out of space)
> 1 fragment dropped after timeout
> 10240 packets reassembled ok
> 48703 packets for this host
> 0 packets for unknown/unsupported protocol
> 0 packets forwarded (0 packets fast forwarded)
> 0 packets not forwardable
> 0 packets received for unknown multicast group
> 0 redirects sent
> 47693 packets sent from this host
> 0 packets sent with fabricated ip header
> 0 output packets dropped due to no bufs, etc.
> 0 output packets discarded due to no route
> 34819 output datagrams fragmented
> 208914 fragments created
> 0 datagrams that can't be fragmented
> 0 tunneling packets that can't find gif
> 0 datagrams with bad address in header
>
> [...]
>
>
> UDP mount
> ---
> root at bsd7:~ netstat -i
> Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
> bge0 1500 <Link#1> 00:0b:cd:37:45:e8 63685 1 119978 0 0
> bge0 1500 xx.xx.xx.xx bsd7 63628 - 119973 - -
> ---
> udp:
> 25880 datagrams received
> 0 with incomplete header
> 0 with bad data length field
> 0 with bad checksum
> 0 with no checksum
> 0 dropped due to no socket
> 7 broadcast/multicast datagrams undelivered
> 0 dropped due to full socket buffers
> 0 not for hashed pcb
> 25873 delivered
> 25882 datagrams output
> 0 times multicast source filter matched
> [...]
> ip:
> 62379 total packets received
> 0 bad header checksums
> 0 with size smaller than minimum
> 0 with data size < data length
> 0 with ip length > max ip packet size
> 0 with header length < data size
> 0 with data length < header length
> 0 with bad options
> 0 with incorrect version number
> 41473 fragments received
> 0 fragments dropped (dup or out of space)
> 1 fragment dropped after timeout
> 6912 packets reassembled ok
> 27818 packets for this host
> 0 packets for unknown/unsupported protocol
> 0 packets forwarded (0 packets fast forwarded)
> 0 packets not forwardable
> 0 packets received for unknown multicast group
> 0 redirects sent
> 27056 packets sent from this host
> 0 packets sent with fabricated ip header
> 0 output packets dropped due to no bufs, etc.
> 0 output packets discarded due to no route
> 18438 output datagrams fragmented
> 110628 fragments created
> 0 datagrams that can't be fragmented
> 0 tunneling packets that can't find gif
> 0 datagrams with bad address in header
> [...]
>
> Then I tried with different versions of FreeBSD and I discovered that:
> -FreeBSD 6.2-p3 has good performance with reclen 2048 and bad with
> 4096 and 8192:
>
> ---
> root at litio:~ uname -a
> FreeBSD litio.xx.xx 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #1: Thu Mar
> 15 14:52:08 CET 2007 root at litio.xx.xx:/usr/obj/usr/src/sys/LITIO
> amd64
>
> TCP
> KB reclen write rewrite read reread read
> write read rewrite read fwrite frewrite fread freread
> 2048 1024 90041 97352 82001 1019428
> 2048 2048 87030 88329 1009366 1053046
> 4096 1024 96139 95218 74186 1032306
> 4096 2048 103038 99700 79950 1003190
> 4096 4096 2897 3354 970821 522038
> 8192 1024 97978 100020 80681 946699
> 8192 2048 95260 104290 82395 1032127
> 8192 4096 3300 3169 149505 860882
> 8192 8192 3371 3090 998300 1012987
> 16384 1024 100072 98446 80503 962737
> 16384 2048 99360 105249 80983 924453
> 16384 4096 3162 3173 106750 936178
> 16384 8192 3233 3223 150394 998913
> 16384 16384 3250 3316 955004 944153
>
> UDP similar
>
>
> -FreeBSD 6.2-p4 and 6.2-p6 has the same performance as 7.0:
> ---
>
> root at xtl:~ uname -a
> FreeBSD xtl.xx.xx 6.2-RELEASE-p6 FreeBSD 6.2-RELEASE-p6 #0: Wed Jul 25
> 14:32:07 CEST 2007 root at xtl.xx.xx:/usr/obj/usr/src/sys/XTL i386
>
> TCP
> random
> random bkwd record stride
> KB reclen write rewrite read reread read
> write read rewrite read fwrite frewrite fread freread
> 2048 1024 15544 14560 846597 841044
> 2048 2048 2769 2649 852225 851211
> 4096 1024 17366 16851 833229 835417
> 4096 2048 2777 2457 49038 841226
> 4096 4096 2527 2813 847868 847492
>
>
>
>
> root at rubidio:~ uname -a
> FreeBSD rubidio.xx.xx 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #0: Tue
> May 8 11:24:10 CEST 2007
> root at rubidio.xx.xx:/usr/obj/usr/src/sys/PROXYCARP1 amd64
>
> TCP
>
> random
> random bkwd record stride
> KB reclen write rewrite read reread read
> write read rewrite read fwrite frewrite fread freread
> 2048 1024 93010 89366 996997 970635
> 2048 2048 94740 90820 943347 946048
> 4096 2048 100036 94668 81065 927548
> 4096 4096 3274 3319 1074493 1047309
> 8192 2048 100990 98212 80574 1054050
> 8192 4096 3078 3071 149890 990428
> 8192 8192 3074 3055 992029 995391
>
>
>
> ---
> Now I will try with a Solaris client and with rsync as protocol.
> Other suggestions?
> Bye
>
If possible, it might help a lot to send the output (or a snippet of it)
or post it somewhere on the web, of a tshark output from the client. You
might try UDP mounting the NFS export also, just to try it.
Eric
More information about the freebsd-performance
mailing list