asymetric speeds over gigE link

[Kip Macy]

One more thing to remember is to raise the socket size on the linux
sender. I found that I got very poor tx performance on linux running
netperf without making the socket buffers fairy large.

       -Kip

On 5/21/07, root <root at jim-liesl.org> wrote:
> Summary:  Using iperf to measure TCP net speed between a linux and
> freebsd box over gigE, I see significant speed difference depending on
> the data direction.  Pushing data from the freebsd box to the linux box,
> I average about 500Gb/s.  Pushing data from the linux box to the freebsd
> box, I see about 300Gb/s.  It would seem that there is something
> limiting either the transmit side of the linux box or the recv side of
> the freebsd box. Any suggestions on how to narrow down who's tuning
> needs more work would also be welcome.  Just for grins, I tried upping
> the txqueuelen on the linux box to 1500, but no help.
>
> I'd like to get closer to the theoretical speed of this link (I'd be
> happy if I could get 600-700 both ways), but the real puzzler is the
> difference.  The client is the linux box @192.168.1.104 and the iperf
> server is the freebsd box @ 192.168.1.15 .  Neither box is cpu or net
> busy during testing.  Both systems only have one nic.  I played with
> larger window sizes, but it only had a minor affect.  I went with the
> Intel Pro/1000's based on their good rep in both the linux and freebsd
> communities.
>
> Client connecting to 192.168.1.15, TCP port 5001
> TCP window size:   128 KByte (default)
> ------------------------------------------------------------
> [  6] local 192.168.1.104 port 34788 connected with 192.168.1.15 port 5001
> [  6]  0.0-10.0 sec    369 MBytes    309 Mbits/sec
> [  5] local 192.168.1.104 port 5001 connected with 192.168.1.15 port 52963
> [  5]  0.0-10.0 sec    597 MBytes    500 Mbits/sec
>
>
> Switch is the Netgear GS105 (5 port, supposedly wire speed, cables are
> Belkin 5e), both systems are on this switch.
>
> FreeBSD box:
> FreeBSD 6.1-RELEASE-p10
> 1.9Ghz Athlon / 1 gig of main mem
> Abit/nforce2 MB+chipset (onboard nic disabled in bios)
> Intel Pro/1000GT NIC
> sysctl.conf:
> kern.ipc.maxsockbuf=8192000
> net.inet.tcp.sendspace=262144
> net.inet.tcp.recvspace=262144
> sysctl reports:
> kern.ipc.nmbjumbo16: 0
> kern.ipc.nmbjumbo9: 0
> kern.ipc.nmbjumbop: 0
> kern.ipc.nmbclusters: 25600
> net.inet.tcp.rfc1323: 1
>
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         options=b<RXCSUM,TXCSUM,VLAN_MTU>
>         inet6 fe80::20e:cff:feda:1a3c%em0 prefixlen 64 scopeid 0x1
>         inet 192.168.1.15 netmask 0xffffff00 broadcast 192.168.1.255
>         ether 00:0e:0c:da:1a:3c
>         media: Ethernet autoselect (1000baseTX <full-duplex>)
>         status: active
>
> Linux box:
> Linux emperor 2.6.17-11-generic #2 SMP Tue Mar 13 23:32:38 UTC 2007 i686
> GNU/Linux
> Soyo Dragon+ MB/ 1.9 Ghz Athlon/ 1 gig main mem.
> I run  tcp_tune.sh at boot on the linux box
> #!/bin/bash
> echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_rmem
> echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_wmem
> echo "131072 262144 8192000" > /proc/sys/net/ipv4/tcp_mem
>
> /proc/sys/net/ipv4/tcp_window_scaling is 1
> /proc/sys/net/ipv4/tcp_timestamps is 1
>
> eth1      Link encap:Ethernet  HWaddr 00:0E:0C:DA:1A:3B
>           inet addr:192.168.1.104  Bcast:192.168.1.255  Mask:255.255.255.0
>           inet6 addr: fe80::20e:cff:feda:1a3b/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:8139539 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:5638407 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:147638720 (140.7 MiB)  TX bytes:1109677958 (1.0 GiB)
>           Base address:0xd800 Memory:e2020000-e2040000
>
> Settings for eth1:
>         Supported ports: [ TP ]
>         Supported link modes:   10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Full
>         Supports auto-negotiation: Yes
>         Advertised link modes:  10baseT/Half 10baseT/Full
>                                 100baseT/Half 100baseT/Full
>                                 1000baseT/Full
>         Advertised auto-negotiation: Yes
>         Speed: 1000Mb/s
>         Duplex: Full
>         Port: Twisted Pair
>         PHYAD: 0
>         Transceiver: internal
>         Auto-negotiation: on
>         Supports Wake-on: umbg
>         Wake-on: g
>         Current message level: 0x00000007 (7)
>         Link detected: yes
>
>
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>