Gigabit ethernet questions?

[Dima Roshin]

Thanks Jon, I did it on both sides, thats much better now:

gate1# sysctl kern.polling.idle_poll=1
kern.polling.idle_poll: 0 -> 1
gate1# sysctl net.inet.ip.fastforwarding
net.inet.ip.fastforwarding: 0
gate1# sysctl net.inet.ip.fastforwarding=1
net.inet.ip.fastforwarding: 0 -> 1
gate1# iperf -c 192.168.0.2 -N -w 196000
------------------------------------------------------------
Client connecting to 192.168.0.2, TCP port 5001
TCP window size:   192 KByte (WARNING: requested   191 KByte)
------------------------------------------------------------
[  3] local 192.168.0.1 port 63941 connected with 192.168.0.2 port 5001
[  3]  0.0-10.0 sec    762 MBytes    639 Mbits/sec

But there is still some bottleneck, and I can't understand where.



-----Original Message-----
From: Jon Otterholm <jon.otterholm at ide.resurscentrum.se>
To: Dima Roshin <roshind at mail.ru>
Date: Wed, 09 Aug 2006 13:54:18 +0200
Subject: Re: Gigabit ethernet questions?

> 
> Dima Roshin wrote:
> >  Greeting colleagues. I've got two DL-360(pciX bus) servers, with BCM5704 NetXtreme Dual Gigabit Adapters(bge). The Uname is 6.1-RELEASE-p3. The bge interfaces of the both servers are connected with each other with a cat6 patchcord.
> >    Here are my settings:
> > kernel config:
> > options DEVICE_POLLING 
> > options HZ=1000 #
> >
> > sysctl.conf:
> > kern.polling.enable=1
> > net.inet.ip.intr_queue_maxlen=5000
> > kern.ipc.maxsockbuf=8388608
> > net.inet.tcp.sendspace=3217968
> > net.inet.tcp.recvspace=3217968
> > net.inet.tcp.rfc1323=1
> >
> > bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 9000
> >         options=5b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,POLLING>
> >         inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
> >         ether 00:17:a4:3a:e1:81
> >         media: Ethernet autoselect (1000baseTX <full-duplex>)
> >         status: active
> > (note mtu 9000)
> >
> > and here are tests results:
> >
> > netperf:
> >
> > TCP STREAM TEST to 192.168.0.1
> > Recv   Send    Send
> > Socket Socket  Message  Elapsed
> > Size   Size    Size     Time     Throughput
> > bytes  bytes   bytes    secs.    10^6bits/sec
> >
> > 6217968 6217968 6217968    10.22     320.04
> >
> > UDP UNIDIRECTIONAL SEND TEST to 192.168.0.1
> > Socket  Message  Elapsed      Messages
> > Size    Size     Time         Okay Errors   Throughput
> > bytes   bytes    secs            #      #   10^6bits/sec
> >
> >   9216    9216   10.00      118851 1724281     876.20
> > 41600           10.00           0              0.00]
> >
> >
> >
> > iperf:
> > gate2# iperf -s -N
> > ------------------------------------------------------------
> > Server listening on TCP port 5001
> > TCP window size: 3.07 MByte (default)
> > ------------------------------------------------------------
> > [  4] local 192.168.0.2 port 5001 connected with 192.168.0.1 port 52597
> > [  4]  0.0-10.1 sec    384 MBytes    319 Mbits/sec
> >
> > Also I can say, that I've managed to achieve about 500mbit.s by tuning tcp window with -w key in iperf.
> >
> > How can we explain such a low tcp performance? What else is to tune? Is there somebody who achieved gigabit speed with tcp on freebsd?
> >
> > Thanks.
> >
> >
> >
> > _______________________________________________
> > freebsd-net at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
> >   
> You also need kern.polling.idle_poll=1 and maybe 
> net.inet.ip.fastforwarding=1 though I never noticed any difference with 
> that one enabled. I got about 840Mbit/s routed through a dell 1850 
> (EMT64 running AMD64) with em-interfaces (I only used one physical IF 
> though with 2 VLAN-if).
> 
>