Flow ID, LACP, and igb

[Adrian Chadd]

Yo,

LRO is an interesting hack that seems to do a good trick of hiding the
ridiculous locking and unfriendly cache behaviour that we do per-packet.

It helps with LAN test traffic where things are going out in batches from
the TCP layer so the RX layer "sees" these frames in-order and can do LRO.
When you disable it, I don't easily get 10GE LAN TCP performance. That has
to be fixed. Given how fast the CPU cores, bus interconnect and memory
interconnects are, I don't think there should be any reason why we can't
hit 10GE traffic on a LAN with LRO disabled (in both software and hardware.)

Now that I have the PMC sandy bridge stuff working right (but no PEBS, I
have to talk to Intel about that in a bit more detail before I think about
hacking that in) we can get actual live information about this stuff. But
the last time I looked, there's just too much per-packet latency going on.
The root cause looks like it's a toss up between scheduling, locking and
just lots of code running to completion per-frame. As I said, that all has
to die somehow.

2c,



-adrian



On 1 September 2013 08:45, Barney Cordoba <barney_cordoba at yahoo.com> wrote:

>
>
> Comcast sends packets OOO. With any decent number of internet hops you're
> likely to encounter a load
> balancer or packet shaper that sends packets OOO, so you just can't be
> worried about it. In fact, your
> designs MUST work with OOO packets.
>
> Getting balance on your load balanced lines is certainly a bigger upside
> than the additional CPU used.
> You can buy a faster processor for your "stack" for a lot less than you
> can buy bandwidth.
>
> Frankly my opinion of LRO is that it's a science project suitable for labs
> only. It's a trick to get more bandwidth
> than your bus capacity; the answer is to not run PCIe2 if you need pcie3.
> You can use it internally if you have
> control of all of the machines. When I modify a driver the first thing
> that I do is rip it out.
>
> BC
>
>
> ________________________________
>  From: Luigi Rizzo <rizzo at iet.unipi.it>
> To: Barney Cordoba <barney_cordoba at yahoo.com>
> Cc: Andre Oppermann <andre at freebsd.org>; Alan Somers <asomers at freebsd.org>;
> "net at freebsd.org" <net at freebsd.org>; Jack F Vogel <jfv at freebsd.org>;
> Justin T. Gibbs <gibbs at freebsd.org>; T.C. Gubatayao <
> tgubatayao at barracuda.com>
> Sent: Saturday, August 31, 2013 10:27 PM
> Subject: Re: Flow ID, LACP, and igb
>
>
> On Sun, Sep 1, 2013 at 4:15 AM, Barney Cordoba <barney_cordoba at yahoo.com
> >wrote:
>
> > ...
> >
>
> [your point on testing with realistic assumptions is surely a valid one]
>
>
> >
> > Of course there's nothing really wrong with OOO packets. We had this
> > discussion before; lots of people
> > have round robin dual homing without any ill effects. It's just not an
> > issue.
> >
>
> It depends on where you are.
> It may not be an issue if the reordering is not large enough to
> trigger retransmissions, but even then it is annoying as it causes
> more work in the endpoint -- it prevents LRO from working, and even
> on the host stack it takes more work to sort where an out of order
> segment goes than appending an in-order one to the socket buffer.
>
> cheers
> luigi
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>