svn commit: r292379 - in head/sys: netinet netinet6

[Steven Hartland]

On 17/12/2015 19:16, Gleb Smirnoff wrote:
>    Steven,
>
> On Thu, Dec 17, 2015 at 02:29:02PM +0000, Steven Hartland wrote:
> S> I would definitely like to understand more about your concerns and learn
> S> from
> S> your knowledge in this area, so thanks for that offer, and while it does
> S> sound
> S> unforgiving I totally understand where you're coming from.
> S>
> S> Hopefully together we can bring this to a satisfactory conclusion as I
> S> would hate
> S> for both carp and lagg to stay as broken, 2 years is long enough :D
>
> Ok, let's get technical. CARP and LAGG were not broken for 2 years. They
> were working very well in the way they were designed to work. The setup
> in the bug 156226 was broken initially.
You may have not read all the detail in the review so you might not have 
noticed that I
identified that carp IPv6 NA was broken by r251584 which was committed 2 1/2
years ago. I'm guessing not may people use it for IPv6.
> The "link aggregation" itself refers to an aggregation of links between
> two logical devices. If you build lagg(4) interface on top of two ports
> that are plugged into different switches, you are calling for trouble.

While multiple switches complicates the matter its not the only issue as 
you can
reproduce this with a single switch and two nics in LAGG failover mode 
with a simple
ifconfig <nic1> down. At this point any traffic entering the switch for 
LAGG member
will back-whole instead of being received by the other nic.

It is much more common in networking now to have multiple physical switches
configured as part of bigger logical devices using protocols such as 
MLAG, which is
what we're using with Cisco's and Arista's, so not some cheepo network ;-)

> All comments in the 156226 from Eugene Grosbein are valid. I would not
> repeat them, but ask you to reread them in bugzilla. There was a good
> reason why for 2 years committers stayed away from this "bug" and related
> patch.
Yes but not confuse the different types, we're talking specifically 
about failover mode
here which has no special configuration hence its reliant on the OS 
implementation
only.
> Nevertheless, someone wants to give a kick to this initially broken
> network design and run it somehow. And this "somehow" implies Layer2
> upcalling into upper layers to do something, since there is no
> established standard layer2 heartbeat packet. I have chatted with
> networking gurus at my job, and they said, that they don't know
> any decent network equipment that supports such setup. However, they
> noticed that Windows is capable for such failover. I haven't yet
> learned on how Windows solves the problem. Actually, those who
> pushed committing 156226 should have done these investigations.
> Probably Windows does exactly the same, sends gratutious ARP or
> its IPv6 analog. Or may be does something better like sending
> useless L2 datagram, but with a proper source hardware address.
Actually our testing here showed both Windows and Linux worked as 
expected and
from my reading doing the GARP / UNA is actually expected in this 
situation, for this
very reason.
> Okay, what if we want same in FreeBSD as in Windows? Should we do the
> following list of evil things:
>
> - put DELAY in context of callout(or in context of any network processing)
> - introduce new notions of a link state, or new KPI for link handling
>    Note that link handling KPI was stable for iver 10 years and satisfied
>    all the different types of interfaces we support
> - create new interface methods
> - call into address families supplying an ifnet that doesn't have this AF
>    instantiated, and then to fix immediate panic putting there a kludge
>    of "if (foo == NULL) return;"
> - etc...
>
> Sorry, I'm putting "etc" here, because tires on details. You would agree
> that the whole process of fixing the "bug" was overcoming the problems
> that the network stack is not designed for the things that you are
> willing to do. Won't you agree?

I am indeed trying to produce feature parity, to prevent the powers that 
be throwing
FreeBSD out as the only OS which fails to work as expected in failover 
mode, even in the
simple case as described above.

Yes we could apply user land work around but then everyone has to be 
aware its need
and to set it up which doesn't sound like the best solution.

> Or should we just write a tiny program, that would observe state of
> networking ports, and if a port changes state then send a tiny packet
> as a bpf(4) write?
This could be done but still means our lagg failover doesn't do what 
people would expect.

I'd like to step back for a second and get you feedback on the changes 
that where
reverted, which didn't have the DELAY in the callout. What where the 
issues as you
saw them? So we don't spam people any more I've reopened the review so 
we can
take this there: https://reviews.freebsd.org/D4111

Apologies if these are very obvious to others but clearly those involved 
with this
didn't spot them so it would be really nice to learn from this.

     Regards
     Steve