Allowing a local subnet route to change to a different ifnet

[Alan Somers]

On Wed, Jan 17, 2018 at 2:56 PM, Ryan Stone <rysto32 at gmail.com> wrote:

> I'm going to prefix this question by noting that I realize that the
> configuration that I am about to describe is quite nonsensical.
> Unfortunately, it seems that under older versions of FreeBSD (possibly
> FreeBSD 7-vintage), the configuration mostly "worked", and now at
> $WORK I am responsible for making the configuration continue to work
> in the future.  The customer in this case is completely unwilling to
> modify their configuration.
>
> I have a customer that has configured two different IPs on the same
> subnet on two different interfaces.  The behaviour that they want is
> that if the link on one of the two interfaces goes down, the route to
> that subnet will migrate to the other IP on the other interface as a
> quasi-failover behaviour.  Under FreeBSD 7, we had a daemon that
> accomplished this by detecting the link loss and then using "route
> change" to move the route to the up interface.  If the subnet in
> question was 192.168.1.0/24, for example, we could run "route change
> 192.1.68.1.0/24 -ifp em1" to migrate the route.
>
> Running on -head I run into two issues.  The first comes out of
> r264986, which changes the behaviour of RTM_CHANGE.  The code path
> changed significantly, but the part that impacts me is that now any
> RTM_CHANGE command with the gateway set NULL gets EINVAL immediately
> where previously it was allowed.  I've hacked around this problem
> locally for testing purposes but I really don't understand the code
> well enough at this point to see what a real fix would look like.
>
> The second issue is quite complex.  The root cause is that the routing
> code sets IFA_ROUTE on any ifaddr that has a local subnet route
> associated with it.  If I don't migrate this flag to the new ifaddr,
> very bizarre behaviour follows.  I've done a prototype that detects
> when this migration is needed in rtrequest1_fib_change() and it works,
> but IFA_ROUTE seems to otherwise be handled in individual L3 protocols
> like in and in6, so I'm worried that this is a layering violation.  My
> patch looks like this:
> https://people.freebsd.org/~rstone/patches/route-change-subnet.diff
>
>
> My first, and most important question, is whether a patch that would
> allow a subnet route to be migrated to a different interface be
> something that would be acceptable in FreeBSD?  If so, I need guidance
> on what a proper fix for both issues would look like so that I can
> implement fixes that I can upstream.
>
> Thanks,
> Ryan
>

I'm sorry for you Ryan; this sounds like a doozy.  I know you said that the
customer is unwilling to change, but would they consider using a lagg(4)
interface?  Using lagg with laggproto=failover is designed to solve exactly
this problem.  They wouldn't have to recable anything, and they could keep
their single IP address.  If not, you should see PR 189088; I think it's
related.

-Alan