Allowing a local subnet route to change to a different ifnet

[Rodney W. Grimes]

> I'm going to prefix this question by noting that I realize that the
> configuration that I am about to describe is quite nonsensical.
> Unfortunately, it seems that under older versions of FreeBSD (possibly
> FreeBSD 7-vintage), the configuration mostly "worked", and now at
> $WORK I am responsible for making the configuration continue to work
> in the future.  The customer in this case is completely unwilling to
> modify their configuration.
> 
> I have a customer that has configured two different IPs on the same
> subnet on two different interfaces.  The behaviour that they want is
> that if the link on one of the two interfaces goes down, the route to
> that subnet will migrate to the other IP on the other interface as a
> quasi-failover behaviour.  Under FreeBSD 7, we had a daemon that
> accomplished this by detecting the link loss and then using "route
> change" to move the route to the up interface.  If the subnet in
> question was 192.168.1.0/24, for example, we could run "route change
> 192.1.68.1.0/24 -ifp em1" to migrate the route.

"route change 192.1.68.1.0/24 -ifp em1" does not appear to be
valid syntax to me, -ifp is not a route option?
Did you mean -interface?

And I think this well work if you use the local IP of the em1 interface
for the gateway?  That should get past the null check

> 
> Running on -head I run into two issues.  The first comes out of
> r264986, which changes the behaviour of RTM_CHANGE.  The code path
> changed significantly, but the part that impacts me is that now any
> RTM_CHANGE command with the gateway set NULL gets EINVAL immediately
> where previously it was allowed.  I've hacked around this problem
> locally for testing purposes but I really don't understand the code
> well enough at this point to see what a real fix would look like.
> 
> The second issue is quite complex.  The root cause is that the routing
> code sets IFA_ROUTE on any ifaddr that has a local subnet route
> associated with it.  If I don't migrate this flag to the new ifaddr,
> very bizarre behaviour follows.  I've done a prototype that detects
> when this migration is needed in rtrequest1_fib_change() and it works,
> but IFA_ROUTE seems to otherwise be handled in individual L3 protocols
> like in and in6, so I'm worried that this is a layering violation.  My
> patch looks like this:
> https://people.freebsd.org/~rstone/patches/route-change-subnet.diff
> 
> 
> My first, and most important question, is whether a patch that would
> allow a subnet route to be migrated to a different interface be
> something that would be acceptable in FreeBSD?  If so, I need guidance
> on what a proper fix for both issues would look like so that I can
> implement fixes that I can upstream.

>From a fundemental standpoint this should work,
that it is now broken is a regression that needs fixed.

You may also run into issues with the kernel
maintain_loopback_route functions.

> Thanks,
> Ryan

-- 
Rod Grimes                                                 rgrimes at freebsd.org