Interface routes and multiple fibs

From: Dan Mahoney <freebsd_at_gushi.org>
Date: Tue, 18 Oct 2022 21:11:14 UTC
All,

Maybe a question for the -net or -rc people.  If I should ask there, let me know.

I'm running with multiple fibs.  One fib is just box management, ssh, etc.  The other fib (which takes BGP routes from peers via BIRD) does DNS anycast things.  The DNS server runs in fib 1.  Our default route is added to both fibs.

My fib0 routing table looks like this:

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.159.249.233    UGS        bge0
127.0.0.1          link#5             UH          lo0
182.159.249.232/29 link#1             U          bge0
182.159.249.236    link#1             UHS         lo0

Fib 1 is missing that final route.:

default            182.159.249.233    UGS        bge0
127.0.0.1          link#5             UH          lo0
182.159.249.232/29 link#1             U          bge0

I've noticed that when I try to do a query (with dig) against it from fib 0, it sends over lo0 to the named process, but the reply packet just gets sent out ON BGE0, and is never received, since dig is listening on the interface it sent the packet over (lo0) to hear the response, which, near as I can tell with tcpdump -i bge0, just goes out on the wire

Obviously, we can add the static route to that second fib with:  route add -host 182.159.249.236 -interface lo0 -fib 1.

Yes, we can also make this stick useing default_routes in rc.conf.

But it feels like we shouldn't have to.  This feels like a glitch, and that if all fibs get the SUBNET route , they should also get the loopback.

-Dan