Re: Custom static route not applied at reboot

Well, I was going to say "thanks a million, that worked great" (and, in
fact, did but from the wrong email address so it bounced).   But then a
weird thing happened.

When I followed Jon's advice and added the -gateway to rc.conf it seemed to
work.  I rebooted, the route was there.  Great.  I starting pinging the
other side.  A few pings in it stopped.  In another console I looked and
the route had disappeared again.  I have heard that when an interface goes
down/up it loses its routes.  I do see the interface (igb0) flapping around
in /var/log/messages:

Nov 26 14:53:04 hero kernel: igb0: <Intel(R) PRO/1000 PCI-Express Network
Driver> port 0xd000-0xd01f mem 0xfc600000-0xfc61ffff,0xfc620000-0xfc623fff
at device 0.0 on pci4
Nov 26 14:53:04 hero kernel: igb0: Using 1024 TX descriptors and 1024 RX
descriptors
Nov 26 14:53:04 hero kernel: igb0: Using 2 RX queues 2 TX queues
Nov 26 14:53:04 hero kernel: igb0: Using MSI-X interrupts with 3 vectors
Nov 26 14:53:04 hero kernel: igb0: Ethernet address: 04:42:1a:06:b8:84
Nov 26 14:53:04 hero kernel: igb0: netmap queues/slots: TX 2/1024, RX 2/1024
...
Nov 26 14:53:04 hero kernel: bridge0: Ethernet address: 58:9c:fc:00:40:4a
Nov 26 14:53:04 hero kernel: lo0: link state changed to UP
Nov 26 14:53:04 hero kernel: igb0: promiscuous mode enabled
Nov 26 14:53:04 hero kernel: bridge0: link state changed to UP
Nov 26 14:53:04 hero kernel: pflog0: promiscuous mode enabled
*Nov 26 14:53:04 hero kernel: igb0: link state changed to UP*
... ntpd ...
Nov 26 14:53:24 hero kernel: epair0a: Ethernet address: 02:a5:63:ce:6f:0a
Nov 26 14:53:24 hero kernel: epair0b: Ethernet address: 02:a5:63:ce:6f:0b
Nov 26 14:53:24 hero kernel: epair0a: link state changed to UP
Nov 26 14:53:24 hero kernel: epair0b: link state changed to UP
Nov 26 14:53:24 hero kernel: epair0a: changing name to 'vnet0.1'
*Nov 26 14:53:25 hero kernel: igb0: link state changed to DOWN*
Nov 26 14:53:25 hero kernel: vnet0.1: promiscuous mode enabled
Nov 26 14:53:25 hero kernel: lo0: link state changed to UP
Nov 26 14:53:29 hero kernel: lo1: link state changed to UP
*Nov 26 14:53:29 hero kernel: igb0: link state changed to UP*
Nov 26 14:53:29 hero kernel: epair1a: Ethernet address: 02:1b:17:53:71:0a
Nov 26 14:53:29 hero kernel: epair1b: Ethernet address: 02:1b:17:53:71:0b
Nov 26 14:53:29 hero kernel: epair1a: link state changed to UP
Nov 26 14:53:29 hero kernel: epair1b: link state changed to UP
Nov 26 14:53:29 hero kernel: epair1a: changing name to 'vnet0.4'
Nov 26 14:53:29 hero kernel: epair1b: changing name to 'epair0b'
Nov 26 14:53:29 hero kernel: vnet0.4: promiscuous mode enabled
Nov 26 14:53:29 hero kernel: lo0: link state changed to UP
Nov 26 14:53:31 hero kernel: pflog0: promiscuous mode enabled
Nov 26 14:53:31 hero kernel: tun0: link state changed to UP
Nov 26 14:53:35 hero dhclient[3919]: New IP Address (igb0): 10.0.0.208
Nov 26 14:53:35 hero dhclient[3923]: New Subnet Mask (igb0): 255.255.255.0
Nov 26 14:53:35 hero dhclient[3927]: New Broadcast Address (igb0):
10.0.0.255
Nov 26 14:53:35 hero dhclient[3931]: New Routers (igb0): 10.0.0.1

*// note: lines below here coming in via remote syslog from the vnet jail
as openvpn starts up*

Nov 26 14:54:04 10.0.0.225 openvpn[89305]: PUSH: Received control message:
'PUSH_REPLY,route 10.0.0.0 255.255.255.0,compress lz4-v2,route-gateway
10.8.0.1,topology subnet,ping 10,ping-restart 120,route 10.0.
0.0 255.255.255.0,ifconfig 10.8.0.8 255.255.255.0,peer-id 0,cipher
AES-256-GCM'
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: OPTIONS IMPORT: timers and/or
timeouts modified
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: OPTIONS IMPORT: compression
parms modified
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: OPTIONS IMPORT: --ifconfig/up
options modified
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: OPTIONS IMPORT: route options
modified
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: OPTIONS IMPORT: route-related
options modified
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: OPTIONS IMPORT: peer-id set
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: OPTIONS IMPORT: adjusting
link_mtu to 1625
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: OPTIONS IMPORT: data channel
crypto options modified
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: Data Channel: using negotiated
cipher 'AES-256-GCM'
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: Outgoing Data Channel: Cipher
'AES-256-GCM' initialized with 256 bit key
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: Incoming Data Channel: Cipher
'AES-256-GCM' initialized with 256 bit key
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: Preserving previous TUN/TAP
instance: tun0
Nov 26 14:54:04 10.0.0.225 openvpn[89305]: Initialization Sequence Completed


Is it possible that the route is applied and then vanishes when that
interface goes up/down?

I have not played with bridging before -- this is my first jail with vnet.
Does all of this stuff about epair[0-1][a-b] and vnet* look reasonable
given this config (rc.conf):

cloned_interfaces="${cloned_interfaces} lo1 bridge0"
ifconfig_bridge0="addm igb0 up"


Have I perhaps messed something up when setting up the vnet?  It "works"
fine once this route is applied and it stays put.

Thx,
Scott




On Fri, Nov 26, 2021 at 2:01 PM Jon Radel <jon@radel.com> wrote:

> On 11/26/21 13:45, Scott Gasch wrote:
> > Hi,
> >
> > I have a FreeBSD 13.0-RELEASE-p4 system that runs openvpn in a vnet jail
> to
> > create a site-to-site VPN.  This works great except for one detail: I
> want
> > the host system to add a static route to send traffic for the other side
> of
> > the VPN to the IP address of the vnet jail to use it as a gateway.  I
> tried
> > this in rc.conf:
> >
> > static_routes="vpn"
> > route_vpn="-net 192.168.0.0/24 10.0.0.225"
> >
> The man page for rc.conf on my 13.0-whatever system seems to say that
>
> static_routes="vpn"
> route_vpn="-net 192.168.0.0/24 -gateway 10.0.0.225"
>
> might work better.  While the details blur after this many versions of
> FreeBSD (and OpenBSD) I vaguely recall learning long ago that even when
> the man page says something like "whose contents will later be passed to
> a 'route add' operation," sometimes it doesn't actually mean that the
> literal string will be passed as is to the mentioned command, but rather
> that the appropriate values will be picked out of the string in rc.conf
> and something useful done with them.
>
>
> --
> --Jon Radel
> jon@radel.com
>
>