Bridge Interfaces and ARPs

[Aleksandr A Babaylov]

On Thu, Dec 03, 2015 at 08:54:10AM -0500, Jason Van Patten wrote:
> Hey gang -
> 
> I posted this to the FreeBSD user forums but figured I'd send a message 
> off to the list to see if anyone has any input, guidance, or ideas. 
> Emailing diagrams around isn't good form (IMHO) but having a diagram 
> handy will help with the discussion.  So please glance at:
> 
> http://pics.lateapex.net/vz.png
> 
> Background: I have a business class Verizon FIOS connection for Internet 
> at home.  Along with that connection, I have 13 (not 14!) static IPs 
> from VZ.  They almost fall within a proper CIDR block, but not quite: 
> 1.2.3.210 - 1.2.3.222.  I don't own .209, so I can't claim 1.2.3.208/28 
> as my IP block (dammit!)  The subnet for the static IPs is a /24, and 
> the default route is *Verizon's* router: 1.2.3.1.
> 
> There are a number of different choices for this network layout: DMZ, 
> bridging, or binat.  I chose bridging so that I don't have the 
> complexity of binatting, and yet have some protection for the servers 
> via my router.  So, per the drawing, the FreeBSD router's em0 is 
> connected to the Verizon equipment, while re0 and re1 are both connected 
> to a managed Cisco switch, on different VLANs.
> 
> VLAN 10 for re0: Public IPs (public services, etc)
> VLAN 20 for re1: Private IPs (NAS, wireless AP, etc)
> 
> Via the router, VLAN 10 and Verizon's network are bridged together.  The 
> bridge interface on the router has IP: 1.2.3.222/24 with a default route 
> set to 1.2.3.1.  All servers on VLAN 10 have IPs within the allocated 
> range (.210 - .220) and the same default route.
> 
> Now: the problem.  I used the LAGG'd server as an example in the 
> diagram, but the same thing is happening with other servers: the router 
> is learning ARP entries for the IPs I own *from* Verizon's router.  As 
> soon as the router caches that bad entry, it no longer routes traffic to 
> those public IPs *from* VLAN 20 (private side).  So, in other words, a 
> laptop on the wireless network won't be able to get to 1.2.3.215.
> 
> My work-around for now has been a series of static ARP entries on the 
> router for each of my public servers.  That seems to work fine, but I 
> wonder if there's something I might be doing wrong?
> 
> If I didn't include enough info, fire away.  Thanks!

May be it is proxy arp from Verison.
Just delete bridge0

ifconfig em0 inet 1.2.3.222/24
ifconfig re0 inet 127.127.127.127/24 or any other fake net

route add 1.2.3.210/31 -iface re0
route add 1.2.3.212/30 -iface re0
route add 1.2.3.216/30 -iface re0
route add 1.2.3.220/31 -iface re0

sysctl net.link.ether.inet.proxyall=1

Default router for your public servers 1.2.3.222
in /28 or wider net.