FreeBSD responding with wrong receiving interface IP

[George Neville-Neil]

On 10 Mar 2015, at 11:26, Paul S. wrote:

> Hi,
>
> I've been deploying FreeBSD as customer edge routers for customers 
> with sites that do not require high throughput (>1g/s).
>
> Each site has two ISPs (Mostly Telstra + Verizon/Optus), and take full 
> routes via OpenBGPd and BIRD. I use next-hop self on all received 
> routes.
>
> The FreeBSD boxes have static routes delegating the announced IP 
> blocks to a L3 switch down the road. i.e: route add -net 10.100.1.0/24 
> 10.0.0.1, and then that /24 is originated via BGP to both upstreams.
>
> Things in general work fine, but I've been receiving reports of 'weird 
> traceroute results' from my customers.
>
> Examples of this would be,
>
> 1 some.random.isp (...) (...)
> 2  gigabitethernet3-3.exi1.melbourne.telstra.net (203.50.77.49) 0.309 
> ms  0.284 ms  0.227 ms
> 3  bundle-ether3-100.exi-core10.melbourne.telstra.net (203.50.80.1)  
> 1.966 ms  1.675 ms  1.852 ms
> 4  bundle-ether12.chw-core10.sydney.telstra.net (203.50.11.124) 16.707 
> ms  15.917 ms  16.360 ms
> 5  customer-gw.syd.ALTER.net (...) (...)
>
> This traceroute seems to claim that the packet was received over the 
> Verizon gateway, which in reality it was not -- it was received 
> directly over the Telstra interface, but my outbound AS-PATH towards 
> some.random.isp uses Verizon.
>
> So FreeBSD replies back with the Verizon address. Another person 
> having the same issue (mostly, but on OpenBSD) can be found at 
> http://openbsd.7691.n7.nabble.com/BGP-responding-with-wrong-IP-address-td90264.html
>
> I would love to know if there's a way to fix this, or if I've missed 
> something, or if there's something wrong in the way I set it up.
>
> Thank you for taking the time to read.

I wonder if we could see some routing tables?  That might help.

Best,
George