Routing outbound IP packets on multihomed box

[Julian Elischer]

Christopher Cowart wrote:
> On Fri, Jun 15, 2007 at 06:30:23PM -0400, Boris Kochergin wrote:
>> Christopher Cowart wrote:
>>> I have a server with two NICs:
>>>
>>> em0:        169.229.79.139/25
>>> vlan526:    169.229.126.9/24
>>>
>>> The default gateway is 169.229.79.129. The router for the 126 subnet is
>>> 169.229.126.1. 
>>>
>>> netstat -rn:
>>> | Destination        Gateway            Flags    Refs      Use  Netif 
>>> Expire
>>> | default            169.229.79.129     UGS         0   102537    em0
>>> | 127.0.0.1          127.0.0.1          UH          0      217    lo0
>>> | 169.229.79.128/25  link#1             UC          0        0    em0
>>> | 169.229.79.129     00:15:c7:b9:f4:80  UHLW        2        4    em0   
>>> 1193
>>> | 169.229.79.139     00:11:25:ab:42:70  UHLW        1      589    lo0
>>> | 169.229.126/24     link#9             UC          0        0 vlan52
>>> | 169.229.126.1      00:15:c7:b9:f4:80  UHLW        1       34 vlan52   
>>> 1200
>>> | 169.229.126.9      00:18:f8:09:d3:a5  UHLW        1        8    lo0
>>>
>>> The IP address on em0 works exactly as one would expect. I have full IP
>>> connectivity to it from other subnets. 
>>>
>>> The problem is I can't get 2-way connectivity with the IP address on
>>> vlan526.
>>>
>>> Using my workstation on a third subnet (169.229.127.38/24), I cannot
>>> ping 169.229.126.9. I leave the ping running and do some tcpdumps on 
>>> the server.
>>>
>>> $ sudo tcpdump -ni vlan526 host 169.229.127.38
>>> | 14:14:37.002920 IP 169.229.127.38 > 169.229.126.9: ICMP echo 
>>> | request, id 15733, seq 35, length 64
>>> | 14:14:38.003037 IP 169.229.127.38 > 169.229.126.9: ICMP echo 
>>> | request, id 15733, seq 36, length 64
>>>
>>> Notice there are no echo replies. That's because they're being sent 
>>> here:
>>>
>>> $ sudo tcpdump -ni em0 host 169.229.127.38
>>> | 14:15:42.006997 IP 169.229.126.9 > 169.229.127.38: ICMP echo reply, 
>>> | id 15733, seq 100, length 64
>>> | 14:15:43.007118 IP 169.229.126.9 > 169.229.127.38: ICMP echo reply, 
>>> | id 15733, seq 101, length 64
>>>
>>> I repeated this last snoop with a -w and loaded it into ethereal. The
>>> echo replies being sent out on em0 indeed have a source address of
>>> 169.229.126.9. The router (169.229.79.139) drops these packets on the
>>> floor, because their source address isn't routable on that interface.
>>>
>>> Because routing is based on destination, not source address, I'm not
>>> sure how to get packets sourced from the 126 subnet to the router on the
>>> 126 subnet. I tried the following ipfw rule right after allow loopback
>>> traffic (my second rule):
>>>
>>> fwd 169.229.126.1 ip from 169.229.126.9 to not 169.229.126.0/24
>>>
>>> Still no luck. Has anyone set up a multihomed box on *different* subnets
>>> before without routing them through the FreeBSD box? Does anyone have
>>> any pointers or things I should be looking at?
>> Hi. I've come across this problem but solved it with a PF rule of this 
>> form, if that's an option for you:
>>
>> pass out route-to (vlan256 169.229.126.1) from 169.229.126.9 to any
>>
>> This tells PF to send all packets sent from 169.229.126.9 through the 
>> vlan256 interface with a next-hop address of 169.229.126.1.
> 
> Unfortunately, I don't think we can use pf. The rest of our
> infrastructure is ipfw and we don't particularly want this to be a
> one-off. I was under the impression that my ipfw rule did exactly this,
> by sending the packets to the 126 router as their next hop.
> 
> Anyone have any ideas on whether an ipfw fwd rule can be used in a
> similar way to this pf rule?
> 
> Thanks again,
> 
he ipfw rule should work, assuming you have the IPFIREWALL_FORWARDING option
(and it's not the couple of versions of the OS where you also needed the
IPFIREWALL_FORWARD_EXTENDED option as well..

also, you need to make it an 'out' rule..i.e. 
 fwd 169.229.126.1 ip from 169.229.126.9 to not 169.229.126.0/24 out