Routing outbound IP packets on multihomed box

[Boris Kochergin]

Hi. I've come across this problem but solved it with a PF rule of this 
form, if that's an option for you:

pass out route-to (vlan256 169.229.126.1) from 169.229.126.9 to any

This tells PF to send all packets sent from 169.229.126.9 through the 
vlan256 interface with a next-hop address of 169.229.126.1.

-Boris

Christopher Cowart wrote:
> Hello,
>
> I have a server with two NICs:
>
> em0:        169.229.79.139/25
> vlan526:    169.229.126.9/24
>
> The default gateway is 169.229.79.129. The router for the 126 subnet is
> 169.229.126.1. 
>
> netstat -rn:
> | Destination        Gateway            Flags    Refs      Use  Netif Expire
> | default            169.229.79.129     UGS         0   102537    em0
> | 127.0.0.1          127.0.0.1          UH          0      217    lo0
> | 169.229.79.128/25  link#1             UC          0        0    em0
> | 169.229.79.129     00:15:c7:b9:f4:80  UHLW        2        4    em0   1193
> | 169.229.79.139     00:11:25:ab:42:70  UHLW        1      589    lo0
> | 169.229.126/24     link#9             UC          0        0 vlan52
> | 169.229.126.1      00:15:c7:b9:f4:80  UHLW        1       34 vlan52   1200
> | 169.229.126.9      00:18:f8:09:d3:a5  UHLW        1        8    lo0
>
> The IP address on em0 works exactly as one would expect. I have full IP
> connectivity to it from other subnets. 
>
> The problem is I can't get 2-way connectivity with the IP address on
> vlan526.
>
> Using my workstation on a third subnet (169.229.127.38/24), I cannot
> ping 169.229.126.9. I leave the ping running and do some tcpdumps on 
> the server.
>
> $ sudo tcpdump -ni vlan526 host 169.229.127.38
> | 14:14:37.002920 IP 169.229.127.38 > 169.229.126.9: ICMP echo 
> | request, id 15733, seq 35, length 64
> | 14:14:38.003037 IP 169.229.127.38 > 169.229.126.9: ICMP echo 
> | request, id 15733, seq 36, length 64
>
> Notice there are no echo replies. That's because they're being sent 
> here:
>
> $ sudo tcpdump -ni em0 host 169.229.127.38
> | 14:15:42.006997 IP 169.229.126.9 > 169.229.127.38: ICMP echo reply, 
> | id 15733, seq 100, length 64
> | 14:15:43.007118 IP 169.229.126.9 > 169.229.127.38: ICMP echo reply, 
> | id 15733, seq 101, length 64
>
> I repeated this last snoop with a -w and loaded it into ethereal. The
> echo replies being sent out on em0 indeed have a source address of
> 169.229.126.9. The router (169.229.79.139) drops these packets on the
> floor, because their source address isn't routable on that interface.
>
> Because routing is based on destination, not source address, I'm not
> sure how to get packets sourced from the 126 subnet to the router on the
> 126 subnet. I tried the following ipfw rule right after allow loopback
> traffic (my second rule):
>
> fwd 169.229.126.1 ip from 169.229.126.9 to not 169.229.126.0/24
>
> Still no luck. Has anyone set up a multihomed box on *different* subnets
> before without routing them through the FreeBSD box? Does anyone have
> any pointers or things I should be looking at?
>
> Thanks,
>
>