Outgoing packets being sent via wrong interface

[Mark Martinec]

On 2015-11-25 09:21, Daniel Bilik wrote:
> It happened again, yesterday, and I can now definitely confirm
> that it's related to default route.
[...]
> ... because again it was pushing outgoing packets wrong way, via public
> interface, where it's dropped by pf...
[...]
> I've tried to just delete default route and enter it back to routing 
> table.
> # route delete default ; sleep 1 ; route add default 82.x.y.29
> ... and voila, ping started to communicate with affected host...

Seems like I have stumbled across the same problem as Daniel,
as all that was said above applies to my case too.

Running a fairly recent 10-STABLE, pf was disabled.
   10.2-STABLE FreeBSD 10.2-STABLE #3 r291378:
   Fri Nov 27 12:45:53 CET 2015 ... amd64

In addition to a regular ethernet interface, I also have a gif
tunnel. Routing directs a tunnel endpoint to the ethernet
interface, and most of the rest goes through tunnel.

Maybe worth mentioning that some processes (like ntpd) run
in FIB 1 with their own routing able to force them to use
a direct route.

I was trying to set up an nginx proxy, but couldn't get it
to respond to a remote client. After tcpdump sessions
it turned out that a TCP SYN from a remote host came in
through an ethernet interface, but a SYN ACK went out
through gif, even through netstat -rn was still telling
the default route is ethernet. Firewall was disabled.

Funny thing is that a sshd process was still sending
responses to the same remote host through the correct
ethernet interface via a default route.

Luckily I came across this thread, and sure enough, a:
   route delete default; route add default x.x.x.x
solved the problem right away.  The netstat -rn output
of before and after the route reset were exactly the same.

Unfortunately I can't reproduce the problem now, and I never
experienced such problem in previous 10-STABLE revisions,
or earlier.

Anyway, just wanted to mention that possibly Daniel
may not be alone with a default route becoming ineffective.

   Mark


On 2015-11-25 09:21, Daniel Bilik wrote:
> On Sun, 22 Nov 2015 13:02:40 +0100
> Daniel Bilik <ddb at neosystem.org> wrote:
> 
>> Well, even though pf may play some role in the problem, I tend to 
>> suspect
>> the routing table as the main trigger. There are several facts to 
>> support
>> this...
> 
> It happened again, yesterday, and I can now definitely confirm that 
> it's
> related to default route.
> 
> In this case, affected address was 192.168.2.33. This host was unable 
> to
> connect to 192.168.2.15 (jail on the router), and router itself was 
> unable
> to even ping the affected host...
> 
> PING 192.168.2.33 (192.168.2.33): 56 data bytes
> ping: sendto: Operation not permitted
> ping: sendto: Operation not permitted
> 
> ... because again it was pushing outgoing packets wrong way, via public
> interface, where it's dropped by pf...
> 
> 00:00:07.091814 rule 53..16777216/0(match): block out on re0:
> 82.x.y.50 > 192.168.2.33: ICMP echo request, id 12037, seq 0, length
> 64
> 00:00:01.011536 rule 53..16777216/0(match): block out on re0:
> 82.x.y.50 > 192.168.2.33: ICMP echo request, id 12037, seq 1, length
> 64
> 
> I've tried to just delete default route and enter it back to routing 
> table.
> In one tmux session ping was running, in another session I've performed
> this...
> 
> # route delete default ; sleep 1 ; route add default 82.x.y.29
> 
> ... and voila, ping started to communicate with affected host...
> 
> ping: sendto: Operation not permitted
> ping: sendto: Operation not permitted
> 64 bytes from 192.168.2.33: icmp_seq=12 ttl=128 time=0.535 ms
> 64 bytes from 192.168.2.33: icmp_seq=13 ttl=128 time=0.264 ms
> 
> Touching nothing else (pf etc.), not rebooting, just "refreshing" the
> default route entry, and the problem disappeared.