Wrong outgoing interface with multiple routing tables

Mon Jul 27 17:26:48 UTC 2009

Oleg Sharoyko wrote:
> Hello!
> 
> I'm having a trouble with multiple routing tables (FreeBSD 7.2 release).
> Either I'm missing something in my setup or packets for daemons started
> with setfib are being sent out via the wrong interface.
> 
> What I'd like to implement:
> 
> em0 - internal management network with ip address 10.2.5.2/24 and
> default route 10.2.5.1
> 
> em1 - public interface to be used in jail with ip address
> 195.208.245.229/27 and default route 195.208.245.225
> 
> Here are my routing tables:
> 
> r61net-fbsdhost-1, / # setfib -0 netstat -rn
> Routing tables
> 
> Internet:
> Destination        Gateway            Flags    Refs      Use  Netif Expire
> default            10.2.5.1           UGS         0      350    em0
> 10.2.5.0/24        link#1             UC          0        0    em0
> 10.2.5.1           00:1e:4a:b4:ea:c0  UHLW        2        0    em0   1182
> 127.0.0.1          127.0.0.1          UH          0       30    lo0
> 
> r61net-fbsdhost-1, / # setfib -1 netstat -rn
> Routing tables
> 
> Internet:
> Destination        Gateway            Flags    Refs      Use  Netif Expire
> default            195.208.245.225    UGS         0        0    em1
> 195.208.245.224/27 link#2             UC          0        0    em1
> 195.208.245.225    link#2             UHLW        2        0    em1

so far, all looks correct.

> 
> Firewall:
> 
> r61net-fbsdhost-1, / # ipfw show
> 00001  0    0 setfib 1 ip from any to any in recv em1

good

> 00010  0    0 count ip from any to any dst-port 2222 fib 0
> 00011  0    0 count ip from any 2222 to any fib 0
> 00012  0    0 count ip from any to any dst-port 2222 fib 1
> 00013  0    0 count ip from any 2222 to any fib 1
> 00100  0    0 allow ip from any to any via lo0
> 00200  0    0 deny ip from any to 127.0.0.0/8
> 00300  0    0 deny ip from 127.0.0.0/8 to any
> 65000 30 2648 allow ip from any to any
> 65535  0    0 deny ip from any to any
> 
> 
> With this setup almost everything works as I expect. For example ICMP
> echo requests and responses are being received and sent via em1. Both
> when ping runs on this host as "setfib 1 ping other_host" and when other
> host pings ip address of em1. Connection attempts (setfib 1 telnet
> other_host) are also being sent out of the right interface. But when it
> comes to the daemons I run into troubles. 
> 
> I use sshd for tests (have also tried other daemons with no luck):
> 
> r61net-fbsdhost-1, / # setfib 1 /usr/sbin/sshd -o 'ListenAddress 195.208.245.229:2222' -D

Are you running this from inetd?. (doesnt look like it)

btw is it 1 or -1? or -F 1?
I can't remember if I supported just '1'.

> 
> sshd is bound only to ip address of em1:
> 
> r61net-fbsdhost-1, / # sockstat | grep 2222
> root     sshd       839   3  tcp4   195.208.245.229:2222  *:*
> 
> While doing telnet 195.208.249.229 2222 from another host I got following packet traces:
> 
> r61net-fbsdhost-1, / # tcpdump -i em0 port 2222
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes
> 17:39:34.872475 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: S 2590499299:2590499299(0) ack 3939022576 win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 3248254533 147282318>
> 17:39:34.902622 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 1:41(40) ack 1 win 8326 <nop,nop,timestamp 3248254533 147282318>
> 17:39:37.572271 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 41:60(19) ack 7 win 8326 <nop,nop,timestamp 3248254593 147282585>
> 17:39:37.572293 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: F 60:60(0) ack 7 win 8326 <nop,nop,timestamp 3248254593 147282585>
> 17:39:37.572986 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: . ack 8 win 8325 <nop,nop,timestamp 3248254593 147282585>
> 
> r61net-fbsdhost-1, / # tcpdump -i em1 port 2222
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes
> 17:39:34.872370 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: S 3939022575:3939022575(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 147282318 0>
> 17:39:34.872803 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 2590499300 win 8326 <nop,nop,timestamp 147282318 3248254533>
> 17:39:35.002882 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 41 win 8326 <nop,nop,timestamp 147282331 3248254533>
> 17:39:37.571659 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: P 0:6(6) ack 41 win 8326 <nop,nop,timestamp 147282585 3248254533>
> 17:39:37.572923 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 61 win 8323 <nop,nop,timestamp 147282585 3248254593>
> 17:39:37.572945 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: F 6:6(0) ack 61 win 8326 <nop,nop,timestamp 147282585 3248254593>
> 
> And firewall counters:
> 
> r61net-fbsdhost-1, / # ipfw show
> 00001  6  326 setfib 1 ip from any to any in recv em1
> 00010  0    0 count ip from any to any dst-port 2222 fib 0
> 00011  5  327 count ip from any 2222 to any fib 0
> 00012  6  326 count ip from any to any dst-port 2222 fib 1
> 00013  0    0 count ip from any 2222 to any fib 1
> 00100  0    0 allow ip from any to any via lo0
> 00200  0    0 deny ip from any to 127.0.0.0/8
> 00300  0    0 deny ip from 127.0.0.0/8 to any
> 65000 60 5057 allow ip from any to any
> 65535  0    0 deny ip from any to any
> 
> So the packets, generated by sshd are being sent out via em0 instead of
> em1.
> 
> With
> 
> ipfw add 2 setfib 1 ip from 195.208.245.229 to any
> 
> outgoing packets are being tagged with correct fib, but still sent via
> em0.

yes becasue on outgoing packets the firewall is too late to influence 
that.

> 
> With
> 
> ipfw add 60003 fwd 195.208.245.225 src-ip me src-ip 195.208.245.224/27 not dst-ip 195.208.245.224/27
> 
> first SYN packet from 195.208.245.229 is being sent correctly via em1,
> but I cannot see any further packets at all:
> 
> r61net-fbsdhost-1, / # tcpdump -i em1 port 2222
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes
> 18:01:56.665341 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: S 2484180116:2484180116(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 147415433 0>
> 18:01:56.665463 IP stat.r61.net.2222 > brain.cc.rsu.ru.50435: S 3905497961:3905497961(0) ack 2484180117 win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 3376909218 147415433>
> 18:01:56.665798 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: . ack 1 win 8326 <nop,nop,timestamp 147415433 3376909218>
> 
> and no packets at em0. TCP connection establishes but no data packets
> come from daemon which is rather weird.
> 
> I would appreciate any help with this issue.

try adding a '-' on the command
and get back to me.

> 