Weired routing issues

[Mikhail]

Hello lists,

(I apologize for cross-posting to freebsd-ipfw).

I came across very weird issue with routing today - somehow, FreeBSD box routes packets it shouldn't be routing.
Here it goes:

One of our customers has VPN connection to one of the remote servers. We are using OpenVPN-2 tunnel for this purpose because it is relatively easy to setup and maintain.
The VPN tunnel is used for customer to access internet through remote VPN server to which he connects, so it basically looks as:

[customer]--->[secure vpn tun via internet]--->[vpn server]--->[internet for customer]

The whole setup is pretty straightforward:

On customer's end, customer connects to the Net through DSL modem, and has a small DMZ locally. There is also a FreeBSD box, which is used to setup OpenVPN tunnel. Both FreeBSD box and customer are in one switch, within one subnet, and see the Net through the DSL modem (which is plugged into the same switch for uplink).

On servers' end, there is a standalone FreeBSD box connected to the Net.

Since customer's FreeBSD box is NAT'ed behind DSL modem, OpenVPN tunnel was setup the way both sides see each other internally (within tunnel) using local IPs:
192.168.10.3, iface tun0 - the IP remote VPN server has once VPN connection gets established
192.168.10.4, iface tun0 - the IP customer's FreeBSD has once VPN connection gets established

The subnet customer uses inside his DMZ is 10.0.0.0/24. The following IPs are used:
10.0.0.1 - DSL modem
10.0.0.2 - FreeBSD VPN box
10.0.0.3 - customer's computer

So my aim was to route 10.0.0.3 to the Net through remote VPN server. This was accomplished with the following steps:

1) run natd on remote VPN server, with divert rules as:
divert 8668 ip from 10.0.0.0/24 to any out xmit rl0
2) add routing entry on remote VPN server for 10.0.0.0/24 net:
/sbin/route add 10.0.0.0/24 192.168.10.4
3) turn on interface forwarding on both sides
4) on customer's FreeBSD box setup packet forwarding to remote VPN server as:
ipfw add fwd 192.168.10.3 ip from 10.0.0.3 to any

So the final step remains is to change default gateway on 10.0.0.3 to FreeBSD VPN box (10.0.0.2) instead of DSL modem. And voila - 10.0.0.3 sees internet through this VPN tunnel and all works beautifully.

So far all seems great, however now I need to route this customer back through his DSL modem. To do that, customer simply changes his default gateway back to DSL modem (10.0.0.1). All works beautifully, but one serious problem just occured - 

customer did not change his route back to 10.0.0.1 (he still had FreeBSD's 10.0.0.2 in his settings), and I removed "fwd" route from ipfw by mistake, and what we saw was that customer was still able to surf the Net! However, his was routed through his normal path (not VPN).
- ipfw was flushed
- there's no natd, nor any other nat on FreeBSD box
- no routing or anything

but it still acted as a gateway, routing packets it shouldn't be.
So after such a long thread (which I could probably cut a lot), my question is - how is it possible that local FreeBSD box passed packets that it shouldn't be passing?
I bashed my head against the wall for few hours now, and still couldn't figure - every time I remove "fwd" rule from ipfw, customer goes normal route with no other changes.
I tweaked all sysctls I could think of, and came to if I disable interface forwarding - customer gets cut, and he does not see the Net; enable interface forwarding back and he sees the Net again. I'm totally lost at this point.

Here's what was used to setup VPN tunnel:

- OpenVPN-2.0
- remote VPN server runs FreeBSD-4.10
- local VPN server runs FreeBSD-5.2
- customer's PC runs Windows XP
- 10/100mbit network switch
- DSL modem

local FreeBSD /etc/sysctl & network:
##############################
kern.corefile=/tmp/%N.code
kern.logsigexit=0
net.inet.ip.forwarding=1
net.inet.ip.fastforwarding=1
net.inet.ip.fw.one_pass=0
net.inet.tcp.inflight_enable=1
##############################

$ ifconfig
sk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::211:2fff:fe87:919d%sk0 prefixlen 64 scopeid 0x1
        ether 00:11:2f:87:91:9d
        media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>)
        status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
        inet6 fe80::211:2fff:fe87:919d%tun0 prefixlen 64 scopeid 0x4
        inet 192.168.10.4 --> 192.168.10.3 netmask 0xffffffff
        Opened by PID 454
$ netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            10.0.0.1           UGS         0       77    sk0
10/24              link#1             UC          0        0    sk0
10.0.0.1           00:a0:c5:9a:d7:49  UHLW        1        0    sk0    841
127.0.0.1          127.0.0.1          UH          0        0    lo0
192.168.10.3       192.168.10.4       UH          0       65   tun0

Anything else I need to specify?
I would highly appreciate any help/tips on how to find and remedy this issue.

Sincerely,
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20050520/1043c61f/attachment.bin