Multiple WAN links - our setup

[Steven Schoch]

I finally got our FreeBSD 5.4 system configured to route everything
the way I want, so I thought I'd share my setup with this group.  This
may not match exactly your setup, but it still may help.

Our background:

We have a T-1 link to our main ISP through a Cisco router and a class
C address.  We've had this for a long time.  Call the class C net
X.Y.Z.0/24.

We have a FreeBSD 5.4 system with three Ethernet cards, fxp0, fxp1,
and fxp2.  Last year, this became our NAT gateway (we had been using a
NetGear box), as well as a web server for a sub-domain, an email
(SMTP) gateway, a DHCP server, and a DNS server.  To support all these
services, the system is a rack-mount, in the closet, with a UPS and
gmirrored SCSI drives.  In hindsight, it would have probably been
better to split the HTTP and SMTP services to a separate machine, but
it works, so I'm not going to change it now.

Recently, we desired to increase our download speed, so we got a SBC
Yahoo!(r) DSL Expert Plus Package, which gives us 6Mb/s download
speed, for a pretty low price ($49.95 for the first six months, then
it goes to something like $65/month).  Now we have two WAN connections
and we needed to find the best way to integrate them.  We have these
factors:

-  The upload speed for the T-1 line is 1.54Mb/s, compared to the DSL,
which is 0.6Mb/s.  Therefore SSH and FTP connections should go over
the T-1 line, because we use that for uploading.

- Many ISP mail servers are suspicious of email (SMTP) originating
from a SBC Yahoo DSL dynamic IP address (from which much spam
originates), so outgoing SMTP should be over the T-1 line.

- The FreeBSD system also does NTP, DNS, and other services, so the
default route for packets from the box itself should be through the
T-1 line.

- We have a few systems to which we connect that authenticate based on
our IP address (X.Y.Z.*), so HTTP connections to these should be
routed through the T-1 line.

- All other HTTP, HTTPS, IMAP, IMAPS should be routed through the new DSL line.

- I don't trust the new DSL line as well, so I want the route to
automatically swith to the T-1 line when the DSL line goes down.

To get this all working, I start with this /etc/pf.conf file:

--------
# Macros: define common values, so they can be referenced and changed easily.
# We have two external interfaces:  fxp0 connects to the T1 router;
# and tun0 connects to PPPoE, which connects to the DSL router.
t1_if="fxp0"
# We know the address of the T1 interface, but the PPPoE gets a dynamic
t1_addr="X.Y.Z..5"
t1_gw="X.Y.Z.2"
# These are my external NAT addresses, used for a variety of servers.
# These are aliases on the fxp0 net
nat1="X.Y.Z.21"
nat2="X.Y.Z.22"
nat3="X.Y.Z.23"
nat4="X.Y.Z.24"
int_if="fxp1"
internal_net="192.168.1.0/24"

table <nat_pool> { $nat1, $nat2, $nat3, $nat4 }
# The table <local_net> is all the addresses local to this machine.  This must
# be labeled 'persist' because it is not used when the DSL line goes down, and
# the kernel will drop it otherwise.  You may wonder what
192.168.0.0/24 is doing
# there.  That's the address of fxp2, which is used to communicate with the
# internals of the DSL modem.
table <local_net> persist { 192.168.0.0/24, 192.168.1.0/24, X.Y.Z.0/24 }

# The ppp.linkup script will load the rules from /etc/pd.dsl.conf, so
don't do it here.
# load anchor DSL:a from "/etc/pf.dsl.conf"

# Translation: specify how addresses are to be mapped or redirected.

# NAT direct to DSL modem.  This is only used to bring up the DSL modem page,
# to see what our current speed is.
nat on fxp2 inet from $internal_net to 192.168.0.1 -> (fxp2)

nat-anchor DSL

# If the DSL is down, then this rule will apply.  It will also apply
to protocols other that
# HTTP, HTTPS, IMAP, IMAPS, as specified in the pf.dsl.conf file.
nat on $t1_if inet from $internal_net to any -> ($t1_if)

rdr-anchor DSL

# SSH to an inside machine
rdr on $t1_if proto tcp from any to $nat1/32 port 22 -> 192.168.1.101

# RDP to another inside machine
rdr on $t1_if proto tcp from any to $nat2/32 port 3389 -> 192.168.1.109

# TAPI to yet another inside machine
rdr on $t1_if proto tcp from any to $nat1/32 port 5000 -> 192.168.1.7

# X11 forwarding on all NAT interfaces
rdr on $t1_if proto tcp from any to ($t1_if) port 6104 -> 192.168.1.104
rdr on $t1_if proto tcp from any to ($t1_if) port 6105 -> 192.168.1.105
rdr on $t1_if proto tcp from any to ($t1_if) port 6106 -> 192.168.1.106

# rdr outgoing FTP requests to the ftp-proxy
# I haven't tested this fully.  It may still have bugs.
rdr on $int_if proto tcp from $internal_net to any port 21 ->
127.0.0.1 port 8021

# Filtering: the implicit first two rules are
#pass in all
#pass out all

anchor DSL

# ftp proxy
pass in on $t1_if inet proto tcp from any port 20 to ($t1_if) user
proxy flags S/SA keep state
--------


When the system comes up, this will route all NAT traffic through the
T-1 interface.

I should mention the stuff I have in /etc/rc.conf.  Here are the
pertinent lines:

--------
# This is the reliable Cisco box that connects to the reliable T-1 line.
defaultrouter="X.Y.Z.2"
gateway_enable="YES"
ifconfig_fxp0="inet X.Y.Z.5  netmask 255.255.255.0"
ifconfig_fxp1="inet 192.168.1.5  netmask 255.255.255.0"
ifconfig_fxp2="inet 192.168.0.5 netmask 255.255.255.0"
ifconfig_fxp0_alias0="X.Y.Z.20 netmask 0xffffffff"
ifconfig_fxp0_alias1="X.Y.Z.21 netmask 0xffffffff"
ifconfig_fxp0_alias2="X.Y.Z.22 netmask 0xffffffff"
ifconfig_fxp0_alias3="X.Y.Z.23 netmask 0xffffffff"
ifconfig_fxp0_alias4="X.Y.Z.24 netmask 0xffffffff"
ifconfig_fxp1_alias0="192.168.1.20 netmask 0xffffffff"
ppp_enable="YES"
ppp_mode="ddial"
ppp_profile="SBCYahooDSL"
ppp_nat="NO"
--------

Now here's the DSL stuff:  In /etc/ppp/ppp.conf:

--------
SBCYahooDSL:
  set device PPPoE:fxp2
  set authname xxx at sbcglobal.net
  set authkey xxx
  set dial
  set login
  enable lqr
  enable echo
# routing NAT is set in the ppp.linkup file
#  add! default HISADDR
# the "enable lqr" and/or the "enable echo" lines are important,
because that's how
# we know when the link goes down.  Note that we remove the "add!" route line.
--------

Here's /etc/ppp/ppp.linkup:
--------
SBCYahooDSL:
  shell /sbin/pfctl -a DSL:a -f /etc/pf.dsl.conf
--------

And here's /etc/ppp/ppp.linkdown:
--------
SBCYahooDSL:
  shell /sbin/pfctl -a DSL -F all
--------

Note that the DSL anchor rule is loaded when the link goes up, and
cleared when the link goes down.  This works amazingly well!  Here's
the /etc/pf.dsl.conf file:

--------
# This file contains rules for the DSL modem.
# These rules will be unloaded if the modem goes down.
# These rulse are labeled with the anchor "DSL".

t1_if="fxp0"
dsl_if="tun0"
# We know the address of the T1 interface, but the PPPoE gets a dynamic
t1_addr="X.Y.Z.5"
t1_gw="X.Y.Z.2"
# Get the DSL gateway.  It's the peer address of the PPP link.
# By the time this file is loaded, the system will know this.
dsl_gw=tun0:peer
int_if="fxp1"
internal_net="192.168.1.0/24"
special_host="A.B.C"

nat on $dsl_if inet from $internal_net to any -> ($dsl_if)

# X11 forwarding on all NAT interfaces
rdr on $dsl_if proto tcp from any to ($dsl_if) port 6104 -> 192.168.1.104
rdr on $dsl_if proto tcp from any to ($dsl_if) port 6105 -> 192.168.1.105
rdr on $dsl_if proto tcp from any to ($dsl_if) port 6106 -> 192.168.1.106

# Filtering: the implicit first two rules are
#pass in all
#pass out all

# If the source address is from our outside net, then route it through
# the T1 gateway (which we know).
pass out on $dsl_if route-to ($t1_if $t1_gw) from ($t1_if) to any

# Pass HTTP traffic coming from inside to the DSL line, because it's faster.
# But not if it's for us!
pass in on $int_if route-to ($dsl_if $dsl_gw) proto tcp from
$internal_net to !<local_net> port {http,https,imaps}
# Don't pass HTTPS traffic to special_host through the DSL link, because
# we authorize based on IP address
pass in on $int_if proto tcp from $internal_net to $special_host port https

# FTP proxy stuff
block in on $dsl_if proto tcp from any to any port 8021
pass in on $dsl_if inet proto tcp from any port 20 to ($dsl_if) user
proxy flags S/SA keep state
--------

That's all!  Our current testing shows that the route will switch
rapidly when the link goes up or down.  Now I may just be a bit biased
and the DSL line may really be almost as reliable as our T-1 line, but
it's natural to be a bit suspicious of new technology.

Comments welcome.
--
Steve