Routing between two physical networks

[Jeremy Chadwick]

(Please keep me CC'd, as I'm not subscribed to freebsd-net)

I'm having some complexity handling 3 separate networks on the same
physical box, which also acts as a gateway.  NAT (pf) is involved, but
only for packets going out the external interface (em0).  The simple
version is that two of the local (physical) networks can't talk.

Before I provide details, please keep in mind I *do not* want to use
bridge(4) to solve this problem.  I have tried it -- yes it works as
expected -- but I'm trying hard to avoid use of it.

OS: amd64 FreeBSD 8.2-PRERELEASE #0: Sat Feb 12 06:13:55 PST 2011

Physical topology and physical interfaces:

  +-------------+
  |             |== em0   <---> cable modem <---> Internet
  | FreeBSD Box |== em1   <---> switch      <---> PC#1
  |             |== wlan0 <---> printer
  +-------------+

  FreeBSD em0   = public Internet IP address
  FreeBSD em1   = 192.168.1.1     netmask 0xffffff00
  FreeBSD wlan0 = 192.168.200.1   netmask 0xffffff00
  PC#1          = 192.168.1.50    netmask 0xffffff00 gateway 192.168.1.1
  printer       = 192.168.200.101 netmask 0xffffff00 gateway 192.168.1.1

Situation:

- PC#1 can talk to the FreeBSD box (as 192.168.1.1 or 192.168.200.1)
- printer can talk to the FreeBSD box (as 192.168.200.1 or 192.168.1.1)
- PC#1 can reach the Internet
- PC#1 **cannot** talk to printer, nor vice-versa

The last item is what I'm trying to fix.

Packets from PC#1 -> Internet are NAT'd using the following pf
statements:

  ext_if="em0"
  int_if="em1"
  nat on $ext_if from $int_if:network to any -> ($ext_if)

gateway_enable="yes" is obviously set in rc.conf.

traceroute from PC#1 to printer shows hop #1 as 192.168.1.1, then
indefinite timeouts.

When PC#1 pings the printer, "tcpdump -p -i em1 -l -n icmp" from the
FreeBSD box shows ICMP ECHO requests from 192.168.1.50 to
192.168.200.101, but no reply.

When doing "tcpdump -p -i wlan0 -l -n icmp", the exact same packets as
above are seen.  I'd love for someone to explain this to me.  :-)

When doing "tcpdump -p -i em0 -l -n icmp", nothing is seen, so I imagine
the packets destined for 192.168.200.101 aren't going out the Internet.

I've tried changing the printer's gateway to 192.168.200.1 but that has
no effect either.

My pf.conf has "set skip on wlan0" and "set skip on em1", so I'm not
exactly sure where the packets are disappearing, and am inclined to
think it's a routing table issue.

I can put up my configuration bits (rc.conf, pf.conf) as well as
"ifconfig -a", "netstat -rn", etc. if someone would like to see them (I
imagine so), just let me know.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |