Dummynet on Bridge on FreeBSD in VMware, its possible right?

Sat May 10 06:13:43 UTC 2008

Hello,
I have been pointed in the right direction that I need to run dummynet 
in a bridge configuration rather than a router configuration.  I have 
carefully followed the instructions for setting up a bridge in 
http://www.freebsd.org/doc/en/articles/filtering-bridges/article.html 
and read numerous man pages, Usenet postings, internet postings, etc.  
Here's a crude schematic of my setup: (switch to fixed width font)

[gateway(.1)]--ether--[le0 (.175) FreeBSD bridge 
le1]<-->VMNet2<-->[(.176)Ubuntu client]
                      |---------------- H O S T Ubuntu P C at 
(.174)-------------------|

The (left) outside end of the bridge (le0) has IP 192.168.111.175 gw 
192.168.111.1, using a VMware Bridged Adapter. The inside end of the 
bridge (on right side) does not have an IP (le1) and is a VMNet2 
adaptor.  My (VMware) Ubuntu client connects to the inside end of the 
bridge via its own VMNet2 adapter at 192.168.111.176.

The bridge is up with both interfaces promiscuous, and in discovery 
mode.  Indeed I can:
- ping OK from the FreeBSD-vm to the gateway(.1), to the Ubuntu client 
(.176), and to the host PC (.174)
- ping OK from the Ubuntu client to the outside end of the bridge 
(.175), and no further
- ping OK from the host PC (.174) to the bridge outside IP (.175) but 
not further to the client

I tried an experiment of using VMNet1 host-only networking for the 
outside-end of the bridge, and adding 3 lines of undecipherable iptable 
commands that had the effect of making the host pc act as a gateway. It 
worked, but I got exactly the same results as above (except gateway was 
local PC (.174)), so I reverted to the more straightforward VMNet 
Bridged adapter architecture for the outside end of the bridge(.175).

I am running 7.0-RELEASE #0, original kernel. 
/boot/loader.conf loads these modules only:
if_bridge_load="YES"
dummynet_load="YES"

/etc/sysctl.conf:
sysctl net.inet.ip.fw.enable=1
sysctl net.link.bridge.ipfw=1
sysctl net.inet.ip.fw.one_pass=1

/etc/rc.conf: (relevant parts)
hostname="freebsdvm"
defaultrouter="192.168.111.1"
gateway_enable="NO"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm le0 addm le1 up"
ifconfig_le0="inet 192.168.111.175 netmask 255.255.255.0 up"
ifconfig_le1="up"
firewall_enable="YES"
firewall_type="open"
firewall_logging="YES"

ifconfig output:
le0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 
mtu 1500
        options=8<VLAN_MTU>
        ether 00:50:56:84:52:ac
        inet 192.168.111.175 netmask 0xffffff00 broadcast 192.168.111.255
        media: Ethernet autoselect
        status: active
le1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 
mtu 1500
        options=8<VLAN_MTU>
        ether 00:0c:29:5c:5e:7f
        media: Ethernet autoselect
        status: active
plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> metric 0 
mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 
1500
        ether 7a:e4:f7:21:7a:14
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: le1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

netstat -rn (ipv4 part only):
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.111.1      UGS         0       52    le0
127.0.0.1          127.0.0.1          UH          0        0    lo0
192.168.111.0/24   link#1             UC          0        0    le0
192.168.111.1      00:0b:46:57:c7:bc  UHLW        2        2    le0   1037
192.168.111.174    00:1d:60:b9:40:07  UHLW        1       98    le0   1199
192.168.111.175    00:50:56:84:52:ac  UHLW        1        4    lo0
192.168.111.176    00:0c:29:96:6c:59  UHLW        1        7    le0   1064

The only thing that seems amiss to me is the above routes indicate the 
Ubuntu client (.176) was reached by the bridge via le0 (outside 
interface) rather than le1 (inside interface) to which the Ubuntu client 
is directly connected via a VMNet2 adapter.  Since the Ubuntu client has 
only the single (VMnet2) interface, it seems impossible, or at least 
undesired, that the FreeBSD bridge host reached the Ubuntu client via 
the outside interface (le0) as indicated in the 'netstat -rn' output, 
but I'm not a networking specialist so its quite possible I'm missing 
something here.

I've regressed from specifying dummynet pipes and queues to plain 
firewall rules (canned from the article quoted above) until I can solve 
this 'FreeBSD bridge on VMWare' networking working.

rc.firewall:
ipfw add 100 pass all from any to any via lo0
ipfw add 200 deny all from any to 127.0.0.0/8
ipfw add 300 deny ip from 127.0.0.0/8 to any

# allow bridge machine to say anything it wants
ipfw add pass tcp from 192.168.111.175 to any setup keep-state
ipfw add pass ip from 192.168.111.175 to any

# allow the inside hosts to say anything they want
ipfw add pass tcp from any to any in via le1 setup keep-state
ipfw add pass ip  from any to any in via le1

# UDP section
# allow DNS only toward the name server
ipfw add pass udp from any to 69.39.192.130 53 in via le1 keep-state

# ICMP section
# pass ping
ipfw add pass icmp from any to any icmptypes 8 keep-state
# pass error messages generated by 'traceroute'
ipfw add pass icmp from any to any icmptypes 3
ipfw add pass icmp from any to any icmptypes 11

ipfw add 65000 allow log all from any to any

BTW, when I say some pings fail, I mean they return the message: 
'Destination Host Unreachable'
Thank you,
Matthew (in Toronto)