Having a problem with getting ipfw fwd to work with vlans and bge - 6.1-RC1 amd64

[. at babolo.ru]

[ Charset ISO-8859-1 unsupported, converting... ]
> Hello,
> I have setup a new firewall and I'm having trouble with it. Perhaps the 
> bge is to blame, perhaps its something else.
> I'll explain my setup, problem and the workaround to get it going.
> 
> Box connects to 2 Internal Lans and 2 External Wans.
> 
> Vlans are mixed untagged and tagged on a single bge0
> 
> Vlan   Network             Desc
> 1      10.255.1.0/24       Admin Lan - No Vlan Tagging
> 2      10.255.2.0/24       VoIP Lan
> 900    67.xxx.xxx.128/27   Internet A - Default Route - Going to be pure 
> VoIP only - thus 10.255.2 boxes get 1:1 NAT to 67.xxx.xxx
> 902    208.xxx.xxx.48/28   Internet B - Web Services
> 
> 1st problem I ran into was pings from vlan 2 through natd to vlan 900 
> were not coming back. I could see the packet enter vlan2 - leave and 
> return on vlan900 - but go nowhere. I tried a tcpdump on bge0 and the 
> pings started coming back. Leading me to putting promisc on my ifconfig bge0
> 
> Now I'm trying to setup up a simple web server on an IP from vlan 902 in 
> combination with fwd rule # 999 to route packets from a vlan902 address 
> back to the router on that internet connection. I try to ping from the 
> outside and can see the icmp echo request. But the replies keep getting 
> sent out vlan900 to the other internet router.
> 
> Hopefully somebody can point me in the right direction. If its the bge, 
> then I can replace it with some em. If its an issue with mixing native 
> vlan and tagged, I can tag everything, If its not me, then who can help 
> getting the code fixed?
> 
> I have put my ifconfig, ipfw rules and natd.conf's below.
Don't know about FreeBSD 6, in FreeBSD 4 you need mtu = 1504
for mtu = 1500 on vlans to work.

This is reason not to use mix tagged/utagged on one bge.

> Thanks -Jon
> 
> ---------------------------------------------------------
> 
> [root at t3031fw ~]# ifconfig -a
> bge0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> mtu 1500
>         options=18<VLAN_MTU,VLAN_HWTAGGING>
>         inet6 fe80::215:f2ff:fed0:d898%bge0 prefixlen 64 scopeid 0x1
>         inet 10.255.1.254 netmask 0xffffff00 broadcast 10.255.1.255
>         ether 00:15:f2:d0:d8:98
>         media: Ethernet autoselect (100baseTX <full-duplex>)
>         status: active
> bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
>         options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
>         ether 00:15:f2:40:d8:35
>         media: Ethernet autoselect (none)
>         status: no carrier
> plip0: flags=108810<POINTOPOINT,SIMPLEX,MULTICAST,NEEDSGIANT> mtu 1500
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
>         inet6 ::1 prefixlen 128
>         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
>         inet 127.0.0.1 netmask 0xff000000
> vlan2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         inet6 fe80::215:f2ff:fed0:d898%vlan2 prefixlen 64 scopeid 0x5
>         inet 10.255.2.1 netmask 0xffffff00 broadcast 10.255.2.255
>         ether 00:15:f2:d0:d8:98
>         media: Ethernet autoselect (100baseTX <full-duplex>)
>         status: active
>         vlan: 2 parent interface: bge0
> vlan900: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
...
>         ether 00:15:f2:d0:d8:98
>         media: Ethernet autoselect (100baseTX <full-duplex>)
>         status: active
>         vlan: 900 parent interface: bge0
> vlan902: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         inet6 fe80::215:f2ff:fed0:d898%vlan902 prefixlen 64 scopeid 0x7
...
>         ether 00:15:f2:d0:d8:98
>         media: Ethernet autoselect (100baseTX <full-duplex>)
>         status: active
>         vlan: 902 parent interface: bge0