Unexpected traffic flow
Göran Löwkrantz
goran.lowkrantz at ismobile.com
Fri Nov 14 07:28:18 UTC 2014
Sorry for top posting but I found the problem. If I remove the bridge
interface everything returns to normal.
Anyone hava any idea? Could there be bad RSTP messages tricking the
forwarding? Bug in if_bridge?
Will do some more network snooping but as this was not a problem with the
previous FBSD version
used here, 8.2, regression?
/glz
--On 13 Nov 2014 17:20:47 +0100 Göran Löwkrantz
<goran.lowkrantz at ismobile.com> wrote:
>
> One of our gateways is behaving oddly after updating to 10.1-PRERELEASE
> r274192 and I just can not understand what is happening.
>
> Internet
> |
> |
> |
> +------+-------+
>| FW |
> +------+-------+
> |
> DMZ
> |
> +------+-------+
>| GW | OpenVPN
> +--+--+--+--+--+
> | | | |
> 2 3 4 9
>
> Net 2 - em2 - 192.168.2.0/24 - servers, server-net switches.
> Net 3 - em1 - 192.168.3.0/24 - workstations, ws-net switches
> Net 4 - em0 - 192.168.4.0/24 - WiFi access points + VLAN switch
> Net 9 - em4 - 192.168.9.0/24 - monitor net w. a few switches.
> DMZ - em5 - XXX.XXX.XXX.128/27 - DMZ and transfer net to outside.
>
> At the bottom of the mail I have included netstat -rn and ifconfig -au
> output.
>
>
> After the upgrade a few hosts got intermittent internet connection and a
> few hosts lost all connectivity.
>
> Specifically, I have one host on the 2-net, 192.168.2.22, that refuses to
> connect to internet or
> lookup names via the gw, which is also the local DNS.
>
> 1: tcpdump on em0, em1, em2, em4 and em5 while sending ping to
> 192.168.2.1 i.e. from the problem host to the gw.
> - incoming traffic on em2:
> 16:54:36.220053 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id
> 9814, seq 1, length 64
> 16:54:37.219705 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id
> 9814, seq 2, length 64
> 16:54:38.219732 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id
> 9814, seq 3, length 64
> 16:54:39.219759 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id
> 9814, seq 4, length 64
> 16:54:40.219783 IP 192.168.2.22 > 192.168.2.1: ICMP echo request, id
> 9814, seq 5, length 64
> 16:54:41.230678 ARP, Request who-has 192.168.2.1 tell 192.168.2.22,
> length 46
> 16:54:41.230682 ARP, Reply 192.168.2.1 is-at 00:1b:21:24:62:4a, length 28
>
> - outgoing traffic on em5:
> 16:54:36.220066 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
> seq 1, length 64
> ...... + 62 repeates .........
> 16:54:36.224436 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
> seq 1, length 64
> 16:54:37.219712 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
> seq 2, length 64
> ...... + 62 repeates .........
> 16:54:37.223711 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
> seq 2, length 64
> 16:54:38.219738 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
> seq 3, length 64
> ...... + 62 repeates .........
> 16:54:38.224984 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
> seq 3, length 64
> 16:54:39.219766 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
> seq 4, length 64
> ...... + 62 repeates .........
> 16:54:39.225757 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
> seq 4, length 64
> 16:54:40.219789 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
> seq 5, length 64
> ...... + 62 repeates .........
> 16:54:40.224282 IP 192.168.2.1 > 192.168.2.22: ICMP echo reply, id 9814,
> seq 5, length 64
>
> Checking on em2, I see
> root at gw01:/usr/local/etc # arp 192.168.2.22
> dbserver-2.lulea.arcticgroup.se (192.168.2.22) at 00:19:99:21:87:62 on
> em2 expires in 1116 seconds [ethernet]
>
> So we have route and arp at em2 but the ping replies are sent out via the
> port connecting to the default gateway.
>
> If I try to ping 192.168.2.22 from the GW, I get this response and only
> traffic going out via em5.
> root at gw01:/usr/local/etc # ping 192.168.2.22
> PING 192.168.2.22 (192.168.2.22): 56 data bytes
> 36 bytes from dmz.xxx.com (XXX.XXX.XXX.129): Redirect Host(New addr:
> XXX.XXX.XXX.158)
> Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
> 4 5 00 0054 39e4 0 0000 40 01 0c2f XXX.XXX.XXX.158 192.168.2.22
>
> For some hosts, 192.168.6 for example, this happens intermittently like
> 3-10 pings go correct, returning on em2, then 1, 2 or more are sent via
> em5.
>
> None of the hosts are using RIP or other active routing, only static
> routes.
> GW host is built with nanobsd and attached kernel configuration.
>
> I have never seen anything like this. Can anyone help? Any more info
> needed?
>
> Routing tables (only IPv4 as that's what's on the inside).
>
> Internet:
> Destination Gateway Flags Netif Expire
> default XXX.XXX.XXX.129 UGS em5 FW DMZ IF
> 10.191.251.0/24 10.191.251.2 UGS tun0 OpenVPN
> net 1
> 10.191.251.1 link#14 UHS lo0 OpenVPN
> local endpoint 1
> 10.191.251.2 link#14 UH tun0 OpenVPN
> remote endpoint 1
> 10.191.252.0/24 10.191.252.2 UGS tun1 OpenVPN
> net 2
> 10.191.252.1 link#15 UHS lo0 OpenVPN
> local endpoint 2
> 10.191.252.2 link#15 UH tun1 OpenVPN
> remote endpoint 2
> 10.191.253.0/24 10.191.253.2 UGS tun2 OpenVPN
> net 3
> 10.191.253.1 link#16 UHS lo0 OpenVPN
> local endpoint 3
> 10.191.253.2 link#16 UH tun2 OpenVPN
> remote endpoint 3
> 127.0.0.1 link#11 UH lo0 localhost
> XXX.XXX.XXX.128/27 link#6 U em5 DMZ network
> XXX.XXX.XXX.157 link#13 UHS lo0
> XXX.XXX.XXX.157/32 link#13 U tap0 OpenVPN
> Server
> XXX.XXX.XXX.158 link#6 UHS lo0 GW extern
> IP, em5
> 192.168.2.0/24 link#3 U em2 server net
> 192.168.2.1 link#3 UHS lo0 GW em2
> 192.168.3.0/24 link#2 U em1 ws net
> 192.168.3.1 link#2 UHS lo0 GW em1
> 192.168.4.0/24 link#1 U em0 wifi net
> 192.168.4.254 link#1 UHS lo0 server em0
> 192.168.9.0/24 link#5 U em4 monitor net
> 192.168.9.254 link#5 UHS lo0 server em4
>
>
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
> ether 00:1b:21:24:62:48
> inet 192.168.4.254 netmask 0xffffff00 broadcast 192.168.4.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
> ether 00:1b:21:24:62:49
> inet 192.168.3.1 netmask 0xffffff00 broadcast 192.168.3.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> em2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0
> mtu 1500
> options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
> ether 00:1b:21:24:62:4a
> inet 192.168.2.1 netmask 0xffffff00 broadcast 192.168.2.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> em4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL
> _MAGIC,VLAN_HWTSO>
> ether 00:30:48:b9:99:c8
> inet 192.168.9.254 netmask 0xffffff00 broadcast 192.168.9.255
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (100baseTX <full-duplex>)
> status: active
> em5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0
> mtu 1500
> options=42098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
> ether 00:30:48:b9:99:c9
> inet XXX.XXX.XXX.158 netmask 0xffffffe0 broadcast XXX.XXX.XXX.159
> inet6 fe80::230:48ff:feb9:99c9%em5 prefixlen 64 scopeid 0x6
> inet6 xxxx:xxxx:101:1::fffe prefixlen 64
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0xb
> inet 127.0.0.1 netmask 0xff000000
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
> ether 02:33:00:4d:00:00
> nd6 options=9<PERFORMNUD,IFDISABLED>
> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> member: em5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> ifmaxaddr 0 port 6 priority 128 path cost 2000000
> member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> ifmaxaddr 0 port 13 priority 128 path cost 2000000
> tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0
> mtu 1500
> options=80000<LINKSTATE>
> ether 00:bd:50:63:00:00
> inet XXX.XXX.XXX.157 netmask 0xffffffff broadcast XXX.XXX.XXX.157
> inet6 fe80::2bd:50ff:fe63:0%tap0 prefixlen 64 scopeid 0xd
> inet6 xxxx:xxxx:101:1::fffd prefixlen 64
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> media: Ethernet autoselect
> status: no carrier
> tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
> options=80000<LINKSTATE>
> inet6 fe80::21b:21ff:fe24:6248%tun0 prefixlen 64 scopeid 0xe
> inet 10.191.251.1 --> 10.191.251.2 netmask 0xffffffff
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> Opened by PID 1565
> tun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
> options=80000<LINKSTATE>
> inet6 fe80::21b:21ff:fe24:6248%tun1 prefixlen 64 scopeid 0xf
> inet 10.191.252.1 --> 10.191.252.2 netmask 0xffffffff
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> Opened by PID 1580
> tun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
> options=80000<LINKSTATE>
> inet6 fe80::21b:21ff:fe24:6248%tun2 prefixlen 64 scopeid 0x10
> inet 10.191.253.1 --> 10.191.253.2 netmask 0xffffffff
> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> Opened by PID 1595
>
> root at gw01:/usr/local/etc # route get 192.168.2.22
> route to: dbserver-2.lulea.arcticgroup.se
> destination: 192.168.2.0
> mask: 255.255.255.0
> fib: 0
> interface: em2
> flags: <UP,DONE,PINNED>
> recvpipe sendpipe ssthresh rtt,msec mtu weight expire
> 0 0 0 0 1500 1 0
>
>
"There are no solved problems; there are only problems that are more
or less solved" -- Henri Poincare
More information about the freebsd-net
mailing list