kern/141843: [em] [vlan] Intel txcsum and assigned vlan invoke wrong dst MAC in TCP packets

Wed Dec 30 15:48:53 UTC 2009

I can reproduce this case.
server1 has nic nfe0
trouble server2 has nic  em0 and  em1
em0 at pci0:13:0:0:        class=0x020000 card=0x108c15d9 chip=0x108c8086 
rev=0x03 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Intel Corporation 82573E Gigabit Ethernet Controller 
(Copper) (82573E)'
    class      = network
    subclass   = ethernet
em1 at pci0:14:0:0:        class=0x020000 card=0x109a15d9 chip=0x109a8086 
rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Intel PRO/1000 PL Network Adaptor (82573L)'
    class      = network
    subclass   = ethernet

scheme:
server1-nfe0 -- switch -- em1-trouble-server2

on server2:
ifconfig  vlan555 create vlandev em1 vlan 555 10.10.10.2/24
and i can't ping from another server1 with ip 10.10.10.1/24 and switch 
has not mac  address of server2 nic.
nic hangs down/up in  /var/log/messages:

Dec 30 16:38:56 server2 kernel: em1: link state changed to DOWN
Dec 30 16:38:56 server2 kernel: vlan555: link state changed to DOWN
Dec 30 16:39:00 server2 kernel: em1: link state changed to UP
Dec 30 16:39:00 server2 kernel: vlan555: link state changed to UP

then ifconfig em1 down && ifconfig em1 up
server1 began to ping and connect to iperf on server2

then
ifconfig  vlan556 create vlandev em1 vlan 556 10.11.11.2/24
in messages:

Dec 30 16:40:28 server2 kernel: em1: link state changed to DOWN
Dec 30 16:40:28 server2 kernel: vlan555: link state changed to DOWN
Dec 30 16:40:28 server2 kernel: vlan556: link state changed to DOWN
Dec 30 16:40:31 server2 kernel: em1: link state changed to UP
Dec 30 16:40:31 server2 kernel: vlan555: link state changed to UP
Dec 30 16:40:31 server2 kernel: vlan556: link state changed to UP

server1 continue ping ip 10.10.10.2 on server2, but not connect to iperf
and really, tcpdump -letn -i vlan555 on server1 shown changing mac 
address server1 in output packets from server2

ifconfig em1 -txcsum -tso resolve this problem, but not resolve hangs 
down/up kern/141285

after apply your patch and rebuild kernel:
on server2:
ifconfig  vlan555 create vlandev em1 vlan 555 10.10.10.2/24
hangs down/up nic, not ping again
ifconfig em1 down && ifconfig em1 up - don't help

ifconfig em1 -txcsum -tso -vlanhwtag help ONLY

 for example tcpdump -letn -i vlan555 on server1
00:1b:fc:af:a1:b4 - mac-address nic of server1
00:30:48:8d:fe:ed - mac-address nic of server2

mac 00:1b:fc:af:a1:b4 exchange on e4:98:fc:af:a1:b4 and 
d8:e0:fc:af:a1:b4 and ... in reply from server2

00:1b:fc:af:a1:b4 > 00:30:48:8d:fe:ed, ethertype IPv4 (0x0800), length 
74: 10.10.10.1.61411 > 10.10.10.2.5001: S 564478084:564478084(0) win 
65535 <mss 1460,nop,wscale 8,sackOK,timestamp 800079145 0>
00:30:48:8d:fe:ed > e4:98:fc:af:a1:b4, ethertype IPv4 (0x0800), length 
74: 10.10.10.2.5001 > 10.10.10.1.61411: S 2331848301:2331848301(0) ack 
564478085 win 65535 <mss 1460,nop,wscale 9,sackOK,timestamp 830508984 
800079145>
00:1b:fc:af:a1:b4 > 00:30:48:8d:fe:ed, ethertype IPv4 (0x0800), length 
74: 10.10.10.1.61411 > 10.10.10.2.5001: S 564478084:564478084(0) win 
65535 <mss 1460,nop,wscale 8,sackOK,timestamp 800082145 0>
00:30:48:8d:fe:ed > d8:e0:fc:af:a1:b4, ethertype IPv4 (0x0800), length 
74: 10.10.10.2.5001 > 10.10.10.1.61411: S 2331848301:2331848301(0) ack 
564478085 win 65535 <mss 1460,nop,wscale 9,sackOK,timestamp 830508984 
800082145>
00:30:48:8d:fe:ed > d8:e0:fc:af:a1:b4, ethertype IPv4 (0x0800), length 
74: 10.10.10.2.5001 > 10.10.10.1.61411: S 2331848301:2331848301(0) ack 
564478085 win 65535 <mss 1460,nop,wscale 9,sackOK,timestamp 830508984 
800082145>
00:1b:fc:af:a1:b4 > 00:30:48:8d:fe:ed, ethertype IPv4 (0x0800), length 
74: 10.10.10.1.61411 > 10.10.10.2.5001: S 564478084:564478084(0) win 
65535 <mss 1460,nop,wscale 8,sackOK,timestamp 800085345 0>
00:30:48:8d:fe:ed > cc:60:fc:af:a1:b4, ethertype IPv4 (0x0800), length 
74: 10.10.10.2.5001 > 10.10.10.1.61411: S 2331848301:2331848301(0) ack 
564478085 win 65535 <mss 1460,nop,wscale 9,sackOK,timestamp 830508984 
800085345>
00:30:48:8d:fe:ed > cc:60:fc:af:a1:b4, ethertype IPv4 (0x0800), length 
74: 10.10.10.2.5001 > 10.10.10.1.61411: S 2331848301:2331848301(0) ack 
564478085 win 65535 <mss 1460,nop,wscale 9,sackOK,timestamp 830508984 
800085345>
00:1b:fc:af:a1:b4 > 00:30:48:8d:fe:ed, ethertype IPv4 (0x0800), length 
62: 10.10.10.1.61411 > 10.10.10.2.5001: S 564478084:564478084(0) win 
65535 <mss 1460,sackOK,eol>

sorry for my english

Pyun YongHyeon wrote:
> The following reply was made to PR kern/141843; it has been noted by GNATS.
>
> From: Pyun YongHyeon <pyunyh at gmail.com>
> To: Dennis Yusupoff <dyr at smartspb.net>
> Cc: bug-followup at FreeBSD.org
> Subject: Re: kern/141843: [em] [vlan] Intel txcsum and assigned vlan invoke wrong dst MAC in TCP packets
> Date: Tue, 29 Dec 2009 10:51:19 -0800
>
>  On Mon, Dec 21, 2009 at 11:53:22PM +0000, linimon at freebsd.org wrote:
>  > Old Synopsis: Intel txcsum and assigned vlan invoke wrong dst MAC in TCP packets
>  > New Synopsis: [em] [vlan] Intel txcsum and assigned vlan invoke wrong dst MAC in TCP packets
>  > 
>  > Responsible-Changed-From-To: freebsd-bugs->freebsd-net
>  > Responsible-Changed-By: linimon
>  > Responsible-Changed-When: Mon Dec 21 23:51:33 UTC 2009
>  > Responsible-Changed-Why: 
>  > Over to maintainer(s).
>  > 
>  > http://www.freebsd.org/cgi/query-pr.cgi?pr=141843
>  
>  I'm not sure whether you're seeing one of edge case of em(4)/igb(4)
>  checksum offload issue. Personally I couldn't reproduce the issue
>  but I guess the checksum offload/TSO handling might cause
>  unexpected result. If you disable Tx checksum offload or TSO em(4)
>  work as expected, right? If so, would you try the following patch
>  and let me know whether it makes any difference?
>  
>  http://people.freebsd.org/~yongari/em.csum_tso.20091229.patch
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>
>
>   