amd64/63188: ti(4) broken on amd64

[Hendrik Scholz]

>Number:         63188
>Category:       amd64
>Synopsis:       ti(4) broken on amd64
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    freebsd-amd64
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Sat Feb 21 13:20:16 PST 2004
>Closed-Date:
>Last-Modified:
>Originator:     Hendrik Scholz
>Release:        FreeBSD 5.2.1-RC amd64
>Organization:
>Environment:
System: FreeBSD slurmcan.wormulon.net 5.2.1-RC FreeBSD 5.2.1-RC #0: Sat Jan 31 12:38:34 GMT 2004 root at ngoc.NUXI.com:/usr/obj/usr/src/sys/GENERIC amd64


	
>Description:

ti0: <Alteon AceNIC 1000baseSX Gigabit Ethernet> mem 0xfc8bc000-0xfc8bffff irq 28 at device 1.0 on pci1
ti0: Ethernet address: 00:60:cf:20:6b:a6
(full dmesg at http://dump.wormulon.net/dmesg/MSI_K8D/dmesg.txt)

$ ifconfig ti0
ti0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
        inet6 fe80::260:cfff:fe20:6ba6%ti0 prefixlen 64 scopeid 0x1
        inet 63.223.8.203 netmask 0xffffff00 broadcast 63.223.8.255
        ether 00:60:cf:20:6b:a6
        media: Ethernet autoselect (1000baseSX <full-duplex>)
        status: active

TCP based protocols (ssh) work but scp is very slow and sometimes the
connection is dropped due to bad packets.

ICMP just misses a bunch of packets:

$ ping 63.223.8.133
PING 63.223.8.133 (63.223.8.133): 56 data bytes
64 bytes from 63.223.8.133: icmp_seq=1 ttl=64 time=0.277 ms
64 bytes from 63.223.8.133: icmp_seq=3 ttl=64 time=0.200 ms
..
11 packets transmitted, 8 packets received, 27% packet loss
round-trip min/avg/max/stddev = 0.200/0.260/0.316/0.039 ms
$

Doing a tcpdump on the host itself reveals:

63.223.8.203 > 63.223.8.34: icmp: echo request (ttl 64, id 29322, len 84, bad cksum 0!)
63.223.8.34 > 63.223.8.203: icmp: echo reply (ttl 128, id 42958, len 84)
truncated-ip - 70 bytes missing! 63.223.7.6.22 > 24.30.48.121.1921: P 4294962720:4294962784(64) ack 4294964481 win 33304 <nop,nop,timestamp[|tcp]> (DF) [tos 0x10]  (ttl 64, id 29087, len 116, bad cksum 0!)
truncated-ip - 38 bytes missing! 63.223.7.6.22 > 24.30.48.121.1850: P 7760:7792(32) ack 1185 win 33304 <nop,nop,timestamp[|tcp]> (DF) [tos 0x10]  (ttl 64, id 29247, len 84, bad cksum 0!)
truncated-ip - 38 bytes missing! 63.223.7.6.22 > 24.30.48.121.1850: P 7760:7792(32) ack 1185 win 33304 <nop,nop,timestamp[|tcp]> (DF) [tos 0x10]  (ttl 64, id 29247, len 84, bad cksum 0!)
truncated-ip - 38 bytes missing! 63.223.7.6.22 > 24.30.48.121.1850: P 3808:3840(32) ack 289 win 33304 <nop,nop,timestamp[|tcp]> (DF) [tos 0x10]  (ttl 64, id 28931, len 84, bad cksum 0!)
truncated-ip - 38 bytes missing! 63.223.7.6.22 > 24.30.48.121.1850: P 4294959456:4294959488(32) ack 4294965025 win 33304 <nop,nop,timestamp[|tcp]> (DF) [tos 0x10]  (ttl 64, id 28359, len 84, bad cksum 0!)

the target sees this:

63.223.8.203 > 63.223.8.34: icmp: echo request (ttl 64, id 29269, len 84)
63.223.8.34 > 63.223.8.203: icmp: echo reply (ttl 128, id 42940, len 84, bad cksum 0!)
63.223.8.203 > 63.223.8.34: icmp: echo request (ttl 64, id 29272, len 84)
63.223.8.34 > 63.223.8.203: icmp: echo reply (ttl 128, id 42943, len 84, bad cksum 0!)
...

bad chksum is ok since its a broadcom card with enabled TXCSUM:

bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
        inet 63.223.8.34 netmask 0xffffff00 broadcast 63.223.8.255

Let's get back to the first tcpdump:

- received packets are missing data (arp is always ok!)
- the bpf sees packets that belong to another interface
$ route get 63.223.7.6
   route to: unknown63223007006.ibis7.net
destination: 63.223.7.0
       mask: 255.255.255.224
  interface: bge1

bge1 is used as main interface and the ssh session runs on this interface.
- amount of missing data reported by tcpdump for packets from other interface
  are somewhat static: 6 (very uncommon), 38, 54, 70 Bytes (all for
  ssh PSH packets)
- filtered traffic on host with ti(4):
  $ tcpdump -ntvi ti0 not arp and not host 63.223.7.6

  (outgoing request and the matching incoming reply)

63.223.8.203 > 63.223.8.133: icmp: echo request (ttl 64, id 32019, len 84, bad cksum 0!)
85:54:6e:39:6e:e2 e1:72:44:af:9e:9d d67f 60:
                         06f0 381a ff4e cd44 c5f0 bef7 d65f f8b0
                         bf1d a26b 4b19 9ce1 ddb7 85ab fef3 6f09
                         c73e c2cf 2f04 a8da 0a85 c241 c9e2

16Bytes (MAC + MAC + 08:00) + 46 Bytes data = 62 Bytes
The reply packet should be 84 Bytes long.

I reproduced this problem on two different Opteron Systems running
5.2.1-RC/amd64 with two different ti cards (same Alteon type) and different
cables.
The same configuration runs fine on FreeBSD/i386.

Hendrik

>How-To-Repeat:
	
>Fix:

	


>Release-Note:
>Audit-Trail:
>Unformatted: