8.1-PRERELEASE 10GE ix driver panics when setting mtu > 1500

[Ronald van der Pol]

On Mon, May 24, 2010 at 10:22:25 -0700, Jack Vogel wrote:

> Try the driver in HEAD, I am going to MFC that shortly anyway.

We are using 8.1-PRERELEASE now with the 2.2.0 ixgbe driver and
have several problems. The setup is:

FreeBSD 8.1-PRERELEASE ixgbe --- UTP -- Arista 7120T -- UTP -- 10G Linux
                       <==========================================

receiver: iperf -s -u -B 10.0.1.1 -l 63K -w200K 
sender: iperf -u -i1 -t 120 -c 10.0.1.1 -l 63K -w200K  -b 10000M

MTU is 9000 on all interfaces. Flow control is on.

This is /var/log/messages on the receiving FreeBSD 8.1-PRERELEASE server:

May 25 15:06:19 flits kernel: ix0: Total Packets Rcvd = 17705828
May 25 15:06:19 flits kernel: ix0: Good Packets Rcvd = 4816
May 25 15:06:19 flits kernel: ix0: Good Packets Xmtd = 64
May 25 15:06:19 flits kernel: ix0: TSO Transmissions = 0
May 25 15:06:20 flits kernel: ix0: Std Mbuf Failed = 0
May 25 15:06:20 flits kernel: ix0: Missed Packets = 17880814
May 25 15:06:20 flits kernel: ix0: Receive length errors = 0
May 25 15:06:20 flits kernel: ix0: Crc errors = 0
May 25 15:06:20 flits kernel: ix0: Driver dropped packets = 0
May 25 15:06:20 flits kernel: ix0: watchdog timeouts = 0
May 25 15:06:20 flits kernel: ix0: XON Rcvd = 0
May 25 15:06:20 flits kernel: ix0: XON Xmtd = 0
May 25 15:06:20 flits kernel: ix0: XOFF Rcvd = 0
May 25 15:06:20 flits kernel: ix0: XOFF Xmtd = 0
May 25 15:06:20 flits kernel: ix0: Total Packets Rcvd = 17888305
May 25 15:06:20 flits kernel: ix0: Good Packets Rcvd = 4817
May 25 15:06:20 flits kernel: ix0: Good Packets Xmtd = 64
May 25 15:06:20 flits kernel: ix0: TSO Transmissions = 0
May 25 15:06:22 flits kernel: ix0: Std Mbuf Failed = 0
May 25 15:06:22 flits kernel: ix0: Missed Packets = 18063137
May 25 15:06:22 flits kernel: ix0: Receive length errors = 0
May 25 15:06:22 flits kernel: ix0: Crc errors = 0
May 25 15:06:22 flits kernel: ix0: Driver dropped packets = 0
May 25 15:06:22 flits kernel: ix0: watchdog timeouts = 0
May 25 15:06:22 flits kernel: ix0: XON Rcvd = 0
May 25 15:06:22 flits kernel: ix0: XON Xmtd = 0
May 25 15:06:22 flits kernel: ix0: XOFF Rcvd = 0
May 25 15:06:22 flits kernel: ix0: XOFF Xmtd = 0
May 25 15:06:22 flits kernel: ix0: Total Packets Rcvd = 18070630
May 25 15:06:22 flits kernel: ix0: Good Packets Rcvd = 4818
May 25 15:06:22 flits kernel: ix0: Good Packets Xmtd = 64
May 25 15:06:22 flits kernel: ix0: TSO Transmissions = 0
May 25 15:06:23 flits kernel: ix0: Std Mbuf Failed = 0
May 25 15:06:23 flits kernel: ix0: Missed Packets = 18245600
May 25 15:06:23 flits kernel: ix0: Receive length errors = 0
May 25 15:06:23 flits kernel: ix0: Crc errors = 0
May 25 15:06:23 flits kernel: ix0: Driver dropped packets = 0
May 25 15:06:23 flits kernel: ix0: watchdog timeouts = 0
May 25 15:06:23 flits kernel: ix0: XON Rcvd = 0
May 25 15:06:23 flits kernel: ix0: XON Xmtd = 0

So, it's receiving 182477 packets per second, only 1 good packet and
182323 missed packets (btw, this does not count up correctly).

What could be a reason for missed packets? It seems to be a counter in
the NIC?

After half a minute or so, the ix0 interface stops responding.
ifconfig ix0 down/ifconfig ix0 up makes it reachable again.

How can I debug this further? Any other statistics?

I set ixgbe_display_debug_stats to 1 in ixgbe.c and recompiled and
installed the kernel. Is there a better way to get these statistics?

The server is a Dell PowerEdge T610.

May 25 13:41:30 flits kernel: ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.2.0> port 0xece0-0xecff mem 0xdf1a0000-0xdf1bffff,0xdf1c0000-0xdf
1fffff,0xdf19c000-0xdf19ffff irq 34 at device 0.0 on pci5
May 25 13:41:30 flits kernel: ix0: Using MSIX interrupts with 9 vectors
May 25 13:41:30 flits kernel: ix0: [ITHREAD]
May 25 13:41:30 flits last message repeated 8 times
May 25 13:41:30 flits kernel: ix0: Ethernet address: 00:1b:21:49:56:61
May 25 13:41:30 flits kernel: ix0: PCI Express Bus: Speed 2.5Gb/s Width x8

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
	options=5bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO>
	ether 00:1b:21:49:56:61
	inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
	media: Ethernet autoselect (10Gbase-SR <full-duplex>)
	status: active

	rvdp