amd64/130483: MSI must be disabled when Myricom 10Gbps Card is used on Dell PowerEdge T300 Server

[John Baldwin]

The following reply was made to PR amd64/130483; it has been noted by GNATS.

From: John Baldwin <jhb at freebsd.org>
To: freebsd-amd64 at freebsd.org
Cc: Xiuchao Wu <wuxiuchao at gmail.com>, freebsd-gnats-submit at freebsd.org
Subject: Re: amd64/130483: MSI must be disabled when Myricom 10Gbps Card is used on Dell PowerEdge T300 Server
Date: Tue, 20 Jan 2009 11:57:57 -0500

 On Tuesday 13 January 2009 12:02:07 am Xiuchao Wu wrote:
 > 
 > >Number:         130483
 > >Category:       amd64
 > >Synopsis:       MSI must be disabled when Myricom 10Gbps Card is used on 
 Dell PowerEdge T300 Server
 > >Confidential:   no
 > >Severity:       serious
 > >Priority:       medium
 > >Responsible:    freebsd-amd64
 > >State:          open
 > >Quarter:        
 > >Keywords:       
 > >Date-Required:
 > >Class:          sw-bug
 > >Submitter-Id:   current-users
 > >Arrival-Date:   Tue Jan 13 05:10:01 UTC 2009
 > >Closed-Date:
 > >Last-Modified:
 > >Originator:     Xiuchao Wu
 > >Release:        FreeBSD 7.0
 > >Organization:
 > National University of Singapore
 > >Environment:
 > FreeBSD 7.0/7.1 Release
 > >Description:
 > Computer: Dell PowerEdge T300 Server
 > NICs: Myricom 10G-PCIE-8AL-C
 > OS: FreeBSD 7.0/7.1 Release
 > 
 > After I rebuild the kernel (to include NIC driver "mxge") and reboot, kernel 
 panic occurs and the computer is automatically rebooted. After contacting 
 with engineers of Myricom, they suggest to disable message signaled interrupt 
 (MSI) in /boot/loader.conf and these NICs can work now.
 > "
 > hw.pci.enable_msix=0
 > hw.pci.enable_msi=0
 > "
 > 
 > However, MSI is really very important for high speed data transmission. Huge 
 number of packets generate many interruptions. Considering that Fedora 9 
 works well on the same computer, it may be a bug of FreeBSD. Below is the 
 screen when kernel panic occurs.
 > 
 > 
 > ......
 > p4tcc3: <CPU Frequency Thermal Control> on cpu3
 > pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
 > pci0: <ACPI PCI bus> on pcib0
 > pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
 > pci3: <ACPI PCI bus> on pcib1
 > pcib2: <ACPI PCI-PCI bridge> at device 3.0 on pci0
 > pci4: <ACPI PCI bus> on pcib2
 > pcib3: <ACPI PCI-PCI bridge> at device 4.0 on pci0
 > pci5: <ACPI PCI bus> on pcib3
 > mxge0: <Myri10G-PCIE-8A> mem 0xd8000000-0xd8ffffff, 0xdfa00000-0xdfafffff 
 irq 16 at device 0.0 on pci5
 > panic: nexus_add_irq: failed
 > .....
 
 How many MSI messages is mxge0 attempting to add and how many other devices in 
 your system are using MSI?  Getting a full copy of a verbose dmesg would be 
 most helpful.  'pciconf -lc' might also help answer the first question.
 
 -- 
 John Baldwin